feat(tools): add logdump tool to scan and inspect raft log (pebbledb based)

[sitano]

During roll out of a new version one replica paniced on a data consistency assert. So we had to understand what led to that object state. We needed a tool to inspect in production environment what RSM operations led to the node crash.

Our objects that failed have an epoch version that corresponds to the Raft commit index. So we were able to disect what Raft commit pushed this specific object version to the Raft SM. Then, by looking around this specific commit with this tool (logdump) we were able to identify specific subset of Raft commits (proposals) that were differently executed by the leader and a replica. And that allowed us to bisect the code changes that led to the bug. And all that was only possible because we were able to scan over the Raft log with this tool.

The patch introduces a new tool /tools/logdump that is capable of describing the current raft state and scan over the commit log (Currently based on PebbleDB). It can also bypass LOCK file flock() lease. It opens PebbleDB in read-only mode. It touches no Dragonboat code. The tool can decode Raft log entries as well as it can abstractly decode Entry.Cmd payload if it is encoded in Protobuf.

$ ./logdump describe ~/raft.1/node-1/dragonboat-1

MANIFEST: Address=10.32.179.88:62901, BinVer=100, HardHash=0, LogdbType=sharded-pebble, Hostname=ieprisyazhniy-dev, DeploymentId=1, StepWorkerCount=16, LogdbShardCount=16, MaxSessionCount=4096, EntryBatchSize=48, AddressByNodeHostId=false
CLUSTER   NODE   TERM   VOTE   COMMIT   ENTRIES   RANGE              SNAPSHOTS   BOOTSTRAP
1         1      20     1      138323   8311      [130013..138323]   3           join=false, 1 addrs, sm=OnDiskStateMachine
2         1      14     2      65289    5284      [60006..65289]     3           join=false, 1 addrs, sm=OnDiskStateMachine
3         1      16     1      65255    5250      [60006..65255]     3           join=false, 1 addrs, sm=OnDiskStateMachine
4         1      21     2      65254    5249      [60006..65254]     3           join=false, 1 addrs, sm=OnDiskStateMachine
5         1      9      1      65249    5244      [60006..65249]     3           join=false, 1 addrs, sm=OnDiskStateMachine
6         1      15     1      65253    5248      [60006..65253]     3           join=false, 1 addrs, sm=OnDiskStateMachine
7         1      12     2      65254    5249      [60006..65254]     3           join=false, 1 addrs, sm=OnDiskStateMachine
8         1      14     2      65256    5251      [60006..65256]     3           join=false, 1 addrs, sm=OnDiskStateMachine

$ ./logdump scan ~/raft.1/node-1/dragonboat-1 --cluster 1 --node 1 --limit 10
── state: cluster=1 node=1 ──
Term: 20, Vote: 1, Commit: 138323

── snapshots ──
INDEX    TERM   MEMBERS
110011   5      1 addrs
120012   5      1 addrs
130013   13     2 addrs

── entries ──
INDEX   TERM   TYPE              KEY                  CLIENT               SERIES   RESPTO   CMD
30013   5      ── EncodedEntry   0xa134bc077ad7f458   0x42ff9364aa44cea2   0        0        000a144a120a... (522 B)
30014   5      ── EncodedEntry   0x1e8a5f4ae48b2c6c   0x42ff9364aa44cea2   0        0        000a98044295... (1039 B)
30015   5      ── EncodedEntry   0x5a47c7aff4c8a919   0x42ff9364aa44cea2   0        0        000a144a120a... (522 B)
30016   5      ── EncodedEntry   0x8806b63e174d9b6c   0x42ff9364aa44cea2   0        0        000a98044295... (1039 B)
30017   5      ── EncodedEntry   0x3da027462db44f57   0x42ff9364aa44cea2   0        0        000a144a120a... (522 B)
30018   5      ── EncodedEntry   0xd6004714bdf7a331   0x42ff9364aa44cea2   0        0        000a98044295... (1039 B)
30019   5      ── EncodedEntry   0x42a4602926019a36   0x42ff9364aa44cea2   0        0        000a144a120a... (522 B)
30020   5      ── EncodedEntry   0x47fc08b20c1f73c8   0x42ff9364aa44cea2   0        0        000a98044295... (1039 B)
30021   5      ── EncodedEntry   0xc1f770be4b966260   0x42ff9364aa44cea2   0        0        000a144a120a... (522 B)
30022   5      ── EncodedEntry   0x91185642ccb77b89   0x42ff9364aa44cea2   0        0        000a98044295... (1039 B)

$ ./logdump scan ~/raft.1/node-1/dragonboat-1 --cluster 3 --node 1 --index 65000 --format json
{"cluster_id":3,"node_id":1,"state":{"term":16,"vote":1,"commit":65255},"snapshots":[{"index":40004,"term":5,"members":1},{"index":50005,"term":5,"members":1},{"index":60006,"term":9,"members":2}],"entries":[{"index":65000,"term":16,"type":"EncodedEntry","key":15160884136415968393,"client_id":11516911269575256109,"series_id":0,"responded_to":0,"cmd":"000a9...<<hex>>"}],"from":65000,"to":65001,"limit":100,"first_index":60006,"last_index":65255,"entry_count":5250}

$ ./logdump scan ~/raft.1/node-1/dragonboat-1 --cluster 3 --node 1 --index 65000 --format json -decode-entry-header
{"cluster_id":3,"node_id":1,"state":{"term":16,"vote":1,"commit":65255},"snapshots":[{"index":40004,"term":5,"members":1},{"index":50005,"term":5,"members":1},{"index":60006,"term":9,"members":2}],"entries":[{"index":65000,"term":16,"type":"EncodedEntry","key":15160884136415968393,"client_id":11516911269575256109,"series_id":0,"responded_to":0,"cmd":{"kind":"encoded_entry","encoded_entry":{"version":0,"compression":"none","size":1038,"payload_hex":"0a9...<<hex>>"}}}],"from":65000,"to":65001,"limit":100,"first_index":60006,"last_index":65255,"entry_count":5250}

$ ./logdump scan ~/raft.1/node-1/dragonboat-1 --cluster 3 --node 1 --index 65000 --format json -decode-entry-header -decode-entry-cmd
{"cluster_id":3,"node_id":1,"state":{"term":16,"vote":1,"commit":65255},"snapshots":[{"index":40004,"term":5,"members":1},{"index":50005,"term":5,"members":1},{"index":60006,"term":9,"members":2}],"entries":[{"index":65000,"term":16,"type":"EncodedEntry","key":15160884136415968393,"client_id":11516911269575256109,"series_id":0,"responded_to":0,"cmd":{"kind":"encoded_entry","encoded_entry":{"version":0,"compression":"none","size":1038,"proto_message":[{"id":1,"value":[{"id":8,"value":[{"id":1,"value":"module.raft.updateCacheProc"},{"id":2,"type_url":"type.googleapis.com/internalpb.ArrayOfAny","value":[{"id":1,"type_url":"type.googleapis.com/...}],"from":65000,"to":65001,"limit":100,"first_index":60006,"last_index":65255,"entry_count":5250}

[kevburnsjr]

The PR description explains what the code does but it does not explain why this tool was created.
What is the problem this tool solves for you?

[sitano]

@kevburnsjr good point! thank you for pointing this out. I will update the description. We needed a tool to inspect in production environment what RSM operations led to the node crash. During roll out of a new version one replica paniced on a data consistency assert. So we had to understand what led to that object state. Our objects that failed had an epoch version that corresponded to the Raft commit index. So we were able to disect what Raft commit pushed this specific object version to the Raft SM. Then, by looking around this specific commit with this tool (logdump) we were able to identify specific subset of Raft commits (proposals) that were differently executed by the leader and a replica. And that allowed us to bisect the code changes that led to the bug. And all that was only possible because we were able to scan over the Raft log with this tool. So I thought other people would appreciate having it too for debugging different issues.