[Storehouse] 007 Payloadless Checkpoint (v7)#8578
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| // Read the V6 checkpoint fully — the V6 reader already reads the 16 subtrie | ||
| // part files concurrently. The resulting tries share sub-tries via Go pointer | ||
| // identity, which lets FromV6Tries memoize and avoid redundant conversion. | ||
| v6Tries, err := LoadCheckpoint(v6Header, logger) |
There was a problem hiding this comment.
This loads the entire checkpoint in memory, which requires a lot of memory. An improvement would be streaming the file, iterate each node, if it's a leaf node, then convert into payload less. This could save lots of memory. Skipped for now, because that requires additional code, can be implemented later.
| return nil, nil | ||
| } | ||
| if index > totalSubTrieNodeCount { | ||
| nodePos := index - totalSubTrieNodeCount |
There was a problem hiding this comment.
This part is same logic as v6 (getNodeByIndex) but slightly different in style, where v6 extracted it into getTopNodeByIndex, which adds little value. Instead, the logic here for getPayloadlessTopNodeByIndex is inlined.
| @@ -0,0 +1,421 @@ | |||
| package wal | |||
There was a problem hiding this comment.
vimdiff ledger/complete/wal/checkpoint_v6_reader.go ledger/complete/wal/checkpoint_v7_reader.go
This file is similar to v6_reader, except some minimal coding style changes. Some read functions are not copied over but reused, such as readCheckpointHeader, allPartFileExist, readSubTriesFooter etc
| @@ -0,0 +1,484 @@ | |||
| package wal | |||
There was a problem hiding this comment.
vimdiff ledger/complete/wal/checkpoint_v6_writer.go ledger/complete/wal/checkpoint_v7_writer.go
v7 is similar to v6 except some comments changes.
| @@ -0,0 +1,372 @@ | |||
| package payloadless | |||
There was a problem hiding this comment.
mirrored from ledger/complete/mtrie/flattener/encoding.go and ledger/complete/mtrie/flattener/iterator.go
| encNodeIndexSize = 8 | ||
| encRegCountSize = 8 | ||
|
|
||
| encLeafHashFlagSize = 1 |
There was a problem hiding this comment.
encRegSizeSize and encPayloadLengthSize in v6 checkpoint node (see ledger/complete/mtrie/flattener/encoding.go) is replaced by encLeafHashFlagSize in payloadless
| return nil, fmt.Errorf("cannot read leaf hash flag: %w", err) | ||
| } | ||
|
|
||
| flag := scratch[0] |
There was a problem hiding this comment.
this part is different from readPayloadFromReader
|
|
||
| type EncodedTrie struct { | ||
| RootIndex uint64 | ||
| RegCount uint64 |
| encLeafHashFlagSize + | ||
| encLeafHashSize |
There was a problem hiding this comment.
encPayloadLengthSize + encPayloadSize is replaced by encLeafHashFlagSize + encLeafHashSize in payloadless
| copy(buf[pos:], path[:]) | ||
| pos += encPathSize | ||
|
|
||
| // Encode leaf hash flag (1 byte) and optional leaf hash (0 or 32 bytes) |
There was a problem hiding this comment.
this part is different from full trie node's encodeLeafNode
7b14692 to
2625248
Compare
| } | ||
|
|
||
| lastCheckpointNum, err := c.checkpointer.LatestCheckpoint() | ||
| lastCheckpointNum, err := c.checkpointer.LatestCheckpointV6() |
There was a problem hiding this comment.
The compactor will eventually need to support both V6 and V7, but this PR only supports V6. To avoid ambiguity, I renamed the function to LatestCheckpointV6.
Uh oh!
There was an error while loading. Please reload this page.