Repository navigation
M1: the record plane and the .yo file #2
Description
Activity
Reopened to audit this checklist against the code rather than against memory. Ten of the eleven boxes are ticked now, two of them because #32 closed them and eight because the code was already there and nobody had come back to mark it.
The seven scope boxes, and where each one lives
The hybrid log with three regions and per shard epochs.
Regionincrates/yo-record/src/log.rswithStable,ReadOnlyandMutable,mutable_fractionandregion_ofdeciding which is which. The epoch is per shard on the log itself and is stamped into every page header. The arena side reclamation epochs are separate and are incrates/yo-shard/src/epoch.rs, one cache line per shard, one relaxed store per batch rather than per operation.Group commit and the four durability modes.
Durabilityincrates/yo-record/src/log.rswithNone,Os,GroupandSync. Group is the default.Recovery from the last good superblock plus a log replay.
Yo::openreads both slots andpickchooses the good one, which is the whole point of the dual superblock. Replay iscrates/yo-record/src/replay.rs, and it is short because the append path did the work: the length is written last, the four bytes past the tail are always zero, and every record carries its own checksum, so the three ways a log can end all look the same from the reader's side.F2 style compaction with dead byte accounting.
crates/yo-record/src/compact.rs.page_wants_compactionreads the page header's own dead fraction againstCOMPACT_TRIGGER, andcompact_uptoruns against a byte budget rather than to completion.The format. Two superblock slots at 0 and 16384,
SUPERBLOCK_LENbeing 16384 throughoutcrates/yo-format/src/superblock.rs. The root flip and its ordering iscrates/yo-file/src/file.rs, and the ordering is the guarantee: data, then the slot, then the slot's own sync, so a checkpoint can be behind the truth and never ahead of it. Page headers arecrates/yo-format/src/page.rs. The collection catalogue iscrates/yo-format/src/catalog.rs.yo check.crates/yo-cli, built on the independent reader and not on the engine, which is the only way a checker can tell you the engine is wrong.The independent minimal reader.
crates/yo-reader, 1420 lines, written from06and07rather than from the engine. Its tests are the one place in the tree where depending on the engine is allowed, because the agreement between the two is the deliverable.The exit gate
The 10 GB open row is measured, not asserted.
cargo bench -p yo-file --bench opengives 263 microseconds at 320 regions, which is ten gigabytes, against a budget of 100 milliseconds. It is not close and it should not be. Open reads two superblock slots and 320 region headers of 32 bytes each and nothing else, so what the number reflects is the design rather than the disk. The fixture is sparse, which is honest for what is being measured, because open never touches a record.The 100k fault row now runs in CI.
ci.ymlruns twenty thousand on every push, which is the right number for a per push job and was never the gate. The gate number runs nightly indeepacross five shapes with seeds that do not overlap, so a failure names one run and one seed and reproduces from the command line. The run on #32 was a hundred thousand trials in 44.4 seconds, all five shapes clean.The reader agreement row was a claim nobody had run, because the gate says every file the fuzzer produces and there was no fuzzer producing files.
fuzz/fuzz_targets/file.rsis that fuzzer. It builds a real.yofile on disk across up to four shards from fuzzer chosen keys, values, TTLs and tombstones, and checks three separate claims: the engine's own replay sees what went in, the independent reader sees what went in, and the two agree on the superblock and the checkpoint entries field for field. Three claims rather than one, so a failure says which side is wrong instead of leaving you with a disagreement and no suspect.The one row that stays open
200k or more durable commits per second in group mode on NVMe with SQPoll. This cannot be attempted yet and will not be ticked on a substitute. SQPoll is io_uring, there is no io_uring sink in the tree, and no machine we have access to meets the gate box definition in
bench/00section 7. The reactor in M2 is where the io_uring path arrives, and the number gets measured then, on a box that qualifies.That is the only thing left in M1. Leaving it open on a milestone whose other ten boxes are done is deliberate: a performance row that gets ticked on a machine that does not qualify is exactly the kind of number the confound registry exists to stop, and aki's four wrong numbers all came from methodology rather than from code.
#34 is merged, so the ring is no longer a crate nobody writes through.
LogFile::use_ringputs a shard's log on it, the shard hands its bytes over instead of stopping for apwriteand anfdatasync, and a commit waiting on durability parks on an address instead of on a syscall.Three rules had to hold before any of that is safe, and each has a test with its name on it. Bytes have to outlive the call, so every write is staged into a pooled buffer the writer owns. Two writes to one page cannot be out at once, because io_uring does not order submissions and every flush rewrites the page header, so the older header landing last is a page that lies about how many records it has. A sync means nothing until the writes it covers have landed, because a queued fsync runs in parallel with the writes queued ahead of it, so
syncrecords the address it has to cover and the fsync goes out frompollonce the last write is back.Two durability modes were quietly wrong under an asynchronous sink and are fixed in the same PR.
syncmode used to sync and reply, which is a reply before the fsync when the sink is asynchronous, and it now parks the caller on the address instead.osmode promised the operating system had the bytes while they sat in the submission queue, and a page boundary now submits as well as writes.The 200 thousand durable commits row stays open, and this is the first update where the reason is hardware rather than missing code. The path now runs end to end on real io_uring, checked on server3 with kernel 6.8, and it lands the same bytes as the synchronous path. What it does not have is a device worth quoting. server1 and server3 are both QEMU virtio disks on shared VPS hosts, and a benchmark row there comes back with error bars around plus or minus 30 percent, wide enough that the ring and
pwritecolumns cannot be told apart in either direction. gamingpc has the fastest disk of the four machines and runs Windows, where there is no io_uring at all.So the row needs the box described in
bench/00section 7, which is NVMe with SQPoll and nothing else running on it, and none of server1, server2, server3 or gamingpc is that box. The number will come from a run on one, not from an argument here. Everything else in M1 is ticked.- added a commit that references this issue
on Sep 6, 2026
The record plane and the file, fronted because this is the thing the lineage has never built. aki spent two years on data types and never started its file, and the durability story was still a document when the project moved on. Building the file first means every type after this is written against a real record plane.
Scope
.yoformat: dual superblock at offset 0 and 16384, the atomic root flip, log page headers, the collection catalogueyo checkExit gate
Notes
The io_uring ladder from the VLDB 2026 study is the reason the commit target is set where it is. The measured rungs go 16.5, 19.0, 183.0 with async exec, 216.0, 238.0 with registered buffers, 300.0, 376.0 with IOPOLL and 546.5 with SQPoll, in thousands of transactions per second. Wrapping io_uring in Tokio erases most of that, which is why there is no async in the engine.
The independent reader is not a nice to have. It is the only test that can catch the format drifting away from what is written down, and it is what makes a yanked release still readable.
Spec:
16-implementation-plan.mdM1,06-record-plane.md,07-yo-file-format.md.