Skip to content

Tags

Tags

v0.1.2026081803

Toggle v0.1.2026081803's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
refactor(index): unified byte-identity format, zero-copy readers (cli…

…ce-io#610)

## What changed

Redesigns the index system around one rule: a file's index rows have
exactly one encoding — the shard blob — produced as final bytes by the
worker and passed as opaque buffers everywhere after.

**Format layer**
- A shard blob is canonical and deterministic: `xxh3_64(bytes)` is the
variant's identity, so equal rows always produce equal bytes.
`validate()` enforces every self-describing choice (packed-range tier,
symbol-id width, ASCII content omission, mask tier) as a strict function
of the data, and rejects non-canonical or hostile blobs at load.
- Row storage compresses hard: packed `u32` ranges (`(begin << 8) |
length` with an escape side table and a wide tier past 16 MB),
`u8/u16/u32` symbol id tiers, and a persisted per-line length table
replacing the runtime `line_starts` derivation.
- Pure-ASCII content is no longer stored (byte offsets already are
UTF-16 offsets); non-ASCII files keep their text in the blob. Position
mapping goes through a new `IndexedLineMap` that does line-table
arithmetic for the omitted-ASCII case. Preview queries (definition text,
reference context) re-read the disk under a `content_hash` check and
degrade to positions-only when the file moved on.

**Envelope layer**
- The per-TU product is a single envelope: `build_tu_index(unit) →
bytes`, carrying the include graph, the TU symbol table, and one
self-contained blob per file (repeated inclusions union into one blob).
The in-memory `TUIndex` type is gone; the name now denotes the zero-copy
reader over envelope bytes, mirroring `Shard`.
- A preamble is just a truncated TU: `build_preamble_index` emits the
same envelope plus preamble-only fields (prefix identity, document
links, inactive regions, open conditionals), and `.pch.idx` stores it
verbatim. The `PreambleIndex` class and its separate format/version are
deleted; loading gates on `from_buffer` + per-section blob verification.

**Server**
- The master merges by byte passthrough: a known variant hash is pure
bookkeeping (91% of merges in practice), a new file or content
generation installs the worker's bytes verbatim, and only a genuinely
new variant of the same generation runs the masked k-way `merge_shards`.
The old decode→re-encode pipeline and the disk-content arbitration at
merge time are gone — a blob is its own generation, and freshness gating
owns disk drift.
- Sessions hold one owning envelope reader; the separate in-memory file
index and symbol table fields are gone.

Also folds in two earlier fixes on this branch base: duplicate global
symbol hashes are rejected at load, and rebuilds re-enqueue TUs whose
pinned variants were discarded.

On a 387-TU workload the old wire format measured 6.75 GB against 880 MB
for the columnar encoding this PR makes the only format (~7.7x), and the
master no longer decodes or re-encodes anything on the common path.

## Tests

All four suites pass locally on both `RelWithDebInfo` and Debug (ASan +
assertions): unit (1213), integration (348), smoke (3), snap (395,
byte-stable across the format change).

The unit suites were migrated to the new readers with coverage extended
rather than moved: deterministic-encoding-as-identity, a battery of
canonical-form and hostile-blob rejections (wrong tiers, stored ASCII
content, mask/escape/line-table mismatches, corrupt section blobs, stale
format versions, out-of-range path ids), k-way merges with mask
re-encoding and dead-variant compaction, preamble prefix-identity
semantics, ASCII preview serve-and-degrade from disk, and direct
`IndexedLineMap` arithmetic tests. Tests pinning deleted behaviors
(merge-time disk arbitration, envelope re-serialization) were rewritten
to pin their replacements: merges keyed purely by content generation,
and byte-passthrough persistence.