Repository navigation
Tags
Tags
refactor(index): unified byte-identity format, zero-copy readers (cli… …ce-io#610) ## What changed Redesigns the index system around one rule: a file's index rows have exactly one encoding — the shard blob — produced as final bytes by the worker and passed as opaque buffers everywhere after. **Format layer** - A shard blob is canonical and deterministic: `xxh3_64(bytes)` is the variant's identity, so equal rows always produce equal bytes. `validate()` enforces every self-describing choice (packed-range tier, symbol-id width, ASCII content omission, mask tier) as a strict function of the data, and rejects non-canonical or hostile blobs at load. - Row storage compresses hard: packed `u32` ranges (`(begin << 8) | length` with an escape side table and a wide tier past 16 MB), `u8/u16/u32` symbol id tiers, and a persisted per-line length table replacing the runtime `line_starts` derivation. - Pure-ASCII content is no longer stored (byte offsets already are UTF-16 offsets); non-ASCII files keep their text in the blob. Position mapping goes through a new `IndexedLineMap` that does line-table arithmetic for the omitted-ASCII case. Preview queries (definition text, reference context) re-read the disk under a `content_hash` check and degrade to positions-only when the file moved on. **Envelope layer** - The per-TU product is a single envelope: `build_tu_index(unit) → bytes`, carrying the include graph, the TU symbol table, and one self-contained blob per file (repeated inclusions union into one blob). The in-memory `TUIndex` type is gone; the name now denotes the zero-copy reader over envelope bytes, mirroring `Shard`. - A preamble is just a truncated TU: `build_preamble_index` emits the same envelope plus preamble-only fields (prefix identity, document links, inactive regions, open conditionals), and `.pch.idx` stores it verbatim. The `PreambleIndex` class and its separate format/version are deleted; loading gates on `from_buffer` + per-section blob verification. **Server** - The master merges by byte passthrough: a known variant hash is pure bookkeeping (91% of merges in practice), a new file or content generation installs the worker's bytes verbatim, and only a genuinely new variant of the same generation runs the masked k-way `merge_shards`. The old decode→re-encode pipeline and the disk-content arbitration at merge time are gone — a blob is its own generation, and freshness gating owns disk drift. - Sessions hold one owning envelope reader; the separate in-memory file index and symbol table fields are gone. Also folds in two earlier fixes on this branch base: duplicate global symbol hashes are rejected at load, and rebuilds re-enqueue TUs whose pinned variants were discarded. On a 387-TU workload the old wire format measured 6.75 GB against 880 MB for the columnar encoding this PR makes the only format (~7.7x), and the master no longer decodes or re-encodes anything on the common path. ## Tests All four suites pass locally on both `RelWithDebInfo` and Debug (ASan + assertions): unit (1213), integration (348), smoke (3), snap (395, byte-stable across the format change). The unit suites were migrated to the new readers with coverage extended rather than moved: deterministic-encoding-as-identity, a battery of canonical-form and hostile-blob rejections (wrong tiers, stored ASCII content, mask/escape/line-table mismatches, corrupt section blobs, stale format versions, out-of-range path ids), k-way merges with mask re-encoding and dead-variant compaction, preamble prefix-identity semantics, ASCII preview serve-and-degrade from disk, and direct `IndexedLineMap` arithmetic tests. Tests pinning deleted behaviors (merge-time disk arbitration, envelope re-serialization) were rewritten to pin their replacements: merges keyed purely by content generation, and byte-passthrough persistence.