fix(autoindex): --no-graph durable path indexed zero documents for every code file - #316
Merged
Merged
Conversation
…napshot blob path (#294) On the durable-generation path (--no-graph) every source-code file was silently dropped: walked, sniffed correctly as family code, counted, and then prepared as ZERO documents. The generation still committed and the run reported success (exit 3), so the loss was invisible - the #204 class of accepted-and-ignored input. Root cause: prepare_artifact sniffs with the logical path (sniff_with_name(snapshot_blob, &source.path)) but extracts with only the blob path. code::extract was the one family extractor that recovered a parameter - the language, via path.extension() - from its content path instead of from Sniffed. Snapshot blobs are content-addressed and extensionless (sync-snapshots/<tx>/blobs/00000000), so the registry lookup never matched and every code file, in every language, became junk. Prose/CSV/JSON files were unaffected, which is what made the symptom look format-specific. Before: preparing a dir with app.py -> prepared.records == 0, prepared.junk == 1, empty prepared/*.ndjson. After: 1 record with language, title app.py, defs 'function alpha_helper'. Fix: Sniffed now carries logical_name, set by sniff_with_name, and code::extract resolves both the grammar and the document title from it (falling back to the content path, which keeps the legacy path byte-identical: there logical == content). Title matters too - without it the fixed durable path would have indexed code docs titled by blob ordinal '00000000'. Files: crates/xerj-autoindex/src/sniff.rs (logical_name), extract/code.rs + extract/mod.rs (signature + lookup), sync_executor.rs + extract/code.rs (regression tests that fail on the parent commit with (records, junk) == (0, 1)). Fixes #294
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On the durable-generation path (
--no-graph), every source-code file was silently dropped: walked, sniffed correctly asfamily: code, counted, and then prepared as zero documents. The generation still committed and the run reported success (exit 3), so the loss was invisible — the #204 class of accepted-and-ignored input.Root cause
prepare_artifactsniffs with the logical path (sniff_with_name(snapshot_blob, &source.path)) but extracts with only the blob path.code::extractwas the one family extractor that recovered a parameter — the language, viapath.extension()— from its content path instead of fromSniffed. Snapshot blobs are content-addressed and extensionless (sync-snapshots/<tx>/blobs/00000000), so the registry lookup never matched and every code file, in every language, became junk.Fix
Sniffednow carrieslogical_name, set bysniff_with_name, andcode::extractresolves both the grammar and the document title from it, falling back to the content path (which keeps the legacy path byte-identical: there logical == content). The title matters too — without it the fixed durable path would have indexed every code doc titled by blob ordinal00000000.Reproduction (run at HEAD, 803c85b)
The new regression test
preparation_extracts_code_from_extensionless_snapshot_blobsfails on the parent commit exactly as the issue describes:End-to-end with this fix (release binary, throwaway server on :9483), the issue's own repro corpus (
app.py+notes.txt) under--no-graph:and the issue's real-tree case (
crates/xerj-autoindex/src, 53.rsfiles) goes from0 records live/count: 0to53 records live/count: 53.Tests
sync_executor::tests::preparation_extracts_code_from_extensionless_snapshot_blobs— the durable-path regression the issue asked for (fails before the fix).extract::code::tests::extracts_from_extensionless_blob_via_sniffed_logical_name— the same contract at the extractor's own boundary.Local gate
cargo test -p xerj-autoindex --lib: 530 passed / 0 failedcargo fmt --all --check: cleancargo clippy -p xerj-autoindex --all-targets -- -D warnings: cleanNoted, out of scope
Other extractors (
txt,html,pdf,docx) use their content path only as a fallback title when the document has no intrinsic one, so on the durable path a titleless file can still get a blob-ordinal title. Cosmetic, no document loss — worth a small follow-up that threadslogical_nameinto those fallbacks.Fixes #294
🤖 Generated with Claude Code