v0.7.0 foundation: refuse redirected bundles and bind sidecars to their dataset - #49
Merged
Conversation
Every bridge check opens a path inside the inspected bundle, and an open follows whatever that path resolves to. A bundle could therefore ship an entry pointing outside itself and have the doctor hash that file and search its content for credentials, reporting the detector category back. Reproduced on Windows with a junction: the hash check walked into the outside directory and the metadata privacy scan reported semantic_secret_key against a file the bundle did not contain. preflight_bundle_tree walks with lstat only and refuses symlinks, reparse points, non-regular files, entries resolving outside the root, and trees over a bounded file count, byte total or depth. It runs before anything opens a file, and a refused bundle reports every check as not_inspected rather than failed: the checks did not run, so they cannot claim a finding.
The portable metadata scan is bounded: it skips files over a size limit, stops at a total byte budget, and gives up on anything it cannot decode or stat. All of those still produced heuristic_passed, so "found nothing" was reported for an inspection that never looked at part of the bundle. Statuses are now heuristic_passed_full, heuristic_failed, heuristic_incomplete and not_inspected, each gap is recorded with its file and reason, and the list is bounded. An incomplete scan is reported but not failed by default; --require-complete-metadata-scan refuses it. Also extend the text gate to the Latin-1 range. A mis-decoded multiplication sign is one character, not the three-byte CJK leaders the v0.6.3 gate looked for, so "State [x] Supervision" and "1.63[x]" survived in CHANGELOG.md through the release. Both are repaired, and the tensor-to-float warnings in test_chunked_equivalence.py are gone.
Sidecars are keyed by row index, so one copied beside a different Parquet file silently means different rows. New sidecars are schema version 2 and carry the digest and row count of the dataset they are published with; reading a v2 sidecar without that binding fails closed. Sidecars published by 0.6.0-0.6.3 declare version 1 and still read. Conversion streams its source over a long period and then publishes a report claiming a source_sha256. If the file is replaced in between, that claim describes bytes the conversion never read while the output rows come from the old ones. Source identity is captured up front and re-checked, digest included, before anything is published. Output-to-source row provenance was one map entry per accepted row, built even for a complete conversion where it is the identity function. It is now contiguous runs -- one run plus at most one per rejection -- with resolve_source_row() to read it. Extension deduplication detail is bounded with an exact total kept alongside. The quality record separated: a local commit and platform where the GPU and Windows paths actually ran, and the exact release commit CI validated. The single measured_commit implied a Windows RTX 4080 count had been taken on the squash commit that was published, which it never was. The pinned bridge workflow filtered on paths at the trigger, so it did not run at all on an unrelated pull request; requiring that check blocked the v0.6.4 state sync permanently. The filter moved to the expensive job and "pinned bridge gate" always reports, reporting not_applicable when no bridge path changed. Its tag trigger was also v0.6.*, which would have skipped the smoke on the v0.7.0 release tag. Development version advances to 0.7.0.dev0. There is no v0.6.4 release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase A of v0.7.0: close the bounded trust and scalability gaps that could
affect benchmark artifacts, before any external benchmark code lands. No
scientific result changes; all eleven frozen artifacts stay byte-identical.
Development version advances
0.6.4.dev0→0.7.0.dev0. There is no v0.6.4release — this hardening ships as part of the external alignment study.
The one that mattered
bridge doctorread files outside the bundle it was inspecting.Every check opens a path inside the bundle, and an open follows whatever that
path resolves to. Reproduced on Windows with a junction:
A bundle you were handed could therefore have your files hashed and their
contents searched for credentials, with the detector category reported back —
an information-disclosure primitive reachable by running a diagnosis.
preflight_bundle_treewalks withlstatonly and refuses file and directorysymlinks, Windows junctions and other reparse points, devices/sockets/FIFOs,
entries resolving outside the root, and trees over a bounded file count, byte
total or depth. It runs before anything opens a file. A refused bundle reports
every check as
not_inspectedrather thanfailed— those checks did not run,so they are not entitled to a finding.
Also in this PR
Privacy scan completeness. The scan skips oversized files, stops at a byte
budget and gives up on anything it cannot decode — and still reported
heuristic_passed. Nowheuristic_passed_full/heuristic_failed/heuristic_incomplete/not_inspected, each gap recorded with file andreason, bounded.
--require-complete-metadata-scanrefuses an incompleteinspection.
Sidecar v2. Sidecars are keyed by row index, so one copied beside a
different Parquet file silently means different rows. v2 binds the digest and
row count of the dataset it is published with. v1 sidecars from 0.6.0–0.6.3
still read.
Source mutation. Conversion captures source identity up front and
re-checks it — stat fields and content digest — before publishing, so a report
can never describe bytes the conversion did not read.
Bounded metadata. Output-to-source row provenance is contiguous runs
instead of one entry per row (it was built even for complete conversions,
where it is the identity function).
resolve_source_row()reads it.Deduplication detail is bounded with an exact total alongside.
Quality provenance. Split into the locally measured commit/platform, where
the GPU and Windows paths actually run, and the exact release commit CI
validated. One
measured_commitimplied a Windows RTX 4080 count had beentaken on the published squash commit; it never was.
Two CI defects. The pinned bridge workflow filtered on paths at the
trigger, so it did not run at all on unrelated PRs — and requiring a check that
never reports blocked the v0.6.4 state sync permanently until it was removed
from the required list. The filter moved to the expensive job;
pinned bridge gatealways reports,not_applicablewhen no bridge path changed, so it issafe to require again. Its tag trigger was also
v0.6.*, which would havesilently skipped the smoke on the
v0.7.0release tag.Text integrity. A mis-decoded
×is a single character, not thethree-byte CJK leaders the v0.6.3 gate looked for, so two CHANGELOG lines
survived the release. Latin-1 range added; both repaired. The tensor-to-float
warnings in
test_chunked_equivalence.pyare gone.Validation
1865 CPU tests pass (35 new), 6 skipped for Windows privileges/POSIX-only
paths. Ruff, format, mypy, text integrity and
release_state.py --checkclean.Frozen artifact digests unchanged.