Skip to content

v0.7.0 foundation: refuse redirected bundles and bind sidecars to their dataset - #49

Merged
DaoyuanLi2816 merged 6 commits into
mainfrom
v0.7.0-foundation
Aug 8, 2026
Merged

v0.7.0 foundation: refuse redirected bundles and bind sidecars to their dataset#49
DaoyuanLi2816 merged 6 commits into
mainfrom
v0.7.0-foundation

Conversation

@DaoyuanLi2816

Copy link
Copy Markdown
Owner

Phase A of v0.7.0: close the bounded trust and scalability gaps that could
affect benchmark artifacts, before any external benchmark code lands. No
scientific result changes; all eleven frozen artifacts stay byte-identical.

Development version advances 0.6.4.dev00.7.0.dev0. There is no v0.6.4
release — this hardening ships as part of the external alignment study.

The one that mattered

bridge doctor read files outside the bundle it was inspecting.

Every check opens a path inside the bundle, and an open follows whatever that
path resolves to. Reproduced on Windows with a junction:

files the hash check walked into : 1
outside file reached by hashing  : ['data/credentials.json']
privacy scan status              : heuristic_failed
privacy detector categories      : ['semantic_secret_key']

A bundle you were handed could therefore have your files hashed and their
contents searched for credentials, with the detector category reported back —
an information-disclosure primitive reachable by running a diagnosis.

preflight_bundle_tree walks with lstat only and refuses file and directory
symlinks, Windows junctions and other reparse points, devices/sockets/FIFOs,
entries resolving outside the root, and trees over a bounded file count, byte
total or depth. It runs before anything opens a file. A refused bundle reports
every check as not_inspected rather than failed — those checks did not run,
so they are not entitled to a finding.

Also in this PR

Privacy scan completeness. The scan skips oversized files, stops at a byte
budget and gives up on anything it cannot decode — and still reported
heuristic_passed. Now heuristic_passed_full / heuristic_failed /
heuristic_incomplete / not_inspected, each gap recorded with file and
reason, bounded. --require-complete-metadata-scan refuses an incomplete
inspection.

Sidecar v2. Sidecars are keyed by row index, so one copied beside a
different Parquet file silently means different rows. v2 binds the digest and
row count of the dataset it is published with. v1 sidecars from 0.6.0–0.6.3
still read.

Source mutation. Conversion captures source identity up front and
re-checks it — stat fields and content digest — before publishing, so a report
can never describe bytes the conversion did not read.

Bounded metadata. Output-to-source row provenance is contiguous runs
instead of one entry per row (it was built even for complete conversions,
where it is the identity function). resolve_source_row() reads it.
Deduplication detail is bounded with an exact total alongside.

Quality provenance. Split into the locally measured commit/platform, where
the GPU and Windows paths actually run, and the exact release commit CI
validated. One measured_commit implied a Windows RTX 4080 count had been
taken on the published squash commit; it never was.

Two CI defects. The pinned bridge workflow filtered on paths at the
trigger, so it did not run at all on unrelated PRs — and requiring a check that
never reports blocked the v0.6.4 state sync permanently until it was removed
from the required list. The filter moved to the expensive job; pinned bridge gate always reports, not_applicable when no bridge path changed, so it is
safe to require again. Its tag trigger was also v0.6.*, which would have
silently skipped the smoke on the v0.7.0 release tag.

Text integrity. A mis-decoded × is a single character, not the
three-byte CJK leaders the v0.6.3 gate looked for, so two CHANGELOG lines
survived the release. Latin-1 range added; both repaired. The tensor-to-float
warnings in test_chunked_equivalence.py are gone.

Validation

1865 CPU tests pass (35 new), 6 skipped for Windows privileges/POSIX-only
paths. Ruff, format, mypy, text integrity and release_state.py --check clean.
Frozen artifact digests unchanged.

Every bridge check opens a path inside the inspected bundle, and an open
follows whatever that path resolves to. A bundle could therefore ship an
entry pointing outside itself and have the doctor hash that file and search
its content for credentials, reporting the detector category back.

Reproduced on Windows with a junction: the hash check walked into the
outside directory and the metadata privacy scan reported semantic_secret_key
against a file the bundle did not contain.

preflight_bundle_tree walks with lstat only and refuses symlinks, reparse
points, non-regular files, entries resolving outside the root, and trees over
a bounded file count, byte total or depth. It runs before anything opens a
file, and a refused bundle reports every check as not_inspected rather than
failed: the checks did not run, so they cannot claim a finding.
The portable metadata scan is bounded: it skips files over a size limit,
stops at a total byte budget, and gives up on anything it cannot decode or
stat. All of those still produced heuristic_passed, so "found nothing" was
reported for an inspection that never looked at part of the bundle.

Statuses are now heuristic_passed_full, heuristic_failed,
heuristic_incomplete and not_inspected, each gap is recorded with its file
and reason, and the list is bounded. An incomplete scan is reported but not
failed by default; --require-complete-metadata-scan refuses it.

Also extend the text gate to the Latin-1 range. A mis-decoded multiplication
sign is one character, not the three-byte CJK leaders the v0.6.3 gate looked
for, so "State [x] Supervision" and "1.63[x]" survived in CHANGELOG.md
through the release. Both are repaired, and the tensor-to-float warnings in
test_chunked_equivalence.py are gone.
Sidecars are keyed by row index, so one copied beside a different Parquet
file silently means different rows. New sidecars are schema version 2 and
carry the digest and row count of the dataset they are published with;
reading a v2 sidecar without that binding fails closed. Sidecars published
by 0.6.0-0.6.3 declare version 1 and still read.

Conversion streams its source over a long period and then publishes a report
claiming a source_sha256. If the file is replaced in between, that claim
describes bytes the conversion never read while the output rows come from
the old ones. Source identity is captured up front and re-checked, digest
included, before anything is published.

Output-to-source row provenance was one map entry per accepted row, built
even for a complete conversion where it is the identity function. It is now
contiguous runs -- one run plus at most one per rejection -- with
resolve_source_row() to read it. Extension deduplication detail is bounded
with an exact total kept alongside.

The quality record separated: a local commit and platform where the GPU and
Windows paths actually ran, and the exact release commit CI validated. The
single measured_commit implied a Windows RTX 4080 count had been taken on
the squash commit that was published, which it never was.

The pinned bridge workflow filtered on paths at the trigger, so it did not
run at all on an unrelated pull request; requiring that check blocked the
v0.6.4 state sync permanently. The filter moved to the expensive job and
"pinned bridge gate" always reports, reporting not_applicable when no bridge
path changed. Its tag trigger was also v0.6.*, which would have skipped the
smoke on the v0.7.0 release tag.

Development version advances to 0.7.0.dev0. There is no v0.6.4 release.
@DaoyuanLi2816
DaoyuanLi2816 merged commit 0be43b0 into main Aug 8, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant