feat(minimal): guest fetch and degraded-mode fallback for min bug - #880

_{Comment @coderabbitai help to get the list of available commands.}

Two fixes pushed to feat/diag-unit7-guest-fetch — 1d696bca and 170bfb27. Both are additive to Unit 7; nothing else is restructured and the PR stays draft.

1. The manifest asserted the daemon logs were absent while carrying them

collect::logs skipped logs/minimald.log* with "no files with this prefix" the moment the host state dir held none. On macOS it never holds any — minimald runs inside the microVM and logs to the guest data volume. From a bundle captured during #869:

SKIPPED: logs/minimald.log* -> no files with this prefix

…with 178 KB of daemon logs sitting on the volume the whole time. Once Unit 7 lands, the same manifest.json carries that claim next to providers/<n>/guest/daemon-diag.tar.zst, whose own logs/ holds minimald.log.<date> — the files the claim says do not exist. The manifest's stated contract is that an absent file is always explainable, so a reader who trusts it stops looking. That is exactly what happened.

This is a regression, not a longstanding gap: the reference implementation (#784) emitted two skips, neither about logs. Unit 2 (#816) introduced this one on a temporal rationale — "the files only begin to exist after Unit 3" — which never considered that on macOS the host will never have them.

What changed

The claim was never wrong about the host; it was wrong about the bundle. Whether it is safe to make is not knowable when the log collector runs — collect_step!(w, "logs", …) fires before the provider loop, so "did a guest bundle arrive?" has not been answered yet. So:

• collect::logs reports prefixes that matched nothing back through an out-param instead of skipping on the spot (the shared collect_step! contract pins its return type at Result<(), _>, so a return value was not available);
• guest::collect now returns a GuestOutcome (BundleNested / NoBundle), which collect_guest propagates — this is the "pass it down" you asked for, just flowing the other way, since the log collector is upstream of the provider loop, not downstream;
• after the loop, collect::explain_absent_log_prefixes records the held-back skips against a DaemonLogSources enum (NoProviders / NoneFetched / Nested(&[name])), so the illegal combination "no providers, yet a nested bundle" does not typecheck.

Plain absence is now claimed only where it holds of the bundle: for a host-only writer (minvmd.log*) or when no provider instance exists at all. Where the logs did arrive, the skip names the carrier. Real output from a min bug run on this machine (throwaway state dir, one provider, no live daemon):

logs/minvmd.log*   -> no files with this prefix in <state>/logs
logs/minimald.log* -> none in <state>/logs — where minimald runs inside the microVM it logs
                      to the guest data volume, not to this host, and no daemon bundle was
                      fetched this run; see providers/*/guest/ for why

and with a bundle nested, the same skip ends … Its own logs are in this bundle, nested in providers/local-0/guest/daemon-diag.tar.zst (see logs/PROVENANCE.txt).

The test is the actual fix

no_skip_claims_absence_for_a_path_the_bundle_carries asserts the invariant, not the wording: it walks every manifest skip whose reason claims absence and requires that no path the archive holds — at any nesting level, nested .tar.zst included — is named by it. The absence phrasing lives in one const NO_SUCH_FILES so the test can find every skip that makes the claim. Non-vacuous in both directions: it also asserts the claim is still made for logs/minvmd.log* (so "fix by never claiming absence" fails it) and that the minimald skip names its carrier. The pre-fix reason was NO_SUCH_FILES verbatim, so the nested logs/minimald.log.2026-07-16 in the test's synthesised guest bundle would trip the assert — though the test cannot literally be compiled against the old code, since the function it drives did not exist.

One level down, untouched: crates/minimald/src/diag.rs:357 emits the same phrasing for logs/minimald.log* inside the guest bundle. There the claim is true of that bundle, so I left it — but note the nested manifest can still say "no files with this prefix" while the outer bundle's providers/<n>/boot.log holds daemon records. That trap is what the provenance note below now spells out.

2. Nothing told a reader which log was which

After Unit 7 a bundle carries two overlapping copies of the daemon's output — providers/<n>/boot.log (hvc0 console mirror, File::create'd per boot) and, inside the guest bundle, the on-volume appender (14 daily files). They are not duplicates and neither is a superset:

• console only — 4 records of boot prologue before the state volume mounts (mounted pseudo filesystem, switched to upstream rootfs, mounted writable state volume, cache + state relocated);
• console only — 5 records at shutdown: the appender is released before quiesce, so state volume quiesced, draining connections and the final close exist nowhere else;
• volume only — nothing in the incident window; the two were byte-identical modulo \r over the overlap;
• and post-Unit-4 the volume log is JSON lines while the console stays human-format, so they are not diffable without normalisation.

logs/PROVENANCE.txt (2.2 KB, one entry in the bundle) states per log path: source, retention, format, and the known holes.

Why the file and not a source field on CollectedEntry

CollectedEntry is #[non_exhaustive], so a field is additive at the type level — but it is not free:

• it changes the manifest shape both producers emit, which is a schema_version question and an amendment to R1.3, which pins the manifest's exact keys;
• every collector in both crates would have to supply a value or it ships mostly-null;
• and a scalar per entry cannot carry "what this source structurally cannot contain", which is the part a reader actually needs — "console mirror" does not tell you the volume log has no boot prologue.

A bundle entry costs no schema: schema_version stays 1, diag-explore needs no change, and the note appears in manifest.collected like anything else. If you'd rather have the field as well, it slots in later without conflicting with this.

Spec amendments I think are needed

Both fixes go beyond what spec 10 (#802, origin/arch/diagnostics-spec) currently says. Flagging rather than silently diverging — I did not edit the spec branch:

1. R2.7 — "Absent files are manifest skips, not errors (the files only begin to exist after Unit 3 — best-effort by design)." The parenthetical is the rationale that produced the bug. Suggest replacing it with the durable reason: an absence claim is scoped to the directory actually searched, and where a daemon may run inside the microVM the skip must point at the guest artifacts rather than assert absence.
2. R2.2 — "absence claims … may only be made on a true NotFound" is necessary but not sufficient; this skip was a true NotFound. Suggest extending: …and only when the bundle does not carry the named path by another route, nested bundles included.
3. New requirement (R2.10 or an R2.7 clause) — the bundle shall carry a log-provenance note naming, per log path, the source, retention, and known holes. Nothing in the spec requires it today, and Unit 8's explorer may want to surface it.

Correction: the degraded-mode debugfs fallback

An earlier review called this fallback inert on macOS because which debugfs fails. The conclusion is right, the cause is not: e2fsprogs is installed, keg-only, at /opt/homebrew/opt/e2fsprogs/sbin/debugfs (debugfs 1.47.4). guest.rs resolves it by bare name (Command::new("debugfs"), in both volume_fallback and logs_tree_bytes), so PATH lookup misses the keg and R7.5 degrades to volume-meta.json plus the offline hint. Measured just now against a fake image:

providers/local-0/guest/volume-logs -> debugfs unavailable (No such file or directory (os error 2)); harvest offline with: …

That is ENOENT on exec, not on the image. I left it alone — out of scope for the two fixes you asked for — but it is a one-liner (probe the keg path when the bare name misses, or honour a MINIMAL_DEBUGFS override) and it is what stands between R7.5 and working on the primary dev platform.

Verification

So: the code paths are verified natively through the real binary, and the test assertions over them are not. The one branch neither covers is DaemonLogSources::Nested end to end — the unit test drives it with a synthesised daemon bundle rather than a real fetch. Explicitly not run: anything needing a live microVM; artifact 4 in the PR body (kill a real VM's daemon mid-session) is still unverified and this does not change that.

Likely conflict ahead: another change in flight adds a --log-tail-bytes flag off main, touching crates/minimal/src/diag/. It will almost certainly want to plumb a cap through collect::logs, whose signature I just changed (added the out-param) and whose body I edited around LOG_TAIL_CAP. Same function, same lines — whoever merges second should expect to resolve it by hand.

@macroscope review

Manual reviews triggered for commit 170bfb2:

• Approvability Check
• Correctness Check

^{All prior checks · these links stay valid even if you push more commits.}

Just FYI for future @mentions, I'm Macroscope-App, not Macroscope.

Review triggered. Results will be posted as check runs when complete.

Approvability

Verdict: Needs human review

2 blocking correctness issues found. This PR introduces new guest bundle fetching and degraded-mode fallback capabilities with substantial new logic. Two unresolved review comments identify potential correctness issues in error handling that should be addressed before merging.

^{You can customize Macroscope's approvability policy. Learn more.}

Review round worked — 2 open findings closed, 8 confirmed already closed, 1 adjacent defect fixed

Five commits on feat/diag-unit7-guest-fetch. Nothing rebased, nothing force-pushed, body untouched.

Verdicts

Open at the start of this pass — both valid, both fixed.

diag/mod.rs:162 (Medium) — fixed. Reproduced first, against a binary built from 170bfb27, with chmod 000 providers on a throwaway state dir:

logs/minimald.log* -> no files with this prefix in <state>/logs
logs/minvmd.log*   -> no files with this prefix in <state>/logs

…in a manifest that also carried two Permission denied errors. The mechanism is as described: absent is filled by the prefix loop before collect::logs reaches its own provider_dirs(...)?, so the prefixes survive the collector's failure and reach a DaemonLogSources computed from a listing that never happened. providers is now Option<Vec<_>> and DaemonLogSources gained ProvidersUnknown. minvmd.log* deliberately still claims plain absence there — host-only writer, unaffected by a provider listing failure.

collect.rs:364 (linked location) — fixed in the same commit. BundleNested was returned for any stored bytes, so a truncated or verification-failed blob got named as the carrier of the daemon's logs. verify_nested_bundle now also reports whether it saw logs/minimald.log* go past — free, since it already walks every entry path for the manifest check — and GuestOutcome splits confirmed from unconfirmed. An unconfirmed bundle is still named, as somewhere to look, never as the answer.

client.rs:284 (Medium) — valid, fixed, but not as suggested. Draining to the end of the channel is unbounded: the daemon only reports a build failure after its tokio::io::copy returns, guest::collect caps this call at --guest-timeout-secs (60s), and the daemon's own stream deadline is 300s. So the suggested drain trades a usable 256 MiB truncated bundle for a timeout and nothing — and the message it would most often recover, client stopped reading the bundle, is one the drain provokes itself. Landed a bounded grace-window drain instead (payload discarded, extended data kept) and corrected the method's contract to state the residual rather than promise what the code does not do. Full reasoning in thread.

Already closed before this pass — verified present in the tree at 170bfb27, read at the call site rather than taken on the resolution marker. collect.rs:332 (High — open_regular_nofollow), guest.rs:138 (High — stream cap under the parser), guest.rs:210 (NotFound distinguished), net.rs:114 (connect stage re-tested), collect.rs:367 (StateDir::open_existing), guest.rs:238 (pre-flight ls -l sizing), plus the two Macroscope self-merged via #889 / #890.

One change that is not from this round

105522fe fixes debugfs resolution. Adjacent rather than requested: two of this round's findings landed inside volume_fallback / logs_tree_bytes, which are its only two callers, and I had previously measured this defect and left it alone as out of scope. Both sites spawned debugfs by bare name, so R7.5 was inert wherever e2fsprogs is not on PATH — keg-only under Homebrew, /sbin on Linux. Measured here against a real ext4 image built with mke2fs -d:

before:  volume-logs -> debugfs unavailable (No such file or directory (os error 2))
after:   volume-logs/minimald.log.2026-07-20
         volume-logs/minimald.log.2026-07-21          (24 entries, 0 errors)

ENOENT on exec, not on the image — on the platform where minimald runs inside the microVM and the volume is the only place its logs exist. Separated into its own commit so it can be dropped independently if it is not wanted here.

Pre-existing, deliberately not fixed

diagnostics::newest_rotated collapses a read_dir failure into an empty result, so a <state>/logs that stats but will not enumerate reaches absent as though it were empty — the same defect class as mod.rs:162, one layer down. It is shared with minimald, so changing its signature belongs with Unit 6.

Verification

The two tests added in 1d696bca that had never been run are now run, and pass:

test diag::collect::tests::no_skip_claims_absence_for_a_path_the_bundle_carries ... ok
test diag::collect::tests::the_log_provenance_note_distinguishes_the_two_daemon_log_sources ... ok

alongside the four added here:

test diag::collect::tests::plain_absence_is_claimed_only_where_the_run_established_it ... ok
test diag::collect::tests::only_a_confirmed_nested_bundle_is_named_as_the_carrier ... ok
test diag::guest::tests::a_pax_extension_bomb_is_stopped_by_the_stream_cap ... ok
test diag::guest::tests::verification_reports_whether_the_daemon_logs_are_present ... ok

test result: ok. 79 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.88s

The three guest-fetch integration tests all pass (bug_with_daemon_nests_a_verified_guest_bundle, bug_with_stale_socket_reports_the_connect_stage_and_falls_back, bug_no_guest_makes_no_daemon_contact). The 2 failures are the Unit 5 tests this branch does not touch, and I re-confirmed the cause directly rather than inheriting the claim — inside the cross image:

ip        MISSING
ifconfig  MISSING
ss        MISSING
netstat   MISSING
lsof      MISSING

so host/net/interfaces.txt cannot be produced and 2 collector errors land against assert_eq!(errors, 0). Same 5/2 split as before this pass — no new failures.

cargo test -p minimal --all-targets still cannot build natively; re-confirmed, and the cause is one layer wider than "procfs": both procfs (Building procfs on an for a unsupported platform) and caps fail on macOS via the minimald dev-dep.

Merge conflict expected

#898 (--log-tail-bytes) touches collect::logs, whose signature this branch already changed in 1d696bca. Not pre-resolved — flagging it so whoever merges second knows it is expected.