feat(minimald,minimald-rpc): daemon bundle serving (DiagBundleTarZst) - #878

sequenceDiagram
  participant Client
  participant SSHSubsystem
  participant DiagnosticsBuilder
  participant BundleWriter
  participant Channel
  Client->>SSHSubsystem: send DiagBundleRequest
  SSHSubsystem->>DiagnosticsBuilder: start bundle build
  DiagnosticsBuilder->>BundleWriter: collect diagnostics
  BundleWriter-->>DiagnosticsBuilder: finalized tar.zst stream
  DiagnosticsBuilder->>Channel: pump bundle bytes
  Channel-->>Client: receive diagnostic bundle

_{Comment @coderabbitai help to get the list of available commands.}

Two collectors added to the guest bundle, from a review of this epic against #869. Both are small and belong here rather than in a follow-up, because Unit 6 is what defines the daemon-side bundle. Nothing else in the PR is touched and it stays in draft.

 crates/diagnostics/src/kmsg.rs  | 247 ++++++++++++++++++++++++++++++++++++++++
 crates/diagnostics/src/lib.rs   |   3 +
 crates/diagnostics/src/procs.rs |   6 +-
 crates/minimald/src/diag.rs     | 165 ++++++++++++++++++++++++++-
 4 files changed, 419 insertions(+), 2 deletions(-)

1. The guest kernel version was in no bundle at all

The whole "why did this start now" of #869 is that the guest kernel moved 6.12.43 → 6.12.94 and crossed a release that turned a silent vsock condition into a fatal connection reset. No bundle carried that fact. meta.json had version, long_version, stdlib_version, pid, uptime_secs, in_microvm — no kernel. diagnostics::system_info does have a kernel field, but it is a host collector and shells out to uname, which the microVM rootfs does not ship.

Three fields added to the meta collector, all read from /proc directly — no external binary:

Raw and parsed, not either/or

The banner stays verbatim and the parse rides beside it. Keeping only the banner is more honest and cheaper, but "which release is this" is precisely the question a bundle-to-bundle diff asks, and eyeballing two lines full of toolchain and build-time noise is how that question gets answered wrongly. Keeping only the parse would let a bad parse destroy evidence.

Riding beside it, the parse can only ever cost a convenience:

fn kernel_release(banner: &str) -> Option<&str> {
    let mut tokens = banner.split_whitespace();
    match (tokens.next(), tokens.next(), tokens.next()) {
        (Some(_sysname), Some("version"), release) => release,
        _ => None,
    }
}

The kernel writes that banner from one fixed format string (linux_proc_banner, "%s version %s …" over utsname()->sysname and ->release), so the third token is the release whenever the second is literally version. Anything else yields None rather than a guess.

/proc/cmdline — checked before collecting it verbatim, then wired to the redaction engine anyway

Confirmed against main: the guest boot line is BASE_KERNEL_CMDLINE (console=hvc0) plus an optional RUST_LOG=<value> from #872, whose kernel_cmdline rejects an empty value, any value containing whitespace, and anything that would overrun COMMAND_LINE_SIZE. Nothing secret can reach it today, and #872's own doc comment states the invariant explicitly.

But that is an invariant about this machine — where /proc/cmdline is world-readable and therefore not a secret from anyone already inside the guest — not about the bundle, which leaves the machine and gets mailed to whoever is helping. A boot parameter added later that happens to be secret-shaped would otherwise ship verbatim. So the line goes through the same fail-closed key=value scrub a process argv does, and meta.json is now recorded as Redaction::Keys rather than a verbatim capture. procs::scrub_flattened was made pub for it — a kernel boot line is a space-joined command line with exactly the same lost-boundary ambiguity ps output has, so it wants that rule and not a new one.

Sample output

meta.json, the new fields (values as collected under a real Linux kernel in the cross test container; in the guest the banner is the microVM kernel and the boot line is console=hvc0 plus the forwarded filter):

{
  "kernel": "Linux version 6.12.76-linuxkit (root@buildkitsandbox) (gcc (Alpine 15.2.0) 15.2.0, GNU ld (GNU Binutils) 2.45.1) #1 SMP Fri May  1 14:35:41 UTC 2026",
  "kernel_release": "6.12.76-linuxkit",
  "kernel_cmdline": "init=/initd loglevel=1 root=/dev/vdb rootfstype=erofs ro vsyscall=emulate panic=0 eth0.dhcp eth1.dhcp linuxkit.unified_cgroup_hierarchy=1 console=hvc0 virtio_net.disable_csum=1 slub_min_order=2 page_reporting.page_reporting_order=2 vpnkit.connect=connect://2/1999 com.docker.VMID=a2621183-8326-4da9-8856-0a7fa2c03882"
}

and the manifest entry for it:

{ "path": "meta.json", "redaction": "keys", "bytes": 1131 }

Had #869's two kernels each produced a bundle, kernel_release alone would have read 6.12.43 and 6.12.94.

2. The guest bundle carried no kernel evidence whatsoever

Kernel messages reach the host only via console=hvc0 → boot.log, which the host bundle collects, so a guest bundle fetched on its own had nothing. And boot.log is File::created per boot and tail-capped, so the kernel banner — the first thing written — is the first thing dropped on a long-lived VM. The one artefact that would have named the kernel is the one that ages out first.

diagnostics::kmsg::kmsg_tail reads the ring buffer straight from /dev/kmsg into logs/kmsg.txt, beside the daemon's own logs and under the same caller-controlled log_tail_bytes cap (so the existing R6.3 clamp bounds it too). The microVM rootfs has no dmesg and needs none: one read(2) per record.

O_RDONLY | O_NONBLOCK, read until EAGAIN. The non-blocking flag is what makes "read to the end" terminate at all — without it the read that drains the buffer blocks until the kernel logs something new, which on the quiet wedged guest this capture exists for may be never. The end of the buffer is EAGAIN, not EOF. EPIPE is not an error either: it means records were overwritten while the loop ran, the file position has already advanced past them, and what was lost is exactly what a tail drops anyway.

Manifest honesty, matching the other log collectors:

None of those is a collector failure, so a bundle from a host that will not show its ring buffer still collects cleanly and explains the gap. kernel_ring_buffer_is_collected_or_explained asserts the archive and the manifest agree on which of the two happened — silence is not an option either way.

Records are kept whole when the cap bites (oldest dropped first): half a kmsg record is a corrupt line rather than a shorter one, and a single record larger than the whole cap is kept rather than losing the newest message.

Sample output

logs/kmsg.txt is the kernel's raw record format — priority,sequence,timestamp,flags;message — not decoded, for the same reason net/routes.txt keeps the raw /proc/net tables: the sequence numbers are themselves evidence of what the ring dropped. Sampled from a live Linux ring buffer:

6,0,0,-;Booting Linux on physical CPU 0x0000000000 [0x610f0000]
5,1,0,-;Linux version 6.12.76-linuxkit (root@buildkitsandbox) (gcc (Alpine 15.2.0) 15.2.0, GNU ld (GNU Binutils) 2.45.1) #1 SMP Fri May  1 14:35:41 UTC 2026
6,2,0,-;OF: reserved mem: Reserved memory: No reserved-memory node in the DT
6,3,0,-;Zone ranges:

Note record 1 — the banner, which is exactly what a tail-capped boot.log loses first.

/dev/kmsg does not exist in the cross test container, so the test run there exercises the absent path and the manifest carries:

{ "what": "logs/kmsg.txt",
  "reason": "no /dev/kmsg — this kernel does not expose the ring buffer here" }

Spec amendments needed

Neither addition changes the bundle layout incompatibly, so schema_version stays at 1 — R1.3's bump condition is not met, both are additive. But R6.2 should be amended on #802 rather than left to diverge, on two points:

1. R6.2's meta.json enumeration currently reads "meta.json (version, uptime, microVM flag)". It should name the kernel identity — kernel / kernel_release / kernel_cmdline — because the guest kernel is a fact no other collector in the spec captures (Unit 2's system_info is host-side and uname-based), and because meta.json is no longer a verbatim capture: the boot line makes it Redaction::Keys, which is R1.5's contract talking.
2. R6.2's entry list should name logs/kmsg.txt. The spec's threat/evidence discussion already leans on "capture (boot.log) independent of the volume" for the wedged-daemon case; that argument has a hole the spec does not state — boot.log is host-side, per-boot and tail-capped, so the guest half of the kernel picture is unavailable from a guest bundle and ages out of the host one. Worth stating and closing in the spec, not only in code.

Neither is a Unit 8 concern: R8.1 keys guest-bundle detection on meta.json presence and R8.2 reads errors from manifest.json, so both additions are transparent to it.

Verification

minimald cannot be built natively on macOS, so it was verified via cross, as this PR already does. Both cross invocations ran with CROSS_CONTAINER_OPTS="--env HOME=/tmp".

New tests:

test diag::tests::meta_carries_the_guest_kernel_identity ... ok
test diag::tests::kernel_release_reads_the_proc_version_banner ... ok
test diag::tests::kernel_cmdline_is_scrubbed_before_it_travels ... ok
test diag::tests::kernel_ring_buffer_is_collected_or_explained ... ok

test kmsg::tests::kmsg_is_either_collected_or_explained ... ok
test kmsg::tests::record_tail_drops_whole_records_from_the_oldest_end ... ok
test kmsg::tests::record_tail_keeps_a_record_larger_than_the_cap ... ok
test kmsg::tests::record_tail_under_the_cap_drops_nothing ... ok

kernel_release_reads_the_proc_version_banner parses #869's two banners specifically (6.12.43 and 6.12.94).

Gating

/proc/version, /proc/cmdline and /dev/kmsg are Linux-only. meta's /proc reads follow the collector's existing best-effort shape (Option, alongside the /proc/uptime and /proc/mounts reads already there) rather than a cfg, so nothing new is Linux-conditional in minimald. In diagnostics, kmsg is a #[cfg(unix)] module whose reader is Linux-only with a non-Linux skip arm — the same shape procs::all_processes already has — so crates/minimal (which builds on macOS and does not call it) is unaffected, and the pure RecordTail tests still run on the macOS host.

Not verified

No live guest fetch: the in-VM values were not observed on a running microVM, which was deliberately left alone. What is proven is the collectors' behaviour under a real Linux kernel in the cross container, plus the parse against #869's literal banners.

@macroscope review

Manual reviews triggered for commit 36cf370:

• Approvability Check
• Correctness Check

^{All prior checks · these links stay valid even if you push more commits.}

Just FYI for future @mentions, I'm Macroscope-App, not Macroscope.

Review is underway. Feedback will be posted through the check runs on this PR when they complete:

• Macroscope - Approvability Check
• Macroscope - Correctness Check

Approvability

Verdict: Needs human review

3 blocking correctness issues found. This PR adds a new daemon diagnostic bundle streaming feature with significant new code and security-sensitive patterns. Three high-severity unresolved review comments identify potential vulnerabilities (symlink following in log directory, missing request timeout, unbounded file reads) that warrant human attention before merging.

^{You can customize Macroscope's approvability policy. Learn more.}

Review round worked — 4 High, 3 Medium

All seven threads are answered in place. Four were fixed in 805727ba before this round; three are fixed here in ba4d9508; one of those three had been marked resolved while still live. One finding is rebutted: its effect is real but its stated cause is not, so I've argued the severity down rather than quietly implementing the suggestion.

Which of these this PR introduced. Everything except the procs.rs scoped defect lands on the bundle-serving code this PR adds (bc36afa7). Nothing in the round touches the two collectors stacked on top of the original review base — the guest kernel identity in meta.json and the /dev/kmsg tail. The procs.rs finding is the split one: hang_triage_including is new here, but the two identical sites in process_tree shipped in Unit 5 (#864) and were surfaced by this round rather than introduced by it — all four sites were fixed together because splitting them would have left a known defect on main to be re-found later.

The one I pushed back on

:481 rests on "the state directory is writable by guest tasks". It isn't, in the part that matters. A session's sandbox receives two bind mounts, home → /home and working → /workbench (sandbox2/src/lib.rs:666-697), which resolve to <state>/sessions/<short>/home and .../tree (sessions/src/store.rs:183-191). record.json is their sibling one level up — outside the task's mount namespace, unnameable from inside the sandbox. SFTP rejects any client path escaping the workspace prefix (minimald/src/sftp.rs:126-146), and the workspace upload unpacks through async_tar, which drops .. components (async-tar-0.6.1/src/entry.rs:473). There is no write primitive, so there is no multi-gigabyte plant and no OOM.

The same premise underpins :306, which is why I declined the offered follow-up PR for the logs/ TOCTOU instead of accepting it. An openat-anchored traversal means hand-rolled rustix plumbing or a new cap-std dependency, spent against a threat that cannot be constructed against this deployment. The symlink_metadata check that shipped stays as what it honestly is — cheap defense in depth, not a patch for a live hole. If a write primitive into the state root ever appears, the fix belongs in diagnostics::newest_rotated, where it also covers the host collector.

I bounded the session reads regardless, because the half of :481 that needs no attacker stands on its own: it was the last caller-triggered read in this subsystem without a bound, in a change whose stated contract (R6.3) is that they all have one, in a pid-1 process with a fixed RAM budget. Routing it through open_regular_nofollow also closed a symlink hole nobody had raised, for free.

Verification

minimald cannot be built natively on macOS (its dep tree pulls procfs), so as with the rest of this PR it was verified through cross:

$ CROSS_CONTAINER_OPTS="--env HOME=/tmp" cross test -p minimald --target aarch64-unknown-linux-musl
test result: ok. 163 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 83.73s
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s

$ ... cross clippy -p minimald --all-targets --target aarch64-unknown-linux-musl -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 18s

Four new tests, all passing:

test diag::tests::a_request_that_never_ends_is_abandoned_on_the_deadline ... ok
test diag::tests::a_symlinked_session_record_is_refused ... ok
test diag::tests::an_oversized_session_record_is_refused_not_buffered ... ok
test diag::tests::an_unlistable_sessions_dir_is_recorded_not_silent ... ok

Two caveats on that evidence, so it is not read as more than it is:

• an_unlistable_sessions_dir_is_recorded_not_silent builds its fixture from mode bits (sessions/ at 0o111), which root ignores, so it skips itself when it detects the fixture did not take. Reproduction under the cross container was confirmed by temporarily replacing the skip with an assertion and watching it pass; the skip is back in place for the benefit of any root test runner.
• Docker died partway through this round and had to be restarted. The first cross invocation afterwards exited 0 while doing nothing at all — cross reports success when it cannot reach the Docker daemon. Worth knowing before trusting a green cross run: check for a test result: line, not an exit code.

Heads-up: main is red as of this merge — E0271 in crates/minimald/src/rpc.rs (:665, :770, :788): a connection future is expected to resolve to Result<(), ConnectionError> but resolves to (). Looks like a semantic conflict with #895 (merged 3 minutes earlier, also touching rpc.rs): each PR was green on its own base, the combination doesn't type-check. Failing main run: https://github.com/gominimal/minimal/actions/runs/29945701386 — a forward fix reconciling the two should clear it (also currently failing all open PR merge-ref builds, e.g. #912).