feat(minimal): add min bug diagnostic bundle command - #784

New streaming RPC: the client writes one JSON DiagBundleRequest and
half-closes; the daemon streams back a zstd-compressed tar of its
diagnostic bundle. Pre-stream errors relay over extended-data stream 1,
mirroring WorkspaceFilesTarZst.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A detached supervisor's stdout goes nowhere, so in-field boot failures
left no log trail. `run --detach` now marks its re-exec'd child with
MINVMD_DETACHED, which routes tracing to
<state>/logs/minvmd.log.<date> (14 retained), mirroring minimald's
scheme so diagnostics find both daemons' logs in one place. Tracing
init moves after CLI parsing because the log dir derives from the
--minimal-state-dir override; the run.log stderr redirect stays and
keeps catching panics and the final error print.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

New diag module streams the daemon's own view for `min bug`: meta
(version, pid, in_microvm, state_volume_mounted), rolling logs
(tail-capped), state-dir listing (metadata only), session records with
secret-shaped values redacted, /proc process and net tables, a gvproxy
gateway probe when in a microVM, and disk/mounts. Collector failures
accumulate into an errors.json entry instead of aborting, so a
half-broken daemon still reports what it can. Served identically by the
native daemon and the in-VM pid-1 instance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

One command for in-field debugging: writes
minimal-diag-<timestamp>.tar.zst containing host system/process/network
state, redacted config and loadouts, state-dir listings (never session
file contents), rolling daemon logs, and — per provider instance — a
staged socket probe (stat -> connect -> handshake -> GetVersion, whose
failing stage distinguishes stale-socket from wedged-vsock from
healthy) plus the daemon's own bundle streamed over the new
DiagBundleTarZst subsystem. Every collector is failure-isolated into
manifest.json errors; the command never autospawns daemons and never
mutates state (reads minvmd.toml raw rather than via effective_state(),
which repairs on read). client.key and host keys are never opened.

The guest download is bounded by --guest-timeout-secs (60s default) and
a 256 MiB cap; `min dirs` now names the minvmd rolling log too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The microVM's pid-1 minimald logged only to stdout (serial console), so
`min bug`'s guest bundle never carried the in-VM daemon's logs: the
`logs` collector reads <state>/logs, which stayed empty. Two causes
stacked — pid-1 is never `--detach`'d (so it took the stdout branch),
and init_tracing runs before the state volume mounts (so a naive file
path would be shadowed by the later mount).

Route the microVM to log to both the console (kept for boot diagnosis
via the host boot.log) and a daily-rotated <state>/logs/minimald.log.
A DeferredFileWriter is installed at init (discarding) and pointed at
the appender once /var/lib/minimal is mounted and state has relocated
onto it. Activation failure only warns and falls back to console — it
must never wedge pid-1.

Verified on macOS via `just dm1`: the guest bundle now contains
logs/minimald.log.<date> with real runtime records, errors.json == [].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

download_diag_bundle only matched Data and ExtendedData in its channel
loop, so when a daemon that predates the RPC refuses the
DiagBundleTarZst subsystem, the resulting SSH_MSG_CHANNEL_FAILURE fell
into `_ => {}` and was dropped. A bare refusal does not close the
channel, so wait() never returned None and the download blocked until
the caller's guest timeout (60s per provider).

Handle ChannelMsg::Failure: bail immediately with an upgrade hint. The
happy path is unchanged — a healthy daemon replies Success and then
streams the bundle.

Verified: against a pre-diagnostics daemon `min bug` now errors in
0.08s (was 60s) with a clear message; against a current daemon the
bundle still downloads with errors[] empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Two review findings let secret-shaped values escape the `min bug`
redaction pass:

- A sensitive-named key holding a table/array (`[tokens]`,
  `api_token = [...]`) recursed without the mask, leaking its members
  unless their own keys also looked sensitive. The sensitive-key check
  now propagates into container values, mirroring env-table semantics.
- Env vars matching the project prefixes were captured verbatim even
  when their names were secret-shaped (`MINIMAL_AUTH_TOKEN`). The
  allowlist now defers to the sensitive-name deny, and the bare
  `MINIMAL` prefix is anchored to `MINIMAL_`.

Also anchors the `public` exemption to `public_key` shapes so
`publication_secret` and friends stay masked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review findings on the DiagBundleTarZst server path:

- If the channel copy failed mid-stream (client disconnect), the
  build task stayed blocked writing into the full duplex pipe forever,
  leaking the task and its ServerStateHandle clone. The pipe's read
  half is now dropped before the task is awaited so its next write
  fails with BrokenPipe, and copy errors propagate first.
- The logs collector bundled the newest five files of any name from
  the shared logs dir; on native installs minvmd's rotated logs sort
  after minimald's and crowded them out entirely. It now takes only
  `minimald.log*` files — minvmd's logs belong to the host collector.
- Collectors used blocking std::fs on the RPC-serving runtime; reads
  now go through tokio::fs and the /proc walk runs in spawn_blocking.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PROCESS_MARKERS were substring-matched against the whole ps args
column, so a user editing or tailing our logs (`vim minimald.log`,
`tail -f minvmd.log.2026-07-15`) had their process — full command
line, plus /proc status on Linux — captured into the bundle. Only the
executable name is matched now, the same discipline the
`min`/`minimal` names already used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review findings on the guest download path of `min bug`:

- A daemon error arriving after some archive bytes had streamed was
  silently discarded, landing a corrupt nested archive with a clean
  manifest. Any daemon-reported error now fails the download, and the
  guest collector records it in the bundle as before.
- A truncated guest bundle was recorded both as skipped and as
  collected-verbatim; it is now a single collected entry with a new
  `truncated` redaction level.
- Guest collector errors were timed from the start of the whole run
  instead of the provider's own collection.
- Host collectors used blocking std::fs in async context; they now
  use tokio::fs (provider_dirs became async accordingly).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The in-VM daemon's rolling log file (persisted to the state volume for
`min bug`) holds a write-open fd under /var/lib/minimal at shutdown.
That fd defeats both the plain unmount and the read-only remount in
quiesce_state_volume, so every clean stop left the ext4 journal dirty
(INCOMPAT_RECOVER set) — caught by
stop_quiesces_volume_leaving_clean_ext4_journal in CI.

pid-1 now parks the appender handle and its worker guard in a release
hook next to the quiesce code; the Shutdown RPC runs it (idempotently)
after the session drain and before the quiesce, flushing and closing
the file. Console logging (serial -> host boot.log) is unaffected, so
the teardown records still land in the host's boot log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Own-IP guest egress rides 100.64/16, inside the CGNAT block that
Tailscale-style VPNs also occupy; published-port and MTU tickets need
to know which host addresses and default route exist. Neither was
visible in the bundle.

host/net/interfaces.txt (`ip addr` / `ifconfig -a`) with every MAC
masked to its vendor OUI — the IPs and subnets are the diagnostic
payload, hardware identifiers are not — and host/net/routes.txt
(`ip route` / `netstat -rn`) verbatim. Same attempt-list-with-timeout
pattern as the listening-sockets collector; hosts without the tools
degrade to a recorded collector error, which the integration test now
accepts as the alternative to the files being present.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The hvc0 console was captured only when the dev-only MINVMD_BOOT_LOG
env var was set; detached production boots discarded it, which is
exactly the evidence lost when a guest wedges before the daemon is
reachable or its transport dies afterwards (#788).

minvmd now defaults the console capture to <provider dir>/boot.log,
truncated per boot so it holds the current VM generation; the env var
still overrides, and a capture failure logs a warning and boots on —
diagnostics must never become a boot dependency. `min bug` bundles the
file per provider (tail-capped) alongside run.log.

Refs: #788

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

On guest-bundle timeout or download failure the bundle carried only an
error.txt — yet the failure domain in that case is often the vsock
transport itself, the very path DiagBundleTarZst rides, so the guest's
logs were lost precisely when they mattered most (#788).

The data volume is plain ext4 and safe to read read-only from the host
while the VM runs. On any guest-collection failure (including a dead
socket probe) `min bug` now records the image's vital signs
(volume-meta.json — its mtime alone dates a stall) and best-effort
harvests /logs from the image via `debugfs -c` into
providers/<n>/guest/volume-logs/ when e2fsprogs is installed; absent
that, the manifest carries the exact offline-extraction command.

Refs: #788

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

"It's frozen" diagnosis runs on thread stacks and socket states of the
min/minvmd/VMM family (#788: vCPUs parked in WFI, proxy in kevent, a
unix socket open with no EOF) — none of which the bundle captured.

For up to eight matched pids: a 1-second `sample` per process on macOS
(host/proc/<pid>.sample.txt); wait channel, current syscall, kernel
stack (root-only — the error is data) and readlink'd fds on Linux
(host/proc/<pid>.stack.txt); plus one `lsof -nP` over the whole set
(host/proc/lsof.txt), accepting lsof's exit-1-with-output convention.
Every miss degrades to a manifest note.

Refs: #788

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Suspend/resume is the most plausible mundane explanation for a
long-lived stream dying silently (#788), yet ruling it in or out meant
asking the user to run pmset by hand. host/power.txt now captures the
last 100 sleep/wake transitions (pmset -g log on macOS, plus sysctl
kern.{boottime,sleeptime,waketime}; journalctl -b -k suspend/resume
lines on Linux), best-effort in the collector idiom. Today's incident
was exonerated this way — the host stayed awake through the wedge —
and that check is now one bundle away instead of a support round-trip.

Refs: #788

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Maintainer review: common is for baseline types used everywhere, and
the diagnostics machinery doesn't belong there. Move the app-agnostic
core into a new `diagnostics` crate:

- redact (was common::redact) — key-based secret masking
- listing (was common::listing) — bounded recursive dir listing
- bundle + manifest (were minimal::diag::{bundle,manifest}) — the
  tar+zstd BundleWriter and its self-describing manifest types

The host-specific collectors (which need a live daemon client, resolved
paths, and the machine's process/network state) stay in minimal::diag
and minimald::diag, now importing from `diagnostics`. BundleWriter no
longer bakes in the CLI version via env!("LONG_VERSION") — the crate is
app-agnostic, so create() takes it as a parameter.

Also hardens minimald/build.rs: git being unavailable (source tarball,
or a worktree whose .git isn't mounted in a cross container) degraded
to an unwrap panic; it now falls back to an empty hash.

Refs: #784

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Maintainer review of the boot-log quiesce fix flagged three linked
smells: a hand-rolled `Arc<RwLock<Option<NonBlocking>>>` deferred
writer ("Wat"), a process-global `static Mutex<Option<Box<dyn FnOnce>>>`
release hook (fork-hazardous), and a free function mutating that global
state (doesn't scale).

Replace all three:
- the deferred writer becomes a `tracing_subscriber::reload` layer that
  starts inert (records still reach the console) and is swapped to the
  on-volume appender post-mount — the library's mechanism, no custom
  MakeWriter. init_tracing returns a `LogActivator` closure so the
  reload handle's generic type never surfaces in a signature.
- the release closure moves from the static into a `VolumeLogRelease`
  field on `ServerState`, threaded async_main -> Server::run ->
  ServerState::new (it can't ride in the serializable `Config`).
- the Shutdown RPC calls `s.release_volume_log().await` — a state
  method, not a free function.

Behavior is unchanged: the on-volume appender's fd is still closed
before the quiesce, keeping the ext4 journal clean on a stop. Needs
the field re-validation (idle attach + `minvmd stop`) before merge.

Refs: #784

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The reload refactor added a parameter to ServerStateHandle::new but the
test-harness caller (gated behind the test-support feature, so the
plain --all-targets clippy missed it) still called it with one arg,
breaking the clippy/tests/root-integration CI lanes. Compile clippy
with --all-features to catch feature-gated callers.

Refs: #784

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>