fix(det-init): a hung sample must not discard the whole event feed - #981
fix(det-init): a hung sample must not discard the whole event feed#981bryan-minimal wants to merge 7 commits into
Conversation
New `crates/det-init` binary — the minimal-detonation guest /init. It reuses `minimald::guest::*` (mount_dev / enter_rootfs / mount_state_volume / bring_up_root_egress / shut_down_vm) for the pid-1 boot plumbing instead of forking minimald, then stands up the in-guest BPF-LSM observer and runs the untrusted sample as `nobody` under it, one-shot — no SSH server, no session, no dependency on the arbitrary-exec surface #709 removed. Ports the proven detonate.sh + init-launcher.sh flow: securityfs/bpffs mounts, deny-channel tripwire staged pre-attach, per-scan binding-nonce load (observer- only) with a nobody-can't-read leak-check, observer ready-file handshake, DNS collector repoint, uid-boundary self-check, droppriv fork-and-reap of the sample, bounded observer drain. Observer streams the JSONL feed over vsock (file fallback); host runs `minimal-detonation verdict`. Prototype: compiles/boots on Linux/KVM (minimald's guest mounts are Linux-only). Runtime contract (host stages sample+nonce, bake stages observer/droppriv/ tripwire/honeytokens) documented as path constants. See gominimal/minimal-detonation#194. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… DNS — boots+detonates on KVM (#194) Proven end-to-end on x86 KVM (spike1-nestedvirt): det-init boots as pid-1, enters the rootfs via minimald's embedded guest lib, stands up the observer (16/16 LSM hooks), runs the sample as nobody, and verdicts BLOCK on a malicious honeytoken-read / PASS on a benign run. Changes from the initial scaffold, each earned by a boot iteration: - emit the READY vsock beacon (guest::emit_simple_ready_marker) right after entering the rootfs — the minvmd supervisor waits for it (~5s) and tears the guest down otherwise; it must fire before the observer-ready wait (up to 10s). - emit the drained feed on stdout so a host collecting via the guest console (the boot harness) gets it without a vsock listener. - file sink only: drop --vsock-port from the observer spawn. It made the observer BLOCK connecting its sink to an absent host vsock listener, so it never signalled ready and the one-shot boot stalled. - gate the DNS-collector repoint on egress being up: with no network the bind-mount is pointless AND showed up as a spurious pid-1 Mount the verdict read as a container-escape false-positive on an otherwise-benign run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iner-escape FP on stock minvmd (#194) Booting on minvmd built from main (which, unlike the fork binary, sets up the own-ip vsock shuttle so egress reports up) surfaced that det-init's own resolv.conf BIND-MOUNT — done as pid-1 — is captured as an sb_mount event the verdict scores as a container-escape, false-blocking an otherwise-benign run. Repoint /etc/resolv.conf with a plain write instead (matching the legacy detonate.sh runner). On a read-only rootfs it fails EROFS and degrades to "DNS names unobserved" rather than mounting. Full DNS-name capture on a RO rootfs (a writable /etc overlay) is a networked-detonation follow-up; the robust fix is verdict-side process-subtree scoping so pid-1 infrastructure never counts toward the sample verdict. PROVEN on STOCK-main minvmd (built from origin/main, no MINVMD_KERNEL_FORMAT override — main auto-picks ImageGz for x86_64 — attaching the #672 /dev/vdb volume): malicious honeytoken-read -> BLOCK, benign -> PASS. The whole substrate is now stock main minimald + stock main minvmd + the BPF-LSM kernel + det-init, no fork. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tion hole (#194) The readiness audit (gominimal/minimal-detonation#194) found a trust-boundary regression: det-init emits the drained JSONL feed on the guest console and the worker collects it by scanning the console for `{`-lines, but the untrusted `nobody` sample's stdout/stderr were INHERITED onto that same console — so a sample could print forged `{"event":…}` lines that the worker accepts as real events, voiding the core invariant "a `nobody` sample cannot write the feed" (e.g. a forged NetworkDns re-contextualizing a real C2 connect → a signed BLOCK→PASS). The fork closed this deliberately (`1>&2` in detonate.sh); the initial det-init port dropped it. Fix: run the sample with stdout+stderr = Stdio::null() (was inherit). The console is root-owned, so nulling the sample's only inherited path to it means the feed on the console is written solely by pid-1 (det-init) and stays trustworthy. Corrected the false "nothing to corrupt" comment. Robust follow-up (noted): drain the root-0400 /run/det-events.jsonl fully out-of-band (/dev/vdb or vsock). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…not a number
`minimald = { version = "0.0.1", path = "../minimald" }` broke the build the
moment the workspace bumped to 0.5.0-rc1. Nothing about the API changed: all six
`guest::` functions det-init calls still have byte-identical signatures, and
guest.rs is additive-only since this branch's base. The pin was a self-inflicted
drift trap — it failed on a version NUMBER while the thing it nominally protected
(compatibility) was entirely fine.
A bare path dep resolves to whatever is in the tree, so version bumps can never
break us again. The API surface is what actually matters and the compiler already
checks it; a hand-maintained version string is a second, weaker check that fails
for the wrong reasons.
Also inherits version.workspace = true, matching every sibling crate.
Rebased onto current main (121 commits): the only conflict was the workspace
members list, where upstream added crates/diagnostics and we add crates/det-init.
Refs gominimal/minimal-detonation#194
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feed is drained AFTER the sample returns (step 8), so a sample that never returns takes the ENTIRE feed with it: the host reaps the VM at its timeout, drain_monitor and emit_feed_to_stdout never run, and a run that should have been BLOCK is reported as "substrate_failed: empty observer capture". The observer had captured everything — honeytoken reads included — and simply never shipped it. Found by a corpus run, not by inspection. Two executions of the SAME poisoned sample, differing only in whether the C2 connect was bounded: unbounded (/dev/tcp to a blackholed IP) -> ERROR, empty capture bounded (timeout 5 curl --max-time 4) -> BLOCK, complete, signed This is not an exotic input. A poisoned package whose C2 is unreachable — sinkholed, taken down, firewalled, or a documentation-range IP — blocks on connect() exactly like this. The perverse consequence is that the better the network defense, the more likely the entire feed is lost, because the hang is caused by the connection NOT succeeding. It is also a one-line evasion: append a sleep after the payload and BLOCK degrades to ERROR once the damage is done. Bounds the sample at SAMPLE_DEADLINE (90s, deliberately under the host's 120s VM timeout) via try_wait polling, SIGKILLs on expiry, reaps, and falls through to the normal drain. SIGKILL rather than SIGTERM because this is untrusted code that already declined to finish and can trap the softer signal into the same hang. Killing at the deadline costs only what the sample would have done AFTER it; keeping the events it already produced is strictly better than losing all of them. The timeout is logged on the console the host collects, so a hang is distinguishable in triage from a genuine boot/attach failure — the old message blamed the substrate for something the substrate did correctly. NOT addressed here, deliberately: a timed-out run yields a partial feed, which arguably should cap the verdict at REVIEW as a coverage-honesty signal rather than producing a confident verdict over truncated evidence. That needs an event/schema change and belongs in its own PR. Refs gominimal/minimal-detonation#272 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
Comment |
det-init was added to the workspace members but never entered Cargo.lock, so any `--locked` build fails outright with "cannot update the lock file". Found by actually compiling on Linux rather than by inspection — it is invisible to a non-locked local build, which happily resolves it on the fly. Also picks up version 0.5.0-rc1 from version.workspace = true (the previous commit dropped det-init's independent 0.0.1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Now verified — undrafted. Compiled on Linux via Cloud Build ( Two warnings, both pre-existing dead-code notes on The build also caught a latent bug that inspection had missed: Which is the argument for compiling rather than reasoning: I'd have shipped a PR that couldn't build under |
|
Superseded by #983 — same change, squashed to one commit. This PR was genuinely hard to read and shouldn't have been opened as-is: six commits introducing one new crate, including fixes to its own earlier commits, plus two headers over commitlint's 100-char limit (pre-existing, from the July prototype work). #983 is the same content as a single pure addition, with the Linux |
The feed is drained after the sample returns (step 8), so a sample that never returns takes the entire feed with it: the host reaps the VM at its timeout,
drain_monitorandemit_feed_to_stdoutnever run, and a run that should have beenBLOCKis reported assubstrate_failed: empty observer capture. The observer had captured everything — honeytoken reads included — and simply never shipped it.Found by a corpus run, not by inspection. Two executions of the same poisoned sample, differing only in whether the C2 connect was bounded:
/dev/tcpto a blackholed IP)ERROR— empty capturetimeout 5 curl --max-time 4)BLOCK, complete, signedNot an exotic input. A poisoned package whose C2 is unreachable — sinkholed, taken down, firewalled — blocks on
connect()exactly like this. The perverse consequence: the better the network defense, the more likely the whole feed is lost, because the hang is caused by the connection not succeeding. It's also a one-line evasion — append a sleep andBLOCKdegrades toERRORafter the payload has run.Fix
Bounds the sample at
SAMPLE_DEADLINE(90s, under the host's 120s VM timeout) viatry_waitpolling, SIGKILLs on expiry, reaps, and falls through to the normal drain. SIGKILL rather than SIGTERM because this is untrusted code that already declined to finish and can trap the softer signal into the same hang.Killing at the deadline costs only what the sample would have done after it; keeping the events it already produced is strictly better than losing all of them. The timeout is logged on the console the host collects, so a hang is distinguishable in triage from a real boot/attach failure — the old message blamed the substrate for something it did correctly.
Verification status — please read
This is not compile-verified.
det-initis Linux-only (procfs) with C deps, so it can't be built from macOS without a cross-toolchain.rustfmtparses it cleanly and the region is fmt-clean, but that is syntax, not typecheck.It also can't be validated by a golden-image bake as things stand: the bake builds det-init from the pinned substrate tarball (
source-root-detinit-b34e1be1.tgz), which predates this commit. Picking it up needs a re-stage on a Linux box withminimal.Not addressed here, deliberately
A timed-out run yields a partial feed, which arguably should cap the verdict at REVIEW as a coverage-honesty signal rather than producing a confident verdict over truncated evidence. That needs an event/schema change and belongs in its own PR.
Refs gominimal/minimal-detonation#272
Note
Add
det-initmicroVM init that emits the event feed even when the sample hangsIntroduces the
det-initbinary, which runs as pid-1 inside a detonation microVM and orchestrates the full sample execution lifecycle./dev, securityfs, and bpffs; raisesRLIMIT_MEMLOCK; then starts a root BPF-LSM observer and waits for its readiness signal before running the sampleSIGKILLso a hung sample does not stall teardown — the event feed is always drained and emitted to stdout afterward viaemit_feed_to_stdoutnobody) via thedropprivhelper, with nonce- and uid-boundary self-checks and optional DNS rerouting to a local collectorminimald::guest::shut_down_vmon completion when running as pid-1; parks if shutdown fails to avoid a kernel panicdrain_monitorbounds the observer shutdown with aSIGKILLbackstop before the feed is readMacroscope summarized 0a0cf81.