Skip to content

fix(det-init): a hung sample must not discard the whole event feed - #981

Closed
bryan-minimal wants to merge 7 commits into
mainfrom
det/det-init-prototype
Closed

fix(det-init): a hung sample must not discard the whole event feed#981
bryan-minimal wants to merge 7 commits into
mainfrom
det/det-init-prototype

Conversation

@bryan-minimal

@bryan-minimal bryan-minimal commented Jul 27, 2026

Copy link
Copy Markdown
Member

The feed is drained after the sample returns (step 8), so a sample that never returns takes the entire feed with it: the host reaps the VM at its timeout, drain_monitor and emit_feed_to_stdout never run, and a run that should have been BLOCK is reported as substrate_failed: empty observer capture. The observer had captured everything — honeytoken reads included — and simply never shipped it.

Found by a corpus run, not by inspection. Two executions of the same poisoned sample, differing only in whether the C2 connect was bounded:

C2 connect Result
unbounded (/dev/tcp to a blackholed IP) ERROR — empty capture
bounded (timeout 5 curl --max-time 4) BLOCK, complete, signed

Not an exotic input. A poisoned package whose C2 is unreachable — sinkholed, taken down, firewalled — blocks on connect() exactly like this. The perverse consequence: the better the network defense, the more likely the whole feed is lost, because the hang is caused by the connection not succeeding. It's also a one-line evasion — append a sleep and BLOCK degrades to ERROR after the payload has run.

Fix

Bounds the sample at SAMPLE_DEADLINE (90s, under the host's 120s VM timeout) via try_wait polling, SIGKILLs on expiry, reaps, and falls through to the normal drain. SIGKILL rather than SIGTERM because this is untrusted code that already declined to finish and can trap the softer signal into the same hang.

Killing at the deadline costs only what the sample would have done after it; keeping the events it already produced is strictly better than losing all of them. The timeout is logged on the console the host collects, so a hang is distinguishable in triage from a real boot/attach failure — the old message blamed the substrate for something it did correctly.

Verification status — please read

This is not compile-verified. det-init is Linux-only (procfs) with C deps, so it can't be built from macOS without a cross-toolchain. rustfmt parses it cleanly and the region is fmt-clean, but that is syntax, not typecheck.

It also can't be validated by a golden-image bake as things stand: the bake builds det-init from the pinned substrate tarball (source-root-detinit-b34e1be1.tgz), which predates this commit. Picking it up needs a re-stage on a Linux box with minimal.

Not addressed here, deliberately

A timed-out run yields a partial feed, which arguably should cap the verdict at REVIEW as a coverage-honesty signal rather than producing a confident verdict over truncated evidence. That needs an event/schema change and belongs in its own PR.

Refs gominimal/minimal-detonation#272

Note

Add det-init microVM init that emits the event feed even when the sample hangs

Introduces the det-init binary, which runs as pid-1 inside a detonation microVM and orchestrates the full sample execution lifecycle.

  • Mounts /dev, securityfs, and bpffs; raises RLIMIT_MEMLOCK; then starts a root BPF-LSM observer and waits for its readiness signal before running the sample
  • Enforces a 90-second sample deadline with SIGKILL so a hung sample does not stall teardown — the event feed is always drained and emitted to stdout afterward via emit_feed_to_stdout
  • Runs the untrusted sample as uid 65534 (nobody) via the droppriv helper, with nonce- and uid-boundary self-checks and optional DNS rerouting to a local collector
  • Shuts down the guest VM via minimald::guest::shut_down_vm on completion when running as pid-1; parks if shutdown fails to avoid a kernel panic
  • Behavioral Change: a hung or killed sample no longer causes the JSONL event feed to be discarded — drain_monitor bounds the observer shutdown with a SIGKILL backstop before the feed is read

Macroscope summarized 0a0cf81.

bryan-minimal and others added 6 commits July 27, 2026 09:26
New `crates/det-init` binary — the minimal-detonation guest /init. It reuses
`minimald::guest::*` (mount_dev / enter_rootfs / mount_state_volume /
bring_up_root_egress / shut_down_vm) for the pid-1 boot plumbing instead of
forking minimald, then stands up the in-guest BPF-LSM observer and runs the
untrusted sample as `nobody` under it, one-shot — no SSH server, no session,
no dependency on the arbitrary-exec surface #709 removed.

Ports the proven detonate.sh + init-launcher.sh flow: securityfs/bpffs mounts,
deny-channel tripwire staged pre-attach, per-scan binding-nonce load (observer-
only) with a nobody-can't-read leak-check, observer ready-file handshake, DNS
collector repoint, uid-boundary self-check, droppriv fork-and-reap of the
sample, bounded observer drain. Observer streams the JSONL feed over vsock
(file fallback); host runs `minimal-detonation verdict`.

Prototype: compiles/boots on Linux/KVM (minimald's guest mounts are Linux-only).
Runtime contract (host stages sample+nonce, bake stages observer/droppriv/
tripwire/honeytokens) documented as path constants. See gominimal/minimal-detonation#194.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… DNS — boots+detonates on KVM (#194)

Proven end-to-end on x86 KVM (spike1-nestedvirt): det-init boots as pid-1,
enters the rootfs via minimald's embedded guest lib, stands up the observer
(16/16 LSM hooks), runs the sample as nobody, and verdicts BLOCK on a malicious
honeytoken-read / PASS on a benign run.

Changes from the initial scaffold, each earned by a boot iteration:
- emit the READY vsock beacon (guest::emit_simple_ready_marker) right after
  entering the rootfs — the minvmd supervisor waits for it (~5s) and tears the
  guest down otherwise; it must fire before the observer-ready wait (up to 10s).
- emit the drained feed on stdout so a host collecting via the guest console
  (the boot harness) gets it without a vsock listener.
- file sink only: drop --vsock-port from the observer spawn. It made the
  observer BLOCK connecting its sink to an absent host vsock listener, so it
  never signalled ready and the one-shot boot stalled.
- gate the DNS-collector repoint on egress being up: with no network the
  bind-mount is pointless AND showed up as a spurious pid-1 Mount the verdict
  read as a container-escape false-positive on an otherwise-benign run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…iner-escape FP on stock minvmd (#194)

Booting on minvmd built from main (which, unlike the fork binary, sets up the
own-ip vsock shuttle so egress reports up) surfaced that det-init's own
resolv.conf BIND-MOUNT — done as pid-1 — is captured as an sb_mount event the
verdict scores as a container-escape, false-blocking an otherwise-benign run.

Repoint /etc/resolv.conf with a plain write instead (matching the legacy
detonate.sh runner). On a read-only rootfs it fails EROFS and degrades to "DNS
names unobserved" rather than mounting. Full DNS-name capture on a RO rootfs (a
writable /etc overlay) is a networked-detonation follow-up; the robust fix is
verdict-side process-subtree scoping so pid-1 infrastructure never counts toward
the sample verdict.

PROVEN on STOCK-main minvmd (built from origin/main, no MINVMD_KERNEL_FORMAT
override — main auto-picks ImageGz for x86_64 — attaching the #672 /dev/vdb
volume): malicious honeytoken-read -> BLOCK, benign -> PASS. The whole substrate
is now stock main minimald + stock main minvmd + the BPF-LSM kernel + det-init,
no fork.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tion hole (#194)

The readiness audit (gominimal/minimal-detonation#194) found a trust-boundary
regression: det-init emits the drained JSONL feed on the guest console and the
worker collects it by scanning the console for `{`-lines, but the untrusted
`nobody` sample's stdout/stderr were INHERITED onto that same console — so a
sample could print forged `{"event":…}` lines that the worker accepts as real
events, voiding the core invariant "a `nobody` sample cannot write the feed"
(e.g. a forged NetworkDns re-contextualizing a real C2 connect → a signed
BLOCK→PASS). The fork closed this deliberately (`1>&2` in detonate.sh); the
initial det-init port dropped it.

Fix: run the sample with stdout+stderr = Stdio::null() (was inherit). The console
is root-owned, so nulling the sample's only inherited path to it means the feed
on the console is written solely by pid-1 (det-init) and stays trustworthy.
Corrected the false "nothing to corrupt" comment. Robust follow-up (noted):
drain the root-0400 /run/det-events.jsonl fully out-of-band (/dev/vdb or vsock).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…not a number

`minimald = { version = "0.0.1", path = "../minimald" }` broke the build the
moment the workspace bumped to 0.5.0-rc1. Nothing about the API changed: all six
`guest::` functions det-init calls still have byte-identical signatures, and
guest.rs is additive-only since this branch's base. The pin was a self-inflicted
drift trap — it failed on a version NUMBER while the thing it nominally protected
(compatibility) was entirely fine.

A bare path dep resolves to whatever is in the tree, so version bumps can never
break us again. The API surface is what actually matters and the compiler already
checks it; a hand-maintained version string is a second, weaker check that fails
for the wrong reasons.

Also inherits version.workspace = true, matching every sibling crate.

Rebased onto current main (121 commits): the only conflict was the workspace
members list, where upstream added crates/diagnostics and we add crates/det-init.

Refs gominimal/minimal-detonation#194

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feed is drained AFTER the sample returns (step 8), so a sample that never
returns takes the ENTIRE feed with it: the host reaps the VM at its timeout,
drain_monitor and emit_feed_to_stdout never run, and a run that should have been
BLOCK is reported as "substrate_failed: empty observer capture". The observer had
captured everything — honeytoken reads included — and simply never shipped it.

Found by a corpus run, not by inspection. Two executions of the SAME poisoned
sample, differing only in whether the C2 connect was bounded:

  unbounded (/dev/tcp to a blackholed IP)  -> ERROR, empty capture
  bounded   (timeout 5 curl --max-time 4)  -> BLOCK, complete, signed

This is not an exotic input. A poisoned package whose C2 is unreachable —
sinkholed, taken down, firewalled, or a documentation-range IP — blocks on
connect() exactly like this. The perverse consequence is that the better the
network defense, the more likely the entire feed is lost, because the hang is
caused by the connection NOT succeeding. It is also a one-line evasion: append a
sleep after the payload and BLOCK degrades to ERROR once the damage is done.

Bounds the sample at SAMPLE_DEADLINE (90s, deliberately under the host's 120s VM
timeout) via try_wait polling, SIGKILLs on expiry, reaps, and falls through to the
normal drain. SIGKILL rather than SIGTERM because this is untrusted code that
already declined to finish and can trap the softer signal into the same hang.

Killing at the deadline costs only what the sample would have done AFTER it;
keeping the events it already produced is strictly better than losing all of them.
The timeout is logged on the console the host collects, so a hang is
distinguishable in triage from a genuine boot/attach failure — the old message
blamed the substrate for something the substrate did correctly.

NOT addressed here, deliberately: a timed-out run yields a partial feed, which
arguably should cap the verdict at REVIEW as a coverage-honesty signal rather than
producing a confident verdict over truncated evidence. That needs an event/schema
change and belongs in its own PR.

Refs gominimal/minimal-detonation#272

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a382de05-9101-464c-be5d-755fe1b8f3af

📥 Commits

Reviewing files that changed from the base of the PR and between b1cde76 and 0a0cf81.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (3)
  • Cargo.toml
  • crates/det-init/Cargo.toml
  • crates/det-init/src/main.rs

Comment @coderabbitai help to get the list of available commands.

@bryan-minimal
bryan-minimal marked this pull request as draft July 27, 2026 19:06
det-init was added to the workspace members but never entered Cargo.lock, so any
`--locked` build fails outright with "cannot update the lock file". Found by
actually compiling on Linux rather than by inspection — it is invisible to a
non-locked local build, which happily resolves it on the fly.

Also picks up version 0.5.0-rc1 from version.workspace = true (the previous commit
dropped det-init's independent 0.0.1).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bryan-minimal
bryan-minimal marked this pull request as ready for review July 27, 2026 19:20
@bryan-minimal

Copy link
Copy Markdown
Member Author

Now verified — undrafted.

Compiled on Linux via Cloud Build (rust:1.97-bookworm, matching the repo's pinned toolchain):

rustc 1.97.0 (2d8144b78 2026-07-07)
    Checking det-init v0.5.0-rc1 (/workspace/crates/det-init)
    Finished `dev` profile ... in 7m 08s

Two warnings, both pre-existing dead-code notes on MainError, unrelated to this change.

The build also caught a latent bug that inspection had missed: det-init was added to the workspace members but never entered Cargo.lock, so any --locked build failed outright with cannot update the lock file. That's invisible to a non-locked local build, which resolves it on the fly. Fixed in the follow-up commit — it would have broken CI here regardless of this PR.

Which is the argument for compiling rather than reasoning: I'd have shipped a PR that couldn't build under --locked, for a reason entirely unrelated to the change I was making.

@bryan-minimal

Copy link
Copy Markdown
Member Author

Superseded by #983 — same change, squashed to one commit.

This PR was genuinely hard to read and shouldn't have been opened as-is: six commits introducing one new crate, including fixes to its own earlier commits, plus two headers over commitlint's 100-char limit (pre-existing, from the July prototype work).

#983 is the same content as a single pure addition, with the Linux cargo check --locked result and the live PASS/BLOCK detonation stated up front.

@bryan-minimal
bryan-minimal deleted the det/det-init-prototype branch July 30, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant