Skip to content

feat(minimald,minvmd): serve sessions directly over the vsock bridge - #374

Merged
norrietaylor merged 1 commit into
mainfrom
minvmd-stage2-vsock-session
Jun 11, 2026
Merged

feat(minimald,minvmd): serve sessions directly over the vsock bridge#374
norrietaylor merged 1 commit into
mainfrom
minvmd-stage2-vsock-session

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jun 9, 2026

Copy link
Copy Markdown
Member

Makes the initramfs guest serve a full SSH session directly on the host-bridged AF_VSOCK port — no socat relay. Stacked on #373 (merged); base main.

What

  • minimald serves SSH directly via Server::run_on_vsock on the bridged vsock port; run_initramfs drops the UDS bind + the in-guest socat relay.
  • Connection::from_stream generalises the russh driver over any byte stream (UDS or vsock).
  • minimald_session_e2e: a russh client over the bridge authenticates, creates a session, execs a command, asserts stdout + exit status — against the GENERIC rootfs.

Requires libkrun ≥ 1.19.0

On 1.18.1 the bridged vsock intermittently stalled a full session (~1 in 5). Root-caused to libkrun's vsock device mishandling multi-descriptor TX chains from Linux 6.2+ guests — fixed upstream by 0ecf4d5f7 (shipped in libkrun 1.19.0, 2026-06-10). On 1.19.0 the direct path is reliable (25/25 + 5/5 local stress runs); the earlier socat workaround is removed.

The self-hosted macOS runner must be on libkrun ≥ 1.19.0 (brew upgrade slp/krun/libkrun) for the session + autospawn e2e to pass — handled as separate infra provisioning. This PR is the transport change only and goes green once the runner is upgraded.

Validation (local, libkrun 1.19.0)

  • direct-vsock minimald_session_e2e: 25/25 then 5/5 on the exact branch build.
  • minvmd build + clippy --all-targets -D warnings + tests; minimald cross clippy (aarch64-musl) clean.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Updated E2E testing and boot docs; sessions now served directly over host vsock (requires libkrun ≥1.19.0).
  • New Features

    • Minimald serves SSH sessions over host vsock on Linux and now surfaces session handshake/transport failures in logs.
  • Tests

    • Added full-session end-to-end tests and new real-hardware autospawn/boot validation with boot-log capture and failure dumps.

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a36ab091-55ca-4508-b607-83f9a1d1cd25

📥 Commits

Reviewing files that changed from the base of the PR and between ebbd76e and 3dba685.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • .github/workflows/ci-macos.yml
  • crates/minimald/src/connection.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/server.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/tests/minimald_session_e2e.rs

📝 Walkthrough

Walkthrough

Serves SSH sessions over guest AF_VSOCK by adding a generic stream Connection, a vsock acceptor, replacing initramfs UDS startup with vsock, introducing a macOS-gated end-to-end session test, and updating CI/docs to build and run the new E2E flows.

Changes

Vsock Bridge and Session E2E

Layer / File(s) Summary
Generic async stream connection abstraction
crates/minimald/src/connection.rs
Connection::from_stream accepts any AsyncRead + AsyncWrite stream and manages russh server setup with configurable local/remote trust; Connection::from_socket refactored to wrap it.
Server-side vsock listener and config refactoring
crates/minimald/src/server.rs
New Server::run_on_vsock listens on AF_VSOCK port and serves sessions via the stream abstraction; build_russh_config centralizes russh config; UDS loop updated to await session futures and warn on errors.
Guest initramfs vsock server startup
crates/minimald/src/main.rs
Boot path switches from UDS to Server::run_on_vsock with new DEFAULT_VSOCK_PORT = 2222; docs updated and SIGCHLD reaper note added.
Full session E2E test with client harness and RPC flow
crates/minvmd/Cargo.toml, crates/minvmd/tests/minimald_session_e2e.rs
New macOS-gated test boots minvmd with minimald init, provides Guest harness and readiness detection, defines CreateSession RPC types, connects via bridge UDS, performs CreateSession subsystem RPC, then execs a command with MINIMAL_SESSION_ID and asserts stdout/exit. Dev-deps added for russh/tokio.
CI workflow and documentation updates
crates/minvmd/README.md, .github/workflows/ci-macos.yml
README shows two-step E2E (boot_e2e then minimald_session_e2e) and notes libkrun >= 1.19.0 requirement; CI boot-e2e builds both test binaries and runs session E2E; new autospawn-e2e job validates autospawn cold/warm runs with diagnostics and artifact uploads.

Sequence Diagram

sequenceDiagram
  participant CI_Test as E2E Test
  participant BridgeUDS as Host Bridge UDS
  participant LibkrunVsock as Libkrun VSOCK (port 2222)
  participant MinimalD as Guest minimald (run_on_vsock)
  participant ConnectionFromStream as Connection::from_stream
  CI_Test->>BridgeUDS: russh connect (UnixStream)
  BridgeUDS->>LibkrunVsock: forward to guest vsock port 2222
  LibkrunVsock->>MinimalD: accept vsock connection
  MinimalD->>ConnectionFromStream: spawn russh server on stream
  ConnectionFromStream->>MinimalD: RunningSession future (awaited)
  CI_Test->>MinimalD: SSH subsystem CreateSession (JSON)
  MinimalD->>CI_Test: CreateSessionResponse (session id)
  CI_Test->>MinimalD: exec channel with MINIMAL_SESSION_ID
  MinimalD->>CI_Test: command stdout + exit status
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

  • gominimal/minspec-test#46: Both add host↔guest vsock↔UDS bridging and end-to-end session roundtrip testing.

Possibly related PRs

  • gominimal/minimal#350: Switches minimald to serve sessions over AF_VSOCK and affects bridge E2E behavior.
  • gominimal/minimal#354: Adds macOS autospawn-e2e CI flow and cold/warm minimal2 checks that complement this PR’s autospawn job.
  • gominimal/minimal#373: Prior initramfs/PID‑1 plumbing that this change modifies to serve sessions over vsock.

Suggested labels

needs-human

Suggested reviewers

  • evanspearman
  • twitchyliquid64

Poem

🐰 I hopped across the vsock span,
From host to guest I drew a plan,
Directly served with no relay,
Tests proved the path — hooray, hooray!
Tiny rabbit, tiny cheer — the sessions run, the coast is clear.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: serving sessions directly over the vsock bridge in minimald and minvmd, which is the central objective of this PR.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

@norrietaylor norrietaylor changed the title feat(minimald,minvmd): serve sessions over the vsock bridge [WIP] feat(minimald,minvmd): serve sessions over the vsock bridge Jun 10, 2026
@norrietaylor
norrietaylor force-pushed the minvmd-stage2-initramfs-boot branch from c6b5b85 to e697a95 Compare June 10, 2026 19:11
@norrietaylor
norrietaylor force-pushed the minvmd-stage2-vsock-session branch from 608afa0 to ae94dbc Compare June 10, 2026 19:11
Base automatically changed from minvmd-stage2-initramfs-boot to main June 10, 2026 20:18
@norrietaylor
norrietaylor force-pushed the minvmd-stage2-vsock-session branch from ae94dbc to 296cf84 Compare June 10, 2026 20:21

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/minimald/src/connection.rs (1)

148-153: ⚡ Quick win

Consider propagating the error instead of unwrap.

The .unwrap() at line 152 will panic if russh::server::run_stream fails during initialization. While this is spawned in a task (per server.rs), an explicit error type would allow the spawning code to log session setup failures rather than silently panic the task.

♻️ Proposed error propagation

Change the return type to propagate initialization errors:

     pub(crate) async fn from_stream<S>(
         s: S,
         c: Arc<RuConfig>,
         serv: ServerStateHandle,
         is_local: bool,
-    ) -> (ConnectionHandle, RunningSession<ConnectionHandler>)
+    ) -> Result<(ConnectionHandle, RunningSession<ConnectionHandler>), russh::Error>
     where
         S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
     {
         let h = ConnectionHandle(Arc::new(Mutex::new(Self {
             auth: if is_local { Auth::Local } else { Auth::Pending },
             ssh_username: None,
             channels: BTreeMap::new(),
             serv,
         })));
 
-        (
+        Ok((
             h.clone(),
-            russh::server::run_stream(c, s, ConnectionHandler(h))
-                .await
-                .unwrap(),
-        )
+            russh::server::run_stream(c, s, ConnectionHandler(h)).await?,
+        ))
     }

Then update call sites in server.rs to handle the Result.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/connection.rs` around lines 148 - 153, The code currently
calls russh::server::run_stream(c, s, ConnectionHandler(h)).await.unwrap(),
which will panic on initialization failure; change this to propagate the error
instead of unwrapping: have the enclosing function return a Result and replace
.unwrap() with ? (or map_err to your crate error type) so the run_stream error
is returned, and update the caller(s) in server.rs to handle/log the Result
rather than assuming success; keep references to h.clone(),
russh::server::run_stream, and ConnectionHandler(h) when locating where to
change the return type and error propagation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/minimald/src/server.rs`:
- Around line 196-204: The build_russh_config function currently unwraps
state.host_key().await which can panic; change build_russh_config to return a
Result<Arc<russh::server::Config>, E> (choose the crate's Error or Box<dyn
std::error::Error>) and replace .unwrap() with ? (or map_err to add context) so
host_key errors are propagated; update all call sites that invoke
build_russh_config (the callers that currently expect Arc from
build_russh_config) to handle the Result (propagate the error or log/return it)
so initialization failures from ServerStateHandle::host_key are handled
gracefully.

---

Nitpick comments:
In `@crates/minimald/src/connection.rs`:
- Around line 148-153: The code currently calls russh::server::run_stream(c, s,
ConnectionHandler(h)).await.unwrap(), which will panic on initialization
failure; change this to propagate the error instead of unwrapping: have the
enclosing function return a Result and replace .unwrap() with ? (or map_err to
your crate error type) so the run_stream error is returned, and update the
caller(s) in server.rs to handle/log the Result rather than assuming success;
keep references to h.clone(), russh::server::run_stream, and
ConnectionHandler(h) when locating where to change the return type and error
propagation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dcd9d9db-788d-470f-be13-f8a35622cc55

📥 Commits

Reviewing files that changed from the base of the PR and between 827bca5 and 296cf84.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (8)
  • .github/workflows/ci-macos.yml
  • crates/minimald/src/connection.rs
  • crates/minimald/src/guest.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/server.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/tests/minimald_session_e2e.rs

Comment thread crates/minimald/src/server.rs
@norrietaylor
norrietaylor force-pushed the minvmd-stage2-vsock-session branch from 296cf84 to a0bad7e Compare June 10, 2026 23:25
@norrietaylor norrietaylor changed the title [WIP] feat(minimald,minvmd): serve sessions over the vsock bridge feat(minimald,minvmd): serve sessions directly over the vsock bridge Jun 10, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/minimald/src/main.rs (1)

13-14: ⚡ Quick win

Misleading "guest relay" terminology.

The doc comment refers to "guest relay," but this PR removes the in-guest socat relay in favor of minimald serving directly on vsock. Consider replacing "guest relay" with "guest server" or "minimald vsock listener" to avoid confusion with the relay architecture that is being removed.

📝 Suggested doc comment clarification
-/// Default AF_VSOCK port the guest relay listens on (the boot-contract bridge
+/// Default AF_VSOCK port minimald listens on in the guest (the boot-contract bridge
 /// port the host registers via `krun_add_vsock_port2`).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/main.rs` around lines 13 - 14, Update the doc comment
that currently calls it the "guest relay" to avoid confusion with the removed
in-guest socat relay—change phrasing above the default AF_VSOCK port constant
(the comment describing the default AF_VSOCK port the guest relay listens on) to
something like "guest server" or "minimald vsock listener" so it accurately
describes minimald serving directly on vsock.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/minimald/src/main.rs`:
- Around line 13-14: Update the doc comment that currently calls it the "guest
relay" to avoid confusion with the removed in-guest socat relay—change phrasing
above the default AF_VSOCK port constant (the comment describing the default
AF_VSOCK port the guest relay listens on) to something like "guest server" or
"minimald vsock listener" so it accurately describes minimald serving directly
on vsock.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c027f970-4f74-4dda-9d34-931c5b9bb4f3

📥 Commits

Reviewing files that changed from the base of the PR and between 296cf84 and a0bad7e.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • .github/workflows/ci-macos.yml
  • crates/minimald/src/connection.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/server.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/tests/minimald_session_e2e.rs
🚧 Files skipped from review as they are similar to previous changes (4)
  • crates/minvmd/Cargo.toml
  • crates/minimald/src/server.rs
  • crates/minimald/src/connection.rs
  • crates/minvmd/tests/minimald_session_e2e.rs

@evanspearman evanspearman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the one coderabbit comment is legitimate and worth fixing.

@norrietaylor
norrietaylor force-pushed the minvmd-stage2-vsock-session branch from a0bad7e to ebbd76e Compare June 11, 2026 15:45

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
.github/workflows/ci-macos.yml (2)

179-186: ⚡ Quick win

Missing boot log capture for session E2E debugging.

The boot_e2e step sets MINVMD_BOOT_LOG (line 171) and autospawn-e2e does too (line 266), but this step omits it. If the session E2E fails, there's no guest console output to diagnose whether the issue is boot-related or session-related.

Consider adding MINVMD_BOOT_LOG="$RUNNER_TEMP/session-boot.log" for consistency and debuggability.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-macos.yml around lines 179 - 186, Add setting
MINVMD_BOOT_LOG for the session E2E run so the guest console is captured for
debugging; in the run block where testbin (resolved from
target/debug/deps/minimald_session_e2e-*) is executed with MINVMD_E2E and
MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH/MINVMD_INITRAMFS, also export
MINVMD_BOOT_LOG="$RUNNER_TEMP/session-boot.log" before invoking the test binary
(the one that runs minimald_session_e2e and the minimald_exec_over_bridge test)
to match boot_e2e and autospawn-e2e behavior.

267-272: 💤 Low value

Temp directory for XDG_RUNTIME_DIR is never cleaned up.

The temp directory created at line 267 is not removed on exit. The EXIT trap only stops minvmd. On a persistent self-hosted runner, this leaks a temp directory per run.

Consider adding cleanup to the trap:

🧹 Suggested fix
           export XDG_RUNTIME_DIR="$(mktemp -d)"
+          _xdg_cleanup() { rm -rf "$XDG_RUNTIME_DIR" 2>/dev/null || true; }
           # ...
-          trap 'minvmd stop >/dev/null 2>&1 || true' EXIT
+          trap 'minvmd stop >/dev/null 2>&1 || true; _xdg_cleanup' EXIT
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-macos.yml around lines 267 - 272, The temp dir created
for XDG_RUNTIME_DIR is never removed; store the result of mktemp -d in a
variable (e.g., TMP_XDG_RUNTIME_DIR), export
XDG_RUNTIME_DIR="$TMP_XDG_RUNTIME_DIR", and extend the existing EXIT trap that
currently stops minvmd to also remove that temp directory (rm -rf
"$TMP_XDG_RUNTIME_DIR") so the temp dir is cleaned up on script exit; ensure the
trap covers all exit signals as before and that the variable name matches where
XDG_RUNTIME_DIR is referenced.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci-macos.yml:
- Around line 173-178: Update the stale workflow comment that still mentions a
"socat vsock→UDS relay + run_on_uds" path: replace that description with the
current design where minimald/Server::run_on_vsock serves SSH directly over
vsock (no in-guest socat relay). Locate the block around the Session E2E comment
and change the text to state the direct vsock path (Server::run_on_vsock) and
remove references to run_on_uds/socat so the comment matches the PR objective.

---

Nitpick comments:
In @.github/workflows/ci-macos.yml:
- Around line 179-186: Add setting MINVMD_BOOT_LOG for the session E2E run so
the guest console is captured for debugging; in the run block where testbin
(resolved from target/debug/deps/minimald_session_e2e-*) is executed with
MINVMD_E2E and MINVMD_KERNEL_PATH/MINVMD_ROOTFS_PATH/MINVMD_INITRAMFS, also
export MINVMD_BOOT_LOG="$RUNNER_TEMP/session-boot.log" before invoking the test
binary (the one that runs minimald_session_e2e and the minimald_exec_over_bridge
test) to match boot_e2e and autospawn-e2e behavior.
- Around line 267-272: The temp dir created for XDG_RUNTIME_DIR is never
removed; store the result of mktemp -d in a variable (e.g.,
TMP_XDG_RUNTIME_DIR), export XDG_RUNTIME_DIR="$TMP_XDG_RUNTIME_DIR", and extend
the existing EXIT trap that currently stops minvmd to also remove that temp
directory (rm -rf "$TMP_XDG_RUNTIME_DIR") so the temp dir is cleaned up on
script exit; ensure the trap covers all exit signals as before and that the
variable name matches where XDG_RUNTIME_DIR is referenced.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1066c98f-dad7-4650-99b0-3f6b3950d9e5

📥 Commits

Reviewing files that changed from the base of the PR and between a0bad7e and ebbd76e.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (7)
  • .github/workflows/ci-macos.yml
  • crates/minimald/src/connection.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/server.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/tests/minimald_session_e2e.rs
🚧 Files skipped from review as they are similar to previous changes (6)
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minimald/src/connection.rs
  • crates/minimald/src/server.rs
  • crates/minimald/src/main.rs
  • crates/minvmd/tests/minimald_session_e2e.rs

Comment thread .github/workflows/ci-macos.yml Outdated
Make the initramfs guest serve a full SSH session directly on the
host-bridged AF_VSOCK port via `Server::run_on_vsock` — no socat relay.
`Connection::from_stream` generalises the SSH driver over any byte
stream (UDS or vsock).

Requires libkrun >= 1.19.0: on 1.18.1 the bridged vsock intermittently
stalled a full session (a multi-descriptor TX-chain bug in libkrun's
vsock device for Linux 6.2+ guests, fixed upstream by 0ecf4d5f7); the
earlier workaround was an in-guest socat vsock->UDS relay, now removed.

- minimald: `run_initramfs` serves via `run_on_vsock` (drops the UDS
  bind + socat relay); `Server::run_on_vsock` + `from_stream` drive
  russh over the bridged vsock stream and log session errors.
- minvmd: `minimald_session_e2e` — a russh client over the bridge
  authenticates, creates a session, and execs a command, asserting
  stdout + exit status, against the GENERIC rootfs.

Validated locally: direct-vsock session e2e 25/25 + 5/5 on libkrun
1.19.0; minvmd build/clippy/tests + minimald cross-clippy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants