Skip to content

fix!: fix socket/path coordination amongst minimal/minvmd/minimald - #690

Merged
twitchyliquid64 merged 3 commits into
mainfrom
tom/release2
Jul 9, 2026
Merged

fix!: fix socket/path coordination amongst minimal/minvmd/minimald#690
twitchyliquid64 merged 3 commits into
mainfrom
tom/release2

Conversation

@twitchyliquid64

@twitchyliquid64 twitchyliquid64 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Bigggg PR to fix all the cross coordination issues across minimal, minvmd, and minimald.

Paths

Everything derived from minimal_state_dir(). Local stuff uses the provider dir, which is state dir/providers/<kind>-<instance num>.

Sockets have consistent names: ssh.sock.

Race-free + death-detecting autospawn

flock now used to concretely determine if the vm supervisor is alive: minvmd.lock for liveness, lifecycle.lock for state transistions in the provider.

Plumbing for state_dir

Now consistent across all binaries.

Summary by CodeRabbit

  • New Features

    • Added an optional global --minimal-state-dir to control where minvmd/minimald state and sockets are stored.
    • Standardized the per-instance directory layout for consistent UDS socket connectivity.
  • Bug Fixes

    • Improved reliability of start/stop/status using lock and liveness checks to automatically repair stale lifecycle state and make stop idempotent.
    • Strengthened startup readiness and single-instance enforcement, including fail-fast behavior for detached runs.
  • Documentation

    • Updated Linux/KVM “Gotchas” and boot notes to reflect minvmd.toml and READY-marker behavior.

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 42d5e115-3b9d-4c83-a989-3e384a028ac9

📥 Commits

Reviewing files that changed from the base of the PR and between 7e6a615 and 44d55f3.

📒 Files selected for processing (8)
  • crates/minimald/src/main.rs
  • crates/minvmd/src/cmd/boot.rs
  • crates/minvmd/src/cmd/run.rs
  • crates/minvmd/src/cmd/status.rs
  • crates/minvmd/src/cmd/stop.rs
  • crates/minvmd/src/cmd/vmm_child.rs
  • crates/minvmd/src/state.rs
  • crates/minvmd/src/volume.rs
✅ Files skipped from review due to trivial changes (1)
  • crates/minvmd/src/cmd/vmm_child.rs
🚧 Files skipped from review as they are similar to previous changes (7)
  • crates/minvmd/src/volume.rs
  • crates/minvmd/src/cmd/stop.rs
  • crates/minimald/src/main.rs
  • crates/minvmd/src/cmd/run.rs
  • crates/minvmd/src/cmd/status.rs
  • crates/minvmd/src/state.rs
  • crates/minvmd/src/cmd/boot.rs

📝 Walkthrough

Walkthrough

This PR moves runtime state, sockets, locks, and volumes to provider-instance directories, adds alive-lock lifecycle handling and single-instance checks, wires --minimal-state-dir, removes vmm.pid management, and updates CI and end-to-end tests.

Changes

Provider-instance state/socket refactor

Layer / File(s) Summary
Paths and state foundation
crates/paths/src/lib.rs, crates/minvmd/src/state.rs, crates/minvmd/src/lifecycle.rs
State resolution, provider-instance helpers, runtime filenames, minvmd.toml, lifecycle activity, and alive-lock repair are added.
minvmd command lifecycle
crates/minvmd/src/main.rs, crates/minvmd/src/cmd/{run,boot,status,stop}.rs
Commands use alive-lock gating, effective state, early socket validation, state-dir forwarding, and no longer manage vmm.pid files.
Minimal client and autospawn wiring
crates/minimal/src/{client,autospawn,lib}.rs
Socket resolution and autospawn use provider-instance paths and propagate the optional state-directory override.
minimald locking
crates/minimald/Cargo.toml, crates/minimald/src/main.rs
Minimald uses provider-instance paths, descriptor-based instance locking, lock-aware detached readiness, and minvmd lock detection.
Socket, volume, and path validation
crates/minvmd/src/{sock,volume,vm}.rs, crates/minvmd/src/cmd/{mod,vmm_child}.rs, crates/minvmd/src/net/shuttle.rs
Bridge sockets, known_hosts, and data volumes use provider directories, with UDS length validation and updated documentation.
CI and end-to-end updates
.github/workflows/*, crates/minvmd/tests/*, crates/minvmd/README.md
CI diagnostics and tests use short temporary state directories and provider-instance socket paths.

Estimated code review effort: 4 (Complex) | ~75 minutes

Possibly related issues

  • feature: minvmd VM provider host daemon #311: Covers the minvmd lifecycle, state-directory, socket, and locking architecture updated here.
  • gominimal/inbox#151: Covers related daemon lifecycle, state-directory, socket discovery, and locking work.

Possibly related PRs

Suggested labels: needs-human

Suggested reviewers: msample, norrietaylor, evanspearman

Poem

I hop through locks and paths so neat,
Past minvmd.toml with tiny feet.
Provider dirs keep state in place,
No stale crumbs remain to chase.
🐇🔒

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately summarizes the main socket/path coordination changes across minimal, minvmd, and minimald.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
crates/paths/src/lib.rs (1)

1458-1469: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Env-mutating test can data-race other parallel tests.

set_var/remove_var mutate process-global env, and cargo test runs tests as threads in one process. Any concurrent test that reads env via getenv (e.g. tempfile::tempdir() consulting TMPDIR, or another minimal_state_dir() caller) races with this set_var — exactly the unsoundness the unsafe requirement flags. The comment addresses value restoration but not the concurrency window.

Consider serializing env-mutating tests (a shared Mutex/serial guard) or using a scoped helper like temp-env to set the variable only for the closure.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/paths/src/lib.rs` around lines 1458 - 1469, The test
minimal_state_dir_honors_xdg_state_home mutates process-global environment state
with set_var/remove_var, which can race with parallel tests. Update this test to
serialize env-mutating access using a shared guard/Mutex or switch to a scoped
helper such as temp-env so XDG_STATE_HOME is only overridden within the test’s
closure; keep the fix localized around minimal_state_dir and the env
save/restore logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci-macos.yml:
- Around line 238-241: The macOS CI job creates a new temporary XDG_STATE_HOME
with mktemp -d in the local provider setup, but the current EXIT trap only stops
the daemon and never removes that directory, so temp state accumulates on the
persistent self-hosted runner. Update the cleanup logic in the same workflow
block that sets XDG_STATE_HOME so the temp directory is removed on exit,
alongside the existing daemon shutdown. Apply the same fix in both local
provider setup sections referenced by the comment, using the existing
trap/cleanup flow around the provider startup steps.

In `@crates/minimald/src/main.rs`:
- Around line 101-103: Guard the Unix domain socket path used by listen_on()
before binding, because
client_instance_dir().sub_path_unchecked(paths::SSH_SOCK_FILE) can produce a
path longer than sockaddr_un.sun_path and cause UnixListener::bind to fail on
long state/temp directories. Add a length check in the listen_on()/socket-path
construction flow and either reject overly long paths with a clear error or
shorten the generated SSH socket path while keeping the existing
client_instance_dir and SSH_SOCK_FILE lookup logic intact.

---

Nitpick comments:
In `@crates/paths/src/lib.rs`:
- Around line 1458-1469: The test minimal_state_dir_honors_xdg_state_home
mutates process-global environment state with set_var/remove_var, which can race
with parallel tests. Update this test to serialize env-mutating access using a
shared guard/Mutex or switch to a scoped helper such as temp-env so
XDG_STATE_HOME is only overridden within the test’s closure; keep the fix
localized around minimal_state_dir and the env save/restore logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cfc45675-0844-4539-850b-f07060b733ec

📥 Commits

Reviewing files that changed from the base of the PR and between 71b364b and 37359f0.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (22)
  • .github/workflows/ci-linux-kvm.yml
  • .github/workflows/ci-macos.yml
  • crates/minimal/src/autospawn.rs
  • crates/minimal/src/client.rs
  • crates/minimal/src/lib.rs
  • crates/minimald/Cargo.toml
  • crates/minimald/src/main.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/src/cmd/boot.rs
  • crates/minvmd/src/cmd/mod.rs
  • crates/minvmd/src/cmd/run.rs
  • crates/minvmd/src/cmd/status.rs
  • crates/minvmd/src/cmd/stop.rs
  • crates/minvmd/src/lifecycle.rs
  • crates/minvmd/src/main.rs
  • crates/minvmd/src/net/shuttle.rs
  • crates/minvmd/src/sock.rs
  • crates/minvmd/src/state.rs
  • crates/minvmd/tests/bridge_e2e.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
  • crates/paths/src/lib.rs

Comment thread .github/workflows/ci-macos.yml Outdated
Comment on lines +238 to +241
# All state (minvmd.toml, locks, the bridge socket) lives under
# $XDG_STATE_HOME/minimal/providers/local-0; a fresh dir gives a
# genuinely clean (NotProvisioned) state on the persistent runner.
export XDG_STATE_HOME="$(mktemp -d)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Orphaned mktemp -d state dirs accumulate on the persistent self-hosted runner.

This job runs on [self-hosted, macOS, ARM64] (a persistent runner, not an ephemeral GitHub-hosted one). Each run now creates a brand-new XDG_STATE_HOME temp directory but nothing ever removes it — the EXIT trap only stops the daemon. Over many CI runs these directories will accumulate indefinitely and can eventually exhaust disk space on the runner.

🧹 Proposed fix: clean up the temp state dir on exit
-                  trap 'minvmd stop >/dev/null 2>&1 || true' EXIT
+                  trap 'minvmd stop >/dev/null 2>&1 || true; rm -rf "$XDG_STATE_HOME"' EXIT

Also applies to: 252-261

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci-macos.yml around lines 238 - 241, The macOS CI job
creates a new temporary XDG_STATE_HOME with mktemp -d in the local provider
setup, but the current EXIT trap only stops the daemon and never removes that
directory, so temp state accumulates on the persistent self-hosted runner.
Update the cleanup logic in the same workflow block that sets XDG_STATE_HOME so
the temp directory is removed on exit, alongside the existing daemon shutdown.
Apply the same fix in both local provider setup sections referenced by the
comment, using the existing trap/cleanup flow around the provider startup steps.

Comment thread crates/minimald/src/main.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/minvmd/tests/boot_e2e.rs (1)

24-31: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider deduplicating short_state_dir() across test binaries.

The same helper (same name, logic, and rationale comment) appears to be duplicated in minimald_session_e2e.rs and bridge_e2e.rs as well. Since each file under tests/ compiles as an independent binary, a shared tests/common/mod.rs (or a small #[path]-included module) would keep the sun_path overflow rationale and prefix in one place instead of three.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minvmd/tests/boot_e2e.rs` around lines 24 - 31, The short_state_dir()
helper is duplicated across multiple test binaries, so consolidate it into a
shared test module instead of keeping the same logic and rationale comment in
boot_e2e.rs, minimald_session_e2e.rs, and bridge_e2e.rs. Move the helper into a
common location such as tests/common/mod.rs or a small #[path]-included module,
then update the affected test files to import and use that shared
short_state_dir() so the tempdir prefix and sun_path overflow workaround stay
consistent in one place.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/minvmd/tests/boot_e2e.rs`:
- Around line 24-31: The short_state_dir() helper is duplicated across multiple
test binaries, so consolidate it into a shared test module instead of keeping
the same logic and rationale comment in boot_e2e.rs, minimald_session_e2e.rs,
and bridge_e2e.rs. Move the helper into a common location such as
tests/common/mod.rs or a small #[path]-included module, then update the affected
test files to import and use that shared short_state_dir() so the tempdir prefix
and sun_path overflow workaround stay consistent in one place.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 55a1cf68-219d-4dc3-b880-f96bfb0132a0

📥 Commits

Reviewing files that changed from the base of the PR and between 37359f0 and e80681f.

📒 Files selected for processing (8)
  • .github/workflows/ci-macos.yml
  • crates/minvmd/src/cmd/boot.rs
  • crates/minvmd/src/cmd/run.rs
  • crates/minvmd/src/sock.rs
  • crates/minvmd/src/vm.rs
  • crates/minvmd/tests/boot_e2e.rs
  • crates/minvmd/tests/bridge_e2e.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
🚧 Files skipped from review as they are similar to previous changes (6)
  • crates/minvmd/tests/bridge_e2e.rs
  • .github/workflows/ci-macos.yml
  • crates/minvmd/src/cmd/boot.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
  • crates/minvmd/src/sock.rs
  • crates/minvmd/src/cmd/run.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/minimald/src/main.rs`:
- Around line 266-289: The readiness check in the detached startup loop is too
weak because `UnixStream::connect` plus `MINIMALD_LOCK_FILE` can succeed before
the child has finished conflict checks and bound its own socket. Update the
`main` startup polling logic to use a child-specific readiness signal tied to
the child process itself—such as having the child write its PID only after
binding and verifying `MINVMD_LOCK_FILE`, and having the parent verify that
signal before returning success. Keep the existing `child.try_wait()` failure
path, but make the success path depend on this post-bind readiness signal
instead of the shared `ssh.sock`/instance-lock combination.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7cb62536-76f6-44ac-925d-c6daf2366378

📥 Commits

Reviewing files that changed from the base of the PR and between e80681f and 7e6a615.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (26)
  • .github/workflows/ci-linux-kvm.yml
  • .github/workflows/ci-macos.yml
  • crates/minimal/src/autospawn.rs
  • crates/minimal/src/client.rs
  • crates/minimal/src/lib.rs
  • crates/minimald/Cargo.toml
  • crates/minimald/src/main.rs
  • crates/minvmd/Cargo.toml
  • crates/minvmd/README.md
  • crates/minvmd/src/cmd/boot.rs
  • crates/minvmd/src/cmd/mod.rs
  • crates/minvmd/src/cmd/run.rs
  • crates/minvmd/src/cmd/status.rs
  • crates/minvmd/src/cmd/stop.rs
  • crates/minvmd/src/cmd/vmm_child.rs
  • crates/minvmd/src/lifecycle.rs
  • crates/minvmd/src/main.rs
  • crates/minvmd/src/net/shuttle.rs
  • crates/minvmd/src/sock.rs
  • crates/minvmd/src/state.rs
  • crates/minvmd/src/vm.rs
  • crates/minvmd/src/volume.rs
  • crates/minvmd/tests/boot_e2e.rs
  • crates/minvmd/tests/bridge_e2e.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
  • crates/paths/src/lib.rs
✅ Files skipped from review due to trivial changes (5)
  • .github/workflows/ci-linux-kvm.yml
  • crates/minvmd/src/cmd/vmm_child.rs
  • crates/minvmd/src/lifecycle.rs
  • crates/minvmd/src/net/shuttle.rs
  • crates/minvmd/README.md
🚧 Files skipped from review as they are similar to previous changes (17)
  • crates/minvmd/src/main.rs
  • crates/minvmd/tests/boot_e2e.rs
  • crates/minvmd/tests/bridge_e2e.rs
  • crates/minimal/src/lib.rs
  • crates/minimald/Cargo.toml
  • crates/minvmd/src/vm.rs
  • crates/minvmd/src/cmd/mod.rs
  • crates/minvmd/src/sock.rs
  • .github/workflows/ci-macos.yml
  • crates/minvmd/Cargo.toml
  • crates/paths/src/lib.rs
  • crates/minvmd/src/cmd/status.rs
  • crates/minimal/src/client.rs
  • crates/minvmd/src/cmd/stop.rs
  • crates/minimal/src/autospawn.rs
  • crates/minvmd/tests/minimald_session_e2e.rs
  • crates/minvmd/src/cmd/run.rs

Comment thread crates/minimald/src/main.rs
@twitchyliquid64
twitchyliquid64 merged commit ec320d6 into main Jul 9, 2026
15 checks passed
@twitchyliquid64
twitchyliquid64 deleted the tom/release2 branch July 9, 2026 23:44
norrietaylor added a commit that referenced this pull request Jul 11, 2026
`just dm3` bridged the CLI socket with `ln -sf <runtime>/minimal/
minimald.sock <state>/providers/local-0/ssh.sock`. Since the socket/path
coordination fix (#690) minvmd binds that <state> ssh.sock DIRECTLY, so
the symlink clobbers minvmd's own live socket with a link to a stale
runtime path — every `minimal` dial then falls through to a native
autospawn that exits 1, failing the recipe on an otherwise-healthy VM.

Drop the symlink block (minvmd's direct bind is already the exact path
the CLI dials) and raise MINVMD_READY_TIMEOUT_SECS to 150 (overridable):
the generic guest kernel can spend 40-50s probing hardware before pid-1
(minimald) starts, overrunning minvmd's 60s READY default on a cold boot.

Validated by the equivalent manual bring-up: `minvmd run --detach`
without the symlink, then `minimal ls` reaches the VM on the first try.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTLRizuWQ3GuZmBBv27Nt7
norrietaylor added a commit that referenced this pull request Jul 13, 2026
* fix(justfile): retry flaky guest-artifact materialize on macOS

`just artifacts` runs `minimal materialize` through the shim, which
executes in a VM and syncs outputs back over an overlay. On a cold cache
that sync can transiently drop the output file, failing the very first
`just dm1` with:

  copying output file I/O error at path …/usr/share/virtio-linux/Image:
  No such file or directory (os error 2)

Re-running succeeds. Wrap the two macOS materialize calls in a small
retry helper (up to 3 attempts, clearing the partial output between
tries) so bring-up survives the flake instead of aborting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E6SWKgFLoRffVAUrkAWtz6

* refactor(justfile): fold the up bring-up recipe into dm1

`dm1` was a macOS gate that just called `just up`, and `up` was the
shared bring-up body. Since `up`'s only caller was `dm1` and `dm3` has
its own body, inline `up` into `dm1` and drop the `up` recipe. `dm1` is
already macOS-gated, so the inlined body sheds the Linux `LD_LIBRARY_PATH`
branch. Build steps are invoked in the recipe body (not as recipe deps)
to preserve the clean Linux SKIP. Comments referencing `just up` now
point at `dm1`/`dm3`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E6SWKgFLoRffVAUrkAWtz6

* fix(justfile): repair dm3 bring-up after the #690 socket move

`just dm3` bridged the CLI socket with `ln -sf <runtime>/minimal/
minimald.sock <state>/providers/local-0/ssh.sock`. Since the socket/path
coordination fix (#690) minvmd binds that <state> ssh.sock DIRECTLY, so
the symlink clobbers minvmd's own live socket with a link to a stale
runtime path — every `minimal` dial then falls through to a native
autospawn that exits 1, failing the recipe on an otherwise-healthy VM.

Drop the symlink block (minvmd's direct bind is already the exact path
the CLI dials) and raise MINVMD_READY_TIMEOUT_SECS to 150 (overridable):
the generic guest kernel can spend 40-50s probing hardware before pid-1
(minimald) starts, overrunning minvmd's 60s READY default on a cold boot.

Validated by the equivalent manual bring-up: `minvmd run --detach`
without the symlink, then `minimal ls` reaches the VM on the first try.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JTLRizuWQ3GuZmBBv27Nt7

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants