Skip to content

fix(minimald): bound the boot-path gvproxy expose so it can't stall SSH accept - #676

Closed
norrietaylor wants to merge 1 commit into
mainfrom
fix/autospawn-bound-gvproxy-expose
Closed

fix(minimald): bound the boot-path gvproxy expose so it can't stall SSH accept#676
norrietaylor wants to merge 1 commit into
mainfrom
fix/autospawn-bound-gvproxy-expose

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 8, 2026

Copy link
Copy Markdown
Member

Problem

autospawn-e2e is green on main but only by a ~40 ms margin, and it's a pre-existing latent flake — surfaced while working the volume PR (#672), whose extra ~60 ms first-boot cost tips it red.

Root cause (from the green-main boot log): start_host_proxies awaits expose_proxy_on_host on Server::run's boot path, before the SSH accept loop serves. When no host gvproxy answers — which is always the case in the autospawn-e2e job (no MINVMD_GVPROXY_BIN) — that await runs to the full 5 s GVPROXY_CONTROL_TIMEOUT, holding the accept loop closed:

16:21:55.511  host-side egress proxy :7654 bindable       ← awaited expose begins
16:22:00.517  WARN expose timed out after 5s              ← accept loop blocked this whole 5s
16:22:00.518  accepted connection … SSH handshake failed: Broken pipe   ← client's 1st retry, gap
16:22:00.607  accepted connection … (a later retry succeeds → cold ls = 6196ms)

The cold minimal ls clears it only because the CLI retries and one retry lands the instant the loop unblocks at ~5 s. Any perturbation past that edge (CI load, or #672's mkfs) flips it to ssh connect: Disconnected.

In production this never happens — a host gvproxy is present, so the expose returns in <100 ms and the accept loop is never delayed.

Fix

Cap the best-effort publish at HOST_EXPOSE_PUBLISH_TIMEOUT (1 s), so a missing/slow gvproxy can't hold the accept loop near the CLI's retry deadline. The accept loop now resumes at ~1.6 s (vs ~5 s) — a multi-second margin instead of 40 ms. Best-effort semantics unchanged (timeout warns and continues); when a gvproxy is present the 1 s cap never triggers.

Validation

Refs: #588

Summary by CodeRabbit

  • Bug Fixes
    • Host-side proxy publication no longer waits indefinitely when the host forwarder is unavailable or unresponsive.
    • Added a bounded wait time for proxy exposure and improved warning messages for timeout versus connection errors.

…SH accept

start_host_proxies awaits expose_proxy_on_host on Server::run's boot path,
before the SSH accept loop begins serving. When no host gvproxy answers
(none configured), that await runs to the full 5s GVPROXY_CONTROL_TIMEOUT,
holding the accept loop closed. The cold `minimal ls` connect-retry
deadline (~5s) then expires just as the loop would resume, so the first
list fails with `ssh connect: Disconnected`. main clears this by only a
~40ms margin, relying on client retries landing the instant the loop
unblocks.

Cap the publish at HOST_EXPOSE_PUBLISH_TIMEOUT (1s) so the accept loop is
never held more than briefly; a present gvproxy answers in well under it.
Best-effort semantics are unchanged (a timeout warns and continues).

Refs: #588
@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4944b8fe-363d-48e9-95c5-f629de44f225

📥 Commits

Reviewing files that changed from the base of the PR and between 1c35174 and b28249c.

📒 Files selected for processing (1)
  • crates/minimald/src/server.rs

📝 Walkthrough

Walkthrough

Added a HOST_EXPOSE_PUBLISH_TIMEOUT constant and wrapped the host gvproxy forwarder publish call in expose_proxy_on_host with tokio::time::timeout, adding distinct warning logs for forwarder errors versus timeouts, replacing the previous unbounded, direct post_json await.

Changes

Publish Timeout Handling

Layer / File(s) Summary
Bounded forwarder publish
crates/minimald/src/server.rs
Adds HOST_EXPOSE_PUBLISH_TIMEOUT constant, updates doc comment, and wraps post_json in tokio::time::timeout with separate warning logs for forwarder errors and timeouts.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: twitchyliquid64

Sequence Diagram(s)

sequenceDiagram
  participant Server as minimald server
  participant Timeout as tokio::time::timeout
  participant Forwarder as host gvproxy forwarder

  Server->>Timeout: expose_proxy_on_host -> post_json(/services/forwarder/expose)
  Timeout->>Forwarder: forward request
  alt success within HOST_EXPOSE_PUBLISH_TIMEOUT
    Forwarder-->>Timeout: Ok response
    Timeout-->>Server: success (no-op)
  else forwarder error
    Forwarder-->>Timeout: Err response
    Timeout-->>Server: warn forwarder error
  else timeout elapsed
    Timeout-->>Server: warn timeout, continue
  end
Loading

Poem

A rabbit hopped to guard the gate,
"No endless waits, we won't be late!"
With timeout snug and warnings clear,
The proxy hops without a fear.
🐇⏱️ Thump-thump — ship it, dear!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is specific, concise, and matches the main change: limiting gvproxy expose time on the boot path to avoid blocking SSH accept.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor deleted the fix/autospawn-bound-gvproxy-expose branch July 8, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant