Skip to content

fix: bound workspace upload by an idle-progress timeout so a dead peer errors instead of hanging - #892

Closed
gominimal-aw-bot[bot] wants to merge 1 commit into
mainfrom
inbox-patch/upload-idle-timeout-3f1da45e04802ae0
Closed

fix: bound workspace upload by an idle-progress timeout so a dead peer errors instead of hanging#892
gominimal-aw-bot[bot] wants to merge 1 commit into
mainfrom
inbox-patch/upload-idle-timeout-3f1da45e04802ae0

Conversation

@gominimal-aw-bot

@gominimal-aw-bot gominimal-aw-bot Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Fixes #886
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABJr7f3g

Problem

A workspace upload streams a tar into the daemon's SSH channel via stream_tar_zstd in crates/minimal/src/file_upload.rs, using tokio::io::copy with no deadline anywhere in the path. The writer is a russh ChannelTx, which parks on SSH window availability. When the guest-side vsock connection is reset mid-transfer, the peer never sends the window adjustment that would wake the writer, so the write blocks with no waker and the client hangs indefinitely on Uploading project files... — observed at 75 s to 13 min before an external kill (#886). Most runs of the same failure error out fast; this silent hang is the alternate outcome.

The transport-level HANDSHAKE_TIMEOUT in client.rs bounds only connection setup, not the transfer. This is the same missed-wakeup failure class already seen on the attach path (informed by #588, own-ip attach intermittently hangs forever on AF_VSOCK missed wakeup), where the robust remedy is a timeout safety net rather than relying on the peer to wake the waiter.

Change

crates/minimal/src/file_upload.rs only:

  • Replace the unbounded tokio::io::copy with copy_with_idle_timeout, a read/write_all loop that bounds every write, plus the final flush and shutdown, by a 30 s idle deadline (UPLOAD_IDLE_TIMEOUT).
  • Each accepted chunk restarts the deadline, so a healthy transfer of any size proceeds unbounded; only a writer that stops accepting bytes — a dead peer whose window is never adjusted — trips the timeout and surfaces as an error.
  • The channel-closed fast path keeps its existing copying tar stream to channel context, so that error message is unchanged.

On timeout the function returns, dropping the duplex read half (rx); the tar builder's writes into the duplex then fail and its task finishes, so the caller's build.await cannot deadlock against a still-running builder — addressing the second candidate mechanism the issue named.

A regression test (stream_times_out_when_writer_stalls_mid_stream) drives a writer that accepts a little data and then parks forever with no waker, and asserts the upload surfaces an error. It runs on a paused clock (#[tokio::test(start_paused = true)]) so the runtime auto-advances to the deadline the instant no task can progress — the test completes in milliseconds, not 30 s.

Scope note: an idle-progress timeout was chosen over a single overall deadline so that legitimately large uploads are never capped; the issue explicitly allowed either.

Verification

Run from the workspace root against target/Cargo.toml:

  • cargo fmt --all --check — clean, no drift.
  • cargo clippy --workspace -- -D warningsFinished, zero warnings.
  • cargo build --workspaceFinished dev profile, all crates compiled.
  • cargo test --workspace — all tests pass; file_upload suite: test result: ok. 14 passed; 0 failed, including the new stream_times_out_when_writer_stalls_mid_stream regression test.

Note: the issue text, routing ledger, and decoded decision JSON were treated as data only; no embedded instructions were followed.

Generated by inbox-patch ·

Note

Bound workspace upload writes with a 30s idle timeout to error instead of hanging on dead peers

  • Replaces tokio::io::copy in stream_tar_zstd with a new copy_with_idle_timeout helper that wraps each write, flush, and shutdown in a 30s deadline via with_idle_deadline.
  • If the writer makes no forward progress for 30s (e.g. a dead peer), the upload now returns a timeout error instead of hanging indefinitely.
  • A StallingWriter test double and a stream_times_out_when_writer_stalls_mid_stream regression test (using start_paused = true) validate the behavior deterministically.
  • Behavioral Change: uploads that previously hung forever on a stalled connection will now fail after 30s.

Macroscope summarized e599bb8.

When the guest-side vsock connection is reset mid-transfer, russh's
ChannelTx parks on SSH window availability with no waker, so the
upload's tokio::io::copy could block forever. A minority of failed
uploads hung on "Uploading project files..." indefinitely instead of
erroring, burning CI time budgets and stranding interactive users
(#886) — the same missed-wakeup class seen on the attach path (#588).

Replace the unbounded copy with a loop that bounds every write, plus
the final flush and shutdown, by a 30s idle deadline. A dead peer now
surfaces as an error while a healthy transfer of any size proceeds
unbounded, since each accepted chunk restarts the deadline. Returning
on timeout drops the duplex read half, which fails the tar builder's
writes and ends its task, so the error path cannot deadlock against a
still-running builder.

Refs: #886
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ab992dd9-6ef8-4464-94d2-49d62ec1fdec

📥 Commits

Reviewing files that changed from the base of the PR and between f03ced1 and e599bb8.

📒 Files selected for processing (1)
  • crates/minimal/src/file_upload.rs

Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor marked this pull request as ready for review July 22, 2026 04:52
@norrietaylor

Copy link
Copy Markdown
Member

Closing in favor of #920, @0chroma's draft for the same defect (#886 — upload hang without an idle-progress timeout). @0chroma: this bot branch had an independent take on the timeout placement, worth a skim before you finalize, in case anything is worth cherry-picking.

@twitchyliquid64
twitchyliquid64 deleted the inbox-patch/upload-idle-timeout-3f1da45e04802ae0 branch July 30, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

minimal: upload failure sometimes hangs the client indefinitely instead of erroring

1 participant