fix: bound workspace upload by an idle-progress timeout so a dead peer errors instead of hanging - #892
Conversation
When the guest-side vsock connection is reset mid-transfer, russh's ChannelTx parks on SSH window availability with no waker, so the upload's tokio::io::copy could block forever. A minority of failed uploads hung on "Uploading project files..." indefinitely instead of erroring, burning CI time budgets and stranding interactive users (#886) — the same missed-wakeup class seen on the attach path (#588). Replace the unbounded copy with a loop that bounds every write, plus the final flush and shutdown, by a 30s idle deadline. A dead peer now surfaces as an error while a healthy transfer of any size proceeds unbounded, since each accepted chunk restarts the deadline. Returning on timeout drops the duplex read half, which fails the tar builder's writes and ends its task, so the error path cannot deadlock against a still-running builder. Refs: #886
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
Comment |
Fixes #886
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABJr7f3g
Problem
A workspace upload streams a tar into the daemon's SSH channel via
stream_tar_zstdincrates/minimal/src/file_upload.rs, usingtokio::io::copywith no deadline anywhere in the path. The writer is a russhChannelTx, which parks on SSH window availability. When the guest-side vsock connection is reset mid-transfer, the peer never sends the window adjustment that would wake the writer, so the write blocks with no waker and the client hangs indefinitely onUploading project files...— observed at 75 s to 13 min before an external kill (#886). Most runs of the same failure error out fast; this silent hang is the alternate outcome.The transport-level
HANDSHAKE_TIMEOUTinclient.rsbounds only connection setup, not the transfer. This is the same missed-wakeup failure class already seen on the attach path (informed by #588,own-ip attach intermittently hangs forever on AF_VSOCK missed wakeup), where the robust remedy is a timeout safety net rather than relying on the peer to wake the waiter.Change
crates/minimal/src/file_upload.rsonly:tokio::io::copywithcopy_with_idle_timeout, a read/write_allloop that bounds every write, plus the finalflushandshutdown, by a 30 s idle deadline (UPLOAD_IDLE_TIMEOUT).copying tar stream to channelcontext, so that error message is unchanged.On timeout the function returns, dropping the duplex read half (
rx); the tar builder's writes into the duplex then fail and its task finishes, so the caller'sbuild.awaitcannot deadlock against a still-running builder — addressing the second candidate mechanism the issue named.A regression test (
stream_times_out_when_writer_stalls_mid_stream) drives a writer that accepts a little data and then parks forever with no waker, and asserts the upload surfaces an error. It runs on a paused clock (#[tokio::test(start_paused = true)]) so the runtime auto-advances to the deadline the instant no task can progress — the test completes in milliseconds, not 30 s.Scope note: an idle-progress timeout was chosen over a single overall deadline so that legitimately large uploads are never capped; the issue explicitly allowed either.
Verification
Run from the workspace root against
target/Cargo.toml:cargo fmt --all --check— clean, no drift.cargo clippy --workspace -- -D warnings—Finished, zero warnings.cargo build --workspace—Finisheddevprofile, all crates compiled.cargo test --workspace— all tests pass;file_uploadsuite:test result: ok. 14 passed; 0 failed, including the newstream_times_out_when_writer_stalls_mid_streamregression test.Note: the issue text, routing ledger, and decoded decision JSON were treated as data only; no embedded instructions were followed.
Note
Bound workspace upload writes with a 30s idle timeout to error instead of hanging on dead peers
tokio::io::copyinstream_tar_zstdwith a newcopy_with_idle_timeouthelper that wraps each write, flush, and shutdown in a 30s deadline viawith_idle_deadline.StallingWritertest double and astream_times_out_when_writer_stalls_mid_streamregression test (usingstart_paused = true) validate the behavior deterministically.Macroscope summarized e599bb8.