fix(diagnostics,minimald): re-pin socket-join pids and make the diag stream timeout idle-based - #927
Conversation
`socket_join` read `/proc/<pid>/fd` for every pid `triage_pids` returned without re-validating the pid against its marker, so a matched process that exited and had its pid recycled between the table snapshot and the fd read would splice an unrelated process's sockets into `sockets.txt`. Its sibling `hang_triage_including` already guards against this: it re-pins each snapshot pid with `still_matches` immediately before reading its kernel state. Share that gate. `repin_live` keeps caller-supplied `always` pids untouched (their identity is not in question) and drops any snapshot pid whose argv0 no longer matches a marker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`stream_diag_bundle` wrapped the whole `tokio::io::copy` pump in a single 300s timeout, but `build_bundle` runs 15 sequential collectors each allowed up to `COLLECTOR_TIMEOUT` (30s) for a ~450s worst case, so a legitimate slow-but-progressing run was killed at 300s and the error unconditionally blamed the client. That timeout exists to bound a non-reading client (a DoS), not total transfer time. Replace it with a manual pump whose per-write deadline resets on every byte that reaches the client: reads from the build task stay unbounded (its collectors self-bound), while each `write_all` is bounded by the renamed `STREAM_IDLE_TIMEOUT` (60s). A stalled write is backpressure from a client that stopped draining the channel — exactly the DoS case — so a slow-but-advancing build now streams to completion and the message describes a stalled stream. The follow-up ordering (shutdown, `drop(rx)` to force `BrokenPipe`, await build) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe PR revalidates snapshot-derived PIDs before socket inspection and replaces the diagnostic bundle’s whole-transfer timeout with a per-write client idle timeout. ChangesLive socket process pinning
Diagnostic bundle streaming
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/diagnostics/src/procs.rs`:
- Around line 433-437: After the repin_live call in socket_join, check whether
pids is empty and immediately call w.skip(...) using the existing skip behavior,
returning before building /proc/net tables or emitting a header-only file.
Preserve the normal socket-join flow when any PID remains.
In `@crates/minimald/src/diag.rs`:
- Around line 64-72: Update the streaming write logic near the diagnostic pump
to loop over the buffer using writer.write rather than a single timeout-wrapped
write_all. Apply timeout(STREAM_IDLE_TIMEOUT, ...) separately to each write
attempt so the deadline resets after every successful partial write, continue
until all bytes are sent, and return an error when a write reports zero bytes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7e5811c1-eff7-4c42-af77-93872977ad6e
📒 Files selected for processing (2)
crates/diagnostics/src/procs.rscrates/minimald/src/diag.rs
| // Re-pin before reading any fds: a snapshot pid recycled since the table | ||
| // would otherwise splice an unrelated process's sockets into the join. The | ||
| // caller's own `always` pids are trusted without a re-read — same gate as | ||
| // `hang_triage_including`. | ||
| let pids = repin_live(pids, markers).await; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Handle an empty set after re-pinning.
When all snapshot PIDs are stale, repin_live returns an empty vector, but socket_join still builds /proc/net tables and emits a header-only file instead of calling w.skip(...). Add the same empty check immediately after Line 437.
Proposed fix
let pids = repin_live(pids, markers).await;
+ if pids.is_empty() {
+ w.skip(path, "no marker-matched processes holding sockets");
+ return Ok(());
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Re-pin before reading any fds: a snapshot pid recycled since the table | |
| // would otherwise splice an unrelated process's sockets into the join. The | |
| // caller's own `always` pids are trusted without a re-read — same gate as | |
| // `hang_triage_including`. | |
| let pids = repin_live(pids, markers).await; | |
| // Re-pin before reading any fds: a snapshot pid recycled since the table | |
| // would otherwise splice an unrelated process's sockets into the join. The | |
| // caller's own `always` pids are trusted without a re-read — same gate as | |
| // `hang_triage_including`. | |
| let pids = repin_live(pids, markers).await; | |
| if pids.is_empty() { | |
| w.skip(path, "no marker-matched processes holding sockets"); | |
| return Ok(()); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/diagnostics/src/procs.rs` around lines 433 - 437, After the repin_live
call in socket_join, check whether pids is empty and immediately call
w.skip(...) using the existing skip behavior, returning before building
/proc/net tables or emitting a header-only file. Preserve the normal socket-join
flow when any PID remains.
| /// How long a single write to the client may stall before the stream is | ||
| /// abandoned. This bounds a *non-reading* peer, not total transfer time: the | ||
| /// pump resets it on every write that makes progress, so a slow-but-advancing | ||
| /// build — up to fifteen 30s [`diagnostics::COLLECTOR_TIMEOUT`] collectors, | ||
| /// ~450s worst case — streams to completion, while a client that stops | ||
| /// draining the channel (backpressure fills the duplex and parks the next | ||
| /// write) trips it. A whole-transfer wall-clock cap, by contrast, would kill a | ||
| /// legitimate slow run and wrongly blame the client for it. | ||
| const STREAM_IDLE_TIMEOUT: Duration = Duration::from_secs(60); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
cargo tree -i tokio
rg -n 'poll_write|WriteAll' \
"${CARGO_HOME:-$HOME/.cargo}/registry/src" \
--glob '*/tokio-*/src/io/util/write_all.rs'Repository: gominimal/minimal
Length of output: 147
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- crates/minimald/src/diag.rs (relevant lines) ---\n'
sed -n '150,220p' crates/minimald/src/diag.rs
printf '\n--- tokio version in this repo ---\n'
cargo metadata --no-deps --format-version 1 | python3 - <<'PY'
import json,sys
m=json.load(sys.stdin)
for p in m["packages"]:
if p["name"]=="tokio":
print(p["name"], p["version"])
PY
printf '\n--- tokio write_all source if present in cargo registry ---\n'
python3 - <<'PY'
import os,glob,sys
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
print("\n".join(paths) if paths else "NOT_FOUND")
PYRepository: gominimal/minimal
Length of output: 246
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the exact call site and surrounding logic.
rg -n 'write_all|timeout\(STREAM_IDLE_TIMEOUT|STREAM_IDLE_TIMEOUT' crates/minimald/src/diag.rs -n -A8 -B8
# Find any local guidance on Tokio or diagnostics behavior.
rg -n 'diagnostics::COLLECTOR_TIMEOUT|STREAM_IDLE_TIMEOUT|write_all' crates -g '!**/target/**'Repository: gominimal/minimal
Length of output: 22091
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read Tokio's write_all implementation from the local cargo registry if present.
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
matches=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
if not matches:
print("NO_TOKIO_SOURCE")
else:
print(matches[0])
PYRepository: gominimal/minimal
Length of output: 171
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- diag.rs excerpt ---'
sed -n '160,210p' crates/minimald/src/diag.rs
echo
echo '--- tokio version from Cargo.lock ---'
rg -n '^name = "tokio"|^version = ' Cargo.lock -A1 -B1
echo
echo '--- local tokio write_all implementation path(s) ---'
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
for p in paths:
print(p)
PYRepository: gominimal/minimal
Length of output: 50373
🌐 Web query:
Tokio AsyncWriteExt::write_all implementation partial writes loop timeout semantics source
💡 Result:
Tokio's AsyncWriteExt::write_all implementation functions by continuously calling the write method on the underlying AsyncWrite type until the entire buffer is exhausted or an error occurs [1][2][3][4]. Regarding your specific areas of inquiry: Partial Writes and Loops The implementation essentially wraps write in a loop [1][2][4]. Because AsyncWrite::poll_write is readiness-based, it may return a partial result (e.g., writing fewer bytes than the length of the provided buffer) [5]. write_all handles this by tracking the remaining data and continuing to call write in subsequent polls until all bytes are written [2][4]. Timeout Semantics write_all does not have built-in timeout semantics [1][2]. If you need a timeout, you must wrap the write_all future in a tokio::time::timeout call. Because write_all is not cancellation-safe, if a timeout occurs, the operation will be aborted mid-progress [1][2][4]. Cancellation Safety write_all is explicitly documented as not cancellation-safe [1][2][4]. If it is used within a tokio::select! block and another branch completes first, the write_all future will be dropped [6]. Any data that was successfully written to the underlying stream remains there, but the future will not resume where it left off [1][2][4]. Subsequent calls to write_all on the same writer will start over from the beginning of the buffer, which can lead to duplicated data if the previous partial write was not accounted for [1][2][4]. Important Considerations - I/O Completion vs. Readiness: Tokio's I/O traits are readiness-based [5]. Even when write_all returns Poll::Ready(Ok(())), it only guarantees that the data has been accepted by the underlying writer (e.g., buffered by the kernel or the runtime), not necessarily that it has been fully persisted to the destination (e.g., disk) [7][8][5]. For files or buffered writers, an explicit flush may be required to ensure completion [7][8][5]. - Errors: write_all will return the first error encountered during its internal loop [1][2][3]. Top results: [1][2][7][6][5]
Citations:
- 1: https://github.com/tokio-rs/tokio/blob/5030b300/tokio/src/io/util/async_write_ext.rs
- 2: https://docs.rs/tokio/latest/tokio/io/trait.AsyncWriteExt.html
- 3: https://docs.rs/tokio/1.16.1/tokio/io/trait.AsyncWriteExt.html
- 4: https://docs.rs/tokio/1.19.2/x86_64-apple-darwin/tokio/io/trait.AsyncWriteExt.html
- 5:
tokio::fs::File::writereturns early before OS says that the operation is completed tokio-rs/tokio#7378 - 6: https://biriukov.dev/posts/async-rust-gocha-tokio-cancelation-select-future-then/
- 7: write_all doesn't guarantee data to be written after returning Poll::Ready tokio-rs/tokio#5531
- 8: Async write_all sometimes silently fails to write to file tokio-rs/tokio#4296
Reset the deadline per write.
At crates/minimald/src/diag.rs:179-180, timeout(STREAM_IDLE_TIMEOUT, writer.write_all(...)) measures the whole buffer write, so a slow-but-progressing client can still be dropped after 60s. Loop on writer.write(...), restart the timeout after each successful partial write, and treat WriteZero as an error.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/minimald/src/diag.rs` around lines 64 - 72, Update the streaming write
logic near the diagnostic pump to loop over the buffer using writer.write rather
than a single timeout-wrapped write_all. Apply timeout(STREAM_IDLE_TIMEOUT, ...)
separately to each write attempt so the deadline resets after every successful
partial write, continue until all bytes are sent, and return an error when a
write reports zero bytes.
Fix socket-join PID re-pinning and switch diagnostic stream to idle-based timeout
socket_joinin procs.rs now callsrepin_livebefore building socket tables, dropping snapshot-derived PIDs that no longer match markers while retaining caller-supplied PIDs unconditionally.Macroscope summarized 96c0d93.
Summary by CodeRabbit