Skip to content

fix(diagnostics,minimald): re-pin socket-join pids and make the diag stream timeout idle-based - #927

Merged
norrietaylor merged 2 commits into
mainfrom
fix/diag-878-followup
Jul 23, 2026
Merged

fix(diagnostics,minimald): re-pin socket-join pids and make the diag stream timeout idle-based#927
norrietaylor merged 2 commits into
mainfrom
fix/diag-878-followup

Conversation

@norrietaylor

@norrietaylor norrietaylor commented Jul 23, 2026

Copy link
Copy Markdown
Member

Fix socket-join PID re-pinning and switch diagnostic stream to idle-based timeout

  • socket_join in procs.rs now calls repin_live before building socket tables, dropping snapshot-derived PIDs that no longer match markers while retaining caller-supplied PIDs unconditionally.
  • The diagnostic stream in diag.rs replaces a 300s whole-transfer timeout with a 60s per-write idle timeout, allowing slow builds to complete as long as the client keeps reading.
  • Behavioral Change: sockets for stale snapshot PIDs are excluded from joined output; streams that stall for 60s are aborted even if the total transfer is under 300s.

Macroscope summarized 96c0d93.

Summary by CodeRabbit

  • Bug Fixes
    • Improved diagnostic process detection to avoid associating sockets with processes that have changed or exited.
    • Diagnostic bundle downloads now remain active for slow transfers while still timing out when the receiving connection stops accepting data.
    • Updated timeout messaging for stalled diagnostic bundle connections.

norrietaylor and others added 2 commits July 22, 2026 22:03
`socket_join` read `/proc/<pid>/fd` for every pid `triage_pids` returned
without re-validating the pid against its marker, so a matched process
that exited and had its pid recycled between the table snapshot and the
fd read would splice an unrelated process's sockets into `sockets.txt`.

Its sibling `hang_triage_including` already guards against this: it
re-pins each snapshot pid with `still_matches` immediately before reading
its kernel state. Share that gate. `repin_live` keeps caller-supplied
`always` pids untouched (their identity is not in question) and drops any
snapshot pid whose argv0 no longer matches a marker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`stream_diag_bundle` wrapped the whole `tokio::io::copy` pump in a single
300s timeout, but `build_bundle` runs 15 sequential collectors each
allowed up to `COLLECTOR_TIMEOUT` (30s) for a ~450s worst case, so a
legitimate slow-but-progressing run was killed at 300s and the error
unconditionally blamed the client.

That timeout exists to bound a non-reading client (a DoS), not total
transfer time. Replace it with a manual pump whose per-write deadline
resets on every byte that reaches the client: reads from the build task
stay unbounded (its collectors self-bound), while each `write_all` is
bounded by the renamed `STREAM_IDLE_TIMEOUT` (60s). A stalled write is
backpressure from a client that stopped draining the channel — exactly
the DoS case — so a slow-but-advancing build now streams to completion
and the message describes a stalled stream. The follow-up ordering
(shutdown, `drop(rx)` to force `BrokenPipe`, await build) is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR revalidates snapshot-derived PIDs before socket inspection and replaces the diagnostic bundle’s whole-transfer timeout with a per-write client idle timeout.

Changes

Live socket process pinning

Layer / File(s) Summary
Re-pin processes before socket correlation
crates/diagnostics/src/procs.rs
repin_live preserves always-selected PIDs, validates snapshot-derived PIDs, and passes filtered IDs into socket joining; a Linux test covers both cases.

Diagnostic bundle streaming

Layer / File(s) Summary
Apply per-write client idle protection
crates/minimald/src/diag.rs
stream_diag_bundle uses buffered reads and per-write STREAM_IDLE_TIMEOUT handling instead of a whole-transfer deadline.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

  • gominimal/inbox#342 — Adds the live PID filtering used before /proc/<pid>/fd inspection.
  • gominimal/minimal#917 — Re-pins snapshot-derived PIDs before socket inspection.
  • gominimal/minimal#915 — Replaces the whole-transfer diagnostic streaming timeout.
  • gominimal/inbox#343 — Implements per-write idle protection for diagnostic bundle delivery.

Possibly related PRs

Suggested reviewers: twitchyliquid64

Poem

A rabbit watched the socket trail,
And trimmed stale PIDs from the tale.
The bundle hops through writes anew,
With idle clocks for clients too.
“No frozen stream!” the rabbit sings,
While moonlight guards the proc-file strings.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning It covers the summary, but omits the required Testing section and Checklist entries from the template. Add a Testing section with commands/output and complete the checklist items for docs and any BREAKING CHANGE footer.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes both the socket re-pinning and idle-timeout changes.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands.

@norrietaylor
norrietaylor marked this pull request as ready for review July 23, 2026 15:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/diagnostics/src/procs.rs`:
- Around line 433-437: After the repin_live call in socket_join, check whether
pids is empty and immediately call w.skip(...) using the existing skip behavior,
returning before building /proc/net tables or emitting a header-only file.
Preserve the normal socket-join flow when any PID remains.

In `@crates/minimald/src/diag.rs`:
- Around line 64-72: Update the streaming write logic near the diagnostic pump
to loop over the buffer using writer.write rather than a single timeout-wrapped
write_all. Apply timeout(STREAM_IDLE_TIMEOUT, ...) separately to each write
attempt so the deadline resets after every successful partial write, continue
until all bytes are sent, and return an error when a write reports zero bytes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7e5811c1-eff7-4c42-af77-93872977ad6e

📥 Commits

Reviewing files that changed from the base of the PR and between 5720bf6 and 96c0d93.

📒 Files selected for processing (2)
  • crates/diagnostics/src/procs.rs
  • crates/minimald/src/diag.rs

Comment on lines +433 to +437
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Handle an empty set after re-pinning.

When all snapshot PIDs are stale, repin_live returns an empty vector, but socket_join still builds /proc/net tables and emits a header-only file instead of calling w.skip(...). Add the same empty check immediately after Line 437.

Proposed fix
     let pids = repin_live(pids, markers).await;
+    if pids.is_empty() {
+        w.skip(path, "no marker-matched processes holding sockets");
+        return Ok(());
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;
// Re-pin before reading any fds: a snapshot pid recycled since the table
// would otherwise splice an unrelated process's sockets into the join. The
// caller's own `always` pids are trusted without a re-read — same gate as
// `hang_triage_including`.
let pids = repin_live(pids, markers).await;
if pids.is_empty() {
w.skip(path, "no marker-matched processes holding sockets");
return Ok(());
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/diagnostics/src/procs.rs` around lines 433 - 437, After the repin_live
call in socket_join, check whether pids is empty and immediately call
w.skip(...) using the existing skip behavior, returning before building
/proc/net tables or emitting a header-only file. Preserve the normal socket-join
flow when any PID remains.

Comment on lines +64 to +72
/// How long a single write to the client may stall before the stream is
/// abandoned. This bounds a *non-reading* peer, not total transfer time: the
/// pump resets it on every write that makes progress, so a slow-but-advancing
/// build — up to fifteen 30s [`diagnostics::COLLECTOR_TIMEOUT`] collectors,
/// ~450s worst case — streams to completion, while a client that stops
/// draining the channel (backpressure fills the duplex and parks the next
/// write) trips it. A whole-transfer wall-clock cap, by contrast, would kill a
/// legitimate slow run and wrongly blame the client for it.
const STREAM_IDLE_TIMEOUT: Duration = Duration::from_secs(60);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

cargo tree -i tokio
rg -n 'poll_write|WriteAll' \
  "${CARGO_HOME:-$HOME/.cargo}/registry/src" \
  --glob '*/tokio-*/src/io/util/write_all.rs'

Repository: gominimal/minimal

Length of output: 147


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- crates/minimald/src/diag.rs (relevant lines) ---\n'
sed -n '150,220p' crates/minimald/src/diag.rs

printf '\n--- tokio version in this repo ---\n'
cargo metadata --no-deps --format-version 1 | python3 - <<'PY'
import json,sys
m=json.load(sys.stdin)
for p in m["packages"]:
    if p["name"]=="tokio":
        print(p["name"], p["version"])
PY

printf '\n--- tokio write_all source if present in cargo registry ---\n'
python3 - <<'PY'
import os,glob,sys
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
print("\n".join(paths) if paths else "NOT_FOUND")
PY

Repository: gominimal/minimal

Length of output: 246


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the exact call site and surrounding logic.
rg -n 'write_all|timeout\(STREAM_IDLE_TIMEOUT|STREAM_IDLE_TIMEOUT' crates/minimald/src/diag.rs -n -A8 -B8

# Find any local guidance on Tokio or diagnostics behavior.
rg -n 'diagnostics::COLLECTOR_TIMEOUT|STREAM_IDLE_TIMEOUT|write_all' crates -g '!**/target/**'

Repository: gominimal/minimal

Length of output: 22091


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Read Tokio's write_all implementation from the local cargo registry if present.
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
matches=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
if not matches:
    print("NO_TOKIO_SOURCE")
else:
    print(matches[0])
PY

Repository: gominimal/minimal

Length of output: 171


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- diag.rs excerpt ---'
sed -n '160,210p' crates/minimald/src/diag.rs

echo
echo '--- tokio version from Cargo.lock ---'
rg -n '^name = "tokio"|^version = ' Cargo.lock -A1 -B1

echo
echo '--- local tokio write_all implementation path(s) ---'
python3 - <<'PY'
import os,glob
home=os.environ.get("CARGO_HOME", os.path.expanduser("~/.cargo"))
paths=glob.glob(home+"/registry/src/*/tokio-*/src/io/util/write_all.rs")
for p in paths:
    print(p)
PY

Repository: gominimal/minimal

Length of output: 50373


🌐 Web query:

Tokio AsyncWriteExt::write_all implementation partial writes loop timeout semantics source

💡 Result:

Tokio's AsyncWriteExt::write_all implementation functions by continuously calling the write method on the underlying AsyncWrite type until the entire buffer is exhausted or an error occurs [1][2][3][4]. Regarding your specific areas of inquiry: Partial Writes and Loops The implementation essentially wraps write in a loop [1][2][4]. Because AsyncWrite::poll_write is readiness-based, it may return a partial result (e.g., writing fewer bytes than the length of the provided buffer) [5]. write_all handles this by tracking the remaining data and continuing to call write in subsequent polls until all bytes are written [2][4]. Timeout Semantics write_all does not have built-in timeout semantics [1][2]. If you need a timeout, you must wrap the write_all future in a tokio::time::timeout call. Because write_all is not cancellation-safe, if a timeout occurs, the operation will be aborted mid-progress [1][2][4]. Cancellation Safety write_all is explicitly documented as not cancellation-safe [1][2][4]. If it is used within a tokio::select! block and another branch completes first, the write_all future will be dropped [6]. Any data that was successfully written to the underlying stream remains there, but the future will not resume where it left off [1][2][4]. Subsequent calls to write_all on the same writer will start over from the beginning of the buffer, which can lead to duplicated data if the previous partial write was not accounted for [1][2][4]. Important Considerations - I/O Completion vs. Readiness: Tokio's I/O traits are readiness-based [5]. Even when write_all returns Poll::Ready(Ok(())), it only guarantees that the data has been accepted by the underlying writer (e.g., buffered by the kernel or the runtime), not necessarily that it has been fully persisted to the destination (e.g., disk) [7][8][5]. For files or buffered writers, an explicit flush may be required to ensure completion [7][8][5]. - Errors: write_all will return the first error encountered during its internal loop [1][2][3]. Top results: [1][2][7][6][5]

Citations:


Reset the deadline per write.
At crates/minimald/src/diag.rs:179-180, timeout(STREAM_IDLE_TIMEOUT, writer.write_all(...)) measures the whole buffer write, so a slow-but-progressing client can still be dropped after 60s. Loop on writer.write(...), restart the timeout after each successful partial write, and treat WriteZero as an error.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/minimald/src/diag.rs` around lines 64 - 72, Update the streaming write
logic near the diagnostic pump to loop over the buffer using writer.write rather
than a single timeout-wrapped write_all. Apply timeout(STREAM_IDLE_TIMEOUT, ...)
separately to each write attempt so the deadline resets after every successful
partial write, continue until all bytes are sent, and return an error when a
write reports zero bytes.

@norrietaylor
norrietaylor merged commit 7fd582e into main Jul 23, 2026
29 checks passed
@norrietaylor
norrietaylor deleted the fix/diag-878-followup branch July 23, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants