Symptom
bun (1.3.14) occasionally hangs during build on the res-servers and never completes — it sits silent until the 6h watchdog or an operator cancels it. Most recent: run b3e5df20-96c1-4ecd-8961-6538931aa627 (2026-08-02), cancelled after ~77 min.
Evidence (from buildbot + res-server logs + host metrics)
1. The hang shape — connect, then dead silence.
16:30:37 build graph loaded / run started
16:31:29 connecting to RES
16:32:30 res-server: "Building package: next" <- last progress log
… 77 minutes, ZERO log output …
17:48:17 res-server: build spec:"bun" … "sandbox error: Execution cancelled" (operator cancel)
A wedged build produces no output, so buildbot just waits. The silence is the signal, not a red herring.
2. It's deterministic-ish, not random. Two hangs the SAME day (12:36 and 17:48 UTC), both the identical spec_hash 7cbbffcce5226d4a57d49517e62dcbb86956b59b4363ffe0096bce478c2f78d3. So bun 1.3.14's build wedges reliably enough that "occasional" means "trips a race often," not "cosmic ray."
3. It is NOT memory / OOM. Res-server host memory across the entire 77-min hang window was flat and low — max 16.4%, mostly 1–3% (Cloud Monitoring agent.googleapis.com/memory/percent_used). A process holding ~16% memory perfectly still for 77 minutes with zero output is not OOM-thrashing (which pins ~100% + swap churn) — it is a lock deadlock: threads blocked on each other (futex/pipe), burning no CPU, no memory movement.
Hypothesis
packages/bun/build.sh runs bun run build:release — bun's own build orchestration (bun install + codegen + cmake native deps + zig + linking + strip) — with no parallelism cap. On the 48-core res-servers that fans cmake/zig/clang out to -j48. This is the high-core-count parallel-scheduler deadlock class (same family as the boost b2 hang, fixed in #313): a race in the parallel build machinery that only trips at high concurrency. The flat 16% memory is the proof it's a correctness bug at high fan-out, not a resource limit — there was enormous headroom.
Occasional-ness fits a race: the same spec_hash builds fine some runs, deadlocks others, depending on scheduling timing.
Proposed fix
Cap the cores the build sees — the established minimal pattern (foundationdb, or-tools, mono all cap at nproc/4). Bun is harder because bun run build:release is an opaque orchestrator (no single -j to pass), so cap the whole process tree via CPU affinity:
JOBS=$(( $(nproc) / 4 )) # 12 on the 48-core res-servers
taskset -c 0-$((JOBS - 1)) bun run build:release
taskset makes nproc report the capped count to every child (cmake/zig/clang/ninja), sidestepping needing bun's internal knob. With only 16% memory used at 48 cores, dropping to 12 costs nothing on resources — it just dodges the race.
(If bun exposes a job/CPU env var, setting that is cleaner than affinity — worth a look at bun 1.3.14's build scripts; I couldn't confirm one from the repo. taskset works regardless.)
Validation — must be on a res-server
This can only be reproduced/validated on a 48-core res-server, not a dev laptop (fewer cores won't trigger the high-core race — a local pass is confirming evidence, not a fix proof). Plan: apply the cap, trigger a res-server rebuild of the hanging spec_hash, confirm it completes. If it still hangs at -j12, escalate to catching a live hang for a stack trace (the res-servers stay warm; the same spec_hash reliably reproduces, so a targeted rebuild + gdb//proc/<pid>/stack on the wedged process would pinpoint which tool deadlocks).
Refs
Symptom
bun(1.3.14) occasionally hangs during build on the res-servers and never completes — it sits silent until the 6h watchdog or an operator cancels it. Most recent: runb3e5df20-96c1-4ecd-8961-6538931aa627(2026-08-02), cancelled after ~77 min.Evidence (from buildbot + res-server logs + host metrics)
1. The hang shape — connect, then dead silence.
A wedged build produces no output, so buildbot just waits. The silence is the signal, not a red herring.
2. It's deterministic-ish, not random. Two hangs the SAME day (12:36 and 17:48 UTC), both the identical spec_hash
7cbbffcce5226d4a57d49517e62dcbb86956b59b4363ffe0096bce478c2f78d3. So bun 1.3.14's build wedges reliably enough that "occasional" means "trips a race often," not "cosmic ray."3. It is NOT memory / OOM. Res-server host memory across the entire 77-min hang window was flat and low — max 16.4%, mostly 1–3% (Cloud Monitoring
agent.googleapis.com/memory/percent_used). A process holding ~16% memory perfectly still for 77 minutes with zero output is not OOM-thrashing (which pins ~100% + swap churn) — it is a lock deadlock: threads blocked on each other (futex/pipe), burning no CPU, no memory movement.Hypothesis
packages/bun/build.shrunsbun run build:release— bun's own build orchestration (bun install + codegen + cmake native deps + zig + linking + strip) — with no parallelism cap. On the 48-core res-servers that fans cmake/zig/clang out to-j48. This is the high-core-count parallel-scheduler deadlock class (same family as the boostb2hang, fixed in #313): a race in the parallel build machinery that only trips at high concurrency. The flat 16% memory is the proof it's a correctness bug at high fan-out, not a resource limit — there was enormous headroom.Occasional-ness fits a race: the same spec_hash builds fine some runs, deadlocks others, depending on scheduling timing.
Proposed fix
Cap the cores the build sees — the established minimal pattern (
foundationdb,or-tools,monoall cap atnproc/4). Bun is harder becausebun run build:releaseis an opaque orchestrator (no single-jto pass), so cap the whole process tree via CPU affinity:tasksetmakesnprocreport the capped count to every child (cmake/zig/clang/ninja), sidestepping needing bun's internal knob. With only 16% memory used at 48 cores, dropping to 12 costs nothing on resources — it just dodges the race.(If bun exposes a job/CPU env var, setting that is cleaner than affinity — worth a look at bun 1.3.14's build scripts; I couldn't confirm one from the repo.
tasksetworks regardless.)Validation — must be on a res-server
This can only be reproduced/validated on a 48-core res-server, not a dev laptop (fewer cores won't trigger the high-core race — a local pass is confirming evidence, not a fix proof). Plan: apply the cap, trigger a res-server rebuild of the hanging spec_hash, confirm it completes. If it still hangs at
-j12, escalate to catching a live hang for a stack trace (the res-servers stay warm; the same spec_hash reliably reproduces, so a targeted rebuild +gdb//proc/<pid>/stackon the wedged process would pinpoint which tool deadlocks).Refs
b3e5df20-96c1-4ecd-8961-6538931aa627b2high-core hang (boost: drop the b2 engine self-test step (deadlocks on high-core boxes) #313)packages/bun/build.sh(bun run build:release, uncapped)