Skip to main content
← Back to list
01Issue
BugOpenSwamp CLI
AssigneesNone

Relationships

#1630 swamp serve dies of descriptor exhaustion on an ordinary workload, having warned about it at startup

Opened by kiljoy · 8/12/2026

Version: 20260723.235316.0-sha.a6927867 Orchestrator: macOS 15 / darwin-arm64 Worker: Ubuntu 24.04 / linux-x86_64

Summary

serve warns at startup that its open-file limit is too low, then fails on exactly that when a placed workflow step writes a few hundred small resources. The workload that triggers it is not large, and the process does not attempt to raise its own soft limit even where the platform would allow it.

Reproduction

Started from a stock macOS Terminal shell (soft limit 256, the macOS default):

$ swamp serve --host 127.0.0.1 --port 9094 --auth-mode token --admins "user:ryan"
warning serve  open-file limit is 256 — swamp serve may crash under load. Raise it to at least 8192
               with 'ulimit -n 8192' or LimitNOFILE=8192 in the systemd unit

Then run a workflow with one placed step whose method writes 332 small JSON resources (~2 KB each) from a remote worker, each proxied back to the orchestrator:

step render failed: Too many open files (os error 24): readfile '/Users/…/UAPResearcher/…'

The step fails, its dependent steps are skipped, and the run fails.

Actual vs expected

  • Actual: 332 sequential resource writes exhaust the descriptor limit and fail the run.
  • Expected: either the workload completes, or the startup warning reflects the real threshold. "may crash under load" reads as a caution about scale; 332 small writes is not scale.

Fix that works

$ ulimit -n 8192
$ swamp serve --host 127.0.0.1 --port 9094 --auth-mode token --admins "user:ryan"

Same workflow then completes. ulimit must be set in the parent shell before launch, which is easy to miss — the warning names the fix but appears after the point at which it can be applied to that process.

Suggestions

  1. Raise the soft limit in-process at startup. On this host the hard limit is unlimited and kern.maxfilesperproc is 122880, so raising the soft limit to 8192 needs no privilege — the warning's own recommended value is already available to the process that is printing the warning.
  2. Or bound concurrent descriptor use in the data path so a workload of a few hundred resource writes cannot exhaust the limit regardless of ulimit.
  3. Failing both, make the warning fatal (--allow-low-fd-limit to override), since continuing produces a failure hundreds of steps later that does not obviously point back to it.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/12/2026, 7:52:34 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.