Skip to content

docs: add worker-pool and data-plane tuning how-to#107

Closed
jmcorgan wants to merge 1 commit into
masterfrom
doc/tune-worker-pools
Closed

docs: add worker-pool and data-plane tuning how-to#107
jmcorgan wants to merge 1 commit into
masterfrom
doc/tune-worker-pools

Conversation

@jmcorgan

@jmcorgan jmcorgan commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

Adds docs/how-to/tune-worker-pools.md, an operator how-to for the
data-plane performance-tuning environment variables introduced with the
off-task encrypt/decrypt worker pools and per-peer connected UDP (#91).
None of these knobs were documented outside source doc-comments. The
guide pairs with the existing tune-udp-buffers.md: the two cover
adjacent concerns (UDP socket buffers vs. per-stage worker queues), and
an operator tuning one will want to find the other. Also adds the guide
to the how-to index.

Closes #101.

What it documents

All of the startup-time runtime knobs, grouped:

  • Worker pool sizing: FIPS_ENCRYPT_WORKERS, FIPS_DECRYPT_WORKERS
  • Per-peer connected UDP: FIPS_CONNECTED_UDP, FIPS_MACOS_CONNECTED_UDP
  • Send-path backpressure: FIPS_SEND_BACKPRESSURE_SLEEP_AFTER /
    _SLEEP_MICROS / _DROP_AFTER
  • Perf profiler: FIPS_PERF, FIPS_PIPELINE_TRACE,
    FIPS_PERF_INTERVAL_SECS
  • macOS send tuning: FIPS_MACOS_ORDERED_SENDER, _WORKER_STRIDE,
    _WORKER_BATCH, _SEND_FLOW_IDLE_MS, _SEND_PACE_MBPS,
    _SEND_PACE_BURST_BYTES, _NET_SERVICE_TYPE

Each knob lists its real default (read from source, including the
macOS/Linux split on the backpressure gates), what it does, and the
observability mode that would prompt tuning it. Includes the
massive-mesh FIPS_CONNECTED_UDP=0 escape-hatch note (one drain thread
per established peer) and a see-also linking the UDP receive buffer
ceiling to the decrypt worker queue capacity.

Review ask

@mmalmi this documents your #91 work, so I would value an accuracy pass
on the prose, especially:

  • the hash-by-destination single-flow caveat (raising encrypt workers
    does nothing for a single flow)
  • the yield, then sleep, then drop backpressure escalation, and that
    DROP_AFTER is the ceiling checked first
  • the macOS batch history (the note that a batch of 32 could trigger TCP
    collapse on Wi-Fi)
  • the QoS-marking regression note for FIPS_MACOS_NET_SERVICE_TYPE

One correction worth flagging: the doc uses 32768 as the decrypt worker
channel capacity (decrypt_worker.rs), not the 1024 that #101's
description listed. 1024 is the encrypt channel capacity; the two
differ.

Notes

Documentation only, no code changes. Defaults and behavior were read
from source at this branch's base.

Document the data-plane performance-tuning environment variables
introduced with the off-task encrypt/decrypt worker pools and per-peer
connected UDP. None of these knobs were documented outside source
doc-comments. The new guide pairs with tune-udp-buffers.md: the two
cover adjacent concerns (UDP socket buffers vs. per-stage worker
queues) and operators tuning one will want to find the other.

Covers worker-pool sizing, per-peer connected UDP, send-path
backpressure, the per-stage perf profiler, and the macOS-specific send
knobs, each with its real default, what it does, and the observability
mode that would prompt tuning it. Includes the massive-mesh
FIPS_CONNECTED_UDP=0 escape hatch note (one drain thread per
established peer) and a see-also linking the UDP receive buffer ceiling
to the decrypt worker queue capacity. Adds the guide to the how-to
index.
@jmcorgan jmcorgan closed this Jun 6, 2026
@jmcorgan jmcorgan deleted the doc/tune-worker-pools branch June 6, 2026 19:49
@jmcorgan jmcorgan restored the doc/tune-worker-pools branch June 7, 2026 00:22
@jmcorgan jmcorgan reopened this Jun 7, 2026
@jmcorgan

Copy link
Copy Markdown
Owner Author

Closing this draft unmerged for now. It has been open without review interest, and the worker-pool tuning knobs it documents are stable, so the how-to can wait. The content is preserved and can be reopened later if we decide to ship operator-facing data-plane tuning docs.

@jmcorgan jmcorgan closed this Jun 19, 2026
@jmcorgan jmcorgan deleted the doc/tune-worker-pools branch June 19, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: document PR #91 env vars and worker-pool tuning (companion to tune-udp-buffers)

1 participant