feat(server): add private pprof debugging toolkit - #53

Summary

This adds an opt-in, Linux-only debugging workflow for answering the questions that aggregate RSS and allocator counters cannot answer: which call stacks consume CPU, and which allocation call stacks retain live heap bytes.

The normal XERJ binary is unchanged. Profiling code is compiled only with the new debug-profiling feature, exposes no network endpoint, starts inactive by default, writes only to an operator-selected local directory, and is bounded to a maximum 300-second delay and 300-second capture.

The workflow is designed for AI-assisted performance investigations. Every bundle contains a reproducibility manifest, exact binary and artifact hashes, server logs, optional copied correctness/telemetry evidence, a machine-readable validation summary, and focused prompts for CPU, heap, and before/after analysis.

Why

XERJ already exposes aggregate resource evidence such as RSS and jemalloc totals, but those values cannot identify the code paths retaining memory or consuming CPU. Full-corpus FinanceBench runs are expensive, so debugging needs to move toward deterministic small-corpus reproduction backed by profiles rather than repeated guess-and-check runs.

This change provides the missing call-stack evidence without adding an unauthenticated pprof endpoint or enabling profiler overhead in shipped builds.

Usage

Build the fast iteration profile from the repository root:

(cd engine && cargo build --profile profiling -j 32 -p xerj-server --no-default-features --features debug-profiling)

Capture CPU and attach correctness plus telemetry evidence:

python3 demo/playbooks/debug-profiling/capture.py --output /tmp/xerj-cpu-run --cpu-seconds 30 --delay-seconds 5 --workload financebench-fb20-v1 --corpus financebench-fb20-sha256 --concurrency 4 --cache-state cold --build-features debug-profiling --build-profile profiling --attach correctness=/tmp/correctness.json --attach telemetry=/tmp/memory.ndjson -- engine/target/profiling/xerj --insecure --data-dir /tmp/xerj-profile-data

Validate the bundle:

python3 demo/playbooks/debug-profiling/inspect.py /tmp/xerj-cpu-run

Repeat into fresh output and data directories with --heap-seconds 30 for the heap profile. Separate CPU-only and heap-only runs are recommended because simultaneous profilers can perturb one another.

Use --profile profiling-final before making a final production-codegen performance claim. profiling-final inherits the release fat-LTO, one-codegen-unit settings but retains symbols.

Artifact contract

cpu.pb is a Google pprof protobuf generated by pprof-rs.

heap.pb.gz is a gzipped Google pprof protobuf generated from jemalloc's sampled in-use heap.

manifest.json records the exact binary SHA-256, command, capture settings, host/kernel/page/cgroup shape, process outcome, artifact hashes, declared workload/corpus/concurrency/cache state, declared Cargo profile/features, repository HEAD/dirty state, attachments, and missing requested artifacts.

server.log captures combined server stdout and stderr.

cpu-error.json and heap-error.json are machine-readable worker failures when a profiler cannot start, dump, convert, or publish.

*.top.txt is generated when Google's pprof executable is available, giving agents an immediately readable top report.

All bundle files are mode 0600; the wrapper-created output directory is mode 0700. Profiles are written through unique temporary files, fsynced, and published with a no-overwrite hard link. A per-directory lock rejects concurrent controllers.

capture.py rejects a missing/non-executable binary before creating the output directory, bounds duration/frequency/delay, requires an explicit five-second post-capture conversion/publication window, terminates the complete server process group with SIGINT/SIGTERM/SIGKILL escalation, and fails when any requested profile is absent.

inspect.py verifies artifact and attachment hashes, validates gzip/non-empty formats, reports evidence gaps, and marks a bundle comparison-ineligible when correctness/benchmark/telemetry evidence is absent.

Profiling behavior

The controller is initialized before config loading, TLS setup, engine construction/replay, and router construction. Delay zero therefore captures initialization. XERJ_DEBUG_PROFILE_DELAY_SECONDS is process-start-relative; it is not a readiness probe.

Jemalloc is compiled with profiling support only in the feature build. Its existing background-thread and one-second dirty/muzzy decay configuration is preserved. Profiling starts with prof_active:false, uses the explicit 512 KiB sampling interval, and leaves cumulative profiling disabled so jemalloc does not retain every unique allocation backtrace.

CPU sampling defaults to 100 Hz, is capped at 1000 Hz, and blocklists libc, libgcc, pthread, and vdso according to upstream pprof-rs guidance.

Invalid requested profiling configuration or controller/thread preflight failure aborts server startup. Later worker failures produce private machine-readable error artifacts and cause the wrapper to report missing requested profiles.

Normal and handled early server exits interrupt delay/capture waits and remove the directory lock. A hostile early config-failure run returned server exit 1, bundle status profile_failed, and left no stale lock.

AI analysis workflow

PROMPT_CPU.md requires absolute and percentage flat/cumulative stacks, source mapping, subsystem grouping, falsifiable follow-up experiments, and explicit CONFIRMED/LIKELY/UNKNOWN sections.

PROMPT_HEAP.md requires byte accounting across HTTP bodies, JSON, prepared documents, vectors/model scratch, memtables, flush snapshots, merges, and caches while separating live heap from allocator active/resident slack and non-jemalloc RSS.

PROMPT_COMPARE.md rejects cross-profile comparisons, requires correctness before speed, requires absolute rather than percentage-only deltas, and distinguishes bytes removed from bytes moved into allocator slack, mmap/file cache, or another process.

All prompts treat the binary-to-source/build relationship as UNVERIFIED unless independently attested. The binary hash is exact, but the wrapper cannot prove that the binary was built from the recorded checkout or declared Cargo profile/features.

Build measurements

The final fast profiling build command was:

cargo build --profile profiling -j 32 -p xerj-server --no-default-features --features debug-profiling

It completed in 3m55.523s and produced a 401,304,832-byte binary with SHA-256 7449fb57a6353335f5375c73221300f4847d125dabbd3b9886db872073b1fe82.

The same worktree's stripped no-default-feature release reference was 39,708,296 bytes with SHA-256 a7c040d5637b22092ed94a73cb7b23fb35dd276cccb74bd835644211f0541dcf.

The fast profiling artifact is 361,596,536 bytes (344.85 MiB) larger, or 10.106x the stripped release reference. This is not pure profiler code size: it is dominated by retained debug symbols and also differs in thin-versus-fat LTO and codegen-unit settings. It is a debugging binary, not a distribution artifact.

No runtime CPU, throughput, latency, or memory overhead claim is made. Those require controlled same-profile A/B measurements. A result found with fast profiling remains provisional until reconfirmed with same-profile before/after runs using profiling-final.

End-to-end smoke evidence

A real two-second combined startup smoke against engine/target/profiling/xerj completed successfully and shut down cleanly.

cpu.pb: 20,113 bytes, SHA-256 21cf24356fbc002d3e7939eee1bcc456dac490a309617fbbd2f4a8e51fdc88d6.

heap.pb.gz: 26,336 bytes, SHA-256 7bbbe2f2c59a5109279daa8e7b866d04495869d72ede6be6b11d9dfcdef79cdd.

server.log: 15,269 bytes, SHA-256 099cbc9e1253056f685ad980fee5b810f1ec7b58478f8e9ba047a01168a4ef0b.

The manifest status was complete, the server exit code was 0, every artifact was mode 0600, and inspect.py validated hashes and formats. It correctly reported comparison_ready: false and an INCONCLUSIVE performance warning because this smoke deliberately attached no correctness or telemetry evidence.

The exact-boundary shutdown experiment happened to publish both profiles once, demonstrating that equality is timing-sensitive rather than a safe contract. The wrapper now rejects stop_after < delay + capture + 5 seconds; the new boundary test proves exact equality is rejected before an output directory is created.

Tests

cargo fmt --all -- --check
cargo check -p xerj-server --no-default-features --features debug-profiling
cargo check -p xerj-server --no-default-features
cargo test -p xerj-server --no-default-features --features debug-profiling debug_profiling::tests -- --test-threads=1
PYTHONDONTWRITEBYTECODE=1 python3 demo/playbooks/debug-profiling/test_tooling.py
git diff --check

Rust result: 6 passed, 0 failed. The tests cover bounded configuration, default sampling rate and delay, interruptible shutdown, no-overwrite atomic publication and 0600 permissions, real CPU capture plus protobuf decoding, and real jemalloc heap capture plus gzip and protobuf decoding.

Python result: 2 passed, 0 failed. The tests cover machine-readable missing-profile failure, manifest privacy/provenance, validator failure behavior, and rejection of a stop time without publication grace.

Both normal and debug-profiling scoped checks pass.

Security and limitations

Linux only.

No network profiling endpoint is added.

Profiles reveal function names, source paths, and workload-dependent execution structure. The manifest records the full server command. Operators must not place passwords, API keys, or tokens in command-line arguments and must transfer bundles only through approved private channels.

CPU profiling uses SIGPROF and libgcc unwinding. Upstream pprof-rs documents that libgcc unwinding is not signal-safe and that restoring the original SIGPROF handler remains a TODO. This is controlled debugging tooling, not an always-on production profiler.

Heap profiles are probabilistically sampled in-use bytes at capture time. They are not every allocation, peak RSS, allocator fragmentation, file cache, mmap residency, or another process.

Combined CPU and heap runs are convenient smoke tests but confounded for performance measurement. Use separate repetitions.

Delay is not a readiness probe. Coordinate the workload externally or use a conservative delay.

A SIGKILL cannot run destructors and can leave .xerj-debug-profile.lock. Before removing it, confirm no XERJ process is using that exact directory, then remove only that lock.

The repository HEAD/profile/features in the manifest are operator evidence, not cryptographic build attestation. source_build_binding is deliberately UNVERIFIED.

No runtime-overhead or performance-improvement claim is included in this PR.