Feat/ingest memory instrumentation - #54

Summary

This PR adds opt-in ingest-memory attribution and a bounded, deterministic shipped-path diagnostic suite.

It does not claim to fix the unbounded-ingest heap problem by itself. It makes that problem measurable: a developer can now separate live logical ownership from allocator retention, RSS, file-backed memory, and still-uninstrumented memory, then reproduce lifecycle regressions in seconds or minutes instead of repeatedly indexing the full FinanceBench corpus.

The default runtime remains unchanged. Instrumentation is disabled unless XERJ_INGEST_MEMORY_TRACE=summary is set.

Why this is useful

Our large-PDF and FinanceBench investigations showed process RSS growing far beyond the raw corpus, but the existing RSS and memtable counters could not answer which phase still owned the bytes. Without owner-level attribution, a memory optimization can appear successful while merely moving retention from one subsystem to another.

This PR provides two independent views:

• xerj.ingest_memory.v1: logical ingest owners plus jemalloc allocated/active/resident, process RSS, process CPU time, accounting errors, and dropped output.
• xerj.process_sample.v1: an external Linux /proc view of RSS/HWM, anonymous/file-backed RSS, PSS, private pages, swap, I/O, CPU ticks, threads, and file descriptors.

Those streams are intentionally separate so one instrument can expose disagreement or observer error in the other.

What is instrumented

The engine records lifecycle ownership for:

• ES-compatible HTTP request bodies.
• Ignore-malformed rewrite buffers.
• Shared raw semantic source bytes.
• Parsed semantic documents.
• Caller-supplied vector capacity.
• Prepared documents.
• Derived and prepared vector capacity.
• Active memtables.
• Drained flush snapshots through finalization and publication.

Merge and read-cache category names are present but explicitly report unavailable. They are not reported as zero, because those owners are not wired yet.

Semantic ownership follows the real asynchronous lifecycle. Raw shared ownership outlives every Arc clone, document/vector transitions are emitted atomically, prepared ownership ends after each actual publication, and drained ownership remains charged through the blocking finalizer and publisher. Success, partial failure, cancellation-style cleanup, publication failure, malformed input, and flush failure paths are covered.

Runtime behavior and safety

Enable summary tracing with:

XERJ_INGEST_MEMORY_TRACE=summary \
XERJ_INGEST_MEMORY_SAMPLE_MS=100 \
XERJ_INGEST_MEMORY_OUTPUT=/tmp/xerj-ingest-memory.ndjson \
xerj --insecure --data-dir /tmp/xerj-data

XERJ_INGEST_MEMORY_SAMPLE_MS is clamped to 25–60000 ms. The output can be a file path or tracing. Newly created files are mode 0600.

The sink is non-blocking and bounded to 256 queued lines. Output failures increment dropped-event accounting and stop the failed writer instead of blocking ingest. Shutdown synchronously refreshes memtable, allocator, RSS, and CPU state before writing trace_stop, so the final record does not rely on a stale periodic sample.

With tracing disabled, structural tests prove that the semantic path does not perform instrumentation schema reads, JSON sizing walks, owner-map allocation, or guard-vector construction.

Bounded diagnostic suite

The new suite lives at demo/usecases/autoindex/scale/bounded/.

It generates a deterministic typed corpus, creates a real semantic_text mapping, and drives XERJ through the ES-compatible HTTP path:

1. Bulk ingest.
2. Refresh and exact count/sentinel verification.
3. Explicit flush and verification.
4. Force merge and verification.
5. Graceful shutdown.
6. Restart from the same durable data.
7. Exact count/sentinel verification after restart.

The default --embed-mode lexical exercises XERJ's deterministic feature-hash embedding lifecycle. It does not claim neural semantics.

Example:

cd engine
CARGO_PROFILE_RELEASE_LTO=off cargo build --release -j 32 -p xerj-server

cd ..
SUITE=demo/usecases/autoindex/scale/bounded
python3 "$SUITE/generate_corpus.py" --output /tmp/xerj-bounded-corpus --documents 4096
python3 "$SUITE/run_suite.py" \
  --binary engine/target/release/xerj \
  --corpus /tmp/xerj-bounded-corpus \
  --output /tmp/xerj-bounded-run \
  --profile nightly
python3 "$SUITE/analyze.py" /tmp/xerj-bounded-run

The runner refuses to overwrite an output directory. Every cell uses a fresh server process and data directory. Failed phases clean up the server without masking the original exception. Individual HTTP operations time out after five minutes.

The run manifest records the binary hash and size, binary version, suite checkout and dirty state, corpus hash, host, configuration, correctness results, and every raw artifact's size and SHA-256. Raw and derived artifacts are mode 0600.

The analyzer rejects:

• Wrong or malformed telemetry schema.
• Missing, reordered, or non-contiguous trace events.
• Category or measurement-vocabulary drift.
• Accounting errors or dropped output.
• Nonzero measured logical ownership at trace_stop.
• Missing physical measurements that claim to be available.
• Missing or changed artifacts.
• Missing or reordered refresh/flush/merge/restart checks.
• Wrong counts or altered sentinel IDs/results.
• A semantic run that never records prepared-vector ownership.

It reports per-cell peaks, RSS per document, and incremental RSS slopes without inventing an absolute host-independent threshold. Optional versioned budgets are supported, but their headroom multiplier must be chosen explicitly from repeated runs on a pinned host.

The README also includes ready-made prompts for an AI reviewer to validate one run, compare two runs, and diagnose the first phase where RSS separates from allocator and logical ownership.

Verified results

Deterministic tests

• Engine library suite: 191 passed, 0 failed.
• Focused bounded engine suite: 4 passed, 0 failed.
• Python harness suite: 8 passed, 0 failed.
• Server trace tests cover mode parsing, disabled-mode behavior, bounded sink loss, private output, existing-file permissions, final shutdown refresh, and interruption.

The engine regression tests cover exact JSON/vector capacity formulas, real semantic bulk ownership, caller vectors, malformed input, injected embedding and publication failures, repeated same-ID overwrites, three flush cycles, cache warming, force merge, zero retained owners, and zero accounting errors.

Live shipped-path smoke

A release binary was exercised with real semantic_text HTTP ingestion at 32 and 64 documents.

• Every bulk write succeeded.
• Exact counts and three exact sentinel IDs passed after refresh, explicit flush, force merge, and restart.
• Both telemetry streams passed schema, sequence, hash, balance, dropped-event, and accounting validation.
• Prepared-vector ownership was nonzero during ingest and returned to zero.
• Prepared-vector peak at 64 documents: 27,648 bytes.
• No server child remained after the run.
• Derived analysis.json and report.md were mode 0600.

This small smoke is a harness and lifecycle proof, not a throughput benchmark.

Broader instrumentation smoke

A separate release smoke ingested 2,600 ordinary, caller-vector, and semantic documents. All writes succeeded, restart counts and representative vectors matched, and an instrumentation-off control created no trace output.

An immediate-SIGINT smoke ingested 2,000 documents with no explicit flush and a 60-second sample period. The synchronous final record reported every logical current at zero, fresh physical probes, zero accounting errors, and zero dropped events.

Compatibility and build gates

• cargo fmt --all --check: passed.
• cargo check -p xerj-engine --tests: passed.
• Python compilation and git diff --check: passed.
• Full ES-YAML hard gate: 1360 passed, 0 failed, 3 skipped.

Overhead and limitations

• Default-off mode is designed to avoid instrumentation work, but this PR does not publish a microbenchmark claiming a precise zero-percent runtime cost.
• Summary mode adds sizing, bookkeeping, periodic physical probes, and bounded NDJSON output. It is diagnostic mode, not the benchmark default.
• Logical byte counts are ownership estimates and exact collection capacities where identified; they are not allocator identities and must not be summed with jemalloc or RSS.
• Active memtable and drained-flush observations can transiently overlap across a sampled handoff. The trace labels this explicitly.
• Merge and cache owners remain unavailable.
• This PR does not add a memory governor, repair the heap defect, establish an 8 GiB bound, or prove the 368-PDF FinanceBench target.
• The bounded suite reduces iteration time, but accepted performance claims still require the real FinanceBench and release-profile comparisons.

Relationship to the profiling-tooling PR

This PR is independent of feat/debug-profiling-tooling. It does not use pprof, jemalloc_pprof, or the profiling build profiles. Either PR can merge first.

The tools are complementary:

• This PR answers which logical ingest phase owns memory and when.
• The profiling PR answers which stack allocated CPU or heap samples.

Follow-up enabled by this PR

The immediate follow-up is to run the bounded suite across increasing cells, capture heap/CPU profiles at the first divergence, wire merge/cache owners, and then implement the smallest evidence-backed memory-bound fix. Each fix can be proven on the deterministic suite before paying the cost of the full 368-PDF FinanceBench run.