Summary

Replace autoindex's raw PDF stream scanner with pdf_oxide 0.3.75 and run each PDF in a fresh same-binary worker process.

The old extractor searched compressed PDF streams and interpreted font character codes without resolving the object graph, page resources, font encodings, or /ToUnicode mappings. On real annual reports this silently produced NUL-separated strings, shifted glyphs, and binary-looking content. Downstream inference then correctly refused to elect body as semantic_text.

This change makes PDF extraction fail closed: parsed Unicode page text is retrieval-sized with page/section provenance, suspicious output is rejected, partial parser failures do not create partial indices, and image-only or damaged/encrypted inputs receive actionable repair/decrypt/OCR guidance. There is no raw-byte fallback.

User-facing behavior

xerj autoindex <folder> now launches a fresh hidden xerj __extract-pdf worker for each PDF. The worker returns a versioned JSON response and the parent validates both xerj-autoindex and pdf_oxide provenance.

Every accepted PDF record includes:

• one-based page;
• stable locator pN-sN;
• pdf_pages_total;
• pdf_pages_with_text;
• pdf_pages_omitted;
• pdf_ocr_warning when pages contain no usable text.

New controls:

--pdf-workers <N>       concurrent parser processes, default min(cores,4), maximum 4
--pdf-timeout-secs <N>  per-document timeout, default 120, maximum 3600

xerj autoindex --help documents limits, OCR behavior, process isolation, the trusted developer override, parent-memory guidance, and the current double-parse cost.

Architecture and safety limits

• Input PDF: 512 MiB maximum.
• Declared pages: 100,000 maximum.
• Extracted page text: 16 MiB maximum.
• Extracted document text: 64 MiB maximum.
• Retained worker stdout: 32 MiB maximum per active PDF.
• Retained worker stderr: 64 KiB maximum.
• Unix worker address space: 1,536 MiB RLIMIT_AS.
• Unix worker cleanup: dedicated process group, killed and reaped on timeout/monitor failure; descendants are also killed after normal leader exit.
• Non-Unix: direct-child cleanup, but no OS memory cap.

The pipe readers continue draining after their retained prefixes overflow, so an oversized response cannot deadlock on a full pipe. This is crash/resource isolation, not a security sandbox: the parser keeps the invoking user's filesystem and network authority.

XERJ_PDF_WORKER_BIN remains a trusted developer/test hook. Its response is schema-, extractor-version-, parser-version-, and JSON-validated, but the executable itself runs with user authority.

Four-report FinanceBench dry run

Command shape:

xerj autoindex ./financebench-pdfs \
  --dry-run --json \
  --url http://127.0.0.1:9200 \
  --state-dir ./autoindex-state \
  --workers 2 \
  --pdf-workers 2

The input contained these previously blocking annual reports:

Result:

• exit 0;
• four files / 41 MiB;
• one PDF dataset;
• zero junk or skipped files;
• 1,779 inference-sampled records;
• body automatically elected as semantic_text;
• aggregate word ratio 0.77;
• mean 226.8 tokens per value;
• average body length 1,540.88 characters;
• 61.567 seconds wall time with two PDF workers.

This was an autoindex dry run. It measured extraction and schema inference only: it did not write an index, generate embeddings, or finalize segments. The per-file record total is greater than 1,779 because inference caps each file's sample contribution.

The omitted-page count does not claim that every omitted page is scanned; it can include intentional blank pages. The warning deliberately says the page may require OCR.

Quality gate

The parser applies a narrow allowlist of known legacy symbol-font corrections, then rejects page text when more than 0.5% of visible characters are controls, replacement glyphs, or private-use codes, or fewer than 5% are alphanumeric.

On the pinned 368-document FinanceBench PDF corpus this accepted 367 documents and rejected one suspicious document. That is narrow calibration evidence, not a claim that the threshold is universal across all PDFs.

Binary and dependency cost

Both binaries were built from e6e8e6ac3b017e9c7e03d40e3b9cf4f4e3bee55e with:

cd engine
cargo build --release -j 32 -p xerj-server --no-default-features

The stripped binary grows 14.22%. The server dependency graph gains 62 unique crate names and the lockfile changes by 747 additions and 33 removals. pdf_oxide is pinned to =0.3.75, uses default-features = false plus legacy-crypto, and is licensed MIT OR Apache-2.0.

Verification

cargo fmt -p xerj-autoindex -p xerj-server -- --check
PASS

cargo clippy -p xerj-autoindex -p xerj-server --no-default-features --all-targets -- -D warnings
PASS

cargo test -p xerj-autoindex --no-default-features
34 passed, 0 failed

cargo test -p xerj-server --test pdf_worker --no-default-features
5 passed, 0 failed

es-yaml-runner --url http://127.0.0.1:9280 --dir tests/es-compat-yaml/yaml
1360 passed, 0 failed, 3 skipped

The subprocess suite covers deterministic output, exact page provenance and protocol versions, corrupt-PDF errors with empty stdout, mixed text/empty-page coverage, non-UTF-8 Unix filenames, stdout and stderr overflow without deadlock, timeout cleanup, descendant cleanup, and process reaping.

Main file pointers

• engine/crates/xerj-autoindex/src/extract/pdf.rs: parser, worker protocol, quality gate, limits, cleanup, and provenance.
• engine/crates/xerj-autoindex/src/cli.rs: PDF controls and discoverability.
• engine/crates/xerj-autoindex/src/lib.rs: autoindex worker configuration.
• engine/crates/xerj-server/src/main.rs: pre-runtime hidden worker dispatch.
• engine/crates/xerj-server/tests/pdf_worker.rs: same-binary process and failure-mode coverage.

Remaining limitations and follow-ups

1. Phase-A sampling still parses and materializes the complete PDF, and phase-B indexing parses it again. Replace the full-response JSON protocol with framed streaming and an explicit sampling limit.
2. Operational error paths clean up explicitly, but a general RAII child/process-group guard would also cover exceptional panics during post-spawn setup.
3. Add broader public fixtures, especially real mixed scanned/text documents, multi-column layouts, rotated text, forms, encrypted files, malformed xrefs, and page-local finance ground truth.
4. Measure peak parent and worker RSS at PDF concurrency 1 and 4, and validate non-Unix behavior in CI.
5. Run the repository's complete transitive-license release audit before release.