From real-world PDFs to embedding-ready JSONL in one call #304

A common question in the Q&A here is "how do I get embedding-ready chunks from a PDF?". The one-liner answer is doc.rag_chunks(). This post shows that one-liner running against five real government and academic PDFs — not synthetic fixtures — and turning each into RAG-ready JSONL you can pipe straight into a vector store.

The full example lives at oxidize-pdf-core/examples/rag_realworld.rs. It downloads each PDF (cached after the first run), chunks it, and writes one chunk per line to ./out/<slug>.jsonl.

The corpus

These are deliberately messy: multi-column scientific layout, dense legal text in three languages, official government formatting. Production RAG ingestion looks like this, not like a clean single-column whitepaper.

The code that matters

The whole chunking step is one call. Everything else in the example is download caching and JSONL serialization:

use oxidize_pdf::parser::{PdfDocument, PdfReader};

let reader = PdfReader::open(path)?;
let doc = PdfDocument::new(reader);

// One line. Default config: 512-token target, heading propagation on.
let chunks = doc.rag_chunks()?;

for chunk in &chunks {
    // chunk.full_text       — heading context prepended, use for embeddings
    // chunk.heading_context — nearest enclosing section heading
    // chunk.page_numbers    — for citation / visual grounding
    // chunk.element_types   — "paragraph", "title", "list_item", "table", ...
    // chunk.token_estimate  — word-count proxy (see note below)
    // chunk.is_oversized    — single element exceeded the token budget
}

What a chunk actually looks like

A real line from out/ens.jsonl (BOE Real Decreto 311/2022), verbatim:

{
  "id": "ens-0077",
  "text": "El marco organizativo está constituido por un conjunto de medidas relacionadas con \nla organización global de la seguridad.",
  "metadata": {
    "page_numbers": [31],
    "heading_context": "3. Marco organizativo [ORG]",
    "element_types": ["paragraph"],
    "token_estimate": 18,
    "is_oversized": false
  }
}

The heading_context is inherited from the nearest enclosing section heading even when that heading appears earlier in the document. That is what lets you embed full_text (heading + body) so a retriever can disambiguate "Marco organizativo" from "Marco operacional" — two sections with near-identical opening sentences.

Aggregate run (oxidize-pdf 2.13.0)

[ok]   ens           → 288 chunks   ~191 tok/avg   35 oversized   200 headings
[ok]   boe-sumario   →  34 chunks   ~292 tok/avg    5 oversized     4 headings
[ok]   higgs         → 290 chunks   ~93 tok/avg     4 oversized   213 headings
[ok]   bsi-tr-02102  → 344 chunks   ~92 tok/avg     0 oversized   149 headings
[ok]   ncsc-caf      → 160 chunks   ~139 tok/avg    8 oversized   114 headings

5/5 documents processed successfully · 1116 total chunks

Every document parsed, chunked, and serialized without the pipeline aborting. Per-file error isolation means one bad document never poisons the batch.

A word on extraction fidelity

Chunking is only as good as the text extraction underneath it, and that varies with the source layout. Clean single-column government text (the BOE documents above) extracts faithfully. Dense two-column scientific PDFs (the Higgs paper) can still reorder fragments within a line where subscripts/superscripts shift the baseline, and some PDFs with non-ASCII typographic quotes can surface replacement characters. These are tracked openly in #302 — if you hit it, an issue with the source PDF attached is the most useful thing you can file.

Running it yourself

The example is committed in the repo — clone, run, inspect the JSONL:

cargo run --example rag_realworld
# writes ./out/<slug>.jsonl, one chunk per line, RAG-ready

A runnable index of the RAG examples (README) and a committed sample of the output shape let you see a chunk without downloading any PDF.

token_estimate is a word-count proxy. BPE/WordPiece tokenisers produce roughly 1.3–1.7x more sub-word tokens than raw words, so the default max_tokens: 512 lands around ~300–390 actual model tokens. Tune with rag_chunks_with(HybridChunkConfig { max_tokens: 256, ..Default::default() }).

Related Q&A: getting embedding-ready chunks (#247), customising chunk size (#250), what metadata survives chunking (#251), and 0-chunk diagnosis (#254).