Background

In the Go ingestion pipeline, a dataflow run can be non-persist (e.g. doc_id == CANVAS_DEBUG_DOC_ID / "dataflow_x", used by the canvas debug / dry-run path). Such runs intentionally skip all persistence (no MinIO image upload, no pipeline log, no index insert) but still run embedding to validate the pipeline.

Problem

Today the non-persist run still parses the entire document (parser + chunker) before returning, even for very large files. Since nothing is persisted and only embedding is exercised for validation, fully parsing a large PDF/DOCX wastes parser and embedding compute/cost.

Proposal

When !persist, inject a limit (e.g. maxPages / maxChunks) through the run context so that:

• the parser only parses the first maxPages pages, and/or
• the chunker only emits the first maxChunks chunks.

This bounds parsing + embedding cost for large documents while still exercising the full pipeline (including embedding) for a quick sanity check.

Notes

• This is a follow-up to the dataflow non-persist zero-side-effect work; it was deliberately excluded from the initial change to keep the diff minimal.
• Embedding must remain running (do not gate it on limit); only the input volume is capped.