Replace per-source extraction parsers with a single LLM path — measured, model-primary across all corpora #205

Question

Can a local, CPU-bound LLM replace the per-source extraction parsers — deleting most of the
~6,195 lines / ~90 regexes in sources/, and surviving City reformatting that silently breaks
regex?

Measured over two days. Answer: no for replacement, yes for drift robustness. The two
findings are not in conflict; they describe different regimes.

TL;DR

Setup

Qwen3-14B-Q4_K_M via llama.cpp, schema-constrained (outlines), 4 CPU threads, i3-13100 / 15 GB,
no GPU. Scored against 20,848 already-validated rows across three deliberately contrasting
structures. Ground truth is each corpus's own validated parser output plus, where available, the
documents' own declared bid counts.

Result 1 — drift robustness: the LLM wins decisively

Formatting mutations applied to the cell grid, both extractors given byte-identical input.
Mutations mimic plausible City reformats, not adversarial tricks.

Three of four reformats take the parser to near-zero — and this is the hardened cell parser
from #151, not a strawman. Inserting a single column zeroes regex and costs the model nothing.

Result 2 — replacement: the incumbent parsers win

Deterministic location (existing anchors) + LLM extraction, grouped per document:

noloc = 0 everywhere — every failure is extraction, not retrieval, which rules out
"better anchoring would fix it". Generic retrieval was also tried and was worse
(29%/56% on TRCA).

Why — the important part

The value in these parsers is not the regex. It is the documented exceptions, each of which
cost real measurement to discover:

• Swap pdftotext for pdfplumber across the PDF sources #116 — the City leaves numbered row 5. blank rather than omitting the bidder; an RFP lists
  proponents with no price at all; a multi-package cell holds a whole column and must be
  zipped positionally or refused.
• Bid Committee bid tables are column-major — 417 BD agendas yield 36 bids #94 — an OUTCOME in the price column (Non-Compliant) is still a bid.
• Link bids to the supplier dimension — 5,096 bidders are free text today #87/Swap pdftotext for pdfplumber across the PDF sources #116 — 2489960 Ontario Inc. is a real firm, not a leaked price.

A general prompt cannot reproduce these, and a larger model will not either: nothing in the
document says that a blank numbered row means an omission rather than an absence. Capability
does not fix this; corpus knowledge does, and corpus knowledge is what the parsers ARE.

Result 3 — GLiNER2 rejected, and why the safety argument failed

GLiNER2 (Fastino, 205M, Apache 2.0, CPU-first) was tried first because it tags spans in the
input and therefore cannot invent a value. That property held and was worthless alone:

• verbatim verification failures: 0
• (bidder, price) pairs correct: 29/80 — 36%

Every name real, every price real, 64% of the relations between them fabricated. It emitted
names in bullet-list order (alphabetical) and zipped them positionally against prices in table
order (ascending), assigning $548,415 to a firm that was disqualified and had no price at all.

A bid record is a relation, and a relation has no verbatim existence in the text to check
against. This archive is almost entirely relations — bidder↔price, table↔solicitation,
award↔supplier. Any future local-model work must be scored on relation accuracy against
independent ground truth, never on whether extracted strings appear in the source.

Cost, if we ever do adopt this

• Steady state is tiny: under a dozen new documents/day (Award Summary Forms ~30/month,
  agency board reports ~400/year). At ~31 s/doc that is ~5 minutes a night — a nightly that
  already drives a headed Chromium will not notice. Model choice is capped by RAM, not
  throughput.
• Backlog is one-time: 6,150 documents ≈ 53 h on this CPU, ≈ 6 h on an RTX 3060.
• Constrained decoding is a hard requirement, not an optimisation. Unconstrained, a 1.7B
  emitted malformed JSON on the simplest table in the corpus. With reasoning enabled the same
  model read that table correctly in prose — comprehension was never the bottleneck,
  serialisation was.

Proposed direction (not implemented)

1. Keep the existing parsers as the primary path. They measure better and encode discoveries.
2. LLM as a drift FALLBACK — fires only when a parser returns zero rows or fails its own
   ground-truth check (a declared bid count it cannot satisfy). Near-zero cost, and it converts a
   City reformat from a silent outage into a degraded-but-working night. This addresses the
   actual motivation.
3. LLM as the FIRST pass for NEW sources — where the O(n)-per-source cost really lives. Every
   new agency has needed a bespoke parser; start new ones on LLM + ground-truth check and write a
   parser only where it measurably underperforms.

Honest limitation: this does not reduce the existing ~6,195 lines, which was the original
goal. The measurement does not support deleting them.

If adopted, one hard constraint

db.rebuild_agency_bids re-derives every row for a source on each store pass — correct when
derivation is regex and costs milliseconds, ruinous when it is a 14B call. It would need a cache
keyed on the PDF's sha256 plus a model+prompt version stamp, never a "skip if rows exist"
guard (which is what makes stale rows persist).

Caveats on this evaluation

• Samples are 30 documents/corpus, not full corpora.

• Unverified: whether award_summary's missing 40% actually clusters on the Swap pdftotext for pdfplumber across the PDF sources #116 exception
  shapes. Cheap to check by diffing missed rows; it separates "buy a bigger model" from "the
  exceptions do not transfer".

• Four apparatus bugs occurred during this evaluation, each producing a plausible-looking
  number: a flattening bug returning exactly one row per document; unconstrained malformed JSON
  scored as extraction (recall appeared to rise under corruption); a patch that silently failed
  while relaunching stale code; and a grouping artifact that scored TRCA documents once per award
  and counted the other awards' bidders as false positives. Three were caught only because a
  number was impossible rather than merely wrong.

  That is the most transferable finding here. A general extraction layer needs ground-truth
  invariants wired in as a runtime property, not an evaluation step — the way schema_check.py
  already works for feed drift. Without that, we would not discover the archive was wrong; it
  would just get quieter.

Full working notes: docs/superpowers/notes/2026-07-28-* and 2026-07-29-*.