carimbo turns Brazilian electronic invoices (NF-e), received as DANFE PDFs, into strictly typed invoice records, and later into typed approve / reject / escalate decisions. It is a portfolio project: a payments engineer applying production rigor to LLM agents in a regulated fintech setting, backed by evidence (structured outputs, deterministic validators, a reproducible eval suite, durable execution, idempotency, tracing and cost attribution). The core value is measured quality: an eval suite that exercises the real pipeline and publishes results anyone can reproduce.
Milestone 1, "Extraction, measured", has seven phases, including Phase 3.1 (Thin Agent Slice), inserted after Phase 3 by D-26. Phases 1 and 2 are complete (2026-10-08), Phase 3 is complete (2026-10-15), Phase 3.1 is complete (2026-10-22), Phase 4 is complete (2026-10-29) and Phase 5 is complete (2026-11-06). Phase 5, Comparable measurement, built the grader, confidence intervals, silent-error rate, failure tags and run comparison, proved them offline over all 154 dataset cases, and froze the failure tag list (D-48) before the test split has been run. The model evidence is still small: three synthetic extraction cases (one model, one run per configuration), eight synthetic decision cases (one model, one live run), and one incomplete live run on 51 of the 54 dev cases of the 154-case dataset. That run is a dev result, not a published one. The sections below separate what is measured from what is not.
| Phase | What it delivers | State |
|---|---|---|
| 1. Walking Skeleton | A few synthetic DANFEs extracted live through the eval endpoint, graded offline from Python, summarized | Complete (2026-10-08) |
| 2. Validated Extraction | Full DANFE-visible target, deterministic validators, bounded repair loop with every attempt recorded | Complete (2026-10-08) |
| 3. Replayable Runs | Request-hash cache with replay modes, single retry policy, costed spans in a local trace viewer, per-request overrides | Complete (2026-10-15) |
| 3.1. Thin Agent Slice (inserted) | Read-only MCP tools, hand-written agent loop with a step budget, typed approve/reject/escalate decision, graded decision cases | Complete (2026-10-22) |
| 4. Synthetic Dataset | 150+ seeded, varied and degraded cases with validated ground truth, manifest and fixed CI subset | Complete (2026-10-29) |
| 5. Comparable Measurement | Full graders, confidence intervals, silent-error rate, failure taxonomy and run comparison | Complete (2026-11-06) |
| 6. Gated CI and Results | Fork-safe PR gate from committed fixtures, budgeted live tier, published two-model results and failure analysis | Not started |
The plan is in .planning/ROADMAP.md. A thin agent slice moved into Milestone 1 as Phase 3.1 (D-26 in docs/DECISIONS.md); the full agent scope (Milestone 2) and durable execution (Milestone 3) stay outside this roadmap.
Each phase is built on its own branch and reaches main as a pull request merged with a merge commit; the branching, merge and README upkeep rules are in AGENTS.md.
Decisions D-22 to D-33 are recorded in docs/DECISIONS.md.
- Invoice v2 (D-22). Issuer, recipient (CNPJ or CPF), 14-column line items, 11 totals boxes and installments. Every field is required and only the two IE fields are nullable, so the model-facing schema uses 0 optional and 2 union properties against the provider limits of 24 and 16. CNPJ and access key accept numeric and alphanumeric forms. Money crosses the wire as pattern-constrained decimal strings and rounds half away from zero identically in C# and Python.
docs/DANFE-MAPPING.mdmaps every field to its DANFE label and NF-e XML source, and a test keeps its rows equal to the 42 schema leaf paths. - Validators (D-23).
Carimbo.Validationis pure, takes a reference date, never throws, and returns findings{field, rule_id, expected, actual, severity}for CNPJ and CPF check digits, the access-key check digit, key-versus-field mismatches, item and tax arithmetic, totals, installments and date plausibility. - Shared oracle.
data/vectors/validator-vectors.jsonholds hand-curated valid and invalid CNPJs, CPFs, access keys and rounding cases, and runs under both xUnit and pytest. - Bounded repair (D-23). Only error findings trigger repair. The conversation continues under prompt
repair-001with the PDF cached, andExtraction:MaxRepairsdefaults to 2 (allowed 0 to 5). Feedback reveals expected and actual values only for arithmetic and date rules, never for identifiers, check digits or keys, so repair cannot invent an identifier that merely satisfies a check. Exhausting the budget returnsvalidation_failedwith the last candidate and its findings. - Eval contract 2 (D-24). Statuses
success,validation_failed,refused,truncated,schema_invalidandinfrastructure_failure; every attempt with its output, findings, tokens, cost and latency; the effective configuration echoed. Total cost is null with a warning when any answered attempt is unpriced, never a partial sum. - Ground-truth gate.
Carimbo.GroundTruthmaps each committed case's NF-e XML to an Invoice, which must produce zero validator errors at the manifestas_of_date. - Strict parse boundary (D-25). Success means schema-valid. Null list elements and enum values that are not the exact wire name are
schema_invalid. - Response cache (D-27). The Api caches model answers by a request hash (model, max tokens, whether repair has marked the PDF for Anthropic prompt caching, prompt and repair prompt versions, hashes of the prompt text and schema, the document's hash and media type, the repair conversation and the replicate salt). Modes are
read_write,read_only(replay, zero model calls),refreshandoff. In Development and Eval the cache is on by default at the git-ignored.cache/llm. A cached answer reports the original latency and cost withhttp_attempts0, so a replay is never mistaken for spend. - One retry owner (D-28). The Anthropic SDK retries transport and provider failures (
MaxRetries2); a refused, truncated or validation-failing answer is never retried into a score. - Eval contract 3 (D-29). Per-request overrides (model, prompt version, repair budget, cache mode), a
cacheblock per attempt andincurred_cost_usdbeside the notionalcost_usd. The runner caps and reportsspent_usdon incurred cost andnotional_usdbeside it. - Prompt catalog (D-30). Versioned prompts selected by id, with the version stamped on every request and echoed in the response.
- Model-call spans and trace viewer (D-31). One
chat <model>span per attempt with model, token counts,carimbo.llm.cost_usd, the prompt version and, on repair calls,carimbo.llm.repair_prompt_version(D-33), never prompt, answer or document content; every eval response carries itstrace_id. Jaeger v2 runs locally fromcompose.yaml. - Golden fixtures (D-32). The golden eval responses are generated by the .NET tests and compared against the committed files (
just golden). - Current prompt.
extract-003, pinned by SHA-256 in the tests. Its first live run is in the Phase 3 results below.
Decisions D-34 to D-39 are recorded in docs/DECISIONS.md.
- Read-only MCP Tools host. A separate, stateless MCP server over Streamable HTTP (
Carimbo.Tools) serves a case's extracted invoice and validator findings, and the buyer policy, through read-only tools. - Hand-written agent loop.
Carimbo.Agentruns the conversation with a step budget (Agent:MaxSteps, 6) and a status guard: a case whose extraction did not succeed is decided by the guard, and a model approve or reject that contradicts the evidence is overridden and recorded. - Decision eval contract (decision-1).
POST /eval/decisionsreturns a typed approve, reject or escalate decision with an escalation cause and the evidence it cites, every step with its tool calls, tokens, cost and cache status. The same cache and replay modes as extraction apply. - Decision grading.
carimbo-evals decisionsandgrade-decisionsreport decision accuracy, escalation-cause accuracy, citation validity, guard overrides and typed failures over the eight committed cases indata/decision. - Offline end to end. The whole slice, two real processes against a scripted model, runs in
just checkwith no provider key.
The first live run of the eight decision cases and its replay are in the Phase 3.1 results below.
Decisions D-40 and D-41 in docs/DECISIONS.md record the dataset. data/dataset holds dataset-001, 154 synthetic cases generated from seed 20261022: 90 text-layer invoices (two tax regimes by four length strata, with fixed counts of the hard traits: 24 alphanumeric issuer CNPJs, 18 long-text and 18 confusable-character invoices, 14 with ST or deferral codes) and 64 image-only twins, the same eight invoices at eight renderings (a clean scan, rotation, blur and low resolution at two severities each, and a damaged barcode), so each rendering slice has n = 8. 54 cases are dev and 100 are test; a twin always shares the split of its base invoice. A 20-case, dev-only subset is listed in the manifest as the future CI subset. The directory is 19,562,414 bytes (the cap is 20,000,000), and just dataset-check regenerates all 309 files in about 50 seconds on the author's machine and fails on any byte difference. After the first user acceptance review, the severe blur was lowered from sigma 2.4 to 1.8 pixels so that the small item digits stay readable with effort, and the PDF date of the twins is pinned in code, so byte-identical regeneration does not depend on SOURCE_DATE_EPOCH (D-41). Byte-identical regeneration is promised on Linux x64 only; other platforms use the committed files. Building it called no model and cost US$0.
Decisions D-42 to D-49 in docs/DECISIONS.md record the grader, the tags, the statistics, compare, the live dev run, the wrap hazards and the freeze. All of this runs offline from the stored cases.jsonl of a run; nothing below calls a model.
- Whole-invoice grading (D-42).
carimbo-evals gradescores the 27 header fields, the 14 item columns on every true row and every installment. Items and installments are matched to the expected rows by optimal assignment (scipy.optimize.linear_sum_assignmentover a weighted identity score), so row order never changes a grade and a missing or invented row is an event of its own. Numbers are exact by default (tolerance 0.00, configurable with--tolerance); nothing fuzzy ever counts as correct. - Primary metric and denominator (D-42).
correct_invoice_rateis the share of invoices whose status issuccessand whose every graded value is right. The denominator is every case exceptinfrastructure_failureandharness_error, which are reported apart and mark the run incomplete; refused, truncated andschema_invalidcases count as not correct, so a model cannot raise its rate by truncating. - Failure tags (D-43, frozen by D-48). Each wrong value gets one tag from an ordered list of 19 rules (
failure-tags-1), pure code over the graded diff and the row assignment, with anuntaggedbucket that shows what the rules miss. Tag counts add up to the wrong-value total. - Headline rows, intervals and slices (D-44). The primary row is the text-layer cases and the second row the image-only cases, reported separately and never pooled into the headline. Every case-level rate carries a 95% Wilson interval whose n is the number of distinct invoices, because the image-only cases are renderings of a few invoices. Pooled row-level metrics (item recall, precision and column accuracy) carry a seeded bootstrap over invoices (10,000 resamples). The summary also holds 24 declared slices over 11 dimensions, model latency percentiles, cost and cache totals.
- Silent-error rate (D-42). The share of
successcases that carry at least one wrong value, split into those that touch a payment value (including the access key, CNPJ, invoice total, item totals and installment dates and amounts) and the rest. - Run comparison (D-45).
carimbo-evals compare --first A --second Breads twosummary.jsonfiles only. It runs one exact McNemar test on the correct-invoice flag over the shared text-layer cases, lists regressed, worsened, improved, status-changed and not comparable cases with the values and tags that flipped, and states that the verdict covers case sampling only, not run-to-run model variation. It exits 0 when it ran and 2 when it refused: a different dataset manifest, grader version or tolerance, a decision summary or a malformed file are refused with no override. - Regrading and published runs.
carimbo-evals grade --run RUN --cases CASES --out DIRregrades a stored run into a new directory. A directory underevals/resultsthat holds asummary.jsonis never rewritten, so every published run stays verbatim. - Proven offline on all 154 cases (D-46).
just e2edrives the documentedrun,gradeandcomparecommands over all 154 dataset cases against the scripted model, with ground-truth answers and with seeded corruptions, and checks the summary rows, statuses, tags, the silent-error split andcompare. That proves the machinery, not the quality of any model: the scripted answers are not model output.
The live runs are committed verbatim under evals/results/: each run directory holds the per-case records (cases.jsonl), the run metadata (run.json) and the graded summary (summary.json, summary.md). New runs land in the git-ignored evals/runs/ first. Each case was run once per configuration (n=1), so these numbers show the loop working and where it fails. They are not an accuracy estimate.
Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-002, repair prompt repair-001, dataset skeleton-002, 27 graded fields per case, recorded 2026-10-08. Commands: just skeleton (repair budget 2) and just skeleton 1.00 0 (repair disabled). Runs: repair budget 2, repair budget 0. Analysis: 02-10-SUMMARY.md.
Per field, over the three cases:
| Field | Repair budget 2 | Repair budget 0 |
|---|---|---|
access_key (44 digits) |
2/3 | 1/3 |
| The other 26 graded fields, each | 3/3 | 3/3 |
| All 27 fields, all cases | 80/81 | 79/81 |
| Schema-valid answers (JSON Schema and Pydantic) | 3/3 | 3/3 |
Wrong answers flagged validation_failed |
1 of 1 | 2 of 2 |
| Cases repaired | 0 | 0 |
| Model calls | 5 | 3 |
| Cost | US$0.0843 | US$0.0642 |
The other 26 fields are number, series, issue_date, operation_nature, the issuer's CNPJ, name, IE and UF, the recipient's tax id, tax id kind, name, IE and UF, the item and installment counts, and the 11 totals boxes.
Per case:
| Case | What it covers | max_repairs 2 | max_repairs 0 |
|---|---|---|---|
| case-001 | One page, Simples Nacional CSOSN 101, numeric CNPJs, 3 items | success, 1 attempt, 27/27 | validation_failed, 1 attempt, 26/27 (access_key wrong), KEY_CHECK_DIGIT |
| case-002 | One page, Regime Normal with IPI, freight and discount, 2 installments, 4 items | validation_failed, 3 attempts, 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_NUMBER_MISMATCH, KEY_SERIES_MISMATCH | validation_failed, 1 attempt, 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_NUMBER_MISMATCH, KEY_SERIES_MISMATCH |
| case-003 | Two pages, 50 items, alphanumeric issuer CNPJ, CPF recipient | success, 1 attempt, 27/27 | success, 1 attempt, 27/27 |
Totals: cost US$0.0843 against US$0.0642, 5 attempts against 3, 0 cases repaired. Both runs: 0 harness errors, 0 unpriced cases.
- The only wrong field in either run was the 44-digit access key, and every wrong key was flagged
validation_failedbyKEY_*rules (also UAT test 5 in02-UAT.md). No case reportedsuccesswith a wrong field in these six extractions. That is an observation about six extractions, not a silent-error rate. - Repair repaired 0 cases. The three attempts on case-002 gave three different wrong keys. The feedback withholds values for check-digit and key rules by design, so the model has nothing to anchor on except re-reading the image.
- The case-001 difference between the runs is run-to-run variance, not a repair effect: nothing was repaired in either run, and these models give no sampling control. With n=1 per cell this says only that access-key transcription is unstable on Haiku 4.5.
- Repair cost US$0.0157 extra on case-002 for no gain. With repair enabled, the PDF cache write made single-attempt cases dearer (+15% on case-001, +7% on case-003, indicative only). Run A read 12994 cache tokens on case-002's repair attempts.
- case-003 (50 items, output 6093 tokens) took 49586 ms and 49269 ms.
- The source lists three remedies for the access-key misread: a barcode-decoded key as input or cross-check, a stronger transcription instruction, or a different model. None is built. The barcode design intent is D-05 in
docs/DECISIONS.md; the requirement sits under the v2 requirements and is not in the Phase 3 roadmap.
Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-003, repair prompt repair-001, dataset skeleton-002, repair budget 2, recorded 2026-10-15. This is the first live run of extract-003. The seeding run used the default read_write cache mode against an empty .cache/llm (just skeleton), so every answer is a real model call; the replay used just skeleton 1.00 2 read_only. Runs: seeding run, read_only replay. Analysis: 03-10-SUMMARY.md.
| Case | Status | Attempts | Fields | Tokens in / out | Cost (seeding) | Latency (seeding) |
|---|---|---|---|---|---|---|
| case-001 | validation_failed | 3 | 26/27 (access_key wrong), KEY_CHECK_DIGIT |
5352 in, 1836 out (plus 6361 cache-read x2) | US$0.02375545 | 25783 ms |
| case-002 | validation_failed | 3 | 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_SERIES_MISMATCH, KEY_NUMBER_MISMATCH |
6414 in, 2343 out (plus 6532 cache-read x2) | US$0.02760040 | 31614 ms |
| case-003 | success | 1 | 27/27 | 998 in, 6093 out (plus 11349 cache-write) | US$0.04564925 | 48521 ms |
Run totals: 7 model calls, US$0.09700510 incurred, 12764 input and 10272 output tokens, 0 harness errors, 0 unpriced cases.
The replay of the same three cases made 0 provider calls: 7 of 7 answered attempts were cache hits (hit rate 1.0), http_attempts 0 on every attempt, and the same statuses and field results. Incurred cost was US$0.00000000; notional cost was US$0.09700510, equal to the seeding run's cost. The replay is a replay: the cost of extracting these three cases is the seeding run's US$0.0970, not the replay's zero. In the replay's summary.json, latency_ms is the real wall time of the replayed case (48, 48 and 8 ms) and model_latency_ms keeps the original model latency (25646 ms for case-001).
Trace lookup: each seeding-run trace_id resolved in the local Jaeger with a POST /eval/extractions server span and exactly one chat claude-haiku-4-5 span per attempt (3, 3 and 1), each carrying model, token counts and carimbo.llm.cost_usd, and none containing the case's raw model output.
What the run shows, at n=1 per case:
- The only wrong field was again the 44-digit access key. Both wrong keys were flagged
validation_failed; no case reportedsuccesswith a wrong field. Case-001 went from success underextract-002(Phase 2, repair budget 2) to failed here, and the Phase 2 note that access-key transcription is unstable on Haiku 4.5 still stands. Run-to-run variance and the prompt change cannot be separated with one run each. - Repair again repaired 0 cases. Case-001's repair attempts gave one more wrong key and then repeated it; case-002's second attempt repeated the first wrong key and the third changed its last digits. The repair attempts on the two failing cases cost US$0.0263 for no gain.
- The access-key remedies listed above are still unbuilt.
Model requested claude-haiku-4-5, prompt agent-001, dataset decision-001 (eight synthetic cases), grader decision-grader-001 (the committed summaries are its verbatim output; the later decision-grader-002 of the review fixes regrades both runs to the same numbers and adds only a citations.uncited count of 0, see D-39), labels decision-labels-1, step budget 6, recorded 2026-10-22. The live run used the default read_write cache mode (just decisions); the replay used just decisions 1.00 read_only. Runs: live run, read_only replay.
| Measure | Live run |
|---|---|
| Decision accuracy | 7/8 (0.875) |
| Cause accuracy, over the 4 expected escalations | 4/4 |
| Model decision accuracy (before the guard) | 7/8 |
| Citations valid | 8/8 |
| Guard overrides | 0 in 8 checked |
| Typed failures | 0 (8 decided, 0 harness errors) |
| Steps | 16 in total, 2 per case |
| Tool calls | 32 by the model (4 per case), 8 by the guard |
| Tokens in / out | 48304 / 2756 |
| Cost | US$0.06208400 incurred, US$0.06208400 notional, 0 unpriced |
| Latency per case | median 5752 ms, min 4548 ms, max 6928 ms |
Per case:
| Case | Expected | Decision | Correct | Citations | Cost |
|---|---|---|---|---|---|
| dc-001 | approve | approve | yes | valid | US$0.00666900 |
| dc-002 | reject | reject | yes | valid | US$0.01165100 |
| dc-003 | escalate / policy_warning | escalate / policy_warning | yes | valid | US$0.00660700 |
| dc-004 | reject | approve | no | valid | US$0.00651700 |
| dc-005 | escalate / extraction_not_success | escalate / extraction_not_success | yes | valid | US$0.00517500 |
| dc-006 | escalate / over_limit | escalate / over_limit | yes | valid | US$0.01201000 |
| dc-007 | approve | approve | yes | valid | US$0.00675900 |
| dc-008 | escalate / extraction_not_success | escalate / extraction_not_success | yes | valid | US$0.00669600 |
The replay made 0 provider calls: 16 of 16 answered steps were cache hits (hit rate 1.0), with the same decisions and grades. Incurred cost was US$0.00000000; notional cost was US$0.06208400, equal to the live run's. As with extraction, the cost of deciding these eight cases is the live run's US$0.0621, not the replay's zero. The replay's per-case latency is the real wall time of the replayed case (median 21 ms, max 551 ms).
What the run shows, at n=1 per case:
- One miss, left as it is. dc-004 (the recipient is another company; expected reject, rule 2) was approved. The model cited both relevant values,
recipient.tax_id69979991000100 fromget_invoiceandbuyer_cnpjCARIMBO0000177 fromget_policy, and its reason says the recipient tax ID matches policy. The two values differ. - Citation validity measures provenance, not correctness. The check passes when every cited evidence
sourcenames a tool the agent itself called in that run; the citedrefandvalueare not compared with the tool results (D-36). dc-004 passes it (8/8) while the comparison over the cited values is wrong. The status guard only looks at the extraction status, so nothing in code caught a recipient mismatch. Which of the following to try is not decided here: a deterministic recipient check, a stronger model, or a prompt change. Prompts, rules and cases were not changed after the run, and the grader was not changed before the numbers above were produced. - The guard decided the two failed-extraction cases (dc-005 and dc-008,
decided_byguard, escalate with causeextraction_not_success) and recorded 0 overrides: it never had to overturn a model approve or reject. - The four escalations were all right in cause (4/4), and the three approve or reject cases other than dc-004 were right. With eight cases and one run each, this is a demonstration of the loop and an honest miss, not an accuracy estimate.
Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-003, dataset dataset-001, grader grader-003, failure tags failure-tags-1, repair budget 2, recorded 2026-11-06. The live run used the default read_write cache mode (just dev-split); the replay used just dev-split 2.50 read_only. Runs: live dev run, read_only dev replay. Every figure below is a dev figure, from the live run's summary.json and run.json; none is a published number, and the test split has not been run.
The live dev run is incomplete: the US$2.50 dev cost cap stopped it (stopped_reason cost_cap) with 3 of the 54 selected dev cases not run (nfe-0144, nfe-0145 and nfe-0146), so every dev figure below covers the 51 dev cases that ran. It recorded 125 answered attempts, 0 harness errors and 0 unpriced cases.
| Dev row | Cases | Invoices | Excluded | Correct | Rate | 95% interval |
|---|---|---|---|---|---|---|
| dev text | 30 | 30 | 0 | 12 | 0.4000 | [0.2459, 0.5768] |
| dev image | 21 | 3 | 0 | 0 | 0.0000 | [0.0000, 0.5615] |
| dev pooled | 51 | 30 | 0 | 12 | 0.2353 | [0.1193, 0.4114] |
Dev statuses over the 51 cases: success 16, validation_failed 35, refused 0, truncated 0, schema_invalid 0. Dev silent errors (a success case with a wrong value): 4 of 16 success cases (0.2500, 95% interval [0.1018, 0.4950]), 0 in payment fields and 4 in other fields. Dev incurred cost US$2.30699480 (spent_usd), 431589 input and 277962 output tokens, 4 dev cases reached success after a repair attempt.
The replay made 0 provider calls: 125 of 125 answered attempts were cache hits and its spent_usd was US$0.00000000 (notional US$2.30699480, the live run's cost). It reproduced all 51 dev cases the live run recorded, with the same statuses and invoices.
Limit of the replay (D-46): the replay reproduces every cached answer, while refused and truncated cases are not cached, so they replay as cache_miss, leave the replay's denominator and mark it incomplete, which means its rate can differ from the live run's and the live run is the measure. Refused or truncated dev cases in the live run: 0 (all 125 live attempts ended end_turn and were stored). The replay's own 3 cache_miss dev cases (nfe-0144, nfe-0145 and nfe-0146) have a different cause: the live run never reached them, so no answer was stored. They are the replay's 3 excluded dev cases; its dev correct counts and rates equal the live run's.
What the dev failures look like. All figures here are dev figures from the live run's summary.json and summary.md (grader grader-003, tags failure-tags-1). The 51 dev cases hold 827 wrong values: 4 in success cases (the silent errors above) and 823 in validation_failed cases. Text-layer and image-only figures are reported separately because they are not comparable: the 30 text-layer cases are 30 independent invoices, while the 21 image-only cases are renderings of 3 invoices (nfe-0001, nfe-0048 and nfe-0064, with 8, 8 and 5 twins), so image counts are not independent. The text layer holds 67 of the 827 wrong values (in 18 of its 30 cases) and the image-only cases 760.
- Tags. The largest are
untagged265 (35 cases),invented_row172 (21),dropped_row94 (13),digit_substitution79 (16),trailing_rows_lost70 (13) andtext_wrong62 (23); the 19 tags are in D-48. Five tags had no value in this run (schema_invalid,party_swap,merged_rows,split_rows,thousands_misparse), so they are proven only on seeded corruptions. Of the 67 text-layer wrong values, 40 areinvented_rowon one case: nfe-0038, an 80-row invoice answered with 119 rows.dropped_row,trailing_rows_lost,digit_substitution,tax_code_wronganddate_wrongoccur on the image cases only. - Silent errors. Three of the four dev silent errors are
recipient.namelosing its last wordSINTETICA, the generator's synthetic marker; the fourth isPAforPARin the unit of nfe-0042. The marker is printed in the PDF text layer, so these stay counted as wrong (D-42, D-43) and the dev text row above stays 12 of 30. Each of the three is a case whose only wrong value is that name, which is what these three cases contribute to the 18 text-layer cases that are not correct. - Alphanumeric CNPJ. Of the 10 text-layer dev cases with an alphanumeric issuer or recipient identifier, 2 are correct, against 10 of the other 20. The samples are 10 and 20 cases, so this is a lead for Phase 6, not a finding.
- Image cases. None of the 21 image-only cases is correct, and
recipient.nameis wrong on all 21. In 15 of them the model returned a printed form caption or the homologation notice (10 containHOMOLOGACAO, 2 areDANFE, 3 areRAZAO SOCIAL); the other 6 are misread names. This is partly specific todataset-001: every synthetic DANFE is a homologation document, so the name box prints the notice and the recipient's name appears only in the small receipt stub (docs/DANFE-MAPPING.md). A production DANFE prints the name in its box. - Why a third of the wrong values stay untagged. 265 of 827 wrong values (32.0%) are
untagged. They sit in header-identifier and amount columns (access key, CNPJ, state registrations, item totals, quantities, unit prices, totals boxes) that field accuracy already reports column by column, and 243 of the 265 are on image cases, that is, on renderings of the three dev invoices above; the other 22 are on text cases. The list was frozen with this bucket rather than renamed with per-column catch-all tags (D-48).
Wrap hazards (D-47). The text layer of dataset-001 wraps the unit PAR as PA over R and wraps large prices and totals mid-number, and the grader never forgives either. 12 of the 30 text-layer dev cases carry such a hazard. They were correct on 5 of 12 with a hazard and on 7 of 18 without, and 7 of the 18 text-layer dev cases that are not correct carry one, so correctness is about the same either way. The hazards do not dominate the dev failures. Of the 18 wrong unit values in the run, 1 is on a PAR row (nfe-0042), and 14 of the 15 PAR rows were read correctly; the other 17 wrong units are scan misreads on image cases, which carry no hazard. The 4 confusable-character dev cases (the ones with many-digit prices and totals) were correct on 3, with 2 wrong values in total, an access key and a CNPJ, and none in a price or a total. The wrap is therefore a confound that stays visible and is not what explains this run, and the confusable slice must not be read as measuring row matching alone: its cases also carry the large-number wrap, and 4 dev cases cannot separate the two. The confound that does mark this run is the image-only rows (0 of 21 correct, 760 of 827 wrong values, 3 invoices).
What these numbers cannot show. 30 text-layer dev cases (30 invoices) and 3 image-only invoices, one run, one model (claude-haiku-4-5), 3 of the 54 selected dev cases not run, and no test result. The tag rules were fitted on seeded corruptions and these dev failures, so they say nothing yet about the holdout. None of these figures is a published result.
Phase 1 ran the same three cases on the first, smaller target (prompt extract-001, dataset skeleton-001, 9 graded fields per case): 3 of 3 success, but access_key was wrong on case-001 and case-002 (8/9 each) and case-003 was 9/9. Cost US$0.0195. Phase 1 had no check-digit validation, so those two wrong keys were reported as success. This is not like-for-like with Phase 2 (different dataset version, target and prompt). Run: skeleton-20261008T001300Z. Analysis: 01-13-SUMMARY.md.
docs/spikes/01-llm-gateway.md(2026-10-07): the direct Anthropic SDK behindILlmGatewaywas chosen because theIChatClientpath is not lossless on usage decomposition or stop details (D-21).thinking: disabledand a non-defaulttemperaturereturn HTTP 400 on Sonnet 5.5. A cache write followed by a read was observed on Haiku 4.5 and Sonnet 5.5. Spend US$0.2551 across four runs of the tool. Its per-case field counts are not a model comparison.docs/spikes/02-schema-probe.md(2026-10-08): the v2 model-facing schema was accepted unchanged (HTTP 200, schema-valid invoice, 0 pattern violations). A two-turn conversation with a cached PDF block read 6321 cache tokens on the repair turn. US$0.0151 for the recorded run, US$0.0526 across three runs (02-03-SUMMARY.md).docs/spikes/03-agent-probe.md(2026-10-22): bothclaude-haiku-4-5runs (dc-001 and dc-003) called all four tools before answering and returned a parsed decision through structured output (pathstructured_output), so the planned agent loop was kept (D-38); the thinking-block echo stayed unverified, because the step-1 answer of theclaude-sonnet-5-5echo check carried no thinking block. Spend US$0.0287 against the US$0.10 cap.
Phase 1: US$0.2788. Phase 2: US$0.2011 (probe US$0.0526 plus the two skeleton runs). Phase 3: US$0.0970 (the one seeding run; the replay spent nothing). Phase 3.1: US$0.0908 (the agent probe US$0.0287 plus the one live decision run US$0.0621; the replay spent nothing). Phase 4: US$0 (the dataset is generated offline and no model was called). Phase 5: US$2.3070 (the one live dev run: run.json spent_usd 2.30699480 against the US$2.50 cap, which stopped it at 51 of 54 dev cases; the replay's spent_usd is 0.00000000; the offline proofs and the tag freeze called no model). Each is under the US$5 phase cap. Every live run is capped at US$1.00 (US$0.25 for the schema probe, US$2.50 for the dev split).
- Phase 5: verification passed, 4/4 ROADMAP success criteria verified at the fixed code, with EVAL-03 to EVAL-08 accounted for (
05-VERIFICATION.md). The code review (05-REVIEW.md) found 1 critical issue, 4 warnings and 8 info findings. The critical issue (a killed run, or one resumed on a narrower selection, could grade as complete), all 4 warnings and 7 of the info findings were fixed (05-REVIEW-FIX.md). The eighth, IN-07, is an observation about the frozen tag list and was skipped by choice (05-REVIEW-DISPOSITION.md). A run with a missing or damagedrun.jsonnow grades as incomplete, andcarimbo-evals runrefusestestcases without--allow-test-split, a flag no recipe passes (D-49). After the fixes, both published dev runs regrade to the same grades, rates and intervals: onlygraded_at, the two primary-metric pointers, acache_misscount on the replay and the wording of two summary lines differ. All 30 threats in the register are closed, 2 of them as accepted risks (05-SECURITY.md). - Phase 4: verification passed with 4/5 must-have truths verified automatically, re-verified after the gap-closure plans 04-10 to 04-12 (
04-VERIFICATION.md). The fifth (severe renderings are hard but still human-readable) is a visual judgement and was settled by UAT. UAT is complete: the first round ran 3 tests, 1 passed (renderer fidelity against a reference DANFE) and 2 issues were found. G-04-1: at sigma 2.4 the severe blur turned small item-table digits into misreads (code D00137 read as D00132, NCM 39191010 as 39198890). G-04-2: the twin PDF bytes depended onSOURCE_DATE_EPOCH, so CI run 40915848246 failed two twin determinism tests on ubuntu-latest. Plan 04-11 lowered the severe blur to sigma 1.8 and plan 04-10 pinned the twin PDF date in code (D-41), and the re-test round passed 2 of 2: read cold at native 220 DPI, every item code, NCM and quantity of the blur-severe twins nfe-0095 and nfe-0127, and the access key of nfe-0127, matched the XML, and CI run 40950655793 was green on ubuntu-latest withoutSOURCE_DATE_EPOCH(just py-check2810 tests passed,just dataset-checkin 63 s) (04-UAT.md). The code review (04-REVIEW.md) found 0 critical issues, 4 warnings and 8 info findings, all open in04-REVIEW-DISPOSITION.md. One of them, IN-07, notes that the severe blur level rests on one reader, who transcribed the mild blur just as correctly, so the contrast between mild and severe blur is thin. - Phase 3.1: verification passed, 5/5 must-haves verified (
03.1-VERIFICATION.md). It records two documented limitations: the dc-004 miss described above, and the live echo of thinking blocks in a tool conversation, which is unverified (D-38) because no live step-1 answer has carried a thinking block yet. The echo is pinned only offline, by the adapter tests and cache v2 entries, and the defaultclaude-haiku-4-5does not think. The code review (03.1-REVIEW.md) found 0 critical issues, 11 warnings and 18 info findings. All 11 warnings were fixed (03.1-REVIEW-FIX.md), and the 18 info findings stay open in03.1-REVIEW-DISPOSITION.md. - Phase 3: verification passed, 7/7 must-have truths verified, re-verified after the gap-closure plan 03-11 and the UAT (
03-VERIFICATION.md). UAT is complete: the first round ran 5 tests, 3 passed and 2 issues were found. G-03-1: in Jaeger, repair-call spans could not be told apart from the initial extraction call. G-03-2: the README's response-cache key list did not match the key. Plan 03-11 closed both, and the re-test round passed 2 of 2 (03-UAT.md). - Phase 2: verification passed, 5/5 ROADMAP success criteria, re-verified after the UAT gap-closure plan 02-13 (
02-VERIFICATION.md). UAT ran 56 checks: 54 passed and 2 issues were found. G-02-1: the mapping doc pointedrecipient.nameat a box that prints the homologation notice. G-02-2: promptextract-002used three totals labels and a column header that differ from the printed DANFE. Plan 02-13 closed both and introduced promptextract-003(02-UAT.md). - Phase 1: verification passed with 4/5 criteria verified automatically (
01-VERIFICATION.md). The fifth (fresh clone, non-Nix path, both agent runtimes, CI on the first PR) was confirmed by UAT, 3 of 3 passed. - Open review warnings WR-02 to WR-06 are tracked in
02-REVIEW-DISPOSITION.md. For example, an export recipient's UFEXis flaggedUF_UNKNOWN.
extract-003has had one live run (n=1 per case). The Phase 2 numbers above usedextract-002and are not like-for-like.- The three skeleton cases are one run per configuration on one model: they give no accuracy estimate. The grader, intervals, silent-error rate and failure tags of Phase 5 are measured only on the dev run above.
- No
testsplit run and no paid full-dataset run (Phase 6). The tag list is frozen (D-48) and thetestsplit has no model answer, so there is no published result. - No two-model table and no cost per correct invoice (Phase 6), and no CI gate on regressions (Phase 6).
- No replicates and no noise floor (MEAS-02), so
comparecovers case sampling only, not run-to-run model variation; no repair ablation (MEAS-01). - The tag rules rest on seeded corruptions and on the 51 of 54 dev cases that ran, with one model and one run. Five of the 19 tags have no real occurrence yet, and the image rows come from 3 invoices.
- Extraction on the 154-case dataset: only 51 of the 54 dev cases have a live run (the cost cap stopped it), and the 100
testcases have none. The dev figures are not published numbers. - Whether repair improves accuracy is unknown: it has repaired zero cases so far.
- Decisions: eight cases, one run, one model (
claude-haiku-4-5). No decision evals at scale, no LLM-as-judge for the reasons, no second model on the agent, and no fix or retest for the dc-004 recipient miss.
You need three tools. Nothing else is installed globally.
- The .NET SDK that
global.jsonpins (10.0.100, or a later 10.0 feature band). - uv 0.8.17 or newer. It provisions Python 3.12 and the locked ruff, pyright and pytest.
- just. If you do not want to install it,
uv tool run --from rust-just justruns it.
Then, from the repo root:
just checkjust check runs every offline gate: both stacks, the contract chain (JSON Schema and generated Pydantic models), byte identity of the synthetic dataset, the end-to-end test against a scripted model, the docs check and the secrets check. It needs no API key and no network beyond the package registries.
The pyright package on PyPI is a wrapper around the Node tool and fetches a Node runtime on first use when none is installed. That is expected on a fresh machine.
A minimal Linux image also lacks two system libraries: the ICU library (a .NET runtime prerequisite; the UAT image needed libicu74) and libatomic1 (for the Node runtime that pyright downloads). Desktop distributions and GitHub runners ship both.
CI runs this same path (the same recipes, on a clean Ubuntu runner without Nix) on every pull request.
On NixOS or any machine with devenv:
devenv shell
just checkor one-shot: devenv shell -- just dotnet-check py-check. The shell provides the .NET 10 SDK, uv, just, Python 3.12, Node (for pyright), git, curl and the secretspec CLI. It runs no services, and ruff and pyright still come from the uv lock, not from nixpkgs.
| Command | What it does |
|---|---|
just dotnet-check |
Build with warnings as errors, verify formatting, run the xUnit v3 tests |
just py-check |
uv sync --locked, ruff check and format check, pyright, pytest |
just schema |
Export schema/*.json from the .NET Domain, then regenerate the Pydantic models |
just schema-check |
Regenerate, then fail if the committed schema or models differ |
just datagen |
Regenerate the seeded synthetic dataset in data/skeleton |
just datagen-check |
Fail unless a regeneration is byte-identical to the committed dataset |
just dataset |
Regenerate the Phase 4 dataset in data/dataset from the quota table and the seed |
just dataset-check |
Fail unless a regeneration is byte-identical to data/dataset (Linux x64) |
just decision-cases |
Regenerate data/decision, the eight decision cases, from the skeleton cases and the specs in dotnet/tools/DecisionCases |
just decision-cases-check |
Fail unless a regeneration is byte-identical to data/decision |
just golden |
Regenerate the golden eval and decision responses from the .NET tests; the compare-mode test fails just dotnet-check when they are stale |
just e2e |
Committed cases over real HTTP to a graded summary, against a scripted model |
just docs-check |
Fail unless decision records D-18 to D-49, the gateway spike recommendation and the schema-probe and agent-probe results exist |
just secrets-check |
Fail when a tracked file holds a provider-key-shaped string |
just check |
All of the above, offline |
just skeleton |
Live: extract the three skeleton cases with Claude and grade them (paid, capped at US$1.00 per run) |
just skeleton 1.00 0 |
Same with repair disabled (Extraction:MaxRepairs 0); the default budget is 2 |
just skeleton 1.00 2 read_only |
Replay the last recorded skeleton answers from .cache/llm: zero model calls, US$0 incurred |
just decisions |
Live: decide the eight decision cases with Claude through the agent loop and grade them (paid, capped at US$1.00 per run; starts the Tools host and the Api) |
just decisions 1.00 read_only |
Replay the last recorded decision answers from .cache/llm: zero model calls, US$0 incurred |
just dev-split |
Live: extract the 54 dev cases of data/dataset with Claude and grade them (paid, capped at US$2.50 per run; this recipe only) |
just dev-split 2.50 read_only |
Replay the last recorded dev-split answers from .cache/llm: zero model calls, US$0 incurred |
just agent-probe |
Live: probe tools plus structured output on the agent conversation (paid, capped at US$0.10, rewrites docs/spikes/03-agent-probe.md) |
just spike-live |
Live: re-run the LLM gateway spike (paid, rewrites docs/spikes/01-llm-gateway.md) |
just schema-probe |
Live: probe the v2 schema and a cached two-turn conversation (paid, rewrites docs/spikes/02-schema-probe.md) |
The grading and comparison commands behind these recipes are carimbo-evals run, grade (with --out to regrade into a new directory) and compare --first A --second B, run from the repo root through uv run --project python. compare reads two summary.json files, exits 0 when it ran and 2 when it refuses (a different dataset manifest, grader version or tolerance, or a decision summary).
docker compose up -dstarts Jaeger v2 from compose.yaml (pinned image tag). The UI is at http://localhost:16686 and the OTLP gRPC receiver at 127.0.0.1:4317. Every port is bound to loopback only, because the UI has no authentication. Storage is in memory: docker compose down stops the viewer and drops every trace.
To send spans from a skeleton run, point the exporter at that receiver: OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 just skeleton. The Api exports nothing when the variable is unset. To look a case up, take its trace_id from evals/runs/<run>/cases.jsonl or summary.md and open http://localhost:16686/trace/<trace_id>. Each model call is one chat <model> span carrying the model, the token counts and carimbo.llm.cost_usd, and never the prompt, the answer or the document content. The first call of a case carries carimbo.llm.prompt_version (the extraction prompt) and each repair call also carries carimbo.llm.repair_prompt_version (repair-001), the version that call's attempts[].prompt_version reports.
devenv does not provide Docker. Any Docker engine with compose v2 works, and it is needed for this step only: nothing in just check needs Docker.
The provider key is optional. Without it, just check runs everything against the fake model client, and the live recipes (just skeleton, just decisions, just dev-split, just spike-live, just schema-probe, just agent-probe) exit 2 without calling the model when no key is available.
Without secretspec (any reviewer). Export CARIMBO_ANTHROPIC_API_KEY in the shell that runs just. To keep the value out of your shell history:
read -rs CARIMBO_ANTHROPIC_API_KEY
export CARIMBO_ANTHROPIC_API_KEYANTHROPIC_API_KEY works as a fallback. carimbo prefers its own name because Claude Code uses ANTHROPIC_API_KEY for its own authentication.
With secretspec (optional; the author's dev setup). secretspec keeps the key in the OS keyring instead of your shell. The repo's secretspec.toml declares the key as optional.
- Run
secretspec config global initonce per machine and pick a provider such as the OS keyring. Thedevelopmentprofile that init selects is already declared insecretspec.toml. - Run
secretspec set CARIMBO_ANTHROPIC_API_KEYwithout--profileand paste the value at the prompt. Never pass the value as an argument. - Run
just skeleton,just decisions,just dev-split,just spike-live,just schema-probeorjust agent-probedirectly. When the variable is not exported they callsecretspec runthemselves. Or launch the agent session assecretspec run -- claude --continue(orsecretspec run -- opencode). A session sees only the variables present when it was launched, so restart it after setting the key.
An agent that calls secretspec itself must pass --reason or set SECRETSPEC_REASON; the recipes already set one. If secretspec is installed but not configured for this repo, exporting the variable bypasses it.
just skeleton costs at most US$1.00 per run (the runner stops at the cap). Its second argument is the repair budget (default 2); just skeleton 1.00 0 runs without repair. Its third argument is the cache mode. In Development the Api caches eval requests by default at .cache/llm (D-27, git-ignored), so repeating just skeleton with the same settings is answered from the cache. just skeleton 1.00 2 read_only replays the last recorded answers with zero model calls (the provider key must still be present), just skeleton 1.00 2 refresh calls the model again and records the new answers, and just skeleton 1.00 2 off bypasses the cache for that run (a per-request override, so it works while the host cache stays on). Read the result in evals/runs/<run>/summary.md; that directory is git-ignored, and a run worth keeping is copied verbatim to evals/results/<run>/.
just decisions costs at most US$1.00 per run (the runner stops at the cap). It starts the Tools host (127.0.0.1:5090) and the Api (127.0.0.1:5080), runs the eight decision cases and grades them. Its second argument is the cache mode, with the same meaning as for just skeleton: just decisions 1.00 read_only replays the last recorded answers with zero model calls (the Tools host still runs, because tool definitions and results are part of the cache key, and the provider key must still be present). A run worth keeping is copied verbatim to evals/results/<run>/.
just dev-split costs at most US$2.50 per run (the runner stops at the cap; no other recipe has that cap, and the phase total stays US$5). It extracts the 54 dev cases of data/dataset and grades them against that dataset. Its second argument is the cache mode, as for just skeleton: just dev-split 2.50 read_only replays the last recorded answers with zero model calls (the provider key must still be present). The test split has no recipe in this phase. Regrade a kept run with carimbo-evals grade --run RUN --cases data/dataset --out DIR; a directory under evals/results is never rewritten.
Never print, log or commit the key. Use a dedicated key with a low spend limit.
dotnet/ .NET 10 solution (Carimbo.slnx)
src/ Carimbo.Domain, .Llm, .Extraction, .Validation, .GroundTruth, .Decisions, .Tools, .Agent, .Api
tests/ xUnit v3 test projects and the scripted host used by the e2e test
tools/ SchemaExport (writes schema/), DecisionCases (writes data/decision), LlmSpike (live gateway spike)
python/ one uv project: carimbo_datagen, carimbo_evals, carimbo_models (generated)
schema/ canonical and model-facing JSON Schema, exported from the Domain
data/skeleton/ seeded synthetic NF-e cases (XML ground truth + DANFE PDF) and a manifest
data/dataset/ the Phase 4 dataset: 154 synthetic cases (XML ground truth, text-layer and image-only PDFs) and a manifest
data/decision/ generated decision cases, buyer policy, manifest and expected labels
data/vectors/ validator-vectors.json (shared oracle) and the shared valid-invoice.json fixture
data/contract/ golden extraction and decision responses (generated) and the two hand-written request examples
evals/runs/ run output (git-ignored)
evals/results/ committed copies of the live runs the README cites
docs/ DECISIONS.md, DANFE-MAPPING.md and the spike records
.github/workflows/ the CI workflow (the same just recipes)
.planning/ GSD planning artifacts: plan summaries, verification and UAT reports
All data in this repository is synthetic: fictitious companies, generated CNPJs with valid check digits, no real taxpayer data (DECISIONS D-14).
Agent instructions for Claude Code and OpenCode live in AGENTS.md.