Skip to content
tarcPublic

About

An AI agent for Brazilian NF-e invoices: extraction, tool-using decisions, durable execution, and a measured eval suite.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

carimbo

carimbo turns Brazilian electronic invoices (NF-e), received as DANFE PDFs, into strictly typed invoice records, and later into typed approve / reject / escalate decisions. It is a portfolio project: a payments engineer applying production rigor to LLM agents in a regulated fintech setting, backed by evidence (structured outputs, deterministic validators, a reproducible eval suite, durable execution, idempotency, tracing and cost attribution). The core value is measured quality: an eval suite that exercises the real pipeline and publishes results anyone can reproduce.

Milestone 1, "Extraction, measured", has seven phases, including Phase 3.1 (Thin Agent Slice), inserted after Phase 3 by D-26. Phases 1 and 2 are complete (2026-10-08), Phase 3 is complete (2026-10-15), Phase 3.1 is complete (2026-10-22), Phase 4 is complete (2026-10-29) and Phase 5 is complete (2026-11-06). Phase 5, Comparable measurement, built the grader, confidence intervals, silent-error rate, failure tags and run comparison, proved them offline over all 154 dataset cases, and froze the failure tag list (D-48) before the test split has been run. The model evidence is still small: three synthetic extraction cases (one model, one run per configuration), eight synthetic decision cases (one model, one live run), and one incomplete live run on 51 of the 54 dev cases of the 154-case dataset. That run is a dev result, not a published one. The sections below separate what is measured from what is not.

Status

Phase What it delivers State
1. Walking Skeleton A few synthetic DANFEs extracted live through the eval endpoint, graded offline from Python, summarized Complete (2026-10-08)
2. Validated Extraction Full DANFE-visible target, deterministic validators, bounded repair loop with every attempt recorded Complete (2026-10-08)
3. Replayable Runs Request-hash cache with replay modes, single retry policy, costed spans in a local trace viewer, per-request overrides Complete (2026-10-15)
3.1. Thin Agent Slice (inserted) Read-only MCP tools, hand-written agent loop with a step budget, typed approve/reject/escalate decision, graded decision cases Complete (2026-10-22)
4. Synthetic Dataset 150+ seeded, varied and degraded cases with validated ground truth, manifest and fixed CI subset Complete (2026-10-29)
5. Comparable Measurement Full graders, confidence intervals, silent-error rate, failure taxonomy and run comparison Complete (2026-11-06)
6. Gated CI and Results Fork-safe PR gate from committed fixtures, budgeted live tier, published two-model results and failure analysis Not started

The plan is in .planning/ROADMAP.md. A thin agent slice moved into Milestone 1 as Phase 3.1 (D-26 in docs/DECISIONS.md); the full agent scope (Milestone 2) and durable execution (Milestone 3) stay outside this roadmap.

Each phase is built on its own branch and reaches main as a pull request merged with a merge commit; the branching, merge and README upkeep rules are in AGENTS.md.

What works today

Decisions D-22 to D-33 are recorded in docs/DECISIONS.md.

  • Invoice v2 (D-22). Issuer, recipient (CNPJ or CPF), 14-column line items, 11 totals boxes and installments. Every field is required and only the two IE fields are nullable, so the model-facing schema uses 0 optional and 2 union properties against the provider limits of 24 and 16. CNPJ and access key accept numeric and alphanumeric forms. Money crosses the wire as pattern-constrained decimal strings and rounds half away from zero identically in C# and Python. docs/DANFE-MAPPING.md maps every field to its DANFE label and NF-e XML source, and a test keeps its rows equal to the 42 schema leaf paths.
  • Validators (D-23). Carimbo.Validation is pure, takes a reference date, never throws, and returns findings {field, rule_id, expected, actual, severity} for CNPJ and CPF check digits, the access-key check digit, key-versus-field mismatches, item and tax arithmetic, totals, installments and date plausibility.
  • Shared oracle. data/vectors/validator-vectors.json holds hand-curated valid and invalid CNPJs, CPFs, access keys and rounding cases, and runs under both xUnit and pytest.
  • Bounded repair (D-23). Only error findings trigger repair. The conversation continues under prompt repair-001 with the PDF cached, and Extraction:MaxRepairs defaults to 2 (allowed 0 to 5). Feedback reveals expected and actual values only for arithmetic and date rules, never for identifiers, check digits or keys, so repair cannot invent an identifier that merely satisfies a check. Exhausting the budget returns validation_failed with the last candidate and its findings.
  • Eval contract 2 (D-24). Statuses success, validation_failed, refused, truncated, schema_invalid and infrastructure_failure; every attempt with its output, findings, tokens, cost and latency; the effective configuration echoed. Total cost is null with a warning when any answered attempt is unpriced, never a partial sum.
  • Ground-truth gate. Carimbo.GroundTruth maps each committed case's NF-e XML to an Invoice, which must produce zero validator errors at the manifest as_of_date.
  • Strict parse boundary (D-25). Success means schema-valid. Null list elements and enum values that are not the exact wire name are schema_invalid.
  • Response cache (D-27). The Api caches model answers by a request hash (model, max tokens, whether repair has marked the PDF for Anthropic prompt caching, prompt and repair prompt versions, hashes of the prompt text and schema, the document's hash and media type, the repair conversation and the replicate salt). Modes are read_write, read_only (replay, zero model calls), refresh and off. In Development and Eval the cache is on by default at the git-ignored .cache/llm. A cached answer reports the original latency and cost with http_attempts 0, so a replay is never mistaken for spend.
  • One retry owner (D-28). The Anthropic SDK retries transport and provider failures (MaxRetries 2); a refused, truncated or validation-failing answer is never retried into a score.
  • Eval contract 3 (D-29). Per-request overrides (model, prompt version, repair budget, cache mode), a cache block per attempt and incurred_cost_usd beside the notional cost_usd. The runner caps and reports spent_usd on incurred cost and notional_usd beside it.
  • Prompt catalog (D-30). Versioned prompts selected by id, with the version stamped on every request and echoed in the response.
  • Model-call spans and trace viewer (D-31). One chat <model> span per attempt with model, token counts, carimbo.llm.cost_usd, the prompt version and, on repair calls, carimbo.llm.repair_prompt_version (D-33), never prompt, answer or document content; every eval response carries its trace_id. Jaeger v2 runs locally from compose.yaml.
  • Golden fixtures (D-32). The golden eval responses are generated by the .NET tests and compared against the committed files (just golden).
  • Current prompt. extract-003, pinned by SHA-256 in the tests. Its first live run is in the Phase 3 results below.

Agent slice (Phase 3.1)

Decisions D-34 to D-39 are recorded in docs/DECISIONS.md.

  • Read-only MCP Tools host. A separate, stateless MCP server over Streamable HTTP (Carimbo.Tools) serves a case's extracted invoice and validator findings, and the buyer policy, through read-only tools.
  • Hand-written agent loop. Carimbo.Agent runs the conversation with a step budget (Agent:MaxSteps, 6) and a status guard: a case whose extraction did not succeed is decided by the guard, and a model approve or reject that contradicts the evidence is overridden and recorded.
  • Decision eval contract (decision-1). POST /eval/decisions returns a typed approve, reject or escalate decision with an escalation cause and the evidence it cites, every step with its tool calls, tokens, cost and cache status. The same cache and replay modes as extraction apply.
  • Decision grading. carimbo-evals decisions and grade-decisions report decision accuracy, escalation-cause accuracy, citation validity, guard overrides and typed failures over the eight committed cases in data/decision.
  • Offline end to end. The whole slice, two real processes against a scripted model, runs in just check with no provider key.

The first live run of the eight decision cases and its replay are in the Phase 3.1 results below.

Synthetic dataset (Phase 4)

Decisions D-40 and D-41 in docs/DECISIONS.md record the dataset. data/dataset holds dataset-001, 154 synthetic cases generated from seed 20261022: 90 text-layer invoices (two tax regimes by four length strata, with fixed counts of the hard traits: 24 alphanumeric issuer CNPJs, 18 long-text and 18 confusable-character invoices, 14 with ST or deferral codes) and 64 image-only twins, the same eight invoices at eight renderings (a clean scan, rotation, blur and low resolution at two severities each, and a damaged barcode), so each rendering slice has n = 8. 54 cases are dev and 100 are test; a twin always shares the split of its base invoice. A 20-case, dev-only subset is listed in the manifest as the future CI subset. The directory is 19,562,414 bytes (the cap is 20,000,000), and just dataset-check regenerates all 309 files in about 50 seconds on the author's machine and fails on any byte difference. After the first user acceptance review, the severe blur was lowered from sigma 2.4 to 1.8 pixels so that the small item digits stay readable with effort, and the PDF date of the twins is pinned in code, so byte-identical regeneration does not depend on SOURCE_DATE_EPOCH (D-41). Byte-identical regeneration is promised on Linux x64 only; other platforms use the committed files. Building it called no model and cost US$0.

Comparable measurement (Phase 5)

Decisions D-42 to D-49 in docs/DECISIONS.md record the grader, the tags, the statistics, compare, the live dev run, the wrap hazards and the freeze. All of this runs offline from the stored cases.jsonl of a run; nothing below calls a model.

  • Whole-invoice grading (D-42). carimbo-evals grade scores the 27 header fields, the 14 item columns on every true row and every installment. Items and installments are matched to the expected rows by optimal assignment (scipy.optimize.linear_sum_assignment over a weighted identity score), so row order never changes a grade and a missing or invented row is an event of its own. Numbers are exact by default (tolerance 0.00, configurable with --tolerance); nothing fuzzy ever counts as correct.
  • Primary metric and denominator (D-42). correct_invoice_rate is the share of invoices whose status is success and whose every graded value is right. The denominator is every case except infrastructure_failure and harness_error, which are reported apart and mark the run incomplete; refused, truncated and schema_invalid cases count as not correct, so a model cannot raise its rate by truncating.
  • Failure tags (D-43, frozen by D-48). Each wrong value gets one tag from an ordered list of 19 rules (failure-tags-1), pure code over the graded diff and the row assignment, with an untagged bucket that shows what the rules miss. Tag counts add up to the wrong-value total.
  • Headline rows, intervals and slices (D-44). The primary row is the text-layer cases and the second row the image-only cases, reported separately and never pooled into the headline. Every case-level rate carries a 95% Wilson interval whose n is the number of distinct invoices, because the image-only cases are renderings of a few invoices. Pooled row-level metrics (item recall, precision and column accuracy) carry a seeded bootstrap over invoices (10,000 resamples). The summary also holds 24 declared slices over 11 dimensions, model latency percentiles, cost and cache totals.
  • Silent-error rate (D-42). The share of success cases that carry at least one wrong value, split into those that touch a payment value (including the access key, CNPJ, invoice total, item totals and installment dates and amounts) and the rest.
  • Run comparison (D-45). carimbo-evals compare --first A --second B reads two summary.json files only. It runs one exact McNemar test on the correct-invoice flag over the shared text-layer cases, lists regressed, worsened, improved, status-changed and not comparable cases with the values and tags that flipped, and states that the verdict covers case sampling only, not run-to-run model variation. It exits 0 when it ran and 2 when it refused: a different dataset manifest, grader version or tolerance, a decision summary or a malformed file are refused with no override.
  • Regrading and published runs. carimbo-evals grade --run RUN --cases CASES --out DIR regrades a stored run into a new directory. A directory under evals/results that holds a summary.json is never rewritten, so every published run stays verbatim.
  • Proven offline on all 154 cases (D-46). just e2e drives the documented run, grade and compare commands over all 154 dataset cases against the scripted model, with ground-truth answers and with seeded corruptions, and checks the summary rows, statuses, tags, the silent-error split and compare. That proves the machinery, not the quality of any model: the scripted answers are not model output.

Results so far

The live runs are committed verbatim under evals/results/: each run directory holds the per-case records (cases.jsonl), the run metadata (run.json) and the graded summary (summary.json, summary.md). New runs land in the git-ignored evals/runs/ first. Each case was run once per configuration (n=1), so these numbers show the loop working and where it fails. They are not an accuracy estimate.

Phase 2 live skeleton runs

Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-002, repair prompt repair-001, dataset skeleton-002, 27 graded fields per case, recorded 2026-10-08. Commands: just skeleton (repair budget 2) and just skeleton 1.00 0 (repair disabled). Runs: repair budget 2, repair budget 0. Analysis: 02-10-SUMMARY.md.

Per field, over the three cases:

Field Repair budget 2 Repair budget 0
access_key (44 digits) 2/3 1/3
The other 26 graded fields, each 3/3 3/3
All 27 fields, all cases 80/81 79/81
Schema-valid answers (JSON Schema and Pydantic) 3/3 3/3
Wrong answers flagged validation_failed 1 of 1 2 of 2
Cases repaired 0 0
Model calls 5 3
Cost US$0.0843 US$0.0642

The other 26 fields are number, series, issue_date, operation_nature, the issuer's CNPJ, name, IE and UF, the recipient's tax id, tax id kind, name, IE and UF, the item and installment counts, and the 11 totals boxes.

Per case:

Case What it covers max_repairs 2 max_repairs 0
case-001 One page, Simples Nacional CSOSN 101, numeric CNPJs, 3 items success, 1 attempt, 27/27 validation_failed, 1 attempt, 26/27 (access_key wrong), KEY_CHECK_DIGIT
case-002 One page, Regime Normal with IPI, freight and discount, 2 installments, 4 items validation_failed, 3 attempts, 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_NUMBER_MISMATCH, KEY_SERIES_MISMATCH validation_failed, 1 attempt, 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_NUMBER_MISMATCH, KEY_SERIES_MISMATCH
case-003 Two pages, 50 items, alphanumeric issuer CNPJ, CPF recipient success, 1 attempt, 27/27 success, 1 attempt, 27/27

Totals: cost US$0.0843 against US$0.0642, 5 attempts against 3, 0 cases repaired. Both runs: 0 harness errors, 0 unpriced cases.

What the runs show

  • The only wrong field in either run was the 44-digit access key, and every wrong key was flagged validation_failed by KEY_* rules (also UAT test 5 in 02-UAT.md). No case reported success with a wrong field in these six extractions. That is an observation about six extractions, not a silent-error rate.
  • Repair repaired 0 cases. The three attempts on case-002 gave three different wrong keys. The feedback withholds values for check-digit and key rules by design, so the model has nothing to anchor on except re-reading the image.
  • The case-001 difference between the runs is run-to-run variance, not a repair effect: nothing was repaired in either run, and these models give no sampling control. With n=1 per cell this says only that access-key transcription is unstable on Haiku 4.5.
  • Repair cost US$0.0157 extra on case-002 for no gain. With repair enabled, the PDF cache write made single-attempt cases dearer (+15% on case-001, +7% on case-003, indicative only). Run A read 12994 cache tokens on case-002's repair attempts.
  • case-003 (50 items, output 6093 tokens) took 49586 ms and 49269 ms.
  • The source lists three remedies for the access-key misread: a barcode-decoded key as input or cross-check, a stronger transcription instruction, or a different model. None is built. The barcode design intent is D-05 in docs/DECISIONS.md; the requirement sits under the v2 requirements and is not in the Phase 3 roadmap.

Phase 3 live skeleton runs: a seeding run and its replay

Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-003, repair prompt repair-001, dataset skeleton-002, repair budget 2, recorded 2026-10-15. This is the first live run of extract-003. The seeding run used the default read_write cache mode against an empty .cache/llm (just skeleton), so every answer is a real model call; the replay used just skeleton 1.00 2 read_only. Runs: seeding run, read_only replay. Analysis: 03-10-SUMMARY.md.

Case Status Attempts Fields Tokens in / out Cost (seeding) Latency (seeding)
case-001 validation_failed 3 26/27 (access_key wrong), KEY_CHECK_DIGIT 5352 in, 1836 out (plus 6361 cache-read x2) US$0.02375545 25783 ms
case-002 validation_failed 3 26/27 (access_key wrong), KEY_CHECK_DIGIT, KEY_SERIES_MISMATCH, KEY_NUMBER_MISMATCH 6414 in, 2343 out (plus 6532 cache-read x2) US$0.02760040 31614 ms
case-003 success 1 27/27 998 in, 6093 out (plus 11349 cache-write) US$0.04564925 48521 ms

Run totals: 7 model calls, US$0.09700510 incurred, 12764 input and 10272 output tokens, 0 harness errors, 0 unpriced cases.

The replay of the same three cases made 0 provider calls: 7 of 7 answered attempts were cache hits (hit rate 1.0), http_attempts 0 on every attempt, and the same statuses and field results. Incurred cost was US$0.00000000; notional cost was US$0.09700510, equal to the seeding run's cost. The replay is a replay: the cost of extracting these three cases is the seeding run's US$0.0970, not the replay's zero. In the replay's summary.json, latency_ms is the real wall time of the replayed case (48, 48 and 8 ms) and model_latency_ms keeps the original model latency (25646 ms for case-001).

Trace lookup: each seeding-run trace_id resolved in the local Jaeger with a POST /eval/extractions server span and exactly one chat claude-haiku-4-5 span per attempt (3, 3 and 1), each carrying model, token counts and carimbo.llm.cost_usd, and none containing the case's raw model output.

What the run shows, at n=1 per case:

  • The only wrong field was again the 44-digit access key. Both wrong keys were flagged validation_failed; no case reported success with a wrong field. Case-001 went from success under extract-002 (Phase 2, repair budget 2) to failed here, and the Phase 2 note that access-key transcription is unstable on Haiku 4.5 still stands. Run-to-run variance and the prompt change cannot be separated with one run each.
  • Repair again repaired 0 cases. Case-001's repair attempts gave one more wrong key and then repeated it; case-002's second attempt repeated the first wrong key and the third changed its last digits. The repair attempts on the two failing cases cost US$0.0263 for no gain.
  • The access-key remedies listed above are still unbuilt.

Phase 3.1 live decision run and its replay

Model requested claude-haiku-4-5, prompt agent-001, dataset decision-001 (eight synthetic cases), grader decision-grader-001 (the committed summaries are its verbatim output; the later decision-grader-002 of the review fixes regrades both runs to the same numbers and adds only a citations.uncited count of 0, see D-39), labels decision-labels-1, step budget 6, recorded 2026-10-22. The live run used the default read_write cache mode (just decisions); the replay used just decisions 1.00 read_only. Runs: live run, read_only replay.

Measure Live run
Decision accuracy 7/8 (0.875)
Cause accuracy, over the 4 expected escalations 4/4
Model decision accuracy (before the guard) 7/8
Citations valid 8/8
Guard overrides 0 in 8 checked
Typed failures 0 (8 decided, 0 harness errors)
Steps 16 in total, 2 per case
Tool calls 32 by the model (4 per case), 8 by the guard
Tokens in / out 48304 / 2756
Cost US$0.06208400 incurred, US$0.06208400 notional, 0 unpriced
Latency per case median 5752 ms, min 4548 ms, max 6928 ms

Per case:

Case Expected Decision Correct Citations Cost
dc-001 approve approve yes valid US$0.00666900
dc-002 reject reject yes valid US$0.01165100
dc-003 escalate / policy_warning escalate / policy_warning yes valid US$0.00660700
dc-004 reject approve no valid US$0.00651700
dc-005 escalate / extraction_not_success escalate / extraction_not_success yes valid US$0.00517500
dc-006 escalate / over_limit escalate / over_limit yes valid US$0.01201000
dc-007 approve approve yes valid US$0.00675900
dc-008 escalate / extraction_not_success escalate / extraction_not_success yes valid US$0.00669600

The replay made 0 provider calls: 16 of 16 answered steps were cache hits (hit rate 1.0), with the same decisions and grades. Incurred cost was US$0.00000000; notional cost was US$0.06208400, equal to the live run's. As with extraction, the cost of deciding these eight cases is the live run's US$0.0621, not the replay's zero. The replay's per-case latency is the real wall time of the replayed case (median 21 ms, max 551 ms).

What the run shows, at n=1 per case:

  • One miss, left as it is. dc-004 (the recipient is another company; expected reject, rule 2) was approved. The model cited both relevant values, recipient.tax_id 69979991000100 from get_invoice and buyer_cnpj CARIMBO0000177 from get_policy, and its reason says the recipient tax ID matches policy. The two values differ.
  • Citation validity measures provenance, not correctness. The check passes when every cited evidence source names a tool the agent itself called in that run; the cited ref and value are not compared with the tool results (D-36). dc-004 passes it (8/8) while the comparison over the cited values is wrong. The status guard only looks at the extraction status, so nothing in code caught a recipient mismatch. Which of the following to try is not decided here: a deterministic recipient check, a stronger model, or a prompt change. Prompts, rules and cases were not changed after the run, and the grader was not changed before the numbers above were produced.
  • The guard decided the two failed-extraction cases (dc-005 and dc-008, decided_by guard, escalate with cause extraction_not_success) and recorded 0 overrides: it never had to overturn a model approve or reject.
  • The four escalations were all right in cause (4/4), and the three approve or reject cases other than dc-004 were right. With eight cases and one run each, this is a demonstration of the loop and an honest miss, not an accuracy estimate.

Phase 5 live dev run

Model requested claude-haiku-4-5 (returned claude-haiku-4-5-20251001), prompt extract-003, dataset dataset-001, grader grader-003, failure tags failure-tags-1, repair budget 2, recorded 2026-11-06. The live run used the default read_write cache mode (just dev-split); the replay used just dev-split 2.50 read_only. Runs: live dev run, read_only dev replay. Every figure below is a dev figure, from the live run's summary.json and run.json; none is a published number, and the test split has not been run.

The live dev run is incomplete: the US$2.50 dev cost cap stopped it (stopped_reason cost_cap) with 3 of the 54 selected dev cases not run (nfe-0144, nfe-0145 and nfe-0146), so every dev figure below covers the 51 dev cases that ran. It recorded 125 answered attempts, 0 harness errors and 0 unpriced cases.

Dev row Cases Invoices Excluded Correct Rate 95% interval
dev text 30 30 0 12 0.4000 [0.2459, 0.5768]
dev image 21 3 0 0 0.0000 [0.0000, 0.5615]
dev pooled 51 30 0 12 0.2353 [0.1193, 0.4114]

Dev statuses over the 51 cases: success 16, validation_failed 35, refused 0, truncated 0, schema_invalid 0. Dev silent errors (a success case with a wrong value): 4 of 16 success cases (0.2500, 95% interval [0.1018, 0.4950]), 0 in payment fields and 4 in other fields. Dev incurred cost US$2.30699480 (spent_usd), 431589 input and 277962 output tokens, 4 dev cases reached success after a repair attempt.

The replay made 0 provider calls: 125 of 125 answered attempts were cache hits and its spent_usd was US$0.00000000 (notional US$2.30699480, the live run's cost). It reproduced all 51 dev cases the live run recorded, with the same statuses and invoices.

Limit of the replay (D-46): the replay reproduces every cached answer, while refused and truncated cases are not cached, so they replay as cache_miss, leave the replay's denominator and mark it incomplete, which means its rate can differ from the live run's and the live run is the measure. Refused or truncated dev cases in the live run: 0 (all 125 live attempts ended end_turn and were stored). The replay's own 3 cache_miss dev cases (nfe-0144, nfe-0145 and nfe-0146) have a different cause: the live run never reached them, so no answer was stored. They are the replay's 3 excluded dev cases; its dev correct counts and rates equal the live run's.

What the dev failures look like. All figures here are dev figures from the live run's summary.json and summary.md (grader grader-003, tags failure-tags-1). The 51 dev cases hold 827 wrong values: 4 in success cases (the silent errors above) and 823 in validation_failed cases. Text-layer and image-only figures are reported separately because they are not comparable: the 30 text-layer cases are 30 independent invoices, while the 21 image-only cases are renderings of 3 invoices (nfe-0001, nfe-0048 and nfe-0064, with 8, 8 and 5 twins), so image counts are not independent. The text layer holds 67 of the 827 wrong values (in 18 of its 30 cases) and the image-only cases 760.

  • Tags. The largest are untagged 265 (35 cases), invented_row 172 (21), dropped_row 94 (13), digit_substitution 79 (16), trailing_rows_lost 70 (13) and text_wrong 62 (23); the 19 tags are in D-48. Five tags had no value in this run (schema_invalid, party_swap, merged_rows, split_rows, thousands_misparse), so they are proven only on seeded corruptions. Of the 67 text-layer wrong values, 40 are invented_row on one case: nfe-0038, an 80-row invoice answered with 119 rows. dropped_row, trailing_rows_lost, digit_substitution, tax_code_wrong and date_wrong occur on the image cases only.
  • Silent errors. Three of the four dev silent errors are recipient.name losing its last word SINTETICA, the generator's synthetic marker; the fourth is PA for PAR in the unit of nfe-0042. The marker is printed in the PDF text layer, so these stay counted as wrong (D-42, D-43) and the dev text row above stays 12 of 30. Each of the three is a case whose only wrong value is that name, which is what these three cases contribute to the 18 text-layer cases that are not correct.
  • Alphanumeric CNPJ. Of the 10 text-layer dev cases with an alphanumeric issuer or recipient identifier, 2 are correct, against 10 of the other 20. The samples are 10 and 20 cases, so this is a lead for Phase 6, not a finding.
  • Image cases. None of the 21 image-only cases is correct, and recipient.name is wrong on all 21. In 15 of them the model returned a printed form caption or the homologation notice (10 contain HOMOLOGACAO, 2 are DANFE, 3 are RAZAO SOCIAL); the other 6 are misread names. This is partly specific to dataset-001: every synthetic DANFE is a homologation document, so the name box prints the notice and the recipient's name appears only in the small receipt stub (docs/DANFE-MAPPING.md). A production DANFE prints the name in its box.
  • Why a third of the wrong values stay untagged. 265 of 827 wrong values (32.0%) are untagged. They sit in header-identifier and amount columns (access key, CNPJ, state registrations, item totals, quantities, unit prices, totals boxes) that field accuracy already reports column by column, and 243 of the 265 are on image cases, that is, on renderings of the three dev invoices above; the other 22 are on text cases. The list was frozen with this bucket rather than renamed with per-column catch-all tags (D-48).

Wrap hazards (D-47). The text layer of dataset-001 wraps the unit PAR as PA over R and wraps large prices and totals mid-number, and the grader never forgives either. 12 of the 30 text-layer dev cases carry such a hazard. They were correct on 5 of 12 with a hazard and on 7 of 18 without, and 7 of the 18 text-layer dev cases that are not correct carry one, so correctness is about the same either way. The hazards do not dominate the dev failures. Of the 18 wrong unit values in the run, 1 is on a PAR row (nfe-0042), and 14 of the 15 PAR rows were read correctly; the other 17 wrong units are scan misreads on image cases, which carry no hazard. The 4 confusable-character dev cases (the ones with many-digit prices and totals) were correct on 3, with 2 wrong values in total, an access key and a CNPJ, and none in a price or a total. The wrap is therefore a confound that stays visible and is not what explains this run, and the confusable slice must not be read as measuring row matching alone: its cases also carry the large-number wrap, and 4 dev cases cannot separate the two. The confound that does mark this run is the image-only rows (0 of 21 correct, 760 of 827 wrong values, 3 invoices).

What these numbers cannot show. 30 text-layer dev cases (30 invoices) and 3 image-only invoices, one run, one model (claude-haiku-4-5), 3 of the 54 selected dev cases not run, and no test result. The tag rules were fitted on seeded corruptions and these dev failures, so they say nothing yet about the holdout. None of these figures is a published result.

Before validators

Phase 1 ran the same three cases on the first, smaller target (prompt extract-001, dataset skeleton-001, 9 graded fields per case): 3 of 3 success, but access_key was wrong on case-001 and case-002 (8/9 each) and case-003 was 9/9. Cost US$0.0195. Phase 1 had no check-digit validation, so those two wrong keys were reported as success. This is not like-for-like with Phase 2 (different dataset version, target and prompt). Run: skeleton-20261008T001300Z. Analysis: 01-13-SUMMARY.md.

Spikes

  • docs/spikes/01-llm-gateway.md (2026-10-07): the direct Anthropic SDK behind ILlmGateway was chosen because the IChatClient path is not lossless on usage decomposition or stop details (D-21). thinking: disabled and a non-default temperature return HTTP 400 on Sonnet 5.5. A cache write followed by a read was observed on Haiku 4.5 and Sonnet 5.5. Spend US$0.2551 across four runs of the tool. Its per-case field counts are not a model comparison.
  • docs/spikes/02-schema-probe.md (2026-10-08): the v2 model-facing schema was accepted unchanged (HTTP 200, schema-valid invoice, 0 pattern violations). A two-turn conversation with a cached PDF block read 6321 cache tokens on the repair turn. US$0.0151 for the recorded run, US$0.0526 across three runs (02-03-SUMMARY.md).
  • docs/spikes/03-agent-probe.md (2026-10-22): both claude-haiku-4-5 runs (dc-001 and dc-003) called all four tools before answering and returned a parsed decision through structured output (path structured_output), so the planned agent loop was kept (D-38); the thinking-block echo stayed unverified, because the step-1 answer of the claude-sonnet-5-5 echo check carried no thinking block. Spend US$0.0287 against the US$0.10 cap.

Spend

Phase 1: US$0.2788. Phase 2: US$0.2011 (probe US$0.0526 plus the two skeleton runs). Phase 3: US$0.0970 (the one seeding run; the replay spent nothing). Phase 3.1: US$0.0908 (the agent probe US$0.0287 plus the one live decision run US$0.0621; the replay spent nothing). Phase 4: US$0 (the dataset is generated offline and no model was called). Phase 5: US$2.3070 (the one live dev run: run.json spent_usd 2.30699480 against the US$2.50 cap, which stopped it at 51 of 54 dev cases; the replay's spent_usd is 0.00000000; the offline proofs and the tag freeze called no model). Each is under the US$5 phase cap. Every live run is capped at US$1.00 (US$0.25 for the schema probe, US$2.50 for the dev split).

Verification and UAT

  • Phase 5: verification passed, 4/4 ROADMAP success criteria verified at the fixed code, with EVAL-03 to EVAL-08 accounted for (05-VERIFICATION.md). The code review (05-REVIEW.md) found 1 critical issue, 4 warnings and 8 info findings. The critical issue (a killed run, or one resumed on a narrower selection, could grade as complete), all 4 warnings and 7 of the info findings were fixed (05-REVIEW-FIX.md). The eighth, IN-07, is an observation about the frozen tag list and was skipped by choice (05-REVIEW-DISPOSITION.md). A run with a missing or damaged run.json now grades as incomplete, and carimbo-evals run refuses test cases without --allow-test-split, a flag no recipe passes (D-49). After the fixes, both published dev runs regrade to the same grades, rates and intervals: only graded_at, the two primary-metric pointers, a cache_miss count on the replay and the wording of two summary lines differ. All 30 threats in the register are closed, 2 of them as accepted risks (05-SECURITY.md).
  • Phase 4: verification passed with 4/5 must-have truths verified automatically, re-verified after the gap-closure plans 04-10 to 04-12 (04-VERIFICATION.md). The fifth (severe renderings are hard but still human-readable) is a visual judgement and was settled by UAT. UAT is complete: the first round ran 3 tests, 1 passed (renderer fidelity against a reference DANFE) and 2 issues were found. G-04-1: at sigma 2.4 the severe blur turned small item-table digits into misreads (code D00137 read as D00132, NCM 39191010 as 39198890). G-04-2: the twin PDF bytes depended on SOURCE_DATE_EPOCH, so CI run 40915848246 failed two twin determinism tests on ubuntu-latest. Plan 04-11 lowered the severe blur to sigma 1.8 and plan 04-10 pinned the twin PDF date in code (D-41), and the re-test round passed 2 of 2: read cold at native 220 DPI, every item code, NCM and quantity of the blur-severe twins nfe-0095 and nfe-0127, and the access key of nfe-0127, matched the XML, and CI run 40950655793 was green on ubuntu-latest without SOURCE_DATE_EPOCH (just py-check 2810 tests passed, just dataset-check in 63 s) (04-UAT.md). The code review (04-REVIEW.md) found 0 critical issues, 4 warnings and 8 info findings, all open in 04-REVIEW-DISPOSITION.md. One of them, IN-07, notes that the severe blur level rests on one reader, who transcribed the mild blur just as correctly, so the contrast between mild and severe blur is thin.
  • Phase 3.1: verification passed, 5/5 must-haves verified (03.1-VERIFICATION.md). It records two documented limitations: the dc-004 miss described above, and the live echo of thinking blocks in a tool conversation, which is unverified (D-38) because no live step-1 answer has carried a thinking block yet. The echo is pinned only offline, by the adapter tests and cache v2 entries, and the default claude-haiku-4-5 does not think. The code review (03.1-REVIEW.md) found 0 critical issues, 11 warnings and 18 info findings. All 11 warnings were fixed (03.1-REVIEW-FIX.md), and the 18 info findings stay open in 03.1-REVIEW-DISPOSITION.md.
  • Phase 3: verification passed, 7/7 must-have truths verified, re-verified after the gap-closure plan 03-11 and the UAT (03-VERIFICATION.md). UAT is complete: the first round ran 5 tests, 3 passed and 2 issues were found. G-03-1: in Jaeger, repair-call spans could not be told apart from the initial extraction call. G-03-2: the README's response-cache key list did not match the key. Plan 03-11 closed both, and the re-test round passed 2 of 2 (03-UAT.md).
  • Phase 2: verification passed, 5/5 ROADMAP success criteria, re-verified after the UAT gap-closure plan 02-13 (02-VERIFICATION.md). UAT ran 56 checks: 54 passed and 2 issues were found. G-02-1: the mapping doc pointed recipient.name at a box that prints the homologation notice. G-02-2: prompt extract-002 used three totals labels and a column header that differ from the printed DANFE. Plan 02-13 closed both and introduced prompt extract-003 (02-UAT.md).
  • Phase 1: verification passed with 4/5 criteria verified automatically (01-VERIFICATION.md). The fifth (fresh clone, non-Nix path, both agent runtimes, CI on the first PR) was confirmed by UAT, 3 of 3 passed.
  • Open review warnings WR-02 to WR-06 are tracked in 02-REVIEW-DISPOSITION.md. For example, an export recipient's UF EX is flagged UF_UNKNOWN.

Not measured yet

  • extract-003 has had one live run (n=1 per case). The Phase 2 numbers above used extract-002 and are not like-for-like.
  • The three skeleton cases are one run per configuration on one model: they give no accuracy estimate. The grader, intervals, silent-error rate and failure tags of Phase 5 are measured only on the dev run above.
  • No test split run and no paid full-dataset run (Phase 6). The tag list is frozen (D-48) and the test split has no model answer, so there is no published result.
  • No two-model table and no cost per correct invoice (Phase 6), and no CI gate on regressions (Phase 6).
  • No replicates and no noise floor (MEAS-02), so compare covers case sampling only, not run-to-run model variation; no repair ablation (MEAS-01).
  • The tag rules rest on seeded corruptions and on the 51 of 54 dev cases that ran, with one model and one run. Five of the 19 tags have no real occurrence yet, and the image rows come from 3 invoices.
  • Extraction on the 154-case dataset: only 51 of the 54 dev cases have a live run (the cost cap stopped it), and the 100 test cases have none. The dev figures are not published numbers.
  • Whether repair improves accuracy is unknown: it has repaired zero cases so far.
  • Decisions: eight cases, one run, one model (claude-haiku-4-5). No decision evals at scale, no LLM-as-judge for the reasons, no second model on the agent, and no fix or retest for the dc-004 recipient miss.

Quick start without Nix

You need three tools. Nothing else is installed globally.

  • The .NET SDK that global.json pins (10.0.100, or a later 10.0 feature band).
  • uv 0.8.17 or newer. It provisions Python 3.12 and the locked ruff, pyright and pytest.
  • just. If you do not want to install it, uv tool run --from rust-just just runs it.

Then, from the repo root:

just check

just check runs every offline gate: both stacks, the contract chain (JSON Schema and generated Pydantic models), byte identity of the synthetic dataset, the end-to-end test against a scripted model, the docs check and the secrets check. It needs no API key and no network beyond the package registries.

The pyright package on PyPI is a wrapper around the Node tool and fetches a Node runtime on first use when none is installed. That is expected on a fresh machine.

A minimal Linux image also lacks two system libraries: the ICU library (a .NET runtime prerequisite; the UAT image needed libicu74) and libatomic1 (for the Node runtime that pyright downloads). Desktop distributions and GitHub runners ship both.

CI runs this same path (the same recipes, on a clean Ubuntu runner without Nix) on every pull request.

With Nix

On NixOS or any machine with devenv:

devenv shell
just check

or one-shot: devenv shell -- just dotnet-check py-check. The shell provides the .NET 10 SDK, uv, just, Python 3.12, Node (for pyright), git, curl and the secretspec CLI. It runs no services, and ruff and pyright still come from the uv lock, not from nixpkgs.

Commands

Command What it does
just dotnet-check Build with warnings as errors, verify formatting, run the xUnit v3 tests
just py-check uv sync --locked, ruff check and format check, pyright, pytest
just schema Export schema/*.json from the .NET Domain, then regenerate the Pydantic models
just schema-check Regenerate, then fail if the committed schema or models differ
just datagen Regenerate the seeded synthetic dataset in data/skeleton
just datagen-check Fail unless a regeneration is byte-identical to the committed dataset
just dataset Regenerate the Phase 4 dataset in data/dataset from the quota table and the seed
just dataset-check Fail unless a regeneration is byte-identical to data/dataset (Linux x64)
just decision-cases Regenerate data/decision, the eight decision cases, from the skeleton cases and the specs in dotnet/tools/DecisionCases
just decision-cases-check Fail unless a regeneration is byte-identical to data/decision
just golden Regenerate the golden eval and decision responses from the .NET tests; the compare-mode test fails just dotnet-check when they are stale
just e2e Committed cases over real HTTP to a graded summary, against a scripted model
just docs-check Fail unless decision records D-18 to D-49, the gateway spike recommendation and the schema-probe and agent-probe results exist
just secrets-check Fail when a tracked file holds a provider-key-shaped string
just check All of the above, offline
just skeleton Live: extract the three skeleton cases with Claude and grade them (paid, capped at US$1.00 per run)
just skeleton 1.00 0 Same with repair disabled (Extraction:MaxRepairs 0); the default budget is 2
just skeleton 1.00 2 read_only Replay the last recorded skeleton answers from .cache/llm: zero model calls, US$0 incurred
just decisions Live: decide the eight decision cases with Claude through the agent loop and grade them (paid, capped at US$1.00 per run; starts the Tools host and the Api)
just decisions 1.00 read_only Replay the last recorded decision answers from .cache/llm: zero model calls, US$0 incurred
just dev-split Live: extract the 54 dev cases of data/dataset with Claude and grade them (paid, capped at US$2.50 per run; this recipe only)
just dev-split 2.50 read_only Replay the last recorded dev-split answers from .cache/llm: zero model calls, US$0 incurred
just agent-probe Live: probe tools plus structured output on the agent conversation (paid, capped at US$0.10, rewrites docs/spikes/03-agent-probe.md)
just spike-live Live: re-run the LLM gateway spike (paid, rewrites docs/spikes/01-llm-gateway.md)
just schema-probe Live: probe the v2 schema and a cached two-turn conversation (paid, rewrites docs/spikes/02-schema-probe.md)

The grading and comparison commands behind these recipes are carimbo-evals run, grade (with --out to regrade into a new directory) and compare --first A --second B, run from the repo root through uv run --project python. compare reads two summary.json files, exits 0 when it ran and 2 when it refuses (a different dataset manifest, grader version or tolerance, or a decision summary).

Trace viewer

docker compose up -d

starts Jaeger v2 from compose.yaml (pinned image tag). The UI is at http://localhost:16686 and the OTLP gRPC receiver at 127.0.0.1:4317. Every port is bound to loopback only, because the UI has no authentication. Storage is in memory: docker compose down stops the viewer and drops every trace.

To send spans from a skeleton run, point the exporter at that receiver: OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 just skeleton. The Api exports nothing when the variable is unset. To look a case up, take its trace_id from evals/runs/<run>/cases.jsonl or summary.md and open http://localhost:16686/trace/<trace_id>. Each model call is one chat <model> span carrying the model, the token counts and carimbo.llm.cost_usd, and never the prompt, the answer or the document content. The first call of a case carries carimbo.llm.prompt_version (the extraction prompt) and each repair call also carries carimbo.llm.repair_prompt_version (repair-001), the version that call's attempts[].prompt_version reports.

devenv does not provide Docker. Any Docker engine with compose v2 works, and it is needed for this step only: nothing in just check needs Docker.

Secrets and live calls

The provider key is optional. Without it, just check runs everything against the fake model client, and the live recipes (just skeleton, just decisions, just dev-split, just spike-live, just schema-probe, just agent-probe) exit 2 without calling the model when no key is available.

Without secretspec (any reviewer). Export CARIMBO_ANTHROPIC_API_KEY in the shell that runs just. To keep the value out of your shell history:

read -rs CARIMBO_ANTHROPIC_API_KEY
export CARIMBO_ANTHROPIC_API_KEY

ANTHROPIC_API_KEY works as a fallback. carimbo prefers its own name because Claude Code uses ANTHROPIC_API_KEY for its own authentication.

With secretspec (optional; the author's dev setup). secretspec keeps the key in the OS keyring instead of your shell. The repo's secretspec.toml declares the key as optional.

  1. Run secretspec config global init once per machine and pick a provider such as the OS keyring. The development profile that init selects is already declared in secretspec.toml.
  2. Run secretspec set CARIMBO_ANTHROPIC_API_KEY without --profile and paste the value at the prompt. Never pass the value as an argument.
  3. Run just skeleton, just decisions, just dev-split, just spike-live, just schema-probe or just agent-probe directly. When the variable is not exported they call secretspec run themselves. Or launch the agent session as secretspec run -- claude --continue (or secretspec run -- opencode). A session sees only the variables present when it was launched, so restart it after setting the key.

An agent that calls secretspec itself must pass --reason or set SECRETSPEC_REASON; the recipes already set one. If secretspec is installed but not configured for this repo, exporting the variable bypasses it.

just skeleton costs at most US$1.00 per run (the runner stops at the cap). Its second argument is the repair budget (default 2); just skeleton 1.00 0 runs without repair. Its third argument is the cache mode. In Development the Api caches eval requests by default at .cache/llm (D-27, git-ignored), so repeating just skeleton with the same settings is answered from the cache. just skeleton 1.00 2 read_only replays the last recorded answers with zero model calls (the provider key must still be present), just skeleton 1.00 2 refresh calls the model again and records the new answers, and just skeleton 1.00 2 off bypasses the cache for that run (a per-request override, so it works while the host cache stays on). Read the result in evals/runs/<run>/summary.md; that directory is git-ignored, and a run worth keeping is copied verbatim to evals/results/<run>/.

just decisions costs at most US$1.00 per run (the runner stops at the cap). It starts the Tools host (127.0.0.1:5090) and the Api (127.0.0.1:5080), runs the eight decision cases and grades them. Its second argument is the cache mode, with the same meaning as for just skeleton: just decisions 1.00 read_only replays the last recorded answers with zero model calls (the Tools host still runs, because tool definitions and results are part of the cache key, and the provider key must still be present). A run worth keeping is copied verbatim to evals/results/<run>/.

just dev-split costs at most US$2.50 per run (the runner stops at the cap; no other recipe has that cap, and the phase total stays US$5). It extracts the 54 dev cases of data/dataset and grades them against that dataset. Its second argument is the cache mode, as for just skeleton: just dev-split 2.50 read_only replays the last recorded answers with zero model calls (the provider key must still be present). The test split has no recipe in this phase. Regrade a kept run with carimbo-evals grade --run RUN --cases data/dataset --out DIR; a directory under evals/results is never rewritten.

Never print, log or commit the key. Use a dedicated key with a low spend limit.

Layout

dotnet/            .NET 10 solution (Carimbo.slnx)
  src/             Carimbo.Domain, .Llm, .Extraction, .Validation, .GroundTruth, .Decisions, .Tools, .Agent, .Api
  tests/           xUnit v3 test projects and the scripted host used by the e2e test
  tools/           SchemaExport (writes schema/), DecisionCases (writes data/decision), LlmSpike (live gateway spike)
python/            one uv project: carimbo_datagen, carimbo_evals, carimbo_models (generated)
schema/            canonical and model-facing JSON Schema, exported from the Domain
data/skeleton/     seeded synthetic NF-e cases (XML ground truth + DANFE PDF) and a manifest
data/dataset/      the Phase 4 dataset: 154 synthetic cases (XML ground truth, text-layer and image-only PDFs) and a manifest
data/decision/     generated decision cases, buyer policy, manifest and expected labels
data/vectors/      validator-vectors.json (shared oracle) and the shared valid-invoice.json fixture
data/contract/     golden extraction and decision responses (generated) and the two hand-written request examples
evals/runs/        run output (git-ignored)
evals/results/     committed copies of the live runs the README cites
docs/              DECISIONS.md, DANFE-MAPPING.md and the spike records
.github/workflows/ the CI workflow (the same just recipes)
.planning/         GSD planning artifacts: plan summaries, verification and UAT reports

All data in this repository is synthetic: fictitious companies, generated CNPJs with valid check digits, no real taxpayer data (DECISIONS D-14).

Agent instructions for Claude Code and OpenCode live in AGENTS.md.

About

An AI agent for Brazilian NF-e invoices: extraction, tool-using decisions, durable execution, and a measured eval suite.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages