padflow-jev-evals

A small, public benchmark of the decisions PadFlow makes inside software.

PadFlow is a SaaS platform for land development companies: schedules, budgets, draws, lot takedowns, document intake, and QuickBooks Online sync. Most of the "intelligence" it needs is not chat. It is a typed decision with a confidence score that the software acts on: post it, or send it to a human.

This repo publishes those decisions as JSON schemas, a labeled set of anonymized examples, and a script that scores any OpenAI-compatible model on them. It exists so a decision model (TypeSafe Jev, or anything else) can be measured against the workload it would replace.

Each schema has the same shape: an input object, a decision enum, and a confidence in [0, 1]. PadFlow posts automatically above a threshold and queues the rest for a human. The threshold is a product decision; the model's job is to be calibrated.

For each model and each decision:

• Accuracy — the share of examples where decision matches the label.
• Calibration — accuracy of the examples the model marked above the auto-post threshold (default 0.90). This is the number that matters. A model that is 85% accurate but knows which 60% it is sure of is more useful than a model that is 90% accurate and always says 0.95.
• Cost per decision — from the provider's usage report.
• Latency — wall time per call, p50 and p95.

Run scripts/run_baseline.py to fill this table. Every cell below was produced by a run on 2026-09-17; nothing here is estimated.

n = 5 / 5 / 7 rows per decision (route_document / code_transaction / classify_import_value). These are the hand-written seed rows, not labeled production rows: the export from PadFlow on 2026-09-17 found 0 human-labeled rows per decision (see data/README.md for why), so the labeled set does not exist yet and neither does the public slice of 25. At n=5 one row is 20 points of accuracy; read the columns as "the runner works and this is the shape of the task", not as a ranking.

All three ran through OpenRouter with provider.data_collection = deny, so no row was routed to a provider that keeps prompts.

Context on the cost side: PadFlow's September 2026 page-reading trial (vision extraction, a different task from the decisions here) ran $3–$33 per 1,000 pages across four frontier models. The decisions in this repo happen far more often than page reads, so per-decision price is the constraint.

pip install -r requirements.txt
export OPENROUTER_API_KEY=...
python scripts/run_baseline.py --model openai/gpt-5.6-luna
python scripts/run_baseline.py --model anthropic/claude-sonnet-5 --decision code_transaction
python scripts/run_baseline.py --model google/gemini-3.7-flash --data-dir private

run_baseline.py talks to any OpenAI-compatible chat endpoint with JSON schema output. Set --base-url to point it elsewhere. Results land in out/<model>.json and the summary prints as a markdown row you can paste into the table above.

data/*.jsonl is public and anonymized. Company names, people, project names, emails, phone numbers, street addresses, and account numbers are replaced with stable placeholders (Company_A (grading), Person_3, Project_1, 123 Placeholder St) by scripts/anonymize.py. Amounts, dates, line-item descriptions, and document structure are kept, because those are what the decision depends on.

The labeled set is split: 25 rows per decision are published here, the rest stay private, and the Baseline table is measured on the full set. Today the files hold seed rows only -- data/README.md has the counts and the reason. The set will be enough to rank models. It is not enough to certify one.

MIT. Data is released for evaluation use.