AI judgment should know how likely it is to be right, and hold back where the evidence runs out.
We build decision models. You state the criterion in plain words and set the accuracy. They answer where the evidence meets it and abstain everywhere else, and every answer a person gives on those cases teaches them your standard.
The calls
AI systems make judgment calls all day.
- Whether a reply meets the standard.
- Which queue a ticket belongs in.
- Whether an agent finished its task.
- Whether an answer is right.
- Whether a listing breaks the rules.
Each is a judgment: a criterion, a thing to hold against it, and a verdict. A careful person making these calls knows which ones are beyond them, and says so. That is what makes their verdicts worth acting on.
The three ways today
There are three ways to automate a judgment. Each costs something.
A trained classifier
It is consistent, and its accuracy is measured before it goes live. But every criterion needs its own labeled data, and someone to train it.
A generative model
It takes any criterion in plain words. But its verdict is sampled: written the way it writes any text. How likely it is to be right is yours to measure, with the same labeled data the classifier needed. Ask again, and the verdict can change.
A decision model
It takes any criterion in plain words, and gives the same answer each time. But the answer comes with a probability, and the threshold that makes it safe to act on is yours to find on labeled data, and to find again for each new version.
One knows its accuracy and learns each criterion from scratch. The other two take any criterion, and leave their accuracy to you to measure.
What we build · A8 (Touchstone), available for preview
A8 is a decision model that takes the criterion in plain words, and answers only where it can carry the accuracy you set.
You call it through the OpenAI or Anthropic SDK. The system message is the criterion. The user message is the thing to judge.
It answers at the accuracy you set, and abstains below it.
You set it on each call, as a number from 0 to 99. At 95, at least 95 of every 100 verdicts A8 gives are correct, and a case the evidence can’t carry that far comes back as an abstention. A8 takes the accuracy and finds the threshold that meets it. The same request returns the same verdict.
It learns your standard from the cases it holds back on.
A case A8 abstains on goes to a person. Their answer, taught in one sentence to your assistant, becomes evidence, and A8 answers cases like it from then on. Other decision models serve every customer the same weights, so the way to change what they answer is to reword the question.
| Another decision model | Teaching A8 |
|---|---|
| Reword the question until it answers right | The answer a person gave, taught from any MCP host |
| Re-run your labeled set and pick a new threshold | A fit in the background, a minute after your workspace goes quiet |
| The same model every customer gets | Your workspace’s own model, served for you |
Each teach is kept the moment you approve it and trains your workspace’s model, while A8 keeps answering. The accuracy it promises is earned again on what you taught. When your standard changes, teach the new answer the same way. Your examples train your model and no other.
Automate the calls A8 answers. Send the ones it abstains on to a person.
How we know
We test the approach against trained human raters, in the open.
In qed-bench, our open benchmark, we scored essays with the approach A8 is built on, and compared the ranking with the one trained human raters gave. A panel of eight generative models judged the same essays.
- Our approach
- 0.815
- The strongest generative judge on the panel
- 0.813
Rank correlation with trained human raters, essay quality. At 1, the two rankings match exactly. The notebooks are public. read qed-bench
When a judgment is automated, its accuracy should be a number you set before it answers.
Tell us where AI judgment falls short in your work.
We are asking the people who build, run and sign off on automated systems how they check them and decide when to trust them.
The AI Oversight Study is open until December 31, 2026. About five minutes, with questions for your role.
take the survey