A small sphere resting at the edge of a ridge, casting one long shadow.

AI judgment should know how likely it is to be right, and hold back where the evidence runs out.

We build decision models. You state the criterion in plain words and set the accuracy. They answer where the evidence meets it and abstain everywhere else, and every answer a person gives on those cases teaches them your standard.

The calls

AI systems make judgment calls all day.

  • Whether a reply meets the standard.
  • Which queue a ticket belongs in.
  • Whether an agent finished its task.
  • Whether an answer is right.
  • Whether a listing breaks the rules.

Each is a judgment: a criterion, a thing to hold against it, and a verdict. A careful person making these calls knows which ones are beyond them, and says so. That is what makes their verdicts worth acting on.

The three ways today

There are three ways to automate a judgment. Each costs something.

A trained classifier

It is consistent, and its accuracy is measured before it goes live. But every criterion needs its own labeled data, and someone to train it.

A generative model

It takes any criterion in plain words. But its verdict is sampled: written the way it writes any text. How likely it is to be right is yours to measure, with the same labeled data the classifier needed. Ask again, and the verdict can change.

A decision model

It takes any criterion in plain words, and gives the same answer each time. But the answer comes with a probability, and the threshold that makes it safe to act on is yours to find on labeled data, and to find again for each new version.

One knows its accuracy and learns each criterion from scratch. The other two take any criterion, and leave their accuracy to you to measure.

What we build · A8 (Touchstone), available for preview

A8 is a decision model that takes the criterion in plain words, and answers only where it can carry the accuracy you set.

You call it through the OpenAI or Anthropic SDK. The system message is the criterion. The user message is the thing to judge.

It answers at the accuracy you set, and abstains below it.

You set it on each call, as a number from 0 to 99. At 95, at least 95 of every 100 verdicts A8 gives are correct, and a case the evidence can’t carry that far comes back as an abstention. A8 takes the accuracy and finds the threshold that meets it. The same request returns the same verdict.

  1. Does the reply answer the customer’s question?

    “Your parcel left our warehouse this morning.”

    yes

  2. Which queue: billing, shipping or account?

    “I was charged twice for one order.”

    billing

  3. Is the answer free of hedging?

    “It may work, but I could be wrong.”

    no

  4. Does the message ask for a refund?

    “Can I swap this for a larger size?”

    abstainsevidence reaches 88

  5. Does the summary keep every figure in the source?

    “Revenue rose in the third quarter.”

    abstainsevidence reaches 82

  6. Is the review about the product or the delivery?

    “Arrived late, but it works well.”

    abstainsevidence reaches 64

answered 3 of 6 abstained 3Illustration. The rows and their evidence are invented to show the setting at work.

It learns your standard from the cases it holds back on.

A case A8 abstains on goes to a person. Their answer, taught in one sentence to your assistant, becomes evidence, and A8 answers cases like it from then on. Other decision models serve every customer the same weights, so the way to change what they answer is to reword the question.

Another decision modelTeaching A8
Reword the question until it answers rightThe answer a person gave, taught from any MCP host
Re-run your labeled set and pick a new thresholdA fit in the background, a minute after your workspace goes quiet
The same model every customer getsYour workspace’s own model, served for you

Each teach is kept the moment you approve it and trains your workspace’s model, while A8 keeps answering. The accuracy it promises is earned again on what you taught. When your standard changes, teach the new answer the same way. Your examples train your model and no other.

Automate the calls A8 answers. Send the ones it abstains on to a person.

How we know

We test the approach against trained human raters, in the open.

In qed-bench, our open benchmark, we scored essays with the approach A8 is built on, and compared the ranking with the one trained human raters gave. A panel of eight generative models judged the same essays.

Our approach
0.815
The strongest generative judge on the panel
0.813

Rank correlation with trained human raters, essay quality. At 1, the two rankings match exactly. The notebooks are public. read qed-bench

When a judgment is automated, its accuracy should be a number you set before it answers.

oversight.study

Tell us where AI judgment falls short in your work.

We are asking the people who build, run and sign off on automated systems how they check them and decide when to trust them.

The AI Oversight Study is open until December 31, 2026. About five minutes, with questions for your role.

take the survey