another decision model
Rewording
- Reword the question until it answers right
- Re-run your labeled set
- Pick a new threshold
available for preview
Any criterion, in plain words. A verdict that clears the accuracy you set, or an abstention.
the stone
Theophrastus described the test around 300 BC. An assayer drew the gold across a black stone and left a streak. Beside it he drew needles of known fineness, one streak each, and read the gold’s color against theirs.
The needle it matched gave the grade. An unmatched streak was set aside.
the same five parts
the grade
Every call carries the accuracy its verdict must meet, from 0 to 99. At 95, at least 95 answered verdicts in a hundred are correct. A case whose evidence falls short of 95 comes back as an abstention.
A8 finds the threshold for the grade itself. A labeled set, if you have one, tests the promise.
The verdict is scored from the evidence, so the same request returns the same verdict.
a call
The system message is the criterion: what you are judging.
The user message is the subject: the thing under judgment.
The schema is the verdict’s shape. A8 answers in JSON that matches it.
The model’s tag sets the grade: a8:95 is 95, and 90 is the default. At 95, at least 95 answered verdicts in a hundred are correct. Below it, A8 abstains.
Here A8 abstains, so a person decides. Tell your assistant their answer, and over MCP it teaches A8, once you approve.
from openai import OpenAI
client = OpenAI(
base_url="https://api.u22a8.ai/v1",
api_key="<your key>",
)
criterion = "Which queue does the ticket belong in?" # 1
subject = "It arrived crushed. I want a refund." # 2
messages = [
{"role": "system", "content": criterion},
{"role": "user", "content": subject},
]
queues = {"enum": ["billing", "shipping", "account"]}
route = {"type": "json_schema", "json_schema": {"name": "route",
"schema": {"properties": {"queue": queues}}}} # 3
resp = client.chat.completions.create(
model="a8:95", # 4: answer only at 95 or above
messages=messages, response_format=route,
)
resp.choices[0].message.refusal
# 'Too close to call at min_accuracy 95; answered at 80.'
# 5: a person decides it's billing. Tell your assistant, over MCP:
# "That ticket belongs in billing. Teach A8."the needles
another decision model
A8
The cases A8 abstains on are the ones to teach. A teach is kept the moment you approve it. About a minute after your workspace goes quiet, it trains a new version of your workspace’s model in the background, and the next verdicts come from it. Until then, the current version answers. The accuracy it promises is earned again on what you taught.
Ask why it answered, and it cites the examples you taught that decided it. Your examples stay in your workspace, and they train only your model.
the method, assayed
On essay quality, we compared the scoring approach A8 is built on with trained human raters and with a panel of eight LLM judges. Its rank correlation with the raters was 0.815. The strongest judge on the panel reached 0.813.
read qed-benchState the criterion and the grade it must clear. Answer the cases A8 holds back on, and your workspace’s model learns from them. A8 is available for preview.
To build A8 into your stack or a tool, write to .