Evaluate
Agent evals from real production traces
Replay a prompt, model, or tool change against the production traces that failed, and see what got better, what got worse, and what it costs before it ships.
Change test
Traces replayed
244
Pass rate
94%+11 pts
Latency
2.9s−21.1%
Tokens per trace
8.2K−21.5%
Better 138
Same 79
Different 14
Worse 9
Inconclusive 4
What is AI agent evaluation?
AI agent evaluation checks whether an agent completes the task it was given, not only whether one reply reads well. That means scoring the whole run: the plan, every tool call, how the agent recovered from errors, and the outcome for the user. Agent evals make that check repeatable: a set of cases, a way to run the agent on each one, and a judge that scores the result. Cases taken from real production traces test what users actually ask for, so evaluation before release pairs with AI agent monitoring after it.
How it works
Turn the traces that failed into evals
Test every change before it ships
Replay a prompt, model, or tool change against the exact traces that failed and compare pass rate, latency, and tokens. Your team reviews the result and decides what ships.
Proposed change
Run comparisonModel
Version
Prompt category
New prompt
You are the support agent for {{company}}.
Answer only from what your tools return.
If a lookup comes back empty, say you could not find it.
Never fill in an order number, date, or amount.
Results
Pass229 of 244 cases
94%
Fail15 of 244 cases
6%
Change vs old (%)
Latency
−21.1%
Token usage
−21.5%
Success score
+10.5%
−30%
−15%
0
+15%
+30%
Compare models on your own traffic
Replay an agent’s recorded calls on other models and see how often each one does worse than the model you run today, next to what it would cost at your volume.
Compare models
Cost estimated at this agent's volume of 412K calls a week.
Check the tasks that already worked
Once a fix passes on the traces that failed, run a regression check: Moda replays the fix on tasks that went well, so a fix that breaks something shows up before it ships, not after.
Regression check
212 of 218 still pass6 regressed
Replays
Replay any change your team makes
Prompt edits
Edit an agent’s prompt and replay it against the traces it handled, as they were and with the new version.
Model swaps
Try one model, or compare several side by side, on the calls your agent actually makes.
Tool description edits
Change how a tool is described to the model and replay the calls where the agent used it.
A verdict on every case
A judge compares each replay with the original and labels it better, same, different, worse, or inconclusive, so you can read the cases that moved.
Real traces, not synthetic tests
Every replay starts from a recorded production trace, including the exact traces behind a problem Moda found.
Your team decides what ships
Engineers make every change. Moda replays it and shows the result, and your team reviews it before anything reaches users.
When the grader is a model
A judge can favor whichever answer it reads first, and a judge that agrees with reviewers 90% of the time can still miss most failures. Read the guide to LLM as a judge for prompt templates, known biases, and how to calibrate a judge against human labels.