Idea
Adopt τ²-bench (Sierra) as our first external eval harness. It's the public benchmark whose shape lines up most cleanly with drua: a tool-using agent talks to a simulated user, calls real APIs against a stateful backend, and is graded by the resulting state diff (not LLM-as-judge alone).
Why it fits drua specifically
drua is fundamentally an MCP gateway routing an agent through real toolsets behind a policy prompt. τ²-bench's runtime shape is the same: tool catalog + policy + stateful backend + scripted user goal + deterministic grader. Most other public benchmarks miss our value prop:
- SWE-bench / Terminal-Bench → pure code editing, doesn't exercise the gateway
- WebArena / OSWorld → wrong surface (browser / desktop)
- GAIA / AgentBench → mostly synthetic, not tool-orchestration shaped
τ²-bench gives us:
- Pass^k reliability metric (not just best-case) — runs the same task k times, only passes if all k succeed
- Deterministic state-diff grading — LLM-judge is only used for the user simulator, not the grade
- Dual-control domain (telecom) — user also has tools, agent has to instruct rather than execute. Maps well to "ops engineer + drua co-debugging"
Proposed plan
- Baseline run — run airline + retail domains as-is through drua's gateway. Gives a public-comparable number and shakes out harness wiring. (~1-2 days)
- Custom drua domain — fork the task format with one of our toolsets. Best candidate:
concourse (we already have benchmarks/concourse/follow-rollout as a seed task). Write ~20 scripted user goals like "find out why the deploy job failed and retry it." (~1 week)
- Compare to internal benches — we already have
benchmarks/notification-domain (code_assistant ablation) and benchmarks/concourse. τ²-bench gives us the missing piece: a reusable, pass^k-graded harness instead of one-off scripts.
Out of scope for now
- Voice / full-duplex evaluation (τ²-bench supports it; not relevant to us yet)
- Multi-toolset tasks — start with one toolset per domain to keep grading deterministic
Links
Idea
Adopt τ²-bench (Sierra) as our first external eval harness. It's the public benchmark whose shape lines up most cleanly with drua: a tool-using agent talks to a simulated user, calls real APIs against a stateful backend, and is graded by the resulting state diff (not LLM-as-judge alone).
Why it fits drua specifically
drua is fundamentally an MCP gateway routing an agent through real toolsets behind a policy prompt. τ²-bench's runtime shape is the same: tool catalog + policy + stateful backend + scripted user goal + deterministic grader. Most other public benchmarks miss our value prop:
τ²-bench gives us:
Proposed plan
concourse(we already havebenchmarks/concourse/follow-rolloutas a seed task). Write ~20 scripted user goals like "find out why the deploy job failed and retry it." (~1 week)benchmarks/notification-domain(code_assistant ablation) andbenchmarks/concourse. τ²-bench gives us the missing piece: a reusable, pass^k-graded harness instead of one-off scripts.Out of scope for now
Links