Coder Eval

Open-source eval framework

Evaluate & benchmark AI coding agents and Claude Code skills.

Coder Eval runs a real agent — Claude Code, Codex, or Gemini — in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. Built for CLI and skill builders who want reproducible, data-driven answers.

uv tool install coder-eval
Get started Read the docs

Apache-2.0 · Python 3.13+ · runs Claude Code, Codex & Gemini

$ coder-eval run tasks/hello_date.yaml tasks/token_check.yamlRun directory: runs/2026-07-21_20-21-48 Running 2 task(s)Mode: Sequential 20:21:50 [INFO] [default/hello_date_smoke_test/00] orchestrator: API routing: anthropic_direct20:22:04 [INFO] [default/hello_date_smoke_test/00] orchestrator: Success criteria: 3/3 passed, weighted score: 1.00020:22:05 [INFO] [default/hello_date_smoke_test/00] orchestrator: Task finished: status=SUCCESS duration=16.4s score=1.000Tasks:  50%|██████████░░░░░░░░░░| 1/2 [00:16<00:16, 16.43s/task]20:22:24 [INFO] [default/token-check/00] orchestrator: Success criteria: 1/1 passed, weighted score: 1.00020:22:25 [INFO] [default/token-check/00] orchestrator: Task finished: status=SUCCESS duration=20.0s score=1.000Tasks: 100%|████████████████████| 2/2 [00:37<00:00, 18.95s/task] Run complete: runs/2026-07-21_20-21-48Results: 2/2 succeededView report: coder-eval report runs/2026-07-21_20-21-48
task_id: "hello_date_smoke_test"description: "Create a Python script that prints 'Hello, Claude!' then the current date."initial_prompt: "Create a Python file named app.py in the current working directory that prints 'Hello, Claude!' on one line, and today's date in YYYY-MM-DD format on the next line. Use the datetime module. Then run the script with: python app.py"tags: [smoke, smoke-pass, golden, basic, pure-python] # Soft target on cumulative SDK inner-loop turns. Surfaces a one-shot# warning + report badge when exceeded; does NOT abort the task or# affect scoring.run_limits:  expected_turns: 5 agent:  type: "claude-code"  permission_mode: "acceptEdits"  # Safer default  allowed_tools: ["Read", "Write", "Bash"] success_criteria:  - type: "file_exists"    path: "app.py"    description: "The file app.py must be created."  - type: "file_contains"    path: "app.py"    includes: ["Hello, Claude!", "datetime"]    description: "The script must contain the required string and import."  - type: "run_command"    command: "python app.py"    timeout: 10    description: "The script must execute successfully."
$ coder-eval report runs/2026-07-21_20-21-48Reading report from runs/2026-07-21_20-21-48/experiment.md               Experiment Report: default Description: Default experiment - provides baseline agent configurationVariants: default Total Duration: 36.5s Aggregate Metrics  Metric            default ────────────────────────────────── Tasks Run         2 Succeeded         2 Failed            0 Errors            0 Success Rate      100.0% Score             1.000 ± 0.000 Avg Duration (s)  18.2 ± 2.5 Assistant Turns   5.5 ± 2.1 Tokens            115,239 ± 33,631 Win Rates  • default: 2/2 tasks (100%) Per-Task Comparison  Task                   default    Best     Spread ───────────────────────────────────────────────── hello_date_smoke_test  1.000 (+)  default  0.000 token-check            1.000 (+)  default  0.000

Declarative YAML tasks

One file per task with pinned dependencies, weighted 0.0–1.0 criteria, and pass thresholds — versioned next to your code.

Sandboxed & reproducible

Isolated runs of Claude Code, Codex, or Gemini with full telemetry: every tool call, token count, and cost.

Skill activation checks

Verify a Claude Code skill actually triggers with skill_triggered, and score SkillsBench-style skill suites.

A/B experiments & CI gates

Agent vs. agent, model vs. model, prompt vs. prompt — side by side. Fail the build when quality regresses.

The problem with “it seems better”

Coding agents are moving fast. Models change, prompts drift, tools get added, skills get rewritten. Every one of those changes can quietly make an agent better — or quietly break it. Almost nobody can answer a simple question with a straight face: is our agent actually better than it was last week?

Fixed benchmarks

SWE-bench and SkillsBench are static datasets. Great for a leaderboard, but you can’t author tasks for your codebase, your skills, or your internal SDK.

LLM-output harnesses

They grade a model’s text against a reference string. But a coding agent isn’t a string generator — it takes turns, calls tools, edits files, and runs commands.

Hand-rolled scripts

They get you started, then rot. No reproducible sandboxes, no weighted scoring, no cost telemetry, nothing to gate CI on.

Coder Eval is a framework for the middle ground: author your own tasks in declarative YAML, run a real agent inside a sandbox with real tool use and multi-turn dialog, and score the actual files and commands it produced. Scoring is continuous, 0.0 to 1.0, with per-criterion weights, fractional credit, and pass thresholds — a run isn’t just green or red, it’s a number you can track, compare, and regress against.

See it in action

A sandboxed agent takes the hello_date task from YAML to a scored run, then the result is browsed in evalboard.

Anatomy of an eval task

One declarative YAML file: the prompt, the agent, and weighted criteria scored against what the agent actually produced.

date_teller_skill.yaml 
task_id: "date-teller-skill"description: "The date-teller skill must trigger — and be right."initial_prompt: "What day of the week was 2026-01-01? Write it to answer.md." agent:  type: "claude-code"  # or "codex" / "gemini" — same task, any agent success_criteria:  - type: "skill_triggered"  # did the agent really engage the skill?    skill_name: date-teller    expected_skill: date-teller    weight: 3.0    description: "The skill fires — not just a lucky answer."   - type: "file_contains"    path: "answer.md"    includes: ["Thursday"]    weight: 1.0    description: "The written answer is correct."

Many ways to say “correct”

Real tasks need more than “does the file exist.” Coder Eval ships criterion types spanning the full range from deterministic checks to model-graded judgment.

Criterion What it checks Scoring
file_exists A file was created at a path Binary
file_contains A file includes required strings Fractional
file_check Existence, includes, excludes and regex in one check Fractional
json_check A JSON document matches an expected shape Fractional
file_matches_regex File contents match a pattern Binary
run_command A command exits successfully Binary
command_executed The agent actually ran a given command or tool Fractional
commands_efficiency The agent stayed inside a tool-call budget Continuous
reference_comparison Code similarity against a reference solution Continuous
classification_match A written label matches ground truth Binary
llm_judge An LLM grades the result against a rubric Continuous
agent_judge An agent inspects the result and scores it Continuous
skill_triggered A target skill was actually engaged Binary
uipath_eval A UiPath agent clears its eval-set thresholds Fractional

Anatomy of a run

One command fans the suite out across isolated sandboxes, in parallel, with every turn, token, and score recorded.

zsh — coder-eval run 
$ coder-eval run tasks/skills/*.yaml --driver docker -j 4Run directory: runs/2026-07-22_09-14-02 Running 12 task(s)Max parallel tasks: 4 09:14:03 Building docker image coder-eval-agent:0.8.7 from docker/Dockerfile09:14:05 [default/date-teller/00] API routing: anthropic_direct09:14:06 [default/invoice-extract/00] API routing: anthropic_direct09:14:31 [default/date-teller/00] Success criteria: 2/2 passed, weighted score: 1.00009:14:32 [default/date-teller/00] Task finished: status=SUCCESS duration=27.1s score=1.000Tasks:  33%|██████░░░░░░░░░░░░░░| 4/12 [01:12<02:24, 18.05s/task]09:15:02 [default/q3-report/00] Success criteria: 1/3 passed, weighted score: 0.33309:15:02 [default/q3-report/00] Task finished: status=FAILURE duration=41.9s score=0.333Tasks: 100%|████████████████████| 12/12 [03:41<00:00, 18.4s/task] Run complete: runs/2026-07-22_09-14-02Results: 11/12 succeededView report: coder-eval report runs/2026-07-22_09-14-02

A/B experiments, with statistics that hold up

The point isn’t one score in isolation — it’s the difference between two. The experiment layer runs the same tasks across variants side by side: Claude vs. Codex, model vs. model, tool-on vs. tool-off, prompt vs. prompt.

model-comparison.yaml 
experiment_id: model-comparisondescription: "Compare Sonnet 4.6 vs Opus 4.6 on coding tasks" defaults:  repeats: 5  # run every (task, variant) pair five times  agent:    type: claude-code    permission_mode: bypassPermissions variants:  - variant_id: sonnet    agent:      model: claude-sonnet-4-6  - variant_id: opus    agent:      model: claude-opus-4-6 # coder-eval run tasks/*.yaml -e experiments/model-comparison.yaml

Reports built into every run

Each run writes its results three ways: browsable HTML, diffable markdown, and CI-native JUnit.

task.html
Self-contained HTML report for a task: SUCCESS badge, score 1.00, agent and model chips, success-criteria table with per-criterion scores, and the start of the conversation trace

Full visibility in evalboard

The runs directory is the database — evalboard is a local dashboard that reads it directly. Success-rate trends across runs, pass rate and cost per run, and per-task drilldowns: success-criteria cards, tool timelines, and message-by-message token and latency breakdowns.

evalboard — localhost:3030
Evalboard task page: score, duration, cost, turns and token cards, tags, the task prompt, three passed success criteria, and the start of the message timeline

evalboard on GitHub →

An eval framework for your tasks — not another leaderboard

Fixed benchmarks like SWE-bench and SkillsBench rank models on shared datasets. Coder Eval evaluates the tasks, skills, and workflows you ship: write a declarative YAML eval task, run Claude Code, Codex, or Gemini in a sandbox, and score the files and commands the agent actually produced with weighted 0.0–1.0 criteria.

Use it to test whether a Claude Code skill actually triggers (skill_triggered), A/B-test agents, models, and prompts side by side, and gate CI on coding-agent quality — a skill that quietly stops triggering fails a build, not a user.

How it compares →

Quick start

uv tool install coder-eval
coder-eval run tasks/hello_date.yaml
coder-eval report runs/latest

Full quick start →

Questions or bugs? Open an issue · or email coder-eval@uipath.com

Already using coder-eval? Tell us →

Frequently asked questions

How is Coder Eval different from SWE-bench and other benchmarks?

SWE-bench, HumanEval, and SkillsBench are fixed datasets for ranking models. Coder Eval is a framework: you write declarative YAML eval tasks for your own repos, skills, and workflows, run a real agent against them in a sandbox, and score the files and commands it actually produced. It measures what leaderboards can't see — your tasks.

Which coding agents does it support?

Claude Code, Codex, and Gemini run out of the box, each in an isolated sandbox with full telemetry: every tool call, turn, token count, and cost. Other agents plug in through the framework's plugin SPI.

Can I gate CI on coding-agent quality?

Yes. A run exits non-zero when weighted criteria or suite thresholds fail, so a plain CI step fails the build when quality regresses — for example when a task's score drops or a Claude Code skill stops triggering.

How do I test that a Claude Code skill actually triggers?

Add a skill_triggered criterion to a task. It scans the run's telemetry for a real skill invocation and scores pass or fail; across a dataset of prompts it aggregates into precision and recall thresholds you can gate on.

How do I benchmark Claude Code against Codex or Gemini?

Define your tasks once, then run an A/B experiment with one variant per agent. Every variant runs the same tasks in the same sandboxes, and the experiment report puts the scores side by side — win rates, cost, duration, and tokens per agent — so the comparison is on your workload, not a public leaderboard.

Can I evaluate AI coding agents on my own codebase?

Yes — that's the point. Task templates seed each sandbox with files from your repo, the agent works on that copy in isolation, and criteria score the result. Runs execute on your machine or your CI, so private code stays in your infrastructure.

How do I know a prompt or model change actually improved my agent?

Run the experiment with repeats: each variant executes the same tasks N times, and reports aggregate the replicates with bootstrap confidence intervals plus a paired mean-difference test for two-variant experiments — a statistically grounded answer instead of a vibe check on one lucky run.