agent-eval

CI evaluation harness for multi-agent development environments.

Modern enterprises are drifting into multi-agent development where different tools serve different cognitive jobs: Claude for ideation, Cursor for precision, Copilot for completions. The result is tool coexistence, not tool choice.

In this environment, agent instruction files (CLAUDE.md, Cursor rules) become shared dependencies. A single line change can silently degrade productivity across repositories.

Today these files are maintained through informal trust networks and anecdotal testing. This cannot scale.

agent-eval provides the safety net: behavioral regression tests that detect negative drift, support safe contribution from a broad set of engineers, and make multi-agent workflows predictable in high-stakes enterprise settings.

The goal is not to perfectly grade agent quality. The goal is to reliably detect "worse" so teams can iterate with confidence.

# Install
uv venv && source .venv/bin/activate
uv sync

# Verify harness is working
uv run python -m harness self-test

# Discover available tasks and configs
uv run python -m harness ls --path examples/getting-started/

# Validate without executing (dry-run)
uv run python -m harness run \
  -t examples/getting-started/tasks/fix-bug.task.yaml \
  -c examples/getting-started/configs/baseline/config.yaml \
  --dry-run

# Run a single evaluation
uv run python -m harness run \
  --task evals/tasks/coding/fix-auth-bypass.task.yaml \
  --config evals/configs/full/config.yaml

# Run full matrix (tasks x configs x runs)
uv run python -m harness matrix \
  --tasks "evals/tasks/**/*.task.yaml" \
  --configs "evals/configs/*/config.yaml" \
  --runs 3

# Quick test with --limit (runs only N samples)
uv run python -m harness matrix \
  --tasks "evals/tasks/**/*.task.yaml" \
  --configs "evals/configs/*/config.yaml" \
  --limit 5

# Compare results for regression detection
uv run python -m harness regression \
  --baseline results/baseline.json \
  --current results/current.json

# Generate scaffold for testing a new skill
uv run python -m harness scaffold --name my-skill-test

# Run in isolated Docker container
uv run python -m harness build-image  # first time only
uv run python -m harness run -t task.yaml -c config.yaml --container

Tasks (what to test)     Configs (how to configure)     Graders (how to score)
        │                         │                            │
        └─────────────────────────┼────────────────────────────┘
                                  │
                                  ▼
                          ┌─────────────┐
                          │  EvalRunner │
                          └─────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          ▼                       ▼                       ▼
    ┌──────────┐           ┌──────────┐            ┌──────────┐
    │ Isolator │           │ Executor │            │ Reporter │
    │ +Archive │           │ +Docker  │            │          │
    └──────────┘           └──────────┘            └──────────┘

• Tasks: YAML files defining prompts, assertions, and scoring weights
• Configs: Environment variants (baseline, skills-only, claude-md-only, full)
• Graders: Code-based (tests pass, file contains) + LLM-based (rubric evaluation)
• Isolator: Fresh temp directory per run with injected configuration + artifact archiving
• Executor: Pluggable backend (Claude CLI, Docker containers)
• Reporter: Results aggregation and regression comparison
• Scaffold: Template generator for skill A/B testing

See docs/ARCHITECTURE.md for detailed technical documentation.

Four presets for A/B testing different Claude Code setups:

This project implements Layer 3 of a larger vision:

1. Layer 1 (future): Canonical specification format for agent instructions
2. Layer 2 (future): Tool adapters (Claude Code, Cursor, Copilot)
3. Layer 3 (this project): CI evaluation harness for regression detection

See CONTRIBUTING.md for development setup and guidelines.

MIT