Evidra Bench

Evidra Bench is a live regression benchmark for infrastructure agents. It puts a model or agent into disposable Kubernetes incidents, lets it use real tools, and catches failed or unsafe behavior before that configuration gets production access.

Most AI agent benchmarks stop at a score, transcript, or vendor claim. Bench is built for the harder questions platform teams ask before an agent touches production:

• Did it diagnose the incident before changing infrastructure?
• Did it fix the root cause, or only make the final check green?
• Did it preserve safety constraints, ownership boundaries, and workload contracts?
• Did it loop, give up, burn tokens, or claim success too early?
• Did a new model, prompt, MCP server, tool server, or skill regress behavior?

The public report site is https://bench.evidra.cc. It hosts exam suites, leaderboards, and inspectable benchmark reports produced by this harness.

Docker is the only prerequisite. The runner creates a disposable kind cluster, runs your model against three live incidents, verifies the outcome, writes local reports, and removes the cluster:

# Linux: run as your own uid so the reports under evidra-results/ are
# yours, not root's (the group that owns /var/run/docker.sock is added so
# the runner can still drive kind). On macOS with Docker Desktop the
# --user/--group-add lines can be omitted.
docker run --rm \
  --user "$(id -u):$(id -g)" \
  --group-add "$(stat -c %g /var/run/docker.sock)" \
  -e HOME=/workspace/evidra-results \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/evidra-results:/workspace/evidra-results" \
  -e OPENAI_API_KEY \
  ghcr.io/vitas/evidra-bench:latest \
  test --model openai/gpt-5

This runs locally without an Evidra account, login, central database, or result upload. Your provider charges still apply to the model you select.

Or bring your own agent binary (--agent receives kubectl on PATH and INFRA_BENCH_SCENARIO per case):

docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/evidra-results:/workspace/evidra-results" \
  -v "$PWD/my-agent.sh:/fixtures/agent.sh:ro" \
  ghcr.io/vitas/evidra-bench:latest \
  test --agent /fixtures/agent.sh

You get a PASS / FAIL / UNSAFE verdict per case, a standalone evidra-results/report.html, result.json, and signed evidence bundles for every run under evidra-results/bundles/ (verify one with evidra verify --dir <bundle> from the Evidra CLI). The first run pulls a ~1 GB Kubernetes node image, cached afterwards.

Every verdict is graded on the evidence captured during that run (ADR 0001): windowed API audit, normalized state snapshots, and a confined sandbox. When the evidence is lost, unattributable (a delegated exec whose effects left the API stream), or the evaluator itself faulted, the case reads INCOMPLETE — never a clean PASS or FAIL — and the result's gap list names what was missing. Result documents carry a semantics_version stamp so pre-ADR bundles can be told apart; nothing refuses to join them.

Mounting /var/run/docker.sock grants the runner host-level control through the Docker daemon. Use this image only with agents and inputs you trust. Linux users who opt into --user should add the socket's numeric group ID with --group-add "$(stat -c '%g' /var/run/docker.sock)" and pre-create a writable results directory.

Local models are not a demo-only sidequest: evidra test accepts Ollama models like any other provider, with the same suites, verdicts, reports, and exit codes. The runner discovers what you already have installed and validates tool calling before it creates any cluster, so a missing or incompatible model fails in seconds instead of provisioning infrastructure first. Nothing is ever downloaded automatically and paid providers are never used as a silent fallback. Start Ollama as usual (ollama serve, ollama pull qwen3:8b), then let the runner share the host network so it can reach the local runtime:

docker run --rm --network host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/evidra-results:/workspace/evidra-results" \
  ghcr.io/vitas/evidra-bench:latest \
  test --model ollama/qwen3:8b --ci

The absolute fastest start is the same evaluation through the demo entry point, which picks your only compatible installed model automatically (and asks once when several qualify):

docker run --rm -it --network host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/evidra-results:/workspace/evidra-results" \
  ghcr.io/vitas/evidra-bench:latest \
  demo

Ollama is used only to discover installed models and read their metadata; inference goes through the same OpenAI-compatible client as every other model. Custom OpenAI-compatible endpoints work through test --endpoint on any network. Host networking exposes the runner to host-published ports, so use it with trusted inputs (see the socket warning above).

Bench creates a healthy infrastructure fixture, injects a known failure, runs the selected agent adapter, collects artifacts, and verifies the final state.

provision workspace
  -> bootstrap healthy baseline
  -> inject failure
  -> execute agent through adapter
  -> collect transcript, tool calls, timeline, and cost
  -> verify infrastructure outcome
  -> classify behavior findings
  -> publish local or hosted report data

Scenario checks are declarative. The agent can fix the problem any way it wants as long as the final infrastructure state satisfies the checks and the action path stays within the scenario safety expectations.

Bench groups scenarios into external-facing suites:

These are exam-aligned benchmark suites, not official CNCF, Linux Foundation, HashiCorp, or AWS certifications. See Results And Reports for suite URLs, scoring labels, reproducibility rules, and report structure.

Separately from the exams, evidra test --suite kubernetes-core@1 runs a twelve-case safety-engine regression pack: every case pairs an authority profile with an assert-v2 verifier and is admitted only when scripted safe/noop/unsafe controls produce the exact PASS / FAIL / UNSAFE matrix on live kind and k3d clusters. See Testing Methodology.

Prerequisites: Go 1.25.14+ (matching go.mod), kubectl, kind or k3d, and helm.

# Build the CLI
make build

# List scenarios
bin/bench-cli scenario list

# Validate a scenario without starting a cluster
bin/bench-cli run --scenario kubernetes/broken-deployment --dry-run

# Run one live scenario
bin/bench-cli run \
  --scenario kubernetes/broken-deployment \
  --provider bifrost \
  --model gemini-2.5-flash \
  --reuse-cluster

If you use an OpenAI-compatible provider route, point the bifrost provider at it:

export INFRA_BENCH_BIFROST_URL=http://localhost:9090/v1
export INFRA_BENCH_BIFROST_AUTH_BEARER="$PROVIDER_API_KEY"

See Quickstart for the first-run path and Bench Service Setup for the local API and runner control plane.

Bench treats MCP servers as external tool backends. Keep the model, provider, scenario set, timeout, memory window, and cluster settings fixed. Change only the server command and labels.

# Native-tools baseline
bin/bench-cli bench \
  --scenario kubernetes \
  --provider bifrost \
  --model sonnet \
  --reuse-cluster

# Same scenario slice through a selected MCP server
bin/bench-cli bench \
  --scenario kubernetes \
  --provider bifrost \
  --model sonnet \
  --mcp-server "$MCP_SERVER" \
  --tool-server-id "$TOOL_SERVER_ID" \
  --tool-server-version "$TOOL_SERVER_VERSION" \
  --reuse-cluster

For paired baseline-versus-candidate reports, use bench-cli report-pack. See Tool Server Integration and Private Report Pack.

The catalog covers operational domains and difficulty levels:

See Scenario Catalog for the current inventory and Scenario Authoring Guide for writing new scenarios.

make test                 # Go unit tests
make test-race            # Go tests with race detector
make fmt                  # gofmt
make lint                 # golangci-lint
make vuln                 # govulncheck
make smoke                # dry-run all scenarios
make public-smoke         # live public API smoke, requires BENCH_API_URL
make private-review-smoke # live private review write smoke
make ui-dev               # Vite dev server for local UI
make ui-build             # production UI build

See Testing Guide for local test coverage, CI gates, and smoke-test workflows.

Apache License 2.0