Org-wide MCP tool-calling evaluation harness. Register any MCP server via a YAML config, and the harness will connect to it, discover its tools, run queries through various LLMs, and score tool-calling accuracy using DeepEval metrics.
- DeepEval — scoring engine (ToolCorrectnessMetric, ArgumentCorrectnessMetric, MCPUseMetric)
- FastMCP — MCP client for tool discovery and server connectivity
- Multi-model providers — Anthropic (Vertex AI + direct API), OpenAI, Ollama
- YAML eval registry — each eval gets a config file with connection details, test cases, and eval settings
mcp-toolbench/
├── evals/
│ ├── aap.yaml # AAP MCP server config + test cases
│ ├── aap-skills.yaml # SEP-2640 skills: construction + use
│ └── dummy.yaml # dummy in-process server for local testing
├── harness/
│ ├── config.py # YAML config loader with ${ENV_VAR} interpolation
│ ├── providers.py # multi-model provider abstraction + auto-detection
│ ├── runner.py # DeepEval scoring engine + result formatting
│ ├── mcp_client.py # MCP tool discovery
│ ├── skills_client.py # skill discovery + citation resolution
│ ├── skills_eval.py # skills construction/use checks
│ ├── lifecycle.py # server clone/setup/start/stop lifecycle
│ └── generate_test_cases.py # standalone test case generator
├── server/
│ └── tools_server.py # dummy in-process FastMCP server (6 tools)
├── results/ # JSON/Markdown results per run
├── main.py # CLI entrypoint
└── requirements.txt
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtThe harness auto-detects available providers based on environment variables:
| Provider | Required env vars |
|---|---|
| Anthropic (Vertex AI) | ANTHROPIC_VERTEX_PROJECT_ID + CLOUD_ML_REGION |
| Anthropic (direct) | ANTHROPIC_API_KEY |
| OpenAI | OPENAI_API_KEY |
| Ollama | Ollama running locally (no env vars needed) |
If no cloud credentials are found, it defaults to local Ollama.
# List registered evals and detected providers
python main.py list
# Run an eval (uses models from YAML config)
python main.py run aap
# Override model
python main.py run aap --model anthropic:claude-sonnet-4-6 --repeats 1
# Run all registered evals
python main.py run-all
# CI mode — exits non-zero if below threshold
python main.py run aap --ci --threshold 0.8
# Output as markdown
python main.py run aap --output md
# Save results as markdown file
python main.py run aap --save-format md
# Generate test cases from discovered tools
python main.py generate aap --model anthropic:claude-sonnet-4-6name: my-eval
# Optional: repo cloning + server lifecycle
repo: https://github.com/org/my-mcp-server
setup: npm install
start: npm start
# Connection
transport: http # http | in_process | stdio
endpoint: http://localhost:3000/mcp
headers:
Authorization: "Bearer ${MY_TOKEN}" # env var interpolation
# Eval settings
models:
- anthropic:claude-sonnet-4-6
- openai:gpt-4o
coverage: all # all | <number> — tools to cover when generating tests
threshold: 0.7
repeats: 3
metrics:
- tool_correctness
# Test cases
test_cases:
- id: TC01-easy
query: "List my available job templates."
expected_tool: job_templates_list
category: easyThe coverage field controls how many tools python main.py generate creates test cases for:
coverage: all— every discovered toolcoverage: 20— 20 tools- omitted — defaults to all
--countCLI flag overrides the YAML value
Each tool gets an easy (keyword-matching) and hard (paraphrased) query.
Test cases live in two places:
evals/<name>.yaml— hand-curated static cases, committed to git. These are your ground truth.evals/<name>.generated.yaml— LLM-generated cases, created bypython main.py generate <name>. Gitignored.
At runtime, both are merged. Static cases take priority — if a generated case has the same ID as a static one, the generated case is skipped.
Use --output md to print results as markdown, or --save-format md to save a .md file. Markdown reports include a Recommendations section that analyzes failure patterns:
- List vs retrieve tool confusion
- Frequently confused tool pairs
- Queries where no tool was selected
- Easy vs hard accuracy breakdown
- Consistently failing tools
http— remote MCP server over streamable HTTP, with optional auth headersin_process— a FastMCP server object importable in this codebasestdio— a local MCP server started via command + args
For each test case, the model receives the full tool list from the MCP server and a natural-language query. A run passes if the model's chosen tool(s) include the expected tool. DeepEval metrics provide additional scoring (tool correctness, argument correctness, MCP use).
Results are saved to results/ as JSON or Markdown.
A server that serves SEP-2640 skills is asked two questions, in two passes over one connection.
Construction is static — no model, no tokens, seconds to run. Per skill it
checks that the skill is named in the server's instructions, that its
frontmatter name matches its URI, that allowed-tools and
mcpToolDependencies agree, that every tool it declares is actually served,
that it declares no write tools, and that every file it cites resolves to a
readable resource URI.
python main.py run aap-skills --construction-onlyUse is agentic. The model gets the server's instructions as its system
prompt and a read_resource tool, and nothing else — the same two things a
conforming client would give it. Each case names the skill the query should
pull in, and the checks are protocol events rather than judgements about prose:
did it read the skill, did it read the files the skill cites, were the tools it
called ones the skill declares, did it avoid writes, did it finish.
python main.py run aap-skillsThe read_resource tool is the harness standing in for a client — no
mainstream client implements SEP-2640 natively yet. Results say whether a
correct client would reach the skills, not what any shipping product does.
Citation resolution follows the rule the server states in its own
instructions: take the skill's URI and replace the final SKILL.md segment
with the path the skill gives. A citation that climbs above the skill root has
no URI under the skill prefix, and is reported as a construction failure with
the address that would have worked.
- Create
evals/<name>.yamlwith connection details and hand-written test cases - Run
python main.py generate <name>to auto-generate additional cases into<name>.generated.yaml - Run
python main.py run <name>— static + generated cases are merged automatically