superpowers-bench

You give your agent a set of skills. You describe a task. Does it figure out which skills to use - or does it need hand-holding?

superpowers-bench measures how well coding agents (Claude Code, Codex, OpenCode) discover and invoke skills from obra/superpowers based on context alone. It tests two conditions: no hints (just the task prompt) and with hints (task prompt + a natural-language hint that implies the right workflow without naming it).

• Skill selection as the unit of measure - not code quality, not pass/fail on tests. Did the agent reach for the right tools?
• Apples-to-apples comparison - same 20 tasks, same skills, same grading. Just swap the agent.
• Baseline vs triggered - quantifies how much a well-written hint improves skill discovery without explicit instruction.

$ npm install
$ npm run fetch-skills            # download skill definitions
$ npm run bench -- run --condition claude --task explain_code
# ... agent runs, result appended to results/results.jsonl

$ npm run bench -- matrix --parallel 4   # run full benchmark
$ npm run bench -- report                # generate report

From source (only method - this is a benchmarking tool, not a published package):

git clone https://github.com/kunchenguid/superpowers-bench.git
cd superpowers-bench
npm install
npm run fetch-skills

Prerequisites: Node.js 16+, and at least one supported agent CLI on your PATH with valid API credentials: claude (Claude Code CLI), codex (OpenAI Codex CLI), or opencode (OpenCode CLI).

┌──────────────────────────────────────────────────┐
│ CLI: run / matrix / report                       │
└──────────────────┬───────────────────────────────┘
                   ▼
┌──────────────────────────────────────────────────┐
│ Runner                                           │
│  1. Clone target repo into isolated workspace    │
│  2. Copy skill definitions into workspace        │
│  3. Build prompt (+ trigger hint if triggered)   │
│  4. Execute agent (multi-turn, auto-respond x3)  │
│  5. Detect which skills were invoked             │
│  6. Grade: precision / recall / F1 vs expected   │
│  7. Append result to results.jsonl               │
└──────────────────────────────────────────────────┘
                   ▼
┌──────────────────────────────────────────────────┐
│ Reporter                                         │
│  Load results.jsonl ► markdown tables            │
│  Summary by condition, per-task, per-skill       │
└──────────────────────────────────────────────────┘

• Isolated workspaces - each parallel worker gets its own git copy of the target repo to prevent interference
• Detection method - Claude: parsed from Skill tool_use events. Codex/OpenCode: fingerprint matching against distinctive phrases from skill bodies
• Grading - set comparison between detected and expected skills. Pass = no missing, no extra

run and matrix start from a fresh results/ directory. report always reflects the latest benchmark batch, not an append-only history.

All config lives in config/:

• conditions.yaml - defines agent variants (agent, model, triggered flag)
• tasks.yaml - 20 tasks with prompts, expected skills, and trigger hints
• fingerprints.yaml - distinctive phrases for Codex/OpenCode skill detection

npm install          # install dependencies
npm run fetch-skills # download skill definitions
npm run bench        # run CLI (see subcommands above)