This directory contains manual evaluation runbooks for each Docker skill. Each runbook documents representative prompts, expected agent behaviors, and verification commands.

1. Phase 1 (current): Manual runbooks. A human evaluator feeds the prompts to an agent with the skill loaded, then walks through the behavioral checklist and runs the verification commands.
2. Phase 2: Automate against one agent target (likely Claude Code) before broadening.
3. Phase 3: Track pass/fail per skill per agent across all four targets (Claude, Codex, Gemini CLI, GitHub Copilot).

task eval runs the static assertions in eval-checks.yaml; it does not execute these prompts, start sandboxes, or prove agent behavior. Asset checks and verification-snippet safety checks complement, but do not replace, the manual evaluations below. Sandbox integration procedures live in each skill's checks/verification.md and require disposable resources. Record the CLI version, agent/model, and observed results when running them.

One runbook per catalogued skill. The table is generated from catalog.yaml by task catalog.

1. Load the skill into the target agent (see install instructions).
2. Open a fresh session with no prior conversation context.
3. Feed each prompt from the runbook to the agent.
4. Walk through every checkbox in the Expected behaviors and Must not sections.
5. Run each Verification command and confirm the expected outcome.
6. Record pass/fail per prompt per agent.