PatchPilot Lite

PatchPilot Lite is a lightweight evaluation harness for AI coding agents. It focuses on SWE-bench-style reliability: test-gated success, pass@k evaluation, tool-call reliability, and explicit failure taxonomy.

See EVIDENCE.md for a benchmark snapshot and example runs.

Many agent failures are evaluation failures:

• Patches apply but don’t fix the bug.
• Tests are missing or never run.
• Tool calls fail silently or inconsistently.

PatchPilot Lite makes these failure modes visible with deterministic, test-gated evaluation and structured metrics.

• Pass@K metrics with attempt tracking (default MAX_ATTEMPTS = 2)
• Tool-call reliability and per-call logging
• Failure taxonomy for consistent root-cause analysis
• Test-gated success (no pass without green tests)
• Minimal footprint (Python stdlib only)

1. Load a task from tasks/<task_id>/.
2. Reset the task repo to a clean baseline.
3. Apply the patch (interactive or patch.diff).
4. Run the task’s test command.
5. Record attempts, tool calls, timing, and failure reason.
6. Write per-task JSON and aggregate summary.csv.

Pass@K

• pass@1: tests pass on attempt 1
• pass@2: tests pass within 2 attempts
• attempts_used: number of attempts executed

Tool Reliability

• tool_calls_attempted, tool_calls_successful, tool_success_rate
• tool_call_log with timing and errors

Failure Taxonomy

• patch_failed
• patch_applied_tests_still_fail
• tests_not_executed
• tool_call_failed
• dependency_or_env_failure
• timeout_or_loop_prevented
• wrong_file_selection
• insufficient_context
• unrelated_code_changes
• tool_schema_error
• unknown_failure

The aggregate row in summary.csv includes a failure distribution string.

python run.py --list
python run.py --all --non-interactive

Results are written to results/.

Demo Video:
Watch here: https://vimeo.com/1158223846?share=copy&fl=sv&fe=ci

• Test-gated success
• Deterministic evaluation
• Simple, isolated tasks
• Reproducibility over complexity
• No external dependencies

patchpilotlite/
├── run.py          # CLI entrypoint
├── agent.py        # Agent loop (placeholder for AI)
├── tools.py        # Tool implementations and patch engines
├── utils.py        # JSON/CSV helpers, timing, taxonomy
├── tasks/          # Task definitions
│   └── <task_id>/
│       ├── task.md
│       ├── repo/
│       ├── test_cmd.txt
│       └── patch.diff
└── results/        # Generated outputs
    ├── <task_id>.json
    └── summary.csv

• Python 3.10+
• Standard library only

python run.py --task <task_id>
python run.py --all
python run.py --all --non-interactive
python run.py --list

• Real LLM agent integration
• Task expansion beyond toy examples
• Richer telemetry (latency distributions, tool failure rates)
• CI integration for regression tracking

MIT