auto-research

A pure-skill (no install, no plugin, no runtime-specific hooks) autonomous experiment loop for any task with a measurable metric — performance tuning, ML training, optimization, hill-climbing.

You state the goal once as a measurable target; the agent then runs unattended:

form a hypothesis → edit → measure → keep wins / revert losses → repeat

until the budget is spent or no honest gain remains.

There are two common ways to ship Karpathy-style autoresearch loops:

• CLI + lifecycle hooks — a binary enforces honest measurement and git bookkeeping, and hooks block the turn from ending. Strong guarantees, but it installs a package, edits .claude/settings.json, and binds to one runtime.
• Pure SKILL.md — a single document the agent follows. Zero install, portable across agents, but every guarantee becomes a rule the agent must follow rather than something a binary enforces.

This repo takes the pure-skill path and deliberately ports back the one thing CLI tooling is best at — measurement honesty — as hard protocol rules, while keeping the long-run resilience (heartbeat, resume-from-disk, anti-stall, direction diversity) that naive loops lack.

It's just a skill file. Drop it where your agent looks for skills, e.g. for Claude Code:

mkdir -p ~/.claude/skills/autoresearch
cp skills/autoresearch/SKILL.md ~/.claude/skills/autoresearch/SKILL.md

Then ask the agent to run an autoresearch loop on a task that has a measurable metric.

• Numbers must be earned. A metric exists only if it came from a real command's real output. Only METRIC name=value lines are trusted; no line means the run failed, not zero.
• Git is the source of truth. Every experiment is a commit; lost ⇒ git reset --hard HEAD~1. The branch history is the result log.
• Don't celebrate noise. Confidence is gated by MAD (median absolute deviation): keep only when |delta| > 2 × MAD.
• Survive long runs. On-disk state + heartbeat + resume-from-disk make context compaction and session restarts a no-op for the loop.
• Don't loop in place. directions_tried.json enforces direction diversity; stale_count escalates from tactical tuning to structural change.

See skills/autoresearch/SKILL.md for the full protocol.

This is a protocol, not a sandbox. The METRIC-only and git-only rules make cheating visible and effortful — they don't make it impossible. Building a small wrapper that emits METRIC lines from real execution is the single most important thing you can do to make the discipline real.

Inspired by Andrej Karpathy's autoresearch loop concept and the surrounding ecosystem of ports. MIT licensed.