Idea

karpathy/autoresearch demonstrates a simple but powerful pattern: give an AI agent a training setup, a single metric, and let it experiment autonomously overnight. The agent modifies code, runs a 5-minute experiment, evaluates the metric, keeps or discards changes, and repeats (~12 experiments/hour).

Can we apply this pattern to harness improvement?

Instead of the agent optimizing a training loop, it optimizes tau itself — specifically the system prompt, tool descriptions, and tool schemas — measured against our eval suite.

How it could work

The loop

1. Agent reads the current system prompt / tool descriptions
2. Agent makes a targeted change (e.g., reword a guideline, adjust a tool description, change output format)
3. Run the eval suite (flask-books + future evals) with 2-3 models (nano + haiku for speed/cost)
4. Compare scores against baseline
5. Keep change if scores improve, discard if they regress
6. Repeat

What the agent can modify

• coding-agent/src/system_prompt.rs — the system prompt text and guidelines
• Tool description() and parameters() return values — how tools present themselves to the model
• Tool output formatting — how results are framed back to the model (error messages, truncation text, etc.)

What stays fixed

• The eval prompts and scorecards (those are the ground truth)
• The core tool implementations (bash execution, file I/O)
• The agent loop logic

The metric

Composite eval score: sum of (pass/total) across all evals × all test models. Single number, higher is better.

Feasibility concerns

• Cost: Each eval run costs ~$0.01-0.07 depending on model. At 12 experiments/hour with 2 models × 2 evals, that's ~$0.50-3.00/hour. Overnight (8h) = $4-24. Manageable.
• Noise: Eval scores are stochastic. May need 2-3 runs per experiment to get reliable signal. Increases cost 2-3x.
• Search space: System prompt changes are high-dimensional. The agent needs good hypotheses, not random perturbation.
• Eval speed: flask-books takes 20-45s per run. With 2 models × 2 evals × 3 runs = 12 runs per experiment = 4-9 minutes. Matches autoresearch's 5-min budget nicely.

Implementation sketch

1. Create a program.md (autoresearch-style instructions) that explains:
   • The tau architecture and what's modifiable
   • The eval suite and how to run it
   • The composite metric and how to compute it
   • Constraints (don't break tests, don't change eval prompts)
2. Create a run_experiment.sh that:
   • Builds tau
   • Runs eval suite across selected models
   • Computes composite score
   • Outputs score in a standard format
3. Point an agent (Claude, Codex) at program.md and let it go

Prerequisite work

• More evals needed first — optimizing against a single eval (flask-books) will overfit
• System prompt needs to be in place (Add dynamic system prompt based on pi-mono's approach #1) so there's something meaningful to optimize
• Eval runner needs to be scriptable and fast

Inspiration

• karpathy/autoresearch — the pattern
• The broader question: can an agent meaningfully improve its own harness? If yes, this is a flywheel.