AutoR: A Recursive Research System

It proposes, tests, and tries to refute itself. The approval gate is the one thing it does not own — by default, that is you.

Start here: The Framework · English Guide · 中文教程 · Full Documentation

AutoR is not a chat demo, not a generic agent framework, and not a markdown-only research toy.

It is a structured research harness over a coding-agent execution layer: the agent handles execution, the human owns the direction, and every run becomes an inspectable research artifact on disk.

docs/framework.md is the single document that describes what this system is: its implementation, its modules, what is new in it, and what it contributes. This README is the overview and the operating manual.

What AutoR is · Quick start · The stage graph · The rigor dial · Self-improvement rounds · Review · The stage contract · Execution model · Run layout · Architecture · Benchmarks · Documentation · Limits · License

Most autoresearch systems optimize for autonomy. AutoR takes a different position: research is too important to hand over as a blind end-to-end loop. The goal is not to remove humans from research. The goal is to give them a stronger execution system.

AutoR runs a research project as eight stages wired into a directed graph. Six of the forward edges are guarded by artifacts on disk; thirteen backward edges let a late finding send the run back — Stage 07 can reopen the literature survey. Hypotheses are frozen and hashed when Stage 04 is approved, every one must be adjudicated at Stage 06 against a named result file that exists, and every paper claim traced at Stage 07; a supported or refuted verdict resting on a single seed is refused unless the run records why one run settles it. An adversarial reviewer attacks Stage 05's results and Stage 06's analysis, and the stage after each must answer every finding in writing or the gate refuses it. Drafts are scored and a refinement that does not improve is reverted. Every stage still stops at an approval gate, and by default that gate is you.

What a default run (--rigor standard) actually uses. Test, Refute, Iterate and Learn are on: the validity chain is unconditional at every rigor level including fast, --evolve defaults on, and the archive records every run — though it only steers under --archive-steer. Localise runs whenever a reviewer quotes a passage, which requires an agent reviewer. Propose and Deliberate need --rigor thorough; Critique's panel needs --rigor max and its cross-model veto is live on the rcb_agent.py path only. See the rigor dial for the exact mapping.

AutoR does not run itself. Manual approval is the default: approval_mode is manual unless a flag opts out. Seven of the eight moves above can only score, refuse, revert or re-order; none of them can approve a stage. The eighth, the review panel, is an approval gate, and it exists only on the runs where you hand it the gate. Recursion did not change who decides; it changed what reaches the desk. The research unit is unchanged: one reproducible run under runs/<run_id>/, isolated, resumable, with redo and rollback.

Approved stage summaries are the only free-text cross-stage memory. Every other cross-stage edge is a typed artifact with a declared reader: sixteen channels in information_flow.py each name the exact stage slugs that consume them, and the eight produced inside the walk name their producing stage as well. obligations.json and review_policy.json cross stages without touching a summary at all — both only behind an agent approval gate.

Many systems aim to generate research outputs that look ready. So the question is not

Does it look ready?

It is

Can you verify every part of it?

The answer is the validity chain — freeze at Stage 04, adjudicate at Stage 06, trace at Stage 07 (preregistration.py) — and the edge into writing stays shut until every frozen hypothesis carries a verdict (_guard_validity_chain, stage_graph.py).

Every number below comes from a named symbol in the source. Re-derive them; that is the point of naming them.

python -m unittest discover -s tests -p "test_*.py" runs 1826 tests in ~69 s across 82 test modules, with no third-party dependency.

• Python 3.10+
• Claude CLI or Codex CLI on PATH for real runs
• Local TeX tools only for --output-format latex; the default markdown output needs no TeX
• pip install google-genai plus a key in GOOGLE_API_KEY or GEMINI_API_KEY — needed by three paths, not only the diagram one: --web-search gemini, required where the backend's own WebSearch tool is disabled (build_genai_client, src/web_search.py); the cross-model veto --cross-review auto|gemini (src/cross_reviewer.py); and --research-diagram, which also reads configs/diagram_config.yaml
• The SDK is not a default dependency. Without it the diagram step prints Diagram generation failed: No module named 'google' and the run continues; cross-review records itself unavailable rather than agreeing

Every flag, its default, and what is preserved on resume: docs/cli-reference.md. Stage identifiers accept 03, 3 or 03_study_design; --venue defaults to neurips_2025.

Three flags put an agent in the approval seat, not two — and a fourth removes the human without replacing them. approval_mode becomes agent for --approval-mode agent, --full-auto and --review-panel, and create_reviewer is called only when it does. --unattended on its own is the odd one: resolve_unattended returns True for all four, but with approval_mode still manual there is no reviewer to install, so the first approval menu raises UnattendedInputError rather than being decided. For a run with nobody at the terminal, pass --full-auto.

Because --rigor is resolved before resolve_unattended runs, a plain --rigor max sets review_panel = True and silently converts an interactive run into an unattended agent-gated one. Under a badge reading Human approval required, the flag that looks like more review is the flag that removes the reviewer. Three headline mechanisms — obligations, the standing review policy, the cross-model veto — also run only behind that agent gate, as do anchored comments. Manual approval is the default and remains the path for work you intend to publish.

For Codex-backed runs AutoR defaults to --codex-sandbox workspace-write. If a verified remote experiment needs SSH or external GPU access, use --codex-sandbox danger-full-access intentionally: it grants the Codex backend unrestricted local and remote execution, so it should not be the default for untrusted tasks.

# Self-improvement is on by default: navigate the graph, score every draft, keep the
# best, and record the route in ~/.autor/archive.
python main.py --goal "..."
python main.py --archive-report                   # what the archive has learned so far
python main.py --goal "..." --evolve-rounds 4     # spend more on improvement
python main.py --goal "..." --evolve-rounds 0     # measure and ratchet, no extra passes
python main.py --goal "..." --archive-steer       # let the archive pick the topology

# Opt out entirely: the strict 01-through-08 sequence, last draft wins.
python main.py --goal "..." --stage-graph linear --routing off --no-evolve --no-archive

A local web UI over the same Claude-backed runs: create a project, watch stages execute, approve or send feedback, read the compiled paper. It needs the Claude CLI on PATH to start a run.

python studio.py                                # http://127.0.0.1:8000/studio/
python studio.py --host 0.0.0.0 --port 8765     # bind externally, see the warning below
python studio.py --runs-dir /path/to/runs       # override runs directory

The Studio API has no authentication. It binds to 127.0.0.1 by default; anything that can reach it can start runs, approve stages, and read every file under the runs directory. For remote access prefer an SSH tunnel over --host 0.0.0.0. See SECURITY.md.

One honest limit, then the walkthrough: the Studio's lazy-resume approve path picks the next stage arithmetically — the first stage with a higher number (src/backend/studio_runner.py) — and never consults the router, so graph routing and backward moves are a CLI capability today. Page-by-page walkthrough and the full HTTP API: docs/studio.md.

Eight stages are the nodes; a finish node closes the walk. Stage 00 intake is not one of them — it runs before the walk starts, and _graph_entry_stage → _select_stages_for_run (src/manager.py) only ever yield the eight. Solid edges advance, dotted edges go back. --stage-graph linear is the eight advance edges plus the conditional terminal — nine in all — and the guards come off with the backward ones (_advance_edges(guarded=False)): one edge out of each node leaves nothing to choose, so a guard there could only halt a run that the stage's own validation is about to fail anyway.

flowchart LR
    S1[01 Literature] --> S2[02 Hypotheses]
    S2 -->|has_hypotheses| S3[03 Design]
    S3 -->|design_artifacts| S4[04 Implementation]
    S4 -->|runnable_code| S5[05 Experiments]
    S5 -->|results_exist| S6[06 Analysis]
    S6 -->|validity_chain| S7[07 Writing]
    S7 -->|report_exists| S8[08 Dissemination]
    S8 --> Z([finish])
    S6 ==>|round abandoned| Z

    S2 -.->|the gap it rests on is not a gap| S1
    S3 -.->|a hypothesis cannot be brought to a decision| S2
    S4 -.->|not executable as specified| S3
    S5 -.->|implementation is at fault| S4
    S5 -.->|comparison cannot distinguish| S3
    S6 -.->|results insufficient to decide| S5
    S6 -.->|confound the results cannot repair| S3
    S6 -.->|evidence refutes, and points somewhere| S2
    S6 -.->|the numbers are wrong, not disappointing| S4
    S7 -.->|claim has no analysis behind it| S6
    S7 -.->|needs a result never produced| S5
    S7 -.->|the survey missed related work| S1
    S8 -.->|deliverable is not what a reader needs| S7

Six of the eight forward edges carry a guard, one per target stage (_ADVANCE_GUARDS); 01→02 and 08→finish are unguarded. Thirteen dotted edges go back (REVISIT_EDGES) — the longest is 07→01: writing it up showed the finding relates to work the survey missed. One conditional terminal (TERMINAL_EDGES, carried by both topologies) lets an abandoned round finish from Stage 06. The Stage 07 guard is the strictest: every preregistered empirical hypothesis needs a verdict and at least one figure under workspace/figures (_guard_validity_chain).

Who decides the move. AutoR decides which moves are admissible, by evaluating each edge's guard against the artifacts on disk. With --routing auto (the default, DEFAULT_ROUTING_MODE) the agent chooses among them and states a reason; --routing off always takes the graph's default. An off-menu choice — an unlisted target, or one with no stated reason — is refused, written to evolution/routing_refusals.jsonl (src/router.py), and replaced by the forward edge.

Two design calls worth naming. Blocked moves are handed to the agent with the reason they are blocked (StageGraph.moves) — the useful thing to say is not "you may go to 06" but "07 is closed because H2 has no verdict", and an agent that sees why writing is closed routes to the analysis that opens it. And a revisit whose justification repeats one already on the path is refused (repeats_a_previous_reason): going again on the same grounds is a loop, not an iteration.

A backward move is only ever a deliberate choice. The default is always the forward edge, and when a guard has closed it the default advances anyway and lets the stage's own validation — still refusing a Stage 07 that writes up unadjudicated hypotheses — be the gate it always was. A guard is a routing preference; the gate is the gate. So a refusal, a routing failure, or a run nobody is steering all come out as the linear pipeline rather than as a stall.

A stage is a node with a visit budget, not a position in a sequence: DEFAULT_MAX_VISITS = 3 (--graph-max-visits); DEFAULT_MAX_STEPS = 20 bounds the whole walk (--graph-max-steps).

--rigor is the single source of truth for which optional machinery a run uses. The table is generated from _LEVEL_FEATURES in src/rigor.py; an explicit --flag / --no-flag always beats the level, which is why those switches use BooleanOptionalAction with default=None.

Two consequences worth stating out loud:

• Effort tiers are on by default. A default run therefore routes 04_implementation, 05_experimentation and 08_dissemination to a lean prompt and a single reviewer (DEFAULT_TIERS, src/effort.py). Under --rigor max the seated panel does not sit at those three gates unless you also pass --no-effort-tiers.
• --rigor max makes the run unattended, because it implies --review-panel. See the note under Common commands.

The scientific-validity chain is not on this dial. It is unconditional at every level, fast included.

Every valid stage draft is measured against a rigour rubric read off disk — do the paths it names resolve, do the numbers it reports appear in a results file, did it produce artifacts during this execution, is the decision ledger four different things rather than one sentence four times. Eight weighted criteria, RUBRIC_VERSION = "2":

min_stage exists so a criterion that cannot apply is not scored zero: Stage 01 has no experiment manifest to produce, and grading it as if it failed to produce one would make every early stage look worse than every late one — which would make the ratchet prefer late drafts for a reason unconnected to quality.

Measuring is free and always on. The rubric reads the run off disk and never calls a backend, so the property it buys costs nothing: the draft that gets promoted is the best one the run produced, not the last one. That is the half that was missing before — AutoR could iterate, but "later" was the only ordering it had, so a refinement that dropped a resolving reference was promoted on exactly the same terms as one that fixed something.

Improvement rounds are the half that costs, and they are budgeted separately from --max-attempts, which bounds a stage that is failing rather than one being improved. Two per stage by default, and a stage whose rubric has no shortfall worth acting on spends none of them — a round aimed at a criterion already at full marks produces churn, so AutoR does not buy one. --evolve-rounds 0 measures without polishing; --no-evolve restores the old behaviour entirely.

One edge of that budget is worth knowing before you resume a run. state() rehydrates the champion and the Pareto frontier from disk and nothing else (src/evolution.py), so --resume-run restarts rounds_spent and the patience counter at zero: the best draft survives the resume, the spend cap does not, and a stage resumed twice can buy the two rounds twice.

A round that scores worse is reverted, so a stage can only improve. A round that changes a hypothesis verdict is rejected outright, whatever it scored — the rubric is blind to what the run concluded, which removes the incentive, and the verdict_drift check removes the possibility. A revision a human asked for always stands, whatever it measures. The ratchet governs AutoR's own rounds, not the direction it is given.

Full mechanism, and the reasoning behind each refusal, in docs/self-improvement.md.

Every finished run is recorded into ~/.autor/archive — the route it took, the rubric fitness it reached, and the set of stages it actually measured (Archive.record_run, from record_into_archive in main.py). edge_payoffs compares runs that took an edge against runs that were offered it and declined (src/decisions.py), and propose_variant turns a payoff that is believable — enough observations, and a delta above min_gain — into a child variant that moves that one edge one step up or down the preference order. When no payoff is believable, propose_exploration proposes an unexplored edge instead, so an edge nobody has taken is not stranded forever.

A variant is only a reordering. It never opens a guarded edge, never adds one that was not declared, and never removes one: the guards are the correctness argument for letting an agent route at all, and the component that learns from outcomes is precisely the one that must not be able to weaken them. Promotion is as conservative — a challenger has to beat the incumbent within every comparability basis rather than on a pooled mean, because "runs that stopped early" is the cheapest composition for a topology to win on.

The archive records and proposes on every run; it steers only when you ask. The proposed variant is written down and reported, but the topology a run walks comes from the archive only under --archive-steer. Without it, resolve_graph returns the declared topology unchanged.

--trial ID --capability NAME --arm LABEL tags a run as one arm of a paired A/B trial in the archive; --trial-report prints the within-pair rubric difference with an exact sign-flip p-value and the smallest p-value that sample size could possibly reach (src/trials.py, src/inference.py). Below MIN_PAIRS_FOR_SIGNIFICANCE = 6 a trial is labelled underpowered rather than reported as a null.

This is the apparatus for answering "does this mechanism help?", not the answer. No paired trial has been run yet. Read docs/self-improvement.md before quoting anything from it.

Only two of the five are the approval gate. The other three cannot approve anything.

The panel's teeth are mechanical. If any final-round seat carries blocking: true and the chair returned "approve", _enforce_blocking_objections() rewrites the approval into a refinement and records chair_overridden. It is enforced in code precisely because the chair is a model that can be argued out of a prompt-level rule. Blocking is read from the seat's own payload and only counts when that payload's decision token is legible, so a seat whose verdict was unreadable cannot veto.

Every panel run carries its own control arm. The chair's round-1 verdict is one model, one call, no peer input; panel_effect.json accumulates solo-vs-panel across the run and writes a verdict sentence deliberately phrased to be unflattering — "it did not earn that cost; consider dropping the panel" — when that is the truth. The same is true of the ideation panel, anchored comments, crux deliberation and effort tiers; scorecard.py reads all five ledgers at the end of every run and writes workspace/reviews/scorecard.md, keeping "could not be measured" separate from "changed nothing".

Two ledgers run underneath. review_policy.json turns every refusal into a standing rule injected into every later solo review, deduplicated on normalized text so a reviewer restating one complaint cannot manufacture the appearance of learning. obligations.json lets an approving reviewer attach a debt to a later stage — "fine, but you owe me a power analysis at design time" — which only a later reviewer may discharge; deferral is counted and shown, never silent.

Anchored comments make a refusal local. A reviewer quotes the exact passage it objects to (minimum 12 characters, and an unfindable quote is dropped as unanchored rather than sent), the revision is told to leave everything else byte-identical, and the next draft is diffed against the quotes so collateral rewriting is counted rather than assumed away.

Full seat charters, the deliberation protocol and the measurement design: docs/review-panel.md, docs/deliberation.md, docs/stage-comments.md, docs/scorecard.md.

AutoR does not consider a run successful just because it generated a plausible markdown summary.

Required stage summary shape. Seven headings, in this order — REQUIRED_STAGE_HEADINGS:

# Stage X: <name>

## Objective
## What I Did
## Key Results
## Files Produced
## Decision Ledger
## Suggestions for Refinement
## Your Options

Also required, and checked: exactly 3 numbered refinement suggestions, exactly the fixed 6 user options, concrete file paths under Files Produced, and no [In progress], [Pending], [TODO] or [TBD] placeholders.

Artifact gates. Most start by asking whether a file is there — the rows that say "valid", "resolving" or "matching" then parse it.

Requirements are cumulative, and the stage that produces a class of artifact must produce it during that stage's execution — a re-run is not credited with the previous attempt's files. The cutoff is stage_execution_started_at feeding recent_in, and the rubric enforces the same rule independently in _fresh_artifact_kinds, so a Stage 07 draft cannot score on Stage 06's figures.

min_report_figures is a run_config.json field with no CLI flag, clamped to [1, 5]. It is 1 for an ordinary run and 3 for a ResearchClawBench run (BENCHMARK_MIN_REPORT_FIGURES).

Validity gates. The same function — validate_stage_artifacts (src/utils.py) — also runs the validators that ask whether a claim is warranted rather than whether output exists. Seventeen validate_* functions are reachable from it in all:

The code labels the split itself: "the scientific-validity chain, distinct from the artifact gates around it". A run can fail because a claim is unwarranted, not only because a file is absent. The 06→07 router edge is closed on top of that, in the adaptive topology only.

A gate that is honest about what it does not check. validate_preregistration compares the manifest's digest to the recorded source_digest; it does not recompute the digest of the frozen file itself. Editing preregistration.json in place, leaving the manifest alone, passes. The honest claim is that a manifest rewrite is detected, not that the frozen file is tamper-proof.

The complete gate, including every JSON schema that is parsed rather than merely counted, is in docs/stage-contract.md.

Context is composed per consumer, not per availability. A stage's inbound block is built by render_inbound(ChannelContext(...), CHANNELS) from the sixteen typed channels in src/information_flow.py. Each channel declares produced_by, a consumed_by set of real stage slugs, and a rationale; test_every_narrowing_is_argued_for (tests/test_information_flow.py) fails a channel that withholds itself from a stage without saying why. Withholding has to be argued for, not just done.

Three narrowings, because the abstraction is not the point:

• the artifact index skips Stages 00-02 — they produce no data, results or figures, so the index is empty noise there
• the writing manifest reaches Stage 07 alone
• the mutable Stage 02 hypotheses stop at 04_implementation, because the freeze at Stage 04's approval supersedes them. Before that edge was typed, the same H1 went into every prompt from Stage 05 on twice — one copy labelled editable, sitting next to the frozen one at exactly the stages where the freeze is the point.

dependency_edges() returns every (producer, consumer, channel key) triple, so the information topology can be printed and diffed rather than reconstructed from a pile of if statements. _record_inbound_channels writes the delivered channel keys per stage into the run log.

Honest scope: sixteen blocks are typed. Six more — obligations_context, intake_context_text, web_search_context, approved_memory, handoff_context, and the # What the Task Asks For block that build_prompt composes inline from format_deliverables_for_prompt — are still delivered by build_prompt itself rather than declared as channels, so each one's delivery rule lives there instead of next to a consumed_by set. Around them, compose_stage_template (src/prompt_fragments.py) assembles the stage's own instructions, the accepted-extension lists generated from the validators' constants rather than hand-copied, and the run-safety rules.

The assembled prompt is written to runs/<run_id>/prompt_cache/, per-stage session IDs to runs/<run_id>/operator_state/, and the selected CLI backend is invoked in live streaming mode. Alongside the prompt, AutoR installs an agent skill pack from src/skills/ into runs/<run_id>/.claude/skills/ — the operator's working directory — so the agent can pull long-form craft guidance when it needs it. Six skills ship today: citation-discipline, latex-repair, paper-writing, reproducibility-check, result-table, venue-checklist. A skill costs nothing in the prompts that do not use it.

claude --model <model> \
  --permission-mode bypassPermissions \
  --dangerously-skip-permissions \
  --session-id <stage_session_id> \
  -p @runs/<run_id>/prompt_cache/<stage>_attempt_<nn>.prompt.md \
  --output-format stream-json \
  --verbose

Web search where the backend has none. Some deployments disable the agent's built-in WebSearch tool. --web-search gemini starts a stdlib JSON-RPC MCP stdio server (src/mcp_web_search.py) that exposes one tool, mcp__autor-search__web_search, backed by Gemini with grounded search, and passes it to the CLI via --mcp-config. assess_search_readiness() refuses to promise a capability the environment cannot deliver: a hard blocker (no key, no SDK) is reported before the run starts rather than discovered at Stage 01.

Important behaviour:

• refinement attempts reuse the same stage conversation whenever possible
• streamed agent output is shown live in the terminal
• raw stream-json output is captured in logs_raw.jsonl
• if resume fails, AutoR can fall back to a fresh session
• if stage markdown is incomplete, AutoR can repair or normalize it locally before failing the stage
• a backend that is unreachable is classified by src/backend_health.py and surfaces as run.backend_unavailable, so "the model was down" never reads as "the research failed"

Every run lives entirely inside its own directory. The tree is build_run_paths (src/utils.py).

runs/<run_id>/
├── user_input.txt      memory.md             run_config.json
├── run_manifest.json   artifact_index.json   intake_context.json
├── obligations.json    review_policy.json    # both per-run; nothing crosses runs
├── report_plan_stamp.json                    # AutoR's copy, outside workspace/ on purpose
├── logs.txt            logs_raw.jsonl
├── prompt_cache/       operator_state/       handoff/        stages/
├── .claude/skills/     # the skill pack, pulled on demand by the agent
├── evolution/          # champion drafts, improvement_ledger.jsonl, summary.json,
│                       # stage_graph.json, routing_refusals.jsonl
└── workspace/
    ├── literature/  code/  data/  figures/  report/  writing/
    ├── bootstrap/   profile/
    ├── notes/       preregistration.json, hypothesis_manifest.json, experimental_protocol.json,
    │                report_plan.json, research_rounds.json, round_decision.json,
    │                deliberation_request.json
    ├── results/     experiment_manifest.json, hypothesis_outcomes.json
    ├── artifacts/   claim_provenance.json, deliverables_coverage.json, citation_verification.json,
    │                self_review.json, report_review.json | layout_review.json, compiled PDFs
    └── reviews/     validity_review_<stage>.json, validity_response_<stage>.json,
                     comment_ledger.json, deliberations.json, scorecard.md, panel/

evolution/ sits outside workspace/ on purpose, and the dataclass records the reason: it is "a record of how the run reached its answer, not part of the answer, and a benchmark export that swept it up would ship the losing drafts alongside the report". report_plan_stamp.json is outside workspace/ for the same class of reason: the agent must not be able to backdate its own declaration.

The only state AutoR writes outside a run directory is the cross-run archive at ~/.autor/archive (--archive, --no-archive).

Workspace semantics. literature/ reading notes, survey tables, benchmark notes · code/ runnable code, scripts, configs · data/ machine-readable datasets, manifests, processed splits · results/ metrics, predictions, ablations, plus the standardized experiment_manifest.json · report/ the markdown deliverable, report.md and the PNGs it embeds under images/ · writing/ LaTeX sources, sections, tables, bibliography · figures/ plots and paper figures · artifacts/ review JSON, build metadata, compiled PDFs, packaged deliverables · notes/ the frozen files of the validity chain plus supporting notes · reviews/ adversarial validity reviews, panel transcripts, self-measurement ledgers, the run scorecard.

Outside workspace/: memory.md is the approved free-text cross-stage memory; handoff/<slug>.md is the second free-text carrier, each approved summary trimmed to Objective / Key Results / Files Produced and sent only on a continuation attempt or when memory is still empty. Every other cross-stage edge is a typed channel or a JSON artifact. run_manifest.json is the lifecycle state that resume, redo and rollback read; prompt_cache/ holds the exact prompt of every attempt, repair, review, panel seat and crux voice.

Full file-by-file reference: docs/run-artifacts.md.

flowchart LR
    P[rigor.py · effort.py<br/>policy: what machinery runs] --> M
    C[information_flow.py<br/>16 typed channels] --> M
    M[manager.py<br/>walks the stage graph] --> W[walk<br/>stage_graph · router]
    M --> G[gates<br/>utils · preregistration · experimental_protocol<br/>report_plan · deliverables · validity_review]
    M --> I[improvement<br/>rubric · evolution · pareto]
    M --> R[review<br/>approval_agent · review_panel · cross_reviewer<br/>obligations · review_policy · stage_comments]
    M --> S[self-measurement<br/>scorecard · archive · decisions · trials · inference]
    M --> X[execution<br/>operator · operator_codex · web_search · backend_health]

Supporting modules: operator_codex.py and operator_protocol.py, intake.py, manifest.py, artifact_index.py, experiment_manifest.py, evidence_ledger.py, hypothesis_manifest.py, mcp_web_search.py, diagram_gen.py, bootstrap.py and project_bootstrap.py, platform/foundry.py, run_skills.py, terminal_ui.py, prompts/, skills/, and backend/ + frontend/ for the Studio. Runnable tools: tools/score_rcb_run.py (score a benchmark run with the reference judge) and tools/archive_sample_complexity.py (how many runs the archive needs before an edge becomes believable).

The full module map, the stage attempt loop and the extension points are in docs/architecture.md. The design rationale — what is new here and why — is in docs/framework.md.

python rcb_agent.py --workspace <WORKSPACE> runs AutoR against a ResearchClawBench workspace with no human in the loop and exports the benchmark's deliverables (report/report.md, report/images/, code/, outputs/). Scoring is the benchmark's own rubric judge; tools/score_rcb_run.py defaults to the reference judge, gpt-5.1, which is what ResearchClawBench itself scores with.

Judge choice can move a score by more than the gap between the top and the bottom third of the leaderboard. On one identical artifact set Gemini 2.5 Flash scored 37.0 where Opus scored 20.8, a spread of 16.2; on another, Opus scored 52.6 where gpt-5.1 scored 46.0, a spread of 6.6. A number carrying the wrong judge is not a smaller number, it is an incomparable one. Quote the judge with every total.

One attempt per task, Claude Opus executing and reviewing, judged by gpt-5.1. The three comparison agents were re-scored from their public runs under that same judge:

AutoR is last, below the bare Codex CLI it can be configured to run on top of. Eight of the forty runs shipped a 197-byte "incomplete run" stub, and the deficit is almost entirely those. Four caveats travel with the number and none can be dropped: it is single-attempt where the leaderboard aggregates the best score per (task, agent) pair; it is cross-model, since all three comparators run GPT-5.4; it is a gpt-5.1 number; and it is pre-repair — #180 and #181 closed the routes that produced the stubs, and a re-run is in flight.

The framework document's §6 is the full account, including the part that is worse than the mean: the two highest scores came from runs that halted at hypothesis generation, and across 133 stage visits the graph took one backward edge.

Measured on Astronomy_000, reference judge gpt-5.1, before and after the export and figure-budget fixes in #147 / #149 / #153:

This is one task out of forty and must not be extrapolated. ResearchClawBench's published leaderboard numbers are means over all 40 tasks; a single-task score is not comparable to one. For what the reported systems actually score and which of their numbers reproduce, see docs/researchclawbench-landscape.md; for the adapter, its output contract and the export rules, see docs/researchclawbench.md.

The docs/ directory is the reference documentation. This README is the overview; everything below is the detail behind it.

runs/20260330_101222 is the full example run the docs work from. Run directories are gitignored, so what ships in the repository is the artifacts lifted out of it, under assets/.

AGSNv2 reached 36.21 ± 1.08 on Actor, and the run preserved the full human approval trail.

Accuracy Comparison

Ablation + Actor Results

Two-Layer Narrative

Four artifact-backed runs, two pages each: the framing page and an evidence page.

Output 1 A complete end-to-end AutoR run.	Page 1	Evidence Page
Output 2 Do More Experts Help? A parameter-matched MoE-LoRA study.	Page 1	Evidence Page
Output 3 Attention Sink Onset in Tiny Transformers A controlled factorial study.	Page 1	Overview Page
Output 4 HSOD: Harmonic Spectral Operator Decomposition A stability-focused time-series study.	Page 1	Analysis Page

The shot above is the real terminal UI: colored stage panels, parsed backend event streams, display-width-aware wrapping, keyboard-selectable menus, and a Stage 00 clarification flow that asks its questions one at a time.

• 2026-08-11 — Documentation rebuilt against the code: docs/framework.md added, and every count in this README re-derived from a named symbol.
• 2026-08-08 — The report plan (src/report_plan.py) and the task-deliverables contract (src/deliverables.py): a run now commits at Stage 03 to the figures and headline numbers its report will carry, and Stage 07 is refused if the report does not answer what the task statement demanded.
• 2026-08-06 — Recursive self-improvement is the default. The eight stages became a directed graph with a router that must justify its move and is refused off-menu; every valid draft is scored and held to a champion ratchet; the cross-run archive keys every fitness comparison on the stages a run actually measured. Opt out with --stage-graph linear, --routing off, --no-evolve, --no-archive.
• 2026-08-04 — --rigor {fast,standard,thorough,max}: one dial over effort tiers, crux deliberation, the ideation panel and the review panel, ordered by what each costs and what evidence there is for it.
• 2026-06-02 — --codex-sandbox danger-full-access for runs that intentionally need remote GPU or SSH execution. Codex still defaults to workspace-write.
• 2026-04-20 — --full-auto: the manual approval gate can be replaced by a strict reviewer agent.
• 2026-04-19 — AutoR Studio merged into main: a local browser workspace over the same run directories.

Things the mechanisms above do not close. Each is also the next thing worth building, named at the code that would have to change.

• A crashed adversarial pass is indistinguishable from a clean result. _write_review(..., failed=True) records reviewer_failed: true (src/validity_review.py) and no production code reads that flag. Zero findings and a reviewer that never returned are the same input to validate_validity_response: nothing owed, gate open.
• Attribution stops at the log. _record_inbound_channels writes which channels reached each stage, but RunRecord (src/archive.py) has no channel field, so "this edge helped" cannot yet become "this information helped".
• The frozen preregistration is not checked against itself. validate_preregistration compares the manifest's digest to the recorded source_digest and never recomputes the digest of the frozen file. Editing preregistration.json in place passes; deleting the manifest silences the check entirely.
• Standing rules and obligations never reach a panel seat. Both are injected only into the solo reviewer's prompt. Under --review-panel, no seat is shown the accumulated rules and no seat is asked whether an inherited obligation was met — and because neither the seat nor the chair prompt asks for carry_forward, a panel run essentially never creates an obligation either.
• The cross-model veto is unreachable from main.py. --cross-review is declared and parsed there, but resolve_cross_reviewer is called only from rcb_agent.py. The feature is live on the benchmark path only.
• The validity chain is bypassable in unattended mode, and past the skip budget it is bypassed deliberately. A stage that burns its attempts against the gate is auto-skipped, up to --max-auto-skips (default 3). Past that, _route_to_deliverable routes to the writing stage around the validity guard rather than exiting with nothing — the rule and its cost are §2.7. What the run owes instead is disclosure: every stage the route stepped over is named in the report. Whether a banner is enough is a fair thing to push on.
• No mechanism here has evidence that it improves a research output. src/trials.py is the apparatus built to produce that evidence; not one paired trial has been run. The scorecard says when a feature did not change a decision in this run, which is a different and weaker claim.
• Most of the recursion is opt-in or partial. --max-rounds defaults to 1, so a round that asks to go back is recorded with acted_on: false and the run continues to writing anyway; the archive steers only under --archive-steer; and REVIEWED_STAGE_NUMBERS = (5, 6), so nothing attacks Stage 07 or 08.
• Studio does not route. Its lazy-resume approve path picks the next stage by stage number and never consults the router, so graph routing is a CLI capability today.

Intentionally out of scope: generic multi-agent orchestration, database-backed runtime state, concurrent stage execution, heavyweight platform abstractions, dashboard-first productization.

Bug reports, feature requests, documentation fixes, and shared runs are all welcome. Setup is one clone and one command — AutoR's runtime imports nothing outside the standard library and there is no build step. Only the optional Gemini-backed paths (--web-search gemini, --research-diagram, the cross-model reviewer) need google-genai:

git clone https://github.com/tangxiangru/AutoR.git
cd AutoR
python -m unittest discover -s tests -p "test_*.py"    # 1826 tests, ~69s, no dependencies

Read CONTRIBUTING.md before opening a pull request, and docs/development.md before changing code. Security issues go through SECURITY.md, not a public issue. Contributions are assigned to the copyright holder under Section 6 of the LICENSE, and running AutoR requires written permission — see below.

AutoR is proprietary software. It is not open source.

Copyright © 2026 Xiangru Tang. All rights reserved. Licensed under the AutoR Proprietary License 1.0 (LicenseRef-AutoR-Proprietary-1.0) — see LICENSE for the full terms and NOTICE for the summary.

This repository is public so that AutoR's design and behaviour can be examined, cited, and discussed. Publication is not a license. No right to use, run, copy, modify, fork, or redistribute the Software is granted by its availability here.

To request permission, open an issue or contact the copyright holder directly. Permission applies only to the specific use, party, and period stated in writing.