DEV Community: GWA

Two Claude Codes Lost to Claude Code + Codex

GWA — Wed, 07 Oct 2026 17:30:33 +0000

If you spend any time around coding agents in 2026, you've absorbed the advice: use multiple agents. One plans, one implements, one reviews. The implication runs one direction. Two agents beat one, and if two are good, three should be better.

Almost nobody has tested that claim with an oracle on the other end. It sounds true, because code review works for human teams. But "more reviewers help" and "reviewers who fail differently help" are two different claims, and the second can be true while the first is false.

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models (Meta, August 2026) is nominally a paper about evolving a recommender model. The part I kept rereading is the benchmark buried inside it. ExecML turns the defects logged during that deployment into 96 oracle-graded tasks on each of two codebases, and then asks a question the multi-agent industry has mostly skipped: does composing two complete coding-agent products buy executable correctness, beyond spending the same budget on one?

The answer is yes. With a catch that matters more than the yes. On 96 tasks over the HSTU codebase, Claude Code → Codex → Claude Code hit 62.5% execution accuracy where Claude Code → Claude Code → Claude Code hit 45.8%, at the same budget. Meanwhile, doubling Claude Code netted roughly zero over a blind best-of-N baseline. More review and different review aren't the same thing.

This is the companion to my last post on harness engineering. That paper described how coding-agent harnesses are built. This one measures what happens when you use one harness to check another.

The Research Process

ExecML is a benchmark, not a leaderboard entry. The design:

Two codebases. The public HSTU recommender implementation (Python/PyTorch) and LitGPT, a non-recommender training framework. HSTU carries the primary claim; LitGPT is a pre-specified transfer test.
96 private tasks per repository, seeded by real incidents logged during a twelve-iteration deployment. The tasks balance six incident families: data provenance and leakage, tensor routing, gradient flow, train/eval mode, metric semantics, and configuration wiring. Half are new implementations, half are repairs of deliberately introduced faults.
A hidden executable oracle per task. A patch passes only if it simultaneously satisfies the regression suite, task-specific behavioral checks, scientific-safety invariants (leakage, dead gradients, wrong evaluation semantics), and evaluator-integrity checks. The primary endpoint is all-or-nothing execution accuracy (EA). The secondary endpoint is the silent critical-defect rate (CDR): oracle-confirmed faults that leave the patch runnable but capable of invalidating a scientific conclusion.
Six conditions, budget-matched. Every flow gets the same three roles, per-node limits, tool permissions, action caps, and wall-clock and dollar caps from a frozen resource envelope. Timeouts and overruns count as failures and stay in the denominators.
Products pinned. Claude Code ran Claude Opus 4.8; Codex ran GPT-5.6; both at maximum reasoning effort, in fresh sandboxes, randomized order, identical prompts and tool policies.

One caveat belongs at the top, not the bottom: "matched budget" means matched observable inference budget. Proprietary products don't expose provider-side compute, so this isn't equality of FLOPs. The authors are explicit about it, and I'll come back to it.

Key Findings

1. Heterogeneous composition wins at parity of budget

Here is the spine of the paper, three roles per condition, all measured on the same envelope:

Condition	HSTU execution accuracy ↑	HSTU silent defects ↓	LitGPT execution accuracy ↑	Cost/task
Claude Code, one pass (unmatched cheap reference)	22.9%	35.4%	20.8%	$0.41
Claude Code, full budget	33.3%	27.1%	29.2%	$0.98
Claude Code best-of-N, oracle-blind selection	43.8%	18.8%	39.6%	$0.95
Claude Code → Claude Code → Claude Code	45.8%	16.7%	43.8%	$0.97
Claude Code → Codex → Claude Code	62.5%	10.4%	56.2%	$1.02
Codex → Claude Code → Codex	56.2%	12.5%	50.0%	$1.00

Look at the ladder beneath the winner. Spending more on one product moves accuracy from 22.9% to 33.3%. Adding independent candidates and an oracle-blind selection step reaches 43.8%. A same-product review chain adds almost nothing on top of that, inside the noise of the step below. Then swapping the reviewer for a different vendor's product jumps to 62.5%.

The paired contrast against the strongest budget-matched baseline is +16.7 points, 95% CI [6.6, 26.7]. The silent-defect rate falls from 16.7% to 10.4%. And it costs $1.02 per task against $0.97. A five-cent difference on a 16.7-point gap.

2. It replicated where it had no obligation to

The LitGPT columns are the pre-specified transfer test, and the authors committed in advance to publishing the result whatever it showed. It showed +12.5 points, 95% CI [3.0, 22.0], p = 0.008. The same direction, on a codebase that has nothing to do with recommenders.

This is the finding that blunts the obvious objection. Recommender code has particular shapes; HSTU has particular failure modes; the authors were from Meta Surely the effect is tuned to their home turf. A pre-registered, independent-domain replication doesn't eliminate that worry, but it puts most of it to rest.

3. The mechanism is not "diversity is good." It is complementary error.

This is the most useful part of the paper, and it's easy to skim past because it's definition-heavy. Strip the notation and the idea is simple.

For a pair where agent A implements and agent B reviews, two quantities matter:

Complementarity (D): the probability mass where A fails and B succeeds. These are the errors B is able to rescue.
Realized gain (G): what accrued to A's patch after review. Rescues minus the damage B's review introduced.

If B's errors track A's errors, D is small no matter how good B is in general: B is blind where A is blind. That's what the two measured pairs show:

Claude Code reviewed by Codex: error correlation ρ = 0.21, complementarity D = 0.18, realized gain G = +0.06.
Claude Code reviewed by Claude Code: error correlation ρ = 0.58, complementarity D = 0.10, realized gain G ≈ 0 once the reviewer's own mistakes are subtracted.

Two products that fail alike aren't a second opinion. They are the same opinion, twice. The paper formalizes the relationship and fits a line through it. The slope on complementarity is β̂₁ = 0.34, 95% CI [0.12, 0.56]. But the plain-English version is the one worth taking to work: composition pays when the second agent fails on the tasks the first one passes.

4. The fancy deployed topology's edge is budget, not topology

RankEvolve's production flow is a plan-merge topology: two planning lanes, Claude Code bound to one and Codex to the other, each lane revising its plan while reading the other's latest draft, then a merge, then a review–follow-up-plan dual. It posts the best raw numbers in the paper: 70.8% HSTU accuracy, 6.2% silent defects, 64.6% on LitGPT.

But it runs at $2.40 per task and 250 actions. Hold it to the matched budget and it ties Claude Code → Codex → Claude Code exactly: 62.5%, paired difference 0.0, 95% CI [-5.5, 5.5]. Against a contemporaneously re-run comparator, its paired lift is +8.3 points, but with McNemar p = 0.077 and CI [-0.7, 17.3]. Not statistically distinguishable.

The authors follow their own result: the confirmatory weight rests on the simple pairing, not the elaborate diagram. What the deployed flow buys is accuracy per task at whatever the extra spend costs, and the paper argues that spend is rational because averted defects protect far more expensive training runs. But don't attribute that to the topology. The topology is the packaging.

5. Order matters

The reversed pair still beats every homogeneous condition at 56.2%, but it trails the forward direction by 6.2 points. Which product implements and which reviews is part of the result.

What This Means for You

Stop doubling the same agent. This is the actionable core. If your setup runs two copies of the same product in an author/reviewer pattern, the paper's numbers suggest you are mostly buying a second invoice. Same-product review had ρ = 0.58 and netted approximately zero after its introduced errors. Pick a reviewer from a different lineage. Different vendor, different harness, different training corpus.

Estimate complementarity before you commit. You don't need an oracle to run a small version of this. Take 20–30 tasks where you already know the correct outcome, run your agent, run the second agent's review, and count two things: how many defects the reviewer caught that the author missed, and how many problems the review introduced. If the first number is small, the pairing isn't buying what you think.

Use the budget ladder deliberately. One pass → extended budget → blind best-of-N → cross-product review chain. Each rung fixes a different failure mode and costs a different amount. The paper's data says the biggest single jump comes from switching the reviewer's identity, not from adding passes to the same one.

The economics are lopsided in your favor. The conditions run at roughly $1 per task to gate training runs that consume hours of GPUs. The paper puts the ratio bluntly: an averted silent defect is worth on the order of 10³ times the inference spend that prevented it. If you're arguing for a second-reviewer budget line, that's the argument.

If you build agent products, this is a competitive surface. A different vendor's reviewer is now part of your product's quality story, and the ability to host other agents behind your interface stops being a curiosity and starts being measurable. Composite products are competing on whose reviewer catches what.

What I'd Want Verified

My last post ended with me opening Pi's docs and checking the paper's claims against the primary source. That move is closed off here: Claude Code, and the oracle are proprietary or private. So the audit shifts from implementation to claims. What I can say after reading the HTML version end to end:

The headline survives its own intervals, but the intervals are wide. Each condition is a binary outcome over 96 tasks, so per-condition means carry roughly ±12-point clustered intervals. The paired +16.7 is the right number to quote, and the authors quote it that way. Paraphrase the result without the interval and you're overselling it.
"Matched budget" is observable budget. Same envelope, same prompts, timeouts as failures. It isn't FLOP equality, and the authors say so.
Oracles are incomplete. The paper reports mutation testing and audits to reduce false passes, which is the correct mitigation. It can't eliminate them. A passing patch is "not caught," not "proven correct."
The mechanism is predictive, not causal. The complementarity slope's interval excludes zero and beats a marginals-only model on held-out data, but the paper explicitly doesn't claim that raising complementarity causes gain. Their words.
The clean result and the messy one are different studies. The HSTU case study ran with a human operator at the protocol's gates. ExecML, where the 62.5% lives, had no human in the loop. Don't blend them.
The model pins will age. Claude Opus 4.8 and GPT-5.6 are the products that ran this experiment. A future model that is better at reviewing its own family's errors would shrink the gap. The method is durable; the numbers are a snapshot.
The replication I want: rerun the pair with open harnesses as author, reviewer, or both. Does the effect survive when the products share more DNA than two vendor stacks do?

Limitations

Condensed from the paper's own section, which is unusually candid. The evidence base is two Python/PyTorch codebases, so cross-domain transfer isn't established. The products are black boxes, so compute equivalence can't be shown. Oracles reduce but don't remove false passes. Complementarity is predictive, not causal. The knowledge layer was active during the deployment but never ablated, so its contribution is unmeasured. The cross-dataset transfer results are single runs. And checkpoint forking understates how far iterations diverge from one another.

Closing

"Two agents are better than one" is false as stated. The paper's numbers show a same-product review chain landing inside the noise of the baseline beneath it. The sentence the data supports is narrower and more useful: a different agent's review is worth more than a duplicate's, and you can measure the difference before you bet a workflow on it.

Call-to-Action

Read the paper yourself: RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models. The benchmark details are in Appendix C and the EOP in Appendix D; Table 4 in Section 4 is the part worth your time first.

I Read a 4-Million-Line Study of AI Coding Agents. It Described My Own Tool.

GWA — Tue, 06 Oct 2026 00:37:31 +0000

Introduction

I asked an AI coding agent to read a paper about AI coding agents. The paper spent a good chunk of its length dissecting the exact tool I was typing into.

That isn't a riddle. The paper is Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents (arXiv:2609.00006), a July 2026 source-code study by Paul Barbaste, Tristan Darrigol, Germain Vu, and Tom Wiltberger of the Wavestone AI Lab. One of the eleven systems it takes apart is Pi, the harness I use every day. Another is OpenCode, the host for oh-my-opencode-slim, the plugin I sent a Korean README fix.

So I did the thing the paper invites. I checked its homework. I opened Pi's local documentation and compared three of its claims about Pi against the primary source. Two matched to the digit. The third was more interesting than either.

Here's what the study found, what it means whether you use these tools or build them, and what happened when I audited the auditors.

The Research Process

This is a descriptive study, not a competition. The authors don't benchmark or rank anything. They read source code.

Eleven harnesses, pinned to July 2026 releases: Claude Code (Anthropic), Codex CLI (OpenAI), Gemini CLI (Google), Mistral Vibe (Mistral), plus OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw.
A twelfth system, Omnigent (Databricks), is analyzed as a "meta-harness", an orchestration layer that drives eleven vendor harnesses behind one API.
Roughly four million lines of Python, TypeScript, and Rust, inspected down to dependency manifests and import statements.
Because eight systems were re-pinned rather than replaced from an April 2026 edition of the same study, the paper also contains a controlled 90-day source diff. The same codebases, three months apart.

Out of that came 13 cross-cutting observations, a catalog of 29 recurring design patterns, 18 design recommendations, and a ~90-line minimum-viable-harness scaffold in Python.

Two disclosures shape how you should read it. First, most performance numbers in the paper are self-reported by the systems' own maintainers. Second, the paper was written with substantial assistance from Claude Code, disclosed in an acknowledgments section, and worth knowing since Claude Code is one of the eleven systems under study.

Key Findings

The paper has thirteen observations. These are the four that changed how I think about the tools on my own machine.

1. An agent is a model plus a harness. The harness is the product.

The paper opens with a one-line equation: Agent = Model + Harness. The harness is everything except the model. The loop, the tools, the context management, the safety controls, the orchestration, the extension surfaces.

It then argues that every system, from Mini-SWE-Agent's hundred-line research baseline to Claude Code, has to take a position on the same seven subsystems:

Agent loop
LLM integration
Memory and context
Tool and action system
Safety and permissions
Extensibility (skills, hooks, plugins, MCP)
Multi-agent orchestration

Even "no position" counts as a position. Pi isn't simply missing a sandbox. The paper documents that the absence is a stated design principle, argued for in its own docs. That's the pattern across the corpus: deliberate refusals are documented as carefully as features.

This is why the tools feel so different while running the same handful of frontier models. The model is shared. Everything you actually touch is the harness.

2. The twin absences

This is the finding I didn't expect.

Across four million lines and eleven production systems:

Zero import a general-purpose agentic framework. Not LangChain, not LangGraph, not AutoGen, not CrewAI, not LlamaIndex, not Pydantic AI, not Semantic Kernel. Gemini CLI uses neither of Google's own frameworks, Genkit and ADK.
Zero use vector embeddings to retrieve code. Instead they use ripgrep, tree-sitter, glob, and auto-discovered Markdown context files like AGENTS.md.

The authors searched for counterexamples for weeks, re-ran the sweep after tripling the corpus, and the result held. Every loop is hand-rolled in the host language's native primitives. asyncio, Tokio, Promises. Every tool registry is custom. Every prompt template is plain Markdown or string concatenation.

There's a footnote that savors its own irony: the term harness engineering was named and defined from inside LangChain, the framework vendor whose libraries appear nowhere in the study's runtime code.

The nuance matters before you argue with this finding. Embeddings do appear in OpenClaw, on by default, but for chat recall, never for reading a source tree. Aider can install llama-index as an optional extra, but only to run RAG over its own documentation, not inside its agent loop.

3. Loop sophistication doesn't predict performance

The paper returns to this point repeatedly. Mini-SWE-Agent's linear loop is roughly fifty lines. Its self-reported SWE-Bench Verified score is 74%+.

Codex's workspace, meanwhile, nearly doubled in a single quarter — from 621,000 to about 1.12 million lines of Rust, across 89 to 126 crates. That growth isn't the loop changing. It's everything around the loop: sandboxing, approvals, memory pipelines, plugin marketplaces.

The paper's conclusion: architectural complexity doesn't predict how well an agent scores, but it does predict production readiness — safety, reliability, extension surfaces, and increasingly the client/server and transport layers, where the largest harnesses now carry most of their mass.

The authors back this up with a scaffold. Listing 3 is roughly 90 lines of Python implementing a linear loop, four tools (bash, read, write, search_replace), root-to-leaf AGENTS.md discovery, and threshold compaction. It deliberately omits sandboxing, multi-agent orchestration, MCP, and skills, because the corpus shows real disagreement there. Their advice is blunt: start here, measure, and add the minimum your observed failure modes demand.

4. The standards resolved, and then everyone copied each other

Two format fights settled during the study's window.

Skills beat MCP. SKILL.md skills are used by 9 of 11 systems; MCP by 8 of 11. The paper credits Pi with breaking the tie by implementing skills while rejecting MCP outright. The skills layer then grew a supply chain: registries, trust tiers, provenance verification, cross-vendor discovery (OpenCode reads Claude Code's skills directory), and the corpus's first agent-authored skills.

ACP found a second job. The Agent Client Protocol ships in 6 of 11 systems. It was designed for the editor-to-agent boundary, but the study documents a role outside its original brief: harness hosting. OpenHands can run Claude Code, Codex, or Gemini CLI as interchangeable backends behind its own interface.

Then there's the 90-day diff, which is the part I keep thinking about. In the April edition, the systems had converged by independent rediscovery. By July, the convergence was traceable. Codex adopted Claude Code's hook event vocabulary verbatim and shipped an importer for its sessions and settings. OpenHands adopted Claude Code's plugin manifest format. Patterns visibly diffused down the corpus.

The paper's assessment: the half-life of a competitive differentiator in this field is currently measurable in weeks.

What This Means for You

If you use coding agents

The study explains behaviors you've probably blamed on the model.

The agent that gets vague after a long session is compacting. Claude Code fires compaction below a 13K-token buffer and post-restores files. Gemini CLI compacts at 50%, preserving the last 30% verbatim. Pi triggers at contextWindow − 16,384 and keeps the 20,000 most recent tokens.
The agent that keeps re-reading your repo instead of "remembering" it has no embeddings. And that's deliberate, not a missing feature.
The agent that refuses to use your favorite framework isn't missing a dependency. It's following the entire field.
Tool count matters more than you'd think. Past roughly 15 tools, the paper says, prompt bloat becomes prohibitive and systems switch to deferred tool loading. Claude Code's deferred flag reportedly cuts the initial prompt by about 40%.

If you build agents

Section 16 is the payoff: 18 recommendations, each anchored to an observed system and an explicit trade-off. The ones that stuck with me:

Start with a linear loop and one bash tool. Add more tools only in response to observed failure modes.
Graduate to a middleware pipeline when you have three or more independent turn-level policies, not before.
Don't build RAG over code. Code has deterministic structure that semantic similarity can't replicate, and it changes minute-to-minute.
Codify safety rules as data or policy files, not imperative code. And if you ship a YOLO mode, keep a floor beneath it.
Stay single-agent until you can point at a specific phase where parallel context isolation clearly beats serial search.

What I Verified Myself

Pi's docs are on my machine, so I compared the paper's claims against the primary source.

Claim 1: Pi compacts at contextWindow − 16,384, keeping 20K recent, and merges summaries iteratively rather than re-summarizing from scratch. Verdict: exact match. docs/compaction.md lists a reserveTokens default of 16384, a keepRecentTokens default of 20k, and a summary step that passes "the previous summary as iterative context."

Claim 2: A session_before_compact hook lets extensions veto or replace a compaction result. Verdict: exact match. It's a documented event.

Claim 3: Pi is the only system in the corpus that documents the absence of safety infrastructure as a design argument. Verdict: the docs agree, in plainer language than the paper uses. From docs/security.md:

"Watching the transcript, using project trust, and reviewing changes do not create a security boundary."

That third check is the one I keep coming back to. The paper described a design philosophy I'd never seen stated so directly in the tool I use every day, and then the primary source said the same thing without any academic hedging. It changed how I read the rest of the paper and how I think about what "safe" means in a terminal that will run rm -rf on my behalf.

Limitations

The paper is careful about its own limits, and you should be too.

It's descriptive. The authors explicitly don't benchmark or rank, so don't read "finding" as "winner."
Most performance figures are self-reported by maintainers, not independently reproduced.
Claude Code is analyzed from a circulated source snapshot, not a public repository. A different class of evidence than the other ten.
The draft was AI-assisted, and one of the assistants is one of the studied systems' closest relatives.
Some claims will age. The authors separate inventory claims (tool counts, feature cells, version pins), which decay in weeks, from structural claims (the loop taxonomy, the subsystem anatomy, the twin absences), which have proven durable and they label which is which.

Closing

The study's biggest claim is that coding agents stopped being tools and became platforms in the first half of 2026. The evidence is all in source: harnesses shipping as importable SDKs while framework vendors ship harnesses; cross-vendor session importers; marketplaces, registries, and enterprise governance layers; and a meta-harness making its own harnesses interchangeable.

Or as the paper puts it in Observation 12: "The competitive unit of the field is no longer the agent loop; it is the ecosystem surface around it."

If you use these tools daily, that's the finding worth sitting with. The loop isn't where the competition is anymore. And the platform around it is being built to hold you.

Call-to-Action

Read the paper yourself: Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — the HTML version is fully readable.

Then do what I did. Pick your own harness, open its docs, and check the paper's claims about it. If you find a discrepancy, that's a better blog post than this one.

If you build with these tools, start with Section 16 and the 90-line scaffold in Listing 3.

My First Open Source PR: How a Korean README Fix Merged in Under an Hour

GWA — Mon, 05 Oct 2026 04:40:57 +0000

Introduction

On October 3, 2026, I merged my first open source pull request. It didn't touch a single line of runtime code, and it didn't fix a bug you could reproduce. It updated one file README.ko-KR.mdin oh-my-opencode-slim, a multi-agent plugin for OpenCode with more than 9,000 stars.

The whole thing went from issue to merged in 59 minutes. I couldn't have planned a softer landing into open source if I'd tried, and that's exactly why it's worth writing down.

If you're circling your own first contribution, here's the honest version of the story: the technical work was the easy part. The hard part was believing my perspective counted. This post walks through how it started, the move that made it painless, and five lessons you can copy for your own first PR.

A README That Fell Behind

I'm a user of the project, not a core contributor. I use oh-my-opencode-slim to coordinate a team of AI agents inside OpenCode, and I read the Korean README the way most users do, to set things up.

That's how I noticed the drift. The Korean README had fallen behind the English one. The gaps weren't dramatic, but they added up:

Setup instructions still described a --skills=force command and installer-managed skill updates that no longer matched how the plugin handles bundled skills.
Several agent-role descriptions were still in English, so half the roster was unexplained in the Korean docs.
Literal translations made terms like "work graph" and "evidence path" harder to parse than the English originals they were supposed to clarify.

Docs are a product surface. When setup steps don't match the software, readers don't blame the docs. They assume they messed up. I caught myself re-reading the same section twice because an instruction no longer applied.

For a while, I sat on it. The project has thousands of stars and an active maintainer. My reasoning went something like: someone closer to the code should handle this. I'm a user.

Then I flipped it. The gap existed because the people closest to the code write in English. I read Korean, I used the tool, and I noticed the mismatch. That's the whole job description for this fix. Nobody else was positioned to spot it.

File the Issue First

My first instinct was to write the translation and open a pull request. Instead, I opened issue #1422 first.

The issue did four things:

It itemized the drift. Not "the docs are outdated". A specific list of stale instructions, untranslated sections, and unclear terms.
It showed history. I referenced the three earlier sync efforts (#502, #727, and #935) so the maintainer could see this was a recurring pattern, not a one-off complaint.
It set the scope. One file. Documentation only. No changes to commands, configuration keys, model IDs, or agent names. Ambiguities in the English source would be flagged, not silently reinterpreted.
It claimed the work. I ended with one sentence: "I'd like to contribute a PR for this update."

That last line changed everything. I wasn't asking permission to touch someone's project. I was telling the maintainer exactly what to expect, in a way they could say no to.

Nobody replied to the issue. Maintainers are busy, and silence is normal. The issue still did its job. It put the scope on record. Ten minutes later, I opened PR #1423.

The translation itself took not that long. The interesting constraint wasn't vocabulary, it was voice. The English README introduces the agents with a mythological tone "seven divine beings" and all that. Korean readers deserve the same voice, not a stiff technical restatement. So I rewrote awkward literal translations, translated the role descriptions, and kept the myth intact.

I also practiced restraint. Contributor-generated sections stayed untouched. Where the English source was ambiguous I flagged it in the PR instead of inventing an answer. The final diff: one file, +149/−132, one commit.

Issue to Merge in 59 Minutes

The public timeline still makes me smile:

Time (UTC)	Event
17:42	Issue #1422 opened
17:52	PR #1423 opened
17:54	Greptile review posted
18:41	Merged

Issue to merge: 59 minutes.

That number is flattering, but it isn't the point. The work happened before the clock started. The issue-first move is what made the merge a formality. The scope was already agreed on in writing, so the maintainer had nothing to negotiate.

The review is worth mentioning too. Two minutes after I opened the PR, Greptile flagged a P2 finding. My new Council instructions told readers to type council or @council, but on a default install there are no Council member presets, so the keyword won't trigger anything until members are configured. Fair catch. The maintainer merged the docs update anyway, and that setup condition is still on my list for a follow-up PR.

The lesson stuck: when you document a feature, check the path that gets a new user to it.

One more detail from the PR that I'd repeat every time: the validation section. I ran git diff --check, verified link and image targets, checked code fences and anchors, and confirmed contributor sections were unchanged. I couldn't run bun run check:ci, the typecheck, or the test suite, because development dependencies weren't installed in my environment. So I said that, in plain text, instead of hiding it.

For a docs-only change, that honesty did more than a green checkmark I couldn't produce.

So what changed? The repo I use every day now has accurate Korean documentation. And in GitHub's eyes, I went from user to contributor. The gap between those two labels turned out to be one issue, one file, and one focused sitting.

Key Takeaways for Readers

If you're planning your first contribution, here's what I'd carry forward.

File the issue before you write the PR. An issue is a cheap way to de-risk a pull request. It gives the maintainer a chance to redirect scope before you invest hours, and it gives your PR an agreed target. Ten minutes of writing saved me a rewrite.
Documentation is a real contribution. Translation drift is a bug for the people who need those docs. You don't have to touch the engine to make the project dramatically better for thousands of readers.
Keep the diff boring. One file, one purpose, one commit. My PR body said it plainly: "Only README.ko-KR.md is changed. No runtime code changes." A reviewer can say yes to that in minutes. Scope creep is what turns a friendly PR into a debate.
Be explicit about what you didn't do and couldn't verify. I listed the checks I ran, named the ones I couldn't run, and flagged source ambiguities instead of guessing. Transparency beats the appearance of completeness especially on your first PR, when trust is the thing you're building.
Treat review feedback as free expertise. A bot found a real gap in my work within two minutes of opening the PR. Read the note, verify it, then fix it or schedule it. Don't treat it as a verdict on you.

Closing

The scariest part of my first contribution wasn't the Korean. It was pressing "Create pull request" on a repo with 9,000 stars.

Then it was over. One file. One afternoon. A maintainer I've never met merged it in less than an hour.

If you've noticed something broken in software you use, you're already qualified to fix it. Your perspective is the qualification. You noticed. Most people didn't, or didn't bother.

If you've been carrying a small fix around in your head, consider this your nudge: file the issue today. Not the PR. The issue.

And tell me in the comments: what's the fix you've noticed but never reported?

If you want more notes on open source, AI tooling, and building things in public, subscribe to the blog. New posts land there first.

Resources

PR #1423 — docs: sync Korean README and polish translation
Issue #1422 — sync Korean README with current English docs
oh-my-opencode-slim on GitHub
Earlier Korean README sync PRs: #502 (initial translation), #727, #935

What I learned from Sol-Pi: A Detailed Review

GWA — Fri, 02 Oct 2026 01:47:43 +0000

Intro

Nvidia released a paper "SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness". It's research of how to improve token efficiency and applying it as a plugin of Pi agent.

In this review, I'll cover the features, benefits, and my overall impressions of Sol-Pi.

Overview of Sol-Pi

Sol-Pi a set of four token-efficiency mechanisms for the harness layer, discovered by letting an AI optimizer search harness designs automatically. It ships as a MIT-licensed standalone extension for Pi from.

Features

The method: RSI-inspired auto-research for harness design.

An optimizer agent watches execution traces from a base harness, proposes harness changes, implements them, and
validates them in real environments.

Scale: ~150 proposed directions across 6 proposal families (context, progress, tools, delegation, prompt/policy,
improvement/eval), ~535 executable search environments (495 GitHub issue→PR repo tasks with hidden fail→pass tests +
40 verifier-driven synthetic tasks), 3,000+ runs, 60,000+ agent–environment interactions.

Broad-to-deep funnel: outer loop = many isolated, disposable search lineages (breadth: cheap to kill failures);
inner loop = repeated implement → independent review → revise (depth).

Anti-overfitting discipline capability metrics and tolerances are fixed up front and isolated from the optimizer. A candidate must stay within capability tolerance and improve an efficiency metric. EdgeBench is frozen and held out and its results never feed back into search. That separation is the paper's main methodological claim.

The four surviving mechanisms

Action Fusion

What it does: edit/write can carry a follow-up then_run (test/build) in the same call and kills the extra model round trip.

Online Context Compact

What it does: at plan-step completion, estimates remaining requests vs. prompt-cache rewrite cost and only compacts when projected savings win (near the window limit it compacts anyway).

ObservationPack

What it does: results >10 KiB get archived locally; sent in full for 2 requests, then replaced by a stable handle + head/tail excerpt.

Evidence-Preserving Reducer

What it does: build/test logs ≥4 KiB → cheap model (GPT-5.6 Luna) extracts a receipt; a deterministic verifier checks schema, hash, exit status, exact quotes, size. Any failure falls back to the original log

The reducer runs before ObservationPack, and ObservationPack recognizes the reducer's marker so verified evidence isn't double-processed. Evidence is never hidden: originals stay on disk.

Who Benefits

1. Anyone running Pi on a metered API, doing long sessions.

If your token bill is dominated by re-read context in a 200-turn session, this is the direct win: ~1/3 less cost at near-identical task quality. The conservative config (actionFusion + observationPack) gets much of it with no extra model calls and no run interruption.

2. Agent fleets / swarms.

The paper's swarm experiment is the telling one: 20 SoL-Pi workers reached better optimization results for 26.8% less cost than 20 Pi workers. When you're paying for N parallel workers over a fixed budget, per-worker efficiency converts directly into more collective exploration. This is the scale where a 1/3 cost cut changes what's affordable.

Configuration	Cycles ↓	Model cost ↓	Speed thresholds
Sol + 20 SoL-Pi	1,127	$60.11	8/8
Single Sol	1,333	$39.20	8/8
Sol + 20 Pi	1,366	$82.12	7/8

3. People running unattended / around-the-clock agents.

The motivating scenario is "supervised code completion" → "unattended, 24/7 exploration." Long-horizon runs are exactly where context accumulates and repeated validation actions show up. Cheaper per hour = more hours inside a budget.

4. The recursive case (research vision, not a product feature).

Cheaper harness → the auto-research loop that builds the next harness costs less → fixed budget covers more environments/ideas. The paper calls this "recursive efficient improvement" and is explicit that it's a long-term vision, not demonstrated. This is the interesting one for RSI research, not for daily use.

5. Extension authors (secondary).

SoL-Pi is also a working reference for how to add harness mechanisms to Pi through public APIs without patching it.

Where it is not the use case

Quality-at-any-cost work. On Terminal-Bench 4 it solved 15 vs 18 tasks. It's cheaper per solved task ($14.07 vs $15.91), but if you want max score and cost is irrelevant, don't gut the context.
Short/small sessions. Nothing accumulates, so no replay to eliminate — the mechanisms never trigger.
Local or free models. The entire value is measured in API cost; at $0/token the wins evaporate.
Sensitive logs, with the reducer on. Evidence-Preserving Reducer ships eligible logs to a configured model. Do not enable it on logs that must stay on the machine (the README's SECURITY.md says this outright).
Latency-sensitive interactive work where you care about wall-clock, not spend. That's not what was optimized.

Verdit

It's a cost-optimization layer for long-horizon Pi agents, sold as ~1/3 off at ~94% of quality. The Terminal-Bench result (fewer tasks solved, cheaper per solve) is the kind of trade you'd want to check against your own workloads before enabling all four mechanisms.

Links

Github: https://github.com/NVlabs/SoL-Pi

Paper: https://arxiv.org/abs/2609.20519

Blog: https://nvlabs.github.io/SoL-Pi/

@misc{liu2026solpirecursivelyscalingautoresearch,
      title={SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness}, 
      author={Haozhe Liu and Tian Ye and Sensen Gao and Qihang Cao and Yitong Li and Mingchen Zhuge and Duomin Wang and Ruihua Zhang and Ping Luo and Jiawang Bian and Lei Zhu and Ligeng Zhu and Enze Xie and Song Han},
      year={2026},
      eprint={2609.20519},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.20519}, 
}