No Agent Reviews Its Own Work

What a four-major-version Angular migration taught me about supervising AI agents on production systems

Most writing about AI coding agents is about speed. This is not that. It is about the part nobody publishes: what you do with the output, and who checks it.

I spent a stretch of last year migrating an enterprise healthcare platform from Angular 11 to Angular 15 — four major versions, on a system that could not go down. Billing, contract administration, and daily operations all depended on it. There was no window in which the application could simply be unavailable while we rebuilt it.

That constraint shaped everything. What follows is the structure I ended up with, why I think it holds, and where I know it is weak.

The mechanical part is not the hard part

Version-by-version framework upgrades are largely mechanical. You run the update, the build breaks, you fix what broke, you move on. The changes are repetitive: renamed APIs, adjusted imports, dependency conflicts, deprecated patterns that now fail compilation.

This is exactly the kind of work AI agents are good at. In most of the version bumps, the update produced errors and the agent resolved them faster than I would have. That part of the story is unremarkable, and it is the part everyone already writes about.

The hard part is different: how do you know the agent's fixes were correct?

Not "did the build succeed" — an agent can always make a build succeed. It can widen a type, swallow an exception, pin a dependency to an old version, or delete the assertion that was failing. Every one of those makes the error go away. None of them is a fix.

The structure

What I arrived at is a separation of roles, borrowed less from software practice than from how controls work in accounting and audit:

An executor agent performs the upgrade and resolves the errors it produces.

A reviewer agent — a separate agent, in a separate context — examines what the executor did and judges it. Not "does it compile," but: is this change sound? Does it follow the conventions of the surrounding code? Did it address the cause or hide the symptom?

A test agent runs the verification suite appropriate to that step.

A visual regression agent compares the rendered interface across versions to catch changes that functional tests cannot see.

Me, at the end, deciding whether we advance to the next version.

The rule that holds this together is one sentence:

No participant validates its own work.

This is not an AI principle. It is the reason the person who writes a check does not also sign it, and the reason the author of a pull request does not approve it. I am applying an old control to a new kind of worker.

Why the separation is not ceremony

The obvious objection is that a reviewer agent is just a second opinion from the same kind of system, and therefore worthless.

In practice it did not behave that way. In one migration step, the reviewer flagged aspects of the executor's changes and gave reasons for its objections. Those reasons were sound. The changes were modified accordingly.

What I take from that is narrower than "the reviewer works." It is that an agent evaluating a change it did not produce behaves differently from an agent evaluating its own output. It has no investment in the approach. It did not spend the last several minutes convincing itself the solution was good. That is the same reason we ask another human to review our code, and it appears to transfer.

I want to be precise about the limit: in my experience the reviewer produced substantive suggestions that changed the result. I cannot yet claim it has rejected work outright and forced a redo. Those are different things, and I would not want the distinction blurred.

Green tests, broken application

The fourth role came from a failure mode the other three could not see.

Some framework boundaries change how things render without changing how they behave. A component library adjusts its default spacing, a typography scale shifts, an overlay positions itself differently. Nothing in the application logic changed. Every test passes. And the interface is subtly wrong in a hundred places, which a user will notice immediately and a suite will never report.

Catching that by hand means someone clicking through the entire application after every version bump — which nobody does honestly by the fourth iteration. It is precisely the kind of high-volume, low-variation attention task where human reliability collapses and an agent does not.

So the visual agent compares renders across versions and reports differences. It does not decide whether a difference is acceptable; that judgment stays with me. A padding change may be a regression or an intended improvement in the new library version. The agent's job is to ensure I am never unaware that something moved.

This is also where risk-proportional gating gets sharper. Visual comparison is cheap to run and matters most at the boundaries where rendering itself changes. On steps that only touch APIs and dependencies, it rarely finds anything.

The symmetry nobody mentions

The usual framing is "human supervises AI." I no longer think that is accurate, because it implies the human is reliable and the agent is not.

The truth is that every participant has blind spots. They are just different ones.

Agents cover for me. They do not get tired on the fortieth dependency conflict. They do not skip a file because it looks familiar. They do not assume something is fine because they already looked at it once. They will compare every screen after every version bump with the same rigor on the fourth pass as on the first, which no human does. Human attention degrades predictably across repetitive work, and mine is no exception.

I cover for them. An agent does not know whether a fix addressed the root cause or silenced a symptom. It does not know that a change is technically correct but architecturally inconsistent with the rest of the system. It does not know that a green test suite can coexist with a broken application.

They cover for each other. Which is why the reviewer is not the executor.

The structure is not a hierarchy. It is a set of mutual checks among participants who fail in different ways.

Verification proportional to risk

The second half of the method concerns how much you verify, and when.

The naive approach is to run the full suite after every version bump. It is also impractical: a complete regression pass on a platform of this size is expensive, and running it four times inflates the migration without buying proportional confidence.

So the gates vary by the blast radius of the jump:

• Low-risk steps — where the changes are confined to APIs and dependencies — get a smoke pass. Does it build, does it start, do the critical paths respond.
• High-risk boundaries get the full regression suite.

The interesting question is what makes a boundary high risk. My first instinct was to key it to the Node version, since a runtime change affects the build and CI environment for everything.

That instinct was incomplete. In this particular trajectory, the jump that actually broke things was structural rather than runtime-related: Angular Material 15 rebuilding its components on MDC, which changed the DOM and CSS classes the application's styles depended on. The major version number is a convention. The blast radius is a property of what actually changed, and the two do not reliably coincide.

Classifying the jumps by real risk, rather than by version number, is the part of this that transfers to other stacks. The Angular specifics do not.

Where this is weak

Two things, stated plainly.

My own gate is the least formalized part of the system. During the migration described here, what I was actually checking at the final review was that the changes seemed reasonable and the tests passed. "Seems reasonable" is a judgment, not a criterion. It cannot be audited, and it is hard to teach. I have since turned it into an explicit checklist — whether a fix addressed cause or symptom, whether the change is consistent with surrounding conventions, whether the diff stayed within the scope requested, whether anything changed behaviour the tests do not cover — and applied it in production on a smaller system. But the checklist came from systematizing an intuition, not the other way around, and it has not yet been tested at the scale of the platform described here.

Reviewer independence is a matter of degree. If the reviewer agent receives the executor's reasoning along with its output, it inherits some of the executor's framing. Reviewing only the diff is more genuinely independent. Using the same underlying model for both roles also limits independence, since the two may share the same blind spots. These are design choices with real consequences, and I do not think the tradeoffs are settled.

Why this matters beyond one migration

Legacy modernization in healthcare stalls for a predictable reason: the systems that most need rebuilding are the ones that can least afford to stop. They sit underneath eligibility, authorizations, claims, and order processing. The cost and risk of touching them exceed the perceived cost of leaving them alone, so organizations defer — and the gap widens each year.

That deferral now has a deadline. CMS-0057-F requires impacted payers to stand up FHIR-based APIs, including one for prior authorization, largely by January 2027. For many organizations, the systems that hold authorization and payer data were never built to expose anything through a modern API. Meeting the rule means changing platforms that cannot stop running, on a fixed federal timeline.

AI agents change that arithmetic, but only if the output can be trusted. In a regulated environment, "the agent did it and the tests passed" is not an answer anyone can act on. What is needed is a structure where every change has a traceable path: who proposed it, who checked it, against what criteria, and who accepted it.

That is what the separation of roles buys. Not speed — traceability.

The methods above are not novel in isolation. Separation of duties is old. Risk-proportional testing is old. What may be new is applying them to participants that are not people, and being explicit that the human is one of the fallible participants rather than the guarantor of the whole thing.

An open question: cost

A structure with four agents consumes considerably more tokens than a single agent doing everything, and as models take on more of the work, that consumption keeps growing. I am currently investigating ways to reduce it without weakening the checks. If the separation of roles is going to be practical beyond a single migration, it also has to be affordable.

I write about legacy modernization in healthcare software. If you are working on similar problems, I would be glad to compare notes.