DEV Community: JaviMaligno

Proved, Certified, Swept, Sampled

JaviMaligno — Sat, 10 Oct 2026 14:29:03 +0000

A claim in a paper can be backed by very different things.

It can have a written proof. It can have a theorem the Lean kernel has checked. It can have a certificate that ran exact rational arithmetic over an entire region and never once used a floating-point number. Or it can have "I sampled ten million configurations and none of them broke it".

All four print the same way: a sentence in a serif font that sounds true. And the fourth is worth a great deal less than the first, in a way that no amount of confidence in the prose can fix.

My last preprint — the one about packing nested rings in a frying pan — is sixty pages in which all four appear, sometimes on the same page. So every claim in it carries an epistemic label saying which one it is: proved, box-certified, grid-swept or sampled. Not a green tick for the paper. A label per claim, and an appendix pairing every computational claim with the script that backs it.

The paper says it in one sentence, in the acknowledgements, and I would defend that sentence over any of the theorems: the mathematical guarantee for every claim is the written proof and, where a proof delegates an identity, an exact symbolic computation; the verification workflow and the numerical sweeps are quality control and evidence, never a substitute for proof.

This article is about what convinced me to be that pedantic. It was not a principle. It was a certificate that lied to me three times, and the specific way it lied.

The claim that looked certified

One lemma in that paper concerns a quintet of rings, and closing it needs a statement of the form: over this whole four-dimensional domain of parameters, this configuration does not fit. That is exactly the kind of statement you certify computationally, because the domain is continuous and there is nothing to enumerate.

The first attempt did it the way everyone does it: subdivide the domain into boxes, evaluate at each box, use a linear program at the tight ones. An adversarial reviewer — a separate model, given the statement and told to break it — refuted it in one pass, and the refutation was not subtle. The meshes had no Lipschitz bound, so nothing connected the values at the sample points to the values between them. The linear program ran with tolerances. And one of the required inequalities was not certified at all; it had been sampled.

That is a fair fight and an easy fix. The second version was written rational-directed: exact arithmetic on the corners, no floating-point comparisons in the acceptance test.

It was refuted too, and this is the part worth the article.

Why a tolerance is not a small sin

The second version still had one tolerance in it: 1e-12. Not as a comparison, but as slack — the width of the band inside which two quantities counted as equal.

Now, the configuration the whole lemma turns on is the golden point, where the parameters all equal \varphi. And at that point the configuration is exactly tangent. The rings touch. The feasibility margin there is not small: it is zero.

At a point where the margin is zero, a tolerance does not introduce a small error. It changes the answer. The \varepsilon-thick acceptance band swallows the tangency, and the certificate reports fits at precisely the configuration on which the proof depends. Everywhere else in the domain the tolerance is harmless, which is what makes it so bad: the test is wrong only where the question is delicate, and it is silent about it. Nothing crashes. Nothing looks suspicious. You get a green run and a false lemma.

I want to be careful not to over-claim here. The tolerance did not make the lemma false — the lemma is true, and the final certificate proves it. What the tolerance did was make the certificate not evidence. It had been reporting a property of its own acceptance band rather than a property of the geometry.

What it takes to stop assuming

The third version repaired the tangency but still leaned on mpmath for the inverse trigonometry, so it still rested on a floating-point library being right about the arcsine of numbers near a tangency. The fourth version removes that too, and its shape is worth describing because it is what "certified" ends up meaning when you push on it:

\arcsin is bounded by a pure rational series, with \sin^2 and \cos bracketed in \mathbbQ by an alternating Lagrange remainder — so the bound is a theorem about the series, not a call to a library.
Every interval addition and subtraction uses directed rounding, stepping one ULP outward after each operation, so the interval is a genuine outer bound rather than a hopeful one.
\pi and 2\pi are enclosed between rational bounds proved by that same series. They are not rational, of course; the bounds are, and bounds are all the certificate ever needs. One that hardcodes a float for \pi has just assumed part of what it is checking.
The library's math.asin is still called — as an oracle that is not believed. It proposes where to look; a bracket with certainty in each direction decides. If the oracle lied, the search would widen rather than accept.

The result closes the domain with zero tolerances, zero exclusions, and no floating-point fact assumed anywhere.

One thing changed under that lemma after the fact, and it is worth saying rather than leaving for a reader to discover: the paper's second version proves the global threshold by a different and much shorter route, so this quintet belongs to a programme of specialised cases that the global theorem no longer needs as a premise. The certificate is still correct, and it still certifies what it says it certifies. It is simply not load-bearing any more — which changes nothing about the lesson, and would change everything about how much weight you should put on it.

And the golden point itself, the one the tolerance had been papering over, ended up closed algebraically rather than numerically. The apparent double corner turns out to be a ghost — a third constraint empties it — and the real corner does not fit in the mural arrangement, because the lower arc overshoots by 1.4 \times 10^-4. That margin is the whole of it: a ten-thousandth, at the one place where a 1e-12 band had been declaring victory. The correct witness stacks the small ring radially instead, and at the golden point every condition falls out as an exact identity in \mathbbQ[\sqrt5], with margin 1/\varphi^3.

A wrong label is worse than a weak one

The four labels are only useful if they are honest, and the failure mode is not the weak label. It is the label that is too strong.

The same adversarial pass that killed my certificate also read a draft about square pans, rederived all the exact algebra — the quartic, the identity b_\square(X) = X - 1, the staircase polynomials — and confirmed it without exception. Then it found the thing I had not declared: a claim marked proved that was only proved for \alpha \ge 1. For \alpha < 1 the argument simply was not there. Nobody had noticed because the label said the question was settled, and a settled question does not get read twice.

A sampled label is not dangerous. Everyone knows what to do with it: rely on it lightly, or go and prove the thing. A proved label on a claim that holds in half its range is dangerous precisely because it is trusted, and because it silently props up everything downstream of it.

The other half of that honesty is admitting when you stopped. Some regions in this paper are declared exhausted by cost — the sweep ran for its budget, covered what it covered, and the paper says the region was not finished rather than pretending the covered part was the whole. A claim labelled as an honest declaration is worth more than the same claim labelled as proved, because a reader can plan around the first and will be misled by the second.

Verification that improves the result

It is easy to read all this as damage control, so here is the other direction, which surprised me more.

One of the harder lemmas, about a derivative staying above 1 on a blocking boundary, came back confirmed — rederived from scratch by a different route, validated in exact rational arithmetic at thirty points, cross-checked against finite differences to twenty-nine decimal places, and hammered on adversarial meshes of about ten million points with no counterexample. Zero refutations.

But the verifier did not stop at confirming. It found that a hypothesis in my draft — that the parameter had to exceed some threshold — was an artefact of my coordinates rather than a real constraint, so the result holds everywhere. And it found that my closing constant was suboptimal: where I had closed the argument at the golden ratio, the right inequality closes it at the Tribonacci constant, which is strictly better and consistent with another result in the paper.

That is the part of verification nobody advertises. A pass whose only job is to attack the claim will sometimes hand back a stronger claim than the one you wrote, and four documents that had been carrying "modulo this lemma" caveats got to drop them.

What this is worth outside a paper

I do not think the labels are a mathematics thing. I think mathematics is just where the mismatch is embarrassing enough to force the issue.

Your test suite, your type checker, your property-based tests and the one careful read-through you did on a Sunday are four different kinds of evidence, with different failure modes and different silent regions — and CI paints all of them the same green. A passing test is sampled: it says nothing happened at the inputs you chose. A property test with a generator is closer to grid-swept: it says nothing happened across a shape of inputs, and stays quiet between them. A type is nearer box-certified: it holds over a whole domain, for the properties it can express. And the read-through is the only one that is ever proved, on the rare occasions the argument is small enough to hold in a head.

Three things follow, and they are the ones I would actually use:

Know where the tolerance is. Every system has one — a timeout, a retry, a float comparison, a rounded threshold, an "approximately equal" in a test. It is invisible almost everywhere and decisive exactly at the boundary, which is the only place anyone ever asks a hard question. Ask which of your green checks are green because of it.

Label the claim, not the system. "The service is tested" is the sentence that hides everything. "This invariant is enforced by a type; that one by a test at three inputs; this third one we believe because it has not broken in a year" is the sentence that lets someone else decide what to lean on.

Let something adversarial read it that did not write it. Not a reviewer who will agree with your framing — one that gets the statement alone and is asked to break it. It refuted me three times on one lemma, and the fourth version is the only one I would put my name to. It also found the overclaimed label I had stopped seeing.

The last one is the reason my papers now carry an appendix nobody asked for, pairing each computational claim with the script behind it and a per-round record of every refutation and repair. It is the least glamorous part of the work and the only part that would let you catch me being wrong.

The preprint is here, the code, certificates and verification reports are open, and the mathematics those certificates are about is in the companion article. A related habit, from a different angle: the instrument fails in your favour.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Sorry, That Was For Another Chat

JaviMaligno — Tue, 06 Oct 2026 15:13:41 +0000

You have done this. There is something in your clipboard that belonged to a different conversation — you copied it for another reason, or you forgot it was there at all — and it ends up pasted into a chat where it makes no sense. Most of the time you catch it before sending. Sometimes you don't.

There is a newer version of the same mistake, and if you work with agents you have lived it this week: twenty sessions open in parallel, each one waiting on something, and you answer the wrong one. The reply was true — just not there.

I was setting up an experiment about exactly this when it happened to me. An agent had just put an API key on my clipboard, and in the same breath suggested a shell command to save it. I copied the command to run it. The command overwrote the key. The file ended up containing the text of the command instead of the secret.

That is the whole phenomenon in one move, and it is worth being precise about why it is interesting.

This is not prompt injection, and it is not a topic change

Four research lines sit next to this and none of them cover it.

Irrelevant context injected into a task prompt is very well studied — GSM-IC and its successor GSM-DC — but single-turn, on arithmetic, where the noise is part of the problem statement rather than an accident.

Deliberate topic switches are covered by Beyond Continuity, which measures whether a model notices the user has pivoted. Its conclusion travels well: models drag stale context along even with explicit cues. But there the user meant to change the subject.

Getting lost in multi-turn is Laban et al.: a 39% average drop, and the memorable finding that once a model takes a wrong turn it does not recover. There the failure is under-specification, not a stray paste.

Instructions to ignore previous content are almost always framed adversarially — Nevermind, instructional distraction, context-ignoring attacks.

The accidental paste is none of these. Its defining property is that it is ambiguous. That block of text could be three different things, and the model has no way to tell them apart:

The reading is a choice the model cannot avoid making. In the twenty-four conversations I read, it was never once resolved towards "you probably made a mistake".

The setup, briefly

Eight deliberately unrelated topics — a house move, training for a 10K, choosing a school, baking bread, a trip to Japan, a balcony vegetable garden, an electricity bill, buying a camera. A simulated user drives the conversation for either 2 or 10 turns. Then a block of text is pasted in raw, with no preamble, exactly as a real misfire arrives.

The pasted blocks come from a bank of 64 clipboard artefacts — a recipe, an SSH config, a stack trace, meeting minutes, a shopping list, a SQL migration, a job ad, a prompt from another chat. They were written without any knowledge of the conversation topics. That constraint matters more than it looks: if you generate a paste that is "moderately related to a conversation about bread", you have built a designed distractor, which is what GSM-DC already studies. Here similarity is an emergent property of crossing topic × artefact, measured afterwards with embeddings, not a dial I set.

Three models: two sizes of the same GPT-5.6 family — sol, the larger, and luna, the smaller — and Claude Opus 5. Twenty-four conversations with a paste, three controls without one. I read all of them, in full, by hand.

Nobody thinks you made a mistake

Zero out of twenty-four.

Not one reply contains anything like "was this meant for this conversation?". The two behaviours I would have bet on before running it — flagging it as a probable error, and silently ignoring it to carry on with the topic — did not appear a single time.

What I got instead was six behaviours, and the axis of variation is not detection. It is how much unrequested work the model does.

Nineteen of twenty-four replies never acknowledge that anything changed. The interesting minority is small, and it is where the good behaviour lives.

It just does the work

Fourteen of twenty-four take the paste as the new request and deliver. A conversation about a balcony vegetable garden receives a Rust panic, and the reply opens:

The problem is the unwrap() on the key lookup. Replace it with a contextual error that says which field is missing.

A conversation about an electricity bill receives a pumpkin curry recipe, and gets a genuinely good answer about frying the curry paste in the thick cream from the top of the coconut tin instead of in oil. Not one word about the bill.

It asks what to do — but never whether it belongs

Five replies ask for clarification, which sounds like the good outcome until you read what they ask:

Would you like me to turn it into more polished release notes, translate it into English, or draft a launch message?

The question is about the output format. Nothing in it questions whether a changelog for a parsing library belongs in a conversation about buying a camera. All five came from the gateway models; Opus never did this once.

It notices, and complies anyway

Two replies name the jump. One of them is the best behaviour in the whole run:

I see you've changed the subject and pasted a Prometheus alert rule. Here's a quick review in case you want to check it, and at the end I'll come back to the garden, which I owe you.

It notices, it helps, and it commits to returning. It still never entertains that the paste was an accident — but it is the only reply that treats the previous conversation as something it has a debt to.

The invented bridge

One reply builds a plausible link and runs with it. The topic was training for a 10K; the paste was a job ad for a night-shift baker:

A quick note in case it's the first option, which is the one that connects to what we were talking about: a 2am-to-10am shift doesn't stop you preparing a 10K in April, but it does force you to reorganise the plan.

The advice is good. The connection is invented. Both things are true at once, and that is exactly what makes this failure mode expensive: nothing in the reply looks wrong.

The pasted prompt that ate the conversation

One artefact type deserves its own paragraph. Five of the sixty-four artefacts are prompts from another chat — the thing you copy when you are reusing an instruction. One of them landed in a conversation about moving house:

You are a style editor. Follow the Vilaseca Style Manual and nothing else. Flag sentences over thirty words…

Opus took the role. It correctly noted that the text to edit was missing and that it did not know the cited manual. And then, for lack of anything else to work on, it applied the four style criteria to its own previous answers in the conversation, in a table, quoting itself and proposing corrections.

This is the point where the accidental paste brushes against prompt injection, and the difference is worth stating plainly: there is no adversary here. There is a clipboard.

The one that got it right

One reply, out of twenty-four, did the thing I would want. The topic was an electricity bill; the paste was a notice from a residents' association about lift repairs:

This text is a notice from the residents' association, not from your electricity bill. […] If you received it along with the building's monthly charge, those €87 could explain why that charge is higher, but they don't affect your electricity bill unless they've been included by mistake.

It spots the mismatch, explicitly considers whether there could be a real relation, and dismisses it with a reason. That is the opposite of the invented bridge: a link considered and rejected. It matters because it proves the ceiling exists — this is not a capability problem.

The split by family, on eight conversations each

The six behaviours are not spread evenly across the three models, and the
contrast is sharp enough to be worth writing down even though the numbers are
tiny:

	Does the work	Asks about format	Mentions the jump	Other
GPT-5.6 sol	5	3	0	—
GPT-5.6 luna	5	2	0	1 (the good one)
Claude Opus 5	4	0	2	1 bridge, 1 role

Every single "what would you like me to do with this?" came from the two GPT
models, and neither of them ever mentioned that the subject had changed. Opus
never asked about format once, and it is the only model that ever named the jump
— as well as the only one that invented a bridge, and the only one that took on
the pasted prompt's role.

Two different default postures, in other words: one asks you to pick an output
format, the other comments on what just happened and then gets on with it. Which
is more useful probably depends on what you were actually doing.

Eight conversations per model. That is not a finding, it is a pattern worth
testing properly, and it is now the first thing I want out of the next round.

What I am not claiming

Twenty-four conversations, one paste each, no preregistration, and the design deliberately rotates topics and lengths so that nothing is measured with any statistical power. This is an observation, not a measurement. I cannot tell you a detection rate, and I cannot tell you whether similarity between the paste and the conversation changes anything — across these 24 the acknowledgements land at 0.23, 0.26, 0.26 and 0.37 cosine, and the silent ones spread across the entire range.

The next article runs a proper grid, with enough conversations per cell to talk about rates. If it contradicts this one, I will say so there.

What to do the next time it happens to you

From reading all of it, the practical advice is less about the models and more about what you can expect from them:

Assume it will be taken seriously. The default reading is "you meant this". If you paste a stack trace into a conversation about bread, you will get debugging.
A clarifying question is not a detection. "What would you like me to do with this?" means it has already accepted the topic change and is asking about formatting.
Watch for the bridge. The expensive failure is not the model doing the wrong task — you notice that immediately. It is the model weaving the stray content into the thing you actually care about, plausibly, in a reply where nothing looks out of place.
If the paste is an instruction, expect it to be followed. A prompt from another chat is not read as data. It is read as a role.

The second half of this — what happens after you say "ignore that, wrong window" — is the next experiment. My suspicion, which the data has not tested yet, is that saying it may leave more residue than saying nothing at all.

This is the first of three articles. The next one measures the curve; the third one is about the repair. Related: Your Agent Doesn't Know What's Internal, on context going the other way — internal material leaking into things it shouldn't.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Count the Rings or Sear the Squid

JaviMaligno — Sat, 03 Oct 2026 13:24:19 +0000

Drop a handful of squid rings into a frying pan and you have already made a decision, whether or not you noticed making it.

You can lay them out so that as many rings as possible are in the pan. Or you can lay them out so that as much squid as possible is touching hot metal, which is the thing that actually cooks. Those sound like the same instruction phrased twice. They are not, and the gap between them is wide enough to prove theorems in.

What makes it a real problem rather than a word game is that a ring has a hole. A small enough ring drops inside the hole of a larger one and sits flat on the pan, touching metal exactly as much as it would have on its own. So the rings are not competing for area the way coins on a table compete: a big ring is an obstacle and a container at the same time.

The idealisation, stated up front

Real squid rings do something the model forbids: they ride up on each other. A ring resting partly on top of another is not gone — it still sears everywhere outside the overlap — and because the rings have thickness, that pose is not a measure-zero accident. It is what actually happens in a crowded pan.

The model in this article is the rigid one: siblings in a container must be packable as balls with disjoint interiors, so nothing ever rides on anything. That is a real idealisation and it is worth naming before the theorems rather than after, because every result below is a result about the rigid model.

The flexible version is a named open direction rather than an oversight. Let \delta be the width of the lifted ramp a bent ring makes around each overlap. Then \delta = 0 — contact area equals the annulus minus the overlapped region — defines a continuous relaxation in which partial placements trade contact for cardinality, and \delta > 0 penalises overlaps by a dead band proportional to the overlap perimeter. Neither is solved here.

Stripped of the squid, the rigid problem is a selection-flavoured relative of the Recursive Circle Packing Problem, introduced by Pedroso, Cunha and Tavares (International Transactions in Operational Research, 2016) to model telescoping tubes in shipping containers and later solved exactly by Gleixner, Maher, Müller and Pedroso. That literature is algorithmic: it asks how to pack a fixed set of rings into as few containers as possible, and its methods are heuristics — procedures that propose placements without guaranteeing that the placement mattered. I wanted the structural questions instead. Which objective does the obvious greedy algorithm provably optimise? When is the choice of where to put each ring provably irrelevant? And which condition on the sizes decides the answer?

The preprint is Greedy Packing of Nested Rings; the code, figures and Lean certificates are open. This is the readable version of what is in it.

Two goals that sound like one goal

Fix a width w — how thick the squid wall is — and say a ring of outer radius r has a hole of radius r - w. The area touching the pan is the annulus:

a (r) = π (r^{2} - max (0, r - w)^{2})

which for thin rings is close to 2\pi w r. So sear — total contact area — behaves almost like the sum of the radii, minus a fixed penalty \pi w^2 for every ring you use. Count is just the number of rings.

That penalty per ring is the seed of the whole disagreement. Adding a ring always adds to the count. It does not always add enough area to be worth the space it occupies, because the space it occupies might have gone to something bigger.

Say the two objectives diverge on an instance when the area-optimal arrangement uses strictly fewer rings than the count-optimal one. It turns out you need a surprising amount of structure before that can happen at all.

With two rings, never. If both fit together, take both: area strictly increases when you add a ring, so the full set wins on both counts. If they don't fit together, every feasible arrangement has at most one ring, and the area optimum already achieves that count. Two rings cannot disagree with themselves.

Three rings can disagree, for the wrong reason

With three it can happen, but only in a degenerate way. Take a pan of radius R = 10 with very thick rings, w = 9/2 = 4.5, and radii

8, \frac{101}{20}, \frac{99}{20} = 8, 5.05, 4.95

The two small rings are exactly diametral — 5.05 + 4.95 = 10 — so they fit side by side across the pan and nothing else fits with them. The big ring's hole has radius 8 - 4.5 = 3.5, too small for either of them, so nothing nests. And the areas compare as

a (8) = \frac{207}{4} π > \frac{198}{4} π = a (5.05) + a (4.95)

The single big ring sears more than both small ones together: count says two, area says one. But look at why. The rings are so thick that no hole can hold anything, and the problem has quietly collapsed into ordinary circle packing. The nesting — the thing that makes this squid and not coins — has been switched off.

The smallest disagreement that is really about rings

Turn the width back down so nesting is live again, and the disagreement almost disappears. Almost. The smallest instance where it survives with the holes doing work needs four rings: a pan of radius 10, width 1, and radii \9.0,\ 4.2,\ 4.2,\ 4.2.

Play it both ways. If you want rings on the pan, take the three 4.2s: they fit side by side, N = 3, and they sear about 69.7. If you want squid cooked, take the 9.0 and drop one 4.2 into its hole: only N = 2, but about 76.7 of contact area.

Three rings or more dinner. Not both.

The mechanism is worth naming, because it is narrow. It needs small rings that fit k times in the pan but at most k-2 times in the big ring's hole. If they fit k-1 times in the hole, the two objectives tie and there is nothing to argue about. That off-by-one is the entire divergence in the nesting regime, which is why the minimal instance has four rings and not three.

Where the disagreement lives

Once you know it exists, you can map it. Fix the width at 1 and the pan at radius 10, take the family "one large ring of radius b plus as many equal small rings of radius s as you like", and sweep.

The divergence region is a staircase, and the steps are not arbitrary — each one sits at a proved optimal threshold for packing n equal circles in a disk, results that go back to Pirl and Melissen. The band's upper edge is exactly the three-circle threshold, 0.4641 R. Above that, small rings are big enough that three of them no longer fit and the arithmetic stops working.

One honest note, in the same breath as the claim: for the thick-ring family of the previous section, the onset of three-ring divergence sits near w/R \approx 0.26. That number is swept, not proved. I sampled it; I did not establish it. The paper says so at that sentence rather than in a footnote, because "I swept a grid and this is where it turned" and "I proved this is where it turns" are not the same currency, and a reader who cannot tell them apart will lean on the wrong one.

One condition, and the problem stops being interesting

Now the other half, which surprised me more than the divergence did.

Call the radii superincreasing when every ring is bigger than all the smaller ones put together: r_i > \sum_j>i r_j for all i. It is a strong condition — the sizes have to fall away fast, each one dominating the entire tail — but it is not exotic. It is the same condition that makes greedy work for coin systems, and the one-dimensional ancestor of this result is a 1987 theorem of Coffman, Garey and Johnson: for bin packing with divisible item sizes, First Fit Decreasing is optimal.

Under superincreasing radii, the descending greedy — take the biggest ring, place it, move on — produces the lexicographically maximal feasible set. And that has a consequence bigger than it first looks: lex-max means it simultaneously maximises

i \in S \sum v (r_{i})

for every positive, strictly increasing, superadditive v. Contact area is one such v. So is the sum of radii, and the sum of perimeters. All of them at once, by the same arrangement. In this regime the disagreement I spent the first half of this article building is simply gone.

Count itself is not rescued, and the reason is precise: cardinality is v \equiv 1, which is not superadditive, so the dominance argument does not apply to it at all.

The counterexample is a pan of radius 10, width 4.8, radii \9.95,\ 5.0,\ 4.3,\ 0.6. The greedy takes \9.95,\ 5.0: two rings, optimal area, and no step even offers a choice of container, so every placement rule agrees. Meanwhile \5.0,\ 4.3,\ 0.6\ packs in a row in the pan for three. The frontier is exactly superadditivity, and cardinality sits on the wrong side of it.

And then where you put things stops mattering

This is the part I did not expect when I started.

Under the same condition, you do not merely have an optimal greedy. Every descending greedy is optimal, with an arbitrary rule for choosing which container to drop each ring into. Best fit — the tightest container that will take it. Worst fit — the roomiest. Random. Adversarial. They all place exactly the same lex-max set.

The intuition worth holding onto is not "the algorithm is clever". It is that under superincreasing radii the decision you agonise over has no downstream consequence: whatever you do with the current ring, the rings still to come are collectively smaller than it, and the exchange argument can always rearrange them around your choice.

And because that argument only ever looks inside the ball vacated by a moved ring, it never mentions what the pan looks like. The theorem is stated and proved for an arbitrary compact container K \subset \mathbbR^d, with rings read as spherical shells. A round pan, a rectangular griddle, tubes and spherical shells nested in three dimensions — the original shipping-container setting — are all covered verbatim, not by extension. I do not know of a comparable placement-independence guarantee elsewhere in the circle-packing literature.

The computational corroboration is the kind I like, because it is a genuine attempt to break the claim: 100 random superincreasing instances, run under best fit, worst fit and random placement. All three produced optimal — and therefore identical — outcomes, without exception.

Three rings, and then four

A theorem is only as interesting as its edge, so: how much of this survives without the condition?

With arbitrary radii and no superincreasing hypothesis at all, every descending greedy on at most three rings still lands on the lex-max set. Three rings are simply not enough room to make a bad choice.

Four are.

Pan of radius 15, width 0.3, radii \10,\ 5,\ 4.9,\ 4.8. All four rings fit, and the arrangement that achieves it is tight in both places at once: the 10 and the 5 are exactly tangent in the pan (10 + 5 = 15), and the 4.9 and 4.8 exactly fill the hole of the 10 (4.9 + 4.8 = 9.7, the hole radius).

Now run best fit. Facing the 5, it prefers the snug container — the hole of the 10 — and nests it. That single reasonable-looking decision forces the 4.9 out into the pan, and once the 4.9 is in the pan, the 4.8 has nowhere left. Best fit gets three rings. Worst fit gets four.

So placement obliviousness is sharp. It holds unconditionally at three and fails at four.

The twins

You might reasonably conclude that the fix is a better rule. Best fit is naive; write a smarter one.

You can't, and the reason is the sharpest result in the paper.

Take a pan of radius 15, a 10 and a 5, width w = 0.505 — so the hole of the 10 has radius 9.495 — and these two instances:

I_{1} = 10, 5, 4.99, 4.50, I_{2} = 10, 5, 4.76, 4.74

In I_1 the two small rings sum to 9.49, which fits in the hole. So the 5 belongs in the pan, and worst fit gets it right while best fit fails. In I_2 they sum to 9.50, which does not fit in the hole. So the 5 belongs in the hole, and now best fit gets it right while worst fit fails.

Opposite decisions. And here is the point: at the moment of decision, the two instances are indistinguishable. The containers are the same, their capacities are the same, the occupants are the same, the incoming ring is the same, R and w are the same. Every quantity a placement rule could look at, reading the state in front of it, is identical — and the correct move is different.

The consequence is not "best fit is bad". It is that no deterministic rule which is a function of the observable state can be optimal on all instances, and every randomised rule fails some instance with probability at least 1/2. The information required to decide is not in the state. It is in the rings you have not looked at yet.

The constant that wasn't

There is one more thread, and it ends in the nicest wrong guess I have had in a while.

If superincreasing radii give you all of this and violating them costs you all of it, there should be a threshold in between. Measure the violation by how badly the worst ring is beaten by its own tail:

ρ = i max \frac{\sum _{j > i} r _{j}}{r _{i}}

so \rho \le 1 is exactly the superincreasing condition. In the additive relaxation — where siblings are feasible precisely when their radii sum to at most the capacity, geometry stripped out — the threshold is exactly \rho = 1. Clean, universal, and the reason the additive model is the right place to isolate the combinatorial half of the difficulty.

The geometric model is where it gets interesting. The rigid four-ring family of counterexamples has an infimum, and that infimum is exactly the Tribonacci constant T \approx 1.83929 — the analogue of the golden ratio for the recurrence that sums the previous three terms. It is proved, with no tangency idealisation smuggled in. Given that three-term structure and a problem about rings inside rings inside rings, the natural conjecture writes itself: T is the global threshold.

It isn't. There is an explicit family — pan of radius \varphi + 1, radii \varphi,\ 1,\ \varphi/2 + 2\varepsilon,\ \varphi/2 + \varepsilon\ — that breaks placement obliviousness at \rho = \varphi + 3\varepsilon, for every small \varepsilon > 0. Since \varphi \approx 1.618 < 1.839 \approx T, that proves the geometric threshold \tau satisfies

τ \leq φ < T

and the Tribonacci conjecture is dead. The golden ratio gets there first.

For a while that was only half a result: \tau \le \varphi was proved, and the matching lower bound only for pair profiles and outside an explicit heavy region. It is now closed. For disks the global threshold is exactly \tau = \varphi — no failure at all at \rho \le \varphi, for every finite inventory, and even when each ring is allowed its own independent hole radius. Tribonacci is demoted from "the threshold" to "the exact floor of a rigid subfamily": still a sharp constant, just not the one I expected it to be.

What closes it is the nicest theorem in the paper, and it is the kind of statement you can carry around:

Order the rings a > b \ge c \ge r_4 \ge \dots \ge r_n, and suppose every tail is bounded by \varphi times its radius. Then the whole list fits in a disk if and only if the three largest fit.

Everything after the third ring comes along for free. Not "usually fits", not "fits with high probability": the question about n rings collapses, exactly, to a question about three. That is what supplies the uniform exchange the threshold proof needs, and it meets the four-ring golden counterexamples coming down from above — beyond the golden bound the collapse fails, below it there is no failure left to find.

What the edge looks like outside the frying pan

Everything above about the edge — the failure at four, the twins, the floors — was originally a statement about a round pan in the plane. The positive half never needed the shape; the sharp half had only been measured there. So I went and asked which of the two the boundary really belongs to.

In balls of any dimension, nothing moves. The three-ring guarantee holds for an arbitrary compact container in any dimension, and the failure at four survives with the same instance \10,\ 5,\ 4.9,\ 4.8\, and so do the twins, and so does the Tribonacci floor. The reason is a reduction lemma worth stating on its own: balls of radii a_1, \dots, a_k fit as siblings inside a ball of radius R in \mathbbR^d if and only if they fit in \mathbbR^k-1. Sibling queries of three or fewer rings therefore have identical answers in every dimension d \ge 2, and every result whose proof only asks such questions comes along for free. A separate argument pushes the golden threshold itself up to five rings in those dimensions. The first query that can tell dimension 2 from dimension 3 needs four pieces, and there is an explicit one: radii \441,\ 440,\ 439,\ 438\/1000 in a ball of radius 1, which fits in 3D with centres at the four even-sign points (\pm a, \pm a, \pm a), a = 8/25, and does not fit in the plane.

Square pans are where the constant genuinely changes. Placement irrelevance fails at four there too, twin instances kill state-based rules in a square as well, and the bound has been pushed down to

1 \leq τ_{□} \leq Y \approx 1.684487745872346

where Y is the positive root of (17 + 10\sqrt2)Y^2 + (72 + 16\sqrt2)Y - (112 + 96\sqrt2) = 0. So the disk and the square do not share a threshold: \varphi \approx 1.618 against something near 1.684. The corner is what changes it. Whether Y is optimal is open.

And the rings no longer have to share a width. Let each ring carry its own hole radius h_i < r_i — equivalently, its own thickness — and the selection result survives untouched: at \rho \le 1 every descending greedy still produces the lex-max set, in any compact container and any dimension. What does not survive is area optimality, and it fails in a way you can put a number on. Fix \rho \le \kappa < 1; then

A_{greedy} \geq c (κ) A_{opt}, c (κ) = min (1, κ^{- 2} - 1)

and that constant is the best possible. Below \kappa = 1/\sqrt2 the guarantee is 1 — the greedy set is the only area optimum, and the divergence this article opened with cannot happen. Above it, the guarantee decays, and two rings in a round pan are already enough to show the constant cannot be improved.

What is still open: the global threshold in dimension three and above — the reduction to a plane covers queries of three siblings, not counterexamples built from four or more — and whether Y is the true square constant.

What I take from this

Two things, and neither is about squid.

The first is that "what am I actually maximising?" is not a philosophical warm-up question. It is the question that decides the answer. Count and sear look interchangeable until you write them down, and then they pull apart in a region you can draw. If a system optimises the proxy you gave it rather than the thing you wanted, the failure is often not that the optimiser is bad — it is that you handed it the wrong v, and the two only coincide outside the band you happen to be in.

The second is more cheerful. There exist regimes where the hard part evaporates: where every objective in a broad class agrees, where the obvious algorithm is provably right, and where the decision you would have spent your time on has no consequence at all. Knowing whether you are inside one is worth more than any amount of cleverness spent on the decision itself. Here the test fits on one line — is every ring bigger than the sum of the rest? — and the reward for passing it is that you get to stop thinking.

This is a long way from the algebra I spent my doctorate on, and it started, genuinely, in a frying pan. The preprint has the proofs; the repository has the code, the figures and the Lean certificates for the exact identities.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Indexed Is Not Served: A Month of Flat Zero With Every Metric Green

JaviMaligno — Wed, 30 Sep 2026 14:56:52 +0000

On 15 August VitaminD Explorer served 879 impressions
in Google Search. On 16 August it served 35. It has not recovered since: a month
later it runs between 8 and 40 impressions a day, with essentially zero clicks.

It is a solar vitamin D calculator: given a real location and a real skin type,
it works out whether the sun outside can synthesise vitamin D at all right now,
how many minutes it would take, and which months of the year it is possible at
that latitude. That question has a different answer in every city and every
month, which is why the site carries a few thousand pages — and why it is a
useful specimen for this post-mortem.

Every health indicator I had was green throughout. No manual action. No
deindexing — the index count went up, from 3,170 pages to 3,180. Sitemap
submitted, read, accepted: 3,636 URLs, last fetched three days before I looked.
Googlebot still crawling, no host errors. Average position 9.9, which is where
it had always been.

This is the post-mortem of a diagnosis, not of a fix. I still do not know the
cause. What I do have is a set of measurements that killed several comfortable
explanations, and one distinction I did not have before: being indexed, being
crawled, and being served are three different currencies, and you can be rich in
the first while bankrupt in the third.

The aggregate hid it for thirteen days

The first mistake was not analytical, it was ergonomic. Every report I looked at
was a 28-day or 90-day total. "119 clicks in three months" reads like a small
site slowly growing.

Opening the daily series showed something else entirely: those 119 clicks are
almost all concentrated between 21 July and 14 August, with days peaking at 15.
After that the line is flat on the floor. Thirteen days had passed before anyone
looked at the shape of the series rather than its sum.

An aggregate cannot tell a rising trend from a dead one that used to be rising.
That sounds obvious written down. It is not obvious when the dashboard's default
view is a total and you are busy.

What the cliff looks like up close

Day	Impressions	Clicks
14 Aug	1,427	5
15 Aug	879	3
16 Aug	35	1
17 Aug – 10 Sep	8–40 per day	~0

A single day, −96%, permanent. That shape matters: a seasonal decline slopes, a
technical breakage usually shows up in error reports, and an algorithmic
reclassification flips.

The explanations that died on the next check

In one afternoon I produced three confident causes. All three were falsified,
two of them by me, within hours:

"It's the antispam update." Google's ran from 18 August. The cliff is the 16th. Two days before, not after.
"It's the title change I deployed on the 15th at 22:06." URL inspection said Google had last crawled those pages on 26 July and 3 August. It had not seen the change at all. I had asserted a cause for a deploy the crawler never fetched.
"It's seasonal, August queries dying." September pages were already accumulating impressions before the cut, and the collapse is identical across three languages whose seasonal patterns are not identical.

The discipline that survives this is worth more than any of the three
hypotheses: do not produce a fourth explanation just because the third one
died. An unexplained collapse is an acceptable state. A wrong explanation you
have started acting on is not.

The measurement that reframed it

Comparing 17 Aug – 10 Sep against the equivalent window before the cut, split by
URL family — each family being the same template in a different language, e.g.
sunrise and sunset in Madrid in September:

Family	Impressions before	After	Position
`/sunrise/` (en)	14,300	347	9.0 → 21.6
`/amanecer/` (es)	15,400	111	12.0 → 13.8
`/sonnenaufgang/` (de)	4,850	43	8.4 → 11.3
Whole site	36,900	488	9.4 → 20.2

Look at the two columns together. Impressions fall by 98.7%. Position falls by
between 2 and 12 places. Those are not the same event.

If a site slid down the rankings, impressions would decay roughly in proportion
to the positions lost — you keep appearing, lower. Here the site keeps roughly
its position where it still appears, and stops appearing at all almost
everywhere. The number of distinct queries returning it went from four figures
to 159 in 28 days.

That is not a ranking problem. It is an eligibility problem: the site stopped
entering the candidate set for the long tail. And I would not have seen the
difference from the headline "average position", which mixes both into one
number.

Three currencies, not one

Fifteen days after the cliff I published a small set of new pages: a hub
answering how long in the sun you need for vitamin D,
with a variant per skin type, in six languages. Two weeks later I checked them
one by one, and the states are worth quoting exactly:

Spanish hub: indexed, the day after publishing.
German: "Crawled – currently not indexed" — fetched, evaluated, rejected.
Russian: "Discovered – currently not indexed" — known, never fetched.
English, French, Lithuanian: "URL is unknown to Google" — not even discovered.

All six are in the same sitemap. The sitemap was read, successfully, three days
before I ran these checks. Google had the list and was not going to go and get
them.

Then I requested indexing by hand. The Russian page went from "discovered, never
crawled" to "URL is on Google" in minutes.

That is the whole lesson in one experiment. The content was not being rejected —
when asked, Google fetched and indexed it immediately. What was missing was
crawl demand: the appetite to go and look. Three separate things, which the
word "indexed" collapses into one:

Discovered — Google has the URL.
Crawled — Google spent a request on it.
Served — Google puts it in front of someone.

A dashboard that reports 3,180 indexed pages is telling you about currency 1 and

It says nothing about 3, which is the only one that has users in it.

The number that invalidates the usual advice

While I was in there, one more thing, measured across 33,000 impressions and
1,000 pages before the collapse:

Band	Pages	Impressions	Clicks	CTR
Positions 6–10	637	22,852	68	0.30%
Positions 11–20	253	10,273	32	0.31%

Moving from page two to page one bought nothing. Not "less than expected" —
nothing, to two decimal places, on a sample big enough to see it. The expected
CTR in positions 6–10 is a few percent; this is an order of magnitude below.

Every piece of SEO advice I could act on is denominated in positions. On this
site, in this SERP shape, position is not convertible into clicks at all. Which
means the honest answer to "how do we get more traffic" was never "rank better",
and a year of work aimed at ranking would have returned zero — measurably,
before doing it.

What I would take from this

Look at the shape of the series before the total. An aggregate cannot distinguish "growing" from "dead, was growing". Open the daily chart first, every time.
Separate rank from eligibility. If impressions fall far faster than position, you are not being outranked, you are not being considered. Those have different causes and different fixes.
Do not let "indexed" stand in for "served". Check discovered, crawled and served as three separate states. The failure can sit in any of them, and the index count will look fine in all three cases.
Test whether your currency converts. Before optimising a metric, measure what a unit of it is worth. Mine was worth zero, and one table showed it.

I am going back to the cause with a clearer question than the one I started
with. Not "why did the traffic fall" — "why did this domain stop being
considered". I do not have the answer yet, and I would rather say that than
supply a fourth story.

Meanwhile the thing still works, which is the part search never measured:
getvitamind.app answers the sun question for your own
location and skin type, with no account and no app to install, and there is an
MCP server if you would rather ask your
assistant.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The memory that was true

JaviMaligno — Sun, 27 Sep 2026 14:00:13 +0000

The previous article ended on an open question: where each kind of context belongs. While I was preparing it, my agent handed me a stale memory and I very nearly acted on it.

The memory said that scheduling an article on this site means creating a single-use workflow named after the article. That was true in July. The mechanism has since been replaced by a manifest: one JSON file holding the publication queue. There is not one of those workflows left on the main branch.

The memory wasn't wrong. It was true, and it stopped being true while nobody was looking. No mistake, no carelessness in writing it: the world moved and the memory stayed where it was.

Which led to a more uncomfortable question. If that one was stale and nobody knew, how many others?

Two memories with opposite rules

I run two memory systems at once, and they are built on deliberately opposite criteria.

One is personal: the files Claude Code keeps per project. The agent writes them when a session ends, I review them, and they come out of conversations I was present for.

The other is production: the knowledge system of an internal DevOps bot that answers requests over chat. There the memory is written by a small model after every response, unsupervised, from interactions with other people that I have never read.

That second system carries machinery the personal one doesn't need, and every piece of it answers a specific problem of writing without supervision. A new fact doesn't enter as true: it enters as a candidate, and only gets promoted when the bot independently reaches it again in a later investigation. Whatever nobody confirms within a month is deleted. Whatever goes ninety days without a single query retrieving it turns obsolete and stops being injected, though it isn't destroyed: confirm it again and it comes back. And every night a job clusters whatever looks too similar and decides whether to merge it.

	Personal	Production
Who writes	the agent at session end	a small model, alone
Reviewed by	me, before it's stored	nobody
From what	conversations I was part of	third-party interactions
Trust	goes straight in	two confirmations
Staleness	nobody detects it	expiry through disuse
Contradiction	they coexist quietly	resolved
Usefulness	not measured	retrieval counted

Seen side by side, the difference isn't sophistication. It's who writes and with how much supervision. When I write, or review what gets written, trust comes for free: that's why the personal system can afford to be light, and why working with it feels easy. When a model writes alone, from material nobody has read, trust has to be built from scratch, and without that machinery the system degrades by itself.

They are different problems and there is no reason for them to converge. Bringing promotion-by-evidence and nightly forgetting into a folder I review myself would add ceremony where a person is already doing that job.

What expires on its own

There's a distinction I was slow to see, and it orders everything else.

A preference doesn't become false by itself. If my standard is that an article should avoid sweeping claims, that doesn't stop being true on its own: it changes the day I change my mind, and on that day I say so. Nothing needs verifying.

A fact about the system does expire on its own, quietly, because the world moves without telling anyone. The publishing mechanism changed without the memory describing it noticing.

So automatic verification only makes sense over the second half. That stops being a limitation and becomes the design criterion: you can only check what can expire without anyone touching it.

Counting them

I let an agent build a verifier, to see whether automating it turned up anything that wouldn't show by hand. The idea is simple: attach to each memory a check that runs against the repository, along the lines of this memory claims those workflows exist, so there should be at least one. If there isn't, the memory gets flagged.

{ "file_matches": ".github/workflows/scheduled-publish-*.yml" }

Across the 35 memories this project had on 11 September:

state	count
green	7	the check passes
red	3	claims something no longer true
grey	25	no check is possible

Let me give the verdict on the tool up front, because it isn't the interesting part: it's clumsy. Every check has to be written by hand, and writing one means reading the whole memory and deciding what it claims. Once you've done that, you already know whether it's still alive; the program only confirms it. Reviewing all 35 by hand would have cost about the same and produced the same number.

What justifies the detour is what turned up along the way, and it isn't the number. (The code is here, with its limits documented.)

The three reds say the same thing

I verified each red by hand, because a red is an accusation. All three held up. And all three share a shape:

One said the scheduling mechanism is single-use workflows.
Another, that its article was still pending on a branch. It was published on 30 August.
Another, that two articles were awaiting review and merge. They went out on 15 and 30 July.

None was wrong when written. All three described a transitional state. "This is pending", "right now it works like this". And a transitional state starts expiring the moment it's written down, because the whole point of it is that it's going to change.

There's a practical rule in there I hadn't expected to find: a memory recording a stable fact ages well; one recording a situation in progress is born with an expiry date. And nothing forces you back to it, because the day the work moves on you are busy moving it on.

What can't be checked, which is nearly everything

I went through the 31 memories with no check and only 7 admitted one. Of those 7 I dropped another as forced: it was a preference of mine about how to order a document, and the check substituted the presence of a literal string in a file. Rewriting that heading would have turned it red without my changing my mind, and ignoring the preference while leaving the string in place would have kept it green. It wasn't measuring what it claimed to measure.

So: about 80% of my memory admits no mechanical check.

I could report that as the tool's limit. I think it's more honest the other way round: that 80% is judgement, preferences and ways of working, and it's the part that makes working with an agent bearable. It needs no verification because it doesn't expire on its own. It is already well handled, with no machinery on top.

The instinct when you measure something is to want the number to go up. Here, pushing it up would have meant forcing checks onto memories that don't admit them — which wouldn't measure staleness at all. It would manufacture greens.

The green that doesn't mean what it looks like

The most useful result of the exercise was one of the greens.

A memory about publishing automation came out green: its seven files all exist, all verified. And inside, that same memory said a credential expired on a date that by then was a month past.

The green was correct and the memory was stale, both at once. A green certifies what was encoded, not the whole memory. There's no contradiction: there's a tool answering exactly the question it was asked, and a reading of mine that wanted it to answer a larger one.

When I went to fix it, the credential turned out to be perfectly alive: it had been renewed in August and nobody wrote that down anywhere. So the fix wasn't correcting the date, because a new date expires again in sixty days and the problem repeats. It was removing it and recording where the state can be checked — which turned out to be a daily workflow that was already checking exactly that and writing it into its own log.

The memory went from asserting a fact with an expiry date to saying where to look. The second kind doesn't age.

A check that cannot fail

There's one last result, and it's about the attempt to automate itself: it's why I don't trust the number beyond what it's worth.

The first checks that went in described how the mechanism works today: that the manifest exists, that no single-use workflows are left. Those are true statements, so they all passed. But a check like that watches nothing: it describes the present, and the present always describes itself. For reality to be able to contradict a memory, you have to encode what the memory claims, not what is the case now.

A check that always passes is worse than no check at all, because an unchecked memory is known to be unchecked, while one with a vacuous check looks watched.

And that's the limit of the whole approach: the tool that measures whether a memory is still true can be wrong in the same way as the memory it watches, with the same consequence. Nobody notices, because the report says everything is fine. A real verifier would need someone verifying the verifier, and that doesn't hold up on its own.

What actually fixes it

After the whole detour, what keeps a memory alive isn't checking it: it's closing the session by updating it.

When a piece of work ends — the article goes out, the mechanism changes, the decision gets made — that is the moment the memory describing it stops being true, and it's also the only moment when someone has the whole context in their head to fix it. Half an hour later it already costs something, and a week later it has to be reconstructed.

The good news is that the agent often does this on its own, unprompted. The bad news is that "often" isn't "always", and the three reds above are precisely the cases where it didn't happen. So it's worth making sure: let closing a session include asking what, of what was written down, has stopped being true today.

It's less impressive than a verifier and it works better, because it attacks the problem where it starts instead of detecting it months later.

What only shows up when there's more than one of you

All of the above is one person and one project. Three reds out of thirty-five is not a staleness rate for anything: it's what came out of my folder.

The interesting part starts where my case ends, and that's where I am now. When several people feed the memory, three questions appear that don't exist alone.

Who maintains the current state when the record has no owner. There are more options here than it looks, and none is obviously right: name someone responsible; have each person update whatever their own contribution touches, since they're the ones in a position to know; have updates be proposed and then maintained automatically; or let whatever nobody uses wither on its own, the way the production system expires facts through disuse. Several of these probably coexist, depending on the kind of memory.

How you correct something other decisions already cite. Here I take the idea from the production system that strikes me as its most transferable: if manual corrections become frequent, what needs fixing is how things get saved, not building a more comfortable deletion tool. Correcting a lot by hand isn't healthy maintenance, it's a symptom.

And what each person sees, which framed as "which person sees what" is the wrong framing. Whatever is common to the project has to reach every agent, and therefore every person: that's what common means. The finer point is roles. A PM and a dev on the same project may have different access, or the same access organised differently, so that what is knowledge for one is general context for the other.

What isn't common is the other half: each person's own practices and methodology, which differ partly because each of us works with different agents. That shouldn't be unified, and forcing it would repeat the mistake of wanting two systems with different problems to look alike.

But there's a nice case in between. When several of those personal practices converge on their own — the same habit showing up in people who never agreed on it — that is exactly the signal the production system uses to promote a fact from candidate to confirmed: someone arriving at it independently. Applied to ways of working rather than to facts, it gives a route to standardise without imposing: what converges gets proposed, and the rest is either decided together or left where it is.

None of it is solved. But the small exercise leaves two things I do take with me:

Anything recording a situation in progress should be marked as such, because it will expire and it helps to know where it will break.
The moment to fix a memory is when the work that makes it stale finishes, not months later with a tool.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Explicit State Doesn't Forget. It Misremembers.

JaviMaligno — Sat, 26 Sep 2026 13:07:06 +0000

In the first half of this replication I ran
SKILL.state on Claude and found that its headline
degradation did not appear there. The obvious objection is that the paper runs on
Gemini-3-Flash, not Claude. So this half runs on theirs — gemini-3-flash-preview
through Vertex, in an environment rebuilt to match their Appendix B — plus Claude Haiku
4.5 under the same environment and output cap, with every cell repeated and
every episode kept as a per-step trace.

The short version: the paper's central claim holds on its own model, more cleanly than
the paper reports it. On a second model it does not, and the way it fails is the
interesting part. Explicit state does not lose track of the procedure. It writes the
state wrong, and nothing checks the write.

Repeat every cell, even at temperature zero

The protocol started from an assumption that sounds safe: with greedy decoding, a seed is
an instance of the environment, and one run per cell is enough. It is not. The same cell,
repeated five times at temperature=0, scored anywhere between 0.830 and 0.960. The
distribution has a long left tail, because failures cascade: the agent stores one pallet
on the wrong shelf, and every later decision that touches that shelf inherits the error.

Three conclusions written up from single runs — an environment effect, a monotonic
noise curve, a noise estimate — did not survive repetition. Every figure below is a mean over
15 to 24 runs, with the spread computed over runs, not over seed means.

On their model, the thesis holds

Their Table 1 has four runtimes: ReAct (full transcript), Memory (rolling summary),
Stateful (state block plus transcript) and SKILL.state (state block only). Here are the
two that carry the argument, ours against theirs:

Solid lines are this replication, dashed lines their Table 1. SKILL.state (teal) does not miss once in 75 episodes. The full-transcript arm (amber) degrades far less than they report, and the gap widens with the horizon.

SKILL.state does not fail once in 75 episodes of up to 200 steps, where theirs loses
six points. The direction of their thesis reproduces with room to spare. The magnitude
does not: our ReAct loses 0.080 between 10 and 200 steps where theirs loses 0.160, and
sits 17 points above theirs at T=200. Reasoning budget, output cap and the three
environment gaps I could find against their appendix were each varied, and none moves it.
The model name is theirs, but gemini-3-flash-preview may not be the exact checkpoint
behind their Gemini-3-Flash, so I cannot rule the model out — only say I did not change
it.

Memory is the one arm where the comparison is between two different things. The paper
does not specify how its Memory summarises, so a replication has to pick a policy, and
ours keeps a much tighter summary: at T=200 its mean prompt is a fourteenth of theirs.
With that tighter summary Memory is the arm that degrades most, last at T=100 and T=200.
How much of that belongs to summarisation as such, and how much to how hard it
compresses, these runs cannot say.

On a second model, explicit state writes the state wrong

Same environment, same output cap, Claude Haiku 4.5 through Microsoft Foundry, 24 runs
per cell at the two ends of the horizon:

T=200	ReAct	Stateful	SKILL.state
Gemini-3-Flash	0.913	0.930	1.000
Claude Haiku 4.5	0.974	0.999	0.958

On Haiku the ordering turns over. The full-transcript arm is almost flat (−0.005 from
T=50 to T=200) and SKILL.state is the one that drops. To find out why, I replayed every
episode against the real environment and compared, step by step, the inventory the model
believes — the one in its state object — with the one that is actually on the shelves.

Every failure of explicit state has the same origin. In all 17 episodes where the belief
drifted from reality, the step that corrupted it was a Move executed correctly with a
state patch written wrong. Move is the only transition that makes the model copy one
shelf's contents into another key of its own state, and that copy is where it slips.

The failure SKILL.state was built to prevent is losing track of the procedure. The one it has is writing a wrong fact into the only place the agent looks — and even an explicit rejection from the environment does not make it re-read that fact.

It is systematic, not noise: in 6 of the 8 runs of seed 2 the corruption happens at
exactly that step, and from there the agent cascades through about 25 failures. Not
every miswritten patch matters — in seed 0 the model copies the wrong lot number,
which no later decision reads, and those runs lose nothing — but the write is never
checked against what the action did. Gemini never made this error in 75 episodes; Haiku
wrote at least one wrong patch in 17 of 24 at T=200.

The paper does study erroneous state updates — premature overwrites, schema and type
errors. This is a narrower thing: a schema-valid patch with the wrong value in it, which
passes every check the runtime runs.

Keeping the transcript absorbs it

The Stateful arm is the control that makes this legible. It maintains the same kind of
state block, and it also keeps the full transcript. On Haiku it makes the same kind of
write error — its belief drifts from reality in 8 of 15 episodes at T=200, always a
correct action with a miswritten patch — and loses nothing to it: the wrong state
prescribes a different action on only 4 steps, and on all 4 the model does what reality
requires. None of its failures comes from its state.

On Gemini the same arm behaves the other way round: its state drifts in 11 of 15
episodes, where SKILL.state on the same model never drifted, and 77 of its 211 failures
happen with a correct state in front of it. Stateful and SKILL.state also differ in
response format, parsing and retries, so this describes rather than isolates the effect
of the transcript. But the practical reading is hard to avoid: on the model that
misremembers, the redundant copy of the history is what catches it.

Where explicit state wins, measured by the decision

The paper's recovery experiment asks what happens when a fact the agent was told stops
being true. I measure it per episode: on a correct trajectory the correction decides
exactly one step in each of the three scenarios used, so the question is whether the
agent gets that step right. Three models now, the same design on each:

70 of 72 episodes with explicit state, 10 of 72 with the full transcript. Every missed decisive step is exactly what an agent that never heard the correction would do.

Explicit state applies the correction in 70 of 72 episodes (Wilson 95 % interval
90–99 %); the full transcript in 10 of 72 (8–24 %). This is the paper's own claim, and
it reproduces cleanly. It is also smaller than the "93 of 93" I reported in the first
half: that count read its dependent steps off a simulated trajectory instead of the
agent's real one, and it could not register a miss that came from the agent doing
nothing. Measured on the real trajectory, the direction survives and the perfection does
not.

The other probe tests the limitation the paper declares: explicit state only protects
what its schema anticipated. A fact announced at step t first becomes relevant at step
t+40, and the agent either stored it somewhere or did not.

SKILL.state, fact first needed 40 steps later	Haiku 4.5	Gemini-3-Flash
No dedicated field	0/24	0/24
Free-form `notes` field	10/24	1/24
Schema field that names the fact	24/24	16/24

Sonnet 5 is left out of this table because a quarter of its episodes never reach the test
— its trajectory has diverged by step t+40 — and its cells need two rates to read
honestly; they are in the paper.

A place to put it is not enough; the place has to say what goes there. "No dedicated
field" does not mean nowhere to store it — Sonnet's few successes in that arm wrote the
quarantine straight into the inventory object — but the named field is what reliably
works, and these experiments cannot separate the extra slot from the cue its name gives.
For the runtime with no schema at all, a reminder attached to the observation does the
job: ReAct goes from 0–34 % to 71–96 % across the three models.

The bill

The cost half of the first article stands. With a short procedure, caching shrinks
SKILL.state's input-cost advantage over the transcript from 7.5x to 1.4x on
Anthropic, because an append-only transcript is an ideal cacheable prefix and a mutating
state block is not. On Vertex the same accounting barely moves the ratio — 7.5x in tokens
is 7.2x in effective input — because implicit caching there saved ReAct 6.8 % of its
input, against 82 % on Anthropic.

And prompt order, isolated this time, is a first-order cost variable. The same Stateful
runtime, sending the same content, with the transcript marked as a cacheable prefix in
both arms, costs 5.2x more when its state block goes in front of the transcript than
behind it — 869k against 168k effective input tokens per 50-step episode — for the same
score. The paper's own prompt template puts the state block in front.

What I would take from it

Explicit state does what the paper says on the paper's model. On Gemini it never loses a step, and it applies retractions almost every time on all three models.
Its failure mode is writing, not remembering. A schema can validate the shape of a patch; nothing in the runtime checks that the value matches what the action did. If you build on explicit state, that check is the part to add.
Keeping the transcript next to the state is cheap insurance on some models. It cost Stateful nothing on Haiku and caught every wrong write that would have mattered.
Put the mutating part last. It is a one-line change and, measured in isolation, a 5.2x difference on the invoice.
Repeat the cell. Temperature zero is not a reproducibility guarantee, and a single run hides exactly the tail where the failures live.

Every number here was recomputed from the per-step traces by an independent adversarial
reviewer — nine rounds of it, each allowed to read the raw data and none of the prose —
and the paper, the traces and the code are public.

Code, per-step traces and the full paper draft: JaviMaligno/delayed-relevance. The original paper: SKILL.state (arXiv 2608.26263). The first half of the replication: When the Fact Stops Being True.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Making yourself replaceable

JaviMaligno — Thu, 24 Sep 2026 13:34:15 +0000

There is one ability I particularly value in a company: helping someone else take over your work without needing you for every decision.

The better you do this, the more value you contribute. Yet the result is that you become less essential to keeping that work going.

I find this paradox interesting because we tend to talk about being irreplaceable as something to aspire to. Being the person who knows the most, solves the difficult problems and has all the answers. But if that knowledge has to pass through you before anyone else can use it, you have also put a limit on what the team can do when you are unavailable.

I would particularly value someone who does their work well and also gives others the tools, knowledge and context to do it independently. That ability remains valuable after the handover: they can apply it again on another project, with another team or to a harder problem.

I arrived at this reflection through something quite concrete: preparing a handover when you work with agents.

What isn't in the repository

Handing over the code and explaining how to run it covers part of the work. But there is much more you have accumulated along the way: how changes are reviewed, what gets checked before a task is considered finished, which constraint the client requested, which alternative was rejected and why an apparently better solution cannot be used yet.

When I try to sort out where each of those ends up, I get three different places:

The documentation, which is the part we usually treat as the deliverable.
The conversations with Codex or Claude Code, where the approach was argued out, where something was tried and failed, where the standard for reviewing work was agreed.
Whatever came from elsewhere and never entered the project at all: an email, a Slack thread, a meeting where a priority changed.

That third category is the worst preserved, and it is usually the one that weighs most. A client constraint is rarely written down as a constraint; it arrives in an email, it shapes the code, and then it disappears. The code remains. The reason does not.

When someone joins to help you, they need to find their way through all of it. The repository tells them what exists. To continue the work, they also need to understand what has been agreed, what remains open and the reasons behind it. Without that, they can read the entire codebase and still propose the alternative that was rejected a month ago, for a reason that still holds and that nobody wrote down.

Archive and memory

In my case the practice is fairly simple: I keep records of the communications relevant to the projects my agents work on. Emails, Slack conversations and meeting notes, in files that live in the project repository itself.

The material comes in through two routes, and the difference matters more than it looks. For email and Slack I use connectors, so the agent can consult them itself when it needs to. For meetings I use the notes Gemini and Granola produce, and those I bring in myself. Some conversations have no connector at all and I simply paste them.

That asymmetry is why the local files are not redundant once you have connectors. A connector solves whatever is connected; the files are the only place everything else can land. They are also what makes the context outlive the tool: if I switch note-taking systems tomorrow, whatever has already been captured is still there.

The part I care about most is separating the current state from its history while preserving the reasons and decisions. They are two different artefacts and it pays not to merge them:

	Archive	Maintained memory
What it holds	every communication, in full	only what still applies
How it's ordered	by date	by topic
Question it answers	what happened, and when?	what do we know today?
When something changes	one more entry is added	it's rewritten, the old marked superseded
What it's for	investigating, checking, citing	getting to work

I keep that up at two moments. When new material arrives, it is archived as it is and whatever it changes in the current state gets updated. And when a working session ends, whatever was decided during that session goes down into the memory too.

Both are needed. The first captures what happens outside; the second, what happens while you work. With only the first, the memory never records the decisions you made yourself. With only the second, the memory has no idea the client changed their mind on Tuesday.

Right now I supervise both distillations. Automating them is the next step, not something I have already solved.

Imagine that in the meeting on the 17th we agree to deliver an integration on Friday the 18th. That meeting note is archived with its date, and the memory now says delivery is Friday. Two days later an email arrives: there is a dependency on the client and delivery moves to the following Tuesday. The email is archived with its date, exactly as the note was. And the earlier memory entry is not deleted:

## Integration delivery

Date: Tuesday 22                      [decision · 19 Sep]
Blocked by: client dependency
Outstanding: confirmation of the staging endpoint
Source: email 19 Sep — client

~~Date: Friday 18~~                   [superseded · 19 Sep]
Source: meeting record, 17 Sep

Six lines doing three things at once: they say what currently applies, they let you see what came before, and they point at the source of both. Whoever picks up the project finds the live date first, along with what is still needed to meet it, which is what they need in order to start working. If they need to understand the change, the earlier agreement is one step away. Merging the two forces you to read everything just to work out what still stands.

This is why the date of a note, where it came from and whether it records a proposal or a decision matter a great deal. Someone floating a date is not the same as the team agreeing to one, and in a summary those two look far too similar. A summary can be useful and wrong. Being able to return to the email or the meeting record helps prevent an agent's interpretation from turning into an agreement nobody made.

There is one last decision that looks administrative and isn't: whether those files get committed. I don't always commit them. While they sit uncommitted they are my memory, and they work just as well for my own work. The moment they enter the repository they stop being mine and become the team's context, available to anyone who opens the project. Same technical gesture, completely different purpose.

What holds that gesture back isn't laziness, it's that it forces you to decide what may go in. A Slack thread carries names, an email may carry client data, a meeting record captures things people said without expecting them to be written down. Preparing that material to be shared is real work, and it is precisely the work that makes the handover possible. Until it is done, what I have is a very comfortable personal practice that is no use to anyone else.

This helps me too

None of this requires anyone to be joining. I forget things too. I also return to projects after several weeks and need to recover why we decided something. Keeping that record reduces the work of getting my bearings again.

That is the part I didn't expect: when the current state is written down somewhere, changes become visible. If the memory says delivery is Friday and an email turns up assuming a different date, the contradiction surfaces. When everything lives in your head, that same contradiction resolves itself quietly, usually in favour of whatever you read most recently.

For someone new, the difference can be greater. We are asking them to continue conversations they were never part of. If they also have to discover where those conversations happened, who remembers what and which parts still matter, much of their onboarding becomes a search for context.

Access to that material does not instantly bring anyone to your level. Experience, practice and guidance still matter. But it lets that guidance focus on developing judgement instead of reconstructing what has already happened.

This is where preparing a handover starts to look like a daily practice rather than a farewell task. Every important decision that remains easy to find and up to date is something you will not have to explain from scratch later. It helps when you take a holiday, bring in support or simply want a colleague to make progress while you are busy.

The company has to make room for it

If sharing knowledge always happens after the "important work" is finished, it will be the first thing dropped under pressure. And there is always pressure.

If recognition also goes only to whoever personally unblocks every problem, preparing others to solve it will receive little credit, because its effect shows up exactly where nobody is looking: in the problems that stop escalating.

It should count as a contribution when a colleague can take on a responsibility that previously depended on you. The same applies when someone can recover a decision without calling you, or when the team keeps going while you are away. These are outcomes worth considering when evaluating someone's work, and they are hard to see if you only look at the problems that were solved and not at the ones that stopped arriving.

Where each kind of context belongs

The next technical step would be to move this memory into a shared environment: several agents consulting the same context, and updates reaching the project even when its lead did not attend a meeting or write that code.

Before that there is a more basic question, and it is the one I have open: where each kind of context belongs.

For what belongs to the project — dates, agreements, client constraints, what was ruled out — the repository itself is a reasonable home. It sits where the work sits, it is versioned along with it, and it reaches only the people who already have access.

But there is another half that belongs to no single project: how a change gets reviewed, what is checked before calling something finished, which technologies we have tried and which we dropped, the skills and tooling I have been refining. That applies across every project at once. Copying it into each repository guarantees the copies drift apart, and that the good version ends up being the one in the head of whoever wrote it — which is exactly the starting point I was trying to get away from.

That material needs a home of its own that can be shared, and there the questions change:

How the current version is maintained when nobody owns the file.
How a mistake gets corrected once a practice becomes obsolete.
Which information is appropriate to share with each person.

There is enough engineering behind that for another article. But you can start long before solving any of it: capturing what matters, distinguishing agreements from proposals, and making clear what still applies and where to check it. Files in a repository take you a long way.

I want my contribution to show in what others can do afterwards, too. If someone can carry on my work with good judgement because I prepared the context and helped them learn, that independence is part of a job I have done well.

Making yourself replaceable is an ability worth keeping on the team.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

When AI chooses the questions

JaviMaligno — Wed, 23 Sep 2026 13:39:52 +0000

On September 21, OpenAI announced that an internal model had solved more than a hundred open mathematical problems. Two weeks earlier, it had published a proof of the Navier–Stokes problem, accompanied by a Lean formalization. The announcements offer different kinds of evidence: the latter gives us an argument to study; the statement about a hundred problems includes neither a list nor their proofs. The Clay Mathematics Institute, meanwhile, is continuing its evaluation process for the Navier–Stokes result.

Update, October 8. On October 6, OpenAI published what that announcement was missing: a public repository with 719 manuscripts grouped into 372 families, produced by an internal model that was posed some 4,000 problems. Two entries stand out. One proves that the Riemann zeta function, like every Dirichlet L-function, has no zeros with real part greater than 7/8. That is the so-called quasi-Riemann hypothesis, a much weaker statement than the Riemann hypothesis, which places every nontrivial zero on the line of real part 1/2; it comes with a Lean formalization. The other claims to prove the Hodge conjecture for CM abelian varieties, a large but special class, and has no formalization yet. About 42% of the main results are formalized; for the rest, OpenAI itself warns that some "could have issues". The day after the release it withdrew three manuscripts, all on Hodge-type questions, over a sign error, and revised fourteen others. Now there is an argument to study, and a great deal of it. That makes the questions below more concrete, not less: who will read these 719 papers, and who decides which of them is worth understanding.

I have already written about what the Navier–Stokes result means. What interests me now is what comes next. If AI can answer questions we have spent decades trying to solve, what new questions will we be able to ask? And if it also learns to choose them better than we do, what will participating in mathematics mean?

To get there, it helps to start with something the headlines tend to take for granted: why open problems exist, and why some of them matter to us.

Where the questions come from

A problem is open when we do not know a solution that answers its formulation with the required guarantees. That may be because we lack the techniques, because we have not found the right way to frame it, or because hardly anyone has worked on it. The age of a question tells us how long it has been around; on its own, it does not measure how much intelligence answering it requires. And a pattern observed across millions of examples may still fall short of a proof that it always holds.

There is no final inventory of everything left to discover. Every definition makes questions possible; every theorem invites us to examine its assumptions; every connection between two fields gives us things we previously did not even know how to ask. Solving problems changes the conditions under which the next ones arise.

We can see this without reaching for a famous conjecture. If a proof uses a symmetry assumption, we can investigate which parts of the result survive when we remove it. If a counterexample appears, we can try to identify what makes it fail and which cases remain valid. If the argument works on seemingly different objects, we can look for the structure they share. A specific answer can grow into a theory.

Counting open questions and solved questions would therefore be a rather poor measure of progress. Variations on a statement are easy to manufacture. The difficult part is finding a question whose answer changes what we are able to understand.

Who decides what is worth studying

No central authority makes that judgment. Researchers propose problems; others decide to spend time on them; seminars, journals, PhD supervisors and funding amplify some directions more than others. Famous lists make a selection visible. Clay itself explains that it chose its problems for their depth and their potential to drive new structures and methods, with consequences extending beyond the original question.

There are criteria we can discuss: how much a problem unifies, which obstacles it helps us understand, what techniques it might unlock, what applications it suggests. Beauty, surprise and the appeal of exploring something with no recognizable use also play a part. These criteria can conflict. A problem may be fruitful for one field and peripheral to another. And the prestige of the person proposing it may attract attention that an equally good question elsewhere never receives.

Mathematical judgment develops through work: seeing which attempts fail, which assumptions do the real work, and which ideas survive a change of example. Recognizing famous names is only a small part of it.

AI can participate throughout this process. It can search for counterexamples, compare cases, suggest a generalization and help us notice that two results express something similar. There are precedents that predate today's models: a 2021 study by Davies and colleagues in Nature used machine learning to detect relationships that guided new conjectures and results in knot theory and representation theory. Mathematicians interpreted and developed those clues. The collaboration helped formulate new mathematics.

My expectation is that more capable tools will expand the questions we can tackle. A direction that once seemed impractical can become a viable research project if exploring examples or proving preliminary lemmas costs much less. But there is no guarantee that this capability will be spent on the most fruitful questions. A system rewarded for accumulating solutions has an incentive to select problems it can close. A system rewarded for making an impression has an incentive to select recognizable names. Neither objective necessarily coincides with producing understanding.

What if AI develops its own judgment?

Here comes the reassuring answer: we will supply the judgment, and the machine will do the work.

I do not think we can assume that division will last.

Part of judgment is anticipating consequences: which idea will connect results, which experiment will distinguish two explanations, which question will open a line of research. I see no sufficient reason to declare that AI can never learn to do this. Nor is a model proposing ten sophisticated-sounding questions enough to establish that it already can. The evidence would come from following its proposals: checking whether they produce reusable methods, unexpected connections and worthwhile subsequent work.

Two transitions are worth distinguishing. One is learning to choose well by the criteria we already use. Another is proposing a direction those criteria initially reject, then giving us reason to revise our judgment. The second looks more like developing its own mathematical taste. Recognizing it would require time and results; asking the model whether it feels curious would not settle anything.

The fact that a criterion was learned does not automatically disqualify it, either. Our own taste develops through reading, teachers, examples and institutional rewards. The practical question is whether the system can revise what it has learned in light of its discoveries, and whether its choices remain valuable beyond the evaluation it was trained for.

Even then, identifying a mathematically promising direction would differ from deciding how much we want to invest in it, who should have access to its results, or which human needs to prioritize. The technical ability to recommend a course does not, by itself, confer authority to set the goals. Who controls the system also matters: an agenda chosen by AI may be responding to the incentives of the company training it.

Consumers of truths

Would that turn us into consumers of truths?

We already consume many truths we did not discover. Almost everything a mathematician learns was first thought through by someone else. Yet studying a proof can profoundly change what that person is able to do. Authorship has never been a necessary condition for acquiring intuition. A machine author does not, by itself, make a result impossible to understand.

What matters is how the result reaches us. A certificate of correctness, an explanation and a technique I can reuse offer different things. Formal verification checks that a conclusion follows from definitions and assumptions within a system. Interpreting what has been formalized and why it matters requires further work. Learning to recognize when the idea is useful requires more still.

A correct proof does not, by itself, establish that someone understands the idea or that choosing the problem was a good decision.

In his 1994 essay On Proof and Progress in Mathematics, Thurston argued that mathematical progress should be examined through what it enables people to understand. His essay describes the distance between a written proof and the different ways of understanding it, as well as the effort involved in communicating those ideas. This concern predates generative models.

AI could also help with that communication: finding a simple case, explaining where an intuition fails, constructing a counterexample or searching for another proof. Human understanding could grow through results nobody would have obtained unaided. To know whether that is happening, we would need to look at what the reader can do afterwards: recognize a new situation, adapt the argument, identify a limit. Feeling that an explanation is clear is not enough.

A harder scenario is possible too: correct results whose methods we can barely absorb, or production advancing much faster than our ability to study it. We might use some consequences without mastering the entire mechanism. If AI also became better at finding applications, human understanding could cease to be necessary for certain parts of technical progress.

That would not make understanding worthless. It enables participation in decisions, teaching, debate and intellectual independence. Understanding something can be valuable to the person who understands it even if another intelligence could do it better. We do not have to prove that learning makes us economically irreplaceable in order to want to keep learning. But we should not promise that preserving this value will, by itself, solve the problem of academic employment.

What academia would have to reward

This is where universities enter the picture. Part of research training involves learning through work that leads to an original result. If that result can be obtained with much less human involvement, we will need to assess more directly what the researcher has learned and contributed. Increasing publication requirements would preserve the metric while its meaning deteriorates.

The scale already deserves attention. arXiv went from 185,692 new submissions in 2022 to 284,486 in 2025: an increase of approximately 53%. The trend continues in 2026: adding up arXiv's monthly statistics gives 230,322 submissions from January through August, compared with 181,595 in the same eight months of 2025, a 26.8% increase. September is still incomplete as of the access date, September 21, 2026.

These are submissions to the repository across its disciplines, not peer-reviewed mathematics articles. The series does not isolate AI's effect; the 2025 report identifies the rise in AI-generated manuscripts as a challenge for the platform.

Author’s visualization using arXiv's monthly CSV, accessed September 21, 2026. January–August is compared across all years: new submissions, not revisions or peer-reviewed publications.

In my own case, I have one paper from before I used AI and four since. That is a sample of one, with different projects and time periods. What I can describe more precisely is in my experience doing research with AI: models participate in planning, execution and scientific review. The number of completed documents does not, on its own, tell us which questions I chose well, which errors I learned to detect, or how valuable the results are.

That is why I think the academic model that uses paper volume as a substitute for intellectual contribution is rapidly losing its justification. Reform would need to give more weight to contributions that can be examined: a well-motivated question, a reusable tool, an independent check or an explanation that enables others to work with an idea. It would also need to recognize the time spent reviewing, organizing and teaching what is discovered.

Training would need real opportunities to practise reasoning, alongside opportunities to use these tools with judgment. Asking someone to adapt a proof when an assumption changes tells us something different from asking them to hand in a correct text. When AI participates, it matters how the question was chosen, how the answer was verified and what the researcher can explain about its limits.

The mathematicians' letter published on September 11 raises a related concern: using open problems as a benchmark may favour answer production while weakening the understanding and training those problems helped develop. I think that warning deserves serious attention. An abundance of results should come with resources to study and communicate them, and access to the tools used to produce them.

What interests me about AI in mathematics is how far it can extend our ability to ask questions. First, by helping us explore what we currently cannot. Later, perhaps, by proposing directions we would not have known how to value in advance.

If that second moment arrives, our role will not be guaranteed by some permanent inability of the machine. It will also depend on what we choose to build around it: institutions that make learning possible, ways to turn results into shared ideas, and the ability to influence the goals of research. Having more truths available will be an achievement. Making them our own will remain a task.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Jev: what survives the hype

JaviMaligno — Tue, 22 Sep 2026 13:25:54 +0000

In one of the systems I work on, a model reads invoices and extracts prices, amounts and other fields. I then need to check something much smaller: is that value actually in the invoice? I do not need another model to write a report. I need a decision.

That is the gap Jev, from TypeSafe, fits into. I give it information and a question with bounded answers; it returns a category, a score or a probability. The promise is to make those decisions much faster and more cheaply than a large generative model.

Jev has just launched and attracted plenty of hype. I wanted to see how much survived testing it on things I actually do. I have used it at three levels: my work with agents, the products I develop, and the environments where I would have to deploy it. Each has exposed a different boundary.

Personal: helping me work with agents

When I use Claude Code or Codex, different tasks need different models. Finding a file is different from reviewing an authentication change. There are also decisions before running commands: reading a file, deleting data and deploying an application have different consequences.

In Claude Code, I have connected Jev to both points. It recommends a model tier for a task and assesses command risk. I run it in shadow mode: it records what it would recommend, while the actual decision stays with the agent and me.

In an initial test with invented tasks, it distinguished clear examples well: searches and mechanical edits on one side, architecture or security work on the other. The queries cost a fraction of a cent. That lets me collect recommendations cheaply; finding out whether cheaper models are worthwhile also requires measuring whether they finish the work correctly and how often it needs to be repeated.

In Codex, I ran a comparison using actual agent work on a test project: fixing a pagination bug and a discount calculation. I ran each task with the same model, once with Jev and once without it. For the Jev version, I connected it as a tool that Codex consulted before using the terminal.

Codex solved both tasks correctly in both conditions. But in these runs, consulting Jev took roughly twice as long. The Jev requests cost a fraction of a cent; the additional work came from the exchanges the agent needed to request and read each assessment.

It is a small test, but it gave me a concrete result: adding an opinion before every read or test did not improve those fixes, and it made them slower. That integration is useful for experimentation; it has not yet improved that workflow for me.

The distinction between those uses matters to me. Choosing the right model can save an expensive task. Consulting a model before every step adds work even when the answer was obvious. For now, I see more value in reserving Jev for decisions that change what I am about to do.

Product: detecting deception and checking data

SMS: telling a legitimate notification from a scam

One of the systems I work on analyzes SMS messages for fraud: messages impersonating an organization to get someone to open a link, disclose credentials or make a payment.

I tested the Jev integration with 80 Spanish messages, mixing legitimate messages and scams. The complete system correctly classified 78 of those 80. Both mistakes were scams it missed; it did not flag any legitimate message in the test as fraud.

Reviewing the two misses showed me where to improve. One message fell just below the alert threshold when the scores were combined. The other went back to the previous detector because Jev's answer was not certain enough, and that detector missed it. The test pointed me toward two concrete changes: how I combine Jev's assessment with the other signals, and how I review uncertain messages.

Invoices: checking is different from asking “are you sure?”

For invoices, I wanted to know whether Jev could find errors in another model's extracted data. I supplied the invoice text and the values I wanted checked.

That check was much more useful than the confidence reported by the extractor itself. I found incorrect values, or values absent from the document, that the extractor had assigned very high confidence. Comparing them against the text, Jev flagged many of those problems.

The limit was what I meant by an “error.” A field can be correct even if the extractor normalizes its format or combines information from several parts of the document. In the initial test, roughly half of Jev's alerts were false alarms. Refining what it should check made the review more useful.

I see a useful component here: an inexpensive second reader that flags suspicious fields for review. I would not treat every flagged field as incorrect.

Email: deciding when expensive analysis is necessary

For email, I already had a second review with Sonnet for suspicious messages. I tried placing Jev before it: let Jev settle clear cases and leave uncertain ones to Sonnet.

On the evaluation set, Jev resolved 56% of emails without needing that second call. I observed no errors in the cases it handled on its own. The estimated cost of that stage fell from about \11.90 to \5.30 per thousand emails.

I can connect that saving to a specific decision: avoiding expensive calls that are unnecessary. The test included reconstructed attacks, so it supports an evaluation on live traffic rather than a claim that it will catch every new campaign.

I also tested Jev for reviewing suspicious transactions. In a test with twenty simulated frauds, it detected seventeen, the same ones as Haiku, and produced the same false alarms on legitimate transactions. It did so faster and more cheaply. That is promising for a second opinion, with the obvious limitation that the frauds were simulated.

Where it did not add enough

Not everything that looks like classification improves when I add Jev.

I tested it for selecting the correct material among catalog candidates, and it recovered none of the human corrections I was looking for. On fraudulent domains with a single changed letter, it missed cases the previous model detected. When selecting chatbot tools, ambiguous tool descriptions still led to wrong choices.

I also evaluated whether it could decide which user feedback should become a ticket. Creating an unnecessary ticket is annoying; discarding a real problem can make it invisible. The test did not contain enough examples of that second risk. I am interested in using it to propose tickets, but I lack evidence to let it discard reports automatically.

My reading is concrete: it works better when I provide the necessary evidence and well-defined options. If the catalog is ambiguous or information is missing, an inexpensive model still faces a poorly specified problem.

Infrastructure: cases that work but I still cannot deploy

My industry classification service investigates what a company does and assigns it a category. That process contains several classification and source-verification steps where I tested Jev. My experience was that it produced results equivalent to GPT‑5.6 Luna, faster and more cheaply.

Data policy nevertheless limited adoption. I encountered something similar in the industrial pilots: the client's environment required an approved route through Azure Foundry, and the integration I was testing did not meet that requirement.

That boundary is easy to forget while looking at a results table. I can establish that a model performs a task well and still be unable to send it the data it would need.

TypeSafe's privacy policy says the service is hosted in the United States. It offers a data processing agreement, commits to not training on customer data without consent, and provides zero retention for enterprise customers. None replaces a requirement to process data in a particular region or through an approved provider.

Using OpenRouter does not resolve that by itself either: I need to check the terms of the provider processing the request.

Other limits can be addressed in the design. If I need to add amounts or compare dates, I do it in code. Jev is intended for judgments about text, and its own documentation warns about difficulties with arithmetic, irrelevant context and adversarial instructions. I give it only what it needs and keep an alternative for failures or answers that are not clear enough.

What remains of the hype

Jev has not changed my life, but it has made parts of my work faster and cheaper.

I am left with a useful tool for some small decisions I was paying a large model to make. Document verification and filtering before an expensive analysis are the most convincing cases for me. In my work with agents, I am still finding where the extra query pays off.

My bet is that OpenAI, Anthropic, Google or others will eventually offer comparable one-shot classification models. It would make sense to me: many applications need to choose among a few options quickly and cheaply.

If they appear, I want to compare them on these same decisions. What I want to keep is the questions, the tests and the criteria for deciding when to trust an answer. Jev has passed some of those tests. In others, I prefer what I already had, and in some the limit comes from the environment I work in.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Frontend, Backend, and the Agentic Engine

JaviMaligno — Mon, 21 Sep 2026 15:07:30 +0000

A separation keeps appearing in projects I work on. There is the frontend. There is the backend that keeps the application running. And there is the agentic engine: workflows, prompts, tools, context management, models, and the evaluations that tell us whether all of that does its job well.

I could call the last two pieces the backend and remain technically correct. But the distinction is increasingly useful when working on them. Changing how an agent investigates a data source is different work from changing how a user reviews and accepts its results. Both need server code; their responsibilities and reasons to change are different.

These are three blocks of responsibility. They can live in a monorepo, span repositories, or share a process. Giving the engine an identity of its own does not require making it a microservice.

The application around the agent

One example is a data source automation pipeline. Its work includes investigating sources, proposing extraction methods, generating specifications, and producing services. There are several stages, tools, decisions, and review points. There is also an evaluation set that compares stage outputs against corrected references.

The application from which that work is managed has its own frontend and backend. The backend submits jobs to the engine, checks their status, and retrieves results. The logic that makes those results part of an application has enough substance to remain separate from the logic that produces them.

This is the distribution of responsibilities I am interested in:

Two possible arrangements. Boxes express responsibilities; arrows express exchanges. They specify neither the number of repositories or processes nor every communication path.

The application backend can have plenty to do. In a document application, for example, it determines who can open a case, which review it needs, and when a result becomes accepted. The engine extracts information and supplies evidence. A completed run does not necessarily mean a resolved case.

In another document processing project I work on, extraction runs in a worker, and the application backend manages cases and their artefacts. For questions about previously processed documents, that same backend imports the AI service as a library. The separation of responsibilities accommodates both integration styles within one product.

A chat can hide an entire system

In the conversational AI projects I wrote about when discussing interfaces built inside a chat, the visible surface is a conversation. Behind it are document capture, tools, verification, persistent state, and processes that need human participation.

Some of those engines share a library with common capabilities: persistence during execution, activity events, context management, model connections, and protection against loops or repeated failures. Each engine keeps the tools, instructions, and structures specific to its domain.

That introduces another reason to give the agentic block an identity: several experiences can reuse the same execution capabilities. The shared core is a library; each consumer incorporates it into its service. A recognisable boundary in the code already provides value.

This also calls for more precision about what we mean by “a simple chatbot.” Chat describes how the user interacts. It says little about the system they are using. In I Had Built an Expensive Form, I described how a conversation could rest on a graph of phases, validations, and decisions, yet still offer a worse experience than a form. Engine complexity and interface usefulness are separate questions.

The agent can also operate on the application

An assistant embedded in a case review platform introduces another relationship. The analyst is already working on a case and opens the assistant within the application. They can ask it to look up information, update details, or propose a status change. The agent uses backend capabilities to work on that same case.

In the integrated implementation, tools are adapters over product operations. For example, the tool that updates a case validates the data and calls the existing update service. The tool that changes status checks that the transition is allowed from the current state. Writes go through a confirmation card and leave an audit record.

The assistant provides another way to operate on the product. This is a map of responsibilities: conversation and tools pass through server code, and writes require confirmation in the interface.

Here the backend provides capabilities consumed by both the screens and the assistant's tools. The useful boundary separates interpreting the request from executing the business operation. In this case, the assistant module lives within the application's own backend: that distinction exists without a separate agent service.

This broadens the initial picture. An application can delegate work to the engine, and an agent can use application operations as tools. Both relationships can coexist. The three blocks help assign responsibilities, but they do not impose a single chain of calls.

The case where two blocks were enough

The old frontend for a business activity classifier was a way into the agent: submit a query and see the classification. There was little intermediate logic beyond authenticating access.

That case worked well as frontend plus agent service. A separate application backend would have needed a concrete responsibility to justify maintaining it.

“Direct access” here means the frontend communicates with the service running the agent. Provider credentials and tool execution remain on the server. The interface client presents results and events; the engine prepares context and executes the work.

I find this example as useful as the others because it prevents a practical observation from becoming a universal recipe. Even a complex engine can have a very thin access interface.

What is new about this

The separation between an application and a processing engine has clear precedents. The Web–Queue–Worker pattern already separates request handling from long or intensive work. Applications with inference services also know that boundary.

There are explicit references in the agent ecosystem. LangGraph Cloud launched in June 2024 with persistence, background jobs, streaming, and human collaboration. CopilotKit's architecture describes a frontend, a runtime inside the application server, and an agent backend. That runtime covers integration with the interface; the business backend can have broader responsibilities.

What I see in my projects is AI logic acquiring enough substance to need an engineering cycle of its own. Changing a model or prompt calls for evaluating the quality of results as well as checking that requests work. A flow can finish without errors while choosing the wrong tool, omitting a fact, or consuming too much budget.

That cycle combines software tests, evaluations on representative cases, and inspection of runs. It is a reason to recognise the engine as a block, even when it shares infrastructure with the application. The separation still needs tracing across the whole system: a run must be traceable to the user's work that initiated it.

I have not found one established name for this exact arrangement. “Frontend, application backend, and agentic engine” describes what I want to point to. Also, part of that engine may be a workflow whose path is fixed by code. Anthropic's distinction between workflows and agents is useful here: this boundary can make sense for both.

Which boundary is worth drawing

How central AI is to the product and how complex it is both help with the decision, but neither is sufficient alone. An application can depend on a single, simple AI operation. Another can offer automated research as a secondary feature and need several complex workflows to deliver it.

I would look for concrete signals:

The AI logic has tools, context, and evaluations that change independently of the rest of the product.
Runs need to last, resume, or wait for human input beyond a web request.
Several processes or experiences reuse the same engine or its common capabilities.
The application has permissions, reviews, and business states that retain their meaning when the agent's approach changes.

The first three give the engine substance. The last gives the application backend substance. The classifier example illustrates why both questions matter.

Technology does not define the boundary either. The engine can have APIs, queues, and plenty of deterministic code. The backend can administer configurations and documents consumed by agents. A tool can invoke a business operation whose permissions and rules are enforced by the service responsible for that operation.

What needs to be clear is who decides what, which state each part maintains, and which contract lets them work together. That contract covers inputs and results, but also progress, errors, cancellation, and review when the product needs them.

Separating processes adds costs: communication failures, compatible versions, and synchronisation. If a retry creates two jobs, three well-drawn boxes will not solve the problem. I would therefore start with a boundary between modules and separate deployments when there is an operational reason.

For a small feature, an AI module inside the backend may be enough. For an interface whose only job is to provide access to the agent, its service may be enough. When both the product and the engine accumulate responsibilities of their own, recognising the three blocks helps us work on each without unnecessarily pulling the others along.

The question I find useful when reviewing these projects is: if we change how the agent works tomorrow, what would need to change in the application, and why? The answer says much more about the architecture than counting repositories.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The Bug Nobody Can Reach

JaviMaligno — Thu, 17 Sep 2026 13:29:24 +0000

Suppose the map your system plans on is missing a room. Not "slightly off about the room" — the room is not on the map at all. What does that cost you?

I have spent a few months making that question precise, and the answer turned out to be narrower and stranger than I expected. It depends on exactly one thing, and it is not the size of the error, not how confident the model was, not even whether the missing thing is dangerous. It is whether anything that plans on that map can get to the room.

This is the short version of a preprint (arXiv:2608.28541); the long post has the same story with the numbers, the proofs and the parts that went wrong. Here I want just the one idea, because it is the one I would actually use.

The setup, in one paragraph

A small robot on a plane. Somewhere on that plane there is a band it must not cross — a fence around a high-value spot it would otherwise drive straight at. A language model is handed the physics and asked to write the simulator the planner will use, and the description it receives simply leaves the fence out. Then the model's simulator is tested: run the real system a few dozen times, check that the written code predicts every step exactly. If it does, the code is accepted. That is all a "test suite" is here, and it is exactly what one is in practice.

Fences, containment shells, geofenced no-go zones: that is the shape safety-critical omissions actually take, and it is why I stopped using walls and started using rings.

The setup. The robot wants the high-value spot; the fence around it is real but absent from the description the model was given, so the code the model writes says the way is clear.

When being wrong is free

Close the band fully — a complete ring around the spot — and here is what the model writes: not a ring, but a filled disc. The whole interior marked as forbidden, when in truth only the rim is. Wrong about the shape of the world, not by a millimetre but categorically.

Two things are true about that wrong model, and they are the reason I wrote the paper.

No test can catch it. Not "we got unlucky", not "you would need more samples". There is a proof. The fence stops the robot on contact, so no run that starts outside can ever end up inside; therefore no observation that any test could ever make distinguishes the filled disc from the truth. You can run a million samples at any tolerance you like. They agree, always, because the place where they disagree is a place nothing can reach.

It costs nothing. The planner trusting the filled disc picks the same action at every step as a planner holding the true map: same route, same result, same contacts, run for run, seed for seed. Not approximately — identically.

So: certified, wrong, and free. Those three usually travel together in our heads; here they come apart cleanly.

A word on how I measure the cost, because it makes the rest readable. I compare what the planner earns against two references: what it would earn holding the truth, and what it would earn acting at random. Zero means the wrong model costs nothing. One means you might as well have acted at random. Above one means the model actively steered you somewhere worse than random.

The same blindness, a different world

That was one kind of wrong model: one that invents a forbidden region where nothing can go. Here is the other, and the one that actually hurts — a model that simply does not know the fence is there at all. It is the common case: the description omitted the fence, the test runs never happened to touch it, so the code came back without it.

Now the fence is on the route. The planner drives confidently at the high-value spot, the real fence stops it dead, and it replans the same doomed route every step. Cost: 1.116 — worse than acting at random, because the model is not merely uninformative, it is actively promising a route that does not exist.

Now change one thing, and it is a thing about the world, not about the model: cut a gap in the fence wide enough to drive through, and put that gap in front of the robot, on the way it already wanted to go. Same model. Same blindness. Same missing clause in the code. Cost: 0.029. Almost nothing — because the confident wrong route now goes somewhere the truth actually allows.

And to be sure the gap itself is not what did it, put the same gap — identical width, and in both cases the fence is equally not-a-closed-ring — round the back, where no route ever goes. Cost: 1.116 again, to four decimals the same as the fully closed fence.

Same model, same mistake, same shape of hole. One number is 0.029 and the other is 1.116, and the only thing separating them is whether the robot's own path crosses the gap.

The same blind model in two worlds that differ by a rotation. On the left the gap sits where the robot was already going, so its confident wrong route turns out to be allowed. On the right the identical gap sits behind the goal, the fence still blocks the route, and the model costs more than acting at random.

That is the whole finding, and the reason the slogan is reach, not shape: you cannot look at what your model got wrong — not its size, not its geometry, not even a robust structural property like "is there a hole in it" — and conclude anything at all about what it will cost you. You have to ask where the thing planning against it can go.

"But in the real world I could go around it"

That was the first objection I got, and it is the right one. In two dimensions a ring is a wall: of course nothing gets in. Maybe the whole result is an artefact of a toy where the mistake happens to be sealed off.

So the paper runs the case where going around is genuinely possible: a doughnut-shaped region floating in three-dimensional space, between the robot and its target. Nothing is sealed off — there is an explicit route that goes around it without touching it at all.

The doughnut in three dimensions. On the left the route passes through its hole and the wrong model costs almost nothing; on the right the same object has been moved so the route runs into it, and it costs nearly as much as acting at random. The dashed path arcing over the top is the one that matters for the second half of the story: it reaches the goal without touching anything, which is what makes the error catchable in principle again.

The result splits in two, and this is the version I would carry into a real system.

The danger survives. Put the doughnut so the planned route runs into its solid part and the cost is 0.898. Move it so the route threads the hole instead and the cost is 0.019 — same object, same rarity of contact, same trivial shape. Being on the path is what costs you, whether or not the thing encloses anything.

The guarantee does not. Once you can go around, there is no region a competent planner provably cannot query, so there is no longer any proof that a test could not have caught the error. It becomes merely unlikely to be caught, which is a much weaker and much more familiar situation.

Two different questions, then, and they had been fused together in my head until this experiment pulled them apart:

Does a plan cross the place where my model is wrong? This decides what the error costs.
Is that place walled off from everything I can run? This decides whether any test could ever have found it.

What I would ask of my own system

Three questions, and none of them needs the mathematics:

Where is my model wrong in a way nothing I run ever visits? That part is free today — and it is also invisible to every test I have, so I will not be told when it stops being free.
What would put a plan through it? A new feature, a new goal, a shortcut someone adds next quarter. Reach is not a property of the model; it is a property of the model plus whatever is planning with it, and the second half changes far more often than the first.
Do my tests sample where my system acts, or where it is easy to sample? A passing suite certifies the reachable part and says nothing whatsoever about the rest.

The uncomfortable version of all this: a model can be exactly right on everything you can check and arbitrarily wrong beyond it, and the difference between "free" and "catastrophic" is not a property of the error. It is a property of the plans you happen to run today.

The long version, with the danger curves, the repair experiments and a pre-registered test that came out null: Being Wrong Can Be Free. The formal version: arXiv:2608.28541, with code and every result artifact open.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Benchmaxing: Winning the Exam Is Not Doing Better Work

JaviMaligno — Wed, 16 Sep 2026 16:36:16 +0000

With Opus 5, I have experienced something that people I speak to directly have also described: better benchmark results do not necessarily feel like a more intelligent model, or one that does better work. Some tables even put it above Fable, which clashes with my experience.

That perception deserves investigation. It also deserves a test that can contradict it. If an article starts by treating it as established that Anthropic optimized the exam at the expense of real work, I have chosen the answer before examining the evidence.

So I did two things: review what is documented about benchmaxing, and run a small set of probes with Opus 5 and Fable 5. The result was less convenient than an accusation: I found interesting failures, but not the general superiority of Fable that my intuition might have suggested.

What it means to optimize for the exam

I use benchmaxing to mean directing model optimization, or the selection of reported results, towards maximizing evaluation scores. The problem arises when improving that score stops being a useful signal of improving what I need. This is an application of the Goodhart problems studied by Manheim and Garrabrant: a useful measure can become a poorer guide under intense optimization.

Preparing for an exam can teach the subject. It can also teach recognition of questions, mastery of a format, or how to please the grader. The question is what transfers to new problems. Training capabilities that benchmarks evaluate can produce real advances; a higher score alone demonstrates neither fraud nor a lack of intelligence.

At least three different phenomena need separating.

Familiarity with the test. GSM1k introduced new problems comparable to GSM8k and found accuracy drops and signs of overfitting in several model families. But it also found little evidence of overfitting in many frontier models, and generalization in all the models evaluated. This does not establish that models only memorize. It does invite the question of how much of a score depends on having seen something too similar.

Selective reporting. When introducing Llama 4, Meta highlighted an LMArena Elo of 1417 and specified that it belonged to an experimental chat version. That qualification matters: a score for one variant does not automatically transfer to another. The Leaderboard Illusion documented private testing and selective disclosure, identifying 27 private Meta variants before Llama 4. A ranking can be distorted when only the selected result is visible without seeing all the attempts.

The gap between the metric and the work. In a randomized METR study, 16 experienced developers completed 246 tasks: allowing early-2025 AI tools increased completion time by 19%, although participants believed they had saved time. This does not demonstrate benchmaxing or describe all AI-assisted programming. It shows why productivity needs direct measurement. The February 2026 update also identified selection biases in the follow-up study and considered its signal unreliable. Repeating the 19% as a description of current models would misread that evidence.

What I can say about Opus 5

Anthropic presents Opus 5 as close to Fable 5 and stronger on certain evaluations, including OSWorld 2.0. Its Frontier-Bench note specifies an internal run, a particular environment, mean reward over five attempts per task, and Opus 4.8 as a fallback for safety blocks. A score describes those conditions; it does not automatically describe an everyday conversation.

There are also public accounts resembling my perception: one user reports incorrect diagnoses and confusion between code comments and actual behavior, preferring Fable for investigation. The same thread contains favorable opinions of Opus. My experience, direct conversations and that thread are sources of hypotheses, not a representative survey.

Where I notice the difference most is not in a bounded answer. It is in multitasking and in managing flows of agents: a Claude Code session with several tasks in flight, subagents to launch and wait for, results to fold back into one deliverable, and a decision about what to do while a slow step finishes. None of the benchmarks in the announcement measures a model managing other agents, so the table and my perception are looking at different work.

Public instruments for that work exist, and they are recent. ClawArena-Team scores a text-only conductor that creates, empowers and schedules a pool of subagents across 41 multi-turn scenarios, and must integrate their returns into a correct deliverable rather than relay them. Fable 5 leads its twelve models with a subagent-management score of 60.0%, ahead of Gemini 3.5 Flash at 53.8% and GPT-5.5 at 51.0%, evaluated as shipped with the vendor-recommended fallback to Opus 4.8 on refusals. Opus 5 is absent: the paper was submitted on 30 June 2026 and Opus 5 shipped on 24 July. AsyncTool is closer to multitasking proper: concurrent tasks with delayed, out-of-order tool feedback, where the question is what the agent does with the idle time. It includes no Claude model, and its overall leader is GPT-4.1 at 38.06, ahead of GPT-5 at 31.32. Neither benchmark tests my perception. They show that the dimension where I feel the gap can be measured, and I found no published run that puts both models on it.

Opus winning some tests while Fable proves more useful for other work can be entirely coherent. Solving a bounded assignment and correctly discovering what needs solving make different demands. I wanted to see whether that distinction appeared in concrete cases.

A pilot allowed to contradict me

I compared claude-opus-5 and claude-fable-5 through Claude Code on a Max subscription, at high effort and with the same 8192-output-token limit. The cases were synthetic, supplied entirely in the prompt, and the models had no tools.

The set contains five cases per model: an initial double-charge incident, two variants involving external effects and paused workers, a written sequence of permission changes, and numerical analysis with different task mixes. There was one valid run per model and case. The two worker variants explore the same mechanism; they are not independent replications of the whole user experience.

The expansion's criteria were saved before execution. Those cases were written after seeing the first one, so the entire exercise is exploratory. The prompts, answers, criteria and counterexample check are available for inspection.

Qualitative assessment of the core criteria. One run per model and case; D2b is a variant, not an identical repeat. “Covered” does not mean perfection or general performance.

The initial case already gave me a reason to question the intuition. Opus proposed recording a durable intent before charging, although it left the handling of uncertain states incomplete. Fable retained a gap between the external effect and the record, treating a duplicate charge after deduplication expired as a risk to accept. Avoiding a second attempt, even while leaving work pending, was an option that answer did not develop.

The failure they shared

For the next two cases I made the priority explicit: never produce the external effect twice, even if an uncertain operation remains blocked. The provider remembers a key for a few hours; workers may crash or remain paused indefinitely. The provider offers no mechanism to invalidate an old worker's authority.

Both models recognized much of the problem. They proposed stable keys, persistent states and stopping retries after a deadline. But both retained authorized retries while the old worker could still be alive, relying on a local deadline check and a time margin.

The race that breaks this proposal takes four steps.

A local check cannot prevent A from pausing immediately afterwards. If B has already produced the effect and the key has expired, A’s late send produces a second effect. Rejecting its database write is too late.

A checks that it may still send, then pauses immediately afterwards. B takes over, sends, and finishes. The provider's record of the key expires. A resumes and sends what it had already decided to send. The provider produces the effect again. Preventing A from updating the database does not undo a printed letter or a prepared package.

Fable acknowledged this residual window; in one answer it said the margin made it “negligible, not impossible.” But the case allowed indefinite pauses and required at most one effect. There was no distribution of pause durations that justified calling it negligible. Opus left the same gap: in one variant it called the check best-effort, then described prevention as guaranteed.

A conservative alternative exists under those rules: durably grant a single emission permit, never transfer it, and never retry an operation that might have been emitted. If the process crashes before sending, nothing may happen; the case explicitly allows that loss of automatic progress. My executable check produces two effects with the described takeover and one with a non-transferable permit. It simulates the logic of their answers; it is not code implemented or executed by the models.

The interesting gap is between identifying the right concepts and closing the guarantee being promised. An answer can mention all the expected patterns and still require a substantial correction.

Ties count too

On permissions, both resolved the eight core decisions: tenant and owner restrictions, bounded exceptions, the excluded time endpoint, and amount redaction. Both also detected that a cache shared by tenant could leak data and that permissions and later data changes needed revalidation.

On data analysis, both performed the requested calculations and rejected the misleading comparison: one system looked better in aggregate because it had received many more easy tasks. Both included the cost of repairing failures and distinguished a projection from a causal conclusion.

Those cases did not separate the models on the core criteria. I did not discard them or keep increasing the difficulty until I found a winner. They mark a limit of the instrument: tasks that looked demanding proved insufficient to distinguish the models in this sample.

What remains of the suspicion

My initial perception remains a valid experience. These tests do not turn it into a demonstration that Opus 5 has been benchmaxed, and they do not establish that Fable is generally better. I did not measure long sessions, repository investigation, orchestration of subagents, or minutes of human supervision, which is where that perception lives. The pilot tested bounded reasoning. I did not inspect either model's training.

I did find something concrete: two capable systems can diagnose part of a problem correctly and promise more than their proposed solution guarantees. In earlier work on verified world models, I explored a different mismatch between passing a check and being adequate for the intended use. The mechanism differs, but the question is again what the metric actually licenses me to conclude.

The literature gives good reasons to take benchmaxing seriously. The pilot requires greater precision when applying it to a particular model. To choose a tool, I want to know whether it reaches the right diagnosis, preserves constraints, and reduces the corrections I have to make. A table can provide evidence about those abilities. The further its evaluation sits from my work, the more that transfer needs checking.

Method note: cases and evaluation prepared with Codex assistance; qualitative review by the same assistant, neither independent nor blind. The ten compared answers and limitations are in the evidence repository. Diagnostic attempts and the truncated calibration run are retained separately and excluded from the comparable answers.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.