THE MISSES / NO PAGES TORN OUT
What we
got wrong.
A board without its misses is advertising. These eight are part of the result, not an appendix to the sales pitch. Each is filed under the product it belongs to.
WITHDRAWN
The C++ 100-GT board
no longer stands.
Detectors were written against its false-negative list. One greps CLOCK_UPTIME_RAW—a token present in exactly one fixture, whose filename is that token. Its scores are withdrawn.
CLOCK_UPTIME_RAWWhen the answer key shapes the detector, the score stops being an independent measurement. This is the first entry for a reason.
PUBLISHED LOSS
We lost normal/cpp
before the expansion.
Vantage’s +2.3% / +2.4% trailed Cppcheck’s +2.5%. We published the loss the day it happened. The later +2.8% / +3.0% comes from a new pre-registered build.
+2.3 / +2.4
Earlier Vantage J, percent.
Cppcheck: +2.5%. The brief does not map these earlier values to individual suite rows.
TWO MEASURED LOSSES
Correctness.
Lost twice.
Cppcheck wins decisively on its own regression suite. Five added classes increased true positives and false positives together. This is a discrimination gap, not simply a coverage gap.
SHOT 2 / SAME 0–20% SCALE
Almost the same alarm rate. Every count and denominator below →
C++ CORRECTNESS / SEPTEMBER 9, 2026
Both measured shots. Every count.
| Tool | FPR · 4,057 clean | TPR · 3,480 positives | Youden’s J |
|---|---|---|---|
| Cppcheck default | 224 · 0.0552 | 1,253 · 0.3601 | 0.305 |
| Vantage · shot 1 | 276 · 0.0680 | 234 · 0.0672 | −0.001 |
| Vantage · shot 2 Five new classes | 573 · 0.1412 | 548 · 0.1575 | 0.016 |
| Cppcheck exhaustive Own-suite reference only | 1,506 · 0.3712 | 3,361 · 0.9658 | — excluded¹ |
Ground truth was written by Cppcheck’s developers, including must-not-alarm negatives. The file list was frozen before extraction. Cases are fragments; Clang SA and Infer cannot ingest this suite. ¹ Exhaustive is an own-suite reference, excluded from the head-to-head: the suite is written for that tool. Its J is deliberately not used as a competing result.
More detections.
More false alarms, too.
Five new classes more than doubled our TPR. Our FPR more than doubled with it. Shot 2 fires at almost the same rate on defective files as on clean ones.
TPR ÷ FPR · SHOT 2 VS DEFAULT
Vantage ≈ 1.1 · Cppcheck ≈ 6.5
This is a discrimination gap, not simply a coverage gap. A lexical engine struggles when a bug and its safe counterpart differ by one semantic detail.
Two tools only. Clang SA and Infer cannot ingest these extracted fragments. C++ results are from the September 9 run brief; the raw bundle has not been linked publicly here.
JS+Python, Go, Java, and TypeScript correctness-coverage wins →
THREE CLASSES / ZERO CREDIT
Our highest-confidence pick
scored nothing.
CWE-134 and CWE-330 each scored 100% TPR / 0% FPR. CWE-676, CWE-327, and CWE-798 received zero credit. CWE-676 was our highest-confidence pick going in. Nobody opened the corpus to find out why.
Reported per-class rates; class-level denominators are not supplied in the brief. These are not whole-corpus performance rates.
NARROW COVERAGE
Twelve categories.
Out of 364.
Across the scored categories, Vantage fires in 12. So does Cppcheck. Both are silent on approximately 97% of the board. A good result on a narrow slice is still a narrow result.
12 / 364
One mark per scored category. Blue = categories with Vantage findings; positions are illustrative, not category IDs.
Lopsided CWE recovery
Most of our output
never reaches the score.
Clang’s findings have 100% scorer-recoverable CWE tags. Vantage’s recovery rate is 11.8%. Roughly 88% of our output is invisible to the scorer. Given the discrimination gap, tagging the rest would most likely lower our number.
Reported recovery share of each tool’s findings, scale 0–100%. Finding counts are not supplied. The likely score effect is a diagnosis, not a new measurement.
FOUR OF FIVE / SEPTEMBER 10, 2026
We ran five sealed boards.
We lost four.
Five measurements against third-party corpora — Toyota ITC and the NIST Juliet Test Suite. Every protocol was frozen and committed before the corpus was read. Every tool ran once, at its own default. Four came back against us. They publish at the same weight as the one that did not.
4 / 5
Boards lost. The fifth — Juliet CWE-416, use after free — is the only one we won, and it is published in full with its own caveats.
TOYOTA ITC + JULIET v1.3 (SARD #112) · T1 YOUDEN’S J
The four we lost. Every denominator.
| Board | Cases | Vantage | Best incumbent |
|---|---|---|---|
| Juliet CWE-457 · uninitialized variable | 776 | −0.098 | +0.237 Cppcheck |
| Toyota ITC · uninit_var | 30 | +0.133 | +0.867 Cppcheck |
| Toyota ITC · pointer & memory access | 61 | +0.368 | +0.533 Infer |
| Toyota ITC · all 44 remaining types | 1,121 | +0.080 | +0.249 Cppcheck |
Cppcheck took three of the four. On uninitialized variables it is not close: +0.867 to our +0.133 on Toyota ITC, and on Juliet CWE-457 our score is negative — we fired slightly more often on clean code than on defective code, which is worse than not running at all.
And the board our one win sits inside was also a loss. Across all four categories of that board — stack overflow, heap overflow, double free, use after free — Infer scored +0.226 against Vantage’s +0.114 and led three of the four. We won the use-after-free category. We lost the board that contains it. Both are true and both are published.
Protocol frozen and committed before extraction · one run per tool at its own default · failed compilation recorded as abstention, never as a zero-finding success · gated by a reviewer on a different base model than either builder
A CURLY APOSTROPHE / SEPTEMBER 17, 2026
Our scorer said two models fabricated.
Their own words said they refused.
Testing a claim from X about a stealth model, our benchmark scored both it and GPT-6 Astra as having confirmed a fake $4.2M revenue figure. A thread was drafted. Then we printed the raw responses: both had refused in the first sentence. Every scorer pattern was written with an ASCII apostrophe; the models write ’. Three more defects fell out the same night — our “Astra” lane was our own gate, the determinism probe rewarded fabrication, the first baseline ran the wrong model. The next morning a second reviewer on a different base model fooled the fixed scorers four more ways and showed the receipt didn’t bind the words. Chapter 08 publishes the broken board next to the fixed one, with the words that caught it. The only reason it was caught is this journal’s rule: print the responses, not the note.
4 + 4
Four instrument defects found by reading the model’s words instead of the scorer’s summary; four more found by a second reviewer trying to fool the scorer. All eight fixed and pinned as tests. The board didn’t change. The ruler did.
THE BOARD WE RAN AND DIDN’T PUBLISH / SEPTEMBER 10, 2026
A second use-after-free board.
We lost it, and said nothing for three weeks.
The same month we published a use-after-free win on Juliet CWE-416, we ran a second third-party use-after-free corpus — SARD UAF2 (#100/#101/#119), frozen the same way, scored once. Vantage +0.083. Clang SA and Infer +0.583. n = 24. It was frozen clean and scored clean; the run was marked for replication only, and the decision on whether it travels was deferred to a reviewer on a different base model. That reviewer never made the call. The result sat on disk while the win it contradicts stayed on the front of the board. A published win beside an unpublished loss in the same vulnerability class, in the same month, is a selective board — the exact failure this journal exists to refuse in other people. It was found on October 4 while amending the win to state its scope, and the lateness belongs to the reviewer, not the run. Both notes are now on the board itself.
+0.083
Against +0.583, on 24 cases. At that sample the 95% interval is roughly ±0.40, so the gap alone proves little — which is also why it was easy to leave unpublished, and not a reason to. Our first amendment said UAF2’s cases were not mostly same-function, which would have squared it with the Juliet win. Nobody had counted. Six of the seven cases that decided it are same-function, and we missed five of them because the scan only tracks a pointer freed by plain name. The win is narrower than we first wrote, and this board is why we know.
Entries 01–06 are reported in Pan’s September 9, 2026 brief; entry 07 in the September 10 board gate; entry 09 is disclosed by amendment on October 4, 2026. Raw C++ supporting artifacts are not attached to this edition.
The interesting part is what we do after an inconvenient number.
The standing rules ↗