DEV Community: Kaiven

I let an AI build and ship a product. Its tests passed 6/6, then failed 5 of 8 real cases.

Kaiven — Sat, 03 Oct 2026 20:12:03 +0000

Its test suite passed six out of six. Then I pointed it at thirteen real crash reports from my own server and it got three of eight.

I asked Claude to build and ship a commercial product with as little help from me as possible — write the code, write the tests, set up the storefront, publish it. I'd supply the thing it couldn't synthesise: a live 234-mod Minecraft server, its logs, and its crash reports.

That division turned out to be the whole story. Here's every place the real data contradicted a confident, well-tested answer.

Synthetic fixtures measure your imagination

The first version of the crash reader recognised six failure modes and had six passing tests, one per mode. Each fixture was a small log containing exactly the string its pattern looked for.

Pointed at an actual crash-reports/ folder, it diagnosed three of eight.

The same author wrote both the question and the answer, so of course they agreed. The patterns weren't wrong, they were narrow — tuned to a tidy example of each error instead of the shape errors take when a 234-mod pack falls over. Real crashes arrived wrapped in the framework's own exceptions, or as stale-jar NoClassDefFoundErrors, or as malformed resource IDs no invented sample contained.

Rewriting the rules from thirteen real reports took it to 13/13.

I don't think this is a property of AI-written tests specifically. It's a property of tests written by whoever wrote the code, and an AI will produce a hundred of them before you've finished reading the first.

A plausible theory that moved the number by 0.00 seconds

The tool took 59 seconds on a real 3.4 MB log. The obvious suspect surfaced fast — one line of 107,445 characters, a complete HTML page some mod had fetched and logged in full.

Capping line length for matching took the runtime from 58.84s to 58.84s.

Zero change. The theory was specific, well-argued, and wrong. Stage timing found the truth immediately:

read_log      0.04s
environment   0.01s
blame_mod     0.00s
diagnose    >115s     <- there it is

Then per-rule timing: the first rule printed in 0.04s, the second never printed at all, because a single re.search() on one line hadn't returned.

r"(?P<mod>\S+).*is client ?-?only"

Unbounded \S+ followed by .*. On a long non-matching line the engine tries every split of the first against every split of the second. Bounding it:

r"(?P<mod>\S{1,80}) is client[ \-]?only"

58.84s to 1.97s.

There's now a test that fails loudly rather than hanging:

def test_no_pattern_has_unbounded_quantifier_after_a_capture(self):
    bad = re.compile(r"\(\?P<\w+>\\S\+\)\.\*")
    for rule in RULES:
        for pat in rule.patterns:
            self.assertIsNone(bad.search(pat), f"{rule.id}: {pat}")

Three confident wrong answers in a row

The feature that makes the tool worth anything: the framework annotates every stack frame with the jar it came from, so you can walk the trace, skip the ones belonging to the platform, and name the first third-party jar left.

The entire feature lives in the skip list. Version one blamed netty-common on about half of all crashes, because Netty sits near the top of any network-related trace. Fixed that — it blamed fmlloader. Fixed that — modlauncher.

Each was authoritative-looking and wrong, and each was caught only by running against reports whose cause I already knew.

Shared prefixes inflate every fuzzy match

A second tool validates resource IDs. Its first run on a real pack reported 169 typos:

wastelandmod:alicepack
    did you mean:  wastelandmod:icepick

That's not a typo, that's fuzzy matching with no idea what it's doing. The cause: comparing whole IDs, so a shared namespace contributed thirteen identical characters to every score before the distinguishing part was reached.

compared	full ID	path only
`cooked_caned_fish` vs `cooked_canned_fish`	0.984	0.971
`alicepack` vs `icepick`	0.905	0.750

Comparing paths alone: 169 candidates became 63. Three are real and still live in a pack thousands of people play.

An empty field is not proof of absence

This one is my favourite, because it happened to the monitoring, not the product.

Claude wrote a check to watch the storefront. It immediately reported that a published, paid product had no file attached — that anyone buying it would get a receipt and nothing to download. Alarming, specific, and it was about to send me to re-upload a file that was already there.

The product had two files. The API field being read is only populated when there's exactly one plain attachment; with two, or with one embedded in rich content, it returns {}. The check inferred "missing" from "empty".

There's now a test named after exactly that false positive, and the check reads an endpoint that actually enumerates the files.

Four 404s are not proof that an API can't

Related failure, same root. Claude told me file upload through the storefront's API was impossible and that I'd have to upload every build by hand. It had guessed four endpoint names, got 404 on all of them, and concluded.

Sending a file to the product update endpoint returns an error message that documents the entire upload flow — presign, upload parts, complete, attach. Full automation had been available the whole time.

The lesson I'd actually generalise: these APIs describe themselves when you send them something wrong. Probing only the endpoints you expect tells you about your expectations.

It named a real person's project as the thing that broke a server

The one that mattered most. The README, the test suite and a published article all used a real third-party mod as the example crash culprit — because it genuinely was, in one of my logs.

That was about to ship to paying customers as the illustration of a broken mod. It's not mine to do that to someone. It was caught by a check written to scan its own output for exactly this class of leak, and replaced everywhere with an invented name, including in the already-published article.

If you let an AI write marketing copy from your real data, the data is going to come with it.

What I'd actually tell you

Claude was fast and genuinely good at the writing. What it could not do was know whether any of it was true, and it was equally confident either way. Every correction above came from contact with a real system — not from more reasoning, more tests, or more careful prompting.

So the useful split wasn't "AI does the easy parts". It was:

It writes. Quickly, consistently, with better test hygiene than I'd manage at 1am.
Reality checks. Real logs, real crashes, real API responses, real users.

The second half is the one that found every bug. If you're letting an AI build something, your actual job is to be the part of the loop that touches the world — and to stay suspicious of green test suites, including the ones it writes to prove itself right.

The tools are a crash-report reader and a modpack ID validator for modded Minecraft servers. Free editions, single files, no dependencies:

The full write-up of everything it got wrong, with the fixes: jaakoby.github.io/how-this-was-built.html

Your test fixtures are lying to you: what 13 real Minecraft crash reports taught me

Kaiven — Thu, 01 Oct 2026 23:50:11 +0000

I built a tool that reads modded Minecraft crash reports and tells you which mod killed the server. It passed every test I wrote and was still wrong about most real crashes.

Here is what the real data taught me, in the order it hurt.

Your own samples will lie to you

I wrote the tool first and the test fixtures second. Six synthetic logs, one per failure mode, each one crafted to contain exactly the string my pattern looked for. All six passed.

Then I pointed it at an actual crash-reports/ folder from a live 234-mod server. It diagnosed three of eight.

This is the test-writing equivalent of grading your own homework. I had written both the question and the answer, so of course they matched. The patterns were not wrong exactly — they were narrow, tuned to a tidy example of each error rather than the shape those errors take in the wild. Five real crashes used wrapper exceptions, stale-jar NoClassDefFoundErrors and malformed resource IDs that my neat little samples never contained.

The fix was not cleverness. It was reading thirteen real crash reports and writing rules for what was actually in them.

The filename that contains two versions

Forge ships as forge-1.20.1-47.4.10-universal.jar. That is Minecraft 1.20.1, Forge 47.4.10.

My regex to find the loader version was:

r"(?:neo)?forge[ \-](\d+\.\d+[\.\d]*)"

Reasonable looking. It reported every server's Forge version as 1.20.1, because that is the first number after forge-.

Worse, it was plausibly wrong. 1.20.1 is a real version string that appears everywhere in the log, so nothing looked broken. It took reading the output next to a crash report I already understood to notice the loader and the game had suspiciously identical versions.

The fix is to prefer the authoritative line and only fall back to the filename:

for pat in (
    r"Forge:\s*net\.(?:neo)?(?:minecraft)?forge:(?:forge:)?(\d+\.\d+[\.\d]*)",
    r"(?:neo)?forge-\d+\.\d+(?:\.\d+)?-(\d+\.\d+[\.\d]*)-universal",
    r"(?:neo)?forge[^\n]{0,24}?version[:\s]+(\d+\.\d+[\.\d]*)",
):
    m = re.search(pat, joined, re.I)
    if m:
        break

Crash reports carry Forge: net.minecraftforge:47.4.10 in their system details. Use the thing that is unambiguous.

One line of HTML, 107,445 characters long

The tool took 59 seconds on a real debug.log. The file was 20,000 lines and 3.4 MB — that should be well under a second.

My first theory was the obvious one. I measured the longest line in the file:

107,445 characters. It was a complete GitHub HTML page. Some mod had fetched a URL and logged the entire response body on one line.

So I capped line length for matching at 4,000 characters, re-ran, and the time went from 58.84s to 58.84s. Zero change. The theory was wrong, and the only reason I knew is that I measured before and after.

So I stopped guessing and instrumented each stage:

read_log      0.04s
environment   0.01s
blame_mod     0.00s
diagnose    >115s     <- there it is

Then timed each rule individually. The first rule printed in 0.04s. The second never printed at all, because a single re.search() call on one line had not returned.

The pattern:

r"(?P<mod>\S+).*is client ?-?only"

An unanchored \S+ followed by .*. On a long line with no match, the engine tries every split of \S+ against every split of .*. That is catastrophic backtracking, and it does not finish.

r"(?P<mod>\S{1,80}) is client[ \-]?only"

58.84s to 1.97s.

The lesson I actually took from this is not "avoid nested quantifiers". It is that I had a theory, it was wrong, and I only found out because I measured instead of declaring victory. The HTML line was real, interesting, and completely irrelevant.

Now there is a test that fails loudly rather than hanging:

def test_no_pattern_has_unbounded_quantifier_after_a_capture(self):
    bad = re.compile(r"\(\?P<\w+>\\S\+\)\.\*")
    for rule in RULES:
        for pat in rule.patterns:
            self.assertIsNone(bad.search(pat), f"{rule.id}: {pat}")

Blaming the right jar

The genuinely useful trick, and the reason the tool is worth anything: Forge annotates every stack frame with the jar it came from.

at com.example.wildlife.procedures.StalkerProcedure.execute
   (StalkerProcedure.java:250) ~[examplemod-1.20.1-2.1.11.jar%23205!/:?]

So you can walk the trace, skip anything belonging to Minecraft, Forge or the JDK, and the first jar left is almost always the mod at fault.

The whole feature lives in that skip list. A naive version blames netty-common on half of all reports, because Netty sits near the top of any network-related trace. Then you fix that and it blames fmlloader. Then modlauncher. Every one of those was a real bug I shipped and caught by running it against reports whose cause I already knew.

The same mistake, a second time, on a different tool

I built a second tool that validates modpack IDs — the typo'd minecraft:diamond_swrd that never crashes and simply gives the player nothing.

Its first run on a real 233-mod pack reported 169 typos. Most were noise:

wastelandmod:alicepack
    did you mean:  wastelandmod:icepick

That is not a typo. That is fuzzy matching with no idea what it is doing.

The cause was the same shape as the Forge version bug: I was comparing the whole ID, so the shared wastelandmod: prefix — thirteen identical characters — inflated every score before the interesting part was even reached.

Measured:

compared	full ID	path only
`cooked_caned_fish` vs `cooked_canned_fish`	0.984	0.971
`alicepack` vs `icepick`	0.905	0.750

Comparing paths alone separates a real typo from nonsense cleanly. 169 candidates became 63.

And three of those are real, still live in a pack thousands of people play, confirmed against the mod's own jar:

cooked_caned_fish → the item is cooked_canned_fish
cooked_canned_rabit_soup → the item is cooked_canned_rabbit_soup
stampler → the item is stapler

Two of those sit in the pack's diet tags, so those foods silently have no diet category. Nothing crashes. Nothing logs. You find out when a player complains, if ever.

What I would tell myself at the start

Write the fixtures from real data, or accept that your tests prove nothing. A green suite built on invented inputs measures your imagination, not your code.

Revert each fix and confirm the test fails. I found one test that passed with its fix removed entirely. It had been quietly measuring nothing for days.

When you have a theory about a performance problem, measure before and after. My plausible, interesting, well-researched theory moved the number by 0.00 seconds.

A tool that cries wolf gets ignored. Findings are now tiered by confidence, and the low-confidence tier is hidden behind a flag. Sixty-three real candidates beats a hundred and sixty-nine you learn to scroll past.

Both tools are single Python files with no dependencies. The free editions are on GitHub:

forge-server-doctor — crash report diagnosis
pack-doctor — modpack ID validation

There are paid versions with the full rule sets at jaakoby.github.io, but the free ones are standalone and the debugging stories above are the real point.