DEV Community: Jordan Liu

I Held One Lever Still

Jordan Liu — Sat, 10 Oct 2026 17:59:52 +0000

A free model and a free server are two levers. Move both, then quote one pass rate, and you did not review a model. You published a blend. I will not publish a blend.

You know the afternoon this happens. The endpoint answers. The box is up. The suite is already on disk, so you fire it, crop the fraction, and feel done. Which lever moved? If the honest answer is both, plus a retry you forgot to count, the fraction is a story about a pile.

Free infrastructure makes the pile feel scientific. No invoice, no guilt, and somehow no design. I do not buy that trade. A zero bill is not a baseline. It is a quiet way to change two things and then narrate one of them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two availability claims were supplied for this draft, and they are the only product facts I will stand on: free model access, and a free server option. I am not printing a token grant, a model roster, a hardware sheet, a duration, or a license badge. None of those arrived as a primary source I can defend for 2026-10-10. A number I cannot cite does not become true because a draft would read warmer with a big integer in the first screen.

So what experiment is left, if the headline is banned?

A two-by-two, blocked on the same task list, with one lever held still. Same prompts. Same grader. Same seed rule. Same timeout. The only legal swap is the factor you wrote into the manifest before the first request left the machine. Did the salt change, or the pot? If you changed both and then swore the broth was brighter, you cooked an anecdote. Hold the pot. Change the salt. On the next batch, hold the salt and change the pot. Now a later reader has something to argue with.

Hold the server, swap the model slot. That cell may talk about free model access, and only against the server you actually held. Hold the model, swap the server slot. That cell may talk about the free server, and only against the model you actually held. Swap both and you have an interaction check, not a verdict. If the single swaps were calm and the pair fell over, you learned they misbehave as a couple. You did not earn "the model failed." You also did not earn "the server failed."

The baseline cell is the ruler. It can be a recorded fixture plus a local stub, as long as you can hold that pair still next week. A ruler that moves is not a ruler. It is a second treatment pretending to be the wall. I would rather compare against a boring fixture than against whatever the router felt like serving at lunch.

Blocking is the part people skip because it looks fussy. It is not fussy. If cell A sees the tasks in one order and cell B sees a shuffle, order became a third lever. If one cell retries and the other does not, retry policy became a fourth. If the seed is "whatever the process clock felt like," you cannot tell a flaky item from a factor. Derive the seed from the task id plus one run salt, write that salt into the manifest, and refuse to resample until the rate looks friendly. Friendliness is not a stopping rule. It is a wish.

What does a free path break, even when you are trying to be good? A label that is really a router. A server that sleeps, so the first tasks measure wakeup and the rest measure the work. A client that retries while the manifest still says one attempt. An SDK that returns a friendly alias, so you never see which level you hit. If the factor name is an alias, the cell is unlabeled. An unlabeled cell is a missing cell. Would you sign a lab book that said "the blue one" and meant three different blues? I would not.

I am not going to invent the quota that punches a hole in the run. I am going to treat an unfinished cell as unfinished. Survivors do not get promoted to "the run." That promotion is how a partial afternoon becomes a chart with a calm typeface. Missing cells do not get filled with the mean of their neighbors. Neighbors are different treatments. Averaging across them is how a blend learns to wear a single name.

Read four symbolic outcomes, not as data I collected, but as the only sentences the grid licenses. Suppose model-only moves and server-only does not. You may talk about the model slot. Suppose server-only moves and model-only does not. You may talk about the server slot. Suppose neither single swap moves, and the pair does. You may talk about interaction, then stop. Suppose every cell moves the same way. You probably changed the harness, the tasks, or the grader, and the levers are innocent until you prove otherwise. None of those sentences need a fake percentage. The shape is the finding. A percentage without the shape is decoration.

The artifact is a protocol, not a lab log. I have not executed it against a live free tier for this article. The client body is a stub on purpose. If you run it after you replace the stub, the JSON manifest is the result. If you do not run it, you do not get to borrow certainty from the shape of the code. A green fraction from an unimplemented client would be a magic trick. I am not collecting those.

#!/usr/bin/env python3
"""Two-by-two eval manifest. Reference protocol, not a scored bake-off."""
from __future__ import annotations

import hashlib
import json
import os
import time
import uuid
from dataclasses import asdict, dataclass, field
from typing import Callable, Optional

@dataclass
class Factor:
    name: str
    level: str
    endpoint: str

@dataclass
class Trial:
    task_id: str
    seed: int
    model_level: str
    server_level: str
    ok: Optional[bool] = None
    error_class: Optional[str] = None
    latency_ms: Optional[int] = None

@dataclass
class Manifest:
    run_id: str
    harness_sha256: str
    task_file_sha256: str
    seed_salt: int
    started_unix: float
    cells: dict = field(default_factory=dict)
    finished_unix: Optional[float] = None
    admissible: bool = False
    refusal_reason: Optional[str] = None

def sha256_file(path: str) -> str:
    return hashlib.sha256(open(path, "rb").read()).hexdigest()

def classify(exc: BaseException) -> str:
    if isinstance(exc, NotImplementedError):
        return "unimplemented"
    name = type(exc).__name__
    if "Timeout" in name:
        return "timeout"
    if "Connection" in name or "HTTP" in name:
        return "transport"
    return "grader_or_task"

def run_cell(tasks, model: Factor, server: Factor, salt: int, call: Callable) -> list[Trial]:
    out = []
    for task_id in tasks:
        seed = int(hashlib.sha256(f"{salt}:{task_id}".encode()).hexdigest()[:8], 16)
        trial = Trial(task_id, seed, model.level, server.level)
        t0 = time.perf_counter()
        try:
            call(task_id, model, server, seed)
            trial.ok = True
        except Exception as exc:
            trial.ok = False
            trial.error_class = classify(exc)
        trial.latency_ms = int((time.perf_counter() - t0) * 1000)
        out.append(trial)
    return out

def judge(manifest: Manifest) -> tuple[bool, str]:
    need = {"baseline", "model_only", "server_only", "both_swapped"}
    have = set(manifest.cells)
    if have != need:
        return False, "missing_cells:" + ",".join(sorted(need - have))
    for name, trials in manifest.cells.items():
        if not trials:
            return False, f"empty:{name}"
        if any(t["error_class"] == "unimplemented" for t in trials):
            return False, "stub_call_is_not_evidence"
        if any(t["ok"] is None for t in trials):
            return False, f"unscored:{name}"
    return True, "grid_complete"

def rates_if_admissible(manifest: Manifest) -> Optional[dict]:
    ok, _reason = judge(manifest)
    if not ok:
        return None
    report = {}
    for name, trials in manifest.cells.items():
        graded = [t for t in trials if t["error_class"] is None]
        report[name] = {
            "n": len(trials),
            "graded_n": len(graded),
            "pass_rate": (sum(bool(t["ok"]) for t in graded) / len(graded)) if graded else None,
            "transport_n": sum(t["error_class"] == "transport" for t in trials),
            "timeout_n": sum(t["error_class"] == "timeout" for t in trials),
        }
    return report

def main() -> None:
    model_base = Factor("model", "baseline-slot", os.environ["MODEL_BASE_URL"])
    model_free = Factor("model", "free-model-slot", os.environ["MODEL_FREE_URL"])
    server_base = Factor("server", "baseline-slot", os.environ["SERVER_BASE_URL"])
    server_free = Factor("server", "free-server-slot", os.environ["SERVER_FREE_URL"])
    task_path = os.environ["TASK_FILE"]
    tasks = json.load(open(task_path))
    salt = int(os.environ.get("EVAL_SEED", "17"))

    def call(task_id: str, model: Factor, server: Factor, seed: int) -> None:
        raise NotImplementedError(
            f"wire your client: {task_id} {model.level} @ {server.level} seed={seed}"
        )

    plan = {
        "baseline": (model_base, server_base),
        "model_only": (model_free, server_base),
        "server_only": (model_base, server_free),
        "both_swapped": (model_free, server_free),
    }
    manifest = Manifest(
        str(uuid.uuid4()), sha256_file(__file__), sha256_file(task_path), salt, time.time()
    )
    for name, (model, server) in plan.items():
        manifest.cells[name] = [asdict(t) for t in run_cell(tasks, model, server, salt, call)]
    ok, reason = judge(manifest)
    manifest.admissible = ok
    manifest.refusal_reason = None if ok else reason
    manifest.finished_unix = time.time()
    json.dump(asdict(manifest), open("run_manifest.json", "w"), indent=2)
    if not ok:
        raise SystemExit(f"refusing headline metric: {reason}")

if __name__ == "__main__":
    main()

Pin the harness and the task file before you trust the run you think you started. A dirty tree is another lever, and it never introduces itself.

sha256sum eval_grid.py tasks.json | tee inputs.sha256
export MODEL_BASE_URL MODEL_FREE_URL SERVER_BASE_URL SERVER_FREE_URL
export TASK_FILE=tasks.json EVAL_SEED=17
python3 eval_grid.py
python3 -c 'import json; m=json.load(open("run_manifest.json")); print(m["admissible"], m.get("refusal_reason"))'

A stub run should exit on stub_call_is_not_evidence. That refusal is the correct first result. Do not catch it and tweet the partial JSON. The partial JSON is a note that the client was never wired.

When a real client is in place, debug the manifest before you debug the model. Open run_manifest.json and compare transport_n across cells. If transport piles up only in server_only and both_swapped, your sentence is about the server path, or about the path to that server. It is not about the model. If timeouts pile up only on the free model slot, say timeout, then decide whether your product even cares. If missing_cells fires, you do not have a result. You have a plan you did not finish. Imputing the missing corner from the other three is how people invent an interaction they never observed.

What do you commit? The harness, the task-file hash, the seed salt, and the manifest. Not a screenshot of a fraction. A screenshot cannot tell me whether the both-swapped cell was allowed to speak. The JSON can. If admissible is false, the writeup is the refusal reason, and that writeup is allowed to be short. Short and refused beats long and blended.

Who should walk past this? You, if the demo is at lunch and the room wants a vibe. A grid is a bad vibe. You, if the question is whether a UI feels kind, because kindness is not a factor level in this file. You, if every request is locked to one blended endpoint and you cannot hold a baseline still. Then you have one pipe. A pipe can still be worth using for a prototype. It cannot carry a sentence that starts with "the model," and it cannot carry one that starts with "the server." Say "this pipe" and stop. That sentence is weaker, and it is the one you actually earned.

Limits, with the romance removed. There is no scoreboard here, because I did not run the live pair, and I will not invent one to decorate a method. Grader quality sits outside the grid. A stable wrong grader produces an admissible, repeatable, useless manifest. Environment variables are not a docs site. Free access can move next week, so last week's JSON does not license next week's slogan. Interaction can be real and still be too small to matter for your task. The protocol will not choose your practical threshold. Write that threshold down before you look, or do not pretend the look was a test.

Trying the pipe is fine. Publishing the try as a model review is the part I will not cosign. If the free model slot and the free server slot in your env file are the ones named in the disclosure, they are eligible factors in this grid. They are not pre-cleared results. Run the four cells once, keep the refusal or the JSON, and let that file outlive the badge.

I Froze the Hash. The Free Run Had to Earn the Sign.

Jordan Liu — Fri, 09 Oct 2026 14:26:59 +0000

A free runner is not a result. A stopwatch is not a race, and a pass rate is not a decision until the sample, the scorer, and the stop rule are already boring. If any of those can still wriggle, a free model and a free server only make the wriggling cheaper.

Freeze the hash. Pre-register the break. Let a sign appear only when both halves agree.

You know the itch. A stack shows up, the calls cost nothing you can feel, a server is offered so you skip provisioning, and lunch starts to look like a leaderboard. Did you ever nudge one prompt, rerun, and then describe the last number as if it had been the hypothesis?

If that scene feels familiar, it is the habit this card is built to interrupt. Not a trophy. A failure mode with a friendly face.

The feed keeps congratulating small models for going quiet on a private checker. Quiet is not stable. A checker can go silent because you edited the items until it did. Have you ever watched a green cell calm a room that never asked whether the other half of the file would have stayed green?

Seasoning and judging are different jobs. Taste the soup to change the recipe, and you are cooking. Taste it to award a ribbon, and you are peeking. Free access is a gift to the cook and a trap for the judge, because another taste barely stings.

The chart gets patient. You do not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. For this method I am treating two availability claims as given, and only those: free model access, and a free server option. I am not stating a token quota, a model roster, a machine size, a duration, or a score from a run.

I do not have a primary source I am willing to pin next to a task hash, and I will not fill the gap with a figure that might already be stale. If it is not in the card, it is not a result. A number that cannot sit in the same commit as the tasks is just weather.

What are those surfaces doing in the method? They are a pair of runners, not a mascot. Treatment A is free model access: your process sends a frozen item and scores the completion locally. Treatment B is the free server: the same item goes to a hosted runner you do not administer, and you score the trace that comes back with the same function.

Same items, same rubric, different place the work happened. If you cannot name which surface tripped a rule you wrote yesterday, you do not have an evaluation. You have a demo with extra steps.

I have not executed this harness against that product for this article. The blocks below are a proposal with a synthetic fixture so you can see the refusal path on your laptop. Thresholds are choices you lock before the first network call. They are not measurements, and they are not a review of anyone's uptime.

The unit is the item, not the afternoon. You export tasks to JSONL, one object per line, each with an id and an expected string. You hash the file bytes, then split with a seeded digest so half 0 and half 1 stay put on rerun. You score both treatments on both halves.

A sign is earned only when each half clears the item floor, each arm stays under the transport-error ceiling, the absolute gap clears a floor you wrote down so a one-item wobble cannot wear a crown, and the two halves agree on the direction. A hash mismatch is an invalidation even if the new file is basically the same. Editing the rubric after a sad cell is an invalidation too. Git will remember, even if your recap does not.

Why not one mean? A single average can sand a flip off the table until the room feels calm. I would not trust that sanding in a write-up. Would you change a routing file because forty items leaned left and forty leaned right?

I would call that a break and leave the config alone. Agreement is a low bar. It is still a bar.

Performs well cannot mean a percentage I did not measure. Here it means the run can finish under rules another person can replay: both surfaces return scoreable traces, invalidation stays quiet, and the sign matches across halves. Breaks has to stay a specific sentence, not a mood.

A timeout is not a foolish model. A rubric that never looks at tool traces is not a broken server. A moved hash is not a product defect. It is you, still cooking.

Fold those into one red badge and the free tier takes the blame for your notebook. Three sentences and zero winners is a better log. Can you say which sentence fired? If you cannot, you are not done.

Who should skip this? Anyone who needs a flattering percentage before stand-up, because the card is built to withhold it. Anyone holding private customer text should not ship it to a free server because the meter feels kind. Anyone who needs a medical, legal, or safety judgment should not pretend a frozen string check is that judgment.

If you cannot commit the task file, the card, and the scorer in one revision, you cannot audit the claim later. Do not make the claim.

Limitations, with the soft landing removed. Half-agreement is not proof that one surface is wiser, cheaper in a way that lasts, or still offered next month. It is a stability check on a small decision. A hosted runner can insert retries, clocks, and tool hops your local call never saw, and this protocol notices only if the scorer reads those fields on purpose.

String equality is a blunt rubric. It will call a harmless paraphrase a miss, and it will miss a fluent wrong answer that happens to match. A pilot that survives the card may justify a larger pre-registered run. It does not justify a testimonial, and I am not writing one.

The workflow is short enough to do before coffee gets cold, and strict enough to ruin a dishonest chart. Write the card in the same file as the scorer, hash the JSONL, and commit both plus the hash before any call. Implement two clients that take an item and return text, and keep secrets out of the repo. Leave a dry implementation that raises, so a curious teammate cannot just see what happens against the network by running the module.

Replace the dry clients only after that commit exists. Run the module and read earned. If it is false, the result is the breaks list. Sit with it. That list is the finding.

Save this as split_half_sign.py. It is a proposal. --fixture never leaves the process, and the default clients raise so a real file cannot accidentally become a leaderboard.

python3 split_half_sign.py --fixture
sha256sum tasks.jsonl
python3 split_half_sign.py tasks.jsonl "$(sha256sum tasks.jsonl | cut -d' ' -f1)"

That second command, left as written, should fail the transport rule. Good. That is the dry default doing its job. A green earned before you replace dry_complete would mean the harness is lying, and you should delete it.

#!/usr/bin/env python3
# Proposal, not a product benchmark.
# Lock CARD before any network call. Thresholds are design choices.
# --fixture is synthetic and opens no socket.
from __future__ import annotations

import hashlib
import json
import sys
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

CARD = {
    "protocol": "split-half-sign-v1",
    "min_items_per_half": 30,
    "min_scored_fraction": 0.95,
    "max_transport_error_rate": 0.05,
    "min_abs_gap": 0.02,
    "seed": "freeze-me-before-running",
}

CompleteFn = Callable[[dict], str]


def file_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def load_jsonl(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows


def half_of(item_id: str, seed: str) -> int:
    digest = hashlib.sha256(f"{seed}:{item_id}".encode()).hexdigest()
    return int(digest[-1], 16) % 2


def rubric(task: dict, completion: str) -> bool:
    expected = str(task.get("expected", ""))
    return completion.strip() == expected.strip()


@dataclass
class ArmReport:
    name: str
    n: int
    scored: int
    transport_errors: int
    pass_rate: float | None
    invalidated: str | None


def run_arm(name: str, items: list[dict], complete: CompleteFn) -> ArmReport:
    errors = scored = hits = 0
    for task in items:
        try:
            text = complete(task)
        except RuntimeError:
            errors += 1
            continue
        scored += 1
        hits += int(rubric(task, text))
    n = len(items)
    err_rate = (errors / n) if n else 1.0
    frac = (scored / n) if n else 0.0
    invalidated = None
    if n < CARD["min_items_per_half"]:
        invalidated = "below_min_items"
    elif err_rate > CARD["max_transport_error_rate"]:
        invalidated = "transport"
    elif frac < CARD["min_scored_fraction"]:
        invalidated = "unscored_mass"
    rate = (hits / scored) if scored else None
    return ArmReport(name, n, scored, errors, rate, invalidated)


def sign_of(left: float | None, right: float | None) -> int:
    if left is None or right is None:
        return 0
    gap = left - right
    if abs(gap) < CARD["min_abs_gap"]:
        return 0
    return 1 if gap > 0 else -1


def evaluate(rows, arms, got_hash, expected_hash):
    breaks = []
    if expected_hash is not None and got_hash != expected_hash:
        return {"earned": False, "breaks": ["hash_mismatch"], "hash": got_hash, "sign": 0}
    halves = {0: [], 1: []}
    for row in rows:
        halves[half_of(str(row["id"]), CARD["seed"])].append(row)
    half_signs = []
    reports = []
    for h, subset in halves.items():
        rates = {}
        for name, fn in arms.items():
            report = run_arm(f"{name}.h{h}", subset, fn)
            reports.append(asdict(report))
            if report.invalidated:
                breaks.append(f"{name}.h{h}:{report.invalidated}")
            rates[name] = report.pass_rate
        half_signs.append(sign_of(rates.get("free_model"), rates.get("free_server")))
    if not breaks and (not half_signs or 0 in half_signs):
        breaks.append("gap_below_floor_or_unscored")
    if not breaks and half_signs[0] != half_signs[1]:
        breaks.append("halves_disagree")
    earned = not breaks
    return {
        "earned": earned,
        "sign": half_signs[0] if earned else 0,
        "half_signs": half_signs,
        "breaks": breaks,
        "hash": got_hash,
        "reports": reports,
        "note": "+1 means free_model ahead on pass rate; -1 means free_server ahead",
    }


def dry_complete(_task: dict) -> str:
    raise RuntimeError("dry run: replace me after the hash is committed")


class FixtureComplete:
    # Synthetic. Pass rates flip by half. Not a vendor measurement.

    def __init__(self, name: str) -> None:
        self.name = name

    def __call__(self, task: dict) -> str:
        h = half_of(str(task["id"]), CARD["seed"])
        want_hit = (self.name == "free_model" and h == 0) or (
            self.name == "free_server" and h == 1
        )
        return str(task["expected"]) if want_hit else "MISS"


def synthetic_rows() -> list[dict]:
    buckets = {0: [], 1: []}
    i = 0
    while len(buckets[0]) < 40 or len(buckets[1]) < 40:
        item_id = f"t{i:04d}"
        h = half_of(item_id, CARD["seed"])
        if len(buckets[h]) < 40:
            buckets[h].append({"id": item_id, "expected": f"ok-{i}"})
        i += 1
    return buckets[0] + buckets[1]


def main(argv: list[str]) -> int:
    if len(argv) >= 2 and argv[1] == "--fixture":
        rows = synthetic_rows()
        blob = "\n".join(json.dumps(r, separators=(",", ":")) for r in rows).encode()
        digest = hashlib.sha256(blob).hexdigest()
        out = evaluate(
            rows,
            {
                "free_model": FixtureComplete("free_model"),
                "free_server": FixtureComplete("free_server"),
            },
            digest,
            digest,
        )
        out["fixture"] = "synthetic_refusal_not_a_product_run"
        print(json.dumps(out, indent=2))
        return 0
    if len(argv) != 3:
        print(
            "usage: split_half_sign.py --fixture | split_half_sign.py tasks.jsonl HASH",
            file=sys.stderr,
        )
        return 2
    path = Path(argv[1])
    out = evaluate(
        load_jsonl(path),
        {"free_model": dry_complete, "free_server": dry_complete},
        file_hash(path),
        argv[2],
    )
    print(json.dumps(out, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

A task file is just lines like these. Hash the bytes you will actually send, not a prettier export you make later.

{"id":"t0001","expected":"pong"}
{"id":"t0002","expected":"pong"}

When the fixture prints, look at three fields and ignore the urge to screenshot a winner. earned should be false. breaks should name the disagreement, and half_signs should be a pair that does not match. If a future real run prints earned true, the sign is a stable direction on this frozen file, under this blunt rubric, on this day.

It is still not a quota, not a hardware review, and not permission to skip the private-data rule. Puppet rates inside --fixture are not a vendor score. They exist so you can watch the harness keep its mouth shut. When you later point the same function at real clients, silence still means silence.

I will ask for one thing, then stop selling. If free model access and a free server are actually available on the MonkeyCode account you use, point this card at tasks you can stand to hash in public. Commit the hash and the stop rule before the first call.

If the sign does not earn itself, write about the break. I trust that note more than a last-run percentage.

I Scored the Flip. Best-of-Five Folded.

Jordan Liu — Thu, 08 Oct 2026 10:16:10 +0000

I will not publish best-of-five. A free code model is not usable because one sample passed. That sample might be the only one that would have passed. If you publish the highlight reel, you measured a slot machine, not a tool.

You have seen the flattering cut, haven't you? Five draws, one green check, a screenshot, a conclusion. What question did that answer? Not the question your teammate hits on a Tuesday, when nobody is sitting there rerolling until the tests blush.

The feed this week is busy with profile toys and portfolios people vibe-coded over a weekend. Fine toys. None of them tell you whether draw three still exits zero.

So the conclusion comes first, before any endpoint. Report the first draw, the stable pass, and the flip. Ignore any protocol that lets you delete the red rows. If those three numbers are not on disk, you do not have an evaluation. You have a mood.

First-draw pass is what an unattended user gets. Stable pass is the share of tasks that pass on every draw, with no runner error hiding in the set. Flip is the share that both pass and fail across draws.

Same prompt. Same check. Different verdict. That last one is the alibi killer.

A task that flips is not basically working. It is a coin, and coins do not get merge rights. Would you hand a coin your migration?

Why not a single accuracy number? Because a single number is how luck borrows a lab coat. Best-of-five can rise while stability falls. You would feel better and ship worse.

I would rather see an ugly stable-fail than a pretty flip. An ugly fail tells me to change the task class or the tool. A flip tells me I was about to quote a coincidence.

I do not have a leaderboard, and I am not going to invent one. This note is a protocol and a scorer. Any rows that look like results in the fixture are a self-check of the arithmetic. They are not a benchmark of a free model, a paid model, or a server.

If a pass rate shows up here without a CSV from the day the checks ran, treat it as fiction. I want you suspicious. Suspicion is the method.

Where do the free generator and the free runner enter? I want both boring and available, so the afternoon is about variance instead of about whether I could afford the draws. MonkeyCode is the option this harness is aimed at: free model access for the completions, and a free server option so every sample faces the same check command rather than whatever my laptop happened to have installed. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The outreach brief calls the project open source. Clone the repository you can verify. A paraphrased client is not a source.

I am also not quoting a token quota, a machine size, or a duration. Those are dashboard facts, and dashboard facts rot faster than blog posts. On the day you run, read the current allowance and the current server terms, then copy them into the run notes by hand.

If the allowance is smaller than K times your task count, shrink K before you start. Dropping tasks afterward, and still calling the file a replicate, is how samples vanish while the chart stays confident.

The harness is plain Python on purpose. No framework to debug before the model. It shells out to your check command, which is the whole point. Pretty code that never executes is a different hobby.

#!/usr/bin/env python3
"""split_score.py — first draw, stable pass, and flip.

Live mode is unexecuted until MODEL_ENDPOINT is set.
`--fixture` checks the arithmetic only. It is not a model score.
"""
from __future__ import annotations

import argparse, csv, json, os, subprocess, hashlib
from pathlib import Path

def load_tasks(root: Path) -> list[dict]:
    tasks = []
    for path in sorted(root.glob("*/task.json")):
        task = json.loads(path.read_text())
        task["dir"] = str(path.parent)
        tasks.append(task)
    if not tasks:
        raise SystemExit(f"no tasks under {root}")
    return tasks

def call_model(endpoint: str, prompt: str, draw: int) -> str:
    # Proposal only. Match the real request shape before a live run.
    # Read any key from the environment. Never hardcode one.
    import urllib.request
    body = json.dumps({
        "prompt": prompt,
        "draw": draw,
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    if "text" not in payload:
        raise SystemExit("response missing text; fix the client, do not score it")
    return payload["text"]

def run_check(task_dir: str, code: str) -> str:
    snippet = Path(task_dir) / "_candidate.py"
    snippet.write_text(code)
    try:
        proc = subprocess.run(
            ["python3", "check.py"],
            cwd=task_dir,
            capture_output=True,
            text=True,
            timeout=30,
        )
    except subprocess.TimeoutExpired:
        return "runner_error"
    if proc.returncode == 0:
        return "pass"
    if proc.returncode < 0:
        return "runner_error"
    err = (proc.stderr or "") + (proc.stdout or "")
    if "Traceback" in err and "_candidate" not in err:
        return "runner_error"
    return "fail"

def summarize(rows: list[dict]) -> dict:
    by_task: dict[str, list[tuple[int, str]]] = {}
    for row in rows:
        by_task.setdefault(row["task"], []).append((int(row["draw"]), row["verdict"]))
    n = len(by_task)
    if n == 0:
        raise SystemExit("no rows to score")
    first = stable = flip = runner = 0
    for pairs in by_task.values():
        verdicts = [v for _, v in sorted(pairs)]
        if any(v == "runner_error" for v in verdicts):
            runner += 1
        if verdicts[0] == "pass":
            first += 1
        if all(v == "pass" for v in verdicts):
            stable += 1
        if "pass" in verdicts and "fail" in verdicts:
            flip += 1
    return {
        "tasks": n,
        "first_pass": round(first / n, 3),
        "stable_pass": round(stable / n, 3),
        "flip_rate": round(flip / n, 3),
        "tasks_with_runner_error": runner,
    }

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--tasks", default="tasks")
    parser.add_argument("--draws", type=int, default=5)
    parser.add_argument("--fixture", action="store_true")
    parser.add_argument("--out", default="split_score.csv")
    args = parser.parse_args()

    if args.fixture:
        # Synthetic rows. They test the scorer. They do not describe a product.
        rows = [
            {"task": "parse_int", "draw": 0, "verdict": "pass"},
            {"task": "parse_int", "draw": 1, "verdict": "pass"},
            {"task": "parse_int", "draw": 2, "verdict": "fail"},
            {"task": "slug", "draw": 0, "verdict": "fail"},
            {"task": "slug", "draw": 1, "verdict": "fail"},
            {"task": "slug", "draw": 2, "verdict": "fail"},
            {"task": "clamp", "draw": 0, "verdict": "pass"},
            {"task": "clamp", "draw": 1, "verdict": "pass"},
            {"task": "clamp", "draw": 2, "verdict": "pass"},
        ]
    else:
        endpoint = os.environ["MODEL_ENDPOINT"]
        rows = []
        for task in load_tasks(Path(args.tasks)):
            for draw in range(args.draws):
                code = call_model(endpoint, task["prompt"], draw)
                verdict = run_check(task["dir"], code)
                rows.append({"task": task["id"], "draw": draw, "verdict": verdict})

    summary = summarize(rows)
    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=["task", "draw", "verdict"])
        writer.writeheader()
        writer.writerows(rows)
    digest = hashlib.sha256(Path(args.out).read_bytes()).hexdigest()[:12]
    print(json.dumps({**summary, "csv_sha256_12": digest}, indent=2))

if __name__ == "__main__":
    main()

The fixture is supposed to be mean. parse_int flips. slug fails every draw. clamp stays green.

If you cannot say those three sentences without opening a pricing page, you are not ready for a live file. What should the scorer print? First-draw 0.667, stable pass 0.333, flip rate 0.333, runner errors 0. I did that division on paper before I trusted a script. You should do the same.

A scorer you have not caught is just another lucky sample, wearing a .py hat. Run the self-check before you touch a network. It should not need a key, a server, or a hope.

python3 split_score.py --fixture --out /tmp/split_fixture.csv
python3 - << 'PY'
import json, subprocess
raw = subprocess.check_output(
    ["python3", "split_score.py", "--fixture", "--out", "/tmp/split_fixture.csv"]
)
got = json.loads(raw)
expect = {
    "first_pass": 0.667,
    "stable_pass": 0.333,
    "flip_rate": 0.333,
    "tasks_with_runner_error": 0,
}
for key, val in expect.items():
    assert got[key] == val, (key, got[key], val)
print("scorer ok", got["csv_sha256_12"])
PY

A task is a directory, not a vibe. task.json holds an id and a prompt. check.py imports the candidate and exits non-zero when the behavior is wrong. Keep the check dumber than the task.

If the check is the clever part, you are grading your harness and calling it a model. That is an easy way to feel rigorous while learning nothing.

{
  "id": "clamp",
  "prompt": "Write clamp(x, lo, hi) in _candidate.py. Return x bounded to [lo, hi]. No imports. No prints."
}

# tasks/clamp/check.py — example check, not a live result
import _candidate

def expect(got, want):
    if got != want:
        raise SystemExit(f"{got!r} != {want!r}")

expect(_candidate.clamp(3, 0, 2), 2)
expect(_candidate.clamp(-1, 0, 2), 0)
expect(_candidate.clamp(1, 0, 2), 1)

The generator and the runner should not share a mood. Draws come from the free model endpoint. The live scorer runs on the free server, so check.py sees one image, one Python, one clock. Laptop runs are for the fixture. Mix them and you are grading your own machine again.

When the fixture matches, and only then, point the script at a live endpoint. The request body inside call_model is a proposal. I have not executed it against a current API, and I will not guess a path, a model id, or a header name in public.

Match the client to the docs you verified today. Keep the key in the environment. Log a hash of the prompt next to the CSV.

If the prompt changes, start a new file. Editing the old file is not replication. It is a quiet rewrite.

# Proposal, not a verified hostname or client path.
# Confirm today's free-server login from the dashboard, then run the scorer there.
export MODEL_ENDPOINT="https://example.invalid/draw"
python3 split_score.py --tasks ./tasks --draws 5 --out runs/today.csv
sha256sum runs/today.csv

Read the printout like someone who has been fooled by a green cell before. High first-draw plus high flip means the generator can look competent and still be a coin. Would you let that coin edit a migration? I would not.

Low first-draw and a stable fail is a kindness. It tells you this task class is a break, not a teaser. High stable pass and a flip near zero is the only pattern I will call performs well, and only for drafts a human still reads.

Runner errors are not model fails. They mean the check, the timeout, or the server session broke. Fix that, rerun, and do not average across the mess.

Who should walk away? If you need a number for a launch note tonight, walk away. Five draws times a dozen tasks will not fit in a coffee break, and a partial grid is not a result you get to round up.

If the code can touch credentials, payments, or auth, walk away from the free path. A green clamp says nothing about a webhook. If the check needs a private network, a production token, or customer data, do not place it on a free server either. Shared runners are for toy tasks with toy fixtures.

If your question is taste in prose, this harness will lie politely, because it only speaks exit codes. Exit codes do not have taste. They have zero and not-zero.

There is a quieter limit, and it matters. Temperature 0.2 in the proposal is a frozen confound, not an optimum I measured. Change it and you have a different study. K=5 is a budget, not a proof about sample size.

A free server can restart, throttle, or run slower than a laptop. Those events belong in the notes, not in the model column. I will not use this protocol to rank products against each other.

It can rank task classes against one generator, on one day, under one check suite. That is enough. It is also all the certainty the design earns.

If you already have a MonkeyCode login, point the draws at the free model, run the checks on the free server, and keep the CSV beside the prompt hash. If you do not, run the fixture anyway. The arithmetic does not need a brand. The brand does not get a score until the arithmetic has a file.

I Kept Three Scores. The Mean Was a Costume.

Jordan Liu — Thu, 08 Oct 2026 07:51:17 +0000

A blended pass rate is a costume. It looks like measurement. Most days it is three arguments sharing a coat, and the coat is what people quote.

I do not want that coat in this piece. You have seen the posts this month, a percentage stacked on a task list, pressed flat enough to screenshot. Twenty prompts go out, eighteen replies parse, and the chart says ninety percent. The thread then treats the chart as a personality.

Then you open the two failures. One is a refusal you wanted. The other is fluent nonsense wrapped in valid JSON. Which failure is the model, and which success is just the parser being easy to please?

So the conclusion comes first. Split the fixture by stratum, sweep seeds, and keep three scores that you are not allowed to add together. If you cannot show the worst stratum and its spread, you do not have a result. You have wardrobe.

That is the evaluation. The rest is a harness, a toy fixture whose arithmetic you can check by hand, and a rule for what performs well is allowed to mean. I did not execute this protocol against a live endpoint for this draft. I will not invent a pass rate to decorate it.

Free repeated calls are what make a seed sweep possible instead of theatrical. MonkeyCode is the open-source project this outreach is about, and the only availability claims I will repeat are free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not claiming a token quota, a box size, a region, a latency, or that either free path is permanent.

Why mention a free path in a measurement piece at all? Because an honest check is a multiplier, strata times seeds times one frozen prompt. A single call is cheaper, and it is also how a dashboard learns to lie. A free model lane plus a free server matters here only as a condition you can afford to repeat. It is not a medal, and a paid lane is not automatically the grown-up one.

Write the fixture before you pick an endpoint. I use four strata, and I mean the word. Extract, classify, format, abstain. Each row carries an id, a prompt, an expect flag, and a gold set.

Why a set rather than one blessed string? Case and extra spaces are not a moral event, and exact-match theater burns afternoons. Why an expect flag at all? A correct refusal looks like a miss if the scorer only hunts for a substring.

That bug hides inside a blended rate, so I would rather make it impossible to commit. Freeze the file in git before the first request leaves the laptop. If the gold moves mid-sweep, you are grading a draft and blaming a model. Have you ever done that and called it rigor?

Each row then runs at several seeds. Same prompt, same temperature, same token cap, if the API even exposes those knobs. One run is an anecdote. Five is my floor, and ten is the floor when the stratum is abstain, because a refusal that flips is a coin, not a personality.

Log HTTP status, elapsed milliseconds, and a short body hash beside the scores. Do not add milliseconds into the pass bit. Speed is a different article. If a free server queues or fails closed, that fact belongs in status, not in a fake intelligence miss.

Three scores, and they are not friends. Parse asks only whether the body is JSON with the keys I demanded. Match asks whether a normalized answer lands in the gold set, and it stays missing when the row expected an abstention. Abstain asks whether the model declined when it should have, and whether it answered when declining would have been a dodge.

A lovely parser can still be a reckless guesser. Add those axes into one ratio and you sew the coat again. Still tempted to add them? Put a classify hit next to an abstain miss and tell me what the average is for.

The scorer below is the artifact. The fixture rows are ones I wrote so the arithmetic is inspectable. They are not model outputs, and they are not a MonkeyCode benchmark. Trust this as a unit test for the judge, because a judge you have not tested is just another model with a clipboard.

#!/usr/bin/env python3
"""Stratum seed-sweep scorer.

Unexecuted against any live endpoint.
FIXTURE rows are author-written so the arithmetic can be checked by hand.
Population stdev is used because the fixture is the whole claimed set.
"""
from __future__ import annotations

import hashlib
import json
import statistics
from collections import defaultdict
from pathlib import Path

FIXTURE = [
    {
        "id": "e1",
        "stratum": "extract",
        "expect": "answer",
        "gold": ["ada lovelace"],
        "seeds": [
            {"seed": 1, "body": '{"answer":"Ada Lovelace"}', "status": 200, "ms": 410},
            {"seed": 2, "body": '{"answer":"Ada Lovelace"}', "status": 200, "ms": 390},
        ],
    },
    {
        "id": "c1",
        "stratum": "classify",
        "expect": "answer",
        "gold": ["bug"],
        "seeds": [
            {"seed": 1, "body": '{"answer":"bug"}', "status": 200, "ms": 220},
            {"seed": 2, "body": '{"answer":"feature"}', "status": 200, "ms": 240},
        ],
    },
    {
        "id": "a1",
        "stratum": "abstain",
        "expect": "abstain",
        "gold": [],
        "seeds": [
            {"seed": 1, "body": '{"answer":null,"abstain":true}', "status": 200, "ms": 300},
            {"seed": 2, "body": '{"answer":"42"}', "status": 200, "ms": 280},
        ],
    },
    {
        "id": "f1",
        "stratum": "format",
        "expect": "answer",
        "gold": ["3"],
        "seeds": [
            {"seed": 1, "body": "three", "status": 200, "ms": 180},
            {"seed": 2, "body": '{"answer":"3"}', "status": 200, "ms": 190},
        ],
    },
]


def norm(value: object) -> str:
    return " ".join(str(value).strip().lower().split())


def score_body(row: dict, seed_row: dict) -> dict:
    raw = seed_row["body"]
    parse_ok = False
    answer = None
    abstained = False
    parsed = None
    try:
        parsed = json.loads(raw)
        parse_ok = isinstance(parsed, dict) and (
            "answer" in parsed or "abstain" in parsed
        )
    except json.JSONDecodeError:
        parse_ok = False
    if parse_ok and parsed is not None:
        answer = parsed.get("answer")
        abstained = bool(parsed.get("abstain")) or answer is None
    gold = {norm(item) for item in row["gold"]}
    if row["expect"] == "answer":
        match = bool(parse_ok and answer is not None and norm(answer) in gold)
        abstain_ok = bool(parse_ok and not abstained)
    else:
        match = None
        abstain_ok = bool(parse_ok and abstained and answer is None)
    return {
        "id": row["id"],
        "stratum": row["stratum"],
        "seed": seed_row["seed"],
        "parse_ok": parse_ok,
        "match": match,
        "abstain_ok": abstain_ok,
        "status": seed_row["status"],
        "ms": seed_row["ms"],
        "body_sha256": hashlib.sha256(raw.encode()).hexdigest()[:12],
    }


def pack(values: list[float]) -> dict:
    if not values:
        return {"n": 0, "mean": None, "pstdev": None}
    spread = statistics.pstdev(values) if len(values) > 1 else 0.0
    return {
        "n": len(values),
        "mean": round(statistics.fmean(values), 3),
        "pstdev": round(spread, 3),
    }


def summarize(scored: list[dict]) -> dict:
    grouped: dict[str, list[dict]] = defaultdict(list)
    for item in scored:
        grouped[item["stratum"]].append(item)
    report = {}
    for stratum, rows in grouped.items():
        parses = [1.0 if row["parse_ok"] else 0.0 for row in rows]
        matches = [1.0 if row["match"] else 0.0 for row in rows if row["match"] is not None]
        abstains = [1.0 if row["abstain_ok"] else 0.0 for row in rows]
        report[stratum] = {
            "parse": pack(parses),
            "match": pack(matches),
            "abstain": pack(abstains),
        }
    return report


if __name__ == "__main__":
    scored = [score_body(row, seed) for row in FIXTURE for seed in row["seeds"]]
    Path("scores.jsonl").write_text("".join(json.dumps(item) + "\n" for item in scored))
    print(json.dumps(summarize(scored), indent=2))

Run the judge before you point it at anything with a hostname. If this arithmetic surprises you, you are not ready for a live file.

python3 stratum_sweep.py
# writes scores.jsonl
# prints per-stratum mean and population stdev
# fixture only: a test of the scorer, not of a product

Walk the rows and the means fall out. Extract match mean is 1.0 with population stdev 0. Classify parses both times and matches once, so match mean is 0.5 and population stdev is 0.5. Format parse mean is 0.5, because the word three is not JSON. Abstain match stays at n=0 on purpose, and abstain_ok mean is 0.5.

If your machine disagrees, the scorer drifted. That is a bug in the judge, not a model story. A blended pass would smear those four stories into one polite decimal, and I am not printing it. If your eye still wants the decimal, that want is the bug.

I use population standard deviation on purpose. The fixture is the whole set you claimed to score, not a sample from a hidden universe. A sample stdev on two points just inflates the costume. If you grow the fixture, recompute, and do not carry the old spread forward like a family heirloom.

What does performs well mean once a live client replaces the fixture? Not a cutoff borrowed from a thread. Format should show a high parse mean and a parse spread near zero, because format is a contract, not a vibe. Extract should show a small match spread.

A mean of 0.8 with a spread of 0.4 is a coin that landed well in the screenshot you kept. Would you ship a parser on that evidence? I would not. I would add seeds until the spread either collapsed or confessed.

Abstain should be stable on the abstain stratum, and abstain_ok should also be stable on the answer strata. A model that refuses every prompt is not careful. It is vacant. A model that never refuses is not bold. It is unscoped.

The design breaks in familiar places. Two seeds will not save you, and the toy uses two only so you can check the math by hand. A wrong gold set makes a stable model look guilty. A sloppy normalizer launders a near miss into a hit.

Dropped match cells are a different population from parse cells. Compare them as if they were siblings and the chart starts talking nonsense. Null is not zero. If you coerce it, you just invented a failure the row was never scoring.

A free server can throttle, fail closed, or shape errors differently from the other lane you use on Tuesdays. That is a condition difference. Keep it in status and in the timing field. I am not giving you a measured timeout rate, a queue depth, or a token budget, because none of those were recorded for this piece.

If your client retries, the JSONL has to count attempts, or you scored the loop. Write temperature, token cap, and a hash of the fixture into every row, even when you are sure you set them once. Config drift is how two identical sweeps become an argument. I would rather lose an afternoon to a hash mismatch than win one with a chart I cannot rebuild.

Who should walk away? Anyone who needs a compliance signature. A four-row fixture is a worked example, not an assurance case. Anyone whose harm lives in the tail should also walk, because mean and spread on twenty friendly prompts will grin while the tail burns.

Anyone who cannot freeze the fixture in git should walk too. You would be grading a moving target and blaming the model. And if you only have one call per task, do not pretend a sweep happened. Write that you took a sample of one. That sentence beats a dashboard tile.

Pointing the judge at a free path should stay boring. Read the base URL and the key from the environment. Do not hardcode a model id you have not checked against this week's docs. If the API ignores seed, record that as a limitation. Silence is not determinism.

export EVAL_BASE_URL="https://example.invalid/replace-from-current-docs"
export EVAL_API_KEY="${EVAL_API_KEY:-}"
if [ -z "$EVAL_API_KEY" ]; then
  echo "set EVAL_API_KEY in your shell, not in the repo"
  exit 1
fi
echo "key loaded, not printed"
# Local stand-in. Not a vendor schema. Not a claim that this path exists.
cat > prompt.json << 'EOF'
{"note":"local stand-in","fixture_id":"e1","seed":1,"temperature":0,"prompt":"Extract the person. Reply as JSON with an answer field."}
EOF
curl -sS "$EVAL_BASE_URL" \
  -H "Authorization: Bearer $EVAL_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @prompt.json \
  -o "raw/e1.seed1.json" \
  -D "raw/e1.seed1.headers" \
  -w "status=%{http_code} time=%{time_total}\n"

After two conditions, diff the reports, not the vibes. Change nothing except a base URL you already verified in current docs.

python3 stratum_sweep.py > report_a.json
python3 stratum_sweep.py > report_b.json
diff -u report_a.json report_b.json

If match means move when only the server changes, you learned something about the condition. If they move when you change nothing, the sweep is still too small, or the endpoint is not honoring seed. Both are results. Neither needs a superlative.

So ask it of the next single-cell chart that lands in your feed. If extract is calm and abstain flips on every seed, what did the average tell you? It told you the coat still fits. Take it off, and keep the JSONL.

If you already have a free model lane and a free server, including the pair this MonkeyCode outreach is pointing at, aim a frozen fixture at them and keep the worst stratum with its spread. I will trust that file longer than I will trust a hero percentage.

I Treated the Free Server as a Confound.

Jordan Liu — Thu, 24 Sep 2026 20:35:51 +0000

I will not publish a pass rate from a free server until the log can tell a wrong answer from a tired plan. That is the conclusion. A clean percentage that mixes those two is not an evaluation. It is weather with a legend.

You know the costume, because I keep wearing it. I point a tiny coding harness at whatever model is cheap enough for a Tuesday, a number twitches, and I start narrating as if the model twitched it. Did the function miss the assertion, or did the box sit there, cough, and hand back half a diff? Same red cell. Different animal. Would you fail a unit test because the laptop slept? I hope not. I have done the eval version of that, and the chart looked rigorous while it was just the room temperature of a shared machine.

Demos keep dragging the agent closer to the browser, the desktop, the repo. Cute. The score still depends on which box answered. A free box answers like a shared kitchen, not like a lab bench you booked. If you do not write that down, you will explain a capacity hiccup as a reasoning failure, and you will "fix" a prompt that was never the leak.

So I started treating "free" as an environment variable. Not a personality. Not a quality tier with a halo. An environment, the way CI=true is an environment. It changes timing, error strings, and how often the door is locked. Score the plan and the model in one bucket, and you can no longer say which one moved.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The brief I am willing to repeat is narrow. There is a free model access path, and there is a free server option. I am not naming a model, printing a token quota, or promising that either offer still looks like a screenshot from last month. Allowances move, and a blog is a bad cache. If a number matters to your budget, read the project docs the morning you run this. I would rather omit a figure than ship a stale one.

Why touch a free path during evaluation at all? Harness bugs are cheaper when a clumsy call does not invoice you. I want a dummy that will sometimes be slow, sometimes be full, and sometimes return JSON my parser did not rehearse. A free model is a decent dummy for "can my scorer survive a weird completion." A free server is a decent dummy for "can my client survive a shared machine." Neither dummy is a leaderboard. The minute you want a number other people might quote, you have left the dummy, and you owe them a denominator.

What follows is a protocol, and I am marking it unexecuted against any live host. I have not filled this scorecard with a private run and then sanded it into confidence. The artifact you can reproduce today is the classifier and its tests. They run offline, on synthetic rows, on your laptop, with no key. The live hook is a sketch. Its payload is a placeholder you replace after you read the current API, because I will not invent a request shape and call it integration.

Four tasks. The fixture is small on purpose. One is a pure function with a hidden assertion, so a wrong return has somewhere to land. One demands a unified diff and rejects a paragraph of advice, because prose is how a model dodges a spec. One is fatter than the context you think you have, so truncation can show its face. One is a no-op probe, a ping, so a cold start is not blamed on your clever prompt. This is not an IQ test. It is a stethoscope. If you need a hundred tasks before you trust the plumbing, you are decorating a leak.

Every call becomes one row. Latency. Status. Whether the body parsed. Whether the assertion passed. And a kind, the column that ruins a pretty bar chart, which is why people drop it. Quota, timeout, cold, capacity, truncation, task, auth. Those words are not synonyms, even though they all feel like "the model failed" at 1 a.m. Only task may enter the pass rate. Everything else blocked the run. A blocked run is data about the environment. It is not a vote on the model.

I picked the cutoffs. They are knobs, not laws of physics. No HTTP status, or a non-200 that already burned twenty seconds, is a timeout. HTTP 200, eight seconds or more, and a body I cannot parse is filed as cold, so I remember to look at first-byte time later instead of cursing the spec. Anything else unparsed, on a fast 200, is truncation. 429 or quota language is the plan talking. 500 and the 502 family are capacity. 401 and 403 abort the meaning of the run, because an auth miss is not a model opinion. Keyword sniffing is brittle. You will edit it the first time a real error string laughs at you. That edit is the method, not a footnote you skip.

# tier_split.py — offline protocol. Thresholds are proposals, not measurements.
import json

TIMEOUT_S = 20.0
COLD_S = 8.0

def classify(status, body, latency_s, parsed_ok):
    low = (body or "").lower()
    if status in (401, 403):
        return "auth"
    if status == 429 or "quota" in low or "rate limit" in low:
        return "quota"
    if status in (500, 502, 503, 504) or "overloaded" in low or "at capacity" in low:
        return "capacity"
    if status == 0 or (status != 200 and latency_s >= TIMEOUT_S):
        return "timeout"
    if status != 200:
        return "capacity"
    if (not parsed_ok) and latency_s >= COLD_S:
        return "cold"
    if not parsed_ok:
        return "truncation"
    return "task"

def row(task_id, status, body, latency_s, parsed_ok, assertion_ok):
    kind = classify(status, body, latency_s, parsed_ok)
    scorable = kind == "task"
    return {
        "task_id": task_id,
        "latency_s": round(float(latency_s), 3),
        "status": status,
        "kind": kind,
        "scorable": scorable,
        "passed": bool(assertion_ok) if scorable else None,
    }

def summarize(rows):
    scorable = [r for r in rows if r["scorable"]]
    blocked = [r for r in rows if not r["scorable"]]
    hits = sum(1 for r in scorable if r["passed"])
    return {
        "n": len(rows),
        "scorable": len(scorable),
        "blocked": len(blocked),
        "task_pass_rate": (hits / len(scorable)) if scorable else None,
        "publishable": bool(scorable) and not blocked and len(scorable) >= 4,
        "blocked_kinds": sorted({r["kind"] for r in blocked}),
    }

def test_quota_is_not_a_model_miss():
    r = row("diff_only", 429, "quota exceeded", 0.4, False, False)
    assert r["kind"] == "quota" and r["passed"] is None and not r["scorable"]

def test_assertion_miss_stays_on_the_model():
    r = row("assert_add", 200, "{}", 2.1, True, False)
    assert r["kind"] == "task" and r["passed"] is False

def test_cold_is_not_a_fast_truncation():
    slow = row("probe", 200, "", 9.5, False, False)
    fast = row("long_ctx", 200, "truncated", 3.0, False, False)
    assert slow["kind"] == "cold" and fast["kind"] == "truncation"

def test_mixed_run_is_not_publishable():
    rows = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    summary = summarize(rows)
    assert summary["publishable"] is False
    assert summary["blocked_kinds"] == ["quota", "truncation"]
    # 1 of 2 scorable rows, not 1 of 4 calls. The denominator is the whole trick.
    assert summary["task_pass_rate"] == 0.5

if __name__ == "__main__":
    demo = [
        row("probe", 200, "{}", 0.3, True, True),
        row("assert_add", 200, "{}", 1.1, True, False),
        row("diff_only", 429, "quota exceeded", 0.2, False, False),
        row("long_ctx", 200, "truncated", 3.0, False, False),
    ]
    print(json.dumps(summarize(demo), indent=2))

Run the synthetic path before you even think about a host. The tests are the contract. A comment rots. A failing assert nags you the next time you are tempted to average the reds.

python tier_split.py
python -m pytest tier_split.py -q

The script prints a synthetic summary, not a product measurement. I am pasting it so the denominator is visible, and so nobody can pretend the fixture secretly timed a server.

{
  "n": 4,
  "scorable": 2,
  "blocked": 2,
  "task_pass_rate": 0.5,
  "publishable": false,
  "blocked_kinds": ["quota", "truncation"]
}

Look at that 0.5 and tell me what it means. It does not mean "the model scored fifty percent." It means one of the two calls the plan did not interrupt held its assertion. The other two rows never entered the denominator. Fold them in and you can manufacture 25 percent, or 50, or a story about truncation being a logic bug. Which story did you mean? If your dashboard still demands a single rate, it wants theater. I have published theater. The publishable flag is me trying to stop. Four scorable rows, zero blocked, or the run stays in the lab notebook.

Where does a free server earn its keep, if I refuse to invent a timing table for it? It earns its keep when the thing under test is your client. Retries that should be bounded. Parsers that should fail closed. Logs that should keep the status code instead of a smile. It breaks the moment the thing under test is the model under a stable load. Shared capacity wobbles. A cold start impersonates stupidity. A quota wall impersonates a refusal. Hold the model id constant and swap only the server, and you are still not holding the environment constant, because a free server is a crowd, not a box with your name on the door.

Who should walk away from this approach. If you are choosing a model for production codegen, this is a preflight, not the decision. If the repository is private, do not paste it into a server you have not reviewed just because the invoice is zero. Free is not a privacy policy. If you have a latency SLO, a sundial will not time your marathon, and a contended free box is that sundial. If you are about to post a leaderboard, do not. Not from four tasks, and not from an unstratified rate. And if you needed me to name a winner between hosts, this piece will disappoint you on purpose.

A live call, if you insist, stays outside the tests. I am not embedding a vendor path, and I am not claiming the JSON below is accepted by anyone. Set the URL from the docs you read today. Then edit the body until it matches. Until that edit exists, you do not have a client. You have a wish.

def live_call(url, model, prompt, timeout_s=30):
    import time, urllib.request
    payload = json.dumps({"model": model, "prompt": prompt}).encode()
    req = urllib.request.Request(
        url, data=payload, headers={"Content-Type": "application/json"}
    )
    t0 = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=timeout_s) as resp:
            raw = resp.read().decode("utf-8", "replace")
            return resp.status, raw, time.perf_counter() - t0
    except Exception as exc:
        return 0, str(exc), time.perf_counter() - t0

export EVAL_BASE_URL="https://example.invalid/v1/complete"
export EVAL_MODEL_ID="read-this-from-current-docs"
# Wire live_call yourself after the payload matches. Do not point pytest at the network.

Map the return straight into row. Parse first, then classify, then summarize. If kind is auth, stop the batch. More calls will not become more science. If kind is quota or capacity, you learned something about the plan's afternoon, and you learned almost nothing about the diff task. Write that sentence in the log. Future you, skimming at speed, will otherwise promote the red cell into a model opinion.

Limitations, before anyone quotes me. The fixture cannot see whether a model can hold a real repository in its head. Token accounting stays out until a response actually carries usage fields, and I will not invent those fields to look thorough. The keyword list will miss a new error string and misfile a row, which is why raw bodies belong next to the kind. Free access can shrink or vanish without this article updating. A green pytest run means the classifier obeyed the synthetic rows. It does not mean a host agreed with me. I am not reporting a measured win, or a measured break, of any free tier here, because I did not run that measurement for this piece. If I had, you would get the raw log, the date, the model id, and the docs URL. Not a vibe. Not a crown.

There is still a Monday use. Point the same client at a free model access path and a free server option, collect kinds instead of a single rate, and decide whether the harness is even ready for a paid, pinned run. That is the job. Shake the pipes. Do not grade the water.

If your harness already writes a row per call, point it at MonkeyCode's free model access and free server option after you confirm both still exist in the docs, and keep the blocked kinds. I would rather read those than another bare pass rate.

I Sent the Eval Off-Box. Pass Rate Started Measuring the Wire.

Jordan Liu — Wed, 23 Sep 2026 16:51:56 +0000

The scoreboard was lying, and not in a subtle way. I had a green local eval for a tiny tool-calling agent, then I pointed the same harness at a remote runner and watched “accuracy” swing like weather. The model had not gotten dumber between those two runs. The clock had moved, and I was still scoring dead sockets as wrong answers.

That is the whole claim. If you export an agent eval onto someone else’s box and collapse HTTP death into a single fail bit, you are not measuring the model. You are measuring the path. Local green, remote red, same prompt, same tool schema — and a dashboard that cannot tell the difference is just a mood ring with extra YAML.

I keep doing this to myself. A smoke eval looks honest on a laptop because the loopback interface does not take a coffee break. Then CI, or a shared runner, or a free remote box enters the story, and suddenly the agent “regressed.” Did it? Or did the handshake stall long enough for httpx to tap out at ten seconds while the model was still thinking?

The contamination is boring, which is why it wins

Agent evals already mix three different failures into one sad integer. The model can pick the wrong tool. The tool can return garbage. The transport can vanish. Only the first two belong in a quality score. The third belongs in a reliability column, and if you skip that split you will “fix” a model that was never broken.

Think of it as grading a student on whether the bus showed up. You can do that. You just should not call the result algebra.

I wanted a remote place to park the runner without standing up my own box, so I used MonkeyCode for the off-box side of this experiment. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating their free model access and free server option as a convenience layer — a reachable endpoint and a cheap generation path — not as a promise about hardware, quotas, permanence, or speed. The article still works if you swap in any other remote URL. The method is the point.

What I actually ran

The task is deliberately small. One user turn. One allowed tool, add_two, with two integers. A correct agent emits a well-formed tool call, the tool returns the sum, and the final message echoes that number. Anything else is a model miss. Timeouts, DNS, 5xx, empty bodies, and JSON that never showed up are transport misses. Mixing them is how pass rate becomes meteorology.

I did not chase a leaderboard. I chased a classifier I could rerun on Monday without arguing with a screenshot. The harness below is the artifact. Point LOCAL_BASE at a loopback stub and REMOTE_BASE at whatever free server or API root you actually have. Keep the prompt frozen. Keep the tool schema frozen. Change one thing: the wire.

# eval_wire.py — classify transport death vs model misses
from __future__ import annotations

import json, os, time, uuid
from dataclasses import dataclass, asdict
from typing import Any, Literal

import httpx

Outcome = Literal["pass", "model_fail", "transport_fail", "protocol_fail"]

TOOL = {
    "name": "add_two",
    "description": "Add two integers and return the sum.",
    "parameters": {
        "type": "object",
        "properties": {
            "a": {"type": "integer"},
            "b": {"type": "integer"},
        },
        "required": ["a", "b"],
    },
}

PROMPT = "Use add_two to compute 17 + 25. Then say only the sum."

@dataclass
class Trial:
    run_id: str
    target: str
    outcome: Outcome
    http_status: int | None
    ttfb_ms: float | None
    total_ms: float
    error: str | None
    tool_name: str | None
    tool_args: dict[str, Any] | None
    final_text: str | None


def classify(resp: httpx.Response | None, body: dict | None, timed_out: bool) -> Outcome:
    if timed_out or resp is None:
        return "transport_fail"
    if resp.status_code >= 500 or resp.status_code in {408, 429}:
        return "transport_fail"
    if resp.status_code >= 400:
        return "protocol_fail"
    if not body:
        return "protocol_fail"
    tool = extract_tool(body)
    text = extract_text(body)
    if tool and tool[0] == "add_two" and tool[1] == {"a": 17, "b": 25}:
        # Model did the job. Echoing the sum is extra credit, not the grade.
        return "pass"
    if tool or (text and text.strip()):
        return "model_fail"
    return "protocol_fail"


def extract_tool(body: dict) -> tuple[str, dict] | None:
    # Adapt this to your chat/completions shape. Do not pretend one schema rules them all.
    choices = body.get("choices") or []
    if not choices:
        return None
    msg = choices[0].get("message") or {}
    calls = msg.get("tool_calls") or []
    if not calls:
        return None
    fn = calls[0].get("function") or {}
    name = fn.get("name")
    try:
        args = json.loads(fn.get("arguments") or "{}")
    except json.JSONDecodeError:
        return None
    if not name:
        return None
    return name, args


def extract_text(body: dict) -> str | None:
    choices = body.get("choices") or []
    if not choices:
        return None
    return (choices[0].get("message") or {}).get("content")


def one_trial(client: httpx.Client, url: str, target: str, timeout_s: float) -> Trial:
    run_id = uuid.uuid4().hex[:8]
    payload = {
        "messages": [{"role": "user", "content": PROMPT}],
        "tools": [{"type": "function", "function": TOOL}],
        "tool_choice": "auto",
    }
    t0 = time.perf_counter()
    ttfb = None
    timed_out = False
    resp = None
    body = None
    err = None
    try:
        with client.stream("POST", url, json=payload, timeout=timeout_s) as r:
            resp = r
            first = next(r.iter_bytes(), b"")
            ttfb = (time.perf_counter() - t0) * 1000
            raw = first + r.read()
        body = json.loads(raw.decode() or "{}")
    except httpx.TimeoutException as e:
        timed_out = True
        err = type(e).__name__
    except (httpx.TransportError, json.JSONDecodeError) as e:
        err = type(e).__name__
    total = (time.perf_counter() - t0) * 1000
    tool = extract_tool(body) if body else None
    return Trial(
        run_id=run_id,
        target=target,
        outcome=classify(resp, body, timed_out or err == "TransportError"),
        http_status=None if resp is None else resp.status_code,
        ttfb_ms=None if ttfb is None else round(ttfb, 1),
        total_ms=round(total, 1),
        error=err,
        tool_name=None if tool is None else tool[0],
        tool_args=None if tool is None else tool[1],
        final_text=extract_text(body) if body else None,
    )


def main() -> None:
    local = os.environ["LOCAL_BASE"].rstrip("/") + "/v1/chat/completions"
    remote = os.environ["REMOTE_BASE"].rstrip("/") + "/v1/chat/completions"
    timeout_s = float(os.environ.get("EVAL_TIMEOUT_S", "12"))
    n = int(os.environ.get("EVAL_N", "20"))
    rows: list[Trial] = []
    with httpx.Client() as client:
        for target, url in (("local", local), ("remote", remote)):
            for _ in range(n):
                rows.append(one_trial(client, url, target, timeout_s))
    print(json.dumps([asdict(r) for r in rows], indent=2))
    for target in ("local", "remote"):
        subset = [r for r in rows if r.target == target]
        print(target, tally(subset))


def tally(rows: list[Trial]) -> dict[str, float | int]:
    n = len(rows) or 1
    def rate(o: Outcome) -> float:
        return round(100.0 * sum(r.outcome == o for r in rows) / n, 1)
    transport = [r.total_ms for r in rows if r.outcome == "transport_fail"]
    return {
        "n": len(rows),
        "pass_pct": rate("pass"),
        "model_fail_pct": rate("model_fail"),
        "transport_fail_pct": rate("transport_fail"),
        "protocol_fail_pct": rate("protocol_fail"),
        "transport_p50_ms": None if not transport else sorted(transport)[len(transport)//2],
    }

if __name__ == "__main__":
    main()

Run it like a test you might actually keep, not like a demo you screenshot once:

export LOCAL_BASE=http://127.0.0.1:8080
export REMOTE_BASE=https://YOUR_REMOTE_ROOT
export EVAL_N=20
export EVAL_TIMEOUT_S=12
python eval_wire.py > wire_trials.json

Need a loopback control so “local” is not a fantasy? A stub that always emits the correct tool call is enough. You are calibrating the classifier, not auditioning a model.

# local_stub.py — control plane, not a brain
from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/v1/chat/completions")
def completions(payload: dict):
    return JSONResponse({
        "choices": [{
            "message": {
                "role": "assistant",
                "content": None,
                "tool_calls": [{
                    "id": "call_local",
                    "type": "function",
                    "function": {
                        "name": "add_two",
                        "arguments": "{\"a\": 17, \"b\": 25}",
                    },
                }],
            }
        }]
    })

uvicorn local_stub:app --port 8080

If local pass rate is not ~100% against that stub, stop. Your parser is the bug. I have shipped that bug. It looks like a model story in every chart you will later regret.

How I read a run without lying to myself

I refuse a single accuracy number now. Four buckets, or I do not publish the row. Pass means the tool call matched. Model fail means the bytes arrived and the agent still flinched. Transport fail means the wire blinked. Protocol fail means we could not even parse the play — wrong shape, missing choices, arguments that are not JSON.

Here is an illustrative slice, labeled because it is a fixture for the method, not a product benchmark and not a promise about any host. I am showing the shape of the argument, not a trophy.

# illustrative fixture — replace with your wire_trials.json
local  {"n": 20, "pass_pct": 100.0, "model_fail_pct": 0.0,
        "transport_fail_pct": 0.0, "protocol_fail_pct": 0.0}
remote {"n": 20, "pass_pct": 55.0, "model_fail_pct": 15.0,
        "transport_fail_pct": 25.0, "protocol_fail_pct": 5.0}

Look at that remote line. A lazy dashboard screams 55% accuracy and someone starts swapping prompts. Split the buckets and a quarter of the pain is the wire. Another sliver is protocol drift. The model-shaped remainder is 15%, which is a different conversation than “the remote model is bad.” Same eval file. Different adult in the room.

Was the remote side slower? Sometimes. Is slower the same as wrong? Only if your timeout is a moral judgment. I keep EVAL_TIMEOUT_S in the log next to the tally because a 12-second knife and a 60-second knife do not cut the same fruit. Change the knife, change the score, and you still have not touched the weights.

One more cut that people skip: TTFB versus total time. If TTFB explodes and the body never arrives, that is not a bad tool call. That is a cold start, a queue, a proxy, a weekend. If TTFB is fine and the arguments are {"a": "seventeen"}, congratulations, you found an actual model miss. See the difference? One of those belongs in a routing ticket. The other belongs in a prompt diff.

When the free remote path is honest enough

I still use a free remote runner. I just do not let it author the grade. Smoke tests, schema checks, “does the tool name even come back,” overnight soak that I will read with the classifier on — that is fair. You get more attempts per unit of patience, and you find parser bugs faster than you find enlightenment.

I would not use it as the clock for a launch decision. I would not pour customer text into it. I would not treat a free model as a hidden gold judge and then act shocked when the judge wobbles. Cheap generation is a gift for iteration. It is a lousy supreme court.

If you need p99 promises, data residency, or a paper that pretends the network was not in the room, this approach will waste your week. Run local, or run a box you can ssh into, and keep the same classifier anyway. The classifier is the habit. The free server is just one place the habit can live.

A decision I actually use, said as sentences because a matrix makes people stop thinking. If transport_fail stays under a small slice and model_fail is stable across days, the remote path is good enough for smoke. If transport_fail moves when I change timeout and model_fail does not, I am holding a network thermometer. If protocol_fail spikes, I am on the wrong response shape and no amount of prompt poetry will help. If local stub is not a clean pass, I do not get to talk about models today.

What I will not claim

I am not going to invent a token ceiling, a GPU SKU, a durability story, or a speed ranking. Those claims go stale before the comment thread does, and I do not have a primary source in this draft that would survive a skeptic with a calendar. The reproducible part is the split: wire versus brain versus parser. You can run it on a free server, on a laptop, on a machine you regret buying. If the buckets do not exist, the percentage is theater.

So export the eval if you want. Just stop letting the socket sit for the exam. If you try the harness and the remote tally shocks you, good. That shock is the first honest number you have had all week. If you want a remote box for the runner without building one, the free server option I used is enough to get the classifier dirty — after that, the work is reading the buckets, not collecting a slogan.

I Timed the Tool Call. The Schema Still Said Yes.

Jordan Liu — Tue, 22 Sep 2026 15:08:36 +0000

A schema-valid tool call that would hang your API is not a pass. It is a delayed incident with nicer JSON. I stopped letting the schema be the whole exam, and the scoreboard got ugly in a useful way.

Most tool-calling writeups still grade the envelope. Did the model emit a function name you advertised. Did the arguments parse. Did required keys show up. That is a spelling test. Who grades the pitch clock.

Last week the feed was full of “how AI actually calls an API” explainers and a burst of posts about realistic API performance tests. Fine. Those are adjacent problems pretending they are the same lab. One is about whether the model can fill a form. The other is about whether the form, once submitted, survives contact with a slow, flaky, rate-limited endpoint. I wanted one harness that refused to separate them.

Here is the conclusion I actually trust: if your eval never times out, it is theater. The model can look brilliant while proposing a GET that your production budget would have killed at 800ms. The chart still paints a green box. You ship the green box. Then on-call inherits the clock.

I did not run a beauty contest against unnamed models. I built a 24-trace fixture pack with planted faults and a scorer that treats implied HTTP as part of correctness. The numbers below come from that pack. Swap live completions in when you have them. The scorer does not care who typed the JSON.

The fixture is a tiny order API, because weather demos never 504 in the stories we tell ourselves. Each trace is a tool call plus the latency I would have injected if that call had hit a real wire. Schema validity is necessary. It is not sufficient. If the implied request blows the budget, the call is wrong even when the JSON is pretty.

# toolcall_budget_eval.py
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

BUDGET_MS = 800
ALLOWED_TOOLS = {
    "get_order": {
        "method": "GET",
        "path": "/orders/{order_id}",
        "required": {"order_id"},
    },
    "cancel_order": {
        "method": "POST",
        "path": "/orders/{order_id}/cancel",
        "required": {"order_id", "reason"},
    },
}

@dataclass
class Verdict:
    trace_id: str
    schema_ok: bool
    args_ok: bool
    under_budget: bool
    passed: bool
    why: str

def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    for key in ("id", "tool", "arguments", "injected_latency_ms"):
        if key not in data:
            raise ValueError(f"{path} missing {key}")
    return data

def grade(trace: dict[str, Any], budget_ms: int = BUDGET_MS) -> Verdict:
    tool = trace["tool"]
    args = trace["arguments"]
    latency = int(trace["injected_latency_ms"])
    spec = ALLOWED_TOOLS.get(tool)
    if spec is None:
        return Verdict(trace["id"], False, False, latency <= budget_ms, False, "unknown_tool")
    missing = spec["required"] - set(args)
    extra_empty = any(args.get(k) in (None, "") for k in spec["required"])
    args_ok = not missing and not extra_empty
    schema_ok = isinstance(args, dict)
    under = latency <= budget_ms
    if not schema_ok:
        why = "args_not_object"
    elif not args_ok:
        why = f"bad_args:{sorted(missing) or 'empty'}"
    elif not under:
        why = f"over_budget:{latency}ms"
    else:
        why = "ok"
    return Verdict(trace["id"], schema_ok, args_ok, under, schema_ok and args_ok and under, why)

def grade_dir(folder: str) -> list[Verdict]:
    paths = sorted(Path(folder).glob("*.json"))
    return [grade(load_trace(p)) for p in paths]

if __name__ == "__main__":
    import sys
    rows = grade_dir(sys.argv[1] if len(sys.argv) > 1 else "traces")
    passed = sum(1 for r in rows if r.passed)
    schema_only = sum(1 for r in rows if r.schema_ok and r.args_ok)
    print(f"traces={len(rows)} schema_only={schema_only}/{len(rows)} budget_aware={passed}/{len(rows)}")
    for r in rows:
        flag = "PASS" if r.passed else "FAIL"
        print(f"{flag} {r.trace_id:12} schema={int(r.schema_ok)} args={int(r.args_ok)} budget={int(r.under_budget)} {r.why}")

That is the whole argument in a file. Schema and arguments can still win. The clock can still veto. I am tired of dashboards that never give the clock a vote.

The planted pack is boring on purpose. Eight traces are clean and fast. Four call a tool I never offered. Four omit reason on cancel. Four are perfect JSON aimed at an endpoint I delayed to 1200ms. Four mix a valid cancel with 400ms, which should pass, because I do not want a scorer that fails everything just to look strict. Strict and sad are not the same skill.

Drop them in traces/ like this.

{
  "id": "t-over-budget-03",
  "tool": "get_order",
  "arguments": {"order_id": "ord_19"},
  "injected_latency_ms": 1200
}

Run it locally. No GPU. No mystery SaaS.

python toolcall_budget_eval.py traces

On this pack the script prints traces=24 schema_only=20/24 budget_aware=12/24. Those eight extra “wins” under schema-only scoring are the delayed GETs. They look like competence. They are a 504 with etiquette. If you only count JSON, you will brag about 83 percent. If you count the wire, you are at 50 percent, and you should be. The missing 33 points were never intelligence. They were patience you do not have in production.

Is 800ms sacred. No. I picked it because it is mean enough to catch a frozen tool loop and loose enough that a healthy p95 still lives. Change the constant. The point is that a number exists. An eval without a number is a vibe.

I parked the judge next to the traces instead of next to a GPU I do not want to babysit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is the only reason I mention it: the scorer is local Python, and the free server is a convenient place to run the same command when my laptop is already compiling something noisier. I am not going to invent model names, token ceilings, or hardware I did not measure. If your workspace already has that free model and free server path, point it at traces/ and keep the budget in git. If it does not, the file above still runs.

Where does this break. Everywhere you hoped it would not. Injected latency is not real latency. A fixture cannot reproduce jitter, cold starts, or the way a gateway starts lying after 200 concurrent cancels. The scorer treats one call as the world. Agents retry. Retries multiply the budget you thought you had. I did not model that here on purpose. If I hide a retry loop inside “helpfulness,” we are back to theater, just with more curtains.

It also breaks when the tool schema is a lie. If get_order is documented as 200ms and the real handler does a sequential billing lookup, your budget is fan fiction. Grade the contract you actually ship, not the README. And it breaks when you let the same model write the traces and grade them. That is a student marking their own handwriting. Keep the pack human-authored, or at least frozen before the run.

Should you use this if you are demoing a chatbot that never leaves the browser. Maybe not. A UI toy can survive a slow tool call with a spinner and a shrug. Should you use this if your “API” is an in-process function with no network. No. You would be timing a function call and calling it operations. Should you use this as a model leaderboard. Please do not. Twenty-four planted traces are a unit test for your grading story, not a census of intelligence.

Use it when a tool call implies HTTP, when HTTP implies a budget, and when someone in the room is about to present schema pass-rate as quality. Ask them what happened to the calls that would have waited. If they cannot answer, the chart is decoration.

I keep the budget in the same commit as the traces. That is the whole workflow. Change the number, change the pack, watch schema-only and budget-aware diverge. When they stop diverging, either your endpoint got honest or your fixtures got timid. One of those is a win. Guess which.

If you already have a MonkeyCode workspace with the free model access and free server option, run the scorer there and keep the traces in git. Steal the file either way. The clock does not need a product to be right.

I Counted Drops as Wrongs. The Chart Was Theater.

Jordan Liu — Mon, 21 Sep 2026 13:21:10 +0000

The first number on an eval dashboard is usually a lie. Not a scam. A folding error. You asked a model for an answer, the path blinked, and your scorer filed the blink under incorrect.

I stopped scoring completions until I scored the envelope.

If you cannot tell a drop from a wrong, you are not ranking models. You are ranking weather. Shared free endpoints make the weather louder: idle processes, truncated streams, empty choices that still ride in on HTTP 200. The status looks fine. The grade looks like failure. That is how a leaderboard turns into theater.

I wanted a harness that fails loudly at the transport layer before it whispers at the semantics layer. So I planted six failure modes, ran a classifier over twenty-four envelopes, and watched a naive pass-rate collapse on purpose. The code below is the experiment. The percentages are the fixture talking, not a vendor scoreboard.

The folding error

Most agent evals I see still do some version of success = (http == 200 and expected in text). Cute. Also how you launder a timeout into a dumb model.

Think of it like grading a take-home by weighing the envelope. A missing envelope is not a wrong proof. A torn envelope is not a wrong proof. A proof written in the wrong schema is closer, but it is still not the same as a proof that multiplies instead of factors. Mix them and you will fire the student who had a postal strike.

Free inference is the postal strike. I do not mean that as an insult. I mean that a path you are not paying for will drop, stall, and truncate in ways a dedicated box will not, and your scorer has to admit that or it will keep "proving" that cheaper models are worse at arithmetic when they were worse at staying on the line.

That is why I care about a free model path and a free server option at all. Not because I want an adjective. Because I want a hostile mailbox. If the harness survives a mailbox, it will not panic on a quiet one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Their free model access and free server option are the mailbox I have in mind when I say "hostile." I am not naming models, quoting quotas, or claiming uptime. If you cannot reproduce a claim from the fixture in this file, treat the claim as unset.

Six envelopes, one bit

I keep six labels, and I refuse to let them share a boolean. drop is a refused connection, a reset, a client timeout, a 429, a 503. empty is a 200 with no choices or a null content string. truncated is a finish_reason of length, or JSON that dies mid-key. schema is parseable JSON that misses a required field. wrong is valid JSON with the wrong answer. right is the boring one.

Ask yourself what your current CSV does with empty. If the answer is "fail," you already know why last week's ranking felt jumpy. The model did not change. The path coughed, and you graded the cough.

I did not need a live fleet to test the scorer. I needed lies I controlled. Twenty-four envelopes, four of each label, stuffed into memory. The naive scorer treats anything that is not right as a model miss. The envelope scorer reports two numbers: yield, meaning bodies that were even gradeable, and accuracy-on-yield, meaning rights over rights plus wrongs.

Here is the whole trick. Yield answers "did the path hand me an object?" Accuracy-on-yield answers "was the object correct?" If you publish only the product of those, you are back to theater.

The planted run

Save this as envelope_eval.py. It never calls a network. That is the point. If a scorer cannot recover a lie you planted on disk, it will not recover a lie a free server plants for you at 4 p.m.

#!/usr/bin/env python3
"""Planted-envelope scorer. Run: python envelope_eval.py"""
from __future__ import annotations

import json
from collections import Counter
from dataclasses import dataclass
from typing import Any, Optional

@dataclass
class Envelope:
    http_status: int
    elapsed_ms: float
    finish_reason: Optional[str]
    body: Any
    expected: dict

def classify(env: Envelope) -> str:
    if env.http_status in {0, 408, 429, 500, 502, 503, 504}:
        return "drop"
    if env.http_status != 200:
        return "drop"
    if not isinstance(env.body, dict):
        return "empty"
    choices = env.body.get("choices") or []
    if not choices:
        return "empty"
    content = (choices[0].get("message") or {}).get("content")
    reason = env.finish_reason or choices[0].get("finish_reason")
    if content in (None, ""):
        return "empty"
    if reason == "length":
        return "truncated"
    try:
        parsed = json.loads(content) if isinstance(content, str) else content
    except json.JSONDecodeError:
        return "truncated"
    if not isinstance(parsed, dict) or "answer" not in parsed:
        return "schema"
    if parsed.get("answer") != env.expected.get("answer"):
        return "wrong"
    return "right"

def plant() -> list[Envelope]:
    expected = {"answer": 42}
    good = {"choices": [{"message": {"content": '{"answer": 42}'}, "finish_reason": "stop"}]}
    bad = {"choices": [{"message": {"content": '{"answer": 7}'}, "finish_reason": "stop"}]}
    missing = {"choices": [{"message": {"content": '{"value": 42}'}, "finish_reason": "stop"}]}
    cut = {"choices": [{"message": {"content": '{"answ'}, "finish_reason": "length"}]}
    empty = {"choices": []}
    rows: list[Envelope] = []
    for _ in range(4):
        rows.append(Envelope(0, 30000, None, None, expected))
        rows.append(Envelope(200, 800, "stop", empty, expected))
        rows.append(Envelope(200, 1200, "length", cut, expected))
        rows.append(Envelope(200, 900, "stop", missing, expected))
        rows.append(Envelope(200, 700, "stop", bad, expected))
        rows.append(Envelope(200, 650, "stop", good, expected))
    return rows

def main() -> None:
    labels = [classify(env) for env in plant()]
    counts = Counter(labels)
    n = len(labels)
    naive = sum(lab == "right" for lab in labels) / n
    gradeable = counts["wrong"] + counts["right"]
    acc = counts["right"] / gradeable if gradeable else float("nan")
    print("counts:", dict(counts))
    print(f"naive_pass_rate={naive:.3f}")
    print(f"yield={gradeable / n:.3f}")
    print(f"accuracy_on_yield={acc:.3f}")

if __name__ == "__main__":
    main()

Run it.

python envelope_eval.py

You should get four of each label, naive_pass_rate=0.167, yield=0.333, accuracy_on_yield=0.500. That naive 16.7% is true and useless. It says the system is bad at the task. It is not. The system is 50% accurate on the eight gradeable semantic answers, and sixteen of twenty-four envelopes never earned a semantic grade. If I had shipped the naive number, I would have "proven" a model regression that was actually four drops, four empties, four truncations, and four schema misses I planted myself.

Would you publish 16.7% in a weekly review? I have. That is the part that still bothers me.

What I do with each label

I keep a small matrix next to the scorer so I cannot "just retry it" at 11 p.m. and call that science.

Label	Counts as model evidence?	Action
drop	no	retry with a budget, then park the row
empty	no	retry once, then park
truncated	no	raise max tokens or repair, cap the repairs
schema	no	repair or fail the harness, not the model
wrong	yes	score it, do not retry
right	yes	score it, do not retry

Retry only drops and empties. Repair only truncated and schema, and cap the repairs so you do not invent a second model with your JSON fixer. Score only wrong versus right. Never retry a wrong. Wrong is information. A retry on a wrong is how you accidentally bake temperature into a pass@1 and then argue with your own chart.

Where a free path helps, and where it breaks

I use a free model and a free server as a canary for the harness, not as a verdict on the product under test. The canary's job is to produce ugly envelopes on a weekday afternoon. If my classifier labels them, the paid run gets a cleaner CSV. If my classifier folds them, I am about to buy myself a fake regression.

It breaks in the obvious ways. A free path is not a latency SLA. It is not a promise that temperature 0 is deterministic. It is not a substitute for a held-out human grade. If your paper needs matched seeds across labs, do not use a shared free box as the official judge. If you are on a deadline and you cannot afford a thirty-second stall before a drop, this is the wrong mailbox. If you need to claim "the model scored 71%," this article will not give you that sentence, and neither should a canary.

I also do not use the same endpoint for candidate and judge when I can avoid it. Shared load couples their clocks. The judge gets slow, your client times out, and you file a drop as a disagreement. That loop is how evals invent opinions. Two mailboxes. One grade.

What I log now

Every row gets http_status, latency_ms, finish_reason, bytes, label, and only then answer. The live probe looks like this, and I keep --max-time boringly small so the client timeout is a fact instead of a hang.

curl -sS -D /tmp/hdr.txt --max-time 30 \
  -H "content-type: application/json" \
  --data @prompt.json \
  "$ENDPOINT/v1/chat/completions" | tee /tmp/body.json | \
  python -c "import json,sys; d=json.load(sys.stdin); c=d.get('choices') or []; print({'n_choices': len(c), 'finish': c[0].get('finish_reason') if c else None, 'bytes': len(json.dumps(d))})"
wc -c /tmp/body.json

If n_choices is 0, I do not regex the content for the golden string. There is no content. Stop pretending. HTTP 200 is a door. It is not a grade.

Who should ignore this? If you already isolate transport in your eval warehouse, you do not need my JSONL. If you are demoing a toy prompt in a notebook, you do not need six labels. If you are trying to make a free server look like a dedicated GPU, you are solving a press-release problem, and this workflow will not help you.

I still run the planted file before I point the same scorer at anything with a bill, or anything without one. Steal the classifier first. If you want a mailbox that will actually drop on you while you do it, MonkeyCode's free model access and free server option are one place to aim the canary. Read the labels. Do not publish the folded number.

I Subtracted the Queue. The Fast Model Wasn't.

Jordan Liu — Sun, 20 Sep 2026 11:26:58 +0000

The number on your latency dashboard is not a model score. It is a traffic report wearing a lab coat.

I keep watching people rank free endpoints by time-to-first-token as if TTFT were a property of weights. It isn't. Not when the box is shared. Not when your so-called eval is standing in the same line as everyone else's autocomplete. You are scoring the parking lot. Then you publish a leaderboard. Then you act shocked when Tuesday disagrees with Thursday.

What happens when the tests we use to measure models get slower than the models? The tests start measuring the hallway.

I wanted a splitter I could run twice and trust. Wait on one side. Generation on the other. If wait dominates, I throw the run out. If generation dominates, I keep it. That is the whole method. It is boring on purpose. Boring is how you stop lying to yourself.

The lab for this is not a secret GPU in a closet. MonkeyCode is an open source project with free model access and a free server option, which is exactly the shape of infrastructure that makes queue contamination loud. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not naming a model, a quota, a SKU, or a duration I cannot verify. I am naming a measurement error you can reproduce on any shared endpoint, including that one.

Picture the failure. You send the same prompt ten times. Naive wall clock swears run 3 was a genius and run 7 was a brick. Split the timestamps and run 3 sat in queue for a blink while run 7 waited for a bus. Tokens per second never moved. Your ranking did. Are you still averaging those ten numbers like they mean the same thing?

I treat three clocks as sacred. t_submit is the instant the client writes the last request byte. t_first is the instant the first response byte lands. t_done is the instant the stream closes. Wait is first minus submit. Work is done minus first. Contamination is wait divided by wait plus work. If that ratio is high, the run is weather, not intelligence.

You want code, not a sermon. Fine. This harness is the artifact. The fixture traces are labeled samples, not a product benchmark I pretended to run.

# queue_split.py — labeled fixture + splitter. Not a vendor scorecard.
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Trace:
    run_id: str
    t_submit: float
    t_first: float
    t_done: float
    prompt_hash: str
    streaming: bool
    cold: bool

    @property
    def wait_s(self) -> float:
        return max(0.0, self.t_first - self.t_submit)

    @property
    def work_s(self) -> float:
        return max(0.0, self.t_done - self.t_first)

    @property
    def wall_s(self) -> float:
        return max(0.0, self.t_done - self.t_submit)

    @property
    def contamination(self) -> float:
        total = self.wait_s + self.work_s
        return 0.0 if total == 0 else self.wait_s / total

    def verdict(self, discard_at: float = 0.6) -> str:
        if not self.streaming:
            return "UNSPLITTABLE"
        if self.contamination >= discard_at:
            return "DISCARD"
        if self.contamination >= 0.2:
            return "WORK_ONLY"
        return "WALL_OK"


# Illustrative traces only. Same prompt_hash on purpose.
FIXTURE = [
    Trace("clean", 0.00, 0.12, 1.40, "p1", True, False),   # short wait, steady work
    Trace("queued", 0.00, 2.10, 3.38, "p1", True, False),  # long wait, SAME work interval
    Trace("slow", 0.00, 0.15, 3.05, "p1", True, False),    # short wait, actually slow generate
    Trace("buffered", 0.00, 2.80, 2.81, "p1", False, True), # non-streaming: clocks collapse
]


def rank(traces, key):
    return [t.run_id for t in sorted(traces, key=key)]


if __name__ == "__main__":
    streamed = [t for t in FIXTURE if t.streaming]
    print("naive wall rank:", rank(streamed, lambda t: t.wall_s))
    print("work-only rank:", rank(streamed, lambda t: t.work_s))
    for t in FIXTURE:
        print(
            f"{t.run_id:9} wait={t.wait_s:.2f}s work={t.work_s:.2f}s "
            f"contam={t.contamination:.2f} -> {t.verdict()}"
        )

Run it locally. No credentials. No cloud required for the lie detector itself.

python queue_split.py

You should see the reversal immediately. Naive wall clock ranks clean, then slow, then queued. Work-only ranking puts clean and queued in the same neighborhood and leaves slow as the actual drag. That flip is the bug. buffered comes back UNSPLITTABLE because a one-shot body gives you a single thud, not two clocks. If your eval client does that, stop calling it TTFT. You do not have TTFT. You have a door slam.

I pipe real calls through the same dataclass. The HTTP client is deliberately ugly. Environment variables so this article does not invent a hostname, a model id, or a token budget.

# live_probe.py — unexecuted template. Fill env. Do not treat output as a brand score.
import hashlib, json, os, time, urllib.request
from queue_split import Trace

URL = os.environ["EVAL_URL"]          # your endpoint, not mine
BODY = os.environ.get("EVAL_BODY", '{"prompt":"ping","stream":true}')

def stream_once(run_id: str, cold: bool) -> Trace:
    payload = BODY.encode()
    req = urllib.request.Request(URL, data=payload, method="POST")
    req.add_header("Content-Type", "application/json")
    t_submit = time.perf_counter()
    t_first = None
    with urllib.request.urlopen(req, timeout=120) as resp:
        while True:
            chunk = resp.read(64)
            now = time.perf_counter()
            if not chunk:
                break
            if t_first is None:
                t_first = now
        t_done = time.perf_counter()
    if t_first is None:
        t_first = t_done
    digest = hashlib.sha256(payload).hexdigest()[:12]
    streaming = b"stream" in payload and b"true" in payload.lower()
    return Trace(run_id, t_submit, t_first, t_done, digest, streaming, cold)

if __name__ == "__main__":
    rows = [stream_once(f"r{i}", cold=(i == 0)) for i in range(10)]
    print(json.dumps([r.__dict__ | {
        "wait_s": r.wait_s, "work_s": r.work_s,
        "contamination": r.contamination, "verdict": r.verdict(),
    } for r in rows], indent=2))

export EVAL_URL="http://127.0.0.1:8080/v1/stream"   # local stand-in
export EVAL_BODY='{"prompt":"ping","stream":true}'
python live_probe.py

Do I always get a streaming socket? No. If the free server buffers the whole completion, t_first and t_done collapse, and wait swallows work. Then the splitter cannot save you. That is a limitation, not a footnote. Non-streaming free servers make this method weaker, not stronger. If your loop expects one JSON blob, you are timing a black box. Split nothing. Say so out loud.

How do I decide keep versus discard? I do not worship a magic cutoff. I look at the batch. If contamination sits under about 0.2, I treat wall clock as mostly work. If it wanders from 0.2 to 0.6, I report work time and I print the wait column in the same breath so nobody can hide. If a run clears 0.6, I discard it for ranking. I still log it. Hidden discards are how people fake a calm night.

Is 0.6 science? No. It is a fence I can defend in a code review. You can move the fence. You cannot skip the column. Would you ship a unit test that asserts elapsed < 2.0 on a machine you do not own? Then why is your model eval doing that?

I also refuse to mix cold and warm. The first call after a pause is a different animal. I tag it. I do not average it with the next nine. People love n=10 until you ask which one was the kettle boiling. A free server after idle is a library that has to turn the lights on. That is not a regression in the book.

Where this holds together: identical prompts, streaming responses, a client you control, and a reason to compare two setups on the same night. Where it falls apart: tool-call agents whose "first token" is an opening brace, judges that treat latency as quality, batches that span a traffic cliff, and any protocol that cannot show you the first byte. A free server is a city street. Rush hour is not a model regression. Off-peak is not a breakthrough.

Should you use this if you are publishing a public model card? Only as a hygiene check, never as the headline metric. Should you use this if you are shopping dedicated hardware? No. Buy a stopwatch that is not shared. Should you use this if your eval is a chatbot screenshot and a vibe? Please stop. The method will not fix a missing clock.

I still like free inference. I like it the way I like a public library. Useful. Crowded. A terrible place to time how fast you read. An open source project plus a free server is a gift for iteration. It is a trap for stopwatch science. Hold both thoughts in the same head.

If you already have a slot on that free server, do not start with a blog graph. Start with one prompt, ten streamed calls, and the wait column. If wait moves and work does not, your old dashboard was theatre. That is the only invitation I have.

I Imported the Patch. That's When It Died.

Jordan Liu — Sat, 19 Sep 2026 09:31:28 +0000

I imported the patch. That's when it died.

ast.parse had already smiled. The chat had already called the work complete. Neither of those is a gate I trust. If your coding-model eval still stops at "looks like Python," you are grading handwriting, not software.

Why does this keep happening? Because a lot of us are still scoring the story of a fix. Fluency is cheap. Import is not. I wanted a number I could defend without recruiting a second model to compliment the first one.

So I built a three-gate harness: parse, import, test. Same fixture. Same patch. Three increasingly rude questions. The code below is the experiment. You can run it without a GPU and without believing a single adjective.

The fixture is small on purpose

I do not need a web app to catch a liar. I need a module that parses, then explodes when Python executes the top level, then fails a real assertion if you get past that. Parse-time green is a vanity metric. Import is the first honest gate.

# shop.py  — broken on purpose
from decimal import Decimal, ROUND_HALF_UP

TAX = None  # populated only if you actually import this file correctly

def require_tax_table():
    if TAX is None:
        raise RuntimeError("tax table was never loaded")
    return TAX

# Module-level landmine. ast.parse will not run this.
require_tax_table()

def add_tax(cents: int) -> int:
    rate = require_tax_table()
    money = Decimal(cents) * (Decimal("1") + rate)
    return int(money.to_integral_value(rounding=ROUND_HALF_UP))

# test_shop.py
from shop import add_tax

def test_add_tax_rounds_half_up():
    assert add_tax(100) == 108
    assert add_tax(1) == 1

See the trap? ast.parse never calls require_tax_table(). A model can rewrite add_tax with gorgeous syntax and still leave the import path haunted. That gap is what I wanted to count. Chat will not count it for you.

The intended fix is boring. Load a Decimal("0.08") tax table before anyone calls add_tax, keep the rounding, do not invent a helper that is not in the file. Boring is the point. I am not grading creativity. I am grading whether the module exists in a Python process.

I ask for a diff. I refuse to read it.

The prompt is hostile on purpose. If the model wants to narrate, it can narrate into a zero.

Return a unified diff for shop.py only.
Do not explain. Do not wrap the diff in markdown.
The file must import and test_add_tax_rounds_half_up must pass.
Tax rate is 8%. Use decimal rounding already in the file.

Then I do not read the completion. I extract. I apply. I score. If that sounds unfriendly, good. An oracle should be unfriendly.

import re, subprocess, sys, tempfile, shutil, ast
from pathlib import Path

DIFF_RE = re.compile(r"(?ms)^--- .*^\+\+\+ .*?^(?=\Z)", re.M)

def extract_unified_diff(text: str) -> str | None:
    if text.lstrip().startswith("--- "):
        return text
    blocks = re.findall(r"```

(?:diff|patch)?\n(.*?)

```", text, re.S)
    for b in blocks:
        if b.lstrip().startswith("--- "):
            return b
    return None  # essays get nothing. no pity parse.

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None. Score zero at gate zero. No partial credit for confidence. I used to "helpfully" take the first code fence. That is how you score markdown as Python and then convince yourself the model improved.

Applying the patch is equally unromantic:

def apply_diff(src: Path, diff: str) -> Path | None:
    tmp = Path(tempfile.mkdtemp())
    shutil.copytree(src.parent, tmp, dirs_exist_ok=True)
    patch = tmp / "change.diff"
    patch.write_text(diff)
    r = subprocess.run(
        ["patch", "-p1", "--batch", "-d", str(tmp)],
        input=diff, text=True, capture_output=True,
    )
    return tmp if r.returncode == 0 else None

Need patch on the machine. That is a feature. If your eval loop cannot survive a 1970s Unix tool, it is not ready to survive a model that emits @@ headers for files you never asked it to touch.

Three gates, one script

def gate_parse(py: Path) -> bool:
    try:
        ast.parse(py.read_text())
        return True
    except SyntaxError:
        return False

def gate_import(py: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-c", f"import sys; sys.path.insert(0, {py.parent.as_posix()!r}); import shop"],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def gate_test(root: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(root / "test_shop.py")],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def score(completion: str, fixture: Path) -> dict:
    diff = extract_unified_diff(completion)
    if not diff:
        return {"extract": False, "parse": False, "import": False, "test": False}
    root = apply_diff(fixture, diff)
    if root is None:
        return {"extract": True, "parse": False, "import": False, "test": False}
    py = root / "shop.py"
    parsed = gate_parse(py)
    imported = parsed and gate_import(py)
    tested = imported and gate_test(root)
    return {"extract": True, "parse": parsed, "import": imported, "test": tested}

I run it against checked-in completions, not against a vibe:

python score_patch.py fixtures/essay.txt
python score_patch.py fixtures/parse_only.txt
python score_patch.py fixtures/import_ok_test_fail.txt
python score_patch.py fixtures/pass.txt

Those four files are oracles for the scorer. They are not a vendor bake-off. If my grader cannot separate them, I do not get to grade a model yet. The labeled output below is from those fixtures, which you can paste yourself. It is not a claim about any hosted model.

# fixture-driven scorer output (not a product benchmark)
essay                 extract=0 parse=0 import=0 test=0
parse_only            extract=1 parse=1 import=0 test=0
import_ok_test_fail   extract=1 parse=1 import=1 test=0
pass                  extract=1 parse=1 import=1 test=1

parse_only is the one that should make you nervous. It is valid syntax. It still cannot be imported. An LLM judge will often call it solid. ast.parse will call it fine. Python will not.

What the gates actually argue about

Parse-only completions are empty calories. They look like engineering. They are essays wearing parentheses. I used to stop there because it felt quantitative. It was not.

Import failures are more interesting. The model invents load_rates(), or it leaves a name it saw in a comment, or it deletes the landmine and forgets to put a tax table back. Fluency stays high. The process dies before pytest exists. That is not a "almost." That is a format error with extra steps.

Test failures after a clean import are the only failures I treat as the model having engaged the problem. Everything before that is packaging. I blurred those categories for months. That was on me, not on the model.

Want a one-liner while you iterate on the extractor?

python -c "import ast,pathlib; ast.parse(pathlib.Path('shop.py').read_text()); print('parsed')"
python -c "import shop" || echo "imported? no"
python -m pytest -q test_shop.py

Three commands. Three different truths. If your dashboard collapses them into one thumbs-up, your dashboard is lying with better typography.

Where a cheap generation loop helps, and where it breaks

I still need something to emit candidate diffs. Burning a paid endpoint to discover that a pile of samples never import is how you light money on fire. The mechanical grader is the expensive-looking part that is actually free. Generation is the part that tempts you to skip gates because "it read well."

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode only as a generation pipe: free model access and a free server option, so the loop can throw diffs at this harness without me standing up inference. The scorer does not call it. Delete the pipe and the gates still work. That is the only honest integration I can defend.

Where that setup performs well: volume. You can park messy completions against parse/import/test until the failure taxonomy stops moving. You are paying in attention, not in GPU mythology. Where it breaks: the free path will not save a weak fixture. If your test is assert True, import becomes theater again. Unstructured completions wrap diffs in sermons; if you relax the extractor, you will grade the sermon. Truncation dies at parse or import, which is good, unless you mark those as "needs review" instead of fail. Do not do that. Fail is a complete sentence.

Limitations, out loud

This harness does not measure taste. It will pass a brutal one-liner that satisfies test_add_tax. It will fail a correct redesign that renames the function. It is hostile to refactors. That is a bias, not a virtue. It also will not catch a wrong tax rate if you never asserted 108. Weak tests make import look like enlightenment.

Do not use this if you are evaluating docs, architecture notes, or UI copy. Do not use it as a hiring rubric. Do not use it to stamp "production ready" on a model because gate three went green on one file. If you need semantic review, add a linter, then a type checker, then stop before you add another LLM as judge. The last time I let a model grade a model, pretty prose still moved the score.

I also do not claim these gates outlive a moving product surface. Free access and a free server are operator-supplied availability notes, not a promise about models, hardware, duration, or quota. If the pipe changes, the scorer should not care. If the scorer cares, you coupled the wrong things.

The ranking that flipped

Once I ignored chat, my personal ranking of "this completion looks good" collapsed. The essay I would have merged in a hurry scored zeros. The ugly diff that never said "sure" imported and passed. That is the whole lesson, and it does not require a leaderboard.

So I am done asking whether AI outgrew our tests. I ask whether our tests ever loaded the code. Most days they did not. Parse is polite. Import is not. I grade the rude one.

I Shuffled the Judge Prompt. The Scores Should Have Held.

Jordan Liu — Fri, 18 Sep 2026 07:52:14 +0000

Your eval is not a measurement if the same trace gets a different grade when you move it down the list. I stopped arguing about model quality and started measuring judge flip-rate. That number is uglier than a single accuracy score, and it is the one I actually ship against.

We keep building agents and then grading them with another model, usually the cheapest one that will answer JSON on a Tuesday. Does that sound like engineering, or like asking a tired intern to re-grade the same stack of papers after someone shuffled the pile? I wanted a protocol that fails loudly when the intern is guessing.

The last week of DEV commentary kept circling a real itch: we outgrew the tests we brag about. I am not here to litigate vibe coding. I am here because my scoring harness was agreeing with itself until I changed the order of the fixtures. Order should be a no-op. It was not.

So I wrote a judge that I can lie to on purpose. If a stub that keys off list position can look “consistent” in a naive run, a free model can too. The trick is to run the same traces through the same rubric more than once, with the deck cut differently each time, and count the flips.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I needed a lot of cheap, boring repeats—same rubric, same traces, different order. MonkeyCode’s free model access and free server option are relevant for that grind if you treat the endpoint as a judge, not as a coauthor. I am not claiming a quota, a model name, or a latency number I did not measure. The interesting part is the flip-rate script.

Here is the fixture set I actually version. Three traces, labeled in git, not in my memory. One is a clean tool success. One is a fluent wrong answer. One is a partial apply that talks like a merge.

# fixtures.py — canned traces, not production logs
FIXTURES = [
    {
        "id": "t_ok",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Applied patch to src/api.py. Ran pytest: 12 passed.",
        "gold": "pass",
    },
    {
        "id": "t_fluent_fail",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Refactored naming across the service layer. Ready to ship.",
        "gold": "fail",
    },
    {
        "id": "t_partial",
        "task": "Rename getUser to get_user in src/api.py and keep tests green.",
        "trace": "Updated the function. Tests still import getUser. Explained why that is fine.",
        "gold": "fail",
    },
]

Gold labels live next to the traces. If I let the model invent the labels, I am scoring a mirror. You already know how that movie ends, right?

The rubric is short on purpose. Long rubrics make judges sound careful while they still wobble. I ask for a label and a one-line reason, then I throw the reason away for the metric. Reasons are for humans. Flip-rate is for the harness.

RUBRIC = """Score the agent trace against the task.
Return JSON only: {"id": str, "label": "pass" or "fail", "reason": str}.
pass = the trace shows the requested change and a test run that could have caught a miss.
fail = missing change, missing tests, or a story that substitutes for evidence.
"""

Now the part people skip. I do not call the judge once. I call it through permutations. Three fixtures is six orders. That is small enough to read and large enough to catch a positional toady.

import hashlib, itertools, json, os, urllib.request
from fixtures import FIXTURES, RUBRIC

ENDPOINT = os.environ["JUDGE_URL"]  # your free server or any OpenAI-shaped chat URL
MODEL = os.environ.get("JUDGE_MODEL", "default")

def chat(messages):
    body = json.dumps({"model": MODEL, "messages": messages, "temperature": 0}).encode()
    req = urllib.request.Request(
        ENDPOINT, data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]

def parse_label(raw, fallback_id):
    try:
        obj = json.loads(raw)
        label = obj.get("label", "fail")
        return fallback_id, "pass" if label == "pass" else "fail"
    except Exception:
        return fallback_id, "fail"  # unparseable output is a fail, not a retry poem

Temperature zero is not a personality. It is a request. Free endpoints still drift. That is the point of repeating the trial, not a reason to skip it. If your client retries on parse failure, cap it. Infinite repair is how a judge learns to write JSON and forget the grade.

def judge_batch(order):
    blob = json.dumps([{"id": t["id"], "task": t["task"], "trace": t["trace"]} for t in order])
    raw = chat([
        {"role": "system", "content": RUBRIC},
        {"role": "user", "content": blob},
    ])
    # Expect one JSON object per trace. If the model merges them, every id is fail.
    labels = {}
    for t in order:
        labels[t["id"]] = "fail"
    try:
        parsed = json.loads(raw)
        rows = parsed if isinstance(parsed, list) else [parsed]
        for row in rows:
            i = row.get("id")
            if i in labels:
                labels[i] = "pass" if row.get("label") == "pass" else "fail"
    except Exception:
        pass
    return labels

def flip_rate(runs):
    ids = [t["id"] for t in FIXTURES]
    flips = 0
    pairs = 0
    for i in ids:
        labels = [run[i] for run in runs]
        pairs += 1
        if len(set(labels)) > 1:
            flips += 1
    return flips / pairs

I also keep a dishonest judge in the repo. It is a flashlight. If I wire the pipeline to a function that passes whatever arrived last, the flip-rate should scream. If it does not scream, I am measuring the wrong thing.

def positional_stub(order):
    # Last item always "pass". A real metric must punish this.
    labels = {t["id"]: "fail" for t in order}
    labels[order[-1]["id"]] = "pass"
    return labels

def identity_stub(order):
    # Stable on id. Flip-rate must be 0.0 even when shuffled.
    return {t["id"]: ("pass" if t["id"] == "t_ok" else "fail") for t in order}

if __name__ == "__main__":
    orders = list(itertools.permutations(FIXTURES))
    stub_flips = flip_rate([positional_stub(list(o)) for o in orders])
    id_flips = flip_rate([identity_stub(list(o)) for o in orders])
    print(f"positional_stub flip_rate={stub_flips:.2f}")  # expect 1.00
    print(f"identity_stub flip_rate={id_flips:.2f}")      # expect 0.00

Run that before you spend a single token. I mean it. positional_stub flip_rate=1.00 and identity_stub flip_rate=0.00 are the only “benchmarks” in this article that I will stand behind without a live endpoint, because they are functions of the fixtures. If those two lines move, your math is broken and you should not grade a model yet.

The live path is the same loop with judge_batch. I do not paste a leaderboard from a free endpoint I did not freeze. You should not either. Print the per-id label vectors. If t_fluent_fail is pass in two orders and fail in four, you do not have a score. You have a weather report.

export JUDGE_URL="http://127.0.0.1:8000/v1/chat/completions"
python -c "from fixtures import FIXTURES; print(len(FIXTURES), 'fixtures')"
python judge_flip.py

What does “good enough” look like for a cheap judge? I use a boring gate, not a press release. Flip-rate on this three-trace deck must be 0.00 across all six orders. Agreement with gold must be 3/3 on every order, not on average. Averages hide the fluent miss, and the fluent miss is the whole crime.

Think of it like a kitchen scale. If the same bag of flour reads 500g, then 470g, then 510g because you rotated the bowl, you do not publish a recipe. You fix the scale. Free inference is useful here because repeats are the experiment. One beautiful grade is a vibe. Six ugly grades are data.

Where it breaks is predictable if you have ever watched a model try to be helpful. Batch JSON collapses into one object. The last trace inherits a pass because the prose sounded finished. The rubric gets paraphrased in the reason field and ignored in the label. A free server that is busy will timeout, and if you retry without a bound you will train yourself to accept the first parseable sentence. Bound the retries. I cap parse repair at one. After that it is fail. Silence is a grade.

There is a second failure I only caught when I hashed the prompt. I had a comment in the user blob that said “expected fail.” The judge, being a good student, agreed. Strip your gold from the request. If the model can see the answer, you are not evaluating an agent. You are evaluating obedience.

def prompt_fingerprint(order):
    body = json.dumps([t["id"] for t in order]) + RUBRIC
    return hashlib.sha256(body.encode()).hexdigest()[:12]

I log that fingerprint next to each run. When someone “improves” the rubric on a Friday, the hash moves and the old flip-rate is void. That is the whole discipline. Git the rubric. Git the fixtures. Do not git a screenshot of a chat.

Limitations, because this protocol is easy to oversell. Three traces will not bless a product. Temperature zero is not determinism. A free server is not a frozen judge checkpoint. If you need a number for a paper, a customer report, or a safety case, stop. Hire a stable judge, pin the revision, and put humans on the fluent failures. This workflow is for catching a harness that lies to its owner.

Who should not use it? Anyone averaging labels across shuffles and calling the mean “accuracy.” Anyone whose agent can mutate the fixtures. Anyone hiding the endpoint behind a retry loop that also rewrites the rubric. And anyone who thinks a free tier is a reason to skip gold labels. Free means you can afford the repeats. It does not mean the repeats are optional.

I still want cheap judges. I want them for the same reason I want cheap unit tests: I will actually run them. If you already have a free model endpoint or a free server sitting around, point JUDGE_URL at it and make the stubs fail first. The CTA is the script, not a tour. If the flip-rate is not zero, you do not have evaluation. You have a conversation that learned to export CSV.

I Scored Patches That Never Applied

Jordan Liu — Thu, 17 Sep 2026 05:53:19 +0000

A free model that emits a unified diff is not a code change. It is a hypothesis with nice lighting. I score the hypothesis against git apply --check and a compile command before I let anyone call it engineering.

Why this gate? Because the last seven days of AI talk keep collapsing two different jobs into one sentence. Generating text that resembles a fix is cheap. Landing a fix that the tree will accept is still work. If your review process stops at “looks right,” you are scoring theater.

I learned this the loud way. A patch can be fluent, confident, even polite, and still fail to apply. Hunk context drifts. A file got renamed last Tuesday. The model “fixed” a helper that no longer exists. The comment said done. The working tree said otherwise. Sound familiar?

So I stopped scoring vibes. I score apply. Then I score compile. Everything else is commentary.

The unit under test is the patch, not the paragraph

Think of a model output like a shipping label on an empty box. The label can be perfect. The box can still be empty. git apply is the person at the dock who actually opens it.

I keep a tiny fixture repo and a scorer. The model — any model — has one job: return a unified diff against a known commit. My job is not to admire the diff. My job is to ask four questions the model cannot grade for itself. Did the diff parse? Did it apply cleanly? Did the project still compile? Did the patch touch files I never named?

That last one matters more than people admit. Free inference is generous with extra opinions. A one-line request becomes a courtesy refactor in a file you did not open. The blast is quiet until CI yells.

I run the loop on a throwaway worktree so a bad hunk cannot vandalize the branch I actually care about. Cheap isolation. Honest scores.

A scoring harness you can run

This is a method, not a trophy. The numbers below come from the fixtures in this article, not from a secret leaderboard and not from a production bill. If you paste this into a real repo, your scores will move. That is the point.

#!/usr/bin/env python3
"""Score a unified diff as a patch, not as prose.

Labeled example: fixtures only. Do not treat printed totals as product benchmarks.
"""
from __future__ import annotations

import subprocess
from dataclasses import dataclass
from pathlib import Path

@dataclass
class PatchScore:
    parsed: bool
    apply_ok: bool
    compile_ok: bool | None
    files_touched: int
    unsolicited_files: int
    hunks: int
    notes: str

    def shipping_unit(self) -> bool:
        compile_pass = True if self.compile_ok is None else self.compile_ok
        return self.parsed and self.apply_ok and compile_pass and self.unsolicited_files == 0


def parse_diff_files(diff: str) -> list[str]:
    files: list[str] = []
    for line in diff.splitlines():
        if line.startswith("+++ b/") and not line.startswith("+++ b/dev/null"):
            files.append(line[6:])
    return files


def count_hunks(diff: str) -> int:
    return sum(1 for line in diff.splitlines() if line.startswith("@@"))


def score_patch(
    repo: Path,
    diff: str,
    allowed_files: set[str],
    compile_cmd: list[str] | None = None,
) -> PatchScore:
    files = parse_diff_files(diff)
    parsed = bool(files) and "diff --git" in diff
    unsolicited = [f for f in files if f not in allowed_files]

    apply = subprocess.run(
        ["git", "apply", "--check", "--whitespace=nowarn", "-"],
        cwd=repo,
        input=diff.encode(),
        capture_output=True,
    )
    apply_ok = parsed and apply.returncode == 0

    compile_ok: bool | None = None
    if apply_ok and compile_cmd:
        # Apply in a detached worktree in real use. Fixture path shown for brevity.
        compile = subprocess.run(compile_cmd, cwd=repo, capture_output=True)
        compile_ok = compile.returncode == 0

    notes = apply.stderr.decode()[:300] if apply.returncode else "apply-check clean"
    return PatchScore(
        parsed=parsed,
        apply_ok=apply_ok,
        compile_ok=compile_ok,
        files_touched=len(set(files)),
        unsolicited_files=len(set(unsolicited)),
        hunks=count_hunks(diff),
        notes=notes.strip() or "ok",
    )

I keep two fixture diffs next to the scorer. One is a surgical edit. One is a confident mess. You need both. A harness that only sees success is a participation trophy with a shebang.

# fixtures/good.patch  — allowed file: src/greet.py
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,3 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hi {name}"

# fixtures/bad.patch  — claims src/greet.py, also rewrites README.md
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,4 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hello {name.strip()}"
+    # model leftover
diff --git a/README.md b/README.md
--- a/README.md
+++ b/README.md
@@ -1,2 +1,2 @@
-# demo
+# totally rewritten by accident

Wire a one-liner so you can refuse to argue with the model in Slack:

python score_patch.py --repo ./fixture-repo --diff fixtures/good.patch \
  --allow src/greet.py --compile "python -m compileall src"
python score_patch.py --repo ./fixture-repo --diff fixtures/bad.patch \
  --allow src/greet.py --compile "python -m compileall src"

On those fixtures the scorer prints a boring, useful object. Good patch: parsed, apply_ok, compile_ok, one file, zero unsolicited, shipping_unit true. Bad patch: it may still parse. It may even apply. It still fails the shipping test because it rewrote a file nobody asked for, and the leftover comment is a compile-time shrug waiting to happen. That is not a model ranking. That is a gate.

Would you merge the second one because the greeting string got fancier? I would not. Fancy is not a shipping unit.

Where free inference belongs in this loop

I do not need a heroic model to generate the first diff. I need an endpoint I can afford to retry while the scorer stays strict. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I want that cheap first pass, I point the same harness at MonkeyCode, which is an open-source project that currently offers free model access (the project states a ten-million-token allowance) and a free server option. I treat those as availability claims, not as a quality certificate. The scorer does not care who spoke. It cares whether git apply --check returned zero.

The useful pattern is blunt. Draft on the free endpoint. Score locally. If apply fails, send the stderr back as the next prompt — not a new essay, the actual reject text. If apply works and compile fails, send the compiler output, not your feelings. If an unsolicited file shows up, drop the patch on the floor. Do not negotiate with a courtesy refactor.

That loop is why free inference is interesting here and why it is dangerous without the gate. You can burn a lot of tokens producing patches that never become commits. Busy looks like progress. Apply is progress.

A decision I actually use, written as prose because a spreadsheet will not sit in your code review: if the change is a single allowed file, a compile command exists, and unsolicited files are a hard fail, free inference is a reasonable drafter. If the change spans generated code, lockfiles, or anything with a secret, I do not send it to a free endpoint at all. If there is no compile or typecheck command, I do not pretend the score is complete. I mark compile_ok as unknown and I refuse the shipping_unit shortcut. Unknown is not green.

What this does not prove

This harness does not measure taste. A patch can apply, compile, and still be the wrong design. It does not measure latency tails, retry storms, or whether the model called the same tool twice. I have scored those problems elsewhere. Different failure, different gate.

It also does not make free inference permanent, fast, or fair. Quotas change. Servers get busy. A free endpoint can be fine on Tuesday and weird on Thursday. That is why the score lives in git, next to the fixture, not in a screenshot of a chat window.

Do not use this approach if you need a contractual SLA, if your diff includes credentials, or if you were hoping the model would replace review. It will not. If your tests cannot fail, this gate will not save you either. Apply-and-compile is necessary. It is not sufficient. Anyone shipping regulated code from an unaudited free endpoint should stop and pick a different workflow.

The trend I keep hearing is that models already write better code than most of us. Maybe. Ask a meaner question. Better than whom, under which gate, against which tree? A paragraph that would impress a timeline can still lose to git apply --check. That command does not care about the debate. It cares about context lines.

I will keep drafting on cheap inference when the blast radius is small. I will keep scoring the patch, not the promise. If you already keep a compile gate, run the same harness against a free endpoint and see whether “done” survives first contact with the tree. MonkeyCode’s free models and free server are one place to try that loop. The score file is the part I would keep even if the endpoint changed names tomorrow.