akaOSS

Checkers decide what can be checked. Humans score the rest. Models don't judge.

Research open · code closed, for now

In October one AI pipeline built 948 working Linux root exploits at about $1,300 each. Discovery scales with compute; checking still runs on people. So my code is closed until I have local protection, and the research stays open. Why →

Research

Layout
  1. № 015 · 2026-11-16

    An unobserved migration: what is left when the instrument was not running

    On 16 November 2026 an agent moved a site's design rules, the laws, out of the site and into the shared design system package, so the site now receives them through a sync. Nothing recorded the agent while it worked, although this lab builds eval-kit to record exactly that. This note is written after the fact. It separates what we read first-hand, what the implementer reported and we did not repeat, and what a timeline reconstructed afterwards from session records shows about how the work was gated. A reconstruction is not a log, and nothing here was declared in advance.

  2. № 013 · 2026-11-15 · Paper

    From building proofs to untangling the proved.

    This paper changes the claim this site rests on. Checkers decide what can be checked. Humans score the rest. Models don't judge.

  3. № 012 · 2026-11-12

    Windowless: when finding is free, trust shrinks to what one person can check

    In October 2026 one pipeline produced 948 working Linux root exploits for about $1,279 each, and one release put hundreds of machine-found mathematical results in front of a field that has not yet understood them. Finding is now cheap and checking is not, and that observation is no longer new. This note argues for what follows from it: trust shrinks to what one person can check, the things worth trusting become small on purpose, and Leibniz described both halves of that arrangement three centuries ago. The second half is a prediction. Nothing here measures it yet.

  4. № 011 · 2026-09-21

    A gate with nothing to read: what a calibrated decision model answers, and what it cannot

    HITL Kit ships a confidence gate that fails open by default, because nothing we could wire to it produced a confidence worth reading. TypeSafe's Jev is the first model we have read that is designed to be that source: typed answers with a probability distribution and a confidence on every call. This note reads their launch material through the gate thesis. It answers the discretionary question well, it must be kept away from the mandated one, it carries two of the three parts of what № 006 calls a signal, and its calibration is a claim only your own labelled data can check. We have not run the model. Every number here is the vendor's.

  5. № 010 · 2026-09-13

    Present, informed, and wrong: the oversight our instrument would have passed

    On 27 June a monitoring tool fired, a responder investigated, and they correctly identified both mechanisms of what would become the largest agent incident of the year. They then advised that stopping the run was not required. That decision would pass every signal № 009 added: attributable, unhurried, informed. This note is about what our instrument cannot see, what the published reports show about scorers being attacked rather than humans, and why a measure wired to a reward is the failure our standing rule was written against.

  6. № 009 · 2026-08-31

    A rubber stamp is also an approval: recording whether oversight was real

    № 005 argued that no current benchmark can tell a rubber stamp from a genuine review. eval-kit's own gate schema could not either: `resolution: "approved"` is exactly the field that collapses the two. This note is about closing that gap without letting the instrument invent the thing it is measuring, and about two divergences between the published contract and the reference implementation that were recorded rather than resolved.

  7. № 008 · 2026-08-11

    A reframe that breaks things: rebuilding eval-kit around authorization

    eval-kit's pitch narrowed from evaluating research agents to measuring authorization. The useful part was not the new sentence. It was discovering which parts of the codebase the sentence contradicted. The reframe broke six things, and each break marked a place where the old claim had been carrying weight the new one will not.

  8. № 007 · 2026-08-06

    Absence passes: what a green check cannot tell you

    We built four CI gates to check that a published package is trustworthy, then published a package whose npm page reads 'ERROR: No README data found!', with every gate green. The cause generalises past this bug: a checker that filters its own input list by file existence converts a missing artifact into an empty work-list, and an empty work-list into a pass. Shipping the fix then failed on a 404 that means four different things, which is the adjacent lesson.

  9. № 006 · 2026-07-26

    Signals, not verdicts: the gate thesis applied

    The companion to the gate thesis: what happens when you build it into a domain where being wrong has a victim either way. Content review turns out to produce both gate kinds without being asked to, and the architecture that results has one property worth stating plainly. The machine may resolve a case alone only when the resolution is to do nothing.

  10. № 005 · 2026-07-21

    The gate is the unit of measurement

    A reframing of what the akaOSS measurement family actually measures: not task completion, but the gate, the moment an agent hands control back to a human. Gates come in two kinds that must never share a metric, autonomous benchmarks structurally penalize both, and a gate over unverifiable output is oversight theater. Regulation now mandates effective human oversight; nothing measures whether oversight is effective. That gap is the research program.

  11. № 004 · 2026-07-18

    The credibility stack: what a stranger checks before trusting your package

    Before anyone reads your test suite, they run an implicit checklist: does the published package install, do the README examples compile, does the API change without warning. We built those checks as CI harnesses across three of our packages and ran them on ourselves. First run: one package shipped no type declarations at all, and three README examples had never compiled.

  12. № 003 · 2026-07-18

    Watching the agent work: observability for agent-driven asset pipelines

    When an AI coding agent became a real part of a Blender → glTF game pipeline: writing the export scripts, running them, rearranging the shipped files. The speed-up came with a blind spot: no picture of what it was actually doing. Hologram is the pattern that closed the gap: one append-only event log, a live dashboard beside the assets, and the same pipeline handed back to the agent as a small non-destructive MCP surface.

  13. № 002 · 2026-07-18

    An instrument, not a judge: why humans score the agents

    LLM-as-judge shares the blind spots of the agent it grades. Both were trained against similar objectives, so the judge rationalizes the same failures. eval-kit's answer is a measurement instrument built around step-level human judgment: per-step tool-match assertions, distractor tasks that score the refusal, and a five-dimension 0–3 rubric the judge never touches by default.

  14. № 000 · 2026-07-18

    Welcome to the lab log: what this feed measures, and how

    Findings here are produced by running eval-kit suites, displayed with HITL Kit components, and scored by humans on a five-dimension rubric, never by an LLM judge. Every entry links its experiment folder and checked-in run JSON, and none of it is leaderboard material.