Every number here was produced by bash scripts/run_all.sh and is reproduced
by it. The generated tables live in results/SUMMARY.md; this page is the
reading of them, including the parts that did not work.
Read METHOD.md first if you intend to quote anything. In particular: the pooled accuracy over the synthetic corpus is an average over four deliberately unlike task families, and its value depends on the mix. The per-family table is the result.
In one paragraph. Blink recognises the form of a decision; it does not
read text the way a pretrained language model does. Where the answer is carried
by form — which queue a ticket's wording points at, whether a claim's verb
agrees with the state's — blink-tiny is near perfect and well calibrated
(routing 0.998, judgment 0.999, ECE 0.028). Where the answer requires
reading — binding a name to the right sentence, comparing quantities, natural
language inference — it is at or a little above chance, and a frozen 4B model
is far ahead (WANLI, SemIf's 256 rows: 0.44 against 0.64). The gap is
pretraining, not capacity: twenty times the parameters did not close it (§5).
Use Blink where form suffices, check that on your own data, and treat its
confidence as the thing to branch on.
Machine. Apple M5 Pro, macOS 27.0, Apple clang, single-threaded.
-std=c99 -O3 -ffp-contract=off. Exact command lines and checksums are in
results/MANIFEST.json.
Weights are mapped read-only and never copied, so the container bytes are paid once per machine rather than once per process or per session. Bytes per parameter sits slightly above 1.0 because the fp32 per-row scales, the layer norms, the depthwise convolutions and the FiLM modulation are not quantized.
| blink-tiny | blink-small | |
|---|---|---|
| parameters | 406,913 | 7,900,033 |
| container on disk | 452.2 KiB | 7.93 MiB |
| bytes per parameter | 1.138 | 1.052 |
| arena, minimum limits | 22.2 KiB | 55.2 KiB |
| arena, embedded limits (128 B state, 4 options) | 129.0 KiB | 300.1 KiB |
| arena, default limits (16 options) | 260.4 KiB | 1.11 MiB |
64 concurrent sessions over one model:
| arena per session (4 options) | 249.9 KiB |
| arenas in total | 15.6 MiB |
| measured resident growth | 16.1 MiB |
| weight bytes counted again | 0 (452.2 KiB, mapped once) |
The resident growth tracks the arenas and nothing else. That is the whole claim about memory: the marginal cost of one more concurrent decision is one arena, whose size you choose by declaring limits.
Single-threaded, p50 over up to 2,000 samples or 1.5 seconds, whichever comes
first, after 50 warm-up iterations. decide re-encodes everything;
cached_state holds the state and the menu and changes only the question;
new_menu holds the state and supplies a fresh option list.
blink-tiny
| state bytes | options | decide | cached_state | cached_batch8 | new_menu | cached decisions/s |
|---|---|---|---|---|---|---|
| 64 | 4 | 169 µs | 43 µs | 42 µs | 110 µs | 22,851 |
| 128 | 4 | 240 µs | 46 µs | 46 µs | 114 µs | 21,061 |
| 256 | 2 | 366 µs | 51 µs | 51 µs | 83 µs | 19,251 |
| 256 | 4 | 406 µs | 54 µs | 53 µs | 121 µs | 18,211 |
| 256 | 16 | 613 µs | 72 µs | 72 µs | 329 µs | 13,666 |
blink-small
| state bytes | options | decide | cached_state | cached_batch8 | new_menu | cached decisions/s |
|---|---|---|---|---|---|---|
| 64 | 4 | 1.25 ms | 291 µs | 265 µs | 859 µs | 3,422 |
| 256 | 4 | 2.47 ms | 300 µs | 275 µs | 869 µs | 3,321 |
| 512 | 2 | 3.98 ms | 306 µs | 281 µs | 575 µs | 3,262 |
| 512 | 4 | 4.29 ms | 314 µs | 290 µs | 887 µs | 3,175 |
| 512 | 16 | 5.89 ms | 373 µs | 347 µs | 2.50 ms | 2,673 |
SIMD. The NEON kernel against the same build with -DBLINK_SCALAR_ONLY=1 (make scalar):
| model | path | state bytes | scalar p50 | NEON p50 | speedup |
|---|---|---|---|---|---|
| blink-tiny | decide |
256 | 1.13 ms | 406 µs | 2.79× |
| blink-tiny | cached_state |
256 | 134 µs | 54 µs | 2.48× |
| blink-tiny | new_menu |
256 | 352 µs | 121 µs | 2.91× |
| blink-small | decide |
512 | 15.42 ms | 4.29 ms | 3.60× |
| blink-small | cached_state |
512 | 1.16 ms | 314 µs | 3.69× |
| blink-small | new_menu |
512 | 3.41 ms | 887 µs | 3.84× |
Three things to read off these tables.
The state cache is the throughput story. The cached path costs the question
and the head, neither of which depends on how long the state was, so the ratio
grows with the state length. A KV prefix cache in a frozen LLM gives the same
shape of win (SemIf: 2.33 → 20.03 decisions/s), but approximately; here the
reuse is exact, because the state and the question are separate sequences and
nothing is being approximated.
tests/c/test_session.c checks it with memcmp.
Cost is linear in the state, not quadratic. There is no self-attention over bytes; the only quadratic terms run after the pool, over a sequence a quarter or an eighth as long.
Projections run four rows at a time, and batching questions fills the
groups. Every int8 projection goes through blink_matmul_i8_rows, which
widens each weight row to fp32 once per group of four input vectors instead of
once per vector; the widening, not the multiply, is what a W8A32 product pays
for. Each vector keeps its own accumulators in the original order, so the
output is bit-identical to the one-vector kernel (tests/c/test_kernels.c
checks it with memcmp) and every number in section 3 is unchanged. Against
the previous build this took 1.3–1.5× off every path. cached_batch8 scores
eight questions per blink_score_batch call, which encodes them side by side
so that short questions still fill the groups; it adds about 8% on
blink-small and almost nothing on blink-tiny, whose per-question cost is
dominated by work a batch does not share (the FiLM product, the queries and
the head). The batch is bit-identical to scoring the questions one at a time,
checked in tests/c/test_session.c. The price is a fixed four-row scratch in
the arena, visible in section 1 as a few KiB more at every limit.
make ACCELERATE=1 (macOS) sends the projections, the layer norms and the
attention reads through Accelerate, which runs matrix products on the Apple
silicon matrix unit. Same machine, one thread, 4 options
(results/bench-latency-accelerate.jsonl):
| model | path | state bytes | default p50 | Accelerate p50 | speed-up |
|---|---|---|---|---|---|
| blink-tiny | decide |
256 | 406 µs | 172 µs | 2.36× |
| blink-tiny | cached_state |
256 | 54 µs | 32 µs | 1.69× |
| blink-tiny | new_menu |
256 | 121 µs | 62 µs | 1.95× |
| blink-small | decide |
512 | 4.29 ms | 824 µs | 5.20× |
| blink-small | cached_state |
512 | 314 µs | 135 µs | 2.33× |
| blink-small | cached_batch8 |
512 | 290 µs | 71 µs | 4.06× |
blink-small gains more because its products are larger: 192 wide against 64, which is where the matrix unit pays for its overhead.
The default build's profile explains the ceiling. On a blink-tiny decide,
60% of the time was the int8 projections; with those moved, what is left is
embeddings, layer norms and attention, and moving those to vDSP and BLAS
brought the remaining time down again, from 238 µs to 172 µs.
The price is described in the README (Accelerate backend): Accelerate
allocates inside its large products, 16 times per blink-tiny decision
(measured by interposing malloc; the default build makes zero); the model
holds a 352 KiB fp32 copy of the projections; the arena is 335 KiB instead of
260 KiB; results agree with the default build to fp32 rounding rather than bit
for bit. On the synthetic test split, 4,000 rows, no argmax changes and the
largest probability difference is 1.5e-6. The accuracy tables in §3 are from
the default build, and they hold for this one to that precision.
make W8A8=1 quantizes the input of every projection to int8, one scale per
row, and multiplies with integer instructions: SDOT, or I8MM on non-Apple
cores that report it. This one has I8MM, but SDOT is the default on Apple
silicon for the reason below; BLINK_W8A8_KERNEL=i8mm selects the other.
Same machine, one thread, 4 options (results/bench-latency-w8a8.jsonl, the
SDOT default):
| model | path | state bytes | default p50 | W8A8, SDOT | W8A8, I8MM | speed-up (SDOT) |
|---|---|---|---|---|---|---|
| blink-tiny | decide |
256 | 406 µs | 221 µs | 217 µs | 1.84× |
| blink-tiny | cached_state |
256 | 54 µs | 33 µs | 33 µs | 1.64× |
| blink-tiny | new_menu |
256 | 121 µs | 64 µs | 63 µs | 1.89× |
| blink-small | decide |
512 | 4.29 ms | 1.45 ms | 1.46 ms | 2.95× |
| blink-small | cached_state |
512 | 314 µs | 115 µs | 116 µs | 2.73× |
| blink-small | new_menu |
512 | 887 µs | 268 µs | 272 µs | 3.31× |
On this core the two instructions are level, which is why SDOT is the
default on Apple silicon. SMMLA does twice the products
per instruction, but a plain loop issues SDOT faster (measured 140 against
193 GMAC/s at the instruction level), and both kernels reach about 170
GMAC/s on blink-small's projections, where loads rather than arithmetic set
the pace. Against the Accelerate backend, W8A8 is slower on a fresh
decision (221 µs against 172 µs for blink-tiny) and level or faster once the
state is cached, and it keeps what Accelerate gives up: nothing allocates
while scoring (checked by interposing malloc), the arena grows by 1 KiB
rather than 75, and it needs no framework. The I8MM kernel reads a copy of
the projections rearranged into SMMLA's layout at open time, 88 KiB for
blink-tiny and 1.3 MiB for blink-small; SDOT needs none, which on a core
where the two are level decides it.
Accuracy. Every published model, evaluated on its own test split with
both builds by the same harness (BLINK_LIBRARY=build-w8a8/libblink.dylib):
| model | rows | top-1, default | top-1, W8A8 | ECE, default | ECE, W8A8 | decisions changed |
|---|---|---|---|---|---|---|
| blink-tiny, synthetic s7 | 4,000 | 0.6530 | 0.6542 | 0.0561 | 0.0558 | 2.1% |
| blink-tiny, synthetic s17 | 4,000 | 0.6215 | 0.6212 | 0.0236 | 0.0222 | 8.0% |
| blink-tiny, synthetic s27 | 4,000 | 0.6258 | 0.6280 | 0.0030 | 0.0059 | 16.1% |
| blink-small, synthetic s7 | 4,000 | 0.6030 | 0.6062 | 0.0134 | 0.0117 | 5.4% |
| blink-small, synthetic s17 | 4,000 | 0.6218 | 0.6260 | 0.0117 | 0.0076 | 3.8% |
| blink-small, synthetic s27 | 4,000 | 0.5620 | 0.5605 | 0.0506 | 0.0509 | 1.5% |
| blink-tiny, WANLI s7 | 4,945 | 0.4861 | 0.4855 | 0.0294 | 0.0300 | 1.0% |
| blink-tiny, WANLI s17 | 4,945 | 0.4967 | 0.4952 | 0.0197 | 0.0168 | 1.0% |
| blink-tiny, WANLI s27 | 4,945 | 0.4910 | 0.4902 | 0.0245 | 0.0239 | 0.6% |
| blink-small, WANLI s7 | 4,945 | 0.4762 | 0.4777 | 0.0258 | 0.0274 | 1.0% |
| blink-small, WANLI s17 | 4,945 | 0.4726 | 0.4730 | 0.0198 | 0.0215 | 1.0% |
| blink-small, WANLI s27 | 4,945 | 0.4720 | 0.4720 | 0.0360 | 0.0360 | 1.0% |
| blink-tiny, Doom | 1,191 | 0.8338 | 0.8346 | 0.0310 | 0.0322 | 0.5% |
| blink-tiny, Snake | 5,693 | 0.7887 | 0.7897 | 0.0025 | 0.0031 | 0.4% |
| blink-tiny, Pilot | 6,981 | 0.8199 | 0.8199 | 0.0291 | 0.0289 | 0.1% |
Top-1 moves by −0.15 to +0.42 points, in both directions and inside the
between-seed spread; calibration is unchanged. The decisions that change are
almost all in the two synthetic families the model does not learn,
comparison and selection, where it sits at chance with near-uniform
probabilities (the largest shift there is 0.016) and a rounding picks a
different near-tie. On blink-tiny s27, the worst case, 31% of comparison
and 34% of selection rows change against 0.1% of judgment and none of
routing. These models were trained with fake quantization of the weights
only; quantization-aware training of the activations is not done.
The projection kernels have x86-64 SIMD paths, chosen at run time so one
binary runs on any x86-64 machine: SSE2, which x86-64 guarantees, and AVX2
with FMA when the CPU reports both (BLINK_X86_SIMD=scalar|sse2|avx2
forces one). In the W8A8 build the integer products get the same two levels.
SSE2 sign-extends to int16 and uses pmaddwd; AVX2 moves the weight's sign
onto the input (vpsignb) and uses vpmaddubsw, which cannot saturate at
these magnitudes. So both are exact and byte-identical to the scalar integer
kernel, as SDOT and I8MM are.
No x86 machine was available for this work. Everything below ran on the
Apple M5 Pro through Docker's linux/amd64 emulation, which is Rosetta and
reports SSE2 through AVX2 and FMA, without AVX-512 or VNNI. Correctness
carries over; speed does not. bash scripts/x86-docker-test.sh reruns all of
it (tests, parity, timings), and on an x86 host its timings become real.
- The C suite passes at every level (scalar, SSE2, AVX2, forced in turn), for
W8A32 and W8A8, with the SIMD compiled out, at
-O0, and under AddressSanitizer and UndefinedBehaviorSanitizer, with GCC 14. The row-blocked product stays bit-identical to the one-vector product within each level. - On 300 test rows of blink-tiny, SSE2 and AVX2 agree with the float64 reference to 6e-7 (the tolerance is 1e-5) and with NEON to 1e-6, with no argmax change. In the W8A8 build, SSE2 and AVX2 give identical bytes and agree with the Arm W8A8 build to 9e-8.
Timings under the emulator, p50, 4 options. They are only meaningful relative to each other within a row, and not even that for AVX2:
| model | path | state | scalar | SSE2 | AVX2 | W8A8 SSE2 | W8A8 AVX2 |
|---|---|---|---|---|---|---|---|
| blink-tiny | decide |
256 | 1,334 µs | 681 µs | 1,006 µs | 686 µs | 1,100 µs |
| blink-tiny | cached_state |
256 | 168 µs | 90 µs | 127 µs | 87 µs | 134 µs |
| blink-small | decide |
512 | 18.1 ms | 6.79 ms | 8.58 ms | 6.85 ms | 10.98 ms |
| blink-small | cached_state |
512 | 1,447 µs | 492 µs | 617 µs | 462 µs | 733 µs |
SSE2 is 2.0–2.9× the scalar loop. AVX2 comes out slower than SSE2 here only because Rosetta executes each 256-bit instruction as 128-bit Arm ones.
On real x86 hardware it is the other way round. The x86 SIMD levels step in
CI times each level on a GitHub-hosted ubuntu-latest runner, built with
GCC. That is a shared virtual machine whose CPU model the run does not record,
so the absolute numbers are indicative, but the ratios within the run are
meaningful. blink-tiny, 256-byte state, 4 options, p50 (first run, commit
a1bdc6c):
| path | scalar | SSE2 | AVX2 + FMA |
|---|---|---|---|
decide |
2,075 µs | 979 µs | 760 µs |
cached_state |
245 µs | 117 µs | 90 µs |
new_menu |
645 µs | 292 µs | 222 µs |
SSE2 is 2.1–2.2× the scalar loop and AVX2 2.7×, so AVX2 is the right default where the CPU has it.
SemIf reports 20.03 decisions/s for a frozen Qwen3.5-4B with parallel suffixes
on an RTX 3090, against 2.33 for fresh scoring. blink-small reaches roughly
3,200 decisions/s on one CPU core, 3,400 with blink_score_batch. This is not a like-for-like comparison —
different models, different hardware, different tasks, and a four-billion-
parameter model is doing something a 7.8-million-parameter model cannot. It is
here to say what changes when the model is small enough to embed, not to claim
a win.
Two open projects expose the same interface as Blink: jevlike, a small PyTorch model trained from scratch, and SemIf, which reads option logits out of a frozen 4B LLM. Blink is measured against both.
Speed. One decision over a 256-byte state, a question and four options, batch one. Blink and jevlike were measured on the same machine, one CPU thread each; the SemIf figures are the ones it publishes, on GPUs.
| system | hardware | fresh decision, p50 | decisions/s, fresh | decisions/s, state reused |
|---|---|---|---|---|
| blink-tiny (C99, int8) | 1 CPU core, Apple M5 Pro | 406 µs | 2,448 | 18,211 |
| blink-tiny, Accelerate backend | 1 CPU core + matrix unit, Apple M5 Pro | 172 µs | 5,808 | 31,290 |
| blink-tiny, W8A8 backend (SDOT) | 1 CPU core, Apple M5 Pro | 221 µs | 4,460 | 29,481 |
| jevlike, width 288 (matched capacity) | 1 CPU core, Apple M5 Pro, PyTorch | 188 µs | 5,275 | no state cache |
| jevlike, width 64 (default) | 1 CPU core, Apple M5 Pro, PyTorch | 96 µs | 10,321 | no state cache |
| SemIf, Qwen3.5-4B BF16 | RTX 3090 | ~430 ms | 2.33 | 20.03 |
| SemIf, Qwen3.5-4B BF16, MLX | Apple M5 Max GPU | ~610 ms | 1.63 | 17.24 |
Memory.
| system | parameters | weights | process peak RSS | allocation while scoring |
|---|---|---|---|---|
| blink-tiny | 407k | 452 KiB int8, mapped read-only, shared across sessions | 2.5 MiB | none (260 KiB arena per session, sized up front) |
| blink-tiny, Accelerate backend | 407k | 452 KiB int8 mapped, plus a 352 KiB fp32 copy of the projections | 7.0 MiB | inside Accelerate: 16 per decision (335 KiB arena) |
| blink-tiny, W8A8 backend (SDOT) | 407k | 452 KiB int8, mapped read-only, no copy | 2.6 MiB | none (261 KiB arena) |
| jevlike, width 288 | 398k | 1.5 MiB fp32 | 199 MiB | PyTorch allocator |
| jevlike, width 64 | 45k | 177 KiB fp32 | 196 MiB | PyTorch allocator |
| SemIf, Qwen3.5-4B BF16, MLX | 4B | 7.83 GiB resident | 9.31 GiB | framework allocator |
| SemIf, Qwen3.5-4B Q4, MLX | 4B | — | 2.82 GiB | framework allocator |
How to read the two tables:
- On a fresh decision the default build is slower than jevlike. jevlike is Python, but its arithmetic runs in PyTorch's native kernels, which on Apple silicon use the CPU's matrix unit. Its model is also a byte embedding and one attention read, with no encoder in between. Blink spends that time on the encoder, and it shows in the accuracy below: at matched capacity it is ten points ahead on top-1 and seven times better calibrated.
- The Accelerate backend closes that gap. With the projections on the same matrix unit, blink-tiny is faster than jevlike on a fresh decision too (172 µs against 188 µs), still on one thread. The W8A8 backend gets to 221 µs without allocating and without a framework.
- Blink is faster once the state is encoded. Encoding a state once and asking it many questions is the common case in software, and the reuse is exact. Blink then answers 3.5× faster than jevlike at matched capacity, and 5.9× with the Accelerate backend.
- Memory is where the runtimes differ most. jevlike's weights are small, but a PyTorch process needs about 80 times Blink's whole footprint. SemIf needs gigabytes and a GPU, and on a fresh decision it is about a thousand times slower.
- A frozen 4B model reads language that Blink cannot. These tables compare cost, not capability.
Blink's figures come from make bench (results/bench-latency.jsonl, and
results/bench-latency-accelerate.jsonl and results/bench-latency-w8a8.jsonl
for the two optional backends) and from
/usr/bin/time -l on blink answering one decision. jevlike's come
from eval/bench_external.py
(results/bench-external-w*.json), on the checkpoints trained by
scripts/experiments/queue_external.sh. SemIf's are quoted from its published
RTX 3090 and MLX results; the fresh latency is the inverse of its fresh rate.
Accuracy, measured head to head: same corpus, same harness, three runs each (details; measured on Blink's previous option head, the current one scores 0.6334 ± 0.0171):
| parameters | top-1 | ECE | |
|---|---|---|---|
| blink-tiny | 407k | 0.6064 ± 0.0149 | 0.0216 ± 0.0036 |
| jevlike, matched capacity | 398k | 0.5004 ± 0.0397 | 0.1514 ± 0.0420 |
On SemIf's 256-row WANLI subset, rebuilt from the identifiers it publishes, a frozen Qwen3.5-4B scores 0.637 balanced accuracy and blink-tiny scores 0.4393 ± 0.0342, with one ten-thousandth of the parameters. Blink does not compete on open-domain language, and External corpora says so with the controls next to it. On jevlike's Wikispeedia next-click task the two are level (0.306 against 0.313), and neither reads the article: the task reduces to matching titles.
Everything between the markers below is generated by
scripts/report_results.py from results/eval-*.json, as are §1 and §2. It
is generated because it was hand-copied for several revisions and went stale
the moment the architecture changed — at one point quoting numbers from models
that no longer loaded. Regenerate it with
python scripts/report_results.py --write.
Every figure is mean ± standard deviation over three independent runs (see the
note on determinism below: on MPS a seed does not pin a run). Reliability
tables, risk–coverage curves and bootstrap intervals are in
results/eval-*.json; row-level predictions are regenerated into
results/predictions/ by the same script and are not committed.
3 runs, 4000 test rows over 2000 scenario groups whose entities appear in neither the training nor the validation split. Nothing clipped.
| blink-tiny | blink-small | |
|---|---|---|
| top-1 | 0.6334 ± 0.0171 | 0.5956 ± 0.0306 |
| balanced accuracy | 0.4909 ± 0.0187 | 0.4583 ± 0.0183 |
| macro F1 | 0.4918 ± 0.0168 | 0.4578 ± 0.0174 |
| NLL | 0.7166 ± 0.0027 | 0.7842 ± 0.0784 |
| Brier | 0.3756 ± 0.0054 | 0.4175 ± 0.0488 |
| ECE | 0.0276 ± 0.0268 | 0.0252 ± 0.0220 |
| uniform baseline | 0.2818 ± 0.0000 | 0.2818 ± 0.0000 |
| majority baseline | 0.2865 ± 0.0000 | 0.2865 ± 0.0000 |
Per family, which is the part that means anything:
| family | rows | blink-tiny | blink-small | uniform |
|---|---|---|---|---|
comparison |
1000 | 0.3043 ± 0.0798 | 0.2547 ± 0.0092 | 0.256 |
judgment |
1500 | 0.9991 ± 0.0015 | 0.9467 ± 0.0924 | 0.333 |
routing |
500 | 0.9980 ± 0.0035 | 0.9413 ± 0.0764 | 0.261 |
selection |
1000 | 0.2317 ± 0.0211 | 0.2370 ± 0.0069 | 0.240 |
Temperature is fitted on validation, whose entity pool is disjoint from both training and test, and is then measured here.
| blink-tiny (3 runs) | |
|---|---|
| fitted temperature | 1.761 ± 0.800 |
| ECE | 0.0276 ± 0.0268 |
| MCE | 0.3723 ± 0.2124 |
| NLL | 0.7166 ± 0.0027 |
| mean confidence | 0.6444 ± 0.0217 |
Reliability, one run (bins holding at least 40 rows):
| confidence bin | rows | mean confidence | accuracy | gap |
|---|---|---|---|---|
| 0.13 – 0.20 | 111 | 0.185 | 0.144 | +0.041 |
| 0.20 – 0.27 | 311 | 0.232 | 0.161 | +0.072 |
| 0.27 – 0.33 | 364 | 0.296 | 0.209 | +0.087 |
| 0.33 – 0.40 | 466 | 0.364 | 0.339 | +0.025 |
| 0.40 – 0.47 | 281 | 0.431 | 0.335 | +0.097 |
| 0.47 – 0.53 | 200 | 0.498 | 0.395 | +0.103 |
| 0.53 – 0.60 | 103 | 0.562 | 0.427 | +0.134 |
| 0.60 – 0.67 | 76 | 0.633 | 0.579 | +0.054 |
| 0.67 – 0.73 | 45 | 0.698 | 0.622 | +0.076 |
| 0.73 – 0.80 | 51 | 0.773 | 0.765 | +0.008 |
| 0.80 – 0.87 | 119 | 0.842 | 0.941 | -0.099 |
| 0.87 – 0.93 | 366 | 0.910 | 0.997 | -0.088 |
| 0.93 – 1.00 | 1507 | 0.973 | 1.000 | -0.027 |
Risk–coverage, averaged over runs:
| confidence threshold | coverage | accuracy when answered |
|---|---|---|
| 0.00 | 1.000 ± 0.000 | 0.6334 ± 0.0171 |
| 0.50 | 0.533 ± 0.049 | 0.9702 ± 0.0400 |
| 0.60 | 0.514 ± 0.024 | 0.9889 ± 0.0180 |
| 0.70 | 0.505 ± 0.009 | 0.9951 ± 0.0080 |
| 0.80 | 0.499 ± 0.001 | 0.9985 ± 0.0022 |
| 0.90 | 0.481 ± 0.032 | 0.9998 ± 0.0003 |
| 0.95 | 0.435 ± 0.104 | 1.0000 ± 0.0000 |
| intact | shuffled state | shuffled question | uniform | |
|---|---|---|---|---|
| blink-tiny | 0.6334 ± 0.0171 | 0.4019 ± 0.0032 | 0.3441 ± 0.0113 | 0.282 |
| blink-small | 0.5956 ± 0.0306 | 0.3771 ± 0.0181 | 0.3295 ± 0.0309 | 0.282 |
| untrained weights | top-1 | uniform |
|---|---|---|
| blink-tiny, random | 0.2900 | 0.282 |
| blink-small, random | 0.2900 | 0.282 |
| perturbation | argmax flip rate | mean total variation | top-1 before | after |
|---|---|---|---|---|
| reversed options | 0.0000 ± 0.0000 | 0.0000 ± 0.0000 | 0.6334 ± 0.0171 | 0.6334 ± 0.0171 |
| irrelevant suffix | 0.2721 ± 0.0285 | 0.1238 ± 0.0402 | 0.6334 ± 0.0171 | 0.5347 ± 0.0429 |
WANLI, trained on its own train split — a generalisation test, not a transfer test. Byte limits were taken from the corpus (--limits-from-data), so nothing is clipped.
| rows | 4945 |
| clipped | 0 |
| top-1 | 0.4913 ± 0.0053 |
| balanced accuracy | 0.4914 ± 0.0055 |
| ECE | 0.0245 ± 0.0049 |
| uniform baseline | 0.333 |
| shuffled state | 0.4260 ± 0.0031 |
| shuffled question | 0.4404 ± 0.0055 |
The same 256 WANLI rows SemIf evaluates. Rebuilt from the row identifiers SemIf commits, in SemIf's display order, from the same pinned revision. No premise in the subset appears in Blink's training split.
| system | parameters | balanced accuracy |
|---|---|---|
| SemIf, Qwen3.5-4B frozen | 4,000,000,000 | 0.637 |
| blink-tiny, trained | 390,401 | 0.4393 ± 0.0342 |
| uniform baseline | — | 0.333 |
Blink's controls on this subset: shuffled state 0.3659 ± 0.0266, shuffled question 0.3906 ± 0.0156. The margin over the controls is about eight points, not the eleven the headline suggests.
Transfer to the external fixtures. A model trained on the synthetic corpus, asked for something it has never seen.
| fixture | rows | clipped | top-1 | uniform |
|---|---|---|---|---|
authored144 |
144 | 96 | 0.3750 ± 0.0069 | 0.333 |
perturbations108 |
108 | 87 | 0.3858 ± 0.0809 | 0.333 |
blink-small on WANLI, same protocol, three runs: top-1 0.4736 ± 0.0023, below blink-tiny's 0.4913, and 0.4467 ± 0.0150 balanced on SemIf's 256 rows against tiny's 0.4393 — inside tiny's spread. It memorises: final training NLL 0.13 against validation 2.86. See §5.
Measured on the previous option head. Everything from here to §4 predates the cosine option head (container format 3). Each subsection compares arms trained under the same head, so the comparisons stand, but the absolute numbers are older than §3 above: blink-tiny was 0.6064 ± 0.0149 pooled then and is 0.6334 ± 0.0171 now, and
judgmentwent from 0.9754 ± 0.0497 to 0.9991 ± 0.0015. The containers these rows came from are format 2 and no longer load; re-running the three comparisons on the current head is §6 item 2.
A note on what "run" means here. PyTorch on the MPS backend is not
deterministic: the same seed, the same corpus and the same architecture diverge
by the first epoch. Measured on a minimal case, two runs at seed 7 give
1.3074637349446614 twice on CPU and 1.307463773091634 against
1.307463755607605 on MPS. So the figures below are independent runs, not
seeds, and every standard deviation in this document conflates the two sources
of variation. See REPRODUCE.md.
Three arms, 60 epochs, identical otherwise:
- cross — the full model. Six runs.
- film — without the question-to-state cross layer. Three runs.
- nofilm — without the cross layer and without the query conditioning. Three runs.
| slice | cross (6 runs) | film (3) | nofilm (3) |
|---|---|---|---|
judgment |
0.9754 ± 0.0497 | 0.8778 ± 0.1819 | 0.7773 ± 0.1928 |
| worst run | 0.8753 | 0.6687 | 0.6653 |
The cross layer raises the mean by about ten points and cuts the spread by a
factor of 3.7. That is a real improvement and it is not the stabilisation
an earlier revision of this page claimed: it reported a factor of 120, from
three runs that had all happened to succeed. Three further runs of the same
configuration scored 0.875, 0.980 and 1.000. judgment is still occasionally
learned only in part; it is no longer occasionally lost entirely.
From the three-run comparison, before the additional runs:
| arm | run | supports | contradicts | insufficient |
|---|---|---|---|---|
| cross | 1 | 1.0000 | 1.0000 | 1.0000 |
| cross | 2 | 1.0000 | 1.0000 | 1.0000 |
| cross | 3 | 0.9920 | 1.0000 | 1.0000 |
| film | 1 | 0.7740 | 0.2320 | 1.0000 |
| film | 2 | 1.0000 | 1.0000 | 1.0000 |
| film | 3 | 0.9060 | 0.9880 | 1.0000 |
| nofilm | 1 | 0.0920 | 0.9040 | 1.0000 |
| nofilm | 2 | 1.0000 | 0.0000 | 1.0000 |
| nofilm | 3 | 1.0000 | 1.0000 | 1.0000 |
insufficient is solved by every run in every arm — the claim mentions
something the state never touches, which is a lexical signal. The other two
differ only in a verb, over the same state, so telling them apart requires
comparing one word in the question against one word in the state. Without the
cross layer a run can collapse the two classes entirely: nofilm run 2 scores
supports 1.000 and contradicts 0.000. That total collapse is what has not
recurred with the cross layer.
The effect is confined to judgment: selection and comparison are
untouched, and routing is very slightly worse, which is the honest cost of
adding capacity a family does not need.
jevlike is the one comparison that is fair without qualification: both models are small and both are trained from scratch on the task data. Its frozen-encoder results measure a pretrained encoder, which is a different experiment.
Same corpus, same 60 epochs, jevlike's own default learning rate, three runs
each, and both sets of predictions scored by eval/evaluate.py so the
arithmetic is identical. --context-tokens was set from the corpus so nothing
jevlike reads is truncated — a comparison against a truncated model measures
the truncation. Reproduce with eval/compare_external.py and
scripts/experiments/queue_external.sh.
| system | parameters | top-1 | balanced | ECE |
|---|---|---|---|---|
| blink-tiny | 407k | 0.6064 ± 0.0149 | 0.4723 ± 0.0085 | 0.0216 ± 0.0036 |
| jevlike, width 288 | 398k | 0.5004 ± 0.0397 | 0.4025 ± 0.0363 | 0.1514 ± 0.0420 |
| jevlike, width 64 (default) | 45k | 0.5116 ± 0.0425 | 0.4161 ± 0.0356 | 0.1755 ± 0.0815 |
| system | routing |
judgment |
comparison |
selection |
|---|---|---|---|---|
| blink-tiny | 0.9900 ± 0.0156 | 0.9518 ± 0.0670 | 0.2657 ± 0.0276 | 0.2373 ± 0.0159 |
| jevlike, width 288 | 0.8420 ± 0.0978 | 0.7191 ± 0.0717 | 0.2607 ± 0.0045 | 0.2413 ± 0.0049 |
| jevlike, width 64 | 0.8733 ± 0.0833 | 0.7493 ± 0.0911 | 0.2507 ± 0.0180 | 0.2350 ± 0.0182 |
| uniform baseline | 0.261 | 0.333 | 0.256 | 0.240 |
At matched capacity Blink is ahead by 15 points on routing and 23 on
judgment, and the intervals do not overlap on judgment. The largest gap is
calibration: 7× lower ECE. jevlike is confident when it is wrong, which for
a model whose output is meant to drive an if statement is the difference
between usable and not.
Two things this table says that are worth more than the win.
Capacity is not jevlike's bottleneck. Nine times the parameters left it slightly worse (0.5004 against 0.5116). It is the clearest evidence that Blink's margin comes from the stem, the pooling, the mixer and the question-to-state layer rather than from spending more weights on the same structure.
The two hard families are hard for both. comparison and selection sit at
chance in all three systems and every run. Blink's advantage is real and it is
confined entirely to the families this class of model can address.
What this does not establish. The corpus is Blink's own. It was built to separate capabilities rather than to flatter anything, but it is still home ground. The comparison in the other direction follows.
The comparison above runs on Blink's corpus. This one runs on Wikispeedia next-click, where jevlike publishes 26% with a frozen Qwen2.5-0.5B encoder and 29% from scratch. It is the direction that does not favour Blink.
eval/build_wikispeedia.py reimplements jevlike's selection exactly — same
hash, same step, same menu sampling, same split by target article — and emits
both formats from one selection. 41,291 training rows (jevlike reports
"40,000 clicks"), 4,373 test rows over 314 target articles, up to 64 options
and 43 on average. Both systems get 512 bytes of article body and nothing
truncated; Blink puts the navigation goal in the question and the body in the
state, jevlike gets the same bytes in its single context. Controls are applied
to both in the same way (compare_external.py score-native), which jevlike's
own shuffle_context does not do: it swaps titles and body together.
| blink-tiny | jevlike, width 288 | |
|---|---|---|
| parameters | 434,049 | 526,752 |
| top-1 | 0.3059 ± 0.0191 | 0.3127 ± 0.0041 |
| ECE | 0.0346 ± 0.0070 | 0.0396 ± 0.0075 |
| shuffled state | 0.2900 ± 0.0178 | 0.3034 ± 0.0043 |
| shuffled question | 0.0690 ± 0.0045 | 0.0725 ± 0.0060 |
| uniform baseline | 0.0362 | 0.0362 |
It is a tie. jevlike is slightly ahead, inside Blink's spread, with 21% more parameters (its position table grows with the 704-byte context) and far lower run-to-run variance. Calibration is close, unlike on the synthetic corpus where Blink's was seven times better.
Neither model reads the article.
| evidence margin: intact − shuffled state | |
|---|---|
| blink-tiny | 0.0159 ± 0.0013 |
| jevlike | 0.0093 ± 0.0010 |
Replacing the article body with another article's costs jevlike 0.9 points out of 31 and Blink 1.6. Both solve the task almost entirely by matching the target title in the question against the link titles. Blink extracts 1.7 times as much from the evidence and the intervals do not overlap, but on a scale of thirty points that difference is real and negligible at once.
The finding worth more than either number is about the benchmark. Wikispeedia next-click, built this way, is a title-matching task. That applies to jevlike's published 26–29% as well: with its default 192-byte context, jevlike saw the titles and almost none of the body. This replication of jevlike reaches 0.3127, slightly above the published 0.29, which is consistent with giving it 704 bytes instead of 192.
Across both corpora, then: Blink is ahead where the evidence matters and the two are level where the task reduces to titles.
The film and nofilm arms above differ only in the query conditioning, and
their difference remains inside their own spread:
| slice | film | nofilm | gap | larger within-arm sd |
|---|---|---|---|---|
| pooled | 0.5796 | 0.5400 | +0.040 | 0.077 |
judgment |
0.8778 | 0.7773 | +0.100 | 0.193 |
comparison |
0.2687 | 0.2583 | +0.010 | 0.036 |
selection |
0.2337 | 0.2357 | −0.002 | 0.008 |
Undecided on every slice, and two earlier single-seed claims in FiLM's favour
— that it rescued judgment, then that it rescued comparison — are retracted.
Note what this comparison is now missing. Both arms lack the cross layer, so
both are bimodal on judgment, and that bimodality is most of their variance.
The experiment that would settle FiLM is cross+FiLM against cross+noFiLM,
now that the noise it has to beat is 0.0015 rather than 0.18. It has not been
run; §6 lists it.
tests/python/test_parity.py, all three presets, on empty questions, empty
states, all 256 byte values, invalid UTF-8 and inputs exactly at the limits:
| comparison | tolerance | observed |
|---|---|---|
| C runtime vs float64 reference, probabilities | 1e-5 | < 1e-6 |
| C runtime vs float64 reference, attention rows | 1e-5 | < 1e-6 |
| PyTorch (fake-quantized) vs float64 reference, logits | 2e-3 | passes |
| bigram hash, C vs NumPy vs pure Python | exact | all 65,536 pairs |
The first two are fp32 accumulation against float64 accumulation and nothing else. The third is looser because PyTorch batches and pads, which changes the order of the additions; it is a check that the export is faithful, not a check of arithmetic.
Stated plainly, because a results page that only lists successes is not a results page.
- Blink does not read. This is the summary of every bullet below. It
learns the form of a decision from its training data; it has no pretrained
knowledge of language, and on tasks that need it — WANLI, SemIf's fixtures,
selection— it is at or near chance. That is the price of 450 KB and 55 µs, and no amount of parameters trained from scratch on a few thousand rows has bought it back. selectionis never learned, by any run, under either option head, in the mixed corpus or trained on nothing else for 100 epochs: 0.2317 ± 0.0211 against a uniform baseline of 0.240. Binding "which city does Omar live in" to the right sentence, when every candidate city also appears in the state, is beyond this architecture as it stands.comparisonis learned once in three. Under the cosine head one run reached 0.396; the other two sat at chance, which is why its spread is 0.0798. It was at chance in every run under the previous head.judgmentis solved on three runs, which is not the same as solved. 0.9991 ± 0.0015 with the cosine head, against 0.9754 ± 0.0497 before it. An earlier revision calledjudgment"stabilised" on three runs that all happened to succeed, and three further runs disproved it; the same caution applies here.- Seeds do not pin a run on MPS. Every standard deviation in this document therefore mixes seed variation with backend non-determinism, and three runs is too few to separate them.
routingcosts a little to add capacity it does not need. 1.0000 ± 0.0000 without the cross layer, 0.9980 ± 0.0035 now — one row in 500 on average. Small, but it is a cost and not a rounding artefact.- The FiLM ablation is undecided, and two earlier single-seed claims in its favour have been retracted. See above.
- Quantization-aware training can destroy a converged model. Switching it on at 70% of the schedule moved validation NLL from 0.87 to 1.51 in one run and from 1.01 to 1.32 in another. The cause in both was a multiplicative weight path being quantized; that path is now fp32, but the abrupt switch remains a sharp edge and the run logs should be read, not assumed.
- Padding the state with irrelevant text costs 10 points. The argmax flips on 27% of rows (0.6334 → 0.5347). The encoder's only long-range path is a masked mean, and a mean is what irrelevant text dilutes; anything that changes the sequence length also moves every pooled window boundary. Measure this on your own data before trusting the accuracy above.
- The model does not transfer. 0.375 and 0.386 on SemIf's two fixtures against a uniform 0.333, and two thirds of their rows are longer than the synthetic model's limits. A 400k-parameter byte model learns what it is trained on.
- Byte limits are a property of the trained container. A corpus with
longer states needs a model trained for it, not a flag at inference time.
--limits-from-datasizes them from the training corpus (WANLI: 0 rows clipped, from 100% with fixed limits). blink-smallis experimental and is not shipped. Its containers are not in the repository (scripts/run_all.sh --smalltrains them). Two reasons. First, it is worse than blink-tiny on every corpus: 0.5956 ± 0.0306 synthetic and 0.4736 on WANLI, against 0.6334 and 0.4913. On the synthetic corpus tiny already saturates the families that are learnable, so capacity has nothing to buy; on WANLI small memorises (training NLL 0.13, validation 2.86). Second, it costs 4.3 ms to encode a 512-byte state against 0.4 ms. Under the previous head it also diverged in one run in three, with the fitted temperature pinned at its bound; the cosine head and 100 epochs removed that failure, and training now warns when the temperature fit hits a bound, but they did not make small better than tiny.- Activations are fp32 by default. The W8A8 build (§2, The W8A8 backend) is about twice as fast with no measurable accuracy cost, but it is not the default: fp32 activations are what make the exact float64 parity check possible.
- One thread. Concurrency is one session per thread over a shared model. There is no intra-decision parallelism.
- The synthetic corpus is templated. Held-out entities show the mechanism generalises past memorised names. They do not show open-domain understanding.
In the order I would do them.
- Settle FiLM against a quiet baseline.
cross+FiLM againstcross+noFiLM, on the current head. The existing ablation compared two arms that were both bimodal onjudgment, so it had to beat a noise floor of 0.18; that floor is now 0.0015, and a real effect of a few points would be visible. Six runs, about two and a half hours. - Re-run the three older comparisons on the cosine head — the
judgmentablation, jevlike on Blink's corpus and jevlike on Wikispeedia — so that every number on this page comes from containers that still load. - A benchmark that requires reading. Wikispeedia turned out to be title matching for both systems. The comparison with jevlike needs a task where the evidence carries the answer, from outside this repository, so neither side designed it.
- Make
selectionlearnable, or establish that it is not. Neither the cross layer nor the cosine head touched it, which is consistent with the diagnosis:selectionneeds binding between two positions inside the state, not between the state and the question. The next thing to try is relative position information in the mixer's attention, so that "Omar" and "Rennes" being adjacent is representable. If that fails, the honest outcome is a documented boundary of the architecture. - Pretraining, if Blink is to read. The WANLI gap to SemIf is not capacity (§5). A byte-level encoder pretrained on unlabelled text, then distilled into the tiny shape, is the only route this page's evidence points at. It is a different project in cost.
- Soften the quantization-aware switch. Ramping the fake-quantization noise in over several epochs, rather than flipping it on at one, would have avoided both blow-ups seen above.
- W8A8. Done as an optional build (§2, The W8A8 backend): 1.8× on blink-tiny and 3.0× on blink-small, accuracy unchanged. What is left is quantization-aware training for the activations, which could let it become the default.
- An x86-64 SIMD path. Done (§2, x86-64): SSE2 always, AVX2 + FMA when the CPU has it, for both W8A32 and W8A8. AVX2 is 2.7× the scalar loop on a GitHub x86 runner. What is left is a benchmark on a dedicated x86 machine, and AVX-512 and VNNI, which are not implemented.
- A Jev comparison, if TypeSafe publishes enough to make one honest. At present nothing here is measured against a Jev endpoint, and this page says so wherever the subject comes up.
Completed from earlier revisions of this list: the three-way disjoint entity
split (closed the ECE gap: 0.131 → 0.026) and corpus-derived byte limits
(--limits-from-data; WANLI went from 100% of rows clipped to 0%, with no
change in accuracy — the limit was not the bottleneck).
Also completed: question-to-state cross attention, which removed the
judgment bimodality that had been the dominant source of variance in every
number on this page.
Also completed: every headline number over three runs; the cosine option head (container format 3), which bounds every logit by construction and ended the blink-small divergence; blink-small at 100 epochs; Wikispeedia against jevlike.