Skip to content

Latest commit

 

History

History
858 lines (711 loc) · 43.5 KB

RESULTS.md

File metadata and controls

858 lines (711 loc) · 43.5 KB

Results

Every number here was produced by bash scripts/run_all.sh and is reproduced by it. The generated tables live in results/SUMMARY.md; this page is the reading of them, including the parts that did not work.

Read METHOD.md first if you intend to quote anything. In particular: the pooled accuracy over the synthetic corpus is an average over four deliberately unlike task families, and its value depends on the mix. The per-family table is the result.

In one paragraph. Blink recognises the form of a decision; it does not read text the way a pretrained language model does. Where the answer is carried by form — which queue a ticket's wording points at, whether a claim's verb agrees with the state's — blink-tiny is near perfect and well calibrated (routing 0.998, judgment 0.999, ECE 0.028). Where the answer requires reading — binding a name to the right sentence, comparing quantities, natural language inference — it is at or a little above chance, and a frozen 4B model is far ahead (WANLI, SemIf's 256 rows: 0.44 against 0.64). The gap is pretraining, not capacity: twenty times the parameters did not close it (§5). Use Blink where form suffices, check that on your own data, and treat its confidence as the thing to branch on.

Machine. Apple M5 Pro, macOS 27.0, Apple clang, single-threaded. -std=c99 -O3 -ffp-contract=off. Exact command lines and checksums are in results/MANIFEST.json.


1. Size

Weights are mapped read-only and never copied, so the container bytes are paid once per machine rather than once per process or per session. Bytes per parameter sits slightly above 1.0 because the fp32 per-row scales, the layer norms, the depthwise convolutions and the FiLM modulation are not quantized.

blink-tiny blink-small
parameters 406,913 7,900,033
container on disk 452.2 KiB 7.93 MiB
bytes per parameter 1.138 1.052
arena, minimum limits 22.2 KiB 55.2 KiB
arena, embedded limits (128 B state, 4 options) 129.0 KiB 300.1 KiB
arena, default limits (16 options) 260.4 KiB 1.11 MiB

64 concurrent sessions over one model:

arena per session (4 options) 249.9 KiB
arenas in total 15.6 MiB
measured resident growth 16.1 MiB
weight bytes counted again 0 (452.2 KiB, mapped once)

The resident growth tracks the arenas and nothing else. That is the whole claim about memory: the marginal cost of one more concurrent decision is one arena, whose size you choose by declaring limits.


2. Latency

Single-threaded, p50 over up to 2,000 samples or 1.5 seconds, whichever comes first, after 50 warm-up iterations. decide re-encodes everything; cached_state holds the state and the menu and changes only the question; new_menu holds the state and supplies a fresh option list.

blink-tiny

state bytes options decide cached_state cached_batch8 new_menu cached decisions/s
64 4 169 µs 43 µs 42 µs 110 µs 22,851
128 4 240 µs 46 µs 46 µs 114 µs 21,061
256 2 366 µs 51 µs 51 µs 83 µs 19,251
256 4 406 µs 54 µs 53 µs 121 µs 18,211
256 16 613 µs 72 µs 72 µs 329 µs 13,666

blink-small

state bytes options decide cached_state cached_batch8 new_menu cached decisions/s
64 4 1.25 ms 291 µs 265 µs 859 µs 3,422
256 4 2.47 ms 300 µs 275 µs 869 µs 3,321
512 2 3.98 ms 306 µs 281 µs 575 µs 3,262
512 4 4.29 ms 314 µs 290 µs 887 µs 3,175
512 16 5.89 ms 373 µs 347 µs 2.50 ms 2,673

SIMD. The NEON kernel against the same build with -DBLINK_SCALAR_ONLY=1 (make scalar):

model path state bytes scalar p50 NEON p50 speedup
blink-tiny decide 256 1.13 ms 406 µs 2.79×
blink-tiny cached_state 256 134 µs 54 µs 2.48×
blink-tiny new_menu 256 352 µs 121 µs 2.91×
blink-small decide 512 15.42 ms 4.29 ms 3.60×
blink-small cached_state 512 1.16 ms 314 µs 3.69×
blink-small new_menu 512 3.41 ms 887 µs 3.84×

Three things to read off these tables.

The state cache is the throughput story. The cached path costs the question and the head, neither of which depends on how long the state was, so the ratio grows with the state length. A KV prefix cache in a frozen LLM gives the same shape of win (SemIf: 2.33 → 20.03 decisions/s), but approximately; here the reuse is exact, because the state and the question are separate sequences and nothing is being approximated. tests/c/test_session.c checks it with memcmp.

Cost is linear in the state, not quadratic. There is no self-attention over bytes; the only quadratic terms run after the pool, over a sequence a quarter or an eighth as long.

Projections run four rows at a time, and batching questions fills the groups. Every int8 projection goes through blink_matmul_i8_rows, which widens each weight row to fp32 once per group of four input vectors instead of once per vector; the widening, not the multiply, is what a W8A32 product pays for. Each vector keeps its own accumulators in the original order, so the output is bit-identical to the one-vector kernel (tests/c/test_kernels.c checks it with memcmp) and every number in section 3 is unchanged. Against the previous build this took 1.3–1.5× off every path. cached_batch8 scores eight questions per blink_score_batch call, which encodes them side by side so that short questions still fill the groups; it adds about 8% on blink-small and almost nothing on blink-tiny, whose per-question cost is dominated by work a batch does not share (the FiLM product, the queries and the head). The batch is bit-identical to scoring the questions one at a time, checked in tests/c/test_session.c. The price is a fixed four-row scratch in the arena, visible in section 1 as a few KiB more at every limit.

The Accelerate backend

make ACCELERATE=1 (macOS) sends the projections, the layer norms and the attention reads through Accelerate, which runs matrix products on the Apple silicon matrix unit. Same machine, one thread, 4 options (results/bench-latency-accelerate.jsonl):

model path state bytes default p50 Accelerate p50 speed-up
blink-tiny decide 256 406 µs 172 µs 2.36×
blink-tiny cached_state 256 54 µs 32 µs 1.69×
blink-tiny new_menu 256 121 µs 62 µs 1.95×
blink-small decide 512 4.29 ms 824 µs 5.20×
blink-small cached_state 512 314 µs 135 µs 2.33×
blink-small cached_batch8 512 290 µs 71 µs 4.06×

blink-small gains more because its products are larger: 192 wide against 64, which is where the matrix unit pays for its overhead.

The default build's profile explains the ceiling. On a blink-tiny decide, 60% of the time was the int8 projections; with those moved, what is left is embeddings, layer norms and attention, and moving those to vDSP and BLAS brought the remaining time down again, from 238 µs to 172 µs.

The price is described in the README (Accelerate backend): Accelerate allocates inside its large products, 16 times per blink-tiny decision (measured by interposing malloc; the default build makes zero); the model holds a 352 KiB fp32 copy of the projections; the arena is 335 KiB instead of 260 KiB; results agree with the default build to fp32 rounding rather than bit for bit. On the synthetic test split, 4,000 rows, no argmax changes and the largest probability difference is 1.5e-6. The accuracy tables in §3 are from the default build, and they hold for this one to that precision.

The W8A8 backend

make W8A8=1 quantizes the input of every projection to int8, one scale per row, and multiplies with integer instructions: SDOT, or I8MM on non-Apple cores that report it. This one has I8MM, but SDOT is the default on Apple silicon for the reason below; BLINK_W8A8_KERNEL=i8mm selects the other. Same machine, one thread, 4 options (results/bench-latency-w8a8.jsonl, the SDOT default):

model path state bytes default p50 W8A8, SDOT W8A8, I8MM speed-up (SDOT)
blink-tiny decide 256 406 µs 221 µs 217 µs 1.84×
blink-tiny cached_state 256 54 µs 33 µs 33 µs 1.64×
blink-tiny new_menu 256 121 µs 64 µs 63 µs 1.89×
blink-small decide 512 4.29 ms 1.45 ms 1.46 ms 2.95×
blink-small cached_state 512 314 µs 115 µs 116 µs 2.73×
blink-small new_menu 512 887 µs 268 µs 272 µs 3.31×

On this core the two instructions are level, which is why SDOT is the default on Apple silicon. SMMLA does twice the products per instruction, but a plain loop issues SDOT faster (measured 140 against 193 GMAC/s at the instruction level), and both kernels reach about 170 GMAC/s on blink-small's projections, where loads rather than arithmetic set the pace. Against the Accelerate backend, W8A8 is slower on a fresh decision (221 µs against 172 µs for blink-tiny) and level or faster once the state is cached, and it keeps what Accelerate gives up: nothing allocates while scoring (checked by interposing malloc), the arena grows by 1 KiB rather than 75, and it needs no framework. The I8MM kernel reads a copy of the projections rearranged into SMMLA's layout at open time, 88 KiB for blink-tiny and 1.3 MiB for blink-small; SDOT needs none, which on a core where the two are level decides it.

Accuracy. Every published model, evaluated on its own test split with both builds by the same harness (BLINK_LIBRARY=build-w8a8/libblink.dylib):

model rows top-1, default top-1, W8A8 ECE, default ECE, W8A8 decisions changed
blink-tiny, synthetic s7 4,000 0.6530 0.6542 0.0561 0.0558 2.1%
blink-tiny, synthetic s17 4,000 0.6215 0.6212 0.0236 0.0222 8.0%
blink-tiny, synthetic s27 4,000 0.6258 0.6280 0.0030 0.0059 16.1%
blink-small, synthetic s7 4,000 0.6030 0.6062 0.0134 0.0117 5.4%
blink-small, synthetic s17 4,000 0.6218 0.6260 0.0117 0.0076 3.8%
blink-small, synthetic s27 4,000 0.5620 0.5605 0.0506 0.0509 1.5%
blink-tiny, WANLI s7 4,945 0.4861 0.4855 0.0294 0.0300 1.0%
blink-tiny, WANLI s17 4,945 0.4967 0.4952 0.0197 0.0168 1.0%
blink-tiny, WANLI s27 4,945 0.4910 0.4902 0.0245 0.0239 0.6%
blink-small, WANLI s7 4,945 0.4762 0.4777 0.0258 0.0274 1.0%
blink-small, WANLI s17 4,945 0.4726 0.4730 0.0198 0.0215 1.0%
blink-small, WANLI s27 4,945 0.4720 0.4720 0.0360 0.0360 1.0%
blink-tiny, Doom 1,191 0.8338 0.8346 0.0310 0.0322 0.5%
blink-tiny, Snake 5,693 0.7887 0.7897 0.0025 0.0031 0.4%
blink-tiny, Pilot 6,981 0.8199 0.8199 0.0291 0.0289 0.1%

Top-1 moves by −0.15 to +0.42 points, in both directions and inside the between-seed spread; calibration is unchanged. The decisions that change are almost all in the two synthetic families the model does not learn, comparison and selection, where it sits at chance with near-uniform probabilities (the largest shift there is 0.016) and a rounding picks a different near-tie. On blink-tiny s27, the worst case, 31% of comparison and 34% of selection rows change against 0.1% of judgment and none of routing. These models were trained with fake quantization of the weights only; quantization-aware training of the activations is not done.

x86-64

The projection kernels have x86-64 SIMD paths, chosen at run time so one binary runs on any x86-64 machine: SSE2, which x86-64 guarantees, and AVX2 with FMA when the CPU reports both (BLINK_X86_SIMD=scalar|sse2|avx2 forces one). In the W8A8 build the integer products get the same two levels. SSE2 sign-extends to int16 and uses pmaddwd; AVX2 moves the weight's sign onto the input (vpsignb) and uses vpmaddubsw, which cannot saturate at these magnitudes. So both are exact and byte-identical to the scalar integer kernel, as SDOT and I8MM are.

No x86 machine was available for this work. Everything below ran on the Apple M5 Pro through Docker's linux/amd64 emulation, which is Rosetta and reports SSE2 through AVX2 and FMA, without AVX-512 or VNNI. Correctness carries over; speed does not. bash scripts/x86-docker-test.sh reruns all of it (tests, parity, timings), and on an x86 host its timings become real.

  • The C suite passes at every level (scalar, SSE2, AVX2, forced in turn), for W8A32 and W8A8, with the SIMD compiled out, at -O0, and under AddressSanitizer and UndefinedBehaviorSanitizer, with GCC 14. The row-blocked product stays bit-identical to the one-vector product within each level.
  • On 300 test rows of blink-tiny, SSE2 and AVX2 agree with the float64 reference to 6e-7 (the tolerance is 1e-5) and with NEON to 1e-6, with no argmax change. In the W8A8 build, SSE2 and AVX2 give identical bytes and agree with the Arm W8A8 build to 9e-8.

Timings under the emulator, p50, 4 options. They are only meaningful relative to each other within a row, and not even that for AVX2:

model path state scalar SSE2 AVX2 W8A8 SSE2 W8A8 AVX2
blink-tiny decide 256 1,334 µs 681 µs 1,006 µs 686 µs 1,100 µs
blink-tiny cached_state 256 168 µs 90 µs 127 µs 87 µs 134 µs
blink-small decide 512 18.1 ms 6.79 ms 8.58 ms 6.85 ms 10.98 ms
blink-small cached_state 512 1,447 µs 492 µs 617 µs 462 µs 733 µs

SSE2 is 2.0–2.9× the scalar loop. AVX2 comes out slower than SSE2 here only because Rosetta executes each 256-bit instruction as 128-bit Arm ones.

On real x86 hardware it is the other way round. The x86 SIMD levels step in CI times each level on a GitHub-hosted ubuntu-latest runner, built with GCC. That is a shared virtual machine whose CPU model the run does not record, so the absolute numbers are indicative, but the ratios within the run are meaningful. blink-tiny, 256-byte state, 4 options, p50 (first run, commit a1bdc6c):

path scalar SSE2 AVX2 + FMA
decide 2,075 µs 979 µs 760 µs
cached_state 245 µs 117 µs 90 µs
new_menu 645 µs 292 µs 222 µs

SSE2 is 2.1–2.2× the scalar loop and AVX2 2.7×, so AVX2 is the right default where the CPU has it.

Context

SemIf reports 20.03 decisions/s for a frozen Qwen3.5-4B with parallel suffixes on an RTX 3090, against 2.33 for fresh scoring. blink-small reaches roughly 3,200 decisions/s on one CPU core, 3,400 with blink_score_batch. This is not a like-for-like comparison — different models, different hardware, different tasks, and a four-billion- parameter model is doing something a 7.8-million-parameter model cannot. It is here to say what changes when the model is small enough to embed, not to claim a win.

Comparison with other systems

Two open projects expose the same interface as Blink: jevlike, a small PyTorch model trained from scratch, and SemIf, which reads option logits out of a frozen 4B LLM. Blink is measured against both.

Speed. One decision over a 256-byte state, a question and four options, batch one. Blink and jevlike were measured on the same machine, one CPU thread each; the SemIf figures are the ones it publishes, on GPUs.

system hardware fresh decision, p50 decisions/s, fresh decisions/s, state reused
blink-tiny (C99, int8) 1 CPU core, Apple M5 Pro 406 µs 2,448 18,211
blink-tiny, Accelerate backend 1 CPU core + matrix unit, Apple M5 Pro 172 µs 5,808 31,290
blink-tiny, W8A8 backend (SDOT) 1 CPU core, Apple M5 Pro 221 µs 4,460 29,481
jevlike, width 288 (matched capacity) 1 CPU core, Apple M5 Pro, PyTorch 188 µs 5,275 no state cache
jevlike, width 64 (default) 1 CPU core, Apple M5 Pro, PyTorch 96 µs 10,321 no state cache
SemIf, Qwen3.5-4B BF16 RTX 3090 ~430 ms 2.33 20.03
SemIf, Qwen3.5-4B BF16, MLX Apple M5 Max GPU ~610 ms 1.63 17.24

Memory.

system parameters weights process peak RSS allocation while scoring
blink-tiny 407k 452 KiB int8, mapped read-only, shared across sessions 2.5 MiB none (260 KiB arena per session, sized up front)
blink-tiny, Accelerate backend 407k 452 KiB int8 mapped, plus a 352 KiB fp32 copy of the projections 7.0 MiB inside Accelerate: 16 per decision (335 KiB arena)
blink-tiny, W8A8 backend (SDOT) 407k 452 KiB int8, mapped read-only, no copy 2.6 MiB none (261 KiB arena)
jevlike, width 288 398k 1.5 MiB fp32 199 MiB PyTorch allocator
jevlike, width 64 45k 177 KiB fp32 196 MiB PyTorch allocator
SemIf, Qwen3.5-4B BF16, MLX 4B 7.83 GiB resident 9.31 GiB framework allocator
SemIf, Qwen3.5-4B Q4, MLX 4B — 2.82 GiB framework allocator

How to read the two tables:

  • On a fresh decision the default build is slower than jevlike. jevlike is Python, but its arithmetic runs in PyTorch's native kernels, which on Apple silicon use the CPU's matrix unit. Its model is also a byte embedding and one attention read, with no encoder in between. Blink spends that time on the encoder, and it shows in the accuracy below: at matched capacity it is ten points ahead on top-1 and seven times better calibrated.
  • The Accelerate backend closes that gap. With the projections on the same matrix unit, blink-tiny is faster than jevlike on a fresh decision too (172 µs against 188 µs), still on one thread. The W8A8 backend gets to 221 µs without allocating and without a framework.
  • Blink is faster once the state is encoded. Encoding a state once and asking it many questions is the common case in software, and the reuse is exact. Blink then answers 3.5× faster than jevlike at matched capacity, and 5.9× with the Accelerate backend.
  • Memory is where the runtimes differ most. jevlike's weights are small, but a PyTorch process needs about 80 times Blink's whole footprint. SemIf needs gigabytes and a GPU, and on a fresh decision it is about a thousand times slower.
  • A frozen 4B model reads language that Blink cannot. These tables compare cost, not capability.

Blink's figures come from make bench (results/bench-latency.jsonl, and results/bench-latency-accelerate.jsonl and results/bench-latency-w8a8.jsonl for the two optional backends) and from /usr/bin/time -l on blink answering one decision. jevlike's come from eval/bench_external.py (results/bench-external-w*.json), on the checkpoints trained by scripts/experiments/queue_external.sh. SemIf's are quoted from its published RTX 3090 and MLX results; the fresh latency is the inverse of its fresh rate.

Accuracy, measured head to head: same corpus, same harness, three runs each (details; measured on Blink's previous option head, the current one scores 0.6334 ± 0.0171):

parameters top-1 ECE
blink-tiny 407k 0.6064 ± 0.0149 0.0216 ± 0.0036
jevlike, matched capacity 398k 0.5004 ± 0.0397 0.1514 ± 0.0420

On SemIf's 256-row WANLI subset, rebuilt from the identifiers it publishes, a frozen Qwen3.5-4B scores 0.637 balanced accuracy and blink-tiny scores 0.4393 ± 0.0342, with one ten-thousandth of the parameters. Blink does not compete on open-domain language, and External corpora says so with the controls next to it. On jevlike's Wikispeedia next-click task the two are level (0.306 against 0.313), and neither reads the article: the task reduces to matching titles.


3. Accuracy

Everything between the markers below is generated by scripts/report_results.py from results/eval-*.json, as are §1 and §2. It is generated because it was hand-copied for several revisions and went stale the moment the architecture changed — at one point quoting numbers from models that no longer loaded. Regenerate it with python scripts/report_results.py --write.

Every figure is mean ± standard deviation over three independent runs (see the note on determinism below: on MPS a seed does not pin a run). Reliability tables, risk–coverage curves and bootstrap intervals are in results/eval-*.json; row-level predictions are regenerated into results/predictions/ by the same script and are not committed.

Synthetic corpus, held-out entities

3 runs, 4000 test rows over 2000 scenario groups whose entities appear in neither the training nor the validation split. Nothing clipped.

blink-tiny blink-small
top-1 0.6334 ± 0.0171 0.5956 ± 0.0306
balanced accuracy 0.4909 ± 0.0187 0.4583 ± 0.0183
macro F1 0.4918 ± 0.0168 0.4578 ± 0.0174
NLL 0.7166 ± 0.0027 0.7842 ± 0.0784
Brier 0.3756 ± 0.0054 0.4175 ± 0.0488
ECE 0.0276 ± 0.0268 0.0252 ± 0.0220
uniform baseline 0.2818 ± 0.0000 0.2818 ± 0.0000
majority baseline 0.2865 ± 0.0000 0.2865 ± 0.0000

Per family, which is the part that means anything:

family rows blink-tiny blink-small uniform
comparison 1000 0.3043 ± 0.0798 0.2547 ± 0.0092 0.256
judgment 1500 0.9991 ± 0.0015 0.9467 ± 0.0924 0.333
routing 500 0.9980 ± 0.0035 0.9413 ± 0.0764 0.261
selection 1000 0.2317 ± 0.0211 0.2370 ± 0.0069 0.240

Calibration

Temperature is fitted on validation, whose entity pool is disjoint from both training and test, and is then measured here.

blink-tiny (3 runs)
fitted temperature 1.761 ± 0.800
ECE 0.0276 ± 0.0268
MCE 0.3723 ± 0.2124
NLL 0.7166 ± 0.0027
mean confidence 0.6444 ± 0.0217

Reliability, one run (bins holding at least 40 rows):

confidence bin rows mean confidence accuracy gap
0.13 – 0.20 111 0.185 0.144 +0.041
0.20 – 0.27 311 0.232 0.161 +0.072
0.27 – 0.33 364 0.296 0.209 +0.087
0.33 – 0.40 466 0.364 0.339 +0.025
0.40 – 0.47 281 0.431 0.335 +0.097
0.47 – 0.53 200 0.498 0.395 +0.103
0.53 – 0.60 103 0.562 0.427 +0.134
0.60 – 0.67 76 0.633 0.579 +0.054
0.67 – 0.73 45 0.698 0.622 +0.076
0.73 – 0.80 51 0.773 0.765 +0.008
0.80 – 0.87 119 0.842 0.941 -0.099
0.87 – 0.93 366 0.910 0.997 -0.088
0.93 – 1.00 1507 0.973 1.000 -0.027

Risk–coverage, averaged over runs:

confidence threshold coverage accuracy when answered
0.00 1.000 ± 0.000 0.6334 ± 0.0171
0.50 0.533 ± 0.049 0.9702 ± 0.0400
0.60 0.514 ± 0.024 0.9889 ± 0.0180
0.70 0.505 ± 0.009 0.9951 ± 0.0080
0.80 0.499 ± 0.001 0.9985 ± 0.0022
0.90 0.481 ± 0.032 0.9998 ± 0.0003
0.95 0.435 ± 0.104 1.0000 ± 0.0000

Controls

intact shuffled state shuffled question uniform
blink-tiny 0.6334 ± 0.0171 0.4019 ± 0.0032 0.3441 ± 0.0113 0.282
blink-small 0.5956 ± 0.0306 0.3771 ± 0.0181 0.3295 ± 0.0309 0.282
untrained weights top-1 uniform
blink-tiny, random 0.2900 0.282
blink-small, random 0.2900 0.282

Perturbations

perturbation argmax flip rate mean total variation top-1 before after
reversed options 0.0000 ± 0.0000 0.0000 ± 0.0000 0.6334 ± 0.0171 0.6334 ± 0.0171
irrelevant suffix 0.2721 ± 0.0285 0.1238 ± 0.0402 0.6334 ± 0.0171 0.5347 ± 0.0429

External corpora

WANLI, trained on its own train split — a generalisation test, not a transfer test. Byte limits were taken from the corpus (--limits-from-data), so nothing is clipped.

rows 4945
clipped 0
top-1 0.4913 ± 0.0053
balanced accuracy 0.4914 ± 0.0055
ECE 0.0245 ± 0.0049
uniform baseline 0.333
shuffled state 0.4260 ± 0.0031
shuffled question 0.4404 ± 0.0055

The same 256 WANLI rows SemIf evaluates. Rebuilt from the row identifiers SemIf commits, in SemIf's display order, from the same pinned revision. No premise in the subset appears in Blink's training split.

system parameters balanced accuracy
SemIf, Qwen3.5-4B frozen 4,000,000,000 0.637
blink-tiny, trained 390,401 0.4393 ± 0.0342
uniform baseline — 0.333

Blink's controls on this subset: shuffled state 0.3659 ± 0.0266, shuffled question 0.3906 ± 0.0156. The margin over the controls is about eight points, not the eleven the headline suggests.

Transfer to the external fixtures. A model trained on the synthetic corpus, asked for something it has never seen.

fixture rows clipped top-1 uniform
authored144 144 96 0.3750 ± 0.0069 0.333
perturbations108 108 87 0.3858 ± 0.0809 0.333

blink-small on WANLI, same protocol, three runs: top-1 0.4736 ± 0.0023, below blink-tiny's 0.4913, and 0.4467 ± 0.0150 balanced on SemIf's 256 rows against tiny's 0.4393 — inside tiny's spread. It memorises: final training NLL 0.13 against validation 2.86. See §5.

Measured on the previous option head. Everything from here to §4 predates the cosine option head (container format 3). Each subsection compares arms trained under the same head, so the comparisons stand, but the absolute numbers are older than §3 above: blink-tiny was 0.6064 ± 0.0149 pooled then and is 0.6334 ± 0.0171 now, and judgment went from 0.9754 ± 0.0497 to 0.9991 ± 0.0015. The containers these rows came from are format 2 and no longer load; re-running the three comparisons on the current head is §6 item 2.

judgment and question-to-state attention

A note on what "run" means here. PyTorch on the MPS backend is not deterministic: the same seed, the same corpus and the same architecture diverge by the first epoch. Measured on a minimal case, two runs at seed 7 give 1.3074637349446614 twice on CPU and 1.307463773091634 against 1.307463755607605 on MPS. So the figures below are independent runs, not seeds, and every standard deviation in this document conflates the two sources of variation. See REPRODUCE.md.

Three arms, 60 epochs, identical otherwise:

  • cross — the full model. Six runs.
  • film — without the question-to-state cross layer. Three runs.
  • nofilm — without the cross layer and without the query conditioning. Three runs.
slice cross (6 runs) film (3) nofilm (3)
judgment 0.9754 ± 0.0497 0.8778 ± 0.1819 0.7773 ± 0.1928
worst run 0.8753 0.6687 0.6653

The cross layer raises the mean by about ten points and cuts the spread by a factor of 3.7. That is a real improvement and it is not the stabilisation an earlier revision of this page claimed: it reported a factor of 120, from three runs that had all happened to succeed. Three further runs of the same configuration scored 0.875, 0.980 and 1.000. judgment is still occasionally learned only in part; it is no longer occasionally lost entirely.

Why, at the sub-case level

From the three-run comparison, before the additional runs:

arm run supports contradicts insufficient
cross 1 1.0000 1.0000 1.0000
cross 2 1.0000 1.0000 1.0000
cross 3 0.9920 1.0000 1.0000
film 1 0.7740 0.2320 1.0000
film 2 1.0000 1.0000 1.0000
film 3 0.9060 0.9880 1.0000
nofilm 1 0.0920 0.9040 1.0000
nofilm 2 1.0000 0.0000 1.0000
nofilm 3 1.0000 1.0000 1.0000

insufficient is solved by every run in every arm — the claim mentions something the state never touches, which is a lexical signal. The other two differ only in a verb, over the same state, so telling them apart requires comparing one word in the question against one word in the state. Without the cross layer a run can collapse the two classes entirely: nofilm run 2 scores supports 1.000 and contradicts 0.000. That total collapse is what has not recurred with the cross layer.

The effect is confined to judgment: selection and comparison are untouched, and routing is very slightly worse, which is the honest cost of adding capacity a family does not need.

Head to head with jevlike

jevlike is the one comparison that is fair without qualification: both models are small and both are trained from scratch on the task data. Its frozen-encoder results measure a pretrained encoder, which is a different experiment.

Same corpus, same 60 epochs, jevlike's own default learning rate, three runs each, and both sets of predictions scored by eval/evaluate.py so the arithmetic is identical. --context-tokens was set from the corpus so nothing jevlike reads is truncated — a comparison against a truncated model measures the truncation. Reproduce with eval/compare_external.py and scripts/experiments/queue_external.sh.

system parameters top-1 balanced ECE
blink-tiny 407k 0.6064 ± 0.0149 0.4723 ± 0.0085 0.0216 ± 0.0036
jevlike, width 288 398k 0.5004 ± 0.0397 0.4025 ± 0.0363 0.1514 ± 0.0420
jevlike, width 64 (default) 45k 0.5116 ± 0.0425 0.4161 ± 0.0356 0.1755 ± 0.0815
system routing judgment comparison selection
blink-tiny 0.9900 ± 0.0156 0.9518 ± 0.0670 0.2657 ± 0.0276 0.2373 ± 0.0159
jevlike, width 288 0.8420 ± 0.0978 0.7191 ± 0.0717 0.2607 ± 0.0045 0.2413 ± 0.0049
jevlike, width 64 0.8733 ± 0.0833 0.7493 ± 0.0911 0.2507 ± 0.0180 0.2350 ± 0.0182
uniform baseline 0.261 0.333 0.256 0.240

At matched capacity Blink is ahead by 15 points on routing and 23 on judgment, and the intervals do not overlap on judgment. The largest gap is calibration: 7× lower ECE. jevlike is confident when it is wrong, which for a model whose output is meant to drive an if statement is the difference between usable and not.

Two things this table says that are worth more than the win.

Capacity is not jevlike's bottleneck. Nine times the parameters left it slightly worse (0.5004 against 0.5116). It is the clearest evidence that Blink's margin comes from the stem, the pooling, the mixer and the question-to-state layer rather than from spending more weights on the same structure.

The two hard families are hard for both. comparison and selection sit at chance in all three systems and every run. Blink's advantage is real and it is confined entirely to the families this class of model can address.

What this does not establish. The corpus is Blink's own. It was built to separate capabilities rather than to flatter anything, but it is still home ground. The comparison in the other direction follows.

Head to head with jevlike, on jevlike's ground

The comparison above runs on Blink's corpus. This one runs on Wikispeedia next-click, where jevlike publishes 26% with a frozen Qwen2.5-0.5B encoder and 29% from scratch. It is the direction that does not favour Blink.

eval/build_wikispeedia.py reimplements jevlike's selection exactly — same hash, same step, same menu sampling, same split by target article — and emits both formats from one selection. 41,291 training rows (jevlike reports "40,000 clicks"), 4,373 test rows over 314 target articles, up to 64 options and 43 on average. Both systems get 512 bytes of article body and nothing truncated; Blink puts the navigation goal in the question and the body in the state, jevlike gets the same bytes in its single context. Controls are applied to both in the same way (compare_external.py score-native), which jevlike's own shuffle_context does not do: it swaps titles and body together.

blink-tiny jevlike, width 288
parameters 434,049 526,752
top-1 0.3059 ± 0.0191 0.3127 ± 0.0041
ECE 0.0346 ± 0.0070 0.0396 ± 0.0075
shuffled state 0.2900 ± 0.0178 0.3034 ± 0.0043
shuffled question 0.0690 ± 0.0045 0.0725 ± 0.0060
uniform baseline 0.0362 0.0362

It is a tie. jevlike is slightly ahead, inside Blink's spread, with 21% more parameters (its position table grows with the 704-byte context) and far lower run-to-run variance. Calibration is close, unlike on the synthetic corpus where Blink's was seven times better.

Neither model reads the article.

evidence margin: intact − shuffled state
blink-tiny 0.0159 ± 0.0013
jevlike 0.0093 ± 0.0010

Replacing the article body with another article's costs jevlike 0.9 points out of 31 and Blink 1.6. Both solve the task almost entirely by matching the target title in the question against the link titles. Blink extracts 1.7 times as much from the evidence and the intervals do not overlap, but on a scale of thirty points that difference is real and negligible at once.

The finding worth more than either number is about the benchmark. Wikispeedia next-click, built this way, is a title-matching task. That applies to jevlike's published 26–29% as well: with its default 192-byte context, jevlike saw the titles and almost none of the body. This replication of jevlike reaches 0.3127, slightly above the published 0.29, which is consistent with giving it 704 bytes instead of 192.

Across both corpora, then: Blink is ahead where the evidence matters and the two are level where the task reduces to titles.

The FiLM ablation: still undecided

The film and nofilm arms above differ only in the query conditioning, and their difference remains inside their own spread:

slice film nofilm gap larger within-arm sd
pooled 0.5796 0.5400 +0.040 0.077
judgment 0.8778 0.7773 +0.100 0.193
comparison 0.2687 0.2583 +0.010 0.036
selection 0.2337 0.2357 −0.002 0.008

Undecided on every slice, and two earlier single-seed claims in FiLM's favour — that it rescued judgment, then that it rescued comparison — are retracted.

Note what this comparison is now missing. Both arms lack the cross layer, so both are bimodal on judgment, and that bimodality is most of their variance. The experiment that would settle FiLM is cross+FiLM against cross+noFiLM, now that the noise it has to beat is 0.0015 rather than 0.18. It has not been run; §6 lists it.


4. Parity

tests/python/test_parity.py, all three presets, on empty questions, empty states, all 256 byte values, invalid UTF-8 and inputs exactly at the limits:

comparison tolerance observed
C runtime vs float64 reference, probabilities 1e-5 < 1e-6
C runtime vs float64 reference, attention rows 1e-5 < 1e-6
PyTorch (fake-quantized) vs float64 reference, logits 2e-3 passes
bigram hash, C vs NumPy vs pure Python exact all 65,536 pairs

The first two are fp32 accumulation against float64 accumulation and nothing else. The third is looser because PyTorch batches and pads, which changes the order of the additions; it is a check that the export is faithful, not a check of arithmetic.


5. What does not work

Stated plainly, because a results page that only lists successes is not a results page.

  • Blink does not read. This is the summary of every bullet below. It learns the form of a decision from its training data; it has no pretrained knowledge of language, and on tasks that need it — WANLI, SemIf's fixtures, selection — it is at or near chance. That is the price of 450 KB and 55 µs, and no amount of parameters trained from scratch on a few thousand rows has bought it back.
  • selection is never learned, by any run, under either option head, in the mixed corpus or trained on nothing else for 100 epochs: 0.2317 ± 0.0211 against a uniform baseline of 0.240. Binding "which city does Omar live in" to the right sentence, when every candidate city also appears in the state, is beyond this architecture as it stands.
  • comparison is learned once in three. Under the cosine head one run reached 0.396; the other two sat at chance, which is why its spread is 0.0798. It was at chance in every run under the previous head.
  • judgment is solved on three runs, which is not the same as solved. 0.9991 ± 0.0015 with the cosine head, against 0.9754 ± 0.0497 before it. An earlier revision called judgment "stabilised" on three runs that all happened to succeed, and three further runs disproved it; the same caution applies here.
  • Seeds do not pin a run on MPS. Every standard deviation in this document therefore mixes seed variation with backend non-determinism, and three runs is too few to separate them.
  • routing costs a little to add capacity it does not need. 1.0000 ± 0.0000 without the cross layer, 0.9980 ± 0.0035 now — one row in 500 on average. Small, but it is a cost and not a rounding artefact.
  • The FiLM ablation is undecided, and two earlier single-seed claims in its favour have been retracted. See above.
  • Quantization-aware training can destroy a converged model. Switching it on at 70% of the schedule moved validation NLL from 0.87 to 1.51 in one run and from 1.01 to 1.32 in another. The cause in both was a multiplicative weight path being quantized; that path is now fp32, but the abrupt switch remains a sharp edge and the run logs should be read, not assumed.
  • Padding the state with irrelevant text costs 10 points. The argmax flips on 27% of rows (0.6334 → 0.5347). The encoder's only long-range path is a masked mean, and a mean is what irrelevant text dilutes; anything that changes the sequence length also moves every pooled window boundary. Measure this on your own data before trusting the accuracy above.
  • The model does not transfer. 0.375 and 0.386 on SemIf's two fixtures against a uniform 0.333, and two thirds of their rows are longer than the synthetic model's limits. A 400k-parameter byte model learns what it is trained on.
  • Byte limits are a property of the trained container. A corpus with longer states needs a model trained for it, not a flag at inference time. --limits-from-data sizes them from the training corpus (WANLI: 0 rows clipped, from 100% with fixed limits).
  • blink-small is experimental and is not shipped. Its containers are not in the repository (scripts/run_all.sh --small trains them). Two reasons. First, it is worse than blink-tiny on every corpus: 0.5956 ± 0.0306 synthetic and 0.4736 on WANLI, against 0.6334 and 0.4913. On the synthetic corpus tiny already saturates the families that are learnable, so capacity has nothing to buy; on WANLI small memorises (training NLL 0.13, validation 2.86). Second, it costs 4.3 ms to encode a 512-byte state against 0.4 ms. Under the previous head it also diverged in one run in three, with the fitted temperature pinned at its bound; the cosine head and 100 epochs removed that failure, and training now warns when the temperature fit hits a bound, but they did not make small better than tiny.
  • Activations are fp32 by default. The W8A8 build (§2, The W8A8 backend) is about twice as fast with no measurable accuracy cost, but it is not the default: fp32 activations are what make the exact float64 parity check possible.
  • One thread. Concurrency is one session per thread over a shared model. There is no intra-decision parallelism.
  • The synthetic corpus is templated. Held-out entities show the mechanism generalises past memorised names. They do not show open-domain understanding.

6. What is still missing

In the order I would do them.

  1. Settle FiLM against a quiet baseline. cross+FiLM against cross+noFiLM, on the current head. The existing ablation compared two arms that were both bimodal on judgment, so it had to beat a noise floor of 0.18; that floor is now 0.0015, and a real effect of a few points would be visible. Six runs, about two and a half hours.
  2. Re-run the three older comparisons on the cosine head — the judgment ablation, jevlike on Blink's corpus and jevlike on Wikispeedia — so that every number on this page comes from containers that still load.
  3. A benchmark that requires reading. Wikispeedia turned out to be title matching for both systems. The comparison with jevlike needs a task where the evidence carries the answer, from outside this repository, so neither side designed it.
  4. Make selection learnable, or establish that it is not. Neither the cross layer nor the cosine head touched it, which is consistent with the diagnosis: selection needs binding between two positions inside the state, not between the state and the question. The next thing to try is relative position information in the mixer's attention, so that "Omar" and "Rennes" being adjacent is representable. If that fails, the honest outcome is a documented boundary of the architecture.
  5. Pretraining, if Blink is to read. The WANLI gap to SemIf is not capacity (§5). A byte-level encoder pretrained on unlabelled text, then distilled into the tiny shape, is the only route this page's evidence points at. It is a different project in cost.
  6. Soften the quantization-aware switch. Ramping the fake-quantization noise in over several epochs, rather than flipping it on at one, would have avoided both blow-ups seen above.
  7. W8A8. Done as an optional build (§2, The W8A8 backend): 1.8× on blink-tiny and 3.0× on blink-small, accuracy unchanged. What is left is quantization-aware training for the activations, which could let it become the default.
  8. An x86-64 SIMD path. Done (§2, x86-64): SSE2 always, AVX2 + FMA when the CPU has it, for both W8A32 and W8A8. AVX2 is 2.7× the scalar loop on a GitHub x86 runner. What is left is a benchmark on a dedicated x86 machine, and AVX-512 and VNNI, which are not implemented.
  9. A Jev comparison, if TypeSafe publishes enough to make one honest. At present nothing here is measured against a Jev endpoint, and this page says so wherever the subject comes up.

Completed from earlier revisions of this list: the three-way disjoint entity split (closed the ECE gap: 0.131 → 0.026) and corpus-derived byte limits (--limits-from-data; WANLI went from 100% of rows clipped to 0%, with no change in accuracy — the limit was not the bottleneck).

Also completed: question-to-state cross attention, which removed the judgment bimodality that had been the dominant source of variance in every number on this page.

Also completed: every headline number over three runs; the cosine option head (container format 3), which bounds every logit by construction and ended the blink-small divergence; blink-small at 100 epochs; Wikispeedia against jevlike.