trynix benchmarks

A benchmark of the QEMU emulator behind trynix, measured across its engine releases to check that each change to it was an improvement.

Real packages, cold and warm

Wall seconds to run each package in a fresh guest, first cold and then again warm, plus the time to boot to a shell.

Instruction classes

Millions of guest instructions per second on a fixed assembly loop per mechanism the emulator pays for, so a change names a mechanism rather than a workload. Higher is better.

What moved the numbers

The changes behind the releases, in order. performance.md and engine-execution.md carry the profiles and the dead ends.

The store share

patch 0002cache=loose on the 9p mount

first ruby -e 1: about 35 s to about 15 s

Packages reach the guest over a 9p share. The guest now caches the share instead of asking the host on every lookup, and resolves a path in one syscall instead of one per component.

Entropy

reseed.crdrand and virtio-rng in machine.json

first run of any binary: about 50 s to 2.5 s

The snapshot is taken before the kernel has seeded its random pool, so the first program to ask for randomness spun in a jitter loop for most of a minute. The guest now has a hardware random source and reseeds at boot.

The clock

patch 0003

guest time was 3.3x slow; sleep 10 took 33 s, now 10

The snapshot was taken on a 3.29 GHz cycle counter and resumed on a 1 GHz one, so the guest's clock ran 3.3x slow. Both builds now count the same clock, and instruction-class numbers from before are corrected by the measured ratio.

Idle, and the engine's size

xterm-pty patchioeventfd=off 路 DWARF stripped

idle guest 1.45 to 0.39 cores; engine download 41 MB to 13 MB

The main loop never slept because the pty poll returned at once; it now waits, with 9p off that loop so the wait is safe. Stripping debug info from the engine cut the download to a third.

Correctness

patch 0004 路 guest CPU raised to x86-64-v3

no speed change; opencode runs at all

The newer CPU model exposed a POPCNT that answered from a stale register, now fixed and checked in CI. opencode needs SSE4.2 and crashed until that fix, so its chart starts here.

The backend

patch 0006

opencode --version 408 s to 171 s; one compiled block 349 to 1163 mips

Blocks now compile in batches of 64 after 32 runs instead of one module per block after 1500, tail-call their successor instead of returning to a C dispatcher, and keep the guest registers in wasm locals. What remains is generated code spread over about 48,000 blocks, cache-missing on every transition.

The transition

patch 0007

two- and four-block loops 1.5x to 1.9x; opencode --version warm 182 s to 173 s here, cold within noise

A jump now caches its successor's function index in its own block, a successor in the same batch is called directly behind a guard on the live jump target, chained entries skip the rewind check only dispatcher entries need, and 64-bit multiplies are emitted inline instead of through a helper. The single hot block and the memory tests do not move; the cold-code rows do because they carry an imul per block.

Linux 7.2.5

nix/guest.nix 路 guest kernel 6.1.187 to 7.2.5

cold jj --version 3.19 s to 3.72 s, cold python 5.30 s to 5.58 s; warm unchanged

The new kernel fixes ELF loading for Fil-C binaries and costs every cold exec: it does about 70% more kernel work per first run over the 9p share, from a forward-only readahead window for executables (6.16), read completion on a work item (6.13), and page-cache code that grew across the folio conversions. No configuration setting recovers it; the profile and the diagnostic builds are in engine-execution.md.

The warm-up

patch 0008

cold browser CPU 4% on jj, 5% on python, 3% on opencode; warm unchanged

A block now interprets 128 times before it joins a batch instead of 32. A cold start spends a third of its vCPU time making code and under six percent interpreting it, so compiling a quarter of the blocks it translates was the wrong side of that trade.

The clock, again

nix/native-qemu.nix 路 a guest clock check in boot-test

two releases ran guest time 3.3x slow; their guest-reported columns are wrong, the host-measured ones are not

The snapshots of the two releases before this one were taken on a native QEMU built before patch 0003, so the guest calibrated its clock against a real cycle counter again. The snapshotter is now a derivation from the same patches as the engine, and CI times a sleep in the guest from the host. A slow clock also made opencode read 15% faster on those two records, which is not the engine: the same engine measures 188 s here with the clock right.

The lookup

patch 0009

call 1.34x, indirect 1.2x; python 3% and opencode 3 to 4% of browser CPU; jj unchanged

A return or indirect jump asked a C helper for its target on every transition. Generated code now probes a per-vCPU cache keyed on the guest state first, and hits 92 to 94% of the time.

The first command

nix/guest/init 路 a background read after the mount

cold jj --version 3.71 s to 2.92 s at this runner's one second of think time; 27% less browser CPU at three seconds

A cold exec is mostly the binary being faulted in over 9p, so init reads what /share/bin links to in the background right after the mount. The first release also read the libraries, which helped small binaries and cost python; the next reads the binaries alone, which is why hello, ripgrep and jj give a little back between those two records.

Appendix