Skip to content

test(benchmarks): add CPython comparison runtime - #178

Merged
siyul-park merged 10 commits into
mainfrom
claude/minipy-benchmark-minivm-port-68o2nj
Aug 8, 2026
Merged

siyul-park merged 10 commits into
mainfrom
claude/minipy-benchmark-minivm-port-68o2nj

Conversation

@siyul-park

@siyul-park siyul-park commented Aug 8, 2026 •

Copy link
Copy Markdown
Owner

Adds a cpython script field and a subprocess-driven comparison row so the
kernel suite can be measured against the interpreter it is actually judged
against. CPython is not a Go library, so it runs as one python3.13 process
per benchmark; the driver checks the result before timing, then times b.N
calls inside its own time.perf_counter() window and prints the elapsed
nanoseconds, keeping CPython's ~13 ms process startup out of the reported
ns/op. PYTHONHASHSEED=0 removes hash-randomization noise. A host without
python3.13 skips the row instead of failing.

Also makes the tengo, gopher_lua, and goja rows conditional on a non-empty
script, matching gpython and yaegi, so a kernel can declare only the
comparisons that are meaningful for it.

Everything stays behind the compare build tag; go test ./... remains
hermetic and needs no Python.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ

Summary by CodeRabbit

  • New Features

    • Expanded benchmark coverage with numeric, memory, N-Queens, and Fannkuch workloads.
    • Added optional CPython comparisons with result validation and consistent timing.
    • Improved performance for typed array and struct reads from local, global, and captured values.
  • Bug Fixes

    • Corrected array-fill stack behavior and strengthened runtime type and bounds validation.
  • Documentation

    • Updated benchmark, fusion, instruction-set, and testing documentation with expanded guidance and results.

claude added 10 commits August 7, 2026 10:44
Adds a `cpython` script field and a subprocess-driven comparison row so the
kernel suite can be measured against the interpreter it is actually judged
against. CPython is not a Go library, so it runs as one `python3.13` process
per benchmark; the driver checks the result before timing, then times `b.N`
calls inside its own `time.perf_counter()` window and prints the elapsed
nanoseconds, keeping CPython's ~13 ms process startup out of the reported
ns/op. `PYTHONHASHSEED=0` removes hash-randomization noise. A host without
python3.13 skips the row instead of failing.

Also makes the tengo, gopher_lua, and goja rows conditional on a non-empty
script, matching gpython and yaegi, so a kernel can declare only the
comparisons that are meaningful for it.

Everything stays behind the `compare` build tag; `go test ./...` remains
hermetic and needs no Python.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
The runtime reads array.fill's operands as array, offset, value, size with
size on top of the stack (see arrayFill in internal/cmd/geninterp/lower.go),
and it fills every typed array element kind. instr.Type declared the popped
kinds in the other order, giving size KindAny and value KindI32, so the
verifier demanded an i32 fill value regardless of the array's element type.

An i32 array hid the defect because both operands are i32 there. Filling an
f64 or i64 array failed verification with array.fill: operand type mismatch,
even though the threaded handler supports bool, i8, i32, i64, f32, and f64.

Found by porting minipy's spectralnorm benchmark, whose `[1.0] * n` is an
f64 fill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
benchmarks/ measured only i32 until now: no f64, i64, string, or map
coverage existed anywhere in the module. These are the first f64 kernels,
translated instruction-for-instruction from minipy's benchmark corpus so the
measurement isolates minivm's execution cost from minipy's code generation.

  NBody         5 bodies, 100 advance() steps   pairwise f64, f64.sqrt
  SpectralNorm  n=24, 2 power iterations        f64 division, nested loops
  Mandelbrot    16x16 grid, max_iter 50         tight f64 loop, early exit
  MatMul        n=16                            f64 multiply-accumulate

Sizes are reduced from minipy's ~1-3 s CPython target to land each at
300-900 us/op, and are part of the benchmark contract. Each kernel projects
its final value to an i32 checksum by truncation, which Python's int(), Go's
int32(), and f64.to_i32_s all agree on exactly, so one number gates
correctness across all three implementations. Comparison rows are native,
cpython, and gpython; the JS, Lua, and Tengo rows stay empty because these
are Python-origin workloads.

Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm
threaded against CPython 3.13.12, -benchtime=200ms:

  SpectralNorm   510 us vs   921 us   0.55x
  Mandelbrot     346 us vs   464 us   0.75x
  MatMul         370 us vs   457 us   0.81x
  NBody          879 us vs   545 us   1.61x

f64 arithmetic itself beats CPython. NBody is the one inversion, and it is
the only kernel that calls a function taking seven array refs from inside a
loop and indexes those locals throughout. That shape is the next thing to
profile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Both are recursion-heavy in a way the existing Call kernels are not:
RecursiveFib and IndirectRecursiveFib recurse over scalars only, so nothing
in the module measured recursion that carries container state through its
frames.

  NQueens   n=7    backtracking over three i1 arrays passed as ref params
                   on every recursive call
  Fannkuch  n=6    Heap's-algorithm permutation search; per-call array copy
                   through array.slice, in-place swaps, three i32 returns

fannkuch's permute returns a 3-tuple, which minivm expresses directly as a
three-entry FunctionType.Returns rather than by threading values through
locals, so the port stays faithful. The gpython script uses perm[:] instead
of perm.copy(), which Python 3.4 does not have; the bytecode and the Go
reference are unaffected.

RecursiveFib gains a cpython row reusing its existing script, since fib is
minipy's fib.py already ported faithfully.

Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm
threaded against CPython 3.13.12, -benchtime=200ms:

  NQueens(7)    597 us vs 402 us   1.48x,    120 B/op,    6 allocs/op
  Fannkuch(6)  1181 us vs 765 us   1.54x, 34,608 B/op, 1442 allocs/op

Both land on the slow side, unlike the pure-scalar float kernels. Together
with NBody they mark the same shape: indexing containers held in locals, and
allocating per call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Completes the corpus port. These three cover the allocation-shaped work the
module measured only narrowly before, and StringBuild is the first string
coverage anywhere in benchmarks/.

  BinaryTrees  min 4, max 6   struct allocation, recursive build and check
  SortStress   n=128 x2       i64 LCG state, in-place i32 array sort
  StringBuild  512 tokens     string.new_utf32, concat, len, encode_utf32

Two translations deviate from the source, both because minivm has no
equivalent opcode, and both noted on the kernel:

  sortstress calls xs.sort(). The sort is written out in bytecode, so the
  kernel measures VM dispatch over an insertion sort rather than a builtin.
  Every compared implementation runs that same insertion sort, so the
  comparison stays VM-against-VM; timing bytecode insertion sort against
  CPython's C Timsort would measure algorithm choice, not either
  interpreter. n is 128 rather than 512 to keep the quadratic sort inside
  the 300-900 us/op band the other ported kernels share.

  strbuild ends with upper, lower, replace, strip, count, find, startswith
  and endswith. minivm has none of them, so the port keeps token generation,
  the per-character checksum, and the concatenation accumulator, and drops
  the method chain. The expected checksum accounts for the omission.

Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm
threaded against CPython 3.13.12, -benchtime=200ms:

  SortStress    625 us vs   687 us  0.91x,     5,136 B/op,   512 allocs/op
  BinaryTrees  2611 us vs  2062 us  1.27x,   556,192 B/op, 8,888 allocs/op
  StringBuild  1592 us vs   691 us  2.30x, 1,724,160 B/op, 5,118 allocs/op

StringBuild is the widest gap in the whole ported corpus, matching strbuild
being minipy's own worst case. String concatenation is the next thing to
profile after the container-indexing shapes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Profiling BenchmarkNumeric_NBody/threaded (3.5 s of samples, linux/amd64,
Intel Xeon 2.80GHz) put ~77% of runtime in four handlers plus dispatch:

  LOCAL_GET of a ref local   19.7% cum, 330 ms of it retainBox
  ARRAY_GET                  16.5% cum
  LOCAL_GET of a scalar      11.1%
  ARRAY_SET                   7.1%
  dispatch                   23.1% flat

`x[i]` where x is an array in a local compiled to three dispatched handlers:
LOCAL_GET pushed the container ref and retained it, ARRAY_GET popped and
released it, and ARRAY_GET then ran a type switch over every typed-array
element kind on every access. The catalog already fused this shape, but only
for a compile-time constant container, and every array in every benchmark
kernel lives in a local.

The container may now also be a LOCAL_GET whose declared type is a concrete
typed array. instr.Kind collapses every ref local to KindRef, so the proof
needs the declared types.Type: Function.Declared exposes the params-then-
locals Type slice that Slots already reduced to Kinds, and the threader
carries it alongside. The container ref is borrowed, per docs/fusion.md's Ref
Ownership rule, because array.get fully consumes it inside the sequence.

Runtime guards are unchanged: the heap slot's concrete array type, the
container's ref kind, bounds, segmentation, and stack room are all still
checked on every execution, in the order the unfused sequence checks them.
Scoped to array.get with a local container; array.set, global.get, and
upval.get containers stay unfused.

Median of five, -benchtime=300ms, same host, threaded mode:

  NBody          830,588 -> 589,620   -29.0%
  Sieve           26,253 ->  22,836   -13.0%
  SpectralNorm   501,189 -> 454,764    -9.3%
  Fannkuch       953,232 -> 921,830    -3.3%
  SortStress     638,744 -> 618,086    -3.2%
  TypedArraySum    6,699 ->   6,506    -2.9%
  BinaryTrees  2,682,567 -> 2,611,588  -2.6%
  Mandelbrot     285,670 -> 280,976    -1.6%
  MatMul         357,257 -> 351,398    -1.6%
  NQueens        537,122 -> 538,394    +0.2%  (noise)

B/op and allocs/op are identical before and after for every kernel. Against
CPython 3.13 on this host NBody moves from 1.61x slower to about 1.08x, and
Sieve from 1.08x slower to faster than CPython.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
The four preceding commits changed the kernel inventory, added a comparison
runtime, and moved measured numbers without updating the documents that own
them. docs/coding-patterns.md §15.2 makes docs/benchmarks.md the owner of
benchmark results and methodology.

docs/benchmarks.md gains a second measurement environment for the linux/amd64
host and a CPython Comparison section carrying the ported kernels' median of
five. The darwin/arm64 tables are left untouched: that host has a live ARM64
JIT and amd64 has none, so the two are not comparable and the arm64 rows stay
the authority for native and adaptive behavior. The summary also records the
array.get fusion result and the array.fill operand-kind defect this port
surfaced.

benchmarks/README.md documents the ten kernels against the minipy program each
came from, their fixture sizes and why they were chosen, the i32 checksum
projection, CPython's subprocess timing boundary, and the two translation
deviations as contract: sortstress writes its sort out in bytecode with every
compared implementation running that same insertion sort, and strbuild drops
the string method chain minivm has no opcodes for. PermutationFlips and
StructTreeWalk were also missing from the kernel table and are added.

The Makefile gains a third benchmark-compare checklist requiring native and
cpython rows for the ported scenarios, and fixes a pre-existing defect in the
two existing checklists: they asked for BenchmarkCall_RecursiveFib/<runtime>,
but that benchmark's rows carry a size level, so the real row is
BenchmarkCall_RecursiveFib/20/<runtime> and the check could never pass. All
three lists were verified against a real -benchtime=1x comparison run.

Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm
threaded against CPython 3.13.12, median of five at -benchtime=300ms:

  SpectralNorm(24)      463 us vs   819 us  0.57x
  Mandelbrot(16x16)     271 us vs   414 us  0.65x
  MatMul(16)            354 us vs   464 us  0.76x
  SortStress(128)       627 us vs   714 us  0.88x
  NBody(100)            589 us vs   573 us  1.03x
  Fannkuch(6)           935 us vs   761 us  1.23x
  NQueens(7)            550 us vs   408 us  1.35x
  BinaryTrees(4,6)    2,628 us vs 1,942 us  1.35x
  StringBuild(512)    1,704 us vs   684 us  2.49x

Scalar and float work beats CPython; allocation and string work does not.
Follow-ups are filed as #172 through #177.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Extends the container fusion 6e5260a added for array.get. Profiling
BenchmarkMemory_BinaryTrees/threaded (3.2 s of samples, linux/amd64), the
kernel that sits at 1.35x CPython and is tracked as #175:

  dispatch                              17.4% flat
  constant-index + STRUCT_GET fused      9.6% cum
  LOCAL_GET of a ref local               9.9% cum
  STRUCT_SET                             7.3% cum

Even with the field index folded in as a constant, every field read popped a
container ref that LOCAL_GET had just retained, then ran two runtime
switches: one over the heap value's type, another over the field's kind. A
local declared as a concrete *types.StructType resolves both at threading
time, so the handler now boxes the field directly and borrows the container.

Also fixes a parity defect this work exposed in 6e5260a itself. The fused
array.get path asserted only types.TypedArray[T], but the unfused handler
also accepts *types.Array; a generic boxed array in a slot declared as a
concrete typed array therefore ran before and trapped after. Confirmed by
A/B against 6e5260a^: verify ok and result 0 before, type mismatch after.
struct.get had the same gap against *HostObject. Both fused paths now fall
back to the unfused arms instead of trapping, sharing one copy of each arm's
bounds, boxing, and ownership rules rather than restating them.

The fallback first cost NBody 12.9% by introducing a join point: a result
variable live across both arms, consumed after the merge, where the fast path
had been straight-line. Giving each arm its own push-and-return removes the
merge and recovers it.

Median of five, -benchtime=300ms, threaded, against HEAD:

  NBody           659,821 ->   641,249   -2.8%
  SpectralNorm    541,985 ->   540,738   -0.2%
  Sieve            27,112 ->    26,957   -0.6%

B/op and allocs/op are unchanged everywhere.

BinaryTrees and StructTreeWalk do not move, because both declare their node
parameter as types.TypeRef and so never reach the fused path -- the same
authoring issue #176 records for the array kernels, and the reason #174
exists. BenchmarkInterpreter_StructGetLocalFusion covers the shape directly
and improves 21.6%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
…tainers

Completes the container fusion started in 6e5260a and extended to structs in
25e75ee. The container may now be a GLOBAL_GET or UPVAL_GET, not only a
LOCAL_GET, whenever its declared type is a concrete array or struct type.

The plumbing was already symmetric: Interpreter.globalTypes and
Function.Captures are both []types.Type, the same shape Function.Declared
supplies for locals. globalTypes is fixed per interpreter so it is set on the
threader once, like globals itself; captureTypes varies per function so it
joins Compile's signature, like captures.

This matters beyond symmetry. The fusion proves the element or field type
from the declared slot type, so a front end that keeps module state in
globals reaches it only if this path exists. minipy, the consumer whose
benchmark corpus prompted this work, is exactly that shape (#174).

Every guard from the local path carries over unchanged: the container's ref
kind, the heap value's concrete type, bounds, and the struct field's runtime
kind, all checked on every execution in the unfused order. The parity
fallback carries over too, so a *types.Array in a slot declared as a typed
array still returns its element rather than trapping, whichever container
holds it. So does the early-return arm shape, so the hot path stays
straight-line.

No canonical kernel holds a container in a global or an upvalue, so none of
them move; their B/op and allocs/op are byte-for-byte unchanged.
BenchmarkInterpreter_ArrayGetContainerFusion covers the two new sources
directly. Median of five, -benchtime=300ms, against HEAD:

  global    11,039,832 -> 7,956,250   -27.9%
  upvalue   11,429,440 -> 7,646,192   -33.1%

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
A review of the container fusion found the parity fallback was still
incomplete. It re-tried the declared TypedArray[T] and the generic
*types.Array, but the unfused ARRAY_GET switch accepts all six TypedArray
variants, and LOCAL_SET, GLOBAL_SET, and closure capture do not re-check a
slot's declared type against the value stored. So a container proven at
threading time to declare i32 elements can hold a TypedArray[float32] at
runtime, and fused execution trapped where unfused returned the element.

Confirmed against a worktree at f0bd50a: a TypedArray[float32], [float64],
[int64], or [bool] in a local declared types.TypeI32Array passes
program.Verify, traps `type mismatch` there, and returns its element here.
Regression cases for all three container sources are in the runTests table,
which exercises standalone and fused modes from one entry.

struct.get does not share the defect. One Go type, *types.Struct, covers
every field layout, and the fused path re-derives the box operation from the
runtime value.Typ.Fields[at].Kind rather than trusting the declared type, so
it is exhaustive by construction.

Making the fallback exhaustive inline cost NBody 8.5% median and 5.0% min,
because a fifty-line switch landed in every fused handler closure and
interp/threaded.go grew to 74,471 lines; the fast path pays for that even
though it returns first. The generic read now lives in two runtime methods,
Interpreter.arrayElem and Interpreter.structField, that the fused fallback
and the unfused handler both call. Neither releases the container: that
difference belongs at the call site, since the unfused path pops its ref and
the fused path borrows it.

interp/threaded.go is 69,599 lines, 1,992 fewer than before this change,
because the unfused struct.get switch collapsed into the same method.

Median and min of seven, -benchtime=300ms, threaded, against f0bd50a:

  NBody         661,966 / 641,963 -> 648,744 / 628,587   -2.0% / -2.1%
  Sieve          26,875 /  26,189 ->  26,405 /  25,965   -1.7% / -0.9%
  SpectralNorm  546,502 / 519,098 -> 542,373 / 530,759   -0.8% / +2.2%

B/op and allocs/op are unchanged on every kernel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
@siyul-park
siyul-park merged commit d56f31b into main Aug 8, 2026
3 of 5 checks passed
@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f29a9311-c88c-41f2-aa9e-0a1406e72a35

📥 Commits

Reviewing files that changed from the base of the PR and between 0820060 and 4b6edd1.

📒 Files selected for processing (22)
  • Makefile
  • benchmarks/README.md
  • benchmarks/benchmark_test.go
  • benchmarks/call_test.go
  • benchmarks/compare_test.go
  • benchmarks/memory_test.go
  • benchmarks/numeric_test.go
  • docs/benchmarks.md
  • docs/fusion.md
  • docs/instruction-set.md
  • docs/testing.md
  • instr/type.go
  • internal/cmd/geninterp/generate.go
  • internal/cmd/geninterp/lower.go
  • internal/cmd/geninterp/pattern.go
  • internal/cmd/geninterp/validate.go
  • interp/interp.go
  • interp/interp_test.go
  • interp/threaded.go
  • interp/trace.go
  • types/function.go
  • types/function_test.go

📝 Walkthrough

Walkthrough

The change expands benchmark coverage with native and CPython comparisons. It also adds declared-type tracking for locals, globals, and captures, enabling typed array and struct fusion with runtime validation and generic fallbacks.

Changes

Benchmark suite expansion

Layer / File(s) Summary
Benchmark kernels and reference implementations
benchmarks/README.md, benchmarks/call_test.go, benchmarks/memory_test.go, benchmarks/numeric_test.go
Adds recursive call, numeric, memory, and string kernels with bytecode builders, native references, and Python comparison scripts.
External runtime comparison
benchmarks/benchmark_test.go, benchmarks/compare_test.go, benchmarks/README.md, docs/benchmarks.md
Adds conditional runtime registration and CPython subprocess execution with result checks, deterministic hashing, timing, and skip behavior.
Benchmark scenario validation
Makefile
Updates recursive Fibonacci validation and adds native and CPython scenario checks.

Declared-type container fusion

Layer / File(s) Summary
Declared type metadata and compilation wiring
types/function.go, types/function_test.go, internal/cmd/geninterp/generate.go, interp/interp.go, interp/trace.go, instr/type.go, docs/testing.md
Adds declared slot types and passes local, global, and capture metadata through compilation paths.
Typed container pattern lowering
internal/cmd/geninterp/pattern.go, internal/cmd/geninterp/lower.go, internal/cmd/geninterp/validate.go
Adds typed array and struct patterns from constants, locals, globals, and upvalues.
Runtime aggregate reads and fallback behavior
internal/cmd/geninterp/lower.go, interp/interp.go, interp/interp_test.go
Adds specialized array and struct reads with runtime checks and generic fallback handlers.
Fusion documentation and coverage
docs/fusion.md, docs/instruction-set.md, docs/benchmarks.md
Documents typed-container fusion rules, runtime checks, fallback behavior, and related benchmark coverage.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Sequence Diagram(s)

sequenceDiagram
  participant Function
  participant Threader
  participant FusedRead
  participant Interpreter
  Function->>Threader: provide declared slot and capture types
  Threader->>FusedRead: specialize array.get or struct.get
  FusedRead->>Interpreter: use generic reader on runtime mismatch
  Interpreter-->>FusedRead: return validated element or field
Loading
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/minipy-benchmark-minivm-port-68o2nj

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@siyul-park
siyul-park deleted the claude/minipy-benchmark-minivm-port-68o2nj branch November 7, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants