Repository navigation
test(benchmarks): add CPython comparison runtime - #178
Merged
Merged
Conversation
Adds a `cpython` script field and a subprocess-driven comparison row so the kernel suite can be measured against the interpreter it is actually judged against. CPython is not a Go library, so it runs as one `python3.13` process per benchmark; the driver checks the result before timing, then times `b.N` calls inside its own `time.perf_counter()` window and prints the elapsed nanoseconds, keeping CPython's ~13 ms process startup out of the reported ns/op. `PYTHONHASHSEED=0` removes hash-randomization noise. A host without python3.13 skips the row instead of failing. Also makes the tengo, gopher_lua, and goja rows conditional on a non-empty script, matching gpython and yaegi, so a kernel can declare only the comparisons that are meaningful for it. Everything stays behind the `compare` build tag; `go test ./...` remains hermetic and needs no Python. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
The runtime reads array.fill's operands as array, offset, value, size with size on top of the stack (see arrayFill in internal/cmd/geninterp/lower.go), and it fills every typed array element kind. instr.Type declared the popped kinds in the other order, giving size KindAny and value KindI32, so the verifier demanded an i32 fill value regardless of the array's element type. An i32 array hid the defect because both operands are i32 there. Filling an f64 or i64 array failed verification with array.fill: operand type mismatch, even though the threaded handler supports bool, i8, i32, i64, f32, and f64. Found by porting minipy's spectralnorm benchmark, whose `[1.0] * n` is an f64 fill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
benchmarks/ measured only i32 until now: no f64, i64, string, or map coverage existed anywhere in the module. These are the first f64 kernels, translated instruction-for-instruction from minipy's benchmark corpus so the measurement isolates minivm's execution cost from minipy's code generation. NBody 5 bodies, 100 advance() steps pairwise f64, f64.sqrt SpectralNorm n=24, 2 power iterations f64 division, nested loops Mandelbrot 16x16 grid, max_iter 50 tight f64 loop, early exit MatMul n=16 f64 multiply-accumulate Sizes are reduced from minipy's ~1-3 s CPython target to land each at 300-900 us/op, and are part of the benchmark contract. Each kernel projects its final value to an i32 checksum by truncation, which Python's int(), Go's int32(), and f64.to_i32_s all agree on exactly, so one number gates correctness across all three implementations. Comparison rows are native, cpython, and gpython; the JS, Lua, and Tengo rows stay empty because these are Python-origin workloads. Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm threaded against CPython 3.13.12, -benchtime=200ms: SpectralNorm 510 us vs 921 us 0.55x Mandelbrot 346 us vs 464 us 0.75x MatMul 370 us vs 457 us 0.81x NBody 879 us vs 545 us 1.61x f64 arithmetic itself beats CPython. NBody is the one inversion, and it is the only kernel that calls a function taking seven array refs from inside a loop and indexes those locals throughout. That shape is the next thing to profile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Both are recursion-heavy in a way the existing Call kernels are not:
RecursiveFib and IndirectRecursiveFib recurse over scalars only, so nothing
in the module measured recursion that carries container state through its
frames.
NQueens n=7 backtracking over three i1 arrays passed as ref params
on every recursive call
Fannkuch n=6 Heap's-algorithm permutation search; per-call array copy
through array.slice, in-place swaps, three i32 returns
fannkuch's permute returns a 3-tuple, which minivm expresses directly as a
three-entry FunctionType.Returns rather than by threading values through
locals, so the port stays faithful. The gpython script uses perm[:] instead
of perm.copy(), which Python 3.4 does not have; the bytecode and the Go
reference are unaffected.
RecursiveFib gains a cpython row reusing its existing script, since fib is
minipy's fib.py already ported faithfully.
Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm
threaded against CPython 3.13.12, -benchtime=200ms:
NQueens(7) 597 us vs 402 us 1.48x, 120 B/op, 6 allocs/op
Fannkuch(6) 1181 us vs 765 us 1.54x, 34,608 B/op, 1442 allocs/op
Both land on the slow side, unlike the pure-scalar float kernels. Together
with NBody they mark the same shape: indexing containers held in locals, and
allocating per call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Completes the corpus port. These three cover the allocation-shaped work the module measured only narrowly before, and StringBuild is the first string coverage anywhere in benchmarks/. BinaryTrees min 4, max 6 struct allocation, recursive build and check SortStress n=128 x2 i64 LCG state, in-place i32 array sort StringBuild 512 tokens string.new_utf32, concat, len, encode_utf32 Two translations deviate from the source, both because minivm has no equivalent opcode, and both noted on the kernel: sortstress calls xs.sort(). The sort is written out in bytecode, so the kernel measures VM dispatch over an insertion sort rather than a builtin. Every compared implementation runs that same insertion sort, so the comparison stays VM-against-VM; timing bytecode insertion sort against CPython's C Timsort would measure algorithm choice, not either interpreter. n is 128 rather than 512 to keep the quadratic sort inside the 300-900 us/op band the other ported kernels share. strbuild ends with upper, lower, replace, strip, count, find, startswith and endswith. minivm has none of them, so the port keeps token generation, the per-character checksum, and the concatenation accumulator, and drops the method chain. The expected checksum accounts for the omission. Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm threaded against CPython 3.13.12, -benchtime=200ms: SortStress 625 us vs 687 us 0.91x, 5,136 B/op, 512 allocs/op BinaryTrees 2611 us vs 2062 us 1.27x, 556,192 B/op, 8,888 allocs/op StringBuild 1592 us vs 691 us 2.30x, 1,724,160 B/op, 5,118 allocs/op StringBuild is the widest gap in the whole ported corpus, matching strbuild being minipy's own worst case. String concatenation is the next thing to profile after the container-indexing shapes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Profiling BenchmarkNumeric_NBody/threaded (3.5 s of samples, linux/amd64, Intel Xeon 2.80GHz) put ~77% of runtime in four handlers plus dispatch: LOCAL_GET of a ref local 19.7% cum, 330 ms of it retainBox ARRAY_GET 16.5% cum LOCAL_GET of a scalar 11.1% ARRAY_SET 7.1% dispatch 23.1% flat `x[i]` where x is an array in a local compiled to three dispatched handlers: LOCAL_GET pushed the container ref and retained it, ARRAY_GET popped and released it, and ARRAY_GET then ran a type switch over every typed-array element kind on every access. The catalog already fused this shape, but only for a compile-time constant container, and every array in every benchmark kernel lives in a local. The container may now also be a LOCAL_GET whose declared type is a concrete typed array. instr.Kind collapses every ref local to KindRef, so the proof needs the declared types.Type: Function.Declared exposes the params-then- locals Type slice that Slots already reduced to Kinds, and the threader carries it alongside. The container ref is borrowed, per docs/fusion.md's Ref Ownership rule, because array.get fully consumes it inside the sequence. Runtime guards are unchanged: the heap slot's concrete array type, the container's ref kind, bounds, segmentation, and stack room are all still checked on every execution, in the order the unfused sequence checks them. Scoped to array.get with a local container; array.set, global.get, and upval.get containers stay unfused. Median of five, -benchtime=300ms, same host, threaded mode: NBody 830,588 -> 589,620 -29.0% Sieve 26,253 -> 22,836 -13.0% SpectralNorm 501,189 -> 454,764 -9.3% Fannkuch 953,232 -> 921,830 -3.3% SortStress 638,744 -> 618,086 -3.2% TypedArraySum 6,699 -> 6,506 -2.9% BinaryTrees 2,682,567 -> 2,611,588 -2.6% Mandelbrot 285,670 -> 280,976 -1.6% MatMul 357,257 -> 351,398 -1.6% NQueens 537,122 -> 538,394 +0.2% (noise) B/op and allocs/op are identical before and after for every kernel. Against CPython 3.13 on this host NBody moves from 1.61x slower to about 1.08x, and Sieve from 1.08x slower to faster than CPython. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
The four preceding commits changed the kernel inventory, added a comparison runtime, and moved measured numbers without updating the documents that own them. docs/coding-patterns.md §15.2 makes docs/benchmarks.md the owner of benchmark results and methodology. docs/benchmarks.md gains a second measurement environment for the linux/amd64 host and a CPython Comparison section carrying the ported kernels' median of five. The darwin/arm64 tables are left untouched: that host has a live ARM64 JIT and amd64 has none, so the two are not comparable and the arm64 rows stay the authority for native and adaptive behavior. The summary also records the array.get fusion result and the array.fill operand-kind defect this port surfaced. benchmarks/README.md documents the ten kernels against the minipy program each came from, their fixture sizes and why they were chosen, the i32 checksum projection, CPython's subprocess timing boundary, and the two translation deviations as contract: sortstress writes its sort out in bytecode with every compared implementation running that same insertion sort, and strbuild drops the string method chain minivm has no opcodes for. PermutationFlips and StructTreeWalk were also missing from the kernel table and are added. The Makefile gains a third benchmark-compare checklist requiring native and cpython rows for the ported scenarios, and fixes a pre-existing defect in the two existing checklists: they asked for BenchmarkCall_RecursiveFib/<runtime>, but that benchmark's rows carry a size level, so the real row is BenchmarkCall_RecursiveFib/20/<runtime> and the check could never pass. All three lists were verified against a real -benchtime=1x comparison run. Measured on linux/amd64 (Intel Xeon 2.80GHz, 4 cores, Go 1.26.2), minivm threaded against CPython 3.13.12, median of five at -benchtime=300ms: SpectralNorm(24) 463 us vs 819 us 0.57x Mandelbrot(16x16) 271 us vs 414 us 0.65x MatMul(16) 354 us vs 464 us 0.76x SortStress(128) 627 us vs 714 us 0.88x NBody(100) 589 us vs 573 us 1.03x Fannkuch(6) 935 us vs 761 us 1.23x NQueens(7) 550 us vs 408 us 1.35x BinaryTrees(4,6) 2,628 us vs 1,942 us 1.35x StringBuild(512) 1,704 us vs 684 us 2.49x Scalar and float work beats CPython; allocation and string work does not. Follow-ups are filed as #172 through #177. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Extends the container fusion 6e5260a added for array.get. Profiling BenchmarkMemory_BinaryTrees/threaded (3.2 s of samples, linux/amd64), the kernel that sits at 1.35x CPython and is tracked as #175: dispatch 17.4% flat constant-index + STRUCT_GET fused 9.6% cum LOCAL_GET of a ref local 9.9% cum STRUCT_SET 7.3% cum Even with the field index folded in as a constant, every field read popped a container ref that LOCAL_GET had just retained, then ran two runtime switches: one over the heap value's type, another over the field's kind. A local declared as a concrete *types.StructType resolves both at threading time, so the handler now boxes the field directly and borrows the container. Also fixes a parity defect this work exposed in 6e5260a itself. The fused array.get path asserted only types.TypedArray[T], but the unfused handler also accepts *types.Array; a generic boxed array in a slot declared as a concrete typed array therefore ran before and trapped after. Confirmed by A/B against 6e5260a^: verify ok and result 0 before, type mismatch after. struct.get had the same gap against *HostObject. Both fused paths now fall back to the unfused arms instead of trapping, sharing one copy of each arm's bounds, boxing, and ownership rules rather than restating them. The fallback first cost NBody 12.9% by introducing a join point: a result variable live across both arms, consumed after the merge, where the fast path had been straight-line. Giving each arm its own push-and-return removes the merge and recovers it. Median of five, -benchtime=300ms, threaded, against HEAD: NBody 659,821 -> 641,249 -2.8% SpectralNorm 541,985 -> 540,738 -0.2% Sieve 27,112 -> 26,957 -0.6% B/op and allocs/op are unchanged everywhere. BinaryTrees and StructTreeWalk do not move, because both declare their node parameter as types.TypeRef and so never reach the fused path -- the same authoring issue #176 records for the array kernels, and the reason #174 exists. BenchmarkInterpreter_StructGetLocalFusion covers the shape directly and improves 21.6%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
…tainers Completes the container fusion started in 6e5260a and extended to structs in 25e75ee. The container may now be a GLOBAL_GET or UPVAL_GET, not only a LOCAL_GET, whenever its declared type is a concrete array or struct type. The plumbing was already symmetric: Interpreter.globalTypes and Function.Captures are both []types.Type, the same shape Function.Declared supplies for locals. globalTypes is fixed per interpreter so it is set on the threader once, like globals itself; captureTypes varies per function so it joins Compile's signature, like captures. This matters beyond symmetry. The fusion proves the element or field type from the declared slot type, so a front end that keeps module state in globals reaches it only if this path exists. minipy, the consumer whose benchmark corpus prompted this work, is exactly that shape (#174). Every guard from the local path carries over unchanged: the container's ref kind, the heap value's concrete type, bounds, and the struct field's runtime kind, all checked on every execution in the unfused order. The parity fallback carries over too, so a *types.Array in a slot declared as a typed array still returns its element rather than trapping, whichever container holds it. So does the early-return arm shape, so the hot path stays straight-line. No canonical kernel holds a container in a global or an upvalue, so none of them move; their B/op and allocs/op are byte-for-byte unchanged. BenchmarkInterpreter_ArrayGetContainerFusion covers the two new sources directly. Median of five, -benchtime=300ms, against HEAD: global 11,039,832 -> 7,956,250 -27.9% upvalue 11,429,440 -> 7,646,192 -33.1% Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
A review of the container fusion found the parity fallback was still incomplete. It re-tried the declared TypedArray[T] and the generic *types.Array, but the unfused ARRAY_GET switch accepts all six TypedArray variants, and LOCAL_SET, GLOBAL_SET, and closure capture do not re-check a slot's declared type against the value stored. So a container proven at threading time to declare i32 elements can hold a TypedArray[float32] at runtime, and fused execution trapped where unfused returned the element. Confirmed against a worktree at f0bd50a: a TypedArray[float32], [float64], [int64], or [bool] in a local declared types.TypeI32Array passes program.Verify, traps `type mismatch` there, and returns its element here. Regression cases for all three container sources are in the runTests table, which exercises standalone and fused modes from one entry. struct.get does not share the defect. One Go type, *types.Struct, covers every field layout, and the fused path re-derives the box operation from the runtime value.Typ.Fields[at].Kind rather than trusting the declared type, so it is exhaustive by construction. Making the fallback exhaustive inline cost NBody 8.5% median and 5.0% min, because a fifty-line switch landed in every fused handler closure and interp/threaded.go grew to 74,471 lines; the fast path pays for that even though it returns first. The generic read now lives in two runtime methods, Interpreter.arrayElem and Interpreter.structField, that the fused fallback and the unfused handler both call. Neither releases the container: that difference belongs at the call site, since the unfused path pops its ref and the fused path borrows it. interp/threaded.go is 69,599 lines, 1,992 fewer than before this change, because the unfused struct.get switch collapsed into the same method. Median and min of seven, -benchtime=300ms, threaded, against f0bd50a: NBody 661,966 / 641,963 -> 648,744 / 628,587 -2.0% / -2.1% Sieve 26,875 / 26,189 -> 26,405 / 25,965 -1.7% / -0.9% SpectralNorm 546,502 / 519,098 -> 542,373 / 530,759 -0.8% / +2.2% B/op and allocs/op are unchanged on every kernel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (22)
📝 WalkthroughWalkthroughThe change expands benchmark coverage with native and CPython comparisons. It also adds declared-type tracking for locals, globals, and captures, enabling typed array and struct fusion with runtime validation and generic fallbacks. ChangesBenchmark suite expansion
Declared-type container fusion
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant Function
participant Threader
participant FusedRead
participant Interpreter
Function->>Threader: provide declared slot and capture types
Threader->>FusedRead: specialize array.get or struct.get
FusedRead->>Interpreter: use generic reader on runtime mismatch
Interpreter-->>FusedRead: return validated element or field
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
cpythonscript field and a subprocess-driven comparison row so thekernel suite can be measured against the interpreter it is actually judged
against. CPython is not a Go library, so it runs as one
python3.13processper benchmark; the driver checks the result before timing, then times
b.Ncalls inside its own
time.perf_counter()window and prints the elapsednanoseconds, keeping CPython's ~13 ms process startup out of the reported
ns/op.
PYTHONHASHSEED=0removes hash-randomization noise. A host withoutpython3.13 skips the row instead of failing.
Also makes the tengo, gopher_lua, and goja rows conditional on a non-empty
script, matching gpython and yaegi, so a kernel can declare only the
comparisons that are meaningful for it.
Everything stays behind the
comparebuild tag;go test ./...remainshermetic and needs no Python.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_015uC534xmef9n7FaQMshuXZ
Summary by CodeRabbit
New Features
Bug Fixes
Documentation