Repository navigation
test(benchmarks): declare concrete array types on kernel container slots - #229
Merged
Merged
Conversation
NQueens, Fannkuch, SortStress, and StringBuild declared their typed-array locals and function params as `any` (types.TypeRef). Container fusion proves an element kind from the declared slot type at threading time, so an `any` slot never reaches the fused array.get / array.set paths however consistently it holds a typed array. These kernels therefore under-reported what the interpreter does with a well-typed program. The Python sources annotate these explicitly (`cols: list[bool]`, `perm: list[int]`), and NBody, SpectralNorm, MatMul, and Sieve already declare `[]f64` / `[]i32` in exactly these positions, so the concrete type is the more faithful translation rather than a tuned one. No interpreter code changes. Interleaved A/B, -benchtime=300ms -count=6, Apple M4 Pro: Call_NQueens/threaded 304.3µs -> 195.7µs -35.66% (p=0.002) Call_NQueens/default 136.4µs -> 107.3µs -21.36% (p=0.002) Call_NQueens/jit 131.9µs -> 104.3µs -20.95% (p=0.002) Memory_SortStress/threaded 310.0µs -> 188.3µs -39.26% (p=0.002) Memory_SortStress/default 52.71µs -> 50.81µs -3.60% (p=0.002) Call_Fannkuch/threaded 513.2µs -> 400.5µs -21.97% (p=0.002) Call_Fannkuch/default 407.6µs -> 401.7µs -1.44% (p=0.002) Memory_StringBuild/threaded 359.4µs -> 336.6µs -6.34% (p=0.002) Control_Sieve and Numeric_NBody, which already declared concrete types, are flat in all three modes. B/op and allocs/op are identical on every row. In threaded mode minivm now beats CPython on NQueens, Fannkuch, and StringBuild, so those three bold marks move in docs/benchmarks.md. The struct-holding slots in StructTreeWalk and BinaryTrees are deliberately left as `any`: their Python type is `Node | None`, which minivm cannot express as a concrete struct type, and declaring one anyway makes the JIT diverge from the threaded interpreter. Filed separately.
Contributor
|
Warning Review limit reachedNext included review available in 13 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 31, 2026
siyul-park
added a commit
that referenced
this pull request
Aug 31, 2026
…t slots (#231) * test(benchmarks): declare the concrete node type on tree-kernel struct slots StructTreeWalk and BinaryTrees declared every Node-holding local, return, and param as `any` (types.TypeRef), so `struct.get` never reached its fused container path. #229 had to leave them that way: declaring the concrete struct type made the ARM64 JIT fault where threaded execution was correct. #230 fixed that, so the declarations can now be what the Python sources say. `left`/`right` stay `any` because the grammar cannot name a recursive type. The `check`/`item_check` params may hold the null ref at a leaf, which is fine: heap slots do not exclude null, `struct.new_default` leaves unset ref fields null, and both tiers now agree on `ref.is_null` for them. No interpreter code changes. Interleaved A/B, -benchtime=300ms -count=6, Apple M4 Pro: Memory_BinaryTrees/threaded 1.145ms -> 1.059ms -7.55% (p=0.002) Memory_BinaryTrees/default 1.187ms -> 1.104ms -7.01% (p=0.002) Memory_BinaryTrees/jit 1.187ms -> 1.106ms -6.85% (p=0.002) Memory_StructTreeWalk/default 160.5us -> 154.6us -3.69% (p=0.002) Memory_StructTreeWalk/threaded 153.0us -> 147.5us -3.58% (p=0.002) Memory_StructTreeWalk/jit 160.2us -> 155.7us -2.81% (p=0.002) Control_Sieve and Numeric_NBody are flat in all three modes. B/op and allocs/op are identical on every row. * test(benchmarks): drop the redundant ref.cast from the tree walkers `check` and `item_check` each downcast their node param with `ref.cast 0` immediately after a `ref.is_null; br_if` guard. The cast only existed because the param was declared `any`; with the param declared as the node struct type it proves nothing the declaration does not already state, and neither Python source has a cast. It was not free. `REF_CAST` is bridged, so every node walk left native code for the interpreter and came back, and the cast also split `local.get 0` from the `i32.const N; struct.get` behind it, blocking that field read from fusing. Interleaved A/B against the previous commit, -benchtime=300ms -count=6, Apple M4 Pro: Memory_BinaryTrees/jit 1108.1us -> 752.7us -32.08% (p=0.002) Memory_BinaryTrees/default 1102.7us -> 751.8us -31.82% (p=0.002) Memory_StructTreeWalk/jit 156.0us -> 135.0us -13.43% (p=0.002) Memory_StructTreeWalk/default 154.8us -> 134.4us -13.20% (p=0.002) Memory_BinaryTrees/threaded 1055.9us -> 950.9us -9.94% (p=0.002) Memory_StructTreeWalk/threaded 147.7us -> 138.1us -6.52% (p=0.002) Control_Sieve is flat in all three modes. B/op and allocs/op are unchanged. * docs(benchmarks): re-measure the tree-kernel rows StructTreeWalk and BinaryTrees moved when their node slots were declared concretely and the redundant ref.cast was dropped. Both now run fastest under default rather than threaded, so they leave that list, and minivm threaded overtakes CPython on BinaryTrees.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the array half of #176.
Four kernels declared their typed-array locals and function params as
any(types.TypeRef).Container fusion proves an element kind from the declared slot type at threading time, so an
anyslot never reaches the fusedarray.get/array.setpaths however consistently it holds atyped array. Those kernels were under-reporting what the interpreter does with a well-typed program.
No interpreter code changes. This is a corrected fixture, not an optimizer gain — the kernels
simply start exercising a path a well-typed program always reaches.
The concrete type is also the more faithful translation:
conformance/testdata/benchmark/{nqueens,fannkuch}.pyannotate
cols: list[bool]andperm: list[int], and NBody, SpectralNorm, MatMul, and Sieve alreadydeclare
[]f64/[]i32in exactly these positions.Changes
solve's three params ->[]i1count_flips/permuteparams + thearray.slicecopy ->[]i32make_listreturn/local +insertion_sortparam ->[]i32codePointsanddigits'arr->[]i32Measured
Interleaved A/B — six alternating rounds with a settle between,
-benchtime=300ms -count=6,Apple M4 Pro. Plain back-to-back runs are not trustworthy on this machine (#227 saw +-169% variance
and a fabricated +50% regression that way).
Memory_SortStress/threadedCall_NQueens/threadedCall_Fannkuch/threadedCall_NQueens/defaultCall_NQueens/jitMemory_StringBuild/threadedMemory_SortStress/defaultCall_Fannkuch/defaultMemory_SortStress/jit,Memory_StringBuild/{default,jit},Call_Fannkuch/jitGeomean -9.51%.
B/opandallocs/opare byte-identical on every row.Controls:
Control_SieveandNumeric_NBodyalready declare concrete types and are flat in allthree modes (p >= 0.09), which is the evidence that the movement comes from the retyped slots and
not from drift.
In
threadedmode minivm now beats CPython onNQueens(196.49 vs 226.15 µs),Fannkuch(399.53 vs 430.81 µs), and
StringBuild(335.8 vs 368.74 µs), so those three bold marks move.Left out, with reasons
StructTreeWalkandBinaryTreeskeepany.binarytrees.pyannotates
t: "Node | None"; minivm has no union type, soanyis the faithful translation. Anddeclaring the parameter as a concrete struct type anyway makes the JIT fault where threaded
execution is correct: filed as perf(jit): a struct-typed slot that can hold null makes the JIT diverge from threaded #228 with a one-line repro. test(benchmarks): kernels declare array and struct params as TypeRef #176 anticipated this case ("or the
kernel keeps
TypeReffor a stated reason").allocationGraph/permutationFlipsany->[]any—arrayKindrecognizes only the sixTypedArray[T]kinds, so ananyelement type cannot fuse. Pure churn on the measurement.stringBuild'sbig/tokany->string— no string container fusion exists.ref.cast 0in the two tree kernels — moot while those kernels keepany, and it changes the instruction stream rather than a declaration.Test plan
cd benchmarks && go test -run='^$' -bench='^(BenchmarkCall|BenchmarkMemory|BenchmarkControl|BenchmarkNumeric)' -benchtime=1x .— every kernel self-verifies its result; all passgo test -race ./...— 19 packages greenmake lint— cleandocs/benchmarks.mdre-measured under its own protocol (-benchtime=300ms -count=3, median, on the machine its header records)🤖 Generated with Claude Code