Skip to content

test(benchmarks): declare concrete array types on kernel container slots - #229

Merged
siyul-park merged 1 commit into
mainfrom
test/176-concrete-container-types
Aug 31, 2026
Merged

siyul-park merged 1 commit into
mainfrom
test/176-concrete-container-types

Conversation

@siyul-park

Copy link
Copy Markdown
Owner

Closes the array half of #176.

Four kernels declared their typed-array locals and function params as any (types.TypeRef).
Container fusion proves an element kind from the declared slot type at threading time, so an
any slot never reaches the fused array.get / array.set paths however consistently it holds a
typed array. Those kernels were under-reporting what the interpreter does with a well-typed program.

No interpreter code changes. This is a corrected fixture, not an optimizer gain — the kernels
simply start exercising a path a well-typed program always reaches.

The concrete type is also the more faithful translation: conformance/testdata/benchmark/{nqueens,fannkuch}.py
annotate cols: list[bool] and perm: list[int], and NBody, SpectralNorm, MatMul, and Sieve already
declare []f64 / []i32 in exactly these positions.

Changes

kernel slots
nqueens 3 module locals + solve's three params -> []i1
fannkuch module local + count_flips/permute params + the array.slice copy -> []i32
sortStress module local + make_list return/local + insertion_sort param -> []i32
stringBuild codePoints and digits' arr -> []i32

Measured

Interleaved A/B — six alternating rounds with a settle between, -benchtime=300ms -count=6,
Apple M4 Pro. Plain back-to-back runs are not trustworthy on this machine (#227 saw +-169% variance
and a fabricated +50% regression that way).

benchmark main branch delta
Memory_SortStress/threaded 310.0 µs 188.3 µs -39.26% (p=0.002)
Call_NQueens/threaded 304.3 µs 195.7 µs -35.66% (p=0.002)
Call_Fannkuch/threaded 513.2 µs 400.5 µs -21.97% (p=0.002)
Call_NQueens/default 136.4 µs 107.3 µs -21.36% (p=0.002)
Call_NQueens/jit 131.9 µs 104.3 µs -20.95% (p=0.002)
Memory_StringBuild/threaded 359.4 µs 336.6 µs -6.34% (p=0.002)
Memory_SortStress/default 52.71 µs 50.81 µs -3.60% (p=0.002)
Call_Fannkuch/default 407.6 µs 401.7 µs -1.44% (p=0.002)
Memory_SortStress/jit, Memory_StringBuild/{default,jit}, Call_Fannkuch/jit ~

Geomean -9.51%. B/op and allocs/op are byte-identical on every row.

Controls: Control_Sieve and Numeric_NBody already declare concrete types and are flat in all
three modes (p >= 0.09), which is the evidence that the movement comes from the retyped slots and
not from drift.

In threaded mode minivm now beats CPython on NQueens (196.49 vs 226.15 µs), Fannkuch
(399.53 vs 430.81 µs), and StringBuild (335.8 vs 368.74 µs), so those three bold marks move.

Left out, with reasons

Test plan

  • cd benchmarks && go test -run='^$' -bench='^(BenchmarkCall|BenchmarkMemory|BenchmarkControl|BenchmarkNumeric)' -benchtime=1x . — every kernel self-verifies its result; all pass
  • go test -race ./... — 19 packages green
  • make lint — clean
  • docs/benchmarks.md re-measured under its own protocol (-benchtime=300ms -count=3, median, on the machine its header records)

🤖 Generated with Claude Code

NQueens, Fannkuch, SortStress, and StringBuild declared their typed-array
locals and function params as `any` (types.TypeRef). Container fusion proves
an element kind from the declared slot type at threading time, so an `any`
slot never reaches the fused array.get / array.set paths however consistently
it holds a typed array. These kernels therefore under-reported what the
interpreter does with a well-typed program.

The Python sources annotate these explicitly (`cols: list[bool]`,
`perm: list[int]`), and NBody, SpectralNorm, MatMul, and Sieve already declare
`[]f64` / `[]i32` in exactly these positions, so the concrete type is the more
faithful translation rather than a tuned one.

No interpreter code changes. Interleaved A/B, -benchtime=300ms -count=6,
Apple M4 Pro:

  Call_NQueens/threaded      304.3µs -> 195.7µs  -35.66% (p=0.002)
  Call_NQueens/default       136.4µs -> 107.3µs  -21.36% (p=0.002)
  Call_NQueens/jit           131.9µs -> 104.3µs  -20.95% (p=0.002)
  Memory_SortStress/threaded 310.0µs -> 188.3µs  -39.26% (p=0.002)
  Memory_SortStress/default   52.71µs ->  50.81µs -3.60% (p=0.002)
  Call_Fannkuch/threaded     513.2µs -> 400.5µs  -21.97% (p=0.002)
  Call_Fannkuch/default      407.6µs -> 401.7µs   -1.44% (p=0.002)
  Memory_StringBuild/threaded 359.4µs -> 336.6µs  -6.34% (p=0.002)

Control_Sieve and Numeric_NBody, which already declared concrete types, are
flat in all three modes. B/op and allocs/op are identical on every row.

In threaded mode minivm now beats CPython on NQueens, Fannkuch, and
StringBuild, so those three bold marks move in docs/benchmarks.md.

The struct-holding slots in StructTreeWalk and BinaryTrees are deliberately
left as `any`: their Python type is `Node | None`, which minivm cannot express
as a concrete struct type, and declaring one anyway makes the JIT diverge from
the threaded interpreter. Filed separately.
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 13 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 11ec4a0b-ce7b-4f5a-a8cb-33bacff70878

📥 Commits

Reviewing files that changed from the base of the PR and between 125b5ba and ca452eb.

📒 Files selected for processing (3)
  • benchmarks/call_test.go
  • benchmarks/memory_test.go
  • docs/benchmarks.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@siyul-park
siyul-park merged commit 88e1dbd into main Aug 31, 2026
2 checks passed
@siyul-park
siyul-park deleted the test/176-concrete-container-types branch August 31, 2026 12:00
siyul-park added a commit that referenced this pull request Aug 31, 2026
…t slots (#231)

* test(benchmarks): declare the concrete node type on tree-kernel struct slots

StructTreeWalk and BinaryTrees declared every Node-holding local, return, and
param as `any` (types.TypeRef), so `struct.get` never reached its fused
container path. #229 had to leave them that way: declaring the concrete struct
type made the ARM64 JIT fault where threaded execution was correct. #230 fixed
that, so the declarations can now be what the Python sources say.

`left`/`right` stay `any` because the grammar cannot name a recursive type.
The `check`/`item_check` params may hold the null ref at a leaf, which is fine:
heap slots do not exclude null, `struct.new_default` leaves unset ref fields
null, and both tiers now agree on `ref.is_null` for them.

No interpreter code changes. Interleaved A/B, -benchtime=300ms -count=6,
Apple M4 Pro:

  Memory_BinaryTrees/threaded     1.145ms -> 1.059ms  -7.55% (p=0.002)
  Memory_BinaryTrees/default      1.187ms -> 1.104ms  -7.01% (p=0.002)
  Memory_BinaryTrees/jit          1.187ms -> 1.106ms  -6.85% (p=0.002)
  Memory_StructTreeWalk/default   160.5us ->  154.6us -3.69% (p=0.002)
  Memory_StructTreeWalk/threaded  153.0us ->  147.5us -3.58% (p=0.002)
  Memory_StructTreeWalk/jit       160.2us ->  155.7us -2.81% (p=0.002)

Control_Sieve and Numeric_NBody are flat in all three modes. B/op and
allocs/op are identical on every row.

* test(benchmarks): drop the redundant ref.cast from the tree walkers

`check` and `item_check` each downcast their node param with `ref.cast 0`
immediately after a `ref.is_null; br_if` guard. The cast only existed because
the param was declared `any`; with the param declared as the node struct type
it proves nothing the declaration does not already state, and neither Python
source has a cast.

It was not free. `REF_CAST` is bridged, so every node walk left native code for
the interpreter and came back, and the cast also split `local.get 0` from the
`i32.const N; struct.get` behind it, blocking that field read from fusing.

Interleaved A/B against the previous commit, -benchtime=300ms -count=6,
Apple M4 Pro:

  Memory_BinaryTrees/jit           1108.1us -> 752.7us  -32.08% (p=0.002)
  Memory_BinaryTrees/default       1102.7us -> 751.8us  -31.82% (p=0.002)
  Memory_StructTreeWalk/jit          156.0us -> 135.0us -13.43% (p=0.002)
  Memory_StructTreeWalk/default      154.8us -> 134.4us -13.20% (p=0.002)
  Memory_BinaryTrees/threaded      1055.9us -> 950.9us   -9.94% (p=0.002)
  Memory_StructTreeWalk/threaded     147.7us -> 138.1us  -6.52% (p=0.002)

Control_Sieve is flat in all three modes. B/op and allocs/op are unchanged.

* docs(benchmarks): re-measure the tree-kernel rows

StructTreeWalk and BinaryTrees moved when their node slots were declared
concretely and the redundant ref.cast was dropped. Both now run fastest under
default rather than threaded, so they leave that list, and minivm threaded
overtakes CPython on BinaryTrees.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant