diff --git a/AGENTS.md b/AGENTS.md index 23106c2..3b5fa58 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -178,6 +178,7 @@ Violations cause silent corruption or invalid execution. - A JIT trace fragment's own `status`, not the root trace's, decides how its ops lower when they run out; `tracePlan` must skip any `aborted` root or branch, so a fragment that recorded a partial, unsupported prefix is never planned or inlined into a parent trace. - `noSpill` (`interp/jit_plan.go`) must scan the whole plan (every block, not just the root's tail) before allowing spilling; when it rejects, `noSpillArch` forces `asm.Build` to fail instead of inserting a spill frame. - A deferred (slot-backed or const) ref operand carries no retain of its own; it must be owned or redeemed before any path that hands the flushed operand stack to the interpreter (ownership transfer, guard exit stub, trap-fallback/module-completion redeem, or a real call), and a committing (loop back-edge) flush rejects any live deferred ref. +- Eligible call-free native loops keep up to seven read-written inline scalar locals authoritative in X19-X25 across the back-edge; every guard exit, fallback, yield, completion, state barrier, or continuation that can expose or reload VM slots must commit or preserve those registers first, and ineligible plans keep the per-iteration slot commit. - Hoisted container registers are valid only within one native loop entry: the prologue re-guards tag and itab on every entry, hoist eligibility requires a call-free loop plan with no store to the container local, `asm.OpPseudoUse` keeps the derived registers live across the native back-edge, and a loop fallback that resumes at the header must run the shadowed threaded handler once (the header slot holds the native stub, so redispatching it would livelock). ## Tests diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 76d5147..2229740 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -21,15 +21,21 @@ For implementation details, see `docs/jit-internals.md`. For profiling counters, ## Measurement Environment -The cross-runtime comparison table was measured on July 30, 2026. The public -API cost tables below it were measured on July 16, 2026: +The cross-runtime comparison table was measured on July 30, 2026. Its +`minivm/default` rows were re-measured the same day after loop-carried scalar +write-back using five interleaved baseline/current pairs. The other +cross-runtime rows use three sequential samples. The public API cost tables +below it were measured on July 16, 2026: - Apple M4 Pro, 12 cores - `darwin/arm64` - macOS 26.4.1 - Go 1.26.2 -Every table reports the median of three sequential samples. Lower `ns/op`, `B/op`, and `allocs/op` are better. The runs were executed serially so concurrent benchmark processes did not compete for CPU time. The comparison numbers come from one `-benchtime=300ms` pass over the whole suite, so a short-warmup adaptive mode reads slightly slower there than in a focused single-kernel run. +Lower `ns/op`, `B/op`, and `allocs/op` are better. Runs were serial so +concurrent benchmark processes did not compete for CPU time. The comparison +numbers use `-benchtime=300ms`; a short-warmup adaptive mode can read slightly +slower there than in a focused single-kernel run. ## Reproduction @@ -38,6 +44,15 @@ Every table reports the median of three sequential samples. Lower `ns/op`, `B/op cd benchmarks go test -tags=compare -run='^$' -bench='.' -benchmem -benchtime=300ms -count=3 ./... +# Issue #163 A/B: alternate this command between the baseline and current +# checkout five times, then take each side's median. +go test -run='^$' \ + -bench='^(BenchmarkControl_IterativeFib|BenchmarkControl_Sieve|BenchmarkCall_IndirectRecursiveFib|BenchmarkCall_ClosureCounter|BenchmarkMemory_TypedArraySum|BenchmarkMemory_AllocationGraph|BenchmarkNumeric_BranchTree)$/^default$' \ + -benchmem -benchtime=300ms -count=1 . +go test -run='^$' \ + -bench='^BenchmarkCall_RecursiveFib$/^(20|35)$/^default$' \ + -benchmem -benchtime=300ms -count=1 . + # Public interpreter and pool API costs go test -run='^$' \ -bench='^(BenchmarkNew|BenchmarkInterpreter_(Reset|Push|Pop|PopBoxed|Peek|Alloc|Retain|Release)|BenchmarkPool_(Get|Put))$' \ @@ -58,12 +73,13 @@ go test -run='^$' -bench='^Benchmark(Array|Struct|TypedMap|Map)_Refs$' \ ## Summary -- `RecursiveFib(35)` places `minivm/default` at **46.18 ms**, within about **3.9%** of wazero's **44.45 ms**, while remaining allocation-free after warmup. -- Adaptive native traces reduce `IterativeFib(30)` from **746.3 ns** threaded to **92.15 ns**, `TypedArraySum(256)` from **6.313 us** to **683.2 ns**, and `BranchTree(96)` from **952.0 ns** to **267.1 ns**. +- `RecursiveFib(35)` places `minivm/default` at **46.95 ms**, within about **5.6%** of wazero's **44.45 ms**, while remaining allocation-free after warmup. +- Adaptive native traces reduce `IterativeFib(30)` from **746.3 ns** threaded to **36.75 ns**, `TypedArraySum(256)` from **6.313 us** to **307.7 ns**, and `BranchTree(96)` from **952.0 ns** to **264.4 ns**. +- Loop-carried scalar write-back keeps eligible call-free locals authoritative in registers and commits their VM slots only on native exit paths. Interleaved A/B (median of five) cuts `IterativeFib(30)` **91.75 -> 36.75 ns (-59.9%)**, `TypedArraySum(256)` **673.2 -> 307.7 ns (-54.3%)**, and `Sieve(256)` **2,645 -> 1,626 ns (-38.5%)**. Every other canonical `default` median stays within **0.9%**, and allocation counts are unchanged. - Fused threaded handlers check stack room once for their own net push instead of once per folded source. That removes about 5,400 generated lines and cuts threaded time on every fusion-heavy kernel: `BranchTree(96)` **-4.3%**, `TypedArraySum(256)` **-4.1%**, `IterativeFib(30)` **-3.6%**, and `Sieve(256)` **-2.8%** (interleaved A/B, median of five). Native and adaptive modes are unchanged within noise. - Primitive array mutation stays on the native loop path in `Sieve(256)`: deferred-ownership elision drops the per-element retain/release pair, so a runtime-allocated array reaches the same cheap native path a typed-array constant already used. All three modes allocate `1,048 B` in `2` allocations. - Loop-invariant container hoisting (issue #153) removes the per-access heap-cell derivation, itab guard, and slice-header reload from hoisted loop bodies. It shrinks the loop callables but leaves wall time unchanged: the removed loads sat off the out-of-order critical path. -- Branch-leg folding (issue #155) records native loop exits as branches and folds hot legs that rejoin the header back into the native loop as real back-edges. On `Sieve(256)` this removes the per-prime entry/exit round trips (scan-loop native entries drop from ~55 to ~1 per run) and cuts `default` from **4.72 us** to **2.68 us**, versus **16.2 us** threaded. The remaining gap to wazero (**687.3 ns**) is dominated by the per-iteration operand flush. +- Branch-leg folding (issue #155) records native loop exits as branches and folds hot legs that rejoin the header back into the native loop as real back-edges. Combined with loop-carried scalar write-back, `Sieve(256)` now runs at **1.626 us** versus **16.2 us** threaded and **687.3 ns** in wazero. - Threshold-zero `jit` is not a warmed-JIT guarantee. It matches `default` on Sieve and BranchTree, but is slower on IterativeFib, TypedArraySum, and recursive Fibonacci because it can compile before representative traces are learned. - Allocation-heavy workloads remain interpreter-bound. `AllocationGraph(128)` is fastest in minivm's threaded mode at **7.774 us**; adaptive and eager modes add profiling cost without native coverage. - Indirect recursion reaches the native self-call path in adaptive mode: `IndirectRecursiveFib(20)` is **54.85 us** in `default`, versus **576 us** threaded and **43.3 us** in wazero. Eager `jit` stays at **594 us**, consistent with the threshold-zero note above. @@ -88,7 +104,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | Workload | Runtime | ns/op | B/op | allocs/op | |---|---|---:|---:|---:| -| IterativeFib(30) | minivm/default | 92.15 | 0 | 0 | +| IterativeFib(30) | minivm/default | 36.75 | 0 | 0 | | IterativeFib(30) | minivm/threaded | 746.3 | 0 | 0 | | IterativeFib(30) | minivm/jit | 95.07 | 0 | 0 | | IterativeFib(30) | native Go | 9.256 | 0 | 0 | @@ -98,7 +114,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | IterativeFib(30) | Goja | 2,226 | 368 | 20 | | IterativeFib(30) | gpython | 2,599 | 2,448 | 88 | | IterativeFib(30) | Yaegi | 2,856 | 2,036 | 101 | -| Sieve(256) | minivm/default | 2,682 | 1,048 | 2 | +| Sieve(256) | minivm/default | 1,626 | 1,048 | 2 | | Sieve(256) | minivm/threaded | 16,239 | 1,048 | 2 | | Sieve(256) | minivm/jit | 2,696 | 1,048 | 2 | | Sieve(256) | native Go | 237.2 | 0 | 0 | @@ -108,7 +124,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | Sieve(256) | Goja | 43,140 | 1,872 | 25 | | Sieve(256) | gpython | 35,636 | 5,704 | 30 | | Sieve(256) | Yaegi | 18,762 | 1,800 | 37 | -| RecursiveFib(20) | minivm/default | 38,266 | 0 | 0 | +| RecursiveFib(20) | minivm/default | 38,280 | 0 | 0 | | RecursiveFib(20) | minivm/threaded | 357,450 | 0 | 0 | | RecursiveFib(20) | minivm/jit | 368,423 | 0 | 0 | | RecursiveFib(20) | native Go | 14,455 | 0 | 0 | @@ -118,7 +134,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | RecursiveFib(20) | Goja | 1,456,275 | 4,680 | 39 | | RecursiveFib(20) | gpython | 3,701,017 | 9,807,924 | 109,494 | | RecursiveFib(20) | Yaegi | 3,791,815 | 8,302,126 | 192,840 | -| RecursiveFib(35) | minivm/default | 46,175,150 | 0 | 0 | +| RecursiveFib(35) | minivm/default | 46,951,773 | 0 | 0 | | RecursiveFib(35) | minivm/threaded | 499,079,076 | 0 | 0 | | RecursiveFib(35) | minivm/jit | 522,672,192 | 0 | 0 | | RecursiveFib(35) | native Go | 20,107,461 | 0 | 0 | @@ -128,7 +144,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | RecursiveFib(35) | Goja | 2,099,479,958 | 375,360 | 46,373 | | RecursiveFib(35) | gpython | 5,578,092,292 | 13,378,028,656 | 149,350,236 | | RecursiveFib(35) | Yaegi | 5,903,047,625 | 11,324,344,728 | 263,043,676 | -| IndirectRecursiveFib(20) | minivm/default | 54,848 | 0 | 0 | +| IndirectRecursiveFib(20) | minivm/default | 54,763 | 0 | 0 | | IndirectRecursiveFib(20) | minivm/threaded | 576,182 | 0 | 0 | | IndirectRecursiveFib(20) | minivm/jit | 594,117 | 0 | 0 | | IndirectRecursiveFib(20) | native Go | 15,981 | 0 | 0 | @@ -138,7 +154,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | IndirectRecursiveFib(20) | Goja | 1,377,150 | 4,680 | 39 | | IndirectRecursiveFib(20) | gpython | 4,018,147 | 10,158,201 | 109,494 | | IndirectRecursiveFib(20) | Yaegi | 11,430,601 | 13,059,854 | 394,041 | -| ClosureCounter(128) | minivm/default | 3,362 | 64 | 2 | +| ClosureCounter(128) | minivm/default | 3,387 | 64 | 2 | | ClosureCounter(128) | minivm/threaded | 2,978 | 64 | 2 | | ClosureCounter(128) | minivm/jit | 3,372 | 64 | 2 | | ClosureCounter(128) | native Go | 38.22 | 0 | 0 | @@ -148,7 +164,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | ClosureCounter(128) | Goja | 10,173 | 1,264 | 13 | | ClosureCounter(128) | gpython | 28,235 | 58,312 | 659 | | ClosureCounter(128) | Yaegi | 34,704 | 34,784 | 786 | -| TypedArraySum(256) | minivm/default | 683.2 | 0 | 0 | +| TypedArraySum(256) | minivm/default | 307.7 | 0 | 0 | | TypedArraySum(256) | minivm/threaded | 6,313 | 0 | 0 | | TypedArraySum(256) | minivm/jit | 621.5 | 0 | 0 | | TypedArraySum(256) | native Go | 73.2 | 0 | 0 | @@ -158,7 +174,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | TypedArraySum(256) | Goja | 13,399 | 2,080 | 238 | | TypedArraySum(256) | gpython | 7,652 | 2,496 | 246 | | TypedArraySum(256) | Yaegi | 4,246 | 296 | 8 | -| AllocationGraph(128) | minivm/default | 9,370 | 5,120 | 256 | +| AllocationGraph(128) | minivm/default | 9,301 | 5,120 | 256 | | AllocationGraph(128) | minivm/threaded | 7,774 | 5,120 | 256 | | AllocationGraph(128) | minivm/jit | 9,362 | 5,120 | 256 | | AllocationGraph(128) | native Go | 943.8 | 1,024 | 128 | @@ -168,7 +184,7 @@ Each minivm kernel times `Interpreter.Run` only. Result extraction, reset, fixtu | AllocationGraph(128) | Goja | 26,551 | 78,016 | 770 | | AllocationGraph(128) | gpython | 5,715 | 5,712 | 266 | | AllocationGraph(128) | Yaegi | 12,190 | 1,492 | 142 | -| BranchTree(96) | minivm/default | 267.1 | 0 | 0 | +| BranchTree(96) | minivm/default | 264.4 | 0 | 0 | | BranchTree(96) | minivm/threaded | 952 | 0 | 0 | | BranchTree(96) | minivm/jit | 264.7 | 0 | 0 | | BranchTree(96) | native Go | 79.98 | 0 | 0 | diff --git a/docs/jit-internals.md b/docs/jit-internals.md index 070d703..9ad6296 100644 --- a/docs/jit-internals.md +++ b/docs/jit-internals.md @@ -338,7 +338,7 @@ Targets still deoptimize when they are unknown or unsupported. Branch lowering may skip hot-path flushes only when the branch state is clean. If locals or operands are dirty, flush first. Learned continuations and side exits must see the same stack image as threaded dispatch. -A committing flush (`selfCall`, `tailLoop`) transfers operand ownership to the VM stack, so it accepts a live `backingStack` ref: that ref already carries the retain taken when it was pushed, and committing hands the same edge to the stack, exactly as the inlined call path does when it stores arguments and drops them from the operand stack. It rejects any live deferred ref (a const marker or a slot-backed operand): a deferred ref carries no retain of its own, and a loop back-edge has no cold stub to take one, so owning it each iteration would leak. A self-recursive function still forwards a ref parameter to itself because the argument is owned into the callee frame (through the call-argument path) before the commit. See Reference Ownership. +A committing flush (`selfCall`, `tailLoop`) transfers operand ownership to the VM stack, so it accepts a live `backingStack` ref: that ref already carries the retain taken when it was pushed, and committing hands the same edge to the stack, exactly as the inlined call path does when it stores arguments and drops them from the operand stack. It rejects any live deferred ref (a const marker or a slot-backed operand): a deferred ref carries no retain of its own, and a loop back-edge has no cold stub to take one, so owning it each iteration would leak. Eligible loop-carried scalars are the exception to local materialization: their registers remain authoritative across the back-edge and cold handoffs commit them separately. A self-recursive function still forwards a ref parameter to itself because the argument is owned into the callee frame (through the call-argument path) before the commit. See Reference Ownership. ### Branch range validation @@ -352,9 +352,11 @@ A loop root is anchored at a loop header: the target of a backward branch. The tracer discovers headers statically. Periodic samples still drive normal hotness, while JIT-enabled unconditional backward `BR` handlers also notify the interpreter after moving the live frame to the exact target header. This prevents a deterministic tick phase from permanently missing those loops. Threshold-zero mode waits for eight exact hits on those headers before capture so the first iteration does not over-specialize the recorded branch path. -Loop lowering builds the normal native prologue, binds a back-edge label, emits the loop body once, commits loop-carried locals to VM stack slots, decrements the safepoint budget, and branches back while budget remains. Eligible trace loops use a true backward branch; when the allocator cannot satisfy the resulting no-spill live ranges, compilation retries with bounded forward chaining. +Loop lowering builds the normal native prologue, loads eligible loop-carried scalar locals into X19-X25, binds a back-edge label, emits the loop body once, decrements the safepoint budget, and branches back while budget remains. The carried registers, not their VM slots, are authoritative inside that native entry; `LOCAL_SET` updates the fixed register and the back-edge emits no slot store. Eligible trace loops use a true backward branch; when the allocator cannot satisfy the resulting no-spill live ranges, compilation retries with bounded forward chaining and the VM slots remain authoritative. -Loop-carried locals round-trip through the VM stack each iteration. This avoids a cross-back-edge register fixpoint. Call-free backward loops cache `journalBudget` in a register and write it back only on yield or another cold exit; other loops retain the memory-backed check. +`loopCarried` decides eligibility from the completed static or trace plan. It requires a real backward edge, a call-free plan, and at most seven root-frame locals that are both read and written inside a backward-loop bytecode range and have inline scalar kinds (`i1`, `i8`, `i32`, `f32`, or `f64`). Refs and `i64` remain slot-backed because their loads can deopt or carry ownership rules. Plans with calls, too many candidates, no backward edge, or unsupported kinds keep the committing per-iteration flush. Register pressure retries the same plan without carried registers before abandoning native compilation. + +Every path that can hand state to threaded execution commits carried registers first: queued guard exits, direct trap fallbacks, native-loop budget yields, and module completion. Calls are ineligible, so real-call plans retain the ordinary pre-call slot commit. State barriers and folded continuations preserve carried registers across local-cache reloads and may not reuse them as scratch. Call-free backward loops also cache `journalBudget` in a register and write it back only on yield or another cold exit; other loops retain the memory-backed check. Module loops at `addr == 0` are valid when their header IP is positive. Only loops whose header is the module or function entry (`ip == 0`) remain threaded. diff --git a/interp/jit.go b/interp/jit.go index fe3fb99..dab563f 100644 --- a/interp/jit.go +++ b/interp/jit.go @@ -78,6 +78,7 @@ type lowering struct { reuseLocals bool spare asm.VReg + carried []carriedLocal // hoist caches one loop-invariant container's slice header, derived by a // per-entry prologue (see arm64Lowerer.hoist). The registers are pure @@ -143,6 +144,14 @@ type activation struct { returns int } +// carriedLocal is one root-frame scalar whose register is authoritative until +// a native loop exits. slot remains the VM home committed by cold paths. +type carriedLocal struct { + value value + local int + slot int +} + type localState uint8 const ( @@ -152,10 +161,11 @@ const ( ) // work is a deferred block whose branch point produced its symbolic state: -// VM stack slots are current, so the block re-enters at label with -// every local unloaded and every operand awaiting reload. If the branch -// returned from an inlined callee, tail keeps the caller path that must run -// after the deferred block stitches back into the caller frame. +// ordinary VM stack slots are current, so the block re-enters at label with +// every ordinary local unloaded and every operand awaiting reload. Carried +// loop locals reconnect to their authoritative registers instead. If the +// branch returned from an inlined callee, tail keeps the caller path that must +// run after the deferred block stitches back into the caller frame. type work struct { label asm.Label block int @@ -322,7 +332,17 @@ func (c *compiler) compile(input *compileInput, plan plan, mod *module, frontend } nativeLoop := plan.kind == entryLoop reason, err := c.emit(input, plan, mod, frontend, arch, nativeLoop) - if reason != prof.CompileReasonRegisterPressure || !nativeLoop { + if reason != prof.CompileReasonRegisterPressure { + return reason, err + } + if len(plan.carried) > 0 { + plan.carried = nil + reason, err = c.emit(input, plan, mod, frontend, arch, nativeLoop) + if reason != prof.CompileReasonRegisterPressure { + return reason, err + } + } + if !nativeLoop { return reason, err } return c.emit(input, plan, mod, frontend, arch, false) diff --git a/interp/jit_arm64.go b/interp/jit_arm64.go index 1a4d029..9dbea9f 100644 --- a/interp/jit_arm64.go +++ b/interp/jit_arm64.go @@ -113,7 +113,7 @@ func (l arm64Lowerer) enter(ctx *lowering) { // snapshot was flushed to its VM stack slot without a retain, and the // interpreter resuming there releases each stack ref it pops, so the stub // takes the retain here — on the cold path only. -func (l arm64Lowerer) emitExits(ctx *lowering) { +func (l arm64Lowerer) emitExits(ctx *lowering) bool { // Every exit's cold stub is a mutually exclusive, straight-line block // (each ends in trapFlushed, an unconditional trap/return), so the // registers used to reload-and-retain a deferred value are safe to reuse @@ -132,6 +132,9 @@ func (l arm64Lowerer) emitExits(ctx *lowering) { if ctx.budget.Width() != asm.WidthUndefined { ctx.assembler.Emit(arm64.STR(ctx.budget, ctx.pin(scratchCtrl), int16(journalBudget*8))) } + if !l.commitCarried(ctx) { + return false + } var addr asm.VReg for j, v := range exit.values { switch v.backing { @@ -170,6 +173,7 @@ func (l arm64Lowerer) emitExits(ctx *lowering) { } l.trapFlushed(ctx, trapFallback, exit.resume, exit.id) } + return true } func (l arm64Lowerer) zero32(ctx *lowering, v asm.VReg) asm.VReg { @@ -190,7 +194,7 @@ func (l arm64Lowerer) emitBlock(ctx *lowering, id int, tail []int) bool { for _, slot := range block.state { ctx.values = append(ctx.values, value{kind: slot.kind, ref: slot.ref, backing: slot.backing, slot: slot.slot}) } - clear(ctx.frame().state) + l.clearLocals(ctx) l.reload(ctx) } done, ok := l.steps(ctx, block.steps) @@ -296,8 +300,7 @@ func (l arm64Lowerer) next(ctx *lowering, from anchor, target edge, tail []int, if ctx.hoist.live { ctx.assembler.Emit(asm.Instruction{Op: asm.OpPseudoUse, Src1: asm.V(ctx.hoist.dataPtr), Src2: asm.V(ctx.hoist.n)}) } - l.back(ctx, ctx.back, target.anchor.ip) - return true + return l.back(ctx, ctx.back, target.anchor.ip) } return l.path(ctx, from, target, tail, opcode) } @@ -916,7 +919,14 @@ func (l arm64Lowerer) localSet(ctx *lowering, op step, pop bool) bool { if !vp.raw { return false } - f.locals[idx] = *vp + if carried := l.carried(ctx, f.base+idx); carried != nil { + if carried.value.reg.ID() != vp.reg.ID() { + ctx.assembler.Emit(arm64.MOV(carried.value.reg, vp.reg)) + } + f.locals[idx] = carried.value + } else { + f.locals[idx] = *vp + } f.state[idx] = f.state[idx]&^localStored | localLoaded | localDirty if pop { ctx.pop() @@ -1095,7 +1105,7 @@ func (l arm64Lowerer) clean(ctx *lowering) bool { for fi := range ctx.frames { f := &ctx.frames[fi] for idx := range f.state { - if f.state[idx]&localDirty != 0 { + if f.state[idx]&localDirty != 0 && l.carried(ctx, f.base+idx) == nil { return false } } @@ -1197,7 +1207,7 @@ func (l arm64Lowerer) arrayGetKnown(ctx *lowering, op step) bool { if !l.flush(ctx, flushSnapshot) { return false } - clear(ctx.frame().state) + l.clearLocals(ctx) fail := ctx.queueExit(nil, op.ip, prof.ExitGuardValue, int(op.op)) a := ctx.assembler @@ -1298,8 +1308,7 @@ func (l arm64Lowerer) path(ctx *lowering, from anchor, target edge, tail []int, return false } if target.anchor.addr == from.addr && target.anchor.ip <= from.ip { - l.back(ctx, label, target.anchor.ip) - return true + return l.back(ctx, label, target.anchor.ip) } ctx.assembler.Emit(arm64.BLabel(label)) return true @@ -1307,7 +1316,7 @@ func (l arm64Lowerer) path(ctx *lowering, from anchor, target edge, tail []int, // back decrements the safepoint budget and continues at label while work remains. // Native loops keep the budget in a register; chained loops update its VM slot. -func (l arm64Lowerer) back(ctx *lowering, label asm.Label, resume int) { +func (l arm64Lowerer) back(ctx *lowering, label asm.Label, resume int) bool { a := ctx.assembler vCtrl := ctx.pin(scratchCtrl) budget := ctx.budget @@ -1323,7 +1332,11 @@ func (l arm64Lowerer) back(ctx *lowering, label asm.Label, resume int) { if ctx.budget.Width() != asm.WidthUndefined { a.Emit(arm64.STR(budget, vCtrl, int16(journalBudget*8))) } + if !l.commitCarried(ctx) { + return false + } l.trapFlushed(ctx, trapYield, resume, -1) + return true } func (l arm64Lowerer) label(ctx *lowering, target edge, tail []int, opcode int) (asm.Label, bool) { @@ -1525,9 +1538,7 @@ func (l arm64Lowerer) directCall(ctx *lowering, op step) bool { regs[idx] = ctx.pinTo(arm64.IntRets[idx]) } ctx.values = ctx.values[:len(ctx.values)-params] - for fi := range ctx.frames { - clear(ctx.frames[fi].state) - } + l.clearLocals(ctx) l.reload(ctx) for idx, typ := range rets { ctx.push(value{reg: regs[idx], kind: typ.Kind(), raw: true}) @@ -2734,6 +2745,9 @@ func (l arm64Lowerer) complete(ctx *lowering) bool { if !l.flush(ctx, flushSnapshot) { return false } + if !l.commitCarried(ctx) { + return false + } // The wrapper preserves this top-level operand stack on trapNone (see // start()), and the interpreter adopts each stack ref as owned, so a // deferred ref left on the stack at module end must re-take its retain. @@ -3361,9 +3375,7 @@ func (l arm64Lowerer) arraySet(ctx *lowering, op step) bool { if !l.flush(ctx, flushSnapshot) { return false } - for idx := range ctx.frames { - clear(ctx.frames[idx].state) - } + l.clearLocals(ctx) ctx.reuseLocals = len(ctx.values) == 3 fail = ctx.queueExit(nil, op.ip, prof.ExitGuardShape, int(op.op)) bounds = ctx.queueExit(nil, op.ip, prof.ExitGuardBounds, int(op.op)) @@ -3628,9 +3640,7 @@ func (l arm64Lowerer) structSet(ctx *lowering, op step) bool { if !l.flush(ctx, flushSnapshot) { return false } - for idx := range ctx.frames { - clear(ctx.frames[idx].state) - } + l.clearLocals(ctx) ctx.reuseLocals = len(ctx.values) == 3 fail = ctx.queueExit(nil, op.ip, prof.ExitGuardShape, int(op.op)) bounds = ctx.queueExit(nil, op.ip, prof.ExitGuardBounds, int(op.op)) @@ -3978,6 +3988,9 @@ func (l arm64Lowerer) trap(ctx *lowering, kind, resume int, reason prof.ExitReas if !l.flush(ctx, flushSnapshot) { return false } + if !l.commitCarried(ctx) { + return false + } id := -1 if kind == trapFallback { // trapFallback hands the flushed operand stack to the threaded @@ -4263,19 +4276,24 @@ func (l arm64Lowerer) flush(ctx *lowering, mode flushMode) bool { } } a := ctx.assembler - vStack := ctx.pin(scratchStack) - addr := l.base(ctx, vStack) + var addr asm.VReg for fi := range ctx.frames { f := &ctx.frames[fi] for idx := range f.kinds { if f.state[idx]&localDirty == 0 { continue } + if l.carried(ctx, f.base+idx) != nil { + continue + } if f.state[idx]&localStored == 0 { boxed, ok := l.boxHome(ctx, f.locals[idx]) if !ok { return false } + if addr.Width() == asm.WidthUndefined { + addr = l.base(ctx, ctx.pin(scratchStack)) + } a.Emit(arm64.STR(boxed, addr, int16((f.base+idx)*8))) f.state[idx] |= localStored } @@ -4291,6 +4309,9 @@ func (l arm64Lowerer) flush(ctx *lowering, mode flushMode) bool { // emitExits). The commit pre-scan above already rejected any deferred // backing, so those cases only run on a non-commit flush. for j, v := range ctx.values { + if addr.Width() == asm.WidthUndefined { + addr = l.base(ctx, ctx.pin(scratchStack)) + } switch v.backing { case backingStack: boxed, ok := l.boxHome(ctx, v) @@ -4309,12 +4330,81 @@ func (l arm64Lowerer) flush(ctx *lowering, mode flushMode) bool { return true } +// carry loads each eligible root-frame local before the loop label and keeps +// its register authoritative until a cold handoff commits it to the VM slot. +func (l arm64Lowerer) carry(ctx *lowering, locals []int, ip int) bool { + regs := [...]asm.PReg{arm64.X19, arm64.X20, arm64.X21, arm64.X22, arm64.X23, arm64.X24, arm64.X25} + if len(locals) > len(regs) { + return false + } + f := ctx.frame() + for idx, local := range locals { + if local < 0 || local >= len(f.kinds) || !l.loadLocal(ctx, f, local, ip) { + return false + } + pinned := ctx.pinTo(regs[idx]) + ctx.assembler.Emit(arm64.MOV(pinned, f.locals[local].reg)) + f.locals[local].reg = pinned + ctx.carried = append(ctx.carried, carriedLocal{ + value: f.locals[local], + local: local, + slot: f.base + local, + }) + } + return true +} + +// commitCarried writes the authoritative loop registers to their VM homes. +// Callers place it only on paths that return control to the interpreter. +func (l arm64Lowerer) commitCarried(ctx *lowering) bool { + if len(ctx.carried) == 0 { + return true + } + addr := l.base(ctx, ctx.pin(scratchStack)) + for _, carried := range ctx.carried { + boxed, ok := l.boxHome(ctx, carried.value) + if !ok { + return false + } + ctx.assembler.Emit(arm64.STR(boxed, addr, int16(carried.slot*8))) + } + return true +} + +func (arm64Lowerer) carried(ctx *lowering, slot int) *carriedLocal { + for idx := range ctx.carried { + if ctx.carried[idx].slot == slot { + return &ctx.carried[idx] + } + } + return nil +} + +// clearLocals invalidates ordinary local caches while preserving root-loop +// registers whose values no longer come from their VM slots. +func (l arm64Lowerer) clearLocals(ctx *lowering) { + for idx := range ctx.frames { + clear(ctx.frames[idx].state) + } + if len(ctx.frames) == 0 { + return + } + f := &ctx.frames[0] + for _, carried := range ctx.carried { + f.locals[carried.local] = carried.value + f.state[carried.local] = localLoaded | localDirty + } +} + // localScratch returns a flushed local register that is no longer live in the // operand stack, or an undefined register when none can be reused safely. -func (arm64Lowerer) localScratch(ctx *lowering) asm.VReg { +func (l arm64Lowerer) localScratch(ctx *lowering) asm.VReg { for fi := range ctx.frames { frame := &ctx.frames[fi] - for _, local := range frame.locals { + for idx, local := range frame.locals { + if l.carried(ctx, frame.base+idx) != nil { + continue + } reg := local.reg if reg.Width() == asm.WidthUndefined { continue @@ -4597,6 +4687,9 @@ func lower(ctx *lowering, plan plan) bool { ctx.budget = ctx.assembler.Reg(asm.RegTypeInt, asm.Width64) ctx.assembler.Emit(arm64.LDR(ctx.budget, ctx.pin(scratchCtrl), int16(journalBudget*8))) } + if len(plan.carried) > 0 && !l.carry(ctx, plan.carried, plan.anchor.ip) { + return false + } if plan.kind == entryLoop && plan.hoist != nil && !l.hoist(ctx, *plan.hoist, plan.anchor.ip) { return false } @@ -4620,13 +4713,13 @@ func lower(ctx *lowering, plan plan) bool { ctx.values = work.values ctx.frames = work.frames ctx.assembler.Bind(work.label) + l.clearLocals(ctx) l.reload(ctx) if !l.emitBlock(ctx, work.block, work.tail) { return false } } - l.emitExits(ctx) - return true + return l.emitExits(ctx) } // hoist derives the plan's loop-invariant container once per native entry: diff --git a/interp/jit_arm64_test.go b/interp/jit_arm64_test.go index 9db4e43..b362971 100644 --- a/interp/jit_arm64_test.go +++ b/interp/jit_arm64_test.go @@ -11,6 +11,7 @@ import ( "testing" "github.com/siyul-park/minivm/asm" + "github.com/siyul-park/minivm/asm/arm64" "github.com/siyul-park/minivm/instr" "github.com/siyul-park/minivm/prof" "github.com/siyul-park/minivm/program" @@ -178,6 +179,125 @@ func TestARM64_Backedge(t *testing.T) { } } +// LoopCarriedLocals protects write-back scalar locals in native loops. Hot +// backedges keep their slots stale; every side exit and safepoint yield must +// commit current registers before threaded execution observes the frame. +func TestARM64_LoopCarriedLocals(t *testing.T) { + if runtime.GOARCH != "arm64" { + t.Skip("native JIT is only available on arm64") + } + + t.Run("folded side exits preserve accumulators", func(t *testing.T) { + const size = int32(16) + b := program.NewBuilder() + b.Locals(types.TypeI32, types.TypeI32) + loop := b.Label() + odd := b.Label() + advance := b.Label() + done := b.Label() + b.Emit(instr.I32_CONST, 0).Emit(instr.LOCAL_SET, 0) + b.Emit(instr.I32_CONST, 0).Emit(instr.LOCAL_SET, 1) + b.Bind(loop) + b.Emit(instr.LOCAL_GET, 0).Emit(instr.I32_CONST, uint64(uint32(size))).Emit(instr.I32_GE_S).BrIf(done) + b.Emit(instr.LOCAL_GET, 0).Emit(instr.I32_CONST, 1).Emit(instr.I32_AND).BrIf(odd) + b.Emit(instr.LOCAL_GET, 1).Emit(instr.I32_CONST, 1).Emit(instr.I32_ADD).Emit(instr.LOCAL_SET, 1).Br(advance) + b.Bind(odd) + b.Emit(instr.LOCAL_GET, 1).Emit(instr.I32_CONST, 2).Emit(instr.I32_ADD).Emit(instr.LOCAL_SET, 1) + b.Bind(advance) + b.Emit(instr.LOCAL_GET, 0).Emit(instr.I32_CONST, 1).Emit(instr.I32_ADD).Emit(instr.LOCAL_SET, 0).Br(loop) + b.Bind(done).Emit(instr.LOCAL_GET, 1) + prog, err := b.Build() + require.NoError(t, err) + + profile := prof.New() + jit := New(prog, WithTick(1), WithThreshold(0), WithProfiler(profile)) + threaded := New(prog, WithTick(1), WithThreshold(-1)) + for iteration := 0; iteration < 32; iteration++ { + require.NoError(t, jit.Run(context.Background())) + require.NoError(t, threaded.Run(context.Background())) + got, err := jit.PopBoxed() + require.NoError(t, err) + want, err := threaded.PopBoxed() + require.NoError(t, err) + require.Equal(t, want, got) + require.Equal(t, types.BoxI32(size+size/2), got) + jit.Reset() + threaded.Reset() + } + require.NoError(t, jit.Close()) + require.NoError(t, threaded.Close()) + + var entries float64 + for _, metric := range profile.Metrics() { + if metric.Name == "vm_jit_native_entries_total" { + entries += metric.Value + } + } + require.Greater(t, entries, float64(0)) + }) + + t.Run("yield commits before WithTick one safepoint", func(t *testing.T) { + const limit = int32(loopBudget + 3) + b := program.NewBuilder() + b.Locals(types.TypeI32) + loop := b.Label() + done := b.Label() + b.Emit(instr.I32_CONST, 0).Emit(instr.LOCAL_SET, 0) + b.Bind(loop) + b.Emit(instr.LOCAL_GET, 0).Emit(instr.I32_CONST, uint64(uint32(limit))).Emit(instr.I32_GE_S).BrIf(done) + b.Emit(instr.LOCAL_GET, 0).Emit(instr.I32_CONST, 1).Emit(instr.I32_ADD).Emit(instr.LOCAL_SET, 0).Br(loop) + b.Bind(done).Emit(instr.LOCAL_GET, 0) + prog, err := b.Build() + require.NoError(t, err) + + profile := prof.New() + jit := New(prog, WithTick(1), WithThreshold(0), WithProfiler(profile)) + for iteration := 0; iteration < 12; iteration++ { + require.NoError(t, jit.Run(context.Background())) + got, err := jit.PopBoxed() + require.NoError(t, err) + require.Equal(t, types.BoxI32(limit), got) + jit.Reset() + } + require.NoError(t, jit.Close()) + + var yields float64 + for _, metric := range profile.Metrics() { + if metric.Name == "vm_jit_native_yields_total" { + yields += metric.Value + } + } + require.Greater(t, yields, float64(0)) + }) +} + +// Flush protects the hot-backedge invariant directly: a dirty carried local +// remains register-authoritative and emits no VM-slot materialization. +func TestARM64_Flush(t *testing.T) { + if runtime.GOARCH != "arm64" { + t.Skip("native JIT is only available on arm64") + } + + assembler := asm.New(arm64.New()) + reg := assembler.Reg(asm.RegTypeInt, asm.Width64) + local := value{reg: reg, kind: types.KindI32, raw: true} + ctx := &lowering{ + assembler: assembler, + frames: []activation{{ + kinds: []types.Kind{types.KindI32}, + locals: []value{local}, + state: []localState{localLoaded | localDirty}, + }}, + carried: []carriedLocal{{value: local}}, + } + + require.True(t, (arm64Lowerer{}).flush(ctx, flushCommit)) + code, err := assembler.Build() + require.NoError(t, err) + require.Empty(t, code) + require.Equal(t, localLoaded|localDirty, ctx.frames[0].state[0]) +} + // AbortedSideExitDoesNotComplete protects partial unsupported traces from // miscompile where a captured side-exit fragment that recorded a few // supported opcodes and then aborted on an unsupported one (MAP_NEW_DEFAULT diff --git a/interp/jit_plan.go b/interp/jit_plan.go index 322a196..099edb7 100644 --- a/interp/jit_plan.go +++ b/interp/jit_plan.go @@ -50,6 +50,7 @@ type plan struct { kind entryKind root int blocks []block + carried []int hoist *hoist noSpill bool } @@ -122,6 +123,7 @@ const ( const ( noBlock = -1 maxHoistSlot = 4095 + maxCarried = 7 ) func input(i *Interpreter, addr int) (*compileInput, bool) { @@ -310,6 +312,7 @@ func staticPlan(input *compileInput) ([]plan, error) { roots[block.anchor] = id } wire(&result, roots) + result.carried = loopCarried(input.function, result.blocks) return []plan{result}, nil } @@ -405,6 +408,7 @@ func tracePlan(input *compileInput) ([]plan, error) { } } wire(&planned, roots) + planned.carried = loopCarried(input.function, planned.blocks) if kind == entryLoop { planned.hoist = hoistable(input.function, planned.blocks) } @@ -414,6 +418,80 @@ func tracePlan(input *compileInput) ([]plan, error) { return plans, nil } +// loopCarried returns the inline scalar locals that a call-free native loop +// may keep authoritative in registers until an exit. The plan must contain a +// real backward edge; straight-line prefixes keep the VM slots authoritative. +// Refs and i64s stay slot-backed because their load and ownership guards can +// deopt while the register set is only partly prepared. +func loopCarried(fn *types.Function, blocks []block) []int { + if fn == nil { + return nil + } + type loopRange struct { + addr int + start, end int + } + var loops []loopRange + for _, block := range blocks { + for _, edge := range block.term.edges { + if edge.block != noBlock && edge.anchor.addr == block.anchor.addr && edge.anchor.ip <= block.anchor.ip { + loops = append(loops, loopRange{addr: block.anchor.addr, start: edge.anchor.ip, end: block.anchor.ip}) + } + } + } + if len(loops) == 0 { + return nil + } + + locals := fn.Slots() + read := make([]bool, len(locals)) + written := make([]bool, len(locals)) + for _, block := range blocks { + inside := false + for _, loop := range loops { + if block.anchor.addr == loop.addr && block.anchor.ip >= loop.start && block.anchor.ip <= loop.end { + inside = true + break + } + } + for _, step := range block.steps { + if step.op == instr.CALL || step.op == instr.RETURN_CALL { + return nil + } + if !inside { + continue + } + switch step.op { + case instr.LOCAL_GET: + local := int(step.args[0]) + if local >= 0 && local < len(read) { + read[local] = true + } + case instr.LOCAL_SET, instr.LOCAL_TEE: + local := int(step.args[0]) + if local >= 0 && local < len(written) { + written[local] = true + } + } + } + } + + var carried []int + for local, ok := range written { + if !ok || !read[local] || local > maxHoistSlot { + continue + } + switch locals[local] { + case types.KindI1, types.KindI8, types.KindI32, types.KindF32, types.KindF64: + carried = append(carried, local) + } + } + if len(carried) > maxCarried { + return nil + } + return carried +} + // hoistable picks the most-accessed loop-invariant container for a loop plan. // A local qualifies when it is a declared ref, no block writes it, and every // recorded ARRAY_GET/ARRAY_SET on it observed one itab. Any call disqualifies diff --git a/interp/jit_plan_test.go b/interp/jit_plan_test.go index 10a0a6b..969af2c 100644 --- a/interp/jit_plan_test.go +++ b/interp/jit_plan_test.go @@ -485,6 +485,88 @@ func TestTracePlan(t *testing.T) { }) } +func TestLoopCarried(t *testing.T) { + loop := func(steps []step) []block { + return []block{{ + steps: steps, + term: terminator{kind: terminateBranch, edges: []edge{{block: 0}}}, + }} + } + + t.Run("selects read-written inline scalars", func(t *testing.T) { + fn := &types.Function{Locals: []types.Type{ + types.TypeI32, types.TypeF64, types.TypeI64, types.TypeRef, types.TypeI32, + }} + blocks := loop([]step{ + {op: instr.LOCAL_GET, args: [2]uint64{0}}, + {op: instr.LOCAL_SET, args: [2]uint64{0}}, + {op: instr.LOCAL_GET, args: [2]uint64{1}}, + {op: instr.LOCAL_TEE, args: [2]uint64{1}}, + {op: instr.LOCAL_GET, args: [2]uint64{2}}, + {op: instr.LOCAL_SET, args: [2]uint64{2}}, + {op: instr.LOCAL_GET, args: [2]uint64{3}}, + {op: instr.LOCAL_SET, args: [2]uint64{3}}, + {op: instr.LOCAL_SET, args: [2]uint64{4}}, + }) + + require.Equal(t, []int{0, 1}, loopCarried(fn, blocks)) + }) + + t.Run("requires a root backedge", func(t *testing.T) { + fn := &types.Function{Locals: []types.Type{types.TypeI32}} + blocks := loop([]step{ + {op: instr.LOCAL_GET, args: [2]uint64{0}}, + {op: instr.LOCAL_SET, args: [2]uint64{0}}, + }) + blocks[0].term.edges[0].block = noBlock + + require.Nil(t, loopCarried(fn, blocks)) + }) + + t.Run("ignores initialization before the loop", func(t *testing.T) { + fn := &types.Function{Locals: []types.Type{types.TypeI32}} + blocks := []block{ + { + anchor: anchor{ip: 0}, + steps: []step{{op: instr.LOCAL_SET, args: [2]uint64{0}}}, + term: terminator{kind: terminateBranch, edges: []edge{{anchor: anchor{ip: 4}, block: 1}}}, + }, + { + anchor: anchor{ip: 4}, + steps: []step{{op: instr.LOCAL_GET, args: [2]uint64{0}}}, + term: terminator{kind: terminateBranch, edges: []edge{{anchor: anchor{ip: 4}, block: 1}}}, + }, + } + + require.Nil(t, loopCarried(fn, blocks)) + }) + + t.Run("rejects calls", func(t *testing.T) { + fn := &types.Function{Locals: []types.Type{types.TypeI32}} + blocks := loop([]step{ + {op: instr.LOCAL_GET, args: [2]uint64{0}}, + {op: instr.LOCAL_SET, args: [2]uint64{0}}, + {op: instr.CALL}, + }) + + require.Nil(t, loopCarried(fn, blocks)) + }) + + t.Run("rejects register overflow", func(t *testing.T) { + fn := &types.Function{Locals: make([]types.Type, maxCarried+1)} + steps := make([]step, 0, 2*len(fn.Locals)) + for local := range fn.Locals { + fn.Locals[local] = types.TypeI32 + steps = append(steps, + step{op: instr.LOCAL_GET, args: [2]uint64{uint64(local)}}, + step{op: instr.LOCAL_SET, args: [2]uint64{uint64(local)}}, + ) + } + + require.Nil(t, loopCarried(fn, loop(steps))) + }) +} + func TestHoistable(t *testing.T) { i32 := itab(types.TypedArray[int32](nil)) fn := &types.Function{Locals: []types.Type{types.TypeI32Array, types.TypeI32}}