fix(interp): make recursive trace boundary explicit - #185

Summary

Fixes #184.

The Fannkuch compare benchmark could produce incorrect results in default and jit modes while threaded remained correct.

Root cause

The trace recorder used skipCall when it encountered a recursive CALL. That is only sound when the recorder can model the call's effects. skipCall supplies placeholder return values and does not reproduce the callee's heap mutations.

A recursive call encountered inside a non-entry loop trace could therefore leave the recorder with a speculative VM/heap state that never existed. Fannkuch exposes this because its recursive permutation routine mutates a shared array and continues execution after the call.

Fix

Treat recursive calls in non-entry loop traces as a trace boundary. The native prefix is retained, the real recursive call executes through the threaded path, and the post-call continuation can be captured from the actual frame and heap state.

Function-entry self-recursion keeps the existing native selfCall path, so the fix does not disable native recursive-call lowering globally or reject heap mutation from JIT plans.

The JIT trace contract documentation is updated to describe the boundary explicitly.

Validation

• go test -race ./...
• go vet ./...
• go test -tags=compare -run='^$' -bench='^BenchmarkCall_Fannkuch$' -benchtime=1x -count=5
• git diff --check

The Fannkuch compare benchmark passes for default, threaded, jit, native, gpython, and cpython on the ARM64 development machine.

Summary by CodeRabbit

• Bug Fixes

  • Improved tracing for recursive calls within loop-based execution paths.
  • Calls to non-entry instructions now terminate the current trace cleanly and resume through standard execution, enabling subsequent continuation tracing.
  • Function-entry recursive calls continue to use native handling.

• Documentation

  • Updated JIT internals documentation to describe the different handling of recursive calls in loop and function-entry traces.