Results for Laya MPS 0.1.0, measured September 20, 2026 on an Apple M5 Pro, 24 GiB RAM, macOS 26.4, Python 3.12.4, PyTorch 2.14.0.

These tests measure memory, latency, output preservation, and agreement with reference labels. The default is Laya typed-decisions, FP32, reduced memory, on the Mac GPU. One model stays loaded and one request runs at a time.

The same complete checkpoint ran in all three modes, with one question per forward:

minimal used 68% less peak process RAM than full. All modes matched exactly on 260/260 decisions, including native probabilities and logits. Default answers also matched diagnostic responses. This verifies output preservation; it does not establish whether the model's answers are correct.

The default reduced mode combines latency similar to full with lower RAM. How memory savings work.

Measurement conditions:

• Fresh processes ran sequentially, one per mode, with two warmups before 111 fixed requests. Cases cover all three answer types, custom Noul descriptions, ambiguous inputs, 26-label choices, 10-level scores, full context, and 12 Pong states. Earlier inputs are repeated afterward to check for stale buffers.
• RAM is peak process physical footprint, including loading. CPU/GPU allocations are not added together. macOS, browsers, and other applications are excluded; these figures do not establish a minimum Mac RAM requirement.
• Latency is engine time for fixed Pong inputs with diagnostics enabled, excluding HTTP. Each mode has one measured run. The small difference between reduced and full is within ordinary run variation.
• OS caches remained intact and other applications were open. minimal requested 190.07 GiB of logical layer reads; process disk reads increased by 5.80 GiB after loading. Logical reads are not physical SSD traffic.

Summary JSON · Inputs, outputs, and runtime settings

Typed-decisions targets English customer service, invoices, security incidents, and agent traces. It had the strongest workflow label agreement in this test. The original English checkpoint is available with --checkpoint english; multilingual and Core ML candidates are benchmark-only in this release. Upstream model scope.

We compared 107 fixed requests / 267 decisions across three original checkpoints and two exports from laya-coreml.

Workflow labels come from the public, synthetic typed-decisions dataset. They measure agreement with a teacher, not independent ground truth. Names, email addresses, and incident details are dataset fixtures, not local user data. We selected ten test cases per workflow by sorting SHA-256 of localjev-0.1.0-model-selection: plus the case ID, taking the first ten, and retaining all five questions and outcome descriptions. The specialist was trained on these workflows' training split, so this is an in-domain check. Score agreement uses the most probable level rather than the expected score.

The remaining 67 decisions cover routing, facts, sentiment, and Pong with fixed expected answers. Non-English routing uses Spanish, French, German, and Japanese with English instructions; this small sample does not establish broad language support. Compact Pong has six observations designed after seeing errors in the original prompt. It is a prompt experiment, not a held-out test or the shipped prompt. Game scores also depend on prompts, control timing, and planner features or safety logic, such as those in the laya-coreml Snake demo.

Each candidate ran in a fresh process, sequentially. Latency is the median of 30 normal engine calls across six short routing requests after two warmups per request. It excludes HTTP and model loading.

For typed-decisions, Core ML and Laya MPS matched 267/267 selected answers on identical token sequences; maximum probability difference was 0.000851. Core ML used CPU_AND_GPU, but its compute plan preferred CPU for all 1,643 attributed operations. A compute plan is not a runtime trace; this result applies to the measured Mac and macOS version.

The ANE export was faster on fitting inputs, but its 96-token limit includes the state, question, and options. It rejected 52/107 requests, including all workflow cases. On the 55 fitting questions, selected answers matched the original multilingual checkpoint; maximum probability difference was 0.007715. Its plan preferred Neural Engine for 6,390 attributed operations.

RAM includes model loading and the accepted workload. ANE's peak excludes the long inputs it rejected; the other candidates processed those inputs. Core ML service processes and other applications are excluded. Both exports returned finite outputs, but the ANE auxiliary action head emitted NumPy warnings and the full-context export emitted shape warnings during plan inspection/cleanup; their causes were not established.

These results support the current typed-decisions / FP32 GPU default. A replacement ANE backend would need the same checkpoint, sufficient context, conversion checks, and end-to-end measurements. Probabilities are not calibrated for every task. Power, battery life, thermals, and M1–M4 performance have not been measured.

Frozen inputs · Recorded outputs and versions · Comparison JSON

A 60-second Chrome run in reduced mode completed 600 responses, averaging 39 ms, 10.0 applied moves/s, and 1.94 GiB current server RAM at completion. Two warmups were excluded. The paddle travelled 364 pixels vertically; chart, pause/resume, reset, and mobile layout checks passed. This measures responsiveness, not decision accuracy. Browser results.

The HTTP checks cover all memory modes and both supported checkpoints: small default responses, opt-in metrics, invalid requests, body limits, busy responses, recovery, and clean shutdown. HTTP results.

Use an Apple Silicon Mac and Node.js 22+. Stop the demo server and run commands from the repository root, one at a time.

uv sync --locked
uv run --locked laya-mps setup
uv run --locked python benchmarks/memory.py

Results go to results/memory/. The script checks FP32 parameters, source and input hashes, complete answers and diagnostics, and repeated-input agreement. A failed comparison exits nonzero.

uv run --locked python benchmarks/model_selection.py typed-decisions --download
uv run --locked python benchmarks/model_selection.py english --download
uv run --locked python benchmarks/model_selection.py multilingual --download

The multilingual revision is pinned for evaluation and does not change server options. To complete the five-candidate comparison, run the Core ML exports on macOS 15+ in a separate environment:

uv venv --python 3.12 /tmp/laya-mps-ane-eval
uv pip install --python /tmp/laya-mps-ane-eval/bin/python \
  'laya-coreml==0.1.0' 'coremltools==9.0' 'numpy==2.1.3' \
  'tokenizers==0.23.2' 'psutil==7.0.0'
/tmp/laya-mps-ane-eval/bin/python benchmarks/model_selection.py ane \
  --model-dir .models/comparison-ane --download
/tmp/laya-mps-ane-eval/bin/python benchmarks/model_selection.py typed-coreml \
  --model-dir .models/comparison-typed-coreml --download
uv run --locked python benchmarks/model_selection.py compare

Results go to results/model-selection/. Downloads precede inference. Later runs can omit --download and use HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1. Converted weights are not shipped in this repository.

uv run --locked laya-mps setup --checkpoint english
uv run --locked python benchmarks/server_smoke.py

The script starts and stops its own servers; results go to results/server-smoke/. For a live workload, start scripts/serve.sh and select Run Benchmark in the demo. The chart shows HTTP response times; the sidebar shows average latency, applied moves per second, and current server RAM. Late replies still move the paddle within their rally.

Measurements predate the rename from LocalJev to Laya MPS. The original relative source paths, hashes, and sampling seed are retained to identify the measured version; they do not describe the current checkout. Published reports live in benchmarks/results/. Scripts write new runs to the ignored root results/ directory; review them before copying them into the published folder.