Apple Silicon AI — Multi-engine LLM benchmark & monitoring CLI

asiai compares inference engines side-by-side on your Mac. Load the same model on Ollama and LM Studio, run asiai bench, get the numbers. No guessing, no vibes — just tok/s, TTFT, power efficiency, and stability per engine.

Share your results with the community (--share), compare against other Apple Silicon users (asiai compare), and get smart engine recommendations (asiai recommend).

Born from the OpenClaw project, where we needed hard data to pick the fastest engine for multi-agent swarms on Mac Mini M4 Pro.

pipx install asiai        # Recommended: isolated install

Or via Homebrew:

brew tap druide67/tap
brew install asiai

Other options:

uvx asiai detect           # Run without installing (requires uv)
pip install asiai           # Standard pip install

Then benchmark and share:

asiai bench --quick --card --share    # Bench + shareable card in ~15 seconds

Auto-detect running inference engines across all known engine ports.

$ asiai detect

Detected engines:

  ● ollama 0.17.4
    URL: http://localhost:11434

  ● lmstudio 0.4.5
    URL: http://localhost:1234
    Running: 1 model(s)
      - qwen3.5-35b-a3b  MLX

Cross-engine benchmark with standardized prompts. Runs 3 iterations per prompt by default, reports median tok/s (SPEC standard) with stability classification.

$ asiai bench -m qwen3.5 --runs 3 --power

  Mac Mini M4 Pro — Apple M4 Pro  RAM: 64.0 GB (42% used)  Pressure: normal

Benchmark: qwen3.5

  Engine       tok/s (±stddev)    Tokens   Duration     TTFT       VRAM    Thermal
  ────────── ───────────────── ───────── ────────── ──────── ────────── ──────────
  lmstudio    72.6 ± 0.0 (stable)   435    6.20s    0.28s        —    nominal
  ollama      30.4 ± 0.1 (stable)   448   15.28s    0.25s   26.0 GB   nominal

  Winner: lmstudio (2.4x faster)
  Power: lmstudio 13.2W (5.52 tok/s/W) — ollama 16.0W (1.89 tok/s/W)

Options:

-m, --model MODEL          Model to benchmark (default: auto-detect)
-e, --engines LIST         Filter engines (e.g. ollama,lmstudio,mlxlm)
-p, --prompts LIST         Prompt types: code, tool_call, reasoning, long_gen
-r, --runs N               Runs per prompt (default: 3, for median + stddev)
    --power                Cross-validate power with sudo powermetrics (IOReport always-on)
    --context-size SIZE    Context fill prompt: 4k, 16k, 32k, 64k
    --share                Share results with the community (anonymous, opt-in)
-Q, --quick                Quick benchmark: 1 prompt, 1 run (~15 seconds)
    --card                 Generate shareable benchmark card (SVG + PNG with --share)
-H, --history PERIOD       Show past benchmarks (e.g. 7d, 24h)
    --agentic-mode         Run the 8-run agentic prefix-cache-reuse protocol
    --agentic-output FILE  Save agentic-mode results as JSON
    --agentic-skip-long    Skip phases 7-8 (50K context) to save ~10 min
    --agentic-only LIST    Run only specified phases (cold,prefix-test-1,...)
    --burst-mode           Concurrency stress: N simultaneous calls (tool-call fan-out)
    --burst-runs N         Repeat the burst protocol N times (median + CV)
    --code                 Dev-quality eval: tool-call, recovery, thinking, coding
    --code-suite LIST      tool-call[-stress],recovery,thinking[,coding[-hard]]
    --instruct             Instruction-following: IFEval-style verifiable + agentic deliverable
    --instruct-scenario L  verifiable,research-brief[,order-control]
    --language CODE        Multilingual retention eval (fr/de/es/it/pt/ja/ko/zh)
    --language-suite LIST  adherence,diacritics[,fluency] (default: deterministic 2)
    --judge-url URL        OpenAI-compat LLM judge for the 'coding'/'fluency' suites

asiai bench --agentic-mode --url http://localhost:8080 --model my-model \
    --agentic-output bench.json

Runs 8 sequential prompts with a fixed long system message and varying user messages to expose how the engine reuses cached prefix tokens. Reads cached_tokens from the streaming usage when the engine exposes it (llama.cpp, mlx-lm), falls back to the TTFT ratio otherwise. Outputs a verdict prefix_cache_reuse: yes | partial | no. The metric that matters when your workload is multi-turn agentic with shared system prompts.

Throughput is not quality. Three deterministic modes (no LLM judge needed for the core signal) measure whether a model is actually usable for real work:

# Dev quality: tool-call reliability (the JSON arg-truncation / empty-object bug),
# agentic error-recovery, thinking discipline — + an optional LLM-judged coding task.
asiai bench --code --url http://localhost:8080 --code-output code.json

# Instruction-following: IFEval-style verifiable instructions (format/length/
# keywords/case…) + an agentic task — does the model produce the primary
# multi-section deliverable AFTER a tool sequence, or only confirm the last step?
asiai bench --instruct --url http://localhost:8080 --instruct-output instruct.json

# Multilingual retention: did a finetune keep the base model's language?
# Adherence (stays in the language) + diacritics (café stays café), 8 languages.
asiai bench --language fr --url http://localhost:8080 --language-output lang.json

--code scores tool-call validity, the empty-object truncation bug, schema conformance and error-recovery deterministically; add --code-suite coding with --judge-url <openai-compat-endpoint> for an LLM-judged code-quality grade (no SDK is bundled; the API key is read from the environment). --instruct runs IFEval-style verifiable instructions (strict + loose, prompt- and instruction-level) plus a tools-then-deliverable scenario that catches a finetune doing the tool work but skipping the primary written output. --language measures language adherence + orthography retention — the catastrophic-forgetting signatures a task-specific finetune can introduce. All JSON-only and compare across models by diffing the output. See Dev-quality benchmarks.

Cross-model comparison — benchmark multiple models in one run and get a ranked summary:

# Cross-model comparison
asiai bench --compare qwen3.5:4b deepseek-r1:7b -e ollama --card

The runner resolves model names across engines automatically — gemma2:9b (Ollama) and gemma-2-9b (LM Studio) are matched as the same model.

List loaded models across all engines. Use --json for machine-readable output.

$ asiai models

ollama  http://localhost:11434
  ● qwen3.5:35b-a3b                             26.0 GB Q4_K_M

lmstudio  http://localhost:1234
  ● qwen3.5-35b-a3b                                 MLX

System and inference metrics snapshot, stored in SQLite. Use --json for machine-readable output.

$ asiai monitor

System
  Uptime:    3d 12h
  CPU Load:  2.45 / 3.12 / 2.89  (1m / 5m / 15m)
  Memory:    45.2 GB / 64.0 GB  71%
  Pressure:  normal
  Thermal:   nominal  (100%)

Inference  ollama 0.17.4
  Models loaded: 1  VRAM total: 26.0 GB

  Model                                        VRAM   Format  Quant
  ──────────────────────────────────────── ────────── ──────── ──────
  qwen3.5:35b-a3b                            26.0 GB     gguf Q4_K_M

Options:

-w, --watch SEC            Refresh every SEC seconds
-q, --quiet                Collect and store without output (for daemon use)
    --json                 Output as JSON (for scripting)
-H, --history PERIOD       Show history (e.g. 24h, 1h)
-a, --analyze HOURS        Comprehensive analysis with trends
-c, --compare TS TS        Compare two timestamps
    --alert-webhook URL    POST alerts on state transitions (memory, thermal, engine down)

Diagnose installation, engines, system health, and database.

$ asiai doctor

Doctor

  System
    ✓ Apple Silicon       Mac Mini M4 Pro — Apple M4 Pro
    ✓ RAM                 64 GB total, 42% used
    ✓ Memory pressure     normal
    ✓ Thermal             nominal (100%)

  Engine
    ✓ Ollama              v0.17.4 — 1 model(s): qwen3.5:35b-a3b
    ✓ LM Studio           v0.4.5 — 1 model(s): qwen3.5-35b-a3b
    ✗ mlx-lm              not installed
    ✗ llama.cpp            not installed
    ✗ vllm-mlx            not installed

  Database
    ✓ SQLite              2.4 MB, last entry: 1m ago

  5 ok, 0 warning(s), 3 failed

Line up each engine's running, installed, and available versions and flag what's behind — including the post-upgrade trap where a live process predates the binary you just upgraded (running-stale).

asiai versions                   # offline: running/installed + brew outdated
asiai versions --check-upstream  # also query PyPI / GitHub (network, opt-in)
asiai versions --engine llamacpp # filter to one engine
asiai versions --json | jq

Engine versions

  ENGINE     RUNNING  INSTALLED  AVAILABLE  STATUS
  ─────────  ───────  ─────────  ─────────  ─────────────────
  llama.cpp  9370     9370       9380       upgrade-available
  Ollama     —        0.24.0     0.24.0     up-to-date

  1 upgrade(s) available

asiai doctor carries an offline recap of this, and asiai web exposes a /versions page with changelog links. Triggering an upgrade is a write and lives in aisctl upgrade <engine> (see docs/versions-mode.md).

Background monitoring via macOS launchd. Collects metrics every minute.

asiai daemon start              # Install and start the daemon
asiai daemon start --interval 30  # Custom interval (seconds)
asiai daemon status             # Check if running
asiai daemon logs               # View recent logs
asiai daemon stop               # Stop and uninstall

Web dashboard with real-time monitoring, benchmark controls, and interactive charts. Requires pip install asiai[web].

asiai web                    # Opens browser at http://127.0.0.1:8899
asiai web --port 9000        # Custom port
asiai web --host 0.0.0.0     # Listen on all interfaces
asiai web --no-open          # Don't auto-open browser

Features: system overview, engine status, live benchmark with SSE progress, history charts, doctor checks, dark/light theme.

Multi-host management across several Macs. Two phases ship today:

• Phase 1 — read-only observability (in asiai). Each remote Mac runs asiai web --host 0.0.0.0; the orchestrator declares the nodes in ~/.config/asiai/fleet.json and polls each one's /api/v1/snapshot in parallel.
• Phase 2 — authenticated writes (Bearer auth in asiai, aisctl serve + aisctl fleet push in asiai-inference-server). Issue purge, stop/start/restart, unload, install/uninstall, upgrade against remote nodes with rate-limited token auth and a per-call audit log.

# --- Read-only (Phase 1) -------------------------------------------
asiai fleet add studio --url http://192.0.2.10:8899 --role workstation
asiai fleet list
asiai fleet status               # parallel poll, aggregated table
asiai fleet status --json | jq   # machine-readable form
asiai fleet ping studio          # single-node check

# --- Writes (Phase 2) ----------------------------------------------
# On the node: initialize the auth surface (prints secret ONCE).
asiai auth init
# On the node: start the loopback companion that runs the commands.
aisctl serve &
# On the orchestrator: register the node with its secret.
asiai fleet add studio --url http://192.0.2.10:8899 --auth-token asai_...
# Issue a write.
aisctl fleet push studio purge
aisctl fleet push studio restart --engine ollama
aisctl fleet push studio unload --engine ollama --model llama3.2

The /fleet page in asiai web is a master-detail cockpit: a card per node with its engines' live lifecycle state, and — once you log in as operator — the write buttons (start/stop/restart, standby/enable, purge, load/unload, install/uninstall) with a type-to-confirm gate on destructive verbs and an audit-journal drawer. The /dashboard and /monitor pages show the whole fleet read-only (a ribbon of aggregates plus a block per node); /monitor adds live token counters and power.

The node-to-node Bearer authenticates machines. A human clicking a write button in the cockpit authenticates separately, with an ephemeral shell-bound code — no password is ever stored:

# In a trusted shell ON the machine running `asiai web`:
asiai auth login                 # prints a single-use code (aop_…, ~60s TTL)
asiai auth login --scope audit:read   # a code that ONLY reads the audit journal

Paste the code into the dashboard's login form (/login). You get a server-side session behind an HttpOnly; SameSite=Lax cookie with per-form CSRF; a process restart or logout ends it. The write funnel is unchanged — the browser session only lets the same-origin proxy forward on your behalf. The audit:read scope is bound to the code at mint: it can be exchanged for one redacted journal read (used by the MCP fleet_audit_tail tool) but can never open a write session.

The dashboard write bind stays on loopback + SSH-forward + the trusted mesh interface; it is never exposed publicly. See docs/fleet-mode.md for the full model.

Phase 3 will add mDNS Bonjour auto-discovery and TLS. Full guide: docs/fleet-mode.md.

⚠️ Phase 1 is unauthenticated read-only; Phase 2 requires Bearer tokens. Both are designed for trusted LANs (or LANs glued together by a VPN like Tailscale/WireGuard). No TLS is enforced between nodes in v1.8 — that's a Phase 3 deliverable.

Browse community benchmarks. Filter by chip or model.

asiai leaderboard                      # All results
asiai leaderboard --chip "M4 Pro"      # Filter by chip
asiai leaderboard --model qwen2.5      # Filter by model

Compare your local results against community medians.

asiai compare --chip "Apple M1 Max" --model qwen2.5:7b

Get engine recommendations based on your hardware and benchmarks.

asiai recommend                                # Best engine for your Mac
asiai recommend --use-case latency             # Optimize for TTFT
asiai recommend --model qwen2.5 --community    # Include community data

Interactive setup wizard — detects hardware, engines, models, and suggests next steps.

asiai setup

Start the MCP server for AI agent integration. 14 tools, 3 resources.

asiai mcp                          # stdio (Claude Code, Cursor)
asiai mcp --transport sse          # SSE (network agents)

Interactive terminal dashboard with auto-refresh. Requires pip install asiai[tui].

asiai tui

Generate a shareable benchmark card image with one flag:

asiai bench --card                    # SVG saved locally (zero dependencies)
asiai bench --card --share            # SVG + PNG via community API
asiai bench --quick --card --share    # Quick bench + card + share

A 1200x630 dark-themed card with your model, chip, specs banner (quantization, RAM, GPU cores, context size), engine comparison bar chart, winner highlight, and metric chips (tok/s, TTFT, power, engine version). Optimized for Reddit, X, Discord, and GitHub READMEs.

Every shared card includes asiai branding — the Speedtest.net model for local LLM inference.

All metrics stored in SQLite (~/.local/share/asiai/metrics.db) with 90-day retention and automatic regression detection.

Following MLPerf, SPEC CPU 2017, and NVIDIA GenAI-Perf standards:

• Warmup: 1 non-timed generation per engine before measured runs
• Runs: 3 iterations per prompt (configurable), median as primary metric
• Sampling: temperature=0 (greedy decoding) for deterministic results
• Power: Always-on via IOReport (no sudo). Per-engine, not session-wide average
• Variance: Pooled intra-prompt stddev (isolates run-to-run noise)
• Metadata: Engine version, model quantization, hardware chip, macOS version stored per result

See docs/benchmark-best-practices.md for the full conformance audit.

Four standardized prompts test different generation patterns:

Use --context-size 4k|16k|32k|64k to test with large context fill prompts instead.

When running asiai web, three REST API endpoints are available for programmatic access. Interactive API documentation (Swagger UI) is available at http://localhost:8899/docs.

# prometheus.yml
scrape_configs:
  - job_name: 'asiai'
    static_configs:
      - targets: ['localhost:8899']
    metrics_path: '/api/metrics'
    scrape_interval: 30s

asiai monitor --json | jq '.mem_pressure'
asiai models --json | jq '.engines[].models[].name'

• macOS on Apple Silicon (M1 / M2 / M3 / M4 families)
• Python 3.11+
• At least one inference engine running locally

The core uses only the Python standard library — urllib, sqlite3, subprocess, argparse. No requests, no psutil, no rich. Just stdlib.

Optional extras:

• asiai[web] — FastAPI web dashboard with charts
• asiai[tui] — Textual terminal dashboard
• asiai[all] — Web + TUI
• asiai[dev] — pytest, ruff

Apache 2.0