Four CLI tools for testing LLM behavior with Producer Pal's MCP tools:

• scripts/eval - Automated evaluation scenarios with scoring and assertions
• scripts/eval --canary - The fragment-sensitive scenario subset, for after you edit a skill fragment (see Measuring a context change)
• npm run probe:skills - Does each fragment / tool / param description still reach the model? Minutes, and needs no Ableton
• scripts/chat - Interactive chat sessions for manual testing

All but probe:skills require Ableton Live running with the Producer Pal device loaded.

dev/Eval-Findings.md records what past runs established — including fixes that were tried and measured as not working. Read it before attacking a scenario that has been failing for a while.

Editing a skill fragment, a tool description, or a param description changes what the model reads. The full suite is the only complete answer, but it costs ~4h and ~130M tokens — too slow to iterate against. Two cheaper gates come first, and neither replaces the other.

1. npm run probe:skills — did the content reach the model at all?

Moving an @include can cost the model a whole fragment even though the text is still in the blob. That is what happened to swing() / quant() / step(), and only a full run caught it. The probe asks one question per fragment, tool and param, answerable only from that source, and takes about a minute:

npm run probe:skills
npm run probe:skills -- -m codex-code/luna                # the model we eval
npm run probe:skills -- -m local/google/gemma-4-26b-a4b   # LM Studio, free
npm run probe:skills -- --small-model                     # basic tier
npm run probe:skills -- --surface param                   # param descriptions only
npm run probe:skills -- -t transforms-expressions         # one source

It runs the real buildSkills() blob, and gets the tool schemas from the real createMcpServer with a stub Live API behind it. Every transport connects to that same server, so an AI-SDK model and codex-cli see identical tools/list output and a difference in results is the model rather than the harness. No Ableton either way. It exits non-zero on a miss, so it works as a gate.

Agent CLIs spawn a subprocess per question and rate-limit under load, so they run one at a time and take a few minutes; AI-SDK providers run in parallel and finish in about one.

It catches content LOSS, not bad behavior — a model can recall legato(tol) perfectly and never use it. Before trusting a new probe, check it fails when the source is absent. Run the standard-tier probes against the basic blob: the fragments basicDriver lacks must MISS. A probe a capable model can answer from the rest of the context is measuring nothing.

2. ./scripts/eval --canary — did behavior change?

./scripts/eval --canary -m codex-code/luna -r 1 --skip-judge  # ~25 scenarios, ~20 min
./scripts/eval --canary -m codex-code/luna -r 3 --skip-judge  # confirm a flagged red

The list is evals/canary-scenarios.txt: every scenario that has ever collapsed between two -r 3 luna runs, plus one witness per standardDriver fragment. -r 1 cannot see 2/3 drift and is not meant to — a bad include position causes total loss, and that is what this gates. Confirm anything it flags at -r 3 on that scenario alone before believing it.

Same caveat as any eval run: it opens Live Sets without saving the current one.

Runs predefined scenarios against Ableton Live and scores the results.

scripts/eval [options]

Models use provider/model format, or just the model name if the provider can be inferred from the prefix:

Only the first / splits provider from model, so a model name can contain slashes of its own: local/qwen/qwen3.8-27b is the qwen/qwen3.8-27b model on the local provider.

# Run all scenarios with a specific model
scripts/eval -a -m gemini-3-flash-preview

# Compare two models on one scenario
scripts/eval -t connect-to-ableton -m gemini-3-flash-preview -m claude-sonnet-4-5

# Compare Codex subscription models (requires `codex login`)
scripts/eval -t connect-to-ableton \
  -m codex-code/sol -m codex-code/terra -m codex-code/luna

# Compare subscription CLIs against each other (requires `codex` and `claude`)
scripts/eval -t connect-to-ableton -m codex-code/terra -m claude-code/sonnet

# Skip Live Set reopening (reuse current MCP connection)
scripts/eval -t connect-to-ableton -s

Local models (Ollama, LM Studio, etc.) need special handling:

1. Always specify the model explicitly with the local/ prefix
2. Enable small-model mode (--small-model) for the basic skills tier and simplified tool descriptions

# Test a local model
scripts/eval -m local/glm-4.7-flash -t connect-to-ableton --small-model

# Test a different local model
scripts/eval -m local/qwen3-8b -t duplicate --small-model

The local provider connects to http://localhost:11434/v1 by default (Ollama). Override with -b / --base-url — both CLIs take it — or set LOCAL_BASE_URL in .env. LM Studio's default port is 1234, so it always needs one of the two:

scripts/eval -m local/qwen/qwen3.8-27b -t connect-to-ableton \
  -b http://localhost:1234/v1 --small-model

claude-code and codex-code are not API providers — they drive an installed coding-agent CLI as a subprocess (claude -p --output-format stream-json, codex exec --json), so a run bills the logged-in subscription instead of a metered API key. The CLI owns the conversation and the MCP connection; each turn is a fresh process that resumes the previous turn's session id.

# Requires `claude` on PATH and a logged-in subscription (`claude auth login`)
scripts/eval -m claude-code/sonnet -t connect-to-ableton

# Requires `codex` on PATH and `codex login`
scripts/eval -m codex-code/terra -t connect-to-ableton

Both transports strip the vendor's API-key environment variables before spawning, so an exported ANTHROPIC_API_KEY / OPENAI_API_KEY cannot silently turn a subscription run into a billed one. Both also run the CLI stripped down to Producer Pal: built-in tools off, settings and plugins off, other MCP servers ignored, and the eval's system instructions REPLACING the CLI's own agent prompt (which is also what keeps the user's memory files out of the run).

On Codex the plugin part takes its own flag, --disable apps. --ignore-user-config does not reach the installed apps, and they come back as MCP tools — a second Producer Pal among them, competing with the eval's server.

A big tool result fails the run on claude-code. Past a size limit the claude CLI saves a tool result to a file and hands the model a 2KB preview instead. The model is what gets the preview, so the run would grade a model that never received the Skills ppal-connect returns. The transport treats that stub as a failed turn rather than let a meaningless score through. It fires on the full toolset today; --tools a subset, or use another provider. No env var raises the limit (MAX_MCP_OUTPUT_TOKENS is a different cap and does not).

Nineteen of the twenty-one published tools fit; all twenty-one does not. Drop library and context and a path scenario runs:

scripts/eval -t path-uncommon-roots -m claude-code/haiku \
  --tools connect,select,read-live-set,read-track,read-clip,read-scene,read-device,create-clip,create-track,create-scene,create-device,update-clip,update-track,update-device,update-scene,update-live-set,duplicate,delete

The cap is on the size of the biggest result, not the tool count, so measure it again after the Skills grow: run connect-to-ableton with the subset you want and see whether the transport rejects the turn.

Give a subset every tool the scenario's RECOVERY path needs, not just the ones it grades. Leaving read-device out of that list is what a model uses to find a drum rack nested inside another rack, and without it the run fails on a check that has nothing to do with the missing tool.

A looping model is bounded the same way it is on the AI SDK path. Neither CLI takes a step limit we can rely on, so the transports count the model's actions (each tool call, each reply) off the event stream and kill the subprocess once the turn goes past the shared budget in evals/shared/step-budget.ts. The run then fails as a blown budget in seconds rather than as a five-minute timeout.

Two caveats when comparing a subscription-CLI run against anything else:

• Token counts are not comparable across transports. Each vendor defines input_tokens differently: Codex reports the total (its cached_input_tokens is a subset of it), while Anthropic reports only the uncached portion, with cache_read_input_tokens / cache_creation_input_tokens alongside. A real claude-code turn that processed ~38k tokens prints tokens: 18 — everything else was a cache read. The mapping deliberately matches the anthropic AI SDK path so the two Anthropic routes agree; it does NOT line up with codex-code.
• Session files outlive the run. Claude Code keys its on-disk session store by working directory, and each eval session uses a fresh temp directory that close() removes. The transcript under ~/.claude/projects/ stays behind, one entry per eval session. (The judge passes --no-session-persistence and leaves nothing; turns cannot, since they resume by session id.)

Point CLAUDE_CODE_BIN / CODEX_BIN at a specific binary when the CLI is not on PATH. The transport tests use the same variables to swap in a fixture that emits canned JSONL, so evals/chat/agent-cli/ is testable with neither CLI installed.

Adding another CLI is a protocol module implementing AgentCliTransport (argv, stream parsing, model names) plus one entry in evals/chat/agent-cli/agent-cli-registry.ts; spawning, session dirs, session-id continuity, turn rendering, and judging are already shared.

A run mirrors the device's settings panel: you choose the environment with CLI flags, then the scenarios run as conversations against it. The environment is server-side state applied before each scenario:

Tests declare what they need via requires (e.g. the transforms DSL, bracket notation, a specific tool, a small-model-excluded param). When the active environment can't satisfy a requirement — e.g. a transforms scenario under --small-model, or a scenario needing a tool you left out of --tools — the scenario is skipped (reported as skipped, not fail) so scores stay apples-to-apples.

A run that never got started is reported as error, and is likewise kept out of pass/fail counts. That means Live would not open the Set or the config would not apply, so the model never took a turn — scoring it would turn an outage into a wall of convincing zeros. The harness retries a failed open three times, waiting once and then killing Live for a cold launch, and stops the whole run after three scenarios in a row fail to start.

# Default environment
scripts/eval -t connect-to-ableton -m gemini-3-flash-preview

# Small-model mode (transforms/bracket scenarios will skip)
scripts/eval -a -m local/qwen3-8b --small-model

# A restricted toolset (scenarios needing other tools will skip)
scripts/eval -a -m gemini-3-flash-preview --tools connect,read-track,create-clip

Know what an environment grades before you pay for the run. --list takes the same environment flags, marks every scenario that environment would skip, and counts what's left:

scripts/eval --list --small-model
# … small-model: grades 52 of 84 (regression 15/25, capability 37/59)

A small-model score is over a much smaller surface than the default run — read it as "of what a small model was given", never as comparable to a default score.

List available scenarios:

scripts/eval -l

Run scripts/eval -l for the current list. Scenarios are tagged as regression (should always pass) or capability (improvement targets, may have low pass rates).

Nearly every scenario opens with "Connect to Ableton Live", and every model answers it the same way: one ppal-connect call, then a sentence acknowledging it. That turn is setup — the behavior under test starts at the next message — but it is the run's most expensive turn, because the connect result (Live Set overview + Producer Pal skills) is re-sent as input on the round trip that produces the acknowledgment.

So the runner writes that turn into the conversation itself, for free. It is not a recording: ppal-connect is called for real over MCP against the Live Set that is actually open, under the run's actual config, so nothing can go stale. Only the assistant's closing sentence is canned, and the model reads the same context either way. Turn numbering is unchanged, so turn: 0 assertions still mean the connect turn.

A scenario is seeded when its first message is the connect message and something follows it. Set seedConnect: false on scenarios that GRADE that turn — its prose, or what it did or didn't write (connect-to-ableton, the context-onboarding-* family). --no-seed-connect disables it for a whole run, which is how to A/B the seeding against real connect turns.

Note that a { type: "tool_called", tool: "ppal-connect", turn: 0 } assertion passes trivially in a seeded scenario. connect-to-ableton is where "does the model reach for ppal-connect" is actually graded.

The agent-CLI providers (claude-code, codex-code) are never seeded: the CLI resumes a session by id and owns its own history, so there is nothing to write into and they fall back to a real connect turn. Worth remembering when comparing one of those runs against any other provider — only one of them paid for that turn.

These assertions decide pass/fail:

• tool_called - Verifies the right tool was called (with optional arg matching). Failed calls don't count — see below.
• state - Verifies Live Set state via MCP tool calls
• custom - Arbitrary callback assertions on turn data

Plus:

• llm_judge - LLM evaluates response quality with pass/fail + issues. It gates the result unless the scenario sets judgeAdvisory: true, which keeps the commentary but stops it flipping a run to fail.
• response_contains - Text/regex patterns in the assistant's prose. Reported as Signals, never gating: the list of acceptable synonyms is unbounded and drifts with every model, so a run that made the right edit and called it "turned those up" instead of "boosted" is not a regression. Pin the outcome with state or custom; keep the pattern for drift signal.
• token_usage - Tracks OUTPUT tokens against a target budget (informational only). Output, not input: the fixed prefix (system prompt, skills, tool schemas) is re-sent on every internal model request, and one conversation turn makes one request per tool-call round trip, so an input total measures the harness rather than the scenario. Set a budget from real runs, not by guess: about 1.3x the median output of a few passing trials, so a normal run reads 60-85% and a run that spirals reads over 100%.

Failed tool calls. A model that hits a tool error, fixes its arguments and calls again still lands the outcome, so it still passes — but the run reports Tool errors and each failed call takes 10% off its score (capped at half). A flat cost, not a share of the calls made: rating the share would pay a model for padding a run with extra successful calls. Grading reads successful calls only: tool_called counts them, and getToolCalls returns them. Use getAllToolCalls when the attempt itself is what's graded ("did it reach for a tool it shouldn't have").

The Score shown per scenario and in the comparison table is the check pass rate (or the trial pass rate under -r N), discounted by that penalty. A clean run outranks a recovered one without either being marked a failure.

The judge defaults to Gemini 3 Flash. Override with -j, or skip it entirely with --skip-judge.

When using -r N, the summary aggregates across trials: checks are totaled, tool errors are summed, efficiency is averaged, and judge shows a pass rate.

Every trial reopens the Live Set, so trial 2 is never graded on trial 1's leftovers. Scenarios that declare reuseLiveSet — they reset whatever they write — skip the reopen and run faster.

Pass -m multiple times to run the same scenarios across models in one run environment. Results are displayed in a comparison table when more than one model is tested.

# 2 scenarios x 2 models = 4 runs, one table
scripts/eval -a -m gemini-3-flash-preview -m claude-sonnet-4-5

To compare environments (e.g. default vs --small-model), do a run per environment and diff them with scripts/eval-report --compare <runId> <runId>.

Each cell tallies passing trials, so a -r 3 run reads 3/3, ~ 2/3, or 0/3 rather than the verdict of one arbitrary trial. The tag on the right compares pass rates between the last two runs: REGRESSION when every passing trial is lost, FIXED when a scenario goes from none passing to some, and WORSE/BETTER for a partial move — usually flakiness rather than a real change.

Interactive chat for manual testing and debugging.

scripts/chat [options] [text...]

Every provider except claude-code and codex-code is supported: those two run through an agent-CLI transport (a spawned claude / codex subprocess), which only the eval CLI drives.

# Quick one-shot test with Gemini
scripts/chat -m gemini-3-flash-preview -1 "list tracks in the set"

# Interactive session with Claude
scripts/chat -m claude-sonnet-4-5

# Test a local model
scripts/chat -m local/glm-4.7-flash -1 "connect to Ableton"

# Local model with custom server URL
scripts/chat -m local/some-model -b http://localhost:1234/v1 -1 "list tracks"

Set these in .env at the project root:

The subscription CLIs take no key. CLAUDE_CODE_BIN and CODEX_BIN override which executable is spawned (see Testing subscription CLIs).

• Ableton Live running with the Producer Pal Max for Live device
• The MCP server must be responsive (eval auto-opens Live Sets and waits for the server)
• Nothing you care about open in Live. A run opens Live Sets without saving the current one, so work in progress is lost.
• API keys configured for the providers you want to test
• For local models: Ollama, LM Studio, or another OpenAI-compatible server running
• For claude-code / codex-code: that CLI installed and logged in

Scenarios are defined in evals/scenarios/defs/. Each file exports an EvalScenario object:

export const myScenario: EvalScenario = {
  id: "my-scenario",
  description: "What this tests",
  kind: "regression",
  liveSet: "basic-midi-4-track", // from evals/live-sets/
  messages: ["Connect to Ableton Live", "Do something specific"],
  assertions: [
    { type: "tool_called", tool: "ppal-connect", turn: 0 },
    // Non-gating drift signal — the state check below is what grades the run.
    { type: "response_contains", pattern: /expected/i },
    {
      type: "state",
      tool: "ppal-read-track",
      args: { trackIndex: 0 },
      expect: { name: "Drums" },
    },
  ],
};

Register new scenarios in evals/scenarios/defs/index.ts and evals/scenarios/load-scenarios.ts.

• Every scenario costs a full run. Each one needs Ableton Live, opens a Live Set, and adds minutes to the suite — and the suite is already long enough that most runs are a filtered subset, not the whole thing. Add a scenario when you find a bug, ship a tool, or need to compare models on something specific; don't add one for coverage's sake.
• Fold a new case into an existing scenario when it fits. An extra turn on a scenario that already opened the right Live Set is far cheaper than a new scenario, and often reads better. Keep it separate when the new case must be measured UNPRIMED — a reach-for probe (which API/idiom does the model pick unprompted?) is worthless once an earlier turn has shown it the answer.
• Default to no judge. tool_called, state, and custom are fast, cheap, and reproducible; a judge costs an LLM call per scenario and miscounts anything musical. Add llm_judge only when the thing being graded is the assistant's PROSE and no state check can see it — did it offer, did it re-ask, did it accept a no. If deterministic checks already pin the outcome and you only want the commentary, mark it judgeAdvisory: true.
• Grade outcomes, not paths. Assert on the final state (e.g., "clip has these notes") rather than the exact sequence of tool calls. This avoids penalizing models that find valid alternative approaches. Grading words is the same mistake one level down, which is why response_contains never gates.
• Keep messages unambiguous. Vague prompts create flaky evals. If a scenario fails at 0%, suspect the prompt before the model.
• Regression vs capability: Tag scenarios as kind: "regression" when they should always pass (use these to catch regressions). Tag as kind: "capability" for aspirational tests that target difficult tasks — these start with low pass rates and graduate to regression once stable.
• Use -r N to diagnose flakiness. If a regression eval fails intermittently, run it 3 times to confirm whether it's flaky or broken before investigating.