CLI vs server: divergent default behavior MUST be fixed (max_tokens, permissive) #130

TL;DR

The CLI and the server MUST behave identically. Same defaults, same flags, same escape hatches, same failure modes. There is no acceptable reason for the server to apply a max_tokens cap and the CLI to skip it, or for the server to hard-code permissive: false while the CLI exposes --permissive. This is a parity bug across two surfaces, not a "discussion point".

This ticket replaces the original discussion-style framing. Concrete actions are below the analyses.

What is broken (v1.3.1)

Code refs (commit 10650d8, v1.3.1):

• CLI builds SessionOptions at main.swift:141-149
• Server builds SessionOptions at Handlers.swift:78-86
• permissive → SystemLanguageModel guardrails at Session.swift:44-48
• The number 512 lives at BodyLimits.swift:17

Six Thinking Hats: is 512 the right cap?

Franz called out the arbitrary "500-something". Honest analysis:

⚪ White Hat (facts only)

• Context window: 4096 tokens, input + output combined.
• defaultOutputReserveTokens (used by the context trimmer) is 512.
• I matched the new defaultMaxResponseTokens to 512 for "architectural consistency" — but the trimmer's reserve and the response cap are two different concepts. The trimmer's 512 reserves space for output; the cap limits how much output the model produces. They can coincide but should not be coupled by definition.
• Typical short instruction reply: 20-200 tokens. Typical paragraph: 200-400. Structured JSON: 300-1500. Code block: 500-2000. Long-form: 1000+.
• OpenAI's own historical default for max_tokens was 16 (and is now effectively the model context max minus the prompt). Anthropic requires it explicitly. There is no industry standard.

🔴 Red Hat (intuition / feelings)

• 512 feels arbitrary because it is arbitrary — picked for code symmetry, not for user need.
• It conflates "trim reservation" with "response cap".
• It will surprise users whose first useful response gets cut at 512 tokens with finish_reason: length.

⚫ Black Hat (risks)

• Silent truncation for any reply longer than 512 tokens. Users who didn't read the new README section will think the model is dumb.
• Coupling bug: if anyone ever changes defaultOutputReserveTokens for trimmer reasons, they will silently change the response cap too. Bad coupling.
• Doesn't actually solve the real problem, which is "the on-device model has no graceful overflow recovery". A cap at 512 only papers over the symptom.
• Hostile to good use cases: structured JSON output, code generation, summarisation — all routinely > 512 tokens.

🟡 Yellow Hat (benefits)

• Eliminates the 50-second hang from Server should cap output at a sensible default when max_tokens is not provided #128.
• Sets finish_reason: "length" so clients can detect truncation and ask for more.
• Conservative — under 512 covers the majority of chat-style replies.

🟢 Green Hat (alternatives, in roughly increasing engineering effort)

1. Bigger fixed default. 1024 or 2048. Covers more legitimate cases. Still safe for typical prompts.
2. No default; reject the request. Return 400 with a message: "max_tokens is required because the on-device model cannot recover from context overflow". Forces the client to make the call. Honest about the limit.
3. Dynamic default: max(64, 4096 − estimatedInputTokens − safetyBuffer). Uses the entire remaining window. Requires TokenCounter on the input before sampling — already available.
4. Per-mode default: 512 for plain chat, larger for response_format=json_object, larger for tool-calling continuations.
5. Decouple from defaultOutputReserveTokens: even if the value stays 512, name it for what it is (responseCap) and document why, so future contributors don't think they are the same thing.
6. Fix the root cause: catch the overflow in the streaming path, end the stream cleanly with finish_reason: "length", and return what was generated up to that point. Then the cap becomes optional rather than load-bearing.

🔵 Blue Hat (process)

• The 512 was a 5-minute patch for a real production bug. It deserves a proper second pass.
• Whatever value we pick, the CLI must apply the same one.
• The decision should be backed by a microbenchmark across short / medium / long prompts so the number has a basis.

Alternatives analysis: what should the cap actually be?

Recommend (3) dynamic default as the right long-term answer, with (5) decoupling the constant as a no-brainer immediate cleanup. Rationale:

If we keep a fixed number for now, 1024 is a defensible compromise — covers typical structured JSON and short-to-medium replies, still leaves 3072 tokens for input.

Permissive mode: what must happen

Server clients have NO way to enable permissive guardrails today. This is a real product gap, not a missing knob — Apple's default guardrails block plenty of benign creative-writing, technical-explanation, and code prompts that the CLI handles fine with --permissive.

The server MUST expose this. Two compatible mechanisms; we should ship both:

1. x_permissive: true request extension — per-request opt-in on the JSON body. Mirrors the existing x_context_* extensions. Cheapest change, immediate parity for OpenAI-SDK callers.
2. --permissive server flag — process-wide opt-in for the rare server operator who wants every request permissive (matches CLI semantics exactly).

Hard-coded permissive: false at Handlers.swift:82 must be deleted in the same PR that ships either of the above.

Actions (the actual work)

In rough order:

• Decouple the constant. Rename defaultMaxResponseTokens to something honest (fallbackResponseCap?) and document that it has nothing to do with defaultOutputReserveTokens. Quick win, lands first.
• Apply the same default to the CLI. Sources/main.swift:143 should use the same fallback the server uses. No more "the CLI streams to stdout in real time" hand-wave — that defence does not hold for piped, scripted, JSON-output, launchd, or large-input invocations. Same bug, same fix, same surface.
• Switch the cap to a dynamic value based on 4096 − estimatedInputTokens − safetyBuffer, with a floor (e.g. 64 tokens) so a too-large input fails loudly rather than silently allowing 0 output. Or, if we keep it fixed for now, raise it to 1024 with explicit rationale in the README.
• Expose permissive mode on the server. Add x_permissive request extension and a --permissive server flag. Delete the hard-coded false at Handlers.swift:82.
• Refactor SessionOptions construction into one shared "apply CLI/server defaults" function. The architectural root cause is that both surfaces hand-roll their own SessionOptions. Today: max_tokens. Tomorrow: the next field someone forgets to wire on one of the two sides.
• Tests asserting parity. A unit test that constructs SessionOptions from "no flags" CLI input and "no fields" server input and asserts they are byte-equal except for the configuration fields (retry, server-only knobs).
• Update the README's Default response cap (max_tokens) section when the above lands. The current text justifies the divergence; that justification needs to disappear.
• Microbenchmark the cap. Measure typical chat / JSON / code response lengths so any future change to the default is data-driven, not vibes.

Acceptance criteria

A future PR that closes this ticket must:

1. Make the CLI and the server apply the same max_tokens fallback (whatever the value), proven by a parity test.
2. Make --permissive reachable from server clients (via extension and/or server flag), proven by an integration test that flips permissive on and verifies a guardrail-loosened reply.
3. Land a single shared applyDefaults(SessionOptions) (or equivalent) so future divergences are mechanically prevented.
4. Update the README to reflect actual parity.

Severity

P1 — Real user impact (silent truncation, no permissive escape hatch on the server, repro of #128 still possible from the CLI). Not a P0 only because both bugs have explicit-flag workarounds. Block any v1.4 release on closing this.

cc @franzenzenhofer

Updated 2026-04-26 after Franz's review on v1.3.1. Original framing was too discussion-heavy; this version states what must happen.