feat: Close the remaining b10621 prompt-cache and slot-scheduling gaps #1473

Part of #1431.

Problem / Background

This continues #1453 (merged as PR #1468) and carries the eleven entries in compat/llama-server/b10621/runtime-and-context.json that #1453 left in state deferred. PR #1468 landed the b10621 spellings and environment bindings for --cache-prompt, --cache-ram, --cache-reuse and --cont-batching, made --no-cont-batching actually pin the decode width, honored per-request cache_prompt on the chat-shaped routes, and added the differential end-to-end gate in tests/prompt_cache_compat_e2e.rs. It also recorded, entry by entry, every way the result still differs from llama-server b10621 in externally observable behavior. That work is merged, correct, and truthfully classified: deferred is epic #1431's fourth compatibility state, "a linked implementation issue exists with explicit acceptance criteria; the current binary must not claim support by silently ignoring it". This issue is the tracked continuation of that state, not a regression report against PR #1468.

The reason it has to exist as its own issue is mechanical. scripts/ci/check_llama_compat_manifest.py --check-issues-open resolves every deferred entry's issue through gh issue view and fails when it is not OPEN (scripts/ci/check_llama_compat_manifest.py:445-468 and scripts/ci/check_llama_compat_manifest.py:624-640), and that flag is what the llama-compat manifest CI job passes (.github/workflows/ci.yml:244-275). Closing #1453 while these eleven entries still point at it turns that job red on every open pull request in the repository, so #1453 is currently reopened at status:in-progress as a stopgap. Repointing the entries here lets #1453 close against what it actually delivered.

Current Behavior

Eleven entries. Every divergence bullet below is quoted verbatim from the manifest as it stands today; if the implementation restates rather than removes one, that is the text it is restating. Two of the eleven are the substantive gaps and are worth reading before the rest.

--cache-reuse cannot be expressed at all today. It refuses a positive minimum chunk size at startup because no KV operation re-bases the rotary positions of a cached span. Upstream reuses a non-prefix chunk by deleting the span between the divergence and the resumption point and shifting the rotation of everything after it back down; mlxcel's prompt cache reuses a strict token prefix. The gap is a missing operation, not a missing switch, and it is the largest single piece of work in this issue.

--cache-prompt controls a cache that covers the chat-shaped routes only. /v1/chat/completions, /v1/messages and /v1/responses look up and donate prompt prefixes; /v1/completions and /completion are never cached whatever the flag says, because nothing on those paths builds a prompt-cache request context. That is a coverage gap in the feature itself, not in the flag, and closing it is what would let the entry leave deferred.

The two substantive gaps, and their request-field siblings

--cache-reuse

Declared at src/cli/cache_args.rs:88-95 with LLAMA_ARG_CACHE_REUSE. CacheCompatArgs::check_cache_reuse (src/cli/cache_args.rs:183-207) accepts 0 and refuses any positive value with a diagnostic naming the missing operation. The operation really is missing, not merely unwired: KVCache::trim_front_keep_sink (src/lib/mlxcel-core/src/cache.rs:2516) advances live_start and deliberately leaves offset alone, and gather_positions (src/lib/mlxcel-core/src/cache.rs:2675) and gather_within_tail (src/lib/mlxcel-core/src/cache.rs:2772) compact surviving slots without re-rotating them.

Recorded divergence:

• a positive minimum chunk size is refused at startup instead of enabling KV-shift reuse; only the upstream default of 0 is accepted
• no KV operation re-bases the rotary positions of a cached span, so chunk reuse cannot be expressed at all

field:n_cache_reuse

Not declared on NativeCompletionRequest (src/server/types/request.rs:1091), which does not set deny_unknown_fields, so serde drops it silently.

Recorded divergence:

• the field is not accepted: KV-shift chunk reuse does not exist, so a per-request threshold has nothing to tune

--cache-prompt

Declared at src/cli/cache_args.rs:62-85 with LLAMA_ARG_CACHE_PROMPT and upstream's enabled default; resolved at src/cli/cache_args.rs:162-181. The cache it switches is reached through build_prompt_cache_request_context (src/server/routes/chat.rs:103-155), which is called from src/server/routes/chat.rs:401, src/server/routes/chat.rs:790, src/server/routes/responses.rs:243, src/server/routes/responses.rs:357, src/server/routes/anthropic.rs:220 and src/server/routes/anthropic.rs:344, and from nowhere in src/server/routes/completions.rs or src/server/routes/native_completion.rs.

Recorded divergence:

• the prompt cache it controls covers the chat-shaped routes only; /v1/completions and /completion are never cached whatever the flag says
• --cache-prompt cannot re-enable the cache over mlxcel's own --no-prompt-cache; between two explicit operator statements the disable wins

field:cache_prompt

Honored on the chat-shaped routes through resolve_cache_prompt (src/server/chat_request.rs:179) and the early return at src/server/routes/chat.rs:111-121. Absent from NativeCompletionRequest.

Recorded divergence:

• accepted on the chat-shaped routes only; native /completion neither declares the field nor caches prompts, so a client sending it there is silently unaffected
• true asserts the default and cannot re-enable the cache for one request against a server-wide disable

Budget and batching

--cache-ram

Declared at src/cli/cache_args.rs:101-104 with LLAMA_ARG_CACHE_RAM; resolve_cache_ram (src/cli/cache_args.rs:210-230) reads MiB with upstream's -1 and 0 sentinels onto the same byte budget as --prompt-cache-capacity-bytes, which wins when both are given.

Recorded divergence:

• the unset default is 2048 MiB (mlxcel's --prompt-cache-capacity-bytes default); b10621 defaults to 8192
• the budget bounds the chat-route prompt-prefix cache; /v1/completions and /completion hold no cached prefixes to bound
• -cram is not accepted: a single-dash multi-character alias has no clap representation

--cont-batching

Declared at src/cli/cache_args.rs:105-137 with LLAMA_ARG_CONT_BATCHING; --no-cont-batching resolves to --max-batch-size 1 at src/cli/cache_args.rs:173, deliberately not to mlxcel's stronger --no-batch.

Recorded divergence:

• -cb and -nocb are not accepted: a single-dash multi-character alias has no clap representation
• disabling pins the decode width to one sequence but leaves chunked prefill, the prompt cache and speculative decoding in place, where llama.cpp's slot loop stops interleaving outright

Slot and checkpoint state that does not exist

Five entries that are not accepted at all. Each one tunes per-slot retained prompts or a ring of context checkpoints, and mlxcel has neither, so accepting any of them inert would claim a policy that does not exist. --slot-prompt-similarity makes the point sharpest: its upstream default is 0.10, not 0, so accepting it inert would mean honoring a script that passes the upstream default while no slot-selection policy exists to honor.

--cache-idle-slots

Not declared. There is no per-slot retained prompt to save: /slots is synthesized from batch_metrics.active_count() and queue_depth() (src/server/routes/slots.rs:29-70) rather than reported from slot objects.

Recorded divergence:

• no flag: there is no per-slot retained prompt to save on a new task or clear on unified KV

--slot-prompt-similarity

Not declared. Reuse goes through a process-wide radix trie over token prefixes (src/server/prompt_cache/trie.rs, src/server/prompt_cache/lookup.rs, src/server/prompt_cache/store.rs) consulted for every request regardless of which sequence last held the tokens.

Recorded divergence:

• no flag: reuse is a process-wide prefix trie, not a scan of what each slot is holding, so there is no slot prompt for a similarity threshold to compare against
• slots are presentation-only (synthesized in /slots from the active count and queue depth) and hold no prompt state

--kv-unified

Not declared. KV is allocated per sequence through the cache pool the scheduler owns (src/server/batch/scheduler.rs, src/server/batch/sequence.rs).

Recorded divergence:

• no flag: KV is allocated per sequence and there is no unified shared buffer to switch to

--ctx-checkpoints

Not declared. capture_history_boundary_snapshot (src/server/batch/scheduler.rs:2873) takes at most one snapshot per sequence, at the prompt/generation boundary.

Recorded divergence:

• no flag: at most one history-boundary snapshot is captured per sequence, not a ring of per-slot context checkpoints
• -ctxcp and --swa-checkpoints are not accepted either

--checkpoint-min-step

Not declared, and there are no successive checkpoints for a spacing to separate; see --ctx-checkpoints.

Recorded divergence:

• no flag: there are no successive context checkpoints for a minimum spacing to separate
• -cms is not accepted either

Proposed Solution

Four pieces, and only the first two are open-ended. Each ends by rewriting the manifest entries it covers into a terminal state; none of them may leave an entry supported with a non-empty divergence, which the validator rejects outright (scripts/ci/check_llama_compat_manifest.py:247-275).

1. Extend the prompt cache to the non-chat routes (--cache-prompt, field:cache_prompt). The cache itself is route-agnostic; what is chat-only is the construction of the request context. Give /v1/completions (src/server/routes/completions.rs) and native /completion (src/server/routes/native_completion.rs) a prompt-cache request context built from the same build_prompt_cache_request_context seam (src/server/routes/chat.rs:103-155), keyed on the raw prompt tokens rather than on a rendered chat template, and declare cache_prompt on NativeCompletionRequest so the per-request opt-out works there too. Extract the seam rather than copying it: it is the single handle the scheduler reaches the store through for both the prefix lookup and the donate-back, and a second, slightly different copy is how one route ends up donating entries it never looks up. The differential gate in tests/prompt_cache_compat_e2e.rs must be extended to the new routes, not merely left passing on the old ones.

2. KV-shift chunk reuse (--cache-reuse, field:n_cache_reuse). This needs a new cache operation: remove a span from the middle of a cached KV set and re-base the rotary positions of every key after it. Decide first whether that is a re-rotation of the stored keys (expensive, exact) or a position-offset the attention path applies at read time (cheap, invasive), and say which in the issue before writing code. If the decision is that neither is worth it, the honest terminal state is not_applicable for both entries with the existing refusal diagnostic and its test, which is a legitimate outcome under the epic's policy and better than a partial implementation. Whichever way it goes, field:n_cache_reuse follows --cache-reuse: a per-request threshold with nothing to tune is worse than an absent field only if it is accepted silently.

3. Budget and batching (--cache-ram, --cont-batching). Both act correctly already; what remains is the recorded divergences. --cache-ram differs in unset default (2048 MiB against upstream's 8192) and inherits the chat-route-only coverage from piece 1, so it may resolve as a side effect of that work plus an explicit decision on the default. --cont-batching differs in that disabling it leaves chunked prefill, the prompt cache and speculative decoding in place where llama.cpp's slot loop stops interleaving outright; decide whether that is close enough to be supported, or whether it stays recorded. Both also carry the single-dash alias divergence below.

4. The five slot and checkpoint entries. These are classification work, not implementation work, unless per-slot state is being built. Per-slot retained prompts and a /slots view that reports real slot objects are #1440's territory in phase 4, and #1440 depends on #1453; if it lands first, --cache-idle-slots and --slot-prompt-similarity become implementable and should be revisited then. Otherwise the terminal state for all five is not_applicable, which the validator requires to name a diagnostic or documentation test and to explain itself in notes (scripts/ci/check_llama_compat_manifest.py:307-311). "Not accepted at all" is not by itself a diagnostic: an unknown-argument error from clap does not tell an operator why, so reaching not_applicable here means adding a hidden argument that rejects with an explanation, or pointing at the documentation test that carries it.

Scope

In scope: the eleven manifest entries listed above and whatever code each terminal state requires. Likely touch points: src/server/routes/completions.rs and src/server/routes/native_completion.rs (prompt-cache coverage), src/server/routes/chat.rs (extracting the request-context seam), src/server/types/request.rs (cache_prompt, and n_cache_reuse if it is accepted), src/lib/mlxcel-core/src/cache.rs (a span-removal-with-re-rotation operation, if KV-shift reuse is implemented), src/cli/cache_args.rs (defaults, diagnostics), tests/prompt_cache_compat_e2e.rs, compat/llama-server/b10621/runtime-and-context.json, and docs/llama-server-compat.md.

Out of scope: the thirteen context, batching and RoPE entries that #1450 left deferred, which are the sibling follow-up #1472; real per-slot state, /slots and slot persistence, which are #1440; control-vector semantics, which are #1449 and share this shard's owner set. Do not change the pinned b10621 reference or compat/llama-server/b10621/pin.json beyond what shard ownership requires.

Implementation Notes

• Reuse: build_prompt_cache_request_context (src/server/routes/chat.rs:103-155) is the one seam through which a request reaches the prompt-cache store, for both the lookup and the donate-back; extend it rather than adding a parallel path. src/server/routes/native_completion.rs:415-435 is the established pattern for a native request field that is declared and then rejected with a named diagnostic (n_cmpl, n_indent, t_max_predict_ms). src/cli/llama_short_flags.rs:60-98 is the argv pre-pass and SHORT_ALIASES table that already translates llama.cpp single-dash multi-character tokens into long spellings.
• Cross-cutting design question: four of the divergences above reduce to one structural limitation, that a single-dash multi-character alias (-cram, -cb, -nocb, -cms, and likewise -ctxcp, -kvu, -no-kvu, -sps) has no clap representation, and the same limitation reaches the sibling issue fix: Close the remaining b10621 context and batching gaps #1472's options. src/cli/llama_short_flags.rs already carries the table and the pre-pass built for exactly this, so the question is whether to extend that table once for every remaining b10621 short token or to handle each option where it is declared. Decide it before either issue starts; solving it twice, differently, in two concurrent branches is the outcome to avoid.
• The issue number is in a shipped diagnostic: src/cli/cache_args.rs:205 ends the --cache-reuse refusal with "Tracked by feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453.", and src/cli/cache_args_tests.rs:106 asserts that string is present. Once feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453 closes, an operator who hits that message is sent to a closed issue. Repoint the message and the assertion to this issue as part of whatever change touches the flag, and do the same for docs/llama-server-compat.md:97, whose shard-ownership table names fix(inference): align context, batching, RoPE, and runtime override semantics #1450, feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453 and feat(inference): implement or explicitly reject control-vector options #1449. The sibling issue fix: Close the remaining b10621 context and batching gaps #1472 has the same problem at src/cli/rope_args.rs:223, src/models/rope_overrides.rs:142 and src/cli/rope_args_tests.rs:56.
• Constraints: the manifest is machine-checked. A supported entry must carry an mlxcel claim, name a conformance test, accept its canonical spelling and record b10621's env binding, and have an empty divergence. An aliased entry must record an mlxcel mapping onto a DIFFERENT identity plus a translation test, and for a request field the mlxcel field name must differ from the b10621 one. A not_applicable entry must name a diagnostic or documentation test and explain itself in notes. An entry that stays deferred must point at an OPEN issue that is a member of pin.json's shards["runtime-and-context"].owners. src/server/llama_compat_tests.rs:208-244 additionally asserts the field claim set in both directions: a field the manifest does not claim must NOT exist on NativeCompletionRequest, so declaring cache_prompt or n_cache_reuse there without flipping the entry fails CI.
• Edge cases: a /completion request whose prompt is a prefix of a cached chat prefix (the two routes must not adopt each other's entries unless the key genuinely covers the rendering difference); cache_prompt: false on a route that now caches, which must both skip the lookup and skip the donate-back; --cache-ram 0 (disable) interacting with --cache-prompt explicitly enabled; --no-prompt-cache against --cache-prompt (the existing rule is that either disable wins, and it must survive the coverage change); a KV-shift reuse that lands on a rotating or quantized cache, where the sidecars must be re-indexed with the keys; concurrent requests reusing the same span while one of them rewrites its rotation.
• Error handling: startup rejections keep the existing shape, one diagnostic naming the flag, the value and the missing operation, as check_cache_reuse does today. Per-request rejections return the native error envelope from src/server/routes/native_completion.rs. Nothing added by this issue may be accepted and ignored.

Acceptance Criteria

Each entry below either leaves deferred for the terminal state named, or stays deferred with its remaining divergence restated against evidence gathered in this issue (a test, a differential run, or a named missing operation), not merely carried over.

• --cache-prompt: supported once /v1/completions and /completion are covered by the same cache, or deferred with the coverage boundary restated and measured.
• field:cache_prompt: supported on every route the cache now covers, including native /completion.
• --cache-reuse: supported with a positive chunk size producing real KV-shift reuse, or not_applicable with the refusal diagnostic and its test after an explicit decision that the operation will not be built.
• field:n_cache_reuse: the same terminal state as --cache-reuse, never a different one.
• --cache-ram: supported after the unset default and the coverage divergence are both resolved, or aliased onto --prompt-cache-capacity-bytes with a translation test.
• --cont-batching: supported, or deferred with the slot-interleaving difference restated against a measured comparison rather than an inference.
• --cache-idle-slots, --slot-prompt-similarity, --kv-unified, --ctx-checkpoints, --checkpoint-min-step: each not_applicable with a diagnostic or documentation test and a notes explanation, or supported if per-slot state lands from feat(server): align props, slots, metrics, health, and slot persistence #1440 first. An unknown-argument error from clap does not satisfy the diagnostic requirement.
• No entry in compat/llama-server/b10621/runtime-and-context.json is supported with a non-empty divergence, and every entry still deferred points at an OPEN issue that is a member of that shard's owner set.
• Every flag this issue declares appears on BOTH mlxcel serve and mlxcel-server with the same spelling, value domain, default and env binding, verified through --dump-flag-surface rather than --help.
• tests/prompt_cache_compat_e2e.rs covers every route the cache now serves, and its differential assertion holds: a cached run's per-position tokens and logprobs equal the cold run's at the same batch width. A cache that is merely fast is not a passing cache.
• Integrated into the real request path, not a parsed-and-stored setting: a coverage change is observable as a cache hit on a route that previously had none, and a reuse change is observable as fewer prefilled tokens for a prompt that diverges in the middle.
• No shipped diagnostic or documentation table still points an operator at a closed issue number.
• docs/llama-server-compat.md states each entry's terminal classification, and any default that now differs from mlxcel's previous behavior is called out as a migration note.

Verification

make verify-llama-compat
python3 scripts/ci/check_llama_compat_manifest.py --check-issues-open
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --profile test-fast --features metal,accelerate

The manifest half of the suite is tests/llama_compat_manifest.rs (claims against both binaries' flag surfaces) and src/server/llama_compat_tests.rs (routes and native request fields in both directions). --workspace is not optional: a bare cargo test at the root resolves to -p mlxcel and skips three of the four members.

The prompt-cache gate needs a real checkpoint and is not part of the unit run:

cargo test --profile test-fast --features metal,accelerate --test prompt_cache_compat_e2e -- --nocapture

A pass is what docs/llama-server-compat.md:265-269 describes: the same-width arms (cache hit, per-request disable) agree with the cold run on every position exactly, and the concurrent arm's first divergence sits below the reference's own top-two gap, which is the batch-width jitter class rather than a cache defect. Extending the cache to a new route means extending these arms to that route; leaving the test passing on the chat routes alone proves nothing about the change.

Manual check for the coverage change, against a real checkpoint:

./target/release/mlxcel-server --model models/Qwen3-4B-4bit --cache-prompt --parallel 1
curl -s localhost:8080/completion -d '{"prompt":"<a long shared prefix> A","n_predict":8}'
curl -s localhost:8080/completion -d '{"prompt":"<the same long shared prefix> B","n_predict":8}'
curl -s localhost:8080/metrics | grep -i cache

A pass is: the second request reports a non-zero cached prefix where it reports zero today, and its generated tokens match a run of the same request against a server started with --no-cache-prompt.

Technical Considerations

• pin.json's shards["runtime-and-context"].owners is currently [1449, 1450, 1453], and the validator requires every entry's issue to be a member of its shard's owner set (scripts/ci/check_llama_compat_manifest.py:423-442). The commit that repoints these entries to this issue also adds this issue number there. That commit is the orchestrator's; do not duplicate it here.
• This shard is shared with feat(inference): implement or explicitly reject control-vector options #1449 and with fix: Close the remaining b10621 context and batching gaps #1472, the sibling follow-up. Two concurrent chains editing one shard is a gate failure by design, not just a merge conflict, so coordinate the ordering rather than rebasing over it.
• feat(server): align props, slots, metrics, health, and slot persistence #1440 (/props, /slots, /metrics, slot persistence) depends on feat(server): implement b10621 prompt-cache and slot-scheduling semantics #1453 and is where per-slot state would come from. If the five slot and checkpoint entries are classified not_applicable here, feat(server): align props, slots, metrics, health, and slot persistence #1440 should revisit that classification rather than inherit it.
• b10621 reference for the option semantics: https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/common/arg.cpp and https://github.com/ggml-org/llama.cpp/blob/c1d0e7a004015f23bc0233470b747b596f29b264/tools/server/server.cpp . The prompt-cache RAM budget is server : host-memory prompt caching ggml-org/llama.cpp#16391 and the context-checkpoint ring is server : add SWA checkpoints ggml-org/llama.cpp#15293.