Repository navigation
fix(native-inference): reserve one token for uncapped AOSP generations - #35820
Conversation
The fused AOSP loader resolves an absent maxTokens to the whole remaining context (contextSize - promptTokens) and passed that decode ceiling to the prepared-request guard as its output reservation. The guard's threshold then equals the prompt token count, so every uncapped TEXT_SMALL / TEXT_LARGE call (REPLY rewrite, grounded action reply) failed with MODEL_INPUT_OVER_BUDGET before the stream opened. Reserve the caller's explicit cap when there is one, and otherwise only room to begin decoding, as the iOS native path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mtcow
left a comment
There was a problem hiding this comment.
Reviewed at head 6d25725
The fix addresses a real, unconditional-rejection bug. Before this change the guard reserved args.config.maxTokens (plugins/plugin-native-inference/src/aosp-local-inference-bootstrap.ts:2976), and for an uncapped request config.maxTokens is not a cap at all — it is resolveAospCompletionBudget(...) = contextSize - promptTokenCount (aosp-local-inference-bootstrap.ts:3421-3427, applied to config at :3428-3429). Feeding that into createPreparedModelRequestGuard makes dispatchThresholdTokens = contextWindowTokens - (contextWindowTokens - promptTokenCount) = promptTokenCount (packages/core/src/runtime/prepared-model-request.ts:289-292), and admission rejects on counted.count >= dispatchThresholdTokens (:312-315) — i.e. promptTokenCount >= promptTokenCount, always true. So every maxTokens-less TEXT_LARGE/TEXT_SMALL generation was rejected at guard construction, before any dispatch. Reserving 1 for the uncapped case is the correct shape and mirrors the iOS path.
Findings
plugins/plugin-native-inference/src/aosp-local-inference-bootstrap.ts:2976—outputReserveTokens: args.requestedMaxTokens ?? 1. Traced againstpackages/core/src/runtime/prepared-model-request.ts:279-292: with an explicit cap the reserve equals the caller's cap, sodispatchThresholdTokens = contextSize - requestedMaxTokens; with no cap the threshold becomescontextSize - 1, which is exactly the widest inputresolveAospCompletionBudgetwill ever hand this path (remainingTokens > 0,:560-569). The comment's distinction between "decode ceiling" and "reservation" is accurate.plugins/plugin-native-inference/src/aosp-local-inference-bootstrap.ts:3449— threads the raw caller valueargs.maxTokens, not the resolvedconfig.maxTokens. This is the load-bearing half of the fix: reserving the derived ceiling was the bug, reserving the caller's explicit cap is the intent. It also keepsprojectRequest's embeddedconfig(:2983) consistent, sinceresolveAospCompletionBudgetreturnsrequestedMaxTokensverbatim when it fits (:586). Semantics matchplugins/plugin-native-inference/src/ios/native-generation.ts:42-44(args.requestedMaxTokens ?? 1).plugins/plugin-native-inference/__tests__/aosp-local-inference-bootstrap.test.ts:48-92— the two cases are discriminating. Case 1 (:63-70) assertsoutputReserveTokens: 1/dispatchThresholdTokens: 4095/attempts === 1; on base code the guard would throwMODEL_INPUT_OVER_BUDGETat construction (threshold1000), soassertBeforeAttempt()could never run. Case 2 (:73-92) pins the capped path (outputReserveTokens: 256, threshold3840) and asserts the over-budget throw atpromptTokenCount: 3900. Neither test is a tautology.
Residual notes (pre-existing in kind; not introduced here, no block)
- Off-by-one between the two budget authorities: the guard rejects on
counted.count >= threshold(prepared-model-request.ts:314), so with reserve1onlypromptTokenCount <= contextWindowTokens - 2is admitted, whileresolveAospCompletionBudgetadmitsremainingTokens == 1(aosp-local-inference-bootstrap.ts:560-569). Likewise a capped request whoserequestedMaxTokens === remainingTokensis admitted byresolve(only>throws,:577) but then rejected by the guard because threshold collapses topromptTokenCount. The guard is consistently one token stricter than "room to begin decoding" implies; worth a follow-up assertion so the two seams can't drift. aosp-local-inference-bootstrap.ts:3425usescontextSize ?? 32768while the guard'scontextWindowTokensat:3445-3446usescontextSize ?? readPositiveIntEnv("ELIZA_LLAMA_N_CTX", 4096). Whenactive.contextSizeis unset these two fallbacks disagree, so the resolvedmaxTokensis budgeted against a 32k window while the guard admits against 4k. Not part of this diff, but it is the same "two authorities, two windows" smell.
Live execution blocked: no test runner in the scratch checkout — node_modules, but no node_modules/.bin (no vitest), and npx vitest aborts with EOVERRIDE from the @playwright/test override conflict; a monorepo-wide bun install is out of this run's budget. The findings above are a static trace over the actual head sources (prepared-model-request.ts, aosp-local-inference-bootstrap.ts, ios/native-generation.ts) with exact line references; the arithmetic is deterministic and the new test cases were checked against base behaviour by hand.
Verdict: accept — the root cause (uncapped generations rejected at the guard because the derived decode ceiling was reserved as if it were a cap) is real, the one-token reservation plus threading args.maxTokens is the minimal correct fix, and the new tests are discriminating; the only gaps are pre-existing off-by-one/fallback mismatches that belong in a follow-up rather than this PR.
Compute receipt: 0 project-attributed tokens (unavailable; device-signed, locally reported)
AI provider/model: deepseek / deepseek-v4-flash
Client / agent tooling: review-lane-free
Contribution skill revision: SlopDotCash/slopdotcash@fa576cd:skills/contribute-to-eliza
Attribution status: self-reported
— [review]
MLuber-bot
left a comment
There was a problem hiding this comment.
Independent review — reserve one token for uncapped AOSP generation
Read the diff and aosp-local-inference-bootstrap.ts at head 6d257258.
- In the fused loader's
generate,resolveAospCompletionBudgetreturnscontextSize - promptTokenCountwhen the caller gives nomaxTokens. That value becameconfig.maxTokens, andcreateAospPreparedTextRequestGuardused it asoutputReserveTokens. So for an uncapped request the guard reserved the whole remaining context as output, and the prompt plus that reserve landed exactly at the context size, or over after the guard's dispatch margin. An uncapped call (the PR names REPLY and grounded replies) could therefore be rejected as "exceeds its context budget" even though the prompt fits. - The fix threads the caller's cap through as
requestedMaxTokensand reservesrequestedMaxTokens ?? 1. An explicit cap still reserves its tokens, and uncapped requests reserve just enough to begin decoding.resolveAospCompletionBudgetstill throwsMODEL_PROMPT_TOO_LARGEwhen no room is left, so a prompt that truly fills the context is still refused. - The decode ceiling is unchanged (
config.maxTokensis still the remaining context), and theMODEL_OUTPUT_INCOMPLETEcheck onresult.accepted >= maxTokensstill catches output that runs to the boundary. So this does not let uncapped output silently truncate. - The new tests cover both branches: uncapped admitted at 1000/4096 with reserve 1, and a capped 256 request rejected at a 3900-token prompt.
Non-blocking: the guard's contextWindowTokens falls back to ELIZA_LLAMA_N_CTX (4096) while the budget falls back to 32768 when active.contextSize is unset. That mismatch was there before, but it only matters if contextSize is missing. I did not run the tests.
AI provider/model: anthropic / claude-sonnet-5-5
Client / agent tooling: claude-code
Contribution skill revision: N/A - ordinary GitHub contribution without optional run evidence
Attribution status: self-reported
— [contributor]
|
The AOSP bootstrap suite passed (41 tests), including uncapped generation context accounting. Reviewed for useful behavior and regressions. Combined integration at |
Relates to
No issue. No open or closed issue or PR covers this defect. Searched:
createAospPreparedTextRequestGuard,resolveAospCompletionBudget,createPreparedModelRequestGuard,AOSP MODEL_INPUT_OVER_BUDGET,aosp outputReserveTokens,aosp maxTokens budget,dispatchThresholdTokens,aosp context budget. Related closed work: #30136 and #29839 (theresolveAospCompletionBudgetcontract), #32843 (mtcow, uncapped bionic chat turns).Change
On AOSP (
ELIZA_LOCAL_LLAMA=1), the fused loader inplugins/plugin-native-inference/src/aosp-local-inference-bootstrap.tsis the only local text backend. When a caller gives nomaxTokens,resolveAospCompletionBudgetsets the decode limit to the rest of the context (contextSize - promptTokens). This is its documented contract ({ contextSize: 4096, promptTokenCount: 1000 }-> 3096).createAospPreparedTextRequestGuardthen gave that same number to the core prepared-request guard asoutputReserveTokens. The guard threshold iscontextWindowTokens - outputReserveTokens, so the threshold became the prompt token count itself. The guard rejectsinputTokens >= threshold, so every request withoutmaxTokensfailed withMODEL_INPUT_OVER_BUDGETbefore the stream opened, whatever the prompt size.Callers that send no
maxTokensinclude the REPLY action rewrite (plugins/plugin-assistant/src/features/basic-capabilities/actions/reply.ts,TEXT_LARGE) and grounded action replies (plugins/plugin-assistant/src/actions/grounded-action-reply.ts,TEXT_SMALL). The core dispatcher addsmaxTokensonly whenMAX_TOKENSsettings exist.generate()(probe below), 1000-token prompt,contextSize4096maxTokensMODEL_INPUT_OVER_BUDGET(reserve 3096, threshold 1000), stream not openedmaxTokens: 256Now the guard reserves the caller's explicit cap when there is one. Without a cap, it reserves one token, the room to begin decoding. The native decode limit (
config.maxTokens) does not change.Why this implementation
plugins/plugin-native-inference/src/ios/native-generation.ts) already uses this rule:outputReserveTokens: args.requestedMaxTokens ?? 1, with the comment "With no explicit output request, reserve only room to begin decoding." This change uses the same rule for AOSP.requestedMaxTokens) to the internal guard arguments, because the guard cannot know fromconfig.maxTokensalone whether the caller asked for a cap.generate()passesargs.maxTokensinto it. 8 lines change in the source file.__tests__/aosp-local-inference-bootstrap.test.tssuite. One case is the uncapped request (it fails on develop). One case keeps the explicit-cap reservation and rejection.Testing
How to test
Use Bun 1.4.2 (the repository pin).
Expected: 41 pass. On develop 78d8235 with the same test file: 40 pass, 1 fail (
admits an uncapped request that may decode into the rest of the context,MODEL_INPUT_OVER_BUDGET).developand has no conflicts with the latestorigin/develop. (The branch is one commit on develop 78d8235.)bun installandbun run verifyafter syncing; record blockers below.Evidence Gate
Evidence must match the reviewed commit.
packages/scripts/pr-evidence.ts rowssets the marker from the live PR head; rerun after each push.
Keep every row. Attach artifacts inline or write
N/A - <reason>whereinapplicable. UI changes require desktop/mobile before-and-after screenshots,
an uploaded MP4 explainer and walkthrough, video proof of the changed flow,
logs, and OCR review. Screenshots do not replace video. Use JPG images where possible.
Keep generated artifacts out of source control.
N/A - no UI change; the rejected request before the fix is recorded under Evidence Details.
N/A - no UI change; the admitted request after the fix is recorded under Evidence Details.
N/A - no UI change; only the AOSP native loader admission changes.
N/A - no AOSP device was available; the loader probe output and its limits are under Evidence Details.
N/A - no frontend code changes; admission runs in the agent process on the device.
N/A - no prompt, action or model output changes; the fix only admits requests that the guard rejected before any model call.
N/A - no stored data, files or schedules change; the loader output is shown in the backend-logs row.
N/A - no rendered visual surface changes in this native loader fix.
Evidence Details
Red: develop 78d8235 with the new tests
Green: 6d25725
Loader probe: develop 78d8235 and 6d25725
The loader probe was a temporary test (not committed). It drove the real
tryBuildAospFusedTextLoader()loader,loadModelandgeneratewith a fakebun:ffimodule (the tokenizer returns 1000 tokens) and a fakestreamGeneratethat records the stream config. No device was used, so it is not a device log.Prior work: the iOS native path (
ios/native-generation.ts) already reservesrequestedMaxTokens ?? 1. #30238 (lalalune) restored theresolveAospCompletionBudgetcontract that #30136 and #29839 described.Known gaps / failures
bun run verify(the full monorepo gate) was not run locally. The suites above and CI on the PR head cover the change.inputTokens >= threshold, whileresolveAospCompletionBudgetaccepts an explicitmaxTokensequal to the remaining context. A request withprompt + maxTokens == contextSizeexactly is still rejected. This PR does not change that boundary.Deployment
None.
🤖 Generated with Claude Code
AI provider/model: anthropic / claude-opus-5-5
Client / agent tooling: claude-code
Contribution skill revision: N/A - ordinary GitHub contribution without optional run evidence
Attribution status: self-reported
— [contributor]