Move a finished conversation's prefix cache to disk - #2261
Open
Lazarus-931 wants to merge 2 commits into
Open
Lazarus-931 wants to merge 2 commits into
Lazarus-931 wants to merge 2 commits into
Conversation
A conversation that has stopped growing still holds the memory its prefix cache needed while it was live. POST /v1/cache/offload writes that prefix to disk, waits for the write, and then releases the memory, so the cache stays available to any later request whose prefix still matches. The conversation is supplied the way a Responses request supplies it, and the prefix is taken from the ids generation actually ran over rather than by re-tokenizing the rendered prompt, which shifts by one whenever a template emits its own BOS. A block is released only once it is on disk and no request is still holding it; anything else stays resident and is reported as retained. Conversations whose cache is kept as a whole-prefix snapshot rather than blocks are spilled the same way, since several model families never occupy the block pool. The salt is recovered from what the manager has stored, because it folds in the model and processor and is never zero.
Lazarus-931
marked this pull request as draft
September 15, 2026 13:19
Lazarus-931
marked this pull request as ready for review
September 16, 2026 14:08
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Free the memory a finished conversation's prefix cache is holding, without losing the cache.
When a conversation goes idle, its KV cache just sits in memory.
POST /v1/cache/offloadwrites that prefix to disk, waits for the write to land, then releases the memory. A later request with the same prefix still reuses it from disk instead of re-prefilling from scratch.A block is only released once it is on disk and no in-flight request is still holding it. Anything that fails either check stays in memory and is reported back as retained, so the endpoint can never pull a cache out from under a running request.
You hand it the conversation the same way you'd hand it to
/v1/responses.Testing
Ran it against 110 checkpoints spanning 40 model families.
Every one freed its memory and handed every block back to the pool — nothing leaked, and the prefix was still reusable afterwards. A typical conversation freed about 110 MB, the largest 660 MB. Most calls finished in under 3 seconds.
One thing worth flagging so it doesn't look like a bug: about a dozen of those families never report
cached_tokensinusage, even on an ordinary warm hit with no offload involved. On those models the reuse shows up as lower latency rather than in the usage field.follows #2260