Tags: ollama/ollama
Tags
mlxrunner: Update XGrammar to 0.2.7 for structured outputs We pick up schema fixes for typed dictionary values and short arrays.
mlx: speed up Qwen 3.8 prompt processing (#18550) * mlx: speed up Qwen 3.8 prompt processing Use MLX's gated-delta kernel for long scans and fold dense MLP global scales into SwiGLU. * address comments
x/transfer, server: tighten redirect handling for registry requests (#… …18512) Redirects for registry and blob transfers now validate the target scheme and resolved addresses before following, re-check DNS on each redirect, and do not follow redirects that switch an https session to plain http. The --insecure option continues to relax address checks for private registries but not scheme checks.
cli: add first-run onboarding shared with the desktop app (#18495)
mlxrunner: Release freed KV buffers during speculative decode The decode loop releases MLX's pool of freed buffers every 256 generated tokens, which is also how often the KV cache grows and drops its previous, smaller buffers. The check fires only when the token count lands exactly on a multiple of 256. Speculative decoding emits several tokens per round, so most rounds step over the boundary and the pool is never released. Each growth at a long context leaves several GB of buffers that no later allocation can reuse, so the runner's footprint keeps climbing over a long generation until the system runs out of memory. We now release the pool whenever a round crosses a multiple of 256 tokens, which is what a single-token round already did. With qwen3.8:27b-mlx at a 98k-token context on a 128 GB machine, a long speculative generation previously grew the runner past 90 GB and panicked the kernel; it now stays flat at 30 GB.
mlxrunner: lay out model by contract, checkpoint and construction model is one package with three jobs: the contract between the runner and the architectures, the opened checkpoint, and building nn layers from checkpoint tensors. Its files did not say which was which. base.go carried the folded package's name over the interfaces and the registry, root.go held the safetensors header scan next to Root, and quant.go mixed the nvfp4 global-scale helpers with quant parameter resolution. base.go becomes model.go, named for what it holds. root.go keeps Root and Open; TensorQuantInfo and the header scan join quant.go, so everything the checkpoint says about quantization is read and resolved in one file. The global-scale helpers move to globalscale.go with their tests. Root.Close, a no-op with one caller, goes. No code changes otherwise.
PreviousNext