Repository navigation
Infill fails to find completions on Qwen3.8 27b with default top_k and top_p #230
Description
Activity
Certainly, they don't mention it in the documentation any more. But the tokenizer still has tokens for it, and if you build a FIM prompt, it'll generate credible completions.
My guess is that they don't consciously train FIM any more, but started from a FIM-trained checkpoint. Does that mean the FIM completions are lower quality than the old Qwen 2.5 coder models, or does Qwen 3.8's better code reasoning compensate? I don't know, and I'm partly doing this to find out.
But Qwen 3.8 is definitely capable of FIM, and it works with Llama.cpp's defaults. It only fails in llama.vscode because it overrides those defaults. It seems a shame to close this off.
@jamespic I would suggest to add new settings top_p and top_k with default values the current ones. So it will be easy to change and experiment. This way you (and other people) could test/experiment and give feedback about Qwen 3.8 and other models. On the other hand it will not affect the other users.
Sounds sensible. I'll get a PR together.
Reacted by igardev
When using Qwen 3.8 27b, completions fail with "no completion". This was surprising, since Qwen 3.8 27b does support FIM, and I'd been able to invoke the
/infillendpoint successfully with curl.Looking into what differed between working curl invocations and non-working llama.vscode invocations, the main culprit was the
top_pandtop_kconfiguration. llama.vscode defaults totop_p = 0.99andtop_k = 40. Successful invocations via curl (that do not include explicittop_pandtop_kparameters) tell me that they usedtop_p = 0.95andtop_k = 20. When I modifyllama-server.tsto use these defaults indefaultRequestParams, llama.vscode is able to successfully obtain completions.I'm happy to raise a simple PR to change these defaults if that's the simplest way forward, but I figured it would be hasty to do so since I don't know the origins of these values, and could imagine that the right fix is more subtle.