server : add speculative decoding support - #10455
Conversation
1973399 to
7dc6ae5
Compare
|
From what I have read the goal is faster inference while retaining quality of the larger model. I am using rx6900xt with vulkan I get about 10-12 t/s with an incorrect configuration.
Flipping the models increased speed and the output looks similar. This makes sense since the -md is the draft model which is supposed to be the smaller model. I get about 16 t/s with the correct configuration.
Setting a lower context 2048, when the limit is reached the server crashed. |
c5ddee2 to
e80f758
Compare
|
@3Simplex What is the output of the following bench on your machine: llama-bench.exe -m "...Qwen2.5-Coder-7B-Instruct-Q8_0.gguf" -p 1,1,2,3,4,5,6,7,8,12,16,32 -r 20 -n 0 -ngl 99 -fa 1 |
|
.\llama-bench.exe -m "...\Qwen2.5-Coder-7B-Instruct-Q8_0.gguf" -p 1,1,2,3,4,5,6,7,8,12,16,32 -r 20 -n 0 -ngl 99 -fa 1
build: 0c745909 (4160) |
|
I tried out commit e80f758e6e4063e0867a5a8d2c929dac48432ede with my P40s, 3xP40s and 3090. These are the commands for the baselines and the tests. Baseline:
With speculative model (just removed the
Tested it with curl using:
Data:
|
e80f758 to
d905266
Compare
|
Currently, it requires The biggest benefit from speculative sampling is when you have more grounding. For example, if you have enough memory for a bigger context, you can try something like this: # get the llama.vim plugin source code
code=$(curl -s https://raw.githubusercontent.com/ggml-org/llama.vim/refs/heads/master/autoload/llama.vim | jq -sRr @json)
# ask qwen to implement something (speculative decoding disabled)
curl --request POST --url http://localhost:8033/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n --arg code "$code" \
'{ messages: [{ role: "system", content: "You are an expert computer scientist. Respond only with code blocks. Do not add any other comments except code." }, { role: "user", content: "Suggest an improvement for the `chunk_sim` function using Levenstein distance: ```\($code)```" }], cache_prompt: true, top_k: 1, samplers: ["top_k"], "speculative.n_max": 0 }')" | jq -r .choices[0].message.content
# speculative decoding enabled
curl --request POST --url http://localhost:8033/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n --arg code "$code" \
'{ messages: [{ role: "system", content: "You are an expert computer scientist. Respond only with code blocks. Do not add any other comments except code." }, { role: "user", content: "Suggest an improvement for the `chunk_sim` function using Levenstein distance: ```\($code)```" }], cache_prompt: true, top_k: 1, samplers: ["top_k"], "speculative.n_max": 16 }')" | jq -r .choices[0].message.contentWith CUDA, you might want to try setting |
|
Thank you for the guidance. Using d905266b05918c709269604b73fe50b41820e965, I reran the tests. Results look quite good.
Server command:
Kept this pretty consistent, except for the 3xP40 run where I added Client side: For the client side curl, I changed Here are the raw results. Some observations first:
3090 data single P40 3xP40 (-sm row) Code generated: function! s:chunk_sim(c0, c1)
let l:lines0 = join(a:c0, "\n")
let l:lines1 = join(a:c1, "\n")
let l:distance = levenshtein(l:lines0, l:lines1)
return 1 - (l:distance / max([strlen(l:lines0), strlen(l:lines1)]))
endfunction
function! levenshtein(s1, s2)
let l:len1 = strlen(a:s1)
let l:len2 = strlen(a:s2)
if l:len1 == 0
return l:len2
endif
if l:len2 == 0
return l:len1
endif
let l:dp = []
for i in range(l:len1 + 1)
call add(l:dp, [])
for j in range(l:len2 + 1)
call add(l:dp[i], 0)
endfor
endfor
for i in range(l:len1 + 1)
let l:dp[i][0] = i
endfor
for j in range(l:len2 + 1)
let l:dp[0][j] = j
endfor
for i in range(1, l:len1 + 1)
for j in range(1, l:len2 + 1)
let l:cost = (strcharpart(a:s1, i - 1, 1) == strcharpart(a:s2, j - 1, 1)) ? 0 : 1
let l:dp[i][j] = min([l:dp[i - 1][j] + 1, l:dp[i][j - 1] + 1, l:dp[i - 1][j - 1] + l:cost])
endfor
endfor
return l:dp[l:len1][l:len2]
endfunction |
|
Also, is |
|
Thanks for the detailed tests. The results are inflated because there is one tricky side effect from the caching - consecutive runs with the same prompt will reuse the previous draft context which combined with greedy sampling would make the drafting instantaneous. So basically, in the following data for example, only the first result is relevant: i.e.
This was a bug - it is fixed now. You should be able to change Btw, here is another fun test that I came up with which uses less context and is suitable for speculation: # get top 10 stories from Hacker News
hn=$(curl -s https://hacker-news.firebaseio.com/v0/topstories.json | jq -r '.[:10] | @tsv' | tr '\t' '\n' | xargs -I {} curl -s "https://hacker-news.firebaseio.com/v0/item/{}.json" | jq -sRr @json)
# make a Markdown table based on some criteria
curl --request POST --url http://localhost:8033/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n --arg hn "$hn" \
'{ messages: [{ role: "system", content: "You are a helpful text-editing assistant. Respond only with the requested text. Do not add any other comments to your response." }, { role: "user", content: "Extract a Markdown table that contains only stories about software engineering, AI or machine learning from the front-page of HN. The table should include: author, title, score, comments and an URL to the story: ```\($hn)```." }], cache_prompt: true, top_k: 1, samplers: ["top_k"], "speculative.n_max": 16 }')" | jq -r .choices[0].message.content |
Thanks. That seems a lot more realistic. I did some tests with a much shorter prompt: "write snake game in swift"
|
|
These numbers look reasonable. The speedup can vary in both ways based on the inputs, but enabling speculative should almost never result in slower than normal decoding. |
With this build I am up to 25t/s on first run generation with speculative decoding using 15/5 draft tokens. |
|
A bit of data with llama-3.1 70B and llama-3.2 1B as the draft model. Prompt: "write a story about the natural resources in Canada".
Server: client (changed speculative.n_max between |
|
Note that I am not very sure what happens with multiple GPUs, but it is possible that the draft model gets split across them, which is not desired (see the logs if that is the case). You would want to keep the draft model fully on one GPU. |
c277c4d to
156aa6d
Compare
I wonder if it is possible to load draft and main model onto different backend. Ie a 7900xtx and P40 in a -cb process |
Hmmm... the zero/one probabilities I got when I ran a 4-bit 8bn drafting model that was fine-tuned on the same data as the verification model. If I use a non-finetuned 1bn or 3bn model, the probabilities vary more, but still no sustained speed-up (just sometimes). Next step: Fine-tune the 1bn model. That shoudl give me more speed than the 8bn bu thten hopefully better proability distributions. |
Okay, so this is already addresse (and solved) here: #9135 |
|
@ggerganov - I've finally got some good news to report. As reported previously, speculative decoding had almost no effect on my M4 Max. I reliably got 8.7 t/s for Llama 3.3 70bn with a 3bn drafting model. But when I started to crank up the -np value, t/s went up significantly. It maxed out at -np 7, when I reliably got 16+ t/s (a doubling of what I had originally, which is amazing). When I went to -np 8, performance cratered and I ended up with 7 t/s. Cranking up the -np value when I run JUST the 70bn model with a drafting model has no effect, so it's definitely an enabler for speculative decoding, at least on my setup. Next up: Getting KV caching to work. With no -np parameter, the system prompt gets cached. With the -np parameter, I see no such effect. I will now try to force the server to use pre-populated KV slots. Will report back when I have that working. |
|
I'm confused. Shouldn't the number of slots available for KV caching be independent of whatever causes spec. decoding to speed up when I set -np to 6 or 7? |
I know I'm fairly late but you may have had better results if your -md was gemma-2-2b-it-Q4_0.gguf |
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
* server : add speculative decoding support ggml-ci * server : add helper function slot.can_speculate() ggml-ci
target #10362
Initial implementation that enables speculative decoding in
llama-server. Test with this command:./bin/llama-server \ -m ../models/qwen2.5-32b-coder-instruct/ggml-model-q8_0.gguf \ -md ../models/qwen2.5-0.5b-coder-instruct/ggml-model-q4_0.gguf \ -ngl 99 -ngld 99 -fa --port 8033 -c 32768 \ --draft-max 16 --draft-min 5--draft-maxand--draft-minmight need tuningllama.cppWeb UI clientTop K = 1-devdargument to put the draft model on only one of them (llama : accept a list of devices to use to offload a model #10497)Feedback is appreciated.
TODO:
server.paramsto something else to avoid confusions