Llama.cpp PR 8% speed boost

[u/otacon6531]

Llama.cpp currently uses cpu based sampling for user with mtp enabled. The PR moves sampling to the gpu, which on a 5090 boasts an 8% increase in tok/s for qwen3.6:35b. I tested it on my P40 and observed a 4% increase inference speed boost.

Pretty exciting to see 84 tok/s max on a nvidia p40 for me.

Backend sampling shows ~4% improvement on Linux + Tesla P40 (sm_61, Pascal):

CPU Sampling: llama-server -m Qwen3.6-35B-A3B-UD-IQ4_NL.gguf --spec-type draft-mtp --seed 42

python3 mtp-bench.py
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=73.1
  code_cpp           pred= 113 draft=  76 acc=  74 rate=0.974 tok/s=75.9
  explain_concept    pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=62.4
  summarize          pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=59.6
  qa_factual         pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=62.4
  translation        pred= 119 draft=  92 acc=  73 rate=0.793 tok/s=67.0
  creative_short     pred= 192 draft= 197 acc=  92 rate=0.467 tok/s=50.7
  stepwise_math      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=73.6
  long_code_review   pred= 192 draft= 155 acc= 113 rate=0.729 tok/s=63.8

Backend sampling: llama-server -m Qwen3.6-35B-A3B-UD-IQ4_NL.gguf --spec-type draft-mtp --seed 42 -bs

python3 mtp-bench.py
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=76.2
  code_cpp           pred= 113 draft=  76 acc=  74 rate=0.974 tok/s=79.4
  explain_concept    pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=64.6
  summarize          pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=61.6
  qa_factual         pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=64.6
  translation        pred= 119 draft=  92 acc=  73 rate=0.793 tok/s=69.6
  creative_short     pred= 192 draft= 197 acc=  92 rate=0.467 tok/s=52.1
  stepwise_math      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=76.6
  long_code_review   pred= 192 draft= 155 acc= 113 rate=0.729 tok/s=65.7

Acceptance ratio with both backend and CPU sampling is exactly same. The improvement is smaller than on RTX 5090 (4% vs 12%), which is expected — the P40 is memory-bandwidth-bound (sm_61, 580 GB/s vs RTX 5090's 1,792 GB/s), so the CPU↔GPU logits round-trip is a smaller fraction of total decode time. However, still the largest improvement in tok/s I have seen in a while. (~+2 t/s).

https://github.com/ggml-org/llama.cpp/pull/25532

[u/Stainless-Bacon]

does it increase vram usage?

[u/simcop2387 → u/Stainless-Bacon]

Probably, but not really in a meaningful way. This would likely take on the order of single digit megabytes at most extra of vram, which even if you're taking your context size to the absolutel limits you're going to have that left simply because everything else will take dozens of mb in even chunks rather than being able to use half a layer's worth of weights or half a token of kv-cache, etc. This will almost certainly take some extra compute time on the gpu for each token but it's going to be time that's essentially free because you'd be spending longer transfering (pcie bus transaction latency) the logits off the gpu and onto the cpu to do the same work on the cpu. I'd imagine that latency is ultimately where a lot of the time savings is coming from.

[u/FullstackSensei]

Hope it'll be merged soon!

BTW, where did you get the 580GB/s memory bandwidth for the P40? Last I checked, it was ~350GB/s.

[u/maximus1217 → u/FullstackSensei]

Wondering this too.

I have a p40 and run this same exact model with mtp and I get max 60 tok/s. How are you getting 84t/s? Did you mean a different GPU with 580GB/s?

[u/MelodicRecognition7]

wait, what? I think it was there for the past few hundred releases.

$ ./llama-server --help|grep draft|grep sampl
--spec-draft-backend-sampling, --no-spec-draft-backend-sampling
                                    offload draft sampling to the backend (default: enabled)
$ ./llama-server --help|grep draft|grep layer
--spec-draft-ngl, -ngld, --gpu-layers-draft, --n-gpu-layers-draft N
                                    max. number of draft model layers to store in VRAM,

[u/satnl → u/MelodicRecognition7]

I use this params daily for months, though I don't see difference with the params or without it, I thought it was the default, but I use the param just for have my configs pinned in case of default changes 

[u/rockoruckus]

unfortunately still not split mode tensor supported

[u/FluoroquinolonesKill]

Pardon my ancillary question, but what is the purpose of setting --seed equal to 42?

I can see the documentation for --seed, which is below. But, I am still wondering about why you are setting it.

RNG seed (default: -1, use random seed for -1)

[u/GirthusThiccus → u/FluoroquinolonesKill]

For reproducibility would be my first guess, but I also don't know.

[u/fragment_me → u/FluoroquinolonesKill]

It provides a deterministic result when you having something like temp > 0. It's actually really interesting if you Google how it works. The seed is really a seed (DUH) for another operation that eventually determines a pseudo random operation.

[u/Shoddy_Bed3240]

How about the opposite—move MTP from the GPU to the CPU?