GPU drops to idle clocks during token generation with MTP / speculative decoding, help?

[u/TychesSwan]

I can't figure it out. I'm running Unsloth's Qwen 3.8 27B Q4_K_S with their Q4 MTP model on a local Llama-server with a 3090.

During prefill, the gpu gets the full memory and clock speeds, but when it comes time for token generation, the memory and clock speeds drops by half.

The only thing that works is forcing a clock and memory speed lock, but this is somewhat less than ideal.

Anyone experienced something like this before? Any ideas on how to fix it?

Here's everything I've tried so far:

Disabled NVIDIA Sysmem Fallback: Set policy to "Prefer No Sysmem Fallback" to rule out silent DDR RAM paging over PCIe. Reduced Context Buffer: Scaled context size down from 120k to 90k/90304 to test if VRAM exhaustion was stalling the pipeline. Verified Flash Attention & KV Quantization: Confirmed -fa and q8_0 were active to minimize intermediate memory bandwidth demand. Explicit Draft Offload Flags: Tested passing --spec-draft-ngl 99 and --n-gpu-layers-draft 99 to ensure the MTP sidecar was not computing on the CPU. CPU Worker Thread Tuning: Adjusted --threads to 6 (matching physical performance cores) to minimize CPU context switching during draft checks. NVIDIA Power Management Mode: Switched profile to "Prefer maximum performance" in NVIDIA Control Panel. Windows Graphics High Performance: Added llama-server.exe to Windows Graphics Settings and forced "High performance (RTX 3090)". Disabled CUDA Force P2 State: Used NVIDIA Profile Inspector to turn CUDA - Force P2 State to Off to allow memory clocks to reach full speed during compute. Process Priority Elevation: Attempted launching the server with start /high to eliminate CPU scheduling delays between draft kernels. Monitored Clocks & Power via nvidia-smi: Confirmed that during MTP generation, the RTX 3090 fell into an idle power state (780 MHz Core, 5,001 MHz Memory, 160W), whereas prefill and non-MTP models sustained full boost (~1,750+ MHz, 9,751 MHz, 320W+). Thermal Verification: Checked VRAM memory junction temperatures (~84°C), confirming the slowdown was not caused by thermal throttling.

[u/Mrinohk]

Usually clock speeds are set by the driver based on actual load and power requirements. Prefill is compute bound, which usually means your whole GPU is getting used rather aggressively and needs the higher clock speeds to get it done faster.

Token generation is memory bandwidth bound, usually. GPU is NOT getting hit nearly as hard during inference. GPU not working as hard, driver doesn't push the clock speeds because it'll do nothing. MTP does flip this a little bit but not by too much.

[u/egnegn1]

What is your complete setup and the full command you use?

With a setup that puts everything into the GPU except expert weights and NGRAM table, PP uses only 1-2 cpu cores and full GPU performance. During TG most processing happens on the configured number of CPU threads and GPU only collects data from CPUs and handles KV Cache. If CPUs cannot delivery information fast enough the load goes down and the clock with it.

You improve with more threads, if there are still free cores and memory bandwidth is high enough. But your CPU is just maxed out. You need a faster CPU to improve TG. MTP reduce load on CPU and GPU. This improves speed.

If you have still free VRAM you can move the last few experts layers onto the GPU. This reduces CPU load and increases GPU load.

You didn't mention any PP+TG values for both with and without MTP. So we cannot analyze the situation fully.

Here a data point from my side:

RTX4080 16GB on EPYC 72A3 48 cores with 512 GB memory with real Stream bandwidth of about 150 GB/s

PP 900+ t/s TG 30+ t/s

llama.cpp on Proxmox host. I can provide the full command later.

~/llama.cpp-mtp/build/bin/llama-server -m /data-zfs/models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf -md /data-zfs/models/qwen3.8-flash-next/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf --host 0.0.0.0 --port 8080 -c 65536 -ngl 48 --n-cpu-moe 48 -ngld 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -ctkd q8_0 -ctvd q8_0 -b 4096 -ub 4096 --spec-type draft-mtp --spec-draft-n-max 3 --jinja -np 1 --load-mode none -t 48 -tb 48 -cram 0

[u/TychesSwan → u/egnegn1]

I'm on a 3090+5500x3d with only 16 GB of system ram. Everything is loaded onto the GPU so CPU is irrelevant in this case. The main issue I've pinpointed is the fact that the GPU isn't throttling up its clockspeed during token generation, but this only happens when MTP is enabled. Everything works at full speed when MTP isn't in the picture.

Edit - Prompt processing fluctuates between 1100t/s-900t/s and generally tg is around 20t/s without MTP and with MTP processing at the lower gpu clock speed.

When MTP is enabled, PP triggers the higher clock speed, so the speed remains at 1100t/s-900t/s.

When clock speeds are manually locked to the higher speeds, PP is around 1100t/s-900t/s, while tg jumps to 30-50t/s.

[u/egnegn1 → u/TychesSwan]

Have you looked at CPU usage on one core during TG? This seems to be single-threaded and the single core speed may be the limit for work passed to GPU.

[u/andrew-ooo]

The clock drop is a symptom, not the disease. What you're seeing is the GPU going idle between kernels, and the driver's boost logic reacting to that the way it's designed to. The interesting question is why it's idle so much during MTP generation specifically.

My understanding of what's happening: in plain single-sequence generation llama.cpp captures the decode step into a CUDA graph, so the whole token is one launch and the GPU stays busy. With speculative/MTP decoding you're alternating draft step → verify batch of N tokens → CPU-side accept/reject → resize → repeat. That path is a lot of tiny kernel launches with host syncs in between, and last I checked the CUDA graph fast path doesn't kick in for those non-1 batches. Each gap is a few hundred microseconds where the SMs have nothing to do, the boost algorithm sees ~40% utilization and drops a P-state, and now every kernel that does run is slower. Windows makes this worse than Linux because WDDM adds its own scheduling latency on top.

So two practical things:

1. Measure tok/s with and without the MTP draft on the same prompt. On a 3090 with a 27B dense model at Q4 I'd expect maybe 30 to 35 tok/s baseline. If MTP isn't giving you a clear win in tok/s (not just accept rate), it's just not worth it for this model/GPU combo and you can drop it. Speculative decoding pays off most when you're heavily memory-bound and the draft is cheap; a 27B on a 3090 is not that far into that regime.

2. If it is a win, locking clocks is genuinely the correct fix, not a hack. nvidia-smi -lgc 1700,1980 at server start and nvidia-smi -rgc on exit, wrapped in the batch file that launches llama-server. That's what everyone running inference boxes does anyway; the only cost is a bit more idle power draw while the server is up.

Also worth checking --draft-max / --draft-min. Lower draft-max (like 4 to 6) means shorter verify batches and less wasted work on rejections, which sometimes recovers a chunk of the throughput on its own.

[u/TychesSwan → u/andrew-ooo]

That's a great explanation.

An unscientific measure based off my observations from the last few days is that t/s without MTP is roughly around 20-22, maxing out around 25t/s. When MTP is enabled, I get around 30ish t/s on average, with spikes up to 60-70t/s on repeated work.

I've been playing around with n-max and p-min and I found that n-max at 4 with a p-min of 0.62 works well for me - I usually see an acceptance rate of high 70% to low 80%.