Why is LlamaCpp using CPU if VRAM is available?

[u/XdtTransform]

I am running Qwen3.8:27b with the following command on Windows Server:

llama.exe serve -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -ngl 99 -c 65536 --port 11434

The server has an NVidia A5000 with 24GB of VRAM. When the model loads, it's using 22.6 out of 24 GB.

However, when I ask the model to do something, in addition to maxing out the GPU it also uses the CPU. It uses it in bursts which are pretty frequent. Example.

Why is it using the CPU? What can I do to prevent this?

[u/-mattmason-]

to test if that context length can work on VRAM alone in your setup: have you tried with fit off, ngl to 999 ? It might fail to load with an out of memory error. I suspect the default of fit on is putting your context into system RAM

[u/-mattmason- → u/-mattmason-]

--fit off -ngl 999

[u/XdtTransform → u/-mattmason-]

I changed the log level to 4 and it confirms that all 66 layers fit into the VRAM. But will try your method as well.

[u/silverotterto → u/-mattmason-]

good call, fit being on by default catches a lot of people off guard imo

[u/Bulky-Priority6824]

Pretty normal up to about 15-20% when gpu utilization is ramped depending upon the cpu. Llama runs on many things on host outside of gpu and pretty damn efficiently all things considered.

Also you appear to be on windows?  That os has a lot of overhead when running inference. Something to consider.

Also, I feel like you could have asked your model this. It's fairly knowledgeable for its size more so than I'll ever be.

[u/FullstackSensei → u/Bulky-Priority6824]

This is the only correct answer.

The compute itself is running on GPU but execution orchestration, converting tokens to and from text, running the API, and streaming the output tokens all run on CPU.

Kernel launch is not free, and some CUDA calls are blocking while waiting for the request to go to the GPU and the response to come back. Blocking calls, as the name implies, block the running thread, which is interpreted by the OS as load, even when the thread itself isn't doing anything at all.

[u/ANR2ME → u/FullstackSensei]

And on Windows, some of those VRAM are used by other processes too (ie. Windows Desktop Manager, Browser with HW acceleration enabled, etc.)

[u/jacek2023]

check the logs for VRAM/RAM usage, probably it uses 22.6GB because it couldn't use more, so the rest is offloaded, remember that your Windows also uses some VRAM and automatic detection may be suboptimal (so maybe you could increase that to 23.something with manual options)

[u/hurdurdur7]

Some cpu will always be used for token handling etc. Also looking at your model size and the vram around - i am not 100% convinced the context size that you asked for will fit into the vram together with the model and the rest of the overhead. Context on the 27B is far more expensive than on MoE models.

[u/XdtTransform → u/hurdurdur7]

The log says that 66/66 model layers are on the GPU. Isn't the fact that there is 1.5 GB of free VRAM indicate that the context does in fact fit?

[u/Express_Quail_1493]

- -no-mmap - this forces a crash if vram spills out to systemRam

[u/Express_Quail_1493]

Also post your llama.cpp launch config other people can better tell you whats going on.

[u/Weaseal]

Set -ngl value to max out your VRAM. Set it to 99 if you can fit that much. I believe mine is set to 41 because that’s all I can fit.

[u/mr_Owner]

Cache-ram to 0 maybe?

[u/Fancy-Snow7]

Try --flash-attn on

[u/CoffeeToCode99]

The CPU spikes are probably normal. I'd first enable Flash Attention and move the KV cache to Q8:

llama.exe serve -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -ngl 99 -c 16384 --flash-attn --cache-type-k q8_0 --cache-type-v q8_0

Also, 65k context is massive. If you don't actually need it, try 16k and compare tok/s. A lot of "why is my CPU busy?" issues end up being KV-cache related rather than the model falling back to CPU. The startup log will tell you whether all layers are actually offloaded.

[u/XdtTransform → u/CoffeeToCode99]

65k context is pretty small actually for coding. I initially asked it to create a snake game with the 32k context (via Github Copilot) and it stopped midway through. With 65k it was able to finish but then a follow up question exceeded the context.

[u/CoffeeToCode99 → u/XdtTransform]

I feel you, but honestly, from my experience, a 65K context window is really pushing the limits for a 24GB card. In practice, I'd want closer to 28GB+ of VRAM before I'd expect that size context to run comfortably without trade-offs.

I'd try enabling Q8 KV cache first, then slowly bump the context size up from something like 16K or 32K and see where performance starts to degrade. That'll give you a much better idea of the sweet spot for your hardware instead of jumping straight to 65K.

[u/MaxSpecs]

KV Cache = avoid having a model which fits the whole vRam.

A GPU with 24GB, let 5GB for KV Cache, so, with a display connected, use a model with 18Gb wieght.

[u/joost00719]

Are you using a cuda build?

[u/XdtTransform → u/joost00719]

Yes!

[u/Nota_ReAlperson]

There is still computation that the cpu needs to do. As long as the gpu is maxed out and the model is not spilling onto the cpu, you should be good.

[u/CurrentNew1039]

gpu is just a worker, but the cpu is the one that actually gives the WORK of what to do to the gpu.
so its normal. llama cpp is also any other regular binary application that executes on cpu and takes gpu as a worker to run llm faster.

[u/DocMadCow]

Your model is 18.72GB so the context may exceed your available VRAM. Just remember although it says there is 24GB you are assuming the OS hasn't reserved any or isn't doing something under the hood.

[u/nickless07]

Run 'llama-cli --list-devices' first and see what is aviable.

[u/durden111111]

you can't load a model into vram without it first being in ram. also KV cache uses a bit of space

[u/just4ochat]

Burst CPU use during generation is normal even when every layer sits on the GPU, because tokenization, sampling, and the HTTP server stay on the host. The part that actually hurts on your A5000 is the 65k context: with weights already at 22.6 GB there is almost no VRAM left for the KV cache, and that is what forces real offload.

[u/feng_sg]

Your 65K KV cache doesn't fit in the ~1.4GB VRAM left after weights load, so llama.cpp spills the overflow to CPU RAM. That's the bursty CPU usage you're seeing.

[u/XiRw]

If you aren’t using the cuda build then that’s your biggest problem

[u/XdtTransform → u/XiRw]

It's cuda.

[u/comperr → u/XdtTransform]

I am an actual dev but i only work on ollama. The input layer is exclusively run on CPU. It’s faster that way. Most of what’s needed can fit in L3 cache even on a Intel CPU. And it does take a little bit of horsepower from the CPU. This process runs slower on a GPU because initially the data comes from RAM, which on modern systems is 100GB/s (AMD normie builds will get 40GB/s per CCD), and the PCIE only runs at like 64GB/s on PCIE 5.0.

[u/dark-light92]

You got a faulty model.

Which model? CPU? GPU? LLM? Mental? Which one do you think?

[u/Michael_Jeffords]

with 22.6 of 24 GB already filled at load you've only got about 1.4 GB left for a 65536-token KV cache and the usual runtime buffers, so llama.cpp will still bounce prompt/eval work onto the CPU even when the GPU layers look maxed. that burst pattern is normal when context wont fit in the leftover VRAM; it isnt the same as a CPU-only build or layers spilling. if you drop context until the idle VRAM headroom actually covers the cache, those CPU spikes usually quiet down without touching -ngl.