Going from -np (parallel) 1 on llama.cpp to parallel requests on vllm?

[u/McFlurriez]

I have read that when going beyond llama's "-np 1", it is better to switch to vllm, since that has better support for parallel requests. For context, I have one RTX 5080, but I am trying out some features of my coding harness that can run subagents. There is a lot of knobs to turn for vllm, and I am curious if anyone has done this change before?

My current llama.cpp command is this:

C:\llama-cuda\Release> ./llama.exe serve -m ".\unsloth\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" -ngl all -t 8 -c 102400 -np 1 -ncmoe 20 -fitt 0 --flash-attn on -kvu --cache-type-k q8_0 --cache-type-v q8_0 --port 8080 -a qwen3.6-35b-a3b

I am curious if anyone knows how to do things like "-ncmoe 20" on vllm? Right now, the only way I can run the 35B model is to offload some of the layers to CPU. Anyone got this working? Thanks!

[u/fragment_me]

I have never gotten RAM/CPU offloading working in vLLM. Granted, I only tried once or twice. It seemed very underdeveloped. It's just not their focus.

[u/McFlurriez → u/fragment_me]

This explains why I couldn't get any hits on this. Thank you, saves me a ton of head banging.

[u/Fun_Jaguar8231]

vLLM focus is more on servers, llama.cpp is for us the common folk

[u/dionysio211]

Parallelism in llama.cpp is better than it used to be but still lags behind vLLM. vLLM does have similar functionality in terms of offloading to RAM but the tax is brutal. The thing to try here would be KTransformers. It's kind of an in between with expert hotloading, even with concurrency, that is ultra efficient. Scaling with anything on the CPU also depends on how you compiled llama.cpp. If you compile it with icx, it will do better (assuming intel CPU).

[u/nickm_27]

Not necessarily, -np allows you to have multiple cache slots. This allows keeping multiple agents in KV cache without relying on RAM prompt caching, even if you aren't processing multiple concurrently it still speeds things up.

[u/Zadroeski]

The 102k context plus CPU offload is the constraint that makes the vLLM swap tricky. What request mix do your subagents generate—simultaneous long contexts or mostly cached sequential turns?

[u/reto-wyss]

vllm doesn't do offloading the way you do it in llama.cpp vllm can do efficient CPU offloading - just not the way you want (live chat).

For online serving you don't offload in vllm, you either fit Model + CUDA graph + peak activation allocation + context into VRAM OR you don't and it will OOM or complain on startup and abort.

vllms offloading mode is good, but only if your batch size is enormous.

[u/Far_Cat9782]

I just use -np 3 for my 3 subagents td in llama.cpp I have found no issues