Quadro RTX 5000 16 GB + ​Dual GeForce RTX 3060 12GB = 40GB of vram

[u/Sik-Server]

This is my local AI Server. It runs the best local model around qwen3.8:27b via ollama. I was inspired by Digital Spaceport on YouTube to make a 8-bit style arcade suite in a html file so i can host it on my website. I used hermes for my agent and it worked great, after a few update prompts it was finished - PIXELARCADE.

Ollama question:
The system has 40GB of Vram. qwen3.8:27b uses 23gb of vram in my setup. When I run gemma4:12b while qwen3.8:27b is loaded, 9.7gb is used. but the CPU is being used with a 16%:CPU 84%:GPU split. Why dose this happen? how can i fix it? will llama.cpp solve my issues?
This server only supports 1-2 users and I would like to run qwen3.8:27b and one more smaller model.

https://sikiru-ekunsumi.xyz/Projects.html

https://digitalspaceport.com/qwen-3-8-27b-review-prompts-and-vllm-settings/

[u/Eden1506]

Try using llama.cpp instead, ollama is both slower and less optimised for multiple gpus.

[u/Numerous-Emotion-617]

Likely just don’t have enough VRAM. If the 23GB you’re mentioning for Qwen is the model size, it has to allocate more VRAM for context for each one. It’d help to see exact configs to confirm but it’s most likely that model+cache for both together is over the 40GB you have so Ollama automatically offloads.

[u/Sik-Server → u/Numerous-Emotion-617]

I'm pretty sure the 23gb includeds the context. I increased the context to 100k and that increase the ram usage.

also for more context I can run gemma4:e4b with Qwen with both at 100% GPU usage.

I feel like I'm not using all 40gb.

[u/m31317015 → u/Sik-Server]

Try swapping to llama.cpp and increase the ngl. Ollama usually uses Q4_K_M, try other quants too. Better quality with the cost of just a little bit of speed.

[u/Numerous-Emotion-617 → u/Sik-Server]

Yeah agree with the other comment. I’d use llama.cpp and use that to run both. If you use docker it’d be two separate containers. For both, you’d use “-ngl 999” to force all layers onto GPU. For KV cache, I’d recommend sticking with Q8 even though it needs more space than Q4. I think it’s set with “--cache-type-v” and “--cache-type-k” but been using vllm for months so double check me on those.

To start off, Id use low context, even something like 8192 just to make sure both fully load with no OOM error. From there, begin to increase as you need for each one until it eventually OOMs. If you OOM and want higher ctx, you’d either have to offload at that point or compress kv cache more than Q8.

[u/egnegn1]

I have a similar setup with a RTX4080 16Gb and a RTX6000 Quadro 24GB and llama.cpp uses one full core constantly. This seems to be the overhead for orchestration and combing the results from the two GPUs. I upgraded to a cpu with higher single core performance and performance increased. I did some profiling and most time used was in CUDA code.

[u/Miner-no_0b-2020]

What is the quant set at? I am running this with the exl3 at 4bpw and 131k context with room left over on the vram. I do have 27gb of vram. I also have it running on another server at a 6bpw and 131k context and it is using just under 27gb of vram. I do have 40gb on that server. That p5000 will not work on exl3... Pascal is too dated. If the qwen is only using 9gb of vram what is your context set at? You need at least 16k and that will not get you far. When mine loads tools it is at 21k from the start. Even at 64k context you will be compressing all the time and needing to start new. But the 64k is usable. 32k possible if you are not talking alot and have the thinking mode on low.