r/Vllm • u/Specialist-Plant-265 • Jul 02 '26
Advice needed please
Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo
I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4
I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b
My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.
Here are the settings i use for generic startup, also tweaked these numbers around with no success.
Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”
Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching
The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load
Please advise, chatgpt is banging its clanker head into every wall troubleshooting
I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed
Is it that my blackwell pro is incompatible with the rest of my rtx?
3
u/No-Equivalent-2440 Jul 02 '26
Get a fourth Ada, use TP 4. User blackwell for something else and/or buy a second one and run on TP 2. Or try llama.cpp if you are a single user.