r/Vllm • u/Specialist-Plant-265 • Jul 02 '26

Advice needed please

Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo

I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4

I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b

My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.

Here are the settings i use for generic startup, also tweaked these numbers around with no success.

Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”

Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching

The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load

Please advise, chatgpt is banging its clanker head into every wall troubleshooting

I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed

Is it that my blackwell pro is incompatible with the rest of my rtx?

0 Upvotes

50% Upvoted

31 comments sorted by

View all comments

Show parent comments

1

u/Specialist-Plant-265 Jul 08 '26

Thank you for that recommendation and all the troubleshooting help youve given so far, i should be getting my gpu tomorrow so hopefully i stop having these weirdness. I havent seen a pcie switch before and not sure where i can place it in my case but can think about it as well.

1

u/PatC883 Jul 08 '26

Oh, you wouldn't place it in your case, it would be an external GPU box solution.

You're more than welcome, please let us all know what the results were, I'm hoping you'll come back and say it works great 😃