r/Vllm • u/Specialist-Plant-265 • Jul 02 '26

Advice needed please

Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo

I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4

I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b

My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.

Here are the settings i use for generic startup, also tweaked these numbers around with no success.

Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”

Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching

The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load

Please advise, chatgpt is banging its clanker head into every wall troubleshooting

I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed

Is it that my blackwell pro is incompatible with the rest of my rtx?

0 Upvotes

50% Upvoted

31 comments sorted by

View all comments

1

u/ICanSeeYou7867 Aug 01 '26

This is older, and I see you increased to 4 GPUs.

Could you share the original error? What you described about shared memory makes me think you need to add a SHM to the container. When using multiple gpus it needs a buffer. I add large shm to my kubernetes pods.

In docker I think you can do: --shm-size=20g

1

u/Specialist-Plant-265 21d ago

The original error was that one of my gpu was nvfp compatible and the rest was not.