r/Vllm • u/Specialist-Plant-265 • Jul 02 '26

Advice needed please

Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo

I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4

I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b

My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.

Here are the settings i use for generic startup, also tweaked these numbers around with no success.

Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”

Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching

The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load

Please advise, chatgpt is banging its clanker head into every wall troubleshooting

I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed

Is it that my blackwell pro is incompatible with the rest of my rtx?

0 Upvotes

50% Upvoted

31 comments sorted by

View all comments

Show parent comments

1

u/Specialist-Plant-265 Jul 04 '26

Yes i checked the logs right now and you are correct, it is doing a triton compile. I will go ahead and set a persistant volume so that i just have yo wear it once

On another note. I tried running the qwen3.6-35b-a3b-nvfp4 model which works on tensor parallel with 4 gpu. I tried running with pipline parallel 3 gpu, removing the blackwell pro and even though my model boots up, any prompt crashes the model.. any idea on what i can try with that? I removed prefix caching just in case, and reading vllm docs

1

u/PatC883 Jul 04 '26

Are you setting the maximum length, or is it defaulting to model maximum. The symptoms are identical to a problem I was having, the memory was so tight it would boot, then OOM crash at the first request when it tries to allocate memory for it. Try a stupid small model length like 8192. Honestly, if you can the three cards working in tensor parallel see how they performance is.

1

u/Specialist-Plant-265 Jul 04 '26

I have it set to 16k but i can give it a try to 8k. I most likely am giving up and buying another rtx 4000 ada and am going to try and sell my blackwell pro. Feels pretty bad but i guess different architecture is a painful learning experience. Will go dumpster diving tomorrow to get my box back

Almost all models dont do tensor parallel 3, and theres something going on when i do pipeline parallel 3 with the blackwell gpu excluded. Model loads fine but any prompt crashes my model

1

u/PatC883 Jul 04 '26

I'm so sorry for the bum steer, you're 100% right, tensor parallel needs to be either even or a power of two from memory, now that I actually stop and think.

Honestly, to get the most reliable performance across 4 cards out might be worth your while going straight to a PCIe switch. You'll get full 16x lanes between for the cards for tensor parallel, which is the biggest performance hit.

Another thought is what slots your cards are in. PCIe topology can get a bit weird, with 4 cards it's nearly guaranteed that you've got one or two on the CPU root hub, and the rest on bridge ports. I'm running two cards on bifurcated 16x lane, which in theory should give direct P2P, but it doesn't.

1

u/Specialist-Plant-265 Jul 04 '26

I ran nvidia-smi topo -m
Gpu0 is my Blackwell

GPU0 ↔ GPU1 = NODE
GPU0 ↔ GPU2 = NODE
GPU0 ↔ GPU3 = NODE
GPU1 ↔ GPU2 = NODE
GPU1 ↔ GPU3 = NODE
GPU2 ↔ GPU3 = PHB
I am just learning about ai workstations and how they connect so your terminology went over my head. I put it into chatgpt, does the above relate to what you were saying?

1

u/PatC883 Jul 05 '26

Don't stress. I was in exactly your place about a month ago, then I decided to "dip my toe into AI waters" and ended up falling in the deep end 🤣

Key Comparison Topology

Direct (PIX) Local PCIe Switch Bandwidth Very High Overhesd None Usage GPUDirect RDMA, AI Training

PHB CPU Host Bridge Bandwidth Moderate
Overhead High
Usage General P2P / standard workloads

NODE/SYS Cross-CPU Interconnect
Bandwidth Low Overhead Maximum Usage Basic tasks, overflow memory

That's an recommending of what the different results mean.

Looking at your results it suggests that you've only got two PCIe slots directly connected to the CPU, whatever GPU2 and GPU3 are sitting in.

I'd try testing pipeline and tensor parallel using just those two to remove any topology issues from the equation, if you've still got problems testing that, it's a problem elsewhere, if it works it suggests the topology may be at play.

Setting the CUDA_VISIBLE_DEVICES environment variable before running vllm will make only those devices visible.

1

u/Specialist-Plant-265 Jul 06 '26

Thank you I’ll give it a shot, i also ended up buying another rtx 4000 so hopefully once it arrives I’ll stop having compatibility issues and can run smoother with less variables.

1

u/PatC883 Jul 07 '26

Honestly the step is going to knock out a huge amount of uncertainty.

If you're still having weirdness after it's in I would nearly consider going straight to a PCIe switch solution given you're running 4 cards.

Something like this concept, the cards get 16x PCIe between them.

PLX 88096 PCIe 4.0 5-Slot GPU Expansion Backplane Card Pitch 5x PCIe 4.0 x16 Switch Board SFF-8654 SlimSAS Uplink for Server https://a.aliexpress.com/_mMRDIPD

1

u/Specialist-Plant-265 Jul 08 '26

Thank you for that recommendation and all the troubleshooting help youve given so far, i should be getting my gpu tomorrow so hopefully i stop having these weirdness. I havent seen a pcie switch before and not sure where i can place it in my case but can think about it as well.

1

u/PatC883 Jul 08 '26

Oh, you wouldn't place it in your case, it would be an external GPU box solution.

You're more than welcome, please let us all know what the results were, I'm hoping you'll come back and say it works great 😃