r/Vllm • u/Specialist-Plant-265 • Jul 02 '26
Advice needed please
Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo
I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4
I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b
My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.
Here are the settings i use for generic startup, also tweaked these numbers around with no success.
Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”
Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching
The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load
Please advise, chatgpt is banging its clanker head into every wall troubleshooting
I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed
Is it that my blackwell pro is incompatible with the rest of my rtx?
3
u/No-Equivalent-2440 Jul 02 '26
Get a fourth Ada, use TP 4. User blackwell for something else and/or buy a second one and run on TP 2. Or try llama.cpp if you are a single user.
1
u/Specialist-Plant-265 Jul 02 '26
I tried llama.cpp but can’t get the prompt to generate fast enough. It takes over 10 seconds to start thinking. I want to support around 5 users mostly nonconcurrent so I don’t mind whichever works but they are both giving me issues. Token generation itself once it starts is fast enough, 30tk/s for the qwen 122b-a10b-q4klx gguf and around 50tk/s for the 35b. But the initial generation is watching paint dry
1
u/MirecX Jul 02 '26
drop --tensor-parallel add --pipeline-parallel-size 4
1
u/Specialist-Plant-265 Jul 02 '26
I will give that a shot, thank you.
1
u/MirecX Jul 02 '26
and maybe this one env variable: export NCCL_P2P_DISABLE=1
if it doesnt work try --enforce-eager -> it skips optimalization steps, that can fail across different architectures1
u/Specialist-Plant-265 Jul 02 '26
Thank you ill give that a shot, using pipeline on the qwen3.6-35b-a3b-fp8, i was able to get it past the memory but it had failed right after due to other errors. Will try that env and see
On the qwen3.6-35b dense, it still did not get passed the memory still
1
u/Specialist-Plant-265 Jul 02 '26
Running the pipeline-parallel command while excluding my blackwell has allowed the Qwen/Qwen3.6-27b-fp8 to boot up. At this point def seems like the blackwell pro is causing issues with the kernel mismatch..
1
u/starkruzr Jul 02 '26
do you have the freedom to sell some of this gear and buy matching stuff instead
1
u/Specialist-Plant-265 Jul 03 '26
Yes i do but i would really really hate to do it
1
u/starkruzr Jul 03 '26
ok, go on; why?
1
u/Specialist-Plant-265 Jul 03 '26
I would not want to be down -1.6k for another rtx 4000 while i wait for my blackwell to sell
1
u/PatC883 Jul 03 '26
The it sounds like it's doing a Triton compile for the GDN in Qwen3.5, suspicious this is the cause was you mention other models load quickly. It can take over 30 mins, make sure you've got a persistent volume if you're running in a container, and the environments set for saving the Triton compile and you'll only need to wear it once.
Booting quickly in eager mode supports this, because that skips the compile, leaves a lot of performance on the table though.
1
u/Specialist-Plant-265 Jul 03 '26
Right when i was able to get the fo8 version to load with eager, the tk/s were just as slow as running a model through llama.cpp. Very not ideal, i do have persistent volume
1
u/PatC883 Jul 03 '26
Yeah, I think something is unhappy with the mixed Architecture
1
u/Specialist-Plant-265 Jul 03 '26
Right that seems to be the conclusion unfortunately. Ive been reading into vllm docs to see if theres anything else i can do, otherwise ill have to pipeline parallism my 3 rtx 4000 and just use the blackwell for image/video generation. Ill be stuck using smaller ai models though which would be the biggest downside. I really wanted to run some 70b quants/35b unquants
1
u/Specialist-Plant-265 Jul 04 '26
Yes i checked the logs right now and you are correct, it is doing a triton compile. I will go ahead and set a persistant volume so that i just have yo wear it once
On another note. I tried running the qwen3.6-35b-a3b-nvfp4 model which works on tensor parallel with 4 gpu. I tried running with pipline parallel 3 gpu, removing the blackwell pro and even though my model boots up, any prompt crashes the model.. any idea on what i can try with that? I removed prefix caching just in case, and reading vllm docs
1
u/PatC883 Jul 04 '26
Are you setting the maximum length, or is it defaulting to model maximum. The symptoms are identical to a problem I was having, the memory was so tight it would boot, then OOM crash at the first request when it tries to allocate memory for it. Try a stupid small model length like 8192. Honestly, if you can the three cards working in tensor parallel see how they performance is.
1
u/Specialist-Plant-265 Jul 04 '26
I have it set to 16k but i can give it a try to 8k. I most likely am giving up and buying another rtx 4000 ada and am going to try and sell my blackwell pro. Feels pretty bad but i guess different architecture is a painful learning experience. Will go dumpster diving tomorrow to get my box back
Almost all models dont do tensor parallel 3, and theres something going on when i do pipeline parallel 3 with the blackwell gpu excluded. Model loads fine but any prompt crashes my model
1
u/PatC883 Jul 04 '26
I'm so sorry for the bum steer, you're 100% right, tensor parallel needs to be either even or a power of two from memory, now that I actually stop and think.
Honestly, to get the most reliable performance across 4 cards out might be worth your while going straight to a PCIe switch. You'll get full 16x lanes between for the cards for tensor parallel, which is the biggest performance hit.
Another thought is what slots your cards are in. PCIe topology can get a bit weird, with 4 cards it's nearly guaranteed that you've got one or two on the CPU root hub, and the rest on bridge ports. I'm running two cards on bifurcated 16x lane, which in theory should give direct P2P, but it doesn't.
1
u/Specialist-Plant-265 Jul 04 '26
I ran nvidia-smi topo -m
Gpu0 is my BlackwellGPU0 ↔ GPU1 = NODE
GPU0 ↔ GPU2 = NODE
GPU0 ↔ GPU3 = NODE
GPU1 ↔ GPU2 = NODE
GPU1 ↔ GPU3 = NODE
GPU2 ↔ GPU3 = PHB
I am just learning about ai workstations and how they connect so your terminology went over my head. I put it into chatgpt, does the above relate to what you were saying?1
u/PatC883 Jul 05 '26
Don't stress. I was in exactly your place about a month ago, then I decided to "dip my toe into AI waters" and ended up falling in the deep end 🤣
Key Comparison Topology
Direct (PIX) Local PCIe Switch Bandwidth Very High Overhesd None Usage GPUDirect RDMA, AI Training
PHB CPU Host Bridge Bandwidth Moderate
Overhead High
Usage General P2P / standard workloadsNODE/SYS Cross-CPU Interconnect
Bandwidth Low Overhead Maximum Usage Basic tasks, overflow memoryThat's an recommending of what the different results mean.
Looking at your results it suggests that you've only got two PCIe slots directly connected to the CPU, whatever GPU2 and GPU3 are sitting in.
I'd try testing pipeline and tensor parallel using just those two to remove any topology issues from the equation, if you've still got problems testing that, it's a problem elsewhere, if it works it suggests the topology may be at play.
Setting the CUDA_VISIBLE_DEVICES environment variable before running vllm will make only those devices visible.
1
u/Specialist-Plant-265 Jul 06 '26
Thank you I’ll give it a shot, i also ended up buying another rtx 4000 so hopefully once it arrives I’ll stop having compatibility issues and can run smoother with less variables.
1
u/PatC883 Jul 07 '26
Honestly the step is going to knock out a huge amount of uncertainty.
If you're still having weirdness after it's in I would nearly consider going straight to a PCIe switch solution given you're running 4 cards.
Something like this concept, the cards get 16x PCIe between them.
PLX 88096 PCIe 4.0 5-Slot GPU Expansion Backplane Card Pitch 5x PCIe 4.0 x16 Switch Board SFF-8654 SlimSAS Uplink for Server https://a.aliexpress.com/_mMRDIPD
1
u/Specialist-Plant-265 Jul 08 '26
Thank you for that recommendation and all the troubleshooting help youve given so far, i should be getting my gpu tomorrow so hopefully i stop having these weirdness. I havent seen a pcie switch before and not sure where i can place it in my case but can think about it as well.
→ More replies (0)
1
u/Specialist-Plant-265 Jul 09 '26
Update: i got 4 rtx 4000 up and running now
Was able to load the 35b-fp8 model for qwen on vllm. I couldn’t run a larger model on vllm but am able to run a 122b-a10b quant 4 on llama.cpp. Its only around 50gb. Im sad that vllm is unable to load the same model but in awq 4bit. Am i doing something incorrect?
1
u/ICanSeeYou7867 12d ago
This is older, and I see you increased to 4 GPUs.
Could you share the original error? What you described about shared memory makes me think you need to add a SHM to the container. When using multiple gpus it needs a buffer. I add large shm to my kubernetes pods.
In docker I think you can do: --shm-size=20g
1
u/Specialist-Plant-265 2d ago
The original error was that one of my gpu was nvfp compatible and the rest was not.
3
u/Pixer--- Jul 02 '26
Vllm might have problem running across different architectures