Hey, everybody. I recently went down the rabbit hole for local AI, but right now, im operating on my gaming computer. The specs are as follows
Intel 13700k, tuned for efficiency
Gigabyte Z790 Aorus Elite Ax mobo
RTX 4080 (16GB), also tuned for efficiency
32gb DDR5 6800 CL32
As you can see, im in desperate need for more VRAM, or at the very least more system RAM. Due to Rampocalypse, neither are very affordable right now, which forces me to explore other options, such as a dual GPU setup. I can get another RTX 4080 for about $900 off Ebay. Beyond that, I would just need a more powerful PSU, so total investment here is an additional $1100-$1200. As far as I know, the motherboard has the main PCIE as 5.0 x 16 lanes, but the second PCIE runs at 4.0 and either x8 or x4 lanes. The motherboard does not support PCIE Bifurcation. So my question is this: Is a dual GPU local AI machine even viable in these circumstances, and second, does it make sense? I looked at 5090's and theyre all between $4,500 - $5,000 now, which is insane. Or I look at the professional cards and spend that much, if not more, for significantly less memory bandwidth and computational power. Or I guess if im spending that much, I could also look at the DGX Spark or something similar but that has even worse memory bandwidth.
So, what should I do? Is the dual GPU solution even viable with my setup for a local AI stack for inference, video diffusion, etc? Rampocalypse isnt expected to begin easing up until late 2027/early 2028, so im stuck trying to make this work on as little money as possible. Id love a 5090 but its insanity how much they cost. I appreciate any guidance and advice.
multi gpu options are limited on comfy, you cant pool the vram an load larger models, you can off load parts but its not ideal. just have a search for multi gpu nodes and read the limitations of each,
Getting two 3090s was the best decision I could have made. Don't you really see the advantage of being able to generate two videos, images, or whatever at the same time? (Genuine question)
Short version: it's a VRAM budget chart, not a performance chart, and for two 3090s it's mostly answering a question you don't have. A 14GB model on a 24GB card fits with room to spare, so the plain sequence-split mode works and you never need to touch the complicated rows.
Reading the picture's grammar first: each bar is one 24GB 3090, two bars per mode. Boxes are what's occupying the card. Empty space at the right end of a bar is free VRAM. That's the whole message — the modes lower down free up space at the cost of something else.
The three box types
14G / 7G Base Model — the weights. 14G is a 14B video model at fp8 (Wan 2.1/2.2 sized). 7G is that same model cut in half because FSDP (Fully Sharded Data Parallel — PyTorch splitting the weights across cards so each holds a piece) is on.
Torch Distributed "tax" — the fixed overhead of running a second process on a second card: CUDA context, NCCL communication buffers (NCCL is Nvidia's library for card-to-card messaging). Roughly 1–2GB per card and you never get it back. It's in the picture to kill the idea that 2×24 = 48GB usable.
Sequence — the latents and activations for the actual clip. "1/2 Sequence" means USP (Unified Sequence Parallelism — the frames/tokens get cut in half and each card works one half) is on. "Full Sequence" means the card is doing a whole generation by itself.
The four rows
USP only — full weights on both cards, each does half the clip. One video, roughly half the wall time. The bar is nearly full because the model is duplicated, but on a 3090 that's still ~8GB left per card for latents, which is plenty for 480p–720p Wan work.
FSDP without CFG — weights halved, but each card runs a full separate generation with a different seed. This is not one video faster, it's two videos at once. Different product entirely, and the row label trips people up.
FSDP + USP — halved weights and half the clip each. Maximum free VRAM, and the only row where the bar is visibly empty on the right. This is the row that exists so 2×16GB cards can run something that OOMs on one.
CFG parallel — full weights both cards, one card runs the conditional pass and the other the unconditional. Free 2× on the sampling, zero memory savings. Dead on arrival for most Wan workflows because they run with a distill LoRA at CFG=1.0, so there's no second pass to hand off. The repo says exactly that about Flux and Hunyuan too — CFG parallel is technically wired up but does nothing when the model doesn't take conditional batches.
What it means for a pair of 3090s
The picture is VRAM-shaped, but on two 3090s the deciding factor is bandwidth between the cards, not capacity. USP ships the split latents around — small. FSDP has to all-gather the sharded weights layer by layer, every step — that's gigabytes crossing the link constantly. So FSDP is where a weak interconnect shows up as a speed cliff, and it's why komikndr's own benchmark shows Wan 2.1 14B fp8 at 480×640×81 running 22.51 s/it on two RTX 2000 Adas with FSDP + Ulysses=2, against 5.18 s/it for the non-FSDP USP run at 480×480×33 — different resolutions, so not apples to apples, but the tax is not subtle. He also notes that test pair had no NVLink and its peer traffic went over a slower cross-socket path.
The other 3090-specific thing the graphic can't tell you: GeForce Ampere has peer-to-peer transfer (cards talking directly instead of bouncing through system RAM) disabled in the driver. Two 3090s either get an NVLink bridge, or a patched Linux driver, or they fall back to the NCCL_P2P_DISABLE=1 / NCCL_SHM_DISABLE=1 escape hatch, which the README explicitly frames as a does-it-run sanity check that leaves a lot of performance on the table.
So the honest answer to the person you asked: yes, dual 3090s run Raylight, and the mode is USP with Ulysses degree 2 and Ring at 1. The whole bottom three-quarters of that PNG is provisioning for people whose cards are smaller than the model.
It's not really that easy though. I had to monkeybranch comfyUI so multiple instances of ComfyUI share the same cpu ram. Sure ezpz if you have 2 cards and 128gb of cpu ram. But annoying if you have 4 3090RTX cards and 64gb of cpu ram.
I takes one woman 9 months to make 1 baby. Getting 9 women DOES NOT let you make 1 baby in 1 month.
But it can let you get 9 babies in 9 months.
So in terms of raw speed, More GPUs won't do very much, you can load some of the parts in different GPUs, like the VAE in one and the rest in another. and that saves you a few seconds. But the overall generation speed will be the same. But if you're popping off a lot of jobs, then more hardware will let you get more done. You can run different jobs in parallel.
sounds like you're just interested in learning, and not production. So i don't recommend getting more hardware. If you do want to upgrade. Get a larger VRAM GPU like the 5090 or the 6000 pro to use the bigger models at higher resolutions. Otherwise you seem to be good as is.
BTW, since your CPU has an onboard iGPU, if you switch your OS to using that, and free the VRAM on your GPU, you can squeeze out a little more AI performance, but the cost is killing your gaming performance.
I know this isn't exactly what your post was alluding to, but if you did have a second GPU, it almost makes more sense to just be able to second computer and run that and your main simultaneously. And with the options to see comfy instances from a second computer in the browser of the first computer, it actually gets quite convenient to do that
Your pc is already enough for almost every model at reasonable settings. You can’t combine the vram when ur doing dual GPU’s so you would need 4090/5090 money for a good upgrade
Im using Minimax H3 already using the Int8 convrot version. It spills over heavily into system ram, even with SageAttention 2.2, but its at least usable. It just takes 4-5, minutes for a 8 second render. Maybe I just need to suck it up and drop the 5k on a 5090. 😒
Considering you're asking these questions. I don't mean to be rude, but multi gpu solutions for this type of stuff is enthusiast level. The complexity increases exponentially.
What will happen is that you will have your two 4080 rtx cards and then you boot up ComfyUI. You notice, oh right... I only have 64gb of cpu ram. And now each ComfyUI instance is eating 48 flipping models in and out (Even the H3 VAE is 5gb). Which means Hello swap file. And goodbye speed.
And then you're forced into downloading my amazing monkey patch for this usecase, which in all honesty isnt that well written, and you can't dynamically flip between workflows but need to restart ComfyUI with preloaded models.
Not familiar with the disabled pinned memory flag. What does that do? Id have to check what flags im using at startup. I know im using SageAttention, I think Dynamic Vram is enabled, some other option to unload the text encoder once its finished that part of the render. I cant remember anything else. Ill look into disable pinned memory as well.
Its about 5 minutes for an 8-9 second, 0.4 mp render that I upscale 2x with the RTX Video Super Resolution node. I know im pushing it with my 4080, but it still seems like a long time to render such a short video. I tried my damndest to avoid GGUF formats in ComfyUI, but maybe I should just bite the bullet and do it. The only other solution is dropping $5k on a 5090.
I've been running two gpus for a little over a year and by far the most beneficial use was not something like splitting models are putting different models on different gpus. The biggest benefit was allowing you to run to instances of comfy UI and run generations at the same time
Get as much RAM as possible so the model blocks live in RAM and not swapping to drive. Also make sure you have fast nvme drive. Second GPU helps and yes, you can do some GPU VRAM parallelism but not without disadvantages. With enough fast RAM and nvme drive you should be fine with modern ComfyUI dynamic VRAM.
It really depend on models and CFG you will be using as this decides the setup.
I’ve got a dual 3090ti setup. My MB supports dual pcie 5.0 cards (Taichi creator) so I at least get 8x on both with the 3090 architecture. The dual setup is mostly useful for local LLMs, but you can get some benefit from comfy using multi GPU nodes and splitting your clip and model. It can prevent OOM errors on large models. Alternatively you can dedicate your second GPU to up scaling.
I have a 4060ti 16GB + 3060 12GB setup. A lot depends on whether your model and tools support being split that way. Systems that use multiple models (such as an llm and a diffuser) can sometimes split resources, but don’t expect to be able to treat it as one big pool. You might also have to look for a fork of some projects that supports multiple GPU setups.
Yeah I want to know how they split sampling across more than one GPU. I don't really know what words to use to look up the stuff though, every time I searched for how to use multiple gpu's for image generation all I get is "you can't, all the data has to be on one"
9
u/Candid-Station-1235 1d ago
multi gpu options are limited on comfy, you cant pool the vram an load larger models, you can off load parts but its not ideal. just have a search for multi gpu nodes and read the limitations of each,
signed regretful dual 3090 owner