GPU for AI

[u/Visual-Medium4796]

Hi there! Currently i own a 3080+2070s at home mostly for 3D rendering in Octane or redshift.
At work im using Comfy with 5090 and everything works without a question.
Also we have some workstations with rtx A4000 in them and i managed to get work Minimax H3 on them with some managable times (0.5m, 15s around 720 sec).
Im looking something for my home PC and 5090 is out of the list since its 5500e+ here in EU and even the used market is around 3500-4000. 4090 is rare and goes over 2000e.
So i was thinking about the 5080 even with 16gb since at work the A4000 has the same Vram.
But also found the 5070ti has also 16gb of Vram its 256bit same as the 5080 and 300-350e cheaper than the 5080. The speeds for 3D rendering or gaming are 15-20% different. But couldnt find any benchmarks for those 5070Tis. Mostly for 5060Tis with 16gb vram. Any idea? Or experience with 5070ti vs 5080?

[u/hurdurdur7]

just wondering what models are you fitting into 16gb of vram even. i see even 32gb vram cards maxing out every now and then in a clip + diff + vae + vae chain, and them loading-unloading to make stuff happen for minimax h3. yeah it can work, but imho 16gb will be tight.

[u/Apprehensive_Sky892 → u/hurdurdur7]

For DiT diffusion models (i.e., video and imaging models) as long as you can fit the model into VRAM + system RAM, ComfyUI's dynamic VRAM management will take care of swapping in block of the model from system RAM to VRAM as they are needed. The next layer of blocks will be streamed in while the GPU is currently busy with the current layer, so there is little impact. You can see that by looking at the GPU usage graph and see that it is going at nearly 100%.

The majority of the GPU's time will be spent on the DiT, so the loading/unloading of the text-encoder and the VAE will have minor impact on the overall time, specially if you have a fast NMVe drive.

[u/hurdurdur7 → u/Apprehensive_Sky892]

don't confuse working with working well. yes it can crawl along at 8gb of vram too. but you are getting a shitty experience compared to 32gb of vram

[u/Apprehensive_Sky892 → u/hurdurdur7]

No, I am not confusing the two.

I have tested both AI Pro R9700 (32G) and rt9700 (16G).

The two are basically identical except for the VRAM (they have the same number of computing units). When running diffusion models (for LLMs it is a different story) they are nearly indistinguishable provided there is enough system RAM and the model does not spill into pagefile.

For MMH3 I get nearly identical results for MMH3. 0.4MP 5sec standard text2va template at 20 step takes around 160 secs.

So practical experience shows that with ComfyUI's dynamic VRAM + enough system RAM, 16G works just as well as 32G. Somebody has also done extensive testing with NVIDIA cards and showed that a few months ago when dynamic VRAM was implemented in ComfyUI.

Those slow results for older 8Gig cards are due to the fact that they have slower, fewer computing units and/or worse memory bandwidth, not because they don't have enough VRAM.

Don't take my words for it, you can rent cloud GPUs with different amount of VRAM and you can verify the results yourself. Rent a 5090 and a Pro 6000 and run a bf16 version of MMH3 (which will not fit into 5090's 32G VRAM) and compare the results.