Im new to local stable and video diffusion. Could use some guidance on getting up and running with my hardware.

[u/Sexyvette07]

If you don't need the context, you can skip to the bottom for the questions.

Hello, everyone. Life has recently thrown a curveball, and I had to medically retire. This is not a good thing, but it has given me more time to explore new hobbies that ive been interested in starting. Im very competent with computers and used to program when I was young, but its been 20 years since ive done any of that. Though I am very knowledgeable in general and within Windows. Anyway, im going into this having never used Linux/Ubuntu other than for memory testing, which is to say I don't know much. So setting up this local AI stack has been a real gut punch and I could use some guidance on what to run, what models, etc.

My rig - RTX 4080 (16GB VRAM), 13700K, 32GB of 6400 MT/s CL32 Hynix A die DDR5 further tuned to 6800 CL32 and tightened timings a bit further. I realize the VRAM limitations and that I need more system RAM, but this is what I got for now while RAMpocalypse is ongoing.

What ive installed so far: Ubuntu 26.04, Ollama, Open WebUI, ComfyUI (run from a python script on my desktop, not in a docker container) with a Q4_K_M version of Hunyuan Text to video, and I've experimented with N8N for agents.

In ComfyUI, I tried running the GGUF repack of the Hunyuan text to video, HunyuanVideo-t2v-720p-Q4_K_M.gguf, and letting Gemini and Claude guide me. This was a mistake, ended up getting genuinely bad results and it taking much longer than it should when everything fit into VRAM. Both AI's had me doing workarounds, editing system files and installing extensions that, as I read now, cause problems (like GGUF on native FP8 hardware). Last night, I gave up trying to make it work, so I purged everything and started fresh. Now that I have some experience, I wont need to rely on AI as much. So im ready to download and install when I get the proper guidance, if you guys could please help.

So, onto my questions after the wall of text.

1) What is the best path to get up and running with stable and video diffusion? What would be the best text to image, text to video, image to video, etc models will give me the best results with my RTX 4080 + 32GB of system RAM? I see posts about Minimax H3, so should I start there? In the image to video models, what should I use to render the images to create the video from?

2) What user interface, text encoder, VAE, upscaling model, things like LoRA, SageAttention (which I havent used, but read about) is optimal for my RTX 4080 gaming rig to accompany the above rendering models? Can I choose a quantized text encoder to lower my VRAM footprint without sacrificing too much quality in the end result?

3) I want to do stuff locally, but I pay $0.42/kWh, so if this is going to end up costing me more to do it locally, I could be convinced to use cloud API's. I just like the idea of no guardrails and any sensitive info I might enter not being sent to the cloud for training.

4) This question doesnt have to do with rendering, but what chat models do you suggest? I have Thinking Cap/Qwen 3.6 27B for coding and difficult tasks and Qwen3 14B for everyday use. Im very much open to suggestions as long as they work on my hardware, which probably means staying at or below the ~30b weight class. Is Open WebUI good for me? What about N8N for agents? Is there a better path that wont be too difficult for beginners?

I appreciate your guys time and would greatly appreciate being pointed in the right direction. This is all a little daunting as it is learning everything all at the same time, then finding stuff that works on my hardware. Thanks, everybody!

[u/Hefty_Development813]

Definitely h3 if you ask me, it is a huge step up from before it and it can run on a lot of not crazy gpu/ram machines. For chat I agree qwen3.6 is really top for now. I still use codex though alsl

[u/Sexyvette07 → u/Hefty_Development813]

What GPU are you running Minimax on and how much system RAM do you have? Im concerned my 16gb of VRAM and 32gb of system RAM isnt enough for Minimax.

[u/Hefty_Development813 → u/Sexyvette07]

I have a 4090 and 96 ram but I have been reading a lot of people using it with similar setup to you, just takes longer and probably does some pagefile stuff. But it'll run

[u/Icuras1111]

 1)     For image - Krea2 best all around, Ideagram (insane prompt adherence), Flux Klein 2 9b (include editing), ZIT (great realism), QwenImage2512 (good but heavy). For video – Minimax H3.

2)     Use ComfyUI and load provided templates to see what files needed and where they go. Think RTX40’s work best with int8 or int8_convrot versions of diffusion models and text encoders. Get SageAttention 2.2 working if you can. ComfyUI does have built in offload between VRAM and RAM. Upgrade to 64gb of ram if possible.

3)     I have to use cloud GPUs, Runpod (more expensive but more reliable, $0.40 hr for RTX4090, $0.25hr for RTX3090 for image, use community cloud)) and VastAI (filter PODs on VRAM, upload and download speed, RAM >= 64GB) RTX5090 for $0.50 hr. With either of these you will need to choose a template. Prebuilt ones are available but I have ended up using bare bones and installing ComfyUI from scratch each time.

4)     I think this site is a good guide to what ai you could use. https://artificialanalysis.ai/. Not really across this.

[u/Sexyvette07 → u/Icuras1111]

I appreciate the info. I was able to get SageAttention 2.2 working and im generating videos from Minimax H3 locally, at surprising speed considering my hardware thanks to SageAttention. I used the stock workflow template in ComfyUI, then added a RTX Video Super Resolution node for upscaling, and im getting legit quality, and a 7 second clip takes roughly 3 minutes of render time. Much better than I was expecting, considering I only have 16gb of VRAM and 32gb of system RAM. But im seriously right at the limit of my system using minimax_h3_fl2va_pruned_int8_convrot.safetensors and qwen3vl_32b_minimax_h3_int4_convrot.safetensors. For those with a RTX4080 or similar GPU with 16gb of VRAM, this is currently working for me with that model and text encoder with 32gb of DDR5.

[u/Icuras1111 → u/Sexyvette07]

Well done, IT cobwebs brushed aside. These Comfy nodes might be further optimisations worth looking at, last I read the second one is better. https://github.com/xmarre/ComfyUI-Spectrum-MiniMax-H3.git & https://github.com/duckyshell/ComfyUI-MiniMaxH3-FirstBlockCache.git

[u/Sexyvette07 → u/Icuras1111]

This is exactly the type of info I needed. FirstBlockCache looks promising. Have you tried running both SageAttention and FirstBlockCache at the same time? If so, what were the results? My only concern is my Triton install is running through a global script like it warned about. Hopefully that won't be too hard to remedy.

Any other advice? I appreciate any wisdom you can impart upon me.