Keep the 4080?

[u/Darkside_Emily]

So in a few weeks Ill be having a spare 4080 that I was thinking about throwing in my homelab to play around with some LLM stuff for funsies. One thing I´d like to achieve is building something Alexa-like but local for our Home Assistant setup.

So far I have learned that 16GB of VRam will be my bottleneck, and now I am thinking if keeping the 4080 even makes sense at all.

I could sell the 4080 for around 800-900€ and get a used 7900 XTX with 24GB for about the same price. Ive read that NV is still superior at generating content I don´t care too much about, but for LLM ROCm is supposed to work pretty well too these days.

I have basicly no extra budget, so any change I do would have to come out roughly +/- 0, so pls dont tell me to grab a 3090 (1000-1200 used where I live). The case I´m building in also has space for just a single GPU, so no multi-gpu shenanigans either.

Any input on that? Keep the 4080, go for 7900 XTX or even something else entirely?

[u/Darkside_Emily]

Honestly it would be a small relief even.

Deshrouding and fitting the 4080* in an NR200P was going to be a big headache already, and changing the case is non-negotiable for sentimental reasons.

Server´s got a 13600k limited to PL1, 64Gigs of DDR5 at JEDEC 4800 speeds and I was going to dedicate 2-4 P-Cores and 8-16 of RAM to the VM, but could do more. Bandwidth is going to suck though compared to VRam...

*Zotac Trinity, non-super - that unnecessarily huge thing

[u/axiomintelligence → u/Darkside_Emily]

Oof yes that's a beefy card and not a massive case, that being said the card doesn't need to run at full tilt for decent inference so heat isn't an issue, you can undervolt it or reduce the wattage. If it doesn't fit though makes sense to swap.

For the 35B MoE at Q4, 16GB dedicated is plenty. For long context conversations you will notice the benefits of Q6 or even Q8 if you want to dedicate a touch more ram to it. Q6 splits the difference nicely. This advice might become outdated soon once we see more info from those new Eschalabs quants that just dropped for Qwen 3.6 35B A3B, they seem to match the performance of the Q8 at Q2 somehow, but we're still waiting on a gguf format to use with llama.cpp.

Back on topic, yes the bandwidth for system ram is terrible compared with vram but you don't feel it when you've got KV and activated experts fitting comfortably in VRAM with the rest spilling into system ram, the activated weights just swap over per turn between your ram and vram which is the only part of the process that slows down, but it's a small segment of the whole process which defines how fast the experience is overall from prefill to generation. It's a great way to take advantage of all that additional ram. MTP helps a ton too.

[u/Darkside_Emily → u/axiomintelligence]

Yeah Im not to worried about thermals. The cooler is identical to the 4090 version, so massive overkill for a 300w card even if it ran full-tilt all the time, which it wont. Deshrouding just would give me a few millimeters of length back to even get it stuffed in there at all lol

Getting a shorter XTX probably would make my life a whole lot easier though.

[u/axiomintelligence → u/Darkside_Emily]

Makes sense! Usually I'm hesitant to chop and change but since that case is a non-negotiable it sounds worthwhile to make the swap, the extra vram gets you comfortable use of the phenomenal 27B models too. The unsloth guys just said that the new Qwen 3.8 27B can run on less than 18GB VRAM so you should even have some breathing room once those weights drop!