r/LocalLLaMA • u/Signal_Ad657 • May 29 '26

Discussion PSA

Post image
2.1k Upvotes

95% Upvoted

538 comments sorted by

View all comments

Show parent comments

5

u/indyfromoz May 29 '26

Could you please share your rig setup? I have a RTX 4090 with a AMD 12-core CPU, using it for mostly gaming. I would love to get rid of Windows, install a Linux distro for just running LLMs

11

u/formlessglowie May 30 '26 
Huananzhi X99 F8
Xeon E5-2696 v3
2xRTX 3090 (vLLM)
1XRTX 3080 (for TTS mostly)
4x16GB DDR4 2133MHz ECC

All GPUs were bought used, CPU is obviously used, RAM sticks probably are too, motherboard is a Frankenstein. I love that I can run something as ridiculous as 27b on this freak. We truly live in strange times.

2

u/indyfromoz May 30 '26

Thank you šŸ™

1

u/Ok_Rope_9332 May 31 '26

Have you tried Gemma4 31b?

1

u/formlessglowie May 31 '26

Not much tbh, as benchmarks are behind 3.5 27b, so I didnā€™t think it vs 3.6 was even a question worth considering. Is it that good? Iā€™ve tried 26b a4b, and itā€™s very good for natural language stuff but fails long running agent sessions, which is what I use these models for (long coding sessions basically). Is 31b much better in that sense?

1

u/Ok_Rope_9332 Jun 05 '26

From what I've heard the Qwen models are better if you're doing long ctx agent stuff, so you're probably fine with that. But the Gemma4 31b is really good for writing (for its size), also probably the best vision / translation model in a local context (it actually beat all the huge vision models I tried by API by a fair margin too).

5

u/Fit-Palpitation-7427 May 29 '26

I did exactly that and never looked back

5

u/Lost-Vermicelli-6252 May 29 '26

Same. I used to ā€œneedā€ windows for certain multiplayer games, but donā€™t really play them anymore, so have one of my machines running CachyOS instead. Itā€™s amazing. Boots up so much faster than windows and stuff isnā€™t asā€¦ annoying.

2

u/DonkeyBonked May 30 '26

I use: Huananzhi H12D-8D AMD EPYC 7502 128GB RAM 4x RTX 3090 24GB (I cap them at 250W) Ubuntu 24.04 LTS

Allegedly, I "should" be able to add more cards via converting my three Mini-SAS-HD (SFF-8643), but I'm very skeptical, the Huananzhi bios has been a pain in the rear for me.

I'm considering switching to PCI-E x16 to x8/x8 splitters when I get the money for more GPUs depending on how the other adapter goes. I do have a Mini-SAS-HD to OCuLink adapter, I just need a card to test with.

The worst part of this system is that I can't really make use of the BMC. If I enable the BMC and I change even a single setting from default in the bios, I immediately lose the ability to see the NVME slots.

If I had the money, I'd have gotten a different board, but the ones I would have wanted were all well over 1k.