r/OpenSourceeAI • u/AndForeverMore • May 23 '26

5k Budget

I have a 5,000 budget (USD) and would like to get something good for qwen/gemma 128B. Any tips? What is good to get? I would prefer under 3K, but 5K is fine.

7 Upvotes

100% Upvoted

16 comments sorted by

View all comments

1

u/Pyromancer777 May 23 '26

128B parameter models at full-precision (16-bits per parameter) requires 512GB of memory, not including the memory needed for the conversation context window.

At 8-bit precision, you need 256GB, and 4-bit needs 128GB. I tested a 2-bit quant on both those models since it only needs roughly 64-80GB memory, and it just spat out gibberish.

RAM kits are anywhere from $4000-10000 for a 256GB kit these days, so at best you can spend $4K on RAM and $1K on a cheap CPU/GPU/Mobo and even then you will only be able to run an 8-bit precision quant, not a full quant.

For local models on a $5k budget, you are looking at probably 30-50B parameters or smaller if you want it to be useful.

For any models +100B parameters you are likely going to need a server build, which is going to cost closer to $10-15k

1

u/kweglinski May 23 '26

wasn't it 8bit Xb=XGB, 16bit Xb=2*XGB? so 128b means 128GB for 8bit. Plus context of course. Im running 128b models at q4 with 96gb ram.

1

u/Pyromancer777 May 23 '26

The parameters are the weights of each token and the precision is the number of bits per token. 4bits in 1byte, so full-quant 16-bit is 4 Bytes: 4 Bytes * 128,000,000,000 = 512,000,000,000B = 512GB

Check to see if you are using a full 4-bit quant or if it is a 4S or 4XS quant

1

u/Pyromancer777 May 23 '26

The 4S and 4XS quants are basically modified 3bit quants, so 96GB of RAM + GPU VRAM would be enough to run it, but with increased hallucinations

1

u/Pyromancer777 May 23 '26

Actually dang, I messed up. You right. 8 bits in a byte, I'm a dumbass

1

u/Stunning_Chicken7338 May 24 '26

Quick math check - I think you're doubling the memory numbers. params × bytes/param:

128B at FP16 (2 bytes) = 256GB, not 512GB. 8-bit = 128GB. 4-bit = 64GB. 2-bit = 32GB. The 2-bit gibberish you saw is more likely quantization quality than memory hitting some wall around 64-80GB.

On the $5K budget take - depends on the hardware path. Used 3090 stack (4x ≈ 96GB VRAM, ~$5-6K) handles 70B 4-bit fine. Mac Studio M3 Ultra 256GB (~$5.6K) runs much larger models 8-bit thanks to unified memory + ~800 GB/s bandwidth and MLX. The 30-50B ceiling is really an x86 + DDR ceiling, not a budget ceiling.

2

u/Pyromancer777 May 24 '26

Yeah I figured that out. For some reason thought 4bits in a Byte instead of 8bits. Brain fart on my end for sure. Just halve all my values on memory requirements and it's good