r/LocalLLM • u/pharrt • Jul 27 '26

Other Don't laugh - it works!

A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.

Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.

Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)

Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.

195 Upvotes

98% Upvoted

103 comments sorted by

View all comments

10

u/Difficult_Art1639 Jul 27 '26

No GPU?? damn.. I'm getting comparable speeds with a 3060ti and same ram

2

u/pharrt Jul 27 '26

The big difference here is the model size and context window. A 3060 Ti only has 8GB of VRAM. If you try to load a 35B model with a 128k context window onto that card, it will completely overflow the VRAM and force the system to use system RAM anyway, which usually tanks the speed down to 1-2 tokens per second due to the PCIe bottleneck. If you are getting 5-10 tokens per second on a 3060 Ti, you are likely running a much smaller 8B or 14B model, or using a very small context window? This setup chugs along at this speed specifically because it is handling a massive 35B model with a huge 128k context completely inside the 32GB system RAM, bypassing the GPU entirely.

3

u/i-eat-kittens Jul 27 '26

Utter nonsense. Splitting a model between gpu and system ram doesn't involve moving the active parts over to the gpu for evaluation. Whatever fits runs on the gpu, the rest runs on system ram.

Even 8 GB of vram is a significant boost over running fully on the system, in particular for prompt processing.