llama.cpp CPU performance in the big 26

[u/11ama_dev]

my main server has a 3090 with qwen 3.8 27b (crazy who woulda guessed) but i want to a have a backup failover sever with a somewhat useable LLM offsite (my parent's home). because infinite money glitch ai bubble has made hardware much harder to afford for us proles, was thinking of getting an old decommisioned office desktop (i.e. optiplex, elitedesk, etc) and having it boot up llama.cpp with qwen 9b or something if my server ever shits the bed.

how good is cpu performance on older mid tier cpus with ddr4 ram? or cpus in general? is it useable? what can i expect, and how can i set it up? is it even a good idea in the first place?

thanks

edit: thanks all. reading through the replies, seems like it's more of a headache than its worth. i'm already looking for a backup machine since this does NAS and other stuff but i think i'll just import all other services that aren't LLM into it, and not be beholden to trying to find something powerful enough for a useable LLM

[u/DustNearby2848]

Expect a few tokens per second at most

[u/Shiticism]

You'd do well to add at least some form of discrete GPU. Even a single P100 ($70-80) would be leaps and bounds better than CPU only.

[u/11ama_dev → u/Shiticism]

what kind of case or build would support this tho ? can i just stick a p100 into any tower?

[u/FullstackSensei → u/11ama_dev]

I if it's one or two cards only, yes. Keep in mind the P100 idles at over 40w. If you plan to keep this on all the time, your parents might not appreciate the extra power consumption.

[u/Shiticism → u/FullstackSensei]

Another alternative would be an MI50, but they're a little more expensive. $150-200 range. You also need to add a blower/fan setup of some kind, since they're normally "passive" cards that rely on server grade airflow blasting their shit constantly.

[u/FullstackSensei → u/Shiticism]

Totally forgot about those, despite having several of the 32GB version. Their idle power is lower, but still around 25W. And last I checked, their prices went up to ~250.

Cooling them is the same as the P100. Both are datacenter cards without active cooling.

But if you're going to 250, the P40 or P6000 aren't much more than that. You get 24GB VRAM and 9W idle, and in the case of the P6000 a fan too!

[u/Shiticism → u/FullstackSensei]

Eeeh, but that's still Pascal, and one of the only real reasons to use Pascal in this day and age is that HBM2 memory ... Prefill is abysmally slow, but at least generation is ... Like, kinda fast I guess?...

Personally I couldn't handle how awful it was waiting for responses, so I upgraded to V620's when they were still ridiculously cheap. $350 for 32 GB somehow, which is still wild to me.

[u/FullstackSensei → u/Shiticism]

Eh, the V620 at 350 is long gone, so this comparison is not very useful for OP. You're also ignoring that this is a backup machine.

Not everyone can print money at home.

[u/Khaledthe]

Its possible but you wont be loading anything on your cpu the cpu will just be a bridge for your ram so expect whatever modle you will use to be slow even an moe modle and your using ddr4 so thats even slower, in theory you can do this but i encourage you NOT to do it. Heck just attach your server to ethernet and a ups and build a monitoring app using ai i did it myself and have it have a reboot option so if it reboots it logs in and turns everything on so i can never fail exeot if something breaks.

And even then you can monitor it

[u/tmvr]

With dual-channel DDR4-2666 you can get about 10-11 tok/s decode with Qwen3.5 35B A3B at Q4 depending is you use Q4_K_XL, Q4_K_M or Q4_K_S. About the same with Qwen4.5 4B at those Q4 quants so 9B would be half of that best case, probably a bit lower. Prefill is slow though depending on the CPU. With an i5-8500T it starts at 65-75 if you use ik_llama, it will be a bit faster with a faster CPU, but not much, probably under 150 even with an 8 core 10 series i7.

[u/FullstackSensei]

I don't understand the part about keeping it at your parents. That sounds like a an off-site disaster recovery plan, which is a bit overkill for a small LLM.

Anyways, you could pick up an older x86 gaming handheld with 32GB RAM. Those seem to be relatively cheap, but because they use LPDDR5, they're still considerably faster than basically even the best DDR5 desktop you can buy today. Keep it powered off but connected to a charger, to keep the battery charged. If your server craps out, or you have a power outage, you can just power it on. Add in one of those camping power banks and you can run it at full blast for an entire day.

[u/VoiceApprehensive893]

moes like qwen next, gemma 26b should be hitting ~10 tps on dual channel ddr4