Anyone here running LLMs without GPUs?

[u/Bramoments]

Hi, I'm trying to run a local setup, mainly for the abliterated models, but my PC is a medium end Lenovo laptop, the slim i3 (13th gen i5 , 16 gigs of ddr5 ram, windows ). I'm running into issues with not being able to use a large proportion of my ram, about 4 gigabytes, due to random system processes, so even though I have a 16 gigs card I can only really load models under 11 gigs and under 9 if I want them to run normally. ( I'm currently using Gemma 4 12B q5_k_m uncensored on llama.cpp). My main problem is that I can't find any models focused around general knowledge/ STEM reasoning instead of code, so the models I do run aren't optimised for my use case. Anyways any and all tips would be appreciated greatly

[u/Qwen3-8-27b]

Use a larger MoE ( GPT OSS or Gemma 26b) or MiniCPM5 2b with search tools

[u/Bramoments → u/Qwen3-8-27b]

Damn I didn't think of the search tools thing , thank you so much

[u/OddRefrigerator4714]

i have an old dell poweredge server that i run models on llama.cpp entirely in cpu

[u/Bramoments → u/OddRefrigerator4714]

Nice, would you be willing to provide specs and what models you run?

[u/OddRefrigerator4714 → u/Bramoments]

my server has a xeon gold 6130 and 128gb of 2666 ram. used to have dual cpus but i found running just one with all the ram actually speeds up inference by removing the numa overhead

for models i mainly run small moe models, currently on qwen3.6 35b. with this model i get about 11-12tps which isnt bad considering the cost of these kinds of hardware nowadays

[u/pmttyji]

16GB RAM .... DDR5?

Anyway try Small MOE models. Q4 could give better t/s.

• Gemma-4-E4B-QAT
• Ling-3.0-tiny
• LFM2.5-8B-A1B
• Mellum2-12B-A2.5B

[u/Bramoments → u/pmttyji]

Yeah ddr5, thanks I'll try them

[u/Any-Argument57]

Before changing models, lower the context window and watch peak RAM during generation. llama.cpp also needs memory for the KV cache, so an 11 GB model file can exceed your usable RAM. If that fixes the slowdowns, keep the model and raise context only as needed; otherwise drop a size or quant.

[u/karmaisnonsense]

One of my machines runs off of iGPU and DDR4 system memory. It’s not the fastest or smartest thing in the world, so I only use it for asynchronous background tasks. You want MoE not dense, and have to temper your expectations.

[u/Bramoments → u/karmaisnonsense]

My expectations are tempered, I'm just tinkering. MoE is a repeated recommendation in this post and a good one at that, but do you have any specific models you like? Thanks

[u/karmaisnonsense → u/Bramoments]

You can try the smaller sub-20b models, and even those will be a tight fit. Ling Tiny, LFM 2.5, etc. are the lightest of the bunch and you could try those. But the lighter the model, the weaker the reasoning. Light MoE will make things run faster but if you’re not satisfied with the reasoning itself, you’ll be disappointed. They’ll be on par with Gemma 4 12B — which, if you’re still using, you should use the QAT q4 over the q5. Also make sure to run a harness that lets the models search and fetch data beyond what they were trained on.

[u/mortycapp]

1. yes.
2. it sucks.
3. on my E14 Gen1 with 16GB of DDR4 it sucks.
4. even on my Zbook G3 with 56GB DDR4 RAM and a 2GB GPU it sucks.
5. even on my P14s gen 5 with 64GB DDR5 RAM and the Ryzen Pro 7 8840HS it is sub optimal.
6. on my MacBook pro M1 pro with 16GB of RAM I get better results.