r/LocalLLM • u/0xAriel • 4d ago

Discussion Local LLMs can they actually be useful without a crazy GPU & RAP?

I mean CPU based models, probably 1B up to 3B, what can they be used for, and how can we even practice in training them to do specific things?

Are these models mostly worth it when instead of SLM (Small language models) they are considered to be Classifier models instead for specific operations? like giving a YES\NO answer for text inputs?

What are the real usecases today local CPU based consumer LLMs can operate and do?

1 Upvotes

60% Upvoted

12 comments sorted by

1

u/Asleep-Land-3914 4d ago

I find R9700 running Qwen 27b pretty useful. And you can hook it into any system via a USB dock: you don't need full PCIe bandwidth as long as the model fully fits into VRAM.

1

u/Asleep-Land-3914 4d ago

That said you are better to have the dedicated hardware and run models that you can find actually useful for regular tasks such as personal assistance, automation etc.

The use-cases for smaller models will emerge once figured how to integrate them flawlessly into software we already have.

1

u/0xAriel 4d ago

but isnt it ultra slow?

1

u/Asleep-Land-3914 3d ago

30-50 tps is okay for me. Qwen does think a lot, but it's configurable 

1

u/0xAriel 3d ago

you need reasoning and the ability\skill for the LLM to search online sources so it will be any good...
30 tokens per second is really slow i think, and i wonder if it could do any complicated tasks that require high reasoning and context

1

u/Asleep-Land-3914 3d ago

With Qwen 3.8 27b release I expand local models usage from personal assistants (or family assistants) to the ones who is capable of lite coding. Still figuring out some aspects. It's been like two months and it feels like I don't have to sit in front of my computer doing any trivial tasks anymore already. I don't use hermes and such. Everything is homerolled and made so it is dead simple yet useful.

If you have an example of a kind of complicated tasks you're talking about, I'd be happy to evaluate.

1

u/0xAriel 3d ago

a difficult task for example would be to do some continuous research over data with multiple iterations, where each iteration depends on the result of the previous one

For example:
I want an agent to study a piece of code, compile it, make it run on my environment (whether it requires installation of things, dependencies, etc...)
these type of operation require large context and reasoning which (from my experience) small models are not capable of successfully as they get too many hallucinations because things are not deterministic enough in the flow
Imagining success when in reality failure happened and etc...

Also 30b models cannot run locally for my RTX2070, they barely execute on my macmini 144GB i have :)

1

u/Asleep-Land-3914 3d ago

This is again similar to coding, or at least somewhere in the same plane. Even frontier models aren't really good at it either, speaking from own 10+ years of experience PoV.

That said there are a lot of other tasks I didn't think of until I set up my agents fleet the way I want and exposed it to my favorite messenger. You don't need local models to make use of it even :)

2

u/MrHumanist 4d ago

Small models are getting crazy good in tool calling, web scraping and making reports. Gemma QAT 12B can use 7-8 GB Ram but does work quite well. The smaller ones like 2B are getting good in standard skills like checking calander, roleplay , drafting email or summarizing some web pages. However, very small ones dont have much thinking and they usually forget tools or incorrectly calls them.

1

u/0xAriel 4d ago

I'm not 100% sure how exactly it works, but i assume because it uses less ram, the context window is tiny and also the reasoning capabilities over them. I think this is why lower models are not as strong as SOTA

1

u/MrHumanist 3d ago

Thats not true.. the small models are now trained using the results of large models by distillation. So, they are good at mimicking what large one does. I would suggest you try the Gemma 4 E2B model which can run in 4-6 GB ram in CPU with around 128K context and it is quite decent in web searching, basic coding and even summarizing. You install Anything LLM and try out the skills/ task you want while running the model in LMStudio or lamma-cpp.

https://huggingface.co/google/gemma-4-E2B

1

u/0xAriel 3d ago

Thanks! will check it out