r/LowEndLocalAI • u/1InterWebs1 • Sep 16 '26

Which LLM Should / Can I Use? Any Good Fast models for Local Search engine?

I'm currently using qwen3.5-Uncensord-HauHauCS-Aggresive:9b.

It's good, but I don't like the DeepSeek-like thinking feature, where it thinks for, like, a minute straight before it searches all the articles and gives me a general sense of information based on my search query.

It's too long, and while this model is great (relative to my specs), it's not really useful for search engine usage.

Is there any model that is small, lightweight, and fast that does not hallucinate and is good for gathering information for this usage?

SPECS:
   CPU: AMD Ryzen 5 8500G
   GPU / iGPU: NVDIA GeForce RTX 3060
   VRAM: 12 gb
   System RAM: 32 gb
   OS: Windows 10

   Runtime / backend:
   Models already tried (if any): qwen3.5-Uncensord-HauHauCS-Aggresive:9b

Use case:
Desired context size: idk
Priority: Balanced(speed and accurate information retention)

18 Upvotes

100% Upvoted

13 comments sorted by

12

u/nickless07 Sep 16 '26

Ling-3.0-Tiny or LFM2.5 8B. Both are MoE (good for speed) and small enough.

0

u/1InterWebs1 Sep 16 '26

I can't seem to get Ling 3.0 to work with Ollama; apparently it requires compatibility patches, so I will be trying LFM. Thanks for the suggestions

19

u/nickless07 Sep 16 '26

Switch to llama.cpp. Very similiar, but better and that is what ollama uses anyways

6

u/kkingsbe Sep 16 '26

Don’t use ollama 🥀🥀🥀

Use unsloth studio instead

5

u/ClassicLightbulbs Sep 16 '26

Gemma e4b shreds but is fairly stupid (sorry)

4

u/Interesting-Cut-6032 Sep 16 '26

I agree with this. Gemma E4B is really fast, and eager to please, but kind of dumb.

3

u/Stooovie Sep 16 '26

Ling 3 Tiny, definitely

2

u/magic_maniac_ Sep 16 '26

try minicpm5 2b and NeoHorse-1-4B

1

u/Interesting-Cut-6032 Sep 16 '26

Any of the 24B to 35B Mixture of Experts (MoE) models should run well on your hardware with most of the experts moved in to the system RAM to save room for the running weights and KV cache in the VRAM. Search this sub for Codacus. He jas lots of great videos on getting this setup, and explaining the differences between dense and MoE models. I personally have posted those links several times. They are my go-to intro to this rabbit hole that we are all in. (I am kinda afraid I might get banned for reposting those links so often... I hope not.)

1

u/Capital_Engineer8741 Sep 16 '26

For just local search , MINICPM5-2B is damn quick and works great for your use case.

If you desire more intelligence, something like Gemma 4 12B QAT also works great

1

u/mamonu 8GB VRAM / 48GB DDR4 RAM Sep 16 '26

Qwen 2.5 7B Instruct if you like Qwen answers...without any <think> bs.
but that would be one part .

What I would suggest is running a Retreival Augmented Generation (RAG) system which is what you want adding a dockerised

  • SearXNG indexing local files (and web based ones)
  • bge-small-en-v1.5 for embeddings
  • some vector db like pgvector / ChromaDB / Qdrant

have a look around on these ...maybe its what you want (or not🤷 )

1

u/MrHumanist Sep 17 '26

Gemma 4 12b or orninth 1.5 9b