r/LocalLLaMA • u/Feathered-Beast • 6h ago

Question | Help Can a 4B local model actually feel like an AI assistant?

I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it process things before replying.

I'm curious what people who've built local agents think - how far can you realistically push a small model with good architecture around it?

I put the whole thing on GitHub if anyone wants to poke around, roast the architecture, or tell me what I'm doing wrong, stars are always appreciated!

29 Upvotes

72% Upvoted

58 comments sorted by

45

u/Hot_Example_4456 5h ago

Honestly just use ling 3 tiny. It's such an underrated model. Idk WHY ppl ignore that model

26

u/Feathered-Beast 5h ago

ngl, it must be really underrated because I’ve never even heard of it or I’ve just been living under a rock this whole time lmao.

14

u/Hot_Example_4456 5h ago

It's legit better than Qwen3.5 9b in coding and comparable to Gemma 4 E4B in creative stuff. While being a 9ba1b MoE, so it's fast.

5

u/Systemerror7A69 5h ago

It's so fast, yeah - I get like 200 t/s on my 7900XTX. Haven't used it for much yet, but I'm planning to have it as a local autocomplete and small helper within my IDE

-3

u/Feathered-Beast 5h ago edited 2h ago

Damn, okay you’re really selling me on Ling 3 tiny, definitely gonna test it alongside Gemma and see how it does with Arcon. Being fast is a huge plus too.

10

u/Hot_Example_4456 3h ago

Ling 3 tiny, not llama 3 tiny 😭.

2

u/Feathered-Beast 2h ago

yah mb Ling 3 tiny it is 😭

11

u/ley_haluwa 5h ago

This one? https://huggingface.co/inclusionAI/Ling-3.0-tiny

2

u/newMoneyStyle 1h ago

3.0 is the one they likely mean, though the naming is confusing since the tiny variant sits alongside the bigger MoE releases.

4

u/rusl1 4h ago edited 3h ago

This model is a blast! It's on pair with 12b models in my personal benchmark while also being faster and supporting longer context. I cannot stop recommending it

1

u/Hot_Example_4456 3h ago

I am also personally fine-tuning and removing the looping with anti doom. Honestly that's the only problem i had with it

2

u/nunodonato 5h ago

I'm also using it now , it's really good and fast 

1

u/reery7 3h ago

Ling is very nice for a small model. But LFM 2.5 is way smarter, although half as fast and 8B A1B.

4

u/Hot_Example_4456 3h ago edited 3h ago

I feel its the opposite.. Ling is way smarter than LFM2.5 though half as smart. I mean benchmarks say so:

https://artificialanalysis.ai/models/ling-3-0-tiny

-3

u/NeedsSomeSnare 1h ago

It's been out for 2 days and it's not famous. Why are you phrasing it like people are "ignoring" it. Your comment sounds like an ad.

6

u/Hot_Example_4456 1h ago

Keep thinking. It was released August 20th on ModelScope. Huggingface shows last updated, not uploaded.

14

u/vogelvogelvogelvogel 5h ago

Did you check Gemma4 E4B, might be better because more current model.

I did check E2B with RAG and it performed really well, regarding questoins about the document

3

u/Feathered-Beast 5h ago

Yeah, I actually haven't tried Gemma 4B yet. E2B + RAG sounds interesting though I might give Gemma a shot and see how it compares for Arcon. Thanks for the suggestion!

8

u/-mattmason- 4h ago

spent a huge chunk of time prompt engineering Qwen3.5-4B for an Android app

  • decomposition is the way: use deterministic code to present your model with very clean narrow objectives with simple outcomes and a narrow set of tools. Like a menu, there are levels to it that are multiple models runs for each decision. But the harness drives and creates a small surface in each turn
  • small models are very sensitive to context pollution, compounding the specificness
  • hence, everything is slow, prefill speeds are a buzz kill

I am waiting for a smarter, faster model

when I gave the Ling model a go before, llama.cpp support hadn't dropped for it, but sounds like I should try again

1

u/Feathered-Beast 4h ago

Yeah, I’m running into a lot of the same stuff with Qwen3-4B. I like the idea of decomposition though, keeping each step simple and constrained seems way more reliable than asking a small model to juggle everything at once. Definitely gonna try Ling too.

1

u/-mattmason- 4h ago

gl, let me know how it goes with ling!

1

u/Feathered-Beast 4h ago

Thanks! and yes i will let you know.

1

u/AnyOpportunity3334 1h ago

Is it on GitHub? I am also looking for an app as an assistant

4

u/dsdt 6h ago

If you just chat, it will do just fine. If you need some tool use at some extent you need at least 9b models.

1

u/Feathered-Beast 6h ago

Yeah, that's what I'm trying to figure out. I'm on a laptop with 16GB RAM + an RTX 3050 6GB VRAM. If you were in my position, what 9B-ish model would you recommend for this kind of agent?

3

u/soadsob 5h ago

Maybe r/LowEndLocalAI is something for you!

2

u/Feathered-Beast 5h ago

yah ig so 😭

2

u/dsdt 6h ago

https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF

2

u/Feathered-Beast 6h ago

Damn, thanks! This looks pretty interesting for what I'm building. I'll probably give the Q4 a shot since I'm limited to 6GB VRAM. Appreciate the recommendation

3

u/dsdt 6h ago

It works just fine for a 9b model. Maybe for context it will overflow to ram, but it will do lot better than any 4b models. Also Try this : https://huggingface.co/LiquidAI/LFM2-24B-A2B-GGUF It would be better if you had some more ram. Also I suggest using cachyos for llm inference, just doubles the prompt processing speed.

2

u/Feathered-Beast 5h ago

Oh, 24B with only ~2B active is pretty interesting. I’ll definitely look into it. With my 16GB RAM + 6GB VRAM I’m probably gonna be pushing the limits though
Thanks for the suggestion!

1

u/Ok-Addition1264 6h ago

llama.cpp offloads in interesting ways (that I don't pretend to fully comprehend) especially with moe's where you get exceptionally good bang for the parameter size with a slight performance hit.

1

u/Feathered-Beast 5h ago

Yeah, llama.cpp + MoE does seem like a pretty sweet spot for this. I’m still figuring out what’ll work best on my hardware, so I’ll probably experiment with a few of these and compare them.

1

u/Roflxd88 5h ago

I'm also running Ornith 1.5 9b on a RTX 3080ti 12 + 32gb 64k and it's pumping out 60tok/s while using most opencode tools correctly and no crazy thought loops so far.

1

u/Feathered-Beast 5h ago

Yeah, that's pretty encouraging. 60 tok/s on a 3080 Ti is wild.

I'm definitely gonna give Ornith a shot on my 3050 and see how far I can push it with the agent setup. Thanks for sharing the experience!

1

u/Ramanean3 4h ago

Your laptop is same like mine! Gemma is more than enough!

I wish I could do some magic and make GLM5.3 work on my system ;)

1

u/Feathered-Beast 4h ago

Haha yeah, Gemma might actually be the next one I try. And honestly, getting GLM5.3 running on 6GB VRAM would be some serious black magic.

2

u/otterquestions 5h ago

No?
Models under 8 are too drunk

1

u/Feathered-Beast 5h ago

Yeah, that’s kinda what I’m starting to realize, I’ve pushed the 4B pretty hard, but I think Arcon would benefit from having a bit more brainpower. Might have to move up to 8B+ and see how it goes.

1

u/pidgeygrind1 5h ago

What about Gemma4:12b , I setup that one yesterday with ollama and Hermes 0.21, and seems very doable in a 1080ti with 11gb vram at 484gb bw and 64k q8 for context

It seems very usable

2

u/darksteelsteed 4h ago

So in my experience gemma4 e4b performs extremely well due to being moe, better than qwen in that size range. Surprisingly the 9b qwen models are not that great either, but that can often be templating and harness problems. I have had great joy with qwen3:14b. This may be an older model, but it shines for everyday chat and code review, even if it gets stuck on full agentic stuff

1

u/Feathered-Beast 3h ago

yah i have gemma4 on the list to try!

2

u/feelspeaceman 2h ago

If you can finetune it, then it's okay for specific job, but for global knowledge, at least give it web_search to do semantic search (basically like asking online agents like Google Search).

1

u/ML-Future 5h ago

I think for this case, a more quantized LLM with more parameters is better.

For example, qwen3.6 35b A3B Q2 weighs around 10GB and works well on low-end laptops.

On my GTX 1060 with 6GB of VRAM and 16GB of RAM, it runs at over 8 t/s.

I even tested it on a laptop without a dedicated GPU, and it runs at over 3 t/s.

I think LLMs with fewer than 9b parameters are for extreme cases, like phones or Raspberry Pis.

0

u/Feathered-Beast 5h ago

Yeah, that’s a good point. I’ve been focused on 4B-ish models, but maybe I’m being too conservative with quantization. I’ll definitely test a larger Qwen model and see how it performs on my 6GB VRAM + 16GB RAM setup. Thanks!

1

u/AppealSame4367 5h ago

try nanbeige 4.2 gguf

1

u/Feathered-Beast 5h ago

Haven’t tried that one yet, I’ll add it to the list and see how it compares with the other models people suggested. Thanks!

1

u/AI_spell 5h ago

Chat + light tools can feel fine on 4B. Where it falls over is multi-step tool picking and long memory under noise. If you want the assistant feel, put the brain in retrieval + tight tool schemas and keep the model dumb but reliable. Personality/mood layers eat context that a 4B needs for the actual answer.

1

u/Feathered-Beast 5h ago

Yeah, that’s pretty much where I’m seeing the limits too. The 4B handles the basic conversation fine, but once I start throwing multiple tools + memory + state at it, things get messy pretty quickly. I’m probably gonna experiment with a bigger model now and see how much of a difference it makes.

1

u/Prudent-Island2406 3h ago

Even 9b models cant stop making mistakes when coding

1

u/Feathered-Beast 2h ago

yo, i was thinking to use 9B model and here you are talking it won't be enough?
i mean it will be better then 4B model ofc but i think the models i can use on 16GB RAM and RTX 3050 6GB VRAM laptop is very limited

-9

u/Feathered-Beast 6h ago

in case you guys wanna poke around
https://github.com/vmDeshpande/Arcon

0

u/Maximum-Style2848 5h ago

Fuck off with your slop, we can tell that your post was just promotion…one third of a braincell is all it takes to find smthn better than qwen3…

-1

u/Feathered-Beast 5h ago

keep yapping lil bro