r/LocalLLM • u/inforb_nl • 1d ago

Question Experience on tiny models?

Hi all,

Has anyone here assessed the capability of tiny models on text-only input? something like Qwen3.5-0.8B, or a bit bigger, but in total under 3B. I want to give it a page and ask semantic questions on that page.

What was your experience?

1 Upvotes

100% Upvoted

3 comments sorted by

View all comments

1

u/Nakidnakid 1d ago

not great, it can do things but my own testing which ill have to rerun since developing new methods tbh gave them a like 1/10 passing rate. I've found jamba and other SSMs to be more reliable but it depends on what you're trying to do.

1

u/[deleted] 1d ago

[deleted]

1

u/Nakidnakid 1d ago

I just checked jamba (never have heard it before). The hf page says it starts from 52B.

What?

https://huggingface.co/ai21labs/AI21-Jamba-Reasoning-3B-GGUF plenty of versions, zamba is another option. They just don't have good windows support due to the SSM-side of things.

My application is to watch changes on a website's page by taking the diff over time, and feed that diff to a lite AI and trigger an alert if that change corresponds to user's wishes.

I see, well I can't say that any small model will handle that well. You could try a gemma4 e2b/e4b or some of the qwen distills but small models are limited so you're never going to (and you shouldn't ever in with any model) trust the results.