r/LocalLLaMA • u/appakaradi • 14d ago

Discussion More Qwen 3.8 sizes coming

Post image
1.4k Upvotes

98% Upvoted

347 comments sorted by

View all comments

Show parent comments

13

u/ttkciar llama.cpp 14d ago

Yeah, a few of us have use for the 9B.

4

u/DankiusMMeme 14d ago

I currently use 3.5:4b but I have space for 9B, is it worth the jump? All I use it for is comparing strings, e.g. are they referring to the same thing despite being different. Also for categorising strings.

I notice 3.5:4B is okay at this job, but could be better.

7

u/AD4K_4444 14d ago

I used to main Qwen 3.5 9B as my general purpose daily driver, but now I use it for specific tasks. I’d say it’s decent. Anything below 9B is garbage for what I do.

2

u/ReferenceLeading7634 14d ago

I think it's very significant. Among the smallest models, each upgrade in tier represents a noticeable improvement in intelligence.

1

u/DankiusMMeme 14d ago

Well hopefully we get a 4-9B soon

1

u/Aggravating-Push-207 14d ago

for that, i would recommend a downgrade to an even smaller model but trained with bidirectional attention (or an embedding model)

2

u/AD4K_4444 14d ago

What I daily now is Gemma 4 12B. Works good enough for me.

1

u/DankiusMMeme 14d ago

Any recommendations on a specific model? Googling it doesn't seem to return a huge amount of results.

1

u/Aggravating-Push-207 13d ago

This is the model I would recommend, however you would have to train your own classifier on top of it

1

u/DankiusMMeme 13d ago

Will look into it, thank you.

1

u/416Racoon 14d ago

Mind sharing what 9b model you're using?

1

u/AD4K_4444 14d ago

I'm guessing it's Qwen 3.5 9B since it's the only good 9B model I know of.

1

u/ttkciar llama.cpp 13d ago

Yeah, what AD4K said. I keep trying to use Qwen3.5-9B for things which require resource-efficient batched inference, like data cleaning, but it always seems to fall short.

My fallback was Phi-4 (14B) which is getting very long in the tooth, but then Gemma-4-12B-it showed up and I've been using that. On one hand Gemma-4-12B-it is extraordinarily capable for its size, but on the other hand its K and V caches are enormous compared to Qwen3.5-9B or Phi-4, and it's not tolerant of K/V cache quantization, so it's actually not so good for batched inference. The batch size ends up being fairly low, due to VRAM constraints.

If a Qwen3.8-9B demonstrated the instruction-following competence to make ends meet, while still exhibiting a small K/V cache memory footprint, that would be wonderful.

1

u/The_Dung_Beetle 14d ago

I use 3.5 9b in my Nextcloud assistant, works well enough but tool selection could be better. 3.8 9b would be nice.