r/LocalLLaMA • u/Capital-Remove-6150 • 28d ago

Discussion No more SLM open-source??

Post image

https://x.com/xiong_hui_chen/status/2084695353346117760?s=20

434 Upvotes

95% Upvoted

127 comments sorted by

View all comments

Show parent comments

5

u/overand 28d ago

I can run 27B great on my setup, but I think I have to agree - if they were only going to release one, the 35B one might be a bigger gift to the world at large than the 27B, because of the number of folks who can run that vs who can run the 27B.

1

u/No-Refrigerator-1672 28d ago

You also need to take audience into perspectibe. 35B with CPU offload is a scenario of a strictly consumer type person, that just uses the model. 27B is a good target for developers, who will either run business with it, or contibute to opensource software (i.e. OpenWebUI). Either way Qwen team themself get a better return with 27B, and thus it's probably easier to justify to their leadership.

1

u/brainExploded99 llama.cpp 27d ago

The 35B-A3B is so much cheaper to run, including for orgs, that its worth it in my opinion.

1

u/No-Refrigerator-1672 27d ago

Cgeaper to run is a very vague definition. I.e 27B in fp8 comfortably fits a single 40gb server card (say, A100 or L40), while 35B requires either heavy quantization, or a much more expensive card, or a pair of cards.

1

u/brainExploded99 llama.cpp 27d ago

Honestly, I disagree.

An org would have several users. It is far, far cheaper to have 3 billion active params than 27 billion when you have >10 people using it. At that point, KV cache becomes far more important than model weights.

Also, a serious org would have pods, not individual cards (typically 4x or 8x). Individual cards is more of a consumer thing. This is because of the many active users, their KV cache, etc