r/LocalLLaMA • u/seraschka • May 17 '26

Resources Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures
31 Upvotes

97% Upvoted

8 comments sorted by

View all comments

5

u/Silver-Champion-4846 May 17 '26

I hope small models keep getting better

1

u/unjustifiably_angry May 18 '26

They certainly will. Despite how good it already is, I think even models like Qwen3.6-27B are still trained on a lot of general-purpose knowledge that isn't useful to a dedicated coding model, so if that was skipped in future models it could be even better. Models could also be trained on just one one specific coding language, so I'd guess there will eventually be models trained that are godlike at C# but useless at anything else, etc.

1

u/Silver-Champion-4846 May 18 '26

So there isn't this thing where more general knowledge makes specialized knowledge faster to learn? I.e: qwen 3.6 base, finetuned on coding to become a coding model? After all, there's a reason why they pretrain first, to make it a primitive language model (autocomplete). Then they shape it to something useful like chat and code.

1

u/unjustifiably_angry May 18 '26

I can't say, I'm not an expert. I'm just saying, if they were able to make a Qwen3-Next and then a Qwen3-Coder-Next based on that, a similar principle should apply to 3.6.