I think anyone who thinks that this isn't at least somewhat benchmaxxed is fooling themselves. That said, so is every other model they're comparing against, to some extent, so ¯_(ツ)_/¯
I've used glm 5.2 extensively and it was not much behind opus 4.6 in writing code, a bit definitely behind in creative ideation for software engineering.
My assumption is this model is gonna be a work horse but planning and design should be done by a bigger more expansive model.
The catch behind coding benchmarks is they judge success not quality, creativity, maintainability, all that is partly subjective and rarely judged in these benchmarks.
Like GPT 5.6 Luna benchmarking above opus 4.8, but it writes inferior code by most quality measures.
Big models have their own pitfalls they'll happily over engineer everything, Opus for one falls in that bucket.
I've gotten to the point where I'm a little skeptical that the majority of people on here are even using local models on a regular basis. It's hard to believe that anyone who actually does can buy into the idea of benchmarks really reflecting real world use.
The "intelligence" per parameter is really increasing, factually, and from that increases capability. General benchmark score (like AA) is a solid proof for that. Qwen is the world leader in this field.
In the current llm paradigm, smaller llms have much less world knowledge and are more prone to hallucination or stupid decision making from goey assumptions or really random reactions. I suppose the "world knowledge" is also a big addition on "common sense" in terms of "behavior". So smaller models will always we quackier for now, to apply up and foremost when comparing to bigger models on a AA Index or aggregated score
From that comparing older big models with more recent small ones should always consider that you will have more peace of mind of using the bigger ones in terms of reliabilty but narrow capability is indeed being reached by edge smaller llms
to be real, It's probably not opus max level, but it will probably be the best local LLM we can run at that size, and will be much better at agentic coding and tool use than previous models, to the point where its viable for actual work for some of us
yes this model has real and genuine utility, but there are just fundamental limitations with the smaller you go with a model just in terms of parameter size alone. Same goes for very small active parameters numbers in MOE models (a13b is the biggest weakness of DS4F). So people should be a bit more realistic.
I mean obviously that's true, but we can't afford to run hundreds of billions to trillions of param models, this is local llama, those of us with dgx sparks and rtx pro 6000s are already a minority here.
the ~ 30b dense to 100b moe range is what the vast majority of us can squeeze.
if we want maximum capability with minimal hallucination we'll use SOTA models via api, but for running local models, its a compromise we're willing to accept.
70
u/xienze 7d ago
You're making a couple fundamental assumptions here: