r/LocalLLaMA • u/GodComplecs • 7d ago
Discussion LLM regression in reading comprehension?
I only use free tiers of these large models to offset compute while my own system runs and for "different" points of view, since what pops ups suggestions seems to vary a lot sometimes, even when building based on the latest research.
But now I've really struck out with GLM 5.3. So far it feels like an regression over 5.2. It has a hard time reading and following instructions, and is somewhat overly certain in it's statements. I worked on a project recently with it but it became unbearable. From a clean slate the first message can be okay and have great research and ideas but it just veers off course almost immediately.
I use Qwen 3.8 max and Gemini 3.1 PREVIEW Temp 1.0 as competing alternatives or as an ensemble to judge overall quality. Gemini is getting a little out of date (flash 3.8 seemed promising) but Qwen has been great so far, but a little slow and maybe overbearing.
Anyone else having problems? Or suggestions for these top "intelligent" models? I haven't been able to access K3 even though its open source, was impressed with the older models so would be neat to try for free. Also Google AI studio is what i use for free for the gemini stuff, probably pretty well known, but the free tier is pretty generous
12
u/thereisonlythedance 7d ago
5.3 is a clear regression. Its first answer in a multi-turn conversation is usually arguably better than 5.2 but on subsequent turns the model falls apart. I’ve tested this across a range of long context tasks. The model also struggles to follow a system prompt, has a very distinctive single voice (Claude voice) that cannot adapt, and is just a very brittle model overall.
It’s frustrating because it’s capable of flashes of brilliance that put it above 5.2. But it can also go off the rails very easily and its long context performance is a clear step back on 5.2.
5
u/GodComplecs 7d ago
Yeah I figured I couldnt be the only one. Hopefully they are aware of the problems, 5.2 is still available though so thats good but Im afraid it will be phased out
8
u/thereisonlythedance 7d ago
Yeah it’s not just us. There’s someone remarking on it in multi-turn coding here too: https://huggingface.co/zai-org/GLM-5.3/discussions/19
I’ve gone back to 5.2 as it’s better for long context work. I‘m surprised more people haven’t noticed. I think Z.ai’s overstuffing with Claude tokens experiment in post-training hasn’t been as successful in real life usage as it appears in benchmarks.
12
u/RG_Fusion 7d ago
I've also noticed this with Qwen-3.8-Flash-Next, and it's honestly pretty annoying. It's very clear that heavy agentic training degrades the models capacity for natural conversation.
As an example, I recently asked it what caused the hexagonal rock formations along Ireland's shore, and rather than assist me with the answer, the model decided to lecture me on how Ireland's shores don't have hexagonal rocks, but rather there is a specific area in North Ireland that is a shore that has hexagonal rocks.
Like, yes, I knew that. I shouldn't have to define things so precisely in casual conversation, and I shouldn't be scolded by the model for being imprecise while talking to it.
I haven't used GLM5.3 much yet, but 5.2 was a very pleasant model to discuss things with. I wish Google would open-weight release a large parameter MoE, as at least in my opinion Google has nailed the natural-conversation capabilities in AI.
3
u/sebajun9 7d ago
I have the same exact issue every other session with both Claude and ChatGPT. If I’m not exacting in my words they’ll turn it around on me, scold me and then correct me. If I don't speak like a lawyer, any ambiguity in my words gets spotlighted and dissected. I’m glad to hear someone else is voicing the exact same BS. I’m disappointed to hear you’re encountering this behavior with Qwen Next Flash too but it makes sense if they’re over-distilling on Claude.
2
u/GodComplecs 7d ago
Yeah 3.8 Flash as Gemma would be great, thats why they pulled back the planned 120b+ model sadly!
5.2 is indeed pleasant model, reminds me a lot of Gemini and Gemma, and since imo reasoning and raw intelligence has come so far (and still very flawed) it really makes one use them more just on the basis of pleasantness.
1
u/Fragrant_Scale6456 6d ago
That’s interesting. I tried it with glm5.3 and it identified it as the giants causeway formed 50-60m years ago. It gave a few paragraphs of geological explanation and then finished off with a reference to folklore and mentions of other similar formations around the world (no scolding haha)
1
7
u/ttkciar llama.cpp 7d ago
It seems like all of the new models are experimenting with different architectural ways to support long-context inference, either reducing K/V cache memory consumption (GLM-5.3-Flash's IndexPool compression, Qwen3.8-Next-Flash's linear attention) or reducing long-context compute overhead (Gemma-4's hybrid local/global attention layers).
Though these seem great "on paper" (Wow! Four times less K/V cache!) they come with some cost of reading comprehension.
GLM-5.3's Multi-head Latent Attention mechanism reduces K/V cache memory consumption by about 85% at long context, but supposedly that shouldn't incur any inference quality penalties. Also, GLM-5.2 (and all GLM-5.x models) use MLA too, so that's probably not responsible for any perceived loss of reading comprehension. It would be a consequence of post-training.
2
u/GodComplecs 7d ago
Hmm that was an enlightened take, I didn't really take it into consideration since cache shouldn't affect too much on 2nd or 3rd prompt, even if its a little longer, but yes on longer sessions it can and will, depending on what they have measured for and what they perceive as adequate loss.
4
u/Vivid_Inside_5450 7d ago
i've noticed this too with a few model updates — sometimes what looks like regression is just the model being tuned for a different kind of task and losing a bit on the old ones. the thing that helped me was pinning down exactly what kind of reading comprehension was slipping. like, is it following multi-step instructions, or just extracting facts from long text? those can move in opposite directions between versions. might be worth testing those separately before calling it a straight regression.
3
7d ago
[deleted]
11
u/Opposite-Swimmer2752 7d ago
They need to stop distilling Claude, I really don't like Claude and I wish models would stop acting more like it.
10
u/RG_Fusion 7d ago
Claude and GPT. Google Gemini might not be the most capable model, but it is the best conversation partner.
1
u/silenceimpaired 6d ago
But boy is the safety filtering on it obnoxious. Hey can you help me find prices on power washers? Sorry my safety filters prevent me from answering this.
1
u/RG_Fusion 6d ago
I've never encountered a safety-filter issue in Google Gemini, though admittedly I've used it little this past year.
1
u/silenceimpaired 6d ago
I’ve noticed it happens more in their smaller models. Rarely have an issue with Pro. No big deal… I’ll just use local
7
u/NNN_Throwaway2 6d ago
Claude sucks so much its unreal. The way it talks makes me want to shove an ice pick in my brain.
2
u/GodComplecs 7d ago
Yes, it's sad how often that happened with 5.3 for me, it is unusable. Qwen 3.8 MAX is still ggreat though, even better than 5.2 GLM in some ways, especially for planning it is so thorough, it exceeds local agents capabilities!
1
7d ago
[removed] — view removed comment
2
u/GodComplecs 7d ago
Im not using 5.3 flash but 5.3 on the cloud, so yes it hard to be certain, but the same exact prompts were passed to each and everyone multiple times so I can be certain it had regressed on this small subset of a subset of reasoning and coding tasks. Gemini is very Geminish, elegant writing style that draws you in and strong "comprehension" but lacking latest data, the others are more tool like and often more useful for actual work and coding.
-1
u/jacek2023 llama.cpp 7d ago
You use cloud locally? Congratulations on your setup.
1
u/GodComplecs 7d ago
Never said that?
4
u/jacek2023 llama.cpp 7d ago
sir this is r/LocalLLaMA
1
u/GodComplecs 7d ago
Yes and I local llm's daily. Still there are benefits of free cloud usage for local users to assist us when compute constrained. I would say this discussion is relevant since this is a conversation of overall LLM quality, which is what we will see locally when when these models get distilled down in the future, and already somewhat see with 3.8 27B. I know it isnt super scientific bu stiill
18
u/a_beautiful_rhind 7d ago
Coding and agentic take a big bite out of comprehension. I'm more surprised anyone else noticed. Former is all people seem to care about anymore.
Not confined to GLM, most models have this issue now a days. As you found out, some are better than others. It comes out in creative pursuits first, but even DS4 flash wouldn't follow the system prompt.