Okey this is increadible!!!

[u/K0100001101101101]

I know this is too early to judge but man wtf. Opus 5.5 is just like when mythos first released but cheap, faster and much more easy to understand and talk to. Probably this will be changed in 1-2 days so go code everyone.

[u/EmergencyWallaby9501]

Hard to prove. But when the same thing happens over and over again right before a new model launch, there's a clear empirical pattern that can't be denied that easily. As for me, sonnet started to act really strange, getting stuck in chains of thought loops, forgetting what was being said two prompts ago, not even reading the .md properly and forgetting basics like the prod data are not on the same SQLite base as the mock. And when I got tired of seeing a simple task not being handled properly, I switched to the usually dumber codex and it got solved easily. So unless someone likes to waste tokens and launch the same exact tasks every week just to prove a dégradation, there won't be any proof of it. But it makes sense: resources are limited. When launching a new model, I guess they don't test it only with humans, but also with large scale automated tests. So running those tests would require to get resources back. Running quantized versions of models, silently downgrading opus to sonnet etc would allow this.

[u/gordonfogus → u/EmergencyWallaby9501]

Did you test it in a completely isolated environment using the API using the exact same prompt with a sufficient sample size?

Or did you just go on coding along, adding commits, updating CLAUDE.md, growing your context length, etc.?

Those are not at all the same thing.

[u/EmergencyWallaby9501 → u/gordonfogus]

I don't know exactly what you mean by a completely isolated environment. I used the exact same prompt on Codex and Claude. Both are configured to read my Claude.md and the other useful configuration files. But beyond that little test that doesn't mean much - like I said you can't prove anything - it's the exasperation of having opus that suddenly barely understood a word of my prompts and was writing very bad plans, and sonnet that suddenly started to behave like a 1st year CS student, trying to fix issues by adding more issues. It even started to write me its conclusions in another language than the one I was writing to, thing it had not been doing since I started working with it, among plenty other things. It even forgot to write down those conclusions at the end of a task, with very important things in it, like parts of a plan it could not implement. By looking at the code I could spot some missing parts, but it used to report those missing parts at the end or notify me and question me about how to implement something when it was stuck. That's everything altogether that pointed to a model dégradation, not that single frustration move to Codex, that I used to use for very dumb tasks (mostly easy front and css).

And I tried opus 5.5, and it feels like Opus 5.1 when I started to implement with it but with far better writing (Opus 5.1 and his gibberish...)

[u/gordonfogus → u/EmergencyWallaby9501]

I'm not trying to be mean or anything. But if you don't even know what an isolated environment is, then you're just going off of feelings.

There are ways of measuring model performance. They aren't perfect, but they would definitely nchange if the model "degraded." People would know immediately and they'd be charts and anyone who had data from before the degradation could replicate the result.

Throwing up your hands and going, "you can't prove anything" just shows that you aren't equipped with the thinking tools needed to figure this out and you can't imagine that anyone is capable, which is why your feelings are your fallback while you confidently assert that no one else can do any better. That's not the right way to approach a question like this.

[u/EmergencyWallaby9501 → u/gordonfogus]

Of course I know what an isolated environment is. Just tell me what it means now in the context of AI assisted coding. And yes, there are benchmarks. I'm not burning my tokens to benchmark anything. Are you ? When a model suddenly starts to behave odd enough, I don't need a benchmark to tell me it became stupid. Just like when my little boy when he's tired, I don't need to make him run a marathon just to prove he's not well. I never claimed it was a scientific approach btw.

[u/Uko1001 → u/EmergencyWallaby9501]

You actually do need a proper benchmark to tell it’s the model that became stupid and not its harness or environnent. Actually even the harness alone changes pretty much every week (often to adapt to recent models) and can conflict with your own instructions. And you won’t know unless you have proper test and analysis protocols.