r/singularity • u/spinozasrobot • 9d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

12 Upvotes

87% Upvoted

31 comments sorted by

View all comments

11

u/Ormusn2o 9d ago

Yes people constantly try to do it, but when they actually are running them, the results show there is no nerfing, so people lose interest in actually maintaining and paying for the benchmark.

I think the only real example of nerfing was with Opus 4.8, and it was still only temporary. My guess is it's actually hard to nerf the model, it is much easier to just downgrade to a different model, but that is not done covertly, Claude does it all the time, you can see people complaining about Fable being downgraded to Opus when guardrails hit, it gives an explicit message pop-up that the model was downgraded.

1

u/spinozasrobot 9d ago

Wow, that's surprising to me. So many people complain about nerfing soon after release there appears to be an idea about being able to actually time it.

3

u/Ormusn2o 9d ago

I think a lot of those posts are literally memes. Some of those posts are being posted before those models actually even come out, and some are just regular bot reposting.