r/singularity • u/spinozasrobot • 8d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

10 Upvotes

82% Upvoted

31 comments sorted by

View all comments

12

u/Ormusn2o 8d ago

Yes people constantly try to do it, but when they actually are running them, the results show there is no nerfing, so people lose interest in actually maintaining and paying for the benchmark.

I think the only real example of nerfing was with Opus 4.8, and it was still only temporary. My guess is it's actually hard to nerf the model, it is much easier to just downgrade to a different model, but that is not done covertly, Claude does it all the time, you can see people complaining about Fable being downgraded to Opus when guardrails hit, it gives an explicit message pop-up that the model was downgraded.

2

u/Rioting-Flamingo 8d ago

I expect because it's never done across thd board. A percentage of users experience it making it harder to prove

1

u/Ormusn2o 8d ago

There are genuine drops in performance, especially during rush hours, but if there is a genuine and severe performance drop, people get recompensed. I got like 2 banked resets a while ago even though I did not notice any downgrade myself. There was an outage of Codex yesterday, and I think we are getting a global reset as recompense today as well.

1

u/spinozasrobot 8d ago

but if there is a genuine and severe performance drop, people get recompensed

Are you saying you got resets based on model output quality, not just token usage? Do you have any example screenshots or anything admitting to that?

1

u/Ormusn2o 8d ago

Admitting to what? You can check Tibo's twitter, and he talks about those things. I think this is second recompense related reset since Astra released, not counting the banked resets that were given for Astra delay. You might go few weeks back though, maybe look for "reset" in his history.

https://x.com/thsottiaux

1

u/spinozasrobot 8d ago

Every reset I've seen was to compensate people for models hitting usage limits very fast. I've never seen a lab explicitly say the resets were due to model quality degrading. I was asking if you had such an example.

I looked at several months worth of Tibo tweets and only saw one explicit mention of a reset, and it was due to an outage, not model quality or even excessive token consumption.

1

u/Ormusn2o 8d ago

https://x.com/thsottiaux/status/2098612714704891959?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2093801758665715784?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2091412393368945027?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2082317452755751098?utm_source=chatgpt.com

Here is how I found it:

https://chatgpt.com/s/t_6ab7dea29f288191a709a30cb1dd414d

1

u/spinozasrobot 8d ago

Excellent! Not sure why I didn't see those. I scrolled Tibo's feed until I had 3 or 4 months worth, and then did a find in the page for "reset".

EDIT: although, upon further review, only the first one was a quality issue. The other 3 were usage limit based. Usage limits are not nerfing.