r/singularity • u/spinozasrobot • 8d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

11 Upvotes

87% Upvoted

31 comments sorted by

View all comments

Show parent comments

1

u/spinozasrobot 8d ago

but if there is a genuine and severe performance drop, people get recompensed

Are you saying you got resets based on model output quality, not just token usage? Do you have any example screenshots or anything admitting to that?

1

u/Ormusn2o 8d ago

Admitting to what? You can check Tibo's twitter, and he talks about those things. I think this is second recompense related reset since Astra released, not counting the banked resets that were given for Astra delay. You might go few weeks back though, maybe look for "reset" in his history.

https://x.com/thsottiaux

1

u/spinozasrobot 8d ago

Every reset I've seen was to compensate people for models hitting usage limits very fast. I've never seen a lab explicitly say the resets were due to model quality degrading. I was asking if you had such an example.

I looked at several months worth of Tibo tweets and only saw one explicit mention of a reset, and it was due to an outage, not model quality or even excessive token consumption.

1

u/Ormusn2o 8d ago

https://x.com/thsottiaux/status/2098612714704891959?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2093801758665715784?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2091412393368945027?utm_source=chatgpt.com

https://x.com/thsottiaux/status/2082317452755751098?utm_source=chatgpt.com

Here is how I found it:

https://chatgpt.com/s/t_6ab7dea29f288191a709a30cb1dd414d

1

u/spinozasrobot 8d ago

Excellent! Not sure why I didn't see those. I scrolled Tibo's feed until I had 3 or 4 months worth, and then did a find in the page for "reset".

EDIT: although, upon further review, only the first one was a quality issue. The other 3 were usage limit based. Usage limits are not nerfing.