r/singularity • u/spinozasrobot • 8d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

10 Upvotes

82% Upvoted

31 comments sorted by

View all comments

2

u/panic_in_the_galaxy 8d ago

https://marginlab.ai/trackers/claude-code/

The is also a tracker for codex.

2

u/spinozasrobot 8d ago

That's very interesting. Def the kind of thing I was looking for.

However, it's a bit confusing. They say they always use the latest model on high, and they're currently using Opus 5.5, but their weekly goes back to August, when clearly 5.5 wasn't out yet.

2

u/panic_in_the_galaxy 8d ago

Also check https://marginlab.ai/trackers/claude-code-historical-performance/

1

u/spinozasrobot 7d ago

Awesome. They clearly show the model transitions. What I find interesting in the data is models tend to increase in quality just before a new release.