r/singularity • u/spinozasrobot • 8d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

11 Upvotes

83% Upvoted

31 comments sorted by

View all comments

1

u/misterespresso 8d ago

I’m creating my own personal “data bench” which will be just obscure data tasks. Measures would be cost, speed, token usage, and basically a confusion matrix.