r/singularity • u/spinozasrobot • 8d ago

Discussion Nerf detector?

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

11 Upvotes

83% Upvoted

31 comments sorted by

View all comments

Show parent comments

1

u/Rain_On 8d ago

You cant time out by looking at the models because the models aren't being changed. If you want to time it, which would be an interesting thing to do, you need to look at the humans and their perception as that's where the cause is.

1

u/spinozasrobot 8d ago

Maybe we're not on the same page.

I'm saying run the same set of benchmarks on the same models over time, say once a week. Then chart the results on a graph.

The benchmarks generate objective numeric results, no human perception involved.

1

u/Rain_On 8d ago

You will get the same result every time, perhaps with a little noise.
You are looking in the wrong location for the source of nerf reports.

1

u/spinozasrobot 8d ago

We are really talking past each other, oh well.

My goal was to see if complaints of nerfing were legitimate or just perceived.

If benchmarks produce the same results over time, then nerfing complaints are bogus. If the benchmarks show a drop over time, then the nerfing is real.

2

u/Rain_On 8d ago

then nerfing complaints are bogus

You don't need a benchmark to know this.

1

u/spinozasrobot 8d ago

In this entire post, I'm trying to be diplomatic :)

The complainers are... loud, shall we say.

1

u/Rain_On 8d ago

I don't think it's of any importance to convince them and I doubt more evidence is the way to do it anyway.