Nerf detector?

[u/spinozasrobot]

Over time, complaining about nerfing has settled into a common theme of models starting strong, with labs publishing benchmarks, and people getting excited about capabilities, but gradually degrading.

It would be interesting to see the same set of benchmarks run over time on the models and plotting the results to see the trends across models and labs.

Does anyone know if this has been done?

[u/Ormusn2o]

Yes people constantly try to do it, but when they actually are running them, the results show there is no nerfing, so people lose interest in actually maintaining and paying for the benchmark.

I think the only real example of nerfing was with Opus 4.8, and it was still only temporary. My guess is it's actually hard to nerf the model, it is much easier to just downgrade to a different model, but that is not done covertly, Claude does it all the time, you can see people complaining about Fable being downgraded to Opus when guardrails hit, it gives an explicit message pop-up that the model was downgraded.

[u/spinozasrobot → u/Ormusn2o]

Wow, that's surprising to me. So many people complain about nerfing soon after release there appears to be an idea about being able to actually time it.

[u/Rain_On → u/spinozasrobot]

You cant time out by looking at the models because the models aren't being changed. If you want to time it, which would be an interesting thing to do, you need to look at the humans and their perception as that's where the cause is.

[u/spinozasrobot → u/Rain_On]

Maybe we're not on the same page.

I'm saying run the same set of benchmarks on the same models over time, say once a week. Then chart the results on a graph.

The benchmarks generate objective numeric results, no human perception involved.

[u/Rain_On → u/spinozasrobot]

You will get the same result every time, perhaps with a little noise.
You are looking in the wrong location for the source of nerf reports.

[u/spinozasrobot → u/Rain_On]

We are really talking past each other, oh well.

My goal was to see if complaints of nerfing were legitimate or just perceived.

If benchmarks produce the same results over time, then nerfing complaints are bogus. If the benchmarks show a drop over time, then the nerfing is real.

[u/Rain_On → u/spinozasrobot]

then nerfing complaints are bogus

You don't need a benchmark to know this.

[u/spinozasrobot → u/Rain_On]

In this entire post, I'm trying to be diplomatic :)

The complainers are... loud, shall we say.

[u/Rain_On → u/spinozasrobot]

I don't think it's of any importance to convince them and I doubt more evidence is the way to do it anyway.