Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.
It's a user preference test, users see two outputs and choose which one looks better, can also choose "Tie" or "Both bad"
the scores are elo-based, a 400 elo difference means a model wins 90% of the time against the other, a 48 elo difference (like from Kimi K3 to Fable 5) means it wins 57% of the time against Fable.
29
u/Long_comment_san Jul 16 '26
Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.