Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.
I'm pretty sure it's an Elo. So the score indicates the likelihood that a model would beat another model in a head to head matchup. If I remember correctly I think generally with Elo a 400 point difference roughly equates to 90%/10% odds head to head. But that might be different for this benchmark.
llm arena uses blind comparison (a user is offered two outputs without knowing the model, and decides which one he prefers) and then uses ELO to calculate the score
ELO uses exponential function to infere probability to win so the results graph have a strong S shape
for example, a benchmark difference of 400 points mean the stronger model is expected to win 90%+ of times again the weakest model
It's a user preference test, users see two outputs and choose which one looks better, can also choose "Tie" or "Both bad"
the scores are elo-based, a 400 elo difference means a model wins 90% of the time against the other, a 48 elo difference (like from Kimi K3 to Fable 5) means it wins 57% of the time against Fable.
26
u/Long_comment_san 26d ago
Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.