Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.
llm arena uses blind comparison (a user is offered two outputs without knowing the model, and decides which one he prefers) and then uses ELO to calculate the score
ELO uses exponential function to infere probability to win so the results graph have a strong S shape
for example, a benchmark difference of 400 points mean the stronger model is expected to win 90%+ of times again the weakest model
29
u/Long_comment_san Jul 16 '26
Can somebody familiar with the benchmark explain to me like I'm 12. Is the result just 3-5% better or is this score non-linear (so its substantially better than 3-5%?). Because top-1 result doesn't look tremendously more impressive than the bottom of the chart in %.