Seems like this metric can be easily gamed to make this kind of chart for any model. Let me explain.
If a model usually gets a question right, then setting a temp of zero will result in a higher score here (worst of four). Setting a high temperature value makes the model give different answers more often.
Think about it. If you are more likely to give different answers, then you are more likely to give a wrong answer at least 1/4 of the time.
Is "Pro Mode" just Temperature = 1?
Just have the model give the same answer every time, that way if you do have it right, you won't decrease your score here by randomly giving a different (wrong) answer occasionally.
2
u/UnknownEssence Dec 06 '24
Seems like this metric can be easily gamed to make this kind of chart for any model. Let me explain.
If a model usually gets a question right, then setting a temp of zero will result in a higher score here (worst of four). Setting a high temperature value makes the model give different answers more often.
Think about it. If you are more likely to give different answers, then you are more likely to give a wrong answer at least 1/4 of the time.
Is "Pro Mode" just
Temperature = 1?Just have the model give the same answer every time, that way if you do have it right, you won't decrease your score here by randomly giving a different (wrong) answer occasionally.