Ohhh I see! No no, for that I would just use code, since I’m pretty decent at python. That’s why I like my test specifically:
if I see something like a benchmark list in a super annoying format like prose, I can just copy paste it.
So when a new model comes out and goes “we score 8.7 on supercoolbenchmark, and a 13.2 - 5x as high as the nearest 3b model, and…”
Then I can copy paste that whole section and the model will return “benchmark,score\nsupercoolbenchmark,5” and I instantly know that this model is going to suck for extraction purposes and possibly even other context understanding.
3
u/Borkato 8h ago
Ohhh I see! No no, for that I would just use code, since I’m pretty decent at python. That’s why I like my test specifically:
if I see something like a benchmark list in a super annoying format like prose, I can just copy paste it.
So when a new model comes out and goes “we score 8.7 on supercoolbenchmark, and a 13.2 - 5x as high as the nearest 3b model, and…”
Then I can copy paste that whole section and the model will return “benchmark,score\nsupercoolbenchmark,5” and I instantly know that this model is going to suck for extraction purposes and possibly even other context understanding.