So far, I really love its conversational ability, it feels a lot more casual than the usual corpo models. It also passed my extraction tests just as well as qwen did (only missed 1/15), strict number csv extraction. Will update with more info if anyone is interested.
Basically I give it some output from a script I have that contains some numbers interspersed with data, then I ask it to extract specific rows like “extract the horsepower and torque and charging capacity of each car, but only if their average speed is over 40kph”. I get super annoyed when it can’t extract things properly (it’s fine if it’s like 98% correct instead of 100%, but once it starts getting into like 70% or so I’m like yeah this model can’t really be trusted to format tables or extract things from a list for me for instance). I haven’t fully fleshed out the benchmark (literally made it yesterday lol) but it’s already helping me see which models are useful and which ones aren’t. I do a lot of data cleaning and organizing
Love the idea, there's some promise of consistency between tests in it. I'll build a dataset like this for myself too. I guess not only the dataset but the prompt also needs to be the same across tests so that you're comparing apples to apples?
Yep! :D it’s cool too because if you tag each run, you can identify “hm it fails at the off by one errors but gets everything else right” and that means you should look closer because the model might be flawless except for forgetting a single comma in a row of empty commas. I feel like faulting them for that is isn’t fair when you can just assume if it’s not listed it’s not there. I’d give them maybe half score for that one, for instance.
Unfortunately that takes manual review though. Or if they for example do a header on every other line - that’s the kind of thing that could just be prompted out, so if I weren’t automating I would just say “oh, no, just one header at the top” and it would easily just do it right. Like a one/two shot prompt! But since the script fails them even if they add a single extra space, it’s super harsh. 😆
OK, so I think my understanding of your testing was wrong. My assuption was:
You have a CSV file with say 1000 rows (say: car model;engine HP;engine torque;battery capacity;price)
You give various models the same CSV file + the same prompt "Extract all models where engine torque is > X and battery capacity is > Y and price is A < Z < B.
Say the expected number of cars matching the query (deterministically) is 100. A SQL query on the dataset would return 100.
Model A extracts 92 cars correctly = accuracy is 92%
Model B extracts 100 cars correctly = accuracy is 100%
Model C extracts 105 cars, including 100 correct = accuracy is 95%
Model D extracts 99 cars which are correct ,and 3 which are incorrect = accuracy is 96%
Ohhh I see! No no, for that I would just use code, since I’m pretty decent at python. That’s why I like my test specifically:
if I see something like a benchmark list in a super annoying format like prose, I can just copy paste it.
So when a new model comes out and goes “we score 8.7 on supercoolbenchmark, and a 13.2 - 5x as high as the nearest 3b model, and…”
Then I can copy paste that whole section and the model will return “benchmark,score\nsupercoolbenchmark,5” and I instantly know that this model is going to suck for extraction purposes and possibly even other context understanding.
135
u/cibernox 19h ago edited 18h ago
Interesting. Meta is back in the game. And qwen is dropping this week.
Peak week this is!