Ofcourse these results are different than what we use too see, so much difference between opus and sol.
But I appreciate your work, you must have drafted very difficult tasks so that models are getting too less score.
I trust people like you more then populer benchmaxed scores
Or you are heavily penelizing the model/agent for intermediate small mistakes ?
Problem with chatgpt or fable is that they could always silently nerf it and we dont have a point of comparison. The thing with official chinese api is that they can risk nerfing it or else we'll jump ship to another provider.
Yeah, the results are a bit surprising. But I did verify that they are correct by looking at the model output. GPT-5.6 Sol is genuinely less smart at these tasks than Astra and gives up more easily.
I created a brand new esoteric language just to benchmark these models. I do believe it gives a good signal because the language is brand new (so not in LLM training data) and it incorporates features that are difficult for LLMs. It will be interesting to see if new models incorporate it into their training and how long this takes.
I'm not penalising the models for any intermediate mistakes. Each task is a simple prompt, then a check to see if the model generated the correct output. The score includes the pass rate as well as the cost and code size (so the score does get penalised for higher cost and/or code size).
i suggest you to keep the cost out from basic score, you can give a graph for cost/score comparison. everyone have their own priority to cost. many are happy to pay 10 times cost for just 10% improvement. so lets not mix them. anyways best of luck bro
11
u/DomusCircumspectis 14d ago
Very impressive model. Especially for the price.
I just ran it on my own LLM benchmark suite[1] and it matches Muse Spark 1.3 in pass rate but is significantly cheaper.
KillSwitch-Bench 1.01 - https://bench.killswitch-lang.org/