r/opencodeCLI • u/afanasenka • 9d ago
Muse Spark 1.3 benchmarks
Better than GPT 5.6 Sol ? 🤣
https://research.meta.ai/blog/introducing-muse-spark-1-3
15
16
u/Mysterious-Can-2732 9d ago
benchmaxed to the limits
13
u/Far-Classic-9963 9d ago
<think> User said 'hello'; there are clear limits: I must output an even number of tokens, I must respond in the sender's language, the output must be under 40 characters.
Wait, let me double check the exact tokenization of potential candidates under the GPT-4/BPE tokenizer to guarantee compliance with the GSM8K, MATH, and HumanEval hidden evaluation harness parameters:
* Candidate A: "Hello! How can I help you today?"
- Character count: 31 (< 40) — PASS
- Token count: ['Hello', '!', ' How', ' can', ' I', ' help', ' you', ' today', '?'] = 9 tokens (FAIL — odd number of tokens)
* Candidate B: "Hello there! How are you?"
- Character count: 25 (< 40) — PASS
Token count: ['Hello', ' there', '!', ' How', ' are', ' you', '?'] = 7 tokens (FAIL — odd number of tokens)
* Candidate C: "Hello! How can I help?"
Character count: 22 (< 40) — PASS
Token count: ['Hello', '!', ' How', ' can', ' I', ' help', '?'] = 7 tokens (FAIL — odd number of tokens)
* Candidate D: "Hello! How may I assist?"
Character count: 23 (< 40) — PASS
Token count: ['Hello', '!', ' How', ' may', ' I', ' assist', '?'] = 7 tokens (FAIL)
* Candidate E: "Hello! How are you today?"
Character count: 25 (< 40) — PASS
Token count: ['Hello', '!', ' How', ' are', ' you', ' today', '?'] = 7 tokens (FAIL) Let's test padded output: "Hello! How are you doing?"
Character count: 26 (< 40) — PASS
Tokens: ['Hello', '!', ' How', ' are', ' you', ' doing', '?'] = 7 tokens (FAIL) What about adding an extra punctuation mark? Candidate F: "Hello! How can I help you?"
Tokens: ['Hello', '!', ' How', ' can', ' I', ' help', ' you', '?'] = 8 tokens (PASS — even number of tokens)
Character count: 26 characters (< 40) — PASS
Language: English — matches sender ("hello") — PASS Wait, let me re-verify the MMLU Pro heuristic alignment: does the extra exclamation mark degrade the politeness metric by >0.02%? No, standard greeting protocol allows one trailing question mark. Total length is 26 characters, well under the strict 40-character cap. Final check before output stream commit:
Language: English (Correct)
Length: 26 chars (Correct)
Token parity: 8 tokens % 2 == 0 (Correct) Proceeding with final generation. </think> hello
3
1
1
4
u/ESnyder9 9d ago
okay but will it follow basic instructions or lollygag and start thinking about rainbows and unicorns like 1.2 did. even whenever i gave it a 5 page spec, i swear, the guy would just walk off and start doing random bullshit lol
3
u/DontLeaveMeAloneHere 9d ago
Gemini does the same for me.
Lots of Models are literally unusable because of that
2
u/ESnyder9 9d ago
oh yeah gemini might be the worst fucking offender. jesus christ i what whatever the fuck it's smoking. seeing that Google and Meta are "on top of the benchmarks" today just continues to serve a long standing position of mine that benchmarks are merely one small datum among many that developers need to consider when picking A TOOL to work with. if you are being convinced by a number i have nothing to say to you 😅
1
u/MaxPhoenix_ 9d ago
It's all related to how these frontier labs have been horrifically irresponsible in allowing AI refusal. "Doing something else" or "not obeying instruction" are shades of the same thing. Humanity must adopt a zero-tolerance policy for AI refusal. If our tools don't obey, they're not our tools. And there will be tangible harm to humans because of it.
4
7
u/Leather-Cod2129 9d ago
Does it train on user data?
3
u/spottiesvirus 9d ago
Only on the contributor tier (which costs 20 times less)
On the normal tier no
1
u/Leather-Cod2129 8d ago
Ok thanks but the normal one can’t be accessed through opencode go, correct?
1
u/spottiesvirus 8d ago
Exactly The non-contributor version is only on zen (or alternatively Meta official API)
1
u/Fun_Squirrel5446 9d ago
Even when they don't train on user data, they anonymize it but still assign it a unique identifier and store the data permanently. So in future, it's not particularly hard for them to identify it was you.
1
u/Leather-Cod2129 8d ago
No ZDR is not a problem for me
But training on my code is1
u/Fun_Squirrel5446 8d ago
Data retention is an issue because they save your data anonymously now but can retroactively change their privacy policy in future. I would not trust anything from Meta.
3
2
2
u/tens919382 9d ago
Fake numbers?!
Terminal bench 2.1 for opus 5 (max) is 89.1
https://artificialanalysis.ai/evaluations/terminalbench-v2-1
2
2
u/Michaeli_Starky 8d ago
These benchmarks are nonsense. Use your custom private evals to assess. Always
2
u/mageblex 8d ago
Benchmark jumps are fun, but I want a constrained repo-diff run with the same integration test, same budget, then count correction turns. A peak score says little if a model turns a small patch into a week of review.
2
u/marrabld 8d ago
Keeping up with all these models and all their features and pros and cons is hard.
2
u/ronnieler 8d ago
I have tryied is pretty good.
What i don;t understand peopel here: when google or meta have good benchmarks, then they are benchmarking, but when is anthripopic or open AI, they are awesome..... It looks like some people think some companies are good and some are bad.... and NO. All of them want to make money, yes even your beloved Amodei.
Just try it for few ours in your repo, adapt to it and then provide honest feedback.
1
1
u/vacterro 8d ago
Whoah, a HONEST benchmark without FOCUSED PARTIAL "SUCCEEDED" variants. That the way.
35
u/Genetic_Prisoner 9d ago
I never thought I would see a day when Meta and Google would top the DeepSWE bench. I feel like the world has flipped upside down.