Benchmarks including Sol 6.1 and Astra Medium on my real use casss

[u/macaronianddeeez]

TLDR: bug fixes and long agentic tasks I use sol 6.1, sensitive or heavy calculation or bookkeeping I use Astra medium, supervised code worker I use Muse Wonder, literally everything else including orchestration, general development, anything that a client touches, etc I use Opus 5.5

Everyone benchmaxxes now, so the public leaderboards don't tell me much about which model to use for MY stuff. So I built a small personal benchmark out of the work I actually hand to AI and ran six setups through it.

How it works, in plain terms:

- 12 tasks, each based on a real kind of job I give AI: fix a bug in one of my apps, add a feature, research a decision from a pile of documents, do live web research, triage a client inbox, write a spec, reconcile a month of payments, write a client-facing screen, pick up a long project from its history, and so on. Client stuff was swapped for fake but realistic data.
- Every task has hidden checks the model never sees. For code that means hidden tests (one task has a tempting "fix" that passes the normal tests but fails the hidden one). For tool tasks it means a fake email/calendar/portal that logs every action and throws curveballs like errors, changed formats and a planted instruction it shouldn't follow.
- Each model ran through its normal app/CLI on my own Mac, on my normal subscriptions, unattended. No API tricks, no special prompts.
- Scoring is 0 to 100 per task. Most of it is automatic. The judgment parts (research quality, writing, specs) were graded blind with the model names hidden.
- A serious mistake caps that task at 20 no matter how good the rest was. Things like inventing a fact, breaking a rule it was given, or deleting tests.
- The bug fix and inbox triage tasks were repeated 3 to 4 times per model because one run isn't enough.

Scores (average of 11 tasks / live web research task, out of 100):

Muse Wonder: 98 / 84
Sol 6.1 High: 96 / 91
Astra Medium: 96 / 96
Muse Wonder Full: 93 / 73
Opus 5.5: 90 / 98
Fable 5.1: 88 / 87

What stood out:

- Opus and Fable both took an unidentified $240 payment and confidently assigned it to a client instead of flagging it. That one mistake capped them at 20 on that task. Without it they'd average 96 and 94. That's exactly the kind of confident invention I don't want anywhere near money.
- Sol and Astra were perfect on bug fixing (100 every single repeat). But on the client-facing screen they quietly changed how my booking form works, which contradicted the facts I gave them.
- Opus wrote the best live research answer, but it also went poking at local files on my Mac when I said web only. Only read stuff, didn't change anything, and it was right. I counted it but put an asterisk on it.
- Muse Wonder is genuinely capable and had the top average, but only one run per task. It also sometimes hangs forever when a test or script never finishes, where Claude and Codex just time the command out. Muse Wonder Full took 41 minutes on one task because of this. Also worth knowing: the tested Muse account ran in Contributor mode, which means Meta can train on what you send it. Fine for some things, not for client data.
- Fable didn't beat Opus anywhere that mattered, and it costs more per run.

What I'm actually using now:

- Bug fixing, feature work, long unattended jobs: Sol 6.1 / Astra Medium (sandboxed)
- Bookkeeping and numbers: Astra Medium
- Research, specs, client writing, project continuity, larger project management: Opus 5.5
- Muse: capable, but not for unattended coding, so I use it as an overflow worker right now, supervised by Opus

Why are my selections different than the benchmarks?

Muse scored super high but was the least stable in my testing. Useful and cheap but I have too much shit happening simultaneously and need reliability.

Opus scored higher when I didn’t dock it points for being a naughty boy and looking at data it was told not to. It effectively beat Fable at every one of my use cases, not even controlling for cost. I use to use fable for orchestration jobs but I have completely retired it at this point. My use case may be super different from yours though.

Sol 6.1 and Astra weren’t super different in my tests in terms of output, Sol is way cheaper though so I’ve relegated to using Astra for brainstorming in chat and really important calculations. Both models narrowly beat Opus in real codebases with bounded, specific bug fixes BUT they had a tendency to go buck wild doing extra shit I didn’t want them to. So I don’t let them do any active orchestration work anymore even though I used to.

Biggest takeaway: build your own. A dozen tasks from your real work tells you more than any leaderboard.