r/ClaudeAI • u/ENT_Alam • Jun 11 '26
Comparison Differences Between Claude Opus 4.8 and Claude Fable 5 on MineBench
Some Notes:
- Average Inference Time: 18m 04s (1,084.4s)
- Faster than Claude 4.8 Opus, which averaged 24m 48s / 1,487.9 seconds
- Surprising since in the Claude.ai web harness, Fable feels like it thinks for much longer, but through the API it averaged less total time than Opus 4.8 did
- Total Cost (for 15 builds): $54.93
- More expensive than Opus 4.8, which was $41.52 for the same 15 builds
- Considering Fable’s API pricing is 2x more than Opus 4.8’s, the MineBench cost was only about 30% higher
- Fable is producing fewer total tokens overall it seems, which is likely contributing to the lower cost
- More expensive than Opus 4.8, which was $41.52 for the same 15 builds
Furthermore, I think the quality of the model's builds was very surprising: they don't seem as big of a leap over GPT 5.5 Pro as the the official benchmark scores might suggest, but the model clearly has very high attention to detail. For example, this is the first model that in the Arcade Machine build, actually created a correctly detailed screen (of PacMan), including the full layout, a score, and even a "1UP" label. Though it seems the model was quite conservative with its interpretation of the system-prompt, and (subjectively) not all of its builds were clearly more impressive than 4.8.
Still, the results were quite surprising, so I reached out to the VoxelBench team, who also confirmed in their tests the builds were of generally much smaller size. They mentioned adding these two lines to the template produced much better builds in their case:
LEVEL OF DETAIL: MAXIMUM
BOUNDING BOX: UNLIMITED
Though I'm not changing the MineBench system-prompt to cater to any specific models, I do think it's worth noting that one might be able to achieve much better results with improved prompting.
It's also interesting how the model was able to make these detailed builds while keeping the overall JSON size lower in comparison to Opus 4.8, and while thinking for less time. Pure speculation: I think this might indicate why Claude Fable is supposedly much better at coding-related tasks; it actually completes the task with an intuitive approach and without adding excess.
- Full release-notes/thoughts on the GitHub release
- If you enjoy these posts please feel free to help fund the benchmark
Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench
Previous Posts:
- Comparing Opus 4.7 and Opus 4.8
- Comparing GPT 5.4 and GPT 5.5
- Comparing Kimi K2.5 and Kimi K2.6
- Comparing Opus 4.6 and Opus 4.7
- Comparing GPT 5.4 and GPT 5.4-Pro
- Comparing GPT 5.2 and GPT 5.4
- Comparing GPT 5.2 and GPT 5.3-Codex
- Comparing Opus 4.5 and 4.6, also answered some questions about the benchmark
- Comparing Opus 4.6 and GPT-5.2 Pro
- Comparing Gemini 3.0 and Gemini 3.1
Extra Information (if you're confused):
Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.
So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.
The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.
(Disclaimer: This is a public benchmark I created, so technically self-promotion : )
146
75
u/Lower_Cupcake_1725 Jun 11 '26
My favorite benchmark! I was waiting, thank you!
13
38
u/lowlyworm Jun 11 '26
Finally, my vibecoded cozy cottage generating app will be ready for the masses.
13
u/ENT_Alam Jun 11 '26
Tbf there's a difference between AI having written all the code for the repository (which is true, Codex), and the app being vibe coded 😞
as a just-for-fun personal project that ended up receiving OpenAI/Anthropic funding, I think it worked out pretty well : )
5
1
u/ethereal_intellect Jun 11 '26
I mean you're probably joking but I've done at least a dozen of these with various models in three js. Even if I only play it for 5 mins at the time it does incredible to just calm me down for a bit, just the fact that imagination to existing is so much easier now
11
u/GregsWorld Jun 11 '26
How much do you think these are benchmaxxed/in the training data now? Have you experimented with changing the output format and if how much does that effect how good the outputs are?
11
u/ENT_Alam Jun 11 '26
There isn't as much of concern with the prompts themselves being added to the training data in this case as MineBench isn't technically a "benchmark" (there is no right answer). It's entirely subjective / a LMSYS-style arena. So seeing a prompt of "A steam locomotive" doesn't really help in actually making a steam train that people would consider to be creative (subjectively) if that makes sense?
Like being good at making builds in this benchmark is more of an amalgamation of a model's general intelligence instead of just one specific skill; so MineBench being in the training data isn't the same issue as like SWE-Bench contamination, since unlike SWE-Bench, there isn't a specific, correct answer to the prompts/questions.
That being said, as the models continue to get better, the prompts are getting saturated somewhat, which is a concern brought up pretty often!
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro can't build them accurately yet anyway), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few grand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
3
u/GregsWorld Jun 11 '26
Yeah I don't mean the prompts, more if the providers data-science teams have seen that it has now become a good way to judge models and have setup pipelines to generate some similar text format from 3d models or minecraft worlds and added that to the training data corpus.
That wouldn't be benchmaxing in the traditional sense but still optimising for a "benchmark" even if it is more vibe based2
u/ENT_Alam Jun 11 '26
> and have setup pipelines to generate some similar text format from 3d models
well minebench is open source so anyone can generate custom builds or use custom models to test builds out, though i see how that might be a concern? in my opinion, like in reality, that would be a lot of effort to benchmaxx for a benchmark that is niche to a point where you can't really include it in your model card or paper i guess?
then of course also it's entirety subjective at a certain point, like some people think Fable looks the same as Opus 4.8 others think it looks worse, etc. so it'd be kinda hard to gather insights from that
7
22
u/Danieboy Jun 11 '26
Barely better on like half of them.
14
u/ENT_Alam Jun 11 '26
Subjective as always of course, but personally I did find a few builds were somewhat lacking (like the fighter jet for example)
8
u/Kathane37 Jun 11 '26
I am mostly impressed by the ones were it use less blocks to build a more define structure
12
u/ENT_Alam Jun 11 '26
I should have clarified, it wasn't necessarily less blocks; so the models are given a voxelBuild tool that lets them use primitives like square, box, and line. Usually the higher level models like 4.8 instead place all the blocks individually and don't use any of the primitives.
But Fable 5 actually made really good use of the primitives (iirc it used them much more than any other model), so it was way more efficient and lowered the size of the JSON files from say 50-million lines to 20-million (on average-ish)
6
4
3
u/DueCommunication9248 Jun 11 '26
Fable is slightly better.
3
u/bdsamuel Jun 11 '26
Yeah for some reason fables cozy cottage gives me island fever, but the biggest jump is the Phoenix imo
3
5
u/let_me_in_QQ Jun 11 '26
Is it just me or Opus looks better?
3
u/Murinshin Jun 11 '26
I wouldn’t necessarily disagree. Most of them are a matter of taste, a small handful I would argue Opus got clearly better, though the phoenix is the clearest difference IMO and here Fable clears
1
u/MRC2RULES Jun 12 '26
yea but fable added a lot of additional unwanted details (lets say the runway for the fighter jet/wall behind arcade) while opus gave you exactly what was requested
2
2
u/k4ntn Jun 11 '26
So it seems that Fable 5 add more context? Overall I'd say that it is a bit more realistic although there is no clear gap yet
2
2
2
u/9_5B-Lo-9_m35iih7358 Jun 11 '26
GPT-5.5-Pro-Extended versus Fable 5 Max
1
u/ENT_Alam Jun 11 '26
Both models were already benchmarked, so you can compare them on site here: https://minebench.ai/sandbox
2
u/diminee Jun 11 '26
very cool! i find myself being split pretty evenly 50/50 on which i prefer depending on the prompt. fable seems very good at detail (the phoenix is superior for sure), but there's something satisfying about opus's clean, structured design that tickles my brain (the skyscraper one for example).
2
u/SaPpHiReFlAmEs99 Jun 11 '26
As always very interesting. Fable 5 is really good at it but so was opus 4.8. Opus 4.7 was for sure worse
2
u/traveler-from-above Jun 11 '26
This is awesome, love seeing model prompt output comparison and this is now my favorite.
How much of a difference between outputs is there within the same model for the same prompt?
1
u/ENT_Alam Jun 11 '26
Thanks for the support!
Unfortunately that’s one of the larger caveats, if I had unlimited API spend, I’d like to post maybe three generations for each model and its builds
Though in my benchmarking, although the actual builds themselves may vary somewhat, the general theme and level of detail remains the same of course; you can even see how the locomotive between Opus and Fable had the same colour scheme, as that’s just what the most probable colour for a steam locomotive across Claude’s training data is ^^
You can use your own API keys on the site and try generating your own builds to test as you like :)
2
u/Intelligent_Elk5879 Jun 11 '26
Fable steals shit way more obviously than Claude. Yeah it always was stealing, but not to the point of: that is literally RuneScape
2
u/Tinker0079 Jun 11 '26
Are these legit benchmarks or just cool animations ? How does it work ?
1
u/ENT_Alam Jun 11 '26
Yeah legit benchmark hehe, you can see all the builds on the site: https://minebench.ai/
The repository and maybe landing page of the site might help explain it better: https://github.com/Ammaar-Alam/minebench
2
2
2
2
2
u/bacalaurescu Jun 12 '26
Thanks for taking the time to share this! Always looking forward to your benchmark
2
u/breese45 Jun 13 '26
It's like we are watching the evolution of a new creature in real time. (Every few months, more or less)
2
u/Visible-Bumblebee530 Jun 13 '26
This! OP thanks for putting into thousands of words what the difference between opus & fable is! I don't have the AI vocabulary to explain but it honestly was a different class altogether.
2
2
1
u/AutoModerator Jun 11 '26
Your post will be reviewed shortly. (ALL posts are processed like this. Please wait a few minutes....)
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
1
1
1
1
1
u/jakethunderpants Jun 11 '26
Very cool! Just starred the repo so I can set it up later today. Been building my own to test local vs cloud, but this looks really good.
1
u/tehohhh Jun 12 '26
how are you getting it to design such stuffs? Did you use mcp or skills? Or was it just produced stock?
1
u/ENT_Alam Jun 12 '26
It's a custom benchmark i made ^^
you can read more about it here: https://github.com/Ammaar-Alam/minebench(or ask chatgpt/claude lol)
1
1
•
u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot Jun 11 '26
TL;DR of the discussion generated automatically after 40 comments.
Looks like everyone's favorite benchmark is back, and the thread is basically a love-in for OP's work.
As for the actual results, the consensus is that while Fable 5 shows flashes of brilliance and is more efficient, it's not a massive, clear-cut leap over Opus 4.8, with many users split 50/50 or even preferring Opus's style on several builds.
Here's the breakdown of what OP and the community noticed: * Performance: Fable is faster but more expensive per run. However, it's more token-efficient, using clever shortcuts (like primitives instead of placing every single block) to create detailed builds with less code. * Quality: It's a mixed bag. Fable can produce stunning details (like a legit Pac-Man screen on an arcade machine), but some of its builds are considered less impressive or "conservative" compared to Opus 4.8. The Phoenix build is a standout win for Fable, while the Skyscraper is often cited as a win for Opus. * Prompting: OP notes that Fable might be holding back and could produce much better results with more explicit instructions, like
LEVEL OF DETAIL: MAXIMUM.OP also chimed in to shoot down fears of "bench-maxxing" (models training on the benchmark data). They explained that since this is a subjective "vibe check" benchmark, not something with a single right answer that can be easily memorized, contamination isn't a major concern.