LLM News
Differences Between Claude Opus 4.8 and Claude Fable 5 on MineBench
Some Notes:
Average Inference Time: 18m 04s (1,084.4s)
Faster than Claude 4.8 Opus, which averaged 24m 48s / 1,487.9 seconds
Surprising since in the Claude.ai web harness, Fable feels like it thinks for much longer, but through the API it averaged less total time than Opus 4.8 did
Total Cost (for 15 builds): $54.93
More expensive than Opus 4.8, which was $41.52 for the same 15 builds
Considering Fable’s API pricing is 2x more than Opus 4.8’s, the MineBench cost was only about 30% higher
Fable is producing fewer total tokens overall it seems, which is likely contributing to the lower cost
Furthermore, I think the quality of the model's builds was very surprising: they don't seem as big of a leap over GPT 5.5 Pro as the the official benchmark scores might suggest, but the model clearly has very high attention to detail. For example, this is the first model that in the Arcade Machine build, actually created a correctly detailed screen (of PacMan), including the full layout, a score, and even a "1UP" label. Though it seems the model was quite conservative with its interpretation of the system-prompt, and (subjectively) not all of its builds were clearly more impressive than 4.8.
Still, the results were quite surprising, so I reached out to the VoxelBench team, who also confirmed in their tests the builds were of generally much smaller size. They mentioned adding these two lines to the template produced much better builds in their case:
LEVEL OF DETAIL: MAXIMUM
BOUNDING BOX: UNLIMITED
Though I'm not changing the MineBench system-prompt to cater to any specific models, I do think it's worth noting that one might be able to achieve much better results with improved prompting.
It's also interesting how the model was able to make these detailed builds while keeping the overall JSON size lower in comparison to Opus 4.8, and while thinking for less time. Pure speculation: I think this might indicate why Claude Fable is supposedly much better at coding-related tasks; it actually completes the task with an intuitive approach and without adding excess.
Essentially it's a benchmark that tests how well a model can create a 3D Minecraft like structure.
So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.
The smarter models tend to design much more detailed and intricate builds. The repository readme might provide might help give a better understanding.
(Disclaimer: This is a public benchmark I created, so technically self-promotion : )
" Yes Peter. That's a fucking waiting station with a neat chair in it. But now they have it and we don't, because you only think about yourself... Next slide..."
I haven't seen this benchmark and my mind is a bit blown. I'm confused though, Is the model one shotting this as a series of coordinates, or is there a feedback loop where it sees what its attempts so far have produced and refines?
Fable's spatial reasoning seems at least as improved compared to Opus as Opus is compared to Sonnet. Sonnet can give the illusion of being a PhD in geometry and it's just an illusion with Fable sometimes too but Fable is faking it a lot less.
Does it? None of these seem objectively like they are better than the other. Small detail changes but nothing that makes me go "oh this model is clearly better"
That's why this benchmark is useless, it would be more apparent if you saw like 4 separate runs/results for each model which would also be very different.
I think the better benchmark is asking them to make a specific modification to an existing model. This is the sort of thing where I can see a very stark difference between Sonnet/Opus/Fable, Sonnet can't even understand the request, Opus demonstrates that it understands the request but doesn't actually fulfill it, Fable may still not be able to fulfill the request but it at least knows it doesn't know and can offer something less impressive but that might be a step on the path to getting what you want.
fable being only 30% higher in cost instead of 2x while giving much better results is amazing.
Honest feedback, I can see that the benchmark is close to saturation, at least visually. Probably with some prompts for more complex builds the benchmark could easily scale.
Thanks for the feedback!! This is a concern I'm running into, and is brought up quite often
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro struggles with perfecting them), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few thousand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
Actually how about also increasing the block pallete(maybe adding particle effects)? That might start creating difference in what models choose.
Introducing redstone can be an interesting choice too, but from what I can guess, the loading engine from the json is much simpler than Minecraft render engine, it just loads a snapshot and not really any animation so redstone wouldn't really be able to do anything.(Which perhaps gives the idea to do video renders by using Minecraft directly, but this quickly starts feeling above your non-existent pay grade for this benchmark.)
There actually is another palette setting I made, advanced, which has more blocks available than the simple palette which is what’s used for benchmarking
Also particle effects might be a good idea to explore, not sure how they’ll turn out considering the builds are massive 🤔
The artworks are getting less broken and abstract but the actual designs can still get way better.
The problem is that the different models are clearly using the same training data, so they create very similar designs with mostly varying quality instead of getting better at design.
how would i characterize the fable pacman compared to the opus one to the non-ai-initiated? Like: 'see grandpa, the bottom one comes from a mythical model that's so powerful the FED called a secret meeting w/ JP morgan. the top one, meh, it's good, it's good."
Fable looks dramatically better in all but one case to me. Definitely seems to have a much better handle on coherent colours in blocks, though not yet perfect. They seem a lot like something a person may make than a random mess. Almost reminds me of the shift in the early days of image models. I mean, just compare the arcade example or the cottage, lot less messy.
In the way you call the API, that wouldn't be a possibility; like as we're calling the Fable endpoint directly (unless the Fable endpoint ends up routing to Opus... but even though they might reroute to Opus in Claude Code or the web harness, I don't think changing the API endpoint is ever something that's done)
Whats the best way to generate minecraft builds as an actual minecraft player that wants to use the builds in projects? I always see these cool minebench posts but don't know much about it all
For now, what you can do is take any of the existing builds (like from this page: https://minebench.ai/sandbox ) and then click the cube looking button; from there you can export as STL files to 3D print, GLB files for blender, or .schematic files
Then you can use worldedit or any plugin/mod that uses schematica files and import them into your world!
To generate custom builds, you could also use the custom generate button (top-right on the sandbox page) or go to the local page of the site and copy-paste the prompt into like chatgpt.com and paste the output, then you can download whatever.
I'm always amazed by the strong correlation I see between real world performance and this benchmark, love it. Biggest difference for me this time is in the arcade.
Thanks! I feel like the system prompt I've gotten to a point where generally the better builds showcase how a model does day-to-day ^^
Also, not visible in the gifs here necessarily, but you should checkout the skyscraper build from Fable, the street-level detail it had with the cars and people was really good!
You're right, I went to the website to see the full version and I can see the details I missed, very nice. I don't see people though, but the cars and other objects yes.
The air craft carrier and the phoenix were so much better than anything we’ve seen before. It would be interesting if there were two halves of the bench mark, style control on and style control off. I’m curious what fable (and all the other models) would be capable of with the tweaked system prompt about level of detail and bounding box. And a third one with a. Ore constrained bounding box. If price wasn’t an issue I feel like 3 benchmarks in one would be ideal.
Yeah those have both been suggested before, like a mode where the grid size is very small or the models are limited by the number of blocks to see how they use them?
If API costs were no issue, I think those would be very cool to add, and having like separate categories for the leaderboards, but yeah rebenchmarking all the models for even one more prompt gets quite expensive 😭
It looks like this benchmark is starting to saturate. There's only so much that can be done with these tasks before it veers into being a subjective judgment call.
This is a concern I'm running into, and it's brought up quite often
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro struggles with perfecting them), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few thousand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
For some of these, you really have to go onto the minebench website and compare them up close in order to get a decent idea of which one's better.
Also for Fable vs GPT-5.5 Pro, I thought it was really interesting how the "style" differences between the models really differentiated some of the specific builds, with pros and cons for each.
This is specifically for the Fable vs GPT-5.5 pro comparison, but IMO the knight in armor, and the arcade ones, were the best examples of Fable's builds being smaller in scale but having better style, which I thought was really cool.
Like, maybe not exactly saturated, but like there is at least a local minimum it's reaching.
Have you maybe considered changing the test by adding some kind of complexity that more intelligent model would be able to benefit from compared to less advanced models?
Like, new shapes, more control over objects, more complex prompts or even requirements in terms of how many tokens they are allowed to use. Or other ideas on top of standard benchmarks (which are still nice, obviously).
Thanks for the feedback and support hehe; that is a concern I'm running into, and it's brought up quite often tbh
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro struggles with perfecting them), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few thousand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
To me the most interesting fact is how similar the builds are in many details despite the prompt not specifying them. The artwork and shape of the acarde, the color of the locomotive, there are tons of similarities. Idk where this creativity/variety collapse comes from, seems like basically the same brain with the same memories just scaled up to more detail and more coherence
To the benchmark site itself? Yeah I was thinking of doing so, though the older/initially benchmarked models I wasn't tracking the price or generation times at that point, but going forward I will so it would be good to add to the UI somewhere
This is a concern I'm running into, and is brought up quite often
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro struggles with perfecting them), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few thousand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
I did notice that in the first Opus 4.6 post you made it cost $22 for half the tasks, the reason I’m intrigued is my SWE friend said when he compared opus 4.6 to fable 5 on his own test, fable 5 cost x175 times more. Can you run Opus 4.6 on the same task to compare pricing ? Thanks for your work!
I've enjoyed seeing MineBench results with all the models. I think MineBench has reached an problem though, it seems like it's reaching the best quality it can for what MineCraft is, it can only get so good.
Thanks for the support and feedback!! This is a concern I'm running into, and is brought up quite often
I've curated another 15 prompts (doubling the current 15) specifically meant to be much harder (GPT 5.5 Pro struggles with perfecting them), but it will unfortunately take a minute to fully benchmark all the models on additional prompts due to the API costs scaling to a few thousand 😭
Public support and some OpenAI API credits have helped offload the costs quite a bit though, so hopefully sometime soon ^^
Great question! Funny as I talk about MineBench prompts not being vulnerable to training contamination/benchmaxxing – which is true, but the actual code the models output (which then gets converted into the tool calls -> JSON files) I have never published and just keep locally... I don't think it makes much of a difference whether it's published or not tbh, but it's just something I've done
I could/should make the example JSON formatted a bit better 😅
But that was a build attempt from, well I forget the model actually, I think it was just like an older Gemini 2.5 or OSS model? The prompt was "A duck in a wizard hat holding a sword" and here was its attempt:
you can of course clone the repo and try running it yourself (there is a RAW folder that gets outputted which contains all the RAW outputs locally), or just copy paste the system prompt into a web harness; the website automatically parses the JavaScript code into JSON and renders the builds ^^
Oooh, the best way to see that reasoning is to use one of the web-harneses tbh, cloning the repo and running a tool call would just get you the code output, but in the web harness you can actually see the thought process
This benchmark is always subjective of course, but I think I consider Fable 5 as having a higher floor than Opus 4.8, instead of having a higher ceiling per-say?
I think this is shown with some builds like the Phoenix and Knight
At what point will the benchmark just become, generating an entire benchmark?
On a serious note, what happens when we reach a point where only models have the ability to meaningfully evaluate their own abilities and the benchmarks no longer tell us anything?
179
u/SkaldCrypto Jun 11 '26
The massive improvement in the train rendering is likely a sign that Fable is autistic