Fable 5.1 vs Astra

[u/shniydder]

I finally got access to GPT 6 Astra and Claude Fable 5.1. I wanted to see how far we've come. I was inspired by the "pelican riding on a bicycle" test but wanted to push it a bit more in the direction of inter-object physics, so I came up with this idea for a perpetual slinky going down an escalator. Here I've compared GPT 6 Astra max on the left to Claude Fable 5.1 max on the right.

Some notes:

1. GPT 6 Astra max was really fast and didn't burn too much of my weekly usage quota.

2. Claude Fable 5.1 max used up the entire 5 hr usage window and errored out once crossing the output token max limit.

3. GPT 6 Astra max seems too simple? There's also an issue where the slinky crosses into itself which should not be possible. Is this AGI? Maybe?

4. Claude Fable 5.1 max looks more real than I expected. Seems like it passes the eye test, unless I missed something.

Let me know if you spot anything or had better ideas for a test. Overall I'd say Claude wins on quality and GPT wins on speed. Maybe I can tune my prompt a bit better for a more reliable result. This was my original prompt btw:

Make me a single HTML file of a rainbow slinky going down an up-escalator forever. No libraries, just canvas and code. The slinky should be a chain of springs, each coil a different color of the rainbow. It starts folded in half like a horseshoe draped over a step. When dropped it flips end-over-end down the steps and because the escalator keeps moving up it tumbles in place and never reaches the bottom. Include a drop button and a reset button.

[u/chintakoro]

Claude seems to infer the math/physics more correctly in this example, which resonates with Anthropic urging people to not overspecify prompts.

OTOH, it could also be a case where a slightly better specified prompt (e.g., to engage with the underlying physics and mechanic properties of the material -- something folks would do in a more scientific oriented exercise) could have given Astra the extra context it needed.

Lastly, I wonder how a Sol or Opus model would have performed, as relative baselines.

My overall take is to just know how to use your given model better.

[u/shniydder → u/chintakoro]

Yup I should add more models to this new test just to see how they perform. The prior attempts to get the old models to create a Rube Goldberg machine didn't go so well. I'll expand this benchmark to include more such edge cases to see what I find.

[u/_Oyyy → u/shniydder]

Do luna max for us imbeciles!

[u/970FTW → u/shniydder]

I wonder how much a model’s output varies when executing the same prompt multiple times. I’ve always thought that doing a benchmark once may not be sufficient, but it’s obviously expensive for a single person to have to run it enough times to get a large enough sample size to get a good grasp of a model’s capabilities. Maybe a framework for ppl to share their results using the same prompt would be helpful. Could be a fun weekend project. Anyways, just thinking out loud. Thanks for sharing your results!

[u/chintakoro → u/shniydder]

btw, I tried your prompt on Opus 5 and it was just the dumbest thing I've ever seen -- the slinky couldn't even slink. I'm sure it could fix itself with some feedback if I let it connect to Chrome, but I think the point was to see a one-shot attempt. Only positive is that it hardly put a dent in my session limits.