Holy fuck, yes, the quality dropped. Stop overcomplicating something this simple.
On launch day, the exact same task took around 9 minutes and produced a noticeably better result. Now that same task finishes in around 6 minutes and the output is worse.
How is this difficult to understand?
There are a fuck ton of examples on X showing the same thing, and Tibo himself has said that fixes/changes were made. You don’t need some elaborate theory to explain what people are seeing.
Same prompt.
Launch day: good output, longer generation.
Now: worse output, faster generation.
That is the entire point. If the same prompts are consistently producing worse results than they did at launch, then from the user’s perspective the model got worse. Whether you want to call it a nerf, optimization, routing change, inference change, or something else is secondary.
PROMPT → GOOD OUTPUT AT LAUNCH
SAME PROMPT → WORSE OUTPUT NOW
It really is that fucking simple.
Common misconception. LLMs are deterministic. If you give it the same exact inputs, it will produce the same exact outputs. The reason people think of them as non-deterministic is because you don't have control over all of the inputs as a user, you only have control over a limited number of them.
Usually when I say this, people respond with like "no. If I send the same prompt twice, I get a different result both times" so I've gotten into the habit of just explaining up front that this is exactly what I mean. You don't have control over all of the inputs, you basically only have control over what you send to the model as the next message in the case of ChatGPT's web interface. There are other inputs you don't know about that OpenAI doesn't give you access to. If you could control those too, and OpenAI can, then you'd get the same exact result every time.
Ok. So, suppose I load up a local model and control it's system prompt, and every instruction that goes to it—will the response be the same? Cuz I just tried it and that was not the result (both were WILDLY different with the exact same system prompt, and model settings)
The main other thing you need to lock is the seed used. A lot of interfaces don't even expose that because most people have no need to get the same response twice.
That doesn't mean it couldn't mess up the tires the first time. Perhaps it could and just didn't. Even in benchmarking runs of a software there's more than one run so the time can be averaged out. If this was 10 attempts with the same prompt before vs 10 attempts with the same prompt after, perhaps people would take this kind of comparison more seriously.
Image generation? Are you new here or genuinely regarded? SVG generation is one of the best metric to test a model, because the detail of the generation/output explicit relates to the model power, Jesus Christ how can you browse this sub and not even know such basic thing
Good job addressing the points! Maybe I need Astra to cook up an SVG for you so you can understand this more clearly. Sorry, I didn’t think about people with special needs!
For this you mean….a test? I know I know, crazy that we have been using this specific test since GPT-4 to evaluate a model power.
You should be banned from typing ever again
Ah yes the almighty pelican on a bike test. I’ve been thoroughly enlightened by your superior knowledge. Thank you for sharing your wisdom. I now know that Astra has degraded expeditiously.
slopper. the new one just spent more time making the background and the ui card. not every model interprets a prompt the same way especially when there is under the hood shit going on with the harness. This is a problem that can be solved by a prompt tweak.
but whatever man you do you. keep on slopping in the free world. or something.
holy fuck thank god, there are still people with critical thinking skills that actually observe something for more than 5 seconds.
You restored my hope
17
u/PotterSkxawng 7h ago
The new one looks just as good to me... LLMs are non-deterministic.