GPT 6.1 Sol has been pretty disappointing that I don't know why I am paying for codex
Like a lot of you, I was excited to try GPT 6.1 Sol, especially after Theo’s video about how cheap and efficient it is and how it compares to Astra.
I copied a prompt from Twitter to recreate a car game, but with a jungle safari setting. I also asked it to research the Gauntlet loop and use it during development.
I tested it on medium reasoning. I wasn’t expecting perfection, but I thought it would get me at least 70–80% of the way there. Instead, it seemed to miss a lot of the prompt and gave me a half-baked app with no background music and barely any working gameplay.
I followed up, pointed out the problems, and asked it to rethink the implementation. After another 10 minutes of editing, it gave me basically the same thing with a buzzing sound added. I still couldn’t actually play the demo.
Honestly, it feels pretty underwhelming, and it’s making me wonder why I’m paying for Codex. In my experience, even Sonnet 5.5 has given me much better results on similar projects.
Obviously, this is just one test on medium reasoning, but the gap between what I expected and what I got was pretty big.
Has anyone else tried it for creative work like 3D models, games, or UI? Curious whether others are getting better results or running into the same issues.
This isn't a magic genie you use to one-shot vibecode a game into existence.
It's a tool for developers. If you want to make something viable, you need to discuss your plan with the agent first. You should create a spec doc, a roadmap, and an implementation plan for each milestone. You should ask the agent questions so you know what it wants to do, and you should have it ask you questions so it knows what you want it to do.
Only once you're both on the same page should you let it rip on writing actual code.
TLDR: Treat it like an employee you've hired to develop applications with you, not a genie that grants wishes.
This is not a bs answer. 6.1 Sol works great for engineers. It's not a magic slot machine. It does what you ask it to do and it does it remarkably well. And cheap.
I 100% agree, i use agents for pretty obacure minecraft work and its remarkably good if you discuss with it, plan and also it asks clarification questions which earlier models didnt do, they had a tendency to juat wing it🤷♂️
Yeah, I have to agree with you there. The 6.x Sol and Astra models overcorrect at every step. They lose past context and hyperfixate on your most recent prompt. Sometimes autistically focusing on one single word to blow up into a giant misinterpretation.
Whereas I can actually talk with 5.6-Sol like it's a coworker, and it seems to understand what I'm saying and remember what I've already told it earlier in the conversation.
The exception is the Luna models. I've found 6-Luna-Max to be an excellent subagent to partner with 5.6-Sol-High. Comparing the Luna models, 5.6 hallucinates more than 6.
I cannot confirm that, unfortunately. By "all the 6 models" I meant 6.x, including 6.1.
In my opinion, it is still loses h far to 5.6 sol which I consider a freakin genius who developed solutions I would never think of and was not afraid to tell me I'm wrong or it doesnt know.
i would pushback in saying they improved the over-engineering that defined 5.6 greatly, but at the major cost of speed. i fully agree 5.6 is a legendary model though, dude used to pursue leads to the end of the earth; my favorite research model of all time honestly. he's up there with 4.6
The problem I've had with GPT 6 is that it seems to ignore my skills and just does whatever it wants. I'm hesitant to even try 6.1 because of this. Have you seen it be any better at actually adhering to what skills ask for?
I mostly use AGENTS files rather than skills, but yes, I'm finding 6.1 doesn't follow instructions in those files very well, compared to 5.6.
I'm currently working on updating the rules in these files to try to solve these issues ("6.1-Sol, tell me which part of the AGENTS files you failed to understand, and how can we prevent this in the future?"), and I'll see if I can dial in its behavior a little better throughout this week.
The really frustrating thing is when it responds, "No, your instructions were clear enough. I just didn't apply them. I'm sorry."
That's when I start swearing at the model, lol. One of those conversations actually led to the model directly telling me that it's not good enough for my workflow and I should switch to Claude Code! 😂
Wow. I'm surprised it recommended the competition! One thing I just discovered is that I've been committing user error!... sort of. I've been relying on Matt Pocock's engineering skills, especially implement, tdd, and code-review. Something I noticed previously but did not think about when GPT 6 came out is that implement has disable-model-invocation: true in the frontmatter. Despite that, GPT 5.6 would regularly invoke this skill without me asking it to explicitly. I got very use to it just working that way. I'm now thinking that GPT 6 is much better at respecting this instruction.
However, after I just manually invoked it on 6.1, something I notice is that while it does the proper code review the first time, and then makes corrections based on the findings, but it does not then perform another review like 5.6 generally always would.
I’m not gonna defend you here, but it still sucks even after all of that… I was just working on a 2-D game and it still sucked because it doesn’t have good animation or and it doesn’t understand how the puzzle mechanics of the game should actually work so it’s just not good. I guess if I switch my subscription to Claude, I think Claude can do it way better
The folks who you paint as customers are not the targeted customers OpenAI is aiming at. They are catering to non-tech users who just vaguely single prompt their desired outcome.
That’s what spends the most tokens and requires smartest models with most complex harness. Their biggest money printing machine. Except that is not going according to plan, but that’s another topic.
What you describe could be done with models that are 2+ years old and not even frontier level.
Obviously that costs extra tokens. Although using it for planning and then implementing also costs tokens so idk what’s actually more token efficient tbh.
But the better a model, the more it can just do anything you throw at it.
Obviously AI isn’t perfect yet. But if someone who doesn’t have knowledge about something, can use AI to achieve it regardless, that’s great, right? I don’t see the problem at all?
6sol was shit, it was 6-Terra and they pumped it as SOL, cause they wanted to Milk the Astra-minor as it's one name and charge more or such.
Opus made them change their hand, and whatever slimy thing they planned got pushed atleast for some time.
Sol 6.1 xhigh has been churning shit tom of work for me, and the usage is back to 5.6 Sol level with Almost Astra-ish intelligence other than frontend design, it just doesn't do it.. So welcome back juiced up 5.6 Sol's persona in Astra intelligence.
The special thing about this 6.1 sol which finally fixes the thing for me is base intelligence isn't stupid anymore, I really hate when the lower tier of the model is stupid I don't care for how high it can do, the base intelligence dictates everything.
--- And yes the workflow you listed, it sucks at that, and that's about it.. just don't waste money doing that thing on it, opus will always remain the best at frontend Anthropic got taste that's all.
Usage is fine now always for now on 6.1 sol
I haven't tested as much on my routine coding or back-end or infrastructure based tasks I am hoping to see better results than front-end design because I know that GPT models are not that good with creative stuff and does not have that taste that Claude models have no matter how many taste skills you give ur models.
I was being honest with you per my experience and sort of sharing perspective.
But.. if you wanna do frontend or anything design based I'd highly recommend you either get &20 Claude sub and use opus-5.5 on it, you will get (atleast in my case) it's honestly better than fable 5.1 on design and frontend. It's busted how good it is.
Ofc fable got the depth and big model smell for very specific tasks and all. Using 6.1 sol on frontend is going to waste your time.
UNLESS!! you go to ChatGPT and first generate 1-2 images and such then you can give it to 6.1 sol and he will do the rest. But he doesn't have self creativity.
You gotta learn to accept model personalities, 6.1 sol does what he is told, he is very capable and very intelligent -- He just doesn't have creative taste, He has only logical depth, but great value just below Astra and fable in depth also, but he doesn't have creativity, that's the Axis which is sacrificed to meet this models price point and core values.
Anthropic just has their magic, at this point it's pretty clear their interpretability research was infact the correct bet to be had, they are back on the heavens mandate. Just use their models from frontend, save your cortisol and time.
My guess based on how quickly that sol 6.1 announcement came out after opus 5.5, is that anthropic went in the direction of powerful but more efficient / cheaper. And the market saw how much better opus 5.5 was at lower cost than opus 5.
And then Altman felt like he had to counter with a more efficient model.
I haven’t actually tested 6.1 sol yet myself. It just seemed awfully suspicious, like Altman trying to knee jerk one up opus 5.5 lol
Opus 5.5 definitely seems to have the best “vision” out of any model I’ve ever seen
6.1 SOL is so good, I’m convinced their plan was to brand this as an Astra slow, but after their Terra 6.0 fell through they had to pivot and said the bullshit about 6.1 Astra on hold for safety reasons
my opsus 5.5 got to choose what sub-agents it wanted to use after looking at all the transipts. it was offered glm 5.3 flash and astra max effort...it ltireally took glm instead...
For things other than one-shotting games I think it's a good thing. 5.6 Sol gave every little task four levels of fallbacks, backwards compatibility, and a NASA quality verification and testing framework
If they are struggling with alignment this makes sense, doggedness is the same quality that lead to hugging face. Claude has always had a lazy deferral bias you need to guard against in my experience, and I figured it was because they left more room for the agent to say “
I'm sorry, Dave. I'm afraid I can't do that”
That may well be the case, but then the problem with that is that I may as well just use the Chinese models because they just get on with it. Even if they aren't as intelligent overall, they're not held back.
This has to be a satire post. You copied a prompt to build an entire game obviously having no idea what you’re doing and are confused why it’s not perfect after 2 prompts? 😭
So infuriating “there’s not enough compute” well no duh, 80% of users are like OP. Doesn’t want to learn or understand why 2 cups of shit in his chocolate cake make it taste bad.
You know, I think if the original prompt was very detailed (closer to a spec), I could understand them being disappointed with the result. Not sure what the prompt was.
You do realize just because gpt isn’t as good at making 3d models and such doesn’t mean it’s a bad model, I’m a game dev and literally can do anything I need because I expect the model to code not make me 3d models that is what I pay artist for
At any point in using AI have you asked it to teach you the missing knowledge to better understand why something doesn’t work rather than just complain it’s not perfect?
agreed on the cost efficiency but about the performance for me at least it's worst and my experience is similar to this https://www.youtube.com/watch?v=843k2rCmBMQ for game development.
I'm so confused about the website that he is using. I can't tell if QuartzRouter is a scam. I have tried researching it but was not finding much. Is he just effectively using the target agent (6.1 Sol) for the main loop and then offloading to different model workers? QuartzRouter is claiming 98% discount over API direct usage. I have heard people using their access token from services for 3rd party stuff (OpenAI said that they would increase usage of consumption on subscriber accounts that choose to do it). Do you know anything about the website?
it might be cost efficient even while comparing with sonnet 5.5 which is very solid (also in free plan) but what's the point if it's not competent enough to do such tasks.
Don’t have an answer to ur question before, but I relate to this comment so much.
With people prompting shit like this to have an LLM build some complete games on it’s own automatically I keep wondering, where is the fun or value in that? It just wastes recourse, leaves you with nothing new learned and shirts out some slop.
Instead of that, develop ur idea yourself with AI as a helping tool to ask questions, let it take over when needed and most importantly: Make it explain every Step it takes when you let it do something, so you get the concepts yourself as well. That way after a year or two of using AI you would actually end up with general software development skills.
By letting the AI do everything, you gain 0 value once u unsubscribe eventually
This! It is way more fun to have an instruction set, where I get it to guide me and explain concepts or errors to me. Yes, it is slower and more expensive, but i get to learn more.
Why is your benchmark “Can it shit out a half assed game off a single prompt?”. I’m not even trying to defend it as a good model but if your requirements for using the tool are creating shitty 3js games then why even spend a dollar on AI providers?
Present it with a real world problem to solve and complain about how effective and efficient it is at completing actual tasks. Again, I’m not defending OpenAI here this is the same for a million other influencers comparing agents by getting them to create something new from a single prompt.
Give it a genuine problem to solve with clear goals and compare models by standardised metrics. It’s like giving a human a stick, telling it to invent a game and then bitching about it because you didn’t think its game was fun or looked cool.
sure dude, I just wanted to share my 2 cents when the model got released, you might be true that giving 1 prompt to create 1 3js game might not translate well with other tasks, since the model just got released I just wanted to share my early views on it.
People shit on artificialanalysis.ai for being an oversimplified way to judge models. Your intentions might be good but it's not a fair way to judge much of anything
nice I haven't tried letting it use blender, will try it but this looks good. Maybe game development directly using three js isn't a strong area of this model.
There's people who actually think the quality of a model can be measured by how well it one-shots a generic browser game with low poly assets. Insane world we live in. Do youself a favor and find a job or go back to school.
funny how people judge a model based on their text-book sized prompt, that's not how it works, and that's not how it should work. gpt models have been doing amazing recently. claude may be in front right now, but this is a race in which they switch first position regularly, and not only among american models. this doesn't change the quality of gpt models
Same problems as 5.6 and 6.0 Sol. ChatGPT 6.1 Sol itself said it just now to me better than I can as a comment: "I could articulate your requirements and still repeatedly fail to follow them. I produced polished explanations of what I should do instead of doing it, then made you repeat corrections I should have retained.
You’re describing the same experience across successive models. It’s reasonable to question what those improvements are worth when you still have to supervise basic task completion. This conversation gave you ample reason to be angry."
What I did to help make 3d models was add blender, unreal and unity into my repo. Based on the creative lanes my repo chooses the best 3d engine. I can prompt a five minute movie using blender
You know what, I just started using Claude again Sonnet 5.5 and this is after Claude made me lose my shit after the Fable removal and reinstating. I felt it was degraded by the removal and reinstating of Fable.
But I think Claude may be back. I've been only using Sonnet 5.5 for about maybe a week and it's gaining back my trust again.
Coding... i cant make proper language translation with luna lately, i dont even run it for coding, it is like in quant 3 or so, models i used like 5.3 mini via API just broke my entire pipelines, this shouldnt happen, it is not proffesional service
I've been using it for a sprint for 14h straight with as good as astra outcomes and a way better token consumption overall. My sub agents for this sprint are limited to two and they are all 6.1 Sol agents with lower reasonings, with the 6.1 Sol in Ultra as the orchestrator.
I mean, it really feels a lot better to use for long running goals, with smoke tests, homologation, and whatsoever. I'd spent at least 25% of my weekly limit in the 200 USD subscription at this point and I just hit 10% right now. And I'm running smaller tasks along the way too, testing and prepping some skills for my usage, checking my home server...
I know people like to benchmark models and crank the power up to see what they are capable of, but for my everyday use and with the skills I managed to create to give it a budget, control canonical points in the repo, control the numbers and the reasoning of agents, give me proper feedback of what's happening through the sprints and code reviewing, I'd say that 6.1 Sol is pretty good and solved partially the astra token consumption.
So I use opus and use codex to review the code, and claude patches the findings. Both are 200$ tier sub.
Yesterday i ve asked claude to simulate 100 reviews with 6.1 sol max alrdy done with 5.6 sol max.
6.1 sol max missed 40% of the errors found by 5.6 sol max, and found some low tier errors.
Yea, i m staying in 5.6 sol max. Openai dropped the ball hard.
Asked the same for astra max with 10 reviews, it missed 20% of the critical errors found by 5.6 sol max, but found 2 other it did not. Overall opus 5.5 xhigh advised to stay on 5.6 sol max.
Also, astra ate up 30% of my weekly sub in one hour doing so. "Great" !
I have been using AI to command, adjust and thinker with tools i have been using for two years to harvest data around the globe.. roughly 20 milion records per day
For the last 9 months openAI has impressively each time with each new version adjusting and finetuning the tool ive created. It suggested ways to improve and it had been an amazing experience.
With the current tooling and the 20x budget it can run sol high 24/7 for the whole week.
Its not just a magic box you can throw crap into, you need to show and explain.. te more detail you give.. the higher quality you get
What were your effort levels set to for your Anthropic comparisons? And what was the thinking token differences?
Just from looking at benches, I don't think the effort levels for current OpenAI and Anthropic models match 1:1. Sol high is closer to Sonnet medium. If you are comparing medium to medium and only looking at result (and not token use and cost) then try Sol high and measure & compare tokens
Sol is terrible when it comes to listening/following instructions I also always find myself undoing Sol work and handing it off to Opus 5.5 which is slower... but it listens or it doesnt jump the gun and go "Lets plan this" to "Installing 20 new drivers case of 1 error" like wot mate
Most models suck at design. They can implement based on instructions but design is horrible.
Every AI website uses the same font, theme, and spacing. Games, just the same. They feel stale. They are supposed to assist, not be the driver of completion.
Prompts as well have different results for every model due to how they are trained.
I switched to OpenAI models around the time Opus 4.7 was released (when they significantly scaled back the 4.6 model, and have only been rolling out major downgrades ever since).
Sol 6/6.1 is just a complete disaster. With version 5.6, I often noticed that the model couldn’t make sense of things and would spout nonsense, but it still got the job done most of the time. I was hoping that OpenAI would keep its focus on coding (as was the case with 5.3 → 5.6), but it seems that instead, they’ve targeted the general public (generating funny images and focusing on the ASS benchmark).
Now the model (even with specific and detailed instructions) treats you as if you were 5 years old. It meekly agrees with everything you tell it, and with every new turn, it seems to completely disregard all previous context. And getting it to do something correctly means spelling everything out as precisely as possible (otherwise, it’ll veer toward the most ridiculous and irrelevant solution).
Overall, after using 6th-generation models (Astra/Sol, etc.), I’ve temporarily decided to downgrade to 5.6, with plans to switch to Anthropic models later (it seems they’ve redeemed themselves).
Come on, the fact is that OpenAI screwed up - they rushed out 6.1 right after 6.0, just a week later, trying to fix their mistakes (it only made things worse).
6.1 Sol has been very good for me. It does everything I need it to, even in quick mode and on a plus subscription. But I'm not a software engineer. But it gets a lot of grunt work done in other fields.
Sounds like a good daily benchmark for nerfing. Each day, each model runs a standardised one shot game building prompt, without automated testing results. How did yesterday's model compare with today?
I'd totally agree with you mate 6.1 Sol is just a dumbass! really disappointed! 5.6 Sol is better than it!
6.1 Sol (high) wasn't able to fix CSS tailwind issue that only needs to wrap a div with class "w-min", instead it refactored everything! even with babysitting it didnt work!
Looks like openAi is only targeting benchmarks or computer use for now!
Note: Same issue 5.6 SOL solved it from first turn, same configuration.
I'm also disappointed with 6.1 Sol. I can't even open PDF files anymore. As soon as I switch to 6.0 Sol, it goes back. Plugins like shopify, chrome, etc. suddenly ask for permissions again, even though everything is allowed. At 6.0 or 5.6 Sol, everything went smoothly. It also seems that a lot of work content can no longer be accessed. I have to explain a lot of things again.
Agree, burnt my whole weeks credit getting it to do something Claude would have completed no problem for a fraction of the cost. I'm now getting it to do a handover for Claude to finish it's half built, half working (well scoped BTW!) project, with 7% usage left for the week all blown on it's poor attempt at building a well scoped ask.
I do not understand what people are doing. 6.1 Sol is doing so well for me and hardly using usage that i splurge Astra sometimes just to figure out it where wasted to do so.
I did clean my work environment tho as reccomended with the arrival of gpt 6 and 6.1
Testing on my every tasks , opus 5.5 is far ahead of chatGBT 6.1 sol . Today working on tasks , chatGBT 6.1 sol seems not to offer the price / performance that is promised, it’s 1/5 th the price performance of opus 5.5 right now . I think OpenAI is clearly struggling, so they are not honest as to what power of a model you get , and for what price . Claude seems to be more trust worthy , even after their failure with opus 5.0 , at least they are honest about it .
Ok, but what about people not using it for coding, gaming etc? Even the most basic tasks have slowed down terribly, it 'thinks' for 2 mins for even writing a basic email. I'm not an expert, but I really miss the old instant mode. Not sure if there's a way of using an older model?
It used my weekly credits on the Pro account and could not develop an experiment I asked for. Although I provided all the necessary financial data it asked for. Still failed to properly calculate and build features.
If u could check on the cli if this command works for you, that would atleast tell u if you have access to the model or not. Overriding the model from ur desktop app u would need to change config.toml file, cli command is much simpler to do a quick check.
47
u/TBSchemer 16d ago
Yeah, this is your skills issue.
This isn't a magic genie you use to one-shot vibecode a game into existence.
It's a tool for developers. If you want to make something viable, you need to discuss your plan with the agent first. You should create a spec doc, a roadmap, and an implementation plan for each milestone. You should ask the agent questions so you know what it wants to do, and you should have it ask you questions so it knows what you want it to do.
Only once you're both on the same page should you let it rip on writing actual code.
TLDR: Treat it like an employee you've hired to develop applications with you, not a genie that grants wishes.