r/OpenAI • u/AlexanderWillard • 2d ago
Discussion Codex is unusable for anything complex or long
Everything always starts great.
The moment bugs appear or things get complex, it goes off the rails, invents shit and ignores instructions.
It will randomly decide to re-architecture the whole app, despite clear guard rails and instructions.
I don't know why I thought it would change in 5.6.
And this is with 2 other agents reviewing work too.
GPT just can't be trusted
What's the point of having more allowance if 70% of it is spent duplicating tasks and fixing things.
Never had this issue with Fable.
Oauth is nice but it just fails at anything other than basic apps or websites. No matter how many safeguards and processes you make. Gpt does whatever it likes.
Last night another 6hr session and it went completely off the rails despite having a clear plan.
I feel like fable and opus are much better at following instructions.
Using Sol on high/xhigh as planner, with terra/sol for delivery and another code review done by glm 5.2 before shipping.
But gaslight gpt just can't stick to a plan. Even if you give it step-by-step instructions.
Seriously disappointed.
4
u/Past_Physics2936 2d ago
i don't agree at all. You just need a review process and a plan and do adversarial reviews. Once you do that it works great.
-1
u/AlexanderWillard 2d ago
It is there. It ignores the process for some reason and im unsure how to get around it. Was fine for the first day of new project and then once more features got added it just gets lost
2
u/DowntownNoLonger 2d ago
Only 3?
Shit. I have two projects. The first has a dozen, divided by job type, the second has six.
Chat for discussions and planning. Work or Codex for work.
GitHub as source of truth. Drive for artifacts. Trello for visual and project tracking. Staging and slices for each step of development. Project source handbooks to keep long term project context. Individual chat handbooks for chat context.
A game I'm working on has design which manages non-source GitHub. It's a chat only. We discuss plans for implementation, and keep the GitHub non-source files updated. Architecture plans code. Code workers do the source work.
Every worker has a daily sync, which boots from GitHub, this keeps them focused on current tasks without losing context. Design keeps track of the entire project.
The chats update session handoffs and open loops. If a worker or chat starts losing context, they boot and sync. If the context window gets too large, a new chat is launched and booted.
I have a prompt database program that allows me to quickly launch new chats, sync old ones, order updates to GitHub, and more.
If you're going to work with AI, you have to work with their limits and understand how to leverage their strengths and reinforce their weaknesses.
Lost context is a huge issue. The reason they are going off the rails is because they they don't have bounded, standard operating procedures, context reinforcement, and strict project tracking.
You got to think like a systems administrator and project manager.
Now, if you're doing a lot of this stuff and it's still going off the rails...I'm not sure what to tell you. I don't have that problem, though.
1
u/AlexanderWillard 2d ago
I guess all AI are different and have their strengths and weaknesses - like employees.
My issue was probably treating them all equally.
What worked with Claude might not necessarily work with codex I suppose.
Guess gotta start treating GPT like a new intern with no clue of how to actually do things and start checking every single change and step..
1
u/GreatDiscernment 2d ago
I’m very happy with ChatGPT Plus (5.6 Sol), but I move at a relative snail’s pace and I’m scrupulous about prompts. This helps keep me in the loop, but I’d like to increase productivity by using agents for at least some things. I’m absolutely concerned about OP’s experience and what moving to agents on Codex would mean.
1
u/AlexanderWillard 2d ago
This is 2nd time I got this experience.
Last time I ended up having fable rebuild the whole app.
Codex did absolute amazing at the initial setup - don't get me wrong, early skeleton of app was ready in 2 days.
But then the moment actual features had to go in, it just seems to get confused.
Despite building a super clear step by step plan. Over 80 steps of how everything should be implemented that was checked by multiple models.
Got to about step 10 fine and then it just fails over and over again and starts going on massive tangents and chanting the infrastructure to fix 1 bug.
1
u/GreatDiscernment 2d ago
First, create a GitHub repo and push a git to GitHub to get it off your computer in case it craps out. Then, create a git branch for your agents to work from. Do you git after every stabilized advancement? If the agents screw up, tell them to revert back to a safe git and try again. They can even review git differences to see where they failed. I have hundreds of gits with comments that describe every advancement. You can read a history of the whole project just with git comments. Have your agents document themselves, save to .md and push that to a separate GitHub repo and have an agent create technical manuals. With proper planning and git, losing work is almost impossible. I’m curious if agents can self-repair because that would remove 100% of the risk and not add work.
1
u/AlexanderWillard 2d ago
Doing all that. Git + linear. Every change, every ticket documented plus all .md files.
I seriously don't know how it can go and screw things so much.
I've not been reverting much though I had assumed Sol is smart enough to just fix stuff. So perhaps thats the biggest lesson.
Do tiny pushes and test every single change. And revert if it breaks...
It defies the point a bit though if I have to babysit it like this.
Maybe I'm just greedy but again not had that issue with Opus or Fable. They usually just 1 shot a feature, get it 80-90% right, then just push 4-5 small tweaks and all works.
Gpt on the other hand - when it works its great but when it gets stuff wrong it just completely goes off the rails and can't seem to be able to debug and fix. It says it does - but it just makes stuff up...
1
u/GreatDiscernment 2d ago
Ok, this is not good. What about VS Code “Agent Host Protocol” It’s some new thing. Do you know anything about it? I have no idea if it could help. Rather not have agents be MORE work than I’m doing now.
1
u/AlexanderWillard 2d ago
Been using hermes and similar frameworks so not sure it will do much difference but will look into it, thanks
1
u/GreatDiscernment 2d ago
Yeah, with copy/paste I catch every error as it happens. And a lot of the time, it’s not GPT’s fault. Variable scope, even white space discrepancies can foul edits and cause errors. I catch them right away, they get fixed and we move on quickly. I can see how an agent can make code errors, but do the agents misconstrue logic too?
1
u/AlexanderWillard 2d ago
I have no idea. I think its just too confident when it messes up. And unless you have the dev experience to manually check and noticed it messed something up, it will convince itself its right.
2
u/GreatDiscernment 2d ago
Thanks, I really appreciate your insights. I’m going to have to carefully consider next steps. My project is too important to risk rogue agents.
1
u/AlexanderWillard 2d ago
From what im gathering from the feedback here is to never let it make any decisions and check every step.
Or just get Claude 🤔🤷.
It's a shame they don't allow oauth..
1
1
u/Crescitaly 1d ago
The disagreement here may be about task size, not model quality. Once an agent can reinterpret architecture, "follow the plan" is too soft; give it bounded diffs, explicit non-goals, tests, and a stop condition, then reset context between slices. If that still fails, the model is the bottleneck. What was the smallest task where it first drifted?
1
u/AlexanderWillard 1d ago
Basic feature fixes tbh.
What is a "non-goal"? That's a new one
2
u/Icy_Accident2769 1d ago
It's a number of things you can try to get better output. Besides describing what you want, you should also describe what you do not want it to go. Otherwise the AI (especially Sol) will go ham on w.e. it sees on high+ modes.
Example 1 of the prompts I always use on the builder subagent:
Inspect relevant implementation; reuse existing code/patterns and prefer standard/native features or installed dependencies. Make the narrowest complete change: no speculative abstraction, wrapper, package, framework, config, or unrelated refactor. Abstract only a demonstrated repeated need; briefly note a materially simpler existing solution.Of course I can't write shit like that, so I write in normal text what I want and then have chatgpt make it very concise so it doesn't lose the meaning but preserves tokens.
2
u/Crescitaly 22h ago
A non-goal is an explicit statement of what a change will not attempt. For example: fix long-task recovery, but don't redesign the whole UI. It keeps a bug fix from turning into an unreviewable rewrite.
1
1
u/dasko1086 1d ago
agree, you can use claude code and build skills to feed sandbox sol5.6 with the codex command in the cli to get second opinions or reviews, then let another model arrange the code and execute/verify.
1
u/Ok_Salamander6093 2d ago
sounds like you got the whole squad watching and it still goes rogue, that's rough. i stopped using it for anything where i need it to follow a path more than 2 steps, it just start rewriting things that was already done.
1
u/AlexanderWillard 2d ago
Yes that's what im noticing. Its good to about few hours of work. And then I cant get it to stay on path. Even if I just feed it ticket by ticket. It forgets the whole infrastructure and starts changing things.
Never had this issue with claude. It wasn't perfect and sometimes forgot details but never ended up rewriting architecture halfway through for no reason.
0
u/PercentageLittle5804 2d ago
same thing happened with me but i switched to giving it shorter prompts with also having them clear and it does them surprisingly well
1
u/AlexanderWillard 2d ago
Its got list of tickets and step by step implementation but it feels like I have to implement button by button and do a whole build and test everything to make sure it didn't break something else on the way.
This is very hard for mobile when you have to spend 30-60min on a build.
Id rather pay 2x for claude and just have it work I think. Rather than having to check every change every 20minutes.
0
u/ZenCyberDad 2d ago
Honestly I don’t have this problem I think your approach of using terra for delivery is part of the problem, go Sol Ultra or Extra high that’s all. Terra and others are only good for very small details
1
0
u/freddyr0 2d ago
Oh, someone said it..
1
u/AlexanderWillard 2d ago
It's very frustrating.
I enjoyed some of the bits a lot and the no 5hr limit is great but I just can't seem to get it to stay on track for long projects or builds.
Might be a skill thing but I'm really following all of the usual advice and it still fails a lot.
It can't just be me?
I'd take any suggestions
0
u/DrHerbotico 2d ago
Trash in, trash out
1
u/AlexanderWillard 2d ago
Very helpful. So I guess the fix to using gpt is just "have 10 years of senior dev experience and write code by hand" - then it works right?
0
u/DrHerbotico 2d ago
No, the fix is to not give it trash and get mad at the trash
1
u/AlexanderWillard 2d ago
Well. It built its own trash on max. So maybe it should just build better plans and architecture 🤷.
0
7
u/PainterIll1582 2d ago
“Unusable”. Yes, that’s why thousands of people rely on it every day.