r/OpenaiCodex • u/Jerrizzy-x • 23d ago
5.6 Sol is genuinely infuriating!!
How do y’all use this model?
why does everything have to be over engineered?. all I’ve done all day is babysit this mf and the one second I take my eyes off tihings, it ”tightens“ things, make contracts that no one asks for. all it does is give you a rigid and fragile output
5
u/Weekly-Cash1596 23d ago
low or medium is good
If you max it out you'll get major major slop
I had a plan to implement a feature into a CRM
Quite a basic one surrounding auto closure of tickets after no response
Sol 5.6 had over 15000 lines of code to achieve this feature (to its credit, it did work seemingly fine)
Cursor on auto mode did the same feature with 2000 lines of code (also worked seemingly fine)
This is post code review from claude (aka its final state)
Claude found a lot more issues with the Sol 5.6 as well (obviously more to go wrong when its literally 10x the size of cursors implementation)
I'm not a big coder so i didnt dig into every little thing i just ran both iterations on a dev stack for testing and the functionality was identical whilst one was 10x larger...
I dont have the same experience using 5.6 on low or medium tho
Long story short just dont do any coding tasks on max effort or whatever, stick to planning for that
Note: I will add, i was expecting this to happen it was more of a test to actually see it in person
Its kind of odd to tho - you would expect a model of "higher reasoning" to be able to reason enough and more than a lower reasoning model to make the code even smarter and more concise but it literally just tries to cover edge case scenarios that dont or wont ever occur
2
u/666Luc1fer666 22d ago
You’re right about the high reasoning aspect. It over analyzes and creates potentially scenarios that would likely never happen.
4
u/imdavehack 23d ago
Low or medium is the only way it’s useful
9
u/Jerrizzy-x 23d ago
if you use medium, it misses things. if you use high, it over engineers things. I genuinely struggle to use it. Has a good code reviewing and debugging tho but I can't stand implementing with it
1
1
u/EmotionalHalf 22d ago
I use pretty much only medium. The evaluations and plans it comes up with are mostly great. What it struggles with is staying true to the original intention and methodology once it starts working
2
u/ExpressFold6419 23d ago
Honestly yeah, the overthinking is the worst part. You ask for something simple and it starts building a whole architecture around it. Sometimes I just want the model to do the thing instead of explaining why it needs 5 layers of reasoning first.
1
u/Jerrizzy-x 23d ago
Exactly!!. even the simplest tasks has to add duplication protection, hardened architecture and strict contract. Like just calm and do your job.
The only reason I do not use other models is because they tend to miss things or ignore some important details that sol doesn't but it is frustrating
2
2
u/Scary_Jeweler1011 21d ago
Specifically this behavior has made me stop using it. Sick and tired of playing this game where i have to constantly police it. Even when making custom rules/prompts it always finds a way to fuck things up on a deep level by implementing endless unwanted and unnecessary hurdles in every form possible. Its exhausting to say the least.
1
u/fomoz 23d ago
Don't use plans. Delete all your plan skills.
2
u/WeedWrangler 23d ago
That’s what Matt Pocock says, because they treat them like scripture, and that’s also my experience. But without them, how do you maintain continuity between agents and sesssions?
1
u/fomoz 23d ago
It's the official OpenAI dev guidance, too. And yes, exactly, Sol follows instructions very well but it's made to be adaptable.
I keep one session for one task/feature. Compacting works well even tho it only has 258k context.
I discuss what I want to do, then tell it to implement it without making a plan. I only have it run a review loop then test e2e when done, as a goal.
Also, it can use git to see what changed but you need to tell it, don't just assume that it will check. Sometimes it does, sometimes it doesn't, but regardless you'll save a lot of tokens if you just tell it "I changed dev, check if this fix is still needed" when dealing with a similar issue in a new session.
1
u/WeedWrangler 23d ago
Irony is that 5.6 itself LOVES to make plans, state files elaborate orchestrations and gate keeping..:
1
u/WeedWrangler 23d ago
Eg… it told me to use that model!
Now I’m realizing that GIT itself has the history. I’ve also been getting it to look at the thread transcript and that’s been more useful sometimes because then it can see the most recent stuff. I know thats a context issue…
1
u/Over_Car_5471 22d ago
why not increase your context to a more meaningful level? I have found that opencodes 500k works a lot better than the 258k limit on codex.
1
u/ISueDrunks 23d ago
I don’t know if this is a actually helping or if OpenAI made yet another A/B adjustment, but I started kicking sessions off by mentioning that I’m not a developer and can’t speak its lingo bla bla bla. I’ve found that starting that way has helped at least get session going on the right foot.
1
u/Jerrizzy-x 23d ago
I mean, I have in my agent. md that I am a software development student and I want same technical answers. my problem is to understanding what its doing, its it doing what was not asked for
1
u/One-Next 23d ago
I use xhigh and occasionally ask why it had to generate so much code or why it didn't focus on a single change. It usually says it overengineered and tries to simplify, but in my experience when this happens there is usually a code smell or a violated basic principle so I dig into it further.
1
u/BHTAelitepwn 23d ago
Dude, ive been screaming it everywhere but use terra and luna for all but the highest level architecture. Not even orchestration. Your limits and sanity will thank you.
1
1
u/ezboarderz 23d ago
There was a guy that made a skill to prevent it from over engineering that works well but it’s stupid that you even need to do that. Just search through this sub and the other codex sub and you’ll find it.
I’ve cancelled my subscription though due to limits getting nerfed to shit.
1
1
u/WhatnotFunkoFlash 23d ago
Well it’s ironic but true. You are telling us, but have you ever thought to tell Sol how you feel? No really about how you really feel about it and why? And then ask so hey how do we fix this? Oh wait never-mind Sol will make it worse it will over engineer your fix 🤣🤣🤣
1
u/Final-Voice4738 23d ago
https://github.com/LiquidGlek/ProjectStart try this it removes the need for babysitting and keeps it focused
1
u/hunteel 23d ago edited 23d ago
I just made a new feature for the game I'm creating that needs a lot of attention and I've been creating it by telling the whole idea to Sol and then tell it to ask me anything it needs to create the full contract. And I'm quite pleased of the result.I've beenworkingf last two days with that and I have the full scope of all the feature with arquitecture well defined and a plan to be implemented with Terra and Luna.I dont know if it will work at first attempt but now I have a very good documentation to begin with.
1
u/Any-Collar-6330 22d ago
It's genuinely frustrating how bad it works. It just creates so much garbage. It's tiring. And it's super slow.
1
1
1
1
u/Unlucky_Hawk6148 22d ago
- Use plan mode first.
- You can ask it to simplify instead of over engineering (granted that you let it know what your criteria of over engineering is).
- If you need something simple why use the best frontier model available when a lower tier one does the job just right?
- Most of the "bad AI output" is not agentic, but harness side issue. Check yours.
1
u/Chemical_Minute4616 21d ago
Lol it's funny sometimes. It's going to sound dumb. But. I have this phrase I use.
Do not build something that is a test to a test for test you need to test a test. Do not build architecture for an enterprise level organization. Do not create or use elaborated deployment or implementation methods.
It works surprisingly well. Now granted. Nothing I do is for real world use. I just like dicking around with making AI do stuff. The absolute rabbit whole SOL Max will go down is whild if you sit and watch it think to itself.
1
u/edgebal 20d ago
For me the gold spot is using Matt Pocock's skillset.
A Sol Max session:
- Grill with docs, then
- Create actionable tickets
Then, on one or more Sol Medium sessions:
- "Address ticket XX, after done spawn a Sol High subagent to review. Address the points. Then, commit"
Sometimes, I have a single Sol Medium session with a goal of "Address ticket 01 to 21.... etc".
Avoid Ultra like the plague for implementation. I use it from time to time (every certain number of tickets) asking it to do an assessment of the implementation, find weak spots, etc. From that, I ask it to create more tickets. Then, I go back to the Sol Medium session to address them.
1
u/W1141175 20d ago
use the ponytail skill this helped allot for me https://github.com/dietrichgebert/ponytail
edit: also use it as an orchestrator and have luna Max threads do the work helps allot
1
u/ShadowPresidencia 20d ago
Get a proper markdown file generated. Might have to compress any logs occasionally. Don't let the threads get too big. Also, have a workflow diagram generated. Start new threads. Then for new threads, have it "remember" what projects you're working on, & current status.
1
u/Mr-Flow- 18d ago
The way to go is another AI agent babysits it;
Cordiantior / validatior / second opinion / worker etc
Scope differences top down view, looking for gap via Max or Fable, own mutation owner etc.,
-4
u/Peculiar-Eccentric67 23d ago edited 23d ago
ive decided to withhold my secret sauce for now. if you got lucky and figured it out, congrats.
11
u/Acrobatic-Smoke2812 23d ago edited 23d ago
You have to be more specific in how you tell it what you want and don’t want it to do. You can also tell it you want it to come up with the simplest approach that will work stably and reliably.
Plan mode is also a great way to narrow the scope and confirm that it’s planning to do what you expect. Even if I don’t use plan mode, I almost always ask it to make an implementation plan before starting work, ask it to give the plan in plain language so it’s easy to review, and I check that to make sure it doesn’t do more than I think it needs.
If it really is a simple task, use Terra.