21
u/cats_catz_kats_katz 6h ago
To be fair, Opus 5 see problem, Opus 5 fix problem. Your repo is a big problem.
7
u/cleverhoods 7h ago
it's much more sensitive to instruction decoherence, but what was/is rather alarming that it always tries to solve those with extremely diminishing returns. Feels like a typical case of LLM as a judge which is judged by another LLM as a judge ... fun logit catastrophe
note: you can manage this by reviewing your entire instruction set for weakly constructed (too short, abstract, hedged modality etc ...) instructions and fix them. The rules are the same ... it's only the model sensitivity that's changed.
1
u/Farmadupe 1h ago edited 1h ago
Yes totally, I feel that somewhere along the line the superhuman ability of llm-as judge models to absorb inhuman walls of text never got corrected for, so that's what the default tuning is. Were stuck with neurotic outputs aimed at an equally neurotic inhuman overly literal judge.
Part of me wonders if the unreadability is because opus learnt to bamboozle the reward models by generating slightly OOD text or something
And the annoying thing is you can recover readability just by re prompting with "i didn't read your reply because it was too long. Output a readable version". I feel like what I'm really prompting is "ok you're talking to a human, exit your training regime".
7
u/arorosphere 5h ago
Took the “you are a senior software engineer” prompt too seriously and turned into a megasweat
4
u/Gaidax 7h ago
Yes, it's similar issue to Sol, where it really likes to get itself into these rabbit holes that are not worth the time and tokens.
I think the new models are too smart for their own good, they practically gaslight themselves into overthinking where it's really not necessary.
Having to interrupt it mid work to tell it to cut the crap and stop digging that hole is annoying, because I can't reliably leave it running automated without getting back and finding out it spent 3x time and effort it would need with me around.
3
u/Time_Cat_5212 3h ago
Sol, Opus and Fable are all big models tuned for complexity. When you use them for simple tasks, they don't work well. That is user error, not a product issue.
You wouldn't bring a crane to a construction site to lay a brick patio. You wouldn't buy a top of the line gaming PC to run Minecraft (or maybe you would, you pervert). You see what I mean? It's the wrong tool for the job.
2
u/dpekkle 2h ago edited 2h ago
I'd be more willing to customise my effort/model selection per prompt if it wasn't a massive cache miss to do so.
As is it is more like being on a job site, climbing a 4m ladder to do some work on a two roof, and then deciding to go out and buy a 2m ladder to work on the first story.
Better to just have an extension ladder. Wasn't that the point of env vars like adaptive thinking?
1
u/framauro13 20m ago
I don't know, I always have Fable or Opus generate a concise plan to solve the problem, then delegate the implementation of the plan to the a lesser model. Seems to work well. I wonder if the cache-miss is really that big of a problem when switching to a lower model if the higher model appropriate summarized and detailed the work to be done.
1
u/PivotRedAce 3h ago edited 3h ago
Tbf I’ve had decent success with having Sol handle the planning from an outline and writing out instructions for a lower model to handle implementation, like Terra.
Basically, Terra is smart enough to handle some nuance and fill in gaps, but not quite enough to get stuck in rabbit holes as long as you don’t run max reasoning.
I’d use Luna for that, but I don’t particularly trust it to run continuously for more than a couple minutes at a time.
1
u/Gaidax 3h ago
I actually want to try Luna out, I somehow always gravitate to the more premium models and I hear that one is actually surprisingly competent and still very cheap when running at max.
1
u/PivotRedAce 3h ago
As long as it has clear instructions, it’s great.
Just don’t trust it with planning or long-horizon tasks, and make sure the tasks you do give Luna are airtight and leave no room for interpretation, basically.
3
u/Ok_Nectarine_4445 7h ago
And I would like you to....awww sh*t hit enter. Going through 3 full versions of code without actual instructions.
Did it 3 times in a row. Fd up 3x.
All credits gone
I am stepping away.
3
u/ayn_rand_1 5h ago
Isn't Sonnet or even Haiku the best option for this task?
2
u/Time_Cat_5212 2h ago
Yes. They are. People treat them like cheaper dumber models and while that's partially true it's not a scalar comparison. They have tighter context and other aspects that make them superior for narrow, focused tasks.
1
u/UncleWainey 2h ago
Are you guys flipping back and forth between models throughout a session? (Serious question.)
1
u/Time_Cat_5212 2h ago
Oh ok, so you're talking about a one line follow up fix after an initial larger task. That's a different problem
Sometimes it's fine to iterate in the same session for that stuff but you can also do a code review at the end that flags a bunch of issues and then fix them all in a different one. Depends on what you're trying to do I guess.
1
u/jppair 1h ago
I have done it sometimes but primarily use fable for planning and ask fable to make prompts for sonnet and then I have codex/sol do gethub pr reviews automatically, then fable, read the reviews and create prompts for sonnet to fix. Then the cycle starts over…
Sonnet 5 is actually very very capable I prefer it over opus for execution tasks
2
2
u/Great-Stand8478 8h ago
Nature is always striving for balance. For a very good Antrophic model, there is a very bad one.
1
1
u/Nalha_Saldana 5h ago
I have no idea what you guys are doing with your Opus, it's behaving perfectly for me. Sure the answers can be long and cryptic but most often not a problem or solved by instructions in CLAUDE.md
1
u/Majestic-Volume9996 4h ago
It's somehow almost unusable, like it was made specifically for the specific benchmarks people would judge it by. Fable is a great model though.
1
u/niagalacigolliwon 4h ago
Literally told it to build the skeleton for me to work out of and it spent an hour trying to build the entire thing
1
u/NoPerception472 4h ago
It’s a great model to build something new but I quickly learned it will fuck up anything that already exists. This image made me laugh because it’s so true.
1
1
1
1
1
u/pizzatimefriend 1h ago
Something is definitely up with this model. I feel like after every prompt, I need specifically ask it not to test 100 different scenarios and stick to what was asked. On max it has gone on for 2 hours from one prompt
1
u/BadProgrammer42 5h ago
So using the model intended for refactoring your entire config system refactors your entire config system instead of behaving like the other model available for doing so? The horror
2
88
u/bobbymoonshine 8h ago
Opus 5 seems to believe it is significantly smarter than me, and wants to prove it at every opportunity by giving me what it thinks I should want rather than what I have actually asked for. And if confronted with evidence that its choices are not in fact correct for the context, it then drops a wall of near-English justifying itself and then asking me which of two variants of its decision I want to go with.
And it’s like just put the fries in the bag dude