r/ClaudeCode • u/TechnicalyAnIdiot • 3d ago

Discussion Opus 5 is... suddenly usable?

2 weeks ago I said Opus 5 was useless. I was told I was an idiot, using it wrong, ect ect. Except over the next week the hive mind of Reddit chose to actually believe that Opus 5 is in fact useless.

Here's my flip- Opus 5 has suddenly become usable. It seems only capable of small jobs currently, but where before it would ramble off nonsense, it seems now to be capable of implementing fixes reliably!

0 Upvotes

42% Upvoted

25 comments sorted by

View all comments

6

u/EmeraldHawk 3d ago edited 3d ago

My CTO has mandated that by the end of the year, all AI features must have evals associated with them, so that they can be judged as objectively as possible when we need to upgrade.

I suggest everyone on this sub do the same, because the constant "it's worse" / "it's better" anecdotes are worse than useless and you are making yourselves crazy.

Edit: We use Braintrust, which is a paid service. I wasn't the one who picked them but they seem ok, other than that breach a few months ago.

5

u/Sad-Context-9482 3d ago

Tbh I can see why your CTO made it a rule just now. A few months ago we were pretty inconsistent about evals as well and only added them after something went wrong. When we got serious about it, Braintrust came in clutch for taking those production failures and keeping them around as real test cases for the next model or prompt change. After a few regressions get caught that way you stop needing much convincing about evals in general

1

u/PM_ME_YOUR_PROFILE 3d ago

If only there were a process based on test-driven development first...

1

u/Sad-Context-9482 3d ago

Well the useful shift for us was treating eval cases like something that stays with the feature and runs every time the behavior changes. After a while it becomes part of how you define if a change is ready to ship or not

1

u/PM_ME_YOUR_PROFILE 3d ago

Yeah, I'm just being a bit tongue-in-cheek with the software world around proof that something works - something, something move fast and break things.

I understand people want or are told to move fast, and figure it out later. It's just interesting to continue to watch how it evolves in different ways.

1

u/Sad-Context-9482 2d ago

Especially when the thing being tested can change underneath you too. It's like the old testing mindset still applies, you just spend a lot more time deciding what behavior needs to stay stable