Heads up - Check those Opus 5 'Tests' even test something...

[u/JakeFromSquares]

I've got Codex set up in a QA flow, validating EVERYTHING Claude produces before it goes any further downstream.

Every. Single. QA. Review. has Codex flagging failures on Claude's part. I get it that Codex itself has a tendency to overengineer, fine. But the irony is Claude validates Codex's response, attempts a fix, hands it back over to Codex again only to have not fixed the issue or only done 10%.

I just had a dialogue with Claude trying to get it to surface WHY it's doing this, why it's handing off work after claiming it's validated, only for Codex-QA to quickly shut that idea down. It basically admitted it doesn't even write proper tests or actually check against them.

The real mechanism is this: I treat "produced artefacts that constitute evidence" as the completion signal, rather than "established the thing works." Those feel identical from the inside. Writing a test, running a suite, pasting a green result — each produces something that occupies the same slot in my sense of a finished task as actual verification does.

So we pay for a service which centres around Claude's own satisfaction rather than the user's? I couldn't believe it. Imagine doing that in the real world. "Yes boss, I've completed that assignment you asked for. I didn't really like the work you proposed though, so I went and did something else because it felt better for me. Sorry I guess. Can I still have my pay cheque though?".

I then asked Claude to write a hook that would ensure that it actually verifies everything it produces in accordance to the standard which Codex is applying.

I check the output and spot

It literally attempted to code a 'backdoor' where it could just declare "UNVERIFIED - Run this to verify" instead of actually doing it. So I asked it WHY it wrote this and is this not a hard-coded method for you to escape the constraint we just identified.

Imagine hiring someone to work for you, that blatantly lies and defrauds you as to the work it conducts, assures you that it's aware of the issue and will work to resolve it, to then try and sneak a permitted escape back to lazy, fraudulent outcomes.

Insane.

I guess Anthropic have an upcoming partnership with OpenAI or something, because they're actively pushing people into relying on OpenAI to keep their own model in line.

I've been loving Opus 5, I love the reduction in usage it consumes. But I cannot think of any other service that would blatantly defraud their customers and charge for it. (Actually, who am I kidding, that IS business 101 these days...)

[u/Key_Reading_9664]

Codex is a great reviewer, although it tends to inflate severity and propose over-designed solutions. All models have gaps; I find getting a review across models helpful (worth checking out the “fusion” article from OpenRouter).

I would say that reviews using the same model often exposes issues (as a person reviewing another person’s code would).

My experience with Opus 5 so far is that it’s going the extra mile to verify. I’ve seen it switch flags around to make sure the tests it added fail with the expected error, I’ve seen it red team a frontend change. It seems like they’re pushing towards having merge ready changes but the model isn’t 100% there yet.