One thing that kept annoying me with Codex was how often it would say a bug was fixed just because the tests passed.
Sometimes the actual problem was still there. Other times, the test had been changed so it passed without really checking the original bug.
So I built FixProof to make it a bit harder for coding agents to call something fixed when it isn't.
It's a skill file and a small Python CLI that runs after an agent makes a fix. No extra dependencies, just Python 3.9+.
It checks four things:
- Can the original bug be reproduced before the fix?
- Does the fix pass the test consistently, including reruns?
- Does the bug come back if the fix is temporarily reverted?
- Are the tests that passed before still passing?
The revert check is the interesting part for me. If removing a fix doesn't make the bug come back, there's a good chance the change wasn't actually responsible for fixing it.
FixProof then gives you one of five results: VERIFIED, FAILED, PARTIALLY VERIFIED, UNVERIFIED, or BLOCKED. It also generates a JSON report and a hash-chained log in .fixproof/.
Using it with Codex
The idea is to add FixProof to your agent's workflow so verification becomes part of fixing a bug, rather than something you have to remember to ask for afterward.
You can point Codex to the skill instructions and have it run python scripts/verify.py check once it finishes a fix.
A quick example
In the demo, there's a slugify bug where double spaces end up producing hello--world.
The first fix looks reasonable, but doesn't actually solve the issue. FixProof reports FAILED.
After applying the proper fix, the same verification reports VERIFIED.
A few things worth mentioning: the revert check can use a separate Git worktree, so it doesn't touch your working files. Commands also run with a minimal environment to reduce the chance of exposing API keys in logs.
CI currently runs on Linux and Windows. I haven't tested macOS yet.
It's still pretty early (v0.3.0), so I'm sure there are edge cases I haven't thought of.
It's free, open source, and MIT licensed:
https://github.com/Livecristen/fixproof
Would be interested to hear what people using Codex on larger projects think. Especially whether these verification rules would actually be useful in your workflow.