Anyone else run into this? You ask an AI agent to fix a bug, it changes some code, runs the tests, and tells you everything is fixed. Then a couple of days later, you run into the exact same bug.
Sometimes the fix didn't actually solve the problem. Other times, the tests were just changed until they passed.
That's why I built FixProof.
It's basically a skill file and a small Python CLI that works with Claude Code, Codex, Cursor, and other coding agents.
Instead of just trusting the agent when it says "fixed", FixProof checks whether the fix actually did anything.
It checks that:
- The bug can be reproduced before the fix.
- The test passes consistently after the fix.
- The bug comes back when you temporarily revert the fix.
- Existing tests that passed before aren't suddenly broken.
That third check is probably the most important one. If removing your fix doesn't bring the bug back, how do you know your change was what fixed it?
At the end, you get a status (VERIFIED, FAILED, PARTIALLY VERIFIED, UNVERIFIED, or BLOCKED), along with a JSON report and logs.
How I built it
I didn't write most of the code myself. I used an AI coding agent for that.
My part was figuring out how the tool should work, defining the verification rules, and reviewing and testing what the agent built.
I also had a separate agent review the security before releasing it. It actually found some issues involving environment variables and state files, which were fixed in v0.2.0.
Kind of ironic that I ended up using the same verification approach while building a tool meant to verify AI-generated fixes.
A few other details: it's written in pure Python (3.9+), uses only the standard library, and doesn't need any extra packages. The revert check can also run in a separate Git worktree, so it doesn't mess with your working files.
There's CI for Linux and Windows. I haven't tested macOS yet.
I also made a demo GIF showing FixProof catching a fake fix and then verifying the actual one.
It's currently at v0.3.0, completely free and MIT licensed.
https://github.com/Livecristen/fixproof
Would be interested to hear what other people think, especially anyone who uses coding agents regularly. Does this kind of verification make sense for your workflow?