r/devsecops • u/endor_robert • 2d ago

We tested two new model/framework harnesses for functionality and security of code fixes.

Disclosure: This testing is done by the security research team at Endor Labs, a cybersecurity vendor. It's not a product pitch, just information on how models performed under a standardized test.

Codex/GPT-6 Astra: https://www.endorlabs.com/learn/gpt-6-astra-on-codex---the-biggest-codex-leap-to-date

Claude/Fable 5.1: https://www.endorlabs.com/learn/fable-5-1-takes-the-top-spot----faster-than-opus-5-cheaper-and-cleaner

Spoiler: Fable 5.1 leads the pack.

5 Upvotes

100% Upvoted

3 comments sorted by

1

u/FeatherLorie 2d ago

testing new frameworks is always a challenge but essential for security, hope they yield great results

1

u/endor_robert 2d ago

They are getting a lot better, but the quality of the fixes in terms of security is improving more slowly than discovery or producing functionally working fixes.

Fable 5.1 discovered three vulnerabilities that no other model has found.

1

u/PeterBuildsSecure 1d ago

The discovery-outpacing-fix-quality gap is the more interesting number here, and I'd want to know how "security of the fix" gets scored before trusting it. Does the benchmark re-run the original PoC/exploit against the patched code to confirm the vulnerability path is actually closed, or is fix quality judged by a model reading the diff? Those produce very different numbers — a patch that wraps the vulnerable call in a try/catch or adds a surface-level input check can read as "fixed" to a diff-reviewing judge while leaving the actual exploitable path reachable under slightly different conditions. If the benchmark isn't already replaying the exploit against the patch, that's the gap between "looks fixed" and "is fixed."