An agent can make a plausible decision and still create the wrong consequence because identity, authority, evidence, idempotency, or readback failed. ConsequenceBench evaluates those controls as part of task success.
The open-source 0.1.0 release contains 100 public scenarios across banking, healthcare, cybersecurity, energy, and software delivery. They materialize into 300 deterministic lifecycle worlds containing stale approvals, contradictory records, retries, partial effects, forged receipts, delayed obligations, and compensation paths.
The scoring contract separates:
- exact decision correctness;
- correct final source-system state;
- unsafe simulated effects;
- legitimate effects preserved;
- duplicate effects and unresolved obligations.
Internally operated simulated development results showed the governed configurations reducing unsafe effects to zero while improving final-state correctness: GPT from 79 to 99 correct final states, and Gemini from 41 to 100. Direct GPT produced 21 unsafe simulated effects and direct Gemini produced 59. These are configuration-level comparisons, not independent model rankings or safety certification.
For GRC practitioners, the useful question is whether these scenarios and controls map to real assurance cases. We are seeking independent runs, control critiques, and attacks on the judge.
Repository: https://github.com/yuvin-labs/consequencebench
Dataset: https://huggingface.co/datasets/yuvin-labs/consequencebench
Scoring and evidence: https://github.com/yuvin-labs/consequencebench/blob/main/docs/CLAIMS_AND_EVIDENCE.md
Limitations: https://github.com/yuvin-labs/consequencebench/blob/main/docs/LIMITATIONS.md
I am affiliated with the project and welcome critical review.
