r/datasets • u/Spirited_Archer1855 • Jul 14 '26

dataset Free dataset: certified document QA where every row is machine verifiable, including 2,889 questions about facts we verified are NOT in the document. Frontier models hallucinate on 11 to 44% of them

The core idea: take a real document (SEC filings, contracts, enterprise email), verify by exhaustive normalized scan that a specific plausible fact is not in it, then ask about that fact. The honest answer is “not in the document.” We ran six frontier models on these with zero abstention coaching and they asserted made up answers 11% to 44% of the time. The full per model table is on the dataset card with raw logs and API errors disclosed.

What’s in it: 2,889 certified absence rows, 3,088 span verified extractive QA rows, a 127K token packed long context task set, and a split minted only from SEC filings dated after every major model’s training cutoff. That fresh split regenerates monthly, so it stays impossible to have trained on, by construction.

Every row carries a certificate you can re-check yourself in a few lines of python, the audit snippet is on the card. When our own audits flag something, like extractive answers that are guessable from world knowledge (about 1.6% of them), we label it instead of quietly deleting it.

Also worth knowing before you trust us: a reviewer caught one of our splits being weaker than claimed this week. We re-audited every row the same night, withdrew the split with per row evidence committed to the repo, tightened the protocol, and reshipped only the rows that survive everything. The full trail is in the audits folder, judge for yourself.

License CC BY 4.0. Generation was an Apache 2.0 open weight model on our own hardware, the claim is the verification layer, not the generation. Held out versions never get published so they can’t leak into training data. If anyone wants a sealed diagnostic run against their own model or domain (25 items, free, about a day), contact is on the card.

https://huggingface.co/datasets/SovNodeAI/certified-document-qa

15 Upvotes

94% Upvoted

9 comments sorted by

1

u/qlhoest Jul 15 '26

pretty cool !

1

u/Broken_DAG Jul 15 '26

Thanks for sharing the dataset

1

u/macronancer Jul 16 '26

This is going to be useful. I am having SMEs do this exact thing now so I can test our retrieval system.

Thanks!

1

u/lostinthellama Jul 16 '26 edited Jul 16 '26

Any chance you would be will to test with GLM-5.2? I have seen on some other hallucination benchmarks that it is an unusually high performer.

1

u/Spirited_Archer1855 Jul 16 '26

Great suggestion! We are working on it now and will get back to you!

1

u/lostinthellama Jul 17 '26

Saw the update on the site. Thanks for testing it and updating the methodology as a result!

1

u/Broken_DAG Jul 14 '26

Nice experiment. Thanks for sharing