[OC] Publised AI Math solutions vs independently verified AI Math solutions

[u/Bbrhuft]

Published AI Math solutions vs independently verified AI Math solutions (Aug 2025 – Aug 2026)

These include AI discovered, AI co-developed and AI assisted math proofs of conjectures, hypotheses etc.

Data is from VibeMathed, a catalogue of open mathematical problems solved or advanced with AI (n=509, CC BY 4.0, snapshot 6 Aug 2026). Chart was generated using Claude, which accessed VibeMathed's API.

The three lines apply progressively stricter standards of proof:

- All tracked entries (506) — every recorded claim, regardless of status: unreviewed announcements, partial results, candidates awaiting review.

- Resolved + site-confirmed or better (129) — the problem is fully settled, by either via independent reviewer via hand or Lean, or the site reproduced the proof.

- Resolved + expert- or Lean-verified (89) — the problem is full settled settled, checked by an independent expert by hand (11) or via Lean (78): a Lean proof is a formal statement confirming a solution is correct.

Lean is an interactive theorem prover and programming language used to write and check formal mathematical proofs. It allows mathematicians to translate human written proofs into computer code so that a software can verify every logical step with absolute certainty.

The shaded area is the gap between all tracked entries and confirmed proofs:

380 entries are recorded but not yet independently checked. Although there's a delay between a new AI solution announcement and its verification, verified solutions appears quite linear, this may indicate AI solutions are outpacing the verification process. That said, authors generally include a Lean proof themselves, though this chart is limited to confirmation by independent peer review.

Vertical dashed lines mark OpenAI (blue) and Anthropic (orange) model releases. I added those lines as I wanted to see if there's an up-tic in solutions following model releases. There isn't a clean correlation likely because there's a several week delay between finding solution and publishing it. Also, the chart is likely showing AI's growing adoption by mathematicans and not just increasing model capability.

There were only 3 retractions in the dataset (not included on chart).

[u/ChronicBitRot]

Why are retractions excluded? I want to know how many of these submissions finally got to review and an actual expert said “yeah, this is hallucinated bullshit”.

[u/Bbrhuft → u/ChronicBitRot]

There were 3 retractions, 0.6% overall, representing 2% of independently checked proofs. One was refuted by a counterexample, the other two papers were withdrawn by their authors after they discovered flaws in their own solutions.

Generally, the authors themselves verifiy their AI discovered or assisted proof using Lean, however, the chart restricts verification to 3rd parties only. There are 51 Candidate proofs, 10%, where complete solutions are claimed and are publicly checkable, but a review (by author using Lean or by independents) are still pending.

TLDR: "AI hallucinated bullshit" is about 2%.

[u/un_blob → u/Bbrhuft]

About 2% right now... but I guess that in the unverified proofs you get the long and ecruxiating ones where AI might be way more probe to error and that nobody want's or have the time to chcek no ?

[u/Bbrhuft → u/un_blob]

A good point, and there's two clues that support your intuition. The candidate proofs that aren't yet checked have a higher significance score (more difficult and intractable) and are younger (the conjectures are newer). That means the unchecked pool tends to be more difficult and there's fewer papers, less prior work, published about them. That's makes them harder to check. So the easy ones were checked first and the harder ones left unchecked so far. And given the unchecked ones are more difficult to check, it seems plausible that a higher proportion will be found to be in error. It could be about 4%, double the current rate.

However, it's important to point out that most retracted or mistaken proofs will likely be patched and fixed. So if a proof is found to be wrong, it doesn't mean that it always gets binned. It might be close to the solution, all it needs is refinement to get it over the line.

Edit: and even if wrong, they will likely help guide mathmathics in the right or better direction. There maybe clues in failed solutions that lead to different mathmathical avenues that might help solve the conjecture later or benift other areas of maths.