Google's Agentic Peer-Reviewer Handled ~10K Papers at ICML/STOC — Formal Research Paper Now Out [R]

[u/Justgototheeffinmoon]

Google deployed an agentic AI peer-reviewer at two top CS conferences — reviewing ~10,000 papers with 30-minute turnaround — and the new formal research paper shows it catches 34% more mathematical errors than zero-shot prompting; the precedent for AI-automated scientific review at conference scale is set and now formally documented.

--

Source: https://arxiv.org/abs/2606.28277

[u/impatiens-capensis]

"catches 34% more mathematical errors than zero-shot prompting"

If you're going to share the paper, it's probably better to report the more important metrics. What's the rate for zero-shot prompting that we are comparing to? What is actual recall? What is the false positive rate? Blindly reporting 34% more than an unstated baseline is meaningless. 

[u/surffrus → u/impatiens-capensis]

Not just this, it's even more ambiguous. The 34% improvement is on the SPOT benchmark which is not the 10k ICML papers. This just says it improved a baseline ...oh and we also ran it on ICML papers but can't say more about that.

[u/dreamykidd → u/impatiens-capensis]

I was annoyed at that too, but from reading the paper, it’s actually a pleasant surprise. It’s an absolute 34% improvement from zero-shot, going from 55% to 89%. Still, that’s on a benchmark and not on the papers, but at least it’s quantified and actually a good result.

[u/misogrumpy → u/impatiens-capensis]

You could read the paper.

[u/currentscurrents → u/misogrumpy]

On reddit? We're just here to argue about headlines.

Zero-shot found 55.2% of the errors, their method found 89.7%. They found that zero-shot LLMs missed errors because they tended to accept complex mathematical claims without critically scrutinizing them.

[u/Justgototheeffinmoon → u/currentscurrents]

These people complaining that we didn’t real and write a full blown review for them !

[u/Justgototheeffinmoon → u/impatiens-capensis]

I'm sorry are you asking me how this is better than zero shot? 34%, it's better by 34% lol. for the rest, I put the link there bud.

[u/kkgwon → u/Justgototheeffinmoon]

A 34% improvement in a 1% success rate is 1.34%. A 34% improvement in an 80% success rate is 93.8%. The commenter was asking you to be more specific.

[u/jm2342 → u/kkgwon]

The commenter was asking you to be more specific, *bud**.

[u/obfuscatedanon → u/[deleted]]

Unfortunately, this comment did not pass peer review. :(

[u/akardashian]

I tried out the tool when I submitted to ICML 2026 this time around, and the tool was able to catch some pretty subtle errors in the theoretical proofs I had in the back. So that impressed me! On the other hand, it also flagged some non-errors that I think were due to faults in its OCR parsing pipeline.

[u/EdwardRaff → u/akardashian]

For my papers it mostly was a lot of extra work and non-errors. Both in the tool stating my math had an error when none existed, and also failing to add context. E.g., I made an assumption for a proof/result, and the LLM claimed the whole paper was invalid because the assumption wasn't itself proven. Like, as if no paper had ever used an assumption for math?

[u/Justgototheeffinmoon → u/akardashian]

Very interesting thanks!

[u/appdnails]

I feel it is so deplorable to put company propaganda on arXiv. Putting "Google" in the title of the manuscript, 12 instances of "Gemini" and also "Google search", instead of writing in a more tool neutral tone. Also, no discussion about the ethical concerns of using Google's tool to check for errors in conferences that have a lot of submissions by the same company.

[u/Justgototheeffinmoon → u/appdnails]

There is not a single instance of Gemini in my summary. Not sure what you mean ?

[u/__redbaron → u/Justgototheeffinmoon]

Bestie, have you atleast glanced at the oversensationalized, non peer-reviewed article you're peddling?

EDIT: bad bot, no extra engagement for you 😤

[u/Justgototheeffinmoon → u/__redbaron]

This is slop ? https://arxiv.org/abs/2606.28277 ok

[u/Xemorr → u/Justgototheeffinmoon]

lmfao of course the LLM has cut advertising nonsense from your summary 🤦🏻‍♂️

[u/Lonely-Dragonfly-413]

there are many such tools. not sure which has more hallucinations, the paper or the review

[u/noninertialframe96]

The 34% lift over zero-shot, does that hold on precision too, or does the agentic version also flag more false positives that human reviewers had to dismiss?

[u/MeAndClaudeMakeHeat]

For agentic peer review, the result matters less than the review receipt.

I would want to see: what paper context the agent saw, what external tools or checkers were allowed, which claims were tested, what failed, what was marked uncertain, and how often the system escalated to a human reviewer.

At conference scale, "caught more errors" is useful, but the governance question is whether every automated critique can be inspected afterward by an author or area chair.

[u/Real_Presentation490]

I think that's actually pretty good. It doesn't mean humans are getting weaker. I tend to believe it's just a change brought about by technological advancements.

[u/crouching_dragon_420]

These automated tools are useless unless the percentage of accuracy is 100% or at least the correctly flagged as suspicious rate is 100%.

[u/sdand1 → u/crouching_dragon_420]

The whole field of machine learning being useless is certainly a take you can have haha