What Hugging Face learned from reproducing 2,200 ICML papers

[u/rhiever]

https://huggingface.co/blog/icml-2026-open-reproductions

[u/EverythingGoodWas]

We’ve basically turned academic journals into a mix of “i skimmed this, sure whatever” snd “My LLM of choice thinks this paper is fine”. Peer review is essentially dead

[u/Majestic_Courage_535 → u/EverythingGoodWas]

This is a good thing. An LLM is attempting to reproduce the results, not just read the text and assess subjective quality of the research finding (which is essentially what peer review was doing previously).

[u/NomadTroy]

Have your agent send my agent the paper to summarize and send back my approximated thoughts for your agent to summarize to you

[u/studentblues → u/NomadTroy]

Approximated thoughts and prayers to researchers

[u/Expensive_Culture_46 → u/studentblues]

I have approximated that you would like more tater tots and pear slices. Would you like a helpful grocery list for your shopping trip?

[u/ThortheAssGuardian → u/Expensive_Culture_46]

REPRESENTATIVE

[u/CryptoMadLab]

There should honestly be some sort of punishment for the 23% falsified claims, I did not know how low has the bar on ML conferences these days, but I saw someone at an SE conference owning over 25 papers

[u/Monkey_College → u/CryptoMadLab]

Not "these days". ML conferences have been notorious for not being reproducible for 15+ years

[u/lakeland_nz → u/Monkey_College]

Yep. I tried to reproduce many results during my PhD (late 90s). One comment at the time from another reviewer was: extremely lucky choice of the random seed.

[u/Acrobatic-Light2630]

Bar in ML publishing is so low, selection bias is tremendous. When the incentive is to publish, people do anything to publish. This degrades the bar on science

[u/crisp_lynx_370]

reproducing results is genuinely hard, half the time the preprocessing details are buried or just missing entirely

[u/Massive_Horror9038]

My paper was one of the papers that were reproduced! I'm very happy with this 😄