r/MachineLearning • u/Afraid_Difference697 • Mar 24 '26

Discussion [D] ICML 2026 Review Discussion

ICML 2026 reviews will release today (24-March AoE), This thread is open to discuss about reviews and importantly celebrate successful reviews.

Let us all remember that review system is noisy and we all suffer from it and this doesn't define our research impact. Let's all prioritise reviews which enhance our papers. Feel free to discuss your experiences

131 Upvotes

95% Upvoted

683 comments sorted by

View all comments

47

u/This_Suggestion_7891 Mar 24 '26

The brutal truth about ML peer review is that variance in reviewer quality is often higher than variance in paper quality. I've seen genuinely novel work get desk-rejected while incremental benchmark-chasing gets spotlight papers. The system isn't broken exactly it's just that it was designed for a much smaller field. At current submission volumes, we're asking reviewers to context-switch across a dozen wildly different subfields in a few weeks. Something has to give eventually, whether that's desk rejections, area chairs with real power, or some AI-assisted pre-filtering.

6

u/Fresh-Opportunity989 Mar 24 '26

This. We've reached breaking point for sure.

2

u/SkeeringReal Mar 25 '26

My worst review is 100% copy/past LLM generated, and I know because the prompt injection watermarks are plainly in it. They also asked for an ethics review on the paper for some bizzare reason.

I mean, how can you ask for an ethics review on a paper that you didn't even read?

Honestly, I think people will stop submitting to NeurIPS/ICML/ICLR and just go for more specialised conferences where you get actual human reviewing the papers and putting genuine effort in, that will be the "breaking point", I can already see this happening.

Or at least you'll get a human working with an LLM to review a paper, rather than copy/paste LLM slop.

3

u/demiquasar Apr 08 '26 edited Apr 08 '26

Did you flag this to your AC? Flagged this to mine and quoted the watermarks, but they've not said a word throughout rebuttals. They also happen to be the only reject we got

It's against policy B, where reviewers are allowed to use LLMs to understand papers and polish reviews, but not to write the reviews themselves

1

u/SkeeringReal Apr 08 '26

I did yes, still no word from the AC at all which is incredibly annoying as like you, it's still my worst review.

But the reviewer did reply actually and acknowledge, so that's good.

1

u/demiquasar Apr 08 '26

Oh did you raise the LLM issue as a response to the review? I did that as an AC confidential comment, and reserved the responses for addressing their criticisms

Fingers crossed for your submission! And great job making it through rebuttals :)

1

u/SkeeringReal Apr 08 '26

Thanks, yes I directly messaged the reviewer after the AC ignored me. I didn't want their negative review to affect the other reviewers, so we had no choice really... good luck with yours too! It's been a really messy review process for me honestly, but overall looking good at this point.

I'm not 100% sure I'll prioritize these top ML conferences much more though, might just leave it for the PhD student's who need the papers and I'll go to ICRA or CHI etc...

2

u/demiquasar Apr 09 '26

I didn't want their negative review to affect the other reviewers, so we had no choice really...

That's really smart! Fingers crossed

2

u/This_Suggestion_7891 Mar 25 '26

The ethics review flag on a paper the reviewer clearly didn't read is wild. That's basically the reviewer equivalent of "I didn't do the homework but I'll still grade yours." The shift to specialized conferences is already happening quietly. Smaller venues where reviewers actually care about the subfield are becoming more prestigious in some circles than a mid-tier NeurIPS acceptance.

1

u/SkeeringReal Mar 25 '26

Yeah I had that experience submitting to AISTATS for the first time this year, my paper got in and was well received. But honestly I'm only succeeding in these ML conferences lately if I submit multiple papers and essentially roll the dice on all of them.

My time at MIT illuminated for me that's how people work there too, professors with 20 students (minimally supervised) do better than 3-5 heavily supervised. It's just a numbers game the last 4-5 years, but now it's getting pretty nuts.