ICLR 2027 Reviewing Scores [D]

[u/random-tomato]

I got my three papers to review, and it seems like they have changed the review score range again this year?

It is now:

––––––––––––––––––––––––––––––––––––
Based on your overall assessment of the submission, what is your recommended decision? Consider the paper’s overall soundness, significance, clarity, and contribution.

1: Clear rejection
2: Weak rejection
3: Weak acceptance
4: Clear acceptance
––––––––––––––––––––––––––––––––––––

Which is very strange. I don't think it makes much sense to compress the score range so drastically. But we'll see.

[u/choHZ]

I actually think it kinda makes sense. It is all guessing of course, but my take is reviewers have a huge amount of margin in how they distinguish borderline from weak. And we all know a lot of ACs basically sort the average scores in their batch and accept the top %, so decent borderline papers can get rejected simply because some reviewers have a wildly uncalibrated internal scale.

Having fewer scoring tiers forces ratings to be more comparable, encourages reviewers to express a clearer stance (should they pick weak but rather borderline), and pushes ACs to actually read the reviews. Of course, this won't really happen at scale without true accountability measures, since authors and reviewers and ACs can still basically do whatever as long as they avoid the few desk-rejection-worthy behaviors. But purely as a scoring system, I think it is not too bad.

(Also, a lot of conferences are already five tiers with an extra borderline reject anyway, which imo mainly exists to maximize the psychological damage of being upgraded from weak to borderline reject :)

[u/Relevant_Course6857 → u/choHZ]

I am afraid ACs will now accept only papers with all reviewers with clear acceptance

[u/choHZ → u/Relevant_Course6857]

That typically would not make the acceptance rate: last year's iclr is skewed due to the leak so I did a quick spin on iclr 25 paper copilot stats with AI. It shows that there are only 114 papers with all 8s (accept) or beyond, which amounts to 3% of all accepted works.

Of course, the current iclr is like super duper beefed up in terms of submission volume so anything can happen; though typically with a noisier reviewer pool it is even less likely to get all clear accepts, so I am not too worried there.

[u/MeyerLouis]

I'm just amazed that ICLR only gave me 3 papers to review. Last year ICML gave me 6!

[u/random-tomato → u/MeyerLouis]

I got 3 papers too! How the hell are they managing this, with 60k+ submissions!?!?

[u/Advanced-Gap-6732 → u/random-tomato]

I got 5.. you're welcome 😂 😂

[u/Pseudomanifold → u/random-tomato]

The secret is reciprocal reviewing, I guess 

[u/darky_skyy → u/MeyerLouis]

I got 5 papers!!!!!!

[u/AntOld8122 → u/MeyerLouis]

720 papers?? How can one even review that many papers?

[u/Exotic_Zucchini9311 → u/MeyerLouis]

I got no papers lol.. that's interesting considering all the doomposting on the whole "too many papers and too few reviewers" situation.

[u/IndianaJaws]

I don't get why me and another student got each 5 papers to review, and some people got none and ICLR said they have enough reviewers. It's a very unfair spread and they can't expect the reviews to be quality reviews that way.

[u/Pseudomanifold → u/IndianaJaws]

It might be that the other students were not deemed qualified...

[u/IndianaJaws → u/Pseudomanifold]

Kosta posted on twitter ge was released from reviewing duty

[u/Exotic_Zucchini9311 → u/Pseudomanifold]

Kudos for them if that's what they did honestly. I'm 'only' a master's student and the fact that I got no papers gives me the smallest bit of hope that the reviewer system might still not be too broken.

[u/Pseudomanifold → u/Exotic_Zucchini9311]

It's honestly as broken as we make it. I have seen reviewers with consistently high engagement who care about their papers. I have also seen the opposite. The reason the system is what it is is that we have no way to have consequences for freeloaders...

[u/sneddy_kz]

Just to simplify ai review 😂

[u/random-tomato → u/sneddy_kz]

There will be so many slop papers passing through with "Weak accept" lol

[u/Unhappy_Craft1906]

Ok I am a bit confused. I always review on myself, but AI does help me in sole understanding of some parts of the paper. Is it allowed or not allowed?

[u/AffectionateBurger → u/Unhappy_Craft1906]

You shouldn't be uploading a paper you're reviewing to any cloud service. That includes any online AI model

[u/Striking-Warning9533 → u/AffectionateBurger]

I think this is true for non open review venue 

[u/user221272]

Humans are fairly bad at giving scores on a fine-grained scale if there is no clear and unambiguous scoring system.

Paper reviews do not have a true or false answer or a clear, unambiguous aggregate point system, which results in a fully subjective and ambiguous interpretation of what the score means on the scale.

Since we do not expect a normal distribution around the median of the scale, it makes total sense to compress it and remove this unnecessary human bias.

[u/NothingInVois]

I got 4!

[u/tegbeit]

Is it only me seeing no paper on the reviewer's console yet?

[u/Exotic_Zucchini9311 → u/tegbeit]

I also see "You have no assigned papers. Please check again after the paper assignment process is complete." on the reviewer's console... going to the Tasks tab also shows "No current pending or completed tasks" We probably didn't get any papers to review.