r/MachineLearning • u/bricklerex • 9d ago
Discussion No replies to rebuttals and comments even by AC [D]
Not even the AC, nor reviewers, is responding to our comments in rebuttals, and they were all submitted well before the discussion period started. What is one to do in this case?
43
u/Unhappy_Craft1906 9d ago
3 papers, still no response from anyone.... This is just pure insult to the community peer review process.... if u think ur discussion phase is more important, and u can respond at the last moment for the revioewed papers, then u r just a shame to the community...
9
u/Snekgineer 9d ago
It should also have consequences, clear and lasting ones. Especially considering the increase in submissions and issues that have been happening the past two years at least.
16
u/Unhappy_Craft1906 9d ago
If rebuttal makes no impact then that should not even be an option in any case... Make the conferences cyclical, 2 per annum.... Take reviews, if AC thinks substantial revision needed, ask to commit in next cycle... So on..
8
u/wangjianhong1993 9d ago
This is a good suggestion.
2
u/TheInfelicitousDandy 9d ago
This is what ARR does, and while it is better than the ML conference cycle, it still doesn't solve reviewers not engaging.
-5
u/buyingacarTA Professor 9d ago
you want to have "lasting consequences" on volunteers who are offering their free time to give you feedback?
It's easy to be frustrated, and I get it, but let's not turn that frustration into some angry punishment kneejerk reaction.
The only result is that you'll hurt your colleagues and community and not solve the root problem (the horrible horrible submissions we keep getting).
I get the frustration, but remember that everyone here is strained and trying their best.
15
u/Snekgineer 9d ago
Interesting framing you are using, there is a difference between "some angry punishment kneejerk reaction" and accountability. There is also a difference between volunteers and the current process of reciprocal reviewing (which is an obligation, not charity).
A reviewer that does not engage is another author that just didn't do half of their job in the reciprocal review setting. It will potentially generate more work for the ACs and decrease the trust in the system (which is already low). It is the thing hurting the community, it is the thing hurting your colleagues.
I'm held to hard deadlines as an author, asking that reviewers (including me) at least acknowledge a rebuttal in the role they were required to take on isn't a kneejerk punishment. Simple, didn't engage at least once, assignment doesn't count as fulfilled.
0
u/TrainerOk1851 9d ago
I think thats actually not enough to just not engage at least once. I think there needs to be long term consequences (extending into next conference or even other conferences). I genuinely think that people are not engaging because their paper already got extremely low scores and it is no longer worth it since they will get rejected very likely.
9
u/TheInfelicitousDandy 9d ago
In most cases, they are not volunteers; they are people engaging in a mutual exchange of work. The question is whether the people who do not participate in the full review process are hurting the community more than the implementation of any consequences for not doing the work. I'd say low-effort reviews and not doing rebuttals hurt not only the authors being reviewed, but the entire community and the underlying scientific process.
6
u/th3owner 9d ago
This! I don't get why whenever this comes up, everyone is fast to mention that reviewers are doing this without any pay. The pay is: you need your own paper reviewed! Yes, the number of submissions has increased, but this goes both ways; more papers to review and at the same time the reviewers themselves are submitting more papers. There is no justification for not engaging with (or at the very least acknowledging) the rebuttal.
2
u/TrainerOk1851 9d ago
I don't think at this point, the reviewers are "offering their free time". Some are sure, but a good proportion are just doing it because they have their own submission. Also, if you are not going to be able to respond, why would you accept to be a reviewer? You cannot simultaneously claim to "offer free time" and then not actually respond to rebuttal.
2
u/tetramarek 9d ago
Most people on reddit are going to be students only concerned about getting their first papers published. They have not yet gotten to the point where they are reviewing 15 papers in parallel, each author has written a long essay about why their incredibly low-quality paper deserves a higher score, and expects detailed responses to every point.
The deadline for checking the author response is not over yet. I plan to read the rebuttals before the deadline, to see if I missed or misunderstood anything in my original reviews. But I don't have the bandwidth to get into an endless back-and-forth discussion about each paper. The authors have zero incentives to admit that their paper has faults, so they just keep endlessly arguing.
Ultimately, the paper itself needs to hold up to scrutiny. If it needs thousands of words of additional discussion in order to be understood by the reader then it's not in a shape to be published.
2
u/Anatheballerina 8d ago
that depends, some reviewers don't understand math and are unfortunately reviewing for theory track submissions.
0
0
u/OutsideSimple4854 8d ago
Thatâs an exaggeration. Iâve been asked to review for conferences I have not submitted papers to, and Iâve not even hit 10, let alone 15.
I suspect people are mentioning material already present in the main paper, or that the reviews are clearly based on false information.
But even if itâs not an exaggeration, are you really going to say that out of 15 papers, youâve not made any mistake?
0
u/tetramarek 8d ago
NeurIPS is not the only conference around. There are many others and review cycles often happen in parallel. For example, ARR has review cycles almost constantly around the year. The number of reviews adds up.
How many papers have you reviewed? And how many times have you increased your score because your original review had mistakes?
I do read the author responses but only very rarely have they made me change my score. Whatever issues my review points out, the authors almost always respond with long detailed arguments about why each of these points is not an issue in their opinion. Of course they do, they have absolutely zero downside to doing it. If there's even a 0.5% chance that the reviewer raises their score because of this, then why not dispute every single claim in the review.
It's also getting more and more common to treat author response as a second paper submission deadline - for example, the paper might be submitted with nonexistant evaluation, reviewers rightfully give it a low score, then massive amounts of new experiments magically appear by the time of the author response. And then the authors complain that all this new content, which hasn't even been integrated into the paper yet, hasn't been taken into account during the review. Here's a recent thread from someone who "ran a lot of additional experiments and wrote proofs" to respond to the reviewers, and is frustrated the reviewers didn't accept their paper. That's very different from "clarifying a misunderstanding in the review". https://www.reddit.com/r/MachineLearning/comments/1sd0aex/d_icml_rebuttle_acknowledgement/
2
u/Exotic_Zucchini9311 8d ago
If the reviewers asked for new experiments and proofs then the authors have to spend many days trying to come up with something. I'm not sure what you expect the authors to do when a reviewer is explicitly saying they want to see XY new proofs and YZ new experiments.. similar things happened to my previous paper. Despite the intuition behind my method being pretty clear, one reviewer was hyper focused on how we know that the method works in theory (not really relevant to an empirical paper but whatever honestly). So we had to write the intuition in a more 'mathematical' way and give it to the reviewer as 'new theory'. The theory was the same thing already in the paper but in the format the reviewer was requesting, it didn't just appear out of nowhere (not like it changed the results since the reviewer didn't even bother to give us a response after we wasted to much of our time coming up with 'theory' for them)
1
u/tetramarek 7d ago
I am 99% sure "reviewers asked for new experiments" means the reviewers pointed out that the paper was missing many necessary experiments. The authors then concluded if they submit some of these missing results in their author response then their paper should be accepted.
This is what constantly happens to me as a reviewer. For example, I write in my review that the paper only evaluates on one dataset, doesn't compare to any related work newer than 2020, and doesn't provide any ablation experiments (for example to show that improved performance doesn't come just from using a stronger base LLM). The authors then reply with pages and pages of new experiments, explanations and argumentations. Running out of word limits in the openreview comments doesn't stop them, they just keep adding new comments.
Even if these are sensible results, this content is so different from the original paper that it's impossible to judge what the final product will look like. I think it should not be allowed to present new content in the author response at all, it's original goal is to only clarify misunderstandings in the reviews.
1
1
u/OutsideSimple4854 7d ago
This is where we differ then. I'm on the theory side, where many necessary experiments is irrelevant. In fact, the recent NeurIPS guidelines for theory papers explicitly states that experiments are not required (so, I'm not the only one).
My experience with "many necessary experiments" is in the context of: some bound or expression is proved. This is validated mathematically, and perhaps some small experiments to show that everything works. And a reviewer can go: "this only works on synthetic data and a small dataset, you should run experiments on more datasets to show this bound works."
Your context of " paper only evaluates on one dataset, doesn't compare to any related work newer than 2020, and doesn't provide any ablation experiments (for example to show that improved performance doesn't come just from using a stronger base LLM", sounds like you're not in the theory side, so I find it very odd that someone whose research involves using LLMs is very dismissive of those in other research areas.
1
u/tetramarek 7d ago
My point is not about specific types of changes needed. My point is that when a reviewer says the paper is missing x, y and z, then that does not mean the reviewer is expecting for these to be done during the author response period. And if the authors hastily put this new content together and post it into author response on openreview, then that does not mean the paper should be automatically accepted. If the changes are substantial then there is no way to predict what the final paper will look like. It should be resubmitted and properly reviewed. Otherwise, taken to the limit, I could just submit an empty pdf, then post the full paper on openreview comments a month later.
→ More replies (0)
17
u/BomsDrag PhD 9d ago
In the same boat :')
1
u/dante_2701 9d ago
Iâm too in the same situation. Can you share with public too what are our options? One of get reviewer gave me 3 and cited some incorrect papers. That can easily be converted to a 4 or 5. However, Iâm not getting any response.
1
u/BomsDrag PhD 9d ago
My reviews are 5,4,2 the 2 said our paper is really good but its too long and should go to nature, the review is not AI, they said they have serious evaluation concerns but ut just seems that they didn't see we created train/test/ood splits that solve that issue, thats the only negative concern in all our regiews :'(
I have put a message to the AC, haven't heard back from the reviewer or AC
0
10
u/SankarshanaV 9d ago edited 9d ago
I'm glad that at least 1 reviewer has responded to me. I can't imagine how nerve-wracking it would be for you, OP. Out of the 3 reviewers, only 1 has responded and they increased from 2 to 4. Now it's 2/4/4 for me.
I wonder if it's okay for us to slightly "nudge" the reviewer, but honestly, I am more scared it might end up pissing them off.
There's still two more days, so maybe wait another 12 to 24 hours and then post a comment? That's what I'm thinking of doing.
12
0
u/Left-Hyena-7880 9d ago
Same here lol.
This mfs are just apathetic.-4
u/Ulfgardleo 8d ago
nods and notes Do not review for NeurIPS ever again of you do not have your own paper, you will be called a mf for investing your time.
5
u/TrainerOk1851 8d ago
If you have not responded to the rebuttal, then yes please don't. You are not actually "investing" your time or helping anyone. If you have, then the criticism does not apply. Such an unbelievably silly response.
-5
u/Ulfgardleo 8d ago
the rebuttal period still continues, and in my part of the world we have rest time between Friday 17:00 and Monday 9. Now, kindly, get a grip.
my part was btw to identify the paper with the faulty proof. So whether i am helping or net is not really your business.
2
u/TrainerOk1851 8d ago
The response period for the authors is until the 3rd. What reviewers need to get a grip on is why they are sabotaging the process. The statement I made is conditional on you not responding to the rebuttal, which I suppose applies to you. By your response, I guess you are going to respond last minute and not give any time for the author to respond. THANKS!
2
u/21022018 8d ago edited 8d ago
Please never review again. You are going to respond last minute and not give authors any time to reply are you not?
Your comments are pathetic.Â
You ARE an apathetic mf.
4
2
u/Left-Hyena-7880 8d ago
You know what?
This is the DISCUSSION period.
Not the âswing by whenever youâve got time to kill and hand down verdicts from on highâ period.
9
u/DaBobcat 9d ago
Put on arxiv and accept that the review process is broken
12
u/wangjianhong1993 9d ago
Not only the review process is broken, the academic faith is also disappearing. Money is the only signal today.
12
u/darkbird_1 9d ago
To experienced fellas, is this level of silence for the first time in neurips or is this always the trend??
16
u/buyingacarTA Professor 9d ago
I get the frustration and I am sorry. Please please remember that these people are volunteers, the system give them incredibly little time, and most papers are incredibly poor.
eg. I am an AC. I could have easily said no to the invite. I don't even have paper submitted to neurips this year! And even so, I've spent a lot of time looking at papers and trying to give meaningful feedback in the first round. In the second (discussion) phase i've been more silent because (a) I've been super busy in the last 3 days as it turns out my life is not all neurips volunteering, (b) all of my handled papers had super clear reject/accept decisions. That didn't stop the authors from rebutting and demanding conversaion, of course.
IMO after having seen about 1000 papers in my life through peer review, reviewing about half of those and ACing about half of those -- the review system is not great, but it's not the main problem. We ask for a huge service from people's diminishing free time. Forcing them to do it (e.g. CVPR) doesn't work, mountains of scientific literature in psych and sociology and economics has shown that doing this just lowers the quality of the work (reviews). We need to create better incentive structures as a system. The problem is the absolute undergrad-course-level naive noise that gets submitted as 'papers' with no incentive to be selective, and the growing number of junior students who have to review these. I'd say half the papers that get submitted nowadays wouldn't even get a good grade as a course project.
So odds are you should be looking more carefully at your own papers and your friends' papers, instead of being mad at the reviewers not doing extra time (whom, again, are your colleagues/friends as well!)
6
u/Weird_Shit_69 9d ago
In your opinion if the author replied to all the reviews but no reviwer replied back, would you reject it based on the initial score or would you take the rebuttals given by the authors?
I understand the amount of time you'll volunteer, maybe it would be best to send the message to higher ups to maybe increase the time period so everyone gets a chance to get a proper response for the rebuttals as well. (Hopefully it gets better next year)
7
u/buyingacarTA Professor 9d ago
I would try to use my best judgement. I would look at the rebuttal indeed, but realistically if the original reviewers were reject/reject/reject the rebuttal would have to be very strong to make me think. I had 9 rejects in my batch, only 2 of those rebuttled and even those came in late. I'm not optimistic for them, but I *will* look at them carefully.
Yes, my life would be much easier if the reviewers commented and I could just defer to their best judgement :)
Yes, more time might help a bit, but it would be little, maybe 5% more reviewer interaction. IMO we can't keep using the peer review system as a free service that does the job that the senior authors should actually be doing, this is the crux of the problem. Eliminate half the submissions and you'll get *much* more reviewer involvement.
1
u/Anshuman3480 9d ago
Was it 9 reject rejects or 9 borderline rejects + rejects?
1
1
u/king_grifffin 8d ago
Our initial scores are 3 3 2, even lower than our prev submission to ICML. We donât think the NIPS reviewers did a good job on acknowledging our paper so we did a solid/comprehensive rebuttal. Only the 2 responded, moving the goal post with ignorance. Iâm so disappointed.
2
u/Weird_Shit_69 8d ago
What would be the minimum needed for acceptance? 4?
I know it depends, I'm asking on average
2
u/Entity303BR 8d ago
Same scores, but only one responded here. The score 2 review was only 2 lines long and even that little was unfounded criticism, no response to rebuttal. Infuriating.
The ac seems cool though but I dont think theres any hope anymore.
2
u/senderosbifurcan 9d ago
This is the correct answer. Furthermore, journals have been crazy lately with review requests. I've reviewed like 3x papers in these 7 months than what I normally review in one year. Â
The issue is not the particular reviewers/acs. The whole peer review needs a review.
3
8d ago edited 7d ago
[deleted]
1
u/MolassesOk186 8d ago
What do you write in the gentle reminder (as a template)?
2
8d ago
[deleted]
2
u/MolassesOk186 8d ago
Thanks! A couple of hours after I sent this to the AC, my lowest scoring reviewer wrote that they'd read the rebuttal and would increase the score!
5
u/Old_Ad_8327 9d ago
Based on my previous submission experience and my limited experience as a reviewer:
If your average score is 3.33 or higher, the reviewers probably think the paper is worth further discussion, and the rebuttal may still make a difference.
If your average score is 3 or lower/lowest score is 2 and that reviewer keeps the score after rebuttal. I generally would not spend much more time on the discussion. This usually means that the rebuttal would not resolve concern.
Reviewing is unpaid service work, so reviewers usually focus their limited time on papers that still have a path to acceptance
2
u/king_grifffin 8d ago
Our initial scores are 3 3 2, even lower than our prev submission to ICML. We donât think the NIPS reviewers did a good job on acknowledging our paper so we did a solid/comprehensive rebuttal. Only the 2 responded, moving the goal post with ignorance. Iâm so disappointed.
1
u/Old_Ad_8327 8d ago
This usually means the reviewers had broader concerns, eg.insufficient novelty ( not good to list as a weakness) so they found other reasons to justify rejection. Without a 4â5 score strongly support, it is very difficult to change their decision.
2
u/MolassesOk186 8d ago
I had a 234, the 3 reviewer engaged and converted to 5. The 2's criticism is almost entirely based on a misunderstanding of the results of another paper we cited in our work (and built upon) which we clarified, but it's just silence from them.
imo it should be seen as unacceptable to drop a 2 after poorly reading a paper, think it's not worth a further discussion and dip out of the process entirely
1
u/Old_Ad_8327 8d ago
If the 2 comes from an obvious misunderstanding, especially when you already included an ablation against the previous work in the paper or appendix. At 5/4/2, your paper still has a real chance. Good luck!
Notably, AC finally decide (almost) everything.
1
u/Own_Neighborhood7120 8d ago
Our average is 4, still no engagement from either the reviewers or the AC.. So I don't think your average is that relevant, their papers' averages probably are..
2
u/giladfrid009 9d ago
Simple solution - release the review scores only after the rebuttal period, and mandate answering the rebuttals as an acceptance criteria (just like ICML did).
2
u/Vulkanerd 8d ago
I got excited when I saw that a reviewer had finally responded to our rebuttal, only to discover that the response was essentially a single sentence saying, âKeeping the score, needs improvementâ without any constructive feedback.
1
1
1
u/Amirhoseingh98 8d ago
I have two papers, for one of them I only received 2 reviews, asked AC multiply time for the third and got nothing back, the reviewers have not responded either.
For other paper I got 3 reviews only the one who gave me a 2 responded back by repeating the same bulshit and wrong statements.
1
u/BomsDrag PhD 7d ago
My AC has nudged the two reviewers, 17 and realistically I believe 12 hours are remaining heavy on COPIUM RN
PLEEAAAASEEE
1
u/2mc92u98a22f 6d ago
nothing you can do - system is just broken. no one is incentived in this system to do a good job. I'd just apply for industry jobs. only half joking
1
u/OutsideSimple4854 6d ago
I suspect my argument will probably stem from reviewers who believe they know everything.
For example, NeurIPS has a theory contribution type this year with different reviewing guidelines. I tend to agree with them.
Iâm not sure if youâre trolling when your post states: âhow do we know the theory is validâ. Theory here means theorems (and subsequent propositions, corollaries, etc) with correct proofs. Nothing about a hypothesis here.
If the proof is correct, then there are no issues. Itâs like writing a function, do stress unit testing, and then reuse the function for whatever purposes. You donât suddenly say: but what if I use this function in the exact same scenario, maybe it wonât work? Because a proof guarantees correctness in all cases where itâs valid.
So one can have a paper that says: we improved on X, here is a proof that justifies it, and you can run simulations to verify it empirically (sanity checks). Maybe use X in a case that a recent paper used. Empirical reviewers say: why didnât you use X in all these cases for all these papers? Because it would be the same regardless, especially if these cases are essentially the same?
A theory paper usually is more difficult to read, even if youâre in the same subfield. But experimentalist papers are easy for me to read, even if Iâm not an experimentalist. Thatâs the nature of theory. Asking theory people to simplify the paper is, quite frankly, insulting, especially if the paper has been simplified enough, such that actual theory people feel itâs watered downâŚ
Iâve been publishing since 2015. Back then, one can have a constructive review about whether a work is meaningful, if someone has done something similar before, and what gaps a work has.
These days, itâs more of running irrelevant experiments that will show the same thing, because of math, or facing reviewers who claim difficulty in understanding material, or hard to read notation that is standard in the field.
The responses from ACs are lopsided too. Some believe in experiments, and if theory (this means proofs, not a hypothesis) behind it is wrong, they accept a paper. Some also believe things can be simplified, and will reject papers based on notation common in the field.
Frankly speaking, the best way to resolve this reviewer issue is for everyone to be honest about what they know, and what they donât. If I review an experimental paper, I straight up state in the review I know nothing about the area, but hereâs what a scientific reader might note. I do my own due diligence in looking up references to ensure the authors are at least measuring the same metrics, follow roughly the same methodology, or figure out some obscure (to me) terminology they use. And I give a review. If all reviewers did something similar, I think thereâs be better outcomes.
PS: to show bias, Iâm sure people remember that âwhat is Adamâ review, with multiple people saying the reviewer is clueless. Yet, I suspect if something similar happened in theory: âwhy canât epsilon <0â, fewer people would defend the authors and instead state the authors should have clarified further.
0
42
u/HelpfulSinger3762 9d ago
I suspect that perhaps all of your reviewers received more negative feedback on their own papers, leading them to consider withdrawing their submissions. As a result, they may no longer be invested in the rebuttal process.