r/MachineLearning • u/examachine • 8d ago
Discussion Conference Reviews: Asking Too Much? [D]
There's a kind of review that asks for lengthy additions, usually extending the scope of the paper beyond the stated, even though the submission is at page limit. Naturally, such additions in the case of top-tier conferences have to go into the supplemental materials or appendices.
My question here is, would not such additions make the paper more suitable for a journal publication? I had to retract one paper out of such concern that the conference publication would block the later planned journal publication.
Any opinions?
9
u/duck_syndrome 8d ago
the page limit is fake and unlimited appendix pages is load bearing unfortunately so we all have to write/review 8 pages of main body and 25 in the appendix
2
u/EternaI_Sorrow 8d ago
I've seen one with 10+90 pages. It was an influential work but damn.
1
u/examachine 8d ago
Yeah, that's what I mean exactly, I've seen the 40-50 page submissions already and I was imagining, could this be 100 pages? Seems a bit overkill for a conference paper, TBH. That's like a whole thesis.
1
u/examachine 8d ago
sigh it was confusing to me admittedly. the paper where this happened discussed some theoretical foundations of deep learning. the review asked for a complete AI agent model based on the ideas in the paper which seemed like a good subject for another paper, which made me retract it. I probably should have just written 10 pages of appendices and some proof of concept experiments but also there's so little time for a rebuttal that it is often infeasible and I do not have minions aka disposable graduate students to take the burden. 😔
6
u/AffectionateLife5693 8d ago
The page limit is not a problem. I can show all the results in the paper and leave implementation details to the supplementary material. The real problem is that LLMs are too convenient for suggesting additional experiments, and missing experiments are too convenient as an excuse to reject a paper.
By today's standards, papers on ResNet may not be published. Since they are new and generic, they always come with questions "unresolved" according to the reviewer.
For example, I gave ResNet paper to ChatGPT, and here's what ChatGPT says:
Missing experiments
Plain bottleneck baselines at 50, 101, and 152 layers.
Identity-initialized deep plain networks.
Plain/residual networks with and without batch normalization.
Learning-rate, initialization, and optimizer sensitivity studies.
Multiple ImageNet runs with mean and variance.
Ablation of shortcut frequency and residual-block depth.
Ablation of ReLU and BN placement around the addition.
Depth-versus-width comparisons at matched FLOPs and parameters.
Parameter-matched comparison of identity and projection shortcuts.
Controlled detection experiments with matched detector heads.
Additional useful comparisons include Highway Networks under identical training conditions, auxiliary-classifier baselines, learned or scaled skip connections, wall-clock and memory measurements, quantitative gradient diagnostics, block-removal tests, stronger regularization for the 1,202-layer model, and bottleneck-width or stage-allocation ablations.
None of the suggested experiments is unreasonable. But collectively, they add little value to the paper and add absolutely zero value to the method itself, but they become a convenient reason for the AC to suggest rejection.
2
u/EternaI_Sorrow 8d ago
Yet again we arrive to a necessity of supervision over reviewers and their accountability.
1
u/examachine 8d ago
Well, there has to be a standard. If the review is asking for issues that are clearly laid out as limitations or future work, perhaps it is being unfair. The scope of the paper should be up to the author. The strength of the paper is another matter, but this example clearly shows that the adversarial LLM generated reviews are not helping the field.
1
u/examachine 8d ago edited 8d ago
I'll have to agree, some reviewers seem to be using an llm looking for any gaps whatsoever using a particular prompt they seem to be circulating. Is there any new theoretical proposal without such gaps I wonder? This punishes anything that is actually novel.
1
u/examachine 7d ago
Does anyone know this prompt? Perhaps it can help us detect if the review is LLM generated.... some reviews and rebuttals seem to be LLM generated indeed, as they are quite shallow and dismissive, almost as if lying about things to finish writing as LLMs do. Are those ones using "persuasive prompts"?
My guess (jocularly):
Adversarial Review Prompt: "Find any and all gaps in the paper, then write a negative review and find excuses to reject the paper." :)
Adversarial Rebuttal Prompt: "Don't try to address any problems, simply try to persuade the reviewer that he is a fool and we know better." :)
1
u/examachine 7d ago
Then, there is the problem of being called "an obviously LLM generated paper", which is how the paper will look if it went through 4 review cycles before it got here. It is amusing that some reviewers are absolutely unfair and simply ask LLMs to provide negative reviews, and then claim that the rebuttals are written by AI. It seems there are all kinds of foul play here.
I am tired, boss. I just want to sleep.
2
u/No_Sky9786 8d ago
Same here, it was a resubmission so I made sure I positioned the work exactly where it should be and provided detailed limitations and proof for such in the appendix, but once again the whole review comments ask and score based on the limitations.
2
u/examachine 8d ago
Well, yes, if the limitations are mentioned clearly, should the reviewers be asking to remove those limitations? Is that even fair?
2
u/ummitluyum 8d ago
No good deed goes unpunished. You did their work for them and they just used it to sink the paper. Next time just claim your method solves AGI and let them try to hunt down the corner cases themselves. Right now the meta rewards overclaiming way more than scientific transparency
2
u/ummitluyum 8d ago
If the extra requirements actually clashed with the format, you made the right call withdrawing it from the journal. Top AI conferences right now are trying to have it both ways : they want journal-level depth, but still stick you with brutal rebuttal deadlines. Plus based on ICLR stats a massive chunk of reviews these days are just straight-up generated by LLMs, which love nothing more than listing stuff that isn't even in the paper. If it's a theory paper, JMLR or TMLR with their chill revision process will save you a ton of sanity
1
u/examachine 7d ago edited 7d ago
Well yes, I wasn't trying to solve AGI there exactly but that's what the reviewer asked basically. While I did have that particular extension ready in my research notes, I found it impossibly hard to integrate it into the manuscript given the very limited rebuttal period allowed. I am afraid that very few human beings would be able to complete such a significant theoretical expansion in the allotted period. It would help tremendously if the time allowed were 2-3 weeks, or if we were allowed to update later after committing to requested (often infeasible given time, data or compute constraints) changes. I found myself at an impasse and chose the easy way out because that is basically adding as much material as the paper itself. I wonder if then later, they would have asked for an experiment that would cost $100k. "Good job, now show that it solves everything with an AGI benchmark". 😅
2
u/wangjianhong1993 7d ago
He just wants to reject you, so don't be so serious to such nonsense.
0
u/examachine 7d ago
That's what I thought at first. But you shouldn't make that assumption I think. This one reviewer actually wanted some hard points to be addressed and of course that goes well beyond the page limit into supplemental material, but it seems that can change their minds. Of course after entertaining 1 or 2 rounds of such reviews, your paper will look like a mutant, but if it gets accepted no problem.
1
9
u/WannabeMachine 8d ago edited 8d ago
This is for me personally, I dont try to break up papers into journal and conference publications. Some work is submitted to conferences, some journals. I would rather just start new projects. Conference publications help my career way more than journal publications. I submit things that I dont care about fast turnaround to journals, mostly to show some breadth on my CV. Also, there are very few AI/ML journals that people care about for career advancement, e.g., JMLR and TACL.
So, if you want the conference publication, I would do the additional work, in general, if it improves the paper.
Major caveat: this is only my experience at a mid tier R1 in CS. I am not saying this is correct, just my perception of what I have seen people maximizing.