[D] ICML 2023 Reviewer-Author Discussion

Submitted by zy415 t3_11ylumz in MachineLearning

Thought it might make sense to create a discussion thread specifically for reviewer-author discussion. It seems that many authors (at least those around me) did not receive any further response from the reviewers. How's everyone's discussion period going?

PS: I understand that ICML reviewers are busy with their own research/work and are managing many submissions at the same time. I just wish they could be more active in the discussion period, because all these submissions are the results of many months of hard work. Personally I am also a reviewer at ICML and have responded to most authors.

24

Comments

You must log in or register to comment.

Mxbonn t1_jd8e2ab wrote

I had all borderline scores with 3/4 reviewers asking at least 4 questions and said they are willing to reconsider rating based on answers.

No one has replied or changed rating so far. Discussion period is not over yet but it's frustrating not knowing whether they even read my rebuttal.

EDIT:
Received no responses in the end.

14

midasp t1_jd8lyq4 wrote

Give it some time. When I was a reviewer, I'd usually sit on responses for at least a day or two to gestate on them before answering.

3

Mxbonn t1_jdbr3gz wrote

Is there anything I can do if they don't reply by the saturday? Should I message the AC?

1

ConsiderationDry7153 t1_jdh569j wrote

I think this is not a good idea to message the AC.

What do you think they can do? They cannot force the reviewers to ask questions. Then they will just lose time with your request while being very busy by the multiple submissions they are managing. It would only make you look impatient or unprofessional which will not help.

1

Mxbonn t1_jdhaoxb wrote

If they ask 5+ questions and don't reply or change review or confidence score, doesn't that imply that they simply haven't read it yet?

2

ConsiderationDry7153 t1_jdhd04u wrote

Maybe, maybe not, we cannot say for sure. Even if they haven't, I think that there is nothing you can do that could really help your case. But this is only my personal opinion.

I am in a similar position: no response from any of the three reviewers. But I think that it does not have to be a bad news: maybe they do not need more details to take their decisions.

You have to remember that maybe they are not that interested by you research so they will just ask enough question to have a global view and not more.

1

Important_Tonight433 t1_jdqamia wrote

>[D]
>
>ICML
>
>2023 Paper Reviews

I spent multiple rounds answering some questions that can be searched from Google, explaining why this is not a question, and many other papers evaluate benchmarks in the same way

Finally the result ends up with "don't argue with me".

I wish they could at least pay me some tuition fee for educating them.

3

nokpil t1_jdcxrhc wrote

Surprisingly, one of my reviewers actually responded to the answer and raised their score from 5 to 6. Hope this will help someone.

6

ILOVETOCONBANDITS t1_jdoj5e7 wrote

I similarly had a reviewer raise their score from a 3 to a 6!

2

OkWrongdoer4091 t1_jdy19mf wrote

What about the other two reviewers? Any word from them?

1

ILOVETOCONBANDITS t1_je3m9ts wrote

I had a total of 4 revieiwers. Went from (6, 6, 4, 3) to (6, 6, 5, 6). The first two reviewers didn't respond at all.

2

OkWrongdoer4091 t1_je3vscq wrote

My only hope that my metareviewers will be diligent enough to reach out to the reviewers that didn't respond and ask if they'd changed their mind after reading the rebuttals. Not responding to the rebuttals destroys the purpose of rebuttal (and the review itself). If you volunteer to serve the community as a reviewer, why not commit to doing a good job of it? I wish I could contact the ACs, but the buttons are no longer available on OpenReview.

1

ILOVETOCONBANDITS t1_je3yo38 wrote

Yeah i feel your pain. it's a bit weird that even though a 5 this year is technically borderline accept, its basically a reject given the score distributions. And not getting a good rebuttal hurts.

1

OkWrongdoer4091 t1_je3w01r wrote

Was it accepted in the end?

1

ILOVETOCONBANDITS t1_je3w70l wrote

Er we'll see soon? I guess decisions come out in april

1

passerby251 t1_jddhszk wrote

Congrats! Do u mean that you didn’t receive any email notification but found them respond and change the scores?

1

nokpil t1_jdfsur1 wrote

I got an auto-generated e-mail from openreview since the reviewer made an official comment below my answer, with the title of "[ICML 2023] Reviewer XXXX commented on your submission. Paper Number ...",

1

OkWrongdoer4091 t1_jdy1392 wrote

Same thing here. Only one of the reviewers answered our rebuttal (which was excruciatingly detailed). They seemed quite happy with it and even raised our paper's score from 3 to 6. The other two reviewers did not respond. Wondering if they even saw our rebuttal.

1

paulgavrikov t1_jdguvmk wrote

Same here, 3 out of 4 reviewers didn’t respond. And with all due respect, there is no excuse for that. If you agree to serve as reviewer, you also agree to do it faithfully. Unfortunately, the last part is often missed.

5

sleeplessinseattle00 t1_jdeir9w wrote

Haven't received any response yet. I've answered all of the raised clarification and extra experimentation, but at this point, I really can't do much if they're just gonna ghost me.

PS: I am also a reviewer and I've finished responding to all of the papers assigned in my pool.

4

zhaoyl18 t1_jdf9tvo wrote

same. None reaction from any reviewer. In this case the 'discussion period' is vague and pretty funny

2

zy415 OP t1_jdfhjy4 wrote

Agreed. It’s basically a joke

3

sleeplessinseattle00 t1_jdfmdtt wrote

Finally got a response saying that you’re rebuttal adequately addressed my concerns. (Yet no change in the rating) 🥲 why do people do this?

3

zy415 OP t1_jdfzxh3 wrote

Sigh, that must be frustrating. What score did the reviewer give?

2

sleeplessinseattle00 t1_jdg0u7p wrote

It was a 6, and we were really hopeful that it can bump to 7

1

paulgavrikov t1_jdgz9be wrote

Adress this! Ask them why they did not increase their score if all concerns were addressed.

2

sleeplessinseattle00 t1_jdi6hot wrote

We’ve written a gentle note, but no reply to that. Literally ran 50 new experiments as suggested

1

zy415 OP t1_jdejz0z wrote

Same situation as yours. Good luck to us

1

OkWrongdoer4091 t1_jdy1kxa wrote

Just out of curiosity, are the deadlines strict for review/rebuttal answer submission? My reviews were release about 24 h past the deadline, and the only response to my rebuttals appeared ~6 h past the end of the reviewer-author discussion period.

1

ILOVETOCONBANDITS t1_jdktauv wrote

If you don’t get any responses to the rebuttal, does that mean the scores will remain the same? I had a borderline paper (6-6-5-3 with conf 4-3-3-3) and the rebuttal questions were all easy to address (theory questions with well known answers) and I ran the experiments they wanted.

Rant: they should at least have a button to acknowledge they don’t need to send a message. For some of us applying for faculty or applying for PhD, admissions to these conferences make huge differences in our life. Such malaise is super stressful and really sucks for some of those who would have benefited greatly from an acceptance.

4

[deleted] t1_jdt9g2f wrote

A reviewer suddenly lowered the score and wrote a concern just a few hours before the end of the discussion period. Since the reviewer revised the original official review, I didn’t get notified.

The reason for lowering the score is that we did not apply to other algorithms and other datasets. If so, I wonder why the reviewer didn’t talk about it earlier and give us a chance to respond during the rebuttal or discussion period.

3

Anonymous_Penguin1 t1_jdty4ed wrote

It seems that you can still make an official comment now on OpenReview :) But not pretty sure if that would violate some rule.

2

OkWrongdoer4091 t1_je2qdt5 wrote

Four days after the end of the reviewer-author discussion period, I'm still wondering how many people (here at least) have received no reply to their rebuttals. Asking this because I hoped that there will be late replies (i.e. a couple of days past the end of the discussion period). But I only got one (6 hours past the end).

3

Firm-Act-3860 t1_je581li wrote

One of my reviewers increased their score by 1 just today and updated their review by editing it. So don't lose hope yet :)

2

Astaligorn t1_jd93sbx wrote

I had a difficult discussion on an NLP paper I submitted with a reviewer whose sole argument for rejecting the paper is, "you did not try your method in computer vision". (Even though it beats all baselines and SOTA in NLP, on a broader NLP benchmark than all previous works).

I hope the AC will step in and will not take this review into account :(

2

StellaAthena t1_jd9emj8 wrote

“We are glad that you view our work as impactful enough to warrant extension to other domains”

6

Astaligorn t1_jd9evse wrote

That is pretty much my rebuttal, along with previous work in CV suggesting that it might not work as well because one the main hypothesis does not seem to hold in image.

​

But they decreased their grade because we did not provide experiment in CV :(

2

Educational-Walk8098 t1_jdf8ctj wrote

We obtain 7/7/5 with confidence score 5/4/4. Our rebuttal addresses the questions and comments of each reviewer but we could not make it on time for an additional experiment that one of the reviewer may find interesting. Our rebuttal was well-written but there are some points we wish that we could express in another way. I'm now extremely worried if it could negatively affect the final recommendations of the reviewers. None of the reviewers respond to my rebuttal yet...

2

underPanther t1_jdgli5w wrote

The 7s would not give these scores already unless they were prepared to argue for the acceptance of your paper in its current state.

Extra experiments are always nice, but I would be proud of yourself for the hard work that you have done already instead of the one experiment that you can't do.

2

Educational-Walk8098 t1_jdh0ah0 wrote

Yeah you're probably right. Thanks a lot for your kind advice! My reviewers' comments are insightful and very helpful to improve our manuscript so I just want to respond to their feedback in the best way that I can.

1

zy415 OP t1_jdhk9t8 wrote

Is it just ICML that reviewers tend to ghost? My experiences (as both reviewers and authors) with NeurIPS and ICLR are that reviewers tend to participate in discussion with authors

2

underPanther t1_jdho40y wrote

I wonder if it's because the author-reviewer discussion period wasn't so clearly communicated at ICML this year compared to those conferences.

3

ILOVETOCONBANDITS t1_jdhsca0 wrote

Do you guys think it would be worth to make an official comment reminding the reviewers to respond or asking the AC to do something?

2

StellaAthena t1_jdi094w wrote

I just posted in response to each reviewer:

> Thank you for taking the time to review our work. We have carefully considered your comments and have provided a thorough rebuttal addressing your concerns. If you feel that your comments have been adequately addressed, we would greatly appreciate it if you could update your score to reflect that. We are also more than happy to continue this conversation over the next few days until the March 26th deadline.

I submitted several papers, all of which got borderline scores (average between 4.3 and 5.3), though one got 7 / 7 / 2 (yikes!). I had been hopeful that a strong rebuttal could judge one of them over the line, but the longer it goes without any response or updates the more discouraged I get.

4

passerby251 t1_jdiao3b wrote

I think reminding could work. I finally receive my first response only after posting reminding comments to each reviewer. Hope the other responses would come in soon. Good luck!

3

zy415 OP t1_jdispob wrote

Do you think sending multiple reminders could urge them to respond (or would it make it sound like I am too desperate/demanding)? I sent one yesterday, and am thinking if I should send another tonight.

1

passerby251 t1_jdiusxb wrote

I am not quite sure about it. Sending multiple times seems a bit demanding. But I think it would be fine to send another one when it is closer to the deadlines.

1

OkWrongdoer4091 t1_jdy2tmx wrote

Two days from the end of the discussion period, have all the reviewers answered?

1

StellaAthena t1_jdydjg4 wrote

I have four papers. Two have no comments, one has all three reviewers say “thanks but I’ll keep my score” with no further elaboration. The 7/7/2 paper had the 2 and one of the 7s argue and the third reviewer remained silent. All tolled, 5/12 responded.

2

OkWrongdoer4091 t1_jdytmme wrote

Let's see what happens next. Given that the reviews were released later than the deadline (at least for me), maybe there will be late responses to the rebuttals

1

zy415 OP t1_jdhzcrf wrote

Definite worth reminding the reviewers to respond

1

Silent-Spirit4827 t1_jdkyuf1 wrote

My experience is the same. The ICLR reviewers were really active in responding.

1

Content-Dog-1985 t1_jdyd0bz wrote

As for me, I got no ICLR 2022 replies but several comments from ICML 2023. It all depends on your luck maybe...

1

passerby251 t1_jdugjjr wrote

How will AC deal with the case when a reviewer shows up in the last 2 hours to ddl and posts some new questions? Though we responded soon, the reviewer didn't reply anymore...

2

underPanther t1_jdukhr6 wrote

Well done on replying quickly. The timings of the last minute interaction should be apparent to the AC.

1

[deleted] t1_jd8ijx5 wrote

[deleted]

1

Nameless1995 t1_jd9scqe wrote

There would be a period of internal discussion after author-reviewer discussion period. So my guess would be (if there is no author-reviewer engangement beyond just the first rebuttal), AC (if they are willing to do their due dilligence) will simply push the reviewers privately and ask what their take is on the rebuttal. If in that private discussion nothing really happens (for example, all reviewers just go MIA), then really it might be upto meta-reviewer's personal judgment how they are going to take into account the strength of rebuttals.

2

UnusualClimberBear t1_jd9109w wrote

First, they know publication is now a big circus and that most papers are clever solutions to problems that don't exist or beautiful explanations that cannot be leveraged. Acceptance is random if your work is not in the top 2% but still in the top 60%.

Publication as proof of work is toxic

1

elegantrium t1_jdd8xyw wrote

I had 4,7,8 and 4's questions are orthogonal to the paper. He does not respond to our rebuttal either... What are my chances?

1

paulgavrikov t1_jdgzg5o wrote

Depends on the quality of the reviews. In theory, the AC could completely ignore all reviews. However, 7&8 are very good and you definitely have a high chance. Be proud of your work!

1

zhaoyl18 t1_jdfh2p5 wrote

So my only hope is the low-score-reviewer will not show up on the last day and say his/her mind is not changing.

1

[deleted] t1_jdplqud wrote

[deleted]

1

ILOVETOCONBANDITS t1_jdpmxfe wrote

Er i'm prob not the best to answer this but I think that's a pretty good score. While the AC can still choose to reject, I think you'd have like a 60% chance? Usually it seems scores of less than 5 are pretty strong reject and greater than 6.5 are pretty safe, but in the middle its more of a toss up. However, that was before the change in rating where 5 went from borderline reject to borderline accept. Now it seems this range has shifted down about a half point. So that's how I got my 60% chance estimate. Best of luck!

1

Anonymous_Penguin1 t1_jdt7fvk wrote

It seems that we could still post an official comment now on OpenReview.

1

Educational-Walk8098 t1_jdv3wbl wrote

Just one quick question, is it ok not having a summary of all the reviews, the rebuttal, and discussions? Normally I would do some kind of a summary of revisions and additional experiments that I made in response to all reviewers but unfortunately I forgot this time since this is my very first time using OpenReview for submission.

1

Unlikely_Hamster_935 t1_jdxk1mc wrote

How can we contact the ACs now? There is no official review button anymore...

1