Rebuttal Advice


REBUTTAL GUIDES

  1. An excellent rebuttal process suggestion from Albrecth Schmidt.
  2. Some more advice from Gene Golovchinsky at FXPAL
  3. Jeff Bigham
  4. Meri Morris
  5. Niklas Elmqvist
  6. Hyunyoung Song

If your paper is definitely getting rejected

Always thank the reviewers anyway.

“We thank the reviewers for their time and efforts in providing thoughtful feedback on our paper. We look forward to using this feedback to improve the paper and look forward to a future submission. Thank you.” (Phrasing suggested by Eric Paulos)

“A sincere thank you to all the reviewers of this paper; your reviews were very thorough and raised excellent points that will help us improve this paper and communicate more effectively for future submissions.” (Phrasing suggested by Payod Panda)

“We thank the reviewers for their valuable and constructive critique and will use it to develop this work further for a future venue. Thank you!” (Phrasing suggested by Wendy Mackay)

REBUTTAL ADVICE

(The following is from Wendy Mackay, Scott Klemmer, Gene Golovchinsky and Bjoern Hartmann)

Wendy: Rebuttals are usually the most emotionally charged part of writing a paper. Remember that, no matter what the reviews say, you need to respond in a polite, clear and factual manner. Your goal is not to argue with the reviewers, but rather to convince the program committee that your paper is worth accepting. This is a tricky writing exercise, no matter how much experience you have.

Two principles from Scott Klemmer at Stanford:

  1. Focus on the changes you will make. i.e., have the rebuttal be forward-looking. (Rather than offering your opinion of the reviews.) Summarizing revisions that will appear is an easy way to generate positive energy. And demonstrate that you understand the concerns. Disagreeing with reviewers is sometimes essential, but most often should be a minor part of the review, not the centerpiece. And don’t recapitulate your favorite part of the review. The reviewers can read.

  2. Show, don’t tell. Don’t say, “In revision, we will compare our work to Cosby ‘83’s Jello Paper.” Say, “The primary distinction between our work and Cosby ‘83 is the use of carrots. We will explain this and add the reference in revision.” Don’t say, “In revision, we will present statistic Y instead of / in addition to statistic X.” Say, “Statistic Y is (BLANK). We will present this instead of / in addition to statistic X.” In other words, don’t say what you will do. Do it, and summarize the upshot in the rebuttal.

Bjoern would add:

  1. Address the most significant issues first, and smaller clarifications last. Issues raised by the 1AC or 2AC are often especially relevant, as they have distilled multiple reviews for you, and they will be present when the paper is discussed.
  2. Use headings (e.g., IN CAPITAL LETTERS) to signal what topic is addressed in which paragraph.
  3. Mention the reviewers who raised each issue (e.g., R1, R3 question our use of a t-test…)
  4. Keep it positive and show that you believe you have a real contribution! Grumbling will not change anyone’s mind.
  5. Start with a few deep breaths. […] Be factual. Rhetoric and invective will not help your case.
  6. Understand the big complaints reviewers might have had, and whether these are fatal flaws in your work, or aspects that can be corrected. Look to the meta-review to see what items the Associate Chair thought were important among the litany collected by the reviewers. The meta-review should highlight the important shortcomings; addressing those will improve your chances of acceptance.
  7. Make sure reviewers know what the takeaway from your paper was. This is the elevator pitch for your paper, and you should have it nailed. Sometimes reviewers don’t read the paper as carefully as you’d like them to, so adding a bit of emphasis here may help alert them to what’s important about the work.
  8. One useful tactic for writing a rebuttal is to offer concrete changes that are responsive to reviewers’ comments.
  9. If the reviewers didn’t understand your analysis, or wanted to see a different cut at it, put some details of the analysis (numbers help!) right into the rebuttal with the explanation that you’ll add it to the paper.
  10. Get someone impartial to read the reviews and your rebuttal, and ask them to comment on the tone as well as on the facts.

A rebuttal writing schedule from Wendy Mackay:

Day one: All authors

Day two: Primary author

Day two: Co-authors

Day three: All co-authors

Remember:

Day four:

Example rebuttal 1

We thank the externals and the 1AC for their thorough and detailed expert reviews. All reviewers find novelty, thoughtful writing, and good coverage of implementation and generalizability. We first discuss issues identified by the 1AC, then additional individual concerns.

VALIDITY/EVALUATION:

HOW PAGES ARE ASSOCIATED TO CODE: DOES N=2 WORK?

RELATED WORK

INDIVIDUAL CONCERNS

Hyunyoung Song

I originally wanted to post this during the SIGCHI rebuttal period. However, I was afraid that it may hurt the acceptance rate of my paper somehow. (Maybe by revealing my identity to the external reviewers? I don’t know. Anything is possible. No?) Hence, I decided to write this immediately after the notification, while everything is still hot in my head. Maybe students that have to write a rebuttal for CHI2012 may find my post useful.

First, I will start by my opinion on why it is difficult to write a convincing rebuttal.

If 90% of writing a paper requires knowing how to explain a research idea and 10% knowing how to convince another researcher, writing a rebuttal requires 10% of the former and 90% of the latter. As a junior researcher (like myself, a graduate student), the only senior researcher you talk to in a daily basis is your advisor. Yet, your advisor is only one sample point among the pool of senior researchers. How on earth would you know how to convince another senior researcher when you do not even know how to initiate a serious conversation with them?

It is always difficult to take criticism. This is especially the case for people like us Ph.D. students who always strive to produce results that are flawless. We seldom hear that our work is a crap in our face. Additionally, the criticism that we have to take in the review seems unjust especially when you believe that the person who’s making the criticism spent maybe a day or two reading a document that you spent endless nights editing over and over again for several months. Some reviewers are actually nasty too. Although it is written in fancy and erudite terms, sometimes the reason that they are rejecting your paper is simply because, “you did not do a good job in impressing me” or “I do not buy your research story”. How would you possibly stay sane when you read these comments?

Despite, many reasons to yell at your reviewers and say, “you are full of s**t~!!”. We should not even express this in any indirect way while writing our rebuttal. In the past, I always had to discard the very first draft that I produced after reading the harsh reviews. My rebuttal was bitter, smirking, cynical and mean. This was not obvious until I slept at least two days crying over it and returned to my calm and reasonable self.

This year, due to my post (in my Korean blog) about my experience in writing a CHI rebuttal in the past years, I’ve been asked by several junior students (outside my institute) to help out in writing their rebuttal. While doing so, there are several tips that I repeated. Here are some of them.

[Understanding and analyzing the review]

Read your reviews with another coauthor and have an in-depth discussion. It is important to address the most important issues first and address only the problems that reviewers raise. Sometimes, I realize that I misunderstood what the reviewer meant and was addressing something that was totally unnecessary. Sometimes, I structured my arguments in the wrong order: order of least importance to most importance. Many authors actually make these mistakes during the writing process, not because they are careless, but because the reviews are somewhat encrypted. Not all of them are kind enough to tell us “A is unconvincing and A’ is my opinion. B and C are what I do not understand but authors should only address B because C is not as important”. It comes more like this, “A is weird, B is weird, C is weird”. Usually, meta-reviewer tries help us by decrypting the dialect of the external reviewers so that the authors are not at a loss. However, not all meta-reviewer are nice either. For this reason, I always talk with my advisor or one of my coauthors for 2~3 hours about the reviews before writing anything. This usually helps a lot.

[Writing process]

Agree with your reviewers. Last year, one of the rebuttal of the paper that I reviewed basically stated that “R1 (myself) is wrong because so and so, and our paper is awesome”. This rebuttal didn’t acknowledge some of the important problems that I pointed out and tried to challenge my judgement. I was offended and became more strict in finding faults of the paper during the discussion period. This is the last thing that the authors want. Making an enemy among the reviewers. To make an ally, you have to tell them how useful their feedbacks are and you have to sincerely mean it too.

Specify how the camera ready version will be reflected based on the reviewer’s request. Often, there are rebuttals that just say, “I understand R2’s point”. This is only half-baked response. The goal of the rebuttal is to demonstrate how the camera ready version will be changed according to the issues raised. Hence, the response should be more specific and go even one step further. Like this: “the question A is raised by R2 because we only explained B in section C when it is also needed in section D. In the camera ready version we will clarify B in section D”

Do not say that the draft will be improved with a major change. I have seen several authors that say in their rebuttal “After the submission we did A,B, and C which addresses all of R1,R2, R3 questions which will be updated in the final draft”. This approach is very bad. First, you are admitting that current draft has many issues. Second, during the PC meeting, papers are discussed “as is”. If it is concluded in the PC meeting that the paper requires major revision, PC members advise that it should be submitted to a future venue. Better approach is to figure out what reviewers misunderstood. Explain why there was a miscommunication and offer ways for reviewers to solve those misunderstandings. Point to a paragraph or a figure in the paper. If needed, direct them to a reference that is not cited in your paper. This is what a rebuttal is for; to clarify.

[Formatting and Style]

Although 5000 characters limit may seem insufficient to explain everything, do not hesitate to allocate some of those characters in white spaces and phrases such as (in response to R1, as mentioned by R2, in our RELATED WORK section, in p8~9). At a glance, they want to see which major raised points have been addressed and which part of the paper they should read again. Sometimes, I use the web browser search tool (namely Ctrl+F) to locate my reviewer id (RX) in the rebuttal and read the accompanying paragraph more carefully to make sure that I didn’t miss anything.

Last but not least, write short and direct sentences. Any sentence that you write to explain in your rebuttal have 50/50 chance of helping your paper and hurting your paper. The longer and indirect a sentence, the higher chance of mis-interpretation. On top of that, reviewers have very short attention span. If sentence become convoluted, they will read what they think the sentence is saying as opposed to what the sentence is actually saying.

The biggest question behind all this is, “Does reviewers actually change their score after reading a rebuttal?”.

And the answer is “YES”~!!. Among the 7 papers that I reviewed this year, I increased the score by 0.5 in one paper because I was happy to learn something that I didn’t know from the rebuttal. Among my 4 papers (2 in previous years and 2 in current year) that have been accepted, 3 paper scores actually increased (by +0.4, +0.1, +0.4) after the rebuttal period.

Although it is painful and tedious process to write a good rebuttal, it is very rewarding once you DO write a good one.

© 2021 / Molly Jane Nicholas / email