When writing a review, it is important to provide positive, constructive, actionable feedback to the authors. When rejecting a paper, imagine that you are writing to next year’s best paper winners. What advice or feedback can you give to the authors to help them turn their paper to award-winning work?

Most robotics conferences work by having a senior member of the community request two to three reviews for each paper. This senior person is sometimes called an Associate Editor (at ICRA) or an Area Chair (at RSS).  This person finds reviewers to look at a paper based on the paper’s topic, with the goal of assembling a balanced panel to evaluate the paper.  Their job is to read the paper, the reviews, and aggregate the reviews to write a meta-review, summarizing the information from the reviews and justifying the decision made for the paper.  Often this senior person then leads a discussion of the paper at a program committee meeting, where a decision to accept or reject is made.  The details vary for each conference, and you should understand how your conference works.

Your goal in writing a review is to help the authors of the paper, and the conference program committee understand the paper’s strengths and weaknesses in order to make a good decision for the paper and the community as a whole. 

The first paragraph of the review should consist of a summary of the paper and its contributions.  This summary helps the authors understand how others view their work, and helps the area chair understand the key contributions for the work.

The next paragraphs should discuss the strengths and weaknesses of the paper.  It is important to be concrete.  For example, do not write “The evaluation is weak.”  Instead write, “The evaluation consisted of a study of only a single user, which was the authors of the paper.  It would be stronger to conduct a double blind study with a cohort of twenty subjects.”

The essence of a robotics paper is “make the robot do something it couldn’t do before, and explain why.” In evaluating the paper, you want to assess whether the robot is doing something new, and why it is doing it. Whenever reviewing a paper, even when rejecting it, consider that you might be giving comments to next year’s best paper.

The final paragraph should state your recommendation for the paper and the high-level reasons.