So, you’ve been asked to review a paper for a journal or conference. How do you approach the process? If you are at the beginning of your reviewing career, please do not review anything until you have carefully read, and digested, the materials described below.
First, try to think of the criteria that I have set out in the paper writing checklist. These are criteria I ask my students to use to decide whether their work is ready for publication, and represent a good framework for deciding whether a paper you are reviewing should be published. Of course the ultimate decision and your review need deeper consideration. A wonderful guide is provided here by the editors of the journal ACM Transactions on Computing Education.
Before I proceed, an important point to make is that you should not review anything that is outside your area of expertise, even if you feel honored to be asked to review it. Providing uninformed reviews, whether positive or negative, harms the scientific community and does a disservice to the authors. If you still want to review a manuscript that is somewhat outside your area of expertise, it is your responsibility to expand your knowledge of the subject so that you can make an informed decision. Do not be surprised if you’ll spend a week or more reading dozens of papers in order to be able to adequately evaluate a manuscript. It will be time well-spent, as at the end you will be able to make a meaningful contribution to the authors, and the journal and conference considering the manuscript for publication, and you will also acquire new knowledge.
Now, a bit about the structure of the review itself. Unless the journal/conference provides more specific guidance, here’s how I like to structure the reviews:
- Brief summary of work. Here you take a couple of sentences to describe what you understand the paper is saying. This is simply an objective statement of your understanding, and does include any value judgement. If you can’t write this short summary, you shouldn’t review the paper, because that means that you don’t understand it. This introduction is useful both for you to ensure that you can review the paper, and to give the authors the confidence that your criticism is grounded in your understanding of what they are trying to say.
- Executive summary of your assessment. Here you summarize your assessment of the paper, touching on the most pertinent factors, positive or negative. Unless you are explicitly asked for this, the text of your review should not contain an actual recommendation (publish/reject). That recommendation is represented elsewhere in the review system and is ultimately the decision of an editor or program committee.
- Specific comments about things the authors can improve. Usually, these comments are broken down into “major” and “minor” recommendations for changes. Note that I emphasized “recommendations”. Here you are not criticizing the paper or the authors, rather identifying flaws with the presentation, experiments, data, etc. and proposing ways to address them. Be specific and constructive in your feedback. Even if the paper is ultimately rejected, the authors will benefit from your advice on how to improve it.
The next thing to consider is the style of the review. By “style”, I mean how you express your opinions about the paper. It is important that you do so with empathy. Write the review you would like to see, rather than the review you believe the paper deserves. Even if you don’t know the authors, you are part of the same, fairly small community, and the well-being of the community depends on helping each other develop and grow. Your criticism should focus on the work and the actual text of the manuscript, not on the authors.
As an example, it is better to say “The manuscript lacks sufficient references to prior work on widget automation. For example, papers X, Y, Z, should be cited and the work described here placed in their context.” instead of “The authors ignore prior work in the field“. The former comment identifies a weakness in the manuscript and proposes a way to address it, while the latter implies malpractice by the authors.
There are many other aspects of the scientific review process that I am not touching upon here, but are covered by others. Here are a few must read resources:
- Niklas Emlqvist’s guide to reviewing HCI/visualization papers.
- Paper describing an experiment at the ICML conference to involve novice reviewers in the evaluation process. The supplementary material contains a useful guide for reviewers that is relevant beyond ICML.
- Tandy Warnow’s guide to reviewing papers.