Why Review a Paper?
- Each paper you submit generates about 3 reviews. So, on average, you and your co-authors should collectively provide a similar number of reviews. Our entire research enterprise would stop functioning without Reviewers.
- When you review a paper you read a manuscript critically, and start to realize potential criticisms reviewers might have of your future papers.
- Gain privileged insight into state-of-the-art in research, and have a say in what is published and what is rejected.
When should you say “yes” and when should you say “no” to a review request?
Reasons to say “yes”:
- The 3 reasons above [be a good citizen!]
- The paper is related to your current interests and/or you are curious about the paper.
- The person asking for the review is someone you would like to do a favor for, for whatever reason. (Sad but true).
A problem is that editors often try to get people whose work is cited to review the paper they are handling. Although this is definitely appropriate in some cases, it creates the following issues: (a) possible conflict of interest, i.e. people like having their papers cited and might be too positive on the paper; (b) often trying to get people to review stuff they are no longer interested in or following closely, or people who get tons of such review requests since they have a seminal paper in that area.
Reasons to say “no”:
- The paper is out of scope of your interests and/or expertise
- You have a conflict of interest with one of the authors, e.g. he/she is a close collaborator. The line here is not always clear, but a decent litmus test is if you think there might be a perceived conflict if someone knew your affiliation with them, then there probably is.
- The paper is submitted to a journal or conference you never publish in (you can still review such papers, but in my mind have no obligation to do so).
A tricky situation is of papers that appear just from the title and/or abstract to be of low quality, e.g. on a topic that has been beaten to death. On the one hand, if all good researchers say no, then the paper might get accepted by the people who eventually review it. In these cases we’d request that you accept the review, and be appropriately critical, per below.
Ok, so you’ve agreed to review the paper. Thanks! Here’s a process and then a format that I recommend in this case.
First of all, do the review ahead of schedule, as soon as you can, and before the arbitrary 6 week deadline.
Second, you should determine in about 10 minutes whether the paper is a definite reject. This is usually based on a careful reading of the abstract and introduction (and perhaps the system model) and is due to the following reasons:
- The paper clearly has little or no novelty, and is just way below the very high standard required for TWC, which is that only the most innovative and interesting papers can be published in this venue.
- The paper’s topic is of little contemporary interest, e.g. is on a well-studied subject that was “hot” ten years ago, and also is weak on novelty.
- The paper is very poorly written, with many grammatical errors, or is just otherwise of poor presentation quality.
- The paper has a major flaw in the assumptions or model that renders the balance of the development irrelevant. This should not be just because you are not a fan of that model.
Note that at the EIC level we are working hard to reject such papers (especially #’s 1-3 above) before they ever reach the Editor or Reviewers. The Executive Editorial Committee does a careful pre-screening of papers and makes immediate reject decisions to the EIC. Additionally, the Area Editor and then the Editor does his/her own screening. Nevertheless, a Reviewer may have particular expertise on a topic that we lack, and might be able to more readily determine that the paper is a trivial iteration on a prior one, or is fundamentally flawed. In such a case, and assuming a high level of confidence, a detailed review can be forsaken and a more concise but very clear and specific review on the order of 2 paragraphs may be sufficient.
Assuming the paper is above the “definite immediate reject” bar, here are some guidelines that are recommended for most papers. This obviously varies depending on the type of paper.
- Be fair, but critical. This is why reviews are anonymous. It gives the editor more freedom to act if you are critical, which he/she will appreciate. It will also generally help the authors to maximally improve their paper as long as the comments are specific and fair.
- No matter how bad the paper is, do not under any circumstances insult the authors or their work, or show any emotion. Be professional and helpful.
- Make sure you maintain your anonymity – for example, citing a bunch of your papers for the authors consideration is a giveaway about who you are (or your circle). Only cite your paper if it is truly essential.
- A serious review should be at least 1-2 pages of single spaced text. The more comments you make the more helpful it is to the editor and authors.
- Your review should be cut and pasted into the provided web review form. Note that pdf files often have tags that identify who/what created them that can be uncovered by mischievous authors. Further, they sometimes are not correctly attached by Manuscript central.
- Do your own check for novelty. Some authors may not cite important papers that might make their paper look less novel.
Typical structure of a review
- Summary of the paper and your review. Begin the paper with 1-2 paragraphs in which you summarize the review in your own words and provide a recommendation and the reasons for it. Typically this includes a summary of the key contributions of the paper (your opinion of the key contributions, not a restatement of the abstract). This shows the authors you understood their paper. Typical recommendations are:
- Accept – almost never given to an initial submission, this is usually given to a strong paper after 1-2 rounds of revision. However you can make it clear that the paper is likely to be accepted in the future.
- Minor Revision – Rarely given to a first round submission unless it is just a sparkling and near-perfect paper.
- Major Revision – This is the typical first round decision for a very good paper, which will be enhanced substantially usually in the next round of revision.
- Reject – This naturally is our most common decision, since we aim to be extremely selective, and the most disappointing one for authors. It means that the paper is just not up to the quality we require for TWC. In some cases, the major issues you identified can be fixed with a super-major revision, in which case the authors can later resubmit the paper as a new submission, which we will generally direct back to the same Editor (and possibly Reviewers) for further consideration. You might wish to clarify whether you think it is such a case, or whether it is an “Unconditional reject” (paper should not be considered again for publication).
- Major comments, enumerated list. These are your major concerns about the paper, and can be either very concrete (I belief proof of Thm 1 has an error) or philosophical (issues with their models, conclusions, etc.). These are the ones that require nontrivial fixes and may in fact not be fixable. They should be described in some detail.
- Minor comments, enumerated list. In as much detail as you are willing to provide, list every thing about the paper that you found in error, unclear, or otherwise problematic. Some of these may be crucial, but by “minor” I mean they can probably be corrected fairly easily and don’t significantly impact your accept/reject recommendation for the paper.
For revisions/responses, the review can usually be simpler.
- Read through all the reviews and the responses to them (you generally can see the other reviews, and they may have identified something crucial you missed in the first round).
- Still provide an overall evaluation of the paper, although it can be a single sentence if it is the same (more or less) as your first evaluations.
- Make sure the authors thoroughly addressed your key concerns, in particular in the manuscript. Too often authors simply respond to the reviewer without changing the manuscript sufficiently. Read all the reviews and check key new parts of the manuscript.
- In cases of a borderline first-round paper with a low quality or careless attempt at revision, argue strongly that the paper should then be rejected and provide the Editor with the details they need to make such a decision.