How to write a peer review Matt Ayres, Dec 2006 (Jan 2009)

The primary literature, which is arguably the backbone of scientific knowledge, is

defined by the peer review process. Many of us contribute as much to science by
serving as reviewers and editors as by publishing our own papers. However, few of us
have received explicit training in how to efficiently write effective reviews. The scientific
community would probably benefit from more structured consideration of the theory and
practice of reviewing papers. This is an effort to make progress. I encourage
comments, critiques, and additions. I encourage anyone to borrow and build on what is
here, and to press their colleagues and mentors to cultivate the development of our
skills as reviewers.

A paper review should be a clear, efficient persuasive argument directed at (1) the
editor, who will be making a decision on the paper, and (2) the authors. Start the
review with a brief paragraph that clearly indicates your judgement regarding the fate of
the manuscript (typically reject, major revisions and possibly accept, or minor revisions
followed by publication). If you recommend rejection, mention another journal where it
might be appropriate and/or indicate what would be needed to make it publishable (e.g.,
different analyses, two more years of data, a proper experiment, etc.). Follow the
summary paragraph with an elaboration of each general criticism (typically 2-5 points in
my reviews). Make these first two sections easy for the editor to understand, evaluate,
and briefly paraphrase. Follow this with detailed comments that are more minor and
which typically reference particular lines or paragraphs. I often label these three
sections as: “Overview”, “General comments”, and “Detailed comments”.
Things to consider in developing your opinion
If what the authors say is true, is it interesting and appropriate for the journal?
C Are the conclusions novel or confirmatory?
C Are the conclusions of interest to a broad audience or mainly to specialists?
C Did the study involve novel questions, hypotheses, techniques, or
interpretations? Could it be said that “This is the first study to ...”?
C Is the work such that the investigators were uniquely qualified to conduct it?
C Is the work such that it is unlikely to be repeated or substantially extended
anytime soon?
To evaluate these points, study the 1st and last sentences of the abstract, the 1st
para of the intro and topic sentences in the discussion. You may wish to read the
goals for the journal (e.g., http://esapubs.org/esapubs/journals/applications.htm for
Ecological Applications and
http://www.blackwellpublishing.com/aims.asp?ref=0030-1299&site=1 for Oikos),
check the impact factor for the journal, and even browse some recent titles and
abstracts .
If the stated conclusions are not sufficiently general and interesting for the journal,
then your review need not go further. If it might be a match for the journal, continue.

Is the research question clear and well justified?

Is the technical approach logical and rigorous?
C How strong is the inference for the important conclusions? (e.g., correlations vs
experiments; were there tests of theoretical predictions that would be improbable
if the hypothesis were not true; were the most important conclusions supported
by more than one line of evidence?)
C Given the research question, did the investigators employ the strongest
imaginable technical approach?
If the question is not compelling or the technical approach is not rigorous, your
review need not go further.

Are the results clear and statistically rigorous?

If there are equivocal results, could the authors realistically do more to nail it before the
work is published?
Does the discussion flow logically from the introduction? Is there a clear and relevant
topic sentence for each paragraph?
How is the clarity, precision, and efficiency of the prose?
Is the ms too long? If so, where? Should the ms address fewer topics or devote fewer
words per topic?
What has been missed? (e.g., literature, analogous systems, connections to other
subdisciplines, alternative interpretations for the data.). Good reviewers evaluate what
is in the manuscript. Excellent reviewers also evaluate what is not in the manuscript.
If I have come this far, and still like the paper, I will probably recommend publication. At
that point, I give the abstract another very careful reading to make sure that every
sentence is justified by the research. I also look again at the title, the figures and
tables, each topic sentence in the discussion, keywords, and references.

How much time to spend on a review? Enough to meet your responsibilities and no
more, unless you altruistically choose to try to help the authors. If your
recommendation is going to be reject, there is no need to write a long and detailed
review. I may do more than I need to if I see that the author is a junior scientist, is not a
native speaker of English, or is from a country or institution that is under-represented in
the literature. I may also do more than I need to if I think the paper has great potential
but is not quite there yet or if it seems like the data really should be published but the
ms has to be fixed first. I find it efficient to read the ms once, rather quickly, and then
reflect for a few days on the higher level questions (whether the conclusions are
important if true, etc.). My reviews tend to be shorter for top tier journals. Do a good
job with your reviews but do not spend too much time. I try to do reviews with 3 to 4
hours of work (but usually with four separate bouts of work - the first to read it, the
second to study it again and outline my response, the third to actually write the review,
and the fourth to read it again for tone and content before submitting it.)
I spend the most time on papers that I recommend publishing (to convince myself that I
am not admitting junk science) and when I am trying to be helpful to authors.

Conflicts of interest and confidentiality. See journal instructions for reviewers (e.g.,
http://esapubs.org/esapubs/reviewers.htm). Every journal with which I have worked has
a strict requirement for confidentiality. Conflicts of interest (COI) for reviewing papers
are not usually very explicit and the norms seem to be less stringent than, for example,
reviewing proposals for the National Science Foundation (see Wiley-Blackwell’s
Publication Ethics). As an editor, I try to avoid asking for reviews from people who are
at the same institution, are identified in the acknowledgements, have co-authored
papers with the authors, or had a student or mentor relationship with one or more
authors. Editors cannot always tell if there is a COI, so the potential reviewer has a
responsibility to decline if there is a conflict of interest. However, knowing an author
(and liking or disliking them) cannot be grounds for declining a review because the
world is small and qualified reviewers will frequently know the authors. My view is that
being a sometime competitor or cooperator with one or more of the authors also cannot
be an automatic conflict of interest for the same reason (but it provides reason for
careful consideration). I generally decline to review papers that were submitted by
colleagues with whom I am a co-PI on proposals because their publication success
might influence my future grant success, but I commonly review papers submitted by
people I know, and I have reviewed papers that I have previously discussed with the
authors. If you might be in conflict, explain the situation to the editor (e.g., under
“Confidential remarks to the Editor”, which will not go to the authors). The topic of
conflicts of interest between reviewers and authors probably deserves more discussion
within the scientific community. For example, I invite discussion motivated by this
paragraph.
When to accept or decline requests to review? Decline any reviews for which you
would not be qualified, but do not underestimate your qualifications – there will usually
be parts of any good manuscript where your expertise is limited. Decline any reviews
that you cannot complete within the approximate time frame that the journal expects.
You have a responsibility to write as many reviews as reviews that you receive (e.g.,
review 2-3 papers for every paper of yours that gets evaluated by two reviewers plus
an editor). I generally review more papers than I need to to meet this requirement
because (1) I find it satisfying to contribute in this way to the primary literature, (2) I
usually learn enough to compensate for the time, (3) it helps my own writing, and (4)
there are some other professional benefits (e.g., you can put it on your CV; and you can
gain the respect of editors, which might lead to seminar invitations, etc.). Also, the
more I do, the better I become. I must be 5x more efficient and competent at reviewing
papers now than I was 5 years ago. This is good in lots of ways.
If you decline to review a paper, do it promptly and try to suggest one or two other
people who could be qualified. Feel free to tell mentors or senior colleagues with
related interests to think of you as a potential reviewer to suggest when they get a
paper that they do not have time to review. Note that many editors will welcome
suggestions for qualified grad students to review papers, especially if it comes from an
advisor who offers to work with their student on the review. I think it would be good for
everybody if this practice becomes more common.

When to sign reviews? The norm is for reviewers to be anonymous to the authors. I
estimate that reviewers elect to retain anonymity in >80% of the cases. Signing a
negative review can create enmity among those who will later be reviewing your
proposals, papers, job applications, etc. Thus it is normal and safe to remain
anonymous. Nonetheless, I sign about half of my reviews. Signing my reviews seems
to enforce a collegial constructive tone in my reviews. Also, I have developed many
positive professional relationships as a result of identifying myself in reviews. Perhaps I
have also created some enemies by signing reviews, but I have never noticed it (not
that I do not sometimes get reviews that read like anonymous hate mail). I will not sign
reviews if I think there is even a small chance that the authors might hold a grudge that
will hurt me later. I do not sign reviews that are highly positive because it might imply
that the authors owe me something later, and it diminishes the impact of the positive
review for the editor, who will probably also wonder if I am trading for future
considerations. Conversely, I find that I am more persuaded as an editor or author by a
negative review that is signed.