How I review a paper
Don't be a jerk and other gentle reminders
Hello darlings!
How is everyone doing? Shall we kick some ass this week?
Let’s talk about reviewing work for academic publications (and grants, etc). A couple of weeks ago, I tweeted some of my philosophies about reviewing here. This is my moral approach to reviewing – a) don’t fix everything, b) use “I” language not “you” language, c) be kind (always bebes), d) be inclusive, e) perfection is the enemy of progress, f) treat others how you want to be treated, g) we are all smart and our voices matter. Feel free to come up with your own philosophies of reviewing!
But then someone asked me: how do you ACTUALLY do a review?
This is my process. Yours might be different! That’s okay. Do you, baby beluga.
Review enough but have a limit
First, do your fucking share. I review a lot. But I also send a lot of articles out for review (and get rejected at lot! #myonlypainischampagne). Doing my fair share means putting into the pot what I take out. I try to review 3+ pieces of work for each piece that I send out for review. I keep track of my “balance” in a spreadsheet and through Publons (although I’m not as good at using Publons as I have been in the past), an idea that I got from Ben Lauderdale. At the same time, I also have a hard cap on how many reviews I can have in my queue at any time – it is currently 4. I cannot handle more than that without getting overwhelmed.
If you aren’t asked to review a lot… the first option is to send more stuff out for review! You can also create accounts in manuscript central for journals that you’d like to review for (making sure to list your interests), tell editors and prolific reviews that you’d like to review, and uh, submit more work so that you can review more. But also know that your rate of reviewing won’t always balance your submissions, particularly early in your career.
My review process:
Step 1: I get a request. I look at my current queue and see whether I have room. If I have 4+ papers waiting, I say no. I ALWAYS suggest alternative reviewers if I can! I try to recommend junior faculty, graduate students, and people at teaching schools that might not get asked as much as research active famous people.
Step 2: If I have room in my queue, I ask myself: do I want to read this paper? The answer to that is probably yes because I’m an eager little beaver who likes to learn new shit. If I want to read it and I have room, I look at whether I have 2.5 to 3 hours in the next 3 weeks. If yes, I say yes. I immediately download the paper into a “reviews” folder so it is sitting there, staring at me like an evil little gremlin. A rule I made for myself is that I never say yes to a review from my phone because then I can’t download the paper and I won’t remember I have to do the review.
Step 3: I immediately go to my calendar over the next 2 or 3 weeks and put three time blocks on the calendar:
- “Read X paper” (1+ hour, usually towards the end of the day)
- “Write review of X paper” (1 hour, at least 2 days after I read the paper)
- “Submit review” (30 minutes)
Step 4: When my calendar tells me to “Read X paper”, I sit down (or sometimes walk or sometimes ride my Peloton) and read it. As I read it, I take voice-to-text notes on the paper. These are a fucking mess. These are just my free-flowing thoughts on the paper: “I don’t like the intro” “What is this figure? I don’t get it” “Page 6 is confusing” “Suggest cite of that one conference paper from APSA 2016 from what’s her name the blond I bet that’s published now check on it” (← a verbatim comment I made last week). After I’ve read the entire paper, I finish with my general thoughts on the paper, including what action I’d suggest to the journal (accept, strong R&R, weak R&R, or reject). Please note that I’m a really fast reader and an hour is generally ALL I need to read the paper through and make these notes. You might be different.
Step 5: When my calendar tells me to write a review (at least 2 days later, so I can marinate on the paper a little bit), I sit down with the paper and the comments and write a review. I try to frame my review around three dimensions: what’s interesting, important, and true in the manuscript and how could the paper be improved on each of those characteristics.
Your criterial might be different. That’s cool. But be clear about what you are asking for when you ask for changes!
Is the paper interesting?
If yes, I discuss that in my review! Remember that your review is an opportunity to advocate for a paper to be published in a journal. I will literally write “this paper is interesting because…” (oh no now everyone will know I’m their reviewer!) If the paper is not interesting, why not? This might be a framing or organization issue. It might be that the authors are burying the lede or hiding exciting details. Works that lack a strong theoretical grounding often need improvement in this area. For example, if a paper is framed around “no one has studied this minor gap,” uh, that’s boring. Instead, why is that gap theoretically or practically interesting?
Is the paper important?
If yes, advocate! What’s important about this? If no, why not? As a scholar of gender & politics, I am particularly sensitive to the comment that work “belongs in a subfield journal” and I don’t say it. That’s something that reviewers say when they are trying to reinforce existing racist and sexist and elitist hierarchies. BUT authors can often be nudged to do a better job of clearly articulating why their work is important and belongs in this journal. As a reviewer, I’ll often recommend that the authors reframe an article around a bigger question or connect it to relevant scholarship. I consider my responsibility as an expert in the field to encourage the authors to ground their work in a broad set of relevant scholarship that will help them make the case that their work is important. I also check the gender composition of citations at this point and point out citation patterns.
Is the work in the paper true?
If yes, tell the editor what it is that makes you confident about the findings or conclusions of the paper! If no, now is the time to raise concerns about whether the data and results are what the authors say they are. Is the author measuring what they say they are measuring? Are they using the correct modeling approach to be able to say what they think they are saying? Are there alternative explanations for the findings that the author can identify? This is NOT the time to play the “what if” game – reviewers love to throw out a “well, what if…” Is this an intellectual exercise or is this a place where the answer to the question can help the author to demonstrate the truth of their findings? If the former – GTFOHWTBS. Or at least say “The authors do not need to do this, but an interesting idea is…”
I do not seek perfection from any paper that I review
There is no such thing as a perfect paper (or, rather, my idea of perfect is very different from other people’s idea of perfect). I want papers to be interesting, important, and true and I structure my comments around those three criteria.
Step 6: After I’ve written my review, I take some time off (15 minutes to 15 days!) and re-read the whole thing. When I submit my review to the journal’s website, I try to write a short note for the editor as to what I see as the main positives and negatives of the piece and whether I think it is possible for the authors to revise the manuscript to be published at that journal. I apply this standard to everything that I write: Would I be embarrassed or feel shame if my name was attached to this review?
Step 7 (optional): if I really liked the paper, I set up a google news alert for the title for it so I can download the final version / citation whenever it is published with the authors’ names attached.
Does this always work? Absolutely not. Am I always on time with my reviews? Fuck no. Do I let things slip through the cracks, especially during a pandemic + natural disasters? Absolutely. Do I feel intense guilt about that? Of course. But this process generally works for me and helps me stay on top of my reviews and their deadlines.
Whatever your process, remember that it is a person who is going to read these comments about their work and it is really easy to be cruel in an anonymous setting. My friend Jeff Guhin once said: "If the mark of your character is what you do when nobody's watching, then how you write an anonymous peer review is a continual test of decency." Be a decent person, my darlings.
XOXO
Mirya