Historiann has been thinking a lot about peer review lately. It seems, as in the old nursery rhyme, that peer review is like the little girl with the curl right in the middle of her forehead: When it is good, it is very very good, but when it is bad, it is horrid.
At its best, peer review helps writers avoid making dunderheaded factual errors and points them to other sources to help bolster their arguments. When it’s done by generous and intellectually engaged reviewers, it helps writers sharpen their arguments, tone down (or rev up) their prose, and to see more big-picture connections and implications of their work that even the writers couldn’t see until someone slightly more expert than they are pointed it out. What’s not to like, with a fair and humane group of supportive senior scholars freely sharing their wisdom with their (usually junior) colleagues? Furthermore, having one’s work reviewed by supportive senior scholars is a really great way of making new friends and influencing influential people. I’ve had that experience a few times–and I’m truly grateful to the people who lent their time and expertise to make me a better historian.
Well, that vision of peer review is very much an ideal, in the way that a 2-2 load at a wealthy institution with brilliant students and lots of leave time is an ideal that most of us will never know outside of our dreams. Peer review is fraught with opportunities for abuse, deception, and caprice. And, when either getting or keeping a job is on the line, that means that the misuse of peer review is not just a playful game of Chutes and Ladders. Here are some of the major problems I’ve seen firsthand or heard about from friends and acquaintances:
- Manuscript reviews at history journals and presses are rarely double-blind, so the reviewers know who the author is, but the author does not know who the reviewers are unless they choose to out themselves. Little wonder, then, that Professor Famousface’s articles are rarely rejected, and that Janey Noname, ABD, undoubtedly doesn’t get the same deference. Now, it’s frequently the case that the Professor Famousfaces, because of their greater knowledge and experience, actually write better articlesthan most of the Janey Nonames out there, but let’s not kid ourselves that these decisions are based strictly on the merits of what’s on the page. (I’ve also been a little surprised to learn that journal editors are very status-conscious: they want to publish only or mostly articles by Professor Famousfaces, so they don’t reserve a lot of room for advancing the careers of the Janey Nonames.) I always thought that taking diamonds-in-the-rough and polishing them up would be the real challenge and great joy of being an editor, and that publishing articles by the already-acclaimed was something that any chump could do. Clearly, I’ll never be a journal editor, with my screwed-up priorities!
- (An aside, but related to the point above: the first time I gave a paper at a conference with the title “Associate Professor” next to my name in the program, I was contacted by the editors of two big journals and invited to submit the revised edition of the paper to them. While I was flattered by the attention, I also thought, “where the heck were you when I was an Assistant Professor trying to get tenure, when an article in your journal would really have been important to me?” Now that I’m a made man, sealed in the temple, a member of the guild, so to speak, you’re dying to publish my incredibly brilliant and insightful work? That kind of sucks!)
- A lot of worthy work by grad students and junior faculty gets rejected because peer review, whether single- or double-blind, serves as an instrument to discipline and punish. In the hands of an unscrupulous or intellectually un-self-confident person, peer review is a means to marginalize arguments and perspectives that appear to conflict with the reviewer’s body of scholarship or that challenge conventional interpretations. In one of my fields, which is still heavily dominated by white men in the senior ranks, I have seen and heard of peer review used to suppress scholarship written by feminist women and people of color. (I’ve also heard about this in many other fields, because most of them are dominated by older white men.) And again, because jobs and tenure are on the line, peer review can be a powerful tool for maintaining status hierarchies and the intellectual homogeneity of the in-group.
- At journals, the function of peer review is extremely unclear, because journal editors are sovereign, and authors should be aware that they’ll publish what they want to publish sometimes in spite of what the reviewers recommend. For the most part, I expect that journal editors who want to publish an article are sure to find friendly reviewers for it, and if they don’t want to publish an article, they are sure to find reviewers who are practically guaranteed to shoot it down. (Most editors, most of the time, are probably looking for some honest opinions to guide them–I should probably emphasize that point a bit, because this post may seem to be only about hatin’ on editors.) However, when reviewers report back unanimously (or nearly unanimously) that an article is great and should be published and the editor rejects the article anyway–what we to make of that? (This just happened to a friend of mine. Coincidentally, another friend of mine heard back from the same journal editor on an article that ze had submitted. This second friend received a very warm and friendly “revise and resubmit” letter from the editor, despite the fact that 2 out of 3 reviews that came back to hir recommended against publication, and one of the negative reviews was extremely short, nasty, and unhelpful. As they say in the military: Whiskey Tango Foxtrot? Oh, and I have to say that both letters explaining both of these, shall we say, eccentric decisions were so opaque, turgid, and contorted that I now have serious reservations about this editor’s judgment about what good writing looks like. The letters looked like the product of a very disordered mind.)
- Journals also seem to have no shared rules or system of peer review. Some journals I’ve published in have showed me the readers’ reports (which is what I always thought was the industry standard for peer review.) Other journals never showed me the readers’ reports, and instead sent me only an editor’s letter of acceptance with hir suggestions for revision. When I asked to see the readers’ reports, for the purposes of satisfying my senior colleagues that this was in fact a peer-reviewed article, I was told by this editor that “we just don’t do that.” Sing it with me: Whiskey Tango Foxtrot? Wev. It was an established journal in one of my fields more than eighty years old, so people just accepted that that article was legitimately peer-reviewed, but to this day I have no idea if any “peers” actually “reviewed” anything. I had another article published in a journal that is accepted as a peer-reviewed journal that never did more than copy-edit my work, so far as I know.
- From what I’ve heard from others, book publishers can be just as capricious, and they certainly are even more clearly sovereign than journal editors. (Historiann loves her editor and publisher, and is always raving about hir/that outfit.) Interestingly, I found publishing my book to be a much more straightforward and less mysterious process than publishing in journals, and so have most (but not all) of my friends.
So, should we all give up and just publish on the non-peer reviewed, open-source internets? Realistically, that’s only an option if you’re already tenured and don’t really care if you ever get another promotion. I guess what I’ve learned in my first decade of working with journal editors and a publisher is that you have to choose your venue for publication carefully. You have to grow a thick skin, because there is a significant minority of creeps who hide behind peer review to suppress, rather than foster, new scholarship. (Cowards.) And it helps to have a lot of friends–at the very least, they can console you when you get a negative verdict or review. Oh, and never do an anonymous review–I always sign my name and invite the author to contact me if ze has any questions. Mostly I do this because it forces me to make constructive criticism. I’m not the voice of God–I’m an individual that was trained in a particular way and have my own axes to grind, and the author should know that so that ze can put my advice in context. It’s a rigged system, but we can each make the process a little fairer and a little more transparent for each other.
What lessons have you all learned–whether you wanted to learn them or not?
28 Responses to “Peer review or smear review? Reflections on a rigged system.”