Comments on: Peer review or smear review? Reflections on a rigged system. History and sexual politics, 1492 to the present Fri, 19 Sep 2014 13:41:03 +0000 hourly 1 By: It’s Beginning to Look a Lot Like Monday Links | Gerry Canavan Mon, 23 Dec 2013 12:48:29 +0000 […] * Peer review or smear review? Reflections on a rigged system. […]

By: Arle Lommel Sun, 22 Dec 2013 18:23:10 +0000 I just found this posting when a friend pointed me to it. It raises some interesting issues. While I am not in the field of history (I was in folklore studies before making a jump to artificial intelligence, but that is another story), I think I can also contribute something from the perspective of journal editors.

First, very often journal editors know what sort of review a piece will get in advance. They receive many contributions and they have to read them to know where to send them. Serious problems are usually apparent from the beginning. In other cases the submission may be perfectly good but the journal represents the wrong audience for it (e.g., it emphasizes certain perspectives and the authors didn’t do their homework before submitting something out of range for the journal).

In such cases it does nobody a service to send those pieces out for a lengthy review process just to confirm that the piece is unpublishable in that journal. Now of course this statement presumes that the editors are acting in good faith and actually can detect bad/misguided papers reliably. There will certainly be a certain number of false negatives—pieces rejected that should have been accepted—at this stage (as there will be for a full peer review process), but eliminating most true negatives reduces the burden spent on making sure that pieces that actually have a chance can be given proper attention.

In the case of the journal I worked at we would take these early negatives and decide how to approach them. In many cases we took a nurturing approach and gave preliminary feedback to the authors telling them what would be required to obtain a full review and consideration for publication. In other cases it was clear that the authors would not stand a chance with their articles in our journal, so we felt it a better service to deliver a polite rejection stating that the article was not a good fit for our journal.

Now we were very friendly and took many submissions, especially by those coming from outside the U.S. and western Europe, and worked extensively with authors even before peer review to help them submit something that would make it through peer review. I realize this approach is far from common, but in many cases we had authors who just needed a little help to navigate writing for an American audience and framing their arguments in appropriate terms. We also took some degree of pride in introducing non-Western perspectives to our audience.

For those that did go out for review, we were very careful in how we handled them. Many, many times we discarded reviews that were clearly biased or unfair. And rather than just handing over the comments, we would synthesize them into actionable summaries for the authors. So if we go “word soup”, we saw it as our responsibility to fix them for the authors. Often we would put authors in touch with reviewers where it made sense to do so (always with permission from both sides).

I suspect we were not typical in any of this. That being said, I think peer review can work, but it requires the commitment of the editors to be fair and to respect both sides and mediate between them.

Because of that experience, I try very hard in doing reviews to do two things:

(1) If there are factual problems, I point them out and, if relevant, supply references. For example, in a recent review some grad students were trying to pass statistically insignificant results off as positive and I had to explain to them why their results were statistically indistinguishable from noise. It won’t be pleasant for them to read, but because I took the time to do the statistical analysis they didn’t, they now know what they need to do. And if they don’t want to do it, I gave them guidance on how to rewrite their finding as negative findings (which would actually be quite valuable in the area).

(2) If there are problems I have with the theoretical perspective, I am careful to label those as my *preference* and offer criticisms that I think will help strengthen the article, even if I disagree with it. After all, tearing down helps nobody. So I try to build up, even those things I disagree with.

So peer review can work, but it requires good faith by all parties and editors willing to intervene to ensure that happens. If peer review fails, as it clearly does in many cases, it is ultimately the editors who are responsible.

By: Andrew Riggsby Sun, 22 Dec 2013 13:07:45 +0000 “There are problems with this system–notably that reviewers might get irritated at being asked to review the same article 3 times–but that doesn’t seem so burdensome. If you’ve read it once, you can send off the same review again to another journal, with a few thoughts as to how the article does or doesn’t fit into the journal in quesiton.)”

I know this isn’t your (good) main point here, but I think I’d describe this problem differently. The victims, such as they are, are not repeat readers but rather the reviewers whose reviews end up being wasted because “their” journal is not the first one to accept. Judging from the complaints of my editor friends, the ecosystem wouldn’t survive if for some reason they collectively had to find 2 or 3 times as many reviewers for each article submitted. Then add to that the fear of being moot would further discourage reviewers from participating.

By: The IPCC: Bogus data on "Climate Sensitivity"!!! - US Message Board - Political Discussion Forum Sat, 30 Mar 2013 00:30:53 +0000 [...] with the "peer review" nonsense. That shit is rigged and has been for centuries. Peer review or smear review? Reflections on a rigged system. : Historiann : History and sexual polit… __________________ "The sword is an emblem of Islam. But Islam was born in an environment [...]

By: From Peer Review to the Wisdom of Crowds? Open Access & Peer Review | History Workshop Fri, 21 Dec 2012 13:17:27 +0000 [...] should take very seriously voices of discontent, and proposals for improvement – for example that reviewers should sign their reviews. But I am not convinced that moving to a universal system of no or light-touch peer review would do [...]

By: Historiann Thu, 06 Sep 2012 13:41:07 +0000 Silly Wabbit: I feel your pain! This is another reason I always sign my reviews. It give the authors an opportunity (if they want it) to correspond with me about my review.

I think you’re well within your rights to correspond with the editor to ask for hir clarification, and/or to ask hir to forward your questions on to the reviewers. Then ze can leave it up to the reviewers as to whether or not they want to be in touch directly with you, or perhaps just convey the response through the editor.

But, disciplinary conventions vary–in my field this would be ok, but I would consult friends and colleagues in your field.

By: Silly Wabbit Thu, 06 Sep 2012 05:29:16 +0000 I just stumbled on this while I should be working….

I find it frustrating that peer review does not allow me to request clarification from reviewers…..I’ve had reviewers make negative comments that don’t make sense. I think there should be a system in place in which you can request for a reviewers to restate a portion of their review if it if unclear.

I’ve found it difficult to dissect a few “word salads” that I have gotten back from reviewers.

By: Multanemo Sat, 06 Feb 2010 02:42:30 +0000 Update: The article was accepted for publication with minor revisions. Not bad for a first year PhD student! I have another article out for review and I will keep everyone posted. But I have a good feeling. Thank you for this blog.

By: Hug an Editor Day: Journal of the History of Sexuality : Historiann : History and sexual politics, 1492 to the present Wed, 03 Feb 2010 22:39:18 +0000 [...] readers who must have read and responded to the article in an extremely timely fashion.  Because I’ve b!tched about the peer-review process at journals generally here, I thought I should recognize a journal that was exceptionally [...]

By: Multanemo Mon, 30 Nov 2009 22:00:00 +0000 Wow! I am currently a PhD candidate in history and have been publishing since my MA days, which were not that long ago. I started with local and regional journals first, then moved on to more prestigious and national ones. I found it easy to publish in peer-reviewed journal at the local historical society because, well, I could walk into their offices and sell the story. However, the national journals have been tougher to crack. First time I submitted to one I got a rejection in three weeks! And this was after the article had been revised at least ten times! Second time I submitted to a journal outside my home state I was told that my approach was “too sociological.”

Well, hopefully, third time will be the charm. I actually met the third editor in person and spoke with him for an hour at a conference I presented at. In addition, I had him read the article first before I formally submitted to the journal. He liked it so I gave it a shot. Finally, I had sent this article out for my own peer review process to several top scholars in my field, Jewish American history; most agreed that it was insightful and more than worthy of being published. Surprise, surprise! I emailed one of those scholars a few days ago with my latest work and he know that he wrote a “very, very positive readers report. This guy is a big deal so I hope the journal listens to him!

I will let you all know what becomes of it. But even at this very early stage in the game, and it is a game, I know not to give up.