I'm now getting to be quite seasoned at the process of peer review. So far this year I've managed (somehow) to submit four papers for review and I await commentary on all of them. But, to be frank, it's a very odd process. I've been joking on Twitter that I've been making sacrifices to the gods of peer review, because, well you might as well. I really do have my tongue firmly stuck in my cheek when I tell students they must use academic journal articles because of the "gold standard" of peer review. The problems are well rehearsed, especially blind peer review: many reviewers are just unhelpful and rude, hidden by the anonymity of the "blind" process; it's not blind at all and could never hope to be – I've raised concerns that I've seen papers before to editors and they've just said go ahead; and the editors have said go ahead as they struggle to get reviewers; you can submit a paper and get rejected and then resubmit it and be accepted. In the social sciences and humanities in particular, peer review also seems to be much more about giving essay feedback than going through findings with a fine-toothed comb (although the scandals of forged results and plagiarism in the natural sciences suggest the process isn't going well there either).
I've had a mix of experiences. Two of my papers accepted so far went into, or were intended for, special issues, so they had lighter-touch peer review. Of my other two papers, my difficult theory paper got the best peer-review I could ever imagine – reams of helpful comments and references to read it up on. The other paper eventually got published in Environment and Planning C after a nice wee peer-review process; and I've discussed before how helpful peer-review was in getting the difficult theory paper out. However, the EPC first got submitted to another journal and I got the worst set of comments I've ever received on anything (until just the other week). In particular this pair (edited down to make them sound particularly horrendous and preserve some vestige of anonymity):
Although this paper covers an interesting area and attempts to tell a convincing story, its failure to follow basic research standards in reporting and contextualisation means that it is not suitable for publication.It does not recognise other ABI approaches which act as benchmarks and alternatives to 'New Life'; critically, several of these pre-date this central government entry into the field. So GEAR, Area Agreements, APTs, etc. were already established along partnership lines, had spawned their own community involvements and fora and with some degree of success had been addressing the regeneration issues this paper discusses. NL was a political project, therefore, to counter the [other authorities] successes.This paper denies that history; apart from a scant few, does not offer any of the many references on … There are other errors of fact or omission.
Were one to use the standard criteria for publishing in an international Journal – clear originality of the main questions posed, a rigorously presented paper likely to appeal to an international audience etc. – this paper would be a reject.It is not:(1) clear what is original(2) why the findings of the paper are agreed around the 2 themes introduced on page 15(3) presented in a way such that the findings have been sufficiently conceptualised – most the front end has been devoted to a review of well known territory…
To add a bit of context – this was a difficult paper. I wrote it in a few evenings, in between my thesis corrections, while working full time for the Scottish Government. One of my supervisors supported my writing and helped a bit and he knew we would be asked for major corrections, but even he said it was the worst set of reviewers' comments he'd ever received. They were so vituperative, bitter and unhelpful that I actually presumed the journal had rejected it based on them. I was quite shocked when I received an email from the editorial assistant, after the paper had been accepted for EPC (in much revised form, but with the same central argument and evidence base), asking why I had not resubmitted it.
In the course of drafting this blog post, I've since received a further rejection from a journal. One of the reviewer's comments were similarly helpful to these. I was so hurt and angry by them I immediately emailed the editor back suggesting they should not have bothered forwarding it on as they were utterly unhelpful. Rather wonderfully, the editor got back very quickly suggesting:
"Reviewers have different styles and some can come across as rude as you observed. However, I think it's often because they are so pressed for time and still trying to be of service to the journal. I thought it was a bit ironic that this reviewer criticized your article for being stream of consciousness when that is what the review itself very much appeared to be. But I hope there are still valuable comments you can take from it."
For the first 18 months of my full-time academic career I managed to stay off the editors' radars for peer review, but papers now roll in about one every two months. I use this horrible experience to try and be a good reviewer. I'm even tempted to do what I know a colleague has started doing: finish the review with their name, to end the pointlessness of the anonymity. Two experiences recently have made me reflect a bit more on my role as a reviewer. Firstly, I recently reviewed a paper that was very much in my research specialism. It needed a lot of work to bring it up to publication. But, if I was horribly uncooperative and frankly evil, I could've rejected it out of hand and used the ideas it contained, with my own knowledge and reading, to produce a very similar paper. But that would have been grossly unfair, especially since it was fairly easy to work out who the authors were from the reference list and they didn't need to be treated like that. I provided about half a page of feedback and comments and a handful of references to help them on their way. In the brief format of a Twitter direct message one of the authors said (once the farce of anonymity had completely broken down): "Thanks, comments very fair & constructive. We knew it needed work but decided rather than miss the opportunity to submit and then revise it". Pretty much what I was doing with my EPC paper.
Another recent incident was when I paper was sent back to me to peer review for the second time. This time, when I logged into the journal submission website* I could see the other reviewer's comments. And, well, basically, they were a lot more detailed than mine and gave me a massive thwack of self-doubt. Am I a good reviewer?
With this in mind, and with the permission of the editor, I thought I'd post my most recent review here:
"First of all, I hope the authors take these comments in the spirit they are intended - as constructive criticism. There is a paper here (if not two or three) and the authors are clearly skilled scholars, but the paper is far off publication as it stands.I will start with two broad comments. Firstly, the paper has been submitted as a "Research Article". As a reviewer I cannot accept it as such as it is a review article. Secondly, the paper really lacks structure - the best word to describe it is "baggy". As a reader it felt like I was being bounced around like a ball in a pinball machine. The paper leapt from policy area to policy area and from theorist to theorist. The abstract, I think, is testimony to this. It could be an abstract to anything - it certainly does not inform the reader as to what to expect from this paper. I certainly was not expecting the conclusion when I got there. To move forward, I think it would be really useful to sit down and get the conclusion nailed, so you know the direction you are writing towards. This means the paper can then have a very thorough edit to give it a strong focus.Some specific comments to pick up on:- the introductory section (pp1-2) makes some sweeping generalisations (some of which you contradict almost immediately) with very light referencing. It also essentially repeats the section from page 2-7 which with editing would make a better introduction - the statement on p7 of what the paper will do and focus on is excellent - keep hold of this and use it to direct your writing- you do need much more clarity on policy areas and what you mean by "science" throughout the paper. It is a bit too sweeping at the moment, and one of the most interesting things across the globe is how different policy areas are amenable to different ranges of evidence and vice versa. This would help support your point made on p7.- throughout you introduce big ideas, like "[theory in here]" and big theorists ([theorist] on p16 was particularly striking) without fully explaining the term/what the theorist says, or justifying why you are using them."
I've edited it down slightly and hidden a couple of bits to preserve anonymity, but I want to know, would this be helpful peer review for you? Am I evil and unhelpful or would you be pleased to receive this sort of feedback, particularly if you're an early career researcher?
In closing, I want to return to my recent rejection. The other very nice, worthwhile thing that the editor said in their reply email was:
"By the way, we also rate the reviews we receive for timely response and helpfulness. While we don't produce a "hall of shame,” we do present annual awards for the best reviewers for the journal. We hope this encourages reviewers to do the best jobs that they can, and also provides them with something they can put on their C.V.s to receive credit for an anonymous service."
This is not the first time I've heard of this – a colleague won a journal award for being their top reviewer. But perhaps this is a way to improve the quality of review and something more journal editors should consider? Or do we need something more radical like open peer review or move to a wiki model of academic publishing?
* I could rant about how crap manuscript central and the like are, but I'd be writing for weeks!
No comments:
Post a Comment