The weird and wonderful world (or World) of peer review

I'm now getting to be quite seasoned at the process of peer review. So far this year I've managed (somehow) to submit four papers for review and I await commentary on all of them. But, to be frank, it's a very odd process. I've been joking on Twitter that I've been making sacrifices to the gods of peer review, because, well you might as well. I really do have my tongue firmly stuck in my cheek when I tell students they must use academic journal articles because of the "gold standard" of peer review. The problems are well rehearsed, especially blind peer review: many reviewers are just unhelpful and rude, hidden by the anonymity of the "blind" process; it's not blind at all and could never hope to be – I've raised concerns that I've seen papers before to editors and they've just said go ahead; and the editors have said go ahead as they struggle to get reviewers; you can submit a paper and get rejected and then resubmit it and be accepted. In the social sciences and humanities in particular, peer review also seems to be much more about giving essay feedback than going through findings with a fine-toothed comb (although the scandals of forged results and plagiarism in the natural sciences suggest the process isn't going well there either).

I've had a mix of experiences. Two of my papers accepted so far went into, or were intended for, special issues, so they had lighter-touch peer review. Of my other two papers, my difficult theory paper got the best peer-review I could ever imagine – reams of helpful comments and references to read it up on. The other paper eventually got published in Environment and Planning C after a nice wee peer-review process; and I've discussed before how helpful peer-review was in getting the difficult theory paper out. However, the EPC first got submitted to another journal and I got the worst set of comments I've ever received on anything (until just the other week). In particular this pair (edited down to make them sound particularly horrendous and preserve some vestige of anonymity):

Although this paper covers an interesting area and attempts to tell a convincing story, its failure to follow basic research standards in reporting and contextualisation means that it is not suitable for publication.It does not recognise other ABI approaches which act as benchmarks and alternatives to 'New Life'; critically, several of these pre-date this central government entry into the field. So GEAR, Area Agreements, APTs, etc. were already established along partnership lines, had spawned their own community involvements and fora and with some degree of success had been addressing the regeneration issues this paper discusses. NL was a political project, therefore, to counter the [other authorities] successes.This paper denies that history; apart from a scant few, does not offer any of the many references on … There are other errors of fact or omission.

Were one to use the standard criteria for publishing in an international Journal – clear originality of the main questions posed, a rigorously presented paper likely to appeal to an international audience etc. – this paper would be a reject.It is not:(1) clear what is original(2) why the findings of the paper are agreed around the 2 themes introduced on page 15(3) presented in a way such that the findings have been sufficiently conceptualised – most the front end has been devoted to a review of well known territory…

To add a bit of context – this was a difficult paper. I wrote it in a few evenings, in between my thesis corrections, while working full time for the Scottish Government. One of my supervisors supported my writing and helped a bit and he knew we would be asked for major corrections, but even he said it was the worst set of reviewers' comments he'd ever received. They were so vituperative, bitter and unhelpful that I actually presumed the journal had rejected it based on them. I was quite shocked when I received an email from the editorial assistant, after the paper had been accepted for EPC (in much revised form, but with the same central argument and evidence base), asking why I had not resubmitted it.

In the course of drafting this blog post, I've since received a further rejection from a journal. One of the reviewer's comments were similarly helpful to these. I was so hurt and angry by them I immediately emailed the editor back suggesting they should not have bothered forwarding it on as they were utterly unhelpful. Rather wonderfully, the editor got back very quickly suggesting:

"Reviewers have different styles and some can come across as rude as you observed. However, I think it's often because they are so pressed for time and still trying to be of service to the journal. I thought it was a bit ironic that this reviewer criticized your article for being stream of consciousness when that is what the review itself very much appeared to be. But I hope there are still valuable comments you can take from it."

For the first 18 months of my full-time academic career I managed to stay off the editors' radars for peer review, but papers now roll in about one every two months. I use this horrible experience to try and be a good reviewer. I'm even tempted to do what I know a colleague has started doing: finish the review with their name, to end the pointlessness of the anonymity. Two experiences recently have made me reflect a bit more on my role as a reviewer. Firstly, I recently reviewed a paper that was very much in my research specialism. It needed a lot of work to bring it up to publication. But, if I was horribly uncooperative and frankly evil, I could've rejected it out of hand and used the ideas it contained, with my own knowledge and reading, to produce a very similar paper. But that would have been grossly unfair, especially since it was fairly easy to work out who the authors were from the reference list and they didn't need to be treated like that. I provided about half a page of feedback and comments and a handful of references to help them on their way. In the brief format of a Twitter direct message one of the authors said (once the farce of anonymity had completely broken down): "Thanks, comments very fair & constructive. We knew it needed work but decided rather than miss the opportunity to submit and then revise it". Pretty much what I was doing with my EPC paper.

Another recent incident was when I paper was sent back to me to peer review for the second time. This time, when I logged into the journal submission website* I could see the other reviewer's comments. And, well, basically, they were a lot more detailed than mine and gave me a massive thwack of self-doubt. Am I a good reviewer?

With this in mind, and with the permission of the editor, I thought I'd post my most recent review here:

"First of all, I hope the authors take these comments in the spirit they are intended - as constructive criticism. There is a paper here (if not two or three) and the authors are clearly skilled scholars, but the paper is far off publication as it stands.I will start with two broad comments. Firstly, the paper has been submitted as a "Research Article". As a reviewer I cannot accept it as such as it is a review article. Secondly, the paper really lacks structure - the best word to describe it is "baggy". As a reader it felt like I was being bounced around like a ball in a pinball machine. The paper leapt from policy area to policy area and from theorist to theorist. The abstract, I think, is testimony to this. It could be an abstract to anything - it certainly does not inform the reader as to what to expect from this paper. I certainly was not expecting the conclusion when I got there. To move forward, I think it would be really useful to sit down and get the conclusion nailed, so you know the direction you are writing towards. This means the paper can then have a very thorough edit to give it a strong focus.Some specific comments to pick up on:- the introductory section (pp1-2) makes some sweeping generalisations (some of which you contradict almost immediately) with very light referencing. It also essentially repeats the section from page 2-7 which with editing would make a better introduction - the statement on p7 of what the paper will do and focus on is excellent - keep hold of this and use it to direct your writing- you do need much more clarity on policy areas and what you mean by "science" throughout the paper. It is a bit too sweeping at the moment, and one of the most interesting things across the globe is how different policy areas are amenable to different ranges of evidence and vice versa. This would help support your point made on p7.- throughout you introduce big ideas, like "[theory in here]" and big theorists ([theorist] on p16 was particularly striking) without fully explaining the term/what the theorist says, or justifying why you are using them."

I've edited it down slightly and hidden a couple of bits to preserve anonymity, but I want to know, would this be helpful peer review for you? Am I evil and unhelpful or would you be pleased to receive this sort of feedback, particularly if you're an early career researcher?

In closing, I want to return to my recent rejection. The other very nice, worthwhile thing that the editor said in their reply email was:

"By the way, we also rate the reviews we receive for timely response and helpfulness. While we don't produce a "hall of shame,” we do present annual awards for the best reviewers for the journal. We hope this encourages reviewers to do the best jobs that they can, and also provides them with something they can put on their C.V.s to receive credit for an anonymous service."

This is not the first time I've heard of this – a colleague won a journal award for being their top reviewer. But perhaps this is a way to improve the quality of review and something more journal editors should consider? Or do we need something more radical like open peer review or move to a wiki model of academic publishing?

* I could rant about how crap manuscript central and the like are, but I'd be writing for weeks!

We are all complainers now

I'm quite interested in those little, boring bits of policy and governance that often go overlooked in the big stories of the day - like my interest in Community Planning in Scotland. One aspect of this is the public service reform agenda in Scotland. Public sector managers are all over it, but I think outside of their bubble, even politicians aren't that up to speed with it. This includes big things like the creation of the Police Scotland, where it looks like, just as the critics said, the service is being cut and standardised across Scotland. 

As aspect of this I've bumped into is the Public Service Reform (Scotland) Act 2010 (stay with me here, it will get marginally more interesting). This was mainly to do with the creation of Creative Scotland and getting rid of another few commissions. It also gave Scottish Ministers rather sweeping powers to create or destroy various public services in Scotland, but that was kept a bit quiet at the time. However, Part 8 of the Act is now coming home to roost - Scrutiny and Complaints. As you'll guess I'll be focusing on the latter bit.

I first came across this when the the housing association I am on the management committee of had to change its complaints handling procedure. Why did we have to do this. Well, if you scroll down to section 119 you'll discover that the act gives the Scottish Public Services Ombudsman power to create "Complaint Handling Procedures" for services which then have to be adopted. Last year the SPSO "consulted" with housing associations on this. By consulted, I mean they told us this was the procedure we must have. The higher education sector is now waking up to the fact the SPSO have done the same for us and so I spent 45 minutes this morning being told about it.

Basically, this complaints handling procedure is three stage - level 1 deal with it there and then; level 2 escalation to management, level 3 ombudsman. At all levels complaints have to be recorded and then reported. Where this gets interesting is the SPSO's attitude. They've been saying that they expect a lot more level 1 complaints and a lot more complaints ending up with them because of the simplified procedures. This is what a lot of people struggle to understand, but what I find particularly interesting.

The beguiling management logic behind this is the view that "complaints are the best thing we get" which I've heard said by quite a few managers in the private sector and public sector. The logic is that, if we get complaints, we know where our processes and service are not right, so we can improve it. In their attitude to the new complaints procedure the SPSO are applying this to all public services in Scotland. We are all now expected to identify and record level 1 complaints, and the SPSO will be all over us if our reporting levels are low. That their website is valuingcomplaints.org.uk says everything you need to know. As such it's another governance tool being applied in a managerial way; another facet of that suite of targets, outcomes, that are meant to make us all pull our socks up and deliver a better service. The trouble is, the SPSO does not seem to have realised the immense administrative burden on organisations, reeling from staff cuts, that this will cause.

The other interesting aspect of this is what it does to citizenship. The SPSO also expect public services to publicise our complaints procedure, using their standardised wording, and enable complaints to be taken. The three-stage procedure also short-cuts many traditional governance institutions within organisations that had a role intervening in complaints - the Council or a committee; management board; University Court etc. Complaints are now a matter of business administration to be adjudicated by the ombudsman. This is creating us a citizen-complainers. We are all expected to read our standardised complaints leaflets and immediately leap to the nearest person and complain heartily. And you'll no doubt guess what my feelings are about this. 

Beyond those questions of equity though, there is a bigger point about governance in the Foucauldian sense. Now, the SPSO are very clear on what is not a complaint - for example matters of academic judgement are not a complaint, although there may be elements of complaint in, say, a student's appeal. Fair enough. But their definition of a complaint is staggeringly broad:

'An expression of dissatisfaction by one or more individuals about the standard of service, action or lack of action by or on behalf of the Institution'

As with my views on the outcomes focus in the National Performance Framework, I do worry that this is depoliticising what should be political decision-making. The standard of service may be a political decision and to not offer it may be a political decision with implications for equity and outcomes. Reducing these to managerial procedures and processes is very dangerous. Yes, an organisation should listen to the views of service users, but our citizenship should not be based on whether we complain about something.