Honest methodology

One of my favourite journal articles is this strip cartoon by Jones and Evans in ACME on creativity and the research process.

In the most recent set of reviewers comments on a paper I’ve been asked to talk more about the methodology. This will involve a lot of fancy-sounding post-hoc rationalisation. I’d like to write an honest methodology, but that would never get published. But I thought it would be fun to do. So, here goes:

Honest methodology

The scope of the review was decided through the following process. The funder hadn’t received any suitable expressions of interest for the first call, so was fishing around for someone to do the work. One of the author’s former supervisor put them in the frame. They figured it would look good on their CV and the funder needed it doing. One of the authors scrabbled together an expression of interest based on vaguely recalled stuff about the topic from their thesis literature review they did six or seven years ago.

The period for the review was chosen because the budget wasn’t very big and X years sounded vaguely enough that the authors would be able to manage the evidence and data within their limited time and budget and it would look half decent. Post-hoc rationalisations for this decision included stuff about the financial crisis, changes in government policy and the useful fact that another major review article was published in ####.

One of the authors searched key terms on Web of Science and got thousands of results and panicked that it was going to be too big a project. After a quick email to the funders they were able to chuck a load of that out as it was decided the project would be more focused. Then the researcher remembered Web of Science wasn’t very good for social science so panicked a bit more and spent the afternoon reference-chasing, sending occasional tweets, and searching specific journals. Even to this day they come across papers and think “bugger, this would’ve been really useful for that project”.  

The research team missed the deadline quite spectacularly because one of them was overwhelmed with teaching and was moving jobs and the other was on sick leave. When they did eventually submit the draft report it then went through endless iterations with the funders where they’d point out really obvious flaws or gaps and then the research team would think “FUCK why didn’t we include that?!” and panic and go away and do some research into it.

Eventually they produced a vaguely convincing narrative on the topic concerned that they were actually quite proud of.