Thinking about evaluation

The project I'm PI on at the moment comes to an end in a fortnight's time. It's been all about evaluation. Wrapping it up, writing and due to a couple of other things, I've been thinking a lot about evaluation. 

The first thing that made me think, a lot, was this story from the BBC - "University study says city's 'Triple P' parenting scheme 'a failure'". Two things I find most interesting are firstly that the evaluation did find triple-p was a failure (more of that later) and secondly the reaction from the partners who commissioned the evaluation. The partners, particularly the Health Board, immediately dismissed the evaluation it seems. I would love to have been studying the whole process from the outside because I'm left with a number of questions:

• Were the partners ready and willing to hear results that were negative?
• How was the evaluation commissioned? And what project oversight was there as it ran? (why wasn't this foreseen?)
• How did the evaluation team work? Did they communicate interim findings?
• Do the partners have a learning culture where they are ready to reflect on failure, develop and move on?
• Why did they spend money commissioning an evaluation and then dismiss the results in this way? Is this an efficient use of limited resources?

Obviously, I've neither read the evaluation, or been part of this process, so these will have to be left as questions. However, I find it interesting that the Health Board are quoted as saying: "The result is an incomplete evaluation based on very limited data and conclusions that do not stand up to scrutiny" which, to me, does not suggest a learning culture at all; it suggests a culture where if you're criticised the first thing you attempt to do is undermine your critics. 

I wonder if a weakness here was the lack of a developmental approach to evaluation overall by all parties. The project I'm leading has fallen into using developmental evaluation - it's essentially just a process of testing things through a process and then stepping back and reflecting on why the worked or did not, and moving forward. It aims to create a learning context within partnerships. 

The second thing that made me reflect on evaluation was a bit of developmental evaluation happening in transport in Edinburgh. For a year, the Council have made George Street a one-way street for vehicles and made the other side of carriageway a two-way segregated cycle way with some huts outside restaurants for the smokers and people foolish enough to want to eat outside in Scottish weather.

I love the George Street cycle path. I can cycle at the speed I want to go, rather than the speed determined by being chased by a Wheeled-Black-Box-Of-Death (taxi). It has some teething issues - for practical reasons, as it's an experiment it swaps side at Frederick Street. You have to follow the path through, something which I've always found very easy. The traffic lights at Hanover Street take an age to change, but they do for everyone. The exit and egress could be a lot better - it's an island of very good cycling provision in and amongst a load of guff though, so at the moment there is nothing to connect it to. In it's pragmatism it reminds me a lot of the Danish and German cycling infrastructure that I've seen, rather than the Dutch. It's far from perfect, but it keeps you a lot safer and it's more pleasant than the road before. 

But what I like most is it is an experiment - the Council are being very open about this. They are actively seeking responses from people who use it. Already they've put in some temporary bollards to stop idiots driving down it. So far, they're doing exactly what the Triple-P parenting folk weren't - listening to criticism and adjusting things as they go along, if they can. I encourage all critiques of it (and seemingly by a controversial twitter exchange there are many) to contact the Council. The last thing I want to happen is for the Council to conclude "well, everyone thinks this is rubbish, so we might as well give up". Let's celebrate some small successes where they've happened and we'll get more and it will be better. And long may Edinburgh's experimental spirit continue. If you want to feedback, email citycentre.vision at edinburgh.gov.uk.

Which doesn't bring me back to Triple-P, but I want to add this. I'm glad the evaluation of Triple-P was so critical. It's been very popular on the Scottish policy scene along with the Family-Nurse-Partnership (FNP). These are imported policy-initiatives that are all about "early intervention" which in Scotland is very much aligned to reducing the long term "burden" on the government from "failure". It's always troubled me that these initiatives can, and do, easily fall back on cultural explanations for poverty, without setting themselves in the broader context of a very economically unequal country, where the biggest cause of poverty is low income. Also, previous evaluations (as I've mentioned on this blog before) like the first incarnation of the FNP, Glasgow's "Starting Well" programme have been similarly critical. Often it is context that makes these interventions successful, as the core of realist evaluation make clear. Finally, as the evaluations of New Start (the inspiration for Sure Start) showed, and a finding repeated by a controversial recent study by the Institute for Education and Institute for Fiscal Studies on childcare showed, the benefits of these early intervention schemes get eroded fairly quickly by the daily grind of poverty and the children end up back where they started.

I don't want to dismiss these schemes completely out-of-hand. I'm currently reading the excellent Good Times, Bad Times by John Hills (BUY THIS BOOK) and he makes the point that the UK state does some of the greatest redistribution of any state in the world; the trouble is the society going into it is also the most unequal in the world, so the society we get out at the other end is very unequal. This suggests, to me, that there is something in Ed Milliband's "predistribution" - if we can increase skills, productivity and pay in the lower end of the labour market then we are probably going to achieve a lot more than relying on the state to redistribute income and wealth. And programmes such as Triple-P and a well-funded health visitor service (as the FNP should actually be) could be part of a programme of constant, tailored, service delivery to support the most vulnerable and poorest in society. Kind of what I argue for in neighbourhood policy.

Defend USS

I teach social policy. I just wrote this blog post for my students on the VLE using the current industrial dispute as a learning point:

Why essay 3 is important in understanding why your essays might not be marked very quickly

At the moment I'm reading this book Good Times, Bad Times by John Hills. It came out last week, so I couldn't add it to the reading list, but it will be pretty much required reading for next year. It's very good indeed and very easy to read. I find reading academic texts hard work. I bought the book from the university booksellers on Saturday and I'm already half way through it which gives an indication of how easy a read it is. Chapters one, two and four would be particularly useful for the Universal Credit essay, four in particular. I've asked the university bookshop to get five in stock and they'll be there early next week in time for your essay, or you could order it from the publisher at a discount by following the above link, or from your favourite non-tax-paying internet bookseller.

The argument the book focuses in on is one that is obscured from public debates on "welfare" (see the current controversy over the way the UK Government is defining "welfare" in its "tax statements" we'll all be getting) is that much of the welfare state actually redistributes resources over time - we don't all earn the same over our life times. All your Beveridge essays on Want focused on how Beveridge wanted to make sure people with blips in their earning potential would not end up being desperately poor. John Hills' analysis unpicks in amazing detail how this is done.

The biggest transfer is between people who are in work and people who have stopped work because they are old - pensioners. They get their income from the state pension, pension tax credit, other benefits, and occupational pensions and individual pensions. At the moment my union - the University College Union - is in dispute with our employers over changes to our pension. You might have read about this. At the moment we have started an assessment boycott, so until the dispute is resolved myself and other academics in the union will not be assessing you - a shorthand for this is a "marking boycott". This has made me realise that next year I must cover pensions in next years' curriculum.

Now, to go back to pensions, they are essentially insurance against getting old. All insurance products share risk among a group of people. Going along that line of pension products, the group that shares the risk steadily shrinks. In the state pension the risk of getting old and not being able to work and therefore needing a pension income is shared among everyone in the country in the past and in the future - an almost infinite number of people. Occupational pensions are often defined benefit schemes - this means we pay into our pension with a guarantee of what income we will get out as a pensioner, which is commonly index-linked; your income will rise with inflation. These share the risk among every single member of the scheme (as employees) in the past and in the future. Therefore the amount employees and employers can pay in might vary over time as the pension scheme has to pay out more money or has increased liabilities (people who might retire in future).

Individual pensions place much more of the risk on individuals. They are defined contribution schemes - you pay in a fixed amount and this in invested for you. The value of your investment can go down as well as up - that is a risk placed on you. When you retire you then have a choice to buy an annuity with your pot which will pay out each year a fixed income that will never change. Actuaries in insurance companies work out how long you are likely to live, compare this to the rest of the people they are paying out pensions too, and work out how much they can afford to pay you without going bust. There is some pooling of risk as with all insurance products, but the risk is much more on the individual than in the state pension or organisational pensions. For example, people retiring right now are finding out that their annuities pay out very little because of the global credit crunch. They did not know the world's economy was going to crash in 2008 when they started paying into their pension 20 years ago. 

What our employers want to do is change our pension to a poorer defined benefit pension for earnings up to c. £40,000 a year (to be technical, care-average-revalued, as opposed to final salary). For earnings over c. £40,000 we will then be forced into a defined contribution scheme. As I described above, this puts the risk on us as individuals, rather than collectively all employee members and employers who are paying into the scheme. Our pensions will be dramatically reduced because of this. However, if we worked for a post-1992 university like Abertay or Edinburgh Napier, we would be in the Teachers' Pension Scheme which is defined benefit. This is while universities are making record incomes from students through the £9,000 tuition fees rUK students have to pay.

Details of the employers' proposals from Universities UK are available here. Details from the University College Union are available here.

We don't take the decision to take industrial action lightly. We all want our students to do their best. But we feel the employers are giving us no choice by forcing a very poor pension on us. We are also concerned that this will affect your education as people will choose to work at post-1992 universities, or elsewhere in the world, where there is a pension scheme. If you are angry that you are not going to be assessed, please contact the students' union and use your routes through to the NUS to put pressure on the employers to negotiate with the UCU.