Evaluating Co-production: Write up of London Workshop

[Adrian Bua]

Members of the London Network met at a workshop on Thursday 9^th July to discuss the theory and practice of evidencing the impact of co-production. The session was attended by over thirty people, and consisted of three speakers:

• Formerly of NEF, Juliette Hough is an independent researcher with extensive experience of conducting evaluations in the voluntary and community sector. As a researcher at NEF, she led on the evaluation of Wandsworth Community Empowerment Network's pilot co-production project working with ethnic minority groups at high risk of cardiovascular disease. In her talk she set out some unique challenges to evaluating co-produced projects, discussed methods, and talked about the lessons she learnt conducting the WCEN evaluation. 
• Johanna Wilson is a graduate trainee at Charityworks - on placement at Together for Mental Wellbeing. She discussed some of the work Together has done around co-production and evidencing its impact, as well as a few of the challenges this has raised, including her involvement in a Social Return on Investment analysis of peer support, a service user Self-Management Steering Group and the creation of an audit form around involvement and impact.
• Holly Welsh is also a graduate trainee at Charityworks who is working across the innovation and communication teams at Richmond Fellowship. Her work has focussed on co-production at a local, service level, and at organisational level. Holly talked about some of the challenges she has faced around embedding co-production at an organisational level and co-producing evaluation as a means of evidencing impact.

Following the presentations, the group engaged in a rich and lively discussion that covered a wide range of issues, and raised some interesting questions and challenges. There was general agreement that the impact of co-produced services is difficult to measure and therefore ‘gold standard’ evidence cannot be expected. Two important reasons for this are (a) that co-production aims to deliver ‘soft’ and longer term outcomes such as improving people’s skills and capabilities, and (b) that intermediate outcomes are adaptable. Much of the value of co-production is that it is a contextualised process, but means it is not amenable to standardised forms of evaluation.

These issues are important barriers to the expansion of co-production, especially considering the risk-averse nature of many commissioners. Participants felt it was important to highlight that ‘not all that counts can be counted, and not everything that can be counted counts’, as well as to highlight both the service benefits of co-production, and its ability to develop enduring social and human capital amongst service users that produces positive ‘spill-overs’ into other areas of social and civic life.

Calls for a different standard of proof do not however preclude the need to develop rigorous evaluation methods. There was some debate around whether Social Return on Investment (SROI) is the best approach to use. This is to be valued for quantifying benefits, but some were preoccupied by the very notion of putting a monetary value on social return – the question on the utility of SROI remained an open one.

Other barriers to the expansion of co-production included that its contextualised nature means that it does not provide a directly transferable model of service delivery. Each instance of co-production looks different, and this makes it a challenging process to implement. The problem of expertise and status asymmetries was also raised. In co-production, professionals share their expertise and decision making power with service users, and participants felt that some professionals are reluctant to take part because of concerns at the potential loss of status. These two problems perhaps explain why, especially in mental health services, the clinical approach remains dominant – something which was especially concerning to some of the mental health practitioners taking part in the workshop.

Finally, it was also a concern that, since co-production builds on human assets, it is communities that are better off and / or have more organisational capacity that stand to benefit the most from its expansion. This mirrors the broader concern that participatory approaches to public administration exacerbate socioeconomic inequality. This issue is specifically important given the context of austerity. Although some argued that austerity has created an appetite for innovation in public services, it is also the case that there are reduced resources for capacity building or providing other essential services - meaning that disadvantaged communities are further disempowered.

Overall, the workshop sparked great interest amongst members of the network and produced a lively debate. Some participants attended with a view to learn more about co-production, and how it might relate to their professional activities – perhaps indicating a rising tide of interest in co-produced approaches to public services.

If you have any feedback on the session, or any ideas for further sessions please get in touch with the network organiser Adrian Bua, at [e-mail].

NEF will be in touch in the near future with information on another event.