Evaluating public health interventions: an urban regeneration case study

How do we evaluate complex health interventions? It is not a new question, as our understanding of the complex nature of health and its determinants has grown, so too has the search for an evidence-based approach. Does a focus on housing and environment work to improve health? How will we know? 

Thankfully a new study in Glasgow (home to the lowest life expectancy in the UK) is tackling not only the social determinants of heath but also the challenge of measuring the impacts of intervention.

In this third article in a series covering the Centre of Excellence in Intervention and Prevention Science (CEIPS) seminars, Rebecca Zosel shares with us the GoWell program. 

Rebecca Zosel writes:

The third of the CEIPS Seminar Series was delivered by Dr Lyndal Bond, Principal Research Officer on Tuesday 26 May: ‘GoWell: the challenges of evaluating urban regeneration as a population health intervention’. The seminar explored the challenges of evaluating complex public health interventions, using an urban regeneration program in Glasglow, Scotland as a case study.

Despite our understanding that the environments where we live, work and play impacts our health, surprisingly, there is little robust evidence to test the policy assumption that housing-led regeneration improves health and wellbeing and reduces health inequalities. This limited evidence base is partly due to the challenges of evaluating regeneration programs.

Launched in 2005, GoWell is a longitudinal research and learning program investigating the impacts of investment in housing and neighbourhood regeneration in Glasgow on the health and wellbeing of individuals, families and communities. GoWell seeks to improve the evidence base by telling us which kinds of improvements lead to the biggest benefits in health and wellbeing, and which may cause unexpected problems; what initiatives have been found to improve health; and how people’s lives are affected by regeneration initiatives (i.e. the ways in which health behaviours and potentially life-enhancing activities such as seeking employment or engaging in voluntary activities are affected by housing and neighbourhood improvements). As part of the evaluation, GoWell will monitor the effects of regeneration policy on area-based health and social inequalities across Glasgow.(1)

The regeneration program is being undertaken across the city of Glasgow, a city known as ‘the sick man of Europe’. GoWell focuses on the 15 most deprived areas with the highest rates of premature mortality in the UK. In these areas, the majority (80-90%) of the population are social renters (the equivalent to public housing), who bear a disproportionate burden of the city’s ill health and poverty. In 2003 before the regeneration program commenced, the condition of social housing in Glasgow was appalling, with 70% of homes non-compliant with Scottish Housing Quality Standards due to damp and mould, and insufficient heating.

The large-scale regeneration program involves over 75,000 homes and £1.4 billion over 10 years to be invested in:

• Housing improvements
• Transformational regeneration (demolition of high-rise blocks and replacement with lower-rise flats and houses)
• Neighborhood renewal
• Resident relocation
• Mixed tenure (mixed income) communities
• Community engagement and empowerment
• Social regeneration (community services).

Lyndal spoke of a number of challenges associated with evaluating the large scale regeneration program and of GoWell’s responses to the challenges. Two overall factors emerged from her presentation as critical to successfully evaluating complex public health interventions.

Flexible and responsive evaluation plans are critical to success

Complex public health interventions are indeed that – complex. Lyndal spoke of a situation familiar to most of us working in public health– the challenges of working with multiple partners within a dynamic and ever-changing context. The emerging and changing nature of interventions. The reality of not having control over everything. The changing political context which influences priorities and agendas. Difficulties in defining boundaries and identifying who receives the intervention. The practical and ethical challenges of finding a control group or randomly allocating at a population level. The list goes on.

The GoWell case study was a good reminder: because of the complex, contextual and dynamic nature of public health interventions, we need evaluation plans that are flexible and responsive – with a good dose of pragmatism

A comprehensive evaluation that captures multiple methods and outcomes is required

Public health interventions are notoriously hard to evaluate, and this is especially true of interventions that tackle the social determinants of health. Like many upstream determinants of health targeted by public health, regeneration strategies are not always amenable to quantification, demonstrable causal pathways or short-term outcomes. Lyndal emphasised the need for comprehensive evaluations that are characterised by multiplicity – evaluations that capture multiple outcomes at multiple levels, use multiple methods and collect multiple data sources.

As a ten-year research and learning program, GoWell has long-term funding from several stakeholders which enables a comprehensive evaluation to be undertaken. Not all interventions are so well resourced however it is interesting to note GoWell’s multitudinous approach to evaluation.

GoWell uses primary and routine data to capture multiple outcomes at multiple levels:

Multiple evaluation methods are used by GoWell including:

• Community health and wellbeing survey (repeat cross-sectional every 2-3 years)
• Longitudinal studies of out movers and remainers
• Ecological monitoring (city and country wide changes to health)
• Environmental audits (capturing change at the neighborhood level)
• Qualitative studies
  • Study of governance, empowerment and participation
  • Experience of demolition and relocation
  • Practitioners and policy makers (mapping policy expectations of regeneration)
  • Evaluations of social regeneration activities (e.g. youth diversionary schemes, local employability schemes)

Further, GoWell data is linked with other data:

• Hospital and general practice linkage for GoWell survey participants
• Mapping crime statistics in GoWell areas
• Mapping food outlets (relating to snacking/diet)
• Education outcome data (impact of regeneration and tenure mix on education outcomes?)

It’s refreshing to see an example of large-scale change with a long-term, comprehensive evaluation to accompany it. This is consistent with CEIPS’ evaluation agenda which aims to build our understanding of systems-change and how to apply the science of systems thinking to population health improvement. GoWell is building the evidence base for complex public health interventions and importantly, helping policy makers to be more explicit and realistic about what regeneration might achieve.

For more information on GoWell visit: www.gowellonline.com

Rebecca Zosel is a public health practitioner, advocate and consultant. She tweets at @rzosel.

(Originally published on CEIPS blog and reproduced here with permission)

 Other articles in this series can be found here.