An innovative evaluation of the major federal investment in Closing the Gap – the Indigenous Chronic Disease Package (ICDP) – has highlighted the tendency of such programs to be of most benefit for higher performing health services.
There was “wide variation in the effective implementation of the ICDP at the local level”, as services with less capability struggled to make the most of the program. Researchers have highlighted “the need to find ways to support development of services in areas of greatest need”.
The evaluation involved establishing a network of 24 geographically defined ‘Sentinel Sites’ across Australia with the aim of identifying problems and innovations at an early stage of the program’s implementation, with a view to informing its refinement and ongoing implementation. These sites included community controlled and government managed Aboriginal health services and general practices.
A huge and demanding exercise, the evaluation involved over 700 interviews with key informants, 72 community focus groups with a total of 670 participants, and 41 health services provided clinical indicator data.
A key message seems to be the importance of developing trust and relationships to enable such a large undertaking to proceed efficiently, according to the report below by Professor Ross Bailie and Jodie Griffin from the Menzies School of Health Research in Darwin.
****
Dynamic evaluation informs refinement of chronic diseases program
Ross Bailie and Jodie Griffin write:
The final reports from the Sentinel Sites Evaluation (SSE) of the Indigenous Chronic Disease Package (ICDP), or more commonly known as the ‘Closing the Gap’ program in health, were recently released.
The ICDP was an unprecedented investment by the Australian Government of $805.5 million dollars from 2009 – 2013, designed to address the gap in life expectancy that is due to high rates of chronic disease among Aboriginal and Torres Strait Islander people.
The evaluation found that the ICDP delivered some notable achievements over the first three to four years of implementation.
These include:
- improved access to primary health care services and to affordable medication;
- improved orientation of the General Practice sector to the needs of Aboriginal and Torres Strait Islander people;
- significant steps towards the establishment of a new workforce that is focused on health promotion and in development of local health promotion initiatives; and
- increased attention to enhancing access to specialist, allied health and team-based approaches to chronic illness care.
One aspect of the ICDP that has implications for health policy and program implementation more generally is innovation in the approach to evaluation.
The establishment of a national network of 24 ‘Sentinel Sites’ across Australia for the purpose of identifying problems, difficulties or innovations at an early stage of implementation, with a view to informing the refinement and ongoing implementation of a major national health program, appears to be unparalleled in Australia, and possibly internationally.
There were some significant challenges to this evaluation approach, including negotiation of agreements for participation of local stakeholder organisations within very tight timeframes, and establishing appropriate processes for ethical oversight of the evaluation in a situation were there were strong differences of opinion regarding these requirements.
The successful establishment of the sites was largely attributable to careful and respectful consultation processes and the goodwill and interest of local site stakeholders, including Aboriginal community controlled organisations.
The definition of site boundaries proved challenging, because Australian Bureau of Statistics boundaries for population census purposes often do not match the boundaries used for health administrative data or with service catchment areas.
Nevertheless, use of consistent boundaries in successive evaluation cycles enabled comparison of trends over time and comparison of trends between sites.
Timely feedback makes a difference
The tight timeframes of six-monthly data collection, analysis, interpretation and feedback to the Commonwealth Department of Health and Ageing (the Department) and to local stakeholders was demanding on the evaluation team. These tight timeframes were also clearly demanding for the Department, where limitations in the capability of data systems and workforce capacity made it difficult to deliver quality data.
However, the regular feedback to local stakeholders was important in maintaining engagement in the evaluation, and it was clear they valued this feedback.
The repeated visits by the evaluation team and engagement with local service providers and managers also had a significant impact at the local site level in increasing awareness and understanding of the ICDP.
The evaluation thus had an important role in supporting implementation within the Sentinel Sites.
The development of good working relationships, trust, and efficient processes were vital to enabling the evaluation to meet its objectives – for the evaluation team working with local site stakeholders as well as within the Department.
The scope of the package required engagement across seven divisions within the Department, with the Office for Aboriginal and Torres Strait Islander Health (now the Indigenous Health Services Delivery Division) having the key coordination role. Different managers had different roles with respect to policy, program implementation and evaluation.
Progress with implementation of some measures was slower than anticipated, presenting some challenges for coordination and evaluation of the overall ICDP. As relationships were developed, there was a substantial improvement in the quality of engagement over the course of the evaluation, with increasing recognition of the significant opportunities to learn from the evaluation process.
The ability to analyse trends over time and to make comparison between sites underpinned a vitally important general finding of the evaluation – that of the wide variation between sites in uptake of various components of the package.
Repeated cycles of data collection, with in-depth exploration with local stakeholders and program managers of reasons for differences between sites, highlighted the wide variation in the capability of different service organisations to use the resources available through the ICDP to provide better quality health care.
The important consequence of different levels of capability is that service organisations with more capability have done relatively well through the ICDP, while those with less capability have continued to struggle.
A major consideration flowing from the evaluation is the need to find ways to support development of services in areas of greatest need.
While it is not clear to what extent the SSE was directly responsible for refinements of the ICDP and the on-going implementation processes, it is clear that various elements of the package have been refined in a way that is consistent with findings presented in interim evaluation reports.
It will be important to continue to monitor implementation across Australia more generally, to interpret the data from the Sentinel Sites in the light of their distinctive nature, and to consider how the positive impact of the SSE in raising awareness and understanding of the ICDP in the Sentinel Sites can be extended to Australia more widely.
The SSE provides a general model of evaluation that appears to have potential to make a significant contribution to the effectiveness of health and other programs, and should be considered for application more broadly in health and other sectors.
A stronger orientation to learning and improvement in the use of evaluation and routine program and administrative data by all stakeholders will add to potential for effective program refinement and implementation.
• Acknowledgement: The SSE was conceived and funded by the Commonwealth Department of Health and Ageing. Successful conduct of the SSE was made possible through the active support and commitment of key stakeholder organisations, community members, individuals who participated in the evaluation, and the contributions made by the SSE project team and the Department staff.
