In a recent speech to the Institute of Public Administration Australia, Professor Ian Anderson AO highlighted the importance of high quality data for local and regional decision-making in Indigenous affairs.
Anderson, who was appointed Deputy Secretary (Indigenous Affairs) in the Department of Prime Minister and Cabinet after a long career in academia, health policy, programs and services, said:
High quality, granular data is key for local and regional decision-making. It is very difficult to build accountability without it – how can you hold someone accountable if you don’t know what’s happening?
Equally, it’s key for generating buy in from leaders in the Indigenous sector and in government – if you can show something is working and having a positive benefit, half your work is done.
And no less importantly, it’s critical for building a case that something is not working, and that the resources and effort going into it would be better redirected into something else.”
No doubt there will widespread interest in concerns raised about the reliability of a key source on Aboriginal Torres Strait Islander health, as outlined below by Alyson Wright and Dr Ray Lovett, from the National Centre for Epidemiology and Population Health at the Australian National University.
Our confidence around “point estimates” matters in Indigenous health
Alyson Wright and Ray Lovett write:
The Aboriginal and Torres Strait Islander Health Performance Framework 2017 Report is the sixth instalment of authoritative statistics for Indigenous health policies and programs.
It’s a report that aims to present national statistics over time and place on key health indicators. For policy makers, the report alleviates time spent wandering through the maze of the Australian Bureau of Statistics website.
Given the report’s potential to inform policy in the critical space of Indigenous health, it is extremely important that the information contained within the report is accurate and well presented.
There are several shortcomings in the most recent Health Performance Framework concerning its statistical analysis; perhaps the most glaring problem relates to the maps presented on page 19 of the report (reproduced below as Figures 1 and 2).
Figure 1. Smoking prevalence amongst Indigenous Australians, by Indigenous Region, 2014-15 from Aboriginal and Torres Strait Islander (Health Performance Framework, p 19)
Figure 2. Prevalence of obesity and overweight amongst Indigenous Australians, by Indigenous Region, 2012-13 (Health Performance Framework p 19)
The maps group the estimates for prevalence of smoking and overweight/obesity in Indigenous Australians by ABS Indigenous regions. The estimates are calculated using national survey data from the Aboriginal and Torres Strait Islander Social Survey 2014-15 and Health Survey 2012-13.
It appears from the maps that the darker shades represent higher rates. However, the comparison between regions is erroneous due to large confidence intervals around the point estimates.
These errors occur due to the reduced statistical power when national survey data is broken down into regional areas. Sample sizes in the Indigenous regions are too small to give reasonable regional estimates.
There are alternative techniques for presenting the prevalence results which might better highlight the lack of certainty around the point estimates. Let’s take the Queensland regions as an example.
The graph (Figure 3) below displays the smoking prevalence for Aboriginal and Torres in 2014, based on the same data as the maps. The maps show prevalence rates by Indigenous region with error bars to present the confidence intervals.
In the smoking prevalence map above, the eight Queensland regions appears to have wide variation; however, the graph below suggests that there may be little variation in smoking rates across the regions, because of the wide overlap in confidence intervals across all the regional estimates.
A line can be drawn at the 40% mark that would intersect six of the eight Queensland regions within the bands of the confidence interval. Only the population in Toowoomba-Roma can be confidently said to have a smoking prevalence below 40% and only Cape York can be claimed to have a higher prevalence.
Figure 3. Smoking prevalence in Aboriginal and Torres Strait adults 15+ by Indigenous Regions, Queensland 2014/15
Visualising data helps, and we are fond of maps to display statistical analysis.
Policy makers want information at their fingertips, including up-to-date statistics, facts and figures, and it is a necessary tool in ‘evidence-based’ decision making.
However, the worry with these maps is that they will be used for simplistic decision making that is not well informed by evidence.
For example, if one was to consider obesity prevention programs for Aboriginal and Torres Strait Islander people, a policy maker might assume that best bang for the buck is spent targeting the regions where it seems obesity rates are highest according to the visualisation in the maps.
For tobacco control, a policy maker might assume that high rates in the map are where the current Tackling Indigenous Smoking program isn’t working.
Having good national survey data that can be used at regional level is important. Given many Indigenous programs are place-based or designed around regional service delivery areas, it makes sense that routinely collected data on health and social indicators for Aboriginal and Torres Strait Islander people ought to be able to tell a story at this regional level.
Professor Ian Anderson, in his recent speech, argued “high quality, granular data is key for local and regional decision-making. It is very difficult to build accountability without it – how can you hold someone accountable if you don’t know what’s happening?”
We agree with him.
Our recent paper published in the Australian Health Review online late last month has suggested options on how national data might be improved to enhance regional estimates, including the following options:
- Increase sample sizes in national Aboriginal Torres Strait Islander social and health surveys.
- Add health questions, including a smoking question or self-assessed body mass index, into the census, or
- Release of Survey microdata (unit record data) for Indigenous region to allow calculation of small area estimates on the national survey data.
In the age of evidence-based policy, we need to be confident in the evidence we use to inform decisions. We also need to ensure our national data can help to evaluate programs at the appropriate program delivery level.
Examining the confidence intervals in point estimates will help policy makers to quickly assess the robustness of the data. Authors of the Health Performance Framework would do well to consider the uncertainty in estimates in future reports.
• Alyson Wright and Dr Ray Lovett are from the National Centre for Epidemiology and Population Health at the Australian National University.