Mark Ragg and Megan Williams write:
Health and social programs developed by and aimed at improving the lives of Aboriginal and Torres Strait Islander people have been credited with bringing about significant change for participants – for examples, see the Healing Foundation, Bulman & Hayes, and Tsey and colleagues.
But we can’t really know of the success of programs unless they are evaluated. Evaluation is an essential part of the cycle of quality improvement in policies and programs – it can lead to deep understanding of what happens, and perhaps why, and hopefully how they could be improved, and sometimes why they need to be maintained exactly as is.
However, many Aboriginal and Torres Strait Islander health and social programs are not evaluated. A 2017 analysis found that only 88 of 1082 such programs (8%) had been evaluated or were under evaluation. Leadership by Aboriginal and Torres Strait Islander people in research and evaluation has the potential to shift this dynamic.
In 2018 Romlie Mokak, a Djugun man and a member of the Yawuru people, was appointed a full-time Commissioner with the Productivity Commission. He had previously been CEO of the Lowitja Institute, and before that CEO of the Australian Indigenous Doctors’ Association.
The next year, the Australian Government asked the Productivity Commission to develop “a whole-of-government evaluation strategy for policies and programs affecting Indigenous Australians, to be used by all Australian Government agencies.” The Commission will also review the performance of agencies against the strategy over time.
After an issues paper released in June 2019, the Commission took 112 submissions into account and released its draft evaluation strategy.
The draft strategy says the overall objective of the Indigenous Evaluation Strategy is to improve the lives of Aboriginal and Torres Strait Islander people by having policy and program decisions informed by high quality and relevant evaluation evidence. It can do this by:
- improving the quality of evaluations of policies and programs affecting Aboriginal and Torres Strait Islander people
- supporting a culture of evaluation and building a body of evidence and data on the effectiveness of policies and programs
- promoting a whole-of-government approach to priority setting and evaluation of policies and programs affecting Aboriginal and Torres Strait Islander people.
The Commission imaged the process as below.
Guiding principles
The overarching principle of the Productivity Commission’s draft strategy involves putting Aboriginal and Torres Strait Islander people, perspectives, priorities and knowledges at the centre of all relevant evaluations of policies and programs.
The Productivity Commission says its strategy means Aboriginal and Torres Strait Islander people have to be involved in making decisions about which policies and programs are evaluated, and how. It means evaluators need skills and experience to be able to work in genuine partnership with Aboriginal and Torres Strait Islander people. It also means evaluation design and reporting needs to reflect the diversity of Aboriginal and Torres Strait Islander people’s priorities and experiences, and that evaluation teams need to incorporate Aboriginal and Torres Strait Islander knowledges.
The Productivity Commission’s other guiding principles are that evaluations need to be:
- credible – “evaluation approaches, methods and processes must be credible if policy and program design and implementation decisions are to be based on evaluation findings. Evaluations should be conducted by evaluators who are technically and culturally capable.”
- useful – “evaluations that do not provide useful results are a waste of resources. When Australian Government agencies plan, commission or conduct an evaluation, the intention should always be to use the evaluation’s findings to inform policy and program decisions.”
- ethical – “all stages of evaluation, from planning, to commissioning, to conduct, to reporting, to use, should be conducted in an ethical way. Applying ethical standards improves the quality and consistency of evaluation and ensures that evaluation has a positive impact on Aboriginal and Torres Strait Islander people.”
- transparent – “transparency increases accountability of agencies and government to the community. It also allows evaluation users to judge the credibility and rigour of evaluation techniques used, and provides incentives for agencies to commission and conduct high quality evaluations.”
Who does the strategy apply to?
Formally, the Indigenous Evaluation Strategy will apply to “Australian Government agencies when they are selecting, planning, commissioning, conducting and using evaluation.”
But informally, the Productivity Commission expects that it will have an impact on everyone involved in the evaluation of Australian Government policies and programs affecting Aboriginal and Torres Strait Islander people, including:
- people and communities who take part in evaluations
- service providers who deliver policies and programs being evaluated
- external evaluators
- peak bodies and community representatives who may contribute to evaluation planning and design or provide input to evaluations
- all involved in the development, evaluation and management of policies and programs
- state, territory and local governments, given the enmeshment of different levels of government.
Office of Indigenous Policy Evaluation
The strategy proposes that an Office of Indigenous Policy Evaluation be established. This office would oversee the implementation of the Strategy and coordinate the proposed whole-of-government approach to evaluating policies and programs.
The office would sit within an existing independent statutory authority, the Commission says. It seems the National Indigenous Australians Agency is the most likely candidate, given its interest in the Indigenous Evaluation Strategy.
The office would have the vital role of identifying evaluation priorities for policies and programs affecting Aboriginal and Torres Strait Islander people. Presumably, hopefully, that would entail broad consultation.
It is essential that the majority of staff, including all senior management, are Aboriginal and Torres Strait Islander people.
The proposed Office of Indigenous Policy Evaluation would be overseen by an Indigenous Evaluation Council (the Council), with a majority of Aboriginal and Torres Strait Islander members.
Anybody wanting to make a submission needs to do so by 3 August 2020.
Dr Mark Ragg is an Adjunct Fellow at Girra Maa, the Indigenous Health Discipline, UTS and director of Ragg & Co. Associate Professor Megan Williams is Research Lead and Assistant Director, National Centre for Cultural Competence, University of Sydney. Together they recently evaluated Legal Aid NSW’s Civil Law Service for Aboriginal Communities using the Indigenous evaluation framework Ngaa-bi-nya, developed by Megan.
