Information about evaluation tenders for Aboriginal and Torres Strait Islander health programs is locked away by governments, according to a new research report.
A review of publicly advertised evaluation tenders over the past ten years found that only five percent of tender documents and 33 percent of evaluation reports were publicly available.
The report, An Evaluation Framework to Improve Aboriginal and Torres Strait Islander Health, makes sweeping recommendations to improve the transparency and accountability of evaluations, as well as the quality of tender processes.
Prepared for the Lowitja Institute by the University of Melbourne’s Margaret Kelaher, Joanne Luke, Angeline Ferdinand and Daniel Chamravi, the report is one of a number of new publications launched at a Community Priorities into Policies forum, convened by the Lowita Institute in Canberra today.
Follow #ResearchIntoPolicy for live tweeting of the discussions, which are being covered by UTS scholar and Croakey contributing editor Dr Megan Williams for the Croakey Conference News Service.
The report calls for tender documents, evaluation reports and responses to evaluation to be stored on a publicly accessible database, so they are accessible to the communities in which data are collected.
Reforms needed
It also calls for sweeping reforms to evaluation of Aboriginal and Torres Strait Islander health programs to ensure they better meet the needs of communities and follow the principles of ethical research.
The researchers said the failure to release evaluation reports was a frustration not only for evaluators, Aboriginal and Torres Strait Islander people, and program implementers – but also commissioners.
“The value of releasing evaluation reports was recognised by all parties,” the report said. “Although decisions not to release evaluation reports are typically made by commissioning agencies, these decisions often reflect political rather than program imperatives.
“Exceptions were cases where there were concerns about the quality of the evaluation; however, this is likely to make up a small proportion of the reports that are not released.”
The researchers said the Department of the Prime Minister and Cabinet was moving to release all evaluations in either report or summary form, but that past evaluations should also be released.
“Lack of access to information about evaluations and their findings is a significant barrier to building the evidence base in Aboriginal and Torres Strait Islander health. It also prevents evidence-based priority setting and quality assurance processes around evaluation.”
Ethical gaps
The report noted that evaluation contracts, particularly around intellectual property, are often at odds with community expectations and ethical frameworks.
“The most important finding from this review of government tenders is that there is no consistency regarding ethics requirements for evaluations involving Aboriginal and Torres Strait Islander populations. Nor is there an ethic to give Aboriginal communities a voice in the evaluation through meaningful engagement or control of the evaluation.”
The report also found that, although there were some positive examples, accepted principles for working with Aboriginal and Torres Strait Islander people are not widely or consistently integrated into programs, tender documents or program evaluations.
For example, principles of holistic concept of health, partnerships and shared responsibility, cultural respect, engagement, capacity building, accountability and governance were not well integrated into evaluations.
“It was not uncommon for a program to stipulate that its outcomes were related to holistic health but then have indicators that were largely biomedical,” the researchers reported.
The report proposes a framework for the evaluation of policies, programs and services for Aboriginal and Torres Strait Islander peoples, noting that the lack of a coherent framework has meant “a reduction in the quantity, quality, scope and use of available evidence”.
While efforts were underway to improve evaluation processes, the researchers said it was recognised that systemic change was required.
They called for tender processes to support evaluation proposals that are most likely to benefit Aboriginal and Torres Strait Islander people, and for evaluation contracts and agreements to be consistent with principles for working with Aboriginal and Torres Strait Islander people and ethical frameworks.
A directory of current evaluations should be developed, and training should be provided to specifically support Aboriginal and Torres Strait Islander leadership in evaluation, the researchers said.
The report gives several examples of positive approaches to evaluation, but notes that “the most constant criticism from Aboriginal and Torres Strait Islander communities about evaluation and other types of research is that the findings are not translated into action and thus not of benefit to communities”.
For example, many of the issues examined in the Royal Commission into the Protection and Detention of Children in the Northern Territory arose from unaddressed recommendations in the 2007 Little Children are Sacred report and the 1991 report of the Royal Commission on Aboriginal Deaths in Custody.
• Please follow the discussions at #ResearchIntoPolicy, @MegBastard, @LowitjaInstitut, @CroakeyNews, and @WePublicHealth. Publications released at the forum will be available from the Lowitja Institute’s website.
