Introduction by Croakey: The Australian Government’s new Aged Care Star Ratings system will help families make informed decisions when choosing an aged care home and keep providers accountable, according to Aged Care Minister Anika Wells.
On closer inspection, however, the system is far from meeting these objectives, and would not be out of place as a plot device in an episode of the television comedy Utopia, writes health and aged care policy analyst Charles Maskell-Knight.
“The only problem is that viewers would find it hard to believe even the Government could invent a system that awarded three stars for an ‘acceptable level of care’ to providers that the regulator found were not meeting the standards for acceptable care,” he says. “The Department needs to go back to the drawing board.”
Charles Maskell Knight writes:
One of the features of the aged care system highlighted by the Aged Care Royal Commission was the limited information available to the public to assist people seeking aged care services.
The only information about the quality of an aged care service was whether or not the service had met the quality standards the last time it was inspected by the quality regulator. Accessing that information required people to leave the MyAgedCare website and search on the Aged Care Quality and Safety Commission website.
There was no publicly available information on other features of services, such as staffing numbers or outcomes against the very limited range of clinical performance indicators. People entering a nursing home were required to hand over an interest-free loan of half a million dollars or more without any real idea whether the home was providing a quality service or not.
The Royal Commission recommended introduction of a system of star ratings to assist people in choosing a nursing home for them or their relatives. The first ratings were released in mid-December.
The following examination of the system concludes that it falls far short of being a useful tool.
Some context
In February 2021 the report from the Aged Care Royal Commission found that a third of residents in aged care homes received substandard care.
In December 2022 the Department of Health released the first results of the star rating system for aged care homes. Based on outcomes for the September 2022 quarter, 90 percent of homes were rated three stars or better. Three stars is supposed to reflect an “acceptable level of care”.
The CEO of industry association ACCPA has claimed that this result “shows that services are providing everything they are funded by the government to provide for older Australians, at a good level of quality”. He added that the result had been achieved “despite extraordinary funding and workforce challenges”.
It is axiomatic that if something sounds too good to be true, it isn’t true.
The real reason for the supposed dramatic turnaround in the quality of care is that the star rating system is deeply flawed in terms of what it measures, how it measures it, and how the results are presented.
How it works
The star ratings manual explains that performance is measured over four dimensions: resident experience (33 percent of overall rating), compliance (30 percent), staffing (22 percent), and quality measures (15 percent).
These weightings are based on the dimension’s relative importance and data maturity. One to five stars are awarded for each dimension. These are then weighted by the relevant percentage to calculate an average rating with two decimal points. This rating is then rounded to the nearest whole number – so 2.51 stars becomes three stars – as does 3.49 stars.
A rating of one star indicates “significant improvement needed”, two indicates “improvement needed”, three is an “acceptable quality of care”, four is “good”, and five is “excellent”.
Resident experience is measured with a 12-question survey administered to at least 10 percent of residents, including questions such as: do you feel safe? do you get the care you need? does it feel like home?
Responses are either: never (1 point), some of the time (2 points), most of the time (3 points), or always (4 points), meaning homes can score between 12 and 48 points. Stars are assigned based on where a home’s score falls: a score of 30 to <36 results in two stars, while 45 or more would receive five stars.
Compliance is assessed on a five-point scale having regard to the severity and currency of compliance action. Imposition of sanctions results in a one-star rating, and a Notice to Remedy (formerly notice of noncompliance) results in two stars. In both cases this ranking limits the overall star rating the home can receive. However, a home that has been free of compliance action for more than a year will receive four stars.
Staffing is assessed using a matrix of total staff care time relative to standard by registered nurse care time relative to standard. For example, a home delivering less than 90 percent of the total staff care time target, but more than 115 percent of the registered nurse (RN) care time target, would receive three stars. (Yes, that’s right, three stars – an “acceptable level of care” – for failing to meet the overall care time target!)
Quality measures is based on 10 indicators including different grades of pressure injuries, physical restraint, unplanned weight loss, falls and major injury, and polypharmacy. The methodology is complex, but essentially involves locating each home’s performance in the distribution of all homes. A home in the bottom quintile receives one star, while one in the top quintile receives five. (It should be noted this is the only element of the rating system that measures relative performance, not performance against absolute standards.)
How rankings are expressed
The first problem with the system is that the use of performance against absolute standards rather than relative performance results in a lack of differentiation.
Some 54 percent of homes are rated as three stars, and 36 percent are rated as four stars. In reality there will be substantial differences in performance between the homes at the top of the three-star group and those at the bottom, and people choosing a home for themselves or their relatives should be aware of these differences.
Consider the following two hypothetical homes.
“Acme” would receive an unrounded score of 2.67 stars, while “Bengello” would receive 3.34. Both would be rounded to three stars, even though they are clearly providing a substantially different overall quality of care.
“Bengello” is performing considerably better than “Acme”, and a slight improvement in its resident experience survey would move it from three stars to four stars, while a slight worsening in its resident experience survey would demote “Acme” to two stars.
A system of relative rankings based on grouping homes into quintiles (or even deciles) would be far more helpful to people seeking care. The 90 percent of homes currently lumped into three and four stars would be split across all five ratings, and “Acme” and “Bengello” would be appropriately differentiated.
This hypothetical example also highlights the issue of rounding the weighted scores to the nearest whole number.
If the ratings are predicated on three stars denoting an “acceptable level of care”, why should a home that receives a weighted score of less than three stars receive the benefit of rounding up? Is “Acme” really providing an “acceptable level of care” if it is providing less than the target number of total care minutes and is in the worst group of homes in terms of quality indicators?
Compliance
The way in which the system uses compliance as a rating criterion is problematic in several ways.
The Aged Care Royal Commission recommended that a star system should include “a graded assessment of service performance against standards” (recommendation 24). In other words, rather than a binary pass or fail approach, the regulator should assess performance on a sliding scale ranging from “significant improvement needed” through to “excellent”.
The first problem is that this recommendation has been ignored, and the star system instead uses the history of compliance action as the basis for rating.
The evidence presented to the Aged Care Royal Commission suggests that compliance action against a home was a sign of bad luck as well as bad quality. A home that was sanctioned was providing poor care, but the fact a home had not been sanctioned did not mean it was providing good care – it might mean that it just hadn’t been found out yet.
It was not unusual for homes to undergo an accreditation inspection and meet all standards, before failing dismally within a year after a second inspection occasioned by an egregious episode of poor care that received media attention.
Giving homes that have avoided compliance action for more than twelve months four stars (or 1.2 stars towards the overall rating after weighting) is akin to giving every child an end of year prize for simply attending school. A home that receives four stars for compliance can receive two stars against the other three dimensions and still receive an overall three-star rating.
The second serious issue is that under the system a home will receive three stars for compliance even if there is an extant direction from the Aged Care Quality and Safety Commission “to revise a plan for continuous improvement”. According to the star ratings manual the Commission “may give a service a Direction to make improvements when there are compliance issues. The service must give the Commission a plan for how they will meet all the Aged Care Quality Standards”.
Remember that three stars represents an “acceptable level of care”. It is ridiculous for the rating system to award a home three stars for compliance if the quality regulator has identified “compliance issues” and required the home to develop a plan to meet the standards.
Assuming the standards specify the requirements for delivering appropriate care, a home found by the regulator not to meet the standards cannot logically be regarded as providing an “acceptable level of care”.
Staffing
The Royal Commission recommended that homes be required to deliver 200 minutes of care per resident per day (increasing to 215 minutes), of which 40 minutes (increasing to 44 minutes) was to be provided by a registered nurse.
The rationale was that quality care requires both an adequate amount of care time to assist people in carrying out the activities of daily living AND an adequate amount of care from RNs to meet their clinical needs.
The rating system ignores this rationale, and awards stars for staffing based on a matrix which conflates total care time and RN time, and allows trade-offs between the two.
A three-star rating is awarded to a home with less than 75 percent of the RN care time target, as long as it provides 115 percent or more of the total care time target. At the other end of the spectrum, three stars are awarded to a home that provides less than 90 percent of the total care time target, as long as it has 115 percent or more of the RN care time target.
These positions are both illogical. A extra 30 minutes of total care time cannot substitute for a shortfall of more than 25 percent or 10 minutes in the RN time required to provide clinical care. And a surplus of six RN care minutes cannot make up for a shortfall in the total care time of more than 10 percent or 20 minutes.
A home that is not meeting both care time targets cannot provide an acceptable level of care, and the ratings system should not imply that it can.
When news of the star ratings was released, unions representing aged care workers were sceptical.
Annie Butler, secretary of the Australian Nursing and Midwifery Federation, said she was “very surprised” that only nine percent of homes were in the bottom two categories, as she didn’t believe the majority of homes were meeting adequate staffing levels.
Carolyn Smith from the United Workers Union said “I don’t think any aged worker would say 91 percent of providers are three stars and above.”
Given a home can receive three stars for staffing while not meeting the care time targets, the unions’ scepticism was well placed.
Work carried out by Professor Kathy Eagar for the Royal Commission found that, based on 2016-17 data, almost 60 percent of aged care residents were in homes with unacceptable staffing levels, and that raising the staffing in those homes to provide adequate three-star care would require an average increase in care staffing in those homes of 37 percent – or an increase of 20 percent in total care staffing across the sector on top of the increase required to staff additional places.
Does anybody seriously think this has happened?
What needs to be done?
The current star rating system would not be out of place as a plot device in an episode of Utopia. The only problem is that viewers would find it hard to believe even the Government could invent a system that awarded three stars for an “acceptable level of care” to providers that the regulator found were not meeting the standards for acceptable care.
The Department needs to go back to the drawing board.
If the current approach of absolute measurement is retained, compliance should be removed as a criterion until such time as the ACQSC can produce graduated measures of compliance against the standards, as recommended by the Royal Commission. The current approach of using the history of compliance action as a proxy distorts the system.
The allocation of stars for staffing needs to be revised so that three stars are only awarded to homes that meet the targets for both total care time and RN care time. A home failing to meet either target should be regarded as failing to provide acceptable care.
The weightings for the different dimensions should be revisited. While the data systems underpinning the staffing and quality measures dimensions may not yet be fully developed, these factors are crucial to quality of care and together should be worth more than 37 percent of the total rating. Their weighting can be increased if compliance is excluded from the system until a proper measure of compliance against standards is developed.
Finally, the rounding up of star scores should be dropped. A home should only receive three stars if its raw score is greater than 3.00 – not if it is 2.51.
The Department should think seriously about redeveloping the star system to adopt the relative approach to ratings it uses for quality measures under the current methodology. A system which assigns homes stars based on their relative performance, such that the best 20 percent of homes receive five stars, the next quintile four stars, and so on, will be much more useful to the public than one which places 54 percent of homes in one category.
Until the system is changed the public need to be aware that a home can still receive three stars overall for providing “acceptable care” if it delivers less than 90 percent of the total care minutes target and less than 100 percent of the RN care minutes target, and if the residents only sometimes feel safe or receive the care they need. This doesn’t sound like “acceptable care” to me.
In its current form the star rating system deserves only one star – “significant improvement needed”.
Charles Maskell-Knight PSM was a senior public servant in the Commonwealth Department of Health for over 25 years before retiring in 2021. He worked as a senior adviser to the Aged Care Royal Commission in 2019-20.
Note from Croakey: Minister Anika Wells has been asked to respond to the concerns raised by this article.
See Croakey’s archive of articles on aged care
