Aged Care Star Ratings for Q3 FY22-23

Analysis
MOA-Benchmarking
Author

Filip Reierson

Published

August 30, 2023

Introduction

In this article I aim to provide some visualisations and analysis of the star rating extract for Q3 FY22-23, released by the Australian Government Department of Health and Aged Care (2023). I show to what extent ratings differ by factors such as organisation purpose, organisation size, state, and remoteness. I also present how star ratings have changed since Q2 FY22-23.

Overview

The number of services given each star rating in each category is summarised, for Q3 FY22-23 in Table 1 and Q2 FY22-23 in Table 2.

Table 1: Number of services that were given each star rating within each category in Q3 FY22-23.
Rating Overall Residents’ experience Compliance Staffing Quality measures
★★★★★ 56 65 1406 225 574
★★★★ 1110 885 480 225 647
★★★ 1230 1384 552 629 1025
★★ 62 178 33 945 163
6 2 6 462 77
Table 2: Number of services that were given each star rating within each category in Q2 FY22-23.
Rating Overall Residents’ experience Compliance Staffing Quality measures
★★★★★ 54 50 1373 240 579
★★★★ 967 737 476 211 612
★★★ 1358 1494 610 578 1017
★★ 115 234 59 934 161
6 5 6 574 163

The same information is also visualised as a bar chart in Figure 1.

  1. Most changes are fairly minor, but overall star ratings have tended to increase with a noticeable decrease in two star ratings.
  2. Residents’ experience results appear to be a driving factor in the increase in star ratings as the survey has now been conducted again for many services.
  3. Compliance ratings haven’t changed significantly, which isn’t very notable since these are generally modified on a rolling basis as audits are conducted.
  4. There is a reduction in one star ratings for staffing.
  5. The overall distribution of quality measures ratings does not appear to have changed much, but there is a notable reduction in one star ratings. This is likely to be due to fewer homes failing to submit data, which would otherwise automatically result in a one star rating.
Figure 1: Number of services that were given each star rating within each category in Q2 FY22-23 and Q3 FY22-23.

Table of star ratings

The following table contains the average star rating in each category grouped by providers. Each service’s individual star ratings can be viewed by expanding the provider rows. The unrounded column corresponds to the unrounded overall star rating as described in this article.

Pattern of change

Instead of just considering the overall change it may be interesting to look at how many services moved from having a low rating to a high rating and vice versa. This can indicate how high mobility is in a particular star rating category, or inversely, how strong the correlation is between subsequent quarters. If correlation is very high then that might indicate the star ratings are rarely updated or homes’ ratings don’t change much. Conversely, a low correlation may indicate that the rating is less sticky and is updated more frequently. However, a very low correlation could suggest a rating is not reliable since it is unlikely that a large proportion of aged care services changed significantly on a quarterly basis.

The count of star ratings that changed by each amount is visualised in Figure 3.

Figure 3: Change in star rating from Q2 FY22-23 to Q3 FY22-23. Positive values represents an increase.

The changes can also be visualised as a heat map, as in Figure 4, where colour represent the frequency. This chart is quite informative for exploring the patterns of star rating changes. 1. There were no homes that went from having a one star overall rating to a four or five star overall rating. 2. The diagonal is darker, representing the tendency of star ratings to stay the same. 3. The tiles above the diagonal are generally darker than below the diagonal signalling an upward trend in star ratings.

Figure 4: Heatmap of homes’ star ratings in Q2 FY22-23 and Q3 FY22-23.

The proportion of star ratings that changed from the previous quarter, for each category, is shown in Figure 5.

Figure 5: Proportion of star ratings that changed within each star rating category.

Sankey charts are a nice way to visualise the most common pathways from one quarter to another. The Sankeys for overall, residents’ experience, compliance, and quality measure star ratings are shown in Figure 6, Figure 7, Figure 8, Figure 9, and Figure 10, respectively. It is similar to the heatmap in what information can be ascertained. The star ratings for Q2 FY22-23 are represented by rectangles on the left and star ratings for Q3 FY22-23 are represented by rectangles on the right. The flows show the number of homes that went from one particular star rating to another. In a desktop browser the flows of the Sankey can be hovered over to get a count, if this doesn’t work refer to the heatmap instead.

Figure 6: Overall star rating changes from Q2 FY22-23 to Q3 FY22-23.
Figure 7: Residents’ experience star rating changes from Q2 FY22-23 to Q3 FY22-23.
Figure 8: Compliance star rating changes from Q2 FY22-23 to Q3 FY22-23.
Figure 9: Staffing star rating changes from Q2 FY22-23 to Q3 FY22-23.
Figure 10: Quality measures star rating changes from Q2 FY22-23 to Q3 FY22-23.

Explanatory factors

In this section I consider the association between various factors provided in the star rating extracts that may be associated with higher or lower ratings.

Summary

The mean of each star rating category stratified by various factors is summarised in Table 4.

Table 4: The mean of each star rating category stratified by various factors.
Group Overall RE Compliance Staffing QM N
Organisation purpose Not for profit 3.43 3.38 4.25 2.41 3.62 1505
Organisation purpose For profit 3.38 3.19 4.35 2.28 3.54 905
Organisation purpose Government 4.03 3.59 4.54 4.17 3.67 214
Organisation size Small 3.53 3.40 4.20 2.71 3.72 940
Organisation size Medium 3.46 3.29 4.33 2.61 3.53 775
Organisation size Large 3.40 3.29 4.42 2.23 3.52 909
State NSW 3.41 3.37 4.14 2.43 3.63 830
State VIC 3.53 3.24 4.44 2.69 3.66 745
State QLD 3.55 3.48 4.53 2.46 3.40 465
State WA 3.33 3.24 4.23 2.38 3.42 248
State SA 3.42 3.34 4.12 2.54 3.68 230
State ACT 3.41 3.15 4.22 2.52 3.96 27
State TAS 3.44 3.26 4.42 2.40 3.91 70
State NT 3.62 2.88 4.00 3.56 4.00 9
Modified Monash Model MM1 3.44 3.23 4.34 2.42 3.61 1649
Modified Monash Model MM2 3.44 3.38 4.36 2.40 3.48 209
Modified Monash Model MM3 3.33 3.41 4.10 2.33 3.57 229
Modified Monash Model MM4 3.44 3.45 4.24 2.66 3.47 189
Modified Monash Model MM5 3.73 3.67 4.34 3.05 3.72 311
Modified Monash Model MM6 3.67 3.33 4.36 3.32 3.54 28
Modified Monash Model MM7 3.67 3.67 4.11 3.44 3.11 9
Modified Monash Model grouped Metropolitan (MM1) 3.44 3.23 4.34 2.42 3.61 1649
Modified Monash Model grouped Regional centres (MM2) 3.44 3.38 4.36 2.40 3.48 209
Modified Monash Model grouped Rural and remote (MM3-7) 3.54 3.53 4.24 2.76 3.60 766

Organisation purpose

The number of services of each organisation type is shown in Table 5. Figure 11 shows the proportion of services that received each star rating and the average star rating along with a 95% confidence interval, stratified by organisation type and star rating category.

The 95% estimated confidence interval, within each star rating category, was calculated as \overline{x} \pm 1.96 s / \sqrt{n}, where \overline x is the sample mean, s is the sample standard deviation, and n is the sample size.

Table 5: The number of aged care services of each organisation purpose with non-missing overall star rating.
Organisation type Count
For profit 849
Government 210
Not for profit 1405
Figure 11: The proportion of services that received each star rating within each category stratified by organisation purpose and the mean star rating along with an estimated 95% confidence interval.

Organisation size

I categorised provider size by the number of unique service names for the provider name in the extract: small (1-4), medium (5-19), large (20+). This may not perfectly capture the size of organisations as there are many business structures in the aged care industry, so the variable should be considered with this limitation in mind. The number of services that are part of an organisation of each size is shown in Table 6. Figure 12 shows the proportion of services that received each star rating and the average star rating along with a 95% confidence interval, stratified by organisation size and star rating category.

Table 6: The number of aged care services of each organisation size with non-missing overall star rating.
Organisation size Count
Small 903
Medium 736
Large 825
Figure 12: The proportion of services that received each star rating within each category stratified by organisation size and the mean star rating along with an estimated 95% confidence interval.

Geography

Remoteness is described by the Modified Monash Model (MMM), which divides Australia into seven categories of remoteness from MM1 to MM7. I use MMM in its original form along with a grouping of MM1, MM2, and MM3-7. I also consider the Australian states and territories. The number of services in each group and how these relate to each other are shown in the alluvial chart in Figure 13.

Figure 13: An alluvial chart showing how the geography categories are related.

The geographical categories introduced above are visualised in an interactive map in Figure 14.

Figure 14: An interactive map showing a simplified version of the geographical boundaries of interest.

Figure 15, Figure 16, and Figure 17 show the proportion of services that received each star rating and the average star rating along with a 95% confidence interval, stratified by geographical groups. It appears that staffing ratings are higher in more remote locations. Residents’ experience ratings are also increasing from MM1 to MM5.

Figure 15: The proportion of services that received each star rating within each category stratified by States and territories and the mean star rating along with an estimated 95% confidence interval.
Figure 16: The proportion of services that received each star rating within each category stratified by the Modified Monash Model and the mean star rating along with an estimated 95% confidence interval.
Figure 17: The proportion of services that received each star rating within each category stratified by a grouped Modified Monash Model and the mean star rating along with an estimated 95% confidence interval.

Stratified trend

The average star ratings along with 95% confidence intervals stratified by the explanatory factors, organisation purpose, organisation size, and remoteness, and some states are visualised in Figure 18, Figure 19, Figure 20, and Figure 21. Corresponding t-tests for differences between Q2 FY22-23 and Q3 FY22-23 star ratings stratified by each factor are shown in Table 7, Table 8, Table 9, and Table 10.

Figure 18: Average of the star ratings, along with a 95% confidence interval, for each organisation purpose in Q2 FY22-23 and Q3 FY22-23.
Table 7: The results of performing a two sample t-test for each category of star rating and organisation purpose between Q2 FY22-23 and Q3 FY22-23.
Category Group Mean Q2 Mean Q3 Difference 95% CI p-value
Overall For profit 3.280 3.383 [0.049, 0.157] <0.001
Government 3.963 4.033 [-0.047, 0.188] 0.238
Not for profit 3.350 3.431 [0.038, 0.124] <0.001
Residents' experience For profit 3.086 3.193 [0.049, 0.165] <0.001
Government 3.581 3.590 [-0.124, 0.143] 0.885
Not for profit 3.272 3.377 [0.059, 0.152] <0.001
Compliance For profit 4.308 4.352 [-0.037, 0.126] 0.287
Government 4.458 4.537 [-0.077, 0.235] 0.320
Not for profit 4.182 4.251 [0.002, 0.136] 0.045
Staffing For profit 2.265 2.281 [-0.075, 0.108] 0.721
Government 4.092 4.173 [-0.157, 0.32] 0.504
Not for profit 2.317 2.414 [0.018, 0.177] 0.017
Quality measures For profit 3.385 3.541 [0.058, 0.253] 0.002
Government 3.771 3.668 [-0.305, 0.1] 0.320
Not for profit 3.539 3.616 [-0.002, 0.155] 0.056
Figure 19: Average of the star ratings, along with a 95% confidence interval, for each organisation size in Q2 FY22-23 and Q3 FY22-23.
Table 8: The results of performing a two sample t-test for each category of star rating and provider size between Q2 FY22-23 and Q3 FY22-23.
Category Group Mean Q2 Mean Q3 Difference 95% CI p-value
Overall Small 3.400 3.530 [0.071, 0.189] <0.001
Medium 3.394 3.458 [0, 0.128] 0.051
Large 3.341 3.402 [0.008, 0.114] 0.023
Residents' experience Small 3.288 3.402 [0.052, 0.174] <0.001
Medium 3.233 3.291 [-0.008, 0.125] 0.085
Large 3.176 3.291 [0.058, 0.174] <0.001
Compliance Small 4.138 4.196 [-0.029, 0.145] 0.192
Medium 4.259 4.330 [-0.017, 0.159] 0.115
Large 4.366 4.420 [-0.027, 0.135] 0.194
Staffing Small 2.635 2.713 [-0.036, 0.193] 0.181
Medium 2.488 2.610 [-0.001, 0.247] 0.053
Large 2.204 2.227 [-0.07, 0.116] 0.629
Quality measures Small 3.589 3.718 [0.027, 0.229] 0.013
Medium 3.461 3.531 [-0.039, 0.178] 0.209
Large 3.453 3.517 [-0.03, 0.158] 0.182
Figure 20: Average of the star ratings, along with a 95% confidence interval, for each MMM grouping in Q2 FY22-23 and Q3 FY22-23.
Table 9: The results of performing a two sample t-test for each category of star rating and MMM grouping between Q2 FY22-23 and Q3 FY22-23.
Category Group Mean Q2 Mean Q3 Difference 95% CI p-value
Overall Metropolitan (MM1) 3.334 3.436 [0.062, 0.143] <0.001
Regional centres (MM2) 3.396 3.437 [-0.073, 0.154] 0.484
Rural and remote (MM3-7) 3.470 3.536 [-0.003, 0.137] 0.062
Residents' experience Metropolitan (MM1) 3.122 3.235 [0.07, 0.156] <0.001
Regional centres (MM2) 3.267 3.377 [-0.006, 0.225] 0.063
Rural and remote (MM3-7) 3.465 3.527 [-0.007, 0.131] 0.078
Compliance Metropolitan (MM1) 4.288 4.337 [-0.012, 0.11] 0.112
Regional centres (MM2) 4.255 4.355 [-0.072, 0.273] 0.252
Rural and remote (MM3-7) 4.165 4.244 [-0.018, 0.176] 0.112
Staffing Metropolitan (MM1) 2.353 2.421 [-0.005, 0.142] 0.068
Regional centres (MM2) 2.426 2.402 [-0.278, 0.229] 0.850
Rural and remote (MM3-7) 2.667 2.758 [-0.046, 0.229] 0.194
Quality measures Metropolitan (MM1) 3.487 3.608 [0.049, 0.194] 0.001
Regional centres (MM2) 3.490 3.477 [-0.227, 0.201] 0.906
Rural and remote (MM3-7) 3.553 3.597 [-0.068, 0.156] 0.442
Figure 21: Average of the star ratings, along with a 95% confidence interval, for select states in Q2 FY22-23 and Q3 FY22-23.
Table 10: The results of performing a two sample t-test for each category of star rating and state between Q2 FY22-23 and Q3 FY22-23.
Category Group Mean Q2 Mean Q3 Difference 95% CI p-value
Overall ACT 3.185 3.407 [-0.104, 0.549] 0.178
NSW 3.319 3.410 [0.03, 0.151] 0.003
NT 3.000 3.625 [-0.078, 1.328] 0.077
QLD 3.436 3.554 [0.043, 0.192] 0.002
SA 3.317 3.423 [-0.012, 0.224] 0.079
TAS 3.429 3.439 [-0.19, 0.212] 0.915
VIC 3.491 3.532 [-0.025, 0.106] 0.221
WA 3.214 3.330 [0.011, 0.222] 0.031
Residents' experience ACT 2.889 3.148 [-0.026, 0.544] 0.073
NSW 3.284 3.368 [0.018, 0.15] 0.013
NT 2.750 2.875 [-0.599, 0.849] 0.717
QLD 3.238 3.484 [0.167, 0.324] <0.001
SA 3.258 3.344 [-0.027, 0.199] 0.136
TAS 3.229 3.258 [-0.176, 0.234] 0.780
VIC 3.201 3.238 [-0.03, 0.104] 0.281
WA 3.210 3.241 [-0.088, 0.151] 0.608
Compliance ACT 4.148 4.222 [-0.44, 0.588] 0.774
NSW 4.080 4.138 [-0.036, 0.151] 0.229
NT 3.917 4.000 [-0.737, 0.904] 0.834
QLD 4.431 4.532 [-0.001, 0.204] 0.052
SA 4.019 4.120 [-0.079, 0.281] 0.270
TAS 4.357 4.424 [-0.226, 0.36] 0.651
VIC 4.405 4.442 [-0.049, 0.124] 0.394
WA 4.196 4.235 [-0.128, 0.205] 0.649
Staffing ACT 2.333 2.519 [-0.406, 0.776] 0.532
NSW 2.304 2.433 [0.028, 0.231] 0.013
NT 2.667 3.556 [-0.46, 2.238] 0.184
QLD 2.444 2.459 [-0.135, 0.164] 0.848
SA 2.426 2.541 [-0.096, 0.326] 0.285
TAS 2.457 2.403 [-0.45, 0.341] 0.787
VIC 2.678 2.691 [-0.132, 0.157] 0.865
WA 2.311 2.385 [-0.108, 0.255] 0.426
Quality measures ACT 3.741 3.963 [-0.292, 0.737] 0.390
NSW 3.553 3.627 [-0.027, 0.176] 0.148
NT 3.167 4.000 [-0.474, 2.141] 0.198
QLD 3.452 3.401 [-0.18, 0.078] 0.437
SA 3.550 3.679 [-0.055, 0.314] 0.170
TAS 3.786 3.910 [-0.282, 0.531] 0.545
VIC 3.633 3.662 [-0.081, 0.139] 0.605
WA 2.963 3.423 [0.243, 0.676] <0.001

Unrounded overall star rating

I calculated the unrounded overall star ratings according to the Star Ratings Provider Manual (Australian Government, Department of Health and Aged Care 2022). The overall star rating is calculated according to the formula, \begin{align*}
\text{Overall star rating} &= 0.33\times \text{Resident's experience star rating}\\
&\quad +0.30\times \text{Compliance star rating} \\
&\quad +0.22\times \text{Staffing star rating} \\
&\quad +0.15\times \text{Quality measures star rating}.
\end{align*} Except when the compliance star rating is two or lower, then the overall star rating can be no greater than the compliance rating. The final step would be to round to the nearest whole number.

Analysing the star ratings prior to the final step of rounding the overall star rating to a whole number can be insightful. In some sense this quantity contains more information than the final overall star ratings. Additionally, unrounded star ratings have the property of resembling a familiar statistical distribution, the normal distribution. This is clear from Figure 22. The graph also shows that the upward shift in star ratings is fairly consistent across the distribution of star ratings. The masses at 1 and 2 star ratings is due to the “compliance rule” step of the algorithm mentioned earlier.

Figure 22: The distribution of unrounded star ratings in each quarter.

The distribution of differences between unrounded star ratings in matching records for Q2 FY22-23 and Q3 FY22-23 are visualised in Figure 23.

Figure 23: The distribution of changes in unrounded star ratings from Q2 FY22-23 to Q3 FY22-23. Positive values represents an increase.

There are many small changes in star ratings that did not result in a change in the overall star rating. This phenomenon is visualised in Figure 24. Since the overall star rating is rounded, there is a range of unrounded overall star ratings that will result in the same star rating.

Figure 24: The unrounded overall star ratings for Q2 FY22-23 compared to for Q3 FY22-23. The colour indicates whether the rounded overall rating changed.

If instead the unrounded changes are of more interest, the magnitude of changes can be visualised as in Figure 25.

Figure 25: The unrounded overall star ratings for Q2 FY22-23 compared to for Q3 FY22-23. The colour indicates the magnitude and direction of change in the unrounded overall star rating.

The unrounded star ratings are also useful for considering the effect of explanatory factors. I performed an analysis of which factors matter for star ratings in the post Multivariable Linear Model for Aged Care Star Ratings. Instead of repeating the analysis, I provide some summary statistics of the unrounded overall star rating stratified by factors considered earlier in this article. Table 11 shows the five point summary, mean, standard deviation, sample size, density plot, and box plot for each stratum.

Table 11: Summary statistics, a density plot, and a boxplot for the unrounded overall star rating stratified by various factors.
Min Q1 Median Q3 Max Mean SD N Density Boxplot
Organisation type
Not for profit 1.0 3.2 3.4 3.8 5.0 3.4 0.5 1405
For profit 1.0 3.1 3.4 3.7 4.7 3.4 0.5 848
Government 2.0 3.7 4.1 4.4 4.8 4.0 0.5 210
Organisation size
Small 1.0 3.2 3.5 3.9 5.0 3.5 0.5 903
Medium 1.0 3.2 3.4 3.8 4.8 3.5 0.5 735
Large 1.0 3.2 3.4 3.7 4.7 3.4 0.4 825
State
NSW 1.0 3.1 3.4 3.7 4.7 3.4 0.5 787
VIC 2.0 3.2 3.5 3.9 4.8 3.5 0.5 697
QLD 2.4 3.3 3.5 3.8 5.0 3.6 0.4 446
WA 1.0 3.1 3.4 3.7 4.5 3.4 0.5 224
SA 1.0 3.1 3.4 3.8 4.7 3.4 0.5 208
ACT 2.0 3.2 3.5 3.7 4.4 3.4 0.5 27
TAS 2.0 3.3 3.4 3.8 4.7 3.5 0.4 66
NT 2.8 3.2 3.6 3.8 4.1 3.5 0.4 8
Modified Monash Model
MM1 1.0 3.2 3.4 3.7 5.0 3.4 0.5 1538
MM2 2.4 3.2 3.4 3.8 4.7 3.5 0.4 197
MM3 1.0 3.1 3.4 3.7 4.7 3.4 0.5 212
MM4 1.0 3.2 3.5 3.9 4.8 3.5 0.6 183
MM5 1.0 3.4 3.8 4.1 4.8 3.7 0.6 297
MM6 2.8 3.4 3.8 3.9 4.4 3.7 0.5 27
MM7 3.0 3.2 3.5 4.2 4.7 3.7 0.6 9
Modified Monash Model grouped
Metropolitan (MM1) 1.0 3.2 3.4 3.7 5.0 3.4 0.5 1538
Regional centres (MM2) 2.4 3.2 3.4 3.8 4.7 3.5 0.4 197
Rural and remote (MM3-7) 1.0 3.2 3.5 3.9 4.8 3.6 0.6 728

Notes on data

Columns

There are new columns available that describe the star ratings in more detail, however I didn’t analyse these here. The home size column was available in Q2 FY22-23, but is not available for Q3 FY22-23, so it was not available for this analysis.

Missing data

Some star ratings are missing in the dataset. Based on the Star Ratings Provider Manual (Australian Government, Department of Health and Aged Care 2022) there are a number of reasons that a star rating may not be available such as: the service has not operated for two reporting quarters of the National Aged Care Mandatory Quality Indicator Program; the service has recently gone through a transfer in ownership; the rating for quality measures or staffing is under review by the department; unavailability of data. Furthermore, the overall star rating can generally only be determined when all other star ratings are available for that service. The number of star ratings missing in each category is shown in Table 12. The patterns of missingness are shown in Table 13. We can confirm from this table that the overall star rating is generally only available when all other star ratings are also available. However, there is one exception, which is the result of a one star compliance rating causing the overall rating to be 1, no matter the residents’ experience rating.

Table 12: The number of star ratings unavailable within each category.
Unavailable star ratings
Overall 160
Residents’ experience 110
Compliance 147
Staffing 138
Quality measures 138
Table 13: Number of services with each combination of missing star ratings.
Missing star rating for Count
Overall, Residents’ experience, Compliance, Staffing, Quality measures 94
Overall, Compliance, Staffing, Quality measures 40
Overall, Residents’ experience 9
Overall, Compliance 8
Overall, Residents’ experience, Compliance 4
Overall, Staffing, Quality measures 2
Overall, Quality measures 1
Overall, Residents’ experience, Compliance, Staffing 1
Overall, Residents’ experience, Staffing, Quality measures 1
Residents’ experience 1

Data linkage

Unfortunately, the overlap between the Q2 FY22-23 extract and Q3 FY22-23 extract is not perfect. Aged care homes can be opened, change name, transfer ownership, or be closed down. Some homes also have two records due to having a low and high care wing. If I only keep records that are directly matched in the extracts then the overall average star rating is 3.364 (instead of 3.379) for Q2 FY22-23 and 3.444 (instead of 3.466) for Q3 FY22-23. Based on this and similar sensitivity analysis among other categories (not shown), I decided to use records where the non-star rating columns matched between the data extracts, whenever linking was required. Among records in the extract for Q3 FY22-23, 97.1% had a matching record for Q2 FY22-23. However, whenever linking between the datasets was not required I used the full unmodified data extract for each quarter.

References

Australian Government, Department of Health and Aged Care. 2022. “Star Ratings Provider Manual.” https://www.health.gov.au/resources/publications/star-ratings-provider-manual?language=en.
Australian Government Department of Health and Aged Care. 2023. “Star Ratings Quarterly Data Extracts.” https://www.health.gov.au/resources/collections/star-ratings-quarterly-data-extracts.