Sources of variability
The world is unpredictable. In a particular month, a person with a low risk of falls may experience more falls than someone with a high risk. If the risk of falls has increased then this should be investigated and responded to. However, if the risk is constant and reasonable, then attention may be directed towards other areas.
The variability inherent in a process, such as the number of falls observed, is known as common cause variability. Common cause variability is also what might be described as noise, since it is not of primary interest. On the other hand, if the risk of falls had increased, then this would be considered special cause variability. You may think of this variability as a signal that something has fundamentally changed, i.e., the underlying risk has changed. Trends can also be considered a form of special cause variability. The aim of control charts is to determine whether a new observation is the result of common cause variability only or both common cause variability and special cause variability.
Introducing the control chart
Control charts are a statistical tool used in quality management for tracking the behaviour of a process. Walter Andrew Shewhart, pictured in Figure 1, described the first control chart in 1924 while working for the Western Electric Company on the manufacturing of telephone equipment (Best 2006). He is considered the father of statistical process control (SPC) and this is the reason control charts are sometimes called Shewhart charts. Since SPC was originally used in manufacturing, many of the terms are related to manufacturing, even though their application is more general. For example, non-conforming units and non-conformities are equivalent to prevalence and incidence respectively, but the former terms are used in SPC.
A process is said to be in-control if it conforms to a specified distribution and out-of-control otherwise. The region between the upper and lower limits define what observations could reasonably be the result of only common cause variability. The regions outside the limits define what observations is likely to be the result of special cause variability. Since the limits define whether to predict that a process is in-control or not, they are known as control limits. Often additional rules are used to highlight potential trends or non-randomness in the data, but these are often not as interpretable as control limits. A typical control chart with a constant sample size might look something like Figure 2. The centre line (CL) indicates the average value posited for the in-control distribution. It may be chosen empirically or based on theory. MOA Benchmarking does the former. Observation six is outside the control limits and so is unlikely to have originated from a distribution with the centre line average. This is what is meant by a process being out-of-control. In the next section I explain how the control chart determines that the highlighted point is improbable based on the centre line.
The normal distribution
The normal distribution pops up everywhere, a reason for this is that for a sufficiently large sample size the sample mean is approximately normally distributed. This holds even when the original data was not normally distributed, which is typically the case. It follows that the estimates of rates and proportions, commonly derived from quality indicators data, are approximately normally distributed. Using this fact, a range defining what are plausible observed rates or proportions can be constructed.
As visualised in Figure 3, about 68% of data from a normal distribution falls within one standard deviations (\(\sigma\)) of the mean, 95% of data falls within two standard deviations, and 99.7% of data falls within three standard deviations. A rule of thumb based on this fact is known as the 68-95-99.7 rule which gives the proportion of data that falls within one, two, and three standard deviations respectively, of the mean for a normal distribution.
From the 68-95-99.7 rule it is clear that observing a rate or proportion that is more than three standard deviations above or below the centre line is evidence that the process mean has changed, since such a result would only occur by chance three times for every 1000 observations. Recall that if the process mean has shifted, the process is considered out-of-control. While other thresholds could be used, for control charts, three standard deviations from the in-control mean is commonly chosen. One caveat is that variability in aged care data also result from a changing case-mix. Therefore, if practicable, adjustment for case-mix should be made.
Control charts in aged care
A process, in quality control, is a series of output units that describe quality in some way. For an aged care service this may include the number of falls with injury in a month, the proportion of aged care consumers receiving 9 or more medications, or the proportion of aged care consumers that experienced significant (3kg or more) unplanned/unexpected weight loss since the prior months weighing. These Quality Indicators (QIs) are available in the MOA Benchmarking standard library and are commonly measured. The use of control charts for monitoring quality in aged care resemble that of Figure 4. In the aged care setting a full cycle generally occurs once a month.
Point prevalence versus incidence rate
The distinction between a count of non-conforming units and a count of non-conformities is important for SPC since it informs the type of control chart that should be used. However, if you are more familiar with terms from epidemiology, note that non-conforming units are a measure of prevalence while non-conformities are a measure of incidence. Therefore, by determining whether a QI is expressed as a point prevalence or incidence rate you know which control chart is appropriate as well. The meaning of a point prevalence or incidence rate is shown in Table 1.
| Point prevalence | Incidence rate | |
|---|---|---|
| Description | Proportion of people in the population that have a characteristic of interest at a point in time. | Rate of occurrence of an event of interest over a period of time. |
| Example from aged care | Number of consumers receiving 9 or more medications. | The number of falls with injury in a home over a month. |
| Typically expressed as | Proportion of beds (%) | Rate per 1000 bed-days |
| Control chart type | p-chart | u-chart |
For an incidence rate, your expectation of the number of events doubles if the period is twice as long. For example, I would typically expect twice as many falls to occur over a year than over a six month period. This is not the case for a point prevalence. This is because a point prevalence is a proportion and it also has no time dimension. Since the point prevalence is expressed as a proportion it allows for some visualisations that are not possible for an incidence rate, such as the waffle plot in Figure 5, or a pie chart. An equivalent plot for incidence rates would not be possible.
Incidence proportion compared to incidence rate
- The falls indicator introduced earlier measures the number of falls observed over a monthly period.
- The unit of interest is the number of falls experienced.
- This gives rise to an incidence rate.
- The falls indicators collected as part of the National Aged Care Mandatory Quality Indicator Program (NACMQIP) measures the number of people who experienced a fall.
- The characteristic of interest is a consumer experiencing at least on fall in the quarter.
- This gives rise to an incidence proportion (or cumulative incidence).
The incidence proportion is less sensitive to outliers, but it comes at the cost of being less sensitive to changes in the underlying risk, especially for common events. Therefore, the incidence rate is preferred from a statistical point of view, especially for control charts.
Types of control charts
The type of data that typically arises in aged care quality benchmarking is known as attribute data and the most common types of charts used for this data are shown in Table 2. Charts that assume a constant risk exposure are not particularly useful in aged care quality benchmarking since the number of residents living in a service can change from month to month. Additionally, even when beds stay the same, bed-days will vary throughout the year since months have different number of days e.g., there are more days in January than February, so a slightly higher number of falls are expected in January, all else equal.
| p-chart | np-chart | u-chart | c-chart | |
|---|---|---|---|---|
| Underlying distribution | binomial | binomial | Poisson | Poisson |
| Exposure | beds | beds | bed days | bed days |
| Exposure | can vary | constant | can vary | constant |
| Plotted value | prevalence as proportion | prevalence as count | incidence as rate | incidence as count |
| Useful for aged care benchmarking | yes | rarely | yes | rarely |
The p-chart
An example of a p-chart is shown in Figure 6 which illustrates what it might look like in practice (using simulated data).
- The y-axis shows proportion of consumers that have a particular characteristic, in this case the characteristic is that the consumer experienced significant weight loss (3kg or more).
- The June observation was determined to be out-of-control based on the chart. In fact the June observation was randomly generated from a distribution with a 15% average whereas the rest were from a distribution with a 5% average.
- After finding that the process is out-of-control, the home can try to mitigate the increase by taking appropriate action, depending on the cause they identify.
- Here the number of beds is assumed to be 75 and constant, a typical number. However, if the number of beds varies then the control limits will not be entirely straight.
- The quality indicator would be measured on a particular day of each month. For each care recipient, the weight is compared with their prior measurement to classify whether they experienced significant weight loss (3kg or more). The consumers that experienced significant unplanned weight loss are then counted, which gives the quality indicator.
The u-chart
An example of a u-chart is shown in Figure 7 which illustrates what it might look like in practice (using simulated data).
- The y-axis shows the observed rate of of falls per 1000 bed-days.
- The June observation was determined to be out-of-control based on the chart. In fact the June observation was randomly generated from a distribution with a 7.5 average whereas the rest were from a distribution with a 2.5 average, per 1000 bed-days.
- The lower control limit is zero, which is often the case for incidence based quality indicators. The reason for this is that events, such as falls with injury, are relatively rare, so a downward shift can’t attain statistical significance. Fortunately, increases are usually more important and these can be detected if the increase is large enough.
- The upper control limit is not straight, even though the number of beds is assumed to be 75, since number of days in each month differs. Note that the 2nd observation (February) has a wider interval between the lower and upper control limits. This is because the number of falls was only observed over 28 days, which gives a less accurate estimate of the rate than if the month had 31 days. This point may seem academic, but it can make a difference.
Conclusion
In aged care quality benchmarking control charts are used to detect changes in risk as measured by quality indicators. The use of control charts in aged care quality benchmarking is motivated by the fact that there is inherent variability in quality indicators, some of which does not reflect a true change in risk. The normal distribution was introduced to provide a justification for the position of control limits. Next, I explained how measures of prevalence and incidence are applied to statistical process control. Finally, I showed what a p-chart and u-chart in the aged care setting may look like and highlighted their key features.
References
Best, M. 2006. “Walter A Shewhart, 1924, and the Hawthorne Factory.” Quality and Safety in Health Care 15 (2): 142–43. https://doi.org/10.1136/qshc.2006.018093.