Performance reporting to help organizations promote quality improvement

Boosted June 2008

The Problem: The greatest quality improvement opportunities happen at the organizational level, where performance reports are scarce

In healthcare, a great deal of time, money and energy go into producing public reports for a wide range of audiences. In Canada, this type of performance reporting – often packaged as public “report cards” – is carried out not only by governments, but also by advocacy groups, independent agencies and, in some cases, arm’s-length organizations established by governments. i All of these organizations try to present their data in a way that suits the needs of a specific audience. However, when it comes to effectively targeting groups that can actually use the data to achieve significant impacts, one audience stands out from the rest: health system managers and providers, who can interpret and apply performance data to improve the quality of care their organizations deliver. i,ii

Reporting strategies often target audiences like the general public, whose behaviour is not readily changed by the information in report cards. iii As well, report cards often provide systems-level or aggregate data that are of little use to managers or providers wanting to make sustainable improvements in individual organizations or facilities. iv,v If the goal, then, is to spur quality improvement activities and enhance quality of care, performance reports are best targeted at hospitals and managed-care organizations. i,ii

Strategy for Change

When it comes to using results to make improvements, process-of-care indicators are often more useful than outcomes indicators. i Take reporting on wait times, for example. Knowing how long people are waiting in some areas of the country compared to other areas may be helpful in gauging the state of healthcare, but it’s not altogether helpful in identifying why wait times vary and where improvement efforts need to be focused. vi In the same way, if the data are not accurately risk-adjusted, the report doesn’t allow a proper “apples-to-apples comparison.” i

Benchmarks are also useful, particularly for identifying top- and bottom-performing facilities. iv Reporting agencies can adopt and adapt the best practices from top performers while working with low performers to improve care. i In fact, this is a common practice of agencies like Cancer Care Ontario, which feeds data on wait times back to organizations across the province and works with them to make quality improvements. To promote a culture of learning, however, reporting should be carried out in a way that celebrates improvement and doesn’t lay blame or condemn individual providers for poor quality of care. vii

It is important that reporting agencies regularly consult with their stakeholders to ensure the relevance and validity of the indicators on which they report. i This is common practice for the Canadian Institute for Health Information (CIHI) – an independent, not-for-profit organization that reports annually to managers, policy makers and others on Canada’s health systems and the health of Canadians. viii, ix

What the Research Says

While report cards don’t appear to influence the healthcare decisions of patients, iii they do have some success with providers, particularly health system managers and groups of providers working in hospitals and other healthcare organizations. i It is generally agreed that most quality improvements happen at the organizational level vii, x, xi, xii and healthcare organizations have been found to be more likely than individual providers to respond to public reports. xi This may be due to the role of organizational culture, which can sometimes be at the root of quality issues. xiii

The evidence of the effectiveness of public reporting on healthcare quality comes mainly from the United States, with some evidence from the United Kingdom and Canada. i, ii, xiv Several U.S. studies that have measured improvements of quality measures have demonstrated small but important effects. xv, xvi, xvii For example, a 2005 U.S. study measuring the effect of public reporting on hospital performance in Wisconsin showed that hospitals receiving a public or private report showed statistically significant quality improvement compared to the control group that received no report. xv One important caveat is that if the data and indicators being reported are limited, reporting can provide an incomplete picture of care and lead to “gaming” of the system – the phenomenon of “what’s measured is what matters.” xviii

Meanwhile, other U.S. studies looking at whether reporting stimulates quality improvement activities have found reporting to be effective. x, xi, xix In a study of 13 hospitals in Rhode Island, researchers found that one of the results of releasing a public report was that the data were used to target new quality improvement activities, evaluate performance and monitor progress. xix A similar study found that more quality improvement activities were launched in hospitals that were reported on publicly and privately versus those that received no report at all. x

Conclusion

Public reporting is about more than mere accountability. Policymakers and reporting agencies wanting to ensure that these reports have an impact should look at the facility or the regional level as their prime target audience. Performance reports can lead to quality improvement activities, and to overall improvement in health services and outcomes, when they are directed at people involved in the delivery of care at the organizational level.