Creative strategies included removing the wheels from trolleys in the ED to make them count as beds, and admitting inpatients via the ED to boost the proportion of patients seen in under four hours (Walley et al. In a 2007 British Medical Association survey of emergency department staff, 31% of respondents reported that their department was manipulating data in order to meet wait time targets. Indicators are often gameableĪnother cause for concern is that many indicators are 'gameable' - that is, staff can misrepresent the data. These results suggest that at least some of the most common measures of hospital quality are of dubious validity. No significant correlations emerged among the different process measures, nor between the process and outcome measures. (2002) compared American hospitals on (a) the quality of various care processes, as assessed by the Joint Council on Accreditation of Healthcare Organizations and (b) several aggregate measures of care outcomes (e.g., adjusted mortality rate, complications). Moreover, different indicators of quality may demonstrate no relationship with one another. However, it concluded that such rates are too unreliable to draw conclusions about the quality of a particular hospital or the relative quality of two hospitals, as calculations are heavily subject to both systematic and random error. A systematic review of 18 relevant studies confirmed that on average, hospitals with exceptionally high risk-adjusted mortality rates do provide poorer care than hospitals with exceptionally low rates (Thomas and Hofer 1998). For example, risk-adjusted mortality rates (such as the Hospital Standardized Mortality Rate, or HSMR) are widely used as an index of hospital safety. An indicator's limitations may not be obviousĮven well-established indicators are sometimes revealed to have serious flaws. In one provincial Community Health Assessment (CHA) planning process, participants identified over 200 indicators through a brainstorming activity, all of which were used - without the further step of applying the above criteria - in the next CHA. Often an indicator may be used simply 'because it is there,' without consideration of its validity or robustness. Unfortunately, such guidelines are not always followed in practice. Mainz (2003) has delineated a rigorous process for developing evidence-based indicators. In reality, these criteria are difficult to achieve, and indicators, at best, are indirect or partial measures of a complex situation. As the Alberta Heritage Foundation for Medical Research (1998: 5) noted: Indicators should actually measure what they are intended to (validity) they should provide the same answer if measured by different people in similar circumstances (reliability) they should be able to measure change (sensitivity) and, they should reflect changes only in the situation concerned (specificity). Getting the Right Answers Not all indicators are created equalĭata derived from an indicator are only as good as the indicator that produced them. Our observations are based on our experience working with decision-makers within a large urban health authority's Research and Evaluation Unit.

As the Canadian Institute for Health Information (CIHI) begins to release data on hospital quality and safety, it is perhaps appropriate to stand back and consider where the emphasis on indicators is taking us. However, in our enthusiasm for quantifiable results, it is easy to overlook the limitations both of particular indicators and of indicators in general. Obviously indicators can provide valuable information. Health Canada has affirmed the value of national indicator reports in promoting informed decision-making ('allow governments … to compare data, track changes, see progress and identify areas for improvement') and enhancing public accountability (Health Canada 2006).

For example, we cannot directly measure the quality of care, but we can measure particular processes (e.g., adherence to best practice guidelines) or outcomes (e.g., number of falls) thought to be related to quality of care. An indicator is a summary statistic used to give an indication of a construct that cannot be measured directly. This trend is not unique to healthcare but parallels a resurgence of interest in social indicators and performance measurement in general (Morris 1998) as governments respond to pressure to cut costs, make evidence-based decisions and be more accountable to the public (Baker et al. The past two decades have seen a growing interest in the use of healthcare indicators to monitor and evaluate health system performance (Lilley 2000 Wait and Nolte 2005).

Health system indicators