The mission of trauma centers is to prevent death and optimize the recovery of injured patients. The performance of trauma centers is therefore of critical interest to patients and their families, and increasingly to those who pay for trauma care as well: insurers and the government.¹ But the evaluation of trauma center performance requires some measure or measures of performance, and such metrics have proved elusive. In part, this is because there are so many possible (and possibly conflicting) outcomes upon which a trauma center might be evaluated. The underlying heterogeneity of trauma patients further compounds the problem because we must always adjust our expectations for an individual patient predicated on his potential for a good outcome. Importantly, the required adjustment will depend on the outcome under consideration.

Perhaps because of these underlying difficulties the measurement of trauma center performance is still in its infancy. Tellingly, although the mostbasic and unequivocal outcome measure, survival, has been extensively examined over the last 25 years there is not yet agreement on the best model for this outcome: Injury Severity Score (ISS), International Classification of Disease Injury Severity Score (ICISS), Trauma and Injury Severity Score (TRISS), and A Severity Characterization Of Trauma (ASCOT) all have their advocates and still other survival models are likely to emerge. (See appendix for a brief discussion of these measures.) Early work suggests that the choice of outcome prediction model affects conclusions about trauma center performance,² and therefore the choice of scoring system is likely to be of considerable interest to all concerned, particularly if trauma center certification or reimbursement becomes contingent upon such measures.

If we consider outcomes for individual patients, we might think of survival as of primary interest, but the degree of residual disability that patients experience may be of even greater importance in the case of injuries that are rarely fatal. Many other outcomes are also of interest: the efficiency with which care is rendered (in terms of length of stay or cost), the effectiveness of care (complication rates, readmission rates, accuracy of medical decision making), satisfaction with care (patients, families, referring physicians), all are important. Developing risk-adjusted measures for any of these outcomes is likely to prove at least as challenging as finding a single risk-adjusted measure for survival to discharge has. Moreover, because the risk factors for different outcomes are likely to be different, every outcome of interest will require its own risk adjustment model. Some thoughtful authors have gone so far as to conclude that “outcome is neither a sensitive nor a specific marker for quality of care” and should be used only “…to help organizations detect trends and spot outliers…”.³ It is sobering to observe that to date we do not have a single risk-adjusted trauma outcome measure that is universally accepted.

Even if performance measures could be defined and measured, methodological difficulties remain. For these measures to be useful metrics of performance, they must be used to make comparisons, either within a single institution over time or between institutions. Deciding if two institutions with different patient mixes differ significantly on a given outcome measure is not trivial, however, as is evidenced by the number of statistical approaches that have been advocated. Although the simple “Z, W, and M statistics”⁴,⁵ approach to comparing institutional mortality rates of 20 years ago
never achieved mathematical respectability, subsequently many other methods have been explored: cumulative sum charts,⁶ hierarchal models,⁷ Bayesian methods,⁸ propensity scoring,⁹ and matching algorithms¹⁰ have all been proposed, and other methods are likely to emerge.

Faced with the enormity of the task of “measuring performance” in terms of numbers of possible outcomes to be measured, the effort required to develop even a single such measure, and the statistical complexities of comparing institutions based on such measures, organizations such as the American College of Surgeons (ACS)¹¹ have wisely eschewed performance measures in favor of measures of structure and process. This approach, first outlined by Donabedian 25 years ago,¹² advocates the evaluation of structures that are believed necessary for excellent care (physical facilities, qualified practitioners, training programs, etc.) and processes that are believed conducive to excellent care (prompt availability of practitioners, expeditious operating room access, patients who underwent postsplenectomy receiving overwhelming postsplenectomy infection [OPSI] vaccines, etc.) Although outcome measures were also included in Donabedian’s schema, he recognized that these would be the most difficult to develop and employ.

Although it might seem that measures of structure and process would be straightforward to posit and use, in practice difficulties immediately arise. The selection of structural characteristics requires that we know which aspects of a trauma center’s design affect performance, and here intuition may mislead us. An instructive example of this was the suggestion of “case volume” as structural criteria for trauma centers. Although case volume has been shown to be associated with outcomes for some high-risk surgical procedures, when this criteria was examined for trauma centers it was unclear how “case volume” should be defined (all trauma admissions?, all “major” trauma cases?, “major” trauma cases per trauma surgeon?, etc.) or what actual volume of cases was adequate. Indeed, it is not even clear that there is an association between volume and outcome in trauma care.¹³ Process measures are subject to these same difficulties with definition and implementation.

An entirely separate question is: “If accurate measures of trauma center performance were developed, how might such measures be employed?” One possible use would be to carefully examine centers with the best performance to determine which characteristics and practices lead to superior outcomes. This approach has been fruitfully employed by the Northern New England (NNE) Cardiovascular Disease Study, where a 24% reduction in hospital mortality rates was observed,¹⁴ and by the National Surgical Quality Improvement Program¹⁵ where a similar decrease in mortality in Veterans Administration Hospitals was described. Other approaches are possible, however. Insurers might simply require trauma centers to attain some benchmark before agreeing to pay for services. Government regulators might follow the “No Child Left Behind” model, and require that all trauma centers meet defined standards or face funding sanctions or perhaps state control. Although the experimental rigor of the NNE approach has been questioned and the unintended consequences of the latter two approaches are difficult to envision, it seems certain that, if predicated on inaccurate or unfair outcome measures, the result of performance evaluation is likely to be worse, not better, care of the injured. It is therefore imperative that any outcome measures be reliable before they are employed.

In this essay we will expand briefly on the philosophic and technical difficulties involved in measuring performance. We will find that measures of structure and process are much better developed at this time, but are not without serious problems in their application. We will then briefly describe how trauma centers are currently evaluated by the American College of Surgeons Trauma Center Verification process. In the end, we will conclude that the perceived need for measures of performance currently outstrips our ability to provide them. Indeed, it is uncertain if such measures can ever be developed.