HISTORY

Codman ² is credited with introducing the concept of outcome assessment in the early part of the 20th century. He espoused the concept of the “end result idea” wherein the clinician critically evaluates the results of treating all surgical cases in order to identify and understand treatment failures so that the care of future patients could be improved.

Until recently, outcome following shoulder reconstruction was usually assessed in terms of treatment efficacy. Observer-based measures such as range of motion, stability, and deformity, together with radiographic assessment of fracture union or prosthesis alignment were used to determine whether or not a procedure had been effective. Such observer-based methodology ignores the consumer of the product when assessing its effectiveness and does not address the larger concept of patient health and the role of the shoulder disorder in affecting the general well-being of the individual.

Scientific analysis has demonstrated that observer-based assessment of shoulder function can be inconsistent. Hayes and coworkers ³ found visual estimation, goniometry, still photography, and reach for six different shoulder motions to have standard errors of measurement of between 14 and 25 degrees between raters and 11 and 23 degrees with the same rater on different occasions. Ostor and colleagues⁴ determined the inter-rater reproducibility of clinical tests for rotator cuff function. Fair concordance between assessments was found in 40 of 55 observations, and moderate concordance was found in only 21 of 55 instances. Tzannes and coauthors⁵ found some tests of shoulder instability to be reliable among examiners, providing pain was not used as a criterion for a positive test. However, their study involved a small group of 13 patients who were preselected by referral to a shoulder specialist with complaints of instability. Terwee and colleagues⁶ found interobserver agreement to be low for assessing active and passive elevation of the shoulder, especially for patients with high pain severity and disability. Rudiger and coworkers⁷ found that even “objective” measurements of shoulder mobility by patients and surgeons correlated poorly in a prospective study. Dowrick and colleagues⁸ compared self-reported and independently observed disability in an orthopaedic trauma population. They found that disability level ratings varied greatly and that observers consistently rated the disability levels lower than participants. Hickey and associates ⁹ found that experienced manipulative physiotherapists had difficulty determining the symptomatic status of patients using observational motion analysis.

The comparison of the results of various health care interventions in different fields of medicine is hampered by the absence of a common measurement instrument. Patient-based outcome assessment tools are widely accepted for assessing general health. General health outcome measures have also been used to determine the effect of shoulder conditions on patient well-being, and patient-based outcome instruments have been developed for the shoulder and the upper extremity as a whole. Cook and colleagues ¹⁰ assessed four shoulder outcome measures and found them to exhibit good internal consistency across surgical status, in contrast to the in-consistency observed when observer-based systems are assessed.

DEVELOPMENT OF OUTCOME MEASURES

The outcome of treatment is best viewed through the paradigm of a disablement scheme. Disablement is a global term that reflects all of the diverse consequences that disease, injury, or congenital abnormalities can have on human function at many different levels. Pathologic conditions can lead to impairment, which is an anatomic or physiologic abnormality or loss that may be the direct result or secondary result of pathology. Outcome measures are developed through a process involving identifying a specific patient population, generating items (questions), reducing items (questions), pretesting the outcome instrument, and determining the instrument’s measurement properties (validity, reliability, and responsiveness).

TYPES OF OUTCOME MEASURES

The decision about which outcome measure to use should be based on many factors, including the population being studied, the purpose of the assessment (routine assessment vs. clinical research), training required, time required for administration and scoring, and availability of normative data. Generic health instruments, joint-specific instruments, limb-specific instruments, and disease-specific instruments are available for outcome assessment in the shoulder. Beaton and Schemitsch ¹¹ have reviewed the development of measures of health-related quality-of-life and physical-function instruments, noting the development of many outcome measures in the 1990s and reports of their reliability, validity, and responsiveness (see later).

Kirkley and coworkers ¹² developed a disease-specific quality-of-life measurement tool for patients with shoulder instability. The steps included identification of a specific patient population, item generation, item reduction and pretesting. The final instrument had 21 items and demonstrated validity, reliability, and responsiveness. The instrument’s responsiveness compared favorably to five other shoulder outcome instruments, a general health outcome measure and measurement of range of motion. The authors suggested that their instrument be used as a primary outcome measure in patients with shoulder instability. In a similar manner Watson and colleagues^12a developed a disease-specific questionnaire to assess outcome of glenohumeral instability termed the Melbourne Instability Shoulder Scale.

Kirkley and coauthors ¹³ also developed disease-specific outcome instrument for patients with rotator cuff disorders (Western Ontario Rotator Cuff index). This instrument has been translated into Turkish, and the instrument has been shown to be valid and reliable when used in this context.¹⁴ Kirkley’s group¹⁵ also developed an outcome measure specifically for patients with glenohumeral osteoarthritis (the Western Ontario Osteoarthritis of the Shoulder Index).

APPLICATION OF OUTCOME MEASURES

The lack of a widely accepted outcome measure can lead to confusion when one attempts to determine the severity of an impairment. Patient-completed functional questionnaires have been developed and tested by psychometric and clinometric methods. When an observer questions a patient with regard to function and then records the response, the possibility of observer bias is introduced. Patient-completed questionnaires can be answered by telephone and mail, do not require a physical examination, and can be used to derive raw scores rather than categorical rankings.

Administration of an Outcome Measure

The response methodology for outcome instruments can include yes or no answers, Likert scales, or visual analogue scales. Visual analogue scales take longer to score than other response methodologies if the value of the patients’ response must be determined manually. Yes-or-no answers limit the number of possible responses and can also limit the instrument’s responsiveness. Many outcome instruments use Likert scales, although the ideal number of response options for a Likert scale has not been determined.

Assessment of Pain

Some investigators question including pain and function in a total score. Because pain is an impairment, some feel that it should not be included to generate a score of functional limitations. This issue can be addressed by scoring and reporting pain and functional limitations separately. Because a clinical result is important to the clinician, the outcome of therapies designed for treating the shoulder should ideally be described on the basis of a separate patient-derived assessment of function, an assessment of pain, and a clinical examination.

Categorical Rankings Versus Aggregate Scores

A review of the literature reveals a plethora of different definitions for categorical rankings that have been used to describe the outcome after operations on the shoulder. The use of categorical ranking might lead clinicians to mistakenly assume that the categorical rankings that have been used in independently designed scoring systems describe similar levels of impairment. The variable definition of terms for categorical rankings hinders accurate communication between investigators and is an impediment to the objective comparison of the results of different studies. For example, the developers of different scoring systems have assigned different weights to each domain and different ranges of values to each categorical ranking. The outcome measures currently in general use for the shoulder are scoring systems based on the assessment domains such as range of motion, pain, and ability to perform daily activities, which are scored separately. Scores are then aggregated and assigned categorical rankings that range from excellent to poor. It is possible that the same patient can have different raw scores and different (or similar) categorical rankings, depending on which scoring system is used.

Scoring systems based on categorical rankings compartmentalize results into several categories or domains that clinicians use in decision making. Such scoring systems may be more appealing to clinicians than other outcome measures such as patient-completed functional questionnaires. However, the admixture of clinical and functional criteria can create a confusing array of variables, and questioning of a patient by an observer introduces the possibility of observer bias during the collection and interpretation of data. The differences among scoring systems are so pervasive that categorical rankings cannot be relied upon to provide meaningful comparisons either with the same cohort of patients or between cohorts. The results of studies that are based on categorical rankings of different scoring systems cannot be compared or combined. Patient-completed functional questionnaires can be valid and reliable instruments for assessing the shoulder and are not limited by observer bias.

ASSESSING OUTCOME MEASURES

A wide variety of outcome measures are available (Box 7-1). Because most clinicians use an outcome measure as an initial assessment tool and as a method of determining a patient’s progress over time, the initial selection of an appropriate outcome measure is important. It is necessary to assess the measurement properties of different instruments to be certain that the outcome is being appraised accurately. Outcome measurements have varying strengths depending on the population being assessed or the reason for using the instrument. An instrument for an outcome measure must be context specific, and selection should be based on evidence that the instrument has the necessary measurement properties in the population being sampled for a study or assessment. The quality of an outcome measure can be assessed objectively, and given the plethora of outcome measures that have been developed, it is advisable to use an outcome measure for which there are data on its measurement properties. The measurement properties of an outcome measure that are important to clinicians are validity, reliability, and responsiveness.

BOX 7-1 Types of Outcome Instruments to Evaluate the Shoulder