Screening tests have three possible applications:

One example of a screening test is the Test of Orientation for Rehabilitation Patients, administered to individuals who are confused or disoriented secondary to traumatic brain injury, cerebrovascular accident, seizure disorder, brain tumor, or other neurologic events (29, 30, 31, 32). This test screens for orientation to person and personal situation, place, time, schedule, and temporal continuity. Another well-developed screening test is the Miller Assessment for Preschoolers (MAP) (33). This test screens preschoolers for problems in the following areas: sensory and motor, speech and language, cognition, behaviors, and visual-motor integration.

The advantages of screening tests are that they are brief and sample a broad range of behaviors, traits, or characteristics. They are limited, however, because of an increased frequency of false-positive results that is due to the small sample of behaviors obtained. Screening tests should be used cautiously for diagnosis, placement, or treatment planning. They are used most effectively to indicate the need for more extensive testing and treatment of specific problem areas identified by the screening assessment.

Assessment tests have five possible applications:

An example of an assessment measure is the Boston Diagnostic Aphasia Examination (34). The advantages of assessment measures are that they have a lower frequency of false-positive results; they assess a representative set of behaviors; they can be used for diagnosis, placement, or treatment planning; and they provide information regarding the functional level of the individual tested. The limitations are that an extended amount of time is needed for testing, and they generally require specially trained personnel to administer, score, and interpret the results.

Criterion-referenced tests are those for which the test score is interpreted in terms of performance on the test relative to the continuum of possible scores attainable (18). The focus is on what the person can do or what he or she knows rather than how he or she compares with others (35). Individual performance is compared with a fixed expected standard rather than a reference group. Scores are interpreted based on absolute criteria, for example, the total number of items successfully completed. Criterion-referenced tests are useful to discriminate between successive performances of one person. They are
conducted to measure a specific set of behavioral objectives. The Tufts Assessment of Motor Performance (which has undergone further validation work and has been renamed the Michigan Modified Performance Assessment) is an example of a criterion-referenced test (36, 37, 38, 39, 40). This assessment battery measures a broad range of physical skills in the areas of mobility, activities of daily living, and physical aspects of communication.

Norm-referenced tests use a representative sample of people who are measured relative to a variable of interest. Norm referencing permits comparison of a single person’s measurement with those scores expected for the rest of the population. The normal values reported should be obtained from, and reported for, clearly described populations. The normal population should be the same as those for whom the test was designed to detect abnormalities (35). Reports of norm-referenced test results should use scoring procedures that reflect the person’s position relative to the normal distribution (e.g., percentiles, standard scores). Measures of central tendency (e.g., mean, median, mode) and variability (e.g., standard deviation, standard error of the mean) also should be reported to provide information on the range of normal scores, assisting with determination of the clinical relevance of test results. An example of a norm-referenced test is the Peabody Developmental Motor Scale (41). This developmental test assesses fine and gross motor domains. Test items are classified into the following categories: grasp, hand use, eye-hand coordination, manual dexterity, reflexes, balance, nonlocomotor, locomotor, and receipt and propulsion of objects.

Interrater or interobserver agreement is the extent to which independent examiners agree exactly on a patient’s performance. By contrast, interrater reliability is defined as the degree to which the ratings of different observers are proportional when expressed as deviations from their means; that is, the relationship of one rated person to other rated people is the same, although the absolute numbers used to express the relationship may vary from rater to rater (45). The independence of the examiners in the training they receive and the observations they make is critical in determining interrater agreement and reliability. When examiners have trained together or confer when performing a test, the interrater reliability or agreement coefficient calculated from their observations may be artificially inflated.

An interrater agreement or reliability coefficient provides an estimate of how much measurement error can be expected in scores obtained by two or more examiners who have
independently rated the same person. Determining interrater agreement or reliability is particularly important for test scores that largely depend on the examiner’s skill or judgment. An acceptable level of interrater reliability or agreement is essential for comparison of test results obtained from different clinical centers. Interrater agreement or reliability is a basic criterion for a measure to be called objective. If multiple examiners consistently obtain the same absolute or relative scores, then it is much more likely that the score is a function of the measure, rather than of the collective subjective bias of the examiners (18).

Pure interrater agreement and reliability are determined by having one examiner administer the test while the other examiner or examiners observe and independently score the person’s performance at the same point in time. When assessing some parameters, when the skill of the examiner administering the test plays a vital role (e.g., sensory testing, range-of-motion testing) or when direct observation of each examiner is required (e.g., strength), it is impossible to assess pure interrater agreement and reliability. In these instances, each examiner must test the individual independently. Consequently, these interrater measures are confounded by factors of time and variation in patient performance.

Test-retest agreement is defined as the extent to which a patient receives identical scores during two different test sessions when rated by the same examiner. By contrast, test-retest reliability assesses the degree of consistency in how a person’s score is rank ordered relative to other people tested by the same examiner during different test sessions. Test-retest reliability is the most basic and essential form of reliability. It provides an estimate of the variation in patient performance on a different test day, when retested by the same examiner. Some of the errors in a test-retest situation also may be attributed to variations in the examiner’s performance. It is important to determine the magnitude of day-to-day fluctuations in performance so that true changes in the parameters of interest can be determined. Variability of the test or how it is administered should not be the source of observed changes over time. Additionally, with quantitative measuring instruments, the examiner must be knowledgeable in the method of and frequency required for instrument calibration.

The suggested test-retest interval is 1 to 3 days for most physical measures and 7 days for maximal effort tests in which muscle fatigue is involved (46). The test-retest interval should not exceed the expected time for change to occur naturally. The purpose of an adequate but relatively short interval is to minimize the effects of memory, practice, and maturation or deterioration on test performance (47).

Intertrial agreement provides an estimate of the stability of repeated scores obtained by one examiner within a test session. Intertrial reliability assesses the consistency of one examiner rank-ordering repeated trials obtained from patients using the same measurement tool and standardized method for testing and scoring results within a test session. Intertrial agreement and reliability also are influenced by individual performance factors such as fatigue, motor learning, motivation, and consistency of effort. Intertrial agreement and reliability should not be confused with test-retest agreement and reliability. The latter involves test sessions usually separated by days or weeks as opposed to seconds or minutes for intertrial agreement and reliability. A higher level of association is expected for results obtained from trials within a test session than those from different sessions.

Alternate form agreement refers to the consistency of scores obtained from two forms of the same test. Equivalent or parallel forms are different test versions intended to measure the same traits at a comparable level of difficulty. Alternate form reliability refers to whether the parallel forms of a test rank order people’s scores consistently relative to each other. A high level of alternate form agreement or reliability may be required if a person must be tested more than once and a learning or practice effect is expected. This is particularly important when one form of the test will be used as a pretest and a second as a posttest.

Population-specific agreement and reliability assess the degree of absolute and relative reproducibility, respectively, that a test has for a specific group being measured (e.g., Ashworth scale scores for rating severity of spasticity from spinal cord injury). A variation of this type of agreement and reliability refers to the population of examiners administering the test (18).

Because measures of reliability and agreement are concerned with the degree of consistency or concordance between two or more independently derived sets of scores, they can be expressed in terms of correlation coefficients (35). The reliability coefficient is usually expressed as a value between 0 and 1, with higher values indicating higher reliability. Agreement statistics can range from −1 to +1, with +1 indicating perfect agreement, 0 indicating chance agreement, and negative values indicating less than chance agreement. The coefficient of choice varies, depending on the data type analyzed. The reader is referred to Bartko and Carpenter (48), Hartmann (49), Hollenbeck (50), Liebetrau (51), and Tinsley and Weiss (45) for discussions of how to select appropriate statistical measures of reliability and agreement. Table 8-1 provides information on appropriate statistical procedures for calculating interrater and test-retest reliability and agreement for discrete and continuous data types. No definitive standards for minimum acceptable levels of the different types of reliability and agreement statistics have been established; however, guidelines for minimum levels are
provided in Table 8-1. The acceptable level varies, depending on the magnitude of the decision being made, the population variance, the sources of error variance, and the measurement technique (e.g., instrumentation vs. behavioral assessments). If the population variance is relatively homogeneous, lower estimates of reliability are acceptable. By contrast, if the population variance is heterogeneous, higher estimates of reliability are expected. Critical values of correlation coefficients, based on the desired level of significance and the number of subjects, are provided in tables in measurement textbooks (52, 53). It is important to note that a correlation coefficient that is statistically significant does not necessarily indicate that adequate reliability or agreement has been established, because the significance level only provides an indication that the coefficient is significantly different from zero (see Table 8-1).

Agreement and reliability both are important for evaluating patient ratings. As discussed earlier, these are distinctly different concepts and require separate statistical analysis. Several factors must be considered to determine the relative importance of each. Decisions that carry greater weight or impact for the people being assessed may require more exact agreement. If the primary need is to assess the relative consistency between raters, and exact agreement is less critical, then a reliability measure alone is a satisfactory index. By contrast, whenever the major interest is either the absolute value of the score or the meaning of the scores as defined by the points on the scale (e.g., criterion-referenced tests), agreement should be reported in addition to the reliability (45). Scores generated from instrumentation are expected to have a higher level of reliability or agreement than scores obtained from behavioral observations.

A test score actually consists of two different components: the true score and the error score (35, 54). A person’s true score is a hypothetical construct, indicating a test score that is unaffected by chance factors. The error score refers to the unwanted variation in the test score (55). All continuous scale measurements have a component of error, and no test is completely reliable. Consequently, reliability is a matter of degree. Any reliability coefficient may be interpreted directly in terms of percentage of score variance attributable to different sources (18). A reliability coefficient of 0.85 signifies that 85% of the variance in test scores depends on true variance in the trait measured and 15% depends on error variance.

There are several statistical measures for estimating interrater agreement and reliability. Four statistics commonly used to determine agreement are the frequency ratio, point-by-point agreement ratio, kappa (κ) coefficients, and Lawlis and Lu’s Χ² and T-index statistics. For reliability calculations, the most frequently used correlation statistics are the Pearson product-moment (Pearson r) and intraclass correlation coefficients (ICCs). When determining reliability for dichotomous or ordinal data, specific ICC formulas have been developed. These nonparametric ICC statistics have been shown to be the equivalent of the weighted kappa (κ_w) (56, 57, 58, 59). Consequently, the κ_w also can be used as an index of reliability for discrete data, and the values obtained can be directly compared with equivalent forms of ICCs (57). The method of choice for reliability and agreement analyses partially depends on the assessment strategy used (45, 48, 51, 60). In addition to agreement and reliability statistics, standard errors of measurement (SEM) provide a clinically relevant index of reliability expressed in test score units. Each statistic is described below.