Clinical Prediction Rules

Their Benefits and Limitations in Clinical Reasoning

Robin Haskins, Chad E. Cook, Peter G. Osmotherly, Darren A. Rivett

An Overview of Statistics in Healthcare Clinical Reasoning

Decision-making in the healthcare context has undergone a long and complex evolution. Perhaps surprisingly, the explicit integration of statistics to inform certain types of clinical decisions is a relatively more recent adjunct. A key driver of this more recent focus and conscientious employment of statistics in the clinical setting has been the evidence-based movement. Statistical data regarding disease and outcome prevalence (Laupacis et al., 1994; Richardson et al., 1999), diagnostic test accuracy (Jaeschke et al., 1994) and the quantification of treatment effect (Guyatt et al., 1994), among many others, have been increasingly used to help inform clinical decision-making.

In contemporary healthcare practice, the application of statistics facilitates the transformation of data into evidence-based diagnostic, prognostic and treatment decisions (Horvitz, 2010). Several different types of statistical prediction tools have been developed for use in the clinical setting, ranging from simple actuarial tables to more computationally complex approaches, such as artificial neural networks (Baxt, 1995; Meehl, 1954). Irrespective of the type, all statistical prediction tools use statistical analysis of prior cases with known outcomes to identify the quantified relationship between predictor variables and a particular diagnosis or outcome, such that they may be used to make future predictions (Swets et al., 2000b). This is simultaneously their strength and limitation.

A notable advantage of statistical prediction tools over unassisted clinician judgement is the control of human cognitive biases that are a common contributor to decision-making errors (Grove et al., 2000; Graber et al., 2005). Such errors in clinical problem solving are thought, at least in part, to be a consequence of limitations in the human cognitive capacity (Elstein and Schwarz, 2002). Simon (1990) described this as the principle of ‘bounded rationality’ – decision-making is limited by human behaviour being only partly rational, thereby causing limitations in information processing and complex problem solving, thus requiring the use of suboptimal approximation methods and heuristics. The need for fast and efficient decision-making ‘shortcuts’ and the resulting cognitive biases are believed, at least in part, to have arisen adaptively through our evolutionary history as a result of their intrinsic advantages for survival (Johnson et al., 2013). Such adaptive cognitive processes may, however, be suboptimal in many modern decision-making contexts, and their identification is frequently cited as central to reducing errors in medical practice (Croskerry, 2009; Ely et al., 2011; Graber et al., 2005, 2002; Hicks and Kluemper, 2011).

A critical limitation, however, of statistical prediction tools is their inherent inflexibility and fragility. That is, their predictions are limited to the specific outcome/diagnosis for which they were designed and are generated based on the limited subset of information considered within the tool. They are consequently not able to inform all categories of clinical judgements and are not able to integrate all of the available information that may be pertinent to a decision. The use of statistical procedures to inform decisions is therefore crucially reliant on a skilled individual’s ability to judge the appropriateness of its application, an awareness of its limitations and assumptions, and the accurate interpretation of its results (Dawes et al., 1989; Swets et al., 2000a, 2000b).

P. E. Meehl (1954) first highlighted the crucial role of the skilled individual in the application of statistical prediction models in what is known as the ‘broken leg countervailing’. This is where a prediction model may normally perform well under usual circumstances (e.g. a model that predicts someone’s attendance at the movies given the day of the week) but will require human adjustment in the light of additional information not accounted for in the model that will influence the predicted outcome (e.g. in the rare case that someone has broken his or her leg, he or she is much less likely to attend the movies) (Grove and Meehl, 1996).

Importantly, statistics are not always available to inform all categories of clinical judgement, limiting the applicability of statistical prediction tools. Greater awareness of common cognitive errors and strategies to reduce some errors may assist in minimizing errors of clinical judgment (see Chapter 1 for further discussion). Rather than being a slave to a mathematical formula, it is suggested that clinicians using statistical prediction models integrate the objective data produced from such tools with all other existing information to facilitate their decision-making (Swets et al., 2000a). That is, statistical predictions do not form a clinical decision but, instead, inform a clinical decision.

The remainder of this chapter focuses specifically on a type of statistical prediction tool most commonly referred to as a ‘clinical prediction rule’.

Clinical Prediction Rules

A clinical prediction rule (CPR) has been defined as a ‘a clinical tool that quantifies the individual contributions that various components of the history, physical examination and basic laboratory results make towards the diagnosis, prognosis, or likely response to treatment in an individual patient’ (McGinn et al., 2008, p. 493). Common synonyms include ‘clinical prediction guides’ (McGinn et al., 2008; US National Library of Medicine, 2009), ‘clinical prediction tools’ (Randolph et al., 1998), ‘clinical decision rules’ (Osmond et al., 2010), ‘clinical decision guides’ (Schneider et al., 2014) and ‘clinical decision tools’ (Thiruganasambandamoorthy et al., 2014).

CPRs may be conceptualized as a method of incorporating research evidence into clinical decision-making (Beattie and Nelson, 2006). They are clinical tools composed of the most parsimonious set of variables that have been empirically identified to predict a meaningful diagnosis or outcome (Childs and Cleland, 2006). Variables are commonly components of the history, physical examination and/or other tests or investigations that may be reliably collected within a standard clinical encounter (Laupacis et al., 1997). Some forms of CPRs enable the calculation of the probability of a given outcome or diagnosis, whilst others function to directly inform a specific course of action (Reilly and Evans, 2006). It is generally considered that CPRs may be of greatest utility when developed to assist in complex clinical decisions (McGinn et al., 2000).

Three major types of CPRs have been identified in the medical literature: diagnostic, prognostic and prescriptive (C. Cook, 2008).

Diagnostic Clinical Prediction Rules

Diagnostic CPRs function to inform clinical decisions regarding an individual patient’s diagnosis or present classification/status. An example of a diagnostic CPR is the Ottawa Knee Rule (Stiell, Greenberg, et al., 1995). This five-item tool is designed to help inform decisions regarding which patients presenting to an emergency department following an acute knee injury require an x-ray. A patient’s status on this CPR is determined by considering the presence or absence of five clinical variables (Table 5.1). In the absence of all five clinical variables, the likelihood of a knee fracture is remote (Bachmann et al., 2004), and consequently, an x-ray of the knee is unlikely to yield valuable clinical information.

TABLE 5.1

EXAMPLE OF A DIAGNOSTIC CLINICAL PREDICTION RULE: THE OTTAWA KNEE RULE
1.	Age ≥55 years
2.	Tenderness at head of fibula
3.	Isolated tenderness of patella
4.	Inability to flex knee to 90 degrees
5.	Inability to bear weight (twice on each limb regardless of limping), both immediately and in the emergency department

(Stiell, Greenberg, et al., 1995)

Prognostic Clinical Prediction Rules

Prognostic CPRs differ from their diagnostic counterparts with respect to their dependence on the dimension of time. Prognostic CPRs function to inform clinical judgements regarding future outcomes or events, such as an individual’s pain severity or likelihood of returning to work in 6 months’ time. An example of a prognostic CPR is the ‘Cassandra rule’ (Dionne, 2005; Dionne et al., 1997, 2011). This CPR was derived in a population of patients with back pain presenting to primary care physicians and aims to identify individuals with differing degrees of risk of developing long-term significant functional limitations. The CPR uses a measure of depression and a measure of somatization from selected items of the Symptoms Checklist 90 Revised (Derogatis, 1977) questionnaire to stratify patients by their degree of risk of having 50% or greater disability on the Roland-Morris Disability Questionnaire (Roland and Morris, 1983) at 2 years.

Prescriptive Clinical Prediction Rules

Prescriptive CPRs are the third major type of these tools and function to sub-classify patient populations by matching patients to treatments based on their predicted responsiveness to that treatment, independent of a diagnostic classification (Foster et al., 2013). As such, prescriptive CPRs inform clinical decisions regarding treatment selection (C. Cook, 2008) and can be conceptualized as a special form of prognostic CPR that specifically relates to treatment effects. The treatment effect is the difference in outcome that is achieved by one intervention in comparison to that achieved by an alternative or control intervention (Kamper et al., 2010). Prescriptive CPRs are thus comprised of treatment effect modifiers (also known as effect moderators) – these are the baseline variables that differentiate patient subgroups which experience differing magnitudes of treatment effect (Kraemer et al., 2006). Such variables are subsequently distinct from prognostic variables, which predict outcomes independent of treatment (Hill and Fritz, 2011).

A patient’s status on a treatment effect modifier predicts the relative benefit the patient will likely achieve from one intervention compared with another. Fig. 5.1 illustrates this relationship. Treatment effect modifiers are identified in randomized clinical trials by exploring interaction effects between candidate baseline variables and treatment groups (Hancock et al., 2009; Sun et al., 2010). The sample sizes required for such trials are, however, very large. To adequately power a study to detect an interaction effect, the sample size needs to be approximately four times that required to detect an overall treatment effect of the same magnitude (Brookes et al., 2004).

Fig. 5.1 Illustration of a treatment effect that is modified by a patient’s status on a baseline variable.

Development of Clinical Prediction Rules

The development a CPR occurs across three main stages: derivation, validation and impact analysis (Fig. 5.2) (Childs and Cleland, 2006; McGinn et al., 2000, 2008). Each stage functions to develop and investigate a specific aspect of a CPR and has crucial implications for its ability to be applied in clinical practice. The following subsections describe the processes involved in each of the main stages of a CPR’s development.

Fig. 5.2 Stages in the development of a clinical prediction rule (CPR) (Adapted with permission from Childs and Cleland [2006]).

Derivation

The first step in the development of a CPR is derivation. This process commences with the identification of a meaningful problem for which the development of a CPR may be perceived as clinically useful. Considerations that help inform the need for a CPR include the complexity of clinical decision-making, the accuracy of unassisted clinician judgement, clinician attitudes, variations in practices and the hypothesized potential for a tool to beneficially impact practice by improving patient outcomes or improving resource efficiencies (Fritz, 2009; Stiell and Wells, 1999).

The study design required to derive a CPR is dependent on the type of CPR under development. Diagnostic CPRs are derived in cross-sectional studies, prognostic CPRs are derived in longitudinal cohort studies and prescriptive CPRs require randomized controlled trials (Hancock et al., 2009; Hill and Fritz, 2011). In all instances, a meaningful, valid and clearly defined dependent outcome that is able to be reliably measured requires selection (Stiell and Wells, 1999). A small number of candidate predictor variables also need to be selected a priori and considered within the context of their hypothesized predictive performance, validity and reliability, as well as their practicality and availability within the clinical environment (C. Cook et al., 2010; Lubetzky-Vilnai et al., 2014; Seel et al., 2012). Clinical judgement, literature reviews, focus groups and questionnaires have been used to select candidate predictor variables in some CPR derivation studies (Dionne et al., 2005; Hewitt et al., 2007; Heymans et al., 2007, 2009).

The patient population sampled in CPR derivation studies needs to represent the spectrum of patients to which the tool is likely to be applied (Stiell and Wells, 1999). Generally, large sample sizes are required to satisfy the assumptions of the statistical techniques that are used and to also generate greater precision of the findings (Childs and Cleland, 2006). Larger sample sizes are particularly required when investigating an outcome with a very low prevalence (e.g. cancer in patients with low back pain), when testing large numbers of candidate predictors and when investigating treatment effect modifiers (Babyak, 2004; Brookes et al., 2004).

Once data collection is complete, statistical analysis is used to identify the candidate variables that have a significant predictive relationship with the dependent outcome. There are several different techniques that have been used to derive CPRs in the medical literature. Table 5.2, adapted from Grobman and Stamilio (2006) and Adams and Leveson (2012), provides an overview of these techniques and their relative advantages and disadvantages.

TABLE 5.2

TECHNIQUES USED TO DEVELOP CLINICAL PREDICTION RULES
Technique	Advantages	Disadvantages
Univariate analysis	Simple to develop. Easy to use.	Predictors may not be independent. Weightings are arbitrary. Less accurate.
Multivariable analysis	Improved accuracy.	Slightly more complicated to develop.
Nomograms	Improved accuracy. Easy to use.	More complicated to develop.
Classification and regression trees (recursive partitioning)	Easy to use. Enables development of rules that are optimized for sensitivity or specificity.	Can often be less accurate than other techniques. Does not work well for continuous variables. Prone to overfitting.
Artificial neural network	Improved accuracy over time with new data. Identifies complex non-linear relationships and interactions.	More complicated to develop. Prone to overfitting. Hard to apply in most clinical settings.

(Adapted from Grobman and Stamilio [2006] and Adams and Leveson [2012])

Univariate analysis, whereby the relationships between each predictor variable and the dependent outcome are examined separately, is the simplest technique but has several limitations. Most notably, it does not account for the relationship among candidate predictor variables. Multivariable analysis overcomes this limitation by examining the independent relationship of each predictor variable with the target outcome, and it also enables the assignment of variable weightings based on the interpretation of the regression coefficients (Laupacis et al., 1997). Various forms of multivariable analysis have been commonly used to derive CPRs (Bouwmeester et al., 2012), and in some cases, automated methods of variable selection (e.g. forward stepwise, backward deletion, best subset) are applied. However, given the increased chance of identifying spurious associations using automated procedures, these approaches may not be well suited for CPR development and may best be reserved for exploratory analysis (Babyak, 2004; Katz, 2003). Multivariable models are generally well suited to construct nomograms, which are graphical calculating tools that facilitate the application of otherwise-complicated mathematical equations (Grobman and Stamilio, 2006).

Classification and regression trees are another approach used to derive CPRs. This analysis uses non-parametric statistical procedures to identify mutually exclusive and exhaustive subgroups based on the variables that predict the dependent outcome (Lemon et al., 2003). Recursive partitioning accounts for interactions between predictor variables (E. F. Cook and Goldman, 1984; Dionne et al., 1997) and is subsequently better suited for deriving CPRs from datasets with interacting variables than logistic regression (Katz, 2006). This approach is also considered to be well suited in instances where a CPR requires optimization of either sensitivity or specificity (Stiell and Wells, 1999).

Artificial neural networks require advanced computational resources and are another approach used to develop CPRs. Artificial neural networks are inherently statistically more flexible than regression approaches and, all else being equal, provide models that better fit the study data (Kattan, 2002). However, as a consequence, they are also more vulnerable to overfitting, thus potentially reducing the likelihood that these approaches will perform well outside of the derivation study data (Tu, 1996).

To illustrate the development of a CPR, the Ottawa Knee Rule (Table 5.1) will be used as an example (Stiell, Greenberg, et al., 1995). A need for a tool to help decide which patients require an x-ray was based on the finding that whilst almost three-quarters of patients presenting with acute knee injury to an emergency department were referred for radiology, only 5% were identified to have a fracture (Stiell, Wells, et al., 1995). This contributes to increased costs of care, increased waiting times and unnecessary radiation exposure. It was also identified that experienced clinicians believed that the probability of a fracture was less than 10% in the majority of patients sent for radiology (Stiell, Wells, et al., 1995).

Consequently, a prospective study was conducted involving 1047 adult patients with acute knee injuries presenting to one of two university hospital emergency departments in Ottawa, Canada. The dependent outcome was any fracture of the knee seen on plain x-ray and was determined blinded to knowledge of the candidate predictor variables. For ethical reasons, patients thought not to require a knee x-ray were not sent for radiology, but follow-up was conducted via a telephone questionnaire with the aim of detecting any missed fractures. Twenty-three candidate predictor variables were selected based on clinician judgement, literature review and pilot study data. Explicit definitions of each variable were provided to clinicians in a handout.

Following data collection, recursive partitioning was used to derive the CPR. The tool was developed to optimize sensitivity, given that a missed fracture would be of greater consequence than an unnecessary x-ray. Many different models were identified to fit the data, and the research team decided to select the model that gave the greatest specificity and used the fewest number of variables whilst maintaining 100% sensitivity. The accuracy of the Ottawa Knee Rule in the derivation study was a sensitivity of 100% (95% confidence interval [CI] 95%–100%) and a specificity of 54% (95% CI 51%–57%).

Only gold members can continue reading. Log In or Register to continue