Validity

Validity is best assessed as the product of a well–thought-out study design and adherence to principles of randomization, blinding, and patient follow-up.

Q: Were patients randomized?

A: Yes. In the SPRINT trial, patients were randomized to receive either a reamed or an unreamed intramedullary nail, helping to balance known and unknown prognostic factors. Factors such as a patient’s age, underlying severity of the injury, presence of comorbidities such as diabetes, or health-related habits such as smoking can influence the outcomes of a planned treatment. If these factors are influential in selecting the course of treatment for a patient, the outcomes of that treatment will either be underestimated or be overestimated. Because known prognostic factors often influence surgeons’ recommendations and patients’ treatment decisions, observational studies may yield misleading results. If these factors (both known and unknown) prove unbalanced between a trial’s treatment groups, the study’s outcome will be biased. The power of randomization is that treatment and control groups are more likely to be balanced with respect to both known and unknown determinants of outcome.

Q: Was randomization concealed?

A: Yes. If those making the decision about patient eligibility are aware of the arm of the study to which the patient will be allocated, they may systematically enroll sicker or less sick patients to either of the two treatment groups. This may yield a biased result. Use of date of birth, odd and even chart numbers, or date of injury, commonly referred to as quasi-randomization methods, results in randomization that is not concealed as the investigator can predict which treatment option the patient will receive. Remote Internet or telephone randomization guarantees concealed randomization. Sealed, opaque envelopes that are tamperproof also result in concealed randomization, although remote randomization is more secure.

SPRINT investigators randomized eligible patients by accessing a 24-hour toll-free remote telephone randomization system. This system ensured concealment as it was impossible to predict the patient’s treatment group. Randomization was further stratified by the clinical center and the severity of soft tissue injury (open, closed, or both open and closed) in randomly permuted blocks of 2 and 4. The patients and the investigators were unaware of the block sizes, thus further ensuring concealment.

Q: Were all patients analyzed in the groups to which they were randomized?

A: Yes. Under the intention to treat principle, all patients were analyzed in the treatment groups to which they were randomized, regardless of the actual treatment received, their eligibility, and whether they followed the study protocol. Excluding patients from analysis in their randomization group for any reason (eg, too sick for surgery) may skew results in favor of a better clinical outcome. Adherence to the intention to treat principle preserves the power of randomization by maintaining that important known and unknown prognostic factors are likely to be equally distributed across treatment groups. The SPRINT investigators used the intention to treat principle.

Q: Were patients in the 2 treatment groups similar with respect to known prognostic factors?

A: Yes. In the ideal clinical investigation, differences between the treatment groups are kept at a minimum with regard to known prognostic variables through the process of randomization. Sometimes in trials with small sample sizes and sometimes by chance, randomization fails to result in balanced treatment groups.

Large differences in prognostic variables between the 2 treatment groups may compromise the study’s validity. Where known prognostic variables have a strong relationship to study outcomes, differences between treatment groups can confound the interpretation of the treatment being studied.

The SPRINT investigators presented multiple prognostic factors by treatment group (eg, age, sex, type of fracture ). They reported that the treatment groups were similar with respect to key prognostic variables, making adjustment for any prognostic factors in their analyses unnecessary.

Q: Were patients blinded?

A: Yes. In the SPRINT trial, the patients were blinded to their treatment allocation (reamed vs unreamed intramedullary nailing). Patients who believe that one treatment method is more efficacious than the other may feel and perform better than those who do not receive it. Measures such as blinding need to be implemented to ensure that patient reporting is free of influence. These problems can be avoided by ensuring that patients are unaware of the treatment they receive, referred to as blinding or masking.

Q: Were treating clinicians blinded?

A: No. Although blinding helps to eliminate the possibility of biased results from differences in patient care, it is particularly challenging to achieve it in randomized controlled trials evaluating different surgical techniques, including the SPRINT trial. This was exacerbated by the preexisting belief among most SPRINT investigators that reamed intramedullary nailing was clinically superior to unreamed intramedullary nailing of tibia fractures. Recognizing that this bias could lead to differences in the threshold for reoperation, a rule was instituted to prevent additional procedures for the treatment of delayed union during the first 6 months after surgery. Although adherence to this rule was only 55%, the number of early reoperations was virtually identical in the reamed and unreamed nailing groups. Therefore, this strategy proved to be effective, and study bias was most likely not introduced by a lack of blinding of the attending orthopedic surgeon.

Q: Were outcome assessors blinded?

A: Yes. Unblinded study personnel may influence the results they are measuring or recording through different interpretations of findings or by offering differential encouragement during performance tests. These study personnel can almost always be blinded to the treatment allocation, even if the patient and the treating surgeon cannot. In addition, investigators can have a blinded Adjudication Committee review clinical data to determine whether a patient experienced a study event.

In the SPRINT trial, the blinded Adjudication Committee reviewed all relevant medical records and radiographs to determine if each reported reoperation met the criteria for being a study event. An initial concern was whether the size of the nail would be sufficient to unmask the allocation of treatment because the nails used for unreamed fixation are smaller in diameter. To resolve this concern and to keep the committee blinded, the digital radiograph was photo edited to include only the fracture site. Finally, the data analysts were blinded in addition to the Writing Committee who interpreted the results of the trial to prevent bias.

Q: Was follow-up complete?

A: Yes. Ideally, investigators would like to know the status of each patient randomized at the conclusion of the trial. But, not all patients return for follow-up and consequently are termed “lost to follow-up.” These patients may disappear because they suffer adverse outcomes, such as death, or because they are doing well and do not return to be assessed. The greater the number of patients who are lost to follow-up, the more a study’s validity is potentially compromised.

Previous randomized controlled trials in orthopedic surgery have typically reported a loss to follow-up rate of 10%, although figures of up to 30% have been reported. The SPRINT investigators implemented numerous strategies to help ensure follow-up, including excluding individuals who were unlikely to keep appointments, obtaining accurate patient contact information and 3 alternate contacts at the time of randomization, organizing the follow-up appointments to coincide with normal surgical fracture clinic visits, and conducting telephone interviews with patients who could not attend the clinic.

As a result, 93% of the patients randomized could be observed. However, even a low rate of loss to follow-up may potentially threaten a study’s validity. The extent of that compromise remains a matter of judgment and depends on how likely it is that patients lost to follow-up in 1 treatment arm did poorly, whereas patients lost to follow-up in the other treatment arm did well. The SPRINT investigators indicate that the characteristics of the patients who were lost to follow-up were similar to those of the patients who were observed for 1 year. Given both this and their low rate of loss to follow-up, the validity of the SPRINT trial results does not seem to be threatened.

What are the Results?

Q: How large was the treatment effect?

A: Investigators conducting randomized clinical trials carefully monitor how often patients experience adverse events or outcomes. These can often be reported as the proportion of patients who experience such an event. The effect of a treatment may also be expressed as a relative risk, which is defined as the ratio of the risk of an event among an exposed population to the risk among the unexposed. The SPRINT investigators appropriately reported their primary composite outcome (reoperation) using relative risks.

Q: How precise was the estimate of the treatment effect?

A: The true risk reduction of a treatment can never be known. The best estimate of it is the point estimate that results from high-quality randomized controlled trials. A point estimate is a single value that is calculated from observations of the sample used to estimate a population value or parameter. The point estimate, although close, is unlikely to be precisely correct. The calculation of confidence intervals indicates the range in which the true effect likely lies. A confidence interval is a range of 2 values in which it is probable that the true value lies for the entire population of patients from which the study patients were selected. A 95% confidence interval defines the range that includes the true relative risk reduction 95% of the time. The SPRINT investigators appropriately reported 95% confidence intervals around each relative risk.

How Can One Apply the Results to Patient Care?

Q: Were the study patients similar to my patient?

A: Sufficient details on both patient and fracture characteristics to allow readers to make a comparison with the patients they see in their own clinical practice were presented. Inclusion and exclusion criteria were also clearly listed.

Q: Were all clinically important outcomes considered?

A: It is necessary to critically assess whether a trial reports all the important outcomes. In the SPRINT trial, the primary composite outcome was measured postoperatively at 12 months and included bone grafting, implant exchange, and dynamization in patients with a fracture gap of less than 1 cm. Infection and fasciotomy were considered as part of the composite outcome, irrespective of the postoperative gap. Adverse events were also reported, including death, deep venous thrombosis, pulmonary embolus, and sepsis. Patient-reported measures, such as health-related quality of life, were assessed but not presented in the primary manuscript.

Q: Are the likely treatment benefits worth the potential harm and costs?

A: Yes. It is also necessary to ascertain if the probable treatment benefits are worth the additional effort and costs. The SPRINT trial results have important implications for clinical practice as a possible benefit for reamed intramedullary nailing in patients with closed fractures was demonstrated. No difference was found between reamed or unreamed approaches in patients with open fractures. In addition, the trial’s results suggest that the level of reoperations may be reduced by allowing increased time for these fractures to heal.