Systematic Reviews and Meta-Analysis

Joseph T. Patterson, MD

Saam Morshed, MD, PhD, MPH

Dr. Morshed or an immediate family member serves as a paid consultant to or is an employee of Synthes and serves as a board member, owner, officer, or committee member of the Orthopaedic Trauma Association. Neither Dr. Patterson nor any immediate family member has received anything of value from or has stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this chapter.

INTRODUCTION

New research relevant to practicing orthopaedic surgeons is constantly being published. A sample of articles published in the top 25 highest impact factor orthopaedic journals each year suggests that the rate of publication of new, primary orthopaedic literature increases annually (Figure 1), which suggests that the body of orthopaedic literature is expanding at an exponential rate. A single practicing surgeon can no longer locate, read, and synthesize all published reports relevant to most clinical questions in a reasonable amount of time.

Published reviews of the scientific literature have come to serve an important role as summaries of information from the flood of primary scientific publications. The most common types of reviews are the narrative review and the systematic review. Quantitative derivatives of the systematic review include meta-analysis of published data, pooled reanalysis, and network meta-analysis. Systematic reviews serve a central role in evidence-based medicine, or the process of integrating individual clinical expertise with the best available external clinical evidence from published research to guide decisions about patient care.¹

The objectives of this chapter are to define the types of literature reviews, to compare the indications and limitations of each type of review within orthopaedic surgery, and to provide the practicing orthopaedic surgeon with an understanding of the methods behind each type of review as well as the skills to interpret and implement review findings.

TYPES OF REVIEWS

NARRATIVE REVIEW

A narrative review summarizes literature selected by an author or group of experts based on the author(s)’ opinion of relevance to a topic of interest. The review may touch on multiple topics around one theme: for example, a narrative review on clavicle fractures might include sections on anatomy, types of clavicle fractures, forms of nonsurgical and surgical fixation, and different techniques for fixation. Narrative reviews are useful for providing a broad overview of evaluating and managing a particular condition, for offering historical context, and for characterizing “what’s new?” or the “state of the art” in a particular area. A critical analysis review is a narrative review that attempts to consider all sides of controversial issues in an even-handed manner and use published evidence to make care recommendations in a standard format.²

A narrative review is inherently biased. The method of selecting studies is unlikely to be objective or systematic. Relevant data, findings, or publications may be selectively presented or omitted, while interpretation is usually qualitative and based on opinion or experience.³ Clinical recommendations from narrative reviews may lag behind or conflict with recently published evidence: an incomplete review of recent literature may fail to identify a new treatment of proven efficacy or recommend a therapy that has
since been proven harmful or useless.⁴ For these reasons, the narrative review is no longer an appropriate means of summarizing the best or most current evidence for answering a specific question.

FIGURE 1 Bar graph demonstrates annual generation of orthopaedic primary scientific publications and reviews 1997-2018 published in the top 25 orthopedic journals ranked by impact factor. (NIH PubMed.)

SYSTEMATIC REVIEW

A systematic review is a scientific investigation of published literature. Published studies are the “subjects” of this type of research. All available publications relevant to a specific question are sought, quantitatively assessed for bias, and synthesized if possible in a quantitative or qualitative manner. The full process is detailed later in the chapter.

The purpose of a systematic review is to answer a specific question using published information, favoring the highest level of available evidence. For example, the American Academy of Orthopaedic Surgeons conducts systematic reviews as the basis for developing clinical practice guidelines.⁵ Major applications to clinical care also include pooling data across studies to improve statistical power or precision while comparing treatment options, diagnostic tests, or measures of outcome.⁶^,⁷

The limitations of a systematic review include its narrow scope and sensitivity to publication bias. A systematic review cannot cover the breadth of topics possible in a narrative review. Additionally, published trials in orthopaedic surgery are more likely to describe positive treatment effects than negative or null treatment effects,⁸ while negative results are less likely to be published⁹—and unpublished results are impossible to include in a systematic review. The efficacy of a treatment, or treatment effect, can therefore be overestimated by a methodologically sound systematic review which has captured publications of selectively published positive results.

META-ANALYSIS

A systematic review pools data from published information. Conventional meta-analysis is the quantitative synthesis of that data. Pooled reanalysis describes the same techniques applied to unpublished raw data provided by original investigators. These techniques can be used when the individual studies evaluate the same test, exposure, or treatment in similar populations using equivalent outcome assessments. For example, three randomized, controlled trials of single versus double bundle anterior cruciate ligament (ACL) reconstruction that used KT-1000 arthrometer measurements and pivot-shift testing as primary outcomes were sufficiently similar in study methodology to permit meta-analysis of the published data.¹⁰ The reported outcomes in the treatment and control groups in each study were tallied. Associations between the type of ACL reconstruction and the two outcomes were then tested with the data aggregated from the three studies using the statistical techniques for meta-analysis. The authors of the meta-analysis reported a statistically significant finding that double-bundle versus single-bundle ACL reconstruction was associated smaller side-to-side differences in KT-1000 arthrometer laxity measurements that were closer to measurements from the uninjured controls knees.

Meta-analysis improves the power of the statistical analysis, increasing the precision of the result. Greater precision increases the likelihood that any association or difference may be interpreted as statistically significant. However, clinically important or functionally relevant differences may not necessarily exist even if statistically significant differences are reached with meta-analysis. As in the ACL example, that statistical difference established by the higher-powered meta-analysis did not translate to a minimal clinically important difference; there was no clinical advantage or functional benefit conferred to patients by the double-bundle ACL reconstruction over the single bundle technique.

Caution should be exercised when interpreting meta-analysis and considering translating the findings to clinical practice. Pooling similar studies to improve power may sacrifice validity if there is heterogeneity among the studies. Heterogeneity refers to differences between studies in selection criteria, characteristics of the sample populations, exposures
or interventions, method of allocating patients to exposures or interventions, distributions of potentially confounding variables across the exposure/intervention and control groups, blinding of the study patients or outcome assessments, outcome assessment techniques, timing of outcome assessment, data collection methods, follow-up, or any other study characteristic or method that differs between studies in a way that could affect—and therefore bias—the measured outcome. Simply, heterogeneity means that a meta-analysis is not comparing apples to apples. Heterogeneity also has a quantitative statistical definition which will be discussed later in this chapter.

Meta-analysis is an additional step performed upon the product of a systematic review. Just as a systematic review does not eliminate any form of bias already present in the included studies, neither does meta-analysis reduce or eliminate bias. The quality of evidence produced from a properly conducted systematic review or meta-analysis is only as good as the quality of the studies that constitute it.

HOW TO PERFORM A SYSTEMATIC REVIEW AND META-ANALYSIS

OVERVIEW

A review begins with a clinical question. In developing a question, a useful starting point is to specifying the target population (condition and context), intervention (treatment or exposure), comparator(s), and primary outcome(s) to be studied (PICO).¹¹ A protocol for the review and a search strategy are developed around the question, often with assistance from a research librarian. Studies are identified, screened, and selected for inclusion. The risk of bias, or the quality, of the included studies is assessed. Data are then abstracted. In a systematic review without meta-analysis, the findings from included studies are summarized and discussed qualitatively. In a quantitative systematic review, the findings are pooled and analyzed with statistical methods. The protocol and findings are reported according to accepted guidelines.

Citation manager software such as EndNote (Thomson Reuters, Carlsbad, CA), RefWorks (RefWorks, Bethesda, MD), Zotero (George Mason University, Fairfax, VA) as well as software specifically designed for systematic reviews such as Covidence (Covidence, Melbourne, Victoria) can be helpful for organizing and performing the search, citation review, study appraisal, and data abstraction portions of a systematic review.

PROTOCOL

As with any scientific investigation, a reproducible literature search strategy or protocol is planned prospectively and must be published with the review. The systematic review protocol details the research question, search strategy, methods of assessment, specific data to abstract, and planned analysis. The review protocol is the key feature that differentiates a systematic review from a narrative review. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P) guidelines represent a consensus standard for the necessary elements of a systematic review and an extensive modification of the QUOROM statement from which PRISMA and PRSIMA-P were developed¹²^,¹³^,¹⁴ (Table 1). For systematic reviews of observational studies and other types of research, we recommend consulting the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network for the appropriate guideline¹⁵ (a brief list is included in Table 2).

HOW TO CRITICALLY READ A SYSTEMATIC REVIEW

Understanding the process involved in generating a systematic review greatly informs the act of critically reading a systematic review.¹⁶ The PRSIMA-P guidelines provide an excellent framework¹² (Table 2). Questions to consider include¹⁷

Does the review address a specific question formulated with the PICO framework?
Was the protocol registered as published?
Is the protocol reproducible?
Is the search strategy comprehensive, or does the strategy systematically omit a body of relevant research such as a key journal or negative outcome?
Is there a clear flowchart of study inclusion and exclusion with explanations?
Are the included studies sufficiently homogenous in their methods and populations to pool the data?
Are the data extraction and synthesis appropriate?
Do forest and inverted funnel plots support meta-analysis?
Did the review address confidence in estimates of effect? Are there sources of funding that could introduce bias?
Do the results apply directly to my patient?

Current practice now favors publishing the systematic review protocol before undertaking the review. Prospective publication of review protocols in a permanent record such as PROSPERO (an international register of systematic reviews supported by the United Kingdom National Health Service) prevents researchers from undertaking overlapping reviews and reduces the opportunity for reporting bias.¹⁸ Before detailing a protocol, the review question should be queried using an electronic web-based search engines as well as the PROSPERO registry to see if an overlapping review has already been published or is underway.

TABLE 1 PRISMA-P 2015 Checklist: Recommended Items to Include in a Systematic Review Protocol

Section/Topic		Item #	Checklist Item
Administrative Information
Title
	Identification	1a’	Identify the report as a protocol of a systematic review
	Update	1b	If the protocol is for an update of a previous systematic review, identify as such
Registration		2	If registered, provide the name of the registry (eg, PROSPERO) and registration number
Authors
	Contact	3a	Provide name, institutional affiliation, and e-mail address of all protocol authors; provide physical mailing address of corresponding author
	Contributions	3b	Describe contributions of protocol authors and identify the guarantor of the review
Amendments		4	If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments
Support
	Sources	5a	Indicate sources of financial or other support for the review
	Sponsor	5b	Provide name for the review funder and/or sponsor
	Role of sponsor/funder	5c	Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol
Introduction
Rationale		6	Describe the rationale for the review in the context of what is already known
Objectives		7	Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO)
Methods
Eligibility criteria		8	Specify the study characteristics (eg, PICO, study design, setting, time frame) and report characteristics (eg, years considered, language, publication status) to be used as criteria for eligibility for the review
Information sources		9	Describe all intended information sources (eg, electronic databases, contact with study authors, trial registers, or other grey literature sources) with planned dates of coverage
Search strategy		10	Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated
Study records
	Data management	11a	Describe the mechanism(s) that will be used to manage records and data throughout the review
	Selection process	11b	State the process that will be used for selecting studies (eg, two independent reviewers) through each phase of the review (ie, screening, eligibility, and inclusion in meta-analysis)
	Data collection process	11c	Describe planned method of extracting data from reports (eg, piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators
Data items		12	List and define all variables for which data will be sought (eg, PICO items, funding sources), any preplanned data assumptions and simplifications
Outcomes and prioritization		13	List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale
Risk of bias in individual studies		14	Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis
Data
Synthesis		15a	Describe criteria under which study data will be quantitatively synthesized
		15b	If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data, and methods of combining data from studies, including any planned exploration of consistency (eg, I², Kendall’s tau)
		15c	Describe any proposed additional analyses (eg, sensitivity or subgroup analyses, meta-regression)
		15d	If quantitative synthesis is not appropriate, describe the type of summary planned
Meta-bias(es)		16	Specify any planned assessment of meta-bias(es) (eg, publication bias across studies, selective reporting within studies)
Confidence in cumulative evidence		17	Describe how the strength of the body of evidence will be assessed (eg, GRADE)
Adapted from Shamseer L, Moher D, Clarke M, et al, PRISMA-P Group: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647. Used under Creative Commons Attribution License 4.0, http://creativecommons.org/licenses/by/4.0/.