Reporting Bias Is Highly Prevalent in Systematic Reviews and Meta-Analyses Related to Medial Patellofemoral Ligament Reconstruction

Purpose

To analyze reporting bias in the form of spin present in systematic reviews and meta-analyses related to medial patellofemoral ligament reconstruction (MPFLR).

Methods

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, systematic reviews were collected from PubMed, Web of Science, and Embase using the search “medial patellofemoral ligament reconstruction” or “MPFLR” AND “systematic review” OR “meta-analysis.” Abstracts were assessed for 15 common spin types. A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2) was used to evaluate the quality of the studies. Characteristics such as Preferred Reporting Items for Systematic Reviews and Meta-Analyses adherence, publication year, and level of evidence were analyzed. Associations between these factors and spin presence or type were determined using statistical tests, including t tests, analysis of variance, Fisher tests, and Spearman rank coefficients.

Results

The initial database search identified 1,044 studies, of which a total of 57 studies were included. Spin was present in the abstract in 51 of 57 studies (89.5%). Each type of spin was observed in at least 1 study’s abstract with the exceptions of spin types 1, 7, 13, and 15. The 3 most common types were type 5 (48/57, 84.2%), followed by type 3 (32/57, 56.1%) and type 9 (30/57, 52.6%). Of the included studies, 91.2% received a critically low AMSTAR 2 confidence rating, and only 5.3% reported a conflict of interest. There was a statistically significant negative correlation between the numerical AMSTAR 2 rating and the presence of spin ( P <.01).

Conclusions

Most systematic reviews on MPFLR received critically low AMSTAR 2 ratings, reflecting the poor quality of evidence in this area. Nearly 90% of abstracts exhibited at least 1 type of spin, with spin types 3, 5, and 9 being the most common, suggesting a tendency to overstate the efficacy of MPFLR for patellar instability.

Level of Evidence

Level IV, systematic review of Level II-IV studies.

Medial patellofemoral ligament reconstruction (MPFLR) is commonly performed to treat recurrent patellar instability. To ensure that the results of MPFLR are accurately portrayed in the literature and are of high quality, it is important to review published studies for reporting bias in the form of spin. Spin is defined as a “specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment in terms of efficacy or safety is greater than that shown by the results (i.e., overstate efficacy and/or understate harm).”

By examining the degree of spin within a study, the precision and accuracy with which findings are reported in the literature can be determined. The assessment of spin scrutinizes research at the level of results reporting. According to Yavchitz et al., spin can be organized into 3 different categories: misleading representation, misleading reporting, and inappropriate extrapolation. Furthermore, prior studies by Lazarus et al. and Boutron et al. have shown that spin is a frequent occurrence in abstracts, appearing in 58.3% of randomized controlled trials with nonsignificant primary outcomes and 84% of nonrandomized intervention studies, respectively. These findings underscore how commonly spin can shape first impressions of research findings, even in otherwise methodologically sound studies. Physicians rely heavily on abstracts as their primary source of information for making therapeutic decisions, potentially influencing the perceived effectiveness of an intervention when spin is present. ^,^,

Although prior studies have evaluated spin in systematic reviews across various orthopaedic procedures, MPFLR remains understudied in this context. The body of literature surrounding MPFLR is relatively diverse and continues to expand, making it important to assess how these studies are being summarized in abstracts. These characteristics may increase susceptibility to spin in abstracts, as authors attempt to reconcile heterogeneous findings. The purpose of this study was to analyze reporting bias in the form of spin present in systematic reviews and meta-analyses related to MPFLR. We hypothesized that spin would be commonly present in abstracts of systematic reviews and meta-analyses related to MPFLR, reflecting trends previously reported in the broader clinical research literature, and that greater levels of spin would be associated with lower methodologic quality.

Methods

This study was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines using a predetermined protocol. The methodology, including eligibility criteria, data extraction strategy, and spin classification approach, was established prior to the conduct of the review to ensure consistency and minimize bias in the assessment process. Two authors (K.T.N. and E.L.B.) conducted a search of the PubMed, Web of Science, and Embase databases using “medial patellofemoral ligament reconstruction” or “MPFLR” AND “systematic review” OR “meta-analysis” in January 2024. The search results were aggregated and deduplicated in Covidence. Two authors (K.T.N. and E.L.B.) independently screened the identified studies for inclusion, while the third author (W.C.R.) resolved remaining conflicts.

Eligibility

Systematic reviews and meta-analyses related to MPFLR published in an English peer-reviewed journal were eligible for inclusion. Databases were queried from inception to January 22, 2024. Studies were excluded if they were not peer-reviewed, were not published in English, were not systematic reviews and/or meta-analyses, were retracted or withdrawn, included nonhuman or cadaver subjects, were unrelated to MPFLR, were published without an abstract, or did not have full text available.

Training

Three authors (K.T.N., E.L.B., and W.C.R.) received training on spin and the AMSTAR 2 grading system by independently reviewing relevant literature on both topics, as well as in the definition and classification of the most common types of spin proposed by Yavchitz et al., as summarized in Table 1 . ^, The authors also learned to assess study quality using A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2). The adoption of AMSTAR 2 for assessing study quality is supported by its impressive inter-rater reliability and high construct validity.

Table 1

Description of Types of Spin Assessed

Category	Type	Description
Misleading interpretation
	1	The conclusion formulates recommendations for clinical practice not supported by the findings.
	2	The title claims or suggests a beneficial effect of the experimental intervention not supported by the findings.
	4	The conclusion claims safety based on nonstatistically significant results with a wide confidence interval.
	9	The conclusion claims the beneficial effect of the experimental treatment despite reporting bias.
	12	The conclusion claims equivalence or comparable effectiveness for nonstatistically significant results with a wide confidence interval.
Misleading reporting
	3	Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention.
	5	The conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies.
	6	Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention.
	10	Authors hide or do not present any conflict of interest.
	11	The conclusion focuses selectively on a statistically significant efficacy outcome.
	13	Failure to specify the direction of the effect when it favors the control intervention.
	14	Failure to report a wide confidence interval of estimates.
Inappropriate extrapolation
	7	The conclusion extrapolates the review findings to a different intervention (e.g., claiming efficacy of one specific intervention, although the review covered a class of several interventions).
	8	The conclusion extrapolates the review’s findings from a surrogate marker or a specific outcome to the global improvement of the disease.
	15	The conclusion extrapolates the review’s findings to a different population or setting.

Using AMSTAR 2, a 16-question critical appraisal tool, studies were graded by each author (K.T.N., E.L.B., and W.C.R.) on their methodologic quality and assigned a confidence rating. This tool evaluates an author’s incorporation of a predetermined study protocol, funding source, conflicts of interest, and authors’ overall ability to adequately characterize the findings of studies included in the review. The full texts of the included studies were used to assess study quality per the AMSTAR 2 checklist, which yielded a numeric score between 0 and 16. This score reflects the extent to which the studies meet quality standards, with the numerical value representing the number of AMSTAR 2 checklist elements found in the study. In addition to the numerical score, the AMSTAR 2 critical appraisal tool distinguishes between deficiencies in critical and noncritical domains. In doing so, the AMSTAR 2 assessment identifies critical flaws in systematic reviews and meta-analyses by assigning studies critically low, low, moderate, and high confidence ratings. ^,

Data Extraction

Three authors extracted data independently (K.T.N., E.L.B., and W.C.R). In the case of disagreement, resolution was achieved after discussion between the 2 authors (K.T.N. and E.L.B.) or input from a third author (W.C.R.). Study characteristics from the full text that were extracted included title, authors, publication year, study design, journal, funding source, level of evidence, reported adherence to PRISMA guidelines, preregistration status, and outcome measures. If not stated within the study, the level of evidence was determined using the American Academy of Orthopaedic Surgeons recommendations.

The full text of each study was assessed for the 15 most common types of spin ( Table 1 ). For studies reporting confidence intervals, an interval was considered wide if it spanned more than 2 units, included both sides of the null value, or exceeded 50% of the point estimate in relative width. The full texts of the included systematic reviews were used to assess study quality per the AMSTAR 2 checklist, which yielded a numeric score between 0 and 16. The AMSTAR 2 confidence ratings were also extracted. Additionally, the impact factor was recorded for the journals in which the included systematic reviews and meta-analyses were published. This metric, which gauges the significance of a journal by totaling the number of citations its selected articles receive over a recent period (typically the past few years), serves as a valuable tool for comparing journals within a specific subject category. A higher impact factor indicates a more esteemed journal.

Data Analysis

Descriptive statistics were used to characterize the frequency of spin occurring in the included studies. Study characteristics, including study type, level of evidence, funding source, PRISMA adherence, PROSPERO registration, impact factor, and AMSTAR 2 confidence rating, were analyzed. Their association with the presence of spin, as well as the number of spin types present, was determined using t tests, analysis of variance, Fisher tests, and Spearman rank coefficients. A P value <.05 was considered statistically significant.

Results

The initial database search identified 1,044 studies, of which 438 duplicates were removed. An additional 499 studies were removed after title and abstract screening because they did not meet the inclusion criteria. The remaining 107 studies were assessed for eligibility. Of the 107, 50 (46.7%) were excluded because they were abstracts only (21, 42.0%), had the wrong study design (19, 38.0%), investigated the wrong intervention (8, 16.0%), or identified the wrong outcomes (2, 4.0%) ( Fig 1 ). The 57 (53.3%) remaining systematic reviews, which were published in 24 different journals with dates of publication ranging from 2010 to 2024, were included for analysis. ^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,^,

Image, AltText currently not available — Fig 1

Most of the included studies were Level of Evidence IV (46 out of 57, 80.7%), 6 (10.5%) were Level of Evidence III, 4 (7.0%) were Level of Evidence II, and 1 (1.8%) was Level of Evidence V. Twenty-nine studies (50.9%) included a systematic review with meta-analysis, while 28 studies (49.1%) were systematic reviews without meta-analyses. Twelve studies (21.1%) did not disclose funding, 16 (28.1%) disclosed at least 1 external source of funding, and 29 studies (50.9%) received no funding. Three studies (5.3%) submitted their protocols to the PROSPERO public register of systematic reviews. Almost all studies reported PRISMA adherence (52/57, 91.2%). Twenty-four different journals were represented among these systematic reviews, for which the mean impact factor was 2.95 ± 2.33 (range, 0.5-12.2) ( Table 2 ).

Table 2

Study Characteristics of Included Systematic Reviews and Meta-Analyses

Author	Year Published	Journal	Impact Factor	Level of Evidence	PRISMA Adherence	PROSPERO Preregistration
Abbaszadeh et al.	2023	World Journal of Clinical Cases	1.1	IV	Yes	No
Abelleyra Lastoria et al.	2023	Knee Surgery & Related Research	3.1	IV	Yes	No
Aicale and Maffulli	2020	Journal of Orthopaedic Surgery and Research	2.7	IV	Yes	No
Aliberti et al.	2021	Orthopaedic Journal of Sports Medicine	2.8	IV	Yes	No
Almeida et al.	2023	Journal of Clinical Orthopaedics and Trauma	3.3	IV	Yes	No
An et al.	2019	Journal of Orthopaedic Surgery	1.8	III	Yes	No
Balcarek et al.	2017	KSSTA	3.4	IV	No	No
Boutefnouche et al.	2016	KSRR	1.8	IV	Yes	No
Buckens and Saris	2010	American Journal of Sports Medicine	3.9	IV	No	No
Burnham et al.	2016	Journal of Arthroscopic and Related Surgery	4.5	IV	No	No
Castagno et al.	2023	The Knee	2.0	IV	Yes	No
Cohen et al.	2022	KSSTA	4.2	IV	Yes	No
D’Ambrosi et al.	2021	Children	2.6	IV	Yes	No
Fancher et al.	2021	Orthopaedic Journal of Sports Medicine	2.8	IV	Yes	No
Fang et al.	2023	Frontiers in Surgery	2.0	III	Yes	No
Fisher et al.	2010	Arthroscopy	3.1	IV	Yes	No
Guevel et al.	2023	Frontiers in Surgery	2.0	IV	Yes	No
Heo et al.	2019	American Journal of Sports Medicine	6.0	IV	Yes	No
Hussein et al.	2018	Journal of ISAKOS	1.9	II	Yes	No
Jiang et al.	2023	Indian Journal of Orthopaedics	1.0	IV	Yes	No
Kahlon et al.	2023	KSSTA	4.2	II	Yes	No
Kalinterakis et al.	2023	European Journal of Orthopaedic Surgery and Traumatology	1.8	IV	Yes	No
Kang et al.	2019	KSSTA	3.4	IV	Yes	No
Kang et al.	2018	Archives of Orthopaedic and Trauma Surgery	2.3	IV	Yes	No
Liu et al.	2021	Orthopaedic Journal of Sports Medicine	2.6	III	Yes	Yes
Mackay et al.	2014	Orthopaedic Journal of Sports Medicine	2.6	IV	Yes	No
Manjunath et al.	2021	American Journal of Sports Medicine	6.0	IV	Yes	No
MarínFermín et al.	2022	Journal of Orthopaedic Surgery and Research	2.7	IV	Yes	No
Migliorini et al.	2022	Journal of Sport and Health Science	13.0	IV	Yes	No
Migliorini et al.	2020	Archives of Orthopaedic and Trauma Surgery	2.5	IV	Yes	No
Migliorini et al.	2022	Journal of Orthopaedics and Traumatology	3.0	IV	Yes	No
Migliorini et al.	2022	Children	2.6	IV	Yes	No
Migliorini et al.	2023	Children	2.4	IV	Yes	No
Migliorini et al.	2021	Journal of Orthopaedic Surgery and Research	2.7	IV	Yes	No
Migliorini et al.	2020	European Journal of Orthopaedic Surgery & Traumatology	1.9	IV	Yes	No
Nha et al.	2019	KSRR	1.6	V	Yes	No
Pang et al.	2023	American Journal of Sports Medicine	6.1	III	Yes	No
Pappa et al.	2023	Journal of ISAKOS	1.8	IV	Yes	No
Plat et al.	2022	American Journal of Sports Medicine	6.1	IV	Yes	No
Ren et al.	2019	Archives of Orthopaedic and Trauma Surgery	2.6	IV	Yes	No
Schneider et al.	2016	American Journal of Sports Medicine	5.3	IV	Yes	No
Shamrock et al.	2019	Orthopaedic Journal of Sports Medicine	2.7	IV	Yes	No
Singhal et al.	2013	The Bone & Joint Journal	3.0	IV	No	No
Smith et al.	2007	KSSTA	1.6	IV	Yes	No
Tan et al.	2022	European Journal of Orthopaedic Surgery & Traumatology	1.8	IV	Yes	No
Tanos et al.	2023	Medical Sciences	2.1	IV	Yes	No
Testa et al.	2017	KSSTA	3.4	III	Yes	No
Ulrich et al.	2020	Journal of Arthroscopy and Joint Surgery	0.6	IV	Yes	No
Vivekanantha et al.	2023	KSSTA	4.3	IV	Yes	No
Walker et al.	2022	KSSTA	4.3	IV	Yes	No
Wang et al.	2024	KSSTA	4.3	IV	Yes	Yes
Weinberger et al.	2017	KSSTA	3.4	IV	Yes	No
Wilkens et al.	2020	KSSTA	3.4	IV	Yes	Yes
Xing et al.	2020	Journal of Orthopaedic Surgery and Research	2.8	II	Yes	No
Xu et al.	2023	Orthopaedic Surgery	2.3	IV	Yes	No
Yoo et al.	2023	Medicine	1.8	II	Yes	No
Zhang et al.	2020	Orthopaedic Journal of Sports Medicine	2.8	III	No	No