“Over-reviewing” of research? An analysis of orthodontic reviews

Introduction

Research overviews may be undertaken to identify gaps in the literature, evaluate existing systematic reviews (SRs), and summarize evidence. This paper aims to profile overviews that have been conducted in orthodontics and related interventions since 2012 and to evaluate the degree of overlap among these overviews.

Methods

Overviews published between January 1, 2012 and June 20, 2023 were identified using an electronic search involving Google Scholar and PubMed. A descriptive summary was produced, and citation matrices were used to evaluate the percentage of overlap between overviews using corrected covered area and covered area. This was classified as slight, moderate, high, or very high.

Results

A total of 35 overviews were identified across a wide range of topics. Eight overviews included <10 SRs; 21 had 10-20 SRs; and 6 included >20 SRs (median no. of SRs per overview, 15; range, 3-62). Meta-analysis was conducted in only 5 overviews. Overlap between overviews on the same topic ranged from slight (2.7%) to very high (53.8%).

Conclusions

Almost all overview topics address treatments and their effects, with a wide variation in the number and quality of SRs included. There is considerable overlap in some orthodontic overviews, suggesting unnecessary duplication and research waste. Researchers should be encouraged to focus on primary data collection to add more high-quality data to SRs, which will ultimately enhance the yield from secondary and tertiary orthodontic research.

Highlights

•

Overviews of systematic reviews have increased in orthodontics in recent years.
•

There is a wide range in the number of systematic reviews included in overviews.
•

Considerable overlap in orthodontic overviews suggests unnecessary duplication.
•

Low-quality primary data underpin many systematic orthodontic reviews.
•

The need for high-quality prospective primary studies is warranted.

Systematic reviews (SRs), which combine data from high-quality clinical studies into a meta-analysis, are regarded as providing the best evidence for the clinical effectiveness of health care interventions. However, the certainty of any findings relies on the extent of good quality primary research data. The number of published SRs in orthodontics has increased considerably in recent years. Specifically, between January 1, 2000 and August 31, 2020, 322 published reviews were identified in 5 major orthodontic journals namely the American Journal of Orthodontics and Dentofacial Orthopedics, The Angle Orthodontist, European Journal of Orthodontics, Journal of Orthodontics, and Orthodontics and Craniofacial Research and the Cochrane Database of Systematic Reviews. A meta-analysis was reported in only 109 of these (34%), with a median of 5 studies combined in each review. The yield from the preponderance of orthodontic SRs has been questioned, particularly given the relatively modest accumulation of high-quality randomized controlled trials compared with other health care fields. This raises a question regarding the priority that is given to the publication of orthodontic SRs, as there is potentially a greater need for the collection of high-quality primary data to ultimately populate future reviews.

As the volume of SRs has risen in health care, so too has the publication of overviews, with an 8-fold increase concerning health interventions reported in the past 2 decades. Overviews use explicit, systematic methods to identify multiple SRs on a specific research question for the purpose of extracting and analyzing their results. Several alternative terms exist, which include umbrella reviews and reviews of reviews, although there are subtle differences between these terms. Three functions for overviews have been described: the identification of gaps in the literature, evaluation of existing SRs, and summarizing evidence. A summary of the evidence in an overview can only be undertaken if the identified SRs meet the following criteria: the included primary trials overlap, the topic aligns with the scope of the overview, the SRs are of high methodologic quality, and the SRs are contemporary.

Concerns have been raised regarding the overlapping and sometimes conflicting content of not only SRs but also of overviews, which by their very nature may be broad. Furthermore, overviews may also lack methodologic rigor and can be difficult to locate without an objectively derived and validated search strategy, resulting in a call for enhanced methodologic and reporting guidance.

In line with the typology of reviews described by Grant and Booth, the broad aim of this paper is to provide an “overview” of overviews of SRs in orthodontics and related interventions. Specifically, it aimed to (1) profile overviews that have been conducted in orthodontics and related interventions since 2012 and (2) evaluate the degree of overlap between these.

Material and methods

Overviews published between January 1, 2012 and June 20, 2023 were identified using an electronic search involving Google Scholar and PubMed. Search terms used included “overviews,” “umbrella reviews,” “review of systematic reviews,” “orthodontics,” “oral surgery,” “periodontology,” “restorative,” “prosthodontics,” “endodontics,” “oral maxillofacial surgery,” “dental public health,” “dental hygiene,” “orofacial pain,” “oral medicine,” “oral pathology,” “paediatric dentistry,” “oral radiology,” “oral immunology,” “gerodontics,” “oral microbiology,” “special care dentistry,” “dental materials,” “dental psychology,” and “regenerative dentistry.”

A descriptive summary was produced, and citation matrices were used to evaluate the percentage overlap area. The index publication, or index case, is the first time a primary publication occurs. The covered area (CA) is ascertained through the division of the total number of included publications in an overview (regardless of their overlap) by the product of the number of index publications (rows in a citation matrix) and the number of included overviews (columns in a citation matrix). As CA may be overly affected by 1 review in an overview containing a large number of primary publications compared with other included reviews, the use of corrected covered area (CCA) has been proposed. The CCA accounts for this effect by subtracting the number of index publications from both the total number of included publications and the product of rows and columns in the citation matrix.

For the purpose of addressing overlap between overviews, in this paper, the index publication or index case refers to the first time an SR publication occured. CCA was classified according to Pieper et al as follows: slight (0-5), moderate (6-10), high (11-15), and very high (>15).

Results

A total of 36 overviews were initially identified ( Table I ), with 1 of these subsequently excluded as it involved a review of randomized controlled trials rather than SRs. Within the included overviews, 30 involved reviews of SRs only, and 5 involved reviews of SRs and meta-analyses. Throughout the text, all were referred to as SRs. The 3 journals that published the most orthodontic overviews were the European Journal of Orthodontics, Clinical Oral Investigations, and Journal of Oral Rehabilitation; all of which published 3 orthodontic overviews. In total, 14 overviews were published from Asia, 13 from Europe, 6 from South America, and 2 from Africa.

Table I

Overviews in orthodontics 2012-2023 (June)

First author, y, country, journal, title	No. of SRs and/or MAs	Ballard and Montgomery function	Methodologic quality of SRs assessment tool	No. of primary studies	Meta-analysis; possible? If not, why?	Overlap assessed
D’Anto, Italy Journal of Oral Rehabilitation Class II functional orthopaedic treatment: a systematic review of systematic reviews	14	2, 3	3 high, 8 moderate, 3 low, ‡	156	No; reason not given	2 studies in 3 searches
Johal et al, United Kingdom Sleep and Breathing Mandibular advancement splint (MAS) therapy for obstructive sleep apnoea—an overview and quality assessment of systematic reviews	8	2, 3	4 high, 3 moderate, 1 low, ‡	338	No; reason not given	No
Bucci et al, Italy Journal of Oral Rehabilitation Dental and skeletal effects of palatal expansion techniques: a systematic review of the current evidence from systematic reviews and meta-analyses	12 (8 SRs and 4 MAs)	2, 3	5 high, 7 moderate, ‡	133	No; reason not given	5 RCTs in 3 SRs/MAs 6 RCTs in 2 SRs/MAs
Jamilian et al, Iran Journal of Orthodontics Methodologic quality and outcome of systematic reviews reporting on orthopaedic treatment for Class III malocclusion: overview of systematic reviews	14 (11 SRs and 3 MAs)	2, 3	10 high, 3 moderate, 1 low, ‡	160	No; reason not given	No
Tan et al, China PLoS One Effects of mandibular setback with or without maxillary advancement osteotomies on pharyngeal airways: an overview of systematic reviews	6	2, 3	3 high, 2 moderate, 1 low, ‡	98	Yes (n = 2)	No
Tan et al, China PLoS One How does mandibular advancement with or without maxillary procedures affect pharyngeal airways? An overview of systematic reviews	11	2, 3	2 high, 6 moderate, 3 low, ‡	64	Yes (n = 2)	Overlapping samples in 2 studies
Yi et al, China Journal of Oral Rehabilitation Effectiveness of adjunctive interventions for accelerating orthodontic tooth movement: a systematic review of systematic reviews	11	2, 3	3 high, 6 moderate, 2 low, ‡	108	No; reason not given	No
Zheng, China Medicine Implants for orthodontic anchorage: an overview	23	2	5 high, 15 moderate, 3 low, ‡	666	No; large numbers and complex outcomes	No
Elkordy, Egypt Seminars in Orthodontics Do fixed orthodontic appliances adversely affect the periodontium? A systematic review of systematic reviews	19	2, 3	2 high, 8 moderate, 7 low 2 critically low, †	206	No; reason not given	No
Haas Junior, Spain International Journal of Oral and Maxillofacial Surgery Hierarchy of surgical stability in orthognathic surgery: overview of systematic reviews	15 (8 SRs, 7 MAs)	2, 3	11 high, 4 moderate, †	148	No; reason not given	No
Niño-Sandoval, Brazil Brazilian Oral Research Incidence of condylar resorption after bimaxillary, LeFort I, and mandibular surgery: an overview	5	2, 3	1 high, 1 moderate, 3 low, † , #	54	No; not possible; high heterogeneity	No
Sato, Japan Japanese Dental Science Review Review of systematic reviews on mandibular advancement oral appliance for obstructive sleep apnea: the importance of long-term follow-up	27	3	Quality assessment not done	466	No, Reason not given	No
Bakdach, Syria Dental and Medical Problems Effectiveness of different adjunctive interventions in the management of orthodontically induced white spot lesions: a systematic review of systematic reviews and meta-analyses	13	3	4 moderate, 3 low, 6 critically low, †	122	No; reason not given	No
Barone, Italy Journal of Stomatology, Oral and Maxillofacial Surgery Surgery-first orthognathic approach vs conventional orthognathic approach: a systematic review of systematic reviews	10	2	1 high, 1 moderate, 3 low, 5 critically low, †	90	No; could not be conducted	No
Francisco, Portugal Journal of Clinical and Experimental Dentistry Condylar form alteration on skeletal Class II patients that underwent orthognathic surgery: an overview of systematic reviews	4	3	2 moderate, 2 low, †	118	Yes (n = 4)	No
Mheissen, Syria Journal of Orthodontics The effectiveness of surgical adjunctive procedures in the acceleration of orthodontic tooth movement: a systematic review of systematic reviews and meta-analysis	14	2, 3	5 moderate, 7 low, 2 critically low, †	118	Yes (n = 4)	No
Ramírez-Ossa, Colombia Journal of Evidence-Based Dental Practice An umbrella review of the effectiveness of Temporary Anchorage Devices and the factors that contribute to their success or failure	17 (7 SRs and 10 MAs)	1, 3	12 high, 5 moderate, †	444	No; reason not given	No
Yassir et al, Iraq European Journal of Orthodontics The impact of labial fixed appliance orthodontic treatment on patient expectation, experience, and satisfaction: an overview of systematic review	9	3	5 moderate, 2 low, 2 critically low, †	158	No; not possible; qualitative data	No
Bravo, Chile British Journal of Oral and Maxillofacial Surgery Effectiveness of glucocorticoids in orthognathic surgery: an overview of systematic reviews	3	3	1 moderate, 2 very low, ¶	19	No, reason not given	The citation matrix showed 4 overlapping studies
Gil et al, Brazil British Journal of Oral and Maxillofacial Surgery Antibiotic prophylaxis in orthognathic surgery: an overview of systematic reviews	4 SRs; 2 MAs	2	All high, †	49	No; reason not given	No
Sardana, China International Journal of Paediatric Dentistry Prevention of demineralization during multi-bracketed fixed orthodontic treatment: an overview of systematic reviews	29	3	2 high, 4 moderate, 4 low, 19 critically low, † , §	128	No; reason not given	CCA determined a slight (2.8%) overlap
Yassir, Iraq European Journal of Orthodontics Orthodontic treatment and root resorption: an overview of systematic reviews	28	3	1 high, 19 moderate, 3 low, 5 critically low, †	379	No; not possible; high heterogeneity	No
Barone, Italy Korean Journal of Orthodontics Incidence and management of condylar resorption after orthognathic surgery: an overview	10	2, 3	7 low, 3 critically low, †	218	No; could not be performed	No
Cremona, Malta European Journal of Orthodontics Quality-of-life improvement, psychosocial benefits, and patient satisfaction of patients undergoing orthognathic surgery: a summary of systematic reviews	12	3	6 moderate, 2 low, 4 critically, low, †	228	No; limited availability of interventional studies	No
Abd El-Ghafour, Egypt Evidence-based Dentistry Is maxillary expansion effective in treatment of obstructive sleep apnoea syndrome? A systematic review of systematic reviews	14	3	1 high, 2 moderate, 5 low 6 critically low, †	163	No; not possible because of high heterogeneity	No
Gasparro, Italy Japanese Dental Science Review Effectiveness of surgical procedures in the acceleration of orthodontic tooth movement: findings from systematic reviews and meta-analyses	28	3	12 high, 8 moderate, 1 low 7 critically low, †	322	No; reason not given	No
Mukhopadhyay, India Journal of Indian Orthodontic Society An umbrella review of systematic reviews with or without meta-analysis assessing treatment outcomes and efficiency of self-ligating brackets	16	2, 3	10 low ROB, 6 unclear ROB, §	165	No; reason not given	CCA determined low (14%) overlap
Togninalli, Switzerland Journal of Stomatology, Oral and Maxillofacial Surgery Condylar resorption following mandibular advancement or bimaxillary osteotomies: a systematic review of systematic reviews	10	2, 3	All low, †	180	No; reason not given	No
Tran, United Kingdom International Journal of Oral and Maxillofacial Surgery Management of temporomandibular disorders: a rapid review of systematic reviews and guidelines	62	3	3 high, 51 moderate, 2 low 6 critically low, †	886	No; reason not given	No
Ventura, Portugal Journal of Clinical Medicine Miniscrew-Assisted Rapid Palatal Expansion (MARPE): an umbrella review	4	2, 3	1 high, 1 low, 2 critically low, †	25	No; reason not given	No
Yassir, Iraq Korean Journal of Orthodontics Which anchorage device is the best during retraction of anterior teeth? An overview of systematic reviews	14	2, 3	2 high, 9 moderate, 1 low, 2 critically low, †	144	No; lack of primary data	No
Yassir, Iraq Clinical Oral Investigations Clinical effectiveness of clear aligner treatment compared to fixed appliance treatment: an overview of systematic reviews	18	3	1 high, 17 moderate, †	180	No; not possible because of a lack of primary data; high heterogeneity	No
Rocha, Brazil Clinical Oral Investigations Efficiency of maxillomandibular advancement for the treatment of obstructive apnea syndrome: a comprehensive overview of systematic reviews	12	2	6 high, 6 moderate, † , #	321	No; reason not given	No
Di Spirito, Italy Dentistry Journal Periodontal management in periodontally healthy orthodontic patients with fixed appliances: an umbrella review of self-care instructions and evidence-based recommendations	17	2, 3	5 low, 12 critically low, †	160	No; not possible; high heterogeneity	No
Ribeiro, Brazil Clinical Oral Investigations Impact of malocclusion treatments on oral health-related quality of life: an overview of systematic reviews	15	2, 3	2 high, 1 moderate, 3 low, 9 critically low, †	340	Yes (n = 4)	No

Note. Overview functions from Ballard and Montgomery : 1, Identify gaps in the literature in which multiple comparable studies may exist but a research synthesis has not been performed; 2, Compare and contrast existing systematic reviews; and 3, Provide a summary of evidence from existing systematic reviews, with or without synthesis.

MAs, meta-analyses ; RCT , randomized controlled trial; ROB , risk of bias.

† A Measurement Tool to Assess Systematic Reviews, version 2.

‡ A Measurement Tool to Assess Systematic Reviews.

§ Risk of Bias in Systematic Reviews.

¶ Grading of Recommendations, Assessment, Development, and Evaluations.

# Glenny scale.

For focus and content:

1.

Five overviews were published in 2 journals listed in the top-ranked orthodontic journals, namely the European Journal of Orthodontics (n = 3) and the Korean Journal of Orthodontics (n = 2) ( https://www.scimagojr.com ).
2.

There were no Cochrane Collaboration overviews of SRs in orthodontics.
3.

Topics covered in the overviews included orthopedic treatment for Class II or III malocclusions (n = 2); adjunctive procedures for accelerated tooth movement (n = 3); orthodontic anchorage, including temporary anchorage devices (n = 3); fixed appliances and the periodontium (n = 2); treatment outcomes and efficiency of self-ligating brackets (n = 1); effectiveness of clear aligner treatment vs fixed appliances (n = 1); iatrogenic effects of orthodontic treatment (demineralization and root resorption) (n = 3); effectiveness of treatments of obstructive sleep apnea (n = 6); rapid maxillary expander and miniscrew-assisted rapid palatal expansion (n = 2); management of temporomandibular disorder (n = 1); condylar form or condylar resorption after orthognathic surgery (n = 4); orthognathic surgery (stability, surgery-first approach, glucocorticoids, and antibiotic prophylaxis; n = 4); and impact of orthodontics or surgical treatment or both on oral health–related quality of life or psychological outcomes (n = 3).
4.

Eight overviews included <10 SRs; 21 had 10-20 SRs; and 6 included >20 SRs (median no. of SRs per overview, 15; range, 3-62 ^,). The overview with the least number of SRs addressed the effectiveness of glucocorticoids in orthognathic surgery, whereas the overview with the greatest number of SRs dealt with the management of temporomandibular disorder.
5.

With regard to the Ballard and Montgomery justification for overviews, 18 overviews aimed to summarize evidence and evaluate existing SRs. Twelve aimed to summarize evidence only; 4 considered the evaluation of existing SRs only; and 1 overview aimed to identify gaps in the literature and summarize the evidence.

For methodologic quality:

1.

A Measurement Tool to Assess Systematic Reviews, version 2 (AMSTAR-2) was used most commonly to assess methodologic quality (n = 24), followed by AMSTAR (n = 8), Risk of Bias in Systematic Reviews (n = 2), Glenny scale (n =2) and Grading of Recommendations, Assessment, Development, and Evaluations (n = 1). One overview used both AMSTAR-2 and the Risk of Bias in Systematic Reviews. Both overviews that used the Glenny scale did so in conjunction with AMSTAR-2.
2.

Across all overviews in which quality assessments were carried out, 99 (20%) SRs were deemed to be of high quality, 209 (42%) of moderate quality, 85 (17%) of low quality, and 94 (19%) of critically low quality. Approximately 2% were deemed to have a low risk of bias, and 1% had an unclear risk of bias.
3.

One overview included 19 (65%) SRs of critically low quality, which addressed the prevention of demineralization during multibracketed fixed appliance orthodontic treatment.
4.

The overview with the greatest number of high-quality SRs (n = 12) was related to temporary anchorage devices, whereas the following topics each had only 1 SR rated as high-quality: Surgery-first vs a conventional orthognathic approach, root resorption, maxillary expansion in the treatment of obstructive sleep apnea, clear aligner treatment compared with fixed appliance treatment, condylar resorption after orthognathic surgery, and miniscrew-assisted rapid palatal expansion.
5.

Meta-analysis was conducted in only 5 overviews addressing the following: (1) the effectiveness of surgical adjunctive procedures in the acceleration of orthodontic tooth movement (4 of 14 SRs) ; (2) the effect of a mandibular setback with or without maxillary advancement osteotomies on the pharyngeal airway (2 of 6 SRs) ; (3) the effect of mandibular advancement with or without maxillary procedures on the pharyngeal airway (2 of 11 SRs) ; (4) Condylar form alteration in skeletal Class II orthognathic surgery patients (all 4 SRs) ; and (5) the impact of treatment of malocclusion on oral health–related quality of life (4 of 15 SRs).

For overlap overviews, there were a number of overlapping SRs in overviews of similar topics, namely orthodontically-related demineralization, orthodontic tooth movement, obstructive sleep apnea, condylar resorption, and anchorage devices ( Table II ). The calculated percentage overlap between individual overviews on the same topic ranged from 2.7% (slight; ^, obstructive sleep apnea) to 53.8% (very high; ^, condylar resorption) ( Supplementary Tables I-X ).

Table II

Summary of overlap for each topic overall and within individual overviews for the topics with >2 overviews overlapping

Overview topic	Overviews with overlap	Total no. of SRs	Overlapping SRs	CCA	CA
Orthodontically-related demineralization	2	31	11	0.355/35.5% ^∗∗∗∗	0.677/67.7%
Orthodontic tooth movement (overall)	3	36	17	0.236/23.6% ^∗∗∗∗	0.490/49.0%
Overview pair ^,		24	1	0.042/4.2% ^∗	0.520/52.0%
Overview pair ^,		33	6	0.181/18.1% ^∗∗∗∗	0.590/59.0%
Overview pair ^,		31	11	0.354/35.4% ^∗∗∗∗	0.677/67.7%
Obstructive sleep apnea (overall)	4	47	9	0.064/6.4% ^∗∗	0.298/29.8%
Overview pair ^,		15	2	0.133/13.3% ^∗∗∗	0.567/56.7%
Overview pair ^,		17	6	0.353/35.3% ^∗∗∗∗	0.676/67.6%
Overview pair ^,		37	1	0.027/2.7% ^∗	0.514/51.4%
Condylar resorption (overall)	4	13	16	0.410/41.0% ^∗∗∗∗	0.558/55.8%
Overview pair ^,		10	4	0.400/40.0% ^∗∗∗∗	0.700/70.0%
Overview pair ^,		10	4	0.400/40% ^∗∗∗∗	0.700/70.0%
Overview pair ^,		13	7	0.538/53.8% ^∗∗∗∗	0.769/76.9%
Overview pair ^,		6	3	0.500/50.0% ^∗∗∗∗	0.750/75.0%
Overview pair ^,		10	5	0.500/50.0% ^∗∗∗∗	0.750/75.0%
Overview pair ^,		10	5	0.500/50.0% ^∗∗∗∗	0.750/75.0%
Anchorage devices (overall)	3	41	12	0.146/14.6% ^∗∗∗	0.431/43.1%
Overview pair ^,		28	3	0.107/10.7% ^∗∗	0.554/55.4%
Overview pair ^,		30	9	0.300/30.0% ^∗∗∗∗	0.650/65.0%
Overview pair ^,		34	2	0.059/5.9% ^∗∗∗	0.529/52.9%