Letters

RESEARCH LETTER
Table. Elements Extracted From Each of 324 Systematic Reviews
and Meta-analyses
Assessment of Publication Bias
and Systematic Review Findings Element No. (%)
in Top-Ranked Otolaryngology Journals Year of publication
Systematic reviews are the highest level of evidence in oto- 2011 21 (6.5)
laryngology clinical practice guidelines.1 However, they gen- 2012 41 (12.7)
erally present only formally published data, which may lead 2013 42 (13.0)
to an inherent problem caused by publication bias, that is, a 2014 51 (15.7)
strong bias to publish only studies that show significant 2015 86 (26.5)
results.2,3 This bias can be mitigated through both the search 2016 83 (25.6)
methods selected and the statistical methods used. Journal
In the present study, we address the following 4 specific American Journal of Otolaryngology 6 (1.8)
research topics: (1) the techniques systematic reviewers used Clinical Otolaryngology 33 (10.2)
to mitigate publication bias during the search process; (2) the International Journal of Otolaryngology 1 (0.3)
statistical methods used to evaluate the existence of publica- JAMA Otolaryngology–Head & Neck Surgery 16 (4.9)
tion bias during data synthesis; (3) whether a difference ex-
Journal of Otolaryngology–Head and Neck Surgery 17 (5.2)
isted in the frequency of publication bias evaluation among
The Laryngoscope 134 (41.4)
systematic reviews that did or did not endorse the Preferred
Otolaryngology–Head and Neck Surgery 117 (36.1)
Reporting Items for Systematic Reviews and Meta-analyses
Top databases searcheda
(PRISMA) reporting guideline; and (4) for systematic reviews
Medline 213 (65.7)
not reporting a publication bias evaluation, whether there was
Embase 205 (63.3)
sufficient information contained within the published report
CENTRAL 62 (19.1)
to conduct an independent evaluation, and if so, the findings
Cochrane 183 (56.5)
of such evaluations.
CINAHL 51 (15.7)
Scotus 74 (22.8)
Methods | The following journals were identified through the
use of Google Scholar h5-index scores: American Journal of No. of databases searched, mean (IQR) 3.7 (3.0-4.0)

Otolaryngology, Otolaryngology–Head and Neck Surgery, JAMA Gray literature searched

Otolaryngology–Head & Neck Surgery, International Journal of Yes 67 (20.7)

Otolar yngolog y, Clinical Otolar yngolog y, Jo ur nal of No 251 (77.5)
Otolaryngology–Head and Neck Surgery, and The Laryngoscope. Unspecified 6 (1.8)
A PubMed search was conducted on October 26, 2016, to Non-English languages searched
identify all relevant systematic reviews within these top- Yes 91 (28.1)
ranked otolaryngology journals. To be eligible for inclusion No 181 (55.9)
as a systematic review, an article must have reported summa- Unspecified 52 (16.0)
rized evidence across numerous studies and must have Hand search performed
provided information on the search strategy, such as search Yes 12 (3.7)
terms, databases, and inclusion and exclusion criteria. Spe- No 310 (95.7)
cific elements were retrieved from each review, including Unspecified 2 (0.6)
the title, authors, year of publication, journal name, type Clinical trials registries searched
of systematic review or meta-analysis, primary study type, Yes 28 (8.6)
whether the systematic reviewers manually searched refer-
No 295 (91.0)
ence lists, methods used to evaluate publication bias, and ad-
Meta-analysis included
ditional elements, which are given in the Table.
Yes 161 (49.7)
After data extraction, evidence for publication bias was as-
No 163 (50.3)
sessed among the systematic reviews that contained at least 1
No. of primary studies in largest meta-analysis, mean (IQR) 14.9 (6.0-14.0)
meta-analysis with 10 or more primary studies that did not
Publication bias mentioned
evaluate for publication bias. The following methods were used
Yes 213 (65.7)
in the present study to assess publication bias: the Egger re-
No 111 (52.1)
gression test,4 Duval and Tweedie trim-and-fill method,5 and
Begg rank correlation test.6 Data were analyzed using Stata, (continued)

jamaotolaryngology.com (Reprinted) JAMA Otolaryngology–Head & Neck Surgery February 2019 Volume 145, Number 2 187

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by a UNIVERSITY OF SYDNEY LIBRARY User on 07/09/2019

 Letters

ranking otolaryngology journals did not often mention, plan

Table. Elements Extracted From Each of 324 Systematic Reviews
and Meta-analyses (continued) for, or formally evaluate for the presence of publication bias.
This conclusion was reached on the basis of 3 findings: most
Element No. (%)
of the SRs included in the present study did not search the gray
Publication bias formally evaluated
literature, did not report the use of a reporting guideline, and
Yes 257 (79.3)
did not assess for publication bias. The limitation to this study
No 67 (20.7)
was that we restricted our search to high-ranking journals in
Publication bias found otolaryngology. Therefore, our findings may not be appli-
Yes 26 (8.0) cable to lower-ranking journals in the field. Publication bias
Probable 1 (0.3) appears to be a pervasive problem in medical research, and in-
No 37 (11.4) terventions may be needed to encourage the publication of
Unspecified 260 (80.2) high-quality research regardless of the strength or direction
Reporting guidelines useda of the study findings.
PRISMA 133 (41.0)
MOOSE 8 (2.5) Andrew Ross, BS
CONSORT 5 (1.5) Craig Cooper, BS
AHRQ 2 (0.6) Harrison Gray, BS
NHS CRD 1 (0.3) Blake Umberham, DO
STREGA 1 (0.3) Matt Vassar, PhD
None mentioned 179 (55.2)
Author Affiliations: Oklahoma State University Center for Health Sciences,
Abbreviations: AHRQ, Agency for Healthcare Research and Quality; Tulsa (Ross, Cooper, Gray, Vassar); Department of Anesthesiology,
CONSORT, Consolidated Standards of Reporting Trials; IQR, interquartile range, University of Minnesota, Minneapolis (Umberham).
MOOSE, Meta-analysis Of Observational Studies in Epidemiology; Accepted for Publication: September 23, 2018.
NHS CRD, National Health Service Centre for Reviews and Dissemination;
PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses; Corresponding Author: Andrew Ross, BS, Oklahoma State University Center
STREGA, Strengthening the Reporting of Genetic Association Studies. for Health Sciences, 1111 W 17th St, Tulsa, OK 74107 (aeross@okstate.edu).
a
Many of the systematic reviews and meta-analyses used more than 1 database Published Online: December 6, 2018. doi:10.1001/jamaoto.2018.3301
or reporting guideline. Author Contributions: Mr Ross and Dr Vassar had full access to all of the data in
the study and take responsibility for the integrity of the data and the accuracy
of the data analysis.
Concept and design: Umberham, Vassar.
version 13.1 (StataCorp) and Comprehensive Meta-Analysis
Acquisition, analysis, or interpretation of data: Ross, Cooper, Gray, Umberham.
(Biostat). Drafting of the manuscript: Ross, Cooper, Gray, Umberham.
Critical revision of the manuscript for important intellectual content: Ross,
Results | In total, 324 systematic reviews were included in the Vassar.
Statistical analysis: Ross, Gray, Umberham.
present study. Extracted data elements are given in the Table.
Administrative, technical, or material support: Ross.
Of the 324 reviews, 67 (21%) searched the gray literature and Supervision: Umberham, Vassar.
145 (45%) mentioned the use of a reporting guideline. Of these Conflict of Interest Disclosures: None reported.
145 reviews, 133 (92%) used the PRISMA reporting guideline. 1. Bhattacharyya N, Gubbels SP, Schwartz SR, et al. Clinical practice guideline:
Only 69 of the 324 reviews (22%) could appropriately evalu- benign paroxysmal positional vertigo (update) executive summary. Otolaryngol
ate for publication bias (they contained at least 1 meta- Head Neck Surg. 2017;156(3):403-416. doi:10.1177/0194599816689660
analysis with at least 10 primary studies). Of these 69 re- 2. Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data.
J R Stat Soc Ser A Stat Soc. 1988;151(3):419. doi:10.2307/2982993
views, 38 (55%) formally assessed for publication bias. There
3. Light RJ. Accumulating evidence from independent studies: what we can win
was no significant association between the evaluation of pub-
and what we can lose. Stat Med. 1987;6(3):221-231. doi:10.1002/sim.4780060304
lication bias and the use of the PRISMA reporting guideline
4. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis
(odds ratio, 1.67; 95% CI, 0.58-4.85). The most common method detected by a simple, graphical test. BMJ. 1997;315(7109):629-634. doi:10.1136/
used by these 38 systematic reviews for evaluating publica- bmj.315.7109.629
tion bias was visualization of a funnel plot (35 [92%]). Only 9 5. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of
of the 31 eligible reviews that did not assess for publication bias testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56
(2):455-463. doi:10.1111/j.0006-341X.2000.00455.x
contained sufficient data for analysis. The present study found
that all 9 of these reviews likely had publication bias. These 9 6. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test
for publication bias. Biometrics. 1994;50(4):1088-1101. doi:10.2307/2533446
reviews included 22 meta-analyses. Using the Duval and
Tweedie trim-and-fill test, we found that 18 of the 22 meta-
analyses had evidence of publication bias. Using the Egger re- Variation in the Quality of Head and Neck Cancer
gression test, Duval and Tweedie trim-and-fill method, and Care in the United States
Begg rank correlation test, we found that 10 of the meta- Variation in clinical care is one of the biggest obstacles facing
analyses showed evidence of publication bias. health care organizations that prevents them from improving
outcomes.1 In previous work, Cramer et al2 validated 5 qual-
Discussion | Our findings indicate that researchers who con- ity metrics for head and neck cancer that meet validity crite-
ducted systematic reviews that were published in high- ria. High adherence to these metrics was associated with a 19%

188 JAMA Otolaryngology–Head & Neck Surgery February 2019 Volume 145, Number 2 (Reprinted) jamaotolaryngology.com

© 2018 American Medical Association. All rights reserved.

Downloaded From: https://jamanetwork.com/ by a UNIVERSITY OF SYDNEY LIBRARY User on 07/09/2019