Journal of Clinical Epidemiology

Available online 24 August 2017

In Press, Accepted Manuscript — Note to users

Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy

  • a Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health, 615 North Wolfe Street, Baltimore, MD 21205
  • b Department of Surgery, Johns Hopkins University School of Medicine, 600 North Wolfe Street, Baltimore, MD
  • c The TMJ Association, Ltd., P.O. Box 26770, Milwaukee, WI 53226
  • d Pharmaceutical Health Services Research, University of Maryland School of Pharmacy, 20 North Pine Street, Baltimore, Maryland 21201
  • e Department of Neurology-Neuromuscular Medicine, Johns Hopkins University School of Medicine, 733 North Broadway, Baltimore, MD 21205
  • f Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, 5510 Nathan Shock Drive, Suite 100, Baltimore, MD 21224
  • g Department of Mental Health, Johns Hopkins University Bloomberg School of Public Health, 624 N Broadway, Baltimore, MD 21205
  • h Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, 624 North Broadway, Baltimore, MD 21205
  • i Department of Psychiatry and Behavioral Sciences, The Women's Mood Disorders Center, The Johns Hopkins Hospital, 550 North Broadway, Baltimore, MD 21205
  • j Welch Medical Library, Johns Hopkins University School of Medicine, 2024 Bldg. 1-213, Baltimore, MD 21287
  • k Johns Hopkins University Peabody Institute, 1 East Mount Vernon Place, Baltimore, MD 21202
  • l Malone Center for Engineering in Healthcare, Johns Hopkins University Whiting School of Engineering, Malone Hall, 3400 N. Charles Street, Baltimore, Maryland 21218
Received 28 June 2017, Revised 8 July 2017, Accepted 29 July 2017, Available online 24 August 2017

Abstract

Objective

To determine whether disagreements among multiple data sources affect systematic reviews of randomized clinical trials (RCTs).

Study Design and Setting

Eligible RCTs examined gabapentin for neuropathic pain and quetiapine for bipolar depression, reported in public (e.g., journal articles) and non-public sources (clinical study reports [CSRs] and individual participant data [IPD]).

Results

We found 21 gabapentin RCTs (74 reports, six IPD) and seven quetiapine RCTs (50 reports, one IPD); most were reported in journal articles (18/21 [86%] and 6/7 [86%], respectively). When available, CSRs contained the most trial design and risk of bias information. CSRs and IPD contained the most results. For the outcome domains “pain intensity” (gabapentin) and “depression” (quetiapine), we found single trials with 68 and 98 different meta-analyzable results, respectively; by purposefully selecting one meta-analyzable result for each RCT, we could change the overall result for pain intensity from effective (standardized mean difference [SMD]=-0.45; 95%CI -0.63 to -0.27) to ineffective (SMD=-0.06; 95%CI -0.24 to 0.12). We could change the effect for depression from a medium effect (SMD=-0.55; 95%CI -0.85 to -0.25) to a small effect (SMD=-0.26; 95%CI -0.41 to -0.1).

Conclusions

Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.

Details Of Contributors

Study conception and design: The study design was first described in the application to the Patient-Centered Outcomes Research Institute (PCORI) in 2013. Kay Dickersin was the principal investigator and worked with Tianjing Li, Swaroop Vedula, and Peter Doshi to design the study, write the application, and obtain the funding. Evan Mayo-Wilson drafted the protocol with contributions from other authors.

Acquisition of data: Terrie Cowley, Kay Dickersin, Nicole Fusco, Gillian Gresham, Jennifer Haythornthwaite, James Heyward, Diana Pham, Tianjing Li, Evan Mayo-Wilson, Jennifer Payne, and Elizabeth Tolbert contributed to developing the data extraction forms. Lori Rosman and Claire Taylor designed and ran the electronic searches. Kay Dickersin, Nicole Fusco, Gillian Gresham, James Heyward, Evan Mayo-Wilson, Tianjing Li, and Swaroop Vedula screened studies for inclusion. Lorenzo Bertizzolo, Joseph K Canner, Jeffrey Ehmsen, Nicole Fusco, Gillian Gresham, James Heyward, Diana Pham, Evan Mayo-Wilson, and Catalina Suarez-Cuervo extracted data.

Analysis and interpretation of data: Joseph K Canner, Nicole Fusco, Hwanhee Hong, Nan Guo, and Evan Mayo-Wilson managed the data. Joseph K Canner, Nicole Fusco, Nan Guo, and Hwanhee Hong analyzed data. Kay Dickersin, Jeffrey Ehmsen, Nicole Fusco, Hwanhee Hong, Tianjing Li, Evan Mayo-Wilson, and Elizabeth Stuart contributed to interpretation and data presentation.

Drafting of manuscript: Evan Mayo-Wilson wrote the first draft, with Evan Mayo-Wilson, Tianjing Li and Kay Dickersin providing subsequent revisions.

Critical revision: All authors reviewed, provided critical revisions, and approved the manuscript for publication.

Evan Mayo-Wilson is the guarantor. All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

ETHICS APPROVAL

The study received an exemption from the Johns Hopkins Bloomberg School of Public Health Institutional Review Board (IRB No: 00006324).

SOURCES OF FUNDING

Supported by contract ME 1303 5785 from the Patient Centered Outcomes Research Institute (PCORI) and a fund established at Johns Hopkins for scholarly research on reporting biases by Greene LLP.

ROLE OF THE FUNDING SOURCE

The funders were not involved in the design or conduct of the study, manuscript preparation, or the decision to submit the manuscript for publication.

Correspondence to: Evan Mayo-Wilson, Department of Epidemiology, Johns Hopkins University Bloomberg School of Public Health, 615 North Wolfe Street, E6036, Baltimore, MD 21205, 443-287-5042.

© 2017 The Authors. Published by Elsevier Inc.

Note to users:
Accepted manuscripts are Articles in Press that have been peer reviewed and accepted for publication by the Editorial Board of this publication. They have not yet been copy edited and/or formatted in the publication house style, and may not yet have the full ScienceDirect functionality, e.g., supplementary files may still need to be added, links to references may not resolve yet etc. The text could still change before final publication.

Although accepted manuscripts do not have all bibliographic details available yet, they can already be cited using the year of online publication and the DOI, as follows: author(s), article title, Publication (year), DOI. Please consult the journal's reference style for the exact appearance of these elements, abbreviation of journal names and use of punctuation.

When the final article is assigned to volumes/issues of the Publication, the Article in Press version will be removed and the final version will appear in the associated published volumes/issues of the Publication. The date the article was first made available online will be carried over.