# A Note on Cherry-Picking in Meta-Analyses

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methods

#### 2.1. Chance of Cherry-Picking in a Meta-Analysis

**Theorem**

**1.**

**Theorem**

**2.**

#### 2.2. Extension to a Random-Effect Model

## 3. Simulation Experiments

#### 3.1. Simulation Settings

#### 3.2. Simulation Results

## 4. Medical Application Studies

#### 4.1. Case 1 Example: Clinical Trials on the Effectiveness of Magnesium for Reducing the Mortality of Acute Myocardial Infarction Patients

#### 4.2. Case 2 Example: Clinical Trials on the Effectiveness of St. John’s Wort for Treating Depression

## 5. Discussion

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## Abbreviations

RCT | randomized clinical trials |

## Appendix A

**Lemma**

**A1.**

**Proof.**

## Appendix B. Proof of Theorem 1

**Proof.**

## Appendix C. Proof of Theorem 2

**Proof.**

## References

- Higgins, J.P.; Thomas, J.; Chandler, J.; Cumpston, M.; Li, T.; Page, M.J.; Welch, V.A. Cochrane Handbook for Systematic Reviews of Interventions; John Wiley & Sons: Hoboken, NJ, USA, 2019. [Google Scholar]
- Whitehead, A. Meta-Analysis of Controlled Clinical Trials; John Wiley & Sons: Hoboken, NJ, USA, 2002; Volume 7. [Google Scholar]
- DerSimonian, R.; Laird, N. Meta-analysis in clinical trials. Control. Clin. Trials
**1986**, 7, 177–188. [Google Scholar] [CrossRef] [PubMed] - Egger, M.; Smith, G.D.; Schneider, M.; Minder, C. Bias in meta-analysis detected by a simple, graphical test. BMJ
**1997**, 315, 629–634. [Google Scholar] [CrossRef] [PubMed] - Hopewell, S.; Clarke, M.J.; Lefebvre, C.; Scherer, R.W. Handsearching versus Electronic Searching to Identify Reports of Randomized Trials. In Cochrane Database of Systematic Reviews; John Wiley & Sons: Hoboken, NJ, USA, 2007. [Google Scholar]
- Ioannidis, J.P. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA
**1998**, 279, 281–286. [Google Scholar] [CrossRef] - Higgins, J.P.; Thompson, S.G. Quantifying heterogeneity in a meta-analysis. Stat. Med.
**2002**, 21, 1539–1558. [Google Scholar] [CrossRef] [PubMed] - Yoneoka, D.; Henmi, M. Meta-analytical synthesis of regression coefficients under different categorization scheme of continuous covariates. Stat. Med.
**2017**, 36, 4336–4352. [Google Scholar] [CrossRef] - Yoneoka, D.; Henmi, M.; Sawada, N.; Inoue, M. Synthesis of clinical prediction models under different sets of covariates with one individual patient data. BMC Med. Res. Methodol.
**2015**, 15, 1. [Google Scholar] [CrossRef] [PubMed] - Sutton, A.J.; Abrams, K.R.; Jones, D.R.; Jones, D.R.; Sheldon, T.A.; Song, F. Methods for Meta-Analysis in Medical Research; Wiley: Hoboken, NJ, USA, 2000. [Google Scholar]
- Hedges, L.V.; Olkin, I. Statistical Methods for Meta-Analysis; Academic Press: Cambridge, MA, USA, 2014. [Google Scholar]
- Borenstein, M.; Hedges, L.V.; Higgins, J.; Rothstein, H.R. Introduction to Meta-Analysis; John Wiley & Sons: New York, NY, USA, 2009. [Google Scholar]
- Cooper, H.; Hedges, L.V.; Valentine, J.C. The Handbook of Research Synthesis and Meta-Analysis; Russell Sage Foundation: New York, NY, USA, 2009. [Google Scholar]
- Page, M.J.; Shamseer, L.; Altman, D.G.; Tetzlaff, J.; Sampson, M.; Tricco, A.C.; Catala-Lopez, F.; Li, L.; Reid, E.K.; Sarkis-Onofre, R.; et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: A cross-sectional study. PLoS Med.
**2016**, 13, e1002028. [Google Scholar] [CrossRef] [PubMed] - Tawfik, G.M.; Giang, H.T.N.; Ghozy, S.; Altibi, A.M.; Kandil, H.; Le, H.H.; Eid, P.S.; Radwan, I.; Makram, O.M.; Hien, T.T.T.; et al. Protocol registration issues of systematic review and meta-analysis studies: A survey of global researchers. BMC Med. Res. Methodol.
**2020**, 20, 213. [Google Scholar] [CrossRef] [PubMed] - Clarke, M.J.; Stewart, L.A. Systematic Reviews: Obtaining data from randomised controlled trials: How much do we need for reliable and informative meta-analyses? BMJ
**1994**, 309, 1007–1010. [Google Scholar] [CrossRef] - Ahmed, I.; Sutton, A.J.; Riley, R.D. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: A database survey. BMJ
**2012**, 344, d7762. [Google Scholar] [CrossRef] [PubMed] - Grégoire, G.; Derderian, F.; Le Lorier, J. Selecting the language of the publications included in a meta-analysis: Is there a Tower of Babel bias? J. Clin. Epidemiol.
**1995**, 48, 159–163. [Google Scholar] [CrossRef] - Egger, M.; Smith, G.D. Meta-analysis bias in location and selection of studies. BMJ
**1998**, 316, 61–66. [Google Scholar] [CrossRef] [PubMed] - Knight, J. Null and void. Nature
**2003**, 422, 554–555. [Google Scholar] [CrossRef] [PubMed] - Komiyama, J.; Maehara, T. A Simple Way to Deal with Cherry-picking. arXiv
**2018**, arXiv:1810.04996. [Google Scholar] - Brockwell, S.E.; Gordon, I.R. A comparison of statistical methods for meta-analysis. Stat. Med.
**2001**, 20, 825–840. [Google Scholar] [CrossRef] - Teo, K.K.; Yusuf, S.; Collins, R.; Held, P.H.; Peto, R. Effects of intravenous magnesium in suspected acute myocardial infarction: Overview of randomised trials. BMJ
**1991**, 303, 1499–1503. [Google Scholar] [CrossRef] [PubMed] - International Study Group of Infarct Survival. ISIS-4: A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction. Lancet
**1995**, 345, 669–685. [Google Scholar] [CrossRef] - Eggar, M.; Smith, G.D. Misleading meta-analysis. Lessons from “an effective, safe, simple” intervention that wasn’t. BMJ
**1995**, 310, 752–754. [Google Scholar] - Linde, K.; Berner, M.; Egger, M.; Mulrow, C. St John’s wort for depression: Meta-analysis of randomised controlled trials. Br. J. Psychiatry
**2005**, 186, 99–107. [Google Scholar] [CrossRef] - Malzahn, U.; Böhning, D.; Holling, H. Nonparametric estimation of heterogeneity variance for the standardised difference used in meta-analysis. Biometrika
**2000**, 87, 619–632. [Google Scholar] [CrossRef] - Cornell, J.E.; Mulrow, C.D.; Localio, R.; Stack, C.B.; Meibohm, A.R.; Guallar, E.; Goodman, S.N. Random-effects meta-analysis of inconsistent effects: A time for change. Ann. Intern. Med.
**2014**, 160, 267–270. [Google Scholar] [CrossRef] [PubMed] - Langan, D.; Higgins, J.P.; Jackson, D.; Bowden, J.; Veroniki, A.A.; Kontopantelis, E.; Viechtbauer, W.; Simmonds, M. A comparison of heterogeneity variance estimators in simulated random-effects meta-analyses. Res. Synth. Methods
**2019**, 10, 83–98. [Google Scholar] [CrossRef] [PubMed] - Henmi, M.; Copas, J.B.; Eguchi, S. Confidence intervals and P-values for meta-analysis with publication bias. Biometrics
**2007**, 63, 475–482. [Google Scholar] [CrossRef] [PubMed] - Copas, J.; Jackson, D. A bound for publication bias based on the fraction of unpublished studies. Biometrics
**2004**, 60, 146–153. [Google Scholar] [CrossRef] [PubMed] - Marchal, O.; Arbel, J. On the sub-Gaussianity of the Beta and Dirichlet distributions. Electron. Commun. Probab.
**2017**, 22, 1–14. [Google Scholar] [CrossRef]

**Figure 1.**Simulation results for the standard hypotheses for Case 1, where meta-analysts overstate the effect of the treatment, regardless of there being no actual effect: Proportion of False Conclusions (i.e., type 1 error). ($a,b$) indicates $\theta =a$ and $\tau =b$.

**Figure 2.**Simulation results for the standard hypotheses for Case 2, where meta-analysts understate the effect of the treatment, regardless of the actual effect: Proportion of False Conclusions (i.e., type 2 error). ($a,b$) indicates $\theta =a$ and $\tau =b$.

**Figure 3.**Meta-analysis of the results of 16 RCTs on the effectiveness of magnesium for reducing mortality following AMI.

**Figure 4.**Meta-analysis of the results from nine RCTs on the effectiveness of St. John’s wort for treating depression.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Yoneoka, D.; Rieck, B.
A Note on Cherry-Picking in Meta-Analyses. *Entropy* **2023**, *25*, 691.
https://doi.org/10.3390/e25040691

**AMA Style**

Yoneoka D, Rieck B.
A Note on Cherry-Picking in Meta-Analyses. *Entropy*. 2023; 25(4):691.
https://doi.org/10.3390/e25040691

**Chicago/Turabian Style**

Yoneoka, Daisuke, and Bastian Rieck.
2023. "A Note on Cherry-Picking in Meta-Analyses" *Entropy* 25, no. 4: 691.
https://doi.org/10.3390/e25040691