Next Article in Journal
A Genome-Wide Association Study of Dementia Using the Electronic Medical Record
Next Article in Special Issue
Big Data in Chronic Kidney Disease: Evolution or Revolution?
Previous Article in Journal
Artificial Intelligence in Bladder Cancer Diagnosis: Current Applications and Future Perspectives
Previous Article in Special Issue
In Silico Protein Structure Analysis for SARS-CoV-2 Vaccines Using Deep Learning
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Towards Automated Meta-Analysis of Clinical Trials: An Overview

Department of Business and Organization Administration, University of Peloponnese, Antikalamos, 24100 Kalamata, Greece
BioMedInformatics 2023, 3(1), 115-140; https://doi.org/10.3390/biomedinformatics3010009
Submission received: 31 December 2022 / Revised: 24 January 2023 / Accepted: 29 January 2023 / Published: 1 February 2023
(This article belongs to the Special Issue Feature Papers in Medical Statistics and Data Science Section)

Abstract

:
Background: Nowadays, much research deals with the application of the automated meta-analysis of clinical trials through appropriate machine learning tools to extract the results that can then be applied in daily clinical practice. Methods: The author performed a systematic search of the literature from 27 September 2022–22 November 2022 in PUBMED, in the first 6 pages of Google Scholar and in the online catalog, the Systematic Review Toolbox. Moreover, a second search of the literature was performed from 7 January 2023–20 January 2023 in the first 10 pages of Google Scholar and in the Semantic Google Scholar. Results: 38 approaches in 39 articles met the criteria and were included in this overview. These articles describe in detail machine learning approaches, methods, and tools that have been or can potentially be applied to the meta-analysis of clinical trials. Nevertheless, while the other tasks of a systematic review have significantly developed, the automation of meta-analyses is still far from being able to significantly support and facilitate the work of researchers, freeing them from manual, difficult and time-consuming work. Conclusions: The evaluation of automated meta-analysis results is presented in some studies. Their approaches show positive and promising results.

1. Introduction

Today clinical trials are considered as an established experimental clinical tool suitable not only for evaluating the effectiveness of interventions, but also for supporting the conduct of an adequately designed systematic review [1]. In addition, meta-analysis is a systematic review of a focused topic in the literature that provides a quantitative estimate of the effect of a therapeutic intervention or exposure [2]. This effect is inferred from outputs usually from more than one previously published clinical trial. A meta-analysis is necessary for making correct medical decisions (such as prognosis, diagnosis, treatment, recording side effects in taking drugs, etc.). It is the prevailing method applied in clinical trials for generating qualitative and quantitative evidence and conclusions. Meta-analysis and synthesis of the results of clinical trials are gaining rapid momentum in the research to generate quantitative information [3]. Thus, clinical trials are at the forefront of clinical decision support.
In parallel, since in the present time the volume of clinical studies is increasing exponentially, automating their processing by applying machine learning (ML) is a great challenge and a dominant research topic.
The automation in the management of clinical studies refers to dealing with the individual processes related to the search, collection, selection, and extraction of results. In detail these tasks are the following: Design Systematic Search, Run Systematic Search, Deduplicate, Obtain full texts, Snowballing, Screen abstracts, Data extraction and Text Mining Tool, Automated bias assessments, Automated Meta-Analysis, Summarize/Synthesis of data (analysis), Write up, and Data Miner/Analysis of Data for General-Purpose [4].
More specifically, the meta-analysis is a systematic approach for understanding a phenomenon by analyzing the results of many previously published experimental studies. Unfortunately, the conventional meta-analysis requires a great deal of human effort, is labor-intensive, and vulnerable to human bias [2,5].
The task of the automated meta-analysis and synthesis of data is the part of their management in which the least research has been done in terms of their mechanical processing and automation [4].
The authors of many studies [6,7,8,9,10,11] demonstrated the feasibility and acceptance of semi-automated and automated meta-analyses yielding promising results. The outcomes of these studies suggest that automated meta-analysis through machine learning approaches reduces the time required for a meta-analysis without altering the expert confidence in methodological and scientific rigor. Moreover, these results suggest acceptance for risk assessment and improve the quality of reporting.
In this direction, the author of this article deals with the application of automated meta-analysis of clinical trials through appropriate machine learning tools to extract the results that can then be applied in daily clinical practice.
In addition, deep learning methods and tools as a subcategory of ML, are also included in this study. Deep learning based on learning data representations are part of the larger family of machine learning algorithms that use multiple layers to progressively extract higher-level features from the raw input [12].
The novelty of this overview is that until yet very few review articles have been published which describe all these mentioned frameworks, techniques, and tools alongside their applications in a complete and effective way in order to contribute to their further development and improvement.
Thus, initially, the author searched for relevant work and described it in detail below in Section 3.1.
More specifically, the author in this article performed an overview exploring the applied state-of-the-art ML methods, approaches, frameworks, and tools in automating the meta-analysis and synthesis of data extracted from clinical trials.
The main research questions were as follows:
  • RQ1. What are the trends and key characteristics of studies showing automation in the meta-analysis and synthesis of clinical trial data.
  • RQ2. What are the most common technologies, methods, tools, and software used in the meta-analysis and synthesis of data extracted from clinical trials.
  • RQ3. What are the impacts that derive from the usage of the automation in the meta-analysis in clinical trials.
  • RQ4. What are the challenges, guidelines, and obstacles to be addressed and what studies and research are proposed to achieve automation and maximum and reliable application of clinical trial results in daily medical practices.
The rest of this study is organized as follows: Section 2 discusses other relevant studies. Section 3 presents the materials and methods of this study. Section 4 summarizes the results. Section 5 discusses the key issues arising from this study. Section 6 concludes the study and presents future directions.

2. Related Work

There are many studies in the field of the management of studies and clinical trials and the extraction of their knowledge [4,13,14,15,16] but only a limited number deal with the automation of the meta-analysis task.
Wang et al. [12] conducted a review and assessment of 18 common deep learning frameworks and libraries (Caffe, Caffe2, Tensorflow, Theano including Keras Lasagnes and Blocks, MXNet, CNTK, Torch, PyTorch, Pylearn2, Scikit-learn, Matlab including MatconvNet Matlab deep learning and Deep learning toolbox, Chainer, Deeplearning4j) and introduced a large number of benchmarking data.
In order to provide a basis for comparing and selecting between software tools that support Systematic reviews, the authors of [17] performed a feature-by-feature comparison of Systematic reviews tools.
Finally, the Systematic Review Toolbox [9] is an online catalog of tools that support various tasks within the systematic review and wider evidence synthesis process. The updated version of the Systematic Review Toolbox was launched on 13 May 2022, with 235 software tools and 112 guidance documents included.

3. Materials and Methods

3.1. Study Design

In this study design the author used the overview approach [18]. An overview is a generic term used for “any summary of the literature” [19] that attempts to survey the literature and describe its characteristics. As such, it can be used for many different types of literature review, with differing degrees of systematicity. Overviews can provide a broad and often comprehensive summation of a topic area and, as such, have value for those coming to a subject for the first time [20]. They are also important in cases where either a subject is not yet mature and well-known enough to be treated with a thorough systematic review or there is not the necessary time to perform it.
Additionally, the forward and backward snowball method is used [21]. It has been proposed that in reviews of complex or heterogeneous evidence in the field of health services research, “snowball” methods of forward (citation) and backwards (reference) searching are powerful. This method allows researchers using the references and citations of an article to find specific literature on an issue quickly and easily.

3.2. Literature Search and Study Selection

The author performed a systematic search of the literature from 27 September 2022–22 November 2022 in PUBMED (http://www.pubmed.org, accessed on 27 December 2022), in the first 6 pages of Google Scholar and in the online catalog: Systematic Review Toolbox (http://www.systematicreviewtools.com/, accessed on 28 December 2022) using combinations of search strings (“automated meta-analysis” AND “trials”). Moreover, a second search of the literature was performed from 7 January 2023–20 January 2023 in the first 10 pages of Google Scholar and in the Semantic Google Scholar using combinations of search strings (“automated meta-analysis” OR “automatic meta-analysis”).
The author did not find records in clinical trials.gov, or in the COCHRANE library.
In addition, forward and reverse citation searches (snowball method) were performed for specific studies to ensure inclusion of the most relevant studies. Snowballing was undertaken, starting from the included citations and from the references of each article.
Restrictions are related to the language (only English articles are included).
In addition, studies involving tools and techniques for image management (e.g., [22,23,24,25,26,27]) were out of the scope of this study and excluded.

3.3. Data Screening

The data were screened in a two-stage review process (Figure 1) that the author performed, (a) initially excluding assignments based on the titles and their abstracts, and (b) then the remaining assignments were screened based on the reading of the full text of the article.
One researcher reviewed the articles.

3.4. Data Extraction and Analyses

The following data were extracted from the included studies:
  • Bibliographic elements of the included articles:
    Name of the studies’ object
    Reference
    Title
    Year
    Author(s)
    Journal
  • Characteristics of the studies’ object:
    Name studies’ object
    Domain
    Type
    (Not)Free/(Not)Open
    Source Code
    Method/Language

4. Results

There were 38 approaches in 39 articles that met the criteria and were included in this overview (Table 1).
These articles describe in detail ML approaches, methods, and tools that have been or can potentially be applied to the meta-analysis of clinical trials.
All articles in the review range from the years 2010–2023, most of which were identified during the years 2016–2022. (There were 3 articles in 2016, 4 in 2017, 6 in 2018, 4 in 2019, 2 in 2020, 5 in 2021, and 7 in 2022) (Table 1).
The dominant technologies used for the development and application of the automated meta-analysis are Python and R programming languages. Some studies also used Java, Excel, and either C++ or another version of it (i.e., C, ANSI C++, C++11). More rarely were found CUDA, Docker environment, Lua, and LuaJIT. In addition, some studies combined the use of several different technologies to achieve their goals (Table 2).
The research for automation of meta-analyses in some studies is specialized to handle strictly specialized issues such as the biomedical domain (5 articles), evidence-based medicine (6 articles), genomics and molecular genetics (1 article), and medical (5 articles) and metabolomic domain (1 article) (Table 2).
According to the findings of this overview, the types of research most frequently encountered to achieve automated meta-analysis are: the development and implementation of appropriate tools (13 studies), the development and implementation of software (5 studies), and the development of appropriate models and methods (7 studies). More analytically this overview basically identified four types of applications related to supporting or developing an automated meta-analysis. These are the following:
A Framework or Tool: this category includes the development of an integrated framework or the development of a specific tool to support automated meta-analysis. Most of the studies included in this review fall into this category.
A Package or Software: this category includes the development of package software to support automated meta-analysis.
A Model, Method, or Approach: this category includes the development of models and/or methods in the field of automated meta-analysis.
A Web application: This category includes web-based applications that implement automated meta-analyses and are either already implemented or may potentially be implemented in the future in clinical studies as well.
Some of the applications may be included in more than one category. Moreover, some of them can be a complete implementation and include all of the above. More analytically, it is worth noting that 2 studies ([10,52]) fully and comprehensively deal with the topic under discussion here by presenting an integrated modeling and application framework (Table 2).
Below are briefly described the applications found in this overview as classified based on the above four categories.

4.1. Framework/Tool (Includes 16 Studies)

  • Amazon SageMaker [30,31]
Description: Amazon SageMaker Studio is the first integrated development environment in the cloud for machine learning and is designed to integrate the following machine learning workflows: data preparation, feature engineering, statistical bias detection, automated machine learning, training, hosting, ML explainability, monitoring, and machine learning operations in one environment.
Features: The features available in Amazon SageMaker Studio include the following issues: build, train, and deploy machine learning models quickly using Amazon SageMaker; analyze, detect, and receive alerts relating to various business problems using machine learning algorithms and techniques; improve productivity by training and fine-tuning machine learning models in production.
Inputs: datasets; csv files; models.
Outputs: models; Python script; data flow; data.
  • Caffe2 [32]
Caffe2 is Facebook’s in-house production framework for training and deploying large-scale machine learning models. Caffe2 is a deep learning framework that provides an easy and straightforward way for you to experiment with deep learning and leverage community contributions of new models and algorithms.
Features: Caffe2 focuses on several key features required by products: performance, cross-platform support, and coverage for fundamental machine learning algorithms and multi-layer perceptions. The design involves a modular approach, where a unified graph representation is shared among all backend implementations.
Inputs: Python and C++ files; models.
Outputs: everything.
  • CINeMA [33]
Description: the Confidence in Network Meta-Analysis (CINeMA) approach is broadly based on the GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework, with several conceptual and semantic differences [5]. It covers the following domains: (i) within-study bias, (ii) reporting bias, (iii) indirectness, (iv) imprecision, (v) heterogeneity, and (vi) incoherence. The reviewer’s input is required at the study level. Then, CINeMA assigns judgments at three levels (no concerns, some concerns, or major concerns) to each domain. Judgments across domains can be summarized to obtain four levels of confidence (very low, low, moderate, or high) for each relative treatment effect.
Features: the CINeMA framework has been implemented in a freely available, user-friendly web application aiming to facilitate the evaluation of confidence in the results from network meta-analysis. The web application applies the Salanti approach and is programmed in JavaScript, uses Docker, and is linked with R; in particular, packages meta and netmeta are used, and an R package to calculate the contribution of studies in network meta-analysis treatment effects.
Inputs: csv files.
Outputs: outputs a downloadable report with a summary of the evaluations.
  • OpenNN [43]
Description: OpenNN is a software library that implements neural networks, a major area of deep learning research.
Features: OpenNN includes: a multilayer perceptron software implementation; many examples; unit testing.
Inputs: C++ code.
Outputs: data; plots.
  • Pymeta [44]
Description: Pymeta is an online meta-analysis tool, as a web-based application it is created and supported with PythonMeta, a Python package of meta-analysis.
Features: performs: combining effect measures (OR, RR, RD for count data and MD, SMD for continuous data); heterogeneity testing (the Q/Chi-square test); subgroup analysis; cumulative meta-analysis; and sensitivity analysis (one or two factors).
Inputs: Python code.
Outputs: data; plots; bar-lines.
  • PythonMeta [45]
Description: PythonMeta package performs the meta-analysis on an open-access dataset from COCHRANE..
Features: meta-analysis package by and for the Python language. This module was designed to perform some evidence-based medicine (EBM) tasks, such as: combining effect measures (OR, RR, RD, MD, SMD), heterogeneity testing (the Q/Chi-squared test), subgroup analysis, and plots (forest, funnel, etc.).
Inputs: dataset from COCHRANE.
Outputs: data; plots.
  • PyTorch [46]
Description: PyTorch is a deep learning research platform that provides maximum flexibility and speed.
Features: PyTorch is a library that consists of the following components: torch which is a Tensor library with strong GPU support; torch.autograd which is a tape-based automatic differentiation library; torch.jit which is a compilation stack; torch.nn which is a neural networks library deeply integrated with autograd designed for maximum flexibility; torch.multiprocessing which is Python multiprocessing useful for data loading and Hogwild training; torch.utils which is a DataLoader; and other utility functions.
Inputs: Python code.
Outputs: data; plots.
  • scikit-learn [47]
Description: Scikit-learn is a Python library.
Features: it includes functions that are integral to the machine learning pipeline such as data preprocessing steps, data resampling techniques, evaluation parameters, and search interfaces for tuning/optimizing an algorithm’s performance.
Inputs: datasets.
Outputs: data; plots.
  • ShinyMDE [48]
Description: ShinyMDE supports an automated meta-analysis of gene expression data facilitating screening and downloading the results.
The tool handles processed and raw data generated from the most widely used data platforms. In addition, the tool provides users with an option of choosing the method of their choice from the list for meta-analysis.
Features: ShinyMDE consists of a web interface, a standalone version to work remotely, and a database holding GPL files. The general workflow of the ShinyMDE system visualizes the steps of the meta-analysis, which is carried out automatically once a user submits the data and selects the necessary parameters.
Inputs: CSV and txt files.
Outputs: data; web.
  • Spark ML [49]
Description: Spark is a unified analytics engine for large-scale data processing. The package spark.ml aims to provide a uniform set of high-level APIs that help users create and tune practical machine learning pipelines.
Features: Spark provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.
Inputs: Java and Python code.
Outputs: data results; plots.
  • TensorFlow [50]
Description: TensorFlow 2 is an end-to-end, open-source machine learning platform which operates as an infrastructure layer for differentiable programming.
Features: It combines the following key abilities: efficiently executing low-level tensor operations on the CPU, GPU, or TPU; computing the gradient of arbitrary differentiable expressions; scaling computation to many devices; and exporting programs (“graphs”) to external runtimes such as servers, browsers, and mobile and embedded devices.
Inputs: Python and C++ code.
Outputs: data; graphs; and plots.
  • Torch [51]
Description: Torch is not in active development. The functionality provided by the C backend of Torch, which are the TH, THNN, THC, and THCUNN libraries, is actively extended and re-written in the ATen C++11 (which is a new version of C++) library (https://github.com/pytorch/pytorch/tree/master/aten, accessed on 30 December 2022). ATen exposes all operators you would expect from torch7, nn, cutorch, and cunn directly in C++11 and includes additional support for sparse tensors and distributed operations. Thus, Torch is the main package in Torch7 where data structures for multi-dimensional tensors and mathematical operations over these are defined. Moreover, it provides many utilities for accessing files, serializing objects of arbitrary types, and other useful utilities.
Features: Torch includes the following libraries: Tensor Library, File I/O Interface Library, and Useful Utilities. Moreover, Torch7 is a versatile numeric computing framework and machine learning library that extends Lua. Its goal is to provide a flexible environment to design and train learning machines. Flexibility is obtained via Lua, an extremely lightweight scripting language. Torch7 can easily be interfaced to third-party software thanks to Lua’s light interface.
Inputs: code; scripts.
Outputs: data results; plots.
  • NeuroSynth [54]
Description: In this article the authors describe and validate an automated brain mapping framework that uses text mining, meta-analysis, and machine learning techniques to generate a large database of mappings between neural and cognitive states.
Features: in this article the authors describe and validate a framework for the brain mapping, NeuroSynth, that takes an instrumental step towards automated large-scale synthesis of the neuroimaging literature. NeuroSynth combines text mining, meta-analysis, and machine learning techniques (naïve Bayes classification) to generate probabilistic mappings between cognitive and neural states.
Inputs: data.
Outputs: data; plots.
  • Automated meta-analysis of the ERP literature [58]
Description: event-related potentials (ERP) are a common signal of analysis in medicine experiments, with a large existing literature of ERP-related work. This work uses automated literature collection and the text-mining of research articles to summarize the ERP literature, examining patterns and associations within and between components.
Features: all code for this project is written in the Python programming language and uses the LISC [63] Python tool to collect and analyze scientific literature. The data is collected from Pubmed, a database of biomedical literature. From there, the authors use text-mining and word co-occurrence analyses to derive data-driven summaries for each ERP, as well as to compare across these profiles to summarize patterns across the literature.
Inputs: Python code.
Outputs: data; plots.
  • CancerMA [59]
Description: CancerMA is an online, integrated bioinformatic pipeline for automated identification of novel candidate cancer markers/targets. CancerMA operates by means of meta-analyzing expression profiles of user-defined sets of biologically significant and related genes across a manually curated database of 80 publicly available cancer microarray datasets covering 13 cancer types. A simple-to-use web interface allows experts to initiate new analyses as well as to view and retrieve the meta-analysis results.
Features: CancerMA consists of a web interface, a set of pipelined analyses, and two relational databases, one holding the analysis data for each user and another one holding the gene annotation data.
Inputs: R code and data.
Outputs: data; plots.
  • CancerEST [60]
Description: CancerEST was developed as a user-friendly and intuitive tool to compute cancer marker/target potential as well as to obtain comprehensive expression profiles and information about the tissue specificity for genes of interest to biologists/clinicians. The CancerEST web interface for viewing the analysis results consists of three sections: the overview, the information, and the results section. The overview section provides basic information about the submitted job and a brief explanation on how to interpret the results.
Features: CancerEST consists of a web interface, pipelined analyses, and three relational databases; one holding the analysis data, one holding the Unigene data, and another one holding the gene annotation data.
Inputs: R code and data.
Outputs: data; plots.

4.2. Package/Software (Includes 7 Studies)

  • Amamida R Package [29]
Description: the Amanida R package allows a meta-analysis of metabolomics data, combining the results of different studies addressing the same question. The Amanida package contains a collection of functions for computing a meta-analysis in R only using significance and effect size. It covers the lack of data provided on metabolomic studies. Amanida also computes qualitative meta-analysis.
Features: Amanida is a meta-analysis approach using only the most reported statistical parameters in this field: P-value and fold-change. The P-values are combined via Fisher’s method and fold-changes by averaging, both weighted by the study size.
Inputs: supported files are csv, xls/xlsx, and txt.
Outputs: the Amanida package includes several visualization options: a volcano plot for quantitative results, a vote plot for total regulation behaviors for each compound, and an explore plot of the vote-counting results.
  • dmetar [34]
Description: the dmetar package using the meta, metafor, netmeta, and meta-SEM packages as a base is provided as a companion to the R package to support more functions that improve the workflow of a meta-analysis.
Features: dmetar provides tools for various stages of the systematic review process, e.g., visualizing the risk of bias, standard inverse variance meta-analysis, network meta-analysis, three-level meta-analysis, and exploration of the between-study heterogeneity.
Inputs: R code.
Outputs: data results.
  • DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA) [35]
Description: MetaDTA is an online interactive application for conducting the meta-analysis of diagnostic test accuracy studies (DTA), requiring no specialist software for the user to install, but leveraging established analysis routines (specifically the lme4 package in R).
Features: the application allows users to upload their own data, customize SROC plots, obtain statistics such as sensitivity and specificity, and conduct sensitivity analyses. All plots and tables are downloadable.
The tool is interactive and uses an intuitive “point and click” interface and presents results in visually intuitive and appealing ways. It is hoped that this tool will assist those in conducting a DTA meta-analysis who are not statistical experts, and, in turn, increase the relevance of published meta-analyses, and in the long term contribute to improved healthcare decision making as a result.
Inputs: csv files.
Outputs: data results; plots.
  • Keras [36]
Description: Keras is a library that provides highly powerful and abstract building blocks to build deep learning networks. It is a deep learning API written in Python, running on top of the machine learning platform TensorFlow. It was developed with a focus on enabling fast experimentation.
Features: Keras supports both CPU and GPU computation and is a great tool for quickly prototyping ideas. It reduces the developer’s cognitive load to free him up to focus on the parts of the problem that really matter. It also adopts the principle of the progressive disclosure of complexity. Finally, it provides industry-strength performance and scalability.
Inputs: Python code.
Outputs: data results.
  • Meta-Essentials [37]
Description: this is a free excel tool for meta-analysis that facilitates the integration and synthesis of effect sizes from different studies.
Features: Meta-Essentials automatically calculates effect sizes from a wide range of statistics and can be used for a wide range of meta-analysis applications, including subgroup analysis, moderator analysis, and publication bias analysis.
Inputs: xls files.
Outputs: xls files.
  • metafor [38]
Description: The metafor package is a free and open-source add-on for conducting meta-analyses with the statistical software environment R.
Features: the package consists of a collection of functions that allow the user to calculate various effect sizes or outcome measures, fit equal-, fixed-, random-, and mixed-effects models to such data, carry out moderator and meta-regression analyses, and create various types of meta-analytical plots.
Inputs: R code.
Outputs: data results; plots.
  • MetaXL [41]
Description: MetaXL is an add-in for meta-analysis in Microsoft Excel for Windows. It supports all major meta-analysis methods, plus, uniquely, the inverse variance heterogeneity and quality effects models. Starting with v4.0, it also implements a powerful, yet easy-to-use way to perform network meta-analyses. Output is in table and graphical formats.
Features: MetaXL employs almost the same meta-analysis methods that can be accessed in general statistical packages and in dedicated meta-analysis software.
Inputs: Excel files.
Outputs: Excel files.

4.3. Model/Method/Approach (Includes 10 Studies)

  • A Logic of the Meta-Analysis approach [28]
Description: in this position paper, the authors propose, the first as far as is known, an approach for automated reasoning in meta-analyses.
Features: thus, they considered the first steps towards a logic for performing automated meta-analysis based on a finite class of confidence intervals and subset relationships as background knowledge.
Inputs: A machine learning problem.
Outputs: The solution of the problem.
  • Causal Learning Perspective [5]
Description: this work demonstrates the efficacy of using causal models to process the outputs of natural language processing (NLP)-based data extraction and achieve the goal of meta-analysis. In this article the authors initially extract information from scientific publications written in natural language. Sequently, from a novel causal learning perspective, they then propose to frame automated meta-analysis—based on the input of the first step—as a multiple causal inference problem where the summary effect is obtained through intervention.
Features: the authors of this article worked toward automating meta-analysis with a focus on controlling for risks of bias. Thus, they proposed the Multiple Causal inference for automated Meta-Analysis (MCMA). MCMA employs existing NLP systems for the extraction of risks of bias and therapeutic association, which are then used to estimate the summary therapeutic association across several Randomized Clinical Trials (RCTs). More analytically, from this perspective, the authors suggest to frame automated meta-analysis—based on the input of the first step—as a multiple causal inference problem where the summary effect is obtained through intervention. Built upon existent efforts for automating the primary steps of the meta-analysis, the proposed approach achieves the goal of automated meta-analysis and largely reduces the human effort involved.
Inputs: free text and data.
Outputs: data; plots.
  • DIAeT [11]
Description: DIAeT (Dynamic Interactive Argumentation Trees) is a method of synthesizing the evidence available in clinical trials in an ad-hoc and on-demand manner that automatically organizes such evidence into a hierarchical argument that recommends a treatment as superior to another based on a series of key dimensions corresponding to the clinical points of interest.
Features: the DIAeT method is an argumentation-based method that contributes to supporting the synthesis of clinical trial evidence. A limitation of the method is that it relies on a manually populated knowledge base. This problem can be addressed by applying natural language processing methods to extract relevant information from publications. The method has been implemented as a web tool.
Inputs: SPARQL queries.
Outputs: results on the web.
  • metamisc [40]
Description: the metamisc package includes the meta-analysis of diagnostic and prognostic modeling studies. In addition, it summarizes estimations of prognostic factors, diagnostic test accuracy, and prediction model performance. Finally, it validates, updates, and combines published prediction models. It also develops new prediction models with data from multiple studies.
Features: This R package deals with the incomplete availability of study-specific results (performance estimates and their precision), and produces summary estimates of the c-statistic and the observed: the expected ratio and the calibration slope. Furthermore, it tackles the implementation of frequentist and Bayesian meta-analysis methods and proposes novel empirically based prior distributions to improve the estimation of between-study heterogeneity in small samples.
Inputs: R code.
Outputs: data results.
  • Comprehensive gene expression meta-analysis [53]
Description: this approach plans a comprehensive gene expression meta-analysis that labels novel immune signatures in patients with rheumatoid arthritis. This pattern suggests meta-analysis to recognize novel gene signatures that take care of providing mechanistic visions into disease initiation, progression, and the development of better therapeutic attacks.
Features: the aim of the meta-analysis method was firstly to extract the intersected genes, then to exclude genes with inconsistent expression, and finally to test them for significance. The weighted Z-method was used to combine the individual q-values of each gene [64] and was implemented using an R package (https://github.com/bhklab/survcomp, accessed on 28 December 2022), [65].
The meta-analysis algorithm was implemented using R.
Inputs: R code.
Outputs: data.
  • Text-mining the neurosynth corpus (NeuroSynth #2) [55]
Description: in this work the authors demonstrate that an unsupervised study of the NeuroSynth text corpus using Deep Boltzmann Machines (DBMs) can be effectively employed to learn the distribution of the text corpus. The results of this study show some of the clusters obtained when k-means clustering is applied to word embeddings obtained from the DBM model. The clusters display clear semantic context.
Features: a two-layer DBM was employed consisting of a visible layer of multinomial visible units followed by two binary hidden layers. During pre-training and model selection, DBMs were trained. Briefly, annealed importance sampling was employed to estimate the partition function for each DBM. Thus, the proposed DBM model can be used to obtain both word as well as document embeddings in a high-dimensional vector space.
Inputs: data.
Outputs: data; plots.
  • Social brain (NeuroSynth #3) [56]
Description: how the human brain processes social information is an increasingly researched topic in psychology and neuroscience, advancing our understanding of basic human cognition and psychopathologies. In this study, the authors investigated whether these brain regions are evoked by the mere presence of social information using an automated meta-analysis and confirmatory data from an independent study. Results of 1000 published fMRI studies containing the keyword of “social” were subject to an automated meta-analysis. The social/non-social contrast in the independent study showed a strong resemblance to the NeuroSynth map. The Region Of Interest (ROI) analyses revealed that a social effect was credible in most of the NeuroSynth regions in the independent dataset.
Features: the first part of the analyses of this study aimed to identify the brain regions that have shown significant activation in published fMRI studies with a prominent social element in the literature. Using the keyword “social” yielded 1000 published fMRI studies to include in an automated meta-analysis on neurosynth.org. The authors used the reverse inference map of the results of the automated meta-analysis, which represent z-scores corresponding to the likelihood that the term “social” is used in a study given the presence of the reported activation. The significant brain regions showing up in the reverse inference map represent those that are more likely to be reported in “social” studies than in “non-social” studies.
Inputs: data.
Outputs: data; plots.
  • MetaCyto [57]
Description: the authors of this article developed MetaCyto for the automated meta-analysis of flow cytometry and mass spectrometry (CyTOF) data.
Features: by combining clustering methods with a scanning method, MetaCyto can identify commonly labeled subsets of cells, thereby enabling meta-analysis. Thus, the application of MetaCyto to a set of cytometric studies allowed for the identification of cell populations that show differences in abundance between demographic groups.
Inputs: R package.
Outputs: data; plots.
  • Research Method Classification [61]
Description: this research work presents a prototype that applies deep transfer learning to predict the research methods in scientific publications, which facilitates an automatic discovery of crucial research information from large numbers of publications. The current state-of-the-art for classification of research methods uses Support Vector Models (SVMs).
This article provides the following research contributions: (a) developing an artifact that uses deep transfer learning and outperforms the state-of-the-art of research method classification, (b) using full papers and classifying them into predefined research methods, and (c) demonstrating the performance based on an extensive Information Systems corpus.
Features: the proposed approach outperforms state-of-the-art research method classification that deploys the Support Vector Model (SVM). The proposed deep transfer learning models can lead to a better recognition of research methods than shallower word embedding approaches such as word2vec or GloVe. The results illustrate the potential of establishing semi-automated methods for meta-analysis.
Inputs: free text and data.
Outputs: data.
  • AUTOMETA [62]
Description: the proposed system for automating meta-analysis employs existing natural language processing methods for identifying Participants, Intervention, Control, and Outcome (PICO) elements. This system can perform advanced meta-analyses by parsing numeric outcomes to identify the number of patients having certain outcomes. In this study, the authors used the BERT-based approach which is a general-purpose language model trained on a large dataset and uses an attention mechanism that learns contextual relations between words in a text.
Features: the proposed system consists of four major components: crawling PubMed articles, NLP module, creating structured data, and aggregation and visualization. First, a user queries the PubMed database and related articles are returned. Abstracts are then extracted from the articles and passed to the NLP module for preprocessing and extraction of PICO elements. The extracted data are then converted into a structured form. It also parses numeric texts to identify the number of patients having certain outcomes. Identification of the number of patients having certain outcomes is important for statistical analysis to determine the effectiveness of an intervention.
Inputs: free text and data.
Outputs: data.

4.4. Web Application and Integrated Systems (Includes 5 Studies)

  • Automated meta-analysis of biomedical texts [10]
Description: in this research article the authors present the results of the automated analysis of the data extracted from abstracts of scientific articles available in PubMed. These results demonstrate the associations between types of tumors and the most used methods for their cell-based immunotherapy.
Features: the proposed method automates the meta-analysis by standardizing the process in a series of steps. In summary, the following are mentioned: (a) crawling abstracts from Pubmed via the Scrapy based web-crawler, (b) rich linguistic features extraction by using the ISANLP framework which is a Python library to obtain the morphology, syntax parsing, and semantic role labeling features [66], (c) combining tumor and cell dictionaries and morphology-based rules to extract entity candidates from the abstracts, d) using syntactic relations and constructing all their possible combinations and applying models (e.g., UMLS Metathesaurus, MetaMap [67] Fasttext model [68]) to map the terms, (e) using syntactic relations and semantic roles to reveal the links between entities and their roles in the sentence, (f) applying a pre-trained sequence-labeling machine learning model to filter uninformative entity candidates, and g) computing co-occurrence statistics and mining associative rules for the extracted entities [69,70] to obtain stable combinations of tumors, therapy, and cell types. We used the Eclat algorithm [71] because of its scalability.
Inputs: biomedical texts; abstracts of scientific articles available in PubMed; Python code.
Outputs: data; plots.
  • MetaInsight [39]
Description: MetaInsight is a new tool that is freely available and that conducts network meta-analysis (NMA) via the web.
Features: MetaInsight is a web-based tool allowing users with only standard internet browser software to be able to conduct NMAs using an intuitive “point and click” interface and present the results using visual plots.
Inputs: .csv files.
Outputs: data results; plots.
  • Nested-Knowledge [72]
Description: Nested Knowledge offers a comprehensive software platform for systematic literature review and meta-analysis.
Features: the software is composed of two parts which work in tandem. Search, screen, tag, and extract data with AutoLit, and visualize, analyze, publish, and share insights with Synthesis.
Inputs: RIS files.
Outputs: data results; plots; RIS or nBIB files.
  • netmeta [42]
Description: an R package for frequentist meta-analysis, this has a comprehensive set of functions providing a lot of methods for network meta-analysis.
Features: this package supports a comprehensive set of functions providing frequentist methods for network meta-analysis such as: the frequentist network meta-analysis; the net heat plot and design-based decomposition of Cochran’s Q; the measurements of characterizing the flow of evidence between two treatments; the ranking of treatments based on the frequentist analogue of SUCRA; the partial order of treatment rankings and the Hasse diagram; and the contribution matrix, etc.
Inputs: R code.
Outputs: data results.
  • Whyis [52]
Description: Whyis is the first framework for creating custom provenance-driven knowledge graphs. Whyis knowledge graphs are based on nanopublications, which simplify and standardize the production of structured, provenance-supported knowledge in knowledge graphs.
To create probabilistic knowledge graphs, Whyis [52] implements a method of automated meta-analysis. The authors refined the methods used in [73] by using Stouffer’s Z-Method [64].
Features: Whyis is written in Python using the Flask framework. The RDF database used by default is Fuseki. Whyis uses the SPARQL Query, Update, and Graph Store HTTP Protocol. Storage is provided using the FileDepot Python library to provide the file-based persistence of nanopublications and uploaded files. Whyis also relies on Celery which is a task queuing system that can be scaled by adding more task workers on remote machines. Thus, knowledge graph developers create their knowledge graphs by generating a Python module that contains the configuration, templates, and code adapted to their purposes.
Inputs: Python code and script modules.
Outputs: Views; data; plots.

5. Discussion

5.1. Purpose of This Study

The aim of this article is to discover the most modern and complete tools used to automate the conduct of meta-analyses of clinical trials. In this way, it will contribute, on the one hand, to the identification and promotion of the most suitable candidates, and on the other hand, to the development of research in this field.

5.2. Benefits Arising from Automated Meta-Analysis

The evaluation of automated meta-analysis results is presented in some studies [5,9,10,11,29,37]. Their approaches show positive and promising results in the feasibility, acceptance, reliability, and time consumption. More analytically, the most important benefits are the ability to process large data sets in shorter times without altering expert confidence in the methodological and scientific rigor [6].

5.3. Comparison of Systems and Tools Currently Available

Built upon existent efforts for automating the basic steps of meta-analysis, the proposed approaches achieve the goal of automated meta-analysis and largely reduce the human effort involved [5].
However, although important steps have been taken to date, currently there is no application that can fully replace the human effort in conducting a systematic review to draw conclusions from clinical trials. Thus, while the other tasks of a systematic review have significantly developed, the automation of meta-analyses is still far from being able to significantly support and facilitate the work of researchers, freeing them from manual, difficult, and time-consuming work.
At the same time, it is worth noting that most of the tools are either open source or some are freely available (Table 2). Therefore, the strengthening of research in this field should be important in the immediate future.
The benefits of automating meta-analysis are expected to be particularly important in all areas of evidence-based medicine and especially in cutting edge areas of medical research such as gene therapy and cancer treatment.

5.4. Limitations of This Study

In addition, this overview has some methodological limitations. Initially the author had difficulty in identifying suitable articles. This limitation was partially addressed using snowballing methods. Secondly, the author included articles written only in English.

6. Conclusions and Future Directions

ML is the fastest growing field in computer science, and Health Informatics is amongst the greatest application challenges, providing significant benefits in improved medical prognosis, diagnosis, and pharmaceutical development [74].
Meta-analysis is a systematic approach for understanding a wonder by resolving the results of many previously published exploratory studies. It is used mainly to extract knowledge and decisions about the summary effect of situations, interventions, and treatments in medicine. Unfortunately, meta-analysis involves excellent human exertion, rendering a process that is extremely inefficient and vulnerable to human bias. To overcome these issues, many researchers are studying and proposing architectures, methods, and tools to automate meta-analysis [5]. The researchers’ main goal is to provide a system for automating the meta-analysis process as much as possible to reduce the time taken in conducting a meta-analysis [62].
Moreover, the development and application of ML in the meta-analysis of clinical trials is a promising approach to implement more effective daily clinical practices.
However, extensive future studies are needed to validate the performance of ML tools in their application domain.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data from this research are not available elsewhere. Please contact the author for more information, if required.

Conflicts of Interest

The author declares no conflict of interest.

References

  1. Sargeant, J.M.; Kelton, D.F.; O’Connor, A. Study Designs and Systematic Reviews of Interventions: Building Evidence Across Study Designs. Zoonoses Public Health 2014, 61, 10–17. [Google Scholar] [CrossRef]
  2. Russo, M.W. How to Review a Meta-analysis. Gastroenterol. Hepatol. 2007, 3, 637–642. [Google Scholar]
  3. Masoumi, S.; Shahraz, S. Meta-analysis using Python: A hands-on tutorial. BMC Med. Res. Methodol. 2022, 22, 193. [Google Scholar] [CrossRef] [PubMed]
  4. Christopoulou, S.C. Machine Learning Tools and Platforms in Clinical Trial Outputs to Support Evidence-Based Health Informatics: A Rapid Review of the Literature. Biomedinformatics 2022, 2, 511–527. [Google Scholar] [CrossRef]
  5. Cheng, L.; Katz-Rogozhnikov, D.A.; Varshney, K.R.; Baldini, I.; Cheng, L.; Katz-Rogozhnikov, D.A.; Varshney, K.R.; Baldini, I. Automated Meta-Analysis: A Causal Learning Perspective. arXiv 2021, arXiv:2104.04633. [Google Scholar]
  6. Ajiji, P.; Cottin, J.; Picot, C.; Uzunali, A.; Ripoche, E.; Cucherat, M.; Maison, P. Feasibility study and evaluation of expert opinion on the semi-automated meta-analysis and the conventional meta-analysis. Eur. J. Clin. Pharmacol. 2022, 78, 1177–1184. [Google Scholar] [CrossRef]
  7. O’Connor, A.M.; Glasziou, P.; Taylor, M.; Thomas, J.; Spijker, R.; Wolfe, M.S. A focus on cross-purpose tools, automated recognition of study design in multiple disciplines, and evaluation of automation tools: A summary of significant discussions at the fourth meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst. Rev. 2020, 9, 100. [Google Scholar] [CrossRef]
  8. Zintzaras, E.; Lau, J. Trends in meta-analysis of genetic association studies. J. Hum. Genet. 2008, 53, 1–9. [Google Scholar] [CrossRef]
  9. Johnson, E.E.; O’Keefe, H.; Sutton, A.; Marshall, C. The Systematic Review Toolbox: Keeping up to date with tools to support evidence synthesis. Syst. Rev. 2022, 11, 258. [Google Scholar] [CrossRef]
  10. Devyatkin, D.; Molodchenkov, A.; Lukin, A.; Kim, Y.; Boyko, A.; Karalkin, P.; Chiang, J.-H.; Volkova, G.; Lupatov, A. Towards Automated Meta-analysis of Biomedical Texts in the Field of Cell-based Immunotherapy. Biomed. Chem. Res. Methods 2019, 2, e00109. [Google Scholar] [CrossRef]
  11. Sanchez-Graillet, O.; Witte, C.; Grimm, F.; Grautoff, S.; Ell, B.; Cimiano, P. Synthesizing evidence from clinical trials with dynamic interactive argument trees. J. Biomed. Semant. 2022, 13, 16. [Google Scholar] [CrossRef] [PubMed]
  12. Wang, Z.; Liu, K.; Li, J.; Zhu, Y.; Zhang, Y. Various Frameworks and Libraries of Machine Learning and Deep Learning: A Survey. Arch. Comput. Methods Eng. 2019, 1–24. [Google Scholar] [CrossRef]
  13. Scott, A.M.; Forbes, C.; Clark, J.; Carter, M.; Glasziou, P.; Munn, Z. Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: A survey. J. Clin. Epidemiol. 2021, 138, 80–94. [Google Scholar] [CrossRef]
  14. Jaspers, S.; De Troyer, E.; Aerts, M. Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA. EFSA Support. Publ. 2018, 15, 1427E. [Google Scholar] [CrossRef]
  15. Khalil, H.; Ameen, D.; Zarnegar, A. Tools to support the automation of systematic reviews: A scoping review. J. Clin. Epidemiol. 2021, 144, 22–42. [Google Scholar] [CrossRef]
  16. Beller, E.; On behalf of the founding members of the ICASR group; Clark, J.; Tsafnat, G.; Adams, C.; Diehl, H.; Lund, H.; Ouzzani, M.; Thayer, K.; Thomas, J.; et al. Making progress with the automation of systematic reviews: Principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst. Rev. 2018, 7, 77. [Google Scholar] [CrossRef]
  17. Cowie, K.; Rahmatullah, A.; Hardy, N.; Holub, K.; Kallmes, K. Web-Based Software Tools for Systematic Literature Review in Medicine: Systematic Search and Feature Analysis. JMIR Med. Inform. 2021, 10, e33219. [Google Scholar] [CrossRef]
  18. Khangura, S.; Konnyu, K.; Cushman, R.; Grimshaw, J.; Moher, D. Evidence summaries: The evolution of a rapid review approach. Syst. Rev. 2012, 1, 10. [Google Scholar] [CrossRef]
  19. Oxman, A.D. Users’ guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA 1994, 272, 1367–1371. [Google Scholar] [CrossRef]
  20. Grant, M.J.; Booth, A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Inf. Libr. J. 2009, 26, 91–108. [Google Scholar] [CrossRef]
  21. Greenhalgh, T.; Peacock, R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ 2005, 331, 1064–1065. [Google Scholar] [CrossRef] [PubMed]
  22. Shehzad, Z.; Kelly, C.; Reiss, P.T.; Craddock, R.C.; Emerson, J.W.; McMahon, K.; Copland, D.A.; Castellanos, F.X.; Milham, M.P. A multivariate distance-based analytic framework for connectome-wide association studies. Neuroimage 2014, 93, 74–94. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Ewendelken, C. Meta-analysis: How does posterior parietal cortex contribute to reasoning? Front. Hum. Neurosci. 2015, 8, 1042. [Google Scholar] [CrossRef] [PubMed]
  24. Chavez, R.S.; Heatherton, T.F. Representational Similarity of Social and Valence Information in the Medial pFC. J. Cogn. Neurosci. 2015, 27, 73–82. [Google Scholar] [CrossRef]
  25. Chawla, M.; Miyapuram, K.P. Comparison of meta-analysis approaches for neuroimaging studies of reward processing: A case study. In Proceedings of the 2015 International Joint Conference, Killarney, Ireland, 12–17 July 2015; pp. 1–5. [Google Scholar] [CrossRef]
  26. Dockès, J.; A Poldrack, R.; Primet, R.; Gözükan, H.; Yarkoni, T.; Suchanek, F.; Thirion, B.; Varoquaux, G. NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife 2020, 9, e53385. [Google Scholar] [CrossRef]
  27. Muller, A.M.; Meyer, M. Language in the brain at rest: New insights from resting state data and graph theoretical analysis. Front. Hum. Neurosci. 2014, 8, 228. [Google Scholar] [CrossRef]
  28. Peñaloza, R. Towards a Logic of Meta-Analysis. In Proceedings of the 17th International Conference on Principles of Knowledge Representation and Reasoning, Rhodes, Greece, 12–18 September 2020. [Google Scholar]
  29. Llambrich, M.; Correig, E.; Gumà, J.; Brezmes, J.; Cumeras, R. Amanida: An R package for meta-analysis of metabolomics non-integral data. Bioinformatics 2021, 38, 583–585. [Google Scholar] [CrossRef]
  30. Hsieh, M. Getting Started with Amazon SageMaker Studio: Learn to Build End-to-End Machine Learning Projects in the SageMaker Machine Learning IDE.; Packt Publishing Ltd.: Birmingham, UK, 2022. [Google Scholar]
  31. Simon, J. Learn Amazon SageMaker: A Guide to Building, Training, and Deploying Machine Learning Models for Developers and Data Scientists; Packt Publishing Ltd.: Birmingham, UK, 2020. [Google Scholar]
  32. Hazelwood, K.; Bird, S.; Brooks, D.; Chintala, S.; Diril, U.; Dzhulgakov, D.; Fawzy, M.; Jia, B.; Jia, Y.; Kalro, A. Applied machine learning at Facebook: A datacenter infrastructure perspective. In Proceedings of the 2018 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Vienna, Austria, 24–28 February 2018; pp. 620–629. [Google Scholar]
  33. Nikolakopoulou, A.; Higgins, J.P.T.; Papakonstantinou, T.; Chaimani, A.; Del Giovane, C.; Egger, M.; Salanti, G. CINeMA: An approach for assessing confidence in the results of a network meta-analysis. PLoS Med. 2020, 17, e1003082. [Google Scholar] [CrossRef]
  34. Harrer, M.; Cuijpers, P.; Furukawa, T.A.; Ebert, D.D. Doing Meta-Analysis with R: A Hands-On Guide; CRC Press: Boca Raton, FL, USA, 2021. [Google Scholar]
  35. Patel, A.; Cooper, N.; Freeman, S.; Sutton, A. Graphical enhancements to summary receiver operating characteristic plots to facilitate the analysis and reporting of meta-analysis of diagnostic test accuracy data. Res. Synth. Methods 2020, 12, 34–44. [Google Scholar] [CrossRef]
  36. Ketkar, N. Introduction to Keras. In Deep Learning with Python; Apress: Berkeley, CA, USA, 2017; pp. 97–111. [Google Scholar]
  37. Suurmond, R.; van Rhee, H.; Hak, T. Introduction, comparison, and validation of Meta-Essentials: A free and simple tool for meta-analysis. Res. Synth. Methods 2017, 8, 537–553. [Google Scholar] [CrossRef]
  38. Viechtbauer, W. Conducting Meta-Analyses in R with the metafor Package. J. Stat. Softw. 2010, 36, 1–48. [Google Scholar] [CrossRef] [Green Version]
  39. Owen, R.K.; Bradbury, N.; Xin, Y.; Cooper, N.; Sutton, A. MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta. Res. Synth. Methods 2019, 10, 569–581. [Google Scholar] [CrossRef] [PubMed]
  40. Debray, T.P.; Damen, J.A.; Riley, R.D.; Snell, K.; Reitsma, J.B.; Hooft, L.; Collins, G.S.; Moons, K.G. A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Stat. Methods Med. Res. 2018, 28, 2768–2786. [Google Scholar] [CrossRef] [PubMed]
  41. Doi, S.A.; Barendregt, J.J.; Khan, S.; Thalib, L.; Williams, G.M. Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model. Contemp. Clin. Trials 2015, 45, 130–138. [Google Scholar] [CrossRef]
  42. Network Meta-Analysis Using Frequentist Methods [R Package Netmeta Version 0.9-8]. Available online: https://CRAN.R-project.org/package=netmeta (accessed on 29 December 2022).
  43. Open NN: An Open Source Neural Networks C++ Library. Available online: http://opennn.cimne.com (accessed on 30 December 2022).
  44. Hongyong, D. PyMeta. 2018. Available online: www.pymeta.com (accessed on 27 November 2022).
  45. Hongyong, D. PythonMeta 1.26. 2018. Available online: https://pypi.org/project/PythonMeta/ (accessed on 28 December 2022).
  46. The Linux Foundation. PyTorch. Available online: https://pytorch.org/ (accessed on 31 December 2022).
  47. Kramer, O. Machine Learning for Evolution Strategies; Springer: Cham, Switzerland, 2016; p. 20. [Google Scholar] [CrossRef]
  48. Shashirekha, H.L.; Wani, A.H. ShinyMDE: Shiny tool for microarray meta-analysis for differentially expressed gene detection. In Proceedings of the 2016 International Conference on Bioinformatics and Systems Biology (BSB), Allahabad, India, 4–6 March 2016; pp. 1–5. [Google Scholar] [CrossRef]
  49. Polak, A. Scaling Machine Learning with Spark: Distributed ML with MLlib, TensorFlow, and Pytorch; O’Reilly Media: Sebastopol, CA, USA, 2023. [Google Scholar]
  50. Hope, T.; Resheff, Y.; Lieder, I. Learning TensorFlow: A Guide to Building Deep Learning Systems; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2017. [Google Scholar]
  51. Collobert, R.; Kavukcuoglu, K.; Farabet, C. Torch7: A Matlab-like Environment for Machine Learning. Available online: https://ronan.collobert.com/pub/matos/2011_torch7_nipsw.pdf (accessed on 31 December 2022).
  52. McCusker, J.P.; Rashid, S.M.; Agu, N.; Bennett, K.P.; McGuinness, D.L. Developing Scientific Knowledge Graphs Using Whyis. In SemSci@ ISWC; Rensselaer Polytechnic Ins.: Troy, NY, USA, 2018; pp. 52–58. [Google Scholar]
  53. Afroz, S.; Giddaluru, J.; Vishwakarma, S.; Naz, S.; Khan, A.A.; Khan, N. A Comprehensive Gene Expression Meta-analysis Identifies Novel Immune Signatures in Rheumatoid Arthritis Patients. Front. Immunol. 2017, 8, 74. [Google Scholar] [CrossRef]
  54. Yarkoni, T.; Poldrack, R.; Nichols, T.; Van Essen, D.C.; Wager, T.D. Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 2011, 8, 665–670. [Google Scholar] [CrossRef]
  55. Monti, R.; Lorenz, R.; Leech, R.; Anagnostopoulos, C.; Montana, G. Text-mining the neurosynth corpus using deep boltzmann machines. In Proceedings of the 2016 International Workshop on Pattern Recognition in Neuroimaging (PRNI), Trento, Italy, 22–24 June 2016; pp. 1–4. [Google Scholar] [CrossRef]
  56. Tso, I.F.; Rutherford, S.; Fang, Y.; Angstadt, M.; Taylor, S.F. The “social brain” is highly sensitive to the mere presence of social information: An automated meta-analysis and an independent study. PLoS ONE 2018, 13, e0196503. [Google Scholar] [CrossRef]
  57. Hu, Z.; Jujjavarapu, C.; Hughey, J.J.; Andorf, S.; Lee, H.-C.; Gherardini, P.F.; Spitzer, M.H.; Thomas, C.G.; Campbell, J.; Dunn, P.; et al. MetaCyto: A Tool for Automated Meta-analysis of Mass and Flow Cytometry Data. Cell Rep. 2018, 24, 1377–1388. [Google Scholar] [CrossRef] [Green Version]
  58. Donoghue, T.; Voytek, B. Automated meta-analysis of the event-related potential (ERP) literature. Sci. Rep. 2022, 12, 1867. [Google Scholar] [CrossRef]
  59. Feichtinger, J.; McFarlane, R.J.; Larcombe, L.D. CancerMA: A web-based tool for automatic meta-analysis of public cancer microarray data. Database 2012, 2012, bas055. [Google Scholar] [CrossRef]
  60. Feichtinger, J.; McFarlane, R.J.; Larcombe, L.D. CancerEST: A web-based tool for automatic meta-analysis of public EST data. Database 2014, 2014, bau024. [Google Scholar] [CrossRef] [PubMed]
  61. Anisienia, A.; Mueller, R.M.; Kupfer, A.; Staake, T. Research Method Classification with Deep Transfer Learning for Semi-Automatic Meta-Analysis of Information Systems Papers. In Proceedings of the 54th Hawaii International Conference on System Sciences, online, 4–9 January 2021; pp. 6099–6108. [Google Scholar] [CrossRef]
  62. Mutinda, F.W.; Yada, S.; Wakamiya, S.; Aramaki, E. AUTOMETA: Automatic Meta-Analysis System Employing Natural Language Processing. Stud. Health Technol. Inform. 2022, 290, 612–616. [Google Scholar] [CrossRef]
  63. LISC-Literature Scanner-Lisc 0.2.0 Documentation. Available online: https://lisc-tools.github.io/lisc/ (accessed on 23 January 2023).
  64. Whitlock, M.C. Combining probability from independent tests: The weighted Z-method is superior to Fisher’s approach. J. Evol. Biol. 2005, 18, 1368–1373. [Google Scholar] [CrossRef] [PubMed]
  65. Schröder, M.S.; Culhane, A.C.; Quackenbush, J.; Haibe-Kains, B. survcomp: An R/Bioconductor package for performance assessment and comparison of survival models. Bioinformatics 2011, 27, 3206–3208. [Google Scholar] [CrossRef] [PubMed]
  66. Larionov, D.; Moscow, R.F.C.R.; Shelmanov, A.; Chistova, E.; Smirnov, I. Semantic Role Labeling with Pretrained Language Models for Known and Unknown Predicates. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019); INCOMA Ltd.: Varna, Bulgaria, 2019; pp. 619–628. [Google Scholar] [CrossRef]
  67. Aronson, A.R.; Lang, F.-M. An overview of MetaMap: Historical perspective and recent advances. J. Am. Med. Inform. Assoc. 2010, 17, 229–236. [Google Scholar] [CrossRef]
  68. Joulin, A.; Grave, E.; Bojanowski, P.; Mikolov, T. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Paper, Valencia, Spain, 3–7 April 2017. [Google Scholar]
  69. Srikant, R. Fast Algorithms for Mining Association Rules and Sequential Patterns; The University of Wisconsin-Madison: Madison, WI, USA, 1996. [Google Scholar]
  70. Srikant, R.; Agrawal, R. Mining quantitative association rules in large relational tables. In Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data—SIGMOD ’96, Montreal, QC, Canada, 4–6 June 1996. [Google Scholar] [CrossRef]
  71. Zaki, M. Scalable algorithms for association mining. IEEE Trans. Knowl. Data Eng. 2000, 12, 372–390. [Google Scholar] [CrossRef]
  72. Nested Knowledge Features. Available online: https://about.nested-knowledge.com/ (accessed on 31 December 2022).
  73. McCusker, J.P.; Dumontier, M.; Yan, R.; He, S.; Dordick, J.S.; McGuinness, D.L. Finding melanoma drugs through a probabilistic knowledge graph. Peer J. Comput. Sci. 2017, 3, e106. [Google Scholar] [CrossRef]
  74. Holzinger, A. Machine Learning for Health Informatics: State-of-the-Art and Future Challenges; Springer International Publishing: Cham, Switzerland, 2016. [Google Scholar] [CrossRef] [Green Version]
Figure 1. The flow diagram of the literature search.
Figure 1. The flow diagram of the literature search.
Biomedinformatics 03 00009 g001
Table 1. The bibliographic elements of the included studies.
Table 1. The bibliographic elements of the included studies.
NameReferenceTitleYearAuthor(s)Journal
A Logic of Meta-Analysis approach[28]Towards a Logic of Meta-Analysis2020Peñaloza, RProceedings of the International Conference on
Amamida R Package[29]Amanida: An R package for meta-analysis of metabolomics non-integral data2022Llambrich, Maria; Correig, Eudald; Gumà, Josep; Brezmes, Jesús; Cumeras, RaquelBioinformatics
Amazon SageMaker[30,31]Getting Started with Amazon SageMaker Studio: Learn to build end-to-end machine learning projects in the SageMaker machine learning IDE2022Hsieh, MPackt Publishing Ltd.
Automated Meta-analysis of Biomedical Texts[10]Towards Automated Meta-analysis of Biomedical Texts in the Field of Cell-based Immunotherapy 2019Devyatkin DA, Molodchenkov AI, Lukin AV et al.Research and Methods
Caffe2[32]Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective2018Hazelwood, K; et al.IEEE International Symposium on High Performance Computer Architecture
Causal Learning Perspective[5]Automated Meta-Analysis: A Causal Learning Perspective2021Cheng, L; Katz-Rogozhnikov, D A; Varshney, K R; othersarXiv preprint
CINeMA[33]CINeMA: An approach for assessing confidence in the results of a network2020Nikolakopoulou, Adriani; Higgins, Julian P T; Papakonstantinou, Theodoros; Chaimani, Anna; Del Giovane, Cinzia; Egger, Matthias; Salanti, GeorgiaPLOS Medicine
DIAeT[11]Synthesizing evidence from clinical trials with dynamic interactive2022Sanchez-Graillet; Witte, Olivia; Grimm, Christian; Grautoff, Frank; Ell, Steffen; Cimiano, Basil; PhilippJ. Biomed. Semantics
dmetar[34]Doing Meta-Analysis with R: A Hands-On Guide2021Harrer, Mathias; Cuijpers, Pim; Furukawa, Toshi A; Ebert, David DCRC Press
DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA)[35]Graphical enhancements to summary receiver operating characteristic plots to facilitate the analysis and reporting of meta-analysis of diagnostic test accuracy data2021Patel, Amit; Cooper, Nicola; Freeman, Suzanne; Sutton, AlexRes Synth Methods
Keras[36]Introduction to keras. In Deep learning with Python2017Ketkar, NikhilApress, Berkeley, CA
Meta-Essentials[37]Introduction, comparison, and validation of Meta-Essentials: A free and simple tool for meta-analysis2017Suurmond, Robert; van Rhee, Henk; Hak, TonyRes Synth Methods
metafor[38]Conducting Meta-Analyses in R with the metafor Package2010Viechtbauer, WolfgangJournal of Statistical Software
MetaInsight[39]MetaInsight: An interactive web-based tool for analyzing, interrogating, and visualizing network meta-analyses using R-shiny and netmeta2019Owen, Rhiannon K; Bradbury, Naomi; Xin, Yiqiao; Cooper, Nicola; Sutton, AlexRes Synth Methods
metamisc[40]A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes2019Debray, Thomas Pa; Damen, Johanna Aag; Riley, Richard D; Snell, Kym; Reitsma, Johannes B; Hooft, Lotty; Collins, Gary S; Moons, Karel GmStat. Methods Med. Res.
MetaXL[41]Advances in the meta-analysis of heterogeneous clinical trials I: The2015Doi, Suhail A R; Barendregt, Jan J; Khan, Shahjahan; Thalib, Lukman; Williams, Gail MContemp. Clin. Trials
Nested-Knowledge[17]Web-Based Software Tools for Systematic Literature Review in Medicine: A Review and Feature Analysis2021Cowie; Rahmatullah, Kathryn; Hardy, Asad; Holub, Nicole; Kallmes, Karl; KevinNested Knowledge, Inc.
netmeta[42]Network Meta-Analysis using Frequentist Methods [R package netmeta version 0.9-82022Rücker, Gerta; Krahn, Ulrike; König, Jochem; Efthimiou, Orestis; Davies, Annabel; Papakonstantinou, Theodoros; Schwarzer, GuidoCRAN package repository
OpenNN[43]Open NN: An Open Source Neural Networks C++ Library2022Lopez, RobertoInternational Center for Numerical Methods in Engineering (CIMNE)
Pymeta[44]PyMeta2018Hongyong, DengPythonMeta Website
PythonMeta[45]PythonMeta 1.262018Hongyong, DengPythonMeta Website
PyTorch[46]PyTorchNot foundPyTorch–Linux Foundation
scikit-learn[47]scikit-learn2016Python Software FoundationPython Software Foundation
ShinyMDE[48]ShinyMDE: Shiny tool for microarray meta-analysis for differentially expressed gene detection2016Shashirekha, H. L.; Wani, Agaz HussainHLS and team
Spark ML[49]Scaling Machine Learning with Spark: Distributed ML with MLlib, TensorFlow, and Pytorch2023Polak, A.O’Reilly Media
TensorFlow[50]Learning TensorFlow: A Guide to Building Deep Learning Systems2017Hope, Tom; Resheff, Yehezkel S.; Lieder, ItayO’Reilly Media
Torch[51]Torch7: A Matlab-like Environment for Machine Learning2019Collobert, Ronan; Kavukcuoglu, Koray; Farabet, ClementNeural Information Processing Systems
Whyis[52]Developing Scientific Knowledge Graphs Using Whyis2018McCusker, J.P., Rashid, S.M., Agu, N., Bennett, K.P. and McGuinness, D.L.SemSci
Comprehensive gene expression meta-analysis[53]A comprehensive gene expression meta-analysis identifies novel immune signatures in rheumatoid arthritis patients2017Afroz, S.; Giddaluru, J.; Vishwakarma, S.; Naz, S.; Khan, A.A.; Khan, N.Frontiers in
NeuroSynth[54]Large-scale automated synthesis of human functional neuroimaging data2011Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TDNat. Methods
Text-mining the neurosynth corpus (NeuroSynth #2)[55]Text-mining the neurosynth corpus using deep boltzmann machines2016Monti R, Lorenz R, Leech R, Anagnostopoulos C, Montana G2016 International Workshop on Pattern Recognition in Neuroimaging
Social brain (NeuroSynth #3)[56]The “social brain” is highly sensitive to the mere presence of social information: An automated meta-analysis and an independent study2018Tso, Ivy F; Rutherford, Saige; Fang, Yu; Angstadt, Mike; Taylor, Stephan FPLoS One
MetaCyto[57]MetaCyto: A Tool for Automated Meta-analysis of Mass and Flow Cytometry Data2018Hu Z, Jujjavarapu C, Hughey JJ, Andorf S, Lee HC, Gherardini PF et al.Cell Rep.
Automated meta-analysis of the ERP literature[58]Automated meta-analysis of the event-related potential (ERP) literature2022Donoghue T, Voytek BSci. Rep.
CancerMA[59]CancerMA: a web-based tool for automatic meta-analysis of public cancer microarray data2012Feichtinger J, McFarlane RJ, Larcombe LDDatabase
CancerEST[60]CancerEST: a web-based tool for automatic meta-analysis of public EST data2014Feichtinger J, McFarlane RJ, Larcombe LDDatabase
Research Method Classification[61]Research Method Classification with Deep Transfer Learning for Semi-Automatic Meta-Analysis of Information Systems Papers2021Anisienia A, Mueller RM, Kupfer A, Staake TProceedings of the Annual Hawaii International Conference on System Sciences
AUTOMETA[62]AUTOMETA: Automatic Meta-Analysis System Employing Natural Language Processing 2022Mutinda FW, Yada S, Wakamiya S, Aramaki EStud. Health Technol. Inform.
Table 2. The characteristics of the studies’ objects.
Table 2. The characteristics of the studies’ objects.
NameDomainType(Not)Free (Not)OpenSource CodeMethod/Language
A Logic of Meta-Analysis approachGeneral purposeApproachNo needNot supportedNot supported
Amamida R PackageMetabolomic studiesPackageOpen source(https://github.com/mariallr/amanida, accessed on 23 December 2022)R package
Amazon SageMakerGeneral purposeToolNot free(https://aws.amazon.com/sagemaker/resources/, accessed on 23 December 2022)Python
Automated Meta-analysis of Biomedical TextsBiomedicalAllNot describedNo needMetaMap; Fasttext model; Eclat algorithm/Python package
Caffe2General purposeFrameworkOpen Source(https://github.com/pytorch/pytorch, accessed on 22 December 2022)Graph representation is shared among all backend implementation; C++ & Python API
Causal Learning PerspectiveGeneral purposeApproachNo needNo needMultiple Causal inference for automated Meta-Analysis (MCMA)
CINeMAGeneral purposeToolOpen source(https://github.com/esm-ispm-unibe-ch/cinema, accessed on 25 December 2022)Salanti approach; JavaScript, Docker, and R package
DIAeTEvidence-based medicine (EBM)Model/MethodOpen source(https://doi.org/10.5281/zenodo.5604516, accessed on 24 December 2022)model Toulmin; Java
dmetarGeneral purposePackageOpen source(https://github.com/MathiasHarrer/Doing-Meta-Analysis-in-R, accessed on 25 December 2022; https://dmetar.protectlab.org/, accessed on 25 December 2022)R package
DTA MA (Diagnostic Test Accuracy Meta-Analysis) (MetaDTA)General purposeSoftwareOpen source(https://github.com/CRSU-Apps/MetaDTA; https://crsu.shinyapps.io/dta_ma/, accessed on 25 December 2022)R package
KerasGeneral purposeSoftwareOpen source(https://keras.io/; https://github.com/keras-team/keras, accessed on 25 December 2022)Python
Meta-EssentialsGeneral purposeSoftwareOpen source(https://www.erim.eur.nl/research-support/meta-essentials/download/, accessed on 26 December 2022; https://www.meta-essentials.com, accessed on 26 December 2022)Excel files
metaforGeneral purposeSoftwareOpen source(https://www.jstatsoft.org/article/view/v036i03, accessed on 27 December 2022)R package
MetaInsightGeneral purposeWeb applicationNot Open; Freely available(https://crsu.shinyapps.io/metainsight, accessed on 22 December 2022)Not described
metamiscGeneral purposeModel/MethodOpen source(https://cran.r-project.org/web/packages/metamisc/index.html, accessed on 25 December 2022; https://github.com/smartdata-analysis-and-statistics/metamisc, accessed on 28 December 2022)R package
MetaXLEvidence-based medicine (EBM)SoftwareFreely available(http://www.epigear.com/index_files/metaxl.html, accessed on 27 December 2022)Excel files
Nested-KnowledgeEvidence-based medicine (EBM)Web applicationNot free(https://nested-knowledge.com/nest/qualitative/371, accessed on 26 December 2022)Not described
netmetaGeneral purposeWeb applicationOpen source(https://cran.r-project.org/web/packages/netmeta/index.html, accessed on 23 December 2022; https://github.com/guido-s/netmeta, accessed on 28 December 2022; https://link.springer.com/book/10.1007/978-3-319-21416-0, accessed on 26 December 2022; https://rdrr.io/cran/netmeta/src/R/netmeta.R, accessed on 26 December 2022)R package
OpenNNGeneral purposeToolOpen source(https://github.com/Artelnics/OpenNN, accessed on 27 December 2022; http://opennn.cimne.com/download.asp, accessed on 28 December 2022)ANSI C++
PymetaEvidence-based medicine (EBM)ToolNot Open(https://www.pymeta.com/, accessed on 28 December 2022Python
PythonMetaEvidence-based medicine (EBM)ToolOpen sourcehttps://pypi.org/project/PythonMeta/, accessed on 28 December 2022)Python
PyTorchEvidence-based medicine (EBM)ToolOpen source(https://github.com/pytorch/pytorch, accessed on 28 December 2022)Python
scikit-learnGeneral purposeToolOpen source(scikit-learn/scikit-learn: scikit-learn: machine learning in Python (github.com), accessed on 21 December 2022)Python
ShinyMDEgenomics, molecular geneticsToolNot Open; Freely available(https://hussain.shinyapps.io/App-1, accessed on 21 December 2022)R package
Spark MLGeneral purposeToolOpen source(https://github.com/apache/spark, accessed on 22 December 2022)Java; Python; R
TensorFlowGeneral purposeToolOpen source(https://github.com/tensorflow/tensorflow, accessed on 22 December 2022)C++; Python
TorchGeneral purposeFrameworkOpen source(https://github.com/torch/torch7, accessed on 22 December 2022)C++11; Lua; LuaJIT, C; CUDA and C++
WhyisGeneral purposeAllOpen(https://whyis.readthedocs.io/en/latest/index.html, accessed on 22 December 2022; https://github.com/tetherless-world/whyis, accessed on 22 December 2022)probabilistic knowledge graphs by using Stouffer’s Z-Method/ Python; Flask framework; Fuseki; SPARQL; Graph Store HTTP Protocol; FileDepot Python library
Comprehensive gene expression meta-analysis BiomedicalMethodOpenNo needWeighted Z-method/ survcomp R package
NeuroSynthMedicalFrameworkOpen(https://github.com/neurosynth, accessed on 26 December 2022)naïve Bayes classification
Text-mining the neurosynth corpus (NeuroSynth #2)MedicalMethodNo needNo needunsupervised study/ Deep Boltzmann machines for text-mining
Social brain (NeuroSynth #3)MedicalMethodNo need(http://neurosynth.org/analyses/terms/social/, accessed on 28 December 2022)Regions Of Interest (ROIs) analysis
MetaCytoBiomedicalMethodNo need(http://bioconductor.org/packages/release/bioc/html/MetaCyto.html, accessed on 28 December 2022)clustering methods with a scanning method/R package
Automated meta-analysis of the ERP literatureMedicalToolOpen(https://erpscanr.github.io/, accessed on 28 December 2022; https://github.com/ERPscanr/ERPscanr, accessed on 28 December 2022)text-mining and word co-occurrence analyses
CancerMABiomedicalToolOpen(http://www.cancerma.org.uk, accessed on 28 December 2022) (not found)HTML/CSS; Twitter Bootstrapp; Javascript/jQuery; Perl; R package; Bioconductor package
CancerESTBiomedicalToolOpen(http://www.cancerest.org.uk/help.html http://www.cancerest.org.uk, accessed on 28 December 2022) (not found)HTML/CSS; Twitter Bootstrapp; Javascript/jQuery; Perl; R package; Bioconductor package
Research Method ClassificationGeneral purposeMethodNo needNo needSupport Vector Models
AUTOMETAMedicalApproachNo needNo needBERT-based model
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Christopoulou, S.C. Towards Automated Meta-Analysis of Clinical Trials: An Overview. BioMedInformatics 2023, 3, 115-140. https://doi.org/10.3390/biomedinformatics3010009

AMA Style

Christopoulou SC. Towards Automated Meta-Analysis of Clinical Trials: An Overview. BioMedInformatics. 2023; 3(1):115-140. https://doi.org/10.3390/biomedinformatics3010009

Chicago/Turabian Style

Christopoulou, Stella C. 2023. "Towards Automated Meta-Analysis of Clinical Trials: An Overview" BioMedInformatics 3, no. 1: 115-140. https://doi.org/10.3390/biomedinformatics3010009

Article Metrics

Back to TopTop