entropy-logo

Journal Browser

Journal Browser

Information Theory for Data Science

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".

Deadline for manuscript submissions: 15 December 2024 | Viewed by 8801

Special Issue Editors


E-Mail Website
Guest Editor
1. Department of Industrial Engineering, The Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Ramat-Aviv 69978, Israel
2. Laboratory of AI Business and Data Analytics (LAMBDA), Tel Aviv University, Ramat-Aviv 69978, Israel
Interests: analytics; machine learning; probability; artificial intelligence
Special Issues, Collections and Topics in MDPI journals

E-Mail Website
Guest Editor
Department of Industrial Engineering, The Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Tel Aviv-Yafo 69978, Israel
Interests: statistical learning; predictive modeling; inference problems; information theory and learning; data analysis and applications
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Data science is a field of study that focuses on the extraction of valuable information from noisy data, and incorporates various disciplines, such as data engineering, data visualization, predictive analytics, data mining, machine learning and statistics. In recent years, there has been a rapidly growing interest in the mathematical and theoretical aspects of data science. This manifests in probabilistic and statistical models striving to provide performance guarantee, robustness, fairness, explainability and to generate reusable and interpretable results. 

For this Special Issue, we invite contributions that focus on information theoretic methods for data science domains. We welcome unpublished original work on both the theory and the practice of the abovementioned areas.

Prof. Dr. Irad E. Ben-Gal
Dr. Amichai Painsky
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • data science
  • data mining
  • machine learning
  • explainable AI
  • statistics and inference
  • predictive modeling
  • big data
  • data analytics

Published Papers (5 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

20 pages, 2184 KiB  
Article
A Comparative Analysis of Discrete Entropy Estimators for Large-Alphabet Problems
by Assaf Pinchas, Irad Ben-Gal and Amichai Painsky
Entropy 2024, 26(5), 369; https://doi.org/10.3390/e26050369 - 28 Apr 2024
Viewed by 482
Abstract
This paper presents a comparative study of entropy estimation in a large-alphabet regime. A variety of entropy estimators have been proposed over the years, where each estimator is designed for a different setup with its own strengths and caveats. As a consequence, no [...] Read more.
This paper presents a comparative study of entropy estimation in a large-alphabet regime. A variety of entropy estimators have been proposed over the years, where each estimator is designed for a different setup with its own strengths and caveats. As a consequence, no estimator is known to be universally better than the others. This work addresses this gap by comparing twenty-one entropy estimators in the studied regime, starting with the simplest plug-in estimator and leading up to the most recent neural network-based and polynomial approximate estimators. Our findings show that the estimators’ performance highly depends on the underlying distribution. Specifically, we distinguish between three types of distributions, ranging from uniform to degenerate distributions. For each class of distribution, we recommend the most suitable estimator. Further, we propose a sample-dependent approach, which again considers three classes of distribution, and report the top-performing estimators in each class. This approach provides a data-dependent framework for choosing the desired estimator in practical setups. Full article
(This article belongs to the Special Issue Information Theory for Data Science)
Show Figures

Figure 1

18 pages, 2746 KiB  
Article
Hybrid DAER Based Cross-Modal Retrieval Exploiting Deep Representation Learning
by Zhao Huang, Haowu Hu and Miao Su
Entropy 2023, 25(8), 1216; https://doi.org/10.3390/e25081216 - 16 Aug 2023
Viewed by 1050
Abstract
Information retrieval across multiple modes has attracted much attention from academics and practitioners. One key challenge of cross-modal retrieval is to eliminate the heterogeneous gap between different patterns. Most of the existing methods tend to jointly construct a common subspace. However, very little [...] Read more.
Information retrieval across multiple modes has attracted much attention from academics and practitioners. One key challenge of cross-modal retrieval is to eliminate the heterogeneous gap between different patterns. Most of the existing methods tend to jointly construct a common subspace. However, very little attention has been given to the study of the importance of different fine-grained regions of various modalities. This lack of consideration significantly influences the utilization of the extracted information of multiple modalities. Therefore, this study proposes a novel text-image cross-modal retrieval approach that constructs a dual attention network and an enhanced relation network (DAER). More specifically, the dual attention network tends to precisely extract fine-grained weight information from text and images, while the enhanced relation network is used to expand the differences between different categories of data in order to improve the computational accuracy of similarity. The comprehensive experimental results on three widely-used major datasets (i.e., Wikipedia, Pascal Sentence, and XMediaNet) show that our proposed approach is effective and superior to existing cross-modal retrieval methods. Full article
(This article belongs to the Special Issue Information Theory for Data Science)
Show Figures

Figure 1

23 pages, 2054 KiB  
Article
On Sequential Bayesian Inference for Continual Learning
by Samuel Kessler, Adam Cobb, Tim G. J. Rudner, Stefan Zohren and Stephen J. Roberts
Entropy 2023, 25(6), 884; https://doi.org/10.3390/e25060884 - 31 May 2023
Cited by 2 | Viewed by 2776
Abstract
Sequential Bayesian inference can be used for continual learning to prevent catastrophic forgetting of past tasks and provide an informative prior when learning new tasks. We revisit sequential Bayesian inference and assess whether using the previous task’s posterior as a prior for a [...] Read more.
Sequential Bayesian inference can be used for continual learning to prevent catastrophic forgetting of past tasks and provide an informative prior when learning new tasks. We revisit sequential Bayesian inference and assess whether using the previous task’s posterior as a prior for a new task can prevent catastrophic forgetting in Bayesian neural networks. Our first contribution is to perform sequential Bayesian inference using Hamiltonian Monte Carlo. We propagate the posterior as a prior for new tasks by approximating the posterior via fitting a density estimator on Hamiltonian Monte Carlo samples. We find that this approach fails to prevent catastrophic forgetting, demonstrating the difficulty in performing sequential Bayesian inference in neural networks. From there, we study simple analytical examples of sequential Bayesian inference and CL and highlight the issue of model misspecification, which can lead to sub-optimal continual learning performance despite exact inference. Furthermore, we discuss how task data imbalances can cause forgetting. From these limitations, we argue that we need probabilistic models of the continual learning generative process rather than relying on sequential Bayesian inference over Bayesian neural network weights. Our final contribution is to propose a simple baseline called Prototypical Bayesian Continual Learning, which is competitive with the best performing Bayesian continual learning methods on class incremental continual learning computer vision benchmarks. Full article
(This article belongs to the Special Issue Information Theory for Data Science)
Show Figures

Figure 1

15 pages, 969 KiB  
Article
Semi-Supervised k-Star (SSS): A Machine Learning Method with a Novel Holo-Training Approach
by Kokten Ulas Birant
Entropy 2023, 25(1), 149; https://doi.org/10.3390/e25010149 - 11 Jan 2023
Cited by 4 | Viewed by 1689
Abstract
As one of the entropy-based methods, the k-Star algorithm benefits from information theory in computing the distances between data instances during the classification task. k-Star is a machine learning method with a high classification performance and strong generalization ability. Nevertheless, as a standard [...] Read more.
As one of the entropy-based methods, the k-Star algorithm benefits from information theory in computing the distances between data instances during the classification task. k-Star is a machine learning method with a high classification performance and strong generalization ability. Nevertheless, as a standard supervised learning method, it performs learning only from labeled data. This paper proposes an improved method, called Semi-Supervised k-Star (SSS), which makes efficient predictions by considering unlabeled data in addition to labeled data. Moreover, it introduces a novel semi-supervised learning approach, called holo-training, against self-training. It has the advantage of enabling a powerful and robust model of data by combining multiple classifiers and using an entropy measure. The results of extensive experimental studies showed that the proposed holo-training approach outperformed the self-training approach on 13 out of the 18 datasets. Furthermore, the proposed SSS method achieved higher accuracy (95.25%) than the state-of-the-art semi-supervised methods (90.01%) on average. The significance of the experimental results was validated by using both the Binomial Sign test and the Friedman test. Full article
(This article belongs to the Special Issue Information Theory for Data Science)
Show Figures

Figure 1

14 pages, 570 KiB  
Article
A Bayesian Mixture Cure Rate Model for Estimating Short-Term and Long-Term Recidivism
by Rolando de la Cruz, Claudio Fuentes and Oslando Padilla
Entropy 2023, 25(1), 56; https://doi.org/10.3390/e25010056 - 28 Dec 2022
Cited by 2 | Viewed by 1704
Abstract
Mixture cure rate models have been developed to analyze failure time data where a proportion never fails. For such data, standard survival models are usually not appropriate because they do not account for the possibility of non-failure. In this context, mixture cure rate [...] Read more.
Mixture cure rate models have been developed to analyze failure time data where a proportion never fails. For such data, standard survival models are usually not appropriate because they do not account for the possibility of non-failure. In this context, mixture cure rate models assume that the studied population is a mixture of susceptible subjects who may experience the event of interest and non-susceptible subjects that will never experience it. More specifically, mixture cure rate models are a class of survival time models in which the probability of an eventual failure is less than one and both the probability of eventual failure and the timing of failure depend (separately) on certain individual characteristics. In this paper, we propose a Bayesian approach to estimate parametric mixture cure rate models with covariates. The probability of eventual failure is estimated using a binary regression model, and the timing of failure is determined using a Weibull distribution. Inference for these models is attained using Markov Chain Monte Carlo methods under the proposed Bayesian framework. Finally, we illustrate the method using data on the return-to-prison time for a sample of prison releases of men convicted of sexual crimes against women in England and Wales and we use mixture cure rate models to investigate the risk factors for long-term and short-term survival of recidivism. Full article
(This article belongs to the Special Issue Information Theory for Data Science)
Show Figures

Figure 1

Back to TopTop