Submit to Entropy Review for Entropy Propose a Special Issue

Journal Menu

Journal Browser

Probabilistic Models in Machine and Human Learning

Special Issue Editors
Special Issue Information
Keywords
Published Papers

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Information Theory, Probability and Statistics".

Deadline for manuscript submissions: closed (14 November 2023) | Viewed by 9465

Share This Special Issue

Special Issue Editor

Prof. Dr. Patrick Shafto

E-Mail Website
Guest Editor

1.Mathematics and Computer Science, Rutgers University, Newark, NJ 07102, USA
2.School of Mathematics, Institute for Advanced Studies, Princeton, NJ 08540, USA
Interests: machine learning; human learning; applied math

Special Issue Information

Dear Colleagues,

Probabilistic models naturally capture the problem of learning from evidence in the presence of uncertainty. As such, probabilistic models feature prominently in machine learning, models of human learning, and AI broadly. Indeed, even neural networks, classically non-probabilistic models, are defined in terms of approximations to probabilistic inference and are adapted to be probabilistic. The depth and breadth of influence of probabilistic models are hard to overstate.

Current applications of probabilistic modeling are evolving rapidly. In human learning, models are developed for ever more nuanced applications from communication, to emotion, to intuitive physics. In machine learning, the successes of deep learning have strongly influenced research by encouraging work on deep Gaussian processes and modifications of neural networks to capture uncertainty. Meanwhile, more classic probabilistic accounts of causality, structure, and nonparametrics continue apace, and connections develop with reinforcement learning. Impressive recent directions span the two fields by considering how to integrate more structured knowledge into less structured approaches, and vice versa.

The goal of this Special Issue is to provide a forum demonstrating the breadth of probabilistic approaches for machine and human learning, and advances at their intersection. In particular, we aim to provide an outlet for rigorous treatments of both fields, in contrast to existing venues which speak from one disciplinary perspective or the other.

Prof. Dr. Patrick Shafto
Guest Editor

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

human learning
machine learning
Gaussian process
learning and development
approximate inference
probabilistic programs
causal reasoning

Published Papers (7 papers)

Download All Papers

Order results

Result details

Show export options Show export options

Select all

Export citation of selected articles as:

Research

Jump to: Review

16 pages, 3876 KiB

Open AccessArticle

Three-Dimensional Reconstruction Pre-Training as a Prior to Improve Robustness to Adversarial Attacks and Spurious Correlation

by Yutaro Yamada, Fred Weiying Zhang, Yuval Kluger and Ilker Yildirim

Entropy 2024, 26(3), 258; https://doi.org/10.3390/e26030258 - 14 Mar 2024

Viewed by 699

Abstract

Ensuring robustness of image classifiers against adversarial attacks and spurious correlation has been challenging. One of the most effective methods for adversarial robustness is a type of data augmentation that uses adversarial examples during training. Here, inspired by computational models of human vision, we explore a synthesis of this approach by leveraging a structured prior over image formation: the 3D geometry of objects and how it projects to images. We combine adversarial training with a weight initialization that implicitly encodes such a prior about 3D objects via 3D reconstruction pre-training. We evaluate our approach using two different datasets and compare it to alternative pre-training protocols that do not encode a prior about 3D shape. To systematically explore the effect of 3D pre-training, we introduce a novel dataset called Geon3D, which consists of simple shapes that nevertheless capture variation in multiple distinct dimensions of geometry. We find that while 3D reconstruction pre-training does not improve robustness for the simplest dataset setting, we consider (Geon3D on a clean background) that it improves upon adversarial training in more realistic (Geon3D with textured background and ShapeNet) conditions. We also find that 3D pre-training coupled with adversarial training improves the robustness to spurious correlations between shape and background textures. Furthermore, we show that the benefit of using 3D-based pre-training outperforms 2D-based pre-training on ShapeNet. We hope that these results encourage further investigation of the benefits of structured, 3D-based models of vision for adversarial robustness. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

18 pages, 626 KiB

Open AccessArticle

A Gibbs Posterior Framework for Fair Clustering

by Abhisek Chakraborty, Anirban Bhattacharya and Debdeep Pati

Entropy 2024, 26(1), 63; https://doi.org/10.3390/e26010063 - 11 Jan 2024

Cited by 1 | Viewed by 848

Abstract

The rise of machine learning-driven decision-making has sparked a growing emphasis on algorithmic fairness. Within the realm of clustering, the notion of balance is utilized as a criterion for attaining fairness, which characterizes a clustering mechanism as fair when the resulting clusters maintain a consistent proportion of observations representing individuals from distinct groups delineated by protected attributes. Building on this idea, the literature has rapidly incorporated a myriad of extensions, devising fair versions of the existing frequentist clustering algorithms, e.g., k-means, k-medioids, etc., that aim at minimizing specific loss functions. These approaches lack uncertainty quantification associated with the optimal clustering configuration and only provide clustering boundaries without quantifying the probabilities associated with each observation belonging to the different clusters. In this article, we intend to offer a novel probabilistic formulation of the fair clustering problem that facilitates valid uncertainty quantification even under mild model misspecifications, without incurring substantial computational overhead. Mixture model-based fair clustering frameworks facilitate automatic uncertainty quantification, but tend to showcase brittleness under model misspecification and involve significant computational challenges. To circumnavigate such issues, we propose a generalized Bayesian fair clustering framework that inherently enjoys decision-theoretic interpretation. Moreover, we devise efficient computational algorithms that crucially leverage techniques from the existing literature on optimal transport and clustering based on loss functions. The gain from the proposed technology is showcased via numerical experiments and real data examples. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

23 pages, 1121 KiB

Open AccessArticle

Automating Model Comparison in Factor Graphs

by Bart van Erp, Wouter W. L. Nuijten, Thijs van de Laar and Bert de Vries

Entropy 2023, 25(8), 1138; https://doi.org/10.3390/e25081138 - 29 Jul 2023

Viewed by 874

Abstract

Bayesian state and parameter estimation are automated effectively in a variety of probabilistic programming languages. The process of model comparison on the other hand, which still requires error-prone and time-consuming manual derivations, is often overlooked despite its importance. This paper efficiently automates Bayesian model averaging, selection, and combination by message passing on a Forney-style factor graph with a custom mixture node. Parameter and state inference, and model comparison can then be executed simultaneously using message passing with scale factors. This approach shortens the model design cycle and allows for the straightforward extension to hierarchical and temporal model priors to accommodate for modeling complicated time-varying processes. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

30 pages, 10160 KiB

Open AccessArticle

Efficient Discretization of Optimal Transport

by Junqi Wang, Pei Wang and Patrick Shafto

Entropy 2023, 25(6), 839; https://doi.org/10.3390/e25060839 - 24 May 2023

Viewed by 882

Abstract

Obtaining solutions to optimal transportation (OT) problems is typically intractable when marginal spaces are continuous. Recent research has focused on approximating continuous solutions with discretization methods based on i.i.d. sampling, and this has shown convergence as the sample size increases. However, obtaining OT solutions with large sample sizes requires intensive computation effort, which can be prohibitive in practice. In this paper, we propose an algorithm for calculating discretizations with a given number of weighted points for marginal distributions by minimizing the (entropy-regularized) Wasserstein distance and providing bounds on the performance. The results suggest that our plans are comparable to those obtained with much larger numbers of i.i.d. samples and are more efficient than existing alternatives. Moreover, we propose a local, parallelizable version of such discretizations for applications, which we demonstrate by approximating adorable images. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

24 pages, 1167 KiB

Open AccessArticle

Seeing the Error in My “Bayes”: A Quantified Degree of Belief Change Correlates with Children’s Pupillary Surprise Responses Following Explicit Predictions

by Joseph Colantonio, Igor Bascandziev, Maria Theobald, Garvin Brod and Elizabeth Bonawitz

Entropy 2023, 25(2), 211; https://doi.org/10.3390/e25020211 - 21 Jan 2023

Cited by 1 | Viewed by 1535

Abstract

Bayesian models allow us to investigate children’s belief revision alongside physiological states, such as “surprise”. Recent work finds that pupil dilation (or the “pupillary surprise response”) following expectancy violations is predictive of belief revision. How can probabilistic models inform the interpretations of “surprise”? Shannon Information considers the likelihood of an observed event, given prior beliefs, and suggests stronger surprise occurs following unlikely events. In contrast, Kullback–Leibler divergence considers the dissimilarity between prior beliefs and updated beliefs following observations—with greater surprise indicating more change between belief states to accommodate information. To assess these accounts under different learning contexts, we use Bayesian models that compare these computational measures of “surprise” to contexts where children are asked to either predict or evaluate the same evidence during a water displacement task. We find correlations between the computed Kullback–Leibler divergence and the children’s pupillometric responses only when the children actively make predictions, and no correlation between Shannon Information and pupillometry. This suggests that when children attend to their beliefs and make predictions, pupillary responses may signal the degree of divergence between a child’s current beliefs and the updated, more accommodating beliefs. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

20 pages, 844 KiB

Open AccessArticle

Stochastic Control for Bayesian Neural Network Training

by Ludwig Winkler, César Ojeda and Manfred Opper

Entropy 2022, 24(8), 1097; https://doi.org/10.3390/e24081097 - 09 Aug 2022

Cited by 3 | Viewed by 1447

Abstract

In this paper, we propose to leverage the Bayesian uncertainty information encoded in parameter distributions to inform the learning procedure for Bayesian models. We derive a first principle stochastic differential equation for the training dynamics of the mean and uncertainty parameter in the variational distributions. On the basis of the derived Bayesian stochastic differential equation, we apply the methodology of stochastic optimal control on the variational parameters to obtain individually controlled learning rates. We show that the resulting optimizer, StochControlSGD, is significantly more robust to large learning rates and can adaptively and individually control the learning rates of the variational parameters. The evolution of the control suggests separate and distinct dynamical behaviours in the training regimes for the mean and uncertainty parameters in Bayesian neural networks. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

► Show Figures

Figure 1

Review

Jump to: Research

23 pages, 361 KiB

Open AccessReview

Probabilistic Learning and Psychological Similarity

by Nina Poth

Entropy 2023, 25(10), 1407; https://doi.org/10.3390/e25101407 - 30 Sep 2023

Viewed by 1255

Abstract

The notions of psychological similarity and probabilistic learning are key posits in cognitive, computational, and developmental psychology and in machine learning. However, their explanatory relationship is rarely made explicit within and across these research fields. This opinionated review critically evaluates how these notions can mutually inform each other within computational cognitive science. Using probabilistic models of concept learning as a case study, I argue that two notions of psychological similarity offer important normative constraints to guide modelers’ interpretations of representational primitives. In particular, the two notions furnish probabilistic models of cognition with meaningful interpretations of what the associated subjective probabilities in the model represent and how they attach to experiences from which the agent learns. Similarity representations thereby provide probabilistic models with cognitive, as opposed to purely mathematical, content. Full article

(This article belongs to the Special Issue Probabilistic Models in Machine and Human Learning)

Show export options Show export options

Select all

Export citation of selected articles as:

Displaying articles 1-7

Journal Menu

Journal Browser

Probabilistic Models in Machine and Human Learning

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Published Papers (7 papers)

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI