entropy-logo

Journal Browser

Journal Browser

MaxEnt 2019—The 39th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering

A special issue of Entropy (ISSN 1099-4300). This special issue belongs to the section "Multidisciplinary Applications".

Deadline for manuscript submissions: closed (30 September 2019) | Viewed by 162459

Special Issue Editors

Max-Planck-Institute for Plasma Physics, Boltzmannstrasse 2, 85748 Garching, Germany
Interests: Bayesian probability theory; surface physics; ion-solid interactions; experimental design and data fusion; data analysis; MCMC-methods; robotics; Gaussian processes
Max-Planck-Institute for Plasma Physics, Boltzmannstrasse 2, 85748 Garching, Germany
Interests: MCMC methods; Bayesian inference; inverse problems and uncertainty quantification (UQ)

Special Issue Information

Dear Colleagues,

The main topics of this Special Issue are the application of Bayesian inference and the maximum entropy principle to inverse problems in science, machine learning, information theory, and engineering.
Inverse and uncertainty quantification (UQ) problems arise from a large variety of applications, such as earth science, astrophysics, material and plasma science, imaging in geophysics and medicine, nondestructive testing, density estimation, remote sensing, Gaussian process (GP) regression, optimal experimental design, data assimilation, and data mining.
This Special Issue thus invites contributions on all aspects of probabilistic inference, including novel techniques and applications, and work that sheds new light on the foundations of inference.

Dr. Udo Von Toussaint
Dr. Roland Preuss
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Entropy is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Published Papers (13 papers)

Order results
Result details
Select all
Export citation of selected articles as:

Research

11 pages, 279 KiB  
Article
Entropic Dynamics in Neural Networks, the Renormalization Group and the Hamilton-Jacobi-Bellman Equation
by Nestor Caticha
Entropy 2020, 22(5), 587; https://doi.org/10.3390/e22050587 - 23 May 2020
Cited by 4 | Viewed by 3187
Abstract
We study the dynamics of information processing in the continuum depth limit of deep feed-forward Neural Networks (NN) and find that it can be described in language similar to the Renormalization Group (RG). The association of concepts to patterns by a NN is [...] Read more.
We study the dynamics of information processing in the continuum depth limit of deep feed-forward Neural Networks (NN) and find that it can be described in language similar to the Renormalization Group (RG). The association of concepts to patterns by a NN is analogous to the identification of the few variables that characterize the thermodynamic state obtained by the RG from microstates. To see this, we encode the information about the weights of a NN in a Maxent family of distributions. The location hyper-parameters represent the weights estimates. Bayesian learning of a new example determine new constraints on the generators of the family, yielding a new probability distribution which can be seen as an entropic dynamics of learning, yielding a learning dynamics where the hyper-parameters change along the gradient of the evidence. For a feed-forward architecture the evidence can be written recursively from the evidence up to the previous layer convoluted with an aggregation kernel. The continuum limit leads to a diffusion-like PDE analogous to Wilson’s RG but with an aggregation kernel that depends on the weights of the NN, different from those that integrate out ultraviolet degrees of freedom. This can be recast in the language of dynamical programming with an associated Hamilton–Jacobi–Bellman equation for the evidence, where the control is the set of weights of the neural network. Full article
16 pages, 2378 KiB  
Article
Statistical Approaches for the Analysis of Dependency Among Neurons Under Noise
by Deniz Gençağa, Sevgi Şengül Ayan, Hajar Farnoudkia and Serdar Okuyucu
Entropy 2020, 22(4), 387; https://doi.org/10.3390/e22040387 - 28 Mar 2020
Cited by 2 | Viewed by 2345
Abstract
Neuronal noise is a major factor affecting the communication between coupled neurons. In this work, we propose a statistical toolset to infer the coupling between two neurons under noise. We estimate these statistical dependencies from data which are generated by a coupled Hodgkin–Huxley [...] Read more.
Neuronal noise is a major factor affecting the communication between coupled neurons. In this work, we propose a statistical toolset to infer the coupling between two neurons under noise. We estimate these statistical dependencies from data which are generated by a coupled Hodgkin–Huxley (HH) model with additive noise. To infer the coupling using observation data, we employ copulas and information-theoretic quantities, such as the mutual information (MI) and the transfer entropy (TE). Copulas and MI between two variables are symmetric quantities, whereas TE is asymmetric. We demonstrate the performances of copulas and MI as functions of different noise levels and show that they are effective in the identification of the interactions due to coupling and noise. Moreover, we analyze the inference of TE values between neurons as a function of noise and conclude that TE is an effective tool for finding out the direction of coupling between neurons under the effects of noise. Full article
Show Figures

Figure 1

21 pages, 706 KiB  
Article
Detrending the Waveforms of Steady-State Vowels
by Marnix Van Soom and Bart de Boer
Entropy 2020, 22(3), 331; https://doi.org/10.3390/e22030331 - 13 Mar 2020
Cited by 2 | Viewed by 3529
Abstract
Steady-state vowels are vowels that are uttered with a momentarily fixed vocal tract configuration and with steady vibration of the vocal folds. In this steady-state, the vowel waveform appears as a quasi-periodic string of elementary units called pitch periods. Humans perceive this quasi-periodic [...] Read more.
Steady-state vowels are vowels that are uttered with a momentarily fixed vocal tract configuration and with steady vibration of the vocal folds. In this steady-state, the vowel waveform appears as a quasi-periodic string of elementary units called pitch periods. Humans perceive this quasi-periodic regularity as a definite pitch. Likewise, so-called pitch-synchronous methods exploit this regularity by using the duration of the pitch periods as a natural time scale for their analysis. In this work, we present a simple pitch-synchronous method using a Bayesian approach for estimating formants that slightly generalizes the basic approach of modeling the pitch periods as a superposition of decaying sinusoids, one for each vowel formant, by explicitly taking into account the additional low-frequency content in the waveform which arises not from formants but rather from the glottal pulse. We model this low-frequency content in the time domain as a polynomial trend function that is added to the decaying sinusoids. The problem then reduces to a rather familiar one in macroeconomics: estimate the cycles (our decaying sinusoids) independently from the trend (our polynomial trend function); in other words, detrend the waveform of steady-state waveforms. We show how to do this efficiently. Full article
Show Figures

Figure 1

17 pages, 2864 KiB  
Article
Mean Shift Cluster Recognition Method Implementation in the Nested Sampling Algorithm
by Martino Trassinelli and Pierre Ciccodicola
Entropy 2020, 22(2), 185; https://doi.org/10.3390/e22020185 - 06 Feb 2020
Cited by 7 | Viewed by 2546
Abstract
Nested sampling is an efficient algorithm for the calculation of the Bayesian evidence and posterior parameter probability distributions. It is based on the step-by-step exploration of the parameter space by Monte Carlo sampling with a series of values sets called live points that [...] Read more.
Nested sampling is an efficient algorithm for the calculation of the Bayesian evidence and posterior parameter probability distributions. It is based on the step-by-step exploration of the parameter space by Monte Carlo sampling with a series of values sets called live points that evolve towards the region of interest, i.e., where the likelihood function is maximal. In presence of several local likelihood maxima, the algorithm converges with difficulty. Some systematic errors can also be introduced by unexplored parameter volume regions. In order to avoid this, different methods are proposed in the literature for an efficient search of new live points, even in presence of local maxima. Here we present a new solution based on the mean shift cluster recognition method implemented in a random walk search algorithm. The clustering recognition is integrated within the Bayesian analysis program NestedFit. It is tested with the analysis of some difficult cases. Compared to the analysis results without cluster recognition, the computation time is considerably reduced. At the same time, the entire parameter space is efficiently explored, which translates into a smaller uncertainty of the extracted value of the Bayesian evidence. Full article
Show Figures

Figure 1

16 pages, 2344 KiB  
Article
Gaussian Process Regression for Data Fulfilling Linear Differential Equations with Localized Sources
by Christopher G. Albert and Katharina Rath
Entropy 2020, 22(2), 152; https://doi.org/10.3390/e22020152 - 27 Jan 2020
Cited by 11 | Viewed by 3837
Abstract
Specialized Gaussian process regression is presented for data that are known to fulfill a given linear differential equation with vanishing or localized sources. The method allows estimation of system parameters as well as strength and location of point sources. It is applicable to [...] Read more.
Specialized Gaussian process regression is presented for data that are known to fulfill a given linear differential equation with vanishing or localized sources. The method allows estimation of system parameters as well as strength and location of point sources. It is applicable to a wide range of data from measurement and simulation. The underlying principle is the well-known invariance of the Gaussian probability distribution under linear operators, in particular differentiation. In contrast to approaches with a generic covariance function/kernel, we restrict the Gaussian process to generate only solutions of the homogeneous part of the differential equation. This requires specialized kernels with a direct correspondence of certain kernel hyperparameters to parameters in the underlying equation and leads to more reliable regression results with less training data. Inhomogeneous contributions from linear superposition of point sources are treated via a linear model over fundamental solutions. Maximum likelihood estimates for hyperparameters and source positions are obtained by nonlinear optimization. For differential equations representing laws of physics the present approach generates only physically possible solutions, and estimated hyperparameters represent physical properties. After a general derivation, modeling of source-free data and parameter estimation is demonstrated for Laplace’s equation and the heat/diffusion equation. Finally, the Helmholtz equation with point sources is treated, representing scalar wave data such as acoustic pressure in the frequency domain. Full article
Show Figures

Figure 1

14 pages, 796 KiB  
Article
Bayesian Uncertainty Quantification with Multi-Fidelity Data and Gaussian Processes for Impedance Cardiography of Aortic Dissection
by Sascha Ranftl, Gian Marco Melito, Vahid Badeli, Alice Reinbacher-Köstinger, Katrin Ellermann and Wolfgang von der Linden
Entropy 2020, 22(1), 58; https://doi.org/10.3390/e22010058 - 31 Dec 2019
Cited by 10 | Viewed by 3493
Abstract
In 2000, Kennedy and O’Hagan proposed a model for uncertainty quantification that combines data of several levels of sophistication, fidelity, quality, or accuracy, e.g., a coarse and a fine mesh in finite-element simulations. They assumed each level to be describable by a Gaussian [...] Read more.
In 2000, Kennedy and O’Hagan proposed a model for uncertainty quantification that combines data of several levels of sophistication, fidelity, quality, or accuracy, e.g., a coarse and a fine mesh in finite-element simulations. They assumed each level to be describable by a Gaussian process, and used low-fidelity simulations to improve inference on costly high-fidelity simulations. Departing from there, we move away from the common non-Bayesian practice of optimization and marginalize the parameters instead. Thus, we avoid the awkward logical dilemma of having to choose parameters and of neglecting that choice’s uncertainty. We propagate the parameter uncertainties by averaging the predictions and the prediction uncertainties over all the possible parameters. This is done analytically for all but the nonlinear or inseparable kernel function parameters. What is left is a low-dimensional and feasible numerical integral depending on the choice of kernels, thus allowing for a fully Bayesian treatment. By quantifying the uncertainties of the parameters themselves too, we show that “learning” or optimising those parameters has little meaning when data is little and, thus, justify all our mathematical efforts. The recent hype about machine learning has long spilled over to computational engineering but fails to acknowledge that machine learning is a big data problem and that, in computational engineering, we usually face a little data problem. We devise the fully Bayesian uncertainty quantification method in a notation following the tradition of E.T. Jaynes and find that generalization to an arbitrary number of levels of fidelity and parallelisation becomes rather easy. We scrutinize the method with mock data and demonstrate its advantages in its natural application where high-fidelity data is little but low-fidelity data is not. We then apply the method to quantify the uncertainties in finite element simulations of impedance cardiography of aortic dissection. Aortic dissection is a cardiovascular disease that frequently requires immediate surgical treatment and, thus, a fast diagnosis before. While traditional medical imaging techniques such as computed tomography, magnetic resonance tomography, or echocardiography certainly do the job, Impedance cardiography too is a clinical standard tool and promises to allow earlier diagnoses as well as to detect patients that otherwise go under the radar for too long. Full article
Show Figures

Figure 1

24 pages, 3803 KiB  
Article
Deep Learning and Artificial Intelligence for the Determination of the Cervical Vertebra Maturation Degree from Lateral Radiography
by Masrour Makaremi, Camille Lacaule and Ali Mohammad-Djafari
Entropy 2019, 21(12), 1222; https://doi.org/10.3390/e21121222 - 14 Dec 2019
Cited by 40 | Viewed by 6854
Abstract
Deep Learning (DL) and Artificial Intelligence (AI) tools have shown great success in different areas of medical diagnostics. In this paper, we show another success in orthodontics. In orthodontics, the right treatment timing of many actions and operations is crucial because many environmental [...] Read more.
Deep Learning (DL) and Artificial Intelligence (AI) tools have shown great success in different areas of medical diagnostics. In this paper, we show another success in orthodontics. In orthodontics, the right treatment timing of many actions and operations is crucial because many environmental and genetic conditions may modify jaw growth. The stage of growth is related to the Cervical Vertebra Maturation (CVM) degree. Thus, determining the CVM to determine the suitable timing of the treatment is important. In orthodontics, lateral X-ray radiography is used to determine it. Many classical methods need knowledge and time to look and identify some features. Nowadays, ML and AI tools are used for many medical and biological diagnostic imaging. This paper reports on the development of a Deep Learning (DL) Convolutional Neural Network (CNN) method to determine (directly from images) the degree of maturation of CVM classified in six degrees. The results show the performances of the proposed method in different contexts with different number of images for training, evaluation and testing and different pre-processing of these images. The proposed model and method are validated by cross validation. The implemented software is almost ready for use by orthodontists. Full article
Show Figures

Figure 1

8 pages, 642 KiB  
Article
Bayesian Determination of Parameters for Plasma-Wall Interactions
by Roland Preuss, Rodrigo Arredondo and Udo von Toussaint
Entropy 2019, 21(12), 1175; https://doi.org/10.3390/e21121175 - 29 Nov 2019
Cited by 4 | Viewed by 1809
Abstract
Within the Bayesian framework a non-intrusive uncertainty quantification is performed to assess the uncertainty of ion–solid interaction simulations. For this we employ a reduced-order spectral expansion approach which is capable of confining the number of model runs to a feasible size. Moreover, the [...] Read more.
Within the Bayesian framework a non-intrusive uncertainty quantification is performed to assess the uncertainty of ion–solid interaction simulations. For this we employ a reduced-order spectral expansion approach which is capable of confining the number of model runs to a feasible size. Moreover, the method facilitates sensitivity examinations regarding to input parameters of the model. It is applied to the ion–solid simulation program SDTrimSP with several uncertain but normally distributed input parameters, i.e., impact angle α , projectile energy E 0 , and surface binding energy E s b . Consequently, the otherwise hardly accessible model parameter E s b can be estimated in combination with recently acquired experimental data. Full article
Show Figures

Figure 1

30 pages, 1174 KiB  
Article
TI-Stan: Model Comparison Using Thermodynamic Integration and HMC
by R. Wesley Henderson and Paul M. Goggans
Entropy 2019, 21(12), 1161; https://doi.org/10.3390/e21121161 - 27 Nov 2019
Cited by 1 | Viewed by 2723
Abstract
We present a novel implementation of the adaptively annealed thermodynamic integration technique using Hamiltonian Monte Carlo (HMC). Thermodynamic integration with importance sampling and adaptive annealing is an especially useful method for estimating model evidence for problems that use physics-based mathematical models. Because it [...] Read more.
We present a novel implementation of the adaptively annealed thermodynamic integration technique using Hamiltonian Monte Carlo (HMC). Thermodynamic integration with importance sampling and adaptive annealing is an especially useful method for estimating model evidence for problems that use physics-based mathematical models. Because it is based on importance sampling, this method requires an efficient way to refresh the ensemble of samples. Existing successful implementations use binary slice sampling on the Hilbert curve to accomplish this task. This implementation works well if the model has few parameters or if it can be broken into separate parts with identical parameter priors that can be refreshed separately. However, for models that are not separable and have many parameters, a different method for refreshing the samples is needed. HMC, in the form of the MC-Stan package, is effective for jointly refreshing the ensemble under a high-dimensional model. MC-Stan uses automatic differentiation to compute the gradients of the likelihood that HMC requires in about the same amount of time as it computes the likelihood function itself, easing the programming burden compared to implementations of HMC that require explicitly specified gradient functions. We present a description of the overall TI-Stan procedure and results for representative example problems. Full article
Show Figures

Figure 1

18 pages, 4321 KiB  
Article
Stochastic Gradient Annealed Importance Sampling for Efficient Online Marginal Likelihood Estimation
by Scott A. Cameron, Hans C. Eggers and Steve Kroon
Entropy 2019, 21(11), 1109; https://doi.org/10.3390/e21111109 - 12 Nov 2019
Cited by 1 | Viewed by 2590
Abstract
We consider estimating the marginal likelihood in settings with independent and identically distributed (i.i.d.) data. We propose estimating the predictive distributions in a sequential factorization of the marginal likelihood in such settings by using stochastic gradient Markov Chain Monte Carlo techniques. This approach [...] Read more.
We consider estimating the marginal likelihood in settings with independent and identically distributed (i.i.d.) data. We propose estimating the predictive distributions in a sequential factorization of the marginal likelihood in such settings by using stochastic gradient Markov Chain Monte Carlo techniques. This approach is far more efficient than traditional marginal likelihood estimation techniques such as nested sampling and annealed importance sampling due to its use of mini-batches to approximate the likelihood. Stability of the estimates is provided by an adaptive annealing schedule. The resulting stochastic gradient annealed importance sampling (SGAIS) technique, which is the key contribution of our paper, enables us to estimate the marginal likelihood of a number of models considerably faster than traditional approaches, with no noticeable loss of accuracy. An important benefit of our approach is that the marginal likelihood is calculated in an online fashion as data becomes available, allowing the estimates to be used for applications such as online weighted model combination. Full article
Show Figures

Figure 1

16 pages, 1776 KiB  
Article
Bayesian Maximum-A-Posteriori Approach with Global and Local Regularization to Image Reconstruction Problem in Medical Emission Tomography
by Natalya Denisova
Entropy 2019, 21(11), 1108; https://doi.org/10.3390/e21111108 - 12 Nov 2019
Cited by 5 | Viewed by 2965
Abstract
The Bayesian approach Maximum a Posteriori (MAP) provides a common basis for developing statistical methods for solving ill-posed image reconstruction problems. MAP solutions are dependent on a priori model. Approaches developed in literature are based on prior models that describe the properties of [...] Read more.
The Bayesian approach Maximum a Posteriori (MAP) provides a common basis for developing statistical methods for solving ill-posed image reconstruction problems. MAP solutions are dependent on a priori model. Approaches developed in literature are based on prior models that describe the properties of the expected image rather than the properties of the studied object. In this paper, such models have been analyzed and it is shown that they lead to global regularization of the solution. Prior models that are based on the properties of the object under study are developed and conditions for local and global regularization are obtained. A new reconstruction algorithm has been developed based on the method of local statistical regularization. Algorithms with global and local regularization were compared in numerical simulations. The simulations were performed close to the real oncologic single photon emission computer tomography (SPECT) study. It is shown that the approach with local regularization produces more accurate images of ‘hot spots’, which is especially important to tumor diagnostics in nuclear oncology. Full article
Show Figures

Figure 1

19 pages, 1831 KiB  
Article
Estimating Flight Characteristics of Anomalous Unidentified Aerial Vehicles
by Kevin H. Knuth, Robert M. Powell and Peter A. Reali
Entropy 2019, 21(10), 939; https://doi.org/10.3390/e21100939 - 25 Sep 2019
Cited by 8 | Viewed by 121220
Abstract
Several Unidentified Aerial Phenomena (UAP) encountered by military, commercial, and civilian aircraft have been reported to be structured craft that exhibit ‘impossible’ flight characteristics. We consider a handful of well-documented encounters, including the 2004 encounters with the Nimitz Carrier Group off the coast [...] Read more.
Several Unidentified Aerial Phenomena (UAP) encountered by military, commercial, and civilian aircraft have been reported to be structured craft that exhibit ‘impossible’ flight characteristics. We consider a handful of well-documented encounters, including the 2004 encounters with the Nimitz Carrier Group off the coast of California, and estimate lower bounds on the accelerations exhibited by the craft during the observed maneuvers. Estimated accelerations range from almost 100 g to 1000s of gs with no observed air disturbance, no sonic booms, and no evidence of excessive heat commensurate with even the minimal estimated energies. In accordance with observations, the estimated parameters describing the behavior of these craft are both anomalous and surprising. The extreme estimated flight characteristics reveal that these observations are either fabricated or seriously in error, or that these craft exhibit technology far more advanced than any known craft on Earth. In many cases, the number and quality of witnesses, the variety of roles they played in the encounters, and the equipment used to track and record the craft favor the latter hypothesis that these are indeed technologically advanced craft. The observed flight characteristics of these craft are consistent with the flight characteristics required for interstellar travel, i.e., if these observed accelerations were sustainable in space, then these craft could easily reach relativistic speeds within a matter of minutes to hours and cover interstellar distances in a matter of days to weeks, proper time. Full article
Show Figures

Graphical abstract

17 pages, 911 KiB  
Article
Pragmatic Hypotheses in the Evolution of Science
by Luis Gustavo Esteves, Rafael Izbicki, Julio Michael Stern and Rafael Bassi Stern
Entropy 2019, 21(9), 883; https://doi.org/10.3390/e21090883 - 11 Sep 2019
Cited by 7 | Viewed by 3917
Abstract
This paper introduces pragmatic hypotheses and relates this concept to the spiral of scientific evolution. Previous works determined a characterization of logically consistent statistical hypothesis tests and showed that the modal operators obtained from this test can be represented in the hexagon of [...] Read more.
This paper introduces pragmatic hypotheses and relates this concept to the spiral of scientific evolution. Previous works determined a characterization of logically consistent statistical hypothesis tests and showed that the modal operators obtained from this test can be represented in the hexagon of oppositions. However, despite the importance of precise hypothesis in science, they cannot be accepted by logically consistent tests. Here, we show that this dilemma can be overcome by the use of pragmatic versions of precise hypotheses. These pragmatic versions allow a level of imprecision in the hypothesis that is small relative to other experimental conditions. The introduction of pragmatic hypotheses allows the evolution of scientific theories based on statistical hypothesis testing to be interpreted using the narratological structure of hexagonal spirals, as defined by Pierre Gallais. Full article
Show Figures

Figure 1

Back to TopTop