Research

22 pages, 2348 KiB

Open AccessArticle

Using the Semantic Information G Measure to Explain and Extend Rate-Distortion Functions and Maximum Entropy Distributions

by Chenguang Lu

Entropy 2021, 23(8), 1050; https://doi.org/10.3390/e23081050 - 15 Aug 2021

Cited by 2 | Viewed by 2668

Abstract

In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual Information (MMI) distributions and ME distributions are expressed by Bayes-like formulas, including Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other [...] Read more.

In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual Information (MMI) distributions and ME distributions are expressed by Bayes-like formulas, including Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other hand, the rate-distortion function has three disadvantages: (1) the distortion function is subjectively defined; (2) the definition of the distortion function between instances and labels is often difficult; (3) it cannot be used for data compression according to the labels’ semantic meanings. The author has proposed using the semantic information G measure with both statistical probability and logical probability before. We can now explain NEFs as truth functions, partition functions as logical probabilities, Bayes-like formulas as semantic Bayes’ formulas, MMI as Semantic Mutual Information (SMI), and ME as extreme ME minus SMI. In overcoming the above disadvantages, this paper sets up the relationship between truth functions and distortion functions, obtains truth functions from samples by machine learning, and constructs constraint conditions with truth functions to extend rate-distortion functions. Two examples are used to help readers understand the MMI iteration and to support the theoretical results. Using truth functions and the semantic information G measure, we can combine machine learning and data compression, including semantic compression. We need further studies to explore general data compression and recovery, according to the semantic meaning. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

13 pages, 309 KiB

Open AccessArticle

Upper Bounds for the Capacity for Severely Fading MIMO Channels under a Scale Mixture Assumption

by Johannes T. Ferreira

Entropy 2021, 23(7), 845; https://doi.org/10.3390/e23070845 - 30 Jun 2021

Cited by 1 | Viewed by 1810

Abstract

A cornerstone in the modeling of wireless communication is MIMO systems, where a complex matrix variate normal assumption is often made for the underlying distribution of the propagation matrix. A popular measure of information, namely capacity, is often investigated for the performance of [...] Read more.

A cornerstone in the modeling of wireless communication is MIMO systems, where a complex matrix variate normal assumption is often made for the underlying distribution of the propagation matrix. A popular measure of information, namely capacity, is often investigated for the performance of MIMO designs. This paper derives upper bounds for this measure of information for the case of two transmitting antennae and an arbitrary number of receiving antennae when the propagation matrix is assumed to follow a scale mixture of complex matrix variate normal distribution. Furthermore, noncentrality is assumed to account for LOS scenarios within the MIMO environment. The insight of this paper illustrates the theoretical form of capacity under these key assumptions and paves the way for considerations of alternative distributional choices for the channel propagation matrix in potential cases of severe fading, when the assumption of normality may not be realistic. Full article

(This article belongs to the Special Issue Measures of Information)

16 pages, 670 KiB

Open AccessArticle

Designing Bivariate Auto-Regressive Timeseries with Controlled Granger Causality

by Shohei Hidaka and Takuma Torii

Entropy 2021, 23(6), 742; https://doi.org/10.3390/e23060742 - 12 Jun 2021

Cited by 1 | Viewed by 1823

Abstract

In this manuscript, we analyze a bivariate vector auto-regressive (VAR) model in order to draw the design principle of a timeseries with a controlled statistical inter-relationship. We show how to generate bivariate timeseries with given covariance and Granger causality (or, equivalently, transfer entropy), [...] Read more.

In this manuscript, we analyze a bivariate vector auto-regressive (VAR) model in order to draw the design principle of a timeseries with a controlled statistical inter-relationship. We show how to generate bivariate timeseries with given covariance and Granger causality (or, equivalently, transfer entropy), and show the trade-off relationship between these two types of statistical interaction. In principle, covariance and Granger causality are independently controllable, but the feasible ranges of their values which allow the VAR to be proper and have a stationary distribution are constrained by each other. Thus, our analysis identifies the essential tri-lemma structure among the stability and properness of VAR, the controllability of covariance, and that of Granger causality. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

14 pages, 690 KiB

Open AccessEditor’s ChoiceArticle

On Representations of Divergence Measures and Related Quantities in Exponential Families

by Stefan Bedbur and Udo Kamps

Entropy 2021, 23(6), 726; https://doi.org/10.3390/e23060726 - 08 Jun 2021

Cited by 1 | Viewed by 2077

Abstract

Within exponential families, which may consist of multi-parameter and multivariate distributions, a variety of divergence measures, such as the Kullback–Leibler divergence, the Cressie–Read divergence, the Rényi divergence, and the Hellinger metric, can be explicitly expressed in terms of the respective cumulant function and [...] Read more.

Within exponential families, which may consist of multi-parameter and multivariate distributions, a variety of divergence measures, such as the Kullback–Leibler divergence, the Cressie–Read divergence, the Rényi divergence, and the Hellinger metric, can be explicitly expressed in terms of the respective cumulant function and mean value function. Moreover, the same applies to related entropy and affinity measures. We compile representations scattered in the literature and present a unified approach to the derivation in exponential families. As a statistical application, we highlight their use in the construction of confidence regions in a multi-sample setup. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

15 pages, 346 KiB

Open AccessArticle

Information Geometry of the Exponential Family of Distributions with Progressive Type-II Censoring

by Fode Zhang, Xiaolin Shi and Hon Keung Tony Ng

Entropy 2021, 23(6), 687; https://doi.org/10.3390/e23060687 - 28 May 2021

Cited by 1 | Viewed by 2031

Abstract

In geometry and topology, a family of probability distributions can be analyzed as the points on a manifold, known as statistical manifold, with intrinsic coordinates corresponding to the parameters of the distribution. Consider the exponential family of distributions with progressive Type-II censoring as [...] Read more.

In geometry and topology, a family of probability distributions can be analyzed as the points on a manifold, known as statistical manifold, with intrinsic coordinates corresponding to the parameters of the distribution. Consider the exponential family of distributions with progressive Type-II censoring as the manifold of a statistical model, we use the information geometry methods to investigate the geometric quantities such as the tangent space, the Fisher metric tensors, the affine connection and the

α

-connection of the manifold. As an application of the geometric quantities, the asymptotic expansions of the posterior density function and the posterior Bayesian predictive density function of the manifold are discussed. The results show that the asymptotic expansions are related to the coefficients of the

α

-connections and metric tensors, and the predictive density function is the estimated density function in an asymptotic sense. The main results are illustrated by considering the Rayleigh distribution. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

19 pages, 366 KiB

Open AccessArticle

Stochastic Order and Generalized Weighted Mean Invariance

by Mateu Sbert, Jordi Poch, Shuning Chen and Víctor Elvira

Entropy 2021, 23(6), 662; https://doi.org/10.3390/e23060662 - 25 May 2021

Viewed by 1704

Abstract

In this paper, we present order invariance theoretical results for weighted quasi-arithmetic means of a monotonic series of numbers. The quasi-arithmetic mean, or Kolmogorov–Nagumo mean, generalizes the classical mean and appears in many disciplines, from information theory to physics, from economics to traffic [...] Read more.

In this paper, we present order invariance theoretical results for weighted quasi-arithmetic means of a monotonic series of numbers. The quasi-arithmetic mean, or Kolmogorov–Nagumo mean, generalizes the classical mean and appears in many disciplines, from information theory to physics, from economics to traffic flow. Stochastic orders are defined on weights (or equivalently, discrete probability distributions). They were introduced to study risk in economics and decision theory, and recently have found utility in Monte Carlo techniques and in image processing. We show in this paper that, if two distributions of weights are ordered under first stochastic order, then for any monotonic series of numbers their weighted quasi-arithmetic means share the same order. This means for instance that arithmetic and harmonic mean for two different distributions of weights always have to be aligned if the weights are stochastically ordered, this is, either both means increase or both decrease. We explore the invariance properties when convex (concave) functions define both the quasi-arithmetic mean and the series of numbers, we show its relationship with increasing concave order and increasing convex order, and we observe the important role played by a new defined mirror property of stochastic orders. We also give some applications to entropy and cross-entropy and present an example of multiple importance sampling Monte Carlo technique that illustrates the usefulness and transversality of our approach. Invariance theorems are useful when a system is represented by a set of quasi-arithmetic means and we want to change the distribution of weights so that all means evolve in the same direction. Full article

(This article belongs to the Special Issue Measures of Information)

13 pages, 274 KiB

Open AccessArticle

Fractional Deng Entropy and Extropy and Some Applications

by Mohammad Reza Kazemi, Saeid Tahmasebi, Francesco Buono and Maria Longobardi

Entropy 2021, 23(5), 623; https://doi.org/10.3390/e23050623 - 17 May 2021

Cited by 35 | Viewed by 2665

Abstract

Deng entropy and extropy are two measures useful in the Dempster–Shafer evidence theory (DST) to study uncertainty, following the idea that extropy is the dual concept of entropy. In this paper, we present their fractional versions named fractional Deng entropy and extropy and [...] Read more.

Deng entropy and extropy are two measures useful in the Dempster–Shafer evidence theory (DST) to study uncertainty, following the idea that extropy is the dual concept of entropy. In this paper, we present their fractional versions named fractional Deng entropy and extropy and compare them to other measures in the framework of DST. Here, we study the maximum for both of them and give several examples. Finally, we analyze a problem of classification in pattern recognition in order to highlight the importance of these new measures. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

14 pages, 284 KiB

Open AccessArticle

Bounds on the Lifetime Expectations of Series Systems with IFR Component Lifetimes

by Tomasz Rychlik and Magdalena Szymkowiak

Entropy 2021, 23(4), 385; https://doi.org/10.3390/e23040385 - 24 Mar 2021

Cited by 2 | Viewed by 1556

Abstract

We consider series systems built of components which have independent identically distributed (iid) lifetimes with an increasing failure rate (IFR). We determine sharp upper bounds for the expectations of the system lifetimes expressed in terms of the mean, and various scale units based [...] Read more.

We consider series systems built of components which have independent identically distributed (iid) lifetimes with an increasing failure rate (IFR). We determine sharp upper bounds for the expectations of the system lifetimes expressed in terms of the mean, and various scale units based on absolute central moments of component lifetimes. We further establish analogous bounds under a more stringent assumption that the component lifetimes have an increasing density (ID) function. We also indicate the relationship between the IFR property of the components and the generalized cumulative residual entropy of the series system lifetime. Full article

(This article belongs to the Special Issue Measures of Information)

10 pages, 777 KiB

Open AccessArticle

Discrete Versions of Jensen–Fisher, Fisher and Bayes–Fisher Information Measures of Finite Mixture Distributions

by Omid Kharazmi and Narayanaswamy Balakrishnan

Entropy 2021, 23(3), 363; https://doi.org/10.3390/e23030363 - 18 Mar 2021

Viewed by 1834

Abstract

In this work, we first consider the discrete version of Fisher information measure and then propose Jensen–Fisher information, to develop some associated results. Next, we consider Fisher information and Bayes–Fisher information measures for mixing parameter vector of a finite mixture probability mass function [...] Read more.

In this work, we first consider the discrete version of Fisher information measure and then propose Jensen–Fisher information, to develop some associated results. Next, we consider Fisher information and Bayes–Fisher information measures for mixing parameter vector of a finite mixture probability mass function and establish some results. We provide some connections between these measures with some known informational measures such as chi-square divergence, Shannon entropy, Kullback–Leibler, Jeffreys and Jensen–Shannon divergences. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

14 pages, 315 KiB

Open AccessArticle

Results on Varextropy Measure of Random Variables

by Nastaran Marzban Vaselabadi, Saeid Tahmasebi, Mohammad Reza Kazemi and Francesco Buono

Entropy 2021, 23(3), 356; https://doi.org/10.3390/e23030356 - 17 Mar 2021

Cited by 8 | Viewed by 1927

Abstract

In 2015, Lad, Sanfilippo and Agrò proposed an alternative measure of uncertainty dual to the entropy known as extropy. This paper provides some results on a dispersion measure of extropy of random variables which is called varextropy and studies several properties of this [...] Read more.

In 2015, Lad, Sanfilippo and Agrò proposed an alternative measure of uncertainty dual to the entropy known as extropy. This paper provides some results on a dispersion measure of extropy of random variables which is called varextropy and studies several properties of this concept. Especially, the varextropy measure of residual and past lifetimes, order statistics, record values and proportional hazard rate models are discussed. Moreover, the conditional varextropy is considered and some properties of this measure are studied. Finally, a new stochastic comparison method, named varextropy ordering, is introduced and some of its properties are presented. Full article

(This article belongs to the Special Issue Measures of Information)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Measures of Information

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Related Special Issues

Published Papers (10 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI