A Dynamical Systems-Based Hierarchy for Shannon, Metric and Topological Entropy

Abstract

A rigorous dynamical systems-based hierarchy is established for the definitions of entropy of Shannon (information), Kolmogorov–Sinai (metric) and Adler, Konheim & McAndrew (topological). In particular, metric entropy, with the imposition of some additional properties, is proven to be a special case of topological entropy and Shannon entropy is shown to be a particular form of metric entropy. This is the first of two papers aimed at establishing a dynamically grounded hierarchy comprising Clausius, Boltzmann, Gibbs, Shannon, metric and topological entropy in which each element is ideally a special case of its successor or some kind of limit thereof.

1. Introduction

Entropy, which can among a variety of other things, be roughly viewed as a measure of uncertainty (cf. [1]), has been and remains a fascinating and still not completely understood concept with an amazing range of applications, including quantum and ecological systems [2]. We have discovered that many mathematical, scientific and engineering colleagues share our abiding curiosity about rigorous connections among the myriad definitions of entropy. For example, what if any is the relationship between classical Clausius entropy and topological entropy or even Shannon entropy? There have been several extensive investigations of various types of entropy including historical accounts of the relevant developments and informative investigations of the linkages among the various forms of entropy such as in [3,4,5,6,7,8,9,10,11,12,13], but they all appear to be somewhat lacking in terms of identification of truly compelling unifying themes for the multifarious definitions.

Our intention in this and a subsequent paper is to provide a rigorous partial answer to this question by proving that there is a dynamical systems thread connecting all of the following definitions of entropy leading, with the possible addition of some mild assumptions, to the hierarchy

$$\mathrm{Clausius}⟵\mathrm{Gibbs}⟵\mathrm{Boltzmann}⟵\mathrm{Shannon}⟵\mathrm{metric}⟵\mathrm{topological}$$

By rigor as used here, we mean that any connections and special case identifications shall be proved. In this paper, we shall establish a dynamical systems thread connecting Shannon, metric and topological entropy; in a forthcoming paper we hope to complete the hierarchy by elucidating the connections among the first three of the above and their link to the last three. In a related note, we recommend a recent interesting functorial connection among these three entropies introduced by Delvenne [14].

The material to be presented is organized as follows. In the interest of completeness, we present the definitions and some basic properties of topological, metric and Shannon entropy. Then, in Section 3, we prove that metric entropy is a special case of topological entropy if one adds just a few assumptions. Equality of metric and topological entropy can and shall be established in the following two forms: Constructively, namely by showing that certain measurable dynamical systems can be given a topology, which might be quite far removed from some of the more usual types, for which the two entropies in question are equal and by comparison, where one can sometimes determine when a given topology on a measurable dynamical system yields equality of the entropies. Next, in Section 4, we show that Shannon entropy is special case of metric entropy when formulated in a certain dynamical context. In particular, one can frame the underlying information theory foundation as a measurable dynamical system in which the metric and Shannon entropies are equal. As an example, we show how the result for a binary alphabet can be obtained essentially from scratch, in stark contrast to the general result due mainly to Kolmogorov and Sinai that involves very deep and extensive analysis. Finally, in Section 5 we summarize some of the conclusions reached and outline our plans for related future work on extending the hierarchy for entropies.

2. Topological, Metric and Shannon Entropy

In this section we define topological, metric and Shannon entropy, list some of their properties, and describe several basic relationships among them. We begin with the topological entropy of a discrete dynamical system on a topological space.

2.1. Topological Entropy

Let X be a nonempty Hausdorff topological space with topology $\mathfrak{T}$ (comprising the open subsets of the space). A topological discrete (semi-) dynamical system (or just discrete dynamical system for short) is a continuous action of the form

$$F:{\mathbb{N}}^{*}\times X\to X,$$

where $F(n,x):=f^n(x)$, $f:X\to X$ is a continuous map, $f^n$ is the n-th iterate of the map under composition ∘, with the convention that $f^0$ is the identity map $id=i{d}_X$ and ${\mathbb{N}}^{*}$ is the abelian semigroup $\{0\}\cup \mathbb{N}$ under addition. There is a standard nomenclature (see, for example [15]) that includes such definitions as (positive semi-) orbits $O(x):=\{f^n(x):n\in$${\mathbb{N}}^{*}\}$ and fixed points where $f(x)=x$, so that $O(x)=\{x\}$. For convenience, we shall denote the dynamical system by $\mathfrak{D}:=\left(f,X\right)$.

Assuming X to be compact, Adler et al. [16] defined the topological entropy of the dynamical system as follows: If $\mathcal{U}=\{U\}$ is any open covering of $X$, the minimum cardinality $N(\mathcal{U})$ over all subcoverings must be finite owing to the compactness of X. Now, if $\mathcal{U}$ and $\mathcal{V}=\{V\}$ are two open coverings of X, so is their common refinement

$$\mathcal{U}\vee \mathcal{V}:=\left\{U\cap V:(U,V)\in \mathcal{U}\times \mathcal{V}\right\},$$

every set of which is contained in a set of both $\mathcal{U}$ and $\mathcal{V}$, with the refinement relation being typically denoted as $\mathcal{U},\mathcal{V}\prec \mathcal{U}\vee \mathcal{V}$. By iterating refinement, for each $n\in \mathbb{N}$ we obtain the open covering ${\mathcal{U}}^n:=\mathcal{U}\vee f^{-1}(\mathcal{U})\vee \cdots \vee f^{-n+1}(\mathcal{U})$ for which it can readily be shown that

$$h_{\mathfrak{T}}\left(\mathcal{U},f\right)=\underset{n\to \infty }{lim}\frac{logN\left({\mathcal{U}}^n\right)}{n}=inf\left\{\frac{logN\left({\mathcal{U}}^n\right)}{n}:n\in \mathbb{N}\right\}$$

(1)

exists. Then, the topological entropy is defined as

$$h_{\mathfrak{T}}\left(f\right):=sup\left\{h_{\mathfrak{T}}\left(\mathcal{U},f\right):\mathcal{U}\in OC(X)\right\},$$

(2)

where $OC(X)$ is the set of all open coverings of X.

There is also another (metric-based) definition of topological entropy (see Bowen [17], Dinaburg [4,5] and [18]), which for compact metric spaces is equivalent to the Adler–Konheim–McAndrew definition above.

It should be remarked that although we have confined ourselves to compact spaces, it is possible to extend the definition of topological entropy to noncompact spaces (cf. [19,20,21]). There is an analogous definition of topological entropy for continuous dynamical systems that applies to continuous (usually continuously differentiable), which we include for possible future reference. It applies to continuous actions of the type

$$\mathsf{\Phi }:\mathbb{R}\times X\to X,$$

where $\mathbb{R}$ is the topological abelian group of real numbers with respect to addition. These actions arise naturally from solutions of autonomous systems of ordinary differential equations satisfying conditions guaranteeing global existence and uniqueness. There is the usual notation for these systems, which includes for example, the positive semiorbit through x defined as $O_{+}(x):=\{{\phi }_t(x):=\mathsf{\Phi }(t,x):t\ge 0\}$ The associated time-1 map ${\phi }_1$, which generates a discrete dynamical system, is defined as ${\phi }_1(x):=\mathsf{\Phi }(1,x)$, and the topological entropy of the continuous dynamical system is

$$h_{\mathfrak{T}}\left(\mathsf{\Phi }\right):=h_{\mathfrak{T}}\left({\phi }_1\right)=sup\left\{h_{\mathfrak{T}}\left(\mathcal{U},{\phi }_1\right):\mathcal{U}\in OC(X)\right\}=sup\left\{h_{\mathfrak{T}}\left(\mathcal{U},f\right):\mathcal{U}\in OC(X)\right\}.$$

(3)

Finally, we note in passing that the topological entropy in both the discrete and continuous cases can be intuitively viewed as the exponential growth in distinct orbits with discrete or continuous time, respectively.

Some Properties of Topological Entropy

Obviously, the topological entropy is nonnegative, and one can use the basic definitions and a bit of straightforward analysis (as in [15,16]) to prove the following properties:

(TE1)

$h_{{\mathfrak{T}}_X}(f)=h_{{\mathfrak{T}}_Y}(g)$ if $f:X\to X$ and $g:Y\to Y$ are topologically conjugate(there is a homeomorphism $\phi :X\to Y$ such that $g\circ \phi =\phi \circ f$)

(TE2)

$h_{\mathfrak{T}}(f)\ge h_{{\mathfrak{T}}_Y}\left(f_{|Y}\right)$ if $Y\subset X$ is closed and f-invariant and ${\mathfrak{T}}_Y$ is the subspace topology of $\mathfrak{T}$ on Y, where $f_{|Y}$ denotes the restriction of f to Y

(TE3)

$h_{\mathfrak{T}}(f)=h_{\mathfrak{T}}(f^{-1})$ if f is a homeomorphism

(TE4)

$h_{\mathfrak{T}}(f^n)=n{h}_{\mathfrak{T}}(f)$ if $n\in \mathbb{N}$

(TE5)

$h_{{\mathfrak{T}}_X\times {\mathfrak{T}}_Y}(f\times g)=h_{{\mathfrak{T}}_X}(f)+h_{{\mathfrak{T}}_Y}(g)$

(TE6)

If a sequence $\{{\mathcal{U}}_n\}$ of open covers is refining - meaning that ${\mathcal{U}}_n\prec {\mathcal{U}}_{n+1}$ for every $n\in \mathbb{N}$ and for every open cover $\mathcal{V}$ of X there exists an n such that $\mathcal{V}\prec$${\mathcal{U}}_n$ - then $\{h_{\mathfrak{T}}\left({\mathcal{U}}_n,f\right)\}$ is a nondecreasing sequence, with $h_{\mathfrak{T}}\left(f\right)={lim}_{n\to \infty }{h}_{\mathfrak{T}}\left({\mathcal{U}}_n,f\right)=sup\{h_{\mathfrak{T}}\left({\mathcal{U}}_n,f\right):n\in \mathbb{N}\}$.

2.2. Kolmogorov–Sinai Metric Entropy

A discrete measurable dynamical system consist of a nonempty set X, a $\sigma$-algebra of $\mu$-measurable subsets $\mathcal{M}$ of X, where $\mu$ is a probability measure (with $\mu (X)=1$) and a measurable function $f:X\to X$ (satisfying $f^{-1}(A)\in \mathcal{M}$ whenever $A\in \mathcal{M}$) such that $\mu$ is f-invariant, which means that $\mu \circ f^{-1}=\mu$ on $\mathcal{M}$. The iterates of f obviously also define a discrete dynamical system, with the usual notions of orbits, fixed points and the like.

The metric(Kolmogorov–Sinai or just K–S) entropy requires a number of introductory elements for its definition. A measurable partition of X is a finite, pairwise-disjoint sequence $\mathcal{P}=\{Q_1,\dots ,Q_m\}$ of $\mu$-measurable sets of positive measure such that $X={\bigcup }_{k=1}^m{Q}_k$, and it is convenient to denote the set of all such measurable partitions of X as $\mathfrak{P}(X)$. The entropy of a measurable partition is defined as

$$H(\mathcal{P}):=-{\displaystyle {\sum }_{Q\in \mathcal{P}}}\mu (Q)log\mu (Q).$$

(4)

As with open coverings in the definition, we can define a common refinement of measurable partitions $\mathcal{P}$ and $\tilde{\mathcal{P}}$ as

$$\mathcal{P}\vee \tilde{\mathcal{P}}:=\left\{Q\cap \tilde{Q}:(Q,\tilde{Q})\in \mathcal{P}\times \tilde{\mathcal{P}}\right\},$$

for which the refinement relation is analogously denoted by $\mathcal{P},\tilde{\mathcal{P}}\prec \mathcal{P}\vee \tilde{\mathcal{P}}$.

Iterating refinement in a manner analogous to that used for topological entropy, we define measurable partitions associated to f of the form

$${\mathcal{P}}^n:=\mathcal{P}\vee f^{-1}(\mathcal{P})\vee \cdots \vee f^{-n+1}(\mathcal{P}),$$

and define

$$H\left(\mathcal{P},f\right):=\underset{n\to \infty }{lim}\frac{H\left({\mathcal{P}}^n\right)}{n},$$

(5)

where the limit can be readily shown to exist. Then, we define the metric entropy as

$$h_{\mu }(f):=sup\left\{H\left(\mathcal{P},f\right)\right\}:\mathcal{P}\in \mathfrak{P}(X)\},$$

(6)

and take note of the strong similarity with the definition of topological entropy.

It should also be mentioned that just as for the case of topological entropy, we can define metric entropy for continuous measurable dynamical systems using the time-1 map of the flow as in (3). As with the topological entropy, there are intuitive characterizations of metric entropy, two of which are the following: The exponential growth rate of typical orbits; and the maximum of the rate of extractable information.

Basic Properties of K–S Entropy

The nonnegativity of K–S entropy is clear from its definition, and we have the following basic properties that are clearly analogs of those for topological entropy, with the last of which following from the concavity of $-xlogx$ (see [10,15,22,23,24,25]).

(ME1)

$h_{\mu }(f)=h_{\nu }(g)$ if $f:X\to X$ and $g:Y\to Y$ are metrically conjugate(there is a measurable bijection $\phi :X\to Y$ such that $g\circ \phi =\phi \circ f$ and $\nu =\mu \circ {\phi }^{-1}$)

(ME2)

$h_{\mu }(f)\ge h_{{\mu }_{|Y}}\left(f_{|Y}\right)$ if $Y\subset X$ is measurable and f-invariant and ${\mu }_{|Y}$ is the restriction of $\mu$ to Y, where $f_{|Y}$ denotes the restriction of f to Y

(ME3)

$h_{\mu }(f)=h_{\mu }(f^{-1})$ if f is a measurable bijection

(ME4)

$h_{\mu }(f^n)=n{h}_{\mu }(f)$ if $n\in \mathbb{N}$

(ME5)

$h_{\mu \times \nu }(f\times g)=h_{\mu }(f)+h_{\nu }(g)$

(ME6)

If a sequence $\left\{{\mathcal{P}}_n=\{Q_1,\dots ,Q_{m_n}\}\right\}$ in $\mathfrak{P}(X)$ is refining - meaning that ${\mathcal{P}}_n\prec {\mathcal{P}}_{n+1}$ for every $n\in \mathbb{N}$ and $∥{\mathcal{P}}_n∥:=max\{\mu \left(Q_k\right):1\le k\le m_n\}\to 0$ as $n\to \infty$ - then $\{h_{\mu }\left({\mathcal{U}}_n,f\right)\}$ is a nondecreasing sequence with $h_{\mu }\left(f\right)={lim}_{n\to \infty }{h}_{\mu }\left({\mathcal{U}}_n,f\right)=sup\{h_{\mu }\left({\mathcal{U}}_n,f\right):n\in \mathbb{N}\}$.

2.3. Shannon Entropy

In contrast with topological and metric entropy, Shannon entropy (see [26,27]) has an information theory rather than a dynamical system foundation. The basic elements can be distilled as follows: Let $S:=\{s_1,\dots ,s_m\}$ be a nonempty finite set of symbols or messages (sometimes referred to as the alphabet) with a discrete probability p assigned to each, such that $p(s_i)\ge 0$ for all $1\le i\le m$ and $p(s_1)+\cdots +p(s_m)=1.$ Then, the Shannon entropy of the message ensemble

$$H\left(S\right):=-{\displaystyle {\sum }_{i=1}^m}p(s_i)logp(s_i)$$

(7)

is just the average or expected value of the information content of the message ensemble X, which is very much like the entropy of a measurable partition used in the definition of metric entropy.

Several Properties of Shannon Entropy

It follows from (7) that the Shannon entropy is nonnegative, and there are several other readily verified properties such as:

(SE1)

$H(S)=H(\tilde{S})$ if there is a bijection $\phi :S\to \tilde{S}$ such that $g\circ \phi =\phi \circ f$ and $\tilde{p}=p\circ {\phi }^{-1}$

(SE2)

$H(S)\ge H\left(\tilde{S}\right)$ if $\tilde{S}\subset S$

(SE3)

$H(S\times \tilde{S})=H(S)+H(\tilde{S})$ if S and $\tilde{S}$ are independent (prob$(s_i,{\tilde{s}}_j)=p(s_i)\tilde{p}({\tilde{s}}_j)$ for all $i,j$)

(SE4)

$H(S)=logn$ is maximized when $p(s_1)=\cdots =p(s_n)=1/n.$

2.4. A Relationship among the Entropies

Here we shall describe a well known relationship between topological and metric entropy obtained by Dinaburg [4,5], Goodman [8], Goodwyn [9] and Misiurewicz [28] using variational techniques (see also [12,13,15]), and often referred to as the variational principle. Let $f:X\to X$ be a continuous map on a compact metric space X with topology $\mathfrak{T}$, which defines a discrete dynamical system that we represent by the triple $\left(f,X,\mathfrak{T}\right)$. To establish the topological-metric entropy relation, it is convenient to define

$$\mathfrak{M}(X):=\left\{\mu :\mu \mathrm{an}\mathrm{invariant}\mathrm{Borel}\mathrm{probability}\mathrm{measure}\mathrm{on}\mathrm{the}\sigma -\mathrm{algebra}{\mathcal{M}}_{\mu }\mathrm{of}\mathrm{subsets}\mathrm{of}X\right\},$$

(8)

where Borel means that $\mathfrak{T}\subset {\mathcal{M}}_{\mu }$ for all $\mu \in \mathfrak{M}(X)$. Then we have the following results, which is proved very efficiently in [28]:

Theorem 1.

If X is a compact metric space with topology $\mathfrak{T}$ and $f:X\to X$ is a continuous map, then

$$h_{\mathfrak{T}}\left(f\right):=sup\left\{h_{\mu }\left(f\right):\mu \in \mathfrak{M}(X)\right\}.$$

(9)

3. When Topological Entropy Equals Metric Entropy

In this section we shall prove there is a topology on numerous discrete measurable dynamical systems (DMDS) of interest such that the corresponding topological entropy is equal to the K–S entropy. Let $\left(f,X,\mathcal{M},\mu \right)$ be a measurable dynamical system having metric entropy $h_{\mu }\left(f\right)$. We shall show that in certain cases there is a topology $\mathfrak{T}$ on X for which $f:X\to X$ is continuous and $h_{\mathfrak{T}}\left(f\right)=h_{\mu }\left(f\right)$. One interesting example, which is an example of type (F2) that concerns Conjecture 3 of [16], which has since been proven (see [17,18,21,29]); namely, if X is a compact separable topological group, with topology $\mathfrak{T}$, and $f:X\to X$ is a continuous automorphism, then the Haar measure $\mu$ is f-invariant, and $h_{\mathfrak{T}}\left(f\right)=h_{\mu }\left(f\right)$. What we also have in mind is a Constructive-type of problem that involves defining a topology on a given DMDS, with no pre-specified topology, such that the topological and K-S entropies coincide and this is where we shall begin.

3.1. Constructive-Type Equality

The question is when can the phase space X of a given DMDS can be endowed with a compact metric topology $\mathfrak{T}$ such that $h_{\mathfrak{T}}\left(f\right)=h_{\mu }\left(f\right)$. Our main result, which follows directly from the Jewett–Krieger theorem (cf. [3,30,31]), seems not to have appeared in the literature.

Theorem 2.

Let $\mathfrak{D}=\left(f,X,\mathcal{M},\mu \right)$ be a measurable dynamical system, with $f:X\to X$ bijective and ergodic. Then, there exists a compact metric topology $\mathfrak{T}$ on X such that $h_{\mathfrak{T}}\left(f\right)=h_{\mu }\left(f\right)$.

Proof. 

The Jewett–Krieger theorem implies that there is a compact metric space $\widehat{X}$ with topology $\widehat{\mathfrak{T}}$ with a continuous map $\widehat{f}:\widehat{X}\to \widehat{X}$, $\widehat{f}$-invariant Borel measure $\widehat{\mu }$, and a sigma-algebra $\widehat{\mathcal{M}}$ of $\widehat{\mu }$-measurable subsets of $\widehat{X}$ defining a measure-theoretic dynamical system $\widehat{\mathfrak{D}}=\left(\widehat{f},\widehat{X},\widehat{\mathcal{M}},\widehat{\mu }\right)$ with the following properties:

(i)

There is a measure-theoretic dynamical system embedding $\mathsf{\Phi }:\mathfrak{D}\to \widehat{\mathfrak{D}}$, which means there is an injective map $\varphi :X\to \widehat{X}$ such that $\widehat{f}\circ \varphi =\varphi \circ f$ and $A\in \mathcal{M}$ implies that $\varphi (A)\in \widehat{\mathcal{M}}$, with $\widehat{\mu }\left(\varphi (A)\right)=\mu (A).$

(ii)

If ${\widehat{\mu }}_{*}$ is the restriction of $\widehat{\mu }$ to $\varphi (X)$ and ${\widehat{\mathfrak{T}}}_{*}$ is the subspace topology induced by $\widehat{\mathfrak{T}}$ on $\varphi (X)$, then $h_{\widehat{\mathfrak{T}}}(\widehat{f})=h_{{\widehat{\mu }}_{*}}(\widehat{f})$.

The choice of the desired topology can be inferred readily from the above; namely, we define $\mathfrak{T}$ to be the topology induced by the injection $\varphi :X\to \widehat{X}$. As is well known, this topology $\mathfrak{T}$ is simply that which is generated by the set $\left\{{\varphi }^{-1}(V):V\in \widehat{\mathfrak{T}}\right\}$. Hence, owing to the properties described above, the systems are rendered both topologically and metrically conjugate on $\varphi (X)$. This completes the proof since it follows that (ii) implies $h_{\mathfrak{T}}\left(f\right)=h_{\mu }\left(f\right)$. □

Although the above is a simple corollary of the Jewett–Krieger result, the proof of the theorem itself is quite long and deep. This suggests that there may be a more direct proof of Theorem 1, which is something worth investigating.

3.2. Comparison-Type Equality

Since we start with a choice of the topology, the construction involves the possibility of noncompactness for which there are approaches in the literature such as [18,19,20,21,29,32,33]. However, we avoid this issue by selecting a compact topology for X.

We begin with the construction of the topology $\mathfrak{T}$ for the phase space X of the DMDS $\mathfrak{D}:=\left(f,X,\mathcal{M},\mu \right)$ that we hope yields the equality of the two entropies. A rather natural choice for $\mathfrak{T}$ is the topology ${\mathfrak{T}}_0(\mu )$ generated by the sets in all possible measurable partitions of X comprising subsets of positive measure, but this has inherent problems, not the least of which concerns compactness. Certainly, $\left(X,{\mathfrak{T}}_0(\mu )\right)$ is not a priori compact, nor as it turns out is it suitable for proving that the corresponding topological and given metric entropies are equal. Consequently, we must be more restrictive in the definition of the chosen topology as well as the DMDS.

First, we deal with the choice of topology and then describe some of the fundamental properties of the DMDS vis a vis the topology that we will use to guarantee the equality of the metric and topological entropies. Our first assumption is that X can be compactly embedded in a metric space Y with metric d, so it may be considered as a compact metric space with metric d restricted to X, with corresponding metric topology ${\mathfrak{T}}_d$. This immediately takes care of the question of compactness, for example, and provides other desirable properties associated with metric topologies. Moreover, it applies to any finite-dimensional, smooth compact manifold. Another convenient assumption connecting the topology ${\mathfrak{T}}_d$ and $\mathfrak{D}$ is the following: The measure is Borel with respect to the topology; i.e., ${\mathfrak{T}}_d$ $\subset \mathcal{M}$ and so all open sets are $\mu$-measurable. It is worth noting that compact smooth manifolds with the usual types of measures typically satisfy these two assumptions. We shall also find it convenient to assume that the map f is continuous with respect to ${\mathfrak{T}}_d$, which implies that the map is also proper; i.e., the preimage of compact sets are compact.

3.2.1. Definitions and Additional Notation

The above assumptions are basic and conveniently serve our needs, but several more subtle properties are required to obtain equality of entropies, which are best delineated by introducing some additional simplifying notation in the context of the assumptions that have so far been made. For example, the following notion shall prove useful.

Definition 1.

A (finite) open cover $\mathcal{U}=\{U_1,\dots ,U_m\}$ of X is minimal with respect to a measurable partition $\mathcal{P}=\{Q_1,\dots ,Q_m\}$ of X if $Q_k\subset U_k$ for every $1\le k\le m$ and no proper subset of $\mathcal{U}$ is a covering of X. For convenience, we denote this as $\mathcal{U}$-min-$\mathcal{P}$.

Definition 2.

A measurable partition $\mathcal{P}=\{Q_1,\dots ,Q_m\}$ is an α-β (with $0<\alpha <\beta$) partition of X with respect to ${\mathfrak{T}}_d$ and denoted as $\mathcal{P}\left(\alpha ,\beta \right)$ if the following properties obtain for all the elements: (i) $\mu (Q_k)>0$; (ii) $Q_k$ is connected in the topology ${\mathfrak{T}}_d$; (iii) the diameter $d(Q_k)<\beta$; and (iv) there is at least one point $x\in Q_k$ such that the (closed ball) ${\overline{B}}_{\alpha }(x):=\{y\in X:d(x,y)\le \alpha \}$, corresponding to the open ball $B_{\alpha }(x):=\{y\in X:d(x,y)<\alpha \}$, is contained in $Q_k$. We denote the set of all α-β partitions as ${\mathfrak{P}}_{\alpha \beta }(X)$.

Definition 3.

The DMDS $\mathfrak{D}$ is of L-type for ${\mathfrak{T}}_d$ if it satisfies the following properties: (i) $\mu (B_{\alpha }(x)>0$ for every $\alpha >0$; (ii) $\mu (B_{\alpha }(x)\to 0$ as $\alpha \to 0$; (iii) for every element $Q_k$ of an α-β partition of X and $ϵ>0$ there is a connected open set $U_k(ϵ)$ such that $Q_k\subset U_k(ϵ),$ $\mu \left(U_k(ϵ)\setminus Q_k\right)<ϵ$ and $d\left(U_k(ϵ)\right)<\beta +ϵ$; (iv) $d\left(x,\partial U_k(ϵ)\right)<ϵ/2$ for every $x\in \partial Q_k$ and (v) $U_k({ϵ}_1)\subset$$U_k({ϵ}_2)$ whenever ${ϵ}_1<{ϵ}_2$, so that ${lim}_{ϵ\downarrow 0}\mu \left(U_k(ϵ)\right)=\mu \left(Q_k\right)$. We note that here the L is for Lebesgue, inasmuch as these properties are well known to apply for the normalized Lebesgue measure on a compact subset of Euclidean space.

Definition 4.

Let $\mathcal{P}=\{Q_1,\dots ,Q_m\}\in {\mathfrak{P}}_{\alpha \beta }(X)$ and $\mathcal{U}(\mathcal{P},ϵ)=\{U_1(ϵ),\dots ,U_m(ϵ)\}$ be a corresponding open covering in accordance with Definition 3. Then we say that open covering $\mathcal{U}(\mathcal{P},ϵ)$ is ϵ-tight with respect to $\mathcal{P}$.

The following result is a readily verified consequence of the above definitions.

Lemma 1.

Let $\mathcal{P}=\{Q_1,\dots ,Q_m\}\in {\mathfrak{P}}_{\alpha \beta }(X)$. Then, for any given integer $n>1$ and $ϵ>0$ there exists a δ with $0<\delta =\delta (ϵ,n)<ϵ$ such that if $\mathcal{U}=\mathcal{U}(\mathcal{P},\delta )=\{U_1(\delta ),\dots ,U_m(\delta )\}$ is δ-tight and (consequently for δ sufficiently small) minimal with respect to $\mathcal{P}$, ${\mathcal{U}}^k={\mathcal{U}}^k(\mathcal{P},ϵ)={\{U_1(ϵ),\dots ,U_m(ϵ)\}}^k$ is ϵ-tight and minimal with respect to ${\mathcal{P}}^k={\{Q_1,\dots ,Q_m\}}^k$ for all $1\le k\le n.$

3.2.2. The Comparison Theorem

We now assemble some of the properties defined above for use as assumptions in the main theorem of this section.

Definition 5.

The measurable dynamical system $\mathfrak{D}:=\left(f,X,\mathcal{M},\mu \right)$ is T-compatible if there exists a compact metric topology ${\mathfrak{T}}_d$ on X such that the following properties obtain: (i) f is continuous with respect to ${\mathfrak{T}}_d$ (ii) μ is Borel with respect to ${\mathfrak{T}}_d$; (iii) ${\mathfrak{P}}_{\alpha \beta }(X)\ne ⌀$; and (iv) $\mathfrak{D}$ is of L-type for ${\mathfrak{T}}_d.$

The stage has now been set for the following result on comparative equality.

Theorem 3.

Suppose that the DMDS $\mathfrak{D}$ is T-compatible with respect to ${\mathfrak{T}}_d$ $on$ X and the following additional properties hold:

(E1) 

There exists a sequence of partitions $\{\mathcal{P}({\alpha }_n,{\beta }_n)\}$ in ${\mathfrak{P}}_{\alpha \beta }(X)$ such that $\mathcal{P}({\alpha }_n,{\beta }_n)\prec \mathcal{P}({\alpha }_{n+1},{\beta }_{n+1})$ for all $n\in \mathbb{N}$ and $\{{\alpha }_n\}$ and $\{{\beta }_n\}$ are decreasing sequences converging to zero, which means that $\mathcal{P}\left({\alpha }_n,{\beta }_n\right)$ is refining in the sense of (ME6).

(E2) 

Moreover, the sequence in (E1) satisfies the following property: For every increasing sequence of positive integers $\{j_k\}$ there exists a dominating sequence of natural numbers $\{n_k\},$ with $n_k>j_k$ for all $k\in \mathbb{N},$ such that ${\mathcal{P}}^{n_k}({\alpha }_k,{\beta }_k)=\left\{Q_{1(k,n_k)},\dots ,Q_{m(k,n_k)}\right\}$ and

$$H\left({\mathcal{P}}^{n_k}({\alpha }_k,{\beta }_k)\right)=log\left(m(k,n_k)\right)-\sigma (k,n_k),$$

(10)

for all $k\in \mathbb{N},$ where $\sigma (k,n_k)>0$ is bounded for all $(k,n_k)\in \mathbb{N}\times \mathbb{N}.$

Then, $h_{{\mathfrak{T}}_d}\left(f\right)=h_{\mu }\left(f\right).$

Proof. 

A key element of our proof is the recognition that (E2) expressed in Equation (10) is tantamount to ${\mathcal{P}}^{n_k}({\alpha }_k,{\beta }_k)$ being nearly equiprobable and that a sufficiently tight open covering $\mathcal{U}$ of $\mathcal{P}({\alpha }_k,{\beta }_k)$ yields a tight open cover ${\mathcal{U}}^{n_k}$ with respect to ${\mathcal{P}}^{n_k}({\alpha }_k,{\beta }_k)$ such that $logN\left({\mathcal{U}}^{n_k}\right)=log\left(m(k,n_k)\right)$. Consequently,

$$\frac{logN\left({\mathcal{U}}^{n_k}\right)}{n_k}-\frac{H\left({\mathcal{P}}^{n_k}({\alpha }_k,{\beta }_k)\right)}{n_k}:=\Delta (\mathcal{U},k,n_k)=\frac{\sigma (k,n_k)}{n_k}\to 0\mathrm{as}{n}_k\to \infty .$$

Now, selecting $\{\mathcal{P}({\alpha }_n,{\beta }_n)\}$ in ${\mathfrak{P}}_{\alpha \beta }(X)$ as in (E1), it follows from (ME 6) that for every $ϵ>0$ there exists a natural number $l=l(ϵ)$ so large that for any integer $k\ge l(ϵ)$ there is a ${\widehat{q}}_k\in \mathbb{N}$ such that

$$\left|\frac{H\left({\mathcal{P}}^{q_k}({\alpha }_k,{\beta }_k)\right)}{q_k}-h_{\mu }(f)\right|<\frac{ϵ}{3}$$

(11)

whenever $q_k$ is an integer greater than ${\widehat{q}}_k$. Moreover, (E2) implies that for each $k\ge l(ϵ)$ there are infinitely instances of Equation (11), which we refer to as ${\tilde{q}}_k$, satisfying

$$\left|\frac{H\left({\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)\right)}{{\tilde{q}}_k}-h_{\mu }(f)\right|=\left|\frac{log\left(m(k,{\tilde{q}}_k)\right)-\sigma (k,{\tilde{q}}_k)}{{\tilde{q}}_k}-h_{\mu }(f)\right|<\frac{ϵ}{3}.$$

(12)

There is, owing to Lemma 1, a decreasing sequence $\{{\delta }_n\}$ with ${\delta }_n\downarrow 0$ as $n\to \infty$ associated to a refining sequence of ${\delta }_k$-tight open coverings $\{{\mathcal{U}}_k\}$ of $\{\mathcal{P}({\alpha }_k,{\beta }_k)\}$ that is refining in the sense of (TE 6). Furthermore, the compactness of X, the continuity of $f:X\to X$, (TE 6), (E 2), Equation (10) and Lemma 1 imply that the $\{{\delta }_n\}$ for the sequence of tight open coverings can be chosen so that ${\mathcal{U}}_k^{{\tilde{q}}_k}$ is minimal for ${\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)$ for some ${\tilde{q}}_k$ for which Equation (12) obtains,

$$\left|\frac{logN\left({\mathcal{U}}_k^{{\tilde{q}}_k}\right)}{{\tilde{q}}_k}-h_{{\mathfrak{T}}_d}\left(f\right)\right|<\frac{ϵ}{3}$$

(13)

and

$$\frac{\sigma (k,{\tilde{q}}_k)}{{\tilde{q}}_k}<\frac{ϵ}{3}.$$

(14)

Accordingly it follows from Equations (12)–(14) that

$$h_{{\mathfrak{T}}_d}\left(f\right)-h_{\mu }(f)=\left|h_{{\mathfrak{T}}_d}\left(f\right)-h_{\mu }(f)\right|$$

$$\begin{array}{cc}\hfill & =\left|\left(h_{{\mathfrak{T}}_d}\left(f\right)-\frac{logN\left({\mathcal{U}}_k^{{\tilde{q}}_k}\right)}{{\tilde{q}}_k}\right)+\left(\frac{logN\left({\mathcal{U}}_k^{{\tilde{q}}_k}\right)}{{\tilde{q}}_k}-\frac{H\left({\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)\right)}{{\tilde{q}}_k}\right)+\left(\frac{H\left({\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)\right)}{{\tilde{q}}_k}-h_{\mu }(f)\right)\right|\hfill \\ \hfill & \le \left|h_{{\mathfrak{T}}_d}\left(f\right)-\frac{logN\left({\mathcal{U}}_k^{{\tilde{q}}_k}\right)}{{\tilde{q}}_k}\right|+\left|\frac{H\left({\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)\right)}{{\tilde{q}}_k}-h_{\mu }(f)\right|+\left|\frac{logN\left({\mathcal{U}}_k^{{\tilde{q}}_k}\right)}{{\tilde{q}}_k}-\frac{H\left({\mathcal{P}}^{{\tilde{q}}_k}({\alpha }_k,{\beta }_k)\right)}{{\tilde{q}}_k}\right|<\frac{ϵ}{3}+\frac{ϵ}{3}+\frac{ϵ}{3}=ϵ.\hfill \end{array}$$

Hence, as $ϵ$ is arbitrary, the proof is complete. □

We note that it appears that Theorem 2 can also be proved using the concept of an f-homogeneous invariant measure introduced in [17]. It is actually likely that the hypotheses in the above theorem are equivalent to Bowen’s f-homogeneous invariant Borel probability measure $\mu$ property for a discrete dynamical system $f:X\to X$ on a compact metric space, which is defined as follows: For every $ϵ>0$, there exist $\delta ,c>0$ such that

$$\mu \left({\displaystyle {\bigcap }_{k=0}^{n-1}}{f}^{-k}\left(B_{\delta }\left(f^k(y)\right)\right)\right)\le c\mu \left({\displaystyle {\bigcap }_{k=0}^{n-1}}{f}^{-k}\left(B_{ϵ}\left(f^k(x)\right)\right)\right)$$

for all $n\ge 0$ and $x,y\in X$, where $B_{\delta }$ and $B_{ϵ}$ are the standard $\delta$ and $ϵ$ balls, respectively, for the metric d generating the topology ${\mathfrak{T}}_d$ on X. He proved that if this property holds, then

$$h_{{\mathfrak{T}}_d}\left(f\right)=h_{\mu }\left(f\right)=\underset{ϵ\to 0}{lim}\left\{lim\underset{n\to \infty }{sup}\left[-\frac{1}{n}log\mu \left({\displaystyle {\bigcap }_{k=0}^{n-1}}{f}^{-k}\left(B_{ϵ}\left(f^k(y)\right)\right)\right)\right]\right\}$$

In the same vein, it is likely that Theorem 2 can be employed to prove the equivalence of the metric and topological entropies for a DMDS comprising a continuous automorphism of a compact separable topological group leaving the Haar measure invariant. Another thing worth noting is the strong indications that this theorem is apt to have analogs for several variants of entropy, such as relative entropy, as well as some extensions to the noncompact versions of entropy in [18,19,20,21,29,32,33] and these seem like they might be an interesting topics for future research.

3.3. Examples of Topological and Metric Entropy Equivalence

We mentioned the case of automorphisms on compact, separable topological groups and now we want to show some examples that follow from Theorems 1 and 2. Most of these examples have already been covered in the literature, so they are mainly meant for illustrative purposes.

Example 1.

Let $f:{\mathbb{S}}^1\to {\mathbb{S}}^1$ be a rotation that is an irrational multiple of $2\pi$, where ${\mathbb{S}}^1$ is the unit circle with the metric d and measure μ both based on the normalized arclength. Then the associated DMDS is ergodic, so Theorem 1 implies that $h_{{\mathfrak{T}}_e}\left({\mathbb{S}}^1\right)=h_{\mu }\left({\mathbb{S}}^1\right)$, which is readily shown to be zero as is that of every rotation of the circle. It is also a simple matter to obtain the same result using Theorem 2. On the other hand, if we chose a rational multiple of $2\pi$ rotation, the DMDS is no longer ergodic, so Theorem 1 cannot be used, but the result can be obtained by employing Theorem 2.

Example 2.

Consider the standard tent map $f=\mathsf{\Lambda }:[0,1]\to [0,1]$ defined as

$$\mathsf{\Lambda }(x):=\left\{\begin{array}{cc}2x,& 0\le x<1/2\\ 2(1-x),& 1/2\le x\le 1\end{array}\right..$$

combining this with the Lebesgue measure space on the unit interval and the euclidean topology ${\mathfrak{T}}_e$, we obtain a DMDS on the compact unit interval. It is easy to see that by successively bisecting the unit interval, we obtain a refining sequence of partitions in satisfying Theorem 2 for all $\{n_k\}$, from which we conclude that $h_{{\mathfrak{T}}_e}\left(\mathsf{\Lambda }\right)=h_{\mu }\left(\mathsf{\Lambda }\right)=log2$.

Example 3.

The same reasoning shows that the $2x(mod1)$ map $f_{2/1}:[0,1]\to [0,1]$ defined as

$$f_{2/1}(x):=\left\{\begin{array}{cc}2x,& 0\le x<1/2\\ 2x-1,& 1/2\le x\le 1\end{array}\right.,$$

with respect to the Lebesque measure and the euclidean topology ${\mathfrak{T}}_e$, is case where $h_{{\mathfrak{T}}_e}\left(f_{2/1}\right)=h_{\mu }\left(f_{2/1}\right)=log2$.

Example 4.

The previous example can be readily generalized to all $mx(mod1)$ maps $f_{m/1}:[0,1]\to [0,1]$ for $m\in \mathbb{N}$, $m\ge 2$, defined as

$$f_{m/1}(x):=\left\{\begin{array}{cc}mx,& 0\le x<1/m\\ mx-1,& 1/m\le x<2/m\\ \cdots ,& 2/m\le x<(m-1)/m\\ mx-(m-1)& (m-1)/m\le x\le 1\end{array}\right.,$$

with the same measure and topological structure above for the $2x(mod1)$ map. The result is $h_{{\mathfrak{T}}_e}\left(f_{m/1}\right)=h_{\mu }\left(f_{m/1}\right)=logm$.

Example 5.

We can neatly recast the $mx(mod1)$ map examples as smooth maps on a smooth manifold and extend them to higher dimensions with ease. To begin, we again let ${\mathbb{S}}^1$ be the unit circle in the complex plane $\mathbb{C}$ with the standard arclength measure μ normalized by a factor of $1/2\pi$, so that it is a probability measure, with the usual Riemannian arclength metric d also scaled by a factor of $1/2\pi$. Then, the $mx(mod1)$ maps can be identified with the smooth maps $z\to z^m$ restricted to the unit circle; namely, $F_m:{\mathbb{S}}^1\to {\mathbb{S}}^1$ defined by ${(z^m)}_{|{\mathbb{S}}^1}$ so that $F_m(e^{i\theta }):=e^{im\theta }$, and this combination defines a DMDS on a compact space. By mimicking the construction in the previous example, it can be readily shown the Theorem 2 implies that $h_{{\mathfrak{T}}_e}\left(F_m\right)=h_{\mu }\left(F_m\right)=logm$. Similarly, given an n-tuple $m=(m_1,\dots ,m_n)\in {\mathbb{N}}^n,$ we define the smooth self map of the n-torus ${\mathsf{\Phi }}_m:{\mathbb{T}}^n\to {\mathbb{T}}^n$ by

$${\mathsf{\Phi }}_m\left(e^{i{\theta }_1},\dots ,e^{i{\theta }_m}\right):=\left(e^{i{m}_1{\theta }_1},\dots ,e^{i{m}_n{\theta }_n}\right),$$

where as usual, ${\mathbb{T}}^n$ is just the cartesian product of n copies of the unit circle with the product topology and product measure associated to the single circle. This then comprises a smooth DMDS on the compact n-torus, which can be easily shown to satisfy the hypotheses of Theorem 2 and yield $h_{{\mathfrak{T}}_e}\left({\mathsf{\Phi }}_m\right)=h_{\mu }\left({\mathsf{\Phi }}_m\right)=log(m_1\cdots m_n)$.

4. Shannon Entropy as a Special Case of Metric Entropy

We shall prove that Shannon’s entropy may be formulated in the context of a Bernoulli scheme; whence, it is equal to the corresponding metric entropy owing to the Kolmogorov–Sinai entropy theorem [23,25], which is also covered in [12,13,15,24]. What we undertake in this section was more or less observed in the process of the development of K–S entropy, although not actually proved in detail as in what follows. In this regard, the work of Frigg [34] should also be noted. To recast Shannon entropy (described above) as a measurable dynamical system called a Bernoulli scheme or Bernoulli shift, we define the phase space X as the set of all bi-infinite sequences of symbols; namely,

$$X:=S^{\mathbb{Z}}:=\left\{\zeta :\zeta :\mathbb{Z}\to S\right\}=\left\{\left(\dots ,{\zeta }_{-2},{\zeta }_{-1},{\zeta }_0.{\zeta }_1,{\zeta }_2,\dots \right):{\zeta }_k\in S=\{s_1,\dots ,s_n\}\forall k\in \mathbb{Z}\right\},$$

and the (bijective) map $\mathfrak{s}:X\to X$ to be the (Bernoulli) shift

$$\mathfrak{s}\left(\zeta \right)(k)=\zeta (k+1)$$

for all $k\in \mathbb{Z}$ together with the following probability distribution $(p_1,\dots ,p_n)$ for the symbol set S, where $p_k:=p(s_k)>0$ for all $k\in \mathbb{Z}$ and $p_1+p_2+\cdots +p_n=1$. This definition fits rather nicely with the information foundation of Shannon entropy inasmuch as it corresponds to reading the string of symbols from left to right one at a time.

To complete the reformulation as a measurable dynamical system, it remains to define a $\sigma$-algebra of $\mu$-measurable subsets of X, denoted as $\mathcal{M}$, where $\mu$ is an $\mathfrak{s}$-invariant probability measure on X. This can be done in many ways, so the trick is to find a definition that yields a metric entropy equal to the Shannon entropy. Toward this end, we start by defining cylinder sets of the form

$$C\left(F,\psi \right):=\left\{\zeta \in X:{\zeta }_{|F}=\psi \right\},$$

where F is a finite set of integers, $\psi \in S^F$ and ${\zeta }_{|F}$ is the restriction of $\zeta$ to $F\subset \mathbb{Z}$, and let $\mathcal{C}$ be the collection of all cylinder sets together with the null set and all of X.

Next, we introduce the set function ${\mu }_0:\mathcal{C}\to \mathbb{R}$ defined as follows:

$${\mu }_0(E):=\left\{\begin{array}{cc}0,& E=⌀\\ 1,& E=X\\ {\displaystyle {\prod }_{k\in F}}p\left(\psi (k)\right),& E=C\left(F,\psi \right)\end{array}\right..$$

Now, there is a theorem of Kolmogorov [22] (see also, for example [24], p. 628) of a rather technical nature stating that the above set function can be uniquely extended to a complete probability measure $\mu :\mathcal{M}\to \mathbb{R}$, called the product measure determined by the distribution$\left(p_1,\dots ,p_n\right)$, where $\mathcal{M}$ is the smallest $\sigma$-algebra containing $\mathcal{C}$ and $p_k:=p(s_k)$ for all $1\le k\le n$.

Finally, to show that this measure is invariant under the shift map, it suffices to verify this for cylinder sets, and this can readily seen by first observing that

$${\mathfrak{s}}^{-1}\left(C\left(F,\psi \right)\right)=C\left(\tilde{F},\tilde{\psi }\right),$$

where $\tilde{F}:=F+1:=\{k+1:k\in F\}$ and $\tilde{\psi }:\tilde{F}\to S$ with $\tilde{\psi }(k+1)=\psi (k)$ for all $k\in F$. Consequently,

$$\mu \left({\mathfrak{s}}^{-1}\left(C\left(F,\psi \right)\right)\right)=\mu \left(C\left(\tilde{F},\tilde{\psi }\right)\right)={\displaystyle {\prod }_{j\in \tilde{F}}}{\mu }_0\left(\tilde{\psi }(j)\right)={\displaystyle {\prod }_{k\in F}}{\mu }_0\left(\psi (k)\right)=\mu \left(C\left(F,\psi \right)\right).$$

Thus we have reformulated the Shannon communication system with alphabet S as the measurable dynamical system $\left(\mathfrak{s},X:=S^{\mathbb{Z}},\mathcal{M},\mu \right)$, where $\mathcal{M}$ and $\mu$ are as defined above, which is called a Bernoulli scheme and often denoted as $B\left(p_1,\dots ,p_n\right)$. The details necessary to prove that the metric entropy of the Bernoulli scheme equals the Shannon entropy of the corresponding communication system are quite extensive and covered rather neatly and completely in [24], so we shall only try to summarize the results in what we hope is a readily understood fashion, at least from an intuitive standpoint.

First, the Kolmogorov–Sinai entropy theorem can be roughly stated as follows: Let$\mathfrak{D}=\left(f,X,\mathcal{M},\mu \right)$be a measurable dynamical system withfinvertible, as it is in the case of$B\left(p_1,\dots ,p_n\right)$, such that there is a measurable partition$\mathcal{P}$for which$\mathcal{M}$is the smallest$\sigma$-algebra containing the union of the unions of all sets of the form

$$f^{-m}\mathcal{P}\vee f^{-m+1}\mathcal{P}\vee \cdots \vee f^{m-1}\mathcal{P}\vee f^m\mathcal{P},$$

as m ranges over the natural numbers $\mathbb{N}.$ Then $h_{\mu }\left(f\right)=H\left(\mathcal{P},f\right).$

Now, it is not difficult to show (see, e.g., [24]) that the measurable partition

$$\mathcal{P}:=\left\{C_k:1\le k\le n\right\},$$

where $C_k:=$$\left\{\zeta \in X=S^{\mathbb{Z}}:{\zeta }_0=s_k\right\}$, satisfies the requirements of the Kolmogorov–Sinai theorem. This follows primarily from the fact that ${\mathfrak{s}}^{-m}\left(C_k\right)=\left\{\zeta \in X:{\zeta }_m=s_k\right\}$ for all $m\in \mathbb{N}$. Now the entropy of the partition $\mathcal{P}$ is

$$H\left(\mathcal{P}\right)=-{\displaystyle {\sum }_{k=1}^n}\mu \left(C_k\right)log\mu \left(C_k\right)=-{\displaystyle {\sum }_{k=1}^n}{p}_{k}log{p}_k.$$

Moreover, we find that

$$H\left({\mathcal{P}}^m\right)=-{\displaystyle {\sum }_{k_0,\dots ,k_{m-1}}^n}{\displaystyle {\prod }_{j=0}^{m-1}}{p}_{k_j}log\left({\displaystyle {\prod }_{j=0}^{m-1}}{p}_{k_j}\right)=-m{\displaystyle {\sum }_{k=1}^n}{p}_{k}log{p}_k.$$

Therefore, it follows from the two expressions above and the entropy theorem of Kolmogorov–Sinai that

$$h_{\mu }(\mathfrak{s})=H\left(\mathcal{P},\mathfrak{s}\right)=\underset{m\to \infty }{lim}\frac{H\left({\mathcal{P}}^m\right)}{m}=-{\displaystyle {\sum }_{k=1}^n}{p}_{k}log{p}_k,$$

which is the desired result.

A Simple Bernoulli Scheme Example

To illustrate the result just obtained, we shall verify it by direct calculation for a binary Bernoulli scheme. In particular, consider $X:=B(p_1,p_2)$, with $S:=\{0,1\}$, in which case it follows readily from the definitions above that the phase space of the dynamical system comprises all bi-infinite binary sequences; namely,

$$X=2^{\mathbb{Z}}=\left\{\zeta :\zeta :\mathbb{Z}\to \{0,1\}\right\}=\left\{\cdots a_{-2}{a}_{-1}{a}_0.a_1{a}_2{a}_3\cdots :a_k\in \{0,1\}\forall k\in \mathbb{Z}\right\},$$

and the map $\mathfrak{s}:X\to X$ is the shift map defined as $\mathfrak{s}(\zeta )(k):=\zeta (k+1)$ or equivalently

$$\mathfrak{s}\left(\cdots a_{-2}{a}_{-1}{a}_0.a_1{a}_2{a}_3\cdots \right):=\cdots a_{-2}{a}_{-1}{a}_0{a}_1.a_2{a}_3\cdots .$$

We know from the above that for the measurable partition

$$\mathcal{P}:=\left\{C_0:=C\left(\{0\},0\right),C_1:=C\left(\{0\},1\right)\right\},$$

where $C_k:=$$\left\{\zeta \in X={\{0,1\}}^{\mathbb{Z}}:{\zeta }_0=k\right\}$ for $k=0,1$, has the following entropy

$$H\left(\mathcal{P}\right)=-p_{1}log{p}_1-p_{2}log{p}_2.$$

Our intention is to show that this is equal to the entropy with respect to the shift map $\mathfrak{s}$, that is

$$h_{\mu }(\mathfrak{s})=H\left(\mathcal{P}\right).$$

Owing to the readily verified fact that every member of ${\bigvee }_{k=-m}^m{\mathfrak{s}}^{-k}\mathcal{P}$ is of the form $C\left({\mathbb{Z}}_{-m}^m,\phi \right)$ with $\phi :{\mathbb{Z}}_{-m}^m\to \{0,1\}$ and the Kolmogorov–Sinai entropy theorem (see e.g., [24]), it suffices to prove that every cylinder $C\left(F,\psi \right)$ satisfies

$$C\left(F,\psi \right)\subset {\displaystyle \bigcup }\left\{C\left({\mathbb{Z}}_{-m}^m,\phi \right):\phi :{\mathbb{Z}}_{-m}^m\to \{0,1\}\right\},$$

where ${\mathbb{Z}}_{-m}^m:=\{l\in \mathbb{Z}:\left|l\right|\le m,m\in \mathbb{N}\}$. This is obvious since F is finite, which proves the desired result.

5. Concluding Remarks

After a brief review of the main features and fundamental properties of topological, metric (Kolmogorov–Sinai) and Shannon entropy, we embarked on our effort to establish dynamical systems theory as their principal connective thread. We began with comparisons between the topological entropy, which may be considered the most general manifestation of the multifarious forms, and the metric entropy, perhaps the most real- world applicable of the embodiments of entropy. For example, we recounted the variational principle that the topological entropy of a discrete dynamical system is the supremum of the metric entropies over all possible corresponding discrete measurable dynamical dynamical systems, which naturally leads to the question of whether or not the supremum is actually assumed. In that vein, we proved our main theorem giving sufficient conditions for the equality of the topological and Komolgorov–Sinai entropies and provided several illustrative examples. Finally, we showed that Shannon’s information entropy is a special case of metric entropy, inasmuch the a dynamical information system can be identified with a Bernoulli scheme for which the Kolmogorov–Sinai entropy theorem provides a formula for the entropy identical to that of Shannon. In summary then, our project aimed at providing a rigorous underlying dynamical system theme for several of the more important entropy definitions.