Directed Data-Processing Inequalities for Systems with Feedback

Abstract

We present novel data-processing inequalities relating the mutual information and the directed information in systems with feedback. The internal deterministic blocks within such systems are restricted only to be causal mappings, but are allowed to be non-linear and time varying, and randomized by their own external random input, can yield any stochastic mapping. These randomized blocks can for example represent source encoders, decoders, or even communication channels. Moreover, the involved signals can be arbitrarily distributed. Our first main result relates mutual and directed information and can be interpreted as a law of conservation of information flow. Our second main result is a pair of data-processing inequalities (one the conditional version of the other) between nested pairs of random sequences entirely within the closed loop. Our third main result introduces and characterizes the notion of in-the-loop (ITL) transmission rate for channel coding scenarios in which the messages are internal to the loop. Interestingly, in this case the conventional notions of transmission rate associated with the entropy of the messages and of channel capacity based on maximizing the mutual information between the messages and the output turn out to be inadequate. Instead, as we show, the ITL transmission rate is the unique notion of rate for which a channel code attains zero error probability if and only if such an ITL rate does not exceed the corresponding directed information rate from messages to decoded messages. We apply our data-processing inequalities to show that the supremum of achievable (in the usual channel coding sense) ITL transmission rates is upper bounded by the supremum of the directed information rate across the communication channel. Moreover, we present an example in which this upper bound is attained. Finally, we further illustrate the applicability of our results by discussing how they make possible the generalization of two fundamental inequalities known in networked control literature.

1. Introduction

The data-processing inequality states that if $x,y,z$ are random variables such that x and z become independent when conditioning upon y, then:

$$\begin{array}{c}\hfill I(x;y)\ge I(x;z)\end{array}$$

(1)

$$\begin{array}{c}\hfill I(y;z)\ge I(x;z),\end{array}$$

(2)

where $I(x;y)$ denotes the mutual information between x and y [1] (p. 252) (a definition of mutual information is provided in Section 2.2 below). Among its many uses, the data-processing inequality plays a key role in the proof of the converse part (i.e., outer bounds) in rate-distortion [1,2,3,4,5] (p. 317), channel capacity [1,6,7,8] (pp. 208, 217, 540 and 566), and joint source-channel coding theorems [1,9,10,11] (p. 221), and has recently been extended to von Newmann algebras, which have applications in quantum field theory (see [12] and the references therein).

It is well known that mutual information has an important limitation in systems with feedback, such as the one shown in Figure 1a. In this system, p, q, r, s, e, u, x, and y are random sequences, and the blocks ${\mathcal{S}}_1,\dots ,{\mathcal{S}}_4$ are deterministic causal mappings with an added delay of at least one sample. These blocks, randomized by their exogenous random inputs p, q, r, s, may yield any causal stochastic dynamic mappings. As pointed out in [13], for sequences inside the loop, such as x and y, $I(x;y)$ does not distinguish the probabilistic interdependence produced by the effect x has on y from that stemming from the influence of y on x. This limitation motivated the introduction of the directed information in [13]. This notion assesses the amount of information that causally “flows” from a given random and ordered sequence to another. For this reason, it has increasingly found use in diverse applications, including characterizing the capacity of channels with feedback [13,14,15,16], the rate distortion function under causality constraints [5], establishing some of the fundamental limitations in networked control [17,18,19,20,21,22,23], determining causal relationships in neural networks [24], and portfolio theory and hypothesis testing [25], to name a few.

The directed information from a randomsequence $x^k$ to a random sequence $y^k$ is defined as:

$$\begin{array}{c}\hfill I(x^k\to y^k)\triangleq \sum _{i=1}^{k}I(y\left(i\right);x^i|y^{i-1}),\end{array}$$

(3)

where the notation $x^i$ represents the sequence $x\left(1\right),x\left(2\right),\dots ,x\left(i\right)$ and $I(x;y|z)$ is the mutual information between x and y conditioned on (or given) z (hereafter we use non-italic letters, such as x, for random variables, denoting a particular realization by the corresponding italic character, x). The causality inherent in this definition becomes evident when comparing it with the mutual information between $x^k$ and $y^k$, given by $I(x^k;y^k)={\sum }_{i=1}^{k}I(y\left(i\right);x^k|y^{i-1})$. In the latter sum, what matters is the amount of information about the entire sequence $x^k$ present in $y\left(i\right)$, given the past values $y^{i-1}$. By contrast, in the conditional mutual information in the sum of (3), only the past and current values of $x^k$ are considered, that is, $x^i$. Thus, $I(x^k\to y^k)$ represents the amount of information causally conveyed from $x^k$ to $y^k$. A related notion is the causally conditioned directed information introduced in [14], defined as:

$$\begin{array}{c}\hfill I(x^k\to y^k\Vert q^k)\triangleq \sum _{i=1}^{k}I(y\left(i\right);x^i|y^{i-1},q^i).\end{array}$$

(4)

In this paper, we derive inequalities involving directed and mutual information within feedback systems. For this purpose, we consider the general feedback system shown in Figure 1a. In this diagram, the blocks ${\mathcal{S}}_1,\dots ,{\mathcal{S}}_4$ represent possibly non-linear and time-varying causal discrete-time systems such that the total delay of the loop is at least one sample. These blocks can model, for example, source encoders, decoders or even communication channels. In the same figure, $r,p,s,q$ are exogenous random signals (scalars, vectors, or sequences), which could represent, for example, any combination of disturbances, noises, random initial states, or side information. We note that any of these exogenous signals, in combination with their corresponding deterministic mapping ${\mathcal{S}}_i$, can also yield any desired stochastic causal mapping (for example, a noisy communication channel, a zero-delay source coder or decoder, or a causal dynamic system with disturbances and a random initial state).

1.1. Main Contributions

• Our first main contribution is the following theorem. It states a fundamental result, which relates the directed information between two signals within a feedback loop, say x and y, to the mutual information between an external set of signals and y:

  Theorem 1.

  In the system shown in Figure 1a, it holds that:

  $$\begin{array}{c}\hfill I(x^k\to y^k)=I(q^k,r^k,p^k\to y^k)-I(q^k,r^k,p^k\to y^k\Vert x^k)\le I(p^k,q^k,r^k;y^k),\forall k\in \mathbb{N},\end{array}$$

  (5)

  with equality achieved if s is independent of $(p,q,r)$.
  The proof is in Section 3. This fundamental result, which for the cases in which $s⫫(p,q,r)$ can be understood as a law of conservation of information flow, is illustrated in Figure 2. (Here, and in the sequel, we use the notation $x⫫y$ to mean “x is independent of y”.) For such a cases, the information causally conveyed from x to y equals the information flow from $(q,r,p)$ to y. When $(p,q,r)$ are not independent of s, part of the mutual information between $(p,q,r)$ and y (corresponding to the term $I(q^k,r^k,p^k\to y^k\Vert x^k)$) can be thought of as being “leaked” through s, thus bypassing the forward link from x to y. This provides an intuitive interpretation for (5).

  Figure 2. The flow of information between exogenous signals $(p,q,r)$ and the internal signal y equals the directed information from $x^k$ to $y^k$ when $s⫫(p,q,r)$.

  Figure 2. The flow of information between exogenous signals $(p,q,r)$ and the internal signal y equals the directed information from $x^k$ to $y^k$ when $s⫫(p,q,r)$.

  Remark 1.Theorem 1 implies that $I(x^k\to y^k)$ is only a part of (or at most equal to) the information “flow” between all the exogenous signals entering the loop outside the link $x\to y$ (namely $(q,r,p)$), and y. In particular, if $(p,q,r)$ were deterministic, then $I(x^k\to y^k)=0$, regardless of the blocks ${\mathcal{S}}_1,\dots ,{\mathcal{S}}_4$ and irrespective of the nature of s.
• Our second main result is the following theorem, which relates directed information involving four different sequences internal to the loop. The proof is in Appendix A on page 19.

  Theorem 2

  (Full Closed-Loop Directed Data-Processing Inequality). Consider the system shown in Figure 1a.

  (a) 

  If $(q,s)⫫(r,p)$ and $q⫫s$, or if $(p,s)⫫(r,q)$ and $p⫫s$, then:

  $$\begin{array}{c}\hfill I(x^k\to y^k)\ge I(e^k\to u^k).\end{array}$$

  (6)

  (b) 

  If $(q,s)⫫(r,p)$ and $q_{i+1}^k\leftrightarrow q^i\leftrightarrow s^i$ for $i=1,2,\dots ,k-1$, then:

  $$\begin{array}{c}\hfill I(x^k\to y^k\parallel q^k)\ge I(e^k\to u^k).\end{array}$$

  (7)

  (The Markov chain notation $t\leftrightarrow v\leftrightarrow w$ means “t and w are independent when v is given”.) To the best of our knowledge, Theorem 2 is the first result available in the literature providing a lower bound to the gap between two instances of nested directed information, involving four different signals inside the feedback loop. This result can be seen as the first full extension of the open-loop (traditional) data-processing inequality, to arbitrary closed-loop scenarios. (Notice that there is no need to consider systems with more than four mappings, since all external signals entering the loop between a given pair of internal signals can be regarded as exogenous inputs to a single equivalent deterministic mapping.)
• Our third main contribution is introducing the notion of in-the-loop (ITL) transmission rate (in Section 6) for the (seldom considered) channel-coding scenario in which the messages to be transmitted and the communication channel are internal to a feedback loop. We show that the supremum of the directed information rate across such a channel upper bounds the achievable ITL transmission rates. Moreover, we present an example in which this upper bound is attainable. This gives further operational meaning to the directed information rate in closed-loop scenarios.
• Finally, we provide additional examples of the applicability of our results by discussing how they allow one to obtain the generalizations of two fundamental inequalities known in networked control literature. The first one appears in [18] (Lemma 4.1) and is written in (12) below. This generalization is a consequence of Theorem 4 and is discussed in Remarks 3 and 5 below. The second generalization applies to [20] (Theorem 4.1) and is described on page 6 below. It is an application of Theorem 2 that has just been carried out by the authors in [26], which is all the more important since, as we also reveal in that note, there is a flaw in the proof of [20] (Theorem 4.1).

A key ingredient in proving most of our theorems is provided by Lemma 1, stated in Section 2.5. It allows one to rigorously establish some of the non-trivial conditional independencies that arise in a feedback loop with several (possibly stochastic) dynamic systems.

Put together, the law of conservation of information flow from Theorem 1, our extension of the data processing inequality to general feedback systems from Theorem 2, and our other results constitute both a conceptual framework and a toolbox for addressing information flow problems in feedback systems. We are convinced that this contribution will be instrumental in establishing new results on, e.g., rate-distortion and channel capacity problems with feedback.

The literature review presented next will allow the reader to further assess the novelty and relevance of our results.

1.2. Existing Related Results

There exist several results characterizing the relationship between $I(x^k\to y^k)$ and $I(x^k;y^k)$. First, it is well known that $I(x^k\to y^k)\le I(x^k;y^k)$, with equality if and only if $y^k$ is causally related to $x^k$ [13]. A conservation law of mutual and directed information has been found in [27], which asserts that $I(x^k\to y^k)+I(0\ast y^{k-1}\to x^k)=I(x^k;y^k)$, where $0\ast y^{k-1}$ denotes the concatenation $0,y\left(1\right),\dots ,y^{k-1}$.

Given its prominence in settings involving feedback, it is perhaps in these scenarios where the directed information becomes most important. For instance, the directed information has been instrumental in characterizing the capacity of channels with feedback (see, e.g., [15,16,28] and the references therein), as well as the rate-distortion function in setups involving feedback [5,20,21,22,29].

For the simple case in which all the systems ${\left\{{\mathcal{S}}_i\right\}}_{i=1}^4$ are linear time invariant (LTI) and stable, and assuming $p,x,q=0$ (deterministically), it was shown in [30] that $I(r^k\to e^k)$ does not depend on whether there is feedback from e to u or not.

Inequalities between mutual and directed information in a less restricted setup, shown in Figure 1b, have been found in [18,19]. In that setting (a networked-control system), G is a strictly causal LTI dynamic system having (vector) state sequence ${\left\{\mathbf{x}\left(i\right)\right\}}_{i=0}^{\infty }$, with $p\triangleq \mathbf{x}\left(0\right)$ being the random initial state in its state-space representation. The external signal r (which could correspond to a disturbance) is statistically independent of s, the latter corresponding to, for example, side information or channel noise. Both are also statistically independent of p.

The blocks labeled E, D, and f correspond to an encoder, a decoder, and a channel, respectively, all of which are causal. The channel f maps $s^k$ and $x^k$ to $y\left(k\right)$ in a possibly time-varying manner, i.e., $y\left(k\right)=f(k,x^k,s^k).$ Similarly, the concatenation of the encoder, the channel and the decoder, maps $s^k$ and $w^k$ to $u\left(k\right)$ as a possibly time-dependent function $u\left(k\right)=\psi (k,w^k,s^k).$ Under these assumptions, the following fundamental result was shown in [19] (Lemma 5.1):

$$\begin{array}{c}\hfill I(r^k,p;u^k)\ge I(r^k;u^k)+I(p;e^k).\end{array}$$

(8)

By further assuming in [19] that the decoder D in Figure 1b is deterministic, the following Markov chain naturally holds,

$$\begin{array}{c}\hfill (p,r^k)⟷y^k⟷u^k,\end{array}$$

(9)

leading directly to:

$$\begin{array}{c}\hfill I(r^k,p;y^k)\ge I(r^k;u^k)+I(p;e^k),\end{array}$$

(10)

which is found in the proof of [19] (Corollary 5.3). The deterministic nature of the decoder D played a crucial role in the proof of this result, since otherwise the Markov chain (9) does not hold, in general, due to the feedback from u to y.

Notice that both (8) and (10) provide lower bounds to mutual information as the sum of two mutual information terms, each of them relating a signal external to the loop (such as $p,r^k$) to a signal internal to the loop (such as $u^k$ or $y^k$). Instead, the inequality:

$$\begin{array}{c}\hfill I(x^k\to y^k)\ge I(r^k;y^k),\end{array}$$

(11)

which holds for the system in Figure 1a and appears in [13] (Theorem 3) (and rediscovered later in [17] (Lemma 4.8.1)), involves the directed information between two internal signals and the mutual information between the second of these and an external sequence.

Remark 2.

By using (22), $I(p^k,q^k,r^k;y^k)=I(r^k;y^k)+I(p^k,q^k;y^k|r^k)$. Then, applying Theorem 1, we recover (11), whenever $s⫫(q,r,p)$. Thus, [13,17] (Theorem 3) and (Lemma 4.8.1)) can be obtained as a corollary of Theorem 1.

A related bound, similar to (10) but involving information rates and with the leftmost mutual information replaced by the directed information from $x^k$ to $y^k$ (which are two signals internal to the loop), has been obtained in [18] (Lemma 4.1) for the networked control system of Figure 1b:

$$\begin{array}{c}\hfill \overline{I}(x\to y)\ge \overline{I}(r;u)+\underset{k\to \infty }{lim}\frac{I(p;e^k)}{k},\end{array}$$

(12)

with $\overline{I}(x\to y)\triangleq {lim}_{k\to \infty }\frac{1}{k}I(x^k\to y^k)$ and $\overline{I}(r;u)\triangleq {lim}_{k\to \infty }\frac{1}{k}I(r^k;u^k)$, provided ${sup}_{i\ge 0}E\left[\mathbf{x}{\left(i\right)}^T\mathbf{x}\left(i\right)\right]<\infty$. This result relies on three assumptions: (a) that the channel f is memory-less and satisfies a “conditional invertibility” property, (b) a finite-memory condition, and (c) a fading-memory condition, the latter two related to the decoder D (see Figure 1).

It is worth noting that, as defined in [18], these assumptions upon D exclude the use of side information by the decoder and/or the possibility of D being affected by random noise or having a random internal state that is non-observable (please see [18] for a detailed description of these assumptions).

Remark 3.

In Section 4 we present Theorem 4, which yields (12) as a special case, but for the general system of Figure 1a and with no other assumption than mutual independence between $r,p,q,s$. Moreover, since with this independence condition Theorem 1 yields $I(x^k\to u^k)=I(r^k,p^k;u^k)$, the same happens with (8).

The inequality (11) has been extended in [16] (Theorem 1), for the case of discrete-valued random variables and assuming $s⫫(r,p,q)$, as the following identity (written in terms of the signals and setup shown in Figure 1a):

$$\begin{array}{c}\hfill I(x^k\to y^k)=I(p^k,y^k)+I(x^k\to y^k|p^k).\end{array}$$

(13)

Letting $q=s$ in Figure 1a and with the additional assumption that $(p,s)⫫q$, it was also shown in [16] (Theorem 1) that:

$$\begin{array}{c}\hfill I(x^k\to y^k)=I(p^k;y^k)+I(q^{k-1};y^k)+I(p^k;q^{k-1}|y^k),\end{array}$$

(14)

for the cases in which $u\left(i\right)=y\left(i\right)+q\left(i\right)$ (i.e., when the concatenation of ${\mathcal{S}}_4$ and ${\mathcal{S}}_1$ corresponds to a summing node). In [16], (13) and (14) play important roles in characterizing the capacity of channels with noisy feedback.

To the best of our knowledge, (8), (10), (11)–(14) are the only results available in the literature that lower bound the difference between internal-to-internal directed information and external-to-internal mutual information. There exist even fewer published results in relation to inequalities between two directed information terms involving only signals internal to the loop. To the best of our knowledge, the only inequality of this type in the literature is the one found in the proof of Theorem 4.1 of [20]. The latter takes the form of a (conditional) data-processing inequality for directed information in closed-loop systems, and states that:

$$\begin{array}{c}\hfill I(x^k\to y^k\Vert q^k)\ge I(x^k\to u^k),\end{array}$$

(15)

provided: $q⫫(r,p)$ and if ${\mathcal{S}}_4$ is such that $y^i$ is a function of $(u^i,q^i)$ (i.e., if ${\mathcal{S}}_4$ is conditionally invertible) $\forall i$.

Inequality (15) plays a crucial role in [20], since it allows [20] (Thm. 4.1) to lower bound the average data rate across a digital error-free channel by a directed information. The setup considered in that theorem is shown in Figure 3, where $\mathcal{F}$ is a plant, and $\mathcal{E}$, $\mathcal{D}$ are the (source) encoder and decoder, respectively. In this figure, the variables that have been adapted to match those in Figure 4a ($r,p,x$ correspond to disturbance, initial state, and plant output, respectively). Assuming $(r,p)⫫(q,s)$ and a conditionally invertible decoder, and letting $R\left(i\right)$ be the expected length (in bits) necessary for a binary representation of $y\left(i\right)$ given $q^i$, it states that $\frac{1}{k}{\sum }_{i=1}^{k}R\left(i\right)\ge \frac{1}{k}I(x^k\to u^k),k=1,2,\dots .$ This is a key result, because, combined with [20] (Equation (9)), it yields:

$$\begin{array}{c}\hfill \frac{1}{k}I(x^k\to u^k)\le \frac{1}{k}\sum _{i=1}^{k}R\left(i\right)\le \frac{1}{k}I(x^k\to u^k)+1[\mathrm{bits}/\mathrm{sample}],k=1,2,\dots .\end{array}$$

(16)

This result highlights the operational meaning of the directed information as a lower bound (tight to within one bit) to the data rate of any given source code in a closed-loop system. This fact has been a crucial ingredient in characterizing the best rate performance achievable in Gaussian linear quadratic networked control [23,31], demonstrating the relevance of directed data-processing inequalities.

Figure 3. The networked control system considered in [20] (Figure 2), slightly simplified. The variables $r,p,x,y$ correspond to $d,x_o,y,s$ in [20], respectively.

Figure 3. The networked control system considered in [20] (Figure 2), slightly simplified. The variables $r,p,x,y$ correspond to $d,x_o,y,s$ in [20], respectively.

Unfortunately, as we will reveal in [26], the proof of [20] (Theorem 4.1) turns out to be invalid, since it relies upon [20] (Lemma 4.2), whose first claim does not hold. In [26] we use Theorem 2 to prove Theorem 4.1 of [20] without requiring a conditionally invertible decoder. This further illustrates the applicability of our results.

In [23] (Lemma 1) another data-processing inequality is stated, which for the system in Figure 1a is equivalent to:

$$\begin{array}{c}\hfill I(x^k\to y^k\parallel u_{+}^{k-1})\ge I(x^k\to u^k),k=1,2,\dots \end{array}$$

(17)

where: $I(x^k\to y^k\parallel u_{+}^{k-1})\triangleq {\sum }_{i=1}^{k}I(x^i;y\left(i\right)|y^{i-1},u^{i-1})$. However, in [23] the blocks ${\mathcal{S}}_3,{\mathcal{S}}_4$ are defined implicitly, writing instead their input–output relation as collections of stochastic kernels $\mathbb{P}(y\left(i\right)|x^i,y^{i-1})$, $\mathbb{P}(u\left(i\right)|y^i,u^{i-1})$, $i=1,2,\dots$. The notation $\mathbb{P}(t|v,w)$ is to be understood as the conditional distribution of t given $(v,w)$. Crucially, this entails the implicit assumption that given $x^i$ and $y^{i-1}$, $y\left(i\right)$ is independent of every other signal in the system (and likewise for $\mathbb{P}(u\left(i\right)|y^i,u^{i-1})$). In the representation of Figure 1a, this corresponds to assuming $q⫫s$ and $(q,s)⫫(r,p)$.

Remark 4.

The conditioning on the side information q in both Theorem 2 and [20] (Theorem 4.1) is motivated by the use of entropy coded subtractively dithered quantization (ECSDQ) in obtaining the upper bound in (16). For such a scenario, the sequences q and s are identical and correspond to the dither signal, which is independent of $r,p$. This satisfies the requirements of (6) in Theorem 2 and of [20] (Theorem 4.1), but not the assumption that $q⫫s$ and $(q,s)⫫(r,p)$ implicit in [23] (Lemma 1), which yields (17). In spite of this, Lemma 1 of [23] is used in that paper to prove the lower bound in [23] (Equation (8)), an analogue of (16) which also considers the use of ECSDQ for the rate term and its upper bound.

1.3. Outline of the Paper

The remainder of the paper continues with some preliminary definitions and results in Section 2, the last of which is Lemma 2, a crucial tool for proving most of our theorems. Then follows the proof of Theorem 1 in Section 3. Section 4 presents additional inequalities relating mutual information between external–internal signal pairs, and directed information from one internal signal to another internal signal. These results can be seen as extensions or consequences of Theorem 1. Then we develop in Section 5 inequalities between two nested directed information expressions. Such results are the precursors of Theorem 2 and, as such, play a key role in its proof (which opens Appendix A). The notions and results associated with in-the-loop channel coding are developed in Section 6. The main conclusions of this work are presented in Section 7. Appendix A provides the proofs that are not written right after their corresponding theorems.

An earlier version of this work was made publicly available on arxiv.org [32] and, as such, it was cited in [23,31,33,34,35].

2. Preliminaries

2.1. Notation

The set of natural numbers is denoted $\mathbb{N}$. Random variables are denoted using non-italic characters, such as x. We write $x^i$ to represent the sequence $x\left(1\right),x\left(2\right),\dots ,x\left(i\right)$. We write $x⫫y$ to express that x and y are independent. We use $Pr\left\{“\mathrm{outcome}”\right\}$ to denote the probability of a specific outcome of one or more random variables. For example, $Pr\{x=x,y\in \mathcal{Y}\}$ is the probability that $x=1$ and y is in a given set $\mathcal{Y}$. Likewise, $Pr\left\{“\mathrm{outcome}”\right|“\mathrm{outcome}2”\}$ is the conditional probability of “outcome 1” given “outcome 2”. The Markov-chain notation $x\leftrightarrow y\leftrightarrow z$ means $Pr\{x\in \mathcal{X},z\in \mathcal{Z}|y\in \mathcal{Y}\}=Pr\{x\in \mathcal{X}|y\in \mathcal{Y}\}Pr\{z\in \mathcal{Z}|y\in \mathcal{Y}\}$, for every choice of the sets $\mathcal{X},\mathcal{Y},\mathcal{Z}$ in the event spaces of $x,y$, and z, respectively. For two probability measures $\mu ,\nu$ on a common event space $\mathcal{U}$ the notation $\mu \ll \nu$ means that $\mu$ is absolutely continuous with respect to $\nu$, i.e., that $\forall \mathcal{U}\in \mathcal{U}:\nu \left(\mathcal{U}\right)=0\Rightarrow \mu \left(\mathcal{U}\right)=0$.

2.2. Mutual Information

Let $(\Omega ,\mathcal{F},P)$ be a probability space, and $(\mathcal{X},{\mathcal{F}}_{\mathcal{X}})$ and $(\mathcal{Y},{\mathcal{F}}_{\mathcal{Y}})$ be measurable spaces, and consider the random variables $x:\Omega \to \mathcal{X}$, $y:\Omega \to \mathcal{Y}$. Define $\mathcal{M}\triangleq {\mathcal{F}}_{\mathcal{X}}\otimes {\mathcal{F}}_{\mathcal{Y}}$, i.e, the $\sigma$-algebra generated by the rectangles $\{A\times B:A\in \mathcal{X},B\in \mathcal{Y}\}$. Consider a probability space $(\mathcal{X}\times \mathcal{Y},\mathcal{M},m)$ where m is the (joint) distribution of $(x,y)$, i.e., $m=P\circ {(x,y)}^{-1}$.

Denote the marginal probability distributions of x and y by $\mu$ and $\nu$, respectively, where:

$$\begin{array}{cc}\hfill \mu \left(A\right)& =m(A\times \mathcal{Y}),A\in {\mathcal{F}}_{\mathcal{X}}\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill \nu \left(B\right)& =m(\mathcal{X}\times B),B\in {\mathcal{F}}_{\mathcal{Y}}\hfill \end{array}$$

(19)

Define the product measure $\pi \triangleq \mu \times \nu$ on $(\mathcal{X}\times \mathcal{Y},\mathcal{M})$.

Definition 1.

With the above definitions, the mutual information between x and y is defined as:

$$\begin{array}{c}\hfill I(x;y)\triangleq \int log\left(\frac{dm}{d\pi }\right)dm,\end{array}$$

(20)

where $\frac{dm}{d\pi }$ is the Radon–Nikodym derivative of m with respect to π [36].

An ensemble of random variables has a standard alphabet if it takes values from a set $\mathcal{A}$ in a standard measurable space $(\mathcal{A},{\mathcal{F}}_{\mathcal{A}})$ [37] (Section 1.4) and its probability measure is defined on ${\mathcal{F}}_{\mathcal{A}}$. For our purposes, it suffices to say that standard alphabets include discrete spaces, the real line, Euclidean vector spaces, and Polish spaces (i.e., complete separable metric spaces) [38].

Lemma 1

(Chain Rule of Mutual Information from [37] (Corollary 7.14)). Suppose $x,y,z$ are random variables with standard alphabets and with joint distribution $P_{xyz}$. Suppose also that there exists a product distribution $M_{xyz}=M_x\times M_{yz}$ such that $M_{xyz}\gg P_{xyz}$. (This is true, for example, if $P_x\times P_{yz}\gg P_{xyz}$.) Then:

$$\begin{array}{c}\hfill I(x;y,z)=I(x;y)+I(x;z|y).\end{array}$$

(21)

From [37] (Lemma 7.4 and Equation 7.28), we have that $I(x;y,z)<\infty \Rightarrow P_x\times P_{yz}\gg P_{xyz}$, and thus (21) also holds if $I(x;y,z)$ is finite.

The conditional version of the chain rule of mutual information [39] (see also [37] (Corollary 2.5.1)) will be extensively utilized in the proofs of our results:

$$\begin{array}{c}\hfill I(t,v;w|z)=I(v;w|z)+I(t;w|v,z).\end{array}$$

(22)

For discrete random variables $x,y$, taking values from the sets $\mathcal{X},\mathcal{Y}$, respectively, the entropy of x is defined as:

$$\begin{array}{c}\hfill H(x)\triangleq -\sum _{x\in \mathcal{X}}Pr\{x=x\}log\left(Pr\{x=x\}\right)\end{array}$$

(23)

and the conditional entropy of x given y is defined as:

$$\begin{array}{c}\hfill H(x|y)\triangleq -\sum _{x\in \mathcal{X},y\in \mathcal{Y}}Pr\{x=x,y=y\}log(Pr\{x=x|y=y\})\end{array}$$

(24)

The entropy satisfies the chain rule:

$$\begin{array}{c}\hfill H(x,y)=H(x)+H(y|x)=H(y)+H(x|y)\end{array}$$

(25)

and is related to the mutual information as:

$$\begin{array}{c}\hfill I(x;y)=H(x)+H(y)-H(x,y)=H(x)-H(x|y)=H(y)-H(y|x).\end{array}$$

(26)

2.3. System Description

We begin by providing a formal description of the systems labeled ${\mathcal{S}}_1\dots {\mathcal{S}}_4$ in Figure 1a. Their input–output relationships are given by the possibly-varying deterministic mappings (For notational simplicity, we omit writing their time dependency explicitly):

$$\begin{array}{cc}\hfill e\left(i\right)& ={\mathcal{S}}_1(u^{i-d_1\left(i\right)},r^i),\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill x\left(i\right)& ={\mathcal{S}}_2(e^{i-d_2\left(i\right)},p^i),\hfill \end{array}$$

(27b)

$$\begin{array}{cc}\hfill y\left(i\right)& ={\mathcal{S}}_3(x^{i-d_3\left(i\right)},s^i),\hfill \end{array}$$

(27c)

$$\begin{array}{cc}\hfill u\left(i\right)& ={\mathcal{S}}_4(y^{i-d_4\left(i\right)},q^i),\hfill \end{array}$$

(27d)

where $r,p,s,q$ are exogenous random signals and the (possibly time-varying) delays $d_1,d_2,d_3,d_4\in \{0,1,\dots \}$ are such that:

$$d_1\left(k\right)+d_2\left(k\right)+d_3\left(k\right)+d_4\left(k\right)\ge 1,\forall k\in \mathbb{N}.$$

That is, the concatenation of ${\mathcal{S}}_1,\dots ,{\mathcal{S}}_4$ has a delay of at least one sample. For every $i\in \{1,\dots ,k\}$, $r\left(i\right)\in {\mathbb{R}}^{n_r\left(i\right)}$, i.e., $r\left(i\right)$ is a real random vector whose dimension is given by some function $n_r:\{1,\dots ,k\}\to \mathbb{N}$. The other sequences ($q,p,s,x,y,u$) are defined likewise.

2.4. A Necessary Modification of the Definition of Directed Information

As stated in [13], the directed information (as defined in (3)) is a more meaningful measure of the flow of information between $x^k$ and $y^k$ than the conventional mutual information $I(x^k;y^k)={\sum }_{i=1}^{k}I(y\left(i\right);x^k|y^{i-1})$ when there exists causal feedback from y to x. In particular, if $x^k$ and $y^k$ are discrete-valued sequences, the input and output, respectively, of a forward channel, and if there exists strictly causal perfect feedback, so that $x\left(i\right)=y(i-1)$ (a scenario utilized in [13] as part of an argument in favor of the directed information), then the mutual information becomes:

$$\begin{array}{cc}\hfill I(x^k;y^k)& =H\left(y^k\right)-H\left(y^k\right|x^k)=H\left(y^k\right)-H\left(y^k\right|y^{k-1})=H\left(y^k\right)-H(y\left(k\right)|y^{k-1})\hfill \\ & =H\left(y^{k-1}\right).\hfill \end{array}$$

Thus, when strictly causal feedback is present, $I(x^k;y^k)$ fails to account for how much information about $x^k$ has been conveyed to $y^k$ through the forward channel that lies between them.

It is important to note that in [13] (as well as in many works concerned with communications), the forward channel is instantaneous, i.e., it has no delay. Therefore, if a feedback channel is utilized, then this feedback channel must have a delay of at least one sample, as in the example above. However, when studying the system in Figure 1a, we may need to evaluate the directed information between signals $x^k$ and $y^k$ which are, respectively, the input and output of a strictly casual forward channel (i.e., with a delay of at least one sample), whose output is instantaneously fed back to its input. In such a case, if one further assumes perfect feedback and sets $x\left(i\right)=y\left(i\right)$, then, in the same spirit as before,

$$\begin{array}{cc}\hfill I(x^k\to y^k)& =\sum _{i=1}^{k}I(y\left(i\right);x^i|y^{i-1})=\sum _{i=1}^k\left[H(y\left(i\right)|y^{i-1})-H(y\left(i\right)|x^i,y^{i-1})\right]=H\left(y^k\right).\hfill \end{array}$$

As one can see, Massey’s definition of directed information ceases to be meaningful if instantaneous feedback is utilized.

It is natural to solve this problem by recalling that, in the latter example, the forward channel had a delay, say d, greater than one sample. Therefore, if we are interested in measuring how much of the information in $y\left(i\right)$, not present in $y^{i-1}$, was conveyed from $x^i$ through the forward channel, we should look at the mutual information $I(y\left(i\right);x^{i-d}|y^{i-1})$, because only the input samples $x^{i-d}$ can have an influence on $y\left(i\right)$. For this reason, we introduce the following, modified notion of directed information.

Definition 2

(Directed Information with Forward Delay). In this paper, the directed information from $x^k$ to $y^k$ through a forward channel with a non-negative time varying delay of $d_{xy}\left(i\right)$ samples is defined as:

$$\begin{array}{c}\hfill I(x^k\to y^k)\triangleq \sum _{i=1}^{k}I(y\left(i\right);x^{i-d_{xy}\left(i\right)}|y^{i-1}).\end{array}$$

(28)

For a zero-delay forward channel, the latter definition coincides with Massey’s [13].

Likewise, we adapt the definition of causally-conditioned directed information to the definition:

$$\begin{array}{c}\hfill I(x^k\to y^k\Vert e^k)\triangleq \sum _{i=1}^{k}I(y\left(i\right);x^{i-d_{xy}\left(i\right)}|y^{i-1},e^i).\end{array}$$

where, as before, $d_{xy}\left(i\right)$ is the delay from x to $y\left(i\right)$.

2.5. A Fundamental Lemma

The following result is an essential ingredient in the proof of most of our theorems:

Lemma 2.

In the system shown in Figure 4, the exogenous signals $r,q$ are mutually independent and ${\mathcal{S}}_1,{\mathcal{S}}_2$ are deterministic (possibly time-varying) causal measurable functions characterized by $y^i={\mathcal{S}}_1(r^i,u^i)$, $u^i={\mathcal{S}}_2(q^i,y^{i-1})$, $\forall i\in \{1,\dots \}$, with $y_0=y_0$ (deterministic). For this system, and for every $0\le j\le i\le k$ such that $i-j\le 1$ and $i\ge 1$, the following Markov chain holds:

$$\begin{array}{c}\hfill r^k⟷(u^i,y^j)⟷q^k,\forall k\in \mathbb{N}.\end{array}$$

(29)

Figure 4. Two arbitrary causal systems ${\mathcal{S}}_1,{\mathcal{S}}_2$ interconnected in a feedback loop. The exogenous signals $r,q$ are mutually independent.

Figure 4. Two arbitrary causal systems ${\mathcal{S}}_1,{\mathcal{S}}_2$ interconnected in a feedback loop. The exogenous signals $r,q$ are mutually independent.

Proof. 

Let $\mathcal{R},\mathcal{Q},\mathcal{U},\mathcal{Y}$ be the event spaces of $r^k,q^k,u^i,y^j$, respectively. Since $y^j={\mathcal{S}}_1(r^j,u^j)$ and $u^i={\mathcal{S}}_2(q^i,y^{i-1})$ are deterministic measurable functions, it follows that for every possible pair of events $\mathcal{U}\in \mathcal{U}$, $\mathcal{Y}\in \mathcal{Y}$, the preimage sets ${\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\triangleq \{r^k:{\mathcal{S}}_1(r^j,u^j)\in \mathcal{Y},u^i\in \mathcal{U}\}$ and ${\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\triangleq \{q^k:{\mathcal{S}}_2(q^i,y^{i-1})\in \mathcal{U},y^j\in \mathcal{Y}\}$ are also deterministic and belong to $\mathcal{R}$ and $\mathcal{Q}$, respectively. Thus, $(u^i,y^j)\in \mathcal{U}\times \mathcal{Y}\iff (r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}})$. This means that for every pair of events $R\in \mathcal{R},Q\in \mathcal{Q}$,

$$\begin{array}{cc}\hfill Pr\{r^k\in R& ,q^k\in Q|y^j\in \mathcal{Y},u^i\in \mathcal{U}\}\hfill \\ & \stackrel{\left(a\right)}{=}Pr\{r^k\in R,q^k\in Q|r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}\hfill \\ & \stackrel{\left(b\right)}{=}\frac{Pr\{r^k\in R\cap {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in Q\cap {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}\hfill \\ & \stackrel{\left(c\right)}{=}\frac{Pr\{r^k\in R\cap {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}·\frac{Pr\{q^k\in Q\cap {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}\hfill \\ & \stackrel{\phantom{\left(d\right)}}{=}\frac{Pr\{r^k\in R\cap {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}Pr\{q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}Pr\{q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}·\frac{Pr\{q^k\in Q\cap {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}\hfill \\ & \stackrel{\left(d\right)}{=}\frac{Pr\{r^k\in R\cap {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}}·\frac{Pr\{q^k\in Q\cap {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}},r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}{Pr\{q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}},r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}}\hfill \\ & \stackrel{\left(e\right)}{=}Pr\{r^k\in R|r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}}\}·Pr\{q^k\in Q|q^k{\mathcal{Q}}_{\mathcal{U},\mathcal{Y}},r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}}\}\hfill \\ & \stackrel{\left(f\right)}{=}Pr\{r^k\in R|y^j\in \mathcal{Y},u^i\in \mathcal{U}\}·Pr\{q^k\in Q|y^j\in \mathcal{Y},u^i\in \mathcal{U}\}\hfill \end{array}$$

where $\left(a\right)$ and $\left(f\right)$ follow because of the equivalence between the events $(y^j\in \mathcal{Y},u^i\in \mathcal{U})$ and $(r^k\in {\mathcal{R}}_{\mathcal{U},\mathcal{Y}},q^k\in {\mathcal{Q}}_{\mathcal{U},\mathcal{Y}})$, $\left(b\right)$ and $\left(e\right)$ follow from Bayes rule, and $\left(c\right)$ and $\left(d\right)$ are true because $r^k⫫q^k$. This completes the proof. □

3. Proof of Theorem 1

It is clear from Figure 1a and from (27) that the relationship between r, p, q, s, x, and y can be represented by the diagram shown in Figure 5.

From this diagram and Lemma 2 it follows that if s is independent of $(r,p,q)$, then the following Markov chain holds:

$$\begin{array}{cc}\hfill y\left(i\right)& ⟷(x^{i-d_3\left(i\right)},y^{i-1})⟷(p^i,q^i,r^i).\hfill \end{array}$$

(30)

Denoting the triad of exogenous signals $p^k,q^k,r^k$ by:

$$\begin{array}{c}\hfill {\theta }^k\triangleq (p^k,q^k,r^k),\end{array}$$

(31)

we have the following:

$$\begin{array}{cc}\hfill I(x^k\to y^i)& =\sum _{i=1}^{k}I(y\left(i\right);x^{i-d_3\left(i\right)}|y^{i-1})\hfill \\ & \stackrel{\left(22\right)}{=}\sum _{i=1}^k\left[I({\theta }^i,x^{i-d_3\left(i\right)};y\left(i\right)|y^{i-1})-I({\theta }^i;y\left(i\right)|x^{i-d_3\left(i\right)},y^{i-1})\right]\hfill \end{array}$$

$$\stackrel{\left(a\right)}{=}\sum _{i=1}^k\left[I({\theta }^i,y\left(i\right)|y^{i-1})-I({\theta }^i;y\left(i\right)|x^{i-d_3\left(i\right)},y^{i-1})\right]$$

(32a)

$$\stackrel{\left(b\right)}{\le }\sum _{i=1}^{k}I({\theta }^i,y\left(i\right)|y^{i-1})\stackrel{\left(c\right)}{\le }\sum _{i=1}^{k}I({\theta }^k,y\left(i\right)|y^{i-1})$$

(32b)

$$=I({\theta }^k;y^k)$$

(32c)

In the above, $\left(a\right)$ follows from the fact that, if $y^{i-1}$ is known, and then $x^{i-d_3\left(i\right)}$ is a deterministic function of ${\theta }^i$. The resulting sums on the right-hand side of (32a) correspond to $I(q^k,r^k,p^k\to y^k)-I(q^k,r^k,p^k\to y^k\Vert x^k)$, thereby proving the first part of the theorem, i.e., the equality in (5). In turn, $\left(b\right)$ stems from the non-negativity of mutual information turn into equality if $s⫫(r,p,q)$, as a direct consequence of the Markov chain in (30). Finally, equality holds in $\left(c\right)$ if $s⫫(q,r,p)$, since y depends causally upon $\theta$. This shows that equality in (5) is achieved if $s⫫(q,r,p)$, completing the proof.

4. Relationships between Mutual and Directed Information

The following result provides an inequality relating $I(x^k\to y^k)$ with the separate flows of information $I(r^k;y^k)$ and $I(p^k,q^k;y^k)$.

Theorem 3.

For the system shown in Figure 1a, if $s⫫(p,q,r)$ and $r^k⫫(p^k,q^k)$, then:

$$\begin{array}{cc}\hfill I(x^k\to y^k)& \ge I(r^k;y^k)+I(p^k,q^k;y^k).\hfill \end{array}$$

(33)

with equality if and only if the Markov chain $(p^k,q^k)\leftrightarrow y^k\leftrightarrow r^k$ holds.

Theorem 3 shows that, provided $(p,q,r)⫫s$, $I(x^k\to y^k)$ is lower bounded by the sum of the individual flows from all the subsets in any given partition of $(p^k,q^k,r^k)$, to $y^k$, provided these subsets are mutually independent. Indeed, both Theorems 1 and 3 can be generalized for any appropriate choice of external and internal signals. More precisely, let $\mathsf{\Theta }$ be the set of all external signals in a feedback system. Let $\alpha$ and $\beta$ be two internal signals in the loop. Define ${\mathsf{\Theta }}_{\alpha ,\beta }\subset \mathsf{\Theta }$ as the set of exogenous signals that are introduced to the loop at every subsystem ${\mathcal{S}}_i$ that lies in the path going from $\alpha$ to $\beta$. Thus, for any $\rho \in \mathsf{\Theta }\backslash {\mathsf{\Theta }}_{\alpha ,\beta }$, if ${\mathsf{\Theta }}_{\alpha ,\beta }⫫\mathsf{\Theta }\backslash {\mathsf{\Theta }}_{\alpha ,\beta }$, we have that (5) and (32) become:

$$\begin{array}{cc}\hfill I(\alpha \to \beta )& =I(\mathsf{\Theta }\backslash \left\{{\mathsf{\Theta }}_{\alpha ,\beta }\right\};\beta ),\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill I(\alpha \to \beta )-I(\rho ;\beta )& \ge I(\mathsf{\Theta }\backslash \{\rho \cup {\mathsf{\Theta }}_{\alpha ,\beta }\};\beta ),\hfill \end{array}$$

(35)

respectively.

To finish this section, we present a stronger, non-asymptotic version of inequality (12):

Theorem 4.

In the system shown in Figure 1a, if $(r,p,q,s)$ are mutually independent, then:

$$\begin{array}{cc}\hfill I(x^k\to y^k)& =I(r^k;u^k)+I(p^k;e^k)+I(q^k;y^k)+I(p^k;u^k|e^k)+I(r^k,p^k;y^k|u^k).\hfill \end{array}$$

(36)

Remark 5.

As anticipated, Theorem 4 can be seen as an extension of (12) to the more general setup shown in Figure 1a, where the assumptions made in [18] (Lemma 4.1) do not need to hold. In particular, letting the decoder D and p in Figure 1b correspond to ${\mathcal{S}}_4$ and $p^k$ in Figure 1a, respectively, we see that inequality (12) holds even if the channel f has memory or D and E have independent initial states, or if the internal state of D is not observable [40].

Theorem 4 also admits an interpretation in terms of information flows. This can be appreciated in the diagram shown in Figure 6, which depicts the individual full-turn flows (around the entire feedback loop) stemming from q, r, and p. Theorem 4 states that the sum of these individual flows is a lower bound for the directed information from x to y, provided $q,r,p,s$ are independent.

5. Relationships between Nested Directed Information

This section presents three closed-loop versions of the data-processing inequality relating two directed information terms, both between pairs of signals internal to the loop. As already mentioned in Section 1, to the best of our knowledge, the first inequality of this type to appear in the literature is the one in Theorem 4.1 in [20] (see (15)). Recall that the latter result stated that $I(x^k\to y^k\Vert q^k)\ge I(x^k\to u^k)$, requiring ${\mathcal{S}}_4$ to be such that $y^i$ is a deterministic function of $(u^i,q^i)$ and that $q⫫(r,p)$. The following result presents another inequality that also relates two nested directed information terms, namely, $I(x^k\to y^k)$ and $I(e^k\to y^k)$, but requiring only that $s⫫(q,r,p)$.

Theorem 5.

For the closed-loop system in Figure 1b, if $(q,r,p)⫫s$, then:

$$\begin{array}{cc}\hfill I(x^k\to y^k)& \ge I(e^k\to y^k).\hfill \end{array}$$

(37)

Notice that Theorem 5 does not require p to be independent of r or q. This may seem counterintuitive upon noting that p enters the loop between the link from e to x.

The following theorem is an identity between two directed information terms involving only internal signals. It can also be seen as a complement to Theorem 5, since it can be directly applied to establish the relationship between $I(e^k\to y^k)$ and $I(e^k\to u^k)$.

Theorem 6.

For the system shown in Figure 1a, if $s⫫(q,r,p)$, then:

$$\begin{array}{c}\hfill I(x^k\to y^k)\ge I(x^k\to u^k)+I(q^k;y^k)+I(r^k,p^k;y^k|u^k)+I(q^k;r^k|u^k,y^k),\end{array}$$

(38)

with equality if, in addition, $q⫫(r,p)$. In the latter case, it holds that:

$$\begin{array}{c}\hfill I(x^k\to y^k)=I(x^k\to u^k)+I(q^k;y^k)+I(r^k,p^k;y^k|u^k).\end{array}$$

(39)

Notice that by requiring additional independence conditions upon the exogenous signals (specifically, $q⫫s$), Theorem 6 (and, in particular, (39)) yields:

$$\begin{array}{c}\hfill I(x^k\to y^k)\ge I(x^k\to u^k),\end{array}$$

(40)

which strengthens the inequality in [20] (Theorem 4.1) (stated above in (15)). More precisely, (40) does not require conditioning one of the directed information terms and holds irrespective of the invertibility of the mappings in the loop.

6. Giving Operational Meaning to the Directed Information: In-the-Loop Channel Coding

In this section we introduce the notions of in-the-loop transmission rate and capacity and show that they are related by the directed information rate across the channel in the same feedback loop. This provides another example to illustrate the applicability of Theorems 1 and 2 and also provides further operational meaning to the directed information rate.

Consider the scheme shown in Figure 7, and suppose $\mathcal{C}$ is a noisy communication channel. Let $\mathcal{E}$ and $\mathcal{D}$ be the channel encoder and decoder, respectively, with r and p being side information sequences causally and independently available to each of them such that $(r,p)⫫(s,q)$. This means that, for $k=1,2,\dots ,n$,

$$\begin{array}{cc}\hfill (p_1^n,r_{k+1}^n)& \leftrightarrow r_1^k\leftrightarrow (w_1^{k+1},x_1^k,y_0^k).\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill p_{k+1}^n& \leftrightarrow p_1^k\leftrightarrow r_1^k\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill p_{k+1}^n& \leftrightarrow p_1^k\leftrightarrow (w_1^{k+1},x_1^k,y_0^k).\hfill \end{array}$$

(43)

A crucial aspect of this scenario is the fact that the messages $w_1^n,w_{n+1}^{2n},\dots$ to be encoded are contained in the sequence w, a signal internal to the loop; they can be regarded as a corrupted version of the decoded messages, which comprise the sequence v. This is a key difference with respect to the available literature on feedback capacity, where, to the best of the authors’ knowledge, the messages are exogenous and the feedback signal only helps in the encoding task (exceptions can be found in some papers on networked control which consider in-the-loop channel coding, such as, e.g., [41,42]). In Figure 7, the latter standard scenario corresponds to encoding the sequence r.

The fact that the messages to be encoded bear information from the decoded message symbols requires one to redefine the notion of information transmission rate commonly used in the standard scenario. To see this, let $w\left(k\right)\in \mathcal{W},k=1,2,\dots$, for some finite alphabet $\mathcal{W}$ of cardinality $\left|\mathcal{W}\right|$, and notice that the transmission rate definitions $log\left(\left|\mathcal{W}\right|\right)$ and $H\left(w_1^n\right)/n$ are unsatisfactory if $w\left(k\right)=y(k-1),k=1,2,\dots$, i.e., if the messages to be transmitted are already available at the decoder (more generally, if there is no randomness in the feedback path). This suggests that a suitable notion of transmission rate for this scenario should exclude information that is already known by the receiver.

In view of the above, we propose the following notion of transmission rate for the case in which the messages to be transmitted are in the loop:

Definition 3.

For the system described in Figure 1, the in-the-loop (ITL) transmission rate is defined as:

$$\begin{array}{c}\hfill R_{\mathrm{ITL}}^n\triangleq \frac{1}{n}\sum _{k=1}^{n}H(w\left(k\right)|w_1^{k-1},y_0^{k-1},p_1^k).\end{array}$$

(44)

The meaning of the ITL transmission rate is further elucidated by considering the following scenarios:

• If the feedback channel is deterministic, then $w\left(k\right)$ is a deterministic function of $y_0^{k-1}$ and thus $R_{\mathrm{ITL}}^n=0$, as desired.
• If the (forward) communication channel is noiseless, then at each time $k-1$, we have $y_1^{k-1}=w_1^{k-1}$. Therefore $R_{\mathrm{ITL}}^n=H\left(w_1^n\right|y_0,p_1^n)/n$. Again, if the feedback channel is deterministic, the ITL transmission rate is zero.
• In the absence of feedback, $R_{\mathrm{ITL}}^n=\frac{1}{n}H\left(w_1^n\right)$, recovering the notion of transmission rate of the case in which the messages are exogenous to the loop.

Thus, $R_{\mathrm{ITL}}^n$ can be interpreted as the sum of the information the encoder attempts to transmit at each sample time (expressed by the conditional entropy $H(w\left(k\right)|w_1^{k-1},y_0^{k-1},p_1^k)$) that is novel for the transmitter (because of the conditioning on $w_1^{k-1}$) and novel for the receiver (due to the conditioning on $y_0^{k-1},p_1^k$).

Theorem 7.

Consider the setup depicted in Figure 7, where $\mathcal{E}$ and $\mathcal{D}$ are the channel encoder and decoder, respectively, and $\mathcal{C}$ is the communication channel. Suppose the message and side-information samples $w\left(k\right)\in \mathcal{W},r\left(k\right)\in \mathcal{R},k=1,2,\dots$, respectively, where $\mathcal{W}$ and $\mathcal{R}$ are finite alphabets. Define the binary random variable $e_n$ to equal 1 if $v_1^n\ne w_1^n$ and 0 otherwise. Then, for every $n\in \mathbb{N}$,

$$\begin{array}{c}\hfill R_{\mathrm{ITL}}^n\ge I(w_1^n\to y_0^n\parallel p_1^n),\end{array}$$

(45)

with equality if and only if $H\left(w_1^n\right|y_0^n,p_1^n)=0$. Moreover,

$$\begin{array}{cc}\hfill Pr\{e_n=1\}& =\frac{R_{\mathrm{ITL}}^n-\frac{1}{n}I(w_1^n\to y_0^n\parallel p_1^n)-\frac{1}{n}H\left(e_n\right|y_0^n,p_1^n)}{\frac{1}{n}H\left(w_1^n\right|y_0^n,p_1^n,e_n=1)}\hfill \end{array}$$

(46)

$$\begin{array}{cc}& \ge \frac{R_{\mathrm{ITL}}^n-\frac{1}{n}I(w_1^n\to y_0^n\parallel p_1^n)-1/n}{{log}_2\left(\left|\mathcal{W}\right|\right)}\hfill \end{array}$$

(47)

Proof. 

Recall that:

$$\begin{array}{cc}\hfill I(w_1^n\to y_0^n\parallel p_1^n)& =\sum _{k=1}^{n}I(w_1^k;y\left(k\right)|y_0^{k-1},p_1^k)=\sum _{k=1}^{n}H\left(w_1^k\right|y_0^{k-1},p_1^k)-\sum _{k=1}^{n}H\left(w_1^k\right|y_0^k,p_1^k)\hfill \end{array}$$

(48)

On the other hand,

$$\begin{array}{cc}\hfill n{R}_{\mathrm{ITL}}^n& =\sum _{k=1}^{n}H(w\left(k\right)|w_1^{k-1},y_0^{k-1},p_1^k)\stackrel{\left(\mathrm{cr}\right)}{=}\sum _{k=1}^{n}H\left(w_1^k\right|y_0^{k-1},p_1^k)-\sum _{k=2}^{n}H\left(w_1^{k-1}\right|y_0^{k-1},p_1^k)\hfill \end{array}$$

(49)

$$\begin{array}{cc}& \stackrel{\left(48\right)}{=}\sum _{k=1}^{n}H\left(w_1^k\right|y_0^k,p_1^k)-\sum _{k=2}^{n}H\left(w_1^{k-1}\right|y_0^{k-1},p_1^k)+I(w_1^n\to y_0^n\parallel p_1^n)\hfill \end{array}$$

(50)

$$\begin{array}{cc}& \stackrel{\left(43\right)}{=}H\left(w_1^k\right|y_0^k,p_1^k)+I(w_1^n\to y_0^n\parallel p_1^n),\hfill \end{array}$$

(51)

where the equality (cr) follows from the chain rule of entropy. This proves the first part of the theorem.

Let us now re-derive the first steps leading to Fano’s inequality, to include the side-information $p_1^n$ and to verify that it is not affected by the fact that w and y are within the loop.

$$\begin{array}{cc}\hfill H\left(w_1^n\right|y_0^n,p_1^n)& \stackrel{\left(\mathrm{cr}\right)}{=}H(w_1^n,e_n|y_0^n,p_1^n)-H\left(e_n\right|y_0^n,p_1^n,w_1^n)\hfill \end{array}$$

(52)

$$\begin{array}{cc}& \stackrel{\left(a\right)}{=}H\left(e_n\right|y_0^n,p_1^n)+H\left(w_1^n\right|y_0^n,p_1^n,e_n)\hfill \end{array}$$

(53)

$$\begin{array}{cc}& \stackrel{\left(b\right)}{=}H\left(e_n\right|y_0^n,p_1^n)+H\left(w_1^n\right|y_0^n,p_1^n,e_n=1)Pr\{e_n=1\},\hfill \end{array}$$

(54)

where the equality (cr) follows from the chain rule of entropy and $\left(a\right)$ holds because $H\left(e_n\right|y_0^n,p_1^n,w_1^n)=0$ and from the chain rule, while $\left(b\right)$ is because $H\left(w_1^n\right|y_0^n,p_1^n,e_n=0)=0$.

Substituting this into (51),

$$\begin{array}{cc}\hfill n{R}_{\mathrm{ITL}}^n& =H\left(e_n\right|y_0^n,p_1^n)+H\left(w_1^n\right|y_0^n,p_1^n,e_n=1)Pr\{e_n=1\}+I(w_1^n\to y_0^n\parallel p_1^n).\hfill \end{array}$$

(55)

Noting that $H\left(e_n\right|y_0^n,p_1^n)\le 1$ and $H\left(w_1^n\right|y_0^n,p_1^n,e_n=1)\le nlog\left(\left|\mathcal{W}\right|\right)$ leads directly to (46), comparing the proof. □

Theorem 7 allows one to draw an additional interpretation of the ITL transmission rate. We extend first the identity of [27] to include causal conditioning by $p_1^n$:

$$\begin{array}{cc}\hfill I(w_1^n;y_0^n,p_1^n)& =I(w_1^n;y_n|y_0^{n-1},p_1^n)+I(w_n;y_0^{n-1}|w_1^{n-1},p_1^n)+I(w_1^{n-1};y_0^{n-1},p_1^{n-1})\hfill \end{array}$$

(56)

$$\begin{array}{cc}& =\sum _{k=1}^{n}I(w_1^k;y\left(k\right)|y_0^{k-1},p_1^k)+\sum _{k=1}^{n}I(w\left(k\right);y_0^{k-1}|w_1^{k-1},p_1^k)=I(w_1^n\to y_0^n\parallel p_1^n)+I(y_0^{n-1}\to w_1^n\parallel p_1^n),\hfill \end{array}$$

(57)

where:

$$\begin{array}{c}\hfill I(y_0^{n-1}\to w_1^n\parallel p_1^n)\triangleq \sum _{k=1}^{n}I(w\left(k\right);y_0^{k-1}|w_1^{k-1},p_1^k).\end{array}$$

(58)

It readily follows from (56) that:

$$\begin{array}{c}\hfill H\left(w_1^n\right)-I(y_0^{n-1}\to w_1^n\parallel p_1^n)=I(w_1^n\to y_0^n\parallel p_1^n)+H\left(w_1^n\right|y_0^n,p_1^n)\stackrel{\left(51\right)}{=}n{R}_{\mathrm{ITL}}^n.\end{array}$$

(59)

Thus, the ITL transmission rate corresponds to the entropy rate of the messages having extracted from it the information flowing from the decoder input to the messages.

The main result of this section is the following theorem, which asserts that the supremum of achievable ITL transmission rates is upper bounded by the directed information across the communication channel.

Theorem 8.

Consider the setup depicted in Figure 7, where $\mathcal{E}$ and $\mathcal{D}$ are the channel encoder and decoder, respectively, and $\mathcal{C}$ is the communication channel. Then the supremum of achievable ITL transmission rates is upper bounded by the supremum of the directed information rate from x to y causally conditioned by $p_1^n$.

Proof. 

The result follows directly from Theorems 2 and 7. □

Thus, the supremum of ${lim}_{n\to \infty }I(x_1^n\to y_1^n\parallel p_1^n)$ is an outer bound to the capacity region of ITL transmission rates.

In the following example, this bound is reachable.

Example 1.

Consider the case in which the forward channel $\mathcal{C}$ in Figure 7 is transparent, i.e., $y\left(k\right)=x\left(k\right)$ for $k=0,1,\dots$, as shown in Figure 8. Let $y\left(k\right)\in \{0,1,2,3\}$, $k=0,1,\dots$. Let $q\left(0\right)=1$ (deterministically) and $q\left(1\right),q\left(2\right),\dots$ be binary and i.i.d. with $Pr\{q(k)=1\}=\alpha =0.9$. The feedback channel $\mathcal{S}$ is defined by the following recursion:

$$\begin{array}{cc}\hfill w\left(k\right)& =\left\{\begin{array}{cc}q\left(k\right)\hfill & ,\mathrm{if}q(k-1)=(y(k-1)mod2)\hfill \\ (y(k-1)mod2)\hfill & ,\mathrm{if}q(k-1)\ne (y(k-1)mod2)\hfill \end{array}\right.,k=1,2,\dots \hfill \end{array}$$

(60)

Thus, $\mathcal{S}$ outputs a new sample of q iff the previous sample of q is matched by the previous sample $mod2$ of y. Otherwise, it lets $y(k-1)mod2$ pass through.

Consider first the following encoder–decoder pair, designed with the aim of achieving zero-error communication while maximizing $H\left(w_1^n\right)/n=I(w_1^n;v_1^n)$.

Encoder ${\mathcal{E}}_1$: Let the side-information sequence r be binary i.i.d. and independent of q, with $Pr\{r(k)=1\}=\beta$, and:

$$\begin{array}{cc}y\left(0\right)\hfill & =r\left(0\right)\hfill \end{array}$$

(61)

$$\begin{array}{cc}y\left(k\right)\hfill & =\left\{\begin{array}{cc}r\left(k\right)\hfill & ,\mathrm{if}w\left(k\right)=(y(k-1)mod2)\hfill \\ r\left(k\right)+2\hfill & ,\mathrm{if}w\left(k\right)\ne (y(k-1)mod2)\hfill \end{array}\right.,k=1,2,\dots \hfill \end{array}$$

(62)

Decoder ${\mathcal{D}}_1$:

$$\begin{array}{cc}\hfill v\left(k\right)& =\left\{\begin{array}{cc}y(k-1)mod2\hfill & ,\mathrm{if}y\left(k\right)\le 1\hfill \\ (y(k-1)mod2)\oplus 1\hfill & ,\mathrm{if}y\left(k\right)>1\hfill \end{array}\right.\hfill \end{array}$$

(63)

where ⊕ is the exclusive–or binary operator. With this choice, $v\left(k\right)=w\left(k\right)$ for $k=1,2,\dots$. In addition,

$$\begin{array}{cc}\hfill w\left(k\right)& =\left\{\begin{array}{cc}q\left(k\right)\hfill & ,\mathrm{if}r(k-1)=q(k-1)\hfill \\ r(k-1)\hfill & ,\mathrm{if}r(k-1)\ne q(k-1)\hfill \end{array}\right.,k=1,2,\dots \hfill \end{array}$$

(64)

Therefore,

$$\begin{array}{c}\hfill Pr\{w(1)=1\}=\alpha \beta .\end{array}$$

(65)

and, for $k\ge 2$,

$$\begin{array}{cc}\hfill Pr& \{w\left(k\right)=1\}=({\alpha }^2+{(1-\alpha )}^2)\beta +\alpha (1-\alpha )\hfill \end{array}$$

(66)

Thus, and since $\alpha =0.9$, the entropy of each $w\left(k\right)$ is maximized by $\beta =0.5055$. However, encoder ${\mathcal{E}}_1$ makes the samples of w interdependent, so finding the value of β that maximizes $H\left(w_1^n\right)/n$ (and thus $I(w_1^n;v_1^n)$ as well) is more involved, and that value does not need to be the same. We have found numerically that (for $n=22$) the maximum of $H\left(x_1^n\right)/n=I(w_1^n;v_1^n)/n$ is (approximately) 0.9941 [bits/sample], attained with $\beta =0.503$, very close to the β which maximizes $H\left(w_1^n\right)/n$.

For later comparison, we also calculate the value of $R_{\mathrm{ITL}}^n$ yielded by this choice of encoder:

$$\begin{array}{c}\hfill R_{\mathrm{ITL}}^n\stackrel{\left(a\right)}{=}I(w_1^n\to y_1^n)\stackrel{\mathrm{Thm}.}{=}I(q_0^n;y_0^n)=\sum _{k=0}^n(H(q\left(k\right)|q_0^{k-1})-H(q\left(k\right)|q_0^{k-1},y_0^n)),\end{array}$$

(67)

where $\left(a\right)$ holds from Theorem 7 because $H\left(w_1^n\right|y_0^n)=0$. Defining the binary random variables $t\left(k\right)\triangleq 1$ when $(y(k)mod2)=q\left(k\right)$ and 0 otherwise, we obtain: -4.6cm0cm

$$\begin{array}{cc}\hfill H(q\left(k\right)|q_0^{k-1},y_0^n)& =H(q\left(k\right)|q_0^{k-1},y_0^n,t(k-1))\hfill \\ & =H(q\left(k\right)|q_0^{k-1},y_0^n,t(k-1)=0)Pr\{t(k-1)=0\}+H(q\left(k\right)|q_0^{k-1},y_0^n,t(k-1)=1)Pr\{t(k-1)=1\}\hfill \end{array}$$

(68)

$$\begin{array}{cc}& \stackrel{\left(64\right)}{=}H(q\left(k\right))Pr\{t(k-1)=0\}+0·Pr\{t(k-1)=1\}\hfill \end{array}$$

(69)

Thus,

$$\begin{array}{cc}\hfill R_{\mathrm{ITL}}^n& =H(q(k\left)\right)(1-Pr\{t(k-1)=0\left\}\right)=H(q(k\left)\right)\left(\alpha \right(1-\beta )+(1-\alpha \left)\beta \right)\hfill \end{array}$$

(70)

$$\begin{array}{cc}& =0.469\times 0.4976=0.2334[bits/sample],\hfill \end{array}$$

(71)

using $\beta =0.503$.

The second encoder/decoder pair is set to maximize $R_{\mathrm{ITL}}^n$, and is defined as follows:

Encoder ${\mathcal{E}}_2$:

$$\begin{array}{c}\hfill y\left(k\right)=\left\{\begin{array}{cc}1\hfill & ,\mathrm{if}k=0\hfill \\ w\left(k\right)\hfill & ,\mathrm{if}k\ge 1\hfill \end{array}\right.\end{array}$$

(72)

Thus, zero-error communication is trivially attained with the simple decoding rule:

Decoder ${\mathcal{D}}_2$:

$$\begin{array}{c}\hfill v\left(k\right)=y\left(k\right),k\ge 1.\end{array}$$

(73)

In addition, encoder ${\mathcal{E}}_2$ yields $w\left(k\right)=q\left(k\right)$, for $k\ge 1$. Therefore,

$$\begin{array}{cc}\hfill \frac{1}{n}I(w_1^n\to y_0^n)\stackrel{\mathrm{Thm}.}{=}{R}_{\mathrm{ITL}}^n& =\frac{1}{n}H\left(q_1^n\right)=0.469[bits/sample]\hfill \end{array}$$

(74)

As expected, encoder ${\mathcal{E}}_2$ yields a higher $R_{\mathrm{ITL}}^n$ than encoder ${\mathcal{E}}_1$. More significant is the fact that encoder/decoder pair 2 achieves the in-the-loop capacity for this channel, since:

$$\begin{array}{c}\hfill \frac{1}{n}I(w_1^n\to y_0^n)\stackrel{\left(a\right)}{=}I(q_1^n;y_0^n)\le H\left(q_1^n\right)\end{array}$$

(75)

The previous example illustrates an important fact that is closely related with the motivation behind the definition of $R_{\mathrm{ITL}}^n$: maximizing the mutual information between the messages to be transmitted and the decoded messages (a leitmotif in traditional channel coding, wherein messages are generated outside the loop) is not suitable when messages are in the loop.

Indeed, (56) provides a mathematically precise meaning to the above observation. It reveals why maximizing $I(w_1^n;y_0^n,p_1^n)$ does not necessarily mean maximizing $I(w_1^n\to y_0^n\parallel p_1^n)$, since the former is the sum of backward and forward information flows (represented in green and red in Figure 8, respectively).

Finally, Theorems 7 and 8 imply that in the design of any encoder for in-the-loop messages, aiming to yield the joint probability distribution of channel input and output sequences that maximizes the directed information is of practical importance: it is necessary for achieving the highest “useful” transmission rate while minimizing the probability of error.

7. Concluding Remarks

The widely used data processing inequality does not hold for systems with feedback. In this work, we provided a very general directed information data processing inequality that is applicable to feedback systems. A key insight to be gained from this new inequality is that, for nested pairs of sequences, the further apart the signals in the feedback system are from each other, the lower is the directed information between them (measuring distance from starting to finishing sequence and in the direction of cause and effect). Thus, post processing signals within a feedback loop cannot increase the information, which is similar to the open loop case. In order to obtain these results, we considered arbitrary causal systems that are interconnected in a feedback loop, with arbitrarily distributed signals. We were able to overcome the generally non-trivial dependencies between the signals in such a scenario by establishing a family of useful Markov chains that conditionally decouple the sequences in the system. These Markov chains are useful by themselves for studies involving interconnected systems. We further used the Markov chains to derive a number of fundamental information inequalities that are applicable to signals that are entirely within feedback loops or where some signals are inside and others outside the loop. With the use of these inequalities, we were able to show that the conventional notion of channel capacity is not adequate for in-the-loop communications. Instead, we provided the new notion of in-the-loop channel capacity, and described a special case where it was achievable. As an additional application of our results, we discussed how they allow one to generalize two known fundamental inequalities in networked control involving directed information. We are confident that our analysis provides useful insights to understand and think about information flows in single-loop feedback systems, and that our results will serve as a toolbox for research in, e.g., networked control systems or communications within a feedback loop.

There are several future research directions stemming from this work, from which we outline the following three:

(1)

Establishing whether (and under which conditions, if any) in the system of Figure 1, each of the following inequalities is true or false:

$$\begin{array}{c}\hfill I(r^k;e^k)\ge I(r^k;x^k)\ge I(r^k;y^k)\ge I(r^k;u^k).\end{array}$$

(76)

(2)

Extending Theorems 1 and 2 to scenarios with more than one feedback loop.

(3)

Exploring if tree codes [43] can be tailored to maximize the ITL data rate instead of the conventional data rate within a feedback loop. If such adaptation is possible, it would be interesting to assess how close to the ITL channel capacity such codes can perform.