Finite-Length Analyses for Source and Channel Coding on Markov Chains ^†

Abstract

We derive finite-length bounds for two problems with Markov chains: source coding with side-information where the source and side-information are a joint Markov chain and channel coding for channels with Markovian conditional additive noise. For this purpose, we point out two important aspects of finite-length analysis that must be argued when finite-length bounds are proposed. The first is the asymptotic tightness, and the other is the efficient computability of the bound. Then, we derive finite-length upper and lower bounds for the coding length in both settings such that their computational complexity is low. We argue the first of the above-mentioned aspects by deriving the large deviation bounds, the moderate deviation bounds, and second-order bounds for these two topics and show that these finite-length bounds achieve the asymptotic optimality in these senses. Several kinds of information measures for transition matrices are introduced for the purpose of this discussion.

1. Introduction

In recent years, finite-length analyses for coding problems have been attracting considerable attention [1]. This paper focuses on finite-length analyses for two representative coding problems: One is source coding with side-information for Markov sources, i.e., the Markov–Slepian–Wolf problem on the system $X^n$ with full side-information $Y^n$ at the decoder, where only the decoder observes the side-information and the source and the side-information are a joint Markov chain. The other is channel coding for channels with Markovian conditional additive noise. Although the main purpose of this paper is finite-length analyses, we also present a unified approach we developed to investigate these topics including asymptotic analyses. Since this discussion is spread across a number of subtopics, we explain them separately in the Introduction.

1.1. Two Aspects of Finite-Length Analysis

We explain the motivations of this research by starting with two aspects of finite-length analysis that must be argued when finite-length bounds are proposed. For concreteness, we consider channel coding here even though the problems treated in this paper are not restricted to channel coding. To date, many types of finite-length achievability bounds have been proposed. For example, Verdú and Han derived a finite-length bound by using the information-spectrum approach in order to derive the general formula [2] (see also [3]), which we term as the information-spectrum bound. One of the authors and Nagaoka derived a bound (for the classical-quantum channel) by relating the error probability to binary hypothesis testing [4] (Remark 15) (see also [5]), which we refer to as the hypothesis-testing bound. Polyanskiy et. al. derived the random coding union (RCU) bound and the dependence testing (DT) bound [1] (a bound slightly looser (coefficients are worse) than the DT bound can be derived from the hypothesis-testing bound of [4]). Moreover, Gallager’s bound [6] is known as an efficient bound to derive the exponentially decreasing rate.

Here, we focus on two important aspects of finite-length analysis:

(A1) 

Computational complexity for the bound and

(A2) 

Asymptotic tightness for the bound.

Both aspects are required for the bound in finite-length analysis as follows. As the first aspect, we consider the computational complexity for the bound. For the BSC (binary symmetric channel), the computational complexity of the RCU bound is $O\left(n^2\right)$, and that of the DT bound is $O\left(n\right)$ [7]. However, the computational complexities of these bounds are much larger for general DMCs (discrete memoryless channels) or channels with memory. It is known that the hypothesis testing bound can be described as a linear programming problem (e.g., see [8,9] (in the the case of a quantum channel, the bound is described as a semi-definite programming problem)) and can be efficiently computed under certain symmetry. However, the number of variables in the linear programming problem grows exponentially with the block length, and it is difficult to compute in general. The computation of the information-spectrum bound depends on the evaluation of the tail probability. The hypothesis testing bound gives a tighter bound than the information-spectrum bound, as pointed out by [8], and the computational complexity of the former is much smaller than that of the latter. However, the computation of the tail probability continues to remain challenging unless the channel is a DMC. For DMCs, the computational complexity of Gallager’s bound is $O\left(1\right)$ since the Gallager function is an additive quantity for DMCs. However, this is not the case if there is a memory (the Gallager bound for finite-state channels was considered in [10] (Section 5.9), but a closed form expression for the exponent was not derived). Consequently, no efficiently computable bound currently exists for channel coding with Markov additive noise. The situation is the same for source coding with side-information.

Since the actual computation time may depend on the computational resource we can use for numerical experiment, it is not possible to provide a concrete requirement of computational complexity. However, in order to conduct a numerical experiment for a meaningful blocklength, it is reasonable to require the computational complexity to be, at most, a polynomial order of the blocklength n.

Next, let us consider the second aspect, i.e., asymptotic tightness. Thus far, three kinds of asymptotic regimes have been studied in information theory [1,11,12,13,14,15,16]:

• A large deviation regime in which the error probability $\epsilon$ asymptotically behaves as $e^{-nr}$ for some $r>0$;
• A moderate deviation regime in which $\epsilon$ asymptotically behaves as $e^{-n^{1-2t}r}$ for some $r>0$ and $t\in (0,1/2)$; and
• A second-order regime in which $\epsilon$ is a constant.

We shall claim that a good finite-length bound should be asymptotically optimal for at least one of the above-mentioned three regimes. In fact, the information-spectrum bound, the hypothesis-testing bound, and the DT bound are asymptotically optimal in both the moderate deviation and second-order regimes, whereas the Gallager bound is asymptotically optimal in the large deviation regime and the RCU bound asymptotically optimal in all the regimes (Both the Gallager and RCU bounds are asymptotically optimal in the large deviation regime only up to the critical rate). Recently, for DMCs, Yang and Meng derived an efficiently computable bound for low-density parity check (LDPC) codes [17], which is asymptotically optimal in both the moderate deviation and second-order regimes.

1.2. Main Contribution for Finite-Length Analysis

We derive the finite-length achievability bounds on the problems by basically using the exponential-type bounds (for channel coding, it corresponds to the Gallager bound.). In source coding with side-information, the exponential-type upper bounds on error probability ${\overline{\mathrm{P}}}_{\mathrm{e}}\left(M_n\right)$ for a given message size $M_n$ are described by using the conditional Rényi entropies as follows (cf. Lemmas 14 and 15):

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{e}}\left(M_n\right)\le \underset{-\frac{1}{2}\le \theta \le 0}{inf}{M}_n^{\frac{\theta }{1+\theta }}{e}^{-\frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)}\end{array}$$

(1)

and:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{e}}\left(M_n\right)\le \underset{-1\le \theta \le 0}{inf}{M}_n^{\theta }{e}^{-\theta H_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)}.\end{array}$$

(2)

Here, $X^n$ is the information to be compressed and $Y^n$ is the side-information that can be accessed only by the decoder. $H_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)$ is the conditional Rényi entropy introduced by Arimoto [18], which we shall refer to as the upper conditional Rényi entropy (cf. (12)). On the other hand, $H_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)$ is the conditional Rényi entropy introduced in [19], which we shall refer to as the lower conditional Rényi entropy (cf. (7)). Although there are several other definitions of conditional Rényi entropies, we only use these two in this paper; see [20,21] for an extensive review on conditional Rényi entropies.

Although the above-mentioned conditional Rényi entropies are additive for i.i.d. random variables, they are not additive for joint Markov chains over $X^n$ and $Y^n$, for which the derivation of finite-length bounds for Markov chains are challenging. Because it is generally not easy to evaluate the conditional Rényi entropies for Markov chains, we consider two assumptions in relation to transition matrices: the first assumption, which we refer to as non-hidden, is that the Y-marginal process is a Markov chain, which enables us to derive the single-letter expression of the conditional entropy rate and the lower conditional Rényi entropy rate; the second assumption, which we refer to as strongly non-hidden, enables us to derive the single-letter expression of the upper conditional Rényi entropy rate; see Assumptions 1 and 2 of Section 2 for more detail (Indeed, as explained later, our result on the data compression can be converted to a result on the channel coding for a specific class of channels. Under this conversion, we obtain certain assumptions for channels. As explained later, these assumptions for channels are more meaningful from a practical point of view.). Under Assumption 1, we introduce the lower conditional Rényi entropy for transition matrices $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ (cf. (47)). Then, we evaluate the lower conditional Rényi entropy for the Markov chain in terms of its transition matrix counterpart. More specifically, we derive an approximation:

$$\begin{array}{c}\hfill H_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)=n{H}_{1+\theta }^{\downarrow ,W}\left(X\right|Y)+O\left(1\right),\end{array}$$

(3)

where an explicit form of the $O\left(1\right)$ term is also derived. Using the evaluation (2) with this evaluation, we obtain finite-length bounds under Assumption 1. Under a more restrictive assumption, i.e., Assumption 2, we also introduce the upper conditional Rényi entropy for a transition matrix $H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$ (cf. (55)). Then, we evaluate the upper Rényi entropy for the Markov chain in terms of its transition matrix counterpart. More specifically, we derive an approximation:

$$\begin{array}{c}\hfill H_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)=n{H}_{1+\theta }^{\uparrow ,W}\left(X\right|Y)+O\left(1\right),\end{array}$$

(4)

where an explicit form of the $O\left(1\right)$ term is also derived. Using the evaluation (1) with this evaluation, we obtain finite-length bounds that are tighter than those obtained under Assumption 1. It should be noted that, without Assumption 1, even the conditional entropy rate is challenging to evaluate. For evaluation of the conditional entropy rate of the X process given the Y process, the assumption of the X process being Markov seems to be not helpful. This is the reason why we consider the Y process being Markov instead of the X process being Markov in this paper.

We also derive converse bounds by using the change of measure argument for Markov chains developed by the authors in the accompanying paper on information geometry [22,23]. For this purpose, we further introduce two-parameter conditional Rényi entropy and its transition matrix counterpart (cf. (18) and (59)). This novel information measure includes the lower conditional Rényi entropy and the upper conditional Rényi entropy as special cases. We clarify the relation among bounds based on these quantities by numerically calculating the upper and lower bounds for the optimal coding rate in source coding with a Markov source in Section 3.7. Owing to the second aspect (A2), this calculation shows that our finite-length bounds are very close to the optimal value. Although this numerical calculation contains a case with a very large size $n=1\times {10}^5$, its calculation is not as difficult because the calculation complexity behaves as $O\left(1\right)$. That is, this calculation shows the advantage of the first aspect (A1).

Here, we would like to remark about the terminologies because there are a few ways to express exponential-type bounds. In statistics or large deviation theory, we usually use the cumulant generating function (CGF) to describe exponents. In information theory, we employ the Gallager function or the Rényi entropies. Although these three terminologies are essentially the same quantity and are related by the change of variables, the CGF and the Gallager function are convenient for some calculations because of their desirable properties such as convexity. On the other hand, the minimum entropy and collision entropy are often used as alternative information measures of Shannon entropy in the community of cryptography. Since the Rényi entropies are a generalization of the minimum entropy and collision entropy, we can regard the Rényi entropies as information measures. The information theoretic meaning of the CGF and the Gallager function are less clear. Thus, the Rényi entropies are intuitively familiar to the readers’ of this journal. The Rényi entropies have an additional advantage in that two types of bounds (e.g., (152) and (161)) can be expressed in a unified manner. Therefore, we state our main results in terms of the Rényi entropies, whereas we use the CGF and the Gallager function in the proofs. For the readers’ convenience, the relation between the Rényi entropies and corresponding CGFs are summarized in Appendix A and Appendix B.

1.3. Main Contribution for Channel Coding

An intimate relationship is known to exist between channel coding and source coding with side-information (e.g., [24,25,26]). In particular, for an additive channel, the error probability of channel coding by a linear code can be related to the corresponding source coding problem with side-information [24]. Chen et. al. also showed that the error probability of source coding with side-information by a linear encoder can be related to the error probability of a dual channel coding problem and vice versa [27] (see also [28]). Since these dual channels can be regarded as additive channels conditioned on state information, we refer to these channels as conditional additive channels (In [28], we termed these channels general additive channels, but we think “conditional” more suitably describes the situation.). In this paper, we mainly discuss a conditional additive channel, in which the additive noise is operated subject to a distribution conditioned on additional output information. Then, we convert our obtained results of source coding with side-information to the analysis on conditional additive channels. That is, using the aforementioned duality between channel coding and source coding with side-information enables us to evaluate the error probability of channel coding for additive channels. Then, we derive several finite-length analyses on additive channels.

For the same reason as source coding with side-information, we make two assumptions, Assumptions 1 and 2, on the noise process of a conditional additive channel. In this context, Assumption 1 means that the marginal system $Y^n$ deciding the behavior of the additive noise $X^n$ is a Markov chain. It should be noted that the Gilbert–Elliott channel [29,30] with state information available at the receiver can be regarded as a conditional additive channel such that the noise process is a Markov chain satisfying both Assumptions 1 and 2 (see Example 6). Thus, we believe that Assumptions 1 and 2 are quite reasonable assumptions.

In fact, our analysis is applicable for a broader class of channels known as regular channels [31]. The class of regular channels includes conditional additive channels as a special case, and it is known as a class of channels that are similarly symmetrical. To show it, we propose a method to convert a regular channel into a conditional additive channel such that our treatment covers regular channels. Additionally, we show that the BPSK (binary phase shift keying)-AWGN (additive white Gaussian noise) channel is included in conditional additive channels.

1.4. Asymptotic Bounds and Asymptotic Tightness for Finite-Length Bounds

We present asymptotic analyses of the large and moderate deviation regimes by deriving the characterizations (for the large deviation regime, we only derive the characterizations up to the critical rate) with the use of our finite-length achievability and converse bounds, which implies that our finite-length bounds are tight in both of these deviation regimes. We also derive the second-order rate. Although this rate can be derived by the application of the central limit theorem to the information-spectrum bound, the variance involves the limit with respect to the block length because of memory. In this paper, we derive a single-letter form of the variance by using the conditional Rényi entropy for transition matrices (An alternative way to derive a single-letter characterization of the variance for the Markov chain was shown in [32] (Lemma 20). It should also be noted that a single-letter characterization can be derived by using the fundamental matrix [33]. The single-letter characterization of the variance in [12] (Section VII) and [11] (Section III) contains an error, which is corrected in this paper.).

As we will see in Theorems 11–14 and 22–25, our asymptotic results have the same forms as the counterparts of the i.i.d. case (cf. [1,6,11,12,13,14]) when the information measures for distributions in the i.i.d. case are replaced by the information measures for the transition matrices introduced in this paper.

We determine the asymptotic tightness for finite-length bounds by summarizing the relation between the asymptotic results and the finite-length bounds in

Table 1. Summary of asymptotic results and finite-length bounds to derive asymptotic results under Assumptions 1 and 2, which are abbreviated to Ass. 1 and Ass. 2.

Problem	First-Order	Large Deviation	Moderate Deviation	Second-Order
SC with SI	Solved (Ass. 1)	${\mathrm{Solved}}^{*}$ (Ass. 2)	Solved (Ass. 1),	Solved (Ass. 1)
		$O\left(1\right)$	$O\left(1\right)$	Tail
CC for Conditional	Solved (Ass. 1)	${\mathrm{Solved}}^{*}$ (Ass. 2)	Solved (Ass. 1)	Solved (Ass. 1)
Additive Channels		$O\left(1\right)$	$O\left(1\right)$	Tail

. The table also describes the computational complexity of the finite-length bounds. “${\mathrm{Solved}}^{*}$” indicates that those problems are solved up to the critical rates. “Ass. 1” and “Ass. 2” indicate that those problems are solved either under Assumption 1 or Assumption 2. “$O\left(1\right)$” indicates that both the achievability and converse parts of those asymptotic results are derived from our finite-length achievability bounds and converse bounds whose computational complexities are $O\left(1\right)$. “Tail” indicates that both the achievability and converse parts of those asymptotic results are derived from the information-spectrum-type achievability bounds and converse bounds of which the computational complexities depend on the computational complexities of the tail probabilities.

In general, the exact computations of tail probabilities are difficult, although they may be feasible for a simple case such as an i.i.d. case. One way to compute tail probabilities approximately is to use the Berry–Esséen theorem [34] (Theorem 16.5.1) or its variant [35]. This direction of research is still ongoing [36,37], and an evaluation of the constant was conducted [37], although its tightness has not been clarified. If we can derive a tight Berry–Esséen-type bound for the Markov chain, this would enable us to derive a finite-length bound that is asymptotically tight in the second-order regime. However, the approximation errors of Berry–Esséen-type bounds converge only in the order of $1/\sqrt{n}$ and cannot be applied when $\epsilon$ is rather small. Even in cases in which the exact computations of tail probabilities are possible, the information-spectrum-type bounds are looser than the exponential type bounds when $\epsilon$ is rather small, and we need to use appropriate bounds depending on the size of $\epsilon$. In fact, this observation was explicitly clarified in [38] for random number generation with side-information. Consequently, we believe that our exponential-type finite-length bounds are very useful. It should be also noted that, for source coding with side-information and channel coding for regular channels, even the first-order results have not been revealed as far as the authors know, and they are clarified in this paper (General formulae for those problems were known [2,3], but single-letter expressions for Markov sources or channels were not clarified in the literature. For the source coding without side-information, the single-letter expression for entropy rate of Markov source is well known (e.g., see [39]).).

1.5. Related Work on Markov Chains

Since related work concerning the finite-length analysis is reviewed in Section 1.1, we only review work related to the asymptotic analysis here. Some studies on Markov chains for the large deviation regime have been reported [40,41,42]. The derivation in [40] used the Markov-type method. A drawback of this method is that it involves a term that stems from the number of types, which does not affect the asymptotic analysis, but does hurt the finite-length analysis. Our achievability is derived by following a similar approach as in [41,42], i.e., the Perron–Frobenius theorem, but our derivation separates the single-shot part and the evaluation of the Rényi entropy, and thus is more transparent. Furthermore, the converse part of [41,42] is based on the Shannon–McMillan–Breiman limiting theorem and does not yield finite-length bounds.

For the second-order regime, Polyanskiy et. al. studied the second-order rate (dispersion) of the Gilbert–Elliott channel [43]. Tomamichel and Tan studied the second-order rate of channel coding with state information such that the state information may be a general source and derived a formula for the Markov chain as a special case [32]. Kontoyiannis studied the second-order variable length source coding for the Markov chain [44]. In [45], Kontoyiannis and Verdú derived the second-order rate of lossless source coding under the overflow probability criterion.

For channel coding of the i.i.d. case, Scarlett et al. derived a saddle-point approximation, which unifies all three regimes [46,47].

1.6. Organization of the Paper

In Section 2, we introduce the information measures and their properties that will be used in Section 3 and Section 4. Then, source coding with side-information and channel coding is discussed in Section 3 and Section 4, respectively. As we mentioned above, we state our main result in terms of the Rényi entropies, and we use the CGFs and the Gallager function in the proofs. We explain how to cover the continuous case in Remarks 1 and 5. In Appendix A and Appendix B, the relation between the Rényi entropies and corresponding CGFs are summarized. The relation between the Rényi entropies and the Gallager function are explained as necessary. Proofs of some technical results are also provided in the remaining Appendices.

1.7. Notations

For a set $\mathcal{X}$, the set of all distributions on $\mathcal{X}$ is denoted by $\mathcal{P}\left(\mathcal{X}\right)$. The set of all sub-normalized non-negative functions on $\mathcal{X}$ is denoted by $\overline{\mathcal{P}}\left(\mathcal{X}\right)$. The cumulative distribution function of the standard Gaussian random variable is denoted by:

$$\begin{array}{c}\hfill \mathsf{\Phi }\left(t\right)={\int }_{-\infty }^t\frac{1}{\sqrt{2\pi }}exp\left[-\frac{x^2}{2}\right]dx.\end{array}$$

(5)

Throughout the paper, the base of the logarithm is the natural base e.

2. Information Measures

Since this paper discusses the second-order tightness, we need to discuss the central limit theorem for the Markov process. For this purpose, we usually employ advanced mathematical methods from probability theory. For example, the paper [48] (Theorem 4) showed the Markov version of the central limit theorem by using a martingale stopping technique. Lalley [49] employed the regular perturbation theory of operators on the infinite-dimensional space [50] (Chapter 7, #1, Chapter 4, #3, and Chapter 3, #5). The papers [51,52] and [53] (Lemma 1.5 of Chapter 1) employed the spectral measure, while it is hard to calculate the spectral measure in general even in the finite-state case. Further, the papers [36,51,54,55] showed the central limit theorem by using the asymptotic variance, but they did not give any computable expression of the asymptotic variance without the infinite sum. In summary, to derive the central limit theorem with the variance of a computable form, these papers needed to use very advanced mathematics beyond calculus and linear algebra.

To overcome the difficulty of the Markov version of the central limit theorem, we employed the method used in our recent paper [23]. The paper [23] employed the method based on the cumulant generating function for transition matrices, which is defined by the Perron eigenvalue of a specific non-negative-entry matrix. Since a Perron eigenvalue can be explained in the framework of linear algebra, the method can be described with elementary mathematics. To employ this method, we need to define the information measure in a way similar to the cumulant generating function for transition matrices. That is, we define the information measures for transition matrices, e.g., the conditional Rényi entropy for transition matrices, etc, by using Perron eigenvalues.

Fortunately, these information measures for transition matrices are very useful even for large deviation-type evaluation and finite-length bounds. For example, our recent paper [23] derived finite-length bounds for simple hypothesis testing for the Markov chain by using the cumulant generating function for transition matrices. Therefore, using these information measures for transition matrices, this paper derives finite-length bounds for source coding and channel coding with Markov chains and discusses their asymptotic bounds with large deviation, moderate deviation, and the second-order type.

Since they are natural extensions of information measures for single-shot setting, we first review information measures for the single-shot setting in Section 2.1. Next, we introduce information measures for transition matrices in Section 2.2. Then, we show that information measures for Markov chains can be approximated by information measures for transition matrices generating those Markov chains in Section 2.3.

2.1. Information Measures for the Single-Shot Setting

In this section, we introduce conditional Rényi entropies for the single-shot setting. For more a detailed review of conditional Rényi entropies, see [21]. For a correlated random variable $(X,Y)$ on $\mathcal{X}\times \mathcal{Y}$ with probability distribution $P_{XY}$ and a marginal distribution $Q_Y$ on $\mathcal{Y}$, we introduce the conditional Rényi entropy of order $1+\theta$ relative to $Q_Y$ as:

$$\begin{array}{c}\hfill H_{1+\theta }\left(P_{XY}\right|Q_Y):=-\frac{1}{\theta }log\sum _{x,y}{P}_{XY}{(x,y)}^{1+\theta }{Q}_Y{\left(y\right)}^{-\theta },\end{array}$$

(6)

where $\theta \in (-1,0)\cup (0,\infty )$. The conditional Rényi entropy of order zero relative to $Q_Y$ is defined by the limit with respect to $\theta$. When X has no side-information, it is nothing but the ordinary Rényi entropy, and it is denoted by $H_{1+\theta }\left(X\right)=H_{1+\theta }\left(P_X\right)$ throughout the paper.

One of the important special cases of $H_{1+\theta }\left(P_{XY}\right|Q_Y)$ is the case with $Q_Y=P_Y$, where $P_Y$ is the marginal of $P_{XY}$. We shall call this special case the lower conditional Rényi entropy of order $1+\theta$ and denote (this notation was first introduced in [56]):

$$\begin{array}{ccc}\hfill H_{1+\theta }^{\downarrow }\left(X\right|Y)& :=& H_{1+\theta }\left(P_{XY}\right|P_Y)\hfill \end{array}$$

(7)

$$\begin{array}{cc}=& -\frac{1}{\theta }log\sum _{x,y}{P}_{XY}{(x,y)}^{1+\theta }{P}_Y{\left(y\right)}^{-\theta }.\hfill \end{array}$$

(8)

When we consider the second-order analysis, the variance of the entropy density plays an important role:

$$\begin{array}{c}\hfill \mathsf{V}\left(X\right|Y):=\mathrm{Var}\left[log\frac{1}{P_{X|Y}\left(X\right|Y)}\right].\end{array}$$

(9)

We have the following property, which follows from the correspondence between the conditional Rényi entropy and the cumulant generating function (cf. Appendix B).

Lemma 1.

We have:

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}{H}_{1+\theta }^{\downarrow }\left(X\right|Y)=H\left(X\right|Y)\end{array}$$

(10)

and (as seen in the proof (cf. (A26)), the left-hand side of (11) corresponds to the second derivative of the cumulant generating function):

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}\frac{2\left[H\left(X\right|Y)-H_{1+\theta }^{\downarrow }\left(X\right|Y)\right]}{\theta }=\mathsf{V}\left(X\right|Y).\end{array}$$

(11)

Proof. 

(10) follows from the relation in (A25) and the fact that the first-order derivative of the cumulant generating function is the expectation. (11) follows from (A25), (10) and (A26). □

The other important special case of $H_{1+\theta }\left(P_{XY}\right|Q_Y)$ is the measure maximized over $Q_Y$. We shall call this special case the upper conditional Rényi entropy of order $1+\theta$ and denote (Equation (13) for $-1<\theta <0$ follows from the Hölder inequality, and Equation (13) for $0<\theta$ follows from the reverse Hölder inequality [57] (Lemma 8). Similar optimization has appeared in the context of Rényi mutual information in [58] (see also [59]).):

$$\begin{array}{ccc}\hfill H_{1+\theta }^{\uparrow }\left(X\right|Y)& :=& \underset{Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)}{max}{H}_{1+\theta }\left(P_{XY}\right|Q_Y)\hfill \end{array}$$

(12)

$$\begin{array}{cc}=& H_{1+\theta }\left(P_{XY}\right|P_Y^{(1+\theta )})\hfill \end{array}$$

(13)

$$\begin{array}{cc}=& -\frac{1+\theta }{\theta }log\sum _y{P}_Y\left(y\right){\left[\sum _x{P}_{X|Y}{\left(x\right|y)}^{1+\theta }\right]}^{\frac{1}{1+\theta }},\hfill \end{array}$$

(14)

where:

$$\begin{array}{c}\hfill P_Y^{(1+\theta )}\left(y\right):=\frac{{\left[{\sum }_x{P}_{XY}{(x,y)}^{1+\theta }\right]}^{\frac{1}{1+\theta }}}{{\sum }_{y^{\prime }}{\left[{\sum }_x{P}_{XY}{(x,y^{\prime })}^{1+\theta }\right]}^{\frac{1}{1+\theta }}}.\end{array}$$

(15)

For this measure, we also have the same properties as Lemma 1. This lemma will be proven in Appendix C.

Lemma 2.

We have:

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}{H}_{1+\theta }^{\uparrow }\left(X\right|Y)=H\left(X\right|Y)\end{array}$$

(16)

and:

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}\frac{2\left[H\left(X\right|Y)-H_{1+\theta }^{\uparrow }\left(X\right|Y)\right]}{\theta }=\mathsf{V}\left(X\right|Y).\end{array}$$

(17)

When we derive converse bounds, we need to consider the case such that the order of the Rényi entropy is different from the order of conditioning distribution defined in (15). For this purpose, we introduce two-parameter conditional Rényi entropy, which connects the two kinds of conditional Rényi entropies $H_{1+\theta }^{\downarrow }\left(X\right|Y)$ and $H_{1+\theta }^{\uparrow }\left(X\right|Y)$ in the way as Statements 10 and 11 of Lemma 3:

$$\begin{array}{c}\hfill {\displaystyle H_{1+\theta ,1+{\theta }^{\prime }}\left(X\right|Y)}\end{array}$$

(18)

$$\begin{array}{cc}:=& H_{1+\theta }\left(P_{XY}\right|P_Y^{(1+{\theta }^{\prime })})\hfill \end{array}$$

(19)

$$\begin{array}{cc}=& -\frac{1}{\theta }log\sum _y{P}_Y\left(y\right)\left[\sum _x{P}_{X|Y}{\left(x\right|y)}^{1+\theta }\right]{\left[\sum _x{P}_{X|Y}{\left(x\right|y)}^{1+{\theta }^{\prime }}\right]}^{\frac{-\theta }{1+{\theta }^{\prime }}}+\frac{{\theta }^{\prime }}{1+{\theta }^{\prime }}{H}_{1+{\theta }^{\prime }}^{\uparrow }\left(X\right|Y).\hfill \end{array}$$

(20)

Next, we investigate some properties of the measures defined above, which will be proven in Appendix D.

Lemma 3.

1.

For fixed $Q_Y$, $\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)$ is a concave function of θ, and it is strict concave iff $\mathrm{Var}\left[log\frac{Q_Y\left(Y\right)}{P_{XY}(X,Y)}\right]>0$.

2.

For fixed $Q_Y$, $H_{1+\theta }\left(P_{XY}\right|Q_Y)$ is a monotonically decreasing (Technically, $H_{1+\theta }\left(P_{XY}\right|Q_Y)$ is always non-increasing, and it is monotonically decreasing iff strict concavity holds in Statement 1. Similar remarks are also applied for other information measures throughout the paper.) function of θ.

3.

The function $\theta H_{1+\theta }^{\downarrow }\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff $\mathsf{V}\left(X\right|Y)>0$.

4.

$H_{1+\theta }^{\downarrow }\left(X\right|Y)$ is a monotonically decreasing function of θ.

5.

The function $\theta H_{1+\theta }^{\uparrow }\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff $\mathsf{V}\left(X\right|Y)>0$.

6.

$H_{1+\theta }^{\uparrow }\left(X\right|Y)$ is a monotonically decreasing function of θ.

7.

For every $\theta \in (-1,0)\cup (0,\infty )$, we have $H_{1+\theta }^{\downarrow }\left(X\right|Y)\le H_{1+\theta }^{\uparrow }\left(X\right|Y)$.

8.

For fixed ${\theta }^{\prime }$, the function $\theta H_{1+\theta ,1+{\theta }^{\prime }}\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff $\mathsf{V}\left(X\right|Y)>0$.

9.

For fixed ${\theta }^{\prime }$, $H_{1+\theta ,1+{\theta }^{\prime }}\left(X\right|Y)$ is a monotonically decreasing function of θ.

10.

We have:

$$\begin{array}{c}\hfill H_{1+\theta ,1}\left(X\right|Y)=H_{1+\theta }^{\downarrow }\left(X\right|Y).\end{array}$$

(21)

11.

We have:

$$\begin{array}{c}\hfill H_{1+\theta ,1+\theta }\left(X\right|Y)=H_{1+\theta }^{\uparrow }\left(X\right|Y).\end{array}$$

(22)

12.

For every $\theta \in (-1,0)\cup (0,\infty )$, $H_{1+\theta ,1+{\theta }^{\prime }}\left(X\right|Y)$ is maximized at ${\theta }^{\prime }=\theta$.

The following lemma expresses explicit forms of the conditional Rényi entropies of order zero.

Lemma 4.

We have:

$$\begin{array}{ccc}\hfill \underset{\theta \to -1}{lim}{H}_{1+\theta }\left(P_{XY}\right|Q_Y)& =& H_0\left(P_{XY}\right|Q_Y)\hfill \end{array}$$

(23)

$$\begin{array}{cc}:=& log\sum _y{Q}_Y\left(y\right)\left|\mathrm{supp}\left(P_{X|Y}(·|y)\right)\right|,\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill \underset{\theta \to -1}{lim}{H}_{1+\theta }^{\uparrow }\left(X\right|Y)& =& H_0^{\uparrow }\left(X\right|Y)\hfill \end{array}$$

(25)

$$\begin{array}{cc}:=& log\underset{y\in \mathrm{supp}\left(P_Y\right)}{max}\left|\mathrm{supp}\left(P_{X|Y}(·|y)\right)\right|,\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill \underset{\theta \to -1}{lim}{H}_{1+\theta }^{\downarrow }\left(X\right|Y)& =& H_0^{\downarrow }\left(X\right|Y)\hfill \end{array}$$

(27)

$$\begin{array}{cc}:=& log\sum _y{P}_Y\left(y\right)\left|\mathrm{supp}\left(P_{X|Y}(·|y)\right)\right|.\hfill \end{array}$$

(28)

Proof. 

See Appendix E. □

The definition (6) guarantees the existence of the derivative of $\frac{d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]}{d\theta }$. From Statement 1 of Lemma 3, $d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]/d\theta$ is monotonically decreasing. Thus, the inverse function (Throughout the paper, the notations $\theta \left(a\right)$ and $a\left(R\right)$ are reused for several inverse functions. Although the meanings of those notations are obvious from the context, we occasionally put superscript Q, ↓ or ↑ to emphasize that those inverse functions are induced from corresponding conditional Rényi entropies. This definition is related to the Legendre transform of the concave function $\theta \mapsto \theta H_{1+\theta }^{\downarrow }\left(X\right|Y)$.) of $\theta \mapsto d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]/d\theta$ exists so that the function $\theta \left(a\right)={\theta }^Q\left(a\right)$ is defined as:

$$\begin{array}{c}\hfill \frac{d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]}{d\theta }{|}_{\theta =\theta \left(a\right)}=a\end{array}$$

(29)

for $\underline{a}<a\le \overline{a}$, where $\underline{a}={\underline{a}}^Q:={lim}_{\theta \to \infty }d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]/d\theta$ and $\overline{a}={\overline{a}}^Q:={lim}_{\theta \to -1}d\left[\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right]/d\theta$. Let:

$$\begin{array}{c}\hfill R\left(a\right)=R^Q\left(a\right):=(1+\theta \left(a\right))a-\theta \left(a\right)H_{1+\theta \left(a\right)}\left(P_{XY}\right|Q_Y).\end{array}$$

(30)

Since:

$$\begin{array}{cc}\hfill & R^{\prime }\left(a\right)=\frac{d{R}^{\prime }\left(a\right)}{da}\hfill \\ \hfill =& \frac{d\theta \left(a\right)}{da}a+1+\theta \left(a\right)-\frac{d\left(\theta H_{1+\theta }\left(P_{XY}\right|Q_Y)\right)}{d\theta }\frac{d\theta \left(a\right)}{da}\hfill \\ \hfill =& \frac{d\theta \left(a\right)}{da}a+1+\theta \left(a\right)-a\frac{d\theta \left(a\right)}{da}=1+\theta \left(a\right),\hfill \end{array}$$

(31)

$R\left(a\right)$ is a monotonic increasing function of $\underline{a}<a\le R\left(\overline{a}\right)$. Thus, we can define the inverse function $a\left(R\right)=a^Q\left(R\right)$ of $R\left(a\right)$ by:

$$\begin{array}{c}\hfill (1+\theta \left(a\left(R\right)\right))a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}\left(P_{XY}\right|Q_Y)=R\end{array}$$

(32)

for $R\left(\underline{a}\right)<R\le H_0\left(P_{XY}\right|Q_Y)$.

For $\theta H_{1+\theta }^{\downarrow }\left(X\right|Y)$, by the same reason as above, we can define the inverse functions $\theta \left(a\right)={\theta }^{\downarrow }\left(a\right)$ and $a\left(R\right)=a^{\downarrow }\left(R\right)$ by:

$$\begin{array}{c}\hfill \frac{d\left[\theta H_{1+\theta }^{\downarrow }\left(X\right|Y)\right]}{d\theta }{|}_{\theta =\theta \left(a\right)}=a\end{array}$$

(33)

and:

$$\begin{array}{c}\hfill (1+\theta \left(a\left(R\right)\right))a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow }\left(X\right|Y)=R,\end{array}$$

(34)

for $R\left(\underline{a}\right)<R\le H_0^{\downarrow }\left(X\right|Y)$. For $\theta H_{1+\theta }^{\uparrow }\left(X\right|Y)$, we also introduce the inverse functions $\theta \left(a\right)={\theta }^{\uparrow }\left(a\right)$ and $a\left(R\right)=a^{\uparrow }\left(R\right)$ by:

$$\begin{array}{c}\hfill \frac{d\theta H_{1+\theta }^{\uparrow }\left(X\right|Y)}{d\theta }{|}_{\theta =\theta \left(a\right)}=a\end{array}$$

(35)

and:

$$\begin{array}{c}\hfill (1+\theta \left(a\left(R\right)\right))a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow }\left(X\right|Y)=R\end{array}$$

(36)

for $R\left(\underline{a}\right)<R\le H_0^{\uparrow }\left(X\right|Y)$.

Remark 1.

Here, we discuss the possibility for extension to the continuous case. Since the entropy in the continuous case diverges, we cannot extend the information quantities to the case when $\mathcal{X}$ is continuous. However, it is possible to extend these quantities to the case when $\mathcal{Y}$ is continuous, but $\mathcal{X}$ is a discrete finite set. In this case, we prepare a general measure μ (like the Lebesgue measure) on $\mathcal{Y}$ and probability density function $p_Y$ and $q_Y$ such that the distributions $P_Y$ and $Q_Y$ are given as $p_Y\left(y\right)\mu \left(dy\right)$ and $q_Y\left(y\right)\mu \left(dy\right)$, respectively. Then, it is sufficient to replace ∑, $Q\left(y\right)$, and $P_{XY}(x,y)$ by ${\int }_{\mathcal{Y}}\mu \left(dy\right)$, $P_{X|Y}\left(x\right|y)p_Y\left(y\right)$, and $q_Y\left(y\right)$, respectively. Hence, in the n-independent and identically distributed case, these information measures are given as n times the original information measures.

One might consider the information quantities for transition matrices given in the next subsection for this continuous case. However, this is not so easy because it needs a continuous extension of the Perron eigenvalue.

2.2. Information Measures for the Transition Matrix

Let ${\left\{W(x,y|x^{\prime },y^{\prime })\right\}}_{((x,y),(x^{\prime },y^{\prime }))\in {(\mathcal{X}\times \mathcal{Y})}^2}$ be an ergodic and irreducible transition matrix. The purpose of this section is to introduce transition matrix counterparts of those measures in Section 2.1. For this purpose, we first need to introduce some assumptions on transition matrices:

Assumption 1

(Non-hidden). We say that a transition matrix W is non-hidden (with respect to Y) if the Y-marginal process is a Markov process, i.e., (The reason for the name “non-hidden” is the following. In general, the Y-marginal process is a hidden Markov process. However, when the condition (37) holds, the Y-marginal process is a Markov process. Hence, we call the condition (37) non-hidden.):

$$\begin{array}{c}\hfill \sum _{x}W(x,y|x^{\prime },y^{\prime })=W\left(y\right|y^{\prime })\end{array}$$

(37)

for every $x^{\prime }\in \mathcal{X}$ and $y,y^{\prime }\in \mathcal{Y}$. This condition is equivalent to the existence of the following decomposition of $W(x,y|x^{\prime },y^{\prime })$:

$$\begin{array}{c}\hfill W(x,y|x^{\prime },y^{\prime })=W\left(y\right|y^{\prime })W\left(x\right|x^{\prime },y^{\prime },y).\end{array}$$

(38)

Assumption 2

(Strongly non-hidden). We say that a transition matrix W is strongly non-hidden (with respect to Y) if, for every $\theta \in (-1,\infty )$ and $y,y^{\prime }\in \mathcal{Y}$ (The reason for the name “strongly non-hidden” is the following. When we compute the upper conditional Rényi entropy rate of the Markov source, the effect of the Y process may propagate infinitely even if it is non-hidden. When (39) holds, the effect of the Y process in the computation of the upper conditional Rényi entropy rate is only one step.):

$$\begin{array}{c}\hfill W_{\theta }\left(y\right|y^{\prime }):=\sum _{x}W{(x,y|x^{\prime },y^{\prime })}^{1+\theta }\end{array}$$

(39)

is well defined, i.e., the right-hand side of (39) is independent of $x^{\prime }$.

Assumption 1 requires (39) to hold only for $\theta =0$, and thus, Assumption 2 implies Assumption 1. However, Assumption 2 is a strictly stronger condition than Assumption 1. For example, let us consider the case such that the transition matrix is a product form, i.e., $W(x,y|x^{\prime },y^{\prime })=W\left(x\right|x^{\prime })W\left(y\right|y^{\prime })$. In this case, Assumption 1 is obviously satisfied. However, Assumption 2 is not satisfied in general.

Assumption 2 has another expression as follows.

Lemma 5.

Assumption 2 holds if and only if, for every $x^{\prime }\ne {\tilde{x}}^{\prime }$, there exists a permutation ${\pi }_{x^{\prime };{\tilde{x}}^{\prime }}$ on $\mathcal{X}$ such that $W\left(x\right|x^{\prime },y^{\prime },y)=W\left({\pi }_{x^{\prime };{\tilde{x}}^{\prime }}\left(x\right)\right|{\tilde{x}}^{\prime },y^{\prime },y)$.

Proof. 

Since the part “if” is trivial, we show the part “only if” as follows. By noting (38), Assumption 2 can be rephrased as:

$$\begin{array}{c}\hfill \sum _{x}W{\left(x\right|x^{\prime },y^{\prime },y)}^{1+\theta }\end{array}$$

(40)

does not depend on $x^{\prime }$ for every $\theta \in (-1,\infty )$. Furthermore, this condition can be rephrased as follows. For $x^{\prime }\ne {\tilde{x}}^{\prime }$, if the largest values of ${\left\{W\left(x\right|x^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$ and ${\left\{W\left(x\right|{\tilde{x}}^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$ are different, say the former is larger, then ${\sum }_{x}W{\left(x\right|x^{\prime },y^{\prime })}^{1+\theta }>{\sum }_{x}W{\left(x\right|{\tilde{x}}^{\prime },y^{\prime })}^{1+\theta }$ for sufficiently large $\theta$, which contradicts the fact that (40) does not depend on $x^{\prime }$. Thus, the largest values of ${\left\{W\left(x\right|x^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$ and ${\left\{W\left(x\right|{\tilde{x}}^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$ must coincide. By repeating this argument for the second largest value of ${\left\{W\left(x\right|x^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$ and ${\left\{W\left(x\right|{\tilde{x}}^{\prime },y^{\prime })\right\}}_{x\in \mathcal{X}}$, and so on, we find that Assumption 2 implies that for every $x^{\prime }\ne {\tilde{x}}^{\prime }$, there exists a permutation ${\pi }_{x^{\prime };{\tilde{x}}^{\prime }}$ on $\mathcal{X}$ such that $W\left(x\right|x^{\prime },y^{\prime },y)=W\left({\pi }_{x^{\prime };{\tilde{x}}^{\prime }}\left(x\right)\right|{\tilde{x}}^{\prime },y^{\prime },y)$. □

Now, we fix an element $x_0\in \mathcal{X}$ and transform a sequence of random numbers $(X_1,Y_1,X_2,Y_2,\dots ,X_n,Y_n)$ to the sequence of random numbers $(X_1^{\prime },Y_1^{\prime },X_2^{\prime },Y_2^{\prime },\dots ,X_n^{\prime },Y_n^{\prime }):=(X_1,Y_1,{\pi }_{x_0;X_1}^{-1}\left(X_2\right),Y_2,\dots ,{\pi }_{x_0;X_1}^{-1}\left(X_n\right),Y_n)$. Then, letting $W^{\prime }\left(x\right|y^{\prime },y):=W\left(x\right|x_0,y^{\prime },y)$, we have $P_{X_i^{\prime },Y_i^{\prime }|X_{i-1}^{\prime },Y_{i-1}^{\prime }}=W^{\prime }\left(y_i^{\prime }\right|y_{i-1}^{\prime })W\left(x_i^{\prime }\right|y_i^{\prime },y_{i-1}^{\prime })$. That is, essentially, the transition matrix of this case can be written by the transition matrix $W\left(y_i^{\prime }\right|y_{i-1}^{\prime })W^{\prime }\left(x_i^{\prime }\right|y_i^{\prime },y_{i-1}^{\prime })$. Therefore, the transition matrix can be written by using the positive-entry matrix $W_{x_i^{\prime }}\left(y_i^{\prime }\right|y_{i-1}^{\prime }):=W\left(y_i^{\prime }\right|y_{i-1}^{\prime })W^{\prime }\left(x_i^{\prime }\right|y_i^{\prime },y_{i-1}^{\prime })$.

The following are non-trivial examples satisfying Assumptions 1 and 2.

Example 1.

Suppose that $\mathcal{X}=\mathcal{Y}$ is a module (an additive group). Let P and Q be transition matrices on $\mathcal{X}$. Then, the transition matrix given by:

$$\begin{array}{c}\hfill W(x,y|x^{\prime },y^{\prime })=Q\left(y\right|y^{\prime })P(x-y|x^{\prime }-y^{\prime })\end{array}$$

(41)

satisfies Assumption 1. Furthermore, if transition matrix $P\left(z\right|z^{\prime })$ can be written as:

$$\begin{array}{c}\hfill P\left(z\right|z^{\prime })=P_Z\left({\pi }_{z^{\prime }}\left(z\right)\right)\end{array}$$

(42)

for permutation ${\pi }_{z^{\prime }}$ and a distribution $P_Z$ on $\mathcal{X}$, then transition matrix W defined by (41) satisfies Assumption 2 as well.

Example 2.

Suppose that $\mathcal{X}$ is a module and W is (strongly) non-hidden with respect to $\mathcal{Y}$. Let Q be a transition matrix on $\mathcal{Z}=\mathcal{X}$. Then, the transition matrix given by:

$$\begin{array}{c}\hfill V(x,y,z|x^{\prime },y^{\prime },z^{\prime })=W(x-z,y|x^{\prime }-z^{\prime },y)Q\left(z\right|z^{\prime })\end{array}$$

(43)

is (strongly) non-hidden with respect to $\mathcal{Y}\times \mathcal{Z}$.

The following is also an example satisfying Assumption 2, which describes a noise process of an important class of channels with memory (cf. the Gilbert-Elliot channel in Example 6).

Example 3.

Let $\mathcal{X}=\mathcal{Y}=\{0,1\}$. Then, let:

$$\begin{array}{c}\hfill W\left(y\right|y^{\prime })=\left\{\begin{array}{cc}1-q_{y^{\prime }}\hfill & ify=y^{\prime }\hfill \\ q_{y^{\prime }}\hfill & ify\ne y^{\prime }\hfill \end{array}\right.\end{array}$$

(44)

for some $0<q_0,q_1<1$, and let:

$$\begin{array}{c}\hfill W\left(x\right|x^{\prime },y^{\prime },y)=\left\{\begin{array}{cc}1-p_y\hfill & ifx=0\hfill \\ p_y\hfill & ifx=1\hfill \end{array}\right.\end{array}$$

(45)

for some $0<p_0,p_1<1$. By choosing ${\pi }_{x^{\prime };{\tilde{x}}^{\prime }}$ to be the identity, this transition matrix satisfies the condition given in Remark 5, which is equivalent to Assumption 2.

First, we introduce information measures under Assumption 1. In order to define a transition matrix counterpart of (7), let us introduce the following tilted matrix:

$$\begin{array}{c}\hfill {\tilde{W}}_{\theta }(x,y|x^{\prime },y^{\prime }):=W{(x,y|x^{\prime },y^{\prime })}^{1+\theta }W{\left(y\right|y^{\prime })}^{-\theta }.\end{array}$$

(46)

Here, we should notice that the tilted matrix ${\tilde{W}}_{\theta }$ is not normalized, i.e., is not a transition matrix. Let ${\lambda }_{\theta }$ be the Perron–Frobenius eigenvalue of ${\tilde{W}}_{\theta }$ and ${\tilde{P}}_{\theta ,XY}$ be its normalized eigenvector. Then, we define the lower conditional Rényi entropy for W by:

$$\begin{array}{c}\hfill H_{1+\theta }^{\downarrow ,W}\left(X\right|Y):=-\frac{1}{\theta }log{\lambda }_{\theta },\end{array}$$

(47)

where $\theta \in (-1,0)\cup (0,\infty )$. For $\theta =0$, we define the lower conditional Rényi entropy for W by:

$$\begin{array}{ccc}\hfill H^W\left(X\right|Y)& =& H_1^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(48)

$$\begin{array}{cc}:=& \underset{\theta \to 0}{lim}{H}_{1+\theta }^{\downarrow ,W}\left(X\right|Y),\hfill \end{array}$$

(49)

and we just call it the conditional entropy for W. In fact, the definition of $H^W\left(X\right|Y)$ above coincides with:

$$\begin{array}{c}\hfill -\sum _{x^{\prime },y^{\prime }}{P}_{0,XY}(x^{\prime },y^{\prime })\sum _{x,y}W(x,y|x^{\prime },y^{\prime })log\frac{W(x,y|x^{\prime },y^{\prime })}{W\left(y\right|y^{\prime })},\end{array}$$

(50)

where $P_{0,XY}$ is the stationary distribution of W (cf. [60] (Equation (30))). For $\theta =-1$, $H_0^{\downarrow ,W}\left(X\right|Y)$ is also defined by taking the limit. When X has no side-information, the Rényi entropy $H_{1+\theta }^W\left(X\right)$ for W is defined as a special case of $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$.

As a counterpart of (11), we also define (Since the limiting expression in (51) coincides with the second derivative of the CGF (cf. (A30)) and since the second derivative of the CGF exists (cf. [22] (Appendix D)), the variance in (51) is well defined. While the definition (51) contains the limit $\theta \to 0$, it can be calculated without this type of limit by using the fundamental matrix [61] (Theorem 4.3.1), [23] (Theorem 7.7 and Remark 7.8).):

$$\begin{array}{c}\hfill {\mathsf{V}}^W\left(X\right|Y):=\underset{\theta \to 0}{lim}\frac{2\left[H^W\left(X\right|Y)-H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]}{\theta }.\end{array}$$

(51)

Remark 2.

When transition matrix W satisfies Assumption 2, $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ can be written as:

$$\begin{array}{c}\hfill H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)=-\frac{1}{\theta }log{\lambda }_{\theta }^{\prime },\end{array}$$

(52)

where ${\lambda }_{\theta }^{\prime }$ is the Perron–Frobenius eigenvalue of $W_{\theta }\left(y\right|y^{\prime })W{\left(y\right|y^{\prime })}^{-\theta }$. In fact, for the left Perron–Frobenius eigenvector ${\widehat{Q}}_{\theta }$ of $W_{\theta }\left(y\right|y^{\prime })W{\left(y\right|y^{\prime })}^{-\theta }$, we have:

$$\begin{array}{c}\hfill \sum _{x,y}{\widehat{Q}}_{\theta }\left(y\right)W{(x,y|x^{\prime },y^{\prime })}^{1+\theta }W{\left(y\right|y^{\prime })}^{-\theta }={\lambda }_{\theta }^{\prime }{Q}_{\theta }\left(y^{\prime }\right),\end{array}$$

(53)

which implies that ${\lambda }_{\theta }^{\prime }$ is the Perron–Frobenius eigenvalue of ${\tilde{W}}_{\theta }$. Consequently, we can evaluate $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ by calculating the Perron–Frobenius eigenvalue of the $\left|\mathcal{Y}\right|\times \left|\mathcal{Y}\right|$ matrix instead of the $\left|\mathcal{X}\right|\left|\mathcal{Y}\right|\times \left|\mathcal{X}\right|\left|\mathcal{Y}\right|$ matrix when W satisfies Assumption 2.

Next, we introduce information measures under Assumption 2. In order to define a transition matrix counterpart of (12), let us introduce the following $\left|\mathcal{Y}\right|\times \left|\mathcal{Y}\right|$ matrix:

$$\begin{array}{c}\hfill K_{\theta }\left(y\right|y^{\prime }):=W_{\theta }{\left(y\right|y^{\prime })}^{\frac{1}{1+\theta }},\end{array}$$

(54)

where $W_{\theta }$ is defined by (39). Let ${\kappa }_{\theta }$ be the Perron–Frobenius eigenvalue of $K_{\theta }$. Then, we define the upper conditional Rényi entropy for W by:

$$\begin{array}{c}\hfill H_{1+\theta }^{\uparrow ,W}\left(X\right|Y):=-\frac{1+\theta }{\theta }log{\kappa }_{\theta },\end{array}$$

(55)

where $\theta \in (-1,0)\cup (0,\infty )$. For $\theta =-1$ and $\theta =0$, $H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$ is defined by taking the limit. We have the following properties, which will be proven in Appendix F.

Lemma 6.

We have:

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}{H}_{1+\theta }^{\uparrow ,W}\left(X\right|Y)=H^W\left(X\right|Y)\end{array}$$

(56)

and:

$$\begin{array}{c}\hfill \underset{\theta \to 0}{lim}\frac{2\left[H^W\left(X\right|Y)-H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)\right]}{\theta }={\mathsf{V}}^W\left(X\right|Y).\end{array}$$

(57)

Now, let us introduce a transition matrix counterpart of (18). For this purpose, we introduce the following $\left|\mathcal{Y}\right|\times \left|\mathcal{Y}\right|$ matrix:

$$\begin{array}{c}\hfill N_{\theta ,{\theta }^{\prime }}\left(y\right|y^{\prime }):=W_{\theta }\left(y\right|y^{\prime })W_{{\theta }^{\prime }}{\left(y\right|y^{\prime })}^{\frac{-\theta }{1+{\theta }^{\prime }}}.\end{array}$$

(58)

Let ${\nu }_{\theta ,{\theta }^{\prime }}$ be the Perron–Frobenius eigenvalue of $N_{\theta ,{\theta }^{\prime }}$. Then, we define the two-parameter conditional Rényi entropy by:

$$\begin{array}{c}\hfill H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y):=-\frac{1}{\theta }log{\nu }_{\theta ,{\theta }^{\prime }}+\frac{{\theta }^{\prime }}{1+{\theta }^{\prime }}{H}_{1+{\theta }^{\prime }}^{\uparrow ,W}\left(X\right|Y).\end{array}$$

(59)

Remark 3.

Although we defined $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ and $H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$ by (47) and (55), respectively, we can alternatively define these measures in the same spirit as the single-shot setting by introducing a transition matrix counterpart of $H_{1+\theta }\left(P_{XY}\right|Q_Y)$ as follows. For the marginal $W\left(y\right|y^{\prime })$ of $W(x,y|x^{\prime },y^{\prime })$, let ${\mathcal{Y}}_W^2:=\{(y,y^{\prime }):W\left(y\right|y^{\prime })>0\}$. For another transition matrix V on $\mathcal{Y}$, we define ${\mathcal{Y}}_V^2$ in a similar manner. For V satisfying ${\mathcal{Y}}_W^2\subset {\mathcal{Y}}_V^2$, we define (although we can also define $H_{1+\theta }^{W|V}\left(X\right|Y)$ even if ${\mathcal{Y}}_W^2\subset {\mathcal{Y}}_V^2$ is not satisfied (see [22] for the detail), for our purpose of defining $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ and $H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$, other cases are irrelevant):

$$\begin{array}{c}\hfill H_{1+\theta }^{W|V}\left(X\right|Y):=-\frac{1}{\theta }log{\lambda }_{\theta }^{W|V}\end{array}$$

(60)

for $\theta \in (-1,0)\cup (0,\infty )$, where ${\lambda }_{\theta }^{W|V}$ is the Perron–Frobenius eigenvalue of:

$$\begin{array}{c}\hfill W{(x,y|x^{\prime },y^{\prime })}^{1+\theta }V{\left(y\right|y^{\prime })}^{-\theta }.\end{array}$$

(61)

By using this measure, we obviously have:

$$\begin{array}{c}\hfill H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)=H_{1+\theta }^{W|W}\left(X\right|Y).\end{array}$$

(62)

Furthermore, under Assumption 2, the relation:

$$\begin{array}{c}\hfill H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)=\underset{V}{max}{H}_{1+\theta }^{W|V}\left(X\right|Y)\end{array}$$

(63)

holds (see Appendix G for the proof), where the maximum is taken over all transition matrices satisfying ${\mathcal{Y}}_W^2\subset {\mathcal{Y}}_V^2$.

Next, we investigate some properties of the information measures introduced in this section. The following lemma is proven in Appendix H.

Lemma 7.

1.

The function $\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff ${\mathsf{V}}^W\left(X\right|Y)>0$.

2.

$H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ is a monotonically decreasing function of θ.

3.

The function $\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff ${\mathsf{V}}^W\left(X\right|Y)>0$.

4.

$H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$ is a monotonically decreasing function of θ.

5.

For every $\theta \in (-1,0)\cup (0,\infty )$, we have $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\le H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$.

6.

For fixed ${\theta }^{\prime }$, the function $\theta H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)$ is a concave function of θ, and it is strict concave iff ${\mathsf{V}}^W\left(X\right|Y)>0$.

7.

For fixed ${\theta }^{\prime }$, $H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)$ is a monotonically decreasing function of θ.

8.

We have:

$$\begin{array}{c}\hfill H_{1+\theta ,1}^W\left(X\right|Y)=H_{1+\theta }^{\downarrow ,W}\left(X\right|Y).\end{array}$$

(64)

9.

We have:

$$\begin{array}{c}\hfill H_{1+\theta ,1+\theta }^W\left(X\right|Y)=H_{1+\theta }^{\uparrow ,W}\left(X\right|Y).\end{array}$$

(65)

10.

For every $\theta \in (-1,0)\cup (0,\infty )$, $H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)$ is maximized at ${\theta }^{\prime }=\theta$, i.e.,

$$\begin{array}{c}\hfill \frac{d\left[H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)\right]}{d{\theta }^{\prime }}{|}_{{\theta }^{\prime }=\theta }=0.\end{array}$$

(66)

From Statement 1 of Lemma 7, $d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]/d\theta$ is monotonically decreasing. Thus, we can define the inverse function ${\theta }^W\left(a\right)={\theta }^{\downarrow ,W}\left(a\right)$ of $d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]/d\theta$ by:

$$\begin{array}{c}\hfill \frac{d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]}{d\theta }{|}_{\theta ={\theta }^W\left(a\right)}=a\end{array}$$

(67)

for $\underline{a}<a\le \overline{a}$, where $\underline{a}:={lim}_{\theta \to \infty }d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]/d\theta$ and $\overline{a}:={lim}_{\theta \to -1}d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]/d\theta$. Let:

$$\begin{array}{c}\hfill R^W\left(a\right):=(1+\theta \left(a\right))a-\theta \left(a\right)H_{1+\theta \left(a\right)}^{\downarrow ,W}\left(X\right|Y).\end{array}$$

(68)

Since

$$\begin{array}{c}\hfill {R^W}^{\prime }\left(a\right)=(1+\theta \left(a\right)),\end{array}$$

(69)

$R^W\left(a\right)$ is a monotonic increasing function of $\underline{a}<a<R^W\left(\overline{a}\right)$. Thus, we can define the inverse function $a^W\left(R\right)=a^{\downarrow ,W}\left(R\right)$ of $R^W\left(a\right)$ by:

$$\begin{array}{c}\hfill (1+\theta \left(a^W\left(R\right)\right))a^W\left(R\right)-{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\downarrow ,W}\left(X\right|Y)=R\end{array}$$

(70)

for $R^W\left(\underline{a}\right)<R<H_0^{\downarrow ,W}\left(X\right|Y)$, where $H_0^{\downarrow ,W}\left(X\right|Y):={lim}_{\theta \to -1}{H}_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$.

For $\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$, by the same reason, we can define the inverse function ${\theta }^W\left(a\right)={\theta }^{\uparrow ,W}\left(a\right)$ by:

$$\begin{array}{c}\hfill \frac{d\left[\theta H_{1+\theta ,1+{\theta }^W\left(a\right)}^W\left(X\right|Y)\right]}{d\theta }{|}_{\theta ={\theta }^W\left(a\right)}=\frac{d\left[\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)\right]}{d\theta }{|}_{\theta ={\theta }^W\left(a\right)}=a,\end{array}$$

(71)

and the inverse function $a^W\left(R\right)=a^{\uparrow ,W}\left(R\right)$ of:

$$\begin{array}{c}\hfill R^W\left(a\right):=(1+{\theta }^W\left(a\right))a-{\theta }^W\left(a\right)H_{1+{\theta }^W\left(a\right)}^{\uparrow ,W}\left(X\right|Y)\end{array}$$

(72)

by:

$$\begin{array}{c}\hfill (1+{\theta }^W\left(a^W\left(R\right)\right))a^W\left(R\right)-{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\uparrow ,W}\left(X\right|Y)=R,\end{array}$$

(73)

for $R\left(\underline{a}\right)<R<H_0^{\uparrow ,W}\left(X\right|Y)$, where $H_0^{\uparrow ,W}\left(X\right|Y):={lim}_{\theta \to -1}{H}_{1+\theta }^{\uparrow ,W}\left(X\right|Y)$. Here, the first equality in (71) follows from (66).

Since $\theta \mapsto \theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ is concave, the supremum of $[-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)]$ is attained at the stationary point. Furthermore, note that $-1\le {\theta }^{\downarrow ,W}\left(R\right)\le 0$ for $H^W\left(X\right|Y)\le R\le H_0^{\downarrow ,W}\left(X\right|Y)$. Thus, we have the following property.

Lemma 8.

The function ${\theta }^W\left(R\right)$ defined in (67) satisfies:

$$\begin{array}{c}\hfill \underset{-1\le \theta \le 0}{sup}[-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)]=-{\theta }^W\left(R\right)R+{\theta }^W\left(R\right)H_{1+{\theta }^W\left(R\right)}^{\downarrow ,W}\left(X\right|Y)\end{array}$$

(74)

for $H^W\left(X\right|Y)\le R\le H_0^{\downarrow ,W}\left(X\right|Y)$.

Furthermore, we have the following characterization for another type of maximization.

Lemma 9.

The function ${\theta }^W\left(a^W\left(R\right)\right)$ defined by (70) satisfies:

$$\begin{array}{c}\hfill \underset{-1\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)}{1+\theta }=-{\theta }^W\left(a^W\left(R\right)\right)a^W\left(R\right)+{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\downarrow ,W}\left(X\right|Y)\end{array}$$

(75)

for $H^W\left(X\right|Y)\le R\le H_0^{\downarrow ,W}\left(X\right|Y)$, and the function $\theta \left(a\right(R\left)\right)$ defined in (73) satisfies:

$$\begin{array}{c}\hfill \underset{-1\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }=-{\theta }^W\left(a^W\left(R\right)\right)a^W\left(R\right)+{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\uparrow ,W}\left(X\right|Y)\end{array}$$

(76)

for $H^W\left(X\right|Y)\le R\le H_0^{\uparrow ,W}\left(X\right|Y)$.

Proof. 

See Appendix I. □

Remark 4.

The combination of (49), (51), and Lemma 6 guarantees that both the conditional Rényi entropies expand as:

$$\begin{array}{ccc}\hfill H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)& =& H^W\left(X\right|Y)-\frac{1}{2}{\mathsf{V}}^W\left(X\right|Y)\theta +o\left(\theta \right),\hfill \end{array}$$

(77)

$$\begin{array}{ccc}\hfill H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)& =& H^W\left(X\right|Y)-\frac{1}{2}{\mathsf{V}}^W\left(X\right|Y)\theta +o\left(\theta \right)\hfill \end{array}$$

(78)

around $\theta =0$. Thus, the difference of these measures significantly appears only when $\left|\theta \right|$ is rather large. For the transition matrix of Example 3 with $q_0=q_1=0.1$, $p_0=0.1$, and $p_1=0.4$, we plotted the values of the information measures in Figure 1. Although the values at $\theta =-1$ coincide in Figure 1, note that the values at $\theta =-1$ may differ in general.

In Example 1, we mentioned that the transition matrix W in (41) satisfies Assumption 2 when transition matrix P is given by (42). By computing the conditional Rényi entropies for this special case, we have:

$$\begin{array}{ccc}\hfill H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)& =& H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(79)

$$\begin{array}{cc}=& H_{1+\theta }\left(P_Z\right),\hfill \end{array}$$

(80)

i.e., the two kinds of conditional Rényi entropies coincide.

Now, let us consider the asymptotic behavior of $H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)$ around $\theta =0$. When $\theta \left(a\right)$ is close to zero, we have:

$$\begin{array}{c}\hfill {\theta }^W\left(a\right)H_{1+{\theta }^W\left(a\right)}^{\downarrow ,W}\left(X\right|Y)={\theta }^W\left(a\right)H^W\left(X\right|Y)-\frac{1}{2}{\mathsf{V}}^W\left(X\right|Y){\theta }^W{\left(a\right)}^2+o\left({\theta }^W{\left(a\right)}^2\right).\end{array}$$

(81)

Taking the derivative, (67) implies that:

$$\begin{array}{c}\hfill a=H^W\left(X\right|Y)-{\mathsf{V}}^W\left(X\right|Y){\theta }^W\left(a\right)+o\left({\theta }^W\left(a\right)\right).\end{array}$$

(82)

Hence, when R is close to $H^W\left(X\right|Y)$, we have:

$$\begin{array}{cc}\hfill R& =(1+{\theta }^W\left(a^W\left(R\right)\right)a^W\left(R\right)-{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(83)

$$\begin{array}{cc}\hfill & =H^W\left(X\right|Y)-\left(1+\frac{{\theta }^W\left(a^W\left(R\right)\right)}{2}\right){\theta }^W\left(a^W\left(R\right)\right){\mathsf{V}}^W\left(X\right|Y)+o({\theta }^W\left(a^W\left(R\right)\right),\hfill \end{array}$$

(84)

i.e.,

$$\begin{array}{c}\hfill {\theta }^W\left(a^W\left(R\right)\right)=\frac{-R+H^W\left(X\right|Y)}{{\mathsf{V}}^W\left(X\right|Y)}+o\left(\frac{R-H^W\left(X\right|Y)}{{\mathsf{V}}^W\left(X\right|Y)}\right).\end{array}$$

(85)

Furthermore, (81) and (82) imply:

$$\begin{array}{cc}\hfill & -{\theta }^W\left(a^W\left(R\right)\right)a^W\left(R\right)+{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(86)

$$\begin{array}{cc}\hfill & ={\mathsf{V}}^W\left(X\right|Y)\frac{{\theta }^W{\left(a^W\left(R\right)\right)}^2}{2}+o\left({\theta }^W{\left(a^W\left(R\right)\right)}^2\right)\hfill \end{array}$$

(87)

$$\begin{array}{cc}\hfill & =\frac{{\mathsf{V}}^W\left(X\right|Y)}{2}{\left(\frac{R-H^W\left(X\right|Y)}{{\mathsf{V}}^W\left(X\right|Y)}\right)}^2+o\left({\left(\frac{R-H^W\left(X\right|Y)}{{\mathsf{V}}^W\left(X\right|Y)}\right)}^2\right).\hfill \end{array}$$

(88)

2.3. Information Measures for the Markov Chain

Let $(\mathbf{X},\mathbf{Y})$ be the Markov chain induced by transition matrix W and some initial distribution $P_{X_1{Y}_1}$. Now, we show how information measures introduced in Section 2.2 are related to the conditional Rényi entropy rates. First, we introduce the following lemma, which gives finite upper and lower bounds on the lower conditional Rényi entropy.

Lemma 10.

Suppose that transition matrix W satisfies Assumption 1. Let $v_{\theta }$ be the eigenvector of $W_{\theta }^T$ with respect to the Perron–Frobenius eigenvalue ${\lambda }_{\theta }$ such that ${min}_{x,y}{v}_{\theta }(x,y)=1$ (since the eigenvector corresponding to the Perron–Frobenius eigenvalue for an irreducible non-negative matrix has always strictly positive entries [62] (Theorem 8.4.4, p. 508), we can choose the eigenvector $v_{\theta }$ satisfying this condition). Let $w_{\theta }(x,y):=P_{X_1{Y}_1}{(x,y)}^{1+\theta }{P}_{Y_1}{\left(y\right)}^{-\theta }$. Then, for every $n\ge 1$, we have:

$$\begin{array}{c}\hfill (n-1)\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)+\underline{\delta }\left(\theta \right)\le \theta H_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)\le (n-1)\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)+\overline{\delta }\left(\theta \right),\end{array}$$

(89)

where:

$$\begin{array}{ccc}\hfill \overline{\delta }\left(\theta \right)& :=& -log\langle v_{\theta }|w_{\theta }\rangle +log\underset{x,y}{max}{v}_{\theta }(x,y),\hfill \end{array}$$

(90)

$$\begin{array}{ccc}\hfill \underline{\delta }\left(\theta \right)& :=& -log\langle v_{\theta }|w_{\theta }\rangle ,\hfill \end{array}$$

(91)

and $\langle v_{\theta }|w_{\theta }\rangle$ is defined as ${\sum }_{x,y}{v}_{\theta }(x,y)w_{\theta }(x,y)$.

Proof. 

This follows from (A29) and Lemma A2. □

From Lemma 10, we have the following.

Theorem 1.

Suppose that transition matrix W satisfies Assumption 1. For any initial distribution, we have (When there is no side-information, (93) reduces to the well-known expression of the entropy rate of the Markov process [39]. Without Assumption 1, it is not clear if (93) holds or not.):

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}{H}_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)& =& H_{1+\theta }^{\downarrow ,W}\left(X\right|Y),\hfill \end{array}$$

(92)

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}H\left(X^n\right|Y^n)& =& H^W\left(X\right|Y).\hfill \end{array}$$

(93)

We also have the following asymptotic evaluation of the variance, which follows from Lemma A3 in Appendix A.

Theorem 2.

Suppose that transition matrix W satisfies Assumption 1. For any initial distribution, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}\mathsf{V}\left(X^n\right|Y^n)={\mathsf{V}}^W\left(X\right|Y).\end{array}$$

(94)

Theorem 2 is practically important since the limit of the variance can be described by a single-letter characterized quantity. A method to calculate ${\mathsf{V}}^W\left(X\right|Y)$ can be found in [23].

Next, we show the lemma that gives the finite upper and lower bounds on the upper conditional Rényi entropy in terms of the upper conditional Rényi entropy for the transition matrix.

Lemma 11.

Suppose that transition matrix W satisfies Assumption 2. Let $v_{\theta }$ be the eigenvector of $K_{\theta }^T$ with respect to the Perron–Frobenius eigenvalue ${\kappa }_{\theta }$ such that ${min}_y{v}_{\theta }\left(y\right)=1$. Let $w_{\theta }$ be the $\left|\mathcal{Y}\right|$-dimensional vector defined by:

$$\begin{array}{c}\hfill w_{\theta }\left(y\right):={\left[\sum _x{P}_{X_1{Y}_1}{(x,y)}^{1+\theta }\right]}^{\frac{1}{1+\theta }}.\end{array}$$

(95)

Then, we have:

$$\begin{array}{c}\hfill (n-1)\frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow ,W}\left(X\right|Y)+\underline{\xi }\left(\theta \right)\le \frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)\le (n-1)\frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow ,W}\left(X\right|Y)+\overline{\xi }\left(\theta \right),\end{array}$$

(96)

where:

$$\begin{array}{ccc}\hfill \overline{\xi }\left(\theta \right)& :=& -log\langle v_{\theta }|w_{\theta }\rangle +log\underset{y}{max}{v}_{\theta }\left(y\right),\hfill \end{array}$$

(97)

$$\begin{array}{ccc}\hfill \underline{\xi }\left(\theta \right)& :=& -log\langle v_{\theta }|w_{\theta }\rangle .\hfill \end{array}$$

(98)

Proof. 

See Appendix J. □

From Lemma 11, we have the following.

Theorem 3.

Suppose that transition matrix W satisfies Assumption 2. For any initial distribution, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}{H}_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)& =& H_{1+\theta }^{\uparrow ,W}\left(X\right|Y).\hfill \end{array}$$

(99)

Finally, we show the lemma that gives the finite upper and lower bounds on the two-parameter conditional Rényi entropy in terms of the two-parameter conditional Rényi entropy for the transition matrix.

Lemma 12.

Suppose that transition matrix W satisfies Assumption 2. Let $v_{\theta ,{\theta }^{\prime }}$ be the eigenvector of $N_{\theta ,{\theta }^{\prime }}^T$ with respect to the Perron–Frobenius eigenvalue ${\nu }_{\theta ,{\theta }^{\prime }}$ such that ${min}_y{v}_{\theta ,{\theta }^{\prime }}\left(y\right)=1$. Let $w_{\theta ,{\theta }^{\prime }}$ be the $\left|\mathcal{Y}\right|$-dimensional vector defined by:

$$\begin{array}{c}\hfill w_{\theta ,{\theta }^{\prime }}\left(y\right):=\left[\sum _x{P}_{X_1{Y}_1}{(x,y)}^{1+\theta }\right]{\left[\sum _x{P}_{X_1{Y}_1}{(x,y)}^{1+{\theta }^{\prime }}\right]}^{\frac{-\theta }{1+{\theta }^{\prime }}}.\end{array}$$

(100)

Then, we have:

$$\begin{array}{c}\hfill (n-1)\theta H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)+\underline{\zeta }(\theta ,{\theta }^{\prime })\le \theta H_{1+\theta ,1+{\theta }^{\prime }}\left(X^n\right|Y^n)\le (n-1)\theta H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y)+\overline{\zeta }(\theta ,{\theta }^{\prime }),\end{array}$$

(101)

where:

$$\begin{array}{ccc}\hfill \overline{\zeta }(\theta ,{\theta }^{\prime })& :=& -log\langle v_{\theta ,{\theta }^{\prime }}|w_{\theta ,{\theta }^{\prime }}\rangle +log\underset{y}{max}{v}_{\theta ,{\theta }^{\prime }}\left(y\right)+\theta \overline{\xi }\left({\theta }^{\prime }\right),\hfill \end{array}$$

(102)

$$\begin{array}{ccc}\hfill \underline{\zeta }(\theta ,{\theta }^{\prime })& :=& -log\langle v_{\theta ,{\theta }^{\prime }}|w_{\theta ,{\theta }^{\prime }}\rangle +\theta \underline{\xi }\left({\theta }^{\prime }\right)\hfill \end{array}$$

(103)

for $\theta >0$ and:

$$\begin{array}{ccc}\hfill \overline{\zeta }(\theta ,{\theta }^{\prime })& :=& -log\langle v_{\theta ,{\theta }^{\prime }}|w_{\theta ,{\theta }^{\prime }}\rangle +log\underset{y}{max}{v}_{\theta ,{\theta }^{\prime }}\left(y\right)+\theta \underline{\xi }\left({\theta }^{\prime }\right),\hfill \end{array}$$

(104)

$$\begin{array}{ccc}\hfill \underline{\zeta }(\theta ,{\theta }^{\prime })& :=& -log\langle v_{\theta ,{\theta }^{\prime }}|w_{\theta ,{\theta }^{\prime }}\rangle +\theta \overline{\xi }\left({\theta }^{\prime }\right)\hfill \end{array}$$

(105)

for $\theta <0$

Proof. 

By multiplying $\theta$ in the definition of $H_{1+\theta ,1+{\theta }^{\prime }}\left(X^n\right|Y^n)$, we have:

$$\begin{array}{c}\hfill {\displaystyle \theta H_{1+\theta ,1+{\theta }^{\prime }}\left(X^n\right|Y^n)}\end{array}$$

(106)

$$\begin{array}{cc}=& -log\sum _{y^n}\left[\sum _{x^n}{P}_{X^n{Y}^n}{(x^n,y^n)}^{1+\theta }\right]{\left[\sum _{x^n}{P}_{X^n{Y}^n}{(x^n,y^n)}^{1+{\theta }^{\prime }}\right]}^{\frac{-\theta }{1+{\theta }^{\prime }}}+\frac{\theta {\theta }^{\prime }}{1+{\theta }^{\prime }}{H}_{1+{\theta }^{\prime }}^{\uparrow }\left(X^n\right|Y^n).\hfill \end{array}$$

(107)

The second term is evaluated by Lemma 11. The first term can be evaluated almost in the same manner as Lemma 11. □

From Lemma 12, we have the following.

Theorem 4.

Suppose that transition matrix W satisfies Assumption 2. For any initial distribution, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}{H}_{1+\theta ,1+{\theta }^{\prime }}\left(X^n\right|Y^n)=H_{1+\theta ,1+{\theta }^{\prime }}^W\left(X\right|Y).\end{array}$$

(108)

3. Source Coding with Full Side-Information

In this section, we investigate source coding with side-information. We start this section by showing the problem setting in Section 3.1. Then, we review and introduce some single-shot bounds in Section 3.2. We derive finite-length bounds for the Markov chain in Section 3.3. Then, in Section 3.5 and Section 3.6, we show the asymptotic characterization for the large deviation regime and the moderate deviation regime by using those finite-length bounds. We also derive the second-order rate in Section 3.4.

3.1. Problem Formulation

A code $\mathsf{\Psi }=(\mathsf{e},\mathsf{d})$ consists of one encoder $\mathsf{e}:\mathcal{X}\to \{1,\dots ,M\}$ and one decoder $\mathsf{d}:\{1,\dots ,M\}\times \mathcal{Y}\to \mathcal{X}$. The decoding error probability is defined by:

$$\begin{array}{ccc}\hfill {\mathrm{P}}_{\mathrm{s}}\left[\mathsf{\Psi }\right]& =& {\mathrm{P}}_{\mathrm{s}}\left[\mathsf{\Psi }\right|P_{XY}]\hfill \end{array}$$

(109)

$$\begin{array}{cc}:=& Pr\{X\ne \mathsf{d}(\mathsf{e}\left(X\right),Y\left)\right\}.\hfill \end{array}$$

(110)

For notational convenience, we introduce the infimum of error probabilities under the condition that the message size is M:

$$\begin{array}{ccc}\hfill {\mathrm{P}}_{\mathrm{s}}\left(M\right)& =& {\mathrm{P}}_{\mathrm{s}}\left(M\right|P_{XY})\hfill \end{array}$$

(111)

$$\begin{array}{cc}:=& \underset{\mathsf{\Psi }}{inf}{\mathrm{P}}_{\mathrm{s}}\left[\mathsf{\Psi }\right].\hfill \end{array}$$

(112)

For theoretical simplicity, we focus on a randomized choice of our encoder. For this purpose, we employ a randomized hash function F from $\mathcal{X}$ to $\{1,\dots ,M\}$. A randomized hash function F is called a two-universal hash when $Pr\{F\left(x\right)=F\left(x^{\prime }\right)\}\le \frac{1}{M}$ for any distinctive x and $x^{\prime }$ [63]; the so-called bin coding [39] is an example of the two-universal hash function. In the following, we denote the set of two-universal hash functions by $\mathcal{F}$. Given an encoder f as a function from $\mathcal{X}$ to $\{1,\dots ,M\}$, we define the decoder ${\mathsf{d}}_f$ as the optimal decoder by $\underset{\mathsf{d}}{\mathrm{argmin}}{\mathrm{P}}_{\mathrm{s}}\left[(f,\mathsf{d})\right]$. Then, we denote the code $(f,{\mathsf{d}}_f)$ by $\mathsf{\Psi }\left(f\right)$. Then, we bound the error probability ${\mathrm{P}}_{\mathrm{s}}\left[\mathsf{\Psi }\left(F\right)\right]$ averaged over the random function F by only using the property of two-universality. In order to consider the worst case of such schemes, we introduce the following quantity:

$$\begin{array}{ccc}\hfill {\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right)& =& {\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right|P_{XY})\hfill \end{array}$$

(113)

$$\begin{array}{cc}:=& \underset{F\in \mathcal{F}}{sup}{\mathbb{E}}_F\left[{\mathrm{P}}_{\mathrm{s}}\left[\mathsf{\Psi }\left(F\right)\right]\right],.\hfill \end{array}$$

(114)

When we consider n-fold extension, the source code and related quantities are denoted with the superscript $\left(n\right)$. For example, the quantities in (112) and (114) are written as ${\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(M\right)$ and ${\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(M\right)$, respectively. Instead of evaluating them, we are often interested in evaluating:

$$\begin{array}{ccc}\hfill M(n,\epsilon )& :=& inf\{M_n:{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)\le \epsilon \},\hfill \end{array}$$

(115)

$$\begin{array}{ccc}\hfill \overline{M}(n,\epsilon )& :=& inf\{M_n:{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)\le \epsilon \}\hfill \end{array}$$

(116)

for given $0\le \epsilon <1$.

3.2. Single-Shot Bounds

In this section, we review existing single-shot bounds and also show novel converse bounds. For the information measures used below, see Section 2.

By using the standard argument on information-spectrum approach, we have the following achievability bound.

Lemma 13

(Lemma 7.2.1 of [3]). The following bound holds:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right)\le \underset{\gamma \ge 0}{inf}\left[P_{XY}\left\{log\frac{1}{P_{X|Y}\left(x\right|y)}>\gamma \right\}+\frac{e^{\gamma }}{M}\right].\end{array}$$

(117)

Although Lemma 13 is useful for the second-order regime, it is known to be not tight in the large deviation regime. By using the large deviation technique of Gallager, we have the following exponential-type achievability bound.

Lemma 14

([64]). The following bound holds: (note that the Gallager function and the upper conditional Rényi entropy are related by (A45)):

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right)\le \underset{-\frac{1}{2}\le \theta \le 0}{inf}{M}^{\frac{\theta }{1+\theta }}{e}^{-\frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow }\left(X\right|Y)}.\end{array}$$

(118)

Although Lemma 14 is known to be tight in the large deviation regime for i.i.d. sources, $H_{1+\theta }^{\uparrow }\left(X\right|Y)$ for Markov chains can only be evaluated under the strongly non-hidden assumption. For this reason, even though the following bound is looser than Lemma 14, it is useful to have another bound in terms of $H_{1+\theta }^{\downarrow }\left(X\right|Y)$, which can be evaluated for Markov chains under the non-hidden assumption.

Lemma 15.

The following bound holds:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right)\le \underset{-1\le \theta \le 0}{inf}{M}^{\theta }{e}^{-\theta H_{1+\theta }^{\downarrow }\left(X\right|Y)}.\end{array}$$

(119)

Proof. 

To derive this bound, we change the variable in (118) as $\theta =\frac{{\theta }^{\prime }}{1-{\theta }^{\prime }}$. Then, $-1\le {\theta }^{\prime }\le 0$, and we have:

$$\begin{array}{c}\hfill M^{{\theta }^{\prime }}{e}^{-{\theta }^{\prime }{H}_{\frac{1}{1-{\theta }^{\prime }}}^{\uparrow }\left(X\right|Y)}\le M^{{\theta }^{\prime }}{e}^{-{\theta }^{\prime }{H}_{1+{\theta }^{\prime }}^{\downarrow }\left(X\right|Y)},\end{array}$$

where we use Lemma A4 in Appendix C. □

For the source coding without side-information, i.e., when X has no side-information, we have the following bound, which is tighter than Lemma 14.

Lemma 16

((2.39) [65]). The following bound holds:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{s}}\left(M\right)\le \underset{-1<\theta \le 0}{inf}{M}^{\frac{\theta }{1+\theta }}{e}^{-\frac{\theta }{1+\theta }{H}_{1+\theta }\left(X\right)}.\end{array}$$

(120)

For the converse part, we first have the following bound, which is very close to the operational definition of source coding with side-information.

Lemma 17

([66]). Let ${\left\{{\mathsf{\Omega }}_y\right\}}_{y\in \mathcal{Y}}$ be a family of subsets ${\mathsf{\Omega }}_y\subset \mathcal{X}$, and let $\mathsf{\Omega }={\cup }_{y\in \mathcal{Y}}{\mathsf{\Omega }}_y\times \left\{y\right\}$. Then, for any $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$, the following bound holds:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{s}}\left(M\right)\ge \underset{\left\{{\mathsf{\Omega }}_y\right\}}{min}\left\{P_{XY}\left({\mathsf{\Omega }}^c\right):\sum _y{Q}_Y\left(y\right)\left|{\mathsf{\Omega }}_y\right|\le M\right\}.\end{array}$$

(121)

Since Lemma 17 is close to the operational definition, it is not easy to evaluate Lemma 17. Thus, we derive another bound by loosening Lemma 17, which is more tractable for evaluation. Slightly weakening Lemma 17, we have the following.

Lemma 18

([3,4]). For any $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$, we have (In fact, a special case for $Q_Y=P_Y$ corresponds to Lemma 7.2.2 of [3]. A bound that involves $Q_Y$ was introduced in [4] for channel coding, and it can be regarded as a source coding counterpart of that result.):

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{s}}\left(M\right)\ge \underset{\gamma \ge 0}{sup}\left[P_{XY}\left\{log\frac{Q_Y\left(y\right)}{P_{XY}(x,y)}>\gamma \right\}-\frac{M}{e^{\gamma }}\right].\end{array}$$

(122)

By using the change-of-measure argument, we also obtain the following converse bound.

Theorem 5.

For any $Q_Y\in \mathcal{P}\left(\mathcal{Y}\right)$, we have:

$$\begin{array}{cc}-\hfill & {\displaystyle log{\mathrm{P}}_{\mathrm{s}}\left(M\right)}\hfill \\ & \le \underset{\genfrac{}{}{0pt}{}{s>0}{\tilde{\theta }\in \mathbb{R},\vartheta \ge 0}}{inf}[(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}\left(P_{XY}\right|Q_Y)-H_{1+(1+s)\tilde{\theta }}\left(P_{XY}\right|Q_Y)\right\}\hfill \end{array}$$

(123)

$$\begin{array}{cc}\hfill & -(1+s)log\left(1-2e^{-\frac{-\vartheta R+(\tilde{\theta }+\vartheta (1+\tilde{\theta }))H_{\tilde{\theta }+\vartheta (1+\tilde{\theta })}\left(P_{XY}\right|Q_Y)-(1+\vartheta )\tilde{\theta }{H}_{1+\tilde{\theta }}\left(P_{XY}\right|Q_Y)}{1+\vartheta }}\right)]/s\hfill \\ \le \hfill & \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}\left(P_{XY}\right|Q_Y)-H_{1+(1+s)\tilde{\theta }}\left(P_{XY}\right|Q_Y)\right\}\hfill \end{array}$$

(124)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(\theta \left(a\left(R\right)\right)-\tilde{\theta })a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}\left(P_{XY}\right|Q_Y)+\tilde{\theta }{H}_{1+\tilde{\theta }}\left(P_{XY}\right|Q_Y)}\right)]/s,\hfill \end{array}$$

(125)

where $R=logM$, and $\theta \left(a\right)={\theta }^Q\left(a\right)$ and $a\left(R\right)=a^Q\left(R\right)$ are the inverse functions defined in (29) and (32), respectively.

Proof. 

See Appendix K. □

In particular, by taking $Q_Y=P_Y^{(1+\theta (a\left(R\right)\left)\right)}$ in Theorem 5, we have the following.

Corollary 1.

We have:

$$\begin{array}{c}\hfill {\displaystyle -log{\mathrm{P}}_{\mathrm{s}}\left(M\right)}\\ \le & \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta },1+\theta \left(a\left(R\right)\right)}\left(X\right|Y)\right\}\hfill \end{array}$$

(126)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(\theta \left(a\left(R\right)\right)-\tilde{\theta })a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow }\left(X\right|Y)+\tilde{\theta }{H}_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}\left(X\right|Y)}\right)]/s,\hfill \end{array}$$

(127)

where $\theta \left(a\right)={\theta }^{\uparrow }\left(a\right)$ and $a\left(R\right)=a^{\uparrow }\left(R\right)$ are the inverse functions defined in (35) and (36).

Remark 5.

Here, we discuss the possibility for extension to the continuous case. As explained in Remark 1, we can define the information quantities for the case when $\mathcal{Y}$ is continuous, but $\mathcal{X}$ is a discrete finite set. The discussions in this subsection still hold even in this continuous case. In particular, in the n-i.i.d. extension case with this continuous setting, Lemma 14 and Corollary 1 hold when the information measures are replaced by n times the single-shot information measures.

3.3. Finite-Length Bounds for Markov Source

In this subsection, we derive several finite-length bounds for the Markov source with a computable form. Unfortunately, it is not easy to evaluate how tight those bounds are only with their formula. Their tightness will be discussed by considering the asymptotic limit in the remaining subsections of this section. Since we assume the irreducibility for the transition matrix describing the Markov chain, the following bound holds with any initial distribution.

To derive a lower bound on $-log{\overline{\mathrm{P}}}_{\mathrm{s}}\left(M_n\right)$ in terms of the Rényi entropy of the transition matrix, we substitute the formula for the Rényi entropy given in Lemma 10 into Lemma 15. Then, we can derive the following achievability bound.

Theorem 6

(Direct, Ass. 1). Suppose that transition matrix W satisfies Assumption 1. Let $R:=\frac{1}{n}log{M}_n$. Then, for every $n\ge 1$, we have:

$$\begin{array}{c}\hfill -log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)\ge \underset{-1\le \theta \le 0}{sup}\left[-\theta nR+(n-1)\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)+\underline{\delta }\left(\theta \right)\right],\end{array}$$

(128)

where $\underline{\delta }\left(\theta \right)$ is given by (91).

For the source coding without side-information, from Lemma 16 and a special case of Lemma 10, we have the following achievability bound.

Theorem 7

(Direct, no-side-information). Let $R:=\frac{1}{n}log{M}_n$. Then, for every $n\ge 1$, we have:

$$\begin{array}{c}\hfill -log{\mathrm{P}}_{\mathrm{e}}^{\left(n\right)}\left(M_n\right)\ge \underset{-1<\theta \le 0}{sup}\frac{-n\theta R+(n-1)\theta H_{1+\theta }^W\left(X\right)+\underline{\delta }\left(\theta \right)}{1+\theta }.\end{array}$$

(129)

To derive an upper bound on $-log{\mathrm{P}}_{\mathrm{s}}\left(M_n\right)$ in terms of the Rényi entropy of transition matrix, we substitute the formula for the Rényi entropy given in Lemma 10 for Theorem 5. Then, we have the following converse bound.

Theorem 8

(Converse, Ass. 1). Suppose that transition matrix W satisfies Assumption 1. Let $R:=\frac{1}{n}log{M}_n$. For any $H^W\left(X\right|Y)<R<H_0^{\downarrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{cc}-\hfill & {\displaystyle log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)}\hfill \\ \hfill & \le \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(n-1)(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right\}+{\delta }_1\hfill \end{array}$$

(130)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(n-1)[({\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta })a^W\left(R\right)-{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\downarrow ,W}\left(X\right|Y)+\tilde{\theta }{H}_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)]+{\delta }_2}\right)]/s,\hfill \end{array}$$

(131)

where $\theta \left(a\right)={\theta }^{\downarrow }\left(a\right)$ and $a\left(R\right)=a^{\downarrow }\left(R\right)$ are the inverse functions defined by (67) and (70), respectively,

$$\begin{array}{ccc}\hfill {\delta }_1& :=& (1+s)\overline{\delta }\left(\tilde{\theta }\right)-\underline{\delta }\left((1+s)\tilde{\theta }\right),\hfill \end{array}$$

(132)

$$\begin{array}{ccc}\hfill {\delta }_2& :=& \frac{({\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta })R-(1+\tilde{\theta })\underline{\delta }\left({\theta }^W\left(a^W\left(R\right)\right)\right)+(1+{\theta }^W\left(a^W\left(R\right)\right))\overline{\delta }\left(\tilde{\theta }\right)}{1+{\theta }^W\left(a^W\left(R\right)\right)},\hfill \end{array}$$

(133)

and $\overline{\delta }(·)$ and $\underline{\delta }(·)$ are given by (90) and (91), respectively.

Proof. 

We first use (124) of Theorem 5 for $Q_{Y^n}=P_{Y^n}$ and Lemma 10. Then, we restrict the range of $\tilde{\theta }$ as $-1<\tilde{\theta }<{\theta }^W\left(a^W\left(R\right)\right)$ and set $\vartheta =\frac{{\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta }}{1+\tilde{\theta }}$. Then, we have the assertion of the theorem. □

Next, we derive tighter bounds under Assumption 2. To derive a lower bound on $-log{\overline{\mathrm{P}}}_{\mathrm{s}}\left(M_n\right)$ in terms of the Rényi entropy of the transition matrix, we substitute the formula for the Rényi entropy in Lemma 11 for Lemma 14. Then, we have the following achievability bound.

Theorem 9

(Direct, Ass. 2). Suppose that transition matrix W satisfies Assumption 2. Let $R:=\frac{1}{n}log{M}_n$. Then, we have:

$$\begin{array}{c}\hfill -log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta nR+(n-1)\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }+\underline{\xi }\left(\theta \right),\end{array}$$

(134)

where $\underline{\xi }\left(\theta \right)$ is given by (98).

Finally, to derive an upper bound on $-log{\mathrm{P}}_{\mathrm{s}}\left(M_n\right)$ in terms of the Rényi entropy for the transition matrix, we substitute the formula for the Rényi entropy in Lemma 12 for Theorem 5 for $Q_{Y^n}=P_{Y^n}^{(1+{\theta }^W\left(a^W\left(R\right)\right))}$. Then, we can derive the following converse bound.

Theorem 10

(Converse, Ass. 2). Suppose that transition matrix W satisfies Assumption 2. Let $R:=\frac{1}{n}log{M}_n$. For any $H^W\left(X\right|Y)<R<H_0^{\uparrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{c}\hfill {\displaystyle -log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)}\\ \hfill \le & \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<{\theta }^W\left(a^W\left(R\right)\right)}}{inf}[(n-1)(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta },1+{\theta }^W\left(a^W\left(R\right)\right)}^W\left(X\right|Y)-H_{1+(1+s)\tilde{\theta },1+{\theta }^W\left(a^W\left(R\right)\right)}^W\left(X\right|Y)\right\}+{\delta }_1\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill & -(1+s)log\left(1-2e^{(n-1)[({\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta })a^W\left(R\right)-{\theta }^W\left(a^W\left(R\right)\right)H_{1+{\theta }^W\left(a^W\left(R\right)\right)}^{\uparrow ,W}\left(X\right|Y)+\tilde{\theta }{H}_{1+\tilde{\theta },1+{\theta }^W\left(a^W\left(R\right)\right)}^W\left(X\right|Y)]+{\delta }_2}\right)]/s,\hfill \end{array}$$

(136)

where ${\theta }^W\left(a\right)={\theta }^{\uparrow ,W}\left(a\right)$ and $a^W\left(R\right)=a^{\uparrow ,W}\left(R\right)$ are the inverse functions defined by (71) and (73), respectively,

$$\begin{array}{cc}\hfill {\delta }_1:=& (1+s)\overline{\zeta }(\tilde{\theta },{\theta }^W\left(a^W\left(R\right)\right))-\underline{\zeta }((1+s)\tilde{\theta },{\theta }^W\left(a^W\left(R\right)\right)),\hfill \end{array}$$

(137)

$$\begin{array}{cc}\hfill {\delta }_2:=& \frac{({\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta })R-(1+\tilde{\theta })\underline{\zeta }({\theta }^W\left(a^W\left(R\right)\right),{\theta }^W\left(a^W\left(R\right)\right))+(1+{\theta }^W\left(a^W\left(R\right)\right))\overline{\zeta }(\tilde{\theta },{\theta }^W\left(a^W\left(R\right)\right))}{1+{\theta }^W\left(a^W\left(R\right)\right)},\hfill \end{array}$$

(138)

and $\overline{\zeta }(·,·)$ and $\underline{\zeta }(·,·)$ are given by (102)–(105).

Proof. 

We first use (124) of Theorem 5 for $Q_{Y^n}=P_{Y^n}^{(1+{\theta }^W\left(a^W\left(R\right)\right))}$ and Lemma 12. Then, we restrict the range of $\tilde{\theta }$ as $-1<\tilde{\theta }<{\theta }^W\left(a^W\left(R\right)\right)$ and set $\vartheta =\frac{{\theta }^W\left(a^W\left(R\right)\right)-\tilde{\theta }}{1+\tilde{\theta }}$. Then, we have the assertion of the theorem. □

3.4. Second-Order

By applying the central limit theorem to Lemma 13 (cf. [67] (Theorem 27.4, Example 27.6)) and Lemma 18 for $Q_Y=P_Y$ and by using Theorem 2, we have the following.

Theorem 11.

Suppose that transition matrix W on $\mathcal{X}\times \mathcal{Y}$ satisfies Assumption 1. For arbitrary $\epsilon \in (0,1)$, we have:

$$\begin{array}{c}\hfill logM(n,\epsilon )=log\overline{M}(n,\epsilon )+o\left(\sqrt{n}\right)=n{H}^W\left(X\right|Y)+\sqrt{{\mathsf{V}}^W\left(X\right|Y)}\sqrt{n}\mathsf{\Phi }(1-\epsilon )+o\left(\sqrt{n}\right).\end{array}$$

(139)

Proof. 

The central limit theorem for the Markov process cf. [67] (Theorem 27.4, Example 27.6) guarantees that the random variable $(-log{P}_{X^n|Y^n}\left(X^n\right|Y^n)-n{H}^W\left(X\right|Y))/\sqrt{n}$ asymptotically obeys the normal distribution with average zero and variance ${\mathsf{V}}^W\left(X\right|Y)$, where we use Theorem 2 to show that the limit of the variance is given by ${\mathsf{V}}^W\left(X\right|Y)$. Let $R=\sqrt{{\mathsf{V}}^W\left(X\right|Y)}{\mathsf{\Phi }}^{-1}(1-\epsilon )$. Substituting $M=e^{n{H}^W\left(X\right|Y)+\sqrt{n}R}$ and $\gamma =n{H}^W\left(X\right|Y)+\sqrt{n}R-n^{\frac{1}{4}}$ in Lemma 13, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(e^{n{H}^W\left(X\right|Y)+\sqrt{n}R}\right)\le \epsilon .\end{array}$$

(140)

□

On the other hand, substituting $M=e^{n{H}^W\left(X\right|Y)+\sqrt{n}R}$ and $\gamma =n{H}^W\left(X\right|Y)+\sqrt{n}R+n^{\frac{1}{4}}$ in Lemma 18 for $Q_Y=P_Y$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{n{H}^W\left(X\right|Y)+\sqrt{n}R}\right)\ge \epsilon .\end{array}$$

(141)

Combining (140) and (141), we have the statement of the theorem.

From the above theorem, the (first-order) compression limit of source coding with side-information for a Markov source under Assumption 1 is given by (although the compression limit of source coding with side-information for a Markov chain is known more generally [68], we need Assumption 1 to get a single-letter characterization):

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}logM(n,\epsilon )& =& \underset{n\to \infty }{lim}\frac{1}{n}log\overline{M}(n,\epsilon )\hfill \end{array}$$

(142)

$$\begin{array}{cc}=& H^W\left(X\right|Y)\hfill \end{array}$$

(143)

for any $\epsilon \in (0,1)$. In the next subsections, we consider the asymptotic behavior of the error probability when the rate is larger than the compression limit $H^W\left(X\right|Y)$ in the moderate deviation regime and the large deviation regime, respectively.

3.5. Moderate Deviation

From Theorems 6 and 8, we have the following.

Theorem 12.

Suppose that transition matrix W satisfies Assumption 1. For arbitrary $t\in (0,1/2)$ and $\delta >0$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}-\frac{1}{n^{1-2t}}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{n{H}^W\left(X\right|Y)+n^{1-t}\delta }\right)& =& \underset{n\to \infty }{lim}-\frac{1}{n^{1-2t}}log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(e^{n{H}^W\left(X\right|Y)+n^{1-t}\delta }\right)\hfill \end{array}$$

(144)

$$\begin{array}{cc}=& \frac{{\delta }^2}{2{\mathsf{V}}^W\left(X\right|Y)}.\hfill \end{array}$$

(145)

Proof. 

We apply Theorems 6 and 8 to the case with $R=H^W\left(X\right|Y)+n^{-t}\delta$, i.e., $\theta \left(a\left(R\right)\right)=-n^{-1}\frac{\delta }{{\mathsf{V}}^W\left(X\right|Y)}+o\left(n^{-t}\right)$. For the achievability part, from (88) and Theorem 6, we have:

$$\begin{array}{cc}\hfill -log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(M_n\right)& \ge \underset{-1\le \theta \le 0}{sup}\left[-\theta nR+(n-1)\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]+\underset{-1\le \theta \le 0}{inf}\underline{\delta }\left(\theta \right)\hfill \end{array}$$

(146)

$$\begin{array}{cc}\hfill & \ge n^{1-2t}\frac{{\delta }^2}{2{\mathsf{V}}^W\left(X\right|Y)}+o\left(n^{1-2t}\right).\hfill \end{array}$$

(147)

To prove the converse part, we fix arbitrary $s>0$ and choose $\tilde{\theta }$ to be $-n^{-t}\frac{\delta }{{\mathsf{V}}^W\left(X\right|Y)}+n^{-2t}$. Then, Theorem 8 implies that:

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n^{1-2t}}log{\mathrm{P}}_{\mathrm{s}}\left(M_n\right)& \le \underset{n\to \infty }{lim\; sup}{n}^{2t}\frac{1+s}{s}\tilde{\theta }\left\{H_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right\}\hfill \end{array}$$

(148)

$$\begin{array}{cc}\hfill & =\underset{n\to \infty }{lim\; sup}{n}^{2t}\frac{1+s}{s}s{\tilde{\theta }}^2\frac{d{H}_{1+\theta }^{\downarrow ,W}\left(X\right|Y)}{d\theta }{|}_{\theta =\tilde{\theta }}\hfill \end{array}$$

(149)

$$\begin{array}{cc}\hfill & =(1+s)\frac{{\delta }^2}{2{\mathsf{V}}^W\left(X\right|Y)}.\hfill \end{array}$$

(150)

□

Remark 6.

In the literature [13,69], the moderate deviation results are stated for ${ϵ}_n$ such that ${ϵ}_n\to 0$ and $n{ϵ}_n^2\to \infty$ instead of $n^{-t}$ for $t\in (0,1/2)$. Although the former is slightly more general than the latter, we employ the latter formulation in Theorem 12 since the order of convergence is clearer. In fact, $n^{-t}$ in Theorem 12 can be replaced by general ${ϵ}_n$ without modifying the argument of the proof.

3.6. Large Deviation

From Theorems 6 and 8, we have the following.

Theorem 13.

Suppose that transition matrix W satisfies Assumption 1. For $H^W\left(X\right|Y)<R$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)\ge \underset{-1\le \theta \le 0}{sup}[-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)].\end{array}$$

(151)

On the other hand, for $H^W\left(X\right|Y)<R<H_0^{\downarrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)& \le & -\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(152)

$$\begin{array}{cc}=& \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)}{1+\theta }.\hfill \end{array}$$

(153)

Proof. 

The achievability bound (151) follows from Theorem 6. The converse part (152) is proven from Theorem 8 as follows. We first fix $s>0$ and $-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)$. Then, Theorem 8 implies:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)\le \frac{1+s}{s}\tilde{\theta }\left\{H_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right\}.\end{array}$$

(154)

By taking the limit $s\to 0$ and $\tilde{\theta }\to \theta \left(a\left(R\right)\right)$, we have:

$$\begin{array}{cc}\hfill & \frac{1+s}{s}\tilde{\theta }\left\{H_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right\}\hfill \end{array}$$

(155)

$$\begin{array}{cc}\hfill & =\frac{1}{s}\left(\tilde{\theta }{H}_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-(1+s)\tilde{\theta }{H}_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right)+\tilde{\theta }{H}_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(156)

$$\begin{array}{cc}\hfill & \to -\tilde{\theta }\frac{d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]}{d\theta }{|}_{\theta =\tilde{\theta }}+\tilde{\theta }{H}_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)(ass\to 0)\hfill \end{array}$$

(157)

$$\begin{array}{cc}\hfill & \to -\theta \left(a\left(R\right)\right)\frac{d\left[\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right]}{d\theta }{|}_{\theta =\theta \left(a\right(R\left)\right)}+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow ,W}\left(X\right|Y)(as\tilde{\theta }\to \theta \left(a\left(R\right)\right))\hfill \end{array}$$

(158)

$$\begin{array}{cc}\hfill & =-\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow ,W}\left(X\right|Y).\hfill \end{array}$$

(159)

Thus, (152) is proven. The alternative expression (153) is derived via Lemma 9. □

Under Assumption 2, from Theorems 9 and 10, we have the following tighter bound.

Theorem 14.

Suppose that transition matrix W satisfies Assumption 2. For $H^W\left(X\right|Y)<R$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }.\end{array}$$

(160)

On the other hand, for $H^W\left(X\right|Y)<R<H_0^{\uparrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)& \le & -\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y)\hfill \end{array}$$

(161)

$$\begin{array}{cc}=& \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }.\hfill \end{array}$$

(162)

Proof. 

The achievability bound (160) follows from Theorem 9. The converse part (161) is proven from Theorem 10 as follows. We first fix $s>0$ and $-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)$. Then, Theorem 10 implies:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)\le \frac{1+s}{s}\tilde{\theta }\left\{H_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)-H_{1+(1+s)\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)\right\}.\end{array}$$

(163)

By taking the limit $s\to 0$ and $\tilde{\theta }\to \theta \left(a\left(R\right)\right)$, we have:

$$\begin{array}{cc}\hfill & \frac{1+s}{s}\tilde{\theta }\left\{H_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)-H_{1+(1+s)\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)\right\}\hfill \end{array}$$

(164)

$$\begin{array}{cc}\hfill & =\frac{1}{s}\left(\tilde{\theta }{H}_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)-(1+s)\tilde{\theta }{H}_{1+(1+s)\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)\right)+\tilde{\theta }{H}_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)\hfill \end{array}$$

(165)

$$\begin{array}{cc}\hfill & \to -\tilde{\theta }\frac{d\left[\theta H_{1+\theta ,1+\theta \left(a\right(R\left)\right)}^W\left(X\right|Y)\right]}{d\theta }{|}_{\theta =\tilde{\theta }}+\tilde{\theta }{H}_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)(ass\to 0)\hfill \end{array}$$

(166)

$$\begin{array}{cc}\hfill & \to -\theta \left(a\left(R\right)\right)\frac{d\left[\theta H_{1+\theta ,1+\theta \left(a\right(R\left)\right)}^W\left(X\right|Y)\right]}{d\theta }{|}_{\theta =\theta \left(a\right(R\left)\right)}+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y)(as\tilde{\theta }\to \theta \left(a\left(R\right)\right))\hfill \end{array}$$

(167)

$$\begin{array}{cc}\hfill & =-\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y).\hfill \end{array}$$

(168)

Thus, (161) is proven. The alternative expression (162) is derived via Lemma 9.□

Remark 7.

For $R\le R_{\mathrm{cr}}$, where (cf. (72) for the definition of $R\left(a\right)$):

$$\begin{array}{c}\hfill R_{\mathrm{cr}}:=R\left(\frac{d\left[\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)\right]}{d\theta }{|}_{\theta =-\frac{1}{2}}\right)\end{array}$$

(169)

is the critical rate, the left-hand side of (76) in Lemma 9 is attained by parameters in the range $-1/2\le \theta \le 0$. Thus, the lower bound in (160) is rewritten as:

$$\begin{array}{c}\hfill \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }=-\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y).\end{array}$$

(170)

Thus, the lower bound and the upper bounds coincide up to the critical rate.

Remark 8.

For the source coding without side-information, by taking the limit of Theorem 7, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)\ge \underset{-1\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^W\left(X\right)}{1+\theta }.\end{array}$$

(171)

On the other hand, as a special case of (152) without side-information, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left(e^{nR}\right)& \le & \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^W\left(X\right)}{1+\theta }\hfill \end{array}$$

(172)

for $H^W\left(X\right)<R<H_0^W\left(X\right)$. Thus, we can recover the results in [40,41] by our approach.

3.7. Numerical Example

In this section, to demonstrate the advantage of our finite-length bound, we numerically evaluate the achievability bound in Theorem 7 and a special case of the converse bound in Theorem 8 for the source coding without side-information. Thanks to the aspect (A2), our numerical calculation shows that our upper finite-length bounds are very close to our lower finite-length bounds when the size n is sufficiently large. Thanks to the aspect (A1), we could calculate both bounds with the huge size $n=1\times {10}^5$ because the calculation complexity behaves as $O\left(1\right)$.

We consider a binary transition matrix W given by Figure 2, i.e.,

$$\begin{array}{c}\hfill W=\left[\begin{array}{cc}1-p& q\\ p& 1-q\end{array}\right].\end{array}$$

(173)

In this case, the stationary distribution is:

$$\begin{array}{ccc}\hfill \tilde{P}\left(0\right)& =& \frac{q}{p+q},\hfill \end{array}$$

(174)

$$\begin{array}{ccc}\hfill \tilde{P}\left(1\right)& =& \frac{p}{p+q}.\hfill \end{array}$$

(175)

The entropy is:

$$\begin{array}{c}\hfill H^W\left(X\right)=\frac{q}{p+q}h\left(p\right)+\frac{p}{p+q}h\left(q\right),\end{array}$$

(176)

where $h(·)$ is the binary entropy function. The tilted transition matrix is:

$$\begin{array}{c}\hfill W_{\theta }=\left[\begin{array}{cc}{(1-p)}^{1+\theta }& q^{1+\theta }\\ p^{1+\theta }& {(1-q)}^{1+\theta }\end{array}\right].\end{array}$$

(177)

The Perron–Frobenius eigenvalue is:

$$\begin{array}{c}\hfill {\lambda }_{\theta }=\frac{{(1-p)}^{1+\theta }+{(1-q)}^{1+\theta }+\sqrt{{\{{(1-p)}^{1+\theta }-{(1-q)}^{1+\theta }\}}^2+4p^{1+\theta }{q}^{1+\theta }}}{2}\end{array}$$

(178)

and its normalized eigenvector is:

$$\begin{array}{ccc}\hfill {\tilde{P}}_{\theta }\left(0\right)& =& \frac{q^{1+\theta }}{{\lambda }_{\theta }-{(1-p)}^{1+\theta }+q^{1+\theta }},\hfill \end{array}$$

(179)

$$\begin{array}{ccc}\hfill {\tilde{P}}_{\theta }\left(1\right)& =& \frac{{\lambda }_{\theta }-{(1-p)}^{1+\theta }}{{\lambda }_{\theta }-{(1-p)}^{1+\theta }+q^{1+\theta }}.\hfill \end{array}$$

(180)

The normalized eigenvector of $W_{\rho }^T$ is also given by:

$$\begin{array}{ccc}\hfill {\widehat{P}}_{\theta }\left(0\right)& =& \frac{p^{1+\theta }}{{\lambda }_{\theta }-{(1-p)}^{1+\theta }+p^{1+\theta }},\hfill \end{array}$$

(181)

$$\begin{array}{ccc}\hfill {\widehat{P}}_{\theta }\left(1\right)& =& \frac{{\lambda }_{\theta }-{(1-p)}^{1+\theta }}{{\lambda }_{\theta }-{(1-p)}^{1+\theta }+p^{1+\theta }}.\hfill \end{array}$$

(182)

From these calculations, we can evaluate the bounds in Theorems 7 and 8. For $p=0.1$, $q=0.2$, the bounds are plotted in Figure 3 for fixed error probability $\epsilon ={10}^{-3}$. Although there is a gap between the achievability bound and the converse bound for rather small n, the gap is less than approximately 5% of the entropy rate for n larger than 10,000. We also plot the bounds in Figure 4 for fixed block length $n=$ 10,000 and varying $\epsilon$. The gap between the achievability bound and the converse bound remains approximately 5% of the entropy rate even for $\epsilon$ as small as ${10}^{-10}$.

The gap between the achievability bound and the converse bound in Figure 3 is rather large compared to a similar numerical experiment conducted in [1]. One reason for the gap is that our bounds are exponential-type bounds. For instance, when the source is i.i.d., the achievability bound essentially reduces to the so-called Gallager bound [64]. However, an advantage of our bounds is that the computational complexity does not depend on the blocklength. The computational complexities of the bounds plotted in [1] depend the blocklength, and numerical computation of those bounds for Markov sources seems to be difficult.

When $p=q$, an alternative approach to derive tighter bounds is to consider encoding of the Markov transition, i.e., $\mathbf{1}[X_i=X_{i+1}]$, instead of the source itself (cf. [45] (Example 4)). Then, the analysis can be reduced to i.i.d. case. However, such an approach is possible only when $p=q$.

3.8. Summary of the Results

The obtained results in this section are summarized in

Table 2. Summary of the bounds for source coding with full side-information. No-side means the case with no side-information.

Ach./Conv.	Markov	Single-Shot	${\mathbf{P}}_{\mathbf{s}}$/${\overline{\mathbf{P}}}_{\mathbf{s}}$	Complexity	Large Deviation	Moderate Deviation	Second Order
Achievability	Theorem 6 (Ass. 1)	Lemma 15	${\overline{\mathrm{P}}}_{\mathrm{s}}$	$O\left(1\right)$		🗸
	Theorem 9 (Ass. 2)	Lemma 14	${\overline{\mathrm{P}}}_{\mathrm{s}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Theorem 7 (No-side)	Lemma 16	${\overline{\mathrm{P}}}_{\mathrm{s}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Lemma 13		${\overline{\mathrm{P}}}_{\mathrm{s}}$	Tail		🗸	🗸
Converse	Theorem 8 (Ass. 1)	(Theorem 5)	${\mathrm{P}}_{\mathrm{s}}$	$O\left(1\right)$		🗸
	Theorem 10 (Ass. 2)	Corollary 1	${\mathrm{P}}_{\mathrm{s}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Theorem 8 (No-side)	(Theorem 5)	${\mathrm{P}}_{\mathrm{s}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Lemma 18		${\mathrm{P}}_{\mathrm{s}}$	Tail		🗸	🗸

. The check marks 🗸 indicate that the tight asymptotic bounds (large deviation, moderate deviation, and second-order) can be obtained from those bounds. The marks 🗸${}^{*}$ indicate that the large deviation bound can be derived up to the critical rate. The computational complexity “Tail” indicates that the computational complexities of those bounds depend on the computational complexities of tail probabilities. It should be noted that Theorem 8 is derived from a special case ($Q_Y=P_Y$) of Theorem 5. The asymptotically optimal choice is $Q_Y=P_Y^{(1+\theta )}$, which corresponds to Corollary 1. Under Assumption 1, we can derive the bound of the Markov case only for that special choice of $Q_Y$, while under Assumption 2, we can derive the bound of the Markov case for the optimal choice of $Q_Y$.

4. Channel Coding

In this section, we investigate the channel coding with a conditional additive channel. The first part of this section discusses the general properties of the channel coding with a conditional additive channel. The second part of this section discusses the properties of the channel coding when the conditional additive noise of the channel is Markov. The first part starts with showing the problem setting in Section 4.1 by introducing a conditional additive channel. Section 4.2 gives a canonical method to convert a regular channel to a conditional additive channel. Section 4.3 gives a method to convert a BPSK-AWGN channel to a conditional additive channel. Then, we show some single-shot achievability bounds in Section 4.4 and single-shot converse bounds in Section 4.5.

As the second part, we derive finite-length bounds for the Markov noise channel in Section 4.6. Then, we derive the second-order rate in Section 4.7. In Section 4.8 and Section 4.9, we show the asymptotic characterization for the large deviation regime and the moderate deviation regime by using those finite-length bounds.

4.1. Formulation for the Conditional Additive Channel

4.1.1. Single-Shot Case

We first present the problem formulation in the single-shot setting. For a channel $P_{B|A}\left(b\right|a)$ with input alphabet $\mathcal{A}$ and output alphabet $\mathcal{B}$, a channel code $\mathsf{\Psi }=(\mathsf{e},\mathsf{d})$ consists of one encoder $\mathsf{e}:\{1,\dots ,M\}\to \mathcal{A}$ and one decoder $\mathsf{d}:\mathcal{B}\to \{1,\dots ,M\}$. The average decoding error probability is defined by:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}\left[\mathsf{\Psi }\right]:=\sum _{m=1}^M\frac{1}{M}{P}_{B|A}\left(\{b:\mathsf{d}\left(b\right)\ne m\}\right|\mathsf{e}\left(m\right)).\end{array}$$

(183)

For notational convenience, we introduce the error probability under the condition that the message size is M:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}\left(M\right):=\underset{\mathsf{\Psi }}{inf}{\mathrm{P}}_{\mathrm{c}}\left[\mathsf{\Psi }\right].\end{array}$$

(184)

Assume that the input alphabet $\mathcal{A}$ is the same set as the output alphabet $\mathcal{B}$ and they equal an additive group $\mathcal{X}$. When the transition matrix $P_{B|A}\left(b\right|a)$ is given as $P_X(b-a)$ by using a distribution $P_X$ on $\mathcal{X}$, the channel is called additive.

To extend the concept of the additive channel, we consider the case when the input alphabet $\mathcal{A}$ is an additive group $\mathcal{X}$ and the output alphabet $\mathcal{B}$ is the product set $\mathcal{X}\times \mathcal{Y}$. When the transition matrix $P_{B|A}(x,y|a)$ is given as $P_{XY}(x-a,y)$ by using a distribution $P_{XY}$ on $\mathcal{X}\times \mathcal{Y}$, the channel is called conditional additive. In this paper, we are exclusively interested in the conditional additive channel. As explained in Section 4.2, a channel is a conditional additive channel if and only if it is a regular channel in the sense of [31]. When we need to express the underlying distribution of the noise explicitly, we denote the average decoding error probability by ${\mathrm{P}}_{\mathrm{c}}\left[\mathsf{\Psi }\right|P_{XY}]$.

4.1.2. n-Fold Extension

When we consider n-fold extension, the channel code is denoted with subscript n such as ${\mathsf{\Psi }}_n=({\mathsf{e}}_n,{\mathsf{d}}_n)$. The error probabilities given in (183) and (184) are written with the superscript $\left(n\right)$ as ${\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[{\mathsf{\Psi }}_n\right]$ and ${\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)$, respectively. Instead of evaluating the error probability ${\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)$ for given $M_n$, we are also interested in evaluating:

$$\begin{array}{c}\hfill M(n,\epsilon ):=sup\left\{M_n:{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)\le \epsilon \right\}\end{array}$$

(185)

for given $0\le \epsilon \le 1$.

When the channel is given as a conditional distribution, the channel is given by:

$$\begin{array}{c}\hfill P_{B^n|A^n}(x^n,y^n|a^n)=P_{X^n{Y}^n}(x^n-a^n,y^n),\end{array}$$

(186)

where $P_{X^n{Y}^n}$ is a noise distribution on ${\mathcal{X}}^n\times {\mathcal{Y}}^n$.

For the code construction, we investigate the linear code. For an $(n,k)$ linear code ${\mathcal{C}}_n\subset {\mathcal{A}}^n$, there exists a parity check matrix $f_n:{\mathcal{A}}^n\to {\mathcal{A}}^{n-k}$ such that the kernel of $f_n$ is ${\mathcal{C}}_n$. That is, given a parity check matrix $f_n:{\mathcal{A}}^n\to {\mathcal{A}}^{n-k}$, we define the encoder $I_{\mathrm{Ker}\left(f_n\right)}:{\mathcal{C}}_n\to {\mathcal{A}}^n$ as the imbedding of the kernel $\mathrm{Ker}\left(f_n\right)$. Then, using the decoder ${\mathsf{d}}_{f_n}:=\underset{\mathsf{d}}{\mathrm{argmin}}{\mathrm{P}}_{\mathrm{c}}\left[(I_{\mathrm{Ker}\left(f_n\right)},\mathsf{d})\right]$, we define $\mathsf{\Psi }\left(f_n\right)=(I_{\mathrm{Ker}\left(f_n\right)},{\mathsf{d}}_{f_n})$.

Here, we employ a randomized choice of a parity check matrix. In particular, instead of a two-universal hash function, we focus on linear two-universal hash functions, because the linearity is required in the above relation with source coding. Therefore, denoting the set of linear two-universal hash functions from ${\mathcal{A}}^n$ to ${\mathcal{A}}^{n-k}$ by ${\mathcal{F}}_l$, we introduce the quantity:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k):=\underset{F_n\in {\mathcal{F}}_l}{sup}{\mathbb{E}}_{F_n}\left[{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[\mathsf{\Psi }\left(F_n\right)\right]\right].\end{array}$$

(187)

Taking the infimum over all linear codes associated with $F_n$ (cf. (113)), we obviously have:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}{\left(\right|\mathcal{A}|}^k)\le {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k).\end{array}$$

(188)

When we consider the error probability for conditionally additive channels, we use notation ${\overline{\mathrm{P}}}_{\mathrm{c}}(n,k|P_{XY})$ so that the underlying distribution of the noise is explicit. We are also interested in characterizing:

$$\begin{array}{c}\hfill k(n,\epsilon ):=sup\left\{k:{\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\le \epsilon \right\}\end{array}$$

(189)

for given $0\le \epsilon \le 1$.

4.2. Conversion from the Regular Channel to the Conditional Additive Channel

The aim of this subsection is to show the following theorem by presenting the conversion rule between these two types of channels. Then, we see that a binary erasure symmetric channel is an example of a regular channel.

Theorem 15.

A channel is a regular channel in the sense of [31] if and only if it can be written as a conditional additive channel.

To show the conversion from a conditional additive channel to a regular channel, we assume that the input alphabet $\mathcal{A}$ has an additive group structure. Let $P_{\tilde{X}}$ be a distribution on the output alphabet $\mathcal{B}$. Let ${\pi }_a$ be a representation of the group $\mathcal{A}$ on $\mathcal{B}$, and let $G=\{{\pi }_a:a\in \mathcal{A}\}$. A regular channel [31] is defined by:

$$\begin{array}{c}\hfill P_{B|A}\left(b\right|a)=P_{\tilde{X}}\left({\pi }_a\left(b\right)\right).\end{array}$$

(190)

The group action induces orbit:

$$\begin{array}{c}\hfill \mathrm{Orb}\left(b\right):=\{{\pi }_a\left(b\right):a\in \mathcal{A}\}.\end{array}$$

(191)

The set of all orbits constitutes a disjoint partition of $\mathcal{B}$. A set of the orbits is denoted by $\overline{\mathcal{B}}$, and let $\mathrm{Orb}:\mathcal{B}\to \overline{\mathcal{B}}$ be the map to the representatives.

Example 4

(Binary erasure symmetric channel). Let $\mathcal{A}=\{0,1\}$, $\mathcal{B}=\{0,1,?\}$, and:

$$\begin{array}{c}\hfill P_{\tilde{X}}\left(b\right)=\left\{\begin{array}{cc}1-p-p^{\prime }\hfill & ifb=0\hfill \\ p\hfill & ifb=1\hfill \\ p^{\prime }\hfill & ifb=?\hfill \end{array}\right..\end{array}$$

(192)

Then, let:

$$\begin{array}{c}\hfill {\pi }_0=\left[\begin{array}{ccc}0& 1& ?\\ 0& 1& ?\end{array}\right],{\pi }_1=\left[\begin{array}{ccc}0& 1& ?\\ 1& 0& ?\end{array}\right].\end{array}$$

(193)

The channel defined in this way is a regular channel (see Figure 5). In this case, there are two orbits: $\{0,1\}$ and $\{?\}$.

Let $\mathcal{B}=\mathcal{X}\times \mathcal{Y}$ and $P_{\tilde{X}}=P_{XY}$ for some joint distribution on $\mathcal{X}\times \mathcal{Y}$. Now, we consider a conditional additive channel, whose transition matrix $P_{B|A}(x,y|a)$ is given as $P_{XY}(x-a,y)$. When the group action is given by ${\pi }_a(x,y)=(x-a,y)$, the above conditional additive channel is given as a regular channel. In this case, there are $\left|\mathcal{Y}\right|$ orbits, and the size of each orbit is $\left|\mathcal{X}\right|$, respectively. This fact shows that any conditional additive channel is written as a regular channel. That is, it shows the “if” part of Theorem 15.

Conversely, we present the conversion from a regular channel to a conditional additive channel. We first explain the construction for the single-shot channel. For random variable $\tilde{X}\sim P_{\tilde{X}}$, let $\mathcal{Y}=\overline{\mathcal{B}}$ and $Y=\varpi \left(\tilde{X}\right)$ be the random variable describing the representatives of the orbits. For $y=\mathrm{Orb}\left(b\right)$ and each orbit $\mathrm{Orb}\left(b\right)$, we fix an element $0_y\in \mathrm{Orb}\left(b\right)$. Then, we define:

$$\begin{array}{c}\hfill P_Y\left(y\right):=P_{\tilde{X}}\left(\mathrm{Orb}\left(b\right)\right),P_{X,Y}(a,y):=\frac{P_{\tilde{X}}\left({\pi }_a\left(0_y\right)\right)}{\left|\right\{a^{\prime }\in \mathcal{A}|{\pi }_a\left(0_y\right)={\pi }_{a^{\prime }}\left(0_y\right)\}|}.\end{array}$$

(194)

Then, we obtain the virtual channel $P_{X,Y|A}$ as $P_{X,Y|A}(x,y|a):=P_{X,Y}(x-a,y)$. Using the conditional distributions $P_{X,Y|B}$ and $P_{B|X,Y}$ as:

$$\begin{array}{cc}\hfill P_{X,Y|B}(a,y|b)& =\left\{\begin{array}{cc}\frac{1}{\left|\right\{a^{\prime }\in \mathcal{A}|{\pi }_a\left(0_y\right)={\pi }_{a^{\prime }}\left(0_y\right)\}|}\hfill & \mathrm{when}b={\pi }_a\left(0_y\right)\hfill \\ 0\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(195)

$$\begin{array}{cc}\hfill P_{X,Y|B}(a,y|b)& =\left\{\begin{array}{cc}1\hfill & \mathrm{when}b={\pi }_a\left(0_y\right)\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(196)

we obtain the relations:

$$\begin{array}{c}\hfill P_{B|A}\left(b\right|a)=\sum _{x,y}{P}_{B|X,Y}\left(b\right|x,y)P_{X,Y|A}(x,y|a),P_{X,Y|A}(x,y|a)=\sum _b{P}_{X,Y|B}(x,y|b)P_{B|A}\left(b\right|a).\end{array}$$

(197)

These two equations show that the receiver information of the virtual conditional additive channel $P_{X,Y|A}$ and the receiver information of the regular channel $P_{B|A}$ can be converted into each other. Hence, we can say that a regular channel in the sense of [31] can be written as a conditional additive channel, which shows the “only if” part of Theorem 15.

Example 5

(Binary erasure symmetric channel revisited). We convert the regular channel of Example 4 to a conditional additive channel. Let us label the orbit $\{0,1\}$ as $y=0$ and $\{?\}$ as $y=1$. Let $0_0=0$ and $0_1=?$.

$$\begin{array}{cc}\hfill P_{X,Y}(x,0)& =\left\{\begin{array}{cc}1-p-p^{\prime }\hfill & ifx=0\hfill \\ p\hfill & ifx=0\hfill \end{array}\right.\hfill \end{array}$$

(198)

$$\begin{array}{cc}\hfill P_{X,Y}(x,1)& =\frac{p^{\prime }}{2}.\hfill \end{array}$$

(199)

When we consider the nth extension, a channel is given by:

$$\begin{array}{c}\hfill P_{B^n|A^n}\left(b^n\right|a^n)=P_{{\tilde{X}}^n}\left({\pi }_{a^n}\left(b^n\right)\right),\end{array}$$

(200)

where the nth extension of the group action is defined by ${\pi }_{a^n}\left(b^n\right)=({\pi }_{a_1}\left(b_1\right),\dots ,{\pi }_{a_n}\left(b_n\right))$.

Similarly, for n-fold extension, we can also construct the virtual conditional additive channel. More precisely, for ${\tilde{X}}^n\sim P_{{\tilde{X}}^n}$, we set $Y^n=\varpi \left({\tilde{X}}^n\right)=(\varpi \left({\tilde{X}}_1\right),\dots ,\varpi \left({\tilde{X}}_n\right))$ and:

$$\begin{array}{c}\hfill P_{X^n,Y^n}(x^n,y^n):=\frac{P_{{\tilde{X}}^n}\left({\pi }_{a^n}\left(0_{y^n}\right)\right)}{\left|\right\{{a^{\prime }}^n\in {\mathcal{A}}^n|{\pi }_{a^n}\left(0_{y^n}\right)={\pi }_{{a^{\prime }}^n}\left(0_{y^n}\right)\}|}.\end{array}$$

(201)

4.3. Conversion of the BPSK-AWGN Channel into the Conditional Additive Channel

Although we only considered finite input/output sources and channels throughout the paper, in order to demonstrate the utility of the conditional additive channel framework, let us consider the additive white Gaussian noise (AWGN) channel with binary phase shift keying (BPSK) in this section. Let $\mathcal{A}=\{0,1\}$ be the input alphabet of the channel, and let $\mathcal{B}=\mathbb{R}$ be the output alphabet of the channel. For an input $a\in \mathcal{A}$ and Gaussian noise Z with mean zero and variance ${\sigma }^2$, the output of the channel is given by $B={(-1)}^a+Z$. Then, the conditional probability density function of this channel is given as:

$$\begin{array}{c}\hfill P_{B|A}\left(b\right|a)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(b-{(-1)}^a)}^2}{{\sigma }^2}}.\end{array}$$

(202)

Now, to define a conditional additive channel, we choose $\mathcal{Y}:={\mathbb{R}}_{+}$ and define the probability density function $p_Y$ on $\mathcal{Y}$ with respect to the Lebesgue measure and the conditional distribution $P_{X|Y}\left(x\right|y)$ as:

$$\begin{array}{cc}\hfill p_Y\left(y\right)& :=\frac{1}{\sqrt{2\pi }\sigma }(e^{-\frac{{(y-1)}^2}{{\sigma }^2}}+e^{-\frac{{(y+1)}^2}{{\sigma }^2}})\hfill \end{array}$$

(203)

$$\begin{array}{cc}\hfill P_{X|Y}\left(0\right|y)& :=\frac{e^{-\frac{{(y-1)}^2}{{\sigma }^2}}}{e^{-\frac{{(y-1)}^2}{{\sigma }^2}}+e^{-\frac{{(y+1)}^2}{{\sigma }^2}}}\hfill \end{array}$$

(204)

$$\begin{array}{cc}\hfill P_{X|Y}\left(1\right|y)& :=\frac{e^{-\frac{{(y+1)}^2}{{\sigma }^2}}}{e^{-\frac{{(y-1)}^2}{{\sigma }^2}}+e^{-\frac{{(y+1)}^2}{{\sigma }^2}}}\hfill \end{array}$$

(205)

for $y\in {\mathbb{R}}_{+}$. When we define $b:={(-1)}^{x}y\in \mathbb{R}$ for $x\in \{0,1\}$ and $y\in {\mathbb{R}}_{+}$, we have:

$$\begin{array}{c}\hfill p_{XY|A}(y,x|a)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(y-{(-1)}^{a+x})}^2}{{\sigma }^2}}=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{({(-1)}^{x}y-{(-1)}^a)}^2}{{\sigma }^2}}=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{(b-{(-1)}^a)}^2}{{\sigma }^2}}.\end{array}$$

(206)

The relations (202) and (206) show that the AWGN channel with BPSK is given as a conditional additive channel in the above sense.

By noting this observation, as explained in Remark 5, the single-shot achievability bounds in Section 3.2 are also valid for continuous Y. Furthermore, the discussions for the single-shot converse bounds in Section 4.5 hold even for continuous Y. Therefore, the bounds in Section 4.4 and Section 4.5 are also applicable to the BPSK-AWGN channel.

In particular, in the n memoryless extension of the BPSK-AWGN channel, the information measures for the noise distribution are given as n times the single-shot information measures for the noise distribution. Even in this case, the upper and lower bounds in Section 4.4 and Section 4.5 are also applicable by replacing the information measures by n times the single-shot information measures. Therefore, we obtain finite-length upper and lower bounds of the optimal coding length for the memoryless BPSK-AWGN channel. Furthermore, even though the additive noise is not Gaussian, when the probability density function $p_Z$ of the additive noise Z satisfies the symmetry $p_Z\left(z\right)=p_Z(-z)$, the BPSK channel with the additive noise Z can be converted to a conditional additive channel in the same way.

4.4. Achievability Bound Derived by Source Coding with Side-Information

In this subsection, we give a code for a conditional additive channel from a code of source coding with side-information in a canonical way. In this construction, we see that the decoding error probability of the channel code equals that of the source code.

When the channel is given as the conditional additive channel with conditional additive noise distribution $P_{X^n{Y}^n}$ as (186) and $\mathcal{X}=\mathcal{A}$ is the finite field ${\mathbb{F}}_q$, we can construct a linear channel code from a source code with full side-information whose encoder and decoder are $f_n$ and ${\mathsf{d}}_n$ as follows. First, we assume linearity for the source encoder $f_n$. Let ${\mathcal{C}}_n\left(f_n\right)$ be the kernel of the linear encoder $f_n$ of the source code. Suppose that the sender sends a codeword $c_n\in {\mathcal{C}}_n\left(f_n\right)$ and $(c_n+X^n,Y^n)$ is received. Then, the receiver computes the syndrome $f_n(c_n+X^n)=f_n\left(c_n\right)+f_n\left(X^n\right)=f_n\left(X^n\right)$, estimates $X^n$ from $f_n\left(X^n\right)$ and $Y^n$, and subtracts the estimate from $c_n+X^n$. That is, we choose the channel decoder ${\tilde{\mathsf{d}}}_n$ as:

$$\begin{array}{c}\hfill {\tilde{\mathsf{d}}}_n(x^{\prime n},y^n):=x^{\prime n}-{\mathsf{d}}_n(f_n\left(x^{\prime n}\right),y^n).\end{array}$$

(207)

We succeed in decoding in this channel coding if and only if ${\mathsf{d}}_n(f_n\left(X^n\right),Y^n)$ equals $X^n$. Thus, the error probability of this channel code coincides with that of the source code for the correlated source $(X^n,Y^n)$. In summary, we have the following lemma, which was first pointed out in [27].

Lemma 19

([27], (19)). Given a linear encoder $f_n$ and a decoder ${\mathsf{d}}_n$ for a source code with side-information with distribution $P_{X^n{Y}^n}$, let $I_{\mathrm{Ker}\left(f_n\right)}$ and ${\tilde{\mathsf{d}}}_n$ be the channel encoder and decoder induced by $(f_n,{\mathsf{d}}_n)$. Then, the error probability of channel coding for the conditionally additive channel with noise distribution $P_{X^n{Y}^n}$ satisfies:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[(I_{\mathrm{Ker}\left(f_n\right)},{\tilde{\mathsf{d}}}_n)\right|P_{X^n{Y}^n}]={\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left[(f_n,{\mathsf{d}}_n)\right|P_{X^n{Y}^n}].\end{array}$$

(208)

Furthermore, (in fact, when we additionally impose the linearity on the random function F in the definition (114) for the definition of ${\overline{\mathrm{P}}}_{\mathrm{s}}\left(M\right|P_{X^n{Y}^n})$, the result in [27] implies that the equality in (209) holds) taking the infimum for $F_n$ chosen to be a linear two-universal hash function, we also have:

$$\begin{array}{cc}\hfill & {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)=\underset{F_n\in {\mathcal{F}}_l}{sup}{\mathbb{E}}_{F_n}\left[{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[\mathsf{\Psi }\left(F_n\right)\right]\right]\le \underset{F_n\in {\mathcal{F}}_l}{sup}{\mathbb{E}}_{F_n}\left[{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[(I_{\mathrm{Ker}\left(F_n\right)},{\tilde{\mathsf{d}}}_n)\right]\right]\hfill \\ \hfill =& \underset{F_n\in {\mathcal{F}}_l}{sup}{\mathbb{E}}_{F_n}{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left[(F_n,{\mathsf{d}}_n)\right]\le \underset{F_n\in \mathcal{F}}{sup}{\mathbb{E}}_{F_n}{\mathrm{P}}_{\mathrm{s}}^{\left(n\right)}\left[(F_n,{\mathsf{d}}_n)\right]={\overline{\mathrm{P}}}_{\mathrm{s}}^{\left(n\right)}\left(\right|{\mathcal{A}}^{n-k}\left|\right).\hfill \end{array}$$

(209)

By using this observation and the results in Section 3.2, we can derive the achievability bounds. By using the conversion argument in Section 4.2, we can also construct a channel code for a regular channel from a source code with full side-information. Although the following bounds are just a specialization of known bounds for conditional additive channels, we review these bounds here to clarify the correspondence between the bounds in source coding with side-information and channel coding.

From Lemma 13 and (209), we have the following.

Lemma 20

([2]). The following bound holds:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\le \underset{\gamma \ge 0}{inf}\left[P_{X^n{Y}^n}\left\{log\frac{1}{P_{X^n|Y^n}\left(x^n\right|y^n)}>\gamma \right\}+\frac{e^{\gamma }}{{\left|\mathcal{A}\right|}^{n-k}}\right].\end{array}$$

(210)

From Lemma 14 and (209), we have the following exponential-type bound.

Lemma 21

([6]). The following bound holds:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\le \underset{-\frac{1}{2}\le \theta \le 0}{inf}{\left|\mathcal{A}\right|}^{\frac{\theta (n-k)}{1+\theta }}{e}^{-\frac{\theta }{1+\theta }{H}_{1+\theta }^{\uparrow }\left(X^n\right|Y^n)}.\end{array}$$

(211)

From Lemma 15 and (209), we have the following slightly loose exponential bound.

Lemma 22

([3,70]). The following bound holds (The bound (212) was derived in the original Japanese edition of [3], but it is not written in the English edition [3]. The quantum analogue was derived in [70].):

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\le \underset{-1\le \theta \le 0}{inf}{\left|\mathcal{A}\right|}^{\theta (n-k)}{e}^{-\theta H_{1+\theta }^{\downarrow }\left(X^n\right|Y^n)}.\end{array}$$

(212)

When X has no side-information, i.e., the virtual channel is additive, we have the following special case of Lemma 21.

Lemma 23

([6]). Suppose that X has no side-information. Then, the following bound holds:

$$\begin{array}{c}\hfill {\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\le \underset{-\frac{1}{2}\le \theta \le 0}{inf}{\left|\mathcal{A}\right|}^{\frac{\theta (n-k)}{1+\theta }}{e}^{-\frac{\theta }{1+\theta }{H}_{1+\theta }\left(X^n\right)}.\end{array}$$

(213)

4.5. Converse Bound

In this subsection, we show some converse bounds. The following is the information spectrum-type converse shown in [4].

Lemma 24

([4], Lemma 4). For any code ${\mathsf{\Psi }}_n=({\mathsf{e}}_n,{\mathsf{d}}_n)$ and any output distribution $Q_{B^n}\in \mathcal{P}\left({\mathcal{B}}^n\right)$, we have:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left[{\mathsf{\Psi }}_n\right]\ge \underset{\gamma \ge 0}{sup}\left[\sum _{m=1}^{M_n}\frac{1}{M_n}{P}_{B^n|A^n}\left\{log\frac{P_{B^n|A^n}\left(b^n\right|{\mathsf{e}}_n\left(m\right))}{Q_{B^n}\left(b^n\right)}<\gamma \right\}-\frac{e^{\gamma }}{M_n}\right].\end{array}$$

(214)

When a channel is a conditional additive channel, we have:

$$\begin{array}{c}\hfill P_{B^n|A^n}(a^n+x^n,y^n|a^n)=P_{X^n{Y}^n}(x^n,y^n).\end{array}$$

(215)

By taking the output distribution $Q_{B^n}$ as:

$$\begin{array}{c}\hfill Q_{B^n}(a^n+x^n,y^n)=\frac{1}{{\left|\mathcal{A}\right|}^n}{Q}_{Y^n}\left(y^n\right)\end{array}$$

(216)

for some $Q_{Y^n}\in \mathcal{P}\left({\mathcal{Y}}^n\right)$, as a corollary of Lemma 24, we have the following bound.

Lemma 25.

When a channel is a conditional additive channel, for any distribution $Q_{Y^n}\in \mathcal{P}\left({\mathcal{Y}}^n\right)$, we have:

$$\begin{array}{c}\hfill {\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)\ge \underset{\gamma \ge 0}{sup}\left[P_{X^n{Y}^n}\left\{log\frac{Q_{Y^n}\left(y^n\right)}{P_{X^n{Y}^n}(x^n,y^n)}>nlog\left|\mathcal{A}\right|-\gamma \right\}-\frac{e^{\gamma }}{M_n}\right].\end{array}$$

(217)

Proof. 

By noting (215) and (216), the first term of the right-hand side of (214) can be rewritten as:

$$\begin{array}{cc}\hfill & \sum _{m=1}^{M_n}\frac{1}{M_n}{P}_{B^n|A^n}\left\{log\frac{P_{B^n|A^n}\left(b^n\right|{\mathsf{e}}_n\left(m\right))}{Q_{B^n}\left(b^n\right)}<\gamma \right\}\hfill \end{array}$$

(218)

$$\begin{array}{cc}\hfill & =\sum _{m=1}^{M_n}\frac{1}{M_n}{P}_{X^n{Y}^n}\left\{log\frac{P_{B^n|A^n}({\mathsf{e}}_n\left(m\right)+x^n,y^n|{\mathsf{e}}_n\left(m\right))}{Q_{B^n}({\mathsf{e}}_n\left(m\right)+x^n,y^n)}\right\}\hfill \end{array}$$

(219)

$$\begin{array}{cc}\hfill & =P_{X^n{Y}^n}\left\{log\frac{Q_{Y^n}\left(y^n\right)}{P_{X^n{Y}^n}(x^n,y^n)}>nlog\left|\mathcal{A}\right|-\gamma \right\},\hfill \end{array}$$

(220)

which implies the statement of the lemma. □

A similar argument as in Theorem 5 also derives from the following converse bound.

Theorem 16.

For any $Q_{Y^n}\in \mathcal{P}\left({\mathcal{Y}}^n\right)$, we have:

$$\begin{array}{cc}-\hfill & {\displaystyle log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)}\hfill \\ \hfill & \le \underset{\genfrac{}{}{0pt}{}{s>0}{\tilde{\theta }\in \mathbb{R},\vartheta \ge 0}}{inf}[(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})-H_{1+(1+s)\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})\right\}\hfill \end{array}$$

(221)

$$\begin{array}{cc}\hfill & -(1+s)log\left(1-2e^{-\frac{-\vartheta R+(\tilde{\theta }+\vartheta (1+\tilde{\theta }))H_{1+\tilde{\theta }+\vartheta (1+\tilde{\theta })}\left(P_{X^n{Y}^n}\right|Q_{Y^n})-(1+\vartheta )\tilde{\theta }{H}_{1+\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})}{1+\vartheta }}\right)]/s\hfill \\ \le & \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})-H_{1+(1+s)\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})\right\}\hfill \end{array}$$

(222)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(\theta \left(a\left(R\right)\right)-\tilde{\theta })a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}\left(P_{X^n{Y}^n}\right|Q_{Y^n})+\tilde{\theta }{H}_{1+\tilde{\theta }}\left(P_{X^n{Y}^n}\right|Q_{Y^n})}\right)]/s,\hfill \end{array}$$

(223)

where $R=nlog\left|\mathcal{A}\right|-log{M}_n$, and $\theta \left(a\right)$ and $a\left(R\right)$ are the inverse functions defined in (29) and (32), respectively.

Proof. 

See Appendix L. □

4.6. Finite-Length Bound for the Markov Noise Channel

From this section, we address the conditional additive channel whose conditional additive noise is subject to the Markov chain. Here, the input alphabet ${\mathcal{A}}^n$ equals the additive group ${\mathcal{X}}^n={\mathbb{F}}_q^n$, and the output alphabet ${\mathcal{B}}^n$ is ${\mathcal{X}}^{\times }{\mathcal{Y}}^n$. That is, the transition matrix describing the channel is given by using a transition matrix W on ${\mathcal{X}}^{\times }{\mathcal{Y}}^n$ and an initial distribution Q as:

$$\begin{array}{c}\hfill P_{B^n|A^n}(x^n+a^n,y^n|a^n)=Q(x_1,y_1)\prod _{i=2}^{n}W(x_i,y_i|x_{i-1},y_{i-1}).\end{array}$$

(224)

As in Section 2.2, we consider two assumptions on the transition matrix W of the noise process $(\mathbf{X},\mathbf{Y})$, i.e., Assumptions 1 and 2. We also use the same notations as in Section 2.2.

Example 6

(Gilbert–Elliot channel with state-information available at the receiver). The Gilbert–Elliot channel [29,30] is characterized by a channel state $Y^n$ on ${\mathcal{Y}}^n={\{0,1\}}^n$ and an additive noise $X^n$ on ${\mathcal{X}}^n={\{0,1\}}^n$. The noise process $(X^n,Y^n)$ is a Markov chain induced by the transition matrix W introduced in Example 3. For the channel input $a^n$, the channel output is given by $(a^n+X^n,Y^n)$ when the state-information is available at the receiver. Thus, this channel can be regarded as a conditional additive channel, and the transition matrix of the noise process satisfies Assumption 2.

Proofs of the following bounds are almost the same as those in Section 3.3, and thus omitted. The combination of Lemmas 10 and 22 derives the following achievability bound.

Theorem 17

(Direct, Ass. 1). Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 1. Let $R:=\frac{n-k}{n}log\left|\mathcal{A}\right|$. Then, we have:

$$\begin{array}{ccc}\hfill -log{\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)& \ge & \underset{-1\le \theta \le 0}{sup}\left[-\theta nR+(n-1)\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)+\underline{\delta }\left(\theta \right)\right].\hfill \end{array}$$

(225)

Theorem 16 for $Q_{Y^n}=P_{Y^n}$ and Lemma 10 yield the following converse bound.

Theorem 18

(Converse, Ass. 1). Suppose that transition matrix W of the conditional additive noise satisfies Assumption 1. Let $R:=log\left|\mathcal{A}\right|-\frac{1}{n}log{M}_n$. If $H^W\left(X\right|Y)<R<H_0^{\downarrow ,W}\left(X\right|Y)$, then we have:

$$\begin{array}{cc}-\hfill & {\displaystyle log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)}\hfill \\ \hfill & \le \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(n-1)(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)-H_{1+(1+s)\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)\right\}+{\delta }_1\hfill \end{array}$$

(226)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(n-1)[(\theta \left(a\left(R\right)\right)-\tilde{\theta })a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow ,W}\left(X\right|Y)+\tilde{\theta }{H}_{1+\tilde{\theta }}^{\downarrow ,W}\left(X\right|Y)]+{\delta }_2}\right)]/s,\hfill \end{array}$$

(227)

where $\theta \left(a\right)={\theta }^{\downarrow }\left(a\right)$ and $a\left(R\right)=a^{\downarrow }\left(R\right)$ are the inverse functions defined by (67) and (70), respectively, and:

$$\begin{array}{ccc}\hfill {\delta }_1& :=& (1+s)\overline{\delta }\left(\tilde{\theta }\right)-\underline{\delta }\left((1+s)\tilde{\theta }\right),\hfill \end{array}$$

(228)

$$\begin{array}{ccc}\hfill {\delta }_2& :=& \frac{(\theta \left(a\left(R\right)\right)-\tilde{\theta })R-(1+\tilde{\theta })\underline{\delta }\left(\theta \left(a\left(R\right)\right)\right)+(1+\theta \left(a\left(R\right)\right))\overline{\delta }\left(\tilde{\theta }\right)}{1+\theta \left(a\right(R\left)\right)}.\hfill \end{array}$$

(229)

Next, we derive tighter bounds under Assumption 2. From Lemmas 11 and 21, we have the following achievability bound.

Theorem 19

(Direct, Ass. 2). Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 2. Let $R:=\frac{n-k}{n}log\left|\mathcal{A}\right|$. Then, we have:

$$\begin{array}{c}\hfill -log{\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta nR+(n-1)\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }+\underline{\xi }\left(\theta \right).\end{array}$$

(230)

By using Theorem 16 for $Q_{Y^n}=P_{Y^n}^{(1+\theta (a\left(R\right)\left)\right)}$ and Lemma 12, we obtain the following converse bound.

Theorem 20

(Converse, Ass. 2). Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 2. Let $R:=log\left|\mathcal{A}\right|-\frac{1}{n}log{M}_n$. If $H^W\left(X\right|Y)<R<H_0^{\uparrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{cc}-\hfill & {\displaystyle log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(M_n\right)}\hfill \\ \hfill & \le \underset{\genfrac{}{}{0pt}{}{s>0}{-1<\tilde{\theta }<\theta \left(a\left(R\right)\right)}}{inf}[(n-1)(1+s)\tilde{\theta }\left\{H_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)-H_{1+(1+s)\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)\right\}+{\delta }_1\hfill \end{array}$$

(231)

$$\begin{array}{c}-(1+s)log\left(1-2e^{(n-1)[(\theta \left(a\left(R\right)\right)-\tilde{\theta })a\left(R\right)-\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y)+\tilde{\theta }{H}_{1+\tilde{\theta },1+\theta \left(a\left(R\right)\right)}^W\left(X\right|Y)]+{\delta }_2}\right)]/s,\hfill \end{array}$$

(232)

where $\theta \left(a\right)={\theta }^{\uparrow }\left(a\right)$ and $a\left(R\right)=a^{\uparrow }\left(R\right)$ are the inverse functions defined by (71) and (73), respectively, and:

$$\begin{array}{ccc}\hfill {\delta }_1& :=& (1+s)\overline{\zeta }(\tilde{\theta },\theta \left(a\left(R\right)\right))-\underline{\zeta }((1+s)\tilde{\theta },\theta \left(a\left(R\right)\right)),\hfill \end{array}$$

(233)

$$\begin{array}{ccc}\hfill {\delta }_2& :=& \frac{(\theta \left(a\left(R\right)\right)-\tilde{\theta })R-(1+\tilde{\theta })\underline{\zeta }(\theta \left(a\left(R\right)\right),\theta \left(a\left(R\right)\right))+(1+\theta \left(a\left(R\right)\right))\overline{\zeta }(\tilde{\theta },\theta \left(a\left(R\right)\right))}{1+\theta \left(a\right(R\left)\right)}.\hfill \end{array}$$

(234)

Finally, when X has no side-information, i.e., the channel is additive, we obtain the following achievability bound from Lemma 23.

Theorem 21

(Direct, no-side-information). Let $R:=\frac{n-k}{n}log\left|\mathcal{A}\right|$. Then, we have:

$$\begin{array}{c}\hfill -log{\overline{\mathrm{P}}}_{\mathrm{c}}(n,k)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta nR+(n-1)\theta H_{1+\theta }^W\left(X\right)+\underline{\delta }\left(\theta \right)}{1+\theta }.\end{array}$$

(235)

Remark 9.

Our treatment for the Markov conditional additive channel covers Markov regular channels because Markov regular channels can be reduced to Markov conditional additive channels as follows. Let $\tilde{\mathbf{X}}={\left\{{\tilde{X}}^n\right\}}_{n=1}^{\infty }$ be a Markov chain on $\mathcal{B}$ whose distribution is given by:

$$\begin{array}{c}\hfill P_{{\tilde{X}}^n}\left({\tilde{x}}^n\right)=Q\left({\tilde{x}}_1\right)\prod _{i=2}^n\tilde{W}\left({\tilde{x}}_i\right|{\tilde{x}}_{i-1})\end{array}$$

(236)

for a transition matrix $\tilde{W}$ and an initial distribution Q. Let $(\mathbf{X},\mathbf{Y})={\left\{(X^n,Y^n)\right\}}_{n=1}^{\infty }$ be the noise process of the conditional additive channel derived from the noise process $\tilde{\mathbf{X}}$ of the regular channel by the argument of Section 4.2. Since we can write:

$$\begin{array}{c}\hfill P_{X^n{Y}^n}(x^n,y^n)=Q\left({\iota }_{y_1}^{-1}\left({\vartheta }_{y_1}\left(x_1\right)\right)\right)\frac{1}{\left|\mathrm{Stb}\right(0_{y_1}\left)\right|}\prod _{i=2}^n\tilde{W}\left({\iota }_{y_i}^{-1}\left({\vartheta }_{y_i}\left(x_i\right)\right)\right|{\iota }_{y_{i-1}}^{-1}\left({\vartheta }_{y_{i-1}}\left(x_{i-1}\right)\right))\frac{1}{\left|\mathrm{Stb}\right(0_{y_i}\left)\right|},\end{array}$$

(237)

the process $(\mathbf{X},\mathbf{Y})$ is also a Markov chain. Thus, the regular channel given by $\tilde{\mathbf{X}}$ is reduced to the conditional additive channel given by $(\mathbf{X},\mathbf{Y})$.

4.7. Second-Order

To discuss the asymptotic performance, we introduce the quantity:

$$\begin{array}{c}\hfill C:=log\left|\mathcal{A}\right|-H^W\left(X\right|Y).\end{array}$$

(238)

By applying the central limit theorem (cf. [67] (Theorem 27.4, Example 27.6)) to Lemmas 20 and 25 for $Q_{Y^n}=P_{Y^n}$, and by using Theorem 2, we have the following.

Theorem 22.

Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 1. For arbitrary $\epsilon \in (0,1)$, we have:

$$\begin{array}{c}\hfill logM(n,\epsilon )=k(n,\epsilon )log\left|\mathcal{A}\right|=Cn+\sqrt{{\mathsf{V}}^W\left(X\right|Y)}{\mathsf{\Phi }}^{-1}\left(\epsilon \right)\sqrt{n}+o\left(\sqrt{n}\right).\end{array}$$

(239)

Proof. 

This theorem follows in the same manner as the proof of Theorem 11 by replacing Lemma 13 with Lemma 20 (achievability) and Lemma 18 with Lemma 25 (converse). □

From the above theorem, the (first-order) capacity of the conditional additive channel under Assumption 1 is given by:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}\frac{1}{n}logM(n,\epsilon )=\underset{n\to \infty }{lim}\frac{1}{n}log\frac{k(n,\epsilon )log\left|\mathcal{A}\right|}{n}=C\end{array}$$

(240)

for every $0<\epsilon <1$. In the next subsections, we consider the asymptotic behavior of the error probability when the rate is smaller than the capacity in the moderate deviation regime and the large deviation regime, respectively.

4.8. Moderate Deviation

From Theorems 17 and 18, we have the following.

Theorem 23.

Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 1. For arbitrary $t\in (0,1/2)$ and $\delta >0$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}-\frac{1}{n^{1-2t}}log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(e^{nC-n^{1-t}\delta }\right)& =& \underset{n\to \infty }{lim}-\frac{1}{n^{1-2t}}log{\overline{\mathrm{P}}}_{\mathrm{c}}^{\left(n\right)}\left(n,\frac{nC-n^{1-t}\delta }{log\left|\mathcal{A}\right|}\right)\hfill \end{array}$$

(241)

$$\begin{array}{cc}=& \frac{{\delta }^2}{2{\mathsf{V}}^W\left(X\right|Y)}.\hfill \end{array}$$

(242)

Proof. 

The theorem follows in the same manner as Theorem 12 by replacing Theorem 6 with Theorem 17 (achievability) and Theorem 8 with Theorem 18 (converse). □

4.9. Large Deviation

From Theorem 17 and Theorem 18, we have the following.

Theorem 24.

Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 1. For $H^W\left(X\right|Y)<R$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{c}}^{\left(n\right)}\left(n,n\left(1-\frac{R}{log\left|\mathcal{A}\right|}\right)\right)\ge \underset{-1\le \theta \le 0}{sup}\left[-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)\right].\end{array}$$

(243)

On the other hand, for $H^W\left(X\right|Y)<R<H_0^{\downarrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(e^{n(log|\mathcal{A}|-R)}\right)& \le & -\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\downarrow ,W}\left(X\right|Y)\hfill \end{array}$$

(244)

$$\begin{array}{cc}=& \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\downarrow ,W}\left(X\right|Y)}{1+\theta }.\hfill \end{array}$$

(245)

Proof. 

The theorem follows in the same manner as Theorem 13 by replacing Theorem 6 with Theorem 17 (achievability) and Theorem 8 with Theorem 18 (converse). □

Under Assumption 2, from Theorems 19 and 20, we have the following tighter bound.

Theorem 25.

Suppose that the transition matrix W of the conditional additive noise satisfies Assumption 2. For $H^W\left(X\right|Y)<R$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{c}}^{\left(n\right)}\left(n,n\left(1-\frac{R}{log\left|\mathcal{A}\right|}\right)\right)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }.\end{array}$$

(246)

On the other hand, for $H^W\left(X\right|Y)<R<H_0^{\uparrow ,W}\left(X\right|Y)$, we have:

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(e^{n(log|\mathcal{A}|-R)}\right)& \le & -\theta \left(a\left(R\right)\right)a\left(R\right)+\theta \left(a\left(R\right)\right)H_{1+\theta \left(a\right(R\left)\right)}^{\uparrow ,W}\left(X\right|Y)\hfill \end{array}$$

(247)

$$\begin{array}{cc}=& \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^{\uparrow ,W}\left(X\right|Y)}{1+\theta }.\hfill \end{array}$$

(248)

Proof. 

The theorem follows the same manner as Theorem 14 by replacing Theorem 9 with Theorem 19 and Theorem 10 with Theorem 20. □

When X has no side-information, i.e., the channel is additive, from Theorem 21 and (245), we have the following.

Theorem 26.

For $H^W\left(X\right)<R$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; inf}-\frac{1}{n}log{\overline{\mathrm{P}}}_{\mathrm{c}}^{\left(n\right)}\left(n,n\left(1-\frac{R}{log\left|\mathcal{A}\right|}\right)\right)\ge \underset{-\frac{1}{2}\le \theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^W\left(X\right)}{1+\theta }.\end{array}$$

(249)

On the other hand, for $H^W\left(X\right)<R<H_0^W\left(X\right)$, we have:

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}-\frac{1}{n}log{\mathrm{P}}_{\mathrm{c}}^{\left(n\right)}\left(e^{n(log|\mathcal{A}|-R)}\right)\le \underset{-1<\theta \le 0}{sup}\frac{-\theta R+\theta H_{1+\theta }^W\left(X\right)}{1+\theta }.\end{array}$$

(250)

Proof. 

The first claim follows by taking the limit of Theorem 21, and the second claim follows as a special case of (245) without side-information. □

4.10. Summary of the Results

The results shown in this section for the Markov conditional additive noise are summarized in

Table 3. Summary of the finite-length bounds for channel coding.

Ach./Conv.	Markov	Single-Shot	${\mathbf{P}}_{\mathbf{c}}$/${\overline{\mathbf{P}}}_{\mathbf{c}}$	Complexity	Large Deviation	Moderate Deviation	Second Order
Achievability	Theorem 17 (Ass. 1)	Lemma 22	${\overline{\mathrm{P}}}_{\mathrm{c}}$	$O\left(1\right)$		🗸
	Theorem 19 (Ass. 2)	Lemma 21	${\overline{\mathrm{P}}}_{\mathrm{c}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Theorem 21 (Additive)	Lemma 23	${\overline{\mathrm{P}}}_{\mathrm{c}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Lemma 20		${\overline{\mathrm{P}}}_{\mathrm{c}}$	Tail		🗸	🗸
Converse	Theorem 18 (Ass. 1)	(Theorem 16)	${\mathrm{P}}_{\mathrm{c}}$	$O\left(1\right)$		🗸
	Theorem 20 (Ass. 2)	Theorem 16	${\mathrm{P}}_{\mathrm{c}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Theorem 18 (Additive)	(Theorem 16)	${\mathrm{P}}_{\mathrm{c}}$	$O\left(1\right)$	🗸 ${}^{*}$	🗸
	Lemma 25		${\mathrm{P}}_{\mathrm{c}}$	Tail		🗸	🗸

. The check marks 🗸 indicate that the tight asymptotic bounds (large deviation, moderate deviation, and second-order) can be obtained from those bounds. The marks 🗸${}^{*}$ indicate that the large deviation bound can be derived up to the critical rate. The computational complexity “Tail” indicates that the computational complexities of those bounds depend on the computational complexities of tail probabilities. It should be noted that Theorem 18 is derived from a special case ($Q_Y=P_Y$) of Theorem 16. The asymptotically optimal choice is $Q_Y=P_Y^{(1+\theta )}$. Under Assumption 1, we can derive the bound of the Markov case only for that special choice of $Q_Y$, while under Assumption 2, we can derive the bound of the Markov case for the optimal choice of $Q_Y$. Furthermore, Theorem 18 is not asymptotically tight in the large deviation regime in general, but it is tight if X has no side-information, i.e., the channel is additive. It should be also noted that Theorem 20 does not imply Theorem 18 even for the additive channel case since Assumption 2 restricts the structure of transition matrices even when X has no side-information.

5. Discussion and Conclusions

In this paper, we developed a unified approach to source coding with side-information and channel coding for a conditional additive channel for finite-length and asymptotic analyses of Markov chains. In our approach, the conditional Rényi entropies defined for transition matrices played important roles. Although we only illustrated the source coding with side-information and the channel coding for a conditional additive channel as applications of our approach, it could be applied to some other problems in information theory such as random number generation problems, as shown in another paper [60].

Our obtained results for the source coding with side-information and the channel coding of the conditional additive channel has been extended to the case when the side-information is continuous like the real line and the joint distribution X and Y is memoryless. Since this case covers the BPSK-AWGN channel, it can be expected that it covers the MPSK-AWGN channel. Since such channels are often employed in the real channel coding, it is an interesting future topic to investigate the finite-length bound for these channels. Further, we could not define the conditional Rényi entropy for transition matrices of continuous Y. Hence, our result could not be extended to such a continuous case. It is another interesting future topic to extend the obtained result to the case with continuous Y.