Utility–Privacy Trade-Offs with Limited Leakage for Encoder

Abstract

The utilization of databases such as IoT has progressed, and understanding how to protect the privacy of data is an important issue. As pioneering work, in 1983, Yamamoto assumed the source (database), which consists of public information and private information, and found theoretical limits (first-order rate analysis) among the coding rate, utility and privacy for the decoder in two special cases. In this paper, we consider a more general case based on the work by Shinohara and Yagi in 2022. Introducing a measure of privacy for the encoder, we investigate the following two problems: The first problem is the first-order rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder, in which utility is measured by the expected distortion or the excess-distortion probability. The second task is establishing the strong converse theorem for utility–privacy trade-offs, in which utility is measured by the excess-distortion probability. These results may lead to a more refined analysis such as the second-order rate analysis.

1. Introduction

1.1. Background

The utilization of database has progressed in our society and includes autonomous cars and the congestion data service over the Internet. At the same time, the risk of accidental or intentional leakage of private information has also increased rapidly. To protect private information, coding with a privacy constraint has been analyzed via an information-theoretic approach. In 1983, Yamamoto [1] introduced a framework to quantify the utility of databases and the privacy of personal information and analyzed the trade-offs between them. Decades later, in 2013, Sankar et al. [2] claimed the necessity of converting databases to protect privacy while maintaining the utility of data. Then, Yamamoto’s framework [1] was re-recognized by Sankar et al. and other researchers. Using the rate-distortion theory in information theory, he revealed the optimal relationships (theoretical limits) among coding rate, utility, and privacy in two cases; (i) public information that can be open to the public and private information that should be protected from a third party are encoded, and (ii) only public information is encoded. However, since a more general case, i.e., where (iii) public information and a part of private information is encoded, had not been clarified, Shinohara and Yagi [3] derived the theoretical limits in such a case (see Figure 1). As a result, our characterization of the achievable region gives a “unified expression” because it includes the characteristics given in [1] in cases (i) and (ii) as special cases.

1.2. Motivation and Contributions

By investigating case (iii), one can compare the theoretical limits corresponding to a variety of patterns of the encoded information. One can see that the achievable region in case (i) is the largest among all patterns. However, this may not be the case if privacy leakage for the encoder is constrained. Motivated by this observation, in this paper, we characterize the optimal trade-offs among coding rate, utility, privacy for the decoder, and privacy for the encoder in Section 3. The addressed problem corresponds to the case where there are some aggregators between the source and the encoder and the aggregator controls the data (source sequence) passing to the encoder. The obtained results indeed suggest that the best-encoded information can be in case (iii) if some restriction is imposed on the privacy leakage for the encoder.

One of the most important tasks in information-theoretic analysis for utility–privacy trade-offs is second-order rate analysis (e.g., [4,5,6]). In general, in second-order rate analysis, the excess-distortion probability is used as a measure of utility [4,5,6]. However, in the first-order rate analysis shown in [3], utility is measured by the expected distortion, so for second-order rate analysis, we need first to conduct first-order rate analysis, which replaces the expected distortion with the excess-distortion probability as the measure of utility. In Section 4, the theoretical limits coincide with the one in which utility is measured by expected distortion.

There is one more problem to solve before tackling second-order rate analysis: we need to clarify whether the boundary of the achievable region may vary or not, depending on the value of the excess-distortion probability. In Section 5, we establish the strong converse theorem, provided that utility is measured by the probability of excess distortion. For the sake of simplicity, we focus on the achievable region of utility and privacy for the decoder or a third party, which reveals an aspect of utility–privacy trade-offs. In the proof, we adopt a change in measure argument developed by Tyagi and Watanabe [7]. Contrary to the standard rate-distortion problem, the alphabets of the encoder’s input and the decoder’s output are different, so we extend the argument to incorporate this discrepancy. Although the strong converse theorem is shown for the rate region of utility and privacy, we can also derive the same result when the privacy of the encoder is involved.

For readers’ convenience, Figure 2 shows the road map to the most important task: the second-order rate analysis. In summary, three contributions of this paper are as follows:

• The rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder in which utility is measured using the expected distortion (Section 3).
• The rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder in which utility is measured using the excess-distortion probability (Section 4).
• The strong converse theorem for utility–privacy trade-offs in which utility is measured using the excess-distortion probability (Section 5).

1.3. Related Work

The analysis of the utility–privacy trade-offs using an information-theoretic approach was initiated by [2], which translates the rate-distortion problem with an equivocation constraint in [1] into the privacy and utility trade-off problem. In information-theoretic studies on coding with privacy and utility constraints, several measures for privacy and utility are adopted. One of the strong measures for privacy is differential privacy [8,9], and an extension and relaxation of differential privacy have been proposed in [10,11]. A weaker but useful privacy measure is the mutual information between the codeword and private information [1,2,12,13,14], which guarantees the average amount of leaked private information. Other examples of well-known privacy measures are maximal leakage [15], maximal $\alpha$-leakage [16,17,18], and total variation [19]. Relationships among several measures for privacy have been revealed in [20]. On the other hand, well-known utility measures are average distortion [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25], hard distortion [16,17], and log-loss distortion [26].

Coding systems in the utility–privacy problem are extended to the ones with the encoder’s side information [2] and with the decoder’s side information [25]. In [14], a related coding problem has been investigated, where both the encoder and the decoder can access a uniform secret key and the decoder can also access side information. Utility–privacy trade-off schemes are applied, for example, to the Internet of Energy [23] and to a system with informational self-determination [24].

A closely related study to this paper was given by Basciftci et al. [13], in which several release mechanisms of encoded information from the database were discussed. In particular, utility–privacy trade-offs (without the coding rate) were compared when the encoded information was (i) both private and public information, (ii) only public information, and (iv) only private information (see also the three cases described in Section 1.1). A sufficient condition under which the utility–privacy trade-offs coincide for cases (i) and (ii) was given.

1.4. Organization

This paper is organized as follows: In Section 2, we begin by introducing the notation and system model that are used in this paper. In Section 3, we give the first-order rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder in which utility is measured by the expected distortion. In Section 4, we tackle the first-order rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder in which utility is measured by the excess-distortion probability. Section 5 focuses on the strong converse theorem for utility–privacy trade-offs in which utility is measured by the excess-distortion probability. In Section 6, we discuss the significance of the encoded information with limited leakage for the encoder. Finally, in Section 7, the conclusion and future work are stated.

2. Notation and System Model

2.1. Information Source

Database d is described by a $K\times n$ matrix whose rows represent K attributes and columns represent n entries of data. Let $\mathcal{K}=\{1,2,\dots ,K\}$ be the set of indexes of K attributes. The random variable for the lth attribute is denoted by $X_l$, which takes a value in a finite alphabet ${\mathcal{X}}_l$. For any subset $\mathcal{B}\subseteq \mathcal{K}$, the tuple of random variables ${\left(X_l\right)}_{l\in \mathcal{B}}$ is abbreviated as $X_{\mathcal{B}}$. Similarly, the Cartesian product of alphabets ${\prod }_{l\in \mathcal{B}}{\mathcal{X}}_l$ is abbreviated as ${\mathcal{X}}_{\mathcal{B}}$.

The K attributes can be divided into two groups; one may be open to the public and the other should be kept secret from a third party. Then, the set $\mathcal{K}$ is divided into disjoint sets $\mathcal{R}$ and $\mathcal{H}$. That is,

$$\begin{array}{c}\hfill \mathcal{K}=\mathcal{R}\cup \mathcal{H},\mathcal{R}\cap \mathcal{H}=\varnothing ,{\mathcal{X}}_{\mathcal{K}}={\mathcal{X}}_{\mathcal{R}}\times {\mathcal{X}}_{\mathcal{H}},\end{array}$$

(1)

where ${\mathcal{X}}_{\mathcal{R}}$ is the set of values that public (revealed) source symbols $X_{\mathcal{R}}$ take and ${\mathcal{X}}_{\mathcal{H}}$ is the set of values that private (hidden) source symbols $X_{\mathcal{H}}$ take.

We assume that the source sequence $X_{\mathcal{K}}^n=(X_{\mathcal{K},1},X_{\mathcal{K},2},\dots ,X_{\mathcal{K},n})$ is generated from a stationary and memoryless source $p_{X_{\mathcal{K}}}$. That is,

$$\begin{array}{c}\hfill P_{X_{\mathcal{K}}^n}\left(x_{\mathcal{K}}^n\right)=Pr\{X_{\mathcal{K}}^n=x_{\mathcal{K}}^n\}=\prod _{i=1}^n{P}_{X_{\mathcal{K}}}\left(x_{\mathcal{K},i}\right),\end{array}$$

(2)

where $x_{\mathcal{K}}^n=(x_{\mathcal{K},1},\dots ,x_{\mathcal{K},n})\in {\mathcal{X}}_{\mathcal{K}}^n$. Taking the partition of attributes in (1) into account, the source sequence $X_{\mathcal{K}}^n$ is described as

$$\begin{array}{c}\hfill X_{\mathcal{K}}^n=(X_{\mathcal{R}}^n,X_{\mathcal{H}}^n),\end{array}$$

(3)

where

$$\begin{array}{cc}\hfill & X_{\mathcal{R}}^n=(X_{\mathcal{R},1},X_{\mathcal{R},2},\dots ,X_{\mathcal{R},n})\in {\mathcal{X}}_{\mathcal{R}}^n,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & X_{\mathcal{H}}^n=(X_{\mathcal{H},1},X_{\mathcal{H},2},\dots ,X_{\mathcal{H},n})\in {\mathcal{X}}_{\mathcal{H}}^n\hfill \end{array}$$

(5)

are referred to as the revealed source sequence and the hidden source sequence, respectively. In the addressed coding system introduced in [22], the revealed symbols and a part of the hidden symbols are input to the encoder, and thus the encoded alphabet $\mathcal{E}$ satisfies $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$. Similar to (3), $X_{\mathcal{K}}^n$ is sometimes described as

$$\begin{array}{c}\hfill X_{\mathcal{K}}^n=(X_{\mathcal{E}}^n,X_{{\mathcal{E}}^{\mathrm{c}}}^n),\end{array}$$

(6)

where $X_{\mathcal{E}}^n$ is the source sequence observed by the encoder and ${\mathcal{E}}^{\mathrm{c}}=\mathcal{K}\setminus \mathcal{E}$.

2.2. Encoder and Decoder

The coding system consists of encoder $f_n$ and decoder $g_n$ as in Figure 1. When the source sequence $X_{\mathcal{K}}^n=(X_{\mathcal{E}}^n,X_{{\mathcal{E}}^{\mathrm{c}}}^n)$ is generated from the stationary and memoryless source $p_{X_{\mathcal{K}}}$, the codeword $J_n=f_n\left(X_{\mathcal{E}}^n\right)$ is generated by the encoder

$$\begin{array}{c}\hfill f_n:{\mathcal{X}}_{\mathcal{E}}^n\to \{1,2,\dots ,M_n\}\end{array}$$

(7)

and the reproduced sequence ${\widehat{X}}_{\mathcal{R}}^n=g_n\left(J_n\right)$ is produced by decoder

$$\begin{array}{c}\hfill g_n:\{1,2,\dots ,M_n\}\to {\widehat{\mathcal{X}}}_{\mathcal{R}}^n,\end{array}$$

(8)

where $M_n$ denotes the number of codewords.

3. First-Order Rate Analysis with Expected Distortion

3.1. Performance Measures

In this section, we mention the measure of the coding rate, utility, privacy for the decoder, and privacy for the encoder. Hereafter, let a pair of the encoder and decoder $(f_n,g_n)$ be fixed.

For a given $M_n$, the coding rate is defined as

$$\begin{array}{c}\hfill r_n≔\frac{1}{n}\log M_n.\end{array}$$

(9)

Let $d:{\mathcal{X}}_{\mathcal{R}}\times {\widehat{\mathcal{X}}}_{\mathcal{R}}\to [0,\infty )$ be a distortion function between $x_{\mathcal{R}}\in {\mathcal{X}}_{\mathcal{R}}$ and ${\widehat{x}}_{\mathcal{R}}\in {\widehat{\mathcal{X}}}_{\mathcal{R}}$. The distortion between sequences $x_{\mathcal{R}}^n\in {\mathcal{X}}_{\mathcal{R}}^n$ and ${\widehat{x}}_{\mathcal{R}}^n\in {\widehat{\mathcal{X}}}_{\mathcal{R}}^n$ is defined as

$$\begin{array}{cc}\hfill d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)≔& \sum _{i=1}^{n}d(x_{\mathcal{R},i},{\widehat{x}}_{\mathcal{R},i}).\hfill \end{array}$$

(10)

Then, the measure of utility is defined as

$$\begin{array}{cc}\hfill u_n≔& \mathbb{E}\left[\frac{1}{n}d(X_{\mathcal{R}}^n,{\widehat{X}}_{\mathcal{R}}^n)\right],\hfill \end{array}$$

(11)

where $\mathbb{E}$ represents the expectation by the joint distribution of $(X_{\mathcal{R}}^n,{\widehat{X}}_{\mathcal{R}}^n)$.

In this system, the privacy of the hidden source sequence $X_{\mathcal{H}}^n$ should be protected when the codeword $J_n$ is observed by decoder $g_n$. The measure of privacy for the decoder is defined as

$$\begin{array}{c}\hfill l_n≕\frac{1}{n}I(X_{\mathcal{H}}^n;J_n),\end{array}$$

(12)

where $I(X_{\mathcal{H}}^n;J_n)$ is the mutual information between $X_{\mathcal{H}}^n$ and $J_n$.

The privacy of the hidden source sequence $X_{\mathcal{H}}^n$ should be protected when the encoded information $X_{\mathcal{E}}$ is observed by encoder $f_n$. The measurement of privacy for the encoder is defined as

$$\begin{array}{c}\hfill e_n≔\frac{1}{n}I(X_{\mathcal{H}}^n;X_{\mathcal{E}}^n),\end{array}$$

(13)

where $I(X_{\mathcal{H}}^n;X_{\mathcal{E}}^n)$ is the mutual information between $X_{\mathcal{H}}^n$ and $X_{\mathcal{E}}^n$.

3.2. Achievable Region and Theorem

We define the achievable region for the first-order rate analysis with the expected distortion and state the obtained results.

Definition 1.

A tuple $(R,D,L,E)$ is said to be $\mathit{ϵ}$-achievable (with respect to the expected distortion measure) if, for any given $ϵ>0$, there exists a sequence of codes $(f_n,g_n)$ satisfying

$$\begin{array}{cc}\hfill r_n& \le R+ϵ,\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill u_n& \le D+ϵ,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill l_n& \le L+ϵ,\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill e_n& \le E+ϵ\hfill \end{array}$$

(17)

for all sufficiently large n.

The technical meanings of each constraint in Definition 1 can be interpreted as follows: Equation (14) evaluates how much the source sequence is compressed, so this rate should be decreased. Equation (15) is the constraint corresponding to distortion being less than $D+ϵ$. The smaller the distortion is, the better the utility is, so this condition should also be decreased. Equation (16) constrains the amount of leaked private information to the decoder. Since private information should be kept secret for the receiver, this quantity should be decreased as well. Equation (17) constrains the amount of private information leaked to the encoder. For the same reason as (16), this quantity should also be decreased.

Remark 1.

The minimum coding rate R for a fixed D corresponds to the rate-distortion function (Section 10 in [27]). Thus, in the proof of achievability, we evaluate the coding rate and the distortion with the argument in rate-distortion theory. This view is also important to correctly understand the numerical results in Section 6.1.

Definition 2.

The closure of the set of ϵ-achievable tuples $(R,D,L,E)$ is referred to as the $\mathit{ϵ}$-achievable region and is denoted by ${\mathcal{C}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$ and defines

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{0<ϵ<1}{\mathcal{C}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}}).\end{array}$$

(18)

To characterize the achievable region, we define the following informational region.

Definition 3.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, ${\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ is defined as

$$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)=\{(R,D,L,E):& R\ge I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill & D\ge \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill & L\ge I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill & E\ge I(X_{\mathcal{H}};X_{\mathcal{E}})\hfill \\ \hfill & \mathrm{for}\mathrm{some}{P}_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}\}.\hfill \end{array}$$

(19)

We establish the next theorem. For the proof of this theorem, please refer to Section 3.3, Section 3.4 and Section 3.5.

Theorem 1.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the achievable region of the coding system is given by

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(20)

To clarify the relationship with the conventional result of Shinohara and Yagi [3], we mention the achievable region among the coding rate, utility, and privacy, which is derived by projecting the result of Theorem 1 onto the $R\text{-}D\text{-}L$ hyperplane.

Definition 4.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}^{RDL}\left(ϵ\right|P_{X_{\mathcal{K}}})≔\{(R,D,L):(R,D,L,E)\in {\mathcal{C}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})\}\end{array}$$

(21)

and

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{0<ϵ<1}{\mathcal{C}}_{\mathcal{E}}^{RDL}\left(ϵ\right|P_{X_{\mathcal{K}}}).\end{array}$$

(22)

Definition 5.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)=\{(R,D,L):& R\ge I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill & D\ge \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill & L\ge I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})\hfill \\ \hfill & \mathrm{for}\mathrm{some}{P}_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}\}.\hfill \end{array}$$

(23)

Corollary 1.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the region ${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ is given by

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(24)

Remark 2.

Corollary 1 suggests that the conventional result [3] can be obtained from ${\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$.

Remark 3.

The derived characterization in (24) reduces to the characterization given in [1] when the encoded attribute $\mathcal{E}$ is either $\mathcal{K}$ or $\mathcal{R}$. Thus, (24) gives its generalization for $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$.

Examples to illustrate this result are shown in Section 6.1.

3.3. Proof Preliminaries for First-Order Rate Analysis

For preliminaries for coding theorems by the first-order rate analysis, we define strongly typical sequences that are necessary for the proof and show some properties. These proof preliminaries are also used in Section 4.

Definition 6

(Definition 2.1, [28]). The type of a sequence $x^n\in {\mathcal{X}}^n$ of length n is the distribution $P_{x^n}$ on $\mathcal{X}$ defined by

$$\begin{array}{c}\hfill P_{x^n}\left(a\right)≔\frac{1}{n}N\left(a\right|x^n),\end{array}$$

(25)

where $N\left(a\right|x^n)$ represents the number of occurrences of symbol $a\in \mathcal{X}$ in $x^n$. Likewise, the joint type of $x^n\in {\mathcal{X}}^n$ and $y^n\in {\mathcal{Y}}^n$ is the distribution $P_{x^n{y}^n}$ on $\mathcal{X}\times \mathcal{Y}$ defined by

$$\begin{array}{c}\hfill P_{x^n{y}^n}≔\frac{1}{n}N(a,b|x^n,y^n),\end{array}$$

(26)

where $N(a,b|x^n,y^n)$ represents the number of the occurrences of $(a,b)\in \mathcal{X}\times \mathcal{Y}$ in the pair of sequences $(x^n,y^n)$.

Definition 7

((Conditional Type), [28], Definition 2.2). We define the conditional type of $y^n$ given $x^n$ as a stochastic matrix $V:\mathcal{X}\to \mathcal{Y}$ satisfying

$$\begin{array}{c}\hfill N(a,b|x^n,y^n)=N\left(a\right|x^n)V\left(b\right|a).\end{array}$$

(27)

In particular, the conditional type of $y^n$ given $x^n$ is uniquely determined and given by

$$\begin{array}{c}\hfill V\left(b\right|a)=\frac{N(a,b|x^n,y^n)}{N\left(a\right|x^n)}\end{array}$$

(28)

if $N\left(a\right|x^n)>0$ for any $a\in \mathcal{X}$.

Definition 8

((Strongly Typical Sequences), [29], Definition 1.2.8). For any distribution P on $\mathcal{X}$, a sequence $x^n\in {\mathcal{X}}^n$ is said to be P-typical with constant $\delta >0$ if

$$\begin{array}{c}\hfill \left|\frac{1}{n}N\left(a\right|x^n)-P\left(a\right)\right|\le \delta \mathrm{for}\mathrm{every}a\in \mathcal{X}\end{array}$$

(29)

and, in addition, no $a\in \mathcal{X}$ with $P\left(a\right)=0$ occurs in $x^n$. The set of such sequences is denoted by $T_{\delta }^n\left(P\right)$. If X is a random variable with values in $\mathcal{X}$, we also refer to P-typical sequences as X-typical sequences and write $T_{\delta }^n\left(X\right)$.

Definition 9

((Conditional Strongly Typical Sequences), [29], Definition 1.2.9). For a stochastic matrix $W:\mathcal{X}\to \mathcal{Y}$, a sequence $y^n\in {\mathcal{Y}}^n$ is said to be W-typical given $x^n\in {\mathcal{X}}^n$ with constant $\delta >0$ if

$$\begin{array}{cc}\hfill \left|\frac{1}{n}N(a,b|x^n,y^n)-\frac{1}{n}N\left(a\right|x^n)W\left(b\right|a)\right|\le \delta & \hfill \\ \hfill \mathrm{for}\mathrm{every}a\in \mathcal{X},b\in \mathcal{Y},& \hfill \end{array}$$

(30)

and, in addition, $N(a,b|x^n,y^n)=0$ whenever $W\left(b\right|a)=0$. The set of such sequences $y^n$ is denoted by $T_{\delta }^n\left(W\right|x^n)$. Further, if X and Y are random variables with values in $\mathcal{X}$ and $\mathcal{Y}$, respectively, and $P_{Y|X}=W$, then they are also said to be $Y|X$-typical and written as $T_{\delta }^n\left(Y\right|X|x^n)$.

Hereafter, the set of conditional strongly typical sequences $T_{\delta }^n\left(Y\right|X|x^n)$ is abbreviated as $T_{\delta }^n\left(Y\right|x^n)$.

We state some lemmas that are used in this proof.

Lemma 1

([29], Lemma 1.2.13). For any positive sequences ${\left\{{\delta }_n\right\}}_{n=1}^{\infty }$ and ${\left\{{\delta }_n^{\prime }\right\}}_{n=1}^{\infty }$ such that ${\delta }_n\to 0$ and ${\delta }^{\prime }\to 0$ as $n\to 0$, there exists a sequence ${ϵ}_n={ϵ}_n\left(\right|\mathcal{X},\mathcal{Y}|,{\delta }_n,{\delta }_n^{\prime })\to 0$$(n\to \infty )$ such that for every distribution P on $\mathcal{X}$ and stochastic matrix $W:\mathcal{X}\to \mathcal{Y}$,

$$\begin{array}{cc}\hfill \left|\frac{1}{n}\log |T_{{\delta }_n}^n\left(P\right)|-H\left(P\right)\right|& \le {ϵ}_n,\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \left|\frac{1}{n}\log |T_{{\delta }_n^{\prime }}^n\left(W\right|x^n)|-H\left(W\right|P)\right|& \le {ϵ}_n.\hfill \end{array}$$

(32)

Lemma 2

([29], Lemma 1.2.7). Let the variational distance between two distributions P and Q on $\mathcal{X}$ be defined as

$$\begin{array}{c}\hfill d_{\mathrm{v}}(P,Q)≔\sum _{x\in \mathcal{X}}|P\left(x\right)-Q\left(x\right)|.\end{array}$$

(33)

If $d_{\mathrm{v}}(P,Q)<\frac{1}{2}$, then

$$\begin{array}{c}\hfill |H\left(P\right)-H\left(Q\right)|\le -d_{\mathrm{v}}(P,Q)·\log \frac{d_{\mathrm{v}}(P,Q)}{\left|\mathcal{X}\right|}.\end{array}$$

(34)

Lemma 3

([29], Lemma 1.2.10). If $x^n\in T_{\delta }^n\left(X\right)$ and $y^n\in T_{{\delta }^{\prime }}^n\left(Y\right|x^n)$, then $(x^n,y^n)\in T_{\delta +{\delta }^{\prime }}^n(X,Y)$ and, consequently, $y^n\in T_{{\delta }^{″}}\left(Y\right)$ for ${\delta }^{″}≔(\delta +{\delta }^{\prime })·\left|\mathcal{X}\right|$.

Lemma 4.

If $(x^n,y^n)\in T_{\delta }^n(X,Y)$, then $x^n\in T_{{\delta }_1}^n\left(X\right)$ and, consequently, $y^n\in T_{{\delta }_2}^n\left(Y\right|x^n)$ for ${\delta }_1≔\left|\mathcal{Y}\right|·\delta$ and ${\delta }_2≔\left(\right|\mathcal{Y}|+1)·\delta$.

Lemma 5.

If $y^n\in T_{\delta }^n\left(Y\right)$ and $(x^n,y^n)\notin T_{2\delta }^n(X,Y)$, then $x^n\notin T_{\delta }^n\left(X\right|y^n)$.

Lemma 6

([29], Lemma 1.2.12 and Remark). For arbitrarily fixed $\delta >0$ and every distribution P on $\mathcal{X}$ and stochastic matrix $W:\mathcal{X}\to \mathcal{Y}$

$$\begin{array}{cc}\hfill Pr\{X^n\in T_{\delta }^n\left(P\right)\}& \ge 1-2\left|\mathcal{X}\right|{\mathrm{e}}^{-2{\delta }^{2}n},\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill Pr\{Y^n\in T_{\delta }^n\left(W\right|x^n)|X^n=x^n\}& \ge 1-2\left|\mathcal{X}\right|·\left|\mathcal{Y}\right|{\mathrm{e}}^{-2{\delta }^{2}n}\hfill \\ \hfill & \mathrm{for}\mathrm{every}{x}^n\in {\mathcal{X}}^n.\hfill \end{array}$$

(36)

3.4. Proof of Converse Part

In this part, we shall prove ${\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$.

Let a tuple $(R,D,L,E)\in {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ be arbitrarily fixed. Then, there exists an $(n,2^{n(R+ϵ)},D+ϵ,L+ϵ,E+ϵ)$ code that satisfies (14)–(17). Let Q be a uniform random variable over $\{1,2,\dots ,n\}$ and let $p_i(x_{\mathcal{E},i},x_{{\mathcal{E}}^{\mathrm{c}},i},{\widehat{x}}_{\mathcal{R},i})$ be the conditional distribution given $Q=i$. Evaluating the inequalities for R, we obtain

$$\begin{array}{cc}\hfill R+ϵ& \stackrel{\left(\mathrm{a}\right)}{\ge }\frac{1}{n}\log M_n\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\ge }\frac{1}{n}H\left(J_n\right)\hfill \\ \hfill & \ge \frac{1}{n}I(J_n;X_{\mathcal{E}}^n)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{=}\frac{1}{n}\{H\left(X_{\mathcal{E}}^n\right)-H\left(X_{\mathcal{E}}^n\right|J_n,{\widehat{X}}_{\mathcal{R}}^n)\}\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{E},i}\right)-\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{E},i}\right|X_{\mathcal{E}}^{i-1},J_n,{\widehat{X}}_{\mathcal{R}}^n)\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\ge }\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{E},i}\right)-\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{E},i}\right|{\widehat{X}}_{\mathcal{R},i})\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{=}\sum _{i=1}^{n}Pr\{Q=i\}H\left(X_{\mathcal{E},i}\right|Q=i)\hfill \\ \hfill & -\sum _{i=1}^{n}Pr\{Q=i\}H\left(X_{\mathcal{E},i}\right|{\widehat{X}}_{\mathcal{R},i},Q=i)\hfill \\ \hfill & =H\left(X_{\mathcal{E},Q}\right|Q)-H\left(X_{\mathcal{E},Q}\right|{\widehat{X}}_{\mathcal{R},Q},Q)\hfill \\ \hfill & \stackrel{\left(\mathrm{g}\right)}{=}H\left(X_{\mathcal{E}}\right)-H\left(X_{\mathcal{E},Q}\right|{\widehat{X}}_{\mathcal{R},Q},Q)\hfill \\ \hfill & \stackrel{\left(\mathrm{h}\right)}{\ge }H\left(X_{\mathcal{E}}\right)-H\left(X_{\mathcal{E}}\right|{\widehat{X}}_{\mathcal{R}})\hfill \\ \hfill & =I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(37)

(a)

follows from (14),

(b)

follows because $H\left(J_n\right)\le \log |J_n|=\log M_n$,

(c)

is due to the fact that ${\widehat{X}}_{\mathcal{R}}^n=g\left(J_n\right)$,

(d)

follows because each $X_{\mathcal{K},i}$ is independent and ${\widehat{X}}_{\mathcal{R}}^n$ is a function of $J_n$,

(e)

follows because conditioning reduces entropy,

(f)

is due to the definition of Q,

(g)

follows because $X_{\mathcal{E}}\perp Q$, and

(h)

follows because conditioning reduces entropy, where $(X_{\mathcal{E}},{\widehat{X}}_{\mathcal{R}})\sim {\sum }_{i=1}^{n}Pr\{Q=i\}{p}_i(x_{\mathcal{E},i},{\widehat{x}}_{\mathcal{R},i})=p(x_{\mathcal{E}},{\widehat{x}}_{\mathcal{R}})$.

Similarly, evaluating D, L, and E, respectively, we obtain

$$\begin{array}{cc}\hfill D+ϵ& \stackrel{\left(\mathrm{i}\right)}{\ge }\mathbb{E}\left[\frac{1}{n}\sum _{i=1}^{n}d(X_{\mathcal{R},i},{\widehat{X}}_{\mathcal{R},i})\right]\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n\mathbb{E}\left[d(X_{\mathcal{R},i},{\widehat{X}}_{\mathcal{R},i})\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{j}\right)}{=}{\mathbb{E}}_Q\left[\mathbb{E}\left[d(X_{\mathcal{R},i},{\widehat{X}}_{\mathcal{R},i})\right|Q]\right]\hfill \\ \hfill & \stackrel{\left(\mathrm{k}\right)}{=}\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill L+ϵ& \stackrel{\left(\mathrm{l}\right)}{\ge }\frac{1}{n}I(X_{\mathcal{H}}^n;J_n)\hfill \\ \hfill & =\frac{1}{n}H\left(X_{\mathcal{H}}^n\right)-\frac{1}{n}H\left(X_{\mathcal{H}}^n\right|J_n)\hfill \\ \hfill & \stackrel{\left(\mathrm{m}\right)}{=}H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{H},i}\right|X_{\mathcal{H}}^{i-1},J_n)\hfill \\ \hfill & \stackrel{\left(\mathrm{n}\right)}{=}H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{H},i}\right|X_{\mathcal{H}}^{i-1},J_n,{\widehat{X}}_{\mathcal{R},i})\hfill \\ \hfill & \stackrel{\left(\mathrm{o}\right)}{\ge }H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{i=1}^{n}H\left(X_{\mathcal{H},i}\right|{\widehat{X}}_{\mathcal{R},i})\hfill \\ \hfill & \stackrel{\left(\mathrm{p}\right)}{=}H\left(X_{\mathcal{H}}\right)-\sum _{i=1}^{n}Pr\{Q=i\}H\left(X_{\mathcal{H},i}\right|{\widehat{X}}_{\mathcal{R},i},Q=i)\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-H\left(X_{\mathcal{H},Q}\right|{\widehat{X}}_{\mathcal{R},Q},Q)\hfill \\ \hfill & \stackrel{\left(\mathrm{q}\right)}{\ge }H\left(X_{\mathcal{H}}\right)-H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})\hfill \\ \hfill & =I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill E+ϵ& \ge \frac{1}{n}I(X_{\mathcal{H}}^n;X_{\mathcal{E}}^n)\hfill \\ \hfill & \stackrel{\left(\mathrm{r}\right)}{=}\frac{1}{n}\sum _{i=1}^{n}I(X_{\mathcal{H},i};X_{\mathcal{E}}^n|X_{\mathcal{H}}^{i-1})\hfill \\ \hfill & \stackrel{\left(\mathrm{s}\right)}{=}\frac{1}{n}\sum _{i=1}^{n}I(X_{\mathcal{H},i};X_{\mathcal{E},i})\hfill \\ \hfill & \stackrel{\left(\mathrm{t}\right)}{=}I(X_{\mathcal{H}};X_{\mathcal{E}}),\hfill \end{array}$$

(40)

where

(i)

is due to (15),

(j)

is derived from the definition of Q,

(k)

follows because $(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\sim {\sum }_{i=1}^{n}Pr\{Q=i\}{p}_i(x_{\mathcal{R},i},{\widehat{x}}_{\mathcal{R},i})=p(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})$,

(l)

is due to (16),

(m)

follows because $i.i.d.P_{X_{\mathcal{K}}^n}$,

(n)

follows because ${\widehat{X}}_{\mathcal{R}}^n=g\left(J_n\right)$,

(o)

follows from the fact that conditioning reduces entropy,

(p)

is derived from the definition of Q, and

(q)

follows because conditioning reduces entropy, where $(X_{\mathcal{H}},{\widehat{X}}_{\mathcal{R}})\sim {\sum }_{i=1}^{n}Pr\{Q=i\}{p}_i(x_{\mathcal{H},i},{\widehat{x}}_{\mathcal{R},i})=p(x_{\mathcal{H}},{\widehat{x}}_{\mathcal{R}})$,

(r)

is due to chain rule for mutual information,

(s), (t)

follow because $i.i.d.P_{X_{\mathcal{K}}^n}$.

It is readily shown that the Markov chain $X_{{\mathcal{E}}^{\mathrm{c}}}$–$X_{\mathcal{E}}$–${\widehat{X}}_{\mathcal{R}}$ holds (cf. Appendix A). We complete the proof of the converse part.

3.5. Proof of Direct Part

In this part, we provide a sketch of the proof of ${\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$.

Under an arbitrarily fixed distribution $P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$, any tuple $(R,D,L,E)\in {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ is chosen such that

$$\begin{array}{cc}\hfill R& >I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill E& >I(X_{\mathcal{H}};X_{\mathcal{E}}).\hfill \end{array}$$

(44)

From (42) and (43), we can choose a sufficiently small $ϵ>0$ such that

$$\begin{array}{cc}\hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]+ϵ,\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+ϵ.\hfill \end{array}$$

(46)

In addition, with this $ϵ$, some constant $0<\tau <\frac{1}{2}$ is fixed such that

$$\begin{array}{cc}\hfill & \tau (\log |{\mathcal{X}}_{\mathcal{H}}|+5)+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }<ϵ.\hfill \end{array}$$

(47)

We can also choose positive numbers $\delta (≔\delta (n\left)\right)$ such that

$$\begin{array}{cc}\hfill & (\delta \left(n\right)+{\delta }_1\left(n\right))|{\mathcal{X}}_{\mathcal{R}}|·|{\widehat{\mathcal{X}}}_{\mathcal{R}}|D_{max}+\tau <ϵ,\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & 2{\delta }^2\left(n\right)\le R-I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}})-\frac{1}{n}-\tau ,\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill & \delta \left(n\right)\to 0,\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill & \sqrt{n}·\delta \left(n\right)\to \infty \hfill \end{array}$$

(51)

as $n\to \infty$, where ${\delta }_1≔\left(\right|{\mathcal{X}}_{\mathcal{E}}|-|{\mathcal{X}}_{\mathcal{R}}\left|\right)·\delta$ and $D_{max}≔\underset{a\in {\mathcal{X}}_{\mathcal{R}},b\in {\widehat{\mathcal{X}}}_{\mathcal{R}}}{max}d(a,b)$. Let $\delta \left(n\right)=\frac{c}{\sqrt{n}}\log n$ where c is a constant, and obviously (50) and (51) are satisfied.

Generation of codebook: Randomly generate ${\widehat{x}}_{\mathcal{R}}^n\left(j\right)$ from the strongly typical sequences $T_{\delta }^n\left({\widehat{X}}_{\mathcal{R}}\right)$ for $j=1,2,\dots ,M_n≔2^{nR}$. Reveal the codebook $\mathcal{C}=\{{\widehat{x}}_{\mathcal{R}}^n\left(1\right),\dots ,{\widehat{x}}_{\mathcal{R}}^n\left(M_n\right)\}$ to the encoder and decoder.

Encoding: If a sequence $x_{\mathcal{E}}^n\in {\mathcal{X}}_{\mathcal{E}}^n$ satisfies $x_{\mathcal{K}}^n=(x_{\mathcal{E}}^n,x_{{\mathcal{E}}^{\mathrm{c}}}^n)$ with some $x_{{\mathcal{E}}^{\mathrm{c}}}^n\in {\mathcal{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n$, we write $x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n$. Given $x_{\mathcal{K}}^n$, the encoder finds j such that $x_{\mathcal{E}}^n\in T_{\delta }^n\left(X_{\mathcal{E}}\right|{\widehat{x}}_{\mathcal{R}}\left(j\right))$ and sets $f_n\left(x_{\mathcal{E}}^n\right)=j$ where $T_{\delta }^n\left(X_{\mathcal{E}}\right|{\widehat{x}}_{\mathcal{R}}\left(j\right))$ is the conditional strongly typical sequences. If there exist multiple such j, $f_n\left(x_{\mathcal{E}}^n\right)$ is set as the minimum one. If there are no such j, then $f_n\left(x_{\mathcal{E}}^n\right)=M_n$.

Decoding: When j is observed, the decoder sets the reproduced sequence as ${\widehat{X}}_{\mathcal{R}}^n={\widehat{x}}_{\mathcal{R}}^n\left(j\right)$.

Evaluation: We define $\mathcal{A}\left(j\right)$, $\mathcal{B}\left(j\right)$, and $\tilde{\mathcal{A}}\left(j\right)$ as

$$\begin{array}{cc}\hfill \mathcal{A}\left(j\right)≔& \{x_{\mathcal{E}}^n:f_n\left(x_{\mathcal{E}}^n\right)=j\},\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill \mathcal{B}\left(j\right)≔& \{x_{\mathcal{K}}^n:x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n,f_n\left(x_{\mathcal{E}}^n\right)=j\},\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill \tilde{\mathcal{A}}\left(j\right)≔& \left\{\begin{array}{c}\{x_{\mathcal{K}}^n:x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n,f_n\left(x_{\mathcal{E}}^n\right)=j,x_{\mathcal{K}}^n\in T_{2\delta }^n\left(X_{\mathcal{K}}\right|{\widehat{x}}_{\mathcal{R}}^n\left(j\right))\}\hfill \\ (j=1,2,\dots ,M_n-1)\hfill \\ \{x_{\mathcal{K}}^n:x_{\mathcal{K}}^n\in {\mathcal{X}}_{\mathcal{K}}^n\setminus \bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\}(j=M_n).\hfill \end{array}\right.\hfill \end{array}$$

(54)

It is easily verified that $\mathcal{A}\left(j\right)$ for $j=1,2,\dots ,M_n$ (also, $\mathcal{B}\left(j\right)$ and $\tilde{\mathcal{A}}\left(j\right)$) is disjoint. From the definitions of $J_n$, $\mathcal{A}\left(j\right)$, and $\mathcal{B}\left(j\right)$,

$$\begin{array}{cc}\hfill Pr\{J_n=j\}=Pr\{X_{\mathcal{E}}^n\in \mathcal{A}\left(j\right)\}=Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}& \hfill \\ \hfill \mathrm{for}j=1,2,\dots ,M_n.& \hfill \end{array}$$

(55)

For sufficiently large n, we can prove (cf. Appendix B)

$$\begin{array}{cc}\hfill |Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}-Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}|\le 2|{\mathcal{X}}_{\mathcal{K}}|·|{\widehat{\mathcal{X}}}_{\mathcal{R}}|{\mathrm{e}}^{-2{\delta }^{2}n}& \hfill \\ \hfill \mathrm{for}j=1,2,\dots ,M_n-1.& \hfill \end{array}$$

(56)

For sufficiently large n, we can show that there exists a code $(f_n,g_n)$ such that (cf. Appendix C)

$$\begin{array}{cc}\hfill & r_n\le R,\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill & u_n\le \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]+(\delta +{\delta }_1)|{\mathcal{X}}_{\mathcal{R}}|·|{\widehat{\mathcal{X}}}_{\mathcal{R}}|D_{max}+\tau ,\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill & e_n\le I(X_{\mathcal{H}};X_{\mathcal{E}}),\hfill \end{array}$$

(59)

$$\begin{array}{cc}\hfill & Pr\left\{X_{\mathcal{E}}^n\notin \bigcup _{j=1}^{M_n-1}\mathcal{A}\left(j\right)\right\}\le \left(2\right|{\mathcal{X}}_{\mathcal{E}}|+1){\mathrm{e}}^{-2{\delta }^{2}n},\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill & Pr\left\{X_{\mathcal{K}}^n\notin \bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\right\}\le \tau ,\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill & |\tilde{\mathcal{A}}\left(j\right)|\ge 2^{n\{H\left(X_{\mathcal{K}}\right|{\widehat{X}}_{\mathcal{R}})-\tau \}}.\hfill \end{array}$$

(62)

For this code $(f_n,g_n)$, we evaluate the privacy leakage against the decoder as

$$\begin{array}{cc}\hfill l_n& ≔\frac{1}{n}I(X_{\mathcal{H}}^n;J_n)\hfill \\ \hfill & =\frac{1}{n}H\left(X_{\mathcal{H}}^n\right)-\frac{1}{n}H\left(X_{\mathcal{H}}^n\right|J_n)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\le }H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \\ \hfill & +4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill & \stackrel{\left(\mathrm{c}\right)}{\le }H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \\ \hfill & +4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}[-\sum _{\begin{array}{c}{x}_{\mathcal{H}}^n\end{array}}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}·\hfill \\ \hfill & \log Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}]·Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau },\hfill \end{array}$$

(64)

where

(a)

follows because of $i.i.d.P_{X_{\mathcal{K}}^n}$,

(b)

is due to the inequality proved in Appendix D,

(c)

follows by removing the term for $j=M_n$.

Here, for any $x_{\mathcal{H}}^n$ satisfying $x_{\mathcal{K}}^n=(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)$ with some $x_{\mathcal{R}}^n$, we can show that

$$\begin{array}{cc}\hfill & Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \\ \hfill & =\frac{Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}}\hfill \\ \hfill & =\frac{{\sum }_{x_{\mathcal{R}}^n:(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n,X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\frac{{\sum }_{x_{\mathcal{R}}^n:(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\le }\frac{{\sum }_{x_{\mathcal{R}}^n\in T_{{\delta }_3}^n\left(X_{\mathcal{R}}\right|x_{\mathcal{H}}^n,{\widehat{x}}_{\mathcal{R}}^n\left(j\right))}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill & \stackrel{\left(\mathrm{f}\right)}{\le }\frac{2^{n\{H\left(X_{\mathcal{R}}\right|X_{\mathcal{H}},{\widehat{X}}_{\mathcal{R}})+\tau \}}·2^{-n\{H\left(X_{\mathcal{R}}\right|X_{\mathcal{H}})-\tau \}}}{2^{n\{H\left(X_{\mathcal{K}}\right|{\widehat{X}}_{\mathcal{R}})-\tau \}}·2^{-n\{H\left(X_{\mathcal{K}}\right)+\tau \}}}·2^{-n\{H\left(X_{\mathcal{H}}\right)-\tau \}}\hfill \\ \hfill & =2^{-n\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}},\hfill \end{array}$$

(66)

where

(d)

follows from the fact that

$$\begin{array}{c}\hfill Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n,X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}=Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\},\end{array}$$

(e)

is due to the inequality proved in Appendix E, and

(f)

follows because of the number of strongly typical sequences.

Therefore, from Equations (61), (64) and (66) we can obtain

$$\begin{array}{cc}\hfill l_n& \le H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}[n\sum _{\begin{array}{c}{x}_{\mathcal{H}}^n\end{array}}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}·\hfill \\ \hfill & \{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}]·Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-Pr\left\{X_{\mathcal{K}}^n\in \left(\bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\right)\right\}·\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & \le H\left(X_{\mathcal{H}}\right)-(1-\tau )\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & \le I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+\tau \left(\log \right|{\mathcal{X}}_{\mathcal{H}}|+5)+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }.\hfill \end{array}$$

(67)

Since constants $ϵ$, $\tau$, and $\delta$ are fixed to satisfy (45)–(48), from (44), (57)–(59) and (67), we obtain

$$\begin{array}{cc}\hfill r_n& \le R,\hfill \end{array}$$

(68)

$$\begin{array}{cc}\hfill u_n& \le \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]+ϵ<D,\hfill \end{array}$$

(69)

$$\begin{array}{cc}\hfill l_n& <I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+ϵ<L,\hfill \end{array}$$

(70)

$$\begin{array}{cc}\hfill e_n& \le I(X_{\mathcal{H}};X_{\mathcal{E}})<E.\hfill \end{array}$$

(71)

Therefore, for the fixed distribution $P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$ any tuple

$$\begin{array}{cc}\hfill (R,D,L,E)\in \left\{\right(R,D,L,E):R& >I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill E& >I(X_{\mathcal{H}};X_{\mathcal{E}})\}≕{\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\hfill \end{array}$$

(72)

is achievable. Consequently, ${\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$. Taking the closure for the left-hand side (l.h.s.), we obtain $Cl\left({\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ because ${\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ is a closed set. We conclude that ${\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)={\bigcup }_{p}Cl\left({\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ because the distribution $P_{X_{\mathcal{K}}}=P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$ is fixed arbitrarily. We complete the proof of the direct part.

4. First-Order Rate Analysis with Excess-Distortion Probability

4.1. Performance Measures

Hereafter, let the pair of the encoder and decoder $(f_n,g_n)$ be fixed.

For a given $M_n$, the coding rate is defined as

$$\begin{array}{c}\hfill r_n≔\frac{1}{n}\log M_n.\end{array}$$

(73)

Let $d:{\mathcal{X}}_{\mathcal{R}}\times {\widehat{\mathcal{X}}}_{\mathcal{R}}\to [0,\infty )$ be a distortion function between $x_{\mathcal{R}}\in {\mathcal{X}}_{\mathcal{R}}$ and ${\widehat{x}}_{\mathcal{R}}\in {\widehat{\mathcal{X}}}_{\mathcal{R}}$. The distortion between sequences $x_{\mathcal{R}}^n\in {\mathcal{X}}_{\mathcal{R}}^n$ and ${\widehat{x}}_{\mathcal{R}}^n\in {\widehat{\mathcal{X}}}_{\mathcal{R}}^n$ is defined as

$$\begin{array}{cc}\hfill d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)≔& \sum _{i=1}^{n}d(x_{\mathcal{R},i},{\widehat{x}}_{\mathcal{R},i}).\hfill \end{array}$$

(74)

Then, the measure of utility is defined as

$$\begin{array}{cc}\hfill u_n≔& Pr\left\{\frac{1}{n}d(X_{\mathcal{R}}^n,{\widehat{X}}_{\mathcal{R}}^n)>D\right\}.\hfill \end{array}$$

(75)

This measurement is called excess-distortion probability for $D\ge 0$.

In this system, the privacy of the hidden source sequence $X_{\mathcal{H}}^n$ should be protected when the codeword $J_n$ is observed by decoder $g_n$. The measure of privacy for the decoder is defined as

$$\begin{array}{c}\hfill l_n≔\frac{1}{n}I(X_{\mathcal{H}}^n;J_n),\end{array}$$

(76)

where $I(X_{\mathcal{H}}^n;J_n)$ is the mutual information between $X_{\mathcal{H}}^n$ and $J_n$.

The privacy of the hidden source sequence $X_{\mathcal{H}}^n$ should be protected when the encoded information $X_{\mathcal{E}}$ is observed by encoder $f_n$. The measurement of privacy for the encoder is defined as

$$\begin{array}{c}\hfill e_n≔\frac{1}{n}I(X_{\mathcal{H}}^n;X_{\mathcal{E}}^n),\end{array}$$

(77)

where $I(X_{\mathcal{H}}^n;X_{\mathcal{E}}^n)$ is the mutual information between $X_{\mathcal{H}}^n$ and $X_{\mathcal{E}}^n$.

4.2. Achievable Region and Theorem

We define the achievable region for the first-order rate analysis with the excess-distortion probability and state the obtained results.

Definition 10.

A tuple $(R,D,L,E)$ is said to be $\mathit{ϵ}$-achievable (with respect to the excess-distortion probability) if, for any given $ϵ>0$, there exists a sequence of codes $(f_n,g_n)$ satisfying

$$\begin{array}{cc}\hfill r_n& \le R+ϵ,\hfill \end{array}$$

(78)

$$\begin{array}{cc}\hfill u_n& \le ϵ,\hfill \end{array}$$

(79)

$$\begin{array}{cc}\hfill l_n& \le L+ϵ,\hfill \end{array}$$

(80)

$$\begin{array}{cc}\hfill e_n& \le E+ϵ\hfill \end{array}$$

(81)

for all sufficiently large n.

The technical meanings of each constraint in Definition 10 can be interpreted as follows: Equation (78) evaluates how much the source sequence is compressed, so this rate should be decreased. Equation (79) is the constraint corresponding to the excess-distortion probability being less than $ϵ$, so this condition should also be decreased. Equation (80) constrains the amount of leaked private information to the decoder. Since private information should be kept secret for the receiver, this quantity should be decreased as well. Equation (81) constrains the amount of leaked private information to the encoder. For the same reason as (80), this quantity should also be decreased.

Definition 11.

The closure of the set of ϵ-achievable tuples $(R,D,L,E)$ is referred to as the $\mathit{ϵ}$-achievable region and is denoted by ${\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$ and define

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{0<ϵ<1}{\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}}).\end{array}$$

(82)

We establish the following theorem. For the proof of this theorem, please refer to Section 4.3 and Section 4.4.

Theorem 2.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the achievable region of the coding system is given by

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(83)

Remark 4.

From Theorems 1 and 2, we find that the achievable region in which utility is measured by the expected distortion is equal to the one in which utility is measured by the excess-distortion probability.

Because in Section 6 we discuss the achievable region among coding rate, utility, and privacy, a characterization of the achievable region is derived by projecting the characterization in Theorem 2 onto the $R\text{-}D\text{-}L$ hyperplane.

Definition 12.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{RDL}\left(ϵ\right|P_{X_{\mathcal{K}}})≔\{(R,D,L):(R,D,L,E)\in {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})\}\end{array}$$

(84)

and

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{0<ϵ<1}{\mathcal{L}}_{\mathcal{E}}^{RDL}\left(ϵ\right|P_{X_{\mathcal{K}}}).\end{array}$$

(85)

Definition 13.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)=\{(R,D,L):& R\ge I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill & D\ge \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill & L\ge I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})\hfill \\ \hfill & \mathrm{for}\mathrm{some}{P}_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}\}.\hfill \end{array}$$

(86)

Corollary 2.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the region ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ is given by

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(87)

Examples of numerical calculation of this result are shown in Section 6.1.

Since we focus on the achievable region between utility and privacy in the next section, a characterization of the achievable region is derived by further projecting the result of Theorem 2 onto the $D\text{-}L$ plane.

Definition 14.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{DL}\left(ϵ\right|P_{X_{\mathcal{K}}})≔\{(D,L):(R,D,L,E)\in {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})\}\end{array}$$

(88)

and

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{0<ϵ<1}{\mathcal{L}}_{\mathcal{E}}^{DL}\left(ϵ\right|P_{X_{\mathcal{K}}}).\end{array}$$

(89)

Definition 15.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, we define

$$\begin{array}{cc}\hfill {\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)=\{(D,L):& D\ge \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill & L\ge I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})\hfill \\ \hfill & \mathrm{for}\mathrm{some}{P}_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}\}.\hfill \end{array}$$

(90)

Corollary 3.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the region ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$ is given by

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(91)

4.3. Proof of Converse Part

From Section 3.4 (proof of the converse part), we have

$$\begin{array}{c}\hfill {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(92)

Let a tuple $(R,D,L,E)\in {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ be arbitrarily fixed and $ϵ>0$ and ${ϵ}^{\prime }>0$ be given. From the argument of the method of types, the sequences $x_{\mathcal{R}}^n$ are divided into two categories: distortion-typical or non-distortion-typical with some ${\widehat{x}}_{\mathcal{R}}^n$. The sequences of the former categories satisfy $\frac{1}{n}d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)<D+ϵ$ and the sequences of the latter one satisfy $\frac{1}{n}d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)<d_{max}$ where $d_{max}≔{\displaystyle \underset{x_{\mathcal{R}}\in {\mathcal{X}}_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}}\in {\widehat{\mathcal{X}}}_{\mathcal{R}}}{max}}d(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})$. Then, the expected distortion is bounded from above as

$$\begin{array}{cc}\hfill \mathbb{E}\left[\frac{1}{n}d(X_{\mathcal{R}}^n,{\widehat{X}}_{\mathcal{R}}^n)\right]& \le D+ϵ+Pr\left\{\frac{1}{n}d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)>D+ϵ\right\}·d_{max}\hfill \\ \hfill & \le D+ϵ+Pr\left\{\frac{1}{n}d(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)>D\right\}·d_{max}\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{\le }D+ϵ+{ϵ}^{\prime }{d}_{max},\hfill \end{array}$$

(93)

where (a) follows from (79) of $ϵ$-achievable in which utility is measured by the excess-distortion probability. Since $ϵ+{ϵ}^{\prime }{d}_{max}$ can be arbitrarily small with proper choices of $ϵ$ and ${ϵ}^{\prime }$, (15) can be derived. This means

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(94)

From both inclusion relations,

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\end{array}$$

(95)

is evidently satisfied.

4.4. Proof of the Direct Part

In this part, we provide a sketch of the proof of ${\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$.

Under an arbitrarily fixed distribution $P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$, any tuple $(R,D,L,E)\in {\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ is chosen such that

$$\begin{array}{cc}\hfill R& >I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(96)

$$\begin{array}{cc}\hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \end{array}$$

(97)

$$\begin{array}{cc}\hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \end{array}$$

(98)

$$\begin{array}{cc}\hfill E& >I(X_{\mathcal{H}};X_{\mathcal{E}}).\hfill \end{array}$$

(99)

From (97) and (98) , we can choose a sufficiently small $ϵ>0$ such that

$$\begin{array}{cc}\hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]+ϵ,\hfill \end{array}$$

(100)

$$\begin{array}{cc}\hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+ϵ.\hfill \end{array}$$

(101)

In addition, with this $ϵ$, some constant $0<\tau <\frac{1}{2}$ is fixed such that

$$\begin{array}{cc}\hfill & \tau (\log |{\mathcal{X}}_{\mathcal{H}}|+5)+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }<ϵ.\hfill \end{array}$$

(102)

We can also choose positive numbers $\delta (≔\delta (n\left)\right)$ such that

$$\begin{array}{cc}\hfill & 2{\delta }^2\left(n\right)\le R-I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}})-\frac{1}{n}-\tau ,\hfill \end{array}$$

(103)

$$\begin{array}{cc}\hfill & \delta \left(n\right)\to 0,\hfill \end{array}$$

(104)

$$\begin{array}{cc}\hfill & \sqrt{n}·\delta \left(n\right)\to \infty \hfill \end{array}$$

(105)

as $n\to \infty$. Let $\delta \left(n\right)=\frac{c}{\sqrt{n}}\log n$ where c is a constant, and obviously (104) and (105) are satisfied.

Generation of codebook: Randomly generate ${\widehat{x}}_{\mathcal{R}}^n\left(j\right)$ from the strongly typical sequences $T_{\delta }^n\left({\widehat{X}}_{\mathcal{R}}\right)$ for $j=1,2,\dots ,M_n≔2^{nR}$. Reveal the codebook $\mathcal{C}=\{{\widehat{x}}_{\mathcal{R}}^n\left(1\right),\dots ,{\widehat{x}}_{\mathcal{R}}^n\left(M_n\right)\}$ to the encoder and decoder.

Encoding: If a sequence $x_{\mathcal{E}}^n\in {\mathcal{X}}_{\mathcal{E}}^n$ satisfies $x_{\mathcal{K}}^n=(x_{\mathcal{E}}^n,x_{{\mathcal{E}}^{\mathrm{c}}}^n)$ with some $x_{{\mathcal{E}}^{\mathrm{c}}}^n\in {\mathcal{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n$, we write $x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n$. Given $x_{\mathcal{K}}^n$, the encoder finds j such that $x_{\mathcal{E}}^n\in T_{\delta }^n\left(X_{\mathcal{E}}\right|{\widehat{x}}_{\mathcal{R}}^n\left(j\right))$ and sets $f_n\left(x_{\mathcal{E}}^n\right)=j$ where $T_{\delta }^n\left(X_{\mathcal{E}}\right|{\widehat{x}}_{\mathcal{R}}^n\left(j\right))$ is the conditional strongly typical sequences. If there exist multiple such j, $f_n\left(x_{\mathcal{E}}^n\right)$ is set as the minimum one. If there are no such j, then $f_n\left(x_{\mathcal{E}}^n\right)=M_n$.

Decoding: When j is observed, the decoder sets the reproduced sequence as ${\widehat{X}}_{\mathcal{R}}^n={\widehat{x}}_{\mathcal{R}}^n\left(j\right)$.

Evaluation: We define $\mathcal{A}\left(j\right)$, $\mathcal{B}\left(j\right)$, and $\tilde{\mathcal{A}}\left(j\right)$ as

$$\begin{array}{cc}\hfill \mathcal{A}\left(j\right)≔& \{x_{\mathcal{E}}^n:f_n\left(x_{\mathcal{E}}^n\right)=j\},\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill \mathcal{B}\left(j\right)≔& \{x_{\mathcal{K}}^n:x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n,f_n\left(x_{\mathcal{E}}^n\right)=j\},\hfill \end{array}$$

(107)

$$\begin{array}{cc}\hfill \tilde{\mathcal{A}}\left(j\right)≔& \left\{\begin{array}{c}\{x_{\mathcal{K}}^n:x_{\mathcal{E}}^n\prec x_{\mathcal{K}}^n,f_n\left(x_{\mathcal{E}}^n\right)=j,x_{\mathcal{K}}^n\in T_{2\delta }^n\left(X_{\mathcal{K}}\right|{\widehat{x}}_{\mathcal{R}}^n\left(j\right))\}\hfill \\ (j=1,2,\dots ,M_n-1)\hfill \\ \{x_{\mathcal{K}}^n:x_{\mathcal{K}}^n\in {\mathcal{X}}_{\mathcal{K}}^n\setminus \bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\}(j=M_n).\hfill \end{array}\right.\hfill \end{array}$$

(108)

It is easily verified that $\mathcal{A}\left(j\right)$ for $j=1,2,\dots ,M_n$ (and also $\mathcal{B}\left(j\right)$ and $\tilde{\mathcal{A}}\left(j\right)$) is disjoint. From the definitions of $J_n$, $\mathcal{A}\left(j\right)$, and $\mathcal{B}\left(j\right)$,

$$\begin{array}{cc}\hfill Pr\{J_n=j\}=Pr\{X_{\mathcal{E}}^n\in \mathcal{A}\left(j\right)\}=Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}& \hfill \\ \hfill \mathrm{for}j=1,2,\dots ,M_n.& \hfill \end{array}$$

(109)

For sufficiently large n, we can prove (cf. Appendix B)

$$\begin{array}{cc}\hfill |Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}-Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}|\le 2|{\mathcal{X}}_{\mathcal{K}}|·|{\widehat{\mathcal{X}}}_{\mathcal{R}}|{\mathrm{e}}^{-2{\delta }^{2}n}& \hfill \\ \hfill \mathrm{for}j=1,2,\dots ,M_n-1.& \hfill \end{array}$$

(110)

For sufficiently large n, we can show that there exists a code $(f_n,g_n)$ such that (cf. Appendix F)

$$\begin{array}{cc}\hfill & r_n\le R,\hfill \end{array}$$

(111)

$$\begin{array}{cc}\hfill & Pr\left\{X_{\mathcal{E}}^n\notin \bigcup _{j=1}^{M_n-1}\mathcal{A}\left(j\right)\right\}\le \left(2\right|{\mathcal{X}}_{\mathcal{E}}|+1){\mathrm{e}}^{-2{\delta }^{2}n},\hfill \end{array}$$

(112)

$$\begin{array}{cc}\hfill & u_n\le \left(2\right|{\mathcal{X}}_{\mathcal{E}}|+1){\mathrm{e}}^{-2{\delta }^{2}n},\hfill \end{array}$$

(113)

$$\begin{array}{cc}\hfill & e_n\le I(X_{\mathcal{H}};X_{\mathcal{E}}),\hfill \end{array}$$

(114)

$$\begin{array}{cc}\hfill & Pr\left\{X_{\mathcal{K}}^n\notin \bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\right\}\le \tau ,\hfill \end{array}$$

(115)

$$\begin{array}{cc}\hfill & |\tilde{\mathcal{A}}\left(j\right)|\ge 2^{n\{H\left(X_{\mathcal{K}}\right|{\widehat{X}}_{\mathcal{R}})-\tau \}}.\hfill \end{array}$$

(116)

For this code $(f_n,g_n)$, we evaluate the privacy leakage against the decoder as

$$\begin{array}{cc}\hfill l_n& ≔\frac{1}{n}I(X_{\mathcal{H}}^n;J_n)\hfill \\ \hfill & =\frac{1}{n}H\left(X_{\mathcal{H}}^n\right)-\frac{1}{n}H\left(X_{\mathcal{H}}^n\right|J_n)\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}H\left(X_{\mathcal{H}}\right)-\frac{1}{n}H\left(X_{\mathcal{H}}^n\right|J_n)\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \mathcal{B}\left(j\right)\}\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\le }H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \end{array}$$

(117)

$$\begin{array}{cc}\hfill & +4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{\le }H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}H\left(X_{\mathcal{H}}^n\right|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right))Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \\ \hfill & +4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}[-\sum _{\begin{array}{c}{x}_{\mathcal{H}}^n\end{array}}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}·\log Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}]·\hfill \end{array}$$

(118)

$$\begin{array}{cc}\hfill & Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau },\hfill \end{array}$$

(119)

where

(a)

follows because of $i.i.d.P_{X_{\mathcal{K}}^n}$,

(b)

is due to the inequality proved in Appendix D, and

(c)

follows by removing the term for $j=M_n$.

Here, for any $x_{\mathcal{H}}^n$ satisfying $x_{\mathcal{K}}^n=(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)$ with some $x_{\mathcal{R}}^n$, we can show that

$$\begin{array}{cc}\hfill & Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}\hfill \\ \hfill & =\frac{Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}}\hfill \\ \hfill & =\frac{{\sum }_{x_{\mathcal{R}}^n:(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n,X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{=}\frac{{\sum }_{x_{\mathcal{R}}^n:(x_{\mathcal{R}}^n,x_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{\le }\frac{{\sum }_{x_{\mathcal{R}}^n\in T_{{\delta }_3}^n\left(X_{\mathcal{R}}\right|x_{\mathcal{H}}^n,{\widehat{x}}_{\mathcal{R}}^n\left(j\right))}Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}}{{\sum }_{({\tilde{x}}_{\mathcal{R}}^n,{\tilde{x}}_{\mathcal{H}}^n)\in \tilde{\mathcal{A}}\left(j\right)}Pr\{X_{\mathcal{R}}^n={\tilde{x}}_{\mathcal{R}}^n,X_{\mathcal{H}}^n={\tilde{x}}_{\mathcal{H}}^n\}}·Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}\hfill \end{array}$$

(120)

$$\begin{array}{cc}\hfill & \stackrel{\left(\mathrm{f}\right)}{\le }\frac{2^{n\{H\left(X_{\mathcal{R}}\right|X_{\mathcal{H}},{\widehat{X}}_{\mathcal{R}})+\tau \}}·2^{-n\{H\left(X_{\mathcal{R}}\right|X_{\mathcal{H}})-\tau \}}}{2^{n\{H\left(X_{\mathcal{K}}\right|{\widehat{X}}_{\mathcal{R}})-\tau \}}·2^{-n\{H\left(X_{\mathcal{K}}\right)+\tau \}}}·2^{-n\{H\left(X_{\mathcal{H}}\right)-\tau \}}\hfill \\ \hfill & =2^{-n\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}},\hfill \end{array}$$

(121)

where

(d)

follows from the fact that

$$\begin{array}{c}\hfill Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n,X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\}=Pr\{X_{\mathcal{R}}^n=x_{\mathcal{R}}^n|X_{\mathcal{H}}^n=x_{\mathcal{H}}^n\},\end{array}$$

(e)

is due to the inequality proved in Appendix E, and

(f)

follows because of the number of strongly typical sequences.

Therefore, from Equations (115), (119), and (121), we can obtain

$$\begin{array}{cc}\hfill l_n& \le H\left(X_{\mathcal{H}}\right)-\frac{1}{n}\sum _{j=1}^{M_n-1}[n\sum _{\begin{array}{c}{x}_{\mathcal{H}}^n\end{array}}Pr\{X_{\mathcal{H}}^n=x_{\mathcal{H}}^n|X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}·\hfill \\ \hfill & \{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}]·Pr\{X_{\mathcal{K}}^n\in \tilde{\mathcal{A}}\left(j\right)\}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & =H\left(X_{\mathcal{H}}\right)-Pr\left\{X_{\mathcal{K}}^n\in \left(\bigcup _{j=1}^{M_n-1}\tilde{\mathcal{A}}\left(j\right)\right)\right\}·\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & \le H\left(X_{\mathcal{H}}\right)-(1-\tau )\{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})-5\tau \}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }\hfill \\ \hfill & \le I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+\tau \{H\left(X_{\mathcal{H}}\right|{\widehat{X}}_{\mathcal{R}})+5\}+4\tau \log \frac{|{\mathcal{X}}_{\mathcal{H}}|·2^R}{2\tau }.\hfill \end{array}$$

(122)

Since constants $ϵ$, $\tau$, and $\delta$ are fixed to satisfy (100)–(102), from (111), (113), and (122), we obtain

$$\begin{array}{cc}\hfill & r_n\le R,\hfill \end{array}$$

(123)

$$\begin{array}{cc}\hfill & u_n\le ϵ,\hfill \end{array}$$

(124)

$$\begin{array}{cc}\hfill & l_n<I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+ϵ<L,\hfill \end{array}$$

(125)

$$\begin{array}{cc}\hfill & e_n\le I(X_{\mathcal{H}};X_{\mathcal{E}})<E.\hfill \end{array}$$

(126)

Therefore, for the fixed distribution $P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$, any tuple

$$\begin{array}{cc}\hfill (R,D,L,E)\in \left\{\right(R,D,L,E):R& >I(X_{\mathcal{E}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill D& >\mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right],\hfill \\ \hfill L& >I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}}),\hfill \\ \hfill E& >I(X_{\mathcal{H}};X_{\mathcal{E}})\}≕{\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\hfill \end{array}$$

(127)

is achievable. Consequently, ${\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\subseteq {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$. Taking the closure for the l.h.s., we obtain $Cl\left({\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\right)\subseteq {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$ because ${\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$ is a closed set. We conclude that ${\mathcal{S}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)={\bigcup }_{p}Cl\left({\mathcal{S}}_{\mathcal{E}}^{*}\left(P_{X_{\mathcal{K}}}\right)\right)\subseteq {\mathcal{L}}_{\mathcal{E}}\left(ϵ\right|P_{X_{\mathcal{K}}})$ because the distribution $P_{X_{\mathcal{K}},{\widehat{X}}_{\mathcal{R}}}=P_{X_{\mathcal{E}},X_{{\mathcal{E}}^{\mathrm{c}}}}·P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$ is fixed arbitrarily. We complete the proof of the direct part.

5. Strong Converse Theorem for Utility–Privacy Trade-Offs

5.1. Another Expression of the Achievable Region

In Section 5.1, we clarify that the achievable region ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$ defined in (89) coincides with the region expressed with a tangent plane.

Definition 16.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the region ${\mathcal{T}}_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)$ is defined as

$$\begin{array}{cc}\hfill & T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)≔min\{I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+\mu \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]\mathrm{for}\mathrm{some}{P}_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}·P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}\},\hfill \\ \hfill & where\hfill \\ \hfill & {\mathcal{T}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)≔\bigcap _{\mu \ge 0}\{(L,D):L+\mu D\ge T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\}.\hfill \end{array}$$

Theorem 3.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, the region ${\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$ defined in (90) is given by

$$\begin{array}{c}\hfill {\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{T}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right),\end{array}$$

(128)

and the achievable region ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$, which is the projection region of the achievable region ${\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ onto the $D\text{-}L$ plane, is given by

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{T}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

(129)

Proof. 

Figure 3 illustrates the proof image using a graph. Let a constance $\mu \ge 0$ be fixed arbitrarily. Like in Figure 3, there exists a boundary point $(D_{\mu },L_{\mu })$ of ${\mathcal{S}}_{\mathcal{E}}^{DL}$ tangent to the line with slope $-\mu$. The intercept of this tangent line is $L_{\mu }+\mu D_{\mu }$.

The minimum $I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+\mu \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]$ characterized by some distribution $P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$ coincides with $L_{\mu }+\mu D_{\mu }$. Therefore,

$$\begin{array}{cc}\hfill {\displaystyle L_{\mu }+\mu D_{\mu }=min}& \{I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})+\mu \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]\mathrm{for}\mathrm{some}{P}_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}·P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}\}.\hfill \end{array}$$

(130)

From (130), we obtain

$$\begin{array}{cc}\hfill & {\displaystyle \{(L,D):L+\mu D\ge L_{\mu }+\mu D_{\mu }\}=\{(L,D):L+\mu D\ge min\{I(X_{\mathcal{H}};{\widehat{X}}_{\mathcal{R}})}\hfill \\ \hfill & +\mu \mathbb{E}\left[d(X_{\mathcal{R}},{\widehat{X}}_{\mathcal{R}})\right]\mathrm{for}\mathrm{some}{P}_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}·P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}\left\}\right\}.\hfill \end{array}$$

(131)

Taking the intersection by $\mu \ge 0$ on the both sides of (131),

$$\begin{array}{cc}\hfill & \bigcap _{\mu \ge 0}\{(L,D):L+\mu D\ge L_{\mu }+\mu D_{\mu }\}=\bigcap _{\mu \ge 0}\{(L,D):L+\mu D\ge T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\}.\hfill \end{array}$$

(132)

The l.h.s. of (131) shows the upper-right region in the first quadrant drawn by the tangent line with a slope $-\mu$ for ${\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$. Since the l.h.s. of (132) is the intersection of the l.h.s. of (131), the l.h.s. of (132) represents ${\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$. From Definition 16, the right-hand side (r.h.s.) of (132) is ${\mathcal{T}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$. As a result, (128) holds. Since ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)={\mathcal{S}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$ from Corollary 3, likewise, (129) holds. □

5.2. Proof Preliminaries

In Section 5.2, we derive two fundamental properties of the minimization about two values and the inequalities about entropy and divergence to prove the strong converse theorem. In Proposition 1, we change the objective function $T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)$ of the region expressed with the tangent plane introduced in Section 5.1 onto the region expressed with divergence.

Proposition 1.

Let $\mu \le 0$ be fixed arbitrarily. For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$,

$$\begin{array}{cc}\hfill T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)& =\underset{\alpha >0}{sup}{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right),\hfill \end{array}$$

(133)

where

$$\begin{array}{cc}\hfill T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)≔& \underset{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}}{min}[I({\tilde{X}}_{\mathcal{H}};{\tilde{\widehat{X}}}_{\mathcal{R}})+\mu \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}},{\tilde{\widehat{X}}}_{\mathcal{R}})\right]\hfill \\ \hfill & +\alpha D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}\Vert Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}})+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}})]\hfill \\ \hfill & =\underset{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}}{min}[I({\tilde{X}}_{\mathcal{H}};{\tilde{\widehat{X}}}_{\mathcal{R}})+\mu \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}},{\tilde{\widehat{X}}}_{\mathcal{R}})\right]]\hfill \\ \hfill & +(\alpha +1)D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}})+\alpha I({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}};{\tilde{\widehat{X}}}_{\mathcal{R}}|{\tilde{X}}_{\mathcal{E}})],\hfill \end{array}$$

(134)

and $Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}$ is the distribution induced from each $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}$.

Proof. 

First, it is clear that $T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\ge T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ for all $\alpha >0$. To prove $T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\le T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ for some $\alpha >0$, for $\alpha >0$, let $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }$ be the distribution that minimizes the r.h.s. of (134) and $Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }=P_{{\tilde{\widehat{X}}}_{\mathcal{R}}|{\tilde{X}}_{\mathcal{E}}}{P}_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}$ be the estimated distribution. Since $G\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\right)I≔({\tilde{X}}_{\mathcal{H}};{\tilde{\widehat{X}}}_{\mathcal{R}})+\mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}},{\tilde{\widehat{X}}}_{\mathcal{R}})\right]$ is non-negative and is bounded above, by setting $a=\log |{\mathcal{X}}_{\mathcal{H}}|+D_{max}$, it must hold that

$$\begin{array}{cc}\hfill & \alpha D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\Vert Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha })\le a\hfill \end{array}$$

and thus

$$\begin{array}{cc}\hfill & D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\Vert Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha })\le (a/\alpha ).\hfill \end{array}$$

Notice that any set of probability distributions on a finite alphabet forms a compact set. Because $G\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\right)$ is a continuous function over a compact set, it is also uniformly continuous. Then, there exists a function $\Delta \left(t\right)$ satisfying $\Delta \left(t\right)\to 0$ as $t\to 0$ such that

$$\begin{array}{cc}\hfill T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)& \ge G\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}{\tilde{X}}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\right)\hfill \\ \hfill & \ge G\left(Q_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}{\tilde{\widehat{X}}}_{\mathcal{R}}}^{\alpha }\right)-\Delta (a/\alpha )\hfill \\ \hfill & \ge T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)-\Delta (a/\alpha ).\hfill \end{array}$$

Consequently, we obtain the desired inequality $T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\le {\displaystyle \underset{\alpha \to \infty }{lim}}{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ by taking $\alpha \to \infty$. □

In the following proposition, we show the inequalities satisfied between $i.i.d.$ source $P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n{X}_{\mathcal{E}}^n}$ and arbitrary source $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}$.

Proposition 2.

For $i.i.d.$ source $P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n{X}_{\mathcal{E}}^n}$, which has the common distribution $P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}$ and arbitrary distribution $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}$, it holds that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n\right|{\tilde{X}}_{\mathcal{E}}^n)+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n{X}_{\mathcal{E}}^n})& \ge n[H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}\right|{\tilde{X}}_{\mathcal{E},J})+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}})],\hfill \end{array}$$

(135)

$$\begin{array}{cc}\hfill H\left({\tilde{X}}_{\mathcal{H}}^n\right)+D(P_{{\tilde{X}}_{\mathcal{H}}^n{\tilde{X}}_{\mathcal{R}}^n}\Vert P_{X_{\mathcal{H}}^n{X}_{\mathcal{R}}^n})& \ge n[H\left({\tilde{X}}_{\mathcal{H},J}\right)+D(P_{{\tilde{X}}_{\mathcal{H},J}{\tilde{X}}_{\mathcal{R},J}}\Vert P_{X_{\mathcal{H}}{X}_{\mathcal{R}}})],\hfill \end{array}$$

(136)

where $J\sim \mathrm{unif}\left(1,\cdots ,n\right)$ is the uniformly random variable over the set $\{1,2,\cdots ,n\}$ for time-sharing and is assumed to be independent of all the other random variables involved.

Proof. 

The l.h.s. of (135) can be represented as

$$\begin{array}{cc}\hfill & H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n\right|{\tilde{X}}_{\mathcal{E}}^n)+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n|{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n|X_{\mathcal{E}}^n}\left|P_{{\tilde{X}}_{\mathcal{E}}^n}\right)+D(P_{{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{\mathcal{E}}^n}).\hfill \end{array}$$

The sum of the first and second terms satisfies the following equation:

$$\begin{array}{cc}\hfill & H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n\right|{\tilde{X}}_{\mathcal{E}}^n)+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n|{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n|X_{\mathcal{E}}^n}\left|P_{{\tilde{X}}_{\mathcal{E}}^n}\right)\hfill \\ \hfill & =\sum _{x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n}{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)\hfill \\ \hfill & ·\left\{\log \frac{1}{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n|{\tilde{X}}_{\mathcal{E}}^n}\left(x_{{\mathcal{E}}^{\mathrm{c}}}^n\right|x_{\mathcal{E}}^n)}+\log \frac{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n|{\tilde{X}}_{\mathcal{E}}^n}\left(x_{{\mathcal{E}}^{\mathrm{c}}}^n\right|x_{\mathcal{E}}^n)}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n|X_{\mathcal{E}}^n}\left(x_{{\mathcal{E}}^{\mathrm{c}}}^n\right|x_{\mathcal{E}}^n)}\right\}\hfill \\ \hfill & =\sum _{x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n}{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)\log \frac{1}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n|X_{\mathcal{E}}^n}\left(x_{{\mathcal{E}}^{\mathrm{c}}}^n\right|x_{\mathcal{E}}^n)}\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}\sum _{x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n}{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)·\left\{\sum _{j=1}^n\log \frac{1}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}|X_{\mathcal{E}}}\left(x_{{\mathcal{E}}^{\mathrm{c}},j}\right|x_{\mathcal{E},j})}\right\}\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{=}n\sum _{x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}}}{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}}(x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}})\log \frac{1}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}|X_{\mathcal{E}}}\left(x_{{\mathcal{E}}^{\mathrm{c}}}\right|x_{\mathcal{E}})}\hfill \\ \hfill & =n\sum _{x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}}}{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}}(x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}})\hfill \\ \hfill & ·\left\{\log \frac{1}{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}|{\tilde{X}}_{\mathcal{E},J}}\left(x_{{\mathcal{E}}^{\mathrm{c}}}\right|x_{\mathcal{E}})}+\log \frac{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}|{\tilde{X}}_{\mathcal{E},J}}\left(x_{{\mathcal{E}}^{\mathrm{c}}}\right|x_{\mathcal{E}})}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}|X_{\mathcal{E}}}\left(x_{{\mathcal{E}}^{\mathrm{c}}}\right|x_{\mathcal{E}})}\right\}\hfill \\ \hfill & =n\{H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}\right|{\tilde{X}}_{\mathcal{E},J})+D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}|{\tilde{X}}_{\mathcal{E},J}}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}|X_{\mathcal{E}}}|P_{{\tilde{X}}_{\mathcal{E},J}}\left)\right\},\hfill \end{array}$$

(137)

where

(a)

follows from the memoryless property of $i.i.d.$ source $P_{X_{{\mathcal{E}}^{\mathrm{c}}}^n{X}_{\mathcal{E}}^n}$;

(b)

holds because $\frac{1}{n}{\sum }_{j=1}^n{P}_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},j}{\tilde{X}}_{\mathcal{E},j}}(x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}})=P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}}(x_{{\mathcal{E}}^{\mathrm{c}}},x_{\mathcal{E}})$.

The third term can be bounded from below as

$$\begin{array}{cc}\hfill D(P_{{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{\mathcal{E}}^n})& =\sum _{j=1}^{n}D(P_{{\tilde{X}}_{\mathcal{E}},j|{\tilde{X}}_{\mathcal{E}}^{j-1}}\Vert P_{X_{\mathcal{E}}}\left|P_{{\tilde{X}}_{\mathcal{E}}^{j-1}}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{c}\right)}{\ge }\sum _{j=1}^{n}D(P_{{\tilde{X}}_{\mathcal{E}},j}\Vert P_{X_{\mathcal{E}}})\hfill \\ \hfill & \stackrel{\left(\mathrm{d}\right)}{\ge }nD(P_{{\tilde{X}}_{\mathcal{E}},J}\Vert P_{X_{\mathcal{E}}}),\hfill \end{array}$$

(138)

where

(c)

follows from the data processing inequality and

(d)

holds because of Jensen’s inequality.

From (137) and (138), (135) can be derived.

Likewise, the l.h.s. of (136) can be represented as

$$\begin{array}{c}\hfill H\left({\tilde{X}}_{\mathcal{H}}^n\right)+D(P_{{\tilde{X}}_{\mathcal{H}}^n}\Vert P_{X_{\mathcal{H}}^n})+D(P_{{\tilde{X}}_{\mathcal{R}}^n|{\tilde{X}}_{\mathcal{H}}^n}\Vert P_{X_{\mathcal{R}}^n|X_{\mathcal{H}}^n}\left|P_{{\tilde{X}}_{\mathcal{H}}^n}\right),\end{array}$$

The sum of the first and second terms satisfies

$$\begin{array}{cc}\hfill & H\left({\tilde{X}}_{\mathcal{H}}^n\right)+D(P_{{\tilde{X}}_{\mathcal{H}}^n}\Vert P_{X_{\mathcal{H}}^n})\hfill \\ \hfill & =\sum _{x_{\mathcal{H}}^n}{P}_{{\tilde{X}}_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)\left\{\log \frac{1}{P_{{\tilde{X}}_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)}+\log \frac{P_{{\tilde{X}}_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)}{P_{X_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)}\right\}\hfill \\ \hfill & =\sum _{x_{\mathcal{H}}^n}{P}_{{\tilde{X}}_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)\log \frac{1}{P_{X_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)}\hfill \\ \hfill & =\sum _{x_{\mathcal{H}}^n}{P}_{{\tilde{X}}_{\mathcal{H}}^n}\left(x_{\mathcal{H}}^n\right)·\left\{\sum _{j=1}^n\log \frac{1}{P_{X_{\mathcal{H}}}\left(x_{\mathcal{H},j}\right)}\right\}\hfill \\ \hfill & \stackrel{\left(\mathrm{e}\right)}{=}n\sum _{x_{\mathcal{H}}}{P}_{{\tilde{X}}_{\mathcal{H}},J}\left(x_{\mathcal{H}}\right)\log \frac{1}{P_{X_{\mathcal{H}}}\left(x_{\mathcal{H}}\right)}\hfill \\ \hfill & =n\sum _{x_{\mathcal{H}}}{P}_{{\tilde{X}}_{\mathcal{H}},J}\left(x_{\mathcal{H}}\right)\left\{\log \frac{1}{P_{{\tilde{X}}_{\mathcal{H}},J}\left(x_{\mathcal{H}}\right)}+\log \frac{P_{{\tilde{X}}_{\mathcal{H}},J}\left(x_{\mathcal{H}}\right)}{P_{X_{\mathcal{H}}}\left(x_{\mathcal{H}}\right)}\right\}\hfill \\ \hfill & =n\{H\left({\tilde{X}}_{\mathcal{H},J}\right)+D(P_{{\tilde{X}}_{\mathcal{H},J}}\Vert P_{X_{\mathcal{H}}})\},\hfill \end{array}$$

(139)

where

(e)

holds because $\frac{1}{n}{\sum }_{j=1}^n{P}_{{\tilde{X}}_{\mathcal{H},j}}\left(x_{\mathcal{H}}\right)=P_{{\tilde{X}}_{\mathcal{H},J}}\left(x_{\mathcal{H}}\right)$.

For the third term, it holds that

$$\begin{array}{cc}\hfill D(P_{{\tilde{X}}_{\mathcal{R}}^n|{\tilde{X}}_{\mathcal{H}}^n}\Vert P_{X_{\mathcal{R}}^n|X_{\mathcal{H}}^n}|P_{{\tilde{X}}_{\mathcal{H}}^n})& =\sum _{j=1}^{n}D(P_{{\tilde{X}}_{\mathcal{R},j}|{\tilde{X}}_{\mathcal{H}}^n{\tilde{X}}_{\mathcal{R}}^{j-1}}\Vert P_{X_{\mathcal{R}}|X_{\mathcal{H}}}\left|P_{{\tilde{X}}_{\mathcal{H}}^n{\tilde{X}}_{\mathcal{R}}^{j-1}}\right)\hfill \\ \hfill & \stackrel{\left(\mathrm{f}\right)}{\ge }\sum _{j=1}^{n}D(P_{{\tilde{X}}_{\mathcal{R},j}|{\tilde{X}}_{\mathcal{H},j}}\Vert P_{X_{\mathcal{R}}|X_{\mathcal{H}}}\left|P_{{\tilde{X}}_{\mathcal{H},j}}\right)\hfill \\ \hfill & \ge nD(P_{{\tilde{X}}_{\mathcal{R},J}|{\tilde{X}}_{\mathcal{H},J}}\Vert P_{X_{\mathcal{R}}|X_{\mathcal{H}}}|P_{{\tilde{X}}_{\mathcal{H}},J}),\hfill \end{array}$$

(140)

where

(f)

follows from the log sum inequality.

From (139) and (140), we obtain (136). □

5.3. Strong Converse Theorem

We shall establish the strong converse theorem, which is the main result of this section. Before proving the theorem, we state the lemma of the key tool in the proof about a single-letterized $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ and a $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)$, which are introduced in Proposition 1.

Lemma 7.

For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$, all $n\in \mathbb{N}$, $\mu \ge 0$ and $\alpha >0$, it holds that

$$\begin{array}{c}\hfill T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)\ge n{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

As the main theorem of this section, we show the strong converse theorem for the utility–privacy trade-offs.

Theorem 4.

Strong converse theorem: For any $\mathcal{E}$ such that $\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$ and all $0<ϵ<1$, it holds that

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathcal{E}}^{DL}\left(ϵ\right|P_{X_{\mathcal{K}}})={\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right).\end{array}$$

Remark 5.

Theorem 4 suggests that regardless of the value of ϵ, the region ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(ϵ\right|P_{X_{\mathcal{K}}})$ is equal to ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$.

5.4. Proof of Lemma 7

Lemma 7 indicates that the function $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)$, whose argument $P_{X_{\mathcal{K}}}^n$ is a probability distribution over ${\mathcal{X}}_{\mathcal{K}}^n$, can be lower-bounded by the n-fold of a single-letterized function $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$. Before describing the detailed proof, we state the outline of the proof: (i) We first express the function $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)$ as the maximum of the difference of two functions denoted by $G_1$ and $G_2$ as in (142). (ii) Then, we show that the first function $G_1$ can be lower-bounded by the n-fold of its single-letterized function as in (143), while the second function $G_2$ can be upper-bounded by the n-fold of its single-letterized function as in (147). This outline of the proof is similar to the Proof of Theorem 4, 16 with a slight modification of the function $G_2$.

For a given distribution $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}$, let functions $G_1\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\right)$ and $G_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}\right)$ be defined as

$$\begin{array}{cc}\hfill G_1\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\right)≔& H\left({\tilde{X}}_{\mathcal{H}}^n\right)+\alpha H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n\right|{\tilde{X}}_{\mathcal{E}}^n)+(\alpha +1)D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n),\hfill \\ \hfill G_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}\right)≔& H\left({\tilde{X}}_{\mathcal{H}}^n\right|{\tilde{\widehat{X}}}_{\mathcal{R}}^n)-\mu \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\right]+\alpha H\left({\tilde{X}}_{\mathcal{E}}^n\right|{\tilde{X}}_{\mathcal{E}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n).\hfill \end{array}$$

(141)

Using these functions, and in view of (134), $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n\right)$ can be written as

$$\begin{array}{cc}\hfill & T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n\right)=\underset{P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}}{min}\left[G_1\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\right)-G_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}\right)\right].\hfill \end{array}$$

(142)

For fixed $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}$, from Proposition 2, it holds that

$$\begin{array}{c}\hfill G_1\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\right)\ge n{G}_1\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}}\right).\end{array}$$

(143)

Next, we consider the function $G_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}\right)$. For the first term on the r.h.s. of (141), it holds that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}_{\mathcal{H}}^n\right|{\tilde{\widehat{X}}}_{\mathcal{R}}^n)& =\sum _{j=1}^{n}H\left({\tilde{X}}_{\mathcal{H},j}\right|{\tilde{X}}_{\mathcal{H}}^{j-1},{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\hfill \\ \hfill & \le \sum _{j=1}^{n}H\left({\tilde{X}}_{\mathcal{H},j}\right|{\tilde{\widehat{X}}}_{\mathcal{R},j})\hfill \\ \hfill & =n·\frac{1}{n}\sum _{j=1}^{n}H\left({\tilde{X}}_{\mathcal{H},j}\right|{\tilde{\widehat{X}}}_{\mathcal{R},j})\hfill \\ \hfill & =nH\left({\tilde{X}}_{\mathcal{H},J}\right|{\tilde{\widehat{X}}}_{\mathcal{R},J},J)\hfill \\ \hfill & \le nH\left({\tilde{X}}_{\mathcal{H},J}\right|{\tilde{\widehat{X}}}_{\mathcal{R},J}).\hfill \end{array}$$

(144)

The second term of (141) can be expressed as follows:

$$\begin{array}{cc}\hfill \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\right]& =\sum _{x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n}{P}_{{\tilde{X}}_{\mathcal{R}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}(x_{\mathcal{R}}^n,{\widehat{x}}_{\mathcal{R}}^n)·\left\{\sum _{j=1}^{n}d(x_{\mathcal{R},j},{\widehat{x}}_{\mathcal{R},j})\right\}\hfill \\ \hfill & =\sum _{j=1}^n\sum _{x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}}}{P}_{{\tilde{X}}_{\mathcal{R},j}{\tilde{\widehat{X}}}_{\mathcal{R},j}}(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})d(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})\hfill \\ \hfill & \stackrel{\left(\mathrm{a}\right)}{=}n\sum _{x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}}}{P}_{{\tilde{X}}_{\mathcal{R},J}{\tilde{\widehat{X}}}_{\mathcal{R},J}}(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})d(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})\hfill \\ \hfill & =n\mathbb{E}\left[d({\tilde{X}}_{\mathcal{R},J},{\tilde{\widehat{X}}}_{\mathcal{R},J})\right],\hfill \end{array}$$

(145)

where

(a)

follows from $\frac{1}{n}{\sum }_{j=1}^n{P}_{{\tilde{X}}_{\mathcal{R},j}{\tilde{\widehat{X}}}_{\mathcal{R},j}}(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})=P_{{\tilde{X}}_{\mathcal{R},J}{\tilde{\widehat{X}}}_{\mathcal{R},J}}(x_{\mathcal{R}},{\widehat{x}}_{\mathcal{R}})$.

Moreover, for the third term of (141), it holds that

$$\begin{array}{cc}\hfill H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n\right|{\tilde{X}}_{\mathcal{E}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)& =\sum _{j=1}^{n}H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},j}\right|{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^{j-1},{\tilde{X}}_{\mathcal{E}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\hfill \\ \hfill & \le \sum _{j=1}^{n}H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},j}\right|{\tilde{X}}_{\mathcal{E},j},{\tilde{\widehat{X}}}_{\mathcal{R},j})\hfill \\ \hfill & =n·\frac{1}{n}\sum _{j=1}^{n}H\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},j}\right|{\tilde{X}}_{\mathcal{E},j},{\tilde{\widehat{X}}}_{\mathcal{R},j})\hfill \\ \hfill & =nH\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}\right|{\tilde{X}}_{\mathcal{E},J},{\tilde{\widehat{X}}}_{\mathcal{R},J},J)\hfill \\ \hfill & \le nH\left({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}\right|{\tilde{X}}_{\mathcal{E},J},{\tilde{\widehat{X}}}_{\mathcal{R},J}).\hfill \end{array}$$

(146)

From (144)–(146), we obtain

$$\begin{array}{c}\hfill G_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}\right)\le n{G}_2\left(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}},J}{\tilde{X}}_{\mathcal{E},J}{\tilde{\widehat{X}}}_{\mathcal{R},J}}\right).\end{array}$$

(147)

Consequently, since (143) and (147) are satisfied for an arbitrary $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n{\tilde{\widehat{X}}}_{\mathcal{R}}^n}$, the proof is completed.

5.5. Proof of Strong Converse Theorem

For any given $ϵ>0$, fix the rate pair $(D,L)\in {\mathcal{L}}_{\mathcal{E}}^{DL}\left(ϵ\right|P_{X_{\mathcal{K}}})$ arbitrarily. Then, by definition, there exists a code $(f_n,g_n)$ satisfying (79) and (80). For this code $(f_n,g_n)$, a set $\mathcal{D}$ is defined as

$$\begin{array}{c}\hfill \mathcal{D}≔\{(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n):d(x_{\mathcal{R}}^n,g_n\left(f_n\left(x_{\mathcal{E}}^n\right)\right))\le nD\}.\end{array}$$

We derive a distribution $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}$ as

$$\begin{array}{c}\hfill P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)≔\frac{P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)1l[(x_{{\mathcal{E}}^{\mathrm{c}}}^n,x_{\mathcal{E}}^n)\in \mathcal{D}]}{P_{X_{{\mathcal{E}}^{\mathrm{c}}{X}_{\mathcal{E}}}}^n\left(\mathcal{D}\right)}.\end{array}$$

It is obvious that the excess-distortion probability measured by $P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}$ is 0; that is, ${\tilde{X}}_{\mathcal{R}}^n$ and ${\tilde{\widehat{X}}}_{\mathcal{R}}^n=g_n\left(f_n\left({\tilde{X}}_{\mathcal{E}}^n\right)\right)$ satisfy $\mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\right]\le nD$. Thus, by imitating the proof approach of the standard weak converse theorem, it holds that

$$\begin{array}{cc}\hfill & n(L+\mu D)\ge I({\tilde{X}}_{\mathcal{H}}^n;{\tilde{\widehat{X}}}_{\mathcal{R}}^n)+\mu \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\right],\hfill \end{array}$$

(148)

$$\begin{array}{cc}\hfill & D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n)=\log \frac{1}{P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n\left(\mathcal{D}\right)}\le \log \frac{1}{1-ϵ}.\hfill \end{array}$$

(149)

From (148), the following equation is obtained:

$$\begin{array}{cc}\hfill n(L+\mu D)& \stackrel{\left(\mathrm{a}\right)}{\ge }I({\tilde{X}}_{\mathcal{H}}^n;{\tilde{\widehat{X}}}_{\mathcal{R}}^n)+\mu \mathbb{E}\left[d({\tilde{X}}_{\mathcal{R}}^n,{\tilde{\widehat{X}}}_{\mathcal{R}}^n)\right]\hfill \\ \hfill & +((\alpha +1)D(P_{{\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n{\tilde{X}}_{\mathcal{E}}^n}\Vert P_{X_{{\mathcal{E}}^{\mathrm{c}}}{X}_{\mathcal{E}}}^n)+\alpha I({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n;{\tilde{\widehat{X}}}_{\mathcal{R}}^n|{\tilde{X}}_{\mathcal{E}}^n))\hfill \\ \hfill & -(\alpha +1)\log \frac{1}{1-ϵ}\hfill \\ \hfill & \stackrel{\left(\mathrm{b}\right)}{\ge }{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)-(\alpha +1)\log \frac{1}{1-ϵ},\hfill \end{array}$$

where

(a)

follows from (149) and $I({\tilde{X}}_{{\mathcal{E}}^{\mathrm{c}}}^n;{\tilde{\widehat{X}}}_{\mathcal{R}}^n|{\tilde{X}}_{\mathcal{E}}^n)=0$,

(b)

is due to (134).

Since $T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}^n\right)\ge n{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ from Lemma 7, we have

$$\begin{array}{cc}\hfill L+\mu D& \ge T_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)-\frac{(\alpha +1)}{n}\log \frac{1}{1-ϵ},\hfill \end{array}$$

and therefore

$$\begin{array}{cc}\hfill \underset{\alpha >0}{sup}(L+\mu D)& \ge \underset{\alpha >0}{sup}\left[T_{ϵ}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)-\frac{(\alpha +1)}{n}\log \frac{1}{1-ϵ}\right].\hfill \end{array}$$

Because $T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)={sup}_{\alpha >0}{T}_{\mathcal{E}}^{\mu ,\alpha }\left(P_{X_{\mathcal{K}}}\right)$ from Proposition 1, it holds that for an arbitrary $\alpha >0$,

$$\begin{array}{cc}\hfill L+\mu D& \ge T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)-\frac{(\alpha +1)}{n}\log \frac{1}{1-ϵ}.\hfill \end{array}$$

Hence, it holds that

$$\begin{array}{cc}\hfill L+\mu D& \ge \underset{n\to \infty }{lim}\left(T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)-\frac{(\alpha +1)}{n}\log \frac{1}{1-ϵ}\right)\hfill \\ \hfill & =T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\mathrm{for}\mathrm{every}\mu \ge 0.\hfill \end{array}$$

(150)

For the set of $(D,L)$ satisfying (150), varying $\mu \ge 0$ arbitrarily and taking the intersection, we have

$$\begin{array}{c}\hfill (D,L)\in \bigcap _{\mu \ge 0}\{(D,L):L+\mu D\ge T_{\mathcal{E}}^{\mu }\left(P_{X_{\mathcal{K}}}\right)\}.\end{array}$$

(151)

From Theorem 3, the r.h.s. of (151) is equal to ${\mathcal{L}}_{\mathcal{E}}^{DL}\left(P_{X_{\mathcal{K}}}\right)$. This proof is completed.

6. Discussion

6.1. Numerical Calculation of Coding Rate, Utility, and Privacy for Decoder

In this section, we show some numerical calculations of the achievable region ${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ in Corollaries 1 and 2, respectively. In general, it is difficult to compute the achievable region ${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$. Nevertheless, to obtain some insight, let us consider the three tractable but essential cases. In these calculations, the number of public attributes is one $\left(\right|\mathcal{R}|=1)$ and the number of private attributes is two $\left(\right|\mathcal{H}|=2)$. We assume that each of the attributes is binary. Here, note again that the coding rate R acts like the rate-distortion function in rate-distortion theory (cf. (Section 10 in [27])). For fixed D and L, a smaller coding rate is better.

In the first example, we calculated the L-D graph of theoretical limits in case (i) $\mathcal{E}=\mathcal{K}$, case (ii) $\mathcal{E}=\mathcal{R}$, and case (iii) $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$ (Figure 4). As a result, the achievable privacy leakage L becomes small as D becomes large if we do not impose any restrictions on the value of R. For a given D, the privacy leakage for the decoder in case (i) $\mathcal{E}=\mathcal{K}$ is the smallest, and the one in case (ii) $\mathcal{E}=\mathcal{R}$ is the largest in all cases. The second example calculated the R-D graph of theoretical limits in cases (i), (ii), and (iii) (Figure 5). We can see that the minimum coding rates for a given D coincide in all cases if we do not impose any restrictions on the value of L. In the third example, we calculated the optimal privacy leakage L for fixed D and the corresponding coding rates R in cases (i), (ii), and (iii) (

Table 1. Minimum L and its corresponding R for $D=0.0500$.

Cases	Leakage L	Coding Rate R
case (ii)	0.019512	0.494629
case (iii)	0.008298	0.527700
case (i)	0.005107	0.539478

,

Table 2. Minimum L and its corresponding R for $D=0.100$.

Cases	Leakage L	Coding Rate R
case (ii)	0.015378	0.368062
case (iii)	0.002656	0.418826
case (i)	0.000000	0.429490

and

Table 3. Minimum L and its corresponding R for $D=0.1500$.

Cases	Leakage L	Coding Rate R
case (ii)	0.011748	0.270436
case (iii)	0.002032	0.294424
case (i)	0.000000	0.382211

). As a result, the optimal privacy leakage in cases (i) and (iii) is smaller than the one in case (ii), whereas for the optimal privacy leakage, the achievable coding rates in cases (i) and (iii) is larger than the one in case (ii).

Next, we discuss these results. In Figure 4, in comparison with each case, we can verify that for a given D, the more private information is encoded, the smaller the achievable minimum privacy leakage is. Figure 5 suggests that if the coding rate should be minimized, it suffices to encode only the public attributes. This result is evident from Corollaries 1 and 2 because the condition on the choice of test channel $P_{{\widehat{X}}_{\mathcal{R}}|X_{\mathcal{E}}}$ in case (i) is weaker than the one in case (ii), and if an appropriate test channel is taken in case (i), it is also appropriate in case (ii). It is indicated that the achievable region in case (ii) is also the one in cases (i) and (iii). The opposite is not the case. From Table 1, Table 2 and Table 3, we can confirm the trade-off between the optimal privacy leakage L for a fixed D and the corresponding coding rate R in comparison with each case.

Summarizing the foregoing arguments, we have discussed the relationship between utility and privacy in Figure 4, the one between utility and coding rate in Figure 5, and the one between privacy and coding rate in Table 1, Table 2 and Table 3. From the discussion about Figure 5, some readers may suspect that case (i) is the best-encoded information because the achievable region in cases (ii) and (iii) is the one in case (i). This is true if we do not consider the leakage for the encoder. However, this is not true if we consider the leakage for the encoder, that is, the measurement of privacy for the encoder (see (12) or (76)). In the next section, we discuss this point in detail.

6.2. Significance of Limited Leakage for Encoder

In this section, we discuss the significance of evaluating the leakage for the encoder. Our goal of this discussion is to show that the best-encoded information may be case (iii) $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$ if we take the limited leakage for the encoder into consideration.

The first issue is the amount of encoded information. Some readers may think that it is better that more encoded information is inputted into the encoder. However, there are pros and cons.

Pros: 

The achievable regions ${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ become larger.

Cons: 

The leakage for the encoder increases.

From this point of view, we can come up with the idea that there exists the best-encoded information in case (iii) $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$ if we impose some constraint on the leakage for the encoder. This idea is the key point of this paper.

The second issue is the significance of the limited leakage for the encoder. Figure 6 shows the Hasse diagram, which represents the inclusion relation about the index sets of attributes. The Hasse diagram is often used to represent inclusion relations, for example, $\mathcal{R}\subset {\mathcal{E}}_2\subset {\mathcal{E}}_1\subset \mathcal{K}$.

We can also regard Figure 6 as the Hasse diagram that represents the inclusion relation for the achievable regions ${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ because the index sets of attributes ($\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}$) corresponds to the encoded information ($X_{\mathcal{E}}$) and the encoded information corresponds to the achievable region (${\mathcal{C}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$). In addition, the diagram in Figure 6 has another property, which is that the superordinate sets have a larger amount of privacy leakage for the encoder than the subordinate sets since the index sets of attributes correspond to the privacy leakage for the encoder.

Let us consider a practical application. We assume that the data aggregator, that is, the encoder, tries to gather encoded information from some application user and hopes to develop the utility of the application while limiting the amount of leakage for $X_{\mathcal{H}}^n$ by $E\ge 0$, that is, $e_n\le E$. More precisely, for a given E, we want to find which subsets of $\mathcal{K}$ are sufficient to characterize

$$\begin{array}{c}\hfill {\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)≔\bigcup _{\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}}\left\{(R,D,L):(R,D,L,E)\in {\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\right\},\\ \hfill {\mathcal{L}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)≔\bigcup _{\mathcal{R}\subseteq \mathcal{E}\subseteq \mathcal{K}}\left\{(R,D,L):(R,D,L,E)\in {\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)\right\},\end{array}$$

where ${\mathcal{C}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}_{\mathcal{E}}\left(P_{X_{\mathcal{K}}}\right)$ are defined in Definitions 2 and 11, respectively. The process is as follows.

Step 1:

Check the user’s requirements and impose the restriction on the privacy leakage for the encoder.

Figure 7 shows the Hasse diagram for Step 1. The blue dotted line means the border line satisfies the restriction of the privacy leakage for the encoder. Therefore, the index sets ${\mathcal{E}}_1$ and $\mathcal{K}$ are not suitable as the index sets of encoded information.

Step 2:

Check the inclusion relation between index sets.

Figure 8 shows the Hasse diagram for Step 2. From Figure 6, we can find that

$$\begin{array}{c}\hfill \mathcal{R}\subset {\mathcal{E}}_2,\mathcal{R}\subset {\mathcal{E}}_3,\mathcal{R}\subset {\mathcal{E}}_5,{\mathcal{E}}_3\subset {\mathcal{E}}_4,{\mathcal{E}}_5\subset {\mathcal{E}}_4.\end{array}$$

Therefore, the index sets $\mathcal{R}$, ${\mathcal{E}}_3$, and ${\mathcal{E}}_5$ are not suitable as the index sets of encoded information.

Figure 9 shows the Hasse diagram obtained after Step 2. From Figure 9, the remaining index sets are ${\mathcal{E}}_2$ and ${\mathcal{E}}_4$. Therefore, if we impose restriction on privacy leakage for the encoder, the index sets ${\mathcal{E}}_2$ or ${\mathcal{E}}_4$ form the Pareto area in this multi-objective optimization problem. In other words, there exists a system that satisfies the user’s requirements E of the maximum amount of leakage to the encoder, and the achievable regions are given by ${\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)={\mathcal{C}}_{{\mathcal{E}}_2}^{RDL}\left(P_{X_{\mathcal{K}}}\right)\cup {\mathcal{C}}_{{\mathcal{E}}_4}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{L}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)={\mathcal{L}}_{{\mathcal{E}}_2}^{RDL}\left(P_{X_{\mathcal{K}}}\right)\cup {\mathcal{L}}_{{\mathcal{E}}_4}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$.

From the discussion above, we mention that the best-encoded information is case (iii) $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$ if we take the limited leakage for the encoder into account. This concept is one of the most important novelties in this paper.

If E satisfies some condition, then ${\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)$ can be characterized by the expressions given by Yamamoto [1] (cf. Remark 3). More specifically, the region ${\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)$ can be given by

$$\begin{array}{cc}\hfill {\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)& ={\mathcal{S}}_{\mathcal{K}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)\hfill \end{array}$$

if $E\ge H\left(X_{\mathcal{K}}\right)$ and

$$\begin{array}{cc}\hfill {\mathcal{C}}^{RDL}\left(P_{X_{\mathcal{K}}}\right|E)& ={\mathcal{S}}_{\mathcal{R}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)\hfill \end{array}$$

if $H\left(X_{\mathcal{R}}\right)\le E<H\left(X_{\mathcal{E}}\right)$ for any $\mathcal{R}\subset \mathcal{E}$ with $\mathcal{R}\ne \mathcal{E}$, where the regions ${\mathcal{S}}_{\mathcal{K}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ and ${\mathcal{S}}_{\mathcal{R}}^{RDL}\left(P_{X_{\mathcal{K}}}\right)$ are given in [1] (cf. Remark 3).

6.3. Discussion on Measures for Privacy Leakage

This paper adopts the mutual information as the measure of privacy leakage as in (12), (13), (76), and (77). However, some less likely data can be leaked even though the database satisfies the theoretical limit of privacy leakage. For example, let $(X,Y)$ be a pair of correlated random variables whose $I(X;Y)$ is very small. However, there may exist a pair of $(x_1,y_1)$ such that $Y=y_1$ can imply $X=x_1$ with high probability. To put it differently, the receiver can tell the value of X if it observes $Y=y_1$. The theoretical limit evaluated with mutual information cannot prevent such a scenario. To circumvent this scenario, we suggest the other measurement adopted in related studies. A promising candidate to avoid this problem is to employ Rényi information of higher orders [30], maximal leakage [15], and maximal $\alpha$-leakage [16,17,18,21].

7. Conclusions

In this paper, we strengthened the results in [3] mainly by establishing three coding theorems in a privacy-constrained source coding problem. In Section 3 and Section 4, two theorems are made about the first-order rate analysis in which utility is measured by the expected distortion or the excess-distortion probability for case (iii), $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$. The novelty is the introduction of the measure of privacy for the encoder along with the use of the excess-distortion probability. The obtained characterization reduces to the one given in [3] derived based on the expected distortion when the leakage for the encoder is not limited, and the result shows that employing an excess-distortion probability does not change the achievable region from the one with an expected distortion. In Section 5, we establish the strong converse theorem for utility–privacy trade-offs. Although the described result is for the projected plane of utility and privacy for the decoder for simplicity, we can also incorporate the measure of privacy for the encoder. Finally, we discuss the significance of the encoded information considering limited leakage for the encoder. The argument suggests that the best-encoded information can be case (iii) $\mathcal{R}\subset \mathcal{E}\subset \mathcal{K}$ if some constraint is imposed on the privacy leakage for the encoder.

As future work, the second-order rate analysis for utility–privacy trade-offs is an interesting research topic [4,5,6]. Moreover, the strong converse theorem and the second-order rate analysis for the four-dimensional region of coding rate, utility, privacy for the decoder, and privacy for the encoder are more challenging tasks. It is also worth analyzing the achievable region with the other privacy measures such as Rényi information [30], maximal leakage [15], and maximal $\alpha$-leakage [16,17,18,21]. This paper analyzed the theoretical limits of coding, but understanding how to achieve the theoretical limits remains open. The construction of good codes is also an important subject. Extensions of this paper’s scenario to coding with side information [2,25] are also of interest.