Mechanisms for Robust Local Differential Privacy

Abstract

We consider privacy mechanisms for releasing data $X=(S,U)$, where S is sensitive and U is non-sensitive. We introduce the robust local differential privacy (RLDP) framework, which provides strong privacy guarantees, while preserving utility. This is achieved by providing robust privacy: our mechanisms do not only provide privacy with respect to a publicly available estimate of the unknown true distribution, but also with respect to similar distributions. Such robustness mitigates the potential privacy leaks that might arise from the difference between the true distribution and the estimated one. At the same time, we mitigate the utility penalties that come with ordinary differential privacy, which involves making worst-case assumptions and dealing with extreme cases. We achieve robustness in privacy by constructing an uncertainty set based on a Rényi divergence. By analyzing the structure of this set and approximating it with a polytope, we can use robust optimization to find mechanisms with high utility. However, this relies on vertex enumeration and becomes computationally inaccessible for large input spaces. Therefore, we also introduce two low-complexity algorithms that build on existing LDP mechanisms. We evaluate the utility and robustness of the mechanisms using numerical experiments and demonstrate that our mechanisms provide robust privacy, while achieving a utility that is close to optimal.

1. Introduction

We consider the setting in which an aggregator collects data from many users with the purpose of, for instance, computing statistics or training a machine learning model. In particular, the data contain sensitive information and users do not trust the aggregator. Therefore, they employ a privacy mechanism that transforms the data before sending it to the aggregator. Users have data $X=(S,U)$ from a finite alphabet $\mathcal{X}=\mathcal{S}\times \mathcal{U}$, where $s\in \mathcal{S}$ is sensitive information and $u\in \mathcal{U}$ is non-sensitive. Data are distributed i.i.d. across users according to the distribution $P^{*}$. In order to preserve their privacy, users disclose a sanitized version Y of X by using a privacy mechanism $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$. The aim is that Y contains as much information about X as possible without leaking too much information about S. The challenge that is addressed in this paper is to develop good privacy mechanisms. This scenario and closely related ones were studied in, for instance [1,2,3,4,5,6,7,8,9,10,11]. In this paper, we use the following version of local differential privacy (LDP), as introduced in [3]:

$$\mathbb{P}(Y=y|S=s)\le {\mathrm{e}}^{\epsilon }\mathbb{P}(Y=y|S=s^{\prime }),$$

(1)

for all $s,s^{\prime }\in \mathcal{S}$ and privacy parameter $\epsilon >0$. In addition, we measure the utility of Y through the mutual information $\mathrm{I}(X;Y)$. We discuss differences with related work in Section 2.

Note that if all information is sensitive, i.e., if $\mathcal{X}=\mathcal{S}$, (1) reduces to

$$\mathbb{P}(Y=y|X=x)\le {\mathrm{e}}^{\epsilon }\mathbb{P}(Y=y|X=x^{\prime }),$$

(2)

which is the traditional LDP constraint [1,2,5]. An important property of (2) is that it does not depend on $P^{*}$, but only on $\mathcal{Q}$. The independence of $P^{*}$ is a key factor in the success of differential privacy, since it leverages the need to make assumptions about the distribution of the data or on the background/side-knowledge available to the aggregator. As is clear from (1), however, independence from $P^{*}$ no longer holds if not all data are sensitive.

Assuming that $P^{*}$ is known, one can develop good privacy mechanisms for various settings with partially sensitive information [3,6,12]. In practice, however, $P^{*}$ has to be modeled using domain knowledge or estimated from data, leading to errors. The prevalent approach in the literature has been to develop privacy mechanisms based on a (point) estimate $\widehat{P}$ and analyze sensitivity with respect to. errors in this estimate. In this work, we follow the approach that was proposed in [13,14], which is to construct a set $\mathcal{F}$ of probability distributions that we are confident contains $P^{*}$. Subsequently, we construct privacy mechanisms that aim to maximize utility, while satisfying (1) for all probability distributions in $\mathcal{F}$. We call the resulting privacy framework robust local differential privacy (RLDP).

In a sense, RLDP is a relaxed form of privacy. Indeed, it may seem appealing, but it is—as we illustrate next—often infeasible to enforce (1) for all possible distributions. To this end, we consider two extreme cases. First, consider a joint distribution of S and U under which $S=U$. Intuitively, we cannot disclose much information about U, since this is directly leaking information about S. As such, the utility of Y is low. Next, consider a joint distribution under which S and U are independent. Intuitively, we can disclose U without additional precautions, providing a high utility on Y. The point is that we need to design a single privacy mechanism $\mathcal{Q}$ that satisfies (1) for all distributions, including the ‘worst case’ in which $S=U$, leading to low utility Y. In this work, we take the mid-ground between, on the one hand, only using a point estimate $\widehat{P}$ and, on the other hand, using all possible distributions. We do so by defining a set of ‘reasonable’ distributions $\mathcal{F}$. In particular, we construct $\mathcal{F}$ based on public side-information. This public side information consists of n pairs of data $(s_1,u_1),\dots ,(s_n,u_n)$, which like the data of users are i.i.d. according to unknown distribution $P^{*}$. Our set $\mathcal{F}$ is constructed as a closed ball under a Rényi divergence around the maximum likelihood point estimate $\widehat{P}$ of $P^{*}$. By doing so, we are (statistically) confident that $\mathcal{F}$ contains $P^{*}$, with the radius of the ball controlling the confidence level.

The RLDP framework is an instance of the more general Pufferfish framework [15]. In Section 2, we make this connection explicit and use it to describe the semantic privacy guarantees that are offered by RLDP.

The main contributions of this paper are as follows:

• We use a Rényi divergence to construct $\mathcal{F}$ and analyze the resulting structure and statistics of $\mathcal{F}$. In particular, we demonstrate that projections of $\mathcal{F}$ are again balls under the same divergence. Moreover, we bound the projected sets in terms of an ${\mathcal{l}}_1$ norm.
• Using these results we approximate $\mathcal{F}$ by an enveloping polytope. We then use techniques from robust optimization [16,17,18] to characterize PolyOpt, the mechanism that is optimal over this polytope.
• A drawback of this method is that it relies on vertex enumeration and is, therefore, computationally unfeasible for large alphabets. Therefore, we introduce two low-complexity privacy mechanisms. The first is independent reporting (IR), in which S and U are reported through separate LDP mechanisms.
• We characterize the conditions that underlying LDP mechanisms have to satisfy in order for IR to ensure RLDP. Furthermore, while IR can incorporate any LDP mechanism, we show that it is optimal to use randomized response [19]. This drastically reduces the search space and allows us to find the optimal IR mechanism using low-dimensional optimization.
• The second low-complexity mechanism that we develop is called secret-randomized response (SRR) and is based on randomized response.
• We show that SRR maximizes mutual information in the low-privacy regime for the case that $\mathcal{F}$ is the entire probability simplex.
• We demonstrate the improved utility of RLDP over LDP with numerical experiments. In particular, we compare the performance of our mechanisms with generalized random response [5]. We provide results for both synthetic data sets and real-world census data.

The structure of this paper is as follows: After discussing related work in Section 2, we describe the model in detail in Section 3. In Section 4, we present results on the structure and statistics of projections of $\mathcal{F}$. These results are used in Section 5 to develop the PolyOpt privacy mechanism. Low-complexity privacy mechanisms are presented in Section 6 and Section 7. In Section 8, we evaluate the discussed methods experimentally. Finally, in Section 9, we provide a discussion of our results and provide an outlook on future work. Most proofs are deferred to Appendix A.

Part of this paper was presented at the IEEE International Symposium on Information Theory 2021 [14]. In this paper, we generalize from a ${\chi }^2$-divergence to an arbitrary Rényi divergence. Moreover, Section 4 and Section 6, most of Section 8, and all proofs are new in the current paper.

2. Related Work

2.1. The Pufferfish Framework

Our RLDP framework is an instance of the more general Pufferfish framework [15]. In this subsection, we make this connection explicit and elaborate on the semantic guarantees offered by RLDP.

A privacy definition following the Pufferfish framework specifies (i) a set of potential secrets, (ii) a set of discriminative pairs of secrets, and (iii) a set of assumptions about how data are generated. In RLDP the potential secrets are the possible values of S, i.e., $\mathcal{S}$. We want to prevent the aggregator from learning anything about S. This means that it should not be able to distinguish the case $S=s$ from $S=s^{\prime }$ for all $s\ne s^{\prime }$, so all non-identical pairs are discriminative. Note that this relies on $\mathcal{S}$ being finite, with extensions to continuous $\mathcal{S}$ discussed in detail in [15].

The set of assumptions on how data are generated consist, in our setting, of probability distributions over $\mathcal{X}$. A key idea in Pufferfish is that this set explicitly models the information that is available to an attacker, i.e., an entity that is trying to infer information about S by observing Y. In our setting, the aggregator is the only attacker and a probability distribution P over $\mathcal{X}$ captures the beliefs that the attacker has about S prior to seeing Y. We can rewrite (1) as

$$\frac{{\mathbb{P}}_{X\sim P}(S=s|Y=y)}{{\mathbb{P}}_{X\sim P}(S=s^{\prime }|Y=y)}\le {\mathrm{e}}^{\epsilon }\frac{{\mathbb{P}}_{X\sim P}(S=s)}{{\mathbb{P}}_{X\sim P}(S=s^{\prime })}$$

(3)

and see that our local differential privacy constraint (1) can be interpreted as the condition that the posterior distribution of S after seeing Y must be very close to the prior distribution. The relevance of P is that it captures a specific set of beliefs of the attacker. As such, we want (3) to hold for various values of P, where each P captures specific background/side-knowledge available to the attacker/aggregator. Note that by doing so we are not making any claims about the actual knowledge available to the aggregator, but instead describing the possible scenarios for which we want to protect the privacy of users. In Pufferfish, these possible scenarios are called the set of assumptions on how data are generated, and in RLDP this is $\mathcal{F}$.

Often, side-information in the form of domain knowledge or existing data is publicly available; i.e., to both the users and the aggregator. This public side-information may suggest, for instance, that there is, at most, limited dependence between S and U. In that case, protecting against attackers who have the belief that $S=U$ incurs an enormous penalty in achieved utility. It is true that those attackers gain a lot of information on S by observing Y. However, they could have also obtained this information from the public side-information directly. Therefore, the approach taken in the Pufferfish framework and in this paper is that we only protect against attackers that have beliefs, i.e., distributions P, that are in line with publicly available side information.

A challenge in working with the Pufferfish framework is that it is often challenging to find good mechanisms. A general mechanism is proposed in [20], but it relies on enumerating over all distributions in $\mathcal{F}$, which is an uncountable set in our setting and cannot be used here. A constrained version of Pufferfish that facilitates analysis and a methodology for finding good mechanisms is proposed in [21]. Another interesting line of work is to model correlations between users in the non-local differential privacy setting [22]. Finally, ref. [23] proposed a modeling framework for capturing domain knowledge about the data. In contrast, in the current work, we impose constraints that are learned from data. Our setting does not fit any of the frameworks for which good mechanisms are known in the literature. One of the main contributions of this paper is to develop such mechanisms.

2.2. Other Privacy Frameworks

Disclosing X through a privacy mechanism that protects sensitive information S has been studied extensively. One line of work starts from differential privacy [24] and imposes the additional challenge that the aggregator cannot be trusted, leading to the concept of local differential privacy [1,2,5]. For this setting, several privacy mechanisms exist, including randomized response [19] and unary encoding [25]. Optimal LDP mechanisms under a variety of utility metrics, including mutual information, are found in [5]. In [1,2,5], all data are sensitive, i.e., $X=S$. The variation of LDP for the case of disclosing $X=(S,U)$, where only S is sensitive, was proposed in [3] and is the setting that we study in this paper. Another line of work connects this setting to the information bottleneck [26], leading to a privacy constraint in terms of mutual information [6,8,9,10]. In these works, it is shown that approaches to optimizing the information bottleneck also work for finding good privacy mechanisms.

Next to differential privacy and mutual information as privacy measures, a multitude of other privacy frameworks and leakage measures exist [27]. Some of these have been studied in the context of privacy mechanisms. In [7,11], privacy leakage is measured through the improved potential of statistical inference by an attacker after seeing the disclosed information. This measure is formulated through a general cost function, with mutual information resulting as a special case. Perfect privacy, which demands the output to be independent of the sensitive data, was studied in [28], and methods were given to find optimal mechanisms in this setting. An estimation-theoretic framework was studied in [29,30]. Our use of a Rényi divergence in the construction of $\mathcal{F}$ may suggest considering a generalization of our privacy definition. This could be achieved by considering, for instance, a Rényi divergence in the privacy constraint, as done in [31]. Along a different line, in [32], the maximal leakage measure with a clear operational interpretation is defined. In [33], this measure is generalized to a parametrized measure, enabling interpolating between maximal leakage and mutual information. A stronger, pointwise, version of the maximal leakage measure is proposed in [34]. These are interesting research directions but not pursued in this paper.

Our setting $X=(S,U)$ is a special case of a Markov chain $S-X-Y$, where only X is observed. This Markov chain is typically studied in the information bottleneck and privacy funnel settings [6,26]. We do not generalize to this setting, because we need observations of S for the estimate of $P_{U|S}$. Without direct observations of s, we can only make worst-case assumptions on $P_{U|S}$, leading to very poor utility. A different type of model, in which only part of the information in X is sensitive, is proposed in [12]. This is a block-structured model in which X is partitioned and information about the partition of an element is sensitive but its index in the partition is not. Our setting of $\mathcal{X}=\mathcal{S}\times \mathcal{U}$ does not fit this model. One can partition $\mathcal{X}$ according to $\mathcal{U}$, but our privacy constraints are different from [12]. We will elaborate on this in Section 6.

2.3. Robustness

The distribution $P_{S,U}^{*}$ is not available in practice. The approach taken in most works is to estimate $P_{S,U}^{*}$ from data and analyze sensitivity with respect to this estimate ${\widehat{P}}_{S,U}$. One of the contributions in [7] is to quantify the impact of mismatched priors, i.e., the impact of not knowing $P_{S,U}^{*}$ exactly. A bound on the resulting level of privacy is derived in terms of the total variational distance between the actual and the estimated ${\widehat{P}}_{S,U}$. The setting in [35] is similar to ours: A ball of probability distributions, centered around a point estimate, was defined that contains $P_{S,U}^{*}$ with high probability. It was then shown that a privacy mechanism that was designed based on the empirical distribution was valid for the entire set for a looser privacy constraint. The privacy slack was quantified and shown to approach zero as the size of the data set increased. An important difference with the current work was that we explicitly optimize the privacy mechanism over the uncertainty set. Another difference is that we base our ball on a Rényi divergence, whereas [35] used an ${\mathcal{l}}_1$ norm. The main technical tool used in [35] was large deviations theory, whereas we rely on convex analysis and robust optimization. We also mention [36,37]. In [36] it is assumed that nothing is known about $P_S^{*}$ and $P_{U|S}^{*}$. It is shown that good privacy mechanisms can be found through a connection to maximal correlation, see also [38]. In [37], sets of probability distributions are not derived from data but carefully modeled such that optimal mechanisms can be derived analytically.

Using robust optimization [16] to find a good mechanism that satisfies privacy constraints for all $P_{S,U}$ in uncertainty set $\mathcal{F}$ was proposed in [13,14]. In this work, we generalize and extend results from [14]. The idea of robust optimization is that constraints in an optimization problem contain uncertain parameters that are known to come from a (a priori defined) uncertainty set. The constraints must hold for possible values of the uncertain parameters. A key result is that, using Fenchel duality, the problem can be expressed in terms of the support function of the uncertainty set and the convex conjugate of the constraint [16,17]. The case where the uncertain parameters are probabilities is known as distributionally robust optimization. Using results from [39], it was shown in [40] how an uncertainty set can be constructed from data using an f-divergence, providing an approximate confidence set. Confidence sets for parameters that are not necessarily probabilities were constructed in [18] under a ${\chi }^2$-divergence. Convergence of robust optimization based on f-divergences was studied in [41] and for the case of a KL-divergence in [42]. In [43], it is shown how distributionally robust optimization problems over Wasserstein balls can be reformulated as convex problems. For the regular differential privacy setting, distributionally robust optimization was used in [44] to find optimal additive privacy mechanisms for a general perturbation cost function. In this paper, we show how robust optimization can be applied to the setting of partially sensitive information with local differential privacy.

2.4. Miscellaneous

Another line of work on privacy mechanisms builds on recent advances in generative adversarial networks [45]. In [46,47], a generative adversarial framework is used to provide privacy mechanisms that do not use explicit expressions for $P_X$. Even though this is not explicitly addressed in [46,47], it is expected that the generalization properties of networks will provide a form of robustness. Closely related approaches are used in the field of face recognition [48,49], with the aim of preventing biometric profiling [50]. The leakage measures that are used in [48,49], however, do not seem to have an operational interpretation.

Disclosing information in a privacy-preserving way is one of the main challenges in official statistics [51,52]. The setting considered in the current paper is closely connected to disclosing a table with microdata, where each record in the table is released independently of the other records. This approach to disclosing microdata was studied in [4] by considering expected error as the utility measure and mutual information as the privacy measure. The resulting optimization problem corresponds to the traditional rate-distortion problem.

3. Model and Preliminaries

In this section, we give an overview of the setting and objectives of this paper. The notation used in this section, as well as the rest of the paper, is summarized in

Table 1. Notation used in this paper. ‘Page’ denotes the page the notation is first defined.

Notation	Meaning	Page
$\mathcal{S}$	sensitive data space	6
$\mathcal{U}$	non-sensitive data space	6
$\mathcal{X}$	$\mathcal{S}\times \mathcal{U}$	6
$a_1,a_2,a,b$	$\left|\mathcal{S}\right|,\left|\mathcal{U}\right|,\left|\mathcal{X}\right|,\left|\mathcal{Y}\right|$	6
$X=(S,U)$	user data	6
ine $\mathcal{Q}$	privacy mechanism	6
Q	matrix of $\mathcal{Q}$	6
Y	$\mathcal{Q}\left(X\right)$	6
$\mathcal{Y}$	output space	6
${\mathcal{P}}_{\mathcal{X}}$	space of prob. dist. on $\mathcal{X}$	6
$P^{*}$	true distribution	6
$\widehat{P}$	estimated distribution	7
I	mutual information	8
$\mathcal{F}$	uncertainty set for P	6
$P_{\mathcal{U}|s}$	condition probability vector	9
${\mathcal{F}}_{\mathcal{U}|s}$	conditional projection of $\mathcal{F}$	9
$L_{u|s}\left(\mathcal{F}\right),{\mathrm{rad}}_s\left(\mathcal{F}\right)$	statistics of $\mathcal{F}$	9
$D_{\alpha }$	Rényi divergence	10
$\mathrm{PolyOpt}$	PolyOpt	13
${\mathrm{SRR}}^{\epsilon }$	Secret Randomized Response	17
${\mathrm{IR}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}$	Independent Reporting	18

.

The data space is $\mathcal{X}=\mathcal{S}\times \mathcal{U}$, where $\mathcal{S}$ and $\mathcal{U}$ are finite sets. We write $\left|\mathcal{S}\right|=:a_1$, $\left|\mathcal{U}\right|=:a_2$, and $\left|\mathcal{X}\right|=a_1{a}_2=:a$. Data items $X=(S,U)$ are drawn from a probability distribution $P^{*}$ in ${\mathcal{P}}_{\mathcal{X}}$, the space of probability distributions on $\mathcal{X}$; here, S represents sensitive data, while U represents non-sensitive data. The aggregator’s aim is to create a privacy mechanism $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$ such that $Y=\mathcal{Q}\left(X\right)$ contains as much information about X as possible, while not leaking too much information about S.

The mechanism $\mathcal{Q}$ is a probabilistic map, which we represent by a left stochastic matrix ${\left(Q_{y|x}\right)}_{y\in \mathcal{Y},x\in \mathcal{X}}$, and we write $\left|\mathcal{Y}\right|=b$. Often, we identify $\mathcal{Y}=\{1,\dots ,b\}$, and likewise for other sets.

The distribution $P^{*}$ is not known exactly. Instead, there is a set of possible distributions $\mathcal{F}\subset {\mathcal{P}}_{\mathcal{X}}$, where ${\mathcal{P}}_{\mathcal{X}}$ denotes the probability simplex over $\mathcal{X}$. We choose $\mathcal{F}$ in such a way that it is likely that $P^{*}\in \mathcal{F}$. The uncertainty set $\mathcal{F}$ captures our uncertainty about $P^{*}$, we guarantee privacy for all $P\in \mathcal{F}$. We denote this as robust local differential privacy (RLDP).

Definition 1 

(Robust Local Differential Privacy). Let $\epsilon \ge 0$ and $\mathcal{F}\subset {\mathcal{P}}_{\mathcal{X}}$. We say that $\mathcal{Q}$ satisfies $(\epsilon ,\mathcal{F})$-RLDP if for all $s,s^{\prime }\in \mathcal{S}$, all $y\in \mathcal{Y}$, and all $P\in \mathcal{F}$ we have

$${\mathbb{P}}_{X\sim P}(Y=y|S=s)\le {\mathrm{e}}^{\epsilon }{\mathbb{P}}_{X\sim P}(Y=y|S=s^{\prime }).$$

(4)

Note that we use the notation ${\mathbb{P}}_{X\sim P}(•)$ to emphasize that X is distributed according to P. If no confusion can arise, we often leave out the subscript $X\sim P$, to improve readability. Note that we can also write

$${\mathbb{P}}_{X\sim P}(Y=y|S=s)=\sum _{u\in \mathcal{U}}{Q}_{y|s,u}{\mathbb{P}}_{X\sim P}(U=u|S=s),$$

(5)

so Definition 1 depends on the conditional probabilities of U given $S=s$ and $S=s^{\prime }$. It does not, however, depend on the realization of U.

For clarity and use in future sections, we give the definition of regular LDP [1], which is used when the goal is to obfuscate all of X, rather than just S.

Definition 2 

(Local Differential Privacy). Let $\epsilon \ge 0$. We say that $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$ satisfies ε-LDP if for all $x,x^{\prime }\in \mathcal{X}$ and all $y\in \mathcal{Y}$ we have

$$\mathbb{P}(Y=y|X=x)\le {\mathrm{e}}^{\epsilon }\mathbb{P}(Y=y|X=x^{\prime }).$$

(6)

Now, for aggregator uncertainty about $P^{*}$, as captured by $\mathcal{F}$, we suppose there is a data base $\overrightarrow{x}=(x_1,\cdots ,x_n)$ accessible to the user, where each $x_{\mathrm{I}}=(s_{\mathrm{I}},u_{\mathrm{I}})$ is drawn independently from $P^{*}$. Based on this, the user produces an estimate $\widehat{P}$ of $P^{*}$. In the experiments, we consider a maximum likelihood estimator, i.e., ${\widehat{P}}_x=\left|\{i\le n:x_{\mathrm{I}}=x\}\right|$. We construct the uncertainty set $\mathcal{F}$ as a closed ball around $\widehat{P}$. In particular, let $D_{\alpha }$ be the Rényi divergence of order $\alpha$ on ${\mathcal{P}}_{\mathcal{X}}$, i.e., for $\alpha \in (0,\infty )$

$$D_{\alpha }(\widehat{P}\left|\right|P)=\left\{\begin{array}{cc}\frac{1}{\alpha -1}log\left(\sum _{x\in \mathcal{X}}\frac{{\widehat{P}}_x^{\alpha }}{P_x^{\alpha -1}}\right),\hfill & \mathrm{if}\alpha \ne 1,\hfill \\ \sum _{x\in \mathcal{X}}{\widehat{P}}_{x}log\frac{{\widehat{P}}_x}{P_x},\hfill & \mathrm{if}\alpha =1.\hfill \end{array}\right.$$

(7)

The case $\alpha =1$ follows, in fact, as a limit from the $\alpha \ne 1$ case. Similarly, the definition can be extended to $\alpha \in \{0,\infty \}$ by taking the corresponding limits, but in this paper we restrict our attention to $\alpha \in (0,\infty )$ to keep the presentation clear. Note that $D_1=D_{\mathrm{KL}}$, the Kullback–Leibler divergence, and $D_2=log{\chi }^2$, where the ${\chi }^2$-divergence is ${\chi }^2(P_1\left|\right|P_2)={\sum }_x{(P_{1,x}-P_{2,x})}^2{P}_{2,x}^{-1}$. In general, a Rényi divergence is a continuous increasing function of a power divergence (a.k.a. Hellinger divergence) [39,53,54], an example of an f-divergence. We omit $\alpha$ from the notation when it is clear from the context.

We define $\mathcal{F}$ by fixing a bound $B\in [0,\infty ]$ and letting

$$\mathcal{F}=\left\{P\in {\mathcal{P}}_{\mathcal{X}}:D_{\alpha }(\widehat{P}\left|\right|P)\le B\right\}.$$

(8)

Since a Rényi divergence is a continuous increasing function of an f-divergence, it follows from [39,40] that $\mathcal{F}$ is a confidence set for $P^{*}$. In particular, for the case of $\alpha =2$, which will be used in our numerical experiments in Section 8, for suitable B, we have

$$\mathcal{F}=\left\{P\in {\mathcal{P}}_{\mathcal{X}}:\sum _x\frac{{({\widehat{P}}_x-P_x)}^2}{P_x}\le \frac{F_{{\chi }^2,a-1}^{-1}(1-\beta )}{n}\right\},$$

(9)

with $\beta \in (0,1)$, where $F_{{\chi }^2,a-1}$ is the cumulative density function of the ${\chi }^2$-distribution with $a-1$ degrees of freedom, resulting in a set $\mathcal{F}$ with significance level $\beta$. This means that the probability of $P^{*}\in \mathcal{F}$ is at least $1-\beta$.

Hence, by designing $\mathcal{Q}$ based on $\mathcal{F}$, we are confident in satisfying (1) for all attackers that have beliefs that are based on the public side-information, as well as for attackers that have beliefs that are closer to $P^{*}$.

As a special case of the above, we will study the case that nothing is known about $P^{*}$. In this case, $B\to \infty$ and $\mathcal{F}={\mathcal{P}}_{\mathcal{X}}$. Regarding privacy, this is the ‘safest’ choice, as we do not make assumptions about $P^{*}$. Another special case is where $\mathcal{F}$ is a singleton, which reflects a situation where $B=0$ and $P^{*}$ is assumed to be known. This setting was studied in [3].

Given $\mathcal{F}$ and $\epsilon$, the goal is now to create a $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$ to be used on new/future data; our setting is depicted in Figure 1. The aim of this paper is to find a satisfactory answer to the following problem:

Problem 1. 

Given $\mathcal{F}$ and ε, find a $\mathcal{Q}$ satisfying $(\epsilon ,\mathcal{F})$-RLDP, while maximizing a given utility function.

Throughout this paper, we follow the original privacy funnel [6] and its LDP counterpart [3] in taking mutual information $\mathrm{I}(X;Y)$ as a utility measure. As is argued in [6], mutual information arises naturally when minimizing log loss distortion in the privacy funnel scenario. As a utility measure of $\mathcal{Q}$, we take ${\mathrm{I}}_{X\sim P}(X;Y)$ (abbreviated to ${\mathrm{I}}_P(X;Y)$), since the aim is to create Y that reflects X as faithfully as possible. This utility measure depends on the distribution P of X that we choose to evaluate. Ideally, one would like to use $P=P^{*}$, but in practice this is not possible, as $P^{*}$ is unknown. In the theoretical part of this paper, we circumvent this issue by proving our results for general P. In the experiments of Section 8, we take $P=\widehat{P}$ as the best available alternative to $P=P^{*}$. We investigate the effect of this choice by comparing ${\mathrm{I}}_{P^{*}}(X;Y)$ to ${\mathrm{I}}_{\widehat{P}}(X;Y)$.

Another option is to use the robust utility measure ${min}_{P\in \mathcal{F}}{\mathrm{I}}_P(X;Y)$ to ensure good utility for every ‘reasonable’ P, see [13]. We do not explicitly study this measure in this paper, but since our results hold for general P, they can also be applied to robust utility.

Example 1. 

We set up an example to illustrate the concepts of this paper. Take $\mathcal{S}=\{s_1,s_2\}$ and $\mathcal{U}=\{u_1,u_2\}$, and suppose

$$P^{*}=\left(\begin{array}{c}{P}_{s_1,u_1}^{*}\\ P_{s_1,u_2}^{*}\\ P_{s_2,u_1}^{*}\\ P_{s_2,u_2}^{*}\end{array}\right)=\left(\begin{array}{c}0.1\\ 0.1\\ 0.2\\ 0.6\end{array}\right).$$

(10)

Moreover, suppose we have a publicly known database of $n=100$ entries, from which we estimate

$$\widehat{P}=\left(\begin{array}{c}{\widehat{P}}_{s_1,u_1}\\ {\widehat{P}}_{s_1,u_2}\\ {\widehat{P}}_{s_2,u_1}\\ {\widehat{P}}_{s_2,u_2}\end{array}\right)=\left(\begin{array}{c}0.07\\ 0.10\\ 0.26\\ 0.57\end{array}\right).$$

(11)

To obtain a $95\%$-confidence set for $\mathcal{F}$ according to a ${\chi }^2$-distribution, we take $\alpha =2$ and $B=log\left(1+\frac{F_{{\chi }^2,3}^{-1}\left(0.05\right)}{100}\right)=0.0752$. In this way, we obtain

$$\begin{array}{}\hfill \mathrm{(12)}& \hfill \mathcal{F}& =\left\{P\in {\mathcal{P}}_{\mathcal{X}}:D_{\alpha }(\widehat{P}\left|\right|P)\le B\right\}\hfill \hfill \mathrm{(13)}& & =\left\{P\in {\mathcal{P}}_{\mathcal{X}}:log\left({\displaystyle \sum _x}\frac{{\widehat{P}}_x^2}{P_x}\right)\le log\left(1+\frac{F_{{\chi }^2,3}^{-1}\left(0.05\right)}{100}\right)\right\}\hfill \hfill \mathrm{(14)}& & =\left\{P\in {\mathcal{P}}_{\mathcal{X}}:{\displaystyle \sum _x}\frac{{({\widehat{P}}_x-P_x)}^2}{P_x}\le \frac{F_{{\chi }^2,3}^{-1}\left(0.95\right)}{100}\right\},\hfill \end{array}$$

which is the desired confidence set (note that the ${\chi }^2$-distribution has $\left|\mathcal{X}\right|-1=3$ degrees of freedom). In this case, we have $D_2(\widehat{P}\left|\right|P^{*})=0.0281<B$, so $P^{*}\in \mathcal{F}$.

4. Conditional Projection of $\mathcal{F}$

In Section 5 and Section 7 below, we will introduce privacy mechanisms that provide $(\epsilon ,\mathcal{F})$-RLDP. These mechanisms depend on the conditional projections of $\mathcal{F}$ on ${\mathcal{P}}_{\mathcal{U}}$ given $S=s$, denoted as ${\mathcal{F}}_{\mathcal{U}|s}$. In this section, we analyze the structure and statistics of these sets. To do so, we introduce, for $s\in \mathcal{S}$, $u\in \mathcal{U}$ and $P\in {\mathcal{P}}_{\mathcal{X}}$.

$$\begin{array}{}\hfill \mathrm{(15)}& \hfill P_s& ={\displaystyle \sum _{u\in \mathcal{U}}}{P}_{u,s},\hfill \hfill \mathrm{(16)}& \hfill P_{u|s}& =\frac{P_{u,s}}{P_s},\hfill \hfill \mathrm{(17)}& \hfill P_{\mathcal{U}|s}& ={\left(P_{u|s}\right)}_{u\in \mathcal{U}}\in {\mathcal{P}}_{\mathcal{U}},\hfill \hfill \mathrm{(18)}& \hfill {\mathcal{F}}_{\mathcal{U}|s}& =\{P_{\mathcal{U}|s}:P\in \mathcal{F}\}\subset {\mathcal{P}}_{\mathcal{U}},\hfill \end{array}$$

We are interested in the following statistics:

$$\begin{array}{}\hfill \mathrm{(19)}& \hfill L_{u|s}\left(\mathcal{F}\right)& ={\displaystyle \underset{R\in {\mathcal{F}}_{\mathcal{U}|s}}{min}}{R}_u\mathrm{for}\mathrm{a}\mathrm{given}u\in \mathcal{U},\hfill \hfill \mathrm{(20)}& \hfill {\mathrm{rad}}_s\left(\mathcal{F}\right)& ={\displaystyle \underset{R\in {\mathcal{F}}_{\mathcal{U}|s}}{max}}\left|\right|R-{\widehat{P}}_{\mathcal{U}|s}{\left|\right|}_1.\hfill \end{array}$$

In (19), $R_u$ is the u-coefficient of $R\in {\mathcal{P}}_{\mathcal{U}}$. It turns out that these statistics give us the information required to construct $(\epsilon ,\mathcal{F})$-protocols efficiently: In Section 5, we use $L_{u|s}\left(\mathcal{F}\right)$ to approximate ${\mathcal{F}}_{\mathcal{U}|s}$ by a polytope, to make computation easier, while in Section 7, we use ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ as a measure for the size of ${\mathcal{F}}_{\mathcal{U}|s}$. While these statistics (or bounds for them) are relatively easy to find for $\mathcal{F}$ itself, the hard part lies in the fact that we have to give bounds for the projection ${\mathcal{F}}_{\mathcal{U}|s}$. The extent to which these bounds can be found explicitly heavily depends on the divergence measure that is used to construct $\mathcal{F}$. In this section, we show how these bounds can be obtained for our case where we construct $\mathcal{F}$ using a Rényi divergence. The reason for this, as we will see below, is that we can give an explicit description of ${\mathcal{F}}_{\mathcal{U}|s}$.

4.1. Structure of ${\mathcal{F}}_{\mathcal{U}|s}$

Recall that, for a given $\alpha \in (0,\infty )$, the Rényi divergence $D_{\alpha }:{\mathcal{P}}_{\mathcal{X}}\to [0,\infty )$ is defined by

$$D_{\alpha }(\widehat{P}\left|\right|P)=\left\{\begin{array}{cc}\frac{1}{\alpha -1}log\left(\sum _{x\in \mathcal{X}}\frac{{\widehat{P}}_x^{\alpha }}{P_x^{\alpha -1}}\right),\hfill & \mathrm{if}\alpha \ne 1,\hfill \\ \sum _{x\in \mathcal{X}}{\widehat{P}}_{x}log\frac{{\widehat{P}}_x}{P_x},\hfill & \mathrm{if}\alpha =1.\hfill \end{array}\right.$$

(21)

The following theorem states that the conditional projections of balls defined by Rényi divergence are themselves Rényi divergence balls:

Theorem 1. 

Let $s\in \mathcal{S}$ be such that ${\widehat{P}}_s>0$. Let $\mathcal{F}$ be defined by Rényi divergence, i.e.,

$$\mathcal{F}=\left\{P\in {\mathcal{P}}_{\mathcal{X}}:D_{\alpha }(\widehat{P}\left|\right|P)\le B\right\}$$

(22)

for a given $\alpha \in (0,\infty )$ and $B\in {\mathbb{R}}_{\ge 0}$. Define the constant $B_s$ by

$$B_s=\left\{\begin{array}{cc}\frac{\alpha }{\alpha -1}log\left(\frac{{\mathrm{e}}^{(\alpha -1)B/\alpha }-(1-{\widehat{P}}_s)}{{\widehat{P}}_s}\right),\hfill & \mathrm{if}\alpha \ne 1,\hfill \\ \frac{B}{{\widehat{P}}_s},\hfill & \mathrm{if}\alpha =1.\hfill \end{array}\right.$$

(23)

Then,

$${\mathcal{F}}_{\mathcal{U}|s}=\left\{R\in {\mathcal{P}}_{\mathcal{U}}:D_{\alpha }({\widehat{P}}_{\mathcal{U}|s}\left|\right|R)\le B_s\right\}.$$

(24)

This theorem gives us a direct description of the ${\mathcal{F}}_{\mathcal{U}|s}$, which is useful because the $L_{u|s}\left(\mathcal{F}\right)$ of (19) and ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ of (20) are defined in terms of these projection sets. A similar bound could also be found for the limit cases $\alpha =0,\infty$, but this is not pursued in this paper, because it does not provide additional insights.

A key property of the Rényi divergence that allows us to prove Theorem 1 is that we can write

$$\frac{{\widehat{P}}_x^{\alpha }}{P_x^{\alpha -1}}=\frac{{\widehat{P}}_{u|s}^{\alpha }}{P_{u|s}^{\alpha -1}}·\frac{{\widehat{P}}_s^{\alpha }}{P_s^{\alpha -1}}.$$

(25)

This allows us to express the divergence $D_{\alpha }({\widehat{P}}_{\mathcal{U}|s}\left|\right|P_{\mathcal{U}|s})$ in terms of $D_{\alpha }(\widehat{P}\left|\right|P)$. For other divergences, which may depend on $\widehat{P}$ and P in a more complicated way, this is typically not possible. Therefore, we cannot generalize our results to uncertainty sets constructed from, for instance, arbitrary f-divergences.

In light of this theorem and the fact that in the following sections we care more about the statistics of ${\mathcal{F}}_{\mathcal{U}|s}$ than about those of $\mathcal{F}$ itself, one might be inclined to think that it is more straightforward to estimate the ${\widehat{P}}_{\mathcal{U}|s}$ from the data and defining uncertainty sets ${\mathcal{F}}_{\mathcal{U}|s}$ around them directly, without going through the intermediate stage $\mathcal{F}$. However, projecting these sets back to ${\mathcal{P}}_{\mathcal{X}}$ results in a larger set. In other words, there are distributions P such that each $P_{\mathcal{U}|s}$ is an element of ${\mathcal{F}}_{\mathcal{U}|s}$, while $P\notin \mathcal{F}$. That is, we have $\mathcal{F}⊊{\mathcal{F}}^{\prime }:=\{P\in {\mathcal{P}}_{\mathcal{X}}:\forall s{P}_{\mathcal{U}|s}\in {\mathcal{F}}_{\mathcal{U}|s}\}$. The reason for this is that, in the proof of Theorem 1, it becomes clear that the $P\in \mathcal{F}$ that project to the boundary points of ${\mathcal{F}}_{\mathcal{U}|s}$ satisfy $P_{\mathcal{U}|s^{\prime }}={\widehat{P}}_{\mathcal{U}|s^{\prime }}$ for $s^{\prime }\ne s$. In other words, elements of $\mathcal{F}$ can be extremal in, at most, one ${\mathcal{F}}_{\mathcal{U}|s}$. By contrast, ${\mathcal{F}}^{\prime }$ also includes P that are extremal in multiple ${\mathcal{F}}_{\mathcal{U}|s}$. We conclude that constructing the ${\mathcal{F}}_{\mathcal{U}|s}$ directly results in a larger ${\mathcal{F}}^{\prime }$, which results in a lower utility. We will give an example of this phenomenon in Example 2.

4.2. Statistics of ${\mathcal{F}}_{\mathcal{U}|s}$

In this section, we analyze statistics of ${\mathcal{F}}_{\mathcal{U}|s}$. More concretely, to find $L_{u|s}\left(\mathcal{F}\right)$ and ${\mathrm{rad}}_s\left(\mathcal{F}\right)$, fix s, $\alpha$ and B and define for $\rho \in [0,1]$ and $\xi \in {\mathbb{R}}_{\ge 0}$ such that $\xi (1-\rho )\le 1$,

$$\begin{array}{}\hfill \mathrm{(26)}& \hfill {\phi }_{B_s}(\rho ,\xi )& =\left\{\begin{array}{cc}\frac{1}{\alpha -1}log\left(\rho {\xi }^{1-\alpha }+(1-\rho ){\left(\frac{1-\rho \xi }{1-\rho }\right)}^{1-\alpha }\right)-B_s,\hfill & \mathrm{if}\alpha \ne 1\mathrm{and}\rho \ne 1,\hfill \\ \rho log\frac{1}{\xi }+(1-\rho )log\frac{1-\rho }{1-\rho \xi }-B_s,\hfill & \mathrm{if}\alpha =1\mathrm{and}\rho \ne 1,\hfill \\ log\frac{1}{\xi }-B_s,\hfill & \mathrm{if}\rho =1,\hfill \end{array}\right.\hfill \hfill \mathrm{(27)}& \hfill {\xi }_{-}\left(\rho \right)& =inf\left\{\xi \in (0,1]:{\phi }_{B_s}(\rho ,\xi )\le 0\right\},\hfill \hfill \mathrm{(28)}& \hfill {\xi }_{+}\left(\rho \right)& =sup\left\{\xi \in [1,{(1-\rho )}^{-1}):{\phi }_{B_s}(\rho ,\xi )\le 0\right\}.\hfill \end{array}$$

Note that the case $\rho =1$ can be obtained via taking the limit. The expressions for ${\xi }_{-}$ and ${\xi }_{+}$ are a bit complicated, but note that, given $\rho <1$, the function ${\phi }_{B_s}(\rho ,\xi )$ is convex in $\xi$. Thus, ${\phi }_{B_s}(\rho ,\xi )=0$ has at most two solutions. Furthermore, ${\phi }_{B_s}(\rho ,1)=-B_s$ and ${\phi }_{B_s}(\rho ,\xi )\to \infty$ as $\xi$ approaches 0 or $\frac{1}{1-\rho }$, so for $\rho <1$ the values ${\xi }_{-}\left(\rho \right)$ and ${\xi }_{+}\left(\rho \right)$ are the two solutions to ${\phi }_{B_s}(\rho ,\xi )=0$.

The following proposition expresses our desired statistics in terms of ${\xi }_{-}$ and ${\xi }_{+}$.

Proposition 1. 

Let $u\in \mathcal{U}$. Then,

$$\begin{array}{}\hfill \mathrm{(29)}& \hfill L_{u|s}\left(\mathcal{F}\right)& ={\widehat{P}}_{u|s}{\xi }_{-}\left({\widehat{P}}_{u|s}\right),\hfill \hfill \mathrm{(30)}& \hfill {\mathrm{rad}}_s\left(\mathcal{F}\right)& =2{\displaystyle \underset{\begin{array}{c}{\mathcal{U}}_1\subset \mathcal{U}:\\ {\mathcal{U}}_1\ne ⌀\end{array}}{max}}{\widehat{P}}_{{\mathcal{U}}_1|s}({\xi }_{+}\left({\widehat{P}}_{{\mathcal{U}}_1|s}\right)-1).\hfill \end{array}$$

As discussed above, ${\xi }_{\pm }\left(\rho \right)$ can be found quickly numerically; however, the calculation of ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ still involves taking the maximum over an exponentially large set.

4.3. Special Case $\alpha =2$

In this section, we show that when $\alpha =2$, we can find explicit expressions for ${\xi }_{\pm }$ and consequently $L_{u|s}$ and ${\mathrm{rad}}_s$. As discussed in (9), for this $\alpha$, the set $\mathcal{F}$ is a confidence set for a ${\chi }^2$-test. To find ${\xi }_{-}\left(\varrho \right),{\xi }_{+}\left(\varrho \right)$, we need to solve ${\phi }_{B_s}(\rho ,\xi )=0$. For $\alpha =2$, we can write this as a quadratic equation in $\xi$, and solving it leads to the following expression:

Lemma 1. 

Suppose $\alpha =2$. Then,

$$\begin{array}{cc}\hfill {\xi }_{-}\left(\rho \right)& =\frac{{\mathrm{e}}^{B_s}+2\rho -1-\sqrt{({\mathrm{e}}^{B_s}-1)({\mathrm{e}}^{B_s}-{(2\rho -1)}^2)}}{2{\mathrm{e}}^{B_s}\rho },\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill {\xi }_{+}\left(\rho \right)& =\frac{{\mathrm{e}}^{B_s}+2\rho -1+\sqrt{({\mathrm{e}}^{B_s}-1)({\mathrm{e}}^{B_s}-{(2\rho -1)}^2)}}{2{\mathrm{e}}^{B_s}\rho }.\hfill \end{array}$$

(32)

Now, we can determine $L_{u|s}\left(\mathcal{F}\right)$ and ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ using Lemma 1 and Proposition 1. For $L_{u|s}\left(\mathcal{F}\right)$, we immediately obtain an expression; for ${\mathrm{rad}}_s\left(\mathcal{F}\right)$, a careful analysis of ${\xi }_{+}$ shows that the optimal ${\mathcal{U}}_1$ of (30) can be found. For large enough $B_s$, the optimum is at ${\mathcal{U}}_1=\left\{u_{min}\right\}$, where $u_{min}$ is the u that minimizes ${\widehat{P}}_{u|s}$. Thus, we obtain a concrete expression for ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ without the need for optimization. For smaller $B_s$, we do not find an exact expression, but we can still derive a lower bound. The results are summarized in the following proposition.

Proposition 2. 

Let $\alpha =2$. Then, the following hold:

1. 

One has

$$L_{u|s}\left(\mathcal{F}\right)=\frac{{\mathrm{e}}^{B_s}+2{\widehat{P}}_{u|s}-1-\sqrt{({\mathrm{e}}^{B_s}-1)({\mathrm{e}}^{B_s}-{(2{\widehat{P}}_{u|s}-1)}^2)}}{2{\mathrm{e}}^{B_s}}.$$

(33)

2. 

Let $u_{min}=arg{min}_{u\in \mathcal{U}}{\widehat{P}}_{u|s}$. If $B_s\ge log(1+{(1-{\widehat{P}}_{u_{min}|s})}^2)$, then

$${\mathrm{rad}}_s\left(\mathcal{F}\right)=\frac{-{\mathrm{e}}^{B_s}+2{\widehat{P}}_{u_{min}|s}-1+\sqrt{({\mathrm{e}}^{B_s}-1)({\mathrm{e}}^{B_s}-{(2{\widehat{P}}_{u_{min}|s}-1)}^2)}}{{\mathrm{e}}^{B_s}}.$$

(34)

3. 

If $B_s<log(1+{(1-{\widehat{P}}_{u_{min}|s})}^2)$, one has ${\mathrm{rad}}_s\left(\mathcal{F}\right)\le \sqrt{{\mathrm{e}}^{B_s}-1}$.

We note that $\alpha =2$ is not the only value of $\alpha$ for which one can bound $L_{u|s}$ and ${\mathrm{rad}}_s$. For instance, for $\alpha \le 1$, one can use Pinsker’s inequality [55,56] and its generalizations [57] to bound ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ in terms of $\left|\right|{\widehat{P}}_{\mathcal{U}|s}-P_{\mathcal{U}|s}{\left|\right|}_1$, which in turn can be used to bound $L_{u|s}\left(\mathcal{F}\right)$. However, unlike $\alpha =2$, these do not result in exact bounds.

Example 2. 

We continue Example 1. We have

$$\begin{array}{cccccc}\hfill {\widehat{P}}_{s_1}& =0.17,\hfill & \hfill {\widehat{P}}_{u_1|s_1}& =0.4118,\hfill & \hfill {\widehat{P}}_{u_2|s_1}& =0.5882,\hfill \\ \hfill {\widehat{P}}_{s_2}& =0.83,\hfill & \hfill {\widehat{P}}_{u_1|s_2}& =0.3133,\hfill & \hfill {\widehat{P}}_{u_2|s_2}& =0.6867.\hfill \end{array}$$

Inserting our values of B and ${\widehat{P}}_s$ into Theorem 2, we find $B_{s_1}=0.3782$, $B_{s_2}=0.0900$. In other words,

$$\begin{array}{cc}\hfill {\mathcal{P}}_{\mathcal{U}|s_1}& =\left\{R=\left(\genfrac{}{}{0pt}{}{R_{u_1}}{R_{u_2}}\right)\in {\mathcal{P}}_{\mathcal{U}}:D_2\left(\left(\genfrac{}{}{0pt}{}{0.4118}{0.5882}\right)\left|\right|\left(\genfrac{}{}{0pt}{}{R_{u_1}}{R_{u_2}}\right)\right)\le 0.3782\right\},\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill {\mathcal{P}}_{\mathcal{U}|s_2}& =\left\{R=\left(\genfrac{}{}{0pt}{}{R_{u_1}}{R_{u_2}}\right)\in {\mathcal{P}}_{\mathcal{U}}:D_2\left(\left(\genfrac{}{}{0pt}{}{0.3133}{0.6867}\right)\left|\right|\left(\genfrac{}{}{0pt}{}{R_{u_1}}{R_{u_2}}\right)\right)\le 0.0900\right\}.\hfill \end{array}$$

(36)

To determine the lower bounds on each $R_{u_i}$, we use Proposition 2 to obtain

$$\begin{array}{cccc}\hfill L_{u_1|s_1}\left(\mathcal{F}\right)& =0.1620,\hfill & \hfill L_{u_2|s_1}\left(\mathcal{F}\right)& =0.2829,\hfill \\ \hfill L_{u_1|s_2}\left(\mathcal{F}\right)& =0.1923,\hfill & \hfill L_{u_2|s_2}\left(\mathcal{F}\right)& =0.5337.\hfill \end{array}$$

In principle, we can also use Proposition 2 to determine the ${\mathrm{rad}}_s\left(\mathcal{F}\right)$. However, in this case, there is a more straightforward approach. Since $\left|\mathcal{U}\right|=2$, every element of ${\mathcal{F}}_{\mathcal{U}|s}$ is a vector of length two whose coefficients sum to 1; thus $P_{\mathcal{U}|s}$ is determined by $P_{u_1|s}$. Since $L_{u_1|s}\left(\mathcal{F}\right)\le P_{u_1|s}\le 1-L_{u_2|s}\left(\mathcal{F}\right)$, it follows that

$$\begin{array}{cc}\hfill {\mathcal{F}}_{\mathcal{U}|s_1}& \cong [L_{u_1|s_1}\left(\mathcal{F}\right),1-L_{u_2|s_1}\left(\mathcal{F}\right)]=[0.1620,0.7171],\hfill \\ \hfill {\mathcal{F}}_{\mathcal{U}|s_2}& \cong [L_{u_1|s_2}\left(\mathcal{F}\right),1-L_{u_2|s_2}\left(\mathcal{F}\right)]=[0.1923,0.4663].\hfill \end{array}$$

Under this identification, ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ is only twice the maximal distance from ${\widehat{P}}_{u_1|s}$ to the endpoint of this interval (the factor two comes from the fact that $\left|\right|P_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s}{\left|\right|}_1=|P_{u_1|s}-{\widehat{P}}_{u_1|s}|+|P_{u_2|s}-{\widehat{P}}_{u_2|s}|=2|P_{u_1|s}-{\widehat{P}}_{u_1|s}|$). Hence,

$$\begin{array}{cc}\hfill {\mathrm{rad}}_{s_1}\left(\mathcal{F}\right)& =2max\{0.4118-0.1620,0.7171-0.4118\}=0.6107,\hfill \\ \hfill {\mathrm{rad}}_{s_1}\left(\mathcal{F}\right)& =2max\{0.3133-0.1923,0.4663-0.3133\}=0.3061.\hfill \end{array}$$

We can also construct the set ${\mathcal{F}}^{\prime }=\{P\in {\mathcal{P}}_{\mathcal{X}}:\forall s{P}_{\mathcal{U}|s}\in {\mathcal{F}}_{\mathcal{U}|s}\}$ of Section 4.1. We can write this as

$${\mathcal{F}}^{\prime }=\left\{\left(\begin{array}{c}{P}_{s_1,u_1}\\ P_{s_1,u_2}\\ P_{s_2,u_1}\\ P_{s_2,u_2}\end{array}\right)\in {\mathcal{P}}_{\mathcal{X}}:\begin{array}{c}0.1620\le P_{u_1|s_1}\le 0.7171,\\ 0.1923\le P_{u_1|s_2}\le 0.4663\end{array}\right\}.$$

(37)

The inequality $0.1620\le P_{u_1|s_1}$ can be written as $0.1620\le \frac{P_{s_1,u_1}}{P_{s_1,u_1}+P_{s_1,u_2}}$, or $0.1620P_{s_1,u_2}\le 0.83830P_{s_1,u_1}$; in other words, this becomes a linear constraint. We can do the same for the other constraints and these, together with inequality constraints of the form $P_{s,u}\ge 0$ and the equality constraint ${\sum }_{s,u}{P}_{s,u}=1$, define the polytope ${\mathcal{F}}^{\prime }\subset {\mathbb{R}}^4$. One can calculate that this polytope is a simplex, spanned by the vertices

$$\left(\begin{array}{c}0.7171\\ 0.2829\\ 0\\ 0\end{array}\right),\left(\begin{array}{c}0.1620\\ 0.8380\\ 0\\ 0\end{array}\right),\left(\begin{array}{c}0\\ 0\\ 0.4663\\ 0.5337\end{array}\right),\left(\begin{array}{c}0\\ 0\\ 0.1923\\ 0.8077\end{array}\right).$$

(38)

The resulting ${\mathcal{F}}^{\prime }$ is considerably larger than $\mathcal{F}$: one way to see this is that, for any of these vertices P, one has $D_2(\widehat{P}\left|\right|P)=\infty$. This example shows the importance of working with the set $\mathcal{F}$, rather than with just its projections ${\mathcal{F}}_{\mathcal{U}|s}$.

5. Polyhedral Approximation: PolyOpt

In this section, we introduce PolyOpt, a family of mechanisms $\mathcal{Q}$ with good utility obtained by enclosing $\mathcal{F}$ by a polyhedron, and then using robust optimization for polyhedra [16] to describe the space of possible $\mathcal{Q}$ as a polyhedron; we then maximize the mutual information over this polyhedron. This approach is related to the polyhedral approach of [3], which finds the optimum for this problem in a non-robust setting.

For a mechanism $\mathcal{Q}$ and $y\in \mathcal{Y}$, we define $Q_y={\left(Q_{y|x}\right)}_{x\in \mathcal{X}}\in {\mathbb{R}}^{\mathcal{X}}$ to be the y-th row of the stochastic matrix Q corresponding to $\mathcal{Q}$, but transposed (i.e., viewed as a column vector). Likewise, we define the column vector $Q_{y|s}={\left(Q_{y|s,u}\right)}_{u\in \mathcal{U}}\in {\mathbb{R}}^{\mathcal{U}}$. In this notation, the condition for $(\epsilon ,\mathcal{F})$-RLDP can be formulated as

$$\forall y\in \mathcal{Y}\forall s_1,s_2\in \mathcal{S}:\underset{P\in \mathcal{F}}{max}\left[P_{\mathcal{U}|s_1}^{\mathrm{T}}{Q}_{y|s_1}-{\mathrm{e}}^{\epsilon }{P}_{\mathcal{U}|s_2}^{\mathrm{T}}{Q}_{y|s_2}\right]\le 0.$$

(39)

Equation (39) boils down to a set of linear constraints in $Q_y$. What makes these difficult to satisfy is that every value $P\in \mathcal{F}$ provides a linear constraint, and each $Q_y$ has to satisfy all infinitely many of these. In this section, we address this difficulty by making the set $\mathcal{F}$ slightly larger, so that robust optimization [16] becomes a convenient tool for optimizing over the allowed $\mathcal{Q}$. More precisely, for every $s\in \mathcal{S}$, let ${\mathcal{D}}_s\subset {\mathcal{P}}_{\mathcal{U}}$ be such that ${\mathcal{F}}_{\mathcal{U}|s}\subset {\mathcal{D}}_s$. Then, certainly

$$\underset{P\in \mathcal{F}}{max}\left[P_{\mathcal{U}|s_1}^{\mathrm{T}}{Q}_{y|s_1}-{\mathrm{e}}^{\epsilon }{P}_{\mathcal{U}|s_2}^{\mathrm{T}}{Q}_{y|s_2}\right]\le \underset{\begin{array}{c}{R}_1\in {\mathcal{D}}_{s_1},\\ R_2\in {\mathcal{D}}_{s_2}\end{array}}{max}\left[R_1^{\mathrm{T}}{Q}_{y|s_1}-{\mathrm{e}}^{\epsilon }{R}_2^{\mathrm{T}}{Q}_{y|s_2}\right].$$

(40)

Thus, we can conclude that $\mathcal{Q}$ is $(\epsilon ,\mathcal{F})$-RLDP whenever

$$\forall y\in \mathcal{Y}\forall s_1,s_2\in \mathcal{S}:\underset{\begin{array}{c}{R}_1\in {\mathcal{D}}_{s_1},\\ R_2\in {\mathcal{D}}_{s_2}\end{array}}{max}\left[R_1^{\mathrm{T}}{Q}_{y|s_1}-{\mathrm{e}}^{\epsilon }{R}_2^{\mathrm{T}}{Q}_{y|s_2}\right]\le 0.$$

(41)

The trick is now to choose the ${\mathcal{D}}_s$ in such a way that the set of $\mathcal{Q}$ satisfying (41) has a closed-form description. To this end, we let each ${\mathcal{D}}_s$ be a polyhedron; that way, we can use robust optimization for polyhedra [16] to give such a description.

There are multiple ways to create a polyhedron ${\mathcal{D}}_s$ that envelops ${\mathcal{F}}_{\mathcal{U}|s}$. Writing $L_{u|s}=L_{u|s}\left(\mathcal{F}\right)$ for convenience, we take

$${\mathcal{D}}_s=\{R\in {\mathcal{P}}_{\mathcal{U}}:\forall u{R}_u\ge L_{u|s}\}.$$

(42)

Since ${\mathcal{D}}_s$ is described by linear equations, it is a polyhedron, and certainly ${\mathcal{F}}_{\mathcal{U}|s}\subset {\mathcal{D}}_{\mathcal{U}|s}$ for all s. Robust optimization for polytopes [16] then allows us to describe the set of mechanisms satisfying (41). To formulate this, we first need the following definition:

Definition 3. 

Let $\epsilon >0$. Then, define ${\Gamma }_{\epsilon }$ to be the convex cone consisting of all $v\in {\mathbb{R}}_{\ge 0}^{\mathcal{X}}$ that satisfy, for all $s_1,s_2\in \mathcal{S}$ and all $u_1,u_2\in \mathcal{U}$:

$$v_{s_1,u_1}-{\mathrm{e}}^{\epsilon }{v}_{s_2,u_2}+\sum _u{L}_{u|s_1}\left(v_{s_1,u}-v_{s_1,u_1}\right)-{\mathrm{e}}^{\epsilon }\sum _u{L}_{u|s_2}\left(v_{s_2,u}-v_{s_2,u_2}\right)\le 0.$$

(43)

Note that, for every choice of $s_1,s_2,u_1,u_2$, (3) is a linear inequality in T and thus defines a half-space in ${\mathbb{R}}^{\mathcal{X}}$. The intersection of these half-spaces, intersected with ${\mathbb{R}}_{\ge 0}^{\mathcal{X}}$, defines the convex cone ${\Gamma }_{\epsilon }$. This definition allows us to formulate the following result:

Theorem 2. 

Let $\mathcal{Q}$ be a privacy mechanism, and for $y\in \mathcal{Y}$, let $Q_y$ be the y-th row of the associated matrix $Q={\left(Q_{y|x}\right)}_{y\in \mathcal{Y},x\in \mathcal{X}}$. Suppose that for all y we have $Q_y\in {\Gamma }_L$. Then, $\mathcal{Q}$ satisfies $(\epsilon ,\mathcal{F})$-RLDP.

The upshot of this theorem is that we have translated the infinitely many constraints of (39) and (41) into the finitely many linear constraints of (3). This makes optimizing utility considerably easier. We perform this optimization by translating it into a linear programming problem. The key inspiration for this optimization is Theorem 4 of [5], where optimal LDP mechanisms are found by translating the problem of optimizing mutual information into linear programming; we use an analogous approach adapted to RLDP. This approach can be sketched as follows: Let $\widehat{\Gamma }=\{v\in {\Gamma }_{\epsilon }:{\sum }_x{v}_x=1\}$, i.e., the intersection of ${\Gamma }_{\epsilon }$ with the hyperplane corresponding to ${\sum }_x{v}_x=1$. This is a polyhedron, and every $\mathcal{Q}$ satisfying the conditions of Theorem 2 has $Q_y={\theta }_y{v}_y$, for some ${\theta }_y\in {\mathbb{R}}_{\ge 0}$ and $v_y\in \widehat{\Gamma }$. The authors of [5] made a number of key observations that also apply to our situation. The first is that, in this case, we can write

$${\mathrm{I}}_{\widehat{P}}(X;Y)=\sum _y{\theta }_y\mu \left(v_y\right),$$

(44)

where

$$\mu \left(v\right)=\sum _{x\in \mathcal{X}}{v}_x{\widehat{P}}_{x}log\frac{v_x}{{\sum }_{x^{\prime }}{v}_{x^{\prime }}{\widehat{P}}_{x^{\prime }}}.$$

(45)

The second observation is that, in order to maximize (44), one can prove from the convexity of $\mu$ that it is optimal to have each $v_y$ be a vertex of $\widehat{\Gamma }$. Thus, once we know the set of vertices $\mathcal{V}$ of $\widehat{\Gamma }$, we find the optimal $\mathcal{Q}$ by assigning a weight ${\theta }_v$ to each $v\in \mathcal{V}$, in such a way that the resulting $Q_y$ form a probabilistic matrix and such that (44) is maximized. Since (44) is linear in $\theta$, this is a linear programming problem. This discussion is summarized in the following theorem:

Theorem 3. 

Let $\widehat{\Gamma }$ be a polyhedron given by $\{v\in {\Gamma }_{L,\epsilon }:{\sum }_x{v}_x=1\}$. Let $\mathcal{V}$ be the set of vertices of $\widehat{\Gamma }$. Define μ as in (45). Let $1_{\mathcal{X}}\in {\mathbb{R}}^{\mathcal{X}}$ be the constant vector of ones. Let $\widehat{\theta }\in {\mathbb{R}}_{\ge 0}^{\mathcal{V}}$ be the solution to the optimization problem

$$\begin{array}{cc}\hfill {\mathrm{maximise}}_{\theta }& \sum _{v\in \mathcal{V}}{\theta }_v\mu \left(v\right)\hfill \\ \hfill \mathrm{satisfying}& \theta \in {\mathbb{R}}_{\ge 0}^{\mathcal{V}},\hfill \\ \hfill & \sum _{v\in \mathcal{V}}{\theta }_{v}v=1_{\mathcal{X}}.\hfill \end{array}$$

(46)

Let the privacy mechanism $\mathcal{Q}$ be given by $\mathcal{Y}=\{v\in \mathcal{V}:{\widehat{\theta }}_v>0\}$ and $Q_{v|x}={\widehat{\theta }}_v{v}_x$. Then, the mechanism $\mathcal{Q}$ maximizes ${\mathrm{I}}_{\widehat{P}}(X;Y)$ among all mechanisms satisfying the condition of Theorem 2. One has $\left|\mathcal{Y}\right|\le a$.

Together, Theorems 2 and 3 show that if we can solve a vertex enumeration problem, we can find a mechanism $\mathcal{Q}$ that maximizes ${\mathrm{I}}_{\widehat{P}}(X;Y)$ among a subset of all $(\epsilon ,\mathcal{F})$-RLDP mechanisms; furthermore, we ensure that the output space $\mathcal{Y}$ is, at most, the size of the input space $\mathcal{X}$. The proof of Theorem 3 is analogous to the proof of Theorem 4 of [5] and is given in Appendix A.5. Note that the results of [5] do not run into the vertex enumeration problem, because the relevant polyhedron there is ${[1,{\mathrm{e}}^{\epsilon }]}^{\mathcal{X}}$, for which the vertices are known.

We remark that a simplex is not the only possible choice for ${\mathcal{D}}_s$. In general, we can make ${\mathcal{D}}_s$ closer to ${\mathcal{F}}_{\mathcal{U}|s}$ by adding more defining hyperplanes. Doing this allows more $\mathcal{Q}$ to satisfy Theorem 2 and in turn increases the utility of the $\mathcal{Q}$ we find via Theorem 3. However, since $\Gamma$ is related to the ${\mathcal{D}}_s$ via duality, adding extra constraints to the ${\mathcal{D}}_s$ will increase the dimension of $\Gamma$ through the addition of auxiliary variables. This makes the vertex enumeration problem of Theorem 3 more computationally involved. Thus, we have a trade-off between utility and computational complexity. Even with the given, ‘simple’ choice of ${\mathcal{D}}_s$, the computational complexity is quite high: recall that we defined $a=\left|\mathcal{X}\right|$. The polytope $\widehat{\Gamma }$ is $(a-1)$-dimensional and is defined by $a^2+a$ inequalities, thus it has $\mathcal{O}\left({(a^2+a)}^{\frac{a-1}{2}}\right)=\mathcal{O}\left(a^a\right)$ vertices [58]. Since this is the dimension of the linear programming problem, we find that the total complexity of finding $\mathcal{Q}$ is $\mathcal{O}(a^{\omega a}log(a^{a+1}/\delta ))$, where $\omega \approx 2.38$ is the exponent of matrix multiplication and $\delta$ the relative accuracy [58]. Clearly, this becomes infeasible rather quickly for large a.

It should be noted that, in general, the increasing utility obtained by decreasing ${\mathcal{D}}_s$ in size does not approach the optimal utility over all $(\epsilon ,\mathcal{F})$-RLDP mechanisms. This is because, as we take increasingly finer ${\mathcal{D}}_{\mathcal{U}|s}$, we approach the set of $\mathcal{Q}$ that satisfy (4) for all P in ${\mathcal{F}}^{\prime }:=\{P:\forall s{P}_{\mathcal{U}|s}\in {\mathcal{F}}_{\mathcal{U}|s}\}$. As discussed in Section 4.1, one has $\mathcal{F}⊊{\mathcal{F}}^{\prime }$. As a result, the set of $(\epsilon ,{\mathcal{F}}^{\prime })$-RLDP mechanisms is strictly smaller than the set of $(\epsilon ,\mathcal{F})$-RLDP mechanisms.

Example 3. 

We continue Example 2 by taking $\epsilon =log2$. To obtain $\widehat{\Gamma }$ in Theorem 3, we need to combine the defining inequalities of ${\Gamma }_{\epsilon }$ in Definition 3, along with the defining equality ${\sum }_x{P}_x=1$. Regarding the inequalities, we have $2^4=16$ inequalities of the form (3), as well as 4 inequalities of the form $v_x\ge 0$. Together with the equality constraint, we obtain a 3-dimensional polytope in ${\mathbb{R}}^{\mathcal{X}}={\mathbb{R}}^4$. Using a vertex enumeration algorithm, one finds that $\mathcal{V}$ consists of the rows of the matrix V below, where the order of the columns is the order of the rows of Example 1. For each row v, we can calculate $\mu \left(v\right)$, resulting in the vector μ below. Solving (46), we obtain the vector $\widehat{\theta }$ below:

$$V=\left(\begin{array}{cccc}0.0744& 0.3227& 0.5603& 0.0426\\ 0.2426& 0.2426& 0.4783& 0.0364\\ 0.3333& 0.3333& 0.1667& 0.1667\\ 0.1091& 0.4737& 0.2086& 0.2086\\ 0.0993& 0.4310& 0& 0.4697\\ 0.1121& 0.4864& 0& 0.4015\\ 0.3404& 0.3404& 0& 0.3191\\ 0.0770& 0.3343& 0.2944& 0.2944\\ 0.2234& 0.2234& 0& 0.5531\\ 0.4875& 0.1434& 0& 0.3690\\ 0.4360& 0.1283& 0& 0.4358\\ 0.4758& 0.1400& 0.1921& 0.1921\\ 0.3437& 0.1011& 0.2776& 0.2776\\ 0.1602& 0.1602& 0.6316& 0.0481\\ 0.1667& 0.1667& 0.3333& 0.3333\\ 0.3325& 0.0978& 0.5294& 0.0403\end{array}\right),\mu =\left(\begin{array}{c}0.1152\\ 0.0942\\ 0.0087\\ 0.0135\\ 0.1097\\ 0.0968\\ 0.0723\\ 0.0080\\ 0.1240\\ 0.0878\\ 0.1014\\ 0.0106\\ 0.0076\\ 0.1240\\ 0.0075\\ 0.1083\end{array}\right),\widehat{\theta }=\left(\begin{array}{c}1.1899\\ 0\\ 0\\ 0\\ 0\\ 0.7670\\ 0\\ 0\\ 0\\ 0\\ 1.4134\\ 0\\ 0\\ 0\\ 0\\ 0.6297\end{array}\right).$$

(47)

We now obtain the privacy mechanism ${\mathcal{Q}}_{\mathrm{PolyOpt}}$ as follows: each row of $Q_{\mathrm{PolyOpt}}$ corresponds to a non-zero coefficient of $\widehat{\theta }$, multiplied by its corresponding row of V. Thus, we obtain

$$\begin{array}{cc}\hfill Q_{\mathrm{PolyOpt}}& =\left(\begin{array}{cccc}{Q}_{y_1|s_1,u_1}& Q_{y_1|s_1,u_2}& Q_{y_1|s_2,u_1}& Q_{y_1|s_2,u_2}\\ Q_{y_2|s_1,u_1}& Q_{y_2|s_1,u_2}& Q_{y_2|s_2,u_1}& Q_{y_2|s_2,u_2}\\ Q_{y_3|s_1,u_1}& Q_{y_3|s_1,u_2}& Q_{y_3|s_2,u_1}& Q_{y_3|s_2,u_2}\\ Q_{y_4|s_1,u_1}& Q_{y_4|s_1,u_2}& Q_{y_4|s_2,u_1}& Q_{y_4|s_2,u_2}\end{array}\right)\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & =\left(\begin{array}{cccc}0.0885& 0.3840& 0.6667& 0.0507\\ 0.0860& 0.3731& 0& 0.3080\\ 0.6162& 0.1813& 0& 0.6159\\ 0.2094& 0.0616& 0.3333& 0.0254\end{array}\right).\hfill \end{array}$$

(49)

Note that indeed we have $4=b\le a=4$. As for the utility, we have ${\mathrm{I}}_{\widehat{P}}(X;Y)=\mu ·\widehat{\theta }=0.4228$. However, the true utility is significantly lower, namely ${\mathrm{I}}_{P^{*}}(X;Y)=0.2804$.

6. An Optimal Policy for $\mathcal{F}={\mathcal{P}}_{\mathcal{X}}$

As PolyOpt mechanisms are obtained via vertex enumeration in a-dimensional space, this can be computationally infeasible for larger a. Thus, there is a need for methods that, given $\widehat{P}$ and $\mathcal{F}$, can find $(\epsilon ,\mathcal{F})$-RLDP mechanisms with reasonable computational complexity.

In this section, we consider the case where $\mathcal{F}$ is maximal, i.e., $\mathcal{F}={\mathcal{P}}_{\mathcal{X}}$. By itself, this represents a situation where we want privacy for every possible probability distribution on $\mathcal{X}$. This scenario may not be very relevant in practice, but any protocol that we in find this way is also $(\epsilon ,\mathcal{F})$-RLDP for any $\mathcal{F}$. As we will see below, this allows us to find $(\epsilon ,\mathcal{F})$-RLDP protocols in a computationally efficient manner.

We show that $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP is almost equivalent to LDP. We exploit this to create SRR, the RLDP analogue to GRR [5], the LDP mechanism that is optimal for $\epsilon \gg 0$. SRR only depends on $\epsilon$ and $\mathcal{X}$ and not on $\widehat{P}$, and as such does not require an optimization procedure to be found; this makes it a good choice when vertex enumeration is computationally infeasible. The downside is that SRR has a stricter privacy requirement than PolyOpt, as it takes $\mathcal{F}$ to be maximal; in Section 8, we investigate numerically to what extent this results in a lower utility.

We start by giving a characterization of $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP. Like LDP, this can be defined by an inequality constraint on the matrix Q.

Proposition 3. 

$\mathcal{Q}$ satisfies $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP if and only if for all $y\in \mathcal{Y}$ and $(s,u),(s^{\prime },u^{\prime })\in \mathcal{X}$ with $s\ne s^{\prime }$ one has

$$\frac{Q_{y|s,u}}{Q_{y|s^{\prime },u^{\prime }}}\le {\mathrm{e}}^{\epsilon }.$$

(50)

Proof. 

Suppose that $\mathcal{Q}$ satisfies $(\epsilon ,\mathcal{F})$-RLDP with respect to ${\mathcal{P}}_{\mathcal{X}}$. Let $(s,u),(s^{\prime },u^{\prime })\in \mathcal{X}$ with $s\ne s^{\prime }$. Let P be given by

$$P_x=\left\{\begin{array}{cc}\frac{1}{2},\hfill & \mathrm{if}x\in \{(s,u),(s^{\prime },u^{\prime })\},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(51)

Then, $P_{u|s}=1$ and $P_{u^{\prime \prime }|s}=0$ for all $u^{″}\ne u$; an analogous statements holds for $P_{u^{\prime }|s^{\prime }}$. It follows that

$$\begin{array}{}\hfill \mathrm{(52)}& \hfill \frac{Q_{y|s,u}}{Q_{y|s^{\prime },u^{\prime }}}& =\frac{Q_{y|s,u}{P}_{u|s}}{Q_{y|s^{\prime },u^{\prime }}{P}_{u^{\prime }|s^{\prime }}}\hfill \hfill \mathrm{(53)}& & =\frac{{\sum }_{u^{″}}{Q}_{y|s,u^{″}}{P}_{u^{″}|s}}{{\sum }_{u^{″}}{Q}_{y|s^{\prime },u^{″}}{P}_{u^{″}|s^{\prime }}}\hfill \hfill \mathrm{(54)}& & =\frac{{\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y|S=s)}{{\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y|S=s^{\prime })}\le {\mathrm{e}}^{\epsilon }.\hfill \end{array}$$

This proves “⇒”. On the other hand, suppose that $\frac{Q_{y|s,u}}{Q_{y|s^{\prime },u^{\prime }}}\le {\mathrm{e}}^{\epsilon }$ for all $s\ne s^{\prime }$ and $u,u^{\prime }$. Then, for all $s\ne s^{\prime }$ and P, we have

$$\frac{{\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y|S=s)}{{\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y|S=s^{\prime })}=\frac{{\sum }_u{Q}_{y|s,u}{P}_{u|s}}{{\sum }_{u^{\prime }}{Q}_{y|s^{\prime },u^{\prime }}{P}_{u^{\prime }|s^{\prime }}}\le {\mathrm{e}}^{\epsilon }.$$

(55)

Hence, $\mathcal{Q}$ satisfies $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP with respect to. $\mathcal{F}$. □

The proposition demonstrates that RLDP is very similar to LDP. The difference is that the condition “for all $x,x^{\prime }\in \mathcal{X}$” from Definition 2 is relaxed to only those x and $x^{\prime }$ for which $s\ne s^{\prime }$.

Before moving on and introducing a new mechanism, note that Proposition 3 clearly illustrates the reason that the setting in this paper cannot be modeled using the block-structured approach from [12]. We see that if $u\ne u^{\prime }$, we still have a privacy constraint, whereas in [12] this is not the case.

Next, we will introduce a mechanism that exploits the difference between LDP and RLDP. Recall that $a=\left|\mathcal{X}\right|$; then generalized randomized response [19] is the privacy mechanism ${\mathrm{GRR}}^{\epsilon }:\mathcal{X}\to \mathcal{X}$ given by

$${\mathrm{GRR}}_{y|x}^{\epsilon }=\left\{\begin{array}{cc}\frac{{\mathrm{e}}^{\epsilon }}{{\mathrm{e}}^{\epsilon }+a-1}\hfill & \mathrm{if}x=y,\hfill \\ \frac{1}{{\mathrm{e}}^{\epsilon }+a-1}\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(56)

This mechanism has been designed such that $\frac{{\mathrm{GRR}}_{y|x}^{\epsilon }}{{\mathrm{GRR}}_{y|x^{\prime }}^{\epsilon }}={\mathrm{e}}^{\pm \epsilon }$ for $x\ne x^{\prime }$, the maximal fractional difference that $\epsilon$-LDP allows. We will see that for RLDP we can go up to a difference of ${\mathrm{e}}^{\pm 2\epsilon }$ if $x=(s,u)$ and $x^{\prime }=(s,u^{\prime })$, as we typically only need to satisfy

$$Q_{y|s,u}\le {\mathrm{e}}^{\epsilon }{Q}_{y|s^{\prime },u^{\prime }}\le {\mathrm{e}}^{2\epsilon }{Q}_{y|s,u^{\prime }}.$$

(57)

We capture the intuition from the necessary condition (57) in a new mechanism called secret randomized response (SRR). Recall that $a_1=\left|\mathcal{S}\right|$, $a_2=\left|\mathcal{U}\right|$.

Definition 4. 

(Secret randomized response (SRR)). Let $\epsilon >0$. Then, the privacy mechanism ${\mathrm{SRR}}^{\epsilon }:\mathcal{X}\to \mathcal{X}$ is given by

$${\mathrm{SRR}}_{s^{\prime },u^{\prime }|s,u}^{\epsilon }=\left\{\begin{array}{cc}\frac{{\mathrm{e}}^{\epsilon }}{{\mathrm{e}}^{\epsilon }+{\mathrm{e}}^{-\epsilon }(a_2-1)+a-a_2},\hfill & if (s^{\prime },u^{\prime })=(s,u),\hfill \\ \frac{{\mathrm{e}}^{-\epsilon }}{{\mathrm{e}}^{\epsilon }+{\mathrm{e}}^{-\epsilon }(a_2-1)+a-a_2},\hfill & if s^{\prime }=s and u^{\prime }\ne u,\hfill \\ \frac{1}{{\mathrm{e}}^{\epsilon }+{\mathrm{e}}^{-\epsilon }(a_2-1)+a-a_2},\hfill & if s^{\prime }\ne s,\hfill \end{array}\right.$$

(58)

It is clear that $\frac{{\mathrm{SRR}}_{y|s,u}^{\epsilon }}{{\mathrm{SRR}}_{y|s^{\prime },u^{\prime }}^{\epsilon }}\in \{{\mathrm{e}}^{-2\epsilon },{\mathrm{e}}^{\epsilon },1,{\mathrm{e}}^{\epsilon },{\mathrm{e}}^{2\epsilon }\}$, and the two extreme cases are only possible when $s=s^{\prime }$. Thus, we can conclude

Lemma 2. 

SRR satisfies $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP.

Example 4. 

We continue Example 3. Although SRR is closely related to GRR, adopting it can still have a significant impact on utility. For instance, in the setting of Example 3, we obtain

$${\mathrm{GRR}}^{\epsilon }=\left(\begin{array}{cccc}0.4& 0.2& 0.2& 0.2\\ 0.2& 0.4& 0.2& 0.2\\ 0.2& 0.2& 0.4& 0.2\\ 0.2& 0.2& 0.2& 0.4\end{array}\right),{\mathrm{SRR}}^{\epsilon }=\left(\begin{array}{cccc}0.444& 0.111& 0.222& 0.222\\ 0.111& 0.444& 0.222& 0.222\\ 0.222& 0.222& 0.444& 0.111\\ 0.222& 0.222& 0.111& 0.444\end{array}\right).$$

(59)

Then,

$$\begin{array}{cccc}\hfill {\mathrm{I}}_{\widehat{P}}(X;{\mathrm{GRR}}^{\epsilon }\left(X\right))& =0.0419,\hfill & \hfill {\mathrm{I}}_{\widehat{P}}(X;{\mathrm{SRR}}^{\epsilon }\left(X\right))& =0.1005,\hfill \end{array}$$

(60)

$$\begin{array}{cccc}\hfill {\mathrm{I}}_{P^{*}}(X;{\mathrm{GRR}}^{\epsilon }\left(X\right))& =0.0412,\hfill & \hfill {\mathrm{I}}_{P^{*}}(X;{\mathrm{SRR}}^{\epsilon }\left(X\right))& =0.0942.\hfill \end{array}$$

(61)

We see that adopting SRR more than doubles the utility. Compared to Example 3, we see that the utility is still significantly lower than that of PolyOpt, but the advantage is that we obtain SRR directly from ε, without having to take $\widehat{P}$ or $\mathcal{F}$ into account; this ensures a significantly faster computation.

The power of SRR, beyond slightly improving on GRR, is that we can prove it maximizes ${\mathrm{I}}_P(X;Y)$ for sufficiently large $\epsilon$; the cutoff point depends on P. This is proven analogously to the result of [5], where GRR is the optimal LDP mechanism for sufficiently large $\epsilon$.

Theorem 4. 

For every P, there is an ${\epsilon }_0\ge 0$ such that for all $\epsilon \ge {\epsilon }_0$, SRR is the $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP mechanism maximizing ${\mathrm{I}}_P(X;Y)$.

The proof of this theorem follows the same lines as the proof of Theorem 14 of [5], in which it is proven that GRR is the optimal LDP mechanism for sufficiently large $\epsilon$. The proof is presented in Appendix A.6. This solves the problem of finding the optimal $(\epsilon ,{\mathcal{P}}_X)$-mechanism, for sufficiently large $\epsilon$. This strategy is similar to the proof of Theorem 3: one can show that the rows $Q_y$ of the optimal $(\epsilon ,\mathcal{F})$-RLDP mechanism $\mathcal{Q}$ correspond to vertices of a polyhedron, and the optimal weights assigned to these vertices are found using a linear programming problem. Unlike in the case of Theorem 3, however, we can give an explicit description of the set of vertices, and we can solve the linear programming problem analytically.

Our result shows that if one wishes to satisfy $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP, then SRR is a solid choice, especially for larger $\epsilon$, since it maximizes ${\mathrm{I}}_{P^{*}}(X;Y)$ for sufficiently $\epsilon$. Thus, we can optimize ${\mathrm{I}}_{P^{*}}(X;Y)$ without having to know $P^{*}$, with the caveat that the cutoff point for ‘large enough’ depends on $P^{*}$.

In [5], the optimal LDP mechanism in the high-privacy regime (i.e., $\epsilon \ll 1$) was also found. In principle, we could also do this for $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP, but this would not be of much use, as the optimal mechanism would depend on $P^{*}$, which we assume to be unknown.

7. Independent Reporting

Section 5 demonstrated the need to find efficiently computable $(\epsilon ,\mathcal{F})$-RLDP mechanisms with decent utility. In Section 6, we approach this problem by considering $(\epsilon ,{\mathcal{P}}_{\mathcal{X}})$-RLDP instead, allowing us to analytically obtain the optimal mechanism. However, when $\mathcal{F}$ is small, this overapproximation might result in a large loss of utility. In this section, we describe independent reporting (IR), a different heuristic that takes the size of $\mathcal{F}$ into account, while still being significantly less computationally complex than PolyOpt.

The basis of IR is to apply two separate LDP mechanisms ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$ to S and U, respectively, reporting both outputs.

Definition 5. 

Let ${\mathcal{Y}}^1,{\mathcal{Y}}^2$ be sets, and let $\mathcal{Y}={\mathcal{Y}}^1\times {\mathcal{Y}}^2$. Let ${\mathcal{R}}^1:\mathcal{S}\to {\mathcal{Y}}^1$ and ${\mathcal{R}}^2:\mathcal{U}\to {\mathcal{Y}}^2$ be probabilistic maps. Then, theindependent reporting of ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$ is the probabilistic map ${\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}:\mathcal{X}\to \mathcal{Y}$ given by ${\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}(s,u)=({\mathcal{R}}^1\left(s\right),{\mathcal{R}}^2\left(u\right))$.

Suppose that ${\mathcal{R}}^i$ satisfies ${\epsilon }_i$-LDP. The composition theorem for differential privacy [59] tells us that ${\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}$ satisfies $({\epsilon }_1+{\epsilon }_2)$-LDP. However, in the RLDP setting, U only indirectly leaks information about S; therefore, we can get away with a higher ${\epsilon }_2$ compared to the LDP setting. How much higher depends on the degree of relatedness of S and U, which is captured by the possible values of P in $\mathcal{F}$. The precise statement is given in the following result:

Theorem 5. 

Let ${\epsilon }_1,{\epsilon }_2\in {\mathbb{R}}_{\ge 0}$. For each s, let $d_s\in [0,\infty )$ be such that $d_s\ge {\mathrm{rad}}_s\left(\mathcal{F}\right)$. Furthermore, define

$$d=\min \left\{2,\underset{s}{max}\left(2d_s\right)+\underset{s,s^{\prime }}{max}\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1\right\}.$$

(62)

Let ${\delta }_2=log\left(1+\frac{2({\mathrm{e}}^{{\epsilon }_2}-1)}{d}\right)$. Suppose that ${\mathcal{R}}^1$ is ${\epsilon }_1$-LDP and that ${\mathcal{R}}^2$ is ${\delta }_2$-LDP. Then, IR is $({\epsilon }_1+{\epsilon }_2,\mathcal{F})$-RLDP.

If $S=U$, then $\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1=2$ for $s\ne s^{\prime }$, so $d=2$ and ${\delta }_2={\epsilon }_2$. In this case, Theorem 5 is the RLDP analogue to the well-known composition theorem for local differential privacy [59]. In general, ${\delta }_2\ge {\epsilon }_2$; this represents the fact the privacy requirement on ${\mathcal{R}}^2$ is less strict when S and U are only partially related. At the other extreme, if S and U are independent in our observation, we have $\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1=0$ for all $s,s^{\prime }$. Still, we cannot fully disclose U, since S and U might be non-independent under $P^{*}$. The term $d_s$ is present in the definition of d to account for this possibility.

In order to prove Theorem 5, we need the following lemma:

Lemma 3. 

Let $\mathcal{Q}:\mathcal{X}\to \mathcal{Y}$ be an ε-LDP mechanism. Then, for all $y\in \mathcal{Y}$ and all $P,P^{\prime }\in {\mathcal{P}}_{\mathcal{X}}$ we have

$$\frac{{\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y)}{{\mathbb{P}}_{X\sim P^{\prime }}(\mathcal{Q}\left(X\right)=y)}\le 1+\frac{{\mathrm{e}}^{\epsilon }-1}{2}\left|\right|P-P^{\prime }{\left|\right|}_1.$$

(63)

Proof. 

Fix y, and let $Q_y^{\max }={max}_x{Q}_{y|x}$ and $Q_y^{\min }={min}_x{Q}_{y|x}$. By the $\epsilon$-LDP property, it holds that $Q_y^{\max }\le {\mathrm{e}}^{\epsilon }{Q}_y^{\min }$. We hence find

$$\begin{array}{}\hfill \mathrm{(64)}& {\mathbb{P}}_{X\sim P}(\mathcal{Q}\left(X\right)=y)-{\mathbb{P}}_{X\sim P^{\prime }}(\mathcal{Q}\left(X\right)=y)\hfill & ={\displaystyle \sum _{x\in \mathcal{X}}}{Q}_{y|x}(P_x-P_x^{\prime })\hfill \hfill \mathrm{(65)}& \hfill & ={\displaystyle \sum _{x:P_x\ge P_x^{\prime }}}{Q}_{y|x}(P_x-P_x^{\prime })-{\displaystyle \sum _{x:P_x^{\prime }>P_x}}{Q}_{y|x}(P_x^{\prime }-P_x)\hfill \hfill \mathrm{(66)}& \hfill & \le \frac{Q_y^{\max }}{2}\left|\right|P-P^{\prime }{\left|\right|}_1-\frac{Q_y^{\min }}{2}\left|\right|P-P^{\prime }{\left|\right|}_1\hfill \hfill \mathrm{(67)}& \hfill & \le \frac{({\mathrm{e}}^{\epsilon }-1)Q_y^{\min }}{2}\left|\right|P-P^{\prime }{\left|\right|}_1\hfill \hfill \mathrm{(68)}& \hfill & \le \frac{({\mathrm{e}}^{\epsilon }-1){\mathbb{P}}_{X\sim P^{\prime }}(\mathcal{Q}\left(X\right)=y)}{2}\left|\right|P-P^{\prime }{\left|\right|}_1,\hfill \end{array}$$

from which the lemma directly follows. □

Proof 

(Proof of Theorem 5). We start by showing that d is an upper bound for $\left|\right|P_{\mathcal{U}|s}-P_{\mathcal{U}|s^{\prime }}{\left|\right|}_1$. If $d=2$, this is certainly the case. Suppose $d={max}_s\left(2d_s\right)+{max}_{s,s^{\prime }}\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1$. Then, for all $s,s^{\prime }\in \mathcal{S}$ and $P\in \mathcal{F}$ we have

$$\begin{array}{}\hfill \mathrm{(69)}& \hfill \left|\right|P_{\mathcal{U}|s}-P_{\mathcal{U}|s^{\prime }}{\left|\right|}_1& \le \left|\right|P_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s}{\left|\right|}_1+\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1+\left|\right|{\widehat{P}}_{\mathcal{U}|s^{\prime }}-P_{\mathcal{U}|s^{\prime }}{\left|\right|}_1\hfill \hfill \mathrm{(70)}& \hfill & \le d_s+d_{s^{\prime }}+\left|\right|{\widehat{P}}_{\mathcal{U}|s}-{\widehat{P}}_{\mathcal{U}|s^{\prime }}{\left|\right|}_1\hfill \hfill \mathrm{(71)}& \hfill & \le d.\hfill \end{array}$$

Combining Lemma 3 with the fact that ${\epsilon }_2=log\left(1+\frac{d({\mathrm{e}}^{{\delta }_2}-1)}{2}\right)$, it follows that for every $y_2\in {\mathcal{Y}}_2$, we have

$$\begin{array}{}\hfill \mathrm{(72)}& \hfill \frac{{\mathbb{P}}_{X\sim P}({\mathcal{R}}^2\left(U\right)=y_2|S=s)}{{\mathbb{P}}_{X\sim P}({\mathcal{R}}^2\left(U\right)=y_2|S=s^{\prime })}& \le 1+\frac{{\mathrm{e}}^{{\delta }_2}-1}{2}\left|\right|P_{\mathcal{U}|s}-P_{\mathcal{U}|s^{\prime }}{\left|\right|}_1\hfill \hfill \mathrm{(73)}& \hfill & \le 1+\frac{d({\mathrm{e}}^{{\delta }_2}-1)}{2}\hfill \hfill \mathrm{(74)}& \hfill & ={\mathrm{e}}^{{\epsilon }_2}.\hfill \end{array}$$

Given S, the random variables ${\mathcal{R}}^1\left(S\right)$ and ${\mathcal{R}}^2\left(U\right)$ are independent. It follows that for every $y_1\in {\mathcal{Y}}_1$ and every $y_2\in {\mathcal{Y}}_2$, we have

$$\begin{array}{}\hfill \mathrm{(75)}& \hfill \frac{\mathbb{P}({\mathcal{R}}^1\left(S\right)=y_1,{\mathcal{R}}^2\left(U\right)=y_2|S=s)}{\mathbb{P}({\mathcal{R}}^1\left(S\right)=y_1,{\mathcal{R}}^2\left(U\right)=y_2|S=s^{\prime })}& =\frac{\mathbb{P}({\mathcal{R}}^1\left(S\right)=y_1,|S=s)}{\mathbb{P}({\mathcal{R}}^1\left(S\right)=y_1|S=s^{\prime })}·\frac{\mathbb{P}({\mathcal{R}}^2\left(U\right)=y_2|S=s)}{\mathbb{P}({\mathcal{R}}^2\left(U\right)=y_2|S=s^{\prime })}\hfill \hfill \mathrm{(76)}& \hfill & \le {\mathrm{e}}^{{\epsilon }_1+{\epsilon }_2},\hfill \end{array}$$

where the last equality holds because of (74) and because ${\mathcal{R}}^1$ is ${\epsilon }_1$-LDP. This shows that ${\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}$ is $({\epsilon }_1+{\epsilon }_2,\mathcal{F})$-RLDP. □

Theorem 5 establishes the privacy of independent reporting. To maximize the utility, we need to determine how to divide the privacy budget $\epsilon$ between ${\epsilon }_1$ and ${\epsilon }_2$, and which LDP mechanisms to use for ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$. To answer both these questions, we first need an expression for the utility of IR, which is given by the following theorem:

Theorem 6. 

For any $P\in {\mathcal{P}}_{\mathcal{X}}$, one has

$${\mathrm{I}}_P({\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}\left(X\right);X)={\mathrm{I}}_P({\mathcal{R}}^1\left(S\right);S)+{\mathrm{I}}_P({\mathcal{R}}^2\left(U\right);U|{\mathcal{R}}^1\left(S\right)).$$

(77)

Proof. 

Since ${\mathcal{R}}^1\left(S\right)$ and U are independent given S, and ${\mathcal{R}}^2\left(U\right)$ and S are independent given U and ${\mathcal{R}}^1\left(S\right)$, we have

$$\begin{array}{}\hfill \mathrm{(78)}& \hfill {\mathrm{I}}_P({\mathrm{I}\mathrm{R}}_{{\mathcal{R}}^1,{\mathcal{R}}^2}\left(X\right);X)& ={\mathrm{I}}_P({\mathcal{R}}^1\left(S\right),{\mathcal{R}}^2\left(U\right);U,S)\hfill \hfill \mathrm{(79)}& \hfill & ={\mathrm{I}}_P({\mathcal{R}}^1\left(S\right);U,S)+{\mathrm{I}}_P({\mathcal{R}}^2\left(U\right);U,S|{\mathcal{R}}^1\left(S\right))\hfill \hfill \mathrm{(80)}& \hfill & ={\mathrm{I}}_P({\mathcal{R}}^1\left(S\right);S)+{\mathrm{I}}_P({\mathcal{R}}^2\left(U\right);U|{\mathcal{R}}^1\left(S\right)).\hfill \end{array}$$

□

We use Theorems 5 and 6 to find high-utility IR protocols that satisfy $(\epsilon ,\mathcal{F})$-RLDP, given $\epsilon$ and $\mathcal{F}$. To do so, we need to choose ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$, and split the privacy budget between them. Since the expression for the utility of IR in Theorem 6 contains a term ${\mathrm{I}}_P({\mathcal{R}}^1\left(S\right);S)$, the ${\mathcal{R}}^1$ that maximizes this is GRR when $\epsilon$ is large enough; thus, we choose ${\mathcal{R}}^1=\mathrm{GRR}$. The second term in the utility expression is

$${\mathrm{I}}_P({\mathcal{R}}^2\left(U\right);U|{\mathcal{R}}^1\left(S\right))={\mathbb{E}}_r\left[{\mathrm{I}}_{U\sim P_U|{\mathcal{R}}^1\left(S\right)=r}({\mathcal{R}}^2\left(U\right);U)\right].$$

(81)

This is the expected value of an expression that is maximized for ${\mathcal{R}}^2=\mathrm{GRR}$, with the caveat that the maximization only holds when $\epsilon$ is large enough, and what ‘large enough’ is depends on the distribution of U. Since this gives us a choice of ${\mathcal{R}}^2$ independent of the distribution, we ignore this caveat and take ${\mathcal{R}}^2=\mathrm{GRR}$ as well.

Having chosen ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$, we are only left with the division of the privacy budget. If we choose ${\epsilon }_2$, then by Theorem 5 the privacy parameters of ${\mathcal{R}}^1$ and ${\mathcal{R}}^2$ are ${\epsilon }_1=\epsilon -{\epsilon }_2$ and ${\delta }_2=log\left(1+\frac{2({\mathrm{e}}^{{\epsilon }_2}-1)}{d}\right)$, respectively. It follows that to find a high-utility IR protocol, we have to solve the following optimization problem:

$$\begin{array}{cc}\hfill {\mathrm{maximize}}_{{\epsilon }_2}& {\mathrm{I}}_P\left({\mathrm{GRR}}_{\epsilon -{\epsilon }_2}\left(S\right),{\mathrm{GRR}}_{log\left(1+\frac{2({\mathrm{e}}^{{\epsilon }_2}-1)}{d}\right)}\left(U\right);S,U\right)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& {\epsilon }_2\in [0,\epsilon ].\hfill \end{array}$$

(82)

This optimization problem is only 1-dimensional. While it is not straightforward to express the complexity of solving this in $\mathcal{O}$-notation, our experiments in Section 8 show this can be quickly performed numerically, and significantly faster than PolyOpt.

Example 5. 

We continue Example 4. Having found ${\mathrm{rad}}_s\left(\mathcal{F}\right)$ and ${\widehat{P}}_{\mathcal{U}|s_1},{\widehat{P}}_{\mathcal{U}|s_2}$ in Example 2, we conclude that, in Theorem 5, we have

$$d=min\left\{2,2·max\{0.6107,0.3061\}+{\left|\left|\left(\genfrac{}{}{0pt}{}{0.4118}{0.5882}\right)-\left(\genfrac{}{}{0pt}{}{0.3133}{0.6867}\right)\right|\right|}_1\right\}=1.4591.$$

(83)

It follows that ${\delta }_2=log\left(1+\frac{2}{1.4591}({\mathrm{e}}^{{\epsilon }_2}-1)\right)=log(1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707)$. For a given value of ${\epsilon }_2$, the matrix corresponding to $\mathrm{I}\mathrm{R}({\mathrm{GRR}}^{log\left(2\right)-{\epsilon }_2},{\mathrm{GRR}}^{{\delta }_2})$ is the Kronecker product

$$\begin{array}{c}\left(\begin{array}{c}\frac{2{\mathrm{e}}^{-{\epsilon }_2}}{2{\mathrm{e}}^{-{\epsilon }_2}+1}\frac{1}{2{\mathrm{e}}^{-{\epsilon }_2}+1}\\ \frac{1}{2{\mathrm{e}}^{-{\epsilon }_2}+1}\frac{2{\mathrm{e}}^{-{\epsilon }_2}}{2{\mathrm{e}}^{-{\epsilon }_2}+1}\end{array}\right)\otimes \left(\begin{array}{c}\frac{1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707}{1.3707{\mathrm{e}}^{{\epsilon }_2}+0.6293}\frac{1}{1.3707{\mathrm{e}}^{{\epsilon }_2}+0.6293}\\ \frac{1}{1.3707{\mathrm{e}}^{{\epsilon }_2}+0.6293}\frac{1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707}{1.3707{\mathrm{e}}^{{\epsilon }_2}+0.6293}\end{array}\right)\hfill \\ =\frac{1}{C}\left(\begin{array}{cccc}2.7414-0.7414{\mathrm{e}}^{-{\epsilon }_2}& 2{\mathrm{e}}^{-{\epsilon }_2}& 1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707& 1\\ 2{\mathrm{e}}^{-{\epsilon }_2}& 2.7414-0.7414{\mathrm{e}}^{-{\epsilon }_2}& 1& 1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707\\ 1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707& 1& 2.7414-0.7414{\mathrm{e}}^{-{\epsilon }_2}& 2{\mathrm{e}}^{-{\epsilon }_2}\\ 1& 1.3707{\mathrm{e}}^{{\epsilon }_2}-0.3707& 2{\mathrm{e}}^{-{\epsilon }_2}& 2.7414-0.7414{\mathrm{e}}^{-{\epsilon }_2}\end{array}\right),\hfill \end{array}$$

(84)

where $C=(2{\mathrm{e}}^{-{\epsilon }_2}+1)(1.3707{\mathrm{e}}^{{\epsilon }_2}+0.6293)$. We now wish to optimize its utility, i.e., find the ${\epsilon }_2\in [0,log2]$ that maximizes ${\mathrm{I}}_{\widehat{P}}(X;Y)$. The optimum occurs at the boundary ${\epsilon }_2=log\left(2\right)$, for which ${\mathrm{I}}_{\widehat{P}}(X;Y)=0.0755$. Notice that now ${\epsilon }_1=0$, so ${\mathcal{R}}^1={\mathrm{GRR}}^0$ is completely random: its output does not depend on the input. In other words, the optimal IR protocol in this case does not transmit any direct information about S at all, only indirectly through ${\mathrm{GRR}}^{{\delta }_2}\left(U\right)$. In this case, we have

$$Q_{\mathrm{I}\mathrm{R}}=\left(\begin{array}{cccc}0.3517& 0.1483& 0.3517& 0.1483\\ 0.1483& 0.3517& 0.1483& 0.3517\\ 0.3517& 0.1483& 0.3517& 0.1483\\ 0.1483& 0.3517& 0.1483& 0.3517\end{array}\right).$$

(85)

Regarding the ‘true’ utility, we have ${\mathrm{I}}_{P^{*}}(X;Y)=0.0718$. Interestingly, ${\mathcal{Q}}_{\mathrm{I}\mathrm{R}}$ yields less utility than SRR. As we will see in Section 8, this is typical for small $\mathcal{S}$ and $\mathcal{U}$.

8. Experiments

In order to gain insight into the behavior of the different mechanisms, we performed several experiments, both on synthetic and real data. We compared the three mechanisms introduced in this paper (PolyOpt, SRR, and IR). Throughout, we let $\mathcal{F}$ be a confidence set for a ${\chi }^2$-test, i.e., for a Rényi divergence with $\alpha =2$. We used the results of Section 4 to find explicit expressions for $L_{u|s}\left(\mathcal{F}\right)$ and (an upper bound for) ${\mathrm{rad}}_s\left(\mathcal{F}\right)$. Recall from Section 3 that

$$\mathcal{F}=\left\{P\in {\mathcal{P}}_{\mathcal{X}}:D_2(\widehat{P}\left|\right|P)\le log\left(1+\frac{F_{{\chi }^2,a-1}^{-1}(1-\beta )}{n}\right)\right\},$$

(86)

where $F_{{\chi }^2,a-1}$ is the cumulative density function of the ${\chi }^2$-distribution with $a-1$ degrees of freedom, and $\beta \in (0,1)$ is a chosen significance level. Throughout the experiments, we took $\beta =0.05$, unless otherwise specified.

We used ${\mathrm{I}}_{\widehat{P}}(X;Y)$ as a utility metric, divided by $\mathrm{H}\left(X\right)$ to obtain the normalized mutual information (NMI). We used this rather than ${\mathrm{I}}_{P^{*}}(X;Y)$, as the aggregator only has access to the former. In fact, while $P^{*}$ is known for the synthetic data, this is not the case for real data, so we cannot even use ${\mathrm{I}}_{P^{*}}(X;Y)$ as a utility metric.

We compared our methods to two existing approaches, each with a slightly different privacy model. First, we compared to an LDP mechanism, to see to what extent the RLDP framework offered a utility improvement over regular LDP. As the LDP mechanism, we chose GRR, because it optimizes ${\mathrm{I}}_P(X;Y)$, our privacy metric, in the low-privacy regime [5]. Second, we compared to the non-robust optimal mechanism of [3]. This mechanism is obtained in a manner similar to PolyOpt, and is the optimal mechanism that satisfies (in our notation) $(\epsilon ,\left\{\widehat{P}\right\})$-RLDP. In other words, it is optimal in the scenario where one knows $P^{*}$ precisely. We shall refer to this mechanism as NR (non-robust). Typically, we would expect NR to have a higher utility than our RLDP mechanisms, (because it only needs to satisfy privacy with respect to. one distribution) and GRR to have worse a utility (because LDP is stricter than RLDP).

8.1. Adult Data Set

We performed numerical experiments on the adult data set (n = 32,561) [60], which contains demographic data from the 1994 US census. Some examples, where we used different categorical attributes from the data set as S and U, are depicted in Figure 2. We omitted PolyOpt from the larger two experiments, as the space complexity became unfeasible: for occupation vs. education, the polyhedron $\widehat{\Gamma }$ was 240-dimensional and was defined by 57,840 inequality constraints; to find its set of vertices Matlab needed to operate on a 57,840 × 57,840 matrix, whose size (24.7 GB) exceeded Matlab’s maximum array size.

We can see that PolyOpt clearly outperformed IR and SRR in the first two experiments, especially in the high-privacy regime (low $\epsilon$). Similarly, IR outperformed SRR in the high-privacy regime, but was slightly overtaken for high $\epsilon$. This is interesting, since SRR satisfies a stronger privacy guarantee, as it provides privacy for all adversary assumptions, so we expected it to offer less utility than IR. An explanation for this is that IR is forced to transmit S and U separately, and so it can be less efficient than SRR, which does not have this restriction. At any rate, the difference between IR and SRR in the low-privacy regime was only marginal compared to the advantage of PolyOpt over both. In the second two experiments, where PolyOpt was infeasible, we can see that IR clearly outperformed SRR. Overall, we see that, especially in the low-privacy regime, PolyOpt was the preferable RLDP mechanism, followed by IR and SRR. Furthermore, we can see that, in all experiments, GRR performed the worst, and the best RLDP mechanism significantly outperformed GRR. This shows that adopting RLDP as a privacy metric results in significantly better utility over LDP. Conversely, NR outperfored the RLDP methods, although the difference between NR and PolyOpt was marginal for higher $\epsilon$. As for PolyOpt, NR was computationally out of reach for larger $\left|\mathcal{X}\right|$.

8.2. Synthetic Data

To study the robustness of our method with respect to utility (Section 8.4) and privacy (Section 8.3), we also needed experiments in which $P^{*}$ was known. For this, we considered experiments on synthetic data. For this, we first randomly created a probability distribution $P^{*}$ on $\mathcal{X}$, where $\mathcal{X}$ was the same as in the experiments on the adult data set. The distribution $P^{*}$ was drawn from the Jeffreys prior on ${\mathcal{P}}_{\mathcal{X}}$, i.e., the symmetric Dirichlet distribution with parameter $\frac{1}{2}$. From $P^{*}$, we then drew $n=32,561$ elements of $\mathcal{X}$, which we used to obtain the estimate $\widehat{P}$; this estimate was then used to create the privacy mechanisms. We carried this out 100 times, and we averaged the NMI of these 100 distributions. The results are shown in Figure 3. The results were similar to those of the experiments of the adult data set: PolyOpt outperformed IR, which outperformed SRR, for small $\left|\mathcal{X}\right|$ SRR could overtake IR in the low-privacy regime. Furthermore, GRR was the worst overall, while NR was the best overall, but only by a small margin.

8.3. Realized Privacy Parameter

In the previous subsections, we saw that NR had a (marginally) better utility than PolyOpt. However, this is not a completely fair comparison, since NR was only designed to give privacy for $X\sim \widehat{P}$ and might result in a larger privacy leakage for $X\sim P^{*}$. For the synthetic data, $P^{*}$ was known, and we could measure the true privacy leakage. For a protocol $\mathcal{Q}$, we defined the realized privacy parameter ${\epsilon }^{*}$ as

$$\begin{array}{cc}\hfill {\epsilon }^{*}& =\underset{\begin{array}{c}y\in \mathcal{Y},\\ s_1,s_2\in \mathcal{S}\end{array}}{max}\frac{{\mathbb{P}}_{X\sim P^{*}}(Y=y\mid S=s_1)}{{\mathbb{P}}_{X\sim P^{*}}(Y=y\mid S=s_2)}\hfill \\ \hfill & =\underset{\begin{array}{c}y\in \mathcal{Y},\\ s_1,s_2\in \mathcal{S}\end{array}}{max}\frac{{\sum }_u{Q}_{y|s_1,u}{P}_{u|s_1}^{*}}{{\sum }_u{Q}_{y|s_2,u}{P}_{u|s_2}^{*}}.\hfill \end{array}$$

Note that this becomes ∞ when there exist $s,y$ such that ${\mathbb{P}}_{X\sim P^{*}}(Y=y\mid S=s)=0$. We compared ${\epsilon }^{*}$ for NR and PolyOpt: the results are shown in Figure 4, where we give the 25% and 75% quantiles for both protocols, out of 100 considered distributions. As one can see, NR’s ${\epsilon }^{*}$ was consistently greater than $\epsilon$, while PolyOpt’s ${\epsilon }^{*}$ was consistently lesser. This is what we expected, as NR does not give privacy guarantees for $P^{*}$, but PolyOpt does when $P^{*}\in \mathcal{F}$, which happens with 95% probability. Note that the privacy leakage was especially bad for low $\epsilon$: at $\epsilon =0.075$, the lowest value of $\epsilon$ we tested, the 75%-quantile of ${\epsilon }^{*}$ of NR was $0.3897$, which is more than 5 times the desired privacy parameter. Overall, we can conclude that NR gave marginally better utility, but this came at quite a privacy cost.

8.4. Utility Robustness

For the synthetic data sets (where we knew $P^{*}$), we also investigated the normalized difference in mutual information $\frac{{\mathrm{I}}_{\widehat{P}}(X;Y)-{\mathrm{I}}_{P^{*}}(X;Y)}{{\mathrm{I}}_{\widehat{P}}(X;Y)}$, to see to what extent we could use ${\mathrm{I}}_{\widehat{P}}(X;Y)$ as a utility metric in lieu of the true utility ${\mathrm{I}}_{P^{*}}(X;Y)$. This is shown for the three methods in Figure 5, at $\epsilon =1.5$. Overall, we can see that the difference was quite minor: for all three methods, the difference in NMI, even at its most extreme, was less than 3% of the NMI value. Furthermore, the differences were very symmetric, with the difference being positive and negative approximately equally often. We can conclude that we were justified in using ${\mathrm{I}}_{\widehat{P}}(X;Y)$ as a utility metric in the other experiments.

8.5. Impact of $\beta$

We also considered the impact of $\beta$ on utility for synthetic data (fixing $\epsilon =1.5$). The results are shown in

Table 2. NMI for synthetic data for various values of $\beta$ ($\epsilon =1.5$).

	$a_1=2,a_2=5$			$a_1=5,a_2=2$
$\beta$	0.1	0.01	0.001	0.1	0.01	0.001
SRR	0.231	0.231	0.231	0.126	0.126	0.126
PolyOpt	0.727	0.723	0.719	0.374	0.372	0.370
IR	0.512	0.501	0.492	0.169	0.165	0.162
	$a_1=15,a_2=16$			$a_1=42,a_2=6$
$\beta$	0.1	0.01	0.001	0.1	0.01	0.001
SRR	0.009	0.009	0.009	0.005	0.005	0.005
IR	0.055	0.053	0.051	0.052	0.052	0.052

, which are averages over 100 runs. Note that SRR does not depend on $\beta$, since it assumes $\mathcal{F}={\mathcal{P}}_{\mathcal{X}}$. Interestingly, we can see that the impact of $\beta$ was quite limited; changing $\beta$ by a factor 100 had at most about $4\%$ impact on NMI. This impact was less for PolyOpt than for IR, and less for larger $\mathcal{X}$. Overall, we can conclude that by choosing $\beta$ closer to 0, we can significantly increase the robustness of privacy without making a considerable impact on utility.

9. Conclusions and Future Work

In this paper, we presented a number of algorithms that, given a desired privacy level $\epsilon$, an estimated distribution $\widehat{P}$, and a bound on the Rényi divergence $D_{\alpha }(\widehat{P}\left|\right|P)$, return privacy mechanisms that satisfy a differential privacy-like privacy constraint for the part of the data that is considered sensitive, for all distributions P within the divergence bound. The first class of privacy mechanisms, PolyOpt, offers high utility, but is computationally complex, as it relies on vertex enumeration. The second class, SRR, satisfies a stronger privacy requirement and is optimal in the low-privacy regime with reference to this requirement, but as a result has less utility than mechanisms that do not satisfy this stronger privacy requirement. The third class, IR, is a general framework for releasing the sensitive and non-sensitive part of the data independently, and the optimal division of the privacy budget between these can be found via 1-dimensional optimization; thus, the optimal IR mechanism can be found quickly, while still offering decent utility. Furthermore, taking RLDP rather than LDP as a privacy constraint, i.e., protecting only the part of the data that is sensitive, significantly improves utility. In particular, we showed that the utility of PolyOpt is close to the utility of the optimal non-robust privacy mechanism. In other words, asking for robustness in privacy comes at only a small performance penalty in utility. At the same time, we showed that not asking for robustness comes at a substantial privacy cost.

There are various interesting directions for future research to build upon the results in this paper. One direction is to find analytical bounds on the performance gap between PolyOpt and optimal mechanisms, in particular on the gap with reference to either the non-robust optimal mechanism from [3] or with reference to an optimal robust mechanism. Note, however, that for the moment we do not have any results on optimal robust mechanisms. Another direction is to improve the performance of the low-complexity algorithms that have been proposed. For instance, in independent reporting, one could change the underlying LDP mechanism from GRR to an optimal mechanism. Since GRR is only optimal in the high-privacy regime, we expect that there would be room for improvement in the low-privacy regime. A significant challenge is incorporating optimal mechanisms along the lines of [5]; however, these mechanisms depend on $P^{*}$ which is inaccessible in the RLDP framework. Yet another interesting direction would be to incorporate robustness in utility in addition to robustness in privacy. This would require finding a mechanism that maximizes ${min}_{P\in \mathcal{F}}{\mathrm{I}}_P(X;Y)$. The challenge in this is that ${\mathrm{I}}_P(X;Y)$ is concave in P, which makes minimizing it over $\mathcal{F}$ difficult. Finally, it would be interesting to apply the RLDP framework to other models. In this work, we studied the model where X splits into a sensitive part S and a non-sensitive part U. It would be interesting to also study the more general case where X is correlated with the sensitive data S, or to apply RLDP to the models that are studied in [12].