Higher-Order Interactions and Their Duals Reveal Synergy and Logical Dependence beyond Shannon-Information

Abstract

Information-theoretic quantities reveal dependencies among variables in the structure of joint, marginal, and conditional entropies while leaving certain fundamentally different systems indistinguishable. Furthermore, there is no consensus on the correct higher-order generalisation of mutual information (MI). In this manuscript, we show that a recently proposed model-free definition of higher-order interactions among binary variables (MFIs), such as mutual information, is a Möbius inversion on a Boolean algebra, except of surprisal instead of entropy. This provides an information-theoretic interpretation to the MFIs, and by extension to Ising interactions. We study the objects dual to mutual information and the MFIs on the order-reversed lattices. We find that dual MI is related to the previously studied differential mutual information, while dual interactions are interactions with respect to a different background state. Unlike (dual) mutual information, interactions and their duals uniquely identify all six 2-input logic gates, the dy- and triadic distributions, and different causal dynamics that are identical in terms of their Shannon information content.

1. Introduction

1.1. Higher-Order Interactions

All non-trivial structures in data or probability distributions correspond to dependencies among the different features, or variables. These dependencies can be present among pairs of variables, i.e., pairwise, or can be higher-order. A dependency, or interaction, is called higher-order if it is inherently a property of more than two variables and if it cannot be decomposed into pairwise quantities. The term has been used more generally to refer simply to complex interactions, as for example in [1] to refer to changes in gene co-expression over time; in this article, however, it is used only in the stricter sense defined in Section 2.

The reason such higher-order structures are interesting is twofold. First, higher-order dependence corresponds to a fundamentally different kind of communication and interaction among the components of a system. If a system contains higher-order interactions, then its dependency structure cannot be represented by a graph and requires a hypergraph, where a single ‘hyperedge’ can connect more than two nodes. It is desirable to be able to detect and describe such systems accurately, which requires a good understanding of higher-order interactions. Second, higher-order interactions might play an important role in nature, and have been identified in various interaction networks, including genetic [2,3,4,5], neuronal [6,7,8,9,10], ecological [11,12,13], drug interaction [14], social [15,16,17], and physical [18,19] networks. Furthermore, there is evidence that higher-order interactions are responsible for the rich dynamics [20] or bistability [21] in biological networks; for example, synthetic lethality experiments have shown that the trigenic interactions in yeast form a larger network than the pairwise interactions [4].

Despite this, purely pairwise descriptions of nature have been remarkably successful, which the authors of [22,23] attribute to the fact that there are regimes in terms of the strength and density of coupling among the variables within which pairwise descriptions are sufficient. Alternatively, it may be attributed to the fact that higher-order interactions have been understudied and their effects underestimated. Currently, perhaps the most promising method of quantifying higher-order interactions is information theory. The two most commonly used quantities are mutual information and its higher-order generalisation (used in, e.g., [24,25]) and the total correlation (introduced in [26] and recently used in [27]). However, one particular problem of interest that total correlation and mutual information do not address is that of synergy and redundancy. Given a set of variables with an nth-order dependency, what part of that is exclusively nth-order (called the synergistic part), and what part can be found in a subset of $m<n$ variables as well (the redundant part)? Quantifying the exact extent to which shared information is synergistic is an open problem, and is most commonly addressed using partial information decomposition [28], which has been applied mainly in the context of theoretical neuroscience [29]. In this article, a different more statistical approach to identifying synergy is taken, which is ultimately shown to be intimately related to information theory while offering significant advantages beyond classical entropy-based quantities.

1.2. Model-Free Interactions and the Inverse Ising Problem

In 1957, E.T. Jaynes famously showed that statistical equilibrium mechanics can be seen as a maximum entropy solution to the inverse problem of constructing a probability distribution that best reproduces a sample distribution [30]. More precisely, the equilibrium dynamics of the (inhomogeneous, or glass-like) generalised Ising model with interactions up to the nth order arise naturally as the maximum entropy distribution compatible with a dataset after observing the first n moments among binary variables. This means that in order to reproduce the moments in the data in a maximally non-committal way, it is necessary to introduce higher-order interactions, i.e., terms that involve more than two variables, in the description of the system. Fitting such a generalised Ising model to data is nontrivial; while the log-likelihood of the Ising model is concave in the the coupling parameters, the cost of evaluating it is exponential in the total number of variables N, which is often intractable in practice [31]. In [32], the authors introduced an estimator of model-free interactions (MFIs) that exactly coincides with the solution to the inverse generalised Ising problem. Moreover, the cost of estimating all nth-order model-free interactions among N variables from M observations scales as $\mathcal{O}\left(M·\left(\genfrac{}{}{0pt}{}{N}{n}\right)\right)=\mathcal{O}\left(M{N}^n\right)$ (i.e., polynomially) in the total system size N. However, this is true only when sufficient data is available. With limited data, certain interactions might require inferring the conditional dependencies from the data, which in the worst case scales exponentially in N again. The definition of MFIs offered in [32] seems to be a general one; in addition to offering a solution to the inverse generalised Ising problem, MFIs are expressible in terms of average treatment effects (ATEs) or regression coefficients. Throughout this article, the general term ‘MFI’ is used, and may be read simply as referring to the maximum entropy or Ising interaction.

1.3. Outline

In Section 2.1, the definition of the MFIs is stated along with a number of their properties. To explicitly link the MFIs to information theory, a redefinition of mutual information in terms of Möbius inversions is provided in Section 2.2, which is then linked to a similar redefinition of the MFIs in Section 3.1 and Section 3.2. A definition in terms of Möbius inversions naturally leads to dual definitions of all objects, which are subsequently explored in Section 3.3. Then, in Section 4, simple fundamental examples are used to demonstrate that MFIs can differentiate distributions that entropy-based quantities cannot. Finally, the results are summarised and reflected upon in Section 5.

2. Background

2.1. Model-Free Interactions

We start by re-defining the interactions introduced in [32]. We define the isolated effect (or 1-point interaction) $I_i^{\left(Y\right)}$ of a variable $X_i\in X$ on an observable Y as

$$I_i^{\left(Y\right)}=\frac{\partial Y}{\partial X_i}{|}_{\underline{X}=0},\underline{X}=X\backslash \left\{X_i\right\}$$

(1)

where the effect of $X_i$ on Y is isolated by conditioning on all other variables being zero. This expression is well-defined, as the restriction of a derivative is the derivative of the restriction. A pair of variables $X_i$ and $X_j$ has a 2-point interaction $I_{ij}^{\left(Y\right)}$ when the value of $X_j$ changes the 1-point interaction of $X_i$ on Y:

$$I_{ij}^{\left(Y\right)}=\frac{\partial I_i^{\left(Y\right)}}{\partial X_j}{|}_{\underline{X}=0}=\frac{{\partial }^{2}Y}{\partial X_j\partial X_i}{|}_{\underline{X}=0},\underline{X}=X\backslash \{X_i,X_j\}$$

(2)

A third variable $X_k$ can modulate this 2-point interaction through what we call a 3-point interaction, $I_{ijk}^{\left(Y\right)}$:

$$I_{ijk}^{\left(Y\right)}=\frac{\partial I_{ij}^{\left(Y\right)}}{\partial X_k}{|}_{\underline{X}=0}=\frac{{\partial }^{3}Y}{\partial X_k\partial X_j\partial X_i}{|}_{\underline{X}=0},\underline{X}=X\backslash \{X_i,X_j,X_k\}$$

(3)

This process of taking derivatives with respect to an increasing number of variables can be repeated to define n-point interactions.

Definition 1 (n-point interaction with respect to outcome Y).

Let p be a probability distribution over a set X of random variables $X_i$ and let Y be a function $Y:X\to \mathbb{R}$. Then, the n-point interaction $I_{X_1\cdots X_n}$ between variables $\{X_1,\dots ,X_n\}\subseteq X$ is provided by

$$\begin{array}{c}\hfill I_{X_1\dots X_n}^{\left(Y\right)}=\frac{{\partial }^{n}Y\left(X\right)}{\partial X_1\dots \partial X_n}{|}_{\underline{X}=0}\end{array}$$

(4)

where $\underline{X}=X\backslash \{X_1,\dots X_n\}$.

This definition of interaction makes explicit the fact that interactions are defined with respect to some outcome. The authors of [32] refer to the interactions from Definition 1 as additive, which they distinguish from multiplicative interactions. However, when the outcome is chosen to be the log of the joint distribution $p\left(X\right)$ over all variables X, then the additive and multiplicative interactions are equivalent and simply related through a logarithm [32]. Setting the outcome to be $logp\left(X\right)$ has other nice properties as well. First, while probabilities are restricted to the non-negative reals, a log-transformation removes this restriction and makes the outcome and subsequent interactions take both positive and negative values, which can have different interpretations. Second, it is this outcome that makes the interactions interpretable as maximum entropy interactions, as they exactly coincide with Ising interactions. Finally, this can be considered the most general outcome possible, as all marginal and conditional probabilities are encoded in this joint distribution. This leads to the following definition of a model-free interaction.

Definition 2 (model-free n-point interaction between binary variables).

A model-free n-point interaction (MFI) is an n-point interaction between binary random variables with respect to the logarithm of their joint probability

$$\begin{array}{c}\hfill I_{X_1\dots X_n}{I}_{X_1\dots X_n}^{(logp(X\left)\right)}=\frac{{\partial }^{n}logp\left(X\right)}{\partial X_1\dots \partial X_n}{|}_{\underline{X}=0}\end{array}$$

(5)

where $\underline{X}=X\backslash \{X_1,\dots X_n\}$.

If the variables $X_i\in X$ are binary, then a definition for a derivative with respect to a binary variable is needed.

Definition 3 (derivative of a function with respect to a binary variable).

Let $f:{\mathbb{B}}^n\to \mathbb{R}$ be a real-valued function of a set X of n binary variables, labelled as $X_i$, $1\le i\le n$. Then, the derivative operator with respect to $X_i$ acts on $f\left(X\right)$ as follows:

$$\begin{array}{c}\hfill \frac{\partial }{\partial X_i}f\left(X\right)=f(X_i=1,X\backslash X_i)-f(X_i=0,X\backslash X_i)\end{array}$$

(6)

The linearity of the derivative operator then immediately and uniquely defines the higher-order derivatives.

Using this definition, the n-point interactions become model-free in the sense that they are ratios of probabilities that do not involve the functional form of the joint probability distribution. For example, writing $X_{ijk}=(a,b,c)$ for $(X_i=a,X_j=b,X_k=c)$, the first three orders can be written out as follows (recall that the notation $\frac{\partial }{\partial X_i}$ here refers to the derivative operator from Definition 3):

$$\begin{array}{}\mathrm{(7)}& \hfill I_i& =\frac{\partial logp\left(X\right)}{\partial X_i}{|}_{\underline{X}=0}=log\frac{p\left(X_i=1\mid \underline{X}=0\right)}{p\left(X_i=0\mid \underline{X}=0\right)}\hfill \mathrm{(8)}& \hfill I_{ij}& =\frac{{\partial }^{2}logp\left(X\right)}{\partial X_j\partial X_i}{|}_{\underline{X}=0}=log\frac{p\left(X_{ij}=(1,1)\mid \underline{X}=0\right)}{p\left(X_{ij}=(0,1)\mid \underline{X}=0\right)}\frac{p\left(X_{ij}=(0,0)\mid \underline{X}=0\right)}{p\left(X_{ij}=(1,0)\mid \underline{X}=0\right)}\hfill & \hfill I_{ijk}& =\frac{{\partial }^{3}logp\left(X\right)}{\partial X_k\partial X_j\partial X_i}{|}_{\underline{X}=0}=\hfill \mathrm{(9)}& \hfill log& \frac{p\left(X_{ijk}=(1,1,1)\mid \underline{X}=0\right)}{p\left(X_{ijk}=(0,0,0)\mid \underline{X}=0\right)}\frac{p\left(X_{ijk}=(1,0,0)\mid \underline{X}=0\right)}{p\left(X_{ijk}=(0,1,1)\mid \underline{X}=0\right)}\hfill & \hfill \times & \frac{p\left(X_{ijk}=(0,1,0)\mid \underline{X}=0\right)}{p\left(X_{ijk}=(1,0,1)\mid \underline{X}=0\right)}\frac{p\left(X_{ijk}=(0,0,1)\mid \underline{X}=0\right)}{p\left(X_{ijk}=(1,1,0)\mid \underline{X}=0\right)}\hfill \end{array}$$

where Bayes’ rule is used to replace joint probabilities with conditional probabilities. This definition of interaction has the following properties:

• It is symmetric in terms of the variables, as $I_S=I_{\pi \left(S\right)}$ for any set of variables S and any permutation $\pi$.
• Conditionally independent variables do not interact: $X_i\perp \perp X_j\mid \underline{X}\Rightarrow I_{ij}=0$.
• If $\underline{X}=\varnothing$, the definition coincides with that of a log-odds ratio, which has already been considered as a measure of interaction in, e.g., [33,34].
• The interactions are model-free; no knowledge of the functional form of $p\left(X\right)$ is required, and the probabilities can be directly estimated from i.i.d. samples.
• The MFIs are exactly the Ising interactions in the maximum entropy model after observing moments of the data. This can be readily verified by setting

  $$p\left(s\right)={\mathcal{Z}}^{-1}exp(\sum _n\sum _{i_1,\dots ,i_n}{J}_{i_1\dots i_n}{s}_{i_1}\dots s_{i_n})$$

  and using Definition 2.

Furthermore, in Appendix A.2 the following two useful properties are introduced and proved:

• An n-point interaction can only be non-zero if all n variables are in each other’s minimal Markov blanket.
• If $\underline{X}$ does not include the full complement of the interacting variables, the bias this induces in the estimate of the interaction is proportional to the pointwise mutual information of states where the omitted variables are 0.

2.2. Mutual Information as a Möbius Inversion

The definition of an n-point interaction as a derivative of a derivative is reminiscent of Gregory Bateson’s view of information as a difference which makes a difference [35]; however, the relationship between information theory and model-free interactions rests on more than a linguistic coincidence. It turns out that interactions and information are generalised derivatives of similar functions on Boolean algebras. To see this, consider the definition of pairwise mutual information and its third-order generalisation:

$$\begin{array}{}\mathrm{(10)}& \hfill MI(X,Y)& =H\left(X\right)-H(X\mid Y)\hfill \mathrm{(11)}& & =H\left(X\right)+H\left(Y\right)-H(X,Y)\hfill \mathrm{(12)}& \hfill MI(X,Y,Z)& =MI(X,Y)-MI(X,Y\mid Z)\hfill & & =H\left(X\right)+H\left(Y\right)+H\left(Z\right)\hfill & & -H(X,Y)-H(X,Z)-H(Y,Z)+H(X,Y,Z)\hfill \end{array}$$

Note that all MI-based quantities can be written thusly as sums of marginal entropies of subsets of the set of variables. Given a finite set of variables S, its powerset $\mathcal{P}\left(S\right)$ can be assigned a partial ordering as follows:

$$\begin{array}{c}\hfill a\le b\iff a\subseteq b\forall a,b\in \mathcal{P}\left(S\right)\end{array}$$

(13)

This poset $P=\left(\mathcal{P}\right(S),\subseteq )$ is called a Boolean algebra, and because each pair of sets has a unique supremum (their union) and infimum (their intersection), it is a lattice. This lattice structure is visualised for two and three variables in Figure 1. In general, the lattice of an n-variable Boolean algebra forms an n-cube. Furthermore, for any finite n, the n-variable Boolean algebra forms a bounded lattice, which means that it has a greatest element, denoted as $\widehat{1}$, and a least element, denoted as $\widehat{0}$.

On a poset P, we define the Möbius function ${\mu }_P:P\times P\to \mathbb{R}$ as

$$\begin{array}{c}\hfill {\mu }_P(x,y)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}x=y\hfill \\ -{\displaystyle \sum _{z:x\le z<y}}{\mu }_P(x,z)\hfill & \mathrm{if}x<y\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(14)

This function type makes ${\mu }_P$ an element of the incidence algebra of P. In fact, $\mu$ is the inverse of the zeta function $\zeta :\zeta (x,y)=1\mathrm{iff}x\le y$, and 0 otherwise. On a Boolean algebra, such as a powerset ordered by inclusion, the Möbius function takes the simple form $\mu (x,y)={(-1)}^{\mid x\mid -\mid y\mid }$ [36,37]. This definition allows the mutual information among a set of variables $\tau$ to be written as follows [38,39]:

$$\begin{array}{}\mathrm{(15)}& \hfill MI\left(\tau \right)& ={(-1)}^{\mid \tau \mid -1}\sum _{\eta \le \tau }{\mu }_P(\eta ,\tau )H\left(\eta \right)\hfill \mathrm{(16)}& & =\sum _{\eta \le \tau }{(-1)}^{\mid \eta \mid +1}H\left(\eta \right)\hfill \end{array}$$

where P is the Boolean algebra with $\tau =\widehat{1}$ and $H\left(\eta \right)$ is the marginal entropy of the set of variables $\eta$. Indeed, this coincides with Equation (11) for $\tau =\{X,Y\}$ and with Equation (13) for $\tau =\{X,Y,Z\}$. Equation (15) is a convolution known as a Möbius inversion.

Definition 4

(Möbius inversion over a poset, Rota (1964) [37]). Let P be a poset $(S,\le )$, let $\mu :P\times P\to \mathbb{R}$ be the Möbius function from Equation (14), and let $g:P\to \mathbb{R}$ be a function on P. Then, the function

$$\begin{array}{c}\hfill f\left(y\right)=\sum _{x\le y}{\mu }_P(x,y)g\left(x\right)\end{array}$$

(17)

is called the Möbius inversion of g on P. Furthermore, this equation can be inverted to yield

$$\begin{array}{c}\hfill f\left(y\right)=\sum _{x\le y}{\mu }_P(x,y)g\left(x\right)\iff g\left(y\right)=\sum _{x\le y}f\left(x\right)\end{array}$$

(18)

The Möbius inversion is a generalisation of the derivative to posets. If $P=(\mathbb{N},\le )$, Equation (18) is just a discrete version of the fundamental theorem of calculus [36]. Equation (18) additionally implies that we can express joint entropy as a sum over mutual information:

$$\begin{array}{c}\hfill H\left(\tau \right)={(-1)}^{\mid \tau \mid -1}\sum _{\eta \le \tau }MI\left(\eta \right)\end{array}$$

(19)

For example, in the case of three variables,

$$\begin{array}{c}\hfill H(X,Y,Z)=MI(X,Y,Z)+MI(X,Y)+MI(X,Z)+MI(Y,Z)+H\left(X\right)+H\left(Y\right)+H\left(Z\right)\end{array}$$

(20)

Instead of starting with entropy, we could start with a quantity known as surprisal, or self-information, defined as the negative log probability of a certain state or realisation:

$$\begin{array}{cc}\hfill S(X=x)=& -logp(X=x)\hfill \end{array}$$

(21)

Surprisal plays an important role in information theory; indeed, the expected surprisal across all possible realisations $X=x$ is the entropy of the variable X:

$$\begin{array}{c}\hfill {\mathbb{E}}_X\left[S(X=x)\right]=H\left(X\right)\end{array}$$

(22)

As we are often interested in the marginal surprisal of a realisation $X=x$ summed over Y, we can write this explicitly as

$$\begin{array}{c}\hfill logp(x;Y):=\sum _{y}logp(x,y)\end{array}$$

(23)

With this, consider the Möbius inversion of the marginal surprisal over the lattice P:

$$\begin{array}{c}\hfill \mathrm{pmi}(T=\tau ):={(-1)}^{\mid \tau \mid }\sum _{\eta \le \tau }{\mu }_P(\eta ,\tau )logp(\eta ;\tau \backslash \eta )\end{array}$$

(24)

This is a generalised version of the pointwise mutual information, which is usually defined on just two variables:

$$\begin{array}{}\mathrm{(25)}& \hfill \mathrm{pmi}(X=x,Y=y)& =log(x,y;\varnothing )-log(x;Y)-log(y;X)+log(\varnothing ;X,Y)\hfill \mathrm{(26)}& & =log\frac{p(x,y)}{p\left(x\right)p\left(y\right)}\hfill \end{array}$$

Summary

• Mutual information is the Möbius inversion of marginal entropy.
• Pointwise mutual information is the Möbius inversion of marginal surprisal.

3. Interactions and Their Duals

3.1. MFIs as Möbius Inversions

With mutual information defined in terms of Möbius inversions, the same can be done for the model-free interactions. Again, we start with (negative) surprisal. However, on Boolean variables a state is just a partition of the variables into two sets: one in which the variables are set to 1, and another in which they are set to 0. That means that the surprisal of observing a particular state is completely specified by which variables $X\subseteq Z$ are set to 1 while keeping all other variables $Z\backslash X$ at 0, which can be written as

$$\begin{array}{cc}\hfill S_{X;Z}& :=logp(X=1,Z\backslash X=0)\hfill \end{array}$$

(27)

Definition 5 (interactions as Möbius inversions).

Let p be a probability distribution over a set T of random variables and let $P=\left(\mathcal{P}\right(\tau ),\subseteq )$, the powerset of a set $\tau \subseteq T$ ordered by inclusion. Then, the interaction $I(\tau ;T)$ among variables τ is provided by

$$\begin{array}{}\mathrm{(28)}& \hfill I(\tau ;T):& =\sum _{\eta \le \tau }{\mu }_P(\eta ,\tau )S_{\eta ;T}\hfill \mathrm{(29)}& & =\sum _{\eta \le \tau }{(-1)}^{\mid \eta \mid -\mid \tau \mid }logp(\eta =1,T\backslash \eta =0)\hfill \end{array}$$

For example, when $\tau$ contains a single variable $X\subseteq T$, then

$$\begin{array}{}\mathrm{(30)}& \hfill I\left(\right\{X\};T)& ={\mu }_P(\left\{X\right\},\left\{X\right\})S_{\left\{X\right\};T}+{\mu }_P(\varnothing ,\left\{X\right\})S_{\varnothing ;T}\hfill \mathrm{(31)}& & =log\frac{p(X=1,T\backslash X=0)}{p(X=0,T\backslash X=0)}\hfill \end{array}$$

which coincides with the 1-point interaction in Equation (7). Similarly, when $\tau$ contains two variables $\tau =\{X,Y\}\subseteq T$, then

$$\begin{array}{}\mathrm{(32)}& \hfill I(\{X,Y\};T)={\mu }_P(\{X,Y\},\{X,Y\})S_{\{X,Y\};T}+{\mu }_P(\left\{X\right\},\{X,Y\})S_{\left\{X\right\};T} & \hfill +{\mu }_P(\left\{Y\right\},\{X,Y\})S_{\left\{Y\right\};T}+{\mu }_P(\varnothing ,\{X,Y\})S_{\varnothing ;T}\mathrm{(33)}& \hfill =log\frac{p(X=1,Y=1,T\backslash \{X,Y\}=0)p(X=0,Y=0,T\backslash \{X,Y\}=0)}{p(X=1,Y=0,T\backslash \{X,Y\}=0)p(X=0,Y=1,T\backslash \{X,Y\}=0)}\end{array}$$

which coincides with the 2-point interaction in Equation (8). In fact, this pattern holds in general.

Theorem 1 (equivalence of interactions).

The interaction $I(\tau ,T)$ from Definition 5 is the same as the model-free interaction $I_{\tau }$ from Definition 2, that is, for any set of variables $\tau \subseteq T$ it is the case that

$$\begin{array}{c}\hfill I(\tau ,T)=I_{\tau }\end{array}$$

(34)

Proof. 

We have to show that

$$\begin{array}{c}\hfill \sum _{\eta \le \tau }{(-1)}^{\mid \eta \mid -\mid \tau \mid }logp(\eta =1,T\backslash \eta =0)=\frac{{\partial }^{n}logp\left(T\right)}{\partial {\tau }_1\dots \partial {\tau }_n}{|}_{\underline{T}=0}\end{array}$$

(35)

Both sides of this equation are sums of ±$logp\left(s\right)$, where s is some binary string; thus, we have to show that the same strings appear with the same sign.

First, note that the Boolean algebra of sets ordered by inclusion (as in Figure 1) is equivalent to the poset of binary strings where for any two strings a and b, $a\le b\iff a\wedge b=a$. The equivalence follows immediately upon setting each element $a\in \mathcal{P}\left(S\right)$ to the string where $a=1$ and $S\backslash a=0$. This map is one-to-one and monotonic with respect to the partial order, as $A\subseteq B\iff A\cap B=A$. This means that Definition 5 can be rewritten as a Möbius inversion on the lattice of Boolean strings $S=({\mathbb{B}}^{\left|\tau \right|},\le )$ (shown for the three-variable case on the left side of Figure 2):

$$I(\tau ;T)=\sum _{s\le {\widehat{1}}_S}{\mu }_S(s,{\widehat{1}}_S)logp(\tau =s,T\backslash \tau =0)$$

(36)

Note that for any pair ($\alpha ,\tau )$ where $\alpha \subseteq \tau$ with respective string representations $(s,t)\in {\mathbb{B}}^{\left|\tau \right|}\times {\mathbb{B}}^{\left|\tau \right|}$, we have the following:

$$\left|\tau \right|-\left|\alpha \right|=\sum _i{(t\wedge \neg s)}_i$$

(37)

Thus, we can write

$$I(\tau ;T)=\sum _{s\le {\widehat{1}}_S}{(-1)}^{\sum \neg s}logp(\tau =s,T\backslash \tau =0)$$

(38)

To see that this exactly coincides with Definition 2, we can define a map

$$e_{i,s}^{\left(n\right)}:{\mathcal{F}}_{{\mathbb{B}}^n}\to {\mathcal{F}}_{{\mathbb{B}}^{n-1}}$$

(39)

where ${\mathcal{F}}_{{\mathbb{B}}^n}$ is the set of functions from n Boolean variables to $\mathbb{R}$. This map is defined as

$$e_{i,s}^{\left(n\right)}:f(X_1,\dots X_i,\dots X_n)\mapsto f(X_1,\dots X_i=s,\dots X_n)$$

(40)

With this map, the Boolean derivative of a function $f(X_1,\dots ,X_n)$ (see Definition 3) can be written as

$$\frac{\partial }{\partial X_i}f\left(X\right)=(e_{i,1}^{\left(n\right)}-e_{i,0}^{\left(n\right)})f\left(X\right)$$

(41)

$$=f(X_1,\dots ,X_i=1,\dots ,X_n)-f(X_1,\dots ,X_i=0,\dots ,X_n)$$

(42)

In this way, the derivative with respect to a set S of m variables becomes function composition:

$$\left(\prod _{i=0}^m\frac{\partial }{\partial X_{S_i}}\right)f\left(X\right)=\left({◯}_{i=0}^m(e_{S_i,1}^{(n-i)}-e_{S_i,0}^{(n-i)})\right)f\left(X\right)$$

(43)

From this, it is clear that a term $f\left(s\right)$ appears with a minus sign iff $e_{i,0}^{\left(n\right)}$ has been applied an odd number of times. Therefore, terms for which s contains an odd number of 0s receive a minus sign. This can be summarised as

$$\left(\prod _{i=0}^m\frac{\partial }{\partial X_{S_i}}\right)f\left(X\right)=\sum _{s\in {\mathbb{B}}^n}{(-1)}^{\sum \neg s}f(X_S=s,X\backslash X_S)$$

(44)

Therefore, we can write

$$I_{\tau }=\sum _{s\in {\mathbb{B}}^n}{(-1)}^{\sum \neg s}log(\tau =s,T\backslash \tau =0)$$

(45)

The sums ${\sum }_{s\le {\widehat{1}}_S}$ and ${\sum }_{s\in {\mathbb{B}}^n}$ contain exactly the same terms, meaning that Equations (38) and (45) are equal. This completes the proof.    □

Note that the structure of the lattice reveals structure in the interactions, as previously noted in [32]. On the right-hand side of Figure 2, two faces of the three-variable lattice are shaded. The green region corresponds to the 2-point interaction between the first two variables. The red region contains a similar interaction between the first two variables, except this time in the context of the third variable fixed to 1 instead of 0. This illustrates the interpretation of a 3-point interaction as the difference in two 2-point interactions ($I_{XYZ}=I_{XY\mid Z=1}-I_{XY\mid Z=0}$; note that $I_{XY\mid Z=0}$ is usually written as just $I_{XY}$). The symmetry of the cube reveals the three different (though equivalent) choices as to which variable to set to 1. Treating the Boolean algebra as a die, where the sides facing up are ⚀, ⚁, and ⚂, we have

$$I_{XYZ}=⚀-⚅=⚁-⚄=⚂-⚃$$

(46)

As before, we can invert Definition 5 and express the surprise of observing a state with all ones in terms of interactions, as follows:

$$logp(\tau =1,T\backslash \tau =0)=\sum _{\eta \le \tau }I(\eta ,T)$$

(47)

For example, in the case where $T=\{X,Y,Z\}$ and $\tau =\{X,Y\}$

$$S(1,1,0)=-logp(1,1,0)=-I_{XY}-I_X-I_Y-I_{\varnothing }$$

(48)

which illustrates that when X and Y tend to be off ($I_X<0$ and $I_Y<0$) and X and Y tend to be different ($I_{XY}<0$), observing the state (1, 1, 0) is very surprising.

3.2. Categorical Interactions

Taking seriously the definition of interactions as the Möbius inversion of surprisal, one might ask what happens when surprisal is inverted over a different lattice instead of using a Boolean algebra. One example is shown in Figure 3; it corresponds to variables that can take three values—0, 1, or 2—where states are ordered by $a\le b\iff \forall i:a_i\le b_i$. To calculate interactions on this lattice, we need to know the value of Möbius functions of type $\mu (s,22)$. It can be readily verified that most Möbius functions of this type are zero, with the exceptions of $\mu (22,22)=\mu (11,22)=1$ and $\mu (21,22)=\mu (12,22)=-1$, which provide the exact terms in the interactions between two categorical variables changing from $1\to 2$ (as defined in [32]). Calculating interactions on different sublattices with $\widehat{1}=\left(21\right),\left(12\right)$ or $\left(11\right)$ provides us with the other categorical interactions. The transitivity property of the interactions, i.e., $I(X:0\to 2,Y:0\to 1)=I(X:0\to 1,Y:0\to 1)+I(X:1\to 2,Y:0\to 1)$, follows immediately from the structure of the lattice in Figure 3 and the alternating signs of the Möbius functions on a Boolean algebra.

3.3. Information and Interactions on Dual Lattices

Lattices have the property that a set with the reverse order remains a lattice; that is, if $\mathcal{L}=(S,\le )$ is a lattice, then ${\mathcal{L}}^{\mathrm{op}}=(S,⪯)$ (where $\forall a,b\in S:a⪯b\iff a\ge b$) is a lattice. This raises the question of what corresponds to mutual information and interaction on such dual lattices. Recognising that a poset $\mathcal{L}=(S,{\le }_{\mathcal{L}})$ is a category $\mathcal{C}$ with objects S and a morphism $f:A\to B$ iff $B{\le }_{\mathcal{L}}A$, these become definitions in the opposite category ${\mathcal{C}}^{\mathrm{op}}$, meaning that they define dual objects.

Let us start with mutual information. We can calculate the dual mutual information, denoted $M{I}^{*}$, by first noting that the dual to a Boolean algebra is another Boolean algebra, meaning that we have $\mu (x,y)={(-1)}^{\left|x\right|-\left|y\right|}$. Simply replacing P with $P^{\mathrm{op}}$ in Equation (15) yields

$$\begin{array}{cc}\hfill M{I}^{*}\left(\tau \right)& =\sum _{\eta ⪯\tau }{(-1)}^{\left|\eta \right|+1}H\left(\eta \right)\hfill \end{array}$$

(49)

The dual mutual information of $\tau ={\widehat{1}}_{P^{\mathrm{op}}}$ is simply $M{I}^{*}(\varnothing )=MI\left({\widehat{1}}_P\right)$, that is, the mutual information among all variables. However, the dual mutual information of a singleton set X is

$$\begin{array}{}\mathrm{(50)}& \hfill M{I}^{*}\left(X\right)& =MI\left({\widehat{1}}_P\right)-MI({\widehat{1}}_P\backslash X)\hfill \mathrm{(51)}& & =\Delta (X;{\widehat{1}}_P\backslash X)\hfill \end{array}$$

where $\Delta$ is known as the differential mutual information and describes the change in mutual information when leaving out X [40], i.e., when marginalising over the variable X. Note that a similar construction was already anticipated in [41] and that the differential mutual information has previously been used to describe information structures in genetics [39]. On the Boolean algebra of three variables $\{X,Y,Z\}$, the dual mutual information of X can be written out as follows:

$$\begin{array}{} & \hfill M{I}^{*}\left(X\right)=\mu (\left\{X\right\},\left\{X\right\})H\left(X\right)+\mu (\{X,Y\},\left\{X\right\})H(X,Y)+\mathrm{(52)}& \hfill \mu \left(\right\{X,Z\},\{X\left\}\right)H(X,Z)+\mu \left(\right\{X,Y,Z\},\{X\left\}\right)H(X,Y,Z)\mathrm{(53)}& \hfill =H\left(X\right)-H(X,Y)-H(X,Z)+H(X,Y,Z)\end{array}$$

Because $\Delta$ is the dual of mutual information, it should arguably be called the mutual co-information; however, the term co-information is unfortunately already in use to refer to normal higher-order mutual information.

To find the dual to the interactions, we start from Equation (36) and construct $S^{\mathrm{op}}=({\mathbb{B}}^{\left|\tau \right|},⪯)$, the dual to the lattice of binary strings $S=({\mathbb{B}}^{\left|\tau \right|},\le )$. A dual interaction of variables $\tau \subseteq T$ is denoted as $I^{*}(\tau ;T)$, and is defined as follows:

$$I^{*}(\tau ;T):=\sum _{s⪯{\widehat{1}}_{S^{\mathrm{op}}}}{\mu }_{S^{\mathrm{op}}}(s,{\widehat{1}}_{S^{\mathrm{op}}})logp(\tau =s,T\backslash \tau =0)$$

(54)

Again, when $\tau ={\widehat{1}}_{S\mathrm{op}}={\widehat{0}}_S=\varnothing$, this is simply (${-1)}^{\mid \tau \mid }I\left({\widehat{1}}_S\right)$, while the dual interaction of a singleton set X is

$$I^{*}(X;T)={(-1)}^{|{\widehat{1}}_S|-1}\left(I({\widehat{1}}_S;T)+I({\widehat{1}}_S\backslash X;T)\right)$$

(55)

For example, on the three variable lattice in Figure 2, the dual interaction of X is

$$I^{*}(X;T)=I(X,Y,Z;T)+I(Y,Z;T)$$

(56)

Writing $p_{ijk}$ for $p(X=i,Y=j,Z=k\mid T\backslash \{X,Y,Z\}=0)$, it can be seen that this is equal to

$$I^{*}(X;T)=log\frac{p_{111}{p}_{100}}{p_{101}{p}_{110}}$$

(57)

which is similar to the 2-point interaction $I_{YZ}$ defined in Equation (8), now conditioned on $X=1$ instead of 0. Note the difference between dual mutual information and dual interactions here; the dual mutual information of X describes the effect on the mutual information from marginalising over X, whereas the dual interaction of X describes the effect on an interaction when fixing$X=1$. This reflects a fundamental difference between mutual information and the interactions, in that the former is an averaged quantity and the latter a pointwise quantity.

Dual interactions should probably be called co-interactions; however, to avoid confusion with the term co-information, we instead refer to them simply as dual interactions. Dual interactions are interactions that are conditioned on certain variables being 1 instead of 0. This makes them no longer equal to the Ising interactions between Boolean variables; however, there are situations in which an interaction is more interesting in the context of $Z=1$ instead of $Z=0$, for example, if Z is always 1 in the data under consideration.

Summary

• Mutual information is the Möbius inversion of marginal entropy on the lattice of subsets ordered by inclusion.
• Differential (or conditional) mutual information is the Möbius inversion of marginal entropy on the dual lattice.
• Model-free interactions are the Möbius inversion of surprisal on the lattice of subsets ordered by inclusion.
• Model-free dual interactions are the Möbius inversion of surprisal on the dual lattice.
• Dual interactions of a variable X are interactions between the other variables where X is set to 1 instead of 0.

To summarise these relationships diagrammatically, note that surprisals form a vector space as follows. Let $\mathcal{P}\left(T\right)$ be the powerset of a set of variables T and let $\left|\mathcal{P}\right(T\left)\right|=n$. This forms a lattice $P=\left(\mathcal{P}\right(T),\subseteq )$ ordered by inclusion, meaning that $\mathcal{P}\left(T\right)$ can be assigned a topological ordering indexed by i as $\mathcal{P}\left(T\right)={\cup }_{i=0}^n{t}_i$. Let $\mathcal{S}$ be the set of linear combinations of surprisals of subsets of T:

$$\begin{array}{c}\hfill \mathcal{S}=\left\{\sum _{i=0}^n{a}_{i}logp\left(t_i\right)\mid a_i\in \mathbb{R}\right\}\end{array}$$

(58)

This set is assigned a vector space structure over $\mathbb{R}$ by the usual scalar multiplication and addition. Note that the set

$$\begin{array}{c}\hfill \mathcal{B}=\left\{logp\left(t\right)\mid t\in \mathcal{P}\left(T\right)\right\}\end{array}$$

(59)

forms a basis for this vector space, because ${\sum }_i{\alpha }_{i}logp\left(t_i\right)=0$ has no non-trivial solutions and a $\mathrm{span}\left(\mathcal{B}\right)=\mathcal{S}$. Only when two variables a and b are independent do we have linear dependencies in $\mathcal{B}$, as it is then the case that $logp(a,b)=logp\left(a\right)+logp\left(b\right)$. To define a map from $\mathcal{S}\to \mathbb{R}$, we only need to specify its action on $\mathcal{B}$ and extend the definition linearly. This means that we can fully define the map ${\mathit{eval}}_T:\mathcal{S}\to \mathbb{R}$ by specifying

$$\begin{array}{cc}\hfill {\mathit{eval}}_T:logp(R=r)& \mapsto logp(R=1,T\backslash R=0)\hfill \end{array}$$

(60)

Similarly, we can define the expectation map $\mathbb{E}:\mathcal{S}\to \mathbb{R}$ as

$$\begin{array}{cc}\hfill \mathbb{E}:logp(R=r)& \mapsto \sum _{r}p(R=r)logp(R=r)\hfill \end{array}$$

(61)

which outputs the expected surprise over all realisations $R=r$. Finally, note that the Möbius inversion over a poset P is an endomorphism of the set ${\mathcal{F}}_P$ of functions over P, defined as

$$\begin{array}{}\mathrm{(62)}& \hfill M_P& :{\mathcal{F}}_P\to {\mathcal{F}}_P\hfill \mathrm{(63)}& \hfill M_P& :f\left(y\right)\mapsto \sum _{x\le y}\mu (x,y)f\left(x\right)\hfill \end{array}$$

Together, these three maps ensure that the following diagram commutes:

For the case where $T=\{X,Y,Z\}$ and $R=\{X,Y\}$, this explicitly amounts to

4. Results and Examples

While mutual information and model-free interactions are related, there are several important differences in terms of how they capture dependencies. Note, for example, that higher-order information quantities are not independent of the lower-order quantities. The mutual information of three variables is bounded by the pairwise quantities as follows:

$$\begin{array}{c}\hfill -min\left\{MI\right(X,Y\mid Z),MI(Y,Z\mid X),MI(X,Z\mid Y\left)\right\}\le MI(X,Y,Z)\le min\left\{MI\right(X,Y),MI(Y,Z),MI(X,Z\left)\right\}\end{array}$$

(64)

This means that there are no systems with zero pairwise mutual information and positive higher-order information. This is not true for the interactions. For example, a distribution with 3-point interactions and no pairwise interactions can trivially be constructed as $p\left(X\right)={\mathcal{Z}}^{-1}exp\left({\sum }_{ijk}{J}_{ijk}{X}_i{X}_j{X}_k\right)$. While this distribution has 3-point interactions with strength $J_{ijk}$ for triplets $\{X_i,X_j,X_k\}$, all pairwise interactions among $\{X_i,X_j\}$ vanish when conditioning on $X_k=0$. In fact, any positive discrete distribution can be written as a Boltzmann distribution with an energy function that is unique up to a constant, and as such is uniquely defined by its interactions; in other words, each interaction, at any order, can be freely varied to define a unique and valid probability distribution, namely, the Boltzmann distribution of the corresponding generalised Ising model. Note that this is closely related to the fact that a class of neural networks known as restricted Boltzmann machines are universal approximators [42,43,44] and exactly (though not uniquely) encode the Boltzmann distribution of a generalised Ising model in one of their layers [31,45]. Therefore, each distribution is uniquely determined by its set of interactions, and should be distinguishable by them. This is famously not true for entropy-based information quantities, as illustrated below through several examples.

4.1. Interactions and Their Duals Quantify and Distinguish Synergy in Logic Gates

Under the assumption of a causal collider structure $A\to C\leftarrow B$, nonzero 3-point interactions $I_{ABC}$ can be interpreted as logic gates. A positive 3-point interaction means that the numerator in Equation (10) is larger than the denominator. Under the sufficient (though not necessary) assumption that each term in the numerator is larger than each term in the denominator, we obtain the following truth table as $I_{ABC}\to +\infty$:

A

B

C

0	0	1
0	1	0
1	0	0
1	1	1

which describes an XNOR gate. Let $p_{\mathcal{G}}$ be the probability of each of the four states in the truth table for a gate $\mathcal{G}$, and let ${ϵ}_{\mathcal{G}}$ be the probability of all other states. Then, the 3-point interaction of an XNOR gate can be written as

$$I_{ABC}^{\mathrm{XNOR}}=log\frac{p_{\mathrm{XNOR}}^4}{{ϵ}_{\mathrm{XNOR}}^4}$$

(65)

Similarly, the truth tables of AND and OR gates imply that

$$\begin{array}{}\mathrm{(66)}& \hfill I_{ABC}^{\mathrm{AND}}& =log\frac{{ϵ}_{\mathrm{AND}}{p}_{\mathrm{AND}}^3}{{ϵ}_{\mathrm{AND}}^3{p}_{\mathrm{AND}}}\hfill \mathrm{(67)}& \hfill I_{ABC}^{\mathrm{OR}}& =log\frac{{ϵ}_{\mathrm{OR}}^3{p}_{\mathrm{OR}}}{{ϵ}_{\mathrm{OR}}{p}_{\mathrm{OR}}^3}\hfill \end{array}$$

If we consider equally noisy gates such that $p_{\mathcal{G}}=p$ and ${ϵ}_{\mathcal{G}}=ϵ$, the gates can be directly compared. Note that when a gate has a 3-point interaction I, its logical negation will have a 3-point interaction $-I$. This determines the 3-point interactions of all six non-trivial logic gates on two inputs, as summarised in

Table 1. The 3-point interactions for all two-input logic gates at equal noise level are related through $I=4log\frac{p}{ϵ}$ and degenerate in AND∼NOR and OR∼NAND.

$\mathcal{G}$	${\mathit{I}}_{\mathbf{ABC}}^{\mathcal{G}}$
XNOR	I
XOR	$-I$
AND	$\frac{1}{2}I$
OR	$-\frac{1}{2}I$
NAND	$-\frac{1}{2}I$
NOR	$\frac{1}{2}I$

. The two gates with the strongest absolute interactions, XNOR and XOR, are the only two gates that are purely synergistic, i.e., knowing only one of the two inputs provides no information about the output. This relationship to synergy holds for three-input gates as well. The three-input gate with the strongest 4-point interaction has the following truth table:

A

B

C

D

0	0	0	0
0	0	1	1
0	1	0	1
1	0	0	1
0	1	1	0
1	0	1	0
1	1	0	0
1	1	1	1

This is a three-input XOR gate, i.e., $D=(A+B+C)mod2$, and is again maximally synergistic, as observing only two of the three inputs provides zero bits of information on the output. Setting this maximum 4-point interaction to I, the three-input OR and AND gates receive a 4-point interaction $I/4$; thus, the hierarchies of interaction and synergy continue to match.

The 3-point interactions are able to separate most two-input logic gates by sign or value, leaving only AND∼NOR and OR∼NAND. Mutual information has less resolving power. Assuming a uniform distribution over all four allowed states from a gate’s truth table, a brief calculation yields

$$\begin{array}{}\mathrm{(68)}& M{I}^{\mathrm{OR}}(A,B,C)=M{I}^{\mathrm{AND}}(A,B,C)=M{I}^{\mathrm{NOR}}(A,B,C)=M{I}^{\mathrm{NAND}}(A,B,C)\hfill & =-log\left(\frac{3^{3/4}}{4}\right)-1\approx -0.189\hfill \mathrm{(69)}& M{I}^{\mathrm{XOR}}(A,B,C)=M{I}^{\mathrm{XNOR}}(A,B,C)=-1\hfill \end{array}$$

That is, higher-order mutual information resolves strictly fewer logical gates by value and none by sign. In fact, the higher-order mutual information of a logic gate can never be positive, because it is bounded from above by the minimum of the pairwise mutual information, which is always zero for the pair of inputs. Because all entropy-based quantities inherit the degeneracy summarised in

Table 2. The marginal entropies of variables in a logic gate are degenerate in XOR∼XNOR and AND∼OR∼NAND∼NOR.

$\mathcal{G}$	$\begin{array}{c}\mathit{H}\left(\mathit{A}\right)\\ =\mathit{H}\left(\mathit{B}\right)\end{array}$	$\mathit{H}\left(\mathit{C}\right)$	$\mathit{H}(\mathit{A},\mathit{B})$	$\begin{array}{c}\mathit{H}(\mathit{A},\mathit{C})\\ =\mathit{H}(\mathit{B},\mathit{C})\end{array}$	$\mathit{H}(\mathit{A},\mathit{B},\mathit{C})$
XNOR	1	1	2	2	2
XOR	1	1	2	2	2
AND	1	$log\frac{3^{3/4}}{4}$	2	$\frac{3}{2}$	2
OR	1	$log\frac{3^{3/4}}{4}$	2	$\frac{3}{2}$	2
NAND	1	$log\frac{3^{3/4}}{4}$	2	$\frac{3}{2}$	2
NOR	1	$log\frac{3^{3/4}}{4}$	2	$\frac{3}{2}$	2

, neither the mutual information nor its dual can increase the resolving power (see

Table 3. While the interactions leave certain gates indistinguishable, the dual J-interactions of the inputs are unique to each gate. The reported decimal values are rounded to three digits; as before, $I=4log\frac{p}{ϵ}$.

$\mathcal{G}$	${\mathit{MI}}_{\mathit{ABC}}$	${\mathit{MI}}_{\mathit{BC}}$	${\mathit{MI}}_{\mathit{A}}^{*}$	${\mathit{I}}_{\mathit{ABC}}^{\mathcal{G}}$	${\mathit{I}}_{\mathit{AB}}^{\mathcal{G}}$	${\mathit{I}}_{\mathit{BC}}^{\mathcal{G}}$	${\mathit{I}}_{\mathit{A}}^{*\mathcal{G}}$	${\mathit{I}}_{\mathit{C}}^{*\mathcal{G}}$	${\mathit{J}}_{\mathit{A}}^{*\mathcal{G}}$	${\mathit{J}}_{\mathit{C}}^{*\mathcal{G}}$	${\overline{\mathit{J}}}^{*\mathcal{G}}$
XNOR	$-1$	0	$-1$	I	$-\frac{1}{2}I$	$-\frac{1}{2}I$	$\frac{1}{2}I$	$\frac{1}{2}I$	$\frac{3}{2}I$	$\frac{3}{2}I$	$\frac{27}{8}{I}^3$
XOR	$-1$	0	$-1$	$-I$	$\frac{1}{2}I$	$\frac{1}{2}I$	$-\frac{1}{2}I$	$-\frac{1}{2}I$	$-\frac{3}{2}I$	$-\frac{3}{2}I$	$-\frac{27}{8}{I}^3$
AND	$-0.189$	0.311	$-\frac{1}{2}$	$\frac{1}{2}I$	$-\frac{1}{4}I$	0	$\frac{1}{2}I$	$\frac{1}{4}I$	$\frac{1}{2}I$	$\frac{3}{4}I$	$\frac{3}{16}{I}^3$
OR	$-0.189$	0.311	$-\frac{1}{2}$	$-\frac{1}{2}I$	$\frac{1}{4}I$	$\frac{1}{2}I$	0	$-\frac{1}{4}I$	$-I$	$-\frac{3}{4}I$	$-\frac{3}{4}{I}^3$
NAND	$-0.189$	0.311	$-\frac{1}{2}$	$-\frac{1}{2}I$	$\frac{1}{4}I$	0	$-\frac{1}{2}I$	$-\frac{1}{4}I$	$-\frac{1}{2}I$	$-\frac{3}{4}I$	$-\frac{3}{16}{I}^3$
NOR	$-0.189$	0.311	$-\frac{1}{2}$	$\frac{1}{2}I$	$-\frac{1}{4}I$	$-\frac{1}{2}I$	0	$\frac{1}{4}I$	I	$\frac{3}{4}I$	$\frac{3}{4}{I}^3$

).

The logic gate interactions and their duals are summarised in Table 3, where it can be seen that neither $I_C^{*\mathcal{G}}=I_{ABC}^{\mathcal{G}}+I_{AB}^{\mathcal{G}}$ nor $I_A^{*\mathcal{G}}$ improve the resolution beyond that of the 3-point interaction. However, the 3-point interaction requires $2^3=8$ probabilities to achieve this resolving power, whereas $I_C^{*\mathcal{G}}=\frac{p_{111}{p}_{001}}{p_{101}{p}_{011}}$ achieves the same resolving power with just four probabilities.

However, note that because of a difference in sign convention dual mutual information is a difference between two mutual information quantities, while dual interactions are a sum of two interactions. Based on this, we can consider the difference of two interactions and define a new quantity $J_A^{*\mathcal{G}}=I_{ABC}^{\mathcal{G}}-I_{BC}^{\mathcal{G}}$. We refer to this as a J-interaction. When the MFIs are interpreted in the context of an energy-based model, such as an Ising model or a restricted Boltzmann machine, then the interactions have dimensions of energy, meaning that the J-interactions correspond to the difference in the energy contribution between a triplet and a pair. These J-interactions of the input nodes A and B assign a different value to each logic gate $\mathcal{G}$, and the symmetric J-interaction ${\overline{J}}^{*\mathcal{G}}=J_A^{*\mathcal{G}}{J}_B^{*\mathcal{G}}{J}_C^{*\mathcal{G}}$, analogous to the symmetric deltas from [40], inherits the perfect resolution from $J_A^{*\mathcal{G}}$.

Note that while $J_A^{*\mathcal{G}}=J_B^{*\mathcal{G}}$ both have perfect resolution, $J_C^{*\mathcal{G}}=I_{ABC}^{\mathcal{G}}-I_{AB}^{\mathcal{G}}$ does not improve the resolution beyond that of the 3-point interaction. This results from the fact that in logic gates we have $I_{ABC}^{\mathcal{G}}=-2I_{AB}^{\mathcal{G}}$, meaning that $I_{ABC}^{\mathcal{G}}$ and $I_{AB}^{\mathcal{G}}$ contain the same information. To see this, note that

$$I_{ABC}^{\mathcal{G}}+2I_{AB}^{\mathcal{G}}=log\frac{p_{111}{p}_{001}{p}_{110}{p}_{000}}{p_{101}{p}_{011}{p}_{010}{p}_{100}}$$

(70)

Because the logic gates are symmetric in their inputs, i.e., $\forall i,j{p}_{ijk}=p_{jik}$, this can be rewritten as

$$I_{ABC}^{\mathcal{G}}+2I_{AB}^{\mathcal{G}}=log\frac{\left(p_{111}{p}_{110}\right)\left(p_{001}{p}_{000}\right)}{\left(p_{101}{p}_{100}\right)\left(p_{101}{p}_{100}\right)}$$

(71)

Each of these terms in brackets has the form $\left(p_{ij1}{p}_{ij0}\right)$. Because these are two contradicting states, this product reduces to $ϵp$ regardless of the truth table of $\mathcal{G}$:

$$I_{ABC}^{\mathcal{G}}+2I_{AB}^{\mathcal{G}}=log\frac{{ϵ}^2{p}^2}{{ϵ}^2{p}^2}=0$$

(72)

Note that this pattern could already be observed in Table 3, though it was not yet explained.

Thus, the J-interactions of the input nodes uniquely assign a value to each gate proportional to the synergy of its logic. The hierarchy is $J_A^{*\mathrm{XNOR}}>J_A^{*\mathrm{NOR}}>J_A^{*\mathrm{AND}}$, which is mirrored for the respective logical complements. XNOR is indeed the most synergistic, while NOR is more synergistic than AND with respect to observing a 0 in one of the inputs; in a NOR gate, a 0 in the input provides no information on the output, while it completely fixes the output of an AND gate. Because the interactions are defined in a context of 0s, they order the synergy accordingly.

4.2. Interactions Distinguish Dynamics and Causal Structures

To illustrate how different association metrics reflect the underlying causal dynamics, consider data generated from a selection of three-node causal DAGs as follows. On a given DAG $\mathcal{G}$, first denote the set of nodes without parents, the orphan nodes, by $S_0$. Each orphan node in $S_0$ receives a random value drawn from a Bernoulli distribution, i.e., $P(X=1)=p$ and $P(X=0)=1-p$. Next, denote the set of children of orphan nodes as $S_1$. Each node in $S_1$ is then set to either the product of its parent nodes (for multiplicative dynamics) or the mean of its parent nodes (for additive dynamics), plus some zero-mean Gaussian noise with variance ${\sigma }^2$. Note that for the fork and the chain this simply amounts to a noisy copying operation. All nodes are then rounded to a 0 or 1. A set $S_2$ is then defined as the set of all children of nodes in $S_1$, and these receive values using the same dynamics as before. As long as the causal structure is acyclic, this algorithm terminates on a set of nodes $S_i$ that has no children. For example, the chain graph $A\to B\to C$ has $S_0=\left\{A\right\}$, $S_1=\left\{B\right\}$, $S_2=\left\{C\right\}$, and $S_3=\varnothing$, at which point the updating terminates.

Figure 4 shows the results for four different DAGs with multiplicative and additive dynamics (though these are the same for forks and chains). The six different dynamics are represented in four different DAGs, two different (Pearson) correlations, four different partial correlations, and two different mutual information structures, which means that each of these descriptions is degenerate in some of the dynamics. The shown pairwise partial correlations are the correlations among the residuals after a linear regression against the third variable. Because this is similar to conditioning on the third variable, it is somewhat analogous to the MFIs; in fact, when the variables are multivariate normal the partial correlations are encoded in the inverse covariance matrix and are equivalent to pairwise Ising interactions [31]. Indeed, it can be seen that the partial correlations are somewhat able to disentangle direct effects from indirect effects, although they fail to distinguish additive from multiplicative dynamics. Note that only the sign of the association and its significance are represented, as the precise value depends on the noise level ${\sigma }^2$. The rightmost column shows that the MFIs assign a unique association structure to each of the dynamics, distinguish between direct and indirect effects, and reveal multiplicative dynamics as a 3-point interaction while identifying additive dynamics as a purely pairwise process. Finally, note that both the partial correlation and the MFIs assign a negative association to the parent nodes in a collider structure. This reflects that two nodes become dependent when conditioned on a common effect (cf. Berkson’s paradox), a phenomenon already found in partial correlations of metabolomic data in [46]. The mutual information is affected by Berkson’s paradox as well, revealed through the negative three-point mutual information. This negative three-point is a direct effect from conditioning on the common effect C, as on colliders $MI(A,B,C)=MI(A,B)-MI(A,B\mid C)=-MI(A,B\mid C)$, because the mutual information among the independent inputs A and B vanishes by definition.

4.3. Higher-Order Categorical Interactions Distinguish Dyadic and Triadic Distributions

That the interactions have such resolving power over distributions of binary variables is perhaps not very surprising in light of the universality of RBMs with respect to this class of distributions. More surprisingly, their resolving power extends to the case of categorical variables. In [47], the authors introduced two distributions, the dyadic and triadic distributions, which are indistinguishable by almost all commonly used information measures (i.e., Shannon, Renyi(2), residual, and Tsallis entropy, co-information, total correlation, CAEKL mutual information, interaction information, Wyner, exact, functional, and MSS common information, perplexity, disequilibrium, and LMRP and TSE complexities).

Figure 4. Different causal dynamics lead to different association metrics. Green edges denote positive values, red edges denote negative values, circles denote a three-point quantity, and dashed lines show edges with marginal significance (depending on ${\sigma }^2$). Correlations and mutual information cannot distinguish between most dynamics, and while partial correlation can identify the correct pairwise relationships for certain noise levels, it falls short of distinguishing additive from multiplicative dynamics. Only MFIs can distinguish between all six scenarios and reveal the combinatorial effect of the multiplicative dynamics as a 3-point interaction. See Appendix A.3 for the simulation parameters and raw numbers. This figure is reproduced with permission from the author of [48].

Figure 4. Different causal dynamics lead to different association metrics. Green edges denote positive values, red edges denote negative values, circles denote a three-point quantity, and dashed lines show edges with marginal significance (depending on ${\sigma }^2$). Correlations and mutual information cannot distinguish between most dynamics, and while partial correlation can identify the correct pairwise relationships for certain noise levels, it falls short of distinguishing additive from multiplicative dynamics. Only MFIs can distinguish between all six scenarios and reveal the combinatorial effect of the multiplicative dynamics as a 3-point interaction. See Appendix A.3 for the simulation parameters and raw numbers. This figure is reproduced with permission from the author of [48].

The two distributions are defined on three variables, each taking a value in a four-letter alphabet $\{0,1,2,3\}$. The joint probabilities are summarised in

Table 4. The joint probability of the dyadic and triadic distributions [47]. All other states have a probability of zero.

Dyadic
X	Y	Z	P
0	0	0	1 / 8
0	2	1	1 / 8
1	0	2	1 / 8
1	2	3	1 / 8
2	1	0	1 / 8
2	3	1	1 / 8
3	1	2	1 / 8
3	3	3	1 / 8
Triadic
X	Y	Z	P
0	0	0	1 / 8
1	1	1	1 / 8
0	2	2	1 / 8
1	3	3	1 / 8
2	0	2	1 / 8
3	1	3	1 / 8
2	2	0	1 / 8
3	3	1	1 / 8

. To construct the distributions, each category is represented as a binary string ($\{0,1,2,3\}\to \{00,01,10,11\}$), leading to new variables $\{X_0,X_1,Y_0,Y_1,Z_0,Z_1\}$. The dyadic distribution is constructed by linking these new variables with pairwise rules $X_0=Y_1,Y_0=Z_1,Z_0=X_1$, while the triadic distribution is constructed with triplet rules $X_0+Y_0+Z_0=0mod2$ and $X_1=Y_1=Z_1$. The resulting binary strings are then reinterpreted as categorical variables to produce Table 4.

The authors of [47] found that no Shannon-like measure can distinguish between the two distributions, and argued that the partial information decomposition, which is different for the two distributions, is not a natural information measure, as it has to single out one of the variables as an output. To calculate model-free categorical interactions between the variables, we can set the probabilities of the states in Table 4 uniformly to $p=(1-(64-8\left)ϵ\right)/8$ and those of the other states to $ϵ$ (i.e., a normalised uniform distribution over legal states). There are a total of $6^3=216$ interactions such that $x_1>x_0,y_1>y_0,z_1>z_0$. Each of these can be written as

$$\begin{array}{cc}\hfill & I_{XYZ}(x_0\to x_1;y_0\to y_1;z_0\to z_1)=\hfill \\ \hfill log& \frac{p\left(X=x_1,Y=y_1,Z=z_1\mid \underline{X}=0\right)}{p\left(X=x_0,Y=y_0,,Z=z_0\mid \underline{X}=0\right)}\frac{p\left(X=x_1,Y=y_0,Z=z_0\mid \underline{X}=0\right)}{p\left(X=x_0,Y=y_1,,Z=z_1\mid \underline{X}=0\right)}\hfill \\ \hfill \times & \frac{p\left(X=x_0,Y=y_1,Z=z_0\mid \underline{X}=0\right)}{p\left(X=x_1,Y=y_0,,Z=z_1\mid \underline{X}=0\right)}\frac{p\left(X=x_0,Y=y_0,Z=z_1\mid \underline{X}=0\right)}{p\left(X=x_1,Y=y_1,,Z=z_0\mid \underline{X}=0\right)}\hfill \end{array}$$

(73)

Of particular interest here are the two quantities $I_{XYZ}(0\to 3;0\to 3;0\to 3)$ and ${\overline{I}}_{XYZ}={\sum }_{x_0,x_1,y_0,y_1,z_0,z_1}{I}_{XYZ}(x_0\to x_1;y_0\to y_1;z_0\to z_1)$, where the sum is over all values such that $x_1>x_0,y_1>y_0,z_1>z_0$, as all possible pairs necessarily sum to zero because $I_{XYZ}(x_0\to x_1;y_0\to y_1;z_0\to z_1)=-I_{XYZ}(x_1\to x_0;y_0\to y_1;z_0\to z_1)$. For the dyadic distribution, we have

$$I_{XYZ}^{\mathrm{Dy}}(0\to 3;0\to 3;0\to 3)=log\frac{p{ϵ}^3}{p{ϵ}^3}=0,$$

(74)

while for the triadic distribution we have

$$I_{XYZ}^{\mathrm{Tri}}(0\to 3;0\to 3;0\to 3)=log\frac{{ϵ}^4}{p{ϵ}^3}=log\frac{ϵ}{p}$$

(75)

Thus, this particular 3-point interaction is zero for the dyadic distribution and negative for the triadic distribution. The sum over all three points (see Appendix A.4 for details) is provided by

$$\begin{array}{}\mathrm{(76)}& \hfill {\overline{I}}_{XYZ}^{\mathrm{Dy}}& =log1=0\hfill \mathrm{(77)}& \hfill {\overline{I}}_{XYZ}^{\mathrm{Tri}}& =64log\frac{ϵ}{p}\hfill \end{array}$$

That is, the additively symmetrised 3-point interaction is zero for the dyadic distribution and strongly negative for the triadic distribution. These two distributions, which are indistinguishable in terms of their information structure, are distinguishable by their model-free interactions, which accurately reflect the higher-order nature of the triadic distribution.

5. Discussion

In this paper, we have related the model-free interactions introduced in [32] to information theory by defining them as Möbius inversions of surprisal on the same lattice that relates mutual information to entropy. We then invert the order of the lattice and compute the order-dual to the mutual information, which turns out to be a generalisation of differential mutual information. Similarly, the order-dual of interaction turns out to be interaction in a different context. Both the interactions and the dual interactions are able to distinguish all six logic gates by value and sign. Moreover, their absolute strength reflects the synergy within the logic gate. In simulations, the interactions were able to perfectly distinguish six kinds of causal dynamics that are partially indistinguishable to Pearson/partial correlations, causal graphs, and mutual information. Finally, we considered dyadic and triadic distributions constructed using pairwise and higher-order rules, respectively. While these two distributions are indistinguishable in terms of their Shannon information, they have different categorical MFIs that reflect the order of the construction rules.

One might wonder why the interactions enjoy this advantage over entropy-based quantities. The most obvious difference is that the interactions are defined in a pointwise way, i.e., in terms of the surprisal of particular states, whereas entropy is the expected surprisal across an ensemble of states. Furthermore, the MFIs can be interpreted as interactions in an Ising model and as effective couplings in a restricted Boltzmann machine. As both these models are known to be universal approximators with respect to positive discrete probability distributions, the MFIs should be able to characterise all such distributions. What is not immediately obvious is that the kinds of interactions that characterise a distribution should reflect properties of that distribution, such as the difference between direct and indirect effects and the presence of higher-order structure. However, in the various examples covered in this manuscript the interactions turn out to intuitively align with properties of the process used to generate the data. While the stringent conditioning on variables not considered in the interaction might make it tempting to interpret an MFI as a causal or interventional quantity, it is important to be very careful when doing this. Assigning a causal interpretation to statistical inferences, whether in Pearl’s graphical do-calculus [49] or in Rubin’s potential outcomes framework [50], requires further (often untestable) assumptions and analysis of the system in order to determine whether a causal effect is identifiable and which variables to control for. In contrast, an MFI is simply defined by conditioning on all observed variables, makes no reference to interventions or counterfactuals, and does not specify a direction of the effect. While in a controlled and simple setting the MFIs can be expressed in terms of causal average treatment effects [32], a causal interpretation is not justifiable in general.

Moreover, the stringency in the conditioning might worry the attentive reader. Estimating $logp(X=1,Y=1,T=0)$ directly from data means counting states such as $(X,Y,T_1,T_2,\dots ,T_N)=(1,1,0,0,\dots 0)$, which for sufficiently large N are rare in most datasets. Appendix A.1 shows how to use the causal graph to construct Markov blankets, making such estimation tractable when full conditioning is too stringent. In an upcoming paper, we address this issue by estimating the graph of conditional dependencies, allowing for successful calculation of MFIs up to the fifth order in gene expression data.

One major limitation of MFIs is that they are only defined on binary or categorical variables, whereas many other association metrics are defined for ordinal and continuous variables as well. As states of continuous variables no longer form a lattice, it is hard to see how the definition of MFIs could be extended to include these cases.

Finally, it is worth noting that the structure of different lattices has guided much of this research. That Boolean algebras are important in defining higher-order structure is not surprising, as they are the stage on which the inclusion–exclusion principle can be generalised [36]. However, it is not only their order-reversed duals that lead to meaningful definitions; completely unrelated lattices do as well. For example, the Möbius inversion on the lattice of ordinal variables from Figure 3 and the redundancy lattices in the partial information decomposition [28] both lead to new and sensible definitions of information-theoretic quantities. Furthermore, the notion of Möbius inversion has been generalised to a more general class of categories [51], of which posets are a special case. A systematic investigation of information-theoretic quantities in this richer context would be most interesting.