Entropy as a Topological Operad Derivation

Abstract

We share a small connection between information theory, algebra, and topology—namely, a correspondence between Shannon entropy and derivations of the operad of topological simplices. We begin with a brief review of operads and their representations with topological simplices and the real line as the main example. We then give a general definition for a derivation of an operad in any category with values in an abelian bimodule over the operad. The main result is that Shannon entropy defines a derivation of the operad of topological simplices, and that for every derivation of this operad there exists a point at which it is given by a constant multiple of Shannon entropy. We show this is compatible with, and relies heavily on, a well-known characterization of entropy given by Faddeev in 1956 and a recent variation given by Leinster.

1. Introduction

In this article, we describe a simple connection between information theory, algebra, and topology. To motivate the idea, consider the function $d:[0,1]\to \mathbb{R}$ defined by

$$d\left(x\right)=\left\{\begin{array}{cc}-xlogx\hfill & \mathrm{if}x>0,\hfill \\ 0\hfill & \mathrm{if}x=0.\hfill \end{array}\right.$$

This map satisfies an equation reminiscent of the Leibniz rule from Calculus, $d\left(xy\right)=d\left(x\right)y+xd\left(y\right)$ for all $x,y\in [0,1]$. In other words, d is a nonlinear derivation [1], (Lemma 2.2.6). This derivation may also bring to mind the Shannon entropy of a probability distribution. Indeed, a probability distribution on a finite set $\{1,\dots ,n\}$ for $n\ge 1$ is a tuple of nonnegative real numbers $p=(p_1,\dots ,p_n)$ satisfying ${\sum }_{i=1}^n{p}_i=1$, and the Shannon entropy of p is defined to be

$$H\left(p\right)=-\sum _{i=1}^n{p}_{i}log{p}_i=\sum _{i=1}^{n}d\left(p_i\right).$$

Although d is not linear, this may prompt one to wonder about settings in which Shannon entropy itself is a derivation. We describe one such setting below by showing a correspondence between Shannon entropy and derivations of the operad of topological simplices.

1.1. Motivation

As evidenced by recent work, the intersection of information theory and algebraic topology is fertile ground. In 2015 tools of information cohomology were introduced in [2] by Baudot and Bennequin who construct a certain cochain complex for which entropy represents the unique cocycle in degree 1. In the same year, Elbaz-Vincent and Gangl approached entropy from an algebraic perspective and showed that what are known as information functions of degree 1 behave “a lot like certain derivations” [3]. A few years prior in 2011, Baez, Fritz, and Leinster gave a category theoretical characterization of entropy in [4], which was recently extended to the quantum setting by Parzygnat in [5]. In preparation of that 2011 result, Baez remarked in the informal article [6] that entropy appears to behave similarly to a derivation in a certain operadic context, an observation we verify and make explicit below. Cohomological ideas are also explored in Mainiero’s recent work, where entropy is found to appear in the Euler characteristic of a particular cochain complex associated to a quantum state [7]. Upon taking inventory, one thus has the sense that entropy behaves somewhat similar to “d of something,” for some (co)boundary-like operator $d.$ The present article is in this same vein. Notably, once a few simple definitions are in place, the mathematics is quite straightforward. Even so, we feel it is worth sharing if for no other reason than to provide a glimpse at yet another algebraic and topological facet of entropy.

1.2. Background

To start, our work is based on a particular characterization of Shannon entropy that is compatible with an operadic viewpoint. Let ${\Delta }^n$ denote the standard topological n-simplex for $n\ge 0$,

$${\Delta }^n:=\{(p_0,p_1,\dots ,p_n)\in {\mathbb{R}}^{n+1}\mid 0\le p_i\le 1\mathrm{and}\sum _{i=0}^n{p}_i=1\},$$

where ${\Delta }^0$ denotes the unique probability distribution on the one-point set. More generally, any probability distribution $p=(p_0,\dots ,p_n)$ on an $n+1$-element set is a point in ${\Delta }^n$. Given $n+1$ probability distributions $q^i=(q_0^i,\dots ,q_{k_i}^i)\in {\Delta }^{k_i}$ where $i=0,1,\dots ,n$, they may be composed with p simultaneously to obtain a point in ${\Delta }^{k_0+k_1+\cdots +k_n+n}$ denoted by

$$p\circ (q^0,q^1,\dots ,q^n):=(p_0{q}_0^0,\dots ,p_0{q}_{k_0}^0,p_1{q}_1^1,\dots ,p_1{q}_{k_1}^1,\dots ,p_n{q}_1^n,\dots ,p_n{q}_{k_n}^n).$$

As shown in [1] and reviewed below, this composition of probabilities finds a natural home in the language of operads. Furthermore, it plays a key role in a well-known 1956 characterization of Shannon entropy due to D. K. Faddeev [8]. A proof of a slight variation of Faddeev’s result was recently given by Leinster [1], (Theorem 2.5.1). That is the version we quote here.

Theorem 1

(Faddeev-Leinster). Let ${\{F:{\Delta }^n\to \mathbb{R}\}}_{n\ge 0}$ be a sequence of functions. The following are equivalent:

1. 

the functions F are continuous and satisfy

$$F(p\circ (q^0,\dots ,q^n))=F\left(p\right)+\sum _{i=0}^n{p}_{i}F\left(q^i\right)$$

(1)

where $n\ge 0$ and $p\in {\Delta }^n$ and $q^i\in {\Delta }^{k_i}$ with $k_0$, $k_1,\dots ,k_n\ge 0$;

2. 

$F=cH$ for some $c\in \mathbb{R}.$

To make the connection with derivations, let us introduce some notation. Given a probability distribution $p\in {\Delta }^n$ let $\overline{p}:{\mathbb{R}}^{n+1}\to \mathbb{R}$ denote the function that maps a point $x=(x_0,\dots ,x_n)$ to the standard inner product $\langle p,x\rangle ={\sum }_{i=0}^n{p}_i{x}_i$. Then, when $F=H$, Equation (1) may be rewritten as

$$H(p\circ (q^0,\dots ,q^n))=H\left(p\right)+\overline{p}(H\left(q^0\right),\dots ,H\left(q^n\right)).$$

(2)

This equation is one hint that entropy might be a derivation, although a “q” is notably absent from the first term on the right-hand side. As a further teaser, Baez explored an algebraic interpretation of Equation (2) in the informal article [6], where the reader is reminded that Shannon entropy is a derivative of the partition function of a probability distribution with respect to Boltzmann’s constant, considered as a formal parameter. In that article, Equation (2) follows in a few short lines from this computation. One is thus motivated to look for a general framework of operad derivations for which Equation (2) is an example. This is what we describe below.

Section 2 reviews the definition of operads and representations of them. We will recall that the collection of topological simplices admits the structure of an operad as in [1] and that $\mathbb{R}$ gives rise to a representation of it. In Section 3, we define an abelian bimodule M over any operad $\mathcal{O}$ and the notion of a derivation of $\mathcal{O}$ with values in M. With these definitions in place, Equation (2) will find a generalization in Proposition 1, and the main result will quickly follow.

Theorem 2.

Shannon entropy defines a derivation of the operad of topological simplices, and for every derivation of this operad there exists a point at which it is given by a constant multiple of Shannon entropy.

2. Background: Operads and Their Representations

In an introduction to operads, it is helpful to first think about algebras. An algebra A is a vector space V equipped with a bilinear map $\mu :V\times V\to V$ thought of as multiplication. Depending on whether $\mu$ satisfies a particular relation, the algebra will usually be described by an approriate qualifier. For instance, if $\mu (v,w)=\mu (w,v)$ for all $v,w\in V$, then A is called a commutative algebra; if $\mu \left(\mu \right(u,v),w)=\mu (u,\mu (v,w\left)\right)$ for all $u,v,w\in V$, then A is a called an associative algebra, and so on. Behind each of these algebras is a particular operad that encodes the behavior of the multiplication map $\mu$. To motivate the formal definition, it is helpful to visualize $\mu$ as a planar binary rooted tree and more generally to imagine an arbitrary n-ary operation as a planar rooted tree with n leaves. There is a natural way to compose such operations. For instance, when f is a 3-ary operation and g is a 4-ary operation, they may be composed to obtain a 6-ary operation by using the output of g as one of the inputs of f as illustrated in Figure 1. There g has been grafted into the second leaf of the tree associated to f, and so we denote that choice with the subscript “${\circ }_2$” in the figure. There are two other composites $f{\circ }_{1}g$ and $f{\circ }_{3}g$, which are not shown but are obtained similarly.

In general, there are n ways to compose an m-ary operation with an n-ary operation, and the resulting operation will always have arity $m+n-1$. This composition should further satisfy some sensible associativity and unital axioms, and the collection of all such operations with their compositions is called an operad. The concept has origins in category theory [9] and has been used extensively in algebraic topology and homotopy theory [10,11,12,13,14] with applications in physics as well [15,16]. Operads may be defined in any symmetric monoidal category, and for ease of exposition below, we will assume all categories $\mathsf{C}$ are concrete (that is, all objects have underlying sets) so that we may refer to elements in a given object of $\mathsf{C}$. Indeed, the main example to have in mind is the category of topological spaces.

Definition 1.

Let $\mathsf{C}$ be a symmetric monoidal category with monoidal product $\otimes .$. Anoperadin $\mathsf{C}$ consists of a sequence of objects $\left\{\mathcal{O}\right(1),\mathcal{O}(2),\dots \}$ together with morphisms

$${\circ }_i:\mathcal{O}\left(n\right)\otimes \mathcal{O}\left(m\right)\to \mathcal{O}(n+m-1)$$

in $\mathsf{C}$ for all $n,m\ge 1$ and $1\le i\le n$ and an operation $1\in \mathcal{O}\left(1\right)$ satisfying the following:

(i) 

[associativity] For all $p\in \mathcal{O}\left(n\right)$ and $q\in \mathcal{O}\left(m\right)$ and $r\in \mathcal{O}\left(k\right)$,

$$\begin{array}{c}\hfill \left(p{\circ }_{j}q\right){\circ }_{i}r=\left\{\begin{array}{cc}\left(p{\circ }_{i}r\right){\circ }_{j+k-1}q\hfill & \mathit{if}1\le i\le j-1\hfill \\ p{\circ }_j\left(q{\circ }_{i-j+1}r\right)\hfill & \mathit{if}j\le i\le j+m-1\hfill \\ \left(p{\circ }_{i-m+1}r\right){\circ }_{j}q\hfill & \mathit{if}i\ge j+m\hfill \end{array}\right.\end{array}$$

(ii) 

[identity] The operation $1\in \mathcal{O}\left(1\right)$ acts as an identity in the sense that

$$1{\circ }_{1}p=p{\circ }_{i}1=p$$

for all $p\in \mathcal{O}\left(n\right)$ and $1\le i\le n.$

The definition is conceptually simple despite its cumbersome appearance. For instance, Figure 2 illustrates the associativity requirements listed in item (i).

As mentioned above, one often thinks of the elements $\mathcal{O}\left(n\right)$ as abstract n-to-1 operations, and the morphisms ${\circ }_i$ specify a way to compose them. It is common to begin indexing the sequence of objects at $n=0$ to account for 0-ary operations, but as we will soon see, our main example of an operad in Example 2 will have no 0-ary operations, and so our definition starts with $\mathcal{O}\left(1\right)$. We do not consider an action of the symmetric group and so $\mathcal{O}$ is sometimes called a non-symmetric operad, but we will simply call it an operad. In the special case when $\mathsf{C}$ is the category of vector spaces with linear maps and ⊗ is the tensor product, $\mathcal{O}$ is often called a linear operad. When it is the category $\mathsf{Top}$ of topological spaces with continuous maps and ⊗ is the Cartesian product, $\mathcal{O}$ is often called a topological operad.

Example 1.

Given a set X, theendomorphism operadis ${End}_X=\{{End}_X\left(1\right),{End}_X\left(2\right),\dots \}$ where ${End}_X\left(n\right):=\mathsf{C}(X^n,X)$ denotes the set of all functions from the n-fold Cartesian product $X^n$ to X. The unit operation in ${End}_X\left(1\right)$ is the identity function ${id}_X:X\to X.$ If $f\in \mathsf{C}(X^n,X)$ and $g\in \mathsf{C}(X^m,X)$ are a pair of functions, then for each $i=1,\dots ,n$ the composition $f{\circ }_{i}g$ is obtained by using the output of g as the ith input of $f.$ Explicitly, given $(x_1,\dots ,x_{n+m-1})\in X^{n+m-1}$,

$$\left(f{\circ }_{i}g\right)(x_1,\dots ,x_{n+m-1}):=f(x_1,\dots ,x_{i-1},g(x_i,\dots ,x_{i+m-1}),x_{i+m},\dots ,x_{n+m-1}).$$

The simultaneous composition of several functions may also be considered. That is, given n functions $g_i\in \mathsf{C}(X^{k_i},X)$ where $i=1,\dots ,n$ they may be composed with f simultaneously to obtain a new function $f\circ (g_1,\dots ,g_n)\in \mathsf{C}(X^{k_1+\cdots +k_n},X)$, which is again defined by using the outputs of the $g_i$ as the inputs of $f.$ Explicitly, given $(x_1,\dots ,x_{k_1+\cdots +k_n})\in X^{k_1+\cdots +k_n}$, we have

$$(f\circ (g_1,\dots ,g_n))(x_1,\dots ,x_{k_1+\cdots +k_n})=f(g_1(x_1,\dots ,x_{k_1}),\dots ,g_n(x_{k_1+\cdots +k_{n-1}+1},\dots ,x_{k_1+\cdots +k_n}))$$

Example 2.

The simplices ${\Delta }^0,{\Delta }^1,{\Delta }^2,\dots$ give rise to a topological operad calledthe operad of topological simplices$\Delta =\{{\Delta }_1,{\Delta }_2,\dots \}$ where ${\Delta }_n:={\Delta }^{n-1}$. The unit operation in ${\Delta }_1$ is the unique probability distribution on a one-point set. If $p=(p_1,\dots ,p_n)\in {\Delta }_n$ and $q=(q_1,\dots ,q_m)\in {\Delta }_m$ are probability distributions, then the composition $p{\circ }_{i}q$ is obtained by multiplying each of the m coordinates of q by $p_i$ and then replacing the $i\mathrm{th}$ coordinate of p with the resulting m-tuple. Explicitly,

$$p{\circ }_{i}q:=(p_1,\dots ,p_i{q}_1,\dots ,p_i{q}_m,\dots ,p_n)\in {\Delta }_{n+m-1}.$$

Equivalently, the distribution p may be visualized as a planar tree with n leaves labeled by the probabilities $p_1,\dots ,p_n$ and similarly for q. Then the composition $p{\circ }_{i}q$ is obtained by “painting” each of the leaves of q with the probability $p_i$ and grafting the resulting tree into the $i\mathrm{th}$ leaf of p as below. Notice the sum of the probabilities on the leaves on the composite tree is 1.

As an example, if $p=\left(\frac{1}{6},\dots ,\frac{1}{6}\right)$ represents the probability distribution of rolling a six-sided die and $q=\left(\frac{1}{2},\frac{1}{2}\right)$ is that of a fair coin toss, then $p{\circ }_{3}q=\left(\frac{1}{6},\frac{1}{6},\frac{1}{12},\frac{1}{12},\frac{1}{6},\frac{1}{6},\frac{1}{6}\right)$ is a point in ${\Delta }_7$, whose picture is shown on the left of Figure 3.

Further recall that if we have n different distributions $q^i=(q_1^i,\dots ,q_{k_i}^i)\in {\Delta }_{k_i}$ where $i=1,\dots ,n$, then we may compose them with p simultaneously to obtain the following point in ${\Delta }_{k_1+\cdots +k_n},$

$$p\circ (q^1,\dots ,q^n)=(p_1{q}_1^1,\dots ,p_1{q}_{k_1}^1,p_2{q}_1^2,\dots ,p_2{q}_{k_2}^2,\dots ,p_n{q}_1^n,\dots ,p_n{q}_{k_n}^n).$$

This simultaneous composition is illustrated by the tree on the right in Figure 3.

Just as groups come to life when considering representations of them, so operads come to life when each abstract n-ary operation is mapped to a concrete n-ary operation on a particular object. This assignment is traditionally called an algebra of the operad, but we prefer the more descriptive name representation.

Definition 2.

Let $\mathcal{O}$ be an operad in the category of sets. Arepresentation of $\mathcal{O}$, or an$\mathcal{O}$-representation, is set X together with functions

$${\phi }_n:\mathcal{O}\left(n\right)\to {End}_X\left(n\right)\mathit{for}n\ge 1$$

that respect the operad unit and compositions. That is, ${\phi }_n\left(1\right)=1$ and

$${\phi }_{n+m-1}\left(p{\circ }_{i}q\right)={\phi }_n\left(p\right){\circ }_i{\phi }_m\left(q\right)$$

for all $p\in \mathcal{O}\left(n\right),q\in \mathcal{O}\left(m\right)$ and $1\le i\le n$.

Importantly, one may also wish to define a representation of an operad in any symmetric monoidal category $\mathsf{C}$ whenever “${End}_X\left(n\right)$” is in fact an object in $\mathsf{C}.$ It must consist of an object X together with a family of morphisms $\mathcal{O}\left(n\right)\to {End}_X\left(n\right)$ in $\mathsf{C}$ that are compatible with the operad unit and compositions. This holds, for instance, when the monoidal category $\mathsf{C}$ is also closed—that is, when it is equipped with an internal hom functor that is compatible with the monoidal product. Monoidal closure, however, will not be required in our work, which primarily concerns the category $\mathsf{Top}$ of topological spaces. Indeed, the main example to have in mind is when $\mathcal{O}=\Delta$ is the operad of simplices and $X=\mathbb{R}$ is the real line in $\mathsf{Top}$. In this case, we define ${End}_{\mathbb{R}}\left(n\right):=\mathsf{Top}({\mathbb{R}}^n,\mathbb{R})$ to be the space of continuous functions ${\mathbb{R}}^n\to \mathbb{R}$ equipped with the product topology. Now, consider the continuous maps ${\phi }_n:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)$ given by $p\mapsto {\phi }_n\left(p\right)$ where ${\phi }_n\left(p\right)\left(x\right):=\langle p,x\rangle ={\sum }_{i=1}^n{p}_i{x}_i$ whenever $x=(x_1,\dots ,x_n)\in {\mathbb{R}}^n$. Then, it is simple to check that ${\phi }_{n+m-1}\left(p{\circ }_{i}q\right)={\phi }_n\left(p\right){\circ }_i{\phi }_m\left(q\right)$ for all $p,q,$ and i and that ${\phi }_n\left(1\right)=1$ for all n, and so $\mathbb{R}$ is a representation of $\Delta .$

3. Derivations of the Operad of Simplices

With these basic definitions in hand, the present goal is to define a mapping d out of the topological operad $\Delta$ that satisfies an appropriate version of the Leibniz rule,

$$\begin{array}{c}\hfill d\left(p{\circ }_{i}q\right)=dp{\circ }_{i}q+p{\circ }_{i}dq\left(\mathrm{desideratum}\right)\end{array}$$

(3)

for all $p\in {\Delta }_n$ and $q\in {\Delta }_m$ and for all $1\le i\le n$. This desired equation suggests the codomain of d should be a (bi)module over $\Delta$ that is, moreover, an abelian monoid. This motivates the following two definitions, the first of which is a slight generalization of that given by Markl in [15].

Definition 3.

Let $\mathcal{O}=\left\{\mathcal{O}\right(1),\mathcal{O}(2),\dots \}$ be an operad in a symmetric monoidal category $\mathsf{C}$. Abimodule over $\mathcal{O}$, or simply an$\mathcal{O}$-bimodule, is a collection of objects $M=\left\{M\right(1),M(2),\dots \}$ in $\mathsf{C}$ together with morphisms

$$\begin{array}{cc}\hfill {\circ }_i^L& =\mathcal{O}\left(n\right)\otimes M\left(m\right)\to M(n+m-1)(\mathit{left}\mathit{composition})\hfill \\ \hfill {\circ }_i^R& =M\left(n\right)\otimes \mathcal{O}\left(m\right)\to M(n+m-1)(\mathit{right}\mathit{composition})\hfill \end{array}$$

in $\mathsf{C}$ for each $1\le i\le n$ such that whenever

$$p\otimes q\otimes r\in \left\{\begin{array}{c}M\left(n\right)\otimes \mathcal{O}\left(m\right)\otimes \mathcal{O}\left(k\right),\mathrm{or}\hfill \\ \mathcal{O}\left(n\right)\otimes M\left(m\right)\otimes \mathcal{O}\left(k\right),\mathrm{or}\hfill \\ \mathcal{O}\left(n\right)\otimes \mathcal{O}\left(m\right)\otimes M\left(k\right)\hfill \end{array}\right.$$

the following holds:

$$\begin{array}{c}\hfill \left(p{\circ }_{j}q\right){\circ }_{i}r=\left\{\begin{array}{cc}\left(p{\circ }_{i}r\right){\circ }_{j+k-1}q\hfill & \mathit{if}1\le i\le j-1\hfill \\ p{\circ }_j\left(q{\circ }_{i-j+1}r\right)\hfill & \mathit{if}j\le i\le j+m-1\hfill \\ \left(p{\circ }_{i-m+1}r\right){\circ }_{j}q\hfill & \mathit{if}i\ge j+m.\hfill \end{array}\right.\end{array}$$

(4)

The associativity requirements displayed in Equation (4)—and hence the intuition behind them—are completely analogous to those defining operads as illustrated in Figure 2. The only difference here is that one of the three operations may come from the bimodule rather than the operad. Here is the main example to have in mind.

Example 3.

As every algebra is a bimodule over itself, so every representation of $\mathcal{O}$ is an $\mathcal{O}$-bimodule in a straightforward way. Indeed, in the case of the topological operad of simplices, the maps comprising the Δ-representation structure on $\mathbb{R}$ induce a Δ-bimodule structure on ${End}_{\mathbb{R}}$. However, we will make use of a slight variant of this bimodule structure. Right composition will be defined in the expected way, though left composition will not. Explicitly, we define the left and right composition maps

$$\begin{array}{cc}\hfill {\circ }_i^L:{\Delta }_n\times \mathsf{Top}({\mathbb{R}}^m,\mathbb{R})& ⟶\mathsf{Top}({\mathbb{R}}^{n+m-1},\mathbb{R})\hfill \\ \hfill {\circ }_i^R:\mathsf{Top}({\mathbb{R}}^n,\mathbb{R})\times {\Delta }_m& ⟶\mathsf{Top}({\mathbb{R}}^{n+m-1},\mathbb{R})\hfill \end{array}$$

as follows. Given a probability distribution $p\in {\Delta }_n$and a continuous function$f:{\mathbb{R}}^m\to \mathbb{R}$, define left composition by $p{\circ }_i^{L}f:=\overline{p}\circ (0,\dots ,0,f,0,\dots ,0)$, where the composition on the right-hand side is defined as in the simultaneous composition in the endomorphism operad of $\mathbb{R}$ illustrated in Example 1, and where each 0 denotes the zero function $\mathbb{R}\to \mathbb{R}$. Here, recall that $\overline{p}:{\mathbb{R}}^n\to \mathbb{R}$ maps a point x to the standard inner product $\langle p,x\rangle$as introduced in Section 1. Unwinding this, left composition thus evaluates explicitly as $\left(p{\circ }_i^{L}f\right)(x_1,\dots ,x_{n+m-1})=p_{i}f(x_i,\dots ,x_{i+m-1})$. In words, the value of the left composite $p{\circ }_i^{L}f:{\mathbb{R}}^{n+m-1}\to \mathbb{R}$ at a point x is computed by evaluating f at the m-subtuple of x beginning at the $i\mathrm{th}$coordinate and scaling that output by $p_i$. All other coordinates of x are ignored. The picture to have in mind is that below, where the bold dots are imagined to be “plugs” that prevent the surplus coordinates from playing a role. In this picture,$n=3$and$m=2$.

Given a probability distribution $q\in {\Delta }_m$ and a continuous function $g:{\mathbb{R}}^n\to \mathbb{R}$, define right composition by

$$\begin{array}{cc}\hfill \left(g{\circ }_i^{R}q\right)(x_1,\dots ,& x_{n+m-1})\hfill \\ \hfill & :=g(x_1,\dots ,x_{i-1},\sum _{k=1}^m{q}_k{x}_{i+k-1},x_{i+m},\dots ,x_{n+m-1}).\hfill \end{array}$$

This may be understood visually as well. The value of the right composite $g{\circ }_i^{R}q:{\mathbb{R}}^{n+m-1}\to \mathbb{R}$ at a point x is computed by taking the inner product of q with the m-tuple of x beginning at the $i\mathrm{th}$ coordinate and using that number as the $i\mathrm{th}$ input of g with all other coordinates of x falling into place as in the picture below. There are no “plugs” in this instance since all coordinates of x play a role.

These examples suggest the inner product notation is a convenient choice. Given $N\ge 1$ and $k\le N$ and a point $x\in {\mathbb{R}}^N$, let ${\mathbf{x}}_{i,k}\in {\mathbb{R}}^k$ denote the k-subtuple of x beginning at the $i\mathrm{th}$ coordinate:

$${\mathbf{x}}_{i,k}:=(x_i,\dots ,x_{i+k-1}).$$

Then given any point $x\in {\mathbb{R}}^{n+m-1}$, the left and right composition maps may be written more succinctly as

$$\begin{array}{cc}\hfill \left(p{\circ }_i^{L}f\right)\left(x\right)& =p_{i}f\left({\mathbf{x}}_{i,m}\right)\hfill \\ \hfill \left(g{\circ }_i^{R}q\right)\left(x\right)& =g(x_1,\dots ,x_{i-1},\langle q,{\mathbf{x}}_{i,m}\rangle ,x_{i+m},\dots ,x_{n+m-1}).\hfill \end{array}$$

We will use this notation below and will always write ${\mathbf{x}}_i$ in lieu of ${\mathbf{x}}_{i,m}$ since the context will make it clear that ${\mathbf{x}}_i$ must be an m-tuple. The boldface font is used to distinguish a tuple ${\mathbf{x}}_i$ from a real number $x_i$. Finally, note that the maps ${\circ }_i^L$ and ${\circ }_i^R$ are continuous since f and g are continuous, and moreover that the associativity requirements in Equation (4) are analogous to those illustrated in Figure 2, so it is straightforward to verify they are satisfied. In particular, the zero functions appearing in the definition of ${\circ }_i^L$ simplify the situation greatly. For instance, several of associativity requirements follow from the simple fact that multiplying an input $x_i$ by a probability and then mapping the result to zero is the same as first mapping the input to zero and then multiplying that zero by a probability. So ${End}_{\mathbb{R}}$ is indeed a Δ-bimodule.

Next, recall that the desired Leibniz rule in Equation (3) suggests the bimodule should be equipped with a notion of addition. This motivates the following definition.

Definition 4.

Let $\mathcal{O}$ be an operad in a symmetric monoidal category $\mathsf{C}$. An $\mathcal{O}$-bimodule M is anabelian $\mathcal{O}$-bimoduleif each $M\left(n\right)$ is an abelian monoid in $\mathsf{C}$; that is, if for each $n=1,2,\dots$ the following hold:

(i) 

[associativity, commutativity] there is a morphism ${\mu }_n:M\left(n\right)\times M\left(n\right)\to M\left(n\right)$ in $\mathsf{C}$ such that ${\mu }_n({\mu }_n(a,b),c)={\mu }_n(a,{\mu }_n(b,c))$ and ${\mu }_n(a,b)={\mu }_n(b,a)$ for all $a,b,c\in M\left(n\right)$,

(ii) 

[identity] there is an element $1\in M\left(n\right)$ such that ${\mu }_n(1,a)=a={\mu }_n(a,1)$ for all $a\in M\left(n\right)$.

As the primary example, consider ${End}_{\mathbb{R}}$ viewed as a $\Delta$-bimodule as described in Example 3. For each $n,$ define ${\mu }_n:{End}_{\mathbb{R}}\left(n\right)\times {End}_{\mathbb{R}}\left(n\right)\to {End}_{\mathbb{R}}\left(n\right)$ by pointwise addition, meaning that for each $f,g\in {End}_{\mathbb{R}}\left(n\right)$ we have ${\mu }_n(f,g)=f+g$ where $(f+g)\left(x\right):=f\left(x\right)+g\left(x\right)$ for all $x\in {\mathbb{R}}^n$. The identity element in ${End}_{\mathbb{R}}\left(n\right)$ is the constant map at zero. Moreover each ${\mu }_n$ is continuous and inherits associativity and commutativity from $\mathbb{R}.$ In this way, ${End}_{\mathbb{R}}$ is an abelian $\Delta$-bimodule.

Remark 1.

Notice that the Δ-bimodule composition maps ${\circ }_i^L$ and ${\circ }_i^R$ distribute over sums in the abelian Δ-bimodule ${End}_{\mathbb{R}}$. In other words, for all continuous functions $f,g\in {End}_{\mathbb{R}}\left(n\right)$ and for all probability distributions $q\in {\Delta }_m$,

$$\begin{array}{c}\hfill (f+g){\circ }_i^{R}q=f{\circ }_i^{R}q+g{\circ }_i^{R}q,1\le i\le n\end{array}$$

and similarly for left composition ${\circ }_i^L$. This follows directly from pointwise addition.

With this setup in mind, our desideratum in Equation (3) is now realized in the following definition.

Definition 5.

Let $\mathcal{O}$ be an operad in a category $\mathsf{C}$ and let M be an abelian $\mathcal{O}$-bimodule. Aderivation of $\mathcal{O}$ valued in M is sequence of morphisms $\{d_n:\mathcal{O}\left(n\right)\to M\left(n\right)\}$ in $\mathsf{C}$ satisfying

$$\begin{array}{c}\hfill d_{n+m-1}\left(p{\circ }_{i}q\right)=d_{n}p{\circ }_i^{R}q+p{\circ }_i^L{d}_{m}q\end{array}$$

(5)

for all $p\in \mathcal{O}\left(n\right),q\in \mathcal{O}\left(m\right)$ and for all $1\le i\le n.$

In the special case when $\mathcal{O}$ is a linear operad, this definition coincides with that given by Markl in [15]. In what follows, we omit the subscripts and simply write d instead of $d_n$. Now, suppose $\mathcal{O}=\Delta$ is the operad of topological simplices and ${End}_{\mathbb{R}}$ is equipped with the structure of an abelian $\Delta$-bimodule given above. Here is the picture to have in mind for Equation (5):

On the right-hand side we have used the “plug” notation introduced in Example 3, which can also be understood explicitly by evaluating d at a point $x\in {\mathbb{R}}^{n+m-1}$,

$$\begin{array}{cc}\hfill d\left(p{\circ }_{i}q\right)\left(x\right)& =\left(dp{\circ }_i^{R}q\right)\left(x\right)+\left(p{\circ }_i^{L}dq\right)\left(x\right)\hfill \\ \hfill & =dp(x_1,\dots ,\langle q,{\mathbf{x}}_i\rangle ,\dots ,x_{n+m-1})+p_{i}dq\left({\mathbf{x}}_i\right).\hfill \end{array}$$

Of particular interest is the behavior of a derivation $\{d:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)\}$ when it is applied to a simultaneous composition of probability distributions. A derivation applied to the composite $\left(p{\circ }_{j}q\right){\circ }_{i}r$ for probability distributions $p\in {\Delta }_n,q\in {\Delta }_m$, and $r\in {\Delta }_k$ can be understood in a convenient picture when q and r are composed onto different leaves of p; that is, when $1\le i\le j-1$ or $i\ge j+m$. This follows straightforwardly from a repeated application of d. Indeed, by definition we have $d\left(\left(p{\circ }_{j}q\right){\circ }_{i}r\right)=d\left(p{\circ }_{j}q\right){\circ }_i^{R}r+\left(p{\circ }_{j}q\right){\circ }_i^{L}dr$ and by applying the Leibniz rule again to the first summand, this is equal to $(dp{\circ }_j^{R}q+p{\circ }_j^{L}dq){\circ }_i^{R}r+\left(p{\circ }_{j}q\right){\circ }_i^{L}dr,$ which we can expand to obtain $\left(dp{\circ }_j^{R}q\right){\circ }_i^{R}r+\left(p{\circ }_j^{L}dq\right){\circ }_i^{R}r+\left(p{\circ }_{j}q\right){\circ }_i^{L}dr$ since composition distributes over sums as noted in Remark 1. We will identify this function with the picture below in lieu of the cumbersome notation.

Importantly, the obvious generalization of the formula holds for any simultaneous composition $p\circ (q^1,\dots ,q^n)$ for any $p\in {\Delta }_n$ and $q^i\in {\Delta }_{k_i}$ where $i=1,\dots ,n$. This again follows directly from repeated applications of Equation (5), as illustrated below.

This is summarized in the following proposition.

Proposition 1.

Let $p\in {\Delta }_n$ and $q^i\in {\Delta }_{k_i}$ for $n,k_1,\dots ,k_n\ge 1$ and let $\{d:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)\}$ be a derivation of the operad of topological simplices. Then for any point $x\in {\mathbb{R}}^{k_1+\cdots +k_n}$,

$$d(p\circ (q^1,\dots ,q^n))\left(x\right)=dp(\langle q^1,{\mathbf{x}}_1\rangle ,\cdots ,\langle q^n,{\mathbf{x}}_n\rangle )+\sum _{i=1}^n{p}_{i}d{q}^i\left({\mathbf{x}}_i\right).$$

Finally, the main result follows.

Theorem 3.

Shannon entropy defines a derivation of the operad of topological simplices, and for every derivation of this operad there exists a point at which it is given by a constant multiple of Shannon entropy.

Proof. 

For each $n\ge 1$ define $d:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)$ by $p\mapsto dp$ where $dp\left(x\right)=H\left(p\right)$ is constant for all $x\in {\mathbb{R}}^n$. Then, d is continuous since H is continuous. Moreover, if $p=(p_1,\dots ,p_n)\in {\Delta }_n$ and $q=(q_1,\dots ,q_m)\in {\Delta }_m$ are probability distributions, then for any $x\in {\mathbb{R}}^{m+n-1}$ and $1\le i\le n$, we have

$$\begin{array}{cc}\hfill d\left(p{\circ }_{i}q\right)\left(x\right)=H\left(p{\circ }_{i}q\right)& =-\left(\sum _{k=1}^{i-1}{p}_{k}log{p}_k+p_i\sum _{k=1}^m{q}_{k}log\left(p_i{q}_k\right)+\sum _{k=i+1}^n{p}_{k}log{p}_k\right)\hfill \\ \hfill & =-\left(\sum _{k=1}^{i-1}{p}_{k}log{p}_k+p_{i}log{p}_i\sum _{k=1}^m{q}_k+p_i\sum _{k=1}^m{q}_{k}log{q}_k+\sum _{k=i+1}^n{p}_{k}log{p}_k\right)\hfill \\ \hfill & =-\left(\sum _{k=1}^n{p}_{k}log{p}_k+p_i\sum _{k=1}^m{q}_{k}log{q}_k\right)\hfill \\ \hfill & =H\left(p\right)+p_{i}H\left(q\right)\hfill \\ \hfill & =(dp{\circ }_i^{R}q+p{\circ }_i^{L}dq)\left(x\right),\hfill \end{array}$$

where the last line follows since $\left(dp{\circ }_i^{R}q\right)\left(x\right)$ is computed by evaluating the function $dp$ at some point, and this function is assumed to be constant at $H\left(p\right).$

Conversely, suppose $\{d:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)\}$ is a derivation. For each $n\ge 1$ define a function $F:{\Delta }_n\to \mathbb{R}$ by $F\left(p\right)=dp\left(0\right)$ where $0=(0,\dots ,0)\in {\mathbb{R}}^n$. Then F is continuous since d is continuous, and Proposition 1 further implies that

$$\begin{array}{cc}\hfill F(p\circ (q^1,\dots ,q^n))& =d(p\circ (q^1,\dots ,q^n))\left(0\right)\hfill \\ \hfill & =dp(\langle q^1,{\mathbf{0}}_1\rangle ,\dots ,\langle q^n,{\mathbf{0}}_n\rangle )+\sum _{i=1}^n{p}_{i}d{q}^i\left({\mathbf{0}}_i\right)\hfill \\ \hfill & =dp\left(0\right)+\sum _{i=1}^n{p}_{i}d{q}^i\left(0\right)\hfill \\ \hfill & =F\left(p\right)+\sum _{i=1}^n{p}_{i}F\left(q^i\right).\hfill \end{array}$$

From the Faddeev–Leinster result in Theorem 1, it follows that $dp\left(0\right)=F\left(p\right)=cH\left(p\right)$ for some $c\in \mathbb{R}$. □

Notice that the important Equation (2) mentioned in the introduction is obtained as a corollary. Indeed, if for each $n\ge 1$ the map $d:{\Delta }_n\to {End}_{\mathbb{R}}\left(n\right)$ is defined to be constant at entropy $p\mapsto dp\equiv H\left(p\right)$, then d is a derivation by Theorem 3 and so Proposition 1 yields the following by evaluating $d(p\circ (q^1,\dots ,q^n))$ at any point.

Corollary 1.

Let $p\in {\Delta }_n$ and $q^i\in {\Delta }_{k_i}$ with $1\le i\le n.$ Then

$$H(p\circ (q^1,\dots ,q^n))=H\left(p\right)+\sum _{i=1}^n{p}_{i}H\left(q^i\right).$$

As a closing remark, Faddeev’s characterization of entropy in Theorem 1 can be reexpressed using the language of category theory and operads as in [1], (Theorem 12.3.1). We have omitted this language here but invite the reader to explore the full category theoretical story in Chapter 12 of Leinster’s book.