Minimum Message Length Inference of the Exponential Distribution with Type I Censoring

Abstract

Data with censoring is common in many areas of science and the associated statistical models are generally estimated with the method of maximum likelihood combined with a model selection criterion such as Akaike’s information criterion. This manuscript demonstrates how the information theoretic minimum message length principle can be used to estimate statistical models in the presence of type I random and fixed censoring data. The exponential distribution with fixed and random censoring is used as an example to demonstrate the process where we observe that the minimum message length estimate of mean survival time has some advantages over the standard maximum likelihood estimate.

1. Introduction

In Type I random censoring we observe for each item i either the true survival time $T_i=t_i$ ($t_i>0$) or the censoring time $C_i=c_i$ ($c_i>0$), where capital letters are used to denote random variables. The data consists of joint realisations of the random variables $(Y_i=y_i,{\Delta }_i={\delta }_i)$ ($i=1,\dots ,n$) where

$$\begin{array}{cc}\hfill Y_i& =min(T_i,C_i),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\Delta }_i& =\mathrm{I}(T_i\le C_i)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{T}_i\le C_i\left(\mathrm{observed}\mathrm{survival}\right)\hfill \\ 0,\hfill & \mathrm{if}{T}_i>C_i\left(\mathrm{observed}\mathrm{censoring}\right).\hfill \end{array}\right.\hfill \end{array}$$

(2)

The censoring time $C_i$ may be fixed (i.e., $C_i=c$ for all $i=1,\dots ,n$) or a random variable that may depend on other factors (e.g., loss to follow-up). The likelihood function of n observed data points $D=\{(y_1,{\delta }_1),\dots ,(y_n,{\delta }_n)\}$ is

$$p\left(D\right)=\prod _{i=1}^n{\left(p_T\left(y_i\right)(1-F_C\left(y_i\right))\right)}^{{\delta }_i}{\left(p_C\left(y_i\right)(1-F_T\left(y_i\right))\right)}^{1-{\delta }_i}$$

where $p_T\left(t\right|\theta )$ and $F_T\left(t\right|\theta )$ denote the probability density and the cumulative density function of the random variable T, respectively. Inference about the survival times $(t_1,\dots ,t_n)$ is of key interest in many areas of science and is commonly done by maximizing the likelihood and dropping terms relevant to C only.

This manuscript examines inference of models in the presence of censored data under the minimum message length (MML) framework. MML (see Section 3) is Bayesian technique for model selection and parameter estimation that is based on data compression and key principles of information theory. MML is known to possess strong theoretical properties [1,2,3] and has previously been successfully applied to a wide range of statistical models [1]. Here, we demonstrate how MML can be used to infer models under fixed censoring as well as type I random censoring. We use the exponential distribution (see Section 2) as a simple example to demonstrate the key steps and compare the MML estimator to the well-known maximum likelihood estimator in this setting (see Section 2.1). Although MML analysis of the exponential distribution is not new (see, for example, [1,4]), the MML principle has not been applied to any kind of survival data with censoring to date.

The main contributions of this manuscript are to: (i) introduce the MML principle of inductive inference and demonstrate how the Wallace–Freeman MML approximation can used to infer exponential models with type I censored data; (ii) show that the MML estimate of the mean lifetime has some advantages over the usual maximum likelihood estimate for small samples and that it converges to the maximum likelihood estimate for large sample sizes, (iii) incorporate the proposed codelengths for censored exponential distributions into MML finite mixture models allowing for inference of all parameters as well as the number of mixture classes; and (iv) compare the MML principle to the closely related minimum description length principle.

2. Exponential Distribution

Consider the case of a randomly censored exponential parameter studied in [5] where the lifetime data and the censoring data are assumed to be exponentially distributed

$$T_i\sim \mathrm{Exp}\left(\beta \right),C_i\sim \mathrm{Exp}\left(\alpha \right),i=1,\dots ,n,$$

(3)

and $\alpha ,\beta >0$ denote the mean censoring time and survival time, respectively. Under this model, the joint probability distribution of $(Y_i=y_i,{\Delta }_i={\delta }_i)$ is

$$\begin{array}{ccc}\hfill p(Y_i=y_i,{\Delta }_i=1)& =& p_T\left(y_i\right)(1-F_C\left(y_i\right))\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill p(Y_i=y_i,{\Delta }_i=0)& =& p_C\left(y_i\right)(1-F_T\left(y_i\right))\hfill \end{array}$$

(5)

where $(Y_i,{\Delta }_i)$ are defined in (1) and (2) respectively. In contrast to random censoring, in fixed censoring an item is observed for a period of time, say $c>0$, and its actual survival time $t_i$ is known if the item fails before time $t_i\le c$; otherwise, we only know that the item survived past $t_i>c$. In the case of exponentially distributed survival times, the observed data ($Y=y_i,{\Delta }_i={\delta }_i)$ follows

$$T_i\sim \mathrm{Exp}\left(\theta \right),C_i=c,i=1,\dots ,n,$$

(6)

where $c>0$ is a fixed constant (the follow up period) known a priori. Given n data points $D=\{(y_1,{\delta }_1),\dots ,(y_n,{\delta }_n)\}$, the aim is to estimate the unknown mean lifetime survival $\beta >0$ (random censoring) or $\theta >0$ (fixed censoring).

2.1. Maximum Likelihood Estimation

The method of maximum likelihood is the most common approach used to obtain parameter estimates in parametric models. Under the censored exponential model, maximum likelihood proceeds by setting the parameter estimate $\widehat{\beta }\left(D\right)$ to the value that maximises the probability of the data. From (4) and (5), the joint probability of the data D is

$$\begin{array}{cc}\hfill p\left(D\right|\alpha ,\beta )& ={\left(\frac{1}{\beta }\right)}^k{\left(\frac{1}{\alpha }\right)}^{n-k}exp\left(-\left(\frac{1}{\beta }+\frac{1}{\alpha }\right)\sum _{i=1}^n{y}_i\right),\hfill \end{array}$$

(7)

where $k=\left({\sum }_i{\delta }_i\right)$ is the number of observed uncensored survival times. Maximizing the likelihood function is equivalent to minimizing the negative log-likelihood function

$$-logp\left(D\right|\alpha ,\beta )=klog\beta +(n-k)log\alpha +\left(\frac{1}{\beta }+\frac{1}{\alpha }\right)\sum _{i=1}^n{y}_i.$$

(8)

Maximum likelihood estimates of the mean survival and censoring times are

$$\widehat{\beta }{\left(D\right)}_{\mathrm{ML}}=\frac{1}{k}\sum _{i=1}^n{y}_i,\widehat{\alpha }{\left(D\right)}_{\mathrm{ML}}=\frac{1}{n-k}\sum _{i=1}^n{y}_i,$$

(9)

respectively. Provided the count of observed survival times $k\in (0,n)$, the maximum likelihood estimates $\widehat{\alpha }\left(D\right)$ and $\widehat{\beta }\left(D\right)$ are finite; otherwise, if $k=0$ or $k=n$, one of the maximum likelihood estimates $\widehat{\alpha }\left(D\right)$ or $\widehat{\beta }\left(D\right)$ is infinite. Kim [5] showed that maximum likelihood estimates have infinite mean and variance in this setting. However, the expected value of the maximum likelihood estimate $\widehat{\beta }\left(D\right)$ is finite if we condition on $k>0$. Kim [5] further showed that, provided $k\in (0,n)$, the maximum likelihood estimates $\widehat{\alpha }\left(D\right)$ and $\widehat{\beta }\left(D\right)$ are unbiased, strongly consistent (without any condition on k) and asymptotically normally distributed.

In the case of Type I censored data with a fixed censoring time $c>0$, the negative log-likelihood function of the data is

$$-logp\left(D\right|\theta ;c)=klog\left(\theta \right)+\frac{1}{\theta }\left(\sum _{i=1}^n{y}_i{\delta }_i\right)+\frac{c(n-k)}{\theta }$$

(10)

where $k={\sum }_i{\delta }_i$ as before. Under fixed censoring, the maximum likelihood estimate of the mean survival time $\widehat{\theta }\left(D\right)$ is (see, for example, [6])

$$\widehat{\theta }\left(D\right)=\frac{c(n-k)+{\sum }_{i=1}^n{\delta }_i{y}_i}{k}.$$

(11)

In case of no censoring (i.e., $k=n$ implying complete data), (11) reduces to $\left({\sum }_i{y}_i\right)/n$, which is the usual maximum likelihood estimate for the exponential distribution with complete data. The sampling distribution of (11) is asymptotically normal with mean $\theta$ and variance

$$\frac{{\theta }^2}{n(1-exp(-c/\theta \left)\right)}=\frac{{\theta }^2}{n{F}_T\left(c\right|\theta )}.$$

(12)

Conditional upon $k>0$, Mendenhall and Lehman [7] obtained the exact mean and variance of the maximum likelihood estimate (11)

$$\mathbb{E}\left\{\widehat{\theta }\right\}=\theta -c\left(\frac{q}{p}-n\mathbb{E}\left\{k^{-1}\right\}+1\right),\mathbb{V}\left\{\widehat{\theta }\right\}={\left(nc\right)}^2\mathbb{V}\left\{k^{-1}\right\}+({\theta }^2-c^{2}q/p^2)\mathbb{E}\left\{k^{-1}\right\},$$

where

$$p=1-exp(-c/\theta ),q=1-p,\mathbb{E}\left\{k^{-a}\right\}=\frac{1}{1-q^n}\sum _{k=1}^n\frac{1}{k^a}\left(\genfrac{}{}{0pt}{}{n}{k}\right)p^k{q}^{n-k},$$

for $a=1,2,\dots$ and $\mathbb{V}\left\{k^{-1}\right\}=\mathbb{E}\left\{k^{-2}\right\}-{\left(\mathbb{E}\left\{k^{-1}\right\}\right)}^2$. However, the large sample normal approximation of the distribution of the maximum likelihood estimates is inaccurate and not representative of the behaviour of the estimate in the small to moderate sample size regime [7]. Balakrishnan and Davies [8] further show that the maximum likelihood estimate computed based on a censoring time $c^{\prime }$ will always produce an estimate which is Pitman closer to the data generating model $\theta$ than the maximum likelihood estimate computed with a shorter censoring time $c<c^{\prime }$. In the next section, we introduce the MML principle of inductive inference (see Section 3) and demonstrate how MML can be used to infer exponential models with censoring (see Section 4).

3. Minimum Message Length

Introduced in the late 1960s by Wallace and Boulton [9], the minimum message length (MML) principle [1,9,10,11] is a framework for inductive inference based on ideas in information theory and data compression. Under the MML framework, the aim is to transmit a set of data (a message) from a hypothetical sender to a receiver over a noiseless transmission channel. The MML message is designed to consist of two parts:

• the assertion: an encoding of the model structure and the associated model parameters $\mathbf{\theta }\in \Theta \in {\mathbb{R}}^p$.
• the detail: a description of the data D using the model $p\left(D\right|\mathbf{\theta })$ that was specified in the assertion.

The length of the assertion measures the complexity of the model, with complex models requiring longer codelengths compared to simpler models, while the detail captures how well a model fits the data. The length of the two-part message, $I(D,\mathbf{\theta })$, is the sum of the length of the assertion, $I\left(\mathbf{\theta }\right)$, and the length of detail, $I\left(D\right|\mathbf{\theta })$; namely,

$$I(D,\mathbf{\theta })=\underset{\mathrm{assertion}}{\underbrace{I\left(\mathbf{\theta }\right)}}+\underset{\mathrm{detail}}{\underbrace{I\left(D\right|\mathbf{\theta })}}.$$

(13)

Within the MML framework we seek the model

$$\widehat{\mathbf{\theta }}\left(D\right)=\underset{\mathbf{\theta }\in \Theta }{arg\; min}\left\{I(D,\mathbf{\theta })\right\}$$

(14)

that minimises the length of this message. Due to the two-part nature of the message, MML automatically balances the trade-off between model complexity and the goodness of fit of the model to the data. By measuring the quality of a model in (say) bits, MML is a yardstick that can be universally used to compare models with different parameters and structures.

There exist several approches to computing message lengths (13), with the strict MML procedure (SMML) [1,12] and the MML87 approximation [1,10] being the most widely known. In contrast to the SMML procedure whose construction is known to be NP hard [13], the MML87 approximation is computationally tractable and most widely used in practice. The MML87 codelength approximation to (13) is

$$I_{87}(D,\mathbf{\theta })=\underset{\mathrm{assertion}}{\underbrace{-log\pi \left(\mathbf{\theta }\right)+\frac{1}{2}log\left|J_{\mathbf{\theta }}\left(\mathbf{\theta }\right)\right|+\frac{p}{2}log{\kappa }_p}}+\underset{\mathrm{detail}}{\underbrace{\frac{p}{2}-logp\left(D\right|\mathbf{\theta })}}$$

(15)

where ${\pi }_{\mathbf{\theta }}\left(\mathbf{\theta }\right)$ is the prior distribution for the parameters $\mathbf{\theta }$, $|J_{\mathbf{\theta }}\left(\mathbf{\theta }\right)|$ is the determinant of the expected Fisher information matrix, $p\left(D\right|\mathbf{\theta })$ is the likelihood function of the model and ${\kappa }_p$ is a quantization constant [14,15] that depends on the number of parameters p. Specifically, for small p we have

$${\kappa }_1=\frac{1}{12},{\kappa }_2=\frac{5}{36\sqrt{3}},{\kappa }_3=\frac{19}{192\times 2^{1/3}},$$

(16)

while ${\kappa }_p$ is well-approximated for large p by [1]:

$$\frac{p}{2}(log{\kappa }_p+1)\approx -\frac{p}{2}log2\pi +\frac{1}{2}logp\pi -\gamma ,$$

(17)

where $\gamma \approx 0.5772$ is the Euler–Mascheroni constant. The MML87 codelength, evaluated at the minimum, is the shortest codelength of a two-part message that encodes both the model parameters $\mathbf{\theta }\in \Theta$ and the data D. The MML87 approximation is known to be invariant under smooth one-to-one reparameterizations of the likelihood function and is asymptotically equivalent to the well-known Bayesian information criterion (BIC) [16] as $n\to \infty$ with $p>0$ fixed; that is,

$$I_{87}(D,\mathbf{\theta })=-logp\left(D\right|\mathbf{\theta })+\frac{p}{2}logn+O\left(1\right)$$

(18)

where the $O\left(1\right)$ term depends on the prior distribution, the Fisher information and the number of parameters p. Unlike model selection criteria such as Akaike’s information criterion (AIC) and BIC, MML allows for both parameter estimation and model selection within the same unified framework. Furthermore, in models where the number of parameters grows with n or the sample size is relatively small, the difference between the MML87 codelength and BIC can be substantial. Examples include analysis of multiple short time series, where several measurements are collected over a period of time for a large number of study participants [17], learning finite mixture models [18] and discriminating between Poisson and geometric distributions based on observed data [19]. In the latter example, both the Poisson and geometric distribution have the same number of free parameters so that model selection with BIC is equivalent to choosing the model with the higher likelihood. In contrast, MML87 takes into account the complexity of each distribution [20] and not just the number of parameters, resulting in improved model selection performance for small sample sizes [19].

MML has been successful applied to a wide range of problems (e.g., decision trees [21], factor analysis [22], linear causal models [23], mixture modelling [18,24]) demonstrating excellent parameter estimation properties and model selection performance that is on par or better than commonly used techniques such as Akaike’s information criterion (AIC) [25] and the Bayesian information criterion (BIC). A brief tutorial overview of minimum message length can be found in [19].

4. Minimum Message Length Inference of Type I Censored Exponential Data

To encode and transmit censored data $D=\{(y_1,{\delta }_1),\dots ,(y_n,{\delta }_n)\}$ between the hypothetical sender and receiver within the MML framework, we have two options:

• Transmit the censoring indicators $({\delta }_1,\dots ,{\delta }_n)$ first and then transmit the lifetime survival data $(y_1,\dots ,y_n)$ given the receiver now knows which of the n data points are censored (see Section 4.1);
• Transmit the censoring indicators and the lifetime data simultaneously (see Section 4.2).

We shall now estimate the MML87 codelength (15) for both the joint and the conditional encoding schemes for the censored exponential distribution setting introduced in Section 2.

4.1. Conditional Encoding of the Data

Under the conditional encoding framework, the sender transmits the censoring indicators ${\delta }_i$ first, and then transmits the lifetime data $y_i$ using the conditional distribution of the data given the observed censoring indicators. The total message length of the data $D=\{(y_1,{\delta }_1),\dots ,(y_n,{\delta }_n)\}$ and the parameters $\mathbf{\theta }$ with the conditional encoding is

$$I_{87}(D,\mathbf{\theta })=I_{87}(\varphi ,\mathbf{\delta })+I_{87}(\psi ,\mathbf{y}|\mathbf{\delta }),$$

(19)

where $\theta =\{\varphi ,\psi \}$ are the model parameters defined below, $I(\varphi ,\mathbf{\delta })$ denotes the message length of the censoring indicators $\mathbf{\delta }=({\delta }_1,\dots ,{\delta }_n)$, and $I(\psi ,\mathbf{y}|\mathbf{\delta })$ denotes the codelength of the survival data $\mathit{y}=(y_1,\dots ,y_n)$, given that the censoring indicators are known to the receiver. From (3), the probability of observing an uncensored datum, say $\varphi >0$, is

$$\varphi =P(T_i\le C_i)=\frac{\alpha }{\alpha +\beta },(i=1,\dots ,n),$$

(20)

implying that the censoring indicators follow a Bernoulli distribution with probability $\varphi$; that is, ${\delta }_i\sim \mathrm{Bernoulli}\left(\varphi \right)$, or equivalently, k follows the binomial distribution $k\sim \mathrm{binomial}(\varphi ,n)$.

The MML87 codelength of the binomial distribution was previously derived in [1,18] and is included here for completeness. Briefly, to compute the MML87 codelength (15) we require the Fisher information $J_{\varphi }\left(\varphi \right)$ and the prior distribution ${\pi }_{\varphi }\left(\varphi \right)$ for the probability of observing an uncensored datum. The Fisher information is well-known

$$J_{\varphi }\left(\varphi \right)=\frac{n}{\varphi (1-\varphi )}.$$

(21)

We assume the prior distribution for the censoring probability $\varphi$ to be the beta distribution ($\varphi \sim \mathrm{beta}(a,b)$) with probability density function

$${\pi }_{\varphi }\left(\varphi \right|a,b)=\frac{{\varphi }^{a-1}{(1-\varphi )}^{b-1}}{B(a,b)}$$

(22)

where $a,b>0$ are the shape and scale parameters respectively and $B(a,b)$ is the usual beta function. Substituting (21) and (22) into the MML87 codelength (15) and noting that ${\kappa }_1=1/12$, yields

$$\begin{gathered} I_{87}(\varphi ,\mathbf{\delta })=-\left(k+a-\frac{1}{2}\right)log\varphi -\left(n+b-\frac{1}{2}-k\right)log(1-\varphi ) \\ +logB(a,b)+\frac{1}{2}(1+logn-log12) \end{gathered}$$

(23)

where, as before, $k=\left({\sum }_i{\delta }_i\right)$. The codelength (23) is minimised at the MML87 estimate

$${\widehat{\varphi }}_{87}\left(\mathbf{\delta }\right)=\frac{k+a-1/2}{n+a+b-1}.$$

(24)

Note that, in the special case of uniform prior distribution ($a=b=1$), the MML87 estimate simplifies to

$${\widehat{\varphi }}_{87}\left(\mathbf{\delta }\right)=\frac{k+1/2}{n+1}.$$

(25)

The shortest MML87 codelength for the censoring indicators is therefore given by $I_{87}({\widehat{\varphi }}_{87},\mathbf{\delta })$. It remains to work out the conditional codelength of the surivival times given the censoring indicators, $I_{87}(\psi ,\mathbf{y}|\mathbf{\delta })$.

We note that the conditional likelihood of the lifetime datum $y_i$ is

$$p\left(y_i\right|\alpha ,\beta ,\delta =0)=p\left(y_i\right|\alpha ,\beta ,\delta =1)=\left(\frac{1}{\beta }+\frac{1}{\alpha }\right)exp\left(-y_i\left(\frac{1}{\beta }+\frac{1}{\alpha }\right)\right),$$

(26)

which is the exponential distribution with mean $\psi ={(1/\beta +1/\alpha )}^{-1}$; that is,

$$y_i|{\delta }_i\sim \mathrm{Exp}\left(\psi \right),i=1,\dots ,n.$$

(27)

The Fisher information of the exponential distribution is

$$J_{\psi }\left(\psi \right)=\frac{n}{{\psi }^2}.$$

(28)

In terms of the prior distribution for $\psi$, Schmidt and Makalic [4] consider the conjugate exponential distribution with a hyperparameter ${\psi }_0$ that controls the prior mean. Here, we would like an objective prior distribution on the mean $\psi$ that is free of hyperparameters and has heavy tails so that large values of $\psi$ are not penalized too severely. Additionally, our choice of the prior distribution should ideally lead to an easy to compute analytic estimate of $\psi$. A reasonable option is the half-Cauchy distribution which has heavy tails however it leads to MML estimates that are roots of polynomial functions of $s={\sum }_i\left(y_i\right)$. Instead, we will use the Fréchet (inverse Weibull) distribution with probability density function

$${\pi }_{\psi }\left(\psi \right)={\psi }^{-2}exp(-{\psi }^{-1}),\psi >0,$$

(29)

which is a type of generalized extreme value distribution and has Cauchy-like heavy tails. Substituting (29), (28) into (3), we obtain the MML87 codelength

$$I_{87}(\psi ,\mathbf{y}|\mathbf{\delta })=(n+1)log\psi +\frac{1}{\psi }\left(1+\sum _{i=1}^n{y}_i\right)+\frac{1}{2}(1+logn-log12).$$

(30)

The MML87 estimate of the mean $\psi$ is

$${\widehat{\psi }}_{87}\left(\mathbf{y}\right)=\frac{s+1}{n+1}$$

(31)

where, as before, $s={\sum }_i{y}_i$. The MML87 estimate corresponds to the usual maximum likelihood estimate ${\widehat{\psi }}_{\mathrm{ML}}\left(\mathbf{y}\right)=s/n$ with one additional data point that has a unit contribution to the mean. The expected mean squared error of the MML87 estimate is

$$\mathbb{E}\left\{{({\widehat{\psi }}_{87}\left(\mathbf{y}\right)-\psi )}^2\right\}=\frac{\psi \left(\psi \right(n+1)-2)+1}{{(n+1)}^2}$$

(32)

which dominates the maximum likelihood estimate for

$$\psi >\frac{n}{n+\sqrt{n(2n+1)}},n>0.$$

(33)

As the sample size n increases, we note that

$$\underset{n\to \infty }{lim}\left\{\frac{n}{n+\sqrt{n(2n+1)}}\right\}=\sqrt{2}-1\approx 0.414$$

(34)

implying the MML87 estimate dominates maximum likelihood for all $\psi >0.414$ in terms of expected mean squared error for large n. However, we note that, unlike the MML87 estimate with this choice of prior distribution, the maximum likelihood estimate is invariant to scaling of the data.

Substituting $I_{87}(\widehat{\varphi },\mathbf{\delta })$ and $I_{87}(\widehat{\psi },\mathbf{y}|\mathbf{\delta })$ into (19) yields the total (conditional) codelength of the data. The MML87 estimates of the mean lifetime ${\widehat{\beta }}_{87}\left(D\right)$ and censoring time ${\widehat{\alpha }}_{87}\left(D\right)$ can be recovered from

$$\alpha \to \frac{\psi }{1-\varphi },\beta \to \frac{\psi }{\varphi },$$

(35)

for $\varphi \in (0,1)$. Next, we examine how the same message can be encoded using joint encoding of lifetime data and censoring indicators.

4.2. Joint Encoding of the Data

Unlike in the conditional encoding, the sender now transmits the survival times and the indicator variables simultaneously. The negative log-likelihood function of the data $D=\{(y_1,{\delta }_1),\dots ,(y_n,{\delta }_n)\}$ is given in (8). The Fisher information in this parameterization is

$$J(\alpha ,\beta )=\frac{n^2}{\beta \alpha {(\alpha +\beta )}^2}.$$

(36)

We would like to use prior distributions for $\alpha$ and $\beta$ that are comparable to those in the conditional coding described in Section 4.1. Noting that $\varphi \sim \mathrm{beta}(a,b)$, $\psi$ has the standard Frechet distribution and

$$\varphi =\frac{\alpha }{\alpha +\beta },\psi ={\left(\frac{1}{\alpha }+\frac{1}{\beta }\right)}^{-1},$$

(37)

the Jacobian of the transformation from $(\varphi ,\psi )\to (\alpha ,\beta )$ is

$$\frac{\alpha \beta }{{(\alpha +\beta )}^3}$$

(38)

implying that a commensurate joint prior distribution for $\alpha ,\beta$ is

$${\pi }_{\alpha ,\beta }(\alpha ,\beta )=\frac{{\alpha }^{a-2}{e}^{-\frac{\alpha +\beta }{\alpha \beta }}{\beta }^{b-2}{(\alpha +\beta )}^{-a-b+1}}{B(a,b)}$$

(39)

where $B(·,·)$ is the beta function. In the special case where $\varphi$ is given a uniform prior (i.e., $a=b=1$), we have

$${\pi }_{\alpha ,\beta }(\alpha ,\beta )=\frac{e^{-\frac{\alpha +\beta }{\alpha \beta }}}{\alpha \beta (\alpha +\beta )}$$

(40)

Substituting (36) and (39) into (15), the MML87 codelength is

$$\begin{array}{cc}\hfill I_{87}(D,\mathbf{\theta })& =klog\beta +(n-k)log\alpha +\left(\frac{1}{\alpha }+\frac{1}{\beta }\right)\left(\sum _{i=1}^n{y}_i\right)-log{\pi }_{\alpha ,\beta }(\alpha ,\beta )\hfill \\ \hfill & +logn-\frac{1}{2}log\left(\alpha \beta {(\alpha +\beta )}^2\right)+log{\kappa }_2+1\hfill \end{array}$$

(41)

where $\mathbf{\theta }=\{\alpha ,\beta \}$ and the quantization constant ${\kappa }_2=5/\left(36\sqrt{3}\right)$. The MML87 estimates that minimize the codelength (41) are

$$\begin{array}{cc}\hfill {\widehat{\alpha }}_{87}\left(D\right)& =\frac{2(s+1)(a+b+n-1)}{(n+1)(2b-2k+2n-1)},{\widehat{\beta }}_{87}\left(D\right)=\frac{2(s+1)(a+b+n-1)}{(n+1)(2a+2k-1)}.\hfill \end{array}$$

(42)

If required, the corresponding estimates of $\varphi$ and $\psi$ can be obtained from (37).

4.3. Properties

First we show the the conditional (19) and joint (41) MML codelengths are equivalent up to a constant to be specified below. From Section 4.1 and Section 4.2, we note the joint density of $(Y,\Delta )$ can be expressed as a product of the binomial $p_{\Delta }\left(\delta \right|\varphi )$ and exponential densities $p_Y\left(y\right|\psi )$

$$p_{Y,\Delta }(y,\delta |\alpha ,\beta )=p_{\Delta }\left(\delta \right|\varphi )p_Y\left(y\right|\psi )=\left({\varphi }^{\delta }{(1-\varphi )}^{1-\delta }\right)\left(exp\left(-y/\psi \right)/\psi \right)$$

where Y and $\Delta$ are independent random variables (see Kim [5] (p. 104)). Consequently, as MML87 is invariant under smooth one-to-one reparameterizations of the sampling model, the MML87 joint codelength (41) and the corresponding conditional codelength (19) are identical (except for the minor efficiency gain in the joint codelength discussed below). Specifically, the relationship between the joint codelength, $I_{87}(\alpha ,\beta ,D)$ and conditional codelength, $I_{87}(\psi ,\varphi ,D)$, can be expressed as

$$I_{87}(\psi ,\varphi ,D)=I_{87}(\alpha ,\beta ,D)+log\left(\frac{3\sqrt{3}}{5}\right)$$

(43)

where the term $log(3\sqrt{3}/5)\approx 0.0385$ arises due to the quantization constant being smaller in higher dimensions since its more efficient to encode multiple parameters simultaneously compared to encoding each parameter independently.

Furthermore, as the MML87 estimate of $\varphi$ is $\widehat{\varphi }\in (0,1)$ (see (24)), MML87 estimates of the mean survival times $(\alpha ,\beta )$ are finite for all $k\in [0,n]$ in contrast to the corresponding maximum likelihood estimates (9) which are finite for $k\in (0,n)$. As $n\to \infty$, it is well-known that the MML87 estimates are equivalent to the maximum likelihood estimates (see (18)) which implies that the MML87 estimates are similarly asymptotically normally distributed and strongly consistent.

The expected mean square error $\mathbb{E}\{{(\widehat{\beta }-{\beta }^{*})}^2\}$ of the ML and MML87 estimates of ${\beta }^{*}$, conditional on $k>0$, is expressible in terms of the generalized hypergeometric function for any $n>0$. Figure 1 (top) depicts the expected mean squared error between the MML87 and ML estimate of ${\beta }^{*}$, expressed as a ratio of MML87 to ML with smaller values indicating preference for the MML87 estimate. The expected mean squared error of the MML87 estimate of ${\beta }^{*}$ was generally lower than the corresponding maximum likelihood estimate (except when the true censoring proportion ${\varphi }^{*}$ was small) with the biggest difference observed for small sample sizes, while the two estimates were practically indistinguishable for larger sample sizes $n\ge 100$.

We also compared the MML87 and the ML estimates in terms of the relative entropy or the Kullback–Leibler (KL) divergence, conditional on $k\in (0,n)$. The KL divergence between the true data generating model $({\alpha }_1,{\beta }_1)$ and the approximating model $({\alpha }_2,{\beta }_2)$ is

$$D_{\mathrm{KL}}({\alpha }_1,{\beta }_1\left|\right|{\alpha }_2,{\beta }_2)=\frac{{\alpha }_1{\beta }_1\left({\alpha }_2+{\beta }_2\right)+{\alpha }_2{\beta }_2\left({\alpha }_{1}log\left(\frac{{\beta }_2}{{\beta }_1}\right)+{\beta }_{1}log\left(\frac{{\alpha }_2}{{\alpha }_1}\right)-{\alpha }_1-{\beta }_1\right)}{{\alpha }_2{\beta }_2\left({\alpha }_1+{\beta }_1\right)},$$

which, as expected, is the sum of the KL divergences between two exponential and two binomial distributions. The KL divergence may be interpreted as the expected amount of extra information required to encode data from $({\alpha }_1,{\beta }_1)$ using the model $({\alpha }_1,{\beta }_2)$. The expected KL divergence for both the ML and MML estimators is shown in the bottom of Figure 1, conditional on $k>0$ and $k<n$. It is clear that for $n=5$ the MML87 estimate dominates the maximum likelihood estimate in terms of the KL divergence for all ${\varphi }^{*}\in (0.05,0.95)$. When the sample size is increased ($n=10$), the MML87 estimate exhibits smaller KL divergence compared to the ML estimate for all ${\varphi }^{*}$ except when ${\varphi }^{*}\to 0$ or ${\varphi }^{*}\to 1$ where the maximum likelihood estimate has smaller KL divergence.

5. Minimum Message Length Inference with Fixed Censoring

Consider now the fixed censoring scenario (6) introduced in Section 2 where the negative log-likelihood function of the data is given in (10). If we wish to encode the data D using the joint MML code (see Section 4.2), we require the negative log-likelihood, the Fisher information and a prior distribution for the mean survival time $\theta >0$. The negative log-likelihood is given in (10) while the Fisher information for Type I censored data with fixed censoring is:

$$J_{\theta }(\theta ;c)=\frac{n(1-exp(-c/\theta \left)\right)}{{\theta }^2}=\frac{n{F}_T\left(c\right|\theta )}{{\theta }^2},$$

(44)

where $F_T(·|\theta )$ is the cumulative distribution function of the survival data T (see Section 2). The reduction in information due to censoring is clearly a function of $\theta$ and the cumulative density function of T, with large c resulting in little information loss compared to small c. As expected, as c gets larger

$$\underset{c\to \infty }{lim}{J}_{\theta }(\theta ;c)=\frac{n}{{\theta }^2},$$

(45)

which is the usual Fisher information for the exponential distribution with no censoring. The prior distribution for $\theta$ is chosen to be the Fréchet ditribution with scale c and probability density function

$${\pi }_{\theta }(\theta ;c)=c^{-1}{\left(\frac{\theta }{c}\right)}^{-2}exp\left(-\frac{c}{\theta }\right).$$

(46)

Substituting (44) and (46) into (15), we obtain the complete MML87 codelength for the joint encoding

$$\begin{array}{cc}\hfill I_{87}(D,\theta )& =(k+1)log\left(\theta \right)+\frac{1}{\theta }\left(c((n-k)+1)+\sum _{i=1}^n{y}_i{\delta }_i\right)\hfill \\ \hfill & +\frac{1}{2}log\left(\frac{1-exp(-c/\theta )}{c^2}\right)+\frac{1}{2}(1+logn-log12).\hfill \end{array}$$

(47)

Due to the form of the Fisher information, the MML87 estimate of $\theta$ that minimizes this codelength is unavailable analytically and must be obtained via numerical optimisation. The maximum likelihood estimate (11) may be used as a starting point for the numerical search.

Consider now the conditional encoding (see Section 4.1) where the probability of observing an uncensored datum, say $\varphi >0$, is

$$\varphi =P(T_i\le c)=F_T\left(c\right|\theta )=1-exp(-c/\theta ),(i=1,\dots ,n),$$

(48)

so that the number of uncensored data points k follows the binomial distribution $k\sim \mathrm{binomial}(\varphi ,n)$. This implies that the mean survival time can then written as

$$\theta =-c/log(1-\varphi ),\varphi \in (0,1).$$

(49)

A naive conditional coding approach proceeds by encoding the censoring indicators following Section 4.1 with codelength (23). To encode $I\left(\mathbf{y}\right|\mathbf{\delta })$, one would use the conditional probabilities of the lifetime data which are

$$\begin{array}{cc}\hfill p_{Y|\Delta }(Y_i=y_i|{\Delta }_i=1)& =\frac{p(T_i=y_i)}{p(T_i\le c)}\mathrm{if}{y}_i\le c,\hfill \end{array}$$

(50)

and

$$p_{Y|\Delta }(Y_i=c|{\Delta }_i=0)=1,p_{Y|\Delta }(Y_i>c|{\Delta }_i=0)=0.$$

(51)

for all $i=1\dots ,n$. The conditional likelihood of the $k=\left({\sum }_i{\delta }_i\right)$ data points is then

$$\begin{array}{cc}\hfill p\left(\mathbf{y}\right|\theta ;\mathbf{\delta }=1)& =\prod _{i:{\delta }_i=1}\frac{(1/\theta )exp(-y_i/\theta )}{1-exp(-c/\theta )}=-\prod _{i:{\delta }_i=1}\frac{{(1-\varphi )}^{y_i/c}log(1-\varphi )}{c\varphi }.\hfill \end{array}$$

(52)

Once the receiver has the censoring data and an estimate of $\varphi$, they implicitly know $\theta$ from (49). The length of the message required to transmit the data $\mathbf{y}$ is

$$\begin{array}{cc}\hfill I_{87}(\theta ,\mathbf{y}|\mathbf{\delta })& =klog\theta +\frac{1}{\theta }\sum _{i:{\delta }_i=1}{y}_i+klog(1-exp(-c/\theta ))\hfill \\ \hfill & =-\frac{1}{c}\left(\sum _{i:{\delta }_i=1}{y}_i\right)log(1-\varphi )+klogc\varphi -klog(-log(1-\varphi ))\hfill \end{array}$$

(53)

which is the negative log-likelihood of the data. However, this codelength is inefficient since the probability of censoring $\varphi \left(\theta \right)$ is not independent of the mean survival time $\theta$. This implies that the precision to which $\varphi \left(\theta \right)$ is encoded must depend on the lifetime data $\mathbf{y}$, which is not the case in the naive approach where the precision quantum for $\varphi \left(\theta \right)$ depends on the censoring data $\mathbf{\delta }$ only. Consequently, joint MML coding should be used instead of the conditional encoding approach for the fixed censoring setup.

5.1. Example

We observe $n=20$ items with an exponential life distribution for $c=150$ h. Out of the 20 items $k=15$ items fail during the observation period and the sum of their lifetimes (in hours) is $s={\sum }_i{y}_i{\delta }_i=835$ [6]. The maximum likelihood estimate of the mean lifetime $\theta$ is

$${\widehat{\theta }}_{\mathrm{ML}}\left(D\right)=\frac{150\left(5\right)+835}{15}=105.6\mathrm{h},$$

(54)

with a negative log-likelihood of $84.9$ at the minimum. The MML87 estimate is obtained by a numerical search and is ${\widehat{\theta }}_{87}\left(D\right)=110.1$ h with a codelength of $124.905$ bits. The MML87 codelength at the maximum likelihood estimate is 124.924 bits suggesting that there is little difference between the two estimates in this example.

5.2. Properties

To evaluate the performance of the MML87 estimate, we computed the mean squared error risk and the expected Kullback–Leibler (KL) divergence for MML87 and the ML estimates under the data generating model ${\theta }^{*}=1$ and sample sizes $n\in \{5,25\}$. Since the ML estimate is undefined for $k=0$, all the results discussed below are conditional on $k>0$. The KL divergence from the ‘true’ model ${\theta }_1$ to the approximating model ${\theta }_2$ is

$$D_{\mathrm{KL}}({\theta }_1\left|\right|{\theta }_2)=\left(1-\frac{{\theta }_1}{{\theta }_2}+log\left(\frac{{\theta }_1}{{\theta }_2}\right)\right)\left(exp\left(-\frac{c}{{\theta }_1}\right)-1\right).$$

(55)

The results are shown in Figure 2 where the x-axis of each plot is the censoring point c set to the p-th percentile of the data generating model Exp(${\theta }^{*}=1$); for example $c=0.69$ corresponds to the $p=0.50$-th percentile of Exp(${\theta }^{*}=1$). It is clear that the MML87 estimate is a reasonable alternative to the maximum likelihood estimate under fixed censoring. For $p\ge 0.20$ (i.e., the 20-th percentile of Exp(${\theta }^{*}=1$)) the MML87 estimate dominates the ML estimate in terms of KL risk, while for $p<0.20$ the estimates are very similar for both sample sizes tested. In terms of the expected mean squared error, the MML87 estimate dominates the ML estimate for all $0.20\le p\le 0.80$ when $n=5$; for $n=25$, the estimates are indistinguishable for $p<0.1$ and $p>0.69$ and the MML87 estimate again dominates ML for all $0.1<p<0.5$.

6. Discussion

This manuscript has demonstrated how minimum message length can be used to infer data with censoring information. Specifically, we have derived MML87 codelengths for the exponential distribution with fixed censoring and random type I censoring. Although information theoretic universal models for the exponential distribution, including those corresponding to MML codes, are known [4], this is the first time MML has been applied to censored data.

The MML87 codelength for the exponential distribution with censoring provides a new means of parameter estimation as well as model selection. In terms of parameter estimation, the MML87 estimate of the mean lifetime $\theta$ under type I censoring described in this paper has some advantages over the usual maximum likelihood estimate for small sample sizes. First, the MML87 estimate is defined for all proportions of censoring unlike the maximum likelihood estimate which does not exist when all observations are censored; i.e., $k=\left({\sum }_i{\delta }_i\right)=0$. In addition, the MML87 estimate has on average lower mean squared error risk and lower KL divergence from the data generating model for a wide range of censoring proportions.

In the case of random censoring, the MML87 estimate is available in closed-form while for fixed censoring, it can only be obtained by numerical optimisation. Although the experiments in the manuscript utilised heavy-tailed prior distributions for the scale parameter as recommended in, for example, [26], the Bayesian nature of MML allows for information prior information to be incorporated directly into the estimation process. The effect of the prior distribution in the examples considered here is expected to be negligible for medium to large sample sizes.

Importantly, the proposed MML87 codelengths can also be used to discriminate between competing models (e.g., exponential vs lognormal) and offer some advantages over the well-known BIC model selection approach. BIC only considers the sample size and the number of parameters when measuring model complexity. In contrast, MML takes into account not just the number of parameters, but also the complexity of the distribution (i.e., the number of random data strings that are fitted well by the distribution). As the sample size $n\to \infty$, the MML87 codelength converges to the BIC and therefore inherits the favourable asymptotic properties of BIC, such as model selection consistency.

The codelengths derived in this manuscript are extendable to MML inference of other censored data types, such as the Weibull and the lognormal distribution, and can be incorporated into more complex models as shown in the next section.

6.1. Clustering Survival Data

To demonstrate the applicability of the codes derived in the manuscript, we implemented the MML87 codes into a Matlab software package for inference of finite mixture models. Our software, called Matlab Snob, features mixture models with categorical data (e.g., multinomial distribution), count data (e.g., geometric, Poisson and negative binomial distributions), continuous data (e.g., normal, Laplace, gamma and Weibull distributions) and survival data (type I fixed and random censored exponential distribution). As a demonstration of Matlab Snob, we used two publicly available survival data sets: (1) Rossi et al.’s criminal recidivism data [27], and (2) survival from malignant melanoma [28].

The crime data was recently analyzed in [29] using variational Bayes estimated finite mixture models. For clustering we used all $n=432$ observations and the following seven attributes: (1) financial aid (no, yes), (2) full-time work experience before incarceration (no, yes), (3) marital status at time of release (married, not married), (4) released on parole (no, yes), (5) number of convictions prior to current incarceration, (6) age in years at time of release and (7) week of first arrest after release (73.6% censored). This is an example of fixed censoring as all censored observations were censored at 52 weeks. We modelled the categorical attributes using a multinomial distribution, number of convictions was modelled with a negative binomial distribution, while a Gaussian distribution was used for age at time of release. For the week of arrest, we used the exponential distribution model with fixed type I censored data (see Section 5).

The melanoma data set consists of $n=205$ patients from Denmark who were diagnosed with malignant melanoma. Five attributes were used for clustering: (1) sex (male, female), (2) ulcer (present, absent), (3) age at diagnosis in years, (4) tumour thickness in mm, and (5) censored survival time in years (65.3% censored). Sex and ulcer were modelled via multinomial distributions, while age and tumour thickness were modelled with univariate Gaussian distributions. For the survival time, we used an exponential distribution with random type I censoring (see Section 4) and combined death due to melanoma and death due to other causes as the primary outcome of interest.

Clustering results for the Crime and the Melanoma data sets with Matlab Snob are shown in

Table 1. MML finite mixture models for Crime and Melanoma data. The attribute modelling censored survival time is seven in the Crime data set and five in the Melanoma data set.

Data	Class	Attributes
		1	2	3	4	5	6	7
Crime	1	(50%, 50%)	(73%, 27%)	( 3%, 97%)	(42%, 58%)	(r: 2.0, p: 0.4)	($\mu$: 20.7, $\sigma$: 2.1)	($\mathbf{\theta }$: 119.0)
	2	(55%, 45%)	(15%, 85%)	(23%, 77%)	(28%, 72%)	(r: 13.4, p: 0.8)	($\mu$: 24.9, $\sigma$: 3.3)	($\mathbf{\theta }$: 249.3)
	3	(40%, 60%)	(16%, 84%)	(18%, 82%)	(51%, 49%)	(r: 2.3, p: 0.5)	($\mu$: 36.3, $\sigma$: 4.9)	($\mathbf{\theta }$: 345.9)
Melanoma	1	(43%, 57%)	(17%, 83%)	($\mu$: 57.3, $\sigma$: 17.1)	($\mu$: 5.4, $\sigma$: 3.4)	($\mathbf{\alpha }$: 12.0, $\mathbf{\beta }$: 7.3)	–	–
	2	(73%, 27%)	(80%, 20%)	($\mu$: 49.5, $\sigma$: 15.8)	($\mu$: 1.4, $\sigma$: 0.9)	($\mathbf{\alpha }$: 8.1, $\mathbf{\beta }$: 37.0)	–	–

. First, since Matlab Snob learns finite mixture models using the MML87 codelength approximation, the same framework is used to estimate all model parameters as well as select the number of classes. For the Crime data, the model with three classes had the smallest codelength, while two classes were selected for the Melanoma data set. We observe that all the classes are relatively well differentiated in terms of average survival time. In the case of the Crime data set, class 1 had the shortest average time to arrest ($\theta =119$ weeks) and consists of younger individuals (mean age 20.7 years, std. dev. 2.1 years) who are primarily unmarried and have no full-time work experience before incarceration. In contrast, class 3 comprised older individuals (mean age 36.3 years, std. dev. 4.9 years) 82% of which had full-time work experience, was estimated to have longest average time to arrest ($\theta =345.9$ weeks). For the melanoma data set, class 1 was estimated to have the shortest average survival time ($\beta =7.3$ years) and consisted of individuals diagnosed at an older age (mean: 57.3 years, std. dev. 17.1 years) with larger tumours (mean: 5.4 mm, std. dev. 3.4 mm). In contrast, patients assigned to class 2 were diagnosed at a younger age, had smaller tumours and were estimated to have longer survival time on average ($\beta =37.0$ years).

The Matlab Snob clustering software is freely available for download from the Mathworks Filexchange website (ID: 72310) and will be extended to incorporate other survival distributions in the future (eg, Weibull and lognormal distribution). We note that the MML87 codelengths for type I censored exponentially distributed data derived in this paper can also be used in decision tree modelling [21,30,31]. For example, one could represent the leaves of the decision tree with a censored exponential distribution and use MML to infer an optimal decision tree for a data set.

6.2. Minimum Message Length and Minimum Description Length

Minimum message length is closely related to minimum description length (MDL), an inductive inference principle independently developed by Rissanen and colleagues [32,33,34,35]. Like MML, the MDL principle is rooted in information theory and, given a data set, seeks a model that would result in the shortest encoding of the data. A recent and popular version of the MDL principle is the normalized maximum likelihood (NML) code which says that the codelength for data $\mathbf{y}$ with respect to model class $\mathcal{M}$ parameterised by models $\mathbf{\theta }\in {\mathbb{R}}^p\in \mathcal{M}$ is

$$-log{p}_{\mathrm{NML}}\left(\mathbf{y}\right|\mathcal{M})=-logp\left(\mathbf{y}\right|\widehat{\theta }\left(\mathbf{y}\right),\mathcal{M})+log\sum _{\mathbf{x}}p\left(\mathbf{x}\right|\widehat{\theta }\left(\mathbf{x}\right),\mathcal{M})$$

(56)

where $\widehat{\mathbf{\theta }}$ is the maximum likelihood estimate of the p parameters and the sum in the second term is taken over the entire data space; we replace the sum with an integral in the case of continuous data. The first term in the NML codelength is the negative log-likelihood of the data evaluated at the maximum likelihood estimate, while the second term represents the parametric complexity of the model class and measures how well models $\mathbf{\theta }\in \mathcal{M}$ within the model class $\mathcal{M}$ approximate random data sequences. In particular, a high parametric complexity says that a large number of data sequences can be well-approximated by models within the class. In contrast, the parametric complexity of a simple model that can only well-approximate a few data sequences will tend to be small.

Rissanen [33] derives an asymptotic approximation for the NML codelength which is accurate for medium to large sample sizes:

$$-log{p}_{\mathrm{NML}}\left(\mathbf{y}\right|\mathcal{M})=-logp\left(\mathbf{y}\right|\widehat{\mathbf{\theta }}\left(\mathbf{y}\right),\mathcal{M})+log{\int }_{\Theta }\sqrt{|J_1\left(\mathbf{\theta }\right)|}+\frac{p}{2}log\left(\frac{n}{2\pi }\right)+o\left(1\right)$$

(57)

where $J_1(·)$ is the per-sample Fisher information matrix. Mera et al. [36] derive a somewhat sharper approximation to the NML codelength using Riemannian geometry tools and apply their new approximation to principal component analysis. Rissanen further shows that, like the MML87 codelength, the NML codelength reduces to the well-known Bayesian information criterion (BIC) in the limit as the sample size $n\to \infty$. Unfortunately, in the case of the exponential distribution with or without type I censoring, the parametric complexity is infinite for both the exact NML codelength and the asymptotic approximation. To circumvent the problem of infinite parametric complexity, one may consider the restricted approximate normalized maximum likelihood (ANML), the two-part ANML or the objective Bayesian code, among others [37].

Although there exist many similarities in the approaches to inference between MML and MDL, there are some important differences which we summmarize below:

• MDL relies on the maximum likelihood estimator and does not offer new means for parameter estimation;
• MDL is decidedly non-Bayesian avoiding the use of any (subjective or objective) prior information;
• MDL nominates the model class $\mathcal{M}$ that would result in the shortest encoding of the data and does not does not infer a fully specified model;
• while MML minimises the expected (average) codelength of the data with respect to the marginal data distribution, MDL minimizes the worst-case codelength relative to the ideal code.

In addition to the NML code, other MDL codes exist including the sequential NML code [38] and the conditional NML distribution [34], among others. Clearly, both MML and MDL approaches to inductive inference have merit, and if used correctly, will result in excellent model selection performance as shown in a wide range of applications. A more detailed discussion of MML and MDL similarities and differences can be found in [39] and [1] (pp. 413–415).