Belief Entropy Tree and Random Forest: Learning from Data with Continuous Attributes and Evidential Labels

Abstract

As well-known machine learning methods, decision trees are widely applied in classification and recognition areas. In this paper, with the uncertainty of labels handled by belief functions, a new decision tree method based on belief entropy is proposed and then extended to random forest. With the Gaussian mixture model, this tree method is able to deal with continuous attribute values directly, without pretreatment of discretization. Specifically, the tree method adopts belief entropy, a kind of uncertainty measurement based on the basic belief assignment, as a new attribute selection tool. To improve the classification performance, we constructed a random forest based on the basic trees and discuss different prediction combination strategies. Some numerical experiments on UCI machine learning data set were conducted, which indicate the good classification accuracy of the proposed method in different situations, especially on data with huge uncertainty.

1. Introduction

Decision trees have been widely used for their good learning capabilities and ease of understanding. In some real world issues, instances may be ill-known for some factors such as randomness, data incompleteness and even expert’s indefinite subjective opinions; however, traditional decision trees can only handle certain samples with precise data. The incompletely observed instances are usually ignored or replaced by a precise one, despite the fact that they may contain useful information [1], which may cause a loss of accuracy.

There have been many attempts to build trees from incomplete data in the past several decades. The probability trees [2,3] were suggested based on probability theory, which is usually intuitively the first tool to modeling uncertainty in practice; however, it has been proven that probability cannot always be adequate for representing data uncertainty [4,5] (often termed epistemic uncertainty). To overcome this drawback, various approaches have been proposed, including: fuzzy decision trees [6,7], the possibilistic decision trees [8] and the uncertain decision trees [9,10]. Besides the aforementioned methods, a more general framework, called the belief function theory [11,12] (also evidential theory or Dempster–Shafer theory), has been proven to have the ability to model all kinds of knowledge. The process of embedding belief functions within decision tree techniques has already been extensively investigated [13,14,15,16,17,18,19,20,21,22,23,24,25] in recent years. Particularly, among these methods, several trees [17,18,19] estimate parameters by maximizing evidential likelihood function using the $E^{2}M$ algorithm [26,27], which is also the basis of part of the trees to be proposed in this paper.

However, the existing methods on incomplete data do not take continuous attributes into full consideration. These proposals deal with uncertain data modeled by the belief function and build trees by extending the traditional decision tree method. The imitation and transformation decides to use existing methods to handle continuous attribute values by discretization, which brings about an issue of losing the detail of the training data. For example, the information gain ratio, the attribute selecting measurement in C4.5, was transformed to adapt the evidential labels of the training set in the Belief C4.5 trees [19], in which the continuous-valued attribute is divided into four intervals of equal width before learning. This issue leads to the purpose of this paper: to learn from uncertain data with continuous attribute values without pretreatment.

To realize this purpose, we firstly, for each attribute, fit the training data to a Gaussian mixture model (GMM), which consists of normal distribution models one-by-one corresponding to class labels, by adopting the $E^{2}M$ algorithm. This step, which significantly differs from other decision trees, confirms the ability to deal with ill-known labels and original attribute values (either discrete or continuous). On the basis of these GMM models, we generate the basic belief assignment (BBA) and calculate belief entropy [28]. The attribute with minimal average entropy, which distinguishes classes from each others most, will be selected as the splitting attribute. The following decision tree induction steps are designed accordingly and logically. To our knowledge, this paper is the first to introduce GMM models and belief entropy to decision trees with evidential data.

Another part of our proposal is adopting the ensemble method for our belief entropy trees. Inspired by the idea of building bagging trees based on random sampling [29], we further choose a more efficient and popular technique—random forest [30]. Under the belief function framework, the basic trees will output either precise or mass (modeled by BBA) label predictions, while traditional random forest can only combine precise labels. Thus, a new method to summarize the basic tree predictions is proposed to combine mass labels directly, instead of voting on precise labels. This combined mass keeps the uncertain information of data as much as possible, which helps to generate a more reasonable prediction. The new combination method is discussed and compared to the traditional majority voting method later.

We note that we have proposed our early work in a shorter conference paper [31]. Compared with our initial conference paper, we have fixed the attribute selection and splitting strategy of a single tree and introduced ensemble learning to our tree method in this paper.

Section 2 recalls some basic knowledge about decision trees, belief function theory, the $E^{2}M$ algorithm and belief entropy. Section 3 details the induction procedure of belief entropy methods and proposes three different instance prediction techniques. In Section 4, we introduce how to expend the single belief entropy tree to random forests and discuss the different predicting combination strategies. In Section 5, we detail experiments that were carried out on some classical UCI machine learning data sets to compare the classification accuracies of proposed trees and random forests. Finally, conclusions are summarized in Section 6.

2. Settings and Basic Definitions

The purpose of a classification method is to build a model that maps an attribute vector $X=\left(x^1,\dots ,x^D\right)\in A^1\times A^2\times \cdots \times A^D$, which contains D attributes, to an output class $y\in \mathcal{C}=\left\{C_1,\dots ,C_K\right\}$ taking its value among K classes. Each attribute discretely has finite values or continuously takes value within an interval. The learning of classification is based on a complete training set of precise data which contains N instances, denoted as

$$T=\left(\begin{array}{c}{X}_1,y_1\\ ⋮\\ X_N,y_N\end{array}\right)=\left(\begin{array}{c}{x}_1^1,\dots ,x_1^D,y_1\\ ⋮\\ x_N^1,\dots ,x_N^D,y_N\end{array}\right).$$

However, the imperfect knowledge about the inputs (feature vector) and the outputs (classification labels) exists widely in practical applications. Traditionally and regularly, the imperfect knowledge is modeled by probability theory, which is considered to be questionable in a variety of scenarios. Hence, we model uncertainty by belief function in this paper. Typically, we consider that attribute values are precise and can be either continuous or discrete, while only the output labels are uncertain.

2.1. Decision Trees

Decision trees [32] are regarded as one of the most effective and efficient machine learning methods and widely adopted for solving classification and regression problems in practice. The success, to a great extent, relays on the easily understandable structure, for both humans and computers. Generally, a decision tree is induced top-down from a training set T, which recursively repeats the steps below:

• Select an attribute, through a designed selection method, to generate a partition of a training set;
• Split the current training set to several subsets and put them into child node;
• Generate a leaf node and determine the prediction label for a child node when a stop criterion is satisfied.

Differing in the attribute selection methods, several decision tree algorithms have been proposed, such as ID3 [32], C4.5 [33] and CART [34]. Among these trees, the ID3 and C4.5 choose entropy as an information measure to compute and evaluate the quality of a node split by a given attribute.

The core of ID3 is information gain. Given training data T and an attribute A with $K_A$ modalities, the information gain will be:

$$Gain\left(T,A\right)=Info\left(T\right)-Inf{o}_A\left(T\right)$$

(1)

where

$$Info\left(T\right)=-\sum _{i=1}^K{\theta }_i{log}_2\left({\theta }_i\right)$$

(2)

and

$$Inf{o}_A\left(T\right)=-\sum _{i=1}^{K_A}\frac{\left|T_i\right|}{\left|T\right|}Info\left(T_i\right)$$

(3)

where ${\theta }_i$ is the proportion of instances in T that are of class $C_i$, $\left|T\right|$ and $\left|T_i\right|$ are the cardinalities of the instance sets belonging to a parent node and to the child node i.

The limitation of information gain is that attributes with largest values will be most promoted  [33], which leads to the $GainRatio$ in the C4.5 algorithm. It is given as:

$$GainRatio\left(T,A\right)=\frac{Gain\left(T,A\right)}{SplitInfo\left(T,A\right)}$$

(4)

where

$$SplitInfo\left(T,A\right)=-\sum _{i=1}^{K_A}\frac{\left|T_i\right|}{\left|T\right|}{log}_2\frac{\left|T_i\right|}{\left|T\right|}.$$

(5)

The attribute with the largest gain ratio will be selected for splitting.

We can easily find the the Equation (2) is actually the Shannon Entropy. Yet in this paper, concerning the feature of evidential data described by the framework of belief function, the attribute selection method is newly designed based on belief entropy [28] instead of Shannon entropy.

2.2. Random Forest

To improve the classification accuracy and generalization ability of machine learning, the ensemble model method is introduced to the learning procedure. One important branch of ensemble method is called $bagging$, which concurrently builds multiple basic models learning from different training sets, which are generated from original data by bootstrap sampling. On the basis of bagging decision trees, random forest (RF) [30] not only chooses the training instance randomly but also introduces randomness into attributes selection. To be specific, traditional decision trees select the best splitting attribute among all D attributes; random forest generates a random attribute subset then chooses the best one within this subset to split the tree node. The size $D^{\prime }$ of this subset is adjustable and generally set as $D^{\prime }=lo{g}_{2}D$.

A detailed description of the mathematical formulation of RF model is found in [30]. The RF model consists of a union of multiple basic trees, where each tree learns from bootstrap samples and selects attribute from a small subset of all attributes. There some advantages of RF: (a) better prediction performance, (b) resistance to overfitting, (c) low correlation of individual trees, (d) low bias and low variance and (e) small computational overhead.

Some existing works have explored the ensemble method on belief decision trees, such as bagging  [29]. In this paper, we apply the random forest technique to the proposed belief entropy trees and discuss the different prediction determining strategies.

2.3. Belief Function Theory

Let the finite set $\mathrm{\Omega }$ denote the frame of discernment containing k possible exclusive values that a variable can take. When considering the output y, the imperfect knowledge about value of y can be modeled by mass function $m_y:2^{\mathrm{\Omega }}\to \left[0,1\right]$, such that $m_y\left(⌀\right)=0$, and

$$\sum _{A\subseteq \mathrm{\Omega }}{m}_y\left(A\right)=1,$$

(6)

which is also called a basic belief assignment (BBA). The subset A is called a focal set where $m_y\left(A\right)>0$, and the $m_y\left(A\right)$ can be interpreted as the support degree of the evidence towards the case that true value is in set A.

There are some typical mass functions need to be attended:

• $Vacuous$ mass: mass function such that $m_y\left(\mathrm{\Omega }\right)=1$, which means total ignorance;
• $Bayesian$ mass: for all focal set A, the cardinality $\left|A\right|=1$. In this case, the mass degenerates to a probability distribution;
• $Logical\left(categorical\right)$ mass: $m_y\left(A\right)=1$ for some A. In this case, the mass is equivalent to the set A.

One-to-one related to the mass function $m_y$, the $belieffunction$ and $plausibilityfunction$ are defined as:

$$Be{l}_y\left(B\right)=\sum _{A\subseteq B}{m}_y\left(A\right),$$

(7)

$$P{l}_y\left(B\right)=\sum _{A\cap B\ne \oslash }{m}_y\left(A\right),$$

(8)

which, respectively, indicate the minimum and maximum belief degree of evidence towards set B. Typically, the function $pl:\mathrm{\Omega }\to \left[0,1\right]$ such that $p{l}_y\left(\omega \right)=P{l}_y\left(\left\{\omega \right\}\right)$ for all $\omega \in \mathrm{\Omega }$ is called $contourfunction$ associated to $m_y$.

For two mass function $m_1$ and $m_2$ induced by evidences independently, they can be combined by the $Dempste{r}^{\prime }srule$ [12]⊕ defined as:

$$\left(m_1\oplus m_2\right)\left(A\right)=\frac{1}{1-\kappa }\sum _{B\cap C=A}{m}_1\left(B\right)m_2\left(C\right),$$

(9)

for all $A\subseteq \mathrm{\Omega },A\ne ⌀$, and $\left(m_1\oplus m_2\right)\left(⌀\right)=0$, where

$$\kappa =\sum _{B\cap C=⌀}{m}_1\left(B\right)m_2\left(C\right),$$

(10)

is called the $degreeofconflict$ between $m_1$ and $m_2$. Obviously, Dempster’s rule is commutative and associative according to the definition.

In the decision making situation, we need to determine the most reasonable hypothesis from a mass. Different decision-making strategies with belief functions [35,36] have been researched. Among these methods, in the transferable belief model (TBM), $pignisticprobability$ [37] was proposed to make decision from a BBA:   

$$BetP\left(\omega \right)=\sum _{A\subseteq \Omega ,\omega \in A}\frac{m\left(A\right)}{\left|A\right|},$$

(11)

where $\left|A\right|$ is the cardinality of A.

When we model uncertain labels of evidential data with mass functions, the training set becomes

$$T=\left(\begin{array}{c}{X}_1,m_1\\ ⋮\\ X_N,m_N\end{array}\right)=\left(\begin{array}{c}{x}_1^1,\dots ,x_1^D,m_1\\ ⋮\\ x_N^1,\dots ,x_N^D,m_N\end{array}\right).$$

2.4. Evidential Likelihood

Consider a discrete random vector Y taking values in $\mathrm{\Omega }$ with a probability mass function $p_Y\left(y;\mathit{\theta }\right)$ assumed to be associated with a parameter $\mathit{\theta }\in \Theta$. After a realization y of Y has been perfectly observed, the likelihood function of $completedata$ is defined as $L:\Theta \to \left[0,1\right]$ such that

$$L\left(\mathit{\theta };y\right)=p_Y\left(y;\mathit{\theta }\right),\forall \mathit{\theta }\in \Theta .$$

(12)

When the observations are uncertain, it is impossible to evaluate parameter $\mathit{\theta }$ from a likelihood function. In this situation, a new statistical tool [38] called evidential likelihood was proposed to perform parameter estimation. Assume that y is not precisely observed, but is known surely that $y\in A$ for some $A\in \mathrm{\Omega }$. Given such $imprecisedata$, the likelihood function will be extended to

$$L\left(\mathit{\theta };A\right)=p_Y\left(A;\mathit{\theta }\right)=\sum _{y\in A}{p}_Y\left(y;\mathit{\theta }\right),\forall \mathit{\theta }\in \Theta .$$

(13)

Furthermore, the observation of instance y could be not only $imprecise$, but also $uncertain$, which is modeled by mass function $m_y$. Thus the evidential likelihood function [27] can be defined as

$$\begin{array}{cc}\hfill L\left(\mathit{\theta };m_y\right)& =\sum _{A\subseteq \mathrm{\Omega }}L\left(\mathit{\theta };A\right)m_y\left(A\right)\hfill \\ \hfill & =\sum _{y\in \mathrm{\Omega }}{p}_Y\left(y;\mathit{\theta }\right)\sum _{A\ni x}{m}_y\left(A\right)\hfill \\ \hfill & =\sum _{y\in \mathrm{\Omega }}{p}_Y\left(y;\mathit{\theta }\right)pl\left(y\right),\forall \mathit{\theta }\in \Theta \hfill \end{array},$$

(14)

where the $pl$ is the contour function related to $m_y$ and the $L\left(\mathit{\theta };m_y\right)$ can be remarked as $L\left(\mathit{\theta };pl\right)$. According to the statement of Denoeux [27], the value $1-L\left(\mathit{\theta };pl\right)$ equals to the conflict between parametric model $p_Y\left(y;\mathit{\theta }\right)$ and the uncertain observation $pl$, which means minimizing $L\left(\mathit{\theta };pl\right)$ is actually a procedure of estimating the best parameter $\mathit{\theta }$ to fit the parametric model to observation as closely as much.

Equation (14) also indicates that $L\left(\mathit{\theta };pl\right)$ can be remarked as the expectation of $pl$ such that

$$L\left(\mathit{\theta };pl\right)={\mathbb{E}}_{\mathit{\theta }}\left[pl\left(Y\right)\right].$$

(15)

Assume that $Y=\left(y_1,\dots ,y_N\right)$ is a sample set containing n cognitively independent [12] and i.i.d. uncertain observations, in which the $y_i$ is model by $m_{y_i}$. In the situation the Equation (15) is written as a product of n terms:

$$L\left(\mathit{\theta };pl\right)=\prod _{n=1}^N{\mathbb{E}}_{\mathit{\theta }}\left[p{l}_n\left(y_n\right)\right].$$

(16)

2.5. E${}^2$M Algorithm

Though an extension of likelihood function, the maximum likelihood estimation of evidential likelihood can not directly be computed by the broadly applied EM algorithm [39]. The $E^{2}M$ algorithms [27] introduced by Denoeux allow us to maximize the evidential likelihood iteratively, which is composed of two steps (similar to EM algorithm):

1.

The E-step require firstly a probability mass function $p_Y\left(·\mid pl;{\mathit{\theta }}^{\left(q\right)}\right)=p_Y\left(·;{\mathit{\theta }}^{\left(q\right)}\right)\oplus pl$, in which the former part means the probability mass function of Y under the parameter ${\mathit{\theta }}^{\left(q\right)}$ estimated from last iteration and the latter part indicates contour function $pl$. The expression is:

$$p_Y\left(y\mid pl;{\mathit{\theta }}^{\left(q\right)}\right)=\frac{p_Y\left(y;{\mathit{\theta }}^{\left(q\right)}\right)pl\left(y\right)}{L\left({\mathit{\theta }}^{\left(q\right)};pl\right)}.$$

(17)

Then calculate the expectation of log likelihood $log{L}_c\left(\mathit{\theta };y\right)=log{p}_Y\left(y;\mathit{\theta }\right)$ of complete data with respect to $p_Y\left(·\mid pl;{\mathit{\theta }}^{\left(q\right)}\right)$,

$$Q\left(\mathit{\theta },{\mathit{\theta }}^{\left(q\right)}\right)=\frac{{\sum }_{y\in \mathrm{\Omega }}log\left(L_c\left(\mathit{\theta };y\right)\right)p_Y\left(y;{\mathit{\theta }}^{\left(q\right)}\right)pl\left(y\right)}{L\left({\mathit{\theta }}^{\left(q\right)};pl\right)}.$$

(18)

2.

The M-step is to maximize $Q\left(\mathit{\theta },{\mathit{\theta }}^{\left(q\right)}\right)$ with respect to $\mathit{\theta }$, obtaining a new estimation that ensures $Q\left({\mathit{\theta }}^{\left(q+1\right)},{\mathit{\theta }}^{\left(q\right)}\right)⩾Q\left(\mathit{\theta },{\mathit{\theta }}^{\left(q\right)}\right)$.

The two steps repeat until $L\left({\mathit{\theta }}^{\left(q+1\right)}\right)-L\left({\mathit{\theta }}^{\left(q\right)}\right)⩽ϵ$, where $ϵ$ is a set threshold.

2.6. Belief Entropy

Inspired by Shannon entropy [40], which can measure uncertainty contained by a probability distribution, a type of belief entropy called Deng entropy is proposed by Deng [28] to handle situation where the traditional probability theory is limited. When the uncertain information is described by the basic belief assignment instead of the probability distribution, Shannon entropy cannot work. Deng entropy is defined on the belief function frame, which makes it able to measure uncertain information described by the BBA efficiently.

Let A be the focal set of belief function, and $\left|A\right|$ be the cardinality of A. Deng entropy E is defined as:

$$E\left(m\right)=-\sum _{A\subseteq \mathrm{\Omega }}m\left(A\right)log\frac{m\left(A\right)}{2^{\left|A\right|}-1}.$$

(19)

We can easily learn from the definition that if the mass function is $Bayesian$, which means $\left|A\right|=1$ for all A, Deng entropy degenerates to Shannon entropy such that

$$E\left(m\right)=-\sum _{A\subseteq \mathrm{\Omega }}m\left(A\right)logm\left(A\right).$$

(20)

The greater the cardinality of the focal set is, the bigger the corresponding Deng entropy is, so that the evidence imprecisely refers to more single elements. Thus, significant Deng entropy indicates huge uncertainty. Powered by this feature, we calculate the average Deng entropy of BBAs to select the best attribute leading to the least uncertainty. The details are shown in the next section.

3. Design of Belief Entropy Trees

Up to now, various decision tree methods have been proposed to deal with evidential data, but many of them consider categorical attributes and transform the continuous attribute values into discrete categories. Some recent works fit the continuous attributes with same class labels into normal distributions [41] and generate BBA from the normal distributions to select the best splitting attribute by calculating belief entropy [42]; however, this method divides samples into each set of certain classes, which can only handle the precise class labels. Our goal is to develop a belief decision tree method learns from data set with continuous and precise attribute values but incomplete class labels directly and efficiently.

This section explains our method in detail, specifically focusing on the procedure of attribute selection. Corresponding splitting strategy, stopping criterion and the leaf structure are also well-designed to accomplish the whole belief entropy decision tree.

3.1. The Novel Method to Select Attribute

The learning procedure of decision trees is generally to decide the split attribute and to decide how to split on this attribute on each node; our method also proceeds in this manner. As a novel decision tree, the most characteristic and core part of our method is the attribution selection, which includes three steps: firstly, for each attribute, fit the values to normal distributions corresponding to each class label, in another words, fit attribute values into K×D normal distribution models, where K is the class number and D is the attributes number of instances; secondly, for every instance, generate D BBAs from each attribute according to the normal distribution-based models; finally, calculate belief entropy from BBAs for each attribute. The attribute with minimum belief entropy will be selected to split.

3.1.1. Parameter Estimation on Data with Continuous Attributes

Powered by the idea of extracting BBAs from normal distribution-modeled attribute values [41], we try to operate similarly on data with ill-known class labels. In the situation that each instance exactly belongs to one class, the d-th attribute values set $\left\{x_1^d,\dots ,x_N^d\right\}$ is divided into K subsets $\left\{x_n^d\mid y_n=C_k\right\},k=1,\dots ,K$ corresponding to each class. It is easy to fit each subset to the normal distribution by calculating means and standard deviations.

Example 1.

Consider the Iris data set [43], a classical machine learning data set, which contains 150 training instances of three classes: ‘Setosa’, ‘Versicolor’, ‘Virginica’, with four attributes: sepal length(SL), sepal width(SW), petal length(PL) and petal width(PW). For the values of attribute SL in the class of Setosa, we can directly calculate the mean value as $\mu =5.0133$ and standard deviation as $\sigma =0.3267$. Similarly, we can obtain normal distribution parameters of class of Versicolor and Virginica. Figure 1 shows the normal distribution model of Iris data set for the SL attribute in three classes.

However, when the labels of training set are ill-known, some instances can not be allocated to a certain class assertively. The evidential likelihood and $E^{2}M$ algorithm introduced in Section 2 make it possible to generate an estimation of model parameters. Because the $E^{2}M$ algorithm uses only contour functions, the label of n-th instance will be represented by plausibility $p{l}_n=\left\{p{l}_{nk}\right\},k=1,\dots ,K$ instead of mass function $m_n$.

For the purpose of comparing attributes, we split the whole training data into D attribute–label pairs and handle D parameter estimation problems. Consider the d-th attribute value vector $X^d={\left(x_1^d,\cdots ,x_N^d\right)}^T,d\in \left\{1,\dots ,D\right\}$, we assume the conditional distribution of $X^d$ when given $y=C_k$ is normal with mean ${\mu }_k$ and standard deviation ${\sigma }_k$:

$$X^d|(y=C_k)\sim \mathcal{N}\left({\mu }_k,{\sigma }_k^2\right),k=1,\dots ,K.$$

Actually the assumption is to build a one-dimensional $Gaussianmixturemodel\left(GMM\right)$ [44]. Similar to the application of $E^{2}M$ algorithm in $lineardiscriminantanalysis$ [45], the following discuss is practically to adopt $E^{2}M$ algorithm to estimate parameters in GMM.

Let ${\pi }_k$ be the marginal probability when $y=C_k$, and $\theta =({\mu }_1,\dots ,{\mu }_K,{\sigma }_1,\dots ,{\sigma }_K,{\pi }_1,\dots ,{\pi }_K)$ the parameter vector to be estimated. The complete-data likelihood is

$$\begin{array}{cc}\hfill L_c\left(\theta \right)& =\prod _{n=1}^{N}p\left(x_n^d\left|Y_n=y_n\right.\right)p\left(y_n\right)\hfill \\ & =\prod _{n=1}^N\prod _{k=1}^K\varphi {\left(x_n^d;{\mu }_k,{\sigma }_k\right)}^{y_{nk}}{\pi }_k^{y_{nk}},\hfill \end{array}$$

(21)

where the $\varphi$ is normal distribution probability density,

$$\varphi \left(x;\mu ,\sigma \right)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{{\left(x-\mu \right)}^2}{2{\sigma }^2}},$$

(22)

and $y_{nk}$ is a binary indicator variable, such that $y_{nk}=1$ if $y_n=C_k$ and $y_{ik}=0$ if $y_N\ne C_k$.

when expended to evidential data, where we use contour function to describe the labels, the evidential likelihood is drew from Equation (16) that,

$$\begin{array}{cc}\hfill L\left(\theta \right)& =\prod _{n=1}^N{\mathbb{E}}_{\theta }\left[p{l}_n\left(y_n\right)\right]\hfill \\ \hfill & =\prod _{n=1}^N\sum _{k=1}^{K}p{l}_{nk}\varphi \left(x_n^d;{\mu }_k,{\sigma }_k\right){\pi }_k.\hfill \end{array}$$

(23)

According to the $E^{2}M$ algorithm, we compute the expectation of complete-data log likelihood

$${\ell }_c\left(\theta \right)=log{L}_c\left(\theta \right)=\sum _{n=1}^N\sum _{k=1}^K{y}_{nk}\left[log\varphi \left(x_i^d;{\mu }_k,{\sigma }_k\right)+log{\pi }_k\right]$$

(24)

with respect to the combined mass probability function

$$p\left(x^d\left|pl;{\theta }^{\left(q\right)}\right.\right)=\prod _{n=1}^{N}p\left(x_n^d\left|p{l}_n;{\theta }^{\left(q\right)}\right.\right).$$

(25)

To simplify the equation, we denote

$${\zeta }_{nk}^{\left(q\right)}=p\left(x_n^d\left|p{l}_n;{\theta }^{\left(q\right)}\right.\right)=\frac{p{l}_{nk}{\pi }_k^{\left(q\right)}\varphi \left(x_n^d;{\mu }_k^{\left(q\right)},{\sigma }_k^{\left(q\right)}\right)}{{\sum }_{k=1}^{K}p{l}_{nk}{\pi }_k^{\left(q\right)}\varphi \left(x_n^d;{\mu }_k^{\left(q\right)},{\sigma }_k^{\left(q\right)}\right)}.$$

(26)

Finally, we obtain the to-be-maximized function

$$Q\left(\theta ,{\theta }^{\left(q\right)}\right)=\sum _{n=1}^N\sum _{k=1}^K{\zeta }_{nk}^{\left(q\right)}\left[log\varphi \left(x_n^d;{\mu }_k,{\sigma }_k\right)+log{\pi }_k\right]$$

(27)

in the E-step.

The formal of $Q\left(\theta ,{\theta }^{\left(q\right)}\right)$ is similar to the function computed in the EM algorithm on the GMM [44]. Because of the similarity, we imitate it and learn that the optimal parameter maximizing $Q\left(\theta ,{\theta }^{\left(q\right)}\right)$ can be iteratively computed by

$${\pi }_k^{\left(q+1\right)}=\frac{1}{n}\sum _{n=1}^N{\zeta }_{nk}^{\left(q\right)},{\mu }_k^{\left(q+1\right)}=\frac{{\sum }_{n=1}^N{\zeta }_{nk}^{\left(q\right)}{x}_i^d}{{\sum }_{n=1}^N{\zeta }_{nk}^{\left(q\right)}},$$

(28)

$${\sigma }_k^{\left(q+1\right)}=\sqrt{\frac{{\sum }_{n=1}^N{\zeta }_{nk}^{\left(q\right)}{\left(x_n^d-{\mu }_k^{\left(q+1\right)}\right)}^2}{{\sum }_{n=1}^N{\zeta }_{nk}^{\left(q\right)}}}$$

(29)

Finally when $L\left({\mathit{\theta }}^{\left(q+1\right)}\right)-L\left({\mathit{\theta }}^{\left(q\right)}\right)⩽ϵ$ is satisfied for some $ϵ$, stop the iteration and remark ${\theta }^{\left(q+1\right)}$ as ${\theta }^d=\left({\mu }_1^d,\dots ,{\mu }_K^d,{\sigma }_1^d,\dots ,{\sigma }_K^d,{\pi }_1^d,\dots ,{\pi }_K^d\right)$, which is the estimation of parameters in the GMM extracted from d-th attribute. Repeat this procedure for every attributes of the training set, $D\times K$ normal distribution

$${\mathcal{N}}_k^d\left({\mu }_k^d,{{\sigma }_k^d}^2\right),d=1,\dots ,D,k=1,\dots ,K$$

will be generated.

The Algorithm 1 shows the procedure of parameter estimation and there is Example 2 to help understand it.

Algorithm 1 Parameter estimation of GMMs.

Input: evidential training set $T_{pl}=(\mathit{x},p{l}_y)$, iteration stop threshold $ϵ$ Output: estimated normal distribution parameter matrix $\left({\mu }_k^d,{\sigma }_k^d\right),d=1,\dots ,D,k=1,\dots ,K$ 1: for each attribute $A^d$ do 2:  initialize parameters as ${{\theta }^d}^{\left(0\right)}=\left({{\mu }_1^d}^{\left(0\right)},\dots ,{{\mu }_K^d}^{\left(0\right)},{{\sigma }_1^d}^{\left(0\right)},\dots ,{{\sigma }_K^d}^{\left(0\right)},{{\pi }_1^d}^{\left(0\right)},\dots ,{{\pi }_K^d}^{\left(0\right)}\right)$; 3:  $q=0;$ Initialize loop variable. 4:  for q do 5:    update the estimation of parameters ${{\theta }^d}^{(q+1)}=\left({{\mu }_1^d}^{(q+1)},\dots ,{{\mu }_K^d}^{(q+1)},{{\sigma }_1^d}^{(q+1)},\dots ,{{\sigma }_K^d}^{(q+1)},{{\pi }_1^d}^{(q+1)},\dots ,{{\pi }_K^d}^{(q+1)}\right)$ by Equations (28) and (29). 6:    if $L\left({{\theta }^d}^{\left(q+1\right)}\right)-L\left({{\theta }^d}^{\left(q\right)}\right)⩽ϵ$ then 7:     break; {End the loop if evidential likelihood increment is less than threshold.} 8:    end if 9:    $q=q+1$; 10:   end for 11:   adopt $\left({{\mu }_k^d}^{(q+1)},{{\sigma }_k^d}^{(q+1)}\right),k=1,\dots ,K$ as estimated normal distribution parameters under attribute $A^d$; 12: end for

Example 2.

Consider the Iris data set mentioned in Example 1. To simulate the situation that labels of training set are not completely observed, we manually introduce uncertainty to the Iris data. In this example, we set that each instance has an equivalent chance (25%) to be $vacuous$, $imprecise$, $uncertain$ or completely observed (the detail of transformation is discussed in Section 5).

Table 1. Uncertain instances in evidential Iris data set.

Number	Attributes				Contour Functions			True Label
	SL	SW	PL	PW	pl (Setosa)	pl (Versicolor)	pl (Virginica)
1	5.1	3.5	1.4	0.2	1	0	1	Setosa
2	4.9	3.0	1.4	0.2	1	0	1	Setosa
3	4.7	3.2	1.3	0.2	1	0	0	Setosa
4	4.6	3.1	1.5	0.2	1	0.7498	0.4073	Setosa
…
51	7.0	3.2	4.7	1.4	0	1	0	Versicolor
52	6.4	3.2	4.5	1.5	0.9519	1	0.7087	Versicolor
53	6.9	3.1	4.9	1.5	1	1	1	Versicolor
54	5.5	2.3	4.0	1.3	1	1	1	Versicolor
…
101	6.3	3.3	6.0	2.5	0	0	1	Virginica
102	5.8	2.7	5.1	1.9	0.4458	0.5088	1	Virginica
103	7.1	3.0	5.9	2.1	0	0	1	Virginica
104	6.3	2.9	5.6	1.8	1	1	1	Virginica
…
150	5.9	3.0	5.1	1.8	0	0	1	Virginica

shows the attribute values and labels described by plausibility $pl$ of some instances in evidential Iris data.

Table 2. Estimated normal distribution parameters for evidential Iris data set.

Attributes	Setosa		Versicolor		Virginica
	$\mathbf{\mu }$	$\mathbf{\sigma }$	$\mathbf{\mu }$	$\mathbf{\sigma }$	$\mathbf{\mu }$	$\mathbf{\sigma }$
SL	5.0031	0.3591	5.9183	0.5419	6.6307	0.6300
SW	3.3822	0.4264	2.7812	0.3201	3.0119	0.2765
PL	1.4640	0.1718	4.3066	0.4961	5.5684	0.5591
PW	0.2408	0.1011	1.3273	0.2074	2.0495	0.2522

shows the mean and standard deviation pairs $\left(\mu ,\sigma \right)$ calculated by $E^{2}M$ algorithm. Figure 2 shows curves of these models.

3.1.2. BBA Determination

This step is to generate D BBAs corresponding to each attribute for every instance in the training set.

Choose an instance $I_n$ with attribute vector $x_n=\left(x_n^1,\dots ,x_n^D\right)$ from the data set, calculate the intersection of $x_n^d\left(d=1,\dots ,D\right)$ and the K normal distribution functions ${\varphi }_k^d=\varphi \left(x^d;{\mu }_k,{\sigma }_k\right),k=1,\dots ,K$, i.e., we obtain K normally distributed probability density function (PDF) values for the attribute $A^d$ and instance $I_n$, denoted as ${\varphi }_{nk}^d,k=1,\dots ,K$.

Due to the property that the probability of a value x sampling from a normal distribution is proportional to the PDF $\varphi \left(x\right)$, we can infer, for the attribute d, the probability that instance $x_n$ belongs to each class is proportional to ${\varphi }_{nk}^d=\varphi \left(x_n^d;{\mu }_k,{\sigma }_k\right),k=1,\dots ,K$. From this opinion of statistical analysis, the rule to assign normal PDFs to some sets was proposed to build BBAs.

Firstly, normalize the ${\varphi }_{nk}^d$ with different class k such that

$$f_k={\varphi }_{nk}^d/\sum _{k=1}^K{\varphi }_{nk}^d.$$

(30)

Then rank $f_k$ in decreasing order $f_r^{\prime }\left(r=1,\dots ,K\right)$, whose corresponding class is denoted as $C_r^{\prime }\left(r=1,\dots ,K\right)$. Assign $f_r^{\prime }$ to the class set by the following rule:

$$\begin{array}{cc}\hfill & m\left(\left\{C_1^{\prime }\right\}\right)=f_1^{\prime }\hfill \\ & m\left(\left\{C_1^{\prime },C_2^{\prime }\right\}\right)=f_2^{\prime }\hfill \\ & \cdots \hfill \\ & m\left(\left\{C_1^{\prime },\dots ,C_K^{\prime }\right\}\right)=m\left(\theta \right)=f_K^{\prime }.\hfill \end{array}$$

(31)

If $f_i^{\prime }=f_{i+1}^{\prime }=\dots =f_j^{\prime }$, then $m\left(\left\{C_1^{\prime },\dots ,C_j^{\prime }\right\}\right)={\sum }_{p=i}^j{f}_p^{\prime }$. By this rule, we obtain a nested BBA of $x_n$ under the select attribute $A^d$, which we denote as $m_n^d$.

Example 3.

Consider the first instance of the evidential Iris data set showed in Table 1, whose attributes are:

$$x^{SL}=5.1,x^{SW}=3.5,x^{PL}=1.4,X^{PW}=0.2.$$

For attribute SL, the intersections of $x^{SL}=5.1$ and three normal distributions are shown in Figure 3 such that

$$\begin{array}{cc}\hfill & {\varphi }_{Setosa}^{SL}\left(x^{SL}\right)=1.0712,\hfill \\ \hfill & {\varphi }_{Versicolor}^{SL}\left(x^{SL}\right)=0.2354,\hfill \\ \hfill & {\varphi }_{Virginica}^{SL}\left(x^{SL}\right)=0.0331.\hfill \end{array}$$

The reader can see in the figure that this instance is closest to class ‘Setosa’, then to the ‘Versicolor’ and ‘Virginica’. Thus, we generate BBA from intersection values, which is intuitive. The BBA is assigned as:

$$\begin{array}{cc}\hfill & m\left(\left\{Setosa\right\}\right)=\frac{1.0712}{1.0712+0.2354+0.0331}=0.7996\hfill \\ \hfill & m\left(\left\{Setosa,Versicolor\right\}\right)=\frac{0.2354}{1.0712+0.2354+0.0331}=0.1757\hfill \\ \hfill & m\left(\left\{Setosa,Versicolor,Virginica\right\}\right)=\frac{0.0331}{1.0712+0.2354+0.0331}=0.0247\hfill \end{array}$$

Similarly, we build BBAs for the rest of the attributes—shown in

Table 3. Generated BBAs of selected instance.

Attributes	BBAs
	$m\left(\left\{Setosa\right\}\right)=0.7996$
SL	$m\left(\left\{Setosa,Versicolor\right\}\right)=0.1757$
	$m\left(\left\{Setosa,Versicolor,Virginica\right\}\right)=0.0247$
	$m\left(\left\{Setosa\right\}\right)=0.6904$
SW	$m\left(\left\{Setosa,Virginica\right\}\right)=0.2329$
	$m\left(\left\{Setosa,Versicolor,Virginica\right\}\right)=0.0767$
	$m\left(\left\{Setosa\right\}\right)=1$
PL	$m\left(\left\{Setosa,Versicolor\right\}\right)=0$
	$m\left(\left\{Setosa,Versicolor,Virginica\right\}\right)=0$
	$m\left(\left\{Setosa\right\}\right)=1$
PW	$m\left(\left\{Setosa,Versicolor\right\}\right)=0$
	$m\left(\left\{Setosa,Versicolor,Virginica\right\}\right)=0$

.

3.1.3. Calculation of Belief Entropy

The last step to determine splitting attribute is to calculate the average Deng entropy

$$E^d=E\left(A^d\right)=\frac{1}{N}\sum _{n=1}^{N}E\left(m_n^d\right),d=1,\dots D$$

(32)

of all instances for each attribute. As mentioned in Section 2.6, Deng entropy measures the uncertain degree contained by BBA, which means the less $E^d$, the more certainty the BBAs contain, and the more separate the division of classes is. Consequently, we choose the attribute $A^{*}$ that minimizes the average Deng entropy such that

$$A^{*}=\underset{A^d}{\arg min}\left(E\left(A^d\right)\right),d=1,\dots ,D$$

(33)

to be the best splitting attribute to proceed the tree building.

Example 4.

Continue the Examples 2 and 3. Calculate Deng entropy of BBAs of selected instance shown in Table 3:

$$\begin{array}{cc}\hfill E^{SL}\left(m\right)& =-0.7996\times log\frac{0.7996}{2^1-1}-0.1757\times log\frac{0.1775}{2^2-1}-0.0247\times \frac{0.0247}{2^3-1}=0.2933\hfill \\ \hfill E^{SW}\left(m\right)& =-0.6904\times log\frac{0.6904}{2^1-1}-0.2329\times log\frac{0.2329}{2^2-1}-0.0767\times \frac{0.0767}{2^3-1}=0.3687\hfill \\ \hfill E^{PL}\left(m\right)& =-1\times log\frac{1}{2^1-1}=0\hfill \\ \hfill E^{PW}\left(m\right)& =-1\times log\frac{1}{2^1-1}=0\hfill \end{array}$$

Similarly proceed same calculation to all instances so that average Deng entropy for attributes are calculated that

$$E\left(A^{SL}\right)=1.3853,E\left(A^{SW}\right)=2.0837,E\left(A^{PL}\right)=0.4275,E\left(A^{PW}\right)=0.2116.$$

According to this result, attribute PW will be chosen to generate child nodes.

Comparing the Deng entropy with the curves in Figure 2, we can intuitively learn that PW has the most distinctive curves for each class, yet curves in SW overlap each other a lot, which conforms to the size of the average Deng entropy above, where PW is the lowest and SW is the highest.

As a matter of fact, Examples 1–4 in this chapter can be orderly combined as a whole calculating example, which shows the procedure of the proposed attribute selecting method.

3.2. Splitting Strategy

The splitting strategy is redesigned according to the selected attribute $A^{*}$ to fit the proposed attribute selection method. Branches will be associated to each class, that is to say, each node to be edged will have K branches. For an instance $I_n$, consider the generated BBAs, the class corresponding to the maximum mass value will be the branch that this instance shall be put into. To put it simply, when splitting the tree under attribute $A^{*}$, the instance $I_n$ will be assigned into the $k_n$-$th$ child node, where the $k_n$ satisfies

$$k_n=\underset{k}{\arg max}{\varphi }_{nk}^{*}\left(x_n^d\right).$$

(34)

The Algorithm 2 summarizes the procedure of selecting attribute and splitting. It should be mentioned that, though the child nodes are associated to each class, this splitting strategy does not mean to determine the affiliation of instances directly and arbitrarily in this step.

Algorithm 2 Attribute selection and splitting.

Input: evidential training set $T_{pl}=(\mathit{x},p{l}_y)$, possible splitting attribute $\mathcal{A}=\left\{A^1,\dots ,A^D\right\}$ Output: selected attribute $A^{*}$, instance sets in child nodes $T_i$,$\left(i=1,\dots ,K\right)$ 1: compute the normal distribution parameters $\left({\mu }_k^d,{\sigma }_k^d\right)$ for each $A^d$ and $C_k$ by $E^{2}M$ algorithm; 2: for each attribute $A^d$ do 3:  for each instance $I^n$ do 4:   generate BBA $m_n^d$ from normal distributions ${\mathcal{N}}_k^d\left({\mu }_k^d,{{\sigma }_k^d}^2\right),k=1,\dots ,K$; 5:   $E_n^d=E\left(m_n^d\right)$; {Calculate Deng entropy for all generated BBAs} 6:  end for 7:   $E\left(A^d\right)=Average\left(E_n^d\right)$;  {Calculate average Deng entropy for each attribute} 8: end for 9: split on attribute $A^{*}=\underset{A^d}{\arg min}\left(E\left(A^d\right)\right)$; {The attribute with minimum average entropy is selected}

3.3. Stopping Criterion and Prediction Decision

After designing the attribute selection and partitioning strategy, we split each decision node to several child nodes. This procedure repeats iteratively until one of the stop criterion is met:

• No more attributes for selection;
• The number of instances in the nodes falls below a set threshold;
• The labels of instances are all precise and fall into the same class;

When the tree building stops at a leaf node L, a class label should be determined to predict the instances that fall into this node. We design two different prediction methods such that:

• The first one is to generate the prediction label from the original training labels of instances contained by this node, which is a similar treatment to traditional decision trees such as C4.5 tree method. Denoting the instances in the leaf node by $\left\{I_1^{\prime },\dots I_P^{\prime }\right\}$ and the corresponding evidential training labels by $\left\{p{l}_p^{\prime }\right\},p=1,\dots P$, the leaf node will be labeled by $\widehat{C}$, where

  $$\widehat{C}=\underset{C_k}{\arg max}\sum _{p=1}^{P}p{l}_p^{\prime }\left(C_k\right),k=1,\dots ,K,$$

  (35)

  which means the class label with maximal plausibility summation will represent this node. This tree predicts from the original labels of training set, which is called $Oringin$-$predictionbeliefentropytree\left(OBEtree\right)$ for short in this paper.
• The first method described above in fact abandons the generated BBAs during the tree build procedure, which will be adopted to generating predicted instance label in the second method. Firstly, the splitting attributes list, which lead instance $I^{\prime }$ to the leaf node from top to down, are denoted by $\left\{A_1^{*},\dots ,A_Q^{*}\right\}$, and the BBAs generated accordingly are denoted by $m_1^{*}\dots ,m_Q^{*}$. Then combine these BBAs by Dempster rule, such that $\widehat{m}=m_1^{*}\oplus ···\oplus m_Q^{*}$, to predict the training instance. On this basis, we continue to combine generated BBAs of all instances in a leaf node such that ${\widehat{m}}_{leaf}={\widehat{m}}_1\oplus ···\oplus {\widehat{m}}_P$, where the once again combined BBA ${\widehat{m}}_{leaf}$ will be the mass prediction label for the whole leaf node. To obtain a precise label for another choice, the last step is making decision on BBA by choosing the class label with maximal pignistic probability computed by Equation (11). We call this tree a $Leaf$-$predictionbeliefentropytree\left(LBEtree\right)$ in this paper.

The Algorithm 3 summarizes the induction of belief entropy trees introduced in this section.

Algorithm 3 Induction of belief entropy trees (BE-tree).

Input: evidential training set $T_{pl}$, classifier type $TYPE$ Output: belief entropy tree $Tree$ 1: construct a root node containing all instances $T_{p}l$; 2: if stopping criterion is met then 3:  if $TYPE=OBE$ then 4:   output precise prediction generated from original plausibility label for the whole node; 5:  else if  $TYPE=LBE$  then 6:   combine BBAs generates during each splitting $\widehat{m}=m_1^{*}\oplus ···\oplus m_Q^{*}$ for each instance; 7:   combine BBAs of all instances in previous node generated in step 6 that ${\widehat{m}}_{leaf}={\widehat{m}}_1\oplus ···\oplus {\widehat{m}}_P$; 8:   output ${\widehat{m}}_{leaf}$ as a mass prediction for the whole leaf node; 9:   output $\widehat{C}=Pignistic\left({\widehat{m}}_{leaf}\right)$ as a precise prediction for the whole leaf node; 10:  end if 11:  return $Tree$=root node; 12: else 13:  apply Algorithm 2 to select splitting attribute $A^{*}$; 14:  induce each subset ${T_{pl}}_{child}$ based on $A^{*}$; 15:  for all ${T_{pl}}_{child}$ do 16:    $Tre{e}_{child}=BE$-$tree\left({T_{pl}}_{child}\right)$; {Recursively build the tree on the new child node} 17:    attach $Tre{e}_{child}$ to the corresponding $Tree$; 18:   end for 19: end if

3.4. An Alternative Method for Predicting New Instance

Two types of belief entropy trees, the OBE tree and the LBE tree, have been described in detail in the last section. Similar to traditional decision trees, a new instance will be classified in a top-down way: starting at the root node and following branches by considering its generated BBA under splitting attribute until reaching a leaf node. The prediction of leaf node will be given to this new instance.

However, differing from the idea of collecting the numerous ‘opinions’ of instances, another method to predict a new instance is considered after a tree has been built. In Section 3.1.2, we introduced how to generate each training instance’s BBA corresponding to attributes. In the same way, we can generate $m_1^{*}\dots ,m_Q^{*}$ corresponding to an attributes list $\left\{A_1^{*},\dots ,A_Q^{*}\right\}$, which orderly splits and leads the new instance to a leaf node. Then, we combine these BBAs such that $\widehat{m}=m_1^{*}\oplus ···\oplus m_Q^{*}$ to predict the new testing instance. It is easy to find that this method performs the same way as the front part of label prediction in LBE trees, yet stops when obtaining a mass prediction from the testing instance’s own attribution values instead of the leaf node it belongs to, which also means testing instances in a same leaf node normally have different mass prediction under this design. For the sake of narrative, a tree predicting in this way is called $Instance$-$predictionbeliefentropytree\left(IBEtree\right)$ in this paper.

Figure 4 and Figure 5 show the procedure of making prediction on leaf node, where Figure 4 is the generation of mass prediction $\widehat{m}$ for each instance, in whether training set or testing set; Figure 5 details the different prediction making in the proposed three belief entropy trees.

4. Belief Entropy Random Forest

We have introduced the induction of belief entropy trees in the previous section, which is regarded as the basic classifier of random forest ensemble method in the following discussion.

The generalization ability of random forest draws from not only the perturbation of sampling, but also the perturbation of attributes selecting. Specific to the proposed belief entropy random forest, for each basic tree, we firstly performs bootstrap sampling on the original training set, which means randomly sampling with replacement for N times on the set T where $\left|T\right|=N$. Secondly, when training on this resampling set, for each to-be-split node, the best splitting attribute will be chosen from a subset $\left\{A_i^{\prime }\right\},i=1,\dots ,D^{\prime }$ of the set of all available attributes $\left\{A_j\right\},j=1,\dots ,D$, where $1<D^{\prime }<D$. If $D^{\prime }=D$, the basic tree splits totally, the same as the belief entropy tree; while $D^{\prime }=1$ means randomly selecting an attribute to split all the time.

Repeat the first and second steps above S times then a ‘forest’ containing variable basic trees will be constructed, where the repeat time S is called forest size. When making a prediction of a new instance on this forest, S primary predictions will be independently generated by S basic trees and finally summarized to one result. It should be mentioned that the OBE tree output precise label directly for testing instances while the LBE trees and IBE trees can provide mass labels described by BBAs or precise labels. This feature inspires two different strategies for making predictions in the last step: the majority voting for precise labels and belief combination of mass labels.

Algorithm 4 shows the procedure of building the complete evidential random forests based on belief entropy trees. Selecting different ensemble prediction strategies and base tree types, we build five random forest lists below:

• label-voting $OBE$ Random Forest$(L$-$OBERF)$, which performs majority voting on precise outputs of OBE trees;
• label-voting $LBE$ Random Forest$(L$-$LBERF)$, which performs majority voting on precise outputs of LBE trees;
• mass-combination $LBE$ Random Forest$(M$-$LBERF)$, which combines BBAs generated by LBE trees and makes decision;
• label-voting $IBE$ Random Forest$(L$-$IBERF)$, which performs majority voting on precise outputs of IBE trees;
• mass-combination $IBE$ Random Forest$(M$-$IBERF)$, which combines BBAs generated by IBE trees and makes decision;

Algorithm 4 Building procedure of evidential random forests.

Input: evidential training set $T_{pl}$, new instance $\mathit{x}$, base classifier number h, base classifier type $TYPE$, base classifier output mode O Output: predicted label $\widehat{y}$ 1:  for  $i=1:h$  do 2:  $T_i=RandomAttributeSampling\left(RandomInstanceSampling\left(T_{pl}\right)\right)$; {The resampling procedure of each base tree.} 3:  if $TYPE=OBE$ then 4:   $Tre{e}_i=OBE\left(T_i\right)$; 5:  else if  $TYPE=IBE$  then 6:   $Tre{e}_i=IBE\left(T_i\right)$; 7.  else if  $TYPE=LBE$  then 8:   $Tre{e}_i=LBE\left(T_i\right)$; 9:  end if 10: end for 11: for  $i=1:h$ do 12:  $L_i=LabelPrediction\left(Tre{e}_i,\mathit{x}\right)$; {Generate predict labels of each base tree.} 13: end for 14: if  $O=preciselabel$ then 15:  $\widehat{y}=Majority\left(L_1,\dots ,L_h\right)$; {Generate prediction from precise labels.} 16: else if  $O=masslabel$  then 17:  $\widehat{y}=Pignistic\left(MassCombination\left(L_1,\dots ,L_h\right)\right)$; {Generate prediction from mass labels.} 18: end if

Figure 6 shows the procedure of constructing the forests, in which the Figure 6a shows generation of basic trees in a random forest, and Figure 6b shows different procedure of combining the final prediction in five forests, which will lead to a different classification performance. We will evaluate them in the next section.

5. Experiments

In this section, we detail experiments to evaluate the performance of the proposed decision tree method. The experiment settings and results are detailed below.

5.1. Experiment Settings

As there are no widely accepted evidential data sets to measure the proposed method, it is necessary to generate a data set with ill-known labels from machine learning databases taken from the UCI repository [46]. We selected several data sets, including: Iris, Wine, Balance scale, Breast cancer, Sonar and Ionosphere.

Denote the true label of a instance by $C_i$, and give its uncertain observation $m_{y_i}$. Due to the characters of belief function, we can simulate several situations from precise data:

•

a $precise$ observation is such that $p{l}_{y_i}\left(C_i\right)=1$, and $p{l}_{y_i}\left(C_j\right)=0,\forall C_j\ne C_i^{*}$;

•

a $vacuous$ observation is such that $p{l}_{y_i}\left(C_j\right)=1,\forall C_j\in \mathcal{C}$;

•

an $imprecise$ observation is such that $p{l}_{y_i}\left(C_j\right)=1$ if $C_j=C_i^{*}$ or $C_j\in {\mathcal{C}}_{rm}$, and $p{l}_{y_i}\left(C_j\right)=0$ otherwise, where ${\mathcal{C}}_{rm}$ is a set of randomly selected labels;

•

an $uncertain$ observation is such that $p{l}_{y_i}\left(C_i^{*}\right)=1$, and $p{l}_{y_i}\left(C_j\right)=r_j,\forall C_j\ne C_i^{*}$, where $r_j$ are sampled independently from uniform distribution $\mathcal{U}\left(\left[0,1\right]\right)$.

To observe the performance on evidential training data sets with different ill-known types and incomplete degrees, we set three variables, vacuousness level $V\in \left[0,1\right]$, imprecision level $I\in \left[0,1\right]$ and uncertainty level $U\in \left[0,1\right]$, to adjust the generation procedure, where $V+I+U⩽1$.

Example 2 shows the transformed Iris data set and listed part of instances in Table 1. In this example, labels of no.53 and no.54 instance are vacuous; labels of no.1 and no.2 instance are imprecise; labels of no.4 and no.52 instance are uncertain.

To improve the reliability and reduce the stochasticity, we performed 5-fold cross-validation on each data set and repeat ten times to compute an average classification accuracy for all experiments. Different tree induction techniques will be compared:

• $traditionalC4.5tree$, which only uses precise data in the training set during tree induction;
• $beliefentropytrees$ described in Section 3.3: OBE tree, LBE tree, IBE tree;
• $beliefentropyrandomforest$ described in Section 4:

  -

  $label$-$votingOBErandomforest(L$-$OBERF)$;

  -

  $label$-$votingLBErandomforest(L$-$LBERF)$;

  -

  $mass$-$combinationLBErandomforest(M$-$LBERF)$;

  -

  $label$-$votingIBErandomforest(L$-$IBERF)$;

  -

  $mass$-$combinationIBErandomforest(M$-$IBERF)$;

We set the maximal size of the leaf node as $\left|T\right|/20$ to avoid overfitting in the belief entropy trees. In the random forests, the forest size was set as 50, and the size of the attributes subset was set as $D^{\prime }=lo{g}_{2}D$.

5.2. Experiments on Vacuous Data

Assuming part of the instances in the training set are totally unobserved while others are completely observed, we performed experiments with different vacuousness levels $V\in \left[0,1\right]$ while $I=U=0$. Generating the training sets and learning on them, the results are shown in Figure 7.

Firstly, we observe the figure as a whole. Obviously, whatever the tree induction method is, it is impossible to learn from data sets whose instances are all vacuous. Thus, the accuracy of all trees decreases gradually as V increases, yet drops sharply when the V approaches nearly to 1. On the contrary, almost all curves keep steady or decrease slightly before the vacuousness level reaches 80%, except for the OBE trees.

Table 4. Classification accuracy on UCI data sets with 90% vacuousness level.

	Iris	Wine	Ionosphere	Balance Scale	Breast Cancer	Sonar
C4.5	0.800	0.6814	0.7221	0.6134	0.8031	0.5247
OBE tree	0.5846	0.4038	0.5775	0.6305	0.7751	0.5553
LBE tree	0.8326	0.6026	0.7763	0.5954	0.8732	0.5793
IBE tree	0.8288	0.6386	0.7797	0.6282	0.8793	0.5841
label voting OBE Random Forest	0.7864	0.4897	0.7404	0.7453	0.8745	0.5611
label voting LBE Random Forest	0.9020	0.8594	0.8555	0.7127	0.9319	0.6149
mass combination LBE Random Forest	0.8989	0.8295	0.8182	0.7138	0.9223	0.6115
label voting IBE Random Forest	0.9053	0.8940	0.8608	0.7454	0.9338	0.6120
mass combination IBE Random Forest	0.9174	0.9082	0.8647	0.6891	0.9330	0.6236

shows the accuracy results when V equals 90%.

Considering the basic belief entropy trees firstly, the LBE trees and IBE trees perform, most of the time, at least as well as the traditional C4.5 decision trees, and better than the traditional decision trees for some time, especially when encountering high vacuousness level V; however, the OBE preforms elusively on different data sets: it has the lowest classification accuracy in Iris, Wine and Ionosphere data set; however, it achieves better results in the Balance scale. It is possible that if all samples in a leaf node are vacuous, the direct combination of all the training labels stays vacuous, which led to the shortage of OBE tree.

It can be observed that the belief entropy random forests perform well overall for their improvement in classification accuracy compared to the corresponding basic tree and the slower accuracy decent rate as V increases. Among these forests, the ones based on IBE and making prediction by mass combination performs better than others in nearly all data sets except the Balance scale.

5.3. Experiments on Imprecise Data

The second situation is that some data are imprecisely observed, i.e., the observation is a set value, while the true value lies in this set (called superset labels [47] in some works). As mentioned before, imprecision level I controls the percentage of imprecise observations.

For the instance to be imprecise, we randomly generate a number $z_k\in \left[0,1\right]$ for each class $C_k$ except the true one. Plausibility of labels with $z_k<I$ will be set to 1. When the $I=1$, a training set becomes totally imprecise, which is, in practice, the same situation as total vacuousness; while $I<1$, instances are in a middle state of transition from precise to vacuous, which indicates a piece of similarity between the vacuous training set and the imprecise training set, i.e., we can tell that the imprecise sample contains more information than the totally vacuous ones. As a result, we can see in Figure 8, that curves of accuracy with changing I are similar to those in experiments with vacuousness in Figure 7, yet more smooth and full.

According to the

Table 5. Classification accuracy on UCI data sets with 90% imprecision level.

	Iris	Wine	Ionosphere	Balance Scale	Breast Cancer	Sonar
C4.5	0.8000	0.6814	0.7221	0.6134	0.8031	0.5247
OBE tree	0.6233	0.5382	0.6786	0.6746	0.8605	0.5841
LBE tree	0.9093	0.7719	0.7858	0.6146	0.8979	0.6303
IBE tree	0.9040	0.7899	0.7892	0.6381	0.9051	0.6413
label voting OBE random forest	0.8647	0.6208	0.8552	0.7536	0.9257	0.6351
label voting LBE random forest	0.9473	0.9124	0.8621	0.7483	0.9359	0.6630
mass combination LBE random forest	0.9327	0.9118	0.8259	0.7437	0.9262	0.6635
label voting IBE random forest	0.9467	0.9219	0.8684	0.7709	0.9364	0.6572
mass combination IBE random forest	0.9447	0.9326	0.8755	0.7237	0.9399	0.6702

, the proposed methods keep pretty good classification results under high-level imprecise observations. OBE still keeps the shortage in almost all data sets while LBE and IBE achieve similar performance. M-$IBERF$ keeps its advantage in most situations, especially in the Iris and Breast Cancer data; the classification accuracy is almost equal to the results on the total precise training set. The balance scale is a particular case to be discussed later.

5.4. Experiments on Uncertain Data

Another type of ill-known label is the uncertain one, which is measured by a plausibility distribution, with the true label having the highest chance among all class labels. To evaluate the performance of the proposed trees and forests in a more general situation with uncertainty, we set $U\in \left[0,1\right]$ and $V=I=0$. For instance, to be transformed into an uncertain one, we assign a value 1 to the plausibility of the true label and random values averagely sampled from $\left[0,U\right]$ to other labels.

Despite the inability to handle total vacuousness and imprecision, the belief entropy trees have the ability to learn from totally uncertain training data sets. The horizontal curves in Figure 9 indicate all methods proposed in this paper keep stable performance with changing uncertainty level U. On the whole, we can learn from the figure that LBE and IBE perform equally well and better than OBE as a single tree in most data sets, except in the Balance scale.

Considering the forests, for the good attribute normality of Iris, Wine and Breast cancer data, classification accuracies of the five forests on these data sets have similar performance according to

Table 6. Classification accuracy on UCI data sets with 90% uncertainty level.

	Iris	Wine	Ionosphere	Balance Scale	Breast Cancer	Sonar
C4.5	0.8000	0.6814	0.7221	0.6134	0.8031	0.5247
OBE tree	0.8847	0.7792	0.8219	0.7387	0.9121	0.6851
LBE tree	0.9400	0.8180	0.7937	0.6235	0.9120	0.6803
IBE tree	0.9513	0.8399	0.7955	0.6342	0.9278	0.6784
label voting OBE random forest	0.9513	0.9590	0.9293	0.8233	0.9381	0.7577
label voting LBE random forest	0.9547	0.9489	0.8803	0.7651	0.9344	0.7370
mass combination LBE random forest	0.9560	0.9478	0.8390	0.7800	0.9225	0.7404
label voting IBE random forest	0.9567	0.9528	0.8826	0.7829	0.9376	0.7346
mass combination IBE random forest	0.9567	0.9596	0.8875	0.7496	0.9394	0..7433

, leading to a heavy overlap of curves in figure. Among these trees, the OBE trees achieve the most significant improvement by building random forest; this improvement helps OBE-RF to surpass other forests in the Ionosphere, Sonar and Balance scale data sets. Particularly, in the Balance scale, the accuracy of OBE-RF even increases slightly as the U decreases, which can be partially explained by the fact that uncertain instances are more informative then absolutely precise instances.

5.5. Summary

By carrying out experiments on training sets with different types and degrees of incomplete observation, we can conclude that the LBE trees and IBE trees, along with four types of random forests based on them, generally possess excellent learning ability on data with ill-known labels. Among the RFs, the ensemble of the IBE tree, L-IBE-RF and M-IBE RF achieve the highest classification accuracy in most situations except on samples with high uncertainty levels, especially on the Balance scale data set. We think there are two reasons: (a) compared to vacuous and imprecise samples, the learning labels of uncertain samples are more information rich, while the OBE use the learning labels to predict directly; (b) the attribute values of Ionosphere, Balance, and Sonar data sets contain less normality than others—the balance scale are totally not normal. We can conclude that the ensemble OBE RF requests less normality of the data set.

The results of experiments indicate that the application of the belief function tool to the prediction of trees and combination of forests is efficient and reasonable; yet there are also some drawbacks. Firstly, the introduction of the belief function and mass combination obviously increases the time cost of learning. The sensitivity to the normality of data makes the proposed trees and RFs unable to handle, to the greatest extent, all situations with one particular structure.

6. Conclusions

In this paper, a new classification tree method based on belief entropy is proposed to cope with uncertain data. This method directly models continuous attribute values of training data by $E^{2}M$ algorithm, and selects a splitting attribute via a new tool–belief entropy. Differing from the traditional decision trees, we redesign the splitting and prediction, making them fit the feature of uncertain labels described by the belief function. Finally, random forests with different combination strategies were constructed on the basis of the proposed tree method to seek higher accuracy and stronger generalization ability.

As the experimental results show, the proposed belief entropy trees are robust to different sorts of uncertainty. They perform closely to traditional decision trees on precise data and keep good results on data with ill-known labels. Meanwhile, the belief entropy random forests, which improve significantly when compared to the basic belief function trees, achieve excellent and stable performance even in the situation with high-level uncertainty. It is proved that the proposed trees and random forests have a potentially broad field of application. In future research, some further improvements will be investigated, such as more reasonable BBA combination methods for the incapacity of Dempster’s rule to handle huge mass conflict, and a boosting ensemble method based on the belief entropy trees.