Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Abstract

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

1. Introduction

Perceptron is the most important neural network model and has been extensively studied in the past six decades. Its history dates to the early 1940s, when McCulloch and Pitts discovered that a neuron could be modeled as a threshold device that could perform logic functions [1]. Later in the 1950s, Rosenblatt [2,3] proposed the perceptron model and its learning algorithm.

In the 1960s, Widrow and Hoff [4] proposed a similar model called Adaline. This model could be trained with the least mean squares (LMS) approach. Interest in neural networks diminished in the 1970s when Minsky and Papert [5] proved that the simple perceptron model [2] was unable to perform complex logic function and could not solve linearly inseparable problems. The revival of interest in the perceptron model took place in 1986, when Rumelhart, Hinton, and Williams [6] trained the multilayer perceptron (MLP) model using the backpropagation (BP) algorithm. Later BP was found to have been described in 1974 by Werbos [7].

For perceptron models, supervised learning is employed. Supervised learning compares the network output and the desired output. It is a closed-loop system, with the error as the feedback signal. In supervised learning, an objective function is first defined. To make the objective cost go toward zero, a gradient-descent procedure, such as LMS [4] and BP algorithms [6], is usually applied.

The MLP model is desirable in regression and classification. When trained with the same input and output patterns, an MLP functions as an autoassociative memory, and the bottleneck layer can serve as a feature extractor [8,9]. LeCun’s six-layer MLP model for handwritten digit recognition is the pioneering work in deep neural network and deep learning.

Since 1986, numerous new results on perceptron learning have been proposed. The research accomplishments on MLPs laid the foundation of the neural networks discipline. The MLP has been the most widely used neural network model in the literature. There is a feature extraction process in the MLP. For example, in [10], the MLP model was used for DDoS attack detection and in [11] it was used for the fault prediction in the energy industry. However, to the best of our knowledge, we have not found a systematic survey of the perceptron model in the literature. We understand that the perceptron model is the classical neural network model, and people may think that the topic is not sufficiently new. However, this model by itself is the most widely investigated, and there are a massive number of publications on the model. Even today, people are still actively conducting research on some aspects of this model such as the basic theory of its learning algorithms and their applications in deep learning. While most people do not have an overall knowledge of the model, it is our motivation to conduct a comprehensive and also state-of-the-art survey of the perceptron model and its learning. After reading this paper, readers will be able to have a big picture of the foundation of the most fundamental model in the neural networks discipline, learn how to use this model, and where to start if they wish to conduct research on the perceptron model.

In this AI or deep learning era, numerous new applications of neural networks have been reported, such as sensor networks [12], production optimization [13], the control of a delay teleoperation system [14], and user behavior analysis [15].

This paper is motivated to give a comprehensive, yet state-of-the-art review and also a tutorial on the perceptron model and its learning. This paper is organized as follows. In Section 2, some basic background on machine learning associated with perceptron models are presented. In Section 3, the single-layer perceptron (SLP) and its learning are dealt with. Section 4 introduces the architecture, properties of the MLP, and the BP algorithm. Section 5 presents some representative results on improving the generalization ability of MLP models. Section 6 deals with the optimization of the MLP architecture. In Section 7, various strategies for speeding up the first-order BP algorithm are introduced. Section 8 discusses several second-order learning methods for MLP. Some other learning algorithms, including expectation–maximization (EM) [16] and natural gradient [17], are briefly described in Section 9. Section 10 describes fault-tolerant learning for MLP. Section 11 features the relation between MLP and deep learning. Section 12 presents an illustrative example, then gives some advice on how to select an MLP learning algorithm for a specific application setting, and finally concludes the entire paper.

2. Background

2.1. Neurons

A McCulloch–Pitts neuron [1] collects all signals from other nodes and generates an output, as shown in Figure 1. Its input–output mapping is given by

$$o=\varphi \left(net\right),$$

(1)

where

$$net={\overrightarrow{w}}^T\overrightarrow{x}+\theta$$

(2)

where $\overrightarrow{x}={\left(x_1,\dots ,x_{J_1}\right)}^T$ is the $J_1$-dimensional input, $\overrightarrow{w}={\left(w_1,\dots ,w_{J_1}\right)}^T$ is the weight vector, and $\theta$ is a threshold or bias. $\varphi (·)$ is a continuous or discontinuous activation function that maps a real number into $(-1,1)$ or $(0,1)$. Popular activation functions are given by

$$\begin{array}{cc}\hfill \varphi \left(a\right)=\left\{\begin{array}{cc}1,\hfill & a\ge 0\hfill \\ -1\mathrm{or}0,\hfill & a<0\hfill \end{array}\right.,& \mathrm{Hard}\mathrm{limiter} \left(\mathrm{threshold}\right),\end{array}$$

(3)

$$\begin{array}{cc}\hfill \varphi \left(a\right)=\frac{1}{1+{\mathrm{e}}^{-\beta a}},& \mathrm{Logistic},\end{array}$$

(4)

$$\begin{array}{cc}\hfill \varphi \left(a\right)=tanh\left(\beta a\right),& \mathrm{Hyperbolic}\mathrm{tangent},\end{array}$$

(5)

$$\begin{array}{cc}\hfill \varphi \left(a\right)=a,& \mathrm{Linear},\end{array}$$

(6)

where $\beta$ is a gain parameter for the steepness control.

The logistic function and the hyperbolic tangent function are sigmoidal functions, which are monotonically increasing and satisfy ${lim}_{a\to +\infty }\varphi \left(a\right)=1$, ${lim}_{x\to -\infty }\varphi \left(a\right)=0$, or $-1$. The McCulloch–Pitts neuron is used in the MLP, the Hopfield network, and many other models that may utilize other activation functions.

2.2. Classification: Linear Separability and Nonlinear Separability

In classification, we have a training data set ${\mathcal{D}}_{train}=\{({\overrightarrow{x}}_1,y_1),\cdots ,({\overrightarrow{x}}_N,y_N)\}$, where ${\overrightarrow{x}}_p$’s are the training input, and $y_p$’s are the training outputs that can be $\pm 1$ (or 1 or 0), denoting whether example p belongs to a specific class.

A two-class classification problem is said to be linearly separable, if there exists a linear separating surface that separates all the input samples of class ${\mathcal{C}}_1$ from class ${\mathcal{C}}_2$. A McCulloch–Pitts neuron with a hard limiter activation, or say a linear threshold gate (LTG), can realize dichotomy, characterized by a hyperplane

$${\overrightarrow{w}}^T\overrightarrow{x}+\theta =0.$$

(7)

An input pattern belongs to ${\mathcal{C}}_1$ if ${\overrightarrow{w}}^T\overrightarrow{x}+\theta \ge 0$, and belongs to ${\mathcal{C}}_2$ if ${\overrightarrow{w}}^T\overrightarrow{x}+\theta <0$.

A linearly inseparable dichotomy may be nonlinearly separable. A dichotomy of set $\mathcal{X}$, denoted $\left\{{\mathcal{C}}_1,{\mathcal{C}}_2\right\}$, is φ-separable if there exists a mapping $\phi :R^{J_1}\to R^{J_2}$ [18]

$${\overrightarrow{w}}^T\phi \left(\overrightarrow{x}\right)=0$$

(8)

such that ${\overrightarrow{w}}^T\phi \left(\overrightarrow{x}\right)\ge 0$ when $\overrightarrow{x}\in {\mathcal{C}}_1$ and ${\overrightarrow{w}}^T\phi \left(\overrightarrow{x}\right)<0$ when $\overrightarrow{x}\in {\mathcal{C}}_2$, where $\overrightarrow{w}\in R^{J_2}$. The mapping $\phi$ is referred to as the concept of functional link network [19], which can map a linearly inseparable dichotomy so that it becomes separable. In particular, when replacing the linear term in an LTG by a high-order polynomial, a polynomial threshold neuron can be used to solve the nonlinearly separable problem. A detailed analysis on the second-order threshold neuron was provided in [20]. However, in the functional link approach, the way to choose the mapping was not addressed [19]. The support vector machine (SVM) algorithm [21] was developed around the same time and could automatically construct the mapping $\phi$ and the weight vector $\{\overrightarrow{w},\theta \}$.

2.3. Boolean Function Approximation

In Boolean function approximation, we would like to use a neural network to model a logic or Boolean function. The input and output values of the neural network are Boolean variables. There exist $2^{J_1}$ combinations for $J_1$ independent Boolean variables, yielding $2^{2^{J_1}}$ different Boolean functions. Some Boolean functions can be realized using LTGs.

For a set of m points in ${\mathbb{R}}^n$, if every subset of m or fewer points is linearly independent, it is known to be in general position. The function-counting theorem [18] estimates the separating capability of an LTG. It counts the number of linearly separable dichotomies of m points that are in general position in ${\mathbb{R}}^n$.

The probability of linear dichotomy is denoted $P(m,n)$, for an n-input LTG to separate m points in general position. According to the function-counting theorem [22], $P(m,n)=1$ when $\frac{m}{n+1}\le 1$, $P(m,n)\to 1$ when $1<\frac{m}{n+1}<2$ and $n\to \infty$, and $P(m,n)=\frac{1}{2}$ when $\frac{m}{n+1}=2$. When characterizing the statistical capability of a single LTG, $\frac{m}{n+1}=2$ is typically adopted.

2.4. Function Approximation

In function approximation, there is a training dataset, ${\mathcal{D}}_{train}=\{({\overrightarrow{x}}_1,y_1),\dots ,({\overrightarrow{x}}_N,y_N)\}$, where ${\overrightarrow{x}}_p$ and $y_p$ are the input and output of the pth sample, respectively. The input and output can be modeled by

$$y_p=f\left({\overrightarrow{x}}_p\right)+{\epsilon }_p,$$

(9)

where $f(·)$ is a mapping, and ${\epsilon }_p$ is a Gaussian noise with zero mean and variance ${\sigma }_{\epsilon }^2$. Our goal is to construct a model $\widehat{f}(·)$ to minimize the mean squared error (MSE):

$${\mathcal{J}}_{\mathrm{MSE}}=\frac{1}{N}\sum _{p=1}^N{(y_p-\widehat{f}({\overrightarrow{x}}_p))}^2.$$

(10)

3. Perceptron

3.1. Simple Perceptron

A simple perceptron is a one-neuron perceptron which is actually the McCulloch–Pitts neuron model [1], as shown in Figure 1. It uses the hard limiter as the activation function. Let ${\mathcal{D}}_{train}=\{({\overrightarrow{x}}_1,y_1),\dots ,({\overrightarrow{x}}_N,y_N)\}$ be the training set, where the ${\overrightarrow{x}}_p$’s are the training inputs, and the $y_p$’s are the training outputs that can be $\pm 1$ (or 1 or 0). In this notation, $y_p=1$ means that ${\overrightarrow{x}}_p\in {\mathcal{C}}_1$, and $y_p=-1$ means that ${\overrightarrow{x}}_p\in {\mathcal{C}}_2$. Our goal is to estimate a weight vector $\overrightarrow{w}$ and a threshold $\theta$ such that a hyperplane

$${\overrightarrow{w}}^T\overrightarrow{x}+\theta =0$$

(11)

separates the two classes:

$$\left\{\begin{array}{ccc}{\overrightarrow{w}}^T\overrightarrow{x}+\theta \hfill & \ge 0,\hfill & \mathrm{when}\overrightarrow{x}\in {\mathcal{C}}_1\hfill \\ {\overrightarrow{w}}^T\overrightarrow{x}+\theta \hfill & <0,\hfill & \mathrm{when}\overrightarrow{x}\in {\mathcal{C}}_2\hfill \end{array}\right..$$

(12)

When more neurons are used, a single-layer perceptron (SLP) is obtained, see Figure 2. An SLP can classify a vector $\overrightarrow{x}$ into more classes. The system state of a $J_1$–$J_2$ SLP is updated by

$$\overrightarrow{net}={\mathbf{W}}^T\overrightarrow{x}+\overrightarrow{\theta },$$

(13)

$$\overrightarrow{o}=\overrightarrow{\varphi }(\overrightarrow{net}),$$

(14)

where $\overrightarrow{net}={\left(ne{t}_1,\dots ,ne{t}_{J_2}\right)}^T$ is the net input vector, $\overrightarrow{o}={\left(o_1,\dots ,o_{J_2}\right)}^T$ is the output vector, $\overrightarrow{\varphi }(\overrightarrow{net})={\left({\varphi }_1\left(ne{t}_1\right),\dots ,{\varphi }_{J_2}\left(ne{t}_{J_2}\right)\right)}^T$, and ${\varphi }_i$ is the activation function of the ith output neuron.

3.2. Perceptron Learning Algorithm

In the perceptron learning algorithm [3,5], the training patterns are sequentially and repeatedly presented. Given a training pattern ${\overrightarrow{x}}_t$ at time t, we have the updating rule

$$ne{t}_{t,j}=\sum _{i=1}^{J_1}{x}_{t,i}{w}_{ij}\left(t\right)+{\theta }_j={\overrightarrow{w}}_j^T{\overrightarrow{x}}_t+{\theta }_j,$$

(15)

$$o_{t,j}=\left\{\begin{array}{cc}1,\hfill & \mathrm{when}ne{t}_{t,j}>0\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(16)

$$e_{t,j}=y_{t,j}-o_{t,j},$$

(17)

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{w}_{ij}(t+1)\hfill & =\hfill & w_{ij}\left(t\right)+\eta x_{t,i}{e}_{t,j}\hfill \\ {\theta }_j(t+1)\hfill & =\hfill & {\theta }_j\left(t\right)+\eta e_{t,j}\hfill \end{array}\right.,\end{array}$$

(18)

where $ne{t}_{t,j}$ stands for the net input of neuron j for example t, $w_{ij}$ the weight from neurons i to j, ${\overrightarrow{w}}_j={\left(w_{1j},w_{2j},\dots ,w_{J_{1}j}\right)}^T$, ${\theta }_j$ the threshold of neuron j, $x_{t,i}$ input i of example t, $o_{t,j}$ and $y_{t,j}$, respectively, the output and the desired output of neuron j for example t, $e_{t,j}$ the error at neuron j for example t, and $\eta$ the learning rate. $w_{ij}$s are initialized at random, and $\eta$ is usually set to $0.5$. Learning terminates when the errors $e_{t,j}$’s are equal to zeros in a learning cycle. The perceptron convergence theorem [3,5] asserts that when the training dataset for input patterns are linearly separable, the perceptron learning algorithm can always find decision hyperplanes to correctly separate classes in finite time.

The perceptron learning algorithm is valid only for the classification of linearly separable patterns. When trained with linearly inseparable patterns, it cannot stop the iteration [5]. Failure to converge for linearly inseparable problems is due to the fact that the perceptron learning algorithm is not able to detect the minimum of the error function [23]. Some variants of the perceptron learning, such as the pocket algorithm [24] and the thermal perceptron learning [25], are used for handling linearly inseparable patterns. The pocket algorithm adds a checking amendment to terminate the perceptron learning algorithm, achieving the minimization of the erroneous classification rate [24]. The weight vector having the longest unchanged run is treated as the best solution so far and is put into the pocket. The pocket convergence theorem asserts the convergence of the pocket algorithm, when the inputs are integers or rational [24,26]. The thermal perceptron learning [25] introduces a temperature annealing factor into the second term of (18).

3.3. Least Mean Squares Algorithm

The LMS algorithm [4] can reliably separate the patterns of different classes by minimizing the MSE by using the gradient-descent method. The linear activation function is used at the training stage, and thus, unlike (17), the error is given by

$$e_{t,j}=y_{t,j}-ne{t}_{t,j},$$

(19)

where $ne{t}_{t,j}$ is the same as (15). The weight update is given by (18). At the operation stage, thresholding is performed on the linear output to generate a binary output. The node containing a linear combiner and thresholding is referred to as Adaline.

The LMS algorithm can also be used for estimating the parameters in linear networks [27,28,29] for linear regression. In linear regression, we have a training set ${\mathcal{D}}_{train}=\{({\overrightarrow{x}}_1,{\overrightarrow{y}}_1),\dots ,({\overrightarrow{x}}_N,{\overrightarrow{y}}_N))$, where ${\overrightarrow{x}}_j\in {\mathbb{R}}^{J_1}$ is the input and ${\overrightarrow{y}}_j\in {\mathbb{R}}^{J_2}$ is the output. Our goal is to find the parameters such that the MSE

$$\mathcal{J}(\mathbf{W},\overrightarrow{\theta })=\frac{1}{N}\sum _{p=1}^N{\parallel {\overrightarrow{y}}_p-{\mathbf{W}}^T{\overrightarrow{x}}_p-\overrightarrow{\theta }\parallel }^2$$

(20)

is minimized.

The LMS rule comprising (15), (18) and (19) is known as the $\mu$-LMS rule. The learning parameter usually takes $0<\eta <\frac{2}{{max}_t{\rho }_{x_t}}$ to ensure its convergence, where ${\rho }_{x_t}={∥\left[{\overrightarrow{x}}_t^{T}1\right]∥}^2$.

The Widrow–Hoff delta rule, known as the $\alpha$-LMS, is obtained by normalizing the input vector [30]:

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{w}_{ij}(t+1)\hfill & =\hfill & w_{ij}\left(t\right)+\eta \frac{x_{t,i}{e}_{t,j}}{{\rho }_{x_t}}\hfill \\ {\theta }_j(t+1)\hfill & =\hfill & {\theta }_j\left(t\right)+\eta \frac{e_{t,j}}{{\rho }_{x_t}}\hfill \end{array}\right.,\end{array}$$

(21)

where $0<\eta <2$ ensures convergence, and $\eta$ is practically selected as $0.1<\eta <1.0$ [30].

In the above, the convergence results [4,30,31] are based on the stochastic setting, where the training samples are independently drawn over time. Under the condition that the training samples are cyclically and sequentially presented, some convergence results were developed in [32,33]. In [32], the result showed that the LMS rules converged to a limit cycle. The limit cycle reduced to an LMS solution when $\eta \to 0$. That meant there existed a tradeoff on $\eta$ between the final error and convergence speed. In [33], the LMS algorithm, with decreasing $\eta$, generated an $ϵ$-optimal weight matrix, which was at most $ϵ$ away from the LMS solution, after $O\left(\frac{1}{ϵ}\right)$ training cycles. For a fixed $\eta$, it needed $\Omega (\frac{1}{ϵ}log\frac{1}{ϵ})$ training cycles to obtain an $ϵ$-optimal weight matrix. The Widrow–Hoff delta rule laid the foundation of modern adaptive signal processing. There is also a complex LMS [34].

3.4. Other Learning Algorithms

Examples of other learning rules for the SLP are Mays’ rule [22,30,35], Ho–Kashyap rule [36,37], adaptive Ho–Kashyap rules [22,38], voted perceptron algorithm [39,40], perceptron algorithm for the perceptron using margins [39,41] or margitron [42], one-shot Hebbian learning [43], and nonlinear Hebbian learning [44].

Other examples for SLP learning algorithms are linear programming (LP) [37,45], convex analysis and nonsmooth optimization [23], constrained steepest-descent algorithm based on quadratic programming (QP) [46], fuzzy perceptron [47,48], fuzzy pocket [48], the conjugate-gradient (CG) method [49,50], control-inspired parameter selection for iterative steepest descent and CG algorithms [51], and shifting perceptron algorithm [52].

3.5. Approximation Ability of Perceptron Variants

The nonlinearity of the activation function introduces local minima in the MSE functions. The number of local minima grows exponentially as the input dimension increases [53].

The classification capability of a sign-constrained perceptron was analyzed in [54]. The authors derived a necessary and sufficient criterion for a sign-constrained perceptron to learn all $2^m$ dichotomies over a given set of m patterns. For a perceptron with n inputs, the VC-dimension is $n+1$, whereas for a sign-constrained perceptron, it is n [54].

When training a perceptron with a set of nonlinearly separable input patterns, the obtained weights may have a limit cycle behavior [55]. The least number of updates for the weights to reach the limit cycle is dependent on the initial weights. The authors of [55] also derived a necessary and sufficient condition for the weights exhibiting a limit cycle behavior, and estimated the number of updates for the weights to reach the limit cycle. An invariant set of the weights that were trained by the perceptron learning algorithm was characterized in [56].

When its majority vote is treated as a binary output, the SLP can compute any Boolean function [57]. When applying a squashing function to the percentage of votes with value 1, the SLP is a universal approximator for any continuous functions in $[0,1]$ [57].

The parallel perceptron implements a soft-winner-take-all gate on the binary outputs of gates on the hidden layer. It is a committee machine. The parallel perceptron is a universal approximator [57]. The parallel delta (p-delta) rule [57] implements a maximization of the margin of individual perceptrons. When trained with the p-delta rule, parallel perceptrons have performance comparable to that of an MLP and an SVM [57]. Direct parallel perceptrons [58] calculate the weights of parallel perceptrons by using an analytical closed-form expression, and they have a linear complexity in terms of both the number of patterns and the input dimension.

4. Multilayer Perceptrons

4.1. Structure

To overcome the linearly inseparable limitation, one can extend the SLP to a multilayered structure, such as an MLP. Madaline models are available by stacking multiple layers of Adalines [30]. Madaline is unable to solve the linearly inseparable problem, as Adaline is a linear network whose consecutive layers can be aggregated into a single layer by multiplying their respective weight matrices.

Madaline I [59,60] and Madaline II [61] are multilayered feedforward networks based on hard limiter neurons or other fixed-logic devices, such as AND, OR, majority-voting elements. Since the hard limiter function is not differentiable, the Madaline I and Madaline II learning rules are based on the LMS rule. The modification of weights is based on the minimal disturbance principle. This disturbance-based updating algorithm is quite slow. They can solve the linearly inseparable problem.

The neuron with the smallest net output in the first hidden layer is selected first and its output is inverted. If the inversion can improve the performance, the input weights of the selected neuron are updated according to the LMS algorithm, then applied to other neurons in the first hidden layer. After all first hidden layer neurons have been considered, the updating is moved to upper layer neurons. The learning speed of Madaline II’s rule is slow because we need to feed the training dataset to the network for each inversion of neurons.

A feedforward neural network (FNN) [6] usually has one or more hidden layers between the input and output layers. Such layered FNN is called an MLP. The architecture of the MLP is shown in Figure 3. There are M layers, and layer m has $J_m$ nodes. All the input nodes are collectively treated as the first layer. We denote the weights from layer $(m-1)$ to layer m by ${\mathbf{W}}^{(m-1)}$, and the bias, output and activation function of neuron i in layer m by ${\theta }_i^{\left(m\right)}$, $o_i^{\left(m\right)}$ and ${\varphi }_i^{\left(m\right)}(·)$, respectively. Subscript p and superscript $\left(m\right)$ correspond to sample p and layer m, respectively. In other words, the input of sample p is

$${\overrightarrow{x}}_p\equiv {\overrightarrow{o}}_p^{\left(1\right)},$$

(22)

the network output of sample p is

$${\overrightarrow{o}}_p\equiv {\overrightarrow{o}}_p^{\left(M\right)},$$

(23)

and the desired output of sample p is denoted as ${\overrightarrow{y}}_p$. For $m=2,\dots ,M$,

$${\overrightarrow{net}}_p^{\left(m\right)}={\left[{\mathbf{W}}^{(m-1)}\right]}^T{\overrightarrow{o}}_p^{(m-1)}+{\overrightarrow{\theta }}^{\left(m\right)},$$

(24)

$${\overrightarrow{o}}_t^{\left(m\right)}={\overrightarrow{\varphi }}^{\left(m\right)}\left({\overrightarrow{net}}_p^{\left(m\right)}\right),$$

(25)

where ${\overrightarrow{net}}_p^{\left(m\right)}=(ne{t}_{p,1}^{\left(m\right)},\dots$, $ne{t}_{p,J_m}^{\left(m\right)}{)}^T$, ${\mathbf{W}}^{(m-1)}\in {\mathbb{R}}^{J_{m-1}\times J_m}$, ${\overrightarrow{o}}_p^{(m-1)}={\left(o_{p,1}^{(m-1)},\dots ,o_{p,J_{m-1}}^{(m-1)}\right)}^T$, ${\overrightarrow{\theta }}^{\left(m\right)}={\left({\theta }_1^{\left(m\right)},\dots ,{\theta }_{J_m}^{\left(m\right)}\right)}^T$, and ${\overrightarrow{\varphi }}^{\left(m\right)}(·)$ applies ${\varphi }_i^{\left(m\right)}(·)$ to the ith entry of the vector within.

For classification, all hidden and output neurons are with a sigmoidal function or hard limiter. For function approximation, all hidden neurons are with a sigmoidal function, but the output neurons in layer M use a linear activation function.

When all hidden neurons and output neurons are with a hard limiter activation, an MLP can be considered as a Madaline II [61].

4.2. Approximation Ability

The MLP model has a universal approximation ability when the number of hidden nodes is sufficiently large. The universal approximation capability of a four-layer MLP can be easily obtained from Kolmogorov’s theorem [62] and has also been proved in [63,64,65]. More importantly, the three-layer MLP model with continuous sigmoidal activation functions has been proved to be capable of approximating any continuous multivariate function to an arbitrary accuracy [66,67,68,69,70].

For approximating a target function by the MLP, the necessary number of neurons relies on the geometry of the target function, as well as the minimal number of line segments/hyperplanes [69] or the number of extrema [71] that can construct the basic geometrical shape of the target function.

4.3. Backpropagation Learning Algorithm

The BP algorithm has long been the most popular and fundamental learning algorithm for MLPs [6,7,72,73]. The BP algorithm is a gradient-search technique that minimizes a cost function. The cost function is defined by the MSE:

$$\mathcal{E}=\frac{1}{N}\sum _{p=1}^N{\mathcal{E}}_p=\frac{1}{2N}\sum _{p=1}^N{∥{\overrightarrow{y}}_p-{\overrightarrow{o}}_p∥}^2,$$

(26)

where N is the size of the training set, ${\overrightarrow{o}}_p$ and ${\overrightarrow{y}}_p$ are, respectively, the actual network output and the desired output for training pattern p,

$${\mathcal{E}}_p=\frac{1}{2}{∥{\overrightarrow{y}}_p-{\overrightarrow{o}}_p∥}^2=\frac{1}{2}{\overrightarrow{e}}_p^T{\overrightarrow{e}}_p$$

(27)

is the error of training pattern p, and

$${\overrightarrow{e}}_p={\overrightarrow{y}}_p-{\overrightarrow{o}}_p,$$

(28)

the element i of ${\overrightarrow{e}}_p$ being $e_{p,i}=y_{p,i}-o_{p,i}$.

When minimizing ${\mathcal{E}}_p$, we have the change of the weight or bias:

$${\Delta }_p{w}_{ij}^{(m-1)}=-\eta \frac{\partial {\mathcal{E}}_p}{\partial w_{ij}^{(m-1)}}\mathrm{and}{\Delta }_p{\theta }_i^{\left(m\right)}=-\eta \frac{\partial {\mathcal{E}}_p}{\partial {\theta }_i^{\left(m\right)}},$$

(29)

where the learning rate $\eta$ is a small positive number. For the output layer, the changes of the parameters are given by

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{\Delta }_p{w}_{ij}^{(M-1)}\hfill & =\hfill & \eta e_{p,i}\frac{\partial o_{p,i}}{\partial w_{ij}^{(M-1)}}\hfill \\ & =\hfill & \eta e_{p,i}{\left(\frac{d{\varphi }_i^{\left(M\right)}\left(a\right)}{da}\right)}_{a=ne{t}_{p,i}^{\left(M\right)}}{o}_{p,j}^{(M-1)}\hfill \\ {\Delta }_p{\theta }_i^{\left(M\right)}\hfill & =\hfill & \eta e_{p,i}\frac{\partial o_{p,i}}{\partial {\theta }_i^{(M-1)}}\hfill \\ & =\hfill & \eta e_{p,i}{\left(\frac{d{\varphi }_i^{\left(M\right)}\left(a\right)}{da}\right)}_{a=ne{t}_{p,i}^{\left(M\right)}}1\hfill \end{array}\right..\end{array}$$

(30)

If we can consider the term $e_{p,i}{\left(\frac{d{\varphi }_i^{\left(M\right)}\left(a\right)}{da}\right)}_{a=ne{t}_{p,i}^{\left(M\right)}}\equiv {\delta }_{p,i}^{\left(M\right)}$ as a generalized error term, the updating rule in BP can be considered as a generalized delta rule [4]. To calculate the changes of parameters in the hidden and input layers, we can apply the chain rule and then get

$${\Delta }_p{w}_{uv}^{(m-1)}=\eta {\delta }_{p,u}^{\left(m\right)}{o}_{p,v}^{(m-1)}\mathrm{and}{\Delta }_p{\theta }_u^{\left(m\right)}=\eta {\delta }_{p,u}^{\left(m\right)}1.$$

(31)

The generalized error terms ${\delta }_{p,u}^{\left(m\right)}$’s can be solved by applying the chain rule. The changes of the parameters in the output layer are first calculated. Then, the generalized error terms are backpropagated by means of the chain rule and are used for calculating the generalized error terms in the hidden layers. The BP algorithm requires a continuous differentiable activation function.

4.4. Batch Mode and Online Mode

There are two versions of the BP algorithm, batch and online modes. In the batch mode, the parameters are updated only after a complete presentation of training patterns. In the online mode, the parameters are updated at once after a training pattern is presented. In the online mode, the usual practice is that all the examples are presented cyclically by epoch and in a random order during each epoch, until the convergence criteria are reached. In either batch or online mode BP, the concept follows that of the gradient-descent technique. In the literature, batch mode BP is commonly known as gradient descent, while online mode BP is known as stochastic gradient descent. In BP, too small an $\eta$ increases the possibility of getting stuck at a local minimum, while too large an $\eta$ may result in oscillatory traps.

Since the batch mode BP follows the standard gradient-descent technique, the convergence (to a local minimum) is guaranteed when a sufficiently small $\eta$ is used. There are several works on the convergence for the online mode BP [74,75,76,77,78,79]. Based on the stochastic gradient theorem [74,75,76], if the training examples are randomly drawn from an infinitely large dataset, the convergence of the online mode BP is guaranteed when a decreasing learning rate ${\eta }_t$ is used, ${\sum }_{t=1}^{\infty }{\eta }_t=\infty$, and ${\sum }_{t=1}^{\infty }{\eta }_t^2<\infty$, where t is the time index for training. For a dataset with finite size, some convergence results are given in [77,78,79,80], where the assumptions are that the learning rate is fixed to a constant within a learning cycle, and that it deceases to zero as learning cycles increase.

Online BP can be implemented when a complete training set is unavailable. This is usually implemented for very large training sets. Online BP has a quasi-simulated annealing property arising from the randomness introduced by the small constant learning rate, and this property helps the algorithm escape local minima [81]. However, online BP is not suitable for parallelization. It is usually orders of magnitude faster than batch BP, but with at least the same accuracy, especially for large training sets [82]. Batch BP is usually too slow for large training sets due to the required small learning rate. Online BP can train more quickly and safely by using a larger learning rate. In [82], a recommendation on the learning rates between online BP and batch BP is given by ${\eta }_{\mathrm{online}}=\sqrt{N}{\eta }_{\mathrm{batch}}$. When $\eta$ is small enough, the number of epochs required for learning has a linear relationship with $\eta$.

With respect to the absolute value of the expected difference and the expected squared difference, and with any analytic sigmoidal function, the convergence for online BP and batch BP are derived and compared for different $\eta$ in [83]. For batch BP, $\eta \ge 2$ is required for convergence, while for online BP, $\eta$ increases up to N. In [84], for online BP training of three-layer feedforward networks subject to the minimum error, a dynamic learning rate ensures the error sequence converges to the global minimum error. In [85], the maximal learning rates are derived as a function of batch size for both gradient descent and adaptive algorithms, such as Adam, by using random matrix theory.

4.5. Momentum Term

By adding a momentum term to the BP algorithm, the BP with momentum (BPM) algorithm is obtained as [6]

$$\begin{array}{c}\hfill \left\{\begin{array}{ccc}{\Delta }_p{w}_{ij}^{(m-1)}(t+1)\hfill & =\hfill & -\eta \frac{\partial {\mathcal{E}}_p}{\partial w_{ij}^{(m-1)}}+\alpha {\Delta }_p{w}_{ij}^{(m-1)}\left(t\right)\hfill \\ {\Delta }_p{\theta }_i^{\left(m\right)}(t+1)\hfill & =\hfill & -\eta \frac{\partial {\mathcal{E}}_p}{\partial {\theta }_i^{\left(m\right)}}+\alpha {\Delta }_p{\theta }_{ij}^{(m-1)}\left(t\right)\hfill \end{array}\right.,\end{array}$$

(32)

where $\alpha$ is the momentum factor; usually $0<\alpha \le 1$ and typically $\alpha =0.9$. The momentum term can effectively improve the convergence when the estimated weights are in almost-flat steady downhill regions of the error surface and has a stabilizing effect when the estimated weights are in regions with high fluctuations. The momentum term introduces second-order information, and thus BPM resembles the CG method [49,50]. BPM is analyzed in [86] and the conditions for convergence are derived.

The convergence rate and MSE performance of BPM was investigated in [87]. BPM can be viewed as standard stochastic gradient method having a rescaled but larger step size. A decaying momentum factor can retain adaptation but avoid performance degradation. A stability analysis of two BPM algorithms for quadratic functions derives the optimal learning rates and the optimal momentum factors simultaneously, leading to the fastest convergence [88].

4.6. Variance Reduction

For stochastic gradient descent, the variance of the randomly drawn gradients will never go to zero. A sublinear convergence rate is obtained only under decreasing step-size sequences. The variance-reduced technique reduces the variance to zero.

The stochastic average gradient (SAG) method [89] uses the sum of the latest individual gradients as an estimator of the descent direction. SAG has a linear convergence rate using a step size $\eta =O(1/L_{max})$, $L_{max}$ being a Lipschitz constant. SAG requires $O\left(nd\right)$ storage and uses a biased gradient estimator, for n training examples in $R^d$. SAG only computes a single stochastic gradient at each iteration. Stochastic variance reduced gradient (SVRG) [90] uses an unbiased gradient estimator. SAGA [91] improves SAG by using an unbiased update of SVRG. SAGA has the same linear convergence and storage as SAG, but with a much simpler proof. SVRG [90] has performance similar to SAG and SAGA, but only requires $O\left(d\right)$ memory. SVRG needs to tune the number of inner iterations, two gradients are computed per iteration, and the full gradient needs to be computed every time the reference point is changed. However, their step size depends on $L_{max}$, which may not be difficult to decide for some problems. The convergence rate of the variance-reduced methods depends on the number of training examples n, while that of classic SGD is not dependent on n. This means that variance-reduced methods can perform worse than SGD in the early iterations when n is very large.

The stochastic dual coordinate ascent (SDCA) method [92] extends coordinate descent methods to the finite-sum problem. Stochastic coordinate descent implements a coordinate-wise gradient descent with respect to a randomly chosen variable and updates this variable while keeping the other variables unchanged.

Recent developments to accelerate the convergence of stochastic optimization by exploiting second-order information are discussed in [93]. This is achieved by stochastic variants of quasi-Newton methods that approximate the curvature of the objective function using stochastic gradient information.

5. Generalization

5.1. Generalization Error

A neural network trained by minimizing the MSE on a training set is not guaranteed to perform well on an unseen test set. Therefore, we are more interested in the generalization ability, which is the performance of a trained network on unseen samples. When a network is over-trained with too many parameters and training epochs, or too few training examples, good results may be produced for the training examples, but the network generalizes poorly. This is known as the overfitting phenomenon. In statistics, when a model has an excessive number of parameters and fits the noise in the data, overfitting occurs. A network’s generalization capability is decided by three factors, namely, the training set size, problem complexity, and network architecture [94,95].

The generalization error consists of a bias and a variance term [96]. The bias term arises from the finite number of parameters of the model, known as the approximation ability of the network. The variance term arises from the finite number of training data. For an MLP with $J_1$ inputs and one output, the total generalization error is bounded as a function of the order of the number of hypothesis parameters, P, and the number of examples, N [97,98]. A larger P leads to a smaller approximation error due to a larger model, but also leads to a larger estimation error due to overfitting. When $P\propto N^{\frac{1}{3}}$, the best tradeoff between the approximation and estimation errors is achieved, and the generalization error for the MLP is maintained at $O\left(\frac{1}{P}\right)$ [97]. This result is in agreement with that of an MLP with a sigmoidal activation [99].

Cross-validation is a traditional way to measure the generalization ability of a trained network. In the simple cross-validation scheme, the dataset is partitioned into a training set and a test set. The training set is used for training the network, while the test set is used for evaluating the generalization ability of the trained network.

5.2. Generalization by Stopping Criterion

Overtraining can be avoided by stopping the training process before the global minimum is reached. Neural networks trained with iterative gradient descent gradually learn a mapping by increasing components of frequency. When stopping the training at a due point, the network will not learn the high-frequency noise. The training error always decreases with time, but the generalization error decreases to a minimum and then rises as the network is overtrained.

Cross-validation is performed to decide when to stop. Three early stopping criteria are empirically compared in [100]. A slower stopping criterion on average yields a small improvement in generalization, but requires a much longer training time [100]. A statistical analysis of overtraining on the three-layer MLP is given in [101].

5.3. Generalization by Regularization

Regularization [95,102,103,104,105] is a popular approach to improving generalization. Let $\overrightarrow{w}$ be the collection of all weights and bias terms. The training error $\mathcal{E}$ is a function of $\overrightarrow{w}$. A positive penalization term ${\mathcal{E}}_{\mathrm{c}}\left(\overrightarrow{w}\right)$ is added to the training objective to penalizes poor generalization. Then, the overall objective function is given by

$${\mathcal{E}}_T\left(\overrightarrow{w}\right)=\mathcal{E}\left(\overrightarrow{w}\right)+{\lambda }_{\mathrm{c}}{\mathcal{E}}_{\mathrm{c}}\left(\overrightarrow{w}\right),$$

(33)

where ${\lambda }_{\mathrm{c}}$ is the regularization parameter. The regularization method is valid for the iterative gradient-based techniques as well as the one-step optimization such as a singular value decomposition (SVD). Regularization decreases the representation capability of the network, i.e., it increases the bias term but improves the variance term (bias–variance dilemma [96]).

Two most common regularizers employed to improve the generalization ability are, respectively, the weight-decay method [106,107] and the input perturbation method [102]. In the simple weight-decay, the constraint term is given by

$${\mathcal{E}}_{\mathrm{c}}\left(\overrightarrow{w}\right)=\sum _{i=1}^P{w}_i^2.$$

(34)

The inclusion of a weight-decay constraint term results in a small magnitude for the trained weights, yielding a smooth network output function and an improved generalization ability [106]. The input perturbation method [102] requires a robustness in the approximation ability to input perturbations. Let $o_{j,p}$ be the jth element of the network output with respect to the pth training sample, and $x_{i,p}$ be the ith element of the pth training input vector. In the input perturbation method [102], the constraint term is given by

$${\mathcal{E}}_{\mathrm{c}}\left(\overrightarrow{w}\right)=\sum _{p=1}^N\sum _{i=1}^{J_1}\sum _{j=1}^{J_M}{\left(\frac{\partial o_{j,p}}{\partial x_{i,p}}\right)}^2.$$

(35)

Network pruning techniques help to improve generalization as well [108,109]. At the end of the training of the weight-decay procedure, those connections with small weights can be removed from the network. One can improve generalization by training a network, when a small amount of noise (jitter) is added to the weights and inputs but no noise is added to the output [104,108,109,110]. In sum, the effect of adding noise in weights is similar to that of the weight-decay regularizer. On the other hand, the effect of adding noise in the input is similar to that of the input perturbation method [104,109].

Early stopping behaves like a simple weight-decay technique [104] in terms of the evolution of the effective number of weights. Weight sharing is used to reduce the number of network parameters, thus improving generalization [6,111].

The step size can be viewed as a regularization parameter [112]. The stochastic gradient method is analyzed in [113]. Regularization is controlled by the step size, the number of passes, and the minibatch size.

5.4. Selection of Regularization Parameter

The variance term can be controlled when the regularization parameter is appropriately chosen. However, too large a regularization parameter will yield an excessive bias term. Therefore, it is crucial to select a regularization parameter.

An appropriate regularization parameter can be selected by cross-validation [100,114,115]. The dataset is randomly divided into a training set and a validation (test) set, with the major portion of the dataset included in the training set and the remaining in the validation set. If only one sample is left for validation, the method is known as leave-one-out cross-validation. A number of networks are trained with different regularization parameters, and then the best trained network is selected based on cross-validation. However, cross-validation may be limited by the dataset size and it is a time-consuming process.

The regularization parameter can also be selected by generalization error estimation methods [94,95,116,117,118,119,120,121,122]. The methods are based on information criteria, such as the Akaike information criterion (AIC) [123], Schwartz’s Bayesian information criterion (BIC) [124], and Rissanen’s minimum description length (MDL) principle [125,126]. Most of these criteria can be expressed as a sum of two components, measuring the training error and penalizing the complexity, respectively [125,127]. For example, a good generalization can be realized when encoding the weights with short bit-lengths based on the MDL principle [110]. In these approaches, we have a generalization error estimation formula which is a function of the training error and trained weights. A number of networks are trained with different regularization parameters, and then the best trained network is chosen based on the estimated generalization errors.

The third approach is to estimate the regularization parameter during training based on a Bayesian framework [103,107,128,129]. In those works, the regularization term comes from the assumption that a prior probability is assigned to the weights. Then, the regularization parameter is related to the variance of the weights. In [103,128,129], the results focused on radial basis function (RBF) networks. The experiments in [128,129] successfully demonstrated the advantage of using this approach to select RBF centers and to automatically select the regularization parameter. In [107], the results can be applied to general MLPs.

6. Optimization on Network Size

Given a problem, an appropriate network size is very important. Too small a network will lead to underfitting, i.e., too large a bias term, corresponding to an inaccurate approximation. In contrast, too large a network will lead to overfitting, i.e., too large a variance term, corresponding to a poor generalization ability. This section introduces some approaches for finding an optimal network size.

6.1. Destructive Approach: Network Pruning

Network pruning starts with a network with a large size, then removes the redundant nodes or weights during or after training. Pruning methods can be either sensitivity-based or regularization-based methods [130,131]. In sensitivity-based methods, the sensitivity of the objective function $\mathcal{E}$ with respect to the removal of a weight or node is first estimated, and the least important weight or node is then removed. The regularization-based methods adds a regularization term to the training objective to punish a network of complex architecture. Some network pruning algorithms are surveyed in [130].

6.1.1. Sensitivity-Based Network Pruning

In the sensitivity-based pruning, the sensitivity measure is utilized to characterize the contribution of individual weights or nodes when solving a problem, and the less important weights or nodes are then removed. Let ${\mathcal{E}}_T$ be the training objective and w be a weight of the output of a hidden node. The normalized sensitivity is defined by

$$S_w^{{\mathcal{E}}_T}=\underset{\Delta w\to 0}{lim}\frac{\frac{\Delta {\mathcal{E}}_T}{{\mathcal{E}}_T}}{\frac{\Delta w}{w}}=\frac{w}{{\mathcal{E}}_{\mathcal{T}}}\frac{\partial {\mathcal{E}}_T}{\partial w}.$$

(36)

In the skeletonization technique [132], the sensitivity measure is defined as $S_w^{{\mathcal{E}}_T}=-w\frac{\partial {\mathcal{E}}_T}{\partial w}$, which utilizes the first-order approximation. This sensitivity measure was applied in Karnin’s pruning method, where no retraining procedure was applied [133]. This method was improved to avoid removing an input node or a particular hidden layer and to incorporate a fast retraining algorithm in [134], or to introduce the local relative sensitivity index in [135]. In [136], a loss-based sensitivity regularization is implemented by shrinking and then pruning parameters with low loss sensitivity. A node-sensitivity regularized node pruning scheme is proposed in [137].

Sensitivity-based methods that use linear models for hidden nodes are developed in [131,138], where redundant hidden nodes are well-modeled as linear models of their net inputs or linear combinations of the outputs of the other nodes. In [139], network pruning is implemented based on orthogonal transforms, e.g., SVD and QR decomposition. In [140], SVD is used to evaluate the significance of the number of hidden layer neurons of a trained network, based on which a pruning/growing technique is implemented. Pruning exploiting PCA of the node activations of successive layers is implemented in [141]. In [142], mutual information is used to prune both the input and hidden nodes. Other sensitivity-based methods utilizing retraining are described in [143,144].

6.1.2. Second-Order Expansion Pruning

Based on the second-order Taylor expansion of the objective function, the optimal brain damage (OBD) [145] and optimal brain surgeon (OBS) [146] procedures were developed around the early 1990’s. Let $\overrightarrow{w}$ be the collection of all weights and biases. At the time of convergence, the gradient approaches zero and thus a change in $\overrightarrow{w}$ leads to an increase in $\mathcal{E}$:

$$\Delta \mathcal{E}\simeq \frac{1}{2}\Delta {\overrightarrow{w}}^T\mathbf{H}\Delta \overrightarrow{w},$$

(37)

where $\mathbf{H}=\left[\frac{{\partial }^2\mathcal{E}}{\partial w_i\partial w_j}\right]$ is the Hessian matrix. A weight $w_i$ can be removed by setting $\Delta w_i=w_i$ and all $\Delta w_j=0$, $j\ne i$, and this introduces a change in $\mathcal{E}$.

OBS is derived from the saliency (37), while OBD is an OBS having diagonal $\mathbf{H}$. A weight with the smallest saliency is deleted. Optimal cell damage (OCD) [147] is obtained by first implementing OBD and further removing irrelevant input and hidden nodes. Unit-OBS [148] modifies OBS by removing a whole node at each step. Based on a block-diagonal approximation of the Hessian, principal components pruning [141] lies between OBD and OBS. Early brain damage (EBD) [149] extends OBD and OBS by implementing early stopping, and it allows the revival of the already pruned weights. Variance nullity pruning [150] is derived from an output sensitivity analysis. In case of a diagonal $\mathbf{H}$, OBD and the output sensitivity analysis are equivalent, if a gradient search and the MSE function are used.

The above approaches rely on the efficient computation of the Hessian matrix. In [151], an efficient procedure for calculating the Hessian matrix was described. Instead of an exact calculation of the Hessian matrix, nowadays, the approximation [152]

$$\mathbf{H}\approx \left[\sum _{p=1}^N\sum _{k=1}^{J_M}\frac{\partial o_{p,k}}{\partial w_i}\frac{\partial o_{p,k}}{\partial w_j}\right]$$

(38)

is usually used.

RLS-based pruning exploits the error covariance matrix obtained during RLS training [152]. RLS-based pruning and OBD have a similar number of pruned weights and similar generalization ability, but with a considerable computational saving and suitable for online situation. In a similar way, extended Kalman filtering (EKF) based pruning techniques for MLPs and recurrent neural networks are given in [153,154].

6.1.3. Regularization-Based Methods

The complexity of a trained network can be measured by its effective number of parameters [95]. The penalty approach aims at reducing the effective number of parameters [95] by adding a regularization term into the objective function. Basically, there are two kinds of regularization approaches.

The first approach drives trained weights to have small magnitudes. The trained weights have a relatively flat distribution around zero. In this case, most trained weights have a contribution to the approximation ability of the trained network. In this approach, the regularization term is given by ${\overrightarrow{w}}^T\mathbf{R}\overrightarrow{w}$, where $\mathbf{R}$ is a positive-definite matrix. The standard weight-decay [155,156] is a representative method of this approach. Its regularization term is the sum of the squares of all the weights. This leads to the assumption that the a priori distribution [107] of weights is zero-mean Gaussian. During training, the decay effect of a weight is proportional to its magnitude. Hence, many trained weights are small but not very close to zero. The effective number of parameters of the trained network is small and the complexity of the trained network is small. The conventional RLS algorithm is actually a weight-decay algorithm [152], since they both use the same objective function, i.e., the sum of the squares of all the weights. The error covariance matrix from the RLS training has similar properties as $\mathbf{H}$. The initial value of the error covariance matrix can be used to control the generalization ability.

The second approach drives unnecessary weights to zero and removes them during training. In [157,158], the regularization term is the sum of the absolute values of the weights. This leads to the assumption that the a priori distribution of weights is zero-mean Laplacian. During training, the decay effect of a weight is a constant. Hence, many trained weights are very close to zero and a skeleton network is obtained.

Weight-smoothing regularization incorporates the Jacobian profile smoothness as a constraint [159]. Several other regularization techniques that use neural Jacobians are the double BP [160], the input perturbation [109], generalized regular network [161], and the approximate smoother [162].

When using the $L_1$-/$L_2$-norm of weight vectors at a group level, the method has a better generalization and pruning efficiency than weight decay, weight elimination, and approximate smoother [163]. In [164], a nonconvex integrated transformed $L_1$ regularizer applied to the weight matrix space is introduced to remove redundant connections and unnecessary neurons simultaneously. It is shown in [165] that compressing the weights of a layer has the same effect as compressing the input of the layer.

6.2. Constructive Approach: Network Growing

In the constructive approach, a small network is first selected and then hidden nodes are added until a specified performance is reached. The constructive approach has the ability to escape a local minimum by adding a new hidden node [166], since the process of adding neuron changes the shape of the error function. The weights of the newly added nodes can be randomly initialized.

Cascade-correlation learning [167] is a well-known constructive learning approach. The constructed network has direct connections between the input and output nodes. In the beginning, there is no hidden node. If the network cannot solve a problem after some training cycles, the one with the maximum covariance among a set of randomly initialized candidate nodes is added to the network. A newly added node is connected to the input nodes and all existing nodes. With previously trained nodes frozen, the network is trained by using the Quickprop algorithm [168]. The error signals are not backpropagated for training.

Many constructive algorithms are inspired from the cascade-correlation learning [169,170]. In [170], constructive BP allows the addition of multiple new nodes at a time, and continuous automatic structure adaptation, including both addition and deletion of nodes, can be performed. Cascade-correlation learning is not suitable for VLSI implementation due to the irregularity in the network architecture. By controlling the connectivity, cascaded-correlation learning generates a strictly layered architecture with a restricted fan-in and a small depth [171].

A quasi-Newton method for constructing the three-layer MLP was reported in [172]. The dependence identification (DI) algorithm [173] is a batch learning process. The MLP training problem is transformed into a set of quadratic optimization problems. In [174], a constructive training algorithm for the three-layer MLP is proposed for classification problems, where the Ho–Kashyap algorithm [37] is used to train the network and a pruning procedure is also incorporated. In [175], the proposed online constructive learning algorithm for the MLP integrates the weight scaling technique [176,177] for escaping local minima, and the quadratic programming (QP) and linear programming (LP)-based procedure for initializing the weights of a newly added neuron. In [178], the proposed constructive approach for MLP learning finds excellent solutions by exploiting the flat regions.

There are also some constructive methods for training MLPs with LTG neurons for classification, such as the tower algorithm [24], tiling algorithm [179], and upstart algorithm [180]. All three algorithms exploit the pocket algorithm [24].

7. Acceleration on BP

Due to the gradient-descent nature, the BP algorithm is slow in convergence. This section describes numerous measures for accelerating the convergence of the BP algorithm.

7.1. Eliminating Premature Saturation

Slow convergence is primarily due to the premature saturation of the output of the sigmoidal functions. When $net$ has a large absolute value, the derivative of $\varphi \left(net\right)$ is small and then the weight update is negligible. This leads to an excessively long learning time. This phenomenon is the well-known flat-spot problem. Premature saturation can be resolved by modifying the slope of the sigmoidal function, or by modifying the objective function.

An analysis of static premature saturation of output nodes is performed in [181]. This premature saturation arises from random initial weights at the beginning of training. In [182], a dynamic mechanism analysis for premature saturation identifies the momentum term as the leading cause of premature saturation. One can prevent premature saturation by temporarily modifying the momentum factor $\alpha$.

In [183], the BP update rule is revised by adding a term relating to the degree of saturation to the objective function. The additional term is a parabolic function of the outputs of the output nodes, given by ${\sum }_{i=1}^{J_M}\lambda {(o_i-0.5)}^n$, where $\lambda$ is a scale factor, n is an exponent, and $o_i$ is the actual output of the ith node in the output layer. Similarly, in [184], the partial derivatives of the logistic activation function are generalized so that the error signals are significantly enlarged when the outputs of the neurons (hidden or output) approach saturation. Other authors also modify the objective functions to avoid premature saturation [76,185]. However, the main drawback of those approaches is that there is no selection guide for the tunable parameters. For example, in [183], the selection rules for $\lambda$ and n were not adjusted.

7.2. Adapting Learning Parameters

The performances of BP and BPM rely on the values of the learning rate $\eta$ and the momentum parameter $\alpha$. There are some heuristics for accelerating the learning by adaptively adjusting $\eta$ and $\alpha$. According to [6], the process of starting with a large $\eta$ and then gradually decreasing it resembles simulated annealing [186]. For gradient-based methods, the learning parameters are usually updated once every epoch.

7.2.1. Globally Adapted Learning Parameters

All the weights are updated by using the global learning parameters, namely, the learning rate $\eta$ and momentum parameter $\alpha$. In [187], $\eta$ has its optimal value as the reciprocal of the largest eigenvalue of the Hessian $\mathbf{H}$ of the objective function. The largest eigenvalue is estimated by an online algorithm, without the calculation of the Hessian.

Instead of using complicated algorithms to update the learning parameters, a simple and popular search-then-converge schedule is developed in [188]. At the beginning, the schedule selects a large $\eta$, then decreases it gradually during the learning process. In the bold-driver technique [189,190], the learning rate $\eta$ and momentum parameter $\alpha$ are varied by four rules according to whether or not an update when training an epoch improves the training objective. The approach is able to escape from local minimum.

The BP rule, or the gradient-descent step, is formulated as

$$\overrightarrow{w}(t+1)=\overrightarrow{w}\left(t\right)-{\eta }_t\overrightarrow{g}\left(t\right),$$

(39)

where the gradient $\overrightarrow{g}\left(t\right)={\left.\frac{\partial \mathcal{E}}{\partial \overrightarrow{w}}\right|}_{\overrightarrow{w}=\overrightarrow{w}\left(t\right)}$. In [191], fast convergence is achieved by adapting $\eta$ for the batch BP according to the instantaneous values of $\mathcal{E}\left(t\right)$ and its gradient. The method, however, leads to a jumpy behavior of the weights, since $\eta$ is very large in the neighborhood of a local or global minimum. In [192], both ${\eta }_t$ and $\alpha$ are updated adaptively by a correlation-based heuristic. This algorithm achieves an exponential change of $\eta$, and outperforms the locally adapted learning-rate-based method proposed in [193].

In [194], $\eta$ is derived from the local approximation of the Lipschitz constant. The algorithm is robust against the jumpy behavior [191] and ensures that the objective function decreases for every weight update. It outperforms BP, delta-bar-delta [195], and the bold-driver technique [189].

The learning parameters are also adapted by the fuzzy inference system (FIS) [196]. The method incorporates Jacobs’ heuristics [195] on the unknown learning parameters by using fuzzy IF–THEN rules. The heuristics are driven by the behavior of $\mathcal{E}\left(t\right)$. $\eta$ and $\alpha$ are, respectively, adjusted by an FIS. Fuzzy BP converges much faster than BP, with much smaller MSE [196].

7.2.2. Locally Adapted Learning Parameters

In the locally adapted learning approach, each weight or bias term $w_i$ can have its own learning rate ${\eta }_{t,i}$ so that

$$\Delta w_i\left(t\right)=-{\eta }_{t,i}{g}_i\left(t\right).$$

(40)

Locally adaptive learning algorithms may use weight-specific learning rates, such as the methods proposed in [193,197], SuperSAB [198], delta-bar-delta [195], Quickprop [168], incremental equalized error BP (EEBP) [199], and the globally convergent strategy [200].

In [197], the authors set the learning rate corresponding to each input weight to a neuron to be inversely proportional to the neuron’s fan-in. This balances the learning speeds of nodes with different fan-ins. The method increases the convergence speed, and its justification is verified from the eigenvalue distribution of $\mathbf{H}$ [201].

In [193] and in SuperSAB [198], ${\eta }_{t,i}$ is heuristically adapted by

$${\eta }_{t+1,i}=\left\{\begin{array}{cc}{\eta }_0^{+}{\eta }_{t,i},\hfill & \mathrm{if}{g}_i\left(t\right)·g_i(t-1)>0\hfill \\ {\eta }_0^{-}{\eta }_{t,i},\hfill & \mathrm{if}{g}_i\left(t\right)·g_i(t-1)<0\hfill \end{array}\right.,$$

(41)

where ${\eta }_0^{+}>1$, $0<{\eta }_0^{-}<1$, and in SuperSAB, ${\eta }_0^{+}\simeq \frac{1}{{\eta }_0^{-}}$. Since ${\eta }_{t,i}$ changes exponentially, multiple successive acceleration steps will yield too large or too small ${\eta }_{t,i}$. For this reason, SuperSAB includes a momentum term. Delta-bar-delta [195] has a similar update for ${\eta }_{t,i}$, but implements linear acceleration and exponential deceleration. When the momentum term is included, delta-bar-delta diverges sometimes, and an adaptive momentum can be introduced to improve delta-bar-delta [202]. However, the selection of the parameters is more difficult for delta-bar-delta.

In [203,204], ${\eta }_{t,i}$ and ${\alpha }_{t,i}$ are derived using the second-order-based, first-order-based, and CG-based approaches, utilizing the first- and second-order derivatives of $\mathcal{E}$ with respect to ${\eta }_{t,i}$ and ${\alpha }_{t,i}$. The computational and storage complexities are at most three times that of standard BP, but with a tenfold speed. In [200], based on Wolfe’s conditions for linear search and the Lipschitz condition, the authors provide a general theoretical result for developing batch BP with local learning rates. Conditions for global convergence are given.

Quickprop [167,168,205] heuristically adapts ${\alpha }_{t,i}$. The method uses error gradient at two consecutive time steps, which is essentially a discrete approximation to second-order derivatives. Thus, Quickprop is a quasi-Newton method. Fahlman improved the flat-spot problem by adding $0.1$ to the derivative of the sigmoidal function [168].

7.3. Initializing Weights

The initial weights should be selected as close as possible to a global minimum before training. Improper weight initialization may lead to slow convergence, and the network may get stuck at a local minimum. All the weights are typically initialized with small random positive values, or with small zero-mean random numbers [6]. Randomness in weights helps to break the symmetry and thus reduce redundancy in the network. Experiments have demonstrated the extreme sensitivity of BP to the initial weight configuration, which has a complex fractal-like structure [206].

In [207], the authors give the maximum amplitude for the initial weights from a statistical analysis. It is theoretically verified in [181] that when the maximal value of the weights increases, the probability of a neuron becoming prematurely saturated increases.

7.3.1. Heuristics for Weight Initialization

An empirical optimal initialization of the weights is to set the weights terminating at a node to be uniformly distributed in $\left[-3/\sqrt{n},3/\sqrt{n}\right]$, where n is the number of weights terminating at the node [208,209]. A similar idea is implemented in [210]. A number of heuristics for weight initialization are also discussed and compared by comprehensive simulations in [208]. In [208], the authors recommend a weight range of $[-0.77,0.77]$ according to their empirical result.

In [211], for the three-layer MLP, the maximum magnitude of the weights between the hidden and output layers is inversely proportional to the hidden-layer fan-in, based on a multidimensional geometry analysis. The mini-max initialization technique [71] approximates continuous functions by using the number of extrema to characterize the complexity of a function.

7.3.2. Weight Initialization Using Parametric Estimation

In [212], a three-layer MLP with prototypes is initialized by clustering the normalized augmented pattern vectors of unit length. In [213], hidden-layer weights are initialized by using clustering and nearest neighbors, and the output-layer weights are initialized by an SVD. In [214], the weights are forward-layerwise-initialized using eigenvectors of the cross-moment matrix based on Stein’s identity, and the weights for the output layer are initialized by generalized linear modeling. In [215,216], the initial hidden-layer weights are calculated in such a way that each hidden node approximates a range of the desired function and the sigmoidal function is replaced by the piecewise-linear approximation. In [217,218], the outputs of the hidden and output neurons are assigned in the nonsaturation region, and the initial weights are estimated using the least squares and linear algebraic methods. In [219], the weight initialization is implemented by using the orthogonal least squares (OLS) method. In [220], two weight initialization techniques that combine random weight initialization with pseudoinverse are given.

An ICA-based weight initialization for the three-layer MLP was derived in [221]. The hidden-layer weights were initialized by extracting the salient feature components from the input data. A weight initialization based on discriminant learning is proposed in [222]. In [223], the maximum covariance initialization method implements a weight initialization procedure similar to the cascade-correlation algorithm [167].

A weight initialization technique based on Taylor series expansion is mathematically well-founded [224]. Based on a Taylor series development of a nonlinear mapping between the input and output of the examples and the sigmoidal function, two weight initialization techniques for the three-layer MLP have been derived from the first- and second-order identifications of the mapping [224].

7.4. Adapting Activation Function

When a node has a large net input, $net$, it is close to a saturation state. In this case, the first-order derivative of the sigmoidal function is very small, leading to very slow weight update. This can be easily solved by adding a bias, say $0.1$, to $\dot{\varphi }\left(net\right)$ [168]. It is also suggested that the designed error function should go to infinity when $\dot{\varphi }\left(net\right)\to 0$ [225] such that a finite nonzero error update is achieved. The flat-spot problem can be effectively solved by defining such an activation function, ${\varphi }_{\mu }\left(net\right)=\mu net+(1-\mu )\varphi \left(net\right)$, with $\mu \in [0,1]$ [226]. When starting with $\mu =1$, every node has a linear activation function, thus the objective function $\mathcal{E}\left(\overrightarrow{w}\right)$ is a polynomial of the weight vector $\overrightarrow{w}$ and has few local minima. BP is used to find a local minimum in $\mathcal{E}$, then $\mu$ is gradually decreased and BP is applied at the same time until $\mu =0$. This is an annealing process, which helps to reach a global or suboptimal minimum.

For the logistic and hyperbolic tangent functions in (4) and (5), the gain $\beta$ is the slope of the activation function. In BP, typically, $\beta =1$. Modified BP with adaptive $\beta$ has improved performance in terms of learning speed and generalization [227,228,229]. According to a theorem given in [230], increasing $\beta$ has the same effect as increasing $\eta$.

A sigmoidal activation function with a wide linear part, called the log-exp function, was derived in [231]. The MLP with such an activation function can learn quickly since the wide nonsaturation region helps to prevent premature saturation. The extended linear part is particularly suitable for implementation. In order to improve BP, a fuzzy system for adapting the gain $\beta$ is given in [230].

7.5. Other Acceleration Techniques

Gradient reuse is a simple strategy for improving the convergence speed [232]. Gradients computed are reused several times until the weight updates do not decrease the objective function. Batch mode is used to generate a more accurate estimate of the true gradient. However, this method is valid only for simple problems [233]. The weight extrapolation technique can accelerate BP by extrapolating each individual weight [234]. This procedure is easy to implement and is activated only a few times during BP iterations.

Inspired by the common PID control algorithm in feedback control, a three-term BP algorithm [235,236] introduces a proportional term to the BP update rule. It can be implemented in both batch and online modes. Compared to BP, it is easy to select learning parameters in three-term BP, and it has a much higher convergence speed and can escape local minima easily.

Successive approximation BP [237] can effectively avoid local minima. Training is performed in $N_{\mathrm{ph}}$ successive BP learning phases, each terminated at a predefined accuracy ${\delta }_i$, $i=1,\dots ,N_{\mathrm{ph}}$. The overall training error is given by $\mathcal{E}<2^{N_{\mathrm{ph}}}{\prod }_{i=1}^{N_{\mathrm{ph}}}{\delta }_i$. The method significantly outperforms BP in terms of convergence speed and generalization.

For gradient descent with delayed updates, the effects of the delay become negligible after a few iterations and the algorithm converges at the same optimal rate as standard gradient descent [238]. This is particularly important for distributed parallel implementations.

Resilient propagation (RProp) [239] is one of the best performing first-order methods, although it is derived from a heuristic. It is a batch-learning algorithm. Each weight is updated according to the sequence of signs of the partial derivatives in each weight $w_i$. RProp is not sensitive to the initial parameters. It converges much faster and has much lower computational complexity than BP, Quickprop [168], and SuperSAB [198]. RProp is comparable to the CG method in terms of performance [240]. It is suitable for hardware implementation and is not susceptible to numerical problems. There are a number of variants of RProp [241].

8. Second-Order Acceleration

The first-order BP learning converges slowly if the error surface is flat in a dimension. Second-order optimization methods theoretically provide significantly faster convergence. The Hessian $\mathbf{H}$ is utilized in second-order methods. The calculation of $\mathbf{H}$ can be implemented based on information from the BP algorithm [151].

Second-order methods can be categorized into matrix-type and vector-type. Matrix-type methods need to store the Hessian and its inverse. However, for MLPs, $\mathbf{H}$ is ill-conditioned [242]. Newton’s methods and the Broyden–Fletcher–Goldfarb–Shanno (BFGS) method [243] are matrix-type methods, and they converge faster than BP typically by two orders of magnitude. The computational complexity is not less than $O\left(P^2\right)$. Vector-type methods need to store a few vectors. Some representative algorithms are limited-memory BFGS [244], one-step secant (OSS) [78,245], CG algorithms [246,247], and scaled CG [248]. They converge faster than BP by typically one order of magnitude. In these algorithms, the Hessian is calculated iteratively, or its structure is implicitly exploited. Line search or trust-region search is implemented.

Second-order methods are implemented in batch mode to maintain the numerical sensitivity of the calculated second-order gradients. Learning parameters are automatically adapted. However, second-order methods become trapped in a local minimum more frequently than the BP algorithm [249].

8.1. Newton’s Methods

To implement Newton’s methods, one has to explicitly compute and store the Hessian [78,250]. As variants of the classical Newton’s method, Newton’s methods are matrix-type algorithms. Some examples are the classical Newton, Gauss–Newton, and Levenberg–Marquardt (LM) methods. They are quadratically convergent.

Classical Newton’s method is obtained by approximating $\mathcal{E}\left(\overrightarrow{w}\right)$ at time $t+1$ into its Taylor’s expansion at time t, and ignoring the third- and higher-order terms. With the approximation, the updating on the weight vector is given by

$$\overrightarrow{w}(t+1)=\overrightarrow{w}\left(t\right)+\overrightarrow{d}\left(t\right),$$

(42)

where

$$\overrightarrow{d}\left(t\right)=-{\mathbf{H}}^{-1}\left(t\right)\overrightarrow{g}\left(t\right)$$

(43)

and the gradient vector is given by $\overrightarrow{g}\left(t\right)={\left.\frac{\partial \mathcal{E}}{\partial \overrightarrow{w}}\right|}_{\overrightarrow{w}=\overrightarrow{w}\left(t\right)}$. For the MLP, the Hessian is nearly singular [251], and thus directly using (43) to find $\overrightarrow{d}\left(t\right)$ is not practical. Instead, we can solve the following set of linear equations for $\overrightarrow{d}\left(t\right)$ by using an SVD or QR decomposition [252]

$$\overrightarrow{g}\left(t\right)=-\mathbf{H}\left(t\right)\overrightarrow{d}\left(t\right).$$

(44)

In the classic Newton method, the computation of $\mathbf{H}\left(t\right)$ needs global information and $O\left(P^3\right)$ floating-point operations are required for computing the search direction.

8.1.1. Gauss–Newton Method

The Gauss–Newton method is obtained by reformulating $\mathcal{E}\left(\overrightarrow{w}\right)$. The updating equations have the same form as (42) and (43), but $\overrightarrow{g}\left(t\right)$ and $\mathbf{H}\left(t\right)$ are, respectively, defined by

$$\begin{array}{ccc}\hfill \overrightarrow{g}\left(t\right)& =& {\mathbf{J}}^T\left(t\right)\overrightarrow{ϵ}\left(t\right),\hfill \end{array}$$

(45)

$$\begin{array}{ccc}\hfill \mathbf{H}\left(t\right)& =& {\mathbf{H}}_{\mathrm{GN}}\left(t\right)={\mathbf{J}}^T\left(t\right)\mathbf{J}\left(t\right),\hfill \end{array}$$

(46)

where $\overrightarrow{ϵ}\left(t\right)=\overrightarrow{ϵ}\left(\overrightarrow{w}\left(t\right)\right)={\left({ϵ}_1,{ϵ}_2,\cdots ,{ϵ}_N\right)}^T$, ${ϵ}_p=∥{\overrightarrow{e}}_p∥$ with ${\overrightarrow{e}}_p={\overrightarrow{o}}_p-{\overrightarrow{y}}_p$, the Jacobian matrix $\mathbf{J}\left(t\right)=\mathbf{J}\left(\overrightarrow{w}\left(t\right)\right)=\frac{\partial \overrightarrow{ϵ}\left(\overrightarrow{w}\right)}{\partial \overrightarrow{w}}=\left[J_{ij}\right]$, and $J_{ij}=\frac{\partial {ϵ}_i}{\partial w_j}$. The Hessian $\mathbf{H}\left(t\right)$ is approximated using first-order derivatives only. It is possible that $\mathbf{J}$ is ill-conditioned and $\mathbf{H}$ is noninvertible. In such a case, we can use the iterative Gauss–Newton method [243] based on the generalized secant method.

8.1.2. Levenberg–Marquardt Method

The LM method [253] adds a small identity matrix to $\mathbf{H}$ to eliminate the possible singularity. The quadratic approximation error to $\mathcal{E}\left(\overrightarrow{w}\right)$ is minimized at step t under the constraint that the step length $\parallel \overrightarrow{d}\left(t\right)\parallel$ is within a trust region, by using the Karush–Kuhn–Tucker (KKT) theorem [243]. That means, we use a modified Hessian, given by

$${\mathbf{H}}_{\mathrm{LM}}\left(t\right)=\mathbf{H}\left(t\right)+\sigma \left(t\right)\mathbf{I},$$

(47)

where $\sigma \left(t\right)>0$ is a small positive number, which determines the size of the trust region. The LM method, as a modification of the Gauss–Newton method, is obtained by replacing ${\mathbf{H}}_{\mathrm{GN}}$ by ${\mathbf{H}}_{\mathrm{LM}}$ [78,254]. Thus, ${\mathbf{H}}_{\mathrm{LM}}$ is always invertible.

For large $\sigma$, the LM method becomes the BP with $\eta =\frac{1}{\sigma }$; on the other hand, for small $\sigma$, it degenerates to the Gauss–Newton method. Thus, one has to balance between the guaranteed convergence of the gradient descent and the rapid convergence of the classical Newton method. In [254], an adaptive rule on $\sigma$ was developed, given by

$$\sigma \left(t\right)=\left\{\begin{array}{cc}\sigma (t-1)\gamma ,\hfill & \mathrm{when}\mathcal{E}\left(t\right)\ge \mathcal{E}(t-1)\hfill \\ \frac{\sigma (t-1)}{\gamma },\hfill & \mathrm{when}\mathcal{E}\left(t\right)<\mathcal{E}(t-1)\hfill \end{array}\right.,$$

(48)

where a constant factor was applied, $\gamma >1$. A typical selection is $0.01$ for $\sigma \left(0\right)$ and 10 for $\gamma$ [254]. The Jacobian $\mathbf{J}$ can be calculated by a modification to BP [254]. $\sigma \left(t\right)$ can also be adapted according to the hook step [253], Powell’s dogleg method [243], and other heuristics [243].

The LM method is an efficient algorithm for medium-sized neural networks [254]. It requires a large space to store the Jacobian, the approximated Hessian, and the inverse of a $P\times P$ matrix at each iteration. Some LM variants, such as the trust-region-based error aggregated training (TREAT) algorithm [255] and a modified LM method given in [256], aim at reducing the storage and computational complexity. The recursive LM given in [257] is used for online training of neural networks. In [258], the LM method is tailored for an arbitrary network architecture.

In [259], the quasi-Hessian matrix and gradient vector are computed directly, and the Jacobian is not used. The computation of the quasi-Hessian matrix and gradient vector reduces the memory requirements by a factor of $N{J}_M$, where $J_M$ is the number of output nodes. In [260], a forward-only LM method utilizes the forward-only computation, while in traditional methods, the Jacobian is calculated by a forward and backward computation. In [261], backpropagation is used for the matrix–matrix multiplication, reducing the computation time of the LM method by a factor of $O\left(J_M\right)$. In a modified ML algorithm [262], a singularity point in the learning rate is eliminated and there is only one learning rate. The error stability and weights boundedness are assured by the Lyapunov technique.

8.1.3. Other Methods

When implementing Newton’s methods for MLP learning, $\mathbf{H}$ is difficult to calculate iteratively, and thus the cost for calculating ${\mathbf{H}}^{-1}$ is also high. For Newton’s methods, the diagonal terms of $\mathbf{H}$ also have the ill-representability problem, and a good initial estimate of the weights is also required. In block Hessian-based Newton’s method [251], a block Hessian ${\mathbf{H}}_{\mathrm{b}}$ is defined to approximate $\mathbf{H}$.

In the LM case, the calculation of the inverse of ${\mathbf{H}}_{\mathrm{b}}+\lambda \mathbf{I}$ can be broken into the calculation of the inverse of each diagonal block ${\mathbf{H}}_{\mathrm{b}}^{\left(m\right)}+\lambda \mathbf{I}$; thus, the problem is decomposed into $M-1$ subproblems, each corresponding to a layer. The inverse in each of the subproblems can be recursively calculated by the matrix inversion lemma.

LM with adaptive momentum (LMAM) and optimized LMAM [263] have the merits of both LM and CG methods, and thus they can help LM escape local minima. LMAM is globally convergent. Both LMAM and optimized LMAM outperform LM, BFGS, and Polak–Ribiere CG with restarts [246].

The attractor-based trust-region method [264] alternates its two phases for MLP learning. The first phase implements a trust-region-based local search for rapid training and global convergence [243], whereas the second phase implements an attractor-based global search to escape from local minima by using a quotient gradient system [265]. Alternating the two phases results in a fast convergence to global optimum. This algorithm outperforms BPM, BP with tunneling [266], and LM algorithms in terms of number of epochs, training time, and MSE [264].

Stochastic second-order methods are attractive due to their low computational cost in each iteration. However, the performance is highly dependent on the approximation of the Hessian. In [267], an accelerated regularized subsampled Newton method is proposed based on Nesterov’s acceleration.

8.2. Quasi-Newton Methods

In quasi-Newton methods, the Hessian or its inverse is approximated iteratively. The Hessian of an MLP is symmetric and usually positive definite. Quasi-Newton methods having positive-definite Hessian are referred to as variable-metric methods. As a class of variable-metric methods, secant methods approximate the Hessian using differences. Quasi-Newton methods have the same storage requirement as Newton’s methods.

Two globally convergent strategies, namely, line search and trust-region search, are employed in quasi-Newton methods. These strategies retain the rapid convergence property of the classical Newton’s method and are generally applicable [243]. In the line-search strategy, given the direction $\overrightarrow{d}\left(t\right)$ from (43) or (44) based on the approximated Hessian, the updating is given by

$$\begin{array}{ccc}\hfill \lambda \left(t\right)& =& arg\underset{\lambda \ge 0}{min}\mathcal{E}\left(\overrightarrow{w}\left(t\right)+\lambda \overrightarrow{d}\left(t\right)\right),\hfill \end{array}$$

(49)

$$\begin{array}{ccc}\hfill \overrightarrow{w}(t+1)& =& \overrightarrow{w}\left(t\right)+\lambda \left(t\right)\overrightarrow{d}\left(t\right).\hfill \end{array}$$

(50)

The line-search strategy guarantees that the objective function decays at each iteration. The optimal $\lambda \left(t\right)$ can be derived as a representation using the Hessian. Inexact line-search and line-search-free methods are also implemented in quasi-Newton methods for MLP learning [268].

Secant methods can be of rank one or rank two. The Broyden family includes many rank-two and rank-one methods [243]. The Davidon–Fletcher–Powell (DFP) and BFGS methods are dual rank-two secant methods in the Broyden family. BFGS has become the most popular variable-metric method [269]. Many properties of DFP and BFGS are common to the Broyden family.

8.2.1. BFGS Method

In BFGS [243,244,269], the Hessian or its inverse is updated iteratively. The inverse of the Hessian is updated by

$$\begin{array}{ccc}\hfill {\mathbf{H}}^{-1}(t+1)& =& {\mathbf{H}}^{-1}\left(t\right)+\left(1+\frac{{\overrightarrow{z}}^T\left(t\right){\mathbf{H}}^{-1}\left(t\right)\overrightarrow{z}\left(t\right)}{{\overrightarrow{s}}^T\left(t\right)\overrightarrow{z}\left(t\right)}\right)\frac{\overrightarrow{s}\left(t\right){\overrightarrow{s}}^T\left(t\right)}{{\overrightarrow{s}}^T\left(t\right)\overrightarrow{z}\left(t\right)}\hfill \\ & & -\left(\frac{\overrightarrow{s}\left(t\right){\overrightarrow{z}}^T\left(t\right){\mathbf{H}}^{-1}\left(t\right)+{\mathbf{H}}^{-1}\left(t\right)\overrightarrow{z}\left(t\right){\overrightarrow{s}}^T\left(t\right)}{{\overrightarrow{s}}^T\left(t\right)\overrightarrow{z}\left(t\right)}\right),\hfill \end{array}$$

(51)

where

$$\begin{array}{ccc}\hfill \overrightarrow{z}\left(t\right)& =& \overrightarrow{g}(t+1)-\overrightarrow{g}\left(t\right),\hfill \end{array}$$

(52)

$$\begin{array}{ccc}\hfill \overrightarrow{s}\left(t\right)& =& \overrightarrow{w}(t+1)-\overrightarrow{w}\left(t\right).\hfill \end{array}$$

(53)

The initial values $\overrightarrow{w}\left(0\right)$, $\overrightarrow{g}\left(0\right)$, and ${\mathbf{H}}^{-1}\left(0\right)$ need to specified. Typically, ${\mathbf{H}}^{-1}\left(0\right)$ is set as the identity matrix. BFGS has a computational complexity of $O\left(P^2\right)$, and it needs to store matrix ${\mathbf{H}}^{-1}$ as well.

The BFGS as well as all the secant methods satisfies the quasi-Newton condition or secant relation [243,269], given by

$${\mathbf{H}}^{-1}(t+1)\overrightarrow{z}\left(t\right)=\overrightarrow{s}\left(t\right).$$

(54)

When using an inexact line search in BFGS, the number of error function evaluations will decrease substantially.

8.2.2. One-Step Secant Method

OSS method [78,245] is a memoryless BFGS.

It is derived by inserting ${\mathbf{H}}^{-1}\left(t\right)=\mathbf{I}$, I being the identity matrix, into the BFGS update equation given by (51), and then multiplying both sides by $-\overrightarrow{g}(t+1)$ to get the search direction. As such, the Hessian is not stored, and the calculation of a matrix inverse is not needed when calculating a new search direction. OSS has a superb computational complexity of $O\left(P\right)$. The price paid is a considerable loss of second-order information, leading to a slow convergence compared to the BFGS.

When implementing an exact line search, OSS generates conjugate directions. Parallel implementations of BFGS and OSS are treated in [270]. In [271], a parallel implementation of a secant method using inexact search is developed for MLP learning.

8.2.3. Other Secant Methods

In limited-memory BFGS methods, second-order information from the most recent iterations are exploited to implement part of the Hessian approximation [244]. Limited-memory BFGS algorithms typically have a memory complexity of $O\left(P\right)$ and do not need accurate line searches [272]. Other examples of quasi-Newton variants are variable-memory BFGS [273], memory-optimal BFGS methods [274], and the trust-region implementation of the BFGS method [275].

In [276], the proposed limited-memory quasi-Newton methods exploit an iterative scheme of a generalized BFGS type method, and approximate the Hessian by using a rank-two formula arising from a fast unitary transform. They have a computational complexity of $O\left(Plog\left(P\right)\right)$ and a memory complexity of $O\left(P\right)$. BFGS and CG have a close relationship, which helps to formulate algorithms with variable or limited memory [269]. One can treat memoryless or limited-memory quasi-Newton methods as a tradeoff between the CG and quasi-Newton methods.

8.3. Conjugate-Gradient Methods

The CG method [78,248,277,278] is a popular alternative to BP. CG implements a series of line searches along noninterfering directions that are built to utilize the Hessian structure but not store it. The method constructs a sequence of P successive search directions that satisfy the so-called $\mathbf{H}$-conjugate property. The memory requirement is $O\left(P\right)$, which is nearly four times that of BP [234]. The computational complexity per weight update cycle is much higher than that of BP. This is because line search with an appropriate step size requires multiple evaluations of $\mathcal{E}$ or its derivative, which require the presentation of the entire training set.

In the CG method for MLPs, the update is given by

$$\begin{array}{ccc}\hfill \overrightarrow{w}(t+1)& =& \overrightarrow{w}\left(t\right)+\lambda \left(t\right)\overrightarrow{d}\left(t\right),\hfill \end{array}$$

(55)

$$\begin{array}{ccc}\hfill \overrightarrow{d}(t+1)& =& -\overrightarrow{g}(t+1)+\beta \left(t\right)\overrightarrow{d}\left(t\right),\hfill \end{array}$$

(56)

where $\lambda \left(t\right)$ is the exact step size to the minimum of $\mathcal{E}$ along $\overrightarrow{d}\left(t\right)$ and it is obtained by a linear search according to (49), and $\beta \left(t\right)$ is a step size for determining the direction $\overrightarrow{d}(t+1)$.

The speed of CG is critically dependent upon the efficiency of the line search. Fast CG algorithms with inexact line searches [272,279] have been used for MLP learning [280]. Scaled CG [248] avoids the line search by using a scalar to keep the Hessian $\mathbf{H}$ positive definite, as implemented in the LM method.

There are many choices of $\beta \left(t\right)$, such as the Hestenes–Stiefel [277], Fletcher–Reeves [281], Polak–Ribiere [282], Dai–Yuan [283], and conjugate descent [243] methods. The popular Polak–Ribiere method is given by [282]

$$\beta \left(t\right)=\frac{{\overrightarrow{g}}^T(t+1)\overrightarrow{z}\left(t\right)}{{\overrightarrow{g}}^T\left(t\right)\overrightarrow{g}\left(t\right)},$$

(57)

where $\overrightarrow{z}\left(t\right)$ is given by (52). Usually, $\overrightarrow{w}\left(0\right)$ is selected as a random vector, and $\overrightarrow{d}\left(0\right)=-\overrightarrow{g}\left(0\right)$. The process is terminated when $\parallel \overrightarrow{g}\left(t\right)\parallel$ is small enough. The computational complexity of the CG method is $O\left(P\right)$.

The CG method can be treated as an extended version of BPM by an automatic selection of the learning parameters $\eta \left(t\right)$ and $\alpha \left(t\right)$ in each epoch [203,247,278,284]. Compared to BP, the CG method gets more easily stuck at a bad local minimum, since it deterministically moves toward the bottom of any valley it reaches, due to the line-search procedure [234,285]. The CG method is usually executed multiple times with different random $\overrightarrow{w}\left(0\right)$, and finally the $\overrightarrow{w}$ yielding the minimum error is taken as the solution [234].

There are some examples of the CG method in MLP learning. In [286], the MLP is decomposed into a set of Adalines. Each Adaline has a local MSE function. The desired output of each Adaline is calculated by error backpropagation. Each Adaline is learned by using a modified line-search-free CG. The localized method is typically one order of magnitude faster than the CG-based global method. In [287], based on the spectral scaling parameter, a self-scaled CG for MLP learning is derived from the principles of several CG methods.

8.4. Extended Kalman Filtering Methods

The Kalman filtering method is an optimum online estimation method for linear systems. The EKF method is obtained from the linearization of a nonlinear system. EKF is a general-purpose incremental training method, and can be used for training any MLP. Recursive least squares (RLS) can be regarded as a reduced form of EKF, and is more popular in adaptation.

8.4.1. Extended Kalman Filtering

In the EKF approach, learning is treated as the parametric identification problem of a nonlinear system. The weight vector $\overrightarrow{w}$ is regarded as a state vector, and the MLP is regarded as an unknown mapping $\overrightarrow{f}(·)$ from the state vector and the input $\overrightarrow{x}\left(t\right)$ onto the output. The behavior of the MLP is described by the following state equations, given by

$$\begin{array}{c}\hfill \overrightarrow{w}(t+1)=\overrightarrow{w}\left(t\right)+\overrightarrow{\upsilon }\left(t\right),\end{array}$$

(58)

$$\begin{array}{c}\hfill \overrightarrow{y}\left(t\right)=\overrightarrow{f}\left(\overrightarrow{w}\left(t\right),\overrightarrow{x}\left(t\right)\right)+\overrightarrow{ϵ}\left(t\right),\end{array}$$

(59)

where $\overrightarrow{x}\left(t\right)$ is the input, $\overrightarrow{y}\left(t\right)$ the observed or desired output, $\overrightarrow{\upsilon }$ the observation noise, and $\overrightarrow{ϵ}$ the measurement noise. $\overrightarrow{\upsilon }=\mathcal{N}(0,\mathbf{Q}\left(t\right))$ and $\overrightarrow{ϵ}=\mathcal{N}(0,\mathbf{R}\left(t\right))$ are assumed to be Gaussian with zero mean and covariance matrices $\mathbf{Q}\left(t\right)$ and $\mathbf{R}\left(t\right)$, respectively.

EKF estimates the weight vector $\overrightarrow{w}$ such that the sum of the squared prediction errors of all past observations is minimized. It is a minimum variance estimator derived from the Taylor series expansion of $\overrightarrow{f}\left(·\right)$ in the vicinity of the previous estimate. EKF [153,154,288,289,290] is given by

$$\begin{array}{ccc}\hfill \overrightarrow{w}(t+1)& =& \overrightarrow{w}\left(t\right)+\mathbf{K}(t+1)\left[\overrightarrow{y}(t+1)-\overrightarrow{f}\left(\overrightarrow{w}\left(t\right),\overrightarrow{x}(t+1)\right)\right],\hfill \end{array}$$

(60)

$$\begin{array}{ccc}\hfill \mathbf{K}(t+1)& =& \mathbf{P}\left(t\right){\mathbf{F}}^T(t+1){\left[\mathbf{F}(t+1)\mathbf{P}\left(t\right){\mathbf{F}}^T(t+1)+\mathbf{R}(t+1)\right]}^{-1},\hfill \end{array}$$

(61)

$$\begin{array}{ccc}\hfill \mathbf{P}(t+1)& =& \mathbf{P}\left(t\right)-\mathbf{K}(t+1)\mathbf{F}(t+1)\mathbf{P}\left(t\right)+\mathbf{Q}\left(t\right),\hfill \end{array}$$

(62)

where $\mathbf{K}\in {\mathbb{R}}^{P\times N_y}$ is the Kalman gain, $N_y$ the dimension of $\overrightarrow{y}$, $\mathbf{P}\in {\mathbb{R}}^{P\times P}$ the conditional error covariance matrix, $\overrightarrow{f}\left(\overrightarrow{w}\left(t\right),\overrightarrow{x}(t+1)\right)$ the estimated output, and $\mathbf{F}(t+1)={\left.\frac{\partial \overrightarrow{f}}{\partial \overrightarrow{w}}\right|}_{\overrightarrow{w}=\overrightarrow{w}\left(t\right)}\in {\mathbb{R}}^{N_y\times P}$. Usually, $\mathbf{P}\left(0\right)=\frac{1}{\epsilon }\mathbf{I}$, $\overrightarrow{w}\left(0\right)=\mathcal{N}(0,\mathbf{P}\left(0\right))$, and a small number $\epsilon >0$.

The above method is the global EKF method for MLP learning [289,291]. BP is a degenerate form of EKF [288]. Some fast algorithms for MLP learning are fading-memory EKF (FMEKF) and U-D factorization-based FMEKF (UD-FMEKF) [292], and the EKF variant reported in [293]. By partitioning the learning task into many small-scaled, separate, localized identification subproblems, localized EKF methods effectively reduce the complexity of the EKF method. There are some localized EKF methods, such as the multiple extended Kalman algorithm (MEKA) [294] and the decoupled EKF algorithm (DEKF) [295]. For localized algorithms, those off-diagonal terms in $\mathbf{P}$ in the corresponding global EKF method are set to zero to remove coupling.

The Kalman filtering method is based on the assumption of a Gaussian noise. The extended H${}_{\infty }$ filtering (EHF) method [296] extends the EKF method to enhance the robustness to non-Gaussian noise.

8.4.2. Recursive Least Squares

When $\mathbf{R}\left(t\right)$ reduces to the identity matrix $\mathbf{I}$ and $\mathbf{Q}\left(t\right)$ to the zero matrix $\mathbf{O}$, the EKF method reduces to the RLS method. The RLS method has been applied to MLP learning in [152,291,297,298]. It is typically an order of magnitude faster than the first-order online BP. For the same accuracy, RLS requires tenfold less epochs than BP [298].

In [152], RLS is derived from the minimization of the sum of the squared prediction errors of all prior observations plus an additional constraint on weights. For the same initialization condition as that of EKF, RLS is inherently a weight-decay technique, and the weight-decay effect is controlled by $\mathbf{P}\left(0\right)$, typically $\mathbf{P}\left(0\right)=\frac{1}{ϵ}\mathbf{I}$. Usually, for smaller $\epsilon$, the training accuracy is better, whereas for larger $\epsilon$, a better generalization is achieved [152]. Two modified RLS algorithms, namely true weight-decay RLS and input perturbation RLS, were developed to improve the generation ability in [299]. Generalized RLS [300] has a general decay term in the energy function. It has the same computational complexity as RLS. Neural networks trained by generalized RLS have a substantially improved generalization ability.

Like localized EKF, a complex problem can be divided into multiple localized subproblems, each of which is solved by RLS. There exist some localized RLS algorithms, such as the local linearized LS (LLLS) method [301] and the block RLS (BRLS) algorithm [302].

9. Other Learning Algorithms

The expectation–maximization (EM) method [16] is the most popular method for providing the maximum-likelihood (ML) solution for the parameters in case of incomplete data. The EM method has been used for MLP learning [303,304]. The EM approach is an iterative statistical technique. In the E-step, the conditional expectation of hidden nodes is calculated based on the observation and current weight vector. In the M-step, a better weight vector is updated based on the conditional expectation of hidden nodes. The BP algorithm is a special case of the generalized EM algorithm for an iterative ML estimation. Injecting carefully chosen noise can speed the average convergence of the EM method as it climbs a hill of probability or log-likelihood [305]. Noise injection is also applied to bidirectional BP, which trains a neural network with backpropagation in both the backward and forward directions using the same synaptic weights [306].

Amari also used the information geometry to explain the learning in MLPs [304]. In their formulation, the learning was regarded as an alternative projection in the manifold of an exponential family.

In a non-Euclidean parametric space with a Riemannian metric structure, the ordinary gradient is not along the steepest direction of the cost function. In such a case, the natural gradient gives the steepest direction [17]. Natural gradient descent can be viewed as a type of second-order optimization method, with the Fisher information matrix acting as a substitute for the Hessian [307]. In many cases, the Fisher information matrix is equivalent to the generalized Gauss–Newton matrix, which approximates the Hessian [307]. Natural gradient descent has a linear convergence in a Riemannian weight space. Online natural gradient learning yields the Fisher efficient estimator; thus, it converges to the optimal batch mode asymptotically. The flat-spot problem, which occurs in BP, will not occur, when using natural gradient [17]. A natural conjugate gradient method for MLP learning is discussed in [308].

MLPs can be trained by iterative layerwise learning methods [309,310,311,312,313]. A weight update is implemented layerwise, and at each layer the weight is updated by solving a set of linear equations. These algorithms typically converge one to two orders of magnitude faster than BP. The layerwise approach can be jointly used with BP [309,313] or Kalman filtering method [310]. In [314], for a three-layer MLP, the hidden-layer weights are updated by BP, while the output-layer weights are updated by solving a set of linear equations. In [315], the hidden-layer weights are updated by a batch-mode Kalman filtering method, while the output-layer weights are updated by solving a set of linear equations. Compared to LM, this method uses an order of magnitude less time at the same accuracy [316].

The parameter-wise algorithm [317] extends the layerwise approach. At each iteration, the weights can be optimized one by one while the other variables are fixed. Compared with the BPM and layerwise algorithms, the parameter-wise algorithm achieves a faster convergence by a factor of more than an order of magnitude.

Lyapunov’s stability theory has been applied for weight update [318,319,320]. A generalization of BP is developed for training MLPs in [318]. Two adaptive versions of BP proposed in [319] converge much faster than BP and EKF. A gradient-descent method without error backpropagation is proposed for MLP learning in [321], and the method has a great potential for concurrency. A linear programming (LP) method is also used for training MLPs [322]. Fuzzy BP (FBP) algorithm [323] outperforms BP considerably in terms of convergence speed and can escape local minima easily, and QuickFBP [324] is a fast version of FBP. Inspired by the chaotic learning process in the brain, chaotic BP integrates the intrinsic chaos of real neurons into BP, leading to a global search ability [325].

The MLP with hard-limiting activation, known as the binary MLP, is typically used for classification. It has the merits of having extremely simple operations, simple internal representations, an easy hardware implementation, and an easy rule extraction. The binary MLP can be trained based on fuzzy logic and error backpropagation [326,327]. In the fuzzy logic implementation, one needs to design a set of fuzzy rules for updating all the weights. For classification, a critical problem in training the binary MLP is to find big linearly separable subsets [328]. Multicore learning (MCL) has been proposed for constructing binary MLPs, based on some lemmas about linear separability [328]. It simplifies the equations of the weights and biases, producing a smaller hidden layer.

10. Fault-Tolerant Learning

In the early 1980s, people believed that MLPs had a built-in ability against node or weight failures. That is, node or weight failures did not affect the performance of a trained MLP much. However, many results [329,330,331,332,333,334] have shown that if special measures are not considered at the training stage, the network faults could result in a rapid performance degradation. Thus, it is important to have a fault-tolerant MLP. Network faults can be in different forms, e.g., weight noise [329] and open-node fault [335,336].

10.1. Open-Node Fault

In the open-node fault model, some of the hidden nodes are disconnected from the output layer [331]. Put another way, the outputs of those damaged hidden nodes get stuck at zero. Some methods to deal with this fault model have been developed.

Injecting a random node fault [333,337] is a typical heuristic. In this approach, during training, a random node fault is artificially included in the output of hidden nodes. Zhou et al. [336] proposed the T3 algorithm to train a network. In T3, the break point is first identified from the fault curve, and then the network is trained by injecting a random node fault. The objective of T3 is to maximize the number of faulty nodes that an MLP can afford. However, when there are a large number of hidden nodes, we have to search over a huge space of potential faulty networks, leading to excessive training time.

It is well known that the output of a network node is very sensitive to large weights. Another approach is to limit the weight magnitude. One can add the classical weight-decay regularizer into the objective function to limit the weight magnitude [333], or one can hard-bound the weight magnitude to a small value during training [338,339]. Limiting weight magnitude, however, makes the theoretical analysis of the method very difficult. For example, in [333], the selection guide for the regularization parameter is unavailable even though the fault statistics are available. To optimize the performance of MLPs in a faulty setting, a number of trained networks need to be trained by using different weight-decay parameters. Then, for each of the trained network, a huge number of faulty networks are generated to study the performance in the faulty setting by feeding the testing or training set to those faulty networks. Clearly, this approach is computationally intensive.

Instead of modified the training algorithm, the replication technique aims at modifying the trained network by replicating hidden nodes from a trained network [335,340]. This approach, however, needs to use additional source.

Another idea to protect MLPs is to modify the objective functions. Neti et al. [341] defined the training of a fault-tolerant MLP as a mini-max problem, which minimizes the maximum of the training errors over all potential faulty networks with a single node fault. An objective function was defined in [341]. A mini-max problem is, however, very difficult to solve. Some authors [342,343] also proposed to modify the objective function. In their approach, the objective function consisted of two terms: an MSE of a fault-free MLP and the sum of MSEs of the faulty networks. In [344], a similar objective function was defined and the corresponding learning algorithm was proposed. The method could improve generalization and fault tolerance [344]. The above techniques, which modify the objective function, are just effective for handling a single node fault. For multinode open-fault situation, the computational cost for handling the second term of the objective function is very high. For example, for an MLP with $J_2$ hidden nodes, for a single node fault, the number of potential faulty networks is equal to $J_2$. For a multinode fault, the number of potential faulty networks becomes ${\sum }_{i=1}^{J_2}\left(\genfrac{}{}{0pt}{}{M}{i}\right)$. Moreover, in [342,343,344], the theoretical guideline to set the weighting factor for MSE terms of the faulty networks was not addressed. In [345], a fault-tolerant regularizer was proposed. The performance of this tolerant regularizer was better than that of Zhou et al.’s approach. However, the result was suitable for RBF networks only [345].

10.2. Multiplicative Weight Noise

Multiplicative weight noise [346,347,348,349,350] refers to a zero-mean symmetrical noise with variance proportional to the weight magnitude. This noise comes from the finite-precision representation of the trained weights in an implementation [329].

There are several studies on the behavior of multiplicative weight noise [196,351,352,353]. The effect of multiplicative weight noise on Adaline and Madaline has been comprehensively analyzed in [351,352]. Choi et al. [196] applied a statistical approach to derive various output sensitivity measures for MLPs. For a binary or bipolar input, the sensitivity to weight perturbation can be accurately computed by using the algorithm proposed in [353].

The output sensitivity is closely related to the performance of a faulty network with multiplicative weight noise. Based on the MSE sensitivity, a fault-tolerance model was proposed for RBF networks in [354], where multiplicative noise was injected in all the parameters. By extending Choi’s results [196], error sensitivity measures were developed for MLPs in [349]. In [355], sensitivity measures were proposed for split-complex-valued MLPs with noise in inputs and weights.

Various fault-tolerant training methods have been developed for multiplicative weight noise on MLPs. The effect of a multiplicative weight noise is amplified by the magnitude of the associated weight. To reduce its effect, the magnitude of the weights can be kept small. In [339], the magnitude of the most harmful weight, defined by a relevance measure, is shrunk at each step of training. In [338], the magnitude of a weight is upper-limited in a modified BP learning algorithm. In [356], a feasible weight range is determined from the average and variance of the weights’ magnitude.

Based on statistics, the effect of multiplicative weight noise can be canceled out when the magnitudes of the input weights of a node are similar. Simon [357] suggested a distributed fault-tolerant training algorithm, in which the training error was minimized subject to an equality constraint on weight magnitudes. Based on the result on MSE sensitivity [354], a fault-tolerant RBF network can be obtained by using a weight-decay regularizer [358]. An explicit regularization method can be used to train a network, such as an MLP [347,349] and an RBF network [350], so that it is able to tolerate multiplicative weight noise. In the explicit regularization method, the objective function is given by

$$\mathcal{J}={\mathcal{J}}_{\mathrm{MSTE}}+\lambda {\mathcal{J}}_{\mathrm{MSES}},$$

(63)

where ${\mathcal{J}}_{\mathrm{MSTE}}$ stands for the mean squared training error, ${\mathcal{J}}_{\mathrm{MSES}}$ for the MSE sensitivity, and $\lambda$ is the regularization parameter. However, the selection rule on $\lambda$ in the regularization term has not been addressed theoretically.

In [359], the authors presented an objective function, which was similar to (63), for training RBF networks to tolerate multiplicative weight noise. The relationship between $\lambda$ and the variance of multiplicative weight noise was proposed. A fault-tolerant training algorithm was derived, and the generalization ability of the trained network was also given.

11. Perceptron in the Deep Learning Era

Hinton proposed the deep belief network in 2006, starting the deep learning era [360]. However, the first deep learning model is known as the convolutional neural network model proposed by LeCun in 1989 [361]. Deep learning and reinforcement learning are indispensable factors to achieve human-level or better AI systems. They both have strong connections with the brain functions.

In this deep learning era, the foundations of all the deep learning methods are laid on the ideas from MLP research. Stochastic gradient descent is the cornerstone of machine learning. Deep neural networks have usually millions of connections and are difficult to scale. For deep learning, the traditional second-order learning methods are not viable due to its superlinear complexity. The error backpropagation is commonly used for the training of deep neural networks.

11.1. Solving the Difficulties

Deep convolutional neural networks [361] are the most popular and important deep learning models due to their excellent performance on many well-known image benchmarks such as ImageNet. Deep MLPs were successfully applied to speech recognition [362]. Deep neural networks are obtained by expanding the traditional neural network models into multiple layers.

Deep neural networks are generally trained in two phases [363]: in the first phase, pretraining one layer at a time by unsupervised learning; in the second phase, fine-tuning the entire network by the ultimate criterion. Layerwise pretraining finds the optimal solution for a layer at a time, by fixing all the weights of the subsequent layers. This is a greedy procedure. A deep network with such unsupervised layerwise pretraining almost always outperforms the cases when there is no pretraining phase [364]. This is explained by the fact that the pretraining phase acts as a regularizer [365] for the supervised optimization problem.

A deep convolutional network [366] utilizes many hierarchical layers of tiled convolutional filters to model receptive fields. The LeNet-5 model [361] is a six-layer MLP model trained with an incremental BP method. It consists of convolutional, pooling, and fully connected layers. Weight sharing and averaging/subsampling are applied, and OBD is employed for the architecture optimization. Other convolutional models are AlexNet, ZFNet, VGGNet, GoogleNet, and ResNet, which have deeper architecture and millions of connections [241].

A deep network can well approximate a target function with high nonlinearity and realize the desired feature representations. However, the high network complexity makes it very difficult to train and easy to overfit. BP or stochastic gradient descent has three difficulties in training a deep network, i.e., overfitting, vanishing gradient, and extreme computational load. Deep convolutional networks have solved all these difficulties. Hinton credited the success to dropout training, ReLU function, the use of GPUs, and techniques for generating more training examples from the existing ones in a talk at NIPS 2012. Dropout is an effective solution to overfitting. It trains some randomly selected nodes at a certain percentage, while the outputs of other nodes are set to be zero to deactivate the nodes. Dynamic regularization of the learning model can also resolve overfitting [367].

A representative solution to the vanishing gradient problem of BP or stochastic gradient descent is the use of the ReLU activation function [368]

$$\sigma \left(x\right)=max(0,x).$$

(64)

The ReLU function is a piecewise linear function, which outputs zero for negative input and retains the positive input. The max operation allows the ReLU function to compute much faster than a sigmoidal function. The ReLU function better transmits the error than a sigmoidal function. It also induces a sparsity in the hidden nodes. The ReLU function has a derivative of one for all active nodes, and has a zero gradient whenever a node is inactive. Some alternatives to the ReLU function are LeakyReLU and the parametric deformable exponential linear unit (PDELU) [369]. PDELU pushes the mean value of activation responses closer to zero, which ensures the steepest descent when training a deep neural network.

11.2. Why Deep Learning Always Achieves Good Results

Standard learning theory suggests that such highly expressive networks should heavily overfit [66] and therefore not generalize at all. Occam’s razor states that simple hypotheses generalize well. Overfitting is a challenge for deep neural networks when the training data are insufficient.

Surprisingly, deep neural networks typically perform the best, with many more parameters than data points. A solid theoretical foundation is still missing. Some attempts have been made to explain this.

In fact, the popular use of the ReLU activation function $\varphi \left(x\right)=max(x,0)$ is used for alleviating the gradient vanishing problem by eliminating the saturation zone, and thus the error propagation over the network is possible. The use of a ReLU activation also substantially reduces the number of functioning nodes, since as long as the weighted sum is negative, its output will be zero. This substantially decreases the total number of parameters and is an inherent regularization process.

To explain the nonoverfitting problem in the overparameterized case, in [370], instead of using the minimal expected value of the square loss, the generalization error is measured by the maximum loss. The exact number of parameters that ensure both zero training error as well as a good generalization error is estimated. A solution of a regularization problem is guaranteed to yield a good training error as well as a good generalization error and estimate the error on test data. In [371], it is proved that gradient descent can find the global minima of the cross-entropy loss for an overparameterized deep fully connected ReLU network for binary classification, trained with a proper random weight initialization, under certain assumptions on the training data.

In [372], random matrix theory is applied to analyze the weight matrices of deep neural networks. The training process implicitly implements self-regularization, even in the absence of explicit regularization, such as dropout or weight norm constraints. For backpropagation-trained deep fully connected networks, noise accumulation is generally bound, and adding additional layers does not worsen the signal-to-noise ratio beyond a limit [373]. Noise accumulation can be suppressed entirely when the slope of the activation function is smaller than unity.

The existence of suboptimal local minima and saddle points in the highly nonconvex loss landscape is important for the performance of trained deep neural networks. This is proved in [374] for deep ReLU networks with a squared loss or cross-entropy loss under reasonable assumptions. In [375], experiments validate that sufficiently deep and wide neural networks are not negatively impacted by suboptimal local minima. Suboptimal local minima, even though degenerated to saddle points, exist for fully connected sigmoid networks. The local minima can be escaped from via a nonincreasing path on the loss curve. This provides a partial explanation for the successful application of deep neural networks.

Deep neural networks generalize remarkably well, indicating a strong inductive bias toward functions with a low generalization error. In [376], $P_{SGD}\left(f\right|\mathcal{S})$, the probability that an overparameterized deep network trained with stochastic gradient descent converges on a function f consistent with a training set $\mathcal{S}$, is empirically calculated. $P_{SGD}\left(f\right|\mathcal{S})$ correlates remarkably well with the Bayesian posterior $P_B\left(f\right|\mathcal{S})$, and $P_B\left(f\right|\mathcal{S})$ is strongly biased towards low-error and low-complexity functions. Thus, a strong inductive bias in the parameter-function map, which determines $P_B\left(f\right|\mathcal{S})$, is the primary explanation for why deep networks generalize so well. $P_B\left(f\right|\mathcal{S})$ is the first-order determinant of $P_{SGD}\left(f\right|\mathcal{S})$.

11.3. Deep versus Shallow

Although most of the recent works advocate a deep architecture, some researchers support a shallow architecture. We here review a few representative papers on their approximation ability as well as performance comparison.

Compared to the three-layer MLP, the four-layer MLP can usually approximate a target with fewer connections, but may introduce more local minima [64,69,377]. Based on a geometrical interpretation of the MLP, a small four-layer MLP can generate better results for a target function with a flat surface located in its domain [69]. The approximation of smooth multivariable functions with an MLP is treated in [378]. For a specified approximation order, explicit results for the necessary number of hidden nodes and its distributions to the hidden layers are given. It turns out that more than two hidden layers are not necessary when minimizing the number of necessary hidden nodes.

A ReLU network is also proved to be universal approximator [379,380]. Explicit estimates on the size of a ReLU neural network are derived for approximating Lipschitz functions to any given accuracy in the $L^{\infty }$-norm [380]. In [381], a deep ReLU neural network is constructed that approximates a function with a specified accuracy, and tight dimension-dependent bounds on the computational complexity are given in terms of the size and depth of this network. For deep neural networks with certain smooth functions including sigmoidal functions and ReLU, the approximation error is proved to decay exponentially in the number of nonzero weights [382]. When approximating sufficiently smooth functions, finite-width deep networks require a strictly smaller connectivity than finite-depth wide networks [382]. Deep convolutional neural networks with general convolution, activation, and pooling operators is analyzed in [383]. The translation-invariance of the features in the resulting feature extractor is proved to become progressively more prominent as the network depth increases.

The capacity of layered, fully connected architectures of linear threshold neurons is investigated in [384]. In general, under the same other conditions, shallow networks compute more functions than deep networks, and the functions computed by deep networks are more regular. According to an abstract theorem in function approximation [385], without the requirement of robust parameter selection, deep networks using a nonsmooth activation function such as ReLU, do not provide any significant advantage over shallow networks in terms of the degree of approximation alone.

Bounds on the approximation error with respect to the size of the tanh neural network are derived in [386]. Tanh neural networks with only two hidden layers suffice to approximate functions at comparable or better rates than much deeper ReLU neural networks [386].

Training deep neural networks with error backpropagation is implausible from a biological perspective. In [387], classification is implemented in a three-layer network with a single readout layer by using biologically plausible, local learning rules. The hidden layer weights are either fixed or trained with unsupervised, local learning rules, and the readout layer is trained with a supervised, local learning rule. Networks with localized receptive fields perform significantly better than fully connected networks and can reach the backpropagation performance on MNIST. A three-layer spiking model with leaky integrate-and-fire neurons and spike-timing-dependent plasticity for training the readout layer achieves $>$98.2% test accuracy on MNIST, which is close to the performance of three-layer rate networks trained with backpropagation. The performance of the shallow spiking network models is comparable to most biologically plausible models of deep learning.

12. Discussion and Conclusions

Before we conclude this paper, we give an example to demonstrate the performance of some popular methods described in this paper.

12.1. An Example—Iris Classification

The iris database from the UCI Repository of machine learning databases was used as a benchmark for evaluating MLP learning algorithms. The iris database contains 150 patterns, which belongs to three classes. Each pattern has four numeric properties. We divided the dataset into a training set ($80\%$) and a testing set ($20\%$). The performance goal was set as $0.001$, and the maximum number of epochs was 1000. Eight popular learning algorithms (RProp, BFGS, one-step secant, LM, scaled CG, CG with Powell–Beale restarts, Fletcher–Powell CG, and Polak–Ribiere CG algorithms) were implemented and compared.

A three-layer (4-4-3) MLP network was selected. When training the network, if an example belonged to class i, the ith output node outputted 1, while all the other output nodes outputted $-1$. The logistic sigmoidal function was used in the hidden layer, and the linear function was used in the output layer. When evaluating the network, the output node with the largest output was set to 1, while the outputs at all the other nodes were set $-1$.

Table 1. A 4-4-3 MLP trained with eight algorithms: performance obtained by averaging 50 random runs. RP—Rprop, SCG—scaled CG, CGB—CG with Powell–Beale restarts, CGF—Fletcher–Powell CG, and CGP—Polak–Ribiere CG.

Algorithm	Mean Epochs	Training MSE	Classifier Accuracy (%)	Std	Mean Training Time (s)	Std (s)
RP	$990.94$	$0.025$	$96.00$	$0.034$	$0.7372$	$0.2262$
LM	$238.54$	$0.007$	$100.00$	$0.000$	$0.3186$	$0.3494$
BFGS	$154.72$	$0.015$	$93.33$	$0.000$	$0.2619$	$0.1778$
OSS	$999.94$	$0.027$	$96.53$	$0.034$	$1.1087$	$0.0555$
SCG	$903.16$	$0.016$	$95.40$	$0.037$	$0.7462$	$0.1720$
CGB	$439.48$	$0.028$	$95.27$	$0.064$	$0.5456$	$0.3567$
CGF	$562.64$	$0.021$	$95.87$	$0.038$	$0.6820$	$0.3278$
CGP	$573.38$	$0.021$	$96.27$	$0.033$	$0.6962$	$0.3565$

lists the training results for 50 independent runs. Figure 4 shows the learning curves for a random run. In this example, BFGS and LM generated the best MSE performance: they used less time for convergence, but as second-order algorithms, they used more memory. All of the algorithms generated good MSE and classification performances. The simulations were performed on a PC with Intel Core i7 CPU 10700 @2.90 GHz and 16 GB RAM based on the MATLAB Neural Networks Toolbox.

12.2. Discussion on the Popular Learning Algorithms

The above iris classification example compared the performance of eight popular MLP learning algorithms. The first-order algorithms, namely, BP and BP with momentum, whether in batch or in online mode, were not compared here, since they are usually slow in convergence, get easily trapped at bad local minima, and it is difficult to select the learning rate and/or the momentum factor. RProp, as the best performing first-order algorithm, was used here for comparison.

Among the eight methods, RProp is a first-order method, LM is a Newton method, BFGS and OSS are quasi-Newton methods, and the other four are CG methods. It is seen that all the algorithms generated good performance. The second-order methods LM and BGFS gave the best performance in terms of convergence speed and total time, but at a cost of higher computational and memory complexities at each epoch. OSS is a memoryless BFGS, and it trades accuracy for lower complexity. The CG algorithms had a performance below that of LM and BFGS but were better than OSS. RProp had a performance comparable to that of several CG algorithms, and this was in agreement with the conclusion given in [240]. The computational complexity of LM is $O\left(N_w^3\right)$ and its memory complexity is $O(N{J}_M+N_w^2)$. The computational and memory complexities for BFGS are, respectively, $O\left(N_w^2\right)$ and $\frac{1}{2}+O\left(N_w\right)$. The computational complexity for OSS is $O\left(N_w\right)$ and it does not need to store the Hessian. The CG methods are closely related to the quasi-Newton method but conduct a series of line searches along noninterfering directions. They have a memory complexity of $O\left(N_w\right)$, which is about four times that of BP. It is well known that BP has linear time and memory complexities. Compared to BP, the computational complexity of CG is significantly increased for each weight update cycle due to the line searches for an appropriate step size. Like BP, CG is also easily trapped at a bad local minimum. CG converges as fast as RLS, but has a lower complexity.

From the complexity and convergence analysis, we suggest that LM be used for medium sized networks, when the number of weights is less than one thousand. BFGS can be used in networks of up to 5000 weights. The CG algorithms can be applied to even larger networks. OSS and RProp are effective heuristic-based methods, and they are not recommended for large networks, since their performance is not theoretically justified. However, for deep learning, with hundreds of thousands or even millions of parameters to adapt, the first-order gradient descent method is the only choice, due to its low computational and memory complexities as well as its proved convergence.

12.3. Summary

The perceptron model was invented by Rosenblatt in 1958, but it is dated to 1943 when McCulloch and Pitts proposed the neuron model. The perceptron model is the most fundamental and also most used neural network model. There have been numerous research publications during the past seven decades. The number of publications on the fundamental theory of the perceptron model and its learning is several thousands, while that on the applications of the perceptron model to various fields is hundreds of thousands. The perceptron model as well as the entire neural network discipline has become a branch of applied mathematics.

Although the perceptron model is so important for the neural network discipline, there are very few comprehensive survey papers in the literature. This paper was motivated to fill the gap. In this paper, we provided a comprehensive, yet state-of-the-art review on the perceptron model, with importance attached to the MLP model and its learning. More specifically, we treated some of the most important topics that are associated with the perceptron model, namely, SLP and its learning, various learning techniques for MLP, including the first-order BP and accelerating techniques for BP, second-order methods, the optimization of network architectures, and fault-tolerant learning. The role of MLP in the deep learning era was also dwelled on, and some important topics on deep learning were also described.

This paper was also meant to serve as a tutorial on the perceptron model. Therefore, we provided details of the equations for the important algorithms that the readers can easily master. We provided an illustrative iris classification example that compared eight most popular MLP learning algorithms, on the basis of which we compared the algorithms and gave our advice on how to select an MLP learning algorithm for a specific application.

Although the perceptron model has become a fundamental method in this deep learning era, it is still a live model. Most popular deep learning models are based on the perceptron model or have a perceptron model as their component layers. We introduced how the difficulties of deep learning were addressed by the research accomplishments of the perceptron model. In recent years, for big data science, many research efforts on first-order learning such as stochastic gradient descent and stochastic coordinate descent, and even on stochastic second-order algorithms, have been conducted on the perceptron model, see Section 4.6. These algorithms are particularly useful for training deep neural networks.

Many deep learning models have been proposed so far. These models are mostly pretrained by using numerous datasets especially ImageNet, and many other image datasets. Most of the models are also based on convolutional neural network models, which use convolution as the major feature extraction operation. Convolution is an effective operation for physical signals and images. The trained models are then used in numerous specific applications by transfer learning. However, transfer learning is based on analogical reasoning. It is only reasonable for applications with different but related tasks. Thus, transferring the learned deep neural network models to non-image-based classification is theoretically unfounded. Theoretical investigations for transfer learning across different domains are needed for the future of deep learning.

While convolutional-neural-network-based deep learning models may be very effective for classification of images, image-based classification is only an important portion of pattern recognition. In most situations where classification and nonlinear regression problems are under consideration [10,11], the inputs are not physical signals and cannot be organized in a rectangular form, and the use of convolutional-neural-network-based deep learning models for handling those problems is not theoretically justified. For example, in DDoS attack detection problems, the commonly used input features are source IP address, prefix of source IP, flow duration, flow number per time interval, packet size, packet number per flow, unique IP address per time interval, etc. These features are not spatial data and are not in an array form. For a major portion of classification problems, the MLP model or MLP-based deep learning models are preferred to deep convolutional neural networks.

Due to the limitation on the length, many topics that are closely associated with perceptron could not be treated in this paper. For example, various application fields of the perceptron model were not discussed in this paper.

Some other topics related to perceptron are mentioned below. A Sigma-Pi network [6] is a generalization of an MLP by using product nodes and summation nodes to build higher-order terms. It is also a universal approximator [67], and provides inherently higher mapping capabilities than first-order models such as MLPs. It can be trained by the BP learning rule. However, for sigma-Pi network, the numbers of product terms and weights have a combinatorial increase.

The perceptron model has been extended to the complex domain. The complex perceptron learning algorithm [388], whose weights take complex values, has a better separating power than the perceptron learning. The convergence of split complex BP [389] for training complex-valued neural networks has been proved. Complex-valued gradient descent algorithm with an adaptive complex-valued step size has been proposed for the training of complex-valued neural networks [390].

In addition to the analysis-based methods described so far, there are numerous MLP learning methods based on metaheuristics [391]. Those methods were not described here. Hardware and parallel algorithm implementations were also not described in this paper.

Neural network models are traditionally treated as black-box models for modeling unknown systems. The knowledge for human experts cannot be easily integrated into the learning process and human cannot easily understand the function of a trained network. There are some attempts to extract rules from trained three-layer MLPs [231]. One can refer to [241] for a recent progress in perceptron and machine learning.