Evidence-Based Regularization for Neural Networks

Abstract

Numerous approaches address over-fitting in neural networks: by imposing a penalty on the parameters of the network (L1, L2, etc.); by changing the network stochastically (drop-out, Gaussian noise, etc.); or by transforming the input data (batch normalization, etc.). In contrast, we aim to ensure that a minimum amount of supporting evidence is present when fitting the model parameters to the training data. This, at the single neuron level, is equivalent to ensuring that both sides of the separating hyperplane (for a standard artificial neuron) have a minimum number of data points, noting that these points need not belong to the same class for the inner layers. We firstly benchmark the results of this approach on the standard Fashion-MINST dataset, comparing it to various regularization techniques. Interestingly, we note that by nudging each neuron to divide, at least in part, its input data, the resulting networks make use of each neuron, avoiding a hyperplane completely on one side of its input data (which is equivalent to a constant into the next layers). To illustrate this point, we study the prevalence of saturated nodes throughout training, showing that neurons are activated more frequently and earlier in training when using this regularization approach. A direct consequence of the improved neuron activation is that deep networks are now easier to train. This is crucially important when the network topology is not known a priori and fitting often remains stuck in a suboptimal local minima. We demonstrate this property by training a network of increasing depth (and constant width); most regularization approaches will result in increasingly frequent training failures (over different random seeds), whilst the proposed evidence-based regularization significantly outperforms in its ability to train deep networks.

1. Introduction

Regularization is a supplementary element added to neural networks in order to improve their ability to generalize, but does not necessarily in their performance on training data [1] (Chapter 5) (see [2] for a comprehensive overview of various approaches to regularization). Broadly speaking, there are five categories of regularization:

• Regularization of the training data: This form of regularization involves augmentation of the data fed to the layers of a neural network or the introduction of transformations to expand the observed set of training samples. The most popular methods in this category are batch normalization [3] and layer normalization [4]. A related form of regularization is the addition of randomness to input data or the labels. One of the most popular methods in this category is the addition of Gaussian noise to input data [5,6], label smoothing [7], and dropout [8,9];
• Regularization via the network architecture: The network topology is selected to match certain assumptions regarding the data-generating process. Such regularization limits the search space of models or introduces certain invariances. Popular methods in this category are limiting the size (either breadth or depth) of the network, the number of convolutional or pooling layers [10,11], or the number of skip connections [12,13];
• Regularization term added to the loss function: This is a term that is independent of the targets that changes the loss, and thereby the gradient step, in a certain manner. Common approaches are weight decay by adding the norm of the weights to the loss (typically either L1, L2, or the sum of those) [14] and weight smoothing by adding the norm of gradients [14];
• Regularization via the error function itself: This includes approaches to make optimization robust to class imbalance [15] or multi-task learning [16,17];
• Regularization via the optimization procedure: The final form of regularization is via the optimization procedure itself, or, more precisely, the interplay between loss function, regularization terms, and optimization iterations. The most common forms of regularization in this category are mini-batch optimization [18], early stopping [19], initialization approach [20,21], learning-rate scheduling [22], and specific update rules such as Momentum, RMSProp, or Adam [23].

2. Evidence-Based Regularization

The basic idea underpinning this approach to regularization is simple; for each neuron, we aim to have a minimum amount of evidence to support the fitted hyperplane dividing the data. To illustrate the need for sufficient evidence, Figure 1 presents the same (2-dimensional) inputs to a neuron, with the starting hyperplane as a dotted line randomly set at initialization. The standard back-propagation algorithm would likely result in the split on the left (Figure 1, left). Since this neuron may represent the original data transformed by numerous layers, we may ask if the left hyperplane could be leading to an over-fitting of the data. Conversely, if we add a penalty when the number of observations on each side of the hyperplane falls below a user-defined threshold, the fitted neuron would likely result in the right configuration (Figure 1, right), which is supported by a larger number of points and thus may represent a better generalization of the data.

Mathematically, we have a wide range of choices to penalize a low number of data points on either side, noting that this can depend on the activation function if we are to insert the penalty after activation. For a $y_i=tanh\left(x_i\right)$ activation, where $y_i$ represents the output of the ith training sample’s activation function, the regularization penalty, r, applicable to the entire batch, can be defined as:

$$r=min\left[\left(-\sum ^{i\in B}{𝟙}_{y_i<0}{y}_i\right)-\lambda ,\left(\sum ^{i\in B}{𝟙}_{y_i>0}{y}_i\right)-\lambda ,0\right]$$

(1)

where $\lambda$ is the minimum number of observations we require on each side of the hyperplane and B is the number of samples in the batch. The term ${𝟙}_{\mathcal{A}}$ stands for the indicator function, which returns 1 when the condition $\mathcal{A}$ is true and is zero otherwise). To back-propagate through the min function, we use the standard $softmin$ substitute or directly use the min operator if implementing this in an environment that supports auto-differentiation, such as TensorFlow. To implement the differentiable version of the min operator, use the softmin version: $\sigma \left(z_i\right)=\frac{e^{z_i}}{{\sum }_{j=1}^B{e}^{z_j}}fori=1,2,\cdots ,B$. In our implementation, we set $z_i=\alpha y_i$ with $\alpha =-0.95/\lambda$ to automatically scale the strength of the min function w.r.t. the minimum number of observations, $\lambda$.

One interesting property of this regularization is that it nudges the dividing hyperplane towards the neuron’s data if it finds itself outside at initialization. This property is key to ensuring that each neuron is actively used in the network and, as we will show in Section 3.1 and Section 3.2, in aiding the training of deep networks that may be prone (fully or in part) to neurons that are saturated and thus have near-zero gradient.

Importantly, this approach to normalization goes across a mini-batch and is not applicable to single-sample updates. Moreover, the minimum number of observations, $\lambda$, which is an intuitive parameter to set, needs to be viewed with respect to the batch size; as an example, a $\lambda =1$ may be quite appropriate for an image recognition task with a batch size of 1000 (as the process may not be overly noisy), whilst a financial data application may require $\lambda =10$ or more for the same batch size. We use a default of $\lambda =0.10B/c$, where B is the batch size and c is the number of categories or $c=1$ for regression problems (noting that for categorization problems that have significantly imbalanced classes, $B/c$ is set to the number of samples in the batch that belongs to the category with the least representation. Other activation functions can be derived in a similar way to Equation (1) when the output is bound. For unbounded activation, such as ReLU, we would need to bound the output in some way. Interestingly, if we consider a radial basis function activation defined for a centroid c over d input dimensions as $y=e^{-\sum {(x_d-c_d)}^2}$, this regularization ensures that the sum of activated samples is greater or equal to $\lambda$—in essence, a minimum number of neighbors if we think of this neuron as a k-nearest neighbor algorithm.

Finally, a note on extrapolation. This regularization is primarily focused on ensuring that there is a minimum amount of evidence when setting each neuron’s hyperplane. When it comes to extrapolation, this approach will not yield a network that can generalize better; if we are concerned with querying samples that may be significantly different from the training inputs, we can employ specialized techniques, such as selective classification [24], specifically designed to address this issue.

3. Empirical Results

Before analyzing the effects of this approach on neuron activation, we explore how it performs as a regularization method. Using the Fashion-MNIST [25] dataset with a $5\%$ label noise [26,27] on a neural network (with a batch size of 1000, 5 hidden layers, 10 hidden units, 50 epochs, and seed 9218 for our data initialization), we test the effectiveness of the evidence-based regularization against standard regularization methods: $L_1$, $L_2$, and $dropout$, as well as against a non-regularized network. Each regularization method is tested on the same network architecture described above. We use the default parameters for all the regularization methods (i.e., the regularization parameter is equal to $0.01$ for both $L_1$ and $L_2$ techniques and the dropout rate is equal to $0.5$ for the $dropout$ method, with 10 threshold for the evidence-based regularization method).

In Figure 2, we can observe the training and validation accuracy results of the evidence-based regularization outperforming the traditional regularization methods.

To analyze how this approach performs as the signal-to-noise ratio decreases (in this case, distorting the inputs), we inject a Gaussian noise component (zero mean; increasing standard deviation) to the normalized input data, using the default parametrization for all the regularization methods. The results in

Table 1. Training and validation accuracy for Fashion-MNIST data with increasing noise for various regularization methods (EBR: evidence-based regularization).

Std. Dev.	0	0.1	0.25	0.5	0.75	1	2	3	4
Training Set
L1	0.80	0.80	0.79	0.76	0.69	0.66	0.32	0.27	0.10
L2	0.86	0.86	0.85	0.82	0.80	0.78	0.68	0.58	0.46
Dropout	0.80	0.80	0.79	0.77	0.75	0.71	0.60	0.48	0.39
EBR	0.89	0.88	0.86	0.84	0.81	0.78	0.68	0.59	0.51
Validation Set
L1	0.80	0.79	0.78	0.75	0.64	0.58	0.24	0.21	0.10
L2	0.84	0.84	0.84	0.83	0.82	0.80	0.70	0.60	0.49
Dropout	0.82	0.82	0.83	0.81	0.80	0.77	0.68	0.58	0.54
EBR	0.85	0.85	0.85	0.85	0.83	0.82	0.74	0.65	0.55

support the use of this approach as a performant regularization method.

3.1. Large Learning Rates

Setting an appropriate learning rate for training is a critical choice to create a robust model. A small learning rate risks overfitting, but a large learning rate risks divergence during training [22]. A common strategy to balance the two is to use cycle learning [28], where the learning rate starts out at a lower limit, then increases to a maximum over a set period of epochs, and finally decreases again to a lower limit several magnitudes below the initial learning rate.

We hypothesize that our proposed regularizer acts as a natural deterrent against rogue learning with high learning rates in the early stages of training. The reason for this is that the additional gradient applied to nodes with a low activation on either side of its hyperplane ensures that the learning algorithm cannot take steps that would result in extreme weights.

To test this hypothesis, we train randomly initialized neural networks we use the same network topology detailed in Section 3) with high and low learning rates on the Fashion-MNIST data. To ensure that there is no contamination from adaptive learning rates, we use vanilla stochastic gradient descent (without momentum). We train the network for 50 epochs with and without regularization and compare the cross-entropy loss (Figure 3), as well as the resulting accuracy (Figure 4). We compare results for a high learning rate of 1.0 and a low learning rate of 0.0001.

The results clearly show that, without regularization, a high learning rate results in erratic improvements of the loss function, while low learning rates result in much better behaved training. However, the accuracy results show that the low learning rate results in very slow progress on the accuracy of the model. Conversely, applying the regularizer with a high learning rate results in smooth learning and rapid progression on accuracy, observable in the early epochs of training.

3.2. Preventing Vanishing and Exploding Gradients

Interestingly, the proposed regularizer ensures that poorly initialized neurons receive an additional nudge during backpropagation. Visually, if the hyperplane of the neuron lies too far away from the data and, therefore, the activation as defined in (A2) is too low, then the gradient of that neuron is increased in the direction that increases activation. A similar issue is encountered in weight initialization. At their core, all approaches to weight initialization have in common that they seek to prevent vanishing or exploding gradients during training. Our regularizer can be seen as a gradient stabilizer, since it ensures that poorly initializer nodes are pushed to a more meaningful configuration. We also suspect that the regularizer prevents vanishing or exploding gradients, as such behavior would indicate that a node does not cut through the data, but rather lies on one side of the data.

We test this by examining the training progress for different levels of variance for the initializer, with and without the EBR. The stabilizing effect of the EBR should allow for higher variances in the initialization and still result in training progress, as the EBR will prevent nodes from remaining dead, i.e., stuck with vanishing or exploding gradients. We use the same network as in other tests and fit the network to the Fashion MNIST data.

Figure 5 presents the resulting cross-entropy loss curves for the training data (left) and validation data (right) at different epochs of training while Figure 6 presents the corresponding accuracy curves. The results show that training of the network with the EBR progresses virtually unchanged for variances as high as five times the stable level. This shows that the EBR has the property of stabilizing training and makes training progress less dependent on the initial state of the network.

3.3. Training Deeper Networks

The improved activation highlighted above has a beneficial impact on our ability to train deep and narrow networks, where the vanishing/exploding gradient can hinder the ability of a network to converge onto an acceptable solution (see [29] for a thorough review of this phenomena). To illustrate this, we explore how the evidence-based regularization technique performs compared to other traditional regularization methods when using a narrow but relatively deep neural network: starting from 5 layers up to a depth of 30, with a width of 10. We perform 100 tests using different seeds for each depth (the 100 different seeds are kept constant across depths and are included in the Appendix B for reproducibility of results), reporting the average validation accuracy at the end of the training period.

We notice from Figure 7 that the evidence-based regularization technique gracefully drops in accuracy as we increase the network depth, with smaller standard errors across the accuracy results. Conversely, other regularization techniques, such as $dropout$, degrade significantly from depth 24 and beyond. See Appendix C for the same experiments run over the Digits MNIST dataset, which shows similar result.

3.4. CIFAR10 Results

The main motivation of the evidence regularizer is to prevent noisy data from rendering nodes insufficiently discriminatory to contribute to the classification task of the network. To examine the effect of the regularizer for noisy data, we utilize both the clean labels of the CIFAR10 data as well as noisy labels obtained from [30]. We use the worst labels as the noisy labels, which have a 40.21% noise rate (percentage of incorrect labels when compared to the clean labels). The noisy labels were created by [30] through human annotation to mimic practical situations where people tasked with annotating data might make mistakes, e.g., due to inattentiveness or lack of skill/knowledge. Complete details of how these labels were created are provided in [30]).

Due to the complexity of the data, we use a much larger network with five hidden dense layers with widths 512, 256, 128, 64, and 32, respectively. Every hidden layer uses the $tanh$ activation function, while the output layer uses $softmax$. Similar to the other tests presented here, we normalize the input data to have a range from −1 to 1, aligned with the value range of the activation function. The networks are trained for 40 epochs using the Adam optimizer and an (initial) learning rate of 0.01.

We compare the unregularized network to a network that uses the evidence regularizer in each hidden layer. The resulting training curves for the cross-entropy loss and the accuracy of the models are shown in Figure 8 for the clean labels and in Figure 9 for noisy labels. Each figure shows the training and test curves, as well as the drop in outcome between training and test data. This drop represents the effect of overfitting on the training data. A successful regularization will move this curve close to zero throughout training, i.e., minimal statistical difference between the data that the network was trained on (training data) and new, unseen data for which the model needs to predict labels (test data). We try three different regularization strengths (1.0, 5.0, and 10.0) to examine how stronger regularization affects results.

For the clean labels, the evidence regularization at unity strength does not strongly affect the learning curves, as results for no regularization (blue line) and EBR with a strength of 1.0 (red line) closely track one another. At stronger regularization, however, overfitting is substantially reduced (left panel), as there is a much smaller gap in results based on training and test statistics. For both the cross-entropy loss and the accuracy, results on training and test data are much closer with stronger regularization. This corroborates the claim that the EBR is a valid regularizer.

As would be expected, overfitting is stronger for noisy labels with the unregularized network or the weakly-regularized network (EBR with strength 1.0): After 40 epochs, the drop in accuracy between training and test data is more than 60 percentage points. With stronger regularization, the accuracy of the test data remains elevated throughout training and there is a much smaller difference between training and test data performance.

4. Concluding Remarks

We propose a simple regularization method based on ensuring that a sufficient amount of evidence supports each neuron’s fitted parameters within an artificial neural network. We show that it performs well compared to the currently available regularization methods. Importantly, we show how this approach nudges each neuron’s hyperplane towards separating its input data, at least to the extent specified by the regularization parameter. This has various beneficial effects: (a) ensuring that each neuron contributes to the overall solution (less saturated neurons with zero gradient); (b) allowing for more latitude in setting the learning rate, since possible large gradient steps that would otherwise bring the separating hyperplane outside of the data will be naturally moved back within the input data; and finally, (c) improving training of deep and narrow neural networks, which would otherwise be at risk of becoming stuck in a suboptimal local minima. An interesting open question remains around how this approach can be accurately normalized w.r.t. the number of inputs to a neuron; each additional input would naturally require a higher number of supporting points (loosely following Occam’s Razor), yet the additional regularization depends both on the number of inputs and the data itself.