A Zeroth-Order Adaptive Learning Rate Method to Reduce Cost of Hyperparameter Tuning for Deep Learning

Abstract

Due to powerful data representation ability, deep learning has dramatically improved the state-of-the-art in many practical applications. However, the utility highly depends on fine-tuning of hyper-parameters, including learning rate, batch size, and network initialization. Although many first-order adaptive methods (e.g., Adam, Adagrad) have been proposed to adjust learning rate based on gradients, they are susceptible to the initial learning rate and network architecture. Therefore, the main challenge of using deep learning in practice is how to reduce the cost of tuning hyper-parameters. To address this, we propose a heuristic zeroth-order learning rate method, Adacomp, which adaptively adjusts the learning rate based only on values of the loss function. The main idea is that Adacomp penalizes large learning rates to ensure the convergence and compensates small learning rates to accelerate the training process. Therefore, Adacomp is robust to the initial learning rate. Extensive experiments, including comparison to six typically adaptive methods (Momentum, Adagrad, RMSprop, Adadelta, Adam, and Adamax) on several benchmark datasets for image classification tasks (MNIST, KMNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100), were conducted. Experimental results show that Adacomp is not only robust to the initial learning rate but also to the network architecture, network initialization, and batch size.

1. Introduction

Deep learning has been highly successful across a variety of applications, including speech recognition, visual object recognition, and object detection [1,2,3]. In general application, deep learning consists of training and inference phases. In the training phase, a predefined network is trained on a given dataset (known as the training set) to learn the underlying distribution characteristics. In the inference phase, the well-trained network is then used for unforeseen data (known as the test set) to implement specific tasks, such as regression and classification. One fundamental purpose of deep learning is to achieve as high accuracy as possible in the inference phase after only learning from the training set. In essence, training a deep learning network is equivalent to minimizing an unconstrained non-convex but smooth function.

$$\begin{array}{c}\hfill \underset{w\in {\mathbb{R}}^n}{min}f\left(w\right):={\mathbb{E}}_{\xi \sim \mathcal{D}}F(w,\xi )\end{array}$$

(1)

where $\mathcal{D}$ is the population distribution, $F(w,\xi )$ is the loss function of sample $\xi$, and $f\left(w\right)$ is the expectation of $F(w,\xi )$ with respect to $\xi$. One effective method to solve problem (1) is mini-batch stochastic gradient descent (SGD) [4]. That is, in each iteration t, the model parameter $w_t$ is updated to $w_{t+1}$, following

$$\begin{array}{c}\hfill w_{t+1}=w_t-{\gamma }_t\frac{1}{|{\mathcal{B}}_t|}\sum _{\xi \in {\mathcal{B}}_t}\nabla F(w_t,{\xi }_i),\end{array}$$

(2)

where ${\gamma }_t$ is the learning rate and ${\mathcal{B}}_t$ is a random mini-batch of size b. Here, based on three advantages discussed below, we consider using problem Equations (1) and (2) to analyze deep learning. First, generality. Because most deep learning networks correspond to non-convex optimization, the derived results for problem (1) can be applied to general deep learning tasks in practice. Second, effectiveness. Because the data scale used in deep learning is usually huge, Equation (2) can achieve a better utility-efficiency tradeoff than SGD or batch GD. Third, simplicity. When using Equation (2), searching for a solution to problem (1) is reduced to setting a proper learning rate ${\gamma }_t$.

Therefore, the research question is how to set a proper ${\gamma }_t$ in Equation (2) to ensure the convergence of problem (1). Without of loss generality, we make the following assumptions.

• Non-convex but smooth $f\left(w\right)$. That is, $f\left(w\right)$ is non-convex but satisfies $f\left(w_{t+1}\right)-f\left(w_t\right)\le \langle \nabla f\left(w_t\right),w_{t+1}-w_t\rangle +\frac{L}{2}{\parallel w_{t+1}-w_t\parallel }^2$.
• Unbiased estimate and bounded variance of $g\left(w\right)\stackrel{▵}{=}\nabla F(w,\xi )$. That is, ${\mathbb{E}}_{\xi \sim \mathcal{D}}\left[g\left(w\right)\right]=\nabla f\left(w\right)$, and ${\mathbb{E}}_{\xi \sim \mathcal{D}}{[\parallel g\left(w\right)-\nabla f\left(w\right)\parallel }^2]\le {\sigma }^2$.

It is well-known that searching for the global minima of Equation (1) is NP-hard [5]. Therefore, one usually aims to search for the first-order stationary point, the gradient of which satisfies $\parallel \nabla f\left(w\right)\parallel <\epsilon$ where $\epsilon$ is a given error bound. For simplicity, denote the average of gradients on mini-batch ${\mathcal{B}}_t$ as ${\overline{g}}_t=\frac{1}{|{\mathcal{B}}_t|}{\sum }_{\xi \in {\mathcal{B}}_t}\nabla F(w_t,{\xi }_i)$. It is well-known that SGD can converge for proper settings of ${\gamma }_t$ [6,7,8,9]. Actually, based on the above assumptions and through direct calculation, we have

$$\begin{array}{c}\hfill \sum _{t=1,\cdots .T}\left({\gamma }_t-\frac{L{\gamma }_t^2}{2}\right)\mathbb{E}{\parallel \nabla f\left(w_t\right)\parallel }^2\le f\left(w_1\right)-f^{*}+\frac{L{\sigma }^2{\sum }_{t=1}^T{\gamma }_t^2}{2b},\end{array}$$

where $f^{*}={min}_{w\in {\mathbb{R}}^n}f\left(w\right)$ and $L,{\sigma }^2$ are defined in the assumptions. Therefore, for any given constant $\gamma =min\{1/L,O(1/\sqrt{L+T})\}$, we can deduce that ${\sum }_{t=1}^T\mathbb{E}\parallel \nabla f\left(w_t\right)\parallel$ is bounded, which implicates that $\parallel \nabla f\left(w_t\right)\parallel$$\to 0$ as $t\to \infty$. That is, SGD is convergent and one can output the stationary solution with high probability [6].

Nevertheless, one challenge of applying these theoretical results in practice comprises the unknown parameters $f\left(w_1\right)$, L and $\sigma$, which are related to network architecture, network initialization, loss function, and data distribution. To avoid computing exact values that are network- and dataset-dependent, there are two common ways to set the learning rate in practice. One way is setting an initial level at the beginning and then adjusting it with a certain schedule [4], such as step, multi-step, exponential decaying, or cosine annealing [10]. However, setting the learning rate typically involves a tuning procedure in which the highest possible learning rate is chosen by hand [11,12]. Besides, there are additional parameters in the schedule that also need to be tuned. To avoid the delicate and skillful parameter tuning, the other way is using adaptive learning rate methods, such as Adagrad [13], RMSprop [14], and Adam [15], in which only the initial learning rate needs to be predefined. However, as shown in our experiments and other studies [16,17,18], they are sensitive to the initial learning rate and each of them has its own effective intervals (refer to Figure 1). Usually, the setting of an initial learning rate is model- and dataset-dependent. This increases the cost of tuning the learning rate and the difficulty of selecting the proper adaptive method in practice.

This motivates us to design an adaptive method, which can reduce the cost of tuning the initial learning rate. Furthermore, the method should achieve a satisfied accuracy no matter what the network architecture and data are. To achieve this, we propose a zeroth-order method, Adacomp, to adaptively tune the learning rate. Unlike existing first-order adaptive methods, which adjust learning rate by exploiting gradients or additional model parameters, Adacomp only uses the values of loss function and is derived from minimizing Equation (1) with respect to ${\gamma }_t$, which has the original expression

$$\begin{array}{c}\hfill \gamma \leftarrow \gamma /2+\frac{max\{0,f\left(w_t\right)-f\left(w_{t+1}\right)\}}{\theta \parallel g\left(w_t\right){\parallel }^2},\end{array}$$

(3)

where $\theta$ is an undetermined parameter and $\theta \parallel g\left(w_t\right){\parallel }^2$ will be further substituted by other explicit factors. Refer to Equation (8) for details. Note that Equation (3) only uses the observable variables $\parallel g\left(w_t\right)\parallel$ and $f\left(w_t\right)-f\left(w_{t+1}\right)$ to adjust $\gamma$. It can be interpreted that when $f\left(w_k\right)-f\left(w_{k+1}\right)$ dominates $(\gamma /2)\parallel g\left(w_t\right){\parallel }^2$, we use an aggressive learning rate to enhance the progress. In contrast, we use an exponential decaying learning rate to ensure convergence. Therefore, in a highly abstract level, $\gamma$ is complementary to loss difference $f\left(t_k\right)-f\left(w_{t+1}\right)$ and we name it Adacomp, which has the following two advantages. Firstly, Adacomp is insensitive to learning rate, batch size, network architecture, and network initialization. Secondly, due to only exploiting values of loss function rather than high-dimensional gradients, Adacomp has high computation efficiency.

In summary, our contributions are as follows.

• We propose a highly computation-efficient adaptive learning rate method, Adacomp, which only uses loss values rather than exploiting gradients as other adaptive methods. Additionally, Adacomp is robust to initial learning rate and other hyper-parameters, such as batch size and network architecture.
• Based on the analysis of Adacomp, we give a new insight into why a diminishing learning rate is necessary when solving Equation (1) and why a gradient clipping strategy can outperform a fixed learning rate.
• We conduct extensive experiments to compare the proposed Adacomp with several first-order adaptive methods on MNIST, KMNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 classification tasks. Additionally, we compare Adacomp with two evolutionary algorithms on MNIST and CIFAR-10 datasets. Experimental results validate that Adacomp is not only robust to initial learning rate and batch size, but also network architecture and initialization, with high computational efficiency.

The remainder is organized as follows. Section 2 introduces related work about typical first-order adaptive methods. Section 3 presents the main idea and formulation of Adacomp. In Section 4, we conduct extensive experiments to validate Adacomp, in terms of robustness to learning rate, network architectures, and other hyperparameters. We conclude the paper and list future plans in Section 5.

2. Related Work

There are many modifications to the gradient descent method, and the most powerful is Newton’s method [19]. However, the requirements of a Hessian matrix and its inverse are prohibitive to compute in practice for large-scale models. Therefore, many first-order iterative methods have been proposed to either exploit gradients or to approximate the inverse of the Hessian matrix.

2.1. Learning Rate Annealing

One simple extension of SGD is the mini-batch SGD, which can reduce the variance by increasing the batch size. However, the proper learning rate is hard to set beforehand. A learning rate that is too small will slow down the convergence, while if it is too large it will cause large oscillation or even divergence. The ordinary method is adjusting the learning rate in the training process, such as by using the simulated annealing algorithm [20,21], or decreasing the learning rate when the loss value is less than a given threshold [22,23]. However, the iteration number and threshold must be predefined. Therefore, the method is not adjustable to different datasets.

2.2. Per-Dimension First-Order Adaptive Methods

In mini-batch SGD, a single global learning rate is set for all dimensions of the parameters, which may not be optimal when training data are sparse and different coordinates vary significantly. A per-dimension learning rate that can compensate for these differences is often advantageous.

Momentum [24,25] is one method of speeding up training per dimension. The main idea is to accelerate progress along dimensions in which gradients are in the same direction and to slow progress elsewhere. This is done by keeping track of past parameters with an exponential decay, $v_t=\rho v_{t-1}+\gamma {\overline{g}}_t,w_{t+1}=w_t-v_t,$ where $\rho$ is an undetermined parameter to control the decay of previous updated parameters. This gives an intuitive improvement over SGD when the cost surface is a long narrow valley.

Adagrad [13,26] adjusts the learning rate according to the accumulated gradients and has shown significant improvement on large-scale tasks in a distributed environment [27]. This method only uses information of gradients with the following update rule: $w_{t+1}=w_t-\frac{\gamma }{\sqrt{{\sum }_{\tau =1}^t{\overline{g}}_{\tau }^2}}·{\overline{g}}_t.$

Here the denominator computes the $L_2$ norm of all previous gradients per-dimension. Since the learning rate for each dimension inversely grows with the gradient magnitudes, the large gradients have a small learning rate and vice versa. This has the nice property, as in second-order methods, that the progress along each dimension evens out over time. However, Adagrad is sensitive to initial model parameters. When some dimensions of gradient are too large, or as gradients accumulate, the learning rate will quickly tend to zero before achieving a good result.

Subsequently, several adaptive methods are proposed to overcome the drawback.

RMSprop [14,28] is a modification of Adagrad, using the root-mean-square (RMS) to replace the denominator in Adagrad, with the following update rule:

$$\begin{array}{c}\hfill \mathbb{E}{\left[{\overline{g}}^2\right]}_t=\rho \mathbb{E}{\left[{\overline{g}}^2\right]}_{t-1}+(1-\rho ){\overline{g}}_t^2,w_{t+1}=w_t-\frac{\gamma }{\sqrt{\mathbb{E}{\left[{\overline{g}}^2\right]}_t}+\epsilon }{\overline{g}}_t.\end{array}$$

This can mitigate the fast decay of learning rate in Adagrad and damp oscillation. The parameter $\rho$ is used to control the decaying speed and $\epsilon$ is a positive small constant to keep the denominator meaningful.

Adadelta [29,30] is another improvement of Adagrad from two aspects. On one hand, Adadelta replaces the denominator in Adagrad by the average of exponential decay of history gradients over a window with some fixed size. On the other hand, Adadelta uses the Hessian approximation to correct units of updates. Adadelta uses the following update rule:

$$\begin{array}{cc}\hfill & \mathbb{E}{\left[{\overline{g}}^2\right]}_t=\rho \mathbb{E}{\left[{\overline{g}}^2\right]}_{t-1}+(1-\rho ){\overline{g}}_t^2,\mathbb{E}{\left[\mathrm{\Delta }{w}^2\right]}_t=\rho \mathbb{E}{\left[\mathrm{\Delta }{w}^2\right]}_{t-1}+(1-\rho )\mathrm{\Delta }{w}_t^2,\hfill \\ \hfill & w_{t+1}=w_t-\frac{\sqrt{\mathbb{E}{\left[\mathrm{\Delta }{w}^2\right]}_t}}{\sqrt{\mathbb{E}{\left[{\overline{g}}^2\right]}_t}+\epsilon }·{\overline{g}}_t.\hfill \end{array}$$

The advantage is no requirement of manually setting the global learning rate. At the beginning and middle phases, Adadelta achieves a good accelerating effect. However, at the late phase, it may oscillate at local minima.

Adam [15,31] can be viewed as a combination of Momentum and RMSprop, with additional bias correction. Adam uses the following update rule:

$$\begin{array}{cc}\hfill m_t& ={\rho }_1{m}_{t-1}+(1-{\rho }_1){\overline{g}}_t,v_t={\rho }_2{v}_{t-1}+(1-{\rho }_2){\overline{g}}_t^2,\hfill \\ \hfill m_t& \leftarrow \frac{m_t}{1-{\rho }_1^t},v_t\leftarrow \frac{v_t}{1-{\rho }_2^t},w_{t+1}=w_t-\gamma \frac{m_t}{\sqrt{v_t}+\epsilon }.\hfill \end{array}$$

Here $m_t,v_t$ are factors of momentum and root-mean-square, and $\frac{m_t}{1-{\rho }_1^t},\frac{v_t}{1-{\rho }_2^t}$ are corresponding bias corrections.

Adamax [15] is an extension of Adam by generalizing the $L_2$ norm to the $L_p$ norm and letting $p\to +\infty$, with the update rule:

$$\begin{array}{c}\hfill m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1){\overline{g}}_t,v_t=max\{{\beta }_2·v_{t-1},\parallel {\overline{g}}_t{\parallel }_{\infty }\},w_{t+1}=w_t-\frac{{\gamma }_t}{(1-{\beta }_1^t)}·\frac{m_t}{v_t}\end{array}$$

Note that the difference between Adamax and Adam is the expression of $v_t$ and there is no bias correction in $v_t$.

However, all these adaptive methods rely on gradients or additional model parameters, which make them sensitive to the initial learning rate and network architectures.

2.3. Hyperparameter Optimization

This aims to find the optimal learning rate values and includes experimental performance analysis [18,32] and Bayesian optimization [33,34] based on mathematical theory. Specifically, [35] combines hyperband and Bayesian optimizations, additionally utilizing the history information of previous explored hyperparameter configurations to improve model utility. Besides, reinforcement learning (RL) and heuristic algorithms are also extensively applied to tune hyperparameters of deep learning. With respect to RL, the focus is how to alleviate the dependence on expert knowledge [16,36,37] or additionally improve the computational efficiency [38,39]. For example, [16] uses RL to learn and adjust the learning rate on each training step and [36] proposes an asynchronous RL algorithm to search the optimal convolutional neural network (CNN) architecture. Ref. [39] reuses the previously successful configuration for reshaping the advantage function and [38] adaptively adjusts the horizon of the model to improve the computational efficiency of RL. With respect to heuristic algorithms, [40] sets an adaptive learning rate for each layer of neural networks by simulating the cross-media propagation mechanism of light in the natural environment. Ref. [41] proposes to use a variable length genetic algorithm to tune the hyperparameters of a CNN. [42] proposes a multi-level particle swarm optimization algorithm for a CNN, where an initial swarm at level-1 optimizes architecture and multiple swarms at level-2 optimize hyperparameters. Ref. [43] combines six optimization methods to improve the performance of forecasting short-term wind speed, where local search techniques are used to optimize the hyperparameters of a bidirectional long short-term memory network. However, these methods suffer unavoidably from more time-consuming space searching.

2.4. Zeroth-Order Adaptive Methods

Like our method, [23,44] propose two methods that adjust the learning rate based on the training loss. In [23], the learning rate at epoch s is set as ${\gamma }_s={\gamma }_0{\mathsf{\Pi }}_{t=1}^{s-1}{r}_{n_t}$, where $r_{n_t}$ is the scale factor. However, the selection of $r_{n_t}$ is through multi-point searching (controlled by beam size), which may be a computation overhead. For example, when the beam size is 4, it has to train the network with 12 different learning rates and to select the optimal one after the current epoch. In [44], the learning rate update is ${\gamma }_{t+1}=(1+\mu ){\gamma }_t$ with $\mu =M·(1-e^{E(t+1)-E\left(t\right)})$, where M is the tracking coefficient and $E(·)$ is the reconstruction error of the WAE (wavelet auto-encoder). Note that the update rule is model restricted and cannot be applied to general cases. For example, when $E(t+1)-E\left(t\right)>0$, ${\gamma }_{t+1}$ may be less than zero and has no meaning. However, the computation overhead of [23] and no meaning of [44] will not occur in Adacomp.

3. Adacomp Method

In the section, we describe our proposed Adacomp through two main steps. First, we deduce the optimal learning rate in each iteration based on theoretical analysis. Then, we design the expression of Adacomp to satisfy several restrictions between learning rate and difference of loss function. For convenience, we summarize the above mentioned and the following used variables in

Table 1. List of variables and explanations.

Variables	Explanations
$T,t$	Number of total iterations and index of the current iteration.
$F(w,\xi ),\nabla F(w,\xi )$	Empirical loss function and gradient at model parameter w and sample $\xi$.
$f\left(w\right),\nabla f\left(w\right)$	Expected function and gradient of $F(w,\xi )$ and $\nabla F(w,\xi )$ with respect to $\xi \sim \mathcal{D}$.
$L,{\sigma }^2$	Smooth constant of $f\left(w\right)$ and variance of $\nabla F(w,\xi )$ with respect to $\xi$.
$g_t,{\overline{g}}_t$	Abbreviations for $\nabla F(w_t,\xi )$ and $\frac{1}{b}{\sum }_{\xi \in {\mathcal{B}}_t}\nabla F(w_t,\xi )$, respectively.
${\mathcal{B}}_t,b$	Mini-batch sampled at t-th iteration with size of b.
${\gamma }_t,\beta$	Learning rate at t-th iteration, parameter of Adacomp used to adjust ${\gamma }_t$.
${\mathrm{\Delta }}_t$	Difference in $f\left(w\right)$ at two consecutive iterations, i.e., $f\left(w_{t-1}\right)-f\left(w_t\right)$.

.

3.1. Idea 1: Search Optimal Learning Rate

For problem (1) in which gradients are Lipschitz continuous, i.e., $f\left(w_{t+1}\right)-f\left(w_t\right)\le \langle \nabla f\left(w_t\right),w_{t+1}-w_t\rangle +\frac{L}{2}{\parallel w_{t+1}-w_t\parallel }^2$, by substituting Equation (2), we have

$$\begin{array}{c}\hfill f\left(w_{t+1}\right)-f\left(w_t\right)\le -{\gamma }_t\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle +\frac{L{\gamma }_t^2}{2}{\parallel {\overline{g}}_t\parallel }^2.\end{array}$$

(4)

To make progress at each iteration t, let the r.h.s. of Equation (4) be less than zero, and we have ${\gamma }_t\le 2\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle /L\parallel {\overline{g}}_t{\parallel }^2$. To greedy search the optimal learning rate, we minimize Equation (4) to obtain

$$\begin{array}{c}\hfill {\gamma }_t=\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle /(L\parallel {\overline{g}}_t{\parallel }^2).\end{array}$$

(5)

This presents an explicit relation between the learning rate and inner product of expected gradient $\nabla f\left(w_t\right)$ and observed gradients ${\overline{g}}_t$. When $\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle \ge 0$, Equation (5) is meaningful (${\gamma }_t\ge 0$) and using this ${\gamma }_t$ can make the largest progress at the current iteration. Elsewhere, we set ${\gamma }_t=0$ when $\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle \le 0$. This means that when ${\overline{g}}_t$ is far away from the correct direction $\nabla f\left(w_t\right)$, we will drop the incorrect direction and do not update the model parameters at the current iteration.

By substituting Equation (5) into Equation (4) and through simple calculation, we obtain Theorem 1, which indicates that $\parallel \nabla f\left(w_t\right)\parallel \to 0$ as $t\to \infty$, i.e., the convergence of nonconvex problem (1).

Theorem 1. 

If the learning rate ${\gamma }_t$ is set as Equation (5), then

$$\begin{array}{c}\hfill \sum _{t=1}^T{\parallel \nabla f\left(w_t\right)\parallel }^2\le 2L{\sigma }^2(f\left(w_1\right)-f^{*})/b\end{array}$$

(6)

where $f^{*}={inf}_{w\in {\mathbb{R}}^n}f\left(w\right)$.

Proof. 

By substituting Equation (5) into Equation (4), we have

$$\begin{array}{c}\hfill f\left(w_{t+1}\right)-f\left(w_t\right)\le -\frac{{\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle }^2}{2L\parallel {\overline{g}}_t{\parallel }^2}.\end{array}$$

Taking expectation with respect to $\xi$ on the current condition $w_t$, we have

$$\begin{array}{c}\hfill \mathbb{E}[f\left(w_{t+1}\right)-f\left(w_t\right)]\stackrel{\left(a\right)}{\le }-\frac{{\langle \nabla f\left(w_t\right),\mathbb{E}{\overline{g}}_t\rangle }^2}{2L\mathbb{E}\parallel {\overline{g}}_t{\parallel }^2}\stackrel{\left(b\right)}{\le }-\frac{\parallel \nabla f\left(w_t\right){\parallel }^4}{2L(\parallel \nabla f\left(w_t\right){\parallel }^2+{\sigma }^2/b)}\stackrel{\left(c\right)}{\le }-\frac{\parallel \nabla f\left(w_t\right){\parallel }^2}{2L{\sigma }^2/b},\end{array}$$

where we use Jensen’s inequality $\mathbb{E}{\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle }^2\ge {\langle \nabla f\left(w_t\right),\mathbb{E}{\overline{g}}_t\rangle }^2$ in (a) and assumption of bounded variance $\mathbb{E}\parallel {\overline{g}}_t{\parallel }^2\le {\parallel \nabla f\left(w_t\right)\parallel }^2+\frac{{\sigma }^2}{b}$ in (b), and the fact $\frac{\parallel \nabla f\left(w_t\right){\parallel }^2}{\parallel \nabla f\left(w_t\right){\parallel }^2+{\sigma }^2}\ge \frac{1}{{\sigma }^2}$ in (c). Taking summation on both sides of the above inequality from $t=1$ to $T-1$ and using the fact that $f\left(w_1\right)-f\left(w_T\right)\le f\left(w_1\right)-f*$ derive the proved result.    □

Some remarks about Theorem 1 are in order.

First, Theorem 1 achieves the optimal convergence rate of smooth nonconvex optimization when using the SGD algorithm. In particular, under a deterministic setting (i.e., $\sigma =0$), Nesterov [45] shows that after running the method for at most $T=O(1/\epsilon )$ steps, one can achieve ${min}_{t=1,\cdots ,T}\parallel \nabla f\left(w_t\right)\parallel \le \epsilon$, where $\epsilon$ is the given error bound. Under a stochastic setting (i.e., $\sigma >0$), the result reduces to $O(1/{\epsilon }^2)$. Ghadimi [6] derives the result $O(1/\epsilon +1/{\epsilon }^2)$ of randomly outputting a solution $w_R$ satisfying $\mathbb{E}\parallel \nabla f\left(w_R\right){\parallel }^2\le \epsilon$ with high probability, where $R\in \left[T\right]$. The optimal rate is improved to $O({\epsilon }^{-7/4}log(1/\epsilon ))$ by using accelerated mirror descent [7]. Here, we obtain that using Equation (6) achieves the optimal convergence rate, i.e., $O\left({\epsilon }^{-2}\right)$, for stochastic optimization when using SGD.

Second, a diminishing learning rate is necessary to ensure convergence, i.e., gradient $\parallel \nabla f\left(w_t\right)\parallel$ will tend to zero as $t\to \infty$. In this case, we have $\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle \to 0$. Additionally, because $\mathbb{E}\parallel {\overline{g}}_t{\parallel }^2={\parallel \nabla f\left(w_t\right)\parallel }^2+{\sigma }^2\to {\sigma }^2$,we deduce that ${\gamma }_t=\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle /(L\parallel {\overline{g}}_t{\parallel }^2)\to 0$ holds almost surely.

Third, gradient clipping is a useful skill to ensure convergence. As proved in [46], using a clipped gradient can converge faster than a fixed learning rate. Here we show an explicit explanation. Based on Equation (5), ${\gamma }_t=min\{\frac{\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle }{L\parallel {\overline{g}}_t\parallel }/\parallel {\overline{g}}_t\parallel ,{\gamma }_t\}$. Therefore, using Equation (5) is equivalent to using threshold $c=\frac{\langle \nabla f\left(w_t\right),{\overline{g}}_t\rangle }{L{\gamma }_t\parallel {\overline{g}}_t\parallel }$ to clip the gradient, which in turn means that clipping the gradient is equivalent to using a varying learning rate to update model parameters. Therefore, for any fixed learning rate ${\gamma }_t\equiv \gamma$, we can adjust it based on Equation (5) to make faster progress. That is, proper gradient clipping can outperform any fixed learning rate update.

3.2. Idea 2: Approximate Unknown Terms

To set learning rate according to Equation (5), two terms, expected gradient $\nabla f\left(w_t\right)$ and smooth constant L, are unknown in practice. Note that in the training process, we only get access to information of stochastic gradient ${\overline{g}}_t$, model parameters $w_t$, and loss value $f\left(w_t\right)$. Most first-order adaptive learning methods exploit stochastic gradient and model parameters, which are usually high-dimensional vectors. Instead, we will use the stochastic gradient ${\overline{g}}_t$ and loss value to reformulate Equation (5). In particular, based on Equation (4), we have

$$\begin{array}{c}\hfill {\gamma }_t=\frac{\langle \nabla f\left(w_t\right),{\gamma }_t{\overline{g}}_t\rangle }{L{\gamma }_t{\parallel {\overline{g}}_t\parallel }^2}\le \frac{\frac{L{\gamma }_t^2}{2}{\parallel {\overline{g}}_t\parallel }^2+f\left(w_t\right)-f\left(w_{t+1}\right)}{L{\gamma }_t{\parallel {\overline{g}}_t\parallel }^2}.\end{array}$$

(7)

In the above inequality, parameters square norm of gradients $\parallel {\overline{g}}_t{\parallel }^2$ and learning rate ${\gamma }_t$ are known, and loss value of expected $f\left(w_t\right)$ at the current iteration can be approximated by empirical value $F(w_t,\xi )$; however, smooth constant L and the loss value $f\left(w_{t+1}\right)$ of the next iteration are unknown. Derived from the basic analysis [6], it is known that when $L{\gamma }_t\le 1$, SGD can converge to the stationary point. Therefore, we introduce a new parameter $\theta :=L{\gamma }_t\in (0,1]$ and use $f\left(w_{t-1}\right)-f\left(w_t\right)$ to approximate $f\left(w_t\right)-f\left(w_{t+1}\right)$ based on the assumption that $f\left(w\right)$ is smooth. Then, for any given learning rate $\gamma$ subject to $\theta \stackrel{▵}{=}L\gamma \in (0,1]$, we adjust the learning rate based on the following formula:

$$\begin{array}{c}\hfill \gamma \leftarrow \frac{\frac{\theta \gamma }{2}{\parallel {\overline{g}}_t\parallel }^2+max\{f\left(w_{t-1}\right)-f\left(w_t\right),0\}}{\theta \parallel {\overline{g}}_t{\parallel }^2},\theta \in (0,1].\end{array}$$

(8)

Equation (8) has straightforward interpretations. First, when $\parallel {\overline{g}}_t{\parallel }^2$ dominates $f\left(w_{t-1}\right)-f\left(w_t\right)$ in the numerator, we will use an exponential decaying learning rate strategy to prevent divergence. This includes the case that $f\left(w_{t-1}\right)-f\left(w_t\right)$ tends to zero, i.e., the model converges to a stationary point. In such case, ${\gamma }_t$ will decay to zero to stabilize the process. As shown in [46], in such case, any fixed large learning larger than a threshold will diverge the training. Second, when $f\left(w_{t-1}\right)-f\left(w_t\right)$ is relatively large, which means that the current model point is located at the rapid descent surface of $f\left(w\right)$, we can use a relatively large learning rate to accelerate the descent speed. This usually happens at the initial phase of the training procedure. Third, the parameter $\theta \in (0,1]$ can control the tradeoff between $\parallel {\overline{g}}_t{\parallel }^2$ and $f\left(w_{t-1}\right)-f\left(w_t\right)$. Note that these two terms have different magnitudes, and $\parallel {\overline{g}}_t{\parallel }^2$ is generally much larger than $f\left(w_{t-1}\right)-f\left(w_t\right)$. In such case, Equation (8) with $\theta =1$ reduces to a totally exponential decaying strategy, which may slow down the convergence. To address it, one can set an adaptive $\theta$ according to $f\left(w_{t-1}\right)-f\left(w_t\right)$.

Although Equation (8) is meaningful, the remaining challenge is how to set the proper $\theta$ according to the observed gradients $\parallel {\overline{g}}_t{\parallel }^2$ and difference in loss function $f\left(w_{t-1}\right)-f\left(w_t\right)$. To address this, we deal with $\theta \parallel {\overline{g}}_t{\parallel }^2$ as a whole and reformulate the term $\frac{max\{f\left(w_{t-1}\right)-f\left(w_t\right),0\}}{\theta \parallel {\overline{g}}_t{\parallel }^2}$ as a pure function of $f\left(w_{t-1}\right)-f\left(w_t\right)$. Here, we propose an effective adaptive schedule Adacomp that only uses the information of loss values and satisfies the following requirements.

• When $f\left(w_{t-1}\right)-f\left(w_t\right)<0$, we should decrease γ to restrict the progress along the wrong direction.
• When $f\left(w_{t-1}\right)-f\left(w_t\right)>0$, we should increase γ to make aggressive progress.
• When γ is too small, we should increase γ to prevent Adacomp from becoming trapped in a local minimum.
• When γ is too large, we should decrease γ to stabilize the training process.

These requirements motivate us to design Adacomp based on the $arctan$ function, which is flexible to small values but robust and bounded to large values. The main principle is as follows:

   Decompose γ into three parts, ${⨿}_1$, ${⨿}_{21}$, and ${⨿}_{22}$. ${⨿}_1$ is used to compensate the learning rate to accelerate training when the loss function decreases. However, the compensation should be inverse to learning rate and bounded. ${⨿}_{21}$ and ${⨿}_{22}$ are used to penalize the learning rate to stabilize training when the loss function increases, but with a bounded amplitude when the learning rate is too large or small.  

Based the above principle, we reformulate Equation (8) as:

$$\gamma \leftarrow \frac{\gamma }{2}+\gamma {\mathbb{I}}_{{\mathrm{\Delta }}_t>ϵ}{⨿}_1+\gamma {\mathbb{I}}_{{\mathrm{\Delta }}_t<-ϵ}({⨿}_{21}+{⨿}_{22}),$$

(9)

where $\mathbb{I}$ is the indicator function of ${\mathrm{\Delta }}_t\stackrel{▵}{=}f\left(w_{t-1}\right)-f\left(w_t\right)$ and $ϵ$ is a small positive constant. Many functions satisfy the above discipline. To reduce the difficulty of designing ${⨿}_1,{⨿}_{21},{⨿}_{22}$, we define the following expressions, where only one parameter $\beta$ needs to be tuned. Furthermore, as experimental results show (refer to Figure A1), Adacomp is not so sensitive to $\beta$.

$$\begin{array}{c}\hfill {⨿}_1=\frac{1}{2}+\frac{arctan({\mathrm{\Delta }}_t+1/\gamma )}{5\pi },{⨿}_{21}=\frac{1}{4}-\frac{arctan\left(\gamma \right)}{\beta \pi },{⨿}_{22}=\frac{1}{4}-\frac{arctan(-{\mathrm{\Delta }}_t+1/\gamma )}{5.5\pi },\end{array}$$

where $1/2<\beta \le 5$ is a parameter used to control the adjustment amplitude of the learning rate.

We first explain the meaning of $\epsilon$ and then ${⨿}_1,{⨿}_{21}$, and ${⨿}_{22}$.

The meaning of $ϵ$. We replace the hard threshold ${\mathrm{\Delta }}_t<0,>0$ with soft threshold ${\mathrm{\Delta }}_t<-ϵ,>ϵ$ on two aspect considerations. First, this can alleviate the impacts of ${\mathrm{\Delta }}_t$’s randomness on learning rate. Second, when ${\mathrm{\Delta }}_t\in [-ϵ,ϵ]$ for small positive values such as $ϵ={10}^{-5}$, we halve $\gamma$. Actually, in such a case the training has converged and halving $\gamma$ can make training more stable.

Now, we explain how to set expressions of ${⨿}_1,{⨿}_{21}$, and ${⨿}_{22}$.

• Expression of ${⨿}_1$. Note that ${⨿}_1$ works only if ${\mathrm{\Delta }}_t>0$. In this case, we should increase $\gamma$ to speed up training. However, the increment should be bounded and inverse to current $\gamma$ to avoid divergence. Based on these considerations, we define ${⨿}_1$ as ${⨿}_1=\frac{1}{2}+\frac{arctan({\mathrm{\Delta }}_t+1/\gamma )}{5\pi }$, where $1/2$ is used to keep $\gamma$ unchanged and $arctan(·)/\left(5\pi \right)$ is used to ensure that the increment amplitude is at most $\gamma /10$.
• Expressions of ${⨿}_{21}$ and ${⨿}_{22}$. Note that ${⨿}_{21},{⨿}_{22}$ work only if ${\mathrm{\Delta }}_t<0$, where the twice $1/4$ is used to keep $\gamma$ unchanged and the remaining terms are used to control the decrement amplitude. In this case, we should decrease $\gamma$ to prevent too much movement along the incorrect direction. However, the decrement should be bounded to satisfy the two following properties.

  −

  When $\gamma$ is small, the decrement of ${⨿}_{21}+{⨿}_{22}$ should be less than the increment in ${⨿}_1$ given the same $|{\mathrm{\Delta }}_t|$ unless $\gamma$ is forced to zero, which potentially leads to training stopping too early. Therefore, we define $\frac{arctan(·)}{5.5\pi }$ in ${⨿}_{21}$ to satisfy $\frac{arctan(·)}{5.5\pi }<\frac{arctan(·)}{5\pi }$. Note that ${⨿}_{21}+{⨿}_{22}\approx \frac{1}{2}-\frac{arctan(-{\delta }_t+1/\gamma )}{5.5\pi }$ when $\gamma$ is small, which satisfies the requirement.

  −

  When $\gamma$ is large, the decrement of ${⨿}_{21}+{⨿}_{22}$ should be larger than the increment in ${⨿}_1$ given the same $|{\mathrm{\Delta }}_t|$. That is, we can fast decrease the large learning rate when the function loss is negative. Therefore, we set $\beta \in (1/2,5]$ in ${⨿}_{21}$ to satisfy that $\frac{arctan\left(\gamma \right)}{\beta \pi }+\frac{arctan(-{\mathrm{\Delta }}_t+1/\gamma )}{5.5\pi }$ (in the case ${\mathrm{\Delta }}_t<0$) is greater than $\frac{arctan({\mathrm{\Delta }}_t+1/\gamma )}{5\pi }$ (in the case ${\mathrm{\Delta }}_t>0$).

  Furthermore, the inferior bound $1/2$ of $\beta$ is used to ensure $\frac{\gamma }{2}+\gamma {\mathbb{I}}_{{\mathrm{\Delta }}_t<-ϵ}({⨿}_{21}+{⨿}_{22})>0$ when ${\mathrm{\Delta }}_t<-ϵ$, i.e., the meaningfulness of $\gamma$.

Remark. In ${⨿}_1,{⨿}_{22}$, the denominator $5\pi ,5.5\pi$ can be generalized as ${\alpha }_1,{\alpha }_2$, with ${\alpha }_1<{\alpha }_2$, while keeping the difference between ${⨿}_1$ and ${⨿}_{22}$ is not significant. Meanwhile, $1/2<\beta \le 5$ is replaced by $1/2<\beta \le {\alpha }_1$. Note that any settings satisfying the above requirements can be used to substitute Equation (9). Here, $5\pi$ and $5.5\pi$ are just one satisfied setting and one can adjust the parameter $\beta$ to control the tradeoff between ${⨿}_1$ and ${⨿}_{21}+{⨿}_{22}$.

In summary, Equation (9) penalizes the large learning rate and compensates the too small learning rate, and presents an overall decreasing trend. These satisfy all the requirements. Note that this adjusting strategy is complementary to ${\mathrm{\Delta }}_t$ and $\gamma$; we named it Adacomp. Although Adacomp is robust to hyperparameters, it may fluctuate at a local minimum. The reason for this is that Adacomp will increase the learning rate when the loss function decreases near the local minima, and the increment potentially makes model parameters skip the local minima. Then, Adacomp in turn will penalize learning rate to decrease loss function. Thus, the alternately adaptive adjustment possibly makes model parameters fluctuate at a local minimum. To achieve both robustness and high-utility, one can view Adacomp as a pre-processing step to reduce the cost of tuning hyperparameters, and combine Adacomp with other methods to improve the final model utility. Algorithm 1 shows the pseudocode, which consists of two phases. Phase one uses Adacomp to fast-train the model by avoiding and carefully tuning the initial learning rate. Phase two uses other methods (e.g., SGD) to improve the model utility. More details are as follows. At the beginning (Input), one should define four parameters, including number of total iterations T with phase threshold $T_1$, initial learning rate ${\gamma }_0$, and one specific value $\beta$ of Adacomp used to control the tradeoff between increment and decrement.

In phase 1 (Lines 2–8), two variables, $los{s}_l,los{s}_c$, are used to track the loss difference ${\mathrm{\Delta }}_t$ in Equation (9), where $los{s}_l=\frac{1}{b}{\sum }_{\xi \in {\mathcal{B}}_{t-1}}F(w_{t-1},\xi )$ is loss of last time and $los{s}_c=\frac{1}{b}{\sum }_{\xi \in {\mathcal{B}}_t}F(w_t,\xi )$ is loss of current time (Lines 3–5). Then ${\mathrm{\Delta }}_t$ is used to update learning rate based on Equation (9) (Line 6). The updated learning rate ${\gamma }_t$ is used to update the model parameters $w_t$ to $w_{t+1}$ based on $w_{t+1}=w_t-{\gamma }_t{\overline{g}}_t$ (Line 7) until t reaches to $T_1$. In phase 2, we set the average of the learning rate used in phase one, ${\gamma }_{avg}$, as the new current initial learning rate (Line 9). Then, one can adopt a decaying scheduler, such as StepLR in mini-batch gradient descent update to stabilize the training process (Lines 11–13). The final model parameter is output when the stopping criterion is satisfied (Line 15).

Algorithm 1 Two-phase Training Algorithm.

Input: Initial ${\gamma }_0$, balance rate $\beta$, iteration number T, and phase threshold $T_1$ Output: ${\mathbf{w}}_T$

4. Experiments

In this section, we describe extensive experiments to validate the performance of Adacomp on five classification datasets, in comparison with other first-order iterative methods, including SGD, Momentum, Adagrad, RMSprop, Adadelta, Adam, and Adamax. Experimental results show that Adacomp is more robust to hyper-parameters and network architectures. Code and experiments are publicly available at https://github.com/IoTDATALab/Adacomp, (accessed on 26 October 2021).

4.1. Experimental Setup

We evaluated the eight first-order methods on five benchmark classification tasks, MNIST [47], KMNIST [48], Fashion-MNIST [49], CIFAR-10, and CIFAR-100 [50], which are used for 10 classification tasks, except CIFAR-100, which is used for 100 classification tasks. Moreover, MNIST, KMNIST, and Fashion-MNIST include 60,000 training samples and 10,000 test samples, while CIFAR-10 and CIFAR-100 include 50,000 training samples and $10,000$ test samples. For MNIST, the network architecture used was borrowed from examples in Pytorch (https://github.com/pytorch/examples, (accessed on 15 May 2021)). In particular, it consists of two convolutionary layers (32, 64 channels with kernel size 3 and relu activation), followed by one max pooling layer with size 2 and two fully connected layers (the first layer with relu activation), ending with the $log\_softmax$ function. For CIFAR-10, we used 6 out of 18 network architectures (https://github.com/kuangliu/pytorch-cifar, (accessed on 15 May 2021)): LeNet, VGG19, ResNet18, MobileNet, SENet18, and SimpleDLA. For CIFAR-100, we used ResNet [51] and the code is available at https://github.com/junyuseu/pytorch-cifar-models.git, (accessed on 15 May 2021). All experiments were deployed on a local workstation with a 10-core Geforce GPU and 128 GB memory, and cuda version 11.0. Note that Adacomp can be implemented by directly replacing Equation (8) in Pytorch’s SGD optimizer.

We set epochs as 10, 20, 100, and 150 for MNIST, KMNIST and Fashion-MNIST, CIFAR-10, and CIFAR-100, respectively. We set $T_1/T=0.6$ in Algorithm 1.

4.2. Results on MNIST Dataset

We conducted Adacomp (set $\beta =0.6$ in Equation (9)) on MNIST dataset to validate its robustness from aspects of learning rate, batch size, and initial model parameters. Predominately, Adacomp works well for a wider range of learning rates than all other compared methods.

4.2.1. Robustness to Initial Learning Rate

We set learning rate (LR) from ${10}^{-5}$ to 15, and used fine granularity in the interval $[0.01,1]$ but loose granularity outside it. Figure 1 shows the results of test accuracy after 10 epochs when LR lies in $[0.01,1]$. It is observed that only Adadelta and Adacomp work when LR is greater than one. Additionally, we illustrate results when LR lies in $[1,15]$ and the effective intervals of all methods are shown in Figure 2. The training batch size was fixed as 64 and the initial seed was set as 1 in all cases.

Two conclusions are observed from Figure 1 and Figure 2.

First, Adacomp is much more robust than the other seven iterative methods. From Figure 1, Adacomp achieves test accuracy greater than $98\%$ for all settings of learning rates. In contrast, RMSprop, Adam, and Adamax are sensitive to learning rate and only work well for very small learning rates. The other remaining methods, except Adadelta, have an intermediate robustness between Adamax and Adacomp. However, when the learning rate is greater than one, only Adadelta and Adacomp still work. Furthermore, from the top of Figure 2, it is observed that Adadelta works well until learning rate increases up to 9 while Adacomp still works even when learning rate is up to 15. The bottom of Figure 2 (logarithmic scale of horizontal axis) illustrates that each method has its own effective interval. From the top RMSprop to the penultimate Adadelta, the effective interval gradually slides from left to right. However, Adacomp (ours) has a much wider interval, which means one can successfully train the model almost without tuning learning rate. The reason for strong robustness is that Adacomp adjusts learning rate to a high value in the first few epochs before shrinking to a small value, no matter what the initial setting.

Second, the adaptive strategy may not always outperform SGD, which uses the fixed learning rate. As shown in Figure 1, adaptive methods, Momentum, Adagrad, RMSprop, Adam, and Adamax, are more sensitive to learning rate than SGD. This means that when using adaptive strategies, proper setting of the initial learning rate is necessary. However, we also observe that adaptive methods, Adadelta and Adacomp, are much more robust to initial learning rate than SGD.

4.2.2. Robustness to Other Hyperparameters

We conducted more experiments to compare the robustness of the eight iterative methods to batch sizes and initial model parameters. Figure 3a shows the impacts when batch sizes were set as $16,32,64,and128$, and Figure 3b shows the impacts when we repeated the experiment four individual times in which the network was initialized using the same random seed ($1,10,30,50$, respectively) for different optimizers. For fairness, we set learning rate to $0.01$. Under the level, all eight methods have a roughly equivalent accuracy (refer to Figure 1). It is observed from Figure 3a that the robustness to batch size, from strong to weak, is in the order $Momentum\approx Adagrad\approx Adamax\approx Adacomp\succ Adadelta\succ SGD\succ RMSprop\succ Adam$ (note that Adam was divergent when the batch size was set to 16). It is observed from Figure 3b that the robustness to initialized seeds is in the order $Adagrad\approx Momentum\approx Adacomp\approx Adamax\succ Adadelta\succ SGD\succ Adam\succ RMSprop$ (note that RMSprop was divergent when the seed was set to $30,50$). In summary, for a certain setting of learning rate, Momentum, Adagrad, Adamax, and Adacomp have the most robustness to batch sizes and initialized seeds, while RMSprop and Adam have the weakest robustness.

4.2.3. Convergence Speed and Efficiency

To compare the convergence speed, we further recorded the number of epochs when prediction accuracy firstly grows up than $98\%$ (Column 3 in

Table 2. Number of epochs when each of the methods first grow greater than $98\%$. Speed is calculated as epochs divided by ${\#}_{\ge 98\%}$. Dataset is MNSIT.

Method	${\#}_{\ge 98\%}$	Epochs	Speed	Training/Total Time (s)
SGD	13	19	1.46	68.13/78.22
Momentum	11	12	1.09	69.11/79.36
Adagrad	12	22	1.83	68.55/78.68
RMSprop	6	16	2.67	69.18/79.34
Adadelta	19	43	2.26	68.56/78.68
Adam	4	7	1.75	68.40/78.49
Adamax	9	28	3.5	68.55/78.67
Ours	25	23	0.92	68.25/78.26

). The convergence speed is defined as the ratio of Epochs to ${\#}_{\ge 98\%}$. Taking SGD as an example, it had 13 out of 25 times where the final prediction was greater than $98\%$ (refer to Figure 1). Meanwhile, among the 13 successful cases, SGD took a total of 19 epochs to achieve $98\%$. Then, the convergence speed was $19/13=1.46$. It is observed from Table 2 that Adacomp has the fastest convergence speed $0.92$, which means Adacomp can achieve at least an accuracy of $98\%$ using no more than one epoch on average. Subsequently, Momentum has a convergence speed of $1.09$ and Adamax has the slowest speed of $3.5$.

To compare the efficiency, we performed each of the methods ten times and recorded the average training and total time in the last column of Table 2. It is also observed that Adacomp achieves the lowest time consumption (excluding SGD), while Momentum and RMSprop have a relatively large time consumption. Although the improvement is slight, it can be deduced that Adacomp performs fewer computations than other first-order adaptive methods to adjust learning rate.

4.3. Results on CIFAR-10 Dataset

In this section, we conducted more experiments to further compare the robustness of eight iterative methods on CIFAR-10. Each of them was performed on three settings of learning rate (i.e., $0.5$, $0.05$, and $0.005$) and six network architectures (namely, LeNet, VGG, ResNet, MobileNet, SEnet, SimpleDLA). In this setting, we chose $\beta =5$ in Adacomp. Figure 4 shows the comparison of test accuracy on six network architectures when LR is $0.5$, $0.05$, $0.005$. Figure 5a,b show improvements in test accuracy when LR is degraded from $0.5$ to $0.05$, and $0.05$ to $0.005$, respectively. We conclude the robustness of eight methods with respect to different network architectures.

Firstly, we separately conclude the model utility for each setting of learning rate. (1) From Figure 4a, it is observed that Adacomp has a similar performance to the best Adadelta. However, Momentum, RMSprop, Adam, and Adamax, are failures for almost all architectures when LR is $0.5$. That is, these adaptive methods are not compatible with a large initial learning rate, which is also observed on MNIST (Figure 1). Among them, Adagrad is much more sensitive to the network architecture than Adadelta, Adacomp, and SGD. Surprisingly, SGD achieves a comparable result to Adadelta and Adacomp, except for the LeNet architecture. (2) From Figure 4b, it is observed that all methods, except RMSprop and Adam, have similar performances on each of the network architectures. Furthermore, except on the MobileNet architecture, Adacomp achieves the highest accuracy, followed by Adadelta and SGD (both have small degradation on SENet and SimpleDLA). Besides, it is observed that RMSprop and Adam are more sensitive to the network architectures than others. (3) From Figure 4c, it is observed that no method diverged when LR was set to $0.005$. Specifically, adaptive methods, Momentum, RMSprop, Adam, Adamax, and Adacomp (ours) perform similarly on each of the six architectures, except that Adacomp has the advantage for SimpleLDA and disadvantage for MobileNet. Subsequently, Adagrad achieves a similar performance except on the MobileNet architecture. However, Adadelta and SGD have a total degradation compared to other methods. This shows that the small setting of the initial learning rate is correct for almost adaptive methods.

Secondly, we combined Figure 4 and Figure 5a,b to conclude the robustness of each of the methods to network architectures and learning rates. (1) It is observed from Figure 4c that when the learning rate is small, all methods have a similar robustness to network architectures. Specifically, they perform poorer on LeNet and MobileNet than others. The reason is that these two architectures are relatively simple and more sensitive to inputs. However, it is observed from Figure 4a that when the learning rate is large, except Adadelta, Adacomp, and SGD, all other methods are sensitive to network architectures. (2) It is observed from Figure 5a,b that Momentum, Adagrad, RMSprop, Adam, and Adamax are more sensitive to learning rate than others. The reason is that when the learning rate is large, the acceleration in Momentum is too large to diverge, and also the accumulation of gradients in Adagrad, RMSprop, Adam, and Adamax is too large. In contrast, SGD, Adadelta, and Adacomp are relatively insensitive to learning rate, but SGD and Adadelta appear to have an overall degradation when the learning rate is small.

In summary, we conclude that the proposed Adacomp is robust both to learning rates and network architectures.

4.4. Results for Other Datasets

We conducted more experiments to further validate the robustness of Adacomp to datasets. In particular, we employed the network architecture that was used for MNIST in Section 4.2 but to datasets KMNIST and Fashion-MNIST. Furthermore, we employed a new network architecture (refer to https://github.com/junyuseu/pytorch-cifar-models.git, (accessed on 20 August 2021)) to fulfill the CIFAR-100 classification task. In each case, the learning rate was set as $0.001,0.01,0.1,1$ and we recorded the average time of CIFAR-100 after repeating 10 times. The final prediction accuracy after 150 epochs and the average time on CIFAR-100 are shown in

Table 3. Prediction accuracy of all methods on three datasets, KMNIST, Fashion-MNIST, and CIFAR-100, is listed, where learning rate was set as $0.001,0.01,0.1,1$. The average time of each method is also recorded, where Tr. and To. are abbreviations for training and total time, respectively.

Dataset	KMNIST				Fashion-MNIST				CIFAR-100
lr	0.001	0.01	0.1	1	0.001	0.01	0.1	1	0.001	0.01	0.1	1	Tr./To. Time (s)
SGD	87.43	92.74	95.07	82.36	83.31	88.60	92.26	10.00	17.91	50.82	57.69	60.59	1004.3/1145.5
Momen.	93.20	94.96	83.54	10.00	91.11	92.24	86.14	10.00	51.63	60.57	61.63	59.07	1052.4/1196.1
Adagrad	92.02	94.34	10.00	10.00	89.36	91.92	90.58	10.00	25.47	52.77	58.23	53.04	1017.0/1158.0
RMSprop	93.78	10.00	10.00	10.00	91.79	10.00	10.00	10.00	59.70	58.63	29.17	1.00	1042.0/1185.9
Adadelta	79.65	91.99	94.58	95.40	83.15	88.31	92.45	92.46	11.48	35.77	54.79	60.02	1025.2/1166.5
Adam	95.35	92.10	10.00	10.00	92.06	88.62	10.00	10.00	61.09	61.04	25.08	1.00	1030.5/1170.9
Adamax	95.09	94.70	10.00	10.00	92.52	91.10	86.82	10.00	57.51	61.41	58.37	1.00	1033.6/1176.0
Ours	93.96	91.90	93.51	93.11	89.94	90.88	90.24	89.25	63.96	62.14	63.50	63.50	1010.1/1148.9

. Moreover, the average and stand variance of prediction accuracy when the learning rate was $0.001,0.01,0.1,1$ are demonstrated in Figure 6. Three conclusions are observed as follows.

First, each method has its effective learning rate setting. For example, as Table 3 shows, Adam and Adamax are more effective when the learning rate value is relatively small (e.g., 0.001). However, SGD and Adadelta are more effective when the learning rate value is relatively large (e.g., 0.1). However, when learning rate equals 1, all methods except Adadelta and Adacomp fail on at least one of the three datasets. Second, as Figure 6 shows, Adacomp is more robust than all other methods on three datasets. In Figure 6, points with large horizontal value mean a high average prediction accuracy and with a small vertical value mean a low variance. Then, it is obvious to observe that Adacomp is more robust to datasets than other methods. Third, Adacomp is computationally efficient. This is directly observed from the last column of Table 3, where the training time and total time are presented. This is because Adacomp only adjusts learning rate according to training loss, while other adaptive methods exploit gradients or additional model parameters.

5. Conclusions and Future Work

We proposed a method, Adacomp, to adaptively adjust learning rate by only exploiting the values of the loss function. Therefore, Adacomp has higher computational efficiency than other gradient-based adaptive methods, such as Adam and RMSprop. From a high abstract level, Adacomp penalizes large learning rates to ensure the convergence and compensates small learning rates to accelerate the training process. Therefore, Adacomp can help escape from local minima with a certain probability. Extensive experimental results show that Adacomp is robust to network architecture, network initialization, batch size, and learning rate. The experimental results show that Adacomp is inferior to Adadelta and others in the maximum validation accuracy over learning rate. Thus the presented algorithm cannot be an alternative to the state-of-the-art. However, the adaptive methods use different learning rates for different parameters and Adacomp determines only the global learning rate.

In future work, we will validate Adacomp using more tasks (besides convolutional neural networks) and extend Adacomp to a per-dimension first-order algorithm to improve the accuracy of SGD. Additionally, we will apply Adacomp to the distributed environments. Note that Adacomp only uses values of loss functions. Therefore, it is suitable for distributed environments where communication overhead is a bottleneck. Furthermore, we will study how to set the parameters of Adacomp in an end-to-end mode. This may be achieved by introducing feedback and control modules.