Training Deep Neural Networks Using Conjugate Gradient-like Methods

Abstract

The goal of this article is to train deep neural networks that accelerate useful adaptive learning rate optimization algorithms such as AdaGrad, RMSProp, Adam, and AMSGrad. To reach this goal, we devise an iterative algorithm combining the existing adaptive learning rate optimization algorithms with conjugate gradient-like methods, which are useful for constrained optimization. Convergence analyses show that the proposed algorithm with a small constant learning rate approximates a stationary point of a nonconvex optimization problem in deep learning. Furthermore, it is shown that the proposed algorithm with diminishing learning rates converges to a stationary point of the nonconvex optimization problem. The convergence and performance of the algorithm are demonstrated through numerical comparisons with the existing adaptive learning rate optimization algorithms for image and text classification. The numerical results show that the proposed algorithm with a constant learning rate is superior for training neural networks.

1. Introduction

Deep neural networks are used for many tasks, such as natural language processing, computer vision, and text and image classification (see also [1,2,3] for applications of neural networks), and a number of algorithms have been presented to tune the model parameters of such networks. The appropriate parameters are found by solving nonconvex stochastic optimization problems. In particular, the algorithms solve these problems in order to adapt the learning rates of the model parameters. Accordingly, they are called adaptive learning rate optimization algorithms [4] (Subchapter 8.5), and they include AdaGrad [5], RMSProp [4] (Algorithm 8.5), Adam [6], and AMSGrad [7].

Recently, reference [8] preformed convergence analyses on adaptive learning rate optimization algorithms for constant learning rates and diminishing learning rates. The convergence analyses indicated that the algorithms with sufficiently small constant learning rates approximate stationary points of the problems [8] (Theorem 3.1). This implies that useful algorithms, such as Adam and AMSGrad, can use constant learning rates to solve the nonconvex stochastic optimization problems in deep learning, in contrast to the results in [6,7] that presented only analyses assuming the convexity conditions of objective functions for diminishing learning rates. The analyses also indicated that the algorithms with diminishing learning rates converge to stationary points of the problems and achieve a certain convergence rate [8] (Theorem 3.2). Numerical comparisons showed that the algorithms with constant learning rates perform better than the ones with diminishing learning rates.

Meanwhile, conjugate gradient methods are useful for unconstrained nonconvex deterministic optimization (see [9] for details on conjugate gradient methods). These methods use the conjugate gradient direction (see also (2) for the definition of the conjugate gradient direction with the Fletcher–Reeves formula), and they accelerate the steepest descent method. Conjugate gradient methods converge globally and generate the descent direction. In particular, the Hager-Zhang, Polak-Ribière-Polyak, and Hestenes–Stiefel methods have efficient numerical performance [9]. It seems that conjugate gradient methods could be applied to constrained optimization, because they might accelerate the existing methods for constrained optimization. However, the inconvenient possibility that the conjugate gradient methods may not converge to solutions to constrained optimization problems [10] (Proposition 3.2) means that we cannot apply them directly. Actually, the numerical results in [10] showed that the conjugate gradient methods with conventional formulas, such as the Fletcher-Reeves, Polak-Ribière-Polyak, and Hestenes-Stiefel formulas, do not always converge to solutions to constrained optimization problems.

The conjugate gradient direction has been modified so that it can be applied to constrained optimization. The modified direction is called the conjugate gradient-like direction [10,11,12,13,14], and it is obtained by replacing the formula used for finding the conventional conjugate gradient direction with a positive real sequence depending on the number of iterations (see (1) for the definition of the conjugate gradient-like direction). The conjugate gradient-like method with the conjugate gradient-like direction can be applied to constrained convex deterministic optimization. In particular, the conjugate gradient-like method converges to solutions to constrained convex deterministic optimization problems when the step sizes (which are called learning rates) are diminishing [10] (Theorem 3.1). Moreover, the numerical results in [10] showed that it converges faster than the existing steepest descent method.

Roughly speaking, the existing adaptive learning rate optimization algorithms [4] (Subchapter 8.5) are first-order methods using the steepest descent direction of an observed function at each iteration. Accordingly, using the conjugate gradient-like method would be useful to accelerate these algorithms. Hence, in this article, we propose an iterative method combining the existing adaptive learning rate optimization algorithms [4] (Subchapter 8.5) with the conjugate gradient-like method [10,11,12,13,14].

This article provides two convergence analyses. The first analysis shows that with a small constant learning rate, the proposed algorithm approximates a stationary point of a nonconvex optimization problem in deep learning (Theorem 1). The second analysis shows that with diminishing learning rates, it converges to a stationary point of the nonconvex optimization problem (Theorem 2). The convergence and performance of the proposed algorithm are examined through numerical comparisons with the existing adaptive learning rate optimization algorithms for image and text classification. The numerical results show that the proposed algorithm with a constant learning rate is superior for training neural networks, while the one with diminishing learning rates is not good for training neural networks.

This article is organized as follows. Section 2 gives the mathematical preliminaries and states the main problem. Section 3 presents the proposed algorithm for solving the main problem and analyzes its convergence. Section 4 numerically compares the behaviors of the proposed learning algorithms with those of the existing ones. Section 5 discusses the relationship between the previously reported results and the results in Section 3 and Section 4. Section 6 concludes the paper with a brief summary.

2. Mathematical Preliminaries

2.1. Notation and Definitions

$\mathbb{N}$ denotes the set of all positive integers and zero. ${\mathbb{R}}^d$ denotes a d-dimensional Euclidean space with inner product $\langle ·,·\rangle$, which induces the norm $\parallel ·\parallel$. ${\mathbb{S}}^d$ denotes the set of $d\times d$ symmetric matrices, i.e., ${\mathbb{S}}^d=\{X\in {\mathbb{R}}^{d\times d}:X=X^{\top }\}$. ${\mathbb{S}}_{++}^d$ denotes the set of $d\times d$ symmetric positive-definite matrices, i.e., ${\mathbb{S}}_{++}^d=\{X\in {\mathbb{S}}^d:X\succ O\}$. ${\mathbb{D}}^d$ denotes the set of $d\times d$ diagonal matrices, i.e., ${\mathbb{D}}^d=\{X\in {\mathbb{R}}^{d\times d}:X=\mathsf{diag}\left(x_i\right),x_i\in \mathbb{R}(i=1,2,\dots ,d)\}$. $A\odot B$ denotes the Hadamard product of matrices A and B. For all $\mathit{x}:=\left(x_i\right)\in {\mathbb{R}}^d$, we have $\mathit{x}\odot \mathit{x}:=\left(x_i^2\right)\in {\mathbb{R}}^d$.

Given $H\in {\mathbb{S}}_{++}^d$, the H-inner product of ${\mathbb{R}}^d$ and the H-norm are defined for all $\mathit{x},\mathit{y}\in {\mathbb{R}}^d$ by ${\langle \mathit{x},\mathit{y}\rangle }_H:=\langle \mathit{x},H\mathit{y}\rangle$ and ${\parallel \mathit{x}\parallel }_H^2:=\langle \mathit{x},H\mathit{x}\rangle$.

The metric projection [15] (Subchapter 4.2, Chapter 28) onto a nonempty, closed convex set X$(\subset {\mathbb{R}}^d)$, denoted by $P_X$, is defined for all $\mathit{x}\in {\mathbb{R}}^d$ by $P_X\left(\mathit{x}\right)\in X$ and $\parallel \mathit{x}-P_X\left(\mathit{x}\right)\parallel ={inf}_{\mathit{y}\in X}\parallel \mathit{x}-\mathit{y}\parallel$. $P_X$ satisfies the nonexpansivity condition, i.e., $\parallel P_X\left(\mathit{x}\right)-P_X\left(\mathit{y}\right)\parallel \le \parallel \mathit{x}-\mathit{y}\parallel$ ($\mathit{x},\mathit{y}\in {\mathbb{R}}^d$), and satisfies $\mathrm{Fix}\left(P_X\right):=\{\mathit{x}\in {\mathbb{R}}^d:\mathit{x}=P_X\left(\mathit{x}\right)\}=X$ [15] (Proposition 4.8, (4.8)). The metric projection onto X under the H-norm is denoted by $P_{X,H}$. When X is an affine subspace, a half-space, or a hyperslab, the projection onto X can be computed within a finite number of arithmetic operations [15] (Chapter 28).

$\mathbb{E}\left[X\right]$ denotes the expectation of a random variable X. The history of the process ${\mathbf{\xi }}_0,{\mathbf{\xi }}_1,\dots$ up to time n is denoted by ${\mathbf{\xi }}_{\left[n\right]}=({\mathbf{\xi }}_0,{\mathbf{\xi }}_1,\dots ,{\mathbf{\xi }}_n)$. For a random process ${\mathbf{\xi }}_0,{\mathbf{\xi }}_1,\dots$, $\mathbb{E}\left[X\right|{\mathbf{\xi }}_{\left[n\right]}]$ denotes the conditional expectation of X given ${\mathbf{\xi }}_{\left[n\right]}=({\mathbf{\xi }}_0,{\mathbf{\xi }}_1,\dots ,{\mathbf{\xi }}_n)$. Unless stated otherwise, all relations between random variables hold almost surely.

2.2. Stationary Point Problem Associated with Nonconvex Optimization Problem

Let us consider the following problem [8] (see, e.g., Subchapter 1.3.1 in [16] for details on stationary point problems):

Problem 1. 

Assume that

(A1)

$X\subset {\mathbb{R}}^d$ is a nonempty, closed convex set onto which the projection can be easily computed;

(A2)

$f:{\mathbb{R}}^d\to \mathbb{R}$, which is defined for all $\mathit{x}\in {\mathbb{R}}^d$ by $f\left(\mathit{x}\right):=\mathbb{E}\left[F\right(\mathit{x},\mathbf{\xi }\left)\right]$, is well defined, where $F(·,\mathbf{\xi })$ is continuously differentiable for almost every $\mathbf{\xi }\in \mathsf{\Xi }$, where $\mathbf{\xi }\in \mathsf{\Xi }$ is a random vector whose probability distribution P is supported on a set $\mathsf{\Xi }\subset {\mathbb{R}}^{d_1}$.

Then, we would like to find a stationary point ${\mathit{x}}^{🟉}$ of the problem of minimizing f over X, i.e.,

$$\begin{array}{c}\hfill {\mathit{x}}^{🟉}\in X^{🟉}:=\{{\mathit{x}}^{🟉}\in X:\langle \mathit{x}-{\mathit{x}}^{🟉},\nabla f({\mathit{x}}^{🟉})\rangle \ge 0(\mathit{x}\in X)\},\end{array}$$

where $\nabla f$ denotes the gradient of f.

We can see that, if $X={\mathbb{R}}^d$, then $X^{🟉}=\{{\mathit{x}}^{🟉}\in {\mathbb{R}}^d:\nabla f({\mathit{x}}^{🟉})=\mathbf{0}\}$ and that, if f is convex, then ${\mathit{x}}^{🟉}\in X^{🟉}$ is a global minimizer of f over X [16] (Subchapter 1.3.1).

Problem 1 is examined under the following conditions [8].

(C1)

There is an independent and identically distributed sample ${\mathbf{\xi }}_0,{\mathbf{\xi }}_1,\dots$ of realizations of the random vector $\mathbf{\xi }$;

(C2)

There is an oracle which, for a given input point $(\mathit{x},\mathbf{\xi })\in {\mathbb{R}}^d\times \mathsf{\Xi }$, returns a stochastic gradient $\mathsf{G}(\mathit{x},\mathbf{\xi })$ such that $\mathbb{E}\left[\mathsf{G}\right(\mathit{x},\mathbf{\xi }\left)\right]=\nabla f\left(\mathit{x}\right)$;

(C3)

There exists a positive number M such that, for all $\mathit{x}\in X$, ${\mathbb{E}[\parallel \mathsf{G}(\mathit{x},\mathbf{\xi })\parallel }^2]\le M^2$.

3. Conjugate Gradient-Like Method

Algorithm 1 is a method for solving Problem 1 under (C1)–(C3).

First, we would like to emphasize that Algorithm 1 uses a conjugate gradient-like direction [10,11,12,13] (see step 3 in Algorithm 1) defined by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_n=\gamma \in \left[0,\frac{1}{2}\right]\mathrm{or}\frac{1}{n},{\mathsf{G}}_n=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n{\mathsf{G}}_{n-1}.\end{array}\end{array}$$

(1)

The direction (1) differs from a conventional conjugate gradient direction using, for example, the Fletcher–Reeves formula,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\gamma }_n^{\mathrm{FR}}=\frac{\parallel \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n){\parallel }^2}{\parallel \mathsf{G}({\mathit{x}}_{n-1},{\mathbf{\xi }}_{n-1}){\parallel }^2},{\mathsf{G}}_n=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n^{\mathrm{FR}}{\mathsf{G}}_{n-1}.\end{array}\end{array}$$

(2)

Although conventional conjugate gradient methods are powerful tools for solving unconstrained smooth nonconvex optimization (see, e.g., [9] for details on conjugate gradient methods), iterative methods with the conjugate gradient-like directions are useful for solving constrained smooth optimization problems [10,11,12,13] (see also Section 1 for details). Since Problem 1 is a constrained optimization problem, we will focus on using conjugate gradient-like directions.

Algorithm 1 Conjugate gradient-like method for solving Problem 1

Require:${\left({\alpha }_n\right)}_{n\in \mathbb{N}}\subset (0,1),{\left({\beta }_n\right)}_{n\in \mathbb{N}}\subset [0,1),{\left({\gamma }_n\right)}_{n\in \mathbb{N}}\subset [0,1/2],\delta \in [0,1)$ 1: $n\leftarrow 0,{\mathit{x}}_0,{\mathsf{G}}_{-1},{\mathit{m}}_{-1}\in {\mathbb{R}}^d,{\mathsf{H}}_0\in {\mathbb{S}}_{++}^d\cap {\mathbb{D}}^d$ 2: loop 3:  ${\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n{\mathsf{G}}_{n-1}$ 4:  ${\mathit{m}}_n:={\beta }_n{\mathit{m}}_{n-1}+(1-{\beta }_n){\mathsf{G}}_n$ 5:  ${\widehat{\mathit{m}}}_n:={(1-{\delta }^{n+1})}^{-1}{\mathit{m}}_n$ 6:  ${\mathsf{H}}_n\in {\mathbb{S}}_{++}^d\cap {\mathbb{D}}^d$ 7:  Find ${\mathsf{d}}_n\in {\mathbb{R}}^d$ that solves ${\mathsf{H}}_n\mathsf{d}=-{\widehat{\mathit{m}}}_n$. 8:  ${\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n+{\alpha }_n{\mathsf{d}}_n)$ 9:  $n\leftarrow n+1$ 10: end loop

We can see that Algorithm 1 with ${\gamma }_n=0$ ($n\in \mathbb{N}$) coincides with the existing algorithm in [8] defined by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\mathit{m}}_n:={\beta }_n{\mathit{m}}_{n-1}+(1-{\beta }_n){\mathsf{G}}_n,\hfill \\ {\widehat{\mathit{m}}}_n:={(1-{\delta }^{n+1})}^{-1}{\mathit{m}}_n,\hfill \\ {\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n-{\alpha }_n{\mathsf{H}}_n^{-1}{\widehat{\mathit{m}}}_n),\hfill \end{array}\right.\end{array}\end{array}$$

(3)

where ${\mathsf{H}}_n\in {\mathbb{S}}_{++}^d\cap {\mathbb{D}}^d$. We can also show that algorithm (3) (i.e., Algorithm 1 with ${\gamma }_n=0$) includes AMSGrad [7] and Adam [6] by referring to [8] (Section 3). For example, consider ${\mathsf{H}}_n$ and ${\mathit{v}}_n$ ($n\in \mathbb{N}$) defined for all $n\in \mathbb{N}$ by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\mathit{v}}_n:=\zeta {\mathit{v}}_{n-1}+(1-\zeta )\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ \hfill & {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},v_{n,i}\}\right),\hfill \\ \hfill & {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \end{array}\end{array}$$

(4)

where ${\mathit{v}}_{-1}={\widehat{\mathit{v}}}_{-1}=\mathbf{0}\in {\mathbb{R}}^d$ and $\zeta \in [0,1)$. Then, algorithm (3) with (4) and $\delta =0$ is the AMSGrad algorithm. When ${\mathsf{H}}_n$ and ${\mathit{v}}_n$ ($n\in \mathbb{N}$) are defined for all $n\in \mathbb{N}$ by

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & {\mathit{v}}_n:=\zeta {\mathit{v}}_{n-1}+(1-\zeta )\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ \hfill & {\overline{\mathit{v}}}_n:={(1-{\zeta }^{n+1})}^{-1}{\mathit{v}}_n,\hfill \\ \hfill & {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},{\overline{v}}_{n,i}\}\right),\hfill \\ \hfill & {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \end{array}\end{array}$$

(5)

Algorithm (3) with (5) resembles the Adam algorithm (The original Adam uses ${\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\overline{v}}_{n,i}}\right)$ and does not always converge [7] (Theorems 1–3). We use ${\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right)$ to guarantee its convergence (see Theorems 1 and 2 for the convergence of Algorithm 1)).

For example, let us consider Algorithm 1 with (4) and $\delta =0$, i.e.,

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n{\mathsf{G}}_{n-1},\hfill \\ {\mathit{m}}_n:={\beta }_n{\mathit{m}}_{n-1}+(1-{\beta }_n){\mathsf{G}}_n,\hfill \\ {\mathit{v}}_n:=\zeta {\mathit{v}}_{n-1}+(1-\zeta )\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},v_{n,i}\}\right),\hfill \\ {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \\ {\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n-{\alpha }_n{\mathsf{H}}_n^{-1}{\mathit{m}}_n).\hfill \end{array}\right.\end{array}\end{array}$$

(6)

From the above discussion, algorithm (6) with ${\gamma }_n=0$ coincides with AMSGrad. We can see that algorithm (6) uses a conjugate gradient-like direction ${\mathsf{G}}_n=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n{\mathsf{G}}_{n-1}$, while AMSGrad (algorithm (3) with (4)) uses a gradient direction ${\mathsf{G}}_n=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)$.

The convergence analyses of Algorithm 1 assume the following conditions.

Assumption 1. 

The sequence ${\left({\mathsf{H}}_n\right)}_{n\in \mathbb{N}}\subset {\mathbb{S}}_{++}^d\cap {\mathbb{D}}^d$, denoted by ${\mathsf{H}}_n:=\mathsf{diag}\left(h_{n,i}\right)$, in Algorithm 1 satisfies the following conditions:

(A3)

$h_{n+1,i}\ge h_{n,i}$ almost surely for all $n\in \mathbb{N}$ and all $i=1,2,\dots ,d$;

(A4)

For all $i=1,2,\dots ,d$, a positive number $B_i$ exists such that $sup\{\mathbb{E}\left[h_{n,i}\right]:n\in \mathbb{N}\}\le B_i$.

Moreover,

(A5)

$D:={max}_{i=1,2,\dots ,d}sup\{{(x_i-y_i)}^2:\left(x_i\right),\left(y_i\right)\in X\}<+\infty$.

Assumption (A5) holds under the boundedness condition of X, which is assumed in [17] (p. 1574) and [7] (p. 2). In [8] (Section 3), it is shown that ${\mathsf{H}}_n$ and ${\mathit{v}}_n$ defined by (4) or (5) satisfies (A3) and (A4).

3.1. Constant Learning Rate Rule

The following is the convergence analysis of Algorithm 1 with a constant learning rate. Theorem 1 can be inferred by referring to the proof of Theorem 3.1 in [8]. The proof of Theorem 1 is given in Appendix A.

Theorem 1. 

Suppose that (A1)–(A5) and (C1)–(C3) hold and ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ is the sequence generated by Algorithm 1 with ${\alpha }_n:=\alpha$, ${\beta }_n:=\beta$, and ${\gamma }_n:=\gamma$ ($n\in \mathbb{N}$). Then, for all $\mathit{x}\in X$,

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim\; sup}\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_n,\nabla f\left({\mathit{x}}_n\right)\rangle \right]\ge -\frac{{\tilde{B}}^2{\tilde{M}}^2}{2\tilde{b}{\tilde{\delta }}^2}\alpha -\frac{\sqrt{Dd}\tilde{M}}{\tilde{b}\tilde{\delta }}\beta -\frac{2\sqrt{Dd}\widehat{M}}{\tilde{\delta }}\gamma ,\end{array}$$

where $\tilde{\delta }:=1-\delta$, $\tilde{b}:=1-\beta$, M is defined as in (C3), ${\widehat{M}}^2:=max\{M^2,\parallel {\mathsf{G}}_{-1}{\parallel }^2\}$, ${\tilde{M}}^2:=max\{\parallel {\mathit{m}}_{-1}{\parallel }^2,4{\widehat{M}}^2\}$, D is defined as in (A5), and $\tilde{B}:=sup\{{max}_{i=1,2,\dots ,d}{h}_{n,i}^{-1/2}:n\in \mathbb{N}\}<+\infty$.

Theorem 1 shows that using a small constant learning rate approximates a solution to Problem 1. The result for $\gamma :=0$ coincides with Theorem 3.1 in [8].

We have the following proposition for convex stochastic optimization.

Proposition 1. 

Suppose that (A1)–(A5) and (C1)–(C3) hold, $F(·,\mathbf{\xi })$ is convex for almost every $\mathbf{\xi }\in \mathsf{\Xi }$, and ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ is the sequence generated by Algorithm 1 with ${\alpha }_n:=\alpha$, ${\beta }_n:=\beta$, and ${\gamma }_n:=\gamma$ ($n\in \mathbb{N}$). Then,

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim\; inf}\mathbb{E}\left[f\left({\mathit{x}}_n\right)-f^{🟉}\right]\le \frac{{\tilde{B}}^2{\tilde{M}}^2}{2\tilde{b}{\tilde{\delta }}^2}\alpha +\frac{\sqrt{Dd}\tilde{M}}{\tilde{b}\tilde{\delta }}\beta +\frac{2\sqrt{Dd}\widehat{M}}{\tilde{\delta }}\gamma ,\end{array}$$

where $f^{🟉}$ denotes the optimal value of the problem of minimizing f over X, and $\tilde{\delta }$, $\tilde{b}$, M, $\widehat{M}$, $\tilde{M}$, D, and $\tilde{B}$ are defined as in Theorem 1.

The previously reported results in [7] showed that AMSGrad, which is an example of Algorithm 1 (see Algorithm (3) with (4) and $\delta =0$), ensures that there exists a positive real number B such that

$$\begin{array}{c}\hfill \frac{R\left(T\right)}{T}=\frac{1}{T}\left(\sum _{t=1}^{T}F({\mathit{x}}_t,{\mathbf{\xi }}_t)-f^{🟉}\right)\le B\sqrt{\frac{1+lnT}{T}},\end{array}$$

(7)

where T is the number of training examples and $F(·,\mathbf{\xi })$ is convex for almost every $\mathbf{\xi }\in \mathsf{\Xi }$. Inequality (7) indicates that the value $R\left(T\right)/T$ generated by AMSGrad has an upper bound; however, it is not guaranteed that AMSGrad solves Problem 1. Meanwhile, Proposition 1 shows that Algorithm 1, which includes Adam and AMSGrad, can approximate a global minimizer of f by using a small constant learning rate.

3.2. Diminishing Learning Rate Rule

The following is the convergence analysis of Algorithm 1 with diminishing learning rates. Theorem 2 can be proven by referring to the proof of Theorem 3.2 in [8]. The proof of Theorem 2 is given in Appendix A.

Theorem 2. 

Suppose that (A1)–(A5) and (C1)–(C3) hold and ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ is the sequence generated by Algorithm 1 with ${\alpha }_n$, ${\beta }_n$, and ${\gamma }_n$ ($n\in \mathbb{N}$) satisfying ${\sum }_{n=0}^{+\infty }{\alpha }_n=+\infty$, ${\sum }_{n=0}^{+\infty }{\alpha }_n^2<+\infty$, ${\sum }_{n=0}^{+\infty }{\alpha }_n{\beta }_n<+\infty$, and ${\sum }_{n=0}^{+\infty }{\alpha }_n{\gamma }_n<+\infty$. Then, for all $\mathit{x}\in X$,

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim\; sup}\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_n,\nabla f\left({\mathit{x}}_n\right)\rangle \right]\ge 0.\end{array}$$

(8)

Moreover, suppose that ${\alpha }_n:=1/n^{\eta }$, ${\beta }_n:={\beta }^n$, ${\gamma }_n:={\gamma }^n$ or $1/n^{\kappa }$, where $\eta \in [1/2,1)$, $\kappa >1-\eta$, and $\beta ,\gamma \in (0,1)$. Then, Algorithm 1 achieves the following convergence rate:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{k=1}^n\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_k,\nabla f\left({\mathit{x}}_k\right)\rangle \right]\ge \left\{\begin{array}{cc}{\displaystyle -\mathcal{O}\left(\sqrt{\frac{1+lnn}{n}}\right)}\hfill & if\eta =\frac{1}{2},\hfill \\ {\displaystyle -\mathcal{O}\left(\frac{1}{n^{1-\eta }}\right)}\hfill & if\eta \in \left(\frac{1}{2},1\right).\hfill \end{array}\right.\end{array}$$

Inequality (8) implies that there exists a subsequence ${\left({\mathit{x}}_{n_j}\right)}_{j\in \mathbb{N}}$ of ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ such that ${\left({\mathit{x}}_{n_j}\right)}_{j\in \mathbb{N}}$ converges to ${\mathit{x}}_{🟉}$ and, for all $\mathit{x}\in X$,

$$\begin{array}{c}\hfill \underset{j\to +\infty }{lim}\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_{n_j},\nabla f\left({\mathit{x}}_{n_j}\right)\rangle \right]=\underset{n\to +\infty }{lim\; sup}\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_n,\nabla f\left({\mathit{x}}_n\right)\rangle \right]\ge 0,\end{array}$$

which implies that ${\mathit{x}}_{🟉}$ satisfies $\langle \mathit{x}-{\mathit{x}}_{🟉},\nabla f\left({\mathit{x}}_{🟉}\right)\rangle \ge 0$ ($\mathit{x}\in X$); i.e., ${\mathit{x}}_{🟉}$ is a solution to Problem 1.

Theorem 2 leads to the following proposition, which indicates that Algorithm 1 converges to a global minimizer of f when $F(·,\mathbf{\xi })$ is convex for almost every $\mathbf{\xi }\in \mathsf{\Xi }$.

Proposition 2. 

Suppose that (A1)–(A5) and (C1)–(C3) hold, $F(·,\mathbf{\xi })$ is convex for almost every $\mathbf{\xi }\in \mathsf{\Xi }$, and ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ is the sequence generated by Algorithm 1 with ${\alpha }_n$, ${\beta }_n$, and ${\gamma }_n$ satisfying ${\sum }_{n=0}^{+\infty }{\alpha }_n=+\infty$, ${\sum }_{n=0}^{+\infty }{\alpha }_n^2<+\infty$, ${\sum }_{n=0}^{+\infty }{\alpha }_n{\beta }_n<+\infty$, and ${\sum }_{n=0}^{+\infty }{\alpha }_n{\gamma }_n<+\infty$. Then,

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim\; inf}\mathbb{E}\left[f\left({\mathit{x}}_n\right)-f^{🟉}\right]=0,\end{array}$$

where $f^{🟉}$ denotes the optimal value of the problem of minimizing f over X. Moreover, suppose that ${\alpha }_n:=1/n^{\eta }$, ${\beta }_n:={\beta }^n$, ${\gamma }_n:={\gamma }^n$ or $1/n^{\kappa }$, where $\eta \in [1/2,1)$, $\kappa >1-\eta$, and $\beta ,\gamma \in (0,1)$. Then, any accumulation point of ${\left({\tilde{\mathit{x}}}_n\right)}_{n\in \mathbb{N}}$ defined by ${\tilde{\mathit{x}}}_n:=(1/n){\sum }_{k=1}^n{\mathit{x}}_k$ almost surely belongs to the solution set $X^{🟉}$, and Algorithm 1 achieves the following convergence rate:

$$\begin{array}{c}\hfill \mathbb{E}\left[f\left({\tilde{\mathit{x}}}_n\right)-f^{🟉}\right]=\left\{\begin{array}{cc}{\displaystyle \mathcal{O}\left(\sqrt{\frac{1+lnn}{n}}\right)}\hfill & if\eta =\frac{1}{2},\hfill \\ {\displaystyle \mathcal{O}\left(\frac{1}{n^{1-\eta }}\right)}\hfill & if\eta \in \left(\frac{1}{2},1\right).\hfill \end{array}\right.\end{array}$$

4. Numerical Experiments

The experiments used a fast scalar computation server (https://www.meiji.ac.jp/isys/hpc/ia.html) at Meiji University. The environment has two Intel(R) Xeon(R) Gold 6148 (2.4 GHz, 20 cores) CPUs, an NVIDIA Tesla V100 (16GB, 900Gbps) GPU, and a Red Hat Enterprise Linux 7.6 operating system. The experimental code was written in Python 3.8.2, and we used the NumPy 1.19.1 package and PyTorch 1.5.0 package.

We compared the existing algorithms, such as the momentum method [18] (9), [19] (Section 2), AdaGrad [5], RMSProp [4] (Algorithm 8.5), Adam [6], and AMSGrad [7] in torch.optim (https://pytorch.org/docs/stable/optim.html) using the default values and learning rate ${10}^{-3}$, with Algorithm 1 defined as follows:

Algorithm 1 with a constant learning rate (Algorithm 1 with ${\gamma }_n=0$, such as Momentum-Ci, Adam-Ci, and AMSGrad-Ci ($i=1,2,3$), is Algorithm 1 in [8]):

• Momentum-C1: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, ${\alpha }_n={\beta }_n={10}^{-1}$, and ${\gamma }_n=0$.
• Momentum-C2: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, ${\alpha }_n={\beta }_n={10}^{-2}$, and ${\gamma }_n=0$.
• Momentum-C3: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, ${\alpha }_n={\beta }_n={10}^{-3}$, and ${\gamma }_n=0$.
• MomentumCG-C1: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-1}$.
• MomentumCG-C2: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-2}$.
• MomentumCG-C3: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-3}$.
• Adam-C1: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), ${\alpha }_n={\beta }_n={10}^{-1}$, and ${\gamma }_n=0$.
• Adam-C2: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), ${\alpha }_n={\beta }_n={10}^{-2}$, and ${\gamma }_n=0$.
• Adam-C3: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), ${\alpha }_n={\beta }_n={10}^{-3}$, and ${\gamma }_n=0$.
• AdamCG-C1: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-1}$.
• AdamCG-C2: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-2}$.
• AdamCG-C3: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-3}$.
• AMSGrad-C1: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), ${\alpha }_n={\beta }_n={10}^{-1}$, and ${\gamma }_n=0$.
• AMSGrad-C2: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), ${\alpha }_n={\beta }_n={10}^{-2}$, and ${\gamma }_n=0$.
• AMSGrad-C3: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), ${\alpha }_n={\beta }_n={10}^{-3}$, and ${\gamma }_n=0$.
• AMSGradCG-C1: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-1}$.
• AMSGradCG-C2: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-2}$.
• AMSGradCG-C3: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\alpha }_n={\beta }_n={\gamma }_n={10}^{-3}$.

Algorithm 1 with diminishing learning rates ${\alpha }_n=1/\sqrt{n}$ and ${\beta }_n=1/2^n$ based on [7] (Theorem 4 and Corollary 1) (Algorithm 1 with ${\gamma }_n=0$, such as Momentum-D1, Adam-D1, and AMSGrad-D1, is Algorithm 1 in [8]):

• Momentum-D1: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\gamma }_n=0$.
• MomentumCG-D1: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\gamma }_n=1/2^n$.
• MomentumCG-D2: Algorithm 1 with $\delta =0$, ${\mathsf{H}}_n=\mathsf{diag}\left(1\right)$, and ${\gamma }_n=1/n$.
• Adam-D1: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\gamma }_n=0$.
• AdamCG-D1: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\gamma }_n=1/2^n$.
• AdamCG-D2: Algorithm 1 with $\delta =0.9$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (5), and ${\gamma }_n=1/n$.
• AMSGrad-D1: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\gamma }_n=0$.
• AMSGradCG-D1: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\gamma }_n=1/2^n$.
• AMSGradCG-D2: Algorithm 1 with $\delta =0$, $\zeta =0.999$, ${\mathsf{H}}_n$ defined by (4), and ${\gamma }_n=1/n$.

Python implementations of the algorithms are available at https://github.com/iiduka-researches/202008-cg-like.

4.1. Image Classification

This experiment used the CIFAR10 dataset (https://www.cs.toronto.edu/~kriz/cifar.html), a benchmark for image classification. The dataset consists of 60,000 color images ($32\times 32$) in 10 classes, with 6000 images per class. There are 50,000 training images and 10,000 test images. The test batch contained exactly 1000 randomly selected images from each class. We trained a 44-layer ResNet (ResNet-44) [20] organized into 43 convolutional layers which had $3\times 3$ filters and a 1000-way-fully-connected layer with a softmax function. We used the cross entropy as the loss function for fitting ResNet in accordance with the commonly used strategy in image classification.

Figure 1, Figure 2 and Figure 3 compare the behaviors of the proposed algorithm with a constant learning rate with those of Momentum, AdaGrad, RMSProp, Adam, and AMSGrad using the default values in torch.optim (i.e., ${\alpha }_n={10}^{-3},{\beta }_n=0.9$). Figure 1 shows that Momentum-C1, MomentumCG-C1, and AMSGrad-C2 minimized the training loss function faster than the existing algorithms, and Figure 2 shows that they decreased the training error rate faster as well. Moreover, AdamCG-Ci (resp. AMSGradCG-Ci) ($i=2,3$) outperformed AdamCG-C1 (resp. AMSGradCG-C1); this implies AdamCG and AMSGradCG require fewer iterations at smaller learning rates. Figure 3 shows that Adam-C2, AdamCG-C2, AMSGrad-C2, AMSGradCG-C2 decreased the test error rate faster than other algorithms. A similar trend was observed in the numerical results in [21].

Figure 4, Figure 5 and Figure 6 plot the behaviors of the proposed algorithms with diminishing learning rates. These algorithms did not work, and thus, it is clear that using diminishing learning rates is not good for training neural networks (see Section 5 for the details). A similar problem was observed in the numerical results in [8].

Table 1. Mean and variance of elapsed time per epoch for the existing algorithms and Algorithm 1 on the CIFAR-10 dataset.

		Existing	C1	C2	C3	CG-C1	CG-C2	CG-C3
Momentum	mean	14.815106	14.766352	14.643343	14.191675	14.370240	14.536258	13.732973
	variance	0.268979	1.144346	0.268576	0.363746	0.180754	0.872769	0.314055
Adam	mean	17.621361	17.388947	18.511805	18.084771	18.106918	18.108820	17.127479
	variance	0.149553	0.044539	1.392942	0.056606	0.213341	0.063594	1.317213
AMSGrad	mean	18.122551	17.650377	17.796328	19.335775	18.855297	18.272888	16.328777
	variance	1.245563	0.313088	0.289944	4.738541	2.820650	1.671705	1.754373

shows the mean and variance of elapsed time per epoch for the existing algorithms and Algorithm 1 with a constant learning rate. This table indicates that the elapsed time of Momentum was almost the same as those of the proposed algorithms, e.g., Momentum-Ci and MomentumCG-Ci ($i=1,2,3$). Adam and AMSGrad also had such a trend.

Table 2. Results of t-test on the training error rates of the existing algorithms (Momentum, Adam, and AMSGrad) and Algorithm 1 (Ci and CG-Ci ($i=1,2,3$)) on the CIFAR-10 dataset (significance level is 5%; the p-values for the proposed algorithms with significantly low error rates are indicated in bold).

		C1	C2	C3	CG-C1	CG-C2	CG-C3
Momentum	t-statistic	3.70879	0.23783	−13.65314	3.34063	−0.17214	−12.77890
(Existing)	p-value	2.38$\times {\mathbf{10}}^{-\mathbf{4}}$	8.12 $\times {10}^{-1}$	4.44 $\times {10}^{-35}$	9.15 $\times {\mathbf{10}}^{-\mathbf{4}}$	8.63 $\times {10}^{-1}$	1.43 $\times {10}^{-31}$
Adam	t-statistic	−10.46006	0.03599	0.20774	−6.70248	0.37493	0.04342
(Existing)	p-value	8.73 $\times {10}^{-23}$	9.71 $\times {10}^{-1}$	8.36 $\times {10}^{-1}$	7.03 $\times {10}^{-11}$	7.08 $\times {10}^{-1}$	9.65 $\times {10}^{-1}$
AMSGrad	t-statistic	−157.96917	−1.59278	−0.16230	−157.97057	−1.59440	−0.00869
(Existing)	p-value	0.00 $\times {10}^0$	1.12 $\times {10}^{-1}$	8.71 $\times {10}^{-1}$	0.00 $\times {10}^0$	1.12 $\times {10}^{-1}$	9.93 $\times {10}^{-1}$

compares the training error rates of the existing algorithms with those of Algorithm 1 by using the scipy.stats.ttest_ind function in Python. The p-value is the probability associated with a t-test, and the significance level is set at 5%. If the value is less than 0.05, then there is a significant difference between the existing algorithm and the proposed algorithms. Table 2 and Figure 2 indicate that Momentum-C1 and MomentumCG-C1 outperformed Momentum and the performance of the existing algorithm (Momentum) was significantly different from the performances of the proposed algorithms (Momentum-C1 and MomentumCG-C1). Adam-Ci and AdamCG-Ci ($i=1,2,3$) had almost the same performance as Adam, while the performance of AMSGrad was not significantly different from that of AMSGrad-Ci and AMSGradCG-Ci ($i=1,2,3$).

4.2. Text Classification

This experiment used the IMDb dataset (https://datasets.imdbws.com/) for text classification tasks. The dataset contains 50,000 movie reviews along with their associated binary sentiment polarity labels. The dataset is split into 25,000 training and 25,000 test sets. We used an embedding layer that generated 50-dimensional embedding vectors and two bidirectional long short-term memory (LSTM) with an affine layer and a sigmoid function as an activation function for the output. To train it, we used the binary cross entropy (BCE) as a loss function minimized by the existing and proposed algorithms.

Figure 7, Figure 8 and Figure 9 compare the behaviors of the proposed algorithm with a constant learning rate with those of Momentum, AdaGrad, RMSProp, Adam, and AMSGrad, using the default values in torch.optim (i.e., ${\alpha }_n={10}^{-3},{\beta }_n=0.9$). These figures show that Adam-C3, AdamCG-C3, AMSGrad-C3, RMSProp, Adam, and AMSGrad all performed well. In particular, Figure 8 shows that AdamCG-C3 (resp. AMSGradCG-C3) performed better than Adam-C3 (resp. AMSGrad-C3), which implies that using conjugate gradient-like directions would be good for training neural networks.

Figure 10, Figure 11 and Figure 12 indicate the behaviors of the proposed algorithms with diminishing learning rates. These figures show that the algorithms did not work, as was the case in Figure 4, Figure 5 and Figure 6 (see Section 5 for the details).

Table 3. Mean and variance of elapsed time per epoch for the existing algorithms and Algorithm 1 on the IMDb dataset.

		Existing	C1	C2	C3	CG-C1	CG-C2	CG-C3
Momentum	mean	19.029660	18.999186	18.957496	19.098836	19.241769	19.286854	18.671163
	variance	0.095132	0.074935	0.107259	0.196841	0.035649	0.058319	0.003906
Adam	mean	20.256827	20.194220	20.193260	20.260705	20.231550	20.388470	19.536741
	variance	0.061552	0.023485	0.041777	0.060461	0.039103	0.174818	0.165803
AMSGrad	mean	20.109489	20.092463	20.102763	20.025613	20.146646	20.136673	19.335856
	variance	0.075432	0.059149	0.059561	0.089540	0.113563	0.098914	0.003543

indicates that the elapsed time for the existing algorithm was almost the same as the one for the proposed algorithms, as seen in Table 1.

Table 4. Results of t-test on the training error rates of the existing algorithms (Momentum, Adam, and AMSGrad) and Algorithm 1 (Ci and CG-Ci ($i=1,2,3$)) on the IMDb dataset (significance level is 5%; the p-values for the proposed algorithms with significantly low error rates are indicated in bold).

		C1	C2	C3	CG-C1	CG-C2	CG-C3
Momentum	t-statistic	13.87142	0.63115	−4.59306	13.22951	1.71477	−4.59306
(Existing)	p-value	5.17 $\times {10}^{-31}$	5.29 $\times {10}^{-1}$	7.76 $\times {10}^{-6}$	4.82 $\times {10}^{-29}$	8.80 $\times {10}^{-2}$	7.76 $\times {10}^{-6}$
Adam	t-statistic	−63.39972	−11.01275	−0.00287	−63.33435	−9.41552	0.11707
(Existing)	p-value	1.79 $\times {10}^{-133}$	2.61 $\times {10}^{-22}$	9.98 $\times {\mathbf{10}}^{-\mathbf{1}}$	2.17 $\times {10}^{-133}$	1.24 $\times {10}^{-17}$	9.07 $\times {\mathbf{10}}^{-\mathbf{1}}$
AMSGrad	t-statistic	−63.53084	−5.68706	−0.63240	−63.42279	−7.93451	−0.06863
(Existing)	p-value	1.21 $\times {10}^{-133}$	4.59 $\times {10}^{-8}$	5.28 $\times {10}^{-1}$	1.67 $\times {10}^{-133}$3	1.53 $\times {10}^{-13}$	9.45 $\times {10}^{-1}$

and Figure 8 show that, although Momentum, Momentum-Ci, and MomentumCG-Ci did not perform better than the existing algorithms such as Adam and AMSGrad, the performance of Momentum was significantly different from that of almost all of proposed algorithms. It can be seen that Adam, Adam-C3, and AdamCG-C3 performed well and that, although AMSGrad, AMSGrad-C3, and AMSGradCG-C3 did not perform better than Adam, AMSGrad-C3 and AMSGradCG-C3 had almost the same performance as AMSGrad.

5. Discussion

Let us first discuss the relationship between the momentum method [18] (9), [19] (Section 2) with MomentumCG used in Section 4. The momentum method [18] (9), [19] (Section 2) is defined by

$$\begin{array}{cc}\hfill & {\mathit{m}}_n:=-ϵ\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)+\mu {\mathit{m}}_{n-1},{\mathit{x}}_{n+1}:=P_X({\mathit{x}}_n+{\mathit{m}}_n),\mathrm{i}.\mathrm{e}.,\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill & {\mathit{x}}_{n+1}:=P_X\left({\mathit{x}}_n-ϵ\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)+\mu {\mathit{m}}_{n-1}\right),\hfill \end{array}$$

(9b)

where $ϵ>0$ is the learning rate and $\mu \in [0,1]$ is the momentum coefficient. We can see that ${\mathit{m}}_n$ defined by (9) is the conjugate gradient-like direction of $ϵ\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)$. Meanwhile, MomentumCG used in Section 4 is as follows:

$$\begin{array}{cc}\hfill & {\mathsf{G}}_n=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{\gamma }_n{\mathsf{G}}_{n-1},\hfill \end{array}$$

(10a)

$$\begin{array}{cc}\hfill & {\mathit{m}}_n:=(1-{\beta }_n){\mathsf{G}}_n+{\beta }_n{\mathit{m}}_{n-1},\hfill \end{array}$$

(10b)

$$\begin{array}{cc}\hfill & {\mathit{x}}_{n+1}:=P_X({\mathit{x}}_n-{\alpha }_n{\mathit{m}}_n).\hfill \end{array}$$

(10c)

Algorithm (10) uses the conjugate gradient-like direction ${\mathsf{G}}_n$ of $\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)$. For simplicity, algorithm (10) with ${\beta }_n=0$ is such that

$$\begin{array}{c}\hfill {\mathit{x}}_{n+1}:=P_X\left({\mathit{x}}_n-{\alpha }_n\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)+{\alpha }_n{\gamma }_n{\mathit{m}}_{n-1}\right),\end{array}$$

(11)

which implies that algorithm (11) is the momentum method with a learning rate ${\alpha }_n$ and momentum coefficient ${\alpha }_n{\gamma }_n$.

The numerical comparisons in Section 4 show that Algorithm 1 with a constant learning rate performed better than Algorithm 1 with diminishing learning rates. For example, let us consider the text classification in Section 4.2 and compare AdamCG-C3 defined by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-{10}^{-3}{\mathsf{G}}_{n-1},\hfill \\ {\mathit{m}}_n:={10}^{-3}{\mathit{m}}_{n-1}+(1-{10}^{-3}){\mathsf{G}}_n,\hfill \\ {\widehat{\mathit{m}}}_n:={(1-{0.9}^{n+1})}^{-1}{\mathit{m}}_n,\hfill \\ {\mathit{v}}_n:=0.999{\mathit{v}}_{n-1}+(1-0.999)\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\overline{\mathit{v}}}_n:={(1-{0.999}^{n+1})}^{-1}{\mathit{v}}_n,\hfill \\ {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},{\overline{v}}_{n,i}\}\right),\hfill \\ {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \\ {\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n-{10}^{-3}{\mathsf{H}}_n^{-1}{\mathit{m}}_n)\hfill \end{array}\right.\end{array}\end{array}$$

(12)

with AdamCG-D1 defined by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-2^{-n}{\mathsf{G}}_{n-1},\hfill \\ {\mathit{m}}_n:=2^{-n}{\mathit{m}}_{n-1}+(1-2^{-n}){\mathsf{G}}_n,\hfill \\ {\widehat{\mathit{m}}}_n:={(1-{0.9}^{n+1})}^{-1}{\mathit{m}}_n,\hfill \\ {\mathit{v}}_n:=0.999{\mathit{v}}_{n-1}+(1-0.999)\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\overline{\mathit{v}}}_n:={(1-{0.999}^{n+1})}^{-1}{\mathit{v}}_n,\hfill \\ {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},{\overline{v}}_{n,i}\}\right),\hfill \\ {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \\ {\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n-n^{-1/2}{\mathsf{H}}_n^{-1}{\mathit{m}}_n).\hfill \end{array}\right.\end{array}\end{array}$$

(13)

AdamCG-C3 (algorithm (12)) works well for all $n\in \mathbb{N}$, since it uses a constant learning rate. Meanwhile, there is a possibility that AdamCG-D1 (Algorithm (13)) does not work for a large number of iterations, because it uses diminishing learning rates. In fact, AdamCG-D1 (algorithm (13)) for a large n is as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}{\mathsf{G}}_n:=\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)-2^{-n}{\mathsf{G}}_{n-1}\approx \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\mathit{m}}_n:=2^{-n}{\mathit{m}}_{n-1}+(1-2^{-n}){\mathsf{G}}_n\approx {\mathsf{G}}_n\approx \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\widehat{\mathit{m}}}_n:={(1-{0.9}^{n+1})}^{-1}{\mathit{m}}_n,\hfill \\ {\mathit{v}}_n:=0.999{\mathit{v}}_{n-1}+(1-0.999)\mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n)\odot \mathsf{G}({\mathit{x}}_n,{\mathbf{\xi }}_n),\hfill \\ {\overline{\mathit{v}}}_n:={(1-{0.999}^{n+1})}^{-1}{\mathit{v}}_n,\hfill \\ {\widehat{\mathit{v}}}_n=\left({\widehat{v}}_{n,i}\right):=\left(max\{{\widehat{v}}_{n-1,i},{\overline{v}}_{n,i}\}\right),\hfill \\ {\mathsf{H}}_n:=\mathsf{diag}\left(\sqrt{{\widehat{v}}_{n,i}}\right),\hfill \\ {\mathit{x}}_{n+1}:=P_{X,{\mathsf{H}}_n}({\mathit{x}}_n-n^{-1/2}{\mathsf{H}}_n^{-1}{\mathit{m}}_n)\approx P_{X,{\mathsf{H}}_n}\left({\mathit{x}}_n\right)={\mathit{x}}_n,\hfill \end{array}\right.\end{array}\end{array}$$

(14)

which implies that algorithm (14) does not work. As can be seen in Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12, Algorithm 1 with diminishing learning rates would not be good for training neural networks.

Finally, let us compare the existing algorithm with Algorithm 1, in particular, AMSGrad in torch.optim using ${\alpha }_n={10}^{-3}$, ${\beta }_n=0.9$, and $\zeta =0.999$ with AMSGrad-C3 using ${\alpha }_n={10}^{-3}$, ${\beta }_n={10}^{-3}$, and $\zeta =0.999$. The difference between AMSGrad and AMSGrad-C3 is the setting of ${\beta }_n$. According to Figure 7, Figure 8 and Figure 9, AMSGrad-C3 performs comparably to AMSGrad, a useful algorithm. These results are guaranteed by Theorem 1, which indicates that Algorithm 1 with a small constant learning rate approximates a stationary point of the minimization problem in deep neural networks, and more specifically, the sequence ${\left({\mathit{x}}_n\right)}_{n\in \mathbb{N}}$ generated by AMSGrad-C3 (Algorithm 1 with $\delta =0$) satisfying

$$\begin{array}{c}\hfill \underset{n\to +\infty }{lim\; sup}\mathbb{E}\left[\langle \mathit{x}-{\mathit{x}}_n,\nabla f\left({\mathit{x}}_n\right)\rangle \right]\ge -\frac{{\tilde{B}}^2{\tilde{M}}^2}{2\tilde{b}}\frac{1}{{10}^3}-\frac{\sqrt{Dd}\tilde{M}}{\tilde{b}}\frac{1}{{10}^3}(\mathit{x}\in X)\end{array}$$

approximates ${\mathit{x}}^{🟉}\in X^{🟉}:=\{{\mathit{x}}^{🟉}\in X:\langle \mathit{x}-{\mathit{x}}^{🟉},\nabla f({\mathit{x}}^{🟉})\rangle \ge 0(\mathit{x}\in X)\}$.

6. Conclusions

We proposed an iterative algorithm with conjugate gradient-like directions for nonconvex optimization in deep neural networks to accelerate conventional adaptive learning rate optimization algorithms. We presented two convergence analyses of the algorithm. The first convergence analysis showed that the algorithm with a constant learning rate approximates a stationary point of a nonconvex optimization problem. The second analysis showed that the algorithm with a diminishing learning rate converges to a stationary point of the nonconvex optimization problem. We gave numerical results for concrete neural networks. The results showed that the proposed algorithm with a constant learning rate is superior for training neural networks from the viewpoints of theory and practice, while the proposed algorithm with a diminishing learning rate is not good for training neural networks. The reason behind these results is that using a constant learning rate guarantees that the algorithm works well, while a diminishing learning rate for a large number of iterations, which is approximately zero, implies that the algorithm is not updated.