Maximum Correntropy Criterion with Distributed Method

Abstract

The Maximum Correntropy Criterion (MCC) has recently triggered enormous research activities in engineering and machine learning communities since it is robust when faced with heavy-tailed noise or outliers in practice. This work is interested in distributed MCC algorithms, based on a divide-and-conquer strategy, which can deal with big data efficiently. By establishing minmax optimal error bounds, our results show that the averaging output function of this distributed algorithm can achieve comparable convergence rates to the algorithm processing the total data in one single machine.

1. Introduction

In the big data era, the rapid expansion of data generation brings data of prohibitive size and complexity. This brings challenges to many traditional learning algorithms requiring access to the whole data set. Distributed learning algorithms, based on the divide-and-conquer strategy, provide a simple and efficient way to address this issue and therefore have received increasing attention. Such a strategy starts with partitioning the big data set into multiple subsets that are distributed to local machines, then it obtains local estimators in each subset by using a base algorithm, and it finally pools the local estimators together by simple averaging. It can substantially cut the time and memory costs in the algorithm implementation, and in many practical applications its learning performance has shown to be as good as that of a big machine that can use all the data. This scheme has been developed in various learning contexts, including spectral algorithms [1,2], kernel ridge regression [3,4,5], gradient descent [6,7], a semi-supervised approach [8], minimum error entropy [9] and bias correction [10].

Regression estimation and inference play an important role in the fields of data mining and statistics. The traditional ordinary least squares (OLS) method provides an efficient estimator if the regression model error is normally distributed. However, heavy-tailed noise and outliers are common in the real world, which limits the application of OLS in practice. Various robust losses have been proposed to deal with the problem instead of least squares loss. The commonly used robust losses mainly include adaptive Huber loss [11], gain function [12], minimum error entropy [13], exponential squared loss [14], etc. Among them, the Maximum Correntropy Criterion (MCC) is widely employed as an efficient alternative to the ordinary least squares method which is suboptimal in the non-Gaussian and non-linear signal processing situations [15,16,17,18,19]. Recently, MCC has been studied extensively in the literature and is widely adopted for many learning tasks, e.g., wind power forecasting [20] and pattern recognition [19]. In this paper, we are interested in the implementation of MCC by a distributed gradient descent method in a big data setting. Note that the MCC loss function is non-convex, so its analysis is essentially different from the least squares method. A rigorous analysis of distributed MCC is necessary to derive the consistency and learning rates.

Given a hypothesis function $f:\mathcal{X}\to \mathcal{Y}$ and the scaling parameter $\sigma >0,$ correntropy between $f\left(X\right)$ and Y is defined by

$$V_{\sigma }\left(f\right):=\mathbb{E}\left[G\left(\frac{{(f\left(X\right)-Y)}^2}{2{\sigma }^2}\right)\right]$$

where $G\left(u\right)$ is the Gaussian function $exp\left\{-u\right\},u\in \mathbb{R}.$ Given the sample set $D={\left\{(x_i,y_i)\right\}}_{i=1}^N\subset \mathcal{Z}:=\mathcal{X}\times \mathcal{Y}$, the empirical form of $V_{\sigma }$ is

$${\widehat{V}}_{\sigma }\left(f\right):=\frac{1}{N}\sum _{i=1}^{N}G\left(\frac{{(f\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right).$$

The purpose of MCC is to maximize the empirical correntropy ${\widehat{V}}_{\sigma }$ over a hypothesis space $\mathcal{H}$, that is

$$\begin{array}{c}\hfill f_{\mathbf{z},\mathcal{H}}:=arg\underset{f\in \mathcal{H}}{max}{\widehat{V}}_{\sigma }\left(f\right).\end{array}$$

(1)

In the statistical learning context, the loss induced by correntropy ${\varphi }_{\sigma }:\mathbb{R}\to {\mathbb{R}}_{+}$ is defined as

$$\begin{array}{c}\hfill {\varphi }_{\sigma }\left(u\right):={\sigma }^2\left(1-G\left(\frac{u^2}{2{\sigma }^2}\right)\right)={\sigma }^2\left(1-exp\left\{-\frac{u^2}{2{\sigma }^2}\right\}\right),\end{array}$$

where $\sigma >0$ is the scaling parameter. The loss function can be viewed as a variant of the Welsch function [21] and the estimator $f_{\mathbf{z},\mathcal{H}}$ of (1) is also the minimizer of the empirical minimization risk scheme over $\mathcal{H}$, that is

$$\begin{array}{c}\hfill \underset{f\in \mathcal{H}}{min}\frac{1}{N}\sum _{i=1}^N{\varphi }_{\sigma }(f\left(x_i\right)-y_i).\end{array}$$

(2)

This paper aims at rigorous analysis of distributed gradient descent MCC within the framework of reproducing kernel Hilbert spaces (RKHSs). Let $K:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ be a Mercer kernel [22], i.e., a continuous, symmetric and positive semi-definite function. A kernel K is said to be positive semi-definite, if the matrix ${\left(K(u_i,u_j)\right)}_{i,j=1}^m$ is positive semi-definite for any finite set $\left\{u_1,\cdots ,u_m\right\}\subset \mathcal{X}$ and $m\in \mathbb{N}$. The RKHS ${\mathcal{H}}_K$ associated with the Mercer kernel K is defined to be the completion of the linear span of the set of functions $\{K_x:=K(x,·),x\in \mathcal{X}\}$ with the inner product ${\langle ·,·\rangle }_K$ given by ${\langle K_x,K_u\rangle }_K=K(x,u).$ It has the reproducing property

$$\begin{array}{c}\hfill f\left(x\right)={\langle f,K_x\rangle }_K\end{array}$$

(3)

for any $f\in {\mathcal{H}}_K$ and $x\in \mathcal{X}.$ Denote $\kappa :={sup}_{x\in \mathcal{X}}\sqrt{K(x,x)}.$ By the property (3), we get that

$$\begin{array}{c}\hfill {\parallel f\parallel }_{\infty }\le \kappa {\parallel f\parallel }_K,foranyf\in {\mathcal{H}}_K.\end{array}$$

(4)

Definition 1.

Given the sample set $D={\left\{(x_i,y_i)\right\}}_{i=1}^N\subset \mathcal{Z}:=\mathcal{X}\times \mathcal{Y}$, the kernel gradient descent algorithm for solving (2) can be stated iteratively with $f_{1,D}=0$ as

$$\begin{array}{c}\hfill f_{t+1,D}=f_{t,D}-\eta \times \frac{1}{N}\sum _{i=1}^N{\varphi }_{\sigma }^{\prime }\left((f_{t,D}\left(x_i\right)-y_i)\right)K_{x_i},t\ge 2\end{array}$$

(5)

where η is the of step size and ${\varphi }_{\sigma }^{\prime }\left((f_{t,D}\left(x_i\right)-y_i)\right)=G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)(f_{t,D}\left(x_i\right)-y_i).$

Divide-and-Conquer algorithm for the kernel gradient descent MCC (5) is easy to describe. Rather than performing on the whole N examples, the distributed algorithm executes the following three steps:

• Partition the data set D evenly and uniformly into m disjoint subsets $D_j$, $1\le j\le m$.
• Perform algorithm (5) on each data set $D_j$, and get the local estimate $f_{T+1,D_j}$ after T-th iteration.
• Take an average ${\overline{f}}_{T+1,D}=\frac{1}{m}{\sum }_{j=1}^m{f}_{T+1,D_j}$ as a final output.

In the next section, we study the asymptotic behavior of the final estimator ${\overline{f}}_{T+1,D}$ and show that ${\overline{f}}_{T+1,D}$ can obtain the minimax optimal rates over all estimators using the total data set of N samples provided that the scaling parameter $\sigma$ is chosen suitably.

2. Assumptions and Main Results

In the setting of non-parametric estimation, we denote X as the explanatory variable that takes values in a compact domain $\mathcal{X}$, $Y\in \mathcal{Y}\subset \mathbb{R}$ as a real-valued response variable. Let $\rho$ be the underlying distribution on $\mathcal{Z}:=\mathcal{X}\times \mathcal{Y}.$ Moreover, let ${\rho }_{\mathcal{X}}$ be the marginal distribution of $\rho$ on $\mathcal{X}$ and $\rho (·|x)$ be the conditional distribution on $\mathcal{Y}$ for given $x\in \mathcal{X}.$

This work focuses on the application of MCC in regression problems, which is linked to the additive noise model

$$\begin{array}{c}\hfill Y=f_{\rho }\left(X\right)+e,\mathbb{E}\left(e\right|X)=0,\end{array}$$

where e is the noise and $f_{\rho }\left(x\right)$ is the regression function, which is the conditional mean $\mathbb{E}\left(Y\right|X=x)$ for $X=x\in \mathcal{X}.$ The goal of this paper is to estimate the mean square error between ${\overline{f}}_{T+1,D}$ and $f_{\rho }$ in $L_{{\rho }_{\mathcal{X}}}^2$-metric, which is defined by ${\parallel ·\parallel }_{L_{{\rho }_{\mathcal{X}}}^2}:={\left({\int }_{\mathcal{X}}{|·|}^{2}d{\rho }_{\mathcal{X}}\right)}^{\frac{1}{2}}.$ For simplicity, we will use $\parallel ·\parallel$ to denote the norm ${\parallel ·\parallel }_{L_{{\rho }_{\mathcal{X}}}^2}$ when the meaning is clear from the context.

Below, we present two important assumptions, which play a vital role in carrying out the analysis. The first assumption is about the regularity of the target function $f_{\rho }$. Define the integral operator $L_K:L_{{\rho }_{\mathcal{X}}}^2\to L_{{\rho }_{\mathcal{X}}}^2$ associated with K by

$$\begin{array}{c}\hfill L_{K}f:={\int }_{\mathcal{X}}{\int }_{\mathcal{X}}f\left(x\right)K_{x}d{\rho }_{\mathcal{X}}\left(x\right),\forall f\in L_{{\rho }_{\mathcal{X}}}^2.\end{array}$$

As K is a Mercer kernel on the compact domain $\mathcal{X}$, the operator $L_K$ is hence compact and positive. So, $L_K^r$ as the r-th power of $L_K$ for $r>0$ is well defined. Our error bounds are stated in terms of the regularity of the target function $f_{\rho }$, given by [3,23]

$$\begin{array}{c}\hfill f_{\rho }=L_K^r\left(h_{\rho }\right),\mathrm{for}\mathrm{some}r>0\mathrm{and}{h}_{\rho }\in L_{{\rho }_{\mathcal{X}}}^2.\end{array}$$

(6)

The condition (6) measures the regularity of $f_{\rho }$ and is closely related to the smoothness of $f_{\rho }$ when ${\mathcal{H}}_K$ is a Sobolev space. If (6) holds with $r\ge \frac{1}{2}$, $f_{\rho }$ lies in the space ${\mathcal{H}}_K$.

The second assumption (7) is about the capacity of ${\mathcal{H}}_K$, measured by the effective dimension [24,25]

$$\begin{array}{c}\mathcal{N}\left(\lambda \right)=\mathsf{Trace}({(L_K+\lambda I)}^{-1}{L}_K),\mathrm{for}\lambda >0,\end{array}$$

where I is the identity operator on ${\mathcal{H}}_K$. In this paper, we assume that

$$\begin{array}{c}\mathcal{N}\left(\lambda \right)\le C{\lambda }^{-s}\mathrm{for}\mathrm{some}C>0\mathrm{and}0<s\le 1.\end{array}$$

(7)

Note that it always holds with $s=1$. For $0<s<1$, it is almost equivalent to that the eigenvalues ${\sigma }_i$ of $L_K$ decay at a rate $i^{-\frac{1}{s}}$. The smoother the kernel function K is, the smaller s and the smaller function space ${\mathcal{H}}_K.$ In particular, if K is a Gaussian kernel, then s can be arbitrarily close to 0, as $K\in C^{\infty }$.

Throughout the paper, we assume that ${\displaystyle \kappa :=\underset{x\in \mathcal{X}}{sup}\sqrt{K(x,x)}\le 1}$ and $\left|y\right|\le M$ for some $M>0$. We denote $\lfloor a\rfloor$ as the smallest integer not less than a.

Theorem 1.

Assume that (6) and (7) hold for some $r>\frac{1}{2}$ and $0<s\le 1$. Taking ${\eta }_t=\eta t^{-\theta }$ with $0<\eta \le 1$ and $0\le \theta <1.$ If $T=\lfloor N^{\frac{1}{(2r+s)(1-\theta )}}\rfloor$ and the number of partition of the data set D

$$\begin{array}{c}\hfill m\le \frac{N^{\frac{r-\frac{1}{2}}{2r+s}}}{{\left(logN\right)}^5},\end{array}$$

(8)

then with confidence at least $1-\delta$,

$$\begin{array}{c}\hfill \parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \le \tilde{C}\left\{N^{-\frac{r}{2r+s}}+N^{\frac{\frac{5}{2}}{2r+s}}{\sigma }^{-2}\right\}{\left(log\frac{12}{\delta }\right)}^4,\end{array}$$

where $\tilde{C}$ is a constant depending on $\theta .$

Remark 1.

The above theorem, to be proved in Section 3, exhibits the concrete learning rates of the distributed estimator ${\overline{f}}_{T+1,D}$ (hence the standard estimator of (5) with $m=1$). It implies that the kernel gradient descent for MCC on the single and distributed data set both achieves the learning rate $O\left(N^{-\frac{r}{2r+s}}\right)$ when σ is large enough. It equals the minimax optimal rates in the regression setting [24,26] in the case of $r>\frac{1}{2}$. This theorem suggests that the distributed MCC does not sacrifice the convergence rate provided that the partition number m satisfies the constraint (8). Thus, the distributed MCC estimator ${\overline{f}}_{T+1,D}$ enjoys both computational efficiency and statistical optimality.

With the help of Theorem 1, we can easily deduce the following optimal learning rate in expectation.

Corollary 1.

Assume that (6) and (7) hold for some $r>\frac{1}{2}$ and $0<s\le 1$, taking ${\eta }_t=\eta t^{-\theta }$ with $0<\eta \le 1$ and $0\le \theta <1.$ If $T=\lfloor N^{\frac{1}{(2r+s)(1-\theta )}}\rfloor$, m satisfies (8) and $\sigma \ge N^{\frac{r/2+5/4}{2r+s}}$, then we have

$$\begin{array}{c}\hfill \mathbb{E}\left[\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \right]=O\left(N^{-\frac{r}{2r+s}}\right).\end{array}$$

By the confidence-based error estimate in Theorem 1, we can obtain the following almost sure convergence of the distributed gradient descent algorithm for MCC.

Corollary 2.

Assume that (6) and (7) hold for some $r>\frac{1}{2}$ and $0<s\le 1$, taking ${\eta }_t=\eta t^{-\theta }$ with $0<\eta \le 1$ and $0\le \theta <1.$ If $T=\lfloor N^{\frac{1}{(2r+s)(1-\theta )}}\rfloor$, m satisfies (8) and $\sigma \ge N^{\frac{r/2+5/4}{2r+s}}$, and for arbitrary $ϵ>0$, we have

$$\begin{array}{c}\hfill \underset{N\to \infty }{lim}{N}^{\frac{r}{2r+s}-ϵ}\left[\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \right]=0.\end{array}$$

3. Discussion and Conclusions

In this work, we have studied the theoretical properties and convergence behaviors of a distributed kernel gradient descent MCC algorithm. As shown in Theorem 1, we derived minimax optimal error bounds for the distributed learning algorithm under the regularity condition on the regression function and capacity condition on RKHS. In the standard kernel gradient descent MCC algorithm ($m=1$), the aggregate time complexity is $O\left(t{N}^2\right)$ after t iterations. However, in the distributed case ($m>1$), the aggregate time complexity reduces to $O\left(t{N}^2/m\right)$ after t iterations. In conclusion, the kernel gradient descent MCC algorithm (5) with the distributed method can achieve fast convergence rates while successfully reducing algorithmic costs.

When the optimization problem arises from non-convex losses, the iteration sequence generated by the gradient descent algorithm is likely to only converge to a stationary point or a local minimizer. Note that the loss induced by correntropy ${\varphi }_{\sigma }$ is not convex. Then, the convergence of the gradient descent method (5) to the global minimizer is not unconditionally guaranteed, which brings difficulties to the mathematical analysis of convergence. Our work on Theorem 1 addresses this issue, which shows that the iterative algorithm ensures the global optimality of its iterations in the theoretical analysis.

For regression problems, the distributed method has been introduced to the iteration algorithm in various learning paradigms and the minimax optimal rate has been obtained under different constraints on the partition number m. For distributed spectral algorithms [1], the lower bound of m that ensures the optimal rates is

$$m\le N^{min\{\frac{2}{2r+s},\frac{2r-1}{2r+s}\}}.$$

(9)

We see from (9) that the restriction on m suffers from a saturation phenomenon in the sense that when $r\ge 3/2$ in the sense that the maximal m to guarantee the optimal learning rate does not improve as r is beyond $3/2.$ Our restriction in (8) is worse than (9) when $r<5/2$ but better when $r>5/2$ as the upper bound in (8) increases with respect to r that overcomes the saturation effect in (9). For distributed kernel gradient descent algorithms with least squares method [6] and minimum error entropy (MEE) principle [9], the restrictions of m are improved as

$$m\le \frac{N^{\frac{r-\frac{1}{2}}{2r+s}}}{{\left(logN\right)}^4+1}$$

(10)

and

$$m\le \frac{N^{\frac{r-\frac{1}{2}}{2r+s}}}{{\left(logN\right)}^5},$$

(11)

respectively. Our bound (8) for MCC differs with (10) for least squares only up to a logarithmic term, which has little impact on the upper bound of m ensuring optimal rates, but numerical experiments show that the distributed kernel gradient descent algorithm for least squares method is inferior to that for MCC in non-Gaussian noise models [15,27,28]. Our bound (8) is the same as (11) that is applied to the MEE principle. As we know, MEE also performs well in dealing with non-Gaussian noise or heavy-tail distribution [13,29]. However, MEE belongs to pairwise learning problems that work with pairs of samples rather than single sample in MCC. Hence, the distributed kernel gradient descent algorithm for MCC has an advantage over MEE in algorithmic complexity.

Several related questions are worthwhile for future research. First, our distributed result provides the optimal rates by requiring a large robust parameter $\sigma$. In practice, a moderate $\sigma$ may be enough to ensure a good learning performance in robust estimation as shown by [17]. It is therefore of interest to investigate the convergence properties of distributed version of algorithm (5) when $\sigma$ is chosen as a constant or $\sigma \left(N\right)\to 0$ as N approaches $\infty .$

Secondly, our algorithm is carried out in the framework of supervised learning; however, in numerous real-world applications, few labeled data are available, but a large amount of unlabeled data are given since the cost of labeling data is high such as time, money. Thus, we shall investigate how to enhance the learning performance of the MCC algorithm by the distributed method and the additional information given by unlabeled data.

Thirdly, as stated in Theorem 1, the choice of the last iteration T and the partition number m depends on the parameters $r,s$, which are usually unknown in advance. In practice, cross-validation is usually used to tune T and m adaptively. It would be interesting to know whether the kernel gradient descent MCC (5) with the distributed method can achieve the optimal convergence rate with adaptive T and $m.$

Last but not least, we should note that here that all the data $D={\left\{(x_i,y_i)\right\}}_{i=1}^N$ are drawn independently according to the same distribution. In the distributed method, we partition D evenly and uniformly into m disjoint subsets. This means that $|D_1|=\cdots =|D_m|=\frac{N}{m}$ and each sample $(x_i,y_i)$ is assigned to the subset $D_j$$(1\le j\le m)$ with the same probability. In the context of uniform random sampling, such randomness splitting strategy should be reasonable and practical. So, our theoretical analysis is based on the uniform random splitting mechanism. However, for the theoretical analysis of other randomness or non-randomness splitting mechanisms, it is necessary to develop new mathematical tools for optimal performance. It is beyond the scope of this paper and will be left for our future work.

4. Proofs of Main Results

This section is devoted to proving main results in Section 2. Here and in the following, let the sample size of each subset $D_1,\cdots ,D_m$ be n; that is, $D=D_1\bigcup \cdots D_m$ and $N=mn.$ Define the empirical operator $L_{K,D}$ on ${\mathcal{H}}_K$ as

$$\begin{array}{cc}\hfill L_{K,D}\left(f\right)& =\frac{1}{N}\sum _{i=1}^N{〈f,K_{x_i}〉}_K{K}_{x_i},\forall f\in {\mathcal{H}}_K,\hfill \end{array}$$

where $x_1\cdots ,x_N\in \left\{x:(x,y)\in Dwithsomey\in \mathcal{Y}\right\}$. Similarly, we can define the operator $L_{K,D_j}$ on ${\mathcal{H}}_K$ for each subset $D_j,1\le j\le m,$

$$\begin{array}{cc}\hfill L_{K,D_j}\left(f\right)& =\frac{1}{n}\sum _{i=1}^n{〈f,K_{x_i}〉}_K{K}_{x_i},\forall f\in {\mathcal{H}}_K,\hfill \end{array}$$

where $x_1\cdots ,x_n\in \left\{x:(x,y)\in D_{j}withsomey\in \mathcal{Y}\right\}$.

4.1. Preliminaries

We first introduce some necessary lemmas in the proofs, which can be found in [3,6,9].

Lemma 1.

Let $g\left(z\right)$ be a measurable function defined on $\mathcal{Z}$ with ${\parallel g\parallel }_{\infty }\le M^{\prime }$ almost definitely for some $M^{\prime }>0$. Let $0<\delta <1;$ then, each of the following estimates holds with confidence at least $1-\delta$,

$$\begin{array}{c}\hfill ∥{(L_K+\lambda I)}^{-\frac{1}{2}}(L_K-L_{K,D})∥\le 2{\mathcal{A}}_{D,\lambda }log\frac{2}{\delta },\end{array}$$

$$\begin{array}{c}\hfill ∥{(L_{K,D}+\lambda I)}^{-1}(L_K+\lambda I)∥\le 2{\left(\frac{2A_{D,\lambda }log\frac{2}{\delta }}{\sqrt{\lambda }}\right)}^2+2.\end{array}$$

and

$$\begin{array}{c}\hfill ∥\frac{1}{N}\sum _{i=1}^N{(L_K+\lambda I)}^{-\frac{1}{2}}\left[g\left(z_i\right)K_{x_i}-L_{K}g\right]∥\le 2M^{\prime }{\mathcal{A}}_{D,\lambda }log\frac{2}{\delta }\end{array}$$

where ${\mathcal{A}}_{D,\lambda }:=\frac{1}{N\sqrt{\lambda }}+\sqrt{\frac{\mathcal{N}\left(\lambda \right)}{N}}$.

Let ${\pi }_i^t$ denote the polynomial defined by ${\pi }_i^t\left(s\right)={\prod }_{j=i}^t(1-{\eta }_{j}x)$ if $i\le t$ and, for notation simplicity, let ${\pi }_{t+1}^t\left(s\right)=1$ be the identity function. In our proof, we need to deal with the polynomial operators ${\pi }_i^t\left(L_K\right)$ and ${\pi }_i^t\left(L_{K,D}\right)$. For this purpose we introduce the conventional notation ${\sum }_{j=T+1}^T:=1$ and the following preliminary lemmas.

Lemma 2.

If $0\le \alpha <1,0\le \theta <1$, then for $T\ge 3,$

$$\begin{array}{c}\hfill \sum _{i=1}^T{i}^{-(\theta +\alpha )}{\left(\sum _{j=i+1}^T{j}^{-\theta }\right)}^{-1}\le C_{\theta ,\alpha }{T}^{-min\{\alpha ,1-\theta \}}logT,\end{array}$$

(12)

where $C_{\theta ,\alpha }$ is a constant depending only on θ and α, whose value is given in the proof. In particular, if $\alpha =0$, we have

$$\begin{array}{c}\hfill \sum _{i=1}^T{i}^{-\theta }{\left(\sum _{j=i+1}^T{j}^{-\theta }\right)}^{-1}\le 15logT.\end{array}$$

(13)

Lemma 3.

If ${\eta }_t=\eta t^{-\theta }$ with $0<\eta <1$ and $0\le \theta <1,$ then for $1\le i\le T-1$,

$$\begin{array}{cc}\hfill & ∥{\pi }_i^t\left(L_{K,D}\right)∥\le 1\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & ∥{\pi }_i^t\left(L_K\right)∥\le 1\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & ∥L_{K,D}{\pi }_{i+1}^T\left(L_{K,D}\right)∥\le {\left(e\eta \sum _{j=i+1}^T{j}^{-\theta }\right)}^{-1},\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & ∥L_K{\pi }_{i+1}^T\left(L_K\right)∥\le {\left(e\eta \sum _{j=i+1}^T{j}^{-\theta }\right)}^{-1},\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & \parallel \sum _{i=1}^T{\eta }_i\left[(L_{K,D}+\lambda I){\pi }_{i+1}^T\left(L_{K,D}\right)\right]\parallel \le 1+\frac{\eta \lambda }{1-\theta }{T}^{1-\theta },\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & \parallel \sum _{i=1}^T{\eta }_i\left[(L_K+\lambda I){\pi }_{i+1}^T\left(L_K\right)\right]\parallel \le 1+\frac{\eta \lambda }{1-\theta }{T}^{1-\theta }.\hfill \end{array}$$

(19)

Define a data-free gradient descent sequence for the least square method in ${\mathcal{H}}_K$ by $f_1=0$ and

$$\begin{array}{c}\hfill f_{t+1}=f_t-{\eta }_t{\int }_{\mathcal{X}}\left(f_t\left(x\right)-f_{\rho }\left(x\right)\right)K_{x}d{\rho }_{\mathcal{X}}=(I-{\eta }_t{L}_K)f_t+{\eta }_t{L}_K{f}_{\rho }.\end{array}$$

(20)

It has been well evidence in the literature [30] that under the assumption (6) with $r>\frac{1}{2}$, there are

$$\begin{array}{c}\hfill \parallel f_t-f_{\rho }\parallel \le h_{\rho }{t}^{-r(1-\theta )}\end{array}$$

(21)

and

$$\begin{array}{c}\hfill \parallel f_t-f_{\rho }{\parallel }_K\le h_{\rho }{t}^{-(r-\frac{1}{2})(1-\theta )},\end{array}$$

(22)

where $h_{\rho }=max\left\{{\parallel g\parallel (2r/e)}^r,\parallel g\parallel {[(2r-1)/e]}^{r-\frac{1}{2}}\right\}$.

Lemma 4.

If ${\eta }_t=\eta t^{-\theta }$ with $0<\eta <1$ and $0\le \theta <1$, then there is a constant $C_{\rho ,\theta ,r}$ such that

$$\sum _{i=1}^T{\eta }_i\parallel L_{K,D}{\pi }_{i+1}^T\left(L_{K,D}\right)\parallel \parallel f_i-f_{\rho }{\parallel }_K\le C_{\rho ,\theta ,r}$$

(23)

and

$$\sum _{i=1}^T{\eta }_i\parallel L_K{\pi }_{i+1}^T\left(L_K\right)\parallel \parallel f_i-f_{\rho }{\parallel }_K\le C_{\rho ,\theta ,r}.$$

(24)

Lemma 5.

If ${\eta }_t=\eta t^{-\theta }$ with $0<\eta <1$ and $0\le \theta <1$, then there is a constant $D_{\rho ,\theta ,r}$ such that

$$\sum _{i=1}^T{\eta }_i{\parallel f_i-f_{\rho }\parallel }_K\le D_{\rho ,\theta ,r}{T}^{1-\theta }.$$

(25)

Recall that the isomorphism between ${\mathcal{H}}_K$ and $L_{{\rho }_{\mathcal{X}}}^2$, which yields in

$$\parallel f\parallel =\parallel L_K^{\frac{1}{2}}{f\parallel }_K\le {\parallel {(L_K+\lambda I)}^{\frac{1}{2}}f\parallel }_K,\mathrm{for}\mathrm{all}f\in {\mathcal{H}}_K.$$

(26)

4.2. Bound for the Learning Sequence

We will need the following bound for the learning sequence in the proof.

Theorem 2.

If the step size sequence ${\eta }_t=\eta t^{-\theta }$ with $0<\eta \le 1$ and $0\le \theta <1,$ then we have the following bound for the learning sequence $\left\{f_{t,D}\right\}$ by (5):

$$\begin{array}{c}\hfill \parallel f_{t,D}{\parallel }_K\le M{t}^{\frac{1-\theta }{2}}.\end{array}$$

(27)

Proof. 

We prove the statement by induction. First note the conclusion holds trivially for $t=1.$ Next, suppose that $\parallel f_{t,D}{\parallel }_K\le M\sqrt{{\sum }_{i=1}^{t-1}{\eta }_i}$ holds. By the updating rule (5) and the reproducing property, we have

$$\begin{array}{cc}\hfill \parallel f_{t+1,D}{\parallel }_K^2=& \parallel f_{t,D}{\parallel }_K^2-\frac{2{\eta }_t}{N}\sum _{i=1}^N{\varphi }_{\sigma }^{\prime }(f_{t,D}\left(x_i\right)-y_i)f_{t,D}\left(x_i\right)\hfill \\ \hfill & +\frac{{\eta }_t^2}{N^2}{∥\sum _{i=1}^N{\varphi }_{\sigma }^{\prime }(f_{t,D}\left(x_i\right)-y_i)K_{x_i}∥}_K^2\hfill \\ \hfill \le & \parallel f_{t,D}{\parallel }_K^2-\frac{2{\eta }_t}{N}\sum _{i=1}^N{\varphi }_{\sigma }^{\prime }(f_{t,D}\left(x_i\right)-y_i)f_{t,D}\left(x_i\right)\hfill \\ \hfill & +\frac{{\eta }_t^2}{N}\sum _{i=1}^N{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2{\left(f_{t,D}\left(x_i\right)-y_i\right)}^2\hfill \\ \hfill =& \parallel f_{t,D}{\parallel }_K^2+\frac{{\eta }_t}{N}\sum _{i=1}^N{Q}_i,\hfill \end{array}$$

(28)

where

$$\begin{array}{cc}\hfill Q_i=& \left[{\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2-2G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right]{\left(f_{t,D}\left(x_i\right)\right)}^2\hfill \\ \hfill & -2\left(G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)+{\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2\right)y_i{f}_{t,D}\left(x_i\right)\hfill \\ \hfill & +{\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2{y}_i^2.\hfill \end{array}$$

The restriction ${\eta }_t\le 1$ implies ${\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2-2G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)<0$. By the property of quadratic function, we have

$$\begin{array}{cc}\hfill Q_i\le & {\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2{y}_i^2-\frac{\left(-G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)+{\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2\right)y_i^2}{{\eta }_t{\left|G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right|}^2-2G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)}\hfill \\ \hfill =& \frac{G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)y_i^2}{2-{\eta }_{t}G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)}\le M^2.\hfill \end{array}$$

Plugging it into (28), we obtain

$$\begin{array}{c}\hfill \parallel f_{t+1,D}{\parallel }_K^2\le {\parallel f_{t,D}\parallel }_K^2+M^2{\eta }_t\le M^2\sum _{i=1}^t{\eta }_i=M^2\eta \sum _{i=1}^t{i}^{-\theta }\le M^2{t}^{1-\theta }.\end{array}$$

This completes the proof. □

4.3. Error Decomposition and Estimation of Error Bounds

Now we are in a position of bounding the error of the distributed kernel gradient descent MCC. For this purpose, we decompose the error $\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel$ into two parts as

$$\begin{array}{c}\hfill \parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \le \parallel f_{T+1}-f_{\rho }\parallel +\parallel {\overline{f}}_{T+1,D}-f_{T+1}\parallel .\end{array}$$

(29)

As we have mentioned in the previous subsection, the first term can be bounded by (21) under the assumption (6) with $r>\frac{1}{2}$. Our key analysis is the second term, which can be bounded with the help of the following proposition.

Proposition 1.

Assume that (6) holds for some $r>\frac{1}{2}.$ Let ${\eta }_t=\eta t^{-\theta }$ with $0<\eta \le 1$ and $0\le \theta <1.$ For $\lambda >0$, there holds

$$\begin{array}{cc}\hfill \parallel f_{T+1,D}-f_{T+1}\parallel & \le C_{r,\theta }^{\prime }\left[{\mathcal{B}}_{D,\lambda }({\mathcal{C}}_{D,\lambda }+{\mathcal{G}}_{D,\lambda })(1+\lambda T^{1-\theta })+T^{\frac{5(1-\theta )}{2}}{\sigma }^{-2}\right],\hfill \end{array}$$

(30)

and

$$\begin{array}{cc}\hfill \parallel f_{T+1,D}-f_{T+1}{\parallel }_K& \le C_{r,\theta }^{\prime }\left[{\mathcal{B}}_{D,\lambda }({\mathcal{C}}_{D,\lambda }+{\mathcal{G}}_{D,\lambda })(1+\lambda T^{1-\theta })/\sqrt{\lambda }+T^{\frac{5(1-\theta )}{2}}{\sigma }^{-2}\right],\hfill \end{array}$$

(31)

where

$$\begin{array}{cc}\hfill {\mathcal{B}}_{D,\lambda }=& \parallel {(L_{K,D}+\lambda I)}^{-1}(L_K+\lambda I)\parallel ,\hfill \\ \hfill {\mathcal{C}}_{D,\lambda }=& \parallel {(L_K+\lambda I)}^{-\frac{1}{2}}(L_K-L_{K,D})\parallel ,\hfill \\ \hfill {\mathcal{G}}_{D,\lambda }=& \parallel {(L_K+\lambda I)}^{-\frac{1}{2}}(L_K{f}_{\rho }-{\widehat{f}}_{\rho ,D}){\parallel }_K,\hfill \\ \hfill {\widehat{f}}_{\rho ,D}=& \frac{1}{N}\sum _{i=1}^N{y}_i{K}_{x_i}=\frac{1}{N}\sum _{(x,y)\in D}y{K}_x,\hfill \end{array}$$

(32)

and $C_{r,\theta }^{\prime }$ is given in the proof, depending on $r,\theta$.

Proof. 

By the definition of $f_{t,D}$ in (5) and the definition of $f_t$ in (20), we have

$$\begin{array}{c}\hfill f_{t+1,D}-f_{t+1}=[I-{\eta }_t{L}_{K,D}](f_{t,D}-f_t)+{\eta }_t[L_K-L_{K,D}]f_t+{\eta }_t[{\widehat{f}}_{\rho ,D}-L_K\left(f_{\rho }\right)]+{\eta }_t{E}_{t,D},\end{array}$$

(33)

where ${\widehat{f}}_{\rho ,D}$ is defined in (32) and

$$\begin{array}{cc}\hfill & E_{t,D}=\frac{1}{N}\sum _{i=1}^N\left(1-G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)\right)\left(f_{t,D}\left(x_i\right)-y_i\right)K_{x_i},\hfill \end{array}$$

Applying (33) iteratively from $t=1$ to $T,$ we obtain

$$f_{T+1,D}-f_{T+1}=I_1+I_2+I_3+I_4$$

(34)

where

$$\begin{array}{cc}\hfill I_1=& \sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_{K,D}\right)[L_K-L_{K,D}](f_i-f_{\rho }),\hfill \\ \hfill I_2=& \sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_{K,D}\right)[L_K-L_{K,D}]\left(f_{\rho }\right),\hfill \\ \hfill I_3=& \sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_{K,D}\right)[{\widehat{f}}_{\rho ,D}-L_K\left(f_{\rho }\right)],\hfill \\ \hfill I_4=& \sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_{K,D}\right)E_{i,D}.\hfill \end{array}$$

For $I_1,$ by (26), Lemmas 4 and 5,

$$\begin{array}{cc}\hfill \parallel I_1\parallel =& {∥\sum _{i=1}^T{\eta }_i{(L_K+\lambda I)}^{\frac{1}{2}}{\pi }_{i+1}^T\left(L_{K,D}\right)[L_K-L_{K,D}](f_i-f_{\rho })∥}_K\hfill \\ \hfill \le & \sum _{i=1}^T\{{\eta }_i∥{(L_K+\lambda I)}^{\frac{1}{2}}{(L_{K,D}+\lambda I)}^{-\frac{1}{2}}∥∥(L_{K,D}+\lambda I){\pi }_{i+1}^T\left(L_{K,D}\right)∥\hfill \\ \hfill & \times ∥{(L_{K,D}+\lambda I)}^{-\frac{1}{2}}{(L_K+\lambda I)}^{\frac{1}{2}}∥∥{(L_K+\lambda I)}^{-\frac{1}{2}}[L_K-L_{K,D}]∥{\parallel f_i-f_{\rho }\parallel }_K\}\hfill \\ \hfill \le & {\mathcal{B}}_{D,\lambda }{\mathcal{C}}_{D,\lambda }\left(\sum _{i=1}^T{\eta }_i\parallel L_{K,D}{\pi }_{i+1}^T\left(L_{K,D}\right)\parallel \parallel f_i-f_{\rho }{\parallel }_K+\lambda \sum _{i=1}^T{\eta }_i{\parallel f_i-f_{\rho }\parallel }_K\right)\hfill \\ \hfill \le & {\mathcal{B}}_{D,\lambda }{\mathcal{C}}_{D,\lambda }\left(C_{\rho ,\theta ,r}+D_{\rho ,\theta ,r}\lambda T^{1-\theta }\right).\hfill \end{array}$$

(35)

For $I_2$, by (26), Lemma 3, and the fact $\parallel f_{\rho }{\parallel }_{\infty }\le M$, we have

$$\begin{array}{cc}\hfill \parallel I_2\parallel =& ∥\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_{K,D}\right)[L_K-L_{K,D}]\left(f_{\rho }\right)∥\hfill \\ \hfill \le & {∥\sum _{i=1}^T{\eta }_i{(L_K+\lambda I)}^{\frac{1}{2}}{\pi }_{i+1}^T\left(L_{K,D}\right)[L_K-L_{K,D}]\left(f_{\rho }\right)∥}_K\hfill \\ \hfill \le & ∥{(L_K+\lambda I)}^{\frac{1}{2}}{(L_{K,D}+\lambda I)}^{-\frac{1}{2}}∥∥\sum _{i=1}^T{\eta }_i(L_{K,D}+\lambda I){\pi }_{i+1}^T\left(L_{K,D}\right)∥\hfill \\ \hfill & \times ∥{(L_{K,D}+\lambda I)}^{-\frac{1}{2}}{(L_K+\lambda I)}^{\frac{1}{2}}∥∥{(L_K+\lambda I)}^{-\frac{1}{2}}[L_K-L_{K,D}]∥{∥f_{\rho }∥}_K\hfill \\ \hfill \le & M\left(1+\frac{\lambda T^{1-\theta }}{1-\theta }\right){\mathcal{B}}_{D,\lambda }{\mathcal{C}}_{D,\lambda }.\hfill \end{array}$$

(36)

Similarly, we can bound $I_3$ as

$$\begin{array}{c}\hfill I_3\le \left(1+\frac{\lambda T^{1-\theta }}{1-\theta }\right){\mathcal{B}}_{D,\lambda }{\mathcal{G}}_{D,\lambda }.\end{array}$$

(37)

For $I_4,$ first note that by the bound (27) of $\left\{f_{t,D}\right\}$, we see

$$\begin{array}{cc}\hfill & {∥\left(G\left(\frac{{(f_{t,D}\left(x_i\right)-y_i)}^2}{2{\sigma }^2}\right)-1\right)(f_{t,D}\left(x_i\right)-y_i)K_{x_i}∥}_K\hfill \\ \hfill \le & \frac{(M+\parallel f_{t,D}{{\parallel }_K)}^3}{2{\sigma }^2}\le \frac{2^2}{{\sigma }^2}{\parallel f_{t,D}\parallel }_K^3\hfill \\ \hfill \le & 2^2{M}^3{t}^{\frac{3(1-\theta )}{2}}{\sigma }^{-2}\hfill \end{array}$$

This implies that

$$\begin{array}{c}\hfill \parallel E_{t,D}{\parallel }_K\le 2^2{M}^3{t}^{\frac{(1-\theta )\left(3\right)}{2}}{\sigma }^{-2}.\end{array}$$

(38)

Thistogether with the estimate $\parallel {\pi }_{i+1}^t\left(L_{K,D}\right)\parallel \le 1$ gives

$$\begin{array}{cc}\hfill \parallel I_4\parallel & \le \sum _{i=1}^T{\eta }_i{\parallel E_{i,D}\parallel }_K\le 2^2{M}^3\eta \sum _{i=1}^T{i}^{\frac{3(1-\theta )}{2}-\theta }{\sigma }^{-2}\hfill \\ \hfill & \le \frac{2^2{M}^3}{(1-\theta )\left(\frac{5}{2}\right)}{T}^{\frac{5(1-\theta )}{2}}{\sigma }^{-2}.\hfill \end{array}$$

(39)

Combining the estimates in (36), (37), (39) and (35), we obtain (30) holds with

$$C_{r,\theta }^{\prime }=C_{\rho ,\theta ,r}+D_{\rho ,\theta ,r}+\frac{2M}{1-\theta }+\frac{2^3{M}^3}{5(1-\theta )}.$$

Following a similar process we can obtain the bound in (31). □

The following theorem provides a bound for the second term in (29).

Theorem 3.

Take $\lambda =T^{-(1-\theta )}.$ There is a constant $C_{r,\theta }^{″}$ such that

$$\begin{array}{cc}\hfill \parallel {\overline{f}}_{T+1,D}-f_{T+1}\parallel \le & C_{r,\theta }^{″}[{\mathcal{G}}_{D,\lambda }+{\mathcal{C}}_{D,\lambda }+{\lambda }^{-\frac{1}{2}}logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })\hfill \\ \hfill & +{\sigma }^{-2}{T}^{\frac{5(1-\theta )}{2}}\left(1+logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }\right)].\hfill \end{array}$$

(40)

Proof. 

For each subset $D_l$ and each $1\le t\le T$, we have

$$f_{T+1,D_l}-f_{T+1}=[I-{\eta }_t{L}_K](f_{T,D_l}-f_t)+{\eta }_T[L_K-L_{K,D}]f_{T,D_l}+{\eta }_T[{\widehat{f}}_{\rho ,D_l}-L_K\left(f_{\rho }\right)]+{\eta }_T{E}_{T,D_l}.$$

This implies that

$$\begin{array}{cc}\hfill f_{T+1,D_l}-f_{T+1}=& \sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)[L_K-L_{K,D_l}]f_{i,D_l}\hfill \\ \hfill & +\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)[{\widehat{f}}_{\rho ,D_l}-L_K\left(f_{\rho }\right)]+\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)E_{i,D_l},\hfill \end{array}$$

and therefore

$$\begin{array}{cc}\hfill \parallel {\overline{f}}_{T+1,D}-f_{T+1}\parallel =& ∥\frac{1}{m}\sum _{l=1}^m\left(f_{T+1,D_l}-f_{T+1}\right)∥\hfill \\ \hfill \le & ∥\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)\frac{1}{m}\sum _{l=1}^m[L_K-L_{K,D_l}]f_{i,D_l}∥\hfill \\ \hfill & +∥\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)\frac{1}{m}\sum _{l=1}^m[{\widehat{f}}_{\rho ,D_l}-L_K\left(f_{\rho }\right)]∥\hfill \\ \hfill & +∥\frac{1}{m}\sum _{l=1}^m\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)E_{i,D_l}∥\hfill \\ \hfill :=& J_1+J_2+J_3.\hfill \end{array}$$

We first estimate $J_2.$ By (26), Lemma 3, and the choice $\lambda =T^{-(1-\theta )}$, we obtain

$$\begin{array}{cc}\hfill J_2\le & {∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)\frac{1}{m}\sum _{l=1}^m{(L_K+\lambda )}^{-\frac{1}{2}}[{\widehat{f}}_{\rho ,D_l}-L_K\left(f_{\rho }\right)]∥}_K\hfill \\ \hfill \le & \left(1+\frac{\lambda T^{1-\theta }}{1-\theta }\right){∥{(L_K+\lambda )}^{-\frac{1}{2}}[{\widehat{f}}_{\rho ,D}-L_K\left(f_{\rho }\right)]∥}_K\hfill \\ \hfill \le & \frac{2M}{1-\theta }\left(1+\lambda T^{1-\theta }\right){\mathcal{G}}_{D,\lambda }\hfill \\ \hfill :=& \frac{4M}{1-\theta }{\mathcal{G}}_{D,\lambda }.\hfill \end{array}$$

(41)

For $J_3,$ by (39) we have

$$\begin{array}{c}\hfill J_3\le \underset{1\le l\le m}{sup}∥\sum _{i=1}^T{\eta }_i{\pi }_{i+1}^T\left(L_K\right)E_{i,D_l}∥\le \frac{2^3{M}^3\eta }{5(1-\theta )}{T}^{5(1-\theta )/2}{\sigma }^{-2}.\end{array}$$

(42)

The estimation of $J_1$ is much more complicated. We decompose it into three parts,

$$\begin{array}{cc}\hfill J_1\le & {∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)\frac{1}{m}\sum _{l=1}^m{(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D_l}]f_{i,D_l}∥}_K\hfill \\ \hfill \le & {∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)\frac{1}{m}\sum _{l=1}^m{(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D_l}](f_{i,D_l}-f_i)∥}_K\hfill \\ \hfill & +{∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right){(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D}](f_i-f_{\rho })∥}_K\hfill \\ \hfill & +{∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right){(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D}]\left(f_{\rho }\right)∥}_K\hfill \\ \hfill :=& J_{11}+J_{12}+J_{13}.\hfill \end{array}$$

By Lemmas 4 and 5 and the fact $\lambda T^{1-\theta }=1$, we obtain

$$\begin{array}{cc}\hfill J_{12}& \le {\mathcal{C}}_{D,\lambda }\left(\sum _{i=1}^T∥{\eta }_i{L}_K{\pi }_{i+1}^T\left(L_K\right)∥\parallel f_i-f_{\rho }{\parallel }_K+\lambda \sum _{i=1}^T{\eta }_i{\parallel f_i-f_{\rho }\parallel }_K\right)\hfill \\ \hfill & \le {\mathcal{C}}_{D,\lambda }\left(C_{\rho ,\theta ,r}+D_{\rho ,\theta ,r}\right).\hfill \end{array}$$

For $J_{13}$, by (19) we have

$$\begin{array}{cc}\hfill J_{13}\le & ∥\sum _{i=1}^T{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)∥∥{(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D}]∥{∥f_{\rho }∥}_K\hfill \\ \hfill \le & M\left(1+\frac{\lambda T^{1-\theta }}{1-\theta }\right){\mathcal{C}}_{D,\lambda }=\frac{2M}{1-\theta }{\mathcal{C}}_{D,\lambda }.\hfill \end{array}$$

Now we turn to $J_{11}.$ We have

$$\begin{array}{cc}\hfill J_{11}\le & \sum _{i=1}^T∥{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)∥{∥\frac{1}{m}\sum _{l=1}^m{(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D_l}](f_{i,D_l}-f_i)∥}_K\hfill \\ \hfill \le & \sum _{i=1}^T∥{\eta }_i(L_K+\lambda ){\pi }_{i+1}^T\left(L_K\right)∥\underset{1\le l\le m}{sup}{∥{(L_K+\lambda )}^{-\frac{1}{2}}[L_K-L_{K,D_l}](f_{i,D_l}-f_i)∥}_K\hfill \\ \hfill \le & \sum _{i=1}^T{\eta }_i\left[{\left(\sum _{j=i+1}^T{\eta }_j\right)}^{-1}+\lambda \right]\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }{∥f_{i,D_l}-f_i∥}_K.\hfill \end{array}$$

(43)

By Theorem 1 and the choice $\lambda =T^{-(1-\theta )}$, for $1\le i\le T$, there holds that $\lambda i^{(1-\theta )}\le 1$ and

$$\begin{array}{cc}\hfill {∥f_{i,D_l}-f_i∥}_K& \le C_{r,\theta }^{\prime }\left[{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })(1+\lambda i^{1-\theta })/\sqrt{\lambda }+i^{\frac{5(1-\theta }{2})}{\sigma }^{-2}\right]\hfill \\ \hfill & \le C_{r,\theta }^{\prime }\left[2{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })/\sqrt{\lambda }+T^{\left(\frac{5(1-\theta )}{2}\right)}{\sigma }^{-2}\right].\hfill \end{array}$$

Plugging it into (43), we obtain

$$\begin{array}{cc}\hfill J_{11}& \le C_{r,\theta }^{\prime }\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }\left[2{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })/\sqrt{\lambda }+T^{\left(\frac{5(1-\theta )}{2}\right)}{\sigma }^{-2}\right]\sum _{i=1}^T{\eta }_i\left[{\left(\sum _{j=i+1}^T{\eta }_j\right)}^{-1}+\lambda \right].\hfill \end{array}$$

From Lemma 2, we see that

$$\sum _{i=1}^T{\eta }_i\left[{\left(\sum _{j=i+1}^T{\eta }_j\right)}^{-1}+\lambda \right]\le 15logT+\frac{\eta \lambda T^{1-\theta }}{1-\theta }=15logT+\frac{1}{1-\theta }\le \left(15+\frac{1}{1-\theta }\right)logT.$$

So, we have

$$J_{11}\le C_{r,\theta }^{\prime }\left(15+\frac{1}{1-\theta }\right)logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }\left[2{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })/\sqrt{\lambda }+T^{\left(\frac{5(1-\theta )}{2}\right)}{\sigma }^{-2}\right].$$

Combining the estimations for $J_{11},$$J_{12}$ and $J_{13}$, we obtain

$$\begin{array}{cc}\hfill J_1\le & \left(\frac{2M}{1-\theta }+C_{\rho ,\theta ,r}+D_{\rho ,\theta ,r}\right){\mathcal{C}}_{D,\lambda }\hfill \\ \hfill & +2C_{r,\theta }^{\prime }\left(15+\frac{1}{1-\theta }\right){\lambda }^{-\frac{1}{2}}logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })\hfill \\ \hfill & +C_{r,\theta }^{\prime }\left(15+\frac{1}{1-\theta }\right){\sigma }^{-2}{T}^{\left(\frac{5(1-\theta )}{2}\right)}logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }.\hfill \end{array}$$

(44)

Now the desired bound for $\parallel {\overline{f}}_{T+1,D}-f_{T+1}\parallel$ in (40) follows by combining the estimations for $J_1$, $J_2$, and $J_3$ and the constant is given by

$$C_{r,\theta }^{″}:=\left(\frac{2M\theta }{1-\theta }+C_{\rho ,\theta ,r}+D_{\rho ,\theta ,r}\right)+3C_{r,\theta }^{\prime }\left(15+\frac{1}{1-\theta }\right)+\frac{2^3{M}^3\eta }{5(1-\theta )}.$$

This proves the theorem. □

4.4. Proofs

Now we can prove Theorem 1.

Proof. 

Firstly, note that with the choice $T=\lfloor N^{\frac{1}{(2r+s)(1-\theta )}}\rfloor$ and $\lambda =T^{-(1-\theta )}$, and under the restriction (8) on m, we have

$$\begin{array}{cc}\hfill {\mathcal{A}}_{D,\lambda }& \le N^{-1+\frac{1}{4r+2s}}+\sqrt{C}{N}^{-\frac{1}{2}+\frac{s}{4r+2s}}\le (\sqrt{C}+1)N^{-\frac{r}{2r+s}}.\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill {\mathcal{A}}_{D_l,\lambda }& \le m{N}^{-1}{N}^{\frac{1}{4r+2s}}+\sqrt{C}{m}^{\frac{1}{2}}{N}^{-\frac{1}{2}}{N}^{\frac{s}{4r+2s}}\hfill \\ \hfill & \le (1+\sqrt{C})m^{\frac{1}{2}}{N}^{-\frac{r}{2r+s}}\hfill \end{array}$$

and

$$\begin{array}{c}\hfill \frac{{\mathcal{A}}_{D_l,\lambda }}{\sqrt{\lambda }}\le (1+\sqrt{C})m^{\frac{1}{2}}{N}^{-\frac{r}{2r+s}}{N}^{\frac{1}{4r+2s}}\le (1+\sqrt{C}).\end{array}$$

By applying Lemma 1, for any $1\le l\le m,$ we have with confidence at least $1-\frac{\delta }{6m}$,

$$\begin{array}{c}\hfill {\mathcal{B}}_{D_l,\lambda }\le 2{\left(\frac{2{\mathcal{A}}_{D_l,\lambda }log\frac{12m}{\delta }}{\sqrt{\lambda }}\right)}^2+2,{\mathcal{C}}_{D_l,\lambda }\le 2{\mathcal{A}}_{D_l,\lambda }log\frac{12m}{\delta },{\mathcal{G}}_{D_l,\lambda }\le 4{\mathcal{A}}_{D_l,\lambda }Mlog\frac{12m}{\delta }.\end{array}$$

Consequently, these bounds hold simultaneously with confidence at least $1-\frac{\delta }{2}.$ This implies that with confidence at least $1-\frac{\delta }{2}$, there holds

$$\begin{array}{cc}\hfill & {\lambda }^{-\frac{1}{2}}logT\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }{\mathcal{B}}_{D_l,\lambda }({\mathcal{C}}_{D_l,\lambda }+{\mathcal{G}}_{D_l,\lambda })\hfill \\ \hfill \le & 2^6(M+1)logT\left[{\left(\frac{{\mathcal{A}}_{D_l,\lambda }}{\sqrt{\lambda }}\right)}^2+1\right]\frac{{\mathcal{A}}_{D_l,\lambda }^2}{\sqrt{\lambda }}{\left(log\frac{12m}{\delta }\right)}^4\hfill \\ \hfill \le & 2^6(M+1){\left[{\left(1+\sqrt{C}\right)}^2+1\right]}^{2}m{N}^{-\frac{2r-\frac{1}{2}}{2r+s}}logT{\left(log\frac{12m}{\delta }\right)}^4\hfill \\ \hfill \le & 2^{10}(M+1){\left[{\left(1+\sqrt{C}\right)}^2+1\right]}^{2}m{N}^{-\frac{2r-\frac{1}{2}}{2r+s}}logT{\left(logm\right)}^4{\left(log\frac{12}{\delta }\right)}^4\hfill \\ \hfill \le & \frac{2^{10}(M+1){\left[{\left(1+\sqrt{C}\right)}^2+1\right]}^2}{(2r+s)(1-\theta )}m{N}^{-\frac{2r-\frac{1}{2}}{2r+s}}{\left(logN\right)}^5{\left(log\frac{12}{\delta }\right)}^4\hfill \\ \hfill \le & \frac{2^{10}(M+1){\left[{\left(1+\sqrt{C}\right)}^2+1\right]}^2}{(2r+s)(1-\theta )}{N}^{-\frac{r}{2r+s}}{\left(log\frac{12}{\delta }\right)}^4\hfill \end{array}$$

(45)

and

$$\begin{array}{cc}\hfill & {\sigma }^{-2}{T}^{\frac{5(1-\theta )}{2}}\left(1+(logT)\underset{1\le l\le m}{sup}{\mathcal{C}}_{D_l,\lambda }\right)\hfill \\ \hfill \le & {\sigma }^{-2}{T}^{\frac{5(1-\theta )}{2}}\left(1+(logT){\mathcal{A}}_{D_l,\lambda }log\frac{12m}{\delta }\right)\hfill \\ \hfill \le & 2{\sigma }^{-2}{N}^{\frac{\frac{5}{2}}{2r+s}}\left(1+\frac{2+2\sqrt{C}}{(2r+s)(1-\theta )}(logN)m^{\frac{1}{2}}{N}^{-\frac{r}{2r+s}}logmlog\frac{12}{\delta }\right)\hfill \\ \hfill \le & 2{\sigma }^{-2}{N}^{\frac{\frac{5}{2}}{2r+s}}\left(1+\frac{2+2\sqrt{C}}{(2r+s)(1-\theta )}{m}^{\frac{1}{2}}{N}^{-\frac{r}{2r+s}}{(logN)}^{2}log\frac{12}{\delta }\right)\hfill \\ \hfill \le & 2{\sigma }^{-2}{N}^{\frac{\frac{5}{2}}{2r+s}}\left(1+\frac{2+2\sqrt{C}}{(2r+s)(1-\theta )}\right)log\frac{12}{\delta }.\hfill \end{array}$$

(46)

By Lemma 1, we have with confidence at least $1-\frac{\delta }{4}$,

$$\begin{array}{c}\hfill {\mathcal{C}}_{D,\lambda }\le 2{\mathcal{A}}_{D,\lambda }log\frac{8}{\delta }\le 2(\sqrt{C}+1)N^{-\frac{r}{2r+s}}log\frac{12}{\delta }\end{array}$$

(47)

and

$${\mathcal{G}}_{D,\lambda }\le 2M{\mathcal{A}}_{D,\lambda }log\frac{8}{\delta }\le 2M(\sqrt{C}+1)N^{-\frac{r}{2r+s}}log\frac{12}{\delta }.$$

(48)

Plugging the estimates (45)–(48) into (40), we obtain with confidence at least $1-\delta ,$

$$\begin{array}{cc}\hfill \parallel {\widehat{f}}_{T+1,D}-f_{T+1}\parallel & \le C\left(N^{-\frac{r}{2r+s}}+{\sigma }^{-2}{N}^{\frac{\frac{5}{2}}{2r+s}}\right){\left(log\frac{12}{\delta }\right)}^4\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill C& =C_{r,\theta ,p}^{″}[2M(\sqrt{C_1}+1)+2(\sqrt{C}+1)\hfill \\ \hfill & +\frac{2^{10}(M+1){\left[{\left(1+\sqrt{C}\right)}^2+1\right]}^2}{(2r+s)(1-\theta )}+2\left(1+\frac{2+2\sqrt{C}}{(2r+s)(1-\theta )}\right)].\hfill \end{array}$$

This, together with the bound

$$\parallel f_{T+1}-f_{\rho }\parallel \le h_{\rho }{T}^{-r(1-\theta )}\le h_{\rho }{N}^{-\frac{r}{2r+s}},$$

leads to the desired conclusion with $\tilde{C}=C+h_{\rho }$. □

Proof of Corollary 1. 

When $\sigma \ge N^{\frac{r/2+5/4}{2r+s}}$, by Theorem 1, we have that with confidence at least $1-\delta ,$ $\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \le 2\tilde{C}{N}^{-\frac{r}{2r+s}}{\left(log\frac{12}{\delta }\right)}^4$. Replacing $2\tilde{C}{N}^{-\frac{r}{2r+s}}{\left(log\frac{12}{\delta }\right)}^4$ by t, then

$$\begin{array}{c}\hfill \mathbf{Prob}\left\{D:\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \ge t\right\}\le 12exp\left\{-{\left(2\tilde{C}\right)}^{-\frac{1}{4}}{N}^{\frac{r}{4(2r+p)}}{t}^{\frac{1}{4}}\right\}.\end{array}$$

Using the probability to expectation formula

$$\begin{array}{c}\hfill \mathbb{E}\left[\xi \right]={\int }_0^{\infty }Pr\{\xi \ge t\}dt\end{array}$$

with $\xi =\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel ,$ we have

$$\begin{array}{ccc}\hfill \mathbb{E}\left[\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \right]& & ={\int }_0^{\infty }\mathbf{Prob}\left\{D:\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \ge t\right\}dt\le 12{\int }_0^{\infty }exp\left\{-{\left(2\tilde{C}\right)}^{-\frac{1}{4}}{N}^{\frac{r}{4(2r+p)}}{t}^{\frac{1}{4}}\right\}dt\hfill \\ & & =324\tilde{C}{N}^{-\frac{r}{2r+s}}{\int }_0^{\infty }{u}^3{e}^{-u}du=324\tilde{C}\Gamma \left(4\right)N^{-\frac{r}{2r+s}},\hfill \end{array}$$

where $\Gamma \left(d\right)$ is the Gamma function defined for $u>0$ by $\Gamma \left(d\right)={\int }_0^{\infty }{u}^{d-1}{e}^{-u}du$.

The proof is complete. □

To prove Corollary 2, we need the following Borel-Cantelli Lemma which is provided in [31].

Lemma 6.

Let $\left\{a_N\right\}$ be a sequence of events in some probability space and $\left\{{\xi }_N\right\}$ be a sequence of positive numbers satisfying ${lim}_{N\to \infty }{\xi }_N=0$. If

$$\begin{array}{c}\hfill \sum _{N=1}^{\infty }\mathbf{Prob}\left\{|a_N-a|>{\xi }_N\right\}<\infty ,\end{array}$$

then $a_N$ will almost certainly converge to a.

Proof of Corollary 2. 

Let $\delta =N^{-2}$ in Theorem 1; then we have

$$\begin{array}{c}\hfill \mathbf{Prob}\left\{N^{\frac{r}{2r+s}}{\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel }_{\rho }\ge 2^5\tilde{C}\left(log12N\right)\right\}<N^{-2}.\end{array}$$

Thus, for any $ϵ>0$,

$$\begin{array}{c}\hfill \mathbf{Prob}\left\{N^{\frac{r}{2r+s}-ϵ}\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel \ge 2^5\tilde{C}\left(log12N\right)N^{-ϵ}\right\}<N^{-2}.\end{array}$$

Applying Lemma 6 with $a_N=N^{\frac{r}{2r+s}-ϵ}\parallel {\overline{f}}_{T+1,D}-f_{\rho }\parallel$, $a=0$ and ${\xi }_N=2^5\tilde{C}\left(log12N\right)N^{-ϵ}$, we can obtain the conclusion of Corollary 2 by noting ${lim}_{N\to \infty }{\xi }_N=0$ and ${\sum }_{N=1}^{\infty }{N}^{-2}<\infty .$

The proof is finished. □