A Modified Gradient Method for Distributionally Robust Logistic Regression over the Wasserstein Ball

Abstract

In this paper, a modified conjugate gradient method under the forward-backward splitting framework is proposed to further improve the numerical efficiency for solving the distributionally robust Logistic regression model over the Wasserstein ball, which comprises two phases: in the first phase, a conjugate gradient descent step is performed, and in the second phase, an instantaneous optimization problem is formulated and solved with a trade-off minimization of the regularization term, while simultaneously staying in close proximity to the interim point obtained in the first phase. The modified conjugate gradient method is proven to attain the optimal solution of the Wasserstein distributionally robust Logistic regression model with nonsummable steplength at a convergence rate of $1/T$. Finally, several numerical experiments to validate the effectiveness of theoretical analysis are conducted, which demonstrate that this method outperforms the off-the-shelf solver and the existing first-order algorithmic frameworks.

1. Introduction

Logistic regression (LR) has been widely considered as a statistical classification model in cross-discipline applications such as machine learning [1,2], signal processing [3,4], and computer vision [5,6], to name a few. It is developed based on the probabilistic relationship between the continuous feature vector and the binary label and highly relates to the maximum likelihood estimation. It is known that the LR model is established based on the training samples, and thus, its out-of-sample performance is deeply affected by the characteristics of the training data. For instance, the LR model presents poor out-of-sample performance when the training data are sparse [7]. In practice, the regularization technique is introduced to combat the overfitting effect triggered by the sparse data. In addition, with the increasing penetration of training data, the so-called “adversarial corruptions” are brought about [8], which describe the situation in which the data uncertainties are unmeasurable through any specific probability distribution. To tackle the sparsity and uncertainty of the training data, the distributionally robust optimization (DRO) approach is introduced, which not only provides a probabilistic interpretation of the existing regularization technique for data sparsity, but also guarantees the LR model to be immune to the risk caused by data uncertainty [7,8].

The basic idea to solve the DRO problem is to reformulate it into its tractable counterpart, then utilize the off-the-shelf solvers to tackle the problem [9,10,11]. It has been shown in [12,13,14] that the DRO problems formulated by statistical measures, e.g., $\varphi$-divergence, f-divergence, Wasserstein metric, and so on, possess exact convex reformulations. In practice, however, computing the large-scale DRO problem via off-the-shelf solvers is a demanding work because most of them rely on the general purpose interior-point algorithm [15]. Thus, it is necessary to devise a fast iterative algorithmic framework for the convex reformulations of DRO problems. In [16], a stochastic gradient descent framework was proposed for the DRO problem with f-divergence. In [17], a stochastic gradient descent algorithm was proposed for distributionally robust learning with $\varphi$-divergence. The $\varphi$-divergence builds on the discrete support of the empirical distribution of uncertainty, whose confidence bounds have an asymptotically inexact confidence region. While the f-divergence is an extension of $\varphi$-divergence, whose confidence bounds have an asymptotically exact confidence region [18]. Nevertheless, both $\varphi$-divergence and f-divergence can merely consider the training data with identical support. To overcome this hindrance, the Wasserstein metric is introduced to formulate the DRO problem [19,20], which is a useful instrument to handle the heterogeneous training data that can be extract from either discrete or continuous probability distributions. In [21], a branch-and-bound algorithm that combines with a linear approximation scheme was presented to reformulate the DRO problem with the Wasserstein metric into decomposable semi-infinite programs (

Table 1. Statistical measures utilized in the DRO problems.

References No.	Statistical Measures
[13,17]	$\varphi$-divergence
[16,18]	f-divergence
[12,19,20]	Wasserstein metric
This paper	Wasserstein metric

).

Despite the benefits brought by the regularization techniques, they simultaneously incur the computation obstacle, since the regularization term may be nonsmooth. For instance, LASSO regularization widely arises in robust regression problems [22,23,24]. In [25], the classical proximal gradient method was proposed, which is proven to be effective for evaluating the objective function and its gradient at a given point. Followed by the idea of a proximal gradient, in [26], a proximal forward-backward splitting (FOBOS) framework was established for signal recovery problems. Then, in [27], a FOBOS algorithm was devised for regularized convex optimization problem, where the subgradient of loss function was incorporated. Likewise, in [28,29], a fast iterative shrinkage-thresholding algorithm was proposed for regularized convex optimization problem, which utilizes the gradient of loss function. In [30], a FOBOS-based quasi-Newton method was proposed for nonsmooth optimization problems. Recently, in [31], a stochastic Douglas–Rachford splitting method was developed for sparse LR, which leverages the proximity operator that includes the stochastic gradient-like method. In [15], a first-order algorithmic framework was developed to address a Wasserstein distributionally robust LR. An ADMM framework that incorporates the gradient of log-loss function was devised, whose convergence performance was demonstrated to outperform the YALMIP solver.

Compared with the gradient-based methods, the nonlinear conjugate gradient (CG) methods are more efficient due to their simplicity and low memory requirement [32,33,34,35,36]. At present, nonlinear CG methods have been generalized to the nonsmooth optimization problems. In [37], a modified HS CG method for nonsmooth convex optimization problems was proposed, whose numerical efficiency was verified via high-dimensional training samples. Furthermore, in [38], a modified CG method that inherits the advantages of both HS and DY CG methods was constructed for a nonsmooth optimization problem. It is noted that the works in [37,38] merely addressed the convergence of the CG methods, while they ignored their convergence rates. Thus, this paper aims to develop a modified CG method for the Wasserstein distributionally robust LR model under the FOBOS framework, which can not only prove its convergence but also estimates its convergence rate. To the best of our knowledge, similar work has not been considered yet (

Table 2. Solution algorithms in the existing works.

References	Solution Algorithms
[25]	Classical proximal gradient method
[26,27,30,31]	FOBOS framework
[15]	First-order ADMM framework
[32,33,34,35,36]	CG method
[37,38]	Modified CG method
This paper	Modified CG method under FOBOS framework

).

The outline of this paper is as follows. Section 2 consists of two parts: Section 2.1 furnishes the basic framework of Wasserstein distributionally robust LR model; Section 2.2 establishes the CG method under FOBOS framework, in which some fundamental results that are vital for the convergence analysis in Section 3 are also derived. Section 3 presents the convergence analysis of the modified CG method, in which the convergence rate is proven to be $1/T$, where T denotes the predetermined iteration round and error bound, respectively, and simultaneously, the modified CG method is proven to converge to the optimal solution of the Wasserstein distributionally robust LR model with nonsummable steplength. Section 4 conducts the numerical experiments. Section 5 concludes the main contributions and provides the future research directions.

2. Preliminaries

2.1. Distributionally Robust Logistic Regression

Denote ${\mathbb{R}}^d$ be a d-dimensional Euclidean space. Let $\widehat{\mathit{x}}\in {\mathbb{R}}^d$ be a feature vector and $\widehat{y}\in \{-1,1\}$ be the associated binary label to be predicted one. In Logistic regression, the conditional distribution of y given $\widehat{\mathit{x}}$ is modeled as

$$\begin{array}{c}\hfill \mathcal{P}\left(\widehat{y}|\widehat{\mathit{x}}\right)={\left(1+e^{-\widehat{y}\langle \mathbf{\beta },\widehat{\mathit{x}}\rangle }\right)}^{-1},\end{array}$$

where the weight vector $\mathbf{\beta }\in {\mathbb{R}}^d$ constitutes a regression parameter [8]. Then, the maximum likelihood estimator of the classical Logistic regression is formulated by solving the following program:

$$\begin{array}{c}\hfill \underset{\mathbf{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{N}\sum _{i=1}^N{l}_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right),\end{array}$$

where ${\left\{({\widehat{\mathit{x}}}_i,{\widehat{y}}_i)\right\}}_{i=1}^N$ are the given training samples, $\mathbf{\beta }$ is the unknown parameter to be estimated, and $l_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right)=log\left(1+e^{-{\widehat{y}}_i\langle \mathbf{\beta },\widehat{{\mathit{x}}_i}\rangle }\right)$. Hereafter, for presentation convenience, define

$$\begin{array}{c}\hfill f\left(\mathbf{\beta }\right)=\frac{1}{N}\sum _{i=1}^N{l}_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right).\end{array}$$

(1)

Throughout this paper, the convex function is defined as follows.

Definition 1.

Function $f:{\mathbb{R}}^d\to \mathbb{R}$ is called a convex function, if for any ${\mathbf{\beta }}_1$, ${\mathbf{\beta }}_2$$\in {\mathbb{R}}^d$ and $0\le \lambda \le 1$, the following inequality holds

$$f(\lambda {\mathbf{\beta }}_1+(1-\lambda ){\mathbf{\beta }}_2)\le \lambda f\left({\mathbf{\beta }}_1\right)+(1-\lambda )f\left({\mathbf{\beta }}_2\right).$$

By Definition 1, it is known that $l_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right)$ is convex with respect to $\mathbf{\beta }\in {\mathbb{R}}^d$. Thus, $f\left(\mathbf{\beta }\right)$ is also convex with respect to $\mathbf{\beta }\in {\mathbb{R}}^d$.

In the standard statistical learning setting, all training and test samples are drawn independently from distribution $\mathbb{P}$ supported on $\Xi ={\mathbb{R}}^d\times \{-1,1\}$. If $\mathbb{P}$ is exactly available, the weight parameter $\mathbf{\beta }$ is found by solving the following stochastic optimization problem:

$$\begin{array}{c}\hfill \underset{\mathbf{\beta }\in {\mathbb{R}}^d}{min}{\mathbb{E}}^{\mathbb{P}}\left[l_{\mathbf{\beta }}(\widehat{\mathit{x}},\widehat{y})\right].\end{array}$$

In practice, however, the distribution $\mathbb{P}$ is always unavailable. Instead, it should be estimated through independently observed training samples ${\left\{({\widehat{\mathit{x}}}_i,{\widehat{y}}_i)\right\}}_{i=1}^N$. Thus, the distributionally robust optimization (DRO) approach provides an alternative perspective, which motivates us to screen the potential true distribution with a high confidence from the ambiguity set that has been constructed by such training samples. In this case, the following DRO problem is considered:

$$\begin{array}{c}\hfill \underset{\mathbf{\beta }\in {\mathbb{R}}^d}{inf}\underset{\mathbb{Q}\in {\mathbb{B}}_{\epsilon }\left({\widehat{\mathbb{P}}}_N\right)}{sup}{\mathbb{E}}^{\mathbb{Q}}\left[l_{\mathbf{\beta }}(\widehat{\mathit{x}},\widehat{y})\right],\end{array}$$

(2)

where ${\mathbb{B}}_{\epsilon }\left({\widehat{\mathbb{P}}}_N\right)=\left\{\mathbb{Q}:W\left(\mathbb{Q},{\widehat{\mathbb{P}}}_N\right)\le \epsilon \right\}$ denotes the Wasserstein ball of radius $\epsilon$ centered at the empirical distribution ${\widehat{\mathbb{P}}}_N$, which is calculated by the training samples ${\left\{({\widehat{\mathit{x}}}_i,{\widehat{y}}_i)\right\}}_{i=1}^N$. Thus, hereafter, this paper only considers $\epsilon >0$. $W\left(·,·\right)$ denotes the Wasserstein distance between two distributions, which is defined as follows.

Definition 2

([7]). Let two distributions ${\mathbb{P}}_1$ and ${\mathbb{P}}_2$ share a common support set Ξ. Given $d(·,·)$ that denotes a metric on $\Xi \times \Xi$. The Wasserstein distance between ${\mathbb{P}}_1$ and ${\mathbb{P}}_2$ is defined as

$$\begin{array}{ccc}& & W\left({\mathbb{P}}_1,{\mathbb{P}}_2\right)\hfill \\ & & =\underset{\Pi \in \Pi \left({\Xi }^2\right)}{inf}\left\{{\int }_{{\Xi }^2}w\left(\xi ,{\xi }^{\prime }\right)\Pi \left(d\xi ,d{\xi }^{\prime }\right):\Pi \left(d\xi ,\Xi \right)={\mathbb{P}}_1\left(d\xi \right),\Pi \left(\Xi ,d{\xi }^{\prime }\right)={\mathbb{P}}_2\left(d{\xi }^{\prime }\right)\right\},\hfill \end{array}$$

where $\xi =\left(\widehat{\mathit{x}},\widehat{y}\right)$ and ${\xi }^{\prime }=\left({\widehat{\mathit{x}}}^{\prime },{\widehat{y}}^{\prime }\right)$. $w(·,·)$ is the distance metric between two vectors ξ and ${\xi }^{\prime }$. Π is the joint distribution of ξ and ${\xi }^{\prime }$, ${\mathbb{P}}_1$ and ${\mathbb{P}}_2$ can be considered as marginal distributions of Π with respect to ξ and ${\xi }^{\prime }$, respectively.

The Wasserstein distance is highly related to the well-known optimal transport problem, in which the distance metric $d(·,·)$ is interpreted as the unit mass transport cost from one element in $\Xi$ to another, which means that $W\left({\mathbb{P}}_1,{\mathbb{P}}_2\right)$ denotes the minimum expected transport cost among the family of joint distributions [39].

Obviously, the DRO problem (2) is intractable in its present form. In order to solve it via numerical methods, its tractable reformulation has be to found. In [7,15], a tractable reformulation of (2) was established by considering the distance metric of type (3),

$$\begin{array}{c}\hfill d\left((\mathit{x},y),({\mathit{x}}^{\prime },y^{\prime })\right)=\parallel \mathit{x}-{\mathit{x}}^{\prime }{\parallel }_p+\frac{\kappa }{2}|y-y^{\prime }|,\end{array}$$

(3)

where $\kappa$ is a positive constant, ${\parallel ·\parallel }_p$ is the p-norm on ${\mathbb{R}}^d$. $\kappa$ represents the trust in the labels of the training samples. It can be seen that $\kappa =+\infty$ avoids different labels to be assigned to a fixed feature vector. Thus, setting $\kappa =+\infty$ is reasonable to reflect the exactness of the label measurements. Hence, the following distance metric on $\Xi$ is considered in this paper:

$$\begin{array}{c}\hfill d\left((\mathit{x},y),({\mathit{x}}^{\prime },y^{\prime })\right)=\left\{\begin{array}{cc}\parallel \mathit{x}-{\mathit{x}}^{\prime }{\parallel }_{\infty },& \mathrm{if}y=y^{\prime },\hfill \\ \infty ,& \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(4)

According to the reformulation results in [24], the DRO problem (2) with distance metric (4) can be reformulated into the following unconstrained optimization problem:

$$\begin{array}{c}\hfill \underset{\mathbf{\beta }\in {\mathbb{R}}^d}{min}r\left(\mathbf{\beta }\right)+f\left(\mathbf{\beta }\right),\end{array}$$

(5)

where $r\left(\mathbf{\beta }\right)={\epsilon \parallel \mathbf{\beta }\parallel }_1$, $f\left(\mathbf{\beta }\right)=\frac{1}{N}{\sum }_{i=1}^N{l}_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right)$.

Remark 1.

The regularization term $r\left(\mathbf{\beta }\right)={\epsilon \parallel \mathbf{\beta }\parallel }_1$ in optimization problem (5) is usually named by LASSO, which is nondifferentiable at the origin. In practice, the LASSO regularization technique is an effective tool and widely investigated for sparse signal processing, including sparse signal reconstructing and sparsity pattern recovering [22].

Remark 2.

In [15], an ADMM-based algorithmic framework was proposed for a modified distributionally robust Logistic regression model, where the gradient of $f\left(\mathbf{\beta }\right)$ was incorporated. In practice, however, the applications in statistical learning are always of large scale. For large-scale problems, the conjugate gradient methods outperform the gradient methods in both numerical efficiency and convergence performance due to their simplicity and low memory requirement [38]. In the following, an improved algorithmic framework takes into account the conjugate gradient methods.

2.2. The Modified CG Method

To address the convex optimization problem (5), where $f\left(\mathbf{\beta }\right)$ is smooth and $r\left(\mathbf{\beta }\right)$ is nonsmooth, the FOBOS framework is employed that incorporates the conjugate gradient steps as a major ingredient, which is an extension of the projected gradient method, by replacing or augmenting the projection step with an minimization problem and alleviates the nonsmooth term $r\left(\mathbf{\beta }\right)$, by taking analytical minimization steps interleaved with conjugate gradient steps [26,27]. The FOBOS framework consists of two steps shown in (6) and (7). One is a conjugate gradient descent step (6) which aims to find an interim vector ${\mathbf{\beta }}_{t+\frac{1}{2}}$ along with the searching direction ${\mathit{d}}_t$ that achieves a lower value of $f\left(\mathbf{\beta }\right)$; the other is a proximal searching step (7) which aims to find an update vector ${\mathbf{\beta }}_{t+1}$ that not only stays close to the interim ${\mathbf{\beta }}_{t+\frac{1}{2}}$, but also attains a low complexity value as expressed by $r\left(\mathbf{\beta }\right)$.

$$\begin{array}{ccc}\hfill {\mathbf{\beta }}_{t+\frac{1}{2}}& =& {\mathbf{\beta }}_t+{\alpha }_t{\mathit{d}}_t\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill {\mathbf{\beta }}_{t+1}& =& {\mathrm{argmin}}_{\mathbf{\beta }\in {\mathbb{R}}^d}\left\{\frac{1}{2}{∥\mathbf{\beta }-{\mathbf{\beta }}_{t+\frac{1}{2}}∥}_2^2+{\alpha }_{t}r\left(\mathbf{\beta }\right)\right\},\hfill \end{array}$$

(7)

where ${\alpha }_t$ is the searching steplength, and ${\mathit{d}}_t$ is the searching direction that is defined as follows:

$$\begin{array}{c}\hfill {\mathit{d}}_{t+1}=-{\mathit{g}}_{t+1}^f+{\vartheta }_{t+1}{\mathit{d}}_t,\end{array}$$

(8)

where ${\mathit{d}}_0=-{\mathit{g}}_0$, ${\mathit{g}}_{t+1}^f$ is the gradient of $f\left(\mathbf{\beta }\right)$ at ${\mathbf{\beta }}_{t+1}$, ${\vartheta }_{t+1}$ is a conjugate parameter, which is defined as

$$\begin{array}{c}\hfill {\vartheta }_{t+1}=\frac{{∥{\mathit{g}}_{t+1}^f∥}_2}{\eta {∥{\mathit{d}}_t∥}_2},\end{array}$$

(9)

where $\eta >1$ is a constant.

Remark 3.

In [37,38], the modified CG methods were proposed under the Moreau–Yosida regularization framework for the nonsmooth convex optimization problems, in which merely the convergence of the proposed modified CG methods was considered, while the convergence rate was ignored. In this paper, an alternative solution way that develops a modified CG method under the FOBOS framework is proposed, in which not only the convergence of the modified CG method can be analyzed, but also its convergence rate can be estimated.

3. Convergence Analysis

The conjugate gradient steps (6) and (7) provide the following two propositions.

Proposition 1.

For the searching direction in (8) and (9), we have

$$\begin{array}{c}\hfill 〈{\mathit{g}}_{t+1}^f,{\mathit{d}}_{t+1}〉\le -\left(1-\frac{1}{\eta }\right){∥{\mathit{g}}_{t+1}^f∥}_2^2\end{array}$$

(10)

and

$$\begin{array}{c}\hfill {∥{\mathit{d}}_{t+1}∥}_2\le \left(1+\frac{1}{\eta }\right){∥{\mathit{g}}_{t+1}^f∥}_2\end{array}$$

(11)

for all $t\ge 0$.

Proof. 

When $t\ge 0$, by (8) and (9), we have

$$\begin{array}{ccc}\hfill 〈{\mathit{g}}_{t+1}^f,{\mathit{d}}_{t+1}〉& \le & -{∥{\mathit{g}}_{t+1}^f∥}_2^2+\frac{{∥{\mathit{g}}_{t+1}^f∥}_2}{\eta {∥{\mathit{d}}_t∥}_2}〈{\mathit{g}}_{t+1}^f,{\mathit{d}}_t〉\hfill \\ & \le & -{∥{\mathit{g}}_{t+1}^f∥}_2^2+\frac{{∥{\mathit{g}}_{t+1}^f∥}_2}{\eta {∥{\mathit{d}}_t∥}_2}{∥{\mathit{g}}_{t+1}^f∥}_2{∥{\mathit{d}}_t∥}_2\hfill \\ & =& -\left(1-\frac{1}{\eta }\right){∥{\mathit{g}}_{t+1}^f∥}_2^2,\hfill \end{array}$$

where the second inequality is obtained by the Cauchy–Schwartz inequality.

On the other hand,

$$\begin{array}{ccc}\hfill {∥{\mathit{d}}_{t+1}∥}_2& \le & {∥{\mathit{g}}_{t+1}^f∥}_2+\frac{{∥{\mathit{g}}_{t+1}^f∥}_2}{\eta {∥{\mathit{d}}_t∥}_2}{∥{\mathit{d}}_t∥}_2\hfill \\ & \le & \left(1+\frac{1}{\eta }\right){∥{\mathit{g}}_{t+1}^f∥}_2.\hfill \end{array}$$

The proof is completed. □

By (10), we know ${\mathit{d}}_t$ is a descent direction, that is, $〈{\mathit{g}}_t^f,{\mathit{d}}_t〉<0$ for all $t\ge 0$. In addition, notice that $f\left(\mathbf{\beta }\right)={\sum }_{i=1}^N{l}_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right)$ and $l_{\mathbf{\beta }}\left({\widehat{\mathit{x}}}_i,{\widehat{y}}_i\right)=log\left(1+e^{-{\widehat{y}}_i\langle \mathbf{\beta },\widehat{{\mathit{x}}_i}\rangle }\right)$. Thus, $f\left(\mathbf{\beta }\right)$ is convex and ${\parallel \nabla f\left(\mathbf{\beta }\right)\parallel }_2$ is bounded by $L^f={max}_{1\le i\le N}{∥{\widehat{y}}_i{\widehat{x}}_i∥}_2$, that is, ${∥{\mathit{g}}_t^f∥}_2\le L^f$ for all $t\ge 0$. Then, by (11), we obtain

$$\begin{array}{c}\hfill {∥{\mathit{d}}_t∥}_2\le \left(1+\frac{1}{\eta }\right)L^f.\end{array}$$

(12)

Proposition 2.

For any vector ${\mathbf{\beta }}^{*}\in {\mathbb{R}}^d$, we have

$$\begin{array}{ccc}\hfill 〈{\mathit{d}}_t,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉& \le & 〈-{\mathit{g}}_t^f,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t〉+f\left({\mathbf{\beta }}^{*}\right)-f\left({\mathbf{\beta }}_t\right)\hfill \\ & & +\frac{1}{2}{∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}∥}_2^2+\frac{1}{2}{\left(\frac{L^f}{{\eta }^2}\right)}^2,\hfill \end{array}$$

(13)

for all $t\ge 0$.

Proof. 

By (8) and (9), we obtain

$$\begin{array}{ccc}\hfill 〈{\mathit{d}}_t,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉& =& 〈-{\mathit{g}}_t^f+{\vartheta }_t{\mathit{d}}_{t-1},{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉\hfill \\ & =& 〈-{\mathit{g}}_t^f,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉+\frac{{∥{\mathit{g}}_t^f∥}_2}{\eta {∥{\mathit{d}}_{t-1}∥}_2}〈{\mathit{d}}_{t-1},{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉\hfill \\ & \le & 〈-{\mathit{g}}_t^f,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t〉+f\left({\mathbf{\beta }}^{*}\right)-f\left({\mathbf{\beta }}_t\right)\hfill \\ & & +\frac{1}{2}{∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}∥}_2^2+\frac{1}{2}{\left(\frac{L^f}{{\eta }^2}\right)}^2,\hfill \end{array}$$

where the third inequality is obtained by utilizing the convexity of $f\left(\mathbf{\beta }\right)$ and $〈\mathit{a},\mathit{b}〉\le \frac{1}{2}〈\mathit{a},\mathit{a}〉+\frac{1}{2}〈\mathit{b},\mathit{b}〉$ by the Cauchy–Schwartz inequality. The proof is completed. □

The optimal condition of (7) gives that

$$\begin{array}{c}\hfill 0\in \partial {\left.\left\{\frac{1}{2}{∥\mathbf{\beta }-{\mathbf{\beta }}_{t+\frac{1}{2}}∥}_2^2+{\alpha }_{t}r\left(\mathbf{\beta }\right)\right\}\right|}_{\mathbf{\beta }={\mathbf{\beta }}_{t+1}},\end{array}$$

(14)

where $\partial \left\{·\right\}$ denotes the set of subgradients. Then, substituting (6) into (14), we have

$$\begin{array}{c}\hfill {\mathbf{\beta }}_{t+1}={\mathbf{\beta }}_t+{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r,\end{array}$$

(15)

where ${\mathit{g}}_{t+1}^r$ is a subgradient of $r\left(\mathbf{\beta }\right)$ at ${\mathbf{\beta }}_{t+1}$. Notice that $r\left(\mathbf{\beta }\right)={\epsilon \parallel \mathbf{\beta }\parallel }_1$, we know that the subgradient of $r\left(\mathbf{\beta }\right)$ is bounded for all $\mathbf{\beta }\in {\mathbb{R}}^d$. Thus, there exists a constant $L^r$ such that ${∥{\mathit{g}}_{t+1}^r∥}_2\le L^r$ for all $t\ge 0$.

Lemma 1.

Suppose vector ${\mathbf{\beta }}^{*}\in {\mathbb{R}}^d$ is the optimal solution of (5). Then, the following inequality holds

$$\begin{array}{ccc}(1-{\alpha }_t){∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}∥}_2^2\hfill & \le \hfill & {∥{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*}∥}_2^2+2{\alpha }_t\left(r\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}_{t+1}\right)\right)\hfill \\ \hfill & \hfill & +2{\alpha }_t\left(f\left({\mathbf{\beta }}^{*}\right)-f\left({\mathbf{\beta }}_t\right)\right)+\frac{L^2}{{\eta }^2}{\alpha }_t+12L^2{\alpha }_t^2,\hfill \end{array}$$

(16)

where $L=max\left\{\left(1+\frac{1}{\eta }\right)L^f,L^r\right\}$.

Proof. 

We begin with deriving the properties of $r\left(\mathbf{\beta }\right)$ and its subgradients. By (15), we have

$$\begin{array}{c}\hfill {\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t={\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r.\end{array}$$

(17)

Notice that $r\left(\mathbf{\beta }\right)$ is convex, we obtain

$$\begin{array}{c}\hfill 〈-{\mathit{g}}_{t+1}^r,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}〉\le r\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}_{t+1}\right).\end{array}$$

(18)

Substituting (17) into (18), we obtain

$$\begin{array}{ccc}〈-{\mathit{g}}_{t+1}^r,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t〉\hfill & =\hfill & 〈-{\mathit{g}}_{t+1}^r,{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r〉\hfill \\ \hfill & \le \hfill & {∥{\mathit{g}}_{t+1}^r∥}_2{∥{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r∥}_2\hfill \\ \hfill & \le \hfill & {\alpha }_t{∥{\mathit{g}}_{t+1}^r∥}_2{∥{\mathit{d}}_t∥}_2+{\alpha }_t{∥{\mathit{g}}_{t+1}^r∥}_2^2\hfill \\ \hfill & \le \hfill & 2{\alpha }_t{L}^2,\hfill \end{array}$$

(19)

where the second inequality is derived by the Cauchy–Schwartz inequality, and the last inequality is obtained by $L=max\left\{\left(1+\frac{1}{\eta }\right)L^f,L^r\right\}$. Similarly, we have

$$\begin{array}{c}\hfill 〈{\mathit{d}}_t,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t〉\le 2{\alpha }_t{L}^2,\end{array}$$

(20)

and

$$\begin{array}{c}\hfill 〈-{\mathit{g}}_t^f,{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t〉\le 2{\alpha }_t{L}^2.\end{array}$$

(21)

We then derive the bound of the difference between ${\mathbf{\beta }}^{*}$ and ${\mathbf{\beta }}_{t+1}$ recursively.

$$\begin{array}{ccc}{∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}∥}_2^2\hfill & =\hfill & {∥{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*}+{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r∥}_2^2\hfill \\ \hfill & =\hfill & {∥{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*}∥}_2^2+2〈{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*},{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r〉+{∥{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r∥}_2^2\hfill \\ \hfill & \le \hfill & {∥{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*}∥}_2^2+2〈{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*},{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r〉\hfill \\ \hfill & \hfill & +2〈{\mathbf{\beta }}_t-{\mathbf{\beta }}_{t+1},{\alpha }_t{\mathit{d}}_t-{\alpha }_t{\mathit{g}}_{t+1}^r〉+{\alpha }_t^2{∥{\mathit{d}}_t-{\mathit{g}}_{t+1}^r∥}_2^2.\hfill \end{array}$$

(22)

Considering the last term in (22), we obtain that

$$\begin{array}{ccc}{∥{\mathit{d}}_t-{\mathit{g}}_{t+1}^r∥}_2^2\hfill & \le \hfill & {\left({∥{\mathit{d}}_t∥}_2+{∥{\mathit{g}}_{t+1}^r∥}_2\right)}^2\hfill \\ \hfill & \le \hfill & 4L^2.\hfill \end{array}$$

(23)

Then, inequality (16) holds by substituting (13), (18)–(21) and (23) into (22). The proof is completed. □

Lemma 1 establishes a fundamental result for the convergence properties of conjugate gradient steps (6) and (7). Now, we are ready to derive the convergence results based on the preceding discussion.

Theorem 1.

Let D be a constant. Suppose (i) ${\alpha }_t\ge {\alpha }_{t+1}$ for all $t\ge 0$; (ii) vector ${\mathbf{\beta }}^{*}\in {\mathbb{R}}^d$ is the optimal solution of (5); (iii) ${∥{\mathbf{\beta }}_0-{\mathbf{\beta }}^{*}∥}_2\le D$. We have

$$\begin{array}{c}\hfill \sum _{t=0}^{T}2{\alpha }_t\left(f\left({\mathbf{\beta }}_t\right)+r\left({\mathbf{\beta }}_t\right)-f\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)\le {\lambda }^2{D}^2+12L^2\sum _{t=0}^T{\alpha }_t^2.\end{array}$$

(24)

Proof. 

Rearranging $f\left({\mathbf{\beta }}^{*}\right)-f\left({\mathbf{\beta }}_t\right)$ and $r\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}_{t+1}\right)$ in (16), we obtain

$$\begin{array}{ccc}(1-{\alpha }_t){∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}^{*}∥}_2^2\hfill & \le \hfill & {∥{\mathbf{\beta }}_t-{\mathbf{\beta }}^{*}∥}_2^2+2{\alpha }_t\left(r\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}_{t+1}\right)\right)\hfill \\ \hfill & \hfill & +2{\alpha }_t\left(f\left({\mathbf{\beta }}^{*}\right)-f\left({\mathbf{\beta }}_t\right)\right)+\frac{L^2}{{\eta }^2}{\alpha }_t+12L^2{\alpha }_t^2,\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill & \hfill & \sum _{t=0}^{T}2{\alpha }_t\left(f\left({\mathbf{\beta }}_t\right)-f\left({\mathbf{\beta }}^{*}\right)+r\left({\mathbf{\beta }}_{t+1}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)\hfill \\ \hfill & \hfill & \le {∥{\mathbf{\beta }}_0-{\mathbf{\beta }}^{*}∥}_2^2-\left(1-\frac{4{\alpha }_{t}L}{\eta }\right){∥{\mathbf{\beta }}_{T+1}-{\mathbf{\beta }}^{*}∥}_2^2+12L^2\sum _{t=0}^T{\alpha }_t^2.\hfill \end{array}$$

(26)

Term ${\sum }_{t=0}^{T}2{\alpha }_t\left(r\left({\mathbf{\beta }}_{t+1}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)$ yields that

$$\begin{array}{ccc}\hfill & \hfill & \sum _{t=0}^{T}2{\alpha }_t\left(r\left({\mathbf{\beta }}_{t+1}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)\hfill \\ \hfill & \hfill & \ge \sum _{t=0}^{T}2{\alpha }_t\left(r\left({\mathbf{\beta }}_t\right)-r\left({\mathbf{\beta }}^{*}\right)\right)+{\alpha }_T\left(r\left({\mathbf{\beta }}_{T+1}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)-{\alpha }_0\left(r\left({\mathbf{\beta }}_0\right)-r\left({\mathbf{\beta }}^{*}\right)\right)\hfill \\ \hfill & \hfill & \ge \sum _{t=0}^{T}2{\alpha }_t\left(r\left({\mathbf{\beta }}_t\right)-r\left({\mathbf{\beta }}^{*}\right)\right)+\left({\alpha }_0-{\alpha }_T\right)r\left({\mathbf{\beta }}^{*}\right)\hfill \\ \hfill & \hfill & \ge \sum _{t=0}^{T}2{\alpha }_t\left(r\left({\mathbf{\beta }}_t\right)-r\left({\mathbf{\beta }}^{*}\right)\right),\hfill \end{array}$$

(27)

where the second inequality is obtained since ${\alpha }_t>0$ for all $t\ge 0$ and $r\left(\mathbf{\beta }\right)\ge 0$ for all $\mathbf{\beta }\in {\mathbb{R}}^d$. The last inequality is obtained since the steplength ${\alpha }_t\ge {\alpha }_{t+1}$ for all $t\ge 0$. Then, by properly choosing ${\mathbf{\beta }}_0$ such that ${∥{\mathbf{\beta }}_0-{\mathbf{\beta }}^{*}∥}_2\le \lambda D$ and combining (25) and (26), we derive (24). The proof is completed. □

In the remainder of this paper, two corollaries are presented to show the convergence properties of the conjugate gradient steps (6) and (7).

Theorem 2.

Suppose the initial value ${\mathbf{\beta }}_0$ is chosen within a $\lambda D$ neighborhood of ${\mathbf{\beta }}^{*}$. Then, for steplength ${\alpha }_t$ that satisfies ${\alpha }_t\ge {\alpha }_{t+1}$, ${\alpha }_t\to 0$ for all $t\ge 0$, and ${\sum }_{t=0}^{\infty }{\alpha }_t=\infty$, the conjugate gradient descent method (6) and (7) converges to the optimal solution of DRO problem (2). Moreover, for predefined T iterations, the conjugate gradient descent method (6) and (7) yields a convergence rate of $1/T$.

Proof. 

By Theorem 1, we have

$$\begin{array}{c}\hfill \underset{t=1,2,\dots ,T}{min}\left(f\left({\mathbf{\beta }}_t\right)+r\left({\mathbf{\beta }}_t\right)-f\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}^{*}\right)\right)\sum _{t=0}^{T}2{\alpha }_t\le {\lambda }^2{D}^2+12L^2\sum _{t=0}^T{\alpha }_t^2.\end{array}$$

(28)

Let $T=\infty$, then (27) implies that

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim\; inf}f\left({\mathbf{\beta }}_t\right)+r\left({\mathbf{\beta }}_t\right)-f\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}^{*}\right)\le \frac{{\lambda }^2{D}^2+12L^2{\sum }_{t=0}^{\infty }{\alpha }_t^2}{2{\sum }_{t=0}^{\infty }{\alpha }_t}.\end{array}$$

(29)

Since ${\sum }_{t=0}^{\infty }{\alpha }_t=\infty$, the right-hand side of (28) tends to 0, which indicates that

$$\underset{t\to \infty }{lim\; inf}f\left({\mathbf{\beta }}_t\right)+r\left({\mathbf{\beta }}_t\right)=f\left({\mathbf{\beta }}^{*}\right)+r\left({\mathbf{\beta }}^{*}\right).$$

Moreover, let ${\alpha }_t=\frac{1}{2(T+1)}$ for all $t\ge 0$ and $\lambda =\frac{1}{\sqrt{T+1}}$. Then, (27) yields that

$$\begin{array}{c}\hfill \underset{t=1,2,\dots ,T}{min}\left(f\left({\mathbf{\beta }}_t\right)+r\left({\mathbf{\beta }}_t\right)\right)\le f\left({\mathbf{\beta }}^{*}\right)-r\left({\mathbf{\beta }}^{*}\right)+\frac{D^2+3L^2}{T+1},\end{array}$$

(30)

which means that the conjugate gradient steps (6) and (7) yield a convergence rate of $1/T$. The proof is completed. □

4. Numerical Experiments

In this section, the infinite norm in distance metric (4) is considered; then, the optimization problem (5) becomes the ${\ell }_1$-regularized convex optimization problem. Then, several numerical experiments are conducted to validate the efficiency and effectiveness of the modified CG method from different aspects. Firstly, the sensitive analysis is carried out to compare the runtime of the modified CG method under different Wasserstein ball radius $\epsilon$ and steplength ${\left\{{\alpha }_t\right\}}_{t=0}^{\infty }$ to show how these parameters affect the performance of the modified CG method. Secondly, the runtime between our modified CG method and the IPOPT solver (https://github.com/coin-or/Ipopt (accessed on 21 January 2023)) is compared, where the IPOPT solver is implemented on the YALMIP platform [40]. Lastly, the runtime among several algorithmic frameworks for ${\ell }_1$-regularized problems is compared, which includes the modified CG method, the standard sub-gradient method, the ISTA, and the FISTA. All the experiments are conducted using MATLAB R2018a on a laptop running Windows 7 with Intel(R) Core(TM) i3-2310 CPU (2.1 GHz) and 10 GB RAM. In addition, the experiments are implemented on real datasets a1a–a9a from LIBSVM (https://www.csie.ntu.edu.tw/∼cjlin/libsvm/index.html (accessed on 23 February 2023)), which are extracted from UCI/Adult (http://archive.ics.uci.edu/ml/index.php (accessed on 23 February 2023)). The data characteristics are listed in

Table 3. The data characteristics of UCI/Adult datasets.

Dataset	Data Characteristics
	Number of Samples	Number of Features
a1a	1605	123
a2a	2265	123
a3a	3185	123
a4a	4781	123
a5a	6414	123
a6a	11,220	123
a7a	16,100	123
a8a	22,696	123
a9a	32,561	123

. To begin with, the conjugate parameter $\eta$ is set to 5 and the initial values of $\mathbf{\beta }$ are chosen randomly within the neighborhood $\left\{\mathbf{\beta }\in {\mathbb{R}}^d:{∥\mathbf{\beta }∥}_{\infty }\le \frac{0.2785}{\epsilon }\right\}$. To eliminate the impact of random choices of ${\mathbf{\beta }}_0$ on the runtime, each experiment is repeated five times and their runtimes are recorded. Lastly, the average runtime is utilized to reflect the numerical efficiency of the solution methods. The programs stop when ${∥{\mathbf{\beta }}_{t+1}-{\mathbf{\beta }}_t∥}_2\le {10}^{-3}$.

4.1. Runtime Comparison: Among Different Parameters

In this part, the average runtime for different Wasserstein ball radius $\epsilon$ and steplength ${\alpha }_t$ are tested to show how such parameters affect the numerical efficiency of the modified CG method. In particular, for fixed steplength ${\alpha }_t=1/(t+1)$, the average runtime for five different choices of $\epsilon$ are tested, including $\epsilon =0.1$, $\epsilon =0.11$, $\epsilon =0.12$, $\epsilon =0.13$, and $\epsilon =0.14$. In addition, for fixed Wasserstein ball radius $\epsilon =0.1$, the average runtime for five different choices of ${\alpha }_t$ is tested, including ${\alpha }_t=1/\sqrt{t+1}$, ${\alpha }_t=1/\sqrt[3]{t+1}$, ${\alpha }_t=1/\sqrt[4]{t+1}$, ${\alpha }_t=1/\sqrt[4]{t+1}$, ${\alpha }_t=1/\sqrt[5]{t+1}$, and ${\alpha }_t=1/\sqrt[6]{t+1}$. The experimental outputs are listed in

Table 4. Comparison of average runtime for different Wasserstein ball radius $\epsilon$ with fixed steplength ${\alpha }_t=1/\sqrt{t+1}$.

Dataset	Average Runtime (s)
	$\mathbf{\epsilon }=\mathbf{0}.\mathbf{1}$	$\mathbf{\epsilon }=\mathbf{0}.\mathbf{11}$	$\mathbf{\epsilon }=\mathbf{0}.\mathbf{12}$	$\mathbf{\epsilon }=\mathbf{0}.\mathbf{13}$	$\mathbf{\epsilon }=\mathbf{0}.\mathbf{14}$
a1a	6.68	4.81	3.45	2.70	1.98
a2a	7.96	6.88	4.95	3.76	2.94
a3a	11.97	8.45	6.30	4.78	3.79
a4a	18.47	13.31	9.30	7.05	5.36
a5a	24.40	16.54	12.51	10.08	7.29
a6a	43.38	31.16	22.36	17.97	14.10
a7a	63.16	46.51	34.91	27.62	20.77
a8a	88.15	70.67	52.28	44.94	37.72
a9a	154.64	115.06	82.86	74.02	67.79

and

Table 5. Comparison of average runtime for different steplengths with fixed Wasserstein ball radius $\epsilon =0.1$.

Dataset	Average Runtime (s)
	${\mathbf{\alpha }}_{\mathbf{t}}=\mathbf{1}/\sqrt{\mathbf{t}+\mathbf{1}}$	${\mathbf{\alpha }}_{\mathbf{t}}=\mathbf{1}/\sqrt[\mathbf{3}]{\mathbf{t}+\mathbf{1}}$	${\mathbf{\alpha }}_{\mathbf{t}}=\mathbf{1}/\sqrt[\mathbf{4}]{\mathbf{t}+\mathbf{1}}$	${\mathbf{\alpha }}_{\mathbf{t}}=\mathbf{1}/\sqrt[\mathbf{5}]{\mathbf{t}+\mathbf{1}}$	${\mathbf{\alpha }}_{\mathbf{t}}=\mathbf{1}/\sqrt[\mathbf{6}]{\mathbf{t}+\mathbf{1}}$
a1a	6.68	2.47	1.75	1.39	1.26
a2a	7.96	3.31	2.44	1.99	1.74
a3a	11.97	4.71	3.46	2.89	2.60
a4a	18.47	7.06	5.59	4.35	4.10
a5a	24.40	10.07	6.68	5.89	5.11
a6a	43.38	17.53	13.46	10.24	9.04
a7a	63.16	26.20	20.93	15.31	13.07
a8a	88.15	39.08	30.47	23.63	20.33
a9a	154.64	59.20	49.64	33.94	28.51

and Figure 1 and Figure 2. Observing the experimental outputs, it can be seen that with the increasing problem scale, the required runtime of the modified CG method increases significantly. For fixed steplength, a larger Wasserstein ball radius means shorter average runtime. The reason is that the Wasserstein ball radius $\epsilon$ is inversely proportional to the radius of the searching neighborhood $\left\{\mathbf{\beta }\in {\mathbb{R}}^d:{∥\mathbf{\beta }∥}_{\infty }\le \frac{0.2785}{\epsilon }\right\}$. Thus, larger $\epsilon$ means smaller searching neighborhood, which results in shorter average runtime. In addition, it is observed from Table 4 and Figure 2 that the average runtime of the modified CG method can be controlled by the steplength ${\alpha }_t$. The average runtime of ${\alpha }_t=1/\sqrt[6]{t+1}$ is the shortest among the five steplengths.

4.2. Runtime Comparison: Modified CG Method versus IPOPT Solver

In this part, the average runtime between the IPOPT solver and the modified CG method are tested. The IPOPT is a typical solver that builds upon the interior-point algorithmic framework. The Wasserstein ball radius $\epsilon$ and the steplength are set to be $0.1$ and $1/\sqrt{t+1}$, respectively. The experimental output is given in

Table 6. Comparison of average CPU time between the IPOPT solver and the modified CG method.

Dataset	Average Runtime (s)
	IPOPT Solver	Modified CG Method
a1a	103.50	6.68
a2a	153.43	7.96
a3a	259.12	11.97
a4a	447.91	18.47
a5a	891.84	24.40
a6a	1460.46	43.38
a7a	2784.30	63.16
a8a	5293.32	88.15
a9a	10,214.28	154.64

and Figure 3. It can be observed that the average CPU time of IPOPT solver far exceeds that of the modified CG method, which demonstrates that the modified CG method is far more effective than the interior-point-based off-the-shelf solvers for large-scale problems.

4.3. Runtime Comparison: Among Different First-Order Algorithmic Frameworks

In this part, the average runtime among four first-order algorithmic frameworks is tested, including the subgradient method, the ISTA, the FISTA, and the modified CG method. The Wasserstein ball radius $\epsilon$ is set to be $0.1$. The steplength of the subgradient method and the modified CG method are set to be $1/\sqrt[6]{t+1}$. The experimental output is illustrated in

Table 7. Comparison of CPU time among different algorithmic frameworks.

Dataset	Average Runtime (s)
	Subgradient Method	ISTA	FISTA	Modified CG Method
a1a	37.26	3.58	1.55	1.26
a2a	57.19	4.53	2.26	1.74
a3a	76.57	6.15	2.74	2.60
a4a	114.14	9.39	3.95	4.10
a5a	164.25	12.89	5.60	5.11
a6a	307.86	23.63	10.14	9.04
a7a	396.50	34.31	16.10	13.07
a8a	578.30	52.76	23.52	20.33
a9a	801.04	79.50	33.89	28.51

and Figure 4. It can be observed that the modified CG method is more effective than the other three widely employed first-order algorithmic frameworks.

5. Conclusions

In this paper, we propose a modified CG method under the FOBOS framework for the Wasserstein distributionally robust LR model. This method consists of two phases: in the first phase, a conjugate gradient descent step is performed, and in the second phase, an instantaneous optimization problem is formulated and solved with a trade-off minimization of the regularization term, while simultaneously staying in close proximity to the interim point obtained in the first phase. With a nonsummable steplength, it is further proven that the modified CG method attains the optimal solution of the Wasserstein distributionally robust LR model. Moreover, the convergence rate of the modified CG method is estimated as $1/T$. Finally, several numerical experiments are conducted to test the numerical efficiency of the modified CG method and demonstrate that this method outperforms the off-the-shelf solver and the exiting first-order algorithmic frameworks. Future research in this field is focused on two aspects: (i) advancing the algorithmic framework through the exploration of advanced optimization techniques, such as the proximal method, accelerated gradient method, and stochastic approximation method. These investigations aim to refine the optimization algorithms, develop more efficient convergence criteria, and enhance numerical performance. (ii) The application of distributionally robust LR to real-world domains, including healthcare, finance, marketing, and transportation. This entails examining the practical implications, interpretability, and performance of these models in various contexts. Furthermore, conducting case studies will allow for an assessment of the benefits and limitations of distributionally robust approaches in comparison to stochastic and robust methods.