A More Efficient and Practical Modified Nyström Method

Abstract

In this paper, we propose an efficient Nyström method with theoretical and empirical guarantees. In parallel computing environments and for sparse input kernel matrices, our algorithm can have computation efficiency comparable to the conventional Nyström method, theoretically. Additionally, we derive an important theoretical result with a compacter sketching matrix and faster speed, at the cost of some accuracy loss compared to the existing state-of-the-art results. Faster randomized SVD and more efficient adaptive sampling methods are also proposed, which have wide application in many machine-learning and data-mining tasks.

1. Introduction

The Nyström method is a widely used technique to speed up kernel machines. Its efficiency in computation has attracted much attention in the past few years [1,2,3,4,5,6,7,8]. Given a kernel matrix $\mathbf{K}\in {\mathbb{R}}^{n\times n}$, the Nyström method tries to approximate the kernel by random sampling to save computation cost. At the cost of computational efficiency, it suffers from a relatively large matrix approximation error in real applications [9,10]. Given the target rank k and target precision parameter $0<ϵ\le 1$, Wang and Zhang [4] gave a theoretical analysis that, with the Nyström method, it is impossible to obtain a $1+ϵ$ bound relative to $\parallel \mathbf{K}-{\mathbf{K}}_k{\parallel }_F^2$ unless the number of sampled columns $c>\Omega \left(\sqrt{nk/ϵ}\right)$. Here, ${\mathbf{K}}_k$ denotes the best rank-k approximation to the kernel matrix $\mathbf{K}$. Several modified Nyström methods were proposed in recent years [3,4,11,12]. In the work of [11], a modified Nyström method just needs $k/ϵ$ columns of the kernel matrix to obtain a $1+ϵ$ bound relative to $\parallel \mathbf{K}-{\mathbf{K}}_k{\parallel }_F^2$. To the best of our knowledge, it is the fastest algorithm, costing $O\left(n{k}^2\right)+T_{Multiply}(\mathrm{nnz}\left(\mathbf{K}\right)logn)$ to achieve a $1+ϵ$ relative error of $\parallel \mathbf{K}-{\mathbf{K}}_k{\parallel }_F^2$, where $\mathrm{nnz}\left(\mathbf{K}\right)$ means the number of non-zero entries of $\mathbf{K}$. Although these modified Nyström methods are superior in approximation accuracy, it needs a much higher computational burden compared to the conventional Nyström method.

In this paper, we propose a much faster modified Nyström method which runs in $\mathcal{O}(n^{\frac{1}{2}}{k}^3/{ϵ}^{\frac{5}{2}})+T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{K}\right)logn)$ time to achieve a $1+ϵ$ bound relative to $\parallel \mathbf{K}-{\mathbf{K}}_k{\parallel }_F^2$. When $ϵ>\sqrt{2}-1$, our algorithm will be accelerated to

$$\mathcal{O}\left(k^3\right)+T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{K}\right)logn),$$

which is guaranteed by Lemma 3. Our algorithm is given in Algorithm 3. It needs $T_{Multiply}\left(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)logn)\right)$ times to conduct matrix multiplication which is easily implemented in parallel. The computation complexity of matrix multiplication in Algorithm 3 is near linear in input sparsity. In addition, for the arithmetic operations which are hard to implement in parallel, such as SVD, pseudoinverse and QR decomposition, Algorithm 3 needs $\mathcal{O}(n^{\frac{1}{2}}{k}^3/{ϵ}^{\frac{5}{2}})$ time which is sublinear in the input size n. At the cost of sacrificing a certain accuracy, $\mathcal{O}\left(k^3\right)$ can be reached with the same computational complexity as the conventional Nyström method, needing $\mathcal{O}\left(k^3\right)$ arithmetic operations when sampling $\mathcal{O}\left(k\right)$ columns. Our empirical studies further validate the efficiency of our algorithm.

In this paper, we improve several key algorithms which constitute a faster modified Nyström method. We summarized our contributions as follow.

• First and most importantly, we propose an efficient modified Nyström method with theoretical guarantees.
• Second, a more computationally efficient adaptive sampling method is proposed in Lemma 2. Adaptive sampling is a cornerstone of column selection, CUR decomposition and the Nyström method [4,5,11,13], and it is also very popular in other matrix problems [14].
• Finally, our proposed practical Nyström method can achieve computation efficiency in real applications, as shown by our experiments.

The rest of this paper is structured as follows. In Section 2, we provide the notations used in this study. Section 3, several key algorithms that constitute the modified Nyström are improved. Section 4 gives our modified Nyström method. We conduct empirical analysis and comparison in Section 5, and conclude our work in Section 6. All detailed proofs are omitted except computation complexity analysis.

2. Notation and Preliminaries [15]

Firstly, we introduce the notation and concepts that will be utilized here and hereafter. ${\mathbf{I}}_m$ is used to represent the identity $m\times m$ matrix. Sometimes we just use $\mathbf{I}$ for simplicity. We also use $\mathbf{0}$ to signify a zero vector or a zero matrix with an appropriate size. The number of non-zero entries in $\mathbf{A}$ is indicated by the notation $\mathrm{nnz}\left(\mathbf{A}\right)$.

Let $k\le \rho$ and $\rho =\mathrm{rank}\left(\mathbf{A}\right)\le \min \{m,n\}$. The singular value decomposition (SVD) of $\mathbf{A}$ may be expressed as

$$\mathbf{A}=\sum _{i=1}^{\rho }{\sigma }_i{\mathbf{u}}_i{\mathbf{v}}_i^T=\left[\begin{array}{cc}{\mathbf{U}}_k& {\mathbf{U}}_{k\perp }\end{array}\right]\left[\begin{array}{cc}{\mathbf{\Sigma }}_k& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{k\perp }\end{array}\right]\left[\begin{array}{c}{\mathbf{V}}_k^T\\ {\mathbf{V}}_{k\perp }^T\end{array}\right],$$

where the top k singular values are represented by ${\mathbf{U}}_k$ ($m\times k$), ${\mathbf{V}}_k$ ($n\times k$) and ${\mathbf{\Sigma }}_k$ ($k\times k$). The best (or closest) rank-k approximation to $\mathbf{A}$ is denoted by ${\mathbf{A}}_k={\mathbf{U}}_k{\mathbf{\Sigma }}_k{\mathbf{V}}_k^T$. The i-th greatest singular value of $\mathbf{A}$ is denoted by ${\sigma }_i={\sigma }_i\left(\mathbf{A}\right)$. The SVD is the same as the eigenvalue decomposition when $\mathbf{A}$ is symmetric positive semi-definite (SPSD), in which case we obtain ${\mathbf{U}}_{\mathbf{A}}={\mathbf{V}}_{\mathbf{A}}$.

Furthermore, let ${\mathbf{A}}^{†}$ be the Moore–Penrose inverse of $\mathbf{A}$, defined as ${\mathbf{A}}^{†}={\mathbf{V}}_{\rho }{\mathbf{\Sigma }}_{\rho }^{-1}{\mathbf{U}}_{\rho }^T$. When $\mathbf{A}$ is non-singular, the matrix inverse is the same as the Moore–Penrose inverse.

The matrix norms are defined in the manner as follows. Assume that the spectral norm is ${\parallel \mathbf{A}\parallel }_2={max}_{\mathbf{x}\in {\mathbb{R}}^n,{\parallel \mathbf{x}\parallel }_2=1},{\parallel \mathbf{A}\mathbf{x}\parallel }_2={\sigma }_1$ and the Frobenius norm is ${\parallel \mathbf{A}\parallel }_F={\left({\sum }_{i,j}{a}_{ij}^2\right)}^{1/2}={\left({\sum }_i{\sigma }_i^2\right)}^{1/2}$.

When given the matrices, $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ and $\mathbf{C}\in {\mathbb{R}}^{m\times r}$ with $r>k$, we explicitly define matrix ${\Pi }_{C,k}^{\zeta }\left(\mathbf{A}\right)$ as the closest representation of $\mathbf{A}$ in the column space of $\mathbf{C}$ with the rank of the most k. The function ${\Pi }_{C,k}^{\zeta }\left(\mathbf{A}\right)$ minimizes the residual $\parallel \mathbf{A}-\widehat{\mathbf{A}}{\parallel }_{\zeta }$ across all $\widehat{\mathbf{A}}$ in the column space of C. Here, “$\zeta$” denotes either the spectral norm or the Frobenius norm.

When given three matrices, $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, $\mathbf{X}\in {\mathbb{R}}^{m\times p}$, and $\mathbf{Y}\in {\mathbb{R}}^{q\times n}$, the projection of $\mathbf{A}$ onto $\mathbf{X}$’s column space is represented as $\mathbf{X}{\mathbf{X}}^{†}\mathbf{A}={\mathbf{U}}_{\mathbf{X}}{\mathbf{U}}_{\mathbf{X}}^T\mathbf{A}\in {\mathbb{R}}^{m\times n}$, and the one onto $\mathbf{Y}$’s row space is denoted by $\mathbf{A}{\mathbf{Y}}^{†}\mathbf{Y}=\mathbf{A}{\mathbf{V}}_{\mathbf{Y}}{\mathbf{V}}_{\mathbf{Y}}^T\in {\mathbb{R}}^{m\times n}$.

We now give the definition of leverage score sampling and subspace embedding, which are key tools to construct our Nyström algorithm.

Definition 1

(Leverage score sampling, [13,15]). Allow $\mathbf{V}\in {\mathbb{R}}^{n\times k}$ to be column orthonormal with $n>k$, and ${\mathbf{v}}_{i,*}$ to signify the i-th row of $\mathbf{V}$. Allow ${\ell }_i={\parallel {\mathbf{v}}_{i,*}\parallel }_F^2/k$. Given that the ${\ell }_i$ are leverage scores, let r be an integer in the range $1\le r\le n$. Create the sampling matrix $\Omega \in {\mathbb{R}}^{n\times r}$ and the rescaling matrix $\mathbf{D}\in {\mathbb{R}}^{r\times r}$ as follows. Pick an index i from the set of $\{1,2\cdots ,n\}$ with probability ${\ell }_i$, for each column $j=1,\cdots ,r$ of $\Omega$ and $\mathbf{D}$, separately and with replacements. Let ${\Omega }_{ij}=1$ and ${\mathbf{D}}_{jj}=1/\sqrt{{\ell }_{i}r}$. The number of operations required by this procedure is $\mathcal{O}(nk+n)$. This procedure is designated as

$$[\Omega ,\mathbf{D}]=LeverageScoreSampling(\mathbf{V},r).$$

Definition 2

([16]). Assuming $\epsilon >0$ and $\delta >0$, define a distribution on $\ell \times n$ matrix $\mathbf{S}$ as $\Pi$, where ℓ depends on n, d, ε and δ. Assume that, any given $n\times d$ matrix $\mathbf{A}$, with a probability of at least $1-\delta$, a matrix $\mathbf{S}$ chosen from distribution Π is a $(1+\epsilon )$ ${\ell }_2$-subspace embedding for $\mathbf{A}$. Meaning that, for every $\mathbf{x}\in {\mathbb{R}}^d$, ${\parallel \mathbf{S}\mathbf{A}\mathbf{x}\parallel }_2^2=(1\pm \epsilon ){\parallel \mathbf{A}\mathbf{x}\parallel }_2^2$ with probability $1-\delta$. After that, we designate Π as an $(\epsilon ,\delta )$-oblivious ${\ell }_2$-subspace embedding.

The sparse subspace embedding matrix $\mathbf{S}$ and subsampled Hadamard matrix $\mathbf{H}$ are the two most popular subspace embedding matrices. For an $n\times k$ matrix $\mathbf{A}$ with k dimension subspace, we can construct a sparse subspace embedding matrix $\mathbf{S}$ for $\mathbf{A}$ with $m=O(k^2/{ϵ}^2)$ rows, and the subsampled Hadamard matrix $\mathbf{H}$ with $m=O(klogk)/{ϵ}^2$ [16]. Combining $\mathbf{S}$ with $\mathbf{H}$ still has the property.

Let’s discussed the computational costs about the matrix operations mentioned above. Matrix multiplication is an intrinsic parallel operation; hence, it can be easily implemented in parallel efficiently just as many mathematical software do. However, SVD decomposition and QR decomposition are much harder to implement in parallel. Hence, we denote the time complexity of such a matrix multiplication by $T_{Multiply}$. For a general $m\times n$ matrix $\mathbf{A}$ with $m\ge n$, computing the full SVD requires $\mathcal{O}\left(m{n}^2\right)$ flops, whereas computing the truncated SVD of rank k ($k<n$), requires $\mathcal{O}\left(mnk\right)$ flops. Additionally, computing ${\mathbf{A}}^{†}$ requires $\mathcal{O}\left(m{n}^2\right)$ flops, too. Given a $m\times m$ Hadamard–Walsh transform matrix $\mathbf{H}$, $T_{Multiply}\left(\tilde{\mathcal{O}}\left(mn\right)\right)$ is the cost for the Hadamard–Walsh transform $\mathbf{H}\mathbf{A}$, which is substantially quicker than $T_{Multiply}\left(\mathcal{O}\left(m^{2}n\right)\right)$ for the typical matrix multiplication. A sparse subspace embedding matrix $\mathbf{S}$ for an $n\times d$ matrix $\mathbf{A}$, $\mathbf{S}\mathbf{A}$ needs $T_{Multiply}\left(\mathcal{O}\left(\mathrm{nnz}\left(\mathbf{A}\right)\right)\right)$ arithmetic operations.

3. Main Lemmas and Theorems

In this part, we will outline our principal theorems and lemmas, which are the key tools to implement Algorithm 3. In addition, these lemmas and theorems are of independent interest and have wide application.

First, we give a fast randomized SVD method which is depicted in Algorithm 1 which is the fastest randomized SVD method as far as we know.

Lemma 1.

Given matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, target rank k and error parameter $0<ϵ\le 1$, $\mathbf{Z}$ is returned from Algorithm 1; then, the following formula holds with high probability.

$$\parallel \mathbf{A}-\mathbf{Z}{\mathbf{Z}}^T{\mathbf{A}\parallel }_F^2\le (1+ϵ){\parallel \mathbf{A}-{\mathbf{A}}_k\parallel }_F^2$$

In addition, $\mathbf{Z}$ can be computed in $\tilde{\mathcal{O}}(k^3/{ϵ}^5)+T_{Multiply}(\mathcal{O}\left(\mathrm{nnz}\left(\mathbf{A}\right)\right)+\tilde{\mathcal{O}}(m{k}^2/{ϵ}^4+k^3/{ϵ}^3))$. We denote Algorithm 1 as

$$\mathbf{Z}=SparseSVD(\mathbf{A},k,ϵ).$$

Algorithm 1 Sparse SVD

1: Input: a real matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, error parameter $ϵ$ and target rank k; 2: Compute $\mathbf{A}{\mathbf{R}}^T$, where $\mathbf{R}=\mathbf{\Pi }\mathbf{S}\in {\mathbb{R}}^{c\times n}$ with $c=\mathcal{O}(klogk/ϵ)$. $\mathbf{S}\in {\mathbb{R}}^{s\times n}$ is a sparse subspace embedding matrix with $s=\mathcal{O}(k^2+k/ϵ)$ and $\mathbf{\Pi }\in {\mathbb{R}}^{c\times s}$ is a subsampled randomized Hadamard matrix with $c=\mathcal{O}(klogk/ϵ)$; 3: Compute an orthonormal basis $\mathbf{U}$ for $\mathbf{A}{\mathbf{R}}^T$ by $\mathbf{U}=\mathbf{A}{\mathbf{R}}^T{\mathbf{C}}^{-1}$, where $\mathbf{C}$ is the Cholesky decomposition of $\mathbf{R}{\mathbf{A}}^T\mathbf{A}{\mathbf{R}}^T$; 4: Compute $\mathbf{\Gamma }={\mathbf{U}}^T\mathbf{A}{\mathbf{W}}^T\in {\mathbb{R}}^{c\times d}$, where $\mathbf{W}=\mathbf{H}\mathbf{F}\in {\mathbb{R}}^{d\times n}$ with $d=\mathcal{O}(klogk/{ϵ}^3)$. $\mathbf{F}\in {\mathbb{R}}^{n\times t}$ is a sparse subspace embedding matrix with $t=\mathcal{O}(k^2{log}^{2}k/{ϵ}^3)$ and $\mathbf{H}\in {\mathbb{R}}^{d\times t}$ is a subsampled randomized Hadamard matrix with $d=\mathcal{O}(klogk/{ϵ}^3)$. 5: Compute the SVD of $\mathbf{\Gamma }$ and let $\Delta \in {\mathbb{R}}^{c\times k}$ contain the top k left singular vectors of $\mathbf{\Gamma }$; 6: Output: $\mathbf{Z}=\mathbf{U}\Delta$.

Proof. 

Lemma A2 shows that $\parallel \mathbf{A}-\mathbf{U}{\mathbf{U}}^T{\mathbf{A}\parallel }_F^2\le (1+ϵ){\parallel \mathbf{A}-{\mathbf{A}}_k\parallel }_F^2$, where $\mathbf{U}$ is of $\mathcal{O}(klogk/ϵ)$ columns. Applying Lemma A1 and replacing $\mathbf{V}$ with $\mathbf{U}$, we can obtain the result that

$$\parallel \mathbf{A}-\mathbf{Z}{\mathbf{Z}}^T{\mathbf{A}\parallel }_F^2\le (1+ϵ){\parallel \mathbf{A}-{\mathbf{A}}_k\parallel }_F^2$$

For computation time analysis, computing $\mathbf{A}{\mathbf{R}}^T$ takes $T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)+\tilde{\mathcal{O}}\left(mk(k+{ϵ}^{-1})\right))$, and then $T_{Multiply}\left(\tilde{O}(m{k}^2/{ϵ}^2+k^3/{ϵ}^3)\right)$ computes the $\mathbf{U}=\mathbf{A}{\mathbf{C}}^{-1}$, where $\mathbf{C}$ is the Cholesky decomposition of ${\mathbf{A}}^T\mathbf{A}$. Computing ${\mathbf{U}}^T\left(\mathbf{A}{\mathbf{W}}^T\right)$ requires $T_{Multiply}\left(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)+m{k}^2/{ϵ}^3+m{k}^2/{ϵ}^4)\right)$. Computing the SVD of $\mathbf{\Gamma }$ requires $\tilde{\mathcal{O}}(k^3/{ϵ}^5)$. In addition, computing $\mathbf{Z}=\mathbf{U}\Delta$ requires $T_{Multiply}\left(\tilde{\mathcal{O}}(m{k}^2/{ϵ}^3)\right)$. Hence, Algorithm 1 takes

$$\tilde{\mathcal{O}}(k^3/{ϵ}^5)+T_{Multiply}(\mathcal{O}\left(\mathrm{nnz}\left(\mathbf{A}\right)\right)+\tilde{\mathcal{O}}(m{k}^2/{ϵ}^4+k^3/{ϵ}^3))$$

computation complexity.    □

A faster adaptive sampling, Algorithm 2, is developed based on the work of [13]. Boutsidis and Woodruff [13] tried to compute norms of each column of $\mathbf{G}\mathbf{B}=\mathbf{G}\mathbf{A}-\mathbf{G}{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}$. To further reduce the computation cost, we introduce the sketched $\mathbf{G}\widehat{\mathbf{B}}=\mathbf{G}\mathbf{A}-\mathbf{G}{\mathbf{C}}_1{\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}\left(\mathbf{R}\mathbf{A}\right)$ to approximate $\mathbf{G}\mathbf{B}$. By such sketching, $\mathbf{G}{\mathbf{C}}_1{\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}\left(\mathbf{R}\mathbf{A}\right)$ can be computed more efficiently than $\mathbf{G}{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}$.

Algorithm 2 Adaptive Sampling

1: Input: a real matrix $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, ${\mathbf{C}}_1\in {\mathbb{R}}^{m\times c_1}$ and the number of selected columns c; 2: Construct $\widehat{\mathbf{B}}=\mathbf{A}-{\mathbf{C}}_1{\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}\left(\mathbf{R}\mathbf{A}\right)$, where $\mathbf{R}=\mathbf{\Pi }\mathbf{S}\in {\mathbb{R}}^{t\times m}$ with $t=2c_{1}log{c}_1$. $\mathbf{S}\in {\mathbb{R}}^{s\times m}$ is a sparse subspace embedding matrix with $s=c_1^2+2c_1$ and $\mathbf{\Pi }\in {\mathbb{R}}^{t\times s}$ is a subsampled randomized Hadamard matrix; 3: Construct $\tilde{\mathbf{B}}=\mathbf{G}\widehat{\mathbf{B}}$ where $\mathbf{G}\in {\mathbb{R}}^{g\times m}$ is a normalized Gaussian matrix with $g=9logn$; 4: Compute sampling probabilities $p_j=\parallel {\tilde{\mathbf{b}}}_j{\parallel }_F^2/{\parallel \tilde{\mathbf{B}}\parallel }_F^2$ for $j=1,\cdots ,n$, where ${\tilde{\mathbf{b}}}_j$ is the $j-$th column of $\tilde{\mathbf{B}}$; 5: Output: Obtain ${\mathbf{C}}_2$ by selecting c columns from $\mathbf{A}$ in c i.i.d. trials; in each trial the index j is chosen with probability $p_j$.

Lemma 2.

Given $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, ${\mathbf{C}}_1\in {\mathbb{R}}^{m\times c_1}$ and $\mathbf{V}\in {\mathbb{R}}^{r\times n}$ such that $rank\left(\mathbf{V}\right)=rank\left(\mathbf{A}{\mathbf{V}}^{†}\mathbf{V}\right)=\rho$, with $\rho \le c\le n$, let ${\mathbf{C}}_2\in {\mathbb{R}}^{m\times c_2}$ be returned from Algorithm 2 containing $c_2$ columns of $\mathbf{A}$. Then, the matrix $\mathbf{C}=[{\mathbf{C}}_1,{\mathbf{C}}_2]\in {\mathbb{R}}^{m\times (c_1+c_2)}$ satisfies that for any integer $k>0$, and with a high probability which is at least $0.9$.

$$\begin{array}{c}\hfill \parallel \mathbf{A}-\mathbf{C}{\mathbf{C}}^{†}\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2\le \parallel \mathbf{A}-\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2+\frac{40\rho }{c_2}{\parallel \mathbf{A}-{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}\parallel }_F^2.\end{array}$$

In addition, this randomized algorithm can be implemented in

$$\tilde{\mathcal{O}}\left(c_1^3\right)+T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)logn+\tilde{\mathcal{O}}(n{c}_1^2+n{c}_{1}logn+c_1^3))$$

computation time. We denote this randomized algorithm as

$${\mathbf{C}}_2=AdaptiveSampling(\mathbf{A},\mathbf{V},{\mathbf{C}}_1,c_2).$$

Proof. 

Let $\mathbf{B}=\mathbf{A}-{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}$ be the residual matrix and ${\mathbf{b}}_i$ is the i-th column of $\mathbf{B}$. By Theorem A4, with high probability, it holds that

$$\begin{array}{c}\hfill {\parallel \mathbf{B}\parallel }_F^2\le \parallel \widehat{\mathbf{B}}{\parallel }_F^2\le {(1+2ϵ)\parallel \mathbf{B}\parallel }_F^2={2\parallel \mathbf{B}\parallel }_F^2\parallel {\mathbf{b}}_i{\parallel }_F^2\le \parallel {\widehat{\mathbf{b}}}_i{\parallel }_F^2\le (1+2ϵ)\parallel {\mathbf{b}}_i{\parallel }_F^2=2{\parallel {\mathbf{b}}_i\parallel }_F^2\end{array}$$

Besides, by the JL property of $\mathbf{G}$, we have $\frac{1}{3}\parallel {\widehat{\mathbf{b}}}_i{\parallel }_F\le \parallel {\tilde{\mathbf{b}}}_i{\parallel }_F\le \frac{4}{3}{\parallel {\widehat{\mathbf{b}}}_i\parallel }_F$. Hence, after utilizing the below distribution for sampling,

$$p_i=\frac{\parallel {\tilde{\mathbf{b}}}_i{\parallel }_F}{\parallel \tilde{\mathbf{B}}{\parallel }_F}\ge \frac{2}{3}·\frac{3}{4}·\frac{\parallel {\widehat{\mathbf{b}}}_i{\parallel }_F^2}{\parallel \widehat{\mathbf{B}}{\parallel }_F^2}\ge \frac{1}{2}·\frac{1}{2}·\frac{\parallel {\mathbf{b}}_i{\parallel }_F^2}{{\parallel \mathbf{B}\parallel }_F^2}=\frac{1}{4}\frac{\parallel {\mathbf{b}}_i{\parallel }_F^2}{{\parallel \mathbf{B}\parallel }_F^2}$$

Using Lemma A3, we obtain

$$\begin{array}{c}\hfill \mathbb{E}\left[\parallel \mathbf{A}-\mathbf{C}{\mathbf{C}}^{†}\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2\right]\le \parallel \mathbf{A}-\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2+\frac{4k}{c_2}{\parallel \mathbf{A}-{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}\parallel }_F^2.\end{array}$$

Using the Markov inequality, we have that

$$\begin{array}{c}\hfill \parallel \mathbf{A}-\mathbf{C}{\mathbf{C}}^{†}\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2\le \parallel \mathbf{A}-\mathbf{A}{\mathbf{V}}^{†}{\mathbf{V}\parallel }_F^2+\frac{40\rho }{c_2}{\parallel \mathbf{A}-{\mathbf{C}}_1{\mathbf{C}}_1^{†}\mathbf{A}\parallel }_F^2.\end{array}$$

holds with a probability of at least $0.9$.

As to the running time, it needs $T_{Multiply}(\mathcal{O}\left(\mathrm{nnz}\left(\mathbf{A}\right)\right)+\tilde{\mathcal{O}}\left(n{c}_1^2\right))$ arithmetic operations to compute $\mathbf{R}\mathbf{A}$. To compute $\mathbf{R}{\mathbf{C}}_1$ costs $T_{Multiply}(\mathcal{O}\left(\mathrm{nnz}\left({\mathbf{C}}_1\right)\right)+\tilde{\mathcal{O}}\left(c_1^3\right))$. To compute ${\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}$, it requires $\tilde{O}\left(c_1^3\right)$. In addition, computing $\mathbf{G}\mathbf{A}$ and $\mathbf{G}{\mathbf{C}}_1$ require $T_{Multiply}$ $\left(\mathcal{O}\right(\mathrm{nnz}\left(\mathbf{A}\right)logn\left)\right)$ and $T_{Multiply}(m{c}_{1}logn)$, respectively. In addition, to compute $\left(\mathbf{G}{\mathbf{C}}_1\right){\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}\left(\mathbf{R}\mathbf{A}\right)$ needs

$$T_{Multiply}\left(\tilde{\mathcal{O}}(n{c}_{1}logn+c_{2}logn)\right)$$

computation. In addition, $\mathbf{G}\mathbf{A}-\mathbf{G}{\mathbf{C}}_1{\left(\mathbf{R}{\mathbf{C}}_1\right)}^{†}\mathbf{R}\mathbf{A}$ needs another $T_{Multiply}\left(\mathcal{O}(nlogn)\right)$ arithmetic operations. Thus, all these need $\tilde{\mathcal{O}}\left(c_1^3\right)+T_{Multiply}\left(\mathcal{O}\right(\mathrm{nnz}\left(\mathbf{A}\right)logn+\tilde{\mathcal{O}}$ $(n{c}_1^2+n{c}_{1}logn+c_1^3))$    □

Lemma 3

([15,17]). Given the matrices $\mathbf{C}\in {\mathbb{R}}^{m\times c}$, $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ and $\mathbf{R}\in {\mathbb{R}}^{n\times r}$, let’s suppose that $\mathbf{S}$ is the leverage-score sketching matrix of $\mathbf{C}$ with $s=\mathcal{O}(c/ϵ+clogc)$ rows, and $\mathbf{T}$ is the leverage-score sketching matrix of $\mathbf{R}$ with $t=\mathcal{O}(r/ϵ+rlogr)$ columns. Let

$${\mathbf{U}}^{★}={\mathbf{C}}^{†}\mathbf{A}{\mathbf{R}}^{†}=\underset{\mathbf{U}}{\mathrm{argmin}}{\parallel \mathbf{A}-\mathbf{C}\mathbf{U}\mathbf{R}\parallel }_F$$

and

$$\widehat{\mathbf{U}}={\left(\mathbf{S}\mathbf{C}\right)}^{†}\mathbf{S}\mathbf{A}\mathbf{T}{\left(\mathbf{R}\mathbf{T}\right)}^{†},$$

then we can obtain

$$\parallel \mathbf{A}-\mathbf{C}\widehat{\mathbf{U}}{\mathbf{R}\parallel }_F\le (1+ϵ){\parallel \mathbf{A}-\mathbf{C}{\mathbf{U}}^{★}\mathbf{R}\parallel }_F.$$

The number of sampled rows in Lemma 3 is independent on the input dimension of $\mathbf{A}$ and is linear to c. By losing some accuracy, a much faster algorithm can be implemented.

4. Practical Modified Nyström Method

We use our new lemmas and theorems developed in Section 3 to implement an efficient modified Nyström algorithm.

4.1. Description of The Algorithm

A $n\times n$ real symmetric matrix $\mathbf{A}$, an error parameter $0<ϵ<1$ and a target rank k are the inputs of Algorithm 3. Meanwhile, a matrix $\mathbf{C}\in {\mathbb{R}}^{n\times c}$ with $c=\mathcal{O}(k/ϵ+klogk)$ columns of $\mathbf{A}$, and a matrix $\mathbf{U}\in {\mathbb{R}}^{c\times c}$ are the results. There are primarily 3 steps in Algorithm 3: (i) using the definition of the leverage score sampling, it samples a number of columns of $\mathbf{A}$ to obtain ${\mathbf{C}}_1$; and using the adaptive sampling method to obtain ${\mathbf{C}}_2$ and ${\mathbf{R}}_2$; (ii) it calculates the leverage scores of $\mathbf{C}$ using the method in [18]; and (iii) it constructs the intersection matrix $\mathbf{U}$. Note that $\widehat{\mathbf{U}}$ in Lemma 3 is asymmetric even when $\mathbf{A}$ is positive semi-definite. Thus, when applied to kernel approximation, we need to construct a positive semi-definite $\mathbf{U}$ shown in Algorithm 3.

Algorithm 3 Practical Nyström

1: Input: a real symmetric matrix $\mathbf{A}\in {\mathbb{R}}^{n\times n}$, error parameter $ϵ$ and target rank k; 2: $\mathbf{Z}=SparseSVD(\mathbf{A},k,1)$; 3: $[\Omega ,\mathbf{\Gamma }]=LeverageScoreSampling(\mathbf{Z},\mathcal{O}(klogk\left)\right)$ and construct ${\mathbf{C}}_1=\mathbf{A}\Omega$; 4: ${\mathbf{C}}_2=AdaptiveSampling(\mathbf{A},{\mathbf{V}}_k^T,{\mathbf{C}}_1,\mathcal{O}(k/ϵ))$ and ${\mathbf{C}}_3=AdaptiveSampling(\mathbf{A},{\mathbf{V}}_k^T,{\mathbf{C}}_1,\mathcal{O}(k/ϵ))$, constructing $\mathbf{C}=[{\mathbf{C}}_1,{\mathbf{C}}_2,{\mathbf{C}}_3]\in {\mathbb{R}}^{n\times \mathcal{O}(k/ϵ+klogk)}$; 5: Compute approximate leverage scores of $\mathbf{C}$ using the method of [18] and construct the leverage sketch matrix ${\mathbf{S}}_1$ and ${\mathbf{S}}_2$ of $n\times s$ size, where $s=\mathcal{O}(\frac{c}{ϵ}+clogc)$; 6: Compute $\widehat{\mathbf{U}}={\left({\mathbf{S}}_1\mathbf{C}\right)}^{†}{\mathbf{S}}_1\mathbf{A}{\mathbf{S}}_2^T{\left({\mathbf{C}}^T{\mathbf{S}}_2^T\right)}^{†}$. 7: Compute $\mathbf{U}={\mathbf{\Pi }}_{{\mathbb{H}}_{+}^s}\left(\widehat{\mathbf{U}}\right)$ by conducting eigenvalue decomposition of $\tilde{\mathbf{U}}=\frac{\widehat{\mathbf{U}}+{\widehat{\mathbf{U}}}^T}{2}$ and setting the negative eigenvalues of $\widehat{\mathbf{U}}$ to zero. 8: Output: $\mathbf{C}$ and $\mathbf{U}$.

4.2. Analysis of Running-Time

Here, we provide a detailed analysis of the Algorithm 3’s arithmetic operations.

1

The computation complexity of Algorithm 3 is $\tilde{\mathcal{O}}\left(k^3\right)+T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)logn+\tilde{\mathcal{O}}(n{k}^2+nklogn+k^3))$ to find $\mathcal{O}(k/ϵ+klogk)$ columns of $\mathbf{A}$ to construct $\mathbf{C}$.

(a)

To obtain $\mathbf{Z}\in {\mathbb{R}}^{n\times k}$ from Theorem 1, it takes $\tilde{\mathcal{O}}\left(k^3\right)+T_{Multiply}(\mathcal{O}\left(\mathrm{nnz}\left(\mathbf{A}\right)\right)+\tilde{\mathcal{O}}(n{k}^2+k^3))$.

(b)

To obtain the leverage score and sample ${\mathbf{C}}_1$ and $C_2$, it takes $T_{Multiply}\left(\mathcal{O}\left(nk\right)\right)$.

(c)

To construct ${\mathbf{C}}_3$ and ${\mathbf{R}}_2$ Lemma 2, it takes $\tilde{\mathcal{O}}\left(k^3\right)+T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)logn+\tilde{\mathcal{O}}(n{k}^2+nklogn+k^3))$.

2

The computation complexity of Algorithm 3 is $\mathcal{O}(k^3/{ϵ}^4)+T_{Multiply}(\mathcal{O}(n{k}^2/{ϵ}^2)+\tilde{\mathcal{O}}(n{k}^2+k^3/{ϵ}^5))$ to construct $\mathbf{U}$ when $s=\mathcal{O}(\frac{c}{ϵ}+clogc)$ is the row dimension of ${\mathbf{S}}_1$ and ${\mathbf{S}}_2$ in Algorithm 3.

(a)

To obtain the leverage scores of $\mathbf{C}$, it takes $\mathcal{O}(k^3/{ϵ}^3)+T_{multiply}(\mathcal{O}(n{(k/ϵ)}^2+\tilde{\mathcal{O}}\left(n{k}^2\right))$.

(b)

To compute ${\left({\mathbf{S}}_1^T\mathbf{C}\right)}^{†}$ and ${\left({\mathbf{S}}_2^T\mathbf{C}\right)}^{†}$, it takes $\tilde{\mathcal{O}}(k^3/{ϵ}^4)$.

(c)

To compute matrix multiplication, it takes $T_{Multiply}\left(\mathcal{O}(k^3/{ϵ}^5)\right)$.

(d)

To compute the eigenvalue decomposition of $\begin{array}{c}\hfill errortypeTitle\mathbf{U}\end{array}$, it takes $\tilde{\mathcal{O}}(k^3/{ϵ}^3)$.

The algorithm’s overall asymptotic arithmetic operation is

$$\begin{array}{c}\hfill T_{Multiply}(\mathcal{O}(\mathrm{nnz}\left(\mathbf{A}\right)logn+n{k}^2/{ϵ}^2+k^3/{ϵ}^5)+\tilde{\mathcal{O}}(n{k}^2+nklogn+k^3/{ϵ}^4)).\end{array}$$

4.3. Error Bound

Primary approximate result regarding Algorithm 3 is shown as the following theorem.

Theorem 1.

Given an error parameter ϵ and a target rank k, run Algorithm 3, then the below inequality holds with high probability.

$$\parallel \mathbf{A}-\mathbf{C}\mathbf{U}{\mathbf{C}}^T{\parallel }_F\le (1+ϵ){\parallel \mathbf{A}-{\mathbf{A}}_k\parallel }_F$$

5. Empirical Study

In this section, we compare our Practical Nyström algorithm with the uniform+adaptive algorithm [11,19], near-optimal+adaptive algorithm [4,11,13] and conventional Nyström using just uniform sampling. All algorithms were implemented in Matlab and experiments were conducted on a workstation with 32 cores of 2G Hz and 24G RAM.

On each data set, we give the approximation error and the execution duration of each algorithm. The approximation error is

$$\begin{array}{c}\hfill \mathrm{Approximation}\mathrm{Error}=\frac{\parallel \mathbf{A}-\mathbf{C}\mathbf{U}{\mathbf{C}}^T{\parallel }_F}{{\parallel \mathbf{A}\parallel }_F},\end{array}$$

where $\mathbf{U}$ is the intersection matrix defined in the Nyström method.

On three data sets we test all three algorithms, and the results are listed in

Table 1. A summary of the datasets for kernel approximation.

Data Set	a9a	USPS	PenDigits
#instance	32,561	11,305	7494
$\gamma$	5	4	30
Source	UCI	TKH96a	UCI

. We create a RBF kernel matrix $\mathbf{A}$ for each dataset, with $a_{ij}=exp\left(\frac{\parallel {\mathbf{x}}_i-{\mathbf{x}}_j{\parallel }_2^2}{2{\gamma }^2}\right)$, where ${\mathbf{x}}_i$ and ${\mathbf{x}}_j$ are data instances and $\gamma$ is the parameter of the RBF kernel function. By the definition of $\mathbf{A}$, the size n of $\mathbf{A}$ is the number of instances of the dataset. Thus, the kernel matrices in our experiments are of large sizes. We set $\gamma$ different values for each data set as Table 1 describes. However, the effectiveness of our algorithm does not depend on the setting of $\gamma$. For each data set, we set $k=10, 30$ and 50. We sampled $c=ak$ columns from $\mathbf{A}$ and a ranges from 8 to 26. We ran each algorithm 5 times and report the average value of approximation error and running time. All results are illustrated in Figure 1, Figure 2 and Figure 3.

As evidenced by the empirical results in the figures, it is clear that our approach is efficient. In terms of accuracy, Our approach is comparable to the state-of-the-art algorithm—the near-optimal+adaptive algorithm [4,11,13]. As to the running time, our approach is much faster than near-optimal+adaptive algorithm and uniform+adaptive algorithm. Our algorithm’s running time grows slower than the near-optimal+adaptive algorithm and uniform+adaptive algorithm. The advantage of the running time of our algorithm grows as the dimension of kernel matrix $\mathbf{A}$ increases. Calculating kernel matrix $\mathbf{A}$ of size $7494\times 7494$ from the ‘PenDigits’ data set, our alogrithm is twice as fast as the near-optimal+adaptive algorithm. As to the ‘a9a’ data set of 32,561 instances, our algorithm is four times faster than near-optimal+adaptive. In addition, as c increases, the running-time superiority of our algorithm also increases. Our algorithm also has similar a advantage over the uniform+adaptive algorithm. Hence, our algorithm is suitable to scale to kernel matrices of high dimensions.

6. Conclusions

In this paper, we proposed an efficient modified Nyström method with a theoretical and emperical guarantee. In a high-level parallel-computation environment with sparse input matrices, our Nyström method can achieve comparable computation efficiency compared to the conventional Nyström method, theoretically. Hence, our Nyström method is suitable for machine-learning algorithms in big-data setting. In addition, we give a sketching generalized matrix approximation which extends the previous work [12]. Faster randomized SVD and more efficient adaptive sampling methods are proposed which have wide application in lots of areas. In addition, our modified Nyström algorithm can be easily extended to CUR decomposition which leads to more efficient CUR decomposition.