Numerical Algorithms for Computing an Arbitrary Singular Value of a Tensor Sum ^†

Abstract

We consider computing an arbitrary singular value of a tensor sum: $T:=I_n\otimes I_m\otimes A+I_n\otimes B\otimes I_{\ell }+C\otimes I_m\otimes I_{\ell }\in {\mathbb{R}}^{\ell mn\times \ell mn},$ where $A\in {\mathbb{R}}^{\ell \times \ell }$, $B\in {\mathbb{R}}^{m\times m}$, $C\in {\mathbb{R}}^{n\times n}$. We focus on the shift-and-invert Lanczos method, which solves a shift-and-invert eigenvalue problem of ${(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn})}^{-1}$, where $\tilde{\sigma }$ is set to a scalar value close to the desired singular value. The desired singular value is computed by the maximum eigenvalue of the eigenvalue problem. This shift-and-invert Lanczos method needs to solve large-scale linear systems with the coefficient matrix $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$. The preconditioned conjugate gradient (PCG) method is applied since the direct methods cannot be applied due to the nonzero structure of the coefficient matrix. However, it is difficult in terms of memory requirements to simply implement the shift-and-invert Lanczos and the PCG methods since the size of T grows rapidly by the sizes of A, B, and C. In this paper, we present the following two techniques: (1) efficient implementations of the shift-and-invert Lanczos method for the eigenvalue problem of $T^{\mathrm{T}}T$ and the PCG method for $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$ using three-dimensional arrays (third-order tensors) and the n-mode products, and (2) preconditioning matrices of the PCG method based on the eigenvalue and the Schur decomposition of T. Finally, we show the effectiveness of the proposed methods through numerical experiments.

1. Introduction

We consider computing an arbitrary singular value of a tensor sum:

$$\begin{array}{c}\hfill T:=I_n\otimes I_m\otimes A+I_n\otimes B\otimes I_{\ell }+C\otimes I_m\otimes I_{\ell }\in {\mathbb{R}}^{\ell mn\times \ell mn},\end{array}$$

(1)

where $A\in {\mathbb{R}}^{\ell \times \ell }$, $B\in {\mathbb{R}}^{m\times m}$, $C\in {\mathbb{R}}^{n\times n}$, $I_n$ is the $n\times n$ identity matrix, and the symbol “⊗” denotes the Kronecker product. The tensor sum T arises from a finite difference discretization of three-dimensional constant coefficient partial differential equations (PDE) defined as follows:

$$\begin{array}{c}\hfill [-\mathit{a}·\left(\nabla \ast \nabla )+\mathit{b}·\nabla +c\right]u(x,y,z)=g(x,y,z)\mathrm{in}\mathrm{\Omega },u(x,y,z)=0\mathrm{on}\partial \mathrm{\Omega },\end{array}$$

(2)

where $\mathrm{\Omega }=(0,1)\times (0,1)\times (0,1)$, $\mathit{a},\mathit{b}\in {\mathbb{R}}^3$, $c\in \mathbb{R}$, and the symbol “*” denotes elementwise products. If $\mathit{a}=(1,1,1)$, then $\mathit{a}·(\nabla \ast \nabla )=\Delta$. Matrix T tends to be too large even if $A,B$ and C are not. Hence it is difficult to compute singular values of T with regard to the memory requirement.

Previous studies [1,2] provided methods to compute the maximum and minimum singular values of T. By the previous studies, one can compute only the maximum and minimum singular values of T without shift. On the other hand, one can compute arbitrary singular values of T with the shift by this work. The previous studies are based on the Lanczos bidiagonalization method (see, e.g., [3]), which computes the maximum and minimum singular values of a matrix. For insights on Lanczos bidiagonalization method, see, e.g., [4,5,6]. The Lanczos bidiagonalization method for T was implemented using tensors and their operations to reduce the memory requirement.

The Lanczos method with the shift-and-invert technique, see, e.g., [3], is widely known for computing an arbitrary eigenvalue $\lambda$ of a symmetric matrix $M\in {\mathbb{R}}^{n\times n}$. This method solves the shift-and-invert eigenvalue problem: ${(M-\tilde{\sigma }{I}_n)}^{-1}\mathit{x}={(\lambda -\tilde{\sigma })}^{-1}\mathit{x}$, where x is the eigenvector of M corresponding to $\lambda$, and $\tilde{\sigma }$ is a shift point which is set to the nearby $\lambda$ ($\tilde{\sigma }\ne \lambda$). Since the eigenvalue problem has the eigenvalue ${(\lambda -\tilde{\sigma })}^{-1}$ as the maximum eigenvalue, the method is effective for computing the desired eigenvalue $\lambda$ near $\tilde{\sigma }$. For successful work using the shift-and-invert technique, see, e.g., [7,8,9,10,11,12,13].

Therefore, we obtain a computing method for an arbitrary singular value of T based on the shift-and-invert Lanczos method. The method solves the following shift-and-invert eigenvalue problem: ${(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn})}^{-1}\mathit{x}={({\sigma }^2-{\tilde{\sigma }}^2)}^{-1}\mathit{x}$, where $\sigma$ is the desired singular value of T, x is the corresponding right-singular vector, and $\tilde{\sigma }$ is close to $\sigma$ ($\tilde{\sigma }\ne \sigma$). This shift-and-invert Lanczos method requires the solution of large-scale linear systems with the coefficient matrix $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$. Here, $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$ can be a dense matrix whose number of elements is $O(n^6)$ even if T is a sparse matrix whose number of elements is $O(n^4)$ when $A,B,C\in {\mathbb{R}}^{n\times n}$ are dense.

Since it is difficult regarding the memory requirement to apply the direct method, e.g., the Cholesky decomposition, which needs generating matrix $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$, the preconditioned conjugate gradient (PCG) method, see, e.g., [14], is applied, even though it is difficult in terms of memory requirements to simply implement this shift-and-invert Lanczos method and the PCG method since the size of T grows rapidly by the sizes of A, B, and C.

We propose the following two techniques in this paper: (1) Efficient implementations of the shift-and-invert Lanczos method for the eigenvalue problem of $T^{\mathrm{T}}T$ and the PCG method for $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$ using three-dimensional arrays (third-order tensors) and the n-mode products, see, e.g., [15]. (2) Preconditioning matrices based on the eigenvalue decomposition and the Schur decomposition of T for faster convergence of the PCG method. Finally, we show the effectiveness of the proposed method through numerical experiments.

2. Preliminaries of Tensor Operations

A tensor means a multidimensional array. Particularly, the third-order tensor $\mathcal{X}\in {\mathbb{R}}^{I\times J\times K}$ plays an important role. In the rest of this section, the definitions of some tensor operations are shown. For more details, see, e.g., [15].

Firstly, a summation, a subtraction, an inner product, and a norm for $\mathcal{X},\mathcal{Y}\in {\mathbb{R}}^{I\times J\times K}$ are defined as follows:

$$\begin{array}{c}\hfill {(\mathcal{X}\pm \mathcal{Y})}_{ijk}:={\mathcal{X}}_{ijk}\pm {\mathcal{Y}}_{ijk},(\mathcal{X},\mathcal{Y}):=\sum _{i=1}^I\sum _{j=1}^J\sum _{k=1}^K{\mathcal{X}}_{ijk}{\mathcal{Y}}_{ijk},\parallel \mathcal{X}\parallel =\sqrt{(\mathcal{X},\mathcal{X})},\end{array}$$

where ${\mathcal{X}}_{ijk}$ denotes the $(i,j,k)$ element of $\mathcal{X}$. Secondly, the n-mode product of a tensor $\mathcal{X}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}$ and a matrix $M\in {\mathbb{R}}^{J\times I_n}$ is defined as

$$\begin{array}{c}\hfill {(\mathcal{X}{\times }_{n}M)}_{i_1\dots i_{n-1}j{i}_{n+1}\dots i_N}=\sum _{i_n=1}^{I_n}{\mathcal{X}}_{i_1{i}_2\dots i_N}{M}_{p{i}_n},\end{array}$$

where $n\in \{1,2,\dots N\},i_k\in \{1,2,\dots ,I_k\}$ for $k=1,2,\dots N,$ and $j\in \{1,2,\dots ,J\}$. Finally, $\mathrm{vec}$ and ${\mathrm{vec}}^{-1}$ operators are the following maps between a vector space ${\mathbb{R}}^{IJK}$ and a tensor space ${\mathbb{R}}^{I\times J\times K}$: $\mathrm{vec}:{\mathbb{R}}^{I\times J\times K}\to {\mathbb{R}}^{IJK}$ and ${\mathrm{vec}}^{-1}:{\mathbb{R}}^{IJK}\to {\mathbb{R}}^{I\times J\times K}.$ $\mathrm{vec}$ operator can vectorize a tensor by combining all column vectors of the tensor into one long vector. Conversely, ${\mathrm{vec}}^{-1}$ operator can reshape a tensor from one long vector.

3. Shift-and-Invert Lanczos Method for an Arbitrary Singular Value over Tensor Space

This section gives an algorithm for computing an arbitrary singular value of the tensor sum T. Let $\sigma$ and x be a desired singular value of T and the corresponding right singular vectors, respectively. Then, the eigenvalue problem of T is written by $T^{\mathrm{T}}T\mathit{x}={\sigma }^2\mathit{x}$. Here, introducing a shift $\tilde{\sigma }\approx \sigma$, the shift-and-invert eigenvalue problem is

$$\begin{array}{c}\hfill {(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn})}^{-1}\mathit{x}=\frac{1}{{\sigma }^2-{\tilde{\sigma }}^2}\mathit{x}.\end{array}$$

(3)

The shift-and-invert Lanczos method (see, e.g., [3]) computes the nearest singular value $\sigma$ based on Equation (3). Reconstructing this method over the $\ell \times m\times n$ tensor space, we obtain Algorithm 1 whose memory requirement is of $O(n^3)$ when $n=m=\ell$.

Algorithm 1: Shift-and-invert Lanczos method for an arbitrary singular value over tensor space

1:  Choose an initial tensor ${\mathcal{Q}}_0\in {\mathbb{R}}^{\ell \times m\times n}$;

2:  $\mathcal{V}:={\mathcal{Q}}_0,{\beta }_0:=\left|\right|\mathcal{V}\left|\right|$;

3:  for $k=1,2,\dots ,$ until convergence do

4:   $Q_k:=\mathcal{V}/{\beta }_{k-1};$

5:   $\mathcal{V}:={\mathrm{vec}}^{-1}\left\{{\left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)}^{-1}\mathrm{vec}({\mathcal{Q}}_k)\right\};$      (Computed by Algorithms 3 or 4 in Section 4)

6:   $\mathcal{V}:=\mathcal{V}-{\beta }_{k-1}{\mathcal{Q}}_{k-1}$;

7:   ${\alpha }_k:=({\mathcal{Q}}_k,\mathcal{V})$;

8:   $\mathcal{V}:=\mathcal{V}-{\alpha }_k{\mathcal{Q}}_k$;

9:   ${\beta }_k:=\left|\right|\mathcal{V}\left|\right|$;

10:  end for

11:  Approximate singular value $\sigma =\sqrt{{\displaystyle {\tilde{\sigma }}^2+\frac{1}{{\tilde{\lambda }}^{(k)}}}}$, where ${\tilde{\lambda }}^{(k)}$ is the maximum eigenvalue of ${\tilde{T}}_k$.

At step k, we have the following ${\tilde{T}}_k$ by Algorithm 1:

$$\begin{array}{c}\hfill {\tilde{T}}_k:=\left(\begin{array}{ccccc}{\alpha }_1& {\beta }_1& & & \\ {\beta }_1& {\alpha }_2& {\beta }_2& & \\ & \ddots & \ddots & \ddots & \\ & & {\beta }_{k-2}& {\alpha }_{k-1}& {\beta }_{k-1}\\ & & & {\beta }_{k-1}& {\alpha }_k\end{array}\right)\in {\mathbb{R}}^{k\times k}.\end{array}$$

To implement Algorithm 1, we need to iteratively solve the linear system

$$\begin{array}{c}\hfill \mathcal{V}:={\mathrm{vec}}^{-1}\left\{{\left(T^{\mathrm{T}}T-\tilde{\sigma }{I}_{\ell mn}\right)}^{-1}\mathrm{vec}({\mathcal{Q}}_k)\right\},\end{array}$$

(4)

whose coefficient matrix is $\ell mn\times \ell mn$, that is, the memory requirement is $O(n^6)$ when $n=m=\ell$. Here, the convergence rate of the shift and invert Lanczos method depends on the ratio of gaps between the maximum, the second maximum, and the minimum singular values ${\sigma }_1,{\sigma }_2,{\sigma }_m$ of ${(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn})}^{-1}$ as follows: $({\sigma }_1^2-{\sigma }_2^2)/({\sigma }_1^2-{\sigma }_m^2)$.

In the next section, we consider solving the linear systems with memory requirement of $O(n^3)$ when $n=m=\ell$.

4. Preconditioned Conjugate Gradient (PCG) Method over Tensor Space

This section provides an efficient solver of Equation (4) using tensors. This linear system is rewritten by $\mathit{v}={\left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)}^{-1}{\mathit{q}}_k$, where $\mathit{v}:=\mathrm{vec}(\mathcal{V})$ and ${\mathit{q}}_k:=\mathrm{vec}({\mathcal{Q}}_k)$. Then we solve $\left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k$, where v and ${\mathit{q}}_k$ are unknown and known vectors. Since the coefficient matrix is symmetric positive definite, we can use the preconditioned conjugate gradient method (PCG method, see, e.g., [14]), which is one of the widely used solvers. However, it is difficult to simply apply the method due to the complex nonzero structure of the coefficient matrix $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$. For applying the PCG method, we consider transforming the linear system $\left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k$ by the eigendecomposition and the complex Schur decomposition as shown in the next subsections.

4.1. PCG Method by the Eigendecomposition

Firstly, T is decomposed into $T:=XD{X}^{-1}$, where X and D are a matrix whose column vectors are eigenvectors and a diagonal matrix with eigenvalues, respectively. Then, it follows that

$$\begin{array}{cc}\hfill \left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k& \iff \left({(XD{X}^{-1})}^{\mathrm{H}}(XD{X}^{-1})-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k\hfill \\ \hfill & \iff \left(\overline{D}{X}^{\mathrm{H}}XD-{\tilde{\sigma }}^2{X}^{\mathrm{H}}X\right)\left(X^{-1}\mathit{v}\right)=X^{\mathrm{H}}{\mathit{q}}_k,\hfill \end{array}$$

where $\overline{D}$ is the complex conjugate of D. We rewrite the above linear system into $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$, where $\tilde{A}:=\overline{D}{X}^{\mathrm{H}}XD-{\tilde{\sigma }}^2{X}^{\mathrm{H}}X$, $\tilde{\mathit{y}}:=X^{-1}\mathit{v}$, and $\tilde{\mathit{b}}:=X^{\mathrm{H}}{\mathit{q}}_k$. Here, X is easily computed by small matrices $X_A,X_B,$ and $X_C$ whose column vectors are eigenvectors of A, B, and C as follows: $X=X_C\otimes X_B\otimes X_A$. Moreover, eigenvalues of T in D are obtained by summations of each eigenvalue of A, B, and C.

The PCG method for solving $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$ is shown in Algorithm 2. Since this algorithm computes $\tilde{\mathit{y}}$, we need to compute $\mathit{v}=X\tilde{\mathit{y}}$. Section 4.1.1 proposes a preconditioning matrix and Section 4.1.2 provides efficient computations using tensors.

Algorithm 2: PCG method over vector space for $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$

1:  Choose an initial vector ${\mathit{x}}_0\in {\mathbb{R}}^{\ell mn}$ and ${\mathit{p}}_0=\mathbf{0}\in {\mathbb{R}}^{\ell mn}$, and an initial scalar ${\beta }_0=0$;

2:  ${\mathit{r}}_0=\tilde{\mathit{b}}-\tilde{A}{\mathit{x}}_0$;

3:  ${\mathit{z}}_0=M^{-1}{\mathit{r}}_0$;

4:  for $k^{\prime }=1,2,\dots ,$ until convergence do

5:   ${\mathit{p}}_{k^{\prime }}={\mathit{z}}_{k^{\prime }-1}+{\beta }_{k^{\prime }-1}{\mathit{p}}_{k^{\prime }-1}$;

6:   ${\widehat{\mathit{p}}}_{k^{\prime }}=\tilde{A}{\mathit{p}}_{k^{\prime }}$;

7:   ${\alpha }_{k^{\prime }}=\left(z_{k^{\prime }-1},{\mathit{r}}_{k^{\prime }-1}\right)/\left({\mathit{p}}_{k^{\prime }-1},{\widehat{\mathit{p}}}_{k^{\prime }}\right)$;

8:   ${\mathit{x}}_{k^{\prime }}={\mathit{x}}_{k^{\prime }-1}+{\alpha }_{k^{\prime }}{\mathit{p}}_{k^{\prime }}$;

9:   ${\mathit{r}}_{k^{\prime }}={\mathit{r}}_{k^{\prime }-1}-{\alpha }_{k^{\prime }}{\widehat{\mathit{p}}}_{k^{\prime }}$;

10:   ${\mathit{z}}_{k^{\prime }}=M^{-1}{\mathit{r}}_{k^{\prime }}$;

11:   ${\beta }_{k^{\prime }}=\left({\mathit{z}}_{k^{\prime }},{\mathit{r}}_{k^{\prime }}\right)/\left({\mathit{z}}_{k^{\prime }-1},{\mathit{r}}_{k^{\prime }-1}\right)$;

12:  end for

13:  Obtain an approximate solution $\tilde{\mathit{y}}\approx {\mathit{x}}_{k^{\prime }}$;

4.1.1. Preconditioning Matrix

Algorithm 2 solves

$$\begin{array}{c}\hfill \left(M^{-1}\tilde{A}{M}^{-\mathrm{H}}\right)\left(M^{\mathrm{H}}\tilde{\mathit{y}}\right)=M^{-1}\tilde{\mathit{b}},\end{array}$$

where $M\in {\mathbb{R}}^{\ell mn\times \ell mn}$ is a preconditioning matrix. M must satisfy the following two conditions: (1) a condition number of $M^{-1}\tilde{A}$ is close to 1; (2) the matrix-vector multiplication of $M^{-1}$ is easily computed.

Therefore, we propose a preconditioning matrix based on the eigendecomposition of T

$$\begin{array}{c}\hfill M:=\overline{D}D-{\tilde{\sigma }}^2{I}_{\ell mn}.\end{array}$$

(5)

Since M is the diagonal matrix, the second condition of the preconditioning matrix is satisfied. Moreover, if T is symmetric, X is the unitary matrix, that is, $X^{\mathrm{H}}X=I_{\ell mn}$. In the case of the symmetric matrix T, we obtain $M=\tilde{A}$. Namely, the proposed matrix satisfies the first conditions when T is symmetric. So, even if T is not exactly symmetric, if T is almost symmetric, then we can expect the preconditioning matrix M to be effective.

4.1.2. Efficient Implementation of Algorithm 2 by the Eigendecomposition

Similarly to obtaining Algorithm 1, to improve an implementation of Algorithm 2, we reconstruct $\ell mn$ dimensional vectors into $\ell \times m\times n$ tensors via ${\mathrm{vec}}^{-1}$ operator as follows: ${\mathcal{X}}_{k^{\prime }}:={\mathrm{vec}}^{-1}({\mathit{x}}_{k^{\prime }})$, ${\mathcal{R}}_{k^{\prime }}:={\mathrm{vec}}^{-1}({\mathit{r}}_{k^{\prime }})$, ${\mathcal{P}}_{k^{\prime }}:={\mathrm{vec}}^{-1}({\mathit{p}}_{k^{\prime }})$, ${\mathcal{Z}}_{k^{\prime }}:={\mathrm{vec}}^{-1}({\mathit{z}}_{k^{\prime }})$, and ${\widehat{\mathcal{P}}}_{k^{\prime }}:={\mathrm{vec}}^{-1}({\widehat{\mathit{p}}}_{k^{\prime }})$. Most computations of vectors are simply transformed into computations of tensors because of the linearity of ${\mathrm{vec}}^{-1}$ operator.

In the rest of this section, we show the computations of ${\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)$ and ${\mathrm{vec}}^{-1}\left(M^{-1}\mathrm{vec}({\mathcal{R}}_{k^{\prime }})\right)$, which are required in the PCG method, using the 1, 2, and 3-mode products for tensors and the definition of T. First, from the definitions of $\tilde{A}$ and X, ${\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)={\mathrm{vec}}^{-1}(\overline{D}{X}^{\mathrm{H}}XD\mathrm{vec}({\mathcal{P}}_{k^{\prime }}))-{\tilde{\sigma }}^2{\mathrm{vec}}^{-1}(X^{\mathrm{H}}X\mathrm{vec}({\mathcal{P}}_{k^{\prime }}))$ holds. Let $\mathcal{D}={\mathrm{vec}}^{-1}(\mathrm{diag}(D))$, where $\mathrm{diag}(D)$ returns an $\ell mn$-dimensional column vector with diagonals of D. Then, ${\mathcal{D}}_{ijk}:={\lambda }_i^{(A)}+{\lambda }_j^{(B)}+{\lambda }_k^{(C)}$, where ${\lambda }_i^{(A)},{\lambda }_j^{(B)},$ and ${\lambda }_k^{(C)}$ denote the eigenvalues of $A,B$, and C. Note that ${({\mathrm{vec}}^{-1}(D\mathrm{vec}({\mathcal{P}}_{k^{\prime }})))}_{ijk}={\mathcal{D}}_{ijk}{({\mathcal{P}}_{k^{\prime }})}_{ijk}$ since we compute ${(D{\mathit{p}}_{k^{\prime }})}_i=D_{ii}{({\mathit{p}}_{k^{\prime }})}_i$ for $i=1,2,\dots ,\ell mn$. Using the relation between the Kronecker product and the mode products via ${\mathrm{vec}}^{-1}$ operator, we compute

$$\begin{array}{cc}\hfill & {\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}(\mathcal{P})\right)\hfill \\ \hfill & =\overline{\mathcal{D}}\ast \left\{(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_1{X}_A^{\mathrm{H}}{X}_A+(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_2{X}_B^{\mathrm{H}}{X}_B+(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_3{X}_C^{\mathrm{H}}{X}_C\right\}\hfill \\ \hfill & -{\tilde{\sigma }}^2\left({\mathcal{P}}_{k^{\prime }}{\times }_1{X}_A^{\mathrm{H}}{X}_A+{\mathcal{P}}_{k^{\prime }}{\times }_2{X}_B^{\mathrm{H}}{X}_B+{\mathcal{P}}_{k^{\prime }}{\times }_3{X}_C^{\mathrm{H}}{X}_C\right),\hfill \end{array}$$

(6)

where “*” denotes elementwise product.

Next, from the definition of the diagonal matrix M in Equation (5), we easily obtain

$$\begin{array}{c}\hfill {\left(M^{-1}\right)}_{ii}=\frac{1}{{(\overline{D})}_{ii}{(D)}_{ii}-{\tilde{\sigma }}^2},i=1,2,\dots ,\ell mn.\end{array}$$

Here, let $\mathcal{M}={\mathrm{vec}}^{-1}(\mathrm{diag}(M^{-1}))$. Then it follows that ${\mathcal{M}}_{ijk}=1/({\overline{\mathcal{D}}}_{ijk}{\mathcal{D}}_{ijk}-{\tilde{\sigma }}^2)$. ${\mathrm{vec}}^{-1}\left(M^{-1}\mathrm{vec}({\mathcal{R}}_{k^{\prime }})\right)$ is computed by

$$\begin{array}{c}\hfill {\mathrm{vec}}^{-1}\left(M^{-1}\mathrm{vec}({\mathcal{R}}_{k^{\prime }})\right)=\mathcal{M}\ast {\mathcal{R}}_{k^{\prime }}.\end{array}$$

(7)

As shown in Algorithm 3, the PCG method can be implemented using the preconditioning matrix M and the aforementioned computations, where the linear system $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$ is transformed into $\tilde{A}\mathrm{vec}(\tilde{\mathcal{Y}})=\mathrm{vec}(\tilde{\mathcal{B}})$, where $\mathrm{vec}(\tilde{\mathcal{B}}):=\tilde{\mathit{b}}=\mathrm{vec}({\mathcal{Q}}_k{\times }_1{X}_A^{\mathrm{H}}+{\mathcal{Q}}_k{\times }_2{X}_B^{\mathrm{H}}+{\mathcal{Q}}_k{\times }_3{X}_C^{\mathrm{H}})$ and $\mathrm{vec}(\tilde{\mathcal{Y}}):=\tilde{\mathit{y}}$. Algorithm 3 requires only small matrices A, B, and C and $\ell \times m\times n$ tensors ${\mathcal{X}}_{k^{\prime }},{\mathcal{R}}_{k^{\prime }},{\mathcal{P}}_{k^{\prime }}$, and ${\mathcal{Z}}_{k^{\prime }}$. Therefore the memory requirement is of $O(n^3)$ in the case of $n=m=\ell$.

4.2. PCG Method by the Schur Decomposition

Firstly, the Schur decomposition of T is $T:=QR{Q}^{\mathrm{H}}$, where R and Q are upper triangular and unitary matrices, respectively. Then,

$$\begin{array}{cc}\hfill \left(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k& \iff \left({(QR{Q}^{\mathrm{H}})}^{\mathrm{H}}(QR{Q}^{\mathrm{H}})-{\tilde{\sigma }}^2{I}_{\ell mn}\right)\mathit{v}={\mathit{q}}_k\hfill \\ \hfill & \iff \left(R^{\mathrm{H}}R-{\tilde{\sigma }}^2{I}_{\ell mn}\right)(Q^{\mathrm{H}}\mathit{v})=Q^{\mathrm{H}}{\mathit{q}}_k.\hfill \end{array}$$

This linear system denotes $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$, where $\tilde{A}:=R^{\mathrm{H}}R-{\tilde{\sigma }}^2{I}_{\ell mn}$, $\tilde{\mathit{y}}:=Q^{\mathrm{H}}\mathit{v}$, and $\tilde{\mathit{b}}:=Q^{\mathrm{H}}{\mathit{q}}_k$. The PCG method for $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$ is shown in Algorithm 2. R and Q are obtained from the complex Schur decomposition of $A,B$, and C as follows: $R=I_n\otimes I_m\otimes R_A+I_n\otimes R_B\otimes I_{\ell }+R_C\otimes I_m\otimes I_{\ell }$ and $Q=Q_C\otimes Q_B\otimes Q_A$ from the definition of T, where $A=Q_A{R}_A{Q}_A^{\mathrm{H}}$, $B=Q_B{R}_B{Q}_B^{\mathrm{H}}$, and $C=Q_C{R}_C{Q}_C^{\mathrm{H}}$ by the Schur decomposition of $A,B$, and C.

Algorithm 3: PCG method over tensor space for the 5-th line of Algorithm 1 [Proposed inner algorithm using the eigendecomposition]

1:  Choose an initial tensor ${\mathcal{X}}_0\in {\mathbb{R}}^{\ell \times m\times n}$ and ${\mathcal{P}}_0=O_{\ell \times m\times n}$, and an initial scalar ${\beta }_0=0$;

2:  ${\mathcal{R}}_0=({\mathcal{Q}}_k{\times }_1{X}_A^{\mathrm{H}}+{\mathcal{Q}}_k{\times }_2{X}_B^{\mathrm{H}}+{\mathcal{Q}}_k{\times }_3{X}_C^{\mathrm{H}})$    $-[\overline{\mathcal{D}}\ast \{(\mathcal{D}\ast {\mathcal{X}}_0){\times }_1{X}_A^{\mathrm{H}}{X}_A+(\mathcal{D}\ast {\mathcal{X}}_0){\times }_2{X}_B^{\mathrm{H}}{X}_B+(\mathcal{D}\ast {\mathcal{X}}_0){\times }_3{X}_C^{\mathrm{H}}{X}_C\}$    $-{\tilde{\sigma }}^2({\mathcal{X}}_0{\times }_1{X}_A^{\mathrm{H}}{X}_A+{\mathcal{X}}_0{\times }_2{X}_B^{\mathrm{H}}{X}_B+{\mathcal{X}}_0{\times }_3{X}_C^{\mathrm{H}}{X}_C)]$;

3:  ${\mathcal{Z}}_0=\mathcal{M}\ast {\mathcal{R}}_0$;

4:  for $k^{\prime }=1,2,\dots ,$ until convergence do

5:   ${\mathcal{P}}_{k^{\prime }}={\mathcal{Z}}_{k^{\prime }-1}+{\beta }_{k^{\prime }-1}{\mathcal{P}}_{k^{\prime }-1}$;

6:   ${\widehat{\mathcal{P}}}_{k^{\prime }}=\overline{\mathcal{D}}\ast \left\{(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_1{X}_A^{\mathrm{H}}{X}_A+(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_2{X}_B^{\mathrm{H}}{X}_B+(\mathcal{D}\ast {\mathcal{P}}_{k^{\prime }}){\times }_3{X}_C^{\mathrm{H}}{X}_C\right\}$    $-{\tilde{\sigma }}^2\left({\mathcal{P}}_{k^{\prime }}{\times }_1{X}_A^{\mathrm{H}}{X}_A+{\mathcal{P}}_{k^{\prime }}{\times }_2{X}_B^{\mathrm{H}}{X}_B+{\mathcal{P}}_{k^{\prime }}{\times }_3{X}_C^{\mathrm{H}}{X}_C\right)$;

7:   ${\alpha }_{k^{\prime }}=\left({\mathcal{Z}}_{k^{\prime }-1},{\mathcal{R}}_{k^{\prime }-1}\right)/\left({\mathcal{P}}_{k^{\prime }-1},{\widehat{\mathcal{P}}}_{k^{\prime }}\right)$;

8:   ${\mathcal{X}}_{k^{\prime }}={\mathcal{X}}_{k^{\prime }-1}+{\alpha }_{k^{\prime }}{\mathcal{P}}_{k^{\prime }}$;

9:   ${\mathcal{R}}_{k^{\prime }}={\mathcal{R}}_{k^{\prime }-1}-{\alpha }_{k^{\prime }}{\widehat{\mathcal{P}}}_{k^{\prime }}$;

10:   ${\mathcal{Z}}_{k^{\prime }}=\mathcal{M}\ast {\mathcal{R}}_{k^{\prime }-1}$;

11:   ${\beta }_{k^{\prime }}=\left({\mathcal{Z}}_{k^{\prime }},{\mathcal{R}}_{k^{\prime }}\right)/\left({\mathcal{Z}}_{k^{\prime }-1},{\mathcal{R}}_{k^{\prime }-1}\right)$;

12:  end for

13:  Obtain an approximate solution $\tilde{\mathcal{Y}}\approx {\mathcal{X}}_{k^{\prime }}$;

14:  $\mathcal{V}=\tilde{\mathcal{Y}}{\times }_1{X}_A+\tilde{\mathcal{Y}}{\times }_2{X}_B+\tilde{\mathcal{Y}}{\times }_3{X}_C$;

4.2.1. Preconditioning Matrix

A preconditioning matrix for $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$ satisfies the conditions in Section 4.1.1. Therefore, we propose the preconditioning matrix based on the Schur decomposition

$$\begin{array}{c}\hfill M:={\overline{D}}_R{D}_R-{\tilde{\sigma }}^2{I}_{\ell mn},\end{array}$$

where $D_R$ is a diagonal matrix with diagonals of R. Since M is also the diagonal matrix, the above second conditions are satisfied. Moreover, if T is symmetric, R is a diagonal matrix, that is, $R=D_R$. Therefore $M=\tilde{A}$ in the case of the symmetric matrix T. From this, we expect that the preconditioning matrix M is effective if T is not symmetric but almost symmetric.

4.2.2. Efficient Implementation of Algorithm 2 by the Schur Decomposition

We show the computations of ${\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)$ and ${\mathrm{vec}}^{-1}\left(M^{-1}\mathrm{vec}({\mathcal{R}}_{k^{\prime }})\right)$ for the PCG method over tensor space using the 1, 2, and 3-mode products for tensors and the definition of T. First, from the definitions of $\tilde{A}$ and R, we have ${\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)={\mathrm{vec}}^{-1}\left(R^{\mathrm{H}}\left(R\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)-{\tilde{\sigma }}^2\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)$. Therefore,

$$\begin{array}{c}\hfill {\mathrm{vec}}^{-1}\left(\tilde{A}\mathrm{vec}({\mathcal{P}}_{k^{\prime }})\right)={\mathcal{P}}_{k^{\prime }}{\times }_1{R}_A^{\mathrm{H}}{R}_A+{\mathcal{P}}_{k^{\prime }}{\times }_2{R}_B^{\mathrm{H}}{R}_B+{\mathcal{P}}_{k^{\prime }}{\times }_3{R}_C^{\mathrm{H}}{R}_C-{\tilde{\sigma }}^2{\mathcal{P}}_{k^{\prime }}.\end{array}$$

Next, from $M={\overline{D}}_R{D}_R-{\tilde{\sigma }}^2{I}_{\ell mn}$, we easily obtain

$$\begin{array}{c}\hfill {\left(M^{-1}\right)}_{ii}=\frac{1}{{({\overline{D}}_R)}_{ii}{(D_R)}_{ii}-{\tilde{\sigma }}^2},i=1,2,\dots ,\ell mn.\end{array}$$

Similarly to Section 4.1.2, let $\mathcal{D}={\mathrm{vec}}^{-1}(\mathrm{diag}(D_R))$ and $\mathcal{M}={\mathrm{vec}}^{-1}(\mathrm{diag}(M^{-1}))$. Then, we have ${\mathcal{M}}_{ijk}=1/({\overline{\mathcal{D}}}_{ijk}{\mathcal{D}}_{ijk}-{\tilde{\sigma }}^2)$, where ${\mathcal{D}}_{ijk}={(R_A)}_{ii}+{(R_B)}_{jj}+{(R_C)}_{kk}$. ${\mathrm{vec}}^{-1}(M^{-1}\mathrm{vec}({\mathcal{R}}_{k^{\prime }}))$ is computed by (7).

As shown in Algorithm 4, the PCG method can be implemented using the preconditioning matrix M and the aforementioned computations, where the linear system $\tilde{A}\tilde{\mathit{y}}=\tilde{\mathit{b}}$ is transformed into $\tilde{A}\mathrm{vec}(\tilde{\mathcal{Y}})=\mathrm{vec}(\tilde{\mathcal{B}})$, where $\mathrm{vec}(\tilde{\mathcal{B}}):=\tilde{\mathit{b}}=\mathrm{vec}({\mathcal{Q}}_k{\times }_1{Q}_A+{\mathcal{Q}}_k{\times }_2{Q}_B+{\mathcal{Q}}_k{\times }_3{Q}_C)$ and $\mathrm{vec}(\tilde{\mathcal{Y}}):=\tilde{\mathit{y}}$. Algorithm 4 just requires small matrices A, B, and C and $\ell \times m\times n$ tensors ${\mathcal{X}}_{k^{\prime }},{\mathcal{R}}_{k^{\prime }},{\mathcal{P}}_{k^{\prime }}$, and ${\mathcal{Z}}_{k^{\prime }}$, namely, do not require large matrix T. Therefore the memory requirement is of $O(n^3)$ in the case of $n=m=\ell$.

Algorithm 4: PCG method over tensor space for the 5-th line of Algorithm 1 [Proposed inner algorithm using the Schur decomposition]

1:  Choose an initial tensor ${\mathcal{X}}_0\in {\mathbb{R}}^{\ell \times m\times n}$ and ${\mathcal{P}}_0=O_{\ell \times m\times n}$, and an initial scalar ${\beta }_0=0$;

2:  ${\mathcal{R}}_0=({\mathcal{Q}}_k{\times }_1{Q}_A+{\mathcal{Q}}_k{\times }_2{Q}_B+{\mathcal{Q}}_k{\times }_3{Q}_C)$    $-\left({\mathcal{X}}_0{\times }_1{R}_A^{\mathrm{H}}{R}_A+{\mathcal{X}}_0{\times }_2{R}_B^{\mathrm{H}}{R}_B+{\mathcal{X}}_0{\times }_3{R}_C^{\mathrm{H}}{R}_C-{\tilde{\sigma }}^2{\mathcal{X}}_0\right)$;

3:  ${\mathcal{Z}}_0=\mathcal{M}\ast {\mathcal{R}}_0$;

4:  for $k^{\prime }=1,2,\dots ,$ until convergence do

5:   ${\mathcal{P}}_{k^{\prime }}={\mathcal{Z}}_{k^{\prime }-1}+{\beta }_{k^{\prime }-1}{\mathcal{P}}_{k^{\prime }-1}$;

6:   ${\widehat{\mathcal{P}}}_{k^{\prime }}={\mathcal{P}}_{k^{\prime }}{\times }_1{R}_A^{\mathrm{H}}{R}_A+{\mathcal{P}}_{k^{\prime }}{\times }_2{R}_B^{\mathrm{H}}{R}_B+{\mathcal{P}}_{k^{\prime }}{\times }_3{R}_C^{\mathrm{H}}{R}_C-{\tilde{\sigma }}^2{\mathcal{P}}_{k^{\prime }}$;

7:   ${\alpha }_{k^{\prime }}=\left({\mathcal{Z}}_{k^{\prime }-1},{\mathcal{R}}_{k^{\prime }-1}\right)/\left({\mathcal{P}}_{k^{\prime }-1},{\widehat{\mathcal{P}}}_{k^{\prime }}\right)$;

8:   ${\mathcal{X}}_{k^{\prime }}={\mathcal{X}}_{k^{\prime }-1}+{\alpha }_{k^{\prime }}{\mathcal{P}}_{k^{\prime }}$;

9:   ${\mathcal{R}}_{k^{\prime }}={\mathcal{R}}_{k^{\prime }-1}-{\alpha }_{k^{\prime }}{\widehat{\mathcal{P}}}_{k^{\prime }}$;

10:   ${\mathcal{Z}}_{k^{\prime }}=\mathcal{M}\ast {\mathcal{R}}_{k^{\prime }}$;

11:   ${\beta }_{k^{\prime }}=\left({\mathcal{Z}}_{k^{\prime }},{\mathcal{R}}_{k^{\prime }}\right)/\left({\mathcal{Z}}_{k^{\prime }-1},{\mathcal{R}}_{k^{\prime }-1}\right)$;

12:  end for

13:  Obtain an approximate solution $\tilde{\mathcal{Y}}\approx {\mathcal{X}}_{k^{\prime }}$;

14:  $\mathcal{V}=\tilde{\mathcal{Y}}{\times }_1{Q}_A+\tilde{\mathcal{Y}}{\times }_2{Q}_B+\tilde{\mathcal{Y}}{\times }_3{Q}_C$;

5. Numerical Experiments

This section provides results of numerical experiments using Algorithm 1 with Algorithm 3 and Algorithm 1 with Algorithm 4. There are the two purposes of this experiments: (1) to confirm convergence to the singular value of T nearest to the shift by Algorithm 1, and (2) to confirm the effectiveness of the proposed precondition matrix in Algorithms 3 and 4. For comparison, the results using Algorithms 3 and 4 in the case of $M=I$ are also given as the results by the CG method. All the initial guesses of Algorithms 1, 3, and 4 are tensors with random numbers. The stopping criteria used in Algorithm 1 was ${\beta }_k\left|\right|{\mathit{e}}_k^{\mathrm{T}}{\mathit{s}}_{\mathrm{MAX}}^k\left|\right|<{10}^{-8}$, where ${\mathit{s}}_{\mathrm{MAX}}^k$ is the eigenvector corresponding to the maximum eigenvalue of ${\tilde{T}}_k$ and ${\mathit{e}}_k$ denotes the k-th canonical basis for k dimensional vector space. Algorithms 3 and 4 were stopped when either the relative residual $\left|\right|{\mathcal{R}}_{k^{\prime }}\left|\right|/\left|\right|\tilde{\mathcal{B}}\left|\right|<{10}^{-12}$ or the maximum number of iterations $k^{\prime }>20,000$ were satisfied.

All computations were carried out using MATLAB R2021a version on a workstation with Xeon processor 3.7 GHz and 128 GB of RAM.

In the following subsection, we show the results computing the 5-th maximum, median, and 5-th minimum singular values $\sigma$ of the test matrices T. For all the cases, for the first purpose, we set the shift value in Algorithm 1 as

$$\begin{array}{c}\hfill \tilde{\sigma }=\sigma -{10}^{-2},\end{array}$$

(8)

where $\tilde{\sigma }$’s and $\sigma$’s are the perturbed singular values of T and the aforementioned singular values computed by the svd function in MATLAB, respectively.

Test matrices T in Equation (1) are obtained from a seven-point central difference discretization of the PDE (2) in over an $(n+1)\times (n+1)\times (n+1)$ grid. The test matrices T in Equation (1), whose size is $n^3\times n^3$, are generated from

$$\begin{array}{c}\hfill A=B=C,A:=\frac{1}{h^2}a{M}_1+\frac{1}{2h}b{M}_2+\frac{1}{3}c{I}_n,\end{array}$$

(9)

where $h=1/(n+1)$, $M_1$ and $M_2$ are symmetric and skew-symmetric matrices given below.

$$\begin{array}{c}\hfill M_1=\left(\begin{array}{ccccc}-2& 1& & & \\ 1& -2& 1& & \\ & \ddots & \ddots & \ddots & \\ & & 1& -2& 1\\ & & & 1& -2\end{array}\right)\in {\mathbb{R}}^{n\times n},M_2=\left(\begin{array}{ccccc}0& 1& & & \\ -1& 0& 1& & \\ & \ddots & \ddots & \ddots & \\ & & -1& 0& 1\\ & & & -1& 0\end{array}\right)\in {\mathbb{R}}^{n\times n}.\end{array}$$

Numerical Results

In all tables, the number of iterations of the shift-and-invert Lanczos method (“the Lanczos method" hereafter) and the average of the number of iterations of the CG or PCG method based on the eigendecomposition or the Schur decomposition are summarized. “Not converged” denotes Algorithm 3 or 4 did not converge.

We show the first results in the case of almost symmetric matrix with $a=c=1$ and $b=0.01$ in Equation (9) for the shift (8). From

Table 1. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th max. singular value of almost symmetric matrix T with $a=c=1$ and $b=0.01$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	4	43.0	4	16.0	4	35.8	4	15.0
	10	4	90.0	4	17.0	4	86.5	4	15.0
	15	3	134.7	3	17.0	3	128.0	3	17.0
	20	4	180.0	4	17.0	4	169.3	4	17.0
	25	3	225.7	3	17.0	3	211.0	3	17.0
	30	3	273.0	3	17.0	3	252.3	3	17.0

,

Table 2. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the median of singular value of almost symmetric matrix T with $a=c=1$ and $b=0.01$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	15	139.3	15	13.0	15	109.1	15	15.0
	10	7	1081.0	7	13.0	7	943.7	7	15.0
	15	41	5201.6	39	15.0	39	4858.1	41	17.0
	20	13	8339.5	4	16.0	4	6847.3	4	15.0
	25	(Not converged.)		48	17.0	(Not converged.)		48	17.0
	30	(Not converged.)		8	19.0	(Not converged.)		8	15.0

and

Table 3. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th min. singular value of almost symmetric matrix T with $a=c=1$ and $b=0.01$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	3	124.7	3	13.0	3	97.9	3	14.6
	10	3	748.1	3	13.0	3	654.9	3	14.1
	15	3	4872.5	3	14.9	3	4531.1	3	16.7
	20	3	775.5	3	13.8	3	2358.8	3	15.0
	25	3	1494.0	3	16.8	3	1472.0	3	16.8
	30	3	2158.0	3	16.8	3	2088.0	3	15.0

, the numbers of iterations of Lanczos methods using any inner algorithms were almost the same. Focusing on the effectiveness of the proposed preconditioning matrix M, the numbers of iterations of both PCG methods were less than 19 regardless of the size of T. On the other hand, the numbers of iterations of both CG methods linearly increased depending on the size of T. From these facts, the preconditioning matrix M is effective in the case of almost symmetric matrix T. Moreover, the number of iterations of the shift and invert Lanczos method for the median singular value is larger than the number for other singular values since the distance between the maximum and second maximum singular values of ${(T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn})}^{-1}$ for the median singular value of T is closer than the cases of other singular values.

Here, the running time of Table 1 is summarized in

Table 4. Running time (seconds) of the Lanczos method using the CG/PCG method in the case of the 5-th max. singular value of almost symmetric matrix T with $a=c=1$ and $b=0.01$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)		Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos with CG	Lanczos with PCG	Lanczos with CG	Lanczos with PCG
n	5	0.114	0.071	0.095	0.061
	10	0.578	0.095	0.566	0.086
	15	1.402	0.137	1.448	0.192
	20	6.639	0.437	6.446	0.335
	25	13.632	0.558	12.686	0.432
	30	35.121	1.145	33.203	0.836

. All the running time by the PCG method were less than the time by the CG method. Moreover, the running time by the PCG methods of Algorithms 3 and 4 were similar since the computational complexities of these algorithms are similar. Thus, the running time is strongly correlated with the number of iterations of Algorithms 3 and 4.

In addition, convergence histories of $n=15$ in Table 1 and Table 2 are shown in Figure 1 and Figure 2. Figure 2 displays that the relative residual norms unsteadily decreased when the number of iterations of the shift and invert Lanczos method is not small.

Next, we show the second results in the case of slightly symmetric matrix with $a=c=1$ and $b=0.1$ in Equation (9) for the shift (8). From

Table 5. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th max. singular value of slightly symmetric matrix T with $a=c=1$ and $b=0.1$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	4	50.0	(Not converged.)		4	37.8	(Not converged.)
	10	4	100.0	(Not converged.)		4	87.3	(Not converged.)
	15	3	152.0	(Not converged.)		3	131.0	(Not converged.)
	20	4	205.5	(Not converged.)		4	172.0	(Not converged.)
	25	3	262.0	(Not converged.)		3	230.0	(Not converged.)
	30	3	306.7	(Not converged.)		3	259.0	(Not converged.)

, both PCG methods did not converge for computing the 5-th maximum singular values of slightly symmetric matrix T. It seems that the linear system for $T^{\mathrm{T}}T-{\tilde{\sigma }}^2{I}_{\ell mn}$ is ill-conditioned since ${10}^{-2}$ in the shift (8) is much less than the 5-th maximum singular values of the matrix. In Appendix A, we show the results using relative shift without the effect of the magnitude of the singular values.

Table 6. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the median of singular value of slightly symmetric matrix T with $a=c=1$ and $b=0.1$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	13	231.8	(Not converged.)		13	193.1	13	73.0
	10	6	1582.7	6	55.0	6	1272.7	6	29.0
	15	30	12,174.6	(Not converged.)		31	11061.9	31	97.0
	20	4	13,777.8	(Not converged.)		4	8799.3	(Not converged.)
	25	(Not converged.)		(Not converged.)		(Not converged.)		83	116.0
	30	(Not converged.)		(Not converged.)		(Not converged.)		27	45.0

shows Algorithms 1 and 4, that is, the algorithms based on Schur decomposition, was more robust than Algorithms 1 and 3, that is, the algorithms based on the eigendecomposition. From

Table 7. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th min. singular value of slightly symmetric matrix T with $a=c=1$ and $b=0.1$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	3	195.5	3	59.0	3	163.9	3	68.4
	10	3	949.8	3	58.0	3	771.2	3	44.0
	15	3	616.0	3	63.0	3	597.3	3	94.5
	20	3	859.8	3	63.0	3	2999.8	3	57.0
	25	3	1695.7	3	63.0	3	1559.3	3	114.4
	30	3	2388.0	3	63.0	3	2262.0	3	46.1

, both PCG methods converged regardless of n, and the numbers of iterations of both PCG methods were less than the number of iterations of both CG method. Namely, it seems that the preconditioning matrix M can be effective in the case of a slightly symmetric matrix T when computing the 5-th minimum and median singular values of T.

Finally, we show the third results in the case of marginally symmetric matrix with $a=c=1$ and $b=0.2$ in Equation (9) for the shift (8). Both PCG methods did not converge for computing the 5-th maximum singular values of T as shown in

Table 8. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th max. singular value of marginally symmetric matrix T with $a=c=1$ and $b=0.2$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	4	52.8	(Not converged.)		4	39.8	(Not converged.)
	10	4	107.0	(Not converged.)		4	93.0	(Not converged.)
	15	3	162.0	(Not converged.)		3	135.7	(Not converged.)
	20	4	217.8	(Not converged.)		4	206.3	(Not converged.)
	25	3	271.3	(Not converged.)		3	248.0	(Not converged.)
	30	3	334.0	(Not converged.)		3	260.0	(Not converged.)

, similarly to Table 5. Moreover, computing the median singular values of T sometimes did not converge from

Table 9. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the median of singular value of marginally symmetric matrix T with $a=c=1$ and $b=0.2$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	11	481.4	(Not converged.)		10	355.6	(Not converged.)
	10	6	2123.8	(Not converged.)		10	1550.8	10	4262.4
	15	(Not converged.)		(Not converged.)		(Not converged.)		(Not converged.)
	20	15	13,268.7	89	6358.9	7	10019.9	108	160.0
	25	(Not converged.)		(Not converged.)		(Not converged.)		(Not converged.)
	30	(Not converged.)		90	1150.0	(Not converged.)		28	11,764.0

. In

Table 10. Number of iterations of the Lanczos method and the average of numbers of iterations of the CG/PCG method in the case of the 5-th min. singular value of marginally symmetric matrix T with $a=c=1$ and $b=0.2$ in (9) for the shift (8).

Method		Algorithms 1 and 3 (by Eigendecompn.)				Algorithms 1 and 4 (by Schur Decompn.)
		Lanczos	CG	Lanczos	PCG	Lanczos	CG	Lanczos	PCG
n	5	5	313.9	5	295.8	5	213.9	5	1077.8
	10	3	1247.8	3	187.0	3	1150.6	3	3048.8
	15	3	650.0	3	201.0	3	606.7	3	177.0
	20	3	929.3	3	6151.3	3	847.8	3	162.8
	25	3	1800.3	3	205.0	3	1683.7	3	249.0
	30	3	2585.7	3	1118.6	3	2371.3	3	181.0

, all methods converged for the 5-th minimum singular value of T. The numbers of iterations by the PCG method with the proposed preconditioning matrix were less than the number of iterations by the CG method in most cases. It seems that the preconditioning matrix M can be effective in the case of the marginally symmetric matrix T when computing the 5-th minimum singular values of T.

6. Conclusions

We considered computing an arbitrary singular value of a tensor sum. The shift-and-invert Lanczos method and the PCG method reconstructed over tensor space. We proposed the preconditioning matrices which are the following two diagonal matrices: (1) whose diagonals of the eigenvalues by the eigendecomposition, and (2) whose diagonals of the upper diagonal matrix by the Schur decomposition. This preconditioning matrix can be effective if the tensor sum is almost symmetric.

From numerical results, we confirmed that the proposed method reduces memory requirements without any conditions. The numbers of iterations of the PCG method by the proposed preconditioning matrix were reduced in most cases of the almost and slightly symmetric matrix. Moreover, the numbers of iterations of the PCG method by the proposed preconditioning matrix were also reduced in certain cases of the marginally symmetric matrix.

For future work, we will consider a robust preconditioning matrix for slightly or marginally symmetric tensor sum, a suitable preconditioning matrix for non-symmetric tensor sum, parallel implementations of the proposed algorithms, and finding real-life applications.