A Missing Data Reconstruction Method Using an Accelerated Least-Squares Approximation with Randomized SVD

Abstract

An accelerated least-squares approach is introduced in this work by incorporating a greedy point selection method with randomized singular value decomposition (rSVD) to reduce the computational complexity of missing data reconstruction. The rSVD is used to speed up the computation of a low-dimensional basis that is required for the least-squares projection by employing randomness to generate a small matrix instead of a large matrix from high-dimensional data. A greedy point selection algorithm, based on the discrete empirical interpolation method, is then used to speed up the reconstruction process in the least-squares approximation. The accuracy and computational time reduction of the proposed method are demonstrated through three numerical experiments. The first two experiments consider standard testing images with missing pixels uniformly distributed on them, and the last numerical experiment considers a sequence of many incomplete two-dimensional miscible flow images. The proposed method is shown to accelerate the reconstruction process while maintaining roughly the same order of accuracy when compared to the standard least-squares approach.

1. Introduction

Missing data is a problematic issue in many applications. This problem may arise from incomplete data collection, unavailability of data, or corrupted data during the transmission process. Recovering missing data accurately is essential, especially for the purposes of data analysis and prediction. Various approaches have been proposed for reconstructing missing data. In [1], the deep convolution neural network was proposed to reconstruct missing data for a remote sensing image. A deep learning approach was used for irregularly and regularly missing data reconstruction in [2]. The concept of fuzzy similarity was used for recovering data in multidimensional time series in [3]. The long short-term memory(LSTM) approach was introduced in [4] for reconstructing time series Landsat images. A methodology based on principles of matrix theory and pseudo-inverses was proposed in [5] for data reconstruction in erasure codes. In [6], a method of tensor completion based on nuclear norm minimization was introduced for 5D seismic data reconstruction. Other approaches for approximating missing data include the regression method [7], K-nearest neighbors (KNN) [8,9], principal component analysis (PCA) [10], and least-squares approximation [11,12,13]. These approaches are mostly specific to certain problems. The goal of this paper is to improve a general framework based on a least-squares method for estimating missing data with an optimal low-rank basis in tje Euclidean norm, which is also called proper orthogonal decomposition (POD) basis.

POD is also known as, for example, Karhunen-Loève decomposition (KLD), principal component analysis (PCA), or singular value decomposition(SVD). POD has been successfully used in many applications, e.g., [14,15,16,17,18], since it can provide an approximation from the basis that extracts the dominant characteristic of the existing data. POD has been used in many applications, such as aerodynamic flow fields [19], chemical engineering [20,21], mechanical engineering [22,23], image processing [24], and optimization of water flooding reservoir [25]. In general, the POD basis can be computed by using singular value decomposition (SVD). However, for high-dimensional data, obtaining SVD by a traditional approach may be computationally costly and memory intensive. Randomized SVD (rSVD) [26] is considered in this work to reduce this computational complexity.

The rSVD algorithm, introduced by Halko et al. [26], employs randomness to obtain a smaller matrix from a high-dimensional data matrix. The smaller matrix is then used to compute the low-dimensional basis. The rSVD algorithm is shown to be computationally efficient for approximating matrices with low-rank structures. The algorithm is divided into two stages. In the first stage, random sampling is used to obtain a reduced matrix whose range approximates the range of the original matrix. A small matrix can be obtained from the orthogonal projection onto the basis of the reduced matrix from the first stage. In the second stage, the smaller matrix is then inexpensively factorized. Randomized methods have been used in many applications, such as low-rank SVD of a large matrix [27], image compression [28,29], and image reconstruction [30]. The implementation and extension of the rSVD have been investigated in many works, e.g., [31,32,33,34]. This work applies rSVD to reconstruct missing data components with the notion of least-squares approximation.

The least-squares method has been used in missing data reconstruction in previous work, e.g., [11,12,13], which is mainly based on employing all remaining available data. In the case of high-dimensional data, the reconstruction process might require high-complexity computation. To accelerate this approximation process further, this work also employs certain important and relevant available data components, which are selected through a greedy iterative procedure called the discrete empirical interpolation method (DEIM). DEIM [35] was originally introduced to estimate nonlinear terms in dynamical systems by selecting the interpolation indices using a greedy algorithm. The index selection process in this algorithm is based on trying to capture most of the variation of the sample set heuristically. DEIM is often used with POD to provide interpolatory projection approximation. Theoretical works on this approximation and its extension can be found in [35,36,37]. DEIM has been used in many applications, such as two-dimensional shallow-water equations [38], four-dimensional variational data assimilation [39], three-dimensional nonlinear aeroelasticity model [40], and other problems [41,42].

This work is an extension of the result given in [43], which only considered one simple preliminary numerical test. This work provides substantially more materials with an additional improved method that incorporates a faster way to compute projection basis based on a randomized algorithm, as well as performs more numerical experiments with detailed analysis and comparisons to the standard approach. The main contributions of this work can be summarized as follows.

• We propose an algorithm that accelerates the missing data reconstruction procedure based on the least-squares method, which can reduce computational time in two aspects. The first one is based on applying the rSVD to obtain an optimal low-dimensional basis used in the least-squares projection. The second aspect arises from applying a greedy point selection procedure that can extract a small subset of important components to reduce the size of the reconstruction problem and, therefore, speed up the computational time.
• We provide numerical experiments that demonstrate the efficiency (in terms of accuracy and speedup) of the proposed method for some standard testing images with different amounts of missing data.
• We perform numerical tests on a sequence of two-dimensional miscible flow images. The proposed method was shown to give 10 to 100 times reduction in reconstruction time with the same order of accuracy. This illustrates the extensibility of the proposed method on other large-scale problems, as well as the applications of video processing.

This paper is organized as follow. Section 2 describes the approach for reconstructing missing components using the least-squares method. The connection between SVD and the projection basis, which is called POD basis, used in the approximation is discussed in Section 3. The rSVD algorithm is explained in Section 4. Section 5 presents a greedy point selection process using DEIM as well as shows how it is used to reduce the computational complexity of missing data reconstruction. Section 6 discusses more details on the complexity reduction of the proposed method in terms of floating-point operations when compared to the standard least-squares approach. The numerical tests demonstrating the effectiveness of the proposed method are performed on three experiments in Section 7. Finally, the conclusion and some possible extensions are discussed in Section 8.

2. Standard Projection-Based Approach for Data Reconstruction

This section describes how SVD is used to reconstruct missing components of incomplete data. We will approximate an incomplete data vector by projecting on a subspace spanned by a basis that represents other related complete data vectors.

Let $\{{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}\}$ be a complete data set. Define $\mathcal{Y}:=\{{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}\}\subset {\mathbb{R}}^n$ and

$\mathbf{Y}=[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}]\in {\mathbb{R}}^{n\times n_s}$. Suppose $\widehat{\mathbf{y}}$ is an incomplete sample that has $n_c$ known components and $n_g=n-n_c$ unknown components. Let $\mathcal{C}:=\{c_1,c_2,\dots ,c_{n_c}\}\subset \{1,2,\dots ,n\}$ be the indices of the known components in $\widehat{\mathbf{y}}$ and define $\mathbf{C}=[{\mathbf{e}}_{c_1},\dots ,{\mathbf{e}}_{c_{n_c}}]\in {\mathbb{R}}^{n\times n_c}$, where ${\mathbf{e}}_{c_i}={[0,\dots ,0,1,0,\dots ,0]}^T\in {\mathbb{R}}^n$ is the $c_i$-th column of the identity matrix ${\mathbf{I}}_n\in {\mathbb{R}}^{n\times n},$ for $i=1,\dots ,m$. Note that pre-multiplying ${\mathbf{C}}^T$ is equivalent to extracting the $n_c$ rows corresponding to the indices $c_1,\dots ,c_{n_c}$. Similarly, let $\mathcal{G}=\{g_1,g_2,\dots ,g_{n_g}\}\subset \{1,2,\dots ,n\}$ be the indices of the unknown components in $\widehat{\mathbf{y}}$ and define $\mathbf{G}=[{\mathbf{e}}_{g_1},\dots ,{\mathbf{e}}_{g_{n_g}}]\in {\mathbb{R}}^{n\times n_g}$. Let ${\widehat{\mathbf{y}}}_c:={\mathbf{C}}^T\widehat{\mathbf{y}}\in {\mathbb{R}}^{n_c}$ and ${\widehat{\mathbf{y}}}_g:={\mathbf{G}}^T\widehat{\mathbf{y}}\in {\mathbb{R}}^{n_g}$. Then, the known components and the unknown components are contained in the vectors ${\widehat{\mathbf{y}}}_c$ and ${\widehat{\mathbf{y}}}_g$, respectively. The missing components contained in ${\widehat{\mathbf{y}}}_g$ will be approximated by first projecting $\widehat{\mathbf{y}}$ onto the column span of a basis matrix $\mathbf{V}$ with rank k, where $k\le r,r:=\mathrm{rank}\left(\mathbf{Y}\right)$. i.e.,

$$\widehat{\mathbf{y}}\approx \mathbf{V}\mathbf{a},\mathrm{or}{\widehat{\mathbf{y}}}_c\approx {\mathbf{V}}_c\mathbf{a}\mathrm{and}{\widehat{\mathbf{y}}}_g\approx {\mathbf{V}}_g\mathbf{a},$$

for some coefficient vector $\mathbf{a}\in {\mathbb{R}}^k$, and where ${\mathbf{V}}_c:={\mathbf{C}}^T\mathbf{V}\in {\mathbb{R}}^{n_c\times k}$, ${\mathbf{V}}_g:={\mathbf{G}}^T\mathbf{V}\in {\mathbb{R}}^{n_g\times k}$.

The known components contained in ${\widehat{\mathbf{y}}}_c$ are then used to determine the coefficient vector $\mathbf{a}$ through the approximation ${\widehat{\mathbf{y}}}_c\approx {\mathbf{V}}_c\mathbf{a}$ from the following least-squares problem:

$$\begin{array}{c}{\displaystyle \underset{\mathbf{a}\in {\mathbb{R}}^k}{min}}{\parallel {\widehat{\mathbf{y}}}_c-{\mathbf{V}}_c\mathbf{a}\parallel }_2^2.\end{array}$$

(1)

The solution obtained from the corresponding normal equation of the above problem is given by $\mathbf{a}={\mathbf{V}}_c^{+}{\widehat{\mathbf{y}}}_c$, where ${\mathbf{V}}_c^{+}={\left({\mathbf{V}}_c^T{\mathbf{V}}_c\right)}^{-1}{\mathbf{V}}_c^T$ is the Moore–Penrose pseudo-inverse. That is,

$$\begin{array}{c}{\widehat{\mathbf{y}}}_g\approx {\mathbf{V}}_g\mathbf{a}={\mathbf{V}}_g{\mathbf{V}}_c^{+}{\widehat{\mathbf{y}}}_c.\end{array}$$

(2)

The steps described above, which is called the POD-LS approach or standard LS approach, are summarized in Algorithm 1.

Note that the dominant computational work in Algorithm 1 generally comes from Step 2 for constructing a set of basis that can accurately approximate the incomplete data and Step 3 for solving the least-squares problem. In this work, Step 2 employs the basis obtained from the singular value decomposition (SVD) because it is optimal in the least-squares sense, as discussed next in Section 3. To reduce the computational cost for Step 2, we will apply a randomized SVD (rSVD), as discussed in Section 4. To reduce the computational cost in Step 3, we will consider a greedy procedure that can decrease the size of the least-squares problem efficiently in Section 5.1.

Algorithm 1 Standard POD-LS approach for approximating missing data

INPUT: - Complete snapshot set ${\left\{{\mathbf{y}}_j\right\}}_{j=1}^{n_s}\subset {\mathbb{R}}^n$ and, dimension $k\le rank\left({\left\{{\mathbf{y}}_j\right\}}_{j=1}^{n_s}\right)$ - Incomplete data $\widehat{\mathbf{y}}\in {\mathbb{R}}^n$ with known entries in ${\widehat{\mathbf{y}}}_c\in {\mathbb{R}}^{n_c}$ and unknown entries in ${\widehat{\mathbf{y}}}_g\in {\mathbb{R}}^{n_g}$, where $n_c+n_g=n$. OUTPUT: - Approximation of ${\widehat{\mathbf{y}}}_g$  1: Create snapshot matrix $:\mathbf{Y}=[{\mathbf{y}}_1,\dots ,{\mathbf{y}}_{n_s}]\in {\mathbb{R}}^{n\times n_s}$ and let $r=rank\left(\mathbf{Y}\right)$.  2: Construct basis $\mathbf{V}$ of rank $k\le r$ for $\mathbf{Y}$.  3: Find coefficient vector $\mathbf{a}$ from ${\widehat{\mathbf{y}}}_c$ using least-squares problem in (1):  ${min}_{\mathbf{a}\in {\mathbb{R}}^k}{\parallel {\widehat{\mathbf{y}}}_c-{\mathbf{V}}_c\mathbf{a}\parallel }_2^2$.  4: Compute the approximation ${\widehat{\mathbf{y}}}_g\approx {\mathbf{V}}_g\mathbf{a}$.

3. Optimal Basis for Data Set

Consider a set of snapshots $\{{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}\}$ where ${\mathbf{y}}_j\in {\mathbb{R}}^n$, $j=1,2,\dots ,n_s$. Suppose we want to approximate a snapshot ${\mathbf{y}}_j$ by using a set of orthonormal vectors $\{{\varphi }_1,{\varphi }_2,\dots ,{\varphi }_k\}\subset {\mathbb{R}}^n$, where $k<n$. Then, the approximation is in the form ${\mathbf{y}}_j\approx {\sum }_{i=1}^k{w}_i{\varphi }_i,$ for some constant $w_i$, $i=1,2,\dots ,k$. Alternatively, we can write this approximation in matrix form as ${\mathbf{y}}_j\approx {\mathbf{\Phi }}_k\mathbf{w},$ where ${\mathbf{\Phi }}_k=[{\varphi }_1,{\varphi }_2,\dots ,{\varphi }_k]\in {\mathbb{R}}^{n\times k}$ is a matrix of basis vectors and $\mathbf{w}={[w_1,w_2,\dots ,w_k]}^T\in {\mathbb{R}}^k$ is the vector of unknown coefficients. To find $\mathbf{w}$, we use the fact that $\mathbf{\Phi }$ has orthonormal columns, i.e., ${\mathbf{\Phi }}^T\mathbf{\Phi }=\mathbf{I}$, and the minimum error occurs when the residual is orthogonal to the column span of ${\mathbf{\Phi }}_k$, i.e., ${\mathbf{\Phi }}_k^T({\mathbf{y}}_j-{\mathbf{\Phi }}_k\mathbf{w})=0,$ which implies that $\mathbf{w}={\mathbf{\Phi }}^T{\mathbf{y}}_j$ and the approximation becomes

$$\begin{array}{ccc}{\mathbf{y}}_j& \approx & {\mathbf{\Phi }}_k{\mathbf{\Phi }}_k^T{\mathbf{y}}_j.\hfill \end{array}$$

(3)

The goal is to find the orthonormal basis that minimizes this approximation error in 2-norm for a given basis rank $k\le \mathrm{rank}\left(\{{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}\}\right)$

$$\underset{{\mathbf{\Phi }}_k\in {\mathbb{R}}^{n\times k}}{min}\mathbf{\sum }_{j=1}^{n_s}{\parallel {\mathbf{y}}_j-{\mathbf{\Phi }}_k{\mathbf{\Phi }}_k^T{\mathbf{y}}_j\parallel }_2^2\mathrm{such}\mathrm{that}{\mathbf{\Phi }}_k^T{\mathbf{\Phi }}_k={\mathbf{I}}_k,$$

(4)

where ${\mathbf{I}}_k\in {\mathbb{R}}^{k\times k}$ is the identity matrix. This optimal basis is often called proper orthogonal decomposition (POD), which is also known by other names, for example, Karhunen-Loève decomposition (KLD), principal component analysis (PCA), or singular value decomposition (SVD).

It can be shown [44,45] that the POD basis defined above can be obtained from the left singular vector of the snapshot matrix $\mathbf{Y}=[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}]$. Let $\mathbf{Y}=\mathbf{U}\mathrm{\Sigma }{\mathbf{V}}^T$ be the singular value decomposition of $\mathbf{Y}$, where matrices $\mathbf{U}=[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_r]\in {\mathbb{R}}^{n\times r}$ and $\mathbf{V}=[{\mathbf{v}}_1,\dots ,{\mathbf{v}}_r]\in {\mathbb{R}}^{n_s\times r}$ are matrices with orthonormal columns and $\mathrm{\Sigma }=diag({\sigma }_1,\dots ,{\sigma }_r)\in {\mathbb{R}}^{r\times r}$ is a diagonal matrix with ${\sigma }_1\ge {\sigma }_2\ge \dots \ge {\sigma }_r>0$. Then the POD basis matrix of dimension k is ${\mathbf{U}}_k=[{\mathbf{u}}_1,\dots ,{\mathbf{u}}_k]\in {\mathbb{R}}^{n\times k}$, $k\le r$, i.e., ${\mathbf{U}}_k=\arg {min}_{{\mathbf{\Phi }}_k\in {\mathbb{R}}^{n\times k}}{\sum }_{j=1}^{n_s}{\parallel {\mathbf{y}}_j-{\mathbf{\Phi }}_k{\mathbf{\Phi }}_k^T{\mathbf{y}}_j\parallel }_2^2\mathrm{with}{\mathbf{U}}_k^T{\mathbf{U}}_k={\mathbf{I}}_k.$ It is well known [44,45] that this minimum error is given by

$$\mathbf{\sum }_{j=1}^{n_s}{\parallel {\mathbf{y}}_j-{\mathbf{U}}_k{\mathbf{U}}_k^T{\mathbf{y}}_j\parallel }_2^2=\mathbf{\sum }_{\ell =k+1}^r{\sigma }_{\ell }^2,$$

(5)

which is the sum of the neglected singular values ${\sigma }_{k+1},\dots ,{\sigma }_r$ from the SVD of $\mathbf{Y}$.

When the dimension n of snapshots is not too large, we can directly obtain the POD basis from the SVD of the snapshot matrix. However, in practice, n can be extremely large and computing the POD basis through a standard SVD might not be efficient. In this work, we use the randomized SVD to handle this problem.

4. Randomized SVD (rSVD)

The randomized SVD (rSVD) was proposed in [26] to reduce the computational complexity of the standard approach for computing SVD. It consists of two stages. In the first stage, random sampling is used to obtain a reduced matrix whose range approximates the range of $\mathbf{Y}$; in the second stage, this reduced matrix is then factorized inexpensively.

In particular, suppose we want to construct a low-dimensional basis with rank k by using rSVD. A random matrix $\mathrm{\Omega }\in {\mathbb{R}}^{n_s\times k}$ is first employed to generate the weights used in the linear combination of the data matrix $\mathbf{Y}$ to obtain a matrix with k columns, where $k<n_s$. Let

$$\mathbf{Z}=\mathbf{Y}\mathrm{\Omega }\in {\mathbb{R}}^{n\times k}.$$

Then the matrix $\mathbf{Z}$ is used to find an orthonormal basis $\mathbf{Q}\in {\mathbb{R}}^{n\times k}$, which can be done through many algorithms, such as Gram–Schmidt process, Householder transformation, or the QR decomposition. At this point, we can use the basis matrix $\mathbf{Q}$ to approximate the column span of $\mathbf{Z}$, as well as the column span of the data matrix $\mathbf{Y}$, i.e.,

$$\mathbf{Y}\approx \mathbf{Q}\mathbf{B},$$

where $\mathbf{B}\in {\mathbb{R}}^{k\times n_s}$ can be obtained from the orthogonal projection of $\mathbf{Y}$ onto the low dimensional subspace spanned by the columns of $\mathbf{Q}$, which gives $\mathbf{B}={\mathbf{Q}}^T\mathbf{Y}$. The second stage of rSVD then computes the SVD of this small matrix $\mathbf{B}$. Suppose $\mathbf{B}=\hat{\mathbf{U}}\hat{\mathrm{\Sigma }}{\hat{\mathbf{V}}}^T$ is the SVD of $\mathbf{B}$, where matrices $\hat{\mathbf{U}}=[{\widehat{\mathbf{u}}}_1,\dots ,{\widehat{\mathbf{u}}}_{\widehat{r}}]\in {\mathbb{R}}^{k\times \widehat{r}}$ and $\hat{\mathbf{V}}=[{\widehat{\mathbf{v}}}_1,\dots ,{\widehat{\mathbf{v}}}_{\widehat{r}}]\in {\mathbb{R}}^{n_s\times \widehat{r}}$ are matrices with orthonormal columns, $\widehat{\mathrm{\Sigma }}=diag({\widehat{\sigma }}_1,\dots ,{\widehat{\sigma }}_{\widehat{r}})\in {\mathbb{R}}^{\widehat{r}\times \widehat{r}}$ is a diagonal matrix with ${\widehat{\sigma }}_1\ge {\widehat{\sigma }}_2\ge \dots \ge {\widehat{\sigma }}_{\widehat{r}}>0$ and $\widehat{r}=\mathrm{rank}\left(\mathbf{B}\right)$. Then, we finally obtain the approximated POD basis or the left singular vectors of $\mathbf{Y}$ from the product $\mathbf{Q}\hat{\mathbf{U}}$.

In practice, the matrix $\mathbf{Z}$ constructed from an n-by-k random matrix may not be a good representation of the original data matrix $\mathbf{Y}$. Consequently, the basis matrix $\mathbf{Q}$ may not give an accurate approximation for the column space of $\mathbf{Y}$. To solve this issue, we consider an improved algorithm for rSVD by using an oversampling strategy given in [26], which uses an n-by-$(k+p)$ random matrix instead of n-by-k, where p is a positive integer of additional random vectors. Algorithm 2 summarizes these two stages of the rSVD.

Algorithm 2 Randomized Singular Value Decomposition (rSVD)

INPUT: ${\left\{{\mathbf{y}}_{\ell }\right\}}_{\ell =1}^{n_s}\subset {\mathbb{R}}^n$, k where $k\le rank\left({\left\{{\mathbf{y}}_{\ell }\right\}}_{\ell =1}^{n_s}\right)$ OUTPUT: POD basis: ${\mathbf{U}}_k$, Singular values: ${\mathbf{S}}_k$   1: Form $\mathbf{Y}=[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_{n_s}]$   2: Construct random matrix $\mathrm{\Omega }=rand(M,K_0)$, $K_0=k+p$, where p is a positive integer   3: $\mathbf{Q}=orthog\left(\mathbf{Y}\mathrm{\Omega }\right)$   4: Set $\mathbf{B}={\mathbf{Q}}^T\mathbf{Y}\in {\mathbb{R}}^{K_0\times n_s}$   5: Compute SVD of $\mathbf{B}$: $\mathbf{B}=\hat{\mathbf{U}}\hat{\mathrm{\Sigma }}{\hat{\mathbf{V}}}^T$   6: Basis: $\mathbf{U}=\mathbf{Q}\hat{\mathbf{U}}$   7: Truncation: ${\mathbf{U}}_k=\mathbf{U}(:,1:k)$, ${\mathbf{S}}_k=\mathrm{\Sigma }(:,1:k)$

5. Accelerated POD-LS Method (Gappy POD)

This section aims to discuss an efficient approach that can reduce the computational complexity for solving the standard LS approach (LS-POD approach) used in the data reconstruction technique described in Section 2. A greedy-based approach called the discrete empirical interpolation method (DEIM) is first discussed. Then, two variants of missing data approximations using DEIM are later introduced.

5.1. Discrete Empirical Interpolation Method (DEIM)

DEIM was first introduced for the purpose of approximating the nonlinear term in the differential equations [35]. In this work, the interpolation indices from this method will be used to reduce the computational complexity of solving the least-squares approximation in Step 3 of Algorithm 1. This section describes DEIM in a general setting and provides the corresponding greedy algorithm for selecting important components that can be used in the approximation.

Consider a vector $\mathbf{f}\in {\mathbb{R}}^n$. For a given orthonormal matrix $\mathbf{U}\in {\mathbb{R}}^{n\times m}$ with $m\le n$, an approximation of this vector $\mathbf{f}$ obtained from projecting onto the subspace $\mathrm{span}\left\{\mathbf{U}\right\}$ is given by

$$\mathbf{f}\approx \mathbf{U}\mathbf{d},$$

(6)

for some vector $\mathbf{d}\in {\mathbb{R}}^m$. The vector $\mathbf{d}$ can be obtained by using an interpolation method. That is, it can be solved from a square system that extracts only m rows of (6). In particular, suppose ${\wp }_1,{\wp }_2,\dots ,{\wp }_m$ are the indices of the selected components in $\mathbf{f}$ and define $\mathbf{P}=[{\mathbf{e}}_{{\wp }_1},\dots ,{\mathbf{e}}_{{\wp }_m}]\in {\mathbb{R}}^{n\times m}$, where ${\mathbf{e}}_{{\wp }_i}={[0,\dots ,0,1,0,\dots ,0]}^T\in {\mathbb{R}}^n$ is the ${\wp }_i$-th column of the identity matrix ${\mathbf{I}}_n\in {\mathbb{R}}^{n\times n},$ for $i=1,\dots ,m$. Note that, as in the previous section, pre-multiplying ${\mathbf{P}}^T$ is equivalent to extracting the m rows corresponding to the interpolation indices ${\wp }_1,\dots ,{\wp }_m$. The coefficient vector $\mathbf{d}$ in the DEIM approximation solves the following minimization problem

$$\underset{\mathbf{d}\in {\mathbb{R}}^m}{min}{\parallel {\mathbf{P}}^T\mathbf{f}-{\mathbf{P}}^T\mathbf{U}\mathbf{d}\parallel }_2^2,$$

which gives $\mathbf{d}={\left({\mathbf{P}}^T\mathbf{U}\right)}^{+}{\mathbf{P}}^T\mathbf{f}$, where ${\left({\mathbf{P}}^T\mathbf{U}\right)}^{+}={\left[{\left({\mathbf{P}}^T\mathbf{U}\right)}^T\left(\mathbf{P}\mathbf{U}\right)\right]}^{-1}{\left({\mathbf{P}}^T\mathbf{U}\right)}^T$ is the pseudo-inverse of ${\mathbf{P}}^T\mathbf{U}$. Notice that ${\mathbf{P}}^T\mathbf{U}$ is a square matrix of size $m\times m$. When the indices ${\wp }_1,\dots ,{\wp }_m$ are selected by the DEIM procedure given in Algorithm 3, it was shown in [35] that ${\mathbf{P}}^T\mathbf{U}$ is invertible. i.e., ${\left({\mathbf{P}}^T\mathbf{U}\right)}^{+}={\left({\mathbf{P}}^T\mathbf{U}\right)}^{-1}$, and the approximation in (6) for $\mathbf{f}$ becomes

$$\begin{array}{c}\hfill \mathbf{f}\approx \mathbf{U}{\left({\mathbf{P}}^T\mathbf{U}\right)}^{-1}{\mathbf{P}}^T\mathbf{f}.\end{array}$$

(7)

This approximation (7) is called the DEIM approximation. The sets of indices $\{{\wp }_1,{\wp }_2,\dots ,{\wp }_m\}$ are obtained by the DEIM index selection algorithm [35], shown in Algorithm 3, which is a greedy procedure that aims to capture the variation of the spatial behavior of the input basis using the infinity norm.

Algorithm 3 Algorithm to create for Interpolation Indices DEIM

INPUT:${\left\{{\mathbf{u}}_{\ell }\right\}}_{\ell =1}^m\subset {\mathbb{R}}^n$ linearly independent OUTPUT: $\overrightarrow{\wp }={[{\wp }_1,\dots ,{\wp }_m]}^T\in {\mathbb{R}}^m$ and $\mathbf{P}$   1: ${\wp }_1=\arg {max}_{i=1,2,\dots ,n}\left\{\right|{\mathbf{u}}_{i1}\left|\right\}$   2: $\mathbf{U}=\left[{\mathbf{u}}_1\right],\mathbf{P}=\left[{\mathbf{e}}_{{\wp }_1}\right],\overrightarrow{\wp }=\left[{\wp }_1\right];$   3: for $j=2$ to m do   4:  Solve $\left({\mathbf{P}}^T\mathbf{U}\right)\mathbf{z}={\mathbf{P}}^T{\mathbf{u}}_j;$   5:  $\mathbf{r}={\mathbf{u}}_j-\mathbf{U}\mathbf{z}$   6:  ${\wp }_j=\arg {max}_{i=1,\dots ,n}\left\{\right|{\mathbf{r}}_i\left|\right\}$   7:  $\mathbf{U}\leftarrow \left[\mathbf{U}{\mathbf{u}}_j\right],\mathbf{P}\leftarrow \left[\mathbf{P}{\mathbf{e}}_{{\wp }_j}\right],\overrightarrow{\wp }\leftarrow \left[\begin{array}{c}\overrightarrow{\wp }\\ {\wp }_j\end{array}\right]$   8: end for

From Algorithm 3, DEIM selects the interpolation indices so that the approximation has the smallest error $\mathbf{r}={\mathbf{u}}_j-\mathbf{U}\mathbf{z}$ in each iteration j. The procedure of DEIM Algorithm 3 can be described as follows. First, the input basis set ${\left\{{\mathbf{u}}_{\ell }\right\}}_{\ell =1}^m$ of rank m. The space spanned by this basis set, in general, is expected to approximately contain the vector $\mathbf{f}$. Then, it selects the first index of a component in the first basis vector ${\mathbf{u}}_1$ with the largest absolute value. Next, each of the other indices is selected from the component with the largest absolute residual error $\mathbf{r}={\mathbf{u}}_{\ell }-\mathbf{U}\mathbf{z}$ in each step.

More details on DEIM approximation and its corresponding error bound can be found in [35]. Some extensions of this error bound to the state-space error estimate can be found in [36,37].

5.2. Accelerated POD-LS Method by Using DEIM

This work particularly focuses on the situation when the incomplete data $\widehat{\mathbf{y}}$ is in a high-dimensional space, i.e., the value of n is large, which may result in a large number of known components $n_c$, even though there are a lot of unknown components $n_g$. In this case, the main computational work in approximating missing components may occur while solving for the coefficient vector $\mathbf{a}$ in (1) or in Step 3 of Algorithm 1. One possible way to reduce this computational work is to use only a small number of known components to specify $\mathbf{a}\in {\mathbb{R}}^k$. These components have to be carefully selected so that they can represent all other $n_c$ components and maintain the same accuracy in the approximation. For this purpose, the procedure shown in Algorithm 3 for selecting DEIM indices, which corresponde to important components, will be used as described.

Recall from Step 3 of Algorithm 1 that it has to solve for a vector $\mathbf{a}$ from the following minimization problem:

$$\underset{\mathbf{a}\in {\mathbb{R}}^k}{min}{\parallel {\widehat{\mathbf{y}}}_c-{\mathbf{V}}_c\mathbf{a}\parallel }_2^2.$$

Suppose ${\mathbf{V}}_c\in {\mathbb{R}}^{n_c\times k}$ has linearly independent columns. Then, for a given DEIM dimension $m\le min\{n_c,k\}$, the columns of ${\mathbf{V}}_c$ can be used as an input of Algorithm 3 to select m DEIM indices that can cover the variations of $n_c$ components.

In many applications, when n is much larger than the number of available complete data $n_s$, i.e., $n>n_s\ge k$, we have $min\{n,k\}=k$. Notice that, based on the standard DEIM approximation, dimension m is limited to the truncated dimension k of $\mathbf{V}$. We can improve the accuracy of the reconstruction by further including more important components of the complete data, i.e., using dimension m that is larger than k. However, using $k>m$ is impossible if we follow the standard DEIM approximation and use ${\mathbf{V}}_c$ as an input to Algorithm 3, because ${\mathbf{V}}_c$ has only k columns, which implies Algorithm 3 can give, at most, k indices. To resolve this issue, instead of using ${\mathbf{V}}_c$ directly, we will consider a matrix constructed from the full set of the POD basis for the complete snapshot matrix $\mathbf{Y}$ with rows selected from the indices of the known components. In particular, if we define $\overline{\mathbf{V}}\in {\mathbb{R}}^{n\times r}$ to be the left singular matrix from SVD of the complete snapshot matrix $\mathbf{Y}$, where $r=rank\left(\mathbf{Y}\right)$, the input to Algorithm 3 would be the matrix obtained from $\overline{\mathbf{V}}$ with rows corresponding to the complete components, i.e.,

$${\overline{\mathbf{V}}}_c:={\mathbf{C}}^T\overline{\mathbf{V}}\in {\mathbb{R}}^{n_c\times r},$$

(8)

which comes from $\overline{\mathbf{V}}$ with selected rows corresponding to the $n_c$ known components in $\mathcal{C}$. In this case, we can use the DEIM algorithm to select up to r indices that correspond to the most (r) relevant components of the reconstruction problem in (1). Note that the columns in $\mathbf{V}$ are the first k columns in $\overline{\mathbf{V}}$.

Suppose ${\overline{\mathbf{V}}}_c\in {\mathbb{R}}^{n_c\times r}$ has r linearly independent columns with $r\le n_c$. Then, the columns of ${\overline{\mathbf{V}}}_c$ can be used in Algorithm 3 to select m DEIM indices, where $m\le r$. In this case, suppose ${\wp }_1,{\wp }_2,\dots ,{\wp }_m$ are the output indices of DEIM Algorithm 3 and let $\mathbf{P}=[{\mathbf{e}}_{{\wp }_1},\dots ,{\mathbf{e}}_{{\wp }_m}]\in {\mathbb{R}}^{n_c\times m}$ as defined earlier in this section. Vector $\mathbf{a}$ in (1) can be computed from a smaller least-squares problem:

$$\underset{\mathbf{a}\in {\mathbb{R}}^k}{min}{\parallel {\mathbf{P}}^T{\widehat{\mathbf{y}}}_c-{\mathbf{P}}^T{\mathbf{V}}_c\mathbf{a}\parallel }_2^2.$$

(9)

Note that we have assumed that ${\overline{\mathbf{V}}}_c\in {\mathbb{R}}^{n_c\times r}$ has linearly independent columns, which may not hold in general cases. To avoid this assumption, we can apply SVD to the matrix ${\overline{\mathbf{V}}}_c$ and use the set of first m corresponding left singular vectors as an input to the DEIM Algorithm 3. The accelerated POD-LS steps are summarized in Algorithm 4 for both cases of (i) using ${\overline{\mathbf{V}}}_c$ directly as an input and (ii) using the set of left singular vectors of ${\overline{\mathbf{V}}}_c$ as an input to Algorithm 3 for selecting the most (m) important components. Note that, as later shown in the numerical tests, the reconstruction results using case (ii) may not be as accurate as the ones using case (i).

Algorithm 4 Accelerated POD-LS method for approximating missing data

INPUT: - Complete snapshot set ${\left\{{\mathbf{y}}_j\right\}}_{j=1}^{n_s}\subset {\mathbb{R}}^n$ and, dimension $k\le rank\left({\left\{{\mathbf{y}}_j\right\}}_{j=1}^{n_s}\right)$ - Incomplete data $\widehat{\mathbf{y}}\in {\mathbb{R}}^n$ with known entries ${\widehat{y}}_j$, $j\in \mathcal{C}$ and unknown entries ${\widehat{y}}_j$, $j\in \mathcal{G}$ OUTPUT: - Approximation: ${\widehat{\mathbf{y}}}_g=\left[{\widehat{y}}_j\right]$, $j\in \mathcal{G}=\{g_1,g_2,\dots ,g_{n_g}\}$   1: Create snapshot matrix $:\mathbf{Y}=[{\mathbf{y}}_1,\dots ,{\mathbf{y}}_{n_s}]\in {\mathbb{R}}^{n\times n_s}$ and let $r=rank\left(\mathbf{Y}\right)$.   2: Compute POD basis $\mathbf{V}$ of rank $k\le r$ for $\mathbf{Y}$ from rSVD in Algorithm 2.   3: Find coefficient vector $\mathbf{a}$     3.1 Find indices $\overrightarrow{\wp }={[{\wp }_1,\dots ,{\wp }_m]}^T\in {\mathbb{R}}^m$ and $\mathbf{P}$ from Algorithm 3 by using input from either      (i) ${\overline{\mathbf{V}}}_c$ defined in (8), or      (ii) Left singular vectors of ${\overline{\mathbf{V}}}_c$ using SVD or rSVD     3.2 Solve $\mathbf{a}$ from (9): ${min}_{\mathbf{a}\in {\mathbb{R}}^k}{\parallel {\mathbf{P}}^T{\widehat{\mathbf{y}}}_c-{\mathbf{P}}^T{\mathbf{V}}_c\mathbf{a}\parallel }_2^2$.   4: Compute the approximation ${\widehat{\mathbf{y}}}_g\approx {\mathbf{V}}_g\mathbf{a}$.

Remarks: In practice, the computation for DEIM indices can be performed in advance and reused for many incomplete snapshots. Once the DEIM indices are found from Algorithm 3, the terms ${\mathbf{P}}^T{\widehat{\mathbf{y}}}_c$ and ${\mathbf{P}}^T{\mathbf{V}}_c$ can be computed without actually performing matrix multiplication since this can be performed through selected row indices.

The next section discusses the computational complexity reduction of the proposed approach.

6. Computational Complexity

In this work, as shown in Algorithm 4, we have reduced the computational complexity in two main parts. The first one is based on using the rSVD for constructing the projection basis, and the other one is based on applying the DEIM procedure to efficiently select a small subset of the most relevant components in the least-squares problem used in the reconstruction process.

In the first part, for a given complete data matrix $\mathbf{Y}\in {\mathbb{R}}^{n\times n_s}$, computing a projection basis of rank k for $\mathbf{Y}$ by using SVD can be performed efficiently via a rank-revealing QR factorization and then manipulating the factors to obtain the final SVD. The cost of this approach is typically $\mathcal{O}\left(n{n}_{s}k\right)$ floating-point operations (flops) [26,46]. The rSVD can approximate SVD with computational cost that is reduced to $\mathcal{O}(n{n}_{s}log\left(k\right)+(n+n_s)k^2)$ flops. This computational cost can be reduced further when using a random matrix $\mathrm{\Omega }$ that has some internal structure to accelerate the computation of $\mathbf{Y}\mathrm{\Omega }$ in Step 3 of Algorithm 2.

In the second part of reducing the computational cost for solving the least-squares problem, we incorporated the DEIM procedure to reduce the size of the problem in the reconstruction process. In general, the well-known approaches for solving least-squares problems are based on using normal equations, QR factorization and SVD [47,48,49]. The last two approaches are efficient for large-scaled problems [48,49]. For the coefficient matrix ${\mathbf{V}}_c$ of size $n_c$-by-k with $n_c\gg k$, the computational cost of solving the least-squares problem in (1) using the QR factorization or the SVD approaches is in the of order $\mathcal{O}\left(n_c{n}_s^2\right)$ flops. Therefore, when the DEIM algorithm is used to reduce the size of the least-squares problem to a coefficient matrix of size $m\times k$ with $m\ll n_c$ as shown in (9), the computational cost is reduced to $\mathcal{O}\left(m{n}_s^2\right)$ flops. Note that the complexity of performing the DEIM procedure is $\mathcal{O}\left(m^3\right)$, which is negligible for $m<n_s$ and $m\ll n_c$ in most practical cases when compared to the overall complexity of the reconstruction process.

The next section investigates the effectiveness of the proposed techniques described in Algorithm 4 through different cases of numerical tests. The reduction in computational cost explained above is also reflected in these reconstruction results.

7. Numerical Results

The numerical experiments in this section compare the efficiency and accuracy of the standard least-squares approach using the standard SVD with the accelerated least-squares approach using rSVD with DEIM. These results are obtained from MATLAB 2021b running on a computer with the following specifications Intel Core i5-1035G4, CPU 1.10, 1.50 GHz, and 8 GB RAM.

We consider three numerical experiments. The first two numerical experiments use data from images with missing pixels. The last one uses data from the solution of 2D nonlinear flow in porous media with some missing components at certain time instances. This dataset can be considered as a sequence of images in the form of movie data. The accuracy of the reconstruction is measured by the average relative error of the reconstructed missing data. The CPU times used in the reconstruction process will be scaled with the ones used by the standard LS approach to see the speedup of the simulation time.

7.1. Numerical Test 1

This section considers a standard test image, called the Lena image, of size $200\times 200$ pixels. The goal is mainly to investigate the accuracy of the reconstruction by the proposed accelerated LS approach for different amounts of missing pixels. The speedup of the reconstruction time will be considered later in the next two numerical tests. We consider the cases when 30%, 50%, and 65% pixels are missing uniformly on the original image, as shown in Figure 1. The reconstruction results for all these cases are shown in Figure 2 when using the standard POD-LS approach with $k=30$ and the proposed accelerated POD-LS approach with $k=30$ and $m=50$. Figure 3 demonstrates the average relative errors of the reconstructed missing components using the accelerated POD-LS approach for $k=30$ and different values of $m=10,20,30,\dots ,80$ for the cases of 30%, 50%, and 65% missing pixels. It shows the convergence of the accelerated POD-LS approach to the standard POD-LS approach as dimension m increases. When there are more missing pixels, this convergence seems to be slower, i.e., the error of the accelerated POD-LS method approaches the error of the standard method at a larger dimension m. Note that, in Figure 3, the accelerated POD-LS approach for both cases (i) and (ii) in Algorithm 4 are performed. The errors for case (ii) seem to be larger than the ones for case (i), and these errors for case (ii) seem to be oscillating with no trend when $m<30$. This suggests that to select the DEIM components it is more appropriate to use the matrix constructed from the rows of POD basis matrix corresponding to the indices of the know components in case (i) than to use its orthogonalized basis in case (ii).

7.2. Numerical Test 2

This section considers a test image of size $807\times 605$ pixels. The corresponding incomplete image has uniform missing pixels equal to 50%, as shown in Figure 4. The goal of this numerical test is to investigate the accuracy and the speedup of the proposed accelerated POD-LS approach when compared to the standard POD-LS approach.

Figure 5 illustrates the reconstructed images by the standard POD-LS approach with basis from SVD using dimension $k=10,50$ and by the accelerated POD-LS approach with basis from rSVD as described in Algorithm 4 case (i) using $k=10$ with $m=10,20,50$ and $k=50$ with $m=50,80,150$. The corresponding values of the average relative errors and the scaled CPU time of the reconstructed images in Figure 5 are given in

Table 1. (Numerical Test 2) The average relative error of the reconstructed missing data and the corresponding scaled CPU times for the reconstruction when the standard POD-LS method and the accelerated POD-LS method with rSVD are used. Dimensions for POD: $k = 10$, DEIM: $m = 10, 20, 50$.

Method	dim POD	dim DEIM	Relative Error	CPU Time (Scaled)
Standard POD-LS	k = 10	-	$0.0627$	1
Accelerated POD-LS	k = 10	m = 10	$0.8566$	$0.01382$
	$k=10$	$m=20$	$0.1239$	$0.02425$
	$k=10$	$m=50$	$0.0969$	$0.10185$

and

Table 2. (Numerical Test 2) The average relative error of the reconstructed missing data and the corresponding scaled CPU times for the reconstruction when the standard POD-LS method and the accelerated POD-LS method with rSVD are used. Dimensions for POD: $k = 50$, DEIM: $m = 50, 80, 150$.

Method	dim POD	dim DEIM	Relative Error	CPU Time (Scaled)
Standard POD-LS	k = 50	-	$0.01753$	1
Accelerated POD-LS	$k=50$	$m=50$	$9.20223$	$0.02120$
	$k=50$	$m=80$	$0.05181$	$0.03578$
	$k=50$	$m=150$	$0.02855$	$0.04114$

, respectively, for the cases of $k=10$ and $k=50$. From these tables, the CPU time of the accelerated POD-LS approach is shown to be roughly reduced by 10–50 times when compared to the standard POD-LS approach for the same level of accuracy. Figure 6 demonstrates the trend of the average relative errors for these 2 approaches for different dimensions $k=10,30,50$ with DEIM dimension m ranging from 10 to 210. Notice that, as the dimensions k and m get larger, the overall errors seem to decrease. Figure 6 also shows that as m increases, the errors of the accelerated POD-LS approach decrease and get closer to the errors of the standard POD-LS approach. As in the previous numerical test, in Figure 6, we consider the errors of the accelerated POD-LS approach for both cases (i) and (ii) in Algorithm 4. The errors for case (ii) seem to be larger in all cases of k considered in Figure 6. This confirms the result in the previous numerical test that we should use the POD basis with rows extracted from corresponding to the known component directly, instead of using its orthogonalized basis, to select the DEIM points used in the accelerated least-squares approach.

7.3. Numerical Test 3

This section uses snapshots from the solutions of miscible flow, which changes continuously with no particular pattern for each time instance, as shown in Figure 7. Each snapshot has $n=15,000$ pixels. The POD basis is constructed from $n_s=200$ complete snapshots and used to reconstruct the missing pixels of 50 incomplete snapshots. Each incomplete snapshot has $30\%$ missing pixels, as demonstrated in Figure 8. The average relative error for the reconstruction and the corresponding CPU time are displayed in Figure 9 for different cases of POD dimensions $k=10,30,50,100$. For each of these cases, different values of DEIM dimensions $m=10,30,\dots ,190$ are used in the accelerated POD-LS approach. Figure 9 demonstrates the convergence of the reconstruction error and the speed up of the CPU time of the accelerated POD-LS and the standard POD-LS approaches as in the previous numerical tests. In this numerical test, the accelerated POD-LS can be used to speed up the CPU time by roughly $\mathcal{O}\left(10\right)$ to $\mathcal{O}\left(100\right)$ times. Note that the dimension of m that is suitable in each case of k is generally selected from the corner of the error plot before the error becomes constant. Therefore, from Figure 9, the error plot from the accelerated least-squares approach using case (i) in Algorithm 4 suggests that we should use an m that is slightly larger than k, for each case of $k=10,30,50,100$.

To clearly illustrate the effect of dimension k on the POD basis, Figure 10 compares the average relative error and the corresponding CPU time used in the reconstruction from the accelerated POD-LS approach using $m=k$ with the ones from the standard POD-LS approach for each k ranging from 10 to 190. Figure 10 shows that the accelerated POD-LS approach can provide almost the same accurate reconstruction as the standard POD-LS roughly 100 times faster. This numerical test, therefore, demonstrates the potential of the proposed approach for other large-scale problems.

8. Conclusions

This work presented an accelerated data reconstruction technique based on the least-squares method, which can reduce computational time in two aspects. The first one is based on using rSVD in computing a low-dimensional basis used in least-squares approximation. The second aspect that can speed up the reconstruction process is based on applying the DEIM greedy point selection procedure on the least-squares approximation. Accuracy and computational time reduction of the proposed method are demonstrated through three numerical experiments, which considered standard test images, as well as a set of sequential two-dimensional flow images in a similar form as video data. These numerical results show that the proposed method can speed up the reconstruction process up to 100 times with a small accuracy trade-off. This work can be extended by employing different approaches to construct the projection basis, e.g., [50,51,52]. It is also possible to adapt other approaches used for solving differential equations, e.g., in [53,54,55,56], to select the relevant components used in the least-squares approximation for data reconstruction. These extensions are left for future investigations.