1. Introduction
Consider estimating
from the overdetermined linear system
where the error exists in both the right-hand side
b and the data matrix
A and
. In this case, the total least squares (TLS) model should be appropriate to adopt (cf. [
1,
2]). The TLS approach is just to find a perturbation with the minimum Frobenius norm to make the system (
1) a compatible system
The TLS method is widely used in various scientific fields, such as physics, automatic control, signal processing, statistics, economics, biology, medicine etc. In essence, a solution of a TLS problem can be expressed by a singular value decomposition of the augmented matrix
When the dimensions of
A are not too large, one can use the truncated-SVD (TSVD) method. When the dimensions of
A become large, this approach becomes prohibitive because the SVD algorithm is of complexity
. The above considerations lead us to consider Krylov iterative methods, that do not alter the matrix
A. The methods have the attractive feature just like the Lanczos methods—that when
n increases, the computed extreme singular elements rapidly become good approximations to the exact ones, and are satisfactorily accurate even if
k is far less than
n theoretically [
1]. Nevertheless, the orthonormal properties of the Krylov basis strongly support the use of these Householder matrix-based algorithms. This is particularly true when we need to be sure that the perturbed problem we are solving has to conserve some spectral similarity properties. This will be especially relevant when we need to compute approximations of the TLS problems. In view of this, we consider applying the Householder bidiagonalization algorithm and the NIPALS PLS algorithm posed by Å. Bj
rck [
3] to TLS problems, the formed Householder bidiagonalization total least squares (HBITLS) algorithm, and NIPALS-TLS algorithm, respectively. Furthermore, we find that the HBITLS and NIPALS-TLS algorithms also compute the same approximate solutions for the TLS problems.
When it comes to practical problems, the arithmetic will be inaccurate and there will be errors in each step of the calculation. Arithmetic operations running on the computer have finite precision, so there will be rounding errors as long as there are numerical computations. These rounding errors cause the calculation quantities to be different from their theoretical values. One of the design principles of the floating-point operation is that it should encourage experts to develop robust, efficient, and portable numerical programs, enable the handling of arithmetic exceptions, and provide for the development of transcendental functions and high-precision arithmetic [
4]. The results in the roundoff error analysis in Lanczos-type methods obtained by Paige [
5,
6,
7] played an important role in interpreting the behavior of the Lanczos method in finite-precision computations. Parlett and Scott [
8] used the results of the roundoff error analysis as the basis for suggesting a modification of the Lanczos method, which they called selective orthogonalization [
8,
9,
10]. In addition, in many practical problems, the stop criterion can be safely selected on the basis of the rounding error analysis of the original problem, thereby diminishing the need for an extremely precise approximation of the algebraic problem solution [
4]. As far as we know, the roundoff error analysis of the approximation TLS solutions obtained by using the Householder bidiagonalization procedure was not systematically performed in the literature. Hence, in this paper, we analyzed the propagation of the roundoff error during the process of the HBITLS algorithm and found that the HBITLS algorithm and NIPALS-TLS algorithm are mixed forward–backward stable.
The paper is organized as follows. The HBITLS algorithm and NIPALS-TLS algorithm were established, by which the same approximate TLS solution was obtained in
Section 2.
Section 3 analyzes the propagation of the roundoff error during the process of the HBITLS algorithm. A brief conclusion is shown in the last section.
3. Roundoff Error Analysis
In this section, we analyze the propagation of roundoff error during the process of the HBITLS algorithm and get the mixed forward–backward stability of the HBITLS algorithm and NIPALS-TLS algorithm naturally. The total roundoff error during the process of the HBITLS algorithm can be divided into the following four parts:
First, we can find that the HBITLS algorithm solves the original TLS problem (
2) to a perturbed TLS problem. The propagation of the roundoff error of a Householder matrix in the HBITLS algorithm is advantageous when performing numerical computations.
From now one, we will denote by
the machine precision under consideration. In [
15], it shows that the computed Householder matrix
comes near the exact Householder matrix
H itself:
Moreover, for a vector
, the computed updates with
are very close to the exact updates with
H:
and, in general,
The following lemma tells us that the reduced system calculated by the HBITLS algorithm is equivalent to the system formed after the original system has been disturbed. , and are the floating-point computation of the matrices , and in HBITLS algorithm, respectively.
Lemma 2. Let be the computed bidiagonalization matrix matrix obtained by the HBITLS algorithm. Then, there comes a perturbation matrix E and exists two column orthogonal matrices and s.t.andwhere n is the number of columns of matrix A. Furthermore, the matrix is an orthonormal basis of with a perturbation vector , where Proof. We prove this theorem by induction. The key point is that we should show the computed matrix, which will be shown by introduction from (
3), for
as follows
For
, first, let
,
, a Householder matrix
is found s.t.
. Set
ref. [
16] tells us that, corresponding to matrix
, we can find a Householder matrix to make
with
Next, let
where
Similarly, for
the computed result can be written as
where
Now, we set the Householder matrix
s.t.
is upper bidiagonal matrix. We know
only works the vector
so there’s no change for the 1st column of
when producted by
and
. Likely, there’s a Householder matrix
s.t.
is bidiagonal matrix in theory, but the algorithm computes a matrix
in practice [
16] such that
where
For the
kth step, assume that the HBITLS algorithm has calculated the matrices
associated with the Householder matrices
. Then, after
k steps, we can get the following result:
where
We know
and
where
For
the floating-point vector is
where
Likely, there is a Householder matrix
, which only works on the vector
, s.t.
is the upper bidiagonal matrix. The algorithm computes a matrix
in practice so that
where
Then the floating-point matrix is obtained, such that
Let
,
be the matrices, such that
and
, respectively, we find that the first
rows of each
is
Let
be the
i-th column of
, then the results are as follows
Then there comes
it is an
upper bidiagonal matrix. And,
we obtain
Since
is the
j-th column
of the matrix
, if we denote by
we have
and so we can obtain
If we cut off the first column of the matrix, we can set
with
such that
, here
is an
upper bidiagonal matrix. If we denote
then
In addition,
let
be the matrix made up of the first
j columns of
, we obtain
and, from the structure of
there comes
Then we can write
owing to
and so
If
we can finish the proof of the first part of the lemma, because
Finally, we prove that the subspace spanned by the columns of the matrix is an orthogonal basis of a Krylov space. Let , and form . We know that and set , then we have
We still prove the rest of the theorem by induction; that is, to prove
where each vector
has only the first
components, which are different from zero.
For
, we have
since the last component of the vector
is zero, in addition, except for the first two components, the rest of the components of vector
are all zero.
Suppose for a given
i the following relation is true,
and in the next step, we will show it is true for
.
From the inductive hypothesis, we know that only the first
components of
are not zero; therefore, the last component is zero of the vector
with
non-zero elements. Then there comes a conclusion that
and, hence, the lemma is proved. □
Based on Lemma 2 and Algorithm 1, for the bidiagonalization matrix
obtained by the HBITLS algorithm, one can find an orthonormal matrix
s.t.
where
is just an orthogonal basis of
. Based on this, we know that the first part of HBITLS, in exact arithmetic, gives the exact basis of the perturbed Krylov space
. A perturbation bound for TLS solutions is given by Xie and Wei [
17], see Lemma 3, which is related to the smallest singular value
of
, the TLS solution
, and the residual
. Let
and
. Then, the unique solution of the perturbed TLS problem can be expressed as
. Denote
and
with
The perturbation bound is obtained under the genericity condition
, where
is the smallest singular value of
ALemma 3 ([
17])
. Consider the TLS problem (2) and assume that the genericity condition holds. If is sufficiently small, then we obtain that Suppose that and are the exact TLS solutions of and respectively. The error introduced in this part of the HBITLS algorithm is the inherent error, so we can give by Lemma 3 and Lemma 2 easily, see Theorem 2.
Secondly, let us consider the error between the TLS solution of the system
and the approximation solution
of the system computed by the HBITLS algorithm at step
k with the exact arithmetic, i.e.,
is the exact solution of the reduced TLS problem
For convenience, define
and let
where
W and
are orthogonal matrices of dimension
m and
, respectively,
is an
diagonal matrix whose diagonal entries
are the singular values of
, sorted in non-increasing order. Let
be the subspace angle between
and
.
Lemma 4. Let denotes the essential TLS solution to the linear system (2) satisfying genericity condition and be the approximation solution obtained from Algorithm 1. Then Then we have an orthonormal matrix
with the partition
such that
(i.e., G “forms a complete space”). From Equation (
27)
there comes
and, therefore
From the CS theorem [
18], we know that
. Then
It was noticed that
denotes the sine of the subspace angle between
and
Hence, the upper bound can be proved as follows
For the lower bound, we have
Since , and this proves the upper bound case. □
Thirdly, we need to consider how to solve problem (
9) and show that the error is between the solution obtained by this method and the theoretical solution. Let the computed solution be
, where
is the computed solution of the problem (
23).
In [
19], James and Kahan posed an algorithm named QR iteration with a zero shift, which guaranteed forward stability. Furthermore, an implicit algorithm about it is given. Error analysis including the singular values and singular vectors are also given, which is just what we’ve needed.
Lemma 5 ([
19])
. Let the matrix obtained by running the implicit zero-shift QR algorithm on a bidiagonal matrix B with . Suppose that all perturbation angles θ emerged from the operations of the algorithm satisfy . Let and are the singular values of B and respectively. Ifthen we have:Moreover, let be the singular values of produced after k steps of the implicit zero-shift QR algorithm. Then if condition (30) holds, and all perturbation angles θ satisfy , we obtain James and Kahan [
19] also give the relative differences between the singular vectors of
B and the ones of
.
Lemma 6 ([
19])
. Let be the singular value of be an unreduced bidiagonal matrix B with and being its corresponding left and right singular vectors, respectively. Let and be the singular vectors computed by the implicit zero-shift QR algorithm. Then the bound of the errors in are shown by Then, combining with the perturbation bound of TLS given in [
20] as shown in Lemma 7, we can give the error estimate
.
If let is a rank-k matrix approximation to , and . Let represent a perturbation of , denote a rank-k matrix approximation to and define , then
Lemma 7 ([
20])
. Let and denote the TLS solution and the perturbed TLS solution. If (the k-th singular value of A) may be provided. Thenwhere and are the smallest right singular vectors of and respectively. In summary, if let be the final computed solution at the k-th step, then roundoff error analysis of HBITLS algorithm for TLS problem can be shown as follows
Theorem 2. Considering the HBITLS algorithm at step k, the roundoff error emerged during the algorithm can be bounded as follows:where and are defined in (26) similarly, is the computed smallest right singular vector of . Proof. The roundoff error can be composed of the following parts
and we analyze these errors separately.
For the first part,
and
are the TLS solutions of the systems
and
, respectively, in line with Lemma 2, so the error of this part is the inherent error. Then, combining with Lemma 3, we have
where
and
see Lemma 3.
For the second part, this error is owing to the approximate solution of
obtained by using HBITLS algorithm after
k steps with the exact arithmetic. Lemma 4 tells us that
For the third part, it is noticed that
we have that
where
is the roundoff error stem from the projected TLS solution. Since
is a special form of bidiagonal matrices, we consider using the implicit zero-shift QR algorithm to perform singular value decomposition. (
31) gives an upper bound of the angle between the solution vectors, and combining Lemma 7, we know
where
is the subspace angle between the sub-spaces produced by the smallest right singular vector and the computed smallest right singular vector of
, respectively.
For the last part, we know
where
is the product of
k Householder matrices. So, on the basis of (
22), we obtain
□
By theorem 2, we get the mixed forward–backward stability of the HBITLS algorithm and NIPALS-TLS algorithm naturally. The backward stability will generate perturbation that will marginally influence the theoretical convergence of the residual to zero.
Remark 2. The bound we introduced in Theorem 2 shows that the total roundoff errors are dominated by the approximation errors . From this, we can know that, in many practical problems, we can safely select the stopping criteria required by the algorithm based on the theoretical nature of the original problem. This shows that, in a great deal of practical studies, the stopping criteria may be effectively selected based on the theoretical properties of the problem itself, thereby reducing the cost required to pursue an extremely accurate approximate solution to the original problem.