Stochastic Model for the LMS Algorithm with Symmetric/Antisymmetric Properties

Abstract

This paper presents a stochastic model for the least-mean-square algorithm with symmetric/antisymmetric properties (LMS-SAS), operating in a system identification setup with Gaussian input data. Specifically, model expressions are derived to describe the mean weight behavior of the (global and virtual) adaptive filters, learning curves, and evolution of some correlation-like matrices, which allow predicting the algorithm behavior. Simulation results are shown and discussed, confirming the accuracy of the proposed model for both transient and steady-state phases.

1. Introduction

In several practical applications, adaptive filtering techniques are used to obtain an approximate representation for the input–output relationship of an unknown system in real-time, such as occurs in system identification problems, echo and noise cancellation, as well as in adaptive control [1,2,3,4,5]. In some of these applications, the system to be identified exhibits known special characteristics (e.g., sparseness, symmetry or antisymmetry, among others), which can be exploited by the adaptive algorithm to improve its convergence and/or reduce computational complexity. For instance, proportionate-type algorithms [6,7,8,9,10,11] exploit the sparseness of the system impulse response. In turn, algorithms operating with bilinear forms [12,13] consider a more efficient representation for characterizing multiple-input/single-output (MISO) spatiotemporal systems. While the least-mean-square with symmetric/antisymmetric properties (LMS-SAS) algorithm and its normalized version [14] exploit the intrinsic symmetric or antisymmetric characteristics of some systems to decompose their impulse responses into two smaller vectors using the Kronecker product. Therefore, considering the practical applicability, theoretical studies on the behavior of these algorithms become relevant.

In this context, stochastic models may serve as a theoretical basis for studying the behavior of adaptive algorithms without relying only on extensive Monte Carlo (MC) simulations [15,16,17,18,19,20,21,22,23]. Such models are useful to establish (through mathematical expressions) stability conditions for the algorithm and a suitable range of values for its parameters, as well as interesting cause-and-effect relationships between performance metrics and algorithm parameters, which can then help the designer [24,25,26,27,28,29]. Moreover, models can be used to identify undesirable behavior of the algorithm, leading thereby to the development of enhanced algorithms [30,31]. However, despite the inherent importance, a stochastic model for the LMS-SAS algorithm [14] has not been discussed (to the best of our knowledge) in the literature so far. So, aiming to fill this gap, the present research work has the following goals:

(i)

To develop a stochastic model describing the behavior of the algorithm, considering (uncorrelated and correlated) Gaussian input data;

(ii)

To derive expressions characterizing the mean weight behavior of the adaptive filters, learning curves, and evolution of some correlation-like matrices; and

(iii)

To verify and discuss the accuracy of the model for different operating scenarios.

Note that performance comparisons with other algorithms from the literature are beyond the scope of this research work (for such, see [14]).

The remainder of the paper is organized as follows. Section 2 introduces the system model and revisits the LMS-SAS algorithm. Section 3 presents the proposed stochastic model. Section 4 provides simulation results for different operating scenarios. Section 5 presents a brief discussion about the results and highlights some algorithm characteristics. Finally, Section 6 summarizes conclusions and suggestions for future research works.

The mathematical notation adopted in this paper follows the standard practice of using lower-case boldface letters for vectors, upper-case boldface letters for matrices, and both italic Roman and Greek letters for scalar quantities. Superscripts $\mathrm{T}$ and $\mathcal{B}$ stand for the transpose and block-transpose operators [32], ⊗ denotes the Kronecker product, $\mathrm{vec}(·)$ characterizes the vectorization operator [33], $\mathrm{Tr}(·)$ represents the trace of a matrix, and $\mathrm{E}(·)$ is the expected value operator. Still, the notation ${\left[\mathbf{A}\right]}_{(i,j)}$ represents the scalar $(i,j)$-entry of matrix $\mathbf{A}$, while ${\mathbf{A}}_{i,j}$ denotes the $i,j$-block/submatrix of $\mathbf{A}$.

2. Problem Formulation

In a system identification setup (as depicted in Figure 1), let us consider a linear single-input/single-output (SISO) system whose output signal $y\left(n\right)$ corrupted by an additive measurement noise $w\left(n\right)$ at time index n yields the so-called desired signal which can be expressed as

$$\begin{array}{cc}\hfill d\left(n\right)& =y\left(n\right)+w\left(n\right)\hfill \\ \hfill & ={\mathbf{h}}^{\mathrm{T}}({\mathbf{h}}_1,{\mathbf{h}}_2)\mathbf{x}\left(n\right)+w\left(n\right)\hfill \end{array}$$

(1)

where $\mathbf{x}\left(n\right)={[x\left(n\right)x(n-1)\dots x(n-L_0^2+1)]}^{\mathrm{T}}$ contains the $L_0^2$ most recent samples of the input signal $x\left(n\right)$, while

$$\begin{array}{cc}\hfill \mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)& ={\mathbf{h}}_1\otimes {\mathbf{h}}_2\pm {\mathbf{h}}_2\otimes {\mathbf{h}}_1\hfill \end{array}$$

(2)

denotes an $L_0^2$-dimensional (unknown) symmetric (+) or antisymmetric (−) system impulse response which depends on two (virtual) impulse responses ${\mathbf{h}}_1$ and ${\mathbf{h}}_2$ of length $L_0$. Note that the symmetric or antisymmetric characteristic considered here is not related to the samples of the impulse response such as in linear-phase finite impulse response filters (for details, see the discussion presented in [14]). So, the error signal can be written as

$$e\left(n\right)=d\left(n\right)-\widehat{d}(n)$$

(3)

with $\widehat{d}(n)$ denoting the output signal of the adaptive filter $\hat{\mathbf{h}}\left(n\right)$ whose weights are adjusted based on $x\left(n\right)$ and $e\left(n\right)$.

In this context, instead of estimating $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ directly, the LMS-SAS algorithm [14] can be used to identify ${\mathbf{h}}_1$ and ${\mathbf{h}}_2$ [in (2)] through two (virtual) adaptive filters with weight vectors ${\widehat{\mathbf{h}}}_1\left(n\right)$ and ${\widehat{\mathbf{h}}}_2\left(n\right)$, both of length $L_0$, in such a way that the (global) adaptive filter can be computed as

$$\begin{array}{cc}\hfill \widehat{\mathbf{h}}\left(n\right)& ={\widehat{\mathbf{h}}}_1\left(n\right)\otimes {\widehat{\mathbf{h}}}_2\left(n\right)\pm {\widehat{\mathbf{h}}}_2\left(n\right)\otimes {\widehat{\mathbf{h}}}_1\left(n\right).\hfill \end{array}$$

(4)

Thus, the adjustment of ${\widehat{\mathbf{h}}}_1\left(n\right)$ and ${\widehat{\mathbf{h}}}_2\left(n\right)$ in (4) is carried out using the following update rules:

$${\widehat{\mathbf{h}}}_1\left(n\right)={\widehat{\mathbf{h}}}_1(n-1)+{\mu }_1{\mathbf{x}}_2\left(n\right)e\left(n\right)$$

(5)

and

$${\widehat{\mathbf{h}}}_2\left(n\right)={\widehat{\mathbf{h}}}_2(n-1)+{\mu }_2{\mathbf{x}}_1\left(n\right)e\left(n\right)$$

(6)

where ${\mu }_1$ and ${\mu }_2$ represent step-size parameters,

$${\mathbf{x}}_1\left(n\right)={\widehat{\mathbf{H}}}_1^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right)$$

(7)

and

$${\mathbf{x}}_2\left(n\right)={\widehat{\mathbf{H}}}_2^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right)$$

(8)

with

$${\widehat{\mathbf{H}}}_1\left(n\right)={\widehat{\mathbf{h}}}_1\left(n\right)\otimes {\mathbf{I}}_{L_0}\pm {\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_1\left(n\right)$$

(9)

and

$${\widehat{\mathbf{H}}}_2\left(n\right)={\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_2\left(n\right)\pm {\widehat{\mathbf{h}}}_2\left(n\right)\otimes {\mathbf{I}}_{L_0}$$

(10)

characterizing matrices of size $L_0^2\times L_0$, in which ${\mathbf{I}}_{L_0}$ denotes the identity matrix with dimension $L_0\times L_0$, while the error signal (3) can conveniently be expressed as

$$\begin{array}{cc}\hfill e\left(n\right)& =d\left(n\right)-{\widehat{\mathbf{h}}}_1^{\mathrm{T}}(n-1){\mathbf{x}}_2\left(n\right)\hfill \\ \hfill & =d\left(n\right)-{\widehat{\mathbf{h}}}_2^{\mathrm{T}}(n-1){\mathbf{x}}_1\left(n\right).\hfill \end{array}$$

(11)

Therefore, the system identification setup and the LMS-SAS algorithm [defined through (4)–(11)] have now been characterized.

3. Proposed Model

In this section, the proposed stochastic model describing the behavior of the LMS-SAS algorithm is derived. Specifically, model expressions are obtained for predicting the mean weight behavior of the (global and virtual) adaptive filters, learning curves, and the evolution of some correlation-like matrices related to the adaptive weight vectors. To this end, the following assumptions and approximations are used:

Assumption 1.

The input signal $x\left(n\right)$ is obtained from a zero-mean (uncorrelated or correlated) Gaussian process with variance ${\sigma }_x^2$ and autocorrelation matrix $\mathbf{R}=\mathrm{E}\left[\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\right]$ [2,5].

Assumption 2.

The measurement noise $w\left(n\right)$ is obtained from a white Gaussian process with variance ${\sigma }_w^2$, which is uncorrelated with any other signal in the system [2,5].

Assumption 3.

The adaptive weight vectors ${\widehat{\mathbf{h}}}_1\left(n\right)$, ${\widehat{\mathbf{h}}}_2\left(n\right)$, and $\widehat{\mathbf{h}}\left(n\right)$ are assumed independent of each other as well as from any other variable in the system [2,5,13,29].

Note that these assumptions have been commonly used in the stochastic modeling of adaptive algorithms to make the development mathematically tractable (see [2,5]) and have led to satisfactory results (as shown later in Section 4).

3.1. Mean Weight Behavior

Taking into account the similar forms of (5) and (6), let us start replacing the subindexes 1 and 2 by $\alpha$ and $\overline{\alpha }$ (or vice-versa) such that both update rules can be expressed through

$${\widehat{\mathbf{h}}}_{\alpha }(n)={\widehat{\mathbf{h}}}_{\alpha }(n-1)+{\mu }_{\alpha }{\mathbf{x}}_{\overline{\alpha }}(n)e(n).$$

(12)

Then, substituting (1) and (11) into (12), taking the expected value of both sides of the resulting expression, and using Assumptions 2 and 3, the mean weight behavior of the (virtual) adaptive filters can be determined as

$$\mathrm{E}[{\widehat{\mathbf{h}}}_{\alpha }(n)]=[{\mathbf{I}}_{L_0}-{\mu }_{\alpha }{\mathbf{S}}_{\overline{\alpha }}(n)]\mathrm{E}[{\widehat{\mathbf{h}}}_{\alpha }(n-1)]+{\mu }_{\alpha }{\mathbf{S}}_{\overline{\alpha }}^{\prime }(n)\mathbf{h}$$

(13)

where

$$\begin{array}{cc}\hfill {\mathbf{S}}_{\alpha }\left(n\right)& =\mathrm{E}\left[{\mathbf{x}}_{\alpha }\left(n\right){\mathbf{x}}_{\alpha }^{\mathrm{T}}\left(n\right)\right]\hfill \end{array}$$

(14)

and

$${\mathbf{S}}_{\alpha }^{\prime }\left(n\right)=\mathrm{E}\left[{\mathbf{x}}_{\alpha }\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\right].$$

(15)

Notice from (13) that the convergence of the (virtual) adaptive filters depends on

• The step sizes ${\mu }_{\alpha }$, the system impulse response $\mathbf{h}$, the estimate of the system impulse response ${\widehat{\mathbf{h}}}_{\alpha }(n)$, as well as the correlation-like matrices ${\mathbf{S}}_{\alpha }\left(n\right)$ and ${\mathbf{S}}_{\alpha }^{\prime }\left(n\right)$;
• The initialization condition ${\widehat{\mathbf{h}}}_{\alpha }(0)$ which may impair the initial convergence or cause instability of the algorithm; and
• Each other, i.e., the behavior of one (virtual) adaptive filter is affected by the behavior of the other one and vice-versa.

Next, substituting either (7) or (8) into (14), using the properties of the Kronecker product [33,34], as well as considering Assumption 3, we have

$${\mathbf{S}}_{\alpha }\left(n\right)={\mathbf{S}}_{\alpha }^{\left(1\right)}\left(n\right)\pm {\mathbf{S}}_{\alpha }^{\left(2\right)}\left(n\right)\pm {{\mathbf{S}}_{\alpha }^{\left(2\right)}}^{\mathrm{T}}\left(n\right)+{\mathbf{S}}_{\alpha }^{\left(3\right)}\left(n\right)$$

(16)

with

$$\begin{array}{cc}\hfill {\mathbf{S}}_{\alpha }^{\left(1\right)}\left(n\right)& =\mathrm{E}\{{[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]}^{\mathrm{T}}\mathbf{R}[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]\}\hfill \\ \hfill & =\left[\begin{array}{ccc}\mathrm{Tr}\left[{\mathbf{R}}_{1,1}{\mathbf{G}}_{\alpha }(n-1)\right]& \cdots & \mathrm{Tr}\left[{\mathbf{R}}_{1,L_0}{\mathbf{G}}_{\alpha }(n-1)\right]\\ ⋮& \ddots & ⋮\\ \mathrm{Tr}\left[{\mathbf{R}}_{L_0,1}{\mathbf{G}}_{\alpha }(n-1)\right]& \cdots & \mathrm{Tr}\left[{\mathbf{R}}_{L_0,L_0}{\mathbf{G}}_{\alpha }(n-1)\right]\end{array}\right]\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mathbf{S}}_{\alpha }^{\left(2\right)}\left(n\right)& =\mathrm{E}\{{[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]}^{\mathrm{T}}\mathbf{R}[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]\}\hfill \\ \hfill & =\left[\begin{array}{c}{\displaystyle \sum _{i=1}^{L_0}{\mathbf{g}}_{\alpha ,i}^{\mathrm{T}}(n-1){\mathbf{R}}_{1,i}}\\ ⋮\\ {\displaystyle \sum _{i=1}^{L_0}{\mathbf{g}}_{\alpha ,i}^{\mathrm{T}}(n-1){\mathbf{R}}_{L_0,i}}\end{array}\right]\hfill \end{array}$$

(18)

and

$$\begin{array}{cc}\hfill {\mathbf{S}}_{\alpha }^{\left(3\right)}\left(n\right)& =\mathrm{E}\{{[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]}^{\mathrm{T}}\mathbf{R}[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]\}\hfill \\ \hfill & =\sum _{i=1}^{L_0}\sum _{j=1}^{L_0}{\left[{\mathbf{G}}_{\alpha }(n-1)\right]}_{(i,j)}{\mathbf{R}}_{i,j}\hfill \end{array}$$

(19)

in which

$${\mathbf{G}}_{\alpha }\left(n\right)=\mathrm{E}[{{\widehat{\mathbf{h}}}_{\alpha }(n)}^{\mathrm{T}}{\widehat{\mathbf{h}}}_{\alpha }(n)]$$

(20)

is the autocorrelation matrix of ${\widehat{\mathbf{h}}}_{\alpha }(n)$, ${\mathbf{g}}_{\alpha ,i}\left(n\right)$ represents the i-th column of ${\mathbf{G}}_{\alpha }\left(n\right)$, ${\left[{\mathbf{G}}_{\alpha }\left(n\right)\right]}_{(i,j)}$ characterizes the $(i,j)$-th element of ${\mathbf{G}}_{\alpha }\left(n\right)$, while ${\mathbf{R}}_{i,j}$ denotes an $L_0\times L_0$-block/submatrix of $\mathbf{R}$, i.e.,

$$\mathbf{R}=\left[\begin{array}{ccc}{\mathbf{R}}_{1,1}& \cdots & {\mathbf{R}}_{1,L_0}\\ ⋮& \ddots & ⋮\\ {\mathbf{R}}_{L_0,1}& \cdots & {\mathbf{R}}_{L_0,L_0}\end{array}\right].$$

(21)

Similarly, from (7) and (8), (15) can be simplified as

$${\mathbf{S}}_{\alpha }^{\prime }\left(n\right)=\mathrm{E}[{\widehat{\mathbf{H}}}_{\alpha }^{\mathrm{T}}(n-1)]\mathbf{R}$$

(22)

with $\mathrm{E}[{\widehat{\mathbf{H}}}_{\alpha }^{\mathrm{T}}(n-1)]$ being obtained by taking the expected value of both sides of either (9) or (10).

Finally, the mean weight behavior of the (global) adaptive filter $\widehat{\mathbf{h}}(n)$ can be determined by applying the expected value to both sides of (4) and using Assumption 3. Thereby, one obtains

$$\mathrm{E}[\widehat{\mathbf{h}}(n)]=\mathrm{E}[{\widehat{\mathbf{h}}}_1(n)]\otimes \mathrm{E}[{\widehat{\mathbf{h}}}_2(n)]\pm \mathrm{E}[{\widehat{\mathbf{h}}}_2(n)]\otimes \mathrm{E}[{\widehat{\mathbf{h}}}_1(n)]$$

(23)

which depends on (13); hence, the characteristics observed in the convergence of the (virtual) adaptive filters hold also for the convergence of the (global) adaptive filter. Therefore, the mean weight behavior of ${\widehat{\mathbf{h}}}_1(n)$, ${\widehat{\mathbf{h}}}_2(n)$, and $\widehat{\mathbf{h}}(n)$ can be predicted [from (13) and (23)] if the evolution of (20) is known.

3.2. Learning Curves

Aiming to characterize the learning curves, let us start by rewriting (11) using the weight-error vector

$$\mathbf{v}(n)=\mathbf{h}-\widehat{\mathbf{h}}(n)$$

(24)

as

$$e\left(n\right)={\mathbf{v}}^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right)+w\left(n\right).$$

(25)

Then, squaring both sides of (25), taking the expected value of the resulting expression, and using Assumption 2, one obtains the following expression describing the evolution of the mean-square error (MSE):

$$J\left(n\right)=J_{\min }+J_{\mathrm{ex}}\left(n\right)$$

(26)

where

$$J_{\min }={\sigma }_w^2$$

(27)

is the minimum MSE attainable in steady state, while

$$J_{\mathrm{ex}}\left(n\right)=\mathrm{E}\left[{\mathbf{v}}^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\mathbf{v}(n-1)\right]$$

(28)

is the excess MSE (EMSE) introduced by the algorithm. Next, considering the properties of the trace operator [33] and using Assumption 3, (28) reduces to

$$J_{\mathrm{ex}}\left(n\right)=\mathrm{Tr}\left[\mathbf{R}\mathbf{K}(n-1)\right]$$

(29)

with

$$\mathbf{K}\left(n\right)=\mathrm{E}\left[\mathbf{v}\left(n\right){\mathbf{v}}^{\mathrm{T}}\left(n\right)\right]$$

(30)

being the autocorrelation matrix of the weight-error vector. Therefore, if the evolution of (30) is known, the MSE and EMSE learning curves are completely characterized by (26), (27), and (29).

3.3. Correlation-Like Matrices

The aim is now to derive recursive expressions describing the evolution of the correlation-like matrices ${\mathbf{G}}_{\alpha }\left(n\right)$ [given by (20)] and $\mathbf{K}\left(n\right)$ [given by (30)]. To this end, substituting (12) into (20), and using Assumptions 2 and 3, we obtain

$$\begin{array}{cc}\hfill {\mathbf{G}}_{\alpha }\left(n\right)& ={\mathbf{G}}_{\alpha }(n-1)-{\mu }_{\alpha }{\mathbf{S}}_{\overline{\alpha }}\left(n\right){\mathbf{G}}_{\alpha }(n-1)\hfill \\ \hfill & -{\mu }_{\alpha }{\mathbf{G}}_{\alpha }(n-1){\mathbf{S}}_{\overline{\alpha }}\left(n\right)+{\mu }_{\alpha }{\mathbf{G}}_{\alpha }^{\prime }(n-1){{\mathbf{S}}_{\overline{\alpha }}^{\prime }}^{\mathrm{T}}\left(n\right)\hfill \\ \hfill & +{\mu }_{\alpha }{\mathbf{S}}_{\overline{\alpha }}^{\prime }\left(n\right){{\mathbf{G}}_{\alpha }^{\prime }}^{\mathrm{T}}(n-1)+{\mu }_{\alpha }^2{\mathbf{F}}_{\overline{\alpha }}\left(n\right)+{\mu }_{\alpha }^2{\sigma }_w^2{\mathbf{S}}_{\overline{\alpha }}\left(n\right)\hfill \end{array}$$

(31)

where

$${\mathbf{G}}_{\alpha }^{\prime }\left(n\right)=\mathrm{E}[{\widehat{\mathbf{h}}}_{\alpha }(n)]{\mathbf{h}}^{\mathrm{T}}$$

(32)

with $\mathrm{E}[{\widehat{\mathbf{h}}}_{\alpha }(n)]$ being computed from (13),

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\alpha }\left(n\right)& =\mathrm{E}[{\widehat{\mathbf{H}}}_{\alpha }^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\mathbf{v}(n-1){\mathbf{v}}^{\mathrm{T}}(n-1)\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right){\widehat{\mathbf{H}}}_{\alpha }(n-1)]\hfill \\ \hfill & \cong \mathrm{E}[{\widehat{\mathbf{H}}}_{\alpha }^{\mathrm{T}}(n-1)\mathbf{E}\left(n\right){\widehat{\mathbf{H}}}_{\alpha }(n-1)]\hfill \end{array}$$

(33)

in which

$$\begin{array}{cc}\hfill \mathbf{E}\left(n\right)& =\mathrm{E}\left[\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\mathbf{K}(n-1)\mathbf{x}\left(n\right){\mathbf{x}}^{\mathrm{T}}\left(n\right)\right]\hfill \\ \hfill & =2\mathbf{R}\mathbf{K}(n-1)\mathbf{R}+\mathbf{R}\mathrm{Tr}\left[\mathbf{R}\mathbf{K}\right(n-1\left)\right]\hfill \end{array}$$

(34)

due to the factorization theorem of Gaussian variables [2,3,5] (also known as Isserlis’ theorem [35]). Thereby, such as in (16), (33) can be expressed [using (34)] as

$${\mathbf{F}}_{\alpha }\left(n\right)={\mathbf{F}}_{\alpha }^{\left(1\right)}\left(n\right)\pm {\mathbf{F}}_{\alpha }^{\left(2\right)}\left(n\right)\pm {{\mathbf{F}}_{\alpha }^{\left(2\right)}}^{\mathrm{T}}\left(n\right)+{\mathbf{F}}_{\alpha }^{\left(3\right)}\left(n\right)$$

(35)

where

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\alpha }^{\left(1\right)}\left(n\right)& =\mathrm{E}\{{[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]}^{\mathrm{T}}\mathbf{E}\left(n\right)[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]\}\hfill \\ \hfill & =\left[\begin{array}{ccc}\mathrm{Tr}\left[{\mathbf{E}}_{1,1}\left(n\right){\mathbf{G}}_{\alpha }(n-1)\right]& \cdots & \mathrm{Tr}\left[{\mathbf{E}}_{1,L_0}\left(n\right){\mathbf{G}}_{\alpha }(n-1)\right]\\ ⋮& \ddots & ⋮\\ \mathrm{Tr}\left[{\mathbf{E}}_{L_0,1}\left(n\right){\mathbf{G}}_{\alpha }(n-1)\right]& \cdots & \mathrm{Tr}\left[{\mathbf{E}}_{L_0,L_0}\left(n\right){\mathbf{G}}_{\alpha }(n-1)\right]\end{array}\right]\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\alpha }^{(2)}\left(n\right)& =\mathrm{E}\{{[{\mathbf{I}}_{L_0}\otimes {\widehat{\mathbf{h}}}_{\alpha }(n-1)]}^{\mathrm{T}}\mathbf{E}(n)[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]\}\hfill \\ \hfill & =\left[\begin{array}{c}{\displaystyle \sum _{i=1}^{L_0}{\mathbf{g}}_{\alpha ,i}^{\mathrm{T}}(n-1){\mathbf{E}}_{1,i}\left(n\right)}\\ ⋮\\ {\displaystyle \sum _{i=1}^{L_0}{\mathbf{g}}_{\alpha ,i}^{\mathrm{T}}(n-1){\mathbf{E}}_{L_0,i}\left(n\right)}\end{array}\right]\hfill \end{array}$$

(37)

and

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\alpha }^{\left(3\right)}\left(n\right)& =\mathrm{E}{\{[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]\}}^{\mathrm{T}}\mathbf{E}\left(n\right)[{\widehat{\mathbf{h}}}_{\alpha }(n-1)\otimes {\mathbf{I}}_{L_0}]\}\hfill \\ \hfill & =\sum _{i=1}^{L_0}\sum _{j=1}^{L_0}{\left[{\mathbf{G}}_{\alpha }(n-1)\right]}_{(i,j)}{\mathbf{E}}_{i,j}\left(n\right).\hfill \end{array}$$

(38)

In turn, calculating $\mathbf{v}\left(n\right){\mathbf{v}}^{\mathrm{T}}\left(n\right)$ from (24) and taking the expected value of both sides of the resulting expression, it is possible to rewrite (30) as

$$\mathbf{K}\left(n\right)=\mathbf{h}{\mathbf{h}}^{\mathrm{T}}-\mathbf{h}\mathrm{E}[{\widehat{\mathbf{h}}}^{\mathrm{T}}(n)]-\mathrm{E}[\widehat{\mathbf{h}}(n)]{\mathbf{h}}^{\mathrm{T}}+{\mathbf{G}}_{}\left(n\right)$$

(39)

where

$$\begin{array}{cc}\hfill {\mathbf{G}}_{}\left(n\right)& =\mathrm{E}[\widehat{\mathbf{h}}(n){\widehat{\mathbf{h}}}^{\mathrm{T}}(n)]\hfill \\ \hfill & ={\mathbf{G}}_1\left(n\right)\otimes {\mathbf{G}}_2\left(n\right)\pm {\{\mathrm{vec}[{\mathbf{G}}_2\left(n\right)]\mathrm{vec}{[{\mathbf{G}}_1\left(n\right)]}^{\mathrm{T}}\}}^{\mathcal{B}}\hfill \\ \hfill & \pm {\{\mathrm{vec}\left[{\mathbf{G}}_1\left(n\right)\right]\mathrm{vec}{[{\mathbf{G}}_2\left(n\right)]}^{\mathrm{T}}\}}^{\mathcal{B}}+{\mathbf{G}}_2\left(n\right)\otimes {\mathbf{G}}_1\left(n\right)\hfill \end{array}$$

(40)

denotes the autocorrelation matrix of $\widehat{\mathbf{h}}(n)$ and ${(·)}^{\mathcal{B}}$ represents the transposition of blocks [32] of dimension $L_0\times L_0$. Note that (40) is obtained by calculating the outer product $\widehat{\mathbf{h}}(n){\widehat{\mathbf{h}}}^{\mathrm{T}}(n)$ from (4), taking the expected value of both sides of the resulting expression, and considering Assumption 3.

Therefore, since the evolution of the correlation-like matrices ${\mathbf{G}}_{\alpha }(n)$, $\mathbf{K}\left(n\right)$, ${\mathbf{G}}_{}\left(n\right)$ has been properly characterized, the behavior of the LMS-SAS algorithm can now be predicted.

4. Simulation Results

This section aims to assess the accuracy of the proposed model by comparing results obtained from MC simulations (average of 200 independent runs) with model predictions. To this end, three examples are presented, covering uncorrelated and correlated Gaussian input data, several signal-to-noise ratio (SNR) values, as well as distinct system impulse responses and initialization conditions for the adaptive filters. Specifically, the input signal $x\left(n\right)$ is obtained from an autoregressive (AR) process [5], given by

$$x\left(n\right)=-a_{1}x(n-1)-a_{2}x(n-2)+v\left(n\right)$$

(41)

in which $a_1$ and $a_2$ denote the AR(2) coefficients, while $v\left(n\right)$ is a white Gaussian noise whose variance is determined as

$${\sigma }_v^2={\sigma }_x^2\left({\displaystyle \frac{1-a_2}{1+a_2}}\right)[{(1+a_2)}^2-a_1^2]$$

(42)

such that ${\sigma }_x^2=1$. The SNR is defined (in dB) as [14,29]

$$\mathrm{SNR}=10{\log }_{10}\left(\frac{{\sigma }_y^2}{{\sigma }_w^2}\right)$$

(43)

with ${\sigma }_y^2={\mathbf{h}}^{\mathrm{T}}({\mathbf{h}}_1,{\mathbf{h}}_2)\mathbf{R}\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ characterizing the variance of the system output signal; in particular, three values of SNR are considered here, i.e., 10, 20, and 30 dB. The (virtual) adaptive filters 1 and 2 are initialized, respectively, as ${\widehat{\mathbf{h}}}_1(0)={[10\dots 0]}^{\mathrm{T}}$ and ${\widehat{\mathbf{h}}}_2(0)=L_0^{-1}{[11\dots 1]}^{\mathrm{T}}$ unless otherwise stated to prevent them from stalling at the beginning of the adaptation process [14], while the step sizes are chosen as ${\mu }_1={\mu }_2={10}^{-3}$.

4.1. Example 1

Here, the proposed model is verified for uncorrelated Gaussian input data, different SNR values, and a system impulse response with antisymmetric characteristic. Specifically, the input signal is obtained from (41) by making $a_1=a_2=0$, which results in an eigenvalue spread [2,3,5] for the input autocorrelation matrix of $\chi =1$. The antisymmetric system impulse response $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ is obtained from (2), with ${\mathbf{h}}_1$ containing the first $L_0=16$ weights of the echo path model 1 given in the ITU-T G.168 Recommendation [36] while ${\left[{\mathbf{h}}_2\right]}_l={(-0.9)}^{l-1}$ for $l=1,2,\dots ,L_0$; consequently, the length of $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ is $L_0^2=256$. The results obtained for this operating scenario are presented in Figure 2.

4.2. Example 2

This example aims now to assess the proposed model for correlated Gaussian input data, different SNR values, and a system impulse response with symmetric characteristic. Particularly, the input signal is taken from (41) with $a_1=-0.6$ and $a_2=0.8$, which results in an eigenvalue spread of $\chi =162.13$ for the input autocorrelation matrix. The symmetric system impulse response $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ is obtained from (2), with ${\mathbf{h}}_1$ containing the first $L_0=32$ weights of the echo path model 1 given in the ITU-T G.168 Recommendation [36] while ${\left[{\mathbf{h}}_2\right]}_l=0.5^{l-1}$ for $l=1,2,\dots ,L_0$; consequently, the length of $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ is $L_0^2=1024$. The results obtained for this operating scenario are presented in Figure 3.

4.3. Example 3

In this example, the accuracy of the proposed model is verified considering different initialization conditions for the adaptive weight vectors. To this end, the input signal is obtained from (41) by making $a_1=-0.6$ and $a_2=0.8$, yielding an eigenvalue spread of $\chi =160.55$ for the input autocorrelation matrix. Symmetric and antisymmetric system impulse responses are used, which are determined from (2) with ${\mathbf{h}}_1$ and ${\mathbf{h}}_2$ given as in Example 2 but now with $L_0=16$ weights; hence, the length of $\mathbf{h}({\mathbf{h}}_1,{\mathbf{h}}_2)$ becomes $L_0^2=256$. In turn, the adaptive weight vectors are initialized as ${\widehat{\mathbf{h}}}_1(0)=L_0^{-1}{[11\dots 1]}^{\mathrm{T}}$ and ${\widehat{\mathbf{h}}}_2(0)=c_{+}{L}_0^{-1}{[11\dots 1]}^{\mathrm{T}}$ with $c_{+}=\{{10}^{-1},1-{10}^{-6},1\}$ when the system impulse response is symmetric, while ${\widehat{\mathbf{h}}}_1(0)={[10\dots 0]}^{\mathrm{T}}$ and ${\widehat{\mathbf{h}}}_2(0)=c_{-}{L}_0^{-1}{[11\dots 1]}^{\mathrm{T}}$ with $c_{-}=\{1,5,7.5\}$ when the antisymmetric impulse response is used. The obtained results are presented in Figure 4, considering only SNR = 20 dB for simplicity.

5. Discussion

Figure 2, Figure 3 and Figure 4 present the results obtained from MC simulations and model predictions for the operating scenarios described in Examples 1, 2, and 3, respectively. Specifically, Figure 2a–c and Figure 3a–c depict the evolution of (five) adaptive weights of ${\widehat{\mathbf{h}}}_1(n)$, ${\widehat{\mathbf{h}}}_2(n)$, and $\widehat{\mathbf{h}}(n)$, while Figure 2d and Figure 3d show EMSE learning curves. [Results obtained for SNR values of 10 and 30 dB have been omitted in Figure 2a–c and Figure 3a–c since they are very similar]. In turn, Figure 4a,b illustrate the impact of the initialization of the adaptive weight vectors on the evolution of (two) adaptive weights of $\widehat{\mathbf{h}}(n)$, while Figure 4c,d depict the effect on the EMSE learning curves, from which one verifies that the convergence speed and the steady-state EMSE are affected by the choice of the initialization vectors of the adaptive filters; in this particular case, when ${\widehat{\mathbf{h}}}_1(0)$ and ${\widehat{\mathbf{h}}}_2(0)$ assume small and distinct values, improved convergence and lower steady-state EMSE are achieved. Despite that, one observes from these figures that the behavior predicted from the proposed model matches very well the one obtained from MC simulations, during both transient and steady-state phases. Therefore, the accuracy of the model is confirmed for uncorrelated and correlated Gaussian input data, different SNR values and initialization conditions for the adaptive weight vectors, as well as for system impulse responses with symmetric and antisymmetric characteristics.

6. Conclusions

A stochastic model for the LMS-SAS algorithm was derived in this paper. The proposed model allows predicting the mean weight behavior of the (global and virtual) adaptive filters, learning curves, and evolution of some correlation-like matrices. Simulation results confirmed the accuracy of the model irrespective of the input data correlation level, SNR value, initialization condition of the adaptive weight vectors, and system impulse response. Based on the proposed model, further research could address the derivation of stability bounds for the step size, analytical expressions characterizing the algorithm behavior in steady state, and the development of models for the normalized version of the LMS-SAS algorithm.