A Robust Hammerstein-Wiener Model Identification Method for Highly Nonlinear Systems

Abstract

The existing results show the applicability of the Over-Parameterized Model based Hammerstein-Wiener model identification methods. However, it requires to estimate extra parameters and performer a low rank approximation step. Therefore, it may give rise to unnecessarily high variance in parameter estimates for highly nonlinear systems, especially using a small and noisy data set. To overcome this corruptive phenomenon. To overcome this corruptive phenomenon, in this paper, a robust Hammerstein-Wiener model identification method is developed for highly nonlinear systems when using a small and noisy data set, where two parsimonious parametrization models with fewer parameters are used, and an iteration method is then used to retrieve the true system parameters from the parametrization models. Such modification can improve the parameter estimation performance in terms of accuracy and variance compared with the over-parametrization model based identification methods. All the above-mentioned developments are analyzed with variance analysis, along with a simulation example to confirm the effectiveness.

1. Introduction

The Hammerstein–Wiener (H-W) models, also called N-L-N models, are known as a particular class of block-oriented nonlinear models where a Linear (L) subsystem is embedded in two static Nonlinear subsystems (N). Due to the use of two nonlinear elements instead of one, H-W models offer convenient higher modeling capabilities for nonlinear systems, especially for nonlinear systems with both actuator and sensor nonlinearities [1]. H-W models identification has attracted considerable attention in the past years and a large amount of works have been explored in different frameworks, see, for example, iterative methods [2,3], and Least-Squares (LS) based Two-Step Algorithms (TSA) [4].

In this paper, the emphasis is put on the LS issue for H-W models with over-parametrization methods, which was perhaps first studied by Bai [4] where first characterizes the original H-W model by an over-parametrization model and then estimates the parameters of this over-parametrization model (OPM) by using a LS method. In consequence, the original system parameters are extracted from the estimated over parametrization model parameters by performing low-rank approximation using Singular Value Decompositions (SVDs). The TSA is attractive because of its numerical simplicity. Thus, following from this pioneering work [4], a number of TSA-based methods have been proposed to identify H-W models under different conditions: For instance, in [5], a auxiliary model multi-innovation stochastic gradient (AM-MISG) based TSA was proposed for H-W models online identification, which has fast convergence rate with robust characteristics and its convergence was proved by using the stochastic process theory. In [6], an instrumental variable based TSA was proposed for H-W continuous-time models in the presence of colored noise. In [7], a modified Bias-Eliminating Least-Squares (BELS) based TSA was proposed for H-W errors-in-variables models.The TSA has also been extensively used for the identification of other types of block-oriented nonlinear systems, e.g., Hammerstein models [8,9], and Wiener models [10].

In these contributions, all existing TSAs provide asymptotically consistent unbiased estimations for H-W models when the data length tends to infinity. However, it is well known that the existing TSAs related to the use of over-parametrization model and low-rank approximation step may result in a degradation of the accuracy and variance of the estimated quantities for highly nonlinear systems, especially for short noisy data length sequences. One reason is that the over-parametrization model has much more parameters than the actual Hammerstein-Wiener model, the extra parameters in TSAs may lead to larger estimation errors and variances, and the number of the extra parameters increase with the degree of nonlinearity. The other is that the low-rank approximation step in TSAs inevitably brings some truncation errors, which will result in some loss of model quality.

In this paper, inspired by the parsimonious models based identfication methods [11,12,13,14] for block-oriented nonlinear Hammerstein models can improve the variance properties of the over-parametrization model based methods, a robust consistent identification formulation for H-W models is thus proposed to overcome the above-mentioned drawbacks for highly nonlinear systems when using short noisy data length sequences, without the necessity of estimating extra parameters and performing low-rank approximation.The main contributions of the paper are stated below:

• Two parsimonious models with fewer parameters constructed to characterize the H-W system instead of an over-parametrization model, and a projection based LS iterative method is proposed to estimate all system parameters in a parallel fashion. The proposed method avoids the estimating extra parameters and a low rank approximation step in classical over-parametrization model based methods, which leads to improved results compared to existing over-parametrization model based methods, because that estimating extra parameters and performing low rank approximation may result into a degradation of the accuracy of the estimated quantities.
• The variance analysis is given to demonstrate that the new method generally gives smaller variance compared to conventional TSA method. Note that the results of variance analysis explains why the proposed identfication methods can lead to improved results compared to existing over-parametrization model based methods for H-W systems.

Note that Hammerstein models and Wiener models are two special cases of Hammerstein–Wiener models, that is, when $n_c=1$ and $f_1\left(u\left(t\right)\right)=u\left(t\right)$ in following (1), the Hammerstein–Wiener models become Wiener models, and when $n_d=1$ and $g_1\left(y\left(t\right)\right)=y\left(t\right)$ in following (1), the Hammerstein–Wiener models become Hammerstein models. The presented methodology in this paper is general for N-L-N models, L-N models and N-L models.

The paper is organized as follows. In Section 2, the identification problem is presented. In Section 3, the proposed method is presented, followed by the Variance analysis in Section 4. Section 5 presents a simulation example to validate and evaluate the performance of the new algorithm. Finally, some conclusions are drawn in Section 6.

2. Problem Statement and Analysis

Consider a particular class of nonlinear H-W model with polynomial nonlinearities, see also [4]

$$\begin{array}{c}\hfill y\left(t\right)=\sum _{j=1}^{n_b}{b}_j\left\{\sum _{m=1}^{n_c}{c}_m{f}_m\left[u(t-j)\right]\right\}+\sum _{i=1}^{n_a}{a}_i\left\{\sum _{l=1}^{n_d}{d}_l{g}_l\left[y(t-i)\right]\right\}+e\left(t\right)\end{array}$$

(1)

where system parameters are $\mathit{a}=[a_1,\dots ,a_{n_a}]$, $\mathit{b}=[b_1,\dots ,b_{n_b}]$, $\mathit{c}=[c_1,\dots ,c_{n_c}]$, and $\mathit{d}=[d_1,\dots ,d_{n_d}]$, $\left\{u\right(t\left)\right\}$ and $\left\{y\right(t\left)\right\}$ are the system input and output sequences, respectively, $\left\{e\right(t\left)\right\}$ is the noise sequence, and $f_m(•)$ and $g_l(•)$ are nonlinear functions. The following general assumptions are made in this paper:

A1

The first nonzero elements of $\mathit{a}$ and $\mathit{b}$ are positive and ${\parallel \mathit{a}\parallel }_2=1$ and ${\parallel \mathit{b}\parallel }_2=1$.

A2

The functions ${\left\{f_m(•)\right\}}_{m=1}^{n_c}$ and ${g_l(•)}_{l=1}^{n_d}$ and degrees $n_a$, $n_b$, $n_c$ and $n_d$ are known.

A3

{$e\left(i\right)$} is a zero-mean white noise sequence with finite variance ${\delta }_e^2$ uncorrelated with {$u\left(j\right)$} for all $i,j$, and uncorrelated with {$y\left(j\right)$} for $i\ne j$.

Assumption 1 ensures uniqueness of the N-L-N nonlinear system model, as the introduction of a factor $q(q\ne 0)$ to the pair ($q\mathit{a}$, $\mathit{d}/q$), the input-output relationship will not change. The terms ${\parallel \mathit{a}\parallel }_2$ and ${\parallel \mathit{b}\parallel }_2$ can be set as any other positive number, the proposed method can be directly used without any change.

Assumption 2 are related to a priori knowledge about the structure of the true system. In this context the static nonlinearity of the N-L-N model is approximated using a linear expansion in terms of basis functions. In generally, the basis functions and system orders lie in the fact that tuning them are more directly related to a priori knowledge about the true system. However, since the unknown coefficients of parameter vectors $\mathit{a},\mathit{b},\mathit{c}$ and $\mathit{d}$ can give the degrees of freedom to accurately modeling the nonlinear systems, the basis functions and system orders can be assumed to known simply for having not directly related knowledge about the nonlinear function in practice. More specifically, the system orders can be set high enough to provide enough degrees of freedom to accurately describing the nonlinear N-L-N systems. The basis functions can be chosen as polynomial functions simply, which is realistic in many applications where the nonlinearities characterizing the systems are smooth enough [9]. Note that the identification method developed under those assumptions in which the nonlinear basis functions as well as the system orders in polynomial model supposed to be known can be regarded as a grey-box method compared with the black-box model approach without any assumption about the structure of the N-L-N model.

Assumption 3 requires the noises to be white, which is realistic in many applications when the main objective is on the plant model.

Based on (1), an over-parametrization model can be formulated as

$$\begin{array}{c}\hfill y\left(t\right)=\mathit{\theta }\Phi \left(t\right)+e\left(t\right)\end{array}$$

(2)

where

$$\begin{array}{cc}\hfill \mathit{\theta }& =[{\mathit{\theta }}_1,{\mathit{\theta }}_2]\in {\Re }^{1\times (n_b{n}_c+n_a{n}_d)}\hfill \end{array}$$

(3a)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_1& =[b_1{c}_1,\dots ,b_1{c}_{n_c},\dots ,b_{n_b}{c}_1,\dots ,b_{n_b}{c}_{n_c}]\in {\Re }^{1\times n_b{n}_c}\hfill \end{array}$$

(3b)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_2& =[a_1{d}_1,\dots ,a_1{d}_{n_d},\dots ,a_{n_a}{d}_1,\dots ,a_{n_a}{d}_{n_d}]\in {\Re }^{1\times n_a{n}_d}\hfill \end{array}$$

(3c)

$$\begin{array}{cc}\hfill \Phi \left(t\right)& ={[f\left(t\right),g\left(t\right)]}^{\top }\in {\Re }^{(n_b{n}_c+n_a{n}_d)\times 1}\hfill \end{array}$$

(3d)

$$\begin{array}{cc}\hfill f\left(t\right)& =[f_1\left[u(t-1)\right],\dots ,f_{n_c}\left[u(t-1)\right],\dots ,\hfill \\ \hfill & f_1{[u(t-n_b),\dots ,f_{n_c}\left[u(t-n_b)\right]]}^{\top }\in {\Re }^{n_b{n}_c\times 1}\hfill \end{array}$$

(3e)

$$\begin{array}{cc}\hfill g\left(t\right)& =[g_1\left[y(t-1)\right],\dots ,g_{n_d}\left[y(t-1)\right],\dots ,\hfill \\ \hfill & g_1{[y(t-n_b),\dots ,g_{n_d}\left[y(t-n_a)\right]]}^{\top }\in {\Re }^{n_a{n}_d\times 1}.\hfill \end{array}$$

(3f)

Define

$$\begin{array}{cc}\hfill Y& =\left[y\right(1),y(2),\dots ,y(N\left)\right]\hfill \end{array}$$

(4a)

$$\begin{array}{cc}\hfill E& =\left[e\right(1),e(2),\dots ,e(N\left)\right]\hfill \end{array}$$

(4b)

$$\begin{array}{cc}\hfill F& =\left[f\right(1),f(2),\dots ,f(N\left)\right]\hfill \end{array}$$

(4c)

$$\begin{array}{cc}\hfill G& =\left[g\right(1),g(2),\dots ,g(N\left)\right]\hfill \end{array}$$

(4d)

$$\begin{array}{cc}\hfill \Phi & ={[F^{\top },G^{\top }]}^{\top }.\hfill \end{array}$$

(4e)

It follows from (1)–(4) that

$$\begin{array}{c}\hfill Y={\mathbf{\theta }}_{1}F+{\mathbf{\theta }}_{2}G+E=\mathbf{\theta }\Phi +E.\end{array}$$

(5)

For the estimation of the unknown system parameters in (5), the following criterion is considered in the TSA:

$$\begin{array}{cc}\hfill & \underset{\mathit{a},\mathit{b},\mathit{c},\mathit{d}}{min}\frac{1}{2}{(Y-\mathbf{\theta }\Phi )}^{\top }W(Y-\mathbf{\theta }\Phi )\hfill \end{array}$$

(6a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\mathrm{rank}\left(\mathit{b}\mathit{c}\right)=1,\mathrm{rank}\left(\mathit{a}\mathit{d}\right)=1\hfill \end{array}$$

(6b)

where W is a symmetric weighting matrix, e.g., an identity matrix of appropriate dimension I.

The TSA is a relaxation method by solving the unconstrained LS problem (6a) without considering the rank constraint (6b), where the system parameters are extracted via SVDs, see [4] for more details. Clearly, the number of extra parameters in the TSA method [4] is $n_a{n}_d+n_b{n}_c-n_a-n_d-n_b-n_c$, and two SVDs have to be used to extract the system parameters from $\mathbf{\theta }$. These may deteriorate the accuracy and variance of estimated quantities. Note that the number of the extra parameters in the TSA will increase as the degree of nonlinearity of H-W models increases. The performance of the TSA method in terms of the accuracy and variance of estimated quantities will be deteriorated for identifying highly nonlinear systems from a limited number of noisy data.

This paper aims to seek a method to estimate unknown parameter vectors $\mathit{a},\mathit{b},\mathit{c}$ and $\mathit{d}$ from the known observed input-output data $\left\{u\right(t),y(t\left)\right\}$ collected at instants $t=1,\dots ,N$ without estimating extra parameters and performing SVDs.

3. The Proposed Method

3.1. Parsimonious Parametrization Model Formulation

We decouple $\mathbf{\theta }$ into two parts ${\mathbf{\theta }}_1$ and ${\mathbf{\theta }}_2$. The over-parametrization model in (2) can be then represented as following two parsimonious parametrization models (PPMs):

$$\begin{array}{c}\hfill y\left(t\right)=\mathit{a}{\Phi }_1\left(t\right)+\mathit{b}{\Phi }_2\left(k\right)+e\left(t\right)=\mathit{d}{\Phi }_3\left(t\right)+\mathit{c}{\Phi }_4\left(k\right)+e\left(t\right)\end{array}$$

(7)

where

$$\begin{array}{cc}\hfill {\Phi }_1\left(t\right)& =(d_1{g}_1\left[y(t-1)\right]+\cdots +d_{n_d}{g}_{n_d}\left[y(t-1)\right],\dots ,\hfill \\ \hfill & d_1{g}_1{[y(t-n_a)+\cdots +d_{n_d}{g}_{n_d}\left[y(t-n_a)\right])}^{\top }\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill {\Phi }_2\left(t\right)& =(c_1{f}_1\left[u(t-1)\right]+\cdots +c_{n_c}{f}_{n_c}\left[u(t-1)\right],\dots ,\hfill \\ \hfill & c_1{f}_1{[u(t-n_b)+\cdots +c_{n_c}{f}_{n_c}\left[u(t-n_b)\right])}^{\top }\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill {\Phi }_3\left(t\right)& ={\left(\sum _{i=1}^{n_a}\left(a_i{g}_1\left[y(t-i)\right]\right),\dots ,\sum _{i=1}^{n_a}\left(a_i{g}_{n_d}\left[y(t-i)\right]\right)\right)}^{\top }\hfill \end{array}$$

(8c)

$$\begin{array}{cc}\hfill {\Phi }_4\left(t\right)& ={\left(\sum _{i=1}^{n_b}\left(b_i{f}_1\left[u(t-i)\right]\right),\dots ,\sum _{i=1}^{n_b}\left(b_i{f}_{n_c}\left[u(t-i)\right]\right)\right)}^{\top }.\hfill \end{array}$$

(8d)

Define

$$\begin{array}{cc}\hfill {\Phi }_1& =[{\Phi }_1\left(1\right),{\Phi }_1\left(2\right),\dots ,{\Phi }_1\left(N\right)]=\underset{{\mathbf{\eta }}_1}{\underbrace{\left[\begin{array}{ccccc}\mathit{d}& \mathbf{0}& \dots & \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathit{d}& \mathbf{0}& \dots & \mathbf{0}\\ :& :& :& :& :\\ \mathbf{0}& \mathbf{0}& \dots & \mathbf{0}& \mathit{d}\end{array}\right]}}\mathit{G}\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill {\Phi }_2& =[{\Phi }_2\left(1\right),{\Phi }_2\left(2\right),\dots ,{\Phi }_2\left(N\right)]\hfill \end{array}$$

(9b)

$$\begin{array}{cc}\hfill {\Phi }_3& =[{\Phi }_3\left(1\right),{\Phi }_3\left(2\right),\dots ,{\Phi }_3\left(N\right)]\hfill \end{array}$$

(9c)

$$\begin{array}{cc}\hfill {\Phi }_4& =[{\Phi }_4\left(1\right),{\Phi }_4\left(2\right),\dots ,{\Phi }_4\left(N\right)].\hfill \end{array}$$

(9d)

It follows from (7) that

$$\begin{array}{c}\hfill Y=\mathit{a}{\Phi }_1+\mathit{b}{\Phi }_2+E=\mathit{d}{\Phi }_3+\mathit{c}{\Phi }_4+E.\end{array}$$

(10)

3.2. Parsimonious Identification Method

We consider an iterative estimation procedure to estimate unknown parameters from (10). Projecting the row space of Y in (10) onto the orthogonal complement of the row space of the matrix F, we have

$$\begin{array}{c}\hfill Y{\Pi }_F^{\perp }=(\mathit{a}{\Phi }_1+\mathit{b}{\Phi }_2+E){\Pi }_F^{\perp }=(\mathit{d}{\Phi }_3+\mathit{c}{\Phi }_4+E){\Pi }_F^{\perp }\end{array}$$

(11)

where ${\Pi }_F^{\perp }=I_N-F^{\top }{\left(F{F}^{\top }\right)}^{-1}F$. Due to $\mathit{b}{\Phi }_2={\mathit{\theta }}_{1}F$ and $\mathbf{c}{\Phi }_4={\mathbf{\theta }}_{1}F$, we have

$$\begin{array}{cc}\hfill Y{\Pi }_F^{\perp }& =\mathit{a}{\Phi }_1{\Pi }_F^{\perp }+E{\Pi }_F^{\perp }=\mathit{d}{\Phi }_3{\Pi }_F^{\perp }+E{\Pi }_F^{\perp }.\hfill \end{array}$$

(12)

Once $\mathit{d}$ is assumed to be known, the parameter vector $\mathit{a}$ can be estimated without knowing $\mathit{c}$ and $\mathit{b}$, while the parameter vector $\mathit{d}$ can be estimated with only knowing $\mathit{a}$, which allows an iterative method to obtain $\mathit{a}$ and $\mathit{d}$ based (12). The details of estimation of $\mathit{a}$ and $\mathit{d}$ are listed as below.

• An LS estimate ${\widehat{\mathit{a}}}^{\left(k\right)}$ can be obtained by substituting ${\widehat{\mathit{d}}}^{(k-1)}$ into (12) as

  $$\begin{array}{c}\hfill {\widehat{\mathit{a}}}^{\left(k\right)}=Y{\Pi }_F^{\perp }{\widehat{\Phi }}_1^{\top (k-1)}{\left({\widehat{\Phi }}_1^{(k-1)}{\Pi }_F^{\perp }{\widehat{\Phi }}_1^{\top (k-1)}\right)}^{-1}\end{array}$$

  (13)

  where the estimate ${\widehat{\Phi }}_1^{(k-1)}$ of ${\Phi }_1$ is a function of ${\widehat{\mathit{d}}}^{(k-1)}$.
• The unique solution ${\widehat{\mathit{a}}}^{\left(k\right)}$ can be obtained by performing a normalization operation as follows:

  $$\begin{array}{c}\hfill {\widehat{\mathit{a}}}^{\left(k\right)}={\widehat{\mathit{a}}}^{\left(k\right)}/{\parallel {\widehat{\mathit{a}}}^{\left(k\right)}\parallel }_2.\end{array}$$

  (14)
• An LS estimate ${\widehat{\mathit{d}}}^{\left(k\right)}$ can be obtained by substituting ${\widehat{\mathit{a}}}^{\left(k\right)}$ into (12) as

  $$\begin{array}{c}\hfill {\widehat{\mathit{d}}}^{\left(k\right)}=Y{\Pi }_F^{\perp }{\widehat{\Phi }}_3^{\top \left(k\right)}{\left({\widehat{\Phi }}_3^{\left(k\right)}{\Pi }_F^{\perp }{\widehat{\Phi }}_3^{\top \left(k\right)}\right)}^{-1}\end{array}$$

  (15)

  where the estimate ${\widehat{\Phi }}_3^{\left(k\right)}$ of ${\Phi }_3$ is a function of ${\widehat{\mathit{a}}}^{\left(k\right)}$.

In similar arguments to estimate $\mathit{a}$ and $\mathit{d}$, by projecting the row space of Y in (10) onto the orthogonal complement of the row space of the matrix G, we get

$$\begin{array}{c}\hfill Y{\Pi }_G^{\perp }=\mathit{b}{\Phi }_2{\Pi }_G^{\perp }+E{\Pi }_G^{\perp }=\mathit{c}{\Phi }_4{\Pi }_G^{\perp }+E{\Pi }_G^{\perp }.\end{array}$$

(16)

• An LS estimate ${\widehat{\mathit{b}}}^{\left(k\right)}$ can be estimated by substituting ${\widehat{\mathit{c}}}^{(k-1)}$ into (16) as

  $$\begin{array}{c}\hfill {\widehat{\mathit{b}}}^{\left(k\right)}=Y{\Pi }_G^{\perp }{\widehat{\Phi }}_2^{\top (k-1)}{\left({\widehat{\Phi }}_2^{(k-1)}{\Pi }_G^{\perp }{\widehat{\Phi }}_2^{\top (k-1)}\right)}^{-1}\end{array}$$

  (17)

  where the estimate ${\widehat{\Phi }}_2^{(k-1)}$ of ${\Phi }_2$ is a function of ${\widehat{\mathit{c}}}^{(k-1)}$.
• The unique solution ${\widehat{\mathit{b}}}^{\left(k\right)}$ can be obtained by performing a normalization operation as follows:

  $$\begin{array}{c}\hfill {\widehat{\mathit{b}}}^{\left(k\right)}={\widehat{\mathit{b}}}^{\left(k\right)}/{\parallel {\widehat{\mathit{b}}}^{\left(k\right)}\parallel }_2.\end{array}$$

  (18)
• An LS estimate ${\widehat{\mathit{c}}}^{\left(k\right)}$ can be estimated by substituting ${\widehat{\mathit{b}}}^{\left(k\right)}$ into (16) as

  $$\begin{array}{c}\hfill {\widehat{\mathit{c}}}^{\left(k\right)}=Y{\Pi }_G^{\perp }{\widehat{\Phi }}_4^{\top \left(k\right)}{\left({\widehat{\Phi }}_4^{\left(k\right)}{\Pi }_G^{\perp }{\widehat{\Phi }}_4^{\top \left(k\right)}\right)}^{-1}\end{array}$$

  (19)

  where the estimate ${\widehat{\Phi }}_4^{\left(k\right)}$ of ${\Phi }_4$ is a function of ${\widehat{\mathit{b}}}^{\left(k\right)}$.

The following general assumption A4 about persistently exciting (PE) conditions is made for system identifiability.

A4

Matrices F and G are full row rank.

The assumption A4 is a general assumption for all identification methods, which can be easily guaranteed by constructing F and G based on random signals. Based on which, once the initial values $\widehat{\mathit{d}}\left(0\right)$ and $\widehat{\mathit{c}}\left(0\right)$ are nonzero, it follows from (10) that matrices (${\widehat{\Phi }}_1\left(0\right),{\widehat{\Phi }}_2\left(0\right)$) will be full row rank, due to that the rank of matrices ${\widehat{\mathbf{\eta }}}_1$ and ${\widehat{\mathbf{\eta }}}_2$ with nonzero parameters $\widehat{\mathit{d}}\left(0\right)$ and $\widehat{\mathit{c}}\left(0\right)$ are full row rank, such that the estimates $\widehat{\mathit{a}}\left(1\right)$ in (13) and $\widehat{\mathit{b}}\left(1\right)$ in (17) are ensured nonzero. Once the nonzero estimated parameters $\widehat{\mathit{a}}\left(1\right)$ and $\widehat{\mathit{b}}\left(1\right)$ are obtained, and then $\widehat{\mathit{c}}\left(1\right)$ and $\widehat{\mathit{d}}\left(1\right)$ are ensured nonzero, due to that the rank of matrices ${\widehat{\mathbf{\eta }}}_3$ and ${\widehat{\mathbf{\eta }}}_4$ with nonzero parameters $\widehat{\mathit{a}}\left(1\right)$ and $\widehat{\mathit{b}}\left(1\right)$ are also full row rank. Following above arguments, we conclude that once the PE matrices and the nonzero initial values are given, the proposed algorithm can provide consistent estimated parameters.

The parameters are estimated asynchronously from the PPMs by two independent projection based iterative methods, thus the proposed method, referred to as Asynchronous Parsimonious Method (APM), is summarized as Algorithm 1.

Algorithm 1:

1. Initialization: $\widehat{\mathit{d}}\left(0\right)$ and $\widehat{\mathit{c}}\left(0\right)$: arbitrary nonzero values. 2. Iteration: for $k=0:\mathrm{convergence}$ a1. Estimate ${\widehat{\mathit{a}}}^{\left(k\right)}$ as (13) and (14). a2. Estimate ${\widehat{\mathit{d}}}^{\left(k\right)}$ as (15). a3. If the stopping criterion is satisfied, and break, else go to step (a1). end 3. Iteration: for $k=0:\mathrm{convergence}$ b1. Estimate ${\widehat{\mathit{b}}}^{\left(k\right)}$ as (17) and (18). b2. Estimate ${\widehat{\mathit{c}}}^{\left(k\right)}$ as (19). b3. If the stopping criterion is satisfied, and break, else go to step (b1). end 4. Let ${\widehat{\mathit{a}}}^{\top }={\widehat{\mathit{a}}}^{\top }\left(k\right)\mathrm{sgn}\left({\widehat{\mathit{a}}}_1^{\top }\left(k\right)\right)$, ${\widehat{\mathit{b}}}^{\top }={\widehat{\mathit{b}}}^{\top }\left(k\right)\mathrm{sgn}\left({\widehat{\mathit{b}}}_1^{\top }\left(k\right)\right)$, ${\widehat{\mathit{d}}}_2={\widehat{\mathit{d}}}_2(k+1)\mathrm{sgn}\left({\widehat{\mathit{a}}}_1^{\top }\left(k\right)\right)$, and ${\widehat{\mathit{c}}}_2={\widehat{\mathit{c}}}_2(k+1)\mathrm{sgn}\left({\widehat{\mathit{b}}}_1^{\top }\left(k\right)\right)$, where $\mathrm{sgn}\left(\right)$ is a sign function and ${\widehat{\mathit{a}}}_1^{\top }\left(k\right)$ and ${\widehat{\mathit{b}}}_1^{\top }\left(k\right)$ are the first element of ${\widehat{\mathit{a}}}^{\top }\left(k\right)$ and ${\widehat{\mathit{b}}}^{\top }\left(k\right)$, respectively.

4. Variance Analysis

Following the Theorem 1 in [11], one can eaily prove that the proposed method with arbitrary nonzero initial values $\widehat{\mathit{d}}\left(0\right)$ and $\widehat{\mathit{c}}\left(0\right)$ provides consistent estimates for H-W models. This section, we analyze the variance of the AMP estimates relative to that of conventional TSA algorithm. Due to all paramters are estimated in a similar way, we only analy the variances of estimated paramter $\widehat{a}$. The LS estimation error variance of $\widehat{\mathbf{\theta }}$ in TSA [4] can be computed as,

$$\begin{array}{c}\hfill cov({\widehat{\mathbf{\theta }}}_1,{\widehat{\mathbf{\theta }}}_2)={\delta }_e^2{\left(\Phi {\Phi }^{\top }\right)}^{-1}\end{array}$$

(20)

Substituting (4e) into (20), we have

$$\begin{array}{c}\hfill cov({\widehat{\mathbf{\theta }}}_1,{\widehat{\mathbf{\theta }}}_2)={\delta }_e^2{\left[\begin{array}{cc}\mathit{F}{\mathit{F}}^{\top }& \mathit{F}{\mathit{G}}^{\top }\\ \mathit{G}{\mathit{F}}^{\top }& \mathit{G}{\mathit{G}}^{\top }\end{array}\right]}^{-1}\end{array}$$

(21)

Based the block matrix inversion relation in [15], we have

$$\begin{array}{c}\hfill cov\left({\widehat{\mathbf{\theta }}}_2\right)={\delta }_e^2{\left(\mathit{G}{\mathit{G}}^{\top }\right)}^{-1}+{\delta }_e^2{\left(\mathit{G}{\mathit{G}}^{\top }\right)}^{-1}\mathit{G}{\mathit{F}}^{\top }{Q}^{-1}{\left(\mathit{G}{\mathit{F}}^{\top }\right)}^{\top }{\left(\mathit{G}{\mathit{G}}^{\top }\right)}^{-1}\end{array}$$

(22)

where $Q=\mathit{F}{\mathit{F}}^{\top }-\mathit{F}{\mathit{G}}^{\top }{\left(\mathit{G}{\mathit{G}}^{\top }\right)}^{-1}\mathit{G}{\mathit{F}}^{\top }$.

Since the solutions of $\widehat{\mathit{a}},\widehat{\mathit{b}},\widehat{\mathit{c}}$ and $\widehat{\mathit{d}}$ are provided by SVD, decompositions of ${\widehat{\mathbf{\theta }}}_1$, and ${\widehat{\mathbf{\theta }}}_2$. Omitting the error terms more than first order which do not contribute to the estimation error variance, deriving based on first-order Taylor expansion of SVD in [16], the variance of the parameter estimation errors of $\widehat{\mathit{a}}$ can be computed in approximately as following,

$$\begin{array}{c}\hfill cov\left({\widehat{\mathit{a}}}_{TSA}\right)={\delta }_e^2{\left({\mathbf{\eta }}_1\mathit{F}{\mathit{F}}^{\top }{\mathbf{\eta }}_1^{\top }\right)}^{-1}+\\ \hfill {\delta }_e^2{\left({\mathbf{\eta }}_1\mathit{F}{\mathit{F}}^{\top }\right)}^{-1}\mathit{F}{\mathit{G}}^{\top }{Q}^{-1}{\left(\mathit{F}{\mathit{G}}^{\top }\right)}^{\top }{\left(\mathit{F}{\mathit{F}}^{\top }{\mathbf{\eta }}_1^{\top }\right)}^{-1}\end{array}$$

(23)

Note that the second terms at the right-hand side of (52) are positive definite, we have

$$\begin{array}{c}\hfill cov\left({\widehat{\mathit{a}}}_{TSA}\right)\ge {\delta }_e^2{\left({\mathbf{\eta }}_1\mathit{F}{\mathit{F}}^{\top }{\mathbf{\eta }}_1^{\top }\right)}^{-1}\end{array}$$

(24)

In a similar way as (20), the estimation error variances of $\widehat{\mathit{a}},\widehat{\mathit{b}},\widehat{\mathit{c}}$ and $\widehat{\mathit{d}}$ in APM are computed as following

$$\begin{array}{c}\hfill cov\left({\widehat{\mathit{a}}}_{APM}\right)={\delta }_e^2{\left({\widehat{\mathbf{\eta }}}_1\mathit{F}{\mathit{F}}^{\top }{\widehat{\mathbf{\eta }}}_1^{\top }\right)}^{-1}\end{array}$$

(25)

Since estimates from AMP are consistent, neglecting the high-order terms in above equations for asymptotic variance, and only considering the noise terms for estimation error variances, we have

$$\begin{array}{c}\hfill cov\left({\widehat{\mathit{a}}}_{APM}\right)={\delta }_e^2{\left({\mathbf{\eta }}_1\mathit{F}{\mathit{F}}^{\top }{\mathbf{\eta }}_1^{\top }\right)}^{-1}\end{array}$$

(26)

Compared (24) and (26), we conclude that the APM estimates generally have a smaller variance than those of the TSA estimates by neglecting high-order terms in estimation error.

5. Case Studies

In this section, we use a benchmark Hammerstein–Wiener model with highly nonlinearities modified from [4] to evaluate the proposed APM and compare it with the TSA. The system model of the benchmark example is presented in the following:

$$\begin{array}{cc}\hfill \mathit{a}& ={[0.1569,0.0934,0.5477,0,-0.7303,-0.3651]}^{\top }\hfill \\ \hfill \mathit{b}& ={[0.1569,0.0934,0.5477,0,0.7303,0.3651]}^{\top }\hfill \\ \hfill \mathit{c}& ={[1,4,3,7,1,3.4]}^{\top }\hfill \\ \hfill \mathit{d}& ={[0.4,0.25]}^{\top }\hfill \\ \hfill f\left(t\right)& ={[u\left(t\right),u^2\left(t\right),u^3\left(t\right),u^4\left(t\right),u^5\left(t\right),u^6\left(t\right)]}^{\top }\hfill \\ \hfill g\left(t\right)& ={[y\left(t\right),\sin \left(y\left(t\right)\right)]}^{\top }.\hfill \end{array}$$

The input $\left\{u\right(t\left)\right\}$ and the measurement noise $\left\{e\right(t\left)\right\}$ are chosen as two zero-mean Gaussian random sequences. The variance of $\left\{u\right(t\left)\right\}$ is fixed as one, while the variance of $\left\{e\right(t\left)\right\}$ is adjusted to achieve a given Signal-to-Noise Ratio (SNR). In this example, three SNR values will be considered, i.e., 15, 20, and 25 dB. For each experimental condition, we conduct a Monte-Carlo (MC) simulation of 100 runs, each with $N=500$ data points. The performance of the methods is evaluated with respect to the accuracy and standard deviations in tracking the system parameters $\mathbf{\theta }=(\mathit{a},\mathit{b},\mathit{c},\mathit{d})$ and output.

More specifically, the quality of the estimates is evaluated by investigating the Parameter Estimation Errors (PEE), Variance Accounted For (VAF) on a validation data set which is different from the one used for model identification. The PEE and VAF are defined as follows:

$$\begin{array}{cc}\hfill \mathrm{PEE}& =\frac{\parallel \mathbf{\theta }-\widehat{\mathbf{\theta }}{\parallel }_2}{{\parallel \mathbf{\theta }\parallel }_2}100\%\hfill \\ \hfill \mathrm{VAF}& =\max \left\{0,1-\frac{\mathrm{var}({\overline{y}}_k-\widehat{y})}{\mathrm{var}\left({\overline{y}}_k\right)}\right\}100\%\hfill \end{array}$$

where the operator $\mathrm{var}(·)$ denotes the variance of its argument, ${\overline{y}}_k$ denotes the noise-free simulated output of the system, and ${\widehat{y}}_k$ denotes the the noise-free simulated model output. The VAF is allowed to reach $100\%$, and PPE is allowed to reach $0\%$ when the model is equal to the true system.

Theoretically, the computational complexity of the proposed APM per iteration is $O\left(N^2\right)$, which is obviously more computationally demanding than $O\left(N\right)$ in the TSA, because of the projection operations [17]. Fortunately, from the averaged estimation results of Euclidean norm $\parallel \mathbf{\theta }\parallel$ for 100 MC simulations at each iteration in APM under different SNRs shown in Figure 1, we observe that APM converges very quickly within 4 iterations for high SNRs (e.g., 20 and 25 dB), and also provides accurate estimation results even for a low SNR of 15 dB. It is concluded that the APM can provide accurate estimation results for highly nonlinear systems from a limited number of noisy samples. The estimated parameters in $\mathbf{\theta }$ along with the mean values and standard deviations and the PEE are listed in

Table 1. Estimated results of $\mathbf{\theta }$ and PEE.

True	APM (SNR = 15)	APM (SNR = 20)	APM (SNR = 25)	TSA (SNR = 15)	TSA (SNR = 20)	TSA (SNR = 25)
$a_1=0.1569$	0.1604 (±0.0793)	0.1654 (±0.0617)	0.1477 (±0.0423)	0.1643 (±0.0840)	0.1669 (±0.0597)	0.1470 (±0.0467)
$a_2=0.0934$	0.0912 (±0.0632)	0.0906 (±0.0443)	0.0979 (±0.0326)	0.0855 (±0.0836)	0.0907 (±0.0485)	0.0928 (±0.0367)
$a_3=0.5477$	0.5442 (±0.1472)	0.5480 (±0.1187)	0.5465 (±0.0277)	0.5162 (±0.1961)	0.5407 (±0.0413)	0.5474 (±0.0280)
$a_4=0$	0.0009 (±0.0500)	0.0032 (±0.0357)	0.0002 (±0.0267)	0.0014 (±0.00826)	0.0041 (±0.0413)	0.0026 (±0.0333)
$a_5=-0.7303$	−0.7218 (±0.0481)	−0.7178 (±0.1473)	−0.7271 (±0.0252)	−0.6758 (±0.2514)	−0.7317 (±0.0419)	−0.7249 (±0.0257)
$a_6=-0.3651$	−0.3597 (±0.07317)	−0.3478 (±0.0839)	−0.3690 (±0.0228)	−0.3164 (±0.1460)	−0.3510 (±0.0398)	−0.3713 (±0.0267)
$b_1=0.1569$	0.1572 (±0.0128)	0.1577 (±0.0068)	0.1558 (±0.0040)	0.2757 (±0.2053)	0.1515 (±0.0770)	0.1620 (±0.0377)
$b_2=0.0934$	0.0930 (±0.0121)	0.0931 (±0.0060)	0.0939 (±0.0038)	0.0039 (±0.3265)	0.0692 (±0.1037)	0.0913 (±0.0367)
$b_3=0.5477$	0.5465 (±0.0155)	0.5478 (±0.0112)	0.5468 (±0.0079)	0.1405 (±0.4996)	0.4736 (±0.2627)	0.5469 (±0.0313)
$b_4=0.7303$	0.7293 (±0.0114)	0.7306 (±0.0074)	0.7299 (±0.0050)	0.1432 (±0.5629)	0.6178 (±0.3892)	0.7220 (±0.0511)
$b_5=0.3651$	0.3676 (±0.0176)	0.3634 (±0.0156)	0.3675 (±0.0093)	0.1299 (±0.4004)	0.3028 (±0.2003)	0.3720 (±0.1045)
$c_1=1$	1.0774 (±0.2146)	0.9999 (±0.1701)	1.0022 (±0.0967)	0.2651 (±0.8143)	0.8495 (±0.5122)	0.9914 (±0.3376)
$c_2=4$	4.0237 (±0.8490)	3.9649 (±0.4590)	4.0077 (±0.2340)	1.1182 (±3.0569)	3.3149 (±1.7629)	3.9646 (±0.4040)
$c_3=3$	2.6377 (±1.3845)	2.9785 (±0.6425)	3.0432 (±0.3804)	0.9981 (±2.5809)	2.5448 (±1.6301)	3.0505 (±0.3848)
$c_4=7$	7.1772 (±1.6055)	7.0295 (±1.4038)	7.0416 (±0.7487)	1.9601 (±10.3455)	6.1131 (±4.6544)	7.1346 (±0.8611)
$c_5=1$	1.0199 (±1.3840)	1.0168 (±0.5661)	0.9645 (±0.3401)	0.0885 (±1.6670)	0.8553 (±0.7665)	0.9455 (±0.3612)
$c_6=3.4$	3.5277 (±1.3443)	3.4013 (±1.0777)	3.3914 (±0.5844)	0.7933 (±5.1063)	2.7925 (±1.8014)	3.2967 (±0.6655)
$d_1=0.4$	0.5043 (±0.1304)	0.4927 (±0.1017)	0.5014 (±0.0040)	0.4731 (±0.1723)	0.5027 (±0.0071)	0.5064 (±0.0045)
$d_2=0.25$	0.2551 (±0.0675)	0.2481 (±0.0595)	0.2500 (±0.0211)	0.2518 (±0.1244)	0.2549 (±0.0330)	0.2501 (±0.0216)
PPE (%)	38.7442 (±21.5544)	18.7887 (±10.3743)	10.8245 (±5.8194)	130.4982 (±71.5442)	36.6713 (±48.9617)	11.8245 (±7.0916)

. From the results shown in Table 1, we can conclude that

• Under the same noise levels, the proposed APM provides better estimation results in terms of accuracy and standard deviations than TSA.
• The two methods yield similar results for $\mathrm{SNR}=25$ dB (the corresponding PEEs are $10.8245 (\pm 5.8194)$ and $11.8245 (\pm 7.0916)$, respectively). The performance of APM is obviously better than the performance of TSA for $c_2,c_4,c_6$, PEE for $\mathrm{SNR}=20$ dB, and $c_1,c_2,c_3,c_4,c_5,c_6,\mathrm{VAF},\mathrm{PPE}$ for $\mathrm{SNR}=15$ dB. The attenuation of the signal levels does not obviously affect the APM while it deteriorates the performance of TSA significantly, showing that proposed APM is more robust to noise and data size.

The calculated VAFs for the proposed APM and TSA are given in Figure 2, while the corresponding histograms of the VAF values are shown in Figure 3. In

Table 2. Estimated Results of VAF under Different SNRs.

Methods	VAF
APM (SNR = 15 dB)	99.79 (±0.0977)
APM (SNR = 20 dB)	99.94 (±0.0270)
APM (SNR = 25 dB)	99.97 (±0.0214)
TSA (SNR = 15 dB)	56.81 (±31.1700)
TSA (SNR = 20 dB)	97.03 (±3.5800)
TSA (SNR = 25 dB)	99.40 (±0.5200)

the mean values and standard deviations of VAF are given. The results show that the VAFs of APM are better than that of TSA under the same noise levels. The results verify again our argument: as the avoidance of introducing extra parameters and performing SVD to estimate the parameters of ($\mathit{a},\mathit{b},\mathit{c},\mathit{d}$) from the products $\mathit{a}\mathit{d}$ and $\mathit{b}\mathit{c}$, the proposed APM based on PPMs with fewer parameters can enhance the accuracy of the estimated quantities and decrease the variance of estimates, for the identification of highly nonlinear Hammerstein-Wiener systems given short noisy data length sequences.

To evaluate the sensitivity of the proposed APM under different sample lengths, the mean values of PEE and VAF for 100 Monte-Carlo simulations with respect to different N and SNRs using proposed APM and TSA are shown in Figure 4 and Figure 5, respectively. It can be seen that the proposed method provides quite good performance when $N>100$. The performance of the proposed APM in terms of PEE and VAF is better than the TSA especially for identifying N-L-N systems from a limited number of noisy data with a low SNR, see PEE and VAF when data point $N<1000$ and SNR = 25 dB. These results show that proposed APM is more robust to noise and data size.

6. Conclusions

A new identification formulation for H-W models, referred to as APM, is presented. The avoidance of using over-parametrization models leads to a number of improvements, i.e., reducing the number of unknown parameters to be estimated and avoiding a truncated SVD step, so improving the estimated quantities in terms of accuracy and variance compared to TSA. These improvements especially hold for highly nonlinear H-W model identification from a limited number of noisy samples. The reasons why the new method results in an improvement model over the previous LS method have been demonstrated by variance analysis. The application to an illustrative example has well demonstrated the effectiveness and merit of the proposed method in comparison with TSA. Future work should focus on identifying other nonlinear systems by taking into account some additional prior knowledge about the systems.