Two-Stage Nested Array Direction of Arrival Estimation for Mixed Signals

Abstract

In this paper, a novel two-stage subspace-based direction of arrival (DOA) estimation algorithm with the nested array is proposed for mixed signals containing circular and non-circular ones. By exploiting the difference between the two types of steering vectors, the DOAs of circular signals are estimated in the first stage. After eliminating the estimated information of circular signals by the covariance matrix reconstruction and oblique projection methods, the dimensions of the noise subspace are increased in estimating the DOAs of non-circular signals in the second stage. Through the two-stage estimation, the DOAs of the circular and non-circular signals are estimated separately and different types of signals with similar or the same DOAs can be distinguished. Furthermore, to avoid the two-dimensional (2-D) search with huge computational burden, a one-dimensional (1-D) search method exploiting the rank deficiency is proposed in the DOA estimation for non-circular signals. The simulation results show that the proposed algorithm can effectively improve the estimation accuracy and resolution probability, especially when circular and non-circular signals have similar DOAs.

1. Introduction

DOA estimation is a key problem in array signal processing and has been widely used in communication, radar, detection, covert transmission, and other fields [1]. A source localization method based on multiple signal classification (MUSIC) was proposed in [2]. The DOAs were calculated by the estimation of signal parameters via the rotational invariant techniques (ESPRIT) in [3], where a novel ESPRIT algorithm based on geometric algebra (GA-ESPRIT) was proposed and the DOA was estimated with double parallel uniform linear arrays (ULA). A fast Fourier transform (FFT)-based method was proposed in [4] with low computational burden. A MUSIC-like method for non-circular signals (NC-MUSIC) was presented in [5], where a virtual array with twice the number of physical array sensors was constructed.

In recent years, the non-uniform linear array (NLA) has been applied to DOA estimation. A compressed sparse array (CSA) was proposed in [6] by combining the compressive sensing and the sparse array. An off-grid DOA estimation method was presented in [7], where a mismatch between true DOA and discrete angular grid was resolved. The estimation performance was improved with NLA and fourth-order cumulants in [8,9]. Specifically, a novel array structure was constructed with the co-prime array in [8], and DOAs was estimated via the sparse Bayesian learning. An array model was proposed in [9], where the fourth-order cumulants and the sparse reconstruction algorithm were exploited to estimate the DOA. An enhanced nested array (ENA) structure was proposed in [10] and a compressed sensing method was applied to DOA estimation. For mixed signals scenarios, the sparse reconstruction and root-MUSIC were used in [11] for non-circular signals and circular signals, respectively. The computational burden of these sparse reconstruction methods is higher than that of the subspace-based ones, although the spatial smoothing is not required.

The methods for co-prime arrays were derived in [12,13], where the DOAs were estimated by exploiting the spatial smoothing and subspace characteristic with the NLA. Specifically, a new filled differential co-prime array (CAFDC) was considered in [12] and the degree of freedom (DOF) was significantly increased by introducing two additional subarrays. The spatial smoothing MUSIC (SS-MUSIC) algorithm was introduced in [14] for circular signals where a ULA with more virtual elements was constructed by the nested array. For 2-D DOA estimation with a coprime cubic array (CCA), a total array-based multiple signals classification (TA-MUSIC) algorithm with a significant increase in accuracy was first proposed in [15]. The traditional nested array was converted into a transformed nested array (TNA) by exchanging the position of the sub-array in [16], and the number of equivalent array elements increased. A method for constructing the continuous differential co-array (cDCA) and continuous sum co-array (cSCA) was proposed for non-circular signals. By discarding the rotation phase of the non-circular signal, the reduced-dimension MUSIC algorithm was proposed, and the computational burden was reduced. To summarize the above description, the sum-difference co-array and reduced-dimension MUSIC (SD-RD-MUSIC) method was illustrated in [17]. To avoid the problem of spatial smoothing with the reduced DOF, a new nested array structure based on quasi-stationary signals (QS) was applied in [18]. The quasi-stationary property of QS was exploited in [19] and the ESPRIT algorithm was applied to estimate the DOAs of non-circular signals. A two-stage DOA estimation algorithm was proposed in [20] with the conformal uniform circular array (UCA), where DOAs were accurately estimated in a multi-target scenario. A spatial-temporal technique was developed in [21] and the low-power spoofing attack and multipath can be distinguished in the global navigation satellite system (GNSS).

In this paper, the scenario of mixed signals containing circular and non-circular ones is considered, and a subspace-based DOA estimation method for two types of signals with the nested array is proposed. The DOAs of the two types of signals are calculated separately based on different types of steering vectors, for the equivalent steering vectors of circular signals only have the cDCA part, while those of non-circular signals contain the cDCA and cSCA parts simultaneously. The DOAs of the circular signals are estimated, while those of non-circular signals are not estimated in the first stage. Then, the covariance matrix containing only the non-circular signals can be obtained by the covariance matrix reconstruction and oblique projection. Thus, the dimensions of noise subspace are increased in estimating the DOAs of non-circular signals in the second stage and two types of signals are distinguished. By separating the DOAs and rotation phases, the DOAs of non-circular signals can be calculated by exploiting the rank deficiency of the matrix, where the 2-D search is avoided and the computational burden is reduced significantly. The estimation accuracy and resolution probability are improved by the separate estimation of different types of signals, especially when circular and non-circular signals have similar DOAs. Even if different types of signals have the same DOAs, the proposed method is still feasible. The methods of estimating different types of signals separately and exploiting the rank deficiency to estimate DOAs in this paper improve the estimation accuracy.

The remainder of this paper is organized as follows. The Section 2 introduces the mixed circular and non-circular signal model, and then explains the structures of the nested array, the cDCA and cSCA. In the Section 3, the method of constructing the cDCA and cSCA using nested array characteristics and estimating DOAs with 1-D search is described. The Section 4 shows the computer simulation results to verify the effectiveness of the proposed method. The Section 5 summarizes the paper.

Notations: We use bold symbols to represent vectors and matrices. The superscripts ${(\cdot )}^{*}$, ${(\cdot )}^T$, ${(\cdot )}^H$ and ${\{\cdot \}}^{†}$ denote the conjugation, transpose, conjugate transpose, and pseudo inverse, respectively. $E\{\cdot \}$ represents the calculation of expectation. $rank(\cdot )$ denotes the rank of a matrix. $diag\{\cdot \}$ denotes the diagonalization operator. I denotes the unit matrix and O denotes the matrix with all zero elements.

2. Data Model

2.1. Mixed Circular and Non-Circular Signals Model

Any digital modulation signal in the complex plane can be expressed as [22]

$$s(t)=\sigma e^{-j\phi }\left(\sqrt{\frac{(1+k)}{2}}{s}_I(t)+j\sqrt{\frac{(1-k)}{2}}{s}_Q(t)\right)$$

(1)

where $\phi$ denotes the rotation phase, k controls the signal amplitude, $s_I(t)$ denotes the co-directional component, and $s_Q(t)$ denotes the orthogonal component. $E\left\{\right|s_I(t){|}^2\}=1$, $E\left\{\right|s_Q(t){|}^2\}=1$ and $E\{s_I(t)s_Q(t)\}=0$.

To measure the degree of non-circularity, the definition of a signal’s non-circular rate $\rho$ is given by [23]

$$\rho =\frac{E\{s^2(t)\}}{E\left\{\right|s(t){|}^2\}}=k{e}^{j2\phi }0\le |\rho |\le 1$$

(2)

where $\left|\rho \right|=0$ and $0<\left|\rho \right|\le 1$ represent the circular and non-circular signals, respectively, and $\left|\rho \right|=1$ indicates the strictly non-circular signal. For convenience, only the case of mixed strictly non-circular and circular signals is considered in this paper. Then the circular signal can be represented as $s_c(t)=e^{-j\phi }{s}_{IQ}(t)$, and the non-circular signal can be represented as $s_n(t)=e^{-j\phi }{s}_{II}(t)$, where $s_{IQ}(t)=\sigma \left(\sqrt{\frac{1}{2}}{s}_I(t)+j\sqrt{\frac{1}{2}}{s}_Q(t)\right)$ and $s_{II}(t)=\sigma s_I(t)$.

For the rest of this paper, the assumptions are required as follows:

(1)

All signals are statistically independent, zero-mean uncorrelated narrowband stationary processes.

(2)

Every sensor has zero-mean, additive white Gaussian noise and the noise is independent from the signals.

2.2. The Nested Array, cDCA, and cSCA Model

Consider a nested array having $N_1$ elements with inter-element spacing $d$ and $N_2$ elements with inter-element spacing $\left(N_1+1\right)d$, where $d$ is $\frac{\lambda }{2}$ in this paper and $\lambda$ denotes the wavelength of the carrier wave. Assume $L$ to be a set of integers, defined as

$$L=\left\{n_1|0\le n_1\le N_1-1\right\}\cup \left\{n_2(N_1+1)-1|1\le n_2\le N_2\right\}$$

(3)

Then the sensor locations of the nested array can be described as

$$L^{\prime }=dL=\left\{l_1,l_2,\cdots ,l_{N^{\prime }}\right\}$$

(4)

where $N^{\prime }=N_1+N_2$ is the total number of array elements.

Figure 1 shows the model of the nested array, where $N_1=N_2=4$.

Assume M uncorrelated narrowband signals impinge on a nested array. Then, the received signals on the array can be expressed as

$$x(t)=s(t)+n(t)=A_C{s}_C(t)+A_N{s}_N(t)+n(t)=x_C(t)+x_N(t)+n(t)$$

(5)

where x(t) and n(t) are $N^{\prime }\times 1$ vectors, and

$$x_C(t)=A_C{s}_C(t)$$

(6)

$$x_N(t)=A_N{s}_N(t)$$

(7)

$$s(t)=A_C{s}_C(t)+A_N{s}_N(t)$$

(8)

$$A_N=[a({\theta }_1),a({\theta }_2),\cdots ,a({\theta }_{M_1})]$$

(9)

$$A_C=[a({\theta }_{M_1+1}),a({\theta }_{M_1+2}),\cdots ,a({\theta }_M)]$$

(10)

$$s_N(t)={[s_1(t),s_2(t),\cdots ,s_{M_1}(t)]}^T$$

(11)

$$s_C(t)={[s_{M_1+1}(t),s_{M_1+2}(t),\cdots ,s_M(t)]}^T$$

(12)

$a({\theta }_m)={[e^{-j2\pi l_{1}sin{\theta }_m/\lambda },e^{-j2\pi l_{2}sin{\theta }_m/\lambda },\dots , e^{-j2\pi l_{N^{\prime }}sin{\theta }_m/\lambda }]}^T$ denotes the m-th steering vector and $M_1$ indicates the number of non-circular signals.

Figure 2 and Figure 3 show the locations and number of elements in the cDCA and cSCA, respectively. According to [17], the total number of elements in the cDCA is $2R_1+1$, and the number of elements in the cSCA is $2(R_2+1)$, where $R_1=N_1{N}_2+N_2-1$ and $R_2=N_1{N}_2+N_1+N_2-1$. The distance between the adjacent virtual elements is d.

3. The Proposed Method

Define the matrix J as

$$J=\left(\begin{array}{l}x(t)\\ x^{*}(t)\end{array}\right)$$

(13)

Then, the covariance matrix of J can be expressed as

$$R_J=E\{J{J}^H\}=\left(\begin{array}{cc}E\{x(t)x^H(t)\}& E\{x(t)x^T(t)\}\\ E\{x^{*}(t)x^H(t)\}& E\{x^{*}(t)x^T(t)\}\end{array}\right)$$

(14)

For convenience, $R_J$ is further rewritten as

$$R_J=R_C+R_N+{\sigma }_n^{2}I$$

(15)

where

$$R_C=\left(\begin{array}{cc}E\{x_C(t)x_C^H(t)\}& E\{x_C(t)x_C^T(t)\}\\ E\{x_C^{*}(t)x_C^H(t)\}& E\{x_C^{*}(t)x_C^T(t)\}\end{array}\right)$$

(16)

and

$$R_N=\left(\begin{array}{cc}E\{x_N(t)x_N^H(t)\}& E\{x_N(t)x_N^T(t)\}\\ E\{x_N^{*}(t)x_N^H(t)\}& E\{x_N^{*}(t)x_N^T(t)\}\end{array}\right)$$

(17)

${\sigma }_n^2$ represents the noise power.

3.1. Non-Circular Signals

For non-circular signals, only $E\{x_N(t)x_N^H(t)\}$, $E\{x_N(t)x_N^T(t)\}$, and $E\{x_N^{*}(t)x_N^H(t)\}$ are analyzed because all valid information of $E\{x_N^{*}(t)x_N^T(t)\}$ is also contained in $E\{x_N(t)x_N^H(t)\}$.

$$E\{x_N(t)x_N^H(t)\}=A_{N}diag\{{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_{M_1}^2\}{A}_N^H={\displaystyle \sum _{m=1}^{M_1}{\sigma }_m^{2}a({\theta }_m)a{({\theta }_m)}^H}$$

(18)

$$E\{x_N(t)x_N^T(t)\}=A_N{\Phi }_{N}diag\{{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_{M_1}^2\}{\Phi }_N^T{A}_N^T$$

(19)

where ${\sigma }_m^2$ represents the power of the m-th non-circular signal and ${\Phi }_N=diag\{e^{-j{\phi }_1},e^{-j{\phi }_2},\cdots ,e^{-j{\phi }_{M_1}}\}$ is a diagonal matrix which contains the rotation phases information of the non-circular signals. We define the merged direction matrix $A_v$ and its m-th steering vector as

$$A_v=A_N{\Phi }_N=[a_v({\theta }_1,{\phi }_1),a_v({\theta }_2,{\phi }_2),\cdots ,a_v({\theta }_{M_1},{\phi }_{M_1})]$$

(20)

$$a_v({\theta }_m,{\phi }_m)=a_N({\theta }_m)e^{-j{\phi }_m}$$

(21)

Then $E\{x_N(t)x_N^T(t)\}$ can be rewritten as

$$\begin{array}{ll}E\{x_N(t)x_N^T(t)\}& =A_{v}diag\{{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_{M_1}^2\}{A}_v^T\\ & ={\displaystyle \sum _{m=1}^{M_1}{\sigma }_m^2{a}_v({\theta }_m,{\phi }_m)a_v{({\theta }_m,{\phi }_m)}^T}\end{array}$$

(22)

And

$$E\{x_N^{*}(t)x_N^H(t)\}={(E\{x_N(t)x_N^T(t)\})}^{*}$$

(23)

According to [17], after vectorizing and removing the redundant elements in (18), (22) and (23), the equivalent signals on the cDCA and cSCA can be expressed as

$$y_v=D_n{p}_n$$

(24)

$$D_n=[d_n({\theta }_1),d_n({\theta }_2),\cdots ,d_n({\theta }_{M_1})]$$

(25)

$$y_{\delta }=C_{\delta }{p}_n$$

(26)

$$y_{\varphi }=C_{\varphi }{p}_n$$

(27)

$$C_{\delta }=[c_{\delta }({\theta }_1,{\phi }_1),c_{\delta }({\theta }_2,{\phi }_2),\cdots ,c_{\delta }({\theta }_{M_1},{\phi }_{M_1})]$$

(28)

$$C_{\varphi }=[c_{\varphi }({\theta }_1,{\phi }_1),c_{\varphi }({\theta }_2,{\phi }_2),\cdots ,c_{\varphi }({\theta }_{M_1},{\phi }_{M_1})]$$

(29)

3.2. Circular Signals

According to the definition of circular signals, we have $E\{x_C(t)x_C^T(t)\}=O$ and $E\{x_C^{*}(t)x_C^H(t)\}=O$ in (16). Then, the information of circular signals only exists in $E\{x_C(t)x_C^H(t)\}$ and the cDCA parts of the steering vectors of circular signals have the same structure as those of non-circular signals. The circular signals on the cDCA are

$$y_c=D_c{p}_c$$

(30)

where $D_c=[d_c({\theta }_{M_1+1}),d_c({\theta }_{M_1+2}),\cdots ,d_c({\theta }_M)]$ and $p_c={[{\sigma }_{M_1+1}^2,{\sigma }_{M_1+2}^2,\cdots ,{\sigma }_M^2]}^T$ represents the power vector of the circular signals.

3.3. DOA Estimation of Mixed Signals

According to the noise assumption, noise only exists on the cDCA. Then, the constructed signals and noise on the cDCA can be expressed as

$$y_d=D_{d}p+{\sigma }_n^2{n}_d$$

(31)

$$D_d=[d_n({\theta }_1),\cdots ,d_n({\theta }_{M_1}),d_c({\theta }_{M_1+1}),\cdots ,d_c({\theta }_M)]$$

(32)

$$p={[{\sigma }_1^2,\cdots ,{\sigma }_{M_1}^2,{\sigma }_{M_1+1}^2,\cdots {\sigma }_M^2]}^T$$

(33)

$n_d$ is a column vector in which only the $R_1+1$ -th element is 1 and the other elements are 0.

The constructed signals on the cSCA can be expressed as

$$y_{\delta }=S_{z}p$$

(34)

$$y_{\varphi }=S_{f}p$$

(35)

$$S_z=\left[c_{\delta }\right({\theta }_1,{\phi }_1),\cdots ,c_{\delta }({\theta }_{M_1},{\phi }_{M_1}),{\nu }_{M_1+1},\cdots ,{\nu }_M]$$

(36)

$$S_f=\left[c_{\varphi }\right({\theta }_1,{\phi }_1),\cdots ,c_{\varphi }({\theta }_{M_1},{\phi }_{M_1}),{\tau }_{M_1+1},\cdots ,{\tau }_M]$$

(37)

where ${\nu }_m (M_1+1\le m\le M)$ and ${\tau }_m (M_1+1\le m\le M)$ are all zero vectors.

So far, the structures of the cDCA and cSCA have been derived. The equivalent received signal vector p can be regarded as the signals of a single snapshot, which needs to be smoothed to increase the rank of the covariance matrix in our method. Here, the method of smoothing the signals on the cDCA, the positive and negative cSCA separately is just like that in [17].

First, the signals on the cDCA are smoothed. In the cDCA, the steering vectors of circular signals and non-circular signals have the same form, then $D_e$ can be expressed as

$$D_e=[d_e({\theta }_1),d_e({\theta }_2),\cdots ,d_e({\theta }_M)]$$

(38)

the m-th steering vector is

$$d_e({\theta }_m)={[1,e^{-j\pi \sin {\theta }_m},e^{-j2\pi \sin {\theta }_m},\cdots ,e^{-j{R}_1\pi \sin {\theta }_m}]}^T$$

(39)

And

$$\begin{array}{ll}{Y}_e& =D_e[p,Gp,\dots ,G^{R_1}p]+{\sigma }_n^2{I}_{R_1+1}\\ & =D_e{S}_d+{\sigma }_n^2{I}_{R_1+1}\end{array}$$

(40)

where

$$G=\left(\begin{array}{cccc}{e}^{j\pi \sin {\theta }_1}& & & \\ & e^{j\pi \sin {\theta }_2}& & \\ & & \ddots & \\ & & & e^{j\pi \sin {\theta }_M}\end{array}\right)$$

(41)

Next, the signals on the positive and negative cSCA are smoothed separately. To ensure that the smoothed matrix has the same number of columns as $Y_e$, the positive and negative cSCA are also divided into $R_1+1$ subarrays, and each subarray contains $R_2-R_1+1$ elements. Then, the constructed equivalent signals can be expressed as

$$y_{z{s}_i}=S_{\delta }{G}^{i-1}p$$

(42)

$$y_{f{s}_i}=S_{\varphi }{G}^{i-1}p$$

(43)

where

$$S_{\delta }=[{\varpi }_1,\cdots ,{\varpi }_{M_1},{\varpi }_{M_1+1},\cdots ,{\varpi }_M]$$

(44)

$$S_{\varphi }=[{\varsigma }_1,\cdots ,{\varsigma }_{M_1},{\varsigma }_{M_1+1},\cdots ,{\varsigma }_M]$$

(45)

$${\varpi }_m={[e^{-j{R}_1\pi \sin {\theta }_m}{e}^{-j2{\phi }_m},e^{-j(R_1+1)\pi \sin {\theta }_m}{e}^{-j2{\phi }_m},\cdots ,e^{-j{R}_2\pi \sin {\theta }_m}{e}^{-j2{\phi }_m}]}^{T}1\le m\le M_1$$

(46)

$${\varpi }_m=O_{(R_2-R_1+1)\times 1}{M}_1+1\le m\le M$$

(47)

$${\varsigma }_m={[e^{j(R_2-R_1)\pi \sin {\theta }_m}{e}^{j2{\phi }_m},\cdots ,e^{j\pi \sin {\theta }_m}{e}^{j2{\phi }_m},e^{j2{\phi }_m}]}^{T}1\le m\le M_1$$

(48)

$${\varsigma }_m=O_{(R_2-R_1+1)\times 1}{M}_1+1\le m\le M$$

(49)

By arranging the equivalent signals on the $R_1+1$ subarrays, the spatially smoothed signal matrices $Y_{\beta }$ and $Y_{\sigma }$ can be rewritten as

$$\begin{array}{l}{Y}_{\beta }=[y_{z{s}_1},y_{z{s}_2},\dots ,y_{z{s}_{R_1+1}}]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=S_{\delta }[p,Gp,\cdots ,G^{R_1}p]=S_{\delta }{S}_d\end{array}$$

(50)

$$\begin{array}{l}{Y}_{\sigma }=[y_{f{s}_1},y_{f{s}_2},\cdots ,y_{f{s}_{R_1+1}}]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=S_{\varphi }[p,Gp,\cdots ,G^{R_1}p]=S_{\varphi }{S}_d\end{array}$$

(51)

$Y_{\beta }$, $Y_e$, and $Y_{\sigma }$ are concatenated as (52) to form a new received signal matrix Y.

$$Y=\left[\begin{array}{l}{Y}_{\sigma }\\ Y_e\\ Y_{\beta }\end{array}\right]=\left[\begin{array}{l}{S}_{\varphi }\\ D_e\\ S_{\delta }\end{array}\right]S_d+\left[\begin{array}{l}{O}_{(R_2-R_1+1)\times (R_1+1)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\sigma }_n^2{I}_{R_1+1}\\ O_{(R_2-R_1+1)\times (R_1+1)}\end{array}\right]$$

(52)

For convenience, we define $A_s=\left[\begin{array}{l}{S}_{\varphi }\\ D_e\\ S_{\delta }\end{array}\right]$ and $N_q=\left[\begin{array}{l}{O}_{(R_2-R_1+1)\times (R_1+1)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\sigma }_n^2{I}_{R_1+1}\\ O_{(R_2-R_1+1)\times (R_1+1)}\end{array}\right]$

$$Y=A_s{S}_d+N_q$$

(53)

where $A_s=[a_{s_1},\cdots ,a_{s_{M_1}},a_{s_{M_1+1}}\cdots ,a_{s_M}]$

$$a_{s_m}=\left[\begin{array}{l}{\varsigma }_m({\theta }_m,{\phi }_m)\\ d_e({\theta }_m)\\ {\varpi }_m({\theta }_m,{\phi }_m)\end{array}\right]1\le m\le M_1$$

(54)

$$a_{s_m}=\left[\begin{array}{l}{O}_{(R_2-R_1+1)\times 1}\\ d_e({\theta }_m)\\ O_{(R_2-R_1+1)\times 1}\end{array}\right]\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{M}_1+1\le m\le M$$

(55)

After these steps, an uneven nested array can be transformed into a ULA, and the number of equivalent elements is significantly increased.

The covariance matrix of Y can be calculated as

$$R_{\chi }=\frac{1}{R_1+1}Y{Y}^H$$

(56)

The eigenvalues decomposition of $R_{\chi }$ can be expressed as

$$R_{\chi }=U_{sc}{\Lambda }_{sc}{U}_{sc}^H+U_{nc}{\Lambda }_{nc}{U}_{nc}^H$$

(57)

${\Lambda }_{nc}=diag\{{\lambda }_{M+1},{\lambda }_{M+2},\cdots {\lambda }_{2R_2-R_1+3}\}$ is a diagonal matrix in which the diagonal elements are the $2R_2-R_1+3-M$ small eigenvalues, and their corresponding eigenvector matrix $U_{nc}$ constitutes the noise subspace.

It can be seen from (54) and (55) that the cSCA parts of the steering vectors of the circular signals are zero vectors while those of the non-circular signals contain the angles and rotation phases information. By exploiting the difference between two types of steering vectors, we can distinguish the two types of signals and estimate their DOAs separately.

3.3.1. DOAs of Circular Signals

When $M_1+1\le m\le M$, $a_{s_m}=\left[\begin{array}{l}{O}_{(R_2-R_1+1)\times 1}\\ d_e({\theta }_m)\\ O_{(R_2-R_1+1)\times 1}\end{array}\right]$. By exploiting the orthogonal property of the signal subspace and noise subspace, the space spectrum $P_1$ can be expressed as

$$P_1(\theta )=\frac{1}{a_{s_m}^H(\theta )U_{nc}{U}_{nc}^H{a}_{s_m}(\theta )}$$

(58)

After the one-dimensional spectral peak search, the angles corresponding to the maximum $M-M_1$ peaks are the DOAs of circular signals. Due to the differences in steering vectors of different types of signals, only the DOAs of circular signals can be estimated in the peak search of (58). Since the cDCA part is contained in the steering vectors of both non-circular and circular signals, the information of circular signals on the cDCA needs to be eliminated after the DOAs of circular signals are estimated.

3.3.2. Eliminate the Information of Circular Signals

Two methods are proposed to eliminate the estimated circular signals based on the total number of signals.

• Reconstruct the covariance matrix

When the total number of circular and non-circular signals does not exceed $N^{\prime }$, the power of the circular signals can be calculated from the covariance matrix $E\{x(t)x^H(t)\}$, and the information of circular signals can be eliminated like the method in [24].

From the previous derivation, the information of the circular signals only exists in $E\{x(t)x^H(t)\}$. Let $R_h$ be the covariance matrix without noise. From the DOAs estimation of circular signals, the power can be calculated as

$${\sigma }_m^2={\{a^H({\theta }_m)R_h^{†}a({\theta }_m)\}}^{-1}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{M}_1+1\le m\le M$$

(59)

Then the covariance matrix $E\{x_N(t)x_N^H(t)\}$ containing only DOAs of non-circular signals is obtained as

$$E\{x_N(t)x_N^H(t)\}=R_h-A_C(\theta )diag\{{\sigma }_{M_1+1}^2,{\sigma }_{M_1+2}^2,\cdots ,{\sigma }_M^2\}{A}_C^H(\theta )$$

(60)

Then, the redundant elements in $E\{x_N(t)x_N^H(t)\}$ are removed. According to the smoothing method in (38)–(41), we have

$$Y_{\epsilon }=[y_{n_1},y_{n_2},\cdots ,y_{n_{R_1+1}}]=D_{\epsilon }[p_n,G_1{p}_n,\dots ,G_1^{R_1}{p}_n]=D_{\epsilon }{S}_n$$

(61)

$$G_1=\left(\begin{array}{cccc}{e}^{j\pi \sin {\theta }_1}& & & \\ & e^{j\pi \sin {\theta }_2}& & \\ & & \ddots & \\ & & & e^{j\pi \sin {\theta }_{M_1}}\end{array}\right)$$

(62)

$$D_{\epsilon }=[\epsilon ({\theta }_1),\epsilon ({\theta }_2),\cdots ,\epsilon ({\theta }_{M_1})]$$

(63)

$p_n={[{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_{M_1}^2]}^T$ represents the power vector of the non-circular signals.

The equivalent signals on the cSCA do not contain the information of circular signals and noise, therefore, it is not necessary to smooth the signals on the cSCA again when calculating the DOAs of non-circular signals.

By splicing $Y_{\sigma }$, $Y_{\epsilon }$ and $Y_{\beta }$ together, we can obtain the equivalent received non-circular signals matrix after removing noise and circular signals information.

$$Y_n=\left[\begin{array}{l}{Y}_{\sigma }\\ Y_{\epsilon }\\ Y_{\beta }\end{array}\right]=A_{\epsilon }{S}_n$$

(64)

where $A_{\epsilon }=[a_{\epsilon }({\theta }_1,{\phi }_1),a_{\epsilon }({\theta }_2,{\phi }_2)\cdots ,a_{\epsilon }({\theta }_{M_1},{\phi }_{M_1})]$ and $a_{\epsilon }({\theta }_m,{\phi }_m)=\left[\begin{array}{l}{\varsigma }_m({\theta }_m,{\phi }_m)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\epsilon ({\theta }_m)\\ {\varpi }_m({\theta }_m,{\phi }_m)\end{array}\right]$

The covariance matrix of the non-circular signals can be calculated as

$$R_{\nu }=\frac{1}{R_1+1}{Y}_n{Y}_n^H$$

(65)

• Oblique projection

The above method is no longer applicable when the number of signals exceeds $N^{\prime }$. Another method is proposed by exploiting the oblique projection method. The proof of the oblique projection method is presented in the Appendix A. The oblique projection operator is constructed as

$${\mathrm{E}}_{A_{SC}(\theta )A_{SN}}=A_{SC}(\theta ){(A_{SC}^H(\theta )R_{\chi }^{†}{A}_{SC}(\theta ))}^{-1}{A}_{SC}^H(\theta )R_{\chi }^{†}$$

(66)

where $A_{SC}(\theta )$ donates the equivalent steering vector matrix of the smoothed circular signals in (53). Then, the covariance matrix is obtained as

$$R_{\nu }=(I-E_{A_{SC}(\theta )A_{SN}})R_{\chi }{(I-E_{A_{SC}(\theta )A_{SN}})}^H$$

(67)

It should be noted that the oblique projection algorithm is also applicable when the total number of signals is no more than $N^{\prime }$.

3.3.3. DOAs of Non-Circular Signals

Through the above derivation, the two methods for eliminating circular signals are proposed, and the covariance matrix $R_{\nu }$ of non-circular signals is obtained. By exploiting the eigenvalue decomposition, $R_{\nu }$ can be expressed as

$$R_{\nu }=U_{sn}{\Lambda }_{sn}{U}_{sn}^H+U_{nn}{\Lambda }_{nn}{U}_{nn}^H$$

(68)

The eigenvectors corresponding to the smaller $2R_2-R_1+3-M_1$ eigenvalues represent the noise subspace. According to the subspace theory, the signal subspace and noise subspace are orthogonal.

$$P_2(\theta ,\phi )=\frac{1}{a_{\epsilon }^H(\theta ,\phi )U_{nn}{U}_{nn}^H{a}_{\epsilon }(\theta ,\phi )}$$

(69)

After performing a 2-D spectral peak search on (69), the DOAs and rotation phases of non-circular signals can be obtained. To reduce the amount of calculation, a method converting the 2-D search into 1-D search is proposed.

$a_{\epsilon }({\theta }_m,{\phi }_m)$ is rewritten as the form of the multiplication of two matrices in (70).

$$a_{\epsilon }({\theta }_m,{\phi }_m)=\left[\begin{array}{c}{e}^{j(R_2-R_1)\pi \sin {\theta }_m}{e}^{i2{\phi }_m}\\ ⋮\\ e^{j\pi \sin {\theta }_m}{e}^{j2{\phi }_m}\\ e^{j2{\phi }_m}\\ 1\\ e^{-j\pi \sin {\theta }_m}\\ ⋮\\ e^{-j{R}_1\pi \sin {\theta }_m}\\ e^{-j{R}_1\pi \sin {\theta }_m}{e}^{-j2{\phi }_m}\\ e^{-j(R_1+1)\pi \sin {\theta }_m}{e}^{-j2{\phi }_m}\\ ⋮\\ e^{-j{R}_2\pi \sin {\theta }_m}{e}^{-j2{\phi }_m}\end{array}\right]=\underset{Q}{\underbrace{\left[\begin{array}{ccc}{e}^{j(R_2-R_1)\pi \sin {\theta }_m}& & \\ ⋮& & \\ e^{j\pi \sin {\theta }_m}& & \\ 1& & \\ & 1& \\ & e^{-j\pi \sin {\theta }_m}& \\ & ⋮& \\ & e^{-j{R}_1\pi \sin {\theta }_m}& \\ & & e^{-j{R}_1\pi \sin {\theta }_m}\\ & & e^{-j(R_1+1)\pi \sin {\theta }_m}\\ & & ⋮\\ & & e^{-j{R}_2\pi \sin {\theta }_m}\end{array}\right]}}\left[\begin{array}{c}{e}^{j2{\phi }_m}\\ 1\\ e^{-j2{\phi }_m}\end{array}\right]$$

(70)

It can be seen from (70) that Q only contains the angle information of the signals, and $\Xi =\left[\begin{array}{l}{\chi }_1\\ {\chi }_2\\ {\chi }_3\end{array}\right]=\left[\begin{array}{c}{e}^{j2{\phi }_m}\\ 1\\ e^{-j2{\phi }_m}\end{array}\right]$ only contains the rotation phase information. From the subspace theory, when both the angle and phase are correct

$$U_{nn}^H{a}_{\epsilon }({\theta }_m,{\phi }_m)=O_{(2R_2-R_1+3-M_1)\times 1}1\le m\le M_1$$

(71)

Let

$$K=U_{nn}^{H}Q$$

(72)

K is considered to be a coefficient matrix and $\Xi$ is the solution vector. Here, the problem can be transformed into the homogeneous equation $K\Xi =O$ existing at least one non-zero solution $\Xi =\left[\begin{array}{c}{e}^{j2{\phi }_m}\\ 1\\ e^{-j2{\phi }_m}\end{array}\right]$, which illustrates that the rank of the coefficient matrix K is less than the number of unknowns. Therefore, the rank estimator method can be used to estimate the DOAs of non-circular signals.

$$rank(K)<3$$

(73)

Since all DOAs are in $[-{90}^{\circ },{90}^{\circ }]$, they can be obtained by searching $\theta$. Because the rank of K changes with different $\theta$, the correct DOAs can obtained when rank of K is deficient. The rotational phases can be further estimated from the singular vectors of K in our method, but they are not considered in many papers [5,14,17] and also in our simulations.

If the traditional 2-D search is applied to (69), it can be summarized as:

$$\begin{gathered} \mathbf{The\; search\; range\; of} \theta \mathbf{is}[-{90}^{\circ },{90}^{\circ }] \\ \mathbf{The\; search\; range\; of} \Phi \mathbf{is}[-{180}^{\circ },{180}^{\circ }] \\ \mathbf{FOR} \theta =-{90}^{\circ },-{90}^{\circ }+\Delta \theta , \cdots { 90}^{\circ } \\ \mathbf{FOR} \varphi =-{180}^{\circ },-{180}^{\circ }+\Delta \varphi , \cdots {180}^{\circ } \\ {P}_2(\theta ,\phi )=\frac{1}{a_{\epsilon }^H(\theta ,\phi )U_{nn}{U}_{nn}^H{a}_{\epsilon }(\theta ,\phi )} \end{gathered}$$

where $\Delta \varphi$ and $\Delta \theta$ denote the search interval of the rotation phase and DOA, respectively.

The proposed 1-D search method in this paper avoids the 2-D search and the computational burden is significantly reduced. The angle search process of the proposed method is summarized as:

$$\begin{gathered} \mathbf{FOR}\theta =-{90}^{\circ },-{90}^{\circ }+\Delta \theta , \cdots { 90}^{\circ } \\ \mathbf{Construct} \mathrm{Q} \mathrm{in} \left(70\right) \\ K=U_{nn}^{H}Q \\ \mathbf{Calculate\; the\; rank\; of\; K} \end{gathered}$$

It can be seen that the proposed method only needs to perform 1-D search for one time in estimating DOAs of non-circular signals and the computational burden is low.

In summary, the main steps of the proposed method can be described as follows:

(1)

Construct matrix J via (13) and calculate covariance matrix $R_J$ via (14).

(2)

Construct $y_d$, $y_{\delta }$, and $y_{\varphi }$ from the covariance matrix and exploit spatial smoothing.

(3)

The equivalent sum-difference co-array steering vectors are constructed by (52), and the DOAs of circular signals are determined by the spectral peak search in (58).

(4)

When the total number of signals does not exceed the actual number of elements $N^{\prime }$, use the method in step $\mathrm{i}$. Otherwise, use the method in step $\mathrm{ii}$.

• The covariance matrix of the reconstructed circular signals is eliminated by (60). After the related elements being extracted and smoothed, the covariance matrix of non-circular signals can be obtained as (65).
• Calculate the oblique projection operator based on the estimated DOAs of the circular signals in (66). Then, the covariance matrix of the non-circular signals can be obtained by (67).

(5)

Perform eigenvalue decomposition on the covariance matrix to obtain the noise subspace. The steering vector of the non-circular signal is decomposed as (70). Then, construct matrix K in (72) and the DOAs of non-circular signals are determined by searching for the rank deficiency of K.

Discussion A: Computational Complexity

For convenience, we define $\Delta \theta$ as the angle search interval and $F$ as the number of virtual array elements. $M_2=M-M_1$ is the number of circular signals. The computational complexity of [17] mainly comes from calculating the inverse of the matrix for both circular and non-circular signals. The overall complexity of this method is $O\left(F^3+F^2(R_1+1)+\frac{\pi }{\Delta \theta }(6F(F-M)+9F+21)\right)$. The complexity of the proposed algorithm mainly comes from the removal of circular signals and the singular value decomposition. If the covariance matrix reconstruction method is used, the complexity is $O\left(2F^3+N^{\prime 3}+M_2(N^{\prime }+2N^{\prime 2})+N^{\prime }{M}_2^2+2F^2(R_1+1)+\frac{\pi }{\Delta \theta }(5F^2-2FM-3F{M}_1-9M_1+10F+9)\right)$. If the oblique projection method is used, the complexity is $O\left(6F^3+M_2^3+2F{M}_2^2+F^2(2M_2+R_1+1)+\frac{\pi }{\Delta \theta }(5F^2-2FM-3F{M}_1-9M_1+10F+9)\right)$.

By estimating different types of signals separately, the complexity is increased, but the accuracy of estimation is also improved. When the different types of signals are estimated simultaneously in our method, the complexity mainly comes from singular value decomposition, which is similar to the complexity in [17].

Discussion B: The Scenario of Unmixed Signals

The proposed method can also work well for unmixed signals. For circular signals, only cDCA part and one parameter $\theta$ exist in this problem, and the traditional MUSIC algorithm can be exploited to estimate the DOA directly through the 1-D spectral peak search. For non-circular signals, the problem in (70) can be transformed into the homogeneous equation existing at least one non-zero solution and the rank of the coefficient matrix K is less than three, which has been discussed in Section 3.3.3. To sum up, the proposed method can be applied to the scenario of unmixed signals, and only the one-stage estimation is needed.

4. Simulation Results

In this section, simulation results are presented to verify the performance of our algorithm where the condition number method is chosen as the indication of the rank deficiency in the estimation of non-circular signals. In this paper, the resolution probability and RMSE are considered. In our simulations, all signals have the same power, and the signal-to-noise ratio (SNR) is defined as

$$SNR=10\lg \left(\frac{{\displaystyle \sum _{i=1}^M{s}_i^2}}{{\displaystyle \sum _{j=1}^{N^{\prime }}{n}_j^2}}\right)$$

(74)

where $s_i^2$ is the power of the i-th signal and $n_j^2$ is the noise power of the j-th sensor.

4.1. Comparison of Resolution Probability

The resolution probability when circular and non-circular signals have similar DOAs is analyzed first. In this simulation, when the angle error is less than $\frac{|{\theta }_n-{\theta }_c|}{2}$, the estimation is considered to be successful [17], where ${\theta }_c$ denotes the DOA of the circular signal and ${\theta }_n$ denotes the DOA of the non-circular one. Figure 4 shows the resolution probability versus the number of snapshots when SNR is −14 dB, where ${\theta }_c={20}^{\circ }$, ${\theta }_n={22}^{\circ }$, the non-circular signal phase $\phi {=5}^{\circ }$, and $N_1=N_2=4$. From simulation results of 500 Monte Carlo experiments, the proposed method has the highest resolution probability and obvious advantages compared with other methods.

When the SNR is 0 dB and the number of snapshots is 200, the resolution probability of different angle intervals is simulated. As shown in Figure 5, when the angle interval is $0{.5}^{\circ }$, the resolution probability of the proposed method is more than 0.4, which is the highest among all methods. With the increase of the angle interval, the proposed method maintains the highest resolution probability.

From the above simulations, the proposed method has the highest resolution probability, and the advantage is more obvious with few snapshots and low SNR. Unlike other methods where all signals are estimated simultaneously, the separate estimation of different types of signals in our method increases the dimensions of the noise subspace in calculating the DOAs of non-circular signals and the two types of signals with similar DOAs can be distinguished. Our method achieves the highest resolution probability through the two-stage estimation.

Next, the resolution probability is simulated when the same types of signals impinge on the array. The resolution probabilities of different methods by estimating the two non-circular signals simultaneously are compared in Figure 6, where ${\theta }_n=\{{20}^{\circ }, {22}^{\circ }\}$ and $\phi {=\{0}^{\circ },{ 5}^{\circ }\}$.

Figure 6 also shows that our method has the best performance, and the highest resolution probability of the proposed method reflects the advantages of searching for the maximum condition number method.

4.2. Comparison of RMSE

In this section, the RMSEs of different methods are compared to analyze the accuracy of the proposed algorithm.

$$RMSE=\sqrt{\frac{1}{VM}\left(\sum _{m=1}^M\sum _{v=1}^V{({\hat{\theta }}_{m,v}-{\theta }_m)}^2\right)}$$

(75)

where V is the simulation times, M is the number of signals, ${\hat{\theta }}_{m,v}$ is the estimated value of the m-th signal angle in the v-th experiment, and ${\theta }_m$ is the theoretical value of the m-th angle.

The performance of RMSE versus SNR and the snapshot number is simulated through 500 Monte Carlo experiments, where the DOAs and rotation phases of sources are shown in

Table 1. Information of the sources.

Source Types	DOA/(Degree)	Rotation Phase/(Degree)
circular	28, 40	0, 0
non-circular	10, 30	0, 5

and $N_1=N_2=4$.

When the number of snapshots is 1000, the RMSE is shown in Figure 7. When the SNR is 0 dB, Figure 8 shows the curve of RMSE versus the number of snapshots. It can be indicated that the proposed method has the best performance.

Unlike the SS-MUSIC method in [14], where only the cDCA is used, the cDCA and cSCA are exploited simultaneously in our method. Furthermore, the circular signals are eliminated before calculating the DOAs of non-circular ones in our method and the RMSE is smaller than estimating all signals together in [14,17]. The small number of snapshots in [18] (QS-MUSIC) leads to poor accuracy. The RMSE of the methods in [2,5] are more than ${10}^{\circ }$. Thus, the proposed method has the best performance.

When only a circular and a non-circular signal have similar DOAs, the proposed method has the minimum RMSE. If there are many signals in different types with similar or the same DOAs, the advantages of our method will be more obvious.

Next, the RMSE is verified in Figure 9 and Figure 10 when the DOAs of circular signals and non-circular signals are distributed uniformly. In this simulation, the source information is shown in

Table 2. Information of the sources.

Source Types	DOA/(Degree)	Rotation Phase/(Degree)
circular	20, 40	0, 0
non-circular	10, 30	0, 5

and $N_1=N_2=4$.

Figure 9 shows the performance of RMSE versus SNR for different methods when the number of snapshots is 800. Figure 10 shows the RMSE when the number of snapshots is changed from 400 to 1200 under the condition of SNR = 0 dB and the other conditions remain unchanged. The RMSE of three methods decreases with the increase of the number of snapshots and our method has the best performance. The dimensions of the noise subspace reach the maximum for non-circular signals in our method, which can improve the accuracy of estimation.

When the angle interval of different types of signals becomes larger, the RMSEs of the different methods are close, because the interference between adjacent impinging signals has been reduced. Of course, in this case, the one-stage angle search without eliminating circular signals can also be exploited in our method with low computational burden.

The relationship between the RMSE and the number of array elements is simulated, and three different cases are compared. The sources information is shown in Table 2. Figure 11 shows the change of RMSE with SNR, and the snapshot number is fixed at 800. It can be seen from the simulation results that when SNR is fixed, the RMSE decreases with the increase of array elements.

To test the performance of the method when the number of signals is more than the actual number of elements of the nested array, further simulations are carried out. When ${\theta }_c{=\{28}^{\circ }\}$ and ${\theta }_n=\{-{40}^{\circ },-{30}^{\circ },-{20}^{\circ },-{10}^{\circ }{, 0}^{\circ }{, 10}^{\circ },30^{\circ },{40}^{\circ }\}$, Figure 12 shows the RMSE of the three methods and $N_1=N_2=4$.

The RMSE of the proposed method is always the smallest and can achieve better accuracy when the number of snapshots and SNR are increased, as all signals can be well distinguished by the two-stage estimation method and the dimensions of the noise subspace are increased by eliminating the circular signals first. In [14,17], the large error makes it unable to distinguish multiple spectral peaks when there are many signals. Even if SNR or the number of snapshots is increased, they cannot obtain the correct estimation of DOAs.

We also analyze the spatial spectrum of the proposed method when a circular signal and a non-circular signal have similar DOAs in Figure 13, where ${\theta }_c={22}^{\circ }$ and ${\theta }_n={20}^{\circ }$. The SNR is −12 dB, the number of snapshots is 500 and $N_1=N_2=4$.

The DOA estimation results of circular and non-circular signals in the proposed method are marked on Figure 13. Unlike other related methods, our method can always distinguish between two spectral peaks with similar DOAs and has high accuracy.

The above methods are comprehensively compared in

Table 3. Comparison of different methods.

Method	The Way of Estimating Two Types of Signals	Whether the Types of Signals Can Be Distinguished	Whether the Rotation Phase Can Be Obtained
The proposed	separately	yes	yes
SD-RD-MUSIC	simultaneously	no	yes
SS-MUSIC	simultaneously	no	no
QS-MUSIC	simultaneously	no	no

and the superiority of the proposed method is illustrated.

The rank deficiency is exploited in calculating the DOAs of non-circular signals; therefore, some different search methods can be evolved. Three representative methods are simulated in Figure 14. Circular and non-circular signals are estimated simultaneously.

Method 1: Find the peak value of the ratio of the maximum singular value to the minimum singular value (the condition number).

Method 2: Find the peak value of the ratio of the sub small singular value to the minimum singular value.

Method 3: Find the peak value of the reciprocal of the minimum singular value.

The variation of RMSE with SNR is analyzed in Figure 14, where the number of snapshots is 800, $N_1=N_2=4$ and the source information is shown in

Table 4. Information of the sources.

Source Types	DOA/(Degree)	Rotation Phase/(Degree)
circular	20, 40	0, 0
non-circular	28, 30	0, 5

. When different non-circular signals have similar DOAs, method 1 and method 2 both have high accuracy. Thus, the method of searching for the maximum condition number is selected to calculate the DOAs in this paper.

5. Conclusions

In this paper, an algorithm is proposed to locate circular and non-circular signals separately with the nested array by exploiting different steering vectors of two types of signals. Only the DOAs of circular signals are estimated in the first stage after the construction of equivalent received signals on the cDCA and cSCA. After eliminating the information of circular signals, the dimensions of noise subspace are increased in estimating DOAs of non-circular signals in the second stage and signals with two types are distinguished. By exploiting the rank deficiency, three ways are proposed to estimate the DOAs of non-circular signals and the method of searching for the maximum condition number is chosen, where the computational burden is significantly reduced by converting 2-D search to 1-D search. Compared with some other methods such as SD-RD-MUSIC and SS-MUSIC, the proposed method has the highest resolution probability and the best estimation accuracy because of the two-stage estimation, especially when two types of sources have similar or the same DOAs. Numerical simulation results demonstrate that the proposed method is more effective and has better performance than other ones.