DOA Estimation for Massive MIMO Systems with Unknown Mutual Coupling Based on Block Sparse Bayesian Learning

Abstract

Obtaining accurate angle parameters using direction-of-arrival (DOA) estimation algorithms is crucial for acquiring channel state information (CSI) in massive multiple-input multiple-output (MIMO) systems. However, the performance of the existing algorithms deteriorates severely due to mutual coupling between antenna elements in practical engineering. Therefore, for solving the array mutual coupling, the array output signal vector is modeled by mutual coupling coefficients and the DOA estimation problem is transformed into block sparse signal reconstruction and parameter optimization in this paper. Then, a novel sparse Bayesian learning (SBL)-based algorithm is proposed, in which the expectation-maximum (EM) algorithm is used to estimate the unknown parameters iteratively, and the convergence speed of the algorithm is enhanced by utilizing the approximate approximation. Moreover, considering the off-grid error caused by discretization processes, the grid refinement is carried out using the polynomial roots to realize the dynamic update of the grid points, so as to improve the DOA estimation accuracy. Simulation results show that compared with the existing algorithms, the proposed algorithm is more robust to mutual coupling and off-grid error and can obtain better estimation performance.

1. Introduction

Massive multiple-input multiple-output (MIMO) technology has been considered one of the most promising technologies for the fifth-generation (5G) and beyond wireless communication networks, because it can obtain a high spatial multiplexing gain and can significantly improve the spatial resolution of array elements [1,2,3]. Accurate angle information obtained through the direction-of-arrival (DOA) estimation plays a crucial role in distinguishing target users from interfering users, reducing pilot pollution, and realizing accurate transmission of effective information in massive MIMO systems [4,5]. Therefore, studies on DOA estimation for massive MIMO systems have important research value and application significance in the field of wireless communications [6,7].

Traditional subspace-based DOA estimation algorithms, such as multiple signal classification (MUSIC) and estimation of signal parameters via rotational invariance techniques (ESPRIT), have been proven to provide high-precision angle estimation, but their computational complexity is relatively high in massive MIMO systems [8,9]. In view of that, numerous improved algorithms have been developed based on these algorithms in recent decades [10,11]. However, as the electromagnetic environment becomes increasingly complex and the direction-finding scenario worsens, these algorithms cause wastage of resources and the severe deterioration of direction-finding performance under low signal-to-noise ratio (SNR), small snapshots, and even large array errors. In addition, these algorithms cannot completely deal with the above-mentioned problems due to the limitation of the subspace framework.

Compressed sensing technology has good applicability and can accurately recover the original signal from low-dimension measurement data using non-zero coefficients [12]. It should be noted that, in compressed sensing technology, the sampling rate is not related to the highest frequency, and a sampling method depends on the sparsity and structure information of a signal. Recently, it has been proposed to use Bayesian compressed sensing [13] to address problems in DOA estimation. Compared with greedy and convex relaxation algorithms, sparse Bayesian learning (SBL) does not require strict information about the number of sources and can accurately reconstruct sparse signals. In view of the two major problems (i.e., small array aperture and large modeling error) existing in the off-grid SBL DOA estimation methods for nested arrays, [14] provided a novel grid refinement method based on a new data formulation to eliminate the modeling error caused by off-grid gaps. In [15,16,17], the authors developed Bayesian learning-based strategies for solving the DOA estimation problem. In [15], it has been proven that SBL can perform remarkably better than traditional beamforming and MUSIC. Since the most frequently used real-valued transformation is suitable for uniform linear arrays only, an innovative real-valued transformation with a lower computational burden was developed in [16], using the virtual steering of linear arrays. In addition, two efficient SBL-based methods for DOA estimation with arbitrary linear arrays were proposed. To diagnose the blocked elements timely under a low SNR, [17] developed an adaptive iterative diagnostic approach based on clustering block SBL, which has a high estimation accuracy in solving the coupling of DOA and antenna blockage estimation.

In practical massive MIMO systems, the performance of conventional DOA estimation algorithms degrades severely or even fails completely due to the mutual coupling between antenna elements [18,19]. The DOA estimation problem for MIMO radar systems with unknown mutual coupling was transformed into a sparse reconstruction problem, in which the off-grid effect and mutual coupling effect are considered simultaneously [20]. Then, a novel sparse Bayesian learning with the mutual coupling (SBLMC) method was proposed. To this end, a $\epsilon$-negative meta-surface superstrate and I-shaped resonators were introduced to reduce mutual coupling [21,22]. Considering that the spatial covariance matrix (SCM), which plays an important role in DOA estimation, cannot be obtained in hybrid massive MIMO systems because the received signals cannot reach the receiver directly, the SCM reconstruction was performed using different algorithms [23,24]. Since random sampling based two-dimensional (2D) DOA estimation in massive MIMO systems can lead to matrix completion, an efficient matrix completion method based on the alternating direction method of multipliers was proposed in [25] to reduce power consumption. In [26], a new higher-order propagator method (HOPM) was designed to estimate the tensor-based signal subspace for 2D DOA estimation, which can achieve high estimation accuracy at low computational complexity. Moreover, a higher-order unitary PM (HOUPM) was proposed to improve the performance further. In [27], a deep-learning-based DOA estimation algorithm was proposed for hybrid massive MIMO systems with a uniform circular array (UCA). This system can realize comparable and, in certain cases, even superior performance at lower computational complexity compared with the maximum likelihood (ML) method. For the 2D DOA and polarization estimation of non-circular signals in polarized massive MIMO systems with uniform rectangular arrays (URAs), an efficient quaternion non-circular MUSIC (QNC-MUSIC) algorithm, which has a high estimation accuracy but low computational complexity, especially for a large source number was proposed in [28]. In [29], a spatial spectrum fusion estimation and localization (SSFEAL) algorithm for massive MIMO systems with UCAs, which can obtain decimeter-level localization accuracy and meet the requirement of future networks, was presented. To solve the two main challenges in Terahertz dynamic array-of-subarray (DAoAS) systems, [30] developed a DAoAS-MUSIC algorithm and a deep convolutional neural network (DCNN)-based algorithm for DOA estimation. Both proposed algorithms can realize super-resolution DOA estimation, and the DCNN-based algorithm can perform better than the DAoAS-MUSIC at a high SNR. In [31], an efficient method for performing the 2D DOA estimation of incoherently distributed sources with gain-phase perturbations was proposed. The proposed method can improve the estimation accuracy efficiently and also is unaffected by the array gain phase disturbance. To overcome the challenges brought by unknown mutual coupling and small samples in accurate DOA estimation for massive MIMO systems, an improved DOA estimation method based on eigenvalue comparison and real-valued MUSIC was proposed in [32]. This method can improve both the performance and the computational efficiency of DOA estimation. It is worth noting that although there have been many studies on DOA estimation algorithms for massive MIMO systems, most of them have ignored the impact of mutual coupling error on DOA estimation performance and have only slightly considered the conditions of low SNR and a small number of snapshots.

To address the shortcomings of the existing work, this paper develops an innovative algorithm based on the block SBL (BSBL) to solve the DOA estimation for massive MIMO systems with unknown array mutual coupling under the conditions of low SNR and small snapshots. Considering the existence of array errors in the actual environment, the array defects are addressed, and the DOA estimation problem is transformed into a block sparse signal reconstruction and parameter optimization problem. Then, structural information of a signal is used, and the cost function of the proposed algorithm is optimized, thus significantly improving the performance of the sparse reconstruction algorithm. The proposed algorithm is verified by simulations, and simulation results demonstrate the excellent performance of the proposed algorithm. The main contributions of this paper can be summarized as follows:

• Considering the array mutual coupling error, which can severely deteriorate the DOA estimation performance, an array output vector is constructed based on the block sparsity of the original signal, where the array flow matrix of the signal is improved by the mutual coupling coefficient. Therefore, the DOA estimation problem is regarded as a block sparse signal reconstruction and parameter optimization problem.
• The expectation-maximization (EM) algorithm is used to estimate hyperparameters and noise parameters to reconstruct the sparse signal. To enhance algorithm convergence speed, the boundary of the cost function is obtained by adopting the approximate approximation, and the hyperparameter representing the signal correlation is optimized, which improves the convergence speed to a large extent.
• Considering the high computational complexity and low estimation accuracy of the equal-spacing angle space division, this study refines the angle space grids by finding the roots of the polynomial to realize the dynamic update of grid points in a discrete space; then a spatial screening of the updated grid is performed to achieve DOA estimation. In addition, the threshold is set to determine the grid points to be updated, thus improving the estimation efficiency. After the updating process, the grid points are closer to the real DOAs, which efficiently eliminates the off-grid error and improves the estimation accuracy. Moreover, an improved SBL-based algorithm (ImBSBL) is proposed for DOA estimation. Simulation results illustrate that compared with the existing algorithms, the proposed algorithm is more robust to mutual coupling and the off-grid error and can obtain better estimation performance.

The rest of this paper is organized as follows. In Section 2, the signal model is introduced. In Section 3, a DOA estimation algorithm for massive MIMO systems based on addressing unknown mutual coupling under a low SNR and small snapshots is introduced. The simulation results are given in Section 4. Finally, the conclusions are drawn in Section 5.

2. System Model

Consider a massive MIMO system with a uniform linear array (ULA) consisting of M antenna elements, receiving K far-field narrowband signals from distinct directions ${\theta }_k$$(\mathit{k}=1,2,\cdots ,\mathit{K})$. From the perspective of spatial sparsity, the whole space can be treated as an angle set ${\left\{{\widehat{\theta }}_i\right\}}_{i=1}^N$ that evenly divides the DOA range $\left[-\frac{\pi }{2},\frac{\pi }{2}\right]$ into N segments, $N(N\gg K)$, where ${\widehat{\theta }}_i$ denotes the possible angle of the incident signal. When the incident signal approaches the array from angle ${\theta }_k={\widehat{\theta }}_i$, the spatial energy at that angle accumulates, forming a significant peak point, while the spatial energy at all other angles is zero. Thus, the array output vector can be expressed by:

$$\mathbf{y}\left(t\right)=\overline{\mathbf{A}}\mathbf{s}\left(t\right)+\mathbf{n}\left(t\right),$$

(1)

where $\mathbf{y}\left(t\right)$ and $\mathbf{n}\left(t\right)$ are the $M\times 1$ observation matrix and observation noise matrix, respectively; $\mathbf{s}\left(t\right)$ is an $N\times 1$ sparse target signal.

This makes it the best represented in the span of the base matrix, $\overline{\mathbf{A}}\in {\mathbb{C}}^{M\times N}$ is the estimation problem to solve. To reconstruct the target signal accurately, the base matrix $\overline{\mathbf{A}}$ is required to be an overcomplete matrix (i.e., $N>M$), and it is defined as follows:

$$\overline{\mathbf{A}}=\left[\mathbf{a}\left({\widehat{\theta }}_1\right),\dots ,\mathbf{a}\left({\widehat{\theta }}_N\right)\right],$$

(2)

where $\mathbf{a}\left({\widehat{\theta }}_i\right)={\left[1,e^{-j2\pi \frac{d}{\lambda }sin{\widehat{\theta }}_i},\dots ,e^{-j2\pi \frac{d}{\lambda }(M-1)sin{\widehat{\theta }}_i}\right]}^{\mathrm{T}}$ and ${[·]}^{\mathrm{T}}$ represents transposition.

Since a unique solution cannot be obtained when $\mathbf{s}\left(t\right)$ is solved according to Equation (1), it is necessary to constrain the signal part of measurement data. In the presence of noise $\mathbf{n}\left(t\right)$ with zero mean and variance ${\sigma }^2$, the DOA estimation problem can be regarded as a ${\ell }_0$-norm optimization problem, which is defined by:

$$\widehat{\mathbf{s}}\left(t\right)=arg\underset{\mathbf{s}}{min}{\parallel \mathbf{s}\parallel }_0\mathrm{s}.\mathrm{t}.\parallel \mathbf{y}-\overline{\mathbf{A}}\mathbf{s}\parallel \le {\sigma }^2.$$

(3)

3. Improved BSBL-Based DOA Estimation Algorithm for Massive MIMO System with Unknown Mutual Coupling

3.1. Mutual Coupling Modeling

Mutual coupling error is common in applications using antenna arrays, and it typically occurs between adjacent elements, particularly when a system operates at a high frequency [16]. The mutual coupling matrix of a ULA can be modeled as a symmetric Toeplitz matrix, which is given by:

$$\mathbf{C}=\left[\begin{array}{cccccc}{c}_1& c_2& \cdots & c_L& \cdots & c_M\\ c_2& c_1& c_2& & \ddots & ⋮\\ ⋮& c_2& c_1& c_2& & c_L\\ c_L& & c_2& \ddots & \ddots & ⋮\\ ⋮& \ddots & & \ddots & c_1& c_2\\ c_M& \cdots & c_L& \cdots & c_2& c_1\end{array}\right],$$

(4)

where $c_i(i=1,2,\dots ,M)$ denotes the mutual coupling coefficient that satisfies the condition of $1=c_1>\cdots >\left|c_L\right|>\left|c_{L+1}\right|=\cdots =\left|c_M\right|=0$; L is the degree of freedom of the mutual coupling matrix, which is unknown in practice. Considering the influence of mutual coupling, Equation (1) can be modified as follows:

$$\mathbf{y}\left(t\right)=\mathbf{C}\overline{\mathbf{A}}\left(t\right)+\mathbf{n}\left(t\right).$$

(5)

Furthermore, to decompose the angle information and mutual coupling coefficient of signal Equation (5), according to the theorem [33], the following equation can be derived:

$$\mathbf{Ca}=\mathbf{Q}\left(\mathbf{a}\right)\mathbf{c},$$

(6)

where $\mathbf{C}$ is a mutual coupling matrix, and $\mathbf{C}=\mathrm{Toeplitz}\left\{\mathbf{c}\right\}\in {\mathbb{C}}^{M\times M}$, where $\mathbf{c}$ is the vector consisting of the first-row elements of $\mathbf{C}$; $\mathbf{a}$ is a complex vector, $\mathbf{a}\in {\mathbb{C}}^{M\times 1}$, which represents the steering vector.

To solve an $M\times L$ transformation matrix $\mathbf{Q}\left(\mathbf{a}\right)$ that can realize the decoupling of position factor and mutual coupling factor, $\mathbf{C}$ is decomposeed into $\mathbf{C}=\mathbf{WZ}$, where $\mathbf{W}=\left[{\mathbf{W}}_1,..,{\mathbf{W}}_L\right]$ and $\mathbf{Z}=\mathbf{c}\otimes \mathbf{I}$ denote $M\times ML$ and $ML\times M$ matrices, respectively; $\mathbf{I}$ represents the identity matrix; ⊗ stands for the Kronecker product.

The $(i,j)$-th element in the l-th subblock of $\mathbf{W}$ can be calculated by:

$${\left[{\mathbf{W}}_l\right]}_{ij}=\left\{\begin{array}{c}1,\mathrm{if}{\left[\mathbf{C}\right]}_{ij}=c_l\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.,$$

(7)

Therefore, Equation (6) can be rewritten as follows:

$$\mathbf{Ca}=\mathbf{W}\mathbf{c}\otimes \mathbf{a}=\mathbf{W}\left[\begin{array}{c}{c}_1\mathbf{a}\\ c_2\mathbf{a}\\ ⋮\\ c_L\mathbf{a}\end{array}\right]=\left[{\mathbf{W}}_1\mathbf{a},{\mathbf{W}}_2\mathbf{a},\dots ,{\mathbf{W}}_L\mathbf{a}\right]\left[\begin{array}{c}{c}_1\\ c_2\\ ⋮\\ c_L\end{array}\right].$$

(8)

According to Equation (6) and Equation (8), $\mathbf{Q}\left(\mathbf{a}\right)$ is a function of the incident signal with $\widehat{\theta }$, so it can be expressed as $\mathbf{Q}\left(\widehat{\theta }\right)$, and then it holds that $\mathbf{Q}\left(\widehat{\theta }\right)=\left[{\mathbf{W}}_1\mathbf{a},{\mathbf{W}}_2\mathbf{a},\dots ,{\mathbf{W}}_L\mathbf{a}\right]$. Thus, Equation (5) can be rewritten as follows:

$$\begin{array}{cc}\hfill \mathbf{y}\left(t\right)& =\mathbf{C}\left[\mathbf{a}\left({\widehat{\theta }}_1\right),\dots ,\mathbf{a}\left({\widehat{\theta }}_N\right)\right]\mathbf{s}\left(t\right)+\mathbf{n}\left(t\right)\hfill \\ \hfill & =\left[\mathbf{Q}\left({\widehat{\theta }}_1\right)\mathbf{c},\dots ,\mathbf{Q}\left({\widehat{\theta }}_N\right)\mathbf{c}\right]\mathbf{s}\left(t\right)+\mathbf{n}\left(t\right)\hfill \\ \hfill & =\mathbf{D}\left(\widehat{\theta }\right)\mathbf{x}\left(t\right)+\mathbf{n}\left(t\right)\hfill \end{array},$$

(9)

where $\mathbf{D}\left(\widehat{\theta }\right)=\left[\mathbf{Q}\left({\widehat{\theta }}_1\right),\dots ,\mathbf{Q}\left({\widehat{\theta }}_N\right)\right]$ is an array flow pattern matrix considering the mutual coupling factor, and the received signal of the array is given by:

$$\mathbf{x}\left(t\right)=\mathbf{s}\left(t\right)\otimes \mathbf{c}=\left[\begin{array}{c}{\mathbf{s}}_1\left(t\right)\mathbf{c}\\ {\mathbf{s}}_2\left(t\right)\mathbf{c}\\ ⋮\\ {\mathbf{s}}_N\left(t\right)\mathbf{c}\end{array}\right].$$

(10)

According to Equation (10), $\mathbf{x}\left(t\right)={\left[{\mathbf{x}}_1^{\mathrm{T}}\left(t\right),\dots ,{\mathbf{x}}_N^{\mathrm{T}}\left(t\right)\right]}^{\mathrm{T}}$ is the signal with a block structure. In addition, since each row in $\mathbf{s}\left(t\right)$ corresponds to a signal in a potential incident direction and the element in a row is non-zero when this direction is the real DOA direction, $\mathbf{s}\left(t\right)$ indicates the row is sparse. Therefore, this paper regards the DOA estimation problem as a problem of block sparse signal reconstruction and parameter optimization.

3.2. Sparse Bayesian Solution

Since block sparsity can reduce the search degree of freedom in the signal space and can improve the reconstruction accuracy and robustness of an algorithm [34], sparse reconstruction algorithms can accurately reconstruct sparse signals. In view of this, this work employs the EM algorithm to reconstruct the sparse signal and realize the DOA estimation for the array output signal model, as shown in Equation (9). For convenience, Equation (9) can be briefly rewritten as follows:

$$\mathbf{y}=\mathbf{D}\mathbf{x}+\mathbf{n},$$

(11)

where $\mathbf{x}$ denotes the block sparse signal.

To avoid over-matching, an SBL method can fully mine and use the prior information on data and constrain the estimated signals, which not only makes the signal more easily meet the sparse condition but also contributes to the high-precision recovery of a sparse reconstruction algorithm [35]. Assuming that the blocks are not correlated with each other; then the block signal $\mathbf{x}={\left[{\mathbf{x}}_1^{\mathrm{T}},\dots {\mathbf{x}}_g^{\mathrm{T}}\right]}^{\mathrm{T}}$ can be modeled as follows:

$$p\left(\mathbf{x}\mid {\gamma }_i,{\mathbf{B}}_i\right)=CN\left(\mathbf{x};\mathbf{0},\mathsf{\Gamma }\right),$$

(12)

where CN denotes the complex Gaussian distribution, ${\gamma }_i$ controls the sparsity of a block, ${\mathbf{B}}_i$ captures the intra-block correlation structure [36], and $\mathsf{\Gamma }={diag}^{-1}\left({\gamma }_1{\mathbf{B}}_1,\dots ,{\gamma }_g{\mathbf{B}}_g\right)$. The observation noise vector $\mathbf{n}$ is assumed to have the distribution of $p\left(n\right)\sim CN\left(0,{\beta }^{-1}\mathbf{I}\right)$, where $\beta$ represents the noise variance. Using the block model Equation (11), the Gaussian likelihood function with mean $\mathbf{D}\mathbf{x}$ and variance ${\beta }^{-1}$ can be obtained as follows:

$$p(\mathbf{y}\mid \mathbf{x};\beta )=CN\left(\mathbf{y};\mathbf{D}\mathbf{x},{\beta }^{-1}\mathbf{I}\right),$$

(13)

where $\mathbf{y}$ is the observation vector, $\mathbf{y}\in {\mathbb{C}}^{M\times 1}$, and it obeys the Gaussian distribution.

The posterior probability can be obtained by Equations (12) and (13) using Gauss identity property as follows:

$$\begin{array}{cc}\hfill & p\left(\mathbf{x}\mid \mathbf{y};\left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)·p\left(\mathbf{y}\mid \left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)\hfill \\ \hfill =& p(\mathbf{y}\mid \mathbf{x},\beta )p\left(\mathbf{x}\mid \left\{{\gamma }_i,{\mathbf{B}}_i\right\}\right)\hfill \\ \hfill =& CN\left(\mathbf{y}\mid \mathbf{D}\mathbf{x},{\beta }^{-1}\mathbf{I}\right)CN(\mathbf{x}\mid 0,\mathsf{\Gamma })\hfill \\ \hfill =& CN(\mathbf{x}\mid \mu ,\mathsf{\Sigma })CN(\mathbf{y}\mid 0,\mathbf{T})\hfill \end{array},$$

(14)

where

$$\mathbf{T}={\beta }^{-1}\mathbf{I}+\mathbf{D}\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}},$$

(15)

$${\mathsf{\Sigma }}^{-1}={\mathsf{\Gamma }}^{-1}+{\mathbf{D}}^{\mathrm{H}}\beta \mathbf{D},$$

(16)

$$\mu =\mathsf{\Sigma }{\mathbf{D}}^{\mathrm{H}}\beta \mathbf{y}$$

(17)

where ${[·]}^{\mathrm{H}}$ represents conjugate transpose.

However, there will occur a large numerical error in calculating ${\mathsf{\Gamma }}^{-1}$ when ${\gamma }_i$ is too small or close to zero; thus, Equation (16) is modified as follows:

$$\mathsf{\Sigma }=\mathsf{\Gamma }-\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}}{\mathbf{T}}^{-1}\mathbf{D}\mathsf{\Gamma }.$$

(18)

Furthermore, to make the calculation of $\mu$ not related to the update of $\mathsf{\Sigma }$, $\mu$ is simplified as follows:

$$\begin{array}{cc}\hfill \mu & =\left({\beta }^{-1}{\mathsf{\Gamma }}^{-1}+{\mathbf{D}}^{\mathrm{H}}\mathbf{D}\right){\mathbf{D}}^{\mathrm{H}}\mathbf{y}\hfill \\ \hfill & ={\mathbf{D}}^{\mathrm{H}}{\left({\beta }^{-1}{\mathsf{\Gamma }}^{-1}+\mathbf{D}{\mathbf{D}}^{\mathrm{H}}\right)}^{-1}\mathbf{y}\hfill \\ \hfill & ={\mathbf{D}}^{\mathrm{H}}\mathsf{\Gamma }{\mathbf{T}}^{-1}\mathbf{y}\hfill \end{array},$$

(19)

where $\mu$ and $\mathsf{\Sigma }$ denote the mean and covariance, respectively, and they are estimated by the EM method.

To find a sparse solution, the global minimum of the cost function [37] is calculated by:

$$\begin{array}{cc}\hfill L\left(\left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)& =-2logp\left(\mathbf{y}\mid \left(\left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)\right)\hfill \\ \hfill & =log\left|\mathbf{T}\right|+{\mathbf{y}}^{\mathrm{H}}{\mathbf{T}}^{-1}\mathbf{y}\hfill \end{array}.$$

(20)

Next, using the Sherman–Morrison matrix identity formula,

$${\left(\mathbf{A}+\mathbf{XB}{\mathbf{X}}^{\mathbf{T}}\right)}^{-1}={\mathbf{A}}^{-1}-{\mathbf{A}}^{-1}\mathbf{X}{\left({\mathbf{B}}^{-1}+{\mathbf{X}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{T}}{\mathbf{A}}^{-1},$$

(21)

$$\left|\mathbf{A}+\mathbf{X}\mathbf{B}{\mathbf{X}}^{\mathrm{T}}\right|=|\mathbf{B}\parallel \mathbf{A}|\left|{\mathbf{B}}^{-1}+{\mathbf{X}}^{\mathrm{T}}{\mathbf{A}}^{-1}\mathbf{X}\right|,$$

(22)

where $\mathbf{A}$ and $\mathbf{B}$ are invertible matrices, Equation (20) can be written as follows:

$$L=\left(\sum _{i=1}^{g}log\left|{\gamma }_i{\mathbf{B}}_i\right|-Nlog\left|\beta \right|+log\left|{\mathsf{\Sigma }}^{-1}\right|\right)+\left({\beta \parallel \mathbf{y}-\mathbf{D}\mathbf{\mu }\parallel }_2^2+{\mathbf{\mu }}^{\mathrm{H}}{\mathsf{\Gamma }}^{-1}\mathbf{\mu }\right).$$

(23)

Then, the solution to paraments ${\gamma }_i$, ${\mathbf{B}}_i$, and $\beta$ is obtained. First, the partial derivative of Equation (23) is given by:

$$\frac{\partial L}{\partial {\gamma }_i}=\frac{d_i}{{\gamma }_i}-Tr\left[\frac{{\mathbf{B}}_i^{-1}\left({\mathbf{\mu }}_i{\mathbf{\mu }}_i^{\mathrm{H}}+{\mathsf{\Sigma }}_i\right)}{{\gamma }_i^2}\right],$$

(24)

$$\frac{\partial L}{\partial {\mathbf{B}}_i}={\mathbf{B}}_i^{-1}-{\gamma }_i^{-1}{\mathbf{B}}_i^{-1}\left[{\mathsf{\Sigma }}_i+{\mathbf{\mu }}_i{\mathbf{\mu }}_i^{\mathrm{H}}\right]{\mathbf{B}}_i^{-1},$$

(25)

$$\frac{\partial L}{\partial \beta }=-\frac{N}{\beta }+Tr\left[\mathsf{\Sigma }{\mathbf{D}}^{\mathrm{H}}\mathbf{D}\right]+{\parallel \mathbf{y}-\mathbf{D}\mathbf{\mu }\parallel }_2^2,$$

(26)

where $Tr[•]$ represents trace, ${\mathbf{\mu }}_i$ denotes the i-th row of $\mathbf{\mu }$, and ${\mathsf{\Sigma }}_i$ denotes the i-th main diagonal element of $\mathsf{\Sigma }$.

Assume that values of Equations (24)–(26) are zeros; then, it can be written as follows:

$${\gamma }_i=\frac{1}{d_i}Tr\left[{\mathbf{B}}_i^{-1}\left({\mathsf{\Sigma }}_i+{\mu }_i{\mathbf{\mu }}_i^{\mathrm{H}}\right)\right],$$

(27)

$${\mathbf{B}}_i=\frac{\left[{\mathsf{\Sigma }}_i+{\mathbf{\mu }}_i{\mathbf{\mu }}_i^{\mathrm{H}}\right]}{{\gamma }_i},$$

(28)

$$\beta =\frac{N}{{\parallel \mathbf{y}-\mathbf{D}\mathbf{\mu }\parallel }^2+Tr\left[\mathsf{\Sigma }{\mathbf{D}}^{\mathrm{H}}\mathbf{D}\right]},$$

(29)

where ${\gamma }_i$ and $\beta$ the parameters that affect the convergence speed of the SBL algorithm.

Compared with the influence of parameter $\beta$ on algorithm performance, parameter ${\gamma }_i$ has a greater influence on algorithm performance; namely, different update approaches of ${\gamma }_i$ affect algorithm performance, altering its convergence speed performance [18]. Moreover, the updated $\beta$ and ${\mathbf{B}}_i$ both affect the reconstruction performance of the sparse recovery algorithm, but the effect of $\beta$ is more significant. If optimal $\beta$ cannot be obtained, the reconstruction and recovery performances of an algorithm cannot be improved even if other parameters are optimized. A number of studies have shown that the parameter $\beta$ in an EM algorithm has poor robustness under a certain SNR because the non-diagonal elements in block elements interfere with the elements on the main diagonal [35]. Parameter $\beta$ is obtained by

$$\beta =\frac{N}{{\parallel \mathbf{y}-\mathbf{D}\mu \parallel }_2^2+{\sum }_{i=1}^{g}Tr\left[{\mathsf{\Sigma }}_i{\mathbf{D}}_i^{\mathrm{H}}{\mathbf{D}}_i\right]}.$$

(30)

It should be noted that ${\mathbf{B}}_i$ affects only the local convergence performance, and the EM algorithm specifies different constraints for different blocks, which can lead to the overfitting problem. Therefore, according to the parameter average idea, so ${\mathbf{B}}_i=\mathbf{B}$, and it holds that:

$$\mathbf{B}=\frac{1}{g}\sum _{i=1}^g\frac{{\mathsf{\Sigma }}_i+{\mathbf{\mu }}_i{\left({\mathbf{\mu }}_i\right)}^{\mathrm{H}}}{{\gamma }_i}.$$

(31)

Since ${\mathbf{B}}_i=\mathbf{c}{\mathbf{c}}^H$ contains the mutual coupling information under unknown mutual coupling, the mutual coupling coefficient can be estimated using the first column of $\mathbf{B}$, and the mutual coupling matrix can be obtained. Finally, when the parameters $\beta$ and ${\left\{{\gamma }_i,{\mathbf{B}}_i\right\}}_i$ are estimated, the Maximum-A-Posteriori (MAP) estimate of $\mathbf{x}$, which is denoted by $\widehat{\mathbf{x}}$, can be obtained by:

$$\widehat{\mathbf{x}}\leftarrow \mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}}{\left({\beta }^{-1}\mathbf{I}+\mathbf{D}\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}}\right)}^{-1}\mathbf{y}.$$

(32)

3.3. Optimized Hyperparameter Updating

In the previous subsection, a sparse Bayesian solution is described in detail, and it is explained that the EM algorithm is applied to obtain the updated parameters and sparse recovery result of a signal. However, convergence speed affects algorithm efficiency significantly. Particularly, in massive MIMO, enhancing the convergence speed can effectively reduce algorithm complexity. Numerous studies have shown that parameter ${\gamma }_i$ affects the convergence speed of an EM algorithm [38,39]. Therefore, this section optimizes parameter ${\gamma }_i$ and then refines the grid by finding roots of polynomials. In view of this, this study proposes an improved ImBSBL algorithm for DOA estimation, whose main aim is to find an alternative method through approximate approximation and obtain the boundary of the cost function.

The following analysis will be conducted based on Equation (20), whose first term is concave while the second term is convex. First, the upper bound of the first term is obtained by:

$$\begin{array}{cc}\hfill log\left|{\beta }^{-1}\mathbf{I}+\mathbf{D}\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}}\right|& \le log\left|{\beta }^{-1}\mathbf{I}+\mathbf{D}{\mathsf{\Gamma }}^{*}{\mathbf{D}}^{\mathrm{H}}\right|\hfill \\ \hfill & +\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right)\left({\gamma }_i-{\gamma }_i^{*}\right)\hfill \\ \hfill & =\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i\hfill \\ \hfill & +log\left|{\mathbf{T}}^{*}\right|-\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i^{*}\hfill \end{array}.$$

(33)

where ${[·]}^{*}$ represents conjugation.

Then, by substituting Equation (33) into Equation (20), the following inequality is obtained:

$$\begin{array}{cc}\hfill L\left(\gamma \right)& \le \sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i+log\left|{\mathbf{T}}^{*}\right|\hfill \\ \hfill & -\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i^{*}\hfill \\ \hfill & +{\mathbf{y}}^{\mathrm{H}}{\left(\beta \mathbf{I}+\mathbf{D}\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{H}}\right)}^{-1}\mathbf{y}\hfill \\ \hfill & =\tilde{L}\left(\gamma \right)\hfill \end{array}.$$

(34)

The abovementioned inequality indicates that $\tilde{L}\left(\gamma \right)$ is a convex function, so its minimum can be obtained at the minimum of the variable $\gamma$ and it should satisfy the condition of $L\left({\gamma }_{min}\right)\le \tilde{L}\left({\gamma }_{min}\right)\le \tilde{L}\left({\gamma }^{*}\right)=L\left({\gamma }^{*}\right)$. Thus, minimizing the right side of Equation (34) will reduce the maximum value of the cost function.

Furthermore, the second term of Equation (20) is given by:

$${\mathbf{y}}^{\mathrm{H}}{\left({\beta }^{-1}\mathbf{I}+\mathbf{D}\mathsf{\Gamma }{\mathbf{D}}^{\mathrm{T}}\right)}^{-1}\mathbf{y}=\underset{\mathrm{x}}{min}\left[\frac{1}{\beta }{\parallel \mathbf{y}-\mathbf{D}\mathbf{x}\parallel }_2^2+{\mathbf{x}}^{\mathrm{H}}{\mathsf{\Gamma }}^{-1}\mathbf{x}\right].$$

(35)

Thus, the substitution function is defined as follows:

$$\begin{array}{cc}\hfill \tilde{L}\left(\gamma \right)& =\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i+log\left|{\mathbf{T}}^{*}\right|\hfill \\ \hfill & -\sum _{i=1}^{g}Tr\left({\mathbf{T}}^{*-1}{\mathbf{D}}_i{\mathbf{B}}_i{\mathbf{D}}_i^{\mathrm{H}}\right){\gamma }_i^{*}\hfill \\ \hfill & +\underset{\mathbf{x}}{min}\frac{1}{\beta }{\parallel \mathbf{y}-\mathsf{\Phi }\mathbf{x}\parallel }_2^2+{\mathbf{x}}^{\mathrm{H}}{\mathsf{\Gamma }}^{-1}\mathbf{x}\hfill \end{array}.$$

(36)

Next, let $\frac{\partial \tilde{L}\left(\gamma \right)}{\partial {\gamma }_i}=0$; then, the update of ${\gamma }_i$ is given by:

$${\gamma }_i\leftarrow \sqrt{\frac{{\mathbf{\mu }}_i^{\mathrm{H}}{\mathbf{B}}_i^{-1}{\mu }_i}{Tr\left[{\mathbf{D}}_i^{\mathrm{H}}{\mathbf{T}}^{-1}{\mathbf{D}}_i{\mathbf{B}}_i\right]}}.$$

(37)

Since the sparse model is divided based on the grid, when the incident angle of a signal is not in the divided grid, there will be an off-grid error in the DOA estimation result. To reduce the influence of the off-grid error on the estimation result, this paper refines the gird using the approach of finding roots of polynomials. To obtain the grid parameter ${\widehat{\theta }}_i(i=1,\dots ,N)$, logarithmic terms irrelevant to formula derivation are ignored, and the following equation is maximized:

$$\begin{array}{c}{E}_{p\left(\mathbf{x}\mid \mathbf{y};\left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)}[logp(\mathbf{y}\mid \mathbf{x};\beta )]\hfill \\ =-\beta E_{p\left(\mathbf{x}\mid \mathbf{y};\left\{{\gamma }_i,{\mathbf{B}}_i\right\},\beta \right)}\left[{\parallel \mathbf{y}-\mathbf{D}\mathbf{x}\parallel }_2^2\right]\hfill \\ ={-\beta \parallel \mathbf{y}-\mathbf{D}\mathbf{\mu }\parallel }_2^2-\beta Tr\left[\mathbf{D}\mathsf{\Sigma }{\mathbf{D}}^{\mathrm{H}}\right]\hfill \end{array}.$$

(38)

Then, to refine ${\widehat{\theta }}_i$, let the partial derivative of Equation (38) with respect to the steering vector ${\mathcal{V}}_{{\widehat{\theta }}_i}=e^{-j2\pi d/\lambda sin{\widehat{\theta }}_i}$ be zero, which can be expressed by:

$$\frac{{\partial \parallel \mathbf{y}-\mathbf{D}\mu \parallel }_2^2}{\partial {\mathcal{V}}_{{\widehat{\theta }}_i}}+\frac{\partial Tr\left[\mathbf{D}\mathsf{\Sigma }{\mathbf{D}}^{\mathrm{H}}\right]}{\partial {\mathcal{V}}_{{\widehat{\theta }}_i}}=0.$$

(39)

According to the idea of a polynomial root, Equation (39) can be expressed in the form of a polynomial with respect to ${\widehat{\theta }}_i$ using algebraic manipulation. Since Equation (39) contains the first derivative of the direction vector with respect to ${\mathcal{V}}_{{\widehat{\theta }}_i}$, its polynomial form has $M-1$ roots in the complex field, and roots of the refined grid parameters should have unit amplitude. However, the roots cannot fall in the unit circle exactly due to the noise in a real environment. Assuming that ${\mathcal{V}}_{{\widehat{\theta }}_i^{*}}$ denotes the root closest to the unit circle, then, it can be written as:

$${\widehat{\theta }}_{i^{*}}^{\mathrm{new}}=arcsin\left(-\frac{\lambda }{2\pi d}·angle\left({\mathcal{V}}_{{\widehat{\theta }}_i^{*}}\right)\right).$$

(40)

Thus, the parameters for refining the grid are obtained. Considering that grid refinement for real DOA, which is close to or equal to the initial grid point, will increase the estimation error of the algorithm, a threshold should be defined to judge whether to perform grid refinement. The initial grid needs to be refined by ${\widehat{\theta }}_{i^{*}}^{new}$ when ${\widehat{\theta }}_{i^{*}}^{\mathrm{new}}\in \left[\frac{{\widehat{\theta }}_{i^{*}-1}+{\widehat{\theta }}_{i^{*}}}{2},\frac{{\widehat{\theta }}_{i^{*}}+{\widehat{\theta }}_{i^{*}+1}}{2}\right]$; otherwise, the refinement is not necessary. Since it is not effective to refine all grid points in each iteration, a threshold $\eta$ is set, and the Frobenius norm is used to select the first $\eta (1\le \eta \le M)$ largest elements of $\mu$ and obtain their indexes, which are used to determine the grid to be updated [40]. In this study, the number of sources is set to $\eta =M$.

For clarity, the proposed ImBSBL algorithm for DOA estimation with unknown mutual coupling is summarized in Algorithm 1. It is worth noting that this research does not involve the synchronization of complex dynamic networks. Different from [14], our work not only takes the off-grid factor into account, but also deals with the influence of array mutual coupling on DOA estimation. In addition, compared with [20], the proposed algorithm improves the convergence speed of the EM algorithm when estimating parameters by approximate approximation.

Algorithm 1: The Proposed ImBSBL Algorithm for DOA Estimation with Unknown Mutual Coupling and Off-grid Error.

1: Input: Received signal $\mathbf{x}$; 2: Initialization: Initialize the number of iterations $\kappa$ and parameters ${\gamma }^{\left(\kappa \right)}$, ${\mathbf{B}}^{\left(\kappa \right)}$, and ${\beta }^{\left(\kappa \right)}$; 3: while not converged do 4:    Calculate ${\mathbf{\mu }}^{(\kappa +1)}$ and ${\mathsf{\Sigma }}^{(\kappa +1)}$ using Equation (19) and Equation (18), respectively. 5:    Update ${\gamma }^{(\kappa +1)}$, ${\mathbf{B}}^{(\kappa +1)}$, ${\beta }^{(\kappa +1)}$ according to Equation (37), Equation (31) and Equation (30), respectively; 6:    Update ${\widehat{\theta }}_{i^{*}}^{(\kappa +1)}$ by Equation (40); 7:    if $∥{\gamma }^{(\kappa +1)}-{\gamma }^{\left(k\right)}∥/∥{\gamma }^{\left(k\right)}∥<{10}^{-4}$ or $\kappa =1000$ then 8:      break 9:    else 10:      Set $\kappa =\kappa +1$; 11:      Return to Step 4. 12:    end if 13: end while 14: Output:${\mathbf{\mu }}^{(\kappa +1)}$ and ${\widehat{\theta }}_{i^{*}}^{(\kappa +1)}$. 15: Perform one dimensional spectrum search on new ${\widehat{\theta }}_{i^{*}}^{(\kappa +1)}$ to find the K maximum values to perform the DOA estimation.

4. Simulation Results

Several experiments were performed to illustrate the superiority of the proposed algorithm over the existing algorithms, including the root SBL (ROSBL) algorithm based on root finding [40], the classical high-resolution high-precision MUSIC algorithm [41], the multi-resolution focal underdetermined system solver (MFOCUSS) algorithm [42] in the convex optimization algorithm, and the sparse Bayesian array calibration (SBAC) algorithm based on the signal observation error model [43].

4.1. Effect of Mutual Coupling on DOA Estimation

The mutual coupling relationship between the elements was described by [44]

$$c_n=\left\{\begin{array}{c}(1+\xi )e^{j\varphi }10\frac{{\alpha }_c(1+0.5n)}{20},n<5\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right.,$$

(41)

where $\xi \sim u\left(\right[-0.05,0.05\left]\right)$, $\phi \sim u\left(\right[0,2\pi \left]\right)$ and ${\alpha }_c$ are used to measure the mutual coupling effect between adjacent elements. A ULA with $M=16$ elements is used to receive $K=2$ far-field signals arriving at the incident angles of $\theta =0^{\circ }$ and $\theta ={25}^{\circ }$. The SNR is set to 5 dB, the number of snapshots is set to 100, and the grid interval is $1^{\circ }$. The spatial spectrum of the MFOCUSS, MUSIC, OMP, and ImBSBL algorithms are presented in Figure 1, where it can be observed that the proposed ImBSBL algorithm has two sharp spectral peaks, which are close to the actual incident positions. Compared with the OMP algorithm, the MFOCUSS algorithm is more accurate but produces many smaller spectral peaks. The MUSIC algorithm performs the worst among all algorithms, having no sharp spectral peak, and the peak is far away from the real position. The reason for these results is that, among the compared algorithms, only the proposed ImBSBL algorithm considers the mutual coupling factor and off−grid error, which could severely deteriorate the DOA estimation performance. Therefore, its estimation performance is optimal.

The incident angle are set to different values, including $\theta =-{15}^{\circ }$, $\theta =-{10}^{\circ }$, $\theta =0^{\circ }$, $\theta ={10}^{\circ }$ and $\theta ={15}^{\circ }$, while the other parameters are set as defined above. The DOA estimation spatial spectrum of the MUSIC algorithm in the presence of mutual coupling and in an ideal environment obtained by 100 Monte−Carlo runs is presented in Figure 2. The performances of the BSBL algorithm and the proposed ImBSBL algorithm in an ideal environment and in the case of mutual coupling are presented in Figure 3 and Figure 4, respectively. As shown in Figure 2, the conventional MUSIC algorithm could accurately estimate the real incident angles of signals from multiple sources in an ideal environment. However, the mutual coupling has a destructive effect and severely affects the resolution and estimation accuracy of the MUSIC algorithm. Furthermore, as presented in Figure 3, although the BSBL algorithm could accurately estimate the real incident angles of signals in the ideal environment, its performance significantly degrades in the presence of mutual coupling. Compared with the BSBL algorithm, the proposed ImBSBL algorithm is more robust to mutual coupling, and could accurately estimate the true incident angles of signals from multiple sources with a lower side lobe and smaller power leakage without pseudo peaks. This is because the mutual coupling factors are decoupled by utilizing the symmetric Toeplitz property and the banded structure of the mutual coupling matrix. Furthermore, the dynamic update of grid points by exploiting the polynomial roots can also improve the DOA estimation precision.

The comparison results regarding the running time and root mean squared error (RMSE) performance of the BSBL and the proposed ImBSBL algorithms are presented in Figure 5. As shown in Figure 5, the running times of both algorithms increase substantially with the number of antennas. However, since the parameters related to the convergence speed are optimized, the proposed ImBSBL algorithm has a faster convergence speed, especially when the antenna number is large. Moreover, as shown in Figure 5b, the RMSE of the proposed ImBSBL algorithm is slightly higher than that of the BSBL algorithm when the SNR is less than 5 dB. However, the two curves overlap when the SNR is higher than 5 dB. Thus, the proposed algorithm is more suitable than the BSBL algorithm for practice applications with a large number of antennas.

4.2. Rmse Performance Analysis

The RMSE results of different DOA estimation algorithms versus the input SNR under different snapshot are presented in Figure 6. The results are obtained under the incident angles of the signals of $-11.3^{\circ }$ and $3.7^{\circ }$ and the number of snapshots of $T=30$, $T=100$, and $T=800$. The SNR varies from 0 to 25 dB, while the other settings are the same as in Section 4.1 As shown in Figure 6, the RMSE performance is significantly improved with the increase in snapshot number for all algorithms; in addition, the proposed ImBSBL algorithm achieves higher estimation accuracy performance than other algorithms. In particular, the proposed algorithm could maintain excellent estimation performance even under low SNR and fewer snapshots. This can be explained by the fact that the block sparsity is used to model the signals, which can improve the reconstruction accuracy and robustness to noise. Moreover, the grid points are dynamically updated by finding the polynomial roots that can reduce the impact of the off-grid error on the estimation accuracy, coupling on estimation results. Since the ROSBL and MFOCUSS algorithms are not robust to mutual coupling, their performances are relatively poor. Although the SBAC algorithm could suppress array defects, its performance is inferior to that of the proposed algorithm because it is based on the point sparse model.

The RMSE results of the algorithms versus the snapshots number are presented in Figure 7. In this experiment, the snapshot number changes from 100 to 1100, the number of antennas is 64 and the other settings are the same as in Section 4.1. The results indicate that the average estimation accuracy of all algorithms increases monotonically with the snapshot number. When the number of snapshots is smaller than 220 and 250, the SBAC algorithm and the ROSBL algorithm fail to separate two signals, respectively, and their DOA estimation precisions are still rougher than that of the proposed ImBSBL algorithm.

The RMSE performance of the algorithms versus the antenna number under different SNR values are presented in Figure 8. In this experiment, the number of snapshots is 1000, the grid interval is $0.5^{\circ }$, the SNR is set to 0 dB and 10 dB, and the other settings are the same as in Section 4.1. When the SNR is fixed, the estimation performance of all methods improves significantly with the increase in antenna number. However, the SBAC algorithm and the proposed ImBSBL algorithm have smaller estimation errors than the ROSBL and MFOCUSS methods. This is because the SBAC and ImBSBL algorithms are robust to the effect of mutual coupling on the estimation results, so their performances are relatively good.

However, since the SBAC algorithm uses the point sparsity of signals for modeling, its performance is slightly worse than that of the ImBSBL algorithm, exploiting the block sparsity of signals for modeling. Thus, the proposed algorithm is suitable for large-scale MIMO systems. In addition, the estimation performances of all methods are improved when the SNR increases, but the proposed algorithm achieves better estimation performance than the other algorithms.

The RMSE performances of the algorithms versus the angle separation are presented in Figure 9. In this experiment, DOAs of two signals are set to ${\theta }_1=-11.3^{\circ }$ and ${\theta }_2={\theta }_1+\mathsf{\Delta }{\theta }^{\circ }$, $\mathsf{\Delta }{\theta }^{\circ }$ varies from $4^{\circ }$ to ${14}^{\circ }$, and the number of antennas is 64; the other settings are the same as those in Section 4.1. As shown in Figure 9, there is an RMSE performance gap between the SBAC and ROSBL algorithms, and HEIR estimation accuracies are much lower than those of the other two algorithms; however, the other two algorithms’ performances are significantly reduced when the angle interval is larger than $8^{\circ }$. In addition, regardless of the angle interval value, the proposed ImBSBL algorithm performs better than the MFOCUSS algorithm. This could be explained by the fact that the proposed algorithm is robust to mutual coupling and off-grid errors and explicitly use the sparsity structure of incident signals.

In Figure 10, the RMSE results of the proposed ImBSBL algorithm obtained by 500 independent Monte−Carlo experiments under different conditions are presented. The RMSE results versus the SNR under different snapshots numbers and array element configurations are presented in Figure 10a,b, respectively. In Figure 10c, the RMSE results versus snapshots number under different SNR are displayed. As shown in Figure 10a,c, when the SNR or snapshot number is fixed, the RMSE value of the proposed algorithm gradually decreases with the snapshots number or SNR, respectively. However, the proposed ImBSBL algorithm could maintain good estimation performance even under a small snapshot number and a low SNR. Moreover, as shown in Figure 10b, the estimation performance of the proposed algorithm could be improved by increasing the number of antennas. Consequently, the proposed algorithm achieves higher estimation accuracy under low SNR and fewer snapshots than the other algorithms; thus, it is more suitable for massive MIMO systems compared with the other algorithms.

5. Conclusions

This paper studies the DOA estimation problem in massive MIMO systems with unknown mutual coupling and off-grid error. Considering the mutual coupling between antennas, the studied problem can be regarded as a problem of block sparse signal reconstruction and parameter optimization. To solve this problem, this paper proposes an improved block sparse Bayesian learning algorithm to estimation the target DOAs. In the proposed algorithm, the cost function boundary is determined by exploiting the approximate approximation. Then, the parameters affecting the convergence speed are optimized to improve the convergence speed of the EM algorithm. Moreover, the effect of the off-grid error on the estimation results of the EM algorithm is considered, and dynamic updating of grid points is realized by finding the polynomial roots. Simulation results confirmed that the proposed algorithm is more robust to mutual coupling and off-grid error compared with the existing algorithms and can significantly outperform the existing DOA estimation algorithms.