Nested Variational Chain and Its Application in Massive MIMO Detection for High-Order Constellations

Abstract

Multiple input multiple output (MIMO) technology necessitates detection methods with high performance and low complexity; however, the detection problem becomes severe when high-order constellations are employed. Variational approximation-based algorithms prove to deal with this problem efficiently, especially for high-order MIMO systems. Two typical algorithms named Gaussian tree approximation (GTA) and expectation consistency (EC) attempt to approximate the true likelihood function under discrete finite-set constraints with a new distribution by minimizing the Kullback–Leibler (KL) divergence. As the KL divergence is not a true distance measure, ’exclusive’ and ’inclusive’ KL divergences are utilized by GTA and EC, respctively, demonstrating different performances. In this paper, we further combine the two asymmetric KL divergences in a nested way by proposing a generic algorithm framework named nested variational chain. Acting as an initial application, a MIMO detection algorithm named Gaussian tree approximation expectation consistency (GTA-EC) can thus be presented along with its alternative version for better understanding. With less computational burden compared to its counterparts, GTA-EC is able to provide better detection performance and diversity gain, especially for large-scale high-order MIMO systems.

1. Introduction

Multiple input multiple output (MIMO) technology has attracted broad attention over the last decade and has been widely applied into practical communication systems. The benefit of MIMO technology lies in the improvement of spectral efficiency and link reliability due to the multiplexing and diversity gain that grows with the number of elements, and a MIMO system is referred to as a massive MIMO system when the scale of array elements grows large enough, which brings increasing difficulty to the signal detection due to huge computational burden, hindering the prevailing usage of massive MIMO systems [1,2].

Many research studies have been carried out for signal detection in massive MIMO systems [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17]. It is well known that the maximum likelihood detection presents the best detection performance with the cost of exponentially growing computational burden [3]. Neglecting the finite-set constraint, the minimum mean square error (MMSE) approach can be applied by solving the least square fit, and a closest lattice point can then be found by treating symbols independently [4]. The MMSE approach normally exhibits a benchmark performance when comparing different detectors, and its performance can be vastly improved by MMSE-SIC when combining with the successive interference cancellation (SIC) technique [5]. However, as MMSE or MMSE-SIC cannot provide satisfied performance, several alternatives have been proposed instead, which can be divided into two major categories, i.e., sub-space searching-based and variational inference-based detectors.

The sub-space searching-based category originates from the idea of reducing the searching space of all possible lattice points with unacceptable complexity. Sphere decoding tries to replicate the maximum likelihood performance by diminishing the searching space, the dimension of which grows up with the number of antennas as well as the modulation order, making it prohibitive for the large-scale or high-order MIMO systems [6,7]. Another two local searching-based approaches were proposed by the name of likelihood ascending search and reactive tabu search [8,9,10], and the basic idea behind them is to search through a proximity sub-space around a given initial solution. They present good performance for a large number of antennas with low-order constellations but poor performance for high-order constellations. A layered tabu search algorithm was proposed in [11] by performing detection over layers, requiring a higher order of complexity for high-order constellations, and a Gibbs sampling-based detector was proposed in [12] by performing a serial of one-dimensional searches over iterations. It may provide good performance for low-order constellations with the cost of enormous processing time. Therefore, algorithms in the former category suffer from poor performance or prohibitive computational burden in large-scale high-order MIMO systems.

Proved to be suitable for the detection problem resulted by high-order constellations, the variational inference based category tries to approximate the true likelihood function into a new distribution that is much easier to handle. A Gaussian tree approximation (GTA) algorithm was proposed in [13,14] by transforming the fully connected factor graph into a tree graph, based on which belief propagation based message passing can be proceeded for inference. The GTA algorithm has comparable performance with MMSE-SIC at a similar complexity only to MMSE. The expectation propagation (EP) algorithm was proposed for MIMO detection in [15] by substituting true priors belonging to a discrete finite set with the introduced Gaussian priors being able to be updated over iterations. EP performs the best at a complexity several times that of MMSE, and its alternative named expectation consistency (EC) was then proposed to provide a more general perspective than EP [16]. Two low-complexity EP/EC-based algorithms were proposed for scenarios when the number of transmit antennas is less than that the number of receiver ones [17,18], and a double-EP based iterative detection and decoding was proposed by iteratively exploiting decoders in [19]. EP/EC-based algorithms can also be applied into channel estimation problems in massive MIMO systems [20,21].

In this paper, we would like to expand the variational inference paradigm by proposing a nested variational chain. The basic idea behind it is that ’exclusive’ and ’inclusive’ KL divergences employed by GTA and EC, respectively, are not exclusive and can be combined in a nested way so as to form an approximation chain, by which both GTA and EC are improved. The major contributions are listed as follows.

• Firstly, the basic idea of the nested variational chain is proposed, and an algorithm is then proposed to establish a general framework. By referring to ’general’, it means this framework is able to combine ’exclusive’ and ’inclusive’ KL divergences, or it degrades to either one as a special case.
• Secondly, providing several examples, we show that existing algorithms, such as MMSE, GTA, and EC, can be regarded as special cases of the variational chain.
• Finally, to provide an initial application of the variational chain into massive MIMO detection, a GTA-embedded Expecatation Consistency (GTA-EC) algorithm is proposed which proves to provide better detection performance, especially for high-order constellations. The complexity of GTA-EC is analyzed as well along with comparisons.

This paper is arranged as follows. Section 2 introduces the system model and MIMO detection problem, based on which the nested variational chain is provided in Section 3 along with a generic algorithm framework. Section 4 derives the GTA-EC algorithm with complexity analyses. Simulation results are demonstrated in Section 5 along with discussions, and the conclusion is presented in Section 6. Throughout this paper, matrices and vectors are denoted by symbols in boldface, and variables are denoted in italics. The notation ${\mathbf{A}}^{\top }$ or ${\mathbf{a}}^{\top }$ is used to represent the transpose of a vector or matrix, and $\mathbf{I}$ represents a unit matrix.

2. Preliminary

2.1. Signal Model

A multiuser MIMO system is considered, without loss of generality, in which $\tilde{N}$ transmitters, each equipped with one antenna, communicate with a base station that is equipped with $\tilde{M}$ antennas. Assume each transmitter transmits any symbol ${\tilde{x}}_i\in \mathbb{C}$ that is selected from a Quadrature Amplitude Modulation (QAM) constellation set $\tilde{\mathcal{A}}$, where $\mathbb{C}$ stands for the complex domain, and the cardinality of the constellation set is $|\tilde{\mathcal{A}}|=A$. The transmitted symbols can be represented as a vector $\tilde{\mathbf{x}}\in {\tilde{\mathcal{A}}}^{\tilde{N}\times 1}$ with the average energy of each QAM symbol defined as ${\tilde{E}}_s$. After propagating through the wireless channels, the received signal $\tilde{\mathbf{y}}\in {\mathbb{C}}^{\tilde{M}\times 1}$ at the base station can be expressed as

$$\begin{array}{c}\hfill \tilde{\mathbf{y}}=\tilde{\mathbf{H}}\tilde{\mathbf{x}}+\tilde{\mathbf{n}},\end{array}$$

(1)

where $\tilde{\mathbf{n}}\in {\mathbb{C}}^{M\times 1}$ stands for additive Gaussian white noises (AWGN), each element having zero mean and ${\sigma }_n^2$ variance. $\tilde{\mathbf{H}}\triangleq [{\tilde{\mathbf{h}}}_1,\dots ,{\tilde{\mathbf{h}}}_i,\dots ,{\tilde{\mathbf{h}}}_N]$ is defined as a matrix by stacking up channel coefficients with ${\tilde{\mathbf{h}}}_i=\mathrm{diag}\{{\tilde{h}}_{i,1},\dots ,{\tilde{h}}_{i,m},\dots ,{\tilde{h}}_{i,M}\}$ being Rayleigh flat-fading channel coefficients of the $i^{}$th symbol. Perfect channel state information (CSI) is assumed such that $\tilde{\mathbf{H}}$ is known at the base station.

The channel model above is usually re-expressed in the real domain by taking into consideration real and imaginary parts, respectively. By defining $\mathcal{R}(·)$ and $\mathcal{I}(·)$ as operations to take the real and imaginary part of a variable or matrix, one can define $\mathbf{y}={\left[\mathcal{R}{\left(\tilde{\mathbf{y}}\right)}^{\top }\mathcal{I}{\left(\tilde{\mathbf{y}}\right)}^{\top }\right]}^{\top }$, $\mathbf{x}={\left[\mathcal{R}{\left(\tilde{\mathbf{x}}\right)}^{\top }\mathcal{I}{\left(\tilde{\mathbf{x}}\right)}^{\top }\right]}^{\top }$, $\mathbf{n}={\left[\mathcal{R}{\left(\tilde{\mathbf{n}}\right)}^{\top }\mathcal{I}{\left(\tilde{\mathbf{n}}\right)}^{\top }\right]}^{\top }$, and that

$$\begin{array}{c}\hfill \mathbf{H}=\left[\begin{array}{cc}\mathcal{R}\left(\tilde{\mathbf{H}}\right)& -\mathcal{I}\left(\tilde{\mathbf{H}}\right)\\ \mathcal{I}\left(\tilde{\mathbf{H}}\right)& \mathcal{R}\left(\tilde{\mathbf{H}}\right)\end{array}\right],\end{array}$$

where $\mathbf{y},\mathbf{n}\in {\mathbb{R}}^{M\times 1}$, $\mathbf{x}\in {\mathbb{R}}^{N\times 1}$, $\mathbf{H}\in {\mathbb{R}}^{M\times N}$, $M=2\tilde{M}$, $N=2\tilde{N}$, and $\mathbb{R}$ stands for the real domain. The equivalent model in the real domain is then given as

$$\begin{array}{c}\hfill \mathbf{y}=\mathbf{H}\mathbf{x}+\mathbf{n},\end{array}$$

(2)

where the variance of each element of $\mathbf{n}$ equals ${\sigma }_n^2={\tilde{\sigma }}_n^2/2$, and $\mathbf{x}$ belongs to the pulse amplitude modulation (PAM) constellation set $\mathcal{A}$ containing real and imaginary parts of the A-QAM alphabets with its cardinality being $\left|\mathcal{A}\right|=\sqrt{A}$. The average energy of a PAM symbol is $E_s={\tilde{E}}_s/2$, and the signal-to-noise (SNR) of the MIMO system is then defined as

$$\begin{array}{c}\hfill \mathrm{SNR}=10·{\log }_{10}\left(\frac{N{\tilde{E}}_s}{{\tilde{\sigma }}_n^2}\right)=10·{\log }_{10}\left(\frac{N{E}_s}{{\sigma }_n^2}\right)\end{array}$$

2.2. MIMO Detection

As the received signal in a MIMO system is a superposition of transmitted symbols weighted by channel coefficients, the purpose of MIMO detection is to estimate successfully all transmitted symbols impaired by channel fading and noises. As is well known, the maximum a posteriori (MAP) detector could achieve the best detection performance by maximizing the a posteriori probability as follows

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}=\underset{\mathbf{x}\in \mathcal{A}}{argmax}\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right),\end{array}$$

where the $aposteriori$ distribution given the received signal $\mathbf{y}$ and CSI $\mathbf{H}$ is expressed as

$$\begin{array}{cc}\hfill \mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)& \propto \mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\mathcal{P}\left(\mathbf{x}\right),\hfill \end{array}$$

(3)

$\mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)$ is defined as Gaussian distribution with a mean vector of $\mathbf{Hx}$ and a covariance matrix of ${\sigma }_n^2\mathbf{I}$, and $\mathcal{P}\left(\mathbf{x}\right)$ is defined as the a priori probability of symbols. When $\mathcal{P}\left(\mathbf{x}\right)={\prod }_{i=1}^N\frac{1}{\sqrt{A}}{\mathbb{I}}_{x_i\in \mathcal{A}}$ is uniformly distributed with ${\mathbb{I}}_{x_i\in \mathcal{A}}$ being an indication function that takes value one if $x_i\in \mathcal{A}$ and zero otherwise, the MAP detection degrades into the maximum likelihood detection, i.e.,

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}=\underset{\mathbf{x}}{argmax}\mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\prod _{i=1}^N{\mathbb{I}}_{x_i\in \mathcal{A}}\end{array}$$

The complexity of maximum likelihood detection grows up exponentially with the number of symbols N, making it prohibitive for middle- or large-scale MIMO systems with especially high-order constellations.

3. Nested Variational Chain

In order to perform low-complexity MIMO detection, one popular approach is to approximate the true posterior with another distribution that is much simpler to perform inference on, and the KL divergence is commonly used to obtain the desired distribution. Defining $\mathcal{Q}\left(\mathbf{x}\right)$ as the distribution utilized to approximate the true posterior, the minimization of the ’exclusive’ and ’inclusive’ KL divergences can be expressed as

$$\begin{array}{cc}\hfill \mathcal{Q}\left(\mathbf{x}\right)& =\underset{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left({\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)\left|\right|\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)\right)\hfill \\ \hfill & =\underset{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{argmin}{\int }_{\mathbf{x}}{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)log\frac{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)},\hfill \end{array}$$

(4)

and

$$\begin{array}{cc}\hfill \mathcal{Q}\left(\mathbf{x}\right)& =\underset{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left(\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)\left|\right|{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)\right)\hfill \\ \hfill & =\underset{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{argmin}{\int }_{\mathbf{x}}\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)log\frac{\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)}{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}.\hfill \end{array}$$

(5)

For instance, the GTA algorithm takes the former way, while the EC algorithm takes the latter one. However, with only one approximation, GTA is unable to update its approximated tree structure, while EC only treats symbols in an independent way rather than exploiting correlation among symbles. In this case, we then would like to demonstrate that the two KL divergences could be combined together, and a nested variational chain is then proposed in what follows.

Suppose there is a desired variational distribution $\mathcal{G}\left(\mathbf{x}\right)$ that can be obtained with ’exclusive’ KL divergengce:

$$\begin{array}{c}\hfill \mathcal{G}\left(\mathbf{x}\right)=\underset{{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left(\mathcal{Q}\left(\mathbf{x}\right)\left|\right|{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\right),\end{array}$$

(6)

which is embedded in an optimization for $\mathcal{Q}\left(\mathbf{x}\right)$ with $\mathcal{Q}\left(\mathbf{x}\right)$ obtained in the first place as

$$\begin{array}{c}\hfill \mathcal{Q}\left(\mathbf{x}\right)=\underset{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left(\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)\left|\right|{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)\right).\end{array}$$

(7)

The processing above actually forms a variational chain with a nested structure given as

$$\begin{array}{c}\hfill \mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)\Rightarrow \mathcal{Q}\left(\mathbf{x}\right)\Rightarrow \mathcal{G}\left(\mathbf{x}\right),\end{array}$$

(8)

indicating $\mathcal{Q}\left(\mathbf{x}\right)$ should be obtained according to the minimization of $\mathrm{KL}\left(\mathcal{P}\left(\mathbf{x}|\mathbf{y},\mathbf{H}\right)\left|\right|{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)\right)$ with respect to ${\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)$, and that the desired $\mathcal{G}\left(\mathbf{x}\right)$ could then be obtained by minimizing $\mathrm{KL}\left(\mathcal{Q}\left(\mathbf{x}\right)\left|\right|{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\right)$ with respect to ${\mathcal{G}}^{\prime }\left(\mathbf{x}\right)$. Following the roadmap, we may derive an algorithm for the nested variational chain combining the two asymetric KL divergences.

3.1. A Generic Framework for Nested Variational Chain

To begin with, a general statistical model should first be defined as follows [12],

$$\begin{array}{c}\hfill \mathcal{P}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right)\mathbf{\prod }_i{t}_i\left(\mathbf{x}\right),\end{array}$$

(9)

where $\mathcal{F}\left(\mathbf{x}\right)$ is a function belonging to the exponential family, and $t_i\left(\mathbf{x}\right)$ for $i=1,\dots ,I$ are non-negative factors. Normally, it is intractable or prohibitive complex to perform inference over $\mathcal{P}\left(\mathbf{x}\right)$ such that the variational inference-based approaches provide another distribution $\mathcal{Q}\left(\mathbf{x}\right)$ that is tractable or easy to handle. The nested variational chain consists of four steps: factor substitution, inner approximation, symbol detection and factor updating.

As for factor substitution, the optimization of the KL divergence should first be achieved, i.e., $\mathcal{Q}\left(\mathbf{x}\right)={argmin}_{{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)}\mathrm{KL}\left(\mathcal{P}\left(\mathbf{x}\right)\left|\right|{\mathcal{Q}}^{\prime }\left(\mathbf{x}\right)\right)$, for which the EC framework can be employed. The EC algorithm assumes a distribution that belongs to the exponential family:

$$\begin{array}{c}\hfill \mathcal{Q}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right)\prod _i{\tilde{t}}_i\left(\mathbf{x}\right),\end{array}$$

(10)

where ${\tilde{t}}_i\left(\mathbf{x}\right)$ instead of $t_i\left(\mathbf{x}\right)$ for $i=1,\dots ,I$ are modified factors, belonging to the exponential family as well.

It should be noticed that the EC framework replaces each non-negative factor $t_i\left(\mathbf{x}\right)$ by another ${\tilde{t}}_i\left(\mathbf{x}\right)$ in the exponential family. However, the distribution $\mathcal{F}\left(\mathbf{x}\right)$ remains constant during this optimization process, which may be further exploited. Based on this idea, another variational distribution could be embedded inside as an inner approximation so as to achieve a final distribution $\mathcal{G}\left(\mathbf{x}\right)$, and the optimization in (6) could be performed:

$$\begin{array}{cc}\hfill \mathcal{G}\left(\mathbf{x}\right)& =\underset{{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left({\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\left|\right|\mathcal{Q}\left(\mathbf{x}\right)\right)\hfill \\ \hfill & =\underset{{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left({\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\left|\right|\mathcal{F}\left(\mathbf{x}\right)\prod _i{\tilde{t}}_i\left(\mathbf{x}\right)\right).\hfill \end{array}$$

(11)

The approximation is normally expressed as $\mathcal{G}\left(\mathbf{x}\right)={\prod }_j{G}_j\left(\mathbf{x}\right)$. When $G_j\left(\mathbf{x}\right)$ for $j=1,\dots ,J$ are defined as disjoint groups, it is mean-field approximation, and structured approximation can be employed when $G_j\left(\mathbf{x}\right)$ for $j=1,\dots ,J$ are overlapped with each other.

Toward symbol detection, a cavity distribution for each factor can then be acquired as

$$\begin{array}{c}\hfill {\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right)=\frac{\mathcal{G}\left(\mathbf{x}\right)}{{\tilde{t}}_i\left(\mathbf{x}\right)}\end{array}$$

(12)

and the final distribution can be represented as

$$\begin{array}{c}\hfill {\tilde{\mathcal{G}}}_i\left(\mathbf{x}\right)\propto \prod _{i=1}^N\underset{p_i\left(\mathbf{x}\right)}{\underbrace{{\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right)t_i\left(\mathbf{x}\right)}},\end{array}$$

(13)

with $p_i\left(\mathbf{x}\right)\triangleq {\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right)t_i\left(\mathbf{x}\right)$ defined as a new distribution by attaching the true factor.

The moments of $p_i\left(\mathbf{x}\right)$ are then obtained by exploiting the true distribution $t_i\left(\mathbf{x}\right)$ as ${\mathbb{E}}_{p_i\left(\mathbf{x}\right)}\left[\varphi \left(\mathbf{x}\right)\right]$, where $\varphi \left(\mathbf{x}\right)$ stands for the sufficient statistics of the exponential family. A new factor ${\tilde{t}}_i^{new}\left(\mathbf{x}\right)$ is updated as well by satisfying the moment-matching condition ${\mathbb{E}}_{p_i\left(\mathbf{x}\right)}\left[\varphi \left(\mathbf{x}\right)\right]={\mathbb{E}}_{q_i\left(\mathbf{x}\right)}\left[\varphi \left(\mathbf{x}\right)\right]$ with

$$\begin{array}{c}\hfill q_i\left(\mathbf{x}\right)\propto {\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right){\tilde{t}}_i^{new}\left(\mathbf{x}\right),\end{array}$$

(14)

such that the distribution $\mathcal{Q}\left(\mathbf{x}\right)$ is able to be updated iteratively.

An algorithm is provided in Algorithm 1, which is used to approximate a statistical model $\mathcal{P}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right){\prod }_i{t}_i\left(\mathbf{x}\right)$. Associating with the four steps described above, the algorithm first substites all factors in Step 1 with much easier accessible ones by using the ’inclusive’ KL divergence, as seen in the EC algorithm. After that, the algorithm further approximates $\mathcal{Q}\left(\mathbf{x}\right)$ with a new distribution in Step 2 such that better detection performance is expected. With detection proceeded on the new distribution, moment matching can be achieved in Step 3 so as to update substituted factors in Step 4. Note that any of the steps, such as factor substitution, inner approximation, or factor updating, may be skipped for a certain purpose so as to form a special case. In the next subsection, we would like to demonstrate that the MMSE, GTA and EC algorithms could be deemed as special cases.

Algorithm 1 An algorithm for nested variational chain

Require: A statistical model $\mathcal{P}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right){\prod }_i{t}_i\left(\mathbf{x}\right)$. Ensure: repeat (1) Step 1: Factor Substitution. Substitute each non-negative factor $t_i\left(\mathbf{x}\right)$ with ${\tilde{t}}_i\left(\mathbf{x}\right)$ for $i=1,\dots ,I$ such that $\mathcal{Q}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right){\prod }_i{\tilde{t}}_i\left(\mathbf{x}\right)$. (2) Step 2: Inner Approximation. Obtain a new distribution to approximate $\mathcal{Q}\left(\mathbf{x}\right)$ as $$\begin{array}{c}\hfill \mathcal{G}\left(\mathbf{x}\right)=\underset{{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left({\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\left|\right|\mathcal{Q}\left(\mathbf{x}\right)\right)\end{array}$$ (3) Step 3: Symbol Detection. for $i\in \left[1,\dots ,I\right]$ do Obtain a cavity distribution as ${\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right)=\mathcal{G}\left(\mathbf{x}\right){\tilde{t}}_i\left(\mathbf{x}\right)$, and then achieve moment matching between $p_i\left(\mathbf{x}\right)\propto {\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right)t_i\left(\mathbf{x}\right)$ and $q_i\left(\mathbf{x}\right)\propto {\mathcal{G}}^{\setminus i}\left(\mathbf{x}\right){\tilde{t}}_i^{new}\left(\mathbf{x}\right)$. end for (4) Step 4: Factor Updating. Substitute ${\tilde{t}}_i\left(\mathbf{x}\right)$ with ${\tilde{t}}_i^{new}\left(\mathbf{x}\right)$ into $\mathcal{Q}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right){\prod }_i{\tilde{t}}_i\left(\mathbf{x}\right)$ and repeat this procedure if necessary. until Convergence is achieved. Output: Detection results on the approximated distribution.

3.2. MMSE, GTA and EC MIMO Detectors as Special Cases

In a MIMO system, the distribution $\mathcal{F}\left(\mathbf{x}\right)$ can be expressed as the likelihood function, i.e., $\mathcal{F}\left(\mathbf{x}\right)\propto \mathcal{N}(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I})$, and each non-negative factor $t_i\left(\mathbf{x}\right)$ for $i=1,\dots ,I$ could be regarded as the $apriori$ probability with respect to symbols. When a factor $t_i\left(\mathbf{x}\right)$ corresponds only to one symbol $x_i$, it reduces to $t_i\left(x_i\right)=\mathcal{P}\left(x_i\right)=\frac{1}{\sqrt{A}}{\mathbb{I}}_{x_i\in \mathcal{A}}$. Hence, as there are N symbols in a MIMO system, there would be N factors or priors as well, and the expression for the substituted factor ${\tilde{t}}_i\left(x_i\right)$ for $i=1,\dots ,N$ depends on any specific algorithm.

(1) Minimum Mean Square Error  

The MMSE approach could be obtained by assuming that each non-negative factor $t_i\left(x_i\right)$ for $i=1,\dots ,N$ can be replaced by a Gaussian distributed factor ${\tilde{t}}_i\left(x_i\right)=\mathcal{N}\left(x_i:0,E_s\right)$ of zero-mean and a variance of $E_s$, and the modified distribution with factor substitution for MMSE is then given as

$$\begin{array}{c}\hfill {\mathcal{Q}}_{MMSE}\left(\mathbf{x}\right)\propto \mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\prod _{i=1}^N\mathcal{N}\left(x_i:0,E_s\right),\end{array}$$

(15)

whose second-order and first-order moments are derived as

$$\left\{\begin{array}{cc}\hfill & {\mathsf{\Sigma }}_{MMSE}={\left({\mathbf{H}}^{\top }\mathbf{H}+\frac{{\sigma }_n^2}{E_s}\mathbf{I}\right)}^{-1}\hfill \\ \hfill & {\mathit{\mu }}_{MMSE}={\mathsf{\Sigma }}_{MMSE}{\mathbf{H}}^{\top }\mathbf{y}\hfill \end{array}\right.$$

Not mentioned though before, there is actually a simple inner approximation for MMSE to approximate the distribution ${\mathcal{Q}}_{MMSE}\left(\mathbf{x}\right)$. With a fully factorized distribution ${\mathcal{G}}_{MMSE}\left(\mathbf{x}\right)={\prod }_i^N{G}_{MMSE}\left(x_i\right)$, each factorized one can be obtained as

$$\begin{array}{cc}\hfill G_{MMSE}\left(x_i\right)& \propto exp\left({\langle ln{\mathcal{Q}}_{MMSE}\left(\mathbf{x}\right)\rangle }_{\sim G_{MMSE}\left(x_i\right)}\right)\hfill \\ \hfill & \propto \mathcal{N}\left(x_i:{\mu }_{i,MMSE},{\mathrm{\Sigma }}_{i,MMSE}\right),\hfill \end{array}$$

(16)

which is known as the mean-field approximation. The expression ${\langle ·\rangle }_{\sim G_{MMSE}\left(x_i\right)}$ refers to expectation with respect to all factors $G_{MMSE}\left(x_j\right)$ for $j=1,\dots ,N$ except for $G_{MMSE}\left(x_i\right)$. This process is equivalent to marginalization of ${\mathcal{Q}}_{\mathrm{MMSE}}\left(\mathbf{x}\right)$ with ${\mu }_{i,MMSE}$ and ${\mathrm{\Sigma }}_{i,MMSE}$ being the $i^{th}$ element of ${\mu }_{\mathrm{MMSE}}$ and of the diagonal of ${\mathrm{\Sigma }}_{MMSE}$.

The MMSE approach skips factor updating, but instead it may output directly the hard detection results. The final distribution of MMSE is expressed as

$$\begin{array}{c}\hfill {\tilde{\mathcal{G}}}_{MMSE}\left(\mathbf{x}\right)\propto \prod _{i=1}^N\underset{p_{MMSE,i}\left(x_i\right)}{\underbrace{G_{MMSE}\left(x_i\right)t_i\left(x_i\right)}},\end{array}$$

(17)

where $p_{MMSE,i}\left(x_i\right)\triangleq G_{MMSE}\left(x_i\right)t_i\left(x_i\right)$ is defined as a new distribution by attaching true priors, based on which symbol detection can be proceeded for each symbol independently.

(2) Expectation Consistency  

The EC algorithm defines a substitution factor for each symbol as well. It replaces the prior $t_i\left(x_i\right)=\frac{1}{\sqrt{A}}{\mathbb{I}}_{x_i\in \mathcal{A}}$ to ${\tilde{t}}_i\left(x_i\right)\propto e^{{\gamma }_i{x}_i-\frac{1}{2}{\mathrm{\Lambda }}_i{x}_i^2}$ so the posterior can be expressed as

$$\begin{array}{cc}\hfill {\mathcal{Q}}_{EC}\left(\mathbf{x}\right)& \propto \mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\prod _{i=1}^N{e}^{{\gamma }_i{x}_i-\frac{1}{2}{\mathrm{\Lambda }}_i{x}_i^2}\hfill \end{array}$$

(18)

Note that ${\tilde{t}}_i\left(x_i\right)\propto e^{{\gamma }_i{x}_i-\frac{1}{2}{\mathrm{\Lambda }}_i{x}_i^2}$ is Gaussian distributed. In this regard, it can be noticed that EC relates essentially to MMSE with the difference that it is able to update priors. The second-order and first-order moments of ${\mathcal{Q}}_{\mathrm{EC}}\left(\mathbf{x}\right)$ are derived as

$$\left\{\begin{array}{cc}\hfill & {\mathsf{\Sigma }}_{EC}={\left({\sigma }_n^{-2}{\mathbf{H}}^{\top }\mathbf{H}+\mathsf{\Lambda }\right)}^{-1}\hfill \\ \hfill & {\mathit{\mu }}_{EC}={\mathsf{\Sigma }}_{EC}\left({\mathbf{H}}^{\top }\mathbf{y}+\gamma \right)\hfill \end{array}\right.$$

where $\mathsf{\Lambda }$ is a diagonal matrix containing ${\mathrm{\Lambda }}_i$, and $\gamma$ is a vector containing ${\gamma }_i$ for $i=1,\dots ,N$.

The EC algorithm employs mean-field approximation for inner approximation as well, by which the fully factorized distribution is defined as ${\mathcal{G}}_{EC}\left(\mathbf{x}\right)={\prod }_i^N{G}_{EC}\left(x_i\right)$, and each factorized distribution $G_{EC}\left(x_i\right)$ is Gaussian distributed such that:

$$\begin{array}{cc}\hfill G_{EC}\left(x_i\right)& \propto exp\left({\langle ln{\mathcal{Q}}_{EC}\left(\mathbf{x}\right)\rangle }_{\sim G_{EC}\left(x_i\right)}\right)\hfill \\ \hfill & \propto \mathcal{N}\left(x_i:{\mu }_{i,EC},{\mathrm{\Sigma }}_{i,EC}\right),\hfill \end{array}$$

(19)

with ${\mu }_{i,EC}$ and ${\mathrm{\Sigma }}_{i,EC}$ being the $i^{th}$ element of ${\mu }_{EC}$ and of the diagonal of ${\mathrm{\Sigma }}_{EC}$, respectively. By doing so, factor updating is then operated with a cavity distribution:

$$\begin{array}{c}\hfill G_{EC}^{\setminus i}\left(x_i\right)=\frac{G_{EC}\left(x_i\right)}{{\tilde{t}}_i\left(x_i\right)},\end{array}$$

(20)

and the final distribution for EC is represented as

$$\begin{array}{c}\hfill {\tilde{\mathcal{G}}}_{EC}\left(\mathbf{x}\right)\propto \prod _{i=1}^N\underset{p_{EC,i}\left(x_i\right)}{\underbrace{G_{EC}^{\setminus i}\left(x_i\right)t_i\left(x_i\right)}},\end{array}$$

(21)

where $p_{EC,i}\left(x_i\right)\triangleq G_{EC}^{\setminus i}\left(x_i\right)t_i\left(x_i\right)$ is defined as a new distribution. Symbol detection can then be performed to achieve the moment-matching condition so the pairs $\left({\gamma }_i,{\mathrm{\Lambda }}_i\right)$ for $i=1,\dots ,N$ are updated in parallel.

(3) Gaussian Tree Approximation  

The GTA algorithm was proposed based on the modified distribution of MMSE, and its distribution with substituted factors can be represented as

$$\begin{array}{cc}\hfill {\mathcal{Q}}_{GTA}\left(\mathbf{x}\right)& ={\mathcal{Q}}_{MMSE}\left(\mathbf{x}\right)\propto \mathcal{F}\left(\mathbf{x}\right)\prod _{i=1}^N{\tilde{t}}_i\left(x_i\right)\hfill \\ \hfill & \propto \mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\prod _{i=1}^N\mathcal{N}\left(x_i:0,E_s\right).\hfill \end{array}$$

(22)

As for inner approximation, the GTA algorithm chooses to optimally approximate the distribution with a tree graph, which can be constructed based on ${\mathcal{Q}}_{GTA}\left(\mathbf{x}\right)$ as

$$\begin{array}{cc}\hfill {\mathcal{G}}_{GTA}\left(\mathbf{x}\right)& =\underset{{\mathcal{G}}^{\prime }\left(\mathbf{x}\right)}{argmin}\mathrm{KL}\left({\mathcal{G}}^{\prime }\left(\mathbf{x}\right)\left|\right|{\mathcal{Q}}_{GTA}\left(\mathbf{x}\right)\right)\hfill \\ \hfill & =\prod _i{G}_{GTA}\left(x_i|x_{pa\left(i\right)}\right),\hfill \end{array}$$

(23)

where $G_{GTA}\left(x_i\right|x_{pa\left(i\right)})$ stands for the conditional probability of $x_i$ given its parent $x_{pa\left(i\right)}$, and $G_{GTA}\left(x_i\right|x_{pa\left(i\right)})=G_{GTA}\left(x_i\right)$ in case that $x_i$ is the root of the tree.

This leads to a result that GTA skips factor updating as well, similar to MMSE, and the performance of the GTA algorithm is subject to the fixed initial distribution ${\mathcal{Q}}_{GTA}\left(\mathbf{x}\right)$ that is not able to be updated. In this case, by directly attaching the true priors $t_i\left(x_i\right)=\frac{1}{\sqrt{A}}{\mathbb{I}}_{x_i\in \mathcal{A}}$ for $i=1,\dots ,N$, the final distribution of GTA is then represented as

$$\begin{array}{cc}\hfill {\tilde{\mathcal{G}}}_{GTA}\left(\mathbf{x}\right)& \propto \prod _{i=1}^N{G}_{GTA}\left(x_i|x_{pa\left(i\right)}\right)\prod _{i=1}^N{t}_i\left(x_i\right)\hfill \\ \hfill & =\prod _{i=1}^N\underset{p_{GTA,i}\left(x_i\right)}{\underbrace{G_{GTA}\left(x_i|x_{pa\left(i\right)}\right)t_i\left(x_i\right)}}.\hfill \end{array}$$

(24)

where $p_{GTA,i}\left(x_i\right)\triangleq G_{GTA}\left(x_i\right)t_i\left(x_i\right)$. Proceeding on such a loop-free tree graph, message passing can then be utilized to perform efficient detection during all but one iteration.

4. Applications into MIMO High-Order Detection

Introducing the nested variational chain for MIMO detection, it can be seen that all existing approaches employ factor substitution. As for inner approximation, MMSE and EC actually perform mean-field approximation with fully factorized distribution, while GTA performs the maximum spanning tree approximation. Finally, only EC performs factor updating, while MMSE and GTA choose to perform direct detection.

This analysis puts forward the question of whether any improvement can be achieved when one enables GTA to update its substituted factors or whether any better inner approximation can be derived for EC rather than being fully factorized. Both thoughts lead us to an idea that it is worth trying to update the GTA factors iteratively since the approximated Gaussian tree is capable of capturing correlation among symbols rather than keeping independence among them. Following this idea, an initial application of the nested variational chain can be performed. By utilizing EC as an outer approximation, an algorithm named GTA-embedded EC (GTA-EC) is proposed in the following.

4.1. The GTA-EC Algorithm

Given $t_i\left(x_i\right)\propto {\mathbb{I}}_{x_i\in \mathcal{A}}$ for $i=1,\dots ,N$, the algorithm starts from the likelihood function with discrete priors as in (3), i.e.,

$$\begin{array}{c}\hfill {\mathcal{Q}}_{GTA-EC}\left(\mathbf{x}\right)\propto \mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right)\prod _{i=1}^N{\mathbb{I}}_{x_i\in \mathcal{A}},\end{array}$$

(25)

which could be divided into two parts, i.e.,

$$\begin{array}{cc}\hfill & f_q\left(\mathbf{x}\right)=\mathcal{N}\left(\mathbf{y}:\mathbf{Hx},{\sigma }_n^2\mathbf{I}\right),\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & f_r\left(\mathbf{x}\right)=\prod _{i=1}^N{\mathbb{I}}_{x_i\in \mathcal{A}}\hfill \end{array}$$

(27)

It is then possible to define a new distribution $q\left(\mathbf{x}\right)$ as

$$\begin{array}{c}\hfill q\left(\mathbf{x}\right)\propto f_q\left(\mathbf{x}\right)exp\left({\gamma }_q^{\top }\mathbf{x}-\frac{{\mathbf{x}}^{\top }{\mathsf{\Lambda }}_q\mathbf{x}}{2}\right),\end{array}$$

(28)

of which the moments can be expressed as

$$\left\{\begin{array}{cc}\hfill & {\mathsf{\Sigma }}_q={\left({\mathbf{H}}^{\top }\mathbf{H}+{\mathsf{\Lambda }}_q\right)}^{-1}\hfill \\ \hfill & {\mathit{\mu }}_q={\mathsf{\Sigma }}_q\left({\sigma }_n^{-2}{\mathbf{H}}^{\top }\mathbf{y}+{\gamma }_q\right)\hfill \end{array}\right.$$

Note that the pair $({\gamma }_q,{\mathrm{\Sigma }}_q)$ acts as priors of all symbols to be updated, and that the definition of $q\left(\mathbf{x}\right)$ actually serves as factor substitution.

To achieve moment consistency, another distribution $s\left(\mathbf{x}\right)$ is then defined as

$$s\left(\mathbf{x}\right)\propto exp\left({\gamma }_s^{\top }\mathbf{x}-\frac{{\mathbf{x}}^{\top }{\mathsf{\Lambda }}_s\mathbf{x}}{2}\right),$$

(29)

where moment matching between $s\left(\mathbf{x}\right)$ and $q\left(\mathbf{x}\right)$ should be achieved so as to obtain ${\gamma }_s$ and ${\mathsf{\Lambda }}_s$. The EC algorithm assumes another distribution:

$$\begin{array}{cc}\hfill r\left(\mathbf{x}\right)& \propto exp\left({\gamma }_r^{\top }\mathbf{x}-\frac{{\mathbf{x}}^{\top }{\mathrm{\Sigma }}_r\mathbf{x}}{2}\right)f_r\left(\mathbf{x}\right),\hfill \end{array}$$

(30)

with moments derived as

$$\left\{\begin{array}{cc}\hfill & {\gamma }_r={\gamma }_s-{\gamma }_q\hfill \\ \hfill & {\mathsf{\Lambda }}_r={\mathsf{\Lambda }}_s-{\mathsf{\Lambda }}_q\hfill \end{array}\right.$$

It can be observed that $exp\left({\gamma }_r^{\top }\mathbf{x}-\frac{{\mathbf{x}}^{\top }{\mathrm{\Sigma }}_r\mathbf{x}}{2}\right)$ partly in $r\left(\mathbf{x}\right)$ actually serves as a cavity distribution of symbols by subtracting their substituted priors $({\gamma }_q,{\mathrm{\Sigma }}_q)$.

The next step involves inner approximation. Since fully factorization for $r\left(\mathbf{x}\right)$ neglects correlation among symbols, we instead propose utilizing the Gaussian approximation tree to perform detection according to the moments ${\mu }_r$, ${\mathrm{\Sigma }}_r$, ${\mu }_q$, and ${\mathrm{\Sigma }}_q$. This is because the Gaussian approximation tree may capture correlation among symbols rather than treating them independently. In this case, we define a new Gaussian tree-based distribution $g\left(\mathbf{x}\right)$ rather than $r\left(\mathbf{x}\right)$ as

$$\begin{array}{cc}\hfill g\left(\mathbf{x}\right)& \propto f_r\left(\mathbf{x}\right)\prod _{i=1}^N{G}^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right)=\prod _{i=1}^N\underset{p_{i\left|pa\right(i)}^{\prime }\left(x_i\right)}{\underbrace{G^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right){\mathbb{I}}_{x_i\in \mathcal{A}}}}\hfill \end{array}$$

(31)

where $p_{i\left|pa\right(i)}^{\prime }\triangleq G^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right){\mathbb{I}}_{x_i\in \mathcal{A}}$ is a new distribution by attaching true priors, and the conditional distribution can be represented as

$$\begin{array}{cc}\hfill & G^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right)\propto exp\left\{-\frac{1}{2}\frac{{\left[(x_i-{\mu }_i^r)-\frac{{\mathrm{\Sigma }}_{i,pa\left(i\right)}}{{\mathrm{\Sigma }}_{pa\left(i\right),pa\left(i\right)}}(x_{pa\left(i\right)}-{\mu }_{pa\left(i\right)})\right]}^2}{{\mathrm{\Sigma }}_{i,i}^r-\frac{{\mathrm{\Sigma }}_{i,pa\left(i\right)}}{{\mathrm{\Sigma }}_{pa\left(i\right),pa\left(i\right)}}}\right\}\hfill \end{array}$$

(32)

where ${\mu }_i^r$ and ${\mathrm{\Sigma }}_{i,i}^r$ for $i=1,\dots ,N$ are taken from ${\mu }_r$ and the diagonal of ${\mathrm{\Sigma }}_r$, respectively, while ${\mu }_i$ and ${\mathrm{\Sigma }}_{i,i}$ for $i=1,\dots ,N$ are taken from ${\mu }_q$ and the diagonal of ${\mathrm{\Sigma }}_q$.

Based on $g\left(\mathbf{x}\right)$, message passing on the Gaussian tree can then be proceeded:

$$\begin{array}{c}\hfill {\mathcal{M}}_{i\to pa\left(i\right)}\left(x_{pa\left(i\right)}\right)=\sum _{\sim x_i}\underset{p_{i\left|pa\right(i)}\left(x_i\right)}{\underbrace{G^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right)t_i\left(x_i\right)}}\prod _{j\left|pa\right(j)=i}{\mathcal{M}}_{j\to i}\left(x_i\right)\end{array}$$

(33)

and

$$\begin{array}{cc}\hfill {\mathcal{M}}_{pa\left(i\right)\to i}\left(x_i\right)& =\sum _{\sim x_{pa\left(i\right)}}\underset{p_{i\left|pa\right(i)}\left(x_i\right)}{\underbrace{G^{\setminus i}\left(x_i|x_{pa\left(i\right)}\right)t_i\left(x_i\right)}}\hfill \\ \hfill & \times \prod _{j|j\ne i,pa\left(j\right)=pa\left(i\right)}{\mathcal{M}}_{j\to p\left(i\right)}\left(x_{pa\left(i\right)}\right){\mathcal{M}}_{pa\left(pa\right(i\left)\right)\to pa\left(i\right)}\left(x_{pa\left(i\right)}\right)\hfill \end{array}$$

(34)

To achieve consistency, the distribution $s\left(\mathbf{x}\right)$ is finally utilized once again to achieve moment matching between $g\left(\mathbf{x}\right)$ and $s\left(\mathbf{x}\right)$ so as to obtain ${\gamma }_s$ and ${\mathrm{\Sigma }}_s$, and the a priori moments can be updated:

$$\left\{\begin{array}{cc}\hfill & {\mathsf{\Lambda }}_q^{new}=\beta \left({\mathsf{\Lambda }}_s-{\mathsf{\Lambda }}_r\right)+\left(1-\beta \right){\mathsf{\Lambda }}_q\hfill \\ \hfill & {\gamma }_q^{new}=\beta \left({\gamma }_s-{\gamma }_r\right)+\left(1-\beta \right){\gamma }_q\hfill \end{array}\right.$$

(35)

The GTA-EC algorithm is concluded and depicted in detail in Algorithm 2. In step 1, the GTA-EC algorithm initiliazes the distribution $q\left(\mathbf{x}\right)$, which behaves as an outer approximation by substituting true factors. In step 2, the inner approximation is applied to $q\left(\mathbf{x}\right)$ by using its moments, such that a maximum spanning tree is constructed. With the derived tree structure, the algorithm repeats step 3 and step 4 over iterations such that factors can be updated by performing symbol detection and moment matching, and hard outputs can then be obtained according to the final distribution.

Algorithm 2 The GTA-EC Algorithm

Require: $\mathbf{y}$, $\mathbf{H}$, $E_s$ and ${\sigma }_n^2$. Initialize ${\mathcal{M}}_{i\to pa\left(i\right)}\left(x_{pa\left(i\right)}\right)=1/\sqrt{A}$, ${\mathcal{M}}_{pa\left(i\right)\to i}\left(x_i\right)=1/\sqrt{A}$, ${\gamma }_i=0$ and ${\mathrm{\Lambda }}_i=E_s^{-1}$ for $i=1,\dots ,N$. Ensure: (1) Step 1: Factor Substitution. Initial $q\left(\mathbf{x}\right)\propto f_q\left(\mathbf{x}\right)exp\left({\gamma }_q^{\top }\mathbf{x}-\frac{1}{2}{\mathbf{x}}^{\top }{\mathsf{\Lambda }}_q\mathbf{x}\right)$. (2) Step 2: Inner Approximation. The maximum Gaussian spanning tree is constructed according to the initial covariance matrix such that the tree structure and relationship among symbols can be obtained. repeat (3) Step 3: Symbol Detection. Obtain $r\left(\mathbf{x}\right)$ by achieving consistency between $q\left(\mathbf{x}\right)$ and $s\left(\mathbf{x}\right)$, and obtain $g\left(\mathbf{x}\right)$ according to the established tree structure and derived moments. Perform message passing in updating ${\mathcal{M}}_{i\to pa\left(i\right)}\left(x_{pa\left(i\right)}\right)$ and ${\mathcal{M}}_{pa\left(i\right)\to i}\left(x_i\right)$ according to (33) and (34), and obtain the $aposteriori$ statistics by achieving consistency between $g\left(\mathbf{x}\right)$ and $s\left(\mathbf{x}\right)$. (4) Step 4: Factor Updating. Update ${\gamma }_q^{new}$ and ${\mathsf{\Lambda }}_q^{new}$ such that $q^{new}\left(\mathbf{x}\right)$ can be updated. until A maximum number of iterations has been achieved. Output: Hard outputs according to the first-order moments of the latest $q^{new}\left(\mathbf{x}\right)$.

4.2. Complexity Analysis

The calculation of GTA-EC resides mainly on three parts. The first one involves the factor substitution step, which necessitates the calculation of second-order and first-order moments in (29), the same as MMSE in (16) or EC in (19). As is well known, its complexity in one iteration can be given as $\mathcal{O}\left(N{M}^2\right)$. The second part involves construction of the tree graph for inner approximation, which needs only to be initialized at the very beginning of iterations. The construction is based on Prim’s algorithm, whose complexity is $\mathcal{O}\left(M^2\right)$. The last part involves the calculation of message passing and factor updating. For each iteration, the major complexity lies in calculating messages in (33) and (34), each requiring the maximum likelihood detection on the conditional distribution with the cardinality of PAM constellation being $\left|\mathcal{A}\right|=\sqrt{A}$. Since there are $M-1$ conditional distributions in the tree graph, the complexity can be represented as $\mathcal{O}\left({M\left|\mathcal{A}\right|}^2\right)=\mathcal{O}\left(MA\right)$. Therefore, by defining $N_{iter}$ as the number of iterations to proceed, the total complexity can be expressed as $\mathcal{O}\left((N_{iter}+1)N{M}^2+M^2+N_{iter}MA\right)\approx \mathcal{O}\left((N_{iter}+1)N{M}^2\right)$ due to the reason that $N{M}^2\gg MA$ is normally satisfied in a massive MIMO system. This indicates that the complexity of GTA-EC is about $N_{iter}$ times more than that of MMSE or GTA, namely $\mathcal{O}\left(N{M}^2\right)$. As a comparison, the complexity of EC can be expressed as $\mathcal{O}((N_{iter}+1)N{M}^2+M+N_{iter}M\sqrt{A})\approx \mathcal{O}\left((N_{iter}+1)N{M}^2\right)$, suggesting that the complexity of GTA-EC is approximately in the same order. The less iterations one algorithm needs to perform, the less complexity it requires. In the next section, when comparing the performance of GTA-EC with EC, the number of iterations should be utilized for complexity comparison. A summary of complexity comparison is demonstrated in

Table 1. Comparisons of complexity.

Algorithm	Factor Substitution	Inner Approximation	Detection and Factor Updating	Total Complexity
MMSE	$\mathcal{O}\left(N{M}^2\right)$	$\mathcal{O}\left(M\right)$	$\mathcal{O}\left(M\sqrt{A}\right)$	$\mathcal{O}\left(N{M}^2\right)$
GTA	$\mathcal{O}\left(N{M}^2\right)$	$\mathcal{O}\left(M^2\right)$	$\mathcal{O}\left(MA\right)$	$\mathcal{O}\left(N{M}^2\right)$
EC	$\mathcal{O}\left(\left(N_{iter}+1\right)N{M}^2\right)$	$\mathcal{O}\left(M\right)$	$\mathcal{O}\left(N_{iter}M\sqrt{A}\right)$	$\mathcal{O}\left(\left(N_{iter}+1\right)N{M}^2\right)$
GTA-EC	$\mathcal{O}\left(\left(N_{iter}+1\right)N{M}^2\right)$	$\mathcal{O}\left(M^2\right)$	$\mathcal{O}\left(N_{iter}MA\right)$	$\mathcal{O}\left(\left(N_{iter}+1\right)N{M}^2\right)$

, in which it can be found that the total complexity is dominated by the complexity of factor substitution as well as the number of iterations.

5. Numerical Results

5.1. Simulation Parameters

In this section, the detection performance of a MIMO system is evaluated in terms of bit error rate (BER). Uncorrelated scattering flat-fading channel model is assumed with channel coefficients being modeled as complex Gaussian distributed variables that are independently generated for all antennas. During the simulation, 20,000 realizations of the channel matrix are employed with each used to send one message. As a comparison, several existing algorithm are evaluated as well such as the MMSE, GTA, and EC algorithms. And we mainly take into consideration the ’worst-case’ scenarios of load $\alpha =N/M=1$ when $N=M=16$ and $N=M=64$ with high-order constellations 16-QAM, 64-QAM, and 256-QAM considered. The factor $\beta$ is set as $0.2$ for all algorithms, and the iteration number of EC and GTA-EC is set as 2, 4, and 6 since convergence can be achieved within $six$ iterations.

5.2. Performance Evaluation

Figure 1, Figure 2 and Figure 3 demonstrate the BER comparison of the GTA-EC algorithm with existing algorithms. The number of antennas deployed at both the transmitter and receiver in the system is set as $N=M=16$ with the constellations being 16-QAM, 64-QAM and 256-QAM, respectively. It can be found that GTA-EC outperforms EC with the same number of iterations, and that GTA-EC with $four$ iterations outperforms EC with $six$ iterations, indicating that GTA-EC may achieve better performances than EC does with lower complexity. While in Figure 2 and Figure 3, GTA-EC with $two$ iterations almost exibits better performance than EC with $six$ iterations, revealing better performance gain when high-order constellations are employed. One can further obseve that the BER slopes of GTA-EC decrease faster than that of EC, demonstrating that superior divergence gain can also be obtained by GTA-EC in a high SNR regime.

Figure 4, Figure 5 and Figure 6 demonstrate a BER comparison of the GTA-EC algorithm with existing algorithms. The number of antennas deployed at both the transmitter and receiver is given as $N=M=64$ with the constellations being 16-QAM, 64-QAM and 256-QAM, respectively. In these figures, it can be found that GTA-EC outperforms EC with the same number of iterations, while GTA-EC with $four$ iterations may have similar performance to that of EC with $six$ iterations. This indicates that GTA-EC exhibits better performance than EC does at the same order of complexity or that GTA-EC presents similar performance to that of EC with lower complexity. And in Figure 4 and Figure 5, one can further observe that the BER slope of GTA-EC decreases faster than that of EC, leading to better performance in a high SNR regime.

By observing and analyzing the figures in different scenarios, we may come to conclusions about the performance comparison of GTA-EC with existing algorithms.

• On one hand, both EC and GTA-EC significantly outperform existing algorithms such as MMSE and GTA. In most scenarios, GTA-EC can obviously outperform EC with either 16-QAM, 64-QAM, or 256-QAM employed. The performance gain of GTA-EC becomes larger when high-order constellation is employed. For example, both the 64-QAM and 256-QAM cases exhibit larger gain than the 16-QAM case when employing 16 or 64 antennas. This indicates that GTA-EC has superior performance gain and is especially suitable for high-order constellations. We believe that the performance gain comes from exploiting additonal relations (correlation) among symbols rather than treating them independently.
• On the other hand, as for the complexity issue, GTA-EC with $four$ iterations may outperform or have comparable performance to EC with $six$ iterations, suggesting that GTA-EC requires less complexity than EC by recalling that their computational burdens are dominated by the number of iterations needed. As a result, $four$ iterations are recommended for GTA-EC according to the simulation results, and hence the complexity of GTA-EC is approximately $four$ times more than MMSE, indicating that it is a practical method for massive MIMO systems.

6. Conclusions

A nested variational chain is proposed along with an algorithm provided, which combines two asymmetic KL divergences. Introduced into MIMO systems, it can be found that several existing algorithms such as MMSE, GTA, and EC can be regarded as special cases. As initial applications for MIMO detection, an algorithm named GTA-EC is proposed with complexity analysis, and numerical results prove that it may achieve better detection performance with less complexity compared to existing algorithms. As for further research topics, it is suggested that one can find better inner approximation that may capture much more correlation among symbols by applying this framework to other detection fields, such as space code multiple access (SCMA), orthogonal time frequency space (OTFS), or low-density parity check (LDPC) decoding systems.