Joint Hybrid Beamforming Design for Millimeter Wave Amplify-and-Forward Relay Communication Systems

Abstract

Hybrid beamforming (HBF) has been regarded as one of the most promising technologies in millimeter Wave (mmWave) communication systems. In order to guarantee the communication quality in non-line-of-sight (NLOS) scenarios, joint HBF design for the mmWave amplify-and-forward (AF) relay communication system is studied in this paper. The ideal case is first considered where the mmWave half-duplex (HD) AF relay system operates with channel state information (CSI) accurately known. In order to tackle the non-convex problem, a manifold optimization (MO)-based alternating optimization algorithm is proposed, where an optimization problem containing only constant modulus constraints in Euclidean space can be converted to an unconstrained optimization problem in a Riemann manifold. Furthermore, considering more practical cases with estimation errors of CSI, we investigate the robust joint HBF design with the system operating in full-duplex (FD) mode to obtain higher spectral efficiency (SE). A null-space projection (NP) based self-interference cancellation (SIC) algorithm is developed to attenuate the self-interference (SI). Different from the traditional SI suppression algorithm, there’s no limit on the number of RF chains. Numerical results reveal that our proposed algorithms has a good convergence and can effectively deal with the influence of different CSI estimation errors. A significant performance improvement can be achieved in contrast with other approaches.

1. Introduction

Compared with microwave communication systems, millimeter wave (mmWave) communication systems have larger bandwidth and higher data rate [1], receiving great attention from both academic and industry. However, high propagation loss is a major characteristic of mmWave signals [2], which should be compensated for by deploying large antenna arrays to guarantee the communication quality. Fortunately, due to the shorter wavelength of mmWave, more antennas can be integrated into the same space, enabling Massive MIMO technology in mmWave communication systems [3]. Nevertheless, the drastic increase in the number of antennas leads to a high cost, based on which researchers proposed the hybrid beamforming (HBF) technology.

mmWave communication is mostly used in line-of-sight (LOS) dominant scenarios owing to their high vulnerability to blockages. However, the source-destination link is probably non-line-of-sight (NLOS). Practically, relay nodes are commonly deployed to solve the problem [3,4]. Combined with the relay technology, hybrid transceivers can combat severe attenuation of mmWave signals and extend network convergence. In addition, an mmWave relay system operating in full-duplex (FD) mode can achieve higher spectral efficiency (SE) than the one based on half-duplex (HD) by enabling simultaneous reception and transmission of relay nodes [5]. However, the self-interference (SI) imposed by simultaneous transmission should be mitigated to ensure the communication quality [6]. With effective suppression of SI, the FD technology can reduce end-to-end communication delay and improve spectral efficiency significantly.

Unfortunately, the HBF design problem in mmWave relay communication systems has not been extensively investigated. In [7], the authors exploited a transceiver HBF design on the basis of orthogonal matching pursuit (OMP), whose complexity is relatively low while the performance significantly relates to the orthogonality of the pre-defined analog beamforming (ABF) matrices. In [8], a wideband HBF algorithm was adopted in the multiple-relays scenario. The HBF design techniques of a relay node has been explored in [9,10]. In particular, the alternating direction method of multipliers (ADMM) algorithm was proposed to reduce the minimum square error (MSE) between received signals and transmit signals in [9]. To overcome high complexity, the optimization problem was decomposed into three sub-problems in [10], which is solved by an iterative successive approximation (ISA) algorithm. However, the above two algorithms only optimize the relay node. The authors in [11] developed an HBF algorithm to jointly design all nodes in the mmWave amplify-and-forward (AF) relay system, including the source node, the relay node and the destination node. In [12], the HBF design in mmWave multi-user MIMO relay system was investigated. The ABF of the base station, relay and users could be iteratively designed by solving the weighted minimum MSE (WMMSE) problem.

It is noteworthy that all of these works were under accurate channel state information (CSI). Due to channel estimation errors and user mobility, CSI is not available perfectly in practice. The works of [13,14] studied the robust HBF design for the mmWave FD relay system with the premise of imperfect CSI. To be specific, the work in [13] introduced a robust OMP HBF scheme for mmWave channels with Gaussian-distributed errors to maximize the average receive signal-to-noise ratio (SNR). To reduce complexity, the authors in [14] designed the radio frequency (RF) and baseband separately and showed a robust WMMSE-based algorithm with respect to the sum of rate for the fully connected AF relay networks. On the other hand, the robust design for mmWave HD relay systems were investigated in [15,16]. Given the impact of SI, a minimum MSE (MMSE) optimization problem for the relay node was deduced in [15]. In [16], the SI power constraint was taken into account in the worst-case sum rate maximization problem. To make it tractable [16], reformulated the problem by using the penalty dual decomposition technique. Currently, there are still few researchers focusing on HBF design for mmWave relay communication systems on the basis of channel estimation errors.

Against this background, the above problem of HBF design for the mmWave relay communication system is addressed in this paper. Instead of considering only relay nodes, all nodes are jointly optimized in this paper. Additionally, we take both perfect and imperfect CSI into account. It is worth noting that our proposed algorithms can be applied to both HD and FD mode. For clarity, the main contributions of our work are summarized as follows:

• We first consider the HD relay system under the assumption of perfect CSI. Different from conventional MSE minimization problems, where the power constraints introduce additional complexity, we model the improved MSE (IMSE) as the optimization objective by designing a power scaling factor.
• To deal with the complicated non-convex problem, a manifold optimization (MO) based alternating optimization algorithm is proposed, which decomposes the problem into three sub-problems. Simulation results demonstrate that our proposed method is superior to its counterparts by approaching the full digital solution.
• In terms of the FD relay communication system in the practical scenario, we develop a robust HBF algorithm. To mitigate SI introduced by the FD model, we further develop a null-space-projection (NP)-based SI cancellation (SIC) method, which has no limit on the number of RF chains in contrast with traditional methods. Simulation shows that the proposed approach can achieve sufficient suppression of SI, providing significant beamforming gain.

The remaining sections are organized as follows. Section 2 presents the system model and the mmWave channel model. Section 3 introduces the HBF design for the HD AF relay communication system. The SIC-NP and HBF-MO-FD-R algorithms for the FD AF relay communication system are proposed in Section 4. Section 5 provides our simulation results. Finally, Section 6 draws conclusions.

Notations: In this paper, vectors and matrixes are denoted by bold lowercase letters and uppercase letters, respectively. $\left|\right|·\left|\right|$ and $\left|\right|·{\left|\right|}_F$ stand for the Euclidean norm of a vector and the Frobenius norm of a matrix. $\mathrm{tr}(·)$ and ${(·)}^H$ represent the trace and conjugate transposition operations, respectively. $|·|$ represents the magnitude of a complex number. $\mathrm{E}\{·\}$ denotes the expectation operator. $\mathrm{d}(·)$ and $\nabla (·)$ denote the differential and the gradient of a function. $\mathbb{R}$ and ${\mathbb{C}}^{m\times n}$ denote the set of all real matrixes and all complex $m\times n$ matrixes. ${\mathbf{I}}_m$ denotes the $m\times m$ identity matrix. Some important symbols and their meanings are listed in

Table 1. Notations.

Symbol	Meaning
$N_{\mathrm{T}}$	The number of antennas of the source node
$N_{\mathrm{R}}$	The number of antennas of the relay node
$N_{\mathrm{D}}$	The number of antennas of the destination node
$N_{\mathrm{RF}}$	The number of antennas of RF chains at each node
$N_{\mathrm{S}}$	The number of data streams
s	The original signal of the source node
$\mathbf{F}$	The HBF matrix of the source node
${\mathbf{F}}_{\mathrm{B}}$	The DBF matrix of the source node
${\mathbf{F}}_{\mathrm{RF}}$	The ABF matrix of the source node
$P_{\mathrm{B}}$	The power threshold of the source node
${\mathbf{H}}_1$	The mmWave channel matrix of the source-to-relay link
$\mathbf{G}$	The HBF matrix of the HD relay node
${\mathbf{G}}_{{\mathrm{RF}}_1}$	The receive ABF matrix of the HD relay node
${\mathbf{G}}_{{\mathrm{RF}}_2}$	The transmit ABF matrix of the HD relay node
${\mathbf{G}}_{\mathrm{B}}$	The DBF matrix of the HD relay node
${\sigma }_{\mathrm{R}}^2$	The nosie variance of the relay node
$P_{\mathrm{R}}$	The power threshold of the realy node
${\mathbf{H}}_2$	The mmWave channel matrix of the relay-to-destination link
$\mathbf{W}$	The HBF matrix of the destination node
${\mathbf{W}}_{\mathrm{RF}}$	The ABF matrix of the destination node
${\mathbf{W}}_{\mathrm{B}}$	The DBF matrix of the destination node
${\sigma }_{\mathrm{D}}^2$	The noise variance of the destination node
$\gamma ,{\gamma }_1,{\gamma }_2$	The scaling factors
${\mathbf{F}}_{\mathrm{U}}$	The unnormalized DBF matrix of the source node
${\mathbf{G}}_{\mathrm{U}}$	The unnormalized DBF matrix of the realy node
$\overline{\mathbf{F}}$	The unnormalized HBF matrix of the source node
$\overline{\mathbf{G}}$	The unnormalized HBF matrix of the realy node
${\mathbf{G}}_{{\mathrm{B}}_1}$	The receive DBF matrix of the FD relay node
${\mathbf{G}}_{{\mathrm{B}}_2}$	The transmit DBF matrix of the FD relay node
${\mathbf{G}}_1$	The receive HBF matrix of the FD relay node
${\mathbf{G}}_2$	The transmit HBF matrix of the FD relay node
${\mathbf{H}}_{SI}$	The SI channel matrix of the FD relay node
$\overline{{\mathbf{H}}_{\mathbf{1}}}$,$\overline{{\mathbf{H}}_{\mathbf{2}}}$,$\overline{{\mathbf{H}}_{\mathbf{SI}}}$	The estimated channel matrixes
$\mathbf{\Theta }$	The covariance matrix of estimation error at the receiver side
$\mathbf{\Phi }$	The covariance matrix of estimation error at the transmit side
$\mathbf{\Delta }$	The unknown part of CSI

.

2. System Model

As shown in Figure 1, an AF mmWave relay communication system is adopted in this paper, including a source node with $N_{\mathrm{T}}$ antennas, an HD AF relay node with $N_{\mathrm{R}}$ antennas and a destination node with $N_{\mathrm{D}}$ antennas. And all nodes are equipped with $N_{\mathrm{RF}}$ RF chains. In our scenario, the blockage between the source–destination link prevents direct communication. Consequently, it is necessary for the two nodes to communicate through the AF relay node.

We assume that $\mathbf{s}\in {\mathbb{C}}^{N_{\mathrm{S}}\times 1}$ denotes the original signal of the source node characterized by $\mathrm{E}\left\{\mathbf{s}{\mathbf{s}}^H\right\}={\mathbf{I}}_{N_{\mathrm{s}}}$, with $N_{\mathrm{S}}$ representing the number of data streams. The signal is processed by the digital beamformer ${\mathbf{F}}_{\mathrm{B}}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_{\mathrm{S}}}$ and then the analog beamformer ${\mathbf{F}}_{\mathrm{RF}}\in {\mathbb{C}}^{N_{\mathrm{T}}\times N_{\mathrm{RF}}}$, which subjects to the constant-modulus constraint, i.e., ${\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1$. Therefore, the source transmit signal can be given by ${\mathbf{x}}_{\mathrm{B}}=\mathbf{Fs}$, where $\mathbf{F}={\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{B}}$ is the HBF matrix of the source node. The source transmit power is stated as $\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{B}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{B}}$.

Through the mmWave channel ${\mathbf{H}}_1\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{T}}}$, the signal is received by the AF relay node, the transmit signal of which can be given by ${\mathbf{x}}_{\mathrm{R}}=\mathbf{G}\left({\mathbf{H}}_1{\mathbf{x}}_{\mathrm{B}}+{\mathbf{n}}_{\mathrm{R}}\right)$, where $\mathbf{G}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{G}}_{\mathrm{B}}{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H$ is the relay node HBF matrix, with ${\mathbf{G}}_{{\mathrm{RF}}_1}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{RF}}}$, ${\mathbf{G}}_{{\mathrm{RF}}_2}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{RF}}}$ and ${\mathbf{G}}_{\mathrm{B}}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_{\mathrm{RF}}}$ being the receive ABF matrix, transmit ABF matrix and digital beamforming (DBF) matrix, respectively. ${\mathbf{n}}_{\mathrm{R}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times 1}$ denotes the addictive complex Gaussian noise with zero mean and covariance matrix $\mathrm{E}\left\{{\mathbf{n}}_{\mathrm{R}}{\mathbf{n}}_{\mathrm{R}}^H\right\}={\sigma }_{\mathrm{R}}^2{\mathbf{I}}_{N_{\mathrm{R}}}$ at the relay node. Similarly, the analog beamformers should subject to the constant-modulus constraints, while the transmit signal ${\mathbf{x}}_{\mathrm{R}}$ should satisfy the power constraint $\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}}$.

With an analog beamformer ${\mathbf{W}}_{\mathrm{RF}}\in {\mathbb{C}}^{N_{\mathrm{D}}\times N_{\mathrm{RF}}}$ followed by a digital beamformer ${\mathbf{W}}_{\mathrm{B}}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_{\mathrm{S}}}$, the received signal at the destination node takes the form of

$${\mathbf{y}}_{\mathrm{D}}={\mathbf{W}}^H{\mathbf{H}}_2\mathbf{G}{\mathbf{H}}_1\mathbf{Fs}+{\mathbf{W}}^H{\mathbf{H}}_2\mathbf{G}{\mathbf{n}}_{\mathrm{R}}+{\mathbf{W}}^H{\mathbf{n}}_{\mathrm{D}},$$

(1)

where ${\mathbf{H}}_2\in {\mathbb{C}}^{N_{\mathrm{D}}\times N_{\mathrm{R}}}$ denotes the mmWave channel of the relay-to-destination link, ${\mathbf{n}}_{\mathrm{D}}\in {\mathbb{C}}^{N_{\mathrm{D}}\times 1}$ is the zero-mean addictive complex Gaussian noise vector with covariance matrix $\mathrm{E}\left\{{\mathbf{n}}_{\mathrm{D}}{\mathbf{n}}_{\mathrm{D}}^H\right\}={\sigma }_{\mathrm{D}}^2{\mathbf{I}}_{N_{\mathrm{D}}}$ at the destination node. For simplicity, we define $\mathbf{W}={\mathbf{W}}_{\mathrm{RF}}{\mathbf{W}}_{\mathrm{B}}$ as the HBF matrix of the destination node.

Channel Model

The mmWave channel model in [17,18,19,20] is considered in this paper, which takes the following form:

$$\mathbf{H}=\sqrt{\frac{N_{\mathrm{T}}{N}_{\mathrm{R}}}{L}}\sum _{l=1}^L{\alpha }_l{\mathbf{a}}_{\mathrm{r}}\left({\phi }_l^{\mathrm{r}}\right){\mathbf{a}}_{\mathrm{t}}^H\left({\phi }_l^{\mathrm{t}}\right),$$

(2)

where L denotes the number of propagation paths, ${\alpha }_l$ denotes the complex channel gain of the $l^{th}$ path. ${\mathbf{a}}_{\mathrm{t}}\left({\phi }_l^{\mathrm{t}}\right)$ and ${\mathbf{a}}_{\mathrm{r}}\left({\phi }_l^{\mathrm{r}}\right)$ are the transmitting and receiving array response vectors. ${\phi }_l^{\mathrm{t}}$ and ${\phi }_l^{\mathrm{r}}$ stand for the elevation angles of departure (AOD) and azimuth angles of arrival (AOA) for the $l^{th}$ path, respectively. It is worth noting that our proposed algorithms can be applied to any antenna array. For simplicity, the commonly applied uniform linear array (ULA) with N elements are considered. The array response vector can be given by

$$\mathbf{a}\left(\phi \right)=\frac{1}{\sqrt{N}}{[1,e^{j\frac{2\pi }{\lambda }dsin\left(\phi \right)},\dots ,e^{j(N-1)\frac{2\pi }{\lambda }dsin\left(\phi \right)}]}^T,$$

(3)

where $\lambda$ represents the wavelength and d stand for the distance between antenna elements, which is assumed to satisfy $d=\lambda /2$. In this section, both ${\mathbf{H}}_1$ and ${\mathbf{H}}_2$ are modelled as (2).

3. Hybrid Beamforming Design for HD AF Relay Systems

3.1. Problem Formulation

A mathematical model of the joint HBF design problem is formulated in this subsection. According to [21,22], the IMSE is introduced as the metric to optimize the HBF matrix, which can be defined as

$$\mathrm{IMSE}\stackrel{\Delta }{=}\mathrm{E}\left\{{∥{\gamma }^{-1}{\mathbf{y}}_{\mathrm{D}}-\mathbf{s}∥}^2\right\},$$

(4)

where $\gamma ={\gamma }_1{\gamma }_2$, with ${\gamma }_1$ and ${\gamma }_2$ being the scaling factors, which are jointly optimized with the HBF matrixes at the source and relay nodes, contributing to simplify and solve the problem. By substituting (1) into (4), we have

$$\begin{array}{cc}\hfill \mathrm{IMSE}=& tr({{\gamma }_2}^{-2}{{\gamma }_1}^{-2}{\mathbf{W}}^H{\mathbf{H}}_2\mathbf{G}{\mathbf{H}}_1\mathbf{F}{\mathbf{F}}^H{\mathbf{H}}_1^H{\mathbf{G}}^H{\mathbf{H}}_2^H\mathbf{W}-{\gamma }_2^{-1}{\gamma }_1^{-1}{\mathbf{W}}^H{\mathbf{H}}_2\mathbf{G}{\mathbf{H}}_1\mathbf{F}\hfill \\ \hfill -& {\gamma }_2^{-1}{\gamma }_1^{-1}{\mathbf{F}}^H{\mathbf{H}}_1^H{\mathbf{G}}^H{\mathbf{H}}_2^H\mathbf{W}+{{\gamma }_2}^{-2}{{\gamma }_1}^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}^H{\mathbf{H}}_2\mathbf{G}{\mathbf{G}}^H{\mathbf{H}}_2^H\mathbf{W}\hfill \\ \hfill +& {{\gamma }_2}^{-2}{{\gamma }_1}^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}^H\mathbf{W}+{\mathbf{I}}_{N_{\mathrm{S}}}).\hfill \end{array}$$

(5)

Similar to that in [22,23,24], the source and relay DBF matrixes can be decomposed as ${\mathbf{F}}_{\mathrm{B}}={\gamma }_1{\mathbf{F}}_{\mathrm{U}}$ and ${\mathbf{G}}_{\mathrm{B}}={\gamma }_2{\mathbf{G}}_{\mathrm{U}}$, with ${\mathbf{F}}_{\mathrm{U}}$ and ${\mathbf{G}}_{\mathrm{U}}$ being the unnormalized DBF matrix of the source and relay nodes, respectively. So we can rewrite the modified MSE as

$$\begin{array}{cc}\hfill \mathrm{IMSE}=& \mathrm{tr}({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}-{\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}-{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}\hfill \\ \hfill +& {\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}^H\mathbf{W}+{\mathbf{I}}_{N_{\mathrm{S}}})\hfill \end{array}$$

(6)

where we define $\overline{\mathbf{F}}={\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{U}}$ and $\overline{\mathbf{G}}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{G}}_{\mathrm{U}}{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H$ for convenience. Hence, we have $\mathbf{F}={\gamma }_1\overline{\mathbf{F}}$ and $\mathbf{G}={\gamma }_2\overline{\mathbf{G}}$. As a further step, the optimization problem can be formulated as

$$\begin{array}{c}\hfill \underset{{\gamma }_1,{\gamma }_2,{\mathbf{F}}_{\mathrm{RF}},{\mathbf{F}}_{\mathrm{U}},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{U}},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2},{\mathbf{W}}_{\mathrm{RF}},{\mathbf{W}}_{\mathrm{B}}}{\max }\mathrm{IMSE}\end{array}$$

(7a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b,\hfill \end{array}$$

(7b)

$$\begin{array}{cc}\hfill & {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}_{c,d}\right|}^2=1,\forall c,d,\hfill \end{array}$$

(7c)

$$\begin{array}{cc}\hfill & {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}_{e,f}\right|}^2=1,\forall e,f,\hfill \end{array}$$

(7d)

$$\begin{array}{cc}\hfill & {\left|{\left[{\mathbf{W}}_{\mathrm{RF}}\right]}_{g,h}\right|}^2=1,\forall g,h,\hfill \end{array}$$

(7e)

$$\begin{array}{cc}\hfill & \mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{B}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{B}},\hfill \end{array}$$

(7f)

$$\begin{array}{cc}\hfill & \mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}},\hfill \end{array}$$

(7g)

where (7b)–(7e) are the constant-modulus constraints of ABF matrixes, (7f) and (7g) are the source and relay transmit power constraints.

3.2. MO-Based HBF Design

It can be observed from the optimization problem (7) that there exist nine variables and four nonconvex constraints, which is tough to get the global optimal solution. In this subsection, an alternating optimization algorithm on the basis of the MO method is proposed. To overcome the difficulty of solving the original problem, it is decomposed into three HBF design sub-problems [25,26].

(1) The Source Node Design: When $\mathbf{G}$ and $\mathbf{W}$ are fixed, the sub-problem can be further expressed as

$$\begin{array}{cc}\hfill \underset{{\gamma }_1,{\mathbf{F}}_{\mathrm{RF}},{\mathbf{F}}_{\mathrm{U}}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b,\hfill \end{array}$$

(8b)

$$\begin{array}{c}\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{B}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{B}}.\hfill \end{array}$$

(8c)

According to $\mathbf{F}={\gamma }_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{U}}$, (8c) can be rewritten as ${\gamma }_1^2$$\mathrm{tr}\left({\overline{\mathbf{FF}}}^H\right)=P_{\mathrm{B}}$, and we can obtain

$${\gamma }_1={\left[\frac{\mathrm{tr}\left({\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{U}}{\mathbf{F}}_{\mathrm{U}}^H{\mathbf{F}}_{\mathrm{RF}}^H\right)}{P_{\mathrm{B}}}\right]}^{-\frac{1}{2}}.$$

(9)

To this end, the Lagrange function $\mathrm{L}1\left({\lambda }_1,{\mathbf{F}}_{\mathrm{U}},{\gamma }_1\right)$ of problem (8) can be constructed. The minimum of $\mathrm{L}1\left({\lambda }_1,{\mathbf{F}}_{\mathrm{U}},{\gamma }_1\right)$ is achieved when its partial derivatives equal to zero, which gives the optimal ${\mathbf{F}}_{\mathrm{U}}$ as

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\mathrm{U}}& ={\left({\mathbf{F}}_{\mathrm{RF}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}{\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}+{\mathbf{B}}_{11}\right)}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W},\hfill \end{array}$$

(10)

with ${\mathbf{B}}_{11}=b_{11}\left({\mathbf{F}}_{\mathrm{RF}}^H{\mathbf{F}}_{\mathrm{RF}}\right)$ and $b_{11}=\frac{{\sigma }_{\mathrm{R}}^2}{P_{\mathrm{B}}}\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H\right.$$\left.{\mathrm{H}}_2^H\mathbf{W}\right)+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{B}}}{\gamma }_2^{-2}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right)$.

One more step, substituting (8a) with (10), the optimization sub-problem can be concisely re-formulated as

$$\begin{array}{cc}\hfill \underset{{\mathbf{F}}_{\mathrm{RF}}}{\min }& \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{s}.\mathrm{t}.{\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b,\hfill \end{array}$$

(11b)

where we denote

$$\begin{array}{cc}\hfill \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)& =\mathrm{tr}{\left\{\left[{\mathbf{I}}_{N_{\mathrm{S}}}+\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}\right)\right.{\mathbf{B}}_{11}^{-1}{\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}\right)}^H\right]}^{-1}\hfill \end{array}$$

(12)

We can see that (11) is an optimization problem with only a constant-modulus constraint. Om the grounds of [21,27], we can tackle it by applying the MO method. However, it is difficult to use MO in the Euclidean space directly. Fortunately, if the gradient of objective function in Euclidean space is derived, the above problem can be easily solved [27]. The gradient of $\mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)$ with respect to ${\mathbf{F}}_{\mathrm{RF}}$ is defined as $\nabla \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)=\frac{\partial \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)}{\partial {\mathbf{F}}_{\mathrm{RF}}^{*}}$ [28], which is given by

$$\begin{array}{cc}\hfill \nabla \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF}}\right)& =\left(b_{11}{\mathbf{F}}_{\mathrm{RF}}{\mathbf{B}}_{11}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H-{\mathbf{I}}_{N_{\mathrm{T}}}\right){\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\right)}^H{\mathbf{B}}_{12}^{-2}{\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{B}}_{11}^{-1},\hfill \end{array}$$

(13)

where ${\mathbf{B}}_{12}\stackrel{\Delta }{=}{\mathbf{I}}_{N_{\mathrm{S}}}+\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}\right){\mathbf{B}}_{11}^{-1}{\left({\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1{\mathbf{F}}_{\mathrm{RF}}\right)}^H$ for brevity. The detailed derivation is provided in Appendix A.

(2) The Relay Node Design: In a similar manner, it is assumed that $\mathbf{F}$ and $\mathbf{W}$ are fixed. The optimization sub-problem of the relay node can be formulated as

$$\begin{array}{cc}\hfill \underset{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{B}},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}_{c,d}\right|}^2=1,\forall c,d,\hfill \end{array}$$

(14b)

$$\begin{array}{c}{\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}_{e,f}\right|}^2=1,\forall e,f,\hfill \end{array}$$

(14c)

$$\begin{array}{c}\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}}.\hfill \end{array}$$

(14d)

For any given ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$ and ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$, we can obtain

$${\gamma }_2={\left[\frac{\mathrm{tr}\left({\gamma }_1^2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H+{\sigma }_{\mathrm{R}}^2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H\right)}{P_{\mathrm{R}}}\right]}^{-\frac{1}{2}}.$$

(15)

This leads to the optimization problem with only power constraint. Based on the Lagrange function, we have

$$\begin{array}{cc}\hfill {\mathbf{G}}_{\mathrm{U}}& ={\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{H}}_2^H\mathbf{W}{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)}^{-1}\hfill \\ \hfill & \times {\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{H}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right)}^{-1}.\hfill \end{array}$$

(16)

By substituting (15) into (14a), the objective function is reduced to

$$\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)=\mathrm{tr}\left({\mathbf{I}}_{N_{\mathrm{RF}}}-{\mathbf{B}}_{22}^H{\mathbf{B}}_{21}^{-1}{\mathbf{B}}_{22}{\mathbf{B}}_{23}^{-1}\right),$$

(17)

where ${\mathbf{B}}_{21}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{H}}_2^H\mathbf{W}{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right)$${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$, ${\mathbf{B}}_{22}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{H}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$ and ${\mathbf{B}}_{23}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$. The partial derivative of $\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)$ for ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^{*}$ and ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^{*}$ can be obtained as

$$\begin{array}{cc}\hfill & {\nabla }_{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}}\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)=\left({\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{B}}_{23}^{-1}{\mathbf{B}}_{22}^H\right.\hfill \\ \hfill & +{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{B}}_{23}^{-1}{\mathbf{B}}_{22}^H\left.-{\mathbf{H}}_1\overline{\mathbf{F}}{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right){\mathbf{B}}_{21}^{-1}{\mathbf{B}}_{22}{\mathbf{B}}_{23}^{-1},\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & {\nabla }_{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}}\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)=\left({\mathbf{H}}_2^H\mathbf{W}{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{B}}_{21}^{-1}{\mathbf{B}}_{22}\right.\hfill \\ \hfill & +\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{B}}_{21}^{-1}{\mathbf{B}}_{22}\left.-{\mathbf{H}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right){\mathbf{B}}_{23}^{-1}{\mathbf{B}}_{22}^H{\mathbf{B}}_{21}^{-1}.\hfill \end{array}$$

(19)

(3) The Destination Node Design: With $\mathbf{F}$ and $\mathbf{G}$ fixed, the optimization sub-problem of the destination node can be expressed as

$$\begin{array}{cc}\hfill \underset{{\mathbf{W}}_{\mathrm{RF}},{\mathbf{W}}_{\mathrm{B}}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(20a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{W}}_{\mathrm{RF}}\right]}_{g,h}\right|}^2=1,\forall g,h.\hfill \end{array}$$

(20b)

Compared with the source and relay nodes, the destination node only transmits signals without receiving signals. Hence, there is no power constraint. When ${\mathbf{W}}_{\mathrm{RF}}$ is constant, (20) can be converted to an unconstrained optimization problem. By taking the derivative, the optimal ${\mathbf{W}}_{\mathrm{B}}$ is given by

$$\begin{array}{cc}\hfill {\mathbf{W}}_{\mathrm{B}}& =\left({\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}\right.+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}\hfill \\ \hfill & {\left.+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{W}}_{\mathrm{RF}}\right)}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}.\hfill \end{array}$$

(21)

Substituting (21) into (20a), let ${\mathbf{B}}_{31}={\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H$${\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{W}}_{\mathrm{RF}}$, we can recast the objective function as

$$\begin{array}{cc}\hfill & \mathrm{T}3\left({\mathbf{W}}_{\mathrm{RF}}\right)=\mathrm{tr}\left[\left({\mathbf{I}}_{N_{\mathrm{S}}}\right.{\left.+{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{B}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}\right)}^{-1}\right].\hfill \end{array}$$

(22)

Likewise, the gradient is given by

$$\begin{array}{cc}\hfill & \nabla \mathrm{T}3\left({\mathbf{W}}_{\mathrm{RF}}\right)=\left({\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{B}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H\right.\hfill \\ \hfill & \left.+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}_{\mathrm{RF}}{\mathbf{B}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H-{\mathbf{I}}_{N_{\mathrm{D}}}\right){\mathbf{B}}_{33}{\mathbf{B}}_{32}^{-2}{\mathbf{B}}_{33}^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{B}}_{31}^{-1},\hfill \end{array}$$

(23)

where we define ${\mathbf{B}}_{32}={\mathbf{I}}_{N_{\mathrm{S}}}+{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{B}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}$ and ${\mathbf{B}}_{33}={\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}$.

(4) Overall Algorithm: On the basis of the above description, the proposed HBF-MO algorithm for the HD relay system is summarized in Algorithm 1. The original joint IMSE problem can be decomposed into three sub-problems, which are optimized alternately until the stopping criterion is met, as shown in Step 3 to Step 12. Moreover, the MO-based approach generates a local optimal solution instead of a global one. As a consequence, the performance of our proposed algorithm depends on the initialization of system parameters. In practice, the HBF matrix keeps approaching to the full DBF matrix with the number of iterations increasing. In this paper, we start with the full DBF matrix calculated by the singular value decomposition (SVD).

Algorithm 1 HBF-MO Algorithm.

Input:  ${\mathbf{H}}_1$, ${\mathbf{H}}_2$, ${\sigma }_{\mathrm{R}}^2$, ${\sigma }_{\mathrm{D}}^2$. Output:  ${\gamma }_1$, ${\gamma }_2$, ${\mathbf{F}}_{\mathrm{RF}}$, ${\mathbf{F}}_{\mathrm{U}}$, ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$, ${\mathbf{G}}_{\mathrm{U}}$, ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$, ${\mathbf{W}}_{\mathrm{RF}}$, ${\mathbf{W}}_{\mathrm{B}}$.    1: $\mathbf{Initialize}$ ${\mathbf{F}}_{\mathrm{RF}}$, ${\mathbf{W}}_{\mathrm{RF}}$, ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$ and ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$ randomly.    2: $\mathbf{Initialize}$ ${\mathbf{F}}_{\mathrm{U}}$, ${\mathbf{G}}_{\mathrm{U}}$ and ${\mathbf{W}}_{\mathrm{B}}$ with full DBF matrixes, and set ${\gamma }_1={\gamma }_2=1$, $k=1$.    3: while ${\mathrm{IMSE}}_k-{\mathrm{IMSE}}_{k-1}>\epsilon$ do    4:    Calculate $\nabla \mathrm{T}1\left({\mathbf{F}}_{\mathrm{RF},k}\right)$ according to (13) and compute ${\mathbf{F}}_{\mathrm{RF},k+1}$ by using the MO method.    5:    Update ${\gamma }_{1,k+1}$, ${\mathbf{F}}_{\mathrm{U},k+1}$ and ${\mathbf{F}}_{k+1}$ according to (9) and (10).    6:    Calculate ${\nabla }_{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}}\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1,k},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2,k}\right)$ according to (18) and compute ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1,k+1}$ by using the MO method.    7:    Calculate ${\nabla }_{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}}\mathrm{T}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1,k+1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2,k}\right)$ according to (19) and compute ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2,k+1}$ by using the MO method.    8:    Update ${\gamma }_{2,k+1}$, ${\mathbf{G}}_{\mathrm{U},k+1}$ and ${\mathbf{G}}_{k+1}$ according to (15) and (16).    9:    Calculate $\nabla \mathrm{T}3\left({\mathbf{W}}_{\mathrm{RF},k}\right)$ according to (23) and compute ${\mathbf{W}}_{\mathrm{RF},k+1}$ by using the MO method.  10:    Update ${\mathbf{W}}_{\mathrm{B},k+1}$ and ${\mathbf{W}}_{k+1}$ according to (21).  11:    $k\leftarrow k+1$.  12: end while

3.3. Algorithm Evaluation

(1) Convergence Analysis: The HBF-MO algorithm mainly includes two loops, which are the outer loop of Step 3 to Step 12 and the inner loop of Step 4, 6, 7, and 9 in Algorithm 1, respectively. With respect to the inner loop, by the definition of Theorem 4.3.1 in [29], the MO-based algorithm guarantees to converge to a point at which the first-order optimality condition is met [21,27]. As a result, the objective function is ensured to decrease in each iteration of the alternating algorithm. For the outer loop, the DBF matrixes are obtained by taking the derivative, guaranteeing the convergence.

(2) Complexity Analysis: The complexity consists of three parts: (a) calculating the gradient, with the complexity of $\mathsf{𝒪}\left(N_{\mathrm{R}}^2{N}_{\mathrm{MAX}2}\right)$, (b) orthogonal projection and retraction, whose complexity is $\mathsf{𝒪}\left(N_{\mathrm{MAX}1}{N}_{\mathrm{RF}}\right)$, and (c) line search, whose complexity is $\mathsf{𝒪}\left(N_{\mathrm{R}}^2{N}_{\mathrm{MAX}2}\right)$, where $N_{\mathrm{MAX}1}=max\left(N_{\mathrm{T}},N_{\mathrm{R}},N_{\mathrm{D}}\right)$, $N_{\mathrm{MAX}2}=max\left(N_{\mathrm{T}},N_{\mathrm{D}}\right)$ and $min\left(N_{\mathrm{T}},N_{\mathrm{R}},N_{\mathrm{D}}\right)>N_{\mathrm{RF}}$. Therefore, the total complexity is $N_{\mathrm{out}}{N}_{\mathrm{in}}\mathsf{𝒪}\left(N_{\mathrm{R}}^2{N}_{\mathrm{MAX}2}\right)$, with $N_{\mathrm{in}}$ and $N_{\mathrm{out}}$ denoting the number of iterations of the inner and outer loops.

4. Hybrid Beamforming Design for FD AF Relay Systems

In this section, a more common case in practical applications is further considered, where the FD AF relay communication system operates with CSI unavailable.

4.1. System Model

(1) System Model: It is inevitable that the AF relay system will lead to severe SI in FD mode. As shown in Figure 2, in order to mitigate the SI, we decompose the DBF matrix ${\mathbf{G}}_{\mathrm{B}}$ of the relay node into ${\mathbf{G}}_{{\mathrm{B}}_1}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_{\mathrm{S}}}$ and ${\mathbf{G}}_{{\mathrm{B}}_2}\in {\mathbb{C}}^{N_{\mathrm{RF}}\times N_{\mathrm{S}}}$ to design individually, which are the relay digital receive and transmit beamforming matrixes, respectively.

The FD relay node can receive its own transmit signal and that of the source node simultaneously. Hence, the received signal of the relay node takes the form of [30,31]

$${\mathbf{y}}_{\mathrm{R}}={\mathbf{G}}_1^H{\mathbf{H}}_1\mathbf{F}{\mathbf{s}}_{\mathrm{B}}+{\mathbf{G}}_1^H{\mathbf{H}}_{\mathrm{SI}}{\mathbf{G}}_2{\mathbf{s}}_{\mathrm{R}}+{\mathbf{G}}_1^H{\mathbf{n}}_{\mathrm{R}},$$

(24)

where ${\mathbf{G}}_1={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{G}}_{{\mathrm{B}}_1}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{S}}}$ and ${\mathbf{G}}_2={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{G}}_{{\mathrm{B}}_2}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{S}}}$ are the relay hybrid receive and transmit beamforming matrixes. ${\mathbf{s}}_{\mathrm{B}}$ and ${\mathbf{s}}_{\mathrm{R}}$ denote the transmit signals of the source and relay nodes, which are subject to $\mathrm{E}\left\{{\mathbf{s}}_{\mathrm{B}}{\mathbf{s}}_{\mathrm{B}}^H\right\}={\mathbf{I}}_{N_{\mathrm{s}}}$ and $\mathrm{E}\left\{{\mathbf{s}}_{\mathrm{R}}{\mathbf{s}}_{\mathrm{R}}^H\right\}=P_{\mathrm{th}}{\mathbf{I}}_{N_{\mathrm{s}}}$, with $P_{\mathrm{th}}$ denoting the power of interference signals. And ${\mathbf{H}}_{\mathrm{SI}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{R}}}$ stands for the SI channel matrix. The received signal at the destination node can be expressed as

$$\begin{array}{cc}\hfill {\mathbf{y}}_{\mathrm{D}}& =\underset{\mathrm{desired}\mathrm{signal}}{\underbrace{{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_2{\mathbf{G}}_1^H{\mathbf{H}}_1\mathbf{F}{\mathbf{s}}_{\mathrm{B}}}}+\underset{\mathrm{SI}}{\underbrace{{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_2{\mathbf{G}}_1^H{\mathbf{H}}_{\mathrm{SI}}{\mathbf{G}}_2{\mathbf{s}}_{\mathrm{R}}}}\hfill \\ \hfill & +\underset{\mathrm{noise}\mathrm{at}\mathrm{the}\mathrm{relay}\mathrm{node}}{\underbrace{{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_2{\mathbf{G}}_1^H{\mathbf{n}}_{\mathrm{R}}}}+\underset{\mathrm{noise}\mathrm{at}\mathrm{the}\mathrm{destination}\mathrm{node}}{\underbrace{{\mathbf{W}}^H{\mathbf{n}}_{\mathrm{D}}}}.\hfill \end{array}$$

(25)

With the SI constraint ${\mathbf{G}}_1^H{\mathbf{H}}_{\mathrm{SI}}{\mathbf{G}}_2=\mathbf{0}$ set in the HBF design, ${\mathbf{y}}_{\mathrm{D}}$ can be further expressed as

$${\mathbf{y}}_{\mathrm{D}}={\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_2{\mathbf{G}}_1^H{\mathbf{H}}_1\mathbf{F}{\mathbf{s}}_{\mathrm{B}}+{\mathbf{W}}^H{\mathbf{H}}_2{\mathbf{G}}_2{\mathbf{G}}_1^H{\mathbf{n}}_{\mathrm{R}}+{\mathbf{W}}^H{\mathbf{n}}_{\mathrm{D}}.$$

(26)

(2) Imperfect CSI Model: In practice, the CSI is usually not available accurately at every node because of user mobility and algorithm processing delay. The channel model with estimation errors is modelled as

$$\mathbf{H}=\overline{\mathbf{H}}+\Delta \mathbf{H},$$

(27)

where $\mathbf{H}$ and $\overline{\mathbf{H}}$ are the true and estimated channel matrixes, respectively. $\Delta \mathbf{H}$ denotes the channel estimation error, which is given by [14,15]

$$\Delta \mathbf{H}={\mathbf{\Phi }}^{\frac{1}{2}}\mathbf{\Delta }{\mathbf{\Theta }}^{\frac{1}{2}},$$

(28)

where $\mathbf{\Theta }$ and $\mathbf{\Phi }$ are the covariance matrixes of estimation error at the receiver and transmitter sides. $\mathbf{\Delta }$ denotes the unknown part of CSI, which is a complex Gaussian matrix with identically distributed zero mean and unit variance entries. Therefore, the true channel matrix satisfies $\mathbf{H}\sim \mathcal{C}\mathcal{N}\left(\overline{\mathbf{H}},\mathbf{\Theta }\otimes \mathbf{\Phi }\right)$.

In this correspondence, the exponential model is adopted to design the covariance matrixes of CSI error [14,15,32,33], i.e., $\mathbf{\Phi }\left(i,j\right)={\sigma }_{\mathrm{e}}^2{\beta }^{\left|i-j\right|}$, $\mathbf{\Theta }\left(i,j\right)={\alpha }^{\left|i-j\right|}$, with $\alpha ,\beta \in \mathbb{R}$ being correlation coefficients and ${\sigma }_{\mathrm{e}}^2\in \mathbb{R}$ being estimation error variance. $\left(i,j\right)$ represents the element at the intersection of row i and column j.

(3) SI Channel Model: In mmWave FD relay communication systems, SI consists of the LOS part and the NLOS part, where the former is much bigger than the latter on account of the path loss of mmWave signals. On one hand, NLOS SI is caused by the reflection of transmit signals on adjacent obstacles. Typically, the reflection channel ${\mathbf{H}}_{\mathrm{NLOS}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{R}}}$ can be modelled as (2), i.e., the mmWave channel [15]. On the other hand, the direct path SI is caused by transmit signals of the FD relay node through the LOS channel ${\mathbf{H}}_{\mathrm{LOS}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{R}}}$ between the transmitter–receiver link. According to [34], it can be formulated as the near-field model, whose ${(m,n)}_{th}$ element can be expressed as

$${\mathbf{H}}_{\mathrm{LOS}}^{\left(m,n\right)}={\mu }_{mn}{e}^{-j\frac{2\pi }{\lambda }{d}_{mn}},$$

(29)

where ${\mu }_{mn}$ is the normalization factor, $d_{mn}$ denotes the distance between the $m_{th}$ receive antenna and the $n_{th}$ transmit antenna. Consequently, the SI channel can be modelled as

$${\mathbf{H}}_{\mathrm{SI}}=\sqrt{\frac{\kappa }{1+\kappa }}{\mathbf{H}}_{\mathrm{LOS}}+\sqrt{\frac{1}{1+\kappa }}{\mathbf{H}}_{\mathrm{NLOS}},$$

(30)

with the scalar κ being the Rician factor.

In practice, the transmit and receive arrays of the FD AF relay node are close and fixed. In this paper, it is assumed that there is an accurate estimation for ${\mathbf{H}}_{\mathrm{LOS}}$ while uncertainty for ${\mathbf{H}}_{\mathrm{NLOS}}$ [35]. As a result, the estimated SI channel is given by [30]

$${\overline{\mathbf{H}}}_{\mathrm{SI}}=\sqrt{\frac{\kappa }{1+\kappa }}{\mathbf{H}}_{\mathrm{LOS}}+\sqrt{\frac{1}{1+\kappa }}{\overline{\mathbf{H}}}_{\mathrm{NLOS}},$$

(31)

where ${\overline{\mathbf{H}}}_{\mathrm{NLOS}}$ denotes the estimated reflection channel.

Due to the channel estimation errors, the SI constraint should be transformed into ${\mathbf{G}}_1^H{\overline{\mathbf{H}}}_{\mathrm{SI}}{\mathbf{G}}_2=\mathbf{0}$, which means the LOS part can be completely eliminated, while the NOLS part cannot. Considering the NLOS SI is much weaker than the LOS SI, we ignore the influence of residual SI in the subsequent problem formulation and algorithm design [30,35], but its impact on spectral efficiency will be shown in the simulation results.

4.2. Proposed HBF-MO-FD-R Algorithm

In this subsection, the IMSE criterion is still adopted in the HBF design for each node in the FD AF relay communication system, where the optimization objective can be written as

$$\begin{array}{cc}\hfill & \mathrm{IMSE}={\mathrm{E}}_{\mathbf{\Delta }{\mathbf{H}}_{\mathbf{1}},\Delta {\mathbf{H}}_{\mathbf{2}}}\left\{\mathrm{tr}\left({\mathbf{W}}^{\mathrm{H}}{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^{\mathrm{H}}{\mathbf{H}}_1^{\mathrm{H}}{\overline{\mathbf{G}}}^{\mathrm{H}}{\mathbf{H}}_2^{\mathrm{H}}\mathbf{W}\right.\right.-{\mathbf{W}}^{\mathrm{H}}{\mathbf{H}}_2\overline{\mathbf{G}}{\mathbf{H}}_1\overline{\mathbf{F}}\hfill \\ \hfill & -{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}^H{\mathbf{H}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{H}}_2^H\mathbf{W}\left.\left.+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}^H\mathbf{W}+{\mathbf{I}}_{N_{\mathrm{S}}}\right)\right\}.\hfill \end{array}$$

(32)

When a matrix $\mathbf{X}$ satisfies $\mathbf{X}\sim \mathcal{C}\mathcal{N}\left(\overline{\mathbf{X}},\mathbf{\Sigma }\otimes \mathbf{\Psi }\right)$, we have [36]

$$\mathrm{E}\left\{\mathbf{XC}{\mathbf{X}}^H\right\}=\overline{\mathbf{X}}\mathbf{C}{\overline{\mathbf{X}}}^H+\mathrm{tr}\left(\mathbf{C}{\mathbf{\Sigma }}^T\right)\mathbf{\Psi },$$

(33)

$$\mathrm{E}\left\{{\mathbf{X}}^H\mathbf{CX}\right\}={\overline{\mathbf{X}}}^H\mathbf{C}\overline{\mathbf{X}}+\mathrm{tr}\left(\mathbf{\Psi }\mathbf{C}\right){\mathbf{\Sigma }}^T.$$

(34)

Since the error matrixes $\Delta {\mathbf{H}}_1$ and $\Delta {\mathbf{H}}_2$ are independent from each other, we can rewrite IMSE as

$$\begin{array}{cc}\hfill \mathrm{IMSE}& =\mathrm{tr}\left[{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}\right.+\mathrm{tr}\left(\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right){\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\hfill \\ \hfill & +\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)\mathrm{tr}\left(\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right){\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\hfill \\ \hfill & -{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}-{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}\hfill \\ \hfill & +{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2\mathrm{tr}\left(\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right){\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\left.+{\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2{\mathbf{W}}^H\mathbf{W}+{\mathbf{I}}_{N_{\mathrm{S}}}\right]\hfill \end{array}$$

(35)

in view of (33) and (34). Therefore, the IMSE problem can be formulated as

$$\begin{array}{c}\hfill \underset{{\gamma }_1,{\gamma }_2,{\mathbf{F}}_{\mathrm{RF}},{\mathbf{F}}_{\mathrm{U}},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{{\mathrm{B}}_1},{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2},{\mathbf{G}}_{{\mathrm{U}}_2},{\mathbf{W}}_{\mathrm{RF}},{\mathbf{W}}_{\mathrm{B}}}{\min }\mathrm{IMSE}\end{array}$$

(36a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b,\hfill \end{array}$$

(36b)

$$\begin{array}{c}{\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}_{c,d}\right|}^2=1,\forall c,d,\hfill \end{array}$$

(36c)

$$\begin{array}{c}{\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}_{e,f}\right|}^2=1,\forall e,f,\hfill \end{array}$$

(36d)

$$\begin{array}{c}{\left|{\left[{\mathbf{W}}_{\mathrm{RF}}\right]}_{g,h}\right|}^2=1,\forall g,h,\hfill \end{array}$$

(36e)

$$\begin{array}{c}\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{B}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{B}},\hfill \end{array}$$

(36f)

$$\begin{array}{c}{\mathrm{E}}_{\Delta {\mathbf{H}}_1}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}},\hfill \end{array}$$

(36g)

$$\begin{array}{c}{\mathbf{G}}_1^H{\overline{\mathbf{H}}}_{\mathrm{SI}}{\mathbf{G}}_2=\mathbf{0},\hfill \end{array}$$

(36h)

where (36f) and (36g) denote the source and AF relay power constraints. Specifically, the former has the same form as that in Section III, while the latter can be given by ${\mathrm{E}}_{\Delta {\mathbf{H}}_1}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}={\gamma }_2^2\mathrm{tr}\left[{\gamma }_1^2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H\right.$$\left.+{\gamma }_1^2\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H+{\sigma }_{\mathrm{R}}^2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H\right]$.

(1) The Source Node Design: While fixing $\mathbf{G}$ and $\mathbf{W}$, the HBF design sub-problem of the source node can be formulated as

$$\begin{array}{cc}\hfill \underset{{\gamma }_1,{\mathbf{F}}_{\mathrm{RF}},{\mathbf{F}}_{\mathrm{U}}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b,\hfill \end{array}$$

(37b)

$$\begin{array}{c}\mathrm{E}\left\{{∥{\mathbf{x}}_{\mathrm{B}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{B}}.\hfill \end{array}$$

(37c)

with $\mathbf{F}={\gamma }_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{U}}$, we have the power constraint as ${\gamma }_1^2\mathrm{tr}\left({\overline{\mathbf{FF}}}^H\right)=P_{\mathrm{B}}$. Therefore, ${\gamma }_1$ can be expressed as

$${\gamma }_1={\left[\frac{\mathrm{tr}\left({\mathbf{F}}_{\mathrm{RF}}{\mathbf{F}}_{\mathrm{U}}{\mathbf{F}}_{\mathrm{U}}^H{\mathbf{F}}_{\mathrm{RF}}^H\right)}{P_{\mathrm{B}}}\right]}^{-\frac{1}{2}}.$$

(38)

Upon having fixed ${\mathbf{F}}_{\mathrm{RF}}$, we can obtain ${\mathbf{F}}_{\mathrm{U}}$ by setting the partial derivatives of the Lagrange function $\mathrm{L}1\left({\lambda }_1,{\mathbf{F}}_{\mathrm{U}},{\gamma }_1\right)$ to zero:

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\mathrm{U}}& ={\left({\mathbf{F}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}+{\mathbf{C}}_{11}\right)}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W},\hfill \end{array}$$

(39)

where ${\mathbf{C}}_{11}=c_{11}{\mathbf{F}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}+c_{12}{\mathbf{F}}_{\mathrm{RF}}^H{\mathbf{\Theta }}_1{\mathbf{F}}_{\mathrm{RF}}+c_{13}{\mathbf{F}}_{\mathrm{RF}}^H{\mathbf{F}}_{\mathrm{RF}}$, $c_{11}=\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right)$, $c_{12}=\mathrm{tr}\left({\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}\right)+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right)\mathrm{tr}\left(\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right)$ and $c_{13}={\gamma }_2^{-2}\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{B}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right)+\frac{{\sigma }_{\mathrm{R}}^2}{P_{\mathrm{B}}}\mathrm{tr}\left({\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}\right)+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{B}}}\mathrm{tr}\left(\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right)\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right)$. Substituting (38) and (39) into (37a), the optimization objective can be further expressed as

$$\begin{array}{cc}\hfill \mathrm{J}1\left({\mathbf{F}}_{\mathrm{RF}}\right)& =\mathrm{tr}\left\{\left[{\mathbf{I}}_{N_{\mathrm{S}}}+\left({\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}\right)\right.{\left.{\mathbf{C}}_{11}^{-1}{\left({\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}\right)}^H\right]}^{-1}\right\}.\hfill \end{array}$$

(40)

The sub-problem with only constant-modulus constraint can be re-formulated as

$$\begin{array}{cc}\hfill \underset{{\mathbf{F}}_{\mathrm{RF}}}{\min }& \mathrm{J}1\left({\mathbf{F}}_{\mathrm{RF}}\right)\hfill \end{array}$$

(41a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{F}}_{\mathrm{RF}}\right]}_{a,b}\right|}^2=1,\forall a,b.\hfill \end{array}$$

(41b)

By mathematical derivation, we can obtain the gradient of $\mathrm{J}1\left({\mathbf{F}}_{\mathrm{RF}}\right)$ with respect to ${\mathbf{F}}_{\mathrm{RF}}$:

$$\begin{array}{cc}\hfill \nabla \mathrm{J}1\left({\mathbf{F}}_{\mathrm{RF}}\right)& =\left(c_{11}{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{C}}_{11}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H\right.\hfill \\ \hfill & \left.+c_{12}{\mathbf{\Theta }}_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{C}}_{11}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H-{\mathbf{I}}_{N_{\mathrm{T}}}+c_{13}{\mathbf{F}}_{\mathrm{RF}}{\mathbf{C}}_{11}^{-1}{\mathbf{F}}_{\mathrm{RF}}^H\right)\hfill \\ \hfill & \times {\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{WC}}_{12}^{-2}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1{\mathbf{F}}_{\mathrm{RF}}{\mathbf{C}}_{11}^{-1}.\hfill \end{array}$$

(42)

(2) The Relay Node Design: First, we consider the HBF design for the receiver of the relay node. It is assumed that $\mathbf{F}$, ${\mathbf{G}}_2$ and $\mathbf{W}$ are fixed. Taking SI constraint out of consideration, the optimization problem can be formulated as

$$\begin{array}{cc}\hfill \underset{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1},{\mathbf{G}}_{{\mathrm{B}}_1}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(43a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}_{c,d}\right|}^2=1,\forall c,d.\hfill \end{array}$$

(43b)

with ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$ fixed, the optimization problem (43) can be recast as an unconstrained optimization problem. By taking derivative, we have the closed-form solution of ${\mathbf{G}}_{{\mathrm{B}}_1}$ as

$$\begin{array}{cc}\hfill {\mathbf{G}}_{{\mathrm{B}}_1}& =\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right.+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{\Phi }}_1{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\left.+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}^{-1}\hfill \\ \hfill & \times \left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\overline{\mathbf{G}}}_2\right)\left[{\overline{\mathbf{G}}}_2^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\overline{\mathbf{G}}}_2\right.{\left.+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right){\overline{\mathbf{G}}}_2^H{\mathbf{\Theta }}_2{\overline{\mathbf{G}}}_2\right]}^{-1}.\hfill \end{array}$$

(44)

Substituting the optimization problem (43) with Equation (44), we have

$$\begin{array}{cc}\hfill \underset{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}}{\min }& \mathrm{Q}1\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right)\hfill \end{array}$$

(45a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right]}_{c,d}\right|}^2=1,\forall c,d,\hfill \end{array}$$

(45b)

$$\mathrm{Q}1\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right)=\mathrm{tr}\left({\mathbf{I}}_{N_{\mathrm{RF}}}-{\mathbf{D}}_{11}^{-1}{\mathbf{D}}_{12}{\mathbf{D}}_{13}^{-1}{\mathbf{D}}_{12}^H\right),$$

(46)

with ${\mathbf{D}}_{11}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{\Phi }}_1$ ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$, ${\mathbf{D}}_{12}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}^H$${\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\overline{\mathbf{G}}}_2$ and ${\mathbf{D}}_{13}={\overline{\mathbf{G}}}_2^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\overline{\mathbf{G}}}_2+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right){\overline{\mathbf{G}}}_2^H{\mathbf{\Theta }}_2{\overline{\mathbf{G}}}_2$. The Euclidean gradient of $\mathrm{Q}1\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right)$ is given as

$$\begin{array}{cc}\hfill \nabla \mathrm{Q}1\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}\right)& =\left[{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{D}}_{11}^{-1}{\mathbf{D}}_{12}\right.\hfill \\ \hfill & +\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{\Phi }}_1{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{D}}_{11}^{-1}{\mathbf{D}}_{12}\hfill \\ \hfill & +{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}{\mathbf{D}}_{11}^{-1}{\mathbf{D}}_{12}\hfill \\ \hfill & \left.-{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\overline{\mathbf{G}}}_2\right]{\mathbf{D}}_{13}^{-1}{\mathbf{D}}_{12}^H{\mathbf{D}}_{11}^{-1}.\hfill \end{array}$$

(47)

After finishing the design of the receiver, we turn to the transmitter. For any given $\mathbf{F}$, ${\mathbf{G}}_1$ and $\mathbf{W}$, we have the optimization sub-problem as follows by ignoring the SI constraint:

$$\begin{array}{cc}\hfill \underset{{\gamma }_2,{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2},{\mathbf{G}}_{{\mathrm{U}}_2}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(48a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}_{c,d}\right|}^2=1,\forall c,d,\hfill \end{array}$$

(48b)

$$\begin{array}{c}{\mathrm{E}}_{\Delta {\mathbf{H}}_1}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}}.\hfill \end{array}$$

(48c)

Likewise, we can obtain ${\gamma }_2$ by taking transposition of (48b), which is given as

$$\begin{array}{cc}\hfill {\gamma }_2& =\left[\mathrm{tr}\left({\gamma }_1^2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H\right.\right.\hfill \\ \hfill & {\left.\left.+{\gamma }_1^2\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H+{\sigma }_{\mathrm{R}}^2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H\right)/\phantom{\left.+{\gamma }_1^2\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H+{\sigma }_{\mathrm{R}}^2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H\right)P_{\mathrm{R}}}{P}_{\mathrm{R}}\right]}^{-\frac{1}{2}}\hfill \end{array}$$

(49)

To this end, the optimization problem (48) can be equivalently recast as an optimization problem (50) with only power constraint by fixing ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$. Based on the Lagrange function, the closed form of ${\mathbf{G}}_{{\mathrm{U}}_2}$ is expressed as (51).

$$\begin{array}{cc}\hfill \underset{{\gamma }_2,{\mathbf{G}}_{{\mathrm{U}}_2}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(50a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathrm{s}.\mathrm{t}.{\mathrm{E}}_{\Delta {\mathbf{H}}_1}\left\{{∥{\mathbf{x}}_{\mathrm{R}}∥}_{\mathrm{F}}^2\right\}=P_{\mathrm{R}},\hfill \end{array}$$

(50b)

$$\begin{array}{cc}\hfill {\mathbf{G}}_{{\mathrm{U}}_2}& =\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\overline{\mathrm{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right.+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{\Theta }}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\hfill \\ \hfill & {\left.+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}^{-1}\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_1\right)\hfill \\ \hfill & \times {\left[{\mathbf{G}}_1^H{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_1+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{G}}_1^H{\mathbf{\Phi }}_1{\mathbf{G}}_1+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_1^H{\mathbf{G}}_1\right]}^{-1},\hfill \end{array}$$

(51)

where we define ${\mathbf{D}}_{23}={\mathbf{G}}_1^H{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_1+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right){\mathbf{G}}_1^H{\mathbf{\Phi }}_1{\mathbf{G}}_1+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2{\mathbf{G}}_1^H{\mathbf{G}}_1$, ${\mathbf{D}}_{21}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{\Theta }}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}+\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$ and ${\mathbf{D}}_{22}={\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}^H{\overline{\mathbf{H}}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\mathbf{G}}_1$. One further step, we can get the simplified optimization objective (52), the optimization problem (53) and its gradient (54) in a similar way.

$$\mathrm{Q}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)=\mathrm{tr}\left({\mathbf{I}}_{N_{\mathrm{RF}}}-{\mathbf{D}}_{21}^{-1}{\mathbf{D}}_{22}{\mathbf{D}}_{23}^{-1}{\mathbf{D}}_{22}^H\right),$$

(52)

$$\begin{array}{cc}\hfill \underset{{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}}{\min }& \mathrm{Q}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)\hfill \end{array}$$

(53a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right]}_{e,f}\right|}^2=1,\forall e,f,\hfill \end{array}$$

(53b)

$$\begin{array}{cc}\hfill \nabla \mathrm{Q}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}\right)& =\left[\left[{\overline{\mathbf{H}}}_2^H\mathbf{W}{\mathbf{W}}^H{\overline{\mathbf{H}}}_2\right.{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{D}}_{21}^{-1}{\mathbf{D}}_{22}\right.+\mathrm{tr}\left({\mathbf{W}}^H{\mathbf{\Phi }}_2\mathbf{W}\right){\mathbf{\Theta }}_2{\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{D}}_{21}^{-1}{\mathbf{D}}_{22}\hfill \\ \hfill & +\frac{{\sigma }_{\mathrm{D}}^2}{P_{\mathrm{R}}}\mathrm{tr}\left({\mathbf{W}}^H\mathbf{W}\right){\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}{\mathbf{D}}_{21}^{-1}{\mathbf{D}}_{22}\left.-{\overline{\mathbf{H}}}_2^H\mathbf{W}{\overline{\mathbf{F}}}^H{\mathbf{H}}_1^H{\mathbf{G}}_1\right]{\mathbf{D}}_{23}^{-1}{\mathbf{D}}_{22}^H{\mathbf{D}}_{21}^{-1}.\hfill \end{array}$$

(54)

It’s worth noting that we temporarily neglect the SI constraint in the HBF design of the relay node. Consequently, we need to design the SI suppression matrix to satisfy ${\mathbf{G}}_1^H{\overline{\mathbf{H}}}_{\mathrm{SI}}{\mathbf{G}}_2=\mathbf{0}$. Severe SI acts as a key factor to affect the performance of FD communication systems. There are strict restrictions on the number of RF chains in traditional SI suppression algorithm based on NP. In [31], it is required to meet $N_{\mathrm{RF},\mathrm{T}}\ge N_{\mathrm{S}}+N_{\mathrm{RF},\mathrm{R}}$, with $N_{\mathrm{RF},\mathrm{T}}$ and $N_{\mathrm{RF},\mathrm{R}}$ being the number of the transmit and receive RF chains at the AF relay node, while [35] needs to satisfy $N_{\mathrm{RF},\mathrm{R}}\ge 2N_{\mathrm{S}}$. The authors of [30] proposed an SI suppression algorithm suitable for the system with $N_{\mathrm{RF},\mathrm{T}}=N_{\mathrm{RF},\mathrm{R}}=N_{\mathrm{S}}$. Nevertheless, its performance remains to be improved. On the basis of the above literature, we further develop the SIC-NP algorithm.

We first construct an equivalent SI channel matrix ${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}\in {\mathbb{C}}^{N_{\mathrm{S}}\times N_{\mathrm{R}}}$, which can be written as

$${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}={\mathbf{G}}_1^H{\overline{\mathbf{H}}}_{\mathrm{SI}}.$$

(55)

It is obvious that the rank of ${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}$ is $N_{\mathrm{S}}\left(N_{\mathrm{R}}\gg N_{\mathrm{S}}\right)$, so its null-space definitely exists. There are a number of ways to obtain the null-space of a matrix. We adopt the singular value decomposition (SVD) method in this paper, which yields

$${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}={\overline{\mathbf{U}}}_{\mathrm{SI}-\mathrm{EQ}}{\overline{\mathbf{\Sigma }}}_{\mathrm{SI}-\mathrm{EQ}}{\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^H,$$

(56)

where ${\overline{\mathbf{U}}}_{\mathrm{SI}-\mathrm{EQ}}\in {\mathbb{C}}^{N_{\mathrm{S}}\times N_{\mathrm{S}}}$ and ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{R}}}$ are unitary matrixes, ${\overline{\mathbf{\Sigma }}}_{\mathrm{SI}-\mathrm{EQ}}\in {\mathbb{C}}^{N_{\mathrm{S}}\times N_{\mathrm{R}}}$ is a diagonal matrix. ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}$ can be further decomposed as ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}=\left[{\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^1{\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^0\right]$, where ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^1\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{S}}}$ and ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^0\in {\mathbb{C}}^{N_{\mathrm{R}}\times \left(N_{\mathrm{R}}-N_{\mathrm{S}}\right)}$ are the right singular matrix for non-zero singular value and zero singular value. And the latter satisfies

$${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}{\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^0=\mathbf{0}.$$

(57)

In this correspondence, we can construct the null-space matrix $\mathbf{P}\in {\mathbb{C}}^{N_{\mathrm{R}}\times N_{\mathrm{R}}}$ of ${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}$ using the column vectors of ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^0$. Subsequently, ${\mathbf{G}}_2$ is projected to $\mathbf{P}$ to mitigate the SI. The projection matrix ${\mathbf{P}}_1$ and SI suppression matrix ${\mathbf{G}}_2^{\mathrm{SIC}}$ can be written as

$${\mathbf{P}}_1=\mathbf{P}{\left({\mathbf{P}}^H\mathbf{P}\right)}^{-1}{\mathbf{P}}^H,$$

(58)

$${\mathbf{G}}_2^{\mathrm{SIC}}={\mathbf{P}}_1{\mathbf{G}}_2.$$

(59)

Hence, the HBF matrix of FD AF relay node takes the form of

$$\mathbf{G}={\mathbf{G}}_2^{\mathrm{SIC}}{\mathbf{G}}_1^H.$$

(60)

For clarity, the above SI suppression procedure is summarized in Algorithm 2.

Algorithm 2 SIC-NP Algorithm.

Input:  ${\mathbf{G}}_1$, ${\mathbf{G}}_2$, ${\overline{\mathbf{H}}}_{\mathrm{SI}}$. Output:  ${\mathbf{G}}_2^{\mathrm{SIC}}$.   1: Calculate the equivalent SI channel matrix ${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}$ according to (55).   2: Calculate ${\overline{\mathbf{V}}}_{\mathrm{SI}-\mathrm{EQ}}^0$ by taking SVD of ${\overline{\mathbf{H}}}_{\mathrm{SI}-\mathrm{EQ}}$.   3: Construct the null-space matrix $\mathbf{P}$.   4: Calculate ${\mathbf{P}}_1$ according to (58).   5: Compute ${\mathbf{G}}_2^{\mathrm{SIC}}$ according to (59).

(3) The Destination Node Design: Similar to that of Section II, the optimization sub-problem of the destination node has only constant-modulus constraint. For any given source and relay HBF matrixes, the destination sub-problem can be formulated as

$$\begin{array}{cc}\hfill \underset{{\mathbf{W}}_{\mathrm{RF}},{\mathbf{W}}_{\mathrm{B}}}{\min }& \mathrm{IMSE}\hfill \end{array}$$

(61a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{W}}_{\mathrm{RF}}\right]}_{g,h}\right|}^2=1,\forall g,h.\hfill \end{array}$$

(61b)

with ${\mathbf{W}}_{\mathrm{RF}}$ fixed, the problem (61) can be transformed into an unconstrained optimization problem. By setting the gradient of the objective function to zero, we can obtain

$$\begin{array}{c}\hfill {\mathbf{W}}_{\mathrm{B}}={\left({\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}+{\mathbf{C}}_{31}\right)}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}},\end{array}$$

(62)

where $c_{31}=\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)$, $c_{32}={\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2$, $c_{33}=\mathrm{tr}\left(\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right)+\mathrm{tr}\left(\overline{\mathbf{F}}{\overline{\mathbf{F}}}^H{\mathbf{\Theta }}_1\right)\mathrm{tr}\left(\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right)+{\gamma }_1^{-2}{\sigma }_{\mathrm{R}}^2\mathrm{tr}\left(\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\mathbf{\Theta }}_2\right)$, $c_{34}={\gamma }_1^{-2}{\gamma }_2^{-2}{\sigma }_{\mathrm{D}}^2$, and we define

$$\begin{array}{cc}\hfill {\mathbf{C}}_{31}& =c_{31}{\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}+c_{32}{\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}\hfill \\ \hfill & +c_{33}{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{\Phi }}_2{\mathbf{W}}_{\mathrm{RF}}+c_{34}{\mathbf{W}}_{\mathrm{RF}}^H{\mathbf{W}}_{\mathrm{RF}}.\hfill \end{array}$$

(63)

Substituting (62) into the equation of IMSE, the objective function is equivalent to

$$\begin{array}{c}\mathrm{J}3\left({\mathbf{W}}_{\mathrm{RF}}\right)=\mathrm{tr}\left[\left({\mathbf{I}}_{N_{\mathrm{S}}}\right.{\left.+{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{C}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}\right)}^{-1}\right].\hfill \end{array}$$

(64)

In the meantime, the optimization sub-problem is reduced to

$$\begin{array}{cc}\hfill \underset{{\mathbf{W}}_{\mathrm{RF}}}{\min }& \mathrm{J}3\left({\mathbf{W}}_{\mathrm{RF}}\right)\hfill \end{array}$$

(65a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\left|{\left[{\mathbf{W}}_{\mathrm{RF}}\right]}_{g,h}\right|}^2=1,\forall g,h.\hfill \end{array}$$

(65b)

Finally, the gradient $\nabla \mathrm{J}3\left({\mathbf{W}}_{\mathrm{RF}}\right)$ can be written as

$$\begin{array}{cc}\hfill \nabla \mathrm{J}3\left({\mathbf{W}}_{\mathrm{RF}}\right)& =\left[\left(c_{33}{\mathbf{\Phi }}_2+c_{31}{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\mathbf{\Phi }}_1{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H\right.+c_{32}{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H+c_{34}{\mathbf{I}}_{N_{\mathrm{D}}}\right)\hfill \\ \hfill & \times {\mathbf{W}}_{\mathrm{RF}}{\mathbf{C}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H\left.-{\mathbf{I}}_{N_{\mathrm{D}}}\right]{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}{\mathbf{C}}_{32}^{-2}{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{C}}_{31}^{-1},\hfill \end{array}$$

(66)

with ${\mathbf{C}}_{32}={\mathbf{I}}_{N_{\mathrm{S}}}+{\overline{\mathbf{F}}}^H{\overline{\mathbf{H}}}_1^H{\overline{\mathbf{G}}}^H{\overline{\mathbf{H}}}_2^H{\mathbf{W}}_{\mathrm{RF}}{\mathbf{C}}_{31}^{-1}{\mathbf{W}}_{\mathrm{RF}}^H{\overline{\mathbf{H}}}_2\overline{\mathbf{G}}{\overline{\mathbf{H}}}_1\overline{\mathbf{F}}$.

The proposed MO based HBF algorithm for the FD relay system is summarized in Algorithm 3.

Algorithm 3 HBF-MO-FD-R Algorithm.

Input:  ${\overline{\mathbf{H}}}_1$, ${\overline{\mathbf{H}}}_2$, ${\overline{\mathbf{H}}}_{\mathrm{SI}}$, ${\sigma }_{\mathrm{R}}^2$, ${\sigma }_{\mathrm{D}}^2$, ${\mathbf{\Phi }}_1$, ${\mathbf{\Phi }}_2$, ${\mathbf{\Theta }}_1$, ${\mathbf{\Theta }}_2$. Output:  ${\gamma }_1$, ${\gamma }_2$, ${\mathbf{F}}_{\mathrm{RF}}$, ${\mathbf{F}}_{\mathrm{U}}$, ${\mathbf{G}}_1$, ${\mathbf{G}}_2^{\mathrm{SIC}}$, ${\mathbf{W}}_{\mathrm{RF}}$, ${\mathbf{W}}_{\mathrm{B}}$.    1: $\mathbf{Initialize}$ ${\mathbf{F}}_{\mathrm{RF}}$, ${\mathbf{F}}_{\mathrm{U}}$, ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1}$, ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2}$, ${\mathbf{G}}_{{\mathrm{B}}_1}$, ${\mathbf{G}}_{{\mathrm{U}}_2}$, ${\mathbf{W}}_{\mathrm{RF}}$ and ${\mathbf{W}}_{\mathrm{B}}$ randomly, and set ${\gamma }_1={\gamma }_2=1$, $k=1$.    2: while  $\mathrm{IMS}{\mathrm{E}}_k-\mathrm{IMS}{\mathrm{E}}_{k-1}>\epsilon$  do    3:    Calculate $\nabla \mathrm{J}1\left({\mathbf{F}}_{\mathrm{RF},k}\right)$ according to (42) and compute ${\mathbf{F}}_{\mathrm{RF},k+1}$ by using the MO method.    4:    Update ${\gamma }_{1,k+1}$, ${\mathbf{F}}_{\mathrm{U},k+1}$ and ${\overline{\mathbf{F}}}_{k+1}$ according to (38) and (39).    5:    Calculate $\nabla \mathrm{Q}1\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1,k}\right)$ according to (47) and compute ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_1,k+1}$ by using the MO method.    6:    Update ${\mathbf{G}}_{{\mathrm{B}}_1,k+1}$ and ${\mathbf{G}}_{1,k+1}$ according to (44).    7:    Calculate $\nabla \mathrm{Q}2\left({\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2,k}\right)$ according to (54) and compute ${\mathbf{G}}_{\mathrm{R}{\mathrm{F}}_2,k+1}$ by using the MO method.    8:    Update ${\gamma }_{2,k+1}$, ${\mathbf{G}}_{{\mathrm{U}}_2,k+1}$ and ${\overline{\mathbf{G}}}_{2,k+1}$ according to (49) and (51).    9:    Calculate $\nabla \mathrm{J}3\left({\mathbf{W}}_{\mathrm{RF},k}\right)$ according to (66) and compute ${\mathbf{W}}_{\mathrm{RF},k+1}$ by using the MO method.  10:    Update ${\mathbf{W}}_{\mathrm{B},k+1}$ and ${\mathbf{W}}_{k+1}$ according to (62).  11:    $k\leftarrow k+1$.  12: end  while  13: Compute ${\mathbf{G}}_2^{\mathrm{SIC}}$ by using the SIC-NP algorithm and update $\mathbf{G}$ according to (60).  14: Power normalization.

5. Simulation Results

To evaluate the performance of the proposed HBF algorithms, simulation results are presented in this section. In the simulations, it is assumed that the SNRs between the relay–destination link and the source-relay link are identical. Unless otherwise mentioned, the number of antennas at the source, relay and destination nodes are $N_{\mathrm{T}}=64$, $N_{\mathrm{R}}=48$ and $N_{\mathrm{D}}=32$, respectively. The number of RF chains and data streams are set as $N_{\mathrm{RF}}=N_{\mathrm{S}}=2$. Due to the limited scattering characteristics of mmWave channels, we assume $L=10$. The Rician factor is chosen as 20 dB. All the AoDs and AoAs are independently and uniformly distributed in $\left[0,2\pi \right]$. Moreover, we define the same source and relay transmit power, i.e., $P_{\mathrm{B}}=P_{\mathrm{R}}=2$. In this paper, all simulation results are averaged over 1000 mmWave channels, which are generated randomly.

5.1. HBF-MO

Firstly, we present the performance of the HBF-MO algorithm (denoted as HBF-MO) for the HD relay system. In this subsection, we will consider a benchmark approach and two other different algorithms, which are the optimal digital processing based on SVD (denoted as DBF), the HBF algorithm in [14] (denoted as HBF) and the OMP based algorithm in [7] (denoted as OMP). Unless otherwise specified, the convergence parameter of the outer loop is set to ${10}^{-4}$, and that of the inner loop is set to ${10}^{-6}$.

Figure 3 validates the convergence of SE and MSE with SNR set to 6dB. To be specific, the number of outer iterations is fixed to 15. As can be observed, our proposed algorithm converges to an optimal solution within about four iterations. SE increases with the growth of iteration number and convergences to almost 5.485 bit/s/Hz, while MSE decreases and maintain at 0.0449 after 15 iterations.

Next, the SE performance versus SNR of our proposed HBF-MO algorithm and three other approaches is investigated in Figure 4. As shown in the figure, the SE of all algorithms increases with growing SNR. Taking the SE of DBF method as a benchmark, the proposed algorithm prevails over two other methods by approaching the DBF algorithm, which is more obvious as SNR becomes higher. When SNR = 10 dB, our HBF-MO algorithm can provide a gain of 7.99% over HBF [14] and 11% over OMP [7].

By varying the number of relay and destiantion antennas, the SE performance of all algorithms are evaluated when SNR = 6dB in Figure 5 and Figure 6. Likewise, the DBF method is adopted as a benchmark. Due to the additional antenna gain, the spectral efficiency of all algorithms improves with the number of antennas increasing. Still, our proposed algorithm yields the best performance over the others.

5.2. HBF-MO-FD-R

In this subsection, simulation results for the proposed robust HBF scheme for the FD relay system is presented. For notation convenience, “HBF-MO-FD-R” represents our proposed method, “HBF-FD-SIC” denotes the HBF-MO-FD-R algorithm with SIC-NP replaced by the SIC method in [30], “HBF-FD-NSIC” stands for the proposed method without SIC-NP, while “HBF-MO-HD” denotes the proposed HD algorithm with imperfect CSI. Additionally, it is assumed that $a_0=b_0=\psi =0$ [15]. Unless otherwise stated, the parameters related to channel estimation errors are set as $\alpha =0.6$, $\beta =0.4$ and ${\sigma }_e^2=0.5$. Meanwhile, the Interference to Noise Ratio (INR) is defined as $\mathrm{INR}=10lg\left(P_{\mathrm{th}}/{\sigma }_{\mathrm{R}}^2\right)$ [30,35].

In Figure 7, we show the convergence performance of HBF-MO-FD-R with respect to iterations at SNR = 6dB and INR = 10dB, where the maximum number of iterations is set to 15. It can be observed that SE converges to a maximum of above 2.5 bits/s/Hz in about seven iterations. On the other hand, MSE decreases to a minimum of almost 0.18 after about five iterations.

Figure 8 compares the SE performance of four different algorithms. As expected, SE of all approaches improves monotonically with increasing SNR. It can be obviously seen that both the HBF-MO-FD-R algorithm and the HBF-FD-SIC algorithm obtain higher SE by suppressing the LOS SI in contrast with the method without SIC. Additionally, the performance of the two methods is still affected by the residual SI because the NLOS channel cannot be accurately estimated. As is shown in the figure, with the increase in INR, the SE of HBF-MO-FD-R and HBF-FD-SIC decreases, but that is still higher than the non-SIC method. Meanwhile, we find that our proposed algorithm outperforms the HBF-FD-SIC approach. The average gap between two approaches is about 0.913 bit/s/Hz when INR = 10 dB and 0.791 bit/s/Hz for INR = 20 dB. Moreover, the non-robust algorithm can obtain more SE than the non-SIC algorithm when SI is severe (INR = 20 dB). It is noteworthy that the value of INR has no impact on the performance of HBF-MO-HD because there is no SI channel in HD mode.

Figure 9 depicts the SE performance with various CSI errors. We analysis how the SE performance is affected by estimation error variance with $\alpha =0.6$, $\beta =0.4$ and INR = 10 dB. As we can see, imperfect CSI will lead to performance degradation in all approaches, whereas our algorithm still has the best performance, which illustrates the effectiveness of the robust design and SIC.

6. Conclusions

We have investigated the joint HBF design for mmWave AF relay communication systems in this paper. Specifically, the optimization-based HBF design for HD relay systems with perfect CSI has been addressed. We have designed an alternating optimization algorithm based on MO, where the modified MSE was adopted as the optimization objective. Simulation results have shown that the performance prevailed other HBF algorithms in the literature. Moreover, we have further extended the scenario to more general cases with channel estimation errors, where the relay system was operated in FD mode. A robust HBF design has been proposed, with SI mitigated by the NP based algorithm. It has been demonstrated via numerical results that the proposed scheme had an improved robustness against SI.