Reinforcement Learning-Aided Channel Estimator in Time-Varying MIMO Systems

Abstract

This paper proposes a reinforcement learning-aided channel estimator for time-varying multi-input multi-output systems. The basic concept of the proposed channel estimator is the selection of the detected data symbol in the data-aided channel estimation. To achieve the selection successfully, we first formulate an optimization problem to minimize the data-aided channel estimation error. However, in time-varying channels, the optimal solution is difficult to derive because of its computational complexity and the time-varying nature of the channel. To address these difficulties, we consider a sequential selection for the detected symbols and a refinement for the selected symbols. A Markov decision process is formulated for sequential selection, and a reinforcement learning algorithm that efficiently computes the optimal policy is proposed with state element refinement. Simulation results demonstrate that the proposed channel estimator outperforms conventional channel estimators by efficiently capturing the variation of the channels.

1. Introduction

The multi-input multi-output (MIMO) system is a key technology in modern communication and can significantly improve channel capacity and communication reliability by using multiple antennas [1,2,3,4,5,6,7]. Spatial multiplexing and diversity gain are representative schemes for this improvement [1,2]. Notably, the channel capacity increases linearly with the number of either transmitter and receiver antennas. However, this increase is based on the unrealistic assumption of perfect channel state information (PCSI) at both the transmitter and receiver.

Many studies have proposed improving the channel estimation accuracy with limited time and frequency resources [8,9,10,11,12,13,14,15]. A representative method is pilot-aided channel estimation, which exploits the information shared between a transmitter and receiver. Linear minimum-mean square-error (LMMSE) channel estimation is a well-known method for pilot-aided channel estimation, owing to its simple structure [8]. However, LMMSE channel estimation exhibits unsatisfactory performance with a limited number of pilots. Thus, many pilots are required to satisfy the performance requirement, which decreases the spectral efficiency.

To overcome this problem, data-aided channel estimation has been investigated in which the detected data symbols are exploited as additional pilot symbols [16,17,18,19,20,21,22,23,24,25,26]. However, the detected data symbols may have errors that degrade the accuracy of channel estimation. The iterative turbo equalizer can overcome this degradation by increasing the maximum-a-posteriori probability (MAP) [16,17,18,19,20,21,22]. However, such an iterative turbo equalizer has considerable complexity and latency at the receivers.

As a non-iterative approach, the reinforcement learning (RL)-aided channel estimator was introduced in [27,28,29,30,31,32,33]. The basic concept of this approach is the sequential selection of detected data symbols to minimize the channel estimation errors. Hence, a Markov decision process (MDP) was defined to solve the sequential selection, and the corresponding optimal policy was derived in a closed-form expression in [31]. In [32], a low-complexity algorithm was investigated by introducing sub-blocks and finite backup samples, and the computational complexity and latency were significantly reduced without performance loss. Recently, a general framework for RL-aided channel estimation was studied in [33] based on Monte Carlo tree search. However, the RL-aided channel estimators in [31,32,33] were originally considered in time-invariant channels; they perform insufficiently in time-varying channels.

In this paper, we propose an RL-aided channel estimator for time-varying MIMO channels. To achieve this, we first introduce an optimization problem for an RL-aided channel estimator in time-varying channels. We then formulate an MDP to solve the optimization problem, and propose an RL algorithm for the MDP that considers the time-varying nature of the channel. The main contributions of this paper are as follows:

• We propose an RL-aided channel estimator for time-varying channels modeled using a first-order Gaussian—Markov process. First, we define the optimization problem in time-varying channels to select the detected data symbols and minimize the estimation error between the estimated and current channels. This optimization problem is different from those in [31,32,33], where the selection of the detected data symbols is unchanged because the current channel remains unchanged with the time slot index.
• We propose an RL algorithm for the optimization problem that captures the time-varying nature of a channel. Because the optimization problem minimizes the estimation error between the estimated and current channels, we adjust the weights of the data symbols to improve the channel accuracy of the current channel. Using this adjustment, we derive the optimal policy as a closed-form solution. Note that the proposed optimal policy differs from those in [31,32,33] because the influence of soft-decision symbols in the virtual state for future rewards gradually diminishes as the time slot index increases.
• We propose a further performance improvement scheme to refine the state elements. This is because the previously selected data symbol degrades the estimation accuracy of the current channel. To improve the estimation accuracy, we refine the previously selected data symbol by reflecting the channel variation. In addition, we remove selected data symbols that are too old by introducing a sliding window, because they have a large noise variance to estimate the current channel. Through simulations, we demonstrated the effectiveness of the proposed channel estimator compared with conventional channel estimators in time-varying channels.

The remainder of this paper is organized as follows. In Section 1, we introduce the system model, optimization problem, and the MDP. The proposed channel estimator, which determines the optimal policy for time-varying channels, is described in Section 2. We propose a further performance improvement scheme in Section 3. In Section 4, we present simulation results to demonstrate the effectiveness of the proposed channel estimator. Finally, we provide our conclusions in Section 5.

2. Preliminaries

This section describes the system model of a data-aided channel estimator for time-varying MIMO channels. We present the considered channel estimation and data detection schemes based on the model and introduce an optimization problem for data-aided channel estimation.

2.1. System Model

We consider MIMO systems in which a transmitter with a number of transmit antennas $N_t$ communicates with a receiver with a number of receive antennas $N_r$ (Figure 1). The information is first encoded and mapped to the symbol constellation where $\mathcal{X}$ is the symbol constellation set. The transmitted symbol at time n denoted by $\mathbf{x}\left[n\right]\in {\mathcal{X}}^{N_t}$ is then sent over a wireless channel. We model the wireless channel using a first-order Gaussian—Markov process as a time-varying channel model [34,35,36,37,38], where the channel matrix $\mathbf{H}\left[n\right]\in {\mathbf{C}}^{N_t\times N_r}$ has its $(t,r)$-th component between the t-th and r-th antennas following a Rayleigh fading $\mathcal{CN}(0,1)$ distribution. The temporal correlation of the wireless channel, denoted by $ϵ\in [0,1]$, increases with velocity. Based on this model, the channel matrix $\mathbf{H}\left[n\right]$ at time slot n is given by

$$\begin{array}{cc}\hfill \mathbf{H}\left[n\right]& =\sqrt{1-{ϵ}^2}\mathbf{H}[n-1]+ϵ\mathsf{\Delta }\left[n\right],\hfill \end{array}$$

(1)

where $\mathsf{\Delta }\left[n\right]$ follows a $\mathcal{CN}(0,1)$ distribution.

When the transmitter sends the symbol $\mathbf{x}\left[n\right]$ to the receiver over the wireless channel $\mathbf{H}\left[n\right]$, the received symbol $\mathbf{z}\left[n\right]$ is given by

$$\begin{array}{cc}\hfill \mathbf{z}\left[n\right]& ={\mathbf{H}}^H\left[n\right]\mathbf{x}\left[n\right]+\mathbf{n}\left[n\right],\hfill \end{array}$$

(2)

where ${(·)}^H$ denotes the conjugate transpose. $\mathbf{n}\left[n\right]$ is the additive white Gaussian noise (AWGN) at time slot n, with distribution $\mathcal{CN}({\mathbf{0}}_{N_r},{\sigma }_n^2{\mathbf{I}}_{N_r})$, where ${\mathbf{0}}_m$ and ${\mathbf{I}}_m$ respectively denote $m\times m$ zero and identity matrices.

The frame consists of one pilot and $M_d$ data blocks (Figure 1). The pilot block contains $N_p$ symbols, whereas each data block contains $N_d$ symbols. ${\mathcal{M}}_p=\{1,\dots ,N_p\}$ is defined as the pilot index set and ${\mathcal{M}}_d=\{(d-1)N_d+1,\dots ,d{N}_d\}$ is defined as the data index set. We consider data-aided channel estimation, where the receiver obtains the initial channel estimates using pilot symbols, and the accuracy of the initial channel estimates is improved by exploiting data symbols.

We adopt the LMMSE method as the basic channel estimation method because it has a simple structure and provides a reasonable performance. Based on the LMMSE method, ${\widehat{\mathbf{h}}}_r$ of the r-th row for the initial channel estimate $\widehat{\mathbf{H}}$ can be obtained as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r& ={\left({\mathbf{X}}^p{\left({\mathbf{X}}^p\right)}^H+{\sigma }_n^2{\mathbf{I}}_{N_t}\right)}^{-1}{\mathbf{X}}^p{\left({\mathbf{z}}_r^p\right)}^H,\hfill \end{array}$$

(3)

where ${(·)}^{-1}$ is the inverse operation. ${\mathbf{X}}^p={[\mathbf{x}\left[1\right],\dots ,\mathbf{x}\left[N_p\right]]}^T$ and ${\mathbf{z}}_r^p={[{\mathbf{z}}_r\left[1\right],\dots ,{\mathbf{z}}_r\left[N_p\right]]}^T$ are the pilot and corresponding received symbols in the pilot block, respectively.

The conventional channel estimator performs data detection at the receiver using the initial channel estimates $\widehat{\mathbf{h}}$. Because the MAP rule guarantees optimal performance, we adopt it for data detection, which is given by

$$\begin{array}{cc}\hfill \widehat{\mathbf{x}}\left[n\right]& =\underset{{\mathbf{x}}_k\in {\mathcal{X}}^{N_t}}{argmax}{\theta }_k\left[n\right].\hfill \end{array}$$

(4)

where $|·|$ is the cardinality of a set. ${\mathbf{x}}_k\in {\mathcal{X}}^{N_t}$ where k belongs to the index set of the symbol vector candidate $\mathcal{K}=\{1,\dots ,|{\mathcal{X}}^{N_t}\left|\right\}$. ${\theta }_k\left[n\right]$ denotes a posteriori probability (APP), which is given by

$$\begin{array}{cc}\hfill {\theta }_k\left[n\right]& =\frac{\mathbb{P}\left[\mathbf{z}\left[n\right]|\mathbf{x}\left[n\right]={\mathbf{x}}_k\right]\mathbb{P}\left[\mathbf{x}\left[n\right]={\mathbf{x}}_k\right]}{{\displaystyle \sum _{j\in \mathcal{K}}}\mathbb{P}\left[\mathbf{z}\left[n\right]|\mathbf{x}\left[n\right]={\mathbf{x}}_j\right]\mathbb{P}\left[\mathbf{x}\left[n\right]={\mathbf{x}}_j\right]},\hfill \end{array}$$

(5)

where the likelihood probability in (5) is calculated by assuming the AWGN channel as

$$\begin{array}{cc}\hfill \mathbb{P}\left[\mathbf{z}\left[n\right]|\mathbf{x}\left[n\right]={\mathbf{x}}_k\right]& =\frac{1}{{\left(\pi {\sigma }_n^2\right)}^{N_r}}{e}^{-\frac{\parallel \mathbf{z}\left[n\right]-{\widehat{\mathbf{h}}}^H{\mathbf{x}}_k{\parallel }^2}{{\sigma }_n^2}}.\hfill \end{array}$$

(6)

where ${\parallel ·\parallel }^2$ denotes the norm operation and $\mathbb{P}(·)$ is the probability of an event. The a priori probability in (5) is also assumed to have an equal probability for possible candidate transmitted symbol $\mathbb{P}\left[\mathbf{x}\left[n\right]={\mathbf{x}}_k\right]=\frac{1}{{\left|\mathcal{X}\right|}^{N_t}}$.

2.2. Problem

In a time-varying channel, the estimation accuracy of $\widehat{\mathbf{h}}$ decreases gradually as time slot index n increases. This degradation results in poor detection performance at the receiver. Because the detected data symbol may have an error owing to the channel, an incorrect use of the detected data symbol severely degrades performance. To overcome this degradation, we consider a data-aided channel estimator that selects the detected data symbols for data-aided channel estimation.

For the selection, we define action $a\in \mathcal{A}=\{0,1\}$ where the detected data symbol is used in channel estimation when $a=1$; otherwise, the detected data symbol is not used. When we define $\mathbf{a}\in {\{0,1\}}^{N_d}$ as a set of actions, the considered data-aided channel estimation can be obtained using this set as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r\left(\mathbf{a}\right)& ={\left(\widehat{\mathbf{X}}\left(\mathbf{a}\right){\widehat{\mathbf{X}}}^H\left(\mathbf{a}\right)+{\sigma }_n^2{\mathbf{I}}_{N_t}\right)}^{-1}\widehat{\mathbf{X}}\left(\mathbf{a}\right){\mathbf{z}}_r^H\left(\mathbf{a}\right)\hfill \end{array}$$

(7)

where ${\mathbf{z}}_r\left(\mathbf{a}\right)={[{\mathbf{z}}_r^p,{\mathbf{z}}_r\left[e_1\left(\mathbf{a}\right)\right],\dots ,{\mathbf{z}}_r\left[e_{{\parallel \mathbf{a}\parallel }_0}\left(\mathbf{a}\right)\right]]}^T$ and $\widehat{\mathbf{X}}\left(\mathbf{a}\right)={[{\mathbf{X}}^p,\widehat{\mathbf{x}}\left[e_1\left(\mathbf{a}\right)\right],\dots ,\widehat{\mathbf{x}}\left[e_{{\parallel \mathbf{a}\parallel }_0}\left(\mathbf{a}\right)\right]]}^T$. The time slot index of the i-th nonzero element is denoted as $e_i(·)$. We then define the optimization problem as

$$\begin{array}{cc}\hfill {\mathbf{a}}^{★}& =\underset{\mathbf{a}}{argmin}\mathcal{E}\{\parallel \widehat{\mathbf{h}}\left(\mathbf{a}\right)-\mathbf{H}\left[n\right]{\parallel }^2\},\hfill \end{array}$$

(8)

where $\mathcal{E}(·)$ is the expectation of a random variable.

Compared with previous studies [31,32,33], the optimization problem in (8) considers the selection to minimize the MSE between the estimated channel and $\mathbf{H}\left[n\right]$. Because the channel is variant with time slot index n, the best action ${\mathbf{a}}^{★}$ may be different with time slot index n. That is, the best action in the previous time slot index may be invalid in the next time slot index. In addition, the optimization problem is difficult to solve because the number of candidate actions increases exponentially with the data symbol length. An exhaustive search for action candidates is not feasible in practical applications. To resolve these difficulties, we introduce a sequential selection of the detected data symbols and a refinement of the selected data.

2.3. Markov Decision Process

We formulate an MDP that solves the optimization problem in (8). To achieve this, we define state ${\mathcal{S}}_n$, transition function ${\mathcal{T}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)$, action $\mathcal{A}$, and reward $\mathcal{R}({\mathsf{S}}_n,{\mathsf{S}}_{n+1})$ [39]. Subsequently, the Q-value function $\mathcal{Q}\left({\mathsf{S}}_n,a\right)$ and the optimal policy ${\pi }^{★}\left({\mathsf{S}}_n\right)$ will be presented. The basic definitions for the MDP are adopted from those in [31,32,33]; however, the RL solution for the MDP is different from those in previous studies, which will be explained in the next section.

The state set ${\mathcal{S}}_n$ is defined as

$$\begin{array}{cc}\hfill {\mathcal{S}}_n=\left\{\left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,\mathcal{C}\right)\right|& {\mathbf{X}}_n=\left[\mathbf{x}\left[1\right]\cdots \mathbf{x}\left[N_p\right],{\mathbf{x}}_{k_{\mathcal{C}\left(1\right)}},\cdots ,{\mathbf{x}}_{k_{\mathcal{C}\left(\left|\mathcal{C}\right|\right)}}\right],\hfill \\ \hfill & {\widehat{\mathbf{X}}}_n=\left[\mathbf{x}\left[1\right]\cdots \mathbf{x}\left[N_p\right],\widehat{\mathbf{x}}\left[\mathcal{C}\left(1\right)\right],\cdots ,\widehat{\mathbf{x}}\left[\mathcal{C}\left(\left|\mathcal{C}\right|\right)\right]\right],\hfill \\ \hfill & \mathcal{C}\subset \left\{1,\cdots ,n-1\right\}\},\hfill \end{array}$$

(9)

where $\mathcal{C}$ is the set of time slot indices where the symbol is used in channel estimation, and $\mathcal{C}\left(i\right)$ is the i-th smallest element. $k_n\in \mathcal{K}=\{1,\dots ,|\mathcal{X}\left|\right\}$ is the transmitted symbol index at time slot n. Based on the expression, we can obtain the proposed channel estimate using the state ${\mathsf{S}}_n\in {\mathcal{S}}_n$ as

$$\begin{array}{cc}\hfill {\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_n\right)& ={\left({\widehat{\mathbf{X}}}_n{\widehat{\mathbf{X}}}_n^H+{\sigma }_n^2{\mathbf{I}}_{N_t}\right)}^{-1}{\widehat{\mathbf{X}}}_n{\mathbf{z}}_r^H\left({\mathsf{S}}_n\right)\hfill \end{array}$$

(10)

where ${\mathbf{z}}_r\left({\mathsf{S}}_n\right)={[{\mathbf{z}}_r^p,{\mathbf{z}}_r\left[\mathcal{C}\left(1\right)\right],\dots ,{\mathbf{z}}_r\left[\mathcal{C}\left(\left|\mathcal{C}\right|\right)\right]]}^T$. Note that ${\mathcal{S}}_n$ is the set of all states and ${\mathsf{S}}_n$ is the state.

The action set is defined as $\mathcal{A}=\{0,1\}$. As explained in the previous subsection, the detected data symbol is used in the proposed channel estimation when $a=1$; otherwise, the detected data symbol is not used. The transition function ${\mathcal{T}}^{(a,j)}\left({\mathsf{S}}_n\right)$ from state ${\mathsf{S}}_n\in {\mathcal{S}}_n$ is defined as

$$\begin{array}{cc}\hfill {\mathcal{T}}^{(a,j)}\left({\mathsf{S}}_n\right)& =\mathbb{P}\left[{\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)|{\mathsf{S}}_n,a\right]=\left\{\begin{array}{cc}\mathcal{I}\left[\mathbf{x}\left[n\right]={\mathbf{x}}_j\right],\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ 1,\hfill & j\in {\mathcal{J}}_a,a=0.\hfill \end{array}\right.\hfill \end{array}$$

(11)

where $\mathcal{I}(·)$ equals one when the event is true and zero otherwise. ${\mathcal{J}}_0\in =\left\{0\right\}$ and ${\mathcal{J}}_1\in \{1,\dots ,K\}$. ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\in {\mathcal{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)$ is a possible candidate for the next state from state ${\mathsf{S}}_n$ and is defined as

$$\begin{array}{cc}\hfill {\mathcal{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)& =\left\{\begin{array}{cc}\left([{\mathbf{X}}_n,{\mathbf{x}}_j],[{\widehat{\mathbf{X}}}_n,\widehat{\mathbf{x}}\left[n\right]],[\mathcal{C}\cup n]\right),\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ \left({\mathbf{X}}_n,{\widehat{\mathbf{X}}}_n,\mathcal{C}\right),\hfill & j\in {\mathcal{J}}_a,a=0.\hfill \end{array}\right.\hfill \end{array}$$

(12)

The reward $\mathcal{R}\left({\mathsf{S}}_n,{\mathsf{S}}_{n+1}\right)$ is defined as the difference between the MSEs at the current state ${\mathsf{S}}_n\in {\mathcal{S}}_n$ and the next state ${\mathsf{S}}_{n+1}\in {\mathcal{S}}_{n+1}$, which is given by

$$\begin{array}{cc}\hfill \mathcal{R}\left({\mathsf{S}}_n,{\mathsf{S}}_{n+1}\right)& =\mathcal{E}\left[\parallel {\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_n\right)-{\mathbf{h}}_r{\left[n\right]\parallel }^2\right]-\mathcal{E}\left[\parallel {\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_{n+1}\right)-{\mathbf{h}}_r{[n+1]\parallel }^2\right]\hfill \\ \hfill & =\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{S}}_n\right)\right]-\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{S}}_{n+1}\right)\right]=\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{S}}_n\right)-\mathbf{B}\left({\mathsf{S}}_{n+1}\right)\right],\hfill \end{array}$$

(13)

where $\mathbf{B}\left({\mathsf{S}}_n\right)=\mathcal{E}\left[({\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_n\right)-{\mathbf{h}}_r\left[n\right]){({\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_n\right)-{\mathbf{h}}_r\left[n\right])}^H\right]$ is error covariance. Unlike in [31,32,33], the error covariance is defined between the estimated channel ${\widehat{\mathbf{h}}}_r\left({\mathsf{S}}_n\right)$ and ${\mathbf{h}}_r\left[n\right]$ at time slot index n.

The Q-value function $\mathcal{Q}\left({\mathsf{S}}_n,a\right)$ is the sum of the rewards, which is given by

$$\begin{array}{cc}\hfill \mathcal{Q}\left({\mathsf{S}}_n,a\right)& =\sum _{j\in {\mathcal{J}}_a}{\mathcal{T}}^{(a,j)}\left({\mathsf{S}}_n\right)\left[\mathcal{R}\left({\mathsf{S}}_n,{\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)+\gamma {\mathcal{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)\right],\hfill \end{array}$$

(14)

where $\mathrm{Tr}(·)$ is a trace operation. ${\mathcal{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)$ is the optimal sum of future reward after ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)$. $\gamma$ is a discounting factor whose value is assumed as one because the proposed channel estimator also considers the effect of future rewards at the ending state [31].

The optimal policy maximizes the Q-value function, which is expressed as

$$\begin{array}{cc}\hfill {\pi }^{★}\left({\mathsf{S}}_n\right)& =\underset{a\in \mathcal{A}}{argmax}\mathcal{Q}\left({\mathsf{S}}_n,a\right).\hfill \end{array}$$

(15)

Solving the optimization problem in (15) is highly difficult because the transition probability ${\mathcal{T}}^{(a,j)}\left({\mathsf{S}}_n\right)$ is unknown, and the number of candidate states exponentially increases with the data length. An effective method to solve this problem is to use a reinforcement learning algorithm. Therefore, the proposed channel estimator also adopts a reinforcement learning algorithm, but the effect of the time-varying channel is also considered in comparison with [31,32,33].

A deep reinforcement learning (DRL) approach is a promising solution for dealing with the dimension explosion of the states by leveraging deep neural networks. To apply the DRL approach to our MDP, an agent needs to interact with an environment to obtain an action-value function for a given action and state. However, both the states and rewards of our MDP are not observable at the receiver. This means that the agent cannot acquire training samples, each of which consists of the state (or the state transition) and the corresponding reward. Consequently, the DRL approach and other data-driven approaches are not directly applicable to solving our MDP.

3. Proposed Optimal Policy

This section describes the proposed optimal policy. The basic concept of the derivation is similar to that in [31,32,33]. However, its direct extension is difficult for time-varying channels. This is because capturing time-variant channels using previously selected data symbols is difficult. To address this, we approximate the first-order Gaussian—Markov process and propose a computationally efficient algorithm.

We employ the approximation in [31,32,33] for the transition function, which is given by

$$\begin{array}{cc}\hfill {\widehat{\mathcal{T}}}^{(a,j)}\left({\mathsf{S}}_n\right)& =\left\{\begin{array}{cc}{\theta }_j\left[n\right],\hfill & j\in {\mathcal{J}}_a,a=1,\hfill \\ 1,\hfill & j\in {\mathcal{J}}_a,a=0,\hfill \end{array}\right.\hfill \end{array}$$

(16)

where ${\widehat{\mathcal{T}}}^{(a,j)}\left({\mathsf{S}}_n\right)\to {\mathcal{T}}^{(a,j)}\left({\mathsf{S}}_n\right)$ as ${\theta }_j\left[n\right]\to 1$.

The main difficulty in analyzing the time-varying channel model is solving element $\mathsf{\Delta }\left[n\right]$. To resolve this difficulty, we approximate the first-order Gaussian—Markov process in (1) as follows:

$$\begin{array}{cc}\hfill \mathbf{H}\left[n\right]& \approx \sqrt{1-{ϵ}^2}\mathbf{H}[n-1],\hfill \end{array}$$

(17)

where $H[n-1]\gg \mathsf{\Delta }\left[n\right]$. This approximation is often adopted in studies because it provides analytical tractableness [36,37,38]. Using this approximation, the received symbol $\mathbf{z}[n+m]$ for $1\le m$ can be expressed in terms of $\mathbf{H}\left[n\right]$ as follows:

$$\begin{array}{cc}\hfill \mathbf{z}[n+m]& ={\mathbf{H}}^H[n+m]\mathbf{x}[n+m]+\mathbf{n}[n+m]\hfill \\ \hfill & \approx {\mathbf{H}}^H\left[n\right]\left({\sqrt{1-{ϵ}^2}}^m\mathbf{x}[n+m]\right)+\mathbf{n}[n+m],\hfill \end{array}$$

(18)

From approximation (18), the virtual state in [31] that mimics the optimal behavior from state ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)$ can be obtained as follows:

$$\begin{array}{cc}\hfill {\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathsf{S}}_n\right)& =\left({\mathbf{X}}_m^{(a,j)},{\widehat{\mathbf{X}}}_m^{\left(a\right)},{\mathcal{C}}_m^{\left(a\right)}\right),\hfill \end{array}$$

(19)

where

$$\begin{array}{cc}\hfill {\mathbf{X}}_m^{(a,j)}& =\left\{\begin{array}{cc}\left[{\mathbf{X}}_n,{\mathbf{x}}_j,\left(\sqrt{1-{ϵ}^2}\tilde{\mathbf{x}}[n+1]\right),\cdots ,\left({\sqrt{1-{ϵ}^2}}^{m-n-1}\tilde{\mathbf{x}}[m-1]\right)\right],\hfill & a=1,\hfill \\ \left[{\mathbf{X}}_n,\left(\sqrt{1-{ϵ}^2}\tilde{\mathbf{x}}[n+1]\right),\cdots ,\left({\sqrt{1-{ϵ}^2}}^{m-n-1}\tilde{\mathbf{x}}[m-1]\right)\right],\hfill & a=0.\hfill \end{array}\right.\hfill \\ \hfill {\widehat{\mathbf{X}}}_m^{\left(a\right)}& =\left\{\begin{array}{cc}\left[{\widehat{\mathbf{X}}}_n,\widehat{\mathbf{x}}\left[n\right],\left(\sqrt{1-{ϵ}^2}\tilde{\mathbf{x}}[n+1]\right),\cdots ,\left({\sqrt{1-{ϵ}^2}}^{m-n-1}\tilde{\mathbf{x}}[m-1]\right)\right],\hfill & a=1,\hfill \\ \left[{\widehat{\mathbf{X}}}_n,\left(\sqrt{1-{ϵ}^2}\tilde{\mathbf{x}}[n+1]\right),\cdots ,\left({\sqrt{1-{ϵ}^2}}^{m-n-1}\tilde{\mathbf{x}}[m-1]\right)\right],\hfill & a=0.\hfill \end{array}\right.\hfill \\ \hfill {\mathcal{C}}_m^{\left(a\right)}& =\left\{\begin{array}{cc}\left[\mathcal{C}\cup \{n+1,\dots ,m\}\right],\hfill & a=1,\hfill \\ \left[\mathcal{C}\cup \{n+2,\dots ,m\}\right],\hfill & a=0.\hfill \end{array}\right.\hfill \end{array}$$

The soft-decision symbol $\tilde{\mathbf{x}}\left[m\right]$ for $m\ge n+1$ is define as

$$\begin{array}{cc}\hfill \tilde{\mathbf{x}}\left[m\right]& =\sum _{k=1}^K{\theta }_k\left[m\right]{\mathbf{x}}_j.\hfill \end{array}$$

(20)

In (19), because $0\le ϵ\le 1$, the effect of soft decision symbol $\tilde{\mathbf{x}}[n+m]$ for estimating $\mathbf{H}\left[n\right]$ is diminished as m increases. Based on the virtual state, the state-action diagram for the proposed channel estimator is shown in Figure 2. In this figure, the number of state transitions at state ${\mathsf{S}}_n$ are one and K for $a=0$ and $a=1$, respectively. However, after $n+2$, the state transition is simplified to one because the virtual state mimics the behavior of state ${\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)$.

Using the definition of virtual state (19), we can compute the future reward ${\mathcal{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)$ as

$$\begin{array}{cc}\hfill {\mathcal{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)& \approx \mathcal{R}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right),{\tilde{\mathsf{U}}}_{n+2}^{(a,j)}\left({\mathsf{S}}_n\right)\right)+\sum _{m=n+2}^{{\mathcal{M}}_d\left(N_d\right)}\mathcal{R}\left({\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathsf{S}}_n\right),{\tilde{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right).\hfill \end{array}$$

(21)

By applying (13) to the future reward, the future reward is simplified as

$$\begin{array}{cc}\hfill {\mathcal{V}}^{★}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)& =\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)-\mathbf{B}\left({\tilde{\mathsf{U}}}_{n+2}^{(a,j)}\left({\mathsf{S}}_n\right)\right)+\sum _{m=n+2}^{{\mathcal{M}}_d\left(N_d\right)}\mathbf{B}\left({\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathsf{S}}_n\right)\right)-\mathbf{B}\left({\tilde{\mathsf{U}}}_{m+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)\right]\hfill \\ \hfill & =\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{U}}_{n+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)-\mathbf{B}\left({\tilde{\mathsf{U}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)\right].\hfill \end{array}$$

(22)

Using the approximations (16) and (22), the Q-value function in (14) is obtained as follows:

$$\begin{array}{cc}\hfill \mathcal{Q}\left({\mathsf{S}}_n,a\right)& =\sum _{j\in {\mathcal{J}}_a}{\widehat{\mathcal{T}}}^{(a,j)}\left({\mathsf{S}}_n\right)\mathrm{Tr}\left[\mathbf{B}\left({\mathsf{S}}_n\right)-\mathbf{B}\left({\tilde{\mathsf{U}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)\right].\hfill \end{array}$$

(23)

The error covariance matrix $\mathbf{B}\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)$ can be computed as

$$\begin{array}{cc}\hfill \mathbf{B}\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)& =\mathcal{E}\{\parallel {\widehat{\mathbf{h}}}_r\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)-{\mathbf{h}}_r\left[n\right]{\parallel }^2\}\hfill \\ \hfill & =\mathcal{E}\left\{({\widehat{\mathbf{h}}}_r\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)-{\mathbf{h}}_r\left[n\right]){({\widehat{\mathbf{h}}}_r\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)-{\mathbf{h}}_r\left[n\right])}^H\right\}\hfill \\ \hfill & =\mathcal{E}\left\{{\widehat{\mathbf{h}}}_r\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right){\widehat{\mathbf{h}}}_r^H\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)-{\mathbf{h}}_r\left[n\right]{\widehat{\mathbf{h}}}_r^H\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right)-{\widehat{\mathbf{h}}}_r\left({\tilde{\mathrm{U}}}_m^{(a,j)}\right){\mathbf{h}}_r^H\left[n\right]+{\mathbf{h}}_r\left[n\right]{\mathbf{h}}_r^H\left[n\right]\right\}\hfill \\ \hfill & \stackrel{\left(a\right)}{=}{\mathbf{Q}}_m^{\left(a\right)}{\widehat{\mathbf{X}}}_m^{(a,j)}\left({\left({\mathbf{X}}_m^{(a,j)}\right)}^H{\mathbf{X}}_m^{(a,j)}+{\sigma }_n^2{\mathbf{I}}_{|{\mathcal{C}}_m^{\left(a\right)}|}\right){\left({\widehat{\mathbf{X}}}_m^{(a,j)}\right)}^H{\mathbf{Q}}_m^{\left(a\right)}\hfill \\ \hfill & -{\mathbf{X}}_m^{(a,j)}{\left({\widehat{\mathbf{X}}}_m^{(a,j)}\right)}^H{\mathbf{Q}}_m^{\left(a\right)}-{\mathbf{Q}}_m^{\left(a\right)}{\widehat{\mathbf{X}}}_m^{(a,j)}{\left({\mathbf{X}}_m^{(a,j)}\right)}^H+{\mathbf{I}}_{N_t}\hfill \\ \hfill & =\left({\mathbf{I}}_{N_t}-{\mathbf{Q}}_m^{\left(a\right)}{\widehat{\mathbf{X}}}_m^{(a,j)}{\left({\mathbf{X}}_m^{(a,j)}\right)}^H\right){\left({\mathbf{I}}_{N_t}-{\mathbf{Q}}_m^{\left(a\right)}{\widehat{\mathbf{X}}}_m^{(a,j)}{\left({\mathbf{X}}_m^{(a,j)}\right)}^H\right)}^H+{\sigma }_n^2{\mathbf{Q}}_m^{\left(a\right)}-{\sigma }_n^4{\left({\mathbf{Q}}_m^{\left(a\right)}\right)}^2\hfill \\ \hfill & \stackrel{\left(b\right)}{=}{\mathbf{Q}}_m^{\left(a\right)}{\mathbf{D}}_m^{(a,j)}{\left({\mathbf{D}}_m^{(a,j)}\right)}^H{\mathbf{Q}}_m^{\left(a\right)}+{\sigma }_n^2{\mathbf{Q}}_m^{\left(a\right)}-{\sigma }_n^4{\left({\mathbf{Q}}_m^{\left(a\right)}\right)}^2\hfill \end{array}$$

(24)

where the distribution of ${\mathbf{z}}_r^H\left({\tilde{\mathsf{U}}}_m^{(a,j)}\left({\mathsf{S}}_n\right)\right)$ is given by $\mathcal{CN}\left({\mathbf{0}}_{|{\mathcal{C}}_m^{\left(a\right)}|},{\left({\mathbf{X}}_m^{(a,j)}\right)}^H{\mathbf{X}}_m^{(a,j)}+{\sigma }_n^2{\mathbf{I}}_{|{\mathcal{C}}_m^{\left(a\right)}|}\right)$ and ${\mathbf{Q}}_m^{\left(a\right)}={\left({\widehat{\mathbf{X}}}_n{\widehat{\mathbf{X}}}_n^H+{\displaystyle \sum _{k=n+1}^{m-1}}{(1-{ϵ}^2)}^{m-n}\tilde{\mathbf{x}}\left[k\right]{\tilde{\mathbf{x}}}^H\left[k\right]+{\sigma }_n^2{\mathbf{I}}_{N_t}\right)}^{-1}$ is applied in $\left(a\right)$. ${\mathbf{D}}_m^{(a,j)}={\left({\mathbf{Q}}_m^{\left(a\right)}\right)}^{-1}-{\widehat{\mathbf{X}}}_m{\mathbf{X}}_m^H={\widehat{\mathbf{X}}}_m{\left({\widehat{\mathbf{X}}}_m-{\mathbf{X}}_m\right)}^H+{\sigma }_n^2{\mathbf{I}}_{N_t}$ is used in $\left(b\right)$.

By applying (24) to the Q-value function, the optimal policy at ${\mathsf{S}}_n$ is computed as

$$\begin{array}{cc}\hfill {\pi }^{★}\left({\mathsf{S}}_n\right)& =\underset{a\in \{0,1\}}{argmax}\mathcal{Q}\left({\mathsf{S}}_n,a\right)=\mathcal{I}\left[\left(\mathcal{Q}\left({\mathsf{S}}_n,1\right)-\mathcal{Q}\left({\mathsf{S}}_n,0\right)\right)\ge 0\right]\hfill \\ \hfill & =\mathcal{I}\left[\mathrm{Tr}\left[\left(\sum _{j=1}^K\mathbf{B}\left({\tilde{\mathsf{U}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(0,0)}\left({\mathsf{S}}_n\right)\right)-{\theta }_j\left[n\right]\mathbf{B}\left({\tilde{\mathsf{U}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(1,j)}\left({\mathsf{S}}_n\right)\right)\right)\right]\ge 0\right].\hfill \end{array}$$

(25)

where $\mathbf{B}\left({\tilde{\mathsf{U}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(a,j)}\left({\mathsf{S}}_n\right)\right)={\sigma }_n^2{\mathbf{Q}}^{\left(a\right)}-{\sigma }_n^4{\left({\mathbf{Q}}^{\left(a\right)}\right)}^2+{\mathbf{Q}}^{\left(a\right)}{\mathbf{D}}^{(a,j)}{\left({\mathbf{D}}^{(a,j)}\right)}^H{\mathbf{Q}}^{\left(a\right)}$. ${\mathbf{Q}}^{\left(a\right)}={\mathbf{Q}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{\left(a\right)}$ and ${\mathbf{D}}^{(a,j)}={\mathbf{D}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(a,j)}$ are defined as

$$\begin{array}{cc}\hfill {\mathbf{Q}}^{\left(a\right)}& ={\left({\widehat{\mathbf{X}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{\left(a\right)}{\left({\widehat{\mathbf{X}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{\left(a\right)}\right)}^H+{\sigma }_n^2{\mathbf{I}}_{N_{\mathrm{tx}}}\right)}^{-1}\hfill \\ \hfill & \stackrel{\left(a\right)}{=}\left\{\begin{array}{cc}{\left({\widehat{\mathbf{X}}}_n{\widehat{\mathbf{X}}}_n^H+{\displaystyle \sum _{m=n+1}^{{\mathcal{M}}_d\left(N_d\right)}}{(1-{ϵ}^2)}^{m-n}\tilde{\mathbf{x}}\left[m\right]{\tilde{\mathbf{x}}}^H\left[m\right]+{\sigma }_n^2{\mathbf{I}}_{N_t}\right)}^{-1},\hfill & a=0,\hfill \\ {\left({\left({\mathbf{Q}}^{\left(0\right)}\right)}^{-1}+\tilde{\mathbf{x}}\left[n\right]{\tilde{\mathbf{x}}}^H\left[n\right]\right)}^{-1},\hfill & a=1.\hfill \end{array}\right.\hfill \\ \hfill {\mathbf{D}}^{(a,j)}& ={\widehat{\mathbf{X}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{\left(a\right)}{\left({\widehat{\mathbf{X}}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{\left(a\right)}-{\mathbf{X}}_{{\mathcal{M}}_d\left(N_d\right)+1}^{(a,j)}\right)}^H+{\sigma }_n^2{\mathbf{I}}_{N_t}\hfill \\ \hfill & \stackrel{\left(b\right)}{=}\left\{\begin{array}{cc}{\widehat{\mathbf{X}}}_n{\left({\widehat{\mathbf{X}}}_n-{\mathbf{X}}_n\right)}^H+{\sigma }_n^2{\mathbf{I}}_{N_t},\hfill & j\in {\mathcal{J}}_a,a=0,\hfill \\ {\mathbf{D}}^{(0,0)}+\widehat{\mathbf{x}}\left[n\right]{\left(\widehat{\mathbf{x}}\left[n\right]-{\mathbf{x}}_j\right)}^H,\hfill & j\in {\mathcal{J}}_a,a=1.\hfill \end{array}\right.\hfill \end{array}$$

Similar to [31], ${\mathbf{Q}}^{\left(1\right)}$ and ${\mathbf{Q}}^{\left(0\right)}$ satisfy ${\mathbf{Q}}^{\left(1\right)}={\mathbf{Q}}^{\left(0\right)}-\frac{{\mathbf{Q}}^{\left(0\right)}\widehat{\mathbf{x}}\left[n\right]{\widehat{\mathbf{x}}}^H\left[n\right]{\mathbf{Q}}^{\left(0\right)}}{1+{\widehat{\mathbf{x}}}^H\left[n\right]{\mathbf{Q}}^{\left(0\right)}\widehat{\mathbf{x}}\left[n\right]}$. In addition, ${\mathbf{D}}^{(1,j)}$ and ${\mathbf{D}}^{(0,0)}$ satisfy ${\displaystyle \sum _{j=1}^K}{\theta }_j\left[n\right]{\mathbf{D}}^{(1,j)}{\left({\mathbf{D}}^{(1,j)}\right)}^H=\left({\mathbf{D}}^{(0,0)}+{\widehat{\mathbf{d}}}_n\right){\left({\mathbf{D}}^{(0,0)}+{\widehat{\mathbf{d}}}_n\right)}^H+{\delta }_n\widehat{\mathbf{x}}\left[n\right]{\widehat{\mathbf{x}}}^H\left[n\right]$ where ${\widehat{\mathbf{d}}}_n=\widehat{\mathbf{x}}\left[n\right]{(\widehat{\mathbf{x}}\left[n\right]-\tilde{\mathbf{x}}\left[n\right])}^H$, and ${\delta }_n={\displaystyle \sum _{j=1}^K}{\theta }_j\left[n\right]\parallel \widehat{\mathbf{x}}\left[n\right]-{\mathbf{x}}_j{\parallel }^2-{\parallel \widehat{\mathbf{x}}\left[n\right]-\tilde{\mathbf{x}}\left[n\right]\parallel }^2$.

Finally, similar to [32], by applying the results in (23) and (24) to (25), we obtain the proposed optimal policy in closed-form as

$$\begin{array}{cc}\hfill {\pi }^{★}\left({\mathsf{S}}_n\right)& =\mathcal{I}\left[\frac{{\sigma }_n^2(1+{\alpha }_n)+{\sigma }_n^4\parallel {\mathbf{a}}_n{\parallel }^2+{\parallel {\mathbf{d}}_n\parallel }^2}{2{\sigma }_n^4{\beta }_n+{\gamma }_n+{\parallel {\mathbf{c}}_n-{\mathbf{b}}_n+{\mathbf{d}}_n\parallel }^2}\ge 1\right].\hfill \end{array}$$

(26)

When we define $\mathbf{Q}={\mathbf{Q}}^{\left(0\right)}$ and $\mathbf{D}={\mathbf{D}}^{(0,0)}$, vectors are computed as ${\mathbf{a}}_n=\frac{\mathbf{Q}\widehat{\mathbf{x}}\left[n\right]}{\sqrt{1+{\alpha }_n}}$, ${\mathbf{b}}_n={\mathbf{D}}^H{\mathbf{b}}_n$, ${\mathbf{c}}_n=\frac{\widehat{\mathbf{x}}\left[n\right]-\tilde{\mathbf{x}}\left[n\right]}{\sqrt{1+{\alpha }_n}}$, and ${\mathbf{d}}_n=\frac{{\mathbf{D}}^H\mathbf{Q}{\mathbf{a}}_n}{\parallel {\mathbf{a}}_n{\parallel }^2}$. In addition, the constants are computed as ${\alpha }_n={\widehat{\mathbf{x}}}^H\left[n\right]\mathbf{Q}\widehat{\mathbf{x}}\left[n\right]$, ${\beta }_n=\frac{{\mathbf{a}}_n^H\mathbf{Q}{\mathbf{a}}_n}{\parallel {\mathbf{a}}_n{\parallel }^2}$, and ${\gamma }_n=\frac{{\delta }_n}{1+{\alpha }_n}$. Note that the expression of the optimal policy in (26) is similar to that in [32]. However, the vectors and constants in the optimal policy is different from those in [32] because the temporal correlation $ϵ$ is considered in $\mathbf{Q}$ and $\mathbf{D}$. When $ϵ=0$, the optimal policy in (26) is equivalent to that in [32].

4. Further Performance Improvement

In this section, we propose a practical method to improve the estimation accuracy of the proposed channel estimator. The proposed method refines state elements to capture the time-varying nature of the channel.

4.1. State Element Refinement

Elements ${\mathbf{X}}_n$ and ${\widehat{\mathbf{X}}}_n$ in state ${\mathsf{S}}_n$ are updated when the detected data symbol is selected based on the optimal policy. However, the elements gradually lose their effectiveness in estimating $\mathbf{H}\left[n\right]$ as time slot index n increases. To address this, we first represent the received symbol for $1\le m$ in terms of $\mathbf{H}\left[n\right]$ as

$$\begin{array}{cc}\hfill \mathbf{z}[n-m]& ={\mathbf{H}}^H[n-m]\mathbf{x}[n-m]+\mathbf{n}[n-m]\hfill \\ \hfill & \approx {\mathbf{H}}^H\left[n\right]\left({\sqrt{1-{ϵ}^2}}^{-m}\mathbf{x}[n-m]\right)+\mathbf{n}[n-m].\hfill \end{array}$$

(27)

Using (27), we refine the elements ${\mathbf{X}}_n$ and ${\widehat{\mathbf{X}}}_n$ in state as the time slot index increases, which is given by

$$\begin{array}{cc}\hfill {\mathbf{X}}_n& \leftarrow {\sqrt{1-{ϵ}^2}}^{-1}{\mathbf{X}}_n\hfill \\ \hfill {\widehat{\mathbf{X}}}_n& \leftarrow {\sqrt{1-{ϵ}^2}}^{-1}{\widehat{\mathbf{X}}}_n.\hfill \end{array}$$

(28)

Regardless of the above refinement, the previously selected data symbols lose their effectiveness as the time slot index increases, particularly for large data lengths. This is because the term $\mathsf{\Delta }\left[n\right]$ in (1) becomes dominant, increasing the uncertainty in estimating the channel. To overcome this, we remove too-old selected data symbols in state by introducing a window size $N_w$. In other words, we maintain the size of the set of time slot indices as $\left|\mathcal{C}\right|=N_w$. Thus, when the optimal action is one at time slot index n, n is included, whereas the first index $\mathcal{C}\left(1\right)$ is removed from set $\mathcal{C}$, which can be expressed as

$$\begin{array}{cc}\hfill {\mathbf{X}}_n& \leftarrow {\mathbf{X}}_n\setminus {\mathbf{X}}_n\left[\mathcal{C}\left(1\right)\right],\hfill \\ \hfill {\widehat{\mathbf{X}}}_n& \leftarrow {\widehat{\mathbf{X}}}_n\setminus {\widehat{\mathbf{X}}}_n\left[\mathcal{C}\left(1\right)\right],\hfill \\ \hfill \mathcal{C}& \leftarrow \mathcal{C}\setminus \mathcal{C}\left(1\right).\hfill \end{array}$$

(29)

4.2. Algorithm

Using the proposed optimal policy and performance improvement strategy, the proposed channel estimator is summarized in Algorithm 1. The receiver obtains the initial channel estimation during pilot transmission. Subsequently, during the data transmission, the receiver sequentially selects a data symbol based on the optimal policy. When the optimal action $a^{★}=1$, the state ${\mathsf{S}}_n$ is updated using the most-probable state transition [31]. In addition, the state element refinement is performed based on this condition. After each data block ends, the channel estimate is updated using the state ${\mathsf{S}}_n$.

Algorithm 1: Proposed channel estimator

1 Obtain the initial channel estimate $\mathbf{H}\leftarrow \widehat{\mathbf{h}}=\left[{\widehat{\mathbf{h}}}_1,\cdots ,{\widehat{\mathbf{h}}}_{N_r}\right]$ from (3) 2 Initialize the state ${\mathsf{S}}_1=\left({\mathbf{X}}^p,{\mathbf{X}}^p,\varphi \right)$. 3 for$d=1$ to $M_d$ do$\phantom{(}$ 4   for$n\in {\mathcal{M}}_d$do$\phantom{(}$ 5    Compute the optimal policy $a^{★}={\pi }^{★}\left({\mathsf{S}}_n\right)$ from (26). 6    Set the optimal values $j^{★}=0$ for $a^{★}=0$ and ${\mathbf{x}}_{j^{★}}=\widehat{\mathbf{x}}\left[n\right]$ for $a^{★}=1$. 7    Update ${\mathsf{S}}_{n+1}\leftarrow {\mathsf{U}}_{n+1}^{(a^{★},j^{★})}\left({\mathsf{S}}_n\right)$ from (12). 8    if$a^{★}==1$ and $N_w<\left|\mathcal{C}\right|$then$\phantom{(}$ 9      Remove the state elements in ${\mathsf{S}}_{n+1}$. 10      ${\mathbf{X}}_{n+1}\leftarrow {\mathbf{X}}_{n+1}\setminus {\mathbf{X}}_{n+1}\left[\mathcal{C}\left(1\right)\right]$, 11      ${\widehat{\mathbf{X}}}_{n+1}\leftarrow {\widehat{\mathbf{X}}}_{n+1}\setminus {\widehat{\mathbf{X}}}_n\left[\mathcal{C}\left(1\right)\right]$, 12      $\mathcal{C}\leftarrow \mathcal{C}\setminus \mathcal{C}\left(1\right)$. 13    end$\phantom{(}$ 14    Refine the state elements in ${\mathbf{X}}_{n+1}\leftarrow {\sqrt{1-{ϵ}^2}}^{-1}{\mathbf{X}}_{n+1}$ and      ${\widehat{\mathbf{X}}}_{n+1}\leftarrow {\sqrt{1-{ϵ}^2}}^{-1}{\widehat{\mathbf{X}}}_{n+1}$. 15  end$\phantom{(}$ 16  Update the channel estimate $\mathbf{H}\leftarrow \widehat{\mathbf{h}}=\left[{\widehat{\mathbf{h}}}_1\left({\mathsf{S}}_n\right),\cdots ,{\widehat{\mathbf{h}}}_{N_r}\left({\mathsf{S}}_n\right)\right]$ from (10). 17 end$\phantom{(}$

In Figure 3, we show a block diagram of the proposed channel estimator, which consists of the LMMSE channel estimator, optimal policy calculator, and state element refinement. The LMMSE channel estimator obtains the initial estimate at pilot transmission and updates the estimate at data transmission using state ${\mathsf{S}}_n$. The optimal policy calculator obtains the optimal action of (26) from the channel estimates and APP from the data detector. The state elements are then refined based on the obtained optimal action, and the refined state is used to estimate the channel and optimal policy for the next step.

Application of other data detection: The proposed RL-aided channel estimator can be universally applied to any other soft-output data detection method. To achieve this, the proposed RL-aided channel estimator relies on the availability of APPs, which can be directly derived from the MAP data detection method. In the case of using other soft-output data detections, the proposed RL-aided channel estimator can utilize the APPs that are computed from the log-likelihood ratios.

Complexity analysis: Complexity is analyzed in terms of real multiplications to provide an implementation perspective. Figure 3 shows the hardware structure of the proposed RL-aided channel estimator, which consists of the LMMSE channel estimator, state element refinement, and optimal policy calculator. Because the exact complexity can vary depending on the implementation details, the complexity order ($\mathcal{O}(·)$) of each component is analyzed.

The complexity order of the LMMSE channel estimator in (7) is $\mathcal{O}\left(\right(N_p+|\mathcal{C}\left({\mathbf{a}}^{★}\right)\left|\right)$$(N_t^2+N_t{N}_r))$ where $\mathcal{C}\left({\mathbf{a}}^{★}\right)$ is the set of selected data symbol vectors. The complexity order of state element refinement in Section 4.1 is $\mathcal{O}\left(4\right(N_p+|\mathcal{C}\left({\mathbf{a}}^{★}\right)\left|\right)$. The complexity of the optimal policy in (25) is primarily determined by the computation of ${\mathbf{Q}}^{\left(a\right)}$. Consequently, the complexity order of the optimal policy in (25) is $\mathcal{O}\left(2N_t^2{T}_d^2\right)$. It is important to note that among the components of the proposed channel estimator, the optimal policy calculator has the highest complexity because it performs every data symbol index n, while the other components perform every data block index d.

5. Simulation Results

This section presents the effectiveness of the proposed channel estimator using simulations. The numbers of transmit and receive antennas used were $N_t=2$ and $N_r=4$. The transmission frame consisted of one pilot block with $N_p=8$ symbols and $M_d=20$ data blocks with $N_d=128$ symbols. Each symbol used 4-quadrature amplitude modulation (QAM) symbol mapping. We adopted turbo channel code with a rate of $1/2$ and 16 cyclic redundancy check bits. For the proposed channel estimator, the window size was set to $N_w=2\times N_d$. The signal-to-noise ratio (SNR) was defined as $E_b/N_0=1/\left({log}_2\left|\mathcal{X}\right|{\sigma }_n^2\right)$ under the power constraint $\mathbb{E}\{\parallel \mathbf{x}\left[n\right]{\parallel }^2\}=1$. The proposed channel estimator was compared with the following methods.

• PCSI: This method is ideal for time-invariant channels in which a perfect initial channel estimate is available at the receiver. Because the initial channel changes during data transmission, it is not optimal for time-varying channels.
• Pilot: This method uses a conventional pilot-aided channel estimator using (3).
• Soft: This method is a data-aided channel estimator when all symbols in (20) are used as additional pilot symbols.
• Conv-RL [31]: This method is a data-aided channel estimator in which the detected data symbol is selected using the RL approach developed for time-invariant channels.

The performance of the methods was compared with that of the proposed channel estimator in terms of the block-error rate (BLER) and normalized MSE (NMSE). In addition, we considered the time-invariant channel $ϵ=0$ and time-variant channel with $ϵ=0.005$ and $ϵ=0.01$. Note that channel was more severely variant when $ϵ=0.01$ than when $ϵ=0.005$.

Figure 4 shows the BLERs for the proposed and other channel estimators in the time-invariant channel, i.e., $ϵ=0$. The conventional pilot-aided channel estimator exhibited a poor performance when the number of pilots was small. Data-aided channel estimators can overcome performance degradation caused by pilot-aided channel estimators. In particular, the RL-based channel estimator [31] showed an outstanding performance compared with other channel estimators. The BLER of the proposed channel estimator was slightly worse than that of [31] because of a reduced window size $N_w=2\times N_d$.

In Figure 5, the proposed channel estimator is compared with other channel estimators in time-varying channels. The proposed channel estimator had a better BLER improvement than the conventional pilot-aided channel estimator. In particular, the performance improvement is more prominent at $ϵ=0.01$ than that at $ϵ=0.005$. This is because the proposed channel estimator can efficiently capture channel variations by selecting and refining detected data symbols. In addition, in time-variant channels, the proposed channel estimator had a slightly higher BLER than the RL-based channel estimator, primarily due to the utilization of a reduced window size (see Figure 5). However, in time-varying channels, this reduction in window size actually contributed to an improvement in BLER by effectively leveraging the most recent data symbols. Consequently, the proposed channel estimator had a lower BLER compared to the RL-based channel estimator. In Figure 6, we show the BLER of the proposed channel estimator for different window sizes $N_w$ in time-varying channels with $ϵ=0.01$ The BLER of the proposed channel estimator gradually degraded as $N_w$ increased. This is because the selected data symbol is undesirable as an additional pilot symbol in fast fading channels; therefore, only the usage of the latest selected data symbol can improve the performance.

To further investigate the effect of window size, we investigated the NMSE of the proposed channel estimator for different window sizes $N_w$ at $ϵ=0.005$ and $E_b/N_0=-2$ dB (Figure 7). We observed that the NMSE improved until $M_d=2$ but decreased as the data block length increased. This is because the old selected data symbol is ineffective for estimating the channel. Thus, when we discard the old data symbol, we can further improve the estimation accuracy (Figure 7).

6. Conclusions

A data-aided channel estimator was proposed for time-varying channels, which involves selecting the detected data symbol. To facilitate efficient selection of the detected data symbol, an optimization problem was initially formulated to minimize the channel estimation error. Subsequently, the MDP for this optimization problem was formulated, and its optimal policy was derived using an RL algorithm. In the derivation process, approximations for the transition probability and a first-order Gaussian–Markov process were utilized. To improve estimation accuracy, a state element refinement was introduced to capture the time-varying nature of the channel by incorporating a window size. Simulation results demonstrated that the proposed channel estimator provides similar performance to the conventional RL-based channel estimator in time-invariant channels when $ϵ=0$, while showing improved performance in time-varying channels when $ϵ=0.01$ and $ϵ=0.005$ compared to conventional RL-based channel estimator.

An interesting direction for further research involves optimizing the frame structure in terms of the spectral efficiency. In this study, the frame structure comprises one pilot and D data block. The proposed RL-aided channel estimator is applied to the data blocks to capture the time-varying nature of the channel. However, in fast fading channels, it can be challenging for the proposed channel estimator to accurately track channel variations. In such cases, reducing the value of D in the frame structure can potentially improve the performance. However, this reduction also leads to a degradation in spectral efficiency. To find an appropriate value for D in time-varying channels, an optimization problem that maximizes spectral efficiency while maintaining acceptable performance levels becomes a suitable criterion. To address this, one approach is to first derive the performance of the RL-aided channel estimator. Subsequently, the solution to the optimization problem can be obtained using the derived performance.