Unscented Kalman Filter with Joint Decision Scheme for Phase Estimation in Probabilistically Shaped QAM Systems

Abstract

A carrier phase estimation method based on the unscented Kalman filter (UKF) is proposed for probabilistically shaped (PS) quadrature amplitude modulation (QAM) systems. We further integrate a joint decision scheme into the proposed UKF−based algorithm to prevent the correlated erroneous decisions in the phase recovery scheme caused by the impact of PS. The proposed method achieves the performance benefit for PS constellations in optical transmissions by partitioning the constellation symbols suitably and utilizing both the maximum a posterior probability (MAP) and maximum likelihood (ML) detection. The results of numerical simulation and experimental verification reveal that the proposed method performs better than the conventional CPR algorithms in PS systems.

1. Introduction

The growing need for communication services that offer both high speed and high capacity has posed challenges for traditional coherent optical communication systems. These challenges primarily revolve around system capacity and spectral efficiency (SE). Consequently, the development of optical communication technology has been driven towards achieving higher capacity and improved spectral efficiency [1,2,3,4]. Probabilistic shaping (PS) has been widely employed in coherent optical communication to improve system capacity and reduce the gap with the Shannon limit. A prominent feature of the PS approach is the occurrence of data symbols with different probabilities. By decreasing the probability of symbols with larger amplitudes, the system can improve its performance and gain a shaping advantage.

However, the introduction of PS results in additional performance degradation and will impair these decision−directed digital signal processing (DSP) algorithms. For instance, Sasai et al. have investigated phase noise tolerances of high−order PS signals and QAM signals in [5] and suggested that we should take the laser linewidth and carrier phase recovery (CPR) algorithms into account when choosing a signal shaping form. In [6], Barbosa et al. have evaluated CPR algorithms for PS transmission and indicated that the blind phase search (BPS) stage is highly affected by PS at moderate and low optical signal−to−noise ratio (OSNR). In [7], Mello et al. have researched the interplay of PS and BPS algorithms and recommended the necessity for alternate CPR methods to be deployed in PS transmissions. Wang et al. have developed an optimized principal component−based phase estimation algorithm which outperforms the BPS algorithm at low signal−to−noise ratio (SNR) for PS QAM systems [8]. As an effective compensation method for linear impairments, the Kalman filter has received interest for applications in coherent optical communication systems [9,10,11]. Because of the fast−tracking capability and high accuracy for parameter estimation, several Kalman−filter−based algorithms have been suggested and implemented such as carrier recovery [12,13,14] and polarization state tracking [15,16]. As an alternative to the BPS algorithm, the CPR algorithm based on the Kalman filter can also function as a blind method for carrier recovery. However, the Kalman filter’s enhanced performance is accompanied by increased computational complexity and additional delay [10]. Moreover, the typical Kalman filter−based CPR algorithms use decision−directed operations, such as ML estimates, to update the measurement state, which is negatively affected by PS. As a potential alternative to these existing CPR algorithms, the Kalman filter in PS transmissions requires additional study.

In this paper, we present a novel CPR algorithm for PS QAM systems that is based on the unscented Kalman filter. The proposed technology combines the UKF−based algorithm with a novel joint decision scheme to mitigate the effects of PS and enhance PS transmission performance. The UKF−based structure ensures the estimation precision of the proposed algorithm. By partitioning the constellation symbols sensibly and employing both the MAP and ML decisions, the joint decision scheme ensures that the proposed technique is immune to the influence of PS in PS transmissions. Compared to conventional carrier phase recovery algorithms, the proposed technology maintains PS resilience and achieves a lower optical signal−to−noise ratio (OSNR) penalty when the normalized generalized mutual information (NGMI) threshold is taken into account. Monte Carlo simulations and experiments are used to assess the efficacy of the proposed algorithm. In comparison to conventional BPS, the proposed technology can reduce the SNR required to meet the 0.8 NGMI threshold for PS 16QAM systems. The proposed UKF−based algorithm can be applied to PS systems more effectively than the conventional CPR algorithm, resulting in an improvement in NGMI performance.

2. Operation Principle

2.1. Phase Estimate Based on the Kalman Filter

The Kalman filter provides rapid tracking capabilities and high precision for parameter estimation, making it a useful instrument for compensating linear impairments. Consequently, numerous Kalman−filter−based CPR techniques have been suggested and confirmed to be effective. In order to use the Kalman filter approach for carrier phase estimation, it is important to define the model of the received signal before to CPR. If the signal has undergone DSP processing, including IQ imbalance compensation, chromatic dispersion (CD) compensation, constant modulation algorithm (CMA), and frequency offset estimation (FOE), the primary impairment is laser phase noise (LPN). The received signal can be defined as

$$r\left(n\right)=s\left(n\right){exp}^{j\varphi \left(n\right)}+\eta \left(n\right)$$

(1)

where $r\left(n\right)$ and $s\left(n\right)$ represents the $n-$th received and transmitted signal. $\varphi \left(n\right)$ represents the phase noise which is modeled as $\varphi \left(n\right)=\varphi (n-1)+\omega \left(n\right)$. $\omega \left(n\right)$ is independent and identically distributed Gaussian random variables with zero mean and variance ${\sigma }^2=2\pi \Delta f\tau$. $\Delta f$ is the combined line width of transmitter and receiver lasers and $\tau$ is the symbols period. $\eta \left(n\right)$ represents the collective amplified spontaneous emission (ASE) noise that is assumed to be additive Gaussian with zero mean. The probability distribution of $s\left(n\right)$ can be provided by

$$P\left(x_i\right)=\frac{exp(-v|x_i{|}^2)}{{\sum }_{k=1}^{M}exp(-v|x_k{|}^2)}$$

(2)

where $x_i$ represents the $i-$th complex M−QAM constellation point and v is the shaping factor.

In this paper, we evaluate the performance of the PS system with various CPR algorithms using generalized mutual information(GMI) and normalized generalized mutual information(NGMI), which are commonly used as performance metrics in PS systems [17]. These two metrics can be represented by [17]

$$\begin{array}{cc}\hfill GMI(X;Y)& =H\left(X\right)-\sum _{j=1}^m\mathbb{H}\left(B_j\mid Y\right)\hfill \\ \hfill NGMI(X;Y)& =1-\frac{\mathbb{H}\left(X\right)-GMI(X;Y)}{m}\hfill \end{array}$$

(3)

In the implementations of linear Kalman filter (LKF) and extended Kalman filter (EKF) proposed in [18,19], the received sampled signal is processed sample−by−sample, and the state parameter vector to be estimated by the Kalman filter is $x_k=\left[{\theta }_k\right]$. The equation for the state update is ${\theta }_{k+1}=A{\theta }_k+{\eta }_p$, where A represents the state transition matrix, which is defined as 1 in the case for a Wiener process. The variable ${\eta }_p$ represents the process noise between two successive samples. The observation model proposed in [18] is then given by $\widetilde{{\theta }_k}={\theta }_k+{\eta }_{\theta }$, where $\widetilde{{\theta }_k}$ represents the actual value for the phase noise and $n_{\theta }$ is the observation noise. Both ${\eta }_p$ and ${\eta }_{\theta }$ are assumed to be white Gaussian with zero mean and co−variances denoted by Q and R, respectively.

In order to accurately estimate the phase noise for each sample and minimize the observation noise, the Kalman filtering algorithms use the prediction−update frame. According to the state update equation indicated above, a prediction ${\widehat{x}}_k^{-}$ of the state parameter vector $x_k$ is made at time instance $k-1$. We get a prediction of the measurement $y_k^{-}$ at time instance k using ${\widehat{x}}_k^{-}$. The actual measurement ${\tilde{y}}_k$ at time instance k is then made and the final estimate of the Kalman filter at time instance k is given as

$${\widehat{x}}_k={\widehat{x}}_k^{-}+K_k({\tilde{y}}_k-y_k^{-})$$

(4)

where $K_k$ represents the Kalman gain which is updated according to the measurement matrix, process, and measurement error covariance in each time instance. By minimizing the innovation vector $({\tilde{y}}_k-y_k^{-})$, the prediction $({\widehat{x}}_k-{\widehat{x}}_k^{-})$ approaches zero and the filter algorithm can lock to the immediate change in state parameters.

The unscented Kalman filter, as opposed to LKF and EKF, employs the unscented transform to obtain a set of ’sigma points’ instead [10]. The probability distribution function and statistical moments of the state parameters $x_k$ can be captured by those points. Although UKF requires more computational effort than LKF and EKF, it outperforms LKF and EKF in terms of system performance and can estimate non−linear systems with greater precision. Thus, the UKF is utilized to develop the proposed algorithm. In [7], Mello investigates the interaction between PS and BPS algorithm and concludes that BPS is influenced by PS due to correlated incorrect decisions made in the phase recovery scheme. PS influences the decisions made by the BPS algorithm, resulting in a performance decrease in excess of the anticipated shaping gain. Moreover, the accuracy of the estimate is influenced by the decisions taken about measurement acquisition during the observation stage. It suggests that the effects of PS also have an influence on a UKF−based CPR algorithm. Applying UKF in the PS transmission thus requires taking the erroneous decision into account.

2.2. Proposed Phase Estimation Algorithm

Maximum a posteriori probability (MAP) detection has recently been suggested [20], and numerous works implementing MAP detection in the CPR scheme have been reported recently [21]. For PS M−QAM modulation formats, the received symbols have different prior probabilities. For the PS constellation, the maximum likelihood detection that follows the rule of choosing the constellation symbols that are closest in terms of Euclidean distance is suboptimal. Figure 1a illustrates how the decision boundaries for MAP detection differ from the conventional boundaries for ML decisions. In comparison with the ML decision, the MAP decision considers the prior probability of the different transmitted symbols to maximize the a posterioriprobability of an accurate determination. The MAP decision rule can be expressed as [20]

$$\underset{i}{min}{|y_k-x_D|}^2+2N_{0}ln\left(P\left(x_D\right)\right)$$

(5)

where $y_k$ represents the $k-$th received symbol, $x_D$ represents the surrounding constellation symbols in terms of Euclidean distance, $N_0$ is the one−dimension noise variance. $P\left(x_D\right)$ represents the prior probabilities which are generally known at the transmitter. The first term in Equation (5) is the squared absolute Euclidean distance which is the same metric used in the decisions of the trational BPS. For constellation symbols under the PS scheme, the second term in Equation (5) is related to the noise variance and prior probabilities, which changes the decision boundary. The symbols with a higher probability occupy larger areas, which decreases the likelihood of making a mistaken decision regarding these symbols and, as a result, reduces the overall probability of decision error.

However, additional investigation and simulation indicate that implementing MAP detection into the UKF−based CPR algorithm does not result in the expected performance improvement. In this observation model, the observed signal’s amplitude noise is accounted for by a complex observation noise, allowing the UKF to monitor both amplitude and phase variations. Despite the fact that a complex observation model improves estimation precision, constellation symbols with larger amplitudes have a greater impact on the innovation vector.

The decision regions for symbols with lower amplitudes and higher probabilities are expanded by MAP detection, whereas the regions for symbols with higher amplitudes in the ideal constellation map are reduced. We infer that additional decision errors of these symbols with larger amplitudes occur after MAP detection which has an impact on the UKF update. Simulation verification is shown in Section 3.1.

We, therefore, propose the jointed decision method to prevent the above−mentioned additional decision errors in order to apply MAP detection to the UKF in order to resolve the PS−induced correlated incorrect decisions. The proposed joint decision scheme is shown in Figure 1.

We set different decision boundaries for the inner and outer constellation points in the proposed joint detection method. Figure 1b illustrates how to detect inner constellation points using MAP’s detection regions, maintaining MAP’s advantage of lowering incorrect decisions in these regions. The boundaries for the outer constellation point, shown in Figure 1c, follow the regular pattern of the ML detection. By doing so, the relative narrowing of the outer constellation decision region could be prevented, the number of erroneous decisions made for the outer constellation symbols could be decreased, and both inner and outer symbols could be recognized using the proper boundaries. The overall decision boundary set is shown in Figure 1d.

Based on the joint detection scheme, we propose a UKF−based CPR algorithm, referred to as UKF−MAP/ML. The operation frame of UKF−MAP/ML is shown in Figure 2.

The majority of the block diagram depicts the method by which the UKF framework iteratively estimates the phase noise via the input signal. The process can be divided into three stages: prediction, observation, and update. The input signal $s\left(k\right)$ is divided into two channels for phase estimate and impairment compensation respectively. At the prediction stage, the prior estimates of the current state vector ${\widehat{x}}_k$ and the error covariance $P_k$ are obtained according to the posterior estimates ${\widehat{x}}_{k-1}$ in the previous instant [10].

$$\begin{array}{cc}\hfill {\widehat{x}}_k^{-}& =A{\widehat{x}}_{k-1}\hfill \\ \hfill P_k^{-}& =A{P}_k{A}^T+Q\hfill \end{array}$$

(6)

where ${(.)}^T$ represents the conjugate transpose operation. The sigma points set is then calculated using the unscented transform and the prior estimations, given as [22]:

$${\mathcal{X}}_k^{-}=\left\{\begin{array}{cc}{\widehat{x}}_k^{-}\hfill & i=0\hfill \\ {\widehat{x}}_k^{-}+{\left(\sqrt{(n+\lambda )P_k^{-}}\right)}^{i-1}\hfill & i=1,\dots ,n\hfill \\ {\widehat{x}}_k^{-}-{\left(\sqrt{(n+\lambda )P_k^{-}}\right)}^{i-n-1}\hfill & i=n+1,\dots ,2n\hfill \end{array}\right.$$

(7)

where ${\left(\sqrt{(n+\lambda )P_k^{-}}\right)}^{i-1}$ represents the $(i-1)$th column of the lower triangular matrix of the matrix $(n+\lambda )P_k^{-}$ after the Cholesky decomposition and n represents the state dimension. The parameter $\lambda$ is given as $\lambda ={\alpha }^2(n+k)-n$. The unscented transformation introduces four external parameters: $\alpha ,\beta ,\kappa ,\lambda$. The values of these external parameters determine the distribution of the sigma points.

At the observation stage, the received signal $s\left(k\right)$ is rotated by the sigma points. The smaller flowchart below in Figure 2 depicts the observation stage and the proposed joint decision scheme in more detail. The rotated signal $s_k^{-}$ is then fed into the radial decision module. According to the threshold, this module calculates the amplitude of the rotated symbols and distinguishes between symbols that correspond to inner constellation points and outer ones. The distinguished signal is then sent to the joint decision module to get the observation of the received signal. Those symbols after the joint decision module $d_k^{-}$ are used to update the measurements:

$${\widehat{Z}}_k^{-}=h\left({\mathcal{X}}_k^{-}\right)=Y_k-d_k^{-}{\mathcal{X}}_k^{-}$$

(8)

where $Y_k$ represents the actual measurement of the received signal. The measurements are utilized to calculate the posterior estimations [22]:

$${\widehat{z}}_k^{-}=\sum _{i=0}^{i=2n}{W}_m^{\left(i\right)}{\widehat{Z}}_{i,k}^{-}$$

(9)

$$P_{z_k,z_k}=\sum _{i=0}^{i=2n}{W}_c^{\left(i\right)}\left[{\widehat{Z}}_{i,k}^{-}-{\widehat{z}}_k^{-}\right]{\left[{\widehat{Z}}_{i,k}^{-}-{\widehat{z}}_k^{-}\right]}^T$$

(10)

$$P_{z_k,x_k}=\sum _{i=0}^{i=2n}{W}_c^{\left(i\right)}\left[{\mathcal{X}}_{i,k}^{-}-{\widehat{x}}_k^{-}\right]{\left[{\widehat{Z}}_{i,k}^{-}-{\widehat{z}}_k^{-}\right]}^T$$

(11)

where $W_m^{\left(i\right)}$ and $W_c^{\left(i\right)}$ represents the the weight of the sigma points when calculating the approximate mean and approximate covariance, respectively. The weights are given as [22]:

$$W_m^{\left(i\right)}=\left\{\begin{array}{cc}{\displaystyle \frac{\lambda }{n+\lambda }}\hfill & i=0\hfill \\ {\displaystyle \frac{1}{2n+\lambda }}\hfill & i=1,\dots ,2n\hfill \end{array}\right.$$

(12)

$$W_c^{\left(i\right)}=\left\{\begin{array}{cc}{\displaystyle \frac{\lambda }{n+\lambda }}+1-{\alpha }^2+\beta \hfill & i=0\hfill \\ {\displaystyle \frac{1}{2n+\lambda }}\hfill & i=1,\dots ,2n\hfill \end{array}\right.$$

(13)

We can calculate the Kalman gain and update the posterior estimate of the mean and covariance in the current instant according to the measurements:

$$\begin{array}{cc}\hfill K& =P_{z_k,x_k}/P_{z_k,z_k}\hfill \\ \hfill {\widehat{x}}_k& ={\widehat{x}}_k^{-}+K(z_k-{\widehat{z}}_k^{-}),z_k=0\hfill \\ \hfill P_k& =P_k^{-}-K{P}_{z_k,z_k}{K}^T\hfill \end{array}$$

(14)

The symbols after phase recovery $s_k=r_k{\widehat{x}}_k^{*}$ are obtained from the output port. The posterior estimate is also fed back to the processing of the next instant.

3. Simulation Analysis

Figure 3 depicts the simulation setup. To evaluate the performance of the proposed algorithm, Monte Carlo simulations were carried out to calculate the GMI and NGMI between the transmitted and received symbols, which are represented as dashed red lines in Figure 3. Square 16QAM symbols following Maxwell−Boltzmann (MB) distribution were generated with 1 sample per symbol.

The channel model considered only LPN and AWGN. The linear phase noise consisted of a constant phase shift of $\pi /6$ and random laser phase noise simulated by the model described in Section 2.1. The constant phase shift is used to test whether the algorithm can compensate for the phase rotation accurately. The random laser phase is used to test whether the algorithm can monitor the change of laser phase noise in the actual system. AWGN is then added to the symbols rotated by the laser phase noise. The output of the phase estimator is used to calculate the GMI and NGMI. The symbol rate was set to 32 GBaud and the combined linewidth between the transmitter laser and the local oscillator (LO) was set to 200 kHz. The SNR values considered only the signal power and AWGN. The GMI and NGMI results were obtained from the mean of 200 realizations with 16,384 symbols each to avoid the random error of the simulation results.

3.1. Optimization of the Decision Threshold at Variables Shaping Factors and Fixed SNR

We used Monte Carlo simulation to evaluate the GMI performance of the proposed algorithm in order to validate the joint decision scheme’s underlying principle. The SNR was fixed to the condition that the NGMI obtained 0.8, which works as the threshold for 27.5 percent soft−decision forward error correction (SD−FEC) [8].

We evaluated various thresholds for each of the four shaping variables for PS 16−QAM, which ranged from 0.1 to 0.2. As the channel model in this simulation only accounted for linear phase noise and AWGN, the signal scale after the channel is equivalent to the ideal 16QAM constellation points. The proposed joint decision scheme requires determining the symbol amplitude and differentiating between symbols based on the threshold. In order to include the amplitude range of the ideal 16QAM constellation points, the threshold was specified to be between 0 and 5 with intervals of 0.5.

Figure 4 illustrates that the GMI performance of the proposed algorithm is related to the threshold value at various shaping factors. By comparing the performance of the GMI at various thresholds of the same shaping factor, it is possible to analyze the relationship between the GMI and the threshold and determine the threshold value at which the GMI reaches its optimal value. The efficacy of the GMI improved as the threshold value rose from 0 to 2.5, reaching its optimum at 2.5. In addition, the GMI efficacy declined when the threshold value increased from 2.5 to 5. When the threshold was too low, the joint detection scheme was unable to take full advantage of the performance enhancement provided by the MAP. Nonetheless, if the threshold exceeded the optimal level, MAP would introduce additional decision errors and degrade performance. This phenomenon demonstrated that the performance of the joint decision method improved when the threshold value was appropriately selected. As the shaping factor increased, so did the threshold at which the GMI reached its optimum value, as shown in Figure 4.

Due to the fact that the proposed scheme is equivalent to MAP if the threshold is set too high, we can verify the inference that MAP would introduce additional decision errors and degrade performance if the threshold was set above the optimal level. In the results with a shaping factor of 0.1, the GMI performance with a threshold of 5 was worse than the GMI performance with a threshold of 0. When the threshold was set to 0, the ML decided all symbols, whereas the MAP decided all received symbols when the threshold was set to 5. Consequently, the results indicated that the performance loss caused by the additional decision error was greater than the performance improvement anticipated by the MAP, thereby establishing the necessity of the proposed joint decision scheme.

By comparing the GMI performance at varying thresholds of various shaping factors, we can verify the effectiveness of the proposed technology under different parameters. When the shaping factor was 0.1 or 0.125, the GMI performance achieved its peak when the threshold was set to 2.5. And when the shaping factor was 0.15, 0.175, or 0.2, the GMI performance reached its peak when the threshold was set to 3. The threshold value at which the GMI reaches its optimal value varies with the shaping factor. This phenomenon occurred because, as the shaping factor increased, the probabilities of exterior constellation symbols decreased further. Given that the KF update was less impacted by the detection error of outside symbols, a high threshold was considered suitable. The phenomenon is consistent with the characteristics of the constellation point distribution varying with the shaping factor, further validating the scheme’s rationality.

3.2. Comparison with Different CPR Algorithms at Variable Shaping Factors

In the preceding section, the performance enhancement of the proposed algorithm was validated. In this section, we compared the performance of the proposed UKF−based CPR algorithm to that of the existing LKF− and EKF−based CPR algorithms, as well as BPS. We examined the performance of the GMI and NGMI for PS 16−QAM, using SNR intervals of 0.5 and various shaping factors, with SNR ranging from 5 to 12.5. The threshold was set appropriately for each shaping factor in order to improve performance. The block length of BPS was 64 and there were 11 test phases. Under varying SNRs, the covariance Q for KF−based algorithms was $1\times {10}^{-5}$, and the covariance R was adjusted.

The performance outcomes were closely correlated with the SNR value, as illustrated in Figure 5. In terms of GMI performance, there was a clear distinction between the BPS and the other algorithms when the SNR was low. Compared to the BPS, these KF−based algorithms demonstrated superior tolerance to the effects of PS in the low SNR region. And performance results demonstrated that the proposed system outperformed the EKF and LKF−based CPR algorithm across the entire SNR region. The NGMI results are displayed in Figure 5. Using the 0.8 NGMI threshold as a benchmark, the proposed method could obtain SNR gains of approximately 0.51 dB, 0.50 dB, and 0.52 dB when compared to the EKF. The required SNR gains were approximately 0.42 dB, 0.52 dB, and 1.11 dB relative to the BPS.

It should be noted that the BPS algorithm performed better than the other methods when the SNR was high enough. This was due to the fact that KF−based techniques, which are independent of test phases, estimate phase noise by updating the state vector, and the estimation’s precision is highly dependent on the initial state vector and the default covariance. Nonetheless, the proposed algorithm maintained superior robustness in the low SNR zone.

Taking into consideration the 0.8 NGMI threshold, the proposed algorithm could accomplish a smaller OSNR penalty than the BPS under the investigated conditions. Increasing the shaping factor decreased the performance disparity between the proposed algorithm and BPS when the SNR was high. It demonstrated that if the shaping factor was high enough, the performance of the proposed algorithm would be comparable to BPS. Consequently, the proposed algorithm is more appropriate for PS systems than the conventional CPR algorithms.

4. Experimental Verification

The experimental verification of the proposed UKF−based CPR algorithm was carried out in a 32 GBaud single polarization coherent optical system, as shown in Figure 6.

At the transmitter end, an offline DSP applied in MATLAB was used to generate PS 16QAM signals following MB distribution. The shaping factor was set as 0.1, 0.15, and 0.2. Digital−to−analog conversion was realized by an arbitrary waveform generator (AWG) with a sampling rate of 64 GSa/s. The analog signals were then amplified by two electrical amplifiers and fed into the IQ modulator. An external cavity laser generated a continuous light wave with a 100 kHz line width. The modulated signals were sent into an Erbium−doped fiber amplifier (EDFA) and then transmitted over 500 km optical fiber link. The optical signal was transmitted through 5 spans of optical fiber, each of length 100 km. An EDFA was set after each span to compensate for optical power attenuation. An ASE source and another VOA were used to adjust the OSNR of the system for the NGMI measurement. At the receiver end, the signals were sent into the coherent receiver to beat with the LO. Then the detected signals were captured by a digital storage oscilloscope (DSO) with a sampling rate of 100 GSa/s. Offline DSP, which was shown in Figure 6b, was used to recover the transmitted data from the received signals.

Figure 7 shows the NGMI performance at different shaping factors in the experimental demonstration. The block length of BPS was 64 and there were 11 test phases. Under varying SNRs, the covariance Q for KF−based algorithms was $1\times {10}^{-5}$ and the covariance R was adjusted. The performance of the NGMI improved as the OSNR increased for all CPR algorithms. Due to the effect of PS, there was a performance disparity between BPS and the other algorithms when the OSNR was low. Using the 0.8 NGMI threshold as a benchmark, the proposed method could obtain SNR gains of approximately 0.10 dB, 0.60 dB, and 0.20 dB when compared to EKF. In comparison to BPS, the requisite SNR enhancements were approximately 0.40 dB, 1.10 dB, and 0.90 dB, respectively. As the OSNR increased, the performance disparity between BPS and the other algorithms decreased. Because the impact of PS on the CPR algorithms decreased as the OSNR increased, and because the performance disparity between KF and BPS was not readily apparent when the OSNR was sufficiently high. Although the performance difference between several algorithms was not readily apparent when the OSNR was high, the proposed method was able to obtain a lower OSNR penalty than the conventional BPS and KF−based algorithm when compared to the 0.8 NGMI threshold. The results of the experiment validate the claim that the proposed UKF−based CPR algorithm with joint detection is more suitable for the PS system.

5. Conclusions

Conventional CPR algorithms are impacted by PS when implemented in PS systems. In this paper, Monte Carlo simulation is utilized to compare the performance of the conventional BPS− and KF−based CPR algorithms during PS transmission. Due to additional decision errors, implementing MAP detection, which has been shown to be optimal for PS systems, directly into the UKF−based CPR algorithm does not result in the anticipated performance enhancement. Therefore, a UKF−based CPR algorithm with a joint detection scheme is proposed in order to prevent making erroneous decisions, and it is verified to be effective for PS systems by both simulation and experiment. By correctly partitioning the constellation symbols and maximizing the use of both MAP and ML detection, the performance of the proposed method can be enhanced. The proposed algorithm is more suitable for PS systems than the conventional BPS− and KF−based algorithms.