Hybrid Cooperative Spectrum Sharing Protocols Based on OMA and NOMA in Cognitive Radio Networks

Abstract

In cognitive radio networks (CRNs), the performance of the primary users (PUs) may suffer adverse effects from the secondary users (SUs) if the spectrum of PUs is haphazardly shared with SUs. In this paper, we propose a hybrid cognitive cooperative protocol based on orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) that improves spectrum utilization by allowing the SUs opportunistic access to the spectrum of the Pus, while guaranteeing the performance of the PU. Specifically, the system can switch between non-cognitive transmission mode, underlay OMA mode, and overlay OMA/NOMA mode, according to the automatic repeat request (ARQ) feedback of PU. The SU has the opportunity to acquire the spectrum to activate the underlay OMA and overlay OMA/NOMA modes only if it listens to the acknowledge (ACK) or negative acknowledge (NACK) feedback from the PU. In order to describe the switching between these three switching modes, a Markov model is developed to analyze the corresponding steady-state probabilities and end-to-end outage probabilities. So, we derive closed-form expressions for the throughput of PU and SU to investigate the spectrum utilization. Numerical and simulation results show that the proposed hybrid cooperative cognitive protocol outperforms the pure OMA hybrid cooperative cognitive protocol.

1. Introduction

In the coming beyond fifth generation (B5G), and sixth generation (6G) networks, the number of mobile devices will show explosive growth. Although massive multiple input multiple output (MIMO) can meet the current access requirements of massive devices and user services [1,2], the low spectrum utilization is still a long-term and common problem in mobile communication networks. However, the proposed cognitive radio technology has shown unique advantages in improving spectrum utilization effectively.

Cognitive radio networks (CRNs) are considered as a promising scheme to adjust user parameters in real time according to environmental changes, which can effectively utilize spectrum and capture spectrum holes. The cognitive radio network consists of primary users (PUs) and secondary users (SUs), and SUs with authorized spectrum can share the spectrum authorized by PUs. CR can improve spectrum utilization by reusing spectrum allocated to authorized users using three dynamic allocation strategies: interweave, overlay, and underlay [3]. In interweave paradigm, SUs access the authorized spectrum that are not used by PUs. However, in underlay mode, PUs and SUs access the spectrum simultaneously under mutual interference, but the interference generated by SUs are not allowed to exceed the maximum range that PUs can bear. Unlike underlay, in overlay mode, SUs opportunistically access the spectrum as a relay. In traditional CRNs, sharing the spectrum of PUs to SUs casually may adversely affect the PU’s performance. Therefore, it is very necessary to develop a new cooperative protocol to allow SUs to opportunistically share PUs spectrum resources while strictly guaranteeing PUs performance.

In addition, integrating non-orthogonal multiple access (NOMA) into CR has shown the possibility to meet 5G requirements of high throughput, low latency, massive connectively and more spectrum sharing [4,5]. In NOMA system, superimpose coding is adopted at the transmitter side to allow multiple users to simultaneously access the spectrum, and successive interference cancellation (SIC) [6,7] is used at the receiver side to separate the signals of the different users. The most popular type of NOMA is the power domain NOMA, where the transmitter sends a superimposed signal to all receivers and serves them at different power factors according to their channel conditions. The method of combining NOMA and cognitive radio is called CR-NOMA. Currently, a wealth of research results has been achieved in CR-NOMA related to underlay NOMA and overlay NOMA.

In the underlay cognitive NOMA network, the secondary transmitter (ST) can provide services for multiple secondary receivers (SR) with NOMA technology under the interference of the primary transmitter (PT). How to minimize the interference of SU on PU link [8,9] is a problem worth pondering. A spectrum sharing protocol with a peak interference constrain was proposed in [10], the results show that the interference channel between the ST and the PU has a more severe effect on the average achievable rate. In [11], the authors use orthogonal spatial modulation technique to improve the spectrum efficiency of the PU and enhance the spectrum usage of the SU. However, none of the work mentioned above takes advantage of the new NOMA and OMA technologies. In recent years, there has also been a lot of work on cooperative spectrum sharing on NOMA technology [12,13]. A novel NOMA assisted cooperative spectrum sharing network with full-duplex ST was presented in [12]. The PT needs the help from the ST to relay its information and as an incentive, ST can access the spectrum of the primary network and send both primary and secondary messages by using NOMA. Similar to the cooperation approach of [12], the authors propose a cooperative spectrum sharing protocol based on NOMA in [14]. The proposed protocol can achieve full-rate at the PU without degrading its outage performance. For better cooperative communication and spectrum sharing, CRNs based on automatic request repeat (ARQ) technology [15] have also been proposed. The throughput of the proposed AF relay and DF relay forwarding models is compared and analyzed. The authors conclude that the proposed protocol provides a more balanced scheme for PU and SU performance. Therefore, it is worth considering the opportunistic access spectrum of SU under the premise of providing a new protocol to guarantee PU transmission.

In the overlay cognitive NOMA networks, the SU acts as a relay to assist transmission of the PU, and as a reward, the SU can send its own signal by using NOMA technology. A hybrid satellite-terrestrial spectrum sharing networks was established in [16]. Secondary networks can maintain stable operation when the outage probability of the system is guaranteed, thus exploring advantages of spectrum sharing opportunities. Similar to [16], the authors also investigate spectrum sharing protocols for hybrid satellite-terrestrial networks in [17]. To improve cooperative spectrum sharing, the NOMA power allocation profile is determined by channel conditioning, thus improving the fairness of spectrum allocation. In addition, the NOMA-based amplify-and-forward cooperative spectrum sharing systems, with imperfect channel state information and continuous interference cancellation, is proposed in [18]. The system performance was significantly improved by using NOMA relay assistance, which demonstrates the effectiveness of the NOMA-based amplify-and-forward cooperation scheme. Based on the above inspiration, what if a new collaborative spectrum sharing protocol was constructed by combining overlay NOMA and underlay OMA strategies?

In this paper, we consider a hybrid cooperative cognitive network based on OMA and NOMA, which is composed of primary user (PU) pairs and secondary user (SU) pairs. PUs consists of a primary transmitter (PT) and primary receiver (PR). SUs consists of a secondary transmitter (ST) and a secondary receiver (SR). It is assumed that the PU’s packet is transmitted at most two times while the SU’s packet is allowed to transmit only once. Automatic repeat request (ARQ) technology is adopted at the PR node. When PT sends a packet, PR replies to acknowledgement (ACK) indicating correct receipt; otherwise, the PR replies to negative acknowledgement (NACK) indicating failure to receive the packets. The packet is discarded if PR sends NACK feedback after PT/ST retransmitting the packet previously failed. In this paper, transmission is divided into non-cognitive radio (non-CR) mode, underlay mode, and overlay cooperative mode. In non-CR mode, PT sends its packet by accessing the spectrum alone, while ST monitors the PU transmission and remains silent. If ACK feedback from PR, as a reward, transmission switch to underlay OMA mode. PT transmits a new packet within the threshold interference range tolerated by PUs, while the SU also sends its own packet; if PR sends NACK, PT retransmits a previously failed packet and ST remains silent. Otherwise, the packet is lost. If PR sends NACK feedback in the non-CR mode and ST has a copy of the PU’s packet, transmission switch to overlay NOMA mode. ST uses NOMA technology to transmit PU’s packet and its own packet. Otherwise, PT retransmits a previously failed packet. For another cooperative protocol proposed, orthogonal multiple access (OMA) is adopted for data transmission in the overlay mode. When PT sends a packet and PR sends NACK feedback, ST only acts as a relay to assist the PUs in transmitting previously failed packets. Otherwise, PT retransmits previously failed packets. The other two transmission modes are similar for hybrid cooperative CR networks based on OMA and NOMA. Finally, the comparison and analysis of the proposed two protocols show that the cooperative cognitive radio network based on NOMA and OMA is superior to the traditional cooperative cognitive radio network based on OMA. The main contributions are summarized as follows:

• We propose a hybrid cooperative cognitive radio networks based on OMA and NOMA and compare with the proposed hybrid cooperative cognitive radio networks base on pure OMA. The difference between the former and the latter is mainly that in overlay mode, ST uses NOMA technology to superimpose PU and SU packets and send them to PR and SR, while the latter uses OMA technology to forward PU packets to PR.
• ARQ technology is used at PR to send received information of the PU’s packet and PT and ST switch cooperative modes according to the feedback information. In the non-CR transmission mode, PT sends the PU’s packet to PR, ST, and SR, while ACK or NACK feedback is sent from PR. If PR sends ACK feedback, the SU and the PU access spectrum in the underlay OMA mode simultaneously. If NACK feedback is sent from PR, ST acts as relay to help forward the PU’s packet to PR in overlay NOMA mode. As a reward, ST can send its own packet simultaneously by using the NOMA principle; or ST only acts as a relay to help forward the PU’s packet to PR in overlay OMA mode.
• We establish a one-dimensional Markov model to analyze the performance of the system through the study of PU mode and state change. We derive the end-to-end outage probability, and then the throughput of PU and SU are derived by using the outage probability and Markov model.
• Finally, we use the MATLAB simulation environment and Monte-Carlo technology to numerically simulate the throughput of PU and SU. Numerical results show that the proposed hybrid cooperative cognitive radio network based on OMA and NOMA is superior to the pure OMA cooperative radio network.

The rest of this paper is organized as follows. Section 2 introduces the system model of hybrid cooperative cognitive radio network based on OMA and NOMA. Section 3 and Section 4 derivate the throughput of PU and SU in terms of the proposed two cooperative protocols. Numerical results and simulations are provided in Section 5. Finally, we conclude the paper in Section 6.

2. System Model and Protocol Descriptions

As illustrated in Figure 1, we consider a hybrid cooperative spectrum sharing system based on OMA and NOMA consisting of a pair of PUs and a pair of SUs. The PUs consist of a primary transmitter (PT), and a primary receiver (PR), while the SUs consist of a secondary transmitter (ST), and a secondary receiver (SR). All nodes are equipped with one antenna and operate in the half-duplex mode.

We supposed that all channels experience independent but not necessarily distributed quasi-static Rayleigh fading. The channel fading coefficients $h_i$ is modeled as independent complex Gaussian random variables denoted by $h_i~CN\left(0,{\lambda }_i\right)$, where $i=PP,PS,PR,SP,SS$ for the channels between node pairs $\mathrm{PT}\to \mathrm{PR},$ $\mathrm{PT}\to \mathrm{ST},$ $\mathrm{PT}\to \mathrm{SR},$ $\mathrm{ST}\to \mathrm{PR}$, and $\mathrm{ST}\to \mathrm{SR},$ respectively. We assume that all channels have an additive white Gaussian noise (AWGN) with mean zero and variance ${\sigma }^2$. The transmit power at PT and ST are denoted by $P_P$ and $P_S$, respectively. The achievable data rate for PUs and SUs are denoted by $R_P$ and $R_S$, respectively.

The proposed spectrum sharing protocol switches between non-CR, underlay mode and overlay NOMA/OMA corporative transmission modes according to feedback information from PR. In the non-CR transmission mode, PUs alone access to the spectrum. The SU remains silent and eavesdrops on the PU’s packets at the same time. In the underlay OMA mode, the PU and SU access the spectrum simultaneously, but the transmit power of the SU must be within the spectrum threshold tolerated by PUs. In overlay NOMA cooperative transmission mode, the PU remains silent and ST superimposes its own data packets with the data packets that the PUs failed to transmit for the first time to PR and SR. PUs’ packets are allowed to be retransmitted once, while SUs’ packets are transmitted only once or not. For another model proposed, OMA is used in overlay cooperation mode. Other modes are similar to those mentioned above. In overlay OMA cooperative transmission mode, the PU stays silent and ST acts as a relay to transmit the PU’s previously failed packets to PR. Therefore, the flow chart of cooperative cognitive radio network based on OMA/NOMA is shown in Figure 2.

2.1. Hybrid Cooperative Cognitive Radio Networks Based on OMA and NOMA

In the non-CR transmission mode, PT transmits a packet while the SU stays silent and eavesdrop on PU transmission. The received packet at PR, ST and SR are given, respectively, by:

$$y_{PR}^{\mathrm{non}-\mathrm{CR}}=\sqrt{P_P}{h}_{PP}{x}_P+n_{PR},$$

(1)

$$y_{ST}^{\mathrm{non}-\mathrm{CR}}=\sqrt{P_P}{h}_{PS}{x}_P+n_{ST},$$

(2)

$$y_{SR}^{\mathrm{non}-\mathrm{CR}}=\sqrt{P_P}{h}_{PR}{x}_P+n_{SR},$$

(3)

where $x_P$ is PU’s packet, $P_P$ is PT’s transmit power, $h_{PP}~\left(0,{\lambda }_{PP}\right)$, $h_{PS}~\left(0,{\lambda }_{PS}\right)$ and $h_{PR}~\left(0,{\lambda }_{PR}\right)$ are the channel coefficients of interference links $\mathrm{PT}\to \mathrm{PR}$, $\mathrm{PT}\to \mathrm{ST}$ and $\mathrm{PT}\to \mathrm{SR}$, respectively, and $n_{PR}~\left(0,{\sigma }^2\right)$, $n_{ST}~\left(0,{\sigma }^2\right)$ and $n_{SR}~\left(0,{\sigma }^2\right)$ are AWGN at the receivers PR, ST and SR, respectively.

If PR sends an ACK, as a reword, the transmission is switched to underlay OMA mode. In this case, PT transmits a new packet while the SUs transmits their own packet. In underlay CRN, the interference power at PR is not allowed to exceed the maximum interference threshold, $I_{th}$. Assuming that the maximum allowable transmit power of ST is $P_{max}$, the transmit power of ST is constrained as [13,19,20]:

$$P_S=min\left\{\frac{I_{th}}{{\left|h_{SP}\right|}^2},P_{max}\right\}$$

(4)

where $h_{SP}~\left(0,{\lambda }_{SP}\right)$ is the channel coefficient of interference link $\mathrm{ST}\to \mathrm{PR}$. In the underlay OMA mode, PT transmits a new packet under the SU’s interference while ST transmits its own packet. The received packet at PR and SR are given by:

$$y_{PR}^{\mathrm{underlay}}=\sqrt{P_P}{h}_{PP}{x}_P+\sqrt{P_S}{h}_{SP}{x}_S+n_{PR},$$

(5)

$$y_{SR}^{\mathrm{underlay}}=\sqrt{P_P}{h}_{PD}{x}_P+\sqrt{P_S}{h}_{SS}{x}_S+n_{SR},$$

(6)

where $x_S$ is SU’s packet, $h_{SS}~\left(0,{\lambda }_{SS}\right)$ is the channel coefficients of interference links $\mathrm{ST}\to \mathrm{SR}$. In the underlay OMA mode, if PR sends a NACK, PT will retransmit the PU’s packet failed previously while SUs remain silent. If the PU’s packet is failed after retransmission, the packet will be lost.

Otherwise, if PR replies by a NACK, the transmission is divided into two cases: (1) if ST correctly decodes the PU’s packet, transmission switches to overlay NOMA mode; otherwise, (2) if ST fails to decode the PU’s packet, PT retransmits that the PU’s packet failed previously while SU stays silent.

(1) If ST correctly decodes the PU’s packet, transmission switches to overlay NOMA mode. In this case, conditioned on ST succeeding in decoding $x_P$, ST combines $x_P$ with its own signal $x_S$ by using NOMA principle. Then, ST transmits the superimposed signal $\sqrt{{\alpha }_P{P}_S}{x}_P+\sqrt{{\alpha }_S{P}_S}{x}_S$ to PR and SR simultaneously. The received packet at PR and SR are given by:

$$y_{PR}^{\mathrm{overlay},\mathrm{NOMA}}=\sqrt{{\alpha }_P{P}_S}{h}_{SP}{x}_P+\sqrt{{\alpha }_S{P}_S}{h}_{SP}{x}_S+n_{PR},$$

(7)

$$y_{SR}^{\mathrm{overlay},\mathrm{NOMA}}=\sqrt{{\alpha }_P{P}_S}{h}_{SS}{x}_P+\sqrt{{\alpha }_S{P}_S}{h}_{SS}{x}_S+n_{SR},$$

(8)

(2) If ST fails to decode the PU’s packet, PT retransmits that the PU’s packet failed previously while SU remains silent. The received packet at PR is similar for non-CR transmission mode.

Based on the above description, the operation of the proposed spectrum sharing protocol is specified by four states, as illustrated in

Table 1. Hybrid cooperative cognitive radio networks based on OMA/NOMA of transmission state.

State	PT	ST
${\mathrm{S}}_1$	PU’s packet	Silent
${\mathrm{S}}_2$	PU’s packet	SU’s packet
${\mathrm{S}}_3$	Silent	PU’s packet and SU’s packet
${\mathrm{S}}_4$	PU’s packet	Silent

. These states are described as follows:

State ${\mathrm{S}}_1$: The PU transmits a packet while SU stays silent and eavesdrops on PU’s transmission. The next state is ${\mathrm{S}}_2$ if an ACK is recevied from PR, otherwise PR sends a NACK and the next state moves to ${\mathrm{S}}_3$ or ${\mathrm{S}}_4$ depending on whether ST decodes the PU’s packet correctly.

State ${\mathrm{S}}_2$: The PU transmits a new packet while ST transmits its own packet. This transmission is operated under interference for the PU and SU. The next state is ${\mathrm{S}}_1$ or ${\mathrm{S}}_4$ depending on the ACK or NACK feedback from PR.

State ${\mathrm{S}}_3$: In the overlay NOMA cooperative mode, ST transmits the superimposed signal $\sqrt{{\alpha }_P{P}_S}{x}_P+\sqrt{{\alpha }_S{P}_S}{x}_S$ to PR and SR, simultaneously. The next state moves to ${\mathrm{S}}_1$ irrespective of whether the PR feedback is ACK or NACK.

State ${\mathrm{S}}_4$: PT retransmits that the packet failed previously while SU keeps silent. The next state moves to ${\mathrm{S}}_1$ irrespective of whether the PR feedback is ACK or NACK.

The four-states Markov model for the proposed protocol is shown in Figure 3, where the state transition probabilities can be defined as follows:

$P_1$: Packet success probability at the channel $\mathrm{PT}\to \mathrm{PR}$.

$P_2$: Packet success probability at the channel $\mathrm{PT}\to \mathrm{PR}$ under interference from the channel $\mathrm{ST}\to \mathrm{SR}$.

$P_3$: Packet failure probability at the channel $\mathrm{PT}\to \mathrm{PR}$ and packet failure probability at the channel $\mathrm{PT}\to \mathrm{ST}$.

$P_4$: Packet failure probability at the channel $\mathrm{PT}\to \mathrm{PR}$ and packet success probability at the channel $\mathrm{PT}\to \mathrm{ST}$.

$P_5$: Packet success probability at the channel $\mathrm{ST}\to \mathrm{PR}$.

The Markov chain of states transition probabilities and the steady state probabilities have following relationships:

$$\left({\pi }_{{\mathrm{S}}_1},{\pi }_{{\mathrm{S}}_2},{\pi }_{{\mathrm{S}}_3},{\pi }_{{\mathrm{S}}_4}\right)\left(\begin{array}{cccc}0& P_1& P_4& P_3\\ P_2& 0& 0& 1-P_2\\ 1& 0& 0& 0\\ 1& 0& 0& 0\end{array}\right)=\left({\pi }_{{\mathrm{S}}_1},{\pi }_{{\mathrm{S}}_2},{\pi }_{{\mathrm{S}}_3},{\pi }_{{\mathrm{S}}_4}\right),$$

(9)

$${\pi }_{{\mathrm{S}}_1}+{\pi }_{{\mathrm{S}}_2}+{\pi }_{{\mathrm{S}}_3}+{\pi }_{{\mathrm{S}}_4}=1,$$

(10)

where ${\pi }_i$ is the steady state probabilities of state i with $i\in I=\left\{{\mathrm{S}}_1,{\mathrm{S}}_2,{\mathrm{S}}_3,{\mathrm{S}}_4\right\}$. The steady-state probabilities of PUs can be obtained by solving the Equations (9) and (10):

$$\{\begin{array}{c}{\pi }_{{\mathrm{S}}_1}=\frac{1}{1+2P_1+P_4+P_3-P_1{P}_2},\\ {\pi }_{{\mathrm{S}}_2}=\frac{P_1}{1+2P_1+P_4+P_3-P_1{P}_2},\\ {\pi }_{{\mathrm{S}}_3}=\frac{P_4}{1+2P_1+P_4+P_3-P_1{P}_2},\\ {\pi }_{{\mathrm{S}}_4}=\frac{P_3+P_1-P_1{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}.\end{array}$$

(11)

2.2. Hybrid Cooperative Cognitive Radio Networks Based on Pure OMA

In the hybrid cooperative cognitive radio network of pure OMA, the transmission is also divided into three modes: non-cognitive transmission mode, underlay NOMA mode, and overlay OMA cooperative mode. In the overlay OMA corporative mode, ST only acts as a relay to assist the transmission of PUs, but does not transmit its own packets. The other two modes are the same as the hybrid cooperative cognitive radio network based on OMA/NOMA, so the operation process of overlay OMA mode is only explained here. The flow chart of the hybrid cooperative cognitive radio network based on pure OMA is shown in Figure 2.

If a NACK is received from PR in the non-CR mode, the transmission is divided into two cases: (1) if ST correctly decodes the PU’s packet, transmission switches to overlay OMA mode; and (2) if ST fails to decode the PU’s packet, PT will retransmit the PU’s packet that failed previously while SUs keeps silent.

(1) If ST correctly decodes the PU’s packet, transmission mode switches to overlay OMA mode. In the overlay OMA, the ST acts as a cooperative relay and retransmits the PU’s packets to PR. The received packet at PR is given by:

$$y_{PR}=\sqrt{P_S}{h}_{SP}{x}_P+n_{PR},$$

(12)

(2) If ST fails to decode the PU’s packet, PT will retransmit that the PU’s packet failed previously while SUs keeps silent. Then, the received packet at PR is the same as the one in non-CR mode.

Based on the above description, the operation of the proposed spectrum sharing protocol is specified by four states, as illustrated in

Table 2. Hybrid cooperative cognitive radio networks based on pure OMA of transmission state.

State	PT	ST
${\mathrm{S}}_1$	PU’s packet	Silent
${\mathrm{S}}_2$	PU’s packet	SU’s packet
${\mathrm{S}}_3$	Silent	PU’s packet retransmission
${\mathrm{S}}_4$	PU’s packet retransmission	Silent

. Specifically, State ${\mathrm{S}}_3$: in this situation it is redefined as ST transmits PU’s packet to PR in the overlay OMA cooperative mode. The next state moves to ${\mathrm{S}}_1$ irrespective of whether the PR feedback is ACK or a NACK. Other states are defined similar to the situation of OMA/NOMA. The Markov model is the same as the hybrid cooperative cognitive wireless network based on OMA/NOMA, except that the transition probability of state ${\mathrm{S}}_3$ to ${\mathrm{S}}_1$ in the model is different. The transition probability of state ${\mathrm{S}}_3$ to ${\mathrm{S}}_1$ is defined as $P_5^{\prime }$.

3. Throughput Analysis of Primary and Secondary Users in Hybrid Cooperative Cognitive Radio Network Based on OMA and NOMA

In non-CR transmission mode, the achieved rates of the channels $\mathrm{PT}\to \mathrm{PR},$, $\mathrm{PT}\to \mathrm{ST}$ and $\mathrm{PT}\to$ $\mathrm{SR}$ are given as:

$$R_{PT\to PR,x_p}^{\mathrm{non}-\mathrm{CR}}={log}_2\left(1+\frac{{\left|h_{PP}\right|}^2{P}_P}{{\sigma }^2}\right),$$

(13)

$$R_{PT\to \mathrm{ST},x_p}^{\mathrm{non}-CR}={log}_2\left(1+\frac{{\left|h_{PS}\right|}^2{P}_P}{{\sigma }^2}\right),$$

(14)

$$R_{PT\to \mathrm{SR},x_p}^{\mathrm{non}-\mathrm{CR}}={log}_2\left(1+\frac{{\left|h_{PR}\right|}^2{P}_P}{{\sigma }^2}\right).$$

(15)

In underlay OMA mode, the achieved rates of the channels $\mathrm{PT}\to \mathrm{PR}$ and $\mathrm{ST}\to \mathrm{SR}$ with interference are given as:

$$R_{\mathrm{PT}\to \mathrm{PR},x_p}^{\mathrm{underlay}}={log}_2\left(1+\frac{{\left|h_{PP}\right|}^2{P}_P}{{\left|h_{SP}\right|}^2{P}_S+{\sigma }^2}\right),$$

(16)

$$R_{ST\to \mathrm{SR},x_s}^{\mathrm{underlay}}={log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{P}_S}{{\left|h_{PR}\right|}^2{P}_P+{\sigma }^2}\right).$$

(17)

In the non-CR mode, if NACK feedback from PR, the transmission is divided into two cases:

(1)

If ST can decode the PU’s packet, transmission switches to overlay NOMA mode. The achieved rate of the channels $\mathrm{ST}\to \mathrm{PR}$ and $\mathrm{ST}\to \mathrm{SR}$ are given as:

$$R_{ST\to \mathrm{PR},x_p}^{\mathrm{overlay},\mathrm{NOMA}}={log}_2\left(1+\frac{{\left|h_{SP}\right|}^2{\alpha }_P{P}_S}{{\left|h_{SP}\right|}^2{\alpha }_S{P}_S+{\sigma }^2}\right),$$

(18)

$$R_{ST\to \mathrm{SR},x_p}^{\mathrm{overlay},\mathrm{NOMA}}={log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_P{P}_S}{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S+{\sigma }^2}\right).$$

(19)

When $R_{\mathrm{ST}\to \mathrm{SR}}\ge R_P$, SR can correctly decode PU’s packet and move it. We consider perfect SIC in practical scenarios, the achieved rate for SU’s packet at SR is expressed as:

$$R_{ST\to \mathrm{SR},x_S}^{\mathrm{overlay},\mathrm{NOMA}}={log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S}{{\sigma }^2}\right).$$

(20)

(2)

If ST fails to decode the PU’s packet, PT will retransmit that the PU’s packet failed previously while SU keeps silent. The achieved rate of the channel $\mathrm{PT}\to \mathrm{PR}$ is the same as Equation (13).

3.1. Throughput of PUs

In this subsection, the throughput of the PU for hybrid cooperative cognitive radio network based on OMA and NOMA is derived. The throughput is defined as the packet transmission success probability multiplied by the steady-state probability. The probability that the target rate exceeds the instantaneous achievable rate of the channel at certain time is defined as the outage probability.

Theorem 1.

The outage probability between any two non-interfering nodes in the channel is given by:

$$p_j^{out}=1-exp\left(-\frac{{\phi }_P}{{\lambda }_i{\rho }_P}\right),$$

(21)

where ${\phi }_P=2^{R_P}-1$,${\rho }_P=\frac{P_P}{{\sigma }^2}$,$j=\mathrm{PT}\to \mathrm{PR},\mathrm{PT}\to \mathrm{ST}, and$ $i=PP,PS$.

Proof. 

See Appendix A.

Theorem 2.

In the underlay OMA mode, PT transmits packet of PU under SU’s interference. The outage probability of $PT\to PR$ channel is expressed as:

$$P_{PT\to PR}^{out,I}=Q_1+Q_2,$$

(22)

where $Q_2=1-exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)-\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right),$ and $Q_1=\left(1-exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\lambda }_{PP}{\rho }_P}-\frac{{\phi }_P}{{\lambda }_{PP}{\rho }_P}\right)\right)exp\left(-\frac{{\rho }_{th}}{{\lambda }_{SP}{\rho }_1}\right)$, with ${\rho }_1=\frac{P_{max}}{{\sigma }^2}$, ${\rho }_{th}=\frac{I_{th}}{{\sigma }^2}$, and ${\phi }_P=2^{R_P}-1$.

Proof. 

See Appendix B.

In the overlay NOMA mode, ST transmits superimposed signal $\sqrt{{\alpha }_S{P}_S}{x}_S+\sqrt{{\alpha }_P{P}_S}{x}_P$ with ${\alpha }_S+{\alpha }_P=1$ to PR and SR simultaneously. The outage probability of $\mathrm{ST}\to \mathrm{PR}$ is expressed as:

$$\begin{array}{c}{P}_{\mathrm{ST}\to \mathrm{PR}}^{out,\mathrm{NOMA}}=Pr\left(R_{\mathrm{ST}\to \mathrm{PR},x_p}^{\mathrm{overlay},\mathrm{NOMA}}<R_P\right)=Pr\left({log}_2\left(1+\frac{{\left|h_{SP}\right|}^2{\alpha }_P{P}_S}{{\left|h_{SP}\right|}^2{\alpha }_S{P}_S+{\sigma }^2}\right)<R_P\right)\hfill \\ =Pr\left({\left|h_{SP}\right|}^2<\frac{{\phi }_P}{{\alpha }_P{\rho }_S-{\phi }_P{\alpha }_S{\rho }_S}\right)=\{\begin{array}{cc}{{\displaystyle \int }}_0^{\frac{{\phi }_P}{{\alpha }_P{\rho }_S-{\phi }_P{\alpha }_S{\rho }_S}}\frac{1}{{\lambda }_{SP}}{e}^{-\frac{x}{{\lambda }_{SP}}}dx,& \frac{{\alpha }_P}{{\alpha }_S}\ge {\phi }_P\\ 1,& \frac{{\alpha }_P}{{\alpha }_S}<{\phi }_P\end{array}\hfill \\ =\{\begin{array}{cc}1-exp(-\frac{{\phi }_P}{{\alpha }_P{\rho }_S{\lambda }_{SP}-{\phi }_P{\alpha }_S{\rho }_S{\lambda }_{SP}}),& \frac{{\alpha }_P}{{\alpha }_S}\ge {\phi }_P\\ 1,& \frac{{\alpha }_P}{{\alpha }_S}<{\phi }_P\end{array}\hfill \end{array}$$

(23)

The PU’s throughput for hybrid cooperative radio wireless network base on NOMA and OMA is derived by the steady state probabilities and state transition probabilities of the Markov chain, given in Figure 3. The outage probabilities of the corresponding channels are given by Equations (21)–(23). Hence the state transition probabilities are expressed by:

$$\begin{array}{c}{P}_1=1-p_{\mathrm{PT}\to \mathrm{PR}}^{out}=exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right),\hfill \\ P_2=1-P_{\mathrm{PT}\to \mathrm{PR}}^{out,I}=exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\lambda }_{PP}{\rho }_P}-\frac{{\phi }_P}{{\lambda }_{PP}{\rho }_P}\right)\cdot exp\left(-\frac{{\rho }_{th}}{{\lambda }_{SP}{\rho }_1}\right),\hfill \\ +\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right),\hfill \\ P_3=p_{\mathrm{PT}\to \mathrm{PR}}^{out}\cdot p_{\mathrm{PT}\to \mathrm{ST}}^{out}=\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\cdot \left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\right),\hfill \\ P_4=p_{\mathrm{PT}\to \mathrm{PR}}^{out}\cdot \left(1-p_{\mathrm{PT}\to \mathrm{ST}}^{out}\right)=\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\cdot exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right),\hfill \\ P_5=1-p_{\mathrm{ST}\to \mathrm{PR}}^{out,\mathrm{NOMA}}=\{\begin{array}{cc}exp\left(-\frac{{\phi }_P}{{\alpha }_P{\rho }_S{\lambda }_{SP}-{\phi }_P{\alpha }_S{\rho }_S{\lambda }_{SP}}\right),& \frac{{\alpha }_P}{{\alpha }_S}\ge {\phi }_P\\ 0,& \frac{{\alpha }_P}{{\alpha }_S}<{\phi }_P\end{array}.\hfill \end{array}$$

(24)

The PU’s packet is successfully transmitted with the probability $P_1$, $P_2$, $P_5$, and $P_1$ at the state ${\mathrm{S}}_1$, ${\mathrm{S}}_2$, ${\mathrm{S}}_3$, and ${\mathrm{S}}_4$, respectively. The PU’s throughput for the hybrid cooperative CR network base on NOMA and OMA can be written as:

$$\begin{array}{c}{\Omega }_{PU}^{\frac{OMA}{NOMA}}=P_1{\pi }_{{\mathrm{S}}_1}+P_2{\pi }_{{\mathrm{S}}_2}+P_5{\pi }_{{\mathrm{S}}_3}+P_1{\pi }_{{\mathrm{S}}_4}\hfill \\ =\frac{P_1}{1+2P_1+P_4+P_3-P_1{P}_2}+\frac{P_1{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}\hfill \\ +\frac{P_5{P}_4}{1+2P_1+P_4+P_3-P_1{P}_2}+\frac{P_3{P}_1+P_1^2-P_1^2{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}\hfill \\ =\frac{P_1+P_1{P}_2+P_5{P}_4+P_3{P}_1+P_1^2-P_1^2{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}=\frac{B}{A}.\hfill \end{array}$$

(25)

Subsisting Equation (24) into Equation (25), we can obtain:

$$\begin{array}{c}A=2+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)-\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\rho }_{th}{\phi }_P}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right)\hfill \\ -exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right).\hfill \end{array}$$

(26)

and the closed-form of B is related to the relationship between ${\alpha }_P/{\alpha }_S$ and ${\phi }_P$, i.e.,

(1)

For ${\alpha }_P/{\alpha }_S\ge {\phi }_P$, we have:

$$\begin{array}{c}B=exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\hfill \\ +exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\left(exp\left(-\frac{{\phi }_P}{{\alpha }_P{\rho }_S{\lambda }_{SP}-{\phi }_P{\alpha }_S{\rho }_S{\lambda }_{SP}}\right)-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\hfill \\ +\left(exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\left(1-\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)+\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)\hfill \\ \times exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right).\hfill \end{array}$$

(27)

(2)

For ${\alpha }_P/{\alpha }_S<{\phi }_P$, we have

$$\begin{array}{c}B=exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\right)\hfill \\ +\left(exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\left(1-\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)+\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)\hfill \\ \times exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right).\hfill \end{array}$$

(28)

3.2. Throughput of SUs

In this section, the throughput of the SU is derived for hybrid cooperative cognitive radio network base on NOMA and OMA. In this network, the SU’s packet is transmitted over the channel $\mathrm{ST}\to \mathrm{SR}$ under the interference of PT’s transmission in underlay OMA transmission mode, and ST transmits a superimposed signal to SR in overlay NOMA cooperative mode. In the overlay NOMA mode, for the successful transmission of SUs packet in channel $\mathrm{ST}\to \mathrm{SR}$, SR can decode its own packet only if it successfully decodes PUs packets. The outage probability of the channel $\mathrm{ST}\to \mathrm{SR}$ is given by:

$$\begin{array}{c}{P}_{\mathrm{ST}\to SR}^{out,x_S}=1-Pr\left(R_{\mathrm{ST}\to \mathrm{SR},x_p}^{\mathrm{overlay},\mathrm{NOMA}}\ge R_P,R_{\mathrm{ST}\to \mathrm{SR},x_S}^{\mathrm{overlay},\mathrm{NOMA}}\ge R_S\right)\hfill \\ =1-Pr\left({log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_P{P}_S}{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S+{\sigma }^2}\right)\ge R_P,{log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S}{{\sigma }^2}\right)\ge R_S\right)\hfill \\ =1-Pr\left({log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_P{P}_S}{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S+{\sigma }^2}\right)\ge R_P\right)\cdot Pr\left({log}_2\left(1+\frac{{\left|h_{SS}\right|}^2{\alpha }_S{P}_S}{{\sigma }^2}\right)\ge R_S\right)\hfill \\ =1-Pr\left({\left|h_{SS}\right|}^2\ge \frac{{\phi }_P}{\left({\alpha }_P-{\alpha }_S{\phi }_P\right){\rho }_S}\right)\cdot Pr\left({\left|h_{SS}\right|}^2\ge \frac{{\phi }_S}{{\alpha }_S{\rho }_S}\right)\hfill \\ =\{\begin{array}{cc}1-{{\displaystyle \int }}_{\varphi }^{\infty }\frac{1}{{\lambda }_{SS}}{e}^{-\frac{x}{{\lambda }_{SS}}}dx,& \frac{{\alpha }_P}{{\alpha }_S}\ge {\phi }_P\\ 1,& \frac{{\alpha }_P}{{\alpha }_S}<{\phi }_P\end{array}=\{\begin{array}{cc}1-exp\left(-\frac{\varphi }{{\lambda }_{SS}}\right),& \frac{{\alpha }_P}{{\alpha }_S}\ge {\phi }_P\\ 1,& \frac{{\alpha }_P}{{\alpha }_S}<{\phi }_P\end{array}\hfill \end{array}$$

(29)

where $\varphi =\max \left\{\frac{{\phi }_P}{\left({\alpha }_P-{\alpha }_S{\phi }_P\right){\rho }_S},\frac{{\phi }_S}{{\alpha }_S{\rho }_S}\right\}$.

Theorem 3.

In the underlay OMA mode, the SU’s packet is transmitted over the channel $ST\to SR$ under the interference of PT’s transmission. The outage probability of $ST\to SR$ channel is given by:

$$P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}=Pr\left(R_{\mathrm{ST}\to \mathrm{SR},x_s}^{\mathrm{underlay}}<R_S\right)=Q_3+Q_4$$

(30)

where $Q_3=exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)-\frac{{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\rho }_P{\lambda }_{SP}{\lambda }_{PR}}\cdot exp\left(-\frac{{\phi }_S}{{\rho }_1{\lambda }_{SS}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\cdot exp\left(DC\right)\cdot Ei\left(-CD\right)$, $C=\frac{{\phi }_S{\rho }_P{\lambda }_{PR}+{\lambda }_{SS}{\rho }_1}{{\lambda }_{SS}{\lambda }_{SP}{\rho }_1{\lambda }_{PR}},D=\frac{{\phi }_S{\lambda }_{SP}+{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\rho }_P{\lambda }_{SP}}$, and $Q_4=\left(1-exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right)\left(1-\frac{{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}+{\phi }_S{\rho }_P{\lambda }_{PR}}\cdot exp\left(-\frac{{\phi }_S}{{\rho }_1{\lambda }_{SS}}\right)\right)$.

Proof. 

See Appendix C.

In the protocol for OMA and NOMA technology, the SU can transmit its own packet at the state $S_2$ and $S_3$, according to the Markov model given in Figure 3. If PR sends ACK feedback in the non-CR transmission mode, the proposed protocol moves on the underlay OMA mode, which corresponds to the state $S_2$ and the SU transmits under the interference of the PU’s transmission. The packet success probability is equal to $1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}$. If a NACK is received from PR in the non-CR transmission mode and ST can successfully decode the PU’s packet, the protocol operates at the state $S_3$. The success probability of the SU’s packet is equal to $1-P_{\mathrm{ST}\to SR}^{out,x_S}$. The SU’s throughput for hybrid cooperative cognitive radio network base on NOMA and OMA can be written as:

$$\begin{array}{c}{\Omega }_{SU}^{\frac{OMA}{NOMA}}=\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}\right){\pi }_{S_2}+\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s}\right){\pi }_{S_3}\hfill \\ =\frac{\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}\right)P_1}{1+2P_1+P_4+P_3-P_1{P}_2}+\frac{\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s}\right)P_4}{1+2P_1+P_4+P_3-P_1{P}_2}\hfill \\ =\frac{\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}\right)P_1+\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s}\right)P_4}{1+2P_1+P_4+P_3-P_1{P}_2}=\frac{E}{A} ,\hfill \end{array}$$

(31)

where the closed-form of A is the same as Equation (26), and subsisting Equations (24), (29) and (30), into Equation (31), we can obtain the closed-form of E is related to the relationship between ${\alpha }_P/{\alpha }_S$ and ${\phi }_P$, i.e.,

(1)

For ${\alpha }_P/{\alpha }_S\ge {\phi }_P$, we have:

$$\begin{array}{c}E=\left(\frac{{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}+{\phi }_S{\rho }_P{\lambda }_{PR}}\left(1-exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right)+\frac{{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\lambda }_{SP}{\rho }_P{\lambda }_{PR}}exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}+DC\right)\cdot \mathrm{Ei}\left(-CD\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}-\frac{{\phi }_S}{{\rho }_1{\lambda }_{SS}}\right)\hfill \\ +\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}-\frac{\varphi }{{\lambda }_{SS}}\right),\hfill \end{array}$$

(32)

where $\varphi =\max \left\{\frac{{\phi }_P}{\left({\alpha }_P-{\alpha }_S{\phi }_P\right){\rho }_S},\frac{{\phi }_S}{{\alpha }_S{\rho }_S}\right\}$.

(2)

For ${\alpha }_P/{\alpha }_S\ge {\phi }_P,$ we have:

$$E=\left(\frac{{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}+{\phi }_S{\rho }_P{\lambda }_{PR}}\left(1-exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right)+\frac{{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\lambda }_{SP}{\rho }_P{\lambda }_{PR}}exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}+DC\right)\cdot \mathrm{Ei}\left(-CD\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}-\frac{{\phi }_S}{{\rho }_1{\lambda }_{SS}}\right),$$

(33)

where $C=\frac{{\phi }_S{\rho }_P{\lambda }_{PR}+{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}{\lambda }_{PR}}$ and $D=\frac{{\phi }_S{\lambda }_{SP}+{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\lambda }_{SP}{\rho }_P}$.

4. Throughput Analysis of Primary and Secondary User in Hybrid Cooperative Cognitive Radio Network Based on Pure OMA

In the section, the throughputs of the PU and SU for a hybrid cooperative cognitive radio network based on pure OMA are derived. In this case, the transmission of non-CR transmission mode and underlay OMA transmission mode are the same as that in the situation of OMA and NOMA. Therefore, only the part of the overlay OMA cooperative mode is analyzed here. In overlay OMA cooperative mode, data transmission can be divided into two cases:

(1)

ST can decode the PU’s packet and act as a relay to help forward the PU’s packet to PR. The achieved date rate for ST to decode the PU’s packet and the channel $\mathrm{ST}\to \mathrm{PR}$ are given as:

$$R_{\mathrm{PT}\to \mathrm{ST},x_p}^{\mathrm{overlay},\mathrm{OMA}}={log}_2\left(1+\frac{{\left|h_{PS}\right|}^2{P}_P}{{\sigma }^2}\right),$$

(34)

$$R_{\mathrm{ST}\to \mathrm{PR},x_P}^{\mathrm{overlay},\mathrm{OMA}}={log}_2\left(1+\frac{{\left|h_{SP}\right|}^2{P}_S}{{\sigma }^2}\right),$$

(35)

(2)

ST fails to decode the PU’s packet, PT will retransmit that the PU’s packet failed previously while SU keeps silent.

4.1. Throughput of PUs

In overlay OMA cooperative mode, ST acts as a relay to help forward the PU’s packet to PR. The outage probability of the channel $\mathrm{ST}\to \mathrm{PR}$ is expressed as:

$$\begin{array}{c}{P}_{\mathrm{ST}\to \mathrm{PR}}^{out,\mathrm{OMA}}=Pr\left(R_{\mathrm{PT}\to \mathrm{ST},x_p}^{\mathrm{overlay},\mathrm{OMA}}>R_P,R_{\mathrm{ST}\to \mathrm{PR},x_P}^{\mathrm{overlay},\mathrm{OMA}}<R_P\right)\hfill \\ =Pr\left({log}_2\left(1+\frac{{\left|h_{PS}\right|}^2{P}_P}{{\sigma }^2}\right)>R_P,{log}_2\left(1+\frac{{\left|h_{SP}\right|}^2{P}_S}{{\sigma }^2}\right)<R_P\right)\hfill \\ =Pr\left({\left|h_{PS}\right|}^2>\frac{{\phi }_P}{{\rho }_P},{\left|h_{SP}\right|}^2<\frac{{\phi }_P}{{\rho }_S}\right)={{\displaystyle \int }}_{\frac{{\phi }_P}{{\rho }_P}}^{\infty }{{\displaystyle \int }}_0^{\frac{{\phi }_P}{{\rho }_S}}\frac{1}{{\lambda }_{PS}}{e}^{-\frac{1}{{\lambda }_{PS}}}\frac{1}{{\lambda }_{SP}}{e}^{-\frac{1}{{\lambda }_{SP}}}dxdy\hfill \\ ={{\displaystyle \int }}_{\frac{{\phi }_P}{{\rho }_P}}^{\infty }\frac{1}{{\lambda }_{PS}}{e}^{-\frac{1}{{\lambda }_{PS}}}\left(1-e^{-\frac{{\phi }_P}{{\lambda }_{SP}{\rho }_S}}\right)dy=exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_S{\lambda }_{SP}}\right)\right).\hfill \end{array}$$

(36)

The PU’s throughput for hybrid cooperative radio wireless network base on pure OMA is derived by the steady state probabilities and state transition probabilities of the Markov chain, given in Figure 3. The outage probabilities of the corresponding channels are given by Equations (21), (22) and (36). Hence, the state transition probabilities are expressed by:

$$\begin{array}{c}{P}_1=1-p_{\mathrm{PT}\to \mathrm{PR}}^{out}=exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right),\hfill \\ P_2=1-P_{\mathrm{PT}\to \mathrm{PR}}^{out,I}=exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\hfill \\ +\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right),\hfill \\ P_3=p_{\mathrm{PT}\to \mathrm{PR}}^{out}\cdot p_{\mathrm{PT}\to \mathrm{ST}}^{out}=(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right))\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\right),\hfill \\ P_4=p_{\mathrm{PT}\to \mathrm{PR}}^{out}\cdot \left(1-p_{\mathrm{PT}\to \mathrm{ST}}^{out}\right)=\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right),\hfill \\ P_5^{\prime }=1-p_{\mathrm{ST}\to \mathrm{PR}}^{out,\mathrm{OMA}}=1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_S{\lambda }_{SP}}\right)\right).\hfill \end{array}$$

(37)

The PU’s packet is successfully transmitted with the probabilities $P_1$,$P_2$, $P_5^{\prime }$, and $P_1$ at the state $S_1$, $S_2$, $S_3$, and $S_4$, respectively. The PU’s throughput for hybrid cooperative radio wireless network base on pure OMA can be written as:

$$\begin{array}{c}{\pi }_{PU}^{OMA}=P_1{\pi }_{S_1}+P_2{\pi }_{S_2}+P_5^{\prime }{\pi }_{S_3}+P_1{\pi }_{S_4}\hfill \\ =\frac{P_1}{1+2P_1+P_4+P_3-P_1{P}_2}+\frac{P_1{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}\hfill \\ +\frac{P_5^{\prime }{P}_4}{1+2P_1+P_4+P_3-P_1{P}_2}+\frac{P_3{P}_1+P_1^2-P_1^2{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}\hfill \\ =\frac{P_1+P_1{P}_2+P_5^{\prime }{P}_4+P_3{P}_1+P_1^2-P_1^2{P}_2}{1+2P_1+P_4+P_3-P_1{P}_2}=\frac{B^{\prime }}{A} ,\hfill \end{array}$$

(38)

where the closed-form of A is the same as Equation (26), and subsisting Equations (37) into Equation (38), we can obtain the closed-form of $B^{\prime }$:

$$\begin{array}{c}{B}^{\prime }=exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\left(exp\left(-\frac{{\phi }_P{\rho }_{th}}{{\rho }_P{\lambda }_{PP}}-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\left(1-\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)+\frac{{\rho }_P{\lambda }_{PP}}{{\phi }_P{\rho }_1{\lambda }_{SP}+{\rho }_P{\lambda }_{PP}}\right)\hfill \\ +exp\left(-\frac{2{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)+exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\right)\hfill \\ +\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PS}}\right)\left(1-exp\left(-\frac{{\phi }_P}{{\rho }_S{\lambda }_{SP}}\right)\right)\right).\hfill \end{array}$$

(39)

4.2. Throughput of SUs

In this section, the throughput of the SU for hybrid cooperative cognitive radio network base on pure OMA is derived. In this network, the SU’s packet is transmitted over the channel $\mathrm{ST}\to \mathrm{SR}$ under the interference of PT’s transmission in underlay OMA transmission mode. The SU can transmit its own packet only at the state $S_2$. Similar to the situation of OMA and NOMA, the success probability of the SU’s packet is equal to $1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}$. The SU’s throughput for the hybrid cooperative cognitive radio network base on pure OMA can be written as:

$$\begin{array}{c}{\pi }_{\mathrm{SU}}^{\mathrm{OMA}}=\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}\right){\pi }_{{\mathrm{S}}_2}\hfill \\ =\frac{\left(1-P_{\mathrm{ST}\to \mathrm{SR}}^{out,x_s,I}\right)P_1}{1+2P_1+P_4+P_3-P_1{P}_2}=\frac{F}{A} ,\hfill \end{array}$$

(40)

where the closed-form of A is the same as Equation (26). Subsisting Equations (30) and (37) into Equation (40), we can obtain the closed-form of F:

$$F=\left(\frac{{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}+{\phi }_S{\rho }_P{\lambda }_{PR}}\left(1-exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}\right)\right)+\frac{{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\lambda }_{SP}{\rho }_P{\lambda }_{PR}}exp\left(-\frac{{\rho }_{th}}{{\rho }_1{\lambda }_{SP}}+DC\right)\cdot Ei\left(-CD\right)\right)exp\left(-\frac{{\phi }_P}{{\rho }_P{\lambda }_{PP}}-\frac{{\phi }_S}{{\rho }_1{\lambda }_{SS}}\right),$$

(41)

where $C=\frac{{\phi }_S{\rho }_P{\lambda }_{PR}+{\rho }_1{\lambda }_{SS}}{{\rho }_1{\lambda }_{SS}{\lambda }_{PR}}$ and $D=\frac{{\phi }_S{\lambda }_{SP}+{\rho }_{th}{\lambda }_{SS}}{{\phi }_S{\lambda }_{SP}{\rho }_P}$.

5. Numerical Results and Simulations

This section provides the MATLAB and Monte-Carlo simulation to simulate and analyze the performance of the proposed hybrid cooperative cognitive radio network based on NOMA and OMA, and to compare the proposed new protocol with the traditional OMA cognitive protocol. We assume the channel gains of all links experience Rayleigh effects with path loss exponent, $v$, i.e., ${\lambda }_{XY}=d_{XY}^{-v}$, where $d_{XY}$ denotes the distance between the transmitter, $X$, and the receiver, $Y$ [5,21]. Without loss of generality, we set $v=4$, PT, ST, PR, and SR are located in two dimensions as $\left(0,0\right)$,$\left(d,0\right)$,$\left(2d,0\right)$ and $\left(d,d\right)$, respectively, where d is distance factor. The number of simulations in this paper is $5\times {10}^5$ times. In fact, ${10}^3$ times of simulation are sufficient to achieve a good fit. For convenience, we use the spectrum sharing protocol based on pure OMA in Section 3 as a basis to compare the performance with the proposed hybrid NOMA/OMA spectrum sharing protocol.

Figure 4 compares the impact of transmit power on throughput performance of the PUs and SUs with different protocols. As can be seen, with the increase of transmit power, the throughput of the PUs continues to increase, while the throughput of the SUs increases first and then decreases. This is because the increase of transmit power leads to the increase of the transmit success probability of the PUs, so the performance of the PUs is improved. However, the SUs opportunistically share the spectrum of the PUs when PUs have spectrum holes. In addition, the throughput of the SUs for the proposed hybrid cooperative cognitive radio network based on OMA and NOMA is better than that of the traditional OMA hybrid cooperative cognitive radio network. This shows that the proposed new protocol can make SUs share the spectrum of PUs opportunistically, on the premise of guaranteeing the performance of PUs.

Figure 5 shows the influence of the transmit power of the PU on the throughput when the transmit power of the PU is 5, 10, and 25, respectively. It can be observed from Figure 5 that with the increase of the PU’s transmit power, the throughput of the PU for the hybrid OMA/NOMA protocol is increasing, and the throughput of both the PU and the SU for the conventional OMA protocol is increasing, while the throughput of the SU for hybrid OMA/NOMA protocol increases first and then decreases. In the hybrid cooperative CRNs based on OMA/NOMA, as the power of the PU increases, the probability of the PU transmitting successfully increases, which results in the SU transmitting in underlay OMA mode, and therefore the throughput of the SU increases. This is attributed to the fact that the higher the power of the PU, the stronger the interference to the SU, resulting in lower throughput of the SU. In addition, when the transmit power of PU is constant, the throughput of PU and SU of the OMA/NOMA hybrid cooperation increases as transmit power of SU increases; the throughput of the PU and SU for the pure OMA hybrid cooperation remains constant. This is because when the transmit power of the PU is fixed, ST uses the NOMA principle to send the superimposed signal of the PU and SU in overlay NOMA mode. In this case, the higher the transmit power of the SU, the better the performance of the PU and SU. However, for the pure OMA hybrid cooperation, in overlay OMA mode, ST only forwards the packet from the PU. In this case, the change of the SU’s transmit power has no effect on the performance of both the PU and SU.

Figure 6 depicts the impact of the transmit power of the PU on the throughput of both PU and SU with different target data rates. As can be seen from the figure, regardless of whether the hybrid OMA/NOMA protocol or the traditional OMA protocol, with the increase of transmit power of the PU, the throughput of the PU shows an increasing trend. This is in line with the general principle that as the PU’s transmit power increases, the throughput of the PU increases. However, with the increase of the PU’s transmit power, the throughput of the SU first increases and then decreases. This is due to the successful transmission of the PU, and as a reward, the SU is transmitted in underlay OMA mode, so the throughput of the SU increases first. Moreover, the PU also transmits and interferes with the SU, which leads to the decrease of SU throughput. In addition, for a given transmit power of the PU, the throughput of the PU decreases with the increase of data rate in both OMA/NOMA protocol and pure OMA protocol. This is due to the increase of data rate would leads to the increase of the outage probability of the PU, and thus the performance of the PU decreases. For the SU, with the increase of transmit power, the SU has the optimal throughput. In this case, only with a small transmit power can the SU transmit successfully, thus increasing the opportunity for the SU to access the spectrum.

Figure 7 demonstrates the impact of distance factor d on the throughput of the PU and SU. The throughput of the PU decreases with the increase of distance factor d in both OMA/NOMA protocol and pure OMA protocol. This indicates that the distance between PUs has a significant impact on performance. In real life, a similar concept can be found: the farther away from a base station, the worse the signal of a cell phone or other terminal will be. For the SU, however, with the increasing of distance, the throughput first increases and then decreases. This is due to the fact that in the cooperative CRNs, ST generally acts as a relay to help the PU, and the SU exists at an optimal distance from the PU to help the PU’s transmission, while at the same time increasing the spectrum access rate.

The impact of the power allocation factor and the transmit power on the throughput PU and SU are shown in Figure 8 and Figure 9. It can be obtained from Figure 8 that increasing the transmit power of the PU will improve the performance of the hybrid OMA/NOMA protocol, while it has no effect on the performance of the pure OMA protocol. The main reason is that the power allocation factor is used in NOMA technology in overlay mode, resulting in the higher power allocated to the PU users, the higher the probability of the PU transmitting successfully, and therefore the better the performance. In the pure OMA protocol, there is no power allocation factor. Therefore, for the pure OMA protocol, the change of power allocation factor will not cause the performance change. In addition, when the power allocation factor is constant, the higher the transmitted power of the SU, the better the PU performance. This is because the data transmission is in overlay NOMA mode and ST is the transmitter of SU, so the higher the transmit power of the SU, the better the system performance. As can be seen from Figure 9, with the increase of the PU’s transmit power, the throughput of the SU in hybrid OMA/NOMA protocol increases first and then decreases, while there is no impact on the performance of SUs in the pure OMA protocol. In this case, the transmission is carried out in overlay NOMA mode. Especially, there exists an optimal power allocation value, i.e., ${\alpha }_P=0.65,{\alpha }_S=0.35$. In addition, the performance change of the PU is consistent with that in Figure 8. For a given power allocation factor, the higher the transmit power of the SU, the better the performance of the SU.

Figure 10 shows the impact of the interference threshold on the system throughput. As can be seen from the figure, with the increase of the interference threshold, the throughput of the PU decreases while the throughput of the SU increases. This is because the higher the interference threshold, the higher the transmitting power of the SU, the higher the probability of successful transmitting of the SU, and therefore the best performance of the SU. For the PU, when the interference threshold is larger, the interference of the SU to the PU is larger, which leads to the degradation of PU performance. As shown in the figure, the throughput of the PU for the hybrid OMA/NOMA protocol is the same as that for the pure OMA protocol. This is mainly because the hybrid OMA/NOMA protocol and the pure OMA protocol distinguish between NOMA technology and OMA technology in overlay mode, but it is the same in the underlay mode. Therefore, the interference threshold has no effect on the throughput for the two protocols, and the throughput of the PU for the two protocols are consistent.

6. Conclusions

In this paper, we proposed a new hybrid cooperative CRNs based on OMA and NOMA technology, if the SU shares the spectrum of the PU directly without any constraints, the PU may be adversely affected. Therefore, the PU is transmitted once in the first time slot, and then the transmission mode is switched according to the ACK/NACK feedback information of PR to improve spectrum utilization. As a reward, the PU can retransmit in overlay NOMA mode or transfer new packets in underlay OMA mode. The proposed cooperative CRNs is divided into three modes: non-CR transmission mode, underlay mode, and overlay cooperative transmission mode. In the non-CR transmission mode, PT sends a packet to PR, ST, and SR, and ACK/NACK feedback from PR. If PR sends ACK, the transmission switches to underlay OMA mode. In this case, the PU and SU accessed the spectrum simultaneously. If PR sends a NACK feedback, the transmission mode switches to the overlay mode. The ST either sends a superimposed signal consisting of the decoded signal from PT and its own signal intending to SR, or acts as a relay to forward the decoded signal from PR. Accordingly, we established a one-dimensional Markov model according to the switching states between cognitive modes and the state of the PU, and formulized the transition probabilities and steady-state probabilities between states. We obtained the end-to-end outage probability, the probability of PU success in each state, and the throughput performance of the PU and SU. Finally, From the numerical results and simulation results, we validated that the performance of the proposed hybrid CRNs based on NOMA and OMA is better than that of the hybrid CRNs based on pure OMA. For future work, we can consider multi-antenna NOMA or next generation wireless communication technologies.