Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks

Abstract

Recently, hybrid band communications have received much attention to fulfil the exponentially growing user demands in next-generation communication networks. Still, determining the best band to communicate over is a challenging issue, especially in the dynamic channel conditions in multi-band wireless systems. In this paper, we manipulate a practical online-learning-based solution for the best band/channel selection in hybrid radio frequency and visible light communication (RF/VLC) wireless systems. The best band selection difficulty is formulated as a multi-armed bandit (MAB) with cost subsidy, in which the learner (transmitter) endeavors not only to increase his total reward (throughput) but also reduce his cost (energy consumption). Consequently, we propose two hybrid band selection (HBS) algorithms, named cost subsidy upper confidence bound (CSUCB-HBS) and cost subsidy Thompson sampling (CSTS-HBS), to efficiently handle this problem and obtain the best band with high throughput and low energy consumption. Extensive simulations confirm that CSTS-/CSUCB-HBS outperform the naive TS/UCB and heuristic HBS approaches regarding energy consumption, energy efficiency, throughput, and convergence speed.

1. Introduction

Recently, hybrid band (HB) systems, mainly radio frequency/visible light communication (RF/VLC) systems, have received attention as an attractive paradigm to boost ultra-high-capacity wireless applications with enhanced connectivity via multi-band standards [1]. Nevertheless, those HB systems encounter fast channel fading, signal attenuation, acute blockage effects, etc. [2]. Still, the optimal hybrid band selection (HBS) of such systems is difficult to model due to the various multi-frequency bands with variable dynamic channel status. Additionally, HB systems should attain satisfying arrangements as soon as possible in order to maintain a high quality of service (QoS) in various application environments. Implementing efficient HBS is not straightforward because of the rapidly adapting environments where their performance relies on a plethora of parameters [3]. As a result, there is a strong direction in the wireless world to use machine learning (ML) instead of classic HBS protocols based on predefined rules or heuristics, where learning from the past is a powerful direction.

1.1. Related Work

Previous formulations of the HBS optimization problem targeting the selection of RF (Wi-Fi/WiGig) or VLC for every transmitter-receiver (Tx-Rx) pair have required full channel knowledge; besides, such formulations are computationally difficult. Former deep learning (DL)-aided solutions such as deep neural networks (DNNs) [4] and convolutional NNs (CNNs) [5] require huge training data sets, which might be hard to obtain till the roll-out of B5G/6G systems with HBs. In [6], a deep Q-learning algorithm is employed to solve the joint optimization problem in HB systems. In [5,7], the coauthors of this manuscript planned a DL-assisted prospective channel assignment framework to tackle the changeable channel environment difficulty in hybrid-band relay systems. However, they neglected the VLC band and considered RF bands only.

Current online-learning-enhanced solutions mostly influenced by reinforcement learning (RL) and multi-armed bandits (MAB)  [8,9,10,11,12] mainly consider dynamic multi-channel access problems. The authors in [13] propose a distributed joint power allocation and channel assignment approach for improved power performance in D2D pairs. In addition, in [14,15], a context-aware transfer learning network selection technique for indoor RF-VLC systems was developed. However, previous work neither found an efficient practical solution nor considered the optimal HBS problem. MAB is an ML technique, where the learner tries to maximize the cumulative rewards and minimize the cost through online learning to handle the famous “exploitation–exploration compromise” [16]. Lately, bandits have been leveraged for vast wireless communication problems, including D2D communications [17,18,19,20], relay probing [21], aerial-enabled communications [22], wireless sensor networks (WSNs) [23], resource allocation [24,25], reconfigurable intelligent surface (RIS) enhanced communications [26,27], and millimeter Wave beamforming [28,29,30]. Furthermore, the authors of this paper recently contributed to the HBS problem using energy-aware MAB solutions [31,32,33] via subtracting the cost from the reward in the exploration part only. However, our main focus was to only handle the exploration–exploitation dilemma, which might cause more energy consumption to obtain the highest cumulative payoff. Hence, in this paper we extend the HBS problem to optimize both cost and payoff metrics and avoid causing exorbitant costs. Specifically, we anticipate the cost in both the exploitation and exploration parts of the bandit formula. This is more applicable especially in applications that have changed types of cost and payoff. Moreover, practical energy-aware HBS applications target both reward maximization (i.e., throughput) and cost minimization (battery life) due to their limited capacity.

1.2. Paper Contributions

In this paper, we attempt to manage the costs (energy consumption) of the HBS problem, where the learner is able to tolerate losing a small amount from the highest payoff (calculated as the payoff that the naive MAB approach could achieve in the absence of costs). This can be modeled as multi-objective optimization, which is important to investigate using online learning solutions, especially advanced proper MAB methods. This paper’s contributions are highlighted as follows:

• We aim to relax/reformulate the HBS multi-objective optimization problem and obtain acceptable solutions (i.e., sub-optimal HBS decisions) in real time without prior channel knowledge using online learning techniques that easily handle blockage and energy consumption during the selection process.
• We reformulate the HBS optimization problem into a cost subsidy multi-armed bandit (CS-MAB) that accounts for the cost during selection in both exploitation and exploration terms.
• We propose CS—upper confidence bound (CSUCB-HBS) and CS—Thompson sampling (CSTS-HBS) algorithms and evaluate their performance compared with the ordinary MAB techniques (UCB and TS), and traditional HBS (i.e., conventional and random choice).
• Simulation results indicate the superior performance of our proposed CS-MAB techniques over classical MAB methods, especially CSTS-HBS, which exhibits better performance than CSUCB-HBS and others.

The paper is organized as follows: Section 2 discusses the considered HBS system model including the formulas of utilized channel models. Section 3 overviews the HBS optimization problem formulation. Section 4 discusses the proposed CS-HBS algorithms. Section 5 discusses the simulation results, followed by the paper’s conclusion in Section 6.

2. System Model

Figure 1 presents the HB design under consideration, whereby a pair of Tx/Rx devices are presented for simplicity. The Tx/Rx pair might consist of base stations, mobile terminals or D2D relays, mounted by hybrid RF (i.e., WLAN/WiGig)/VLC frequency bands. The RF channel is IEEE 802.11ac/n (WiFi in 5.25 GHz and 2.4 GHz), and we leverage the linkage formula of [19]. The WiFi received power can be expressed as

$$P_{R_x}^F\left[dBm\right]=P_{T_x}^F\left[dBm\right]-P_{L_0}^F-10{\alpha }^F{log}_{10}\left(r\right)-{\psi }^F,$$

(1)

where $P_{R_x}^F$, $P_{T_x}^F$, and $P_{L_0}^F$ reflect the received power at Rx, the transmitted power from Tx, and the referenced path loss, respectively. ${\alpha }^F$ is the WiFi path loss exponent, while ${\psi }^F\backsim \mathcal{N}\left(0,{\sigma }^F\right)$ is the log-normal shadowing with zero mean and ${\sigma }^F$ standard deviation. Tx–Rx separation distance is denoted by r.

For the WiGig linkage model, we leveraged the 38 GHz band [19]. The WiGig delivered power ($P_{R_x}^G$) with accounting both beamforming (BF) and blocking influences [19] formulated by

$$P_{R_x}^G=\mu \left({\mathbb{P}}_{LOS}\left(r\right)\right)P_{T_x}^G{\Lambda }_{T_x}{\Lambda }_{R_x}/P{L}^G\left(r\right),$$

(2)

where $\mu \left({\mathbb{P}}_{LOS}\left(r\right)\right)$ is a random variable (RV) that follows Bernoulli distribution and reflects the blockage influence via proximity-dependent line-of-sight (LOS) likelihood denoted by ${\mathbb{P}}_{LOS}\left(r\right)$. $P_{T_x}^G$ is the Tx’s WiGig transmitted power. The Tx and Rx BF gains are ${\Lambda }_{T_x}$ and ${\Lambda }_{R_x}$, respectively. The distance-based path loss, $P{L}^G\left(r\right)$, is expressed in dB by:

$${10log}_{10}\left(P{L}^G\left(r\right)\right)=P{L}_0^G+10{\alpha }^G{log}_{10}\left(r\right)+{\chi }^G,$$

(3)

where $P{L}_0^G,{\alpha }^G$, and ${\chi }^{G,L}\backsim \mathcal{N}\left(0,{{\sigma }^2}^G\right)$ are the WiGig path loss at a benchmark distance $r_0$, WiGig path loss exponent, and WiGig-based log-normal shadowing having zero mean and variance of ${{\sigma }^2}^G$, respectively. We take into account only the LOS path because of its dominant nature [34]. ${\Lambda }_{T_x}$ and ${\Lambda }_{R_x}$ are the 2D steerable antenna model with a Gaussian main loop profile, which can be written as [19]:

$$\Lambda \left(\theta \right)={\Lambda }_0{e}^{-4ln\left(2\right){\left(\frac{\theta }{{\theta }_{-3dB}}\right)}^2},{\Lambda }_0={\left(\frac{1.6162}{sin\left(\frac{{\theta }_{-3dB}}{2}\right)}\right)}^2,$$

(4)

where $\theta ,{\theta }_{-3dB}$, and ${\Lambda }_0$ reflect the azimuth angle, half power beamwidth, and the largest total radiated power by the antenna, respectively.

Regarding VLC linkage formulation, we employ the Lambertian model suitable for light-emitting diode (LED) transmitters [1], where the LoS channel gain for indoor VLC using Lambertian layouts is calculated as follows:

$$H_{\mathit{VLC}}=\frac{(g+1)A_R}{2\pi r^2}{G}_{T_x}\left({\varphi }_{vlc}\right)g\left({\varphi }_{vlc}\right){cos}^m{\theta }_{vlc}cos{\varphi }_{vlc}\forall {\varphi }_{vlc}<{\varphi }_c,$$

(5)

where $A_R$ is the optical detector’s physical area. ${\theta }_{vlc}$ and ${\varphi }_{vlc}$ are the angles of incidence and radiance, respectively. ${\varphi }_c$ identifies the Rx field vision’s width and $(g=-\frac{ln\left(2\right)}{ln(cos{\theta }_{vmax})})$ reflects the Lambertian model’s order. The gains of the optical filter and concentrator are $G_{T_x}\left({\varphi }_{vlc}\right)$ and $g\left({\varphi }_{vlc}\right)$, respectively. Hence, the delivered optical power, $P_{R_x}^{vlc}$, is expressed as:

$$P_{R_x}^{vlc}=H_{\mathit{vlc}}{P}_{T_x}^{vlc}.$$

(6)

Finally, regarding the hybrid band blockage model, we leveraged the urban frequency-dependent blockage model given in [35]. This model is almost static, where the Tx is fixed and the Rx moves away and the blocker (small/large car) is located at a fixed distance from the Tx. Therefore, the generalized frequency-dependent blockage formulation can be expressed as [35]:

$$Blockingloss\left[dB\right]={\beta }_b+{\gamma }_{b}log(1+\frac{f_{c,n}}{1\mathrm{GHz}}),$$

(7)

where ${\gamma }_b$ and ${\beta }_b$ define the slope and the intercept of the linear relationship, and b reflects the blocker dimensions (i.e., small or large). To estimate the blocking loss, the above equation was utilized whilst changing the blocker size and the utilized frequency.

3. Problem Formulation

The aforementioned HBS problem in our HB system is formulated as follows:

$$\begin{array}{cc}\hfill \underset{m,n}{arg\; max}& \sum _{m=1}^M\sum _{n=1}^{N_m}{x}_{mn}\mathbb{E}\left({\psi }_{mn}\left(t\right)\right)\hfill \\ \hfill s.t.\end{array}$$

(8a)

$$\begin{array}{cc}\hfill & \underset{m,n}{arg\; min}{\mathcal{C}}_{T_x,mn}\left(t\right)\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill & {\Xi }_{T_x,mn}\left(t\right)>{\Xi }_{th},\hfill \end{array}$$

(8c)

$$\begin{array}{cc}\hfill & \sum _{m=1}^M{N}_m\le J\hfill \end{array}$$

(8d)

$$\begin{array}{cc}\hfill & \sum _{m=1}^M\sum _{n=1}^{N_m}{x}_{mn}=1,\hfill \end{array}$$

(8e)

$$\begin{array}{cc}\hfill & x_{mn}\in \{0,1\},\hfill \end{array}$$

(8f)

where M defines the number of heterogeneous frequency bands and $N_m$ is the channels count in each band m. J defines all the available channels over whole bands. ${\psi }_{mn}\left(t\right)$ is the Tx-Rx linkage throughput at time t using channel n of band m. ${\Xi }_{T_x,mn}\left(t\right)$ is the remaining energy (in Joules) of Tx at time t upon the used channel n of band m, and ${\Xi }_{th}$ is the threshold level energy at which the transmitter will not be capable of setting up wireless linkages and preserve its energy for its core purpose. $x_{mn}$ is a decision variable for choosing the best band and its related channel that maximizes the throughput $\mathbb{E}\left({\psi }_{mn}\left(t\right)\right)$ expressed as:

$${\psi }_{mn}\left(t\right)=\frac{B_{mn}{T}_D{\Gamma }_{mn}\left(t\right)}{U\left(t\right)T_{h,mn}+T_D},$$

(9)

where $T_D$ is the data transmission duration while $T_{h,mn}$ is the Tx-Rx time overhead due to the operating frequency. $B_{mn}$ is the bandwidth, and ${\Gamma }_{mn}\left(t\right)$ is the spectral efficiency (SE) in bps/Hz upon the selected band/frequency at t, given by

$${\Gamma }_{mn}\left(t\right)={log}_2(1+\frac{P_{R_x}^{mn}\left(t\right)}{N_0+I\left(t\right)}),$$

(10)

where $P_{R_x}^{mn}\left(t\right)$ is the Rx received power at time t due to the chosen band, $N_0$ is the Rx’s noise power, and $I\left(t\right)$ is the interference from surrounding equipments that use that frequency. Only the interference from the two Wifi channels is considered here. The WiGig and VLC systems have negligible interference compared to the random noise as they are directional systems. As a result, there is a trade-off between investigating alternative frequency bands/channels and keeping the selected band to obtain the maximal throughput while reducing energy usage and so extending the battery lifespan. Such a trade-off can be addressed using sequential self-decision-making online learning algorithms. Therefore, we handle the above problem via reformulating it as a CS-MAB to not only deliver ultra-fast decisions under the influence of variability and dynamic blocking, but also to save the battery life of the transmitter. MAB is a confidence algorithmic approach that perfectly manipulates the exploration–exploitation compromise [16,17,29]. The main CS-MAB layout includes N probable actions (i.e., arms) to decide the best one within T trials. In each trial $t\in T$, selecting an arm, the learner fetches a payoff from the picked arm and consumes a cost due to the choice of that arm  [16]. The player tries not only to maximize his reward but also to minimize his cost consumption. The energy consumption of Tx according to the selected band, ${\mathcal{C}}_{T_x,N_{\mathit{MAB}}^{*}}\left(t\right)$ given in (8b), is expressed as follows:

$${\mathcal{C}}_{T_x,N_{\mathit{MAB}}^{*}}\left(t\right)=\frac{P_{T_x}^N{L}_D}{{B_{N_{\mathit{MAB}}^{*}}\Gamma }_{N_{\mathit{MAB}}^{*}}\left(t\right)},$$

(11)

where $\mathit{MAB}$ refers to the applied MAB technique (e.g., CSUCB, CSTS, UCB, and TS) and $C_{T_x,N_{\mathit{MAB}}^{*}}\left(t\right)$ is the consumed energy to transmit a datum of $L_D$ bits with a transmission speed of ${B_{N_{\mathit{MAB}}^{*}}\Gamma }_{N_{\mathit{MAB}}^{*}}\left(t\right)$ bps.

The objective function (8a) contains several constraints. For each Tx, constraint (8b) indicates that the selected band should have low power consumption. Constraint (8c) states that Tx’s standing energy should be greater than a certain level, ${\Xi }_{th}$. The following constraint (8d) indicates that the overall number of channels over all the frequency bands analyzed should be equal to or less than the observed accessible number of channels, J. Afterwards, constraints (8e) and (8f) indicate that a relay node can only choose one band and its associated channel at a time, and $x_{m,n}$ is a binary decision variable. We should always draw the arm with the lowest cost subsidized from a of the maximum mean reward ${\mu }_{m,n}$. Note that $0\le a\le 1$. If $a=0$, then the priority is the highest reward arm regardless cost and if $a=1$, the priority will be the lower-cost arm regardless its reward value. Hence, the optimal multi-objective arm is the cheapest arm of the high-quality arms that achieves

$${\{m,n\}}^{*}=\underset{m,n}{arg\; max}(1-a){\mu }_{m,n}S.T\underset{m,n}{arg\; min}{\mathcal{C}}_{m,n}.$$

(12)

4. Envisioned CS-HBS Methods

In this section, we discuss our planned CSUCB-HBS and CSTS-HBS schemes. Both methods modify the naive UCB and TS stochastic bandit schemes, respectively, to be cost subsidy, where the Tx can tolerate a small loss from the largest payoff (subsidy, i.e., the amount of payoff the Tx may forgo to refine costs). Since UCB and TS are the most common stochastic MAB schemes, we present the cost subsidy concept of both of them to check their performance regarding the HBS issue.

4.1. CSUCB-HBS Algorithm

Our proposed CSUCB-HBS algorithm makes use of the UCB strategy, which provides optimism under unknown channel constrains through heterogeneous bands. Rather than using the same confidence bound for each arm in every trial, the UCB strategy selects the best arm via a conservative estimate [16]. Changed from the original UCB, CS-UCB first chooses the highest reward according to the subsidy parameter, and then investigates the cheapest arm from the feasibility set to play with it. This can be expressed as:

$$\begin{array}{c}{\mu }_N^{score}\left(t\right)={\overline{\psi }}_N\left(t\right)+\sqrt{\frac{2lnt}{{\rho }_{N,t}}},\end{array}$$

(13a)

$$\begin{array}{c}{m}_t=\underset{N}{arg\; max}{\mu }_N^{score}\left(t\right),\end{array}$$

(13b)

$$\begin{array}{c}F\left(t\right)=\{N:{\mu }_N^{score}\left(t\right)-(1-a){\mu }_{m_t}^{score}\left(t\right)\ge 0\}\end{array}$$

(13c)

$$\begin{array}{c}{N}_{CSUCB-HBS}^{*}\left(t\right)=\underset{N\in F\left(t\right)}{arg\; min}{\mathcal{C}}_N;\end{array}$$

(13d)

where ${\overline{\psi }}_N\left(t\right)$ is the average throughput collected from the transmission band N until time t. ${\rho }_{N,t}$ is a counter for arm N if it has been drawn until time t. Equation (13a) is the original UCB equation and its arm selection $m_t$ policy in Equation (13b) without arm cost consideration [16]. $F\left(t\right)$ is the feasibility set according to subsidy factor a from reward. Equation (13d) is the CSUCB-selected cheapest arm within $F\left(t\right)$ computed from (11).

4.2. CSTS-HBS Algorithm

Similar to CSUCB-HBS, the CSTS-HBS makes use of the TS algorithm [16] Bayesian strategy through prior Gaussian reward assumption. After defining the maximum reward arm, the feasibility set is constructed according to the subsidy value a, and then the cheapest arm is drawn from this set. This can be mathematically expressed as:

$$\begin{array}{c}{N}_{TS-HBS}^{*}\left(t\right)=arg\underset{N}{max}\left\{{\theta }_n\left(t\right)\right\},{\theta }_N\left(t\right)\sim \mathbb{N}({\overline{\psi }}_N\left(t\right),\frac{1}{{\rho }_{N,t}+1}),\end{array}$$

(14a)

$$\begin{array}{c}F\left(t\right)=\{N:{\theta }_N^{score}\left(t\right)-(1-a){\theta }_{N_{TS-HBS}^{*}}^{score}\left(t\right)\ge 0\}\end{array}$$

(14b)

$$\begin{array}{c}{N}_{CSTS-HBS}^{*}\left(t\right)=\underset{N\in F\left(t\right)}{arg\; min}{\mathcal{C}}_N;\end{array}$$

(14c)

where $\mathbb{N}({\overline{\psi }}_N\left(t\right),\frac{1}{{\rho }_{N,t}+1})$ is a normal distribution with ${\overline{\psi }}_N\left(t\right)$ mean and $\frac{1}{{\rho }_{N,t}+1}$ variance.

Algorithm 1 outlines the key steps of our suggested online HBS algorithms. First, we try each arm once (i.e., channel) ($N=N_{ch}$) and fetch its payoff. If the conditions $(N_{ch}+1)<t<T$ and $E_{T_x}>E_{th}$ are not fulfilled, no more steps are completed and the algorithm terminates. On the other hand, it extracts a better payoff (channel index) $N_{MAB}^{*}\left(t\right)$ in the time trial $t\in T$ based on the CSUCB-HBS and CSTS-HBS policies, which select the lowest-cost arm from $(1-a)$ of the maximum reward, respectively. Afterwards, the bandit-based parameters are upgraded and the remaining energy levels of the transmitting device are estimated taking into account the drown band/channel. As a result, when the Tx decides to send new data frames to Rx, the process begins again.  

Algorithm 1: CSUCB/CSTS-HBS Algorithms

5. Results

This section investigates the performance of our proposed CSUCB-HBS and CSTS-HBS algorithms compared to the original UCB/TS [16] schemes and traditional (e.g., optimal, conventional, and random) HBS schemes. The optimal HBS is achieved through the instantaneous selection of the best channel, while the conventional HBS scheme investigates all available bands first and then decides which one to link with. Finally, the random HBS approach selects a random band without any channel quality curiosity.

Table 1. Simulation parameters.

Simulation Parameters		Value
Number of channels		4 (WiFi 2.4 GHz, 5.25 GHz, WiGig 38 GHz, VLC ${10}^5$ GHz)
T, $L_D$, $E_{th}$		1000, 1 TB, 1%
Operating frequencies of each channel		5.25, 2.4, 38, ${10}^5$ GHz
$BW$		40, 20, 40, 20 MHz
r		{10–100} m
Blocking model [35]	Small blocker: {length, width, height}	{5.07, 1.69, 1.93} m
	Large blocker: {length, width, height}	{7.01, 2.04, 2.63} m

lists the utilized simulation parameters including blockage information. The average throughout, convergence rate, and the normalized energy consumption per selected band/channel in mjoule are the evaluation metrics.

Figure 2 presents the energy consumption against different values of the cost subsidy parameter a for both CSUC-HBS/CSTS-HBS. At low a values, the energy consumption is increased, especially the CSUCB-HBS scheme due to the main focus of the maximum reward. At higher values of a, the energy consumption is reduced according to the algorithm selection policy.

Figure 3 presents the throughput performance evaluation of our proposed CSUCB-HBS and CSTS-HBS algorithms at $a=0.5$ with UCB, TS, optimal, conventional, and random HBS schemes at different r values at three changeable blockage types (i.e., none, small, and large). The attained reward, i.e., th average throughput, is reversely proportional to the blockage value which is apparent in the downward transition of the average throughput values due to blocked objects, i.e., no-blocking, small blocking, and large blocking, as illustrated in Figure 3a–c, respectively. Compared to the traditional and random HBS schemes, the envisioned CSUCB-HBS and CSTS-HBS techniques demonstrate promising performances (near optimum). The attained average throughput throughout all distance range is fairly close compared to the ideal case. The CSTS-HBS and CSUCB-HBS techniques deliver up to 99% and 80% average throughputs, respectively, within all separation distances values compared to the optimal case. Thus, CSTS-HBS arises as the best economic approach for the HBS difficulty due to not only the TS’s Bayesian learning policy of TS but also its ability to save energy consumption. The second best performance is the CSUCB-HBS due to applying both upper-bound and cost subsidy policies. Nerveless, the random HBS attains only 60% of the average throughput, offering the worst performance due to the randomized selection policy without any considerations.

Figure 4 presents the energy consumption of the HBS schemes at no blocking and $a=0.7$. Our proposed CSTS-HBS algorithm outperformed all the other schemes due to the proper channel selection strategy while minimizing the consumed energy of the transmitter. As r is incremented, the energy consumption of all traditional HBS methods are increased relatively with an increase in distance, whereas CSTS-HBS/CSUCB-HBS methods persisted with considerably superior low consumption performance. Still, the conventional HBS technique offers the highest energy consumption due to attempting all the available bands in every trial without energy awareness, opposite to our proposed MAB policies. Hence, the promising practical results enhance the affordability of the proposed CSTS-HBS then CSUCB-HBS schemes in HBS in B5G/6G systems.

The convergence curves of the proposed CSUCB-HBS and CSTS-HBS schemes are drawn in Figure 5 at $r=10$ m. The results show that the suggested CSTS-HBS method exhibits a superior performance over all the trials t. As t approaches 400, the proposed CSTS-HBS scheme converges to 99.5% of the optimal throughput due to Bayesian and cost subsidy strategies. Furthermore, the CSUCB-HBS reaches 97.2% due to the upper bound and cost subsidy conceptualizations. This result demonstrates the practicality of our envisioned methods for HBS in B5G/6G systems with prolonging the battery life time.

Finally, Figure 6 exhibits the viability of our envisioned HBS schemes in terms of energy efficiency (defined as the average payoff (throughput) over energy expenditure per chosen band in bit/sec/joule [32,33]) in Gbps/mJ over distinct Tx-Rx distances. Here, Figure 6a–c preview the energy efficiency performance for no-blocking, small blocking, and large blocking layouts, respectively. For all approaches, the energy efficiency is inversely related to the separation distance because of the path loss effect. For all the plotted blocking layouts, the CSTS-HBS method outperforms other approaches due to its better performance and appropriate channel selection policy, reflecting the lowest energy consumption. Meanwhile, TS-HBS, CSUCB-HBS, and UCB-HBS displayed encouraging energy efficiency performances over all separation distances, respectively. However, due to the traditional choice strategies, the random and conventional approaches show much worse performances due to randomization and attempting whole available bands, respectively. Therefore, these promising experimental results ensure that our suggested CSUCB/TS-HBS approaches, especially CSTS-HBS, are the most feasible HBS methods for multi-band wireless communication networks.

6. Conclusions and Outlook

As heterogeneous band wireless communication systems are being introduced, we examined the significant problem of efficient, realistic online channel allocation in this paper. Furthermore, we emphasize the necessity to design an online HBS technique. Hence, the HBS problem is restructured as a stochastic cost subsidy MAB, where each band selection consumes energy from the player/Tx. This paper investigated a realistic solution to the optimal band decision problem in hybrid (RF/VLC) systems by introducing two cost subsidy MAB algorithms, i.e., CSUCB, and CSTS-HBS, which outperform the primary UCB and TS implementations, respectively. Especially, CSTS-HBS confirmed not only the near-optimal performance but also a faster convergence. The results of our extensive simulation highlight the near-optimal performance of the advised CSTS-HBS algorithm. Future work includes a multi-Tx-Rx scenario extension and further real data investigations throughout hybrid network analysis.