Use of Logarithmic Rates in Multi-Armed Bandit-Based Transmission Rate Control Embracing Frame Aggregations in Wireless Networks

Abstract

Herein, we propose the use of the logarithmic values of data transmission rates for multi-armed bandit (MAB) algorithms that adjust the modulation and coding scheme (MCS) levels of data packets in carrier-sensing multiple access/collision avoidance (CSMA/CA) wireless networks. We argue that the utilities of the data transmission rates of the MCS levels may not be proportional to their nominal values and suggest using their logarithmic values instead of directly using their data transmission rates when MAB algorithms compute the expected throughputs of the MCS levels. To demonstrate the effectiveness of the proposal, we introduce two MAB algorithms that adopt the logarithmic rates of the transmission rates. The proposed MAB algorithms also support frame aggregations available in wireless network standards that aim for a high throughput. In addition, the proposed MAB algorithms use a sliding window over time to adapt to rapidly changing wireless channel environments. To evaluate the performance of the proposed MAB algorithms, we used the event-driven network simulator, ns-3. We evaluated their performance using various scenarios of stationary and non-stationary wireless network environments including multiple spatial streams and frame aggregations. The experiment results show that the proposed MAB algorithms outperform the MAB algorithms that do not adopt the logarithmic transmission rates in both the stationary and non-stationary scenarios.

1. Introduction

Multi-armed bandit (MAB) algorithms are the algorithms developed to solve MAB problems [1]. The name MAB problem resulted from an analogy of a scenario in which a player selects one slot machine (also known as a one-armed bandit) among many of them sequentially to maximize the total profit at the end of the game. MAB algorithms attempt to find the best strategies in such scenarios by adequately determining the trade-off between the exploitation of prior knowledge and the exploration of new opportunity. MAB algorithms have attracted significant interest from not only academia but also the industry because they can be applied to solve many practical problems in various areas, including commercial applications such as online advertisements and financing.

The epsilon-greedy algorithm ($ϵ$-greedy) [1], upper confidence bound (UCB) [2], exponential-weight algorithm for exploration and exploitation (Exp3) [3], Kullback–Leibler UCB (KL-UCB) [4], and Thompson sampling [5,6] are examples of MAB algorithms. Among the MAB algorithms, KL-UCB and Thompson sampling might be the most popular ones owing to their performance in maximizing rewards while minimizing regrets [7].

Today, the demand for wireless communications continues to increase partly because wireless communications have become essential to the daily lives of most people in this highly connected world. Furthermore, recent advances in Internet of Things (IoT) technologies [8] and their usage [9] have placed an even higher demand for wireless communications. To mitigate the ever-increasing demand for wireless communications, technologies such as edge/fog computing [10,11,12] are becoming important parts of communication infrastructures.

In addition to cellular wireless networks such as long-term evolution (LTE) [13] and 5G [14], carrier-sensing multiple access/collision avoidance (CSMA/CA) wireless communication standards (i.e., IEEE 802.11 [15]) play important roles by providing an alternative method of relatively easy and cheap wireless communication. However, because of the lack of the central control among mostly self-installed and self-managed CSMA/CA wireless networks, they often result in low resource utilization of network resources such as bandwidth and frequency channels, particularly when there are multiple users and access points in the vicinity. Furthermore, when CSMA/CA wireless networks operate in a purely distributed manner, such as the distributed coordination function (DCF) mode [15], inadequate selections of data transmission rates can significantly affect the performance of the wireless networks, particularly when the wireless channel status is dynamically changing or users are involved with movement.

To overcome the difficulties, many studies have introduced algorithms for adaptive transmission rate control such as Minstrel [16], collision-aware rate adaptation (CARA) [17], Q-learning-based rate control [18], and deep learning-based approaches [19]. These algorithms control the transmission rates of data frames by adaptively selecting an appropriate modulation and coding scheme (MCS) in response to the changes in the status of the wireless channel between a sender and receiver of the data frames. Furthermore, if multiple communication links are available between a sender and receiver through the use of multiple antennas and the multiple-input multiple-output (MIMO) mode, most transmission rate control algorithms also determine the number of spatial streams (e.g., 1 × 1 or 2 × 2) together with the MCS to be used for each transmission among possible combinations to achieve better performance in the given wireless environment. Hereinafter, we use the term MCS level to designate not only the MCS but also the number of communication links to be used for a transmission.

In recent years, researchers have introduced MAB algorithms to improve the performance of CSMA/CA wireless communication networks by considering the selection of the adequate MCS level as a special case of the MAB problem. In CSMA/CA wireless networks, the sender of a data packet waits for the arrival of an acknowledgment (ACK) frame for the data packet from the receiver. If the ACK frame is received in a given time interval, the data packet is considered to be delivered successfully (i.e., reward is 1). Otherwise, the sender assumes the packet is lost (i.e., reward is 0) and retries sending the packet until the maximum limit (e.g., seven times for the short-retry limit case) is reached. Graphical-optimal rate sampling (G-ORS) [20], modified Thompson sampling (MTS) [21], and constrained Thompson sampling (CoTS) [22] are some of the MAB algorithms proposed to adaptively control the data transmission rates by adjusting the MCS levels.

Most existing MAB algorithms for wireless communications use the expected throughputs of available MCS levels as a criterion to select the best MCS level for the next data transmission. As an example, if we consider a CSMA/CA wireless communication standard IEEE 802.11g [23], there are eight possible MCS levels that the sender can select for its data transmission to the receiver, i.e., MCS level $i\in$$\{0,\dots ,7\}$. The (physical) data transmission rate $r_i$ of each MCS level i is one of $\{6,9,12,18,24,36,48,54\}$ Mbps, respectively, when we consider only the extended-rate physical-orthogonal frequency-division multiplexing (ERP-OFDM) in the standard. If we assume that the sender adopts the MTS algorithm to control its transmission rate, the sender uses the transmission rate $r_i$ of each MCS level i and its success probability estimate $p_i\left(t\right)\in [0,1]$, which is obtained from the history of the transmission experiences up to time t, and selects the MCS level i that maximizes the expected throughput $r_i·p_i\left(t\right)$ as follows:

$$i=\underset{i}{argmax}(r_i·p_i\left(t\right)).$$

(1)

However, we notice that there is strong preference to high MCS levels because the data transmission rates of high MCS levels are significantly higher than those of the low MCS levels if the estimated success probabilities are similar. Although packets can be transmitted faster using higher MCS levels, the strong preference to use high MCS levels might result in a low utilization of network resources because high MCS levels require a high signal-to-interference and noise ratio (SINR) at the receiver of the packets. Therefore, we argue that the utilities of the MCS levels may not be proportional to their nominal transmission rates. To mitigate the strong preference to the high MCS levels, as a remedy, we propose the use of the logarithmic (base e) values of the transmission rates when computing the expected throughputs as follows:

$$i=\underset{i}{argmax}(log\left(r_i\right)·p_i\left(t\right)).$$

(2)

Here, we assume that $r_i>1$.

To demonstrate the effectiveness of the proposal, we apply the logarithmic transmission rates to two types of well-known MAB algorithms for the transmission rate control; one is based on KL-UCB and the other on Thompson sampling. We used the event-driven network simulator ns-3 [24] to evaluate the proposed MAB algorithms in various wireless communication scenarios including stationary and non-stationary cases, and with different standards such as IEEE 802.11g and 802.11ax [25]. The experimental results show that the use of the logarithmic values of the transmission rates improves performance of both types of the MAB algorithms.

The remainder of this paper is organized as follows: In Section 2, we explain works related to the MAB algorithms for the transmission rate control in CSMA/CA wireless networks. In Section 3, we propose two MAB algorithms that adopt the logarithmic transmission rates and embrace frame aggregations. The algorithms also show a sliding window scheme in detail to adapt to dynamically changing wireless environments. In Section 4, we present experimental results with the IEEE 802.11g standard in both of the stationary and non-stationary scenarios. In Section 5, we evaluate the performance of the proposed algorithms with the IEEE 802.11ax standard where wireless nodes can use frame aggregations. Section 6 concludes this paper.

2. Related Works

In this section, we briefly explain some of the MAB algorithms for the transmission rate control in CSMA/CA wireless networks.

In [20], the authors introduced the G-ORS algorithm that uses the Kullback–Leibler (KL) divergence measure and graphical unimodality to select the best MCS level for the data transmission. Because each transmission of the data packet results in an outcome of success or failure, it is modeled as a Bernoulli trial, and the KL divergence d between the two Bernoulli distributions with means of p and q can be measured as follows:

$$d(p,q)=plog\frac{p}{q}+(1-p)log\frac{1-p}{1-q}.$$

(3)

If the graphical unimodality is not used, the algorithm becomes a variant of the KL-UCB algorithm.

In [21], the authors proposed the MTS algorithm to control the data transmission rates of wireless nodes based on Thompson sampling. Thompson sampling computes the success probability distribution of each arm i of a given MAB problem using a beta distribution of which the probability density function f is defined using parameters ${\alpha }_i,{\beta }_i>0$ as follows:

$$f(x;{\alpha }_i,{\beta }_i)=\frac{x^{{\alpha }_i-1}{(1-x)}^{{\beta }_i-1}}{B({\alpha }_i,{\beta }_i)},x\in [0,1].$$

(4)

Here, $B({\alpha }_i,{\beta }_i)$ is the beta function, which is defined as follows:

$$B(\alpha ,\beta )=\frac{\mathrm{\Gamma }\left(\alpha \right)\mathrm{\Gamma }\left(\beta \right)}{\mathrm{\Gamma }(\alpha +\beta )},$$

(5)

where $\mathrm{\Gamma }(·)$ is the gamma function. The two parameters ${\alpha }_i$ and ${\beta }_i$ are used to record the histories of the successes and failures of the transmissions from each MCS level i. Whenever a transmission with an MCS level i is successful, ${\alpha }_i$ is increased by one; otherwise, ${\beta }_i$ is increased by one.

In [22], the authors proposed the CoTS algorithm to improve the MTS algorithm by incorporating a graphical unimodality to reduce the size of the search space. However, finding an adequate graphical unimodality graph for wireless networks with many MCS levels (including multiple streams) in dynamically changing wireless networks is another challenge that needs to be addressed. In [26], the authors proposed to use the received signal strength indicator (RSSI) for a Thompson sampling-based MAB algorithm. It reduces the search space by considering only the data rates suitable for the measured RSSI range.

In [27], the authors proposed to consider only a compact set of optimal data rates [28] decided from the configuration of a given wireless network. In addition, it also uses a threshold-based approach to further reduce the search space by using the transmission success histories of data rates for a Thompson sampling-based MAB algorithm. In [29], the authors proposed a two-stage approach for a Thompson sampling-based MAB algorithm. In the first stage, a group of data rates that could include the optimal MCS is identified, and then, in the second stage, the optimal MCS is searched only in the reduced search space.

However, in this paper, we do not consider these search space reduction techniques to focus on the effect of the proposed use of the logarithmic transmission rates.

In contrast to most classical MAB problems, the main characteristic of wireless communication networks is that they are inherently non-stationary. The status of a wireless communication channel can change quickly, particularly when wireless nodes are moving and/or when multiple wireless nodes interfere each other. To solve non-stationary MAB problems, in [30], the authors proposed discounted-UCB (D-UCB), which discounts the amount of rewards and number of selection times of arms with a discount parameter $\gamma \in (0,1)$. In [31], the authors proposed sliding-window UCB (SW-UCB) that uses a sliding window $\tau$ and counts only the last $\tau$ events in addition to the discounting when it computes the empirical averages of the rewards of the arms.

However, in wireless communications, the time intervals between transmission events are different depending on the used MCS levels because their transmission durations are decided by the selected MCS levels. To overcome this problem, the authors of G-ORS proposed sliding-window G-ORS (SW-G-ORS) in [20] that adopts a sliding window over time. They also suggested approximation methods for the implementation of the sliding window over time to reduce its computational cost. In [32], the authors proposed to use RSSI and ACK/negative ACK (NACK) signals to detect the changes of the channel status. Then, the results were used for a D-UCB-like rate adaptation. In [33], the authors proposed an adaptive online Bayesian learning to detect changes in non-stationary communication channels. The rate adaptation is conducted with a Thompson sampling-based MAB algorithm.

Nonetheless, in our proposed MAB algorithms, we directly use the sliding window over time because it is simple and the time interval we use for the sliding window is not large (e.g., a few tenths of a second) considering the simulated network environments, as shown in Section 4 and Section 5. Thus, we explicitly include the usage of the sliding window in the proposed MAB algorithms as described in Section 3.

The contributions of this paper can be summarized as follows:

• We propose the use of log values of data transmission rates in MAB algorithms for the rate control in CSMA/CA-based wireless networks.
• We propose two MAB algorithms that embrace the use of the frame aggregations developed for high-throughput wireless standards (explained in Section 3).
• We evaluate the performance of the proposed MAB algorithms using the ns-3 network simulator for IEEE 802.11g and 802.11ax wireless networks with various scenarios.

3. Multi-Armed Bandit Algorithms Using Logarithmic Rates

In this section, we introduce two MAB algorithms that adopt the logarithmic values of the transmission rates when they compute the expected throughput of each MCS level. One algorithm is based on KL-UCB and the other on Thompson sampling. In contrast to the two MAB algorithms presented in [34], in this paper, we modify the two algorithms to accommodate frame aggregation schemes available in high-throughput wireless network standards such as IEEE 802.11n [35] and IEEE 802.11ax. If the frame aggregation is enabled, (multiple) aggregated packets can be sent as one aggregated MAC service data unit (A-MSDU) or aggregated MAC protocol data unit (A-MPDU) frame [15].

When A-MSDU is used, all the packets transmitted as one A-MSDU frame are considered to be successfully delivered if an ACK frame is received on time. Otherwise, all the aggregated packets in the A-MSDU frame are assumed to be lost. If A-MPDU is used, a block acknowledgment (BAck) frame is used to monitor the success and failure of each packet transmitted in one A-MPDU frame. When a BAck frame is received by the sender of the A-MPDU frame, the sender can identify successfully delivered packets and lost packets among the transmitted packets using a bitmap structure. Otherwise, all the transmitted packets in the A-MPDU frame are assumed to be lost, and the sender sends a control message called block acknowledgment request (BAckR) frame to the receiver.

Thus, to embrace the frame aggregation, we use the number of successfully delivered packets by one aggregated packet transmission as the amount of the reward if a BAck or ACK frame is received on time. When the packet aggregation is disabled or not available as in the IEEE 802.11g standard, the proposed MAB algorithms in this paper reduce to the previous ones because the reward of each transmission is one if the delivery is successful (i.e., an ACK frame is received on time).

3.1. KL-UCB Using Logarithmic Rates

We first introduce an MAB algorithm for the transmission rate control in CSMA/CA wireless networks using the logarithmic values of the transmission rates based on KL-UCB. We call it KL-UCB-logR. We assume that there are K MCS levels, of which one can be selected for each (physical) transmission. Each transmission can be for aggregated (multiple) packets if the frame aggregation is enabled; otherwise, it is for one packet. Algorithm 1 shows the pseudocode of the algorithm.

At the initial stage of Algorithm 1, an empty list $R_i$ is assigned for each MCS level i to record the rewards obtained when the MCS level is used for transmissions. After the initial K steps, for each transmission at time t, the algorithm computes the total number of transmitted packets N that occurred during the sliding window time $\tau$. The time interval defined by the sliding window $\tau$ is between $t-\tau$ and t, i.e., $[t-\tau ,t)$ if time t is greater than $\tau$; otherwise it is [0, t). Note that if $\tau$ is set as $\tau \ge t$, the sliding window is disabled and the time interval of interest is always [0, t).

In addition, the algorithm computes the sum of the rewards ($S_i$) and number of packets transmitted ($N_i$) with each of the MCS level i during $\tau$. Subsequently, they are used to estimate the success probability $p_i\left(t\right)$ of each MCS level, as shown in line 13. The configuration parameter c in line 13 is set to one by default. The function $d(·,·)$ computes the Bernoulli KL divergence as in Equation (3). Line 15 shows that the expected throughput of each i is computed using the logarithmic value of its transmission rate. Then, an MCS level i with the maximum expected throughput is selected for transmission. Ties are broken arbitrarily.

After each transmission, if the data sender receives an ACK or BAck frame before the timeout period expires, the transmission is considered a success and the number of successfully delivered packets in the transmission is appended to $R_i$; otherwise, 0 is appended. Finally, the time at which the outcome is observed is recorded using a separate list (not shown in the pseudocode) to keep track of all the events during the sliding window time.

If the nominal transmission rates are used directly without the logarithm for computing the expected throughput in line 15, we call the algorithm KL-UCB-R for performance comparison. KL-UCB-R may appear similar to the KL-R-UCB algorithm in [36]. However, KL-UCB-R uses a slightly different formula to compute $p_i\left(t\right)$ in line 13, and KL-UCB-R explicitly describes how the sliding window is used. In addition, KL-UCB-R accommodates the use of the frame aggregation.

Algorithm 1 KL-UCB-logR algorithm

1: for each MCS level $i\in \{0,\dots ,K-1\}$ do 2:     $R_i\leftarrow \left[\right]$                         ▹ Prepare empty lists 3: end for 4: $step\leftarrow 0$ 5: for each transmission at time t do 6:     if $step\le K-1$ then 7:         use $i=step$ for transmission and wait for the outcome. 8:     else 9:         N = number of all the transmitted packets during $\left[\max \right(0,t-\tau ),t)$. 10:         for each i do 11:            $S_i$ = sum($R_i$ during $\left[\max \right(0,t-\tau ),t)$). 12:            $N_i$ = number of the transmitted packets using i during $\left[\max \right(0,t-\tau ),t)$. 13:            $p_i\left(t\right)$ = max{$q\in$ [0,1]: $d(S_i/N_i,q)\le (log\left(N\right)+c·log(log\left(N\right)))/N_i$}. 14:         end for 15:         use $i={argmax}_i(log\left(r_i\right)·p_i\left(t\right))$ for transmission and wait for the outcome. 16:   end if 17:   if success then 18:         append the number of successfully delivered packets to $R_i$. 19:   else 20:         append 0 to $R_i$. 21:   end if 22:     record the time at which the outcome is observed. 23:     $step\leftarrow step+1$ 24: end for

3.2. Thompson Sampling Using Logarithmic Rates

In this subsection, we introduce another MAB algorithm for the transmission rate control in CSMA/CA wireless networks using the logarithmic values of the transmission rates based on Thompson sampling. We call it Thompson sampling with logarithmic rates (TS-logR). We assume that there are K MCS levels. Each transmission can be for aggregated (multiple) packets if the frame aggregation is enabled. Algorithm 2 shows the pseudocode of TS-logR.

In line 2 of Algorithm 2, ${\alpha }_i$ and ${\beta }_i$ are lists used to monitor the success and failure events of each MCS level i, respectively. They are initialized as empty lists. In line 6, the success probability estimate $p_i\left(t\right)$ of each MCS level i is sampled from the beta distribution shown in Equation (4) using the histories of the success and failure events during the sliding window time $\tau$. The two arguments used by the beta distribution, sum(${\alpha }_i$ during $\left[\max \right(0,t-\tau ),t))$ and sum(${\beta }_i$ during $\left[\max \right(0,t-\tau ),t))$, indicate that we only consider the numbers of success and failure events, respectively, that occur between $t-\tau$ and t, i.e., $[t-\tau ,t)$, if time t is greater than $\tau$; otherwise, during [0, t). Again, if $\tau$ is set as $\tau \ge t$, the sliding window is disabled, i.e., all the events in [0, t) are considered.

Line 8 shows the computation of the expected throughput of each MCS level using the log values of $r_i$ and $p_i\left(t\right)$. An MCS level i with the maximum expected throughput is selected for transmission. Ties are broken arbitrarily. Lines 10 and 12 track the success and failure events, respectively. If the data sender receives an ACK or BAck frame from the receiver before the timeout period expires, the event is considered a success and the number of successfully delivered packets in the transmission is appended to ${\alpha }_i$. Otherwise, the transmission is considered a failure, and the number of the transmitted packets in the transmission is appended to ${\beta }_i$. Finally, the time of each event is recorded on a separate list (not shown in the pseudocode) to be used for the sliding window.

If the transmission rate $r_i$ was used without the logarithm in line 8, we call the algorithm TS-R for performance comparison. The difference between TS-R and MTS is that TS-R accommodates the use of the frame aggregation. In addition, TS-R explicitly describes the use of the sliding window over time to adapt to wireless channel environments that can be rapidly changing.

Algorithm 2 TS-logR algorithm

1: for each MCS level $i\in \{0,\dots ,K-1\}$ do 2:     ${\alpha }_i\leftarrow \left[\right],{\beta }_i\leftarrow \left[\right]$.                         ▹ Prepare empty lists 3: end for 4: for each transmission at time t do 5:     for each i do 6:         sample $p_i\left(t\right)$ from beta(sum(${\alpha }_i$ during $\left[\max \right(0,t-\tau ),t)$)+1, sum(${\beta }_i$ during $\left[\max \right(0,t-\tau ),t)$)+1). 7:     end for 8:     use $i={argmax}_i(log\left(r_i\right)·p_i\left(t\right))$ for transmission and wait for the outcome. 9:     if success then 10:         append the number of successfully delivered packets to ${\alpha }_i$. 11:   else 12:         append the number of transmitted packets to ${\beta }_i$. 13:   end if 14:   record the time at which the outcome is observed. 15: end for

To evaluate the performance of the two proposed MAB algorithms in non-aggregated and aggregated packet transmission situations, we used IEEE 802.11g and IEEE 802.11ax standards in experiments, respectively. For the experiments, we used the event-driven network simulator ns-3 (version 3.35). Moreover, we used ns3-gym [37] (version 1.0.1), which provides a simulation environment for reinforcement learning in wireless communication networks through cooperation with the ns-3.

4. Experiments without Frame Aggregations

In this section, we show the experimental results of the proposed MAB algorithms when used in IEEE 802.11g wireless networks. We used the IEEE 802.11g standard to evaluate the proposed MAB algorithms when packets were transmitted one at a time because the standard does not support the frame aggregation. In addition, IEEE 802.11g supports only a single wireless link, i.e., the single-input single-output (SISO) mode, between a sender and receiver. According the standard, there are eight possible MCS levels, i.e., K = 8. The data transmission rate can be one of {6, 9, 12, 18, 24, 36, 48, 54} Mbps. Here, we use only the ERP-OFDM scheme in the 802.11g standard for simplicity.

The carrier frequency is set to 2.412 GHz and the channel bandwidth is set to 20 MHz. For the path loss of the radio signal, we used the urban microcell non-line-of-sight model in [38]. Thus, with the carrier frequency and a given distance d (in m) between a sender and receiver, the path loss (in dB) of the signal was computed as follows:

$$40.198+38{log}_{10}\left(d\right).$$

(6)

Here, we assumed that $d>1$.

In all the experiments in this study, all wireless nodes operated in the DCF mode. The request-to-send/clear-to-send (RTS/CTS) scheme was not used for simplicity. As an application in all the experiments, we used a fixed constant bit-rate (CBR) traffic from a sender to its receiver during entire simulation time. The data packet size was set to 1500 bytes including the Internet protocol (IP) and user diagram protocol (UDP) header. All wireless nodes used 100 mW (i.e., 20 dBm) to transmit packets. The noise figure was set to 7 dB.

Table 1. Simulation parameters used for experiments.

Parameter	Value
Carrier frequency	2.412 GHz
Channel bandwidth	20 MHz
Transmit power	100 mW
Receiver sensitivity	−101 dBm
Slot time	20 us
Noise floor	∼−94 dBm
Clear channel threshold	−82 dBm
Data packet size (including IP/UDP header)	1500 bytes
Device queue size	500 packets
Guard interval	800 ns
Preamble detection model threshold	−30 dB
Preamble detection model minimum RSSI	−99 dBm

provides more details about the simulation parameters for the experiments in this study. All other parameters for simulations remained at their default values in the ns-3 version unless stated otherwise.

4.1. Packet Success Rate

For the packet error probabilities of arriving packets, we used the National Institute of Standards and Technology (NIST) error rate model in the ns-3. For the experiments, we used two wireless nodes that operated with the IEEE 802.11g standard. Initially, the distance between the sender and receiver was set to 5 m. During the simulation time of 13 s, one node (sender) moved away from the other node (receiver) at a speed of 6 $\mathrm{m}$/$\mathrm{s}$ while it sent CBR traffic of 54 Mbps to the receiver. Thus, the distance between the two nodes changed from 5 to 83 m. The sender used one (fixed) MCS level among the eight MCS levels for the data packet transmissions in each simulation. As a result of the movement, we can expect the SINRs of the received packets to deteriorate while the distance between the sender and receiver increases.

In Figure 1, we show the experimental results that show the packet success rates (i.e., 1—packet error rates) according to the SINRs of the received packets and distances between the sender and receiver. The experimental results in Figure 1a show that the packet success rates decreased when the SINRs of the arrived packets at the receiver decreased regardless of the MCS levels used by the sender. The experimental results in Figure 1b show that the packet success rates decreased when the distance between the sender and receiver increased, regardless of the used MCS levels. The experimental results clearly showed that packets may be more successfully delivered to the receiver when the sender adopts low MCS levels until the distance between the two nodes becomes greater than approximately 73 m.

4.2. Stationary Scenario

In this subsection, we evaluate the performance of the proposed MAB algorithms in a stationary scenario. For the experiments in this scenario, we used two wireless nodes at fixed positions during the simulation time of 2.5 s. Both nodes use the IEEE 802.11g standard. During the simulation time, the sender transmitted CBR traffic of 54 Mbps to the receiver. The distance between the two nodes was fixed at 60 m. Because the two nodes were not moving and no other interfering nodes existed, we disabled the sliding window in the proposed MAB algorithms by setting $\tau =2.5$.

Because of the large distance of 60 m between the two nodes, as shown in Figure 1b in Section 4.1, the lowest MCS level (i.e., MCS 0) had higher packet success rates than the other MCS levels. In particular, the packet success rates of the MCS levels higher than 2 were almost zero with the distance. Thus, MCS 0 is the best MCS level in this scenario that can be selected by the sender. Figure 2 shows the experimental results when four different MAB algorithms were adopted by the sender.

Figure 2a shows the changes in the application throughputs (0.1 s average) achieved by the receiver during a simulation time of 2.5 s. These results indicate that the application throughput can be approximately 5 Mbps in its maximum. The results show that TS-logR and KL-UCB-logR reached the maximum throughput faster than TS-R and KL-UCB-R, respectively.

Figure 2b shows the changes in the average MCS levels (0.1 s average) selected by the four MAB algorithms. The results clearly show that the average of the MCS levels selected by TS-logR and KL-UCB-logR converged to MCS 0 faster than TS-R and KL-UCB-R, respectively. In addition, the results in Figure 2a,b show that TS-logR and KL-UCB-logR achieved a similar performance. However, TS-R had a better performance than KL-UCB-R in this scenario.

4.3. Non-Stationary Scenario

In this subsection, we evaluate the performance of the proposed MAB algorithms in a non-stationary scenario. In this scenario, we let one node (the sender) move away from or toward the other node (the receiver) at a speed of 6 $\mathrm{m}$/$\mathrm{s}$ during a simulation time of 10 s. The receiver was located at a fixed position during the simulation time. The initial distance between the sender and the receiver was set to 5 m. As a result, when the sender moved away from the receiver, the distance between the two nodes changed from 5 to 65 m. In the opposite direction, the distance began at 65 m and ended at 5 m. During the simulation time, the sender transmitted CBR traffic of 54 Mbps to the receiver. The sliding window time was set to 100 ms considering the simulated network environment.

Figure 3 shows the experimental results when the sender adopted different MAB algorithms. Figure 3a shows the changes in the application throughputs (0.1 s average) achieved by the receiver when the sender moved away from the receiver. It shows that TS-logR had a better performance than TS-R, particularly when the distance between the sender and receiver was large. In addition, KL-UCB-logR had a better performance than KL-UCB-R, which exhibited low performance from approximately the middle of the simulation time. The experimental results in Figure 3a imply that the proposed MAB algorithms select low MCS levels more accurately than their counterparts when the SINR is low.

Figure 3b shows the changes in the application throughputs (0.1 s average) achieved by the receiver when the sender moved toward the receiver. The experimental results in Figure 3b clearly show that TS-logR and KL-UCB-logR achieved better performance than TS-R and KL-UCB-R, respectively, particularly when the distance between the sender and receiver was large. The experimental results in Figure 3a,b indicate that by adopting the logarithmic values of the transmission rates, the proposed MAB algorithms performed better than their counterparts in the rapidly changing channel environments.

In Figure 4, we show the final application throughputs achieved by the receiver when the sender adopted different transmission rate control algorithms during the 10 s simulation time. For the experiments, we repeated the simulations ten times with different seed values for randomness. The results in Figure 4a,b show their averages and standard deviations. The figure also shows the simulation results from two non-MAB-based transmission rate control algorithms in the ns-3 simulator called Minstrel and Ideal for performance comparison purpose.

Minstrel controls data transmission rates by adjusting the MCS levels of data packets using the results from periodic probing data packets with different MCS levels. Minstrel is one of the most well-known transmission rate control algorithms because it is widely adopted in practical operating systems such as Linux [39]. The Ideal transmission rate control algorithm in the ns-3 simulator assumes that the SINRs of all the arrived packets at the receivers are known to the sender of the packets. Thus, using the SINR information, the Ideal transmission rate control algorithm can achieve almost ideal performance in a given wireless environment.

The experimental results in Figure 4a,b clearly show that, by adopting the logarithmic values of the transmission rates, the two proposed MAB algorithms performed better than their counterparts in the rapidly changing channel environments. The results also show that TS-logR performed better than KL-UCB-logR and Minstrel in both directions. Additionally, TS-logR achieved almost as high a performance as the Ideal transmission rate control algorithm in both directions. The overall performance improvements from the results in Figure 4 are summarized in

Table 2. Performance improvements in IEEE 802.11g networks.

	Moving Away	Moving Toward	Average
KL-UCB	+30.4%	+36.2%	+36.3%
Thompson sampling	+10.9%	+6.6%	+8.8%

.

5. Experiments with Frame Aggregations

In this section, we evaluate the performance of the proposed MAB algorithms when the frame aggregation and multiple spatial streams are available. For this, we use the IEEE 802.11ax standard that is recently developed to achieve high efficiency in wireless networks (it is also known as the high-efficiency (HE) standard). The IEEE 802.11ax standard can support the frame aggregation using A-MPDU and A-MSDU. However, we considered only A-MPDU in the experiments because A-MPDU is known for performing better than A-MSDU, particularly in high packet error rate situations [40]. This is primarily because A-MPDU can selectively acknowledge successfully delivered packets among the aggregated packets transmitted as a single frame. For the experiments described in this section, we set the A-MPDU size to 65,535 bytes, which is the maximum size in the ns-3 version we used for the experiments.

When the A-MPDU frame aggregation is enabled, in addition to the BAck and BAckR frames, several control frames are also used. They are add block acknowledgment (AddBA) request/response frames and delete block acknowledgment (DelBA) frames [15]. These control messages are exchanged between a sender and receiver to manage aggregated packet transmission sessions. Among these control messages, we control the MCS levels of BAckR frames as if they were data packets and their transmission results are used in the proposed MAB algorithms. This is because the BAckR frames play an important role in the frame aggregation particularly when the packet error rates are high. For this, we modified the ns-3 implementation to make the sender adjust the MCS levels of BAckR frames in the same way as it does for the data packets when the sender adopts the proposed MAB algorithms.

In addition, the IEEE 802.11ax standard supports multiple spatial streams (i.e., multiple links) between a sender and receiver. Therefore, wireless nodes can use the MIMO technology if they are equipped with multiple antennas in the IEEE 802.11ax standard. According to the standard, there are 12 possible MCSs for each combination of the bandwidths and guard intervals. Although the IEEE 802.11ax standard supports up to eight spatial streams, for simplicity, we assume that wireless nodes can be equipped with maximum two antennas each, such that they can use the 2 × 2 MIMO mode in addition to the 1 × 1 SISO mode. By fixing the bandwidth to 20 MHz and the guard interval to 800 ns, senders can choose one MCS and the number of spatial streams to be used for each frame transmission. Thus, in our experiment setting, K = 24 if 2 × 2 MIMO mode is available and the set of all the corresponding data transmission rates is {8.6, 17.2, 25.8, 34.4, 51.6, 68.8, 77.4, 85.0, 103.2, 114.7, 129.0, 143.4, 17.2, 34.4, 51.6, 68.8, 103.2, 137.6, 154.9, 172.1, 206.5, 229.4, 258.1, 286.8} Mbps. However, if each wireless node is equipped with only one antenna, 1 × 1 SISO mode is used. In this case, K = 12 and the set of all the corresponding data transmission rates is {8.6, 17.2, 25.8, 34.4, 51.6, 68.8, 77.4, 85.0, 103.2, 114.7, 129.0, 143.4} Mbps.

Except for the IEEE 802.11ax standard, we used the same wireless network environments as we used for the experiments with the IEEE 802.11g standard described in Section 4. Specifically, we used the same path loss model (i.e., Equation (6)) with the same parameters. We used the same packet error probability model (i.e., NIST error rate model) for the packet error rates. In addition, all wireless nodes operated in the DCF mode. Thus, Table 1 also shows the simulation parameters used for the experiments with the IEEE 802.11ax standard in this section.

5.1. Packet Success Rate

To demonstrate the effect of the IEEE 802.11ax standard, we let a wireless node (sender) move away from the other node (receiver). The sender moved at a speed of 6 $\mathrm{m}$/$\mathrm{s}$ during a simulation time of 12 s while sending CBR traffic of 150 Mbps to the receiver. The receiver remained at a fixed location. The initial distance between the sender and receiver was set to 5 m. Thus, the distance between the two nodes changed from 5 to 77 m during the simulation time. Both wireless nodes are quipped with one antenna each. Thus, 1 × 1 SISO mode was used by the sender. The sliding window time was set to 300 ms considering the simulated network environment.

Figure 5a shows the experimental results of the packet success rates according to the SINRs of the received packets when the sender adopted one of the 12 MCS levels during the simulations. The experimental results in Figure 5a clearly show that the packet success rates of the packets decreased when the SINRs of received packets deteriorated regardless of the used MCS levels. Figure 5b shows the changes in the packet success rates according to the distances between the sender and receiver. The results in Figure 5b also show that the packet success rates decreased when the distance between the sender and receiver increased independently of the used MCS levels.

The experiment results indicated that, by using low MCS levels, the packets are more likely to be successfully delivered to the receiver than when using high MCS levels when two nodes are far from each other. In addition, by comparing the experiment results in Figure 1a in Section 4.1 with those in Figure 5a in Section 5.1, we observe that the sender with the IEEE 802.11ax standard could utilize higher SINRs. This is because of the improved performance by the IEEE 802.11ax standard.

5.2. Stationary Scenario

In this subsection, we evaluate the performance of the proposed MAB algorithm in a stationary scenario in which wireless nodes use the IEEE 802.11ax standard. We used two wireless nodes located in fixed positions during a simulation time of 2.5 s. The two nodes are are quipped with one antenna each. The distance between the two nodes was set to 70 m. During the simulation time, one node (sender) sends CBR traffic of 150 Mbps to the other node (receiver). We disabled the sliding window in the proposed MAB algorithms by setting $\tau$ = 2.5 because the two nodes were not moving, and no interfering nodes existed. Because of the large distance between the two nodes, the experimental results of the packet success rates shown in Figure 5b indicated that only the lowest MCS level (i.e., MCS 0) had packet success rates higher than zero. All other MCS levels exhibited zero packet success rates with the given distance. Thus, MCS 0 is the only MCS level that can result in successful reception of packets at the receiver in this scenario.

Figure 6a shows the changes in the application throughputs (0.1 s average) achieved by the receiver when each of the four different MAB algorithms was adopted by the sender. The experimental results in Figure 6a show that the receiver achieved approximately 5.8 Mbps in its maximum. The experimental results clearly show that TS-logR and KL-UCB-logR reached the maximum throughput faster than TS-R and KL-UCB-R, respectively. Figure 6b shows the changes in the average MCS levels (0.1 s average) used by the four MAB algorithms. The MCS level used by TS-logR and KL-UCB-logR converged to MCS 0 faster than TS-R and KL-UCB-R, respectively.

However, by comparing the results in Figure 6b with Figure 2b in Section 4.2, we observe that it took a longer time for the MAB algorithms to converge with the IEEE 802.11ax standard than with the IEEE 802.11g standard, regardless of the MAB algorithms. This was primarily because of the increased size of the search space, i.e., K was 12 instead 8.

5.3. Non-Stationary Scenario

In this subsection, we evaluate the performance of the proposed MAB algorithms in a non-stationary scenario while all wireless nodes operated with the IEEE 802.11ax standard. For this scenario, we let a sender move away from or toward its receiver at a speed of 6 $\mathrm{m}$/$\mathrm{s}$ during the simulation time of 11 s. The initial distance between the sender and receiver was set to 5 m. Thus, when the sender moved away from the fixed receiver, the distance between the two nodes changed from 5 to 71 m. In the opposite direction, the distance started at 71 m and ended at 5 m. The sliding window time $\tau$ was set to 300 ms considering the simulated network environment.

In contrast to the previous experiments, we let all the wireless nodes be equipped with two antennas each. Thus, both the 2 × 2 MIMO and 1 × 1 SISO modes could be used by the sender and K was 24 in the experiments. Because of the increased speed of the high MCS levels by the 2 × 2 MIMO mode, the sender transmitted CBR traffic of 300 Mbps to the receiver during the simulation time of 11 s.

Figure 7 shows the changes in the application throughputs (0.1 s average) achieved by the receiver from different MAB algorithms when the sender moved away or toward the receiver. Figure 7a shows the experimental results when the sender moved away from the receiver during the simulation time of 11 s. In Figure 7a, for the better visualization of the results during the time interval in which we are more interested, we also show the changes in the application throughputs during 9 s to 11 s in a separate window. The results in the figure show that TS-logR and KL-UCB-R achieved better performance than their counterparts, particularly when the distance became large.

Figure 7b shows the experimental results when the sender moved toward the receiver during the simulation time of 11 s. In Figure 7b, we also show the changes in the application throughputs during 0 s to 2 s in a separate window. Figure 7b clearly shows that TS-logR and KL-UCB-logR achieved better performance than TS-R and KL-UCB-R, respectively, particularly when the distance between the sender and receiver was large.

In Figure 8, we show the final application throughputs achieved by the receiver when the sender adopted different transmission rate control algorithms during the 11 s simulation time. For the experiments, we repeated the simulations ten times with different seed values for randomness. Figure 8 shows their averages and standard deviations. The figure also shows the simulation results of the Ideal and Minstrel-HT rate control algorithm in the ns-3 simulator for the performance comparison purpose. Minstrel-HT is an upgraded version of the Minstrel for high-throughput (HT) wireless network standards such as IEEE 802.11n.

The experimental results in Figure 8a show that TS-logR and KL-UCB-logR outperformed their counterparts when the sender moved away from the receiver. The experimental results in Figure 8b also show that TS-logR and KL-UCB-logR outperformed their counterparts when the sender moved toward to the receiver. In addition, TS-logR achieved a higher throughput than KL-UCB-logR and Minstrel-HT in both directions.

Overall, the experimental results in Figure 8 show that the proposed MAB algorithms performed better than their counterparts in both directions in this scenario. The performance improvements in both directions from the results in Figure 8 and their averages are summarized in

Table 3. Performance improvements in 2 × 2 IEEE 802.11ax networks.

	Moving Away	Moving Toward	Average
KL-UCB	+30.9%	+25.2%	+28.1%
Thompson sampling	+15.1%	+11.6%	+13.4%

.

6. Conclusions

In this study, we aimed to control the transmission rates of data packets in CSMA/CA wireless networks more accurately using MAB algorithms. We proposed the use of the logarithmic values of the transmission rates of MCS levels instead their nominal values in the computation of their expected throughput estimations. To evaluate the effectiveness of the proposed method, we introduced two MAB algorithms which adopt the logarithmic rates and a sliding window over time. The proposed MAB algorithms were also designed to support frame aggregation schemes available in wireless network standards for high throughput such as IEEE 802.11ax.

The experimental results using the event-driven network simulator, ns-3, demonstrated that the proposed MAB algorithms outperformed their counterparts that do not adopt the logarithmic rates in both the stationary and non-stationary scenarios. Additionally, the proposed MAB algorithms achieved better performance than their counterparts when frame aggregation was enabled. Furthermore, the proposed MAB algorithms achieved comparable performance to non-MAB-based transmission rate control algorithms such as Minstrel in rapidly changing wireless channel environments.

We expect that the use of the logarithmic rates of the transmission rates can be adopted to improve the performance of MAB algorithms that involve other types of wireless communications such as cellular networks. However, by comparing the experimental results of the proposed algorithms with those of the Ideal algorithm in the IEEE 802.11ax network, we can see that there is room for improvement. As a future work, we consider the adoption or development of techniques to reduce the size of the search space to improve the performance particularly when the search space is large. In addition, we would like to investigate whether the adoption of sophisticated change detection techniques could result in better performance than using the sliding window.