Multi-Objective UAV Positioning Mechanism for Sustainable Wireless Connectivity in Environments with Forbidden Flying Zones

Abstract

A communication system based on unmanned aerial vehicles (UAVs) is a viable alternative for meeting the coverage and capacity needs of future wireless networks. However, because of the limitations of UAV-enabled communications in terms of coverage, energy consumption, and flying laws, the number of studies focused on the sustainability element of UAV-assisted networking in the literature was limited thus far. We present a solution to this problem in this study; specifically, we design a Q-learning-based UAV placement strategy for long-term wireless connectivity while taking into account major constraints such as altitude regulations, nonflight zones, and transmit power. The goal is to determine the best location for the UAV base station (BS) while reducing energy consumption and increasing the number of users covered. Furthermore, a weighting method is devised, allowing energy usage and the number of users served to be prioritized based on network/battery circumstances. The suggested Q-learning-based solution is contrasted to the standard k-means clustering method, in which the UAV BS is positioned at the centroid location with the shortest cumulative distance between it and the users. The results demonstrate that the proposed solution outperforms the baseline k-means clustering-based method in terms of the number of users covered while achieving the desired minimization of the energy consumption.

1. Introduction

It was a truism that the number of subscriptions to mobile communication networks and the amount of data consumed per user were increasing over the years, and that the newer generations were becoming more dominant than the legacy after a few years of their first deployment [1]. In the fifth generation of mobile communications (5G), on the other hand, such increase is more highlighted due to the fact that there are more demanding emerging applications, including tactile internet, 4K video streaming, online gaming, etc., and the concept of Internet of Things (IoT) was seriously proliferating and pervading our daily lives with a large inclusion in various domains, such as healthcare, manufacturing, etc. [2]. There were various concepts and technologies proposed in the literature to address the aforementioned capacity issues. The use of millimeter-wave (mmWave) frequencies, massive multi-input multioutput (mMIMO), and network densification are some of the most popular and practical ones among others [3,4,5,6]. Each of these technologies has a different set of advantages and disadvantages, however, they mainly target capacity enhancement in mobile communication networks. With mmWave communications, for example, an additional spectrum added to 5G networks—it was already included in 5G New Radio (NR) as frequency range-2 [7]—, and thus the capacity is increased with this additional bandwidth. The use of higher carrier frequencies also enable smaller antenna sizes, which subsequently enables mMIMO antenna arrays, enhancing the capacity further [3,5]. Network densification, on the other hand, offers deployments of smaller base stations (BSs) with comparatively less antenna transmit power to reuse the frequency band, leading to a great deal of capacity enhancement [3].

Even though all these solutions are quite beneficial in enhancing the capacity of mobile communication networks, there is still room for improvements, since the spatio-temporal changes in wireless networks pose another type of challenges. More specifically, unusual circumstances, including expositions, sport competitions, and musical concerts, where much more people than normal gather together and significantly increase the demand for wireless communications are required to be tackled in a more sophisticated and intelligent manner. This is mainly because such kind of events do not happen pretty often (only few times a year); hence, it is not a good idea to design the network by taking them into consideration. In this regard, BS mounted on UAVs (which will be referred as to UAV BS hereafter) was a promising solution to meet the strict user requirements of coverage, capacity, and quality of service (QoS). With the dawn of 5G network and related technologies, the user requirements are getting diverse as the users are from a diverse groups including conventional user equipments (UEs), IoT devices, machines, vehicles, etc. The UAV-assisted communication system is a solid use-case for the next generation of mobile communications given that UAVs are flexible, easy-to-deploy, and cost-efficient [8]. This provides a boost to the terrestrial cellular infrastructure as UAVs can be deployed to provide the extra coverage, or increase the capacity in the given area.

The primary research challenges regarding the UAV-assisted communication systems are as follows: (i) determining optimal positions of the UAVs; (ii) finding optimal UAV trajectory; and (iii) meeting the regional regulations for UAVs. Energy efficient networking with UAV BSs is at the core of the discussion, since the UAVs are battery operated and have limited energy capacity. This is one of the most important limitations with UAV-assisted communications, requiring a proper management; otherwise, the concept can get infeasible if the the flight time of UAVs cannot be sufficiently prolonged (The sufficiency mentioned here should be discussed according to the scenario; i.e., the conditions of the networks and the requirements of mobile network operators as well as users, and hence, it is quite hard to put a formal and strict definition and/or a numerical value for it. However, the main idea is to maximize the flight time of UAVs as much as possible). Furthermore, UAV-based communication systems represent even a more complex case, because the total energy consumption depends on both the communication to the ground users and flying on a predefined trajectory or simply hovering over a fixed point. Besides, to cover as much as users as possible while maintaining the energy efficiency is an important aspect of UAV-assisted networking, since the the main objective is to enhance the capacity of the network to serve more users in temporally-dense networks. Put it another way, energy efficiency is required to realize the main objective, which is capacity enhancement in this case, an thus the idea is to keep the UAV BSs more in the air to serve more users in total. Therefore, to capture this phenomena in our work, we consider both the energy consumption and the total number of users covered as objective functions of our novel problem formulation.

No-fly-zones (NFZs), which are restricted or prohibited areas where UAV is not allowed to fly—such as military bases—are considered as a practical constraint in the deployment phase of UAV BSs. This is because the optimal UAV trajectory and positioning are affected by NFZs given that UAVs need to avoid those places even if they are optimal positions. In other words, UAV BSs are supposed to be positioned considering the NFZ constraints, bringing additional challenge to the optimization process. In addition, as aforementioned, there are also regulations on the minimum and maximum altitude of the UAVs; such that the UAVs are supposed to be within the allowed range in terms of their altitude (The altitude regulations vary for different countries and regions). In this regard, to seize this idea of NFZs and altitude regulations in this work, we consider them as constraints in the problem formulation.

Machine learning has a considerable place in optimizing wireless communications networks [9,10,11,12,13], due to its strong capabilities in terms of convergence, dynamism, and agility. Moreover, it is envisioned that it will play rather a more critical role in the upcoming generations of mobile communication networks, such as sixth generation of mobile communication (6G) [14,15]. Therefore, we employed reinforcement learning (RL), a subset of machine learning, in this study to provide more intelligent, dynamic, and effective solution.

1.1. Related Work

The literature on UAV-assisted communication systems is thoroughly evaluated in this subsection. In recent years, numerous number of studies were done in the field of UAV-assisted wireless networking [16,17,18,19,20,21,22,23,24,25], but only a few studies have looked at the flying regulations [26,27]. In [8], the authors presented a survey on the most recent research possibilities and problems in the field of UAV aided wireless networks. The key difficulties in UAV-assisted networking are investigated, including 3D deployment, performance analysis, channel modeling, and energy efficiency. These topics are investigated due to huge demands in terms of network performances for 5G services, which results, among other factors, in a considerable increase in energy consumption. Consequently, energy saving is a fundamental design requirement for 5G networks. Although, in the case of ground base stations, solutions as low-cost, low-complexity cell sleep scheduling algorithms may help to minimize the energy consumption while guaranteeing the service requirements [28,29], for UAV-assisted networks it is still an open-issue. For the coexistence of UAVs and under-laid device-to-device (D2D) communication networks, a tractable analytical framework is proposed in [22]. The authors showed that flying a UAV at the ideal altitude can result in the highest system sum-rate and coverage probability. Furthermore, an optimal trajectory design can reduce transmit power; however, networking under the UAV altitude regulations did not receive enough attention in the literature. The lowest and maximum authorized altitudes of flying UAVs vary by country; for example, European laws for flying UAVs establish the limits of minimum and maximum allowed altitudes, which may fluctuate in different regions of the world, and in [26], the authors looked at the status of UAV-related regulations.

In the previous few years, numerous surveys and tutorials were released. The findings reveal that creating air route networks is a scientifically sound and efficient way to standardize and improve the efficiency of low-altitude UAV operations [27]. The most significant approach for UAV regulation in urban regions, in terms of safety and efficiency, is to enhance research that heavily relies on urban remote sensing and Geographic Information System (GIS) technology, as well as application demonstrations of low-altitude public air route networks [26]. In [27], the authors discussed the standardization initiatives for UAV-assisted UEs, UAV-assisted BSs, UAV communication prototypes, and UAV-assisted cellular communications cyber-physical security. The usage of UAV-assisted communication was suggested as a possible approach for Internet of Things (IoT) networks in the literature [8,30,31,32]. In [30], it was demonstrated how to collect data in an energy-efficient manner for IoT networks, and the best way to deploy and move several UAVs was examined. The authors developed a framework for concurrently optimizing UAV 3D positioning and mobility, device-UAV association, and uplink power regulation in their paper. Firstly, the ideal UAV location is identified based on the locations of active IoT devices at each time instant. Next, the optimal UAV mobility patterns were studied to dynamically serve the IoT devices in a time-varying network. The goal is to utilize as little energy as possible for the UAVs’ mobility while serving IoT devices. For the coverage and rate analyses, a tractable analytical framework is developed [21], wherein the UAV’s coexistence with a D2D communication network is taken into account. The authors in [33], reviewed the literature of machine learning integration to UAV-assisted communications, and they discussed various aspects including physical layer, security, resource management, and positioning.

The interfering UAVs are considered in [34], while in [35], the authors investigated the optimal 3D placement of multiple UAVs, that use directional antennas, to maximize total coverage area. The authors in [36] analyzed the impact of a UAV’s altitude on the sum-rate maximization of a UAV-assisted terrestrial wireless network. The 3D placement of drones to achieve the goal of maximizing the number of ground users served by the drones was investigated in [37]. The minimum number of drones needed for serving all the ground users within a given area was determined in [38]. In [39], evolutionary algorithms were employed to find the optimal placement of low-altitude platforms (LAPs) and portable BSs. The optimal locations and minimum number of UAVs required to completely cover the desired area are investigated for disaster relief scenarios. The authors in [40] determined the optimal location of the UAV by maximizing the average rate while ensuring that the bit error rate does not exceed a specified percentage. The UAVs are categorized as access and gateway UAVs in [41], wherein a disaster scenario, where the terrestrial infrastructure is completely down, was considered. The authors formulated the problem of gateway UAV selection (gateway UAVs use mmWave backhaul) and tried to solve it with multiarm bandit.

Different considerations, such as flight time, energy limits, ground user demands, flying regulations, and avoiding NFZs, have a substantial impact on a UAV’s trajectory. For maximizing the minimal average rate among ground users, the authors in [42] proposed a simultaneous optimization of user scheduling and UAV trajectory. While a number of jammers with unknown locations sent jamming signals, the authors in [42] presented a combination UAV and ground users’ scheduling and transmit power allocation optimization technique. The optimal trajectory of UAVs with multiple antenna for maximum sum-rate in uplink communication was researched [43]. The throughput maximization problem in mobile relaying systems was investigated in [44], where authors optimized the transmit power along with the UAV trajectory considering the practical mobility constraints such as speed and UAV’s locations. The authors in [45] proposed an E-Spiral algorithm for accurate photogrammetry. This technique used an energy model to determine different optimal speeds for straight parts of the road, thereby lowering energy consumption and improving the energy model’s ability to estimate overall path energy. To characterize the practical path planning requirements of UAVs in difficult situations, the authors in [46] developed an energy-aware, multi-UAV, multiarea coverage path planning model. A bipartite cooperative coevolution (BiCC) algorithm was suggested in this regard, which coevolves interarea and intra-area path planning components to generate good solutions. In [47], the authors proposed a geometric planning-based iterative trajectory optimization technique. To begin, graph theory was used to generate all potential UAV-ground BS association sequences, and candidate association sequences were chosen based on the topological link between UAV and ground BSs. Following that, an iterative handover location design based on the triangle inequality property is given to calculate the shortest flying route with quick convergence and minimal computation complexity. After that, by comparing all of the possible trajectories, the optimal flight trajectory can be determined. The authors presented a tradeoff between mission completion time and flight energy usage [47].

In addition, recent research looks on multiobjective optimization of UAV-assisted communication [48,49]. Over the course of a flight, a multiobjective optimization problem is constructed to jointly optimize three objectives [49]: (1) maximization of cumulative data rate, (2) maximization of total gathered energy, and (3) reduction of UAV energy consumption. Because these goals are incompatible, the authors suggested an enhanced deep deterministic policy gradient (DDPG) technique for learning UAV control policies with multiple goals. In [50], the authors developed a mathematical propulsion energy model for rotary-wing UAVs with the goal of minimizing the total energy consumption of the UAV while keeping all ground node data rates in consideration. The authors in [51] take the problem of UAV-based communicating systems in a more complete manner, where they optimize both access and backhaul links simultaneously. In particular, they consider low earth orbit (LEO) satellites as a backhauling option and try to obtain an optimal backhaul link through BS-satellite association. Besides, the transmit power, channel to be used, and the trajectory of UAV BSs are also optimized with the help of RL.

1.2. Contributions

In this paper, a smart UAV positioning mechanism is proposed by taking such regulation constraints into account to provide sustainable wireless coverage and services to the ground users under more realistic conditions. In particular, we propose a Q-learning-based approach for UAV-assisted communication systems. The optimal position of UAVs are determined under the constraints of altitude regulations, NFZs, and transmit power. The main contributions of the paper are as follows:

• A smart UAV positioning mechanism for a sustainable UAV communication system is proposed, under certain constraints.
• A multiobjective optimization model is formulated, that is, minimizing the energy consumption of UAV, while maximizing the number of users covered.
• A weighting mechanism is developed to prioritize the two objectives given in the previous item over each other for different scenarios.
• Q-learning-based algorithm is used to find the optimal position of UAV. The convergence of the developed algorithm is first tested, followed by comparing its performance with the baseline k-means method in terms of number of users covered and energy consumption.

1.3. Organization of the Paper

The remainder of this paper is organized as follows. Section 2 describes the system model including propagation and energy consumption models, while Section 3 presents the problem formulation. Section 4 presents the proposed Q-learning-based UAV positioning mechanism, followed by discussing the simulation scenario and the results in Section 5. Section 6 concludes the the paper.

2. System Model

In this section, we elaborate on the system modeling of the work, including the scenario used, propagation, and energy consumption modeling.

2.1. Scenario

We consider a UAV mounted BS to provide coverage to $n_{\mathrm{u}}$ ground users that are distributed over a geographical area of size $a\times b$ square-meter, Figure 1. Let $U=\{1,2,3,\dots ,n_{\mathrm{u}}\}$ be a set of $n_{\mathrm{u}}$ users, and the UAV can move in the (x, y, or z) direction to cover maximum number of ground users based on user density. The total time of service $T_{\mathrm{t}}$ (in mins) is divided into consecutive time-slots with equal duration of $T_{\mathrm{d}}$ (in mins), such that $n_{\mathrm{ts}}=T_{\mathrm{t}}/T_{\mathrm{d}}$ is the number of time slots, and $\overrightarrow{T}$ becomes a vector containing the consecutive time slots as $\overrightarrow{T}=[t_0,t_1,\dots ,t_{n_{\mathrm{ts}}}]$. The location of a user is represented by $(x_{\mathrm{u}},y_{\mathrm{u}},z_{\mathrm{u}})$, where $x_{\mathrm{u}}\in {\mathbb{R}}^{+}$ is in the range of $x_{\mathrm{u}}=[0,a]$, and similarly $y_{\mathrm{u}}\in {\mathbb{R}}^{+}$ is in the range of $x_{\mathrm{u}}=[0,b]$. $z_{\mathrm{u}}\in {\mathbb{R}}^{+}$ is assumed to be a constant number as $z=h_{\mathrm{u}}$, since we consider the conventional mobile handsets, which are carried around a similar height. In this work, we assume the height of UEs to be $h_{\mathrm{u}}=1.5$ m.

The altitude of the UAV $h_{\mathrm{d}}\in {\mathbb{R}}^{+}$ is in the range of $[h_{\min },h_{\max }]$ where $h_{\min }$ and $h_{\max }$ are the minimum and maximum allowed altitude (According to the regulations of concerning country/region) of the UAV, respectively. For instance, according to the European regulations for flying UAV, $h_{\min }$ and $h_{\max }$ are 30 and 120 m, respectively (Easy Access Rules for Unmanned Aircraft Systems (Regulation (EU) 2019/947 and Regulation (EU) 2019/945) by EASA (European Union Aviation Safety Agency)).

Specifically, the association between the UAV and the ground users is based on time division multiple access (TDMA). Although orthogonal frequency division multiple access (OFDMA) or nonorthogonal multiple access (NOMA) are the most used approaches for the UAV networks, we adopt TDMA to fully exploit the above-mentioned time slots for the total time of service.

2.2. Propagation Model

The propagation model is inspired from [19,24], wherein the average path loss model for air-to-ground communication can be given below in terms of the line of sight (LoS) links and non-LoS (NLoS) links.

$$\begin{array}{c}\hfill L_{\mathrm{LoS}}^k=20log\left(\frac{4\pi f_c{d}_k}{2}\right)+{\eta }_{\mathrm{LoS}},\\ \hfill L_{\mathrm{NLoS}}^k=20log\left(\frac{4\pi f_c{d}_k}{2}\right)+{\eta }_{\mathrm{NLoS}},\end{array}$$

(1)

where $f_c$ is the carrier frequency and $d_k$ is the Euclidian distance between the UAV and user k, c is the speed of light, ${\eta }_{\mathrm{LoS}}$ and ${\eta }_{\mathrm{NLoS}}$ are the mean value of the excessive path loss for LoS and NLoS, respectively. The probability of LoS link is given as,

$$\begin{array}{c}\hfill P_{\mathrm{LoS}}^k\left({\vartheta }_k\right)=\frac{1}{1+\psi exp(-\varsigma ({\vartheta }_k-\psi ))},\end{array}$$

(2)

where $\psi$ and $\varsigma$ are constant values depend on the environment, ${\vartheta }_k=\frac{180}{\pi }arcsin(\frac{h_{\mathrm{d}}}{d_k})$ is the elevation angle. Besides, the probability of NLoS link can be calculated as

$$\begin{array}{c}\hfill P_{\mathrm{NLoS}}^k\left({\vartheta }_k\right)=1-P_{\mathrm{LoS}}^k\left({\vartheta }_k\right).\end{array}$$

(3)

Therefore, the average path loss can be expressed as

$$\begin{array}{c}\hfill L_k(h_d,{\vartheta }_k)=20log\left(\frac{4\pi f_c{d}_k}{2}\right)+P_{\mathrm{LoS}}^k\left({\vartheta }_k\right){\eta }_{\mathrm{LoS}}+\\ \hfill P_{\mathrm{NLoS}}^k\left({\vartheta }_k\right){\eta }_{\mathrm{NLoS}}.\end{array}$$

(4)

2.3. Energy Consumption Model

The energy consumption model is inspired from [24] where it is modeled as a combination of the energy consumption resulting from communication, UAV hovering, and UAV mobility.

Communication Energy Consumption

The communication energy is needed to communicate with the ground users; i.e., transmit/receive the signals to/from the users. As such, the communication energy consumption of UAV $E_{\mathrm{C}}$ can be calculated as follows:

$$\begin{array}{c}\hfill E_{\mathrm{C}}\left(t_j\right)=(n_{u,t_j}{P}_{\mathrm{t}}+P_{\mathrm{cu}})t_{\mathrm{cm}},\end{array}$$

(5)

where $P_{\mathrm{t}}$ is transmission power, $P_{\mathrm{cu}}$ is the on-board circuit power, $t_{\mathrm{cm}}$ is the duration to communication of UAV to user j, and $n_{u,t_j}$ is the number of users served by UAV during time slot $t_j$.

2.4. Hovering Energy Consumption

The hovering energy is required to keep the UAV up in the air and stay at the right altitude, and the hovering energy consumption of the UAV during time slot $t_j$ can be given as

$$\begin{array}{c}\hfill E_{\mathrm{H}}\left(t_j\right)=P_{\mathrm{H}}{t}_{\mathrm{H}},\end{array}$$

(6)

where $t_{\mathrm{H}}$ is the duration of hovering of UAV. $P_{\mathrm{H}}$ (in Watts) is the instantaneous hovering power consumption that can be determined by

$$\begin{array}{c}\hfill P_{\mathrm{H}}=\frac{RU\frac{3}{2}}{\sqrt{2\rho \pi {\beta }^2}},\end{array}$$

(7)

where R is the number of rotors of the helicopter, U is the thrust, $\rho$ is the fluid density of the air, and $\beta$ is the rotor disk radius.

2.5. Mobility Energy Consumption

The mobility energy is needed to move the UAV to the optimal position to serve the ground users. From [24], the mobility energy consumption of the UAV can be given as

$$\begin{array}{c}\hfill E_{\mathrm{M}}\left(t_j\right)=P_{\mathrm{h}}\frac{d\left(t_j\right)}{v_{\mathrm{h}}}+I(\Delta h\left(t_j\right))P_{\mathrm{a}}\frac{\Delta h\left(t_j\right)}{v_{\mathrm{a}}}\\ \hfill -(1-I(\Delta h\left(t_j\right)))P_{\mathrm{d}}\frac{\Delta h\left(t_j\right)}{v_{\mathrm{d}}},\end{array}$$

(8)

where $P_{\mathrm{h}}$ is instantaneous power consumption for mobility in the horizontal direction, $P_{\mathrm{a}}$ is the ascending power, $P_{\mathrm{d}}$ is descending power. $d\left(t_j\right)$ the horizontal moving distance at $t_j$, while $\Delta h\left(t_j\right)$ is the changes in the altitude of the UAV at $t_j$. $v_{\mathrm{h}}$, $v_{\mathrm{a}}$, and $v_{\mathrm{d}}$ are the horizontal, vertical (ascending), and vertical (descending) velocities of the UAV, respectively (For the details on the calculations of these velocities, please refer to [24]). $I(\Delta h\left(t_j\right))$ is the indicator function, such that [24]

$$\begin{array}{c}\hfill I(\Delta h\left(t_j\right))=\left\{\begin{array}{cc}1\hfill & \Delta h\left(t_j\right)⩾0,\hfill \\ 0\hfill & \Delta h\left(t_j\right)<0.\hfill \end{array}\right.\end{array}$$

(9)

Lastly, the power consumption of the horizontal direction is as follows:

$$\begin{array}{c}\hfill P_{\mathrm{h}}=P_{\mathrm{P}}+P_{\mathrm{I}},\end{array}$$

(10)

where $P_{\mathrm{P}}$ is the parasitic power for overcoming the parasitic drag due to the aircraft’s skin friction [24].

3. Problem Formulation

The primary objective of this work is to maximize the number of connected users while minimizing the total energy consumption of the UAV BS to prolong its flight time. In this regard, we aim to find the optimal position of the UAV BS and associate the ground users, which are normally out-of-service due to the congestion in the terrestrial network, to it so that the number of unconnected users are reduced; however, it is important to consider the total energy consumption of the UAV BS to maximize the service duration given they they are battery operated and have limited flight time. We also consider certain constraints, including the NFZs (e.g., the UAV BS cannot fly over those forbidden regions), the altitude regulations for UAVS, etc., thereby determining the optimal positioning of the UAV BS by taken into account both the requirements and constraints becomes a nontrivial objective.

Theorem 1.

The number of connected users, $n_{\mathrm{c}}$, can be controlled by the altitude of the UAV BS, $h_{\mathrm{d}}$.

Proof. 

Let K be a rectangular prism with the base area of $A_{\mathrm{K}}=x_{\mathrm{K}}{y}_{\mathrm{K}}$, $x_{\mathrm{K}}$ and $y_{\mathrm{K}}$ are the x and y dimensions of the base of K that is placed on $z=0$ plane. If we place the UAV BS—with a directivity angle of $\theta$—at any point inside K, the radius of the footprint of the UAV BS can be calculated as follows [16]:

$$R_{\mathrm{d}}=h_{\mathrm{d}}tan\left(\frac{\theta }{2}\right),$$

(11)

where $h_{\mathrm{d}}$ is determined by

$$h_{\mathrm{d}}=|\overrightarrow{\mathbf{N}}|,$$

(12)

where $|\overrightarrow{\mathbf{N}}|$ is the length of the normal vector, $\overrightarrow{\mathbf{N}}$, from UAV BS to the $z=0$ plane. Then, the footprint of the UAV BS can be found as

$$A_{\mathrm{d}}=\pi R_{\mathrm{d}}^2.$$

(13)

If a random point is selected on $z=0$ plane, then the probability of falling inside the footprint of the UAV BS can be given as

$$p_{\mathrm{f}}={\displaystyle \frac{A_{\mathrm{d}}}{A_{\mathrm{K}}}}={\displaystyle \frac{\pi R_{\mathrm{d}}^2}{x_{\mathrm{K}}{y}_{\mathrm{K}}}},$$

(14)

where $A_{\mathrm{K}}$ is the base area of the rectangular prism K. Let $p_{\mathrm{q}}$ is the probability of receiving sufficient signal-to-noise ratio (SNR (It is worth noting that we assumed that the UAV BS is operating at an out-of-band spectrum, and thus it does not cause any interference to the terrestrial networks, or vice versa.)) for a UE, such that

$$p_{\mathrm{q}}=P(S_{\mathrm{r}}\ge T_{\min }),$$

(15)

where $S_{\mathrm{r}}$ is the received SNR, while $S_{\min }$ is the minimum required SNR value to establish a connection between the UAV BS and the UE. We assume the ground BS uses a different frequency band than the UAV BS, thereby it does not create any interference to the UAV BS. Therefore, SNR is a better choice here. Moreover, note that $T_{\mathrm{s}}$ captures the receiver sensitivity of the user equipment (UE), and $p_{\mathrm{q}}$ encompasses small-scale and large-scale fading effects. Therefore, for a UE, the probability of being served by the UAV BS can be determined as

$$p_{\mathrm{c}}=p_{\mathrm{f}}{p}_{\mathrm{q}}{p}_{\mathrm{r}},$$

(16)

where $p_{\mathrm{r}}$ is the probability of having enough resource for the UE at the UAV BS, such that $p_{\mathrm{r}}=P(B_{\mathcal{L}}\ge B_{\mathcal{R}})$, where $B_{\mathcal{L}}$ is the remaining radio resources at the UAV BS and $B_{\mathcal{R}}$ is the required radio resources for the UE. By substituting (11), (13), and (14) into (16), we get:

$$p_{\mathrm{c}}=p_{\mathrm{q}}{p}_{\mathrm{r}}{\displaystyle \frac{\pi z_{\mathrm{d}}^2{tan}^2\left(\frac{\theta }{2}\right)}{x_{\mathrm{K}}{y}_{\mathrm{K}}}}.$$

(17)

Hence, it is obvious from (17) that the probability of being served by the UAV is a function of the height of the UAV BS and they have a direct proportionality. □

Theorem 2.

The total energy consumption of the UAV BS, $E_{\mathrm{T}}$, can be controlled by the altitude of the UAV BS, $h_{\mathrm{d}}$.

Proof. 

Let $E_{\mathrm{T}}$ be the total energy consumption of the UAV BS, $E_{\mathrm{C}}$ be the communication energy, $E_{\mathrm{M}}$ be the energy consumption during the UAV mobility. Suppose UAV moves to the optimal position and attain optimal altitude to serve the ground users.

The total energy consumption of the UAV BS can be calculated as follows:

$$E_{\mathrm{T}}=E_{\mathrm{C}}+E_{\mathrm{H}}+E_{\mathrm{M}}.$$

(18)

By substituting the $E_{\mathrm{M}}$ from (8) into (18), we get

$$\begin{array}{c}\hfill E_{\mathrm{T}}=E_{\mathrm{C}}+E_{\mathrm{H}}+P_{\mathrm{h}}\frac{d\left(t_j\right)}{v_{\mathrm{h}}}+I(\Delta h\left(t_j\right))P_{\mathrm{a}}\frac{\Delta h\left(t_j\right)}{v_{\mathrm{a}}}\\ \hfill -(1-I(\Delta h\left(t_j\right)))P_{\mathrm{d}}\frac{\Delta h\left(t_j\right)}{{}_{\mathrm{d}}}.\end{array}$$

(19)

It is obvious from (19) that the total energy consumption of the UAV directly depends on the changes in the height of the UAV as well as the movement in the horizontal direction. □

3.1. Optimization Problem Formulation

There are two primary objective functions considered in this work; namely, (i) maximization of the number of served users by the UAV BS ($n_{\mathrm{c}}$) and (ii) minimization of the total energy consumption of the UAV BS ($E_{\mathrm{T}}$). Therefore, these two objective functions can be formulated as follows:

3.1.1. Maximization of Number of Served Users

The number of served users by the UAV, $n_{\mathrm{c}}$ is supposed to be maximized at each time slot. Let F be the NFZ and a nonself-intersecting convex quadrilateral that is defined by its vertices as $\overrightarrow{V_i}=(x_i,y_i,z_i)$, where $i=1,2,3,4$. Moreover, let ${\overrightarrow{C}}_{3\mathrm{d}}\in {\mathbb{R}}^3$ be a vector containing the 3-dimensional (3-D) coordinates of the UAV, and $C_{2\mathrm{d}}$ be a point in $xy$-plane, representing the projection of the UAV on the $xy$-plane, and imagine we draw straight lines from each vertex of F to the point $C_{2\mathrm{d}}$. Then, the optimization problem can be modeled as follows:

$$\begin{array}{cc}\hfill \underset{{\overrightarrow{C}}_{3\mathrm{d}}\in {\mathbb{R}}^3}{max}& f(\overrightarrow{C})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.{\mathrm{C}}_{\mathbf{1}}\mathbf{:}& h_{\min }\le h_{\mathrm{d}}\le h_{\max },\hfill \\ \hfill {\mathrm{C}}_{\mathbf{2}}\mathbf{:}& \angle A_1+\angle A_2+\angle A_3+\angle A_4<2\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{3}}\mathbf{:}& \theta <\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{4}}\mathbf{:}& P_{\mathrm{t}}\le P_{{\mathrm{t}}_{\max }},\hfill \end{array}$$

(20)

where $\angle A_1=\angle V_1{C}_{2\mathrm{d}}{V}_2$, $\angle A_2=\angle V_2{C}_{2\mathrm{d}}{V}_3$, $\angle A_3=\angle V_3{C}_{2\mathrm{d}}{V}_4$, and $\angle A_4=\angle V_1{C}_{2\mathrm{d}}{V}_4$. $f:{\mathbb{R}}^3\to \mathbb{R}$ is the objective function, and $f(\overrightarrow{C})=n_{\mathrm{c}}$ in this case.

Explanations of Constraints in (20)

• ${\mathrm{C}}_1$: The altitude of the UAV ($h_{\mathrm{d}}$) is regulated in many countries and regions, such that the maximum ($h_{\max }$) and minimum ($h_{\min }$) altitudes that UAVs can flight are determined. Therefore, in this work, the UAV is supposed to obey these limitations in terms of the altitude.
• ${\mathrm{C}}_2$: Since F is defined as the NFZ, it means that the UAV BS cannot fly over it. As such, this constraint confirms that the UAV BS is flying out of F, such that the projection of the UAV BS on the $xy$-plane, $C_{2\mathrm{d}}$, is not within F.
• ${\mathrm{C}}_3$: The directivity angle of the antenna of the UAV BS can be $\pi$ at maximum (The use of an isotropic antenna is not a good idea for UAV BSs as they are serving to the users under them in terms of height, and there is no sense to provide a radiation above the UAV BS. Therefore, we assume that the maximum antenna angle for the UAV BSs should be $\pi$), but practically it should be less than that to have a better antenna gain. Though this could be normally not a hard constraint, in this work we deal with the case where the antenna angle is less than $\pi$, thereby this becomes a constraint for the optimization problem.
• ${\mathrm{C}}_4$: Given that the maximum transmit power of the BSs are regulated, this constraint captures such regulations, meaning that the transmit power of the UAV BS has an upper bound.

3.1.2. Minimization of Energy Consumption

It is crucial to minimize the energy consumption of the UAV BS for it to stay in the air for a longer time so that the service that the ground users get is prolonged. Put it another way, the optimization objective elaborated in Section 3.1.1 focuses on maximizing the number f connected users, $n_{\mathrm{c}}$, however, such objective is instantaneous (i.e., for a duration of a single time slot, $T_{\mathrm{d}}$) and does not aim to maximize $n_{\mathrm{c}}$ for a period of time. The total number of connected users over a period of time considered can be calculated by

$$n_{\mathrm{c},\mathrm{t}}=\sum _{i=1}^{n_{\mathrm{ts}}}{n}_{\mathrm{c},i},$$

(21)

where $n_{\mathrm{c},i}$ indicated the number of served users by the UAV BS during time slot i from $\overrightarrow{T}$. In (21), $n_{\mathrm{ts}}$ is a function of $T_{\mathrm{s}}$, such that $n_{\mathrm{ts}}=f\left(T_{\mathrm{s}}\right)$, thereby although $T_{\mathrm{s}}$ is assumed to be fixed here, normally it is dependent on the energy stored in the UAV battery (i.e., battery capacity) as well as the energy consumption of the UAV BS. Since the battery capacity is fixed (We acknowledge that different UAVs have different battery capacities, but here we mean that once a UAV is selected out of various options, the battery capacity becomes something that cannot be changed/controlled), the only way left to prolong the UAV flight time is reducing the energy consumption. Therefore, the second objective of our problem formulation becomes the minimization of the total energy consumption of the UAV BS ($E_{\mathrm{T}}$), and that can be modeled as follows:

$$\begin{array}{cc}\hfill \underset{{\overrightarrow{C}}_{3\mathrm{d}}\in {\mathbb{R}}^3}{min}& g(\overrightarrow{C})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.{\mathrm{C}}_{\mathbf{1}}\mathbf{:}& h_{\min }\le h_{\mathrm{d}}\le h_{\max },\hfill \\ \hfill {\mathrm{C}}_{\mathbf{2}}\mathbf{:}& \angle A_1+\angle A_2+\angle A_3+\angle A_4<2\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{3}}\mathbf{:}& \theta <\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{4}}\mathbf{:}& P_{\mathrm{t}}\le P_{{\mathrm{t}}_{\max }},\hfill \end{array}$$

(22)

where $g:{\mathbb{R}}^3\to \mathbb{R}$ is the objective function, and $g(\overrightarrow{C})=E_{\mathrm{T}}$ in this case.

3.1.3. Multiobjective Problem Formulation

As detailed in Section 3.1.1 and Section 3.1.2, there are two distinctive objectives included in our problem; i.e., maximization of connected users—as given in (20)—and minimization of the energy consumption of the UAV BSs—as given in (22). In this work, we aim at optimizing the both objectives—(20) and (22), simultaneously. In this regard, we developed the following optimization model:

$$\begin{array}{cc}\hfill \underset{{\overrightarrow{C}}_{3\mathrm{d}}\in {\mathbb{R}}^3}{max}& h(\overrightarrow{C})\hfill \\ \hfill \mathrm{s}.\mathrm{t}.{\mathrm{C}}_{\mathbf{1}}\mathbf{:}& h_{\min }\le h_{\mathrm{d}}\le h_{\max },\hfill \\ \hfill {\mathrm{C}}_{\mathbf{2}}\mathbf{:}& \angle A_1+\angle A_2+\angle A_3+\angle A_4<2\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{3}}\mathbf{:}& \theta <\pi ,\hfill \\ \hfill {\mathrm{C}}_{\mathbf{4}}\mathbf{:}& P_{\mathrm{t}}\le P_{{\mathrm{t}}_{\max }},\hfill \end{array}$$

(23)

where $h:{\mathbb{R}}^3\to \mathbb{R}$ is the objective function, and $h(\overrightarrow{C})=w_{1}f(\overrightarrow{C})-w_{2}g(\overrightarrow{C})=w_1{n}_{\mathrm{c}}-w_2{E}_{\mathrm{T}}$ in this case. Here, $w_1,w_2\in \mathbb{R}$ are coefficients used for two purposes:

• To prioritize one objective over the other. For example, a mobile network operator may not be interested in the energy consumption much and focuses only on covering as much as users as possible for a short duration, and it would choose $w_1\gg w_2$. On the other hand, if the operator ranks both objectives equally, then it would choose $w_1=w_2$. Therefore, $w_1$ and $w_2$ allow the operators to rank the objectives according to their requirements.
• To make the units of both $f(\overrightarrow{C})$ (unitless) and $g(\overrightarrow{C})$ (in Joules) the same, since $h(\overrightarrow{C})$ includes the summation of $f(\overrightarrow{C})$ and $g(\overrightarrow{C})$. To this end, while $w_1$ is chosen to be unitless, $w_2$ is in ($1/\mathrm{Joules}$).

4. Proposed Q-Learning Based UAV Positioning Mechanism

RL has a special place in machine learning, as it is structurally quite different than supervised and unsupervised learning methods. RL consists of a set of policy-based goal-seeking algorithms, where an agent takes actions in a given environment to maximize its reward or minimize the penalty, and therefore RL is predominantly used in optimization problems rather than time-series analysis, classification, or clustering as supervised and unsupervised learning algorithms do.

RL has unique advantageous characteristics making it more preferable than other types of optimization methodologies, including heuristics. Firstly, RL algorithms, such as Q-learning and state–action–reward–state–action (SARSA), are predominantly model-free, meaning that they do not require the model of the environment-of-interest in advance, they instead interact with the environment to capture its dynamics [52]. Moreover, since RL algorithms include learning in their body, they do not have to start from scratch every time there is a change in the environment, they rather adapt themselves to the changes, giving them a strength of optimization with reasonable time complexity. This is an essential feature for an optimization algorithm especially for dynamic scenarios, where network conditions change rapidly and frequently. To this end, we employ Q-learning, one of the most common RL algorithms, in this work to take the benefit of above-mentioned features.

In RL, there is an agent taking actions to find the optimum policy for a given problem. Based on the action of the agent, first, corresponding state is observed, followed by evaluating the subsequent penalty/reward function. Then, the action-value function, storing calculated penalty/reward values for all the states and actions, is updated [52]. The agent takes action in two different ways: explore and exploit. In the initial phases of the implementation, the agent is expected to explore more to discover the environment better. However, after a sufficient exploration, the agent should start exploiting the available information to be able to focus on finding the best policy.

We adopted OpenAI Gym [53] tool for building environment for this study. It is based on episodic RL, where experience of each agent is divided into episodes. In initial state of each episode, we randomly localize the UAV BS and the users in a grid, and learning proceeds until the environment reaches one of the stopping criteria (this will be detailed in the following paragraphs). The main goal here is to maximize the total reward per episode and to decrease the number of episodes for achieving desired performance. RL steps in each episode are given in Algorithm 1, where $s_t$ and $s_{t+1}$ are the current and next states, respectively, and $a_t$ is the current state while $R_{t+1}$ is the expected value of the reward function.

Algorithm 1:Q-learning algorithm [52]

In this study, states refer to the position of the UAV in the grid. The agent in the developed Q-learning algorithm has seven action values for each state, which denote the agent action $a_t$ in UAV state of $s_t$ at time t. The possible actions for each state $s_t$ are hold, move up, move down, move left, move right, move forward, and move backward. The agent follows $ϵ$-greedy [52] policy to take random actions initially—which is referred to as exploring— and decays $ϵ$ through iterations—which is referred to as exploiting— for decreasing random actions. Given that the main goal of this study is to optimize energy consumption of the UAV along with maximizing user coverage, the reward function in the proposed method is inline with the objective function in (23), and depends on the energy consumption and coverage.

The components of the developed Q-learning algorithm for the problem of UAV BS positioning are detailed in the following paragraphs.

4.1. Environment

We create discrete environment with finite size (grid) representing the state of UAV in OpenAI-Gym. The size of the grids in the environment-of-interest in this study is $(25,25,12)$, which is simulated with 10 m-resolution in each axis. Therefore, the real environment size becomes $(250,250,120)$ in meters. These certain dimensions of the environment are chosen by considering both the computational burden and the reality of the work; such that, the environment should be in a size of some realistic area (and the UAV BS should be able to have sufficient degree of freedom in movement) while not bringing much computational burden (the simulation time should be reasonable for us to make some tuning during the design of the algorithm). However, we intuitively confirm that the developed algorithm would work in any environment size, as the UA BS can only move slightly at one iteration thereby extending the size of the environment would not affect the performance of the algorithm other than prolonging the simulation time.

4.2. Agent

The UAV BS in the state $s_t$ corresponds to the agent in this study. It will take an action, $a_t$, in state $s_t$, and it receives an observation and reward from the environment. Accordingly, it updates Q-table to learn the dynamics of the environment, and adapt itself to the changes. It is quite convenient to choose the UAV BS as the agent in the developed Q-learning algorithm, as it is the only one taking different actions; e.g., moving in different directions.

4.3. Actions

We consider seven different actions that agents can take. Let ${\overrightarrow{C}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}},z_{\mathrm{u}})$ be the current position of the UAV BSs and ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}$ be the position after an action taken, while r (in meters) denotes the step size in any direction. Then, the set of the actions, $\mathbf{A}$, that the agent takes is as follows:

• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}},z_{\mathrm{u}}+r)$: Move up (in z direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}},z_{\mathrm{u}}-r)$: Move down (in z direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}}+r,y_{\mathrm{u}},z_{\mathrm{u}})$: Move left (in x direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}}-r,y_{\mathrm{u}},z_{\mathrm{u}})$: Move right (in x direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}}+r,z_{\mathrm{u}})$: Move forward (in y direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}}-r,z_{\mathrm{u}})$: Move backward (in y direction)
• ${\overrightarrow{\widehat{C}}}_{3\mathrm{d}}=(x_{\mathrm{u}},y_{\mathrm{u}},z_{\mathrm{u}})$  :Hold

4.4. States

We denote state s as the position of the UAV in a 3D space. We divide 3D space into grids (i.e., we discretized the state space) for having finite set of state that can be used in Q-learning. This state selection is inline with the criterion given in [52], such that the state should be affected by the actions that the agent takes. As such, the actions of the agent is fundamentally altering the 3D position of the UAV BS, which changes the state of the agent, which is also defined to be the 3D position of the agent.

4.5. Reward

To avoid the limitations of the work (or respect the constraints, in other words), a penalty mechanism is developed, such that the agents obtains a reward of -1 when the UAV BS

• goes beyond the dimensions of the environment,
• flies on the NFZ,
• does not respect any other constraint in (23).

On the other hand, a reward function is designed for the cases where the UAV BS is not in one of the states listed above. Since the main goal is to optimize energy consumption along with maximizing number of user covered, the reward is defined inline with the optimization objective in (23), such that

$$\begin{array}{c}\hfill R_{t+1}=h(\overrightarrow{C})=w_1{n}_{\mathrm{c}}-w_2{E}_{\mathrm{T}}.\end{array}$$

(24)

The selection of the reward function as in (24) (i.e., making it equal to $h(\overrightarrow{C})$) is a legitimate decision, because the the objective of the developed Q-learning algorithm is to maximize the reward, R, and the objective function in (23) is the maximization of $h(\overrightarrow{C})$. Thus, making the reward equal to $h(\overrightarrow{C})$ is completely inline with the model in (23).

4.6. Policy

There are mainly two phases in the search of an agent to find optimal solution: (i) exploration and (ii) exploitation. In the former, agents enhance their current knowledge for a long-term benefit and make more accurate estimations of action-values. On the other hand, in the latter, agents take advantage of their experiences and perform greedy actions to obtain the most reward by exploiting the action-value estimate of agents. This behavior may not result in getting the most award and leads to suboptimal solutions. Namely, agents make more accurate action-values predictions when they explore, and receive more reward when they exploit. It is not possible to perform exploration and exploitation simultaneously which is known as exploration-exploitation dilemma. $ϵ$-greedy is a simple policy to create a balance between exploration and exploitation. $ϵ$ identifies the probability of agent’s action to choose explore. This method is formulated as follows:

$$\begin{array}{c}\hfill A_t=\left\{\begin{array}{cc}{Q}_t\left(a\right)\hfill & (1-ϵ)\hfill \\ \kappa \hfill & ϵ,\hfill \end{array}\right.\end{array}$$

(25)

where $\kappa$ is a random variable with a discrete uniform distribution on $\mathbf{A}$.

We follow an $ϵ$-greedy policy [52] to explore the environment by taking random actions in earlier iterations (exploration phase). As the iterations proceed (e.g., the number of the iterations get larger), we turn the exploration phase into the exploitation phase by decreasing $ϵ$ with a decay-rate of $0.95$. This is done to allow the agent explore and acquire new experiences during the exploration phase, while in the exploitation phase it uses the obtained experience to converge to an optimal value.

4.7. Q-Table Update

We update the Q-table according the action $a_t$ in the state $s_t$ using:

$$\begin{array}{c}\hfill Q(s,a)=:Q(s,a)+\\ \hfill \alpha [R_{t+1}+\gamma max\left(Q\left(s_{t+1}\right)\right)-Q(s,a)]\end{array}$$

(26)

where $\alpha$ is the learning-rate, and $\gamma$ is the discount rate. Q-table update is crucial in storing the obtained experience as well as modifying it with the new data.

4.8. Initialization

In each episode, the UAV BS and the users are located randomly in the grid, so that the agent does not “memorize" (or it is called as overfitting in more technical terminology) a certain environment, instead produce a more generic model.

4.9. Episodes

The episode is considered as a snapshot of the environment in the problem formulation. The agent takes random actions in each episode and learn the environment using Q-table with (26) by evaluating the reward, R, through (24). When the the agent reaches stopping criteria, a new episode begins.

4.10. Stopping Criteria

If the predefined maximum number of iterations is reached or all the users are covered by the UAV BS, the current episode is terminated, and the algorithm goes into a new episode. The maximum number of iteration is set to 2000 in this work.

4.11. Complexity

The space complexity of Q-learning algorithm is the number of states times the number of actions. In this study, the UAV BS can take 7 actions as described earlier in Section 4. The 3D environment is discretized into (25, 25, 12), and thus the total number of states is 25 × 25 × 12 = 7,500.

The time complexity of proposed algorithm is the same as Q-learning which is $O\left(N_a\right)$ where $N_a$ is the number of actions. The current position of the UAV BS is the current state of the UAV BS and the action is performed accordingly by looking up the Q-table after training.

5. Performance Evaluation

In this section, we present the performance evaluation of the proposed methodology. After describing the simulation scenario, we introduce the benchmark method as well as the performance metrics, followed by presenting the obtained results and corresponding discussions. The parameters used in the simulation campaigns are given in

Table 1. Simulation parameters.

Parameter	Value
General
Carrier frequency, $f_{\mathrm{c}}$	1 GHz
Antenna directivity angle, $\theta$	60${}^{\circ }$
Minimum UAV height, $h_{\min }$	30 m
Maximum UAV height, $h_{\max }$	120 m
Urban Area	250 × 250 m${}^2$
Total number of users, $n_{\mathrm{t}}$	100
Height from ground for all UEs	1.5 m
Speed of light, c	$3\times {10}^8$ m/s
$\eta$LoS	1.6 dB
$\eta$NLoS	23 dB
Parameter of A2G path loss model, $\Psi$	12.08
Parameter of A2G path loss model, $\zeta$	0.11
Number of rotors, M	4
Fluid density of the air, $\rho$	1.2 Kg/m${}^3$
Rotor disk radius, $\beta$	0.25 m
Weight of the frame	1.5 Kg
Weight of the battery and payload	2 Kg
Bandwidth	180 kHz
Transmit power, $P_{\mathrm{t}}$	30 dBm (1 W)
On-board circuit power, $P_{\mathrm{cu}}$	0.01 W
Duration of hovering of UAV, $t_{\mathrm{h}}$	1 s
Duration to communication of UAV, $t_{\mathrm{cm}}$	1 s
Velocity of the UAV, v	30 m/s
Angular velocity, $\omega$	40 rad/s
Drag coefficient	0.025
Rotor chord, $c_{\mathrm{b}}$	0.022 m
Reference frontal area of the UAV	0.192 m${}^2$
$\mathit{Q}$-learning
Discount rate, $\gamma$	0.9
Epsilon, $ϵ$	1
Epsilon decay, $ϵ$-decay	0.95
Learning rate, $\alpha$	0.9
Learning rate decay, $\alpha$-decay	$1\times {10}^{-4}$

.

The speed of the UAV BS is fixed in our model, as seen in Table 1. However, it is also an important parameter to considers, as it can have multiple impacts on the performance. For example, if the speed of the UAV BS is less than the average speed of the ground users, the UAV BS positions itself after the perfect time, which would degrade the overall performance as there is a delay in finding the optimal position. In the extreme scenario where the speed of the UAV BS is too less than the average speed of the users, a new positioning would be needed by the time the UAV BS arrives at the optimal position, as the users already moved, and the position of the UAV BS is not the optimal anymore. The same principle also applies to the case where the speed of the UAV BS is more than the average speed of the ground users, since there will be delay between the time that the UAV BS positions to an optimal point and the time that such point becomes the optimal position (because the users move slower than the UAV BS, which arrives the future optimal position early). The speed of the UAV BS requires a complex and detailed discussion, which would make this work lengthy and would also distract the main focus of the work. Therefore, we kept that discussion for a further study and focused only on how the UAV BS can be located in the case of moving users and NFZs.

5.1. Simulation Scenario

We implement a simulation scenario to evaluate the proposed Q-learning algorithm. An urban area of $250\times 250$ m${}^2$—which is discretized by means of square-shaped grid—and $n_{\mathrm{t}}=$ 100 total number of users (that are normally out-of-service from the terrestrial network) are considered. Consequently, the UAV can move in the discretized 3D space in terms of x, y, and z coordinates. Furthermore, due to the regulations, we impose a minimum and a maximum altitude of $h_{\min }=$ 30 and $h_{\max }=$ 120 m, respectively, and a certain number of NFZs, corresponding to specific not allowed grids. The users in the considered model are mobile and move at each time slot, and we consider a random walk model for the user mobility, meaning that the users move by a certain amount of distance from their original location in a random manner; such that the directions that they can move to have equal probability. The height from ground for all UEs is fixed to 1.5 m. Furthermore, we assume the directivity angle $\theta =$ 60${}^{\circ }$ and the carrier frequency $f_c=$ 1 GHz. An initialization procedure is performed to explicate the initial conditions. In particular, the values for all the involved parameters are determined, considering arbitrary positions for the UAV BS and the users. An outage threshold is calculated considering a required minimum received power to establish and maintain with a certain QoS a connection between the UE and the UAV BS, called $P_{{\mathrm{r}}_{\min }}$. For a given transmitted power for the UAV BS, $P_{\mathrm{t}}$—lower than the maximum allowed value, $P_{{\mathrm{t}}_{\max }}$—, and the above-mentioned $P_{{\mathrm{r}}_{\min }}$, the path loss experienced by the UE, L, can be expressed by the following relation, $L=P_{\mathrm{t}}-P_{{\mathrm{r}}_{\min }}$. Considering the 2D position of the UAV BS ($C_{2\mathrm{d}}$), the QoS constraint can be expressed in terms of L lower than $L_{\max }$ [19], where $L_{\max }$ is the path loss experienced by edge users. The footprint of the UAV BS, on the other hand, can be considered as a circle centered in the 2D position of the UAV BS ($C_{2\mathrm{d}}$).

5.2. Benchmark and Metrics

In this work, k-means algorithm is employed as a benchmark method, since it was widely used in similar problems [16]. k-means is an unsupervised clustering algorithm, where the data points are clustered according to certain features. In k-means, a centroid is assigned to each cluster and the objective is to place the centroids to the position which yields minimum cumulative distance to the data points. In particular, in the initialization of the algorithm, the centroids are placed randomly and the data points are assigned to each centroid to form a cluster, and the assignment is done in a way that each data point is assigned to the cluster that is closest to it in terms of Euclidian distance. Then, the centroids are moved towards to the center of their clusters, and this process iteratively continues until the convergence, where the centroids cannot be moved anymore.

Therefore, as this algorithm finds the position of the centroid, where the cumulative distance between the centroid and the data points is the minimum, it serves as a strong benchmark for this problem. In particular, if the UAV BS is considered as the centroid, while the ground users are the data points, the k-means algorithm positions the UAV BS at a point where it is closest to the ground users in terms of distance. Given that the distance is the primary parameter affecting the link quality between the transmitter and receiver, k-means algorithm becomes a appropriate benchmark. With this algorithm, we compute the centroid position related to the actual ground users’ positions. Consequently, the centroid corresponds to the best 2D position for UAV, in terms of respective distances between UAV and ground users.

Two different phases, one for training and one for testing, respectively, are performed to demonstrate the efficiency of the proposed Q-learning algorithm in terms of coverage and energy consumption. Regarding the coverage, we count the number of ground users, which are normally out of service from the terrestrial network, connected to the UAV BS, and the more users covered means a better performance in terms of the coverage. In the energy consumption, we measure the total energy consumption, $E\mathrm{T}$, of the UAV BS while it is providing service to the users, and the less energy consumption refers to a better performance in terms of the energy consumption as the flight time of the UAV BS is prolonged.

During the training phase of the developed Q-learning algorithm, the simulation is conducted through a certain number of episodes to populate the related Q-table and consequently achieve the needed learning. A trade-off between coverage and energy consumption prioritization is considered both in training and testing phases. Specifically, five different experiments are performed with different values of the weights (e.g., $w_1$ and $w_2$) that are responsible of prioritizing the coverage or the energy consumption.

5.3. Simulation Results

Figure 2 demonstrates the averaged and normalized results in terms of energy consumption (orange bars) and covered users (green bars) for different altitudes through k-means algorithm. Since the k-means algorithm determines the 2D position of the UAV BS, the altitude of the UAV BS should also be determined. Although there are different methods to determine the altitude, such as a trigonometric approach is used in [16], those usually do not consider the altitude regulations for the UAVs. In this work, on the other hand, considering such regulations, we used different fixed levels for the altitude of the UAV BS. In particular, three different altitude levels are considered: i) minimum allowed altitude ($h_{\min }=$30), maximum allowed altitude ($h_{\max }=$120 m), and the middle point between the two $\left({\displaystyle \frac{h_{\max }+h_{\min }}{2}}\right)=$75 m.

From (11) it is understood that the value in the second case, i.e., with maximum allowed height of 120 m, can be assumed as the upper-bound in terms of coverage, since in this case the UAV is placed in the best 2D position with the maximum allowed height, that means the maximum achievable coverage area with respect to the size of the considered urban area, consequently obtaining the maximum number of covered users. A similar consideration can be done for this case (i.e., $h_{\max }=$120 m) in terms of energy consumption. Since all ground users are covered at the first iteration, one of the stopping criteria is readily matched, thereby no movement is performed by the UAV, resulting in energy consumption due to mobility equals to zero. For the two remaining cases, with altitudes of 30 and 75 m, respectively, the results in terms of coverage can be considered as lower-bound and median values, since as previously stated, the coverage area, and subsequently, the number of covered user are highly dependent on the considered altitude of the UAV BS. Lastly, considering the energy consumption results, the UAV BS exploits the maximum number of allowed iterations attempting to match the stopping criteria for coverage, consequently resulting in the maximum value for energy consumption due to mobility.

We acknowledge that the number of users considered in the simulation campaigns would intuitively change the results in terms of the percentage of covered users, since the UAV BS has a certain amount of radio resources available and cannot serve all the users in case the number of users increased (when the total demand exceeds its available capacity). However, the goal of the agent (the UAV BS in our case) in this design is to maximize the number of users covered when $w_2$ is prioritized, and the simulation results confirm that the agent performs according to the prioritization conditions. Therefore, the scope of this work is to prove that the UAV BS is capable of maximizing the number of users covered and/or minimizing the energy consumption according to the prioritization policy. In the cases where the total demand is more than the capacity of a single UAV BS, then deploying multiple UAV BSs would be considered to accommodate more users in the system.

Figure 3 presents the results in terms of achieved rewards, after an initial phase, the convergence of the Q-learning algorithm occurs, demonstrating the effectiveness of the Q-learning algorithm. One of the important takeaway from the findings in Figure 3 is that, regardless of the weighting approach (e.g., for different $w_1$ and $w_2$ values), the designed Q-learning algorithm converges to the final reward. This confirms the proper design of the algorithm, and is a clear sign that it can work in various scenarios.

Following the above assumptions, the efficiency of the proposed Q-learning algorithm is verified through the testing phase, with the UAV BS positioning optimization through k-means as a benchmark. In this phase, the UAV BS is located in the above-mentioned simulated scenario and an arbitrary uniform distribution for the ground users is considered. The testing phase is conducted performing a certain number of runs, to average the results with regard to the specified parameters. Figure 4 shows the single UAV BS position optimization for different set of weights and the results are the normalized version of the average values between 0 and 1. The goodness of considering a trade-off between coverage and energy consumption, achieved by means of the two different weights, is mostly visible in two of the five experiments. In particular, for the $w_1=$0.2 and $w_2=$0.8, the best average energy consumption is achieved, whereas for the weights $w_1=$0.0 and $w_2=$1.0, best overall coverage, and rewards are obtained. In other words, the proper performances of the above-mentioned trade-off can be observed from the results in Figure 4. Effectively, when energy consumption is not prioritized, the UAV BS finds the optimum position in fewer episodes but at the expense of a higher energy consumption, conversely in the remaining cases. Therefore, the designed weighting mechanism works well, as the performance of the Q-learning algorithm is deeply affected by the numerical values of the weights. However, these results does not only affirm that the weighting mechanism works, but also gives superiority to the proposed approach, as it can converge to a solution according to the requirements of the network operators.

6. Conclusions

In this paper, a smart UAV BS positioning mechanism was proposed by taking altitude regulations as well as NFZs into account along with some hard constraints, including maximum transmit power and directivity of the UAV BS antenna, to provide sustainable wireless coverage and services to the ground users under more realistic conditions. As such, firstly, two different optimization models were developed for the minimization of the energy consumption and maximization of the number of users covered. Then, these two distinctive models are combined with a weighting mechanism, and a multiobjective optimization problem formulation was developed. With the developed weighting mechanism, wireless networks operators become capable of positioning the UAV BSs according to their requirements by relatively ranking the energy consumption and the number of users covered. We proposed a Q-learning-based approach for UAV-assisted communication systems, and the OpenAI Gym tool was used to build the RL environment. The objective is to find the optimal position of the UAV and minimize the energy consumption while maximizing the number of users covered. The results demonstrate that the proposed solution outperforms the baseline k-means method in terms of covered users, while achieving the desired minimization of the energy consumption.

Since a single UAV BS has a certain amount of radio resources and is capable of accommodating a limited number of users, in future we plan to investigate the UAV-assisted coverage extension problem in multi-UAV scenario, wherein more than one UAV BS are deployed to serve to the ground users to further maximize the number of connected users. That would be a more challenging problem due to the fact that the interference between the UAV BSs will be another aspect to consider, as such interference will have an impact on the communication performance and so the positions of the UAV BSs. Future work would also consider to investigate the impacts of the speed of the UAV BS in the overall performance. Moreover, as in a realistic case, having a dynamic speed for the UAV BS, in which case the speed should also be optimized, will be also considered.