FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures

Round 1

Reviewer 1 Report (Previous Reviewer 4)

Comments and Suggestions for Authors

This is a resubmitted manuscript. It would be better for the reviewer if the authors attach a reply letter with a manuscript that highlights the changes made in response to the reviewer in the previous round.

Comments on the Quality of English Language

It is fine with the English quality of the current manuscript.

Author Response

Please see the attachment.

 

Author Response File: Author Response.pdf

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

This article considers network communication in emergency rescue scenarios and constructs an emergency network with unmanned aerial vehicles (UAVs) as the core communication equipment using a layered multi-domain data transmission architecture. It proposes a routing decision strategy based on Federated Reinforcement Distillation (FedRDR) for network communication.

1) The authors need to provide relevant literature to demonstrate that UAVs can effectively provide communication services in disaster rescue scenarios and clarify the differences between this article and existing UAV rescue networks.

2) The authors should clarify their own contributions, whether the focus of the article is on system construction or algorithm design.

3) Regarding the mentioned routing algorithm in the article, does it involve communication between different domains or only within the same domain? Please clarify this part of the question.

4) Is this article about routing communication in an ideal state? Has the possibility of emergencies rendering UAVs unusable been considered?

5) The authors should review the writing of the paper more carefully and modify any unprofessional descriptions.

6) Please carefully check the formatting of formulas in the paper and ensure consistency.

7) This paper's organization is not very clear, especially in the introduction section. The research motivation and system construction process are not well defined.

8) Has this paper considered other environmental factors, such as harsh weather? How would these factors affect the proposed system and algorithm?

9) Can the UAVs maintain a constant connection?

10) Does the proposed architecture and algorithm work equally well in highly complex network scenarios?

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report (New Reviewer)

Comments and Suggestions for Authors

In this paper, the authors proposed the hierarchical multi-domain data transmission architecture for an emergency network, with UAVs employed as core communication devices. In the architecture, the UAV controllers perceive the network status and learn the spatio-temporal characteristics of air-to-ground network links. Also, the authors presented a routing algorithm based on federated reinforcement distillation which enhances the generalization capability of the routing decision model by increasing the training data samples.

Taken overall, the paper is readable and the proposed scheme sounds reasonable. However, the description of the technique is somewhat complicated, making it difficult to understand the authors’ contribution and the novelty of the proposed technique. Thus, this manuscript should be improved further, and some modifications are required as follows:

1.     I think that the authors need to clarify the difference between the existing researches and the proposed research. In addition, the authors should clearly highlight the novelty of the study.

2.     There are many grammatical and presentation errors which degrade the level of completion of the manuscript. The followings are just some examples.

A.     In 105, no spacing

B.      In 115, the reference position is inconsistent.

C.      In 155, an imperative sentence was used.

D.     Figure 1 can be improved

E.      In 285, no spacing.

F.      In 293, 12 -> 1

G.     In 297, target -> target

H.     In 325, there is an error.

I.       In 329, no spacing

J.       In 350, make up -> makes up

K.      In 378, the past tense should be used.

L.      In 387, no spacing

3.     There are some expressions that are rarely used in the paper. These need to be modified.

4.     Many informal notations and expressions were used in mathematical equations and algorithms

Comments on the Quality of English Language

There are many grammatical and presentation errors which degrade the level of completion of the manuscript. Also, there are some expressions that are rarely used in the paper. These need to be modified.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report (New Reviewer)

Comments and Suggestions for Authors

The authors addressed all problems my concern.

Reviewer 3 Report (New Reviewer)

Comments and Suggestions for Authors

This manuscript was considerably improved through the revision.

No more comments.

Comments on the Quality of English Language

Minor editing and polishing up is recommended.

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper  proposes a hierarchical multi-domain data transmission architecture for emergency networks, using unmanned aerial vehicles (UAVs) as core communication devices. It introduces the Federated Reinforcement Distillation (FedRDR) routing algorithm. The authors claim that it    reduces parameter transmission size by approximately 29% compared to other algorithms.  The paper covers an important topics; however, there are few points to be considered before acceptance.

-         Drones are prone to failure by themselves; therefore, the multilevel architecture of the drones could cause many other problems due to failure or energy depletion of one or more drone. The authors did not consider such problems during their design. 

-         The experiments are conducted on a small network; however, in real situations, the number of drones could be large.  So, can the proposed model be affective. 

-         There are other parameters that might affect the radio transmission of the drones such as environmental conditions, especially in disaster and emergency situations.    The paper approach did not consider such parameters. 

-         Assuming ideal cases of interconnection drones is not viable assumption .      

 

Comments on the Quality of English Language

NA

Author Response

Response to Review Comments

We express our gratitude to the editor and reviewers for their insightful and constructive review comments.The feedback provided by the reviewers is critical in refining our paper and shaping the direction of our research.We have carefully reviewed the comments, and have made necessary corrections to address the concerns raised by the reviewers.The revised manuscript incorporates the major corrections suggested by the reviewers. The following section outlines our response to each of the comments provided by the reviewers.

Reviewer #1:

1.Drones are prone to failure by themselves; therefore, the multilevel architecture of the drones could cause many other problems due to failure or energy depletion of one or more drone. The authors did not consider such problems during their design. 

Response： First of all, I would like to thank the experts for their suggestions on our work. In our next steps, we will take into consideration the factor of energy consumption in the UAV and conduct further experiments. In this paper, our focus is on deploying a fast multi-domain data transmission architecture among different UAVs using the strategy model. Therefore, the energy consumption of the UAVs is sufficient to support the deployment of the entire architecture in this study.

Modified place：We have revised the first paragraph of the introduction as follows:

For example, in September 2023, certain regions in China were affected by Typhoon "Dusuwei," resulting in floods and geological disasters. Relevant departments utilized unmanned aerial vehicles to play a crucial role in communication support, disaster reconnaissance, and material delivery in challenging rescue environments and complex weather conditions.

2.The experiments are conducted on a small network; however, in real situations, the number of drones could be large.  So, can the proposed model be affective. 

Response： Because this paper considers a distributed multi-domain data transmission architecture, the focus of the strategy work in data transmission lies in the efficiency of the routing decision process. Therefore, we need to minimize the amount of parameters to be transmitted in order to generate decision models quickly. Regarding its effectiveness, it is not necessarily required to have a large number of drones as the backbone network for coverage within a certain area.

Modified place：In the 5.2 simulation system design section, we can add the following paragraph in the first paragraph:

This paper considers a distributed multi-domain data transmission architecture, in order to make the routing decision process efficient, we need to minimize the amount of parameters to be transmitted, thus enabling quick generation of decision models. The number of drones required for the backbone network part does not necessarily have to be very large.

 

3.  There are other parameters that might affect the radio transmission of the drones such as environmental conditions, especially in disaster and emergency situations. The paper approach did not consider such parameters. 

Response：We appreciate the reviewer's suggestions on wireless transmission factors that may affect drones. This article focuses on the situation where infrastructure is damaged due to natural disasters. The network status information obtained by the network processor is then transformed into a traffic matrix, which is used as the basis for making our routing decision. However, no consideration was given to this complex channel condition, and this was left for further work.

Modified place：In the 3. System Model, We change the end of the first paragraph as follows：

For situations in which infrastructure is damaged due to natural disasters, the network status information obtained through network processors (such as topology, link, node, or network traffic status) is converted into a traffic matrix. Based on this, decisions are made for our routing selection. In order to protect the privacy of information regarding the network status of different domains, a federated learning architecture is deployed on domain controllers and global controllers for the overall decision-making model. This enhances the data privacy and security of the network while reducing network transmission costs.

4. Assuming ideal cases of interconnection drones is not viable assumption .

Response：This paper does not assume interconnection between unmanned aerial vehicles. The communication between drones is conducted through wireless channels. After the infrastructure is damaged, we rely on the communication devices carried by the drones and communicate using wireless communication protocols. In the simulation algorithm, we configure the parameters of such wireless links accordingly in the experiments.

Modified place：We have revised the first paragraph of the introduction

In emergency situations and natural disaster conditions, traditional IoT communication infrastructure, including sensor facilities, may be damaged. This can lead to communication service disruptions. One urgent issue to address is the ability to transition quickly from a faulty state to a normal state and ensure that ongoing communication services can continue smoothly without being affected by network failures. After the infrastructure is damaged, drones communicate through wireless channels. These drones carry communication devices and communicate using wireless communication protocols. This enables the entire IoT system to restore its normal communication state. For example, in September 2023, certain regions in China were affected by Typhoon "Dusuwei," resulting in floods and geological disasters. Relevant departments utilized drones to play a crucial role in communication support, disaster reconnaissance, and material delivery in challenging rescue environments and complex weather conditions.[22]

 

Reviewer 2 Report

Comments and Suggestions for Authors

The first reaction to the paper, is that the motivation is not clear.. Sometimes authors talk about IoT communications, other times about generic air-to-ground support cellular networks for special situations such as disaster areas, or even about fast moving UAVs and 6G . All these use-cases are very different.

More on the technical side, the role of federated learning is not clear. Authors talk about using it to ensure user privacy. Authors also talk about collaborative learning, to reduce communication costs. It is assumed to be of federated learning models, although this is not clearly mentioned in the paper.

Later on, authors mention that actually the goal of the paper is to apply air-to-ground collaborative networking and mobile edge computing to routing mechanisms. At this point it is not clear what does it means collaborative networking (is like cooperative relaying of opportunistic networks as found in the literature). Moreover the new topic, MEC, is not directly correlated with all the AI topics mentioned before, although it is not difficult to build that correlation. But it is not clear how  the authors propose to do it.

Later on, authors talk about intra-domain and multi-domain issues, which is new to the reasoning presented in the paper... Are authors talking about clustered networks? How does this related with all the previous topics?

Moreover it is not clear the difference between the method proposed in this paper and other papers about federated reinforcement distillation such as "Federated Reinforcement Distillation with proxy experience memory" in Arxiv Aprl 2020.

The paper needs a better statement about its motivation. It is difficult to follow the argumentation with so many different topics mentioned. From a technical point of view it would be good to have a clear distinction with other papers proposing federated learning distillation and the usage of Q-routing.

 

Comments on the Quality of English Language

The paper should be revised in order to improve the way sentences are written and to improve the connection between sentences and paragraphs. Sometime the reading flow is stopped since sentences are referring to different uncorrelated issues.

Author Response

Response to Review Comments

We express our gratitude to the editor and reviewers for their insightful and constructive review comments.The feedback provided by the reviewers is critical in refining our paper and shaping the direction of our research.We have carefully reviewed the comments, and have made necessary corrections to address the concerns raised by the reviewers.The revised manuscript incorporates the major corrections suggested by the reviewers. The following section outlines our response to each of the comments provided by the reviewers.

Reviewer #2:

1. The first reaction to the paper, is that the motivation is not clear. Sometimes authors talk about IoT communications, other times about generic air-to-ground support cellular networks for special situations such as disaster areas, or even about fast moving UAVs and 6G . All these use-cases are very different.

Response: For the motivation of this article, it discusses the possibility of signal interruption in traditional IoT communication in emergency situations and natural disasters. To address this issue, we introduce drones as mobile communication nodes. So as to maintain the normal communication of the entire Internet of Things. In reference [3], it is explained that fast drones, as one type of drone, are applied in 6G based IoT communication environments.

Modified place：We have made modifications to the introduction section Second Paragraph.

Unmanned aerial vehicles (UAVs) have the ability to create collaborative air-to-ground networks above disaster areas, regions with infrastructure challenges, or densely populated communication hotspots. They support and enhance existing cellular infrastructure to connect previously unconnected networks [1,2].Due to their high mobility, strong line-of-sight signal transmission, rapid deployment, and strong adaptability to various environments, UAVs play a critical role in Internet of Things. For instance, after Hurricane Ida hit Louisiana, AT\&T used UAV-mounted cellular on wings (COW) to restore LTE coverage for cellular users.[3] Moreover, He et al. utilized multiple UAVs to provide services to a wide range of users.[4] To extend the overall duration of a communication network, load balancing is implemented between adjacent UAVs, and resource consumption is shared, significantly increasing the overall coverage time of the UAV network. In addition, UAV cluster networks are a crucial component of air-to-ground collaborative networks and provide an essential foundation for achieving aerial networking and data transmission.[5,6]

2. More on the technical side, the role of federated learning is not clear. Authors talk about using it to ensure user privacy. Authors also talk about collaborative learning, to reduce communication costs. It is assumed to be of federated learning models, although this is not clearly mentioned in the paper.

Response: The role of federated learning: on the one hand, federated learning can reduce model training costs, and on the other hand, it can also protect user privacy. For collaborative learning, it can improve the transmission efficiency of federated learning and reduce the transmission of model parameters. Collaborative learning is one of the modes of distillation learning. The distillation learning mode adopted in this article is collaborative learning. In order to improve the efficiency of federated learning, distillation learning was introduced.

Modified place：We have made modifications to the introduction section Third Paragraph.

In the construction of a ground-based collaborative network for drone clusters, priority should be given to the privacy protection of user data. Federated Learning (FL) provides an effective method for training machine learning models while ensuring user privacy. In an FL system, clients have various computing and communication resources [7]. In recent years, researchers have been devoted to improving the communication efficiency of federated learning. One commonly used method is gradient compression, which directly reduces the size of model updates. Another widely adopted method is collaborative learning, where clients share local model predictions instead of transmitting model updates, thus reducing communication costs. Collaborative learning is one mode of distillation learning. For example, in literature [8], FL introduces Knowledge Distillation (KD) to achieve efficient and low-cost information exchange, especially when dealing with heterogeneous models, with the aim of reducing communication expenses.

 

3. Later on, authors mention that actually the goal of the paper is to apply air-to-ground collaborative networking and mobile edge computing to routing mechanisms. At this point it is not clear what does it means collaborative networking (is like cooperative relaying of opportunistic networks as found in the literature). Moreover the new topic, MEC, is not directly correlated with all the AI topics mentioned before, although it is not difficult to build that correlation. But it is not clear how  the authors propose to do it.

Later on, authors talk about intra-domain and multi-domain issues, which is new to the reasoning presented in the paper... Are authors talking about clustered networks? How does this related with all the previous topics?

Response: Regarding the above two issues. For collaborative networks, this article is designed as follows: Drones mainly provide an access service to ground nodes. Due to the wide coverage required for the service, the range of communication supported by drones is limited. Therefore, we adopt multiple drones for multi domain collaboration to achieve multi domain coverage. At the same time, the drone is equipped with a domain controller, which enables relay communication in the collaborative network. In addition, regarding the MEC section, as it is not closely related to the topic of this paper, MEC will not be mentioned in this paper.

Modified place：We have made modifications to the fourth paragraph of the introduction section.

In the design of air-to-ground collaborative networking, drones mainly provide an access service to ground nodes. Due to the wide coverage area required for the service, the communication range supported by drones is limited. Therefore, we use multiple drones for multi-domain collaboration to achieve multi-domain coverage. At the same time, the drones are equipped with domain controllers to achieve relay communication of the collaborative network through the domain controllers. The network management of distributed and heterogeneous nodes may be challenging. Moreover, due to the dynamic changes in network conditions, link interruptions can easily occur, making transmission reliability difficult to ensure. This paper focuses on the application of air-to-ground collaborative networking, and uses mobile edge computing technology to study network transmission strategies and routing mechanisms in highly dynamic networks. To address the issues of low intra-domain data volume and poor generalization ability of local decision models, a joint strategy is proposed to aggregate multi-domain data and accelerate intra-domain training, while protecting intra-domain privacy and avoiding network information leakage.

The main contributions of this paper are as follows:

• First, we design an air--ground collaborative network architecture for UAV-assisted networks. This architecture utilizes SDN's hierarchical multi-domain networking technology, with a super domain controller responsible for obtaining global network state information and cross-domain business requests. The domain controller is responsible for collecting network state information within its domain and aggregating both local and cross-domain business requests, ultimately enhancing the network's information processing capabilities and reducing communication latency.

4. Moreover it is not clear the difference between the method proposed in this paper and other papers about federated reinforcement distillation such as "Federated Reinforcement Distillation with proxy experience memory" in Arxiv Aprl 2020.

The paper needs a better statement about its motivation. It is difficult to follow the argumentation with so many different topics mentioned. From a technical point of view it would be good to have a clear distinction with other papers proposing federated learning distillation and the usage of Q-routing.

Response: The difference between this paper and "Federated Reinforcement Distillation with proxy experience memory" lies in the improvement of traditional federated reinforcement distillation in this paper. Instead of traditional distillation methods, knowledge distillation is employed to reduce communication costs.

Modified place：We have made modifications to the fifth paragraph of the algorithm model section.

Therefore, we have designed a federated reinforcement distillation [18] routing -- a distributed machine learning architecture that protects privacy and has high communication efficiency. We constructed a proxy for domain experience memory and exchanged data between the domain controller and the server. The proposed federated reinforcement distillation routing algorithm does not leak sensitive intra-domain data when combining multi-domain proxy experience memories, and reduces the amount of data that needs to be transmitted compared to traditional federated distillation algorithms, thus lowering communication costs.

Reviewer 3 Report

Comments and Suggestions for Authors

The article presents a UAV-based RL-based Federated Distillation scheme for routing.

The article's strength is the targeted topic and its timeliness.

The article's main limitations are the limited performance evaluation, statistical analysis, and novelty.

In particular, the presented results should include confidence intervals and other statistical analysis tools for any reported result. Otherwise, it is hard for the reader to convince themselves about the superiority of the proposed approach over the selected baselines. Furthermore, a broader range of scenario parameters should be explored, and more varied data visualization styles should be explored to strengthen the article's claims. 

The article's novelty and differences from the related works and other state-of-the-art studies should be further highlighted.

Consider removing Table 1 if the hardware specifications are not indispensable for interpreting the results. Any wall-clock metric should be possibly converted into a relative (hardware-independent) metric.

Comments on the Quality of English Language

The article's structure, conciseness, and storyline should be improved. At least 5 pages can be removed without any loss of important information.

Author Response

Response to Review Comments

We express our gratitude to the editor and reviewers for their insightful and constructive review comments.The feedback provided by the reviewers is critical in refining our paper and shaping the direction of our research.We have carefully reviewed the comments, and have made necessary corrections to address the concerns raised by the reviewers.The revised manuscript incorporates the major corrections suggested by the reviewers. The following section outlines our response to each of the comments provided by the reviewers.

Reviewer #3:

1. In particular, the presented results should include confidence intervals and other statistical analysis tools for any reported result. Otherwise, it is hard for the reader to convince themselves about the superiority of the proposed approach over the selected baselines. Furthermore, a broader range of scenario parameters should be explored, and more varied data visualization styles should be explored to strengthen the article's claims. 

Response： Thanks for the reviewer's suggestions. We really need to provide more explicit visual data analysis. Therefore, we add relevant visualizations to compare the proposed multi-domain intelligent routing algorithm with traditional federated reinforcement learning algorithms. So we added the corresponding visual image to better illustrate. As for the suggestion of confidence interval you mentioned, since the confidence interval is to describe the model accuracy rate, it is for the type of algorithm with clear correct labels. While the algorithm proposed in this paper for federated reinforcement distillation learning is not explicitly labeled. The algorithm proposed in this paper is an optimal strategy that maximizes the return value through continuous exploration. Therefore, confidence intervals cannot be used for evaluation.

Modified place：We added the following figure and text before the penultimate paragraph of 5.3 Performance Evaluation

Figure 18.The communication triffic between FRL and FRD

According to Figure 18, the multi-domain intelligent routing algorithm proposed in this paper can reduce communication volume by 25% - 33% compared to FRL. The results indicate that this algorithm reduces the amount of data that needs to be transmitted, improving model training performance.

 

2. The article's novelty and differences from the related works and other state-of-the-art studies should be further highlighted.

Response：Thanks for the reviewers' suggestions on the novelty of our paper, we think it is necessary to further emphasize the novelty, so we further explain the difference between the proposed algorithm and other algorithms

Modified place：We added the corresponding text before the last paragraph of the Related Work section

Since most of the existing studies focus on single UAV, and the communication coverage of single UAV is limited, it is not applicable to resource-constrained networks, and many studies on multi-UAV networks need the support of ground base stations. In disaster scenarios, ground base stations may be destroyed at any time, so the above studies are not applicable to disaster scenarios, due to the emergence of federal reinforcement learning. Protect user privacy and reduce communication costs. However, these studies are presented for their own scenarios, and do not consider the overall workload problem from the perspective of the amount of data transmitted parameters. Therefore, it does not apply to resource-constrained networks.

 

3. Consider removing Table 1 if the hardware specifications are not indispensable for interpreting the results. Any wall-clock metric should be possibly converted into a relative (hardware-independent) metric.

Response： Thank you for the suggestions provided by the reviewer. We believe that it is necessary to describe the hardware specifications, as different hardware devices will have different impacts on the final results. We also need to introduce what kind of hardware devices and software resources are needed to validate the algorithm proposed in this article, as there are many software devices available for this experiment. Therefore, we need to make a special note to facilitate the replication of the experiment for future generations

 

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

This paper proposed a federated reinforcement distillation based routing algorithm for UAV-assisted resilient networks to prevent communication infrastructure failures. The paper is in overall well-written with supporting numerical results. The following points can be considered to improve the academic quality of the paper. 

1) Regardless of many attempts to improve the algorithm, the absolute value of the improvement is quite small in the understandings of the reviewer. 

Federated reinforcement learning 63.6KB (Conventional)

Federated reinforcement distillation 45.3KB (Proposed)

While the communication speed had been improved much in recent years e.g. Mbps or even Gbps, the absolute value of the above data size is quite small and does not significantly affect communication latency etc. Please elaborate this point if the improvement of the proposed scheme is really worthwhile or not. 

2)  The paper said that it dealt with resilient networks to prevent communication infrastructure failures but the reviewer cannot find any numerical results explicitly investigated on the resiliency of the proposed mechanisms. 

Author Response

Response to Review Comments

We express our gratitude to the editor and reviewers for their insightful and constructive review comments.The feedback provided by the reviewers is critical in refining our paper and shaping the direction of our research.We have carefully reviewed the comments, and have made necessary corrections to address the concerns raised by the reviewers.The revised manuscript incorporates the major corrections suggested by the reviewers. The following section outlines our response to each of the comments provided by the reviewers.

Reviewer #4:

1. Regardless of many attempts to improve the algorithm, the absolute value of the improvement is quite small in the understandings of the reviewer. 

Federated reinforcement learning 63.6KB (Conventional)

Federated reinforcement distillation 45.3KB (Proposed)

While the communication speed had been improved much in recent years e.g. Mbps or even Gbps, the absolute value of the above data size is quite small and does not significantly affect communication latency etc. Please elaborate this point if the improvement of the proposed scheme is really worthwhile or not. 

Response： Thank you for the expert advice. Although the reduction rate is not particularly high, the scenario in this article is based on a resource-constrained network. Therefore, while the reduction rate may not be high compared to other scenarios, it is still a breakthrough development in resource-constrained environments. Therefore, the proposed solution in this article has certain practical value.

Modified place：We have made changes to the abstract section

Therefore, the FedRDR routing algorithm helps to facilitate knowledge transfer, accelerate the training process of intelligent agents within the domain, reduce communication costs in resource-constrained scenarios for UAV networks, and has practical value.

 

2. The paper said that it dealt with resilient networks to prevent communication infrastructure failures but the reviewer cannot find any numerical results explicitly investigated on the resiliency of the proposed mechanisms. 

Response： Firstly, thank you for the expert's input on elastic networks. The scenario we have considered previously is based on a self-organizing network using unmanned aerial vehicles (UAVs), which exhibit network elasticity due to their mobility. We appreciate the expert's suggestions, and as a result, we have decided to change the title to: "FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures".

Modified place：We have made changes to the title

FedRDR: Federated Reinforcement Distillation-Based Routing Algorithm in UAV-Assisted Networks for Communication Infrastructure Failures

 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The paper is still very confusing tackling a large set of technologies, each one of which with their own challenges, which are not properly considered in the paper, namely in the areas of:

- IoT

- Air-to-ground UAV communication

- Network clustering

- Federated learning

- Cooperative learning

- 6G

For instance, the paper never mention how UAV clusters are created and maintained, what routing is used to keep UAVs connected inside and between clusters.  Authors mention that they analysed  depth first search (DFS) algorithm and Q-learning-based routing algorithm using reinforcement learning. Are these routing algorithms used inside or between clusters? Why were these routing algorithms selected? When introducing DDQN, authors mention that it is used to compute routing decision, but no references to intra-cluster or inter-cluster operations. Later on, before section 5, authors mention that DDQN is used for intra-domain routing. What is the algorithm used for inter-cluster, and how do the two cooperate?

In what concerns performance, authors mention that Q-learning requires more time to converge than DDQN, but this statement is not proved. For an algorithm that is based on learning based on a significant amount of data, it is important to understand how fast is it to update a network topology, in an way that is efficient to react to the need to forward packets in very short time windows (e.g. ms).

Authors do not provide any information about what 6G properties are considered in the paper. What 6G architecture is assumed in the envisioned scenario, including considerations about the UAV to BS (air-to-ground) links.

Comments on the Quality of English Language

Could be improved, but it is not a major limitation.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf