Scalable and Cooperative Deep Reinforcement Learning Approaches for Multi-UAV Systems: A Systematic Review

Round 1

Reviewer 1 Report

Scalable and Cooperative Deep Reinforcement Learning

Approaches for Multi-UAV Systems: A Systematic Review

 

1. The main question addressed by the research

- Deep Reinforcement Learning (DRL) techniques for cooperative and scalable multi-UAV systems;

- multi-UAV systems;

- comparative analysis of cooperative DRL-based multi-UAV systems;

 

2. Relevance and the originality of the topic

- the topic is relevant in the field of UAV;

- the manuscript is clear and well structured;

 

3. The added value of the article compared with other published material

- there are a lot of applications already published for multi-UAV systems. Some of those applications were presented also by the authors in the introduction;

- the challenge and the identified gap in the field is the cooperative multi-UAV system related to Deep Reinforcement Learning;

 

4. The methodology

- the main methodology used is the multi-agent Reinforcement Learning.

- the results are reproductible based on the presented information;

 

5. The figures and tables

- the data presented in the figures are relevant and are well interpreted by the authors;

 

6. The conclusions

- the conclusions are consistent in the field of the Multi-UAV Systems;

 

7. The references

- the used references are recently  and relevant in the field;

Author Response

Dear Reviewer,
Thanks for the valuable comments. We really appreciated your positive review.

Reviewer 2 Report

please consisder the critics

Comments for author File: Comments.pdf

Author Response

Dear Reviewer,
Thanks for the valuable comments and suggestions. Below we report the answers to your questions:

1. We added more horizontal lines in figure 1 for a better understanding, and additionally, we added the number of published papers per year on top of each bar.
2.  We numbered the formulas.
3.  The diagram in figure 7 was generated through a software called Litmaps, and thus it is not possible to modify it further to make it more readable. However, a detailed description of figure 7 is already provided in the caption of the figure and the "Discussion" section. 
4.  The main results are described in detail in the latest sections. However, we added the following sentence in the abstract: "The suggested lines of research help improve the sim-to-real knowledge transferability and the UAV systems responsiveness and safety. "
5.  We added the missing abbreviations associated with the indicated algorithms. Thanks for pointing it out. 
6.  For what concerns the suggested paper, we cannot include them inside the uploaded manuscript since they are outside of the scope of our review:
   1. The first paper refers to the improvement of a single hexarotor UAV without using any deep reinforcement learning techniques;
   2. The second paper suggested is about the design of a single HUAV, and it is not DRL-based;
   3. The goal of the third paper suggested is simultaneously determining the optimum shape of blade tip swept and anhedral. The article deals with a single agent, and it is not DRL-based;

Reviewer 3 Report

This research paper is a review paper.

It is meaningful in that it systematically reviewed the approach to multiple UAV systems.

 

I will give my opinion in minor parts.

 

However, when reviewing the thesis as a whole, there are parts that are listed at length. This should be summarized in a more concise style.

Such is the nature of review papers, but they are quite verbose, and it is difficult for the reader to determine which parts are being emphasized for review.

 

In addition, it would be good to suggest the direction in which research should be conducted in relation to multiple UAV systems. It would be nice to present that part in the conclusion.

 

Review the wording, etc. throughout the paper.

 

Also, it would be nice to add suggestions to overcome the limitations.

Author Response

Dear Reviewer,
Thanks for the valuable comments and suggestions. Below we report the answers to your questions:
Q: However, when reviewing the thesis as a whole, there are parts that are listed at length. This should be summarized in a more concise style.
Such is the nature of review papers, but they are quite verbose, and it is difficult for the reader to determine which parts are being emphasized for review.
A: For each multi-UAV application, we briefly described each paper but focusing only on the most relevant among them for the considered application. The distinction between the most relevant articles and the minor ones (belonging to the same class) is already highlighted by a new line and introductive sentences such as "more appreciable modifications to well-known algorithms can be found instead in [...]." 
For a general and quick overview, you can refer to the summarizing tables present in each application class. A further synthesis of the works could lead to an incomplete and not fully understandable description of the analyzed articles, which deal with complex topics.

Q: In addition, it would be good to suggest the direction in which research should be conducted in relation to multiple UAV systems. It would be nice to present that part in the conclusion.
A: We added the following part in the conclusion of the last version of the manuscript:
"In order to satisfy these systems' requirements, we suggested directing the future works towards mainly solutions which are not fully centralized but cooperative, and that explicitly consider the delay in the algorithm design: the former ensures the system scalability, safety, relatively low computational resources, and the independence from a single central control unit; while the latter eases the sim-to-real knowledge transferability process."
A more detailed description is provided in the "Discussion" section

Q: Review the wording, etc., throughout the paper.
 A: We reviewed the wording by modifying the sentences where needed

Q: Also, it would be nice to add suggestions to overcome the limitations.
A: The suggestions for overcoming the limitations are present in the "Discussion" section, and we added them in the "Conclusion" section: some possible solutions are cooperative and not fully centralized approaches and algorithms considering the delays.