Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph Attention Network for UAV Swarms

Abstract

Multiple unmanned aerial vehicles (Multi-UAV) systems have recently demonstrated significant advantages in some real-world scenarios, but the limited communication range of UAVs poses great challenges to multi-UAV collaborative decision-making. By constructing the multi-UAV cooperation problem as a multi-agent system (MAS), the cooperative decision-making among UAVs can be realized by using multi-agent reinforcement learning (MARL). Following this paradigm, this work focuses on developing partially observable MARL models that capture important information from local observations in order to select effective actions. Previous related studies employ either probability distributions or weighted mean field to update the average actions of neighborhood agents. However, they do not fully consider the feature information of surrounding neighbors, resulting in a local optimum often. In this paper, we propose a novel partially multi-agent reinforcement learning algorithm to remedy this flaw, which is based on graph attention network and partially observable mean field and is named as the GPMF algorithm for short. GPMF uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step. The graph attention module consists of a graph attention encoder and a differentiable attention mechanism, outputting a dynamic graph to represent the effectiveness of neighborhood agents against central agents. The mean field module approximates the effect of a neighborhood agent on a central agent as the average effect of effective neighborhood agents. Aiming at the typical task scenario of large-scale multi-UAV cooperative roundup, the proposed algorithm is evaluated based on the MAgent framework. Experimental results show that GPMF outperforms baselines including state-of-the-art partially observable mean field reinforcement learning algorithms, providing technical support for large-scale multi-UAV coordination and confrontation tasks in communication-constrained environments.

1. Introduction

The Multi-UAV system [1] has gained significant traction in various domains in recent years, in which multiple unmanned aerial vehicles collaborate to accomplish tasks. It has been widely applied in many fields, including Multi-UAV transportation, Multi-UAV area search and Multi-UAV formation flying [2]. It is noteworthy that a Multi-UAV cooperative transportation system [3,4,5] has been developed and implemented by our research group following the trend.

Multi-agent systems (MAS) [6,7] consist of multiple entities called agents that interact in a shared environment aiming to achieve some individual or collective objective. Multi-UAV system can be considered as a specific type of MAS. In a Multi-UAV system, each UAV is treated as an intelligent agent that perceives the environment, makes decisions, and executes actions to collaboratively accomplish tasks. Therefore, the theories and methods in MAS research can provide guidance for the coordinated control of Multi-UAV systems. Among them, reinforcement learning, as a widely applied approach, can be used to achieve decision-making and collaboration among agents. Recently, reinforcement learning [8] has found widespread applications in various domains, including video games [9,10,11], task division [12], and education [13]. Multi-Agent Reinforcement Learning (MARL) [14] is an extension of reinforcement learning in MAS, and it has been applied in some real-world scenarios such as autonomous mobile [15,16] and multi-UAV collaborative scenarios. Shi et al. [3] successfully migrated MARL to real-world multi-UAV handling tasks by employing Recurrent Multi-Agent Deep Deterministic Policy Gradient (R-MADDPG) and domain randomization techniques. Liu et al. [17] proposed the Extensible Multi-Agent Deep Deterministic Policy Gradient (Ex-MADDPG) algorithm to address dynamic task assignment problems in UAV swarms, in which the practicality and effectiveness of the Ex-MADDPG algorithm were validated using a swarm of nine UAVs.

For large-scale MAS [18], it is unrealistic for agents to observe the entire environment globally limiting their ability to find appropriate actions. Furthermore, as the number of agents increases, joint optimization of all agents in a multi-agent environment may result in a huge joint state-action space, which also brings scalability challenges. Therefore, traditional MARL is difficult to apply to large-scale multi-agent environments, especially when the number of agents increases exponentially. To tackle this drawback, recent studies address the scalability issues of multi-agent reinforcement learning [19,20,21] by introducing mean field theory, i.e., the multi-agent problem is reduced to a simple two-agent problem. However, Yang et al. [19] assumes that each agent can observe global information, which is unreasonable in real-world MAS. As shown in Figure 1, multiple UAVs cooperate to perform search for fire detection, in which each UAV cannot get the global environment information, but it can obtain the information of other UAVs within the limited communication range. Thus, the adaptive communication strategy are required in the multi-UAV tasks such as collaborative search and rescue mission, which are typical partially observable scenarios.

Therefore, it is necessary to study large-scale multi-agent reinforcement learning algorithms in partially observable cases [22]. In addition, researchers have intensively studied mean field-based multi-agent reinforcement learning algorithms to improve performance in partially observable cases. One way is to further decompose the Q-function of the mean field-based multi-agent reinforcement learning algorithm [23,24]. Another way uses probability distribution or weighted mean field to update the mean action of neighborhood agents [25,26,27,28]. The graph attention and mean field mechanism are also utilized in [29], which measure the interaction strength between agents with fixed relative positions and global information. Differently, we consider the dynamic changes of the agent’s position and the death scene of the agent and construct a more flexible partial observable graph attention network based on the mean field theory for partially observable scenarios.

This paper focuses on identifying the neighborhood agents that may have the greater influence on the central agent in a limited observation space, in order to avoid the local optimum issue. Since the graph neural network [30,31,32] can fully aggregate the relationship between the central agent and its surrounding neighbors, we propose a graph attention-based mechanism to calculate the importance of neighbor agents to estimate the average actions more efficiently.

The main contributions of this paper are as follows:

• We propose a Graph attention network supported Partially observable Mean Field Multi-agent reinforcement learning (GPMF) algorithm, which can learn a decentralized agent policy from an environment without requiring global information of an environment. This is particularly valuable in scenarios with large-scale agents and limited observability, where existing methods lack sufficient judgment of the importance of neighbor agents.
• We provide theoretical evidence that the settings of the GPMF algorithm approach Nash equilibrium, showcasing its sound theoretical foundation.
• Experiments on three challenging game tasks in the MAgents framework show that GPMF outperforms two baseline algorithms as well as the state-of-the-art partially observable mean field reinforcement learning algorithms. These results validate the effectiveness of GPMF in solving dynamic cooperation problems in multi-UAV scenarios and facilitating collaboration among UAVs.

2. Related Work

2.1. Multi-UAV System Based on Multi-Agent Reinforcement Learning

The development of multi-agent reinforcement learning provides new solutions for multi-UAV missions. Qie et al. [33] formulated the Multi-UAV Target Assignment and Path Planning (MUTAPP) problem as a multi-agent system, and adopted the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to solve the MUTAPP problem. For large-scale multi-UAV systems, Azzam et al. [34] developed MARL-based cooperative navigation of UAV swarms via centralized training and decentralized execution (CTDE), and the method was extended to work with a large number of agents, without retraining or changing the number of agents during training. Wang et al. [35] proposed a novel MARL paradigm, weighted mean field reinforcement learning, and conducted experiments on a large-scale UAV swarm confrontation environment to verify the effectiveness and scalability of the method. However, most of the group goals of existing MARL-based multi-UAV cooperation are based on a single cooperation or confrontation environment, and the complex environment of cooperation and confrontation has not been fully considered. In addition, the UAV observation range in the real scene is limited, so it is necessary to consider the solution of large-scale multi-UAV system cooperation and confrontation under the limitation of communication.

2.2. Large-Scale Partially Observable Multi-Agent Reinforcement Learning

Large-scale partially observable multi-agent reinforcement learning can provide technical support for the realization of large-scale multi-UAV tasks under communication constraints. For large-scale multi-agent environments, Yang et al. [19] introduced the mean–field theory, which approximates the interaction of many agents as the interaction between the central agent and the average effects from neighboring agents. However, partially observed multi-agent mean–field reinforcement learning algorithms still have a space to improve. Some researchers further decompose the Q-function of the mean field based multi-agent reinforcement learning algorithm. Zhang et al. [23] trained agents through the CTDE paradigm, transforming each agent’s Q-function into its local Q-function and its mean field Q-function, but this approach is not strictly partially observable. Gu et al. [24] proposed a mean field multi-agent reinforcement learning algorithm with local training and decentralized execution. The Q-function is decomposed by grouping the observable neighbor states of each agent in a multi-agent system, so that the Q-function can be updated locally. In addition, some researchers have focused on improving the mean action in mean field reinforcement learning. Fang et al. [25] added the idea of mean field to MADDPG, and proposed a multi-agent reinforcement learning algorithm based on weighted mean field, so that MADDPG can adapt to large-scale multi-agent environment. Wang et al. [28] proposed a weighted mean field multi-agent reinforcement learning algorithm based on reward attribution decomposition by approximating the weighted mean field as a joint optimization of implicit reward distribution between the central agent and its neighbors. Zhou et al. [26] used the mean action of neighbor agents as a label, and trained a mean field prediction network to replace the mean action. Subramanian et al. [27] proposed two multi-agent mean field Q-learning algorithms based on partially observable (POMFQ) settings: Fixed Observation Radius (FOR) and Probabilistic Distance-based Observability (PDO), extracting partial samples from Dirichlet or Gamma distribution to estimate partially observable mean action. Although these methods achieved good results, they did not fully consider the feature information of surrounding neighbors.

2.3. Graph Neural Networks (GNNs)

Graph Neural Networks (GNNs) are able to mine graph structures from data. In multi-agent reinforcement learning, GNNs can be used to model interactions between agents. In recent works, graph attention mechanism has been used for multi-agent reinforcement learning. Zhang et al. [36] integrated the importance of the information of surrounding agents based on the multi-head attention mechanism, effectively integrated the key information of the graph to represent the environment and improved the cooperation strategy of agents with the help of multi-agent reinforcement learning. DCG [37] decomposed the joint value function of all agents into gains between pairs of agents according to the coordination graph, which can flexibly balance the performance and generalization ability of agents. Li et al. [38] proposed a deep implicit coordination graph (DICG) structure that can adapt to dynamic environments and learn implicit reasoning about joint actions or values via graph neural networks. Ruan et al. [39] proposed a graph-based coordination strategy, which decomposes the joint team strategy into a graph generator and a graph-based coordination strategy to realize the coordination behavior between agents. MAGIC [40] accurately represented the interactions between agents during communication by modifying the standard graph attention network and compatible with differentiable directed graphs.

Our approach differs from related works above in that it uses a graph attention mechanism to select surrounding agents that are more important to the central agent in a partially observable environment. GPMF uses a graph attention module and a mean field module to describe how an agent is influenced by the actions of other agents at each time step, where graph attention consists of a graph attention encoder and a differentiable attention mechanism, and finally outputs a dynamic graph to represent the effectiveness of the neighborhood agent to the central agent. The mean field module approximates the influence of a neighborhood agent on a central agent as the average influence of the effective neighborhood agents. Using these two modules together is able to efficiently estimate the mean action of surrounding agents in partially observable situations. GPMF does not require global information about the environment to learn decentralized agent policies from the environment.

3. Motivation and Preliminaries

In this section, we model discrete-time non-cooperative multi-agent task as a stochastic game (SG). SG can be defined as a tuple $(S,A^1,\dots ,A^N,r^1,\dots ,r^N,p,\gamma )$, where S represents the true states of the environment and $A^j$ is the set of actions of the j-th agent. The reward function for agent j is $r^j:S\times A^1\times \cdots \times A^N\to R$. State transitions are specified as $p:S\times A^1\times \cdots \times A^N\to \Omega \left(S\right)$. $\gamma$ is the discount factor, that represents the importance of future rewards. It is a constant between 0 and 1. The disadvantage is that it cannot be applied to the coexistence of multiple agents. Yang et al. [19] introduced mean field theory, which approximates the interaction of many agents as the interaction between the average effect of a central agent and neighboring agents, and solves the scalability problem of SG.

The Nash equilibrium of general and random games can be defined as a strategy tuple $\left({\pi }_{*}^1,\cdots ,{\pi }_{*}^N\right)$, for all $s\in S$ and $\forall {\pi }^i\in {\Pi }^i$, there is $v^j\left(s,{\pi }_{*}^1,\cdots ,{\pi }_{*}^i,\cdots ,{\pi }_{*}^N\right)\ge v^j\left(s,{\pi }_{*}^1,\cdots ,{\pi }^i,\cdots ,{\pi }_{*}^N\right)$. This shows that when all other agents are implementing their equilibrium strategy, no one agent will deviate from this equilibrium strategy and receive a strictly higher reward. When all agents follow the Nash equilibrium strategy, the Nash Q-function of agent j is $Q_{*}^j(s,a)$. Partially observable stochastic games can generate a partially observable Markov decision process (POMDP), we review POMDP in Section 3.1 and analyze the partially observable model from a theoretical perspective. Section 3.2 first introduces the globally observable mean field multi-agent reinforcement learning, and then introduces the partially observable mean field Q-learning algorithm (POMFQ) based on the POMDP framework, and analyzes the existing part of the observable in detail. The limitation of mean field reinforcement learning POMFQ(FOR) [27] is that the feature informations of surrounding neighbors are not fully considered. In a partially observable setting, the observable neighborhood agent information $o^j$ of agent j can be used to better mine the relationship between features through a graph attention network. Introducing graph attention networks into partially observable mean field multi-agent reinforcement learning can further improve their performance, and Section 3.3 briefly describes graph attention networks.

3.1. Partially Observable Markov Decision Process

We mainly study Partially Observable Markov Decisions Process (POMDP) [14,22,41,42]. The POMDP of n agents can be represented as a tuple $〈N,S,{\left\{A^i\right\}}_{i=1}^n,T,Z,R,O,\gamma 〉$, where $N=\{1,\dots ,n\}$ represents the set of agents, S denotes the global state, $A^j$ is the set of action spaces of the j-th agent, Z represents the observation space of the agents, and the agent j receives observation $o^j\in O^j$ through the observation function $Z(s,j):S\times N\to O$, and the transition function $T:S\times A^1\times \dots \times A^N\times S\mapsto [0,1]$ represents the environment transitions from a state to another one. At each time step t, the agent j chooses an action $a_t^j\in A^j$, gets a reward $r_t^j:S\times A^j\mapsto R$ w.r.t. a state and an action. $\gamma \in [0,1]$ is a reward discount factor. Agent j has a stochastic policy ${\pi }^j$ conditioned on its observation $o^j$ or action observation history ${\tau }^j\in \left(Z\times A^j\right)$, and according to the all agents’ joint policy $\pi \stackrel{\Delta }{=}\left[{\pi }^1,\dots ,{\pi }^N\right]$, the value function of agent j under the joint strategy $\pi$ is $v_{\pi }^j\left(s\right)={\sum }_{t=0}^{\infty }{\gamma }^t{E}_{\pi ,p}\left[r_t^j|s_0=s\right]$, and then the Q-function can be formalized as $Q_{\pi }^j(s,a)=r^j(s,a)+\gamma E_{s^{\prime }\sim p}\left[v_{\pi }^j\left(s^{\prime }\right)\right]$. Our work is based on the POMDP framework.

3.2. Partially Observable Mean Field Q-Learning

Mean field reinforcement learning [19] approximates interactions among multiple agents as two-agent interactions, where the second agent corresponds to the average effect of all other agents. Yang et al. [19] decomposes the multi-agent Q-function into pairwise interacting local Q-functions as follows:

$$Q_{\pi }^j(s,a)=\frac{1}{N^j}\sum _{k\in N\left(j\right)}{Q}_{\pi }^j\left(s,a^j,a^k\right)$$

(1)

where $N^j$ is the index set of the neighbors of the agent j and $a^j$ denotes the discrete action of the agent j and is represented by one-shot coding. Mean field Q-function is cyclically updated according to Equations (2)–(5):

$$Q_{\pi }^j\left(s_t,a_t^j,{\overline{a}}_t^j\right)=(1-\alpha )Q_{\pi }^j\left(s_t,a_t^j,{\overline{a}}_t^j\right)+\alpha \left[r_t^j+\gamma v^j\left(s_{t+1}\right)\right]$$

(2)

where

$$v^j\left(s_{t+1}\right)=\sum _{a_{t+1}^j}{\pi }^j\left(a_{t+1}^j\mid s_{t+1},{\tilde{a}}_t^j\right)Q_{\pi }^j\left(s_{t+1},a_{t+1}^j,{\tilde{a}}_t^j\right)$$

(3)

$${\overline{a}}_t^j=\frac{1}{N}\sum _{k\ne j}{a}_t^k,a_t^k\sim {\pi }^k\left(·\mid s_t,{\overline{a}}_{t-1}^k\right)$$

(4)

$${\pi }^j\left(a_t^j\mid s_t,{\overline{a}}_{t-1}^j\right)=\frac{exp\left(-\beta Q_{\pi }^j\left(s_t,a_t^j,{\overline{a}}_{t-1}^j\right)\right)}{\sum _{a_t^{j^{\prime }}\in A^j}exp\left(-\beta Q_{\pi }^j\left(s_t,a_t^{j^{\prime }},{\overline{a}}_{t-1}^j\right)\right)}$$

(5)

where ${\overline{a}}_t^j$ is the mean action of the neighborhood agents, $r_t^j$ is the reward for agent j at time step t, $v^j$ is the value function of agent j, and $\beta$ is the Boltzmann parameter. Yang et al. [19] assumed that each agent has global information, and for the central agent, the mean action of the neighboring agents is updated by Equation (4). However, in a partially observable multi-agent environment, the way of calculating the mean action in Equation (4) is no longer applicable. In the case of partial observability, Subramanian et al. [27] take U samples from the Dirichlet distribution to update the mean action of Equation (4), and achieve better performance than the mean field reinforcement learning algorithm. The formula is as follows:

$$D^j\left(\theta \right)\propto {\theta }_1^{{\eta }_1-1+c_1}\dots {\theta }_L^{{\eta }_L-1+c_L};{\tilde{a}}_{i,t}^j\sim D^j(\theta ;\eta +c);{\tilde{a}}_t^j=\frac{1}{U}\sum _{i=1}^{i=U}{\tilde{a}}_{i,t}^j$$

(6)

where L denotes the size of the action space, $c_1,\dots ,c_L$ denotes the number of occurrences of each action, $\eta$ is the Dirichlet parameter, $\theta$ is the classification distribution. But the premise of the Dirichlet distribution is to assume that the characteristics of each agent are independent to achieve better clustering based on the characteristics of neighboring agents. In fact, in many multi-agent environments, the characteristics of each agent have a certain correlation, but the Dirichlet distribution does not consider this correlation, which makes it unable to accurately describe the central agent and the neighborhood agents. There will be some deviations in the related information. Figure 2 shows a multi-robot system, each robot has a limited observation range and does not know all the information of the environment. The goal of the robot is to find the exit by observing the state of the surrounding robots. The action space of the agent is $\{up,down,left,right\}$. A central agent surrounded by a red circle is influenced by surrounding agents. We use the Dirichlet distribution to simulate and calculate the probability of the central agent moving in all directions, where the Dirichlet parameter $\eta$ is set to 1, and the result is as follows:

$$\left\{\begin{array}{c}{p}_{up}=0.25\hfill \\ p_{down}=0.5\hfill \\ p_{left}=0.143\hfill \\ p_{right}=0.1\hfill \end{array}\right.$$

(7)

It can be seen that the probability of the agent moving $down$ is the highest, which is essentially due to a large number of agents moving to $down$, so the robot is more inclined to move in the direction of exit 3. However, the closest exit to it is actually exit 1, and moving $up$ is the best move. The Dirichlet distribution results in a local optimal solution rather than finding the optimal action.

Zhang et al. [43] believe that the correlation between two agents is crucial for multi-agent reinforcement learning. First, the paper [43] calculated the correlation coefficient between each pair of agents, and then shielded the communication among weakly correlated agents, thereby reducing the dimensions of the state-action value network in the input space. Inspired by Zhang et al. [43], for large-scale partially observable multi-agent environments, it is more necessary to select important neighborhood agents. In our paper, we adopt a graph attention network to filter out more important neighborhood agents, discard unimportant agent information, and achieve more accurate estimation of the mean action of neighborhood agents.

3.3. Graph Attention Network

Graph neural network [30] can better mine the graph structure form among data elements. Graph attention network (GAT) [44] is composed of a group of graph attention layers, which model the effect of neighbor nodes on central node. The attention score can be computed as Equations (8) and (9).

$$e_{ij}=(W{m}_i\parallel W{m}_j)$$

(8)

$${\alpha }_{ij}=\mathrm{softmax}j\left(e_{ij}\right)=\frac{exp\left(e_{ij}\right)}{{\displaystyle \sum _{k\in N_i}}exp\left(e_{ik}\right)}$$

(9)

where $e_{ij}$ is the attention coefficient of each pair of nodes, indicating the importance of node j to node i. Finally, the output features are obtained by weighting the input features $h_j$, and the update rule for each node i is:

$$e_i=\sigma \left(\sum _{j\in N_i}{\alpha }_{ij}W{h}_j\right)$$

(10)

where $e_i$ represents the feature of node i, $N_i$ is the set of adjacent nodes of node i, and $\sigma (·)$ is a nonlinear activation function.

4. Approach

In this section, we propose a Graph attention network supported Partially observable Mean Field Multi-agent reinforcement learning (GPMF) algorithm, which can be applied to large-scale partially observable multi-UAV cooperative pursuit tasks, where he communication range of each UAV is limited, and it can only observe the characteristic information of other UAVs within a fixed range. The multi-UAV cooperative pursuit problem is essentially a mixed scenario of competition and cooperation. Competition means that each UAV competes for resources to maximize rewards, while cooperation means that UAVs cooperate to achieve a common goal. Incorporating the multi-UAV cooperative pursuit problem into a multi-agent system (MAS), as shown in Figure 3, each UAV is abstracted as an agent, and then trained in a dynamic environment based on our proposed GPMF framework. The overall architecture of the GPMF algorithm is depicted in Figure 4. It has two important components: the Graph Attention Module and the Mean Field Module. In the Graph Attention Module, the information observed locally by each agent is spliced firstly. Then the high-dimensional feature representations are obtained by a latent space mapping process which followed by a one-layer LSTM network to obtain the time-series correlation of the target agent, and the hidden layer of the LSTM is used as the input of the graph attention module to initialize the constructed graph nodes. Then to enhance the aggregation of neighbor agents to target agent, a similar process is implemented as a FC mapping network followed by a GAT layer. After that, the final representation of agents are obtained by a MLP layer with the input of the representations of target agent and other observable agents. Finally, we adopt layer-normalized method to obtain the adjacency matrix ${\left\{G^t\right\}}_1^N$ via Gumbel Softmax. In the Mean Field Module utilizes the adjacency matrix ${\left\{G^t\right\}}_1^N$ from Graph Attention Module to obtain adopting action from important neighbor agents, in which the joint Q-function of each agent j approximates the Mean Field Q-function $Q^j(s,a)\approx Q_{\mathrm{POMF}}^j\left(s,a^j,{\tilde{a}}^j\right)$ of important neighbor agents, where the Q-value is Partially Observable Mean Field(POMF) Q-value, and ${\tilde{a}}^j$ is the mean action of the important neighborhood agents that are partially observed by agent j. Each component is described in detail below.

4.1. Graph Attention Module

To more accurately determine the influence of agent j’s neighbor $N_j$ on itself, we need to be able to extract useful information from the local observations of agent j. The local observations of each agent include the embedding information of neighboring agents. For each agent j and each time step t, the information of a local observation of length $L_j$ is expressed as $o_j^t=\left(x_1^t,x_2^t,\cdots ,x_{N_j}^t\right)$, where $x_{N_j}^t$ represents the feature of the $N_j$-th neighbor agent of agent j, and $o_j^t\in R^{N_j\times D}$, $x_i^t\in R^{1\times D}$. $L_j$ is concatenation of the embedding features of each neighbor. Our goal is to learn an adjacency matrix ${\left\{G^t\right\}}_1^N$ to extract more important embedding information for the agent j from local observations at each time step t. Since graph neural networks can better mine the information of neighbor nodes, we propose a graph attention structure suitable for large-scale MAS. This structure focuses on information from different agents by associating weights with observations based on the relative importance of other agents in their local observations. The graph attention structure is constructed by overlaying a graph attention encoder and a differentiable attention mechanism. For the local observation $o_j^t$ of agent j at time step t, $o_j^{t^{\prime }}$ is first encoded using a fully connected layer (FC), and is passed to the LSTM layer to generate the hidden state $h_j^t$ and cell state $c_j^t$ of agent j, where $h_j^t$ serves as the input of the graph attention module to initialize the constructed graph nodes:

$$h_j^t,c_j^t=LSTM\left(e\left(o_j^t\right),h_j^t,c_j^t\right)$$

(11)

where $e(·)$ is a fully connected layer representing the observed encoder. $h_j^t$ is encoded as a message:

$$m_j^t=e\left(h_j^t\right)$$

(12)

where $m_j^t$ is the aggregated information of the neighborhood agents observed by agent j at time step t. The input encoding information $M^t$ is passed to the GAT encoder and hard attention mechanism, where the hard attention mechanism consists of MLP and Gumbel Softmax function. Finally, the output adjacency matrix ${\left\{G^t\right\}}_1^N$ is used to determine which agents in the neighborhood have an influence on the current agent. The GAT encoder helps to efficiently encode the agent’s local information, which is expressed as:

$${\left\{M^t\right\}}_1^N=f_{Sched}\left(m_1^t,\cdots ,m_N^t\right)$$

(13)

Additionally, we take the form of the same attention mechanism as GAT [44], expressed as follows:

$${\alpha }_{ij}^S=\frac{exp\left(Leaky\mathrm{Re}LU\left(a_S^T\left[W_S{m}_i^t\left|\right|W_S{m}_j^t\right]\right)\right)}{{\displaystyle \sum _{k\in N_j^t\cup \left\{j\right\}}}exp\left(Leaky\mathrm{Re}LU\left(a_S^T\left[W_S{m}_j^t\left|\right|W_S{m}_k^t\right]\right)\right)}$$

(14)

where $LeakyReLU(·)$ is the activation function, $a_S\in R^D$ is the weight vector, $N_j^t\cup \left\{j\right\}$ represents the central agent j and its observable neighborhood agent set, and $W_S\in R^{D\times D}$ is the weight matrix. The node feature of agent j is expressed as:

$$e_j^t=ELU\left(\sum _{i\in N_j^t\cup j}{\alpha }_{ij}^S{W}_S{m}_i^t\right)$$

(15)

where $ELU(·)$ is an exponential linear unit function. Connecting the features of each node in pairs: $E_{i,j}^t=\left(e_i^t\left|\right|e_j^t\right)$, we can get a matrix $E^t\in R^{N\times N_j\times 2D}$, where $E_{i,j}^t$ represents the relevant features of agent j. Taking $E^t$ as the input of MLP, which is followed by a Gumbel Softmax function, then the connected vector $G_j^t$ can be obtained. The connected vector $G_j^t$ consists of elements $g_{ij}$, where i represents the neighbors of the central agent j. The element $g_{ij}^t=1$ in the adjacency matrix indicates that the action of the agent i will have an impact on the agent j. Conversely, $g_{ij}^t=0$ means that the agent’s actions have no effect on the agent j.

4.2. Mean Field Module

This Graph Attention Module selects important $M_j$ agents from the neighbors $N_j$ of agent j, and compute the mean of the actions of the choosed neighbor agents:

$${\tilde{a}}_t^j=\frac{1}{M_j}\sum _{k\in N_j}{a}_t^k·G_j^t,a_t^k\sim {\pi }^k\left(·\mid s_t,{\tilde{a}}_t^k\right)$$

(16)

where · is the element-wise multiplication.

In the above formula, $a^k$ represents the important neighborhood agent for agent j. Then the Q-value of each agent is shown in Equation (17). Note that the Q-value here is a partially observable Q-value.

$$Q_{\mathrm{GPMF}}^j\left(s_t^j,a_t^j,{\tilde{a}}_t^j\right)=(1-\alpha )Q_{\mathrm{GPMF}}^j\left(s_t^j,a_t^j,{\tilde{a}}_t^j\right)+\alpha \left[r_t^j+\gamma v\left(s_{t+1}^j\right)\right]$$

(17)

where the value function $v^j$ is expressed as

$$v^j\left(s_{t+1}^j\right)=\sum _{a_{t+1}^j}{\pi }^j\left(a_{t+1}^j\mid s_{t+1}^j,{\tilde{a}}_t^j\right)Q_{\mathrm{GPMF}}^j\left(s_{t+1}^j,a_{t+1}^j,{\tilde{a}}_t^j\right)$$

(18)

According to the above graph attention mechanism, more important neighborhood agents are obtained. The new mean action ${\tilde{a}}_t^j$ is calculated by Equation (16), and then the strategy ${\pi }_t^j$ of agent j is updated by the following formula:

$${\pi }^j\left(a_t^j\mid s_t^j,{\tilde{a}}_{t-1}^j\right)=\frac{exp\left(-\beta Q_{\mathrm{GPMF}}^j\left(s_t^j,a_t^j,{\tilde{a}}_{t-1}^j\right)\right)}{{\displaystyle \sum _{a_t^{j^{\prime }}\in A^j}}exp\left(-\beta Q_{\mathrm{GPMF}}^j\left(s_t^j,a_t^{j^{\prime }},{\tilde{a}}_{t-1}^j\right)\right)}$$

(19)

4.3. Theoretical Proof

This subsection is devoted to proving that the setting of GPMF is close to the Nash equilibrium [45]. Subramanian et al. [27] showed that in partially observable cases, the fixed observation radius (FOR) setting is close to a Nash equilibrium, where the mean action of each agent’s neighborhood agents is approximated by a dirichlet distribution. First, we state some assumptions, which are the same as literature [27], and are followed by all the theorems and analyses below.

Assumption 1.

For any i and j, there is ${lim}_{t\to \infty }{\tau }_j^i\left(t\right)=\infty$$w.p.$1.

This assumption guarantees a probability of 1 that old information is eventually discarded.

Assumption 2.

Suppose some measurability conditions are as follow: (1) $x\left(0\right)$ is $\mathcal{F}\left(0\right)$-measurable. (2) For each i, j and t, $w_i\left(t\right)$ is $\mathcal{F}(t+1)$-measurable. (3) For each $i,j$ and t, ${\alpha }_i\left(t\right)$ and ${\tau }_j^i\left(t\right)$ are $\mathcal{F}\left(t\right)$-measurable. (4) For each i and t, $\mathrm{B}\left[w_i\left(t\right)|\mathcal{F}\left(t\right)\right]=0$. (5) $\mathrm{B}\left[w_i^2\left(t\right)|\mathcal{F}\left(t\right)\right]\le A+B{max}_j{max}_{\tau \le t}{\left|x_j\left(\tau \right)\right|}^2$, where A and B are deterministic constants.

Assumption 3.

The learning rates satisfy $0\le {\alpha }_i\left(t\right)<1$.

Assumption 4.

Suppose some conditions for the F mapping are as follows: (1) If $x\le y$, then $F\left(x\right)\le F\left(y\right)$, that is, F is monotonic; (2) F is continuous; (3) When $t\to \infty$, F is limited to the interval $\left[x^{*}-D,x^{*}+D\right]$, where $x^{*}$ is some arbitrary point; (4) If $e\in {\mathcal{R}}^n$ is a vector that satisfies all components equal to 1, then $F\left(x\right)-pe\le F(x+pe)\le F(x+pe)+pe$, where p is a positive scalar.

Assumption 5.

Each action-value pair can be accessed indefinitely, and the reward is limited.

Assumption 6.

Under the limit $t\to \infty$ of infinite exploration, the agent’s policy is greedy.

This assumption ensures that the agent is rational.

Assumption 7.

In each stage of a stochastic game, a Nash equilibrium can be regarded as a global optimum or saddle point.

Based on these assumptions, Subramanian et al. [27] give the following lemma:

Lemma 1

 ([27]). When the Q-function is updated using the partially observable update rule in Equation (2), and Assumptions 3, 5, and 7 hold, the following holds for $t\to \infty$:

$$|Q_{*}(s_t,a_t)-Q_{POMF}(s_t,a_t,{\tilde{a}}_t)|\le 2D$$

(20)

where $Q_{*}$ is the Nash Q-value, $Q_{POMF}$ is the partially observable mean field Q-function, and D is the bound of the F map. The probability that the above formula holds is at least ${\delta }^{L-1}$, where $L=\left|A\right|$.

In our GPMF setting, for partially observable neighborhood agents, we choose to select a limited number of important agents by using graph attention, and then update the POMF Q-function. The following theorem proves that the setting of GPMF Q-function is close to Nash equilibrium.

Theorem 1.

The distance between the MFQ (globally observable) mean action $\overline{a}$ and the GPMF (partially observable) mean action $\tilde{a}$ satisfies the following inequality:

$$\left|{\tilde{a}}_t^j-{\overline{a}}_t^j\right|\le \sqrt{\frac{1}{2N_j}log\frac{2}{\delta }}$$

(21)

where $N_j$ is the number of observed neighbor agents, $\tilde{a}$ is the partially observable mean action obtained by graph attention in Equation (16), $\overline{a}$ is the globally observable mean action in Equation (4), when $t\to \infty$, the probability $>=\delta$.

Proof of Theorem 1.

Assuming that each agent is globally observable, the mean of important agents selected by graph attention is close to the true underlying global observable $\overline{a}$. Since the GPMF Q-function is updated by taking finite samples through graph attention, the empirical mean is $\tilde{a}$.    □

Theorem 2.

If the Q-function is Lipschitz continuous with respect to the mean action, i.e., M is constant, then the MF Q-function $Q_{MF}$ and GPMF Q-function $Q_{GPMF}$ satisfy the following relation:

$$\left|Q_{GPMF}\left(s_t,a_t,{\tilde{a}}_{t-1}\right)-Q_{MF}\left(s_t,a_t,{\overline{a}}_{t-1}\right)\right|\le M\times L\times log\frac{2}{\delta }\times \frac{1}{2N_j}$$

(22)

where $L=\left|A\right|$, A is the action space of the agent, when the limit $t\to \infty$, the probability is $\ge {\left(\delta \right)}^{L-1}$.

Proof of Theorem 2.

To prove Theorem 2, first consider a Q-function that is Lipschitz continuous for all $\overline{a}$ and $\tilde{a}$. According to Theorem 1, the above formula can further deduce the result of Theorem 2. The total number of components is equal to the action space L. The bound of Theorem 1 is probability $>=\delta$, and since there are L random variables, the probability of Theorem 2 is at least ${\left(\delta \right)}^{L-1}$. When the first $L-1$ random variable is fixed, the deterministic last $\overline{a}$ component satisfies the relationship that the sum of the individual components is 1. Since each agent’s action is represented by a one-hot encoding, the ${\tilde{a}}^{\prime }$ component of GPMF also satisfies the relationship that the sum of the individual components is 1, and the component of the agent’s mean action does not change due to the application of graph attention. The proof of Theorem 2 ends.    □

Theorem 3.

A stochastic process in form $x_i(t+1)=x_i\left(t\right)+{\alpha }_i\left(t\right)\left(F_i\left(x^i\left(t\right)\right)-x_i\left(t\right)+w_i\left(t\right)\right)$ remains bounded in the range $[x_{*}-2D,x_{*}+2D]$ on limit $t\to \infty$ if Assumptions A1, A2, A3 and A4 are satisfied, and are guaranteed not to diverge to infinity. Where D is the boundary of the F map in Assumption A4.

Proof of Theorem 3.

This theorem can be proved by extending the results in Tsitsiklis [46]. The result of Theorem 3 can then be used to derive Theorem 4.    □

Theorem 4.

When the Q-function is updated using the partially observable update rule in Equation (17), and Assumptions 3, 5, and 7 hold, the following holds for $t\to \infty$:

$$|Q_{*}(s_t,a_t)-Q_{GPMF}(s_t,a_t,{\tilde{a}}_t)|\le 2D$$

(23)

where $Q_{*}$ is the Nash Q-function, $Q_{GPMF}$ is the partially observable mean field Q-function, and D is the bound of the F map. The probability that the above formula holds is at least ${\delta }^{L-1}$, where $L=\left|A\right|$.

Proof of Theorem 4.

Theorem 4 shows that the GPMF update is very close to the Nash equilibrium at the limit $t\to \infty$, i.e., reaching a plateau for stochastic policies. Therefore, the strategy of Equation (19) is approximately close to this plateau. Theorem 4 is an application of Theorem 3, using Assumptions 3, 5 and 7. However, in MARL, a Nash equilibrium is not optimal, but only a fixed-point guarantee. Therefore, to achieve better performance, each selfish agent will still tend to pick a limited number of samples. To balance theory and performance when selecting agents from the neighborhood, an appropriate number of agents (more efficient agents) need to be used for better multi-agent system performance. This paper uses the graph attention structure to filter out more important proxies, which can better approximate the Nash equilibrium.    □

4.4. Algorithm

The implementation of GPMF follows the related work of the previous POMFQ [27], the difference is that the graph attention structure is used to select the neighborhood agents that are more important to the central agent when updating the mean action. Algorithm 1 gives the pseudocode of the GPMF algorithm. It obtains effective neighbor agents by continuously updating the adjacency matrix $G_j^t$ to update the agent’s strategy.

Algorithm 1 Partially Observable Mean Field MARL Based on Graph Attention Network

Initialize the weights of Q-function $Q_{{\varphi }^j}$, $Q_{{\varphi }_{-}^j}$, replay buffer B, GAT encoder, MLP layers and mean action ${\overline{a}}^j$ for each agent $j\in 1,\dots ,N$. for$episode=1,2,\dots ,E$do      for $t\le T$ and not terminal do         For each agent j, calculate the hidden state $h_j^t$ according to Equation (11), and encode $h_j^t$ as a message $m_j^t$ (Equation (12)).         For each agent j, sample $a^j$ fron policy induced by $Q_{{\varphi }^j}$(Equation (19)).         For each agent j, pass the encoded information $m_j^t$ to the GAT encoder and hard attention mechanism to output the adjacency matrix $G_j^t$.         For each agent j, calculate the new neighborhood agent mean action ${\overline{a}}^j$ by Equation (16).         Receive the full state of environment $s_t$, action $a=\left[a^1,\dots ,a^N\right]$, reward $\left[r=r^1,\dots ,r^N\right]$, and the next state $s^{\prime }=\left[s^1,\dots ,s^N\right]$.         Store transition $〈s,a,r,s^{\prime },\overline{a}〉$ in B, where $\overline{a}=\left[{\overline{a}}^1,\dots ,{\overline{a}}^N\right]$ is the mean action.      for $j=1,\dots ,N$  do         Sample a minibatch of K experiences $〈s,a,r,s^{\prime },\overline{a}〉$ from replay buffer B.         Set $y^j=r^j+\gamma v_{{\varphi }_{-}^j}^{GPMF}\left(s^{\prime }\right)$ according to Equation (18).         Minimize the loss $L\left({\varphi }^j\right)=\frac{1}{K}\sum \left(y^j-\right.{\left.Q_{{\varphi }_j}\left(s^{\prime },a^j,{\overline{a}}^j\right)\right)}^2$ to update Q network.     For each agent j, update params of target network:${\varphi }_{-}^j\leftarrow \tau {\varphi }^j+(1-\tau ){\varphi }_{-}^j$.

In Algorithm 1, for each agent j, the Q-function is parameterized by $\varphi$ and trained by minimizing the loss function $L\left({\varphi }^j\right)$, where $y^j$ is the target mean field value calculated with the weights ${\varphi }_{-}^j$ and finally trained by the gradient optimizer.

5. Experiments

For the multi-UAV cooperative pursuit scenario, the multi-agent reinforcement learning training platform Magent is used to simulate and verify the GPMF algorithm. The same experimental environments as [27] are adopted in our work, in which three three different strategies are designed in the MAgent framework [47]. It is noteworthy that the simulation environment is a partially observable multi-agent cooperative task environment, which is used to simulate the cooperative pursuit of multi-UAV swarms. In these three tasks, the map size is set to 28*28, where the observation range of each UAV is 6 units. The state space is the concatenation of the feature information of other UAVs within each UAV’s field of view, including location, health, and grouping information. The action space includes 13 move actions and 8 attack actions. In addition, each UAV is required to handle at most 20 other UAVs that are closest. To ensure a comprehensive understanding of the evaluation process, we provide detailed descriptions of the experimental setup and the training procedures are employed within the MAgent framework. This allow us to analyze the performance of the GPMF and evaluate its effectiveness in dealing with dynamic multi-UAV cooperative pursuit tasks.

5.1. Environment and Tasks

We will evaluate on three tasks: Multibattle game, Battle-Gathering and Predator-Prey.

• Multibattle game: There are two groups of UAVs participating in the competition scenario, each consisting of 25 UAVs. UAV gets −0.005 points for each movement, −0.1 points for attacking open space, 200 points for collision causing enemy UAV to fall, and 0.2 points for successful collision. Each UAV has a size of 2 × 2, a maximum health of 10 units, and a speed of 2 units. At the end of the mission, the team with the most surviving UAVs wins. If two teams have the same number of surviving UAVs, the team with the highest reward wins. Each team’s reward is the sum of the rewards of the individual UAVs in the team.
• Battle-Gathering game: The distribution of resources in the environment is uniform, and each UAV can observe the location of all resources. In addition to rewards for shooting down enemy UAVs, each UAV can also occupy resources to obtain rewards. UAVs get 5 points for collides enemy UAVs, and the rest of the reward settings are the same as the Multibattle environment.
• Predator-Prey game: There are 40 predators (big UAVs) and 20 preys (small UAVs), respectively simulating two groups of UAVs. Each large UAV is a square with a size of 2 × 2, with a maximum health of 10 units and a speed of 2 units. The small UAV is a 1 × 1 square with a maximum health of 2 units and a speed of 2.5 units. In order to complete the task, the large UAV must collide with more small UAVs, and the small UAVs must try to evade. In addition, large UAVs and small UAVs have different reward functions. Specifically, large UAVs get −0.3 points for attacking spaces, 1 point for successful collisions with small UAVs, 100 points for shooting down small UAVs, −1 point for collisions with small UAVs, and large UAV drops earn 0.5 points. Unlike the Multibattle environment, when the round ends for a fairer duel, if the two teams have the same number of surviving UAVs, it is judged as a draw.

5.2. Evaluation

We consider four algorithms for the above three games: Mean Field Q-learning (MFQ), Mean Field Actor-Critic (MFAC) [19], Partially Observable Mean Field Q Learning-Fixed Observation Radius (POMFQ(FOR)) [27] and GPMF, where MFQ and MFAC are baselines and POMFQ(FOR) is the state-of-the-art algorithm.

The original baselines MFQ and MFAC were proposed by Yang et al. [19] based on global observability, and the idea was to approximate the influence of the neighborhood agents on the central agent as their mean actions, thereby updating the actions of the neighborhood agents. We fix the observation radius of each agent in the baseline MFQ and MFAC and apply it to a partially observable environment, where neighbor agents are agents within a fixed range. The POMFQ(FOR) algorithm introduces noise in the mean action parameters to encourage exploration, uses Bayesian inference to update the Dirichlet distribution, and samples 100 samples from the Dirichlet distribution to estimate partially observable mean field actions. The GPMF algorithm evaluates the effectiveness of neighborhood agents within a fixed range through the graph attention mechanism, selects more important neighborhood agents, and updates the mean action by averaging the actions of these agents.

5.3. Hyperparameters

In the three tasks, each algorithm was trained for 2000 epochs in the training phase, generating two sets of A and B sets of models. In the test phase, 1000 rounds of confrontation were conducted, of which the first 500 rounds were the first group A of the first algorithm against the second group B of the second algorithm, and the last 500 groups were the opposite. We would like to emphasize that our primary objective was to compare the performance of different algorithms, rather than conducting adversarial experiments between the same algorithms. Our goal was to assess the effectiveness of various approaches and identify the algorithm that performs best in the given tasks. The hyperparameters of MFQ, MFAC, POMFQ(FOR) and GPMF are basically the same.

Table 1. Hyperparameters for four algorithms training.

Parameter	Value	Description
$\alpha$	${10}^{-4}$	learning rate
$\beta$	decays linearly from 1 to 0	exploration rate
$\gamma$	0.95	discount rate
B	1024	replay buffer
h	64	the hidden layer size in GAT
K	64	mini-batch
$temperature$	0.1	the soft-max layer temperature of the actor in MFAC

lists the hyperparameters during training of the four algorithms, and the remaining parameters can be seen in [27].

6. Results and Discussion

In this section, we evaluate the performance of GPMF in three different environments, including Multibattle, Battle-Gathering, and Predator-Prey.

We benchmark against three algorithms, MFQ, MFAC and POMFQ(FOR), in which POMFQ(FOR) achieved the state-of-the-art performance to our best knowledge.

We implement our method and comparative methods on three different tasks. Note that we only used 50 UAVs in our experiments and did not test more UAVs, this is because the proportion of other UAVs that each UAV can see is more important than the absolute number.

6.1. Reward

Figure 5 shows how the reward changes as the number of iterations increases during training. We plot the reward changes for the four algorithms in different environments during the first 1000 iterations. Since each algorithm is self-training which results in a large change in the reward, we use the least squares method to fit the reward curve. In Figure 5, the solid black line represents the reward of the GPMF algorithm. From Figure 5a–c, it can be seen that the reward of the GPMF algorithm increases rapidly, indicating that the GPMF algorithm can converge rapidly in the early stage, and the convergence performance is better than the other three algorithms.

6.2. ELO Calculation

We use ELO Score [48] to evaluate the performance of the two groups of agents, the advantage of which is that it takes into account the strength gap between the opponents themselves. ELO ratings are commonly used in chess to evaluate one-on-one situations, and this approach can similarly be extended to N-versus-N situations. For the algorithm proposed in the paper, we record the total rewards of the two teams of agents during each algorithm confrontation, which are $R_1$ and $R_2$, respectively. Then the expected win rates of the two groups of agents are:

$$E_1=\frac{1}{1+{10}^{(R_2-R_1)/400}},E_2=\frac{1}{1+{10}^{(R_1-R_2)/400}}$$

(24)

where $E_1+E_2=1$. By analyzing the actual and predicted winning rates of the two groups of agents, the new ELO score of each team after the game ends can be obtained:

$${R_1}^{\prime }=R_1+K(S_1-E_1),{R_2}^{\prime }=R_2+K(S_2-E_2)$$

(25)

where $R_1$ represents the actual winning or losing value, 1 means the team wins, 0.5 means the two teams are tied, and 0 means the team loses. K is represented as a floating coefficient. To create a gap between agents, we set K to 32. For each match, we faced off 500 times and calculated the average ELO value for all matches.

Table 2. The ELO Score of four algorithms in Multibattle environment.

Algorithm 1	Algorithm 2	ELO Score 1	ELO Score 2
GPMF-1	POMFQ(FOR)-2	3579	820
GPMF-2	POMFQ(FOR)-1	2696	2838
GPMF-1	MFQ-2	2098	1508
GPMF-2	MFQ-1	2535	1695
GPMF-1	MFAC-2	1350	−49
GPMF-2	MFAC-1	−856	−78
POMFQ(FOR)-1	MFQ-2	3145	2577
POMFQ(FOR)-2	MFQ-1	2569	2857
POMFQ(FOR)-1	MFAC-2	−205	−64
POMFQ(FOR)-2	MFAC-1	826	−42
MFQ-1	MFAC-2	−142	−49
MFQ-2	MFAC-1	610	−46

,

Table 3. The ELO Score of four algorithms in Battle-Gathering environment.

Algorithm 1	Algorithm 2	ELO Score 1	ELO Score 2
GPMF-1	POMFQ(FOR)-2	7770	8931
GPMF-2	POMFQ(FOR)-1	8293	9310
GPMF-1	MFQ-2	6374	10,870
GPMF-2	MFQ-1	8510	8313
GPMF-1	MFAC-2	5525	10
GPMF-2	MFAC-1	10,751	−31
POMFQ(FOR)-1	MFQ-2	8526	8760
POMFQ(FOR)-2	MFQ-1	8632	8227
POMFQ(FOR)-1	MFAC-2	12,722	0
POMFQ(FOR)-2	MFAC-1	12,171	−88
MFQ-1	MFAC-2	12,649	49
MFQ-2	MFAC-1	13,788	−48

and

Table 4. The ELO Score of four algorithms in Predator-Prey environment.

Algorithm 1	Algorithm 2	ELO Score 1	ELO Score 2
GPMF-1	POMFQ(FOR)-2	421	−32
GPMF-2	POMFQ(FOR)-1	16	7
GPMF-1	MFQ-2	714	−27
GPMF-2	MFQ-1	−15	−94
GPMF-2	MFAC-1	16	16
POMFQ(FOR)-1	MFQ-2	66	18
POMFQ(FOR)-2	MFQ-1	13	24
POMFQ(FOR)-1	MFAC-2	16	−16
POMFQ(FOR)-2	MFAC-1	47	16
MFQ-1	MFAC-2	16	−16
MFQ-2	MFAC-1	174	17

show the ELO scores of the four algorithms on the three tasks. It can be seen from Table 2 that in Multibattle environment, the GPMF algorithm has the highest ELO score of 3579, which is significantly better than the other three algorithms. As shown in Table 3, in Battle-Gathering environment, the ELO score of the MFQ algorithm is the highest, and the ELO score of the GPMF algorithm is average. This is because some algorithms favor capturing resources for a quick reward rather than colliding with enemy UAVs. However, the final game winning or losing decision is made by comparing the number of remaining UAVs between the two teams of UAVs. As shown in Table 4, in Predator-Prey environment, the ELO score of the GPMF algorithm has the highest ELO score of 860, which is significantly better than the other three algorithms. From the experimental results in the three environments, we can summarize that ELO score of the GPMF algorithm is better than other three algorithms, showing better performance.

6.3. Results

Figure 6 shows the face-off results of the four algorithms in the three tasks. Figure 6a shows the faceoff results of Multibattle game. The different colored bars for each algorithm represent the results of an algorithm versus others. The vertical lines in the bar graph represent the standard deviation of wins for groups A and B over 1000 face-offs. Figure 6a shows GPMF against three other algorithms, all with a win rate above 0.7.

Figure 6b shows the faceoff results of Battle-Gathering game. In addition to the rewards for shooting down UAVs, UAVs can also get rewards for occupying resources. It can be seen that MFQ loses to all other algorithms, MFAC and POMFQ(FOR) perform in general, and our GPMF is clearly ahead of other algorithms.

Figure 6c shows the faceoff results of Predator-Prey game.The standard deviation of this game is significantly higher than the previous two games, due to the fact that both groups A and B are trying to beat each other in the environment. It can be seen that the GPMF algorithm is significantly better than other three algorithms, reaching a winning rate of 1.0.

6.4. Visualization

To visualize the effectiveness of the GPMF algorithm, we visualize the confrontation between GPMF and POMFQ(FOR) in a Multibattle environment, as shown in Figure 7, where the red side is GPMF and the blue side is POMFQ(FOR). It can be seen from the confrontation process that for the GPMF algorithm, when a UAV decides to attack, the surrounding UAVs will also decide to attack under its influence, forming a good cooperation mechanism. On the contrary, for the POMFQ(FOR) algorithm, some blue-side UAVs are chosen to attack, some are chosen to escape, and no common fighting mechanism was formed. Similarly, in the Battle-Gathering environment of Figure 8, GPMF can learn the besiege mechanism well. In the Predator-Prey environment of Figure 9, when GPMF acts as a predator, the technique of surrounding the prey POMFQ(FOR) can be learned. On the contrary, when POMFQ(FOR) acted as a predator, it failed to catch the prey GPMF.

6.5. Ablation Study

Figure 10 is an ablation study that investigates the performance of the GPMF algorithm for different observation radius in a Multibattle environment. where the solid line represents the least squares fit of the reward change. It can be seen from the figure that when the number of training episodes is small, the performance of the algorithm is better as the observation distance increases. But with the increase of training times, when $R=4$, the performance of the algorithm is the best, so the appropriate observation distance can achieve better performance. What is more important in this paper is to explore the effect of the ratio of observable distance to the number of agents on the performance of the algorithm, so there is no experiment with more agents.

7. Conclusions

In this paper, we propose a novel multi-agent reinforcement learning algorithm, Graph attention network supported Partially observable Mean Field Multi-agent reinforcement learning (GPMF), to address large-scale partially observable multi-UAV cooperative pursuit problem. While existing methods approach Nash equilibrium, they fail to consider direct inter-agent correlations. By abstracting the UAVs of a multi-UAV system into agents, exploiting inter-agent correlations, a graph attention module is incorporated to capture the influence of actions taken by other agents at each time step. Experimental results on three challenging tasks within the MAgents framework demonstrate that our proposed approach outperforms baselines and state-of-the-art partially observable mean field reinforcement learning algorithms in all of these tasks. This further indicates the ability of our algorithm to improve communication and cooperation among UAVs.

In addition, we recognize the importance of investigating the robustness of multi-UAV system and improving the resilience of the GPMF algorithm against various interferences. One key aspect we aim to address is enhancing the reliability and effectiveness of communication protocols [7]. By developing more robust and efficient communication protocols, we can improve the overall resilience and robustness of multi-UAV system, enabling UAVs to effectively coordinate and adapt to changing environments and disturbances.