UAV Air Game Maneuver Decision-Making Using Dueling Double Deep Q Network with Expert Experience Storage Mechanism

Abstract

Deep reinforcement learning technology applied to three-dimensional Unmanned Aerial Vehicle (UAV) air game maneuver decision-making often results in low utilization efficiency of training data and algorithm convergence difficulties. To address these issues, this study proposes an expert experience storage mechanism that improves the algorithm’s performance with less experience replay time. Based on this mechanism, a maneuver decision algorithm using the Dueling Double Deep Q Network is introduced. Simulation experiments demonstrate that the proposed mechanism significantly enhances the algorithm’s performance by reducing the experience by 81.3% compared to the prioritized experience replay mechanism, enabling the UAV agent to achieve a higher maximum average reward value. The experimental results suggest that the proposed expert experience storage mechanism improves the algorithm’s performance with less experience replay time. Additionally, the proposed maneuver decision algorithm identifies the optimal policy for attacking target UAVs using different fixed strategies.

1. Introduction

The advancement of Unmanned Aerial Vehicle (UAV) technology has led to its emergence as a novel weapon system in air games. In comparison to manned aircraft, UAVs offer benefits such as no risk of human casualties, extended flight duration, and low cost [1]. Nevertheless, UAVs do not possess the same degree of flexibility and adaptability as manned aircraft and suffer from limited autonomous gaming capabilities. Consequently, enhancing the autonomous decision-making capacity of UAVs is a primary area of interest in contemporary UAV air game research. UAV air game maneuver decision-making algorithms utilize sensor data as input to generate the optimal strategy for successfully executing target UAV strike missions. Broadly, maneuver decision algorithms employed by UAVs can be classified into three categories: game theory methods, optimization methods and artificial intelligence methods [2].

Game theory is a discipline that studies mathematical models of conflict and cooperation between intelligent decision makers [3]. Given the inherent game-like nature of interactions between two UAVs in air game scenarios, employing game theory to address air game challenges has emerged as a principal research focus. Currently, prominent approaches to maneuver decision algorithms grounded in game theory encompass those based on influence diagrams and differential games.

Influence diagrams are graphical models that represent uncertain variables and decision problems [4]. The maneuver decision approach based on influence diagrams integrates prior knowledge of pilot’s habits and preferences for behavioral reasoning [5]. Virtanen et al. [6] introduced a multi-stage influence diagram game to simulate a pilot’s maneuver decision-making in one-on-one air games, graphically illustrating aircraft dynamics and pilot preferences under uncertain conditions. A moving-horizon control method was employed to solve the influence diagram game. To solve multi-level influence diagrams, Zhong et al. [7] transformed the multi-stage influence diagrams into a two-level optimization problem based on multi-stage influence-diagram decision-process modeling. A study by Pan et al. [8] utilized influence diagrams in the context of multi-aircraft air games and suggested a maneuver model based on state prediction influence diagrams. This model incorporated a state acquisition model using an unscented Kalman filter based on belief states. To resolve the state prediction influence diagrams, the moving horizon approach was applied, and the model was substantiated through a simulation of a two-on-two cooperative air game. Xie et al. [9] adopted a Bayesian theorem for real-time situation assessment, transforming the multi-aircraft air game model into a one-on-one air game model, and they employed influence diagrams to represent the multi-aircraft air game process. A maneuver decision algorithm based on influence diagrams integrates pilots’ prior knowledge and offers strong simulation outcomes and interpretability. However, a notable drawback of this approach is its high computational complexity, which poses challenges when applying it to intricate air game environments.

Differential game theory [10] is commonly employed to address maneuver decision-making challenges in three-dimensional air games. Mukai et al. [11] devised a sequential linear quadratic method to identify local Nash solutions of general differential games and applied this technique to air game simulations. Simulation experiments demonstrated that solution estimation could converge to the local Nash (saddle) solution of the original game. Fu et al. [12] established a fixed-duration difference game model with speed, elevation angle, and yaw angle as control variables. In this model, the integral of the deviation angle and the departure angle was utilized as the cost function, while a penalty function was used to deal with constraints. Park et al.’s [13] maneuver decision algorithm features a hierarchical decision structure and leverages a differential game theory to compute the scoring function matrix, which reflects the degree of UAV air superiority. Başpınar et al. [14] employed the flatness property of the dynamical system to generate control and state sequences using a B-spline curve for the flat output. Then, by employing the security policy approach in game theory, the air game problem was transformed into an optimization problem of parameterized curves. He et al. [15] established a target–defender–attacker model and designed a nonlinear guiding rule based on differential game theory, verifying the algorithm’s effectiveness through simulation experiments. Although the application of a differential games model to solve air game problems has yielded numerous theoretical results, the expansion of decision spaces can lead to a dimension explosion issue, resulting in reduced solution efficiency.

Optimization methods treat air game problems as multi-objective optimization problems subject to aerodynamic constraints. Smith et al. [16] developed a genetics-based machine learning system that employs digital simulation models of one-on-one air games and genetic algorithms to generate maneuvers. Experimental results demonstrated that the proposed system enabled the X-31 to successfully counter traditional fighter opponents. Sprinkle et al. [17] framed the control problem as a cost-minimization problem and proposed a non-linear model predictive tracking controller algorithm for evasive maneuvers in fixed-wing UAVs. McGrew et al. [18] utilized approximate dynamic programming (ADP) for UAVs to learn the optimal strategies for air games, facilitating rapid response to quickly changing tactical situations with long planning horizons and good performance. Duan et al. [19] introduced a predator-based particle swarm optimization method that effectively addresses the dynamic task assignment of UCAVs as validated by air game simulations. To enhance the efficiency and accuracy of air game confrontation evolution, Ji et al. [20] applied biological immunity to air game decision-making and developed an improved Tactical Immune Maneuver System (TIMS) model that combines sequential-association data mining with the TIMS model. Han et al. [21], considering the impact of uncertainty factors and decision-making lag in UAV air games, proposed a robust maneuver decision-making method based on target-accessibility domain-state prediction. This method designed a robust multi-objective optimization function grounded in state function and statistical principles and utilized the grey wolf optimizer to solve real-time optimization issues. Tan et al. [22] introduced the Model Predictive Control (MPC) theory and introduced the LSHADE-TSO-MPC air game maneuver decision-making method, which predicted target UAV trajectories and accelerated air game victories. Ruan et al. [23] combined the bionic intelligent optimization algorithm with air game decision-making problems and designed a transfer-learning pigeon-inspired optimization model that demonstrated good convergence, high precision, and reasonable maneuver decisions in one-on-one dogfights. Machmudah et al. [24] solved the optimization problem of flight trajectories by using the bank-turn mechanism of fixed-wing UAVs at a constant altitude. The experimental results showed that the particle swarm optimization algorithm had good performance in path planning and flight trajectory planning. Despite the numerous methods for solving optimization problems, air game decision-making that employs optimization theory faces challenges such as difficulty in convergence and poor real-time performance due to the complexity of the air game state space.

The advancement of UAV technology and artificial intelligence has spurred research of maneuver decision algorithms. These algorithms can be categorized into reinforcement learning methods and other machine learning methods. In the latter category, machine learning algorithm are combined with autonomous maneuver decision-making to address the problem of UAV autonomous decisions in air games. Some researchers have employed neural networks to store air game rules, enabling the replication of air game behavior [25,26,27]. This data-driven approach does not necessitate a detailed aircraft dynamics model. However, the trained agent’s decision-making ability is influenced by the training data. Machine learning algorithms, including Bayesian networks and fuzzy theory, have also been utilized in generating air game strategies. Considering the uncertainty and real-time nature of the UAV air game decision-making process, Geng et al. [28] proposed a hybrid tactical decision-making method based on rule sets and fuzzy Bayesian networks (FBN), preserving the advantages of expert systems. Similarly, Wu et al. [29] developed a situation assessment model utilizing a fuzzy reasoning machine, enabling the UAVs to conduct thorough assessment of air game benefits. Çintaş et al. [30] used deep learning to automatically detect and track another moving UAV in the air. The experimental results showed that the proposed method provided the highest accuracy rate of 82.7% and a mean fps speed of 29.6.

However, approaches relying on training data or expert systems are constrained by the training data of human experts or pilots. Reinforcement-learning-based algorithms simulate the air game environment as a Markov Decision Process (MDP) and derive optimal strategies through continuous trial and error in the simulation environment, making them suitable for various complex environments without depending on pilot training data. Recently, reinforcement learning technology has rapidly advanced, and its integration with air games has become a prominent research area.

Acknowledging sensor errors in air game environments, Kong et al. [1] represented air games as a state-adversarial Markov Decision Process (SA-MDP) and introduced an autonomous maneuver strategy generation algorithm for aircraft based on the state-adversarial deep deterministic policy gradient algorithm (SA-DDPG). The shaping reward obtained by the inverse reinforcement learning algorithm for addressing reward sparsity effectively enhances air game strategy learning speed. To solve reinforcement learning algorithm convergence speed issues, Li et al. [31] improved the deep deterministic policy gradient algorithm [32] (DDPG) convergence speed by proportionally returning the final reward value to other reward values in the same air game process. Targeting the DDPG exploration strategy’s insufficient exploration and low data utilization, Wan et al. [33] incorporated a heuristic exploration strategy to enhance the algorithm’s exploration capacity. Yang et al. [34] applied the proximal policy optimization algorithm [35] (PPO) to UAV air games and verified the algorithm’s effectiveness through air game confrontation simulations. Zhang et al. [36] integrated the concept of final reward with proximal strategy optimization, proposing an FRE-PPO-based air game agent training framework to improve air game agents’ maneuver decision-making ability through self-play training methods. Fan et al. [37] proposed an autonomous maneuver decision-making model based on the Asynchronous Advantage Actor–Critic (A3C) algorithm, establishing a two-layer reward mechanism combining internal rewards and sparse rewards to promote faster convergence. By combining deep reinforcement learning with stochastic game theory, Cao et al. [38] designed a MinimaxDDQN network to effectively enhance the UCAV’s autonomous game capability. Aiming at the problem of the generalization performance of the SAC algorithm, Li et al. [39] proposed a PSP-SAC algorithm, which can realize sample sharing and strategy sharing in various game environments, and it significantly improves the generalization performance compared with independent training. In order to solve the diversity of air game samples, Li et al. [40] also proposed a SAC algorithm based on expert actors, using a small amount of expert experience to increase the diversity of samples, which can greatly improve the exploration and utilization efficiency of deep reinforcement learning. In general, reinforcement-learning-based methods are applicable to various complex confrontation scenarios and can achieve online real-time decision-making. However, these methods exhibit low training data utilization efficiency, and the reinforcement learning algorithms employed are challenging to converge.

Traditional methods for UAV maneuver decision-making suffer from high computational complexity and exploding dimensions. However, for air games, reinforcement-learning-based maneuver decision-making can leverage the benefits of this learning approach in complex environments, thereby improving the dynamic adaptability of UAVs. Therefore, reinforcement learning is used for UAV maneuver decision-making in this paper. Despite the advantages of this approach, there are challenges related to the low utilization efficiency of training data and difficulties with algorithm convergence when applying reinforcement learning to air game decision-making. In real-world environment, the flying experience of air game experts is more effective than the strategies explored by novice pilots, which means incorporating expert experience into the training of UAV agents may provide a more effective strategy. Based on this idea, this paper proposes a maneuver decision algorithm based on a Dueling Double Deep Q Network (D3QN) with an expert experience storage mechanism (EESM). Firstly, the air game problem is modeled based on the MDP framework, encompassing the aircraft dynamics model, attack determination area, state space, action space, and reward function. To prevent the waste of expert experience in the replay buffer during reinforcement learning algorithm training, an expert experience storage mechanism is introduced, which aims to enhance experience storage efficiency and algorithm convergence speed. Subsequently, D3QN is incorporated into UAV air game maneuver decision-making, leading to the design of a D3QN maneuver decision algorithm based on the expert experience storage mechanism. Lastly, simulation experiments are conducted, with results indicating that the proposed mechanism accelerates the algorithm’s convergence, and the suggested maneuver decision algorithm can effectively strike target UAVs by adopting various fixed strategies.

2. Air Game Environment Modeling

2.1. Markov Decision Process

The goal of reinforcement learning is to facilitate the agent’s ability to learn strategies that maximize numerical rewards through continuous trial and error. The Markov decision process serves as a mathematical framework ideally suited for addressing reinforcement learning problems [41].

Within this framework, the agent selects an action and the environment responds by presenting a new state based on the chosen action. Concurrently, the environment generates reward that continuously encourage the agent to learn a strategy that maximizes the total reward. The interaction between the environment and the agent is illustrated in Figure 1.

As depicted in Figure 1, at each discrete time t, the agent observes the environment’s state $S_t\in \mathcal{S}$ and chooses an action $A_t\in \mathcal{A}\left(s\right)$ based on the current strategy. In the following moment, the agent receives the immediate reward $R_{t+1}\in \mathcal{R}\subset \mathbb{R}$, which is given by the environment. Subsequently, the environment transitions to a new state $S_{t+1}$ according to the action $A_t$.

Typically, the function p is employed to define the dynamic characteristics of MDP. For any $s^{\prime },s\in \mathcal{S}$, $r\in \mathbb{R}$, and $A_t\in \mathcal{A}\left(s\right)$, the following definitions are given:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill p\left(s^{\prime },r\mid s,a\right)\doteq Pr\left\{S_t=s^{\prime },R_t=r\mid S_{t-1}=s,A_{t-1}=a\right\}\end{array}\end{array}$$

(1)

The agent aims to maximizes the sum of future rewards, considering discount factors. The expected return is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill G_t\doteq R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}\end{array}\end{array}$$

(2)

To assess the agent’s performance in a particular state, the concepts of action–value function (Q-function) and value function are introduced as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill v_{\pi }\left(s\right)& \doteq {\mathbb{E}}_{\pi }\left[G_t\mid S_t=s\right]={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}\mid S_t=s\right],\hfill \\ \hfill Q_{\pi }(s,a)& \doteq {\mathbb{E}}_{\pi }\left[G_t\mid S_t=s,A_t=a\right]\hfill \\ \hfill & ={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}\mid S_t=s,A_t=a\right]\hfill \end{array}\end{array}$$

(3)

The advantage function refers to the advantage of an action a relative to the average value in a particular state; it can be described by the following formula:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill A_{\pi }(s,a)=Q_{\pi }(s,a)-v_{\pi }\left(s\right)\end{array}\end{array}$$

(4)

2.2. Aircraft Motion Modeling

Within the air game environment, our UAV and the target UAV engage in an air game within a three-dimensional space. To accurately depict the reinforcement-learning air game environment, modeling the motion of the UAV is essential.

The aircraft motion model outlines the updated state of the UAV under specific actions, signifying the transition from $S_t$ to $S_{t+1}$. The primary focus of this article is the maneuver decision-making algorithm, with an emphasis on updating the UAV’s position and attitude. As a result, this article introduces a particle model and treats the UAV as a particle. The UAV coordinate system employs local east, north, and up (ENU) coordinates OXYZ, where the OX-axis represents east, the OY-axis represents north, and the OZ-axis points vertically skyward. The following kinematic equations for position information in the OXYZ coordinate system can be derived as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill \frac{\mathrm{d}x}{\mathrm{d}t}& =vcos\theta sin\psi \hfill \\ \hfill \frac{\mathrm{d}y}{\mathrm{d}t}& =vcos\theta cos\psi \hfill \\ \hfill \frac{\mathrm{d}z}{\mathrm{d}t}& =vsin\theta \hfill \end{array}\right.\end{array}\end{array}$$

(5)

where x, y, and z denote the UAV’s coordinates in the ENU coordinates OXYZ; v denotes the UAV’S velocity vector; $\mathrm{d}x/\mathrm{d}t$, $\mathrm{d}y/\mathrm{d}t$, and $\mathrm{d}z/\mathrm{d}t$ are the projections of v in the ENU coordinates OXYZ; and $\theta$ and $\psi$ symbolize the pitch and yaw angles of the UAV, respectively.

As shown in Figure 2, $\theta$ is defined as the angle between the UAV velocity vector v and the projections of v in the O-XY-plane, and $\psi$ is defined as the angle between the projections of v in the O-XY-plane and and the OY-axis. In the ENU coordinate system, the kinetic equations can be derived as below:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill \frac{\mathrm{d}v}{\mathrm{d}t}& =g\left(n_x-sin\theta \right)\hfill \\ \hfill \frac{\mathrm{d}\theta }{\mathrm{d}t}& =\frac{g}{v}\left(n_{z}cos\varphi -cos\theta \right)\hfill \\ \hfill \frac{\mathrm{d}\psi }{\mathrm{d}t}& =\frac{g{n}_{z}sin\varphi }{vcos\theta }\hfill \end{array}\right.\end{array}\end{array}$$

(6)

where $\mathrm{d}v/\mathrm{d}t$ denotes the change rate of velocity with time, $\mathrm{d}\theta /\mathrm{d}t$ denotes the change rate of pitch angle with time, $\mathrm{d}\psi /\mathrm{d}t$ denotes the change rate of yaw with time, and g refers to the acceleration of gravity. The control variables of the UAV are represented by $[n_x,n_z,\varphi ]$, where $n_x$ refers to the longitudinal acceleration of the UAV, $n_z$ refers to the normal acceleration of the UAV, and $\varphi$ refers to the roll angle of the UAV. In summary, given the current state of the UAV and its control variables, the state of the UAV at the next moment can be determined through Equations (5) and (6).

2.3. Maneuver Decision Modeling by Deep Reinforcement Learning

2.3.1. Framework of Model

The air game maneuver decision algorithm proposed in this paper is based on reinforcement learning, which typically utilizes MDP as a theoretical framework. Thus, it is necessary to model the maneuver decision process and clarify the interaction between the UAV agent and the air game simulation environment. In the air game simulation environment, the goal of our UAV agent is to complete the task of striking the target, which entails keeping the target within attack range as much as possible while ensuring the UAV’s safety. Figure 3 illustrates the framework of the UAV agent interacting with the environment.

As depicted in Figure 3, the UAV agent receives state information from the environment and selects action using the maneuver decision algorithm. The air game environment then employs the aircraft motion equation described in Section 2.2 to calculate the next state of the UAV, taking the current state and maneuvering action of both UAVs as input while simultaneously calculating the current reward value using the reward function. After the calculation, the air game environment provides feedback to the UAV agent in terms of state and reward value. The UAV agent updates the decision-making algorithm based on the reward value and state information and employs the updated algorithm to complete the maneuver decision for the next moment.

2.3.2. Attack Determination Model

The attack decision model outlines the ultimate goal of the UAV agent and provides a detailed definition of the conditions required for the UAV agent to complete the air game cycle. The objective of the UAV agent is to acquire a higher reward value and successfully strike the target UAV.

Figure 4 defines the attack range of the UAV, where ${\theta }_{attack}$ denotes the attack angle and $D_{attack}$ denotes the attack distance. When the target UAV enters the attack zone of our UAV, this indicates that our UAV has successfully executed the attack. Equation (6) mathematically represents the decision condition for the success of a UAV attack:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill d& <D_{attack}\hfill \\ \hfill q_U& <{\theta }_{attack}\hfill \end{array}\right.\end{array}\end{array}$$

(7)

where $q_U$ denotes the angle between the UAV’s speed and the line connecting the UAV and the target, and d denotes the distance between the UAV and the target. Given UAV information $(x_U,y_U,z_U,{\theta }_U,{\psi }_U,{\varphi }_U)$ and target information $(x_T,y_T,z_T,{\theta }_T,{\psi }_T,{\varphi }_T)$, $q_U$ and d can be calculated by the following equation:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill d& =\sqrt{{\left(x_U-x_T\right)}^2+{\left(y_U-y_T\right)}^2+{\left(z_U-z_T\right)}^2}\hfill \\ \hfill q_U& =arccos\left\{\left[\left(x_T-x_U\right)cos{\varphi }_{U}cos{\theta }_U\right.\right.\hfill \\ \hfill & \left.\left.+\left(y_T-y_U\right)sin{\varphi }_{U}cos{\theta }_U+\left(z_T-z_U\right)sin{\theta }_U\right]/d\right\}\hfill \end{array}\end{array}$$

(8)

2.3.3. State Definition

The UAV state space represents the environment information perceived by the UAV agent. When designing the state space, it is necessary to filter and extract the environmental information. This information includes both directly related information about the goal and indirectly related information. The goal of the UAV agent is to complete a strike on the target, and the directly related information includes the distance d between the UAV and the current attack angle $q_U$ of the UAV. The indirectly relevant information is the environment information that is not directly related to the agent’s goal, but the agent can learn the strategy to achieve the goal through this indirect information. The indirectly related information for the UAV agent includes position information $(x_U,y_U,z_U,x_T,y_T,z_T)$, attitude information $({\theta }_U,{\psi }_U,{\varphi }_U,{\theta }_T,{\psi }_T,{\varphi }_T)$, velocity information $(v_U,v_T)$, and attack information $q_T$. By integrating and normalizing the directly related information and profile-related information, the designed state space is illustrated in

Table 1. The state space of the UAV agent.

State	Definition	State	Definition
$s_1$	$\frac{x_U-x_T}{d_0}$	$s_8$	$\frac{q_T}{\pi }$
$s_2$	$\frac{y_U-y_T}{d_0}$	$s_9$	$\frac{{\theta }_U}{2\pi }$
$s_3$	$\frac{z_U-z_T}{d_0}$	$s_{10}$	$\frac{{\psi }_U}{2\pi }$
$s_4$	$\frac{d}{d_0}$	$s_{11}$	$\frac{{\varphi }_U}{2\pi }$
$s_5$	$\frac{v_U-v_0}{v_1}$	$s_{12}$	$\frac{{\theta }_T}{2\pi }$
$s_6$	$\frac{v_T-v_0}{v_1}$	$s_{13}$	$\frac{{\psi }_T}{2\pi }$
$s_7$	$\frac{q_U}{\pi }$	$s_{14}$	$\frac{{\varphi }_T}{2\pi }$

.

Where $d_0$, $v_0$, $v_1$ are normalized constants. The position information adopts normalized relative coordinate information $(\frac{x_U-x_T}{d_0},\frac{y_U-y_T}{d_0},\frac{z_U-z_T}{d_0})$, which can make the UAV agent have good performance in different coordinate systems. The variable $q_T$ is the departure angle of the target, which is obtained by calculating the angle between the target’s speed and the line connecting the UAV and the target:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill q_T& =arccos\left\{\left[\left(x_T-x_U\right)cos{\varphi }_{T}cos{\theta }_T\right.\right.\hfill \\ \hfill & \left.\left.+\left(y_T-y_U\right)sin{\varphi }_{T}cos{\theta }_T+\left(z_T-z_U\right)sin{\theta }_T\right]/d\right\}\hfill \end{array}\end{array}$$

(9)

2.3.4. Action Definition

According to Equation (6), $[n_x,n_z,\varphi ]$ are selected as the control variables of the UAV: $n_x$ represents the longitudinal overload of the UAV, indicating the acceleration capability of the UAV along the direction of the flight speed; $n_z$ represents the normal overload of the UAV, which is vertical to the flight speed direction; and $\varphi$ is the roll angle of the UAV. Attitude correction can be achieved through the control of $\varphi$ and $n_z$. Using $[n_x,n_z,\varphi ]$ as the maneuver action implies that the action space of the UAV agent is continuous. Directly discretizing the continuous action space can lead to the curse of dimensionality, making it challenging for the algorithm to converge.

In real air games, NASA has defined a library of fundamental flight maneuvers for aircraft, including straight flight, acceleration, deceleration, left turn, right turn, upward flight, and downward flight [42]. The combination of different basic maneuvers can create more advanced stunts (e.g., Immelmann turns, loops, etc.). In this paper, taking into account air game background knowledge, the action space is defined as a set of seven basic actions, which avoids the curse of dimensionality caused by directly discretizing the continuous action space. The basic maneuvering action of the UAV can be expressed by setting the values of different control variables.

Table 2. The action space and the settings of the corresponding control values.

Action	Definition	${\mathit{n}}_{\mathit{x}}$	${\mathit{n}}_{\mathit{z}}$	$\mathit{\varphi }$
$a_1$	straight flight	0	1	0
$a_2$	acceleration	2	1	0
$a_3$	deceleration	−2	1	0
$a_4$	left turn	0	8	$-arccos(1/8)$
$a_5$	right turn	0	8	$arccos(1/8)$
$a_6$	upward flight	0	8	0
$a_7$	downward flight	0	8	$\pi$

displays the action space of the UAV in this study and the corresponding control value settings.

2.3.5. Reward Function Definition

The reward function’s design should ensure that the agent can accomplish its objective while maximizing the reward. In this study, the UAV agent’s goal is to complete the task of striking the target, which entails keeping the target within attack range as much as possible while ensuring the UAV’s safety. For the UAV agent’s goal, the reward function design can be separated into objective rewards and sub-objective rewards.

The objective reward pertains to the positive reward $R_0$ given to the UAV agent when the goal is achieved, which is typically unbiased towards the goal. When the target is within the UAV’s attack range, a positive reward value $R_p$ is given to the UAV agent; when the UAV is within the target’s attack range, a negative reward value $R_n$ is given to the UAV agent. To encourage the UAV agent to complete the strike mission as quickly as possible, a time penalty $R_t$ is introduced to guide the UAV towards swift mission completion. $R_0$ can be expressed using the following equation:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_0=\left\{\begin{array}{cc}{R}_p,& \mathrm{if}\mathrm{the}\mathrm{UAV}\mathrm{completes}\mathrm{attack}\mathrm{mission};\hfill \\ R_n,& \mathrm{if}\mathrm{the}\mathrm{UAV}\mathrm{is}\mathrm{wrecked};\hfill \\ R_t,& \mathrm{else}.\hfill \end{array}\right.\end{array}\end{array}$$

(10)

However, relying solely on objective rewards may result in sparse rewards, which could leave the UAV agent without proper guidance in exploration direction. By incorporating additional reward or penalty components based on the objective reward, the reward function becomes denser, providing adequate guidance to the UAV agent. This can accelerate the convergence of the deep reinforcement learning algorithm and enhances performance. Auxiliary reward components are designed by breaking down the objectives into sub-objectives and assigning different weights according to each sub-objective’s contribution to the overall goal realization. The goal of the UAV strike mission is divided into three sub-objectives:

• The distance between the UAV and the target should be within the attack range;
• The angle between the UAV and the target should be outside the attack angle of the target;
• The angle between the UAV and the target should be within the attack angle of the UAV.

According to the sub-goals of the UAV agent, $R_1$, $R_2$, $R_3$ are introduced. $R_1$ is the distance reward function, which increases as the UAV gets closer to the target. The distance reward function $R_1$ can guide the UAV agent to reduce the distance to the target and to rapidly enter the UAV’s attack range. The distance reward function $R_1$ is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_1=1-{\left(\frac{d}{D_r}\right)}^{0.4}\end{array}\end{array}$$

(11)

where $D_r$ is the distance reward constant. $R_2$ is the angle reward function, considering both the UAV’s attack angle and the target’s departure angle. The angle reward function $R_2$ can direct the UAV agent to maintain an offensive position and approach the target while ensuring its own safety. The angle reward function $R_2$ is defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_2=\frac{\pi -(q_U+q_T)}{\pi }\end{array}\end{array}$$

(12)

$R_3$ is the attack reward function, which increases as the target approaches the attack angle. The attack reward function $R_3$ encourages the UAV agent to actively attack and is defined using the following equation:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_3=1-{\left(\frac{q_U}{\pi }\right)}^{0.4}\end{array}\end{array}$$

(13)

Based on the contributions of $R_1$, $R_2$, and $R_3$ in the process of achieving the goal, $c_1$, $c_2$, and $c_3$ are introduced as the weights of each sub-objective reward function. The reward function $R_t$ of the UAV agent is composed of the objective reward function and the sub-objective reward functions as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R_t=R_0+c_1\ast R_1+c_2\ast R_2+c_3\ast R_3\end{array}\end{array}$$

(14)

3. UAV Maneuver Decision Algorithm

3.1. Expert Experience Storage Mechanism

Traditional reinforcement learning algorithms tend to discard transitions after learning from them once, leading to inefficient use of experience. Moreover, strong correlation between adjacent transitions is not conducive to the agent learning. In order to alleviate the problems of data correlation and experience waste, experience replay technology is introduced in the DQN algorithm [43]. Experience replay stores the agent’s experience at each time step in a dataset, accumulating transitions from multiple episodes into a replay buffer R, and uniformly samples randomly from R when performing an update, where $R={\tau }_1,\dots ,{\tau }_N,{\tau }_t=(s_t,a_t,r_t,s_{t+1})$. The current oldest transition is deleted when a new transition is stored and the replay buffer is full.

Fedus et al. [44] studied the impact of three factors in the replay buffer, namely replay capacity, age of the oldest policy, and replay ratio. The experimental results proved that an increase in replay capacity greatly improves the performance of n-step DQN series algorithms. However, not all transitions hold equal importance. Based on this idea, Schaul et al. [45] proposed a prioritized experience replay (PER) mechanism, using TD-error as a judgment basis. The experimental results showed that DQN with prioritized experience replay outperformed the original DQN. Continuing with this idea, Karimpanal et al. [46] optimized the experience storage process by collecting high-reward sequences during interactions and using prioritized experience replay during the learning process.

However, in aerial combat games, the goal is not to obtain higher reward values but to successfully attack the target UAV. In a real air game environment, the flying experience of an air game expert is often more valuable than the strategies independently explored by novice pilots. Consequently, when training the UAV agent, the replay buffer should ideally contain a higher proportion of expert experience. Although obtaining genuine expert experience can be challenging, the agent may successfully strike the target during its exploration process. Compared to random exploration experience, instances of successful target strikes can be considered as the UAV agent’s expert experience.

Based on the above analysis, this paper proposes an expert experience storage mechanism to optimize the experience storage process and uses the expert experience collected by UAV agents during the exploration process to enhance algorithm performance. When storing transitions in the replay buffer, the sequence of transitions is also recorded. In addition to experience replay technology, the proposed mechanism incorporates goal attainment determination; specifically, it assesses whether the UAV has successfully completed its strike mission. When the goal is achieved, the recorded sequence $\tau$ is regarded as an expert experience sequence from which the agent can learn. The recorded sequence $\tau$ is stored in both the original replay buffer and the expert replay buffer. Same as for the original replay buffer, the oldest expert experience sequence will be removed when the expert replay buffer is full. To balance the UAV agent’s exploration and utilization of expert experience in the air game environment, two measures are proposed: centralized learning and replay buffer replacement.

Centralized learning refers to setting multiple learning points in the replay buffer, and when the amount of stored experience reaches the learning point, the existing expert experience is learned multiple times. During UAV exploration, updates typically follow the principle of obtaining higher reward values. However, inaccurately designed reward functions can lead to situations where the agent achieves high reward values but fails to accomplish the goal. Learning from the expert replay buffer can rectify the strategy update direction, shifting the agent from being reward-oriented to goal-oriented.

Nonetheless, during the initial stage of training, learning from an expert experience sequence may not effectively correct the direction or might even steer it in the wrong direction. Hence, a centralized learning method is necessary, which entails the agent learning multiple times from the experience sequence. However, this strategy can cause the agent to become trapped in a local optimum at the beginning of training.

An intuitive solution is to perform centralized learning on the agent each time a certain amount of successful experience is gathered. As illustrated in Figure 5, the replay buffer is evenly divided into h parts, with the capacity line of each part serving as the UAV agent’s learning point. When the stored experience reaches the learning point, the existing expert experience is learned k times by the agent.

During the exploration process, the UAV agent employs the expert replay buffer to conduct gradual update learning. After centralized learning, the agent’s strategy direction is adjusted. At this point, continuing to use the original replay buffer for exploration is not suitable; instead, exploration should be based on the expert experience replay buffer. Consequently, the proposed algorithm replaces the original replay buffer simultaneously with centralized learning each time, facilitating exploration grounded in expert experience.

See Algorithm 1 for a more formal description of the proposed algorithm.

Algorithm 1: Expert Experience Storage Mechanism

Input: a reinforcement learning off-policy algorithm $\mathbb{A}$, self-learning time k, storage constant h, and expert replay buffer size N Initialize $\mathbb{A}$ Initialize replay buffer R Initialize expert replay buffer $R^{\prime }$ for $\mathrm{episode}=1,M$ do    Initialize transition sequence $\tau =\left\{\right\}$    Initialize state $s1$    for $t=1,T-1$ do       Select action $a_t$ using the policy from $\mathbb{A}$       Excute action $a_t$ and observe reward $r_t$ and $s_{t+1}$       Store transition $(s_t,a_t,r_t,s_{t+1})$ in the replay buffer R       Add transition $(s_t,a_t,r_t,s_{t+1})$ to the transition sequence $\tau$       Sample a minibatch transition ${\tau }_{batch}$ from the replay buffer R       Update weights through a minibatch transition ${\tau }_{batch}$ and $\mathbb{A}$    end    if goal achieved then       Store transition sequence      $\tau =\left\{(s_1,a_1,r_1,s_2),(s_2,a_2,r_2,s_3),\dots ,(s_t,a_t,r_t,s_{t+1},\dots )\right\}$ in the expert replay buffer $R^{\prime }$       $m=\mathrm{current}\mathrm{buffer}\mathrm{memory}\mathrm{mount}$       if $\mathrm{current}\mathrm{buffer}\mathrm{memory}\mathrm{mount}m>\frac{N}{h}\ast n$  then         $n\leftarrow n+1$     $R\leftarrow R^{\prime }$     for $j=1,k$ do       Sample a minibatch transition ${\tau }_{batch}$ from the replay buffer $R^{\prime }$       Update weights through a minibatch transition ${\tau }_{batch}$ and $\mathbb{A}$     end       end    end end

3.2. Dueling Double Deep Q Network Algorithm

DDPG [32] and PPO [35] are algorithms widely used for decision-making in continuous action space games, while DQN series of algorithms are typically used for tasks with a discrete action space. In this paper, Dueling Double Deep Q Network is selected as the algorithm for UAV maneuver decision-making. D3QN algorithm integrates the central concepts of both Double DQN [47] and Dueling DQN [48] algorithms, enhancing performance beyond the original DQN. This section provides an overview of the key components of the D3QN algorithm.

In conventional Q-learning, a Q-function is introduced, which is the expected future reward for performing a specific action in a specific state. The value of each Q-function is recorded through the Q-table, which determines the agent’s decisions based on an $ϵ$-greedy strategy [43]. The Q-table iteratively updates $Q\left(s_t,a_t\right)$ utilizing the Bellman equation:

$$\begin{array}{c}\hfill Q(s,a)\leftarrow Q(s,a)+l\left[r+\gamma \underset{a^{\prime }}{max}Q\left(s^{\prime },a^{\prime }\right)-Q(s,a)\right]\end{array}$$

(15)

where l denotes the learning rate, a hyperparameter that influences if and when the objective function converges to a local minimum. Through continuous trial and error, the agent eventually learns a convergent Q-table, which enables it to select the action that yields the maximum expected reward.

The advancement of deep learning has facilitated the integration of reinforcement learning and deep learning. The DQN algorithm, proposed by the DeepMind team in 2015 [43], employs a neural network in place of the Q-table. The current state serves as the input to the neural network, which produces Q values as output. The DQN model comprises two networks: the predict Q network and the target Q network. The predict Q network is employed to predict the Q value of each action corresponding to the current state, while the target Q network is used to predict the Q value of each action in the subsequent state. When calculating the loss function in the network, the target Q network will not be updated with the model at any time. After a certain number of iterations, the parameters of the predict Q network are transferred to the target network for correction. The objective function of network update can be defined as follows:

$$\begin{array}{c}\hfill Y_t^{\mathrm{DQN}}\equiv R_{t+1}+\gamma \underset{a}{max}Q\left(S_{t+1},a;{\theta }_t^{-}\right)\end{array}$$

(16)

However, there is an overestimation problem in DQN. After Q network initialization, the estimation of the Q value is random. Due to the maximization operation used in updating the Q value, the final Q value is often overestimated. To alleviate this problem, the Double DQN algorithm separates action selection and evaluation. In the predict Q network, it finds the action that corresponds to the maximum Q value and then uses this selected action to calculate the target Q value in the target Q network. Based on this idea, Double DQN modifies the objective function:

$$\begin{array}{c}\hfill Y_t^{\mathrm{DoubleDQN}}\equiv R_{t+1}+\gamma Q\left(S_{t+1},\underset{a}{argmax}Q\left(S_{t+1},a;{\theta }_t\right),{\theta }_t^{-}\right)\end{array}$$

(17)

The model structure of Double DQN is identical to that of DQN, encompassing both the predict Q network and the target Q network. The optimal action selection in DQN is based on the parameter ${\theta }^{-}$ of the target Q network, while the optimal action selection in Double DQN is based on the parameter $\theta$ of the predict Q network currently being updated.

Double DQN retains the model structure of DQN while enhancing the algorithm’s performance through the modification of the objective function. Contrasting with Double DQN, Dueling DQN concentrates on the neural network structure and optimizes the Q network model. The Dueling DQN algorithm introduces a new neural network structure known as the dual network.

As shown in Figure 6, the output of the Dueling DQN algorithm comprises two branches, which are the state value function V and the advantage function value A for each action. The Q value is represented by the sum of V and A. To enhance the distinguishability of V and A, Dueling DQN centralizes the advantage function and imposes constraints using the following equation:

$$\begin{array}{c}\hfill Q(s,a;\theta ,\alpha ,\beta )=V(s;\theta ,\beta )+\left(A(s,a;\theta ,\alpha )-\underset{a^{\prime }\in \left|\mathcal{A}\right|}{max}A\left(s,a^{\prime };\theta ,\alpha \right)\right)\end{array}$$

(18)

where $\beta$ is a local network parameter unique to the value function and $\alpha$ is a local network parameter unique to the advantage function.

The D3QN algorithm is established upon the foundation of the DQN algorithm, integrating concepts from both the Double DQN algorithm and the Dueling DQN algorithm to optimize the DQN model structure and optimize the objective function.

3.3. Maneuver Decision Algorithm Based on Dueling Double Deep Q Network with Expert Experience Storage Mechanism

Based on D3QN and EESM, a UAV air game maneuver decision algorithm is developed. The comprehensive algorithm framework is illustrated in Figure 7.

The current state and reward values are calculated from the UAV air game environment through Equations (5) and (6) and the reward function. The predict Q network takes the current state value as input. After processing through two fully connected layers, the predict Q network calculates the value function and advantage function separately and employs Equation (18) to derive a series of Q values. The $ϵ$-greedy strategy is used to complete the action selection and interact with the UAV air game environment.

Throughout the training process, the UAV agent acquires strategy by sampling from the replay buffer, which comprises both the original replay buffer and an expert replay buffer. The original replay buffer serves to disrupt the correlation between samples and update the network parameters at each training step, while the expert replay buffer encompasses the experience of the UAV successfully attacking the target aircraft. Centralized learning is conducted when the mission is accomplished. Upon the accumulation of a specific amount of expert experience, the contents of the expert replay buffer are duplicated into the original replay buffer.

During the process of sampling from the replay buffer, the UAV state value is simultaneously transmitted to both the target Q network and the predict Q network. The predict Q network supplies the predict Q value for action selection, while the target Q value is used for calculating the objective function in the network parameter update. Through the target Q network and the predict Q network outputs, the parameter update for the predict Q network can be accomplished based on the objective value in Equation (17). Utilizing the fixed target network method, the parameters of the predict Q network are copied to the target Q network at regular intervals. When the UAV successfully strikes the target during the training phase, expert experience is sampled from the expert replay buffer for centralized learning. The learning process is identical to that of ordinary experience sampled from the replay buffer.

The training process of the proposed algorithm is presented in Algorithm 2.

Algorithm 2: UAV Maneuver Decision Algorithm

Input: self-learning time K, storage constant h, expert replay buffer size N, discount factor $\gamma$, batch size B, learning rate l, copy network step C, $\alpha$, and $\beta$ Initialize predict Q network parameter $\theta$ Initialize target Q network parameter ${\theta }^{-}$ Initialize replay buffer R Initialize expert replay buffer $R^{\prime }$ for $\mathrm{episode}=1,M$ do    Initialize transition sequence $\tau =\left\{\right\}$    Initialize state $s_1$    for $t=1,T-1$ do       Calculate the Q value through the predict Q network       Select action $a_t$ using $ϵ$-greedy policy       Excute action $a_t$ and observe reward $r_t$ and $s_{t+1}$       Store transition $(s_t,a_t,r_t,s_{t+1})$ in the replay buffer R       Add transition $(s_t,a_t,r_t,s_{t+1})$ to the transition sequence $\tau$       Sample a minibatch transition ${\tau }_{batch}$ from the replay buffer R       for $j=1,B$ do          Compute TD-error $$Y_j^{\mathrm{DoubleDQN}}\equiv R_{j+1}+\gamma Q\left(S_{j+1},\underset{a}{argmax}Q\left(S_{j+1},a;{\theta }_j\right),{\theta }_j^{-}\right)$$          Update weights $${\theta }_{j+1}={\theta }_j+l\left(Y_j^{\mathrm{DoubleDQN}}-Q\left(S_j,A_j;{\theta }_j\right)\right){\nabla }_{{\theta }_j}Q\left(S_j,A_j;{\theta }_j\right)$$       end       Every C steps, copy weights into target network ${\theta }^{-}\leftarrow \theta$    end    if goal achieved then       Store transition sequence      $\tau =\left\{(s_1,a_1,r_1,s_2),(s_2,a_2,r_2,s_3),\dots ,(s_t,a_t,r_t,s_{t+1},\dots )\right\}$ in the expert replay buffer $R^{\prime }$       $m=\mathrm{current}\mathrm{buffer}\mathrm{memory}\mathrm{mount}$       if $\mathrm{current}\mathrm{buffer}\mathrm{memory}\mathrm{mount}m>\frac{N}{h}\ast n$ then          $n\leftarrow n+1$          $R\leftarrow R^{\prime }$          for $k=1,K$ do             Sample a minibatch transition ${\tau }_{batch}$ from the replay buffer $R^{\prime }$             for $j=1,B$ do                Compute TD-error $$Y_j^{\mathrm{DoubleDQN}}\equiv R_{j+1}+\gamma Q\left(S_{j+1},\underset{a}{argmax}Q\left(S_{j+1},a;{\theta }_j\right),{\theta }_j^{-}\right)$$                Update weights $${\theta }_{j+1}={\theta }_j+l\left(Y_j^{\mathrm{DoubleDQN}}-Q\left(S_j,A_j;{\theta }_j\right)\right){\nabla }_{{\theta }_j}Q\left(S_j,A_j;{\theta }_j\right)$$             end          end       end    end end

4. Simulation Experiment and Analysis

4.1. Simulation Experiment Basic Setup

This study establishes a simulation environment on a Windows 10 operating system with hardware support from an NVIDIA RTX2060 GPU and an Intel Core i7-9700k processor. The proposed UAV maneuver decision-making algorithm is implemented using the Pytorch module.

The basic parameters of the simulation environment are shown in

Table 3. The basic parameters of the simulation environment.

Parameter	Value
Threshold distance $D_{threshold}$	5000 m
State space normalization parameter $v_1$	20 m/s
State space normalization parameter $v_0$	80 m/s
UAV velocity range	[60 m/s, 100 m/s]
attack distance $D_{attack}$	500 m
learning rate l	$0.0003$
initial greedy coefficient $ϵ$	1
greedy coefficient $ϵ$ decay rate	$3\times {10}^{-5}$
discount factor $\gamma$	$0.98$
reward factor $c_1$	$0.7$
reward factor $c_2$	$0.15$
reward factor $c_3$	$0.15$
replay buffer size	${10}^6$
batch size	256
storage constant h	10
self-learning time k	10
number of hidden layers	2
number of network cells in each layer	256

. The threshold distance $D_{threshold}$ = 5000 m, state space normalization parameter $v_1$ = 20 m/s, $v_0$ = 80 m/s, UAV velocity $v\in$ [60 m/s, 100 m/s], attack distance $D_{attack}$ = 500 m, attack angle ${\theta }_{attack}={60}^{\circ }$.

The D3QN network and evaluation network in this study both have two hidden layers, with each layer consisting of 256 network cells. The Adam optimizer is used to train the D3QN network with a learning rate ł set to 0.0003. The initial greedy coefficient $ϵ$ is set to 1 and decays to 0.05 with a decay rate of $3\times {10}^{-5}$. The discount factor $\gamma =0.98$, while the reward factors are $c_1=0.7$, $c_2=c_3=0.15$, The replay buffer size is set to ${10}^6$ and the batch size to 256. The self-learning time k is set to 10.

Two simulation experiment were conducted in order to verify the proposed method using the basic parameters mentioned above. The ablation experiment was designed to test the effectiveness of the expert experience storage mechanism, while an air game simulation experiment was set up in a simulation environment to verify the performance of the maneuver decision algorithm.

4.2. Expert Experience Storage Mechanism Ablation Experiment

In this paper, an expert experience storage mechanism is proposed to solve exploration efficiency in the training model. Prioritized experience replay [45] is an experience-replay-based reinforcement learning technique that adjusts the sampling probability of experience samples based on their priority. To evaluate the effectiveness of the expert experience storage mechanism, ablation studies are performed on the basis of two mechanisms, and four groups of experiment are compared:

• Training and testing with both PER and the expert experience storage mechanism;
• Training and testing with only PER and without the expert experience storage mechanism;
• Training and testing with only the expert experience storage mechanism and without PER;
• Training and testing without either PER or the expert experience storage mechanism.

In each experiment, the initial UAV situation of each training step is shown in

Table 4. The initial UAV situation at each training step.

	Initial Position $(\mathit{x},\mathit{y},\mathit{z})$	v (m/s)	$\mathit{\theta }$	$\mathit{\psi }$	$\mathit{\varphi }$
Our UAV	(10,300, 20,000, 25,000)	60	0	0	0
Target UAV	(11,300, 21,000, 26,000)	60	0	0	0

. The target UAV adopts the strategy of uniform speed and direct flight, which mean the target UAV executes $a_1$ in the action space. Our UAV is initially behind the target UAV, and the previously stated maneuver decision algorithm training method is used to learn the decision-making process.

During the training process, the total reward value and experience replay time obtained are recorded in each episode. The total reward value received in each episode can be used to reflect the performance of the algorithm under different sampling mechanisms. The experience replay time refers to the time it takes for the replay buffer to be sampled from storage, indicating the efficiency of different sampling mechanisms. The effectiveness of the proposed mechanism is validated by comparing the total reward values and experience replay times obtained for each experiment.

The training curves for various sampling mechanisms are presented in Figure 8.

As shown in Figure 8, without the use of EESM and PER mechanisms, the maximum average reward value obtained by the UAV agent is −13.5. When the original sampling mechanism was used, the UAV agent received a relatively low reward, indicating that the agent did not learn the strategy to obtain higher rewards as the training steps increased. On the other hand, when the PER mechanism was used, the UAV agent obtained a maximum average reward value of 42.5, which was higher than that obtained by using the original sampling mechanism. The proposed EESM mechanism allowed the UAV agent to achieve a maximum average reward value of 102.3 in the air game simulation environment, outperforming the PER mechanism and the original sampling mechanism. This suggests that the EESM mechanism is effective in helping the UAV agent learn better attack strategies. The maneuver decision-making algorithm that uses both PER and EESM mechanisms achieves a maximum average reward value of 106.6, which is similar to the reward obtained by using the EESM alone. However, with PER and EESM, the algorithm’s overall convergence speed is faster, suggesting that PER has a certain acceleration effect on the algorithm’s convergence speed when fully utilizing expert experience.

Table 5. The details of each ablation experiment.

	PER	EESM	Experience Replay Time	Max Average Reward
Experiment 1	✓	✓	1577.0 ms	106.6
Experiment 2	✓		1149.2 ms	42.5
Experiment 3		✓	178.5 ms	102.3
Experiment 4			101.0 ms	−13.5

displays the experience replay times for different sampling mechanisms. The EESM mechanism has an experience replay time of 178.5 ms, slightly higher than the experience replay time without the EESM and PER mechanisms, which is 101.0 ms. The experience replay times for the PER mechanism and the combined PER and EESM mechanisms are 1149.2 ms and 1577.0 ms, respectively. These replay times indicate that the proposed EESM mechanism has a slightly slower replay speed compared to the original mechanism but is much faster than the PER mechanism, which is typically implemented using a sumtree.

The overall algorithm performance should be assessed by considering the training curve and experience replay time. By comparing the reward curves of the four experimental groups, it is observed that the EESM and PER mechanisms improve the UAV intelligent agent’s learning effect, and the maneuver decision algorithm that uses the EESM mechanism obtains higher reward values. Then, the algorithm that uses both EESM and PER mechanisms achieves approximately the maximum reward value compared to the algorithm that only uses the EESM mechanism but with a faster convergence rate. In addition, it is found that the PER mechanism is time-consuming, while the experience replay time using the EESM mechanism is slightly higher than the original sampling mechanism but reduced by 81.3% compared to the time taken using the PER mechanism. Taking into account both the maximum reward value and sampling time, the following can be concluded:

• Without using the EESM and PER mechanisms, although the experience replay time of the algorithm is less, the UAV intelligent agent has difficulty learning an effective strategy for attacking the target UAV.
• The UAV intelligent agent can achieve a certain training effect when only using the PER mechanism, but the replay experience time is greatly increased. The algorithm’s performance is further improved after introducing the proposed EESM mechanism.
• The EESM mechanism proposed in this study can significantly improve the performance of the algorithm with less replay time: it reduces the replay time of the algorithm by 81.3% compared to the PER mechanism and makes the UAV intelligent agent obtain a higher maximum average reward value.

4.3. Air Game Simulation Experiment Based on Maneuver Decision Algorithm

In this section, an air game simulation experiment was conducted using the maneuver decision algorithm with the proposed expert experience storage mechanism. In the simulation experiment, our UAV maneuver decision algorithm adopts the proposed expert experience storage mechanism. The initial situation of the UAV in each training step was the same as the parameters presented in Table 3. The target UAV adopts three different strategies:

• Uniform speed and direct flight: Target UAV executes $a_1$ in the action space.
• Uniform speed and hovering: Target UAV executes $a_4$ in the action space.
• Uniform speed and serpentine maneuver: Target UAV executes $a_4$ and $a_5$ periodically, with the period being 35 simulation steps.

Figure 9 and Figure 10 show the training curve and typical air game flight trajectory of our UAV agent when the target UAV adopts the strategy of uniform speed and direct flight. As shown in Figure 9, as the number of training steps increases, the proposed autonomous maneuver decision-making algorithm gradually converges and finally obtains the maximum reward value of 298.7. As depicted in Figure 10, the initial position of the target UAV is above our UAV in horizontal flight. Our UAV observes the state information of the target UAV and uses the proposed autonomous maneuver decision-making algorithm to select the appropriate maneuvering actions. After flying straight for a brief period, our UAV initiates a climb and attacks the target UAV positioned from below.

The results of the air game simulation experiment with the target UAV adopting the uniform speed and hovering strategy is shown in Figure 11 and Figure 12. As illustrated in Figure 11, the proposed autonomous maneuver decision-making algorithm gradually converges with increasing training steps, achieving a maximum reward of 262.3. Figure 12 depicts the target UAV hovering with our UAV’s initial position below. Using the proposed autonomous maneuver decision-making algorithm, our UAV takes the state information of the aerial game as inputs and outputs real-time maneuvering actions. Then, our UAV adjusts its flight direction and performs a climb to occupy the optimal attack position above the target UAV to complete the attack.

The results of the air game simulation experiment with the target UAV adopting uniform speed and serpentine maneuver is shown in Figure 13 and Figure 14. As illustrated in Figure 13, the proposed autonomous maneuver decision-making algorithm gradually converges, ultimately achieving a maximum reward of 299.6. Figure 14 shows the target UAV performing a serpentine maneuver while our UAV adopts the proposed algorithm. As the target UAV performs the serpentine maneuver, our UAV adjusts its direction and moves straight for a short period. Finally, our UAV climbs and positions itself above the enemy UAV for an optimal attack.

5. Conclusions

When utilizing deep reinforcement learning for UAV air game maneuver decision-making, poor algorithm performance may result from sample exploration and utilization inefficiency. To address this issue, this paper proposes an expert experience storage mechanism that can effectively enhance algorithm performance with less experience replay time by utilizing successful expert experience in UAV air games. Based on the Dueling Double Deep Q Network and expert experience storage mechanism, a UAV air game maneuver decision-making model is established in this study, and its effectiveness is verified through simulation experiments. The results demonstrate that the proposed expert experience storage mechanism can improve the algorithm’s performance with less experience replay time. Furthermore, the proposed maneuver decision-making algorithm is effective in providing decisions to strike target UAVs employing different fixed strategies.

In future work, an analysis of the algorithm’s computational complexity and inference speed will be conducted with respect to the limited computational load of UAVs, and the aircraft will be modeled in detail on a semi-physical simulation platform to simulate the real flight conditions of the UAV.