Bio-Inspired Fission–Fusion Control and Planning of Unmanned Aerial Vehicles Swarm Systems via Reinforcement Learning

Abstract

Swarm control of unmanned aerial vehicles (UAV) has emerged as a challenging research area, primarily attributed to the presence of conflicting behaviors among individual UAVs and the influence of external movement disturbances of UAV swarms. However, limited attention has been drawn to addressing the fission–fusion motion of UAV swarms for unknown dynamic obstacles, as opposed to static ones. A Bio-inspired Fission–Fusion control and planning via Reinforcement Learning (BiFRL) algorithm for the UAV swarm system is presented, which tackles the problem of fission–fusion behavior in the presence of dynamic obstacles with homing capabilities. Firstly, we found the kinematics models for the UAV and swarm controller, and then we proposed a probabilistic starling-inspired topological interaction that achieves reduced overhead communication and faster local convergence. Next, we develop a self-organized fission–fusion control framework and a fission decision algorithm. When dealing with various situations, the swarm can autonomously re-configure itself by fissioning an optimal number of agents to fulfill the corresponding tasks. Finally, we design a sub-swarm confrontation algorithm for path planning optimized by reinforcement learning, where the sub-swarm can engage in encounters with dynamic obstacles while minimizing energy expenditure. Simulation experiments demonstrate the capability of the UAV swarm system to accomplish self-organized fission–fusion control and planning under different interference scenarios. Moreover, the proposed BiFRL algorithm successfully handles adversarial motion with dynamic obstacles and effectively safeguards the parent swarm.

1. Introduction

UAV swarm systems exhibit collective behavior among drones and their interaction with the environment [1]. Nature frequently serves as an inspiration source for the development of these systems, drawing from phenomena like insect movement [2], bird flocks [3], and fish schools [4,5]. The motion of such swarms is inherently dynamic, with the group size and composition undergoing frequent changes as individual members split or merge, referred to as “fission-fusion” behavior [6,7,8]. Fission–fusion of the swarm holds significant importance in the natural world for various animal species. For instance, tree-dwelling bats rely on shared survival space within the group [9], while bird flocks enhance their chances of survival [10], and herds of bison and giraffes evade predators [11].

Recently, researchers from diverse fields have increasingly recognized the significance of fission–fusion behaviors. They have successfully emulated these behaviors to achieve robot-controlled motion [12,13,14], enhance the efficiency of swarm resource search [15], and accomplish planning objectives, such as task allocation and obstacle avoidance [1,16,17]. For example, Wang et al. [13] incorporate the fission–fusion motion into the formation controller of an underwater vehicle, Nauta et al. [15] study the population resource search with the fission–fusion concept, whereas Reséndiz-Benhumea et al. [16] integrate the fission–fusion movements observed in ant colonies into the task assignment algorithm for robot swarms. As such, the investigation of fission–fusion behavior in swarm systems holds substantial theoretical research and practical application values.

Swarm fission–fusion behavior involves the ability of individuals within a certain range to gather into a cohesive group with synchronized movement through specific interactions. Reynolds et al. [18] propose three fundamental behavioral rules to address this issue: collision avoidance, speed consistency, and mutual aggregation. Subsequently, extensive research has been conducted based on these rules for implementing swarm controls [19,20,21] and exploring swarm fission–fusion dynamics [17,22,23,24]. Nevertheless, most of these studies primarily focus on collision avoidance for static obstacles within a single swarm or during the fission–fusion process in a known environment with complete knowledge. There has been limited research on swarm planning in the presence of unknown dynamic obstacles, particularly of those with tracking capabilities. This is primarily because existing path planning strategies for static or dynamic obstacles often rely on global environment information [25,26,27,28]. For instance, Wang et al. [26] perform path planning through a sampling-based algorithm, while Garrett et al. [27] propose to address the continuous space subproblems and integrate the discrete and continuous aspects of the search process to achieve path planning objectives. In the real world, however, many dynamic obstacles remain unknown or possess tracking capabilities, continually intruding upon and pursuing the targeted swarm to disrupt its progress. Previous methods [22,23,24,25] for swarm control and planning may be significantly affected by limited success rate and resource utilization in reaching the intended destination.

Typical swarm controllers incorporate fixed-neighbor distance interactions for fission–fusion motions [1,22,29,30,31]. However, this interaction manner imposes a communication load that increases with the swarm size, making it unfeasible for large-scale swarms [25,29,30,31]. In practice, large-scale swarms are usually achieved through limited interactions among a subset of individuals, enabling the formation of expansive swarms [32,33,34]. Notably, studies have shown that each starling in a flock interacts with only six to seven of its closest neighbors, allowing for the swarm formation comprising thousands of individuals through topological interaction structures [32]. As such, numerous researchers have investigated the starling topology and its utilization in swarms [35,36,37]. However, in the context of UAV swarms, the reliance on a seven-nearest-neighbor topological interaction model often leads to local convergence, as shown in Figure 1. It still remains a problem of the local convergence to maintain swarm integrity while reducing the communication load among UAV swarm members.

Based on the above discussion, this paper focuses on investigating the bio-inspired fission–fusion control and planning of UAV swarm systems by enhancing the starling topology and incorporating a reinforcement learning algorithm. The proposed approach, referred to as the Bio-inspired Fission–Fusion Control and Planning via Reinforcement Learning (BiFRL), encompasses several key contributions. Firstly, we introduce a probabilistic starling-inspired topological interaction (PSTI) structure, which deviates from the traditional fixed-range neighbor interactions in swarm motion while effectively reducing the communication load and likelihood of local convergence. Next, a self-organized fission–fusion control framework and a fission decision algorithm are developed, which enables the swarm to autonomously divide into sub-swarms when encountering dynamic obstacles. It allows for precise control of sub-swarm units. Finally, we propose a sub-swarm confrontation via reinforcement learning (SCRL), which can efficiently plan confrontation movement with minimal energy loss through reinforcement learning techniques. This characteristic facilitates the original objectives of the parent swarm remaining unaffected by dynamic obstacles, while the sub-swarms can seamlessly reintegrate into the parent swarm once the interference ceases. Finally, the effectiveness and robustness of the proposed BiFRL algorithm for UAV swarm are validated through simulation models that consider dynamic obstacles approaching from different directions.

The present study provides the following contributions:

• We introduced a probabilistic starling-inspired topological interaction structure, which effectively reduces the communication load and likelihood of local convergence.
• A self-organized fission–fusion control framework for UAV swarms is presented that extends the existing social force model, where a fission decision algorithm is designed to manipulate the composition of sub-swarms and target the dynamic obstacles.
• We developed a reinforcement learning sub-swarm confrontation algorithm to achieve a self-organized sub-swarm against dynamic obstacles in unknown environments, which significantly improves the adaptability of UAV swarms.
• The proposed method’s feasibility and validity are demonstrated through extensive numerical simulations, accompanied by the development of several numerical evaluation indicators.

The subsequent sections of this paper are structured as follows: In Section 2, we present the formulation of the problems and review of existing models pertaining to fission–fusion control and planning in UAV swarms. Section 3 presents the proposed BiFRL framework and its components in detail. We then conduct the simulation experiments to analyze the proposed method in Section 4. Finally, the findings and conclusions of this study are presented in Section 5.

2. Problem Formulations

2.1. Unmanned Aerial Vehicles Kinematic Model

In the event that the UAV is endowed with a tri-loop autopilot for velocity, altitude control, and heading angle. The kinematic model of the UAV can be simplified as follows [38]:

$$\left\{\begin{array}{l}{\dot{x}}_i^{\mathsf{\varpi }}=v_i^{\mathsf{\varpi }}\mathit{cos}{\psi }_i^{\mathsf{\varpi }}\\ {\dot{y}}_i^{\mathsf{\varpi }}=v_i^{\mathsf{\varpi }}\mathit{sin}{\psi }_i^{\mathsf{\varpi }}\\ {\dot{h}}_i^{\mathsf{\varpi }}={\lambda }_i^{\mathsf{\varpi }}\\ {\dot{v}}_i^{\mathsf{\varpi }}=\frac{1}{{\alpha }_v}\left(v_i^{{\mathsf{\varpi }}_{inp}}-v_i^{\mathsf{\varpi }}\right)\\ {\dot{\psi }}_i^{\mathsf{\varpi }}=\frac{1}{{\alpha }_{\psi }}\left({\psi }_i^{{\mathsf{\varpi }}_{inp}}-{\psi }_i^{\mathsf{\varpi }}\right)\\ {\dot{\lambda }}_i^{\mathsf{\varpi }}=+\frac{1}{{\alpha }_h}\left(h_i^{{\mathsf{\varpi }}_{inp}}-h_i^{\mathsf{\varpi }}\right)-\frac{1}{{\alpha }_{\lambda }}{\lambda }_i^{\mathsf{\varpi }}\end{array}\right.$$

(1)

where ($x_i^{\mathsf{\varpi }},y_i^{\mathsf{\varpi }},h_i^{\mathsf{\varpi }}$) denote the position of UAV $i$ in $\mathsf{\varpi }\in \left[\mathit{sub}-\mathit{swarm}, \mathit{parent}-\mathit{swarm}\right]$; $v_i^{\mathsf{\varpi }}$, $v_i^{{\mathsf{\varpi }}_{inp}}$ are the rate and control input commands of horizontal velocity; ${\psi }_i^{\mathsf{\varpi }}$, ${\Psi }_i^{{\mathsf{\varpi }}_{inp}}$ are the rate and control input commands of the heading angle; ${\lambda }_i^{\mathsf{\varpi }}$ is altitude change rate and control input commands of agent $i$ in $\mathsf{\varpi }$; ${\alpha }_{\psi }$, ${\alpha }_v$, ${\alpha }_h$, and ${\alpha }_{\lambda }$ are the self-driving instrument control parameters. The following are the UAV flight condition constraints that are taken into account.

$$\left\{\begin{array}{l}{v}_{min}<v_i^{\mathsf{\varpi }}<v_{max}\\ \left|\dot{\chi }\right|\le {\varphi }_{max}\mathrm{g}/v_i^{\mathsf{\varpi }}\\ {\lambda }_{min}<{\lambda }_i^{\mathsf{\varpi }}<{\lambda }_{max}\end{array}\right.$$

(2)

where $v_{min}$ and $v_{max}$ are the minimal and ceiling horizontal speeds, respectively; ${\phi }_{max}$ indicates the maximum lateral overload; $\mathrm{g}$ denotes the acceleration of gravity; ${\lambda }_{min}$ and ${\lambda }_{max}$ present the minimum and maximum height change rates, respectively, which are all greater than zero.

2.2. Dynamic Obstacle Movement Model

In this research, a dynamic obstacle with an expansive sensing limit is established, and the UAVs in its proximity are selected as tracking targets. The kinematic equation governing the dynamic obstacle is formulated as follows:

$$\left\{\begin{array}{l}{\dot{x}}_{invader}=v_{invader}\\ {\dot{v}}_{invader}=u_{invader}-{\delta }_{invader}{\Vert v_{invader}\Vert }^2{v}_{invader}\end{array}\right.$$

(3)

$$\begin{gathered} u_{invader}={\kappa }_{invader}^{aut}\frac{v_{invader}}{\Vert v_{invader}\Vert }+{\kappa }^{invader} \frac{x_{min}^{son\_subgroup}-x_{invader}}{\Vert x_{min}^{son\_subgroup}-x_{invader}\Vert } \\ +{\delta }_{invader}{\Vert v_{invader}\Vert }^2{v}_{invader} \end{gathered}$$

(4)

where $x_{invader}=\left(x_{invader},y_{invader},h_{invader}\right)\in R^n$ represents the three-dimensional position coordinates of the dynamic obstacle, $v_{invader}\in R^n$ and $u_{invader}\in R^n$ are the velocity vector rate and control input of the dynamic obstacle, ${\kappa }_{invader}^{aut}$ and ${\kappa }^{invader}$ are the inertia coefficient and racking factor of the dynamic obstacle, and $-{\delta }_{invader}{\Vert v_{invader}\Vert }^2{v}_{invader}$ is the frictional force generated by the interaction between the dynamic obstacle and its surrounding environment.

2.3. Traditional Unmanned Aerial Vehicles Swarm Dynamics Model

In the present investigation, two swarm systems are present $\mathsf{\varpi }\in \left[sub-swarm, parent-swarm\right]$. In the present investigation, two swarm systems comprising a total of N agents are examined. It is observed that these swarms traverse through a three-dimensional space without any imposed boundary constraints. It is noteworthy that the two swarms under consideration are equivalent and autonomous, with each adhering to its own set of swarm movement regulations. The subsequent double integrator governs the motion of the agent:

$$\left\{\begin{array}{l}{\dot{\chi }}_i^{\mathsf{\varpi }}=v_i^{\mathsf{\varpi }}\\ m_i^{\mathsf{\varpi }}{\dot{v}}_i^{\mathsf{\varpi }}=u_i^{\mathsf{\varpi }}-\xi {\Vert v_i^{\mathsf{\varpi }}\Vert }^2{v}_i^{\mathsf{\varpi }}\end{array}\right.$$

(5)

where ${\chi }_i^{\mathsf{\varpi }}=\left(x_i^{\mathsf{\varpi }},y_i^{\mathsf{\varpi }},h_i^{\mathsf{\varpi }}\right)\in R^3$ is the position of UAV $i$ in $\mathsf{\varpi }\in \left[sub-swarm, parent-swarm\right]$, $v_i^{\mathsf{\varpi }}\in R^3$, indicating the velocity vector, $u_i^{\mathsf{\varpi }}\in R^3$ denotes the control input acting on; $m_i^{\mathsf{\varpi }}$ is the quality, $-\xi {\Vert v_i^{\mathsf{\varpi }}\Vert }^2{v}_i^{\mathsf{\varpi }}$ is the friction against air, and $\xi$ is the damping factor of air.

The majority of internal interactions between agents adhere to the cohesion alignment [22] and separation rules [21]. A prototypical implementation is outlined as follows:

$$u_i^{\mathsf{\varpi }}=u_i^{{\mathsf{\varpi }}_{vel}}+u_i^{{\mathsf{\varpi }}_{gui}}+u_i^{{\mathsf{\varpi }}_{pos}}+\xi {\Vert v_i^{\mathsf{\varpi }}\Vert }^2{v}_i^{\mathsf{\varpi }}+{\gamma }^{ine}{v}_i^{\mathsf{\varpi }}$$

(6)

where $u_i^{{\mathsf{\varpi }}_{vel}}$ is the velocity alignment, $u_i^{{\mathsf{\varpi }}_{gui}}$ is the navigation function, $u_i^{{\mathsf{\varpi }}_{gui}}$ is navigational force, $u_i^{{\mathsf{\varpi }}_{pos}}$ is the position of cooperation, and ${\gamma }^{ine}$ is the inertia coefficient. $u_i^{{\mathsf{\varpi }}_{pos}}$ is defined as follows:

$$u_i^{{\mathsf{\varpi }}_{pos}}={\displaystyle \sum }_{j\in N_i^{\mathsf{\varpi }}\left(t\right)}{\Gamma }^{pos}\Vert d_{ij}^{\mathsf{\varpi }}\Vert \left(1-\left({(\frac{l_a^{\mathsf{\varpi }}\Vert {\chi }_j^{\mathsf{\varpi }}-{\chi }_i^{\mathsf{\varpi }}\Vert }{{\chi }_j^{\mathsf{\varpi }}-{\chi }_i^{\mathsf{\varpi }}})}^2\right)\mathit{exp}(\frac{l_c^{\mathsf{\varpi }}}{d_{ij}^{\mathsf{\varpi }}})\right)$$

(7)

where ${\Gamma }^{pos}$ is the coefficient of position cooperation, $l_a^{\mathsf{\varpi }}$, $l_c^{\mathsf{\varpi }}$ are the agent desire spacing and the motion attenuation factor, $d_{ij}^{\mathsf{\varpi }}$ is the distance between two agents, $N_i^{\mathsf{\varpi }}$ is utilized to represent the number of agents that are interacting with agent $i$ in $\mathsf{\varpi }$, and $N_i^{\mathsf{\varpi }}\left(t\right)$ is the collection of the neighbors in $\mathsf{\varpi }$ that are interacting with agent $i$ at time $t$.

Typical definitions of $u_i^{{\mathsf{\varpi }}_{vel}}$ is given as follows:

$$u_i^{{\mathsf{\varpi }}_{vel}}=\frac{{\Gamma }^{vel}}{N_i^{\mathsf{\varpi }}}{\displaystyle \sum }_{j\in N_i^g\left(t\right)}\left(v_j^{\mathsf{\varpi }}-v_i^{\mathsf{\varpi }}\right)$$

(8)

where ${\Gamma }^{vel}$ is the parameter of $u_i^{{\mathsf{\varpi }}_{vel}}$. Equation (8) represents the widely recognized “velocity consensus” algorithm, which allows the swarm to quickly reach a common speed and sustain a homogeneous state [18]. The present study aimed to investigate the ability of a swarm using Equation (8) to perform self-organized fission movements when only a few agents in the swarm detect the presence of an obstacle. However, researcher findings revealed that the swarm was unable to execute self-organized fission movements in such a scenario [34]. To address this limitation, Yang et al. [17] introduced an intermittent selective mechanism that allows swarms encountering static obstacles to engage in fission–fusion motion. However, when confronted with dynamic obstacles possessing tracking capabilities or uncertain interference directions, the UAV swarm often struggles to complete its flight with optimal resource utilization. Furthermore, such situations may severely disrupt the normal movement of the swarm. The effects of dynamic obstacles on traditional swarm motion are illustrated in Figure 2.

Hence, it is of great interest to develop a new control algorithm that facilitates the self-organized fission–fusion of UAV swarms in three-dimensional space while maintaining a lightweight structure and low communication costs. In this study, we propose a probabilistic starling-inspired topological interaction approach, which enables the UAV swarm to execute fission–fusion motions while minimizing communication requirements and reducing local convergence among swarm members. Additionally, we propose a sub-swarm confrontation algorithm based on reinforcement learning that is designed specifically for unknown dynamic obstacles with tracking capabilities. This algorithm allows the sub-swarm to self-organize and effectively confront unknown dynamic obstacles with minimal energy loss while simultaneously safeguarding the parent swarm from the disruptive influence of these obstacles.

2.4. Conversion Relations between Swarm Controller and Kinematic Model

The $u_i^{\mathsf{\varpi }}$ obtains the autopilot control input for UAV is given as follows:

$$u_i^{\mathsf{\varpi }}=\left(\begin{array}{c}{u}_i^{{\mathsf{\varpi }}_x}\\ u_i^{{\mathsf{\varpi }}_y}\\ u_i^{{\mathsf{\varpi }}_h}\end{array}\right)$$

(9)

$$\left\{\begin{array}{l}{v}_i^{{\mathsf{\varpi }}_{inp}}={\alpha }_v\left(u_i^{{\mathsf{\varpi }}_x}\mathit{cos}{\psi }_i^{\mathsf{\varpi }}+u_i^{{\mathsf{\varpi }}_y}\mathit{sin}{\psi }_i^{\mathsf{\varpi }}\right)+v_i^{\mathsf{\varpi }}\\ {\psi }_i^{{\mathsf{\varpi }}_{inp}}=\frac{{\alpha }_{\psi }}{v_i^{\mathsf{\varpi }}}\left(u_i^{{\mathsf{\varpi }}_x}\mathit{cos}{\psi }_i^{\mathsf{\varpi }}-u_i^{{\mathsf{\varpi }}_y}\mathit{sin}{\psi }_i^{\mathsf{\varpi }}\right)+{\psi }_i^{\mathsf{\varpi }}\\ h_i^{{\mathsf{\varpi }}_{inp}}=h_i+\frac{{\alpha }_h}{{\alpha }_{\lambda }}{\lambda }_i^g+{\alpha }_h{u}_i^{{\mathsf{\varpi }}_h}\end{array}\right.$$

(10)

where $u_i^{{\mathsf{\varpi }}_x}$, $u_i^{{\mathsf{\varpi }}_y}$ are the swarm control inputs in the horizontal direction and $u_i^{{\mathsf{\varpi }}_h}$ is the control inputs in the height direction.

The output values from the unmanned aerial vehicle dynamics model can be translated into vectors denoting both the position and velocity. These vectors serve as the inputs for the UAV swarm controller, as outlined below:

$$\left\{\begin{array}{l}{x}_i^{\mathsf{\varpi }}=\left(x_i^{\mathsf{\varpi }},y_i^{\mathsf{\varpi }},h_i^{\mathsf{\varpi }}\right)\\ v_i^{\mathsf{\varpi }}=\left(v_i^{\mathsf{\varpi }}\mathit{cos}{\psi }_i^{\mathsf{\varpi }},v_i^{\mathsf{\varpi }}\mathit{sin}{\psi }_i^{\mathsf{\varpi }},{\lambda }_i^{\mathsf{\varpi }}\right)\end{array}\right.$$

(11)

3. Bio-Inspired Fission–Fusion Control and Planning via Reinforcement Learning Algorithm

The proposed BiFRL algorithm consists of four parts. First, we propose a probabilistic starling-inspired topological Interaction that provides a starling communication structure for UAV swarms that avoids local convergence. Second, in the SFCRL algorithm, we have developed a self-organized fission–fusion control framework tailored for unmanned aerial vehicle swarms. Then, on this basis, we propose a fission decision algorithm and a sub-swarm confrontation via reinforcement learning to realize the effect of a controllable number of sub-swarm in fission–fusion swarm movement and confrontation movement facing unknown dynamic obstacles.

3.1. Probabilistic Starling-Inspired Topological Interaction

In practical scenarios, interaction structures that rely on fixed-distance neighbors often result in large communication overheads, thus hindering the possibility of large-scale UAV swarming in real-world applications. To address the above issues, we present a probabilistic starling-inspired topological interaction approach against the existing seven-nearest-neighbor topological interaction structure in Algorithm 1. It draws inspiration from the topological interaction observed in starling flocks and offers a solution to mitigate the communication burden associated with fixed-distance neighbors while reducing the occurrence of localized swarming. It introduces a probabilistic decision model that performs a probabilistic decision when a swarm of UAVs swarms a localized convergence, and if two swarms are identified as locally convergent, the UAVs in the swarm will interact with the UAVs that are farther away from them as the new topology.

Algorithm 1: Probabilistic starling-inspired topological interaction algorithm

Input: $N_i^{\mathsf{\varpi }}$, $N_{top}$, $d_N^{\mathsf{\varpi }}$ Output: $N_i^{\mathsf{\varpi }}\left(t\right)$ Function topological interaction  if  $N_i^{\mathsf{\varpi }}\le N_{top}$    $n_{now}$ $\leftarrow N_i^{\mathsf{\varpi }}$    for  $o \leftarrow$ N − 1 do       find $\min (d_{ij}^{\mathsf{\varpi }})$ in $d_N^{\mathsf{\varpi }}$       $n_{d_{ij}}^{\mathsf{\varpi }} \leftarrow \min (d_{ij}^{\mathsf{\varpi }})$     if  $d_{ij}^{\mathsf{\varpi }}$ < $R_{Radius}$ && $N_i^{\mathsf{\varpi }}\le N_{top}$ − 2 then       $N_i^{\mathsf{\varpi }}\left(t\right)$ $\leftarrow$ the agent j from swarm{}       $n_{now}$ $\leftarrow n_{now}$ + 1       $N_i^{\mathsf{\varpi }} \leftarrow \le N_{top}$     end if     if $N_i^{\mathsf{\varpi }}=N_{top}$ − 1 then         $\mathit{\varrho }={\mathit{e}}^{{{\rm Y}}_{\mathit{rad}}}\left(\max \left(d_{ij}^{\mathsf{\varpi }}\right)-\min (d_{ij}^{\mathsf{\varpi }}\right))$      if $\mathit{\varrho }>{\mathit{\varrho }}_{\mathit{rad}}$       $p_{d_{ij}}^{\mathsf{\varpi }}$ $\leftarrow$ choose the $\min (d_{ij}^{\mathsf{\varpi }})$ agent from             swarm{}       $o_{d_{ij}}^{\mathsf{\varpi }}$ $\leftarrow$ choose the $\min (d_{ij}^{\mathsf{\varpi }})$ agent from             swarm{except($n_{d_{ij}}^{\mathsf{\varpi }}$)}       if $\left|d_{ip}^{\mathsf{\varpi }}-d_{io}^{\mathsf{\varpi }}\right|\le {\varrho }_{max}$ then         $N_i^{\mathsf{\varpi }}\left(t\right)$ $\leftarrow$ the agent p from swarm{}        $n_{now}$ $\leftarrow n_{now}$ + 1        $N_i^{\mathsf{\varpi }} \leftarrow N_{top}$       else       $N_i^{\mathsf{\varpi }}\left(t\right)$ $\leftarrow$ the agent o from swarm{}       end if      end if      o $\leftarrow N$ − 1    end if   end for  end if  return sub-swarm end function

3.2. Self-Organized Fission–Fusion Control Framework

The fission–fusion dynamics within a swarm involve two contrasting and competitive behaviors. The fusion behavior necessitates the formation of a collectively coordinated ensemble among all individuals, while the fission behavior requires a disruption of the original order, giving rise to distinct smaller sub-swarms [5,19]. To account for the influence of dynamic obstacles, we integrate intrusive forces into the self-organized fission–fusion control algorithm, as delineated below:

$$\left\{\begin{array}{l}{\dot{\chi }}_i^{\mathsf{\varpi }}=v_i^{\mathsf{\varpi }}\\ m_i^{\mathsf{\varpi }}{\dot{v}}_i^{\mathsf{\varpi }}={\gamma }^{ine}{u}_i^{\mathsf{\varpi }}+\underset{u_i^{{\mathsf{\varpi }}_{pos}}}{\underbrace{{\wp }_i^{\mathsf{\varpi }}{u}_i^{{\mathsf{\varpi }}_{pls}}+\left(1-{\wp }_i^{\mathsf{\varpi }}\right)u_{\mathrm{invaders}}^{pos}}}\\ \hspace{1em}\hspace{1em}\hspace{1em}+\underset{u_i^{{\mathsf{\varpi }}_{vel}}}{\underbrace{{\wp }_i^{\mathsf{\varpi }}{u}_i^{{\mathsf{\varpi }}_{vll}}+\left(1-{\wp }_i^{\mathsf{\varpi }}\right)u_{\mathrm{invaders}}^{vel}}}\\ \hspace{1em}\hspace{1em}\hspace{1em}-\xi {\Vert v_i^{\mathsf{\varpi }}\Vert }^2{v}_i^{\mathsf{\varpi }}+\zeta {\vartheta }_i^{\mathsf{\varpi }}+{\wp }_i^{\mathsf{\varpi }}{u}_i^{{\mathsf{\varpi }}_{gui}}\end{array}\right.$$

(12)

where ${\gamma }^{ine}{v}_i^{\mathsf{\varpi }}$ represents the inertial term associated with the velocity of the UAV; $\zeta {\vartheta }_i^{\mathsf{\varpi }}$ pertains to the stochastic disturbance term generated in the context of the UAV. To address the interference posed by dynamic obstacles, we extend velocity coordination in the following manner:

$$u_i^{{\mathsf{\varpi }}_{vel}}={\displaystyle \sum }_{j\in N_i^{\mathsf{\varpi }}\left(t\right)}{\Gamma }^{vel}\left(v_j^{\mathsf{\varpi }}-v_i^{\mathsf{\varpi }}\right)+\left(1-{\wp }_i^{\mathsf{\varpi }}\right)\underset{u_{\mathrm{invaders}}^{vel}}{\underbrace{\left(u_{tra}+u_{lur}\right)}}$$

(13)

where $u_{lur}$ represents the attractive force generated when a sub-group identifies dynamic obstacles. $u_{tra}$ is the capture force generated when a sub-swarm engages in the capture of dynamic obstacles, with a detailed analysis of the specific mechanisms behind the capture force provided in Section 3.4. ${\wp }_j^{\mathsf{\varpi }}$ is the state of the agent. We have taken into consideration the impact of dynamic obstacles on sub-swarm and expanded the position cooperation term based on Equation (7). The definition of position cooperation is outlined as follows:

$$\begin{array}{l}{u}_i^{{\mathsf{\varpi }}_{pos}}=\left(1-{\wp }_i^{\mathsf{\varpi }}\right){\epsilon }_{invader}\mathit{exp}(-d_{iinvader}^{\mathsf{\varpi }})\\ \hspace{1em}\hspace{1em}+{\displaystyle {\displaystyle \sum }_{j\in N_i^{\mathsf{\varpi }}\left(t\right)}}{\Gamma }^{pos}\Vert d_{ij}^{\mathsf{\varpi }}\Vert \left(1-\left({(\frac{l_a^{\mathsf{\varpi }}\Vert x_j^{\mathsf{\varpi }}-x_i^{\mathsf{\varpi }}\Vert }{x_j^{\mathsf{\varpi }}-x_i^{\mathsf{\varpi }}})}^2\right)\mathit{exp}(\frac{l_c^{\mathsf{\varpi }}}{d_{ij}^{\mathsf{\varpi }}})\right)\end{array}$$

(14)

where ${\epsilon }_{invader}$ is the position interference coefficient of the dynamic obstacle, and $d_{iinvader}^{\mathsf{\varpi }}$ represents the distance between the dynamic obstacle and agent $i$. According to Equation (14), when the sub-swarm is tighter from the dynamic obstacle, the further the UAVs of the sub-swarm are from each other to prevent the agents in the sub-swarm from getting too close to each other to be more vulnerable togetherly to attack.

3.3. Fission Decision Algorithm

Algorithm 1 has depicted the establishment of the interaction topology for the entire swarm through a limited number of interactions among agents. We further illustrate the sub-swarm selection mechanism employed by the swarm when encountering an obstacle, as shown in Algorithm 2. Under the influence of topological interactions, the algorithm selectively organizes sub-swarm based on the interference direction of dynamic obstacles, concurrently achieving a controllable number of agents within the swarm.

Algorithm 2: Fission decision algorithm

Input: $d_N^{\mathsf{\varpi }}$, $N_i^{\mathsf{\varpi }}\left(t\right)$ Output: sub-swarm Function sub-swarm selection if  $\min (d_{ij}^{\mathsf{\varpi }})$ < $R_{Radius}$ then   $n_{now}$ $\leftarrow$ 0   for i $\leftarrow$ 1: N do    if $n_{now}$ $\le n_{ex-now}$ && ${\phi }_i^{\mathsf{\varpi }}\ne 0$ then     {}sub-swarm $\leftarrow$ choose $\min (d_{ij}^{\mathsf{\varpi }})$ agent     from swarm{except(${\phi }_i^{\mathsf{\varpi }}=0$)}     ${\phi }_i^{\mathsf{\varpi }}=0$      $n_{now}$ $\leftarrow n_{now}$ + 1     $N_i^{\mathsf{\varpi }}\left(t\right) \leftarrow$ Algorithm 1      if ${\phi }_{N_i^{\mathsf{\varpi }}\left(t\right)}^{\mathsf{\varpi }}\ne 0$ then      {}sub-swarm $\leftarrow$ choose $N_i^{\mathsf{\varpi }}\left(t\right)$ agent      {1,$n_{now}$}sub-swarm $\leftarrow$ {}sub-swarm       $n_{now}$ $\leftarrow$ + $n${}sub-swarm      ${\phi }_{\mathrm{choose} N_i^{\mathsf{\varpi }}\left(t\right) \mathrm{agent}}^{\mathsf{\varpi }}=0$      end if    end if    if $n_{now}$ == $n_{ex-now}$ then     break    end if   end for   if $min\left(d_{iinvader}^{\mathsf{\varpi }}\right)\in {ʑ}_{iva}$ then           $u_{tra}=-v_{iinvader}^{\mathsf{\varpi }}/\left|v_{iinvader}^{\mathsf{\varpi }}\right|+-\xi {\Vert v_i^{\mathsf{\varpi }}\Vert }^2{v}_i^{\mathsf{\varpi }}$   else                $u_{lur} \leftarrow \mathrm{Algorithm} 3$   end if end if return sub-swarm end function

3.4. Sub-Swarm Confrontation via Reinforcement Learning

After splitting the sub-swarms, we formulate the sub-swarm path planning problem with a Markov decision process (MDP). An MDP provides a mathematical framework for modeling sequential decision-making problems where an agent must choose actions in a sequence to achieve a desired goal. The MDP can be denoted as a 4-tuple: $〈S, A, P, R〉$, where $S$ is the set of state $s_t$, $A$ is the set of available actions ($a_t$) of the UAV, $P$ is the transition probability distribution, and $R$ defined as $S\times A\to R$, which is the reward function.

In time slot $t$, the state $S$ in the environment can be denoted as $S_t=\left\{c_{ps}, c_{ss}, c_e, c_t\right\}$, which indicates the coordinate of the parent swarm, sub-swarm, dynamic obstacle, and target, respectively. The action of a UAV can be written as $A_t=\left\{v_t, {\theta }_t\right\}$, which denote the flying speed and direction of the UAV, respectively.

The reward function in time slot $t$ is comprised of three parts, which are described as

$$R_t=r_{s2e}+r_{s2t}+r_{e2p}$$

(15)

The first term $r_{s2e}$ is a reward for the distance between the sub-swarm and the dynamic obstacle, denoted as

$$r_{s2e}=2\times \left(e^{-\frac{{\left(d_{s2e}-d_{s2e}^{mean}\right)}^2}{2{\sigma }^2}}-0.5\right)+\left\{\begin{array}{r}\alpha \log (\frac{d_{s2e}}{d_{s2e}^{safe}}), d_{s2e}<d_{s2e}^{safe}\\ 0, d_{s2e}\ge d_{s2e}^{safe}\end{array}\right.$$

(16)

where $d_{s2e}$ is the distance between the sub-swarm and the dynamic obstacle, $d_{s2e}^{safe}$ and $d_{s2e}^{capture}$ are the safe and capture distance between the sub-swarm and dynamic obstacle, respectively. This term is parameterized through $\sigma$ and $\alpha$.

The second term ($r_{s2t}$) is a reward for the distance between the sub-swarm and the target, given by

$$r_{s2t}=-\beta d_{s2t}+c_1$$

(17)

where $d_{s2t}$ is the distance between the sub-swarm and the target, and $\beta$ and $c_1$ are parameters for adjustment.

The third term ($r_{e2p}$) is a reward for the distance between the dynamic obstacle and the parent swarm, and it is described as

$$r_{e2p}=\left\{\begin{array}{r}\gamma clip(\log \left(\frac{d_{e2p}}{d_{e2p}^{min}}-1\right),-1, 1), d_{e2p}>d_{e2p}^{min}\\ c_2, d_{e2p}\le d_{e2p}^{min}\end{array}\right.$$

(18)

where $d_{e2p}$ means the distance between the dynamic obstacle and the parent swarm, $d_{e2p}^{min}$ means the minimum distance between the dynamic obstacle and parent swarm, $\gamma$ and $c_2$ are parameters for adjustment, and $clip\left(\log \left(\frac{d_{e2p}}{d_{e2p}^{min}}-1\right),-1, 1\right)$ is a mathematical operation which removes the incentive for moving $\log \left(\frac{d_{e2p}}{d_{e2p}^{min}}-1\right)$ outside of the interval $\left[-1, 1\right]$. The sub-swarm is captured by the dynamic obstacle if $d_{e2p}<=d_{e2p}^{min}$.

To address the path planning problem of the sub-swarm, we propose an RL-based algorithm. Reinforcement learning is an effective approach for tackling sequential decision problems in MDPs, aiming to maximize the cumulative reward within an episode. Among various RL algorithms, the proximal policy optimization (PPO) algorithm stands out as one of the most efficient methods for policy optimization. Another notable off-policy RL algorithm is the soft actor-critic (SAC) algorithm, which demonstrates high sample efficiency. Additionally, deep Q-learning (DQN) is a widely used RL method, which, however, is limited to discrete action spaces. In this study, we introduce the SAC algorithm to realize the antimotion of sub-swarm with dynamic obstacles.

Algorithm 3: Sub-swarm confrontation algorithm

Input: ${\mathit{c}}_{\mathit{t}}^{\mathit{parent}-\mathit{swarm}}$, ${\mathit{c}}_{\mathit{t}}^{\mathit{sub}-\mathit{swarm}}$, ${\mathit{c}}_{\mathit{t}}^{\mathit{enemy}}$, ${\mathit{c}}_{\mathit{t}}^{\mathit{target}}$ Output: $u_{lur}$ Function: sub-swarm Confrontation with dynamic obstacles   Initialize parameter vectors $\mathit{\psi }$, $\overline{\mathit{\psi }}$, $\mathit{\theta }$, $\mathit{\varphi }$   Initialize replay buffer $\mathcal{D}$   for each iteration do      for each time step do         Sample ${\mathit{a}}_{\mathit{t}}$ according to ${\mathit{a}}_{\mathit{t}}~{\mathit{\pi }}_{\mathit{\varphi }}\left({\mathit{a}}_{\mathit{t}}|{\mathit{s}}_{\mathit{t}}\right)$ from target network         Observe ${\mathit{s}}_{\mathit{t}+1}$ according to ${\mathit{s}}_{\mathit{t}+1}~\mathit{p}\left({\mathit{s}}_{\mathit{t}+1}|{\mathit{s}}_{\mathit{t}}, {\mathit{a}}_{\mathit{t}}\right)$ from transition         probability distribution         Obtain ${\mathit{r}}_{\mathit{t}}={\mathit{r}}_{\mathit{s}2\mathit{e}}+{\mathit{r}}_{\mathit{s}2\mathit{t}}+{\mathit{r}}_{\mathit{e}2\mathit{p}}$ from the environment         Store transitions to replay buffer $\mathcal{D}$      end for      for each gradient step do         Update $\mathit{\psi }$, $\mathit{\theta }$, $\mathit{\varphi }$, $\overline{\mathit{\psi }}$ in turn      end for   end for end function

4. Simulation Studies

4.1. Swarm Campaign Evaluation Metrics

4.1.1. Order Parameters

The concept of swarm motion order proposed by Vicsek [38] is typically assessed using an order parameter. In this section, we employ a set of quantitative metrics as order parameters to analyze the fission–fusion motion of the UAV swarm.

Polarization Index. $\phi \in \left[0,1\right]$ denotes the degree to which all drones tend to move in the same direction at this moment. Given the introduction of stochastic disturbances, we have established a benchmark (${\phi }_{flock}$) for the polarization index. When surpassing ${\phi }_{flock}$, it indicates the formation of a stable swarm.

$$\phi =\Vert {\displaystyle \sum }_{i=1}^{N_i^{\mathsf{\varpi }}}\frac{v_i^{\mathsf{\varpi }}}{\Vert v_i^{\mathsf{\varpi }}\Vert }\Vert \frac{1}{N_i^{\mathsf{\varpi }}}$$

(19)

where $v_i^{\mathsf{\varpi }}\in R^3$ is the velocity vector.

Differentiation Index. The single polarization index does not adequately capture the movement characteristics of the swarm during sub-swarm movements. To address this limitation, we incorporate the differentiation index [17] to assess the velocity variation among agents within the swarm, defined as follows:

$$\lambda =\frac{{\Gamma }^2+1}{\hslash }$$

(20)

$$\Gamma =\frac{1}{N}{\displaystyle \sum }_{i=1}^N{(v_i-\overline{v})}^3/{(\frac{1}{N}{\displaystyle \sum }_{i=1}^N{(v_i-\overline{v})}^3)}^{-\frac{3}{2}}$$

(21)

$$\hslash =\frac{1}{N}{\displaystyle \sum }_{i=1}^N{(v_i-\overline{v})}^4/{(\frac{1}{N}{\displaystyle \sum }_{i=1}^N{(v_i-\overline{v})}^2)}^{-2}$$

(22)

where $\Gamma$ and $\hslash$ indicate the skewness and kurtosis of the velocity distribution of the UAV swarm, respectively, for differentiation index $\lambda \in [0,1$], when $\lambda =1$ denotes two independent Renoulli distributions signifies the complete fission of velocities; In this study, when $\lambda >0.9$ we determine that two independent swarms have been formed.

4.1.2. Performance Evaluation Index

To assess the effectiveness of the swarm in resisting the interference from dynamic obstacles, this section proposes the precision of stimuli as an indicator of the swarm’s resilience to interference. Additionally, we establish a communication load to quantify the communication pressure.

Precision of Stimuli. The precision of stimuli denotes the proximity of the swarm’s motion direction to that of dynamic obstacles. When $\mathsf{\Lambda }=0$, it signifies a lack of interference as the motion directions are dissimilar. Conversely, when $\mathsf{\Lambda }=1$, it indicates severe interference, as the motion directions are entirely congruent. The definition is as follows:

$$\Lambda =\frac{A-A_0}{1-A_0} A_0=\frac{1+r_o{r}_{att}}{2}A=\frac{1}{N_i^{\mathsf{\varpi }}}{\displaystyle \sum }_{i=1}^{N_i^{\mathsf{\varpi }}}\frac{1+r_i{r}_{att}}{2}$$

(23)

where $r_i$, $r_{att}$ and $r_o$, respectively signify the unit velocity directions of unmanned aerial vehicle $i$, the swarm, and the dynamic obstacles.

Communication Load. This term represents the average communication cost incurred by the UAV fleet, where a lower communication load indicates a reduced average communication cost. It is defined as follows:

$$N_{cl}=\frac{1}{N}\Vert {{\displaystyle \sum }}_{i=1}^N{{\displaystyle \sum }}_{j=1}^{N_{il}^{\mathsf{\varpi }}}{d}_{ij}^{\mathsf{\varpi }}\Vert$$

(24)

where $N_{il}^{\mathsf{\varpi }}$ represents the number of unmanned aerial vehicles interacting with unmanned aerial vehicle $i$.

4.2. Detailed Simulation Parameters

Table 1. Swarm detailed simulation parameters.

Parameters	Description	Numerical Value
${\gamma }^{ine}$	Inertia coefficient	0.75
$\xi$	Environmental damping factor	0.006
$\zeta$	Random noise factor	0.003
${\Gamma }^{pos}$	Position cooperation factor	1
${\Gamma }^{vels}$	Velocity alignment factor	1
$l_c^{\mathsf{\varpi }}$	Decay coefficient	0.2
$l_a^{\mathsf{\varpi }}$	Desired spacing	0.2
${\alpha }_{\psi }$		0.85
${\alpha }_v$	Self-driving instrument conrol parameters	3.15
${\alpha }_h$		0.4
${\alpha }_{\dot{h}}$		1
${\mathrm{v}}_{\min }$	Min horizontal speed	0.02
${\hslash }_{\max }$	Max height change rate	2.1
${\mathsf{\phi }}_{\max }$	Max lateral overload	10
${\mathrm{v}}_{\max }$	Max horizontal speed	2
g	Gravitational acceleration	9.8
${\hslash }_{\min }$	Minimum height change rate	−2.2
$R_{Radius}$	Perception radius	70
${\phi }_{flock}$	Polarization index threshold	0.85
${ʑ}_{iva}$ ${ʑ}_{ina}$	Lead-in distance Safe confrontation range	70–65 3–5
Ntop	Maximum number of neighbors	7
${\mathrm{{\rm Y}}}_{rad}$	Random number	0–1
${\varrho }_{rad}$	Probability thresholds	1.75

and

Table 2. Sub-swarm simulation parameter setting.

Parameters	Description	Numerical Value
W	Simulation region size	100
$\mathcal{T}$	Maximal simulation steps	28
$d_{e2p}^{min}$	Minimal distance between parent swarm and obstacle	5
$d_{s2e}^{safe}$	Safe distance between the sub-swarm and obstacle	15
$d_{s2e}^{capture}$	Capture distance between the sub-swarm and dynamic obstacle	10
$v_{sub}^{max}$	Max speed of sub-swarm	2
$v_p^{max}$	Max speed of parent swarm	1
$v_e^{max}$	Max speed of dynamic obstacle	1.5
$\mathcal{D}$		10
$\mathsf{\sigma }$		2
$\mathsf{\alpha }$		5
$\mathsf{\beta }$	Discrete actions	0.1
$\mathsf{\gamma }$		5
$c_1$		5
$c_2$		−10

show the detailed simulation parameters of the experiments and RL-based path planning of the sub-swarm, respectively. It is worth noting that the purpose of the parameter list we have provided is in the hope that researchers will be able to reproduce the methodology of this study more quickly. In fact, the parameters in the table can be modified according to different research situations, which does not affect the validity of the proposed algorithm.

4.3. Simulation Results Analysis

In this section, we assess the ability of the UAV swarm to achieve fission–fusion and perform confrontation movements, ensuring that the parent swarm movement remains unaffected during anti-confrontation scenarios.

4.3.1. Evaluation of the Probabilistic Starling-Inspired Topological Structure

For the validation of the PSTI structure, we conducted multiple tests with varying birth ranges for the agents (ranging from 4 to 13). Each test consisted of 1000 trials, with 130 iterations per trial. Figure 3 displays the probabilities of generating a stable swarm using the previous seven-nearest-neighbor and our proposed PSTI structures, respectively, after randomly generating 20 agents at different birth ranges.

From Figure 3, it is observed that both methods achieve high swarm rates within a small range. However, as the range increases, the number of seven-nearest-neighbour topological interaction models gradually exhibits local convergence problems. Even at a range of 15, the probability of local convergence reduces to 50 percent. Conversely, the proposed PSTI structure consistently produces superior results across all tested ranges, which demonstrates its ability to enhance the swarming efficiency of UAV swarms while reducing the occurrence of local convergence. It is worth noting that the PSTI structure does exhibit some instances of local convergence due to the random characteristics of the incorporated probability coefficients. However, we can maintain a low communication load while achieving superior swarming performance with our proposed approach by balancing the probability factors.

4.3.2. Simulation of the Sub-Swarm Confrontation via Reinforcement Learning Algorithm

We build a simulation environment to satisfy the demand for abundant interactions between the swarm and the dynamic obstacle. Assuming that the mission area is $100\times 100$. The coordinates of the parent swarm, sub-swarm, and dynamic obstacle are generated by the planning algorithms mentioned above. We use the center point of the sub-swarm to optimize its trajectory, and the sub-swarm flies at a constant altitude of its initial height. When an episode starts, the coordinates of both swarms and the dynamic obstacle are generated. Once the parent swarm is captured by the dynamic obstacle, when $d_{e2p}\le d_{e2p}^{min}$ or the sub-swarm runs for more than $\mathcal{T}=28$ steps, the episode ends. The maximal speed of the parent swarm, sub-swarm, and the dynamic obstacle are $v_p^{max}=1$, $v_{sub}^{max}=2$, $v_e^{max}=1.5$, respectively.

We then evaluate the performance of RL-based algorithms in the simulation environment, including the PPO, SAC, and DQN methods. For the sake of inadaptability to the continuous action space of the DQN algorithm, we discretize each action of sub-swarm (i.e., the flying speed ($v_t$) and direction (${\theta }_t$)) into 10. The three algorithms have the same hyperparameters of $\gamma =0.99$, a learning rate of $l_r=0.0003$, and network architecture of $\mathcal{N}=128\times 128\times 128$. Figure 4 shows the accumulative reward of these RL algorithms over training steps for the same network depth. It can be observed that these algorithms achieve similar accumulated rewards, while the agent based on the SAC algorithm converges faster and is as stable as the agent of the PPO algorithm.

Figure 5 illustrates the trajectory of the sub-swarm planned by using the PPO algorithm. The visualization demonstrates the efficient response of the sub-swarm to dynamic obstacles, considering various obstacle coordinates. The sub-swarm consistently approaches the target successfully by the end of each episode.

4.3.3. Simulation of the Bio-Inspired Fission–Fusion Control and Planning via Reinforcement Learning Algorithm

Simulation results of swarm motion are depicted in Figure 6a–f. Figure 6a showcases the whole process of fission–fusion motion of the UAV swarm when encountering dynamic obstacles. This motion is achieved through our proposed BiFRL algorithm. The sub-swarm successfully executes the confrontation movement and seamlessly integrates back into the parent swarm upon completion. The point of origin for the dynamic obstacle is (40, 60, 30). When the swarm perceives the presence of dynamic obstacles, it initiates a self-organizing fission into two swarms, which include a parent swarm and a sub-swarm. The two swarms swiftly synchronize their positions. During the confrontation process, the sub-swarm effectively confronts the interference from dynamic obstacles utilizing the SFCRL algorithm. Finally, following the resolution of the sub-swarm’s engagement, it autonomously converges back into the parent swarm, fusion into a stable and cohesive swarm.

Figure 6b illustrates random initial spots of all agents. Figure 6c demonstrates the rapid formation of a stable swarm by intelligent agents under random initial spots. In Figure 6d, the swarm perceives the obstacle, leading to its self-organization and fission into two state-stable swarms. The confrontation motion against dynamic obstacles is executed based on the SFCRL algorithm. Figure 6e exhibits the sub-swarm completing its confrontation movement and initiating its return to the parent swarm after the dynamic obstacle stops tracking. Lastly, Figure 6f illustrates the self-organization of the sub-swarm as it returns to the parent swarm following the conclusion of the antagonistic movement, culminating in a fusion.

4.3.4. Evaluation of the Bio-Inspired Fission–Fusion Control and Planning via Reinforcement Learning Algorithm

Figure 7 depicts the temporal evolution of the polarization index of the UAV swarm combined with reinforcement learning. The polarization indices in Figure 7 demonstrate that the entire swarm successfully completes the swarming within 10 s. After the fission campaign, both swarms have strong robustness. Upon the sub-swarm’s return to the parent swarm, there is a brief decrease in the polarization index, followed by stabilization around 1. Notably, the parent swarm is unaffected by the dynamic obstacle throughout the entire process.

Figure 8 illustrates the evolution of the differentiation index in the swarm, integrated with reinforcement learning.

The differentiation index showed that 20 randomly generated UAVs rapidly established stable swarms within the first 0–3 s of the initial, which persists until approximately 24 s. At around 27 s, the sub-swarm undergoes rapid fission, forming two stable sub-swarms, while maintaining a constant differentiation index of approximately 1. Subsequently, at approximately 92 s, after completing the confrontation and returning to the parent swarm, it seamlessly fuses back into a stable swarm, thereby validating the effectiveness of our proposed algorithm.

Figure 9 illustrates the precision of stimuli exhibited by both swarms under dynamic obstacle interference. Both swarms demonstrate commendable precision of response stimuli throughout the fission–fusion process. The precision of stimuli of the sub-swarm, influenced by the SFCRL algorithm, exhibits some variability; however, it consistently demonstrates a high level of accuracy. In contrast, the parent swarm experiences slight fluctuations in response stimuli when dynamic obstacles are detected. These fluctuations primarily arise from the need for the parent swarm to stabilize into a new formation during the fission–fusion process and are not directly caused by the dynamic obstacles themselves. Furthermore, it is observed that the obstacles do not significantly impact the parent swarm during the remaining period duration.

Figure 10 presents the dynamics of communication load in UAV swarms employing different interaction structures. In this study, we compare the communication load of the PSTI structure with various fixed-distance communication structures. The simulation results demonstrate that in the absence of fission–fusion, the probabilistic starling-inspired topological interaction achieves a significant reduction in communication load, ranging from 50% to 85% compared to other structures. Even during the fission–fusion process, our proposed interaction structure continues to exhibit advantages. Conversely, as the fixed distance interaction structure adopts smaller interactive distances, the likelihood of local convergence increases during the swarming process.

5. Conclusions

In recent years, researchers have shown significant interest in the UAV swarm, particularly its self-organized fission–fusion control methods. However, dealing with dynamic obstacles that possess tracking capabilities remains a challenge in UAV swarm control. As such, we present a bio-inspired fission–fusion control and planning of a UAV swarm system via reinforcement learning algorithm to tackle this issue. In contrast to existing methodologies, our approach is targeted at resolving the interference posed by unknown dynamic obstacles through the utilization of fewer resources. The proposed self-organized fission–fusion framework facilitates autonomous grouping and separation in response to dynamic obstacles while maintaining control over the size of the sub-swarm. Additionally, we introduce sub-swarm confrontation via reinforcement learning to handle the selection of confrontation paths in the presence of unknown disturbance directions. After completing the confrontation, the sub-swarm seamlessly integrates back into the parent swarm through self-organization. Furthermore, we propose a probabilistic starling-inspired topological interaction structure, which effectively mitigates the issue of swarm local convergence encountered by existing seven-nearest-neighbor algorithms. To validate the competitiveness of our approach, we conduct extensive simulations involving different swarm initial ranges and evaluate the communication load as a performance metric. The results demonstrate the effectiveness and feasibility of our proposed BiFRL algorithm, which combines reinforcement learning with UAV swarm fission–fusion control and planning to handle unknown dynamic obstacle disturbances. We believe that our proposed algorithm has a positive effect on improving the efficiency of swarm control in dynamic environments and the ability of clusters to combat dynamic disturbances. This has a positive effect on the research in the field of multi-swarm.

However, the only resource terms we considered during the study are limited, and it is clear that we need to consider more resource terms, e.g., electromagnetic interference in the environment, leakage rate during recognition of dynamic obstacles, and other factors. These issues will make the existing dynamic environment more complex but more realistic. We will introduce more AI algorithms to try to solve such complex dynamic problems. This will, in turn, improve the robustness and applicability of the algorithms. We will perform in-depth sensitivity analyses for more complex environments to observe the robustness of the algorithms [39]. In future work, we will further investigate the interference of complex dynamic obstacles within the swarm fission–fusion approach and study the influence of specific parameters on the algorithm in greater depth.