A Geometrical, Reachable Set Approach for Constrained Pursuit–Evasion Games with Multiple Pursuers and Evaders

Abstract

This paper presents a solution strategy for deterministic time-optimal pursuit–evasion games with linear state constraints, convex control constraints, and linear dynamics that is consistent with linearized relative orbital motion models such as the Clohessy–Wiltshire equations and relative orbital elements. The strategy first generates polytopic inner approximations of the players’ reachable sets by solving a sequence of convex programs. A bisection method then computes the optimal termination time, which is the least time at which a set containment condition is satisfied. The pursuit–evasion games considered are games with (1) a single pursuer and single evader, (2) multiple pursuers and a single evader, and (3) a single pursuer and multiple evaders. Compared to variational methods, this reachable set strategy leads to a tractable formulation even when there are state and control constraints. The efficacy of the strategy is demonstrated in three numerical simulations for a constellation of satellites in close proximity in low earth orbit.

1. Introduction

This paper studies time-optimal pursuit–evasion games with linear dynamics and convex state and control constraints. Cases with a single pursuer and single evader, multiple pursuers and a single evader, as well as a single pursuer and multiple evaders are considered. The paper takes a geometrical approach in solving these games by modeling them as reachable set problems as first introduced in [1]. By using modern, convex optimization techniques for the reachable set construction of constrained systems, it is now possible to solve constrained, multiplayer games not considered in [1].

Pursuit–evasion games commonly arise in aerospace engineering as well as in economics. The history of differential pursuit–evasion games begins with the seminal work by Isaacs [2]. His work describes several applied problems and their differential pursuit–evasion formulations. An overview of the development of differential pursuit–evasion games is given in [3], which covers seminal work conducted in the 1970s as well as current state-of-the-art work.

Aerial warfare is one of the typical problems that can be modeled as a pursuit–evasion game [4,5]. Multiple assumptions are often made to simplify dynamical complexity. Some common assumptions restrict the motion to be planar, model the aircraft as a point mass, linearize the dynamics, and require an instantaneous control response [6]. Examples of different aerial warfare problems modeled as pursuit–evasion games consist of missile versus aircraft [4,6,7,8], aircraft versus aircraft interception [9,10], and fighter maneuvering [11].

Pursuit–evasion games have also been studied in the context of astrodynamics, particularly collision avoidance and interception [12,13]. Rather than use a nonlinear two-body formulation, linearization and the use of the Clohessy–Wiltshire (CW) equations simplifies the problem so that minimax formulations lead to open-loop and closed-loop control strategies based on kriging [14]. Pursuit–evasion strategies have been used for the tracking of space objects and selection of sensor management strategies [15]. A hybrid global–local technique has been developed for a two-phase (long distance and short distance) game [16]. The two-phase approach has been extended in [17] using reinforcement learning under assumptions of impulse control, incomplete information, and a quadratic objective.

A pursuit–evasion game can be seen as an optimal control problem where the pursuer(s) and evader(s) have different objectives. The addition of multiple agents with opposite objectives generally makes the optimal control problem more difficult to solve than those with an individual pursuer or evader would be. It should be noted that sometimes the single-agent optimal control problems can be complicated to solve as is. A technique utilizing calculus of variations may be used to solve differential games. This technique requires setting an optimal control problem for all the players with the underlying necessary conditions and then finding a solution simultaneously for all the players. Linear–quadratic as well as time-optimal scalar games were solved using this methodology in [18]. A case of one of the players not playing the game optimally based on the variational approach is also included in [18] as part of a short discussion on games with stochastic behavior.

Because of their complexity, it is often difficult to solve differential games using variational theory. This fact is aptly summarized in [17]: “However, for the space [pursuit-evasion] game, it is a challenge to solve the transformed [two-point boundary value problem] due to its high dimensionality and strong nonlinearity.” This is particularly true when there are pointwise constraints on the states and controls. The pointwise constraints introduce switches as trajectories enter and leave the control boundaries, which may violate the smoothness assumptions required for the Newton method. Consequently, direct methods [19,20] and semidirect methods have been introduced to leverage finite-dimensional optimization packages. These methods involve the use of genetic algorithms and neural networks [8,11,21,22].

In contrast to the optimization-based approaches, Mizukami [1] has shown that a two-player time-optimal pursuit–evasion game terminates when the evader’s reachable set is contained in the pursuer’s reachable set, which motivates a geometric approach to solving the game based on reachable set computation. More precisely, the termination point in the state space lies on the boundary of the players’ reachable sets. For problems with linear dynamics and integral-constrained (rather than pointwise-constrained) controls, analytical solutions can be derived for the termination time and optimal trajectories. Motivated by this work, reachable set analyses have been used in dynamic flowfields [23], coordinate control [24], missile/sensor trade studies [25], and other game scenarios [26,27,28].

In the CW setting and with control magnitude constraints, it is possible to approximate reachable sets with a sequence of analytical computations [29]. The presence of state constraints and other types of control constraints complicates the process of computing reachable sets, and in general, numerical methods are utilized to approximate them. Multiple numerical algorithms to calculate a reachable set for control affine systems are provided in [30]. An algorithm for the computation of reachable sets for linear systems with bounded inputs, which approaches the problem by finding inner and outer approximations, is introduced in [31]. Optimal control has also been used to calculate reachable sets [32,33,34,35]. A polytopic approximation of the actual reachable set is computed. In [34], a single semidefinite program (SDP) and multiple sequential second-order cone programs (SOCPs) are solved to find an inner approximation of the reachable set for linear dynamics and convex state and control constraints. An outer approximation is added in [33] as a heuristic to determine the precision of the approximation. Nonconvex control constraints are considered in [35]. The approach from [34] is used as a basis for solving multiplayer games in this paper.

The primary contribution of this work is the construction of reachable set approximations to solve constrained pursuit–evasion games. The inclusion of pointwise, convex constraints on the state and control make the variational approach difficult, while such constraints add minimal complexity to the geometric approach. The inclusion of multiple pursuers and evaders adds complexity to the problem as well, especially if variational techniques were to be used. It is shown that with reachable set methods, constrained pursuit–evasion problems with multiple pursuers and evaders remain tractable. It is assumed that the pursuers and evaders have perfect information about the other players and all players perform optimally. Pursuit–evasion problems with (1) a single pursuer and single evader, (2) multiple pursuers and a single evader, and (3) a single pursuer and multiple evaders are considered.

The remainder of this paper is structured as follows. Section 2 describes a numerical method based in convex optimization for approximating reachable sets. The algorithm is based on the one introduced in [34]. The constrained pursuit–evasion game is introduced in Section 3 and an explanation on how reachable set theory can be used to solve it for a varying number of pursuers and evaders is explained. Section 5 applies the method to aerospace problems whose dynamics and constraints are explained in Section 4. The results are summarized and conclusions are drawn in Section 6.

2. Reachable Sets

An introduction to reachable sets is given in this section. Reachable sets for linear time-varying (LTV) dynamics with second-order cone control constraints and linear state constraints are considered. This system of interest is given as

$$\begin{array}{cc}\hfill \dot{x}& =A\left(t\right)x+B\left(t\right)u\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill y& =C\left(t\right)x\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill x& \in \mathcal{X}\left(t\right)=\left\{x\in {\mathbb{R}}^n:D\left(t\right)x\le b\left(t\right)\right\}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill u& \in \mathcal{U}\left(t\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill x\left(t_0\right)& =x_0.\hfill \end{array}$$

(5)

The state vector is $x\in {\mathbb{R}}^n$, the control vector is $u\in {\mathbb{R}}^m$, the output vector is $y\in {\mathbb{R}}^p$, the system matrix is $A\left(t\right):{\mathbb{R}}^n\to {\mathbb{R}}^n$, the control influence matrix is $B\left(t\right):{\mathbb{R}}^m\to {\mathbb{R}}^n$, and the output matrix is $C\left(t\right):{\mathbb{R}}^n\to {\mathbb{R}}^p$. The quantities $D\left(t\right)\in {\mathbb{R}}^{q\times n}$ and $b\left(t\right)\in {\mathbb{R}}^q$ define the linear state constraint $\mathcal{X}\left(t\right)$. The second-order cone control constraint is $\mathcal{U}\left(t\right)$. The state and control constraints limit the values that can be taken by the state and control pointwise in time. The initial state vector is $x_0$ at initial time $t_0$. A feasible control is any control satisfying (1)–(5).

Definition 1.

For the system in (1)–(5), the reachable set $\mathcal{R}\left(t\right)$ is the set of all outputs the system can reach at time t from the initial state $x_0$ using feasible controls, i.e.,

$$\begin{array}{cc}\hfill \mathcal{R}\left(t\right)=\{y\in {\mathbb{R}}^p:y& =C\left(t\right)\Phi (t,t_0)x_0+C\left(t\right){\int }_{t_0}^t\Phi (t,\tau )B\left(\tau \right)u\left(\tau \right)d\tau \mathit{for}\mathit{all}\mathit{feasible}u\}\hfill \end{array}$$

(6)

where $\Phi (t_2,t_1)$ is the state-transition matrix.

The reachable set is the set of all points that can be reached from a given initial condition with a feasible control. The first term on the right-hand side of (6) is $C\left(t\right)\Phi (t,t_0)x_0$. Because it is independent of the control, this term is associated with the translation of the reachable set. The second term on the right-hand side is the integral term. This term depends on the control and is associated with the size and shape of the reachable set. In reachable set theory, sets that grow in size are said to be expanding. An illustration of a reachable set in two-dimensional space that is translating and growing in time is shown in Figure 1.

Reachable sets do not always expand, and the strategy presented herein has no such requirement. The simplest sufficient condition to ensure the expansion of the reachable set is time invariance [36]. For time-varying systems, a sufficient condition for expansion is properness (see Corollary 17.1 of [37]). Furthermore, because the dynamics of the system are linear and the state and control constraints are convex, the reachable set is convex [36].

2.1. Algorithm for Reachable Set Calculation

An algorithm to calculate an inner polytopic approximation of the reachable set for the system described in (1)–(5) is given next. The algorithm first solves an SDP to compute a largest possible size simplex that fits inside the reachable set. The algorithm then solves a sequence of SOCPs to compute a better inner approximation of the reachable set based on the initial simplex. A more detailed explanation of the algorithm can be found in [34].

Consider the discretized version of the continuous system given in (1)–(5). The discrete-time system is

$$\begin{array}{cc}\hfill x_{k+1}& ={\overline{A}}_k{x}_k+{\overline{B}}_k{u}_k,k=0,...,N-1\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill y_k& =C_k{x}_k,k=0,...,N\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill x_k& \in {\mathcal{X}}_k=\left\{x\in {\mathbb{R}}^n:D_{k}x\le b_k\right\}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill u_k& \in {\mathcal{U}}_k\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill x_{k=0}& =x_0\hfill \end{array}$$

(11)

where N represents the number of time intervals used in discretization. Appendix A explains how discretization is conducted in this paper to obtain the discrete, barred quantities based on their continuous counterparts and discretization time step. Discretization allows for the use of numerical optimization. The reachable set definition given in Definition 1 for the continuous system is analogous to that for the discrete system by replacing t with $t_k$.

2.1.1. Initial Simplex

The computation of the initial simplex is explained first. Computing the initial simplex requires solving a single SDP. Let $\mathcal{Z}$ be the set containing the $p+1$ vertices of an initial simplex in ${\mathbb{R}}^p$. Furthermore, define a matrix Q as

$$Q=\left[\begin{array}{ccc}(z_2-z_1)& \dots & (z_{p+1}-z_1)\end{array}\right]:z_i\in \mathcal{Z}.$$

(12)

The volume of the simplex may then be calculated as

$$v=\frac{1}{p!}|detQ|.$$

(13)

With an expression for the simplex volume, maximizing (13) gives an initial simplex with a maximum volume. This also leads to the vertices of the simplex being on the boundary of the system’s reachable set:

$$z_i\in \partial \mathcal{R}\subset \mathcal{R}.$$

(14)

This maximum volume simplex is used as the initial simplex. The volume maximization problem can be written as the following SDP:

$$\begin{array}{cc}\hfill \underset{x_i,u_i,z_i}{min}& -\mathrm{logdet}\left(Q\right)\hfill \\ \hfill s.t.& Q=Q^T=\left[\begin{array}{ccc}(z_2-z_1)& \dots & (z_{p+1}-z_1)\end{array}\right]⪰0\hfill \\ \hfill & z_i=y_i\left(t_k\right),i=1,...,p+1\hfill \\ \hfill & \mathrm{Equations}\left(7\right)–\left(11\right).\hfill \end{array}$$

(15)

The log determinant is denoted as logdet, which acts on square symmetric matrices. The initial simplex requires the calculation of $p+1$ trajectories emanating from the initial point, $x_0$, to $p+1$ final states. The initial simplex is constructed from the $p+1$ final state vectors which form the $z_1,\dots ,z_{p+1}$ vertices of the initial simplex.

2.1.2. Growing Simplices

Following the construction of the initial simplex, improved approximations are achieved by computing additional simplices out of the open faces of the current polytopic approximation. Whereas the computation of the initial simplex required a solution of an SDP, the computation of additional simplices is achieved by solving an SOCP for each additional vertex in the polytopic approximation. This is possible because maximizing the volume of an additional simplex is analogous to maximizing the length of a vector that connects the open face to a point on the boundary of the reachable set and is orthogonal to the face. The point on the boundary of the reachable set is then added as a new vertex to the current polytopic approximation of the reachable set:

$$\begin{array}{cc}\hfill \underset{\alpha ,\lambda ,x,u}{min}& -\alpha \hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \nu =Z_i\lambda ,1^{\top }\lambda =1,\lambda \ge 0\hfill \\ \hfill & \nu +\alpha h_i=z\hfill \\ \hfill & z=y\left(t_k\right)\hfill \\ \hfill & \mathrm{Equations}\left(7\right)–\left(11\right).\hfill \end{array}$$

(16)

The quantity to be maximized is the scaling factor $\alpha$. The point on the open face $\nu$ is such that the minimum distance between it and the reachable set boundary is maximized. The matrix $Z_i$ contains the p vertices corresponding to the open face i. The vector $\lambda \in {\mathbb{R}}^p$ identifies the point $\nu$ from the vertices in $Z_i$. A vector normal to the ith open face is given by $h_i$. A point on the boundary of the reachable set is z. Figure 2 gives a graphical representation of these variables with $\nu$ being represented as a vector from one vertex.

Overall, the algorithm requires the solution of a single SDP defined in (15) to obtain an initial approximation of the reachable set. A tighter approximation is then reached by solving a sequence of SOCPs given in (16), with each SOCP adding a new vertex to the current polytopic approximation. The first three reachable set approximations are illustrated in Figure 3 along with the actual reachable set. Further explanation and illustration of the approximation process may be found in [34].

3. Game Theory

This section introduces the pursuit–evasion game of interest. The three pursuit–evasion games considered in this paper are games with (1) a single pursuer and single evader, (2) multiple pursuers and a single evader, and (3) a single pursuer and multiple evaders. The pursuers’ objective is to minimize the capture time of all the evaders. The evaders’ objective is to maximize the time it takes for the pursuers to capture all the evaders. It is assumed that a solution exists, i.e., capture occurs, such that it is a game of degree rather than a game of kind [2].

3.1. Single Pursuer and Single Evader

Consider the simplest case of a single pursuer and single evader. The game may be rewritten as two time-optimal control problems. The pursuer’s problem is:

$$\begin{array}{cc}\hfill \underset{u_P(·)}{min}& t^{*}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\dot{x}}_P=A_P\left(t\right)x_P+B_P\left(t\right)u_P\hfill \\ \hfill & y_P=C_P\left(t\right)x_P\hfill \\ \hfill & x_P\in {\mathcal{X}}_P\left(t\right)=\left\{x\in {\mathbb{R}}^n:D_P\left(t\right)x\le b_P\left(t\right)\right\}\hfill \\ \hfill & u_P\in {\mathcal{U}}_P\left(t\right)\hfill \\ \hfill & x_P\left(t_0\right)=x_{0,P}\hfill \\ \hfill & y_P\left(t^{*}\right)=y_E\left(t^{*}\right).\hfill \end{array}$$

(17)

The evader’s problem is:

$$\begin{array}{cc}\hfill \underset{u_E(·)}{max}& t^{*}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\dot{x}}_E=A_E\left(t\right)x_E+B_E\left(t\right)u_E\hfill \\ \hfill & y_E=C_E\left(t\right)x_E\hfill \\ \hfill & x_E\in {\mathcal{X}}_E\left(t\right)=\left\{x\in {\mathbb{R}}^n:D_E\left(t\right)x\le b_E\left(t\right)\right\}\hfill \\ \hfill & u_E\in {\mathcal{U}}_E\left(t\right)\hfill \\ \hfill & x_E\left(t_0\right)=x_{0,E}\hfill \\ \hfill & y_E\left(t^{*}\right)=y_P\left(t^{*}\right).\hfill \end{array}$$

(18)

The subscript P refers to the pursuer, the subscript E refers to the evader, and $t^{*}$ is the termination time of the game. It is assumed that both the pursuer and evader have perfect information about the other.

It was shown by Mizukami [1] that such time-optimal games can be recast as reachable-set inclusion problems.

Theorem 1

([1]). If a solution to the game exists, then the game terminates in the least time such that

$${\mathcal{R}}_E\left(t\right)\subseteq {\mathcal{R}}_P\left(t\right).$$

(19)

The question of existence is not explored in this paper. It is assumed that a solution exists, and the theorem is used to find the solution by comparing the players’ reachable sets. The termination time is the first time at which the evader’s reachable set is contained in the pursuer’s reachable set. The game is therefore reduced to a one-dimensional search for $t^{*}$. This search is performed in practice by using a bisection, which is a special case of the more generic Algorithm 1 presented in the next subsection, and the method of Section 2.1 for approximating the reachable sets. In the state space, the game terminates at a point on the boundary of both players’ reachable sets, i.e., $y^{*}\in \partial R_E\left(t^{*}\right)\cap \partial R_P\left(t^{*}\right)$, where $y^{*}$ is the capture point.

The simplest situation occurs when $C\left(t\right)\Phi (t,t_0)x_0$ is zero (so that the reachable sets do not translate) and the pursuer’s control set is larger than the evader’s (so that it enlarges at a faster rate). Prior to the termination time, the evader’s reachable set is not contained in the pursuer’s. After the termination time, the evader’s reachable set is in the interior of the pursuer’s. At the termination time, the evader’s set is contained in the pursuer’s set, and they share a boundary point. These relationships are demonstrated in Figure 4.

3.2. Multiple Pursuers and Single Evader

When there are multiple pursuers and a single evader, the game terminates when the evader’s reachable set is contained in the union of the pursuers’ reachable sets [23], i.e.,

$${\mathcal{R}}_E\left(t\right)\subseteq \left\{\bigcup _{i=1}^{N_P}{\mathcal{R}}_{P_i}\left(t\right)\right\}$$

(20)

where $N_P$ is the number of pursuers. For later use, the bracketed term on the right-hand side is defined to be ${\mathcal{R}}_{\cup P}\left(t\right)$. If the game has only one pursuer, (20) reduces to (19) as expected. Some of the possible capture scenarios are shown in Figure 5 and Figure 6. In Figure 5, the capture happens at the boundary of the two pursuers’ and evader’s sets. In Figure 6, the capture happens at the boundary of the four pursuers’ sets but is in the interior of the evader’s reachable set. In both cases, however, the capture happens at the first time instance when the evader’s reachable set is contained in the union of the pursuers’ reachable sets.

A line search is conducted to find the capture time. An algorithm that performs this line search is shown in Algorithm 1. The algorithm uses the method from Section 2.1 to generate the inner approximations of the pursuers’ and evader’s reachable sets and checks if (20) is satisfied or not. The algorithm uses a bisection method to find the capture time by checking the set inclusion and tightening the minimum and maximum allowable times.

Algorithm 1 Line Search for Capture Time in Multiple Pursuer and Single Evader Game

Input:  Dynamics for evader and pursuers, initial states, $ϵ$, $t_{min}$, $t_{max}$ Output:  $t^{*}$ 1: Set $t_0=(t_{min}+t_{max})/2$ and $i=0$. 2: while  $\parallel t_i-t_{i-1}\parallel <ϵ$ do 3:    $i=i+1$ 4:    Calculate ${\mathcal{R}}_E\left(t_{i-1}\right)$ 5:    Calculate ${\mathcal{R}}_{P_1}\left(t_{i-1}\right),\dots ,{\mathcal{R}}_{P_{Np}}\left(t_{i-1}\right)$ and ${\mathcal{R}}_{\cup P}\left(t_{i-1}\right)$ 6:    if ${\mathcal{R}}_E\left(t_{i-1}\right)\subset {\mathcal{R}}_{\cup P}\left(t_{i-1}\right)$ then 7:        $t_i=(t_{min}+t_{i-1})/2$, $t_{max}=t_{i-1}$ 8:    else 9:        $t_i=(t_{max}+t_{i-1})/2$, $t_{min}=t_{i-1}$ 10:    end if 11: end while 12: $t^{*}=t_i$

3.3. Single Pursuer and Multiple Evaders

Lastly, a pursuit–evasion game with a single pursuer and multiple evaders is considered. This game is split into multiple single pursuer and single evader games where the pursuer’s reachable set is reset at the capture point when a capture of one of the evaders happens. Figure 7 demonstrates the evolution of the pursuer’s and evaders’ reachable sets where the pursuer’s reachable set is reset after capturing an evader.

To consider all the possible strategies for the pursuer to capture the evaders, it is in general required to consider $N_E!$ possibilities for capturing sequences, where $N_E$ is the number of the evaders. This means that in a case of three evaders, the pursuer could capture the evaders in six different orders: 1,2,3; 1,3,2; 2,1,3; 2,3,1; 3,1,2; or 3,2,1. Figure 8 shows the possible capture order of the pursuer in a three evader game. t with subscripts represents the capture time of the evaders in the order of numeric values in the subscript. $t^{*}$ is the termination time of the game associated with the shallowest branch, i.e., the branch offering the least time compared to all other branches. In this example, the selected order for capturing the evaders is 2,1,3.

Now consider a case where catching four evaders in the order 1,2,3,4 takes $t_{1,2,3,4}$ time units. If catching only evaders 4 and 3 in this order takes $t_{4,3}$ time units and $t_{1,2,3,4}<t_{4,3}$, then it is not optimal for the pursuer to first capture evader 4 and then evader 3. As a consequence, there is no need to further compute $t_{4,3,1,2}$ and $t_{4,3,2,1}$ since it is guaranteed that both are greater than $t_{1,2,3,4}$. This logic leads to a pruning strategy that may allow the termination time for the game to be found without enumerating all the capture strategies completely. This pruning strategy is analogous to that in branch and bound, which is used to solve integer optimization problems [38].

An algorithm to find the termination time $t^{*}$ for the multiple evader and single pursuer game is provided in Algorithm 2. The algorithm begins by finding the termination time of the game by using a greedy approach when choosing the next evader to be captured. The other capturing sequences are then compared to this. If the capture time of an evader is larger than the current estimate for the termination time of the game $t_{current}^{*}$, that capturing sequence is excluded from further analysis and the branch is pruned. On the other hand, if the game termination time using that sequence is less than the current estimate for the game termination time, the current estimate for the game termination time is updated to that.

Algorithm 2 Termination Time in Single Pursuer and Multiple Evader Game

Input:  Dynamics for evaders and pursuer, initial states Output:  $t^{*}$ 1: Set $t_{current}^{*}=\infty$ 2: for  $i=1$ to $N_E!$ do 3:    while Free evaders left do 4:       Find capture time, t of the next evader such that this capture order is not yet considered. 5:       Add the captured evader in the capture sequence. 6:       if $t\ge t_{current}^{*}$ then 7:           Set this capture sequence considered. 8:           break while 9:         end if 10:       Reset pursuer’s reachable set at capture point. 11:    end while 12:    if $t<t_{current}^{*}$ then 13:        $t_{current}^{*}=t$ 14:    end if 15: end for 16: $t^{*}=t_{current}^{*}$

4. Dynamics

The dynamics used for the numerical simulations in Section 5 are explained in this section. A planar constellation of satellites near a circular orbit is considered. The dynamics can be modeled linearly with the Clohessy–Wiltshire (CW) equations [39]

$$\begin{array}{cc}\hfill \dot{x}& =Ax+Bu=\left[\begin{array}{cc}0& I\\ M_1& M_2\end{array}\right]x+\left[\begin{array}{c}0\\ I\end{array}\right]u\hfill \\ \hfill y& =Cx=\left[\begin{array}{cc}I& 0\end{array}\right]x\hfill \end{array}$$

(21)

where $x\in {\mathbb{R}}^4$ is a state vector with the first two elements corresponding to a satellite’s relative position in a local vertical local horizontal (LVLH) frame and the last two corresponding to a satellite’s relative velocity in the LVLH frame. The external control acceleration is $u\in {\mathbb{R}}^2$. The matrices $M_1$ and $M_2$ are given by

$$M_1=\left[\begin{array}{cc}3{\omega }^2& 0\\ 0& 0\end{array}\right],M_2=\left[\begin{array}{cc}0& 2\omega \\ -2\omega & 0\end{array}\right]$$

(22)

where $\omega$ is the mean motion of the circular reference orbit and is set to 4 rad/h for the remainder of this paper. This corresponds to a low earth orbit. Other relative motion models such as those based on time-varying relative orbital elements [40,41] are also applicable within the reachable set-based solution strategy. In all the simulations, the output vector y is to be the relative position of the players, which means that the output matrix is $C=\left[\begin{array}{cc}I& 0\end{array}\right]$, with I being the 2 × 2 identify matrix and 0 being the 2 × 2 zero matrix. The relative position is chosen as the output because capture occurs in position space.

5. Numerical Simulations

Numerical simulations of the three cases introduced in Section 3 are provided in this section. All the agents evolve according to CW dynamics as explained in Section 4. Each agent’s control set is of the form

$$\mathcal{U}\left(t\right)=\{u\in {\mathbb{R}}^2:|\left|u\right||\le \rho \}$$

(23)

with $\rho$ being a prescribed upper bound on the control magnitude. This is a second-order cone constraint. Additional constraints consistent with those in Equations (1)–(5) are also present and described in the following subsections.

5.1. Single Pursuer and Single Evader

Consider a single pursuer and single evader pursuit–evasion problem with the players’ dynamics given in (21). The initial states of the pursuer and evader are

$$\begin{array}{cc}\hfill x_{0,P}& ={[0\mathrm{km},1\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top }\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill x_{0,E}& ={[0\mathrm{km},0\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top }.\hfill \end{array}$$

(25)

The pursuer’s and evader’s states are constrained by a linear inequality as in (3) with

$$D=\left[\begin{array}{cccc}0& -1& 0& 0\\ 0& 1& 0& 0\end{array}\right],b=\left[\begin{array}{c}1\\ 1\end{array}\right]\mathrm{km}.$$

(26)

The D matrix is dimensionless. The control of the pursuer is bounded by ${\rho }_P=1 {\mathrm{km}/\mathrm{h}}^2$, and the control of the evader is bounded by ${\rho }_E=0.5 {\mathrm{km}/\mathrm{h}}^2$.

Figure 9 shows the reachable sets of the pursuer and evader when the capture happens. The trajectories that the pursuer and evader take to reach the capture point are also shown in the figure. The capture happens at $t^{*}=1.42$ h. It is evident from the figure that the capture happens at the boundary of both the pursuer’s and evader’s reachable sets.

5.2. Multiple Pursuers and Single Evader

Now consider a pursuit–evasion problem with four pursuers and a single evader. The dynamics of the pursuers and evader are given by the CW Equation (21). The initial states of the players are

$$\begin{array}{cc}\hfill x_{0,P_1}& ={[0\mathrm{km},0.5\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top },\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill x_{0,P_2}& ={[0\mathrm{km},-0.5\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top },\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill x_{0,P_3}& ={[{\omega }^{-2}\mathrm{km},0.5\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top },\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill x_{0,P_4}& ={[-{\omega }^{-2}\mathrm{km},-0.5\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top },\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill x_{0,E}& ={[0\mathrm{km},0\mathrm{km},0\mathrm{km}/\mathrm{h},0\mathrm{km}/\mathrm{h}]}^{\top }.\hfill \end{array}$$

(31)

The states of all the players are constrained with the same D and b specified in (26) as in the previous simulation. The control input of all the pursuers is constrained by ${\rho }_P=1 {\mathrm{km}/\mathrm{h}}^2$, and the control input of the evader is constrained by ${\rho }_E=0.5 {\mathrm{km}/\mathrm{h}}^2$.

Figure 10 shows the reachable sets of all the pursuers as well as the evader. The capture happens at the origin. The evader could be captured by either pursuer 1 or pursuer 2, but only pursuer 1’s trajectory from its initial point to the capture point is shown. Interestingly, the capture happens at the evader’s initial location as that is the last point covered by the union of the pursuers’ reachable sets. The capture happens at $t^{*}=0.84$ h.

5.3. Single Pursuer and Multiple Evaders

Lastly, a game with three evaders and a single pursuer is considered. The dynamics of all the agents are again given by the CW equations. The initial states of the pursuer and evaders are

$$\begin{array}{cc}\hfill x_{0,P}& ={[0,0,0,0]}^{\top },\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill x_{0,E_1}& ={[0,0.5,0,0]}^{\top },\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill x_{0,E_2}& ={[0,-0.75,0,0]}^{\top },\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill x_{0,E_3}& ={[{\omega }^{-2},0,0,0]}^{\top }.\hfill \end{array}$$

(35)

There are no state constraints for the agents but the control inputs are constrained with ${\rho }_P=1 {\mathrm{km}/\mathrm{h}}^2$ for the pursuer and ${\rho }_E=0.25 {\mathrm{km}/\mathrm{h}}^2$ for the evaders.

Figure 11 shows the reachable sets of all the agents at the time instance when the first evader is captured. Figure 12 shows the reachable sets of the remaining two evaders as well as the pursuer’s set when the second evader is captured. Figure 13 shows the reachable sets of the pursuer and the third evader at the time of the last capture, which terminates the game. The captures happen at $t_1=0.92$ h, $t_{1,2}=2.32$ h, and $t_{1,2,3}=t^{*}=6.24$ h.

5.4. Computational Performance

The reachable set strategy for solving time-optimal pursuit–evasion games relies upon bisection (see Algorithm 1) and reachable set calculations (see Section 2.1). At each time within a bisection, a reachable set calculation is required for each player in the game. Each reachable set calculation requires the solution of a single SDP (for the initial simplex) and multiple SOCPs (for growing the faces). SDPs are solved by using the software SDPT3 [42], and SOCPs are solved by using the software Gurobi [43]. Problems are parsed using YALMIP [44] in the MATLAB environment [45].

Table 1. Reachable set calculation times.

	Average	Average	Average	Average
Case	SDP Time (s)	SOCP Time (s)	SOCP Number	Total Time (s)
Section 5.1	0.59	0.0068	40	0.86
Section 5.2	0.49	0.0064	44	0.77
Section 5.3	0.34	0.0059	59	0.69

shows the average SDP solve time, average SOCP solve time, average number of SOCPs solved, and average time to calculate each reachable set for the three simulations described in Section 5.1, Section 5.2 and Section 5.3. The computations were performed on an HP laptop with a 10th generation Intel i7 1.8 GHz processor.

The overall run time of the algorithm is affected primarily by Algorithm 1 parameters $t_{min}$, $t_{max}$, and $ϵ$ because these dictate the number of bisection iterations and hence number of reachable sets that must be calculated. The best way to reduce the total run time is to provide good estimates of the lower and upper bounding times $t_{min}$ and $t_{max}$.

Table 2. Total run time for the reachable set algorithm.

	Total Time (s)	Total Time (s)	Total Time (s)	Total Time (s)
Case	for 5 itr	for 10 itr	for 25 itr	for 100 itr
Section 5.1	8.6	17.2	43.0	172
Section 5.2	19.3	38.5	96.3	385
Section 5.3	13.8	27.6	69.0	276

shows the total run time for the algorithm for the three simulations described in Section 5.1, Section 5.2 and Section 5.3 on instances requiring 5, 10, 25, and 100 bisection iterations (itr).

To summarize, the reachable set strategy takes minutes (on a personal laptop) to solve constrained time-optimal pursuit–evasion games with two to five players.

6. Conclusions

This paper presented a technique using convex optimization to numerically construct reachable sets and to solve time-optimal pursuit–evasion games when the dynamics are linear and all the constraints are convex. These requirements are consistent with linearized relative orbital motion models such as the Clohessy–Wiltshire equations and relative orbital elements. Games with (1) a single pursuer and single evader, (2) multiple pursuers and a single evader, and (3) a single pursuer and multiple evaders were considered. Traditional formulations, such as those based on variational equations, are not tractable in the presence of practical actuator and state constraints. On the contrary, such constraints add minimal complexity to the reachable set strategy described herein. This is true because construction of the reachable sets has been reduced to a sequence of convex programs. Three numerical simulations were presented to demonstrate the strategy for a constellation of satellites in close proximity in low earth orbit. The computation times for the reachable sets were in subseconds, and the total run times for the algorithm were in minutes.