Detection of Hidden Moving Targets by a Group of Mobile Agents with Deep Q-Learning

Abstract

In this paper, we propose a solution for the problem of searching for multiple targets by a group of mobile agents with sensing errors of the first and the second types. The agents’ goal is to plan the search and follow its trajectories that lead to target detection in minimal time. Relying on real sensors’ properties, we assume that the agents can detect the targets in various directions and distances; however, they are exposed to first- and second-type statistical errors. Furthermore, we assume that the agents in the group have errorless communication with each other. No central station or coordinating agent is assumed to control the search. Thus, the search follows a fully distributed decision-making process, in which each agent plans its path independently based on the information about the targets, which is collected independently or received from the other agents. The suggested solution includes two algorithms: the Distributed Expected Information Gain (DEIG) algorithm, which implements dynamic Voronoi partitioning of the search space and plans the paths by maximizing the expected one-step look-ahead information per region, and the Collective Q-max (CQM) algorithm, which finds the shortest paths of the agents in the group by maximizing the cumulative information about the targets’ locations using deep Q-learning techniques. The developed algorithms are compared against previously developed reactive and learning methods, such as the greedy centralized Expected Information Gain (EIG) method. It is demonstrated that these algorithms, specifically the Collective Q-max algorithm, considerably outperform existing solutions. In particular, the proposed algorithms improve the results by 20% to 100% under different scenarios of noisy environments and sensors’ sensitivity.

1. Introduction

Target searching is a fundamental problem in mathematics, which can be traced back to the origins of calculus [1] when it was considered a purely academic task. During World War II, the search problem for static and mobile targets became practical when Koopman [2] established a new research scheme to find German submarines in the Atlantic Ocean.

Formally, the target searching problem can be considered from two viewpoints. The first perspective requires distributing a given search effort over a search domain so that targets are detected with maximal probability. The second perspective requires planning search and navigation paths for the search agents such that they detect the targets with maximal probability in a minimal time.

In 1975, Stone presented the main concepts and ideas in his search theory [3] and substantiated optimizing techniques for distributing search efforts. In 1989, Washburn summarized the used search methods and presented formal techniques for tracking targets using mobile agents [4]. In later research works, search and screening methods, as well as target detection and tracking methods, were somewhat unified into a general probabilistic framework [5,6,7,8] that allowed the consideration of search and detection in different settings and with various levels of certainty.

Further developments in target-searching theory addressed multi-agent multi-target systems. However, along with the obvious advantages of the cooperative search, there is a considerable challenge to the high complexity of the involved algorithms for planning the optimal paths of a group of agents (the terms group, team, set and fleet are used interchangeably). To overcome such a challenge, cooperative search methods have implemented different heuristic approaches [7,9,10,11] and learning techniques [12,13,14].

In this paper, we consider the search for multiple static and moving targets by a group of mobile agents and propose new algorithms for path planning and agent navigation. Following conventional formulations, we consider a search with constrained paths [15] and simulate the detection uncertainty by using intermittently emitting targets [16]. Similar to previously obtained solutions [17,18], the agents utilize an occupancy grid [19,20], which represents their knowledge about the targets’ locations, and we implement methods of deep Q-learning for planning the paths and navigating in the grid [21].

In contrast to other existing algorithms, the suggested solution considers the target-searching problem of mobile targets by a team of agents, such that detecting the targets includes statistical errors of the first and the second types. The solution includes two algorithms:

-

A quick online reactive algorithm that implements dynamic Voronoi partitioning of the search domain and plans the search paths by maximizing expected one-step information gain in each region;

-

A reinforcement learning algorithm that finds the shortest paths of the agents in the group by maximizing the cumulative information about the targets’ locations using deep Q-learning techniques.

In the first algorithm, at each moment, the search domain is divided into Voronoi regions [22] with respect to the current probability map of the domain, and each agent searches in its region independently from the other agents. During the search, the location probabilities of the targets over the search domain change, causing the Voronoi regions to change, thus directly affecting the agents’ search movements and plan over the updated regions. We call this version of the algorithm the Distributed Expected Information Gain (DEIG) algorithm in contrast to the previously developed versions of the centralized EIG algorithm [17,18] that did not use the Voronoi diagram. A clear benefit of this heuristic is its simplicity and low complexity, which allow online implementation over simple agents and components.

In the second algorithm, the decision regarding the next step of the agent is obtained via a deep Q-learning scheme over a neural network based on agent-by-agent value iteration [23]. The network receives the agent’s location, the current probability map and the networks’ parameters of previous agents as input and outputs the preferred move of the agent. This algorithm maximizes the network Q-value over a group of agents and is called the Collective Q-max algorithm.

The proposed algorithms are defined by a set of equations and are illustrated with numerical simulations that are compared with existing methods. These algorithms are implemented in the Python programming language using the PyTorch machine learning library. It is found that the novel deep Q-learning algorithm effectively governs the collective behavior of the search agents and substantially outperforms existing algorithms of collective detection without learning.

The rest of the paper is organized as follows. In Section 2, we introduce the required concepts and notation and formulate the problem. Section 3 is the main section where we present the suggested algorithms. Section 3.1 presents the algorithm of collective detection based on the Voronoi regions, and Section 3.2 presents the algorithm of collective detection using deep Q-learning. Section 4 describes the numerical simulations of the suggested algorithm and its comparisons with the known methods. Section 4.1 addresses the collective detection of static targets, and Section 4.2 addresses the detection of moving targets. In Section 4.3, we consider the training time required by the algorithm with deep Q-learning. Section 5 includes a general discussion about the suggested algorithms, and Section 6 concludes the paper.

2. Problem Formulation

Let $C=\left\{c_1,c_2,\dots ,c_n\right\}$ be a finite set of cells that represent a grid over a two-dimensional domain. In the domain, there are $\xi$ targets, $\xi \le n-1$, and $\eta$ agents, $\eta \le n-1$ (in practice, $\eta \ll n$), each of which can be located in one cell. The agents are equipped with sensors that can detect close-enough targets, and accordingly, each agent plans its motion over the domain, aiming to detect the targets as fast as possible. The mobile targets, in contrast, are not aware of the agents and move independently of the agents’ actions (i.e., we do not consider a game).

Each agent is equipped with sensors that can detect the targets in different directions and distances. Following a conventional detection sensor approach, as presented by Koopman [2,3], we assume that the detection probability of the target increases as the agent moves closer to the target and as the agent is exposed to the target location for a longer period of time. These assumptions are reflected by the simplified Koopman equation of the diction probability of a target in a cell:

$$Pr\left\{target detected in c_i | target located in c_i\right\}=1-\mathit{exp}\left[-\kappa \left(c_i,c_j,\tau \right)\right],$$

(1)

where $\kappa \left(c_i,c_j,\tau \right)~\tau /d\left(c_i,c_j\right)$ is the search effort applied to cell $c_i$ when the agent is in cell $c_j$; $\tau$ is the observation period; and $d\left(c_i,c_j\right)$ is the distance between cells $c_i$ and $c_j$. If all the cells are observed during the same period, $\tau$ can be omitted from the equation, as implemented below.

Moreover, we assume that the detection of a target is not perfect but is exposed to statistical errors of the first and second types, which implies that the agent can erroneously miss an existing target and can erroneously detect a target that does not exist in the cell.

To represent this assumption, the state of cell $c_i\in C$, $i=1, 2, \dots ,n$, at time $t=\mathrm{1,2},\dots$ is denoted by $s\left(c_i,t\right)$. Using occupancy grid techniques [19,20], the state $s\left(c_i,t\right)$ is considered to be a random variable with the values $s\left(c_i,t\right)\in \left\{\mathrm{0,1}\right\}$, such that $s\left(c_i,t\right)=0$ indicates that cell $c_i$ at time $t$ is empty, and $s\left(c_i,t\right)=1$ indicates that cell $c_i$ at time $t$ is occupied by a target. Because these two events are clearly complementary, their probabilities satisfy

$$Pr\left\{s\left(c_i,t\right)=0\right\}+Pr\left\{s\left(c_i,t\right)=1\right\}=1.$$

(2)

We assume that the occupied cells at time $t$ broadcast an alarm signal $\tilde{a}\left(c,t\right)=1$ with the following probability:

$$p_{TA}=Pr\left\{\tilde{a}\left(c,t\right)=1│s\left(c,t\right)=1\right\},$$

(3)

and the empty cells at time $t$ broadcast an alarm signal $\tilde{a}\left(c,t\right)=1$ with the following probability:

$$p_{FA}=Pr\left\{\tilde{a}\left(c,t\right)=1│s\left(c,t\right)=0\right\}=\alpha p_{TA},$$

(4)

where $0\le \alpha <1$. The first alarm is called a “true alarm”, and the second alarm is called a “false alarm”. The probabilities $p_{TA}$ and $p_{FA}$ are the probabilities of detection errors of the first and the second type, respectively.

Relying on Koopman Formula (1), the probability of perceiving the alarms is

$$Pr\left\{alarm percieved at c_j by agent k| alarm sent from c_i\right\}=\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right],$$

(5)

where ${\lambda }^k$, $k=\mathrm{1,2},\dots , \eta$, is the sensitivity of the sensor installed on the agent located in cell $c_j$.

The agents’ knowledge about the targets’ locations at time $t$ is represented by a probability vector $P\left(t\right)=\left\{p_1\left(t\right),p_2\left(t\right),\dots ,p_n\left(t\right)\right\}$, where $p_i\left(t\right)=Pr\left\{s\left(c_i,t\right)=1\right\}$ is the probability that, at time $t$, cell $c_i\in C$ of the domain is occupied by the target. The vector $P\left(t\right)$ is called the “probability map”. We assume that all agents in the group are exposed to the same map $P\left(t\right)$ and share and update it in real time.

Accordingly, the probability of an event ${\tilde{x}}_j\left(c_i,t\right)=1$, $i,j=\mathrm{1,2},\dots ,n$, implying that at time $t$, an agent $k$ located in cell $c_j$ receives a signal from cell $c_i$, is computed as follows:

$$Pr\left\{{\tilde{x}}_j^k\left(c_i,t\right)=1\right\}=p_i\left(t-1\right)p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]+\left(1-p_i\left(t-1\right)\right)p_{FA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right],$$

(6)

and the probability of the event ${\tilde{x}}_j\left(c_i,t\right)=0,$ i.e., implying that the agent does not receive a signal at time $t$ from that cell, is

$$Pr\left\{{\tilde{x}}_j\left(c_i,t\right)=0\right\}=1-Pr\left\{{\tilde{x}}_j^k\left(c_i,t\right)=1\right\}.$$

(7)

The event ${\tilde{x}}_j^k\left(c_i,t\right)$ represents a realistic assumption that the agent cannot distinguish between true and false alarms but only receives a signal, which can be either true or false.

Following the Bayesian scheme, when agent $k$ located in cell $c_j$ receives a signal from cell $c_i$, the probability that cell $c_i$ is occupied by the target is

$$Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=1\right\}=\frac{p_i\left(t-1\right)p_{TA}}{p_i\left(t-1\right)p_{TA}+\left(1-p_i\left(t-1\right)\right)p_{FA}},$$

(8)

and when this agent does not receive a signal from $c_i$, the probability that cell $c_i$ is occupied by the target is computed as follows:

$$Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=0\right\}=\frac{p_i\left(t-1\right)\left(1-p_{TA}\mathit{exp}\left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)}{p_i\left(t-1\right)\left(1-p_{TA}\mathit{exp}\left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)+\left(1-p_i\left(t-1\right)\right)\left(1-\alpha p_{TA}\mathit{exp}\left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)},$$

(9)

Therefore, the targets’ location probabilities are:

$$p_i\left(t\right)=\left\{\begin{array}{c}Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=1\right\} \mathrm{if agent} k \mathrm{received a signal at time} t, \\ Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=0\right\} \mathrm{if agent} k \mathrm{had not received a signal at time} t.\end{array}\right.$$

(10)

Note that, if the true and false alarms are sent with equivalent probabilities $p_{TA}=p_{FA}$, then the agents’ knowledge about the target location does not depend on the received alarms and is represented only by the probability map, as follows:

$$Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=1\right\}=Pr\left\{s\left(c_i,t\right)=1|{\tilde{x}}_j^k\left(c_i,t\right)=0\right\}=p_i\left(t-1\right).$$

(11)

The described process of receiving signals and updating the probability map is illustrated in Figure 1.

In the figure, each agent receives true and false alarms by its onboard sensors and updates the targets’ location probabilities, which form a shared probability map.

In the case of static targets, the targets’ location probabilities $p_i\left(t\right)$, $i=1, 2, \dots , n$, depend only on the agents’ positions at time $t$ and their movements, and in the case of moving targets, these probabilities are defined both by the targets’ and by the agents’ motion.

Following the conventional formulation of search and detection problems [3,4], we assume that the targets are unaware of the agents’ activities and move independently over the domain.

The agents’ goal is to detect all $\xi$ targets in a minimal time. Note that, in the problem of detecting the targets, the agents are not required to chase the targets or to reach their locations physically but rather are required to specify the locations of the targets as definitively as possible by using the information obtained from their sensors.

3. Cooperative Detection: Using Voronoi Regions and Deep Q-Learning

The considered detection process follows the outline of a decision-making procedure and is specified as follows: at time $t$ in cell $c\left(t\right)$, each agent $k$ obtains the probability map $P\left(t\right)$, receives the alarms $\tilde{a}\left(c,t\right)$ from the available cells and decides which cell $c\left(t+1\right)$ it should move to. Accordingly, a main challenge for an efficient cooperative search process is how to divide domain $C$ and distribute the search paths among $\eta$ agents and how to plan each agent’s path to achieve a minimum detection time.

Below, we present two algorithms that address the challenge: the first considers the Voronoi regions of the agents and plans the agents’ motion in their own regions, whereas the second uses the shared probability map and implements deep Q-learning techniques to control the agents’ search.

3.1. Agents’ Actions and Decisions

Consider agent $k$ to be at time $t$ in cell $c\left(t\right)\in C$. We assume that the action $\mathbb{a}\left(t\right)$, which can be chosen by the agent, is one of nine possible movements from cell $c\left(t\right)$, which are “forward”, “right-forward”, “right”, “right-backward”, “backward”, “left-backward”, “left”, “left-forward” and “stay in the current cell”. In other words, the action of the $k$th agent is denoted by

$${\mathbb{a}}^k\left(t\right)\in \mathbb{A}=\left\{\uparrow ,\nearrow ,\to ,\searrow ,\downarrow ,\swarrow ,\leftarrow ,\nwarrow ,\odot \right\}.$$

(12)

A probability map that represents the targets’ locations at time $t+1$ is denoted by $P_{\mathbb{a}}^k\left(t+1\right),$ given that, at time $t$, the $k$th agent chooses action ${\mathbb{a}}^k\left(t\right)$. Then, given action ${\mathbb{a}}^k\left(t\right)$, the immediate expected informational reward of the $k$th agent is given by the Kullback–Leibler distance, namely

$$R_{\mathbb{a}}^k\left(t\right)=D_{KL}\left(P_{\mathbb{a}}^k\left(t+1\right)\left|\right|P\left(t\right)\right),$$

(13)

between the expected agent’s map $P_{\mathbb{a}}^k\left(t+1\right)$ and the current probability map $P\left(t\right)$. This informational reward $R^k\left(t\right)$ forms a basis for making decisions about the agents’ next steps.

In the search by a single agent or by several agents making independent decisions over a shared probability map [18], the choice of the next action is governed by a simple rule:

$${\mathbb{a}}^k\left(t\right)=\underset{\mathbb{a}\in \mathbb{A}}{\mathrm{argmax}}{R}_{\mathbb{a}}^k\left(t\right),$$

(14)

applied by each agent $k=\mathrm{1,2},\dots ,\eta$. This rule represents the agents’ immediate one-step reaction to the changes in the probability maps and states of the targets.

In more complicated search cases by a single agent [21], the decision-making process considers the cumulative reward that is obtained in the sequence of the agent’s actions. Given a policy $\pi$, which is a sequence of the agent’s movements starting from its current position $c\left(t\right)$, the expected cumulative discounted reward obtained by the agent is

$$q_{\pi }\left(c\left(t\right),P\left(t\right),\mathbb{a}\left(t\right)\right)={\mathbb{E}}_{\pi }\left\{{\sum }_{\tau =0}^{\infty }{\gamma }^{\tau }R\left(\mathbb{a},t+\tau \right)\right\},$$

(15)

where the discount factor is $0<\gamma \le 1$. The goal is then to find a maximum value

$$Q\left(c\left(t\right),P\left(t\right),\mathbb{a}\left(t\right)\right)=\underset{\pi }{\mathit{max}}{q}_{\pi }\left(c\left(t\right),P\left(t\right),\mathbb{a}\left(t\right)\right)$$

(16)

of the expected reward $q_{\pi }$ over the policies $\pi$ that can be obtained after action $\mathbb{a}\left(t\right)$ is chosen at time $t$.

Next, we extend reactive rule (13) and decision-making rules (15) and (16) to a search of multiple targets by a group of several interacting agents. The first approach is based on the Voronoi diagrams [22], and the second uses deep-learning techniques [21].

3.2. Reactive Decision Making in Voronoi Regions: A Distributed EIG Algorithm

Let $C=\left\{c_1,c_2,\dots ,c_n\right\}$ be a two-dimensional domain and $P\left(t\right)=\left\{p_1\left(t\right),p_2\left(t\right),\dots ,p_n\left(t\right)\right\}$ be a probability map at time $t$. We assume that the map $P\left(t\right)$ is shared among all $\eta$ agents, $\eta \le n-1$, and is updated with respect to the information obtained by each $k$th agent, $k=\mathrm{1,2},\dots ,\eta$.

The Voronoi region for each agent $k$ is defined as follows. We assume that, at time $t$, the $k$th agent is in the cell $c^k\left(t\right)$, and $d\left(c^k\left(t\right),c\right)$are the distances between cell $c^k\left(t\right)$ and cell $c\in C$ of the domain. Then, the Voronoi region of agent $k$ is subdomain $C^k\left(t\right)\subset C$ of domain $C$, where

$$C^k\left(t\right)=\left\{c | d\left(c^k\left(t\right),c\right)<d\left(c^j\left(t\right),c\right), j=\mathrm{1,2},\dots \eta , j\ne k\right\},$$

(17)

which includes the cells that are closer to the position of the $k$th agent than to the positions of the other agents. Because the agents change their positions with time, the Voronoi regions of the agents are updated accordingly.

The part of the probability map corresponding to the Voronoi region $C^k\left(t\right)\subset C$, $k=\mathrm{1,2},\dots ,\eta$, is denoted by $P^k\left(t\right)\subset P\left(t\right)$. The probability maps $P^k\left(t\right)$ are updated simultaneously with the updates of regions $C^k\left(t\right)$, and the values of the probabilities $p\left(t\right)\subset P^k\left(t\right)$ are updated according to the detection results. The set of Voronoi regions $C^k\left(t\right)$ is called the Voronoi diagram and is denoted by $\mathfrak{C}$, and the set of probability maps $P^k\left(t\right)$ is called the probability atlas and is denoted by $\mathfrak{P}$.

Based on the Voronoi regions, the detection process is conducted with several simple steps. Given positions $c^k\left(t\right)$ of agents $k=\mathrm{1,2},\dots ,\eta$, domain $C$ is divided into Voronoi regions $C^k\left(t\right)$. Each $k$th agent focuses on the corresponding part $P^k\left(t\right)$ of the probability map $P\left(t\right)$, and rule (14) is used to make an independent decision regarding its next movement. When all $\eta$ agents make their decisions, it moves to the next positions $c^k\left(t+1\right)$ and observes the sensors’ output. They update the location probabilities to $p\left(t+1\right)$ to form an updated probability map $P\left(t+1\right)$, and the process continues.

Formally, the Distributed EIG algorithm (Algorithm 1) based on the Voronoi regions is outlined as follows.

Algorithm 1. Cooperative detection with reactive decision making: Distributed EIG algorithm

Input: domain $C=\left\{c_1,c_2,\dots ,c_n\right\}$,   number of agents $\eta$,   initial agents’ positions $c^1\left(0\right),c^2\left(0\right),\dots ,c^{\eta }\left(0\right)$,   set $\mathbb{A}=\left\{\uparrow ,\nearrow ,\to ,\searrow ,\downarrow ,\swarrow ,\leftarrow ,\nwarrow ,\odot \right\}$ of possible actions,   probability $p_{TA}$ of true alarms,   rate $\alpha$ of false alarms and their probability $p_{FA}=\alpha p_{TA}$,   sensor sensitivity $\lambda$,   initial probability map $P\left(0\right)=\left\{p_1\left(0\right),p_2\left(0\right),\dots ,p_n\left(0\right)\right\}$ on $C$,   number of targets $\xi$. Output: target locations ${\widehat{c}}^1\left(T\right), {\widehat{c}}^2\left(T\right),\dots ,{\widehat{c}}^{\xi }\left(T\right)$ at a termination time $T$. Start with $t=0$, initial agent positions $c^1\left(t\right), c^2\left(t\right),\dots ,c^{\eta }\left(t\right)$ and initial probability map $P\left(t\right)=\left\{p_1\left(t\right),p_2\left(t\right),\dots ,p_n\left(t\right)\right\}$. Create the Voronoi diagram $\mathfrak{C}\left(t\right)=\left\{C^1\left(t\right),C^2\left(t\right),\dots ,C^{\eta }\left(t\right)\right\}$, $C^k\left(t\right)\subset C$, $k=\mathrm{1,2},\dots ,\eta$. Create the probability atlas $\mathfrak{P}\left(t\right)=\left\{P^1\left(t\right),P^2\left(t\right),\dots ,P^{\eta }\left(t\right)\right\}$, $P^k\left(t\right)\subset P\left(t\right)$, $k=\mathrm{1,2},\dots ,\eta$. Decision making 4. For each agent $k=\mathrm{1,2},\dots ,\eta$, do: 5. Choose action ${\mathbb{a}}^k\left(t\right)=\underset{\mathbb{a}\in \mathbb{A}}{\mathrm{argmax}}{R}_{\mathbb{a}}^k\left(t\right)$, where $R_{\mathbb{a}}^k\left(t\right)=D_{KL}\left(P_{\mathbb{a}}^k\left(t+1\right)\left|\right|P^k\left(t\right)\right)$. 6. End for Acting 7. For each agent $k=\mathrm{1,2},\dots ,\eta$, do: 8. Apply action ${\mathbb{a}}^k\left(t\right)$: move to new position $c^k\left(t+1\right)$. 9. End for Updating 10. Set $t=t+1$. 11. For each agent $k=\mathrm{1,2},\dots ,\eta$, do: 12. Screen the domain $C$ with respect to the sensor’s abilities. 13. Update the probability map $P\left(t\right)$ to $P\left(t+1\right)$. 14. End for 15. If all $\xi$ targets are detected, then 16. Set $T=t$ and terminate (go to line 20). 17. Else 18. Continue with line 2. 19. End if. 20. Return targets’ locations ${\widehat{c}}^1\left(T\right), {\widehat{c}}^2\left(T\right),\dots ,{\widehat{c}}^{\xi }\left(T\right)$

In the presented Algorithm 1, it is assumed that the number $\xi$ of targets is known, and this number is used to terminate the process. If the number of targets is unknown, then to define the termination, one can use certain measures over the probability map $P^{*}$, such as the entropy of the location probability, which represents sufficient knowledge about the targets’ locations. In this case, the condition in line 15 is substituted or completed using the condition concerning the equivalence of the current probability map $P\left(t\right)$ and the objective map $P^{*}$. In the simulations below, we assume that $P^{*}=\left(p_1^{*},p_2^{*},\dots ,p_n^{*}\right)$ is constant with $p_i^{*}=0.95$, $i=\mathrm{1,2},\dots ,n$, and we use this map as a termination condition. In the search with deep Q-learning, the objective probability map $P^{*}$ is used both for learning and termination; below, we consider this scenario in detail. Note that, in real-world search tasks, the map $P^{*}$ is not necessarily known to the agents, and they can continue their activity. However, it is necessary to control the performance of the algorithms.

The activity of the Distributed EIG algorithm is illustrated in Figure 2, which shows four detection stages of $\xi =30$ static targets by $\eta =6$ agents from a starting time $t=0$ until $t=90$, when all the targets are detected. The figures on the left side show the positions of the targets and the agents, and the Voronoi regions and the right-side figures show the probability maps. The detected targets are denoted by white squares, and the targets for which the detection probability is less than $0.95$ are denoted by gray squares with the brightness proportional to the detection probability.

The presented Algorithm 1 is a direct extension of a previously developed algorithm [17,18] that maximizes the expected information gain over a domain $C$. However, as is demonstrated in Section 4, the use of a Voronoi diagram $\mathfrak{C}\left(t\right)$ and the independent activity of the agents in their Voronoi regions lead to changes in the agents’ motion and a serious decrease in the detection time.

3.3. Collective Deep Q-Learning Approach

Now, we assume that, in addition to sharing the probability map, each agent makes its decisions with respect to the decisions made by the other agents. In this setting, a direct solution to the detection problem is computationally hard, and the exact activity of the group of agents can be specified only for small domains $C$. To overcome this problem, we apply agent-by-agent value iteration [23] together with reinforcement and deep learning techniques [24,25].

The suggested Collective Q-learning approach extends the Q-max search algorithm by a single agent described in detail [21]. It presents similar performances to those of the known optimal algorithms of search for moving targets based on dynamic programming techniques [26,27]. In particular, in comparison with the algorithms based on dynamic programming [26,27] and the algorithms with deep Q-learning, it was demonstrated that, in the search by a single agent and with a $10\%$ false alarm ratio, the algorithm with deep Q-learning results achieves the same solutions as those of the known optimal dynamic programming algorithms. However, for $25\%$ and $50\%$ false alarm ratios, the algorithms with dynamic programming can plan only $7$ steps for the agent in $2$ h of simulation time, and the algorithm with Q-learning plans up to $32$ steps. Consequently, the algorithm with Q-learning results in higher detection probabilities for the targets than those of the algorithm with dynamic programming: $p_1=0.99$ and $p_2=0.95$ versus $p_1=0.84$ and $p_2=0.68$ at best, respectively.

Then, based on the obtained results at a given time, the algorithm with Q-learning plans more agent steps and results in higher detection probabilities than those of the algorithm with dynamic programming. We extended the Q-learning algorithm for the search using multiple cooperating agents.

In contrast to the algorithms with dynamic programming [26,27] and the previously developed Q-max algorithm [21], the new Collective Q-max algorithm is not limited to searching for a single target in small domains; it defines the search by a group of agents acting in relatively large domains.

Each $k$th agent, $k=\mathrm{1,2},\dots ,\eta$, deals with two neural networks: the prediction network and the target or the Q-max network. The input layer of each of the networks includes $2n$ neurons, where $n$ is the size of the domain. The first chunk of $n$ inputs $\left(\mathrm{1,2},\dots n\right)$ receives a binary vector that represents the agent’s position. If the agent is in cell $c_j^k$, then the $j$th input of the network is equal to $1$, and the other $n-1$ inputs are equal to $0$. For convenience, the cell occupied by the considered agent is denoted by the value $10$ instead of $1$. The second chunk of $n$ inputs $\left(n+1,n+2,\dots 2n\right)$ receives the target location probabilities; the $\left(n+i\right)$th input receives the target location probability $p_i$, $i=\mathrm{1,2},\dots ,n$, as it appears in probability map $P$.

The hidden layer of each network is a fully connected linear layer, which consists of $2n$ neurons and the sigmoid activation function $f\left(x\right)=1/\left(1+e^{-x}\right)$ that returns values in the range $\left(0, 1\right)$. The other possibility is to use the SoftPlus techniques with the activation function $f\left(x\right)=\ln \left(1+e^x\right)$ that returns values in the range $\left(0, \infty \right)$. In the simulations, it was observed that both functions result in similar performances. However, for both functions, the observed performance is significantly better than the performance based on the step activation function.

The output layer includes nine neurons with respect to the number $\#\mathbb{A}$ of possible actions: the first output corresponds to the action ${\mathbb{a}}_1=\mathrm{“}\uparrow \mathrm{”}$, “step forward“; the second output corresponds to the action ${\mathbb{a}}_2=\mathrm{“}\nearrow \mathrm{”}$, “step right-forward”, up to action ${\mathbb{a}}_9=\mathrm{“}\odot \mathrm{”}$, which implies “stay in the current cell”. The value of the $i$th output is the maximal expected cumulative discounted reward $Q\left(c_j^k,P,{\mathbb{a}}_i^k\right)$ obtained by the agent if it is in cell $c_j^k$, $j=\mathrm{1,2},\dots ,n$, and given the probability map $P=\left(p_1,p_2,\dots ,p_n\right)$, it chooses action ${\mathbb{a}}_i^k$, $i=\mathrm{1,2},\dots ,9$.

The scheme of the network of the $k$th agent is shown in Figure 3. For the input, the cells occupied by the agents have values of $1$, and the cell occupied by the acting agent is denoted by $10$.

The interaction between the agents is conducted with two channels. The first uses the shared probability map $P$, and the second shares the updated weights of the links in the prediction network. This means that the $\left(k+1\right)$th agent starts training the network with weights that were specified at the end of the training by the $k$th agent.

The Bellman equation used for calculating the maximal cumulative discounted reward is as follows:

$$Q\left(c^k\left(l\right),P\left(l\right),{\mathbb{a}}^k\left(l\right)\right)=\left\{\begin{array}{l}\begin{array}{cc}{R}_{\mathbb{a}}^k\left(l\right)+\gamma \underset{\mathbb{a}\in \mathbb{A}}{\mathit{max}}Q\left(c^{k+1}\left(l+1\right),P\left(l+1\right),\mathbb{a}\right)& k=\mathrm{1,2},\dots ,\eta -1,\hfill \end{array}\\ \begin{array}{cc}{R}_{\mathbb{a}}^1\left(l\right)+\gamma \underset{\mathbb{a}\in \mathbb{A}}{\mathit{max}}Q\left(c^1\left(l+1\right),P\left(l+1\right),\mathbb{a}\right) & k=\eta , \hfill \end{array}\end{array}\right.$$

(18)

where $l=\mathrm{1,2},\dots$ enumerates the steps at the learning stage. This equation forms a basis for updating the weights of the links in the networks. Thus, in the suggested Collective Q-max algorithm, the target network of the $k$th agent is considered a prediction network for the $\left(k+1\right)$th agent up to the $\eta$th agent, which is a predecessor of the $1^{st}$ agent.

The agents’ rewards are calculated using the Voronoi diagram $\mathfrak{C}$. Each $k$th agent considers its region $C^k\in \mathfrak{C}$ and calculates the reward $R_{\mathbb{a}}^k$ within the relevant section $P^k$ of the probability map. Updating the Voronoi diagram $\mathfrak{C}$ is conducted after completing the calculations for all $\eta$ agents.

The SoftMax policy is implemented to select an action, where the probability $p\left({\mathbb{a}}_i|Q;\theta \right)$ of choosing action ${\mathbb{a}}_i$ is defined as follows:

$$p\left({\mathbb{a}}_i|Q;\theta \right)=\frac{\exp \left[Q\left(c,P,{\mathbb{a}}_i\right)/\theta \right]}{\sum _{j=1}^9\exp \left[Q\left(c,P,{\mathbb{a}}_j\right)/\theta \right]},$$

(19)

where $\theta \in \left.\left[0,+\mathrm{\infty }\right.\right)$ is a parameter that governs the randomness of the choice. If $\theta \to 0$, then $p\left({\mathbb{a}}_i|Q;\theta \right)\to 1$ for ${\mathbb{a}}_i=\arg \underset{\mathbb{a}\in \mathbb{A}}{\max }Q\left(c,P,\mathbb{a}\right)$, and $p\left({\mathbb{a}}_i|Q;\theta \right)\to 0$ for all other actions. If $\theta \to \infty$, then $p\left({\mathbb{a}}_i|Q;\theta \right)\to \frac{1}{9}$, which corresponds to a randomly chosen action.

In the learning process, it is assumed that the value of the parameter $\theta$ decreases in the number of steps $l$from its maximal value to zero. Then, in the first learning stages, the agent chooses actions randomly, and then, along with learning, the agent uses the information about the targets’ locations learned by the networks. The first stages with randomly chosen actions are usually interpreted as exploration stages, and the later stages, based on the learned information, are considered exploitation stages.

Finally, $Q\left(c^k\left(l\right),P\left(l\right),{\mathbb{a}}^k\left(l\right);w\right)$ denotes the maximal cumulative discounted reward calculated at step $l=\mathrm{1,2},\dots$ by the network with weights $w$, and $Q^{+}\left(c^{k+1}\left(l+1\right),P\left(l+1\right),{\mathbb{a}}^{k+1}\left(l+1\right);w^{\prime }\right)$ denotes the expected maximal cumulative discounted reward calculated using the vector $w^{\prime }$ of the updated weights following the recurrent Equation (17). Note that the value of $Q^{+}$ is calculated for the next step $l+1$ and for agent $k+1$. Training of the networks is conducted using the temporal difference learning error:

$$∆\left(Q,l+k-1\right)=Q^{+}\left(c^{k+1}\left(l+1\right),P\left(l+1\right),{\mathbb{a}}^{k+1}\left(l+1\right);w^{\prime }\right)-Q\left(c^k\left(l\right),P\left(l\right),{\mathbb{a}}^k\left(l\right);w\right),$$

(20)

which allows sequential computation of the rewards obtained by the agents.

The learning process is illustrated in Figure 4.

The prediction network of the $k$th agent is used for choosing the action and specifying the expected position of this agent at step $l$, and the target network of the $k$th agent is used for calculating the reward after selecting and conducting the action that leads to step $l+1$. Then, the cumulative reward calculated by the target network is used to update the prediction network, and its weights are then used to update the weights of the trained network. Note that, together with the position of the $k$th agent, the target network considers the position of the $\left(k+1\right)$th agent. The target network of the $k$th agent, which at step $l$ is trained with respect to the positions of the $k$th and $\left(k+1\right)$th agents, is considered a prediction network of the $\left(k+1\right)$th agent at step $l+1$.

At the learning stage, we assume that the agents share the probability map, and that each agent updates the probabilities of the targets’ locations over the entire domain. In addition, the reward of the $k$th agent is calculated over its Voronoi region. The Voronoi diagram is updated after updating the positions and the probability map by all $\eta$ agents.

In the above definitions, the cumulative rewards do not depend on the previous trajectories of the agents, and the process that governs the activity of each agent is a Markov process with states that include the positions of the agent and the probability maps. This property allows the use of the offline learning procedure. In this process, at step $l$, instead of the target location probabilities defined by Equations (8)–(10), the networks use the probabilities of the expected targets’ locations $Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=1\right\}$ and $Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=0\right\}$ at step $l$ given the states of the cells in the previous step $l-1$. These probabilities are defined as follows using a Bayesian scheme:

$$\begin{array}{l}Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=1\right\}=\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{p_i\left(l-1\right)p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]}{p_i\left(l-1\right)\left(1-\alpha \right)+\alpha }+\frac{p_i\left(l-1\right){\left(1-\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)}^2}{p_i\left(l-1\right)\left(1-p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)+\left(1-p_i\left(l-1\right)\right)\left(1-\alpha p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)}\end{array},$$

(21)

$$\begin{array}{l}Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=0\right\}=\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{p_i\left(l-1\right)\alpha p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]}{p_i\left(l-1\right)\left(1-\alpha \right)+\alpha }+\frac{p_i\left(l-1\right)\left(1-\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)\left(1-\alpha p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)}{p_i\left(l-1\right)\left(1-p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)+\left(1-p_i\left(l-1\right)\right)\left(1-\alpha p_{TA}\exp \left[-d\left(c_i,c_j\right)/{\lambda }^k\right]\right)}\end{array},$$

(22)

Then, at the learning stage, instead of Equation (10), the targets’ location probabilities are specified by the following condition:

$$p_i\left(l\right)=\left\{\begin{array}{l}Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=1\right\} \mathrm{if\; the\; target\; was\; in} c_i \mathrm{at} l-1, \\ Pr\left\{s\left(c_i,l\right)=1|s\left(c_i,l-1\right)=0\right\} \mathrm{if\; the\; target\; was\; not\; in} c_i \mathrm{at} l-1.\end{array}\right.$$

(23)

The learning process is terminated when the updated probability map $P\left(l\right)$ becomes equal to the objective map $P^{*}$. The detection process, similar to the considered above reactive procedures, can terminate either when the updated probability map $P\left(t\right)$ becomes equal to the objective map $P^{*}$ or when all $\xi$ targets are detected. Note that, in the learning scenario, the objective map $P^{*}$ is necessary for learning and for the control of the algorithm’s performance.

The Collective Q-max algorithm used for the detection of multiple targets includes two stages: the learning stage, during which the agents’ neural networks are trained, and the acting stage, which is an application of the agents (with the trained neural networks) for detecting the targets. The Algorithm 2 is outlined as follows.

Algorithm 2. Collective detection with deep learning: Collective Q-max algorithm

Network structure:   input layer: $2n$ neurons ($n$ agent positions and $n$ target location probabilities, both relative to the size $n$ of the domain),   hidden layer: $2n$ neurons,   output layer: $9$ neurons (in accordance with the number of possible actions). Activation function:   asymmetric sigmoid function $f\left(x\right)=1/\left(1+e^{-x}\right)$. Loss function:   mean squared error (MSE) function. Input: domain $C=\left\{c_1,c_2,\dots ,c_n\right\}$,   number of agents $\eta$,   sensor sensitivities ${\lambda }^1,{\lambda }^2,\dots ,{\lambda }^{\eta }$,   initial agents’ positions $c^1\left(0\right),c^2\left(0\right),\dots ,c^{\eta }\left(0\right)$,   set $\mathbb{A}=\left\{\uparrow ,\nearrow ,\to ,\searrow ,\downarrow ,\swarrow ,\leftarrow ,\nwarrow ,\odot \right\}$ of possible actions,   probability $p_{TA}$ of true alarms,   rate $\alpha$ of false alarms and their probability $p_{FA}=\alpha p_{TA}$,   initial probability map $P\left(0\right)=\left\{p_1\left(0\right),p_2\left(0\right),\dots ,p_n\left(0\right)\right\}$ on $C$,   objective probability map $P^{\mathrm{*}}$,   number of targets $\xi$ (or objective probability map $P^{\mathrm{*}}$). Output: target locations ${\widehat{c}}^1\left(T\right), {\widehat{c}}^2\left(T\right),\dots ,{\widehat{c}}^{\xi }\left(T\right)$ at a termination time $T$. Learning 1. Generate training data set: agents’ positions $c\left(0\right)=\left(c^1\left(0\right), c^2\left(0\right),\dots ,c^{\eta }\left(0\right)\right)$, probability map, $P\left(0\right)=\left\{p_1\left(0\right),p_2\left(0\right),\dots ,p_n\left(0\right)\right\}$. 2. For each agent $k=1,\dots ,\eta$, do: 3. Create the prediction network. 4. Create the target network as a copy of the prediction network. 5. End for. 6. Start with $l=0$. 7. For each pair $\left(c,P\right)$ from the training dataset, do: 8. Create the Voronoi diagram $\mathfrak{C}\left(l\right)=\left\{C^1\left(l\right),C^2\left(l\right),\dots ,C^{\eta }\left(l\right)\right\}$. 9. Create the probability atlas $\mathfrak{P}\left(l\right)=\left\{P^1\left(l\right),P^2\left(l\right),\dots ,P^{\eta }\left(l\right)\right\}$. 10. For each agent $k=1,\dots ,\eta$, do: 11. For each action ${\mathbb{a}}^k\in \mathbb{A}$, do: 12. Calculate the value $Q\left(c^k\left(l\right),P\left(l\right),{\mathbb{a}}^k\left(l\right)\right)$ with the prediction network. 13. Choose action ${\mathbb{a}}^k\left(l\right)$ with the value $p\left(\mathbb{a}|Q;\theta \right)$ and the SoftMax policy. 14. Apply the chosen action to the current position $c^k\left(l\right)$ and obtain the next position $c^k\left(l+1\right)$. 15. Update the probability map $P\left(l\right)$ to $P\left(l+1\right)$. 16. If $P\left(l+1\right)=P^{\mathrm{*}}$, then 17. Set immediate reward $R_{\mathbb{a}}^k\left(l\right)=0$. 18. Set cumulative reward $Q\left(c^k\left(l\right),P\left(l\right),{\mathbb{a}}^k\left(l\right),w\right)=0$. 19. Else 20. Calculate immediate reward $R_{\mathbb{a}}^k\left(l\right)$ with respect to the probabilities of the agent’s parts $P^k\left(l\right)\in \mathfrak{P}\left(l\right)$ and $P^k\left(l+1\right)\in \mathfrak{P}\left(l\right)$ of the probability map. {The Voronoi diagram and the cells associated with the $k$th agent remain, but the values of the probabilities change.} 21. End if. 22. End for. 23. Calculate the values $Q^{+}\left(c^k\left(l+1\right),P\left(l+1\right),{\mathbb{a}}^k;w^{\prime }\right)$ with the target network. 24. Calculate the temporal difference learning error ${∆}_l\left(Q\right)$ for maximal $Q^{+}$. 25. Update the prediction network with respect to the error ${∆}_l\left(Q\right)$. 26. Update the target network with the weights of the prediction network. 27. Update the prediction network of the $\left(k+1\right)$th agent with the target network of the $k$th agent. 28. End for. 29. If the training epochs ended 30. For each agent $k=1,\dots ,\eta$, do: 31. Update the target network with the target network of the $\eta$t agent. 32. End for. 33. Start acting (go to line 36). 34. End if. 35. End for Acting 36. Start with $t=0$. 37. Obtain the initial agents’ positions $c^1\left(t\right),c^2\left(t\right),\dots ,c^{\eta }\left(t\right)$. 38. Obtain the initial probability map $P\left(t\right)=\left\{p_1\left(t\right),p_2\left(t\right),\dots ,p_n\left(t\right)\right\}$. 39. For each agent $k=1,\dots ,\eta$, do: 40. Obtain the values $Q\left(c^k\left(t\right),P\left(t\right),{\mathbb{a}}^k\left(t\right),w\right)$ using the trained network. 41. Choose action ${\mathbb{a}}^k\left(t\right)$, which provides the maximum $Q\left(c^k\left(t\right),P\left(t\right),{\mathbb{a}}^k\left(t\right),w\right)$. 42. Apply the chosen action to the current position $c^k\left(t\right)$ and obtain the next position $c^k\left(t+1\right)$. 43. Screen the domain $C$. {The $k$th agent screens all of domain $C$ with respect to the abilities of the on-board sensors.} 44. Update the targets’ locations ${\widehat{c}}^1\left(t\right),{\widehat{c}}^2\left(t\right),\dots ,{\widehat{c}}^{\xi }\left(t\right)$. 45. Update the probability map $P\left(t\right)$ to $P\left(t+1\right)$. 46. End for. 47. If all $\xi$ targets are detected (or if $P\left(t\right)=P^{\mathrm{*}}$), then 48. Set $T=t$ and terminate (go to line 53). 49. Else 50. Set $t=t+1$. 51. Continue detection (go to line 39). 52. End if. 53. Return targets’ locations ${\widehat{c}}^1\left(T\right),{\widehat{c}}^2\left(T\right),\dots ,{\widehat{c}}^{\xi }\left(T\right)$.

The presented Algorithm 2 extends a previously developed detection algorithm by a single agent [21] and follows the same techniques for training the networks. However, instead of using individual prediction and target networks, it considers the prediction network of the next agent as its target network. Thus, each agent uses the training results of the previous agent and shares its knowledge with the next agent.

4. Numerical Simulations

The suggested algorithm 1 and algorithm 2 were implemented and tested in several settings, and their functionality was compared against a previously developed heuristic algorithm based on the same expected information gain.

Numerical simulations were implemented using the Python programming language, including the PyTorch machine learning library. The trials were executed on a PC Intel^® Core™ i7-10700 CPU with 16 GB RAM (eight cores) with a GPU Nvidia GeForce GTX 1650Super (1280 CUDA Cores). Using this computer system, we measured the run time of the simulations over different datasets, demonstrating that the suggested algorithms can be implementable on conventional computers with CUDA parallel computing architecture and do not require specific parameters for their functionality.

In the Collective Q-max algorithm, the initial weights $w$ of neural networks were generated via the corresponding procedures of the PyTorch library. The optimizer used in the simulation was the ADAM optimizer from the PyTorch library. The size of the training data set was $\mathrm{100,000}$, the number of training epochs was $30$, and the average time required for the offline training of the prediction network was approximately $10$ h on the described computation platform. After an offline training period, online decision making was conducted directly by applying immediate selection without additional calculations.

In the simulations, we compared four algorithms: random search, in which the agents move randomly in the domain; centralized and Distributed EIG algorithms; and the Collective Q-max algorithm.

In all algorithms, we used 40 × 40 and 50 × 50 cell grid sizes, which correspond to practical military and civil tasks. In the case of military applications requiring search and detection in active operations, the size of the cell in the detection tasks is up to $0.5 {\mathrm{k}\mathrm{m}}^2$; thus, the grid of the indicated size represents a city district or the natural terrain of up to $1.0 {\mathrm{k}\mathrm{m}}^2$. In the case of civil applications, for example, in the search and rescue tasks and military logistic tasks, such grids represent the terrain of more than $6.0 {\mathrm{k}\mathrm{m}}^2$ while searching objects the size of a human and up to $25.0 {\mathrm{k}\mathrm{m}}^2$ while searching objects the size of an automobile.

The sensitivity of the sensor is specified by the parameter $\lambda$ (see Equation (5)), and in the simulations, we used the values $\lambda =10$ and $\lambda =15$, which are associated with the sensitivity of the Lidar sensor, for which the probability of detecting the target decreases from $1$ (detection at a distance of $0$ m) to $1/e$ (detection at a distance of $15$ m). Certainly, in real-world tasks, the sensors’ sensitivity $\lambda$ should be defined with respect to the type and quality of the sensor.

4.1. Detection of Static Targets

The first set of simulations dealt with the detection of static targets. The results of these simulations are illustrated by the detection of $\xi =30$ targets by $\eta =6$ agents in the domain of size $n=40\times 40=1600$ cells. The probability of a true alarm is $p_{TA}=1$, the sensor sensitivity is $\lambda =15$, the initial target location probabilities are $p_i\left(0\right)=0.05$ and $i=\mathrm{1,2},\dots ,n$, and the initial agent positions are $c^k\left(0\right)=\left(\mathrm{0,0}\right)$ and $k=\mathrm{1,2},\dots ,\eta$.

Let us consider the cumulated rewards obtained by the agents. The dependence of the cumulative reward on the time for the ratio of false alarms $\alpha =0.5$ (see Equation (4)) is shown in Figure 5.

The fastest growth of the cumulative reward is provided by the Collective Q-max algorithm, and the slowest growth is provided by the random search method. The random search is slightly outperformed by the grid search, in which the paths of the agents are determined in advance. At the beginning of the search process at $t=0$, the domain is divided into equal areas with respect to the number of agents, and each agent conducts the center of gravity search in its area according to the initial probability map. Finally, intermediate growth is provided by the EIG algorithms, for which the Distributed EIG algorithm outperforms the centralized EIG algorithm.

The same tendency was observed for the number of search actions (moves) conducted by the algorithms until detecting all the targets. The results of these simulations with different $\alpha$ ratios of false alarms, for which the sensitivity of the agents’ sensors is defined by $\lambda =15$, are summarized in

Table 1. Number of actions up to the detection of $30$ static targets by $6$ agents for different ratios of false alarms. Sensor sensitivity is defined by $\lambda =15$; the domain size is $40\times 40$.

Detection Algorithm	Number of Actions up to the Detection of 30 Static Targets $\mathit{n}=40\times 40$$, \mathit{\lambda }=15$
	$\mathit{\alpha }=0.25$	$\mathit{\alpha }=0.5$	$\mathit{\alpha }=0.75$
Random search	$120$	$175$	$450$
Grid search	$112$	$170$	$436$
Centralized EIG	$88$	$122$	$232$
Distributed EIG	$79$	$98$	$152$
Collective Q-max	$75$	$88$	$102$

.

As expected, the worst results are obtained for a random search, in which the agents move randomly in the domain, whereas the best results are provided by the Collective Q-max algorithm. Note that the difference between the best results provided by the Collective Q-max algorithm and the other methods increases with the ratio of false alarms.

For comparison,

Table 2. Number of search actions up to the detection of $30$ static targets by $6$ agents for different ratios of false alarms. Sensor sensitivity is given by $\lambda =10$; the domain size is $40\times 40$.

Detection Algorithm	Number of Actions up to the Detection of 30 Static Targets $\mathit{n}=40\times 40$$, \mathit{\lambda }=10$
	$\mathit{\alpha }=0.25$	$\mathit{\alpha }=0.5$	$\mathit{\alpha }=0.75$
Random search	$205$	$310$	$>600$
Grid search	$198$	$305$	$590$
Centralized EIG	$145$	$221$	$321$
Distributed EIG	$134$	$210$	$285$
Collective Q-max	$106$	$138$	$178$

presents the number of search actions executed by the benchmarked algorithms to detect all the targets with a lower sensitivity of the agents’ sensors given by $\lambda =10$.

Because the sensitivity $\lambda =10$ of the agents’ sensors is lower than that in the previous scenario, the agents need more search actions to detect the targets.

To stress the difference in the activity of the considered algorithms, Figure 6 illustrates the effectiveness of the search algorithms in comparison to Collective Q-max, which achieves a maximum efficiency of 100%.

The figure shows that the proposed Collective Q-max algorithm outperforms the other algorithms. The difference in the performance of the Collective Q-max algorithm and the other algorithms depends on the sensitivity of the sensors. For example, for $\lambda =15$ and $\alpha =0.25$, the Collective Q-max algorithm requires nearly two times less time to detect the targets than that of the random search, and for $\lambda =15$ and $\alpha =0.75$, it requires nearly three times less time than that of the random search. Similar ratios are observed for the other algorithms and values of $\lambda$ and $\alpha$.

The same relations were demonstrated in the other scenarios. For example, the number of actions conducted by the algorithms detecting up to $100$ static targets by $10$ agents equipped with sensors of sensitivity $\lambda =10$ are presented in

Table 3. Number of search actions up to the detection of $100$ static targets by $10$ agents for different ratios of false alarms. Sensor sensitivity is given by $\lambda =10$; the domain size is $50\times 50$.

Detection Algorithm	Number of Actions up to the Detection of 100 Static Targets $\mathit{n}=50\times 50$$, \mathit{\lambda }=10$
	$\mathit{\alpha }=0.25$	$\mathit{\alpha }=0.5$	$\mathit{\alpha }=0.75$
Random search	$202$	$275$	$>600$
Grid search	$196$	$272$	$>600$
Centralized EIG	$156$	$215$	$322$
Distributed EIG	$135$	$198$	$305$
Collective Q-max	$102$	$135$	$165$

; the domain size is $n=50\times 50$.

Similar to the above, in this scenario, the Collective Q-max algorithm leads to essentially better results than those of the other algorithms, and the difference in the number of search steps required by the known methods and the suggested algorithm depends on the ratio of false alarms. It is seen that, for higher ratios of false alarms, this difference is higher. For example, for $\alpha =0.75$, the difference in the search steps in the random search and in the suggested algorithm is greater than $400$, and for $\alpha =0.25$, this difference is $100$. Therefore, when considering the detection of static targets, the suggested Collective Q-max algorithm is shown to be preferable with respect to the other methods, and such preference is more significant in tasks with a high ratio of false positive errors. If the ratio of false positive errors is relatively low, then the suggested Distributed EIG algorithm can also be applied.

4.2. Detection of Moving Targets

The second set of simulations dealt with the detection of moving targets. As above, we present the results of simulations with $\xi =30$ targets searched by $\eta =6$ agents in a domain of size $n=40\times 40=1600$ cells. The probability of a true alarm is $p_{TA}=1$, the sensor sensitivity is given by $\lambda =15$, the initial targets’ location probabilities are uniformly distributed with $p_i\left(0\right)=0.05$, $i=\mathrm{1,2},\dots ,n$, and the initial agents’ positions are $c^k\left(0\right)=\left(\mathrm{0,0}\right)$, $k=\mathrm{1,2},\dots ,\eta$. For simplicity, here, we consider slowly moving targets such that the probability of staying in the same cell is substantial, $Pr\left\{\widehat{\mathbb{a}}\left(t\right)=\odot \right\}=0.9$, and the probability of taking a search step in any direction is $Pr\left\{\widehat{\mathbb{a}}\left(t\right)\in \mathbb{A}\backslash \odot \right\}=\frac{1-Pr\left\{\widehat{\mathbb{a}}\left(t\right)=\odot \right\}}{\#\mathbb{A}-1}=0.0125$.

For convenience, we consider the results of the simulations with the same sensor sensitivities, i.e., with $\lambda =15$ and $\lambda =10$ and three ratios of false alarms. However, in this study, we limit the false alarm ratios to lower values: $\alpha =0.1$, $\alpha =0.15$ and $\alpha =0.25$. The reason for this is that, in scenarios with higher false alarm ratios, the detection process takes too much time, and the resulting high numbers of search actions, despite the use of the same relations, are less illustrative.

The results of the simulations with different false alarm ratios $\alpha$ and a fixed sensitivity of the agents’ sensors are given by $\lambda =15$. These results are summarized in

Table 4. Number of search actions up to the detection of $30$ moving targets by $6$ agents for different ratios of false alarms. Sensor sensitivity is given by $\lambda =15$; the domain size is $40\times 40$.

Detection Algorithm	Number of Actions up to the Detection of 30 Moving Targets $\mathit{n}=40\times 40$$, \mathit{\lambda }=15$
	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=0.15$	$\mathit{\alpha }=0.25$
Random search	205	310	720
Grid search	$190$	$301$	$710$
Centralized EIG	140	208	366
Distributed EIG	132	180	340
Collective Q-max	$115$	$132$	188

.

As seen above, the worst results correspond to the random search, and the best results are provided by the Collective Q-max algorithm. In addition, note that, for moving targets, when $\alpha =0.25$, the random search method requires six times more actions ($720$) than those of the case of the detection of static targets ($120$, see Table 1). For the same $\alpha$ value, the EIG algorithm requires nearly four times more search actions ($366$ vs. $88$ and $340$ vs. $79$). Finally, the ratio between these numbers of required actions between the static and the dynamic case when applying the suggested Collective Q-max algorithm is only $\frac{188}{75}\approx 2.5$.

For comparison,

Table 5. Number of search actions required up to the detection of $30$ moving targets by $6$ agents for different probabilities of false alarms. Sensor sensitivity is given by $\lambda =10$; the domain size is $40\times 40$.

Detection Algorithm	Number of Actions up to the Detection of 30 Moving Targets $\mathit{n}=40\times 40$$, \mathit{\lambda }=10$
	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=0.15$	$\mathit{\alpha }=0.25$
Random search	280	410	850
Grid search	$268$	$405$	$844$
Centralized EIG	202	245	475
Distributed EIG	185	225	450
Collective Q-max	$128$	$155$	220

presents the number of search actions conducted by the algorithms until detecting all targets, with lower sensitivity for the agents’ sensors, $\lambda =10$.

As expected, because the sensitivity of the agents’ sensors is given by $\lambda =1$0, the ratios between the number of required actions are lower compared to this ratio in the previous scenario, and the agents need more actions to detect the targets. In addition, the number of search actions follows the same tendency as above. Thus, the suggested Collective Q-max algorithm strongly outperforms all the other methods.

The difference in the activity of the considered algorithms is exemplified in Figure 7.

Finally, parallel to the results presented in Table 3, the number of actions executed by the algorithms up to the detection of $100$ moving targets by $10$ agents equipped with sensors of sensitivity $\lambda =10$ is presented in

Table 6. Number of search actions required up to the detection of $100$ moving targets by $10$ agents for different ratios of false alarms. Sensor sensitivity is given by $\lambda =10$; the domain size is $50\times 50$.

Detection Algorithm	Number of Actions up to the Detection of 100 Moving Targets $\mathit{n}=50\times 50$$, \mathit{\lambda }=10$
	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=0.15$	$\mathit{\alpha }=0.25$
Random search	$295$	$422$	$865$
Grid search	$285$	$411$	$861$
Centralized EIG	$205$	$247$	$465$
Distributed EIG	$190$	$232$	$447$
Collective Q-max	$132$	$162$	$225$

. The size of the search domain is $n=50\times 50$.

In this scenario, the Collective Q-max algorithm also obtains the best results in comparison with the other algorithms, and its advantage is higher as the ratio of false alarms increases.

4.3. Learning Errors and Run Time of the Collective Q-Max Algorithm

We consider two characteristics of the suggested Collective Q-max algorithm, which emphasize the feasibility of its implementation in solving real-world problems.

As indicated above, in the presented simulations, we used $30$ learning epochs with $\mathrm{100,000}$ training data sets. This choice of parameters is based on the dependence of the percentage of learning errors on the number of learning epochs. The graph of this dependence is shown in Figure 8.

The percentage of learning errors decreases exponentially and converges to the value $0.1\%$ after $30$ training epochs.

Next, we consider the training run time. The run times required for training the neural networks in different simulation settings for the abovementioned computation system are summarized in

Table 7. The training run times and learning errors for different data sets and search domains.

$\mathbf{Domain} \mathbf{Size} {\mathit{n}}_{\mathit{x}}\times {\mathit{n}}_{\mathit{y}}$	Number of Nonzero Weights in the Neural Network	Size of the Data Set	Run Time for One Epoch [minutes]	Mean Squared Error *
$20\times 20$	$648,009$	$50,000$	$5$	$0.20$
		$\mathrm{100,000}$	$9$	$0.13$
$40\times 40$	$\mathrm{10,272,009}$	$\mathrm{50,000}$	$12$	$0.28$
		$\mathrm{100,000}$	$20$	$0.17$
$50\times 50$	$\mathrm{25,050,009}$	$\mathrm{50,000}$	$14$	$0.30$
		$\mathrm{100,000}$	$22$	$0.19$

* Error was calculated over the temporal difference errors at the validation stage at epoch $t=30$.

.

Whereas the size of the domain and the number of links in the network increase exponentially, the mean squared error increases very slowly and remains relatively small. Even on a commodity PC system, as described above, the computations require a reasonable amount of time. Note again that, after training, decision making is processed using the already trained networks that allow the application of the suggested techniques in online algorithms.

5. Discussion

In this paper, we consider the problem of detecting and tracking multiple hidden static and moving targets by a team of mobile agents and suggest two algorithms for agent navigation under uncertainty. In contrast to existing methods, the suggested algorithms effectively resolve uncertainty about the expected target’s locations and navigate the agents in the presence of false-positive and false-negative detection errors.

The first algorithm, the Distributed EIG algorithm (DEIG), is a reactive online procedure in which each agent observes the environment and makes its decision using the obtained Expected Information Gain (EIG) value. This algorithm extends the previously developed greedy centralized EIG algorithm.

The DEIG algorithm uses Voronoi diagrams in parallel with the probability map. As a result, the search efforts are distributed in such a manner that the agents first consider their neighborhoods and then continue with detection in the other regions.

The main advantage of the DEIG algorithm is its computational simplicity and short run time. The conducted simulation studies demonstrate the algorithm’s effectiveness, especially in cases with a low ratio of detection errors of the second type.

The second algorithm, the Collective Q-max (CQM) algorithm, includes deep Q-learning abilities that can be used both online and in the offline stage. The algorithm extends the previously developed Q-max algorithm for a team of agents.

The simulations of the algorithms were conducted using grids of sizes representing practical military and civil situations. A series of simulations show that both proposed algorithms outperform the known greedy and learning procedures and require reasonable and relatively moderate computation time. Note that, in the same situations, the existing algorithms for search either do not converge or require an extremely long computation time, which makes these algorithms unusable.

The CQM algorithm utilizes the learned information about the targets’ location probabilities, the detection errors and the targets’ motion. In the considered simulations, the targets’ motion was characterized as 90% static (remaining stationary) and 10% as a random walk without any specific movement patterns. It is reasonable to expect that, if the targets move according to certain movement patterns, the CQM algorithm would be able to learn and detect these patterns, which can result in a shorter search time. Such scenarios, especially the case of the search game in which targets attempt to evade the search agents, require additional considerations.

The basic idea of the suggested learning procedure is the sequential training of the prediction neural network. Each agent is trained by its predecessor in the array of agents and trains its successor by simultaneously counting the expected actions. The training can be conducted both offline and online or can be processed in a mixed regime with online updating of the offline training results. In the considered scenarios, the order of the agents varies and is specified with respect to the distance of the agent from the starting point. Such a definition allows for the inclusion of all agents in minimal time in the search. In the other scenarios, the order can be predefined by the enumeration of the agents or can vary with respect to information measures or distances between the agents. Studies on different ordering schemes remain for further research.

The further development of the suggested algorithms will include their extension to detection in a domain with shadowing. For example, such tasks appear in the search in the terrain where certain areas are shadowed and exposed only during the agents’ motion by certain trajectories.

This problem, as well as the search by marine on-water and underwater agents, gives rise to the detection problem with piecewise sensing, which requires combining the probability maps from the parts obtained by different agents at various times.

Finally, we plan to consider the influence of the expected information that is obtained by the agents in the next steps of the search and to determine the dependence of the discount factor used in the learning procedure with this information.

6. Conclusions

In this paper, we suggest two algorithms for detecting multiple static and moving targets by a team of mobile agents. The first algorithm is a reactive procedure, which implements the Voronoi diagrams, and the second algorithm is a procedure that implements deep Q-learning, which can be conducted both on- and offline.

Numerical simulations of the suggested algorithms in different scenarios of search for static and mobile targets demonstrate that the algorithm with deep Q-learning considerably outperforms the benchmark algorithms, including the algorithm based on Voronoi diagrams.

The main advantages of the algorithm are demonstrated in a noisy environment with the use of sensors with low sensitivity with many statistical errors of the first and second types.

The suggested algorithms can be used both for the further development of probabilistic search and detection methods and for practical applications for navigating autonomous drones and ground vehicles in the protection of facilities, smart city maintenance, mapping and surveying, precision agriculture, etc.