Multiple-Network-Based Control System Design for Unmanned Surveillance Applications

Abstract

Networks are essential components in the surveillance applications of control systems. In unmanned surveillance applications, numerous agents are employed to provide unmanned services. These agents secure large areas and communicate with a control system, checks their status and sends/receives data via multiple networks. These networks need to assign roles based on the application characteristics. In this study, we propose the design of a multiple-network-based control system for large surveillance areas. To this end, an interface for transmitting mission commands to agents needs to be developed because it can allow users to monitor and assign tasks to all agents. The proposed system is developed as a test bed connected to fixed/mobile agents using LoRa, Wi-Fi, Bluetooth, and LTE communication methods; moreover, its usability was tested in a real environment.

1. Introduction

Recently, owing to the Fourth Industrial Revolution, interest in unmanned services provided by multiple agents has been increasing. Numerous studies have been conducted to autonomously perform missions using fixed and mobile agents. As these services enhance human convenience, their use is increasing in various fields, including social, medical, security, information, care, manufacturing, maritime, aviation, and disaster fields [1,2]. In particular, security services are major autonomous services that can reduce social anxiety and monitor high-risk, unattended areas. Currently, most surveillance systems mainly detect intruders or monitor fires based on images [3,4]. With the development of communication technology, surveillance systems composed of only fixed camera devices are being replaced with those composed of robots and cameras [5,6,7]. However, most surveillance systems are used in indoor environments or limited areas and mainly perform single-sided image transmission for surveillance [8]. In order to compensate for these shortcomings, more active security services using ground or air agents are required [9]. This improved service must utilize a multiple communication method, be capable of direct and rapid intervention through fixed and mobile agents, and monitor through a communication system [10,11]. The existing data collection system that uses multi-band communication is difficult to apply to robot services [12]. Moreover, the fusion system with multiple data-sharing methods, although originally designed to enable high-speed connection, can be used with a large number of databases [13]. Data and transmission security is always a consideration [14,15]. Nevertheless, for the unmanned security robot service, a communication system capable of bidirectional communication and covering large areas is required.

Meanwhile, the control system must provide a useful interface for the user. To perform security missions, the user must be able to interact with agents on the interface; therefore, the interface must be designed with the objective of facilitating the monitoring of transmission and reception data (e.g., agent status). The interface of the monitoring system is intended for user-centered interaction such as gestures [16,17]. Although an interface with a monitoring-robot system using wireless communication exists, its application to actual sites is limited in a wide space, or by data size [18]. Therefore, surveillance systems and interfaces are required to analyze the actual security mission and define and design the necessary tools. Herein, we propose a control system that incorporates various communication techniques; with this system, numerous robots can communicate in both directions in large outdoor areas. This system employs four types of communication methods, namely, long range (LoRa) [19], wireless fidelity (Wi-Fi), Bluetooth, and long-term evolution (LTE), for communication between the control system and fixed/mobile agents. Further, it provides an interface that enables users to monitor the image data received from the agents in real time and command the defined mission by selecting a surveillance area. For performance analysis, unmanned security mission tests were conducted by installing the proposed system at two different sites.

This study presents a multiple network-based control system for unmanned surveillance applications. The system performed 48 h continuous guarding and collected data for more than 6 months. The contributions of this paper are as follows:

• The long-term cost scenario demonstrated using multiple networks.
• We realized a real-time unmanned security system by simultaneously using fixed and mobile agents.
• We present examples of applying this system to multiple sites at the same time.

The rest of this paper is structured as follows. The roles of the configuration and communication method of the multiple-network system and data transmission code are detailed in Section 2. The control system and interface are introduced in Section 3. The results of building the control system are presented in Section 4. Finally, the conclusions are presented in Section 5.

2. Multiple-Network System

2.1. Four Types of Communication Methods

Figure 1 shows the four types of communication methods (LoRa, Wi-Fi, Bluetooth, and LTE) that play a role in the consideration of each characteristic for the mobile agents. The proposed system uses Bluetooth communication (first priority) for direct user control, LoRa communication (second priority) for emergencies, Wi-Fi communication (third priority) for normal performance, and LTE communication for sensor data transmission. The flowchart of the control data for the mobile agent is shown in Figure 2. The fixed agent sends sensor and cognitive data through one-way communication using the Internet.

Therefore, the mobile agents execute control commands according to the priority of each communication method. First, Bluetooth is employed for short-distance control, because the operation of multiple robots requires reliable communication. Second, normal control and status communication are performed to cover a large outdoor area using multiple Wi-Fi modules. Finally, the mobile agent is configured to transmit sensor data using LTE with a wide bandwidth. Details of the communication methods used are as follows:

• Bluetooth (controller): short-range (within 20 m) wireless control (mode (manual/autonomous/remote), linear speed, angular speed, speed increase, speed decrease);
• LoRa (emergency): emergency start/stop, return command, reboot command when communication is impossible in normal status;
• Wi-Fi (usual): mission command (robot mission (patrol, monitoring), target point list, mission start, emergency stop) and normal message (robot current position, previous target point, current target point, robot status message), remote control (start/stop, linear speed, angular speed, speed increase, speed decrease);
• LTE (for sensor data): multimodal sensor module-based image data (three types: color, night vision, thermal image) and 10 Hz 3D LiDAR Point Cloud data.

2.2. Implementation of the Network System

Table 1. Comparison of the communication module specifications.

Method	Model	Specifications
		Distance	Frequency	Bandwidth
LoRa	uLory	10 km	917–923 MHz	125 kHz
	LoryG	10 km	917–923 MHz	125 kHz
Wi-Fi	Spider	500 m (Hand over)	5 GHz	20 MHz
LTE	ME-Y51Wl	100 km	8,502,600 MHz	100 MHz
Bluetooth	Logitechf710	10 m	2.4 GHz	1–2 MHz

lists the details of each communication module constituting the system used with the multiple communication method. For LoRa, two models for transmission and reception are used, both of which employ frequencies in the range of 917–923 MHz and have a bandwidth of 125 kHz. Although this method has a narrow bandwidth, it is suitable for communicating control commands in emergencies because a coverage of ≥10 km is possible, even in an urban environment. The communication method is implemented following the transmission protocol and using send/receive messages; the details are presented in

Table 2. LoRa transmission protocol.

Variable Name	Type	Contents
Robot ID	uint16	Robot identification
Timestamp	int32	Time
Millisecond	uint16	Time
Message ID	uint16	Message identification
cmd	int32	0: mission stop
		1: mission start
		2: surveillance start
		3: return start

and Figure 3.

By contrast, the Wi-Fi communication module has the advantage of enabling handover—with a coverage of 500 m and performance of 300 Mbps—as shown in Figure 4. Since reception and transmission are possible with a single module, it can be mounted onto mobile agents to easily address blind spots. To transmit robot control commands via Wi-Fi communication, message queuing telemetry transport (MQTT) is used. This method follows the TCP/IP protocol and is very useful for operating multiple agents in a manner that focuses on embedded devices.

Table 3. MQTT transmission protocol.

	Variable Name	Type	Contents
	Path Count	int32	Path Amount
Robot path protocol	Path info (100)	double	Map coordinate
			(x, y, heading)
Robot mission protocol	cmd	int32	0: mission stop
			1: mission start
			2: surveillance start
			3: return start

and Figure 5 present the protocols for communication using this method. In other words, it is possible to transmit and receive control commands through communication between the control system (cloud or local server) and agents.

Next, the LTE module transmits three types of image data (color, thermal image, and night vision) using the multimodal sensor module [20] mounted onto the fixed and mobile agents and the ME-Y51WL model. It is constructed on the security site using several agents (e.g., three fixed and three mobile), and sensor data are transmitted as a file at a speed of 10 fps to monitor and recognize abnormal situations; the details are shown in Figure 6. Finally, Bluetooth allows the user to control the robot with direct control commands using a wireless controller.

3. Unmanned Surveillance Control System

The architecture of operating an integrated control system for guarding an outdoor environment using multiple robots is shown in Figure 7. It comprises four layers: the network layer that uses the four types of communication methods mentioned in Section 2; the system layer that includes middleware for connecting with several agents; and the UI that enables monitoring, reporting, commands, and mobile agent control. Further, it comprises an application layer and a data layer for storing monitoring data. Therefore, the system is designed to perform various security missions. Moreover, monitoring and positioning interfaces are added for access to the status information of the mobile agents (e.g., battery level, number of agents available, working hours, and agent failure).

To perform the different security missions, allocating a path for each mobile agent mission is necessary. Therefore, the control system assigns missions (e.g., patrol, surveillance, and guidance), receives sensor data and status information from fixed and mobile agents, confirms with the administrator, and configures the interface to perform the task, as shown in Figure 8. In the COMMAND modal window, the function to give control commands to the robot is included. In the ROBOT ENABLE modal window, the user can select the mobile agent they want to control on the security site. In the WORK modal window, the user can send commands such as: patrol, monitoring, guidance, emergency, access control, return to all, and stop all tasks, to the mobile agent. Finally, in the MISSION modal window, the user can select mission stop, mission start, watch start, and return. Therefore, the control system has an interface that the user can utilize to comprehensively control the agent’s mission (e.g., surveillance, patrol, guidance, and return) and to select the numbers of agents and the mission of each agent. Further, the user can control the agents at any time while monitoring.

Moreover, the control system configures the Map UI by receiving the user’s input, such that the user can allocate the mission and route information to command the mobile agents to perform the mission. Further, it uses the coordinates on the map to perform the mission and transfers it to the global path-planning algorithm [8]; thereafter, it displays the individual routes received on the map, allowing the administrator to intuitively set the mission (Figure 9).

Control monitoring is possible using real-time location information received from a robot while performing a given mission. In Figure 10, this is achieved using Google API, and the real-time location of the robot can be displayed on Google Maps through the label function; consequently, the progress of the security mission can be known. Furthermore, by managing the state of multiple mobile agents based on the status information, the cost efficiency for covering the large security area is improved by simultaneously linking with other agents as they have autonomous charge. In Figure 11, the upper part is a window to check the communication status of the control system and the status of each robot, and the lower part displays the working time, work type, fault status, and battery level.

4. Control System Implementation

The proposed control system in the outdoor environment based on multiple fixed and mobile agents created a unique tag for message transmission between the sending and receiving agents. Each agent was connected to one agent core and transmitted a message by recognizing the tag, regardless of the IP and port number, as shown in Figure 12. When creating an agent, the ID was registered in the agent core, and when the sending agent transmitted the message, the agent core separated the message and transmitted it to the receiving agent.

With the control system built using the agent core-based framework, it was possible to communicate with an agent using Linux (communication server) and Windows (UI and status check). Therefore, by integrating the data of the agents, an integrated control system was built on two locations; this allowed the user to check the agent status, mission information, and abnormal situation information or to assign tasks to multiple agents (Figure 13).

Figure 14 shows the proposed system performing repeatedly over a long period of time (Continuous 48 h or 6 months long term).

5. Conclusions and Future Works

We proposed a system that can be used to control multiple agents (fixed and mobile agents) using multiple communication methods. Further, this system can also be used to provide security services. An efficient UI and control system for security services in large outdoor environments was developed, and the necessary factors were analyzed. The proposed system was applied to two sites (Pohang and Gwangju). The multimodal sensor dataset built using the proposed system is used openly [21] and the data are stored in the form of a probability map to learn and respond to abnormal situations [22]. This system is expected to be used not only for practical application of domestic robots and surveillance systems but also to impact related research areas. Further, different communication methods will have to be developed to suit various purposes.

In future, this application will be expanded to a wide-area space communication system in an extreme environment. To this end, we are developing a long-distance Wi-fi module (communication distance greater than 50 km) that can be used in Antarctica by applying Iridium communication to replace LoRa and by working to minimize latency to enable real-time remote control.