An Application of IoT in a Drone Inspection Service for Environmental Control

Abstract

This paper presents an exploratory activity with a drone inspection service for environmental control. The aim of the service is to provide technical support to decision-makers in environmental risk management. The proposed service uses IoT for the interaction between a mobile application, a Smart City platform, and an Unmanned Aircraft System (UAS). The mobile application allows the users to report risky situations, such as fire ignition, spills of pollutants in water, or illegal dumping; the user has only to specify the class of the event, while the geographical coordinates are automatically taken from device-integrated GPS. The message sent from the mobile application arrives to a Smart City platform, which shows all the received alerts on a 3D satellite map, to support decision-makers in choosing where a drone inspection is required. From the Smart City platform, the message is sent to the drone service operator; a CSV file defining the itinerary of the drone is automatically built and shown through the platform; the drone starts the mission providing a video, which is used by the decision-makers to understand whether the situation calls for immediate action. An experimental activity in an open field was carried out to validate the whole chain, from the alert to the drone mission, enriched by a Smart City platform to enable a decision-maker to better manage the situation.

1. Introduction

Environmental control of a vast rural territory is a challenging task, particularly during bad weather conditions, in rural and mountainous zones, or at sea. Too many alarms are received daily by all the law enforcement agencies, making it difficult to discern between what is really dangerous and what is a false alarm or a hoax. Modern technologies can help in this task in many ways. This paper presents an experimental activity based on the combination of a Smart City platform, used as a decision support system, with a mobile application and a drone inspection service to support decision-makers in understanding where an intervention is needed. Smart Cities make great use of the Internet of Things (IoT) and open data to optimize, for instance, the flow of energy, services, and data; in this context, even drones could be considered as sources of information. In the presented activity, a Smart City platform is the control room of the system, providing completeness to the service.

An Unmanned Aircraft System (UAS) is an aircraft flying without a pilot on board and including all the elements required to operate: the aircraft itself (technically the Unmanned Aerial Vehicle, in short, UAV), the ground control system, tools, such as one or more cameras, sensors, a GPS, and all the related software. Henceforth it will be referred to as a drone.

Drones have proved to be very effective in managing safety in public places thanks to some of their characteristics:

• they are fast;
• they can reach considerable heights;
• they are equipped with cameras that return high definition images;
• they are cheaper and more effective than other airborne monitoring systems (e.g., helicopters).

These features make drones a valuable ally in security management, from surveillance in contexts such as major events to logistical support for interventions in difficult-to-access sites that have suffered natural disasters.

Thanks to the different types of sensors with which drones can be equipped, it is easy to detect, for example:

• illegal dumping and abandonment of waste;
• spills of pollutants in water;
• fire ignition.

Drones are therefore greatly effective in cases where monitoring is particularly expensive and/or difficult: they can provide live images and videos to operational centers and promptly activate risk mitigation operations.

However, a drone has many limitations, the most important of which are summarized below. It has a short flight time, which is about 15 to 25 min depending on the type of drone. The batteries need to be frequently replaced in order to use the drones for a longer period. Their endurance strongly depends on the type of drone: multirotor drones are subject to greater energy consumption, while fixed-wing drones can last longer, having better performance at the expense of maneuverability.

On rainy or windy days, it is difficult to fly a drone as the battery as well as the electrical components are sensitive to moisture, and water droplets can cause malfunctions. Moreover, a strong wind could interfere with the flight and even cause a crash.

Another issue is related to safety. In fact, there have been cases of drones injuring the drone operator or other people, as described in many studies, such as, for instance, [1,2,3,4]. For this reason, not everyone takes well to seeing a drone flying over their surroundings, and for this reason flying over people not involved or not informed is generally forbidden or strictly restricted. Several countries regulate the use of drones in order to reduce the chances of accidents. In fact, regulations are a major reason for the slow development of drone usage; a significant limitation on drone operations is the requirement that they must be conducted within view of an operator. It is possible to request proper authorizations, but these are difficult to obtain.

Some drones, such as the one used for the presented activity, make it possible to minimize safety issues during flight; for example, in case of signal loss, the drone can automatically go back to the position recorded as “Return to Home”.

This paper presents a system based on drone inspection to optimize environmental surveillance that has been integrated with an existing platform, dedicated to the management of a Smart City. Starting from alerts sent by users through a mobile application, all the alerts are collected by the Smart City platform, and shown in a 3D satellite view to a decision-maker; the platform supports the decision-maker in choosing the alerts which require a drone inspection; he/she must then select the alert and click the ‘send drone’ button. This request is sent by the Smart City platform to the drone operating room where a mission is automatically planned through an algorithm properly developed for the presented activity, which outputs a CSV file defining the route and other parameters for the drone mission.

Related Work

The use of drones with an on-board camera or sensors for terrain surveillance purposes has become increasingly widespread. For instance, in manufacturing monitoring, an interesting application is described in [5], which is a paper about the use of drone inspection of power facilities and structures, dealing with common difficulties: flight stability, control accuracy, and safety; in particular, it provides a detailed state of the art review dating back to 2018, based on a selection of 27 papers where the use of drones allows safe outdoor and indoor location inspection of difficult/hazardous areas, mentioning also application of methodologies based on Artificial Intelligence, and 3D reconstructions; however, none of the described applications uses a Smart City platform to make the results of drone inspections available on a map to be consulted by a decision-maker. In a more recent work [6], a Geographic Information System is used coupled with drones for bridge inspections.

Drones can also help in early fire detection, as described in some recent works [7,8] discussed below, which detail technical improvements in inspections, but do not involve the use of a Smart City platform to manage the whole controlled area in a 3D map, integrating all the information coming from other available sources. One of these studies is [7], where forest fire surveillance is performed with live video streaming from a UAV through multiple LTE Networks, based on Scalable Video Coding (SVC); the authors tested the proposed forest fire surveillance system in Indonesia using a cheap set of lightweight devices for high-quality video streaming, that allows firefighters to easily control a wide area. Another experimental activity with a fire surveillance system based on drones is presented in [8]; in this case the system is based on magnetometer sensors, temperature sensors, and night vision cameras on drones to notify the controller.

An example of drone-assisted IoT is presented in [9], which describes the design and implementation of an online drone-based target detection system that performs online data processing and autonomous navigation through an adaptive motion planner. The paper also describes an experimental activity of the system at high altitude. Another example of drone-based surveillance IoT for industrial security application is presented in [10]; the work describes a system to allow authenticated users to login from any device, anywhere, and view real-time video or images sent from drones. A similar work on drone surveillance applied to smart cities is presented in [11], which focuses on the challenging issue related to energy conservation in UAVs in order to enhance flight time, and provides an interesting discussion of two case studies of the use of drones in the smart cities of Singapore and India, only mentioning the importance of a control room to manage the complete technical infrastructure.

A major survey of the potential techniques and applications of drone and IoT collaboration is presented in [12], based on an analysis of recent and ongoing research activities aimed at improving real-time applications for smart cities, including the generation of high-resolution 3D maps; in particular, the focus of the study is on the measurement of the smartness of the cities based on quality of life, healthcare, public safety, disaster management, environmental aspects (such as energy efficiency, air quality, traffic monitoring, etc.), and services; less importance is given to the visualization of all the information on a 3D map in order to support decision-making.

In the activity described in [13], drones are used for detection and identification of pollutants on the water surface, by pictures taken from a camera on the drone. Another paper on this topic is [14], which provides an interesting state-of-the-art survey of drones and sensor systems in relation to the petroleum industry, including functions such as offshore oil spill detection, oil leakage detection, pipeline monitoring, gas emission sensing, remote facility inspection, petroleum exploration (i.e., land surveying, geologic mapping, and petroleum exploration), and other related environmental monitoring.

Some significant EU-funded projects involving the use of drones in environmental control are described below.

5G!Drones [15] uses 5G to enhance drones and UAVs in many real-world applications, such as: traffic management; public safety; situation awareness (equipping drones with IoT devices, such as HD camera, gas sensors, humidity/temperature sensors, and devices for remote environmental monitoring interacting with a large network of sensors on the ground); connectivity during crowded events where cellular networks face an extremely high demand for communication capacity.

ResponDrone [16], coordinated by the German Aerospace Center (DLR www.dlr.de, accessed on 7 July 2022), describes a complete system to be used in emergency situations to provide first responders with a system that manages multiple synchronized missions of several drones. This project is particularly interesting because it involves a web-based cloud technology in order to gather, store, and provide collected data to all stakeholders.

InterAct [17] is related to Arctic stations, dedicating a Work Package to the use of drones, involving also issues of current legislation and UAV sensors suitable to operate in the Arctic environment. In this project drones are used for climate change observations, such as tracking glacier height, meteorological observations, etc.

Geo-SAFE [18], coordinated by the University of Greenwich, has the aim of developing tools to support the response phase with resource optimization; this project also involves drones to obtain high-resolution images.

AURORA-UAM [19] focuses on emergency-related applications, dealing also with the integration of drones into the smart cities context.

The interest in the many possible applications of drones, in various situations in which there is a need to face complex or prohibitive scenarios at low cost, is boosting innovation. Some best practices reported below highlight this aspect.

A significant example dates to 2011, in Fukushima, Japan [20], when robotic aerial vehicles were used to take videos and photos to get high-resolution information about the damage.

Another interesting example is Tararua, New Zealand [21], which is considered a Smart City because in 2018 it started using drones to capture images of flooding along rural roads in order to achieve faster solutions.

A more recent application is in Singapore [22], where in 2021 a testing activity with drones was started for inspections at six reservoirs. One of the aims of the service is to enhance response time to the great number of alerts received daily, similarly to the activity presented in this paper.

The aim of the presented activity is to integrate drone service management with a Smart City platform in order to provide a holistic view of what is happening in the area; the whole chain, starting from the alert sent through a mobile application and ending with the inspection by a drone, requested by a decision maker, has been tested in an open field.

2. Materials and Methods

In this paper, a drone inspection service for environmental control is presented. The system consists of three main parts (as shown in Figure 1) developed during the activity:

• a mobile application to report potentially dangerous events;
• a Smart City platform to collect and show all the alerts in a satellite 3D map, and send the request of an inspection by drone;
• a system to manage the inspections made by a drone, including the definition of a CSV file of the itinerary.

Each of these parts and their interactions are described in detail below.

2.1. The Mobile Application

The typical use case starts when an end user of the mobile application observes a risk for the environment and reports it to a control room. The present release of the application allows the reporting of the following environmental risks:

• fire ignition;
• spills of pollutants in water;
• illegal dumping;
• other (in this case a further textual message is required).

Figure 2 shows the mobile application.

Each alert message is in JSON format and also includes the geographical coordinates of the mobile phone, along with the timestamp, and two unique identification strings: one related to the device (“id”), and the other to the alert itself. An example of alert type ‘waste’, in JSON format, is as follows:

{

“id”: “ae0af7284b27fd6c”,

“ts”: “17 February 2022 16:57:17”,

“id_event”: “ae0af7284b27fd6c17-Feb-2022 16:57:17”,

“lat”: 39.2686312,

“lng”: 9.1757257,

“type”: “Waste”,

“msg”: ““

}

2.2. The Platform to Support the Decision Maker

The Smart City platform used in the presented activity was developed by CRS4 in the context of the city of Cagliari to manage data acquisition, persistence, processing, and presentation.

An ambitious infrastructure that can connect a vast territory and acquire large quantities of data in many different contexts necessarily requires the use of a modular approach that deals with individual aspects, from data management to the presentation of the processed information. In order to cope with the huge amount of data arriving from the territory, the key points in the development of the platform are flexibility, stability, and scalability. In this context, this platform covers a city but can easily be scaled to a larger area. These requirements are achieved by following the strategies of openness and reusability of the software in order to avoid technological lock-in, and thus extend its use to multiple contexts. The purpose of the platform is twofold: in the background it deals with managing the data that can be conveyed to the platform; at the front end it collects and presents to the coordinator all the available information for decision-making support.

2.2.1. Benchmarking of the Smart City Platform

The data used for the loading tests refer to a real-world installation for a project on the Smart City of Cagliari. In a single instance, without high availability or load balancer systems, the incoming traffic was about 500 JSON blocks per second, each one of about 10 KB; the traffic was acquired from the back end through a RESTFul API, which is the slowest among those that the platform makes available, deliberately chosen to get a perspective on the worst case scenario. The platform is modular, so it is horizontally scalable throughout its stack.

As for the specific use case related to this document, the data traffic load required is almost inconsiderable and therefore has little impact on an infrastructure already dedicated to Smart Cities. In fact, the order of magnitude of a single message is about 5 kB, and its impact is irrelevant because the actual flow requires 14 messages divided into: new event incoming, flight request, 11 Points of Interest created to monitor the emergency scene, end mission.

2.2.2. Technical Details of Data Ingestion and Front end

In the specific scenario linked to drones, the platform’s back end receives the stream of alerts sent by means of the mobile application, through a rule chain implemented with the open source IoT platform ThingsBoard [23] and shown in Figure 3. The approach used was to manage information and communications as if they were data generated by IoT sensors, and design the flow with a dedicated IoT platform to exploit their versatility and robustness.

The platform’s front end is a powerful tool to support the decision-maker in the selection of the alerts which require a drone inspection (see Figure 4).

In particular, the front end of the platform is made of the following:

• a main menu, on the top, center;
• two sidebars, one on the left and the other on the right, showing analytical and statistical information about all the received requests;
• a navigable 3D satellite map, in the center.

Figure 4 shows different statistics regarding the use case of drones. The data shown in the sidebars refer to a time span that embraces the previous 30 days; however it is possible to modify the value of the time span parameter. In the left bar it is possible to find information on the types of reports and their cardinality; the status ID of the reports is also displayed making it possible to understand how many of them are new and how many are still pending. The bar on the right shows a pie chart based on the cardinality of the report types and a table with the log of the reports and their status. On the central map, different geo-referenced pins are shown to indicate the points of the reports, the type of the icon matching with the type of report and the color with the progress of the request.

Whenever a message is sent through the mobile application, it is received by the Smart City platform, which works as control room, showing all the alerts in the 3D satellite view. A proper algorithm for decision-making has not been developed yet, but from the overall view, shown in Figure 4, the decision-maker is supported in the careful analysis of all the messages received in order to single out the most dangerous ones that call for an inspection by drone. In order to do this, he/she simply has to select the alert shown in the view, which opens the related widget (also called, ‘infobox’); the widgets of all the new alerts have an extra button to activate the request for a drone flight. On the map, the alerts are shown in different colors, according to their status. Four progress statuses are possible:

• new requests, for which it is possible to select and eventually activate the alert mechanism linked to the flight of the drone;
• mission requested (when the button has been clicked by the decision-maker);
• ongoing missions, showing the route of the drone;
• completed missions, after the drone completes its mission.

Figure 5 shows an example of a selected notification on the 3D satellite map, with the related widget where the button ‘send drone’ allows the decision-maker to ask for the inspection by drone.

2.3. Programming a Mission

For the on field experimental activity, a DJI Inspire 2 drone was used, which can fly for 25 min and is able to take the required images through the DJI Zenmuse x5s optics.

According to the manufacturer, this is the first filmmaking drone in the world to integrate an HD video transmission system, a 360° rotating gimbal, and a 4K camera. This model can detect obstacles up to 30 m ahead and 5 m above. When the drone operating room receives a request for a new inspection by drone, the mission is planned and a CSV file with the route is automatically built, starting from the coordinates of the place of the original alert.

The algorithm to build the optimized route was designed and developed for the presented application, and should be explained in more detail. In fact, it can elaborate a set of itineraries for a fleet of drones, to inspect an area which includes several points of interest. In this scenario, the drones are considered as a multi-agent system, and each drone as a resource to be dynamically allocated to a sequence of tasks related to each of the points of interest in the area. To do this, the area is split in sub-areas, called ‘equitable partitions’, each one including only one point of interest, following the Voronoi criteria [24]. Each drone is assigned a mission made of a sequence of tasks minimizing two parameters: the time from the request to the mission and the sum of the distances traveled by all the drones involved. Each itinerary includes a circular flight around a point of interest, where the radium of the circle is calculated as half the minimum diameter of that Voronoi cell. The developed algorithm returns in output the set of itineraries (one for each drone) defined in a CSV file described in detail later in this paragraph. The presented experimental test involved only one drone, and its itinerary was built with the described criteria, around the only point of interest considered. But a further development will cover a greater area, with more point of interests and an inspection by a fleet of drones.

The generated CSV file must be downloaded and imported on the ‘‘Litchi’’ platform [25], a third-party software that allows mission planning and which integrates with most of the drones on the market. In the CSV file, the coordinates of the points defining the drone’s mission are listed; an example of a CSV file is shown in Figure 6.

In the file, each row represents a waypoint position and a set of actions to perform over it.

Apart from the obvious meaning of latitude, longitude, and altitude, a brief description of the nontrivial parameters can be given as follows:

• heading(deg) controls the starting rotation of the aircraft;
• curvesize(m) controls the curve angle of the UAV passing way points. By default, it is set to (=0.0);
• rotationdir camera control parameter sets the UAV basic turn direction to right (0) or left (1);
• gimbalmode camera control parameter; 0 deactivates the gimbal control; 1 activates the gimbal for focusing POIs; 2 activates the gimbal for focus and interpolates a field of view in an angle of gimbalpitchangle
• gimbalpitchangle vertical angle of camera +30 deg.−90 deg;
• actiontypeX (with X from 0 to 5) specifies the type of action of the camera (defined later);
• actionparamX (with X from 0 to 5) provides corresponding parameter for the individual actiontype.

The actiontype parameter could be:

0: stay hovering;

1: take photo;

2: start recording;

3: stop recording;

4: rotate aircraft;

5: tilt camera.

After loading the CSV file on the platform, the mission, the point of interest, and the related waypoints configured around the target, will be displayed on a georeferenced map of Litchi and then made available on any device connected to the platform, as shown in Figure 7.

The flight plan of the mission is therefore available on the device that is connected to the drone controller, which allows the management of all the functions of the aircraft.

Before starting the aerial monitoring of the specified point of interest, it is possible to modify or add waypoints around the target and/or complete the configuration of the specific mission by varying some parameters.

From this moment on, the mission and all the functions of the drone can also be managed directly via the controller (see Figure 8).

The drone sends the real-time video via YouTube, through the mobile phone connected to the drone controller. After the mission’s end, its status changes from ‘ongoing’ to ‘completed’ and the video shot from the drone becomes available to the decision-maker on the map of the smart city (from the widget related to the mission).

3. Results

The on-field testing activity covered the whole presented solution, from the alert to the end of the drone mission, managed through the Smart City platform. Unfortunately, a large no-fly zone, due to restrictions on drones over military and urban areas, led to the choice of a rural place about 15 km from Cagliari. There, three people used a drone simulating an operational base (see Figure 9), another person went nearby with a mobile phone to send alerts, while a person from the office pretended to be the decision-maker and selected one of the alerts to request an inspection by drone.

It must be specified that in a real context the number of people required is lower than the number of people that were involved in the on-field simulation; in fact, only one person is sufficient to make the report, who is supposed to be a common citizen willing to communicate a risk situation. To manage the smart city platform, the decision-maker can lead the environmental inspection along with a flight planner, and a pilot (but they can be the same person). It must be said that in some countries (such as in Italy), due to the already mentioned safety issues, the drone operator must guide the drone without losing sight of it, but drones are already technically able to replace humans in environmental inspections.

The experimental chain was completed in less than half an hour. An issue related to the precision of the GPS coordinates, arising because the mobile phone was not connected to Wi-Fi, requires some more consideration.

The accuracy of the GPS system of smartphones depends on several factors. First of all, when a GPS is activated or if the GPS has been inactive in the background for too long, the GPS needs to fetch data from the satellites that describe the position and timing of all of the satellites in the system. This can take up to 4 min to be corrected, and during this time the GPS is not reliable (this is the so called “cold start”). Usually, to cope with GPS inaccuracy the Android subsystem uses several methods; one of these is the Wi-Fi connection that can provide a first georeferentiation, which is then refined with satellite data; the absence of Wi-Fi slows down the “cold start” process.

Another factor is related to the Android energy management and its customizations: the more aggressive it is, the more the sensor will tend to be turned off to improve battery performance.

The mobile app presented in this project is POC and so prone to several enhancements; to fix several of these issues it is possible to enforce the application development to force the app to be always active, and thus with the GPS coordinates updated.

Figure 10 shows the drone during the inspection.

Figure 11 shows two alerts and the points for the inspection.

A detail of the points of the drone mission are shown in Figure 12.

As already mentioned, at present, the drone can share a streaming video of the area on YouTube through the mobile phone connected to the drone controller. Once the mission ends, the video is also made available on the platform, as shown in Figure 13. Based on the video, the decision-maker can determine if an inspection by a team is necessary, and this closes the chain.

It can be said that there are two kinds of limitations common to any drone-based service: technical and regulatory. The technical limitations can be overcome by innovative evolution, while the regulatory ones can be bypassed with a special license issued by the authorities to operate in emergency situations. The on-field test was sufficient to better understand the advantages and disadvantages of the proposed solution, which are the challenges encountered by any similar system, and which are summarized below.

The main advantage of the presented solution is that in the operations center, the decision-maker can manage many kinds of reports coming from all the vertical applications implemented on the Smart City platform, not only the ones related to the presented use case. In fact, the information generated by this specific application is only one of many, built on data collected from sensors and other sources, which together compound the control room of a Smart City, providing the decision-maker with a holistic view of what is happening in the area.

An important advantage related to the use of a drone is that it is easy to fly at low and ultra-low altitudes, compared to manned aircraft; for this reason, thanks to the on-board video camera, it was possible to get high-resolution images and play the video.

Another advantage is related to the possibility for the drone to reach difficult-to-access sites, such as the lake nearby (clearly visible in Figure 11), if necessary.

The main disadvantage is related to the battery: the drone needed to be recharged for a couple of hours before the mission, and its autonomy would not have lasted more than 25 min (as already mentioned in Section 2.3); this also implies that the drone should be kept fully charged in order to be always ready for unexpected missions.

Another disadvantage is related to the Beyond Visual Line of Sight (BVLOS) restrictions, because in some real-world situations it would be useful to have the possibility to guide the drone for an inspection in impervious zones.

Lack of connection is also a critical issue of the proposed solution, significantly limiting the operational territory, which is already reduced by other types of no-flight zones, such as urban areas, military zones, etc.

Finally, harsh weather conditions can significantly complicate if not completely obstruct operations.

4. Discussion

An experimental activity testing the use of drones in environmental control was described. The activity is made up of three main components: a mobile application to send on-field alerts, a Smart City platform to collect alerts and show them on a 3D satellite map to support a decision-maker in the choice of the alerts requiring an inspection by drone, and the operating base to manage the drone in its mission. A complete use case was tested in the field, starting from an experimental alert, and ending with a flight by drone. The application could have great potential to support a decision-maker in centralizing and optimizing a control room which receives information about dangerous situations which can be risky for the environment.

In the experiment, a multirotor drone was selected, but a fixed-wing drone, which can cover longer distances thanks to its greater autonomy, could be particularly suitable in scenarios of distant wildfires that are difficult to reach quickly.

The present version of the application could be improved in many ways. To begin with, the mobile device could send more information about what is going on. Object detection algorithms could be applied to analyze the video shot by the video camera on the drone. A further improvement would be achieved by adding sensors to allow the application of remote sensing algorithms, and so expand the potential use cases. It is worth noting that without the limit of visual flight, many operational bases could be located on a territory, where all the actions to be taken can be coordinated directly from the same central station; depending on weather conditions and other parameters, the decision-maker can choose which missions call for an inspection by drone, avoiding sending a human operator in the field.