Automated Fire Extinguishing System Using a Deep Learning Based Framework

Abstract

Fire accidents occur in every part of the world and cause a large number of casualties because of the risks involved in manually extinguishing the fire. In most cases, humans cannot detect and extinguish fire manually. Fire extinguishing robots with sophisticated functionalities are being rapidly developed nowadays, and most of these systems use fire sensors and detectors. However, they lack mechanisms for the early detection of fire, in case of casualties. To detect and prevent such fire accidents in its early stages, a deep learning-based automatic fire extinguishing mechanism was introduced in this work. Fire detection and human presence in fire locations were carried out using convolution neural networks (CNNs), configured to operate on the chosen fire dataset. For fire detection, a custom learning network was formed by tweaking the layer parameters of CNN for detecting fires with better accuracy. For human detection, Alex-net architecture was employed to detect the presence of humans in the fire accident zone. We experimented and analyzed the proposed model using various optimizers, activation functions, and learning rates, based on the accuracy and loss metrics generated for the chosen fire dataset. The best combination of neural network parameters was evaluated from the model configured with an Adam optimizer and softmax activation, driven with a learning rate of 0.001, providing better accuracy for the learning model. Finally, the experiments were tested using a mobile robotic system by configuring them in automatic and wireless control modes. In automatic mode, the robot was made to patrol around and monitor for fire casualties and fire accidents. It automatically extinguished the fire using the learned features triggered through the developed model.

1. Introduction

Fire accidents are drastically increasing in all parts of the world. This impacts people in large numbers and also affects the environment. Therefore, there are inevitable risks involved in industries and other organizations due to fire. With reference to the statistics reported in [1], fire risk dropped to the 10th rank in 2019 compared with the previous year, where it was ranked 3rd. In addition to these statistics, fire risks are comparatively high in private industries. Although there is a drop in ranking compared with other risks, a risk of fire is inevitable. The oxygen content in the air, the heat emitted from surfaces, and sources of fuel are the three main ingredients of most fires. As heat can come from any hot surfaces, electrical equipment, static electricity, as well as open flames, appropriate countermeasures are required to resist them. Fuel can be any flammable liquid, gas, wood, paper, dust, and certain metals. The oxygen requirements are readily available in the air [2]. A fire only needs 16% of oxygen in the atmosphere to ignite, whereas humans need about 21% oxygen in the air in order to breath. Once a fire has started, it can be stopped if any one of these three elements is eliminated [3]. This brings up probable solutions for fire prevention and shows the importance of keeping these three ingredients under control [4]. Hazardous fire accidents are happening in textile industries, chemical industries, forests, and even in subways. Subway fire accidents also occur all around the world and the number of accidents keep increasing over the year [5]. All fire accidents are severe life hazards and cause long-term effects in some cases. Extinguishing a fire after it spreads over a large area is a very tedious task and consumes too much time to extinguish the fire completely. Early fire detection is the only feasible solution to address such issues. Among the numerous fire detection techniques, most fire detection systems are implemented based on sensors to detect the fire [6]. Sensor-based fire detection systems require close proximity to detect the fire and cannot be used over a large area. Deep learning (DL) is excessively deployed in most of the domains [7], including natural language processing and image processing [8,9]. DL techniques are characterized by their capability to learn from data, creating a mathematical model with the provided data with identity recognition [10] as one of its crucial applications. DL models exhibit high accuracy of detection in all applications. Therefore, we used DL techniques to detect fire by building a convolutional neural network (CNN) model [6].

A DL approach, along with spatial and temporal features, were considered for fire and smoke detection through surveillance cameras in [11]. Here, region-based CNN was used for extracting local and global features to reduce false positives in fire and smoke detection. In another similar work [12], DL models were used in video-based fire detection using region-based CNN and LSTM models. Their method interpreted the dynamic behavior of fire and smoke through the temporal changes in their features. The experiments were well-performed over long- and short-range videos, thereby significantly reducing false alarms. Later on, Govil et al. [13] used remote cameras to monitor and detect wildfires using a DL approach. In addition, they used cloud-based task management in handling a stream of images from ground-based and terrestrial cameras. The complexity of the DL is described in detail in [14,15].

Salameh et al. [16] surveyed the usage of IoT-based systems for fire and gas leakage detection applications, particularly focusing on wireless sensor networks. In [17], the authors experimented with fire smoke detection in aircraft cargo for fire and non-fire aerosol identification. Further, they distinguished between black and white smoke based on asymmetry ratios. Oil storage tank monitoring with an IoT-based framework, presented in [18], focused on the detection of smoke and fire, as well as monitored temperature. Further, it also assisted with real-time surveillance to alert industrial authorities to prevent accidents.

1.1. Research Gap and Motivation

Although many studies have been conducted on the use of DL techniques for disaster management [19], studies on fire detection using robotic systems with onboard sensors are rather scarce. Few studies were found to address fire disasters through a genetic algorithm approach [20]. In addition, the growing interest in DL techniques and their applications for fire detection needs to be explored by addressing their potential and challenges. To better estimate the prediction accuracy for fire accidents, we need to explore the inherent capabilities of DL frameworks and their properties and impact on the accidents [21].

In this paper, the investigations were carried out using a fire dataset collected from the Kaggle benchmark dataset [22], named outdoor-fire and non-fire images for computer vision tasks (Version 1) to analyze and predict the occurrence of fires in a specific environment. The collected dataset was composed of 3638 images. Moreover, deep neural network implementation was performed by introducing various activation functions, optimizers, and learning rates that were applied to the collected dataset [23].

1.2. Objectives and Goal of the Study

Based on an in-depth analysis of the literature, it was revealed that many studies have used DL techniques for fire-extinguishing applications. However, none of these studies explored a fire accident scenario with automated extinguishing nor employed robotic systems for maneuvering across the environment for the detection and extinguishing of the fire. This work provides the following set of outcomes, with primary focus on the DL technique for fire detection tasks and initiating extinguishing operations:

• Fire detection is carried out using a CNN model configured to operate on the chosen fire dataset.
• A DL-based framework is developed and trained to automatically classify the fire scenarios.
• Human detection is carried out using Alex-net with a pre-trained set of data from the fire datasets with a human involved during fire accidents.
• A mobile robotic system with onboard sensors is employed for detecting the fire.

The remaining sections of this article are as follows: Section 2 provides a description of the proposed CNN for fire and human detection. Section 3 presents and compares the results obtained using different optimizers, activation layers, and learning rates. Finally, our conclusions are provided in Section 4, along with future research directions.

2. Proposed Framework

This section is dedicated to presenting the proposed framework for the automated fire extinguishing system and further discussions on the overall functional modules of the system.

2.1. Learning-Based Automated Fire Extinguishing System

The overall functional block diagram of the proposed system is shown in Figure 1. Initially, the system is in patrol mode around the suspicious fire-prone area. The data acquisition unit collects the data (image/video sequences) to be assessed for fire and humans in the environment. The processing unit is configured to access the neural network via a cloud service. The corresponding neural network model predicts the output scenario and sends back the predicted output to the processing unit, which in turn operates corresponding actuation and alert systems [24]. The main focus of this work was to analyze a fire detection model with various metrics. Alex-net architecture has shown great results for human detection [25]. Therefore, human detection was performed using the predefined Alex-net model, utilizing a transfer learning framework, and further analysis was performed on the proposed neural network model for fire detection.

2.2. Fire Detection Model

Deep CNN methods are extensively applied in the fire detection and classification process. A large number of CNN models have been built with their own advantages and limitations over the last decade, during which most research was focused on fundamental DL models.

Our proposed framework for fire detection using the CNN model was driven by a pre-processed sequence of images from the fire dataset and tested using a portion of the dataset to evaluate its performance. Further, during the testing phase, the fire images acquired from the camera mounted on the robotic system were fed to the CNN model to decide the actuation subsystem for extinguishing the fire [26].

2.2.1. Description of the Dataset

The dataset of outdoor-fire and non-fire images for computer vision tasks (Version1) was created by a team of experts during the NASA Space Apps Challenge in 2018 [22], with the goal of using them in a benchmark dataset to develop a model that could recognize the fires in these scenarios. The chosen dataset with 999 images (755 fire and 244 non-fire) was augmented with a few more images, so that the final dataset comprised 3638 images (2528 fire, 737 non-fire, and 373 smoke) collected under different environmental conditions, such as heavy smoke. Further, the non-fire images consisted of natural images with trees, rivers, people, animals, roads, and waterfalls. Few studies have used this dataset among the research community [27,28] for fire detection applications. Some of the sample images in the training set are shown in Figure 2.

In the pre-processing stage, in order to classify the fire and non-fire images in the dataset, a histogram of the images was taken, which was meant to eliminate noise and other residual features. Further, based on the intensity of the background content in the images, redundant background features were eliminated. Subsequently, image enhancement was performed on the background-separated images. As a final stage input to the CNN model, object representation tasks were performed to identify the fire pattern in the images.

2.2.2. Proposed CNN Architecture

The CNN architecture used here is similar to the architecture used in [29], with important tweaks in the layers and filter size to suit the input pre-processed fire dataset, and is shown in Figure 3. This developed model included three convolution layers, two pooling layers, and two dense layers. The input color image (RGB), shaped to $150\times 150\times 3$ in size, was applied to the first convolution layer, which had 32 filters with a kernel size of $3\times 3$. The size of the kernel depends on the stride and slide, considering the size of the input image as $W\times W\times D$. Further, in a spatial dimension of F kernels, P padding, and S strides, the output volume could be computed as shown in Equation (1).

$$W_o=\frac{W-F+2P}{S}+1$$

(1)

This would produce an output of size $W_o$×$W_o$×$D_o$. The resulting ($148\times 148$) feature maps were passed through the MaxPooling layer with strides (2,2). For the MaxPooling layer, strides S and kernel size F could be specified based on the depth of the number of activation maps D. Equation (2) specifies the output dimension of the image, after passing through the MaxPool layer.

$$P_o=\left(\frac{W_o-F}{S}+1\right)\times D$$

(2)

The resultant 74 × 74 feature maps were passed through the second convolutional layer, which had 64 filters with the same 3 × 3 kernel size. Further, this was applied to a MaxPooling layer of the same specifications used in the first MaxPooling layer. The resultant 36 × 36 feature maps were passed on to the last convolutional layer. Finally, for the last convolutional layer, fully connected layers were included for performing the classification tasks. In the fully connected layer, all the inputs had separable weights for each of the output units. The final output layer consisted of two neurons engaged in classifying the fire and non-fire images.

3. Analysis of Results and Discussion

This section discusses the experimental results of our proposed CNN model with various metrics. The metrics used to analyze the model included different optimizers, activation functions, and learning rates, which are elaborated on in the subsequent sections. All the metrics were evaluated by experimenting with the model for 100 epochs. The rest of this section elaborates on the various metrics used in the proposed model.

3.1. Optimizer Analysis

Optimizers are algorithms used to change the attributes of neural network such as weights and learning rates to reduce losses. Various optimizers are available for neural networks that could be used, based on the data and demands of the users [31]. The optimizers used for the analysis of the proposed network were Adam, Adamax, Nadam, RMSprop, and Sgd. Adam is an optimizer for CNN, developed to overcome the disadvantages of the stochastic gradient descent [32], which was first published in 2014 and presented at the machine learning conference ICLR in 2015.

The performance evaluation of the developed model was analyzed with an appropriate choice of optimizer. Based on the choice of each optimizer, the corresponding observations in performance were discussed for each of the models, as follows:

Adam optimizer: With the choice of the Adam optimizer for analysis, Figure 4a shows a plot with the number of epochs vs accuracy (%) observed for 100 epochs. It is observed that an accuracy of approximately 90% was achieved at the 13th epoch in the training phase. However, up to 85 epochs, there were a few distortions observed, which considerably affects the accuracy. After the 85th epoch, the training accuracy was saturated and only less deviation from training to validation accuracy was observed. Figure 4b shows the number of epochs vs. loss (%) graph. Similar to the accuracy plot, minimum loss was achieved in a fewer number of epochs. However, there were more distortions up to the 85th epoch. Further, the loss was also saturated and deviations from training loss to validation loss were minimized.

Adamax: Adamax is one of the variants of Adam based on the infinity norm feature [33]. The crucial advantage of Adamax is that it is less sensitive to hyperparameters such as learning rate. Figure 5a shows the observed accuracy with a plot showing the number of epochs vs. accuracy (%) for 100 epochs of the Adamax optimizer. Figure 5b shows the observed loss with a plot of the number of epochs vs. loss (%) for 100 epochs of the Adamax optimizer. While using the Adamax optimizer, the accuracy and loss were more or less saturated in fewer epochs with negligible deviations. The average percentage of accuracy and loss was also similar to Adam. However, the Adam optimizer was chosen for further analysis because the change in learning rate may not have produced notable changes in the results, as it was less sensitive to its hyperparameters [34].

Nadam: This is an acronym for Nesterov and Adam and was developed by modifying Adam’s momentum concept by incorporating the Nesterov momentum, which is the significant advantage of NAG (Nesterov’s accelerated gradient) [35]. Nadam works better for complex models; however, its prediction time is larger than the time taken for prediction using Adam [36]. Figure 6a,b shows the epochs vs. accuracy and loss graph for 100 epochs of the Nadam optimizer. In our case, the accuracy and loss were not saturated until 100 epochs. The deviations between the training and validation accuracy were permissible; however, loss variation was high.

RMSprop: RMSprop is an unpublished, adaptive learning rate algorithm; however, [37] shows its popularity until 2017 and it remains popular now. It uses the magnitude of recent gradients to normalize the gradients. Figure 7a shows a plot of the number of epochs vs. accuracy (%) based on the observation of 100 epochs of the RMSprop optimizer. Figure 7b shows a plot of the number of epochs vs. accuracy (%) observed for 100 epochs of the RMSprop optimizer. Unfortunately, both accuracy and loss were not saturated up to 100 epochs.

SGD: Stochastic gradient descent (SGD) is a simple, classical method that calculates the gradient of each weight on the network [38]. This method is very slow compared with other optimizers; nevertheless, it is good for shallow networks. Figure 8a shows a plot with the number of epochs vs. accuracy (%) observed for the SGD optimizer. Figure 8b shows the performance observations for the losses incurred in the SGD optimizer. The maximum range of accuracy was achieved gradually (epoch 50) compared with the other optimizers used for the analysis. Minimum loss was also obtained compared with another optimizers. SGD produced better accuracy and loss characteristics. However, due to its slow convergence rate, SGD is not commonly used nowadays as numerous advanced optimization algorithms have evolved.

We further compared the average accuracy and loss values of each optimizer by taking the average of values generated for all epochs.

Table 1. Comparison with various optimizers.

Optimizer	Accuracy (%)	Validation Accuracy (%)	Loss (%)	Validation Loss (%)
Adam	89.6463	89.3862	11.0794	12.8375
Adamax	89.7150	89.3163	10.8713	12.8152
Nadam	89.5313	89.2661	11.4997	13.3637
RMSprop	89.5390	89.2379	12.2243	17.0270
SGD	88.1809	89.0939	15.2573	12.7738

shows the performance of the developed model with various optimizers, tabulated with the average accuracy, validation accuracy, and validation losses. It was observed that the losses in Adamax were significantly lower than in the other variants.

Although Adam and Adamax produced nearly the same accuracy (as shown in Table 1), the Adam Optimizer was used for further analysis due to having the best accuracy among the different optimizers, considering their validation accuracy and time.

3.2. Activation Layer Analysis

The activation function is a node that is placed at the end or in between the neural networks, which decides whether the neuron will fire or not. Different activation functions can be used for different purposes [39]. Different activation functions produce different results based on their operations. Sigmoid, Softmax, and Softplus activation functions were used here to analyze our neural network. The Sigmoid function is mathematically expressed, as shown in Equation (3):

$$S\left(x\right)=\frac{1}{1+e^{-x}}$$

(3)

where $S\left(x\right)$ is the sigmoid function and e is Euler’s number.

The sigmoid function exists between 0 and 1. Therefore, it is especially used for models where the output prediction falls under ranges of probabilities. Figure 9a,b shows the number of epochs vs. accuracy (%) and loss graph for 100 epochs of the Adam optimizer using the Sigmoid activation function. From the graph, high accuracy and minimum loss were achieved in a fewer number of epochs, and accuracy and loss were saturated.

Softmax is used as the activation function for multi-class classification problems where class membership is required for more than two class labels. Although it can be used as an activation function in the hidden layers of networks, it is less commonly used. Here, the Softmax layer was used at the end of our neural network model. Figure 10a,b shows the number of epochs vs. accuracy (%) and loss graph for 100 epochs of the Adam optimizer using the Softmax activation function.

The Softplus function is mathematically expressed as shown in Equation (4),

$$f\left(x\right)=ln(1+e^x)$$

(4)

It is an alternative to traditional activation functions and is less commonly used. Figure 11 shows the number of epochs vs. accuracy (%) and loss graph for 100 epochs of the Adam optimizer using the Softplus activation function.

The average accuracy and loss were calculated and tabulated below for the previously discussed activation functions.

Table 2. Comparison chart for various activation layers of Adam optimizer.

Activation Function	Accuracy (%)	Validation Accuracy (%)	Loss (%)	Validation Loss (%)
Sigmoid	89.6257	89.2181	11.2117	13.8817
Softmax	89.6462	89.3862	11.0794	12.8375
Softplus	88.3582	88.4686	18.0761	19.6317

shows the accuracy, validation accuracy, loss, and validation loss for the various activation layers used.

As shown in Table 2, Softmax produced better accuracy than the other activation layers. It is also the most commonly used activation function at the end of neural networks. The Adam optimizer and Softmax activation function were used for further analysis.

3.3. Learning Rate Analysis

The learning rate or step size is the number of weights that are updated during the training phase. Usually, it lies in a range of 0 to 1. This hyperparameter controls how quickly the model learns a problem. A smaller learning rate requires more epochs, whereas a larger learning rate requires fewer epochs. A larger learning rate may cause the model to converge too quickly, whereas a very small learning rate may also have negative effects [40]. Therefore, learning rate is the most important hyperparameter and should be carefully selected.

Table 3. Comparison with different learning rates of Adam optimizer and Softmax activation layer.

Learning Rate	Accuracy (%)	Validation Accuracy (%)	Loss (%)	Validation Loss (%)
0.01	88.3164	88.7505	14.6927	14.9022
0.001	89.6462	89.3862	11.0794	12.8375
0.0001	89.5501	89.3503	11.4771	11.9575

shows the comparative analysis on the performance of the model considering different learning rates of Adam optimizer and Softmax activation.

The learning rates of $0.01$, $0.001$ (default), and $0.0001$ were used for analysis.

Table 4. Comparison with different learning rates of Adam optimizer and Softmax activation layer.

Learning Rate	Accuracy (%)	Validation Accuracy (%)	Loss (%)	Validation Loss (%)	Precision	Recall	F1-Score
0.01	88.3164	88.7505	14.6927	14.9022	0.89	0.88	0.87
0.001	89.6462	89.3862	11.0794	12.8375	0.91	0.90	0.85
0.0001	89.5501	89.3503	11.4771	11.9575	0.90	0.89	0.86

shows the accuracy, validation accuracy, loss, and validation loss for the various learning rates of the model, along with the precision, recall, and F1-score statistics observed with the choice of the Adam optimizer and Softmax activation layer for the network. Figure 12 shows the confusion matrix representing the comparison among the predicted fire, non-fire, and smoke images in the dataset.

As shown in Table 4, the learning rate of $0.001$ produced better results than the others. Usually, a lower learning rate produces better results, but needs a large number of epochs. The best learning rate is based on the number of epochs used for training. In our case (for 100 epochs), the default learning rate of $0.001$ for Adam produced the highest accuracy. Generally, it is obvious that a large dataset could have significant improvements in classification accuracy with reduced error. However, with the dataset chosen, with a small set of data on fire, non-fire, and smoke images, it was evident from the results that high error rates were observed. However, if the model involved larger standard datasets on fire and non-fire images, it could drastically enhance the classification accuracy and thereby reduce the higher error rates.

4. Conclusions and Future Work

Deep learning frameworks have been under the spotlight in recent years because of their flexibility and suitability for classification tasks. DL frameworks learn useful representations of features directly from the input image, text, audio, or video data using neural networks. In this article, fire detection was carried out using the proposed CNN architecture, which consisted of six layers. We experimented and analyzed the neural network using various optimizers, activation functions, and learning rates. From our results, the best optimizer, activation function, and learning rate for this neural network were observed, based on accuracy and loss metrics generated for both training and validation. Further, the robotic system was configured for navigation in autonomous and manual mode toward the fire zone and for actuation of the extinguishing unit based on the learned features triggered through the developed model. It was evident from the experimental results that the Adam optimizer with Softmax activation, driven with a learning rate of $0.001$, provided enhanced accuracy compared with the other parameters tested. In addition, the chosen dataset investigated with the aforementioned configuration provided a better accuracy of 89.64% compared with the other techniques. Even though the system imposes limitations and concerns regarding its applicability in real-world scenarios due to its high error rates, consideration of larger datasets will enhance the accuracy of the system.

In future studies, we plan to devise a robust DL architecture for combined fire and human detection, resulting in a single model that can detect both humans and fire for saving the lives of humans while also extinguishing fires.