Deep Learning- and IoT-Based Framework for Rock-Fall Early Warning

Abstract

In recent years, several strategies have been introduced to enhance early warning systems and lower the risk of rock-falls. In this regard, this paper introduces a deep learning- and IoT-based framework for rock-fall early warning, devoted to reducing rock-fall risk with high accuracy. In this framework, the prediction accuracy was augmented by eliminating the uncertainties and confusion plaguing the prediction model. In order to achieve augmented prediction accuracy, this framework fused prediction model-based deep learning with a detection model-based Internet of Things. This study utilized parameters, namely, overall prediction performance measures based on a confusion matrix, to assess the performance of the framework in addition to its ability to reduce the risk. The result indicates an increase in prediction model accuracy from 86% to 98.8%. In addition, the framework reduced the risk probability from 1.51 × 10⁻³ to 8.57 × 10⁻⁹. Our findings demonstrate the high prediction accuracy of the framework, which also offers a reliable decision-making mechanism for providing early warning and reducing the potential hazards of rock falls.

1. Introduction

Rock fall is a complex natural phenomenon that threatens humans and infrastructure in many mountain regions of the world. Because rock-fall events are random, finding reliable mechanisms for monitoring, predicting, and managing the geological risk due to rock fall is still a challenging task. In recent years, many approaches have been used to model and assess rock-fall hazards. For example, the study [1] developed a hazard-assessment model based on the frequency of rock falls, rock structure, and bounce height. By using a dynamic computational technique, the suggested model evaluated the risk of rock fall and the quantification of uncertainties. Quantitative models were also created to evaluate and control the danger of rock falls [2,3].

There are other techniques to detect falling rocks, such as seismic signal detection. The literature reports many methods to track seismic waves produced by falling rocks, as geophysical sensors were developed to track the seismic signals caused by falling boulders and determine how the rock’s effect on the surface was estimated [4]. A micro-seismic approach was introduced to identify rock-fall occurrences in another investigation [5,6,7]. Although these methods are excellent at detecting rock-fall occurrences, doing so requires a large number of seismic sensors. Through the use of micro-electromechanical and micro-seismic networks, new methods to circumvent micro-seismic limits have been developed. Camera-based monitoring techniques have recently been used to monitor and track fallen rocks in real time. More recently, the development of photogrammetric monitoring systems for rock-fall monitoring purposes has received increased attention in the scientific literature [8,9,10,11,12]. A test version of an artificial intelligence camera was used to track and monitor falling rocks in real-time [13]. The camera has outperformed many technologies, even micro-seismic networks, regarding its capability to monitor many rocks simultaneously. Seismic and camera-based monitoring systems are used to identify falling rocks at the time of impact; nevertheless, they must be sufficiently effective to alert vehicles to the risk of rock fall before it happens. Unfortunately, this technology responds to events after causing severe damage to the road and pedestrians. In order to move around the limitations of monitoring approaches, rock-fall occurrences must be predicted. Recently, excellent models to forecast rock-fall dangers have been created using machine learning technology. Various machine learning techniques, including logistic regression, have been utilized for predicting rock falls [14], as support vector machines (SVM) [15,16]. Another approach [17] developed a tool for predicting the spatiotemporal distribution of rock fall using artificial neural networks and linear regression. The rock-fall risk was assessed using several approaches, such as a hybrid early warning system for rock-fall risk reduction [18]; this system uses three models to predict the likelihood of rock falls: a logistic regression model, a computer vision model, and a hybrid risk-reduction model, which also provides early warning and hazard level classification. Because the model was created using insufficient historical data about a particular location, the current prediction approaches are ineffective at reducing the rock-fall risk in real-time. Although this system contributed to reducing the risk of rock falls, the prediction process still needs to be more accurate and uncertain because it inherited previous models’ limitations. In order to overcome the constraints of all earlier models, offering an accurate prediction is necessary.

This study proposed a framework to reduce rock-fall risk. The primary purpose of this framework is to augment prediction accuracy by eliminating the uncertainties and confusion that plague the prediction model. In order to achieve prediction accuracy augmentation, four different techniques were integrated, namely, detection model-based (computer vision and micro-seismic wave), prediction-based (deep learning model), an Internet of Things network (IoT), and a decision-making algorithm.

Finally, the following are the key contributions of this study:

• We propose an IoT-based framework for rock-fall early warning.
• We created a deep learning model to predict the likelihood of rock-fall events.
• We created a detection model-based micro-seismic wave and computer vision.
• We have augmented the accuracy of a prediction model by fusing the detection model with a prediction model.
• We developed a decision-making algorithm.
• We provide a baseline methodology and a prediction accuracy benchmark for future related work.

This study’s remaining sections are structured as follows. In Section 2, the study area and issues are presented. Section 3 introduces data acquisition. Section 4 presents the methodology. Section 5 presents the results and discussion. The evaluation of the study is presented in Section 6.

2. Study Area and Problems

The study targeted two sites in southern Saudi Arabia along the Sarawat Mountain, constituting a natural obstacle to communication between cities and population centers above the mountains and those scattered in the plains and valleys [19]. The first site (Aqabat Shaar) is located on the road linking the cities of Abha and Mahayel Asir, and this obstacle extends to a length of 14 km on mountain ranges with a height of 2160 m. The second site (Aqabat Dhala) is located on the road linking the cities of Abha and Jazan. This obstacle extends 11 km along a steep mountain at a height of 2220 m. These two sites have many bends and tunnels, crowded with high traffic intensity, which increases the possibility of exposed cars at the moment of a rock fall.

One of the most important reasons for the fall of rocks in this area is the constant rainfall throughout the summer on the mountain range, which causes an absolute nightmare for passers-by in Aqabat Shaar and Dhala. It is stable and becomes suitable for the fall of rock masses. Furthermore, one of the causes of rock fall in this area is the nature of the rock formation in some locations, where it consists of large rocky blocks interspersed with rocky rubble, in addition to other areas permeated by a limestone layer. This formation makes it weak against natural factors. The difference in temperatures between night and day characterizes these areas. These temperature differences, in addition to ground movements, cause rock stress, which leads to cracks. These cracks expand with time, and rainwater retracts into the cracks, causing pressure on the rocks, which reduces cohesion strength and results in successive rock slides [20].

The construction work in the area has led to the emergence of edges with sharp sloping angles, which has reduced the gravity of the rock masses and made it easier to roll them downward, increasing the possibility of rocks falling at any moment.

3. Data acquisition

3.1. Data Collection and Preparation

The historical data on rock-fall events, in addition to meteorological data, was gathered between January 2015 and August 2021. Different sources, such as KSA Civil Defense and the Geological Hazards Research Center, were used as data sources. This period was divided into 2040 samples, which included 415 rock-fall events. During the initial filtering of the data, three non-dependent variables (slope angle, rainfall, and temperature variation) and one dependent variable (rock-fall event) were selected. The dataset from the rock-fall inventory was split into a training dataset of 70% (1428 samples) and a test dataset of 30% (612 samples) in order to train and test the model.

3.2. Rock-Fall Condition Factors

The decision to use rock-fall conditioning factors directly affects mathematical models’ accuracy [21,22]. This study utilized three rock-fall conditioning factors: hydrological (rainfall), topographic (slope angle), and weather-related (temperature variation) [23], as in

Table 1. Rock-fall conditioning factors.

Type	Factor	Unit	Factor Class
Topographic	slope angle	degree	(range 20–60)
Hydrological	rain full	mmh−1	(range 0–46)
Weather	temperature variation	°C	(range 0–21)

.

In considerable rock-fall events, rainfall directly affects the movement and rolling of rocks [24]. The rock-fall events are directly proportional to the slope angle of the mountain. The higher the slope angle, the less rock stability [25]. The difference in temperature between day and night exposes rocks to expansion and contraction, which leads to cracks in the rocks [26].

4. Methodology

4.1. Rock-Fall Early Warning Framework Design

This section presents a framework-based Internet of Things (IoT) for rock-fall early warning. Figure 1 shows our proposed framework for detecting and predicting rock-fall incidents. The framework consists of five layers: the field layer, the edge layer, the fog computing layer, the cloud computing layer, and the data presentation layer.

1. The Internet of Things layer includes the actual sensors to measure the physical parameters of interest; these include rainfall, air temperature, air humidity, a seismic sensor for detecting seismic waves, and a camera for rock movement detection. The Field layer was also expanded to accommodate an intelligent voice module and a light panel to run out the early warning action.

2. Edge Computing Layer: In order to decrease network traffic and energy consumption, the edge nodes gather input from the sensors and conduct data and image processing algorithms. The main goal is to locally generate fundamental model properties of the specific process which are then passed on to the fog computing. Furthermore, it is necessary to execute commands incoming from higher-level systems to deliver an early warning action and make local decisions, thus preventing upper-layer latency.

3. Fog Computing Layer: It bridges the gap between the cloud and edge nodes by enabling computations such as rock-fall monitoring, rock-fall prediction, deep learning models, rock-fall risk assessment, networking, data management, and decision making. Blynk Cloud was used in this study; it is a comprehensive software package needed to deploy and remotely manage linked electrical devices at any size, from small-scale home IoT projects to millions of commercially connected items.

4. Cloud computing layer: The cloud was chosen to provide communication services by streaming data between the fog computing and data presentation layers and providing a medium for data storage. Blynk utilizes HTTPS (API) to report telemetry and regularly fetch data streams. Additionally, it offers open-source hardware libraries so that any device can connect to Blynk Cloud.

5. Data presentation layer: This is concerned with the software systems that convey the data analysis results to end users or decision makers. In this study, web and mobile app dashboards are used to present the early warning alarm in the case of event detection, in addition to affording access privileges to the system administrator and decision making.

4.2. Rock-Fall Detection Model

This study obtained a robust rock-fall detection model by gathering two detection processes. First, a computer vision algorithm was used to detect the rock-fall event. Secondly, seismic wave sensors were used to detect the vibrations from rock cracking or falling.

4.2.1. Rock-Fall Detection-Based Computer Vision

The computer vision algorithms detect the rock-fall events in three steps, as shown in Figure 2. They filter the image frame, perform background subtraction, and perform frame manipulation. In the first step, the blurring Gaussian filter was used to filter out the noise from the captured images. Secondly, the moving rocks were detected from the video frame sequence. Due to weather conditions and daytime, the video frame sequence suffers from background illumination variations, so the adaptive Gaussian mixture model was used to overcome this problem.

The model treats each pixel as a composite of Gaussians before learning the image’s backdrop and categorizing each pixel as background or foreground. Equation (1) represents the background model.

$$\widehat{P}\left(\overrightarrow{x}|{\chi }_{T },BG\right)={\sum }_{m=1}^M{\widehat{\pi }}_m\mathcal{N}\left(\overrightarrow{x};{\widehat{\overrightarrow{\mu }}}_m,{\widehat{\sigma }}_m^{2}I\right)$$

(1)

where $\widehat{P}\left(\overrightarrow{x}|{\chi }_{T },BG\right)$ represents the estimated background, $\overrightarrow{x}$ is the grayscale of the pixel value at time t, M is the number of the Gaussian components, ${\chi }_T$ represents the training set, the weight ${\widehat{\pi }}_m$ indicates how many data are part of the GMM m component of the, ${\widehat{\overrightarrow{\mu }}}_m$ is estimated means, I represents an identity matrix, and ${\widehat{\sigma }}_m^2$ is the estimated variances.

Bayesian analysis was used to classify pixels as background or foreground from moving rock video frames [27]. Frame manipulation was used to overcome imperfections in the segmentation process.

4.2.2. Rock-Fall Detection-Based Micro-Seismic Wave

The seismic wave is generated in two cases: at the moment rocks crack or when rocks fall. Thus, the seismic wave sensor can be used to detect rock-fall events. This study characterized the micro-seismic wave by its frequency component, classified into three frequency domains. A significant frequency spectrum band between 100 and 1000 Hz was present in the first domain. These signals are generated several hours prior to the rock’s fall. The second domain is in the higher frequency band, between 500 and 1000 Hz.

The third domain is in the lower frequency spectrum, 100 to 500 Hz. These signals precede the rock-fall event by a few moments. The relationship between rock-fall incidents and seismic wave frequency domains was quantified using the spectral amplitude ratio (R). The spectral amplitude ratio (R) is calculated according to Equation (2) [28].

$$R=\frac{{\mathrm{A}}_{\mathit{MAX}}(100 \mathrm{H}\mathrm{z}–500 \mathrm{H}\mathrm{z})}{{\mathrm{A}}_{\mathit{MAX}}(500 \mathrm{H}\mathrm{z}–1000 \mathrm{H}\mathrm{z})}$$

(2)

where A_MAX (100 Hz–500 Hz) is the maximum amplitude of the frequency spectrum 100 Hz to 500 Hz, and A_MAX (500 Hz–1000 Hz) is the maximum amplitude of the frequency spectrum 500 Hz to 1000 Hz. The average amplitude ratio R for all frequency domains is shown in

Table 2. The average spectral amplitude ratio R.

Frequency Domain	Frequency Spectrum	R
first domain	100 Hz–1000 Hz	1.5 ± 0.08
second domain	500 Hz–1000 Hz	2.7 ± 0.32
third domain	100 Hz–500 Hz	7.1 ± 0.68

.

4.3. Rock-Fall Prediction Model

Due to the randomness of a rock-fall occurrence, the function of the rock-fall event probability P = f_x(r, s, t) is uncertain; therefore, mapping the relationship between the rock-fall possibility P, the slope angle s, the rain-fall r, and the temperature variation t cannot be strictly described. To solve the uncertainty of the mapping function, the BP (backpack) artificial neural network can learn the relationships between the input and output of the corresponding procedure by analyzing sample data and adopting a model that gives the expected output value when given the input value [29].

4.3.1. Deep Learning Model

In this paper, we propose a deep learning model for rock-fall occurrence prediction. It is part of a machine learning process. It is a complex mathematical model that simulates the biological neuron structure and self-learning function. It uses mathematical methods that are based on the idea of linked layers of nodes. [30]. As seen in Figure 3, it has an output layer, three hidden layers, and a three-parameter input layer.

As shown in Figure 4, each node in this neural network is a neuron, and each neuron has six major components, including inputs (x_i), biases (b_j), weights (w_ij), sum functions (u_j), activation functions (f), and outputs (y_j). The information from neurons or the outside world that is used as a decision variable is referred to as an input. Weights are values that translate the influence of inputs on one another. The sum function (Equation (3)) is an operation that takes into account a bias value and reflects the impacts of inputs and weights [31].

$${\left(u\right)}_j={\sum }_{i=1}^n{w}_{ij}{x}_i+b_j$$

(3)

where:

• i is the ith input neuron;
• j is the jth output neuron;
• n is the number of elements in the ith input vector;
• b_j is the bias value (also known as the activation threshold) connected to the jth node.

According to the equation in Equation (4), the activation function is in charge of converting the node’s summed weighted input into the activation of the node or output for that input.

$$y_j=f(u_j)=\frac{1}{1+e^{-\propto (u_j)}}$$

(4)

where y_j represents the output of the jth neuron, controls the slope of the rectified linear activation function, and is typically equal to 1.

4.3.2. Training Methods

The SciKit-Learn neural network package was used to create the neural network models that were used in this study. The model was generated with three hidden layers. In our method, the training process used 70% of the entire data, while the validation process used 30% of the remaining data. The training process was operated using the multilayer perceptron (MLP). The MLP was chosen because it provides quick predictions after training. It utilizes a supervised learning technique called backpropagation for training with the rectifier linear unit (ReLU). The learning algorithm performs backpropagation, which calculates the correct gradient for nonlinear multilayer networks to reduce errors (the gap between prediction and actual values) [32].

In this study, a gradient descent method was used as an optimization model. This method updates the variables iteratively in the opposite direction of the gradients of the objective function. Equation (5) performs the gradient descent algorithm. It updates the weight and bias parameters iteratively in the negative gradient direction to minimize the loss function f(θ).

$${\theta }_i={\theta }_i-\alpha \ast \frac{d}{d{\theta }_i} f(\theta )$$

(5)

where α is a learning rate, f(θ) is a loss function, and θ_i is the weight or bias parameter, which we need to update.

4.3.3. Model Performance Validation

In this section, the overall model performance (recall, specificity, precision, F1-score, and accuracy), as well as the mean squared error (MSE) and area under a receiver operating characteristic (ROC) curve (AUC), were used to validate the model’s ability to distinguish between the occurrence of a rock-fall and a non-rock-fall event. The system’s performance was calculated using the confusion matrix [33]. The first metric is recall, which is referred to as sensitivity or the true-positive rate. The following calculation measures how well the model predicted the rock-fall event:

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{TP}{TP+FN}\times 100\%$$

(6)

The second metric, specificity, is used to assess a system’s capacity to verify the absence of a rock-fall event, which is described as:

$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}=\frac{TN}{TN+FP}\times 100\%$$

(7)

The third metric is precision, it is used to determine how many samples really fall into the positive class out of all those that the model projected would.

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{TP}{TP+FP}\times 100\%$$

(8)

The fourth metric is F1-score, is used as a model’s predictive power assessment. The F1-score is derived by combining the accuracy score and the recall score of a model, and its definition is as follows:

$$\mathrm{F}1 \mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\ast \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\ast \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$

(9)

The fifth metric is accuracy, which is a reflection of how accurately the system can identify the rock-fall event, its defined as follows:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{(\mathrm{T}\mathrm{N}+\mathrm{T}\mathrm{P})}{(\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}+\mathrm{T}\mathrm{N}+\mathrm{T}\mathrm{P})}\times 100\%$$

(10)

where true positive (TP) indicates that all events were indeed discovered, false negative (FN) indicates that some events took place but went undetected, and true negative (TN) indicates that no events took place. A false positive (FP) event is absent, yet the system records it as present. The system reports that the event is absent.

Equation (11) uses the mean squared error (MSE) to calculate the average squared difference between the values of real and predicted data points in order to quantify the degree of error in the learning model.

$$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{\mathrm{n}}{\sum }_{\mathrm{i}=1}^{\mathrm{n}}{({\mathrm{Y}}_{\mathrm{i}}-{\widehat{\mathrm{Y}}}_{\mathrm{i}})}^2$$

(11)

where n is the total number of data points in the dataset, Yi is the actual data point values, and i is the projected data point values. The overall performance of a prediction is measured using the area under a receiver operating characteristic (ROC) curve (AUC), which is based on a confusion matrix.

4.4. Rock-Fall Risk Assessment

The likelihood that a rock-fall event will occur at a specific location and at a specified time and impose a specific level of damage to roads, automobiles, and pedestrians was characterized as rock-fall risk. Then, the risk was calculated in terms of the temporal and spatial information on the effect of precipitation. Based on the possibility that cars are available in a certain position and time period affected by rocks falling, the temporal-spatial probability and susceptibility were determined [34]. Equation (12) shows the risk probability’s value.

$$P_{\left(Risk\right)}=f_r\times P(S:T)\times P\left(r\right)\times V\left(u\right)$$

(12)

where P_(Risk) denotes the likelihood that a rock-fall incident will occur within a given hour and fr denotes the frequency of rock falls. The possibility that a rock will fall and impact the car is P(r). The vulnerability of the vehicle regarding rock-fall incidents V(u), has two possible values: 1 if the rock actually hits the car, and 0 if it does not. The likelihood that automobiles will be accessible at a particular location and time is known as P(S:T). There is a temporal–spatial probability that a car traveling the entire path will be impacted at the moment of impact. It is determined using Equation (13) [35].

$$p_{(S:T)}=\frac{N_V}{24}\times \frac{L_v}{1000}\times \frac{1}{V_v}$$

(13)

where N_V is the average daily number of vehicles, L_v is the average length of a vehicle in meters, and V_v is the average speed of a vehicle in kilometers per hour.

4.5. Rock-Fall Prediction Model Augmentation

The prediction made by the deep learning model is subject to noise, model errors, and uncertainty [36]. Therefore, it is highly desirable for any AI-based system to represent uncertainty in a reliable manner. In this part of the article, the prediction model has been enhanced by increasing the overall level of model accuracy so that a precise decision can be made to reduce the chance of a rock falling.

For our proposed model, noise in data and inadequate knowledge lead to uncertainty in data; subsequently, the naturally uncertain nature of the data is modeled by the predictions, making it irreducible. In this study, to address this problem, a new method was proposed to decrease uncertainty by diminishing two instances of confusion: situations where certain events occur but are identified as undetected are false negatives (FN) and cases where some events not occur but are identified as present are false positives (FP).

The augmented rock-fall prediction probability P(J) is obtained by applying the union of independent probabilities theory between the probability of the rock-fall risk according to the detection models P(D), and the possibility of a rock fall according to the prediction models P(p). Figure 5 shows the steps involved in calculating the joint rock-fall probabilities utilizing detection and prediction models.

The rock-fall probability of the detection models is calculated from the micro-seismic detection model and the computer vision detection model, and then the joint probability is calculated as in Equation (14).

$$P\left(D\right)=P\left(S\cup V\right)=P\left(S\right)+P\left(V\right)-P(S\cap V)$$

(14)

where P(D) is the rock-fall probability of the detection models, P(S ∪ V) is the union probability, P(S) is the rock-fall probability determined by the micro-seismic detection, P(V) is the rock-fall probability determined by the computer vision detection model, and P(S ∩ V) is the probability that P(S) and P(V) are mutually exclusive. In this study, P(S) was determined from the spectral amplitude ratio (R) of the micro-seismic event, as in Equation (15).

$$P\left(S\right)=\frac{R}{R_{MAX}}$$

(15)

where R is the spectral amplitude ratio of a micro-seismic event, and takes the values from (1.5 ± 0.08) to (7.1 ± 0.68), as mentioned in Section 4.2, and R_MAX is the ratio of the spectral amplitude when the rock-fall occurrence is confirmed, and its value is equal to (7.78). The P(V) value is 0 in cases where no rock fall is detected, and it is valued at 1 in cases where rock fall is detected. By substituting the values of the probabilities into Equation (6), we obtain Equation (16).

$$P\left(D\right)=\frac{R}{\left(7.78\right)}+P\left(V\right)-\frac{R}{\left(7.78\right)}\times P\left(V\right)$$

(16)

The rock-fall occurrence probability (P*), which was obtained from the prediction model (artificial neural network), was used to determine the rock-fall risk probability. Finally, the augmented rock-fall prediction probability P(j) was calculated by combining the rock-fall probability of the detection models P(D) with the rock-fall risk probability of the prediction models P(P), Equation (17).

$$P\left(j\right)=P\left(D\right)+(1-P\left(D\right))\times P\left(P\right)$$

(17)

The overall accuracy of the augmented model can be obtained from Equation (18).

$$Au=\frac{(\mathrm{T}\mathrm{N}+\mathrm{T}\mathrm{P})}{(\mathrm{T}\mathrm{N}+\mathrm{T}\mathrm{P})+\delta (\mathrm{F}\mathrm{N}+\mathrm{F}\mathrm{P})}\times 100\%$$

(18)

where Au is the augmented model accuracy, and δ is the uncertainty-decreasing factor.

$$\delta =(1-P(D\left)\right)$$

(19)

4.6. Rock-Fall Risk Reduction Process

The risk reduction process is carried out by preventing cars and pedestrians from entering the vulnerable area. This study used early warning to prevent cars from entering a hazard zone. The likelihood that vehicles would not enter the danger zone after receiving the early warning signal at the time of the incident was used to determine the risk reduction [37]. In this study, the value of a risk reduction was calculated using the risk reduction probability, which includes the system reliability, the average number of cars, and the likelihood of vehicle response, as in Equation (20).

$$P_{\left(r\right)}=A_u\times N_v\times P_{\left(rs\right)}$$

(20)

where P(R) refers to the possibility that the risk will be reduced, P_(rs) refers to the probability that a given vehicle will not reach the affected road segment after receiving the warning signal, Nv refers to the average number of vehicles, and Au refers to the overall accuracy of the enhanced model. The value of P_(rs) is determined by applying Equation (21):

$$P_{\left(rs\right)}=1-\left(\frac{Total Stopping Distance}{Safe Distance to Stop}\right)$$

(21)

Based on the physical force distance, response time, and brake contact distance, the overall stopping distance was calculated. The safe distance is determined by the reaction time of the vehicle driver. The reaction time of the driver varies randomly in a range between 0.4 and 2 s. The physical force distance can be calculated by dividing the vehicle’s speed by the amount of time it takes for the brakes to react. An acceleration of 10 m/s² is implied [38].

4.7. Decision-Making Algorithm

This algorithm was developed to make the appropriate decision about reducing the rock-fall risk. It fuses the outputs of the rock-fall prediction model (the deep learning model) with the outputs of the detection model in order to obtain the augmented prediction, as well as calculating the risk of a rock fall, categorizing it into three levels, and creating a warning strategy to handle a severe hazard situation. The subsequent steps demonstrate how the proposed Algorithm 1 utilizes and controls a rock-fall danger level.

Algorithm 1: Figure out the rock-fall risk, identify the risk level, and carry out the rock-fall risk reduction process

The first step: Gathering information with the IoT layer

Read rainfall by rain sensors

Read temperature by temperature sensors

Read IoT camera video frames

Read seismic waves by seismic sensor

The second step: Detection of falling rocks

in accordance with Equation (1)

The third step: Determine the rock-fall occurrence probability (P)

in accordance with deep dearning model

The fourth step: Compute the total rock-fall risk probability P(j)

in accordance with Equation (17)

The fifth step: Classifying the hazard in to three levels:

When P(Risk) is greater than or equal to (1 × 10−3)

then hazard is at an unacceptable level.

When P(Risk) is greater than (1 × 10−6) and less than (1 × 10−3)

then hazard is at a tolerable level.

When P(Risk) is less than or equal to (1 × 10−6)

then hazard is at an acceptable level.

The sixth step: performing the risk reduction action

Reducing the risk of rock falls by sounding and lighting warnings

Turn on the red light + sound when the hazard is at an unacceptable level.

Turn on the yellow light when the hazard is at a tolerable level.

Turn on the green light when the hazard is at an acceptable level.

The seventh step: Return to first step.

5. Results and Discussion

The research, findings, and framework discussion are presented in this section. The findings of the experiment provide demonstrations for three different terms. First is the deep learning model validation. First comes the validation of the deep learning model. The second is a risk assessment for rock falls. The assessment of the risk reduction comes last.

5.1. Deep Learning Model Validation

Performance metrics were obtained based on the four possibilities from the confusion matrix of TP, TN, FP, and FN, and were used to validate the proposed deep learning model (

Table 3. The confusion matrix.

		Predicted Event
		Does Not Occur 0	Occurs 1
Actual Event	Does not occur 0	TN = 304	FP = 51
	Occurs 1	FN = 35	TP = 222

). These metrics included specificity, accuracy, precision, F1-score, and area under a receiver operating characteristic (ROC) curve (AUC) metrics, in addition to mean squared error (MSE).

A report on the deep learning model for the 612 testing samples obtained through the confusion matrix (Table 3). It shows that the actual detected events (TP) numbered 222, the number of events that occurred but were not detected (FN) was 35, the number of events that did not occur and the system generated an absence event report (TN) was 304, and the number of events that were absent but the system reported as present (FP) was 51. Additionally, the outcome demonstrates an ability to ignore fake occurrences, with an average specificity of 85.6%. The accuracy of 86% reflects the percentage of times a model produced a prediction throughout the entirety of the dataset.

The classification report (

Table 4. Classification report.

Class	Precision	Recall	F1-Score	Support
Rock Fall (Not occur 0)	91%	86%	88%	355
Rock Fall (Occurs 1)	81%	86%	84%	275
Accuracy			86%	612
Macro avg	85%	86%	86%	612

) shows the validation data average recall (sensitivity) for both classes (rock fall does not occur is 0 and rock fall occurs is 1) is 86%. That means, at the lowest sensitivity levels, only 14% of the rock-fall occurrences were improperly recognized. The average precision of 85% shows that the majority of the positive predictions made by the model were correct. The average F1-score is 86%.

The deep learning model’s loss curve for 200 iterations is shown in Figure 6. In contrast, during the neural network training session, the cost value drops with each iteration, reflecting the performance of the learning through time. Finally, the cost value eventually dropped to less than 0.14 points, which is regarded as an acceptable mean squared error (MSE) score.

The ROC curve shown in Figure 7 demonstrates the model’s accuracy in predicting the occurrence of rock falls. The area under the ROC curve (AUC) value reached 0.946.

5.2. Rock-Fall Risk Assessment Result

The method outlined in Section 4.4 (Equation (12)) of this study was applied to calculate the rock-fall risk probability. The Python environment was used as a simulation tool. The settings and configurations used for the simulation are listed in

Table 5. Simulation setups.

Parameter	Value
Average daily number of vehicles on the road (NV)	8325 vehicles
Average vehicle lengths	5.4 m
Brake engagement time	2 s
Driver reaction time	0.4 to 2 s
Average acceleration	10 m/s2

.

In this study, we used the normal distribution of vehicle traffic for twenty-four hours a day and assumed the value of the vulnerability of the vehicle to be equal to one, considering that any rock-fall event will cause damage to cars. The result in Figure 8 shows the rock-fall risk probability per hour during the day. The findings indicate that the values for the highest and lowest rock-fall risk probability were, respectively, 1.51 × 10⁻³ and 7.98 × 10⁻⁶. Moreover, the relationship between the local time and the rock-fall risk probability (Figure 8) shows that the high-risk probabilities are concentrated in the local time period between 12 p.m. and 6 p.m.

In order to classify the risk values into three levels, the outcomes were compared to the triangle of safety-critical regulation and management thresholds, which is shown in Figure 9 (ALARP) [39]. The result shows that the risk probability values were spread among all ALARP levels. Overall, 29.1% of values are unacceptable, and the remaining risk values are divided between acceptable and tolerable levels by 16.6% and 54.2%, respectively.

5.3. Model Uncertainty Reduction

As mentioned in Section 4.5, in order to improve prediction accuracy, we have to reduce the uncertainties and confusion that plague the prediction model by using two independent detection models (computer vision and micro-seismic wave).

Table 6. Micro-seismic wave response.

Frequency Spectrum	The Signal Generation Moments	The Average Spectral Amplitude Ratio R	P(S) Rock-Fall Detection
500 Hz–1000 Hz	Several hours prior to the rock’s fall	2.7 ± 0.32	0.31–0.39
100 Hz–500 Hz	Precede the rock fall by a few moments OR moment of rock fall occurrence is confirmed	7.1 ± 0.68	0.83–1.00

confirms that the detection-based micro-seismic wave has the ability to predict the fall a few moments before as well as confirm the rock-fall occurrence.

The results in Table 6 show the micro-seismic wave response to two events with different frequency spectrums. In the first case, when detecting vibrations in the frequency spectrum 500 Hz–1000 Hz, this frequency appears several hours prior to the rock’s fall. In this case, the probability of detecting a rock fall ranges between 0.31 and 0.39. Second, when detecting vibrations in the frequency spectrum 100 Hz–500 Hz, this frequency precedes the rock fall by a few moments or at the moment of rock-fall occurrence; in this case, the probability of detecting rock fall ranges between 0.83 and 1.00. These results confirm the ability of the seismic detector to detect rock-fall events before and after their occurrence.

To obtain the uncertainty-decreasing factor (δ), we first conducted experiments on the micro-seismic and computer vision detection models. The results indicate that the average probability of rock fall when occurrence is confirmed is 0.96 and 0.97, respectively, as in

Table 7. Prediction uncertainty reduction by detection model.

The Detection Model	Detection Probability	Probability Value When Occurrence is Confirmed	The Average Probability	Uncertainty-Decreasing Factor (δ)
Micro-seismic	P(S)	0.93–1.00	0.96	-
Computer vision	P(V)	0.94–1.00	0.97	-
The overall detection models	P(D)	0.87–1.00	0.93	0.07

. Then, the average rock-fall probability of overall detection models was calculated as the joint probability of two independent events; the result indicates that the average probability of overall detection models is 0.93. Finally, by substituting the overall detection model probability into Equation (19), we obtain a minimum uncertainty-decreasing factor of about 0.07.

5.4. Overall Model Accuracy Augmentation

In order to perform prediction model augmentation, the decision-making algorithm was used to obtain the uncertainty-decreasing factor by computing the overall detection model probability from the real-time data. In the case of obtaining an uncertainty-decreasing factor in the range of 0.07, it decreases the FP from 51 to 3.57, and in the same context, it decreases the FN from 35 to 2.45. The overall accuracy of the augmented model was obtained by substituting the new values of FP and FN in Equation (18). We found that the overall accuracy after augmentation increased from 86% to 98.8%, as shown in

Table 8. Effect of augmentation in increasing overall model accuracy.

Rock-Fall Risk Prediction Model	FP	FN	Accuracy
Before Augmentation	51	35	86%
After Augmentation	3.57	2.45	98.8%

. This result confirms the effectiveness of the augmentation in raising the model’s accuracy and reducing the confusion associated with the prediction process.

5.5. Rock-Fall Risk Reduction

In this study, the risk reduction procedure begins with detecting and predicting rock fall, and then the decision-making algorithm issues an early warning to prevent vehicles from entering the danger zone a few moments before or at the time the event occurs.

Results indicated that, after using our proposed framework, the rock-fall risk probability values decreased from 1.51 × 10⁻³ to 7.98 × 10⁻⁶, to new values in a range of 8.57 × 10⁻⁹ to 8.21 × 10⁻⁷, as shown in Figure 8 and Figure 10. Thus, we found that during the rush hour (1 p.m. to 6 p.m.), when the probability of risk increases according to the intensity of vehicular traffic, our proposed framework reduces the risk by a 10⁻³ factor, from 1.51 × 10⁻³ to 8.21 × 10⁻⁷, as shown in

Table 9. Rock-fall risk probability before and after reduction.

Rock-Fall Risk Probability	Minimum	Maximum
Before Reduction	7.98 × 10−6	1.51 × 10−3
After Reduction	8.57 × 10−9	8.21 × 10−7

.

When comparing the reduced risk values with ALARP levels, we found that all risk values after reduction were less than 1 × 10⁻⁶, and therefore all values were located at acceptable levels.

6. Conclusions

In this study, an early warning framework was developed to reduce the risk of rock falls. First, 2040 samples of historical rock-fall event data were gathered from various sources, and randomly split into two groups, with 70% going for deep learning model training and 30% going for model validation. Next, the model prediction accuracy was augmented using a real-time rock-fall detection model based on IoT and computer vision. Finally, the decision-making algorithm assesses the risk of a rock fall, categorizes it into three levels, and produces a warning reaction to handle the critical hazard situation.

This study utilized parameters and overall prediction performance measures based on a confusion matrix to compare the performance of the model before and after augmentation. The results showed that the models had acceptable performance. The results show that the overall model accuracy before augmentation was 86%, which became 98.8% after the augmentation. Additionally, as shown in Table 9, a framework can lower the risk probability from 1.51 × 10⁻³ to 8.57 × 10⁻⁹.

By comparing our new augmented model with our model in a previous study [14] based on performance and the ability to reduce the risk of rock fall, we discovered that the new proposed model’s prediction accuracy was 98.8%. In comparison, the previous study’s accuracy was 97.9%. In addition, the new model reduced the risk probability to (8.57 × 10⁻⁹), while the previous study reduced the risk to (1.13 × 10⁻⁸). This result indicates that although the previous model had acceptable performance, the augmented model outperformed it. It can be considered a promising technique for predicting and reducing rock-fall risks.

In future studies, we suggest enhancing the real-time rock-fall detection process by increasing the number of seismic sensors to achieve a higher detection accuracy and enhancing the current DL prediction model’s accuracy by adding more rock-fall conditioning factors.