Detection of Blockages of the Belt Conveyor Transfer Point Using an RGB Camera and CNN Autoencoder

Abstract

In the material transfer area, the belt is exposed to considerable damage, the energy of falling material is lost, and there is significant dust and noise. One of the most common causes of failure is transfer chute blockage, when the flow of material in the free fall or loading zone is disturbed by oversized rock parts or other objects, e.g., rock bolts. The failure of a single transfer point may cause the entire transport route to be excluded from work and associated with costly breakdowns. For this reason, those places require continuous monitoring and special surveillance measures. The number of methods for monitoring this type of blockage is limited. The article presents the research results on the possibility of visual monitoring of the transfer operating status on an object in an underground copper ore mine. A standard industrial RGB camera was used to obtain the video material from the transfer point area, and the recorded frames were processed by a detection algorithm based on a neural network. The CNN autoencoder was taught to reconstruct the image of regular transfer operating conditions. A data set with the recorded transfer blockage state was used for validation.

1. Introduction

A belt conveyor allows bulk materials to be moved over long distances [1,2] while providing optimal electricity consumption [3,4] and reliability [5] indicators. For this reason, belt conveyors are one of the most popular means of transport in industry and mining [6]. The length of the belt conveyor route is limited by the permissible belt stresses in the upper belt [7], resulting from the resistance to movement along the length of the route [8]. As a standard, it is assumed that the operating stresses should be 10% of the nominal belt strength [9]. Intermediate drives are a solution that extends the length of the route [10], but their use is not always possible or cost effective [11]. In such cases, it is necessary to use parallel transfer chutes, which allows the material to be reloaded from the feed conveyor to the receiving conveyor in the transport line [12]. In a situation where it is required to feed the material to the conveyor at a significant angle and change the flow direction, angular chutes are used [13].

Transfer chutes are crucial infrastructure elements that connect conveyors that are part of complex transport systems [14]. Proper operation of transfers determines the continuity of production. Blockage causes production shutdown and, consequently, losses related to downtime and removal of failures [15]. In addition, at the transfer points, there are significant energy losses [16] associated with additional resistance to accelerating the loaded material to the speed of the receiving belt [17] and friction against skirt boards [18]. The appropriate shape of the transfer chute structure allows for feeding the material steadily and at the right speed tangential to the receiving belt. Thanks to this, the resistance to movement is reduced, as well as the wear of the belt. Statistical studies and a review of available failure analyses show that transfer points are where the conveyor belt most often fails [19]. Due to the described issues, transfer chute projects are subject to increased supervision compared to other route elements. The design of modern transfer points is supported by numerical methods [20,21], thanks to which it is possible to optimize new [22] and already working structures [23,24]. There is also considerable dust [25,26] and noise [27] at the transfer points.

A well-designed transfer chute increases production capacity by controlling dust and noise [28], reducing belt wear [28] and mistracking [29], and eliminating potential blockages of the loading zone [30]. Blockages occurring is one of the most dangerous failures of chutes, the conveyor belt may be completely cut or broken, and significant damage to the steel structure of the route may occur [30,31]. Blockages are caused by the appearance of oversized or foreign bodies in the transported material or the feeding of too much material that the conveyor cannot receive [32]. At the moment of blockage, the loading space is gradually filled until it is filled. In this case, the receiving conveyor stops due to a significant increase in resistance or damage or a breakng of the blocked belt [33].

Most standard monitoring methods and non-contact blockage detection are adapted only to chutes with a gutter in the free fall zone and loading zone [34]. The most basic solution is proximity sensors, emitting a beam in the infrared band [35]. Information on the status of regular operation and blockage is sent with the detected movement of material. RF sensors are less and less used due to numerous operation problems in conditions of extreme dust and vibration, which generate false failure states. Over time, radiometric detectors have replaced RF sensors, which are considered one of the most effective solutions [36]. However, the sensor’s construction requires using radioactive isotopes to detect the presence of a blockage in the hopper. The radiation source must be shielded and placed in the direction of the material flow in a chute, and the detected radiation level on the opposite side allows for blockage state determination [37]. The measurement method’s most significant disadvantage is the cost, safety, and environmental impact. The use of radiometric sensors requires increased supervision of external services that allow the device to work and numerous training of service employees. The most popular solution is microwave sensors, consisting of a low-power microwave emitter and a receiver [38]. This device is relatively cheap compared to radiometric detectors, does not require additional training and safety regulations, and is not sensitive to vibration and dust. Among the contact methods, the most popular are limit switches signaling the status of the chute blockage at the moment of contact with the material at a safety level.

Many transfer points are structures without a chute in the zone of free fall of material; therefore, the measurement methods mentioned in the previous paragraph cannot be used. The article presents the research results on monitoring the transfer operation in an underground copper ore mine using images from an industrial RGB camera. The anomaly detection was based on the CNN autoencoder, which was taught the parameters of regular transfer operation. The detection effectiveness was validated based on collected data frames illustrating the actual lock states. The results of the neural network operation were illustrated based on the network learning loss function, the algorithm’s effectiveness, and the error matrix was determined. Anomalous operating states were visualized using color maps superimposed on the anomalous test dataset.

2. Related Works

Vision methods in industry and mining are widely used in production quality supervision [39,40] and condition monitoring [41,42]. One of the most developed areas is detecting damage to idlers [43], pulleys, and conveyor belt drives based on anomalies in the image of the thermal imaging camera [44,45]. Increasingly, monitoring using thermal imaging methods is carried out using robots [46,47] and automated diagnostic platforms [48]. The thermal image can also be used to identify people [49]. The RGB image is also used for diagnostic purposes to determine rollers’ anomalous operating states [50] and estimate the rotational speed [51]. In the case of conveyor belts, vision algorithms are used to detect surface damage [52,53,54]. Industrial cameras are used to detect the type of machines and vehicles [55] and to monitor the state of their operation. It is also possible to forecast performance and plan production in real-time based on parameters read from the image [56]. Another area of application is assessing grain size composition [57] based on images from drones [58], industrial cameras on conveyors [59], or cyclic transport vehicles [60]. Cameras installed on machines are used to monitor the wear status of components [61,62,63]. The issue most similar to the subject discussed in the article is the monitoring of crusher blockages based on the image from the chute [64,65]. Monitoring based on CNN is very often used to evaluate images when supervising production quality [66], monitoring the condition of structures [67], medicine [68] or nanotechnologies [69]. Many of the presented works are limited only to laboratory conditions. This article focuses on the practical use of industrial cameras and artificial intelligence in image processing to detect blockages under operating conditions.

3. Materials and Methods

3.1. Research Object

The research was carried out on a perpendicular, angular transfer (approx. 90°) in an underground copper ore mine (Figure 1). The nominal capacity of the feeding conveyor is 600 $\mathrm{t}/\mathrm{h}$, while the capacity of the receiving conveyor is 1700 $\mathrm{t}/\mathrm{h}$. The increased efficiency of the receiving conveyor is because several transfer chutes are located along its route. The receiving conveyor collects material from three feeding conveyors from several branches in the mine. The analyzed transfer is the first construction along the route of the receiving conveyor. The belt speed of the conveyor is 2 $\mathrm{m}/\mathrm{s}$, and the speed of the receiving conveyor is 2.7 $\mathrm{m}/\mathrm{s}$. The density of the transported material is approx. 1.7 $\mathrm{t}/{\mathrm{m}}^3$.

The transfer is a Rock-Box construction, where the material going into the box loses its kinetic energy and then slides onto the conveyor belt. At the moment of contact of the material with the belt, its acceleration occurs. The zone where the material is accelerated has been built with skirt boards, preventing material spillage and guiding material in the acceleration zone.

The selected transfer station is a construction with a tendency to blockades. Transported material moves the phase of preliminary grinding and separation on the grate [70], but this stage does not allow for maintaining a balanced grain size distribution of the ore stream. For this reason, many elongated lumps of rock in the material can cause blockage of the transfer point. In addition, there are often wastes in the stream that are the remnants of exploitation, such as rock bolts. The object also has a disadvantage in its design, as the falling stream of material hits the chute plate directly and does not slide freely down the side chute plate onto the receiving conveyor belt in the loading zone.

3.2. Measuring Equipment

Images were captured with an AXIS P1354-E standard industrial RGB camera (Figure 2). The camera is a regular part of the mine monitoring infrastructure, used to monitor the transfer space by the mine dispatcher. The article also tested the recorded image regarding the possibility of automatic detection of transfer blockages and unwanted foreign bodies in material drop and loading areas. The recorded images were in the mine’s internal data acquisition system, while the processing was already performed in external software. The Ethernet protocol carried out power, communication, and data transmission. The camera was placed under the roof of the excavation at a distance of approx. 6 m from the transfer point. The camera is resistant to dust, vibrations, and climatic conditions, which is important due to the constantly deteriorating micro climate [71] and temperature above 30 °C. For this reason, no additional cooling of the apparatus was required. The camera has been able to record images with a frame rate of 10 $\mathrm{FPS}$. The data rate was 13,197 $\mathrm{kb}/\mathrm{s}$. The video image is distributed into data frames measuring 1920 × 1080 $\mathrm{px}$, 96 $\mathrm{dpi}$ resolutions in two directions, and 24 bits deep.

3.3. Methodology

Data acquisition was the first stage of the work. Data frames in video materials divided over individual frames were collected for different periods of transfer chute operation so that the training data set contained all the characteristic features of the regular operating state. The training set consisted of 1078 photos. To validate the model, 205 images were used, on which there were various types of blockages in the form of oversized lumps and foreign bodies blocking the free flow of material. Figure 3 shows sample images later used to create training and test collections.

The original images were processed in the second stage to isolate distinctive features and reduce interference. Through subsequent trials with different image processing methods and subsequent iterations of algorithm learning, the best results were obtained by degrading the complexity of the input images. The data frames were cropped and resized to represent the area of free fall and loading of material onto the receiving conveyor. Frequency domain filtering is used to reduce noise and image features while preserving all edges of objects. Then, using threshold operators and morphology, the elements of the transfer chute and the flowing material were separated from the background.

The last, third stage included neural network learning in order to obtain the best results of the reconstruction of the input image. A previously trained convolutional neural network VGG16 [72] was used, which was additionally modified for the purposes of the article. The network consists of six blocks. Each block contained convolutional layers and a pooling layer (MaxPooling). Some layers were preserved in a learned state, and the remaining layers were trained. After the section of subsequent blocks, one more layer of pooling (Global Average Pooling) was added. The last layer defined the class of the object being analyzed. Multi-class classification was used for two predicted states with Softmax activation. The autoencoder was validated against a test data set containing anomalous operating states. The loss of image reconstruction and the performance metrics of the prediction model were determined. The results were visualized using modified code [73] that allowed the density of outliers in images classified as abnormal to be determined using a color scale. Figure 4 presents the algorithm of conduct and subsequent tasks discussed in the article.

3.4. Image Processing

3.4.1. Crop

In the first stage of processing, the recorded data frames were cut only to the area where the transfer box was visible, and therefore the moment of free fall of the material and loading on the receiving belt was analyzed. Images from the original size were cropped to 1000 × 1000 for network training, and the input image size was further reduced to 225 × 225 without changing the aspect ratio. Figure 5 shows what an example image of regular (Figure 5a) and failure working state (Figure 5b) looks like after cropping and resizing.

3.4.2. Smoothing

The Gaussian averaging operator was chosen as the most optimal way to smooth the image. Using Gaussian blur is the same as combining an image with Gauss. Thanks to this filter, it is possible to reduce components from high-frequency images (low-pass filter), which allows limiting interferences recorded in underground camera conditions. It is a type of image blur filter that uses the Gaussian function to determine the normal distribution in statistical issues to calculate transformations for each pixel of the image being processed. The Gaussian function g for a one-dimensional system with a coordinate x, controlled by the variance of the ${\sigma }^2$, is [74]:

$$g(x,\sigma )=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\left(\frac{x^2}{2{\sigma }^2}\right)}$$

(1)

Image processing requires the use of the Gaussian function g for a two-dimensional system with coordinates $x,y$, which is described by the equation:

$$g(x,y,\sigma )=\frac{1}{2\pi {\sigma }^2}{e}^{-\left(\frac{x^2+y^2}{2{\sigma }^2}\right)}$$

(2)

Equation (2) allows the calculation of the Gaussian template, which is then entangled with the image. For processing, use a 7 × 7 template that reduces many image features. Once entangled, each pixel has a new weighted average neighborhood value for that pixel. The original pixel has the highest weight value, and the weight of adjacent pixels is based on the distance from the original pixel. Thanks to this, the image obtains blurring and reduces many components and distortions, while maintaining the most important feature, which is the objects’ edges and boundaries [74]. Figure 6 shows the picture of regular transfer operation and its spectrum before and after filtration.

3.4.3. Thresholding

In the next stage, to distinguish the characteristic features of the images, a threshold was used, where the pixels above the designated level received white and below the black color [74]. Thanks to this procedure, it is possible to separate the background elements of the transfer from its construction and excavation, which isolates objects of interest from the monitoring point of view. In this case, binary thresholding combined with the Otsu method [75] and color inversion was used. The Otsu method allows the determination of the optimal value of the threshold of pixel separation and separation of the object from the background based on the difference in intensity. The research shows that the Otsu method maximizes the probability that the threshold will be chosen to divide the image between the object and its background [76,77,78]. This is achieved by selecting a threshold that provides the best class separation for all pixels in the image. This method uses a standardized histogram in which the number of each level point is divided by the total number of image points. Therefore, this represents the probability distribution for intensity levels as [74]:

$$p\left(l\right)=\frac{\mathbf{N}\left(l\right)}{N^2}$$

(3)

The formula is used to calculate the cumulative moments of the zero and first-order normalized histogram to the k-th level as:

$$\omega \left(k\right)=\sum _{l=1}^{k}p\left(l\right)$$

(4)

$$\mu \left(k\right)=\sum _{l=1}^{k}l·p\left(l\right)$$

(5)

The total average image level is:

$$\mu T=\sum _{l=1}^{N_{mat}}l·p\left(l\right)$$

(6)

The variance of class separation then has the form:

$${\sigma }_B^2\left(k\right)=\frac{{(\mu T·\omega \left(k\right)-\mu \left(k\right))}^2}{\omega \left(k\right)(1-\omega (k\left)\right)}\forall k\in 1,N_{max}$$

(7)

The optimal threshold is the level at which the variance of class separability is maximum. That is, the optimal threshold is one where the variance is:

$${\sigma }_B^2\left(T_{\mathrm{opt}}\right)=\underset{1\le k<N_{max}}{max}\left({\sigma }_B^2\left(k\right)\right)$$

(8)

The threshold results are shown in Figure 7 on the example of an image for a normal operating state and blockage.

3.4.4. Morphology

The operators used in mathematical morphology were developed based on set [79]. Thanks to them, we process images according to shape, treating both of these elements as separate sets of points [80]. Local transformations defined that changing the value of the pixels represented as sets. Changing pixel values is strictly formalized by a miss-and-hit transformation. The object is represented as a X set and tested by a structural element represented by the B set. The transformation is defined as the operator point [74]:

$$X\otimes B=\left\{x\mid B_x^1\subset X\cap B_x^2\subset X^c\right\}$$

(9)

In Equation (9), x represents one pixel in the image, which is one element from the set of X. $X^c$ is the complement of X, that is, the set of pixels of the image that are not in the set X. Dement structuring B is represented by two parts $B^1$ and $B^2$, which apply to the set of X or its complement $X^c$. The structural element is the shape, which is how mathematical morphological operations process images according to the properties of the shape. An operation $B^1$ on a set of X is a hit transformation, and $B^2$ on a X is a miss. The lower index indicates that the structuring element has moved to the position of the x element. The B dement defines the window that is moved through the image [74]. In image processing, the simplest forms of morphological operators, i.e., erosion and extension, were used. Erosion, otherwise known as reduction, occurs when $B^1$ is empty:

$$X\ominus B=\left\{x\mid B_x^1\subset X\right\}$$

(10)

In the erosion operator, the hit or miss transformation determines whether a pixel x belongs to the set. It is possible when each point of the element $B^1$ transferred to x is in the set of X. Since all points in $B^1$ must be in X, this operator removes pixels on the boundaries of objects in the X set. Thus, it erodes or shrinks the collection. The operator was used to remove noise from images after thresholding. The erosion operator sometimes causes the loss of valuable edges, so the image has been corrected using dilation, otherwise known as increase, which occurs when $B^2$ is empty [74]:

$$X\oplus B=\left\{x\mid B_x^2\subset X^c\right\}$$

(11)

The results of image processing using morphological operators are shown in Figure 8.

3.5. CNN Autoencoder Structure

An autoencoder based on a convolutional neural network was used to detect anomalies on subsequent frames from the video material. This algorithm was taught to reconstruct the input data. Autoencoders are taught to encode data in an unsupervised manner, which is important in the case of binary classification of anomalous events at the transfer point. Preparing a dataset containing regular- and anomalous-state labels is difficult. Blockages and other anomalous events are rare, so it is difficult to prepare a proper database. In addition, anomalous events in the case of transfer are so diverse that detection and manual labeling of all possible cases are practically impossible [81].

Autoencoders consist of three main parts. The first part, an encoder, compresses the input into an encoded representation. The size of the encoded representation is usually several times smaller than the size of the input data. The second part, called the bottleneck, is the latent space where the encoded representation of the input in a compressed form is stored. The last, third part, called the decoder, is used to reconstruct compressed data. The reconstruction data are then compared with the input data, and a reconstruction error is determined [82]. Figure 9 shows a diagram of a typical autoencoder used to reconstruct an input image.

Iterative neural network learning involves specifying a loss function that determines the error of the decoder reconstruction relative to the input image. The autoencoder loss function can be written as [83]:

$$Loss={∥x-d_{\varphi }\left(z\right)∥}^2={∥x-d_{\varphi }\left(e_{\theta }\left(x\right)\right)∥}^2$$

(12)

The proposed neural network model determines the probability of anomalies in the studied image. For this reason, categorical cross-entropy was chosen as the parameter characterizing the autoencoder loss function. The categorical cross-entropy function can be used for two-class or multi-class classification. It allows for calculating the average difference between the actual value and the predicted probability distributions for all classes in the analyzed classification problem. In subsequent iterations, the result is minimized, and the ideal cross-entropy value is 0 [84]. The equation can describe the cross-entropy for a class problem:

$$Loss=-\sum _{i=1}^{\begin{array}{c}N\end{array}}{t}_i·log\left(p_i\right)$$

(13)

where $t_i$ is the truth label and $p_i$ is the Softmax probability for each class i from the class set N.

4. Results

Figure 10 shows how the autoencoder loss function changed during subsequent iterations of neural network learning. For the first 10 iterations of the model for training data, the cross-entropy value varies quite a lot from 0.58 to 0.01. After this learning stage, the loss function’s course is smoothed and stabilized at an average level of 0.002. The loss for validation data is less than 0.01 in a complete iterative cycle, and for the waveform stabilization range, it is only 0.001 on average. The model was trained until the cross-entropy value was approximately 0, which occurred in about 50 iterations.

As an indicator of the quality of the prediction model, a parameter called accuracy was chosen, which allows the determination of how often the predictions were equal to the actual labels defining the class of the analyzed image. Figure 11 shows how the accuracy value changed during subsequent iterations of model learning. As with the loss function, the most significant changes were recorded in the first 10 iterations. After this stage, the mileage for test and training data stabilized. For iterations, when the autoencoder loss function was 0, the forecast accuracy was 1.

The results of the prediction algorithm were visualized using a matrix of errors (Figure 12). The error matrix is used for assessing the quality of binary classification, where there were regular and anomalous transfer chute operation classes. Data labeled positive and negative are classified as either a predicted positive class or a predicted negative class. A given originally marked as positive may be mistakenly classified as negative. Due to the effectiveness of network detection equal to unity, there are only two classes in the error table: truly positive (TP) and truly negative (TN). The TP class accounted for 72.45% of the forecasted data, while the TN 27.55%.

The prediction algorithm allows the determination of the location of anomalous pixels on reconstructed images. Heatmaps were plotted based on their location and using the function of detecting maximum values on forecasted images, where colors show the density of outlier pixels. Figure 13 shows the detected anomaly caused by the blockage of an oversized rock block in the zone of free fall of material. Figure 14 shows the detected foreign body as a rock bolt stuck between the transfer box and the feed conveyor structure. The last example (Figure 15) shows the detected complete blockage. Each of the presented cases was positively classified as an anomaly, and the additional marking in the form of a field with a red border indicates only the area of the detected most significant number of outlier pixels.

5. Conclusions

The article presents the possibility of detecting blockages of the conveyor transfer point using the proposed detection algorithm. The best results were obtained by filtering the images with a Gaussian filter and reducing complexity by thresholding and transforming using morphology operators. By pointing out, the image examples used for validation turned out to be a trivial problem for the trained autoencoder algorithm. The effectiveness and loss of learning validation data were more beneficial than training data because the anomalous images contained typical examples of blockages and the presence of solids in the zone of fall and loading of material. The real anomalous states vary widely, and it is impossible to record them all for model validation. The algorithm managed to detect the blockage of transfer through an oversized rock fragment and a rock bolt. Blocks caused by these objects are the most common reason for reduced capacity and can be considered the beginning of a potential blockage.

At this stage, the possibility of using the method proposed in industrial conditions should be considered. Since this method is in the initial stage of development, the detection algorithm should provide a security alert before the state of total blockage occurs. The tested model also detected total blockages when the receiving conveyor loading space was filled with excavated material, causing material spillage. The following stages of research will focus on validating the algorithm based on real-time image processing. In much more diverse operating conditions, there will undoubtedly be a decrease in detection efficiency and the occurrence of false alarm states. The presented algorithm will require modification and taking into account other parameters, e.g., information about the movement of the material.