Ensemble Deep Learning for Wear Particle Image Analysis

Abstract

This technical note focuses on the application of deep learning techniques in the area of lubrication technology and tribology. This paper introduces a novel approach by employing deep learning methodologies to extract features from scanning electron microscopy (SEM) images, which depict wear particles obtained through the extraction and filtration of lubricating oil from a 4-stroke petrol internal combustion engine following varied travel distances. Specifically, this work postulates that the amalgamation of ensemble deep learning, involving the combination of multiple deep learning models, leads to greater accuracy compared to individually trained techniques. To substantiate this hypothesis, a fusion of deep learning methods is implemented, featuring deep convolutional neural network (CNN) architectures including Xception, Inception V3, and MobileNet V2. Through individualized training of each model, accuracies reached 85.93% for MobileNet V2 and 93.75% for Inception V3 and Xception. The major finding of this study is the hybrid ensemble deep learning model, which displayed a superior accuracy of 98.75%. This outcome not only surpasses the performance of the singularly trained models, but also substantiates the viability of the proposed hypothesis. This technical note highlights the effectiveness of utilizing ensemble deep learning methods for extracting wear particle features from SEM images. The demonstrated achievements of the hybrid model strongly support its adoption to improve predictive analytics and gain insights into intricate wear mechanisms across various engineering applications.

1. Introduction

The integration of machine learning (ML) techniques offers the potential to revolutionize lubricant oil or wear particle image analysis, thus potentially contributing to lubrication interval decisions and enhancing equipment longevity and operational efficiency [1]. ML, a subset of artificial intelligence (AI), equips systems with the ability to autonomously learn from data and improve their performance over time [2]. In the area of tribology and lubrication technology, ML holds the promise of analysing intricate datasets derived from real-world operating conditions to derive more accurate and contextually relevant lubrication interval strategies [2,3]. This deviation from rule-based and static approaches to adaptive and data-driven decision making has the potential to mitigate the adverse effects of under- or over-lubrication, resulting in reduced friction, wear, and maintenance costs.

Deep learning (DL), a subset of ML, involves the use of artificial neural networks (ANN) to model and solve intricate problems. Its ability to handle large datasets and capture intricate patterns has led to remarkable advancements in diverse domains. In tribology, DL techniques offer the promise of enhanced predictive capabilities, quicker analysis of complex data, and novel insights into the underlying mechanisms governing friction, wear, and lubrication. Thereby, it is estimated that patterns and relationships between the (micro-) wear particles and the health of, for example, engines, as well as the prediction of the distance travelled by a vehicle can be identified. This might facilitate a more precise detection of wear particles and contaminants, potentially leading to engine damage and the prediction of the remaining useful life (RUL), as well as maintenance scheduling. As such, Hu et al. [4] employed ML to predict the mileage of a vehicle based on the wear particles present in the engine oil. Thereby, the researchers used a support vector machine (SVM) to classify the wear level and then used a linear regression model to predict the mileage with an accuracy of around 90%. Moreover, Sun et al. [5] employed deep learning methods for detecting and classifying wear of tungsten-carbide-copper matrix composites with high accuracy, whereby the algorithms learned from scanning electron microscopy (SEM) images.

Ensemble deep learning involves combining multiple DL models to improve accuracy and reduce overfitting by reducing the variance or errors that may be present in any one model; this has already been successfully employed in other disciplines [6,7]. In ensemble DL, the individual models are typically neural networks that are trained on different subsets of the data or with different configurations. Once the models are trained, the predictions made are combined in various ways to produce the final output. This can be performed using a simple average or weighted average of the individual model predictions, or by using more complex methods such as stacking or boosting. Ensemble DL are increasingly attracting attention, especially in competitions such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Winning models in such competitions often incorporate ensemble techniques due to their ability to improve the generalization ability of models, particularly when training data are limited or noisy. By leveraging the ability of convolution neural networks (CNNs) to extract features from images and classify them accurately, several studies have demonstrated the importance of utilizing this tool to detect relevant features [8,9,10]. Generally, CNNs are useful for image classification problems due to their capability to learn and extract meaningful features from input images automatically [11]. CNNs process images through multiple convolutional layers that enable them to learn different levels of features from input images in a hierarchical manner. Low-level features, such as, edges and corners and high-level features, such as shapes and objects, can be extracted from CNNs more effectively than traditional ML algorithms. Additionally, CNNs can handle the spatial dependencies between pixels in an image that are crucial for recognizing objects and patterns accurately. Overall, the powerful capabilities of CNNs make them an effective tool for image classification, contributing to their widespread use in various applications, such as computer vision, self-driving cars, medical image analysis and many others.

To summarize, ML methods are increasingly being employed in the context of tribology and have the potential to revolutionize wear particle image analysis to correlate features with the components’ health. In this context, this contribution is based on the hypothesis that ensemble deep learning methods can identify relevant features from SEM images of wear particles with higher accuracy than individually trained ML and DL methods, thus representing a prospective tool for identifying patterns and relationships between the wear particles and the components’ health, predicting the RUL and improving maintenance practices. To this end, we employed a SEM image dataset from the wear particles present in the lubricating oil at different conditions of a 4-stroke petrol engine, artificially increased the size of the image collection by data augmentation, and trained an ensemble DL model made up of Inception V3, Xception, and MobileNet V2, as well as trained the three mentioned methods individually and compared their prediction accuracies.

2. Materials and Methods

2.1. Experimental Procedure, Data Acquisition and Augmentation

The experimental data were obtained using a newly bought scooter’s air-cooled and BS IV compliant single-cylinder 4-stroke petrol engine (TVS Motors, Chennai, Tamil Nadu, India) with overhead cam, 109.7 cm³, a max. power of 5.88 kW, a max. torque of 8.4 Nm, and a force of 1755 N. The scooter was regularly operated in the field at speeds of 700–900 min⁻¹ and the distance travelled by the vehicle was tracked through global positioning system (GPS) and odometer readings. For engine lubrication, new and fully formulated SAE 10W-30 lubricating oil was utilized. A 10 ml syringe with a 110 mm-long, 3 mm-diameter tube was put into the lubricating oil tank to collect the lubricant samples (Figure 1a). Oil samples were collected from the engine at regular intervals of 300 km, 600 km, 900 km, and 1200 km (Figure 1b) and wear particle studies were carried out. To this end, oleic acid, acting as a dispersant, was mixed with extracted oil in a ratio of 1:10, ultrasonicated for 30 min to ensure a steady dispersion of wear particles, and then filtered using the filtergram technique (Figure 1c). The employed filtering flask had a 10 mm outlet conduit, a capacity of 250 ml, and a rubber tubing connecting it to the vacuum pump (VE-115N, Value, Zabrze, Poland). The flask’s entrance was sealed with a laboratory rubber stopper with a hole that could be filled with a Buchner funnel containing PTFE filter paper (Nupore, Ghaziabad, Uttar Pradesh, India) with a diameter of 47 mm and a pore size of 2 μm. Following the filtering procedure, the filter paper was removed from the Buchner funnel and dried for an hour in a warm oven (WIST, Palghar, Maharashtra, India) at 35 °C. The wear particles were first removed from the filter paper using conductive carbon adhesive tape and subsequently analyzed using SEM imaging (Supra 55, Carl Zeiss, Oberkochen, Germany). The SEM images, as shown in Figure 1d, were collected using an electron current of 100 nA, an accelerating voltage of 0.02–30 kV, and a working distance of 8.5 mm. The images were then categorized/labelled and stored as *.jpg to create a uniform dataset at a scale of 10 μm. Subsequently, the dataset was transformed into binary images using Mathworks Matlab to enhance interpretability and expanded artificially by data augmentation [9] to yield a total of 400 images (100 per class) through various image transformation techniques, including rotation, shifting, flipping, adding noise, warping, blurring, zooming, etc., using AI [10] to obtain sufficient data for training. The resulting augmented dataset, which is made available under https://github.com/Sangharatna786/SEM-Images.git (accessed on 22 August 2023), was further split into 80% for training and 20% for testing the CNNs (Figure 1e), whereby the objective of the CNN was to correctly classify the wear particles to the engine condition.

2.2. Deep Learning

The employed DL CNNs were composed of artificial neurons in multiple convolution, pooling, as well as fully linked layers and utilized convolution to scale down the SEM images into a more manageable size without losing information. Thereby, the input pictures were run through a number of convolutional layers, each of which applies a different set of filters to the input image to extract key features. These filters were learned during the training process to typically capture simple features, such as edges and corners in the lower layers, and more complex features, like shapes and patterns in the higher layers. Generally, more complex features can be recognized with the growing number of layers. The spatial size of the convolved features could be decreased by the pooling layer, lowering the dimensions allowed to decrease the computational costs of data processing. After the convolutional layer, the output was passed through one or more fully connected layers to perform the classification task [12]. The final output was a probability distribution over the possible classes. Within the scope of this contribution, we employed three different CNN models, namely Inception V3, Xception, and MobileNetV2. These models, which are described in more detail in the following, reflect different advantages in terms of extraction capability, computational efficiency, and model size; these choices align with the specific needs of wear particle feature extraction from SEM images, where diverse particle sizes and complex patterns demand a range of architectural strengths while considering practical deployment and computational demands.

2.2.1. Inception V3

The deep neural network architecture Inception was introduced by Google in 2015 and is intended for tasks requiring picture recognition [13]. Inception V3 (GoogleNet V3) is based on a combination of convolutional layers of different sizes and pooling operations that extract features from the input image at different scales. At the onset of the network, the architecture employs a "stem" module, which comprises a series of convolutional and pooling layers that work together to decrease the spatial dimensions of the input image and increase the number of channels in the feature maps. InceptionV3 also uses a series of “Inception” modules that include multiple parallel convolutional and pooling operations of different sizes and aspect ratios. These operations are concatenated together along the channel dimension, allowing the network to capture features at different scales and resolutions. In addition, Inception V3 uses batch normalization and regularization techniques such as dropout and weight decay to improve the training stability and prevent overfitting. Thus, it is effective at capturing both fine-grained and global features in images due to its multi-scale approach and balance between model size and performance. Inception V3 has attained leading-edge results on various image identification benchmarks. Additionally, the architecture has been utilized as a feature extractor for different vision tasks, such as object detection and segmentation, and has been incorporated into well-known DL frameworks like TensorFlow 2.14.0 and PyTorch 2.1.0 + vu118.

2.2.2. Xception

Xception is a deep neural network architecture proposed by Google in 2016, extending the Inception architecture to use depth-wise separable convolutions in place of standard convolutions [14]. This means a factorization of standard convolutions that split the convolution into two separate operations: a depth-wise convolution, where one filter is applied to each input channel, followed by a point-wise convolution, where the output of the depth-wise convolution is subjected to a linear combination of 1 × 1 filters. This keeps the convolution’s accuracy high while reducing the number of parameters and calculations. The Xception architecture replaces each Inception module with a series of depth-wise separable convolution blocks. Each block comprises a depth-wise convolution layer, followed by a batch normalization layer, a rectified linear unit (ReLU) activation layer, a pointwise convolution layer, another batch normalization layer, another ReLU activation layer, and a skip connection that adds the input to the output of the convolution. These blocks can be stacked to form a deep network that can learn intricate feature representations using fewer parameters and computations than traditional convolutional networks, providing strong feature extraction capabilities, especially when dealing with complex patterns in images.

2.2.3. MobileNetV2

MobileNet is a deep neural network architecture designed by Google in 2018 for mobile and embedded vision applications that require low latency and low power consumption [15]. MobileNetV2 uses a combination of depth-wise separable convolutions and linear bottleneck blocks to reduce the number of parameters and computations required for inference, while increasing the nonlinearity and preserving the information flow, thus maintaining high accuracy on image classification tasks. MobileNetV2 also introduces a new inverted residual structure that improves the accuracy and efficiency of the network. The inverted residual block consists of a linear bottleneck layer, followed by a depth-wise separable convolution and another linear bottleneck layer. The input and output of the block are connected by a shortcut connection that skips the depth-wise separable convolution, similar to the ResNet architecture. MobileNet V2 is significantly smaller and faster compared to models like Inception V3 and Xception. Also, it is a feature extractor that has been pre-trained on the Image Net dataset and may be adjusted for a range of vision tasks, including facial recognition, semantic segmentation, and object detection. MobileNet V2 has been implemented in popular DL frameworks, such as TensorFlow and PyTorch, and has achieved state-of-the-art results on mobile and embedded platforms with limited computational resources.

2.2.4. Transfer Learning and Fine-Tuning

Transfer learning is a technique that involves utilizing pre-trained models (Section 2.2.1, Section 2.2.2 and Section 2.2.3) as the starting point for a new model on a different task [16]. The rationale behind this approach is that the pre-trained model has already learned informative features from a vast dataset and these features can serve as a foundation for learning new features in a related task with less data and computational resources. Fine-tuning is a specific type of transfer learning that entails further training of the pre-trained model on the new task by adjusting the weights of some or all of its layers, whereby the degree of fine-tuning is dependent on the similarity between the initial and new tasks. After transferring pre-trained weights for Inception V3, Xception, and MobileNet V2, the model architectures were adjusted in accordance with the collected dataset. Generic image features were used in the initial layers of the pre-trained models, while domain-specific features were used for training in the following levels. Thereby, a minimum learning rate was applied for the pre-trained models to extract picture characteristics in the first few layers and encourage slow learning in the following ones. According to the chosen test circumstances, fully linked layers of pre-trained networks with 1000 neurons were changed and fixed to six neurons. A detailed specification of the pre-trained CNNs that were finally employed is summarized in

Table 1. Detailed specification of pre-trained networks employed in this study.

Deep Learning Model	Number of Parameters	Depth
Inception V3	23.8 Million	159
Xception V2	22.9 Million	71
MobileNet V2	3.4 Million	53

.

2.2.5. Ensemble Learning

In order to enhance the overall performance, ensemble learning was utilized by combining the outputs of three pre-trained DL models Inception V3, Xception, and MobileNet V2 in accordance with [17]. As depicted in Figure 2, the features obtained from these models were concatenated and passed through a dropout layer with a 0.5 dropout rate, followed by a classification layer. The dropout layer helped to prevent overfitting while reducing computational time.

3. Results and Discussions

The overall test accuracies of Inception V3, Xception, and MobileNet V2 when trained individually were 93.75%, 93.75%, and 85.93%, respectively. Thus, these models already feature superior accuracy compared to other ML approaches, such as SVM, when employed in a comparable scenario [4] (however, it should be noted that the underlying data were different and a direct comparison is not fair). The training (blue) and validation (orange) accuracies, as well as losses over training epochs for the three pre-trained models, are depicted in Figure 3a–f, whereby smooth curves could generally be observed. Furthermore, confusion matrices comparing the predicted and actual classes (i.e., travelled distances) of the testing data in its rows and columns as illustrated in Figure 4a–c were employed to assess the level of prediction of each model. Despite featuring good overall accuracy, the MobileNet V2 featured more than double or even triple the number of misclassifications (12), which indicates a lack of confidence throughout the classification in all four categories (300, 600, 900, and 1200 km), in comparison with Inception V3 (5) and Xception (4).

In comparison to the individually trained DL approaches, the ensemble methods combining the three pre-trained deep neural networks featured a superior accuracy of 98.75%, which points towards a higher generalizability of the technique. This can also be seen in the initially already very high and fast converging training (blue) and validation (orange) accuracies, as well as losses over training epochs as shown in Figure 5a,b. As can be seen from the confusion matrix in Figure 6, the ensemble method only featured one misclassification that occurred in one of the classes (where the vehicle had travelled 600 km) and achieved perfect classification in all other classes. These findings suggest that the image features of these classes were well learned during training. The superiorness can be attributed to the ensemble’s ability to capture a broader range of patterns and relationships within the data. Additionally, the model diversity mitigates the risk of overfitting by preventing it from memorizing the training data. The proposed model employed depth-wise separable convolution layers, which implemented the factorization concept resulting in reduced design dimensions and computational costs. These findings indicate that the proposed model may outperform each model regarding classification accuracy.

4. Conclusions

The increasing integration of ML methodologies within the area of tribology shows great potential, for example, in reshaping decisions pertaining to lubrication intervals. This advancement carries the capacity to significantly augment equipment longevity and amplify operational efficacy. A promising avenue for future research involves scrutinizing wear images to discover meaningful correlations between wear particles, contaminants, and overall component health. In accordance with our investigation, predicated upon the hypothesis that ensemble DL can yield more precise prognostications of pertinent parameters in contrast to individually trained DL convolutional neural networks (CNNs), this technical note aimed to contribute to this trajectory. Leveraging SEM images depicting wear particles sourced from a diverse array of distances covered by an IC engine, our methodology encompassed the utilization of various pre-trained and fine-tuned CNN architectures, namely Inception V3, Xception, and MobileNet V2. These individual models yielded commendable classification accuracies for distance estimation of 93.75%, 93.75%, and 85.93%, respectively. In contrast, the collaborative framework of ensemble learning, harnessing the collective outputs of these three pre-trained DL models, resulted in a remarkable predictive accuracy of 98.75%. Notably, this ensemble model exhibited a substantial reduction of up to 91% in misclassifications, attributable to its inherent capacity to encapsulate a wider spectrum of patterns within the data, all while mitigating overfitting concerns and preserving a commendable level of generalizability. Thus, we postulate that the application of ensemble DL strategies emerges as a sanguine avenue for assessing the condition of lubricating oils by analysing wear particles. This, in turn, has significant implications for prognosticating, for example, the RUL of equipment, as well as refining the landscape of maintenance practices. From a research and understanding point of view, one of the primary drawbacks of ML approaches as used within this study is the lack of interpretability in “black-box” models. They generate results based on complex mathematical operations and patterns that are often difficult to decipher, making it challenging to gain insights into the underlying mechanisms. These models do not incorporate prior domain knowledge or physical principles explicitly, which can result in a disconnect between the extracted features and the actual phenomena being observed. This limitation can hinder the model’s ability to provide accurate explanations or insights. Future research should, therefore, focus on making the models more transparent and interpretable. Yet, the presented approaches already can perform image feature extraction at high speed and scale. It should be emphasized that this technical note sought to demonstrate the applicability of one exemplary use case scenario. However, potential applications are not limited to analyzing wear particles from SEM images, but can be extended to extract features from any sort of images from tribo-technical systems, e.g., for predicting the wear mechanisms or surface conditions from SEM [6] or even optical microscopy images, etc., where we also assume that the presented ensemble deep learning technique features superior accuracy compared to other approaches. To fully exploit the (commercial) potential, the approach should be integrated into actual predictive maintenance systems automotive, aerospace, manufacturing, and energy sectors. Additionally, future work can focus on real-time analysis, user-friendly interfaces, cloud-based solutions, and data integration for a holistic view of equipment health.