Using Deep Learning Networks to Identify Cyber Attacks on Intrusion Detection for In-Vehicle Networks ^†

Abstract

With rapid advancements in in-vehicle network (IVN) technology, the demand for multiple advanced functions and networking in electric vehicles (EVs) has recently increased. To enable various intelligent functions, the electrical system of existing vehicles incorporates a controller area network (CAN) bus system that enables communication among electrical control units (ECUs). In practice, traditional network-based intrusion detection systems (NIDSs) cannot easily identify threats to the CAN bus system. Therefore, it is necessary to develop a new type of NIDS—namely, on-the-move Intrusion Detection System (OMIDS)—to categorise these threats. Accordingly, this paper proposes an intrusion detection model for IVNs, based on the VGG16 classifier deep learning model, to learn attack behaviour characteristics and classify threats. The experimental dataset was provided by the Hacking and Countermeasure Research Lab (HCRL) to validate classification performance for denial of service (DoS), fuzzy attacks, spoofing gear, and RPM in vehicle communications. The proposed classifier’s performance was compared with that of the XBoost ensemble learning scheme to identify threats from in-vehicle networks. In particular, the test cases can detect anomalies in terms of accuracy, precision, recall, and F1-score to ensure detection accuracy and identify false alarm threats. The experimental results show that the classification accuracy of the dataset for HCRL Car-Hacking by the VGG16 and XBoost classifiers (n = 50) reached 97.8241% and 99.9995% for the 5-subcategory classification results on the testing data, respectively.

1. Introduction

With the rapid development of Internet of Vehicles (IoV) technology, electronic devices with in-vehicle networks (IVN) are becoming increasingly connected to the Internet to provide online communication and real-time updates on traffic. However, security and privacy issues, such as remote network intrusions into the controller area network (CAN), are critical concerns for IoV networks. In other words, while equipping existing vehicles with mobility services will create new business opportunities, it will also raise new cyber security threats for car manufacturers and consumers with the advent of the era of ubiquitous network connections and on-demand service such as over-the-air (OTA). Security vulnerabilities for OTA updates have been mitigated by process enhancements via user authentication with a system firmware examination.

According to a research report by Fortune Business Insights, the global electric vehicle (EVs) market size reached USD 246.70 billion in 2020. More than 10 million vehicles throughout the world’s roads in 2020 were battery-powered electric models. The market is anticipated to grow from USD 287.36 billion in 2021 to USD 1318.22 billion in 2028, at a CAGR of 24.3% in the 2021–2028 periods [1]. A series of security incidents targeting vehicular communications have occurred recently using denial of service (DoS), fuzzers, insertion, and spoofing gear/RPM attack techniques. Information security hazards include not only individual hackers highlighting their technical capabilities, but also group attacks aiming to extract financial gain. These attacks by hackers have concerns with car manufacturers. The hackers behind the aforementioned attacks demanded a large ransom from car manufacturers, and threatened to detonate money using implanted malicious codes.

Network communications are widely used in IVNs, which deploy CAN as a protocol for electronic control units (ECUs). ECUs can be used to control the transmission, engine, speed, airbags, powertrain, and many other subsystems of a vehicle. In practice, CAN suffers from multiple security issues, including the lack of data encryption and user authentication in inter- and intra-vehicle communications [2]. Generally, CAN buses are designed to guarantee reliable, rather than secure, communication in IVNs. For example, CAN uses the carrier sense multiple access with collision detection (CSMA/CD) protocol as a network communication, which allows nodes to capture unencrypted messages going through the network by a hacker. Furthermore, the ECUs in the CAN bus are vulnerable to cyberattacks because they lack security features such as user authentication and message encryption. Therefore, the improved CAN bus incorporates the network-based IDS on in-vehicle network system, namely, on-the-move intrusion detection system (OMIDS) into in-vehicle networks to help users detect abnormal connections in a timely manner [3].

The network-based IDS on an in-vehicle network (i.e., OMIDS) plays a role of data collector and data analyzer over the CAN bus that identifies security threats and attacks of in-vehicle network systems by using detection of anomalous CAN bus messages with protocol analyses [4]. Thus, distinct NIDSs on in-vehicle network systems are being developed for detecting threats of the CAN bus network attacks within in-vehicle communication protocol associated with potential attacks exploited, categorising the anomaly messages in securing in-vehicle networks and the related information systems on vehicles [2,3,4,5,6,7].

The concept of IDS deployed to the automotive system was first introduced by Hoppe et al. [5]; later, IDS deployment strategies were discussed by [6,7]. For NIDS to monitor and inspect the communication messages in the CAN network from different sources, it is recommended to be deployed to the central gateway [2]. As shown in Figure 1, the OMIDS was placed at a central gateway that monitors activities in the CAN network and identifies the attacks.

In the identification of possible attacks, OMIDS can be used to collect routing messages on the CAN Bus and identify attack categories for the IVN successfully. Therefore, a number of message records in the CAN bus must be collected, analysed, and classified in real time. Many classification approaches incorporate machine learning (ML) algorithms—such as the naïve Bayes classifier, decision tree, logistic regression (LR), support vector classifier (SVC), and deep convolutional neural networks [7,8,9,10,11,12]—to help managers precisely identify network attacks. In the future, OMIDS may encounter a more diverse range of attacks and extortions.

Most existing ML approaches for detecting cyber-attacks on OMIDS involve cyber-threat analyses that match routing information to potential attack profiles based on behavioural analysis techniques with packet collection, filtering, and feature comparisons. The routing information of CAN devices is used to classify possible attack categories, identify the real attack type, mark the compromised ECU, and initiate countermeasures.

Consequently, deep convolutional neural network techniques, such as recurrent neural networks (RNN) [8], long short-term memory (LSTM) [7], and convolutional neural networks (CNN) [9,10,11,12], have been adopted to help security managers detect complex threats from various sources. Typically, deep convolutional neural networks provide a new approach to improve threat recognition accuracy of network intrusion detection and reduce the false positive rate (FPR). However, a single base classifier does not sufficiently match the data distribution, either with a high bias (low-degree-of-freedom models) or high variance to be robust (high-degree-of-freedom models) [12]. Considering the increasing security threats in the CAN bus systems, the challenges for automatic ML-based intrusion detection on the in-vehicle network systems are summarized as:

(i)

The idea of machine learning-based IDS deployed to the CAN bus network was first introduced by Kang et al. in 2016 [13]. In this case, they used unsupervised pre-training of deep belief networks (DBN) model in detecting any deviations from normal frequencies of CAN message. Later, Taylor et al. used a support vector machine for binary classification to classify the CAN traffic flows [14]. Recently, Hossain et al. (2020) developed an LSTM to detect the threat predict using sequence data inputs and achieved an overall detection accuracy of 99.995% [7].

With the proliferation of attacks on an in-vehicle network, the traditional artificial neural network (ANN) failed to detect in some complex attack cases, such as fuzzy attack in [15]. In this case, Song et al. (2020) used the deep neural network models, deep convolutional neural network (DCNN), and LSTM to achieve high accuracy with low error rate for prediction results.

Theoretically, assembling multiple classifiers can reduce false positives and produce more accurate classification results than single classifiers [16]. For example, Rajadurai and Gandhi used a stacked ensemble model of different base classifiers to build a stronger learner and showed that the stacked ensemble model produced more accurate results than that of a single algorithm [17]. Therefore, ensemble learning-based techniques such as boosting [18], random forest [19], gradient boost DT (GBDT) [20], and XGBoost [21,22,23,24] have been developed to reduce the bias and variance.

(ii)

In a supervised ML model, it requires complete labeled data in the training process. There are difficulties in predicting and generating attack behaviour in evaluating the CAN bus system [2]. Practically, existing signature-based approaches for the NIDS are based on behavioural features to categorize the threats [25]. Importantly, high accuracy of the OMIDS needs continuous updating for high-resolution feature set inputs extracted from attack scenarios of CAN bus, which is not a trivial task. Moreover, it cannot efficiently work if an unknown message is abnormal. Thus, the development of an accurate and robust approach for automatic threat detection for in-vehicle network systems is still a challenge.

In this study, we used a CNN-based scheme called VGG16 [10] (a specific CNN model specialising in image recognition) to handle a diverse range of threat classification problems. The system validation incorporates the XBoost and naïve Bayes classifiers, decision tree, LR, and SVC to compare their performance.

In summary, the primary contributions of this study are as follows:

• For vehicle network security, this study addresses a diverse range of threat classification problems in intrusion detection systems using the VGG16 model to verify performance;
• In our experiment, the accuracy of the VGG16 model for intrusion detection is 100%/100% (Table 9) for binary classification on the training and testing data;
• As shown in Table 10, the proposed VGG16 approach provides higher prediction accuracy (97.9420%/97.8241%) for multiclass classification (five categories) than those of Naïve Bayes (91.0095%/91.0273%) and SVC classifier (91.0095%/91.4137%) on training and testing data;
• To compare the classification accuracy of competing approaches, including the XBoost classifier, Naïve Bayes classifier, DT, LR, and SVC, the test cases are capable of intrusion detection in terms of accuracy, precision, recall, and F1-score;
• In our experiment, the XBoost classifier’s prediction error for intrusion detection decreased most steadily among the four models, followed by the VGG16 model, naive Bayes, decision tree, logistic regression, and SVC;
• The average accuracy of the classification of intrusion detection by the VGG16 and XBoost classifiers (n = 50) for the testing data was 97.8241% and 99.9995%, respectively.

The remainder of this paper is organised as follows: Section 2 reviews other relevant studies in the field. Section 3 introduces the proposed CNN-based model for in-vehicle network security analysis. Section 4 presents the performance analysis and its results. Finally, Section 5 provides concluding remarks.

2. Overview of Intrusion Detection for In-Vehicle Network

This section reviews approaches for addressing threats to in-vehicle networks, and introduces a CNN-based model, VGG16, and an ensemble method, XGBboost, for intrusion detection to categorise possible attacks.

2.1. Potential Vulnerabilities for In-Vehicle Network

The problem of identifying potential threats in an automotive communication system, specifically for the CAN bus protocol, is referred to as the security of the in-vehicle network (SOVN) problem. Typically, the SOVN problem involves a guarantee of data confidentiality, integrity, and availability that can never be breached within the ECUs or CAN bus in the security management of existing vehicles [3]. Solving the SOVN problem is of crucial concern in identifying possible attacks, given a constraint on both the quantity of routing CAN packets collected by OMIDS, and the computational time. Consequently, many detection methods for discovering true IVN attacks have recently been proposed.

Currently, the CAN bus fails to ensure all three essential security levels that would maintain necessary protection from diverse and complex threats. The CAN protocol does not have inherent cryptographic protections to ensure that only authorised parties have access to information. Consequently, it allows intruders to access sensitive user data and invade privacy. Furthermore, the CAN protocol does not feature a comprehensive integrity check, and fails to maintain the accuracy, completeness, and validity of data. Although the CAN bus has a CRC for the verification of integrity against transmission errors, it cannot prevent the injection of data by malicious parties. Given the nature of priority-based messaging in CAMA/CD, if a message with the highest priority is inserted, the network will be inaccessible by the lower-priority nodes, thus violating availability. Therefore, an OMIDS must be established for in-vehicle networks to detect possible threats under certain real-time constraints.

For instance, Miller and Valasek first demonstrated that they can hack into a Ford Escape and Toyota Prius, and control the brakes and steering through the use of physical access to the onboard diagnostics (OBD-II) computer port on each vehicle [26]. Later, Mahaffey and Rogers exploited the keyless flaws of Tesla Model S to remotely unlock the vehicle’s doors, start the vehicle, and drive away by issuing a kill command to shut down the vehicle’s systems, bringing it to a stop [27]. Recently, the Keen Security Lab of Tencent reported that attackers can exploit these flaws to take full control of a Tesla vehicle’s infotainment system without any user interaction. In one experiment, they gained entrance from wireless Wi-Fi/Cellular, compromised of many in-vehicle systems, including firmware on IC, Gateway, and injected malicious messages into the CAN bus to perform a series of operations such as opening the car door, window, and trunk.

These security cases showed the existence of weak points in EVs that may affect the car manufacturers’ reputations with substantial financial implications such as recalls. Safety and security are the highest short- and mid-term challenges in the automotive industry [28]. Therefore, extensive studies have been conducted to find possible solutions to the vulnerabilities of the CAN bus. Some of these studies have performed successful experimental attacks on cars [2,23,24,25,29,30]. Furthermore, researchers have proposed preventative methods for known attacks, including network segmentation, encryption, authentication, and intrusion detection systems (IDSs).

To address the lack of data encryption and user authentication of the CAN protocol in inter- and intra-vehicle communications, many studies on network detection have primarily focused on the use of digital signature mechanisms, such as symmetric secret key techniques, to defend against cyber threats [28]. In practice, digital signatures with pair-wise symmetric secret keys require a high computation time and communication overhead for in-vehicle networks. Because the bandwidth of CAN is limited to 500 kbps in the original design, this is not a practical approach for data encryption and user authentication. However, this approach may be realised for CAN-XL, which will support high transmission speeds of 10 Mbps in the future. However, most current approaches have been conducted with a low communication overhead on the intrusion detection system.

2.2. In-Vehicle Detection of Targeted CAN Bus Attacks

Modern vehicles’ initiatives taken by the automotive manufacturers have increased the number of ECUs per vehicle. For example, there are nearly 70 ECUs deployed in the modern vehicle. With its increasing applications, CAN bus has become a standard choice for automobiles, as well as for other applications too such as EV batteries, planes, ships, machineries, and many more [31]. Practically, modern vehicles often use the CAN bus for communication among their components. From the intrusion reports [4,5,6,7,26,29,30], hackers exploited the vulnerabilities and intruded the in-vehicle network to compromise the targeted ECU of the vehicle and issue attack commands. As shown in Figure 2, an example of DoS attack injected high priority of CAN messages (0x000) in a short cycle from the compromised ECU node thru the use of OTA update and delayed the normal message communications.

Miller and Valasek [26] opened a new era of security analyses for intrusion detection in in-vehicle networks using open intrusion datasets published in 2013. In the experiments, we used well-known open datasets for the targeted intrusion detection of car hacking developed by the Hacking and Countermeasure Research Lab [32]. The datasets include normal flow as well as attack data, with the five major represented attack types being denial-of-service (DoS), fuzzers, insertion, spoofing gear/RPM, and hybrid attacks. Related details regarding attacks and detection methods are found in [3,4,5,6,29,30,31,32]. For instance, attackers often use fuzzy attacks to recognise the reaction of ECUs to certain data packets. In a fuzzy attack, spoofed random CAN ID and data packets are rapidly inserted into the CAN bus, causing network nodes to receive a large number of meaningless messages, which results in vehicle failure. Network attack types on the CAN bus are summarised in

Table 1. Common attacks for in-vehicle network.

	Features	Results/Countermeasures
DoS	Typically, DoS attacks are prepared by injecting high-priority CAN messages in a short cycle.	As the ECU that attempts to send a message with the most dominant CAN ID always wins the bus in the arbitration phase, other ECUs are prevented from transmitting their messages. The agent and SVM were used to improve the detection precision of intrusive attacks for network intrusion detection (NID).
Fuzzers	The fuzzy attack is similar to the DoS attack; however, the CAN ID and data values of messages are entirely random. Generally, fuzzy attacks are used to learn how ECUs react to certain packet types.	Spoofed random CAN ID and data packets are rapidly inserted into the CAN bus, causing failure. IDS detects the fuzzy attacks based on the differences from the insertion attack, because the random ID might not appear benign.
Insertion attacks	The method inserts packets to the CAN bus with an arbitrary ID and data frame. Unlike the fuzzy attack, insertion attack packets feature truly arbitrary IDs and data frame.	The use of insertion attacks might cause the vehicle to malfunction. Indeed, it is difficult to detect insertion attacks due the true arbitration of ID and data frame.
Spoofing gear/RPM	The spoofing attack enabled us to deceive the original ECU and change the RPM (revolution per minute) gauge and drive gear on the instrument panel.	The spoofing attack enabled us to deceive the original ECU and change the RPM gauge and the drive gear on the instrument panel.
Hybrid attacks	This method interleaves benign data with DoS attacks, fuzzers, spoofing gear/RPM, and insertion attacks.	The use of hybrid attacks may cause serious malfunctions in vehicles. In the identification of possible attacks, defenders used IDS on the CAN bus to collect sufficient routing information to identify attack categories successfully in the optimal time.

.

2.3. VGG16

VGG Net is a pre-trained CNN invented by Simonyan and Zisserman from the Visual Geometry Group (VGG) at the University of Oxford in 2014 [33]. This model obtained 92.7% test accuracy for ImageNet [34], winning it the ImageNet 2014 competition. To date, VGG16 is still considered an excellent vision model architecture, and has successfully been used in many real-world applications [10,35,36].

The standard VGG16 architecture consists of 13 convolutional layers, five max-pooling layers, three fully-connected layers (FCLs), and a softmax classifier. The number of filters in the original block was 64, though it was doubled in subsequent blocks until it reached 512 [10]. Thus, VGG16 offers a straightforward and simple architecture that sufficiently categorises cyber threats according to collected feature sets.

Inspired by [33], the VGG16 model parameters were used in the model training phase. In the developed model, the feature vectors were transformed into matrices, which formed VGG16 input images to accurately classify cyberattacks. Specifically, VGG16 uses a cascade of numerous layers of nonlinear processing units for feature extraction and transformation, and exhibits superior results in image recognition applications. The data were then clustered into categories according to classification weights. Consequently, the proposed model can minimise the classification error and maximise the generalisability of learning using convolutional features.

Classification is the most frequently encountered issue in ML. Most existing classification approaches for intrusion detection employ data mining algorithms to categorise cyber threats using feature sets by training a classifier. Classification problems based on convolutional features are illustrated as follows:

Consider a given training dataset $D(x_i,y_i)$, where $x_i$ denotes the number of observations of a sample ($x_i\in R^N,i=1,\dots ,n$), and $y_i$ indicates the class to which the point $x_i$ belongs $y_i,\begin{array}{c}\end{array}i=1,\dots ,n,$ $y_i$ is assigned to each observation $x_i$. Each facial feature $x_i$ is of a dimension that corresponds to the number of propositional variables:

$$f_{\theta }={\{(x_{i,}{y}_i)\}{x}_{i,}\in R^n,y_i\in \{0,1,\dots m\}\}}_{i=1}^n,$$

(1)

Generally, a mapping function $f_{\theta }(x_i)=W·x_i+b$ is defined for classification as follows:

$${\hat{y}}^{(i)}\cong y^{(i)}=f_{\theta }\left(x_i\right)=s\left(W·x_i+b\right){,}^{}$$

(2)

where ${\hat{y}}^{(i)}$ is the final class of the output; $y^{(i)}$ represents the output result of the training process; W represents the convolution kernel of the feature matrix; b is the bias of the feature vector; $f_{\theta }$ represents a mapping function in the CNN; and s represents the softmax function.

2.4. XGBoost Classifier

Most previous studies [2,3,7,8,9] on intrusion detection in in-vehicle networks focus on detecting abnormal CAN messages using a base classifier such as DT, LR, SVC, CNN, or LSTM, as shown in Figure 3.

As shown in Figure 3, a base (individual) classifier in ML models use the selected feature vectors from the training instances to train the model, determine the hyperparameters of the trained model, and predict the possible categories of test data. In the training process, there are important topics for the ML model: (i) feature selection. Feature selection is the process of reducing the number of input variables to develop a predictive model using a selected feature set to improve the performance of the model by reducing the computational costs of modelling [37]. (ii) hyperparameter tuning. Essentially, the prediction performance of the machine learning network model is influenced quite heavily by the choice of hyperparameters, hence it can corporate the grid search optimisation process to improve the searching efficiency in model development [38].

As mentioned above, ensemble learning approach allows the production of better predictive performance compared to a single model [16]. Ensemble learning algorithms, such as extreme gradient boosting (XGBoost), can overcome overfitting in threat data. In particular, XGBoost energises machine learning model performance and computational speed, where decision trees are built in parallel instead of sequentially, as in GBDT.

To overcome overfitting in threat data and increase detection accuracy of normal and intrusive patterns, this paper proposes an improved ensemble-based learning algorithm associated with an XGBoost classifier to identify features from qualified threats.

XGBoost is an efficient implementation of the gradient boosting machine learning algorithm that employs stochastic gradient or tree boosting to create a powerful machine learning technique that performs well on a wide range of challenging problems [21]. It has been shown to provide state-of-the-art results on several standard classification benchmarks. Furthermore, XGBoost is a scalable and highly accurate implementation of gradient boosting that limits the computing power of boosted tree algorithms. In other words, XGBoost provides parallel tree boosting, and is the leading machine-learning library for regression, classification, and ranking problems (Figure 4).

As shown in Figure 2, XGBoost is an ensemble of decision tree algorithm, where new trees fix the errors present in trees that were already part of the model. New trees are generated until no further improvements can be made to the model. XGBoost provides a highly efficient implementation of the stochastic gradient boosting algorithm, as well as access to a suite of model hyperparameters designed to provide control over the model-training process [21]. The most significant factor behind the success of XGBoost is its scalability in all scenarios. The system runs more than ten times faster than existing popular solutions on a single ML machine, and scales to billions of examples in distributed and memory-limited settings.

XGBoost Features

XGBoost improves upon the gradient boosting algorithm, where the Newton–Raphson–Gradient method is used to explore the solution of the loss function by expanding its Taylor series to the second order, as well as adding a regularisation term. The objective function during training consists of two parts: the loss of the gradient boosting algorithm and the regularisation term.

The loss function of the XGBoost model is defined as [21,23]:

$$L\left(t\right)={\displaystyle \sum }_{i=1}^{n}l(y_i^{\prime }{y}_i+f_k\left(x_i\right))+{\displaystyle \sum }_k\mathsf{\Omega }\left(f_k\right),$$

(3)

where $l$ is the loss function, $f_k$ is the kth tree output, and Ω is a normal term. Assuming that L$\left(t\right)$ is a convex function, $y_i^{\prime }$ represents the predicted value for the training sample at time t − 1, and $y_i$ denotes the true value of the training sample at time t. One major approach to ensure fast calculation is the approximation of the Taylor expansion as:

$$L\left(t\right)\approx {\displaystyle \sum }_{i=1}^{n}l(y_i^{\prime }{y}_i)+g_i{f}_k\left(x_i\right)+\frac{1}{2}{h}_i{f}_k{}^2+{\displaystyle \sum }_k^{}\mathsf{\Omega }\left(f_k\right),$$

(4)

where $g_i$ and $h_i$ are the loss function’s first and second derivatives, respectively. The normal term defines the model’s complexity, and can be simplified as

$$\mathsf{\Omega }\left(f_k\right)=\mathsf{\gamma }\mathrm{T}+\frac{1}{2}\mathsf{\lambda }{\left|\left|w\right|\right|}^2,$$

(5)

where γ and λ are the given parameter values, w is the vector formed by the values of all leaf nodes of the decision tree, and T is the number of leaf nodes. XGBoost was developed into an optimised distributed function library for the gradient boosting model, with the purpose of achieving high efficiency, flexibility, and portability.

3. An Analysis Model for Intrusion Detection of an In-Vehicle Network

The proposed network intrusion detection model combines the CNN model with an ensemble learning XGBoost algorithm to maintain high precision in predicting the stability of network intrusion detection after the collection of suspicious network flows. The overall structure of the model is exhibited in Figure 5, which illustrates the three sub-phases in the behaviour classification process: (1) data pre-processing, (2) model training, and (3) model validation.

Step 1. Data Preprocessing

First, the training sample data were obtained from two data sources: the common HCRL intrusion detection archive [32], and the HCRL Car-Hacking dataset, the latter of which were used in several other CAN IDS case studies [39,40]. The HCRL dataset contains 30–40 min of CAN traffic, with approximately 4 million total messages. For our approach, we only evaluated spoofing attacks on the driving gear and RPM gauge from this set. In these attacks, only the messages for a single ID out of approximately 25 were attacked in each case. Therefore, only messages with these IDs were evaluated.

Step 1.1. Experiment Data Sources

In the intrusion detection experiment, the overall HRCL dataset was selected as a comprehensive dataset to examine the performance of the developed XGBoost classifier. The HRCL dataset was compiled by Culture Makers and the Korea Internet and Security Agency for car security to include synthetic contemporary attack behaviours in real-world in-vehicle network traffic.

To extensively examine the developed model’s performance, we reconstructed the experimental dataset to generate five major attack types by comprising a CAN-intrusion dataset and a CAN-hacking dataset. To simplify the amount of training data and improve efficiency of the training process, redundant normal messages were removed and injected messages were extracted.

Step 1.2. Normalisation

All fields in the HRCL dataset were regarded as symbolic features. First, we performed symbol conversion of the network packets. Then, the attack categories of Flag were converted to a numerical format (1–5) according to different attack types, where normal traffic was assigned to ‘0’ through a one-hot encoding process.

Step 2. Model Training Phase

In this step, the experimental data were divided into a training set and a testing set. The training set contained 175,341 records (68.05%), whereas the testing set contained 82,332 records (31.95%). Furthermore, the training set contained 119,341 (68.06%) and 56,000 (31.95%) attack and normal files, respectively, whereas the testing set had 45,332 (55.06%) attack files and 37,000 (44.94%) normal files.

For the model training phase, the VGG16 model was used to accurately categorise cyber threats. In the experiment, the revised VGG16 was trained to detect network intrusion based on the behavioural patterns of collected samples. The softmax function was used to evaluate the model, and a cross-entropy function was used to adjust the learning rate to minimise the classification error and increase the speed of the training process.

Step 3. Model Validation Phase

In the model validation phase, the trained model’s performance was assessed and compared using the XGBoost classifier with Equations (3)–(5). After training the base classifiers, the results were aggregated from randomly selected sub-datasets of the training data; finally, a meta-classifier was trained to perform threat classification of the samples.

In practice, the DLN training process and classification performance of the model are significantly influenced by the choice of hyperparameters. Therefore, we used GridSearchCV to set the model parameters for multi-classification on XGBoost. The learning results were used as a basis for the parameters in the validation phase, including weight matrix, batch size, epochs, and classification accuracy.

4. Experimental Results

All experiments were conducted in Python using the ML library of the scikit-learn package, which is an open-source library for classification algorithms. The software, presented in

Table 2. Experimental environment.

Numerical and Machine Learning Library
Python 3.8.10	Tensor flow 2.1.0
	scikit-learn
	Numpy 1.21.5
	scipy 1.0.1
	Pandas 1.2.0

, was run on an Intel Core i3-4160 dual core CPU clocked at 3.0 Ghz and 8 GB of DDR3 RAM. The operating system was Ubuntu Desktop 20.04.3 LTS, and the database platform was MongoDB 5.0.3.

Step 1. Data Pre-Processing Phase

Step 1.1. Experiment Data Sources

The experimental dataset for HCRL Car-Hacking comprises four major types of network attacks: DoS, fuzzy, spoofing the gear gauze (gear spoofing), and spoofing the RPM gauze (RPM spoofing), as shown in

Table 3. Overview of the experiment HCRLCar-Hacking dataset.

Attack Type	# of Injected Messages	# of Normal Messages	Total
DoS Attack	587,521 (16.03%)	3,078,250 (83.97%)	3,665,771 (100.0%)
Fuzzy Attack	491,847 (12.81%)	3,347,013 (87.19%)	3,838,860 (100.0%)
Gear Spoofing	597,252 (13.44%)	3,845,890 (86.56%)	4,443,142 (100.0%)
RPM Spoofing	654,897 (14.17%)	3,966,805 (85.83%)	4,621,702 (100.0%)

.

We selected the HRCL dataset because it offers three advantages over other benchmark datasets. First, it contains up-to-date behavioural features with existing attack sequences of CAN messages. Furthermore, it includes a set of features from the CAN IDs (header of packets) and DATA to reflect the network packets efficiently. In addition, it contains many complex features that the model can learn to discriminate more accurately, as shown in

Table 4. Details of experiment messages.

Attack Type	No. of Record	Message Details
Normal	14,237,958(85.93%)	Normal CAN messages
DoS	587,521(3.55%)	Inject the message with CAN ID of ‘0x000’
Fuzzy	491,847(2.97%)	Randomly inject deceptive messages with CAN ID and DATA
Gear Spoofing	597,252(3.6%)	Inject the messages of imitating nodes with Arbitrary ID = ‘0x43f’
RPM Spoofing	654,897(3.95%)	Inject the messages of imitating nodes with Arbitrary ID = ‘0x316’
Total	16,569,475(100%)

, which lists the numbers of normal and malicious messages in the experimental datasets. Referring to HRCL attack scenarios, we reconstructed and aggregated four major attack datasets: DoS attack, fuzzy attack, spoofing the drive gear, and spoofing the RPM gauge. Each dataset was generated by simulating ECUs while injecting fabricated CAN messages in a controlled environment. These four basic attack messages were then mixed and grouped for our experimental dataset, which was subsequently divided into training and testing sets.

Then, four CSV files of HCRL Car-Hacking were merged into 16,569,475 messages with a balanced sampling policy to enhance the precision of the multiclass classification model. In other words, the four types of attacks are equivalent to distributing with similar probabilities in the experiment dataset. Relevant data are presented in Table 4.

The CAN message data attributes in Table 4 include the timestamp, CAN ID, DLC, DATA [0], DATA [1], DATA [2], DATA [3], DATA [4], DATA [5], DATA [6], DATA [7], and flag fields which are summarized as

Table 5. Data attributes for CAN message.

Field	Description of CAN Message
Timestamp	recorded time (s)
CAN ID	identifier of CAN message in HEX (ex. 043f)
DLC	number of data bytes, from 0 to 8
DATA [0~7]	data value (byte)
Flag	T or R, where ‘T’ represents injected special attack messages while ‘R’ represents normal messages

. To perform multi-label classification, we specified the class label for each CAN message in the model training data, as shown in Figure 6. Therefore, a set of reduced features with 10 feature sets, including items 2–5, was used for the threat classification of in-vehicle networks:

Timestamp: recorded time (s)

CAN ID: identifier of CAN message in HEX (ex. 043f)

DLC: number of data bytes, from 0 to 8

DATA [0~7]: data value (byte)

Flag: T or R, T represents injected special attack messages while R represents normal messages

The training set contained 13,255,580 records (80.0%), whereas the testing set contained 3,313,598 records (20.0%), including malicious and normal files. The training set had 11,390,366 (85.93%) and 1,865,214 (14.07%) normal and intrusion attack files, respectively. The testing set had 2,847,375 (85.93%) and 466,223 (14.07%) normal and intrusion attack files, respectively.

Step 1.2. Normalisation

All fields in the HRCL dataset were regarded as symbolic features. First, we performed symbol conversion on the network packets. Then, the attack categories of Flag were converted to a numerical format (1–5) according to different attack types, and normal traffic was assigned to ‘0′ through a one-hot encoding process.

Step 1.3. Feature Conversion

In the following, the feature vectors were transformed into feature images (i.e., image matrix transformation), which formed the input images of the VGG16 model to accurately categorise cyber threats in in-vehicle networks. To recognise the feature differences between malicious attacks and normal applications in binary format, we used a feature matrix to represent the behavioural features of traffic connections using the quantile normalisation formula. In statistics, the quantile normalisation process is a technique for equalising the statistical properties of two distributions. Thus, quantile normalisation was implemented to form a feature map for model training. As shown in Figure 7, the feature maps generated by the normal application and malware from the HRCL dataset are significantly different. Therefore, they can be used for image classification.

Step 2. Model Training and Optimization Phase

Inspired by [33], the VGG16 model parameters were used in the model training phase, as listed in

Table 6. Architecture of the revised VGG16 model.

Layer	No. of Filter	Filter Size	Feature Map Generated
Convolution layer C1	64	3 × 3 × 3	150 × 150 × 64
Pooling layer P1	1	2 × 2	75 × 75 × 64
Convolution layer C2	128	3 × 3 × 64	150 × 150 × 128
Pooling layer P2	1	2 × 2	37 × 37 × 128
Convolution layer C3	256	3 × 3 × 128	37 × 37 × 256
Pooling layer P3	1	2 × 2	18 × 18 × 256
Convolution layer C4	512	3 × 3 × 256	18 × 18 × 512
Pooling layer P4	1	2 × 2	9 × 9 × 512
Convolution layer C5	512	3 × 3 × 512	9 × 9 × 512
Pooling layer P6 (Max)	1	2 × 2	4 × 4 × 512
Pooling layer P7 (Avg)	1	2 × 2	512
Dense (256)	-	-	256
Dropout (rate = 0.5)	-	-	256
Classification (Softmax)	-	-	2 for binary classification 5 for multi-classification

. VGG16 is a CNN architecture used to win the ImageNet 2014 competition. To date, it is considered an excellent vision model architecture.

VGG16 comprises 13 convolutional layers, five max-pooling layers, and three fully-connected layers. The number of filters in the first block is 64, and is doubled in subsequent blocks until it reaches 512. As listed in Table 6, each VGG block consists of a sequence of convolutional layers, which are followed by a max-pooling layer. The same kernel size (3 × 3) was applied to all convolutional layers.

In the experiment, the input size is a 150 × 150 × 3 image converted from a set of network features, batch_size = 32, epoch = 20, strides = (1, 1), activation function= ‘tanh’, loss = ‘categorical_crossentropy’, optimiser = ‘dadelta’. The learning results were used as a basis for the validation parameters, including weight matrix, batch size, epochs, and classification accuracy. The experimental results of model training using VGG-16 are shown in Figure 8 and

Table 7. Binary classification accuracy of VGG16 model.

Accuracy (%)	Precision (%)	Recall (%)	F1 Score (%)	Training Time (s)
100.0	100.0	100.0	100.0	389 sec

.

Step 3. Model Validation Phase

To validate the prediction accuracy of the developed model, we applied the XGBoost classifier to ensure that the prediction accuracy of a meta-classifier, such as XGBoost, is superior to that of a base classifier. In practice, it is necessary to first determine the number of trees (component classifiers) to be generated in the XGBoost classifier. Very few practical cases involve more than 300 trees, and such cases may significantly increase the computation time for learning [41]. In this study, GridSearchCV with a 10-fold cross-validation scheme was used to set the model parameters of multi-classification on XGBoost. The search parameters for the XGBoost classifier are listed in

Table 8. Initial parameter search for XGBoost classifier using GridSearchCV.

	Parameter	n-Estimators	Max-Depth	Learning Rate	Optimizer
Model
XGBoost		[50, 100, 150, 200, 250, 300]	[4, 5, 6, 7, 8]	0.0001	Negative Log Likelihood Loss

and Figure 9.

In this study, experiments to determine the appropriate number of trees on the XGBoost classifier were conducted by examining overall accuracy with different numbers of trees ranging from 50 to 300. The low classification error of the XGBoost classifier shown in Figure 10 was obtained with data approximately 50 trees.

From

Table 9. Multi-classification performance with the number of trees selected for the XGBoost classifier.

	Accuracy (%)	Precision (%)	Recall (%)	F1 Score (%)	ROC AUC (%)
n = 50	99.7880	99.9480	98.5739	99.2347	99.7084
n = 100	99.7881	99.9480	98.5745	99.2350	99.7084
n = 150	99.7881	99.9480	98.5745	99.2350	99.7084
n = 200	99.6991	99.9284	97.9733	98.8968	99.9450
n = 250	99.7888	99.9482	98.5790	99.2375	99.7084
n = 300	99.7888	99.9482	98.5790	99.2375	99.7084

, it is seen that there is almost the same performance (i.e., accuracy precision, recall, F1 Score, and ROC AUC) of multi-classification in six experiments (n = 50, 100, 150, 200, 250, 300). Notably, there is a low loss of cost function when training with the training set for 50 trees (n = 50), compared to n =100, 150, 200, 250, 300. Accordingly, the number of trees for the XGBoost classifier was set to 50, to produce the optimal benefit in prediction performance from learning additional trees. The performance of the XGBoost classifier was then evaluated using Equations (3)–(5) by training component classifiers and aggregating their results by randomly selecting sub-datasets of the training data. Finally, a meta-classifier was trained with a majority voting approach to perform threat classification of the samples.

The average accuracy for both VGG16 and XGBoost classifiers (n = 50) was approximately 100.0% (

Table 10. Binary classification accuracy when 10 features were used.

	Training	Testing
VGG16	100.0%	100.0%
Naïve Bayes	100.0%	100.0%
CART	100.0%	100.0%
Logistic Regression	100.0%	100.0%
SVC	100.0%	100.0%
XGBoost classifier	100.0%	100.0%

) for the binary classification results of the training and testing data. The accuracy results (in %) associated with the optimal C and γ values were obtained through a cross-validation scheme for the SVC. The binary classification accuracy rates for the training and testing datasets were approximately 100.0% and 100.0% (C = 1000, γ = 0.1), respectively.

The optimal parameters C = 1000 and γ = 0.1, for SVC were examined using Python GridSearchCVparam = {‘C_range’:(0.1, 10, 100, 1000), ‘Gamma_range’:(0.1, 10, 100, 1000)}, and the fitting error with mean_squared_error was analysed for different values of C and γ. After analysing the experimental results, C = 1000 and γ = 0.1 for SVC were selected. Table 10 shows the same binary classification performance (i.e., accuracy = 100%, recall = 100%, precision = 100%, F1 Score = 100%, and ROC AUC = 100%) for the VGG16 model and the four base classifiers, compared to the XGBoost classifier (n = 50).

Similarly, using the VGG16 and XGBoost classifiers (n = 50) on the testing data, the multiclass classification accuracy levels for the five subcategories decreased to 97.8241% and 99.9995%, respectively, as listed in

Table 11. Multiclass classification accuracy when 10 features were considered.

	Training	Testing
VGG16	97.9420%	97.8241%
Naïve Bayes	91.0095%	91.0273%
CART	99.3259%	99.3170%
Logistic Regression	99.3170%	98.9261%
SVC	91.0095%	91.4137%
XGBoost classifier	99.9997%	99.9995%

.

In addition, the multiclass accuracy levels (%) of naïve Bayes decreased to 91.01% and 91.03% for the testing data. Similarly, the multiclass accuracies (%) of CART decreased to 99.32% on the testing data. Obviously, there is better multi-classification performance for the XGBoost model (n = 50) compared to VGG16 and the four base classifiers (i.e., naïve Bayes, CART, logistic regression, and SVC).

5. Conclusions

This paper presents an intrusion detection model that incorporates an XGBoost classifier with an ensemble of a decision tree algorithm to enhance the precision of a multiclass classification model for in-vehicle network security. Moreover, the proposed approach minimises classification errors using a balanced class with a set of reduced features to accelerate intrusion detection. Overall, the results indicate that the precision of the proposed model for the XGBoost classifier in the intrusion detection analysis of in-vehicle networks is higher than that for the VGG16 model, and those of the four base classifiers, on the HCRL Car-Hacking dataset from Table 11.