Lane Line Detection Based on Object Feature Distillation

Abstract

In order to meet the real-time requirements of the autonomous driving system, the existing method directly up-samples the encoder’s output feature map to pixel-wise prediction, thus neglecting the importance of the decoder for the prediction of detail features. In order to solve this problem, this paper proposes a general lane detection framework based on object feature distillation. Firstly, a decoder with strong feature prediction ability is added to the network using direct up-sampling method. Then, in the network training stage, the prediction results generated by the decoder are regarded as soft targets through knowledge distillation technology, so that the directly up-samples branch can learn more detailed lane information and have a strong feature prediction ability for the decoder. Finally, in the stage of network inference, we only need to use the direct up-sampling branch instead of the forward calculation of the decoder, so compared with the existing model, it can improve the lane detection performance without additional cost. In order to verify the effectiveness of this framework, it is applied to many mainstream lane segmentation methods such as SCNN, DeepLabv1, ResNet, etc. Experimental results show that, under the condition of no additional complexity, the proposed method can obtain higher $F{1}_{Measure}$ on CuLane dataset.

1. Introduction

With the increasing popularity of automobiles, the frequent occurrence of road traffic accidents and the aggravation of traffic congestion are becoming increasingly prominent. According to relevant statistics, about 1.35 million people died in traffic accidents worldwide in 2018, and tens of millions of people were injured or disabled in traffic accidents [1]. The occurrence of road traffic accidents not only causes huge economic losses to individuals, families, and even the whole country [2], but also seriously threatens people’s safety. According to a statistical analysis of traffic accidents by the Ministry of Public Security, about 50% of significant traffic accidents are related to lane line departure. In this context, a lane-keeping system is favored by more and more drivers. In a lane-keeping system, lane line detection is closely related to the realization of lane-keeping function, which directly affects the robustness and real-time performance of the whole system.

In recent years, the advanced driver assistance system (ADAS) and autonomous driving are becoming more important to reducing traffic accidents. As a key technology for intelligent vehicles, lane detection has attracted widespread attention from plenty of institutes and automobile technology companies [3]. Generally, ADAS technology includes tasks such as lane changing (LC), object detection system (ODS), collision avoidance system (CAS), adaptive cruise control (ACC), lane departure warning system (LDWS) [4]. Among the research, vision-based lane detection has always been a hot topic in the field of lane line detection [5]. Lane markings are an essential traffic safety feature with the functions of separating road areas, stipulating the direction of travel, and providing guidance information for pedestrians. In addition, as one of the core elements of an auxiliary environment awareness module, lane detection technology is crucial in the autopilot field [6] and is a powerful tool not only for building high-precision maps, but also for verifying their accuracy [7]. Researchers use different sensors such as radar, lidar, single camera, and stereo camera to obtain environmental data to identify the environment around the vehicle accurately. This paper will focus on vision-based methods because they are cheaper, but can correctly locate local locations in environmental perception for any subsequent lane departure detection and path/trajectory planning. Constantly monitoring the vehicle’s position in the lane lines can be very helpful in avoiding collisions caused by accidental lane deviation, fatigue, or driving under the influence of controlled substances.

Lane line detection becomes very challenging and confusing due to the following factors: (a) The appearance of lane line marking variants, such as dashed or solid lines, white or yellow lines, and road boundaries with less knowledge of road geometry [8,9,10]. (b) The intrusion of nearby obstacles either concealed the lane line markings or was mistakenly regarded as the lane line-markings. (c) The existence and shadow effect of street signs or words are often considered lane lines feature points [11,12]. (d) The change in illumination, which will mutate the color and intensity.

The main contributions of this paper are as follows: (a) Considering the importance of the decoder in recovering the boundary information of the target pixel, the decoder branch is added based on the existing direct up-sampling lane line detection algorithm. It improves the classification and prediction ability of the lane line boundary pixel. (b) Considering the limitation of the decoder in the computational cost, the idea of knowledge distillation is adopted to use the classification probability graph generated by the decoder. It is a soft target to guide the direct up-sampling branch to learn the boundary classification ability of the lane line without increasing the time of the model in the inference stage. (c) The proposed method can be applied to all current lane line detection algorithms using a direct up-sampling structure. In this paper, the generality and effectiveness of the proposed method are verified on the most challenging CuLane dataset by conducting experiments on several mainstream lane line detection algorithms using a direct up-sampling structure.

The remaining part of the research is organized as follows:

• Section 2—Discusses the related work.
• Section 3—Describes the methodology and multi-tasks optimization.
• Section 4—Discusses the CuLane dataset.
• Section 5—Reports the experimental results and analysis.
• Section 6—Provides our conclusions.

2. Related Work

The automatic driving sensing module can obtain the environmental information around the vehicle. One of the most important components of the module is lane line detection, which is the premise to control the safe driving of the automatic driving vehicle within the lane line. Therefore, lane line detection has become a research hotspot in the field of autonomous driving perception. Existing in the lane line detection method can be roughly divided into three categories: traditional image processing method, image processing, and convolutional neural networks (CNN) with the neural network and the lane line detection method based on deep learning.

Among the traditional lane line detection methods, some works of literature [13,14,15] have made a comprehensive summary of it summarized that the traditional lane line detection methods [16] are usually divided into three steps: image pre-processing, local feature extraction, and lane line fitting. Among them, local feature extraction is to capture local lane line information by using edge [17], texture [18], and color [19] features on the region of interest (ROI) [20] in the image, which is a key step of traditional lane line detection methods. However, it needs to analyze the distribution characteristics of lane lines in the image, and manually design and combine feature extraction algorithm. The traditional lane line detection method was selected for specific road scenes. It has been difficult to adapt to the complex scenes in the practical application of autonomous driving, and the design of the extraction algorithm requires a high professional level.

In the method combining traditional image processing and CNN, CNN relies on large-scale datasets [21], and its powerful nonlinear fitting ability, which overcomes the limitations of the traditional lane line detection method and improves the generalization ability of the algorithm. Early lane line detection methods [17,22] based on the convolutional neural network only use a neural network to replace the local feature extraction step in traditional ways, but they still need complex post-processing. For example, Kim et al. [17] combined neural network and RANSAC algorithm to detect lane lines and used CNN to learn lane line features in edge images. The RANSAC algorithm was used to remove outliers and fit the lane lines, but since CNN input is a lane line edge detection image, the accuracy of this method is directly affected by the edge detection algorithm. In literature [23], the top and front view of the lane lines are taken as input, and the dual-view convolutional neural network is used to detect the lane line. However, it requires complex pre-processing of the input image.

The lane line detection method based on deep learning is defined as a dense classification prediction problem. In order to better classification from the supervision of the sparse signal out of the lane line, Pan [24] proposed a spatial convolution neural network (SCNN). SCNN transmits messages between neurons from different directions in the space so that they can capture the spatial relationship between pixels, but it cannot recover the lane line boundary pixels well because of direct up-sampling. Therefore, most lane line segmentation networks [25,26] adopt the encoder–decoder network structure. For example, Kim et al. [27] redefined the point detection problem as a region segmentation problem. SegNet [28] was used to segment the lane lines on both sides in a sequential end-to-end transmission learning method without any post-processing. However, it relies heavily on the lane line segmentation graph as the supervision signal and the training process is very complicated. Zhang et al. [29] use the network structure of two decoders sharing the same encoder to segment the lane line and the driving area and use the link encoder to transfer the lane line information of the two decoders with complementary branches, while the decoder with multi branches will bring huge computing overhead. Literatures [30,31] also adopted the encoder–decoder structure and extended lane line detection to instance segmentation. They need to accurately predict pixel categories and assigned correct instance labels for each pixel. However, they used clustering or complex label allocation strategies, which increased the computational complexity of the model.

To solve the problem that the direct up-sampling algorithm method in the literature as mentioned above cannot recover the lane line features well, and the problem that the network using the encoder–decoder method increases the huge computational cost, this paper proposes an object feature distillation (OFD) framework. It uses a direct up-sampling structure network based on the addition of a decoder branch. It also needs to fuse the features of each stage of the encoder to supplement the detailed information of the target and better restore the classification prediction of the target. Meanwhile, considering the requirement of the autonomous driving system for the reasoning time of lane line detection algorithm, this paper is inspired by the knowledge distilling algorithm [32,33,34]. The classification probability graph generated by the decoder is used as the soft target of the distillation loss function. The classification probability graph of the up-sampling branch is directly used as input to learning the refined classification information of the target in the encoder branch. In addition, only the forward calculation is performed on the direct up-sampling branch in the model inference state. The decoder is not used, which will not bring any additional computational cost.

3. Methodology

Firstly, detail the SCNN architecture along with network details and then show how we fuse object feature distillation to get better results.

3.1. Spatial Convolution Neural Network (SCNN)

The target feature distillation method proposed in this paper is mainly applied to the method of directly up-sampling the feature of the image to the input original image size for pixel-to-pixel classification. The SCNN network is one of the representatives of this kind of lane line segmentation algorithm. As shown in Figure 1, it adds 4 SCNNs in the spatial direction after the top convolution layer (“FC7 layer”) of LargeFOV [35] to introduce spatial message propagation so that it can better capture the spatial relationship between pixels. Its backbone network is the modified VGGNet [36], which better preserves spatial information by removing the largest pooling layer in the 3rd and 4th stages, and the standard convolution is replaced by Dilated Convolution (DC) [37] to obtain a larger receiver field. Finally, the directly up-samples branch samples the feature image up by eight times through bilinear interpolation. The resolution is equal to the size of the input image to obtain a larger receptive field. Another existing prediction branch will pass the $softmax$ operating to the feature image and input it to the average pooling layer and the fully connected layer (FC), the $sigmoid$ function is used to predict whether there is a lane line.

3.2. Knowledge Distillation (KD) Method

Knowledge distillation (KD) is to extract knowledge from complex networks to guide compact network learning, which can significantly improve the performance of compact models and has been extended from image classification tasks to segmentation tasks [38,39,40]. Usually, in the knowledge distillation method, the compact student network takes the intermediate output of the teacher network as the soft target and supervises the student network to learn the knowledge extracted from the teacher network. In the literature [39], the geometric information of target depth predicted by the teacher network is used to guide the semantic segmentation tasks. However, lane lines do not have similar target depth information in natural images. In lane line segmentation, knowledge distillation is used to use the more abstract feature images generated in the later stage as the soft target of the distillation [41], which enables the network to learn richer context information. However, it ignores the shallow local detail information, which is very important for lane line boundary restoration. The target feature distillation method proposed in this paper does not need to train the complex teacher network alone, and each branch shares the feature coding network. We only need to use the classification probability map generated by the encoder branch with lane line boundary thinning information like the soft target of distillation to guide the direct up-sampling branch. In addition, the method in this paper does not require additional annotation of the training dataset.

3.3. Methodology Follow in This Paper

This paper proposes a method of distilling lane line boundary refinement information from the decoder. It can be applied to various algorithms that directly up-samples the feature image to the input image size for lane line segmentation to learn more accurate lane line boundary information from the decoding branch and do not increase the reasoning time of the network. Next, the design details of the overall framework of the proposed method, the structure of the added decoder branch, and the implementation principle of distilling lane line feature information from the decoder are described in detail.

3.3.1. The Overall Structure Design of the Target Characteristic Distillation Network

In this paper, lane line detection is defined as a pixel-level classification problem; that is, lane line features are extracted from the input image through the backbone network. Then, the resolution of the feature image is gradually restored, and the lane line information in the feature image is refined in the decoder. Finally, distilling the classification results of each pixel predicted by the decoder to the direct up-sampling branch to enhance its prediction ability of lane line features. As shown in Figure 2, the whole network is composed of four parts, including 1 encoder and 3 output branches.

In the existing method, the encoder is usually a modified mainstream Convolutional Neural Network (CNN) [42,43]. For example, to preserve the spatial information of the target, VGGNet and ResNet will remove the maximum pooling layer of the last two stages in VGGNet and Resnet or modify the convolution step size to 1 and replace the standard convolution layer with hole convolution with a different void ratio, to obtain a larger receptive field. In addition, “FC6” and “FC7” fully connected convolution layer, as shown in Figure 1, are added. Their output with high-dimensional feature images enables the network to learn more abundant lane line features.

The direct up-sampling branch changes the channel of the output feature image of the backbone network through a $1\times 1$ convolution layer. It then uses bilinear interpolation to directly restore the resolution of the feature image to the size of the input image. However, the resolution of the feature image is continuously reduced in the encoder, and the representation of the feature becomes more and more abstract. Therefore, the lane line detection network using the direct up-sampling method often fails to sufficiently recover the lane line boundary information. Therefore, a decoder branch is added in this paper, which can restore the resolution of the feature image and refine the lane line features through three times of two times up-sampling and convolution operations.

It is predicted that the lane lines have branches that will act on the backbone network. The design details are shown in Figure 1. Firstly, the feature graph output by the decoder is subjected to the $softmax$ operation; then, it is stretched into a one-dimensional vector and sent to the average pooling layer and the two-layer fully connected layer. Finally, the output result of the full connection layer is used as a $sigmoid$ operation input to the binary cross-entropy loss function. The lane line branch prediction results are used for the lane line evaluation stage, and only the segmentation image that predicts the presence of lane lines is obtained for lane line coordinates to reduce the network’s misdetection of non-existent lane lines.

3.3.2. Network Decoder Structure Design of Target Feature Distillation Network

The design of the encoder structure in the lane line detection network determines the ability of the network to extract the target features, and the design of the decoder structure affects the ability to restore and predict the details of the target features. In order to explore the influence of different decoder structures on the feature prediction of recovery lane lines, this paper added decoders to the SCNN network. The decoder is designed with three stages, and its structure is shown in Figure 3a. The output of the encoder is directly refined by three times of convolution layer and two times of bilinear interpolation so that the resolution of the feature image is restored to the original image size. The structure of the 5-stage decoder is shown in Figure 3b. In addition to the above operations, also fully consider the different characteristics of the target feature expression at a different stage of the encoder. The decoder features of the later stage are used to guide the feature learning of the previous stage to recover the prediction of target details better. This decoder design can be applied to various lane line segmentation algorithms using the direct up-sampling method. In addition, when using ResNet as the backbone network, only four stages of decoders are needed. In Figure 3, “×” is used to define that how many convolutional layers we used, and “*” is used to define the convolutional kernel size we use in that specific layer.

3.3.3. Loss Function of the Target Feature Distillation Network

Inspired by using the classification probability map output by the teacher network as the soft target of the pixel-level distillation loss function in the existing research, the classification probability map generated by the decoder branch is distilled in this paper. The direct up-sampling branch learns the lane line boundary information to enhance the detection of lane line boundary. Therefore, the final convolution output of decoder branching is defined as $A\in R^{c\times h\times w}$, where $c$ represents the number of channels of convolution output. $h$ represents the height of the output feature graph; $w$ represents the width of the output feature graph. Similarly, the directly up-samples branch convolution output is defined as $B\in R^{c\times h\times w}$. In order to enable the direct up-sampling branch to learn the pixel-level classification results of the decoder branch, the mean square error loss function is used as the distillation loss function of the two branches, and its definition is shown in Equation (1):

$$l_{dist}\left(A_{ij},B_{ij}\right)=\frac{1}{w\times h}{\displaystyle \sum }_i^h{\displaystyle \sum }_j^w\Vert \phi \left(A_{ij}\right)-\phi {(B_{ij})\Vert }^2$$

(1)

where $\phi (.)$ represents $softmax$ operation. After the convolution output passes through $softmax$, the sum of all pixels in the channel dimension is 1, and any pixel value on the feature image represents the classification probability that the current position belongs to one of the lane lines or background; $\phi \left(A_{ij}\right)$ represents the target value of the decoder distillation loss function, and $\phi \left(B_{ij}\right)$ is the input value of the distillation loss function.

The distance between the classification probabilities of the two branch pixels is measured by Equation (1), and the network minimizes the distance between the input value and the target value through continuous iteration.

The definition of the total loss function of the method in this paper is shown in Equation (2):

$$l_{total}=l_{up}\left(B-\widehat{B}\right)+l_{decoder}\left(A-\widehat{A}\right)+\alpha l_{dist}\left(A,B\right)+\beta l_{exit}\left(E-\widehat{E}\right)$$

(2)

The total loss function is composed of four parts, in which $l_{up}(.)$ and $l_{decoder}(.)$ represent the cross-entropy loss function of the direct up-sampling branch and the decoder branch, respectively. $\widehat{A}$ and $\widehat{B}$ both represent the true annotation of the current input image; $l_{dist}(.)$ represents the decoder information of distillation loss function in Equation (1), which means that the lane line has a branch loss function, using the binary cross loss function. $E$ is the output of the $sigmoid$ function; $\widehat{E}$ represents the true label of whether the lane line exists in the current input image. Finally, parameters α and β were used to balance the distillation task and the presence of lane lines to predict the influence of the task on the performance of the whole network.

4. Dataset

Despite the importance and complexity of traffic lane line detection, current datasets are either too limited or too easy to compare various approaches, and a large public annotated benchmark is needed [13]. KITTI [44] and CamVid [45] both have pixel-level annotations for lane markings, but only hundreds of images, making deep learning approaches ineffective. Caltech Lanes Dataset [16] and TuSimple Benchmark Dataset [46] include 1224 and 7000 photos with annotated lane markings, respectively, in a restricted scenario with light traffic and transparent lanes marking. Furthermore, none of these datasets annotates lane markings that are occluded or unseen due to abrasion. However, such lane markings can be inferred by humans and are extremely useful in real-world applications.

In this paper, we prefer to use the CuLane dataset [24]. Cameras were installed in six different vehicles driven by six different drivers and captured videos while driving in Beijing on various days. More than 55 h of video were collected, yielding 133,235 frames, more than 20 times the size of the TuSimple Dataset. The dataset was divided into 88,880 for preparation, 9675 for validation, and 34,680 for testing. These input images have a resolution of $1640\times 590$. We remove the sky and very close areas from the image and resize accordingly to the lane line network model. Our initial experiments found that traditional photometric and geometric recipes for data augmentation were not providing better driving results, so we do not use augmentation. Figure 4 depicts a variety of scenes, including urban, agricultural, and highway settings. As one of the world’s largest and most congested cities, Beijing presents many daunting traffic scenarios for lane line detection. The dataset was divided into two categories: standard and difficult, which correspond to the nine examples in Figure 4. The proportion of each scenario is shown in Figure 5. The eight difficult scenarios account for the majority of the dataset (72.3%). We manually annotate the traffic lane lines with cubic splines for each frame. As previously mentioned, lane markers are often obscured by automobiles or are not visible. Even in these difficult situations, lane line detection algorithms must estimate lane locations from the background in real-world applications. As a result, we continue to annotate the lanes according to the context in these situations, as shown in Figure 4.

CuLane dataset is available at https://drive.google.com/drive/folders/1mSLgwVTiaUMAb4AVOWwlCD5JcWdrwpvu (accessed on 1 May 2021).

Table 1. Comparison among different lane line datasets.

Dataset	Training	Frame	Sequences	M. Weather	M. Time	Attributes
Caltech Lanes	-	1224	4	Yes	No	2
Road Marking	-	1443	29	Yes	Yes	10
KITTI Road	289	579	-	No	No	2
TuSimple	3626	7000	7000	No	No	2
VPGNet	14,783	21,097	-	Yes	Yes	17
BDD100K	70,000	100,000	100,000	Yes	Yes	11
CuLane	88,880	133,235	55 h Video	Yes	Yes	9

compares and prefers using the CuLane dataset [24] with existing lane marking datasets [47].

5. Experimental Result and Analysis

The training, validation, and testing of the experimental model in this paper are all built by TensorFlow [48] framework and cuDNN [49] kernel used for calculation. Hardware equipment mainly includes a high-performance workstation host. The workstation is configured with an Intel@CoreTM i7-6800K CPU@3.40 GHz, NVIDIA 2080 Ti graphics card.

5.1. Evaluation Standard

To verify the effectiveness of the proposed method, the official evaluation terminology is fully used when evaluating the CuLane dataset. In the literature [25], to make the evaluation more reasonable, both the predicted and the real lane line are labeled as lines with a width of 30 pixels and calculated the intersection over union (IOU) between them. True-positive (TP) cases are selected by setting IOU threshold, which is usually set to 0.3 or 0.5. Then, the $F{1}_{Measure}$ is used to quantify the results of lane line detection, whose definition is shown in Equation (3):

$$F{1}_{Measure}=\left(1+r^2\right)\frac{P\times R}{r^{2}p+R}$$

(3)

Precision (P) is defined as

$$Precision \left(P\right)=\frac{TP}{TP+FP}$$

(4)

where FP represents false-positive cases, and Recall Rate (R) is defined as:

$$Recall \left(R\right)=\frac{TP}{TP+FN}$$

(5)

where FN denote false negative (FN), γ is usually set to 1. In addition, during the test, the coordinates of each lane line need to be determined from the probabilistic map predicted by the network, so the results with branches need to be used. If the existing value of the lane line in the predicted probability map is greater than the threshold value, then the pixel coordinate points of the corresponding lane line are obtained from the probability map for subsequent calculation of $F{1}_{Measure}$ which can effectively reduce the false detection rate.

5.2. Experimental Environment & Network Parameter Setting

This paper uses the standard Stochastic Gradient Descent (SGD) training model on the CuLane data set. The batch size is set as 12, and the basic learning rate is 0.01, the momentum is 0.9. The weight decay is 0.0001. The learning rate strategy adopts “Poly,” and the learning index and the iteration times are set as 0.9 and 60 K, respectively.

The backbone network used in the experiment is VGGNet and ResNet, both of which use the pre-trained weights of the ImageNet dataset and enhance the input data. First, random rotation of [−2°, 2°] is performed on the batch data; then, random cropping is performed, and the cropped image size is equal to 0.05 times the original input image resolution. Finally, the image resolution is adjusted to $288\times 800$ pixels. Through a large number of experiments, α in Equation (2) is set as 10, and β is set as 0.1. All experiments in this paper were carried out under the PyTorch framework, and two NVIDIA 2080 Ti GPUs were used. In addition, in training the network, the gradient of distillation loss during backpropagation does not update the decoder branch parameters; it only affects the weights of the parameters of the direct up-sampling branch.

5.3. Experimental Result and Comparative Analysis

5.3.1. Quantitative Analysis

To verify the effectiveness of the proposed method in this paper and calculate its $F{1}_{Measure}$on the CuLane dataset,

Table 2. Model performance comparison.

Method	Scenario
	Normal	Crowded	Dazzle Light	Shadow	No Line	Arrow	Curve	Crowded	Night	Total
SCNN [24]	90.6	69.7	58.5	66.9	43.4	84.1	64.4	69.7	66.1	71.6
GCJ [29]	89.7	76.5	67.4	65.5	35.1	82.2	63.2	76.5	68.7	73.1
SAD [41]	90.7	70.0	59.9	67.0	43.5	84.4	65.7	70.0	66.3	71.8
Our Smooth	92.1	72.6	61.6	71.7	46.0	86.9	68.9	72.6	69.4	74.1

shows the calculation results of the target feature distillation method applied on the SCNN [24]. In nine complex scenarios of the CuLane dataset, the $F{1}_{Measure}$ of the proposed method in six scenarios, and the whole dataset is ahead of the other algorithms. It verifies the effectiveness of the proposed method without increasing the additional computation cost. However, in-crowd and dazzle light scenario GCJ [29] method because $F{1}_{Measure}$ is higher than the other methods. This is because GCJ method uses the geometric prior information between the lane line and the driving area, which increases the context reasoning ability of the algorithm for occluding lane line. By adding geometric information, the segmentation effect of lane lines is improved, which is the next work of this paper.

As shown in columns 6–9 of

Table 3. Experimental results of the application of target feature distillation method to the main stream lane segmentation method.

Scenario	Methods
	DeepLabv1 [25]	ResNet50 [25]	ResNet101 [25]	SCNN [25]	DeepLabv1 + OFD	ResNet50 + OFD	ResNet101 + OFD	SCNN + OFD
Arrow	74.0	79.0	84.0	84.1	76.8	82.5	86.2	86.8
Cross Road	2060	2505	2183	1990	2445	2229	2004	2222
Crowded	61.0	64.1	68.2	69.7	62.8	67.1	69.8	71.5
Curve	61.0	59.8	65.5	64.4	61.8	64.5	68.5	67.0
Dazzle Light	49.9	54.1	59.8	58.5	55.3	58.6	63.3	64.3
Night	56.9	60.6	65.9	66.1	57.57	63.6	67.1	67.6
No Line	34.0	38.1	41.7	43.4	35.4	40.6	43.8	45.0
Normal	83.1	87.4	90.2	90.6	84.5	89.1	90.9	91.6
Shadow	54.7	60.7	64.6	66.9	64.7	67.6	68.3	73.0
Total	63.2	66.7	70.8	71.6	64.7	69.4	72.2	73.2

, $F{1}_{Measure}$ of the target feature distillation method applied to DeepLabv1, ResNet50, ResNet101, and SCNN are shown, respectively [25]. The results show that the target feature distillation method proposed in this paper can improve the lane line detection performance in each sub-scene and the total dataset, especially in the shadow and dazzle light scene. This method brings significant improvement, which verifies its effectiveness and versatility. In addition, columns 2–5 in Table 3 are the evaluation results of literature [24] on the corresponding lane line detection algorithm. Since there is no lane line labeling in the crossroad sub-scene data, Table 3 only shows its FP measure.

As shown in Figure 6, the improved $F{1}_{Measure}$ of the four groups of comparative experiments in Table 3 in each scene were calculated, and their mean and standard deviation were calculated. It can be seen from the figure that the average value of the improvement is the highest in the Resnet50 method, and it can be found that the network performance of the decoder with a relatively simple structure is improved more obviously, which indicates the importance of the decoder for the prediction of lane line feature recovery, and also proves the effectiveness of the target feature distillation method. Additionally, in the Resnet101 method, the performance of each sub-scene has the closest improvement range, so the standard deviation of its $F{1}_{Measure}$ improvement is the smallest, which may be due to its encoder’s strong feature expression ability, which enables it to extract lane line features in each traffic scene effectively.

5.3.2. Qualitative Analysis

The target feature distillation method was applied to DeepLabv1 and SCNN methods, and lane line detection results before and after application were compared. As shown in Figure 7, column 1 represents the actual label of the input image. The images in column 2 represent the results without using the target feature distillation method. The image in column 3 represents the experimental results after applying the method. By comparing the lane line detection results at the red circle in column 2 image of the DeepLabv1 method, it can be seen that the target feature distillation method can more effectively recover the prediction of lane line pixels in congestion and night scenes. From the red circle in the lower right corner of the SCNN prediction result image, it can be seen that the method presented in this paper can effectively reduce the false detection rate of lane lines on the shadow road without clear lane line identification.

Figure 8 shows lane line detection results after each algorithm applying the proposed method in different complex road scenarios of the CuLane dataset. Column 1 is the original image input, column 2 is the corresponding segmented image, and column 3–5 is the probability graph of lane line segmentation of each algorithm. To better see the probability of lane line classification, the probability diagram of the SCNN network in the fifth column is visualized by a heat map and displayed in the last column. The blue represents the minimum probability value, and the red represents the maximum. From left to right, as the algorithm complexity increases, the lane line boundary segmentation and lane line continuity in the lane line classification probability diagram get better and better. In particular, as the visualization result of line 6 in Figure 8 shows, with the enhancement of the network’s ability to extract lane line features, the detection effect of the blue lane line on the right of the figure gets better and better. In addition, as shown in lines 4–6 in Figure 8, the method proposed in this paper can effectively detect lane lines under harsh conditions with no obvious illumination at night.

As different decoders have different degrees of recovery and refinement to the encoder feature classification prediction results. Various decoder networks are designed to add to the SCNN method, and the influence of lane boundary refinement information on lane detection performance in the distillation is explored. As shown in

Table 4. Influence of different decoder structures on lane line detection performance %.

Method	Total (%)
SCNN	71.6
SCNN (3 Stage)	72.4
SCNN (5 Stage)	73.2
SCNN (Smooth)	74.1

, row 2 shows the experimental results when the decoder with three stages is up-sampled, row 3 shows the results of using a five stage decoder network. Row 4 shows the results after using the smooth decoder network proposed in the literature [50]. Due to the highest complexity of decoder design and channel attention module, the attention weight vector can be learned and generated. By adjusting the channel weight, the classification probability of lane line pixel can be enhanced to improve the lane line detection performance and to distill the information of smooth decoder, the $F{1}_{Measure}$ of the new method is 74.1%, which is 2.5% higher than that of the original SCNN. It also shows the effectiveness of the proposed method.

Since the final output result of the convolutional layer of the network is operated by $softmax$, the classification probability of the pixel belonging to each lane line and background in the channel dimension of the feature map satisfies the probability distribution. Therefore, in the design of distillation loss function and using the mean square error (MSE) to measure the distance of classification probability of each pixel directly. The KL divergence loss function can also measure the distance between the distribution of classification probability map of lane line output from direct up-sampling branch and decoder branch, which measures the distance between the output results of two branches from the perspective of a probability distribution. As shown in

Table 5. Influence of different distillation loss functions on lane line detection performance %.

Method	Total (%)
SCNN	71.6
SCNN_5Stage_MSE	73.2
SCNN_5Stage_KL	73.6

, row 2 is the experimental result of using the mean square error (MSE) loss function as the distillation loss function. Row 3 shows the experimental results using the KL divergence loss function as the distillation loss function, which improves the $F{1}_{Measure}$ on the dataset by 0.4% compared to the results using the mean square error (MSE) as the distillation loss function.

6. Conclusions and Future Work

In this paper, lane detection is defined as the problem at a pixel-level classification. To solve the problem of lane line segmentation without using a decoder to recover the lane line boundary refinement information. By adding the decoder branch, the target of the lane line can be better recovered from the low-resolution feature image. The feature pixel classification prediction and the pixel-level distillation loss function are applied. The direct up-sampling branch learns the lane line segmentation probability map generated by the decoder branch to make up for the lack of lane line boundary refinement information. Therefore, without increasing any computational cost, the network’s ability to detect lane lines is improved. The experimental result shows that the target feature distillation method proposed in this paper improves the $F{1}_{Measure}$ and can better adapt the various types of road scenes. In straight lines, curves, backlit scenes, and vehicle occlusion scenes, the algorithm has good detection accuracy, strong robustness, and higher detection speed. In addition, the prior geometric information between lane lines can effectively improve the performance of lane detection.

In future work, the lane detection and classification results will fuse with a forward-collision warning strategy. It will be helpful for assisted driving in avoiding collision caused by accidental lane deviation, fatigue, or driving under the influence of controlled substances in complex or straight road environments structured.