A Lightweight Network for Real-Time Rain Streaks and Rain Accumulation Removal from Single Images Captured by AVs

Abstract

In autonomous driving, object detection is considered a base step to many subsequent processes. However, object detection is challenged by loss in visibility caused by rain. Rainfall occurs in two main forms, which are streaks and streaks accumulations. Each degradation type imposes different effect on the captured videos; therefore, they cannot be mitigated in the same way. We propose a lightweight network which mitigates both types of rain degradation in real-time, without negatively affecting the object-detection task. The proposed network consists of two different modules which are used progressively. The first one is a progressive ResNet for rain streaks removal, while the second one is a transmission-guided lightweight network for rain streak accumulation removal. The network has been tested on synthetic and real rainy datasets and has been compared with state-of-the-art (SOTA) networks. Additionally, time performance evaluation has been performed to ensure real-time performance. Finally, the effect of the developed deraining network has been tested on YOLO object-detection network. The proposed network exceeded SOTA by 1.12 dB in PSNR on the average result of multiple synthetic datasets with 2.29× speedup. Finally, it can be observed that the inclusion of different lightweight stages works favorably for real-time applications and could be updated to mitigate different degradation factors such as snow and sun blare.

1. Introduction

Autonomous driving is considered the next-generation transportation breakthrough due to its multiple benefits. According to the World Health Organization (WHO), approximately 1.3 million people die each year as a result of road traffic accidents [1]. Autonomous vehicles (AVs) have the potential to enhance road safety by applying artificial intelligence (AI) techniques in its different tasks [2]. The Society of Automotive Engineers (SAE) ranks autonomous driving from Level 0 (no automation) to Level 5 (full automation under any operational design domains (ODDs), including adverse weather conditions) [3]. Until now, Level 5 AVs have not been introduced to the market; they should be reliable in all challenging conditions in order to gain the public’s confidence that they are worth the investment. Most AV applications are tested in clear weather conditions. However, adverse weather conditions cause sensors (ex: camera, LiDAR, etc.) report inaccurate data and limit their functionality.

Many survey papers have reviewed different algorithms used for object detection in AVs [4,5,6,7,8,9,10,11]. On the other hand, the performance of object-detection algorithms is greatly affected by the presence of challenging weather conditions.

Challenging weather conditions that cause images distortions can be classified into steady and dynamic [12]. Steady weather conditions include fog, mist, and haze. While dynamic weather conditions include rain and snow. Dynamic weather conditions impose greater degradations to the scene’s visibility as the droplets are bigger and of a changing pattern. These conditions reduce scene visibility and contrast, which causes a significant degradation in the process of objects detection performance by AVs.

Rain is the most common dynamic challenging weather condition, especially in the United Kingdom [2]. Even though many researchers have achieved advancements in developing deraining methods, real-time video deraining for AV tasks have been largely understudied.

Rain can be described as the presence of countless drops with different sizes, complex shapes, and varying speeds. Rain causes two different types of visibility degradations. Rain streaks distort the images by generating specular highlights and distorting background scene elements. On the other hand, rain streaks may accumulate generating a hazy effect, similar to fog, hence, reducing the visibility of the scene [13]. Characteristics of rain streaks and rain streaks accumulation are different; therefore, distortions they impose on images are different. Eventually, researchers address them as two separate problems.

In this paper we aim to present a lightweight multi-stage network for single image deraining which mitigates the main two types of rain degradations. Our approach utilizes different progressively stages to reliably restore images affected by different rain degradations while maintaining low inference time. Additionally, extensive experiments are conducted to validate the effectiveness of our network on both synthetic and real rainy datasets. Moreover, despite the existing research towards the development of deraining frameworks, the effect of deraining on object detection was not sufficiently studied especially in the context of autonomous driving; therefore, we also apply object detection for comprehensively evaluating the deraining performance.

The paper is organized as follows. Section 2 we review related deraining work. In Section 3, we introduce and discuss the proposed network. In Section 4, we evaluate the proposed network over different synthetic and real rainy datasets, also we perform sped comparisons between the proposed network and other SOTA networks, experimental results are quantitively and qualitatively presented. In Section 5, we summarized our findings. Moreover, we tabulated used abbreviations in Abbreviations section.

2. Related Work

Deraining problem can be addressed as either a video-based problem or a single image-based one. Video based methods may make use of temporal information present in subsequent frames. On the hand, single image-based deraining is a more challenging task as it lacks the temporal redundancy knowledge and less information is available [14,15]. In order to handle this problem, more research attention has focused on developing different algorithm designs. Moreover, single image-based approaches can be divided into three methods [16]:

• Filter-based [17,18,19,20]: It is based on identifying different rain properties and designing appropriate filters to achieve rain-free images;
• Prior-based [21,22,23,24,25,26]: It uses prior knowledge of rainy and clear images. Prior-based methods includes, but not limited to, morphological component analysis (MCA) [27], histogram of oriented gradients (HOGs) [28], structural similarities, and sparse representation models [26];
• Deep-learning-based models [29,30].

Several approaches have been proposed to achieve rain detection and removal [31,32,33,34,35,36,37]. They have been summarized in

Table 1. Rain detection and removal approaches.

Approach	Image/ Video	Addressed Degradation	Method	Limitation
Liao et al. [31]	Video	Rain drops	They used Hough transformation to detect raindrops in the context of lane scenes, and Sobel filter to detect raindrops in the context of a building scene	It requires the interaction of a human operator to process images
Chen et al. [32]	Video	Rain streaks	They exploited both spatial and temporal information. For dynamic scenes, they applied motion segmentation. Rain removal filters are applied after photometric and chromatic constraints are used for rain detection	Depends on temporal information
Fu et al. [33]	Image	Rain streaks	De-rainNet: They used image processing domain knowledge to improve deraining with a modestly sized CNN	Low speed
Yang et al. [34]	Video	Rain streaks and accumulation	A rain-free frame is estimated from a single rain frame and then taken as a guidance along with previously recovered clean frames	Makes use of previously cleared frames to obtain a more accurate clean one
Li et al. [35]	Video	Rain streaks	Applied a multiscale convolutional sparse coding for rain removal	Low Speed
Yeh et al. [36]	Image	Rain streaks	Frequency division, non-negative matrix factorization	Limited to rain streaks
Chen et al. [37]	Video	Rain streaks	They applied super pixel segmentation to perform scene decomposition into depth consistent units	Limited to rain streaks.

.

However, most of them did not consider AVs system requirements, and suffered from the below listed deficiencies:

• High computation time [33,35,38];
• Only addressed the problem of rain detection;
• Assumes static scenarios [39]; ref. [40]’s approach does not work with highly dynamic raindrops. The approach of [41] fails to separate dynamic textures and moving objects;
• Not applicable for real-time scenarios:
  • The approach of [31] requires the involvement of a human operator to process the captured frames, also it is bound by certain scenes (lane and building scenes);
  • The approach of [32,33] exploits temporal information during the recovery process;
  • The approach of [34] makes use of previously cleaned frames;
  • The approach of [35] has high computational time;
• Limited to certain degradation factors: [36,37,38,42].

2.1. Single Image Deraining

Most of the state-of-the-art algorithms are based on convolutional neural networks (CNNs). Different modules can be used in deraining networks, for example:

• Residual blocks are used in [43] and in [44,45,46,47];
• Dilated convolution is used in [48,49];
• Dense blocks are used in [50];
• Recurrent layers are used in [51,52].

Some approaches used lightweight networks in either a cascaded or Laplacian pyramid manner aiming to enhance the computational efficiency [53,54]; however, this degraded the deraining performance. On the other hand, deep networks are complicated and makes analyzing single modules harder.

2.2. State-of-the-Art Deraining Networks

2.2.1. PRENET

In 2019, Ren et al. [14] provided a simple baseline for deraining networks. They proposed a novel network which repeatedly unfolds a shallow residual network (ResNet). By using a progressive ResNet, they took advantage of recursive computations. Moreover, they introduced recursive layers which exploits dependencies of deep features across stages. Due to its simplicity, this model is suitable as a baseline for future deraining research.

2.2.2. MSPFN

In 2020, Jiang et al. [44] addressed single image rain streak removal. They explored the muti-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework. Recurrent calculations capture global textures of similar rain streaks at different locations.

2.2.3. MPRNET

In 2021, Zamir et al. [55] developed a multi-stage network, which progressively learns restorations for degraded images; therefore, the whole process is broken down into smaller more manageable sub-processes. In addition to sequentially exchanging information from early stages to late stages; lateral connections between feature processing blocks are also deployed to maintain all information. It works on different image restoration tasks such as denoising, deblurring, and deraining. At earlier stages, encoder–decoders are used to extract multi-scale contextualized features. Later stages operate at the original image resolution to generate spatially accurate outputs. Moreover, they used a supervised attention module (SAM) between every two stages which refines features before passing them to another stage.

2.2.4. HINET

In 2021, HINet was designed by Chen et al. [56] as a simple and powerful multi-stage network. It consists of two subnetworks. Each subnetwork is a U-Net [57]. They introduced a novel block called the half instance normalization block (HIN). The developed network surpasses SOTA on multiple image restoration tasks, such as denoising, deblurring, and deraining.

SOTA deraining networks succeeded in achieving deraining of single images; however, their performance was not evaluated on the performance of object-detection process. Some of them can process different image restoration tasks, and not limited to deraining. Moreover, the running time was not the main concern for the reviewed SOTA networks.

To summarize, many deraining networks and approaches have been developed. However, reviewed approaches suffered from different limitations, such as slow running time, limited to a single rain degradation, or the requirement of a human operator. These limitations disable their employment on AVs. Therefore, the main objective of this proposed research is to develop and evaluate a lightweight deraining network that can mitigate different rain degradations in real-time and enhancing the following object-detection process.

3. Materials and Methods

In Figure 1, we elaborate the proposed deraining framework which involves the following main stages: (i) rain streak removal and (ii) rain accumulation removal. Separate training for both stages has been made as the degradation of rain accumulation is multiplicative and its removal will negatively affect the rain streak removal stage causing larger errors. On the other hand, both stages have been jointly implemented in the testing phase.

When rain streaks accumulate, they generate a veiling effect, similar to the degradation fog imposes on image (see Equation (1)) [58]. Therefore, the rain accumulation removal process is similar to defogging. The first step is performing rain streak removal, followed by rain accumulation removal, and finally reapplying rain streaks removal.

$$\begin{gathered} I\left(x\right)=J\left(x\right) \alpha \left(x\right)+A\left(1-\alpha \left(x\right)\right), \\ J(x)=(B+{{\displaystyle \sum }}_{t_r=1}^s{\tilde{s}}_{t_r}° R), \end{gathered}$$

(1)

where $I\left(x\right)$ is the observed hazy image at pixel location $x$; $J$ is scene radiance; $\alpha$ is the transmission map; $A$ is the global atmospheric light; $B$ is the background layer; ${\tilde{s}}_{t_r}$ represents a layer of rain streaks having the same direction; $t_r$ is the index of rain-streak layers; $s$ is the maximum number of rain-streak layers, and $R$ is a region-dependent variable that indicates the locations of individually visible rain streaks; finally, $°$ is an element-wise multiplication operation.

Removal of rain accumulation may lead to strengthening the appearance of existing rain streaks. Therefore, in the proposed organization, another rain streak removal stage was re-applied after rain accumulation removal, as the rain accumulation removal stage will cause the rain streaks that may be missed in the first stage of rain streaks removal to be more apparent. The proposed deraining framework is shown in Figure 1.

3.1. Evaluation Metrics

3.1.1. Evaluation of Detection

The detection methods generate bounding boxes to identify the detected region. There are four classification categories, true positive ($TP$), true negative (TN), false positive ($FP$), false negative ($FN$). Precision can be also called positive predictive values (PPVs), as it represents the ratio of positive results that are $TP$ (Equation (2)). Moreover, recall corresponds to the ratio of positive results that are correctly predicted to be positive (Equation (3))

$$Precision=\frac{TP}{TP+FP},$$

(2)

$$Recall=\frac{TP}{TP+FN},$$

(3)

Moreover, intersection over union (IoU) is an evaluation metric which is used to compare the predicted bounding box (detection output area) with the ground truth annotated area. It is defined in Equation (4):

$$IoU=\frac{A_{gt}{\displaystyle \cap } A_P}{A_{gt}{\displaystyle \cup } A_P},$$

(4)

where $A_{gt}$ is the area of the ground-truth of experts annotation and $A_p$ is the predicted bounding box from the detetcion method.

3.1.2. Evaluation of Deraining

In order to compare deraining performance, two main assessment indices are used based on reference rain-free images: peak-signal-to-noise ratio and structural similarity index (SSIM) [59]. PSNR is the ratio between the maximum power of a signal and the power of the noise distorting this signal. In order to calculate the PSNR of a test image $g$ and a reference image $f$. Both images are of size M × N, the PSNR between $f$ and $g$ is defined by Equation (5).

$$PSNR\left(f,g\right)=10 {\log }_{10}\left(\raisebox{1ex}{${255}^2$}\!\left/ \!\raisebox{-1ex}{$MSE\left(f,g\right)$}\right.\right),$$

(5)

where

$$MSE\left(f,g\right)=\frac{1}{MN}{\displaystyle \sum }_{i=1}^M {\displaystyle \sum }_{j=1}^N{\left(f_{ij}-g_{ij}\right)}^2,$$

(6)

As the mean square error (MSE) approaches zero, the PSNR value approaches infinity; therefore, high PSNR value corresponds to a better image quality, on the other hand, a small PSNR value implies big numerical differences between images.

Another well-known evaluation metric that is used to measure the similarity between two images is the SSIM. It was developed by Wang et al. [60], It is correlated with the quality perception of the human visual system (HVS). SSIM models image distortion in terms of three factors: loss of correlation, luminance distortion, and contrast distortion. It is defined in Equation (7)

$$SSIM\left(f,g\right)=l\left(f,g\right)c\left(f,g\right)s\left(f,g\right),$$

(7)

where$l\left(f,g\right)$ corresponds to the luminance comparison, $c\left(f,g\right)$ corresponds to the contrast comparison, and finally $s\left(f,g\right)$ corresponds to the structure comparison.

However, in order to evaluate the performance of the deraining process on real rainy images, where their corresponding clear images are not available, we used two other different evaluation metrics: the naturalness image quality evaluator (NIQE) [61] and spatial spectral entropy-based quality (SSEQ) [62].

3.2. Architecture

3.2.1. Rain Streaks Removal

The rain streak removal step is based on residual networks (ResNets) [43]. In order to maintain real-time performance and not adding to the computational complexity of the object-detection framework The base architecture of progressive residual networks has been adopted. So, instead of utilizing deeper and complex networks, rain streak removal is performed in multiple stages, such that ResNet is used at each stage. Additionally, in order to avoid the increase in the number of network parameters and the susceptibility of overfitting, recursive computations have been applied by sharing same network parameters with multiple stages.

A basic ResNet has three parts: (i) a convolutional layer $f_{in}$, (ii) multiple ResBlocks $f_{res}$ to extract deep representations, and (iii) a convolutional layer $f_{out}$, as shown in Figure 2. Inference of the PRN at stage $t$ is shown in Equation (2).

$$\begin{gathered} x^{t-0.5}=f_{in}\left(x^{t-1},y\right), \\ {x}^t=f_{out}\left(f_{res}\left(x^{t-0.5}\right)\right), \end{gathered}$$

(8)

Network parameters are reused across different stages, as $f_{in}$, $f_{res}$, and $f_{out}$ are stage invariant. As shown in Equation (2), $f_{in}$ takes the concatenation of the current estimation $x^{t-1}$ and rainy image $y$ as input, the inclusion of $y$ further improves the deraining performance. The progressive network architecture is shown in Figure 3:

• Filter sizes $3\times 3$, padding $1\times 1$;
• $f_{in}$ is a convolutional layer with ReLU non-linearity [63], due to the concatenation of 3 channels from $y$ and another 3 channels from $x^{t-1}$, the convolution of $f_{in}$has 6 input channels and 32 output channels;
• $f_{res}$ includes 5 ResBlocks;
• $f_{out}$is a single-layer convolution, it takes the output of $f_{res}$ with 32 channels as input and outputs a 3-channel RGB image.

3.2.2. Rain Accumulation Removal

Rain streaks and rain accumulation impose different visual distortion effects on captured images. Distant rain streaks accumulate and impose a distortion similar to fog, which also causes visibility degradation and leads to further hiding of image and present objects’ features, thus should also be removed. Since its distortion effect caused by rain accumulation is similar to fog (Equation (1)), then removal of rain accumulation may follow same algorithm (e.g., [13,58,64]).

The rain accumulation removal network is based on a transmission-guided lightweight neural network (TGL-Net) [65]. In this network, instead of producing simulated transmission maps from both depth data and clear images, guided transmission maps are automatically computed using a filter-refined dark-channel-prior (F-DCP) method which facilitates network training on both synthetic and natural images.

The architecture is based on the very deep residual encoder–decoder network (RED-Net) [66] which owns a symmetric architecture and employs skip connections; however, TGL-Net has fewer layers and parameters and is very lightweight. TGL-Net is composed of three stages: (i) downsampling, (ii) encoder–decoder, and (iii) upsampling, as shown in Figure 4 [65].

The downsampling phase includes a convolutional layer ($3\times 3$) then a max pooling layer. The convolutional layer extracts images features, and then the max-pooling layer reduces the total amount by 25 times by using a stride of 5. This leads to improving the computational efficiency of the network.

In the encoder–decoder phase, three convolutional and three deconvolutional layers are connected through a condensed encoder–decoder connection. This stage performs feature extraction and transmission estimation. The convolutional layers extract further image features and also clear noise. On the other hand, the deconvolutional layers recover the details of transmission maps. In the encoder, the sizes of the convolutional kernel are $3\times 3$, $5\times 5$, and $5\times 5$ sequentially, which are equal to the corresponding deconvolutional kernels in the decoder, but in reverse.

The symmetric structure of the convolutional and deconvolutional links retains the benefits from the residual network characteristics [43], including rapid convergence and high accuracy. Using skip connections that directly propagate signals to subsequent layers reduces the common problem of vanishing gradients which commonly happens in deep neural networks (DNNs) [67]. Moreover, using skip connections helps maintain useful image features and details.

After the encoder–decoder layers, another convolutional layer is used, it combines a three-channel feature map into a single-channel transmission map. Additionally, a non-linear sigmoid activation function has been applied, it reaches a saturation region when the input is either too small or too big (i.e., larger than 5 or smaller than −5), which increases the risk of vanishing gradient when adding more layers. However, the employment of ELU units and skip connections in the encoder–decoder layers prevent the vanishing gradient problem. The non-linear sigmoid function outputs a preliminary estimation of a reduced transmission map which is fifth the size of the input image.

In order to enlarge the transmission map to the same size as the input image, the upsampling stage is applied. The upsampling stage is composed of two steps. First, image expansion is performed using bilinear interpolation to maintain the computational efficiency of the network. Second, a convolutional layer is used to further refine the output of the transmission map.

4. Experimental Results

4.1. Experimental Setting

Both rain streak removal and rain accumulation removal stages were trained separately. However, they were tested progressively. The rain streak removal network (PRN) was trained on 1800 images of the Rain100H dataset, while the rain accumulation removal network was trained on a synthetic set of hazy images generated from the NYU dataset and augmented real hazy data generated from the NTIRE 2018 dataset. Both networks used the ADAM algorithm optimizer for training with an initial rate of 0.001. The full proposed network was tested and evaluated using a computer equipped with NVIDIA GTX 1050Ti GPU.

4.1.1. Number of Progressive Stages in the Rain Streak Removal Stage

The charts in Figure 5 show that the effect of the number of PRN stages ($T$) on the PSNR and SSIM evaluation metrics on the Rain100H dataset. It can be noted that the more stages in the network, the higher the values of both PSNR and SSIM. However, when $T$ > 8, both PSNR and SSIM stop improving. Additionally, Figure 6, lists the average running time when tested on the Rain100H dataset on models with stages $T$ = 2, 3, 4, 5, 6, 7, 8, 9, and 10 on a computer equipped with NVIDIA GTX 1050Ti GPU. Considering the trade-off between efficiency and deraining performance, we set $T$to 8 in the following experiments.

4.1.2. Effect of Integrating Different Stages Progressively

Two experiments were made to correctly choose the right organization of the network’s components. The first suggestion was to apply rain accumulation removal as an initial pre-processing step; however, as shown in Figure 7, this has negatively affected the whole deraining process. This is because the rain accumulation removal step alters the quality of rain streaks, it boosts their contrast and appearance, this makes rain streaks look different compared with their original appearance in both real-world and synthetic datasets. Therefore, the second proposed suggestion was to apply rain accumulation removal progressively in combination with rain streak removal, shown in Figure 8. Additionally, the corresponding PSNR and SSIM values for both suggestions are shown in

Table 2. Effect of applying rain accumulation removal as pre-processing vs. applying it progressively.

Approach	PSNR	SSIM
Rain accumulation removal as an initial stage	15.683	0.683
Rain accumulation removal performed progressively with rain streak removal	33.01	0.901

.

4.2. Experimental Results

4.2.1. Evaluation on Synthetic Datasets

Our proposed network is evaluated on five synthetic datasets, i.e., Test100 [68], Rain100H [48], Rain100L [48], Test2800 [42], and Test1200 [50]. Both PSNR and SSIM values have been compared and tabulated in

Table 3. Average PSNR and SSIM comparison on the synthetic datasets Test100, Rain100H, Rain100L, Test2800, and Test1200.

Method	Test100 [68]		Rain100H [48]		Rain100L [48]		Test2800 [42]		Test1200 [50]		Average
	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
HINet [56]	30.29	0.906	30.65	0.894	37.28	0.970	33.91	0.941	33.05	0.919	33.036	0.926
MPRNet [55]	30.27	0.897	30.41	0.890	36.40	0.965	33.64	0.938	32.91	0.916	32.726	0.921
MSPFN [44]	27.05	0.876	28.66	0.860	32.40	0.933	32.82	0.930	32.39	0.916	30.726	0.903
PreNet [14]	24.81	0.851	26.77	0.858	32.44	0.950	31.75	0.916	31.36	0.911	29.426	0.897
PRN [14]	23.51	0.761	28.07	0.884	36.98	0.977	23.79	0.812	19.73	0.709	26.417	0.829
Proposed	32.43	0.930	33.01	0.901	38.21	0.969	33.89	0.932	34.23	0.953	34.154	0.937

. It can be seen that our proposed model achieves remarkable improvement over SOTA deraining algorithms.

Qualitative results on different datasets, shown in Table 3, are shown in Figure 9. Our proposed model exhibits improved restoration performance especially on Test100, Rain100H, and Test1200 datasets, as these datasets include diverse raining conditions including rain streaks and rain streaks accumulation. Moreover, our proposed network showed equally improved performance on Rain100L and Test2800 datasets. Other networks tend to either blur the images by over-smoothing or leave some rain streaks visible in the images.

4.2.2. Evaluation on Real-World Datasets

Additional experiments have been performed on real-world rainy datasets RID and RIS [69] in order to evaluate the generalization capabilities of our proposed model. Moreover, due to the absence of ground truth (clear with no rain) images, no-reference metrics are required to evaluate the performance of deraining (NIQE and SSEQ). Quantitative results are shown in

Table 4. Comparison results of average NIQE/SSEQ on real-world dataset (RID and RIS). The smaller scores indicate better perceptual quality.

Dataset	Evaluation Metrics	HINet	MPRNet	MSPFN	PreNet	Proposed
RID	NIQE ↓	4.985	4.856	6.518	7.007	4.562
	SSEQ ↓	81.162	50.648	40.47	43.04	48.213
RIS	NIQE ↓	6.887	6.045	6.135	6.722	6.148
	SSEQ ↓	52.983	51.689	43.47	48.22	52.681

.

4.2.3. Time Performance Evaluation

Due to the tight real-time constraints of AVs tasks—in our case, object detection—any additional processing should not impose high computational complexity that may affect the real-time performance. Therefore, deraining process should take this into consideration. Moreover, another challenge imposed by the nature of AVs, is that both Avs and the surrounding objects are in a dynamic/moving state. Since computational efficiency is crucial for autonomous driving which requires real-time performance.

Table 5. Comparison of running time (sec) on NVIDIA GTX 1050Ti GPU.

Image Size	HINet	MPRNet	MSPFN	PreNet	Proposed
500 × 500	0.574	0.792	0.415	0.464	0.251
1024 × 1024	2.065	2.782	1.538	1.578	0.843

lists the running time of different methods based on a computer equipped with an NVIDIA GTX 1050Ti GPU. Our proposed model achieves competitive performance compared with other models.

Authors of these SOTA deraining algorithms have evaluated the inference time in their research on different GPUs; therefore, we had to re-evaluate their running times on the same image sizes on a unified computer to produce valid comparisons.

4.2.4. Effect on Object-Detection Performance

Deraining of images captured by AVs in real driving scenarios is considered an effective enhancement of images quality which subsequently enhances the performance of object-detection task performed by AVs, which is the main objective of our work. Therefore, we investigated the effect of applying deraining algorithms, including our proposed network, on the accuracy of object detection.

In order to perform this evaluation, we used samples collected by Jian et al. in [44]. They have collected samples from datasets that include driving scenes with diverse conditions (850 samples from COCO and BDD datasets to create new synthetic rain datasets: COCO350, and BDD350), such as diverse rain orientations and magnitudes, in addition to complex scenes (e.g., night scenes). First, we applied our proposed deraining algorithm, as well as other SOTA deraining algorithms in order to restore rain-free images. Afterwards, we apply the object-detection framework—YOLOv3—to assess the detection efficiency on de-rained images. Quantitative results of both the deraining performance as well as the detection efficiency are shown in

Table 6. Comparison results of joint image deraining and object detection on samples of COCO and BDD datasets.

Methods	Rainy Input	HINet	MPRNet	MSPFN	PreNet	Proposed
Deraining; Dataset COCO350/BDD350
PSNR	14.79/14.13	18.068/15.61	18.01/16.65	18.23/17.85	17.53/16.9	18.879/17.98
SSIM	0.648/0.470	0.786/0.567	0.786/0.779	0.782/0.761	0.765/0.652	0.801/0.823
Object Detection; Algorithm: YOLOv3; Dataset: COCO350/BDD350
Precision (%)	23.03/36.86	32.87/41.52	31.45/40.05	32.56/41.04	31.31/38.66	33.54/41.97
Recall (%)	29.6/42.8	39.54/50.21	38.21/49.32	39.31/50.40	37.92/48.59	40.12/50.67
IoU (%)	55.5/59.85	60.57/62.53	62.54/62.21	62.54/62.21	60.75/61.08	62.34/61.23

. In addition, visual comparisons are shown in Figure 10.

The object-detection accuracy is greatly degraded by the presence of unclear weather conditions, especially rain. Rain drops cover and distort underlying visual features of surrounding objects, which are used by detection methods to classify and localize objects. Figure 10 demonstrates the performance of object detection with and without draining in advance.

Not all deraining algorithms boosts the performance of object detection; however, Hnewa et al. in [70] and Li et al. in [69] have shown that some deraining algorithms [43,51] may degrade the detection performance when compared with directly using the rainy images as input to the corresponding detection frameworks, as they are not optimized towards the end goal of object detection. One factor for this degradation is that the tested deraining algorithms tended to smooth out the images, and hence, distorts important visual features of a scene, and smooths out edges of objects leading to missing detecting them.

Therefore, in our research it is crucial to evaluate the performance of our proposed deraining model by jointly evaluating the object-detection performance on the de-rained output. As shown in Table 6, the detection precision of the produced de-rained images by our proposed model shows noticeable improvement compared with that of original rainy inputs by 68%. Additionally, when compared with other SOTA deraining models, the de-rained images generated by our proposed model shows more dependable contents with clearer features, which efficiently boosts the detection performance.

5. Conclusions

A novel deraining network is presented; it mitigates different rain effects—rain streaks with different directions and intensities and rain accumulations—and targets images captured under rainy driving scenes. The proposed network comprises of three main phases. The first phase performs rain streak removal to remove most visible rain streaks in an image; the second phase is rain accumulation removal which is similar to a defogging operation; lastly, we perform a second rain streak removal operation in order to remove any left rain streaks that could be boosted by the previous rain accumulation removal phase.

The network is based on baseline lightweight network in order to maintain real-time computational performance, as this is a crucial constraint for the performance of deraining networks along with other subsequent operations performed by AVs.

Experiments showed that the proposed network has higher efficiency when compared with SOTA deraining networks, regarding both qualitative and quantitative evaluation metrics, and running time. Moreover, we evaluate the proposed network in addition to the SOTA deraining networks regarding object-detection performance. The presented methodology is considered to be the first to deal with both real-time efficiencies and enhanced performance.

Further research direction includes the investigation of enhancing the model to mitigate more diverse weather conditions such as snow, sun glare, and lightning. Moreover, the proposed network performs removal of both rain streaks and rain streaks accumulation even if only one of these degradations exist. Therefore, another direction to our research is to add initial step of detecting the type of degradation exists in the captured images. Additionally, we plan to test the network on different hardware in order to evaluate the speedup enhancement that could be achieved using more powerful hardware.