A Semantic Segmentation Method with Emphasis on the Edges for Automatic Vessel Wall Analysis

Featured Application

Cerebrovascular.

Abstract

To develop a precise semantic segmentation method with an emphasis on the edges for automated segmentation of the arterial vessel wall and plaque based on the convolutional neural network (CNN) in order to facilitate the quantitative assessment of plaque in patients with ischemic stroke. A total of 124 subjects’ MR vessel wall images were used to train, validate, and test the model using deep learning. An end-to-end architecture network that can emphasize the edge information, namely the Edge Vessel Segmentation Network (EVSegNet) for automated segmentation of the arterial vessel wall, is proposed. The EVSegNet network consists of two workflows: one is implemented to achieve finely and multiscale segmentation by combining Dense Upsampling Convolution (DUC) and Hybrid Dilated Convolution (HDC) with different dilation rates modules, and the other utilizes edge information and is fused with another workflow to finally segment the vessel wall. The proposed network demonstrates robust segmentation of the vessel wall and better performance with a Dice (%) of 87.5, compared with the traditional U-net that has a Dice (%) of 81.0 and other U-net-based models on the test dataset. The results suggest that the proposed segmentation method with an emphasis on the edges improves segmentation accuracy effectively and will facilitate the quantitative assessment of atherosclerosis.

1. Introduction

Magnetic resonance (MR) vessel wall imaging (VWI) has been proven to visualize plaques and quantitatively analyze their morphologic and signal characteristics [1]. One of the potential important applications of quantitative morphologic and signal measurements of the arterial vessel wall and plaques is to monitor intracranial atherosclerotic disease progression and regression during medical management, which can greatly assist a comprehensive study of atherosclerosis. Quantitative morphologic measurements require the segmentation of the arterial vessel wall [2]. However, the manual segmentation method can only be performed slice by slice based on the 2D images, which is inefficient and costly, and heavily depends on expert knowledge and experience. Therefore, a fast and precise computer-aided automatic segmentation method is potentially needed.

Most previous studies were based on semi-automatic method. Cheng et al. proposed a Hough transform-based boundary identification method to automatically detect the carotid artery boundary in MR images. The method can realize the analysis of a tremendous number of images and all results are repeatable [3]. Luo et al. presented a modified traditional level set method with a double adaptive threshold (DATLS) to semi-automatically segment the carotid artery on TOFMRA images, which performed well on images with a weak boundary and outperformed traditional level sets in terms of segmentation computing time, robustness, and accuracy [4]. Sakellarios et al. utilized active contours and knowledge-based methodology to segment and reconstruct the lumen and vessel based on the axial TOFMRA and T1W images of the carotid arteries and perform semi-automatic characterization of the plaque type, which can realize the fast and accurate characterization of the plaque type in the carotid MRI images [5]. However, these methods still need extensive manual input, such as seed points initialization, interested artery identification, and registration of image sequences.

Recently, deep learning technology has been applied in arterial vessel and vessel wall segmentation because of its ability to use fully automated methods to solve complex and labor-intensive problems [6,7,8]. These methods utilize the superior learning capability of the neural network and outperform other traditional segmentation methods. Wu et al. developed a deep morphology aided diagnosis (DeepMAD) network for automated segmentation of the vessel wall of the carotid artery, which automatically segmented the lumen and the outer wall together and achieved promising segmentation performances [9]. Li et al. proposed a fully automated tracking-by-detection approach to detect artery locations and segment the vessel wall in the polar coordinate system, which achieved a high accuracy [10]. Alblas et al. presented a multi-task regression in a polar coordinate system method for carotid artery vessel wall segmentation in black-blood MRI using anatomical priors; the results show that it is feasible to automatically obtain anatomically plausible segmentations of the carotid vessel wall with high accuracy [11]. Zhou et al. developed an intracranial vessel wall segmentation method with deep learning using a novel tiered loss function to incorporate class inclusion [12]. However, the arterial vessel wall segmentation based on the MR vessel wall image is still a challenging task due to the relatively low signal-to-noise ratio and contrast between vessel walls and surrounding tissues, which often lead to blurred boundaries and are harder to identify [13]. Specifically, edge information plays a critical role in biomedical image quantification as well as precise segmentation because it often carries abundant physiological information. However, the color, texture, and shapes are processed together by these previous CNN approaches. The diverse type of information related to recognition may lead to unsatisfying results. Disentangling the edge information from the fused features is essential for improving the performance of the CNN architecture.

In this work, we proposed a two-stream deep network architecture entitled EVSegNet, which emphasizes the edges for vessel segmentation. One stream incorporates the Dense Upsampling Convolution (DUC) and Hybrid Dilated Convolution (HDC) for texture feature extraction, which can process multiscale objects. Meanwhile, another stream focuses on the feature extraction of edge information. The outputs of the two streams will be ultimately fused. The proposed EVSegNet was evaluated and compared with the standard U-Net and modified U-Net to validate the effectiveness of vessel wall segmentation based on MR vessel wall images. In summary, the major contribution of this work is that we propose a fully automated vessel wall segmentation used for MR vessel wall images that emphasizes edge information. The paper is organized as follows. The materials and methods are introduced in Section 2. Section 2.1, Section 2.2, Section 2.3, Section 2.4, Section 2.5 describe how to segment the vessel wall. Section 2.6 is implementation. Next, results are given in Section 3. We then discuss the results in Section 4. Finally, the conclusion is given in Section 5.

2. Materials and Methods

2.1. Network Architecture

We proposed the Edge Vessel Segmentation Network (EVSegNet), a highlighted edge convolution network for arterial vessel wall and plaque segmentation. The architecture of the EVSegNet is illustrated in Figure 1. It consists of a regular stream and an edge stream. The regular stream has an encoder-decoder architecture to process texture information. In the encoder, four downsampling convolutional blocks are followed by a bottleneck block. Overall, the downsampling block reduced the spatial dimension down to 1/16 of the input size to extract high-level semantic information. In the decoder, we introduced Dense Upsampling Convolution (DUC) to replace traditional bilinear interpolation upsampling to predict segmentations from features extracted by the encoder. Motivated by Fang et al. [14], the bottleneck block is a hybrid dilated convolution (HDC) module, which is composed of three stacked HDC blocks to expand receptive fields but to not increase the weights number in the filter. The HDC block consists of three convolution layers with kernel size 3 and dilation rates of r = 1, 2, 3, respectively, three normalization layers, and three ReLU activation functions. The edge stream is composed of a sequence of residual blocks followed by an edge gated layer, except for the bottleneck, which was fed into the edge stream as the first residual block input of the edge stream. Each resolution upsampling feature map from the decoder and its corresponding downsampling feature map from the encoder of the regular stream were fused and combined with previous residual block as the edge gated layer inputs. Specifically, the edge information was generated by utilizing the Canny filter. Finally, the output of the edge stream was concatenated with the output of the regular stream. Two streams shared a dual loss function besides their dedicated loss layers.

2.2. Dense Upsampling Convolution (DUC) Module

Traditional upsampling methods utilize bilinear interpolation upsampling. Due to the unlearnably of bilinear interpolation upsampling parameters, it is likely to miss the essential low-level information and hard to carry out the recovery by common upsampling operations. To resolve this problem, we introduced the DUC module, which can capture and compensate for the fine-detailed information that is commonly missed in the bilinear interpolation operation to generate a dense pixel-wise prediction map and further final prediction.

As shown in Figure 2, we performed the DUC on the h × w × c feature map from the output of the encoder part, the original size of which is the size of the input image with H × W × C. We set the downsampling factor as d, so h = H/d, w = W/d. Then, we utilized a 1 × 1 convolutional layer to adjust the channel number c to d² × L in order to obtain the dimension h × w × d² × L of the feature map, where L represents the class numbers in segmentation. Subsequently, the dimension h × w × d² × L feature map is reshaped to H × W × L (here, our L is 1, d is 2). In particular, the main idea of DUC is to divide the label map into the number of d² subparts that are equal in size to the input feature map where the length is the same as the width. Thus, DUC can make up for the loss of detail information in the length and width of the feature map through the channel dimension.

2.3. Hybrid Dilated Convolution (HDC) Module

Recent researches have demonstrated that combining with dilated convolutions is a promising method for image analysis tasks [15,16]. Dilated convolution replacing pooling operation, allows maintaining receptive fields without extra parameters. One problem that exists in the dilated convolution framework with a fixed dilation rate may cause “gridding artifacts”. The use of large dilation rate information may only be beneficial for some large regions, but it may have disadvantages for small areas. As can be seen from Figure 3, when dilation rate (r) = 2, kernel size (k) = 3, not all the pixels in the receptive field are involved in the computation, leaving some holes that did not cover the entire receptive field. Therefore, the information will be lost in the places where these holes are in the image. However, these holes are the places on the image where small information exists. If the dilated rate is larger, the number of pixels that take part in computation will be smaller. For instance, if k = 3 and r = 2, only 9 of 25 pixels are involved in the convolutional operation. Here, inspired by Wang [17], we adopt the HDC block. The HDC model is built by stacking the dilated convolution kernels with different rates. As can be seen in Figure 4, instead of using the same dilation rate, HDC employs different dilation rates for the dilated convolution layer. It can effectively solve the information loss and precision reduction problem caused by the holes in the dilated convolution kernels because it can make a series of convolution operations to cover the area completely without holes when expanding the receptive field. Figure 5 depicts the detail of the HDC block; we can see that the HDC block consists of three convolution layers with kernel size 3 and dilation rates of r = 1, 2, 3, respectively, three batch normalization layers, and three ReLU activations.

2.4. Edge-Gated Layer

Gated convolution solves the problem of traditional convolutional methods that treat all the pixels as equally important; the gated convolution layer (GCL) is capable of filtering irrelevant information and focus on partial information by providing a selection mechanism. Motivated by this, we introduced a gated layer to our work. The edge gated layer receives the inputs from its previous gated layer outputs with the corresponding resolution feature maps of the main stream. GCL enables collaboration between two streams. The regular stream has a high ability to extract high-level semantic information; GCL deactivates the features from each resolution feature map of the regular stream. In this work, edge ground truth was generated by leveraging a Canny filter [18,19] to the original vessel wall images’ ground truth masks.

Let i be the number of convolutional layers, i ∈ 0, 1, …, j. Where $r_i$ and $e_i$ represent the inputs from resolution i of the regular stream and edge stream, respectively. We obtained an attention map ${\alpha }_i\in {ℝ}^{H\times W}$ by feeding $r_i$ and $e_i$ into a 1 × 1 convolutional layer, $C_{1\times 1}$, then concatenating them and passing them into a rectified linear unit (ReLU) [20] activation function.

$${\alpha }_i=\sigma \left(ReLU\left(C_{1\times 1}\left(e_i\right)+C_{1\times 1}\left(r_i\right)\right)\right)$$

(1)

As can be seen in the following, GCL is operated on a shape stream, where $e_i$ pixel-wise multiplies the attention map ${\alpha }_i$ followed by a residual block. Thus, the output is computed as:

$$e_{out}=e_i \odot {\alpha }_i+e_i$$

(2)

In our work, there will be the same number of edge-gated layers as the number of resolutions in the regular stream.

2.5. Loss Function

Semantic loss: The area of the vessel wall only accounts for 0.45–30.4% of the total area. Therefore, there is an imbalance between the foreground and background of our vessel wall images. We defined the foreground and background as easily classified samples and difficultly classified samples, respectively. The Focal loss proposed by Lin [21] is evolved from the cross entropy (CE) loss [22] to address the category imbalance problem. In this work, we introduce the focal loss as the semantic loss. Focal loss is defined as following:

$$FL\left(p_t\right)=-{\alpha }_t{\left(1-p_t\right)}^{\gamma }{\mathit{log}}_e\left(p_t\right)(\gamma \ge 0)$$

(3)

Focal loss introduces a modulating factor ${\left(1-p_t\right)}^{\gamma }$ that is compared with CE; its purpose is to reduce the weight of the background so that the model can focus more on foreground samples during training. When $p_t$ >> 0.5, the contribution from the background to the loss is downweighted. The hyperparameter γ can be used to tune the weight of different samples. As γ increases, fewer easily classified samples will be involved in the training loss computation. Intuitively, when γ is 0, the focal loss is the same as the BCE loss. It balances easy and difficult samples by tuning α. Focal loss function with α = 0.25 and γ = 0.5.

Edge loss: We supervised the loss of edge stream in the vessel wall image by adding an auxiliary loss term that encourages the prediction of edges. Using the focal loss along with the BCE loss to supervise the edge stream, as follows:

$$L_{Edge}={\lambda }_1{L}_{FL}+{\lambda }_2{L}_{BCE}$$

(4)

Here ${\lambda }_1$, ${\lambda }_2$ are hyper-parameters.

Dual loss: The Dual loss is defined as follows:

$$L_{dual}={{\displaystyle \sum }}_{i\in p^{+}}\begin{array}{c}(\parallel \nabla \left(\mathit{arg}\mathit{max}\left(P\left(y_{pre,j}=1|p;c\right)\right)\parallel \right)\\ -\parallel \nabla \left(y_{gt,i}\right)\parallel )\end{array}$$

(5)

where $y_{pre,j}$ denotes the prediction result of the regular stream, c represents the segmentation class. However, the function argmax is non-differentiable, so we use Gumbel softmax to back-propagate. Thus, we approximate the argmax operator as follows:

$$\frac{\partial \mathit{arg}ma{x}_{t}P\left(y^t\right)}{{\partial }_{{\eta }_i}}={\nabla }_{{\eta }_i}\frac{e^{\left({\mathit{log}}_{e}P\left(y_t\right)+g_t\right)/\tau }}{e^{\left({\mathit{log}}_{e}P\left(y_j\right)+g_j\right)/\tau }}$$

(6)

where τ is a hyper-parameter, $g_j$ denotes the Gumbel density function.

2.6. Implementation

2.6.1. Image Acquisition

We collected our MR vessel wall images from 124 subjects on a 3-T whole-body MR system (uMR780, United Imaging Healthcare Co., Ltd., Shanghai, China) equipped with a 32-channel head coil using a T1-weighted 3D MATRIX sequence. The parameters of the imaging are repetition time (TR)/echo time (TE) = 800/13.92 ms, field of view = 230 × 192 × 154 mm³, matrix size = 384 × 320 × 256, spatial resolution = 0.6 × 0.6 × 0.6 mm³. With this, the entire scan is completed within 5 min.

2.6.2. Image Preprocessing

The acquired 3D MR vessel wall images were transferred to a dedicated plaque analysis software (uWS PlaqueTool, United Imaging Healthcare Co., Ltd., Shanghai, China) for image preprocessing. We obtained the Contrast-enhanced MR angiography (CE-MRA) for a clear morphological structure of the blood vessels to extract the blood vessel centerline. Therefore, through the registration relationship, we mapped the centerline of the CE-MRA to the MRVWI to obtain the centerline of the MR vessel wall images.

According to the centerline of the MRVWI to the Curved multi-planar reformation (MPR), we reconstructed the entire vessel by straightening the image of the curved vessel wall and displaying it in a two-dimensional plane on a curved surface to obtain the cross-sectional 2D image. The total process was completed automatically by the system.

Then, the cross-sectional 2D images were manually delineated from the outer wall contours by five experienced radiologists with more than six years of experience.

2.6.3. Experimental Settings

We split 124 patients (13,962 slices) into 75 patients for training (8377 slices); 37 patients for validation (4189 slices), and 12 patients for testing (1396 slices). To avoid model overfitting, the training dataset was expanded by five times through rotation, translation, and padding, using Kaiming as initialization function. The parametric rectified linear units (PReLU) function was selected as the activation function. The SGD algorithm was used to optimize the network with a momentum of 0.9. We set a weight decay of 0.0001, a batch size of 32, and 20 epochs. We set the initial learning rate as 3 × 10⁻⁵. We performed the segmentation network by implementation with PyTorch on a computer with Intel i7-10700 CPU @ 2.9 GHz and an Nvidia GeForce TX 2080Ti 11 GB GPU.

2.6.4. Evaluation and Metrics

We employed five commonly used metrics to quantitatively compare different methods for vessel wall segmentation: Dice coefficient (denoted as Dice, %), recall, precision, accuracy, and mIoU.

2.6.5. Ablation Analysis of Our EVSegNet

In order to justify the effectiveness of the principal components of our network, i.e., DUC, HDC, and edge stream in the proposed EVSegNet, we used Dice, recall, precision, and accuracy as evaluation matrices. The U-net was compared in the testing dataset with an approach involving additional components including Resnet, DUC, HDC, and edge stream for further evaluations.

3. Results

Figure 6 shows the segmentation results by the proposed ESegNet from the test dataset as well as their corresponding original image, mask, binary mask, and edgemap. Edgemaps are generated online by computing the Canny filter. As can be seen from Figure 6, our examples included multiscale vessel walls and low-contrast images.

The compared results segmentation outputs with different backbone architecture, U-net (None), U-net (Res), U-net (DUC), U-net (DUC + HDC), and EVSegNet, respectively, and are illustrated in

Table 1. The Comparison of Segmentation Results of Different Backbone Architecture.

Backbone	U-Net (None)	U-Net (Res)	U-Net (DUC)	U-Net (DUC + HDC)	EVSegNet
Dice coefficient %	81.0 (2.5)	84.5 (1.2)	85.6 (1.4)	86.5 (1.2)	87.5 (1.4)
Accuracy %	96.5 (0.3)	96.8 (0.3)	97.3 (0.4)	97.4 (0.4)	97.6 (0.3)
Recall %	78.6 (2.3)	83.8 (1.7)	85.2 (1.7)	86.1 (1.7)	86.8 (1.7)
Precision %	88.4 (2.9)	89.0 (1.7)	90.9 (1.5)	91.3 (1.2)	91.5 (1.1)

. Table 1 shows the comparison results of the four metrics in our method with different components of our collected dataset. As shown in Table 1, the baseline U-net achieved a Dice, accuracy, recall, and precision score of 81.0, 96.5, 78.6, and 88.4, respectively, for the vessel wall. The performance kept improving when successively adding Resnet, DUC, HDC, and edgestream modules. As a result, our finalized network included DUC, HDC, and edgestream for image segmentation. Compared to the other backbone architecture, our EVSegNet has larger Dice, accuracy, the most stable recall, and great precision scores. For the test dataset, EVSegNet achieved the best performance with a Dice of 87.5 for the vessel wall, followed by U-net (None), U-net (Res), U-net (DUC), U-net (DUC + HDC), achieving a Dice of 81.0, 84.5, 85.6, and 86.5, respectively. Figure 7 shows the violin plot of segmentation performance evaluation of different methods. As can be seen from the figure, the wider violin shape regions represent more aggregated values than those in the narrower zones. EVSegNet aggregates places with values larger than those aggregated by other methods.

Ablation study for adopting the DUC block: Researchers have shown the DUC leads to better performance. Therefore, there is a concern that combining the DUC module might improve our network performance. The proposed EVSegNet employs the DUC block to replace the traditional bilinear interpolation for preserving the learning capability when upsampling. Table 1 shows the segmentation results; our model performance can be improved by adopting the dense upsampling convolution. For outer vessel wall segmentation, the Dice, accuracy, recall, and precision scores are increased from 81.0, 96.5, 78.6, and 88.4 to 85.6, 97.3, 85.2, and 90.9, respectively. The results indicated that the DUC block is beneficial, which can help promote the model’s performance.

Ablation study for HDC block: We employed the HDC block with different dilation rates to replace the original bottleneck block of U-Net aiming at extracting multiscale features from the MR vessel wall images. As depicted in Table 1, our model performance can be improved by adopting HDC. Our proposed HDC module (referred to U-Net + DUC) improves the Dice, accuracy, recall, and precision scores are increased from 85.6, 97.3, 85.2, and 90.9 to 86.5, 97.4, 86.1, and 91.3, respectively. Figure 8 shows how the Dice curves change of 11 epochs for different models for the test dataset. From Figure 8, it can be seen that the HDC model has a higher convergence speed than the traditional CNN model in terms the performance index of Dice and has a better performance. We validate that HDC contributes to the performance of the model and accelerate convergence.

Ablation study for edge stream: Motivated by Takikawa et al. [23], the proposed EVSegNet combines an additional edge stream that could supply the edge knowledge information before segmenting for extracting edge information. Table 1 also shows the effect of the edge stream, which boosts the performance of the MR vessel wall imaging segmentation. Compared to the U-net + DUC + HDC, the Dice, accuracy, recall, and precision scores are increased from 86.5, 97.4, 86.1, and 91.3 to 87.5, 97.6, 86.8, and 91.5, respectively. Experimental results showed that our method was effective and presented superior performance over other methods. It serves to improve the performance of the model noticeably.

Furthermore, we also give a few examples for visual comparison in Figure 9 to show the segmentation results from 12 patients: the first column is the original image, and the second is the corresponding binary mask for references. The following columns are vessel wall segmentation images predicted by U-net (None), U-net (Res), U-net (DUC), U-net (DUC + HDC), and EVSegNet. Their corresponding evaluation results of Dice and mIoU are shown in

Table 2. Evaluation Results of the Visualization Examples with Different Backbones (mean (std)).

		Row1	Row2	Row3	Row4	Row5
U-net	Dice %	80.4	82.1	80.6	80.7	83.3
	mIoU %	68.9	72.3	69.6	69.6	76.7
U-net (Res)	Dice %	81.0	85.3	83.4	84.0	86.2
	mIoU %	71.1	79.7	76.7	77.5	80.9
U-net (DUC)	Dice %	82.2	86.8	84.5	84.5	87.4
	mIoU %	72.4	81.4	77.8	77.8	82.0
U-net (DUC + HDC)	Dice %	83.6	87.7	85.5	86.1	88.2
	mIoU %	75.6	81.8	79.6	80.7	82.5
EVSegNet	Dice %	85.7	91.5	86.1	86.9	90.4
	mIoU %	80.3	85.3	80.8	81.6	83.1

. In particular, as can be seen from the second row and fifth row in Figure 9, all backbones do well in large-sized vessel wall segmentation. Meanwhile, in the first, third, and fourth row, the performance of U-Net is significantly lower than that of the other backbone networks and suffered from unsatisfactory segmentation accuracy due to the small vessel wall. The U-net is not precise enough to discriminate between the foreground and background, which leads to the edge being non-closed. After adding the HDC module, the performance of small vessel walls has been improved. This indicates that thanks to the advantages of employing the HDC module, we were able to obtain a better and more stable performance for multiscale vessel walls. In addition, for the first and third row image, there is low contrast surrounding the boundaries in the neighboring tissues that were essentially invisible to naked eyes; it is observed that EVSegNet was able to locate the vessel wall region and produce the cleanest and most accurate segmentation results and is superior to the other backbones.

4. Discussion

We evaluated the EVSegNet through ablation study by adding Resnet, DUC, HDC, and edge stream block. For the outer vessel wall contour segmentation, as can be seen in the first, third, and fourth row in the case of the MR vessel wall image segmentation results from Figure 9 and Table 2, the EVSegNet achieved superior quantitative results than the other methods for the segmentation of small-sized vessel walls in rows 1, 3, and 4.

More careful observation suggests that adding the edge stream as an auxiliary output for the vessel wall segmentation task is vital for improving the accuracy. The use of a subnetwork of edge stream before doing the segmentation is beneficial; this approach improves the shape accuracy of the resulting segmentation. In particular, our edge stream does not require additional annotation, since edge information can be generated from the ground truth segmentation masks. This means that adding the edge module as prior knowledge into our method can further improve the segmentation accuracy by generating refined boundaries. Therefore, it improves the segmentation quality surrounding the edge, which results in the overall better segmentation performance achieved; our results are most consistent with ground truths.

5. Conclusions

Overall, the proposed fully automatic vessel wall segmentation method, which emphasizes the edges information, can improve segmentation accuracy by utilizing edge gated layers in combination with DUC and HDC modules. This will facilitate more accurate quantitative morphologic and signal measurements of arterial vessel walls and plaque.