SARFNet: Selective Layer and Axial Receptive Field Network for Multimodal Brain Tumor Segmentation

Abstract

Efficient magnetic resonance imaging (MRI) segmentation, which is helpful for treatment planning, is essential for identifying brain tumors from detailed images. In recent years, various convolutional neural network (CNN) structures have been introduced for brain tumor segmentation tasks and have performed well. However, the downsampling blocks of most existing methods are typically used only for processing the variation in image sizes and lack sufficient capacity for further extraction features. We, therefore, propose SARFNet, a method based on UNet architecture, which consists of the proposed SL_iRF module and advanced AAM module. The SL_iRF downsampling module can extract feature information and prevent the loss of important information while reducing the image size. The AAM block, incorporated into the bottleneck layer, captures more contextual information. The Channel Attention Module (CAM) is introduced into skip connections to enhance the connections between channel features to improve accuracy and produce better feature expression. Ultimately, deep supervision is utilized in the decoder layer to avoid vanishing gradients and generate better feature representations. Many experiments were performed to validate the effectiveness of our model on the BraTS2018 dataset. SARFNet achieved Dice coefficient scores of 90.40, 85.54, and 82.15 for the whole tumor (WT), tumor core (TC), and enhancing tumor (ET), respectively. The results show that the proposed model achieves state-of-the-art performance compared with twelve or more benchmarks.

1. Introduction

Brain tumors are usually caused by the abnormal growth of a large quantity of cells in or around the brain and pose a considerable risk to human health [1,2]. Gliomas, the most common brain tumors [3], are invasive [4] and classified into four levels by the World Health Organization (WHO) [5]. Low-grade gliomas (LGGs) with a grade of 1 or 2 and high-grade gliomas (HGGs) with a grade of 3 or 4 are generally associated with increased mortality [6,7]. Medical imaging is often used in clinical diagnosis [8] to reduce brain tumor mortality. The medical images used for brain tumor diagnosis include X-rays [9], computed tomography (CT), and magnetic resonance imaging (MRI) [10]. Compared with other auxiliary methods, MRI is a primary tool for analyzing brain tumors [11,12,13] and has important clinical value in early diagnosis and treatment [14]. Conventional MRI in the BraTS2018 dataset involves fusing four sequences—T1-weighted (T1), T1 contrasted-enhanced (T1ce), T2-weighted (T2), and fluid attenuated inversion recovery (FLAIR) sequences—to assess the brain structure [15]. A common problem is that it takes a long time for radiologists to segment medical images accurately, and the accuracy is highly dependent on the radiologist’s experience [16,17]. Additionally, segmentation by hand is subject to human error, and evaluation varies among radiologists [18], which accounts for the inconsistent results [19].

Many studies have proposed alternative segmentation methods for brain tumors. It is widely acknowledged that convolutional neural networks (CNNs) can be applied for image segmentation [20,21,22]. A CNN customizes feature prediction into a classification problem that can effectively capture image details [23]. The input image size must be fixed for a neural network with a well-designed parameter structure at all levels, such as AlexNet [24], VGGNet [25], and GoogleNet [26]. Fully convolutional network (FCN) is the first network applied to deep learning for semantic segmentation [27]. FCN enables a designed network to input images of any size [28]. Unlike FCN, which combines shallow features with deep features via addition operations, UNet combines shallow features with deep features by splicing. UNet, a network focused on medical image segmentation, has achieved excellent performance [29]. UNet has a U-symmetrical structure where the left branch performs the encoder task while the right branch conducts the decoder work. Another aspect of the architecture is that the encoder concatenates the corresponding layer of the decoder. UNet combines shallow features with deep features through a U-shaped network structure, which is used to perform the final semantic segmentation [30]. The blocks of UNet are generally composed of multiple convolutional layers [31]. A convolutional layer usually contains multiple feature maps with different weight vectors, which makes it possible to retain abundant image features. After the convolutional layer, the pooling layer is applied to the downsampling operation, which can reduce the size of the image and the number of parameters, thereby increasing the robustness of translation and deformation [32]. However, in the downsampling process, the features cannot be effectively extracted; therefore, important information and correlations between local and global features may be lost [33,34]. The UNet bottleneck features can be decomposed in space or channels [35]. Spatial features produce the most important position information. In contrast, channel features focus on the semantic categories of the segmented objects [36]. An appropriate increase in the receptive field and global dependence benefits spatial feature extraction [37]. However, a problem is that the spatial features of the bottleneck block exploit only global dependencies or local features. The main contributions of our work are as follows:

(1)

We proposed SARFNet, which can fully utilize the properties and characteristics of local and global features, to improve brain tumor segmentation performance.

(2)

We designed the SL_iRF block, which is composed of different convolution kernels, a group convolution, an attention mechanism of spatial features, and channel features in each downsampling layer to capture detailed features, avoid losing important information and enhance the correlations between local and global features during the downsampling process.

(3)

We developed the AAM module in the bottleneck layer, which uses a global receptive field and prior knowledge with a multiscale receptive field to integrate more contextual information between the axial features.

(4)

We exploited CAM and deep supervision to improve the segmentation effect further.

The remainder of this paper is organized as follows: related work is described in Section 2. The methods and details of SARFNet are presented in Section 3. The datasets and preprocessing, implementation details, evaluation metrics, comparisons with other methods, and ablation experiments are presented and analyzed in Section 4. Subsequently, Section 5 provides the discussion and conclusions.

2. Related Work

2.1. Deep Learning-Based Methods for Medical Image Segmentation

In recent years, deep learning has become an important branch of artificial intelligence. It is widely used in image processing, natural language processing, and other fields. Image segmentation is an important application in deep learning. It divides an image into several regions; the pixels in each region have the same characteristics, providing more refined region-level information for subsequent processing [38]. In 1996, Sahiner et al. [39] proposed a CNN that was applied to medical image processing. In this work, regions of interest (ROIs) containing biopsy-confirmed masses were extracted from mammograms. A traditional CNN is mainly used for image classification; all images are assigned corresponding class labels separately, while UNet mainly focuses on end-to-end pixel-level medical image segmentation. The UNet structure is generally composed of an encoder and a decoder. The encoder usually contains a convolutional block and a maximum pooling layer. In contrast, the decoder contains upsampling blocks that integrate deep and shallow semantic information and convolutional layers. Compared with 2D images, 3D images can provide more detailed brain tumor information. Özgün Çiçek et al. [40] proposed 3D UNet, which is a variant of the UNet architecture, to process volume data from CT or MR images with depth information. The network layer gradually deepens to learn more abstract and complex features and improve the network expression ability. However, with the gradual increase in network depth, problems such as gradient disappearance or gradient explosion will occur during backpropagation. Many UNet variants have been studied to address such challenges and improve brain tumor image segmentation results. A network architecture named UNet++ with dense connections was proposed by Zhou et al. [41] and makes full use of the characteristics of different levels to improve accuracy. In UNet++, the network parameters are greatly reduced to an acceptable range using the deep supervision mechanism. The V-Net architecture based on UNet, which uses a convolutional layer to replace upsampling and downsampling layers, was proposed by Milletari et al. [42]. Chen et al. [43] proposed a 3D dilated multifiber network (DMF) architecture to reduce the computational overhead and solve the problem that the model cannot be used in practical large-scale applications. The MBANet, which was developed by Cao et al. [44], achieved more accurate segmentation results using spatial and channel attention. Zhang et al. [45] constructed HMNet, which effectively focused on enhancing the relationship between low-level and high-level semantic features. The above methods fully demonstrate the effectiveness of the U-shaped structure in medical image processing, so we chose the U-shaped network as our framework.

2.2. The Attention-Based Module for Medical Image Segmentation

The attention mechanism is able to focus on some key points and reduce the loss of detailed features; this approach has attracted attention and has been increasingly applied to medical images. Cao et al. [44] used a novel multibranch 3D shuffle attention (SA) module as the attention layer in the encoder-constructed MBANet to obtain more accurate segmentation results. Liu et al. [46] proposed SGERes UNet, in which SGE attention is employed to enhance the ability to learn semantic features. Tian et al. [47] proposed AABTS-Net with an axial attention mechanism, which makes it easier for models to provide local–global contextual information and capture abundant semantic information. Zhuang et al. [48] proposed ACMINet to adaptively refine and fuse multimodal features. To reduce the semantic gap, Kuang et al. [49] presented UCA-Net to enhance the contextual extraction ability among distinct slices. Yang et al. [50] proposed a hybrid pooling (HP) module and a hybrid upsampling (HU) module to capture more contextual features. Wang et al. [51] designed a residual block by fusing large and small kernels to simultaneously capture global and local features while retaining the advantages of ViT. As to research in other fields, Li et al. [52] developed an attention mechanism that enhanced the ability of the downsampling module to extract the essential features of maize diseases.

Dosovitskiy et al. [53] introduced the Transformer, which has achieved great success in the NLP field, into computer vision. At present, efforts are being made to apply the Transformer to medical image segmentation tasks; these models are theoretically efficient. Chen et al. [54] proposed TransUNet, which combines the Transformer and UNet, to enhance finer details by recovering localized spatial information from medical image segmentation. Zhang et al. [55] exploited a novel multimodal medical Transformer (mmFormer) to realize segmentation of incomplete modalities in medical images; it performed well. Lu et al. [56] combined the local modeling of CNN and the long-range representation of Transformer in an auxiliary MetaFormer decoding path. Zhang et al. [57] incorporated an efficient spatial-channel attention layer in the bottleneck for global interaction to further capture high-level semantic information and highlight important semantic features from the encoder path output. Undoubtedly, many UNet variants have shown great success in brain tumor segmentation. However, downsampling, which reduces the image size, performs poorly. An architecture with an effective mechanism needs to be developed to address this challenge and help networks efficiently perform further brain tumor feature extraction.

3. Methodology

3.1. Network Architecture

The suggested U-shape structure with the encoder on the left, the decoder on the right, and the skip connections in the middle achieves good segmentation performance in many image segmentation tasks. As shown in Figure 1, the network we proposed is similar to the UNet structure. Firstly, the images of the four brain tumor modalities (T1, T1ce, T2, and FLAIR) are processed using two 3 × 3 × 3 convolutions with a stride of 1, and the dimensions are adjusted from 4 × 128 × 128 × 128 to 32 × 128 × 128 × 128. Group Normalization (GN) and Gaussian error linear unit (GeLU) are employed after each convolution. Empirically, a residual block is used in the first layer. Sequentially, the SL_iRF module is used for downsampling. Four layers are used in the encoder to perform the feature extraction operation. In the bottleneck stage, the three AAM modules and squeeze-and-excitation (SE) are used to improve the model performance by explicitly constructing the interdependence between feature channels. The four upsampling blocks in the decoder path alter the image resolution via deconvolution with a stride of 2 based on the previous image pixels. Each layer in the decoder contains two convolutions, a GN and a GeLU, and the size after convolution is the same as that of the equivalent encoder. There is a residual block between convolutions. The SE module, used in the residual block, can dynamically adjust the weights of different channels by learning the relationships between different channels. Skip connections are constructed that can integrate high-level semantic information and low-level semantic information between the encoder and decoder layers to accurately locate the target image. In the third and fourth layers of the skip connections, we applied CAM [58], which is shown in Figure 2, to guide high-level features to the selection of low-level semantic features. Naturally, deep supervision is introduced to solve convergence problems and improve training performance. At the end of the network, a 1 × 1 × 1 convolution is utilized to restore the dimensions of the obtained feature map to the original 4 × 128 × 128 × 128.

3.2. Selective Layer Receptive Field Attention Module (SL_iRF)

Spatial features are very important for medical image segmentation; however, the challenge of losing effective information during the downsampling stage has still not been addressed. Receptive field attention (RFA) [59] was introduced to address the issue of considering the significance of each feature in the receptive field to further improve the efficiency of the extraction features and the performance of the model. Inspired by RFA, we propose the selective layer receptive field attention module (SL_iRF), which is shown in Figure 3. The module is used in downsampling, which utilizes convolutions with different kernels to reduce the image size and capture more useful information on each channel and spatial feature. Generally, downsampling uses pooling or convolution with a stride of 2, which leads to information loss.

The SL_iRF module consists of SL_iConv, group convolution, and spatial and channel attention. SL_iConv, which adopts different convolution kernels in each layer, can expand the receptive field while completing the size change. One of its main functions is to retain important feature information during downsampling. Group convolution can extract spatial features with a larger receptive field. Batch Normalization (BN) and Rectified Linear Unit (ReLU) are used together to enhance the model’s ability to learn nonlinear mappings. The spatial and channel weights can be obtained while ensuring a large receptive field to improve segmentation performance using the global average pooling and maximum pooling operations in the spatial and channel attention mechanism. Based on the above operations, the SL_iRF module can ensure a large receptive field to effectively reduce the loss of important information during downsampling operations.

Formally, as shown in Figure 3, x is the input feature $x\in R^{C\times H\times W\times D}$ of the SL_iRF module, where C, H, W, and D represent the number of channels, height, width, and depth of the feature map, respectively. The SL_iConv algorithm selects different convolution kernels according to the position of the layer, which can be represented as follows:

$${U_i=f}_{sli}\left(x\right)$$

(1)

where $f_{sli}$ denotes the convolution with different kernels of the SL_iConv module, whose kernel sizes are {3 × 3 × 3, 5 × 5 × 5, 7 × 7 × 7, 7 × 7 × 7}. U_i is the result calculated by the SL_iConv module. Index i represents the position of each encoder layer. We define the convolution kernels according to the different positions of the layer as follows:

• Layer 1: Kernel Size is 3 × 3 × 3 → U1 → $f_{sl1}\in R^{C\times \mathrm{H}\times \mathrm{W}\times \mathrm{D}}$
• Layer 2: Kernel Size is 5 × 5 × 5 → U2 → $f_{sl2}\in R^{2C\times \frac{1}{2}\mathrm{H}\times \frac{1}{2}\mathrm{W}\times \frac{1}{2}\mathrm{D}}$
• Layer 3: Kernel Size is 7 × 7 × 7 → U3 → $f_{sl3}\in R^{4C\times \frac{1}{4}\mathrm{H}\times \frac{1}{4}\mathrm{W}\times \frac{1}{4}\mathrm{D}}$
• Layer 4: Kernel Size is 7 × 7 × 7 → U4 → $f_{sl4}\in R^{8C\times \frac{1}{8}\mathrm{H}\times \frac{1}{8}\mathrm{W}\times \frac{1}{8}\mathrm{D}}$

After SL_iConv, group convolution with the same number of groups as the number of input channels is used to quickly capture features with receptive fields. The specific expression is as follows:

$$G{U}_i=ReLU\left(BN\left(GCon{v}_{3\times 3\times 3}\left(U_i\right)\right)\right)$$

(2)

where GConv_3×3×3 denotes Group Convolution with 3 × 3 × 3 kernel, with the number of groups the same as the number of input channels. BN represents batch normalization; ReLU is Rectified Linear Unit.

The top part in Figure 3 is the channel attention weight section, which captures the weights of the channel features. The specific expression is as follows:

$$C{A}_i=LNS\left(LNR\left(GP\left(U_i\right)\right)\right)$$

(3)

where LNR denotes linear operation and ReLU. LNS represents linear operation and sigmoid. GP is the global average pooling.

The spatial attention weight is presented in the bottom branch in Figure 3. The specific description is as follows:

$$S{A}_i=GW\left(CS\left(concat(AP\left({GU}_i\right),MP(G{U}_i)\right)\right))$$

(4)

where AP denotes average pooling; MP is max pooling; CS is defined as convolution with 3 × 3 × 3 kernel and sigmoid; and GW indicates the get weight operation, which can be written as follows:

$${GW}_i(•)=AP\left(GC\left(•\right)\right)$$

(5)

where GC is defined as group convolution, whose convolution kernel is 1, and the number of groups is equal to the number of input channels.

We combine the three results via multiplication to accurately extract spatial and channel features with receptive fields. The specific description is as follows:

$$Z_i=U_i\times S{A}_i\times C{A}_i$$

(6)

At the end of the SL_iRF block, a convolution with a 3 × 3 × 3 kernel and stride of 3 is used to extract the spatial features of the receptive field.

3.3. Axial Atrous Spatial Pyramid Pooling Module (AAM)

The image in the bottleneck layer contains considerable details, which plays an important role in image restoration during upsampling. The ordinary convolution affects the final restoration effect due to the lack of global information when dealing with detailed features. Axial attention [60] using self-attention can establish global dependence and expand the receptive field of the image on each axial feature. Its receptive field is larger than the convolution, and more contextual information can be obtained. However, the axial attention mechanism cannot effectively use prior knowledge of the scale invariance, translation invariance, or feature locality of the image, which leads to the inability to construct an accurate local relationship in some cases. ASPP [61] uses features with different expansion sampling rates generated by atrous convolution, so that the features contain multiscale receptive fields. It includes multiscale local information and retains the properties of ordinary convolution. The above two attention mechanisms are fused to create the axial atrous spatial pyramid pooling module (AAM) shown in Figure 4. It uses axial attention to extract global information and retains the characteristics of ordinary convolution and multiscale receptive fields. It can be represented as follows:

$$Z=con{v}_{1\times 1\times 1}\left(cat\right(atrous(AP\left(self\_attn(self\_atten(x,w),h)\right),\theta )\left)\right)$$

(7)

where x is the input feature $x\in R^{C\times H\times W\times D}$; Z is the result processed by the AAM module; Conv_1×1×1 denotes convolution with a kernel of 1 × 1 × 1; Atrous is expressed as an atrous convolution operation with a rate of θ, $\theta ϵ\{1,2,3,4\}$; AP represents the average pooling; and self_attn describes the self-attention mechanism. Here, parameter h represents the column, and parameter w represents the row. Self-attention can be defined as follows:

$$Z=\alpha \times torch.bmm(Conv\left(x\right),sigmoid\left(torch.bmm\left(Conv\left(x\right),{Conv\left(x\right)}^T\right)\right))+x$$

(8)

where Z denotes the output result of the module, x denotes the input feature, and Conv represents convolution with a kernel of 1 × 1 × 1. α indicates learning parameters, which can learn suitable parameter values through model training. The torch.bmm operation is defined as a matrix product.

4. Experiments and Results

4.1. Datasets and Preprocessing

The BraTS dataset refers to the brain tumor segmentation challenge dataset, which is a public medical image dataset used to research and develop brain tumor segmentation algorithms. The dataset is provided by multiple medical centers and contains image data from multiple patients who underwent brain tumor MRI. The BraTS 2018 [62,63,64,65,66], which consists of 285 patients used for training and 66 patients used for online validation, has become popular. Generally, the cases all contain ground truths labeled by board-certified neuroradiologists. In comparison, the ground truths of 66 cases are hidden from the public, and the results of the model can be obtained only by online validation. Our training strategy was to use all 285 cases for training. Our results were evaluated on the official BraTS platform (https://www.med.upenn.edu/sbia/brats2018.html, accessed on 13 May 2024).

The brain tumor competition team conducted data preprocessing, including registration, resampling, and skull removal. However, further data preprocessing was also necessary for segmentation tasks. First, we reduced as much of the background as possible while ensuring that all the brain images were included and randomly re-cropped the fixed patch size of the image to 128 × 128 × 128. Second, the Z-score method was used to standardize each image to reduce the contrast differences between the four sequences. All intensity values were clipped to the 1st and 99th percentiles of the nonzero voxel distribution of the volume. Similarly, we applied data augmentation techniques to prevent overfitting. During image processing, we performed channel rescaling between 0.9 and 1.1, a channel intensity shift between −0.1 and 0.1, additive Gaussian noise of a centered normal distribution with a standard deviation of 0.1, channel dropping, and 80% random flipping along each spatial axis.

4.2. Implementation Details

Our network was constructed in Python 3.8.10 and PyTorch 1.10.0. A single NVIDIA GeForce RTX 3090 (Nvidia, Santa Clara, CA, USA) with 24 GB of memory and Intel(R) Xeon(R) Platinum 8255C (Intel, Santa Clara, CA, USA) were used during training. As shown in

Table 1. Model parameter configuration.

Basic Configuration	Value
PyTorch Version	1.10.0
Python	3.8.10
GPU	NVIDIA GeForce RTX 3090(24 G)
Cuda	11.3
Learning Rate	1.00 × 10−4
Optimizer	ranger
Batch Size	1
Input Size	128 × 128 × 128
Output Size	128 × 128 × 128

, the initial learning rate was 1.00 × 10⁻⁴, and the batch size was 1. Ranger was used to optimize our network. Unlike hybrid loss, only the ordinary soft Dice loss was trained in our network. The sizes of the input in the first stage and the output of the last layer were 128 × 128 × 128.

4.3. Evaluation Metrics

Quantitative and qualitative analyses were carried out by evaluation metrics, including the Dice similarity coefficient (Dice) score and the Hausdorff distance (HD).

The Dice coefficient is a measure of the similarity between the network-predicted segmentation results and manual masks in image segmentation, which can be represented as follows:

$$Dice=\frac{2\mathit{TP}}{2\mathit{TP}+\mathit{FP}+\mathit{FN}}$$

(9)

where TP, FP, and FN are the values of true positive, false positive, and false negative voxels, respectively.

HD represents the maximum distance between the predicted segmentation region boundary and the real region boundary. The smaller the value, the smaller the prediction boundary segmentation error and the better the quality. The HD can be written as follows:

$$HD(P,T)=max\{su{p}_{t\in T }in f_{p\in P}d\left(t,p\right), su{p}_{p\in P }in f_{t\in T}d\left(t,p\right)\}$$

(10)

where t and p represent the real region boundary and predicted segmentation region boundary, respectively; d(•) represents the distance between t and p; and sup denotes the supremum.

4.4. Comparison with Other Methods

To evaluate the advantages of the proposed model, we compared twelve advanced models. The numbers of networks compared were 2, 4, 2, 1, 1, and 2 for 2024, 2023, 2022, 2020, 2019, and 2016, respectively. 3D UNet, UNet++, V-Net, DMFNet, MBANet, HMNet, GMetaNet, RFTNet, SPA-Net and ETUNet are architecture variants based on basic UNet, and TransUNet and mmformer are structures based on Transformer.

Table 2. Comparison of different methods on BraTS2018 validation dataset.

Methods	Dice Score (%)				Hausdorff Distance (mm)
	WT	TC	ET	AVG	WT	TC	ET	AVG
3D U-Net [40] (2016)	88.53	71.77	75.96	78.75	17.10	11.62	6.04	11.59
UNet++ [41] (2020)	88.76	82.05	77.94	82.92	6.31	8.64	4.82	6.59
V-Net [42] (2016)	89.60	81.00	76.60	82.40	6.54	7.82	7.21	7.19
DMFNet [43] (2019)	89.90	83.50	78.10	83.83	4.86	7.74	3.38	5.33
TransUNet [54] (2022)	89.95	82.04	78.38	83.46	7.11	7.67	4.28	6.35
mmformer [55] (2022)	89.56	83.33	78.75	83.88	4.43	8.04	3.27	5.25
GMetaNet [56] (2023)	90.10	84.20	82.00	85.40	5.16	5.26	2.62	4.34
MBANet [44] (2023)	89.80	85.47	80.18	85.15	5.13	5.65	2.47	4.42
HMNet [45] (2023)	90.10	84.30	78.60	84.33	4.73	7.73	2.70	5.05
RFTNet [67] (2023)	90.30	82.15	80.24	84.23	5.97	6.41	3.16	5.18
SPA-Net [68] (2024)	89.63	85.89	79.90	85.14	4.79	5.40	2.77	4.32
ETUNet [57] (2024)	90.00	85.20	81.00	85.40	6.67	7.40	6.01	6.69
SARFNet (Ours)	90.40	85.54	82.15	86.03	5.95	7.30	2.72	5.23

and Figure 5 and Figure 6 show that the Dice coefficients of SARFNet are 90.40, 85.54, and 82.15, and the HDs are 5.95, 7.30, and 2.72 for the three tumor subregions (WT, TC, and ET), respectively. We compared the traditional 3D UNet as the baseline, and our WT, TC, and ET results increased by 1.87, 13.77, and 6.19, respectively. In the experiment, we conducted multiple comparisons, including the variants based on UNet that perform brain tumor image segmentation tasks well, as well as CNN + Transformer architecture based on Transformer. The implementation foundation of Transformer is self-attention, which is an attention mechanism that can establish global dependencies and expand the receptive field of images. Compared to CNN, it has a larger receptive field and can capture more contextual information. It achieves better performance by selecting important information and ignoring unimportant information. However, this results in its ability to capture effective local information to be lower than CNN’s. Recently, many networks based on Transformer have also incorporated CNN for local characteristics. Although there has been a significant improvement in the results of brain tumor segmentation tasks, it has not achieved the best performance of brain tumor segmentation tasks, due to only adding local features without considering maintaining the receptive field. As a result, some networks based on Transformer have poorer results than the networks based on UNet. The attention mechanism is currently working well in networks based on UNet architecture. However, due to its inferior ability to establish accurate global relationships compared to Transformer, the loss of global relationship information affects the final segmentation performance.

In brain tumor segmentation tasks, both global relationships and local detail features are crucial. The purpose of the network we propose is to maintain global relationships as much as possible, thereby improving performance in brain tumor segmentation. The results show that our network dramatically improved in TC and ET; that is, our network performs better than the baseline in small target segmentation.

Figure 7 shows the visualization results of the SARFNet model on the BraTS2021 [69] dataset, from which five cases are randomly chosen. The medical case numbers 003_107, 020_84, 56_76, 62_83, and 66_80, according to sequences a, b, c, d, and e, are segmented by the SARFNet. Green, yellow, and red represent the whole tumor, tumor core, and enhancing tumor, respectively. The results of SARFNet are close to the labeled ground truth. Compared with the variant network based on UNet, the best MBANet has lower indicators. We also overcame the networks based on Transformer. In general, our architecture and modules achieved excellent results on BraTS2018, which can provide a good basis for subsequent research.

4.5. Ablation Experiments

4.5.1. Ablation Study of Each Module in SARFNet

We conducted ablation experiments to verify the effect of different modules in this architecture.

Table 3. Dice result of ablation study of each module in SARFNet.

	SLiRF	AAM	CAM	DP	WT	TC	ET	AVG
A		√			89.62	81.79	80.98	84.13
B	√	√			90.42	84.00	80.66	85.03
C	√	√	√		90.13	84.19	81.14	85.15
D	√	√		√	90.83	85.09	82.00	85.97
SARFNet	√	√	√	√	90.40	85.54	82.15	86.03

shows that, when AAM is used for UNet, the results are 89.62, 81.79, 80.98, and 84.13 for the WT, TC, ET, and average Dice, respectively. The results of UNet with AAM are better than baseline, which shows that the AAM module is effective at brain tumor segmentation tasks because of its ability to use axial attention to extract global information and retain the characteristics of ordinary convolution and multiscale receptive fields. After the SL_iRF module is added, the results are 90.42, 84.00, 80.66, and 85.03 for the WT, TC, ET, and average Dice, respectively, which indicates that it has been further improved on the basis of the previous module. The module plays a role in further feature extraction during the downsampling stage.

After CAM was added, the TC, ET, and average Dice increased, with values of 90.13, 84.19, 81.14, and 85.15 for the WT, TC, ET, and average Dice, respectively. The results of the ablation study of deep supervision show that the results are improved at WT, TC, ET, and average Dice. We fused the above modules to the network; the results are 90.40, 85.54, 82.15, and 86.03 for WT, TC, ET, and average Dice, respectively. The best results show that our network and all the modules can be effectively applied to brain tumor segmentation tasks.

4.5.2. Ablation Experiments for SL_iRF Module

In the previous ablation experiment, the SL_iRF module was added to the network to improve the effect of the network. However, using different convolution kernels for each layer changes the effect on the SL_iRF module. Four groups of experiments were performed to verify the convolution kernel size in different layers. The four experimental settings are as follows:

The convolution kernel sizes are 3 × 3 × 3 in all layers.

The convolution kernels sizes are 5 × 5 × 5 in all layers.

The convolution kernels sizes are 7 × 7 × 7 in all layers.

The convolution kernel sizes from the first to the fourth layers are 3 × 3 × 3, 5 × 5 × 5, 7 × 7 × 7, and 7 × 7 × 7, respectively.

The results of SL_iRF with the last line convolution kernel sizes were the best. The values in

Table 4. Dice results of ablation experiments of selective kernel for SL_iRF module.

	Layer Number:Kernel Size				WT	TC	ET	AVG
A	Layer 1:3	Layer 2:3	Layer 3:3	Layer 4:3	90.32	85.43	80.51	85.42
B	Layer 1:5	Layer 2:5	Layer 3:5	Layer 4:5	89.86	85.45	80.65	85.32
C	Layer 1:7	Layer 2:7	Layer 3:7	Layer 4:7	89.97	85.50	79.37	84.96
SLiRF	Layer 1:3	Layer 2:5	Layer 3:7	Layer 4:7	90.40	85.54	82.15	86.03

are 90.40, 85.54, 82.15, and 86.03 for the WT, TC, ET, and average Dice, respectively. The results show that, as the number of layers increases, the increase in the size of convolution kernels has a greater benefit for the segmentation performance.

4.5.3. Ablation Experiments for CAM Module

The CAM, which is added to skip connections between the encoder and decoder, can fuse high-level features to guide the feature selection of the low-level features. The deeper the layer, the richer the features. For this reason, we studied the number of CAM applied to the skip connection layer.

As shown in

Table 5. Dice results of ablation experiments for CAM module.

	Layer				WT	TC	ET	AVG
	4th	3rd	2nd	1st
A					90.83	85.09	82.00	85.9
B	√				90.65	85.31	80.00	85.32
C (SARFNet)	√	√			90.40	85.54	82.15	86.03
D	√	√	√		89.24	84.10	81.74	85.02
E	√	√	√	√	89.71	85.53	78.66	84.63

, when CAM is employed simultaneously on the third and fourth layers, the effect is the best, and the results are 90.40, 85.54, 82.15, and 86.03 for the WT, TC, ET, and average Dice, respectively. We speculated that the shallow semantic information in first layer and second layer is less helpful to the decoder layer. CAM is beneficial to our architecture, but its effectiveness is limited.

4.5.4. Ablation Experiments for Axial Dimension of AAM Module

We conducted three experiments to analyze the relationship between axial dimensions and receptive fields. The experiments included:

A: expanding receptive fields on all three axial operations;

B: not expanding receptive fields on all three axial operations;

C: expanding receptive fields on two axial operations.

The depth axial operation lacking sufficient information on each slice has an impact on the results, as shown in

Table 6. Dice results of ablation experiments for axial dimension of AAM module.

Experiment	Dice Score (%)
	WT	TC	ET	AVG
A	90.61	85.25	79.87	85.25
B	90.20	84.95	78.39	84.52
C (SARFNet)	90.40	85.54	82.15	86.03

and Figure 8. Therefore, the operations of receptive fields applied to all three axial operations are not as effective as the horizontal and vertical axial operations alone. That is to say, AAM works best on the horizontal and vertical axial operation.

5. Discussion and Conclusions

An appropriately scaled receptive field is crucial for brain tumor segmentation tasks, and making good use of the receptive field can effectively enhance the performance of brain tumor segmentation. In this paper, we proposed a SARFNet architecture based on UNet that integrates the SL_iRF module, AAM module, CAM, and deep supervision to improve brain tumor segmentation performance. The SL_iRF module is developed for downsampling to reduce the image size and expand the receptive field to extract multi-scale information. The AAM module in the bottleneck layer can capture global information while preserving the advantages of convolution. The CAM is used in the skip connections of the third and fourth layers, using high-level semantic information to guide low-level semantic information to help restore images during the upsampling stage. In addition, we compared it with networks based on the UNet and Transformer from 2022 to 2024. Our results show Dice values of 90.40, 85.54, and 82.15 and HD values of 5.95, 7.30, and 2.72, respectively, for the three tumor subregions (WT, TC, and ET). The results show that the proposed model achieved state-of-the-art performance compared with more than twelve benchmarks. This indicates that the model and modules we proposed can achieve accurate brain tumor segmentation and enhance the performance of the segmentation.

By comparing with a traditional 3D UNet, we found that there were significant improvements in the segmentation of small targets. Compared with variant networks based on UNet and Transformer, our achievement lies in the fact that our proposed network uses different convolution kernels according to the characteristics of each layer in the network, effectively maintains the relationship between the receptive field and global features, and prevents the loss of important information during the feature processing. Specifically, we also conducted ablation experiments on the SL_iRF module, AAM module, CAM, and deep supervision, which demonstrated the effectiveness of our modules. From Table 4, it is known that, in our network, features become more detailed as the layers deepen. In this case, the information extracted by larger convolutional kernels is sufficient, and there is also an improvement in performance. This indicates that, in the deeper layers of the network, the size of the receptive field has an impact on the effective extraction of features. The results in Table 5 show that, in the network we proposed, the deeper the network layers, the more necessary it is to use high-level semantic information to guide the selection of lower-level semantic information.

However, our research also had some limitations. The extensive use of attention mechanisms will inevitably increase the complexity of the model. Therefore, we aim to perform lightweight research on models in the future. Although our network exhibited better performance when tested on the brain tumor dataset for tumor segmentation tasks, we have not attempted to evaluate images unrelated to brain tumors, such as RGB images. In the next stage, we aim to increase the scope of our experiments to make the network we proposed more practical. We believe that the encouraging results obtained with SARFNet will inspire further research into brain tumor segmentation.