Automated Hybrid Model for Detecting Perineural Invasion in the Histology of Colorectal Cancer

Abstract

Perineural invasion (PNI) is a well-established independent prognostic factor for poor outcomes in colorectal cancer (CRC). However, PNI detection in CRC is a cumbersome and time-consuming process, with low inter-and intra-rater agreement. In this study, a deep-learning-based approach was proposed for detecting PNI using histopathological images. We collected 530 regions of histology from 77 whole-slide images (PNI, 100 regions; non-PNI, 430 regions) for training. The proposed hybrid model consists of two components: a segmentation network for tumor and nerve tissues, and a PNI classifier. Unlike a “black-box” model that is unable to account for errors, the proposed approach enables false predictions to be explained and addressed. We presented a high performance, automated PNI detector, with the area under the curve (AUC) for the receiver operating characteristic (ROC) curve of 0.92. Thus, the potential for the use of deep neural networks in PNI screening was proved, and a possible alternative to conventional methods for the pathologic diagnosis of CRC was provided.

1. Introduction

Perineural invasion (PNI) in colorectal cancer (CRC) is a well-established independent prognostic factor [1,2], with an incidence range from 9% to 30% [3,4]. PNI is defined as tumor invasion into, around, and through neural structures [5]; it is the distinct route through which cancer cells spread and metastasize to adjacent or distant organs [6,7]. PNI detection is associated with response to adjuvant chemotherapy [8]. Therefore, a meticulous evaluation of PNI and prognostication on the basis of the standardized pathology report are mandatory in routine pathology practice [9,10].

Despite its importance, the histologic evaluation of PNI is a cumbersome and time-consuming process, with a high risk of misdiagnosis [11]. Peng et al. [12] reported that only 7.5% of patients were PNI-positive in original pathologic reports; however, reviewing their PNI status revealed that 24.3% of patients were PNI positive. Thorough histological inspections are necessary for reducing the misdiagnosis rate. However, pathologic diagnosis of PNI is tedious and increases the workload of pathologists. Given the increasing workload for pathologists and their critical shortage nationally and globally [13,14], developing an automated screening tool for PNI is crucial.

Deep learning (DL) methods have achieved promising results in medical image analysis [15,16], surpassing human performance [17]. In computational pathology, histology-based DL approaches have facilitated computer-aided diagnostics, including tumor detection, classification, segmentation, and even quantification of established biomarkers, such as tumor-infiltrating lymphocytes [18,19,20]. Until now, limited studies have focused on PNI detection in computational pathology [21,22]. However, as with DL-based approaches, these studies exhibited the drawback of failing to interpret unpredictable failures. The lack of interpretability in “black-box” modeling limits real-world application, which may lead to user distrust [23,24].

In this study, an interpretable DL-based PNI detector was developed through CRC histology, demonstrating the potential of computer-aided diagnosis for PNI screening. The proposed approach could become an alternative to the conventional methods of pathologic diagnosis of CRC.

2. Materials and Methods

2.1. Data Acquisition

A total of 77 whole-slide images (WSIs) of 63 patients with CRC who underwent surgical resection at International St. Mary’s Hospital, Catholic Kwandong University in Incheon Metropolitan City, Republic of Korea, were selected for the study. The specimens were formalin-fixed, paraffin-embedded, and stained with hematoxylin and eosin. An Aperio AT2 slide scanner (Leica Biosystems, Buffalo Grove, IL, USA) was used to scan the WSIs at 40× magnification. From the 77 WSIs, 530 regions were selected, including PNI with tumor and nerve tissue, non-PNI with tumor, non-PNI with nerve tissue, and normal tissue excluding neural tissue, respectively, denoted as “PNI”, “tumor”, “nerve”, and “normal”, (

Table 1. Composition of regions and patches.

		No. of Regions	No. of Patches
PNI		100	362
Non-PNI	Nerve	204	687
	Tumor	207	7547
	Normal	19	880
Total		530	9476

). All PNIs inside the region were annotated; non-PNI with tumor tissues, and non-PNI with nerve tissues were randomly extracted and annotated. Tumor, nerve, and normal tissues were annotated by board-certified pathologists (J.J. and S.A.) using the automated slide analysis platform (ASAP). All results in this study have undergone two rounds of reviews. Inconsistencies were discussed with another board-certified pathologist (S.H.L.). A total of 490 regions were used for training and validation, and the remaining 40 were used to test the model.

2.2. Patch Generation

We used a half-overlap sliding window algorithm for model input. This method can overcome the loss of information between adjacent areas. Moreover, the summation of the probability of adjacent regions, which comes from deep learning models, can increase overall accuracy. For the strategy of patch generation, we employed 1.0 mpp (micron per pixel) for resolution and 512 × 512 × 3 pixels for size. Patch prediction or extraction is skipped when the mean pixel value of the target patch is too high (>235) or too low (<50), because a high mean of the pixel value is mostly due to the background, and a low mean of the pixel value is mostly due to the low quality of the whole slide image.

2.3. Image Preprocessing

Each patch generated from the same WSI follows a similar color distribution, but patches from different WSIs may not. To overcome the difference of color distribution among WSIs, we employ color augmentation, such as HSV shift and random brightness. To obtain a sufficient geometric pattern of patches, geometric augmentations, such as elastic transformation, shift scale rotation, random rotation, horizontal flipping, and vertical flipping, were used. All Augmentations were implemented using the Albumentation open-source library https://github.com/albumentations-team/albumentations (accessed on 20 October 2021) [25]. Scaling (0–1) was employed for data normalization.

2.4. Segmentation Network Development

A general scheme of the proposed model is displayed in Figure 1. A hybrid model was proposed to detect PNI using histology, comprising a semantic segmentation network and a rule-based PNI classifier.

A multiclass semantic segmentation network was trained to detect tumors and nerves. As an alternative approach, experiments were conducted using binary segmentation models for tumors and nerves. For the segmentation frameworks, we used the U-Net [26], Deeplabv3+ [27], and SegFormer [28] networks. U-Net, Deeplabv3+, and SegFormer were built on a pre-trained backbone. For an ablation study, SegFormer was trained from scratch, comparing the performance of a transformer-based segmentation network with transfer learning. To train the U-Net, three backbone models, namely Inception-Resnet-v2, EfficientNet-B0, and SE-ResNeXt-101, were used. For the training of the Deeplabv3+, two models—MobileNet and Xception—were used as backbones. A pre-trained model using the ImageNet database was used [29]. For the training of SegFormer, MiT-B0 (Mix Transformer Encoder) was used. An adaptive moment estimation optimizer was used, with an initial learning rate of 10 × 10⁻³. The batch size for training was set to 32 and the maximum number of epochs was set to 200. In the multi-class segmentation network, a multi-loss function, calculated as a weighted sum of the dice loss and categorical focal loss (L_Multi $=$ L_Dice $+$ L_Focal) was used. To train the binary segmentation network, a combination of dice loss and binary cross-entropy loss (L_Binary $=$ L_Dice $+$ L_CE) was used.

2.5. PNI Classifier

To generate tumor and nerve masks, which were input into the PNI classifier, we implemented six combinations of segmentation networks for tumors and nerves, denoted as Module1 (Md1) to Module6 (Md6). Md1 to Md4 employed binary segmentation networks for tumors and nerves, whereas Md5 and Md6 used multiclass segmentation networks. In the Md1 framework, U-Net was used for both nerve and tumor segmentation. For Md2, U-Net and DeepLabv3+ were used for nerve and tumor segmentation, respectively. In Md3, DeepLabv3+ and U-Net were utilized for nerve and tumor segmentation, respectively. In Md4, DeepLabv3+ was used for both nerve and tumor segmentation. In Md5 and Md6, U-Net and SegFormer were employed, respectively.

Based on the trained segmentation network, binary probability maps were inferred for tumors and nerves using a half-overlapped sliding window. The average probability was applied for the overlapped window. Tiny areas predicted as tumors or nerves (probability threshold = 0.5) were removed using morphological analysis. Nerves with PNI were extracted according to a rule-based approach, where the distances between the binary map of the tumor and the dilated nerve were calculated (Figure 2). PNI and non-PNI groups were defined as follows:

PNI: $Are{a}_{dilate nerve}\cap$ $Are{a}_{tumor}$ $\ne \varnothing$

Non-PNI: $Are{a}_{dilate nerve}\cap Are{a}_{tumor}$ $=\varnothing$

$Are{a}_{tumor}$ and $Are{a}_{dilate nerve}$ denote the binary map of the tumor and the dilated nerve, respectively.

2.6. Evaluation Metrics

The performance of the trained segmentation model using pixel-wise accuracy, intersection over union (IoU), sensitivity, precision, and the F1-score was compared. All metrics are defined as follows:

$$\mathrm{Accuracy}= \sum p|p=g,\forall p\in \mathrm{P}, g\in \mathrm{G}$$

$$\mathrm{IoU}= \frac{\mathrm{G}\cap \mathrm{P}}{\mathrm{G}\cup \mathrm{P}}$$

$$\mathrm{Sensitivity}= {{\displaystyle \mathbf{\sum }}}_{\mathit{i}}\frac{\mathbf{\sum }\mathbf{(}\mathit{p}\mathbf{|}\mathit{p}\mathbf{=}\mathit{i}\mathbf{,}\mathit{g}\mathbf{=}\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }G\mathbf{)}}{\mathbf{\sum }\mathbf{(}\mathit{p}\mathbf{|}\mathit{p}\mathbf{=}\mathit{i}\mathbf{,}\mathit{g}\mathbf{=}\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }\mathbf{G}\mathbf{)}\mathbf{+}\mathbf{\sum }\mathit{p}\mathbf{|}\mathit{p}\mathbf{\ne }\mathit{i}\mathbf{,}\mathit{g}\mathbf{=}\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }G)}\mathbf{\forall }\mathit{i}\mathbf{=}\mathbf{0}\mathbf{,}\mathbf{1}\mathbf{,}\mathbf{2}$$

$$\mathrm{Precision}= {{\displaystyle \mathbf{\sum }}}_{\mathit{i}}\frac{\mathbf{\sum }\mathbf{(}\mathit{p}\mathbf{|}\mathit{p}\mathbf{=}\mathit{i}\mathbf{,}\mathit{g}\mathbf{=}\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }G\mathbf{)}}{\mathbf{\sum }\mathbf{(}\mathit{p}\mathbf{|}\mathit{p}\mathbf{=}\mathit{i}\mathbf{,}\mathit{g}\mathbf{=}\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }G\mathbf{)}\mathbf{+}\mathbf{\sum }\mathit{p}\mathbf{|}\mathit{p}\mathbf{=}\mathit{i}\mathbf{,}\mathit{g}\mathbf{\ne }\mathit{i}\mathbf{,}\mathbf{\forall }\mathit{p}\mathbf{\in }P\mathbf{,} \mathit{g}\mathbf{\in }G\mathbf{)}}\forall \mathit{i}\mathbf{=}\mathbf{0}\mathbf{,}\mathbf{1}\mathbf{,}\mathbf{2}$$

$$\mathrm{F}1-\mathrm{score}:{\mathrm{F}}_1\left(precision, recall\right)=\frac{2 precision recall}{precision+recall}$$

P and G denote predicted value and ground truth, respectively.

To compare the performance of the PNI classifier, we used region-wise Accuracy_R, Sensitivity_R, Specificity_R, Precision_R, Negative Predictive Value_R (NPV_R), F1-score_R, and the area under the curve (AUC). The metrics used to evaluate the region-wise performance are defined as follows:

$${\mathrm{Accuracy}}_{\mathrm{R}}=\frac{\sum \left(\mathrm{TP}+\mathrm{TN}\right)}{\sum (\mathrm{TP}+\mathrm{FN}+\mathrm{FP}+\mathrm{TN})}$$

$${\mathrm{Sensitivity}}_{\mathrm{R}}=\frac{\sum \left(\mathrm{TP}\right)}{\sum \left(\mathrm{TP}+\mathrm{FN}\right)}$$

$${\mathrm{Specificity}}_{\mathrm{R}}=\frac{\sum \left(TN\right)}{\sum \left(FP+TN\right)}$$

$${\mathrm{Precision}}_{\mathrm{R}}=\frac{\sum \left(TP\right)}{\sum \left(TP+FP\right)}$$

$${\mathrm{NPV}}_{\mathrm{R}}=\frac{\sum \left(TN\right)}{\sum \left(FN+TN\right)}$$

$$\mathrm{F}1-{\mathrm{score}}_{\mathrm{R}}: \mathrm{F}1(\mathrm{precision},\mathrm{recall}{)}_{ }=\frac{2 \mathrm{precision} \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}$$

True positive, true negative, false positive, and false negative are denoted as TP, TN, FP, and FN, respectively.

Instead of a stochastic model, a rule-based model was designed. Thus, we used a simple receiver operating characteristic (ROC) curve, which indicates that the classifier predicts PNI as positive when the distance between the tumor and the nerve is zero, and the classifier predicts PNI as negative when the distance is infinite. The confidence interval was calculated by assuming that the distribution of the AUC is similar to that of the accuracy, which is a binomial distribution of sample length and probability.

2.7. Inference Timing

The inference times required in U-Net and SegFormer were measured by averaging the five execution times for a randomly selected patch.

3. Results

3.1. Results of Segmentation Networks

Table 2. Performance of the segmentation models.

		Accuracy	IoU	Sensitivity	Precision	F1-Score
Nerve	U-Net a	0.987	0.887	0.943	0.937	0.940
	DeepLabv3+ a	0.985	0.837	0.892	0.931	0.911
	U-Net (m) b	0.893	0.801	0.867	0.924	0.891
	SegFormer (m) b	0.921	0.829	0.921	0.893	0.907
Tumor	U-Net a	0.900	0.676	0.887	0.740	0.805
	DeepLabv3+ a	0.922	0.769	0.903	0.839	0.869
	U-Net (m) b	0.893	0.611	0.856	0.681	0.757
	SegFormer (m) b	0.838	0.686	0.838	0.791	0.814

^a Binary semantic segmentation; ^b Multi-class semantic segmentation.

details the performance of the algorithms for the trained segmentation models. For identifying nerves, the U-Net-based binary segmentation model exhibited excellent performance, with an IoU of 0.887. For identifying tumors, the DeepLabv3+ binary segmentation model outperformed other models, with an IoU of 0.769. The overall performance of the multiclass semantic segmentation models was lower than that of the binary semantic segmentation models.

3.2. Region-Wise Performance

By inputting the tumor and nerve masks into the segmentation models, we extracted PNIs based on the distance between tumors and nerves (Figure 2). The pipeline using the multiple segmentation model, Md5, exhibited excellent performance, with an AUC of 0.92 (

Table 3. Performance of PNI classifier according to the various combinations of segmentation models.

Module a	AccuracyR	SensitivityR	SpecificityR	NPVR b	PrecisionR	F1-ScoreR	AUC (95% CI)
Md1	0.85	0.85	0.85	0.85	0.85	0.85	$0.85 \pm 0111$
Md2	0.80	0.85	0.75	0.83	0.77	0.81	$0.80 \pm 0.124$
Md3	0.80	0.75	0.85	0.77	0.83	0.79	$0.80 \pm 0.124$
Md4	0.72	0.75	0.70	0.74	0.71	0.73	$0.72 \pm 0.138$
Md5	0.92	0.90	0.95	0.90	0.95	0.92	$0.92 \pm 0.078$
Md6	0.88	0.80	0.95	0.83	0.94	0.87	0.88$\pm 0.102$

^a Architectures used for nerve and tumor segmentation in each sequential binary model are as follows: Md1: U-Net + U-Net; Md2: U-Net + DeepLabv3+; Md3: DeepLabv3+ + U-Net; Md4: DeepLabv3+ + DeepLabv3+. In Md5, U-Net is adopted for simple multiple segmentation; Md6: SegFormer is adopted for simple multiple segmentation. ^b Negative Predictive Value.

, Figure 3). Moreover, the standard deviation was the lowest among the models, indicating that the multiple segmentation model was more stable than the combined binary segmentation model.

We found that the pipeline using the multiclass segmentation network showed better performance, although the binary segmentation network outperformed the simple multiclass segmentation model regarding pixel-wise performance for tumors and nerves. The combined networks are assumed to cause error accumulation and degrade performance.

3.3. Analysis of False Results

Falsely predicted images using to Md5 are presented in Figure 4. One of the FN regions exhibited relatively small tumor clusters and neural bundles (Figure 4b). In another region, the surrounding inflammatory cells and nerve cells were misclassified as tumor cells (Figure 4d). An FP region included thick blood vessels around a tumor, which were misclassified as PNI. The model falsely identified the smooth muscle cells of the wall of vessels as nerve bundle Schwann cells because of their similar spindle shapes. All the predicted results regarding Md1 to Md6 can be accessed through the web page (http://pni.ssus.work/, accessed on 13 September 2022).

In the current pipeline, falsely predicted results can be classified into six subgroups according to the types of tissue in error. FN prediction occurred in tumor, nerve, or both tissues (Figure S2a,c,e). FP originated from errors in tumor or nerve tissues (Figure S2b,d). The FP result was not observed in either the tumor or nerve tissues of our model. This classification allowed us to intuitively distinguish the tasks the model performed incorrectly, providing the interpretability of false results in the current model.

3.4. Effects of Pre-Training Tasks

We studied the impact of the pre-trained model on the performance of SegFormer (Table S2). The overall performance for identifying tumors increased when the pre-trained model weights were used. For identifying nerves, the F1 score was also improved. With transfer learning, improved performance was obtained.

3.5. Inference Time Comparison

Figure S3b reports the average inference time per patch for each architecture. SegFormer inference achieves the average of 126.754 ms, compared to the time of 1616.037 ms using the U-Net model. The number of parameters in SegFormer is about that in U-Net (3,714,915 vs. 6,251,759 for SegFormer and U-Net, respectively) (Figure S3a).

4. Discussion

In this study, a DL-based hybrid model was developed to detect PNI in CRC patients. The proposed framework exhibited excellent performance (accuracy of 0.92, sensitivity of 0.90, AUC of 0.92), with potential for computer-aided diagnosis in PNI screening. Considering the prognostic implications of PNI and the difficulty in detecting PNI in pathology slide images, the automated PNI detector exhibited a high potential for utility, and can potentially save medical resources.

In practice, PNI detection is a time-consuming and cumbersome task, with high intra- and inter-observer variation. A review of CRC slides in a study revealed that 46 of 55 PNI-positive cases (from a total of 249 cases) were reported as PNI-negative in the original pathology reports [2]. In another study, Peng et al. also revealed the differences in the PNI-positive rate (24.3%) after review compared to the PNI-positive rate (7.5%) recorded in the initial reports [12]. Furthermore, considerable variation is perceived between observers in defining PNI [11,30]. Some of the variations depend on the evaluation criteria among pathologists in terms of the distance between the cancer source and nerve cells [6]. Uncertainty in defining a nerve is another cause of poor inter-observer PNI detection reproducibility. These difficulties in detecting PNI can be solved by standardizing the evaluation criteria in the algorithms and refining the pixel-wise prediction of neural bundles. Thus, expectedly, inter-observer reproducibility would increase, and underestimations could be reduced.

Some attempts have been made to use DL-based approaches to extract PNI histologically. Ström et al. used a convolutional neural network to classify PNI in prostate biopsies, and achieved a discrimination AUC of 0.98 [21]. Recently, an international Medical Image Computing and Computer Assisted Intervention Society-Pathology Artificial Intelligence Platform (MICCAI-PAIP) challenge was held to detect PNI in multiple organ cancers (https://paip2021.grand-challenge.org, accessed on 13 September 2022). The top-ranked team achieved the best F1 score of 41.55% using a feature pyramid network (FPN) [22]. This result revealed the capacity of the multi-resolution network, FPN, to detect PNI in histological images. However, these algorithms, where the representative images of PNI itself were learned, could not provide sufficient interpretation for false prediction because of the “black-box” nature of the DL methods.

An interpretable DL-based model was proposed to provide interpretability through semantic classification of the tissue type and the calculation of distances. This process sequence is similar to a diagnosis by a pathologist, enabling us to interpret the predicted results outputted by the proposed DL-based model, increasing model reliability. Thus, the method provided considerable advantages for medical image analysis and applying the models in practice [31,32].

Numerous challenges exist while using the DL algorithm for clinical applications, including its use in the current model to detect PNI in colon cancer. Therefore, these challenges should be resolved. First, an automated and efficient workflow using digitalized pathology images should be established to effectively use the DL model in practice. This high-cost digital pathology system does not exhibit any benefits, hindering most institutions from establishing this system [33]. Therefore, definite clinical values, such as reduced diagnostic time and improved quality and efficiency, should be achieved in DL implementation to incentivize DL adoption in clinical applications.

Second, safety should be guaranteed in the clinical application of the DL model. The robustness and generalizability of the trained DL model are critical for clinical application. Centralized digitalized data archives, which store large-scale biomedical images from various institutions, can be used to overcome this obstacle. The stored digital images can be used for model validation and bias optimization, allowing the algorithms to achieve generalized performance. Digital pathology guidelines for DL implementation and quality assessment have been established [34,35,36,37]. However, the digital pathology system is typically being established in large-scale university hospitals because the capital outlay for the system cannot be borne by every hospital. Schömig-Markiefka et al. investigated the accuracy of a deep-learning-based algorithm for datasets from various institutions digitized by different scanner systems. Although a model with high overall accuracy of >98% was used, substantial losses occurred in accuracy because of dependence on HE-staining quality, brightness, and contrast [38]. Therefore, national planning and systemic support for developing large-scale centralized biomedical image archives or databases is crucial for future clinical applications.

This study had some limitations. First, the dataset was limited in size. We used 77 WSIs to train the semantic segmentation network for extracting tumors and nerves. However, the proposed model exhibited performance comparable to that of a prior study, where 80k biopsy cores were used to achieve an AUC of 0.98 [21]. Considering the dataset size used in this study and the performance achieved, extremely large-scale data may not be required for the convergence of the proposed pipeline. Second, the results obtained were not externally validated. External validation with additional datasets can improve interpretability and generalizability. Finally, the classifier using distance calculation exhibits a drawback. Since the PNI classifier is a rule-based classifier, distinguishing whether tumor cells actually infiltrated a nerve sheath or are just adjacent to a nerve bundle is not possible. Despite these limitations, the PNI classifier can play a significant role as a screening tool.

Therefore, as a trial of PNI detection, this study provides researchers with new possibilities for the development and improvement of data-driven algorithms for PNI detection. Furthermore, by enabling the detection of accurate PNI status, this study can improve the clinical decisions made for individual patients, positively impacting their prognosis.

5. Conclusions

A novel DL-based approach was proposed to detect PNI in CRC using histopathological images. The hybrid model consists of two components: a segmentation network for tumor and nerve tissues, and a PNI classifier. The proposed framework exhibited high performance (with an accuracy of 0.92, a sensitivity of 0.90, and an AUC of 0.92), as well as the potential for computer-aided diagnosis in PNI screening. Considering the prognostic implication of PNI and the difficulty in detecting it in pathology slide images, the automated PNI detector exhibits significant potential for PNI diagnosis.