ImmunoAIzer: A Deep Learning-Based Computational Framework to Characterize Cell Distribution and Gene Mutation in Tumor Microenvironment

Simple Summary

A comprehensive evaluation of immune cell distribution in the tumor microenvironment (TME) and tumor gene mutation status may contribute to therapeutic optimization of cancer patients. In this study, we aimed to demonstrate that deep learning (DL)-based computational frameworks have remarkable potential as a tool to analyze the spatial distribution of immune cells and cancer cells in TME and detect tumor gene mutations. TME analysis can benefit from the computational framework, mainly due to its efficiency and low cost. Cells distribution in TME and tumor gene mutation status can be characterized accurately and efficiently. This may lead to a reduced working load of pathologists and may result in an improved and more standardized workflow.

Abstract

Spatial distribution of tumor infiltrating lymphocytes (TILs) and cancer cells in the tumor microenvironment (TME) along with tumor gene mutation status are of vital importance to the guidance of cancer immunotherapy and prognoses. In this work, we developed a deep learning-based computational framework, termed ImmunoAIzer, which involves: (1) the implementation of a semi-supervised strategy to train a cellular biomarker distribution prediction network (CBDPN) to make predictions of spatial distributions of CD3, CD20, PanCK, and DAPI biomarkers in the tumor microenvironment with an accuracy of 90.4%; (2) using CBDPN to select tumor areas on hematoxylin and eosin (H&E) staining tissue slides and training a multilabel tumor gene mutation detection network (TGMDN), which can detect APC, KRAS, and TP53 mutations with area-under-the-curve (AUC) values of 0.76, 0.77, and 0.79. These findings suggest that ImmunoAIzer could provide comprehensive information of cell distribution and tumor gene mutation status of colon cancer patients efficiently and less costly; hence, it could serve as an effective auxiliary tool for the guidance of immunotherapy and prognoses. The method is also generalizable and has the potential to be extended for application to other types of cancers other than colon cancer.

1. Introduction

The tumor microenvironment (TME) is the internal environment in which tumor grows [1,2,3]. The understanding of TME, especially the spatial distribution of tumor infiltrating lymphocytes (TILs) within TME, can be of great importance to offer guidance for immunotherapy [4,5,6,7] and evaluating prognosis of cancer [8]. Moreover, tumor gene mutation status is also of vital importance for the prediction of drug efficacy [9], disease-free survival [10], and immunotherapy response [11,12]. As such, a comprehensive evaluation of immune cell distribution in TMEs and tumor gene mutation status may offer guidance for patient selection and therapeutic optimization.

Currently, the histological analysis of hematoxylin and eosin (H&E)-stained tissue is considered the gold standard for pathologic diagnoses. Multiplexed immunohistochemistry (mIHC) techniques can help pathologists interpret clinically relevant cellular subtypes and biomolecules, and are widely applied in TME analysis [13,14,15]. The mIHC-based methods allow simultaneous display of the cellular expressions of several biomarkers on tissue slides; thus, they offer more comprehensive insight into disease heterogeneity. However, mIHC has certain limitations [16]. First, the multiplexed staining method is both time- and cost-consuming. Second, spectrum overlapping can affect the accuracy of the staining results. Third, a professional pathologist is often required to interpret mIHC-based images; thus, the interpretation is subject to individual subjectivity. Therefore, there is an urgent need to develop an alternative and more efficient method to complement the existing mIHC approach to characterize different cell types in the TME.

A computational framework may provide a more comprehensive, accurate, and objective method to assist in therapeutic optimization [17]. The application of deep learning (DL) in medical image processing, especially histological image analysis, has been shown to extract valuable information that is imperceptible to humans [18,19,20]. Recent advances in DL and digital pathology have made it possible to derive fluorescence images from transmitted light images of unlabeled fixed or live biological samples [21], thus demonstrating the capability and reliability of DL methodology as a tool to facilitate biomarker prediction tasks. Burlingame et al. introduced an adversarial deep learning method that they used to infer fluorescence images based on the H&E-stained tissue from a patient with pancreatic ductal adenocarcinoma [22], further verifying the practicability of DL-based methods as a type of prediction tool for complex human tissues. Moreover, previous research has also suggested that a DL neural network can detect gene mutations based on H&E images [23], demonstrating the potential application of DL-based methods in the field of pathological image analysis. Hence, we hypothesized that DL may facilitate the evaluation of immune cell and cancer cell distribution in TMEs by predicting the cellular biomarker distribution, meanwhile offer guidance for tumor gene mutation analysis.

Despite the advancements of DL-based methods in medical image analysis, most existing DL-based methods are fully supervised and require a large amount of labeled data for training to achieve an acceptable level of performance, especially when they are applied to pixel-level segmentation tasks. However, the annotation of medical image data can be very laborious and time-consuming, and it often requires professional pathologists to ensure accuracy [24]. Thus, it would be very beneficial to develop semi-supervised DL- based computational methods [25] to leverage unlabeled image data to increase the efficiency. Moreover, The Cancer Genome Atlas (TCGA) dataset contains a massive amount of H&E staining data from cancer patients; however, it remains underutilized and can be useful for a semi-supervised approach.

In this study, we proposed a computational framework, termed ImmunoAIzer, which can use H&E images to provide comprehensive and valuable information for pathologists to aid TME analysis. ImmunoAIzer contains two components: (1) a semi-supervised cellular biomarker distribution prediction network (CBDPN) which can make predictions of the spatial distribution of CD3, CD20, PanCK, and DAPI on H&E images; (2) a tumor gene mutation detection network (TGMDN), which can detect APC, TP53, and KRAS gene mutations from H&E images. The flowchart of ImmunoAIzer is described in Figure 1. In general, our proposed computational framework can reveal TILs and cancer cells distribution in TME, and facilitate tumor gene mutation assessment. These factors allow the framework to be used as an effective auxiliary tool for the comprehensive evaluation of TME and as a reference for the therapeutic optimization. Furthermore, our method can be extended for application to different cancer types.

2. Materials and Methods

2.1. Dataset Establishment

The data we used in our work includes two parts: (1) data used for CBDPN training and validation, and (2) data used for TGMDN training and validation.

For the CBDPN, the dataset included two types of data: (1) pixel-level labeled H&E image data of 8697 H&E image patches and their corresponding label masks of 8 colon cancer patients from Peking University Cancer Hospital and Institute, and (2) unlabeled image data of 50,801 H&E image patches of 60 colon cancer patients acquired from the TCGA dataset, Colon Adenocarcinoma (COAD) project. For the tissue samples acquired from Peking University Cancer Hospital and Institute, the H&E and mIHC stains were conducted. PanCK was used to characterize tumor cells, and CD3 and CD20 were used to characterize TILs [26], and DAPI for nuclei. The staining and scanning protocols are described in Appendix A. First, in order to facilitate annotation, a registration procedure was performed after acquisition of the H&E and mIHC images to ensure that the H&E and mIHC images were spatially registered. Second, the registered H&E and mIHC whole slide images (WSI) were tiled into 512 × 512-pixel patches, and were pixel-level annotated by 3 board certified pathologists. Third, a color normalization [27,28] procedure was conducted to improve the image quality. Finally, we conducted a data augmentation procedure to obtain 8697 H&E image patches of which the training, validation, and testing image data ratio was set to be approximately 8:1:1, yielding 6937 image patches in the training set, 946 image patches in the validation set, and 814 images in the test set. The whole data processing pipeline is illustrated in Figure S1. The registration procedure and other post-processing details are described in Appendix B. Additionally, for the unlabeled image data from the TCGA dataset, the WSIs of 60 patients were tiled into 512 × 512-pixel patches, yielding 50,801 image patches in total, among which 10,200 image patches of 35 patients with background proportion below 50% were utilized for semi-supervised training, and 40,601 image patches of the other 25 patients were utilized for performance validation.

For the TGMDN, H&E-stained WSIs from 446 colon cancer patients with gene mutations were selected from the TCGA dataset COAD project. In order to make sure that the dataset contains sufficient images with mutations, we chose those which were mutated in at least 10% of the tumor areas. The WSIs were tiled into 512 × 512-pixel patches. For each H&E-stained slide, only patches predicted through our CBDPN to have a PanCK-positive area of more than 50% were selected for implementation. This condition was necessary to choose the images of the tumor area. Finally, we used CBDPN to select 44,534 image patches from 339 patients for the task. The training, validation, and testing image ratio was set to be approximately 14:3:3, respectively, yielding 31,174 H&E image patches in the training set, 6680 H&E images in the validation set, and 6680 H&E images in the test set. All of the H&E patches were intensity-normalized [27,28] prior to training.

2.2. CBDPN Training and Validation

2.2.1. Semi-Supervised Mechanism

In order to tackle the problem of insufficient labeled data and the heavy burden of annotation work, our semi-supervised biomarker distribution prediction network utilizes an adversarial learning paradigm [29], the structure of which consists of a generator and a discriminator. In order to better extract useful features from pathological images and optimize performance, the generator was structured to have an encoder–decoder form. The generator can be any form of segmentation network which takes in the original H&E image and outputs the class probability maps and the discriminator is organized the same as [29]. Note that the generator can also be used as an independent, fully supervised biomarker distribution prediction network. The semi-supervised network structure is illustrated in Figure 2.

Given an input H&E image, $Xn,$ of size H × W × 3, the segmentation network of the generator is denoted as $S\left(·\right)$; the predicted probability map, $S\left(Xn\right),$ of size H × W × C where $C$ is the category number; the applied categories were background, PanCK-positive area, CD3 and CD20-positive area, and DAPI. Note that CD3 and CD20 positive cells were regarded as one class as TILs in our study [26]. The discriminator is denoted as $D\left(·\right)$, which takes in a probability map of size H × W × C and outputs a confidence map of size H × W × 1. In this study, we applied two possible inputs to the discriminator: (1) a biomarker distribution prediction, $S\left(Xn\right),$ and (2) a one-hot-encoded mask vector, $Yn$. Each pixel $p$ of the discriminator output map indicates whether the pixel was sampled from the ground truth labeled by the pathologists p = 1 or from the predictions made by our segmentation network p = 0. In this way, we can leverage both labeled data and unlabeled data to train the network.

The generator was trained by minimizing the multitask loss function:

$$L_{seg}= L_{ce} +{\lambda }_{adv}{L}_{adv} +{\lambda }_{semi}{L}_{semi},$$

(1)

where $L_{ce}$, $L_{adv}$, and $L_{semi}$ represent multiclass cross-entropy loss, adversarial loss, and semi-supervised loss, respectively. In (1), ${\lambda }_{adv}$ and ${\lambda }_{semi}$ are two weighted parameters. As the network is trained on the labeled data, the cross-entropy loss is calculated as:

$$L_{ce}=-{\displaystyle \sum }_{h,w}{\displaystyle \sum }_{c\in C}{Y}_n^{\left(h,w,c\right)}\log (S{\left(X_n\right)}^{\left(h,w,c\right)}).$$

(2)

The adversarial learning strategy is applied to maximize loss $L_{adv}$, as follows:

$$L_{adv}=-{{\displaystyle \sum }}_{h,w}\log \left(D{\left(S\left(X_n\right)\right)}^{\left(h,w\right)}\right).$$

(3)

This loss function was applied to fool the discriminator by maximizing the probability of the predicted results being generated from the ground-truth label mask.

The discriminator was trained by applying the spatial cross-entropy loss of the two causes; this was achieved by performing the following operation:

$$L_D=-{{\displaystyle \sum }}_{h,w}\left(1-y_n\right)\log (1-D{\left(S\left(X_n\right)\right)}^{\left(h,w\right)})+y_n\log \left(D{\left(Y_n\right)}^{\left(h,w\right)}\right),$$

(4)

under the condition that the input is given as the output of the generator, $y_n=0$; alternatively, under the condition that the sample input was taken from the ground truth labels, $y_n=1$. In (4), $D(S{\left(X_n\right)}^{\left(h,w\right)}$ represents the confidence map of $X$ at coordinates $\left(h,w\right)$; $D{\left(Y_n\right)}^{\left(h,w\right)}$ is similarly defined.

Note that $L_{ce}$ was not applied in the case of unlabeled data because there was no ground-truth mask. To prevent the prediction from being overcorrected by the adversarial loss, we applied a smaller ${\lambda }_{adv}$ to the loss function. The segmentation network outputs an initial prediction, $S\left(X_n\right)$, and the discriminator generates a confidence map, $D\left(S\left(X_n\right)\right)$, to indicate the regions that are sufficiently close to the ground-truth biomarker distribution masks. A threshold was applied to establish the trustworthy region, i.e., the region that can be used as the label to reasonably train the segmentation network in a self-taught manner. The resulting semi-supervised loss is defined as:

$$L_{semi}=-{\displaystyle \sum }_{h,w}{\displaystyle \sum }_{c\in C}I\left(D{\left(S\left(X_n\right)\right)}^{\left(h,w\right)}>T_{semi}\right)·{\widehat{Y}}_n^{\left(h,w,c\right)}\log \left(S{\left(X_n\right)}^{\left(h,w,c\right)}\right).$$

(5)

In (5), $I\left(·\right)$ is the indicator function, $T_{semi}$ is the threshold used to select the trustworthy region, ${\widehat{Y}}_n$ is the one-hot-encoded ground truth, and ${\widehat{Y}}_n^{\left(h,w,\mathrm{c}\right)}=1$ if $c=argma{x}_{c}S{\left(X_n\right)}^{\left(h,w,c\right)}$.

2.2.2. CBPDN Structure

In order to better extract features from pathological images, we equipped CBDPN with a combination of residual blocks [30], and Inception V3 [31] architecture as the backbone. The Inception V3 architecture comprises inception units, with each inception unit consisting of several nonlinear convolutional modules at various resolutions, which makes it very useful for pathology-related image-processing tasks. We added two residual blocks after the first max-pooling layer to improve the feature-extraction performance. The network structure is illustrated in Figure 3. In addition, the structure details of each block are described in Figure S3. CBPDN can be used as the generator in the semi supervised mechanism described in Section 2.2.1 and it can also be used independently as a fully supervised segmentation network.

For the discriminator, it consists of 5 convolution layers with 4 × 4 kernel in the stride of 2. The channels of these layers are 64, 128, 256, 512, and 1, respectively. Each convolution layer is followed by batch normalization and Leaky ReLU. An up-sampling layer is added to the last layer to rescale the output to the size of the input map to transform the model into a fully convolutional network.

2.2.3. Test Set Validation

For CBPDN performance evaluation, first, we applied the dataset with labeled images in independent training and testing experiments using CBDPN and three conventional segmentation networks, i.e., UNet [32], DeepLabV3 [33], and DeepLabV3+ [34] in the fully supervised manner, to prove the superiority of our proposed approach in making predictions of the biomarker distribution. Second, to prove the effectiveness of the semi-supervised mechanism, we incorporated CBDPN and the other three networks with the semi-supervised structure. It is worth mentioning that in this case, the segmentation networks were used as the generator of the entire structure described in Figure 2. Further, 10,200 unlabeled H&E image patches were added in the training phase. In addition, 814 H&E images with labels were used for validation.

The metrics used to evaluate the performance of the biomarker distribution prediction network were the dice coefficient, and intersection-over-union (IoU), precision, recall. We used TP, FP, TN, and FN to denote the true-positive, false-positive, true-negative, and false-negative predictions, respectively. In addition, $x,y$ are used to represent the prediction result and the ground truth map, respectively. Further, $k$ denotes the total number of classes. The IoU and Dice metrics are defined as:

$$IoU=\frac{{{\displaystyle \sum }}_{i=1}^k\frac{\left|x_i\cap y_i\right|}{\left|x_i\cup y_i\right|}}{k},$$

(6)

$$dice=\frac{2·{{\displaystyle \sum }}_{i=1}^k\frac{\left|x_i\cap y_i\right|}{\left|x_i\right|+\left|y_i\right|}}{k},$$

(7)

which measure the overlap between prediction $x$ and ground truth $y$.

The precision score was calculated as follows:

$$precision=\frac{{{\displaystyle \sum }}_{i=1}^k\frac{T{P}_i}{T{P}_i+F{P}_i}}{k}.$$

(8)

The recall score was calculated as follows:

$$recall=\frac{{{\displaystyle \sum }}_{i=1}^k\frac{T{P}_i}{T{P}_i+F{N}_i}}{k}.$$

(9)

For each patch, the prediction result and the ground truth label were compressed into one channel and each pixel was labeled to represent one specific marker. For the prediction result, the label with the highest probability of each pixel point being predicted is considered to be the final marker classification of that pixel. In addition, the final metrics were calculated on pixel-level by averaging the metric of each prediction class.

2.2.4. TCGA Dataset Validation

In order to prove the robustness and generalization ability of CBDPN, we used 43,601 H&E image patches of 25 colon cancer patients from the TCGA dataset for further validation. Because the corresponding mIHC staining images of the TCGA WSIs were not available, we were unable to make label masks and calculate the accuracy and dice coefficient. Instead, we calculated the PanCK-positive cell fraction and CD3- and CD20-positive cell fractions given by our optimal model, and then compared the results to the cell fractions determined based on the molecular information available in the TCGA dataset. The TCGA benchmark cell fraction was obtained from molecular genomics assays based on DNA methylation arrays and RNA sequencing data. This is a reasonable approach because PanCK is typically used to characterize tumor cells, and CD3 and CD20 are used to characterize TILs [26].

2.2.5. CBDPN-Based Cell Quantification

In order to prove the utility of CBDPN in clinical scenarios, considering programmed cell death protein 1 (PD-1) is a crucial biomarker for TILs [35,36,37], we also tested the ability of CBDPN to aid quantification of PD-1-expressing cells in TME in the presence of anti-PD-1 immunofluorescence staining. We first acquired H&E images and anti-PD-1 immunofluorescence images of the same tissue. We used CBDPN to acquire cell-specific biomarker distribution in TME, and then we used ImageJ software [38] to preprocess the anti-PD-1 immunofluorescence image to determine PD-1-positive cell distribution. Finally, we merge the cell-specific biomarker distribution generated by CBDPN and the PD-1-positive cell distribution image to gain visualization of the spatial distribution of PD-1-expressing TILs. The quantification can then be achieved by using ImageJ software (Version 1.52p, Fiji-64bit).

In clinical settings, pathologists often use mIHC staining images and Inform software [39] (PerkinElmer, Waltham, MA, USA) to characterize and quantify the PD-1 expression in TME. Inform software (Version 2.4) uses mIHC image as input to experimentally measure cell proportion. The basic workflow is that Inform software first localizes nuclei distribution on the mIHC WSI and then detects the type of fluorescence marker closest to each nucleus to determine the biomarker expression status at cellular level.

To validate the accuracy of our CBDPN-based cell quantification method, we compared our results with the results calculated using the commercial Inform software based on mIHC WSIs of the same tissues.

2.3. TGMDN Training and Validation

2.3.1. TGMDN Structure

For the TGMDN, we used the ShuffleNet V2 [40] network to conduct the multilabel training process to detect APC, TP53, and KRAS mutations in colon cancer patients. ShuffleNet V2 was selected because it has demonstrated good performance on gene mutation detection tasks [41]. The network learned the features of each of the three mutations, with each mutation classification corresponding to an independent binary classification. We implemented a binary cross-entropy loss function and sigmoid layer to replace the softmax layer; this allowed each H&E patch to be associated with several binary labels. The entire network structure is illustrated in Figure 4.

2.3.2. TGMDN Evaluation

To test the performance of the TGMDN, we tested the detection accuracy on the test set containing 6680 H&E image patches from TCGA dataset and calculated the receiver operating characteristic (ROC) of these three mutations.

Moreover, in order to better understand the gene mutation association, we also employed a t-SNE method [42], utilizing the values of the last fully connected layer of our TGMDN as input, to visualize how the mutations and patches were organized in the multidimensional space of the network.

2.4. Implementation Details

Our method is implemented based on the PyTorch DL framework using Python [43]. Two pieces of TESLA V100 GPUs were employed to accelerate the training. For semi-supervised CBDPN training, the batch size was set as 8 and the initial learning rate was set as 0.001 for the generator, and 10⁻⁶ for the discriminator. ${\lambda }_{semi}$ was set as 0.1; ${\lambda }_{adv}$ was set as 0.01 and 0.001 for labeled and unlabeled data, respectively, and $T_{semi}$ was set as 0.5. To train the predictive network, we use the Adam optimizer method, where the momentum is 0.9, and the weight decay is 10⁻⁸. We first train the generator for 10,000 steps and then train the generator and discriminator simultaneously for 300,000 steps. We start the semi-supervised training after training 100,000 steps with labeled data. For TGMDN training, we implemented the RMSprop optimization to train the network for 100 epochs with batch size as 16. We set the momentum as 0.9, and the weight decay as 10⁻⁸. The initial learning rate is 0.01. Statistical analysis is conducted using Prism 6 software.

3. Results

3.1. CBDPN for the Prediction of Cellular Biomarker Distribution in TME

3.1.1. Fully-Supervised Experiment Results

We used CBDPN, UNet, DeeplabV3, and DeeplabV3+ to train the biomarker prediction model. The results on our test set (including 814 H&E image patches) are summarized in

Table 1. Fully-supervised learning comparisons of the four networks on the test set.

Network	Accuracy	Precision	Recall	Dice	IoU
UNet	0.873	0.799	0.871	0.825	0.714
DeepLab V3	0.856	0.776	0.836	0.797	0.677
DeepLab V3+	0.863	0.778	0.858	0.805	0.690
CBPDN	0.875	0.806	0.865	0.827	0.717

The bold font indicates the best result.

. As observed, our CBDPN yielded optimal results in the fully supervised experiments. Our proposed network achieved the highest scores for accuracy, precision, IoU, and dice among the four networks. Although CBDPN did not achieve the highest score for recall, it achieved the best dice coefficient, which provides information on the trade-off between precision and recall. The results prove the effectiveness and reliability of CBDPN in predicting cell biomarker distributions.

3.1.2. Semi-Supervised Experiment Results

We incorporate CBDPN, UNet, DeeplabV3, and DeeplabV3+ with the semi-supervised structure to train the biomarker prediction model. Note that in this case, the segmentation networks were used as the generator of the whole structure. By incorporating with the semi-supervised structure, CBDPN was demonstrated to be capable of predicting cellular biomarker distributions that are similar to the corresponding mIHC staining results (Figure 5). An example of an H&E-stained slide is shown in Figure 5A; the corresponding mIHC image and prediction generated by CBDPN are shown in Figure 5B,C, respectively. In Figure 5D, we compared results generated by CBDPN and other three conventional segmentation networks under the semi-supervised structure. The visualization of the feature maps obtained after softmax operation are illustrated in Figure S4. It can be seen that our method yielded cellular biomarker distribution prediction results that were similar to the results of mIHC staining analysis.

The quantified results of our test set (including 814 H&E patches with labels) are summarized in

Table 2. Semi-supervised learning comparisons of the four networks on the test set.

Network	Accuracy	Precision	Recall	Dice	IoU
UNet	0.891	0.804	0.863	0.824	0.711
DeepLab V3	0.868	0.789	0.859	0.813	0.701
DeepLab V3+	0.872	0.803	0.865	0.823	0.712
CBPDN	0.904	0.854	0.901	0.872	0.788

The bold font indicates the best result.

. As observed, our CBDPN also achieved optimal results in the semi-supervised experiments. The results of all performance indicators revealed that CBDPN consistently outperformed the other networks. Our semi-supervised strategy also proved to be effective in the biomarker distribution prediction task, as it increased accuracy by 2.9%, up to 90.4% in the case of CBDPN. Increases were also observed in the tests conducted using the other three networks. The semi-supervised mechanism enables unlabeled images being trained in a self-taught manner, so that more image data can be leveraged in the training phase.

3.1.3. TCGA Dataset Validation Results

To confirm the generalizability and robust of our model, we also used data of 25 patients from TCGA dataset for validation. A prediction result visualization of TCGA whole slide image (WSI) is shown in Figure 6. Board certified pathologists were asked to make judgements of the spatial distribution of TILs on the H&E WSI. It can be seen that TIL clusters predicted by our method is consistent with the judgment of pathologists. We compared proportions of cancer cells and TILs calculated based on CBDPN predictions and molecular information provided by TCGA. The details are illustrated in Figure 7. The scatter plot is shown in Figure 7A, p values analysis and Pearson correlation coefficients are shown in Figure 7B. Pearson correlation coefficients for PanCK and TILs experiments are 0.7942 and 0.5875, respectively, accompanied by significant p values. The results of the statistical analysis revealed agreement between the cell proportions calculated by the CBDPN and those determined based on the molecular information of TCGA dataset. The number of image patches of the WSIs used in this experiment is described in Figure 7C and races and genders distribution of these 25 patients is shown in Figure 7D. These results prove that our proposed biomarker prediction network has good generalizability and robustness across various tissue samples.

3.1.4. CBDPN-Based Cell Quantification Analysis

PD-1, a critical immune checkpoint molecule, plays an important role in determining the TME immune status. In this study, we tested the ability of CBDPN to help quantify PD-1 expression in TME in the presence of anti-PD-1 immunofluorescence staining.

In total, all of the 94,662 H&E image patches of WSIs of four colon cancer patients from Peking University Cancer Hospital and Institute were analyzed by CBDPN in the cell quantification experiment. One sample image is shown in Figure 8. Figure 8A shows the H&E image and Figure 8B shows the anti-PD-1 immunofluorescence-stained image of the same tissue. The prediction result generated by CBPDN is shown in Figure 8C, and the merged image showing PD-1 expressing TILs is shown in Figure 8D. The sizes of the four WSIs are described in Table S1. We compared results obtained via our approach, and those obtained by applying the commercial Inform software to analyze the mIHC image of the same tissue. CD3 and CD20-positive cell percentage comparison results are described in Figure 9A,B. In addition, PD-1-expressing CD3 and CD20-positive cell percentage comparison results are described in Figure 9C,D. The results showed that most of the cells expressing PD-1 were TILs. Further, agreement can be seen between results based on CBDPN and Inform software, with Pearson correlation values as 0.9998 and 0.9834 for the two quantification experiments accompanied by highly significant p values. This further proves that our proposed method can help quantify TILs and PD-1-expressing TILs in TME accurately. Moreover, quantification using CBDPN was achieved at a lower cost and faster speed than conventional methods based on mIHC, and thus it can be beneficial for TME analysis. We believe that, given the ability of our network to quantify PD-1 expression, CBDPN could also be used to facilitate the quantification of other biomarkers in the presence of certain immunochemistry stains.

3.2. TGMDN for the Detection of Tumor Gene Mutations

3.2.1. Detection of Tumor Gene Mutations from H&E Images

We next focused on predicting APC, KRAS, and TP53 gene mutations in the colon cancer tumor areas by using H&E image data as the input to a multilabel prediction network. We selected 446 patients in total, and the gene mutation status of the dataset is illustrated in Supplementary Figure S2. In total 44,534 H&E patches were used in the gene mutation detection experiment, of which 31,174 H&E image patches were included in the training set, 6680 H&E image patches in the validation set, and 6680 H&E image patches in the test set.

The receiver operating characteristic (ROC) curves are shown in Figure 10. The area-under-the-curve (AUC) values for APC, TP53, and KRAS were 0.76, 0.79, and 0.77, respectively, indicating that these three gene mutations were detectable in the tumor areas analyzed by TGMDN.

3.2.2. Visualization of Network Features

The cluster results based on t-SNE are shown in Figure 11A–C, with each dot representing an image patch and its color intensity indicating the probability of our model to predict the tumor gene mutation. The H&E patch-embedded representation (Figure 11D) can be used to visualize patches predicted to have similar mutations, and thus possess the potential to help reveal the associations of gene mutations. In Figure 11D, the clusters in the top-left enlargement comprise patches constituting APC and TP53 gene mutations. The clusters in the center reveal the co-existence of TP53 and KRAS gene mutations. Finally, the clusters in the bottom-right enlargement comprise patches that confirm the co-existence of APC and KRAS gene mutations. These findings suggest that our ImmunoAIzer-TGMDN can be used to identify such genotype–phenotype association in colon cancer.

4. Discussion

In this work, we proposed a computational framework, termed ImmunoAIzer, which takes H&E image patches as inputs and consists of two components: (1) CBDPN to make predictions of the spatial distribution of CD3, CD20, PanCK, and DAPI in TME, and (2) TGMDN to detect APC, TP53, and KRAS mutations.

Our study demonstrates that CBDPN can make predictions of distributions of CD3, CD20, PanCK, and DAPI with an accuracy of 90.4%, demonstrating that it can be used to reveal the spatial distributions of TILs and cancer cells in TME, and is therefore of significant value as a tool for TME analysis. To maximize the utility of our proposed ImmunoAIzer framework as a tool for clinical application, we incorporated a tumor gene detection network. We used TGMDN to predict the mutation status of three of the most common mutations in TCGA COAD project, namely, APC, TP53, and KRAS. The predicted AUC values for these three mutations were 0.76, 0.79, and 0.77, respectively; these results demonstrate the feasibility of TGMDN as a tool to predict tumor gene mutation status. In general, our approach demonstrates that DL-based methods can be reasonably applied to predict the spatial distribution of TILs and cancer cells in TME, meanwhile to detect gene mutations of colon cancer patients based on histological image data. This can be highly beneficial for the guidance of immunotherapy and prognoses.

The spatial distribution of TILs and cancer cells in TME is of crucial importance for the guidance of immunotherapy and prognosis. The acquisition of such information often requires mIHC staining, which can be time- and cost-consuming, and the interpretation of the staining result can be compromised by individual subjectivity; this highlights the necessity for the development of a standardized methodology to provide pathologists with comprehensive references for TME analysis. Despite advances in DL-based computational methods, current fully supervised DL-based methods often require large amount of annotated image data to achieve good generalization ability and robustness. However, it can be extremely time consuming to annotate each cell on histological image in practice. Thus, our proposed computational framework ImmunoAIzer includes a semi-supervised CBDPN to leverage both labeled and unlabeled image data to make accurate and robust predictions of spatial distribution of TILs and cancer cells in TME, which highly reduces the burden of annotation work and makes it more suitable for clinical use. The increasing data size can contribute to the robustness of the model and improve the generalization performance as well as prediction accuracy on the test set. These account for the improvement of models when switching from fully to semi-supervised strategy. Both CBDPN and UNet adopted the idea of multi-scale feature fusion, which makes both of them superior in pathological image data processing tasks. However, CBDPN not only conducts a feature concatenation process between the encoder and decoder, but also used a series of multi-resolution convolutional modules within the encoder structure to enhance the feature extraction performance, which made the performance advantage of CBPDN more obvious when the data size is increased. The results demonstrate the effectiveness of the semi-supervised mechanism to make full use of information encoded in the unlabeled image data to boost prediction performance. To the best of our knowledge, ImmunoAIzer is the first DL-based computational framework that utilizes a semi-supervised learning strategy to make predictions of the spatial distribution of immune cells and cancer cells in TME. Moreover, the incorporation of anti-PD-1 immunofluorescence staining image enables the CBDPN to be used to aid quantification of the PD-1-positive cells in TME. The implementation of CBDPN offers significant time and cost savings compared to stain-based methods. These findings demonstrate that the ImmunoAIzer can serve as an auxiliary tool for providing guidance for TME analysis.

Meanwhile, the results of incorporating a gene mutation detection network that enables detections about APC, TP53, and KRAS mutations revealed the potential of computational methods for implementation in gene mutation detection tasks. The t-SNE cluster results provided further insight into the associations between APC, TP53, and KRAS mutations; this type of information can be well utilized in tumor gene mutation association studies. Thus, this study is also useful as a reference for tumor gene mutation analysis, and as a general introduction to computational methods as a valuable tool for clinical usage.

The necessity of a tool with the ability to increase the amount of useful information that can be acquired from histological images is increasingly being acknowledged as critical for therapeutic optimization and patient selection [44]. ImmunoAIzer is reproducible and provides an efficient tool for the cellular biomarker distribution prediction and tumor gene mutation detection. Moreover, the dataset consists of H&E images with pixel-level labels which can be used to facilitate development of other DL-based computational tools.

It is worth mentioning that our model was initially trained on H&E data from colon cancer patients, the performance of the model on other cancer types has yet to be validated. However, we believe this methodology is generalizable and could be used to analyze data of other cancer types. In our future work, we will extend the application of ImmunoAIzer to other cancer types and immune biomarkers. For the network structure, the attention mechanism [45] can be further incorporated into the convolution process to optimize the model and thus improve the prediction accuracy. We hope that by applying our computational framework to recognize a wider range of pathological features in TME, we will be able to gain more insights and offer more guidance for therapeutic optimization and prognoses.

5. Conclusions

ImmunoAIzer can utilize H&E images to provide comprehensive information about spatial distribution of TILs and cancer cells in TME, meanwhile detect tumor gene mutations as APC, KRAS, and TP53. Our work demonstrates the potential of DL-based tools to aid pathologists in clinical use.