Minimum Sample Size Estimate for Classifying Invasive Lung Adenocarcinoma

Abstract

Statistical Learning Theory (SLT) plays an important role in prediction estimation and machine learning when only limited samples are available. At present, determining how many samples are necessary under given circumstances for prediction accuracy is still an unknown. In this paper, the medical diagnosis on lung cancer is taken as an example to solve the problem. Invasive adenocarcinoma (IA) is a main type of lung cancer, often presented as ground glass nodules (GGNs) in patient’s CT images. Accurately discriminating IA from non-IA based on GGNs has important implications for taking the right approach to treatment and cure. Support Vector Machine (SVM) is an SLT application and is used to classify GGNs, wherein the interrelation between the generalization and the lower bound of necessary sampling numbers can be effectively recovered. In this research, to validate the interrelation, 436 GGNs were collected and labeled using surgical pathology. Then, a feature vector was constructed for each GGN sample through the fully connected layer of AlexNet. A 10-dimensional feature subset was then selected with the p-value calculated using Analysis of Variance (ANOVA). Finally, four sets with different sample sizes were used to construct an SVM classifier. Experiments show that a theoretical estimate of minimum sample size is consistent with actual values, and the lower bound on sample size can be solved under various generalization requirements.

1. Introduction

An important task in machine learning and artificial intelligence is to build a classifier for prediction and classification given available training samples or other prior information [1]. Statistical Learning Theory (SLT) plays an important role in prediction estimation and machine learning in that it provides a unified framework for solving finite-sample learning problems [2,3,4]. In the past few decades, Support Vector Machine (SVM), a method based on SLT, has been widely applied for the construction of a classifier when limited samples are available [5,6,7]. Unlike other classifiers, which assume that the number of samples can tend to infinity, SVM offers the best generalization and prediction capability after it is trained using limited samples. SVM is one among many classification methods capable of handling large or small samples, nonlinear, or those with high dimensional issues [8]. A vital question, however, in using SVM is still unresolved, namely, how many samples are necessary for a certain standard of prediction accuracy. To address this question, we devise here a new idea that not only pays attention to the performance of the model on the training set, but also puts additional focus on the model’s prediction accuracy of unknown new data. Typically, a model overfits when it is trained so well on the training data that it fails to generalize. Accordingly, the ability of the model to adapt to new data decreases. It is generally accepted that the primary factor in generalization performance is sample size. However, very little study has focused on studying the quantitative relationship between sample size and generalization. In this paper, we use the medical diagnosis of lung cancer based on computed tomography (CT) images as an example to solve the relationship problem; wherein, it is difficult to obtain large-scale samples in practice.

The construction of any classifier must include training and testing processes according to available samples. In this context, training error refers to the learning ability whereas testing error reflects the prediction and generalization ability [9]. In SLT, a learning error is called an empirical risk (EMP) while a generalization error is called an expected risk (EXP) [10]. Minimizing EMP is a necessary condition for building a classifier with generalization ability or an EXP, but it is not a sufficient condition [11]. In fact, a low EMP may lead to a high EXP, wherein the prediction and generalization ability involves both the number of samples and choice of the classifier type. Under the same EMP, a classifier with simple structure can often lead to a high generalization ability [12]. Nevertheless, SLT can act as the theoretical basis that connects EMP and EXP [13].

Recently, a number of methods have been proposed to diagnose lung cancer using CT images [14,15]. Invasive adenocarcinoma (IA) is a primary type of lung cancer and often presents as ground glass nodules (GGNs) in patient’s CT images [16]. Accurately discriminating IA from non-IA based on GGNs has critical implications for providing an effective treatment plan. Existing studies on classifying IA and non-IA have mostly focused on how minimizing EMP can improve classification accuracy and with very little consideration given to EXP. Moreover, the generalization and prediction ability of the classifier is not guaranteed in practice. To date, little is known about the interrelation between EXP and EMP, and more specifically, it remains unclear as to how many samples are necessary under a certain requirement of prediction accuracy. In this paper, the effect of different sample sizes on EXP is explored, and the minimum sample size is estimated to achieve the desired generalization requirements.

2. Materials and Methods

For this study, we retrospectively collected 436 early GGNs samples from the Tianjin General Hospital, in which the sample rate between IA and non-IA was 216:220. The segmented CT images of GGNs are shown in Figure 1. All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and all experiments were approved by the ethics committee of General Hospital of Tianjin Medical University.

After obtaining a sample set of GGNs, we extracted features from each GGN sample for classifying IA and non-IA. Previous studies have demonstrated how the powerful modeling and feature learning capabilities of deep learning lead to remarkable results [17,18,19,20,21], especially in biomedical image processing applications based on quantitative CT images [22,23,24].

Transfer learning (TL) is a new research area in deep learning. TL transfer the knowledge obtained by a pre-trained network to other image processing tasks [25]. TL methods, when combined with a convolutional neural network (CNN), have been shown as effective solutions to problems associated with relatively small datasets [26]. Specifically, the deep feature extraction capability of TL methods can activate layers of a CNN model to construct the related feature vectors [27]. In the medical field, use of mainstream networks as feature extractors has become a typical and general method to deal with classification problems [28,29,30].

In this work, we use the AlexNet model as the feature extractor to construct a 4096-dimensional feature vector for each GGN sample. The AlexNet model is the first deep CNN model that was trained on the ImageNet competition. It has shown efficacy in image processing [31,32,33] and is convenient in terms of implementation and the ability to fine-tune its processes for new tasks. AlexNet has no complex or deep architectures, which means reduced computation time and the need for robust hardware. In addition, our previous research [34] has shown the effectiveness of features extracted by the fully connected (FC) layer of AlexNet for GGNs classification.

To find the key features in the sample set, we first use the p-value calculated by the Analysis of Variance (ANOVA) as a criterion for feature selection [35]. Typically, lower p-values correspond to larger between-class significant differences. The top ten features with the lowest p-value are selected to form a feature subset.

According to the general principle in SLT, EXP and EMP obey the following relationship with at least (1 − η) probability for a limited number of samples n [36]:

$$R_{exp}(w)\le R_{emp}(w)+\sqrt{\frac{h(\ln (2n/h)+1)-\ln (\mathsf{\eta }/4)}{n}},$$

(1)

where the value of EXP is denoted as R_exp(w) and the value of EMP as R_emp(w).

Equation (1) can be simplified as

$$R_{exp}(w)\le R_{emp}(w)+\Phi (n/h).$$

(2)

where Φ(n/h) is the confidence range; η is confidence limits, expressed as the probability that the classifier predicts the outcome to occur; n is the number of training samples; and h is VC dimension, indicating the maximum number of samples that a selected type of classifier can distinguish all possible 2^h sample combinations [37]. Only few types of classifiers have the known VC dimensions. Essentially, for the most used linear classifiers, the VC dimension is (d + 1), and d is the number of used features. The widely used SVM is just a linear classifier. When dealing with multi-classification problems using the linear classifier, the actual VC dimension in a one-to-rest manner is (d + 1) × c, where c is the number of classes [38]. From Equation (1), the EXP of the classifier consists of two parts: one part is the EMP; the other part is the confidence range, which is related to the VC dimension and the number of training samples. As the number of samples n increases, the confidence interval decreases monotonically, and the probability that Equation (1) holds increases monotonically.

We build four subsets of different sample sizes by random sampling. The SVM, which is the applicable form of SLT, is used for classifying IA and non-IA. We quantitatively estimate the effect of sample size to the generalization ability and compare the results between the computed EXP using Equation (2) and the actual test from the experiments. The schematic diagram of the experimental design is shown as Figure 2.

3. Results

First, 36 samples in the sample set were randomly selected from the overall 436 GGNs samples as a fixed test set. Four training subsets with different but increasing sample numbers were drawn from the remaining 400 samples as training sets; the proportion between positive (IA) and negative (non-IA) samples was 1:1. Therefore, the number of samples in the four group of training sets were 100, 200, 300, and 400, respectively. We repeated four sets of experiments with different training sets of four sample sizes and a fixed test set. With four sets of sample sizes, the 10-dimensional feature subsets obtained from the AlexNet were inputted into SVM for classifying IA and non-IA. The EXP error of GGNs in the test set and the EMP error in the training set under different sample sizes were calculated separately. We compared the actual value from the test set with the theoretical value computed by Equation (1) at 90% (i.e., η = 0.10) and 95% (i.e., η = 0.05) confidence levels. The experimental results are shown in

Table 1. Comparison of theoretical and actual values of R_exp(w).

Number of Samples	Remp(w)	EXP (η = 0.05)		EXP (η = 0.10)		Rexp(w) in Test Set
		Φ(n/h) in Equation (2)	Theoretical Value	Φ(n/h) in Equation (2)	Theoretical Value
n = 100	0	0.6877	0.6877	0.6826	0.6826	0.361
n = 200	0.005	0.524	0.529	0.5207	0.5257	0.222
n = 300	0.003	0.4449	0.4479	0.4423	0.4453	0.194
n = 400	0.002	0.3954	0.3974	0.3932	0.3952	0.139

.

It can be seen from Table 1 that the actual values from the testing samples and the computed theoretical values of R_exp(w) in Equation (1) show a nearly consistent trend as the sample size increases. Their differences gradually decrease with increase in sample size n, while not exceeding the upper limit of the theoretical value of R_exp(w). The value of confidence range Φ(n/h) is inversely proportional to the sample size, given the same number of features used. Experimental results demonstrate that the generalization ability of the invasive classification model is proportional to the sample size.

From the above conclusions, we can see that it is a necessary condition to minimize only EMP, but it is not a sufficient condition when the number of samples is limited. To increase the generalization and prediction capability of the model, the selected types of classifiers must have as small a VC dimension as is possible. Moreover, there is a requirement to minimize the confidence range as well. This process of selecting the classifier model and learning algorithm can be referred to as adjusting the confidence range. However, due to a lack of theoretical guidance, currently this process in most existing methods is based on prior knowledge and experience, resulting in an excessive dependence on researchers.

In practice, another critical challenge for building an effective classifier is the question of the number of samples necessary under a certain requirement of prediction accuracy. According to Equation (1), we continue to explore the lower bound on sample size under various generalization requirements. The experimental results are shown in

Table 2. The relationship between EXP and sample size.

	Number of Classes	n = 3000	n = 6000	n = 9000	n = 12,000	n = 15,000	n = 18,000	n = 21,000	n = 24,000
95% confidence (d = 2)	c = 2	0.1103	0.0833	0.0710	0.0635	0.0583	0.0544	0.0514	0.0490
	c = 4	0.1841	0.1386	0.1176	0.1047	0.0957	0.0890	0.0838	0.0795
	c = 6	0.2157	0.1624	0.1377	0.1225	0.1120	0.1040	0.0978	0.0928
	c = 8	0.2416	0.1820	0.1542	0.1372	0.1253	0.1164	0.1094	0.1037
	c = 10	0.2639	0.1989	0.1686	0.1499	0.1369	0.1272	0.1195	0.1132
90% confidence (d = 2)	c = 2	0.1091	0.0825	0.0704	0.0630	0.0578	0.0540	0.0510	0.0486
	c = 4	0.1835	0.1382	0.1172	0.1044	0.0955	0.0888	0.0836	0.0793
	c = 6	0.2151	0.1621	0.1374	0.1223	0.1117	0.1038	0.0976	0.0926
	c = 8	0.2411	0.1817	0.1540	0.1370	0.1251	0.1163	0.1093	0.1036
	c = 10	0.2634	0.1986	0.1683	0.1497	0.1367	0.1270	0.1193	0.1131

Note: The numbers in orange refer to the achievement of 90% generalization ability.

.

From Table 1, we see that the value of R_emp(w) is inversely proportional to the sample size when the VC dimension and confidence limits are unchanged, and the values are all lower than 1%. Therefore, we suppose R_emp(w) is 1% taking into consideration a degree of classification error, which means that the classification accuracy of the training set reaches 99%. Then, we compute the classification task using only 2 features, meaning the value of d is 2. We take into account that under the binary classification task and the multi-classification task by the linear classifiers, the corresponding VC dimensions are (d + 1) and (d + 1) × c, respectively. When dealing with classification tasks of 2, 4, 6, 8, 10 classes, VC dimension is calculated as 3, 12, 18, 24, 30, respectively. Finally, by plugging the VC dimensions, confidence limits η (η = 0.05, 0.1) and supposed R_emp(w) to Equation (1), the lower limit of the achievable generalization requirements can be calculated under different sample sizes as shown in Table 2. Table 2 also shows the minimum sample size required to meet different generalization requirements under various classification tasks. It also shows that if GGNs are divided into two classes, four classes, and six classes, it respectively requires at least 6000, 12,000, and 18,000 samples to achieve 90% generalization ability when using two features.

Furthermore, we compare the sample sizes between the actually used and the computed theoretical values from Equation (1) under a certain expected prediction accuracy and η = 0.05, where the actually used cases are taken from the research of Diego et al. [39], Mohammad et al. [40], and Dai [41], as shown in

Table 3. The comparison results between the actual and the computed theoretical values on the lower bound of the sample size estimated in Equation (1).

Case	Number of Classes	Number of Features	EXP	Number of Samples (η = 0.05)
				Actual Value	Theoretical Value
Diego et al. [39]	c = 2	d = 2	0.32	301	235
Mohammad et al. [40]	c = 2	d = 10	0.31	1000	776
Dai [41]	c = 2	d = 40	0.47	1000	958

. We can see that all the actual sample sizes are greater than the lower bound of the computed sample sizes. This validates the estimated bound of sample size by Equation (1).

4. Conclusions

SLT plays an important role in building a classifier for prediction and classification in the medical diagnosis of lung cancer under available training samples or other prior information. In this paper, the SLT tool was used to analyze two key issues for design classification, i.e., one regarding the computation of the generalization ability of a classifier based on available samples, and its inverse problem, i.e., how many samples are necessary under a certain requirement of prediction accuracy. The research has guiding significance for establishing a necessary sample size under a certain requirement of prediction accuracy, such as the study of the medical diagnosis of lung cancer. Our experimental results validate our method in the process for classifying IA and non-IA. In the future, we will continue exploring the connection between the type of classifiers and generalization ability with SLT.