Image Information Contribution Evaluation for Plant Diseases Classification via Inter-Class Similarity

Abstract

Combineingplant diseases identification and deep learning algorithm can achieve cost-effective prevention effect, and has been widely used. However, the current field of intelligent plant diseases identification still faces the problems of insufficient data and inaccurate classification. Aiming to resolve these problems, the present research proposes an image information contribution evaluation method based on the analysis of inter-class similarity. Combining this method with the active learning image selection strategy can provide guidance for the collection and annotation of intelligent identification datasets of plant diseases, so as to improve the recognition effect and reduce the cost. The method proposed includes two modules: the inter-classes similarity evaluation module and the image information contribution evaluation module. The images located on the decision boundary between high similarity classes will be analysis as high information contribution images, they will provide more information for plant diseases classification. In order to verify the effectiveness of this method, experiments were carried on the fine-grained classification dataset of tomato diseases. Experimental results confirm the superiority of this method compared with others. This research is in the field of plant disease classification. For the detection and segmentation, further research is advisable.

1. Introduction

At present, smart agriculture and digital agricultural technology have provided a lot of help in improving crop production and scientific planting [1]. One important application is the identification of plant diseases, the existence of pests and diseases has seriously affected the world’s food production. Timely control and prevention can avoid losses to the greatest extent. Generally speaking, the identification of pests and diseases needs to be entrusted to experts, which will cost a lot. In addition, if it is not discovered, entrusted, and prevented in time, will expand the loss [2]. Therefore, intelligent agriculture and digital agriculture have been well developed. Combining the plant disease identification and computer vision methods, can prevent timely and accurately. Finally, the purpose of reducing costs is achieved [3,4]. In general, the combination of deep learning and plant disease identification has been widely used [5,6,7,8,9], and has achieved many good results. However, plant disease identification combined with deep learning also faces the problems of insufficient data and inaccurate classification of fine-grained similar classes [10,11].

In order to solve the problem of insufficient data, scholars have proposed many deep learning methods based on few-shot learning. Active learning is a technology to solve the shortage of images in the field of high image labeling costs [12,13,14]. The process of active learning is that the network selects the most valuable images to improve the performance of the model from unlabeled images according to the image selection strategy [15,16,17,18]. The most widely used active learning image selection strategies are methods based on uncertainty [19,20], diversity [21,22,23], and model parameters change [24]. The method based on uncertainty is to select the images with the most inaccurate network prediction. The method based on diversity is to select the images with the lowest redundancy with the existing images. The method based on model parameters change is to select the images that can most affect the model parameters. In addition, the existing few-shot learning methods are mainly based on data enhancement and metric learning [25,26].

The method based on data enhancement is to use the existing image to create more images, so that the neural network can achieve a better generalization effect, which is often applied in the network training stage [27]. The few-shot learning method based on metric learning completes the classification task by measuring the distance between the images in the train set and the images in the test set. Prototype network [28] is a metric learning method proposed by Snell. It calculates the average value of the characteristics of all images in each class of image as the characteristic prototype representing this type of image, and calculates the Euclidean distance from the image characteristics in the test set to the characteristic prototype to predict the label of this image. The relationship network [29] is proposed by Sung. Through the relationship network composed of two modules, the similarity scores of the two images are calculated, and the similarity scores are used to judge whether the two images are from the same class.

Fine-grained image classification is a more detailed sub-classification of coarse-grained classes. Take the identification of tomato leaf diseases as an example. All the images belong to the coarse-grained class of tomato leaves. It is needed to specifically subdivide whether they are healthy leaves or which disease leaves. The classification and recognition of plant diseases is mostly based on the nuances of the same species, which will bring to the classification problem [30,31,32]. In order to solve the fine-grained classification problem of specific classification of sub classes under such a large class, researchers have proposed some methods base on the models. The initial method is to widen and deepen the network for fine-grained classification, and to improve the representation ability of the network for fine-grained image classification. On this basis, some people also use the method of network integration [33,34] to improve the accuracy of fine-grained classification and discrimination by using multiple neural networks. They also proposed the high-order coding method of convolution features [32,35,36] to convert the CNN features to high-order and then classify them.

In general, in order to improve the accuracy of intelligent recognition of plant diseases and pests, we need to pay attention to few-shot learning and fine-grained classification at the same time. The existing model-based fine-grained classification problems do not take into account the number of images and the image information quality. Fine-grained classification is the process of further classifying samples with high similarity. From the perspective of image information quality, adding images near the decision boundary of the more confusing category is more helpful to establish a clearer decision boundary. The present research proposes an image contribution evaluation method based on the analysis of inter-class similarity. This method first calculates the similarity between the various classes of images, then evaluates the information contribution of the images according to the inter-class similarity relationship. At the same time, the image information contribution evaluation method is combined with the active learning strategy. This few-shot learning method of plant diseases can use fewer images to achieve a better recognition effect.

Our contributions are as follows:

(1)

We propose an image information contribution evaluation method, which focuses on the inter-class similarity, defines the images located on the decision boundary between high similarity classes as high contribution images. This can effectively alleviate the problem of inaccurate fine-grained classification of plant disease identification.

(2)

We combine the image information contribution evaluation method with the active learning image selection strategy, which can effectively solve the problem of insufficient data for plant disease identification.

(3)

We have carried out experiments on plant disease datasets. This method has achieved better experimental results than the traditional active learning methods. It can also achieve better experimental results with fewer data, which can provide guidance for the collection and annotation of plant disease datasets.

2. Materials

2.1. Dataset

In the plant disease classification, most of them encounter fine-grained image classification. In order to facilitate the verification of the effect of this method on fine-grained plant disease classification, we selected the tomato disease dataset. The data of this dataset is selected and sorted out from PlantVillage dataset [37]. In order to distinguish, we call the newly established dataset tomat10 dataset (T10 dataset). Some data of the T10 dataset are shown in Figure 1.

T10 dataset contains 10 categories of tomato leaves, including 1 category of healthy leaves, 4 fungal diseases (Tomato early blight, Tomato leaf mold, Tomato septoria leaf spot, Tomato target spot), 1 bacterial disease (Tomato bacterial spot), 1 mold disease (Tomato late blight), 1 mite disease (Tomato spider mites two-spotted spider mite) and 2 viral diseases (Tomato mosaic virus, Tomato yellow leaf curl virus). Each class of T10 dataset contains 500 samples, a total of 5000 images. The pixel size of each image is adjusted, and the size is 128 × 128.

Characteristics of tomato diseases mainly focus on the color and shape of leaves. Tomato early blight and tomato late blight are the two typical tomato diseases. The disease spots of tomato early blight have obvious concentric rings, which appear dark green or yellow at the beginning, and grow a black mold layer when it is wet. The disease spots of tomato late blight are dark green at first, irregular water stains, and gradually expanded to brown. According to the color and shape of tomato leaf disease spots, the types of tomato leaf disease can be better distinguished. Figure 1 also shows that each tomato leaf disease has its own characteristics.

2.2. Dataset Segmentation

First, we divide 5000 images into pool set and test set according to the ratio of 4:1, take 20% of the pool set as the initially labeled dataset, and set the budget of each tag to 20% of the pool set. For the first time, 800 images are randomly selected for training, and then 800 images are screened and added to the train set according to different methods each time. In other words, the data of the test set is 1000, the data of the five train sets are 800, 1600, 2400, 3200 and 4000.

The special note is that the number of images in the test set and the initially labeled dataset is balanced. The remaining image selection methods default to no label information, and the number of each class of images may be different when added. This is more in line with the actual dataset collection and labeling process.

3. The Proposed Method

We propose an image information contribution evaluation method for fine-grained classification of plant diseases. This method fully considers the similarity difference between image categories in fine-grained image classification, evaluates the similarity contribution of unlabeled images according to the similarity difference of image categories. Select and label pool images in combination of the evaluation results and labeling budget. The final training set is composed of the initially labeled dataset and the labeled images selected from the pool set. The image selection strategy is shown in Figure 2.

Next, we will introduce the construction method of active learning fine-grained image classification dataset based on the inter-class similarity. In Section 3.1, we will introduce the inter-class similarity evaluation method, in Section 3.2, we will introduce the image information contribution evaluation method.

3.1. Inter-Class Similarity Evaluation

The inter-class similarity evaluation is to obtain the similarity matrix between various images by using the statistical characteristics of the initially labeled dataset. When calculating, first calculate the core area of each class, and then calculate the statistical characteristics of other images entering the core area. After normalization, obtain the similarity matrix between various images. Then calculate the overall similarity between each image class and other images according to the similarity matrix. The specific calculation method is as follows:

First, the feature prototype $c_i$ is calculated by using the characteristics of images $f_i^{\left(k\right)}$ in each class in the initially labeled dataset as the center of this class:

$$c_i={\displaystyle \frac{{\displaystyle \sum _{k=1}^{n_i}}{f}_i^{\left(k\right)}}{n_i}}$$

(1)

where, $f_i^{\left(k\right)}$ represents the characteristic of the k-th image in class i, $c_i$ represents the center of class i and $n_i$ represents the number of class i images.

The core area radius of a class is expressed by the mean of the distance from the image feature to the prototype of the class of images:

$$c_{r_i}={\displaystyle \frac{{\displaystyle \sum _{k=1}^{n_i}}{\left|\right|c_i-f_i^{\left(k\right)}\left|\right|}_2}{n_i}}$$

(2)

Calculate the entry of each image into other core areas in the initially labeled dataset:

$$d(f_i^{\left(k\right)},c_j)={\left|\right|f_i^{\left(k\right)}-c_j\left|\right|}_2$$

(3)

$$m(x_i^{\left(k\right)},c_{\left(j\right)})=\left\{\begin{array}{c}1,d(f_i^{\left(k\right)},c_j)<c_{r_i}\hfill \\ 0,d(f_i^{\left(k\right)},c_j)\ge c_{r_i}\hfill \end{array}\right.$$

(4)

Get the statistical characteristics of inter-class similarity:

$$m_{ij}=\left\{\begin{array}{c}{\displaystyle \sum _{k_i=1}^{n_i}}m(x_i^{\left(k_i\right)},I_j)+\sum _{k_j=1}^{n_j}m(x_j^{\left(k_j\right)},I_i),i\ne j\hfill \\ 0,i=j\hfill \end{array}\right.$$

(5)

Secondly, normalize the inter-class similarity matrix of images in the dataset:

$$m_{ij}^{{}^{\prime }}={\displaystyle \frac{m_{ij}}{{\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^n}{m}_{ij}}}$$

(6)

$$M=M_{n\ast n}={\left(m_{ij}^{{}^{\prime }}\right)}_{n\ast n}$$

(7)

where, n is the number of classes of images in the dataset.

Finally, measure the similarity between each class of images and all other classes of images:

$$s_i={\displaystyle \sum _{j=1}^n}{m}_{ij}^{{}^{\prime }}$$

(8)

$$s_i^{{}^{\prime }}={\displaystyle \frac{s_i}{{\displaystyle \sum _{i=1}^n}{s}_i}}$$

(9)

$$S=S_{n\ast 1}={\left(s_i^{{}^{\prime }}\right)}_{n\ast 1}$$

(10)

3.2. Image Information Contribution Evaluation

High information contribution refers to the image that can provide more information for the network. If task indicators are oriented, it is the high information image that can better serve the model training. On the contrary, low information contribution has a large overlap with the existing images in the train set, or the image itself can provide less help for network classification. For low information images, if task indicators are oriented, they are images that help less to improve the performance of model classification tasks.

The core idea of unlabeled image information contribution evaluation is to evaluate the contribution of adding this image to the train set to the establishment of the class decision boundary without supervised information. It is more inclined to select images that enter the core area of more classes and the core area of class that is more easily confused with other classes for labeling. The implementation method is as follows:

First, calculate the distance between the unlabeled image feature vector $g_i^k$ and the centers of each class:

$$d(g_i^{\left(k\right)},I_j)={\left|\right|g_i^{\left(k\right)}-c_j\left|\right|}_2$$

(11)

Secondly, compare the distance with the radius of the core area, and get the situation that the image enters the core area of each classes:

$$B=B_{n\ast 1}={\left(b_j\right)}_{n\ast 1}$$

(12)

$$b_j=\left\{\begin{array}{c}1,d(g_i^{\left(k\right)},I_j)<c_{r_i}\hfill \\ 0,d(g_i^{\left(k\right)},I_j)\ge c_{r_i}\hfill \end{array}\right.$$

(13)

Finally, add the similarity evaluation results of the classes where the image is located at the decision boundary to obtain the similarity contribution of the image:

$$D_j=B\ast S^T$$

(14)

4. Experiments

4.1. Experimental Parameter Setting

In the present research, training is carried out on a 3.2-GHz CPU and a Titan Xp GPU. In each cycle, the model is trained for 200 epochs with cosine annealing learning rate. Resnet-18 [38] is used in both feature extraction network and training network. Pay attention to ensure that the batchsize in the comparative experiment is consistent with the training enhancement method to ensure the effectiveness of the comparative experiment.

4.2. Validation Experiment of Image Contribution Evaluation Method

In the present research, we propose an image information contribution evaluation method, which is based on the results of inter-class similarity evaluation. When calculating the image contribution, we believe that the image between the two classes of images with high similarity is a high information contribution image, and the image between the two classes of images with low similarity, or within a class of images is a low contribution image. In order to verify the effectiveness of the image contribution evaluation method proposed in the present research. Under the same budget, we compare the experimental results of high contribution images, low contribution images, and randomly selected images. Each experiment is repeated 3 times, and the experimental result is the average value of these three experiments. The experimental results are shown in

Table 1. Validation experiment on T10.

$\mathit{Method}$	20%	40%	60%	80%	100%
Randomly		88.27%	91.17%	92.70%
High information contribution	84.13%	90.87%	93.03%	93.87%	93.97%
Low information contribution		87.03%	89.73%	91.77%

.

Through the analysis of the results in Table 1, it can be seen that the experimental effect of selecting high contribution images is the best, and the experimental effect of selecting the low contribution images is the worst.

In addition, the experimental effect of screening 40% of the high contribution images is better than that of selecting 60% of the low contribution images, and the difference between the number of labeled training data is more than 800. Using this method to select and label data, we can achieve a higher test accuracy with less labeled data. It fully proves the effectiveness of the image information contribution evaluation method based on the inter-class similarity proposed in the present research.

4.3. Comparative Experiment

The image information contribution evaluation method based on inter-class similarity evaluation proposed in the present research can be combined with active learning strategies, we compare this method with traditional active learning methods. The experimental methods involved in the comparison include uncertainty based method [20] and diversity based method [21]. Each experiment is repeated 3 times, and the experimental result is the average value of these three experiments. The comparative experimental results are shown in Figure 3.

Through the analysis of the comparative experimental results, it can be seen that the test accuracy of all active learning image selection methods is higher than that of random selecting methods, and the image selection strategy formed by the image information contribution evaluation method based on the inter-class similarity evaluation proposed in the present research has the best experimental effect.

5. Discussion

In the discussion part, we will discuss the motivation and reasons.

5.1. Motivation

In terms of agricultural plant disease control, digital and intelligent methods such as deep learning are needed. However, the method combined with deep learning faces the problems of insufficient data and inaccurate fine-grained classification. Most of the existing fine-grained classification methods are implemented by enhancing the network, and the consideration of image information quality is still insufficient. In order to supplement this research, it is necessary to propose an image information quality evaluation method for fine-grained classification to guide the collection and annotation of data.

5.2. Reasons

The reason why this method can achieve good results is that it fully considers the image inter-class similarity, and assigns higher information contribution scores to the images near the class decision boundary with high similarity. At the same time, the images around the more class decision boundary have higher contribution scores. In this section, it will be further analyzed in combination with the experimental results.

5.2.1. Class Similarity Calculate Results

Figure 4 shows the similarity evaluation results between classes calculated from the initially labeled dataset. In order to more intuitively understand the similarity evaluation results, we analyze the similarity between classes from the perspective of maximum and minimum values. The analysis results are shown in Figure 5.

From the results of similarity evaluation, the prominent characteristics are as follows: For the images of “Healthy” (Class 0), the inter-class similarity evaluation results between them and the images of “Tetranychus urticae” (Class 9) are the largest, and the two classes of images are the most similar. For the images of “Target spot” (Class 4), the inter-class similarity evaluation results between them and the images of “Tetranychus urticae” (Class 9) are the largest, and the two classes of images are the most similar. For the images of “Tetranychus urticae” (Class 9), the inter-class similarity evaluation results between them and the images of “Target spot” (Class 4) are the largest, and the two classes of images are the most similar. The inter-class similarity evaluation results between “Healthy” (Class 0) images and “Septoria leaf spot” (Class 3) images are the smallest, and the two classes of images are the most dissimilar.

Figure 6 shows the actual situation of tomato leaves, which is consistent with the inter-class similar evaluation results.

5.2.2. The Models Test Results

Figure 7 shows the test results of the models which are trained on the initially labeled dataset. Figure 8 shows the mean value of the test results of three training times for three classes of images in the initially labeled dataset and marks the maximum and minimum values of the number of false predictions.

Comprehensively consider the mean value of the test results of the three models. Class 0 (“Healthy”) images will be easily predicted as Class 9 (“Tetranychus urticae”). Class 4 (“Target spot”) images will be easily predicted as Class 9 (“Tetranychus urticae”). Class 9 (“Tetranychus urticae”) images will be easily predicted as Class 4 (“Target spot”). There is no misclassification between images of Class 0 (“Healthy”) and Class 3 (“Septoria leaf spot”). It is basically consistent with the results of the inter-class similar evaluation.

5.2.3. Different Budget Test Results

Figure 9 shows the test results of the model trained with different size train sets formed according to the random method and the method proposed in this paper.

Table 2. Comparison of the mean value of the number of images misclassified by different methods.

Truth = 0 & Prediction = 9		Truth = 4 & Prediction = 9		Truth = 9 & Prediction = 4
Randomly	Proposed	Randomly	Proposed	Randomly	Proposed
15.00	6.00	10.67	7.67	2.67	2.67

is the mean value of the number of images misclassified between the above confusing classes under different budgets. Compare the test results between the random method and the method proposed in this research.

From the results analysis, it can be seen that compared with random addition, the recognition accuracy of confusing classes will be higher by adding images with the method in this paper. According to the above analysis, our method can evaluate the inter-class similar well. Adding images near the decision boundary of Class 4 (“Target spot”) and Class 9 (“Tetranychus urticae”) is more conducive to fine-grained image classification than adding images near the decision boundary of Class 0 (“Healthy”) and Class 3 (“Leaf mold”). The images near the decision boundary of Class 4 (“Target spot”) and Class 9 (“Tetranychus urticae”) are high information contribution images.

6. Conclusions

In the work of the present research, in order to solve the problems of insufficient data and fine-grained classification accuracy in plant disease classification, we propose a new image information contribution evaluation method based on inter-class similarity analysis. In order to verify the effectiveness of the method, an active learning image selection experiment was carried out. Experiments on fine-grained tomato classification datasets show that the proposed method can achieve better fine-grained classification results than the existing methods under the same budget. The effectiveness of this method shows that in the task of fine-grained image classification, we can not only research from the perspective of strengthening network performance, but also from the perspective of data analysis. In the future, we will continue to study few-shot learning and fine-grained classification problems, carry out data-centric image information quality evaluation methods. Strive to achieve better results with less labeled data, provide efficient sampling guidance in the field of digital agricultural data acquisition and labeling, and improve the overall effect of datasets. Propose more image information quality evaluation methods to promote the development of smart agriculture and digital agriculture.