Time Series Feature Extraction Using Transfer Learning Technology for Crop Pest Prediction

Abstract

Following the rapid development of information and communication technology, and the huge amounts of data that have undergone explosive growth, artificial intelligence and machine learning have been used for predictive analysis in many fields. However, the prediction accuracy of these machine learning recognition models depends on the quality of the features selected for training. It is therefore very important to analyse characteristics that are meaningful and in line with the target variables as the training conditions for machine learning recognition models. In this paper, we analyse the correlation between features and target variables using the Pearson product-moment correlation coefficient, and integrate transfer learning technology for sequential feature extraction to enhance the prediction accuracy of a machine learning recognition model for the prediction of multiple crop pests and diseases as the performance verification target of the proposed method. The performance of our machine learning recognition model is compared with schemes in related work, and our approach is shown to increase the prediction accuracy by between 3% and 15%.

1. Introduction

Benefiting from the development of artificial intelligence and big data technologies, which are now being implemented in many different fields, many prior studies have used artificial intelligence and machine learning to construct recognition models to solve the prediction problems that arise in these fields. Due to the slight differences in the data types that are suitable for various machine learning prediction algorithms, the basic requirement is to select the most suitable machine learning prediction algorithm for a specific data type [1,2]. The most important aspect that affects the prediction accuracy of various machine learning recognition models is feature selection, as selecting features that are uncorrelated or not highly correlated with the target variable will affect the predictive performance of the machine learning recognition model, meaning that the feature selection technology used for training of a machine learning recognition model is a very important research issue [3,4,5,6,7,8,9,10,11,12,13,14]. The prediction of crop diseases and insect pests is a key topic in the field of machine learning algorithms for intelligent agriculture [15,16,17]. One previous study used climate conditions, cotton pests and disease datasets, provided by the Crop Pest Decision Support System website, to develop a machine learning recognition model based on LSTM technology to predict the occurrence of cotton diseases and insect pests [18]. Features were selected that were highly correlated with the target variable to construct a machine learning recognition model, and it was proven that the occurrence of cotton diseases and insect pests was related to sequential changes in the climate. Another study used the random forest algorithm in a machine learning model to predict and select the characteristics of atmospheric circulation parameters and construct climate combinations, and Bi-LSTM was used to construct a machine learning recognition model of the degree of damage done by cotton pests [19]. From the above two previous studies, the machine learning recognition model was updated and selected features were used for training and prediction of machine learning recognition models. However, factors such as the overall climate change and pest evolution were considered, which led to the original high correlation features of the data to change the target variable. The use of changed features for the construction of a machine learning recognition model will lead to a decrease in prediction accuracy. In this paper, we therefore propose to use transfer learning technology for sequential feature extraction to dynamically adjust the highly correlated features selected by the machine learning recognition model for new data. We verify the effectiveness of the proposed method for the application of crop disease and insect pest prediction, and conduct an efficiency analysis involving the prediction of Bactrocera dorsalis pests on red dates and the machine learning recognition model of cotton pests from the study described above. The remainder of this paper is structured as follows: we review related work in Section 2 and explain time series feature extraction using a transfer learning technology method in Section 3. We introduce our experimental method and analyse its performance in Section 4. Finally, conclusions and prospects for future work are presented in Section 5.

2. Related Work

The primary task for predictive analysis with artificial intelligence based on machine learning is feature selection for the training of a recognition model. Selecting feature variables that are highly correlated with target variables for training of a recognition model can improve the prediction accuracy. This can be achieved by using the filter, wrapper or embedded methods, features which have a high level of correlation with the target variable; removing uncorrelated features or redundant features can make recognition model training simpler and improve the prediction performance [20,21]. One study in the literature used machine learning to predict the occurrence or non-occurrence of crop pests based on climate conditions and cotton pest datasets provided by the Crop Pest Decision Support System website [18]. The authors used the eight climate conditions provided by this website as features for the training of an LSTM machine learning recognition model. The K-means algorithm was used to discretise the feature values for these eight climatic conditions in advance, while the Apriori algorithm was used to verify whether or not these climatic conditions affected the occurrence of cotton pests. The experimental results of the study proved that the occurrence of cotton pests was related to sequential changes in the climate. However, the authors did not remove the features of the machine learning recognition model training; the use of redundant features can increase the complexity of training a recognition model and reduce the prediction performance. In this paper, we use the Pearson product-moment correlation coefficient for feature analysis and selection to obtain features that are highly correlated with the target variable for training and prediction of a machine learning recognition model, in order to reduce the complexity of training and improve the prediction performance. Another paper extended the research results from the study described above, which considered atmospheric circulation conditions and cotton pest datasets, by adding four further climatic factor combinations to the eight climatic conditions provided by the website for training of a machine learning recognition model [19]. In addition, the random forest algorithm was applied to the 74 atmospheric circulation conditions to observe the contributions of each feature, and the top 25 atmospheric circulation conditions with high correlation were selected. The 12 climatic conditions described above and 25 atmospheric circulation conditions were used as the training features for a Bi-LSTM machine learning recognition model. However, the authors did not take into account the fact that the data may change due to factors such as overall climate change and sudden changes in the evolution of insect pests, which lead to changes in the original features with strong correlation with the target variable. The use of historical features and data was not sufficient to accurately predict the degree of future crop pest damage when used as input to train the machine learning recognition model. In this paper, we use transfer learning techniques with sequential feature extraction to adjust the highly correlated features and data dynamically, thus adjusting the machine learning recognition model to real-time data.

A study in the related literature [22] used the number of pests (captured daily) and weather station information as a forecast dataset to construct a machine learning recognition model. An observation period of three to five days was selected based on the occurrence of pests to randomly divide the data, while retaining the data to carry out training of the machine learning identification model in a sequential manner. Real data obtained from different locations were used as the test dataset to adjust the machine learning identification model to predict the dates of pest occurrence. The goal was to use the climate temperature and relative humidity data information collected by the sensor to predict the occurrence of pests in different places in real time, and to provide enough warning to allow farmers to choose methods to prevent the occurrence of pests and to control the degree of damage. However, the abovementioned study did not consider the changes in sensing values caused by climate change, which can affect the occurrence of pests. Another study [23] used a multi-data type fusion model to predict disease in rice. The authors used information from agricultural meteorological sensors and camera images as a dataset, and an MLP architecture was applied to integrate the images with the sensor data. A CNN architecture was employed to classify rice diseases. The classification results from the MLP and CNN were input as two connection layers to obtain the final rice disease classification. However, the acquisition of camera images in this study was prone to noise from outdoor light and occlusion of the line of sight, resulting in a decrease in classification accuracy. A method [24] was proposed to calculate the number of insects based on density maps from deep learning, in which the RGB colour space and the DenseNet model were used to identify the leaves and the background. The first method used the colour space and VGG16 to classify the pests affecting the leaves. Deep learning was applied to identify the location of insect pests and to construct a number density map for the leaves. However, the acquisition of insect images in this study was prone to problems from outdoor illumination and line of sight occlusion problems, resulting in a decline in classification accuracy. Other authors [25] carried out research on the classification and prediction of pests and diseases based on existing machine learning recognition models. As images are captured in the outdoor environment, the process is often affected by outdoor light or problems with the line of sight. This paper focuses on integrating transfer learning technology for sequential feature extraction to enhance the prediction accuracy of machine learning recognition models.

3. Time Series Feature Extraction Using a Transfer Learning Technology Method

3.1. Overview of the Proposed System

We propose a sequential feature extraction method based on the use of transfer learning. A diagram of the system architecture is shown in Figure 1. After applying data processing, such as cleaning and feature aggregation from historical data, the data are passed to the feature selection stage. At the feature selection stage, many methods can be used to compute the feature variables that are highly correlated with the target variable, to obtain information on the strength of the correlation between each feature variable and the target variable. Multiple highly correlated features are selected as candidate features for our machine learning recognition model. Since real-time data will be added to the dataset one by one, the correlation between the features and the target variables will change. Sequential feature extraction by transfer learning technology involves using the features retained by the previous machine learning recognition model as candidate features, and then using these candidate features for the historical and real-time data to select the top N features for training of the machine learning recognition model and prediction. At the same time, since the data types that are suitable for various machine learning prediction algorithms are slightly different, we construct a complex prediction model for these top N features, and use those with the highest prediction accuracy for the machine learning recognition model as our sequential feature extraction method for transfer learning.

3.2. Design of the Recognition Model

In the data processing phase, the data are cleaned to avoid missing values affecting the subsequent modelling process and resulting in a decrease in prediction accuracy. At this stage, we pre-check the various characteristic types of data and identify abnormal and missing values to perform value-added correction processing. Furthermore, multiple forms of feature data with the same physical meaning are aggregated into a single feature. The method used in this paper to calculate the characteristic variables that are highly correlated with the target variable involves applying the Pearson product-moment correlation coefficient to obtain the correlation between feature variables and target variables. We can use the above method to obtain the correlation between variables; the degree of correlation index that observes the same changes among the above variables has directionality, and ranges from −1 (negative correlation) to 1 (positive correlation). A larger absolute value of the Pearson product-moment correlation coefficient indicates higher correlation, while a lower value indicates lower correlation. The proposed approach applies a sequential feature extraction method with transfer learning in which the Pearson product-moment correlation coefficient is used to analyse the correlation strength between each feature condition (X) and the target variable (Y). The value of the correlation coefficient between each feature condition and the target variable can be obtained as shown in Equation (1). Based on this value, we select the top N features for training and prediction, as shown in Equation (2).

$$r=\frac{\sum _{i=1}^n(X_i-\stackrel{-}{X})(Y_i-\stackrel{-}{Y})}{\sqrt{\sum _{i=1}^n{(X_i-\stackrel{-}{X})}^2}\times \sqrt{\sum _{i=1}^n{(Y_i-\stackrel{-}{Y})}^2}}$$

(1)

topN _{(j = 1, 2, 3, …, n)} = r _{(i = 1, 2, 3, …, n)}

(2)

When constructing the machine learning recognition model, the dataset that remains after the data processing stage is randomly divided into a training dataset and a test dataset, based on a 6:4 ratio, to build the machine learning recognition model. We apply leave-one-out cross-validation and use the recognition model with the highest prediction accuracy as a sequential feature extraction method for transfer learning. Our sequential feature extraction method with transfer learning takes into consideration the fact that real-time data will be added to the dataset one by one, meaning that the correlations between the features and target variables will change. Transfer learning technology is used for sequential feature extraction and to retain the features of the previous model as the candidate features. We integrate the values of the correlation coefficient for historical and real-time data between each feature condition and target variable to select the top N features for training and prediction. A complex prediction model is constructed based on the above selected top N features, and the k nearest neighbours (KNN), support vector machine (SVM), naive Bayes and random forest algorithms are used for training and prediction of the machine learning recognition model. The approach with the highest prediction accuracy is used as the sequential feature extraction method for transfer learning in the machine learning recognition model, as shown in Figure 2.

3.3. Examples of Recognition Models

When the historical data have passed through the data processing stages, such as data cleaning and feature aggregation, the data have features A, B, C, D, E, and F. The Pearson product-moment correlation coefficient is used to sort the highly correlated features at the feature selection stage, and the top four features are selected (in this case, features A, C, D, and F) for training and prediction by the machine learning recognition model. If real-time data points are added to the dataset one by one, the same Pearson product-moment correlation coefficient will be used to sort the highly correlated features. We assume that the top four candidate features at this time are features A, C, D, and E. We then use the sequential feature extraction method of transfer learning to integrate these four features from the historical data with those of the real-time data, giving features A, C, D, E and F, as shown in Figure 3. The proposed machine learning recognition model uses the KNN, SVM, naive Bayes and random forest algorithms.

3.4. Performance Assessment

The proposed method applies the Pearson product-moment correlation coefficient to analyse the correlations between features and target variables, and uses transfer learning to extract sequential features to strengthen the prediction accuracy of the machine learning recognition models. The accuracy (ACC), precision (Precision), recall rate (Recall) and F1-score are considered to assess the accuracy and effectiveness of the prediction model. For binary classification model outputs, there are only two types of results: correct (P) and incorrect (N). Our model has four outcomes for the case predictions: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). The definitions of ACC, Precision, Recall and F1-score are shown in Equations (3)–(6). Multiclass classification tasks of precision, recall, and F1-score can be applied to each label independently, which can be obtained the information through combine results across labels, specified by the average argument.

$$\mathrm{ACC}=\frac{TP+TN}{P+N}$$

(3)

$$\mathrm{Precision}=\frac{TP}{TP+FP}$$

(4)

$$\mathrm{Recall}=\frac{TP}{TP+FN}$$

(5)

$$\mathrm{F}\text{1-score}=\frac{2Precision\ast Recall}{Precision+Recall}$$

(6)

4. Experiments and Results

4.1. Prediction of Bactrocera Dorsalis Crop Pests

Bactrocera dorsalis is the primary pest affecting red dates during the growth process, and often causes the fruits to rot or drop, thereby reducing the harvest. Our first experiment was therefore carried out to predict the number of Bactrocera dorsalis pests on red dates to verify the performance of the proposed method. We used climate feature datasets from Taiwan’s agricultural meteorological observation and monitoring platform system (https://agr.cwb.gov.tw/NAGR/history/station_day, accessed on 17 August 2022), as shown in

Table 1. Climate characteristics used in Experiment 1.

Climate Characteristics
1. Mean pressure (hpa)	13. Maximum gust wind direction (winddirmax)
2. Mean sea-level pressure (hpasea)	14. Cumulative rainfall (rainday)
3. Maximum pressure (hpamax)	15. Maximum precipitation (10 min) (rainmax10)
4. Minimum pressure (hpamin)	16. Maximum precipitation (60 min) (rainmax60)
5. Mean temperature (temp)	17. Accumulative irradiation time (sunhr)
6. Maximum temperature (tempmax)	18. Cumulative solar radiation (solar)
7. Minimum temperature (tempmin)	19. Mean ground temperature (0 cm) (templand)
8. Dew point temperature (dew)	20. Mean ground temperature (5 cm) (templand)
9. Mean relative humidity (RH)	21. Mean ground temperature (10 cm) (templand)
10. Mean wind speed (vrl)	22. Mean ground temperature (20 cm) (templand)
11. Mean wind direction (winddir)	23. Mean ground temperature (50 cm) (templand)
12. Gust peak speed (windmax)	24. Mean ground temperature (100 cm) (templand)

. We applied data cleaning and feature aggregation to the historical data with the aim of improving the learning ability and generalisability of our machine learning recognition model. The average values were calculated for all mean ground temperature feature conditions, and these were combined into a single value representing the mean ground temperature feature. In this experiment, the Pearson product-moment correlation coefficient was used to analyse the climate characteristics provided by the original Taiwan agricultural meteorological observation and monitoring platform system. We then obtained the correlation between these and the activity of Bactrocera dorsalis pests. A heat map is shown in Figure 4.

Based on the absolute values of the Pearson product-moment correlation coefficients, the correlations were sorted and the top six climate characteristics were selected to construct a complex prediction model. In this experiment, the leave-one-out method was used for cross-validation to evaluate the prediction accuracy of the recognition model, in order to avoid overfitting or underfitting of the model. A dataset containing the numbers of oriental fruit flies from 2020 to 2022, as shown in Table 1, provided by the Farmers Association of Gongguan Township, Miaoli County, was used. The target of abovementioned oriental fruit fly pest number prediction is shown in

Table 2. Classification of the numbers of pests.

Number of Pests	Category
0~10	0
10~20	1
20~30	2
30~50	3
50~100	4
100~200	5
200~500	6
500 or more	7

. The proposed model takes into consideration the fact that real-time data will be added one by one, thus affecting the correlation of the climate characteristics with the activity of the fruit fly pests. The model uses transfer learning technology for sequential feature extraction to retain the features of the previous machine learning recognition model as candidate features, integrates historical and real-time data, and selects the top six features for training and prediction. In this way, it dynamically adjusts the highly correlated climatic features from the machine learning recognition model to the real-time data. In this experiment, the learning of the top six climate characteristics obtained from the 2020 historical data was transferred to the 2021 real-time data, as shown in Figure 5, and the learning of the top six climate characteristics obtained from the 2021 historical data was transferred to the 2022 real-time data. The top six climate characteristics obtained from this process are shown in Figure 6.

In this experiment, KNN, random forest, SVM and naive Bayes classifier algorithms were constructed to carry out a performance analysis of the selected climate characteristics. The parameters for the KNN algorithm were weights = distance, n_neighbors = 3, algorithm = auto and p = 2. The parameter used for the random forest algorithm was n_estimators = 100. The parameters for the SVM algorithm were kernel = linear and C = 1, while for the Naive Bayes algorithm, we used priors = None. In this experiment, transfer learning technology was integrated with sequential feature extraction to strengthen the machine learning recognition model, and the prediction results are shown in

Table 3. Results from complex forecasting models for sequential feature extraction from data for 2021.

2021	KNN with Proposed Method	Random Forest with Proposed Method	SVM with Proposed Method	Naive Bayes with Proposed Method
ACC	0.9196	0.9700	0.9700	0.9112
Precision	0.9396	0.9805	0.9800	0.9364
Recall	0.9392	0.9803	0.9803	0.9330
F1-score	0.9280	0.9772	0.9700	0.9250

and

Table 4. Results from complex forecasting models for sequential feature extraction from data for 2022.

2022	KNN with Proposed Method	Random Forest with Proposed Method	SVM with Proposed Method	Naive Bayes with Proposed Method
ACC	1.0000	1.0000	1.0000	1.0000
Precision	1.0000	1.0000	1.0000	1.0000
Recall	1.0000	1.0000	1.0000	1.0000
F1-score	1.0000	1.0000	1.0000	1.0000

. From these experimental results, it can be seen that the random forest algorithm gave the highest accuracy for the numbers of oriental fruit fly pests. The learning curves for the model are shown in Figure 7 and Figure 8, and it can be seen that the curve gradually converges as the number of samples increases. We therefore used it in the remainder of the experiment in our machine learning recognition model with the sequential feature extraction method and transfer learning. In addition, the climate conditions were used without feature selection for modelling, and the prediction results are shown in

Table 5. Results from complex forecasting models for sequential feature extraction from data for 2021, without feature selection.

2021	KNN	Random Forest	SVM	Naive Bayes
ACC	0.7300	0.88	0.881	0.88
Precision	0.6892	0.9565	0.85	0.9555
Recall	0.8380	0.9142	0.9143	0.9142
F1-score	0.6954	0.9204	0.9200	0.9200

and

Table 6. Results from complex forecasting models for sequential feature extraction from data for 2022, without feature selection.

2022	KNN	Random Forest	SVM	Naive Bayes
ACC	0.8055	0.8050	0.8055	0.8000
Precision	0.7666	0.9247	0.7666	0.9247
Recall	0.7407	0.7410	0.7400	0.7407
F1-score	0.6083	0.7454	0.6080	0.7450

. A comparison of the results in Table 3, Table 4, Table 5 and Table 6 shows that the use of feature selection was important.

4.2. Prediction of Whitefly Crop Pests

The second experiment aimed to predict the number of whitefly pests on cotton crops, and was used to validate the proposed method. The Crop Pest Decision Support System (http://www.icar-crida.res.in:8080/naip/index.jsp, accessed on 30 July 2022) from the literature provided the climatic characteristics and datasets on the activity of whitefly pests in cotton, as shown in

Table 7. Climate characteristics used in Experiment 2.

Climate Characteristics
MaxT (°C) MinT (°C) RH1 (%) RH2 (%) RF (mm) WS (kmph) SSH (hrs) EVP (mm)

. The Pearson product-moment correlation coefficients of the abovementioned climate characteristics were analysed to obtain the correlations between various climate characteristics and the effects of whitefly pests, and a heat map is shown in Figure 9.

The results were sorted based on the Pearson product-moment correlation coefficient, from high to low, after taking the absolute values. The top four climate characteristics were selected to construct a complex prediction model, involving the transfer learning of the top four climate characteristics obtained from the historical data in the first year to the real-time data of the second year. The top four climate characteristics obtained in this way are shown in Figure 10. This experiment combined transfer learning technology with sequential feature extraction to strengthen the machine learning recognition model. The prediction results are shown in

Table 8. Results for prediction of the numbers of whitefly pests.

Whitefly	KNN with Proposed Method	Random Forest with Proposed Method	SVM with Proposed Method	Naive Bayes with Proposed Method	Related Work [18]
ACC	0.95555	0.94444	0.84444	0.7888	0.9244
F1-score	0.95553	0.94438	0.84413	0.7886	0.9243

, and it can be seen that the KNN algorithm had the highest prediction accuracy for the numbers of whitefly pests. The model learning curve is shown in Figure 11. It was therefore used in the machine learning recognition model with the sequential feature extraction method and transfer learning. Compared with the performance of another scheme in the literature, the method proposed in this paper could improve the prediction accuracy by more than 3%.

4.3. Prediction of Aphid Crop Pests

The third experiment aimed to predict the number of aphid pests on cotton crops, as a validation of the proposed method. We used data from the Crop Pest Decision Support System (http://www.icar-crida.res.in:8080/naip/index.jsp, accessed on 30 July 2022), which provided the climate characteristics and datasets for whitefly pests in cotton, as shown in

Table 9. Climate characteristics used in Experiment 3.

Climate Characteristics
1. MaxT (°C)	7. SSH (h)
2. MinT (°C)	8. EVP (mm)
3. RH1 (%)	9. LTP
4. RH2 (%)	10. TF
5. RF (mm)	11. PTR
6. WS (kmph)	12. THC

. In this experiment, the Pearson product-moment correlation coefficients for the abovementioned climate characteristics were analysed to obtain the correlation between the various climate characteristics and the activity of whitefly pests. A heat map is shown in Figure 12.

The Pearson product-moment correlation coefficients were sorted from high to low after taking the absolute values, and the top four climate characteristics were selected to construct a complex prediction model in this experiment. Transfer learning was applied to the top four climate characteristics obtained from the historical data from the first year, and these were transferred to the real-time data for the second year. The top four climate characteristics obtained in this way are shown in Figure 13. In this experiment, we integrated transfer learning technology with sequential feature extraction to strengthen the machine learning recognition model. The prediction results are shown in

Table 10. Results for prediction of the number of aphid pests.

Aphid	KNN with Proposed Method	Random Forest with Proposed Method	SVM with Proposed Method	Naive Bayes with Proposed Method	Related Work [19]
ACC	0.8898	0.9661	0.8983	0.8305	0.8298
F1-score	0.8959	0.9725	0.9048	0.8430	0.8234

. From these experimental results, it could be seen that the random forest algorithm had the highest prediction accuracy for the numbers of aphid pests. The model learning curve is shown in Figure 14. We therefore used this in the machine learning recognition model with our sequential feature extraction method and transfer learning. Compared with the performance of the related work, the proposed method in this paper was able to improve the prediction accuracy by more than 14%.

5. Conclusions

In this study, we used the Pearson product-moment correlation coefficient to analyse the correlation between features and target variables. Transfer learning technology was integrated with time-series feature extraction to enhance the prediction accuracy of our machine learning recognition models. From a comparison of the performance of our machine learning recognition models for the prediction of multiple types of crop pests with a scheme from the literature, we found that the prediction accuracy value was increased by at least 3% and up to 15%.