Next Article in Journal
Agronomic and Physicochemical Properties Facilitating the Synchronization of Grain Yield and the Overall Palatability of Japonica Rice in East China
Previous Article in Journal
Structural Polymorphisms of Chromosome 3Am Containing Lr63 Leaf Rust Resistance Loci Reflect the Geographical Distribution of Triticum monococcum L. and Related Diploid Wheats
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Classification of Maize Lodging Extents Using Deep Learning Algorithms by UAV-Based RGB and Multispectral Images

1
Research Center of Information Technology, Beijing Academy of Agriculture and Forestry Sciences, Beijing 100089, China
2
School of Science, China University of Geosciences, Beijing 100089, China
3
National Engineering Research Center for Information Technology in Agriculture, Beijing 100089, China
*
Authors to whom correspondence should be addressed.
Agriculture 2022, 12(7), 970; https://doi.org/10.3390/agriculture12070970
Submission received: 16 June 2022 / Revised: 1 July 2022 / Accepted: 5 July 2022 / Published: 6 July 2022

Abstract

:
Lodging depresses the grain yield and quality of maize crop. Previous machine learning methods are used to classify crop lodging extents through visual interpretation and sensitive features extraction manually, which are cost-intensive, subjective and inefficient. The analysis on the accuracy of subdivision categories is insufficient for multi-grade crop lodging. In this study, a classification method of maize lodging extents was proposed based on deep learning algorithms and unmanned aerial vehicle (UAV) RGB and multispectral images. The characteristic variation of three lodging extents in RGB and multispectral images were analyzed. The VGG-16, Inception-V3 and ResNet-50 algorithms were trained and compared depending on classification accuracy and Kappa coefficient. The results showed that the more severe the lodging, the higher the intensity value and spectral reflectance of RGB and multispectral image. The reflectance variation in red edge band were more evident than that in visible band with different lodging extents. The classification performance using multispectral images was better than that of RGB images in various lodging extents. The test accuracies of three deep learning algorithms in non-lodging based on RGB images were high, i.e., over 90%, but the classification performance between moderate lodging and severe lodging needed to be improved. The test accuracy of ResNet-50 was 96.32% with Kappa coefficients of 0.9551 by using multispectral images, which was superior to VGG-16 and Inception-V3, and the accuracies of ResNet-50 on each lodging subdivision category all reached 96%. The ResNet-50 algorithm of deep learning combined with multispectral images can realize accurate lodging classification to promote post-stress field management and production assessment.

1. Introduction

According to the data released by China’s National Bureau of Statistics, the planting area of maize reached 43.324 million hectares in 2021, increasing by 2.059 million hectares compared with 2020. The total output of maize achieved 272 million tons, which made it the most productive of China’s major crops. The yield variance of maize has an important impact on national food security and agricultural economic development. However, crop lodging is one of the major negative elements to affect maize output. It is stated as the displacement of the above-ground stems from their upright position or failure of root-soil attachment [1]. Lodging is generally caused by rainstorms, loose soil, high planting density and unreasonable fertilization [2,3,4]. Lodging hinders the growth of maize [5], reduces grain quality [6] and affects mechanized harvesting [7], which is becoming an important restricting issue to increase maize yield [8]. Therefore, precise and efficient classification with different maize lodging extents can help agricultural departments to investigate the influence of maize growth, guide farmers to implement post-stress field management and facilitate insurance firms to settle disputes properly [9,10].
The traditional lodging assessment methods widely used are mainly visual inspection and artificial measurement [11], which are inefficient, time-consuming and environment-constraining [12]. The inaccuracy and subjectivity of them may lead to compensation disputes between farmers and insurance companies, which cannot meet the needs of precision agriculture. Remote sensing technology, as a new approach, has greatly promoted the development of crop lodging detection [13]. The lodging incidence in wheat, rice and barley were detected using visible and thermal infrared images based on the ground-based and space-borne platforms [14,15,16,17]. In recent decades, the unmanned aerial vehicle (UAV) has been increasingly applied for lodging monitoring due to its advantages of convenient, flexible, low cost and high resolution [18,19]. It can timely and accurately obtain centimeter-level images with multiple sensors, which plays a powerful role in lodging detection [20]. Many studies detected crop lodging based on a UAV system equipped with a digital camera. They discriminated lodging from non-lodging and evaluated the extents of crop lodging by analyzing color and texture features [21,22,23]. However, compared to RGB images with only three visible bands, multispectral images with red edge and infrared bands reflecting the growth capacity of crops can offer more information in crop lodging [12,24,25]. Both spatial and spectral information of ground targets are obtained in the meantime. Therefore, the information richness of lodging features between these two types of images are different. It is worth studying to verify the performance of discriminating lodging severity extents using RGB and multispectral images.
The appropriate classification methods for crop lodging extents are significant as well as the selection of data source. Traditionally, machine learning algorithms consisting of a support vector machine (SVM) [8], decision tree [26] and nearest neighbor [5] were used to classify lodging by extracting crop morphology and spectral characteristics [21,23,27]. However, these manual approaches for extracting features often required empirical knowledge and were typically suboptimal in the results [28]. With the development of machine learning, the convolutional neural network (CNN) of deep learning has gradually become the mainstream. CNN algorithms can automatically extract image features, and depict rich intrinsic information with strong nonlinear modeling ability. Xia hao et al. [29] proposed a classification model named GL-CNN on account of convolutional neural networks to determine the optimal growth stage of leafy vegetables. Ananda et al. [30] used the Visual Geometry Group (VGG) model to achieve the disease detection and classification of grapes and tomatoes. CNN has been proved to be superior to existing traditional machine classification algorithms [31]. The Inception and ResNet algorithms were proposed with better performance, which could automatically extract target features from images more accurately. They have been widely used in disease detection and crop classification in intelligent agriculture [32,33]. However, there are few studies on maize lodging classification based on deep learning algorithms. The maize lodging characteristics of multiple data types need to be analyzed. The performance difference using RGB and multispectral images will be compared. Previous studies have often focused on the overall classification accuracy of crop lodging, which were unable to fully embody the quality of the model. The classification effects of algorithms under subdivision categories are also worthy of attention.
The purpose of this study is to use deep learning algorithms to monitor the lodging extents of maize based on RGB or multispectral images. The lodging extents are discriminated as non-lodging, moderate lodging and severe lodging by lodging angle. The specific objectives are as follows: (1) to analyze the characteristics of obtained images with different lodging extents, (2) classify lodging extents of maize based on RGB and multispectral images through VGG-16, Inception-V3 and ResNet-50 algorithms and (3) evaluate classification performance in different lodging extents to determine the optimal algorithm.

2. Materials and Methods

The processes of classifying maize lodging extents in the study were showed in Figure 1. The RGB and multispectral images were acquired via UAV, which were respectively cropped, augmented and labeled to build the datasets. The difference of each band of RGB and multispectral images caused by maize lodging extents were analyzed. The classification results of maize lodging extents using three deep learning algorithms were compared and validated.

2.1. Study Area

The study area is located in Lishu County, Siping City, southwest Jilin Province, China (Figure 2). The geographical coordinates of it are 43°02′ N–43°46′ N, 123°45′ E–124°53′ E. Lishu is in the hinterland of Songliao Plain and the major grain producing county with a maize planting area of 213,300 hectares. During the maize growth period, sunshine and precipitation are sufficient, which can fully meet the growth needs of one ripe a year. From late August to early September in 2020, strong winds and heavy rain caused crop lodging.

2.2. Data Acquisition

The data collection of maize lodging canopy images in this study was performed with a DJI Phantom 4 Pro (DJI-Innovations, Inc., Shenzhen, China) at 12:00 am on 12 September 2020. The weather was cloudless and windless. The overall weight of UAV system is 1388 g, and the duration of flight is about 30 min. In this study, the flight altitude was 30 m above the ground. The forward and lateral overlap was 80%. The digital camera had three color channels of red, green and blue with a resolution of 1 cm/pixel. The multispectral images were collected by a Parrot Sequoia camera (MicaSense, Inc., Seattle, DC, USA). It consisted of four multispectral channels of green (550 nm), red (660 nm), red edge (735 nm) and near-infrared (790 nm) with a resolution of 2 cm/pixel. The global positioning system (GPS) and irradiance sensors were equipped at the same time. Before and after each flight, radiometric calibration images were obtained by a calibrated reflectance panel. The field inspection was taken after UAV data acquisition. Lodging has a huge impact on both yield and grain quality. Lodging caused a maize yield loss of approximately 0–50% at different lodging angles [3]. In general, the smaller the lodging angle, the smaller the yield loss. Lodging classification can provide a basis for predicting future harvest yield. According to the investigation of maize lodging in the study area, we categorically defined three lodging extents based on crop lodging angle: non-lodging (NL) maize with a crop angle <10°, moderate lodging (ML) maize with a crop angle between 10–50° and severe lodging (SL) maize with a crop angle >50° (Figure 3).

2.3. Data Cleaning and Augmentation

RGB and multispectral images of the entire study area were obtained by Agisoft Photoscan software. RGB images were resampled to 2 cm/pixel to match the resolution of multispectral images. Then, the images of the entire study area were cropped into small images with a resolution of 300 × 300 pixels. The actual spatial size of each image was 6 m, achieving a precise classification of maize lodging. Considering the partial areas of the images were not related to maize lodging, the original dataset of 1326 images was acquired by deleting the cropped images containing roads and weeds. Then, each sample was labeled as non-lodging, moderate lodging and severe lodging by an expert through visual interpretation (Figure 4).
For the purpose of improving the overall generalization ability of the model, abundant training images are needed in the deep learning algorithms to avoid over-fitting. Data augmentation undertakes a more crucial improvement upon the classification accuracy in the dataset [28]. Therefore, we performed data augmentation on the obtained dataset to expand the number of samples. In this study, we enhanced image numbers by random rotation, horizontal inversion and vertical inversion. A dataset of 5000 RGB images and 5000 multispectral images was generated by data augmentation without introducing extra labeling costs. The dataset included 1616 non-lodging samples, 1684 moderate lodging samples and 1700 severe lodging samples. The results of image augmentation taking an RGB image as an example are shown in Figure 5.

2.4. Deep Learning

2.4.1. Convolutional Neural Networks

Convolutional neural networks (CNN) have been essential to the development of deep learning. Remarkable advancements have been made on image classification [34]. CNN architecture is mainly divided into convolution layer, pooling layer and fully connected layer (Figure 6). The various aspects in the whole image are assigned importance for establishing a distinction between different objects in convolution layer. The weights of convolution kernels (not directly accessible to users) are constantly updated during algorithm iterations. After the convolution, the pooling operation can reduce the spatial size of the convolved features. It can help reduce the computing power requirements of data processing. We generally use two pooling methods, including maximum pooling and average pooling. Maximum pooling was superior in this study, because it could suppress noise while reducing dimension. Convolution and pooling layers were combined to extract image features of different levels. The last layer is the fully connected layer, which identifies the extracted features and provides the predicted label by using Softmax regression classifier eventually.

2.4.2. VGG-16

VGG-16 is a CNN algorithm proposed by the Visual Geometry Group of Oxford University [35]. It consists of thirteen convolution layers (extracting image features), five maximum pooling layers (reducing image spatial size) and three fully connected layers (classifying images into labels) (Figure 7). Compared with traditional convolutional neural networks, this algorithm uses a 3 × 3 convolution kernel to replace the larger one (e.g., 5 × 5, 7 × 7). This optimization effectively reduces the number of model parameters and extracts the detail features of the images more accurately. Hence, it can improve the computing speed and has good generalization performance.

2.4.3. Inception-V3

Inception-V3 is the most representative algorithm among inception algorithms [36]. It uses the Inception module, which performs multiple convolution and max pooling operations in parallel to obtain a deeper feature map. The Inception-V2 references VGG net using small convolution kernels (e.g., 1 × 1, 3 × 3) to reduce the computational cost effectively. On the basis of that, Inception-V3 decomposes the 3 × 3 convolution kernel into 1 × 3 and 3 × 1 convolution kernels (Figure 8). The depth and nonlinearity of the network increase, which makes the network classification ability stronger.

2.4.4. ResNet-50

ResNet-50 is proposed to solve the degradation problem in neural network training, which means the performance of the algorithm decreases with the deepening of network layers [37]. Residual block is the core of ResNet network (Figure 9), which mainly connects the convolution layer across layer by jumping connection and short circuit methods. It can transfer the input x as the initial result directly to the output, ensuring the integrity of the information. The output result is H(x) = F(x) + x, where F(x) is the residual function, which helps to transmit information to deeper neural networks and improve the accuracy of the algorithm.
These three algorithms were used to classify different lodging extents and test their accuracy performance. The ReLU function was used as the activation function, and the dropout layer was imported to prevent the algorithms from overfitting (dropout_ratio = 0.5). The last layer (fully connected layer) was replaced by three classification categories to adapt to the dataset of this study.
To demonstrate the algorithms’ validity and reliability, 70% of the samples (without substitution) were randomly selected as the training set and the remaining 30% of samples were the test set.

2.5. Validation

The image classification results of the dataset are evaluated by the confusion matrix, test accuracy and Kappa coefficient. The test accuracy is figured by the ratio between the number of correctly classified samples and the total number of samples in the test set. Kappa coefficient is a robust measure of the extents of agreement. In order to evaluate these indicators more persuasively, we repeated the experiments 10 times. The test accuracy and Kappa coefficients were calculated by the following formula and recorded as the average of ten repetitions:
Test   Accuracy = i = 1 n x i i N
Kappa = i = 1 n x i i / N i = 1 n ( j = 1 n x i j j = 1 n x j i ) / N 2 1 i = 1 n ( j = 1 n x i j j = 1 n x j i ) / N 2
where x i i refers to the correctly predicted samples, x i j refers to the elements of the i-th row and j-th column of the confusion matrix, n is the number of classifications and N is the total number of samples in test set.

3. Results

3.1. Research Images Analysis

In order to obtain an understanding of the lodging features under different types of images better, all the samples were used to observe the characteristics variation of maize canopy in the different lodging extents. The intensity values of RGB images and the reflectance of multispectral images were extracted directly by the statistical function of the ENVI 5.3 software.

3.1.1. RGB Images Analysis

RGB images contain the intensity values in red, green and blue color channels ranging from 0 to 255. Different intensity values of the three channels are combined into different colors. The means and standard deviations of three channels with different lodging extents were calculated in Figure 10. The intensity values of lodging (moderate lodging and severe lodging) were all significantly higher than that of non-lodging in three bands, but those of moderate and severe lodging were close relatively. In the non-lodging area, there were interspaces between maize plants along with shadows, and the soil was exposed to aerial photography, which made the intensity values low. After lodging, the plants tilted and piled each other, causing the soil to be covered. The intensity values increased with the decrease of soil bareness and the increase of plant density. Meanwhile, the changes of intensity values with different lodging extents were consistent in three bands, which were the lowest in blue and highest in green band. In Table 1, compared with the intensity values of non-lodging maize in three bands, those of the moderate lodging increased by 37.64%, 21.68% and 27.73%, and those of the severe lodging increased by 53.81%, 32.89% and 40.81%, respectively. It showed that the intensity values increased rapidly after lodging, and the increase rate of the values in the blue band was the highest.

3.1.2. Multispectral Images Analysis

Multispectral images show the reflectance in green, red, red edge and near-infrared bands with different lodging extents. The reflectance ranges from 0 to 1. The means and standard deviations of four channels with different lodging extents were calculated in Figure 11. The spectral reflectance increased following the enhancement of lodging extents in four bands. The reason was that lodging has changed the morphological structure of the maize population. The original maize canopy was damaged with the stems exposed. As the severity of maize lodging increased, more stems were exposed in aerial images taken by the UAV. Furthermore, the reflectance of the leaf was lower than that of the stem [24]. In Table 2, the reflectance of the red edge and near-infrared bands was significantly higher than that of the green and red bands. Compared with the reflectance of non-lodging maize in four bands, that of the moderate lodging increased by 6.45%, 12.50%, 19.51% and 13.20%, and that of the severe lodging increased by 19.35%, 25%, 36.58% and 20.75%, respectively. It indicated that in different lodging extents, the variation of reflectance in the red edge band was more evident than that in the visible band, which meant the increase rate of reflectance in the red edge band was the largest as well.

3.2. Lodging Classification Using RGB Images

VGG-16, Inception-V3 and ResNet-50 had pre-trained CNN models to deal with RGB images. Their weight parameters were trained and identified based on a huge number of RGB images from the ImageNet Dataset (http://image-net.org/index, accessed on 4 March 2022). The transfer-learning method could achieve sharing of model features through the hyperparameter transfer. Therefore, the backbone parameters of three CNN algorithms were initialized using the pre-trained weights, which could save algorithm training time and obtain accurate results. The PyTorch framework with Python 3.6 was used to support all experiments, and GTX 1070 6G GPU was employed to accelerate the overall process. The learning rate, batch size and the number of iterations of the three algorithms were set to 0.0001, 20 and 100, respectively. The networks were trained with the Adam optimizer and cross-entropy loss function to optimize the objectives.
The changes of classification accuracy and loss in three algorithms during 100 iterations were shown in Figure 12. Due to the use of pre-training models, the initial training accuracies were all more than 0.6. With the continuous optimization of the algorithms, the classification accuracy improved rapidly. Eventually, the training accuracy of the three algorithms reached 86.16%, 91.89% and 94.16%, respectively. In addition, we chose cross-entropy as the loss function, and loss gradually decreased following the opposite overall trend to accuracy curves. Both of them began to maintain stability after approximately 20 iterations. The convergence rate of ResNet-50 algorithm was obviously faster than the other two algorithms. In Table 3, the test accuracies of the three algorithms were 83.55%, 87.32% and 90.08% with Kappa coefficients of 0.7421, 0.8040 and 0.8599, respectively. The overfitting phenomenon did not occur in the training process. ResNet-50 obtained the optimal performance in three algorithms, whose test accuracy was 7.81% and 3.16% higher than VGG-16 and Inception-V3. However, in addition to discussing the overall classification accuracies of the three algorithms, the classification performance of different categories was also worth further analysis.
The confusion matrices of the three algorithms are shown in Figure 13. It indicated that the performance varied for different lodging severity extents in three algorithms. The identifications of non-lodging all achieved good results, whose classification accuracies were more than 90%. The accuracies of Inception-V3 and ResNet-50 in moderate lodging were improved over 10% compared to that of VGG-16. The three algorithms had no distinct differences in severe lodging. However, the classification error of the three algorithms between moderate lodging and severe lodging was high with almost over 10%, especially VGG-16, which made it difficult to identify the subdivision of lodging effectively.

3.3. Lodging Classification Using Multispectral Images

For multispectral images, the backbone parameters of the three algorithms needed to be randomly initialized to retrain models by the Xavier initialization method [38]. The last layer (fully connected layer) was replaced by three classification categories as well. The software environment and hyperparameter settings were the same as the operation on the RGB images.
The fluctuations of classification accuracy and loss of the three algorithms during 100 iterations using multispectral images were represented in Figure 14. In the early stage of algorithm optimization, the accuracy and loss curves showed an oscillating trend. That was because the algorithms were quickly adjusting the parameters to meet the classification requirements at the beginning. Then, the training accuracy of the three algorithms gradually increased and converged after 60 iterations with 92.34%, 94.70% and 98.55%, respectively. With the continuous optimization, the loss decreased quickly and ResNet-50 was the first to realize convergence. In Table 4, the test accuracies of the three algorithms were 89.91%, 92.36% and 96.32% with Kappa coefficients of 0.8318, 0.8935 and 0.9551, respectively. There was no over-fitting phenomenon in the training process as well. The test accuracy of ResNet-50 was 7.12% and 4.28% higher than VGG-16 and Inception-V3, respectively.
The confusion matrices of the three algorithms through multispectral images are shown in Figure 15. The three algorithms still performed well in the classification of non-lodging, which was higher than 92%. Compared with the RGB images, the classification of moderate lodging and severe lodging was significantly improved by multispectral images, and the accuracies of Inception-V3 and ResNet-50 were more than 90%. The accuracy error of ResNet-50 between moderate lodging and severe lodging was less than 5%, which can better classify the three extents of maize lodging.

3.4. Classification Results

In this study, the experiment results indicated that the overall performance of three deep learning algorithms using multispectral images in classification of different maize lodging extents was better than that of RGB images with an increase of 6.42%, 5.77% and 6.93% (Figure 16). The maize lodging classification based on RGB images using three algorithms realized high accuracy in non-lodging, which was suitable for the binary classification of lodging and non-lodging. Among the three deep learning algorithms, ResNet-50 was efficient and robust to classify the different lodging extents with the fastest convergence rate and highest classification accuracy during algorithm training. ResNet-50 also had the highest improvement in classification accuracy of multispectral images compared with RGB images, which could extract the lodging features more effectively. Therefore, ResNet-50 was the optimal algorithm to realize the classification of maize lodging extents.

4. Discussion

Lodging is a major factor in decreasing the crop yields worldwide. Accurate classification of lodging extents is beneficial to monitoring crop production and conducting reasonable decision-making. Timely and effectively obtaining experimental data plays a crucial role in it. Some researchers used satellite data to conduct crop lodging studies [14,39]. However, it is susceptible to clouds and the revisiting time is long with low spatial resolution. With the development of UAV technology, remote sensing research based on UAVs platform has been highly valued and become a hotspot [40]. The wide application of UAVs has indeed facilitated the monitoring of crop lodging. Tan et al. [23] used RGB images for grading lodging severity with the accuracy of 79.1%. Sun et al. [25] realized the detection of maize lodging with the overall accuracy of 86.61% and the Kappa coefficient of 0.8327 using maximum likelihood classification (MLC) by multispectral images. Furthermore, through applying machine learning methods, such as nearest neighborhood classification and Support Vector Machine (SVM), Chauhan et al. [24] and Rajapaksa et al. [41] reported the wheat lodging classification using multispectral images with 90% and 92.6% accuracies, respectively. Multispectral images had more potential to explore the characteristics of crop lodging. The lodging feature extraction is also of great significance for the classification result. Canopy texture, crop height, spectral reflectance and vegetation indices were extracted separately to research in the above study. The extraction process was both time-consuming and subjective. The features extracted for different crops were also different. It created difficult problems for further research of crop lodging.
We further realized the maize lodging classification based on deep learning algorithms. Deep learning algorithms can automatically extract intrinsic features from massive data through supervised learning to classify different lodging extents. Among the three deep learning algorithms in this study, the ResNet-50 algorithm performed best, with a test accuracy of 96.32% and Kappa coefficients of 0.9551, which was significantly better than traditional machine learning algorithms. On the type of images used above, although the lodging classification using multispectral images was more accurate, the low cost of RGB images acquisition and more than 80% test accuracy made it more beneficial for smallholders to detect crop lodging. The application of transfer-learning method can greatly shorten the training time of the models, which can facilitate more timely agricultural disaster assessment and management. In addition, through using multispectral images, the reflectance variation in red edge band was more evident than that in visible band with the increase of lodging severity extents, which may be an important factor for better lodging classification using multispectral images. Using red edge band to extract sensitive features for classifying lodging extents is worth further study.
There are still some deficiencies that need to be improved. We divided the experimental plots into three lodging extents. Further detailed classification of the lodging extents is necessary, which meets the requirements of precision agriculture. Moreover, the models presented in this study need to be tested and validated in other crop lodging classifications. The solution of them can serve crop yield prediction and precise agricultural insurance claim.

5. Conclusions

In this study, unmanned aerial vehicles (UAVs) provided convenience for multiple types of data acquisition. The RGB and multispectral images of maize lodging canopy were tested to classify different lodging extents. The images were preprocessed by cropping, cleaning and enhancing to generate the dataset containing 5000 subimages. The experimental results indicated that the spectral reflectance increased with the increase of lodging severity on the multispectral images of maize lodging. The red edge band was the most sensitive to the change of lodging severity extents. The classification performance of the three algorithms using RGB images, although good for non-lodging with over 90% accuracy, was unsatisfactory for moderate and severe lodging. The test accuracies of VGG-16, Inception-V3 and ResNet-50 were 89.91%, 92.36% and 96.32% with Kappa coefficients of 0.8318, 0.8935 and 0.9551, respectively, by using multispectral images. The accuracy of ResNet-50 on each lodging subdivision category all reached 96%. Therefore, ResNet-50 outperformed the Inception-V3 and VGG-16 algorithms, and multispectral images were more suitable for crop lodging classification than RGB images. This study provides a more accurate and effective method for the classification of crop lodging extents. Further detailed lodging classification and the general applicability of the method will be the focus of subsequent research.

Author Contributions

X.G. and T.C. designed and initiated the experiments; X.Y. wrote the article; Q.S. collected the data; J.Z. processed the data and prepared the figures; S.G. and Y.P. helped in revising the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Key Research and Development Program of China (2021YFD1500203) and Beijing Talents Project (2020A58).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to the use of subsequent studies.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Wu, W.; Ma, B. A new method for assessing plant lodging and the impact of management options on lodging in canola crop production. Sci. Rep. 2016, 6, 31890. [Google Scholar] [CrossRef] [Green Version]
  2. Ma, D.; Xie, R.; Liu, X.; Niu, X.; Hou, P.; Wang, K.; Lu, Y.; Li, S. Lodging-related stalk characteristics of maize varieties in China since the 1950s. Crop Sci. 2014, 54, 2805–2814. [Google Scholar] [CrossRef]
  3. Sun, Q.; Gu, X.; Chen, L.; Xu, X.; Wei, Z.; Pan, Y.; Gao, Y. Monitoring maize canopy chlorophyll density under lodging stress based on UAV hyperspectral imagery. Comput. Electron. Agric. 2022, 193, 106671. [Google Scholar] [CrossRef]
  4. Shu, M.; Zhou, L.; Gu, X.; Ma, Y.; Sun, Q.; Yang, G.; Zhou, C. Monitoring of maize lodging using multi-temporal Sentinel-1 SAR data. Adv. Space Res. 2020, 65, 470–480. [Google Scholar] [CrossRef]
  5. Han, L.; Yang, G.; Yang, X.; Song, X.; Xu, B.; Li, Z.; Wu, J.; Yang, H.; Wu, J. An explainable XGBoost model improved by SMOTE-ENN technique for maize lodging detection based on multi-source unmanned aerial vehicle images. Comput. Electron. Agric. 2022, 194, 106804. [Google Scholar] [CrossRef]
  6. Islam, M.S.; Peng, S.; Visperas, R.M.; Ereful, N.; Bhuiya, M.S.U.; Julfiquar, A.W. Lodging-related morphological traits of hybrid rice in a tropical irrigated ecosystem. Field Crop. Res. 2007, 101, 240–248. [Google Scholar] [CrossRef]
  7. Guo, Y.; Hu, Y.; Chen, H.; Yan, P.; Du, Q.; Wang, Y.; Wang, H.; Wang, Z.; Kang, D.; Li, W.-X. Identification of traits and genes associated with lodging resistance in maize. Crop J. 2021, 9, 1408–1417. [Google Scholar] [CrossRef]
  8. Liu, T.; Li, R.; Zhong, X.; Jiang, M.; Jin, X.; Zhou, P.; Liu, S.; Sun, C.; Guo, W. Estimates of rice lodging using indices derived from UAV visible and thermal infrared images. Agric. For. Meteorol. 2018, 252, 144–154. [Google Scholar] [CrossRef]
  9. Sposaro, M.M.; Berry, P.M.; Sterling, M.; Hall, A.J.; Chimenti, C.A. Modelling root and stem lodging in sunflower. Field Crop. Res. 2010, 119, 125–134. [Google Scholar] [CrossRef]
  10. Zhang, P.; Gu, S.; Wang, Y.; Yang, R.; Yan, Y.; Zhang, S.; Sheng, D.; Cui, T.; Huang, S.; Wang, P. Morphological and mechanical variables associated with lodging in maize (Zea mays L.). Field Crop. Res. 2021, 269, 108178. [Google Scholar] [CrossRef]
  11. Bock, C.H.; Poole, G.H.; Parker, P.E.; Gottwald, T.R. Plant disease severity estimated visually, by digital photography and image analysis, and by hyperspectral imaging. Crit. Rev. Plant Sci. 2010, 29, 59–107. [Google Scholar] [CrossRef]
  12. Chu, T.; Starek, M.; Brewer, M.; Murray, S.; Pruter, L. Assessing lodging severity over an experimental maize (Zea mays L.) field Using UAS images. Remote Sens. 2017, 9, 923. [Google Scholar] [CrossRef] [Green Version]
  13. Jay, S.; Maupas, F.; Bendoula, R.; Gorretta, N. Retrieving LAI, chlorophyll and nitrogen contents in sugar beet crops from multi-angular optical remote sensing: Comparison of vegetation indices and PROSAIL inversion for field phenotyping. Field Crop. Res. 2017, 210, 33–46. [Google Scholar] [CrossRef] [Green Version]
  14. Chauhan, S.; Darvishzadeh, R.; Lu, Y.; Boschetti, M.; Nelson, A. Understanding wheat lodging using multi-temporal Sentinel-1 and Sentinel-2 data. Remote Sens. Environ. 2020, 243, 111804. [Google Scholar] [CrossRef]
  15. Sakamoto, T.; Shibayama, M.; Takada, E.; Inoue, A.; Morita, K.; Takahashi, W.; Miura, S.; Kimura, A. Detecting seasonal changes in crop community structure using day and night digital images. Photogramm. Eng. Remote Sens. 2010, 76, 713–726. [Google Scholar] [CrossRef]
  16. Liu, Z.; Li, C.; Wang, Y.; Huang, W.; Ding, X.; Zhou, B.; Wu, H.; Wang, D.; Shi, J. Comparison of spectral indices and principal component analysis for differentiating lodged rice crop from normal ones. In Computer and Computing Technologies in Agriculture V. IFIP Advances in Information and Communication Technology; Li, D., Chen, Y., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; Volume 369, pp. 84–92. [Google Scholar]
  17. Murakami, T.; Yui, M.; Amaha, K. Canopy height measurement by photogrammetric analysis of aerial images: Application to buckwheat (Fagopyrum esculentum Moench) lodging evaluation. Comput. Electron. Agric. 2012, 89, 70–75. [Google Scholar] [CrossRef]
  18. Xiang, H.; Tian, L. Development of a low-cost agricultural remote sensing system based on an autonomous unmanned aerial vehicle (UAV). Biosyst. Eng. 2011, 108, 174–190. [Google Scholar] [CrossRef]
  19. Chapman, S.; Merz, T.; Chan, A.; Jackway, P.; Hrabar, S.; Dreccer, M.; Holland, E.; Zheng, B.; Ling, T.; Jimenez-Berni, J. Pheno-copter: A Low-altitude, autonomous remote-sensing robotic helicopter for high-throughput field-based phenotyping. Agronomy 2014, 4, 279–301. [Google Scholar] [CrossRef] [Green Version]
  20. Burkart, A.; Aasen, H.; Alonso, L.; Menz, G.; Bareth, G.; Rascher, U. Angular dependency of hyperspectral measurements over wheat characterized by a novel UAV based goniometer. Remote Sens. 2015, 7, 725–746. [Google Scholar] [CrossRef] [Green Version]
  21. Wang, J.-J.; Ge, H.; Dai, Q.; Ahmad, I.; Dai, Q.; Zhou, G.; Qin, M.; Gu, C. Unsupervised discrimination between lodged and non-lodged winter wheat: A case study using a low-cost unmanned aerial vehicle. Int. J. Remote Sens. 2018, 39, 2079–2088. [Google Scholar] [CrossRef]
  22. Molaei, B.; Chandel, A.; Peters, R.T.; Khot, L.R.; Vargas, J.Q. Investigating lodging in spearmint with overhead sprinklers compared to drag hoses using entropy values from low altitude RGB-imagery. Inf. Process. Agric. 2021, 9, 335–341. [Google Scholar] [CrossRef]
  23. Tan, S.; Mortensen, A.K.; Ma, X.; Boelt, B.; Gislum, R. Assessment of grass lodging using texture and canopy height distribution features derived from UAV visual-band images. Agric. For. Meteorol. 2021, 308, 108541. [Google Scholar] [CrossRef]
  24. Chauhan, S.; Darvishzadeh, R.; Lu, Y.; Stroppiana, D.; Boschetti, M.; Pepe, M.; Nelson, A. Wheat Lodging Assessment Using Multispectral UAV Data. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2019, XLII-2/W13, 235–240. [Google Scholar] [CrossRef] [Green Version]
  25. Sun, Q.; Sun, L.; Shu, M.; Gu, X.; Yang, G.; Zhou, L. Monitoring maize lodging grades via unmanned aerial vehicle multispectral Image. Plant Phenomics 2019, 2019, 1–16. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Yang, M.-D.; Huang, K.-S.; Kuo, Y.-H.; Tsai, H.; Lin, L.-M. Spatial and spectral hybrid image classification for rice lodging assessment through UAV imagery. Remote Sens. 2017, 9, 583. [Google Scholar] [CrossRef] [Green Version]
  27. Wilke, N.; Siegmann, B.; Klingbeil, L.; Burkart, A.; Kraska, T.; Muller, O.; van Doorn, A.; Heinemann, S.; Rascher, U. Quantifying lodging percentage and lodging severity using a UAV-based canopy height model combined with an objective threshold approach. Remote Sens. 2019, 11, 515. [Google Scholar] [CrossRef] [Green Version]
  28. Zhang, H.; Li, Y.; Zhang, Y.; Shen, Q. Spectral-spatial classification of hyperspectral imagery using a dual-channel convolutional neural network. Remote Sens. Lett. 2017, 8, 438–447. [Google Scholar] [CrossRef] [Green Version]
  29. Hao, X.; Jia, J.; Khattak, A.M.; Zhang, L.; Guo, X.; Gao, W.; Wang, M. Growing period classification of Gynura bicolor DC using GL-CNN. Comput. Electron. Agric. 2020, 174, 105497. [Google Scholar] [CrossRef]
  30. Paymode, A.S.; Malode, V.B. Transfer learning for multi-crop leaf disease image classification using convolutional neural network VGG. Artif. Intell. Agric. 2022, 6, 23–33. [Google Scholar] [CrossRef]
  31. Zhang, Z.; Flores, P.; Igathinathane, C.; Naik, D.L.; Kiran, R.; Ransom, J.K. Wheat lodging detection from UAS imagery using machine learning algorithms. Remote Sens. 2020, 12, 1838. [Google Scholar] [CrossRef]
  32. Subetha, T.; Khilar, R.; Christo, M.S. A comparative analysis on plant pathology classification using deep learning architecture–Resnet and VGG19. Mater. Today Proc. 2021. [Google Scholar] [CrossRef]
  33. Zhao, Y.; Sun, C.; Xu, X.; Chen, J. RIC-Net: A plant disease classification model based on the fusion of inception and residual structure and embedded attention mechanism. Comput. Electron. Agric. 2022, 193, 106644. [Google Scholar] [CrossRef]
  34. Zhang, H.; Feng, L.; Zhang, X.; Yang, Y.; Li, J. Necessary conditions for convergence of CNNs and initialization of convolution kernels. Digit. Signal Process. 2022, 123, 103397. [Google Scholar] [CrossRef]
  35. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2015, arXiv:1409.1556. [Google Scholar]
  36. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2818–2826. [Google Scholar]
  37. Lu, Z.; Bai, Y.; Chen, Y.; Su, C.; Lu, S.; Zhan, T.; Hong, X.; Wang, S. The classification of gliomas based on a pyramid dilated convolution resnet model. Pattern Recognit. Lett. 2020, 133, 173–179. [Google Scholar] [CrossRef]
  38. Glorot, X.; Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; pp. 249–256. [Google Scholar]
  39. Kumpumaki, T.; Linna, P.; Lipping, T. Crop lodging analysis from UAS orthophoto mosaic, sentinel-2 image and crop yield monitor data. In Proceedings of the IGARSS 2018: IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 23–27 July 2018. [Google Scholar]
  40. Ampatzidis, Y.; Partel, V. UAV-based high throughput phenotyping in citrus utilizing multispectral imaging and artificial intelligence. Remote Sens. 2019, 11, 410. [Google Scholar] [CrossRef] [Green Version]
  41. Rajapaksa, S. Classification of crop lodging with gray level co-occurrence matrix. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018. [Google Scholar]
Figure 1. Flowchart of RGB and multispectral dataset acquisition and classification of different lodging extents using deep learning algorithms.
Figure 1. Flowchart of RGB and multispectral dataset acquisition and classification of different lodging extents using deep learning algorithms.
Agriculture 12 00970 g001
Figure 2. Overview of the study area (a) Geographical location of the study area. (b) The UAV RGB image. (c) The UAV multispectral image (false color composite, R: Red, G: NIR, B: Green).
Figure 2. Overview of the study area (a) Geographical location of the study area. (b) The UAV RGB image. (c) The UAV multispectral image (false color composite, R: Red, G: NIR, B: Green).
Agriculture 12 00970 g002
Figure 3. Maize lodging data collection (left), aerial imagery collection; (right), classification of three lodging extents based on crop lodging angle.
Figure 3. Maize lodging data collection (left), aerial imagery collection; (right), classification of three lodging extents based on crop lodging angle.
Agriculture 12 00970 g003
Figure 4. Maize lodging samples after data cleaning.
Figure 4. Maize lodging samples after data cleaning.
Agriculture 12 00970 g004
Figure 5. Original UAV image and three augmented images.
Figure 5. Original UAV image and three augmented images.
Agriculture 12 00970 g005
Figure 6. Structure diagram of the convolutional neural network.
Figure 6. Structure diagram of the convolutional neural network.
Agriculture 12 00970 g006
Figure 7. Structure diagram of VGG-16.
Figure 7. Structure diagram of VGG-16.
Agriculture 12 00970 g007
Figure 8. The optimization procedure of Inception module: (left), architecture of the initial Inception module; (middle), module architecture in Inception-V2; (right), module architecture in Inceptiom-V3.
Figure 8. The optimization procedure of Inception module: (left), architecture of the initial Inception module; (middle), module architecture in Inception-V2; (right), module architecture in Inceptiom-V3.
Agriculture 12 00970 g008
Figure 9. Architecture of the residual block to solve the degradation problem.
Figure 9. Architecture of the residual block to solve the degradation problem.
Agriculture 12 00970 g009
Figure 10. The intensity values variation (left) and the maize canopy under the UAV aerial with different lodging extents (right).
Figure 10. The intensity values variation (left) and the maize canopy under the UAV aerial with different lodging extents (right).
Agriculture 12 00970 g010
Figure 11. Spectral reflectance of four bands under different lodging extents.
Figure 11. Spectral reflectance of four bands under different lodging extents.
Agriculture 12 00970 g011
Figure 12. Accuracy and loss for the training and test sets of the three algorithms through RGB images.
Figure 12. Accuracy and loss for the training and test sets of the three algorithms through RGB images.
Agriculture 12 00970 g012
Figure 13. The confusion matrices of the three algorithms through RGB images. Types of lodging extents: NL is non-lodging, ML is moderate lodging, SL is severe lodging.
Figure 13. The confusion matrices of the three algorithms through RGB images. Types of lodging extents: NL is non-lodging, ML is moderate lodging, SL is severe lodging.
Agriculture 12 00970 g013
Figure 14. Accuracy and loss for the training and test sets of the three algorithms through multispectral images.
Figure 14. Accuracy and loss for the training and test sets of the three algorithms through multispectral images.
Agriculture 12 00970 g014
Figure 15. Confusion matrix for the three algorithms through multispectral images. NL is non-lodging, ML is moderate lodging, SL is severe lodging.
Figure 15. Confusion matrix for the three algorithms through multispectral images. NL is non-lodging, ML is moderate lodging, SL is severe lodging.
Agriculture 12 00970 g015
Figure 16. Classification accuracy of the three algorithms with two image types.
Figure 16. Classification accuracy of the three algorithms with two image types.
Agriculture 12 00970 g016
Table 1. The average intensity values with three lodging extents in different bands.
Table 1. The average intensity values with three lodging extents in different bands.
BandsBlueGreenRed
Extents
Non-lodging82.49103.9994.84
Moderate lodging113.54126.54121.14
Severe lodging126.88138.20133.55
Table 2. The average reflectance with three lodging extents in different bands.
Table 2. The average reflectance with three lodging extents in different bands.
BandsGreenRedRed EdgeNear-Infrared
Extents
Non-lodging0.310.240.410.53
Moderate lodging0.330.270.490.60
Severe lodging0.370.300.560.64
Table 3. Performance of the three algorithms for the test sets of RGB images.
Table 3. Performance of the three algorithms for the test sets of RGB images.
AlgorithmsTest AccuracyKappa
VGG-1683.55%0.7421
Inception-V387.32%0.8040
ResNet-5090.08%0.8599
Table 4. Performance of the three algorithms for the test sets of multispectral images.
Table 4. Performance of the three algorithms for the test sets of multispectral images.
AlgorithmsTest AccuracyKappa
VGG-1688.91%0.8318
Inception-V392.36%0.8935
ResNet-5096.32%0.9551
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Yang, X.; Gao, S.; Sun, Q.; Gu, X.; Chen, T.; Zhou, J.; Pan, Y. Classification of Maize Lodging Extents Using Deep Learning Algorithms by UAV-Based RGB and Multispectral Images. Agriculture 2022, 12, 970. https://doi.org/10.3390/agriculture12070970

AMA Style

Yang X, Gao S, Sun Q, Gu X, Chen T, Zhou J, Pan Y. Classification of Maize Lodging Extents Using Deep Learning Algorithms by UAV-Based RGB and Multispectral Images. Agriculture. 2022; 12(7):970. https://doi.org/10.3390/agriculture12070970

Chicago/Turabian Style

Yang, Xin, Shichen Gao, Qian Sun, Xiaohe Gu, Tianen Chen, Jingping Zhou, and Yuchun Pan. 2022. "Classification of Maize Lodging Extents Using Deep Learning Algorithms by UAV-Based RGB and Multispectral Images" Agriculture 12, no. 7: 970. https://doi.org/10.3390/agriculture12070970

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop