# A Prediction Model of Maize Field Yield Based on the Fusion of Multitemporal and Multimodal UAV Data: A Case Study in Northeast China

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

_{75–100}in the later part of the V period, Point

_{80–100}in the R period, and Point

_{50–100}in the M period, complemented by corresponding meteorological data as inputs. The resulting yield estimation demonstrates exceptional performance, with an R

^{2}value of 0.78 and an rRMSE of 8.27%. These results surpass previous research and validate the effectiveness of multimodal data in enhancing yield prediction models. Furthermore, to assess the superiority of the proposed model, four machine learning algorithms—multiple linear regression (MLR), random forest regression (RF), support vector machine (SVM), and backpropagation (BP)—are compared to the CNN-attention-LSTM model through experimental analysis. The outcomes indicate that all alternative models exhibit inferior prediction accuracy compared to the CNN-attention-LSTM model. Across the test dataset within the study area, the R

^{2}values for various nitrogen fertilizer levels consistently exceed 0.75, illustrating the robustness of the proposed model. This study introduces a novel approach for assessing maize crop yield and provides valuable insights for estimating the yield of other crops.

## 1. Introduction

- Build the CNN-attention-LSTM network model. The model is used to fuse relevant growth parameters and climate data for multiple fertility stages of maize and to make yield predictions.
- Provide a comparison of the effects of different reproductive stages and sensor combinations on the yield prediction model. An evaluation of optimal multitemporal and multimodal maize yield predictor combinations is performed.
- Evaluate the model robustness using data collected in the test area; the adaptability of the proposed CNN-attention-LSTM model to predict maize yield under different fertilization treatments is also verified.

## 2. Materials and Methods

#### 2.1. Study Area and Field Experimental Design

^{2}, resulting in a total of 18 experimental blocks (Figure 1b). During the V

_{3}–V

_{6}phenological periods, 5 rounds of liquid nitrogen fertilizer and 1 round of solid nitrogen fertilizer were applied. Based on the soil conditions in the experimental area and regional recommendations, the 5 rounds of liquid nitrogen fertilizer (45% N in liquid urea solution) had the following application rates: 0, 50, 100, 150, and 200 kg/ha. The solid nitrogen fertilizer (45% N in solid urea mixture) was applied at a rate of 150 kg/ha. The same N fertilizer treatments were used in the validation zone, and five replicate trials were conducted for each group of N fertilizer. Each experimental plot in the training zone had an area of 5 × 20 m

^{2}.

#### 2.2. Data Collection

#### 2.2.1. UAV Data Collection

_{i}is the i-th stage of the vegetative stage, and R

_{i}is the i-th stage of the reproductive stage. The UAV platform was equipped with both a multispectral sensor and a LIDAR sensor.

^{2}. The sampling was performed at a frequency of 160 Hz (Figure 2), and the spatial separation rate is about 1500 pixels/m. All flight missions were autonomously conducted, maintaining a flight altitude of 30 m and a flight speed of 1.2 m/s.

#### 2.2.2. Field Data Collection

^{2}. LAI was then calculated using Equation (2).

_{i}). D is the planting density of maize, and LAI is the leaf area index of maize leaves.

^{2}) based on the planting density.

_{6}). During the R

_{6}of maize, a designated area measuring 10 m × 2.2 m (4 rows in the middle of the plot) was manually harvested in each plot to determine the actual yield [24]. The harvested maize crops were transported to the laboratory, threshed, and adjusted to a standard moisture content of 14%. The number of rows, number of grains per row, and weight of 500 grains were measured from the harvested maize ears. The maize yield was calculated by weighing the grains from the sampled area and applying a weightage factor (Figure 2). Additionally, this study assessed the impact of different nitrogen application rates on maize yield by evaluating the apparent nitrogen use efficiency (aNUE) using Equation (3).

_{Ni}represents the maize yield at the ith nitrogen level, C

_{N0}represents the maize yield without nitrogen application, and N

_{i}represents the nitrogen content at the ith level.

#### 2.2.3. Meteorological Data

^{−2}day

^{−1}). These variables were collected and used as input variables for yield prediction.

#### 2.3. Data Processing and Dataset Construction

#### 2.3.1. Point Cloud Data Processing

_{0–20}, Point

_{20–40}, Point

_{40–60}, Point

_{60–80}, and Point

_{80–100}. The four-level point cloud data were labeled as Point

_{0–25}, Point

_{25–50}, Point

_{50–75}, and Point

_{75–100}. The two-level point cloud data were labeled as Point

_{0–50}and Point

_{50–100}.

_{ext}(X,t) and the internal driving factor F

_{int}(X,t). Assuming that only the external factor has an influence, the internal driving factor is set to 0, and their relationship is described by Equation (5).

_{int}(X,t) can control the issue of particle inversion in blank areas (Equation (6)). By incorporating both the internal and external factors, the height differences between the LIDAR point cloud and particles are calculated, resulting in the classification of ground points.

#### 2.3.2. Multispectral Image Processing

_{i}× DN

_{i}+ b represents radiance and E

_{0}denotes the solar irradiance, which varies for different spectral bands. Typically, d is set to 1.

#### 2.4. CNN-Attention-LSTM for Maize Yield Prediction

_{t}denotes the i-th hidden state and y is the corresponding feature map mapping.

_{t}(Equation (10)). In Equation (10), W denotes the weight matrix and b represents the bias vector. The weights, α

_{t}, are obtained by normalizing using the softmax function (Equation (12)). Ultimately, the vector z represents the weighted sum of the hidden states.

#### 2.5. Modelling and Evaluation Indices

^{2}), root mean square error (RMSE), relative root mean square error (rRMSE), and relative percentage difference (RPD) to evaluate the yield prediction results of the model in a comprehensive manner [31]. Usually, models are considered to have better prediction when they have a higher R

^{2}and lower rRMSE. The relevant equations are shown in Table 3.

## 3. Results

#### 3.1. Analysis of Growth Characteristics and Yield Prediction under Different Nitrogen Levels

_{1}to N

_{4}range, while a decrease in yield is observed at the N

_{5}level. The yield at the N

_{5}level is similar to that at the N

_{3}level. In contrast, the distribution pattern of aNUE shows an opposite trend to yield. Within the N

_{1}to N

_{4}range, aNUE decreases as nitrogen levels increase, but it increases at the N

_{5}level. These findings indicate a correlation between maize yield and nitrogen content, underscoring the importance of monitoring maize growth under different nitrogen levels and predicting yield. Moreover, the measured data from the experimental and validation zones exhibit similar distribution patterns, indicating the suitability of the selected validation plots for the experiment and the usability of the extracted data for the subsequent evaluation of yield prediction models.

#### 3.2. Results of Maize Grain Yield Prediction Based on Different Reproductive-Stage Remote Sensing Data

#### 3.2.1. Maize Grain Yield Prediction Results Based on Different Reproductive-Stage Multispectral Data

^{2}value shows a clear increasing trend. Using the full growth period as input variables yields better prediction results compared to using data from only two or three stages of the growth period. When utilizing the full growth period’s multispectral data as input variables, the model achieves an R

^{2}value of 0.67 and an rRMSE of 15.7%. As more reproductive stages are included, the prediction performance improves. Additionally, including data from the initial part of the R period as input variables significantly enhances the yield prediction. Among the experiments using three reproductive stages as input variables, the combination of the later parts of the V, R, and M periods demonstrates the best performance, with an R

^{2}value of 0.65 and an rRMSE of 19.4%. Conversely, excluding R period data from the reproductive stage combination (specifically using the initial part of the V period, the later part of the V period, and the M period) leads to the worst performance, with an R

^{2}value of 0.51 and an rRMSE of 23.9%.

#### 3.2.2. Maize Growth Parameter Extraction and Yield Prediction Results Based on LIDAR Data

_{50–100}had the highest correlation coefficient with CHM, reaching 0.52, and a correlation coefficient of 0.49 with LAI. In the later part of the V period, Point

_{75–100}showed a correlation coefficient of 0.65 with LAI and 0.71 with CHM. For the R period, Point

_{80–100}was selected for parameter extraction, and for the M period, Point

_{50–100}was selected. To conduct yield prediction experiments, this research work extracted three-dimensional features from the selected point cloud data.

^{2}value of 0.48 and an rRMSE of 29.1%. When using point cloud data for two time periods as input variables, the combination of the later parts of the V and R periods showed the best training performance, with an R

^{2}value of 0.54 and an rRMSE of 24.3%. When using point cloud data for three time periods as input variables, the combination of the later parts of the V, R, and M periods yielded the best training performance, with an R

^{2}value of 0.62 and an rRMSE of 19.6%. These results showed only a slight difference compared to the prediction results using the full growth period’s LIDAR data. The experimental results demonstrated that the point cloud data for the later parts of the V, R, and M periods achieved higher accuracy and stability. Therefore, this study selected the combination of these three reproductive stages as the model parameters for LIDAR data. The experimental results are presented in Table 5. Better yield prediction results can provide more accurate and detailed information on productivity crops in the field while allowing for more significant yield variation at different N levels.

#### 3.3. Yield Prediction Results Based on CNN-Attention-LSTM Model

#### 3.3.1. Impact of Attention Mechanism on Yield Prediction Results

^{2}values for window intervals of 5, 10, and 15 s were 0.62, 0.73, and 0.59, respectively. In contrast, the CNN-attention-LSTM model with an attention mechanism achieved R

^{2}values of 0.69, 0.78, and 0.64 for the respective window intervals. Notably, the CNN-attention-LSTM model demonstrated an improvement of 0.05 in R

^{2}for the 10 s window interval compared to the CNN-LSTM model. The experimental results consistently demonstrated the superior performance of the CNN-attention-LSTM model with an attention mechanism. The attention mechanism effectively fused the multimodal data by assigning different weights to different features, leading to positive impacts on the prediction results. Based on these findings, the 10 s window interval was selected for subsequent data processing.

#### 3.3.2. Comparative Analysis of Yield Prediction Using Different Regression Methods

^{2}value of 0.42 when using the data from the initial part of the V period and the later parts of the V, R, and M periods as inputs in the training area. RF achieved the highest R

^{2}value of 0.71 and an rRMSE of 13.4%, demonstrating higher stability and accuracy. However, the CNN-attention-LSTM model with an ElasticNet layer outperformed all the other models, achieving an R

^{2}value of 0.78 and an rRMSE of 8.27%. The comparative results of the multiple regression models are presented in Table 6.

#### 3.4. Yield Prediction Results of Multimodal Data Fusion

^{2}value of 0.78 and an rRMSE of 8.27% for yield prediction. These results demonstrate the strong predictive capability of the multimodal yield prediction model, making it an effective tool for maize yield estimation. The fusion of multiple datasets significantly improved the R

^{2}value compared to using single-type remote sensing data alone. Additionally, the multimodal fusion outperformed the combination of single-type remote sensing data with meteorological data, highlighting the complementary nature of different modalities when fused together.

^{2}value of 0.78.

## 4. Discussion

#### 4.1. Correlation Analysis between UAV Remote Sensing Data and Maize Growth

^{2}was selected as the standard for point cloud extraction to fulfill the requirements of feature extraction throughout the entire reproductive period.

#### 4.2. Analysis of the Impact of Different Reproductive Stages on Maize Yield Prediction

^{2}of 0.36 and an rRMSE of 40.2%. The R

^{2}value significantly increased during the later part of the V period, reaching 0.59 during the R period. The R

^{2}value slightly decreased to 0.54 during the M period, with an rRMSE of 26.1%. These experimental results indicate that the R period is the optimal reproductive stage for yield prediction, and assigning more weight to features from the R period can improve the accuracy of yield prediction. The overall R

^{2}for yield prediction across the entire experimental area was 0.78, which is 0.19 higher than the R period alone. These findings, combined with the results from Section 3.2, demonstrate that a single reproductive stage cannot fully capture crop growth characteristics, and the fusion of multiple time-period features is the optimal approach for yield prediction.

^{2}value of 0.75 was achieved when the independent variables included the later parts of the V, R, and M periods. The lowest R

^{2}value of 0.62 and the highest rRMSE of 21.3% were observed when the independent variables included the initial part of V, the later part of the V period, and the M period. These experimental results confirm that the R period is the optimal stage for yield prediction, while the initial part of the V period has the least impact on yield prediction. These findings are consistent with the results obtained from the single-reproductive-stage experiments. The yield prediction results are closely related to the monitored parameters during the selected reproductive stages in this study. As the maize growth cycle progresses, the acquired multispectral data and LIDAR point cloud data also undergo changes. Therefore, the accuracy of the proposed CNN-attention-LSTM yield prediction model may vary accordingly. During the R period, the yield prediction performance in the validation area is optimal and aligns with the results from the training area. However, the prediction accuracy in the validation area is slightly lower, indicating the need to further enhance the robustness of the model. During the later part of the V period, the rapid changes in canopy coverage and maize pH result in the point cloud data receiving more attention from the attention mechanism in the yield prediction model. As the maize crop enters the R period, the emergence of tassels can alter the canopy structure, thereby increasing the correlation between spectral features and various maize growth parameters. Consequently, this modification in the relationship between the response characteristics of remote sensing data, such as VIs, and maize growth parameters enhances the impact on yield prediction [39].

#### 4.3. Correlation Analysis between UAV Remote Sensing Data and Maize Growth

#### 4.4. Potential of Multimodal Data Integration with Deep Learning for Maize Yield Prediction

^{2}reached 0.78 in the first 24 days of harvest. However, the multimodal fusion estimation of UAV remote sensing data is more applicable to yield prediction at the field scale. Previous studies have often relied on manual feature extraction methods for multimodal feature fusion, which require large-window data and do not align with the practicality of field-scale crop information collection. Moreover, large-window data are less compatible with small-scale variations in data, and can only be effectively utilized when there are significant changes in multidimensional features due to substantial differences in nitrogen content. However, the CNN-attention-LSTM model, which enables the automatic extraction of multimodal features, simultaneously processes different modalities using CNN, allowing the use of relatively smaller window data to improve yield prediction performance under different nitrogen treatments. The finer feature extraction and fusion techniques facilitate a more effective utilization of multidimensional maize structures [44].

## 5. Conclusions

^{2}value of 0.78 and an rRMSE of 8.27% [18,45]. The fusion of multimodal data also outperforms individual modalities. The self-attention mechanism in the model allows for the assignment of varying weights to different features. In comparison, the CNN-LSTM model without self-attention achieves an R

^{2}value of 0.73 and an rRMSE of 13.2%. These results indicate that self-attention effectively balances the information from multiple dimensions.

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- van Dijk, M.; Morley, T.; Rau, M.-L.; Saghai, Y. A meta-analysis of projected global food demand and population at risk of hunger for the period 2010–2050. Nat. Food
**2021**, 2, 494–501. [Google Scholar] [CrossRef] - De Schutter, O. The political economy of food systems reform. Eur. Rev. Agric. Econ.
**2017**, 44, 705–731. [Google Scholar] [CrossRef] - Ranum, P.; Peña-Rosas, J.P.; Garcia-Casal, M.N. Global maize production, utilization, and consumption. Ann. N. Y. Acad. Sci.
**2014**, 1312, 105–112. [Google Scholar] [CrossRef] - Zhou, X.; Zheng, H.B.; Xu, X.Q.; He, J.Y.; Ge, X.K.; Yao, X.; Cheng, T.; Zhu, Y.; Cao, W.X.; Tian, Y.C.; et al. Predicting grain yield in rice using multi-temporal vegetation indices from UAV-based multispectral and digital imagery. SPRS J. Photogramm. Remote Sens.
**2017**, 130, 246–255. [Google Scholar] [CrossRef] - Jin, X.; Li, Z.; Yang, G.; Yang, H.; Feng, H.; Xu, X.; Wang, J.; Li, X.; Luo, J.J. Winter wheat yield estimation based on multi-source medium resolution optical and radar imaging data and the AquaCrop model using the particle swarm optimization algorithm. ISPRS J. Photogramm. Remote Sens.
**2017**, 126, 24–37. [Google Scholar] [CrossRef] - Ziliani, M.G.; Altaf, M.; Aragon, B.; Houborg, R.; Franz, T.E.; Lu, Y.; Sheffield, J.; Hoteit, I.; McCabe, M.F. Early season prediction of within-field crop yield variability by assimilating CubeSat data into a crop model. Agric. For. Meteorol.
**2022**, 313, 108736. [Google Scholar] [CrossRef] - Huang, J.; Tian, L.; Liang, S.; Ma, H.; Becker-Reshef, I.; Huang, Y.; Su, W.; Zhang, X.; Zhu, D.; Wu, W.J.A.; et al. Improving winter wheat yield estimation by assimilation of the leaf area index from Landsat TM and MODIS data into the WOFOST model. Agric. For. Meteorol.
**2015**, 204, 106–121. [Google Scholar] [CrossRef] - Nakajima, K.; Tanaka, Y.; Katsura, K.; Yamaguchi, T.; Watanabe, T.; Tatsuhiko, S. Biomass estimation of World rice (Oryza sativa L.) core collection based on the convolutional neural network and digital images of canopy. Plant Prod. Sci.
**2023**, 26, 187–196. [Google Scholar] [CrossRef] - Hassan, M.A.; Yang, M.; Rasheed, A.; Yang, G.; Reynolds, M.; Xia, Z.; Xiao, Y.; He, Z. A rapid monitoring of NDVI across the wheat growth cycle for grain yield prediction using a multi-spectral UAV platform. Plant Sci.
**2019**, 282, 95–103. [Google Scholar] [CrossRef] - Wan, L.; Cen, H.; Zhu, J.; Zhang, J.; Zhu, Y.; Sun, D.; Du, X.; Zhai, L.; Weng, H.; Li, Y.; et al. Grain yield prediction of rice using multi-temporal UAV-based RGB and multispectral images and model transfer—A case study of small farmlands in the South of China. Agric. For. Meteorol.
**2020**, 291, 108096. [Google Scholar] [CrossRef] - López García, P.; Ortega, J.; Pérez-Álvarez, E.; Moreno, M.; Ramírez, J.; Intrigliolo, D.; Ballesteros, R. Yield estimations in a vineyard based on high-resolution spatial imagery acquired by a UAV. Biosyst. Eng.
**2022**, 224, 227–245. [Google Scholar] [CrossRef] - Gong, Y.; Duan, B.; Fang, S.; Zhu, R.; Wu, X.; Ma, Y.; Peng, Y. Remote estimation of rapeseed yield with unmanned aerial vehicle (UAV) imaging and spectral mixture analysis. Plant Methods
**2018**, 14, 70. [Google Scholar] [CrossRef] - Zhang, Y.; Yang, Y.; Zhang, Q.; Duan, R.; Liu, J.; Qin, Y.; Wang, X. Toward Multi-Stage Phenotyping of Soybean with Multimodal UAV Sensor Data: A Comparison of Machine Learning Approaches for Leaf Area Index Estimation. Remote Sens.
**2023**, 15, 7. [Google Scholar] [CrossRef] - Luo, S.; Liu, W.; Zhang, Y.; Wang, C.; Xi, X.; Nie, S.; Ma, D.; Lin, Y.; Zhou, G. Maize and soybean heights estimation from unmanned aerial vehicle (UAV) LiDAR data. Comput. Electron. Agric.
**2021**, 182, 106005. [Google Scholar] [CrossRef] - Klompenburg, T.V.; Kassahun, A.; Catal, C. Crop yield prediction using machine learning: A systematic literature review. Comput. Electron. Agric.
**2020**, 177, 105709. [Google Scholar] [CrossRef] - Hta, B.; Pwa, B.; Kt, C.; Jza, B.; Sz, D.; Hl, D.J.A.; Meteorology, F. An LSTM neural network for improving wheat yield estimates by integrating remote sensing data and meteorological data in the Guanzhong Plain, PR China. Agric. For. Meteorol.
**2021**, 310, 108629. [Google Scholar] - Tian, H.; Wang PTansey, K.; Zhang, S.; Zhang, J.; Li, H. An IPSO-BP neural network for estimating wheat yield using two remotely sensed variables in the Guanzhong Plain, PR China. Comput. Electron. Agric.
**2020**, 169, 105180. [Google Scholar] [CrossRef] - Yang, W.; Nigon, T.; Hao, Z.; Paiao, G.D.; Fernandez, F.G.; Mulla, D.; Yang, C. Estimation of corn yield based on hyperspectral imagery and convolutional neural network. Comput. Electron. Agric.
**2021**, 184, 106092. [Google Scholar] [CrossRef] - Mou, L.; Zhou, C.; Zhao, P.; Nakisa, B.; Gao, W. Driver Stress Detection via Multimodal Fusion Using Attention-based CNN-LSTM. Expert Syst. Appl.
**2020**, 173, 114693. [Google Scholar] [CrossRef] - Ma, J.; Liu, B.; Ji, L.; Zhu, Z.; Wu, Y.; Jiao, W. Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery. Int. J. Appl. Earth Obs. Geoinf.
**2023**, 118, 103292. [Google Scholar] [CrossRef] - Fei, S.; Hassan, M.A.; Xiao, Y.; Su, X.; Chen, Z.; Cheng, Q.; Duan, F.; Chen, R.; Ma, Y. UAV-based multi-sensor data fusion and machine learning algorithm for yield prediction in wheat. Precis. Agric.
**2023**, 24, 187–212. [Google Scholar] [CrossRef] - Lou, Z.; Quan, L.; Sun, D.; Li, H.; Xia, F. Hyperspectral remote sensing to assess weed competitiveness in maize farmland ecosystems. Sci. Total Environ.
**2022**, 844, 157071. [Google Scholar] [CrossRef] [PubMed] - Li, D.; Cheng, T.; Zhou, K.; Zheng, H.; Yao, X.; Tian, Y.; Zhu, Y.; Cao, W. WREP: A wavelet-based technique for extracting the red edge position from reflectance spectra for estimating leaf and canopy chlorophyll contents of cereal crops. ISPRS J. Photogramm. Remote Sens.
**2017**, 129, 103–117. [Google Scholar] [CrossRef] - Li, R.; Zheng, J.; Xie, R.; Ming, B.; Peng, X.; Luo, Y.; Zheng, H.; Sui, P.; Wang, K.; Hou, P.; et al. Potential mechanisms of maize yield reduction under short-term no-tillage combined with residue coverage in the semi-humid region of Northeast China. Soil Tillage Res.
**2022**, 217, 105289. [Google Scholar] [CrossRef] - Meteorological Data Network in China. Available online: http://www.nmic.cn/ (accessed on 12 November 2022).
- Cao, L.; Coops, N.C.; Sun, Y.; Ruan, H.; She, G. Estimating canopy structure and biomass in bamboo forests using airborne LiDAR data. ISPRS J. Photogramm. Remote Sens.
**2019**, 148, 114–129. [Google Scholar] [CrossRef] - Elsayed, S.; Thyssens, D.; Rashed, A.; Schmidt-Thieme, L.; Jomaa, H.S. Do We Really Need Deep Learning Models for Time Series Forecasting? arXiv
**2021**, arXiv:2101.02118. [Google Scholar] - Shi, X.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada, 7–12 December 2015. [Google Scholar]
- Saini, P.; Nagpal, B.; Garg, P.; Kumar, S. CNN-BI-LSTM-CYP: A deep learning approach for sugarcane yield prediction. Sustain. Energy Technol. Assess.
**2023**, 57, 103263. [Google Scholar] [CrossRef] - Enwere, K.; Ogoke, U. A Comparative Approach on Bridge and Elastic Net Regressions. Afr. J. Math. Stat. Stud.
**2023**, 6, 67–79. [Google Scholar] [CrossRef] - Tabrizchi, H.; Razmara, J.; Mosavi, A. Thermal prediction for energy management of clouds using a hybrid model based on CNN and stacking multi-layer bi-directional LSTM. Energy Rep.
**2023**, 9, 2253–2268. [Google Scholar] [CrossRef] - Fitzgerald, G.J.; Rodriguez, D.; Christensen, L.K.; Belford, R.; Sadras, V.O.; Clarke, T.R. Spectral and thermal sensing for nitrogen and water status in rainfed and irrigated wheat environments. Precis. Agric.
**2006**, 7, 233–248. [Google Scholar] [CrossRef] - Siegmann, B.; Jarmer, T. Comparison of different regression models and validation techniques for the assessment of wheat leaf area index from hyperspectral data. Int. J. Remote Sens.
**2015**, 36, 4519–4534. [Google Scholar] [CrossRef] - Martins, G.D.; Sousa Santos, L.C.; dos Santos Carmo, G.J.; da Silva Neto, O.F.; Castoldi, R.; Machado, A.I.M.R.; de Oliveira Charlo, H.C. Multispectral images for estimating morphophysiological and nutritional parameters in cabbage seedlings. Smart Agric. Technol.
**2023**, 4, 100211. [Google Scholar] [CrossRef] - ten Harkel, J.; Bartholomeus, H.; Kooistra, L. Biomass and Crop Height Estimation of Different Crops Using UAV-Based Lidar. Remote Sens.
**2020**, 12, 17. [Google Scholar] [CrossRef] - García, M.; Saatchi, S.; Ustin, S.; Balzter, H. Modelling forest canopy height by integrating airborne LiDAR samples with satellite Radar and multispectral imagery. Int. J. Appl. Earth Obs. Geoinf.
**2018**, 66, 159–173. [Google Scholar] [CrossRef] - Maimaitijiang, M.; Sagan, V.; Sidike, P.; Hartling, S.; Fritschi, F.B. Soybean yield prediction from UAV using multimodal data fusion and deep learning. Remote Sens. Environ.
**2019**, 237, 111599. [Google Scholar] [CrossRef] - Jin, X.; Liu, S.; Baret, F.; Hemerlé, M.; Comar, A. Estimates of plant density of wheat crops at emergence from very low altitude UAV imagery. Remote Sens. Environ.
**2017**, 198, 105–114. [Google Scholar] [CrossRef] - Sagan, V.; Maimaitijiang, M.; Bhadra, S.; Maimaitiyiming, M.; Brown, D.R.; Sidike, P.; Fritschi, F.B. Field-scale crop yield prediction using multi-temporal World View-3 and PlanetScope satellite data and deep learning. ISPRS J. Photogramm. Remote Sens.
**2021**, 174, 265–281. [Google Scholar] [CrossRef] - Xia, F.; Lou, Z.; Sun, D.; Li, H.; Quan, L. Weed resistance assessment through airborne multimodal data fusion and deep learning: A novel approach towards sustainable agriculture. Int. J. Appl. Earth Obs. Geoinf.
**2023**, 120, 103352. [Google Scholar] [CrossRef] - Ju, S.; Lim, H.; Ma, J.; Kim, S.; Lee, K.; Zhao, S.; Heo, J. Optimal county-level crop yield prediction using MODIS-based variables and weather data: A comparative study on machine learning models. Agric. For. Meteorol.
**2021**, 307, 108530. [Google Scholar] [CrossRef] - Sun, Z.; Li, Q.; Jin, S.; Song, Y.; Xu, S.; Wang, X.; Cai, J.; Zhou, Q.; Ge, Y.; Zhang, R.; et al. Simultaneous Prediction of Wheat Yield and Grain Protein Content Using Multitask Deep Learning from Time-Series Proximal Sensing. Plant Phenomics
**2022**, 2022, 1–13. [Google Scholar] [CrossRef] - Cheng, M.; Penuelas, J.; McCabe, M.F.; Atzberger, C.; Jiao, X.; Wu, W.; Jin, X. Combining multi-indicators with machine-learning algorithms for maize yield early prediction at the county-level in China. Agric. For. Meteorol.
**2022**, 323, 109057. [Google Scholar] [CrossRef] - Rischbeck, P.; Elsayed, S.; Mistele, B.; Barmeier, G.; Heil, K.; Schmidhalter, U. Data fusion of spectral, thermal and canopy height parameters for improved yield prediction of drought stressed spring barley. Eur. J. Agron.
**2016**, 78, 44–59. [Google Scholar] [CrossRef] - Ma, Y.; Zhang, Z.; Kang, Y.; Özdoğan, M. Corn yield prediction and uncertainty analysis based on remotely sensed variables using a Bayesian neural network approach. Remote Sens. Environ.
**2021**, 259, 112408. [Google Scholar] [CrossRef] - Jiang, Y.; Wei, H.; Hou, S.; Yin, X.; Wei, S.; Jiang, D. Estimation of Maize Yield and Protein Content under Different Density and N Rate Conditions Based on UAV Multi-Spectral Images. Agronomy
**2023**, 13, 421. [Google Scholar] [CrossRef]

**Figure 1.**Experimental area overview: (

**a**) geographic location of the study area; (

**b**) experimental layout of the test blocks; and (

**c**) experimental layout of the validation blocks.

**Figure 4.**(

**a**) Yield variation under different nitrogen levels. (

**b**) Difference in aNUE under different nitrogen levels, with aNUE being 0 under N1 treatment.

**Figure 5.**Correlation between remote sensing data and field yield at different nitrogen levels. (

**a**) Correlation between multispectral data at different nitrogen levels and field yield during various reproductive stages. (

**b**) Correlation between LIDAR point cloud data at different nitrogen levels and field yield during various reproductive stages.

**Figure 6.**Heat map of correlation coefficients between VIs and yield across multiple reproductive stages.

**Figure 7.**Comparison of predictions with and without an attention mechanism at different window intervals.

**Figure 8.**Maize yield prediction results based on the fusion of multispectral, LIDAR, and meteorological data using the CNN-attention-LSTM model.

**Figure 9.**Maize yield prediction results at different reproductive stages. (

**a**) Maize yield prediction results for individual reproductive stages. (

**b**) Maize yield prediction results for combinations of three reproductive stages.

Dates | Phenological Stages |
---|---|

15 June 2022 | V3–V4 |

29 June 2022 | V6–V7 (plucking stage) |

13 July 2022 | V9 (growth rate rapidly increases) |

18 July 2022 | V12 (trumpeting stage) |

24 July 2022 | V12 (trumpeting stage) |

1 August 2022 | VT (tasseling stage) |

8 August 2022 | R1 (silking stage) |

16 August 2022 | R3 (milk stage) |

12 September 2022 | R4–R5 (dough stage) |

6 October 2022 | R6 (physiological maturity) |

Index | Formulas |
---|---|

NDVI | $({R}_{NIR}-{R}_{RED})/({R}_{NIR}+{R}_{RED})$ |

DVI | R_{NIR} − R_{RED} |

GNDVI | (R_{NIR} − R_{RED})/(R_{NIR} + R_{GRE}) |

RDVI | $({R}_{NIR}-{R}_{RED})/\sqrt{{R}_{NIR}+{R}_{RED}}$ |

RVI | ${R}_{NIR}/{R}_{RED}$ |

EVI | $2.5\times ({R}_{NIR}-{R}_{RED})/({R}_{NIR}+6.0\times {R}_{RED}-7.5\times {R}_{BLUE}+1)$ |

SAVI | $(({R}_{NIR}-{R}_{RED})/({R}_{NIR}+{R}_{RED}+0.16))\times (1+0.5)$ |

GCI | $\frac{NIR}{G\mathrm{R}E}-1$ |

NDRE | $NDRE=({R}_{NIR}-{R}_{RED\_EDGE})/({R}_{NIR}+{R}_{RED\_EDGE})$ |

Evaluation Metric | Calculation Formula |
---|---|

R^{2} | $1-\frac{{\displaystyle \sum _{\mathrm{i}=1}^{n}{({Y}_{i}-{X}_{i})}^{2}}}{{\displaystyle \sum _{\mathrm{i}=1}^{n}{({Y}_{i}-\overline{Y})}^{2}}}$ |

RMSE | $\sqrt{\frac{{\displaystyle \sum _{i=1}^{n}({Y}_{\mathrm{i}}-{X}_{i}{)}^{2}}}{n}}$ |

$\mathrm{r}RMSE$ | $\frac{RMSE}{\overline{\mathrm{y}}}$ |

RPD | $\frac{STD}{RMSE}$ |

**Table 4.**Maize yield prediction results based on different VI combinations at various reproductive stages.

Different Fertility Combinations | The Reproductive Period | R^{2} | RPD | rRMSE |
---|---|---|---|---|

Two reproductive stages | Later part of V period, R period | 0.33 | 1.20 | 37.9% |

Later part of V period, M period | 0.45 | 1.36 | 15.7% | |

R period, M period | 0.61 | 1.47 | 20.1% | |

Three reproductive stages | Initial part of V period, Later part of V period, R period | 0.56 | 1.45 | 21.7% |

Initial part of V period, Later part of V period, M period | 0.51 | 1.34 | 23.9% | |

Later part of V period, R period, M period | 0.65 | 1.71 | 19.4% | |

All reproductive stages | Initial part of V period, Later part of V period, R period, M period | 0.67 | 1.83 | 15.7% |

**Table 5.**Maize yield prediction results based on different LIDAR data combinations at various reproductive stages.

Different Fertility Combinations | R^{2} | RPD | rRMSE |
---|---|---|---|

One reproductive stage (R period) | 0.48 | 1.38 | 29.1% |

Two reproductive stages (later part of V period, R period) | 0.54 | 1.66 | 24.3% |

Three reproductive stages (later part of V period, R period, M period) | 0.62 | 1.87 | 19.6% |

Yield Forecasting Models | R^{2} | RPD | rRMSE |
---|---|---|---|

ElasticNet | 0.78 | 2.31 | 8.27% |

MLR | 0.42 | 1.26 | 34.6% |

RF | 0.71 | 1.89 | 13.4% |

SVM | 0.63 | 1.64 | 19.7% |

BP | 0.53 | 1.47 | 25.6% |

Different Modal Combinations | R^{2} | RPD | rRMSE |
---|---|---|---|

Multispectral data | 0.67 | 1.83 | 15.7% |

LIDAR data | 0.62 | 1.87 | 19.6% |

Multispectral data, LIDAR data | 0.74 | 2.23 | 10.1% |

Multispectral data, meteorological data | 0.71 | 2.05 | 13.5% |

LIDAR data, meteorological data | 0.66 | 1.98 | 16.6% |

Multispectral data, LIDAR data, meteorological data | 0.78 | 2.31 | 8.27% |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zhou, W.; Song, C.; Liu, C.; Fu, Q.; An, T.; Wang, Y.; Sun, X.; Wen, N.; Tang, H.; Wang, Q.
A Prediction Model of Maize Field Yield Based on the Fusion of Multitemporal and Multimodal UAV Data: A Case Study in Northeast China. *Remote Sens.* **2023**, *15*, 3483.
https://doi.org/10.3390/rs15143483

**AMA Style**

Zhou W, Song C, Liu C, Fu Q, An T, Wang Y, Sun X, Wen N, Tang H, Wang Q.
A Prediction Model of Maize Field Yield Based on the Fusion of Multitemporal and Multimodal UAV Data: A Case Study in Northeast China. *Remote Sensing*. 2023; 15(14):3483.
https://doi.org/10.3390/rs15143483

**Chicago/Turabian Style**

Zhou, Wenqi, Chao Song, Cunliang Liu, Qiang Fu, Tianhao An, Yijia Wang, Xiaobo Sun, Nuan Wen, Han Tang, and Qi Wang.
2023. "A Prediction Model of Maize Field Yield Based on the Fusion of Multitemporal and Multimodal UAV Data: A Case Study in Northeast China" *Remote Sensing* 15, no. 14: 3483.
https://doi.org/10.3390/rs15143483