Use of CNN for Water Stress Identification in Rice Fields Using Thermal Imagery

Abstract

Rice is a staple food in many Asian countries, but its production requires a high water demand. Moreover, more attention should be paid to the water management of rice due to global climate change and frequent droughts. To address this problem, we propose a rice water stress identification system. Since water irrigation usually affects the opening and closing of rice leaf stomata which directly affects leaf temperature, rice leaf temperature is a suitable index for evaluating rice water stress. The proposed rice water stress identification system uses a CNN (convolutional neural network) to identify water stress in thermal images of rice fields and to classify the irrigation situation into three classes: 100%, 90%, and 80% irrigation. The CNN was applied to extract the temperature level score from each thermal image based on the degree of difference between the three irrigation situations, then these scores were used to further classify the water-stress situation. In the experiments in this study, we compare CNN classification results without considering the degree between each class. The proposed method considerably improves water stress identification. Since rice leaf temperature is relative to air temperature and is not an absolute value, the background temperature is also important reference information. We combine two different methods for background processing to extract more features and achieve more accurate identification.

1. Introduction

Many studies have shown that water stress in rice reduces its yield and quality [1,2,3]. Wang et al. [4] observed that drought reduced the average rice yield by 6.8% in China. In addition, Zubaer et al. [5] evaluated the effects of three water levels (100%, 70%, and 40%) on three rice varieties, showing that the grain yield was reduced by 20.74–32.51% at the 70% water level and by 45.34–53.08% at the 40% water level. Furthermore, Hossain et al. [6] reported that under 75% and 50% water saturation, the panicle number/rice plant was approximately one, while it was five under saturated conditions. He et al. [7] also showed that reduced irrigation significantly decreased rice yield. These studies illustrate the importance of water to rice yield; however, unfortunately, many meteorologists forecast that global droughts will become more severe in future [8,9,10,11], thereby causing food supply problems in countries where rice is the staple food. Taking other plants as an example, Rani et al. [12] emphasized that chickpea yields were drastically reduced due to drought. In response to this predicament, many studies have proposed various solutions, such as the stricter management of irrigation [13]. In addition, botanists are attempting to cultivate new drought-tolerant rice varieties [14,15,16] to reduce water usage. However, these proposals require an automatic irrigation monitoring system; therefore, the purpose of this article was to develop water stress identification for rice fields.

The proposed system is based on leaf temperature. Since plant leaves contain stomata, the opening and closing of the stomata are affected by water irrigation. When there is sufficient water, the stomata often open; otherwise, the stomata are mostly closed. If the stomata are open, the leaf temperature is lower, and vice versa [17,18]. A thermal imaging camera is a suitable choice for simple measurement of leaf temperature as it can record leaf temperature by taking pictures and does not require the manual measurement of each leaf. In addition, it is a non-contact measurement, so causes no damage to the leaf. Therefore, thermal imaging cameras are widely used to measure the temperature of various plants [19].

Recently, the CNN has become the most successful method for image recognition [20] and is commonly used to identify plant phototypes. For example, the VGG16 model has been used to identify chickpea seed varieties [21] and different varieties of grape leaves [22], while DenseNet has been used to identify PlantVillage (a well-known plant disease database) [23]. These applications show that CNNs are advantageous for plant phototype identification. Typically, a CNN extracts features by training convolution kernel weights and uses these features for recognition by training fully connected layers. In this article, a CNN was used to identify water stress in rice from thermal images, adopting three well-known CNN architectures as the network backbone [24,25,26]. In addition, Taheri-Garavand et al. [27] proposed a plant leaf water content estimation system based on visible light cameras. In their method, six texture features were extracted in three color spaces and used to estimate the leaf water content. This usually requires more prior knowledge, while texture features require stricter shooting conditions; for example, the shooting distance must be sufficient to capture the texture. Since thermal images directly reflect the water content of the leaves, we do not need to consider prior knowledge or shooting conditions. Therefore, using thermal imaging is a relatively easy way to estimate the water content of plant leaves. Recently, due to the rapid development of CNNs, the technology has been applied in many studies to identify the problem of water stress in plants. Chandel et al. [28] used three common CNN architectures (AlexNet, GoogLeNet, and Inception v3) to identify stress and non-stress in RGB images of corn, okra, and soybean. In addition, Zhuanga et al. [29] built a model for water stress identification in corn using a simple six-layer convolutional network model to extract RGB image features to evaluate water stress with an SVM and an extra trees regressor. Both these methods are based on RGB images to build models; however, water stress is usually identified in such images by observing shrinkage on the surface of the plant leaf. For rice, such water stress is already very serious and greatly affects the production quality, so it is more appropriate to use thermal images to identify rice water stress.

Several previous studies have used CNN to identify plant drought problems with thermal images. First, Niu et al. [30] classified the irrigation of pomegranate trees as high (100% and 75%) and low (35% and 50%), then used a shallow (three-layer) CNN to identify the classes. Chandel et al. [31] built a water stress identification system for winter wheat. They classified irrigation conditions as stressed or non-stressed, then trained CNN models based on RGB and thermal images. They tested five state-of-the-art CNN architectures (AlexNet, GoogLeNet, Inception v3, MobileNetV2, and ResNet-50) and five well-known classifiers (logistic, KNN, ANN, SVM, and LSTM), showing that the model using thermal images performed better than using RGB images as input. Melo et al. [32] identified the water stress of sugarcane using transfer learning to train thermal images on an Inception-ResNet-v2 architecture.

In order to clarify the above literature, we list several important factors [27,28,29,30,31,32] in

Table 1. Comparison of literature on water content of plant leaves.

Method	Plants	Data Type of Input	Features and Classifiers	Remark
Taheri-Garavand et al. [27]	Spathiphyllum wallisii	RGB image	Texture features selected by PCA + MLP	Stricter shooting conditions and prior knowledge requirements.
Chandel et al. [28]	Corn, okra, and soybean	RGB image	CNN	RGB image lacks features and ignores degree information.
Zhuanga et al. [29]	Corn	RGB image	CNN features + SVM
Niu et al. [30]	Pomegranate trees	Thermal image	CNN	Too rough classification of water stress and lack of information on degree of consideration
Chandel et al. [31]	Winter wheat	Thermal and RGB image
Melo et al. [32]	Sugarcane	Thermal image

, as well as some remarks.

For a more detailed study on the drought tolerance ability during the development of drought-tolerant varieties, the degree of rice water stress must be classified precisely. In addition, rice depends heavily on water; therefore, if water stress is very high, rice will not survive. Consequently, in our work, 100% and 90% irrigation are regarded as well-watered and slightly stressed for rice, while 80% normal irrigation is considered mildly stressed. In addition, leaf temperature is affected by the amount of irrigation to a different degree. The above studies [30,31,32] either refer to two classes or their target plants do not need such fine classification, so they usually ignored degree information. To pay more attention to degree information and to make the identification of water stress more accurate, a framework was designed to consider the degree information. In addition, leaf temperature is not an absolute value and is affected by the surrounding environment, which is also reflected in CWSI calculations used in many studies on plant water stress. CWSI [33,34] takes the leaf temperature minus the air temperature as normalization, while this study refers to the background as the temperature reference. If acceptable accuracy can be achieved using the background as a reference, then it is not necessary to record the current temperature.

Referring to Table 1, we have sorted out the main contributions of this study:

• We chose thermal images instead of RGB images as the input data and explain its advantages in our subsequent discussion section.
• A CNN was selected as a feature extractor and classifier, which does not require prior knowledge of hand-crafted features. In addition, we used a scoring method to retain degree information.
• We used two background processing methods: early fusion and late fusion which complement each other.
• For other plants or species, a framework is provided for the assessment of the water content of other plant leaves.

This study comprises five sections: Section 2 describes the data collection and their classification methods as well as the methods used in this study; in Section 3, we designed several experiments for comparison to illustrate the effect of the method in this study; Section 4 discusses the proposed method in-depth; and the conclusion and future work are provided in Section 5.

2. Materials and Methods

2.1. Dataset

The experiments were conducted in central Taiwan rice fields, mainly for three rice varieties, Tainan 11, HVA1 gene transgenic rice, and TNG67. Referring to the irrigation amount setting in [7], the irrigation was classified as W1–W6, corresponding to 100–50% of normal irrigation. In addition, the grain yield decreased significantly after W4 (i.e., 70% irrigation). The goal of this study was to detect when water shortage is not serious; therefore, our experiment targets the relatively light water stress situations of W1–W3 (i.e., 100–80% irrigation). Well-watered, slight stress, and mild stress conditions were used to represent 100%, 90%, and 80% of the normal irrigation, respectively, and the data for each class are shown in

Table 2. Dataset for our experiments.

Water Stress Situation	Number of Data
Well-watered	136
Slight stress	137
Mild stress	130
Total	403

.

We conducted experiments on shooting from March 2020 to October 2020, where the planting distance of each plant was approximately 15 to 20 cm. An AVIO G100EX/R300 infrared thermal imaging camera (AVIO, Turin, Italy) was used to capture the data in Table 2. For each picture, the thermal image (8–14 µm) and the RGB image (400–700 nm) were captured at the same time, as shown in Figure 1 (taking HVA1 as an example). The upper and lower rows are the RGB image and the thermal image, respectively, with 100% irrigation to 80% irrigation from left to right. At the far right, the temperature is represented by color, with blue to red representing low to high temperatures, respectively.

From Figure 1, it is difficult to distinguish the water stress from the rice RGB image. However, the thermal image in the lower row shows a significant temperature difference (rice fields reflect relatively low temperatures when watering is sufficient), confirming that the thermal image reflects the water irrigation situation.

2.2. An Overview of the Proposed Framework

The proposed method is shown in Figure 2. Since the background was used as a reference temperature, different background processing methods have their own advantages and disadvantages. Therefore, our architecture consists of two branches that achieve complementary effects. The processing of these two branches is early fusion and late fusion [35] according to the background, with each branch calculating a temperature level score (TLS). For convenience, we use symbols to represent TLS_ef (Temperature Level Score of the Early Fusion Branch) and TLS_lf (Temperature Level Score of the Late Fusion Branch), and then use MLP to identify the final water stress classes according to these scores. On TLS_ef and TLS_lf ground truth scores, well-watered is 1.0 point, slight stress is 0.5 points, and mild stress is 0.0 points. In addition, in the late fusion branch, the foreground is separated from the background, so RGB images are used to segment the thermal image.

2.3. The Early Fusion Branch

Early fusion usually refers to the fusion of data before the network input. In this study, leaf temperature and background temperature were considered as two types of data. Therefore, in this branch, we directly input each thermal image into the CNN, which can be regarded as a type of early fusion, and the leaf temperature and the background temperature are combined in the early stage. This branch plays a role when background pixels are much smaller than leaf pixels. When there are too few background pixels, the background branch of the later fusion will not obtain sufficient background temperature; therefore, this branch compensates for shortcomings in the late fusion branch.

2.4. The Late Fusion Branch

2.4.1. Foreground and Background Segmentation

In order to separate the background and the foreground, we use a simple method to separate the two. First, since rice leaves are green, we use Equation (1) to convert the RGB image to $I^{\prime }$. $I^{\prime }$ mainly converts stronger green pixels. Then we use the Otsu thresholding algorithm [36] on $I^{\prime }$ to get a mask, M. Finally, we use M to cover the thermal image. The segmentation process is shown in Figure 3 and several examples are shown in Figure 4

$$I^{\prime }\left(x,y\right)=2G\left(x,y\right)-\left(R\left(x,y\right)+B\left(x,y\right)\right),$$

(1)

In Equation (1), $G\left(x,y\right)$, $R\left(x,y\right)$, and $B\left(x,y\right)$ are the intensity values of the three channels of each pixel of the RGB image at the image coordinate $\left(x,y\right)$, while $I^{\prime }\left(x,y\right)$ represents the converted intensity value at the image coordinate $\left(x,y\right)$. In addition, when $G\left(x,y\right)\gg R\left(x,y\right)$ and $G\left(x,y\right)\gg B\left(x,y\right)$, $I^{\prime }\left(x,y\right)$ will have a greater intensity; therefore, the pixel in $\left(x,y\right)$ is likely to be a leaf. When $G\left(x,y\right)<\left(R\left(x,y\right)+B\left(x,y\right)\right)$, the pixel is probably not a leaf, therefore, we regard $I^{\prime }\left(x,y\right)$ as 0.

We roughly cut out the foreground and background. Although the cutting is not perfect, we believe it is enough for the application in this study. In order to explain why the cutting results are sufficient for our use, as shown in

Table 3. Comparison of the accuracy of the three CNN architectures.

CNN Architecture	Early Fusion with Softmax	Late Fusion with Softmax	MLP with TLSef + TLSlf
VGG16	36.59%	75.61%	100.0%
ResNet34	51.22%	73.17%	87.80%
DenseNet	34.15%	75.61%	85.37%

, compared with the classification results in Section 2.3, the classification ability of the late fusion branch is much higher. We speculate that since the objects we identify are thermal images, if the edges contain errors, they might be weakened by the pooling operation in the CNN. If some leaves are missed, since the temperature difference between the leaves in the whole rice field is usually not large, such slight omissions do not affect the result.

2.4.2. The Architecture of the Late Fusion Branch

The late fusion branch combines different types of data after extracting features from the network. In this branch, the background and the foreground (leaves) are input into their respective CNN models to concatenate two fully connected layers together. This is a typical late fusion method, as shown in Figure 5.

The leaf temperature and background temperature are independent information, so it is more appropriate to extract leaf temperature and background temperature features separately. This also explains the results presented in Table 3.

2.5. The Combination of TLS_ef and TLS_lf

After obtaining TLS_ef and TLS_lf through two CNNs, their scores must be combined to determine the final irrigation classification. Simple MLP is used for this and its architecture is shown in Figure 6. This MLP has three hidden layers, with 1024, 512, and 256 neurons in each layer. The activation functions of these three layers are all ReLU. The output layer contains three neurons representing three water stress conditions and uses softmax as the activation function.

2.6. CNN Training Details

In the CNN above, we experimented with three architectures: the architecture VGG16 is simply connected to the next layer, while ResNet and DenseNet are designed to be connected across layers to alleviate the gradient problem. Each CNN was trained for 300 epochs, and SGD was used to learn the network parameters with a learning rate of 0.001. The MLP was trained for 1000 epochs, while the rest of the settings are the same as the CNN settings.

3. Experimental Results

The results of the three well-known CNN architectures, VGG16, ResNet34, and DenseNet121, are in the second column. The last layer of the early fusion branch model in Section 2.3 was changed into three output neurons and the softmax function was used for classification. This approach is similar to [30,31,32], with some minor differences such as the different CNN architectures. The third column is the result of changing the last layer to three output neurons and using the softmax function in the model in Section 2.4. The fourth column is the method proposed in this study, as shown in Figure 2. Under the three CNN architectures, the method of separating the background and the foreground (the third column of Table 3) is significantly better than in the second column of Table 3, which shows that the late fusion of the background and the foreground significantly improves identification. The results in the fourth column are much higher than those in the previous two columns and are a significant improvement on the results of the late fusion method, which shows that the early fusion branch compensates for the deficiencies of the late fusion method.

To further examine these results, we calculated precision, recall, and F1-score. The calculation of these three evaluation metrics are shown in Equations (2)–(4):

$$Precision=\frac{TP}{TP+FP},$$

(2)

$$Recall=\frac{TP}{TP+FN},$$

(3)

$$F1-score=2\frac{Precision\times Recall}{Precision+Recall}.$$

(4)

TP is true positive, which means that the rice field is facing water stress and the model correctly predicted water stress. FP is false positive, which means that the rice field is well-watered but the model wrongly identified water stress. FN is false negative, which means that the rice field is facing water stress but the model wrongly predicted that it is well-watered. TN is true negative, which means that the rice field is well-watered and the model correctly predicted that it is well-watered.

The precision in

Table 4. Precision, recall and F1-score under the three CNN architectures.

CNN Architecture	Precision	Recall	F1-Score
VGG16	1.0000	1.0000	1.0000
ResNet34	0.9200	1.0000	0.9019
DenseNet121	1.0000	0.9231	0.8695

is excellent, indicating that the proposed model performs well for the misjudgment of water stress. The recall is also relatively good, indicating that the system has a good ability to detect water stress.

The confusion matrix of the three CNN architectures was drawn to describe detailed identification results, as shown in Figure 7. Most of the misjudgments were between slight stress and mild stress for ResNet34 and DenseNet121, thereby requiring further enhancements.

In addition, VGG16 performed better than the other two architectures. Since VGG16 does not have a cross-layer connection structure in the network, it makes the temperature information less mixed when it is in the convolutional layer, also indicating that the temperature data is more suitable for late fusion processing.

4. Discussion

The softmax feature was also used for classification to explore whether the degree information features considered in this study are useful. The softmax feature is the model output in the second and third columns of Table 3. Using these two results, we use the same MLP as in Section 2.5 and the same setting for the classification task. The accuracy, shown in

Table 5. Accuracy with softmax feature under the three CNN architectures.

CNN Architecture	VGG16	ResNet34	DenseNet121
Accuracy	95.12%	82.93%	75.61%

and compared with the results of the proposed model in the last column in Table 3, is less accurate under the three CNN architectures.

We also conducted an experiment to illustrate the necessity of MLP. By directly using the two scores of TLS_ef and TLS_lf for water stress classification, we combined these two scores in a simple statistical way, using the mean, maximum, and minimum of the two scores. For these three scores, the following rules were used to identify the degree of water stress: if the score was within 1–0.667 we considered it well-watered; within 0.667–0.333 we considered it mildly stressed; and within 0.333–0 we considered it slightly stressed. The results are shown in

Table 6. Result of accuracy of three statistical values.

CNN Architecture	Mean of TLSef and TLSlf	Maximum of TLSef and TLSlf	Minimum of TLSef and TLSlf
VGG16	80.48%	78.05%	92.68%
ResNet34	65.85%	82.93%	43.90%
DenseNet121	68.29%	68.54%	60.98%

. We reviewed the MLP performance in the last column of Table 3, which outperformed the results in Table 6, demonstrating that the MLP can effectively fuse the two scores in this application. Furthermore, it mostly outperformed the CNN with the softmax output, and the combination also outperformed the softmax feature-based in Table 5. This shows that our method retains more degree information.

In order to clearly compare Table 5 and Table 6 and the results of the method in this study (mean of TLS_ef and TLS_lf), we drew them in Figure 8. Through this figure, it can be clearly seen that the mean of TLS_ef and TLS_lf is superior to the others under the three CNN architectures.

Next, we explored several single cases to better understand the advantages and disadvantages of the proposed method and provide directions for future work. First, let us discuss a few cases where identification was correct. The TLS_ef and TLS_lf complemented each other’s deficiencies through the use of two scores. Incidentally, although we only selected the correct cases due to the addition of TLS_ef in the

Table 7. Correct case sample TLS_ef and TLS_lf for 3 CNN architectures.

Case Number	CNN Architecture	TLSef	TLSlf	Ground Truth
Case1	VGG16	0.94414753	0.41299862	Well-watered
Case2	VGG16	0.49225223	0.93656325	Slight stress
Case3	VGG16	0.51609260	0.99480104	Slight stress
Case4	VGG16	0.00144600	0.95988095	Mild stress
Case5	ResNet34	0.81816900	0.28788370	Well-watered
Case6	ResNet34	0.74497569	0.48903641	Well-watered
Case7	ResNet34	0.48903641	0.22066300	Slight stress
Case8	ResNet34	0.45508286	0.28079394	Slight stress
Case9	DenseNet121	0.97530377	0.14977294	Well-watered
Case10	DenseNet121	0.48703855	0.29539412	Slight stress
Case11	DenseNet121	0.54188645	0.18642481	Slight stress
Case12	DenseNet121	0.16764741	0.44175833	Mild stress

, there are also many clearly classified cases. For the convenience of comparison, we reraised the corresponding scores again, in ground truth, where well-watered was 1.0 point, slight stress was 0.5 points, and mild stress was 0.0 points.

From Table 7, we can see the cases 1, 5, 6, and 9 all have unsatisfactory TLS_lf, and all rely on the TLS_ef score being closer to well-watered (1.0 point) to identify the case to correct. In cases 2, 3, 7, 8, 10, 11, and also cases 4 and 12, the ground truths were slightly stressed and mildly stressed, respectively, also to be corrected by TLS_ef.

Then we listed several cases of wrong identification (

Table 8. Error case samples of TLS_ef and TLS_lf for the three CNN architectures.

CNN Architecture	TC	TS	Ground Truth	Misjudgment
ResNet34	0.00104090	0.14355020	Slight stress	Mild stress
ResNet34	0.54493946	0.48646522	Mild stress	Slight stress
DenseNet121	0.99440467	0.99999380	Slight stress	Well-watered
DenseNet121	0.78365338	0.99712139	Mild stress	Slight stress
DenseNet121	0.67452341	0.99999666	Mild stress	Slight stress

) and found that when a wrong prediction occurred, their two scores were far from the correct result. This result indicates that to make the system more accurate, we need to start with two scoring branches.

In the MLP, we expected that the output layer uses softmax, while all other layers use ReLU. We also tested whether this setting is reasonable through our experiments. In

Table 9. Accuracy of replacing MLP activation functions.

CNN Architecture	Tanh	Sigmoid
VGG16	100.00%	100.00%
ResNet34	87.80%	80.49%
DenseNet121	82.92%	85.37%

, we replaced all the activation functions of the MLP with tanh and sigmoid, except for the output layer. Since the values were all positive after ReLU conversion, and since the two scores (TLS_ef and TLS_lf) were all positive, we predicted that ReLU should have a better performance than tanh and sigmoid.However, there was no significant difference in the results of the replaced activation functions in Table 9, which shows that our MLP’s number of layers and neuron settings provide enough capacity for this application.

Compared with previous research, the method of this study mainly lies in the improvement of two points. First, we used two scores, TLS_ef and TLS_lf, to retain degree information, explained by comparisons between Table 3, Table 5, and Table 6. Second, we demonstrated through experiments that using thermal images is better than using RGB images. As shown in

Table 10. Accuracy using RGB images.

CNN Architecture	VGG16	ResNet34	DenseNet121
Accuracy	48.78%	56.10%	46.34%

, we used the same architecture as a MLP with TLS_ef and TLS_lf in Table 3, and using RGB images as the input. These results show that it is difficult to directly use RGB images to identify the water content of plant leaves. A more suitable and easier solution is to use the response of water to leaf temperature and obtain temperature information through thermal imagery.

5. Conclusions

This study proposed an identification system of water stress in rice fields. Leaf temperature is an important reference index for identifying water stress in rice and can be obtained relatively simply and conveniently using a handheld thermal imaging camera. The proposed automatic rice water stress identification system was developed for thermal images based on a CNN and achieved an accuracy of 85% to 100%, thus meeting the needs of practical applications. In the past, crop temperature-based water identification systems usually ignored degree information which can provide useful features in thermal images. We proposed a framework to incorporate degree information into the system, demonstrating through experiments that our method utilizes this information effectively. Since leaf temperature is relative, background temperature was used as a reference temperature, with two background processing methods used to obtain two temperature degree scores which complemented each other to achieve better results.

In the future, the proposed system will be improved with better leaf temperature degree score extraction. Furthermore, since a MLP was used in the final score combination, other simpler methods can be considered to improve the computing speed. After further improvement, we believe that this framework may also be applied to other crops. Finally, for future field applications, we suggest replacing the handheld thermal imager with an external thermal imager connected to a mobile phone, thereby allowing for upload to the GPU host for real-time identification. This configuration is low-cost and easy to operate, making it accessible to farmers, scientists, and students.