Multiscale Superpixel-Based Fine Classification of Crops in the UAV-Based Hyperspectral Imagery

Abstract

As an effective approach to obtaining agricultural information, the remote sensing technique has been applied in the classification of crop types. The unmanned aerial vehicle (UAV)-based hyperspectral sensors provide imagery with high spatial and high spectral resolutions. Moreover, the detailed spatial information, as well as abundant spectral properties of UAV-based hyperspectral imagery, opens a new avenue to the fine classification of crops. In this manuscript, multiscale superpixel-based approaches are proposed for the fine identification of crops in the UAV-based hyperspectral imagery. Specifically, the multiscale superpixel segmentation is performed and a series of superpixel maps can be obtained. Then, the multiscale information is integrated into image classification by two strategies, namely pre-processing and post-processing. For the pre-processing strategy, the superpixel is regarded as the minimum unit for image classification, whose feature is obtained by using the average of spectral values of pixels within it. At each scale, the classification is performed on the basis of the superpixel. Then, the multiscale classification results are combined to generate the final map. For the post-processing strategy, the pixel-wise classification is implemented to obtain the label and posterior probabilities of each pixel. Subsequently, the superpixel-based voting is conducted at each scale, and these obtained voting results are fused to generate the multiscale voting result. To evaluate the effectiveness of the proposed approaches, three open-sourced hyperspectral UAV-based datasets are employed in the experiments. Meanwhile, seven training sets with different numbers of labeled samples and two classifiers are taken into account for further analysis. The results demonstrate that the multiscale superpixel-based approaches outperform the single-scale approaches. Meanwhile, the post-processing strategy is superior to the pre-processing strategy in terms of higher classification accuracies in all the datasets.

1. Introduction

Agriculture is the foundation of the national economy, and crop production affects the quality of human life. Specifically, obtaining the spatial distribution and growing status of crops is crucial for agricultural monitoring and policy development [1,2]. However, the traditional field measurement, investigation and statistic methods are time-consuming and labor-intensive, making it difficult to obtain the agricultural information of a large area in the required time [3,4].

Thanks to the development of earth observation technology, the remote sensing approach has been widely applied in agriculture for years, as it is able to achieve a large area of farmland with higher data collection frequency and lower costs [5,6,7]. In this context, the researchers pay attention to the interpretation of remote sensing images for achieving crop information. Moreover, state-of-the-art machine learning methods are utilized and evaluated for the agricultural crop classification in remote sensing images. Ok et al. [8] analyzed the performance of the random forest and maximum likelihood classification methods for crop recognition with multispectral SPOT 5 images. Zhao et al. [9] compared three deep learning models for early crop classification on Sentinel-1A imagery. Piedelobo et al. [10] explored the high-resolution crop mapping in big areas by fusing the open-source remote sensing data from Sentinel-2 and Landsat-8 satellites. Moreover, Chakhar et al. [11] combined Landsat-8 and Sentinel-2 information for irrigated crop classification and assessed the performance of 22 nonparametric algorithms for classifying crops. Kussul et al. [12] proposed a multilevel deep learning architecture for crop type classification based on the images acquired by Landsat-8 and Sentinel-1A RS satellites. Sonobe et al. [13] used the data provided by Sentinel-1A C-SAR and Sentinel-2A MultiSpectral Instrument for the identification of six crop types. Kumar et al. [14] viewed the Resourcesat-2 as a highly suitable satellite for crop classification studies owing to its improved features and capabilities and compared the classification performance given by several algorithms performed on this imagery.

In recent years, the unmanned aerial vehicle (UAV) opens a new avenue to precision agriculture owing to its flexibility and intelligence [15,16]. To extract the crop distribution from UAV-based imagery, a lot of research has been conducted. Senthilnath et al. [17] investigated the application of a UAV imaging platform for vegetation analysis based on spectral-spatial methods. In this study, vertical take-off and landing (VTOL) quadcopters and fixed-wing were used to acquire images for vegetation analysis, and experiments illustrated the effectiveness of the spectral-spatial methods. Ye et al. [18] used a UAV equipped with a five-band multi-spectral sensor to capture imagery for the identification of banana fusarium wilt using supervised classification algorithms. Moreover, the UAV-based hyperspectral imagery shows abundant spectral properties as well as detailed spatial information [19,20], making it a satisfactory data source for the accurate recognition of crops [21,22]. A survey focused on the combination of UAV and hyperspectral sensors was proposed, in which the hyperspectral sensors, inherent data processing and applications focusing both on agriculture and forest were investigated [23]. Ishida et al. [24] used a liquid crystal tunable filter to select the optimal combination of spectral bands for vegetation classification. Wei et al. [25] proposed a spectral-spatial-location fusion method based on conditional random fields, in which the spectral information, the spatial context, the spatial features, and the spatial location information were integrated into the conditional random field for crop recognition. Zhong et al. [26] built a UAV-borne hyperspectral dataset with high spectral and spatial resolution and proposed a deep convolutional neural network with a conditional random field for precise crop identification.

Meanwhile, the concept of superpixel has been introduced in hyperspectral image interpretation. A superpixel can be regarded as a region consisting of several spatial-coherent pixels with similar properties. It is able to avoid outliers and preserve the boundary of objects in the image [27,28]. The researchers viewed the superpixel as the minimum processing unit for image classification. Li et al. [29] developed a superpixel-level sparse representation classification framework with multitask learning for hyperspectral imagery. Fang et al. [30] proposed a superpixel-based discriminative sparse model for spectral-spatial classification of hyperspectral images, where pixels within each superpixel were represented via a joint sparse regularization and the label of superpixel was determined by the recovered sparse coefficients. Cui et al. [31] proposed a hyperspectral image classification method on the basis of superpixel and multi-classifier fusion, which made use of the spectral information of superpixels and the spatial information of hyperspectral images. Li et al. [32] combined the probability outputs of the pixel-level and superpixel-level classification in a maximum a posteriori estimation model. The aforementioned research illustrated the superiority of superpixel-based approaches compared to the conventional pixel-based ones. However, few works have paid attention to the superpixel-based crop fine classification. Meanwhile, the performance of superpixel-based approaches relys on the segmentation result, and it is difficult to select optimal parameters for the description of different kinds of objects in the agricultural hyperspectral image.

In this manuscript, multi-scale superpixel-based approaches are developed for the fine classification of crops in UAV-based hyperspectral imagery. On the basis of the spectral similarity and spatial relationship among pixels, the image is segmented as a series of superpixels. To exploit the multiscale information of remote sensing image, several segmentation maps with different numbers of superpixels are generated. The superpixel information can be introduced in classification by two different approaches, namely the pre-processing method and the post-processing method. Specifically, the pre-processing method regards each superpixel as a minimum processing unit instead of a pixel, and the post-processing method combines the superpixel segmentation maps with pixel-wise classification results by using a voting strategy. For the pre-processing method, the classification is performed on superpixels, and the feature of a superpixel is calculated based on the pixels located in it. For each scale, both the crisp and soft classification outputs can be obtained. Therefore, label-based and probability-based approaches are proposed to fuse the multiscale information. For the post-processing method, pixel-wise classification is first performed to obtain the label and probability information of each pixel. Based on the superpixel segmentation map at each scale, label-based and probability-based voting can be implemented. Then, the multiscale information is fused by combining the voting results obtained at different scales. To test the effectiveness of the proposed method, three UAV hyperspectral images obtained by UAV over agricultural areas are adopted in the experiments.

The rest of this paper is organized as follows: The multiscale superpixel-based classification approaches are introduced in Section 2. Section 3 and Section 4 show the experimental results and discussions. The conclusions are drawn in Section 5.

2. Methodology

2.1. Superpixel Segmentation

To obtain the superpixel segmentation result, the entropy rate superpixel (ERS) algorithm is employed in this research. For the ERS algorithm, superpixel segmentation is regarded as a clustering problem. An image can be as mapped as an undirected graph $G=\left(V,E\right)$, where $V$ is the vertex set and $E$ is the edge set. The vertices denote the pixels in the image, and the edge weights denote the similarity between vertices given in the form of a similarity matrix. To segment an image into $K$ superpixels, we search for a subset of edges $A\subseteq E$ that makes the resulting graph $G=\left(V,A\right)$ contain $K$ connected subgraphs. The superpixel segmentation result can be obtained via optimizing the following objective function with respect to the edge set

$$\begin{array}{l}\underset{A}{\max }H(A)+\lambda B(A)\\ \mathrm{subject}\mathrm{to}A\subseteq E\end{array}$$

(1)

where $H(A)$ represents the entropy rate of a random walk on graph $G=\left(V,A\right)$ and $B\left(A\right)$ denotes the balancing function. $\lambda \ge 0$ is the weight of the balancing term. Specifically, the entropy rate of the random walk is employed as a criterion to achieve compact and homogeneous clusters, which encourages the division of images on perceptual boundaries and favors superpixels overlapping with only one object. While the balancing function is used to encourage clusters with similar sizes and reduce the number of unbalanced superpixels. By combining the entropy rate and the balancing function, the objective function favors compact, homogeneous, and balanced clusters.

Note that, since the inclusion of any edge will increase the uncertainty of a jump of the random walk, the entropy rate is monotonically increasing. On the other hand, the balancing function is also a monotonically increasing and submodular function under the given graph construction. Therefore, as a linear combination with non-negative coefficients, the objective function is submodular and monotonically increasing that can be optimized by a Greedy algorithm. Starting with an empty edge set, the algorithm adds edges to the set sequentially. At each iteration, the edges that yield the largest gain in the objective function are selected. With the update of the edge set, the number of connected graphs is also changed. When the number of connected graphs reaches the preset K, the iterations are stopped and the superpixel segmentation result is achieved. More detailed information about the objective function and the resulting algorithm of ERS segmentation can be found in [33].

2.2. Superpixel-Based Classification

A superpixel is composed of several spatial adjacent pixels with similar spectral properties that should be assigned to the same label. Thus, the classification can be performed on the basis of the superpixel instead of the original pixels, where the superpixel is used as a minimum processing unit to avoid the salt-and-pepper noises and preserve the boundary of objects. Specifically, to describe the properties of each superpixel, we take the average value of spectral responses of pixels within it into account. The spectral properties of the superpixel are input into the pre-trained classifier to predict its label information, and then a classification map can be generated.

On the other hand, inspired by the object-based voting strategy [34], the superpixel-based voting strategy is also developed. Firstly, a pixel-wise classification is conducted on the original image to obtain a classification map. Then, the superpixel segmentation result is introduced to improve the classification performance. For the superpixel-based voting strategy, the pixels located in the same superpixel should be assigned to the same class of the superpixel. In this work, we use two methods to obtain the label of superpixels. Specifically, the label of a superpixel can be determined using the dominated class of pixels within it, where the most frequently occurred class is used as the label of the superpixel. On the other hand, the probability outputs can also be used to determine the label of the superpixel. The posterior probabilities of pixels belonging to different classes are given by the classifier. For a superpixel, its class-specific probabilities are calculated using the mean probabilities of the pixels within it. Then, the probabilities of the pixel located in the superpixel are modified and can be described as

$$\overline{p}(x)=\frac{1}{N_{sp}}{\displaystyle \sum _{x\in sp}p(x)}$$

(2)

where $p(x)$ is the probability output of pixel $x$ given by the pixel-wise classification, $sp$ is the superpixel that pixel $x$ belongs to, and $N_{sp}$ is the number of pixels contained in the superpixel $sp$. Then, the label of the pixel is assigned to the class with the highest probability. Therefore, the pixel-wise classification map is refined by tuning the label of pixels according to the superpixel-voting strategy.

2.3. Multiscale Superpixel-Based Classification

The objects in remote sensing images always show different characteristics in different scales, making it difficult to select the optimal scale to represent different kinds of objects. Hence, the multiscale superpixel-based approaches are proposed for the fine classification of crops in the UAV-based hyperspectral image. Similar to the traditional superpixel-based classification approaches, the proposed approaches can be divided into the pre-processing and post-processing methods. The flow chart of multiscale superpixel-based approaches is showed in Figure 1.

For the pre-processing method, the average value of the spectral feature of the pixels within a superpixel is used to represent its characteristics at each scale. Subsequently, a classifier is used to give the posterior probabilities and the label output of superpixels. Moreover, each pixel obtains the label and the probability information according to the superpixel it belongs to. Therefore, two approaches are developed to mine the multiscale information of hyperspectral images according to the label and probability outputs. Specifically, the multiscale label fusion (MLF) approach regards the dominant label of a pixel obtained at a different scale as its final class label. Moreover, the multiscale probability fusion (MPF) combines the posterior probabilities obtained at a series of scales to generate a probability output, which can be described as

$$p_{MPF}(x)=\frac{1}{S}{\displaystyle \sum _{s=1}^S{p}_s(\overline{x})}$$

(3)

where $\overline{x}$ is the average value of feature of the pixels within the superpixel that pixel $x$ is located in, $p_s(\overline{x})$ is the corresponding probability output obtained at $s$-th scale, $S$ is the number of scales, and $p_{MPF}(x)$ is the fused probability output of pixel $x$. For a pixel, the class with the highest fused probability is assigned as its label.

For the post-processing method, the pixel-wise classification is performed on the original hyperspectral image to obtain the label and probabilities of each pixel. Based on the superpixel segmentation maps with different scales, superpixel-based voting is conducted and a series of voting results are generated. Moreover, the label-based and probability-based voting strategies are developed in this work. In the multiscale label voting (MLV) strategy, the superpixel-based voting result is calculated using the label information of pixels at each scale. Then, the voting results generated at a series of scales are fused, where the class occurred with the highest frequency is selected as the final label of pixel. Meanwhile, in the multiscale probability voting (MPV) strategy, the initial pixel-wise posterior probabilities are modified using probability voting on the basis of the superpixel segmentation map at each scale. Then, for a pixel, the average of the fused probabilities obtained at several scales is leveraged to represent its membership belonging to a different class.

$$p_{MPV}(x)=\frac{1}{S}{\displaystyle \sum _{s=1}^S{\overline{p}}_s(x)}$$

(4)

where ${\overline{p}}_s(x)$ is the modified output obtained by probability voting at $s$-th scale. The final label of pixel is assigned as the class with the highest multiscale fused probabilities.

3. Experiments and Discussion

3.1. Dataset

In the experiments, the Wuhan UAV-borne hyperspectral image (WHU-Hi) dataset was employed to test the effectiveness of the multiscale superpixel-based methods for the fine classification of crops, which was collected and shared by the Intelligent Data Extraction, Analysis and Applications of Remote Sensing (RSIDEA) research group of Wuhan University. For the WHU-Hi dataset, the preprocessing, including radiometric calibration and geometric correction, was conducted with the HyperSpec software provided by the instrument manufacturer. In the radiometric calibration, the raw digital number values were converted into radiance values with the laboratory calibration parameters of the sensor. Specifically, this dataset was acquired over farming areas in Hubei province, China, by a Headwall Nano-Hyperspec sensor equipped on a UAV platform, including three individual UAV-borne hyperspectral datasets, namely LongKou, HanChuan, and HongHu datasets. An overview of these datasets is provided in Figure 2, Figure 3 and Figure 4 and

Table 1. Ground truth classes for the LongKou dataset and the corresponding sample number.

No.	Class Name	Samples
C1	Corn	34,511
C2	Cotton	8374
C3	Sesame	3031
C4	Broad-leaf soybean	63,212
C5	Narrow-leaf soybean	4151
C6	Rice	11,854
C7	Water	67,056
C8	Roads and houses	7124
C9	Mixed weed	5229

,

Table 2. Ground truth classes for the HanChuan dataset and the corresponding sample number.

No.	Class Name	Samples
C1	Strawberry	44,735
C2	Cowpea	22,753
C3	Soybean	10,287
C4	Sorghum	5353
C5	Water spinach	1200
C6	Watermelon	4533
C7	Greens	5903
C8	Trees	17,978
C9	Grass	9469
C10	Red roof	10,516
C11	Gray roof	16,911
C12	Plastic	3679
C13	Bare soil	9116
C14	Road	18,560
C15	Bright object	1136
C16	Water	75,401

and

Table 3. Ground truth classes for the Honghu dataset and the corresponding sample number.

No.	Class Name	Samples
C1	Red roof	14,041
C2	Road	3512
C3	Bare soil	21,821
C4	Cotton	163,285
C5	Cotton firewood	6218
C6	Rape	44,557
C7	Chinese cabbage	24,103
C8	Pakchoi	4054
C9	Cabbage	10,819
C10	Tuber mustard	12,394
C11	Brassica parachinensis	11,015
C12	Brassica chinensis	8954
C13	Small Brassica chinensis	22,507
C14	Lactuca sativa	7356
C15	Celtuce	1002
C16	Film covered lettuce	7262
C17	Romaine lettuce	3010
C18	Carrot	3217
C19	White radish	8712
C20	Garlic sprout	3486

.

The LongKou dataset was acquired over a simple agricultural scene in Longkou Town, Hubei province, China. The size of the image is 550 × 400 pixels, with 270 bands from 400 to 1000 nm. The UAV flew at an altitude of 500 m, and the spatial resolution of the image is about 0.463 m. The Hanchuan dataset was acquired over a rural-urban fringe zone in Hanchuan City, Hubei province, China. The size of the image is 1217 × 303 pixels, with 274 bands from 400 to 1000 nm. The UAV flew at an altitude of 250 m, and the spatial resolution of the image is about 0.109 m. The Hanchuan dataset was acquired over a complex agricultural scene in Hanchuan City, Hubei province, China. The size of the image is 940 × 475 pixels, with 270 bands from 400 to 1000 nm. The UAV flew at an altitude of 100 m, and the spatial resolution of the image is about 0.043 m.

3.2. Experimental Setup

To test the performance of the proposed multiscale superpixel-based classification approaches, two classifiers, namely the support vector machine (SVM) and random forest (RF), are utilized. For SVM, the radial basis kernel is used, and 5-fold cross-validation is employed to select the optimal value of bandwidth and penalty factor. For RF, the number of trees is set to 500. Meanwhile, the superpixel-based classification with a single scale is also taken into consideration. The number of scales is set as 12, and the number of superpixels for each scale is related to the number of pixels contained in the image. In the s-th scale, the number of superpixels is set as $2^{⌊{\log }_{2}N⌋-s}$, where N is the total number of image pixels. The training set is provided by the RSIDEA group, which contains seven sets with 25, 50, 100, 150, 200, 250, and 300 labeled samples per class. All of the experiments are implemented using Matlab/Simulink on a personal computer with Intel(R) Core(TM) i7-8700K CPU and 32 GB RAM.

In this work, overall accuracy (OA), Kappa coefficient and class-specific accuracy are employed to evaluate the performance of different approaches [35]. OA represents the probability that an individual sample will be correctly recognized by a classifier, that is, the number of corrected predicted samples divided by the total number of testing samples. Kappa takes both the omission and commission errors into account, which is a more robust evaluation measurement than OA. The value of Kappa ranges from −1 to 1. Specifically, a value of 0 indicated that the classification is equal to a random classification. A negative number indicates the classification is worse than random, and a value close to 1 indicates that the classification is significantly better than random. For the class-specific accuracy, the F-score is utilized, which can be expressed as

$$F=\frac{2\cdot PA\cdot UA}{PA+UA}$$

(5)

where PA and UA are the producer’s accuracy and user’s accuracy, respectively. The producer’s accuracy represents the probability of reference samples being correctly predicted, and the user’s accuracy indicates the probability that a predicted sample in the classification map actually represents the class on the ground.

3.3. Results

To analyze the superpixel-based classification method, we compare the performance of the single-scale approaches that only use one superpixel segmentation map for the crop classification. The classification results obtained using SVM are shown in Figure 5, Figure 6 and Figure 7 for Longkou, Hanchuan, and Honghu datasets, while the results with RF are presented in Figure 8, Figure 9 and Figure 10 for the three datasets. In each figure, the vertical axis donates the OA obtained by different approaches, and the horizontal axis represents the superpixel segmentation scale. Meanwhile, with the increase in scale, the number of superpixels included in the segmentation map decreases. From all of the figures, it can be found that the OAs increase steadily at the first few scales, then it reaches the best accuracy and begins to decrease. This phenomenon can be attributed to the fact that, in the first few scales, the image is over-segmented, and one object may be divided into several superpixels. Moreover, the superpixel number is much less in the last few scales, resulting in the under-segmentation phenomenon that different objects may be included in one superpixel. Both the over-segmentation and under-segmentation are unfavorable for the fine classification of crops. Notably, the single-scale superpixel-based approaches also show different performances. The superpixel-based classification approach which views the superpixel as the minimum unit gives an unsatisfactory result. On one hand, its best classification accuracy is lower than that of the other approaches. On the other hand, the OA curve decreases rapidly with the increase in scale. Especially on a larger scale, its OA is even lower than the pixel-wise classification. For this reason, the number of superpixels on a larger scale is small, and a superpixel may contain lots of objects belonging to different crop types. The inclusion of different kinds of objects distorts the average spectral feature of the superpixel, resulting in the misclassification of the superpixel and unsatisfactory performance. Moreover, the accuracy curves of voting approaches decrease as the scale increases, but the descent rate is much slower. The classification accuracy given by the superpixel-based approaches is better than the original pixel-wise classification since the employment of the superpixel introduces spatial information for the crop classification that avoids the isolated misclassified noises. In particular, the highest accuracies of the single-scale approaches are all given by the post-processing methods. Among the 21 results obtained on 3 datasets with 7 training sets using SVM, the probability-based and label-based voting methods give 16 and 5 best performances, respectively. Specifically, the best results achieved with 25, 50, 100, 150, 200, 250, and 300 training samples per class are 97.77%, 98.56%, 98.45%, 98.53%, 98.45%, 98.68%, and 98.77% in the Longkou dataset, respectively. In the Hanchuan dataset, the highest OAs provided by the single-scale approach are 67.19%, 76.17%, 85.20%, 87.52%, 88.84%, 88.52%, and 88.85% with different training sets, respectively. As for the Honghu dataset, the optimal accuracies are 85.31%, 84.77%, 88.04% 88.81%, 89.92%, 90.68%, and 90.85% obtained with different training sample numbers, respectively. When using RF, 17 of the optimal 21 results with different datasets and training sets are provided by the probability-based voting method, and the rest are given by the label-based voting method. For the Longkou dataset, the best results provided by the single-scale approach are 95.44%, 96.41%, 95.07%, 96.83%, 97.99%, 97.27%, and 97.99% using 25, 50, 100, 150, 200, 250, and 300 training samples per class, respectively. In the Hanchuan dataset, the highest OAs achieved with the seven predefined training sets are 69.32%, 79.22%, 81.06%, 81.76%, 84.48%, 84.51%, and 85.13%, respectively. As for the Honghu dataset, the optimal accuracies obtained are 81.57%, 82.68%, 85.10%, 84.37%, 85.13%, 85.71%, and 87.07% with different numbers of training samples per class, respectively.

In the meantime, the OAs and Kappa coefficients provided by the multiscale superpixel-based approaches, including MLF, MPF, MLV, and MPV, as well as the pixel-wise spectral-based approach based on SVM are reported in

Table 4. OAs (%) and Kappa coefficients obtained by different methods for the Longkou dataset using SVM with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	90.32	0.875	95.42	0.940	96.09	0.949	97.18	0.963	97.54	0.968
50	92.83	0.907	97.15	0.963	97.79	0.971	98.70	0.983	98.96	0.986
100	94.22	0.925	97.44	0.966	97.94	0.973	98.47	0.980	98.52	0.981
150	95.64	0.943	97.75	0.970	98.38	0.979	98.91	0.986	99.03	0.987
200	96.17	0.950	98.47	0.980	98.87	0.985	98.85	0.985	99.07	0.988
250	96.23	0.951	98.43	0.979	98.86	0.985	99.08	0.988	99.21	0.990
300	96.98	0.961	98.68	0.983	98.99	0.987	99.27	0.990	99.33	0.991

,

Table 5. OAs (%) and Kappa coefficients obtained by different methods for the Hanchuan dataset using SVM with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	59.28	0.541	63.06	0.583	64.36	0.597	64.43	0.598	66.60	0.622
50	68.67	0.643	73.30	0.695	75.04	0.715	76.65	0.733	77.09	0.738
100	73.93	0.700	82.91	0.802	83.15	0.805	86.41	0.842	86.21	0.840
150	78.24	0.749	85.59	0.833	86.23	0.840	89.75	0.881	89.65	0.880
200	80.08	0.770	87.06	0.850	87.48	0.855	90.88	0.894	90.67	0.891
250	80.66	0.776	86.64	0.845	87.02	0.849	90.47	0.889	90.68	0.892
300	80.89	0.779	87.77	0.858	87.96	0.860	90.64	0.891	90.79	0.893

and

Table 6. OAs (%) and Kappa coefficients obtained by different methods for the Honghu dataset using SVM with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	66.98	0.605	81.51	0.770	82.56	0.783	83.93	0.799	83.39	0.794
50	64.08	0.581	82.16	0.779	83.35	0.793	81.63	0.774	81.80	0.777
100	73.42	0.681	88.26	0.853	88.74	0.859	88.87	0.861	89.31	0.866
150	74.74	0.695	88.50	0.856	88.90	0.861	89.71	0.871	90.06	0.876
200	77.05	0.721	90.24	0.877	90.65	0.882	90.87	0.885	91.05	0.888
250	77.43	0.726	90.80	0.884	91.28	0.890	91.70	0.896	91.86	0.898
300	79.77	0.752	90.88	0.885	91.06	0.887	92.07	0.900	92.40	0.904

for Longkou, Hanchuan, and Honghu datasets, respectively. The classification results obtained with RF are shown in

Table 7. OAs (%) and Kappa coefficients obtained by different methods for the Longkou dataset using RF with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	79.79	0.746	86.10	0.823	87.17	0.837	91.08	0.886	91.31	0.888
50	84.14	0.799	85.10	0.812	87.14	0.837	92.02	0.898	94.69	0.931
100	86.03	0.823	89.84	0.870	91.60	0.892	93.80	0.920	95.33	0.940
150	89.15	0.861	94.32	0.926	94.54	0.929	97.07	0.962	97.13	0.963
200	90.67	0.880	94.41	0.927	94.85	0.933	97.47	0.967	97.42	0.966
250	89.64	0.867	94.38	0.927	94.77	0.932	96.83	0.959	97.14	0.963
300	90.76	0.881	94.02	0.922	94.74	0.932	97.48	0.967	97.75	0.971

,

Table 8. OAs (%) and Kappa coefficients obtained by different methods for the Hanchuan dataset using RF with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	57.69	0.524	62.69	0.578	63.03	0.582	64.55	0.599	64.54	0.599
50	69.38	0.649	71.36	0.671	72.73	0.686	76.98	0.734	78.49	0.751
100	71.46	0.673	74.00	0.702	75.64	0.720	79.62	0.765	81.03	0.781
150	74.61	0.708	78.41	0.751	79.45	0.762	82.72	0.800	83.53	0.809
200	77.19	0.736	80.26	0.771	80.91	0.779	85.96	0.837	86.16	0.839
250	77.24	0.737	79.27	0.760	80.03	0.769	85.44	0.831	85.98	0.837
300	77.15	0.736	79.93	0.768	81.19	0.782	85.23	0.829	86.24	0.840

and

Table 9. OAs (%) and Kappa coefficients obtained by different methods for the Honghu dataset using RF with different numbers of training samples per class.

Training Samples	Spectral		MLF		MPF		MLV		MPV
	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa	OA	Kappa
25	62.79	0.559	76.74	0.713	78.15	0.730	79.50	0.745	78.56	0.736
50	59.99	0.537	77.75	0.727	78.69	0.738	78.57	0.739	78.47	0.739
100	67.28	0.612	80.64	0.760	82.53	0.783	83.63	0.797	84.08	0.803
150	68.60	0.626	81.69	0.773	82.43	0.782	84.47	0.807	84.48	0.808
200	69.96	0.640	82.12	0.778	82.46	0.782	84.75	0.810	85.69	0.822
250	70.44	0.646	83.16	0.791	84.05	0.802	85.32	0.817	86.45	0.832
300	72.08	0.664	83.76	0.798	84.42	0.806	86.60	0.833	87.71	0.847

for the three datasets. Obviously, compared to the aforementioned single-scale approaches, the multiscale approaches show more satisfactory classification performance. The proposed multiscale approaches give similar and even better accuracies to the optimal result achieved by the single-scale approaches. The phenomenon reveals that the employment of multiscale information benefits the recognition of crops, and the multiscale superpixel-based method avoids the optimal scale selection problem in image analysis. Comparing the result obtained with different sample sets, it can be observed that the accuracy is improved as the number of training samples increases, which illustrates that sufficient samples are conducive to the construction of a discriminative classification model. Among the proposed multiscale superpixel-based methods, MLV and MPV show much better results than MLF and MPF in terms of higher OAs and kappa coefficients. This phenomenon is similar to the results of the single-scale approaches, demonstrating that the post-processing strategy is more effective than pre-processing strategy in identifying the crops in the hyperspectral image. It can be also found that MLF and MPF show similar results, and the accuracies given by MLV and MPV are very close. Actually, the OAs and Kappa coefficients of probability-based approaches are slightly higher than that of label-based approaches, meaning the probability-based approaches may be more suitable for crop classification. Moreover, the classification results obtained using SVM are better than that using RF, especially for the pixel-wise spectral-based approach. Although SVM and RF show different distinguishing abilities in recognizing the crops in hyperspectral images, the accuracy improvements achieved by introducing the multiscale superpixel information are still evident. Overall, the proposed multiscale superpixel-based approaches give satisfactory results in the testing datasets, while MPV shows the most promising performance as it obtains the highest accuracies in most cases.

4. Discussion

For further discussion and analysis, we compare the classification performances of different approaches with 100 training samples per class in this subsection.

Table 10. Classification accuracies given by different methods for the Longkou dataset using SVM and RF with 100 training samples per class.

No.	SVM					RF
	Spectral	MLF	MPF	MLV	MPV	Spectral	MLF	MPF	MLV	MPV
C1	97.66	99.51	99.54	99.53	99.55	93.28	99.14	99.48	99.44	99.55
C2	79.16	97.08	97.81	95.96	96.04	68.39	68.17	68.25	97.09	95.37
C3	73.64	97.31	99.00	97.49	97.90	33.07	45.04	56.06	52.03	69.91
C4	92.21	98.18	98.55	98.19	98.10	81.64	88.90	91.50	90.19	92.86
C5	72.65	71.20	75.02	90.04	89.39	51.38	43.89	44.52	63.97	65.75
C6	98.44	98.91	98.88	99.15	99.21	90.44	98.15	98.57	98.85	98.61
C7	99.96	99.96	99.96	99.96	99.96	99.95	99.96	99.96	99.95	99.95
C8	88.15	83.72	85.38	94.37	95.24	84.91	79.55	82.13	91.99	93.48
C9	85.16	84.71	90.89	91.84	93.89	59.24	67.55	77.62	86.38	87.67
OA	94.22	97.44	97.94	98.47	98.52	86.03	89.84	91.60	93.80	95.33
Kappa	0.925	0.966	0.973	0.980	0.981	0.823	0.870	0.892	0.920	0.940

reports the OAs, kappa coefficient, and class-specific accuracies given by SVM and RF for the Longkou dataset. The spectral-based classification accuracy with SVM is 94.22%, which is higher than 86.03% obtained by RF. By introducing the multiscale superpixel information in crop classification, the MLF, MPF, MLV and MPV increase the accuracy by 3.22%, 3.72%, 4.25% and 4.30% with SVM. For RF, the accuracy improvements obtained by the proposed methods are 3.81%, 5.57%, 7.77% and 9.30%, respectively. In this image, the sesame and narrow-leaf soybean achieve the lowest accuracies, where the accuracies given by SVM are 73.64% and 72.65%, while the accuracies given by RF are 33.07% and 51.38%. However, MPV with SVM gives 97.90% and 89.39% for sesame and narrow-leaf soybean, which is a satisfactory result, and MPV with RF achieve 69.91% and 65.75% for these two crops, which is much better than the original spectral-based result. The classification accuracies for Huanchuan datasets obtained with 100 training samples are shown in

Table 11. Classification accuracies given by different methods for the Hanchuan dataset using SVM and RF with 100 training samples per class.

No.	SVM					RF
	Spectral	MLF	MPF	MLV	MPV	Spectral	MLF	MPF	MLV	MPV
C1	78.83	90.95	91.59	92.66	92.09	74.45	81.98	84.30	86.96	87.37
C2	60.19	72.79	73.71	81.51	78.99	44.52	41.14	42.44	55.55	60.12
C3	64.99	78.88	80.58	88.49	88.51	62.62	61.74	61.14	75.57	78.15
C4	87.91	91.43	91.66	91.61	92.67	82.06	92.27	94.86	93.52	95.45
C5	25.04	25.74	28.46	39.83	41.60	18.65	17.92	18.61	22.03	25.00
C6	28.57	51.90	53.21	55.93	55.84	23.28	36.14	42.73	40.11	42.44
C7	64.32	75.81	77.23	73.50	74.34	65.58	75.31	76.28	73.77	74.08
C8	60.59	73.35	75.93	78.77	80.09	61.74	67.70	68.87	71.71	73.01
C9	62.25	75.22	74.87	82.02	82.70	50.31	56.59	56.28	61.84	64.69
C10	85.52	88.78	90.34	92.38	92.16	79.61	85.34	86.64	84.57	86.17
C11	75.98	85.81	86.35	84.79	84.33	78.28	80.12	83.56	83.76	85.03
C12	33.12	55.45	50.31	65.09	58.88	32.32	37.47	38.62	50.22	49.86
C13	41.02	55.47	54.25	56.99	58.51	44.55	47.32	48.05	57.53	59.04
C14	68.35	69.45	68.90	78.00	78.99	66.54	60.62	64.03	69.07	73.31
C15	58.34	72.87	71.78	73.08	77.65	58.56	61.23	64.36	71.61	71.67
C16	95.70	97.96	97.84	98.06	98.08	97.05	96.53	97.46	98.54	98.69
OA	73.93	82.91	83.15	86.41	86.21	71.46	74.00	75.64	79.62	81.03
Kappa	0.700	0.802	0.805	0.842	0.840	0.673	0.702	0.720	0.765	0.781

. The OAs given by MLF, MPF, MLV and MPV with SVM are 82.91%, 83.15%, 86.41%, and 86.21%, while the OA of the spectral-based approach is only 73.97%. As for RF, the spectral-based approach gives 71.46%, while MLF, MPF, MLV and MPV give 74.00%, 75.64%, 79.62%, and 81.03%, respectively. In this dataset, the post-processing methods show much better performance than the pre-processing ones, as the accuracies obtained by MLV and MPV are 3% higher than that obtained by MLF and MPF. For the class-specific accuracy, all of the approaches achieve unsatisfactory results in water spinach, watermelon, plastic and bare soil with accuracies lower than 60%. Especially for the water spinach, the accuracies given by MLF, MPF, MLV and MPV with RF are only 17.92%, 18.61%, 22.03%, and 25.00%. The classification accuracies given by different methods for the Honghu dataset using SVM and RF with 100 training samples per class are presented in

Table 12. Classification accuracies given by different methods for the Honghu dataset using SVM and RF with 100 training samples per class.

No.	SVM					RF
	Spectral	MLF	MPF	MLV	MPV	Spectral	MLF	MPF	MLV	MPV
C1	92.20	96.01	95.90	96.73	97.37	85.09	91.06	90.74	93.78	94.31
C2	70.67	78.46	80.86	74.72	81.43	65.80	67.89	71.36	70.53	77.66
C3	83.69	87.96	87.61	91.34	91.68	80.79	82.25	84.30	90.16	91.08
C4	86.04	97.93	97.91	97.07	97.05	82.06	94.92	96.10	94.25	93.57
C5	34.13	72.91	78.04	67.70	69.78	26.02	50.26	60.04	50.97	47.77
C6	87.15	92.92	93.53	93.69	94.16	81.96	89.03	90.33	90.80	92.48
C7	68.43	77.39	77.86	80.53	81.07	56.80	62.95	61.55	67.51	70.17
C8	24.83	54.03	57.93	59.51	59.15	19.43	33.24	39.23	43.20	45.94
C9	94.55	95.58	95.49	95.87	96.13	92.11	94.73	94.99	95.94	96.15
C10	57.51	84.56	85.29	84.38	85.89	42.71	50.35	52.68	74.86	78.58
C11	41.06	73.91	73.23	73.82	72.99	33.37	52.18	57.80	67.53	64.05
C12	53.33	70.30	71.64	71.53	71.38	51.89	61.94	63.36	64.82	67.93
C13	59.75	70.38	72.71	72.08	74.06	58.60	68.62	72.61	69.54	72.22
C14	70.24	74.01	76.56	77.16	81.97	61.34	69.22	68.60	73.30	76.48
C15	13.66	63.98	63.89	78.17	76.22	13.62	32.29	31.13	80.04	63.29
C16	85.03	94.67	95.18	95.85	96.28	81.64	90.01	92.72	93.85	94.81
C17	70.28	88.99	89.70	93.41	94.33	68.13	78.13	84.79	91.22	93.47
C18	45.13	73.34	76.86	76.17	75.62	33.28	49.41	48.18	51.45	52.33
C19	75.09	89.08	88.87	87.49	88.25	67.27	83.61	83.23	81.09	84.80
C20	58.31	76.98	77.50	82.70	82.63	36.62	56.63	58.68	76.96	75.15
C21	21.25	29.08	29.77	36.58	41.69	20.39	21.20	22.54	32.80	38.83
C22	38.83	64.57	60.78	61.94	57.14	28.63	55.37	54.76	44.54	43.38
OA	73.42	88.26	88.74	88.87	89.31	67.28	80.64	82.53	83.63	84.08
Kappa	0.681	0.853	0.859	0.861	0.866	0.612	0.760	0.783	0.797	0.803

. It can be observed that the best result of the Honghu image is achieved by MPV whether using SVM or RF. Specifically, the MPV with SVM gives an accuracy of 89.31%, which is 5.23% higher than the accuracy of 84.08% given by MPV with RF. As for the pixel-wise spectral-based classification, the classification accuracies obtained with SVM and RF are 73.42% and 67.28%. The comparison between the spectral-based and multiscale superpixel-based approaches illustrates the effectiveness of the employment of spatial information. Moreover, among the 22 classes, 7 classes achieve a satisfactory accuracy that is higher than 90% and only 3 classes’ accuracies are lower than 60% in the result given by MPV with SVM. For visual interpretation, the classification maps obtained with SVM for Longkou, Hanchuan, and Honghu datasets are shown in Figure 11, Figure 12 and Figure 13. Meanwhile, Figure 14, Figure 15 and Figure 16 show the classification maps given by different approaches on the testing dataset using RF. It is observed that the pixel-wise spectral-based classification is subject to the salt-and-pepper misclassification noises. While the superpixel-based approaches show satisfactory results with less salt-and-pepper noises and more accurate object boundaries, which illustrate the superiority of the proposed multiscale superpixel-based methods in identifying the crops in hyperspectral imagery.

In addition, the effectiveness of the multiscale superpixel-based methods relays on the superpixel segmentation results. Obviously, different superpixel segmentation algorithms will result in different performances, and thus, it is important to select a suitable segmentation algorithm to generate the superpixel results. Meanwhile, the number of scales used in the proposed method restricts the final classification performance. A larger scale number always indicates a higher computational burden and time cost, while a smaller scale number cannot comprehensively exploit the spectral-spatial information of images.

5. Conclusions

In this manuscript, multiscale superpixel-based approaches were developed for the fine recognition of crop types in UAV-based hyperspectral images. Superpixel segmentation was performed with different parameters to exploit the multiscale information of objects, and several superpixel maps can be obtained. To fuse the multiscale superpixel information, the pre-processing and post-processing strategies were proposed according to different principles. Specifically, the pre-processing strategy views the superpixel as the minimum image processing unit, and the classification was conducted on the superpixel level at each scale. Then, the label of each pixel was assigned to the domain class among multiscale results. Moreover, the post-processing strategy was inspired by the voting approach, and the class information of the superpixel was determined by the majority classes of pixels within it. By fusing the voting result obtained at different scales, we can obtain the final classification map. Note that, for the pre-processing and post-processing methods, the class probability output and label information were taken into consideration to generate the final classification results by different approaches.

The experiments were conducted on the WHU-Hi dataset provided by the RSIDEA research group, which contains three individual UAV-based hyperspectral images. Moreover, for each dataset, seven training sets with different number of labeled samples were supplied, as well as the hyperspectral image. Meanwhile, SVM and RF were employed to test the effectiveness of the proposed methods. The comparison of the single-scale approaches demonstrates that it is hard to select an optimal scale for a complex image scene. Moreover, the best result among the single-scale superpixel-based approaches was inferior to the multiscale superpixel-based approaches. Furthermore, it is found that the post-processing strategy shows better result than the pre-processing strategy, which illustrates the effectiveness of voting methods. Additionally, the classification maps show that the proposed method is able to preserve the object boundaries while avoiding the discrete misclassification pixels. Future work will focus on the extraction of superpixel-based features for better classification of crops.