Synergy of Sentinel-1 and Sentinel-2 Imagery for Crop Classification Based on DC-CNN

Abstract

Over the years, remote sensing technology has become an important means to obtain accurate agricultural production information, such as crop type distribution, due to its advantages of large coverage and a short observation period. Nowadays, the cooperative use of multi-source remote sensing imagery has become a new development trend in the field of crop classification. In this paper, the polarimetric components of Sentinel-1 (S-1) decomposed by a new model-based decomposition method adapted to dual-polarized SAR data were introduced into crop classification for the first time. Furthermore, a Dual-Channel Convolutional Neural Network (DC-CNN) with feature extraction, feature fusion, and encoder-decoder modules for crop classification based on S-1 and Sentinel-2 (S-2) was constructed. The two branches can learn from each other by sharing parameters so as to effectively integrate the features extracted from multi-source data and obtain a high-precision crop classification map. In the proposed method, firstly, the backscattering components (VV, VH) and polarimetric components (volume scattering, remaining scattering) were obtained from S-1, and the multispectral feature was extracted from S-2. Four candidate combinations of multi-source features were formed with the above features. Following that, the optimal one was found on a trial. Next, the characteristics of optimal combinations were input into the corresponding network branches. In the feature extraction module, the features with strong collaboration ability in multi-source data were learned by parameter sharing, and they were deeply fused in the feature fusion module and encoder-decoder module to obtain more accurate classification results. The experimental results showed that the polarimetric components, which increased the difference between crop categories and reduced the misclassification rate, played an important role in crop classification. Among the four candidate feature combinations, the combination of S-1 and S-2 features had a higher classification accuracy than using a single data source, and the classification accuracy was the highest when two polarimetric components were utilized simultaneously. On the basis of the optimal combination of features, the effectiveness of the proposed method was verified. The classification accuracy of DC-CNN reached 98.40%, with Kappa scoring 0.98 and Macro-F1 scoring 0.98, compared to 2D-CNN (OA reached 94.87%, Kappa scored 0.92, and Macro-F1 scored 0.95), FCN (OA reached 96.27%, Kappa scored 0.94, and Macro-F1 scored 0.96), and SegNet (OA reached 96.90%, Kappa scored 0.95, and Macro-F1 scored 0.97). The results of this study demonstrated that the proposed method had significant potential for crop classification.

1. Introduction

Agriculture is the foundation of the national economy and the basic condition for ensuring social development [1]. It is of great significance to obtain the spatial distribution information of crops in time for phenology monitoring, yield estimation, disaster assessment, soil moisture inversion, and other fields [2,3,4,5]. Remote sensing (RS) technology, as a macro-real-time, large-scale Earth observation technology, has an obvious advantage over traditional manual statistical methods in identifying crop categories [6]. Therefore, it is widely used by scholars in the study of crop classification [7].

Multispectral (MS) images and Synthetic Aperture Radar (SAR) images are two effective RS data sources. MS images obtained by optical sensors contain rich spectral information, which can identify the biochemical characteristics of different crops [8]. In 2017, Sonobe et al. [9] classified crops in Hokkaido, Japan, and demonstrated that the final accuracy could reach 94.5% when using data obtained from the Operational Land Imager of Landsat-8. In [10], the sown area and spatial distribution of the main crops were extracted using MODIS data in Hebei Province, which verified the feasibility of extracting provincial crop planting information. Kenichi et al. [11] used Landsat-7 data to study crop classification in the Iac area, Peru. The difference in spectral characteristics of different crops in MS images is the basis of crop classification. However, MS imagery is weather-dependent and is affected by clouds, rain, and fog, which hinders its performance when classifying crops [12].

SAR, an active microwave sensor without limitations caused by weather conditions due to its penetration capability, can provide different information from MS images for crop classification [13,14]. In [15], rice was identified based on RADARSAT-2 data with the threshold set according to the ratio of HH to VV, and the identification accuracy reached 92.64%. In order to improve the classification accuracy, Xiang et al. [16] combined Sentinel-1 (S-1) data with elevation and slope information to extract ground feature information in the study area. Based on the polarimetric components of different crops in PolSAR data, Guo et al. [17] put forward a new parameter to realize crop classification. As can be seen from the above studies, SAR images can achieve high accuracy in crop classification, but due to the lack of spectral features and sensitivity to surface parameters, the applicability of SAR in crop classification is limited, and it is often combined with other auxiliary data.

As there is a very complicated nonlinear relationship between environmental elements, a single type of RS data cannot fully and accurately reflect the comprehensive information on the ground [18]. The diversity of local, regional, and global agricultural landscapes and their site-specific challenges have been reflected in many studies, including spectral similarity, crop diversity, weather conditions, and farming systems [19,20,21,22]. In recent years, the information provided by multi-source RS data has been complementary and cooperative [23,24]. The collaborative application of multi-source RS can reduce or eliminate the problems of target features, such as ambiguity, incompleteness, and uncertainty [25]. Multi-source RS image fusion can make use of complementary information from different sources to achieve accurate and comprehensive crop classification [21,24,26,27,28,29].

Recently, several studies have used MS and SAR data for crop classification. In particular, S-1 and Sentinel-2 (S-2) have similar spatial resolution, which has made their synergistic use a research hotspot [30,31,32,33,34]. In [35], wheat and oilseed rape were monitored based on the spectral information of optical data and the backscattering coefficient of SAR, and the accuracy of S-1 and S-2 data combined could reach 92%. Sun et al. combined the spectral information in optical data with the backscattering coefficient of SAR data and used a machine learning algorithm to classify crops. The results showed that VV and VH were effective in the classification of wheat and other crops, and the classification results could reach 93% [36]. Ghassemi, B. et al. used optical features from S2 and composites from S1 together to provide a full coverage map with appropriate OA [37]. However, the polarimetric components have not been fully utilized in the task of crop classification. There are many polarimetric decomposition methods for obtaining polarimetric information from SAR images, such as H/A/α decomposition [38,39] and Freeman decomposition [40], but most of them have been proposed for quad-pol SAR sets and are not suitable for S-1 data. On the other hand, with the continuous development of deep learning technology, it is widely used in the field of classification [41,42], and some scholars use deep learning technology to identify crops through multi-source RS images [43,44,45,46]. Kussul et al. classified corn and wheat in Ukraine and found that the classification result of a convolutional neural network (CNN) was better than that of RF [47]. In [48], CNN, RF, SVM, etc. were used to classify soybeans and wheat in an agricultural region of Canada. The results showed that CNN had the best classification result and that the overall accuracy could reach 96.72%. However, its ability to process multi-source RS data was still limited [49]. The insufficient integration of complementary information between multi-source data, for example, was prone to redundant input. Moreover, most studies analyze crop phenology changes based on time-series remote sensing data to achieve classification. For some areas affected by weather conditions, it is difficult to obtain time series images with quality assurance, and the effectiveness of single-phase and multi-source remote sensing data to improve crop classification accuracy needs to be verified.

Regarding these issues, this paper adapted the polarimetric components of S-1 images, which were extracted based on the newly developed model-based decomposition method [50]. Meanwhile, a Dual-Channel CNN (DC-CNN) based on a combination of the features of single-date S-1 and S-2 was constructed by extracting and fusing the multi-source features. On this basis, the best feature combination of the backscattering components, polarimetric components, and MS features was analyzed. The main contributions of this paper are given as follows:

• To properly exploit the polarimetric content of S-1 SAR data in crop classification, the outputs of the new polarimetric decomposition conceived in [50] by Mascolo et al., which is adapted for dual-polarimetric SAR data, are extracted from VH-VV S-1 observations. These, along with the VH and VV backscattering coefficients, are combined with MS features, and the best combination strategy was analyzed.
• A dual-channel CNN model, namely DC-CNN, with shared parameters based on multi-source RS data was constructed. Specifically, the features obtained from S-1 and S-2 data were fed into two CNN channels for independent learning, and they were transformed into high-dimensional feature expressions. Furthermore, the sharing of parameters in the convolution layer made the two branches learn cooperatively. The correlation of multi-source features was maximized while maintaining the unique features of each data source.

The rest of this paper is arranged as follows: In Section 2, the study area and adopted data are described, and the crop classification method is recommended in detail. The results of crop classification are shown in Section 3. In Section 4, the results obtained were discussed and analyzed. Section 5 presents the final conclusion.

2. Materials and Methods

2.1. Study Area

Tongxiang (120°17′E–120°39′E, 30°28′N–30°47′N) is located in the Hangjiahu Plain in the northern Zhejiang Province, China, as shown in Figure 1. The land is flat and fertile, which makes it suitable for the cultivation of wheat, rice, and oilseed rape. Wheat and oilseed rape were sown in late November and harvested in June and July of the following year. Due to the abundant rain in this study area, the availability of optical data is greatly affected. Therefore, this study adopted the method of single-time phase data to achieve high precision crop distribution mapping.

2.2. Data and Preprocessing

2.2.1. Sentinel-1A SAR Data and Preprocessing

Sentinel-1A (S-1A), a SAR satellite, was launched by the European Space Agency (ESA) on 3 April 2014. It is able to operate day and night with a high spatial resolution of 5 m × 20 m in Interferometric Wide (IW) swath mode. The sensor carried by the S-1A can provide large-scale images of 250 km × 250 km.

In this study, the IW-mode Single Look Complex (SLC) S-1A image on 6 May 2021, was downloaded from the Copernicus Open Access Hub (https://scihub.copernicus.eu/). The detailed parameters are shown in

Table 1. S-1A image used in the study.

S-1A Parameters	S-1A
Product type	SLC
Imaging mode	IW
Polarization	VV VH
Pixel size	10 m $\times$ 10 m
Pass direction	Ascending
Wave band	C
Dates	2021-05-06

. The main processes include: (1) Orbit correction. Using accurate satellite orbit data to correct orbit information can effectively remove systematic errors caused by orbit errors. (2) Thermal noise removal. In order to reduce the influence of noise in SAR images, noise removal technology [48] was used. (3) Radiometric calibration. Eliminated all kinds of distortions associated with the radiant brightness in the image data as much as possible. (4) Deburst. Each burst that had effective signal parts was merged. (5) Generating a polarimetric matrix C2 according to the complex band in step (3). (6) Multi-looking. The number of range looks and azimuth looks was 4 and 1, respectively. (7) Speckle filtering. (8) Range-Doppler terrain correction. (9) Data conversion to convert the ${\sigma }_0$ band into the common dB standard. (10) Extraction of the study area. After preprocessing the S-1 data, the backscattering coefficient ${\sigma }_0$ was obtained, and the saved C2 matrix was derived for polarimetric decomposition.

2.2.2. Sentinel-2B Data and Preprocessing

Sentinel-2 (S-2) is a multi-spectral imaging satellite that consists of two satellites, 2A and 2B. Each satellite has a revisit period of 10 days, with a complementary double satellite for a temporal resolution of 5 days.

The study area is located in the south of China, which has a subtropical monsoon climate. Due to the influence of weather, the quality of optical data is seriously affected during the critical period of crop growth (April 2021 to June 2021). The weather conditions counted are shown in Figure 2, and the statistical data was obtained from http://www.weather.com.cn/ and https://weather.cma.cn (accessed on 11 April 2022). It can be shown that there were just 14 days from April to June that were sunny, while other days were cloudy, overcast, or rainy.

S-2 images will be affected by weather factors such as clouds, rain, and fog. In order to select effective S-2 images in the study area, the cloud cover percentage of S-2 images was counted (obtained from https://scihub.copernicus.eu/dhus/#/home (accessed on 11 April 2022)), as shown in

Table 2. Cloud cover data from S-2 images during the critical period of crop growth from April 2021 to June 2021.

S-2 Satellite	Date	Cloud Cover Percentage (%)	S-2 Satellite	Date	Cloud Cover Percentage (%)
A	8 April 2021	99.94	B	3 April 2021	99.89
A	8 April 2021	71.47	B	3 April 2021	99.97
A	18 April 2021	87.58	B	13 April 2021	53.41
A	18 April 2021	56.11	B	13 April 2021	88.98
A	28 April 2021	98.98	B	23 April 2021	98.98
A	28 April 2021	94.74	B	23 April 2021	98.46
A	8 May 2021	99.99	B	3 May 2021	11.42
A	8 May 2021	99.89	B	3 May 2021	19.96
A	18 May 2021	99.77	B	13 May 2021	99.33
A	18 May 2021	99.57	B	13 May 2021	98.58
A	28 May 2021	100	B	23 May 2021	67.47
A	28 May 2021	94.12	B	23 May 2021	88.47
A	7 June 2021	93.43	B	2 June 2021	97.03
A	7 June 2021	96.48	B	2 June 2021	99.29
A	17 June 2021	93.91	B	12 June 2021	83.16
A	17 June 2021	99.64	B	12 June 2021	90.27
A	27 June 2021	97.58	B	22 June 2021	82.89
A	27 June 2021	94.57	B	22 June 2021	9.47

.

As can be seen from Table 2, most S-2 images with a high cloud cover percentage cannot be used for crop classification. Only three S-2B images were suitable for crop classification from 3 April to 27 June 2021. As the study area is large, one scene in the S-2B image cannot completely cover the whole study area. Therefore, S-2B images taken on 3 May 2021, with a low cloud cover percentage were chosen for crop classification.

In this study, cloud-free level 2A atmospheric effect-corrected S-2B data on 3 May 2021, were acquired. S-2 data preprocessing processes included: (1) Resampling. Resampling the resolution of all bands to 10 m. (2) Layer stacking. Combining the resampled multiple bands into an MS image. (3) Mosaicking. Two images were stitched together to obtain an image covering the whole study area. (4) Extraction of the study area. The spectral bands (

Table 3. S-2B image used in the study.

S-2B Parameters	Spatial Resolution (m)	S-2B Spectral Description
Band 2	10	Blue
Band 3	10	Green
Band 4	10	Red
Band 5	20	Vegetation red edge
Band 6	20	Vegetation red edge
Band 7	20	Vegetation red edge
Band 8	10	Near Infrared
Band 8A	20	Vegetation red edge
Band 11	20	Short-Wave Infrared
Band 12	20	Short-Wave Infrared
Dates		2021-05-03
Processing Level		Level 2A

) were extracted for further crop classification.

2.2.3. Ground-Truth Data and Preprocessing

The vector data for farmland were obtained by manual measurement and the statistics of the local government. A reliable farmland boundary is beneficial to the final mapping of crop distribution. A field investigation was conducted from 10 May to 12 May 2021, to collect samples of crops in the study area. The farmland we studied was a government-planned field provided by the local government. During the study period, the planting status of farmland can be completely divided into three categories: oilseed rape, wheat, and bare land. There were also economic crops such as mulberry in Tongxiang city, but they were not involved in our experiment. In order to ensure the uniform distribution and quantity of samples, three experts interpreted and supplemented the samples according to a 3.8-m high-resolution remote sensing image (GF-1 optical image on 1 May 2021) and Google images (Landsat-8 optical image on 29 April 2021). The number of expert interpretation samples accounted for about 10% of the total sample. Finally, 305 plots were obtained as samples (19,540 pixels), including 101 oilseed rape (6038 pixels), 103 wheat (7418 pixels), and 101 bare lands (6084 pixels). The sample distribution is shown in Figure 3.

The samples were randomly divided into three parts: the training set was 60% (11,713 pixels), the validation set was 20% (3906 pixels), and the testing set was 20% (3921 pixels). In particular, the validation set was used to adjust the hyperparameters to prevent overfitting of the model. The testing set did not participate in the training process and was used to evaluate the performance of the model independently. In

Table 4. Ground-truth data and sample allocation of crop types.

Label	Type	Number of Fields	Total Number of Pixels	Number of Training Samples	Number of Validation Samples	Number of Testing Samples
1	Oilseed rape	101	6038	3601	1207	1230
2	Wheat	103	7418	4458	1483	1477
3	Bare land	101	6084	3654	1216	1214
Total	-	305	19,540	11,713	3906	3921

, the detailed parameters of the sample were described.

2.3. Crop Type Classification

2.3.1. Overview

The flow chart of the proposed method is shown in Figure 4. There were three steps in this crop classification method. Data acquisition and preprocessing should be completed beforehand. In step 1, on the one hand, the MS image was obtained from S-2 data. On the other hand, the VH and VV backscattering components and the polarimetric components from the model-based decomposition in [50] were obtained from S-1 data. The above features were grouped into different combinations. In step 2, the DC-CNN model was constructed, which included three modules: a feature extraction module, a feature fusion module, and an encoder-decoder module. In step 3, the optimal feature combination was obtained by analyzing and evaluating the classification results of different feature combinations, and the final crop distribution map was acquired based on the trained DC-CNN model. In order to measure the accuracy of the classification results, qualitative and quantitative evaluations were made. Qualitative evaluation was applied to evaluate the classification results intuitively [51]. Compared with the samples obtained from ground-truth data, the classification results were interpreted and verified intuitively. To evaluate quantitatively the accuracy of each combination, various parameters (i.e., Macro-F1, Overall Accuracy (OA), and Kappa) were considered [52,53].

2.3.2. Polarimetric Decomposition and Feature Combination

The abundant polarimetric information extracted from the SAR data has great potential for the classification of crop types [54]. However, there were few applicable polarimetric decomposition methods for S-1. Mascolo et al. proposed a novel model-based decomposition in [48] that is adapted for S-1 and by which any Stokes vector can be decomposed into a partially polarized and polarized wave component. In this study, this decomposition was applied to exploit the polarimetric components of S-1, which were introduced into crop classification for the first time. The specific process of decomposition was as follows:

The Sentinel-1 dual-polarimetric SAR data can be represented by the following polarimetric covariance matrix [55,56]:

$$C_{2X2}=\left[\begin{array}{cc}{c}_{11}& c_{12}\\ {c_{12}}^{*}& c_{22}\end{array}\right]=\left[\begin{array}{cc}⟨{\left|S_{VH}\right|}^2⟩& ⟨S_{VH}{S_{VV}}^{*}⟩\\ ⟨S_{VV}{S_{VH}}^{*}⟩& ⟨{\left|S_{VV}\right|}^2⟩\end{array}\right]$$

(1)

where $<>$ is multilook and/or speckle filtering.

The random dipole cloud model, which was widely used in quad-pol decompositions [57,58], was used for the dual-pol decomposition.

Transforming $C_{2X2}$ into the Stokes vector:

$$\underset{\_}{S}=\left[\begin{array}{c}{s}_1\\ s_2\\ \begin{array}{c}{s}_3\\ s_4\end{array}\end{array}\right]=\left[\begin{array}{c}{c}_{11}+c_{22}\\ c_{11}-c_{22}\\ \begin{array}{c}2Re\left(c_{12}\right)\\ 2Im\left(c_{12}\right)\end{array}\end{array}\right]$$

(2)

where $Re\left(c_{12}\right)$ and $Im\left(c_{12}\right)$ represent the real part and the imaginary part of $c_{12}$, respectively.

The Stokes vector $\underset{\_}{S}$ can be decomposed according to the model-based decomposition in [50].

$$\underset{\_}{S}=m_v{\underset{\_}{s}}_v+m_s{\underset{\_}{s}}_p=m_v\left[\begin{array}{c}1\\ \pm 0.5\\ \begin{array}{c}0\\ 0\end{array}\end{array}\right]+m_s\left[\begin{array}{c}1\\ cos2\alpha \\ \begin{array}{c}sin2\alpha cos\delta \\ sin2\alpha sin\delta \end{array}\end{array}\right]$$

(3)

where, on the left side, ${\underset{\_}{s}}_v$ and ${\underset{\_}{s}}_p$ represent the partially polarized and the completely polarized components, respectively, with $m_v$ and $m_s$ being the corresponding powers. Note that, on the right side, the volume term is modeled according to the random cloud of dipoles model. $m_v$ is the power of the partially polarized volume term (also referred to as volume scattering), and $m_s$ is the power of the polarized term (also referred to as surface scattering). The $\alpha$ angle measures the separation between the transmitted and received waves [49], and $\delta$ is the cross-polarized phase.

For different crop types in the early stages of planting, surface scattering is dominant. With the growth and development of crops, the proportion of volume scattering increases gradually. For leafy crops, the proportion of volume scattering is higher than that of surface scattering. The characteristics of different crop canopies will affect the specific value of volume scattering [59,60].

Therefore, as shown in [50], $m_v$ can be obtained by solving the quadratic in Equation (4), where the coefficients $a$, $b,$ and c, are calculated from the random dipoles cloude model.

$$a{m}_v^2+b{m}_v+c=0,\left\{\begin{array}{c}a={\underset{\_}{s}}_v^{T}G{\underset{\_}{s}}_v=0.75\\ b=-2{\underset{\_}{s}}^{T}G{\underset{\_}{s}}_v=-2s_1\pm 0.5s_2\\ c={\underset{\_}{s}}^{T}G\underset{\_}{s}=s_1^2-s_2^2-s_3^2-s_4^2\end{array}\right.$$

(4)

Only one root of the quadratic equation satisfies energy conservation ($m_v\le s_1$), which provides a unique solution for $m_v$.

The $m_s$ can be calculated as follows by hinder qual-pol model-based approaches [61] to avoid all the negative eigenvalue issues:

$$m_s=s_1-m_v$$

(5)

The backscattering components VV/VH were obtained from the preprocessed S-1 data, and $m_s$/$m_v$ were calculated using model-based decomposition. The MS feature (Bands 2, 3, 4, 5, 6, 7, 8, 8A, 11, and 12) was acquired via preprocessed and band-sifted S-2 data.

Table 5. Different combinations composed of S-1 and S-2 features were used.

Combination	Abbreviation	Comment
A	S-1 (VV, VH)	Only the intensity components VV and VH of S1
B	S-1 ($m_s$, $m_v$)	Only the polarimetric components $m_s$ and $m_v$ of S1
C	S-1 (VV, VH, $m_s$, $m_v$)	The intensity components VV, VH, and the polarimetric components $m_s$, $m_v$, of S1
D	S-1 (VV, VH) + S-2(MS)	The intensity components VV, VH of S1 + MS of S-2
E	S-1 ($m_s$, $m_v$) + S-2(MS)	The polarimetric components $m_s$, $m_v$ of S1 + MS of S-2
F	S-1 (VV, VH, $m_s$) + S-2(MS)	The intensity components VV, VH, and the polarimetric components $m_s$ of S1 + MS of S-2
G	S-1 (VV, VH, $m_v$) + S-2(MS)	The intensity components VV, VH, and the polarimetric components $m_v$ of S1 + MS of S-2
H	S-1 (VV, VH, $m_s$, $m_v$) + S-2(MS)	The intensity components VV, VH, and the polarimetric components $m_s$, $m_v$ of S1 + MS of S-2

describes in detail different combinations of extracted features that would be used as different input data sets for subsequent experiments.

Combinations A to C were represented using only the different components extracted from the S-1, which made it convenient to analyze the performance of polarimetric components $m_s$ and $m_v$ in crop classification using only S-1 data. Combinations D to H were different combinations of multi-source features. Combinations F and G were set to evaluate the contribution of polarization features $m_s$ and $m_v$ to the classification of multi-source crops, respectively.

2.3.3. Framework of DC-CNN

CNNs, which are widely used in deep learning approaches for crop classification [62,63], are helpful for effectively discovering salient features in data. However, the features of crops in multi-source RS images were different. In order to effectively use the features extracted by S-1 and S-2, DC-CNN was constructed to investigate the effects of S-1 and S-2 on crop classification. Due to the different characteristics of crops in S-1 and S-2 images, the features obtained from S-1 were used as input for one branch, and the spectral information obtained from S-2 was used as input for the other. Each of the streams learned sensor-specific representations. Through parameter sharing, the representations can be learned from other branches in order to achieve the best classification effect.

As shown in Figure 5, the structure of the DC-CNN included a feature extraction module, a feature fusion module, and an encoder-decoder module and was connected to a SoftMax layer for classification. The feature extraction and feature fusion modules realized feature learning and achieved the purpose of the deep fusion of multi-source features.

Because the characteristics of crops in S-1A and S-2B are different, two convolution kernels were set in the feature extraction module. The feature extraction process was shown in Equation (6):

$${\mathrm{Z}}_{S,i}^{\left(l\right)}=\left\{\begin{array}{c}f\left(w_S^{\left(l\right)}{⨂X}_{S,i}+b_S^{\left(l\right)}\right), l=1,\\ f\left(w_S^{\left(l\right)}⨂{\mathrm{Z}}_{S,i}^{\left(l-1\right)}+b_S^{\left(l\right)}\right), l=2,\dots ,n\end{array}\right.$$

(6)

where $X_S$ represents the source data in the two channels and $S=1,2$. ${\mathrm{Z}}_{S,i}^{\left(l\right)}$ is the output feature after the $l$-layer convolution; and $f\left(x\right)$ is the activation function in the model structure; $X_{S,i}$ represents the $i$-th pixel in the $S$-th channel; $w_S^{\left(l\right)}$ is the weight of the $l$ convolutional layer; $b_S^{\left(l\right)}$ is the bias of the convolutional kernel; $⨂$ represents the convolutional operation; $n$ is the number of layers in the CNN.

The Batch Normalization (BN) layer was set after the convolution layer to speed up the training process. Following that, there was the pooling layer, which uses the 2 $\times$ 2 max pooling to reduce the data variance and computational complexity. Furthermore, ReLU [63] was selected as the activation function; that is, $f\left(x\right)=\mathrm{m}\mathrm{a}\mathrm{x}(0,x)$, which maintains a nonlinear mapping relationship and avoids the problem of gradient disappearance by changing the negative value to zero.

The convolution layer consists of two layers, in which two branches share the parameters $w$ and $b$ in the second layer. At this time, the information in the two branches of the network influences each other, and the loss functions of the two branches jointly determine the gradient update in the back propagation, so as to learn more distinguishing features. Following that, the features extracted from the two channels were spliced and used as inputs for the encoder. The spliced feature was denoted as $\left[{\mathrm{Z}}_{1,i}^{\left(\mathrm{n}\right)}+{\mathrm{Z}}_{2,i}^{\left(\mathrm{n}\right)}\right]$.

$${\mathrm{F}}_i^{\left(l\right)}\left(Z_1,Z_2\right)=f\left(w_S^{\left(l\right)}⨂\left[{\mathrm{Z}}_{1,i}^{\left(\mathrm{n}\right)}+{\mathrm{Z}}_{2,i}^{\left(\mathrm{n}\right)}\right]+b_S^{\left(l\right)}\right),l=n+1,\dots ,m$$

(7)

The fused feature of ${\mathrm{F}}_i^{\left(l\right)}\left(Z_1,Z_2\right)$ was used as the input of the decoder. The feature map was mapped back to the spatial information of the original image through the decoder module. The SoftMax classifier was used to get the final classification result.

The loss function adopted the categorical cross-entropy (CCE) commonly used in multi-classification [64].

$$CCE\left(\widehat{y,}y\right)=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^C{y}_{i,j}·\log \left(p\left({\widehat{y}}_{i,j}\right)\right)$$

(8)

where N represents the number of samples and C is the number of classes. The value $\widehat{y}$ represents the predicted class, with y being the training label.

The detailed parameters of DC-CNN are shown in

Table 6. Parameters of the DC-CNN.

S-1		S-2
Conv: 3 $\times$ 3 $\times$ 16	(n, 7, 7, 16)	Conv: 3 $\times$ 3 $\times$ 16	(n, 7, 7, 16)
BN	(n, 7, 7, 16)	BN	(n, 7, 7, 16)
ReLU	(n, 7, 7, 16)	ReLU	(n, 7, 7, 16)
Max-Pooling: 2 $\times$ 2	(n, 4, 4, 16)	Max-Pooling: 2 $\times$ 2	(n, 4, 4, 16)
Conv: 3 $\times$ 3 $\times$ 32	(n, 4, 4, 32)	Conv: 3 $\times$ 3 $\times$ 32	(n, 4, 4, 32)
BN	(n, 4, 4, 32)	BN	(n, 4, 4, 32)
ReLU	(n, 4, 4, 32)	ReLU	(n, 4, 4, 32)
Max-Pooling: 2 $\times$ 2	(n,2, 2, 32)	Max-Pooling: 2 $\times$ 2	(n,2, 2, 32)
Flatten	128	Flatten	128
Layer	Parameters		Output shape
Joint Layer			(n, 256)
Encoder1	128, activation = ‘ReLU ‘		(n, 128)
Encoder2	64, activation = ‘ReLU ‘		(n, 64)
Encoder3	32, activation = ‘ReLU ‘		(n, 32)
Compressed features	16, activation = ‘ReLU ‘		(n, 16)
Decoder1	32, activation = ‘ReLU ‘		(n, 32)
Decoder2	64, activation = ‘ReLU ‘		(n, 64)
Decoder3	128, activation = ‘ReLU ‘		(n, 128)
Classification	SoftMax		(n, 3)

.

2.3.4. Model Accuracy Evaluation

In order to select the best combination of multi-source features and measure the effectiveness of the DC-CNN model, qualitative and quantitative evaluations were conducted.

For the qualitative evaluation, three regions were selected and compared with the visual interpretation map; each region contained three types of farmland to be classified. Examples of these regions are shown in Figure 6.

For the quantitative evaluation, Macro-F1, Overall Accuracy (OA), and Kappa were employed, which were illustrated in Equation (9), Equation (13) and Equation (14), respectively.

$$\mathrm{Macro}-\mathrm{F}1=\mathrm{A}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e}(\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e})$$

(9)

Here, $\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}$ can be expressed as Equation (12):

$$\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\frac{2\ast Precision\ast Recall}{Precision+Recall}$$

(10)

$$Precision=\frac{TP}{TP+FP}$$

(11)

$$Recall=\frac{TP}{TP+FN}$$

(12)

$$\mathit{OA}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}}$$

(13)

$$\mathit{Kappa}=\frac{\mathit{OA}-p_e}{1-p_e}$$

(14)

$$p_e=\frac{(\mathit{TP}+\mathit{FN})\times (\mathit{FP}+\mathit{TP})+(\mathit{FP}+\mathit{TN})\times (\mathit{FN}+\mathit{TN})}{(\mathit{FP}+\mathit{TN}+\mathit{TP}+\mathit{FN}{)}^2}$$

(15)

where $Precision$ and $Recall$ were calculated via confusion matrices.

3. Classification Results

3.1. Implementation Details

All the experiments were completed in the TensorFlow environment with a GeForce RTX 3080Ti. The weight decay was set to 0.004, and the learning rate was set to 0.001. DC-CNN was trained by an Adam optimizer [65,66] and trained on 100 epochs to obtain the final classification result. The batch size was 32. The hardware and software configurations are presented in

Table 7. Hardware and software configurations of the experiments.

Configuration	Version
GPU	GeForce RTX 3080Ti
Memory	64G
Language	Python 3.8.3
Frame	Tensorflow 1.14.0

.

3.2. Comparison of Feature Combinations

3.2.1. Polarimetric Components in a SAR-Only Image

Figure 7 shows the local comparison of the classification results of three different feature combinations based on DC-CNN, among which the features were extracted from S-1. Obviously, when Combination A was used as input data, the classification result was not ideal. An OA of 76% with a Kappa of 0.462 was achieved. In the three types, the result of bare land classification was better than the other two types, and there was a serious confusion phenomenon between wheat and oilseed rape, as shown in the oval area in Figure 7a(1)–a(3), where a large amount of wheat was mistaken for oilseed rape. After the polarimetric components were introduced into the classification task (Combination B and Combination C), the confusion phenomenon was obviously reduced.

According to

Table 8. Precision, Recall, F1-score, Macro-F1, OA, and Kappa coefficients corresponding to the crop classification results in different Combinations, using only S-1 data. Combination A: S-1 (VV, VH); Combination B: S-1 ($m_s$, $m_v$); Combination C: S-1 (VV, VH, $m_s$, $m_v$).

		Oilseed Rape	Wheat	Bare Land	Macro-F1	OA	Kappa
Combination A	Precision	0.783	0.694	0.806	0.7603	0.7609	0.462
	Recall	0.715	0.771	0.802
	F1-score	0.747	0.730	0.804
Combination B	Precision	0.832	0.752	0.835	0.806	0.8061	0.572
	Recall	0.765	0.825	0.834
	F1-score	0.797	0.787	0.834
Combination C	Precision	0.855	0.780	0.843	0.828	0.8312	0.628
	Recall	0.792	0.854	0.852
	F1-score	0.822	0.815	0.847

, when Combination C was used as the input data set, the accuracy of various classes was significantly improved (Figure 7c(1)–c(3)). Compared with Combination A, the overall accuracy of Combination C increased by about 4.5%, and Kappa reached 0.628. These results indicated that the polarimetric component had a positive effect on crop classification and could effectively improve classification accuracy. It should be noted that the sample set for testing was independent of the training sample and did not participate in the model training process.

3.2.2. Polarimetric Components in SAR-Optical Images

It can be seen from Table 8 that polarimetric components played a positive role in crop classification tasks based on SAR data only. In order to explore its potential in multi-source crop classification tasks, the Combinations D, E, F, G, and H in Table 4 were compared. As shown in Figure 8, the misclassification phenomenon of Combination E (e(1)–e(3)), Combination F (f(1)–f(3)), and Combination G (g(1)–g(3)) was obviously reduced compared with the classification result of Combination D (d(1)–d(3)). The only difference between Combinations D, F, and G was that the polarimetric component was added to Combinations F and G. When all the features of S-1 and S-2 were used (h(1)–h(3)), the classification results were more accurate for both the interior and edge of farmland.

The predicted results of the model with different feature combinations were compared with the ground-truth map on the ground. Following that, make a quantitative evaluation. The specific accuracy results are shown in

Table 9. Precision, Recall, F1-score, Macro-F1, OA, and Kappa coefficients corresponding to the crop classification accuracy in different Combinations.

		Oilseed Rape	Wheat	Bare Land	Macro-F1	OA	Kappa
Combination D	Precision	0.909	0.889	0.911	0.9030	0.9030	0.8545
	Recall	0.867	0.905	0.937
	F1-score	0.888	0.897	0.924
Combination E	Precision	0.918	0.941	0.943	0.9340	0.9340	0.9010
	Recall	0.919	0.937	0.946
	F1-score	0.919	0.939	0.945
Combination F	Precision	0.940	0.933	0.956	0.9433	0.9430	0.9145
	Recall	0.928	0.946	0.955
	F1-score	0.934	0.940	0.956
Combination G	Precision	0.967	0.972	0.972	0.9707	0.9703	0.9554
	Recall	0.961	0.967	0.983
	F1-score	0.964	0.970	0.978
Combination H	Precision	0.976	0.990	0.986	0.9840	0.9840	0.9760
	Recall	0.986	0.974	0.992
	F1-score	0.981	0.982	0.989

.

For visually observing the difference between different input precisions, a visual comparison of accuracy evaluation indicators is presented in Figure 9. The accuracy of Combination E can reach more than 90%, which indicates the efficiency of crop classification by combining only the polarimetric components and MS features. Apparently, compared with Combination D, the classification accuracy of Combination F, Combination G, and Combination H was obviously improved. That is, the addition of $m_s$ and $m_v$ can increase the discrimination between crops and reduce the misclassification. Especially when both $m_s$ and $m_v$ were added at the same time, the classification result was the best, and the accuracy could reach 98%. This showed that the scattering characteristics reflected by polarimetric information played an effective role in crop classification. Combination H was used as the optimal feature combination in subsequent experiments.

In order to verify the effect of separating polarimetric features $m_s$ and $m_v$ from SAR data on improving classification accuracy, the feature group without a polarimetric feature (Combination D) and the feature group with a polarimetric feature (Combination H) were analyzed visually, as shown in Figure 10. The multi-dimensional features are mapped to two-dimensional space by the Principal Component Analysis (PCA) method. In the Figure 10, the horizontal and vertical coordinates were the first and second principal components that provided most of the information about the data.

There were three different colored dots in the figure, which represented three kinds of samples. It can be seen that when only the MS of S-2 and the intensity features (VV and VH) of S-1 were combined, confusion between the three categories was evident. In particular, oilseed rape was confused with bare land and wheat to different degrees. Confusion between the three crop types was improved when polarimetric components ($m_s$ and $m_v$) were added. In particular, the differentiation between bare land and oilseed rape samples increased markedly. More specifically, in areas comprising bare land without sowing, the scattering characteristics mainly reflect surface scattering, so the $m_s$ is always larger than the $m_v$. Oilseed rape and wheat were mature during the study period, and thus the $m_v$ was dominant, meaning that they could be clearly distinguished from bare land. The differences in plant height and canopy between oilseed rape and wheat mean that their $m_v$ values differ to some extent. The results showed that polarimetric components can enhance the differentiation between samples, which is conducive to improving classification accuracy.

3.3. Accuracy Comparison with Other Classifiers

Figure 11 shows the classification results of the DC-CNN model when using Combination D. This taxonomic map clearly identified three crop categories: oilseed rape, wheat, and bare land. The results showed that the fields in the southwest of Tongxiang were complete and wide and that wheat was mainly planted here. Wheat and oilseed rape fields spread throughout the region; some bare land was scattered.

In order to prove the effectiveness of the proposed method in crop classification with multi-source remote sensing data, three deep learning methods were selected for comparison, including the traditional 2D-CNN method, CNN-based FCN, and SegNet [42,43,44,45]. For fairness of comparison, the input data sets of all methods were Combination G, and the training samples, verification samples, and test samples used in classification were consistent.

3.3.1. Qualitative Evaluation

In qualitative evaluation, three areas were selected for comparison with ground-truth in order to visually compare the classification results. As shown in Figure 11, the 2D-CNN, FCN, and SegNet methods all had many outliers and misclassified crops. In the rectangular area in Figure 12a(1)–c(1), oilseed rape was misclassified as bare land to different degrees. The classification results obtained by the proposed method were the closest to the actual situation when the training samples, test samples, and input data were all the same. This indicated that the proposed method can effectively utilize the multi-dimensional information of multi-source images and achieve a good effect.

3.3.2. Quantitative Evaluation

To verify the validity of the DC-CNN method, 2D-CNN, FCN, and SegNet classifiers were used for comparison. All these classifiers were quantitatively evaluated, and confusion matrices were created for them (Figure 13). The results were shown in

Table 10. Precisions, Recalls, F1-scores, Macro-F1, OA, and Kappa coefficients corresponding to the crop classification accuracy in different methods (the input data sets are all Combination G).

		Oilseed Rape	Wheat	Bare Land	Macro-F1	OA	Kappa
2D-CNN	Precision	0.940	0.934	0.958	0.9467	0.9487	0.9230
	Recall	0.933	0.955	0.958
	F1-score	0.937	0.945	0.958
FCN	Precision	0.963	0.953	0.972	0.9630	0.9627	0.9440
	Recall	0.954	0.963	0.971
	F1-score	0.959	0.958	0.972
SegNet	Precision	0.965	0.978	0.964	0.9693	0.9690	0.9534
	Recall	0.966	0.959	0.982
	F1-score	0.966	0.969	0.973
DC-CNN	Precision	0.976	0.990	0.986	0.9840	0.9840	0.9760
	Recall	0.986	0.974	0.992
	F1-score	0.981	0.982	0.989

.

According to the indicators in Table 10, the accuracy of the DC-CNN model was the highest. The Macro-F1 was 0.9840, the OA was 0.9840, and the Kappa was 0.9760; the accuracy metrics of the other models were slightly lower than those of the DC-CNN. FCN and SegNet acquired Macro-F1 of 0.9630 and 0.9693, respectively; OA of 0.9627 and 0.9690, respectively; and Kappa scored 0.9440 and 0.9534, respectively. Among the models tested, the 2D-CNN model exhibited the worst classification performance, with three indicators of 0.9467, 0.9487, and 0.9230, respectively. The Macro-F1 score of the proposed method was increased by about 4%, the OA was increased by 5%, and the Kappa was increased by around 5% compared with the 2D-CNN. For the oilseed rape and wheat categories, which were easily confused, the accuracy was improved by about 5% and 2%, respectively, and the accuracy of bare land identification was improved by about 3% compared with the traditional CNN. This experiment proved that the ability of the DC-CNN classifier to classify crops using multi-source data was improved to some extent when compared with other existing classifiers.

4. Discussion

Compared with a single data source, the advantages of using multi-source data synergistically in classification have been confirmed in many studies [30,31,32,33]. In fact, we also evaluated the effects of using only SAR data, only optical data, and a SAR-optical combination on crop classification. The results were shown in

Table 11. Overall classification accuracy, using only S-1, only S-2, and both combined.

	Macro-F1	OA	Kappa
S-1 (VV, VH, $m_s$, $m_v$)	0.8280	0.8312	0.6280
S-2(MS)	0.9430	0.8854	0.7403
S-1 (VV, VH, $m_s$, $m_v$) + S-2(MS)	0.9840	0.9840	0.9760

.

The classification accuracy obtained by only optical was about 5% higher than that obtained by only SAR, and the target type can be better reflected in optical data, which is the same conclusion as [67,68,69,70,71]. The cooperative use of SAR-optical provided multi-dimensional information, including the physical scattering mechanism of SAR and the MS of optical data, and the classification result was better than that of a single sensor. As shown in Table 11, compared with S-1 and S-2 alone, the OA of synergy between S-1 and S-2 is improved by 15% and 10%, respectively. [72] also showed that compared to the sole use of S-1 and S-2 data, combining S1 and S2 can improve OA by 3% and 10%, respectively. The classification result in this paper was more accurate (OA improved by almost 5%) than that in the literature [45]. On the one hand, there may be relatively few crop species in the study area; on the other hand, the importance of SAR polarization information has been frequently discussed in many crop classification studies, such as wheat and oilseed rape [73,74,75]. The experiment in this paper showed that the polarimetric components obtained by the new decomposition method [49] had a more outstanding contribution to classification than VV/VH. Polarimetric components were highly sensitive to crop canopy structure. Since the vertical state of canopy structure in wheat maturity is obviously different from that in oilseed rape, components performed well in classification tasks. As can be seen from Table 9, the final classification results obtained by the proposed method were more accurate than those obtained by the other comparison methods. Compared with 2D-CNN, FCN, and SegNet, the Macro-F1, OA, and Kappa improved by 2–4%, 2–4%, and 2–5%, respectively. However, in the early growth stages of wheat and oilseed rape, due to their similar physical characteristics, the contribution of polarimetric components will be affected to some extent. In the future, other features that can effectively distinguish wheat from oilseed rape will be used for classification.

Not only feature quality and correlation were important aspects of crop classification with multi-source data, but also the classification model affected classification results. The effectiveness of 2D-CNN, FCN, and SegNet in crop classification has been proven [43,44,45,46]. Good classification results have also been obtained in this experiment, and the accuracy rate is over 90%. However, under the condition of using the same input data set, the DC-CNN obtained more accurate classification results. The proposed method can effectively utilize and fuse the features in multi-source data, integrate the physical and structural properties of the target surface contained in SAR, and utilize the spectral information contained in optics. Meanwhile, 2D-CNN, FCN, and SegNet made insufficient use of features and lost feature information. Nevertheless, the training samples in the proposed method were collected by manual investigation, which required a lot of manpower and material costs. Therefore, in the future, it will be important to design an accurate classification method for crops with a small number of samples.

5. Conclusions

In this paper, DC-CNN was constructed to classify crops more accurately based on S-1 and S-2 data. Advanced features were extracted using the feature extraction and feature fusion modules of the model, and the deep fusion of multi-source features was realized. Moreover, the influence of polarimetric information from dual-pol SAR data on model-based decomposition on crop classification was observed for the first time. Experiments showed that the classification of farmland type was best when the polarimetric components, backscattering coefficient, and spectral information were simultaneously used as model inputs. The difference between different sample categories was augmented, and the probability of correct classification was elevated when polarimetric information was utilized. The results showed that the polarimetric information in SAR images can play an important role in crop classification. Meanwhile, the advantages of the proposed method involving the fusion of multi-source data were verified by comparing the classification results with those of other common methods. By using single-date imagery of S-1 and S-2 for crop classification, the results of the proposed method were more accurate compared with other methods. The proposed method classifies crops more accurately. The method proposed in this paper also provides an idea for areas with similar weather conditions, namely, using limited optical and SAR images to classify crops.

Along with the development and application of multi-source RS for the task of crop classification, which data to use and how to effectively use multi-source data are key issues. The results of this paper clearly showed the effectiveness and superiority of this method in SAR-optical combination crop classification. However, its application ability needs to be further developed to adapt to the classification task in a complex environment with more modalities. Future research will focus on whether other features that can be extracted from multi-source data, such as vegetation index extracted from optical data and texture information extracted from SAR data, can be used to improve classification accuracy. Another aspect is that the model is extended to make it suitable for classification tasks under more data conditions, such as data fusion of more than two modalities or collaboration of multi-source and multi-temporal data.