Multi-Source Remote Sensing Pretraining Based on Contrastive Self-Supervised Learning

Abstract

SAR-optical images from different sensors can provide consistent information for scene classification. However, the utilization of unlabeled SAR-optical images in deep learning-based remote sensing image interpretation remains an open issue. In recent years, contrastive self-supervised learning (CSSL) methods have shown great potential for obtaining meaningful feature representations from massive amounts of unlabeled data. This paper investigates the effectiveness of CSSL-based pretraining models for SAR-optical remote-sensing classification. Firstly, we analyze the contrastive strategies of single-source and multi-source SAR-optical data augmentation under different CSSL architectures. We find that the CSSL framework without explicit negative sample selection naturally fits the multi-source learning problem. Secondly, we find that the registered SAR-optical images can guide the Siamese self-supervised network without negative samples to learn shared features, which is also the reason why the CSSL framework outperforms the CSSL framework with negative samples. Finally, we apply the CSSL pretrained network without negative samples that can learn the shared features of SAR-optical images to the downstream domain adaptation task of optical transfer to SAR images. We find that the choice of a pretrained network is important for downstream tasks.

1. Introduction

Today, with the developments of earth observation technology and sensor hardware platforms, a large number of Earth observation satellites are operating [1]. There are an increasing number of remote-sensing images from different sources that we can obtain [2]. However, there are many limitations in the practical application of the images and spectral information from a single sensor, which has made it difficult to meet the daily requirements [3] Therefore, the application of multi-source images has become a developmental trend in the field of remote sensing [4] These can fuse the information of different types of images obtained from different sensors, improving the accuracy and richness of the information contained in the same area [5,6] The registered multisource scene images can provide information about the structural consistency of the scene, and the effective use of this information can improve the classification accuracy of the scene images. However, traditional fusion methods rely on manually designed filters to extract features [7,8] and it is difficult to design ideal feature extraction methods or fusion rules. With their powerful data processing capability, convolutional neural networks are increasingly used in multi-source remote sensing and have become the main method for remote sensing image analysis [9] The most primitive method of convolutional neural networks in image processing is to send a large number of images and their unique corresponding tags into the network and extract image features from the neural network by supervised training [10]. The accuracy and efficiency of this end-to-end network are greatly improved. Nevertheless, the supervised training process consumes a large amount of time, memory, and annotated data. Consequently, the most commonly used image classification method in deep learning is to transfer the pretrained network on a large-scale annotated dataset to the downstream classification task for finetuning [11].

The pretrained network can bring initial model parameters, so that helps to achieve higher task accuracy and speed up model convergence [12]. The most widely used pretraining network is based on ImageNet [13], a large, labeled dataset in the computer vision (CV) field. The ImageNet pretrained network learned generalized feature representations from a mass of natural annotation images [14]. There are many CNN classification methods pretrained on the ImageNet dataset that have been widely used on remote-sensing scene images for finetuning [15]. However, due to the huge difference in imaging principles between natural optical images and remote-sensing images, remote-sensing images and natural images have great differences in color features and texture features, especially in data distribution between SAR images and natural optical images [16]. The prior knowledge of the model brought by transferring the network pretrained on the natural optical image is not reliable, and system bias will be introduced [17]. The remote-sensing image finetuning with the ImageNet pretrained network may introduce strong class-discriminative biases for shapes [18] and textures [19] from the pretraining dataset [20]. Those problems are difficult to solve only by using remote-sensing data for finetuning. Therefore, the most direct solution is to construct a remote-sensing annotation dataset of the same size as the ImageNet dataset and then train a general used pretrained network on this remote-sensing dataset.

The number of remote-sensing images we can obtain increases exponentially, but it is time-consuming to annotate them and requires a lot of expert knowledge [21,22]. In addition, there are many types of remote-sensing images, and the imaging methods obtained by various sensors are different [23]. Also, in SAR images, there are differences in the data distribution of images from different wavebands and imaging angles [24]. Hence, the idea of training a general used network on such a complete remote-sensing dataset is not tenable. The key to solving these problems is how to train large datasets without the need for annotations.

In recent years, self-supervised learning has provided a new idea for unlabeled data utilization [25]. Self-supervised learning is based on the idea of training a neural network to solve an artificially designed pretext task by constructing pseudo-labels from unlabeled samples to drive model learning features [17]. The pretext task is the most important role in self-supervised learning, which should prompt the neural network to learn meaningful image representations. Self-supervised learning can be divided into contrastive self-supervised learning (CSSL) and generative self-supervised learning (GSSL) according to different pretext tasks [26]. The core concept of GSSL is feature learning by restoring artificially damaged images to the original images. In addition, CSSL learns features through contrasting latent representations of different transformation augmentations. Consequently, GSSL heavily focuses on pixel-level information, which will lead to redundancy and memory wasting. Since CSSL focuses on extracting easily distinguishable features between categories, it saves more memory than GSSL and is widely used by researchers [27,28]. CSSL can be divided into CSSL methods with negative samples and CSSL methods without negative samples according to whether there are negative samples or not. Typical CSSL methods with negative samples include MOCO [29], SimCLR [30] and CMC [31], while typical CSSL methods without negative samples include BYOL [32] and SimSiam [33]. The self-supervised learning pretrained neural network can be used in downstream supervised tasks [16]. Accordingly, self-supervised learning can be used to develop the pretraining network on unlabeled multi-source remote-sensing data.

The strategy of multi-source image fusion can fuse the information of different types of images obtained from different sensors to improve the accuracy and richness of the information contained in the same area [34]. Stojni proposed applying the CMC structure in multi-source remote-sensing images [17]. However, in the face of CSSL methods under different frameworks, no research shows which framework is more suitable for multi-source images. In the meantime, BYOL and SimSiam achieve superior performance on single-source images by both using a Siamese network and discarding the negative samples during the network training. The Siamese network structure of BYOL and SimSiam can better utilize the consistency information brought by registered SAR-optical images so that the network can learn the shared features of SAR-optical images. Consequently, we apply SAR-optical images to the CSSL framework without negative samples. The features extracted from SAR-optical remote-sensing images in the network with and without negative samples are compared experimentally, and the pretrained network is transferred to the downstream single-source scene classification task to compare their accuracy.

In fact, self-supervised pretraining networks are widely used [12]. In the downstream transfer task of optical to SAR images in the field of remote-sensing, domain adaptation is usually applied to improve the transfer accuracy, such as in DANN [35], DAN [36] and CDAN [37], because of the large difference in the distribution of optical and SAR images. However, in domain adaptation, the use of ImageNet pretrained networks as feature extractors to align the data distributions of optical and SAR images suffers and deteriorates the transfer accuracy. In this paper, a contrastive self-supervised pretraining network based on registered SAR-optical images is used to replace the ImageNet pretraining network to extract features of optical and SAR images and verify whether the accuracy of the domain adaptation transfer task can be improved.

To address the aforementioned issues, we conducted extensive experiments on six popular datasets. The three main contributions are summarized as follows:

(1)

We introduce the CSSL method with negative samples and the CSSL method without negative samples into multi-source remote-sensing image applications simultaneously, and we systematically compare the effectiveness of the two different frameworks for remote-sensing image classification for the first time.

(2)

We determine through analysis that the registered SAR-optical images have structural consistency and can be used as natural contrastive samples that can drive the Siamese network without negative samples to learn shared features of SAR-optical images.

(3)

We explore the application of self-supervised pretrained networks to the downstream domain adaptation task of optical transfer to SAR for the first time.

2. Methods

We address the problem of improving unlabeled SAR-optical data utilization in a contrastive self-supervised fashion. This paper applies the two frameworks of CSSL to SAR-optical images. The CSSL framework with negative samples requires the encoder to decrease the distance between views mapped to instances in the latent space while increasing the distance between other instances (see Figure 1a) [38]. Another framework is CSSL without negative samples, which tasks the encoder with predicting different augmented views of input samples to learn features (see Figure 1b). However, multiple views in single-source images often require the use of random augmentation to simulate them (see Figure 2a). Multi-source remote-sensing images naturally have multiple views, especially the registered multi-source remote-sensing scene images. The images obtained by different sensors in the same scene have strong structural consistency, forming a natural positive sample pair as shown in Figure 2b.

In this section, we introduce different contrastive frameworks on SAR-optical data augmentation strategies (see Figure 1c). We propose to explore which contrastive framework can effectively improve the utilization of SAR-optical images. We also investigate the principles of those methods and propose a novel application scenario of SAR-optical contrastive self-supervised pretraining network on domain adaptation tasks.

2.1. Multi-Source Contrastive Self-Supervised Method with Negative Samples

In a set of data, different augmented views of the same image constitute positive sample pairs, and augmented views of different images constitute negative sample pairs. The purpose of contrastive learning is to reduce the distance between positive samples and features and increase the distance between negative samples to learn instance-level features.

Given a dataset of X that consists of a collection of image pairs $\{x_1,x_2,\dots ,x_n\}$,${\{x_i,x_i^{+}\}}_{i=1,2,\dots ,n}$ are positive pairs where $x_i^{+}$ is an augmentation sample of $x_i$, ${\{x_i,x_j\}}_{i\ne j}$ are negative pairs. In contrastive loss function, $f(\cdot )$ is a feature extractor consists of neural network. Contrastive loss function is sampled as Equation (1):

$$L=-E_x·\left[\log \frac{\exp (f{(x_i)}^{T}f(x_i^{+}))}{\exp (f{(x_i)}^{T}f(x_i^{+}))+{\displaystyle \sum _{j=1}^{n-1}\exp (f{(x_i)}^{T}f(x_j))}}\right]$$

(1)

CMC (contrastive multi-view coding) is a multi-view contrastive method. In multisource image contrastive application, images in the same scene are regarded as positive sample pairs, while different source images in different scenes are regarded as negative sample pairs. As shown in Figure 3, $f_{{\theta }_1}(\cdot )$, $f_{{\theta }_2}(\cdot )$ are the same feature extractors, and $p_{{\theta }_1}(\cdot )$, $p_{{\theta }_2}(\cdot )$ are the same projectors with different parameters ${\theta }_1$, ${\theta }_2$. Given two datasets of optical images $X_1$ and SAR images $X_2$ that consist of a collection of samples ${\{x_1^i,x_2^i\}}_{i=1}^N$, we take $\{x_1^i,x_2^i\}$ as positive sample pairs, and take $\{x_1^i,x_2^j\}$ as negative pairs. The features of images are extracted as $y_1^i=f_{{\theta }_1}(x_1^i)$, $y_2^i=f_{{\theta }_2}(x_2^i)$, and then projected as $z_1^i=p_{{\theta }_1}(y_1^i)$, $z_2^i=p_{{\theta }_2}(y_2^i)$.

In this paper, ResNet18 [15] is selected as the feature encoder, and we use a fully-connected linear layer for the projection network. After the feature encoder extracts image features, feature vectors are mapped into space, and the distance between feature vectors is calculated by the loss function.

Cosine similarity is used in discriminating functions to measure similarity between samples. The discriminating function $h(\cdot )$ is trained to achieve a high value for positive pairs and a low for negative pairs and can be defined with the temperature parameter τ as:

$$h(\{x_1^i,x_2^i\})=\exp \left(\frac{z_1^i{z}_2^i}{‖z_1^i‖‖z_2^i‖}\cdot \frac{1}{\tau }\right)$$

(2)

Loss function of CMC can be defined as:

$$L_{x_1,x_2}=-\underset{\{x_1^1,x_2^2,\dots ,x_2^{M+1}\}}{E}\left[\log \frac{h(\{x_1^1,x_2^1\})}{{\displaystyle {\sum }_{j=1}^{M+1}h(\{x_1^1,x_2^j\})}}\right]$$

(3)

where view $x_1$ is an anchor, and there are M negative samples all sampled from $x_2$. Therefore, the whole loss in the training set can be defined as:

$$L(x_1,x_2)=L_{x_1,x_2}+L_{x_2,x_1}$$

(4)

At the end of the calculation of the loss function, the distance between the positive sample pairs is increasing, and the distance between the negative sample pairs is decreasing.

2.2. Multi-Source Contrastive Self-Supervised Method without Negative Samples

The contrastive learning method without negative samples can not only eliminate the impact of negative sample data volume on performance but also greatly reduce memory consumption. In contrastive learning methods without negative samples, asymmetric Siamese networks are generally used to map the features from different augmentations of the same image for mutual prediction. We apply SAR-optical images to the CSSL framework without negative samples, as shown in Figure 4. The network extracts features from optical and SAR images registered in the same scene and drives different branches of the Siamese network to predict each other to learn the shared features of the optical and SAR images.

The improved model of the SAR-optical multi-source structure of BYOL and SimSiam is shown in Figure 5 and Figure 6, which improves the original single-source input image into the optical and SAR multi-source input structure. There are two different parameter update strategies between the BYOL and SimSiam structures.

The working method of multi-source BYOL is as follows: Remote-sensing images from different sources are input to the online network and the target network respectively. Features of different enhanced views are extracted by the encoder $f_{\theta }$ and $f_{\xi }$ and get feature representation $y_{\theta }$ and $y_{\xi }$. The feature representations are mapped by the projector $g_{\theta }$ and $g_{\xi }$, and finally get feature vectors $z_{\theta }$ and $z_{\xi }$. Resnet-18 is used as the encoder output feature, and then the projector and predictor are constructed by a multi-layer perceptron (MLP) network [39]. The online network needs to go through MLP mapping again and finally get vector $q_{\theta }(z_{\theta })$. The target network finally get vector $z_{\xi }$. The vector obtained from the online network predicts the vector of the target network and calculates the loss through the loss function:

$$L_{\theta ,\xi }=2-2\cdot \frac{〈q_{\theta }(z_{\theta }),z_{\xi }〉}{{‖q_{\theta }(z_{\theta })‖}_2\cdot {‖z_{\xi }‖}_2}$$

(5)

where $\left|\right|\cdot |{|}_2$ is $l_2$-norm. Similarly, we exchange online network inputs with target network inputs and define the loss as $L_{\theta ,\xi }^{\prime }$, the total loss in BYOL can be defined as:

$$L_{BYOL}=L_{\theta ,\xi }+L^{\prime }{}_{\xi ,\theta }$$

(6)

Only the online network parameter $\theta$ is updated by an optimizer for each calculated loss $L_{BYOL}$, and the target network parameter $\xi$ is updated by exponential average strategy depending on $\theta$:

$$\theta \leftarrow optimizer(\theta ,{\nabla }_{\theta }{L}_{BYOL},\eta )$$

(7)

$$\xi \leftarrow \omega \theta +(1-\omega )\theta$$

(8)

where optimizer (·) in Equation (7) denotes an optimizer. Equation (7) represents the use of a selected optimizer to update the gradient and parameters of the loss function $L_{BYOL}$. In this paper, we used the SGD (stochastic gradient descent) optimizer [40]. According to the derivation of the parameter θ in the loss function $L_{BYOL}$ in Equation (7), this parameter is updated according to the stochastic gradient descent strategy and the set learning rate η. And ω in Equation (8) denotes the target decay rate.

SimSiam differs from BYOL in that the two branch networks in SimSiam share parameters and only update online network parameters according to the loss function.

2.3. Domain Adaption Based on Multi-Source Contrastive Self-Supervised Pretraining

When there is a huge gap between the data distribution of a source domain and the target domain, the method of domain adaptation is generally used to align their data distribution and therefore improve the classification accuracy of the target domain. In the case of the optical and SAR image transfer tasks, unsupervised domain adaptation methods are adopted when the annotation of SAR data is difficult. An ImageNet pretrained network is used for feature extraction domain adaptation. This approach ignores the difference between natural optical images and remote-sensing images, which will lead to low classification accuracy in the transfer task of optical to SAR images.

In this paper, the SAR-optical contrastive framework without negative samples in Section 2.2 can theoretically learn the shared features of SAR-optical images. Therefore, we propose to replace the ImageNet pretraining network with the contrastive self-supervised pretraining network without negative samples to extract features of optical images and SAR images, as shown in Figure 7 [41]. The upper part exemplified the contrastive self-supervised training process of SimSiam. The pretrained feature extractor was transferred to the domain adaptation method based on metric learning exemplified in the lower part to extract image features more effectively to improve the transfer accuracy.

3. Experiments

3.1. Datasets

(1) So2Sat LCZ42 [42]: So2Sat LCZ42(So2Sat) contains 17 categories of data and is a dataset composed of 400,673 pairs of corresponding Sentinel-1 SAR and Sentinel-2 multispectral image patches with local-climate-zone (LCZ) labels. The dataset covers 42 city-dense areas on all continents except Antarctica. Sentinel-1 consists of two polar-orbiting satellites equipped with C-band SAR remote-sensing systems, covering an area of less than 400 km. Sentinel-1 data in the So2Sat dataset contain 8 real-valued bands, while the Sentinel-2 data in the So2Sat LCZ42 dataset contain 10 real-valued bands. SAR-optical image pairs and the corresponding LCZ labels were sampled to 10 m spatial resolution and cropped into 32 × 32 pixels patches. We selected five types of data with obvious characteristics for the experiments, compact high rise, heavy industry, dense trees, bare soil, and sand and water. A sample diagram of the So2Sat LCZ42 dataset is shown in Figure 8.

(2) SEN1-2 [43]: SEN1-2 is a dataset composed of 282,384 pairs of corresponding Sentinel-1 SAR and Sentinel-2 multispectral image patches without labels, collected from across the globe and throughout all meteorological seasons. SAR-optical image pairs were sampled to 10 m spatial resolution and cropped into 256 × 256 pixels patches. A sample diagram of the SEN1-2 dataset is shown in Figure 9.

(3) QXS-SARPORT [44]: QXS-SARPORT(QXS) is a dataset composed of 20,000 pairs of SAR and optical images without labels. SAR patches from SAR satellite GaoFen-3 and optical patches from Google Earth. Those images are collected from three ports: San Diego, Shanghai, and Qingdao. SAR-optical image pairs were sampled to 1 m spatial resolution and cropped into 100 × 100 pixels patches. A sample diagram of the QXS dataset is shown in Figure 10.

(4) UCMerced_LandUse [45]: UCMerced_LandUse (UCM) is a dataset composed of 21 types of optical remote-sensing scene images, with an image resolution of 0.3 m and an image size of 256 × 256 pixels. We selected six types of data with obvious characteristics for the experiments, beach, forest, river, dense residential, medium residential, and storage tank. A sample diagram of the UCMerced_LandUse dataset is shown in Figure 11.

(5) AID [46]: AID is a dataset composed of 30 types of optical remote-sensing scene images, with an image resolution of 0.5–0.8 m and an image size of 600 × 600 pixels. We selected six types of data with obvious characteristics for the experiments, beach, forest, river, dense residential, medium residential, and storage tank. A sample diagram of the AID dataset is shown in Figure 12.

(6) OpenSarUrban [47]: OpenSarUrban is a Sentinel-1 dataset composed of 10 different target area categories of urban SAR images, with a resolution of 20 m and image size of 100 × 100 pixels. A sample diagram of the OpenSarUrban dataset is shown in Figure 13.

3.2. Experimental Setup

The experiments of this research are divided into three parts: the first part is the process of CMC, BYOL, and SimSiam contrastive self-supervised feature extractor training. The second part is to evaluate self-supervised methods by transferring those pretrained networks to the downstream single-source SAR and optical classification tasks. Then, the third step is transferring BYOL and SimSiam self-supervised pretrained networks to the downstream task of optical images transfer to SAR images, replacing the original feature extractor pretrained on the ImageNet dataset in the domain adaption network.

3.2.1. Self-Supervised Network Training

The pretraining data in self-supervised learning can be presented as 20,000 pairs of SAR-optical images. For CMC, the optical and SAR images of the same scene are positive sample pairs, while different scene images are negative sample pairs. We used SGD [40] as the optimizer in the experiment: The SGD was represented by a learning rate of 0.05, the weight decay was given at 0.0004, and the momentum was set to 0.9 in the CMC training step. We used a batch size of 256, and models were run for 200 epochs. For multi-source BYOL and SimSiam, SAR-optical image pairs were divided into two branches, one for online network input and the other for target network input. The SGD with a learning rate of 0.05, a weight decay of 0.0004, and a momentum of 0.9 was adapted as our optimizer, and Gaussian blurring augmentation was used to improve the performance.

3.2.2. Downstream Classification Task

We assessed the performance of three SAR-optical self-supervised methods after self-supervised pretraining on the training set of the So2Sat LCZ42 dataset. We first evaluated their representation by training a linear classifier on top of the frozen representation without updating the network parameters and the batch statistics. Then, we evaluated our representation by finetuning the pretrained network respectively pretrained on SEN1-2, So2Sat and QXS datasets to assess whether the features learned on different SAR-optical fusion datasets were generic and ultimately useful or not in classification tasks of single-source So2Sat, UCM, AID, and OpenSarUrban datasets. The total finetuning training data amount in the downstream SAR image and optical image classification tasks was 6684, the data amount of the training set increased successively, and the data amount of the test set was 6654. In detail, we trained the linear classifier for 50 epochs.

3.2.3. Downstream Domain Adaption Task

In this paper, the domain adaptation method based on metric learning and the domain adaptation method based on adversarial learning were adopted. We used a SAR-optical self-supervised pretraining network without negative samples as a feature extractor in the transfer task of optical to SAR images. The source domain data were optical remote-sensing images from So2Sat, and the target domain data were SAR images from So2Sat.

4. Results

We assessed the performance of BYOL’s representation after self-supervised pretraining on the training set of the So2Sat dataset. We first evaluated it on both RGB and SAR images from So2Sat dataset in a linear evaluation setup. We then measured its transfer capabilities on other datasets as well.

Firstly, we wanted to analyze the influence of the augmentation contrastive strategies of single-source and multi-source data in CSSL; we also wanted to explore the effectiveness of multi-source augmentation strategies in CSSL frameworks with and without negative samples through the classification accuracy by training a linear classifier of downstream single-source remote-sensing scene images in So2Sat dataset. Secondly, we wanted to investigate the reason which multi-source strategy is more effective under which CSSL framework. Thirdly, we measured the transfer capabilities of the CSSL framework without negative samples on other datasets. Finally, we evaluated the possibility of extracting features in the task of domain adaptation transfer from optical to SAR images using a previously trained CSSL model without negative samples.

4.1. Comparison on the Linear Classification Task

The effectiveness of the encoders pretrained by BYOL, SimSiam, CMC, SimCLR, and MOCO [48] was evaluated by performing the linear classification protocol on features extracted in an unsupervised way. We compared self-supervised methods with pure supervised methods and the ImageNet pretrained network, and we also compared single-source CSSL methods with SAR-optical CSSL methods on single-source SAR and optical image classification tasks.

Firstly, we compared the CSSL methods with the ImageNet pretrained network and the pure supervised method on the linear classification downstream task of remote sensing images. We compare the CSSL methods in

Table 1. Overall accuracy (%) obtained by different methods on the test set with a linear classifier.

Method		Test Data
		So2Sat LCZ42-SAR	So2Sat LCZ42-OPT
Single-source without negative samples	BYOL-SAR	79.44%	-
	BYOL-OPT	-	91.62%
	SimSiam-SAR	73.85%	-
	SimSiam-OPT	-	86.23%
Single-source with negative samples	SimCLR-SAR	77.23%	-
	SimCLR-OPT	-	86.71%
	MOCO-SAR	74.43%	-
	MOCO-OPT	-	86.51%
Multi-source with negative samples	CMC	74.03%	87.09%
Multi-source without negative samples	Multi-source-BYOL	81.22%	92.37%
	Multi-source -SimSiam	80.52%	91.04%
	ImageNet	72.95%	84.61%
	Supervised	73.71%	86.16%

with the accuracy of ImageNet and supervised methods in the last two rows. We found that the CSSL methods achieved higher accuracy than that of ImageNet pretrained and pure supervised. Therefore, it is much better to train the self-supervised models on remote-sensing images than the ImageNet pretrained network and pure supervised.

Secondly, we compared the CSSL methods with negative samples of the single-source augmentation strategy such as MOCO, SimCLR and the contrastive methods without negative samples such as BYOL and SimSiam. We can see that in the linear classification downstream task of single-source remote-sensing images, BYOL achieved the best accuracy, while the SimSiam method without negative samples had low accuracy. We speculate that the parameter update strategy is important in the upstream training of contrastive learning from single-source remote-sensing images without negative samples. Thirdly, we compared the effectiveness of BYOL without negative samples in single-source and multi-source augmentation strategies. It can be seen that the SAR-optical augmentation strategy greatly improved the linear classification accuracy of single-source optical and SAR images, and the SAR-optical augmentation made the upstream feature encoder learn more effective features.

Thirdly, we compared the effectiveness of the SAR-optical augmentation strategy in the negative sample method CMC and the CSSL methods without negative samples, such as BYOL and SimSiam. In the linear classification results, the SAR-optical method without negative samples achieved higher classification accuracy. It was proven that the SAR-optical method without negative samples learned more effective features, and the shared features of SAR and optical images learned by BYOL and SimSiam are more conducive to transfer to the downstream single-source classification task.

Finally, we compared the upstream self-supervised training time of different CSSL methods on a GeForce RTX 3070 GPU with a memory size of 16G, as shown in

Table 2. Training time of CSSL methods (h: hours, m: minutes, s: seconds).

Methods	MOCO	SimCLR	CMC	BYOL		SimSiam
				Single-Source	Multi-Source	Single-Source	Multi-Source
Upstream CSSL training time	3 h 37 m	3 h 05 m	3 h 17 m	4 h 12 m	4 h 13 m	4 h 12 m	4 h 14 m
Downstream linear classification time	1 m 17 s	1 m 17 s	1 m 17 s	1 m 17 s	1 m 17 s	1 m 17 s	1 m 17 s

. For single-source methods, we only compared the training time on optical remote-sensing images. It can be seen from Table 2 that the BYOL and SimSiam methods without negative samples had longer training times than the methods with negative samples. The reason for the longer training time of BYOL and SimSiam is that the batch normalization (BN) layer in the MLP network increases the amount of computation, while MoCo, SimCLR and CMC do not need to be based on global BN, so the feedforward part that consumes the most computation is much faster. However, in the method without negative samples, the multi-source method does not consume more time than the single-source method and can obtain better accuracy.

4.2. Comparison on Finetuning Results

We wanted to analyze the influence of the number of images used for finetuning the self-supervised models for our downstream tasks. To do this, we finetuned self-supervised models pretrained on the So2Sat dataset using subsets of 500, 2000, 3500, 5000, and 6654 annotation images from the So2Sat dataset. We can see from Figure 14 that self-supervised models can achieve higher accuracy than the ImageNet pretrained network, especially with a small number of annotation images.

The results in

Table 3. Overall accuracy (%) obtained by BYOL and SimSiam pretrained on different dataset on the downstream test sets, which are finetuned evaluation.

Finetuning Initialization	Test Data
	So2Sat LCZ42-RGB	So2Sat LCZ42-SAR	OpenSAR-Urban	UCMerced_LandUse	AID
BYOL-QXS-SARPORT	94.02	81.15	57.35	93.33	95.75
SimSiam-QXS-SARPORT	92.38	81.69	51.13	83.33	88.92
BYOL-SEN1-2	92.73	81.27	48.52	63.33	83.96
SimSiam -SEN1-2	92.83	81.49	48.70	64.17	83.73
BYOL-So2Sat LCZ42	93.25	81.22	53.35	86.67	90.57
SimSiam-So2Sat LCZ42	93.15	81.21	53.40	88.00	91.98
ImageNet	92.58	80.64	59.07	97.50	95.87
Supervised	92.74	81.09	58.58	79.17	90.80

show that the choice of images used for the training of the BYOL and SimSiam models is crucial to the performance on the downstream tasks if we use the models as feature extractors. The results of the downstream finetuning task on the So2Sat dataset show little difference between different pretraining datasets. We can see that in the case of UCM and AID dataset images, the best results were obtained using neural networks pretrained in a supervised fashion. The QXS dataset has a resolution of 1 m, which is much higher than the 10-m resolution of the SEN1-2 and So2Sat datasets. It is also interesting that those self-supervised models gave good results when trained with images that had similar resolutions as the images in the downstream task. When the spatial resolution between pretraining and downstream images significantly differed, we obtained opposite results. On the downstream task of the OpenSarUrban dataset, resolution was also a key point in the pretrained dataset.

4.3. Visualization

We wanted to verify that the SAR-optical CSSL without negative samples had learned the shared features by visualizing the image features obtained from the convolution layer. We took two SAR-optical image pairs from QXS as the input of the pretraining network to visualize the features extracted after the first convolution layer. Comparing Figure 15b,d with Figure 15c, we can see that the SAR images and optical features extracted by each channel of the convolutional layer in the BYOL and SimSiam methods are the same. We can conclude that features learned by BYOL and SimSiam are shared features of SAR-optical images: BYOL and SimSiam outperformed CMC because the shared features can bring better stability and robustness in the transfer process, as well as bring higher accuracy to optical and SAR single-source image classification tasks. We also see that BYOL and SimSiam pay more attention to texture features instead of background messages, which is beneficial for remote-sensing image scene classification.

4.4. Transfer Task of Optical to SAR Images

In the previous experiments, we found that BYOL and SimSiam could obtain very robust shared features of optical and SAR images and could obtain high accuracy in single-source optical and SAR image classification tasks at the same time. Therefore, we applied the multi-source supervised pretraining network to the transfer task of optical remote-sensing images to SAR images. Since there is a big difference between the data distribution of optical images and SAR images, the common solution is to introduce a domain adaptation algorithm to align the data distribution of the source domain and the target domain. Among the domain adaptation algorithms listed in the

Table 4. Overall accuracy obtained by DAN, DANN and CDAN domain adaption methods based on different pretrained networks.

Initialization Networks	Transfer Methods
	Source-Only	DANN	DAN	CDAN
BYOL	66.04	74.88	77.67	78.60
SIMSIAM	57.61	79.58	80.67	81.45
ImageNet	51.56	75.12	73.03	81.35

, DANN and CDAN are based on adversarial learning. While DAN is based on metric learning, the source-only is the control group that does not adopt the domain adaptive algorithm. In the experiment, we replaced the ImageNet pretraining network with a self-supervised pretraining network to extract the features of images in the source domain and target domain. As seen in Table 4, self-supervised pretraining networks can extract more effective features from the source domain and target domain and eventually improve the transfer classification accuracy of optical to SAR images.

We also analyzed the confusion matrix of classification results for each method, as shown in Figure 16. It can be seen from the results of the confusion matrix that dense trees and water have the highest transfer accuracy. Through horizontal comparison, we found that the self-supervised contrast learning network could assist the domain adaptation algorithm to maximize the classification accuracy of the successfully transferred categories. However, it cannot improve the transfer results of easily confused categories.

5. Conclusions

In this paper, we mainly study the application of self-supervised contrastive learning methods with and without negative samples to multi-source remote-sensing images. We show that if we use a feature extractor trained on the CSSL method as a pretrained network, it outperforms both the ImageNet pretrained network and the purely supervised method. We also verify that the multi-source CSSL method achieves better results than the single-source CSSL method. We prove that CSSL can effectively use the information from SAR-optical remote-sensing images and that the contrastive Siamese network without negative samples can more effectively utilize the consistency information brought by the registered images.

We arrive at the conclusion that the CSSL method without negative samples performs better than the CSSL method with negative samples because it is able to learn the shared features of multi-source remote-sensing images. Shared features of multi-source images learned by upstream self-supervised training bring robustness and stability in downstream tasks, which can be effective for single-source image classification tasks. However, with the finetuning self-supervised network in different datasets downstream, it was constrained by resolution when it was transferring to remote-sensing datasets of different resolutions. We found that in the domain adaptation method of optical transmission to SAR images, using a multi-source CSSL pretrained network without negative samples can greatly improve the transfer accuracy due to the fact that this pretrained network can learn the shared features of optical and SAR images.

Accordingly, for future development, we plan to minimize the finetuning performance degradation caused by data resolution differences for classification tasks and further improve transfer performance from optical images to SAR images.