Deep Contrastive Self-Supervised Hashing for Remote Sensing Image Retrieval

Abstract

Hashing has been widely used for large-scale remote sensing image retrieval due to its outstanding advantages in storage and search speed. Recently, deep hashing methods, which produce discriminative hash codes by building end-to-end deep convolutional networks, have shown promising results. However, training these networks requires numerous labeled images, which are scarce and expensive in remote sensing datasets. In order to solve this problem, we propose a deep unsupervised hashing method, namely deep contrastive self-supervised hashing (DCSH), which uses only unlabeled images to learn accurate hash codes. It eliminates the need for label annotation by maximizing the consistency of different views generated from the same image. More specifically, we assume that the hash codes generated from different views of the same image are similar, and those generated from different images are dissimilar. On the basis of the hypothesis, we can develop a novel loss function containing the temperature-scaled cross-entropy loss and the quantization loss to train the developed deep network end-to-end, resulting in hash codes with semantic similarity preserved. Our proposed network contains four parts. First, each image is transformed into two different views using data augmentation. After that, they are fed into an encoder with the same shared parameters to obtain deep discriminate features. Following this, a hash layer converts the high-dimensional image representations into compact binary codes. Lastly, a novel hash function is introduced to train the proposed network end-to-end and thus guide generated hash codes with semantic similarity. Extensive experiments on two popular benchmark datasets of the UC Merced Land Use Database and the Aerial Image Dataset have demonstrated that our DCSH has significant superiority in remote sensing image retrieval compared with state-of-the-art unsupervised hashing methods.

1. Introduction

In recent years, various remote sensing images have emerged with the rapid development of sensors and aerospace technology [1]. Therefore, searching for desired images from such a big data archive has become a challenging task [2,3]. At present, remote sensing image retrieval, which aims at fulfilling the task in an effective and efficient manner, has received significant attention in the remote sensing community.

Remote sensing image retrieval methods are divided into two paradigms: text-based and content-based image retrieval. In the early days, since text-based image retrieval approaches [4,5] search images by matching text keywords of data, they require domain experts to annotate images with high-quality descriptors in advance. However, remote sensing images have high complexity and cannot be fully represented by only a few keywords. Instead of manually describing each image, content-based image retrieval [6,7] can directly take a raw image as input and automatically extract its visual information.

Generally, content-based image retrieval includes two main steps: feature extraction and similarity ranking. In the first step, we aim to explore an effective algorithm to represent all database RS images and query images by discriminative visual features. In the second step, based on these features, we measure the similarity between the query and each image in the retrieval database and then return a ranked list of similar images. For feature extraction, numerous methods have been proposed to describe image content. For instance, early studies focus on low-level features based on three aspects: spectrum [8], texture [9,10] and shape [11]. These features tend to be sensitive to scale, rotation or illumination, leading to poor retrieval accuracy. Then a variety of encoding methods, such as locally aggregated descriptors [12] and bag-of-visual words [13], have been proposed to improve robustness by aggregating low-level features into holistic representations. All of the above features are usually real-valued vectors with thousands of dimensions in the above methods, resulting in high storage costs. In addition, retrieval speed is inefficient since it is time-consuming to retrieve similar images in the high-dimensional Euclidean space. Both of them would be a barrier in real-world retrieval practice, especially in the era of big data.

To address the above issue, hash-based retrieval methods (also called hashing) have been developed to compress the images into compact binary codes rather than only high-dimensional features. Besides lowering storage costs, short binary representations also shorten the retrieval time due to searching in low dimensional Hamming space rather than high dimensional Euclidean space. The key to hash-based retrieval [2,14,15,16] is to learn hash functions, which need to preserve the similarity relation from image space to Hamming space. In other words, similar images are projected to nearby hash codes, and dissimilar images are far away. Hashing can be generally divided into supervised [17,18,19,20,21,22] and unsupervised [23,24,25,26] based on whether predefined labels are used during training.

Deep learning has achieved dramatic success in various computer vision tasks, such as image classification [27,28,29]. Recently, deep learning has also been introduced to hashing methods for image retrieval, dubbed deep hashing. It also falls into unsupervised and supervised. Deep supervised hashing methods construct an end-to-end supervised deep model to generate hash code by modifying existing convolutional neural networks such as AlexNet, VGG, GoogLenet and ResNet. To make the generated hash codes preserve semantic similarity, it is essential to utilize predefined manual labels as supervision information. Pairwise or triplet-wise labels are the two representative types. For instance, Refs. [21,30,31] provide pairwise labels that show whether two images are similar or dissimilar. Based on this type of label, pairwise cross-entropy loss is developed to train deep networks to preserve semantic similarity. Refs. [32,33,34,35] construct a triplet metric learning deep network with the aid of triplet-wise labels. Each of them contains three images: an anchor image, a positive image and a negative image, and indicates a relative relation: an anchor is more similar to the positive image than the negative.

Even deep supervised hashing methods have achieved promising performance; they are heavily dependent on supervision information, which is scarce and expensive, especially in large-scale data archives. Thus, unsupervised deep hashing, adopting deep neural networks to generate semantically-preserving hash code with unlabeled images, will be the focus of this study. Existing deep unsupervised hashing methods use pre-trained convolutional neural networks that are trained on a natural dataset of ImageNet to extract deep features of images and a non-deep structure for coding them. For instance, Fernandez et al. [26] provided an unsupervised algorithm named probabilistic latent semantic hashing, which adopted a pre-trained network of ResNet-18 to obtain features and then used probabilistic topic models to produce the hash code. It cannot simultaneously learn feature and hash code, resulting in unsatisfactory results. Due to the lack of predefined labels, it is challenging to develop an objective function that forces related/unrelated images to be close/far apart in the Hamming space, thus failing to build a deep hashing network that can be trained end-to-end.

Fortunately, self-supervised contrastive learning, which achieves advanced success in feature representation, has recently provided creative thinking for solving the above problem. It can generate powerful image representations from vast amounts of unlabeled data by creating a deep unsupervised network. In addition, it eliminates the need for a predefined label by maximizing the agreement between different views of the same image. However, contrastive learning cannot be directly applied to hashing since it focuses on learning continuous features rather than binary codes. The continuous features are high-dimensional and real-valued, while our aim of binary codes must be short and binary (each element is either −1 or +1).

In this paper, we propose a deep unsupervised hash learning method for remote sensing image retrieval by modifying contrastive learning, namely deep contrastive self-supervised hashing (DCSH). Using only unlabeled images, it can build a deep network with end-to-end training. Motivated by contrastive learning, we assume that the two hash codes for different views generated from the same image are similar, while the two hash codes for views generated from different images are not similar. Based on the hypothesis, we design a new loss function to preserve the generated hash codes with semantic similarity and train the unsupervised network end-to-end. In the proposed framework, four parts are included in the training process, as shown in Figure 1. First is the data augmentation part; each image is randomly transformed into two different views. Secondly is the encoder part; the augmented images are fed into an encoder network with shared parameters to obtain deep discriminative features. Thirdly, a hash layer is used to convert the high-dimensional image representations into low-dimensional compact binary codes. Lastly, a novel contrastive loss function is introduced to train the proposed network end-to-end and thus guide the generated hash codes with semantic similarity. In summary, there are three main contributions in this paper:

• We present a novel deep unsupervised hashing method for remote sensing image retrieval. Motivated by contrast learning, we hypothesize that the hash codes for different views generated from the same image are similar, while the hash codes for views generated from different images are not similar. To the best of our knowledge, we are the first to implement this idea for remote sensing image retrieval. According to the hypothesis, we can build a deep unsupervised hash network with end-to-end training, which can learn discriminative hash codes from a large number of unlabeled data. It avoids the problem of annotating images compared to most deep hashing algorithms.
• We introduce a novel hashing objective loss to train our deep network. This gives each image a more effective hash code, improving the efficiency of image searching. Additionally, instead of the conventional relaxation strategy, we propose a continuous strategy that converges a non-differentiable sign function using a sequence of differentiable functions, allowing us to explicitly enforce binary constraints on hash codes.
• In contrast to existing unsupervised hashing methods for the retrieval of remote sensing images, we achieve the state-of-the-art results on benchmark datasets of the UC Merced Land Use Database and the Aerial Image Dataset.

The article is arranged in the following manner. In Section 2, the related work is reviewed. Section 3 gives a detailed description of our proposed unsupervised deep hashing method. Section 4 presents extensive experimental results on two popular benchmark datasets of the UC Merced Land Use Database and the Aerial Image Dataset. Section 5 discusses several hyper-parameters and shows the retrieval results of multiple methods in a visual manner. Section 6 gives a conclusion of this research.

2. Related Work

Unsupervised hashing aims at encoding images into data-aware binary codes with only unlabeled images, and it is not well-developed in the remote sensing community. The existing methods are limited and mainly contain two modules. The first step is the feature learning module. Each remote sensing image is described by a visual representation. The second step is the hash learning module. The visual feature is fed into an unsupervised hashing algorithm to produce a hash code. For instance, in [36], Demir et al. proposed a kernel-based unsupervised locality-sensitive hashing method, which characterized each image by a bag-of-visual-words feature and then encoded it into hash code by a set of hash functions learned in kernel space. Later, Li et al. [37] presented a partial randomness hashing approach, which first adopted the hand-crafted holistic descriptor [38] with 512-dimension as visual representation, and then constructed hash functions based on the framework approximated auto-encoder structure [39]. Unlike a traditional auto-encoder, only parameters on the decoder require optimization. Its parameter strategy, partial randomness and partial training makes hash function construction efficient and effective. In [25], Reato et al. represented each image by multi-hash codes instead of single-hash codes. More specially, it described images by primitive-sensitive clusters and then encoded them into multi-hash codes by kernel-based unsupervised locality-sensitive hashing, at the cost of more hash bits to represent each image.

However, the above-mentioned methods have not achieved satisfactory retrieval performance. First, the feature learning module produces low-level hand-crafted features primarily based on textures without sufficient semantic information. Secondly, the modules of feature learning and hash learning are separated without interaction with each other. Thus, hash codes of the above approaches are incompetent when describing the complex semantic content of the remote sensing image. In order to embed more semantics into hash codes, Fernandez et al. [26] provided an unsupervised algorithm named probabilistic latent semantic hashing, which adopted pre-trained ResNet-18 as a feature learning module and added an extra probabilistic topic model. As proven in various remote sensing imagery tasks, the intermediate outputs of pre-trained deep convolutional neural networks often have stronger capability than hand-crafted descriptors on the characterization of high-level semantic information of RS images. In addition, probabilistic topic models [40] can further extract higher-level semantic content via uncovering hidden semantic patterns from visual features in feature learning modules. However, Ref. [26] may still suffer from two limitations. On the one hand, when the pre-trained convolutional neural network poorly generalizes in the target dataset, the pre-extracted deep visual feature may lead to inferior performance. On the other hand, the feature learning module is still independent of the hash learning module. In other words, when optimizing the loss function of the hash learning module, the feature vectors are fixed and unable to be optimized simultaneously. Thus, visual features from semantically dissimilar/similar image pairs might not have a relatively satisfactory distance to well preserve the similarity. To solve the above two problems, in our paper, we aim to create a deep unsupervised network that can simultaneously optimize feature and hash code learning with only unlabeled images.

3. Proposed Method

Given a training set of N images $\mathit{X}=\left\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_N\right\}$ without any labels, for the similarity retrieval task, we expect to learn an appropriate hash function that maps each image ${\mathit{x}}_i$ into a compact K-bit binary codes ${\mathit{b}}_i\in {\left\{-1,1\right\}}^K$, where N denotes the number of images from the training set and K represents the length of the binary codes. Additionally, the hash function should be such that semantically similar images are projected to nearby hash codes while dissimilar images are encoded far away in the Hamming space. Then, image retrieval, searching images semantically similar to a query image, can be efficiently fulfilled by Hamming distance computed with bit-wise XOR operations.

In our paper, we propose a novel deep end-to-end framework to conduct unsupervised hash learning, as shown in Figure 1. Firstly, in Section 3.1, each image in the sampled mini-batch training images would be transformed into two different views by different data augmentation strategies. After augmentation, we set two views from the same image as the positive pairs and two views from different images as negative pairs. Secondly, we utilize the encoder module (Section 3.2) to extract the discriminative deep feature representation and then the hashing layer (Section 3.3) to transform the feature vectors to corresponding hash codes. Finally, the novel loss function is proposed in Section 3.4 to train our network. In Section 3.5, we give the optimizing strategy for our proposed network.

3.1. Data Augmentation

Generally, to equip learned hash code with similarity-preserving properties, the supervised hash learning [31,41,42] needs to leverage the label information to construct similar relations for the training data as supervised information. Specifically, two points are considered similar if they have at least one label in common; otherwise, they are dissimilar. However, due to the absence of label annotations, unsupervised hashing methods [37,43,44,45] can not model the similarity relations of data points by this means.

To address this problem, inspired by SimCLR [46], we introduce the data augmentation strategy to generate similar/dissimilar pairs for the training dataset. Specifically, we consider that two augmented views generated from the same image are similar, and inversely, two views produced from different images are dissimilar. We use common visual transformations to augment images. More specifically, we sequentially apply the following data augmentation techniques; that is, random cropping and resizing, random horizontal flipping, random color jittering, random grayscale, and random Gaussian blur, as shown in Figure 2.

Given the original image ${\mathit{x}}_i$, we get two augmented views ${\mathit{v}}_i^{\left(1\right)}$ and ${\mathit{v}}_i^{\left(2\right)}$ by randomly selecting two different augmented transformations $\varphi$ and $\widehat{\varphi }$ presented in Figure 2, which can be modeled by the following expression:

$$\begin{array}{cc}\hfill & {\mathit{v}}_i^{\left(1\right)}=\varphi \left({\mathit{x}}_i\right)\hfill \\ \hfill & {\mathit{v}}_i^{\left(2\right)}=\widehat{\varphi }\left({\mathit{x}}_i\right)\hfill \end{array}$$

(1)

3.2. Encoder

Recently, deep convolutional neural networks (CNNs) have shown powerful feature learning capability. Therefore, we tailor one of them to our encoder, which aims to project each input augmented image into discriminative representation. There exist various well-known neural networks, such as AlexNet [47], VGG [48], GoogLeNet [49], and ResNet [50]. Among them, ResNet-based networks have achieved remarkable performance in diverse computer vision tasks, so we extend our encoder network from ResNet-50, which has a convolutional layer conv1, four residual blocks conv2_x–conv5_x, a global average pooling layer, and a fully connected layer (classifier layer). We remove all layers after the global average pooling layer, and the detailed structural information of the encoder is shown in

Table 1. The Architecture for the encoder.

Layer Name	Output Size (Width × Height × Channel)	Configuration
conv1	112 × 112 × 64	7 × 7, 64, stride 2
conv2_x	56 × 56 × 256	3 × 3 max pool, stride 2
		$\left[\begin{array}{c}1\times 1,64\\ 3\times 3,64\\ 1\times 1,256\end{array}\right]$× 3
conv3_x	28 × 28 × 512	$\left[\begin{array}{c}1\times 1,128\\ 3\times 3,128\\ 1\times 1,512\end{array}\right]$× 4
conv4_x	14 × 14 × 1024	$\left[\begin{array}{c}1\times 1,256\\ 3\times 3,256\\ 1\times 1,1024\end{array}\right]$× 6
conv5_x	7 × 7 × 2048	$\left[\begin{array}{c}1\times 1,512\\ 3\times 3,512\\ 1\times 1,2048\end{array}\right]$× 3
global average pooling layer	2048	global average pooling

. Given an augmented image ${\mathit{v}}_i^{\left(c\right)}$, we can get a deep feature vector ${\mathit{e}}_i^{\left(c\right)}$ with 2048-dimension, which can be formulated as:

$${\mathit{e}}_i^{\left(c\right)}=E_n\left({\mathit{v}}_i^{\left(c\right)};{\theta }_e\right)$$

(2)

where ${\theta }_e$ denotes the encoder parameters, and $E_n$ is our encoder network.

3.3. Hashing Layer

The hashing layer aims to compress high-dimensional continuous vectors $\mathit{e}$ into low-dimensional compact binary hash codes $\mathit{b}$, resulting in efficient retrieval and storage. We use two steps to attain this goal. The first step is feature reduction. In this paper, we adopt Multi-Layer Perception (MLP) with two fully connected layers to compress the 2048-dimensional feature vector to hash-like representation $\mathit{z}$ with K-dimension, where K is the length of hash codes. The process is shown as follows:

$$\mathit{z}=f_h\left(\mathit{e};W\right)=W^{\left(2\right)}\sigma \left(W^{\left(1\right)}\mathit{e}\right)$$

(3)

where $W^{\left(1\right)}$ and $W^{\left(2\right)}$ denote parameters of the two fully connected layers, respectively, and $W=\left[W^{\left(1\right)};W^{\left(2\right)}\right]$, as shown in Figure 3a, $\sigma$ is a non-linear function of rectified linear units (ReLU); that is,

$$\sigma \left(x\right)=max\left(0,x\right)$$

(4)

Note that $\mathit{z}$ in Equation (3) is still continuous rather than binary. Thus the second step is to quantize the continuous vector into a binary code by an activation function, which is generally performed by the element-wise sign function. The function can be formulated as:

$$\begin{array}{c}\hfill \mathit{b}=sgn\left(z\right)=\left\{\begin{array}{cc}+1,\hfill & z\ge 0\hfill \\ -1,\hfill & z<0\hfill \end{array}\right.\end{array}$$

(5)

3.4. Loss Function

The goal of our proposed framework is to produce binary hash codes of the input images. The generated hash codes should meet two constraints. The first one is preserving similarity. We satisfy this limit by the normalized temperature-scaled cross-entropy loss called NT-Xent loss due to their consistent aim of semantic preservation, i.e., pulling similar images together and pushing dissimilar ones apart in embedding space. The second one is quantization loss, which forces embedded features to be −1 or +1.

For a minibatch image ${\left\{{\mathit{x}}_i\right\}}_{i=1}^M$ randomly sampled from the training set, we can get the corresponding M pairs of binary codes ${\left\{\left({\mathit{b}}_i^{\left(1\right)},{\mathit{b}}_i^{\left(2\right)}\right)\right\}}_{i=1}^M$. We apply NT-Xent loss on all M pairs; that is,

$$L_{NT-Xent}=\frac{1}{M}\sum _{i=1}^{M}l\left(i\right)$$

(6)

where $l\left(i\right)$ denotes an expression of loss function on i-th pair, which aims at forcing related pairs $\left\{{\mathit{b}}_i^{\left(1\right)},{\mathit{b}}_i^{\left(2\right)}\right\}$ from the same image ${\mathit{x}}_i$ close and unrelated pairs ${\left\{{\mathit{b}}_i^{\left(c\right)},{\mathit{b}}_j^{\left(k\right)}\right\}}_{j\ne i;c=1,2;k=1,2}$ from different images are far away. The definition is as follows:

$$\begin{array}{cc}\hfill l\left(i\right)& =-\frac{1}{2}\sum _{c=1,2}log\frac{exp\left(sim\left({\mathit{b}}_i^{\left(1\right)},{\mathit{b}}_i^{\left(2\right)}\right)/\tau \right)}{exp\left(sim\left({\mathit{b}}_i^{\left(1\right)},{\mathit{b}}_i^{\left(2\right)}\right)/\tau \right)+{\sum }_{j\ne i}{\sum }_{k=1,2}exp\left(sim\left({\mathit{b}}_i^{\left(c\right)},{\mathit{b}}_j^{\left(k\right)}\right)/\tau \right)}\hfill \\ \hfill & =\frac{1}{2}\sum _{c=1,2}\left[log(1+\sum _{j\ne i}\sum _{k=1,2}exp\left(\frac{sim(b_i^{\left(c\right)},b_j^{\left(k\right)})-sim(b_i^{\left(1\right)},b_i^{\left(2\right)})}{\tau }\right))\right]\hfill \end{array}$$

(7)

where $\tau$ is a temperature parameter, and $sim\left(·,·\right)$ is a metric function to measure the pairwise similarity of two binary codes. It must be such that the smaller Hamming distance between two binary codes, the larger the output of $sim\left(·,·\right)$. In order to meet this condition, we choose the cosine similarity:

$$sim\left({\mathit{b}}_i,{\mathit{b}}_j\right)=\frac{{\mathit{b}}_i^T{\mathit{b}}_j}{∥{\mathit{b}}_i∥∥{\mathit{b}}_j∥}\in \left[-1,1\right]$$

(8)

where $∥·∥$ denotes the L2-norm, since there is a relationship between the Hamming distance $dis{t}_H\left({\mathit{b}}_i,{\mathit{b}}_j\right)$ and cosine similarity $sim\left({\mathit{b}}_i,{\mathit{b}}_j\right)$: $dis{t}_H\left({\mathit{b}}_i,{\mathit{b}}_j\right)=\frac{K}{2}\left(1-sim\left({\mathit{b}}_i,{\mathit{b}}_j\right)\right)$.

Equation (7) resembles softmax loss. It can shorten the similarity distance of positive pairs $\left({\mathit{b}}_i^{\left(1\right)},{\mathit{b}}_i^{\left(2\right)}\right)$ and enlarge the similarity distance of $2\left(M-1\right)$ negative pairs $\left\{\left({\mathit{b}}_i^{\left(c\right)},{\mathit{b}}_j^{\left(k\right)}\right)\right\}$ where $j\ne i;k=1,2;c=1,2$. Thus, compared to the triplet loss [51] with only one negative example, each sample in our loss function (Equation (7)) can interact with multiple negative examples, which eases the problem of slow convergence and poor local optima [52].

Based on Equation (7), when $l\left(i\right)\to 0$, we get $sim(b_i^{\left(c\right)},b_j^{\left(k\right)})-sim(b_i^{\left(1\right)},b_i^{\left(2\right)})\to -2$, where $\left(b_i^{\left(c\right)},b_j^{\left(k\right)}\right)$ is a dissimilar pair, and $\left(b_i^{\left(1\right)},b_i^{\left(2\right)}\right)$ is a similar pair. Thus the loss function used in this paper can make the hash codes of similar pairs close and the hash codes of dissimilar pairs far away.

3.5. Optimization by Continuation

Note that Equation (7) can not be optimized directly since the binary codes are produced by the sign function (Equation (5)) that suffers from the vanishing gradient problem. Therefore, our proposed deep hashing network fails to be trained by the back-propagation algorithm [53,54,55] in a truly end-to-end manner. However, numerous research studies [42,56,57,58] have shown that end-to-end training significantly improves the retrieval accuracy of deep hashing.

To train our framework end-to-end, we must address the binary constraint, and there are two kinds of methods. The first kind is an iterative alternating optimization algorithm [59,60], which alternately optimizes continuous vector $\mathit{e}$ and binary hash codes $\mathit{b}$ while fixing the other. That is to say, it skips the ill-posed gradient challenge caused by the sign function but substantially weakens the deep hashing model flexibility. The second kind is a widely-used relaxation scheme [61,62,63,64,65], which relaxes the discrete constraint $\left\{-1,+1\right\}$ to a continuous interval $\left[-1,+1\right]$ by seeking a smooth function (sigmoid or tanh) to approximate the sign function. Most studies [64,65,66] adopt a scaled tanh function $tanh\left(\beta x\right)$, where $\beta$ is a hyper-parameter controlling the trade-off between the function smoothness and the binary quantization error. Therefore, choosing optimal $\beta$ is vital yet difficult [64,65,66] because a small value for $\beta$ tends to a large quantization loss, while a large value for $\beta$ tends to an almost vanishing gradient problem similar to the sign function. Hence, we adopt a novel way to solve the binary constraint.

According to the verification of recent works [67,68], the complex problem of non-smooth optimization can be simplified. Specifically, as shown in Figure 3b, we gradually increase the non-smoothness level $\beta$ in $tanh\left(\beta \mathit{z}\right)$, which would converge to the sign function optimization problem. This method is effective due to a nice relationship between the $sgn\left(.\right)$ and $tanh\left(.\right)$ function:

$$sgn\left(\mathit{z}\right)={lim}_{\beta \to \infty }tanh\left(\beta \mathit{z}\right)$$

(9)

Therefore, in the training process, to use standard back-propagation to train our deep hashing network in an end-to-end manner, we adopt a sequence of scaled tanh as our activation function instead of sign function, that is:

$$\mathit{h}=tanh\left(\beta \mathit{z}\right)$$

(10)

where $\mathit{h}$ denotes a K-dimensional real-valued feature vector before converting to binary (termed as a hash-like feature) and $\beta$ changes gradually.

$$sgn\left(\mathit{z}\right)={lim}_{\beta \to \infty }tanh\left(\beta \mathit{z}\right)$$

(11)

The details of the training process are as follows. We start with $\beta =1$ in Equation (10). For each stage, after the network converges in the last stage, we enlarge $\beta$ and initialize the current stage parameters with the parameters of the last converged network. By involving $tanh\left(\beta x\right)$ with $\beta \to \infty$, the network obtains the same results as adopting $sgn\left(x\right)$, which can produce exact binary hash codes as we promise. When $\beta$ is increased to 10, we can already obtain expectant convergence. Therefore, for each stage, we use the hash-like feature h instead of binary hash code b, and Equation (7) can be rewritten as:

$${\widehat{l}}_{NT-Xent}\left(i\right)=-\frac{1}{2}\sum _{c=1,2}log\frac{exp\left(sim\left({\mathit{h}}_i^{\left(1\right)},{\mathit{h}}_i^{\left(2\right)}\right)/\tau \right)}{exp\left(sim\left({\mathit{h}}_i^{\left(1\right)},{\mathit{h}}_i^{\left(2\right)}\right)/\tau \right)+{\sum }_{j\ne i}{\sum }_{k=1,2}exp\left(sim\left({\mathit{h}}_i^{\left(c\right)},{\mathit{h}}_j^{\left(k\right)}\right)/\tau \right)}$$

(12)

Quantization Loss: In order to accelerate the approximation process between the hash-like code $\mathit{h}$ and the binary code $\mathit{b}$, we add quantization loss:

$$l_Q\left(i\right)=\frac{1}{2}\left({∥\left|{\mathit{h}}_i^{\left(1\right)}\right|-\mathbf{1}∥}^2+{∥\left|{\mathit{h}}_i^{\left(2\right)}\right|-\mathbf{1}∥}^2\right)$$

(13)

where $∥·∥$ and $\left|·\right|$, respectively, denote the L2-norm and element-wise absolute value operation. The quantization loss on the i-th positive pair aims at pulling the element of hash-like features to desired discrete values $\{-1,+1\}$.

Total Loss: We combine NT-Xent loss (Equation (12)) and quantization loss (Equation (13)) to get the total loss function:

$$L_{total}=\frac{1}{M}\sum _{i=1}^M\left({\widehat{l}}_{NT-Xent}\left(i\right)+\alpha l_Q\left(i\right)\right)$$

(14)

where $\alpha$ is a weighting parameter to balance the two terms. Algorithm 1 is the main learning procedure of our proposed method.

Algorithm 1 DCSH’s main learning algorithm.

Require: batch size M, initial and trainable encoder parameters ${\theta }_e$, initial and trainable hash layer parameters $\mathit{W}$, hyper-parameter of temperature $\tau$ and weighting parameter $\alpha$ 1: for sampled mini-batch ${\left\{{\mathit{x}}_i\right\}}_{i=1}^M$ do 2:    for i in $\left\{1,2,\dots ,M\right\}$ do 3:      draw two different augmented transformation $\varphi$ and $\widehat{\varphi }$, then get a pair of two related images from the same image $x_i$: 4:      ${\mathit{v}}_i^{\left(1\right)}=\varphi \left({\mathit{x}}_i\right)$ 5:      ${\mathit{v}}_i^{\left(2\right)}=\widehat{\varphi }\left({\mathit{x}}_i\right)$ 6:      learn deep image representation by Encoder: 7:      ${\mathit{e}}_i^{\left(1\right)}=E_n\left({\mathit{v}}_i^{\left(1\right)};{\theta }_e\right)$ 8:      ${\mathit{e}}_i^{\left(2\right)}=E_n\left({\mathit{v}}_i^{\left(2\right)};{\theta }_e\right)$ 9:      get hash-like feature ${\mathit{h}}_i^{\left(1\right)}$ and ${\mathit{h}}_i^{\left(2\right)}$ via hashing layer of Equation (3) and activation function of $tanh$ described in Equation (10): 10:      ${\mathit{z}}_i^{\left(1\right)},{\mathit{z}}_i^{\left(2\right)}=f_h\left({\mathit{e}}_{\mathit{i}}^{\left(\mathbf{1}\right)};W\right),f_h\left({\mathit{e}}_{\mathit{i}}^{\left(\mathbf{2}\right)};W\right)$ 11:      ${\mathit{h}}_i^{\left(1\right)},{\mathit{h}}_i^{\left(2\right)}=tanh\left(\beta {\mathit{z}}_i^{\left(1\right)}\right),tanh\left(\beta {\mathit{z}}_i^{\left(2\right)}\right)$ 12:    end for 13:    for i in $\left\{1,2,\dots ,M\right\}$ do 14:      compute NT-Xent loss on i-th related pair ${\widehat{l}}_{NT-Xent}\left(i\right)$ via Equation (12) and quantization loss on i-th related pair $l_Q\left(i\right)$ via Equation (13) 15:    end for 16:    total M pairs loss: $L_{total}=\frac{1}{M}{\sum }_{i=1}^M\left({\widehat{l}}_{NT-Xent}\left(i\right)+\alpha l_Q\left(i\right)\right)$ 17: end for

Inference Process: Note that, in the inference process, we get the desired discrete code by binarizing the hash-like feature h produced by Equation (10), i.e.,

$$\mathit{b}=sgn\left(\mathit{h}\right)$$

(15)

where $sgn$ is a point-wise function defined by Equation (5).

4. Experiments

To verify the effectiveness of our proposed DCSH, we perform extensive retrieval experiments on two popular benchmark databases, the UC Merced Land Use Database (UCMD) and the Aerial Image Dataset (AID).

4.1. Datasets

The performance of our proposed method is evaluated on two remote sensing benchmark datasets. We briefly introduce these datasets as follows.

• UCMD [69]: It is publicly free and provided by the University of California, which collected surface images from the United States national city map produced by the United States Geological Survey. This dataset comprises 21 challenging land cover concepts, where each concept consists of 100 images of size $256\times 256$ with a spatial resolution of 0.3 m per pixel. We show images from UCMD in Figure 4.
• AID [70]: It is a large-scale remote sensing publicly available dataset gathered by Wuhan University from Google Earth imagery. The images are completely annotated by specialists in the remote sensing image interpretation field. By contrast with the UC Merced dataset, AID is significantly more challenging. Specifically, the dataset comprises a total of 10,000 aerial images with a fixed size of $600\times 600$ pixels and a spatial resolution varying from 8 m to about 0.5 m. It is categorized into 30 land-use scene classes such as airport, bare land, dense residential, desert, and so on; each class contains a number of images ranging from 220 to 420. Furthermore, the AID is derived from diverse remote sensing imaging sensors and chosen from different countries, under various times and distinct seasons, which makes the intra-class diversity larger and inter-class dissimilarity smaller as well. We show images from AID in Figure 5.

4.2. Evaluation Protocols

To perform the quantitative evaluation of image retrieval, we adopt three standard evaluation metrics: precision-recall curves (PR-curves) [71], mean average precision (MAP), and precision within different numbers of top retrieved examples. For the query database, the MAP is computed as follows:

$$MAP=\frac{1}{\left|Q\right|}\sum _{i=1}^{\left|Q\right|}A{P}_i$$

where $\left|Q\right|$ denotes the total number of query databases, and $A{P}_i$ is the average precision of query $x_i$ based on ranking, i.e.,

$$A{P}_i=\frac{1}{N_{+}}\sum _{k=1}^n\frac{N_{+}^k}{k}\times pos\left(k\right)$$

where $N_{+}$ is the total number of images related to query $x_i$ in the retrieval database, n denotes the number of images in the retrieval database, $N_{+}^k$ is the number of related examples within the top k retrieved examples, and $pos\left(k\right)$ is an indicator function and indicates if the returned image at position k is related to the query. $pos\left(k\right)=1$ if the relationship is relevant, otherwise $pos\left(k\right)=0$.

4.3. Implementation Details

We implement our DCSH method using the open source Pytorch (https://pytorch.org/) on Ubuntu 16.04 with 2 NVIDIA GeForce GTX TITAN X GPUs. The DCSH method contains four main components: data augmentation, encoder, hashing layer, and contrastive loss function. We conduct the experiment on two benchmark datasets, UCMD and AID. We randomly select a mini-batch of images from the training dataset; the selected images are first resized into $224\times 224\times 3$ as the input. The data augmentation module randomly transforms each image into two different views with the size of $224\times 224\times 3$. Then the augmented mini-batch training set will be fed into our encoder network. The architecture of the encoder is shown in Table 1; we construct the encoder network from the original ResNet-50 by removing all layers after the global average pooling layer. We initialize it with the pre-trained ResNet50, which has been well-trained on the ImageNet dataset [72]. The hash layer contains two fully connected (FC) layers; the first FC layer has 1024 neurons with ReLU activation function, and the second FC layer has K neurons (the length of hash code) and deploys scaled tanh function as the activation function. In addition, the hash layer is randomly initialized with zero-mean Gaussian distribution with a standard deviation of 0.01. We use Adam (Adaptive moment estimation) [73] as the optimizer in our experiment; the learning rates of the encoder network and the hash layer are set to ${10}^{-5}$ and ${10}^{-4}$, respectively. The training epochs for UCMD and AID are set to 100 and 120, respectively. The hyper-parameters of batch size and the temperature $\tau$ are set to 64 and 0.3, respectively. The detailed analysis of these settings will be further investigated in Section 5. The weighting parameter $\alpha$ is set to 1.

4.4. Comparative Experiments with State-of-the-Art Methods

We compare our proposed unsupervised deep hashing framework for remote sensing image retrieval with six state-of-the-art methods, including three unsupervised hashing methods specifically designed for remote sensing, i.e., kernel-based unsupervised hashing (KULSH) [36], partial randomness hashing (PRH) [37] and probabilistic latent semantic hashing (PLSH) [26]; and three representative hashing methods for natural images widely used in computer vision, i.e., density sensitive hashing (DSH) [74], locality sensitive hashing (LSH) [75] and iterative quantization (ITQ) [59]. For a fair comparison, all approaches are unsupervised hashing without utilizing any label information in the training stage, which is vital in practical application since the label annotation is highly labor-intensive and prone to errors.

According to the settings in Shan’s recent paper [14], we take both hand-crafted features of 512-dimensional GIST descriptors [38] and fully-connected features with 4096 dimensions of a pre-trained VGG16 as input for all these baseline methods and use ‘-CNN’ and ‘-GIST’ as the notation, respectively. Then, we demonstrate the experimental results of all the compared methods and our proposed method under the indicators above on UCMD and AID datasets. Moreover, we conduct several ablation analyses and discussions on two significant hyper-parameters appearing in our proposed method in Section 5.

4.4.1. Results on UCMD

For the UCMD dataset, there are 21 classes and each class consist of 100 images. We randomly choose 80 images for every class as training data (1680 images), and in the retrieval database, the remaining images (420 images in total) are set as the test queries.

Table 2. The mean average precision (MAP) retrieval results on the UCMD dataset; we give separate hash retrieval results for 16, 32, 48, 64 bits. The testing set has 420 images (each category randomly takes 20% of the images).

Method	Hash Code Length
	16-bits	32-bits	48-bits	64-bits
DSH-GIST [74]	16.38	17.09	19.24	19.18
LSH-GIST [75]	17.55	18.25	19.78	21.46
ITQ-GIST [59]	19.35	20.45	20.89	20.78
KULSH-GIST [36]	28.37	33.56	34.98	34.16
PRH-GIST [37]	31.77	33.38	35.76	36.92
PLSH-GIST [26]	40.49	44.17	47.32	46.37
DSH-CNN	28.35	33.90	34.24	34.66
LSH-CNN	32.44	38.58	45.48	51.67
ITQ-CNN	42.38	45.99	47.28	47.49
KULSH-CNN	53.68	55.37	58.23	64.58
PRH-CNN	55.39	59.45	61.89	67.77
PLSH-CNN	62.28	65.35	70.44	73.07
OUR	75.80	78.17	80.49	80.09

shows the MAP scores obtained by our approach and other baseline methods for hash codes with different lengths on the UCMD dataset. First, we can observe a significant superiority of our proposed method over the baseline methods in all hash code lengths. To be specific, the MAP score of our method shows a large improvement of 7.02–13.52% compared to the second-best competitor, PLSH-CNN. In addition, the methods specially designed for remote sensing images, such as KULSH, PRH and PLSH, outperform the methods for natural images; this phenomenon is also observed in a recent paper [14]. Third, all the ‘CNN’-based hash methods outperform their ‘GIST’-based versions, which indicates the CNN feature is more discriminative than the hand-crafted feature of GIST descriptors under the same hash method. Finally, even in general, the MAP score increases when the length of the hash code increases; the MAP score will sometimes decline when the length of the hash code reaches a certain value, such as 64-bits, as shown in Table 2.

In addition to the MAP indicator, we demonstrate two important evaluation metrics to further validate the effectiveness of our proposed method, i.e., precision curves and precision-recall curves. Figure 6 shows the UCMD retrieval results of precision curves with respect to a different number of retrieved images for different methods. As shown in Figure 6a,b, our proposed method outperforms all baseline methods by significant margins on the precision curve with 32 and 64 bits, respectively. Compared to methods with 32 bits hash codes, the same methods with 64 bits hash codes have small gains in precision with the same number of returned images, which indicates that appropriately increasing the length of hash codes can contribute to improving the retrieval performance. Figure 7 reveals the precision-recall curves by ranking on Hamming distance. From Figure 7a,b, we can find that our method obtains the best performance on both 32 and 64 bits, respectively. Therefore, compared to the state-of-art algorithm of PLSH, we obtain higher precision with the same recall, which is desirable for the remote sensing image retrieval task.

4.4.2. Results on AID

The AID dataset consists of 30 classes, and each class contains a number of images ranging from 220 to 420. We randomly choose 50% of the samples from every class as training data (5000 images) and retrieval database, and the remaining images (500 images in total) are set as the test queries. Each image has a fixed size of $600\times 600$, and we resize it to $256\times 256$.

Table 3. MAP retrieval results on the AID dataset; we give separate hash retrieval results for 16, 32, 48, 64 bits. The testing set has 5000 images (each category randomly takes 50% of the images).

Method	Hash Code Length
	16-bits	32-bits	48-bits	64-bits
DSH-GIST [74]	9.42	9.87	10.06	10.35
LSH-GIST [75]	10.53	10.89	12.74	13.77
ITQ-GIST [59]	9.67	10.49	11.74	11.91
KULSH-GIST [36]	11.43	13.46	14.63	15.86
PRH-GIST [37]	13.42	15.37	15.74	17.26
PLSH-GIST [26]	13.26	17.32	18.64	18.99
DSH-CNN	16.25	18.74	19.63	19.57
LSH-CNN	25.44	29.79	35.58	40.66
ITQ-CNN	23.28	27.29	28.74	29.65
KULSH-CNN	38.47	41.37	46.64	50.62
PRH-CNN	43.89	47.58	50.27	55.74
PLSH-CNN	48.72	52.35	55.83	60.14
OUR	66.17	70.04	71.48	73.83

shows the MAP scores for hash codes with different lengths ranging from 16 to 64 bits on the AID dataset. First, as shown in Table 3, our method surpasses all baseline methods. Compared to the second-best competitor PLSH-CNN, MAP indicates a large advantage of 13.69–$17.69\%$. Second, the methods specifically designed for remote sensing images are significantly superior to the methods for natural images such as DSH, LSH and ITQ. Third, all the ‘CNN’-based hash methods outperform their ‘GIST’-based versions, which indicates that the CNN feature is a more discriminant feature than hand-crafted features of GIST descriptors under the same hash method on the AID dataset. For example, KULSH-CNN gets more than $27\%$ than KULSH-GIST at all hash code lengths. Finally, even in general, the MAP score increases when the length of the hash code increases, the MAP score will sometimes decline when the length of the hash code reaches a certain value, such as 64-bits, as shown in Table 3.

Figure 8 shows the AID retrieval results of precision curves with respect to the different numbers of retrieved images for different methods. As shown in Figure 8a,b, the proposed method outperforms all baseline methods by significant margins on precision curves with 32 and 64 bits, respectively. Compared to methods with 32-bit hash codes, the same methods with 64 bits hash codes have a small gain in the precision with the same number of returned images, which indicates that an appropriate increase in the length of hash codes would contribute to the promotion of the retrieval performance.

Figure 9a,b reveal the precision-recall curves by ranking the Hamming distance. Since high precision corresponding to high recall has been a sign of a desirable retrieval algorithms, our method is advantageous over other compared methods on both 32 bits (red line) and 64 bits (red line), respectively.

5. Discussion

5.1. The Analysis and Setting of Temperature $\tau$

Similar to previous work [46,76], $\tau$ in Equation (7) is used to scale the sensitivity of the loss function, which plays an important role in our unsupervised contrastive learning on remote sensing image retrieval. In view of this, we further make an ablation analysis to examine the influence of hyperparameter $\tau$ on MAP scores on the UCMD dataset. We fix the batch size to 64 and evaluate the MAP score by varying $\tau$ from 0.1 to 1. From Figure 10, it is inspiring that the model trained with the optimal temperature can improve performance by nearly 10%. More specifically, with the increase in $\tau$, the MAP score first shows a sharp increase before $\tau =0.3$ and then remains at the top level from $\tau =0.3$ to $\tau =0.6$, then is followed by a moderate drop afterward. The best two values of the MAP score (0.8049 and 0.8051) are almost equal. Since recent research shows that a smaller temperature can increase the model’s penalty on difficult negative examples [77] and then generate more discriminative image features and hash codes, we set $\tau =0.3$.

5.2. The Analysis and Setting of Training Batch Size

As can be seen in the detail of training implements, our proposed method needs to set the proper batch size to train an optimal remote sensing retrieval model. In order to evaluate the effect of the batch size in the training process, we conduct several experiments to demonstrate the retrieval MAP metric under various batch sizes. Due to the conclusion that in the original feature space larger batch size can benefit the contrastive learning [46], we also design the batch size from small to large to study whether it still has this property in the Hamming space. Figure 11 shows the influence of batch size when the model is trained for different numbers of epochs, and this parameter setting makes a vast difference in getting excellent retrieval performance. We can observe that when the batch size is too small or too large, the performance is not optimal. On the other hand, as the batch size and epochs increase to a proper value, the performance rises obviously and then converges to the maximum MAP.

5.3. Visualization Study

To further emphasize the effectiveness of our proposed method intuitively, we illustrate the quantitative top-10 matching results and the number of correctly returned images out of the top-60 results of the query samples, respectively, from the UCMD dataset and AID dataset in Figure 12 and Figure 13. The query images are all randomly sampled in the related dataset, and the returned retrieval results are ranked by the Hamming distances between the query image and the images in the retrieval database. We select three competitive baseline methods, KULSH-CNN, PRH-CNN and PLSH-CNN, to compare with our proposed method in terms of the visual, quantitative retrieval results.

As shown in Figure 12 and Figure 13, the first column displays the query image, and the remaining columns present the retrieval results for the different baseline methods. The red boxes indicate the false positive data points. Particularly, in view of the MAP results obtained by different hash bits presented in Table 2 and Table 3, we can observe that the 48-bit hash codes can achieve the best result of 80.49% in the UCMD dataset and the 64-bit hash codes perform best in AID dataset. Therefore, we select 48-bit and 64-bit hash codes to demonstrate the performance of the two datasets, respectively. From the two figures, it can be seen that our proposed method can obtain all true positive results for two query samples (baseballdiamond from UCMD dataset and bridge from AID dataset) in the top-10 and top-60. The other three methods also obtain good results, but they have several wrong retrieval images. In addition, it is worth noting that our method can obtain more discriminative hash codes, which can distinguish confusing remote sensing images. For instance, the KULSH-CNN and PRH-CNN have an error in identifying the “tenniscourt” as the “baseballdiamond”, and the PLSH-CNN misclassifies the residential as the bridge. Our method demonstrates obviously superior performance over other competitive baseline methods.

6. Conclusions

In this paper, we propose an unsupervised deep hash learning method for remote sensing image retrieval, namely deep contrastive self-supervised hashing. By creating a pretext task of preserving data visual augmentation invariance, we solve the problem of lacking supervised information, which is a significant challenge in unsupervised retrieval research. More specially, based on the task of pulling two augmented views generated from the same image and simultaneously pushing two views produced from different images, we develop the first deep unsupervised hash learning framework by introducing the normalized temperature-scaled cross-entropy loss and quantization loss. Extensive experiments on two popular benchmark datasets of UCMD and AID have demonstrated that our DCSH has overwhelming superiority in remote sensing image retrieval compared with state-of-art unsupervised hashing methods. For instance, on AID, the MAP performance for 16, 32, 48 and 64 bits of our DCSH method is 66.17%, 70.04%, 71.48% and 73.83%, respectively, which enables it to outperform KUSH, PRH and PLSH by at least 13.69%.

The paper only focuses on unimodal hash retrieval; that is, a query image must be provided to find the desired image. Nevertheless, retrieval system users sometimes prefer to enter text rather than images, so we plan to extend our method to the problem of retrieving images via text in the future, i.e., cross-modal hash retrieval.