Conformal Test Martingale-Based Change-Point Detection for Geospatial Object Detectors

Abstract

Unsupervised domain adaptation for object detectors addresses the problem of improving the cross-domain robustness of object detection from label-rich to label-poor domains, which has been explored in many studies. However, one important issue in terms of when to apply the domain adaptation algorithm for geospatial object detectors has not been fully considered in the literature. In this paper, we tackle the problem of detecting the moment or change-point when the domain of geospatial images changes based on conformal test martingale. Beyond the simple introduction of this martingale-based process, we also propose a novel transformation approach to the original conformal test martingale to make change-point detection more efficient. The experiments are conducted with two partitions of our released large-scale remote sensing dataset and the experimental results empirically demonstrate the promising effectiveness and efficiency of our proposed algorithms for change-point detection.

1. Introduction

Object detection for geospatial images is a fundamental task in remote sensing [1]. Nowadays, many deep learning-based object detectors have been proposed to improve the performance of geospatial object detectors drastically when the training and test data are independent and identically distributed (IID assumption) [2,3,4,5,6]. Since the test images may be distributed differently from the training images because of the variation of illumination, image quality, and background, the domain shift problem has been considered for object detection [7]. To overcome the large drop in detection precision, the strategy of unsupervised domain adaptation has been utilized in many works [8,9,10,11], and which improves the cross-domain robustness of object detection from label-rich to label-poor domains in an unsupervised way. The existing literature builds on the assumption that the domain shift is known in advance and the research mainly focuses on how to make the trained object detector adapt to a new domain. In practice, however, the domain shift phenomenon may not appear immediately after the deployment of an object detector, and the time when the domain changes should be detected automatically. In other words, one important issue of detecting domain shift and deciding when to call domain adaptation algorithms intelligently has not been fully addressed. Figure 1 shows the flowchart of deploying object detectors in a situation where the domain of the test images changes, and our focus in this paper is to design the change-point detector. This is essential for geospatial object detectors, which are deployed in satellites in military or civilian areas without human intervention. In this case, automatically detecting the change point of two domains is important for timely avoiding the considerable drop in detection precision caused by domain shift.

Detecting the change point of data streams is important and challenging for both statistics and machine learning areas [12,13]. Without the detailed assumptions of underlying distributions, many statistical methods can fail to effectively and efficiently detect the changes [14,15]. Also, the mainstream work in statistics focuses on the batch setting where the observations are real numbers, which limits usability. The existing algorithms in machine learning, on the other hand, do not test IID assumption directly, which lacks theoretical bases and interpretability [16,17]. Based on the idea of conformal prediction [18,19], one promising non-parametric framework of testing IID assumption online named conformal test martingale has been proposed, which does not require any detailed assumptions of the data-generating process [20,21].

The conformal prediction-based method can construct exchangeability martingales with the assumption of the test data stream, satisfying IID assumptions, and makes use of the martingale inequalities to test this null assumption [22]. Each martingale inequality can define a corresponding little probability event, and the IID assumption can be rejected if the event happens. These martingales are built on the p-values calculated by conformal prediction, which are uniformly distributed when the IID assumption holds. Thus, these conformal prediction-based martingales for testing IID assumption are called conformal test martingales (CTMs), and have been successfully used for change-point detection in the area of classification or regression problems. The first work of using CTMs to detect change-point was proposed in [14], and more explorations [17,23,24] were conducted. The recently proposed CTMs with Simple Jumper [17,25] were successfully applied to change-point detection for regression problems, and further extended to probabilistic prediction for classification problems to protect the predictors from significant deterioration after the change point [26]. Despite their success in detecting a change point, CTMs have only been tested for data with a relatively simple structure, such as vectors or handwritten digits. Therefore, whether they can be used to detect change-point for high-resolution images with complex contents and backgrounds such as geospatial images is worth exploring.

Therefore, we introduce CTMs to detect change-point of different domains for geospatial object detectors, such that the domain adaptation algorithm can be called automatically. Moreover, we develop transformations on top of conformal test martingales using momentum [27] to make change-point detection faster. To test the proposed approaches, we propose using clustering analysis on a remote sensing dataset to construct different domains, since there is a lack of datasets, especially for domain adaptive geospatial object detectors.

In summary, this work is the first to attempt to address the problem of detecting the change point of different domains for geospatial object detectors, and which employs conformal test martingales and proposes a novel transformation method to accelerate the detection process. The contributions of this paper are as follows:

• Clustering analysis for partitioning our released large-scale remote sensing dataset for geospatial object detection [28] is proposed for domain shift problems in the field of geospatial object detection.
• Conformal test martingales and a novel transformation method are first introduced to detect change-point for object detectors, with few false alarms for the target domain.
• Experimental results not only demonstrate the effectiveness and efficiency of our proposed methods for change-point detection, but also verify the partitions of the datasets in a statistical way.

The rest of this paper is organized as follows. Section 2 introduces conformal test martingale and change-point detection. Section 3 proposes the methods of constructing different domains, calculation of nonconformity measure, and improving detection efficiency with momentum. In Section 4, the experiments are conducted to verify the partitions and test our proposed algorithm for change-point detection. The conclusions are drawn in Section 5.

2. Conformal Test Martingale

We first introduce conformal test martingale following [17]. $(z_1,z_2,\dots )$ denote a sequence of test data and an inductive nonconformity measure is a measurable function A, which maps any $z_i$ to a real number ${\alpha }_i=A\left(z_i\right)$. ${\alpha }_i$ is known as the nonconformity score and measures the strangeness of $z_i$ compared with a dataset S. In the literature, A is often related to training data, and accordingly, ${\alpha }_i$ measures the difference of $z_i$ from training data. After obtaining $z_n$, the p-value of $z_n$ can be calculated based on the existing collected test data $(z_1,\dots ,z_n)$ as follows,

$$p_n=\frac{|\left\{i\right|{\alpha }_i>{\alpha }_n\}|+{\theta }_n\left|\left\{i\right|{\alpha }_i={\alpha }_n\}\right|}{n},$$

(1)

where i ranges over $\{1,\dots ,n\}$. ${\theta }_n$ is a random number independent of $(z_1,\dots ,z_n)$ and distributed uniformly on $[0,1]$. If $(z_1,z_2,\dots )$ satisfy IID assumption, then the p-values $(p_1,p_2,\cdots )$ are independent and uniformly distributed on $[0,1]$. Based on this property, conformal test martingales $S_n$ can be constructed by

$$S_n=F(p_1,\dots ,p_n),n=0,1,\dots ,$$

where $F:{[0,1]}^{\ast }\to [0,\infty ]$ is a measurable function named betting martingale and ${[0,1]}^{\ast }$ denotes the set containing the sequences of any length whose elements taking values in $[0,1]$. Also, by definition, F maps empty sequence to 1 and for each $n\ge 1$ and each sequence $(u_1,\dots ,u_{n-1})\in {[0,1]}^{n-1}$, there holds

$${\int }_0^{1}F(u_1,\dots ,u_{n-1},u)du=F(u_1,\dots ,u_{n-1}).$$

One effective betting martingale is the Simple Jumper proposed in [17], whose definition is as follows,

$$F(u_1,\dots ,u_n)=\int \prod _{i=1}^n{f}_{{ϵ}_i}\left(u_i\right)\mu \left(d({ϵ}_1,{ϵ}_2,\dots )\right),$$

where

$$f_{ϵ}\left(p\right)=1+ϵ(p-0.5)$$

and $\mu$ is the probability measure on ${\{-1,0,1\}}^{\infty }$ defined by a Markov chain with state space $\{-1,0,1\}$ and parameter J. The martingale constructed above is summarized in Algorithm 1, which is the conformal test martingale with a simple jumper.

Algorithm 1: Conformal test martingale with a simple jumper.

Input: The p-values $p_1,p_2,\dots$ defined by Formula (1) and the parameter J. Output: Conformal test martingale $S_1,S_2,\dots$. 1: Set $S_0=1$, $C_{-1}=C_0=C_1=\frac{1}{3}$ and $C=1$. 2: for $n=1,2,\dots$ do 3:  for $ϵ\in \{-1,0,1\}$ do 4:   $C_{ϵ}=(1-J)C_{ϵ}+(J/3)C$ 5:  end for 6:  for $ϵ\in \{-1,0,1\}$ do 7:   $C_{ϵ}=C_{ϵ}{f}_{ϵ}\left(p_n\right)$ 8:  end for 9:  $C=C_{-1}+C_0+C_1$ 10:  $S_n=C$ 11: end for

Conformal test martingales are martingales by mathematical definition, since they satisfy the following equation:

$$E\left(S_n\right|S_1,\dots ,S_{n-1})=S_{n-1}.$$

Furthermore, by Ville’s inequality, for any constant $c>1$, there holds

$$P(\exists n:S_n\ge c)\le \frac{1}{c}.$$

(2)

Therefore, we can use Algorithm 1 to detect the change point [29,30]. Formally, suppose that $(z_1,z_2,\dots )$ are random variables, where the first $\theta -1$ variables, $(z_1,z_2,\dots ,z_{\theta -1})$, are each distributed according to $f_0\left(z\right)$ and the remaining variables $(z_{\theta },z_{\theta +1},\dots )$ are each distributed according to $f_1\left(z\right)$, i.e., the distribution changes from $f_0\left(z\right)$ to $f_1\left(z\right)$ at the change-point $\theta$. Thus, $(z_1,z_2,\dots ,z_{\theta -1})$ come from one domain and $(z_{\theta },z_{\theta +1},\dots )$ come from another domain. To test whether the distribution of $(z_1,z_2,\dots )$ satisfies the IID assumption, one can set $c=20$ to make a false alarm rate no more than $5\%$. After the change-point $\theta$, $S_n$ can become very large as the violation of IID assumption, and the change can be detected when $S_n\ge 20$.

3. Method

In this section, we first propose using clustering to partition the remote sensing dataset to different domains, since there lacks the datasets for domain adaptation research for geospatial object detectors. Then, we introduce a nonconformity measure based on the existing research about domain adaptation, which indicates how to measure the domain difference for different domain samples. At last, we develop a transformation of the original CTM to accelerate the change-point detection process based on the idea of momentum.

3.1. Partitions for Different Domains

We partition the dataset released in [28] by image quality or illumination, since these two indicators can be defined properly by mathematics, which avoids subjectivity in partitions.

Clear Domain vs. Blur Domain: We employ a Laplace operator to obtain the blur length for each image [31] and use K-mean algorithms with $K=2$ to cluster the blur lengths of all of the images, which leads to the clear domain (6278 samples) and blur domain (11,909 samples) in this paper.

Bright Domain vs. Dark Domain: The illumination intensity cannot be defined properly as a single value like blur length, since the dark or bright background may dominate the calculation. Thus, we propose using the distribution of illumination intensity to partition. We employ L values of images in LAB (Lightness, Red/Green Value, Blue/Yellow Value) color space [32] to define the illumination intensity for every pixel. For each image, all of the L values can be seen as its distribution of illumination intensity. To describe the distribution, we use the $0.1,0.2,\dots ,1.0$ quantiles to form the ten-dimensional feature vector. After vectorization of all of the images, we also use K-mean algorithms to cluster them with $K=2$ and the final partition leads to the bright domain (8159 samples) and dark domain (10,028 samples) in this paper.

3.2. Nonconformity Measure with Image- and Instance-Level Representations

The nonconformity measure is of great importance to efficient change-point detection, since it measures the strangeness between the current test datum and the obtained data, which is the key to increasing $S_n$ after the change point. Existing works about domain adaptive object detectors mainly use the hidden feature map output by the base convolutional layers as the image-level representation of the original input image, and all of the final output vectors with high confidence of objects as the corresponding instance-level representation [7]. Building on those representations, the domain adaptation algorithms are called to decrease the gap in the representations between domains. In this spirit, we propose using the two representations to calculate the nonconformity measure, since the representations between two domains are different before the domain adaptation process.

Our notations follow [7], which both concerns the image-level representation and instance-level representation. Define ${\varphi }_{u,v}\left(z_i\right)$ as the activation at position $(u,v)$ of the feature map of the ith image $z_i$ output by the backbone, and let $p_{i,j}$ be the output vector of the jth object of ith image detected by the object detector. Following [7], we construct the image-level representation $\overline{\varphi }\left(z_i\right)$ and the instance-level representation $\overline{p}\left(z_i\right)$ as follows:

$$\overline{\varphi }\left(z_i\right)=\frac{1}{N_{\varphi }}\sum _{u,v}{\varphi }_{u,v}\left(z_i\right),$$

$$\overline{p}\left(z_i\right)=\frac{1}{N_{p_i}}\sum _j{p}_{i,j},$$

where $N_{\varphi }$ is the total number of pixels of the feature map, $N_{p_i}$ is the total number of detected objects and $\overline{p}\left(z_i\right)$ is set to zero vector if $N_{p_i}=0$. Finally, our whole representation vector $v\left(z_i\right)$ of each image is the concatenation of $\overline{\varphi }\left(z_i\right)$ and $\overline{p}\left(z_i\right)$. After this vectorization of each image, we calculate the nonconformity score of $z_i$ as

$${\alpha }_i=A\left(z_i\right)=\underset{j}{min}\sqrt{{(v\left(z_i\right)-v\left(z_j^{val}\right))}^2}$$

(3)

where $z_j^{val}$ represents the jth sample in the source validation set and j ranges over $\{1,\dots ,N^v\}$ with $N^v$ being the validation size. This nonconformity measure is the 1-nearest neighbor nonconformity measure mentioned in [17], which measures the mathematical distance between $v\left(z_i\right)$ and the point set $\{v\left(z_1^{val}\right),\dots ,v\left(z_{N^v}^{val}\right)\}$ with the fact that larger ${\alpha }_i$ is related to stranger $z_i$ compared with the source validation data.

3.3. Accelerating Change-Point Detection with Momentum

According to the existing research in the literature [23,24,29] and our empirical implementation, before the change point, $S_n$ tends to decrease to a low value. As a result, after the change point, the $S_n$ needs many steps to recover from the low value to the significant value c in Formula (2). To reduce the delay and accelerate the detection process, two methods based on the transformations of the original CTMs have been proposed. One method [23] is to restrict $S_n$ such that it will not take values below some preset threshold. The other method [24] first sets $S_n=1$ when there is not enough evidence available to reject the IID assumption, and then starts the process of conformal test martingales when the evidence is sufficient.

To address this issue, we propose a novel transformation method considering the variation of the martingale values. The motivation of our method is from the observation that after the change point, the martingale values $S_n$ tend to increase and exceed the value c. To make this process efficient, we construct a transformation based on the idea that if the change in $S_n$ is positive, one should make $S_n$ increase faster, which can be realized using momentum [27]. Specifically, let ${\delta }_n=S_n-S_{n-1}$ be the current change. To bring in the idea of momentum, one can employ a new process $S_n^{\ast }$ with the following iteration equations:

$$\begin{array}{cc}\hfill & {\delta }_n^{\ast }={\delta }_n+r_n\times {\delta }_{n-1}^{\ast },\hfill \\ \hfill & S_n^{\ast }=S_{n-1}^{\ast }+{\delta }_n^{\ast },\hfill \end{array}$$

where $r_n$ is the momentum parameter of the iterations. However, this performs poorly since the momentum both accelerates the increases and decreases of the original $S_n$ and $S_n^{\ast }$ can also be a small value. Therefore, we introduce Algorithm below, where $a\vee b=max\{a,b\}$ and $S_n^{\ast }$ only concentrates on the non-negative part of the momentum. Also, $S_n^{\ast }$ is always no less than $S_0$ for all n.

Algorithm 2: The transformation of conformal test martingale with momentum.

Input: The p-values $p_1,p_2,\dots$ defined by Formula (1), the momentum parameters $r_1,r_2,\dots$ and the parameter J. Output: The transformation process $S_1^{\ast },S_2^{\ast },\dots$. 1: Set $S_0=1$, $C_{-1}=C_0=C_1=\frac{1}{3}$, $C=1$, $S_0^{\ast }$ = 1 and ${\delta }_0^{\ast }=0$. 2: for $n=1,2,\dots$ do 3:  Calculate ${\delta }_n$ using Algorithm 1. 4:  Calculate ${\delta }_n^{\ast }={\delta }_n+r_n\times ({\delta }_{n-1}^{\ast }\vee 0)$. 5:  Obtain $S_n^{\ast }=(S_{n-1}^{\ast }+{\delta }_n^{\ast })\vee S_0$. 6: end for

This transformation in Algorithm 2 can significantly accelerate the detection speed if the momentum parameters are chosen or calculated properly. Based on the experimental observation that the rolling mean of the p-values are close to or far away from $0.5$ before or after the change point, we further propose a method to calculate $r_n$ with p-values in Section 4.2. Although the output of Algorithm 2 is not a martingale theoretically, which might violate the validity of the original CTM, our experiments empirically demonstrate the practicability and efficiency of this method for fast change-point detection.

4. Experiments

This section conducts experiments for empirically testing the proposed methods in Section 3. We first verify the partition of the dataset based on the performance drop of YOLOv3 (You Only Look Once Object Detector, Version 3) [33]. Then we explore the proposed algorithms from clear to blur domain to visualize their characteristics. Finally, we demonstrate the performance of different methods to show the improvement in our momentum-based approaches.

4.1. Verification of the Partitions

To verify the partitions proposed in Section 3.1 and prepare the datasets to test the algorithms in this paper, we split each domain into training, validation, and test sets with the ratio being 6:2:2. For each domain pair, we trained YOLOv3 with learning rate being $0.0001$ and epoch number being 120 on the source training set and used the total loss in the source validation set to choose the final model. All chosen models were tested on the test sets from source and target domains, respectively, to see whether there is a detection precision drop between the domains. The mAP (Mean Average Precision) for each test set is summarized in

Table 1. Mean Average Precision of YOLOv3.

Domain Shift Case	Source mAP	Target mAP
From Clear to Blur Domain	$94.15\%$	$69.62\%$
From Blur to Clear Domain	$95.34\%$	$91.69\%$
From Bright to Dark Domain	$87.48\%$	$80.87\%$
From Dark to Bright Domain	$95.54\%$	$88.64\%$

. The large drop of the mAP verifies our partition for different domains.

4.2. Exploratory Experiments of CTM

We explore Algorithm 1 from clear to blur domain to see whether it can detect change-point. The YOLOv3 trained on the training set of the clear domain is used to calculate the p-values based on the conformity measure defined by Formula (3). The first group data (1256 samples) of the test data stream is the shuffled test data from the clear domain, and the remaining data stream is the shuffled test data from the blur domain, i.e., the change-point $\theta$ is 1257. The parameter J is set to $0.01$ according to [17] and c is set to 20 to make the false alarm rate no more than $5\%$, referring to Formula (2). The rolling mean of the p-values is shown in Figure 2 and $S_n$ output by Algorithm 1 is shown in Figure 3, which demonstrate that the rolling mean is close to and running away from $0.5$ before and after the appearance of the change point, and the martingale $S_n$ increases considerably after the change point. This phenomenon was also shown in [14], since the p-values are distributed uniformly on $[0,1]$ if the test data are IID based on the theory analysis of conformal prediction.

Based on the observations from Figure 2 and Figure 3, we propose using the rolling mean of the p-values to calculate the momentum parameter $r_n$ in Algorithm 2. Denote the rolling mean of the p-values at step n as ${\gamma }_n$. We propose calculating the momentum parameters of Algorithm 2 as follows,

$$\begin{array}{c}\hfill r_n=\left\{\begin{array}{cc}0,\hfill & if{\gamma }_n-0.5\ge R\hfill \\ M,\hfill & if{\gamma }_n-0.5<R\hfill \end{array}\right.\end{array}$$

(4)

where R is the threshold to decide whether to accelerate the increase of the process and M is the momentum for acceleration. The reason for setting $r_n=0$ for large ${\gamma }_n$ is that $S_n$ can increase a little at the first few steps and a positive $r_n$ can cause false alarms of the change point. We set the window size of the rolling mean as 100 in this paper, so the first 99 $r_i$s are set to 0. This transformation process from clear to blur domain is also shown in Figure 3. By comparison, it can be seen that the transformation process can detect the change point more quickly than the original conformal test martingale.

4.3. Comparison Experiments

The comparison experiments were conducted for all of the four domain shift cases proposed in Section 3.1. For each domain shift case, the test data from the source domain came sequentially, followed by the test data from the target domain, and ten experiments were conducted by shuffling the order of the samples within each domain. The experimental results are recorded in

Table 2. Change-Point Detection From Clear to Blur Domain.

Algorithm	Mean Delay	POFD	POD
Alg.1	41.3	0.0003	0.9827
CTM-C($ϵ=5$)	37.3	0.0000	0.9843
CTM-C($ϵ=10$)	33.5	0.0000	0.9855
CTM-C($ϵ=15$)	34.7	0.0000	0.9854
Alg.2($r_n=0$)	41.2	0.0003	0.9827
Alg.2($r_n=0.1$)	40.8	0.0007	0.9829
Alg.2($r_n=0.5$)	$\mathbf{30}.\mathbf{5}$	0.1806	0.9872
Alg.2(Adaptive)	34.5	0.0003	0.9855

The lowest Mean Delay is shown in bold and underlined.

,

Table 3. Change-Point Detection From Blur to Clear Domain.

Algorithm	Mean Delay	POFD	POD
Alg.1	53.6	0.0008	0.9573
CTM-C($ϵ=5$)	46.0	0.0000	0.9633
CTM-C($ϵ=10$)	37.3	0.0000	0.9703
CTM-C($ϵ=15$)	37.5	0.0000	0.9700
Alg.2($r_n=0$)	53.5	0.0008	0.9574
Alg.2($r_n=0.1$)	48.6	0.0949	0.9613
Alg.2($r_n=0.5$)	37.4	0.2493	0.9702
Alg.2(Adaptive)	$\mathbf{31}.\mathbf{0}$	0.0008	0.9753

The lowest Mean Delay is shown in bold and underlined.

,

Table 4. Change-Point Detection From Bright to Dark Domain.

Algorithm	Mean Delay	POFD	POD
Alg.1	40.0	0.0000	0.9801
CTM-C($ϵ=5$)	27.1	0.0002	0.9864
CTM-C($ϵ=10$)	26.6	0.0000	0.9867
CTM-C($ϵ=15$)	27.9	0.0000	0.9860
Alg.2($r_n=0$)	40.0	0.0000	0.9801
Alg.2($r_n=0.1$)	39.9	0.0000	0.9801
Alg.2($r_n=0.5$)	37.7	0.0000	0.9812
Alg.2(Adaptive)	$\mathbf{26}.\mathbf{0}$	0.0000	0.9870

The lowest Mean Delay is shown in bold and underlined.

and

Table 5. Change-Point Detection From Dark to Bright Domain.

Algorithm	Mean Delay	POFD	POD
Alg.1	51.1	0.0003	0.9687
CTM-C($ϵ=5$)	41.6	0.0002	0.9745
CTM-C($ϵ=10$)	36.7	0.0000	0.9774
CTM-C($ϵ=15$)	37.8	0.0000	0.9768
Alg.2($r_n=0$)	50.8	0.0003	0.9689
Alg.2($r_n=0.1$)	48.6	0.0153	0.9702
Alg.2($r_n=0.5$)	42.9	0.0970	0.9737
Alg.2(Adaptive)	$\mathbf{31}.\mathbf{8}$	0.0003	0.9805

The lowest Mean Delay is shown in bold and underlined.

, where Mean Delay refers to the mean value of the delay steps to raise the first true alarm of domain shift [23]. POD and POFD are widely used indicators for detection in statistics [34], where POD represents probability of detectionand POFD represents probability of false detection. All of the input features are $v\left(z_i\right)$ introduced in Section 3.2. Alg.1 is the conformal test martingale calculated by Algorithm 1, whose nonconformity measure is Formula (3) and parameter J is set to $0.01$ according to [17]. CTM-C is a recently proposed transformation process of CTM based on a cautious betting strategy, which takes the evidence of violating IID assumption into account [24]. CTM-C is also a martingale theoretically whose performance is close to or even better than existing well-performed change-point detection techniques. Therefore, CTM-C is the state-of-the-art approach of CTM-based methods. The parameter $ϵ$ of CTM-C is the threshold related to the evidence of violating IID assumption, and we set it as 5, 10, or 15 in the comparison experiments and denote the algorithms as CTM-C($ϵ=5$), CTM-C($ϵ=10$) and CTM-C($ϵ=15$), respectively. Alg.2 is the transformation of the outputs of Alg.1 based on Algorithm, where Alg.2($r_n=a$) means $r_n=a$ for any n. Alg.2(Adaptive) is Alg.2 with $r_n$ calculated by Formula (4), where $R=-0.1$, $M=10$. Since we focus on the detection speed of each algorithm, the lowest Mean Delay is shown in bold and underlined in the Tables.

In Table 2, we can see that the performance of Alg.1 and Alg.2($r_n=0$) is nearly the same, which indicates that only limiting the lower bound of $S_n^{\ast }$ cannot improve much. By increasing $r_n$, the change point can be detected quickly, whereas larger $r_n$ leads to more false alarms. By setting $r_n=0.5$, the violation of validity with Algorithm 2 is observed, since the POFD is higher than the $5\%$ significance level. This implies that the momentum parameter of Algorithm 2 should be chosen properly for empirical validity. Finally, our proposed Alg.2(Adaptive) has low POFD and Mean Delay simultaneously, as it only accelerates the increase of $S_n^{\ast }$ when the distribution of p-values is far away from uniformity on $[0,1]$, i.e., the $r_n$ is far away from $0.5$. In comparison, the Mean Delay of CTM-C is really low and CTM-C($ϵ=10$) performs best, since it achieves the lowest Mean Delay with the POFD lower than $5\%$. In this domain shift case, our proposed Alg.2(Adaptive) is close to the performance of CTM-C($ϵ=10$).

With similar analysis, the experimental results shown in Table 3, Table 4 and Table 5 demonstrate that Alg.2(Adaptive) performs best for the other three domain shift cases, which confirms the effectiveness and efficiency of our proposed transformation process with adaptive momentum. Since Mean Delay is the most important indicator under the premise of low POFD, we also show Mean Delay in Figure 4 for visualization.

All of the experiments demonstrate the worth of introducing conformal test martingale to the detection of change-point for geospatial object detection with few false alarms, and the proposed Alg.2(Adaptive) can achieve equivalent and in some cases superior performance to the state-of-the-art CTM-based methods for change-point detection. The experimental results also show that although the proposed approach violates validity, it empirically performs well in most cases with proper momentum parameters. Besides, since Alg.1 can test IID assumption with a theoretical basis, the experimental results of Alg.1 also verify our partition for different domains proposed in Section 3.1 statistically.

5. Conclusions

This paper focuses on the problem of detecting domain shift for geospatial object detectors automatically with few false alarms. To do so, we first partition our released large-scale remote sensing dataset using clustering analysis, which attempts to build different domains for geospatial object detection. After this preparation, the original CTM and its transformation are tested for four different domain shift cases. The experimental results confirm the effectiveness and efficiency of our proposed algorithms, and the variant with adaptive momentum parameters demonstrates equivalent and in some cases superior performance to the state-of-the-art CTM-based methods for change-point detection. In addition, since CTM is a standard statistical hypothesis testing for detecting domain shift and all four domain shift cases are detected by CTM, we can reject the null hypothesis of the IID assumption and verify the partitions for different domains statistically. Our methods of constructing the domain shift datasets and detecting change-point are valuable for research on geospatial object detection in practice.