Siamese Visual Tracking with Spatial-Channel Attention and Ranking Head Network

Abstract

Trackers based on the Siamese network have received much attention in recent years, owing to its remarkable performance, and the task of object tracking is to predict the location of the target in current frame. However, during the tracking process, distractors with similar appearances affect the judgment of the tracker and lead to tracking failure. In order to solve this problem, we propose a Siamese visual tracker with spatial-channel attention and a ranking head network. Firstly, we propose a Spatial Channel Attention Module, which fuses the features of the template and the search region by capturing both the spatial and the channel information simultaneously, allowing the tracker to recognize the target to be tracked from the background. Secondly, we design a ranking head network. By introducing joint ranking loss terms including classification ranking loss and confidence&IoU ranking loss, classification and regression branches are linked to refine the tracking results. Through the mutual guidance between the classification confidence score and IoU, a better positioning regression box is selected to improve the performance of the tracker. To better demonstrate that our proposed method is effective, we test the proposed tracker on the OTB100, VOT2016, VOT2018, UAV123, and GOT-10k testing datasets. On OTB100, the precision and success rate of our tracker are 0.925 and 0.700, respectively. Considering accuracy and speed, our method, overall, achieves state-of-the-art performance.

1. Introduction

Object tracking is important work in the field of computer vision [1,2]. With the advancements of deep learning, object tracking has a wide range of applications in human–computer interaction [3], intelligent driving [4], video surveillance [5], virtual reality [6], and other fields [7,8]. Although object tracking has made significant progress at present, it still faces challenges from two aspects: (1) the target itself, such as deformation, scale variation, etc; (2) the external environment, such as occlusion, low resolution, illumination variation, etc.

Due to its outstanding performance, deep learning has gotten much attention in recent years. The object tracker based on deep learning has also become the mainstream tracker at present, among which, the Siamese tracker is one of the most famous tracking frameworks. Although Siamese trackers have achieved a large improvement in performance, there is still a problem, which is that Siamese trackers are trained offline, meaning that the templates of Siamese trackers cannot be updated online. Therefore, when the tracker encounters distractors that bear a resemblance to the tracked target, it may struggle to maintain accurate tracking, leading to a decrease in overall precision. However, introducing an attention mechanism into most Siamese trackers can effectively address these issues. Specifically, we propose the Spatial Channel Attention Module (SCAM), which can output an enhanced fusion feature map by combining spatial and channel information from template and search region features. Finally, we feed the output of SCAM into the ranking head network for classification and regression to achieve more robust tracking.

In object tracking, the loss function can make the predicted bounding box as close as possible to the ground-truth bounding box. By computing the ratio of intersection and union between the predicted bounding box and the ground-truth bounding box, the optimization of the regression branch of the model can be achieved by Intersection over Union (IoU) loss [9]. Cross entropy loss [10] is the most commonly used loss term of the classification branch. It guides the predicted category of the classification branch to be the same as the true category. However, in the tracking process, there is a mismatch between the predicted value of the classification branch and the predicted value of the regression branch. For example, a positive sample after a huge deformation, which has a low IoU, may not be selected as a result. In fact, considering the predicted values of both classification and regression branches is conducive to the selection of the final results. Inspired by RBO [11], we introduce joint ranking losses and design the ranking head network. Specifically, the classification ranking loss allows us to better eliminate distractors and select the correct target. The confidence&IoU ranking loss links the classification and regression branches and selects the most suitable predicted bounding box as the result, improving the accuracy of our tracker.

In summary, we propose a Siamese visual tracker with spatial-channel attention and a ranking head network. The contributions of our work are shown below:

(1)

We propose SCAM. Specifically, we first calculate the spatial similarity matrix between two feature maps, and then we use this similarity matrix to filter the information in the search region’s feature. We concatenate the filtered and original search region’s features, and send it to the channel attention module. This approach is used to achieve spatial channel attention. This enhances the representation of fusion features and makes them more discriminative.

(2)

We design a ranking head network. Specifically, we introduce joint ranking loss terms into our approach. We use the mutual guidance of classification confidence score and IoU to select the final result, which can solve the problem of mismatch between the predicted values of the classification branch and the regression branch. Through the ranking head network, we can obtain more precise results and achieve a more robust tracking performance.

(3)

We train our tracker on ImageNet DET [12], ImageNet VID [12], COCO [13], YouTube-BB [14], and GOT-10k training set [15]. Excellent results have been achieved on five challenging datasets, including the GOT-10k testing set, UAV123 [16], OTB100 [17], VOT2016 [18], and VOT2018 [19]. Our code and data are available at https://github.com/csust7zhangjm/lyf2021 (accessed on 9 October 2023).

SCAM can enhance feature representation by combining spatial and channel information. It can be applied as a module in other tasks based on deep learning. The classification ranking loss can optimize the predicted values of the classification branch, and the confidence&IoU ranking loss can link the classification branch and regression branch to output the most suitable result. Therefore, in some tasks with multiple branches, the predicted values of multiple branches can be considered simultaneously, making the results of related tasks more accurate. The following is the structure of our paper. Section 2 presents the related work of this research, Section 3 describes SCAM and the joint ranking loss terms, Section 4 is the experimental proof of the proposed method, and Section 5 gives our conclusions.

2. Related Work

We will review the work related to object tracking in the following three areas: single-object Siamese tracking, the attention mechanism, and the head network.

2.1. Single Object Siamese Tracking

The Siamese object tracker has received extensive attention. SiamFC [20] was the pioneer of Siamese object tracking. It converted the traditional tracking process into a similarity matching problem, and determined the location of subsequent tracking by calculating the similarity. The success of SiamFC inspired many subsequent trackers, and they applied Siamese networks to object tracking to achieve the most advanced performance. SiamRPN [21] introduced the Region Proposal Network (RPN) [22], which, in turn, transformed the similarity matching problem into an independent classification and regression problem, where classification and regression branches serve different purposes, with the former used for distinguishing the background and foreground, and the latter for locating bounding boxes. In addition, trackers such as DaSiamRPN [23] and SiamRPN++ [24] improved SiamRPN to achieve better performance, but the above RPN-based algorithms designed multi-scale anchor boxes to obtain accurate bounding boxes, which would undoubtedly consume a large amount of time and cost and bring a large computational burden.

In 2019, the anchor-free Siamese tracking algorithm was proposed. Unlike the anchor-based algorithm, the anchor-free algorithm directly calculated the position of the target. Since no anchor box was introduced in the process of determining the target position, the anchor-free Siamese object tracking algorithm could reduce the computational burden and improve the tracking speed for trackers. SiamBAN [25] and SiamCAR [26] were among the most outstanding algorithms, and their performance reached the advanced level at that time. Overall, compared to RPN-based algorithms, anchor-free tracking algorithms could reduce the computational burden and, to some extent, reduce computational time. Anchor-free Siamese object tracking is a trend for the future.

2.2. Attention Mechanism

An important module in deep learning is the attention mechanism module, which is plug-and-play and very powerful [27]. SENet [28] established interdependence between channels by compressing feature maps. STMTrack [29] proposed a pixel-level correlation method; through the matrix reshape and matrix multiplication operation between the template and the search region’s feature, we could obtain the weight information of the spatial position, and then the weight information was weighted with the template’s feature to realize the spatial attention operation. CBAM [30] was a lightweight attention mechanism module that could perform attention operations on spatial information and channel information, making the model more concerned with the object itself. NLNet [31] was also a common attention module that improved the model’s understanding of global contextual information by capturing global dependencies.

The recently popular Transformer [32] was a powerful attention mechanism module, including a self-attention mechanism and cross-attention mechanism. Its application had enabled many algorithms to achieve state-of-the-art performance. Overall, the attention mechanism was applied to many deep learning tasks, and its introduction could undoubtedly bring performance improvements. However, we should also note that the introduction of attention mechanisms may also result in elevated computational complexity, and make the algorithm fail to achieve real-time performance.

2.3. Head Network

The head network is a crucial component in object tracking. Its function is to predict the location of the target based on the input information. Nowadays, most of the mainstream trackers rely on anchor-free algorithms, which directly calculate the position of the bounding box. This method effectively reduces computation and boosts tracking performance. CornerNet [33] was an object detection method that used the two corner points of the bounding box as key points to achieve accurate object detection. ExtremeNet [34], on the other hand, was also an object detection method that predicted the center point and the four corner points of all objects simultaneously.

The Head network can be further divided into a classification branch and a regression branch. The function of the classification is to distinguish the foreground from the background, and the purpose of regression is to locate the bounding box’s position. Some current trackers, such as SiamRPN, SiamFC++ [35], etc., determined the position of the bounding box through classification and regression branches in the head network. However, in the inference phase, mismatches between the predicted values of the classification and regression branches could interfere with the selection of results. PrDiMP [36] used probabilistic methods to model the labels of the targets, but this approach could worsen the discrepancy between classification and regression even more [4]. Therefore, if the mismatch between these two branches can be successfully resolved, our tracker’s robustness can be further improved.

3. Methods

3.1. Overview

Figure 1 shows our overall tracking process. In the feature extraction stage, ResNet-50 [37] was chosen as our feature extraction network. SiamBAN selects the feature maps of C3, C4, and C5 layers outputted by the feature extraction network, but we observe that the weight values of C5 in the classification and regression network are very small, and as such, we only used C3 and C4. The utilization of features from different layers can make varying impacts on the tracking process. The features extracted by the shallow layer contain richer spatial information, whereas the features extracted by the deep layer contain richer semantic information. Therefore, we select the output features of C3 and C4 for feature fusion, and our approach takes both spatial information and semantic information into consideration.

In the feature fusion network, our SCAM adopts pixel-wise correlation [29] instead of depth-wise cross-correlation, and utilizes a channel attention mechanism to enhance our feature’s expression. In the classification and regression network, classification feature maps and regression feature maps from different layers are added with weights, and fed into our improved loss function, which further improves the accuracy of the tracker results.

3.2. Spatial Channel Attention Module

The attention mechanism is helpful for most visual tasks. The inputs to SCAM are the feature maps output by feature extraction network. As shown in Figure 2, let $Z\in {\mathbb{R}}^{C\times h\times w}$ be the template’s feature map, and $X\in {\mathbb{R}}^{C\times H\times W}$ be the search region’s feature map. We use the Reshape operation for the template’s feature Z and the search region’s feature X to obtain feature maps $Z^1$, $Z^2$, and $X^1$, as shown in Equation (1):

$$\begin{array}{c}{Z}^1=\mathrm{Reshape}1\left(\mathit{Z}\right),\hfill \\ Z^2=\mathrm{Reshape}2\left(\mathit{Z}\right),\hfill \\ X^1=\mathrm{Reshape}3\left(\mathit{X}\right),\hfill \end{array}$$

(1)

where, $Z^1\in {\mathbb{R}}^{C\times hw}$, $Z^2\in {\mathbb{R}}^{hw\times C}$, and $X^1\in {\mathbb{R}}^{C\times HW}$. The function of $\mathrm{Reshape}i\left(.\right)$, $i\in \{1,2,3\}$, is to change the dimensions of Z and X. Here, we utilize $\mathrm{Reshape}i\left(.\right)$ to reduce the dimensions of Z and X, so that their dimensions change from three dimensions to two dimensions.

Next, the similarity matrix $W\in {\mathbb{R}}^{hw\times HW}$ is obtained through matrix multiplication, and the matrix entry $W_{ij}$ is calculated according to Equation (2):

$$W_{ij}=\frac{exp\left[\left(Z_{i.}^2\odot X_{.j}^1\right)/\sqrt{C}\right]}{{\sum }_{\forall k}exp\left[\left(Z_{k.}^2\odot X_{.j}^1\right)/\sqrt{C}\right]},$$

(2)

where $Z_{i.}^2$ represents i-th row on $Z^2$, $X_{.j}^1$ represents j-th column on $X^1$, ⊙ denotes the vector dot-product operation, and C represents the number of channels.

By concatenating the filtered and original search region’s feature in the channel dimension, we can obtain the fusion feature S, as shown in Equation (3):

$$S=\mathrm{Concat}\left(\mathrm{Reshape}4\left(\left({\mathit{Z}}^1\otimes \mathit{W}\right)\right),\mathit{X}\right),$$

(3)

where Concat $\left(·,·\right)$ represents the concatenate operation.

The input of SCAM are C3 and C4’s output features from the feature extraction network, and the output is the fusion feature $F_i$. Its operation is shown in Equation (4):

$$F_i=\mathrm{SCAM}\left({\mathit{Z}}_{\mathrm{i}},{\mathit{X}}_{\mathrm{i}}\right),$$

(4)

where $\mathrm{SCAM}\left(·,·\right)$ is the module we proposed, $Z_i$ represents the template’s feature of the i-th layer, and $X_i$ represents the search region’s feature of the i-th layer, $i\in \left\{3,4\right\}$.

Through the classification and regression network, we get fusion feature $F_i$’s classification feature map $f_{cls}^i$ and regression feature map $f_{reg}^i$, and we perform weighted addition operations on the results from different layers, as shown in Equation (5):

$$\begin{array}{c}\hfill f_{cls}=\sum _{i=3}^4{\beta }_{cls}^i{f}_{cls}^i,\\ \hfill f_{reg}=\sum _{i=3}^4{\beta }_{reg}^i{f}_{reg}^i,\end{array}$$

(5)

where $f_{cls}$ and $f_{reg}$ are the final classification and regression feature maps, respectively. In addition, ${\beta }_{cls}^i$ and ${\beta }_{reg}^i$ are the weights obtained after optimization together with the network.

3.3. Ranking Head Network

Regression loss. For every position within the regression’s feature map $f_{reg}$, we can find the corresponding region in the search region’s feature map X. Our method can predict the bounding box at each position through regression. Figure 3 shows the loss function we used. Our regression loss is shown in Equation (6):

$$L_{reg}=1-\frac{y_{label\_reg}\cap y_{reg}}{y_{label\_reg}\cup y_{reg}},$$

(6)

where $y_{label\_reg}$ represents the ground-truth bounding box and $y_{reg}$ represents the predicted bounding box.

Classification loss. In the classification branch, we usually use cross entropy loss as the classification loss function in the head network of our tracker. Our classification loss is shown in Equation (7):

$$\begin{array}{c}\hfill L_{cls}={\lambda }_{pos}\frac{1}{N_{pos}}\sum _{i\in P_{pos}}CE\left(y_{cls}^i,y_{label\_cls}\right)\\ \hfill +{\lambda }_{neg}\frac{1}{N_{neg}}\sum _{i\in P_{neg}}CE\left(y_{cls}^i,y_{label\_cls}\right),\end{array}$$

(7)

where $CE$ represents the cross entropy loss, and $N_{pos}$ and $N_{neg}$ represent the number of positive samples and negative samples, respectively. $P_{pos}$ and $P_{neg}$ represent the positive and negative sample sets, respectively. $y_{label\_cls}$ represents the classification label, and we use $y_{cls}$ to denote the output of the classification branch. ${\lambda }_{pos}$ and ${\lambda }_{neg}$ are control parameters; we set them to 0.5 here.

Classification ranking loss. In our experiment, we will filter out negative samples with lower classification confidence scores, but some negative samples do have higher classification confidence scores. We refer to these negative samples as hard negative samples. These hard negative samples may have an appearance that closely resembles the tracking target. In the process of tracking, these hard negative samples are difficult to distinguish when performing classification. In order to give our tracker a stronger ability to distinguish hard negative samples, we introduce classification ranking loss as Equation (8), and our goal is to minimize the influence of these hard negative samples on the tracking results:

$$L_{rk-cls}=\frac{1}{\gamma }log\left(1+exp\left(\left(E_{neg}-E_{pos}+\delta \right)\times \gamma \right)\right),$$

(8)

where $E_{neg}$ represents the expectation of negative samples, and it is obtained by using weighted addition method for the classification confidence score of each negative sample [11]. $E_{pos}$ is as well. $\gamma$ and $\delta$ are two parameters; we set them here to 4 and 0.5, respectively.

Confidence & IoU ranking loss. The branches of the tracker are relatively independent. This sometimes leads to a mismatch between the classification and the regression branches. In order to address this problem, as shown in Figure 4, we introduce the confidence&IoU ranking loss. The positive sample that ranks high in both classification confidence score and IoU is selected as the final result. We hope to improve the accuracy of our tracking results in this way. The loss function is shown in Equation (9):

$$\begin{array}{c}\hfill L_{rk-ci}=\frac{1}{N_{pos}}\sum _{i,j\in P_{pos},p_i^{IoU}>p_j^{IoU}}exp\left(-\alpha \left(p_i^{conf}-p_j^{conf}\right)\right)\\ \hfill +\frac{1}{N_{pos}}\sum _{i,j\in P_{pos},p_i^{conf}>p_j^{conf}}exp\left(-\alpha \left(p_i^{IoU}-p_j^{IoU}\right)\right),\end{array}$$

(9)

where $p_i^{conf}$ represents the classification confidence score of each positive sample, and $p_i^{IoU}$ represents the predict bounding box of each positive sample. $\alpha$ is a parameter; we set it to 3.

Finally, our overall loss function is shown in Equation (10):

$$L_{\mathrm{total}}=L_{reg}+L_{cls}+{\lambda }_{rk-cls}{L}_{rk-cls}+{\lambda }_{rk-ci}{L}_{rk-ci}.$$

(10)

where ${\lambda }_{rk-cls}$ is set to 0.5 and ${\lambda }_{rk-ci}$ is set to 0.25.

4. Experiments

4.1. Implementation Details

Our algorithm is based on CUDA 10.1, Pytorch 1.3.1, and Python 3.7, implemented on three 2080 Ti GPUs. ResNet-50 was chosen as our feature extraction network, which was trained on ImageNet and preserves its weights. We use images with a size of 127 × 127 pixels as input into our template feature extraction network and images with a size of 255 × 255 pixels as input into our search region feature extraction network. We train our tracker using five training datasets, including YouTube-BB, COCO, ImageNet DET, ImageNet VID, and GOT-10k training set. Also, we utilize the stochastic gradient descent algorithm to optimize our model. Considering our hardware environment, we set the batch size to 28. We trained for a total of 20 epochs. We used warming learning rates from 0.001 to 0.005 for the first epoch to the fifth epoch. When training the first epoch to the tenth epoch, we train only the classification and the regression branches in the head network. In training the eleventh epoch to the twentieth epoch, we make it possible for the backbone network to also be included in the training by fine-tuning the weights of the backbone network. In the testing phase, we select the first frame as our template and then perform similarity matching on the subsequent video sequences. We evaluate the performance of our proposed algorithm on OTB100, UAV123, VOT2016, VOT2018, and the GOT-10k testing set. We achieved satisfactory results on these datasets.

4.2. Results on OTB100 Benchmark

There are 100 fully annotated sequences in OTB100 dataset. Each sequence in the dataset is annotated with challenges, including occlusion, deformation, motion blur, etc. On OTB100, we use the precision and the success rate to test the performance of our tracker. On OTB100, we compare our tracker with nine excellent trackers such as SiamBAN [25], SiamCAR [26], SiamFC++ [35], DaSiamRPN [23], SRDCF [38], SiamFC [20], CFNet [39], SiamRPN++ [24], and SiamDW [40].

We can see from Figure 5 that our tracker achieves satisfactory performance on the OTB100 dataset. Compared with SiamDW, SiamCAR, SiamBAN, and SiamRPN++, our tracker improves the accuracy by 0.3%, 1.6%, 1.6%, and 2.2%, respectively. At the same time, our trackers have the highest success rate, reaching up to 0.700. Figure 6 and Figure 7 show the names of the 11 challenges and the results of our tracker on 11 challenges.

4.3. Results on UAV123 Benchmark

Another excellent dataset in the field of object tracking is UAV123. There are 123 video sequences included in UAV123, all of which are captured from a low-altitude aerial perspective. Among them, the 1st video to the 103rd video are stable, the 104th video to the 115th video are unstable, and the 116th video to the 123rd video are synthetic. On UAV123, we compare our tracker with eight excellent trackers, SiamCAR, ECO [41], ECO-HC [41], SiamRPN, SiamRPN++, STRCF [42], TADT [43], and DaSiamRPN. Figure 8 clearly demonstrates that compared to SiamRPN++, SiamCAR, and DaSiamRPN, our tracker improves the precision by 5.3%, 3.6%, and 7.8%, respectively. In addition, our tracker has achieved a success rate of 0.631, which is 1.2% higher than SiamCAR. Both are the best among the eight compared trackers. Figure 9 and Figure 10 show the names of the 12 challenges and the results of our tracker on 12 challenges.

4.4. Results on VOT2016 Benchmark

The VOT dataset holds a position of authority within the area of object tracking. There are 60 types of videos in the VOT2016 dataset. Before the emergence of VOT, the mainstream tracking strategy was to use the first frame of the video sequence to initialize the tracker, and then until the end of the video. However, due to the presence of distracting objects, the tracker is likely to lose its localization of the target. Therefore, when the tracker loses the target, the VOT is delayed back 5 frames. In VOT2016, when we evaluate the performance of a tracker, we use the expected average overlap (EAO). In this way, we can balance the accuracy and robustness of the tracker at the same time. We compare our tracker with other advanced trackers, including SiamRPN, DaSiamRPN, ECO-HC, MCCT-H [44], SiamRPN++, SiamMask [45], SiamR-CNN [46], ECO, and MCCT [44]. The data in

Table 1. Evaluation of our tracker and other trackers on VOT2016. E is EAO, A represents accuracy, R denotes robustness. Higher values of EAO and A represent greater accuracy, so we use ↑. The smaller the R value, the greater the immunity to interference, so we use ↓. The three best results are highlighted in bold, bold and italic, and italic.

	MCCT-H	ECO-HC	SiamRPN	ECO	MCCT	DaSiam-RPN	SiamMask	SiamRPN++	SiamR-CNN	Ours
E↑	0.299	0.322	0.337	0.374	0.393	0.401	0.425	0.437	0.460	0.530
A↑	0.570	0.542	0.578	0.555	0.579	0.609	0.634	0.644	0.645	0.634
R↓	0.331	0.303	0.312	0.200	0.186	0.224	0.214	0.219	0.172	0.084

clearly show that compared with SiamR-CNN, SiamRPN++, and SiamMask, our tracker improves by 15.2%, 21.2%, and 24.7% in EAO, and our robustness is the lowest among these algorithms at only 0.084, which is enough to prove that our tracker can achieve stable tracking in complex environments. Figure 11 shows the names of the six challenges and the results of our tracker on six challenges.

4.5. Results on VOT2018 Benchmark

VOT2018 is one of VOT’s latest object tracking datasets. Compared to VOT2016, the video sequences in VOT2018 exhibit a higher level of complexity and contain more difficult challenges. VOT2018 and VOT2016 use the same evaluation criteria, and we conducted comparative experiments on VOT2018, comparing state-of-the-art trackers such as SiamCAR, SiamMask, SiamFC++, SiamKPN [47], SiamRPN++, SiamR-CNN, SiamRPN, DaSiamRPN, and ATOM [48]. As shown in

Table 2. Evaluation of our tracker and other trackers on VOT2018. E is EAO, A represents accuracy, R denotes robustness. Higher values of EAO and A represent greater accuracy, so we use ↑. The smaller the R value, the greater the immunity to interference, so we use ↓. The three best results are highlighted in bold, bold and italic, and italic.

	DaSiam-RPN	ATOM	SiamR-CNN	SiamMask	Siam-RPN++	SiamCAR	SiamFC++	SiamKPN	SiamRPN	Ours
E↑	0.383	0.400	0.405	0.406	0.415	0.423	0.426	0.428	0.383	0.430
A↑	0.586	0.590	0.612	0.598	0.601	0.578	0.583	0.596	0.586	0.587
R↓	0.276	0.203	0.220	0.248	0.234	0.197	0.173	0.187	0.276	0.164

, compared with SiamCAR, SiamKPN, and SiamFC++, our tracker improves by 1.7%, 0.5%, and 0.9% in the average overlap rate. In terms of robustness, our tracker performs the best among the eight compared trackers. The results on the six challenges are shown in Figure 12.

4.6. Results on GOT-10k Benchmark

GOT-10k is a large dataset which was published by the Chinese Academy of Sciences that contains numerous video sequences. There are 560 kinds of moving objects in the training set, with 87 different motion patterns between them. The total video sequence exceeds 10,000 and there are more than 1,500,000 manually labeled bounding boxes. The GOT-10k testing set has a total of 420 video sequences. These video sequences contain a total of 31 different motion categories and 84 different object categories. There are three indicators in the GOT-10k testing set, including average overlap (AO), success rate ($S{R}_{0.5}$, $S{R}_{0.75}$), and frames per second (FPS).

Table 3. Evaluation of our tracker and other trackers on GOT-10k. ↑ indicates that a higher value is better. The three best results are highlighted in bold, bold and italic, and italic.

	SiamDW	DaSiamRPN	SiamRPN++	SiamCAR	ATOM	SiamRPN	SiamMask	Ours
AO↑	0.416	0.444	0.517	0.569	0.556	0.483	0.453	0.618
$S{R}_{0.5}$↑	0.475	0.536	0.616	0.670	0.634	0.581	0.550	0.722
$S{R}_{0.75}$↑	0.144	0.220	0.325	0.415	0.402	0.270	0.248	0.491
FPS↑	66.67	134.40	3.18	17.21	20.71	97.55	15.37	50.91

clearly shows the results of our tracker compared to SiamCAR, SiamMask, DaSiamRPN, SiamDW, SiamRPN, SiamRPN++, and ATOM.

4.7. Ablation Experiment

We performed five sets of ablation experiments on UAV123. We use the backbone of Figure 1 to replace the backbone in SiamBAN, and make the modified SiamBAN as our baseline. As shown in

Table 4. We verified the validity of each part on UAV123. $\Delta$s represents increases in success plot and $\Delta$p represents increases in precision plot.

Method	Success	$\Delta$s	Precision	$\Delta$p
baseline	0.608	−	0.805	−
baseline + SCAM	0.620	$+1.2\%$	0.817	$+1.2\%$
baseline + SCAM + $L_{rk-cls}$	0.624	$+1.6\%$	0.823	$+1.8\%$
baseline + SCAM + $L_{rk-ci}$	0.622	$+1.4\%$	0.827	$+2.2\%$
baseline + SCAM + $L_{rk-cls}$ + $L_{rk-ci}$	0.631	$+2.3\%$	0.842	$+3.7\%$

, our method has increases in the success rate and precision by $2.3\%$ and $3.7\%$, respectively. In order to verify the ability of our proposed SCAM to distinguish similar objects, we evaluated the SOB challenge on UAV123, as shown in Figure 13. The experiments prove that our proposed SCAM has good discrimination ability for similar objects.

4.8. Real-Time Analysis

We conducted a speed test on three 2080Ti GPUs.

Table 5. The speed of our tracker on different datasets. ↑ indicates that a higher value is better.

	OTB100	UAV123	VOT2016	VOT2018	GOT-10k
FPS↑	51.1	63.1	48.5	59.4	50.91

shows that our tracker achieves real-time tracking on five datasets. We also measured the speed of our tracker and other advanced trackers on GOT-10k. As shown in Table 3, the speed of our tracker reached 50.91 FPS, achieving a compromise between accuracy and real-time performance.

4.9. Experimental Summary

In Section 4, we validate our proposed algorithm from four different perspectives. Specifically, in Section 4.1, we introduce the corresponding training strategies and the experimental environment. From Section 4.2 to Section 4.6, we evaluate the effectiveness of our tracker on five datasets: OTB100, UAV123, VOT2016, VOT2018, and the GOT-10k testing set. In comparison to mainstream trackers, our tracker achieves an excellent performance. In Section 4.7, we perform ablation experiments on our proposed SCAM and the introduced joint ranking loss terms on the UAV123 dataset, and the experimental results prove that the method we propose is effective. Finally, in Section 4.8, we test the speed of our tracker on three 2080Ti GPUs, and the results emphasize its real-time capability. The visualization of our algorithm is shown in Figure 14.

When evaluating the performance of trackers on different datasets, differences in tracker performance may be due to different proportions of video sequences with the same challenge. For example, OTB100 has a large number of video sequences containing an occlusion challenge, so when a tracker can solve the problem of target occlusion, it will have a high success rate on OTB100.

5. Conclusions

Overall, we proposed a Siamese visual tracker with spatial-channel attention and a ranking head network, and trained our tracker using five authoritative datasets. Our proposed SCAM can not only fuse template’s feature and search region’s feature, but can also establish long-term dependencies between spatial position information and channel information. The introduced confidence&IoU ranking loss and classification ranking loss loss can link the classification and the regression branches, use the classification confidence score and IoU to guide the selection of the final result, and improve the performance of the tracker. Overall, our proposed SCAM can combine information from both the spatial and channel to achieve feature enhancement. The joint ranking loss terms we introduce can consider both classification confidence score and IoU to output the most suitable result.

In addition, we also conducted a series of experiments on OTB100, VOT2016, VOT2018, UAV123, and GOT-10k. On the OTB100 dataset, our tracker achieved a success rate of 0.700, which is the best of all trackers. Our tracker achieved a precision of 0.842 on UAV123. The EAO of our tracker is 0.530 on VOT2016 and 0.430 on VOT2018, respectively, which are the best among all trackers. Our tracker’s AO reached 0.618 on GOT-10k. Therefore, our tracker achieves decent performance on these datasets.

However, our tracker’s performance is not satisfactory when the video has a low resolution or the target is out of view. We are considering introducing an online update module in future work; the introduction of an online update module could improve the stability of our tracker in the face of complex environments. These issues are the direction of our future efforts.