Multi-Camera Vehicle Tracking Based on Deep Tracklet Similarity Network

Abstract

Multi-camera vehicle tracking at the city scale has received lots of attention in the last few years. It has large-scale differences, frequent occlusion, and appearance differences caused by the viewing angle differences, which is quite challenging. In this research, we propose the Tracklet Similarity Network (TSN) for a multi-target multi-camera (MTMC) vehicle tracking system based on the evaluation of the similarity between vehicle tracklets. In addition, a novel component, Candidates Intersection Ratio (CIR), is proposed to refine the similarity. It provides an associate scheme to build the multi-camera tracking results as a tree structure. Based on these components, an end-to-end vehicle tracking system is proposed. The experimental results demonstrate that an $11\%$ improvement on the evaluation score is obtained compared to the conventional similarity baseline.

1. Introduction

With the recent advancement of computer vision, city-scale automatic traffic management is now possible. Real-time multi-target multi-camera (MTMC) vehicle tracking can be improved by techniques for automatic traffic monitoring and management [1,2,3,4,5,6]. Automatic video analytics can enhance traffic infrastructure design and congestion handling through the pervasively deployed traffic cameras.

Real-time multi-target multi-camera tracking is one of the crucial tasks in traffic management. Its purpose is to achieve better traffic design and traffic flow optimization by tracking many vehicles in a network across multiple surveillance cameras, as shown in Figure 1. Most approaches in MTMC follow the tracking by detection pipeline. Firstly, a detector is adopted to obtain all vehicle detections. After vehicle detection, a single-camera tracker needs to form vehicle tracklets of the same vehicle in each view. Then, these vehicle tracklets are associated across cameras.

There are also several difficulties for the MTMC task. The problems of how to eliminate unreliable vehicle tracklets and deal with view variations are significant in these tasks. Large-scale automatic video analytic systems must handle a large variability of vehicle types and appearances to meet the accuracy and reliability requirements in the real world. For applications such as vehicle re-identification, large view variations cast a significant challenge in vehicle re-identification across views. Similarly, how best to perform space–time vehicle tracklet association across views is important for vehicle counting and traffic analysis. In addition, images are captured by different cameras. The vehicle may have different poses and illumination conditions, resulting in different colors of the appearances. Different weather conditions, such as raining or hazing, make vehicle tracking problems more challenging.

Existing works [7,8] evaluate the connectivity between tracklets across cameras by simple Euclidean distance and cosine similarity. However, these metrics are not robust enough to measure the connectivity in tracklets. Moreover, when one tracklet is associated only with a tracklet having the highest similarity score, it does not fully use the information of the whole ranking list. To address these problems, a novel Tracklet Similarity Network (TSN) and Candidate Intersection Ratio (CIR) metric is proposed. TSN can better learn robust similarity scores. CIR and its corresponding association scheme can better leverage the ranking information of the tracklets.

In this paper, the proposed MTMC system is proposed via a complete pipeline. Firstly, Faster-RCNN is adopted as the vehicle detection module. Then, the TrackletNet tracker can generate tracklets based on vehicle detection results. Filters are applied to eliminate unreasonable tracklets. Finally, the proposed TSN evaluates similarities between tracklets, and the CIR matching scheme associates tracklets across cameras. To sum up, the contributions of this paper include the following. (1) Several novel components are proposed in this paper. Re-Id Block learns the feature representation for each tracklet rather than applying CNN to extract image features. Disentangle Block separates the tracklet feature from cameras to learn a camera-independent feature by exploiting camera loss and the ranked list loss functions. Similarity Measurement Block leverages strong similarities information from the ranking list of tracklets rather than using the tracklet with the highest score alone via a random walk scheme. (2) The aforementioned components are exploited to build an end-to-end deep multi-camera vehicle tracking model. (3) The experimental results demonstrate that an $11\%$ improvement on the evaluation score is obtained compared to the conventional similarity baseline. The proposed CIR outperforms the pairing method over $3.6\%$ by the Cosine Similarity and nearly $2\%$ by the IDF1 score.

2. Related Work

2.1. Single-Camera Tracking

The tracking-by-detection paradigm has been dominant in single-camera tracking tasks. DeepSORT [9] combines the Kalman filter [10] and Hungarian algorithm [11]. The Kalman filter helps predict the next position of a tracklet. The Hungarian algorithm is a data association method. It is used to associate tracklets. However, tracking-by-detection is a time-consuming process. The JDE tracker [12] is proposed to address this problem. The JDE tracker can generate vehicle detection and embeddings simultaneously so that it can save significant time. Yang et al. [2] propose a tracklet reconnection technique in SCT. They leverage GPS information and pre-defined zone areas to refine tracklet results.

2.2. Multi-Camera Tracking

Multi-camera tracking aims to associate tracklets across cameras. Most approaches follow the pipeline of vehicle detection, vehicle feature extraction, single-camera tracking, and multi-camera tracking. These approaches can be classified into three categories: vehicle feature extraction, hierarchical structure, and exception handling methods. Many multi-camera tracking methods focus on representative feature extraction for tracking. He et al. [1] propose a spatial–temporal attention mechanism in their feature extraction module. Therefore, they can generate more robust tracklet features. Ristani et al. [13] proposed adaptive weighted triplet loss for learning appearance features and hard identity mining for sampling hard negatives more frequently. Peri et al. [14] introduced an unsupervised excitation layer for highlighting the discriminative saliency region. The saliency map can increase the robustness and performance of the model. However, these methods fail to consider feature disentanglement and the variations brought by different cameras.

The second category is hierarchical structure methods. Li et al. [3] develop a hierarchical match strategy by the rule-based method. When associating tracklets, many rules can be considered, such as vehicle speed, directions, motion, etc. Xu et al. [5] presented a hierarchical composition model to composite the geometry, appearance, and motion of trajectories for multi-view multi-object tracking. Xu et al. [15] exploited semantic attributes by building a parse tree to relate the same people on different camera views for multi-camera people tracking. Wen et al. [16] achieve multi-camera multi-target tracking by formulating the problem as a sampling-based search of the space–time view hypergraph. The hypergraph is able to encode higher-order constraints on 3D geometry, motion continuity, and trajectory among 2D tracklets within and across different camera views. These methods capture multi-view objects by adding the physical properties of objects. However, their formulations usually have many parameters for different terms. The parameter tuning is difficult and time consuming.

The third category is exception handling methods. You et al. [17] utilizes local pose matching to solve the occlusion problem for multi-target multi-camera tracking. Optical flow is applied to reduce the distance caused by fast motion. The idea can be extended from pedestrians to vehicles easily. Specker et al. [18] consider that most tracking errors of false detection occur in occlusions. They develop an occlusion handling approach by labeling the overlap higher than a non-maximum suppression threshold as an occlusion pair. The occlusion information is leveraged for removing false detections and multi-camera clustering for tracking.

Bredereck et al. [6] proposed a data association method with the greedy algorithm to associate trackers for multi-object tracking. Some physical constraints are used to make the tracklet reliable. The aforementioned research evaluates the similarity between tracklets by simple cosine similarity, Euclidean distance, or handcrafted metric. Instead of using evaluation metrices, Hou et al. [4] proposed a network to measure the similarity. They adopt temporal sliding windows in their data sample method to capture the local temporal feature of tracklets. In comparison, our method proposes deep neural network learning features across cameras and exploits the Random Walk method to refine the similarity by further considering the relationships between tracklets in the target gallery.

2.3. Random Walk

The Random Walk algorithm is a graphical method to determine the probability of one point to another point. An initial probability can be defined for the graph that decides which point to start. For each walking step, the point-to-point probability is updated. After a few iterations, the probability will be stabilized. Ref. [19] proposed a random walk layer in deep neural networks. The network calculates the refined affinities between probe image and gallery images. By calculating cross-entropy loss with refined affinities, the network can be updated with stronger supervision. Motivated by [19], we exploit and design an additional random walk layer in our model.

2.4. Ranked List Loss

Triplet Loss [20] is widely used for learning similarity and dissimilarity semantic information of data. The Triplet Loss is computed to ensure that an anchor image $x_i^a$ is closer to positive samples $x_i^p$ while pushing away from negative samples $x_i^n$ and maintaining a margin distance m in feature space f:

$${\mathcal{L}}_{triplet}=\sum _i{\left[\parallel f\left(x_i^a\right)-f\left(x_i^p\right){\parallel }_2^2-{\parallel f\left(x_i^a\right)-f\left(x_i^n\right)\parallel }_2^2+m\right]}_{+}$$

(1)

where ${[·]}_{+}$ denotes the hinge function. However, it is known to cause slow convergence. Ranked List Loss (RLL) [21] is proposed to address this problem by pushing negative samples from a boundary $\alpha$, maintaining a margin m between positive samples and negative samples, and pulling closer positive samples within a boundary $\alpha -m$. This paper adopts Ranked List Loss for features disentanglement to maintain intra-class compactness and inter-class separation; this method helps learn more discriminative features.

3. Proposed Method

Following the tracking-by-detection paradigm, the framework of vehicle tracking is composed of three stages: (1) vehicle detection, (2) single-camera tracking, and (3) multi-camera tracking. In the first stage, video frames are extracted and used as the input into a vehicle detector. The bounding boxes are determined and passed into a feature extraction module. In the single-camera tracking stage, the bounding boxes under the same camera are associated altogether to build vehicle tracklets. Parked cars or bounding boxes with no car are likely to appear in the vehicle tracklets. Therefore, three filters are designed to remove false positive samples. In the last stage, the Tracklet Similarity Network is used to calculate the similarity between tracklets. The Candidates Intersection Ratio is used to refine the similarity and obtain the final similarity results. A CIR association scheme can represent the same tracklet under different cameras as a tree structure. The pipeline of the system framework is depicted in Figure 2.

3.1. Revisiting of Random Walk

The Random Walk algorithm is widely used to refine the probability to a stable status. Let $A\in {\mathbb{R}}^{n\times n}$ denote the pairwise affinities of gallery objects, where n is the number of gallery objects, and $f_0$ denote the initial affinities between the query object and other objects. To keep the sum of each row equal to one ${\sum }_{j}A(i,j)=1$, let S store the value of A. $A(i,j)$ can be normalized as follows.

$$A(i,j)=\frac{expS(i,j)}{{\sum }_{i\ne j}expS(i,j)}$$

(2)

$A(i,j)$ is set to 0 for $i=j$. Then, the distribution can be updated by Equation (3).

$$f_t=f_0{A}^t$$

(3)

where $f_t$ denotes the distribution after t iterations.

To avoid the affinities deviating too far away from the initial affinities, the refined affinities and initial affinities are combined.

$$f_t=\lambda A{f}_{t-1}+(1-\lambda )f_0$$

(4)

After several derivation steps, the equation can be re-written as follows.

$$f_{\infty }=(1-\lambda ){(I-\lambda A)}^{-1}{f}_0$$

(5)

In this paper, we adopt Equation (5) to integrate the Random Walk algorithm in the proposed deep network.

3.2. Tracklet Similarity Network

The Tracklet Similarity Network (TSN) is proposed to calculate the similarity between tracklets. TSN consists of three building blocks, including ReID Block, Disentangle Block, and Similarity Measurement Block, as depicted in Figure 3.

ReID Block. The ReID block aims to learn the feature representation of the tracklet. The ResNet series includes the most popular CNN models in the computer vision field. IBN-Net is usually used to enhance ResNet, since it is appearance-invariant without additional computational consumption. Therefore, ResNet-101-IBN-Net becomes a common backbone model for feature extraction in the vehicle re-identification problem. ResNet101-IBN-Net-a [22] is exploited to extract the feature for each video frame, which is denoted as $\tilde{f}$. The mean and standard deviation of features extracted from each frame within the tracklet are calculated and concatenated. This forms the tracklet feature by the 4096-dimension. Denote $\overrightarrow{f}$ as the tracklet feature. The feature extraction module is fixed during the training process.

Disentangle Block. Videos from different cameras may have different color tones or brightness. Basically, camera specification has its impact on the image feature learned from images from that camera. Therefore, using the camera ID as the answer of classification enforces the deep model to learn camera-specific features. The is the reason to introduce Camera Cross-Entropy Loss to learn the classifier and extract the camera feature. To reduce the impact brought by cameras, we can disentangle the camera feature from the tracklet feature. Technically, the following process can be applied.

The feature representation learned from the ReID Block is used as the input into the camera encoder to learn the camera feature, $\overrightarrow{f_{cam}}$. Several fully connected layers are served as the classifier for camera ID classification. The learning of camera features is optimized via the cross-entropy loss of camera ID. Next, the camera feature is subtracted from the tracklet feature; the feature without camera information denotes $\overrightarrow{f^{\prime }}$. Finally, we use the ranked list loss to ensure that the learned disentangled feature can distinguish the difference between tracklets. The details of ranked list loss are introduced in the next subsection.

Similarity Measurement Block. The similarity measurement block leverages the disentangled feature to measure the similarity between each tracklet. Denote $\overrightarrow{f_{a-e}}$ the residual by subtracting $\overrightarrow{f_e^{\prime }}$ from $\overrightarrow{f_a^{\prime }}$. This extracts more discriminant features for each tracklet. Three fully connected layers are exploited to measure the similarity between each tracklet. Each dimension of the output represents the similarity score between one tracklet to another tracklet. Finally, a random walk is used to refine the similarity. The network is optimized with the refined similarity by cross-entropy loss. Although the random walk matrix provides full affinities for each pair of the tracklets, only the similarity scores between the query and the gallery are required. One can note that during the calculation of similarity between the query and the gallery, the similarities between gallery tracklets are considered altogether. Once we consider the similarity between the query tracklet A to the gallery tracklet B, the similarity is affected by the similarities between tracklet B and tracklet C, D, and E as well. This leverages the information not only between query and gallery but also similarities between gallery tracklets.

Loss Function. The cross-entropy loss is used when learning the camera features for each tracklet. The camera cross-entropy loss can be formulated as follows.

$${\mathcal{L}}_{CAM}=-\frac{1}{N}\sum _{i=1}^N{y}_{cam}^{i}log({\widehat{y}}_{cam}^i)$$

(6)

where ${\widehat{y}}_{cam}^i$ is the camera ID of the prediction, and $y_{cam}^i$ is the target.

The binary classifier distinguishes tracklet similarity through refined affinities from the random walk. Similarly, the classifier is optimized by the similarity binary cross-entropy loss using the following formula:

$${\mathcal{L}}_{SIM}=-\frac{1}{N}\sum _{i=1}^N{y}_{sim}^{i}log({\widehat{y}}_{sim}^i)+(1-y_{sim}^i)log(1-{\widehat{y}}_{sim}^i)$$

(7)

where ${\widehat{y}}_{sim}^i$ is the similarity of the prediction, and $y_{sim}^i$ is the target.

The ranked list loss is adopted to distinguish positive and negative samples. The ranked list loss is defined as:

$${\mathcal{L}}_m(x_i,x_j)=(1-y_{ij}){[\alpha -d_{ij}]}_{+}+y_{ij}{[d_{ij}-(\alpha -m)]}_{+}$$

(8)

where ${[·]}_{+}$ refers to the hinge function, and

$${\mathcal{L}}_{RLL}=\frac{1}{N}\sum _{\forall c,\forall i}\left(0.5\sum _{x_j^c\in \left|P_{c,i}^{*}\right|}\frac{1}{P_c}{\mathcal{L}}_m(x_i^c,x_j^c)\right)+\left(0.5\sum _{x_j^c\in \left|N_{c,i}^{*}\right|}\frac{1}{N_c}{\mathcal{L}}_m(x_i^c,x_j^c)\right)$$

(9)

where x denotes the tracklet feature, and $y_{ij}\in \{0,1\}$ denotes whether the identity of $x_i$ and $x_j$ is the same or not. If $x_i$ and $x_j$ are the same identity, $y_{ij}$ is 1. Otherwise, it is 0. $d_{ij}$ denotes the Euclidean distance between $x_i$ and $x_j$. $\alpha$ is the negative boundary, and m represents the margin of the positive and the negative sample. To minimize ${\mathcal{L}}_m$, the distance between the data sample and the positive sample is constrained to be lower than $\alpha -m$, while the distance between the data sample and the negative sample aims to be larger $\alpha$. In the loss function ${\mathcal{L}}_{RLL}$, $x_i^c$ means data sample $x_i$ is from class c. $|P_{c,i}^{*}|$ denotes the set of all positive samples to $x_i^c$. Thus, the first term of Equation (9) calculates the average distance of data sample $x_i$ and all its positive samples over all data samples and all classes. Similarly, $|N_{c,i}^{*}|$ denotes the set of all negative samples to $x_i^c$. ${\mathcal{L}}_{RLL}$ calculates the average distance from data sample to its positive negative samples.

The total loss for the Tracklet Similarity Network is defined as follows.

$${\mathcal{L}}_{total}={\lambda }_{RLL}{\mathcal{L}}_{RLL}+{\lambda }_{CAM}{\mathcal{L}}_{CAM}+{\lambda }_{SIM}{\mathcal{L}}_{SIM}$$

(10)

3.3. Candidates Intersection Ratio

We develop the CIR tracklet matching metric to evaluate the similarity between two tracklets for the calculation and ranking of tracklet associations iteratively, and the association will be performed hierarchically similar to the standard agglomerative clustering algorithm in hierarchical clustering. We follow a similar notation in considering a tracklet A with matching candidates $L_A$ and another tracklet B with matching candidates $L_B$. The CIR metric for evaluating the association similarity of the matching list of tracklets $L_A$ and $L_B$ is defined as:

$$CIR(L_A,L_B)=\frac{size(L_A\cap L_B)}{min(size\left(L_A\right),size\left(L_B\right))}$$

(11)

Figure 4 shows an example of the CIR metric calculation, where the intersection of the matching list of two tracklets is calculated according to Equation (11) to determine if the two tracklets should be associated together (i.e., merged or linked, and regarded as the tracklets of the same vehicle across views).

The CIR tracklet association algorithm performs iteratively by considering all such $(L_A,L_B)$ pairs across views. Initially, all distinct (unassociated) tracklets are regarded as individual nodes without edges connecting them. The matching list of associated tracklets is constructed by associating the pair $(L_A,L_B)$, and repeated association yields a binary tree-like structure, where the root of the tree indicates the vehicle ID of this association tree. This way, the association tree is constructed by regarding tracklets and their CIR similarity as nodes and edge weight. During tree construction, we enforce that the size of the parent node must be larger or equal to the size of the children nodes. This way, a unique vehicle ID should be retained for each tree throughout the whole iterative association process.

We iteratively compute the CIR score between every tracklet and the tracklets of sizes smaller or equal to the first one. Figure 5 illustrate an example of the tracklet association. In the association of tracklet A to its candidates, we find that tracklet B contains a larger CIR score with tracklet A (where the score > threshold). Therefore, tracklet A is assigned as the parent node of tracklet B. By repeating this process, we can cluster the same tracklets as a tree.

In contrast to a naive greedy best-first tracklet association algorithm that only considers the association of the best-matching pair of tracklets, the proposed CIR tracklet association considers the matching of the whole matching list of tracklets for optimization.

Avoid cyclic tracklet associations. Repeated tracklet association following the above steps by associating tracklets with the best-first selection of CIR scores might result in an unwanted case of cycle associations, where the resulting vehicle ID of the associated set cannot be uniquely defined, causing problematic MTMC results. To avoid such an issue, we enforce the following rules during the CIR association steps to explicitly check and avoid cyclic associations.

• When the size of the matching list of tracklet A is larger than the size of the matching list of tracklet B, we assign tracklet B as the child node of tracklet A.
• If the size of the matching list of tracklet A is equal to the size of the matching list of tracklet B, we check and make sure that tracklet B is not the ancestor of A. If tracklet B is indeed the ancestor of A, we assign tracklet B to be the parent of tracklet A. Otherwise, we assign tracklet A to be the parent of tracklet B.

The CIR association iteration terminates when all pairwise tracklet lists are examined. Afterwards, we assign the vehicle ID of each tree root to all of its descent nodes. This completes the CIR multi-camera vehicle tracking. Algorithm 1 shows the detailed steps of the CIR vehicle tracklet association algorithm in pseudo code.

Algorithm 1: CIR Tracklet Association.

Input: tracklets $T_1,\dots ,T_N$, number of tracklets N   Output: tracklets after association $T_1^{\prime },\dots ,T_N^{\prime }$   for i = 1 to N do    Calculate the pair-wise cosine similarity $S_{i,j}$    Obtain $T_i.L$ the matching list of $T_i$ using $S_{i,j}$    $T_i.L$:= L    //initialize CIR threshold    $T_i.max\_score:=0.33$   for i = 1 to N do    for j = 1 to N do     if (i == j) or ($T_i$.camera == $T_j$.camera) or (size($T_i$) < size($T_j$)) then      continue     $score:=CIR(T_i.L,T_j.L)$     if size($T_i$) == size($T_j$) then      if $is\_ancestor(T_i,T_j)$ then       if $score>T_i.max\_score$ then         $T_i.parent:=T_j$         $T_i.max\_score:=score$       else        if $score>T_j.max\_score$ then         $T_j.parent:=T_i$         $T_j.max\_score:=score$     else      if $score>T_j.max\_score$ then       $T_j.parent:=T_i$       $T_j.max\_score:=score$   for i = 1 to N do    $T_i^{\prime }.id:=T_i.root\_id$

3.4. Other System Components

Vehicle Detection and Single-Camera Tracking. In the begining of the multi-target multi-camera tracking pipeline of Figure 2, Faster R-CNN [23] is adopted as the detector using ResNet-101 as the backbone model. TrackletNet Tracker [24] is used for single-camera tracking. TrackletNet Tracker is a graphical model that effectively reduces computational complexity with temporal information and predicts bounding boxes with epipolar geometry.

Single-Camera Tracklets Filtering. As shown in Figure 6, there is an empty bounding box. The empty bounding box occurs in two situations: (1) the car is at high speed, and (2) the car stays for a long time. We propose the tracklet speed filter and the tracklet stay time filter to deal with the two situations. In addition, we propose the tracklet IoU filter to eliminate the parked car.

The tracklet speed filter is defined as:

$$\frac{v-{\mu }_v}{{\sigma }_v}>{\tau }_v$$

(12)

where v is the tracklet speed, ${\mu }_v$ is the average speed in the same camera, ${\sigma }_v$ is the standard deviation of the speed, and ${\tau }_v$ is the threshold. The tracklet speed is computed with the GPS position and duration time. The GPS position and duration time are projected from the calibration homography matrix provided by AICITY Organization. In this way, the tracklet speed filter removes the outlier of the tracklet speed. The number of removed tracklets is $3\%$ manually controlled. Similarly, the tracklet stay time filter is defined as:

$$\left|\frac{t-{\mu }_t}{{\sigma }_t}\right|>{\tau }_t$$

(13)

where t is the stay time of the tracklet, ${\mu }_t$ is the average stay time in the same camera, ${\sigma }_v$ is the standard deviation of the stay time, and ${\tau }_v$ is the threshold. With the two filters, we can remove most of the empty bounding boxes.

The proposed tracklet IoU filter is defined as:

$$\frac{bo{x}_s\cap bo{x}_e}{bo{x}_s\cup bo{x}_e}>{\tau }_{iou}$$

(14)

where $bo{x}_s$ refers to the first bounding box, and $bo{x}_e$ refers to the last bounding box. If the IoU is greater than the threshold, the car in the tracklet does not move. The filters eliminate the candidate bounding boxes, thereby reducing mismatched tracklets.

The overview pipeline is described in Algorithm 2. The Single-Camera Tracking module (SCT) uses video frames as input and outputs the raw tracklets $T_1$ to $T_{N_{raw}}$. By conducting three filters: speed, staytime, and IoU, tracklets can be refined as $T_1$ to $T_N$. Then, they are used as input to the Tracklet Similarity Network (TSN). The output of TSN is the top N rank list of each query tracklet, which is represented as a matrix of $S^{N\times N}$. After processing the Candidates Intersection Ratio (CIR), it outputs the final associated tracklets.

Algorithm 2: Multi-target multi-camera tracking pipeline.

Input: video frames $f_1,\dots ,f_K$   Output: tracklets after association $T_1^{\prime },\dots ,T_N^{\prime }$   $[T_1,\dots ,T_{N_{raw}}]:=\mathrm{SCT}(f_1,\dots ,f_K)$   $[T_1,\dots ,T_{N_{sp}}]:={\mathrm{filter}}_{speed}\left([T_1,\dots ,T_{N_{raw}}]\right))$   $[T_1,\dots ,T_{N_{st}}]:={\mathrm{filter}}_{staytime}\left([T_1,\dots ,T_{N_{sp}}]\right))$   $[T_1,\dots ,T_N]:={\mathrm{filter}}_{iou}\left([T_1,\dots ,T_{N_{st}}]\right))$   ${\mathbf{S}}^{N\times N}:=\mathrm{TSN}\left([T_1,\dots ,T_N]\right)$   $[T_1^{\prime },\dots ,T_N^{\prime }]:={\mathrm{CIR}}_{association}([T_1,\dots ,T_N],\mathbf{S})$

4. Experimental Results

4.1. Implementation Details

The Disentangle Block comprises fully connected layers; the number of neurons is given by 4096-4096-32. After this, the Similarity Measurement Block comprises three fully connected layers of 4096-2048-1024-1, as illustrated in Figure 3. All fully connected layers in TSN except the classifier are initialized following [25]. In the classifier, the weights are sampled from the normal distribution with a mean of 0 and a standard deviation of 0.001, and the biases are set to 0.

In the single-camera filtering stage, we set the three threshold ${\tau }_{iou}=0.05$, and ${\tau }_s={\tau }_v=1$. If the number of removed tracklets exceeds $3\%$, the thresholds are added to $0.5$. This procedure is repeated until the number of removed tracklets is less than $3\%$. We adopt Stochastic Gradient Descent (SGD) as the optimizer, set the learning rate to $0.001$, and set the weight decay to $0.3$. The $\alpha$ and m are set to $1.4$ and $0.8$, respectively. Following the hyperparameter settings, the training process is converged in five epochs. In the CIR association scheme, we set ${\tau }_{sim}=0.8$ to determine which candidates can be selected into the match list. The hyperparameters are listed in

Table 1. Hyperparameter settings.

Hyperparameter	Value
Hidden layers in the Disentangle Block	4096-4096-32
Hidden layers in the Similarity Measurement Block	4096-2048-1024-1
Optimizer	SGD
Learning rate	0.001
Weight decay	0.3
Batch size	16
Number of epochs	5
$\alpha$	1.4
m	0.8
${\tau }_{iou}$	0.05
${\tau }_s$	1
${\tau }_v$	1
${\lambda }_{RLL}$	1.0
${\lambda }_{CAM}$	1.0
${\lambda }_{SIM}$	1.0

.

The performance analysis of our method is 12.94 fps to process one image during testing and 6.6 h for training. The evaluation is performed with an AMD Ryzen 9 5950X CPU and NVIDIA GeForce RTX 3090 GPU. All experiments are repeated three times, and the average results are reported in the following sections.

4.2. Dataset

We evaluate our method on the CityFlowV2 dataset [26,27]. The dataset contains six scenes captured by a total of 46 cameras. The length of the videos is 215 min. In the dataset, three scenes are used for training, two are used for validation, and one is used for testing. The car is annotated if at least two cameras capture it.

The dataset comprises hard samples and easy samples. A hard sample is a sample that cannot be correctly sorted by cosine similarity. Otherwise, it is an easy sample. For each sample, we choose a query tracklet and the corresponding positive and negative gallery tracklets. The data sample obeys the following rules:

• The number of positive and negative gallery tracklets must be consistent.
• The data sample in which the similarity between any negative gallery tracklet and a query tracklet is greater than the similarity between the positive gallery tracklet and the query tracklet is a hard sample; otherwise, it is an easy sample.
• The ratio of hard samples and easy samples is kept to 1:1.

We sample 668 training data and 668 validation data following these rules.

4.3. Evaluation Metric

IDF1, IDP, and IPR [28] are common evaluation indicators for multi-camera tracking and single-camera tracking and are usually the default indicators for evaluating tracker capabilities. The formula is defined as follows:

$$IDF1=\frac{2IDTP}{2IDTP+IDFP+IDFN}$$

(15)

$$IDP=\frac{IDTP}{IDTP+IDFP}$$

(16)

$$IDR=\frac{IDTP}{IDTP+IDFN}$$

(17)

where $IDTP$, $IDFP$, and $IDFN$ represent the true positive, false positive, and false negative of the vehicle ID, respectively.

MOTA is one of the indicators used to evaluate a single-camera tracker. It does not calculate the accuracy of the object position but uses the missing rate, misjudgment rate, and mismatch rate to evaluate the quality of a tracker. The formula of MOTA is listed as follows.

$$MOTA=1-\sum _t\frac{(F{N}_t+F{P}_t+{\varphi }_t)}{G{T}_t}$$

(18)

where t is the index of the number of frames, $FN$, $FP$, and $GT$ are the number of false negatives, false positives, and ground truths. $\varphi$ is the number of ID switches.

4.4. Ablation Study

Table 2. The results of single-camera tracking filtering.

Filters	IDF1
baseline	40.99%
speed	44.10%
speed + staytime	51.41%
speed + staytime + IOU	61%

presents the results of single-camera tracking filtering. For the baseline method, parked cars or bounding boxes with no car are likely to appear in the vehicle tracklets and thus decrease the performance. The proposed three filters improve the single-camera tracking task by $20\%$ in IDF1 scores. Especially, the IoU filter provides the most significant improvement.

Table 3. Comparison of CIR tracklet association against the top-1 baseline.

Method	IDF1	IDP	IDR
Top-1	42.17%	55.19%	34.13%
CIR	45.80%	64.64%	35.46%

presents the effect of the proposed CIR metric. In this experiment, the similarity metric is cosine similarity, and we use the CIR metric for refinement. The top-1 method in the table means the tracklet is directly paired with the highest cosine similarity candidate.

The proposed CIR association scheme can increase the IDF1 score by $3.6\%$ on the validation set. It significantly reduces the number of false positive samples. The number of reduced samples is more than 10,000, which brings significant improvement by our method. It shows that the CIR association scheme can avoid mismatches due to insufficient pairing information. The $IDP$ and $IDR$ of CIR are also improved by the top-1 method.

Table 4. Performance comparison of TSN modules. DE denotes the disentangle module.

Similarity Metric	Association	IDF1	IDP	IDR
Cosine	Top-1	42.17%	$--$	$--$
Cosine	CIR	45.80%	$--$	$--$
Baseline	CIR	47.55%	59.40%	39.64%
Baseline + RW reranking	CIR	50.08%	69.03%	36.66%
Baseline + RW training	CIR	50.83%	65.36%	41.58%
Baseline + RW training + DE	CIR	52.68%	71.49%	41.72%
Baseline + RW training + DE + RLL	CIR	53.15%	73.49%	41.62%

presents the ablation study brought by each TSN module. The first two rows are the results of the previous experiments. Under the same similarity metric Cosine Similarity, the proposed CIR outperforms the pairing method by over $3.6\%$. The baseline only uses fully connected layers to predict the results. With the proposed CIR, the IDF1 can be improved close to $2\%$. Next, we compare the results by adding random walk in the reranking phase or training phase. IDF1 scores of $50.08\%$ and $50.83\%$ can be obtained, respectively. Among them, the random walk refined results can be used to calculate the loss to achieve more substantial supervision. Therefore, a better IDF1 score is achieved when adding random walk in the training phase.

Finally, we use the disentangle module (DE) to separate the camera feature and the ranked list loss in the training stage. With the disentangle module, we obtain the result of $53.15\%$. This result is $11\%$ higher than the cosine and top-1 methods. In summary, our TSN can separate camera features from tracklet features and uses Ranked List Loss (RLL) to limit negative and positive boundaries. Exploiting random walk into the network allows gallery-to-gallery information to be considered during the training. It makes the similarity of the prediction from the model more accurate. The $IDP$ goes up to $73.49\%$ when all components are considered. The $IDR$ score is relatively close when RW, DE, and RLL are exploited.

Table 5. The results of the data sampling method.

Dataset	IDF1	IDP	IDR
Easy	48.43%	76.81%	35.36%
Hard	49.90%	78.69%	36.54%
Merge	53.15%	76.89%	41.62%

shows the results of using different samples to train TSN. Easy represents that the dataset contains only easy samples, while Hard represents that the dataset contains only hard samples. Merge represents the mixture of easy samples and hard samples. The results indicate that sampling with an easy sample achieves a result of $48.43\%$, while using a hard sample can achieve a result of $49.90\%$. After mixing the hard and easy samples, the model benefits from the cosine similarity and avoids TSN from being biased toward hard samples. The IDF1 can be improved to $53.15\%$. The IDP is decreased a little while IDR is improved over $5\%$.

Figure 7 shows the loss curves of training and validation; following the settings, the training converges at five epochs.

5. Discussion

The proposed Tracklet Similarity Network (TSN) includes several novel components. Instead of learning image features from a conventional CNN model, Re-Id Block learns the feature representation for each tracklet. Then, Disentangle Block learns the camera-independent feature by exploiting camera loss and ranked list loss functions to achieve feature disentanglement. Lastly, Similarity Measurement Block leverages strong similarities information from the ranking list of tracklets via a random walk scheme rather than using the tracklet with the highest score alone from a query tracklet. The experimental results demonstrate that each component provides incremental improvement for the IDF1 score. The associations of vehicles across cameras from TSN can be further refined by the CIR Tracklet association scheme to achieve multi-camera vehicle tracking. Compared to the recent multi-camera vehicle tracking framework in

Table 6. Comparison to the state-of-the-art methods.

Method	IDF1
Ren et al. [29]	57.63%
Yang et al. [2]	54.58%
Ours	53.15%

, our work is more devoted to developing a multi-camera tracking model and scheme. However, in the front end, the results of single-camera tracking also play an important role in the whole framework, since multi-camera tracking is applied based on the results of single-camera tracking. If the accuracy of vehicle detection and single-camera tracking models can be improved, the overall IDF1 can be also significantly increased. In this paper, only baseline methods of vehicle detection and vehicle re-identification are exploited in the multi-target multi-camera tracking framework. For interested readers, please refer to [26] for the state-of-the-art results. In the future, our model can be extended to include the single-camera tracking model as a larger end-to-end tracking system. This can be expected to see higher tracking accuracy.

6. Conclusions

The Deep Tracklet Similarity Network is proposed to calculate the similarity between tracklets. Disentangle techniques are applied to reduce the camera information from the tracklet features. In addition, the ranked list loss is adopted and shows significant improvement. The results are refined by the random walk to obtain better tracklet similarities. The Candidates Intersection Ratio scheme is proposed for tracklet association. It leverages the information of the tracklet match list to adjust the tracklet similarity. We integrate tracklets with the same ID into a tree structure in the CIR association scheme. Compared with the existing method, our method significantly improves the IDF1 score by $11\%$.