Absent Color Indexing: Histogram-Based Identification Using Major and Minor Colors

Abstract

The color histogram is a statistical behavior for robust pattern search or matching; however, difficulties have arisen in using it to discriminate among similar objects. Our method, called absent color indexing (ABC), describes how to use absent or minor colors as a feature in order to solve problems while robustly recognizing images, even those with similar color features. The proposed approach separates a source color histogram into apparent (AP) and absent (AB) color histograms in order to provide a fair way of focusing on the major and minor contributions together. A threshold for this separation is automatically obtained from the mean color histogram by considering the statistical significance of the absent colors. After these have been separated, an inversion operation is performed to reinforce the weight of AB. In order to balance the contributions of the two histograms, four similarity measures are utilized as candidates for combination with ABC. We tested the performance of ABC in terms of the F-measure using different similarity measures, and the results show that it is able to achieve values greater than 0.95. Experiments on Mondrian random patterns verify the ability of ABC to distinguish similar objects by margin. The results of extensive experiments on real-world images and open databases are presented here in order to demonstrate that the performance of our relatively simple algorithm remained robust even in difficult cases.

1. Introduction

Pattern search or matching is the task of finding targets through the use of images or through statistic or deterministic features extracted from images. Proposed image features for pattern search have included gray features [1], texture features [2], color features [3], and convolution features [4], where color features provide universally successful cues for identification of individuals in many applications. These have been extensively utilized in pattern search, computer vision, and image processing [5,6,7]. Analysis of color features plays a vital role in various tasks, including object matching, background subtraction, video tracking, and image retrieval [8,9,10].

Among color features, color histograms, a statistical measure of color distribution in images, have been widely used to describe color information. Color histograms are beneficial in that the color distribution in the template image is recorded without complicated learning processes, they feature strong robustness against object deformation and scale changes, and they provide effective statistics for utilizing discrete color distributions or histograms over a given color space. Color histogram-based approaches [11] can thus be effectively used to search for objects. However, these approaches reduce performance in discriminatin between similar objects, because any histogram trades positional information about the pixels for flexibility in matching. Swain et al. [12] proposed a combination of color histograms and intersection for searching a target location by histogram, where each bin represents coarse color frequencies in a given color space or system. Although such color indexing (CI) works well with changes in size or posture, it has difficulties handling changes in noise and illumination. To address these problems, Stricker et al. [13] used a relation of color histogram bins to generate a cumulative color histogram (CCH), whereby fixing the color order accumulates bins with smaller color frequencies into bins with relatively larger color values. This reduces sensitivity to noise interference and illumination changes, while the accumulated histogram reduces sensitivity in the process of object discrimination. The color co-occurrence histogram (CH) [14] method utilizes distance information of pairs of certain colored pixels in an image space to generate a color co-occurrence histogram and set a tolerance parameter to apply occlusion and deformation conditions in the algorithm. This can be costly in terms of computation time, and although the process of classifying colors uses Euclidian distances in RGB space, it can sometimes misclassify colors. Han et al. [15] introduced the fuzzy color histogram (FCH) based on the fuzzy c-means clustering algorithm, where an adjustable fuzzy membership matrix is defined to deal with different noise interferences and applications. Verma et al. [16] improved the fuzzy color histogram method (TFCM) and combined it with a spatial filter to solve the illumination problem for template matching. A triangular membership function is proposed to connect each bin for fuzzy logic implementation in a color histogram. The above-mentioned color histogram-based methods focus on bins with high contribution or frequency to design similarity measures.

Meanwhile, numerous methods based on the merits of color histograms have been proposed. Wang [17] introduced an image retrieval method based on color histograms of local feature regions and combined it with a Harris–Laplace detector to obtain spatial information before using the color histograms. The scheme in the method proposed by Varish et al. [18] used a color histogram and the wavelet transform method, applying color and texture features to enhance performance. Liu et al. [19] combined color histograms with local binary pattern-based features for better classification. This approach generally shows the possibility of combining color histograms or features with other schemes in classification tasks, to which our method for utilizing absent colors as a new feature type may contribute. Generally, these are not pure ways of using color histograms, and instead combine them with other features to achieve experimental results based on color histograms. This makes the overall algorithm more complex, preventing advantages in terms of computation time.

In this research, we explore ABC as a color feature for use in robust search or matching methods for objects with varying sizes, postures, and levels of occlusion in the scene. The key contribution of this study is that the proposed approach provides a balanced method of focusing on the relative importance of AP and AB by separating the original color histogram. In our prior work [20,21], ABC was formulated as the decomposition of a color histogram into two disjointed histograms using fixed parameters to achieve good performances in terms of feasible matching. In this study, we developed an automatic threshold, $h_{\mathrm{T}}$, designed to obtain more effective absent colors. In particular, we propose a more sophisticated version involving four specific techniques, including a mean color histogram, a novel algorithm based on statistical significance as defined for the sorted mean color histogram, a trial to investigate the availability of representative similarity measures combined with the proposed method, and a statistical treatment of the margin obtained by matching. The results of an evaluation of actual scenes and datasets are provided here in order to demonstrate the robust performance of our proposed ABC method.

2. Absent Color Indexing

The motivation behind our proposed approach is to enhance colors in cases where the objects to be searched have few positive features with existing color features. For example, when identifying individuals, eye color is not a major color feature by volume, however, it can provide a significant feature for identification.

2.1. Definition of Absent Colors

By introducing absent colors to realize this idea, we propose a novel approach to utilizing color histograms for robust pattern identification. This approach focuses on low-frequency colors in any two histograms of the target image. When evaluating histogram similarity, there must be four conditional combinations with respect to high and low frequencies in their bins. If both bins include high frequencies, they have high similarity, and if one is low and the other is high, it provides a low contribution to the total similarity. The case where both have low frequencies is conventionally evaluated as having a low contribution to similarity, however, the recognition of this as a common characteristic in our trials formalizes their treatment in similarity evaluations as an effective new approach. However, contamination by additional noise must be prevented in histograms when designing algorithms, because noise can easily influence such low frequencies.

Table 1. Combinations of apparent (AP) and absent (AB) colors.

Histograms		H
		${\mathit{h}}^{\mathbf{AP}}$	${\mathit{h}}^{\mathbf{AB}}$	0
G	$g^{\mathrm{AP}}$	$(h^{\mathrm{AP}},g^{\mathrm{AP}})$	$(h^{\mathrm{AB}},g^{\mathrm{AP}})$	$(0,g^{\mathrm{AP}})$
	$g^{\mathrm{AB}}$	$(h^{\mathrm{AP}},g^{\mathrm{AB}})$	$(h^{\mathrm{AB}},g^{\mathrm{AB}})$	$(0,g^{\mathrm{AB}})$
	0	$(h^{\mathrm{AP}},0)$	$(h^{\mathrm{AB}},0)$

defines histograms H and G in the same color space or specification as consisting of high-frequency bins ($h^{\mathrm{AP}}$ and $g^{\mathrm{AP}}$), low-frequency bins ($h^{\mathrm{AB}}$ and $g^{\mathrm{AB}}$), and bins with no entities (0 and 0). Below, we propose a detailed scheme for defining these using a reasonable threshold value. As Table 1 shows, we define apparent colors (AP) as those in high-frequency bins, while colors in low- and null-frequency bins provide candidate absent colors (AB). There are nine possible arrangements of bins having the same colors, as shown in Table 1. In the top row and the leftmost column, every pair includes $h^{\mathrm{AP}}$ or $g^{\mathrm{AP}}$, which are evaluated for similarity in almost all methods based on color histograms as features; if both bins are AP, then their contribution to similarity may be larger, while those including an AB element make a lower contribution. As mentioned in the previous section, our motivation here is to focus on other items in the table, namely, those such as $(h^{\mathrm{AB}},g^{\mathrm{AB}})$ where both elements are AB. Such pairs are conventionally evaluated as minor elements in similarity calculations. Note that the last possible combination, 0 and 0, is never evaluated in any similarity evaluation scheme, because they represent colors not present in target images and are thus beyond the scope of consideration.

2.2. AP and AB Histograms

Illumination effects are handled by transforming RGB images to CIE L*a*b* color space and removing the L* channel, avoiding sensitivity reduction by enhancing the contribution of absent colors. We use the a* and b* channels to establish a two-dimensional color space from the CIE L*a*b* color space. When establishing many color features in color statistics, the CIE L*a*b* color space has a broader color range that is closer to human vision, allowing brightness to be separated as an independent coordinate (the L* channel).

As an example, Figure 1 shows two images, 1 and 2. Let their color space have ${\beta }_1\times {\beta }_2$ bins or quantization. In order for images 1 and 2 to be evaluated as matched or unmatched, their two-dimensional color histograms $H={\left\{h_{ij}\right\}}_{\left(i,j\right)=\left(1,1\right),\cdots ,\left({\beta }_1,{\beta }_2\right)}$ and $G={\left\{g_{ij}\right\}}_{\left(i,j\right)=\left(1,1\right),\cdots ,\left({\beta }_1,{\beta }_2\right)}$ are represented in terms of the relative frequencies of classified colors in their bins, confirming their summation to be unity. Moving forward, for explanation purposes we use H here as a representative example. Figure 2 shows the relative histograms, H and G, for images 1 and 2, respectively.

From the original two-dimensional color histogram H, we create a pair of two-dimensional histograms, $H=H^{\mathrm{P}}+H^{{}^{\prime }}$, $H^{\mathrm{P}}=\left\{h_{ij}^{\mathrm{P}}\right\}$, and $H^{{}^{\prime }}=\left\{h_{ij}^{{}^{\prime }}\right\}$ as complements. We have omitted the subscripts i and j in the formula for simplicity and to avoid confusion.

$$h^{\mathrm{P}}=h(1-\varphi \left(h\right))=h\overline{\varphi \left(h\right)},$$

(1)

$$h^{\prime }=h\varphi \left(h\right),$$

(2)

where $\varphi \left(x\right)$ is an indicator function that shows classification such that $\varphi \left(x\right)=1$ if $x\le h_{\mathrm{T}}$ and $\varphi \left(x\right)=0$ otherwise. The threshold value, $h_{\mathrm{T}}$, is an important parameter in this study and is defined in Section 2.3. Note that the initial values of the other elements without any indication in the above definition are set to 0. $H^{\mathrm{P}}$ includes major colors frequently observed in image 1, while $H^{{}^{\prime }}$ contains minor colors that are denoted as “absent” colors because their occurrence is infrequent within the image. Both have the same structure as the two-dimensional histogram H. The elements $h^{\mathrm{P}}$ and $h^{{}^{\prime }}$ represent the color frequencies. We expect to systematically and effectively utilize information included in the histogram at low frequencies through the decomposition process.

From $H^{{}^{\prime }}$, we intend to make an opposite counterpart for the major color histogram as a complementary feature in the original histogram. However, certain special cases, such as zero frequency, should be considered. To briefly explain the cases of zero frequency ($h^{{}^{\prime }}=0$) during the inversion process of the absent color histogram $H^{{}^{\prime }}$, we temporarily assume that $H^{{}^{\prime }}$, H, and G are one-dimensional histograms only in Figure 3.

In the first case (I) shown in Figure 3, if $h^{{}^{\prime }}=0$ and $h>h_{\mathrm{T}}$, then $h^{\mathrm{N}}=0$; In cases (II) and (III), if $h^{{}^{\prime }}=0$, $h=0$ and $g>0$, then $h^{\mathrm{N}}=h_{\mathrm{T}}$; finally, in case (IV), if $h^{{}^{\prime }}=0$, $h=0$ and $g=0$, then $h^{\mathrm{N}}=0$.

Table 2. Value of $h^{\mathrm{N}}$ in conditions of $h^{{}^{\prime }}=0$.

${\mathit{h}}^{\mathbf{N}}$	${\mathit{h}}^{{}^{\prime }}=0,\mathit{h}>{\mathit{h}}_{\mathbf{T}}$	${\mathit{h}}^{{}^{\prime }}=0,\mathit{h}=0$
$g>0$	0	$h_{\mathrm{T}}$
$g=0$	0	0

summarizes these three transformations to provide a clear explanation.

After the above-mentioned inverting process, the absent color histogram $H^{\mathrm{N}}=\left\{h_{ij}^{\mathrm{N}}\right\}$ is defined to represent small or zero frequencies in the original one, as follows:

$$h^{\mathrm{N}}=(h_{\mathrm{T}}-h^{{}^{\prime }})\varphi \left(h\right)\psi \left(h\right)+h_{\mathrm{T}}\overline{\psi \left(h\right)}\psi \left(g\right).$$

(3)

Here, $\psi \left(x\right)$ is another indicator function that satisfies the following conditions: $\psi \left(x\right)=1$ if $x>0$; otherwise, $\psi \left(x\right)=0$. Finally, it is necessary to normalize both $H^{\mathrm{P}}$ and $H^{\mathrm{N}}$ to satisfy the condition that all components should sum to 1.

Figure 4 shows major color histograms, $H^{\mathrm{P}}$ and $G^{\mathrm{P}}$, and absent color histograms, $H^{\mathrm{N}}$ and $G^{\mathrm{N}}$, for images 1 and 2, respectively.

2.3. Threshold Definition

In the algorithm described in the last section, the threshold, $h_{\mathrm{T}}$, plays a main role in defining apparent and absent colors. This section describes how to define $h_{\mathrm{T}}$ in order to provide a meaningful algorithm with effective performance. We first introduce the mean color histogram, M, to find an averaged tendency of the color distributions in two histograms to be compared, then use it realize a stable definition of the threshold. $M={\left\{m_{ij}\right\}}_{\left(i,j\right)=\left(1,1\right),\cdots ,\left({\beta }_1,{\beta }_2\right)}$ is defined as

$$m_{ij}=\frac{h_{ij}+g_{ij}}{2}.$$

(4)

Generating the mean color histogram is a critical phase before threshold selection. The proportion of each color in the histogram is statistically analyzed for matching images, thereby improving the rationality and dynamism of threshold determination and guaranteeing the accuracy of the final similarity measurement.

We next convert the two-dimensional histogram M to a sorted one-dimensional histogram $M^{\mathrm{sorted}}$, as follows:

$$M^{\mathrm{sorted}}=\left\{m_{i-1}^{\mathrm{sorted}}\ge m_i^{\mathrm{sorted}}\right\}.$$

(5)

The threshold value, $h_{\mathrm{T}}$, can be defined by the following equation through use of an order index s related to a significant rate $\alpha$, by which we can separate the set of all bins into sets of apparent and absent colors by considering the rarity of absent colors in the images.

$$h_{\mathrm{T}}=\frac{m_s^{\mathrm{sorted}}+m_{s+1}^{\mathrm{sorted}}}{2}.$$

(6)

$$s=argmin\left\{\sum _{i=1}^s{m}_i^{\mathrm{sorted}}\ge 1-\alpha \right\}.$$

(7)

Using parameter s, a stable decomposition can be performed with no “chattering” near the threshold value in comparison with a constant threshold. In the definition of the absent color histogram, because zero frequency plays an important role in eliminating any noise effects, we must remove near-zero frequencies in the absent color histogram. For example, we compare frequencies as $0.2\times h_{\mathrm{T}}$ in our experiments. Figure 5 shows the mean color histogram for histograms H and G. Figure 6 is a Pareto chart [22] for this example. We can use this information to determine the significant rate, $\alpha$, which represents effectiveness in revealing the rareness of absent colors and contributes to setting of the threshold value.

Note that the mean color histograms can be used both to derive the threshold, $h_{\mathrm{T}}$, and to calculate similarities based on apparent and absent colors, as described in the next section.

2.4. Similarity Measures

Many measures for testing similarity between two histograms or probability density functions have been proposed, including intersection, chi-square distance, Jensen–Shannon divergence, and Bhattacharyya distance. This section describes these measures in combination with our ABC in order to demonstrate its universality in evaluating the similarity of images, as described in Section 3.2. We expect this universality of ABC to make it useful as an effective scheme for many applications.

2.4.1. Intersection

Intersection [23] has been used in many studies and applications because of its simplicity. It is defined for two same-sized histograms, H and G, as follows:

$$\begin{array}{c}\hfill I\left(H,G\right)=\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}min\left\{h_{ij},g_{ij}\right\}.\end{array}$$

(8)

For the two histogram types proposed in this paper, we define a scheme for combining the two intersections using weighting coefficients, as follows:

$$S=w_{\mathrm{P}}I\left(H^{\mathrm{P}},G^{\mathrm{P}}\right)+w_{\mathrm{N}}I\left(H^{\mathrm{N}},G^{\mathrm{N}}\right),$$

(9)

where $w_{\mathrm{P}}$ and $w_{\mathrm{N}}$ are weights for balancing the two types of intersections using the constraint $w_{\mathrm{P}}+w_{\mathrm{N}}=1$.

2.4.2. Chi-Square Distance

The chi-square test is a non-parametric test that is mainly used to compare two or more sample rates. In order to measure similarity between two histograms, we use the ${\chi }^2$ statistic to observe frequencies. We define our version using the mean color histogram as follows:

$$\begin{array}{c}\hfill {\chi }^2\left(H,G\right)=\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}\frac{{\left(h_{ij}-m_{ij}\right)}^2}{m_{ij}}+\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}\frac{{\left(g_{ij}-m_{ij}\right)}^2}{m_{ij}},\end{array}$$

(10)

where $m_{ij}$ is the element of the mean color histogram. Equation (10) can be simplified as

$${\chi }^2\left(H,G\right)=\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}\frac{{\left(h_{ij}-g_{ij}\right)}^2}{\left(h_{ij}+g_{ij}\right)}.$$

(11)

The total distance between the histograms is defined using the following weights:

$$D_{{\chi }^2}=w_{\mathrm{P}}{\chi }^2\left(H^{\mathrm{P}},G^{\mathrm{P}}\right)+w_{\mathrm{N}}{\chi }^2\left(H^{\mathrm{N}},G^{\mathrm{N}}\right).$$

(12)

2.4.3. Jensen–Shannon Divergence

JS divergence [24] is a symmetric divergence measurement based on Kullback–Leibler divergence [25]. By calculating divergences between histograms, a larger divergence indicates smaller correlative relation and smaller similarity between the histograms. This is defined as

$$D_{\mathrm{KL}}\left(H\Vert G\right)=\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}{h}_{ij}log\frac{h_{ij}}{g_{ij}}.$$

(13)

We define a particular version of JS divergence as follows using the mean color histogram M:

$$D_{\mathrm{JS}}\left(H,G\right)=\frac{1}{2}{D}_{\mathrm{KL}}\left(H\Vert M\right)+\frac{1}{2}{D}_{\mathrm{KL}}\left(G\Vert M\right).$$

(14)

In this formula, the antilogarithm cannot be zero in logarithm calculations such that $log\frac{h_{ij}}{g_{ij}}$ can take a relatively minimum value. The JS divergence-based distance between the histograms is defined as

$$D_{\mathrm{JSD}}=w_{\mathrm{P}}{D}_{\mathrm{JS}}\left(H^{\mathrm{P}},G^{\mathrm{P}}\right)+w_{\mathrm{N}}{D}_{\mathrm{JS}}\left(H^{\mathrm{N}},G^{\mathrm{N}}\right).$$

(15)

2.4.4. Bhattacharyya Distance

In statistics, the Bhattacharyya distance [26] is often used to measure the dissimilarity of two discrete or continuous probability distributions. It is closely related to the Bhattacharyya coefficient, which measures overlap between two statistical samples or populations. The Bhattacharyya distance can be used to determine relative relationships between two samples or to determine differences between two classes. Thus, the two histograms can be considered as discrete probability distributions; the formula is

$$\mathit{BD}=-ln\left(BC\left(H,G\right)\right),$$

(16)

where $0\le \mathit{BD}\le \infty$, $0\le \mathit{BC}\le 1$ and $\mathit{BC}\left(H,G\right)$ is the Bhattacharyya coefficient,

$$\mathit{BC}\left(H,G\right)=\sum _{\left(i,j\right)=\left(1,1\right)}^{\left({\beta }_1,{\beta }_2\right)}\sqrt{h_{ij}{g}_{ij}}.$$

(17)

The distance between two sets of apparent and absent color histograms is defined as

$$D_{BD}=w_{\mathrm{P}}\mathit{BD}\left(H^{\mathrm{P}},G^{\mathrm{P}}\right)+w_{\mathrm{N}}\mathit{BD}\left(H^{\mathrm{N}},G^{\mathrm{N}}\right).$$

(18)

3. Experimental Evaluation

In this section, we consider the numerous challenges of ABC matching. In order to analyze the performance of our method, we derive a signal-to-noise (SNR) formula for adding reasonable noise to the Mondrian random pattern, as shown in Figure 7. We utilize three common measurements, namely, the F-measure, Margin, and Fisher ratio (FR), to explain the essential strengths of the ABC. Finally, we utilize real data to evaluate the matching effect of the ABC. The pseudo-code of ABC approach is shown in Algorithm 1.

Algorithm 1: Proposed ABC approach

Input: Reference image $S_{\mathrm{R}}$ and compared image $S_{\mathrm{S}}$.          Initial parameters: ${\beta }_1=10,{\beta }_2=10,\alpha =0.2,$          $w_{\mathrm{P}}=0.6,w_{\mathrm{N}}=0.4$ Output: Target location $L_{\mathrm{T}}$ in the searched image. repeat    1: Crop the compared image from position $(i,j)$ in the scene.    2: Generate two-dimensional color histograms H and G by a* and b* channels.    3: Divide color histograms into apparent color histograms $H^{\mathrm{P}}$, $G^{\mathrm{P}}$ and absent color histograms $H^{{}^{\prime }}$, $G^{{}^{\prime }}$.    4: Invert absent color histograms $H^{{}^{\prime }}$ and $G^{{}^{\prime }}$ to $H^{\mathrm{N}}$ and $G^{\mathrm{N}}$.    5: Calculate similarity $R_{(i,j)}$ by $H^{\mathrm{P}}$ and $G^{\mathrm{P}}$, $H^{\mathrm{N}}$ and $G^{\mathrm{N}}$. until all locations are scanned, then find position $L_{\mathrm{T}}$ with $max\left(R_{(i,j)}\right)$;

3.1. Preparation of Signal and Noise

We first describe a color image, $I_{\mathrm{t}}=\left\{t_{ij}\right\}$. We denote $I_{\mathrm{t}}$ by signal $I_{\mathrm{s}}=\left\{s_{ij}\right\}$, while noise $I_{\mathrm{n}}=\left\{n_{ij}\right\}$ is defined as

$$I_{\mathrm{t}}=I_{\mathrm{s}}+I_{\mathrm{n}}.$$

(19)

The variance of image $I_{\mathrm{t}}$ is represented as

$${\sigma }_{\mathrm{t}}^2=E{\left(I_{\mathrm{t}}-{\mu }_{\mathrm{t}}\right)}^2=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{\left[I_{\mathrm{t}}\left(i,j\right)-{\mu }_{\mathrm{t}}\right]}^2$$

(20)

where ${\mu }_{\mathrm{t}}={\mu }_{\mathrm{s}}+{\mu }_{\mathrm{n}}$ is the average value in image $I_{\mathrm{t}}$, ${\mu }_{\mathrm{s}}$ and ${\mu }_{\mathrm{n}}$ are the average values of $I_{\mathrm{s}}$ and $I_{\mathrm{n}}$, $E\left(I_{\mathrm{n}}\right)={\mu }_{\mathrm{n}}=0$ describes the noise distributions obeyed by certain balanced and unbiased Gaussian distributions, and m and n are the sizes of the images. As mentioned in Equation (20),

$$\begin{array}{cc}\hfill {\sigma }_{\mathrm{t}}^2& =E{\left(\left(I_{\mathrm{s}}+I_{\mathrm{n}}\right)-\left({\mu }_{\mathrm{s}}+{\mu }_{\mathrm{n}}\right)\right)}^2\hfill \\ \hfill & =E\left({\left(I_{\mathrm{s}}-{\mu }_{\mathrm{s}}\right)}^2\right)+2E\left(\left(I_{\mathrm{s}}-{\mu }_{\mathrm{s}}\right)\left(I_{\mathrm{n}}-{\mu }_{\mathrm{n}}\right)\right)+E\left({\left(I_{\mathrm{n}}-{\mu }_{\mathrm{n}}\right)}^2\right)\hfill \\ \hfill & =E\left({\left(I_{\mathrm{s}}-{\mu }_{\mathrm{s}}\right)}^2\right)+E\left({\left(I_{\mathrm{n}}-{\mu }_{\mathrm{n}}\right)}^2\right)\hfill \\ \hfill & ={\sigma }_{\mathrm{s}}^2+{\sigma }_{\mathrm{n}}^2,\hfill \end{array}$$

(21)

where ${\sigma }_{\mathrm{s}}^2$ and ${\sigma }_{\mathrm{n}}^2$ represent the variance of the signal and noise in image $I_{\mathrm{s}}$ and $I_{\mathrm{n}}$. The formulas are calculated as

$${\sigma }_{\mathrm{s}}^2=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{\left[I_{\mathrm{s}}\left(i,j\right)-{\mu }_{\mathrm{s}}\right]}^2.$$

(22)

$${\sigma }_{\mathrm{n}}^2=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{\left[I_{\mathrm{n}}\left(i,j\right)-{\mu }_{\mathrm{n}}\right]}^2.$$

(23)

The signal-to-noise ratio (SNR) is calculated as

$$SNR=10log\frac{{\sigma }_{\mathrm{s}}^2}{{\sigma }_{\mathrm{n}}^2}=10log\frac{{\sigma }_{\mathrm{t}}^2-{\sigma }_{\mathrm{n}}^2}{{\sigma }_{\mathrm{n}}^2}=10log\left(\frac{{\sigma }_{\mathrm{t}}^2}{{\sigma }_{\mathrm{n}}^2}-1\right).$$

(24)

Next, we analyzed the standard deviations ${\sigma }_{\mathrm{n}}$ of noise in stationary background regions of the datasets [27,28], where we collected the small regions of interest (ROI) with sizes of $\left(100,100\right)$ at the same positions in their background without any moving things: $\left(61,531\right)$ in 50 frames of Biker, and $\left(1,1\right)$ in 50 frames of Walking, respectively. Then, ${\sigma }_{\mathrm{n}}$ can be calculated from numerous estimated noise values in the difference images; $I_{\mathrm{n}}=I_{\mathrm{t}}-\overline{I_{\mathrm{t}}}$. ${\sigma }_{\mathrm{t}}$ is from one of 50 images in each dataset.

Table 3. Statistical characteristics of background noise.

		R	G	B
Biker	${\sigma }_{\mathrm{n}}$	0.958	0.903	1.143
	SNR	35.09	35.20	34.17
Walking	${\sigma }_{\mathrm{n}}$	1.021	0.953	1.028
	SNR	36.48	37.90	37.23

shows the SNR in the sampled images of these databases. The average value of SNR in these two datasets is 36.01. In order to add noise to the noise-free Mondrian random pattern, we utilized the knowledge on noise obtained from Table 3 and thereafter added the noise of unbiased Gaussian distributions to the individual channels of the color image by SNR = 36, 33, and 30, where a reduced SNR implies increased noise.

3.2. Evaluation of ABC in Combination with Four Similarity Measures

In these experiments, we added artificial noise to the original images in order to investigate the performance of our method against others. Figure 7 shows an example of a synthetic Mondrian random pattern with no structures except for a colored circle as an elemental shape.

We next add noise to the color image. We first separate the individual channels of the color image and then add unbiased Gaussian noise of SNR = 36, 33, and 30 to each channel before finally merging the channels into a color image. Therefore, CI, CCH, TFCM, and ABC can be tested by use of these Mondrian random patterns embedded as typical additional noise. We tested four similarity measures, as follows: first, reference images of size $50\times 50$ in Figure 7 were randomly selected, then searches were performed for the best matching positions in each noisy version. To evaluate each similarity measure, we use the F-measure, as follows:

$$Precision=\frac{TP}{TP+FP}$$

(25)

$$Recall=\frac{TP}{TP+FN}$$

(26)

$$F-measure=\frac{2\times Precision\times Recall}{Precision+Recall},$$

(27)

where TP and FP indicate the numbers of true positives and false positives, respectively, defined by a threshold value of intersection over union (IoU). TP can thus have an IoU exceeding 0.75, while FP has a value below threshold 0.25. FN represents the number of false negatives, where TP+FN is the entirety of the target image. F-measure is the harmonic mean of Precision and Recall.

Table 4. Performance of ABC with various similarity measures.

	F-Measure
SNR	36	33	30
ABC + Intersection	0.973	0.965	0.952
ABC + Chi-square	0.976	0.971	0.961
ABC + JS divergence	0.980	0.974	0.971
ABC + Bhattacharyya	0.974	0.970	0.968

shows a performance evaluation in matching using ABC-based similarity measures under different noise conditions. We found that in all cases, ABC could be used in combination with these similarity measures using experimental parameters $I=10$, $J=10$, $\alpha =0.2$, $w_{\mathrm{AP}}=0.6$, and $w_{\mathrm{AB}}=0.4$. We use Intersection in later experiments because of its similarity in the range from 0 to 1, which makes it easier to compare with other approaches.

3.3. Analysis by Margin and Fisher Ratio

In this section, we evaluate performance by comparing ABC with three other color histogram-based matching methods, CI [12], CCH [13], and TFCM [16], as these methods are frequently cited and have been used in many papers and studies.

Margin [29] is defined as the difference in similarity between the best and second-best matching positions. The best-matching position is the maximum value at which the area of IoU between ground truth and the searched position should exceed 0.90. The second-best matching position is the maximum value at which the area of IoU is less than 0.15. A larger margin provides stronger distinguishability in avoiding interference by similar objects. Figure 8 shows example profiles. These are projected onto the horizontal axis by 3D similarity profiles for ABC, CI, CCH, and TFCM. We found that the margin of ABC was much larger than that of the other three methods.

In order to statistically investigate the performance of ABC and others in terms of margin, we independently extracted from random positions 100 reference images sized $50\times 50$, then searched for them within the reference image itself or in versions contaminated with noise. For each image, we were able find the best position or peak as peak1 and the second-best position as peak2. From the noisy image, we separated 100 pairs of peak1 and peak2 and created their histograms, as shown in Figure 9. The similarity ranges of ABC, CI, and CCH are the same from 0.3 to 1, and that of TFCM is the same from 0.8 to 1. In order to better clarify this observation, we utilized Fisher’s ratio (FR) to evaluate the discriminating power of any two-class discriminator. FR is defined as

$$FR=\frac{{\left(m_{\mathrm{p}1}-m_{\mathrm{p}2}\right)}^2}{{\sigma }_{\mathrm{p}1}^2+{\sigma }_{\mathrm{p}2}^2},$$

(28)

where $m_{\mathrm{p}1}$, $m_{\mathrm{p}2}$, ${\sigma }_{\mathrm{p}1}^2$, and ${\sigma }_{\mathrm{p}2}^2$ represent the means and variances, respectively, of the peak1 and peak2 classes. Here, an increased numerator indicates an increased distance or interval between classes, and a reduced denominator indicates the greater compactness of each class; thus, FR can be greater when the feature space for classification is better for discrimination or identification.

Table 5. Margins in CI, CCH, TFCM and ABC.

		CI		CCH		TFCM		ABC
SNR	Class	Mean	Std	Mean	Std	Mean	Std	Mean	Std
36	Peak1	0.952	0.034	0.935	0.041	0.999	$3.6\times {10}^{-4}$	0.946	0.049
	Peak2	0.583	0.060	0.590	0.069	0.902	0.020	0.456	0.055
	FR	28.67		18.28		21.75		43.77
33	Peak1	0.937	0.038	0.915	0.042	0.999	$2.2\times {10}^{-4}$	0.935	0.057
	Peak2	0.575	0.058	0.591	0.072	0.910	0.021	0.455	0.064
	FR	26.52		14.80		16.95		31.18
30	Peak1	0.919	0.040	0.898	0.045	0.998	$5.7\times {10}^{-4}$	0.914	0.058
	Peak2	0.579	0.053	0.592	0.065	0.908	0.024	0.465	0.064
	FR	25.79		14.85		14.07		26.58

summarizes the FR obtained by CI, CCH, TFCM, and ABC in cases of noisy Mondrian random patterns in which we varied the SNR from 36 to 30.

Table 5 indicates that the FR of ABC was superior to those of CI, CCH, and TFCM with the case of $\mathit{SNR}=$ 36, 33, and 30, and furthermore, that it maintained this performance even in cases with a reduced SNR. As shown in Figure 10, the horizontal axis represents the change in variance, and the vertical axis represents the change in mean value, which is the size of the average margin. In these experiments, we used every set of ten images to observe those distributions in the means and variances that construct the FR values. As a result, there are two types of similarity measures, corresponding to the two approaches used to increase FR: first, to increase the difference or distance between the means or centers of the two classes, or second, to decrease their own variances. ABC adopts the former approach, while TFCM uses the latter approach.

3.4. Search in Cluttered Scenes

We designed search tasks in cluttered scenes and with actual objects under five conditions: varied illumination, rotation, deformation, scaling, and occlusion [30,31,32]. In these experiments, the reference in Figure 11a is of size $88\times 44$, and the five scenes are of the same size, $360\times 640$. We first tested ABC performance under illumination change, as shown in Figure 11. Except for TFCM, all were able to correctly find the position of the reference under the illumination change condition. Observing their profiles, it can be seen that the difference between the highest and second-highest peak under the ABC-based search was larger than that under the other two. This observation can be confirmed from its profile in Figure 12a, which verifies that ABC may have the best discrimination performance among these classes. Figure 13 shows the other challenges of searching for the reference. In Figure 13a, the doll was rotated by a small angle at the same location. In Figure 13b, the doll’s clothing was deformed, and in Figure 13c the doll’s apparent size was reduced by moving it back. Finally, in Figure 13d the doll was occluded by placing it behind another doll; nonetheless, the different methods were able to find its true position.

3.5. Tracking in Open Data

Tracking is a difficult task requiring a stable and reliable matching scheme for consecutive following of objects of interest. Many conventional tracking algorithms [33,34,35] track targets by accurately updating the reference; here, however, our aim was to test the fundamental feasibility of ABC in tracking tasks with no modification of the single reference. Liquor [27] is an open dataset that includes interesting sequences of bottles being picked up and moved by human hands in which rotation, deformation, scaling, and occlusion happen over many consecutive frames. We used frames #1301–#1400 to test the tracking performances of CI, CCH, TFCM, and ABC. In frames #1301–#1380, the target bottle remains in the leftmost position. Other bottles are picked up by hands, passed in the front of the target bottle, then moved to other positions. In frames from #1381 to the end, the target is picked up and moved forward, including small out-of-plane rotations to the left and right.

In the first frame, we defined the reference shown at the top left in Figure 14. We used horizontal displacement, $\Delta x$, of matched positions from the central ground truth positions as a fundamental evaluator, because in these frames the bottle of interest was moved mainly horizontally and other candidate mismatched bottles were placed on the same level. Therefore, a smaller $\Delta x$ can show better tracking performance in the algorithms. In this tracking experiment, we chose the following five typical situations for the target bottle in order to demonstrate ABC performance: (1) In frame #1304, all three methods were able to find reasonable positions, because the bottle is not moved; (2) because of the large occlusion in frame #1320, only ABC was able to capture the lower-left position of the target, while the other three methods were misled to unreasonable places; (3) because of the partial occlusion in frame #1351, the result from CCH was largely shifted; (4) in frame #1354, possibly the most difficult case of full occlusion, none of the four methods wre able to identify the target; and (5) in frame #1395, where the bottle is lifted and moved to the right, only ABC maintained stable tracking of the bottle.

We next tested the effectiveness of introducing significance level, $\alpha$, to determine $h_{\mathrm{T}}$ using the same data frames. In preparation, we scanned all possible positions in frames #1301–#1400 to find the average, ${\overline{h}}_{\mathrm{T}}$, and its variance, ${\sigma }_{{\mathrm{h}}_{\mathrm{T}}}^2$, which were obtained as $0.05$ and $6.96\times {10}^{-4}$, respectively, by introducing $\alpha =0.02$. Based on this result, it is reasonable to compare the two cases of $\alpha =0.02$ and $h_{\mathrm{T}}=0.05$, which can be selected as representative constant values. Figure 15 shows that in these two cases, there was no apparent difference in performance except for the case of frame #1373, where the constant $h_{\mathrm{T}}$ failed to find the correct position.

Table 6. Number of mismatches in 100 frames.

	CI	CCH	TFCM	ABC
$\left|\Delta x\right|\ge 20$	15	28	49	16
$\left|\Delta x\right|\ge 30$	15	22	40	7
$\left|\Delta x\right|\ge 40$	15	20	18	4
$\left|\Delta x\right|\ge 50$	15	19	16	3

shows the number of mismatches in the Liquor dataset for frames #1301–#1400. The size of the reference image is $210\times 73$ pixels. If $\left|\Delta x\right|$ exceeds 73, the searched position does not overlap with the ground truth. We thus set 20, 30, 40, or 50 pixels to observe the number of mismatches. For example, if $\left|\Delta x\right|$ exceeds 20, the result is incorrect matching. As a result, ABC could not find targets under large occlusion conditions, although it remained robust in other frames. In Figure 16, we show the performance of the Skiing [27] dataset for frames #31–#60. The reference image is defined at the top left, and the size is $41\times 39$ pixels. The challenges are deformation and rotation. (1) In frame #32, ABC, CI, CCH, and TFCM can search for the correct positions because the skiier does not change significantly in the sky. (2) In frames #38, #45, and #58, only ABC can match the target. This is because ABC provides a good balance between the major and absent colors. In the comparison process, red and yellow colors are major colors for the reference image, and are the absent colors for the compared images. (3) In frame #50, the proportion of yellow pixels is decreased because of the large deformation. Therefore, four methods cannot match the target.

Table 7. Number of mismatches in 30 frames.

	CI	CCH	TFCM	ABC
$\left|\Delta x\right|\ge 20$	25	24	23	5
$\left|\Delta x\right|\ge 30$	24	24	23	4
$\left|\Delta x\right|\ge 40$	23	23	23	4
$\left|\Delta x\right|\ge 50$	23	23	23	4

shows the number of mismatches in the Skiing dataset for frames #31–#60. The number of mismatches proves the performance of our proposed ABC approach. ABC is more robust in the matching process.

ABC has several merits: technical simplicity, invariance with respect to in-plane rotation and distortion, and scaling as shown in the previous sections; however, it retains demerits as well, particularly the necessity of color and a reasonable image size in order to keep the histograms effective and avoid loss of positional information. In many of our experiments, we showed its effectiveness in tracking and searching problems, where a rapid method is generally needed in both the preparation or off-line processing and in on-line or real-time calculation. When sufficient data can be obtained from the training images, it may be possible to utilize different methods of machine learning, such as CNN or effective classifiers, to solve these problems. We suggest the combination of the proposed technique with these methods in order, for instance, to reduce the size and cost of training as well as to raise the total performance in processing. For example, in certain hierarchical approaches, ABC can represent an effective way of nominating possible candidates for continuing either the following detailed classification or other applications. In this type of utilization, it is preferable for absent colors to be somewhat independent of or orthogonal to any other features in order to maintain better performance in total.

3.6. Computational Cost

We calculated the computation cost of ABC using Visual Studio 2015 and the OpenCV 2.4.13 library, without parallel processing or GPU acceleration. The hardware used was a Windows 10 PC with a 2.81 GHz Intel Core i5-8400 CPU and 8 GB of RAM. We tested reference images of size $50\times 50$ that were scanned and searched pixel-by-pixel in a scene of size $100\times 100$. The entire computation required 0.1029 s per the OpenCV timing function and 0.1030 s per the QueryPerformanceCounter function.

4. Conclusions

Here, we have proposed a novel method based on color histogram, called absent color indexing for robust pattern search. By reorganizing a color histogram into two complementary histograms, we observed that the new feature of absent colors can effectively increase the margins, resulting in high reliability or distinguishability in matching in many different tasks. In the proposed method, low-frequency colors or ones that are relatively non-existent in the color histogram bins are enhanced by inversion for fair treatment. A novel way of obtaining an important parameter for histogram decomposition is provided through specifying the statistical significance level in the mean histogram. Mondrian random patterns were effectively used as a general type of pictures for fundamental evaluation of the proposed method in comparison experiments with several representative competitors. Experimental results using real-world images and datasets showed that the proposed method has good performance. In future works, we hope to investigate the applicability of ABC to real tasks such as human tracking and WEB search, as well as to problems where color features are highly important and greater reliability may be necessary.