Partial Order Rank Features in Colour Space

Abstract

Partial orders are the natural mathematical structure for comparing multivariate data that, like colours, lack a natural order. We introduce a novel, general approach to defining rank features in colour spaces based on partial orders, and show that it is possible to generalise existing rank based descriptors by replacing the order relation over intensity values by suitable partial orders in colour space. In particular, we extend a classical descriptor (the Texture Spectrum) to work with partial orders. The effectiveness of the generalised descriptor is demonstrated through a set of image classification experiments on 10 datasets of colour texture images. The results show that the partial-order version in colour space outperforms the grey-scale classic descriptor while maintaining the same number of features.

1. Introduction

It is, at first sight, peculiar that one of the most robust tools for image description, namely rank features, have only seen limited application to colour images. The problem is, of course, that while they are very effective at dealing with noise, rank features run afoul of the main theoretical difficulty associated with colour spaces—that is the absence of a natural order.

In recent years there has been a revival of interest in ranking of colour pixels. Notably, Ledoux et al. [1] published an extensive comparative study in the use of total orders as rank features for texture recognition. However, interest has been keenest in the field of colour morphology, where several solutions have been proposed—from adaptive orders that work around the ‘false colour problem’ to the natural mathematical structure for ordering higher-dimensional sets—that is partial orders [2,3,4,5,6].

In partially-ordered sets we simply admit that there will be couples of elements incomparable to each other. Partial orders are therefore particularly suitable for dealing with colour spaces, where statements like “yellow is greater than green” make little or no sense at all.

The objective of this work is to introduce a novel category of rank features based on partial orders. In the remainder, after providing some background on partial orders (Section 2), we detail the ways in which rank features can be defined (Section 2.5) and extend a classical descriptor (the Texture Spectrum) to work with partial orders (Section 3.1). We demonstrate the feasibility of the method through a set of experiment on 10 datasets of colour texture images (Section 3.2) and show that partial orders in colour space can outperform grey-scale total ordering (Section 4).

2. Background

2.1. Rank Features

Rank features are a well established technique for dealing with noise in images, enforcing invariance to all sorts of contrast or illumination variations and sensor nonlinearities [7]. Because of their robustness, they were first developed in the context of wide-baseline stereo matching—see for instance the census and rank transforms [8]. More recently, descriptors in the popular Local Binary Pattern (LBP) family, including Texture Spectrum, Binary Gradient Contours, etc. [9,10] have turned rank features into a general purpose tool, with applications—among others—in texture classification, face recognition, surface inspection and content-based image retrieval [11]. The descriptive power of rank features has been expanded to explicitly capture orientation (Ranklets [12]) and second-order stimuli (Variance Ranklets [13]), all types of information that were seen as the preserve of linear filters or ad-hoc algorithms.

Common to all rank features is the fact that they are defined in terms of ordinal information between pixels only, with the actual pixel values being discarded. This can be done in terms of pairwise pixel comparisons (rank and census transform, LBP), pixel ranks (Ranklets) or a permutation of ranks (Variance Ranklets), but it is easy to see that the two approaches are equivalent [12] and that all descriptors rely on the natural order relation (≤) between pixel values.

Before proceeding to definitions it is worth noting that, notwithstanding the trend towards the use of convolutional neural networks as feature extractors [14], rank features are still competitive in texture applications [15]. In the following section we recall the axioms for an order relation.

2.2. Order Relations

An order relation is an abstraction of the common notion of “greater than” used to compare numerical values, in our case pixel values in $\mathsf{P}$ (typically the set of 8-bit intensity values). In order to be called a (total) order, a binary relation ≤ needs to satisfy the following four conditions:

Definition 1

(Order axioms). For all $(x,y,z)\in {\mathsf{P}}^3$,

1. 

$x\le x$ (reflexivity).

2. 

if $x\le y$ and $y\le x$ then $y=x$ (antisymmetry).

3. 

if $x\le y$ and $y\le z$ then $x\le z$ (transitivity).

4. 

either $x\le y$ or $y\le x$ (totality).

The last condition guarantees that we know how to compare any pair of pixel values.

2.3. Ordering High-Dimensional Data

The application of rank features to multi-channel images or higher dimensional data is hindered by the fact that there is no natural way of ordering multivariate data. It is certainly possible to provide a total order for a colour space; for instance, one could order RGB data lexicographically using the R channel as the primary sorting key, followed by G and finally by B. However, like other similar options, this has a disadvantage, namely, there are colours that are very close to each other in colour space, but very far in the order—and is therefore of limited practical interest (a sub-relation of the lexicographical order, the product order, is indeed of practical interest and will be discussed in detail in this paper, see Section 2.5.1). In general, it is best to resort to some sort of sub-ordering principle. These can be broadly divided in four categories [16]:

• Marginal ordering (M-ordering).
• Reduced (aggregate) ordering (R-ordering).
• Conditional (sequential) ordering (C-ordering).
• Partial ordering (R-ordering).

In marginal ordering, ranking is carried out on one or more components (marginals) of the multivariate data. Ranking colour data in the RGB space by the value of red is an example of M-ordering; lexicographical ordering is another one. Reduced (aggregate) ordering relies on converting multivariate data to univariate through suitable transformations. A common way to do this consists of establishing a reference point in the data space and using the distance from that point to rank the data. Conditional ordering occurs when we sort a random multivariate sample based on the corresponding (usually marginally-sorted) values of another sample. C-ordering is closely related to the concept of concomitants in Statistics [17]. Partial ordering will be discussed in detail in Section 2.5.

Interestingly, many of the common ways of dealing with order in colour space fall under the first two categories, i.e., marginal and reduced (aggregate) ordering. For instance, ranking based on intensity can be seen as a marginal ordering of the HSV space along the V axis; or as a reduced or aggregate ordering over the RGB space, where the aggregating function is the grey-level intensity. Other examples of aggregating orders will be given in Section 3.1.

In this paper, we will focus on rank features based on partial orders. Before introducing these, we review recent approaches to using multivariate orders on colour images.

2.4. Rank-Based Approaches to Colour Processing

Previous approaches to rank-based colour features typically extend grey-scale rank-based methods to the colour domain by considering either the colour channels separately (intra-channel features) and/or in pairwise combination (inter-channel features). Mäenpää and Pietikäinen [18] for instance extended classic LBP by applying it both to each R, G and B colour channel separately and pairwise between each of the R–G, R–B and G–B pairs. Bianconi et al. [19] adopted the same approach for extending grey-scale ranklets [12] to the colour domain. Lee et al. [20] defined Local Colour Vector Binary Patterns (LCVBP) by decomposing the colour triplets into a norm and angular component and by computing LBP on each of them. More recently, Cusano et al. [21] introduced Local Angular Patterns (LAP) which consider the angular component only and discard the norm part altogether.

Another possible strategy consists of establishing some sort of a priori total ordering on the colour data. This approach is not uncommon in colour morphology—see for instance Angulo [4], van De Gronde and Roerdink [6]—and has been advocated for extending LBP to colour images by Barra [22]. Of late, this family of methods has been extensively investigated by Ledoux et al. [1] and Bello-Cerezo et al. [23]. The problem is that imposing a total ordering on the colour data inevitably entails a certain degree of arbitrariness, with the consequence that the results tend to be dataset-dependent. On the other hand, morphology for tensor-valued images (that arise from certain magnetic resonance techniques) has relied on the Loewner order, that is in fact a partial order (see for instance Burgeth et al. [24]; more on this in Section 2.5.2). More recently, this approach has been extended to morphology for colour images Burgeth and Kleefeld [5]. Partial ordering circumvents the problem of ordering multivariate data totally, at the expense of not allowing comparisons between some colour values.

2.5. Partial Orders

A partial order differs from a total order in that the fourth axiom in Definition 1 is waived, i.e., there are pairs of elements in the set that are incomparable. In order to distinguish this from a total order we use the notation $x⪯y$. If the elements x and y are incomparable, we shall write $x\nsim y$.

In the following, we describe two types of partial orders that are applicable to colour spaces with Cartesian and polar coordinates respectively.

2.5.1. Product Order

By product order we mean the relation obtained from the component-wise comparison of colour values. Given $\mathbf{u}=\left(c_{1u},c_{2u},c_{3u}\right)$ and $\mathbf{v}=\left(c_{1v},c_{2v},c_{3v}\right)$ two triplets representing colours in a generic space we write:

$$\left\{\begin{array}{cc}\mathbf{u}{⪯}_{\times }\mathbf{v}\hfill & \mathrm{if}{c}_{1u}\le c_{1v},c_{2u}\le c_{2v},c_{3u}\le c_{3v}\hfill \\ \mathbf{u}{\nsim }_{\times }\mathbf{v}\hfill & \mathrm{if}\mathrm{neither}\mathbf{u}{⪯}_{\times }\mathbf{v}\mathrm{nor}\mathbf{v}{⪯}_{\times }\mathbf{u}.\hfill \end{array}\right.$$

(1)

Note that this is a subset of the lexicographical order introduced in Section 2.3; it is however of higher practical interest as it treats all three channels symmetrically. In the RGB space, for instance, a given colour $\mathbf{u}$ weakly dominates the rectangular parallelepiped $\mathsf{C}\left(\mathbf{u}\right)$ with three edges along the axes and a vertex in the colour itself (see Figure 1). For any colour $\mathbf{v}$ that does not dominate all of $\mathsf{C}\left(\mathbf{u}\right)$, $\mathbf{v}\nsim \mathbf{u}$.

The product order can of course be applied to any colour space, giving relations of various degree of interpretability and effectiveness for pattern recognition (see Section 3.1 and Section 4).

2.5.2. Loewner Order

The Loewner (partial) order is defined on symmetric matrices. Given two symmetric matrices $\mathbf{A}$, $\mathbf{B}$ we write:

$$\left\{\begin{array}{cc}\mathbf{A}{⪯}_{\mathcal{L}}\mathbf{B}\hfill & \mathrm{if}\left(\mathbf{B}-\mathbf{A}\right)\in {\mathcal{S}}_{+}\hfill \\ \mathbf{A}{\nsim }_{\mathcal{L}}\mathbf{B}\hfill & \mathrm{if}\mathrm{neither}\mathbf{A}{⪯}_{\mathcal{L}}\mathbf{B}\mathrm{nor}\mathbf{B}{⪯}_{\mathcal{L}}\mathbf{A}\hfill \end{array}\right.$$

(2)

where ${\mathcal{S}}_{+}$ indicates the set of positive semi-definite matrices. Applying this to a colour space requires mapping colour values to symmetric matrices. Following [5], we start from a modified colour space $\mathrm{HC}\tilde{\mathrm{L}}$ obtained from HSL ([25], Section 4.6) by setting $\tilde{L}=2L-1$ for the (modified) luminance and replacing saturation with chroma $C=max\{R,G,B\}-min\{R,G,B\}$. The resulting colour gamut fills a bicone with axis $\tilde{\mathrm{L}}$ and opening angle 90${}^{\circ }$. We isometrically map colours to the space Sym$\left(2\right)$ of symmetric $2\times 2$ matrices by setting [5]:

$$\mathbf{M}\left(h,c,\tilde{l}\right)=\frac{1}{\sqrt{2}}\left(\begin{array}{cc}\tilde{l}-c& h\\ h& \tilde{l}+c\end{array}\right).$$

(3)

For two colours $\mathbf{u}=\left(h_u,c_u,{\tilde{l}}_u\right)$ and $\mathbf{v}=\left(h_v,c_v,{\tilde{l}}_v\right)$ in the $\mathrm{HC}\tilde{\mathrm{L}}$ space we therefore write:

$$\left\{\begin{array}{cc}\mathbf{u}{⪯}_L\mathbf{v}\hfill & \mathrm{if}\mathbf{M}\left(h_v,c_v,{\tilde{l}}_v\right){⪯}_{\mathcal{L}}\mathbf{M}\left(h_u,c_u,{\tilde{l}}_u\right)\hfill \\ \mathbf{u}{\nsim }_L\mathbf{v}\hfill & \mathrm{if}\mathrm{neither}\mathbf{u}{⪯}_L\mathbf{v}\mathrm{nor}\mathbf{v}{⪯}_L\mathbf{u}\hfill \end{array}\right.$$

(4)

where ${⪯}_{\mathcal{L}}$ is defined in Equation (2). Geometrically (Figure 2), a given colour $\mathbf{v}$ weakly dominates all colours of lower luminance that fall in a cone with its vertex in $\mathbf{v}$ and its axis parallel to the $\tilde{L}$ axis.

3. Materials and Methods

3.1. Rank Features on Partial Orders

In this section we show how to generalise existing rank-based descriptors by replacing total order in grey-scale with suitable partial orders in colour space. In the remainder we shall use the Texture Spectrum [26] as our reference model—though other descriptors such as Local Binary Patterns and Local Ternary Patterns are amenable to the same procedure with virtually no effort.

In Texture Spectrum, a local image pattern $\mathcal{P}=\left\{{\mathbf{p}}_0,{\mathbf{p}}_1,\dots ,{\mathbf{p}}_n\right\}$ is assigned a unique decimal code as follows:

$$f_{\mathrm{TS}}\left(\mathcal{P}\right)=\sum _{i=1}^n{3}^i\tau \left[g\left({\mathbf{p}}_0\right),g\left({\mathbf{p}}_i\right)\right]$$

(5)

where ${\mathbf{p}}_0$ represents the central pixel and ${\mathbf{p}}_i$, $i\in \{1,\dots ,n\}$ the peripheral pixels, which we assume to be arranged on a circle around the central pixel. We also assume that $\mathbf{p}$ represents a point in a 3D colour space, though again extension to multi-spectral data is straightforward. In Equation (5) the function $g\left({\mathbf{p}}_0\right)$ stands for a generic conversion from colour into grey-scale, whereas $\tau \left(u,w\right)$ indicates the ternary thresholding function:

$$\tau \left(w,z\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}w<z\hfill \\ 1\hfill & \mathrm{if}w=z\hfill \\ 2\hfill & \mathrm{if}w>z.\hfill \end{array}\right.$$

(6)

An image is represented by the dense, orderless statistical distribution over the set of possible codes. For Texture Spectrum, the number of (directional) features generated by the method is clearly $3^n$. Invariance under rotations and/or reflections can be obtained by grouping together all those codes that represent patterns which can be transformed into one another by such transforms. The corresponding mathematical structures are necklaces and bracelets, respectively for invariance under rotations (i.e., cyclic group of order n; $C_n$) and under rotations + reflections (i.e., dihedral group of order n; $D_n$). For general formulas about the number of resulting $C_n$- and $D_n$-invariant features and for other mathematical details please refer to González et al. [27], Zelenyuk and Zelenyuk [28]. Specifically, for $n=8$ (which is the case considered herein—see below) the number of features is respectively 834 and 498.

A ternary rank feature for partially ordered data analogous to Texture Spectrum—the Partial Order Texture Spectrum (POTS)—can easily be defined in the following way:

$$f_{\mathrm{POTS}}\left(\mathcal{P}\right)=\sum _{i=1}^n{3}^i\phi \left({\mathbf{p}}_0,{\mathbf{p}}_i\right)$$

(7)

$$\phi \left(\mathbf{u},\mathbf{v}\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}\mathbf{u}\prec \mathbf{v}\hfill \\ 1\hfill & \mathrm{if}\mathbf{u}⪰\mathbf{v}\hfill \\ 2\hfill & \mathrm{if}\mathbf{u}\nsim \mathbf{v}\hfill \end{array}\right.$$

(8)

where ⪯ indicates a generic partial order relation in the colour space (see Section 2.5). Notably, the number of features generated by this formulation is the same as generated by the Texture Spectrum.

In the experiments we considered the following partial order/colour space combinations: product order (Section 2.5.1) in the RGB, Ohta’s and opponent spaces [25]; Loewner order (Section 2.5.2) in the $\mathrm{HC}\tilde{\mathrm{L}}$ space. When reporting experimental results we use subscripts ‘RGB’, ‘ohta’ and ‘opp’ to indicate the colour spaces, and superscripts ${⪯}_{\times }$ and ${⪯}_L$ respectively for the product and Loewner orders (see Equations (1) and (4)). No superscript was used to indicate the natural total order on greyscale values.

Conversion from RGB to grey-scale was also performed in three different ways: (1) through the standard PAL/NTSC formula ([25], Section 4.3.1); (2) by computing the average of the three channels; and (3) by determining, for each image, the principal axes of the colour distribution in the RGB space and projecting each $\left(r,g,b\right)$ triplet onto the first axis. In the remainder we denote the corresponding variations of Texture Spectrum respectively as TS${}_{\mathrm{grey}}$, TS${}_{\mu }$ and TS${}_{\mathrm{p}1}$.

Finally, we computed $C_n$- and $D_n$-invariant features over $3\times 3$, non-interpolated, square neighbourhoods of radius 1px and 2px. The overall feature vector was obtained by concatenating the feature vectors obtained at each resolution—see also González et al. [27] for details. These settings respectively generates $834\times 2=1668$ and $498\times 2=996$ features.

3.2. Experiments

To test the effectiveness of the partial-order rank features described in Section 3.1 we ran a set of supervised image classification experiments. Datasets, classification strategy and accuracy estimation are described in the following subsections.

3.3. Datasets

We used ten datasets of colour texture images from different sources as described below. The main properties of each dataset are summarised in

Table 1. Datasets used in the experiments: round-up table.

ID	Name	No. of Classes	No. of Samples per Class	Variations in Imaging Conditions	Sample Images
1	Epistroma	2	825/551	Unspecified
2	KTH-TIPS	10	81
3	KTH-TIPS2b	11	432
4	Kylberg-Sintorn	25	6	None
5	MondialMarmi	25	16	None
6	Outex-13	68	20	None
7	Outex-14	68	60
8	Pap smear	2	204	Unspecified
9	PlantLeaves	20	60	None
10	RawFooT	68	184

KEY TO SYMBOLS: = illumination, = rotation, = scale.

.

3.3.1. Epistroma

Contains 1376 histopathological images from colorectal cancer representing either epithelium (825 images) or stroma (551 images). The image size ranges from 93 px to 2372 px in width and from 94 px to 2373 px in height. Further details about tissue preparation and digitisation procedure are available in Linder et al. [29].

3.3.2. KTH-TIPS

Includes 10 classes of common materials (e.g., aluminum foil, bread, corduroy, etc.) with 81 image samples for each class [30,31]. Each material was acquired under nine scales, three rotation angles and three lighting directions.

3.3.3. KTH-TIPS2b

Features 11 classes of materials (432 sample images per class) and is actually an extension of KTH-TIPS. The image acquisition settings were the same as in KTH-TIPS, but four rather than three illumination conditions were used in this case [32].

3.3.4. Kylberg–Sintorn

Is composed of 25 classes of heterogeneous materials, such as food (e.g., lentils, oatmeal and sugar), fabric (e.g., knitwear and towels) and tiles [33,34]. For each class one sample image was acquired using invariable illumination conditions and under nine different rotation angles—of which only the images at 0${}^{\circ }$ were included in our experiments. Each image was further subdivided into six non-overlapping sub-images of dimension 1728 × 1728 px.

3.3.5. MondialMarmi

Comprises 25 classes of marble and granite products identified by their commercial denominations, e.g., Azul Platino, Bianco Sardo, Rosa Porriño and Verde Bahía [35]. Each class is represented by four tiles; ten images for each tile were acquired under steady illumination conditions and at rotation angles from 0${}^{\circ }$ deg to 90${}^{\circ }$ in steps of 10${}^{\circ }$. In the experiments we only used the images at 0${}^{\circ }$; moreover, we subdivided each image into four non-overlapping sub-images therefore obtaining 16 image samples for each class.

3.3.6. OUTEX-13 and OUTEX-14

Are based on the same sets of images that respectively make up the OUTEX_ TC_00013 and OUTEX_TC_00014 test suites—see Ojala et al. [36] for details. Specifically, OUTEX-13 features 68 classes of materials with 20 images per class acquired under invariable illumination conditions; OUTEX-14 contains the same classes—but in this case the image samples were acquired under three different illumination conditions—therefore there are 60 samples per class. Please notice, however, that in order to maintain the same evaluation protocol for all the datasets considered here (see Section 3.4), the subdivisions into train and test sets used in our experiments were not the same as in the OUTEX_TC_00013 and OUTEX_TC_00014 test suites.

3.3.7. Pap Smear

Consists of 917 PAP-stained images of variable dimension representing cells from the cervix [37]. The images represent either abnormal cases—675 samples or normal cases—242 samples. The dataset also comes with a further subdivision into seven sub-classes which was not considered in our experiments. The image size ranges from 84 × 88 px to 392 × 262 px. In our experiments we considered a balanced sub-set containing 204 samples for each of the two classes.

3.3.8. Plant Leaves

Includes a total of 1200 samples of plant leaves from 20 different classes with 60 samples per class [38]. The images were acquired using a planar scanner and have a dimension of 128 × 128 px.

3.3.9. RawFooT

Comprehends 68 classes of raw food and grains such as corn, chicken breast, pomegranate, salmon and tuna [39,40]. The materials were acquired under 46 different illumination conditions resulting in as many image samples for each class. We further subdivided the images into four non-overlapping sub-images, thus obtaining 184 samples for each class. The dimension of the resulting image tiles was 400 × 400 px.

3.4. Classification and Accuracy Estimation

For each dataset described in Section 3.3 we performed supervised classification using a nearest neighbour classifier (1-NN) with the $L_1$ (‘Manhattan’) distance. In detail, after extracting a feature vector from all images according to one of the descriptors tested, we computed the distance between such vectors as the sum of the absolute differences between components. We then assigned each test vector to the class of the closest training vector. The absence of tuning parameters, the ease of implementation and other desirable asymptotic properties make the 1-NN particularly appealing for comparison purposes. Its use in related works is indeed customary: see for instance Cusano et al. [39], Kandaswamy et al. [41], Liu et al. [42].

Accuracy estimation was based on split-half validation with stratified sampling—for each dataset we used half of the samples of each class to train the classifier (train set) and the other half (test set) to compute the accuracy. This was defined as the ratio between of number of samples of the test set correctly classified ($N_c$) and the total number of samples of the test set (N):

$$a=\frac{N_c}{N}.$$

(9)

For a stable estimation we averaged the above value over a hundred different subdivisions into train and test set:

$$\hat{a}=\frac{{\sum }_{i=1}^{100}{a}_i}{100},$$

(10)

where $a_i$ indicates the accuracy achieved in the i-th subdivision into train and test set. In

Table 2. Overall classification accuracy: confidence intervals for the cross-validated accuracy $\hat{a}$. Best results highlighted for grey-level (orange) and colour space features (blue). Boldface figures indicate statistically significant differences.

Descriptor	Inv.	Dataset
		1	2	3	4	5	6	7	8	9	10
LBP${}_{\mathrm{grey}}$	$C_n$	91.3–91.7	93.2–93.8	92.6–92.8	93.4–94.6	79.5–79.9	82.5–82.7	97.4–97.7	81.5–82.2	72.9–73.5	94.3–94.5
LBP${}_{\mathrm{grey}}$	$D_n$	91.5–91.9	93.3–93.9	92.5–92.7	93.4–94.6	79.7–80.1	82.4–82.7	97.3–97.6	81.1–81.9	73.0–73.6	94.3–94.5
TS${}_{\mathrm{grey}}$	$C_n$	91.3–91.8	92.4–93.0	94.3–94.5	95.2–96.2	79.5–79.9	82.5–82.7	97.7–98.1	83.5–84.3	74.5–75.1	96.1–96.2
TS${}_{\mathrm{grey}}$	$D_n$	91.8–92.2	92.3–92.9	94.2–94.3	95.3–96.3	79.3–79.8	82.4–82.6	97.8–98.2	84.0–84.9	74.8–75.4	96.1–96.2
TS${}_{\mu }$	$C_n$	91.6–92.0	92.7–93.3	93.7–93.9	92.8–94.2	79.2–79.6	79.5–79.7	97.1–97.5	83.1–83.8	73.9–74.4	95.8–96.0
TS${}_{\mu }$	$D_n$	91.9–92.2	92.8–93.4	93.5–93.7	92.8–94.2	78.8–79.3	79.5–79.7	97.0–97.4	83.7–84.5	74.3–74.8	95.9–96.0
TS${}_{\mathrm{p}1}$	$C_n$	91.3–91.6	92.9–93.4	93.0–93.2	94.4–95.6	78.6–79.0	80.0–80.2	96.8–97.3	80.6–81.4	73.2–73.8	95.1–95.2
TS${}_{\mathrm{p}1}$	$D_n$	91.5–91.8	92.9–93.5	92.8–93.0	94.6–95.8	78.4–78.9	80.0–80.3	96.8–97.3	80.9–81.7	73.0–73.6	95.1–95.3
POTS${}_{\mathrm{ohta}}^{{⪯}_{\times }}$	$C_n$	86.9–87.3	89.3–89.8	94.7–94.9	93.3–94.4	82.2–82.6	82.0–82.2	97.6–98.0	77.6–78.6	69.3–69.9	94.5–94.6
POTS${}_{\mathrm{ohta}}^{{⪯}_{\times }}$	$D_n$	87.0–87.3	89.4–89.9	94.9–95.1	93.3–94.4	82.4–82.8	82.3–82.5	97.7–98.1	78.3–79.3	69.3–69.9	94.5–94.6
POTS${}_{\mathrm{opp}}^{{⪯}_{\times }}$	$C_n$	90.3–90.6	89.9–90.5	95.1–95.3	95.3–96.5	80.1–80.5	78.9–79.2	97.5–97.9	73.7–74.7	64.1–64.7	93.1–93.3
POTS${}_{\mathrm{opp}}^{{⪯}_{\times }}$	$D_n$	90.2–90.6	89.7–90.3	95.2–95.3	94.9–96.0	80.4–80.9	79.0–79.2	97.5–97.9	74.6–75.6	64.4–65.0	93.2–93.4
POTS${}_{\mathrm{HC}\tilde{\mathrm{L}}}^{{⪯}_L}$	$C_n$	91.4–91.8	93.6–94.1	95.3–95.5	90.7–92.5	73.8–74.3	73.4–73.7	96.1–96.7	82.9–83.7	67.7–68.3	95.3–95.5
POTS${}_{\mathrm{HC}\tilde{\mathrm{L}}}^{{⪯}_L}$	$D_n$	92.0–92.4	93.5–94.0	95.2–95.4	90.7–92.5	74.3–74.7	74.0–74.3	96.2–96.7	83.9–84.7	68.1–68.6	95.4–95.5
POTS${}_{\mathrm{RGB}}^{{⪯}_{\times }}$	$C_n$	90.4–90.8	94.1–94.6	94.9–95.1	87.9–90.0	72.6–73.0	74.5–74.7	96.0–96.6	82.6–83.4	75.2–75.8	96.4–96.5
POTS${}_{\mathrm{RGB}}^{{⪯}_{\times }}$	$D_n$	91.3–91.7	94.0–94.5	94.8–95.0	87.9–90.0	73.0–73.5	74.8–75.0	96.2–96.8	83.6–84.5	75.9–76.4	96.3–96.5

we report the 95% confidence intervals for $\hat{a}$ (computed under the simplifying assumption of normal distribution).

4. Results and Discussion

Table 2 reports the confidence intervals for the means of the overall classification accuracy (see Section 3.4). For each dataset we highlighted in orange the best result obtained by total-order grey-scale rank features; in blue the best result obtained by partial order rank features (POTS) in colour space. When there was a statistically significant difference between the two, the best figure was indicated in boldface. As can be seen, partial order rank features in colour space performed significantly better in five datasets out 10, whereas the reverse occurred in one dataset only (dataset six). In the remaining four datasets there was no significant difference between the two methods.

As for grey-scale rank features, the results show that in most cases (i.e., 8 datasets out of 10) the best performance was obtained using standard PAL/NTSC grey-scale conversion. By contrast, partial-order rank features denoted a higher dependence on the colour space used.

The computational cost of all the descriptors considered is roughly equivalent, as the number of features is the same and the complexity of computing a partial or total order comparison in colour space is comparable to the cost of a colour space transformation. Indeed, as we have just described, even the traditional TS requires a grey-scale conversion, the choice of which can be seen as an integral part of the descriptor.

In

Table 3. Comparison with the results obtained by other ordering methods as reported in the references indicated. Key to symbols: ‘cvn’ = colour vector norm, ‘lex’ = lexicographic ordering, ‘rcl’ = preorder based on white as reference colour. Please refer to the cited works for further details.

Dataset	Best Result (Literature)			Best Result (This Paper)
	LBP${}_{\mathbf{cvn}}$	LBP${}_{\mathbf{lex}}$	LBP${}_{\mathbf{rcl}}$
KTH-TIPS	94.3 [23]	94.3 [23]	94.0 [23]	94.1–94.6 (POTS${}_{\mathrm{RGB}}^{⪯\times }/C_n$)
KTH-TIPS2b	92.3 [23]	92.3 [23]	92.1 [23]	95.3–95.5 (POTS${}_{\mathrm{HCL}}^{⪯\times }/C_n$)
Kylberg-Sintorn	N/A	99.1 [43]	N/A	95.3–96.5 (POTS${}_{\mathrm{RGB}}^{⪯\times }/C_n$)
Outex-13	85.3 [1]	86.3 [1]	85.9 [1]	82.3–82.5 (POTS${}_{\mathrm{opp}}^{⪯\times }/D_n$)
Outex-14	74.3 [1]	73.4 [1]	72.3 [1]	97.7–98.1 (POTS${}_{\mathrm{ohta}}^{⪯\times }/D_n$)
PapSmear	N/A	N/A	66.2 [43]	83.6–84.5 (POTS${}_{\mathrm{RGB}}^{⪯\times }/D_n$)
PlantLeaves	69.9 [23]	65.2 [23]	71.9 [23]	75.9–76.4 (POTS${}_{\mathrm{RGB}}^{⪯\times }/D_n$)
RawFoot	N/A	N/A	80.5 [43]	96.4–96.5 (POTS${}_{\mathrm{RGB}}^{⪯\times }/C_n$)

we compare our results to published results obtained using rank-based descriptors in conjunction with other ordering methods in colour spaces. As can be seen, in most cases our partial-order based approach improves significantly over previous results. We should here emphasise that the computational requirements of our partial-order descriptors are not higher than those of the other ordering methods cited.

5. Conclusions and Future Work

The lack of a natural order among colours represents an intrinsic impediment to the definition of rank features in colour space. In this paper we have introduced a novel and general approach based on partial orders. Partial orders overcome the problems inherent to ordering multivariate data at the expense of admitting that not all pairs of colours can be compared to each other. We showed that this scheme fits in well with existing grey-scale local image descriptors, that are amenable to extension to the colour domain with little effort. Taking the Texture Spectrum as a model, we showed that its partial-order version in colour space (POTS) can outperform the grey-scale classic descriptor while maintaining the same number of features and with comparable computational complexity. Previous studies have also demonstrated that the use of colour can improve texture discrimination, but at the expense of employing a higher number of features [44,45,46]. Notably, our approach improves on published results that use descriptors based specifically on (total) colour space ordering (see Table 3).

To the best of our knowledge this is the first time that partial orders have been used to define rank features for pattern recognition. The method is conceptually simple, fairly general and shows potential for application in a wide number of computer vision tasks. Future studies will be focussed on extending the approach to the broader class of descriptors known as Histograms of Equivalent Patterns [9]. The effect of the colour space on the performance of rank features based on partial orders is also an important topic for further investigation. Finally, the insertion of partial order based algorithms in more involved image processing pipelines (e.g., convolutional neural networks) also represents an interesting opportunity for future research; integration at the level of matching [47] has so far been successful.