Fuzzy Windows with Gaussian Processed Labels for Ordinal Image Scoring Tasks

Abstract

In this paper, we propose a Fuzzy Window with the Gaussian Processed Label (FW-GPL) method to mitigate the overlap problem in the neighboring ordinal category when scoring images. Many published conventional methods treat this challenge as a traditional regression problem and make a strong assumption that each ordinal category owns an adequate intrinsic rank to outline its distribution. Our FW-GPL method aims to refine the ordinal label pattern by using two novel techniques: (1) assembling fuzzy logic to the fully connected layer of convolution neural networks and (2) transforming the ordinal labels with a Gaussian process. Specifically, it incorporates a heuristic fuzzy logic from the ordinal characteristic and simultaneously plugs in ordinal distribution shapes that penalize the difference between the targeted label and its neighbors to ensure a concentrated regional distribution. Accordingly, the function of these proposed windows is leveraged to minimize the influence of majority classes that mislead the prediction of minority samples. Our model is specifically designed to carefully avoid partially missing continuous facial-age segments. It can perform competitively when using the whole continuous facial-age dataset. Extensive experimental results on three facial-aging datasets and one ambiguous medical dataset demonstrate that our FW-GPL can achieve compelling performance results compared to the State-Of-The-Art (SOTA).

1. Introduction

Image scoring, typically known as ordinal classification, is a supervised learning problem aiming to predict a discrete set of ordinal labels. The main difference from the classification task is that the categories are related in a natural or implied order. For example, the apparent age group estimation grades face images based on an ordinal scale: “infants”, “children”, “teenagers”, “youth”, “young adults”, “adults”, “middle-aged”, and “aged”. Ordinal classification can be viewed as a special case of metric regression, where the regression targets are discrete and finite. The differences in features between adjacent labels are not always equal to each other. The difference in facial features between “infants” and “children” being more obvious than facial features between “young adults” and “adults” is one example. However, if the ordinal relationship of labels is ignored, the ordinal regression problem will only become a simple multi-class classification issue. In Figure 1, when learning with ordinal images, a common problem is that the ambiguity between two neighboring categories usually has a negative effect on the training convergence. Therefore, the performance of the learned model tends to degrade in ordinal classes. This challenge has motivated us to develop a robust ordinal image classification approach to analyzing ordinal data.

Ordinal image classification approaches or ordinal models [1] can be roughly divided into two aspects, Single Label Learning with Specific Loss (SLL-Loss) [2,3,4,5] and the Label Distribution Based Learning (LDBL) [2,6,7,8,9,10,11,12,13]. SLL-Loss methods typically rely on independently processing a single facial image. This ignores gradual changes in human faces, and thus, facial appearance is usually ambiguous as regards adjacent age classes. The LDBL methods tend to map ordinal ground-truth learning based on a Gaussian or Gaussian-like label distribution. But under such a long-tailed case, they also ignore the processing of ordinal neighbors or overlapping features.

To address the ambiguous and overlapping features in the ordinal data, we propose a Fuzzy Window with the Gaussian Processed Labels (FW-GPL) approach to the ordinal classification issue. This also aims to stretch the semantic margins (or enlarge their interclass variance) of ordinal classes, which represent the shared features of neighbor categories. As shown in Figure 1, we assume that two neighboring ordinal classes have a closely shared feature region which can increase the difficulty of the ordinal classification task. A fuzzy window with Gaussian processed labels is carefully designed on the top of deep neural networks so as to reduce the effect of the overlapping features but preserve the age distribution information. In Figure 2, our proposed FW-GPL is composed of two crucial branches, including a defuzzifier window and a learning strategy using Gaussian processed labels. The defuzzifier window attempts to reduce the influence the ambiguously overlapping features have on ordinal neighbors, and simultaneously, it tries to retain the internal ordinal features which can represent the real category.

Practically, the Gaussian processed labels allow the incorporation of prior knowledge (e.g., Gaussian-like age distribution) to concentrate on the major class and weaken the influence of remote neighbor classes. To validate the effectiveness of our proposed method, we perform extensive experiments on three widely-used face-aging datasets, including MORPH II [14], FG-NET [15], and CACD [16], as well as one medical ordinal dataset: Curated Breast Imaging Subset of Digital Database of Screening Mammography (CBIS-DDSM). This paper proposes a novel Fuzzy Window with a Gaussian Processed Label (FW-GPL) method for ordinal image scoring, and we achieve a competitive performance compared to State-Of-The-Art (SOTA) methods. The main contributions of this work are summarized as follows:

• We directly face the internal challenge of the ordinal image classification task and clarify the theoretical reason why the ordinal image classification task is difficult.
• FW-GPL can indirectly reduce the influence of the overlapping features among the ordinal neighbor classes. This process can effectively improve the scoring performance of the ordinal images.
• When the ordinal sequence of the images is not consecutive, FW-GPL can achieve an equivalent performance to wholly sequential ordinal data by setting a proper length for the fuzzy window.

2. Related Work

The objective of this learning architecture for the ordinal regression problem is to weaken the influence of the overlapping features $F=\{f_{\mathit{1}},f_{\mathit{2}},\dots ,f_{ϵ}\}$ extracted from the neighboring ordinal categories: $C=\{C_{\mathit{1}},C_{\mathit{2}},\dots ,C_i,\dots ,C_K\}$ ($ϵ$ is the number of quantized features, and K is the number of categories). Each $C_{\mathit{i}}$ is an ordinal category containing overlapping features with its neighbors, $\{C_{\mathit{i}-\mathit{a}},\dots ,C_{\mathit{i}-\mathit{a}+\mathit{1}},C_{\mathit{i}-\mathit{1}}\}$ and $\{C_{\mathit{i}+\mathit{1}},\dots ,C_{\mathit{i}+\mathit{b}-\mathit{1}},C_{\mathit{i}+\mathit{b}}\}$, where values a and b are related to the relationship between the feature strength of the specific category $C_{\mathit{i}}$ and its closeness to neighboring categories (In prior work [17,18], the boundary of the window is $\{a,b\}$. In this paper, we also set the upper bound of the window as a and the lower bound of the window as b). Moreover, Gaussian processed labels can prevent extracted features from roughly slipping into one category, which means they make neighboring ordinal categories meticulously divided up according to shared overlapping features.

2.1. Ordinal Classification

In the machine learning field, ordinal classification models are reassembled by reformulating the problem to utilize multiple binary classifiers [19]. There are some earlier studies working on constructing Convolution Neural Networks (CNNs) [20,21], which have replaced the last layer of the ordinal classification model with a number of binary classifiers [22]. In this Ordinal Regression CNN (OR-CNN) architecture, the ordinal classification problem has been converted to a number of K binary classification tasks. If the maximum value of the ordinal label is K, we rearrange the labels with a set $k=\{0,1,\dots ,K-1\}$ and define the binary classifier as whether the output is greater than k or not. All K binary tasks share the same intermediate layers, but they are assigned distinct weight parameters in the output layer [23]. This Ordinal Regression CNN (OR-CNN) architecture deeply relies on the ordinal continuity of the data. If the training dataset has insufficient and intermittent input ordinal labels, and if the dataset has missing data (for example, 150-year-old facial-age data), the fitted OR-CNN cannot recognize the intermittent or missing segment, which inevitably leads to a classification crash.

2.2. Windows for Ordinal Classification

MWR [18] uses five neurons and a local window to estimate facial age. It proposes the notion of relative rank ($\rho$ rank), a new order representation scheme for input and reference instances. This relative rank was estimated iteratively by selecting two reference instances to form a search window and then estimating the $\rho$ rank within the window. In other words, MWR applies two overlapping windows with reference centers to limit the influence of the relative rank (or “intrinsic rank”), and it also uses a search process to find the most proper position of the centers to reduce the influence of the overlapping “rank”. This has inspired us to develop a fuzzy window that can reduce the overlapping features of neighboring ordinal classes.

2.3. Fuzzy Scoring for Ordinal Classification

Before using fuzzy logic to disjoint the characteristic adhesion between two neighbor categories, an OR-CNN is typically designed to be used for age estimation [24]. There is an expectation layer that takes the predicted distribution and label set as input and emits its expectation:

$$\tilde{y}=\sum _{k=0}^{K-1}{P}_k{l}_k,$$

(1)

where $P_k$ denotes the prediction probability that the input image belongs to label $l_k$. Given an input image, the expectation regression module minimizes the error between the expected value $\tilde{y}$ and ground truth $y_{true}$. We use the below loss as the error measurement:

$${\mathit{Loss}}_{err}=|\tilde{y}-y_{true}|,$$

(2)

where $|·|$ denotes absolute value. Note that this module does not introduce any new parameters. OR-CNN adopts a general image classification framework that maximizes the probability of the ground-truth class during training. However, because each class is naturally influenced by its neighbors (in Figure 3, we can see that the 20–39 age group has a feature overlap with the 40–59 age group), the training would become unstable.

When using fuzzy logic to solve the ordinal regression problem, one strategy achieved outstanding performance by extracting a set of fuzzy rules from an example set and using it as the basic model with a genetic algorithm [25]. Moreover, a method based on monotonicity indexes, an evolutionary fuzzy systems algorithm, was used for ordinal classification and ordinal regression tasks [26]. However, there is no common approach that can cover most ordinal image classification problems because most researchers prefer to develop particular methods or systems to target specific problems.

Inspired by Deep Expectation (DEX), a fuzzy scoring method was used to reduce the influence of the tails and the shared features in each class and thus weaken the feature overlaps during the training steps [17]. Our previous work proposed a fuzzy window that focused on softly pulling the shared features to the optimal position by balancing the position of features with the distance away from the center. Under the conditions in Figure 1a, we set the length of the fuzzy window to 3, and the ascending (or descending) trend of the high score (or low score) was 1. Alternatively, under the condition of Figure 1c, in order to reduce the influence of the overlapping features, we set the length of the fuzzy window to 5. With this setting, the ascending or descending respective high or low score trend was 2. Eventually, the output value modified by the fuzzy window tended to slip forward to the global average position, and they optimized the redistributed probabilities as follows:

$$\tilde{P}\left(x_i\right|y_i=i)=\frac{|i-{\tilde{V}}_o|}{b-a}\times \sum _{j=i-a}^{i+b}\frac{e^{-E(y_j,x_j)}}{\sum {}_{y_1}^{y_K}{e}^{-E(y_j,x_j)}},$$

(3)

where b is the upper bound of the fuzzy window, and a is the lower bound. $\tilde{P}$ is the probability after using fuzzy windows. $E(y_j,x_j)$ is the expectation that $x_j$ is predicted as $y_j$. ${\tilde{V}}_o$ is used to reduce the conglutination between two either neighbors or remote classes. This was calculated with:

$${\tilde{V}}_o\left(x_j\right|y_j=j)=\frac{j\times P\left(x_j\right|y_j)}{{\sum }_{j=i-a}^{i+b}P\left(x_j\right|y_j)}.$$

(4)

2.4. Soft Labels and Gaussian Processes

Hard labels. This type of label is traditionally used as a one-hot vector. For example, the encoding $h_{\mathit{i}}=[0;1;0]$ means that $x_{\mathit{i}}$ is annotated to be the second class and $y_{\mathit{i}}=2$. However, the undesirable application in classifying ordinal images is its disadvantage, as many of them are ambiguous or have an unclear “borderline”. Thus, it brings difficulties to researchers as to which class they should belong to [27]. These hard labels tend to create an artificial gap that ponderously defines the borderline, and this instinctual drawback might then reduce the ability of the network to adapt [28].

Soft labels. By contrast, soft labels annotate categories by representing corresponding classes with a probability vector. For example, the encoding $h_{\mathit{i}}=[0.1;0.7;0.2]$ indicates that $P(Y=2|X=x_i)=0.7$. Thus, the value of the true label has been switched from 1 to $0.7$ [27]. Instead of using a single bit, soft labels with probabilities can provide extra information to the training models [29]. Meanwhile, they have information inheritance, which can resist disturbance during inference [30,31,32].

Gaussian Processes. Gaussian process approaches for ordinal regression have been studied based on support vector machines [33], deep neural networks [7], and deep learning models with Gaussian distribution labels [34,35,36]. One partial label machine learning study used the Gaussian Processed (GP) approach to address and disambiguate the vague labeling information conveyed by the training data [8]. It assumed that there was already an unobservable latent function depending on the Gaussian process in the feature space of each class label. The essential problem, however, is that manually affected and annotated ambiguous labels have been ignored. The reason is that the Gaussian distribution, at times, could not really represent realistic labels without logic deblurring.

For facial-age detection, many technical articles, for example, RCNN [22,37,38,39], Distribution Learning (DLDL) [9,30], and DLDL2 [2], reported that all of them trend towards the implicit use of the learning label distribution method, and data distribution is a Gaussian-like distribution.

The distribution of facial ages is represented by a Gaussian distribution for which a lookup table is generated beforehand to store multi-part integrals [40]. These integrals can explain the probability of whether an input image should belong to the true chronological age of a given person whose multiple age samples have been provided. In [10], a label distribution learning with a normal distribution variance $\sigma$ was used and proposed $p_{\mu }(y,\sigma )$ to represent the k-th $(k\in [0,99\left]\right)$ element of $p(y,\sigma )$:

$$p_{\mu }(y,\sigma )=1/\sqrt{2\pi {\sigma }^2}{e}^{-\frac{{(y-\mu )}^2}{2{\sigma }^2}},$$

(5)

where $p_{\mu }$ is the probability that the true age is $\mu$ years old. It represents the connection between the classes $\mu$ and y in a normal distribution view. The optimal $\sigma$ in each iteration depends on the optimal model parameter ${\theta }^{*}$:

$${\theta }^{*}\left(\sigma \right)=argmi{n}_{\theta }{L}_{KL}(H,y_{true},\theta ,\sigma ),$$

(6)

where $L_{KL}(H,y_{true},{\theta }^{*},\sigma )$ denotes the train loss. H is the training input image, while $y_{true}$ is its label. $KL$ is the Kullback–Leibler divergence.

3. Our Method

For ordinal regression, the most effective and popular method is using multiple binary classifiers to determine the ordinal category for each input (the K-rank approach) [2,4,30]. But the fundamental principle of this strategy is the consistency of the ordinal regression data [22]. In this section, we propose a simple and intuitive method that frames ordinal regression as a traditional classification problem, uses the Gaussian processed labels to stretch the shared features between two ordinal neighbors, and, finally, combines these Gaussian processed labels with a fuzzy window [17] to stabilize the weights on shared features.

3.1. Normalized Gaussian Processed Labels

After we set the equivalent double wings of the fuzzy window, which means $i-a=b-i$, we get the fuzzy $window=\left\{wi{n}_1=x_{i-a},\dots ,wi{n}_{a+1}=x_i,\dots ,wi{n}_{a+b+1}=x_{i+b}\right\}$. The true label is defined as:

$$Label\left(x_i\right|y_i=i)=\left\{\begin{array}{ccc}\hfill 0& \mathrm{for}& x_i\ne i\hfill \\ \hfill 1& \mathrm{for}& x_i=i,\hfill \end{array}\right.$$

(7)

and then the Gaussian processed label $Labe{l}_g$ can be:

$${Label}_G\left(x_i\right|y_i=i)=\left\{\begin{array}{ccc}\hfill 0& \mathrm{for}& x_i\notin window\hfill \\ \hfill \frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{{(x_i-\mu )}^2}{2{\sigma }^2}}& \mathrm{for}& x_i\in window,\hfill \end{array}\right.$$

(8)

where $x_i$ is predicted to be $y_i$, $y_i$ is the annotated label, i is the ordinal number, ${\mu }_o$ is the serially ordinal number of the true label, and $\sigma$ is the variance. Here, we set $\sigma =1/\sqrt{2\pi }\approx 0.4$ so as to ensure that when $x_i=\mu$, $Labe{l}_g$, it can be 1.

In

Table 1. An example of using Gaussian labels. There are seven categories ($C_{\mathit{1}}$ to $C_{\mathit{7}}$), a probability vector, original labels, original labels’ errors, Gaussian windows ($\mu$ = 4, and $\sigma$ = 0.5), Gaussian processed labels ($\mu$ = 4, and $\sigma$ = 0.5), and Gaussian processed labels’ errors.

Category	${\mathit{C}}_{\mathit{1}}$	${\mathit{C}}_{\mathit{2}}$	${\mathit{C}}_{\mathit{3}}$	${\mathit{C}}_{\mathit{4}}$	${\mathit{C}}_{\mathit{5}}$	${\mathit{C}}_{\mathit{6}}$	${\mathit{C}}_{\mathit{7}}$
Probability Outputs	0.19	0.1	0.01	0.4	0.18	0.09	0.03
Original Labels	0	0	0	1	0	0	0
Errors	0.19	0.1	0.01	−0.6	0.18	0.09	0.03
Gaussian Window	0	0.05	0.1	0.7	0.1	0.05	0
Gaussian Labels	0	0.07	0.14	1	0.14	0.07	0
${\mathbf{Errors}}_{\mathbf{G}}$	0.19	0.03	−0.13	−0.6	0.04	0.02	0.03

, we illustrate the essential utilization difference between GP labels and traditional original labels. We assume there are seven categories in this example. The output probabilities of these seven categories are generated using an artificial design. In Table 1, the traditional back-propagation error vector (Errors of the output layer = probabilities of the output—original labels) should be $[0.19,0.1,0.01,-0.6,0.18,0.09,0.03]$. After we use the traditional original labels, there is only one negative error resulting from the back-propagating calculation. If, however, we apply the GP labels on the back-propagation processing, the original hard label vector will switch from $[0,0,0,1,0,0,0]$ to the soft label vector $[0,0.07,0.14,1,0.14,0.07,0]$. After using GP labels, the back-propagation error vector ($Error{s}_G$ of the output layer = probability outputs—Gaussian labels) will turn to $[0.19,0.03,-0.13,-0.6,0.04,0.02,0.03]$. The output probability of $C3$ is lower than a systematic value—here, we assume this value was generated from the Gaussian function—and there would be two negative errors, which, in the next step, are used for back-propagation.

The ordinal vector is $Ordinal$ = $\{1,2,\dots ,n\}$, and m is the total number of ordinal categories. Because we used cross-entropy as the loss function, the back-propagation error between the output and the last layer after using the $Labe{l}_g$ was:

$${▽}_g\left(L\right)=|P\times Ordinal-y_i|.$$

(9)

The gradient of the weight from the ${\alpha }_{th}$ neuron in the layer $L-1$ to the ${\beta }_{th}$ neuron in the layer L after using the $Labe{l}_g$ was:

$${▽}_g(L-1)=\left\{\begin{array}{ccc}\hfill P\times \frac{\partial E_m}{\partial W_{L-1}^t(\alpha ,\beta ))}& \mathrm{for}& x_i\notin window\hfill \\ \hfill {▽}_g\left(L\right)\times \frac{\partial E_m}{\partial W_{L-1}^t(\alpha ,\beta ))}& \mathrm{for}& x_i\in window,\hfill \end{array}\right.$$

(10)

where $E_m$ is the expectation output of the $m_{th}$ category, and $W_{L-1}^t(\alpha ,\beta )$ is the weight matrix of the ${\alpha }_{th}$ neuron in the layer $L-1$. We find the value of ${▽}_g\left(L\right)$ cannot always stay positive, which means when $x_i\in window$, ${▽}_g(L-1)$ should be merged using the multiplication product of ${▽}_g\left(L\right)$, $sign(P-e^{-\pi {(x_i-\mu )}^2})$ and $\frac{\partial E_m}{\partial W_{L-1}^t(\alpha ,\beta ))}$.

The difference between using GP labels and original labels is presented in Figure 4. We assume that there are two adjacent ordinal categories, $C_{i-1}$, and $C_{\mathit{i}}$, and, at the same time, shared quantized features, which are represented by the grey area. In the first round of Gradients Decent Directions (GDDs), the original center of the shared quantized features is located at $C_a\left(0\right)$. If the true label is $C_{\mathit{i}}$, with the updating of the weights of models when using the original labels and back-propagation of errors, the initialized location of $C_a\left(0\right)$ would slide to $C_a\left(1\right)$ (see it in the Figure 4a). In the second GDDs round, if the true label is $C_{i-1}$, according to the vector direction of the pulling force, the center will slide to $C_a\left(2\right)$. Finally, after inducting the location of $C_a\left(2\right)$, the center of the shared features would stay close to either $C_{\mathit{i}}$ or $C_{i-1}$ but not near the borderline.

Alternatively, when using the GP labels, the center of the shared features will fluctuate around the borderline of $\{C_{\mathit{i}},C_{i-1}\}$. By using the original labels, the pulling force of back-propagation is unidirectional, which means the center of the shared features will move toward $C_{\mathit{i}}$ or $C_{i-1}$ during every updating step. However, when using the GP labels, the pulling force of back-propagation is a resultant force generated from the $C_{\mathit{i}}$ side and the $C_{i-1}$ side.

This part is very similar to the Fast Gradient Sign Method (FGSM) in both non-targeted and targeted adversarial attacks [41,42,43]:

$$H^{adv}=H+ϵ·sign\left({▽}_{H}J(H,y_{non-target})\right)$$

(11)

and

$$H^{adv}=H+ϵ·sign\left({▽}_{H}J(H,y_{target})\right),$$

(12)

where x is the input image, $x^{adv}$ is the perturbed adversarial image, J is the classification loss function, $y_{non-target}$ or $y_{target}$ is the true label for the input H and $ϵ$ can control the steps toward to the targeted or non-targeted image. In our method, this step depends on $sign(P-e^{-\pi {(x_i-\mu )}^2})$, and the targeted category is the Gaussian processed neighbor of the true label.

3.2. Fuzzy Windows with Normalized Gaussian Processed Labels

In order to make models more stable, practically, we should weaken the influence of gradients and use a lower learning rate or a smaller updating gradient. Additionally, when considering the optimal global strategy of only using a fully connected layer, the Fuzzy Fully Connected Layer (FFCL) can bring weaker influence into whole neural networks [17]. Thus, regardless of the strength of the pulling force (the gradient matrices in every layer), the center of the shared features can slide relatively smoothly to the optimal position. Then, this combined method is more beneficial to do classification in the output OR-CNN layer.

We use the DEX method as the base, and the true label y is quantized to different label groups, which are treated as a class. To train DEX with fuzzy windows and normalized Gaussian processed labels, we replaced the expectation module—the last output layer—with fuzzy windows of different lengths, used a Gaussian function ( $\sigma =1/\sqrt{2\pi }\approx 0.4$) to process the ordinal labels, and, finally, modified the loss function with a typical cross-entropy loss. The back-propagation error between the output and the last layer after using the $Labe{l}_g$ was:

$${\widetilde{▽}}_{x_i}l(x_i,y_{true})=\left\{\begin{array}{ccc}\hfill \tilde{P}-0& \mathrm{for}& x_i\notin window\hfill \\ \hfill \tilde{P}-Labe{l}_g& \mathrm{for}& x_i\in window,\hfill \end{array}\right.$$

(13)

where $\tilde{P}$ is calculated from Equations (3) and (4).

In Algorithm 1, we provide the pseudo-code of the fuzzy window with a normalized Gaussian processed label algorithm on processing the ordinal regression issue. The first step is to process the labels with Gaussian distribution. After setting the length of the Gaussian window, $L_{Win}$, the $Labe{l}_G$ can be calculated according to the Gaussian processing template in Table 1. However, if the Gaussian window reaches the head or the tail of the whole age sequence (0 or m), the element which is out-of-range (for example, if the front side of the window $Frt<0$ ($Frt=i-L_{hWin}$) or if the back side of the window $Bk>m$ ($Bk=i+L_{hWin}$)) should be removed. The second step is to use fuzzy logic to eliminate the influence of overlapping features in ordinal neighbor classes. $\widetilde{P_i}$ can be calculated out using Equation (3), and ${\tilde{V}}_o$ is able to be computed from Equation (4). During the inference time of the fuzzy window, an expected value, the sum of the multiplication of two elements—the position of the binary classifier and the prediction probability of this specific classifier—is used for the final estimation.

Algorithm 1 Fuzzy Windows with Normalized Gaussian Processed Labels

4. Experiments

In this section, we introduce one medical image dataset (CBIS-DDSM) and three facial-age datasets (IMDB-WIKI, FG-NET, MORPH-2, CACD). In the following, there are three experimental ablation results. The first shows the performance on the selection of the hyperparameter $L_{Win}$. The second ablation study presents the performance of FW-GPL in processing a designed fragmentary ordinal dataset. The last one demonstrates comparison results with SOTA methods on three facial-age datasets.

4.1. Datasets

In this study, there are one medical image dataset and four different facial-age estimation datasets (one for pretraining).

4.1.1. Ordinal Medical Dataset

Table 2. Sample distribution of CBIS-DDSM dataset based on BI-RADS assessment.

Scores (BI-RADS)	0	1	2	3	4	5
Training Set (Mass  +  Calcification)	192 (129 + 63)	1	559 (77 + 482)	368 (279 + 89)	1286 (533 + 753)	458 (299 + 159)
Testing Set (Mass  +  Calcification)	46 (33 + 13)	2	85 (14 + 71)	109 (85 + 24)	347 (169 + 178)	115 (75 + 40)

shows the size of one ordinal medical dataset and its corresponding splits for training and testing.

CBIS-DDSM. CBIS-DDSM (Curated Breast Imaging Subset of DDSM) is a large collection of digitized film mammography images, which includes 3572 images referring to 2689 patient cases. According to BI-RADS, overall BI-RADS assessment from 0 to 5 has been described in this dataset, including BI-RADS score 0 (Incomplete cases), BI-RADS score 1 (Negative cases), BI-RADS score 2 (Benign cases), BI-RADS score 3 (Probably Benign cases), BI-RADS score 4 (Suspicious Abnormal cases) and BI-RADS score 5 (Highly Suspicious Malignant cases); (This CBIS-DDSM dataset is available at https://wiki.cancerimagingarchive.net/display/Public/CBIS-DDSM, accessed on 18 February 2023).

4.1.2. Facial-Age Estimation Datasets

Table 3. Facial-age datasets used to evaluate the proposed FW-GPL.

Datasets Name	Train	Test	Val	Total	Label Range
IMDB-WIKI	260,282	⊗	⊗	523,051	0–100
FG-NET	990	12	⊗	1002	0–69
MORPH 2	4380	1095	⊗	5475	16–70
CACD	145,275	10571	7600	163,446	⊗

shows the size of each dataset, and the corresponding splits for training and testing.

IMDB-WIKI. For the IMDB-WIKI dataset (IMDB-WIKI can be downloaded from http://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/, accessed on 18 February 2023), the authors crawled images of celebrities from IMDB (www.imdb.com, accessed on 18 February 2023) and Wikipedia (https://en.wikipedia.org/, accessed on 18 February 2023).

FG-NET. The Face and Gesture Recognition Research Network (FG-NET) [15] aging database consists of 1002 color and grey-scale images, which were taken in a totally uncontrolled environment. On average, there are 12 images for each of the 82 subjects, whose age ranges from 0 to 69 (FG-NET is available at https://yanweifu.github.io/FG_NET_data/, accessed on 18 February 2023).

MORPH-2. The Craniofacial Longitudinal Morphological Face Database (MORPH) [14] is the largest publicly available longitudinal face database containing more than fifty thousand mug shots ( You can find MORPH-2 from https://www.faceaginggroup.com/morph/, accessed on 18 February 2023).

CACD. The Cross-Age Celebrity Dataset (CACD) [16] collected from the Internet contains 163,446 images from 2000 celebrities. This dataset splits into three parts, 1800 celebrities are used for training, 80 for validation, and 120 for testing (The link of CACD is http://bcsiriuschen.github.io/CARC/, accessed on 18 February 2023).

4.2. Evaluation Metrics

For model evaluation and comparison [44], we computed the Mean Absolute Error (MAE) [45] and Root-Mean-Square Error (RMSE) [46], on the test set after the last training epoch:

$$MAE=\frac{1}{N}\sum _{n=1}^N|\tilde{y}-y|$$

(14)

$$RMSE=\sqrt{\frac{1}{N}{\sum }_{n=1}^N{|\tilde{y}-y|}^2}$$

(15)

where $\tilde{y}$ is the output value of OR-CNNs, y is the real facial age label, and N is the total number of test samples.

4.3. Experiment Settings

Following DEX [21], SSR [5], Mean-Variance Loss [3], and C3AE [4], the model can be first pre-trained on the IMDB-WIKI dataset. This method can be embedded into any CNN ordinal classification model. We respectively set the length of the fuzzy window $L_{Win}$ as 10 for facial-age detection and 3 for breast cancer detection. We used the Adam optimizer in all the experiments, and similarly to SSR and C3AE, the initial learning rate, dropout rate, momentum, and weight decay were set to 0.002, 0.2, 0.9, and 0.0001, respectively. The learning rate was $0.001$ with a decay every 10 epochs by a factor of $0.9$. Compared with the SOTA methods, each model totally trained two hundred epochs with a batch size of 50. During the training steps, to avoid overfitting the overlapping features, we adjusted the training strategy according to Algorithm 2.

Algorithm 2 Training $Model$

4.4. Hardware and Software

All loss functions and neural network models were implemented in MATLAB2019b and PyTorch 1.7 and trained on four Tesla V100 graphics cards (The source code is available at https://github.com/ChengKang520/FW-with-GPL-for-Ordinal-Regression, accessed on 18 February 2023).

5. Results and Analysis

So as to compare with the SOTA results, we respectively summarize the comparison result of CBIS-DDSM in

Table 4. Comparison with existing methods on DDSM in terms of ACC.

Method	CNN  + FCL	CNN  +  FFCL	CNN  + W-GPL
Geras [47] (BI-RADS: 0/1/2)	68.8%	70.1%	70.3%
Akselrod-Ballin [48] (BI-RADS: 2/(3-4-5))	60.0%	62.3%	62.4%
Kang [17] (BI-RADS: 0/(2-3)/(4-5))	72.0%	74.1%	74.2%
Kang [17] (BI-RADS: 0/1/2/3/4/5)	56.34% ±1.4%	57.40% ±1.7%	58.29% ±1.9%

and the comparison result of facial-age detection in

Table 5. In terms of MAEs, our approach is compared with different SOTA methods. (* indicates the model was pre-trained on the IMDB-WIKI dataset.)

Type	Method	MORPH 2	FG-NET	CACD	Paras
Bulky	DEX [21]	3.25	4.63	-	138 M
	DEX * [21]	2.68	3.09	6.52	138 M
	MV [3]	2.41	4.10	-	138 M
	MV * [3]	2.16	2.68	-	138 M
	DLDL-v2 [2]	1.969	-	-	138 M
	FP-Age [49]	2.04	5.60	5.60	138 M
	FP-Age * [49]	1.90	4.68	4.33	138 M
	DRF [6]	2.80	3.47	5.63	-
	PML [50]	2.31	2.16	-	-
	JREAE [13]	2.71	3.390	4.596	-
	MWR [18]	2.13	-	5.68	-
	FW-GPL [Ours]	2.71	4.27	-	138 M
	FW-GPL * [Ours]	2.24	2.73	6.10	138 M
Compact	ORCNN [3]	3.27	6.44	-	479.7 K
	MRCNN [3]	3.42	-	-	479.7 K
	SSR [5]	3.16	-	-	40.9 K
	C3AE [4]	2.78	4.09	-	39.7 K
	C3AE * [4]	2.75	2.95	-	39.7 K
	AVDL * [10]	2.37	2.32	-	11 M
	MWR [18]	2.00	2.23	-	-
	FW-GPL[Ours]	2.72	3.71	-	40.9 K

.

5.1. Scoring Breast Cancer Images

As we set the hyperparameter $L_{Win}=3$ when scoring BI-RADS, only when the categories are greater than 3 in number can our FW-GPL work well to reduce the influence of overlapping features among neighboring ordinal classes—this can also be seen in Table 4. We find that only when scoring the BI-RADS of six categories does FW-GPL show a weak but obvious improvement. The distance d between BI-RADS score 2 (benign) and BI-RADS score 3 (probably benign) is probably beyond the “boundary”, as is the distance d between BI-RADS score 4 (suspicious abnormal) and BI-RADS score 5 (highly suspicious malignant); therefore, the classification task for BI-RADS is difficult.

5.2. Scoring Facial-Age Images

With respect to scoring facial-age images, we set the number of neurons to 10 and the length of the window to 5. The results of a comparison between our model and SOTA models on three facial-age datasets are summarized in Table 5.

Compared with label distribution learning methods such as Deep Label Distribution Learning V2 (DLDL-V2), ref. [2] and MV Loss [3], FW-GPL leverages a fixed pattern (Gaussian processed labels) to learn features that have considered the issue of age distribution. Our FW-GPL does not need to know beforehand the age distribution of the image data. Compared with some models that use special loss functions, we find that our FW-GPL, in particular, achieves competitive results compared with most SOTA methods, for example, MV Loss [3], SSR [5], and C3AE [4]. This is because the fuzzy window reduces the influence of the conjugation among neighboring ordinal categories, and it does not like DLDL-V2 [2], MV Loss [3], and SSR [5] considering the whole probability, or as with C3AE [4], focusing only on the two highest output probabilities. The second reason is that the Gaussian processed labels do not need to fit a proper hyperparameter $\sigma$ [10] that should approximate the true age probability distribution. Compared with our FW-GPL, MWR [18] developed global and local relative ordinal regressors ($\rho$ regressors) to predict $\rho$ ranks within the entire and specific rank ranges. Furthermore, MWR first refined an initial search window, iteratively moved it by selecting two reference instances, and, lastly, estimated the $\rho$ rank within the window.

6. Ablation and Discussion

Based on the facial-age image classification, we used the ordinal IMDB-WIKI data to do the ablation analysis. The ablation study was conducted in three parts: (1) to analyze the influence of the number of neurons, (2) to analyze the influence of the length of window $L_{Win}$, and (3) to figure out how this model could process incomplete ordinal data.

6.1. Ablation Study I (Influence of the Number of Neurons)

We used the classical pre-trained DEX model as the base. In

Table 6. Test performance of the FW-GPL method, with the $L_{Win}=10$ (set length of output neurons N as [100, 50, 20, 10, 5]).

Method	DEX	DEX with FW-GPL
$\mathit{N}$	100	50	20	10	5
RMSE	12.46	13.36	12.65	12.60	12.80
MAE	8.94	8.67	8.79	8.62	8.59

and

Table 7. Test performance of the DEX method (set length of output neurons N as [100, 50, 20, 10, 5]).

Method	DEX	DEX without FW-GPL
$\mathit{N}$	100	50	20	10	5
RMSE	13.57	13.38	12.86	12.67	12.71
MAE	8.96	8.83	8.77	8.64	8.74

, we see that when the neuron number N is 10 or 5, the DEX model can get the best performance. This finding echoes prior research showing that when the number of neurons in the output layer is 10 or 5, DEX-family age detection models can achieve better performance [18]. In other words, a smaller N has a better error tolerance.

6.2. Ablation Study II (Influence of the Length of the Window $L_{Win}$)

We used two types of output layers (when $N=100$ and $N=10$) to analyze the performance of FW-GPL under different $L_{Win}$, with the results summarized in

Table 8. Test performance of FW-GPL on testing data sets (length of output neurons set as 100).

Method	DEX with FW-GPL
${\mathit{L}}_{\mathit{Win}}$	5	10	20	50	100
RMSE	15.17	15.10	14.58	13.65	13.68
MAE	10.18	10.11	9.78	9.69	9.71

and

Table 9. Test performance of FW-GPL on testing data sets (length of output neurons set as 10).

Method	DEX with FW-GPL
${\mathit{L}}_{\mathit{Win}}$	5	10	20
RMSE	12.91	12.60	12.60
MAE	8.78	8.62	8.62

. As the length of the half window $L_{hWin}$ should be greater than the number of neurons in the output layer, we find that when the number of the neurons in the output layer is 100, a wider window ($L_{Win}=50$ or $L_{Win}=100$) gets better performance. Additionally, when the number of neurons in the output layer is 10, we can find the same trend. That is because the wider window can contain sufficient information to estimate the facial age. But if $N\le L_{hWin}$, there is no improvement. These denote that a proper window length can improve the performance of the FW-GPL model.

6.3. Ablation Study III (Incomplete Ordinal Image Data)

We manually removed some age segments of the IMDB-WIKI to train the model and test it in the complete ordinal text data, as shown in Figure 5. In

Table 10. Test performance between the DEX and the FW-GPL with fragmentary IMDB-WIKI dataset. (length of output neurons set as 100).

Method	DEX	FW-GPL
${\mathit{L}}_{\mathit{Win}}$	0	5	10	20	50
RMSE	12.46	14.73	14.30	13.64	12.72
MAE	8.94	9.43	9.13	8.78	8.81

, we can see that when the number of neurons is 100, the most proper window is 20. In

Table 11. Test performance between the DEX and the FW-GPL with fragmentary IMDB-WIKI dataset. (length of the window set as 10).

Method	DEX	FW-GPL
$\mathit{N}$	101	50	20	10	5
RMSE	12.46	13.36	12.65	12.60	12.80
MAE	8.94	8.67	8.79	9.08	8.59

, when we set the length of the window as 10, the lowest MAE appears when the number of neurons is 5. Consequently, there is no obvious difference between the incomplete (this section) and complete (Ablation Study II) ordinal image data, and the result can only be affected by the number of neurons N and the length of the window $L_{Win}$.

6.4. Advantage and Limitation

By directly facing the challenge of ordinal image classification, our method attempts to reduce the influence of the overlapping features. The length of the window controls the defuzzification of the ordinal neighbor categories. JREAE [13] used two covariance matrices to capture the underlying correlations from both aspects of input facial features and output age labels, but this family of methods (e.g., DRF [6] and AVDL [10]) should first take the age distribution of the dataset into account. This consideration is necessary because after fitting the distribution of the facial age dataset, there is an inevitable deviation between the real distribution of the age and the fitted one. To avoid such a problem, our method used a Gaussian distribution within the window to approximate the relationship between input facial features and output age labels. From Table 5, our method outperforms other LDBL methods and presents the advantage of using label distribution-based learning methods.

However, the disadvantage is that we only use a naive fuzzy logic window to leverage the challenge of ordinal image classification tasks. By adaptively adjusting the distance between the real age and the center of the moving window, MWR [18] moved the window to fit the $\rho$-ranks within entire and specific age ranks. Our method constrained the center of the window by using naive fuzzy logic to adjust the distribution of the facial age in the window. That would ignore the influence of the remote but highly related feature, which is beyond the window. Even though we tried to use longer windows, our method failed to overcome this problem.

7. Conclusions

In this paper, we have proposed a novel method for ordinal image scoring named fuzzy window with Gaussian processed label learning (FW-GPL). FW-GPL introduces a method to reduce the influence of the overlapping features between two ordinal neighbors. It achieves better performances than others on multiple age estimation datasets and one ambiguously annotated medical dataset. Our experiments also show that FW-GPL can process discontinuous ordinal regression by setting the proper length of the windows.

The idea of using fuzzy logic and a Gaussian process strategy to guide ordinal image classification is inspirational, and we will explore more possibilities for it. There are many directions for future work. (1) There are many other ordinal medical tasks, for example, scoring the severity of depression and grading the injury of spinal cords. We will use this method on such medical tasks in the coming research. (2) Our method cannot achieve the best SOTA result. We will try to overcome this challenge by infusing FW-GPL into other SOTA models. (3) To save the computing cost, we will fine-tune the pre-trained models which have been inserted with FW-GPL.