WB Score: A Novel Methodology for Visual Classifier Selection in Increasingly Noisy Datasets

Abstract

This article addresses the challenges of selecting robust classifiers with increasing noise levels in real-world scenarios. We propose the WB Score methodology, which enables the identification of reliable classifiers for deployment in noisy environments. The methodology addresses four significant challenges that are commonly encountered: (i) Ensuring classifiers possess robustness to noise; (ii) Overcoming the difficulty of obtaining representative data that captures real-world noise; (iii) Addressing the complexity of detecting noise, making it challenging to differentiate it from natural variations in the data; and (iv) Meeting the requirement for classifiers capable of efficiently handling noise, allowing prompt responses for decision-making. WB Score provides a comprehensive approach for classifier assessment and selection to address these challenges. We analyze five classic datasets and one customized flooding dataset in São Paulo. The results demonstrate the practical effect of using the WB Score methodology is the enhanced ability to select robust classifiers for datasets in noisy real-world scenarios. Compared with similar techniques, the improvement centers around providing a visual and intuitive output, enhancing the understanding of classifier resilience against noise, and streamlining the decision-making process.

1. Introduction

Computational classification methods utilizing machine learning algorithms have gained significant popularity due to their ability to learn patterns and make accurate predictions automatically [1,2]. However, real-world scenarios often present challenges in the form of uncertainties and noise inherent in the datasets used for training and testing these classifiers [3]. This article focuses on a scenario where classifiers are trained once and deployed in production environments, continuously facing datasets with increasing noise levels over time.

When dealing with datasets prone to increasing noise, several challenges arise in selecting the most appropriate classifiers. The following problems are commonly encountered: (i) Classifiers must exhibit robustness to noise, ensuring accurate predictions despite noisy instances or attributes in the data. The selection of classifiers that can effectively handle varying levels of noise is crucial to maintaining high prediction performance over time [4]; (ii) It is often difficult to collect and label large amounts of data representative of the noise encountered in the real world. As a result, classifiers may be trained on datasets that are not representative of the noise that they will encounter in production, which can lead to poor performance [5]; (iii) Noise can take many different forms making it difficult to distinguish between noise and real data variation [6]; (iv) There are scenarios where efficient handling of real-time data streams with increasing noise requires classifiers that can process incoming instances promptly. In cases like this, solutions like ensemble methods or active learning techniques may not be effective since retraining time can be highly costly [7].

This paper presents the novel “WB Score” methodology for selecting a robust classifier in increasingly noisy environments. The methodology inputs a dataset and a list of classifiers to test, determining the most reliable despite significant noise. By providing an intuitive graph that visually represents classifier performance, decision-makers can efficiently assess and choose an appropriate classifier based on their specific needs and the overall noise levels. Although there are graphic or visual methodologies such as robustness curves [8], noise tolerance plots [9], box plots [10], stacked bar charts [11], and heatmaps [12], none of them were designed to take into account the increase in noise nor to analyze a set of classifiers at the same time to select the best one for the task to be performed.

This paper is organized as follows: Section 2 introduces relevant related works; Section 3 presents the Data and Methods; Section 4 shows the results and relevant discussions of this analysis; finally, Section 5 contains the conclusions.

2. Related Works

As an application of machine learning, classification techniques are used to distinguish instances of a dataset into classes or groups of similar elements [13]. These techniques are widely applied in several areas of knowledge area, for example, biology [14], biometric authentication [15], computer vision [16], document classification [17], development of new drugs [18], pattern recognition [19], and natural language processing [20]. For each of these areas, there are peculiarities in their explanatory variables (or features) that make a classification algorithm better than another for a given purpose or dataset.

However, for the composition of a dataset, there may have needed to be more accurate readings in the collected values, causing errors or imprecision. This work presents a ranking method for classification algorithms that consider adding noise in the original dataset, thus enabling the choice of the most robust algorithm under this condition. The selected algorithm can then be of great value in datasets with known noisy or inaccurate information.

The “No Free Lunch theorem” [21] says that no single algorithm always performs better than others in all datasets. Depending on its data, a classifier can give the best response among all others due to some of its internal implementations, which deal better with the existing dataset bias [22]. Finding the better solution for a practical problem is formally known as the Algorithm Selection Problem in literature [23].

At first, the most logical approach for selecting a classification algorithm for a new problem is to compare the performance of previous similar ones, choosing the best option historically found. In this case, subjects like ranking [24,25,26,27], meta-learning [28,29], problem difficulty [30], and knowledge discovery [31] can also be addressed.

Another approach found is using multiple classifier systems as an alternative to improve accuracy. The idea is to have a classifier pool that can be used to compose the result, selecting the best classifier for each sample to be classified. Several studies have demonstrated its advantages over individual classification models [32].

The literature needs to include the approach of selecting the best classifier for the case in which we have a noisy dataset or is error-prone in its data acquisition [33,34]. However, some articles demonstrate that using MCS helps to reduce the effects of noise in the analyzed datasets [35,36,37]. The proposed methodology is a solution to fill this gap, allowing the analysis of several classifiers simultaneously to choose one that is more robust to errors or noise within the used dataset. The classifiers’ accuracy response on the original dataset and the datasets in which noise is introduced are considered to find the best suitable algorithm with the used dataset.

Some works use sensitivity analysis [38,39] to identify the relationship of all fields that make up the input data, trying to find out the most important ones and discard those who contribute with a tiny fraction, which can then be considered negligible [40]. Methods like One-Factor-at-a-Time [41] and Elementary Effects [42] are examples of this, and they are helpful in large modeled systems with dozens of inputs or more, but this work is different. Here, the objective is to check how the output response degrades from the original ones as the noise levels grow up, regardless of how the input data interact with themselves.

3. Data and Methods

The data are presented in Section 3.1, whereas the methods are explained from Section 3.2 through to Section 3.7.

3.1. Used Datasets

Five well-known classic public datasets were used in this experiment: IRIS, GLASS, IONOSPHERE, IMAGE SEGMENTATION, and SEEDS. All of them are available to download from the UC Irvine Machine Learning Repository (https://archive.ics.uci.edu (accessed on 1 July 2023)). A customized flooding dataset in São Paulo was also analyzed (FLOODINGS SP) [43].

Table 1. Description of datasets used in the experiment.

Dataset	Description	Instances	Attributes
IRIS	Classification of iris flowers into three species: setosa, versicolor, and virginica.	150	4
GLASS	Classification of glass types into six categories: window, bottle, table, vehicle, laboratory, and other.	214	9
IONOSPHERE	Classification of ionospheric conditions into three categories: quiet, disturbed, and very disturbed.	351	34
IMAGE SEGMENTATION	The instances were drawn randomly from a database of 7 outdoor images and were hand segmented to create a classification for every pixel.	210	19
SEEDS	Classification of seeds into three species: setaria italica, digitaria sanguinalis, and eleusine indica.	210	7
FLOODINGS SP	Classification of floodings events between 2015 and 2016 in São Paulo.	825	6

gives a short description of each dataset and the size of elements.

The attributes in each dataset vary in type and number. However, they all allow being tampered with generated noise, a necessary characteristic for evaluating the proposed methodology. All numerical values were tampered with multiplicative and additive noise.

It is interesting to note that all variables observed in real life are analog by nature, and we depend on instruments or sensors to measure them correctly [44]. In this sense, any data expressing some environmental characteristic will be expressed in real numerical values, making the numerical framework presented here perfectly applicable and coherent. It is impossible to apply the noises if the attributes were nominal or strings without probably changing their class, which would reflect in a dataset utterly different from the original.

3.2. WB Score Explained

Let $D_0=\left\{\right(x_{\mathrm{i}},y_{\mathrm{i}})ϵX\times Y:i=1,\dots ,m\}$ an original considered dataset; where x_i are observations and y_i a class label. Additionally, let $D_{\mathrm{e}}=\left\{\right(x_{\mathrm{i}}^{\left(\mathrm{e}\right)},y_{\mathrm{i}})\in X\times Y:i=1,\dots ,m\}$ a dataset with observations tampered with e% noise-level.

Assuming $f_{\mathrm{k}}\left(x\right):X\to Y$ a trained classifier through D_k, for $k=0,1,\dots ,\beta$; the accuracy regarding the original and noise-corrupted datasets, herein denoted by A₀ and A_c, respectively, are defined as:

$$A_0=\frac{1}{m}\sum _{i=1}^m\delta (f_{\mathrm{k}}\left(x_{\mathrm{i}}\right),y_{\mathrm{i}})$$

$$A_{\mathrm{c}}=\frac{1}{\beta .m}\sum _{k=1}^{\beta }\sum _{i=i}^m\delta (f_{\mathrm{k}}\left(x_{\mathrm{i}}\right),y_{\mathrm{i}})$$

where $\delta (a,b)=\left\{\begin{array}{c}1\mathrm{if}a=b\hfill \\ 0\mathrm{otherwise}\hfill \end{array}\right.$ is the Kronecker operator.

The basic idea of the proposed methodology is the analysis of possible models that may be used in a classification problem from a dataset that may contain noise in its data. The central evaluation metric used is the overall accuracy [45].

Applying different random noise intensities is a convenient procedure to assess the robustness of the model in this scenario. One can consider the original dataset as the baseline to be compared with other noise levels. Consequently, these perturbed datasets allow for analyzing how the error rates increase due to the added noise intensities.

For this discussion, we consider noise intensities ranging from ${\ell }_{inf}$% to ${\ell }_{sup}$%, by a progressive increase of ${\ell }_{step}$%. Regarding the perturbation process, the following expression is employed:

$${\tilde{\mathbf{x}}}_i={\mathbf{x}}_i+\frac{{\displaystyle ϵ\ell {\mathbf{x}}_i}}{{\displaystyle 2}};i=1,2,\dots ,m$$

(1)

where ${\tilde{\mathbf{x}}}_i$ is the resulting perturbation on ${\mathbf{x}}_i$; $ϵ$ is a random scalar sampled from a uniform distribution in $[-1,+1]$; and $\ell \in [{\ell }_{inf},{\ell }_{sup}]$ represents the noise level ranging from ${\ell }_{inf}\%$ to ${\ell }_{sup}\%$, respectively.

It is worth observing that the respective assignment between ${\mathbf{x}}_i$ and its “label” (${\mathbf{y}}_i$) is still persistent to ${\tilde{\mathbf{x}}}_i$, for $i=1,\dots ,m$.

In addition to calculating the accuracy achieved by the model when using the original dataset (A₀), the average of the accuracies of this same dataset with added noise ranging from ${\ell }_{inf}$% to ${\ell }_{sup}$%, with increment step of ${\ell }_{step}$%, is also calculated (A_c). Limiting the error or noise threshold in data measurement readings to a minimal or negligible value is one of the main goals when building a dataset, ensuring the assertiveness of the classification algorithm output.

Once the accuracy values are calculated, we can derive the $\rho$ and $\theta$ values that comprise the indicators of the proposed methodology, herein called the WB Score. Figure 1 describes the above-discussed methodology in a flow chart diagram.

In practice, the accuracy values are placed on a Cartesian graph with the original accuracy on the y-axis and the average of the accuracies with noise on the x-axis, as depicted in Figure 2. The vector resulting from the sum of these values indicates the evaluation of the classification algorithm tested, where $\rho$ is the value of its module and $\theta$ is the angle formed between the vector and the abscissa axis.

It is worth noting that, in this configuration, the $\rho$ module can be greater than 1. In order to keep it between [0, 1], we can normalize it. In this way, the $\rho$ and $\theta$ values can be expressed as follows:

$$\rho ={\left(\frac{A_{0^2}+A_{{\mathrm{c}}^2}}{2}\right)}^{\frac{1}{2}};[0,1]$$

(2)

$$\theta =arctan\left(\frac{A_0}{A_{\mathrm{c}}}\right);(0,\pi /2).$$

(3)

Analyzing Figure 2, we can point to the following properties:

(i)

Vectors with $\theta <\frac{\pi }{4}$ generally represent algorithms with robustness in relation to noisy datasets;

(ii)

Vectors with $\theta >\frac{\pi }{4}$ generally represent algorithms with robustness in relation to noiseless datasets;

(iii)

Vectors with $\theta \approx \frac{\pi }{4}$ represent algorithms with a balanced response between noisy and noiseless datasets.

Figure 3 graphically express these properties, highlighting regions for each one of them. The region comprising the balanced response was arbitrarily defined to $\theta$ between 43 and 47 degrees (45 ± 2 degrees).

The indicative vectors may have the same modulus value (length of $\rho$) but can be differentiated by their angular part (value of $\theta$). Thus, it is possible to choose a more robust algorithm for the case of a noisy dataset, a noiseless dataset, or a balanced solution between both.

The methodology will always return a position of the tested classifier within the delimited graphic area, except for the case in which the accuracies have a value of zero. In this case, we will have a mathematical uncertainty to calculate the arc whose tangent would be zero divided by zero. However, this is an undesirable situation, meaning that the classifier always gets its predictions wrong, and is therefore useless.

Lastly, it is worth mentioning that the accuracy evaluation metric is used in this introduction to the WB Score methodology. However, it can be changed to any other evaluation metric among dozens of others described in the literature, such as Kappa, Total Cost, Average Cost, KB, MAE, RMSE, RAE, RRSE Precision, Recall, F-measure, ROC, PRC, and others [46].

3.3. Selected Classifiers for Testing

Eight supervised classification algorithms were tested: KNN (K-Nearest Neighbors) [47]; Naïve Bayes (NB) [48]; Random Forest (RF) [49]; J48 (JAVA implementation of the original C4.5 classifier [50]); Random Tree (RT) [51]; Multilayer Perceptron (MLP) [52]; SVM with linear kernel [53]; and SVM with Radial Basis Function kernel [54]. One is a linear classifier (SVM with linear kernel), one implements the Bayes theory (Naïve Bayes), and all others are nonlinear-based classifiers.

Table 2. List of selected classifiers for testing.

Classifier Name	Acronym	Classification Type
Naive Bayes	NB	Bayes Theory
Support Vector Machine (linear kernel)	SVM (linear)	Linear
C4.5 (ported in JAVA)	J48	Nonlinear
K-Nearest Neighbor	KNN	Nonlinear
MultilayerPerceptron	MLP	Nonlinear
Random Forest	RF	Nonlinear
Random Tree	RT	Nonlinear
Support Vector Machine (Radial Basis Function kernel)	SVM (RBF)	Nonlinear

lists all of them.

3.4. The WEKA Software

WEKA (Waikato Environment for Knowledge Analysis) is a widely used and highly regarded open-source software suite for machine learning and data mining tasks. Developed at the University of Waikato in New Zealand, WEKA provides a comprehensive set of tools and algorithms for machine learning, including data preprocessing, classification, regression, clustering, association rules mining, and feature selection [55].

One of the primary strengths of WEKA lies in its extensive collection of classification algorithms. It offers a diverse range of techniques that can be employed to build predictive models from labeled datasets. These algorithms encompass traditional and state-of-the-art approaches, allowing researchers and practitioners to choose the most suitable method for their tasks. WEKA includes popular classification algorithms such as decision trees, random forests, Support Vector Machines (SVM), k-Nearest Neighbors (k-NN), naive Bayes, and neural networks.

WEKA’s classification algorithms excel in their flexibility and configurability. The software provides users with numerous options to customize the learning process, such as adjusting parameters, handling missing values, nominal and numeric attributes, and imbalanced datasets. This level of control allows for fine-tuning the models and adapting them to different types of data and problem domains.

In this experiment, the WEKA exploration module was extensively used to follow the methodology described in Section 3.2. All the classifiers’ implementations came from this module, included in the WEKA 3.9.6 version.

3.5. Fine Tuning with Grid Search

The Grid Search procedure [56] was used under the WEKA tool to fine-tune the classifiers’ core parameters.

Table 3. Best parameters’ configuration found using the Grid Search method.

Classifier	Fine-Tuned Parameter	Search Space	Best Configuration (IRIS, GLASS, IONOSPHERE, SEGMENTATION, SEEDS, FLOODINGS SP)
KNN	KNN distanceWeighting	{3, 5, 7, 9, 11} {None, 1/distance, 1-distance}	11, 3, 7, 3, 3, 3 1/distance (all)
NB	useKernelEstimator useSupervisedDiscretization	{0, 1} {0, 1}	1, 0, 1, 0, 0, 0 0, 1, 0, 1, 1, 1
RF	numIterations maxDepth	{10, 20, …, 190, 200} {1, 2, 3, …, 8, 9, 10}	200, 30, 60, 200, 80, 200 3, 7, 6, 10, 10, 7
J48	minNumObj unpruned	{1, 3, 5, 7, 9, 11} {0, 1}	3, 5, 5, 5, 1, 1 1 (all)
RT	breakTiesRandomly maxDepth	{0, 1} {1, 2, 3, …, 8, 9, 10}	1, 1, 0, 1, 0, 0 2, 6, 6, 5, 5, 9
MLP	learningRate momentum	{0.1, 0.2, …, 0.5} {0.1, 0.2, …, 0.5}	0.5, 0.5, 0.4, 0.5, 0.5, 0.5 0.5, 0.2, 0.1, 0.1, 0.1, 0.4
SVM (linear)	cost coef0	{1, 10, 100, 1000, 10000} {0, 1}	10000, 10, 100, 1, 100, 100 1 (all)
SVM (RBF)	cost gamma	{1, 10, 100, 1000, 10000} {0.01, 0.1, 1}	10, 10, 10, 10000, 10000, 1 0.01, 1, 0.1, 10000, 0.01, 0.1

shows the key parameters each classifier was fine-tuned, the original search space entered, and the actual best configuration found. All parameter names are referenced as they are found within implementations of the classifiers in WEKA.

3.6. Introduced Noises

In this experiment, three types of noise were introduced to generate the noisy test datasets:

• Multiplicative: random variations to the original value ensuring that the noise is centered around zero;
• Additive: random variations to the original value with the variations centered around the mean value;
• Both multiplicative and additive: noises are added and divided by two, so the result will remain within the desired noise level.

These types of noises appear when an output response depends on a linear combination of input data, the typical scenario in production environments. A linear combination involves the accumulation of additive and multiplicative errors, which is the behavior reproduced by this approach.

For the generation of each test dataset with a specified noise level, the original dataset was read line by line, and all numerical values were tampered with these three noise types, generating 100 new instances each. Only the label value remained the same. Ultimately, each test dataset was 300 times bigger than the original one.

More information about this procedure, the code, and the public datasets used can be found in the GitHub repository from this experiment: https://github.com/wagnerbilla/WBScore (accessed on 1 July 2023).

3.7. Accuracies Calculation

For the application of the methodology described in Section 3.2, the following values were used: ${\ell }_{inf}$ = 1, ${\ell }_{sup}$ = 20 and ${\ell }_{step}$ = 1. Noises of progressive intensities from 1 to 20% were considered, with an incremental step of 1%. It was supposed that noise levels above 20% are no longer a concrete basis for comparing with the original dataset.

For each of the datasets described in Section 3.1 and for each classifier described in Section 3.3, the following steps were applied to calculate the accuracies:

(i)

The classifier was trained on the original dataset;

(ii)

The classifier was tested on the original dataset;

(iii)

For each of the 20 datasets with different noise levels, the classifier was tested, and the accuracies were registered;

(iv)

A graph of the classifier as a function of the noise level was plotted.

The graph plots showed that the accuracy of the classifier generally decreases as the noise level increases.

The described approach is quick and direct for measuring accuracy values as a function of introduced noise. The intention was not to guarantee the absence of data overfits by the classifiers; it is a reference implementation only. One can intend to replace it completely, using any other technique found in literature, splitting the original dataset into three others (train, validation, and test), and applying noise only in the tested instances or including a cross-validation scheme during the training phase.

4. Results and Discussion

4.1. Classic Datasets

Figure 4 shows the accuracy curves of each classifier tested over the variation of the inserted noise, using the best parameters’ configurations found with the Grid Search method in all selected testing datasets, as detailed earlier in Table 3.

In all graphs, a generalized degradation of the accuracy of the classifiers is visible due to the increase in noise. The response curves of the classifiers also change concerning the dataset type because it is a data-driven process, and each dataset type generates a different “signature” by each classifier.

The classifiers that stood out in these tests were the KNN and the RF. KNN won in the IRIS, IONOSPHERE, IMAGE SEGMENTATION, and SEEDS datasets, whereas in the GLASS dataset, the RF gives the best response. They all have better overall accuracy than their competitors, almost always remaining as the highest curves in the performance graphs.

Figure 5 shows the position of each classifier in the WB Score graph space for each tested dataset.

Each graph represents a visual interpretation of all classifiers as a function of the tested dataset and the chosen evaluation metric, which, in this case, is accuracy. It is possible to perceive a greater or lesser spread of the position of the classifiers in each of the results, which may indicate the internal structure of the dataset.

We could analyze all of these graphical results, but for brevity, let us look closer at the SEEDS dataset, as the spread of the classifiers is concentrated on two regions of the graph that can be explored: Balanced Response (BR) and Strong Noiseless Response (SNLR). Remember that what is discussed here for this dataset is also valid for the others.

Table 4. Summarized results of WSB Score’s parameters for the SEEDS dataset.

Algorithm	A0	Ac	$\mathit{\rho }$	$\mathit{\theta }$	Accuracy Drop %
KNN	0.971	0.946	0.958	45.74	6.78
NB	0.933	0.875	0.904	46.82	9.40
RF	1	0.920	0.960	47.38	15.05
J48	0.985	0.884	0.936	48.10	20.49
RT	0.990	0.897	0.945	47.80	19.36
MLP	1	0.854	0.929	49.49	28.72
SVM (linear)	0.985	0.863	0.926	48.78	24.85
SVM (RBF)	0.990	0.849	0.922	49.37	27.70

shows the summarized results and

Table 5. Performance rank of the eight tested classifiers against the SEEDS dataset.

Rank #	Overall Performance ($\mathit{\rho }$)	Strong Noiseless Response ($\mathit{\rho }sin\mathit{\theta }$)	Balanced Response (${43}^{\circ }\le \mathit{\theta }\le {47}^{\circ }$)
1	RF	RF	KNN
2	KNN	MLP	NB
3	RT	RT
4	J48	SVM (RBF)
5	MLP	J48
6	SVM (linear)	SVM (linear)
7	SVM (RBF)	KNN
8	NB	NB

the classifiers’ performance rank for the SEEDS dataset.

For the SEEDS dataset, we have the ranking of the best classifiers for three possible situations: overall performance (when considering only the $\rho$ value), strong noiseless response (an environment without noise), and a balanced response. Thus, the practitioner can decide which is the best classifier for his dataset in the environment in which it should be inserted.

Finally, it is also worth noting that the KNN classifier, in addition to presenting the best accuracy, also had the lowest drop in accuracy of all classifiers tested, making it a great choice for classifying this dataset. This information can be seen in the “Accuracy drop %” column in Table 4.

4.2. Customized Flooding Dataset

Figure 6 shows the accuracy curves of each classifier tested against a custom flooding dataset using the WB Score methodology. These data came from a real production environment and were taken from the Climate Emergency Management Center (CGE) between 2015 and 2016 (https://www.cgesp.org/). Additional information on this dataset can be found in Section 2.1 of [43].

Again we have different responses for each type of classifier, showing that the internal and logical implementation of each has different approaches. The SVM (RBF), SVM (linear), and MLP classifiers practically behaved indifferently to the applied noise level, always presenting similar accuracy throughout the process. This situation did not occur in any of the five classic datasets analyzed earlier, indicating that this behavior is linked to the data contained in the dataset and how its random variables behave. The other classifiers had their outputs degraded, as expected. KNN was the winning classifier, followed by RF.

Figure 7 shows the position of each classifier in the WB Score graph space for the FLOODINGS SP dataset.

It is possible to verify that almost all classifiers are located in the Balanced Response (BR) area; only the classifiers based on decision trees RT and J48 are located in the Strong Noiseless Response Area (SNRA). At the top of the graph, we can easily see the icons corresponding to the KNN and RF algorithms.

Table 6. Summarized results of WSB Score’s parameters for the FLOODINGS SP dataset.

Algorithm	A0	Ac	$\mathit{\rho }$	$\mathit{\theta }$	Accuracy Drop %
KNN	0.975	0.958	0.967	45.49	3.74
NB	0.886	0.839	0.862	46.56	6.37
RF	0.961	0.923	0.942	46.15	6.41
J48	0.941	0.867	0.905	47.36	10.33
RT	0.927	0.864	0.896	47.00	9.49
MLP	0.827	0.822	0.825	45.17	1.55
SVM (linear)	0.717	0.717	0.717	45.00	0
SVM (RBF)	0.884	0.885	0.885	44.98	0

shows the summarized results and

Table 7. Performance rank of the eight tested classifiers against the FLOODINGS SP dataset.

Rank #	Overall Performance ($\mathit{\rho }$)	Strong Noiseless Response ($\mathit{\rho }sin\mathit{\theta }$)	Balanced Response ${43}^{\circ }\le \mathit{\theta }\le {47}^{\circ }$)
1	KNN	KNN	SVM (RBF)
2	RF	RF	SVM (linear)
3	J48	J48	MLP
4	RT	RT	KNN
5	SVM (RBF)	NB	RF
6	NB	SVM (RBF)	NB
7	MLP	MLP	RJ
8	SVM (linear)	SVM (linear)	J48

the classifiers’ performance rank for the FLOODING SP dataset.

According to the results presented in the tables above, the nonlinear classifiers KNN and RF presented the best results for the FLOODING SP dataset, with KNN being the best option. It is interesting to note that the SVM’s classifiers presented no accuracy drop in all noise levels tested, indicating that they are immune to the noise applied to this dataset.

4.3. Comparison of Classifier Selection Methods

The proposed WB Score methodology addresses the challenges of selecting robust classifiers in noisy real-world scenarios. To provide a comprehensive comparison, we will evaluate the WB Score methodology alongside other standard methods: ensemble learning, Grid Search with cross-validation, performance-based selection, algorithm selection heuristics, algorithm ranking based on statistical tests, and portfolio selection.

WB Score Methodology: The WB Score methodology stands out for explicitly addressing noise-related challenges. It focuses on classifier performance and noise robustness, which is crucial in real-world applications. Utilizing a visually intuitive graph for performance representation aids decision-making.

Ensemble Learning: Ensemble learning combines predictions to improve performance. It enhances robustness by reducing noise impact. However, it may increase computational complexity and needs explicit insights into individual classifier performance [57,58].

Grid Search with Cross-Validation: Grid Search with cross-validation systematically explores hyperparameter combinations for classifiers. It helps find optimal configurations but can be computationally expensive [59].

Performance-Based Selection: Performance-based selection trains classifiers on a subset of data and evaluates their performance on a validation set. It prioritizes the best-performing classifiers, assuming consistent noise levels between training and validation data [60].

Algorithm Selection Heuristics: Algorithm selection heuristics use dataset characteristics to guide classifier choice. Although they can be effective, they may not account for subtle noise patterns [61].

Algorithm Ranking Based on Statistical Tests: This method ranks classifiers based on their performance across multiple data splits and uses statistical tests to determine significance. However, it may not fully capture noise-related challenges [62].

Portfolio Selection: Portfolio selection treats classifier selection as an optimization problem, distributing resources based on historical classifier performance. It requires careful consideration of noise and performance history [3].

Table 8. Comparison of classifier selection methods.

Method	Pros	Cons
WB Score Methodology	Explicitly addresses noise-related challenges.	Specific details of noise robustness and efficient handling are not detailed.
	Emphasizes robustness to noise and efficient noise handling.
	Utilizes a visually intuitive graph for performance representation.
Ensemble Learning	Robustness through combination of multiple classifiers.	Increased computational complexity.
	Effective noise reduction.	Limited insight into individual classifier performance.
Grid Search with Cross-Validation	Systematic exploration of hyperparameter space.	Computationally expensive for large search spaces.
	Can find optimal configurations.
Performance-Based Selection	Prioritizes best-performing classifiers.	Assumes consistent noise levels between training and validation data.
	Evaluates classifiers on validation set.
Algorithm Selection Heuristics	Guided by dataset characteristics.	May not capture subtle noise patterns.
	Tailored to specific characteristics.
Algorithm Ranking Based on Statistical Tests	Ranks classifiers based on performance.	May not fully capture noise-related challenges.
	Uses statistical tests for significance.
Portfolio Selection	Optimizes resource allocation based on historical performance.	Requires consideration of noise and performance history.
	Addresses classifier performance over time.

summarizes the method’s main characteristics, highlighting each approach’s pros and cons.

The WB Score methodology offers a specialized approach explicitly designed to address noise-related challenges, making it a promising solution for real-world scenarios with increasing noise levels. Although other methods, such as ensemble learning, Grid Search with cross-validation, performance-based selection, algorithm selection heuristics, algorithm ranking based on statistical tests, and portfolio selection, contribute valuable techniques, their application may require careful consideration of specific noise-related concerns.

5. Conclusions

According to the type of dataset used for a classification task, there is a better algorithm for the problem to be attacked in which the bias is minimized. In the vast majority of data, noise insertions make it challenging to choose the best robust single solution for the purpose, and there needs to be more literature on this type of approach. The WB Score methodology presented here makes it possible to choose the best classifier for a given dataset among a set of options, considering its accuracy response when introducing noise levels representing a scenario where classifiers are trained once and deployed in production environments, continuously facing increasing noise levels over time. Graphical or visual methodologies are found in the literature. However, none were designed to consider the increase in noise levels nor to analyze a set of classifiers simultaneously to select the best one.

Five classic datasets and a customized one were used to show that the methodology will always present its graphical output as long as the accuracy of the tested classifiers is nonzero and the dataset is based on numerical values. Two classifiers stood out in the tests, drawing much attention: KNN and RF, both considered nonlinear classifiers.

In real-life applications, the analyzed objects are usually represented as random variables. Contaminating it with additive and multiplicative noise is a convenient process to produce scenarios with distinct variability. In this sense, generalizing the results for tens of datasets in a noisy environment involves positioning the tested classifiers in the areas defined in the WB Score output graph. This straightforward procedure enables users to select the best classifier for their specific needs.

Possible WB Score methodology application fields are medical settings where accurate and reliable classification of patient data is paramount. The ability to explicitly account for noise and ensure robustness can lead to improved diagnostic accuracy; financial forecasting, where markets are noisy and subject to unpredictable fluctuations, the methodology’s focus on noise robustness empowers financial analysts to identify classifiers better suited for handling noisy market data, leading to more accurate predictions and informed investment strategies; environmental monitoring, where the choice of the best classifier aids in building reliable models for predicting natural events, such as rainfall patterns or air quality levels, contributing to effective environmental management and disaster preparedness; and industrial settings, where data from sensors and machinery may be contaminated with noise, the WB Score methodology can optimize predictive maintenance processes by selecting robust classifiers that identify potential equipment failures and maintenance needs more accurately, minimizing downtime and maximizing operational efficiency.

This article introduced a basic reference implementation to present an overview of the WB Score methodology. This implementation can be changed according to the needs of practitioners. One can change, for example, how the training and testing phases of the classifiers are performed, as well as choose another evaluation metric.

6. Future Works

Well-known classic datasets and a custom dataset were explored here. However, there are a multitude of types with different characteristics among them. We can mention new scenarios to be studied: unbalanced datasets, noise in classes not yet seen, and the addition of between-class samples.

The types of noise introduced in this study were the traditional ones found in the literature, especially when the classifier’s response is a function of a linear combination of random variables in the dataset’s attributes. However, other types should be explored, such as the generation of convolutional noise and the reinforcement method commonly used in reinforcement learning scenarios.