Analysis of Colorectal and Gastric Cancer Classification: A Mathematical Insight Utilizing Traditional Machine Learning Classifiers

Abstract

Cancer remains a formidable global health challenge, claiming millions of lives annually. Timely and accurate cancer diagnosis is imperative. While numerous reviews have explored cancer classification using machine learning and deep learning techniques, scant literature focuses on traditional ML methods. In this manuscript, we undertake a comprehensive review of colorectal and gastric cancer detection specifically employing traditional ML classifiers. This review emphasizes the mathematical underpinnings of cancer detection, encompassing preprocessing techniques, feature extraction, machine learning classifiers, and performance assessment metrics. We provide mathematical formulations for these key components. Our analysis is limited to peer-reviewed articles published between 2017 and 2023, exclusively considering medical imaging datasets. Benchmark and publicly available imaging datasets for colorectal and gastric cancers are presented. This review synthesizes findings from 20 articles on colorectal cancer and 16 on gastric cancer, culminating in a total of 36 research articles. A significant focus is placed on mathematical formulations for commonly used preprocessing techniques, features, ML classifiers, and assessment metrics. Crucially, we introduce our optimized methodology for the detection of both colorectal and gastric cancers. Our performance metrics analysis reveals remarkable results: 100% accuracy in both cancer types, but with the lowest sensitivity recorded at 43.1% for gastric cancer.

1. Introduction

Cancer, a longstanding enigma in human history, has experienced a notable upsurge in its prevalence in recent decades due to several contributing causes. These reasons encompass the inexorable aging of populations, the embracing of detrimental lifestyles, and heightened exposure to carcinogens in the environment, food, and beverages [1,2]. The term “cancer” has its origins in the Greek word “kapkivoc”, which carries a dual meaning, referring to both a neoplasm and a crustacean of the crab genus. This nomenclature was first introduced in the medical lexicon in the 17th century and signifies a condition characterized by the invasive spread of cells to different anatomical sites, potentially causing harm [3,4,5]. In the human anatomy, composed of countless innumerable cells, cancer can emerge in diverse locations, from the extremities to the brain. While cells typically divide and multiply to meet the body’s needs and undergo programmed cell death, when necessary, deviations can lead to the uncontrolled replication of damaged or abnormal cells, resulting in the formation of a neoplasm or tumor. These tumors can be categorized as benign (non-malignant) or malignant (cancerous), with the latter having the potential to travel to distant body parts from the original location, often affecting nearby tissues along the way. Notably, blood cancers, like leukemia, do not follow the typical pattern of solid tumor formation but rather tend to involve the proliferation of abnormal blood cells that circulate within the body and may not form solid masses as seen in other types of cancer. Cancer arises from genetic anomalies that disrupt the regulation of cellular proliferation. These genetic anomalies compromise the natural control mechanisms that prevent excessive cell proliferation. The body has inherent mechanisms designed to remove cells that possess damaged DNA, but, in certain cases, these fail, allowing abnormal cells to thrive and potentially develop into tumors, disrupting regular bodily functions; these defenses can diminish with age or due to various factors [6].

Each instance of cancer exhibits a distinct genetic modification that evolves as the tumor grows. Tumors often showcase a diversity of genetic mutations across various cells existing within the same cluster. Genetic abnormalities primarily affect three types of genes: DNA repair genes, proto-oncogenes, and tumor suppressor genes. Proto-oncogenes are typically immersed in healthy cell division and proliferation. The transformation of these genes into oncogenes, brought on by specific alterations or increased activity, fuels uncontrolled cell growth and plays a role in cancer development. Meanwhile, tumor suppressor genes meticulously manage cellular division while imposing restraints on unbridled and unregulated cellular proliferation, and mutations in these genes disable their inhibitory function, increasing the risk of cancer. Mutations in DNA repair genes are significant in rectifying DNA damage, and these genes can lead to the accumulation of further genetic abnormalities, making cells more prone to developing cancer. Metastasis is the movement of cancer cells from the initial site to new parts. It includes cell detachment, local tissue invasion, blood or lymph system entry, and growth in distant tissues [7,8]. Understanding the genetic and cellular mechanisms underlying cancer development and metastasis is crucial for improving diagnostics, developing effective treatments, and advancing cancer research. Researchers can work toward better strategies for prevention, early detection, and targeted therapies by unraveling the intricacies of cancer at the molecular level. The early diagnosis of cancer developments across different body areas requires accurate and automated computerized techniques. While numerous researchers have made significant strides in cancer detection, there remains substantial scope for improvement in this field. In this manuscript, we have scrutinized colorectal and gastric cancers employing conventional ML techniques solely based on medical imaging datasets. Medical images offer finer and more specific details compared to other medical data sources.

Literature Review

This section provides an evaluative comparison of the most recent review articles available, analyzing current review articles dedicated to the utilization of machine learning and deep learning classifiers for cancer detection across diverse types. The objective is to summarize the positive aspects and limitations of these review articles, as per the review presented, on various cancer types. The papers selected for analysis include those that cover more than two cancer types, are peer-reviewed, and were published between 2019 and 2023. This present study extends our prior works [9,10] by providing an extensive review that now encompasses seven distinct cancer types. Levine et al. (2019) [9] focused on cutaneous, mammary, pulmonary, and various other malignant conditions, emphasizing radiological practices and diagnostic workflows. The study detailed the construction and deployment of a convolutional neural network for medical image analysis. However, limitations included a relative underemphasis on malignancy detection, sparse literature sources, and examination of a limited set of performance parameters. Huang et al. (2020) [10] explored prostatic, mammary, gastric, colorectal, solid, and non-solid malignancies. The study presented a comparative analysis of artificial intelligence algorithms and human pathologists in terms of prognostic and diagnostic performance across various cancer classifications. However, limitations included a lack of literature for each malignancy category, the absence of consideration for machine learning and deep learning classifiers, and a lack of an in-depth literature review. Saba (2020) [11] examined mammary, encephalic, pulmonary, hepatic, cutaneous, and leukemic cancers, offering concise explanations of benchmark datasets and a comprehensive evaluation of diverse performance metrics. However, limitations included a combined treatment of machine learning and deep learning without a separate analysis and the absence of a comparative exploration between the two methodologies. Shah et al. (2021) [12] proposed predictive systems for various cancer types but had limitations. It used a data, prediction technique, and view (DPV) framework to assess cancer detection. The focus was on data type, modality, and acquisition. However, the study included a limited number of articles for each cancer type, lacked a performance evaluation, and only considered deep learning-based methods.

Majumder and Sen (2021) [13] centered its focus on the domains of mammary, pulmonary, solid, and encephalic malignancies. The findings embraced the demonstration of artificial intelligence’s application in the domains of oncopathology and translational oncology. However, limitations included a limited consideration of cancer types and literature sources, along with variations in performance metrics across different sources. Tufail et al. (2021) [14] evaluated astrocytic, mammary, colorectal, ovarian, gastric, hepatic, thyroid, and various other cancer types, emphasizing publicly accessible datasets, cancer detection, and segmentation. However, the exclusive focus on deep learning-based cancer detection limited a comprehensive examination of each cancer type. Kumar and Alqahtani (2022) [15] examined mammary, encephalic, pulmonary, cutaneous, prostatic, and various other malignancies, detailing diverse deep learning models and architectures based on image types. However, limitations included the exclusive focus on deep learning methods and variations in performance metrics across different literature sources. Kumar et al. (2022) [3] evaluated various malignancies, offering comprehensive coverage across diverse cancer categories. The study drew from numerous literature sources, presenting a wide array of performance metrics and acknowledging challenges. However, limitations included the amalgamation of all cancer types in a single analysis and the absence of a separate assessment of machine learning and deep learning approaches. Painuli et al. (2022) [16] concentrated on mammary, pulmonary, hepatic, cutaneous, encephalic, and pancreatic malignancies. The study examined benchmark datasets for these cancer types and provided an overview of the utilization of machine learning and deep learning methodologies. The research identified the most proficient classifiers based on accuracy but unified the examination of deep learning and machine learning techniques instead of offering individual assessments.

Rai (2023) [17] conducted a comprehensive analysis of cancer detection and segmentation, utilizing both deep neural network (DNN) and conventional machine learning (CML) methods, covering seven cancer types. The review separately scrutinized the strengths and challenges of DNN and CML classifiers. Despite limitations, such as a limited number of research articles and the absence of a database and feature extraction analysis, the study provided valuable insights into cancer detection, laying the foundation for future research directions. Maurya et al. (2023) [18] assessed encephalic, cervical, mammary, cutaneous, and pulmonary cancers, providing a comprehensive analysis of the performance parameters and inherent challenges. However, it lacked an independent assessment of machine learning and deep learning techniques and a dataset description. Mokoatle et al. (2023) [19] focused on pulmonary, mammary, prostatic, and colorectal cancers, proposing novel detection methodologies utilizing SBERT and the SimCSE approach. However, limitations included the study’s focus on four cancer types, the lack of a dataset analysis, and reliance on a single assessment metric. Rai and Yoo (2023) [20] enhanced cancer diagnostics by classifying four cancer types with computational machine learning (CML) and deep neural network (DNN) methods. The study reviewed 130 pieces of literature, outlined benchmark datasets and features, and presented a comparative analysis of CML and DNN models. Limitations included a focus on four cancer types and reliance on a single metric (accuracy) for classifier validation.

This study offers an expansive and in-depth examination of the current landscape and potential prospects for diagnosing colorectal and gastric cancers through the application of traditional machine learning methodologies. The key contributions and highlights of this review can be distilled into the following key points.

• Mathematical Formulations to Augment Cognizance: Inaugurating the realm of mathematical formulations, meticulously addressing the most frequently utilized preprocessing techniques, features, machine learning classifiers, and the intricate domain of assessment metrics.
• Mathematical Deconstruction of ML Classifiers: Engaging in a profound exploration of the mathematical intricacies underpinning machine learning classifiers commonly harnessed in the arena of cancer detection.
• Colorectal and Gastric Cancer Detection: Dedicating an analytical focus to the nuanced landscape of colorectal and gastric cancer detection. Our scrutiny unfurled a detailed examination of the methodologies and techniques germane to the diagnosis and localization of these particular cancer types.
• Preprocessing Techniques and Their Formulation: Penetrating the intricate realm of preprocessing techniques and probing their pivotal role in elevating the quality and accuracy of models employed in cancer detection.
• Feature Extraction Strategies and Informative Features: Embarking on a comprehensive journey, scrutinizing the multifaceted domain of feature extraction techniques, meticulously counting and discerning the number of features wielded in research articles.
• A Multidimensional Metrics Analysis: Conducting an holistic examination encompassing a spectrum of performance evaluation metrics, encapsulating accuracy, sensitivity, specificity, precision, negative predictive value, F-measure (F1), area under the curve, and the Matthews correlation coefficient (MCC).
• Evaluation Parameters for Research Articles: Systematically analyzing diverse parameters, including publication year, preprocessing techniques, features, techniques, image count, modality nuances, dataset details, and integral metrics (%).
• Prominent Techniques and Their Effectiveness: Expertly identifying the techniques most prevalently harnessed by researchers in the realm of cancer detection and meticulously pinpointing the most effective among the gamut of options.
• Key Insights and Ongoing Challenges: Highlighting key insights from the scrutinized research papers, encompassing advances, groundbreaking revelations, and challenges in cancer detection using traditional machine learning techniques.
• Architectural Design of Proposed Methodology: Laying out in meticulous detail an architectural blueprint derived from the reviewed literature. These architectural formulations present invaluable guides for the enhancement of cancer detection models.
• Recognizing Opportunities for Improvement: Executing a methodical comparative analysis of an array of metrics, meticulously scrutinizing their zenith and nadir values, as well as the interstitial chasm. This granular evaluation aids in the strategic pinpointing of areas harboring untapped potential for enhancement in cancer detection practices.

2. Materials and Methods

2.1. Literature Selection Process

In this section, we will provide a broad overview of the procedures involved in selecting and employing research articles for the purpose of cancer classification through traditional ML approaches. These selection criteria encompass both inclusion and exclusion standards, which we will delineate in depth. The PRISMA flow diagram delineates the systematic review process employed for the detection of colorectal and stomach (gastric) cancer utilizing conventional machine learning (CML) methodologies, as illustrated in Figure 1. Commencing with an initial identification of 571 records through meticulous database searching, the subsequent removal of 188 duplicates yielded 383 distinct records. Through a rigorous screening process, 197 records were deemed ineligible, prompting a detailed assessment of eligibility for 186 full-text articles. Within this subset, the exclusion of 150 articles on various grounds culminated in the inclusion of 36 studies. This select group of 36 studies served as the foundational basis for the scoping review, offering a comprehensive exploration of cancer detection methods employing CML approaches for both colorectal and stomach cancers.

2.1.1. Inclusion Criteria

The inclusion criteria for the review of research articles focused on cancer detection were defined across several specific parameters. Firstly, the articles had to pertain exclusively to the classification of cancer using conventional machine learning classifiers. These articles were specifically chosen if they were peer-reviewed and published between 2017 and 2023. The selection was limited to journal articles, omitting conference papers, book chapters, and similar sources to maintain the analytical scope. The studies selected for review utilized medical image datasets related to colorectal and gastric cancers. Additionally, a key criterion was the inclusion of accuracy as a performance metric in the chosen articles. Accuracy stands as a fundamental measure in evaluating the effectiveness of cancer detection models. The selected studies also strictly employed traditional machine learning classifiers for their classification tasks. The review was narrowed down to studies covering two specific high-mortality cancer types: colorectal and gastric cancer. Furthermore, articles were required to be in the English language, a criterion implemented to ensure the enhanced accessibility and comprehension of the research, thereby contributing to clarity and accuracy in the assessment process. Figure 2 illustrates the parameters governing the inclusion and exclusion of research articles in the selection process employed in this manuscript.

2.1.2. Exclusion Criteria

The exclusion criteria, a pivotal aspect of the research review process for cancer detection, served as a strategic filter to ensure the selection of high-quality, pertinent articles. Omitting conference papers and book chapters was a deliberate choice to uphold a superior standard, guided by the in-depth scrutiny and comprehensive nature typically associated with peer-reviewed journal articles. Additionally, the requirement for digital object identifiers (DOIs) within the selected studies aimed to guarantee the reliability and accessibility of the articles, facilitating easy citation, retrieval, and verification processes. The temporal boundary set the scope within a specific timeframe, excluding research published before 2017 or after 2023, with the intention of focusing on the most recent advancements within the field of cancer detection. Language limitations were incorporated, allowing only English publications to ensure a consistent understanding and analysis. Moreover, the exclusion of deep learning classifiers in favor of traditional machine learning methods aligned with the specific objective of assessing the performance and effectiveness of the latter in cancer detection. By narrowing the focus exclusively to colorectal and gastric cancers, the exclusion criteria aimed to ensure a concentrated and comprehensive analysis across these specific high-mortality cancer types. This approach facilitated a deeper understanding of the efficacy of traditional machine learning methods in the context of different cancer types.

To illuminate the research hotspots, we have detailed the quantity of literature references pertaining annually to each cancer category (colorectal and gastric), along with the cumulative total, visually represented in Figure 3. This visual aid is designed to aid readers in identifying pertinent literature related to these specific cancer categories, fostering a more nuanced analysis within the specified years.

2.2. Medical Imaging Datasets

Data collection is the essential first step in any machine learning endeavor, and the performance of classifiers and detection tasks depends on the characteristics of the datasets used. The approach for identifying or classifying diseases, particularly cancers, is closely linked to the nature of the dataset. Various data types, such as images, text, and signal data, may require distinct processing methods. In the context of cancer detection, medical image datasets are of paramount importance. These datasets contain images that provide valuable information about the presence and characteristics of cancerous tissues. Specialized techniques, including image segmentation and feature extraction, are applied to extract relevant information for classification or detection. Analyzing image datasets differs significantly from text or signal datasets due to differences in data structures and feature extraction techniques. Dataset availability can be categorized as offline or real-time. In the domain of cancer detection, most research relies on offline datasets sourced from healthcare institutions, research centers, and platforms like Kaggle and Mendeley. Researchers often use local datasets from these sources to conduct studies and develop innovative cancer detection methods. In

Table 1. Benchmark and public medical imaging datasets for colorectal and gastric cancer with download links.

Dataset	Cancer Category	Modality	Downloadable Link	No. of Data Samples	Pixel Size
NCT-CRC-HE-100K	Colorectal	H&E	https://zenodo.org/record/1214456 (accessed on 15 September 2023)	100,000	224 × 224
Lung and colon histopathological images (LC25000)		H&E	https://academictorrents.com/details/7a638ed187a6180fd6e464b3666a6ea0499af4af (accessed on 15 September 2023)	10,000	768 × 768
CRC-VAL-HE-7K		H&E	https://zenodo.org/record/1214456 (accessed on 15 September 2023)	7180	224 × 224
Kather-CRC-2016 (KCRC-16)		H&E	https://zenodo.org/record/53169#.W6HwwP4zbOQ (accessed on 15 September 2023)	5000 10	150 × 150 5000 × 5000
Kvasir V-2 dataset (KV2D)	Stomach (Gastric)	Endoscopy	https://dl.acm.org/do/10.1145/3193289/full/ (accessed on 15 September 2023)	4000	720 × 576 to 1920 × 1072
HyperKvasir dataset (HKD)		Endoscopy	https://osf.io/mh9sj/ (accessed on 15 September 2023)	110,079 images and 374 videos	----
Gastric histopathology sub-size image database (GasHisSDB)		H&E	https://gitee.com/neuhwm/GasHisSDB	245,196	160 × 160, 120 × 120, 80 × 80

, we have described some benchmarked imaging datasets of lung and colorectal cancers.

2.3. Preprocessing

In cancer detection, preprocessing is essential to prepare data for analysis and classification. It refines diverse data types, like medical images and genetic and clinical data, addressing noise and inconsistencies. Medical image preprocessing includes noise reduction, enhancement, normalization, and format standardization. Augmentation enhances data diversity. Quality preprocessed data improves cancer detection model performance. Common tasks include noise reduction, data cleaning, transformation, normalization, and standardization. Preprocessing optimizes data for analysis, contributing to effective cancer diagnosis. Key preprocessing techniques are summarized in

Table 2. Fundamental preprocessing techniques, associated formulas, and detailed descriptions.

Preprocessing Technique	Formula	Description
Image Filtering	$I_{\mathrm{filtered}}\left(A,B\right)={\displaystyle \sum _{x=-N}^N}{\displaystyle \sum _{y=-N}^N}I\left(A-x,B-y\right)\cdot K\left(x,y\right)$	$I_{\mathrm{filtered}}\left(A,B\right)$ epitomizes the clean image pixel at location $\left(A,B\right).$ $I\left(A-x,B-y\right)$ is the pixel significance at location $\left(A-x,B-y\right)$ in the original image. $K\left(x,y\right)$ is the value of the convolution kernel at location $\left(x,y\right).$ The summation is performed over a window of size $\left(2N+1\right)\times \left(2N+1\right)$ centered at $\left(A,B\right)$.
Image Denoising	$I_{\mathrm{denoised}}=\arg \min \left(E\left(I_{\mathrm{denoised}}\right)+R\left(I_{\mathrm{denoised}}\right)\right)$	$I_{\mathrm{denoised}}$ represents the denoised image. $E\left(I_{\mathrm{denoised}}\right)$ is the data fidelity term, which measures how well the denoised image matches the noisy input image. $R\left(I_{\mathrm{denoised}}\right)$ is the regularization term, which imposes a prior on the structure of the denoised image [21].
Gaussian Filtering	$Filtere{d}_{value}=\frac{1}{\left(2\pi {\sigma }^2\right)}\ast e^{\frac{-(x^2+y^2)}{2{\sigma }^2}}$	$Filtere{d}_{value}$ represents the resulting value after applying Gaussian filtering. $x$ and $\mathrm{ }y$ are the spatial coordinates. $\mathsf{\sigma }$ is the standard deviation, controlling the amount of smoothing or blurring.
Contrast Enhancement of Images (CEI)	${Pixel}_{OP}=\frac{{Pixel}_{IP}-{Min}_{IP}}{{(Max}_{IP}-{Min}_{IP})}\ast \left({Max}_{OP}-{Min}_{OP}\right)+{Min}_{OP}$	$Pixe{l}_{OP}$ is the enhanced pixel value, derived from $Pixe{l}_{IP}$ in the input image. $Mi{n}_{IP}$ and $Ma{x}_{IP}$ are the minimum and maximum pixel values in the input image. $Mi{n}_{OP}$ and $Ma{x}_{OP}$ represent the desired minimum and maximum pixel values in the output image [22].
Linear Transformation	$T\left(v\right)=Av$	where $T$ is the transformation operator, $v$ is the input vector, and $A$ is a matrix defining the transformation.
Contrast Limited Adaptive Histogram Equalization (CLAHE)	$O\left(A,B\right)=T\left(I\left(A,B\right)\right)$	$O\left(A,B\right)$ is the enhanced output pixel at $\left(A,B\right)$ using contrast-enhancing transformation function $T\left(\cdot \right)$ based on pixel intensity using cumulative distribution function (CDF).
Discrete Cosine Transform (DCT)	$X\left[m\right]={\displaystyle \sum _{k=0}^{N-1}}x\left[k\right]\cdot \cos \left(\frac{\pi \left(2k+1\right)m}{2N}\right)$	$X\left[m\right]$ represents the DCT coefficient at frequency index $m$. $x\left[k\right]$ is the input signal. $N$ is the number of samples in the signal. The summation is performed over all samples in the signal
Wavelet Transform (WT)	$W\left(x,y\right)={\displaystyle \sum _{a=0}^{N-1}}{\displaystyle \sum _{b=0}^{M-1}}I\left(a,b\right)\cdot {\psi }_{x,y}\left(a,b\right)$	$W\left(x,y\right)$ is the DWT coefficient, $\left(I\left(a,b\right)\right)$ is the pixel value at $\left(a,b\right)$, and ${\mathsf{\psi }}_{x,y}\left(a,b\right)$ is the 2D wavelet function.
RGB to Gray Conversion (RGBG)	$\mathit{\text{Gray\_value}}=(0.2989\ast {Red}_{value})+(0.5870\ast {Green}_{value})+(0.1140\ast {Blue}_{value})$	$Gra{y}_{value}$ is the converted gray value from RGB channels ($Re{d}_{value},Gree{n}_{value},Blu{e}_{value}$). Coefficients 0.2989, 0.5870, and 0.1140 are weights assigned to the R, G, and B channels, respectively [23].
Cropping (ROI)	$I_{\mathrm{cropped}}=I\left[y:y+h,x:x+w\right]$	The cropped image $I_{\mathrm{cropped}}$ is obtained by cropping the input image $I$ at coordinates $\left(x,y\right)$ with width $w$ and height $h$.

.

2.4. Feature Engineering

Feature engineering is a critical component in solving classification problems, particularly with traditional machine learning methods. Features represent dataset attributes used by the model to classify or predict. Instead of using the entire dataset, relevant features are extracted and serve as classifier inputs, delivering the desired outcomes. Proper preprocessing is essential before feature engineering to ensure data quality. Feature engineering involves selecting which features to extract, choosing methods, defining the domain, and specifying the number of features. Categories of feature engineering include extraction, selection, reduction, fusion, and enhancement. Commonly used features for predicting lung and colorectal cancers in medical images are outlined below.

2.4.1. Histogram-Based First-Order Features (FOFs)

These are statistical features extracted from an image’s histogram, providing valuable information about the distribution and characteristics of pixel intensities [24]. Here are some significant FOFs, along with their mathematical formulae presented in Equations (1)–(4).

Skewness (s): Skewness quantifies the asymmetry of the histogram and is calculated as:

$$s=\frac{1}{{\sigma }^3}\sum _{i=1}^{G_{max}}\{{(i-\mu )}^3\ast h_i\}$$

(1)

Here, $i$ is the gray level, $h_i$ is its frequency, $G_{\max }$ is the highest grayscale intensity, and $\mathsf{\mu }$ and ${\mathsf{\sigma }}^2$ are the mean and variance, respectively.

Excess Kurtosis (k): Excess kurtosis measures the peakedness of the histogram and is calculated as:

$$k=\frac{1}{{\sigma }^4}\sum _{i=1}^{G_{max}}\left\{{\left(i-\mu \right)}^4\ast h_i\right\}-3$$

(2)

Energy: Energy reflects the overall intensity in the image and is computed as:

$$Energy=\sum _{i=1}^{G_{max}}\left\{{\left[h_i\right]}^2\right\}$$

(3)

Entropy (HIST): Entropy quantifies the information or randomness in the histogram and is calculated as:

$$Entropy=\sum _{i=1}^{G_{max}}\{h_i\ast \ln \left(h_i\right)\}$$

(4)

2.4.2. Gray-Level Co-Occurrence Matrix (GLCM) Features

GLCM is a technique used for texture analysis in image processing. It assesses the association between pixel values in an image, relying on the likelihood of specific pixel pairs with particular gray levels occurring within a defined spatial proximity [25,26,27]. Here are some important GLCM features, along with their mathematical formulas as provided in Equations (5)–(10).

Here, (x, y) pairs typically refer to the intensity values of adjacent or neighboring pixels.

Sum of Squares Variance (SSV): SSV quantifies the variance in gray levels within the texture.

$$\mathrm{S}\mathrm{S}\mathrm{V}=\sum _{x,y}{\left(x-\mathsf{\mu }\right)}^2\ast GLCM\left(x,y\right)$$

(5)

Inverse Different Moment (IDM): IDM measures the local homogeneity and is higher for textures with similar gray levels.

$$IDM=\sum _{x,y}\frac{1}{1+{(x-y)}^2}\ast GLCM(x,y)$$

(6)

Correlation (Corr): Correlation quantifies the linear dependency between pixel values in the texture. It spans from −1 to 1, with 1 signifying flawless positive correlation.

$$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}=\frac{\sum _{x,y}(x\ast y \ast GLCM(x,y)\left)\right({\mu }_a\ast {\mu }_b)}{{\sigma }_a\ast {\sigma }_b}$$

(7)

Dissimilarity: Dissimilarity quantifies how different neighboring pixel values are.

$$Dissimilarity=\sum _{x,y}\left|(x-y)\right|\ast GLCM(x,y)$$

(8)

Autocorrelation (AuCorr): Autocorrelation measures the similarity between pixel values at different locations in the texture.

$$AuCorr=\sum _{x,y}x\ast y\ast GLCM(x,y)$$

(9)

Inverse Difference (ID): ID measures the local homogeneity and is higher for textures with similar gray levels at different positions.

$$ID=\sum _{x,y}\frac{GLCM(x,y)}{1+\left|(x-y)\right|}$$

(10)

2.4.3. Gray-Level Run Length Matrix (GLRLM)

This is a statistical procedure employed in image processing and texture assessment to quantify the distribution of run lengths of specific gray levels within an image. Here are some significant GLRLM features along with their corresponding mathematical formulas, as presented in Equations (11)–(22).

Short Run Emphasis (SRE): SRE evaluates the dispersion of shorter runs characterized by lower gray-level values.

$$SRE=\sum _{x,y}\frac{C(x,y)}{x^2}$$

(11)

Here, $(x, y)$ are gray levels, and $C\left(x,y\right)$ is the co-occurrence matrix value reflecting the frequency of each gray-level combination.

Long Run Emphasis (LRE): LRE assesses the presence of extended runs marked by higher gray-level values [28].

$$LRE=\sum _{x,y}C\left(x,y\right)\ast x^2$$

(12)

Gray Level Nonuniformity (GLN): GLN Quantifies the nonuniformity of gray-level values in runs.

$$GLN=\sum _{x,y}{C\left(x,y\right)}^2$$

(13)

Run Length Nonuniformity (RLN): RLN evaluates the irregularity in the lengths of runs.

$$RLN=\sum _{x,y}\frac{C\left(x,y\right)}{y^2}$$

(14)

Run Percentage (RP): RP represents the percentage of runs in the matrix.

$$RP=\sum _{x,y}\frac{C\left(x,y\right)}{N^2}$$

(15)

Run Entropy (RE): RE calculates the entropy of run lengths and gray levels.

$$RE=-\sum _{x,y}(C\left(x,y\right)\ast \log C\left(x,y\right) +\in )$$

(16)

Low Gray-Level Run Emphasis (LGRE): LGRE accentuates shorter runs with lower gray-level values.

$$LRGE=\sum _{x,y}\frac{C\left(x,y\right)}{y^2},\hspace{1em}\hspace{1em}\mathit{for}y\le \frac{N+1}{2}$$

(17)

High Gray-Level Run Emphasis (HGRE): HGRE highlights longer runs with higher gray-level values.

$$HRGE=\sum _{x,y}C\left(x,y\right)\ast y^2,\hspace{1em}\hspace{1em}\mathit{for}y>\frac{N+1}{2}$$

(18)

Short Run Low Gray-Level Emphasis (SRLGLE): SRLGLE highlights shorter runs that contain lower gray-level values.

$$SRLGLE=\sum _{x,y}\frac{C\left(x,y\right)}{{(x}^2\ast y^2)},\hspace{1em}\hspace{1em}\mathit{for}x,y\le \frac{N+1}{2}$$

(19)

Short Run High Gray-Level Emphasis (SRHGLE): SRHGLE highlights shorter runs that contain higher gray-level values.

$$SRHGLE=\sum _{x,y}\frac{C\left(x,y\right)\ast x^2}{y^2},\hspace{1em}\hspace{1em}\mathit{for}x,y\le \frac{N+1}{2}, y>\frac{N+1}{2}$$

(20)

Long Run Low Gray-Level Emphasis (LRLGLE): LRLGLE emphasizes longer runs featuring lower gray-level values.

$$LRLGLE=\sum _{x,y}\frac{\frac{C\left(x,y\right)}{x^2}}{y^2},\hspace{1em}\hspace{1em}\mathit{for}x>\frac{N+1}{2}, y\le \frac{N+1}{2}$$

(21)

Long Run High Gray-Level Emphasis (LRHGLE): LRHGLE highlights extended sequences with higher gray-level values.

$$LRHRGLE=\sum _{x,y}C\left(x,y\right)\ast {x^2\ast y}^2,\hspace{1em}\hspace{1em}\mathit{for}x,y>\frac{N+1}{2}$$

(22)

2.4.4. Neighborhood Gray-Tone Difference Matrix (NGTDM)

This is another texture analysis method used in image processing to characterize the spatial arrangement of gray tones in an image. Here are some key NGTDM features along with their respective mathematical formulas, as outlined in Equations (23)–(27).

Coarseness: Measures the coarseness of the texture based on differences in gray tones.

$$Coars={\sum }_{x=1}^{N_g}\frac{C(x,y)}{{\left(\Delta x\right)}^2}$$

(23)

Ng refers to the highest achievable discrete intensity level within the image.

Contrast (NGTD): Quantifies the contrast or sharpness in the texture.

$${Contrast}_{NGTD}={\sum }_{x=1}^{N_g}{\sum }_{y=1}^{N_g}C\left(x,y\right)\ast \left|x-y\right|$$

(24)

Busyness: Represents the level of activity or complexity in the texture.

$$Busyness={\sum }_{x=1}^{N_g}{\sum }_{y=1}^{N_g}C\left(x,y\right)\ast y$$

(25)

Complexity: Measures the complexity or intricacy of the texture.

$$\mathrm{C}\mathrm{o}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{x}\mathrm{i}\mathrm{t}\mathrm{y}={\sum }_{x=1}^{N_g}{\sum }_{y=1}^{N_g}\frac{P\left(x,y\right)}{1+{\left|x-y\right|}^2}$$

(26)

Texture Strength (TS): Quantifies the strength or intensity of the texture.

$$\mathrm{T}\mathrm{S}=\sqrt{{\sum }_{x=1}^{N_g}{\sum }_{y=1}^{N_g}P\left(x,y\right)\ast {\left(\frac{x}{N_g}-\frac{y}{N_g}\right)}^2}$$

(27)

These features provide a detailed analysis of texture patterns in images, making them valuable for various applications, including image classification, quality control, and texture discrimination in fields such as geology, material science, and medical imaging.

2.5. Traditional Machine Learning Classifiers

Machine learning-based classifiers, renowned for their advanced capabilities in detecting cancer, notably stand out in their effectiveness when harmonized with non-invasive diagnostic techniques, providing a significant edge in the domain of cancer detection. Researchers have employed a range of ML classifiers to identify different malignancies and disorders. Some commonly used classifiers include:

2.5.1. K-Nearest Neighbors (KNN)

K-Nearest Neighbors (KNN) is a widely used and simple machine learning algorithm, suitable for classification and regression tasks. It relies on the assumption that similar inputs lead to similar outputs, assigning a class label to a test input based on the prevalent class among its k closest neighbors. The formal definition involves representing a test point ‘x’ and determining its set of ‘k’ nearest neighbors, denoted as ‘Nx’, where ‘k’ is a user-defined parameter.

The Minkowski distance is a flexible distance metric that can be tailored by adjusting the value of the parameter ‘p.’ The Minkowski distance between two data points ‘x’ and ‘z’ in a ‘d’-dimensional space is defined by Equation (28):

$$dist\left(x,z\right)={\left(\sum _{r=1}^d{\left|x_r-z_r\right|}^p\right)}^{1/p}$$

(28)

The “1-NN Convergence Proof” states that, as your dataset grows infinitely large, the 1-Nearest Neighbor (1-NN) classifier’s error will not be more than twice the error of the Bayes optimal classifier, which represents the best possible classification performance. This also holds for k-NN with larger values of k. It highlights the ability of the K-Nearest Neighbors algorithm to approach optimal performance with increasing data [29]. As $n$ approaches infinity, $Z_{NN}$ converges to $Z_t,$ and the probability of different labels for $Z_t$ when returning $\left(Z_{NN}\right)$’s label is described in Equation (29) [30].

$$\begin{gathered} {\in }_{NN}=P(y\ast |Z_t\left)\right(1-P(y\ast |Z_{NN}\left)\right)+P(y\ast |Z_{NN}\left)\right(1-P(y\ast |Z_t\left)\right)\le (1-P(y\ast \left|Z_{NN}\right))+(1-P(y\ast |Z_t\left)\right) \\ =2(1-P(y\ast \left|Z_t\right))=2{\in }_{BO} \end{gathered}$$

(29)

Here, BO is the Bayes optimal classifier. If the test point and its nearest neighbor are indistinguishable, misclassification occurs if they have different labels. This probability is outlined in Equation (30) and Figure 4 [29,31].

$$\left(1-p\left(s|x\right)\right)p\left(s|x\right)+p\left(s|x\right)\left(1-p\left(s|x\right)\right)=2p\left(s|x\right)\left(1-p\left(s|x\right)\right)$$

(30)

Equation (30) represents the misclassification probability when the test point and its nearest neighbor have differing labels.

2.5.2. Multilayered Perceptron (MLP)

In contrast to static kernels, neural network units have adaptable internal parameters for an adjustable structure. A perceptron, inspired by biological neurons, comprises three components: (i) weighted edges for individual multiplications, (ii) a summation unit for calculating the sum, and (iii) an activation unit applying a non-linear function [32,33,34]. The single-layer unit function involves a linear combination passed through a non-linear activation, represented by Equation (31) and Figure 5 [33,34].

$$y^{\left(1\right)}f=\left({w_0}^{\left(1\right)}+\sum _{j=1}^N{w_j}^{\left(1\right)}{x}_j\right)$$

(31)

In a single-layer neural network unit, $y^{\left(1\right)}f$ is the output, $w_0^{\left(1\right)}$ is the bias, and ${\sum }_{j=1}^N{w_j}^{\left(1\right)}{x}_j$ is the weighted sum of inputs. In general, we compute $U_1$ units as feature transformations in learning models, described in an Equation (32) [33,34].

$$\mathrm{model}\left(x,w\right)=w_0+y_1^{\left(1\right)}\left(x\right)w_1+\cdots +y_{U_1}^{\left(1\right)}\left(x\right)w_{U_1}$$

(32)

The input vector x can be denoted as represented in Equation (33) [33,34].

$$x=\left[\begin{array}{c}1\\ x_1\\ .\\ .\\ .\\ x_N\end{array}\right]$$

(33)

The vector representation comprises input values $x_1$ to $x_N,$ and an additional element of 1. Internal parameters of single-layer units include bias $w_{0,j}^{\left(1\right)}$ and weights $w_{1,j}^{\left(1\right)}$ through $w_{N,j}^{\left(1\right)}$. These parameters form the $jth$ column of a matrix $W^{\left(1\right)}$ with dimensions $\left(N+1\right)\times U_1$, as demonstrated in Equation (34) below [34]:

$$W_1=\left[\begin{array}{cccc}{w}_{\mathrm{0,1}}^{\left(1\right)}& w_{\mathrm{0,2}}^{\left(1\right)}& \cdots & w_{0,U_1}^{\left(1\right)}\\ w_{\mathrm{1,1}}^{\left(1\right)}& w_{\mathrm{1,2}}^{\left(1\right)}& \cdots & w_{1,U_1}^{\left(1\right)}\\ ⋮& ⋮& ⋮& ⋮\\ w_{N,1}^{\left(1\right)}& w_{N,2}^{\left(1\right)}& \cdots & w_{N,U_1}^{\left(1\right)}\end{array}\right]$$

(34)

Notably, the matrix–vector product $W_1^{T}x$ encompasses all linear combinations within our $U_1$ units as given in Equation (35) [33].

$${\left(W_1^{T}x\right)}_j=w_{0,j}^{\left(1\right)}+\sum _{n=1}^N{w}_{n,j}^{\left(1\right)}{x}_n,\hspace{1em}j=1,\dots ,U_1$$

(35)

We extend the activation function $f$ to handle a general $d\times 1$ vector $v$ in Equation (36) [34]:

$$f(v)=\left[\begin{array}{c}f\left(v_1\right)\\ ⋮\\ f\left(v_d\right)\end{array}\right]$$

(36)

In Equation (37), $f\left(W_1^{T}x\right)$ is a $U_1\times 1$ vector containing all $U_1$ single-layer units [33,34]:

$$f{\left(W_1^{T}x\right)}_j=f\left(w_{0,j}^{\left(1\right)}+\sum _{n=1}^N{w}_{n,j}^{\left(1\right)}{x}_n\right),\hspace{1em}j=1,\dots ,U_1$$

(37)

The mathematical expression for an $L$-layer unit in a general multilayer perceptron, built recursively from single-layer units, is given by Equation (38) [33,34].

$$y^{\left(L\right)}\left(x\right)=f\left(w_0^{\left(L\right)}+\sum _{i=1}^{U^{\left(L-1\right)}}{w}_i^{\left(L\right)}{f}_i^{\left(L-1\right)}\left(x\right)\right)$$

(38)

2.5.3. Support Vector Machine (SVM)

SVMs, employed for regression and classification tasks, stand out in supervised machine learning for their precision with complex datasets. Particularly effective in binary classification, SVMs aim to discover an optimal hyperplane, maximizing the boundary between classes. Serving as a linear classifier, SVMs build on the perceptron introduced by Rosenblatt in 1958 [35,36,37]. Unlike perceptrons, SVMs identify the hyperplane (H) with the maximum separation margin, defined in Equation (39).

$$h\left(x\right)=\mathrm{sign}\left(w^{Tx}+b\right)$$

(39)

The SVM classifies in $\{+1,-1\}$, emphasizing the key concept of finding a hyperplane with maximum margin $\sigma$. Figure 6 illustrates this importance, with the margin expressed in Equation (40) [35]

$$\sigma =\underset{(x_j,y_j)\mathit{ϵD}}{\mathit{min}}\left|w.x_j\right|$$

(40)

where input vectors $x_j$ are within the unit sphere, $\sigma$ is the closest data point from the hyperplane, and the vector $w$ resides on the unit sphere.

Max Margin Classifier: We formulate our pursuit of the maximizing-margin hyperplane as a constrained optimization task, aiming to enhance the margin while ensuring correct classification of all data points. This is expressed in Equation (41) [35,37]:

$$[\underset{\mathrm{maximize} \mathrm{margin}}{\underbrace{\underset{u,\mathsf{\delta }}{\max }\mathsf{\sigma }\left(u,\mathsf{\delta }\right)}}\hspace{1em}\mathrm{such} \mathrm{that}\hspace{1em}\underset{\mathrm{separating} \mathrm{hyperplane}}{\underbrace{\forall i\hspace{0.17em}{y}_i\left(u^T{x}_i+\mathsf{\delta }\right)\ge 0}}]$$

(41)

Upon substituting the definition of σ, Equation (42) is derived, as given below.

$$[\underset{\mathit{maximize}\mathit{margin}}{\underbrace{\underset{\mathit{u},\delta }{\max }\underset{\sigma (u,\delta )}{\underbrace{\frac{1}{{‖u‖}_2}\underset{{\mathit{x}}_{\mathit{i}}\in D}{\min }\left|u^T{x}_i+\delta \right|}}}}\hspace{1em}\mathrm{s}.\mathrm{t}.\hspace{1em}\underset{\mathrm{separating} \mathrm{hyperplane}}{\underbrace{\forall i\hspace{0.17em}{y}_i\left(u^T{x}_i+\mathsf{\delta }\right)\ge 0}}]$$

(42)

Scaling invariance enables flexible adjustment of $u$ and $\mathsf{\delta }$. Smart value selection ensures $(\underset{x\in D}{\min }\left|u^{T}x+\mathsf{\delta }\right|=1),$ introduced as an equality constraint in the objective per Equation (43) [37]:

$$[\underset{u,\mathsf{\delta }}{\max }1\cdot |u{|}_2=\underset{u,\mathsf{\delta }}{\min }|u{|}_2=\underset{u,\mathsf{\delta }}{\min }{u}^{\mathrm{\top }}\mathit{u}]$$

(43)

Utilizing the fact that $f\left(z\right)=z^2$ is monotonically increasing for $z\ge 0$ and $|u|\ge 0$, where $u$ maximizing $|u{|}_2$ also maximizes $u^{\mathrm{\top }\hspace{0.25em}}\mathbf{u}.$ This reformulates the optimization problem in Equation (44), and a structural diagram of a multi-SVM has been visualized in Figure 7.

$$\underset{u,\mathsf{\delta }}{\min }{u}^{\mathrm{\top }}\mathbf{u}\hspace{1em}\mathrm{subject} \mathrm{to}\hspace{1em}\forall i,y_i\left(u^T{x}_i+\mathsf{\delta }\right)\ge 0, \underset{i}{\min }\left|u^T{x}_i+\mathsf{\delta }\right|=1$$

(44)

2.5.4. Bayes and Naive Bayes (NB) Classifier

The Bayes classifier, an ideal algorithm, assigns class labels based on class probabilities given observed features and prior knowledge. It predicts the class with the highest estimated probability, often used as a benchmark but requiring complete knowledge of underlying probability distributions. To estimate $P\left(y|\overline{x}\right)$ for the Bayes classifier, the common approach is maximum likelihood estimation (MLE), especially for the discrete variable y, as outlined in Equation (45) [37]:

$$P\left(y|\overline{x}\right)=\frac{{\sum }_{k=1}^{m}I\left(\overline{x_k}=\overline{x} \wedge \overline{y_i}=y\right)}{{\sum }_{i=1}^{n}I\left(\overline{x_i}=\overline{x}\right)}$$

(45)

Naive Bayes addresses MLE’s limitations with sparse data by assuming feature independence. It estimates $P\left(y\right)$ and $P\left(\overline{x}|y\right)$ instead of $P\left(y|\overline{x}\right)$ using Bayes’ rule (Equation (46)) [37]:

$$P\left(y|\overline{x}\right)=\frac{P\left(\overline{x}|y\right)P\left(y\right)}{P\left(\overline{x}\right)}$$

(46)

Generative learning estimates $P\left(y\right)$ and $P\left(\overline{x}|y\right)$, with $P\left(y\right)$ resembling tallying occurrences for discrete binary values (Equation (47)).

$$P\left(y=c\right)=\frac{{\sum }_{i=1}^{n}I\left(y_i=c\right)}{n}={\mathsf{\pi }}^c$$

(47)

To simplify estimation, the Naive Bayes (NB) assumption is introduced, a key element of the NB classifier. It assumes feature independence given the class label, formalized in Equation (48) for $P\left(\overline{x}|y\right).$

$$P\left(\overline{x}|y\right)=\prod _{\mathsf{\alpha }=1}^{d}P\left(x_{\mathsf{\alpha }}|y\right)$$

(48)

Here, $x_{\mathsf{\alpha }}$ is the value of feature $\mathsf{\alpha }$, assuming feature values, given class label y, are entirely independent. Despite potential complex relationships, NB classifiers are effective. The Bayes classifier, defined in Equation (49), further simplifies to (50) due to $P\left(\overline{x}\right)$ independence from y, and using logarithmic property, it can be expressed as (51).

$$h\left(\overline{x}\right)={\mathrm{argmax}}_{y}P\left(y|\overline{x}\right)={\mathrm{argmax}}_{y}P\left(\overline{x}|y\right)P\left(y\right)$$

(49)

$$h\left(\overline{x}\right)={\mathrm{argmax}}_y\prod _{\mathsf{\alpha }=1}^{n}P\left(x_{\mathsf{\alpha }}|y\right)P\left(y\right)$$

(50)

$$h\left(\overline{x}\right)={\mathrm{argmax}}_y\sum _{\mathsf{\alpha }=1}^n\log \left(P\left(x_{\mathsf{\alpha }}|y\right)\right)+\log \left(P\left(y\right)\right)$$

(51)

Estimating $\log \left(P\left(x_{\mathsf{\alpha }}|y\right)\right)$ is straightforward for one dimension. $P\left(y\right)$ remains unaffected and is calculated independently. In Gaussian NB, where features are continuous $\left(x_{\mathsf{\alpha }}\in R\right),P\left(x_{\mathsf{\alpha }}|y\right)$ follows a Gaussian distribution (Equation (52)). This assumes each feature $\left(x_{\mathsf{\alpha }}\right)$ follows a class-conditional Gaussian distribution with mean ${\mathsf{\mu }}_{\mathsf{\alpha }c}$ and variance ${\mathsf{\sigma }}_{\mathsf{\alpha }c}^2$ (Equations (53) and (54)), using parameter estimates in the Gaussian NB classifier for each class [37].

$$P\left(x_{\mathsf{\alpha }}|y=d\right)=\frac{1}{\sqrt{2\mathsf{\pi }{\mathsf{\sigma }}_{\mathsf{\alpha }c}^2}}\exp \left(-\frac{{\left(x_{\mathsf{\alpha }}-{\mathsf{\mu }}_{\mathsf{\alpha }c}\right)}^2}{2{\mathsf{\sigma }}_{\mathsf{\alpha }c}^2}\right)$$

(52)

$${\mathsf{\mu }}_{\mathsf{\alpha }c}=\frac{1}{n_d}\sum _{k=1}^{n}I\left(y_k=d\right)x_{i\mathsf{\alpha }}$$

(53)

$${\mathsf{\sigma }}_{\mathsf{\alpha }c}^2=\frac{1}{n_d}\sum _{k=1}^{n}I\left(y_k=d\right){\left(x_{i\mathsf{\alpha }}-{\mathsf{\mu }}_{\mathsf{\alpha }c}\right)}^2$$

(54)

2.5.5. Logistic Regression (LR)

Logistic regression, commonly used in classification, calculates the probability of a binary label based on input features. In logistic regression (LR), the logistic (sigmoid) function transforms a linear combination of input features $x$, weights $w$, and a bias term $b$ into a likelihood estimate between 0 and 1. Mathematically, logistic regression is defined in Equation (55) [38]:

$$P\left(y=1|x\right)=\frac{1}{1+e^{-\left(w^{T}x+b\right)}}$$

(55)

Equation (71): $P\left(y=1|x\right))$ is the likelihood of class 1 given features $x$. $w$ and $b$ are estimated using statistical methods, minimizing assumptions about $P\left(x_i|y\right)$, allowing flexibility in underlying distributions [38].

The Maximum Likelihood Estimate (MLE): MLE maximizes $P\left(y\mathrm{ }|\mathrm{ }x,w\right)$, the probability of observing $y\in R^n$ given feature values $x_i$. It aims to find parameters maximizing this function, assuming independence among $y_i$ given $x_i\mathrm{ }$ and $\mathrm{ }w$. Equation (56) captures the mathematical expression for the conditional data likelihood.

$$P\left(y|x,w\right)=\prod _{k=1}^{m}P\left(y_k|x_k,w\right)$$

(56)

Now, by taking the logarithm of the product of Equation (57), we obtain Equation (73):

$$log\left(\prod _{k=1}^{m}P\left(y_k|x_k,w\right)\right)=-\sum _{k=1}^{m}log\left(1+e^{-y_k{w}^T{x}_k}\right)$$

(57)

To find the MLE for $w,$ we aim to minimize the function provided in Equation (58):

$$w^{MLE}=argmax\left(w\right)-\sum _{k=1}^{m}log\left(1+e^{-y_k{w}^T{x}_k}\right)=argmin\left(w\right)\sum _{k=1}^{m}log\left(1+e^{-y_k{w}^T{x}_k}\right)$$

(58)

Minimizing the function in Equation (58) is our goal, achieved through gradient descent on the negative log likelihood in Equation (59).

$$\mathcal{L}\left(w\right)=\sum _{k=1}^{m}log\left(1+e^{-y_k{w}^T{x}_k}\right)$$

(59)

Maximum a Posteriori (MAP): In maximum a posteriori (MAP), assuming a Gaussian prior, the objective is to find $w^{MAP}$ that maximizes the posterior probability, represented mathematically in Equation (60). Reformulating, this becomes an optimization problem, as shown in Equation (61), where $\mathsf{\lambda }=\frac{1}{2{\mathsf{\sigma }}^2}$, and gradient descent is employed on the negative log posterior $l\left(w\right)$ for parameter optimization [32,37].

$$w^{MAP}=\arg \underset{w}{\max }\log \left(P\left(y|x,w\right)P\left(w\right)\right)\propto P\left(y|x,w\right)P\left(w\right)$$

(60)

$$w^{MAP}=\arg \underset{w}{\min }\sum _{k=1}^m\log \left(1+e^{-y_k{w}^T{x}_k}\right)+\mathsf{\lambda }{w}^{T}w$$

(61)

2.5.6. Decision Tree (DT)

Decision trees, used for regression and classification, form a hierarchical structure with nodes for decisions, branches for outcomes, and leaves for predictions. The goal is a compact tree with pure leaves, ensuring each contains instances from a single class. Achieving consistency is computationally challenging due to the NP-hard complexity of finding a minimum-size tree [37]. Impurity functions in decision trees, evaluated on a dataset D with pairs $\left(a_1,b_1\right),\dots ,\left(a_n,b_n\right)$, where $b_i$ takes values in $\{1,\dots ,m\}$ representing m classes, are crucial for assessing tree quality.

Gini Impurity: Gini impurity in a decision tree is calculated for a leaf using Equation (62), and the Gini impurity for the entire tree is given by Equation (63).

$$I\left(D\right)=\sum _{m=1}^k{q}_m\left(1-q_m\right)$$

(62)

$$G_T\left(D\right)=\frac{\left|D_L\right|}{\left|D\right|}{G}_T\left(D_L\right)+\frac{\left|D_R\right|}{\left|D\right|}{G}_T\left(D_R\right)$$

(63)

where: $D=D_L\cup D_R$, $D_L\cap D_R=\mathrm{\varnothing }$, $\frac{\left|D_L\right|}{\left|D\right|}$ represents the fraction of inputs in the left subtree, and $\frac{\left|D_R\right|}{\left|D\right|}$ represents the fraction of inputs in the right subtree. The binary decision tree with class levels has been visualized in Figure 8.

Entropy in Decision Trees: Entropy in decision trees measures disorder using class fractions. Minimizing entropy aligns with a uniform distribution, promoting randomness. KL-Divergence $KL\left(p\right|\left|q\right)$ gauges the closeness of $p$ to a uniform distribution $q$, as in Equation (64).

$$\begin{array}{cc}\hfill KL\left(p\right|\left|q\right)& ={\displaystyle \sum _{n=1}^c}{p}_n\mathit{log}\frac{p_n}{q_n}>0\leftarrow KL-Divergence\hfill \\ & ={\displaystyle \sum _n}{p}_{n}log\left(p_n\right)-p_{n}log\left(q_n\right),\mathit{where}{q}_n=\frac{1}{c}\hfill \\ & ={\displaystyle \sum _n}{p}_{n}log\left(p_n\right)+p_{n}log\left(c\right)\hfill \\ & ={\displaystyle \sum _n}{p}_{n}log\left(p_n\right)+log\left(c\right){\displaystyle \sum _n}{p}_n,\mathit{where}log\left(c\right)\leftarrow constant,{\displaystyle \sum _n}{p}_n=1\hfill \\ \hfill \underset{p}{\mathit{max}}KL(p||q)& =\underset{p}{\mathit{max}}{\displaystyle \sum _n}{p}_{n}log\left(p_n\right)=\underset{p}{\min }-{\displaystyle \sum _n}{p}_{n}log\left(p_n\right)=\underset{p}{\min }H\left(s\right)\leftarrow Entropy\hfill \end{array}$$

(64)

ID3 Algorithm: The ID3 algorithm stops tree-building when all labels are the same or no more attributes can split further. If all share the same label, a leaf with that label is created. If no more splitting attributes exist, a leaf with the most frequent label is generated (Equation (65)) [39].

$$ID3(S):\left\{\begin{array}{c}if \exists \overrightarrow{y} s.t. \forall (x,y) \in S, y=\overrightarrow{y}, return leaf with label \overrightarrow{y} \hfill \\ if \exists \overrightarrow{x} s.t. \forall (x,y) \in S, x=\overrightarrow{x} return leaf without mode (y: (x,y)\in S) \hfill \end{array}\right.$$

(65)

CART (Classification and Regression Trees): CART (classification and regression trees) is suitable for continuous labels ($y_i\in R$), using the squared loss function (Equation (66)). It efficiently finds the best split (attribute and threshold) by minimizing the average squared difference from the average label $y_s$ [37].

$$L\left(S\right)=\frac{1}{\left|S\right|}\sum _{\left(i,j\right)\in S}{\left(y-\overrightarrow{y_S}\right)}^2\leftarrow \mathit{Average}\mathit{squared}\mathit{difference}\mathit{from}\mathit{average}\mathit{label}$$

(66)

where $\overrightarrow{y_S}=\frac{1}{\left|S\right|}{\sum }_{\left(i,j\right)\in S}y\leftarrow \mathit{average}\mathit{label}$

2.5.7. Ensemble Classifier (EC)

Ensemble classifiers represent a sophisticated class of machine learning techniques aimed at enhancing the precision and resilience of predictive models. Their fundamental premise revolves around the amalgamation of predictions from multiple foundational models. Below, we delve into several prominent types of ensemble classifiers, each with its distinct modus operandi.

Bagging (Bootstrap Aggregating): Bagging orchestrates the training of multiple foundational models in parallel. Each model operates independently on distinct, resampled subsets of the training data. This decomposition helps us understand the sources of error in our models. Bias/variance decomposition is described by Equation (67) [37,40]:

$$\underset{Error}{\underbrace{E\left[{(f_k(x)-b)}^2\right]}}=\underset{\mathrm{V}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}{\underbrace{E\left[{(f_k(x)-\overrightarrow{f}(x))}^2\right]}}+\underset{\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}{\underbrace{E\left[{(\overrightarrow{f}(x)-\overrightarrow{c}())}^2\right]}}+\underset{\mathrm{N}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}{\underbrace{E\left[{(\overrightarrow{c}(x)-d(x))}^2\right]}}$$

(67)

In Equation (67), we decompose the error into four components: “Error”, “Variance”, “Bias”, and “Noise”. Our primary objective is to minimize the “Variance” term, which is expressed as Equation (68):

$$\underset{\mathrm{V}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}}{\underbrace{E[{(f_k(x)-\overrightarrow{f}(x))}^2]}}$$

(68)

Ensemble learning minimizes variance by averaging individual predictions $f_k\left(x\right).$ Bagging enhances ML classifiers by creating multiple datasets, training individual classifiers $h_i\left(\right)$, and aggregating predictions in the final ensemble classifier $h\left(z\right)$, through averaging (Equation (69)) [40]:

$$h\left(z\right)=\frac{1}{n}\sum _{i=1}^n{h}_i\left(z\right)$$

(69)

In practice, a larger value of n often leads to a better-performing ensemble, as it leverages diverse base models for more robust predictions.

Random Forest (RF): RF stands as one of the most renowned and beneficial bagging algorithms. The RF algorithm entails creating multiple datasets, building decision trees with random feature subsets for each dataset, and averaging their predictions for the final classifier [37,40] $\left((h\left(x\right)=\frac{1}{m}\sum h_j\left(x\right))\right)$.

Boosting: Boosting addresses high bias in machine learning models, specifically when dealing with the hypothesis class H. Boosting reduces bias by iteratively constructing an ensemble of weak learners $\left(H_T\left(\overrightarrow{x}\right)={\sum }_{t=1}^T{\alpha }_t{h}_t\left(\overrightarrow{x}\right)\right)$ with each iteration introducing a new classifier, guided by gradient descent in function space [37,41].

Gradient descent: Gradient descent in functional space optimizes the loss function ℓ within hypothesis class H by finding the appropriate step size α and weak learner h that minimizes $\mathrm{l}\left(H+\mathsf{\alpha }h\right).$ The technique uses Taylor approximation to approximate the optimal weak learner h with a fixed α around 0.1 (Equation (70)) [34].

$$argmi{n}_{h\in H\mathrm{l}}\left(H+\mathsf{\alpha }h\right)\approx argmi{n}_{h\in H}<\nabla \mathrm{l}\left(H\right),h\ge argmi{n}_{h\in H}\sum _{i=1}^n\frac{\partial \mathrm{l}}{\partial \left[H\left(x_i\right)\right]}h\left(x_i\right)$$

(70)

Here, each prediction serves as an input to the loss function. The function ℓ(H) can be expressed by Equation (71).

$$l\left(H\right)\mathrm{ }=\mathrm{ }\sum _{i\mathrm{ }=\mathrm{ }1}^{nl}\left(H\left(x_i\right)\right)\mathrm{ }=\mathrm{ }l\left(H\left(x_1\right),\mathrm{ }\dots \mathrm{ },\mathrm{ }H\left(x_n\right)\right)$$

(71)

This approximation enables the utilization of boosting as long as there exists a method, denoted as A, capable of solving Equation (72).

$$h_{t+1}=argmi{n}_{h\in H}\sum _{i=1}^n\underset{r_i}{\underbrace{\frac{\partial \mathrm{l}}{\partial \left[H\left(x_i\right)\right]}}}h\left(x\right)$$

(72)

where $A\left(\{\left(x_1,r_1\right),\dots ,\left(x_n,r_n\right)\}\right)=argmi{n}_{h\in H}{\sum }_{i=1}^n{r}_{i}h\left(x_i\right)$; progress is made as long as ${\sum }_{i=1}^n{r}_{i}h\left(x_i\right)<0$, even if h is not an excellent learner.

AnyBoost (Generic Boosting): AnyBoost, a versatile boosting technique, iteratively combines weak learners, prioritizing challenging data points for enhanced accuracy. It creates a strong learner from weak ones, effectively reducing bias and improving predictions. See Algorithm 1 for the pseudo-code [41].

Algorithm 1: Pseudo-code for the AnyBoost.

Input: l, a, {(xi, yi)}, A H0 = 0 for t = 0: T − 1 do   $\forall I:r_i=\frac{\partial l\left(\right(H_t\left(x_1\right),y_1),\dots ,(H_t\left(x_n\right),y_n) }{\partial H\left(x_i\right)}$    $h_{t+1}=A\left(\right\{(x_1,r_1),\dots .,(x_n,r_n)\left\}\right)=argmi{n}_{h\in H}{\sum }_{i=1}^n{r}_{i}h\left(x_i\right)$    if ${\sum }_{i=1}^n{r}_i{h}_{t+1}\left(x_i\right)<0$ then     $H_{t+1}=H_t+{\alpha }_{t+1}{h}_{t+1}$    else     return Ht (Negative gradient orthogonal to descent direction.)    end end return HT

Gradient Boosted Regression Trees (GBRT): GBRT, a sequential regression algorithm, combines decision trees to correct errors iteratively for precise predictions. Applicable to both classification and regression, it uses weak learners, often shallow regression trees, with a fixed depth. The step size (α) is a small constant, and the loss function (l) must be differentiable, convex, and decomposable over individual samples. The ensemble’s overall loss is defined in Equation (73) [41].

$$\mathcal{L}\left(H\right)=\sum _{i=1}^{n}l\left(H\left(x_i\right)\right)$$

(73)

GBRT minimizes the loss by iteratively adding weak learners to the ensemble. Pseudo-code is in Algorithm 2 [41].

Algorithm 2: Pseudo-code for GBRT

Input: l, α, {(xi, yi)}, A H = 0 for t = 1: T do   $\forall i : t_i=y_i-H\left(x_i\right)$   $h={argmin}_{h\in H}{\left(h\right(x_i)-t_i)}^2$   $H\leftarrow H+\alpha h$ end return H

AdaBoost: AdaBoost is a binary classification algorithm utilizing weak learners h producing binary predictions. Key components include step-size α and exponential loss ℓ(H), given by Equation (74):

$$l\left(H\right)=\sum _{i=1}^n{e}^{-y_{i}H\left(x_i\right)}$$

(74)

The gradient function $r_i$ needed to find the optimal weak learner is computed using Equation (75).

$$r_i=\frac{\partial \mathcal{L}}{\partial H\left(x_i\right)}=-y_i{e}^{-y_{i}H\left(x_i\right)}$$

(75)

Introducing $w_i=\frac{1}{Z}{e}^{-y_{i}H\left(x_i\right)}$, for clarity and convenience, where $Z={\sum }_{i=1}^n{e}^{-y_{i}H\left(x_i\right)}$, normalizing the weights. Each $w_i$ signifies the role of $\left(x_i,y_i\right)$ in the global loss. To find the next weak learner, we solve the optimization problem in Equation (76) with $h\left(x_i\right)\in \{+1,-1\}$ [42].

$$\begin{gathered} h\left(x_i\right)=argmi{n}_{h\in H}\sum _{i=1}^n{r}_{i}h\left(x_i\right)\hspace{1em}\left(\mathit{substitute}\mathit{in}:r_i=e^{-H\left(x_i\right)y_i}\right) \\ =argmi{n}_{h\in H}-\sum _{i=1}^n{y}_i{e}^{-H\left(x_i\right)y_i}h\left(x_i\right)\hspace{1em}\left(\mathit{substitute}\mathit{in}:w_i=\frac{1}{Z}{e}^{-H\left(x_i\right)y_i}\right) \\ =argmi{n}_{h\in H}-\sum _{i=1}^n{w}_i{y}_{i}h\left(x_i\right)\hspace{1em}\left(y_{i}h\left({\mathit{x}}_{\mathit{i}}\right)\in \{+1,-1\}\mathit{with}h\left({\mathit{x}}_{\mathit{i}}\right)y_i=1⟺h\left({\mathit{x}}_{\mathit{i}}\right)=y_i\right) \\ =argmi{n}_{h\in H}\sum _{i:h\left({\mathit{x}}_{\mathit{i}}\right)\ne y_i}{w}_i-\sum _{i:h\left(x_i\right)=y_i}{w}_i\hspace{1em}\left(\sum _{i:h\left(x_i\right)=y_i}{w}_i=1-\sum _{i:h\left(x_i\right)\ne y_i}{w}_i\right) \\ =argmi{n}_{h\in H}\sum _{i:h\left({\mathit{x}}_{\mathit{i}}\right)\ne y_i}{w}_i\hspace{1em}\left(\mathit{This}\mathit{is}\mathit{the}\mathit{weighted}\mathit{classification}\mathit{error}.\right) \end{gathered}$$

(76)

In (76), $ϵ={\sum }_{i:h\left(x_i\right)\ne y_i}{w}_i$, representing the weighted classification error. AdaBoost seeks a classifier minimizing this error without requiring high accuracy. The optimal step size, denoted as $\mathsf{\alpha },$ minimizes the loss $l$ most effectively in the closed-form optimization problem (77) [41].

$$\begin{gathered} \alpha =argmi{n}_{\alpha }l\left(H+\alpha h\right) \\ =argmi{n}_{\mathsf{\alpha }}\sum _{i=1}^n{e}^{-y_i\left[H\left(x_i\right)+\mathsf{\alpha }h\left(x_i\right)\right]} \end{gathered}$$

(77)

Taking the derivative with respect to α and setting it to zero, as shown by Equations (78)–(80):

$$\sum _{i=1}^n{y}_{i}h\left(x_i\right)e^{-y_i\left[H\left(x_i\right)+\mathsf{\alpha }{y}_{i}h\left(x_i\right)\right]}=0\hspace{1em}\hspace{1em}(y_{i}h(x_i)\in \{+1 \mathit{or}-1\})$$

(78)

$$-\sum _{i:h\left(x_i\right)y_i=1}{e}^{-\left(y_{i}H\left(x_i\right)+\mathsf{\alpha }\underset{1}{\underbrace{y_{i}h\left(x_i\right)}}\right)}+\sum _{i:h\left(x_i\right)y_i\ne 1}{e}^{-\left(y_{i}H\left(x_i\right)+\mathsf{\alpha }\underset{-1}{\underbrace{y_{i}h\left(x_i\right)}}\right)}=0\hspace{1em}\hspace{1em}\left(w_i=\frac{1}{Z}{e}^{-y_{i}H\left(x_i\right)}\right)$$

(79)

$$-\sum _{i:h\left(x_i\right)y_i=1}{w}_i{e}^{-\mathsf{\alpha }}+\sum _{i:h\left(x_i\right)y_i\ne 1}{w}_i{e}^{\mathsf{\alpha }}=0\hspace{1em}\hspace{1em}\left(ϵ=\sum _{i:h\left(x_i\right)y_i=-1}{w}_i\right)$$

(80)

For further simplification, with ε representing the sum over misclassified examples, as given in Equation (81):

$$-\left(1-\mathsf{ϵ}\right)e^{-\mathsf{\alpha }}+\mathsf{ϵ}{e}^{\mathsf{\alpha }}=0$$

(81)

Solving for α, as shown in Equation (82):

$$e^{2\mathsf{\alpha }}=\frac{1-\mathsf{ϵ}}{\mathsf{ϵ}}$$

(82)

$$\mathsf{\alpha }=\frac{1}{2}\ln \frac{1-\mathsf{ϵ}}{\mathsf{ϵ}}$$

(83)

The optimal step size α, derived from the closed-form solution in (83), facilitates rapid convergence in AdaBoost. After each step $H_{t+1}=H_t+\alpha h$, recalculating and re-normalizing all weights is crucial for the algorithm’s progression. The pseudo-code for AdaBoost Ensemble classifier is presented in Algorithm 3 [37,41].

Algorithm 3: Pseudo-code for AdaBoost

Input: l, α, {(xi, yi)}, A H = 0 $\forall i : w_i=\frac{1}{n}$ for t = 1: T do     $h=A \left(w_1, x_1, y_1\right),\dots \dots \dots , \left(w_n, x_n, y_n\right)$      $ϵ={\displaystyle \sum _{i:h\left(x_i\right)\ne y_i}}{w}_i$     if $ϵ<\frac{1}{2}$  then       $\mathsf{\alpha }=\frac{1}{2}\ln \frac{1-\mathsf{ϵ}}{\mathsf{ϵ}}$       $H_{t+1}=H_t+\alpha h$       $\forall i : w_i\leftarrow \frac{w_i{e}^{-\alpha h\left(x_i\right)y_i}}{2ϵ{\left(1-ϵ\right)}^{\frac{1}{2}}}$     else        return (Ht)     end     return H end

2.6. Assessment Metrics

The crucial next step in evaluating machine learning classifiers is the use of a separate test dataset that has not been part of the training process. Evaluation involves various parameters, with the confusion matrix being a widely adopted tool. This matrix forms the basis for determining assessment metrics, essential for validating model performance, whether it is a traditional or deep neural network classifier. In cancer prediction tasks, numerous metrics are employed to assess effectiveness, including error rate, accuracy, sensitivity, specificity, recall, precision, predictivity, F1 score, area under the curve (AUC), negative predictive value (NPR), false positive rate (FPR), and false negative rate (FNR), and Matthews correlation coefficient (MCC) [43]. These metrics quantify predictive capabilities and are vital for diverse prediction tasks. Multiple performance evaluation metrics rely on the confusion matrix, as visualized in Figure 9, for multiclass classification.

Accuracy (Acc): This metric is a fundamental indicator of a model’s overall performance. It measures the ratio of accurately categorized cases (both cancer and non-cancer) to the overall cases in the test dataset. It may not be suitable when the dataset is imbalanced.

$$\mathit{Accuracy} (\mathrm{\%}ACC)=\frac{(\mathrm{TP}+\mathrm{TN}) }{ \mathrm{Total} \mathrm{Samples}}\times 100$$

Error Rate (ER): The reciprocal of accuracy equates to the error rate. It quantifies the proportion of instances that the model incorrectly classifies. A lower error rate suggests a more accurate model, and it is especially useful when you want to know how often the model makes incorrect predictions.

$$\mathit{Error}\mathit{rate}\left(ER\right)=1-Acc$$

$$\mathrm{\%}ER=\frac{FP+FN}{\mathit{Total}\mathit{Samples}}\times 100=100-(\mathrm{\%}ACC)$$

Specificity (% Spe): True negative rate, commonly known as specificity, is a metric that evaluates a model’s accuracy in correctly identifying true negative cases. This is crucial in minimizing false alarms.

$$Specificity\left(\mathrm{\%}\mathrm{S}\mathrm{p}\right)=\mathrm{True}\mathrm{Negative}\mathrm{Rate}(\mathrm{\%}\mathrm{T}\mathrm{N}\mathrm{R})=\frac{TN}{\mathrm{Total}\mathrm{Negative}}\times 100$$

Sensitivity (% Sen): This metric, also termed recall or the true positive rate (TPR), gauges the model’s capability to accurately identify true positive values, which correspond to cases of cancer, among the total positive cases within a dataset [42].

$$Sensitivity\left(\mathrm{\%}Sen\right)=Recall\left(\mathrm{\%}Re\right)=\mathit{True}\mathit{Positive}\mathit{Rate}\left(\mathrm{\%}TPR\right)=\frac{TP}{\mathrm{Total}\mathrm{Positive}}\times 100$$

Precision (% Pr): Precision, also recognized as positive predictive value (PP), denotes the ability to accurately predict positive values among the true positive predictions. A high precision score signifies that the model effectively reduces false positive errors.

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{\%}\mathrm{P}\mathrm{r}\right)=\mathit{Positive}\mathit{Predictivity}(\mathrm{\%}PP)=\frac{TP}{\mathrm{True}\mathrm{Prediction}}\times 100$$

F1 Score (% F1): An equitable metric that amalgamates positive predictive value and recall forms the F1 score [44]. It is particularly valuable when you require a singular metric that contemplates both incorrect positive predictions and missed positive predictions.

$$\text{F1-score} (\%\mathrm{F}1)=\frac{2\times \mathrm{TP}}{(2\times \mathrm{TP}+\mathrm{FP}+\mathrm{FN})}\times 100=\frac{2\mathrm{PP} \times \mathrm{TPR}}{(\mathrm{PP}+\mathrm{TPR})}\times 100$$

Area Under the Curve (AUC): The AUC assesses the classifier’s capacity to differentiate between affirmative and negative occurrences. It gauges the general efficacy of the model concerning receiver operating characteristic (ROC) graphs. A superior AUC score signifies enhanced differentiation capability.

Negative Predictive Value (% NPV): It measures the classifier’s capability to accurately predict negative instances among all instances classified as negative. A high NPV suggests that the classifier is effective at identifying non-cancer cases when it predicts them as such, reducing the likelihood of unnecessary treatments.

$$\mathit{Negative}\mathit{Predictive}\mathit{Value}\left(\mathrm{\%}NPV\right)=\frac{TN}{\mathit{Total}\mathit{Negative}}\times 100$$

False Positive Rate (%FPR): This quantifies how often the classifier falsely identifies a negative instance as positive. It provides insights into the model’s propensity for false positive errors. In cancer detection, a high FPR can lead to unnecessary distress and treatments for individuals who do not have cancer.

$$\mathit{False}\mathit{Positive}\mathit{Rate}\left(\mathrm{\%}FPR\right)=\frac{FP}{\mathit{Total}\mathit{Negative}}\times 100$$

False Negative Rate (%FNR): It determines the classifier’s tendency to falsely identify a positive instance as negative. It reveals the model’s performance regarding false negative errors, which is critical in cancer detection to avoid missing real cases. High FNR can lead to undiagnosed cancer cases and potentially delayed treatments.

$$\mathit{False}\mathit{Negative}\mathit{Rate} (\mathrm{\%}FNR)=\frac{\mathrm{FN} }{ \mathrm{Total} \mathrm{Positive}}\times 100$$

Matthews Correlation Coefficient (MCC): The Matthews correlation coefficient (MCC) represents a pivotal metric utilized for evaluating the effectiveness of binary (two class) predictions, prominently beneficial when dealing with scenarios where classes are asymmetrically distributed in their volume and representation within the dataset. The formula to calculate MCC is:

$$\frac{[(TN\ast TP)-(FN\ast FP)]}{\sqrt{\left((\mathrm{True}\mathrm{Prediction})\ast (\mathrm{False}\mathrm{Predication})\ast (\mathit{Total}\mathit{Positive})\ast (\mathit{Total}\mathit{Negative})\right)}}$$

where TN (True Negative) is accurately recognized negatives, TP (True Positive) is accurately recognized positives, FP (False Positive) is negatives incorrectly identified as positives, FN (False Negative) is positives incorrectly recognized as negatives, Total Positive is the Sum of TP and FN (all actual positives), Total Negative is the Sum of TN and FP (all actual negatives), True Prediction: Sum of TP and FP (correctly identified positives), False Predication: Sum of FN and TN (incorrectly identified negatives), Total Samples: Sum of TP, TN, FP, and FN (entire dataset).

3. Review Analysis

In this section, we present a thorough and extensive analysis of cancer detection utilizing conventional machine learning models applied to medical imaging datasets. Our study is focused exclusively on the detection of two specific types of cancer: colorectal and stomach cancer. For each of these cancer types, we have meticulously compiled a comprehensive review table that encompasses the relevant literature published during the period spanning 2017 to 2023. This table encompasses a range of crucial review parameters, including the year of publication, the datasets utilized, preprocessing methods, feature extraction techniques, machine learning classifiers employed, the number of images involved, the imaging modality, and various performance metrics. In total, our review encompasses 36 research articles that have harnessed medical imaging datasets to detect these specific types of cancer. Our primary emphasis lies in scrutinizing the utilization of traditional machine learning methodologies in the context of cancer detection using image datasets. We have conducted this analysis based on the meticulously assembled review tables. Subsequent subsections provide in-depth and comprehensive reviews for both colorectal and stomach cancer. Within our analysis, we delve into the intricate application of machine learning approaches for the intent of cancer prediction. Our overarching goal is to furnish valuable insights into the efficacy and constraints of conventional machine learning models when applied to the realm of cancer detection using medical imaging datasets. Through a meticulous examination and comparative analysis of results derived from various studies, our objective is to make a meaningful contribution to the evolution of cancer detection methodologies and to offer guidance for future research endeavors in this critical domain.

3.1. Analysis of Colorectal Cancer Prediction

Table 3 showcases 20 studies conducted from 2017 to 2023, focusing on machine learning-based colorectal cancer detection. These studies underscore the vital role of preprocessing methods in enhancing detection accuracy. The highest accuracy achieved is 100%, with the lowest at 76.00%. Various techniques, including cropping, stain normalization, contrast enhancement, smoothing, and filtering, were employed in conjunction with segmentation, feature extraction, and machine learning algorithms like SVM, MLP, RF, and KNN. These approaches successfully detect colorectal cancer using modalities such as endocytoscopy, histopathological images, and clinical data. The studies employed varying quantities of images, patients, or slices, ranging from 54 to 100,000. The “KCRC-16” datasets are prominently featured in these analyses.

In a comparative analysis of colorectal cancer detection studies, (Talukder et al., 2022) [45] stood out with an impressive accuracy of 100%. Their approach included preprocessing steps like resizing, BGR2RGB conversion, and normalization. Deep learning models such as DenseNet169, MobileNet, VGG19, VGG16, and DenseNet201 were employed. Performance assessment was conducted using a combination of voting, XGB, EC, MLP, LGB, RF, SVM, LR, and hybrid techniques on a dataset comprising 2800 H&E images from the LC25000 dataset. Their best model achieved a flawless 100% accuracy. In contrast, (Ying et al., 2022) [46] achieved thelowest accuracy of 76.0% in colorectal cancer detection. Their approach involved manual region of interest (ROI) selection and various preprocessing techniques. They leveraged multiple features, including FOS, shape, GLCM, GLSZM, GLRLM, NGTDM, GLDM, LoG, and WT. Classification was carried out using the MLR technique on a dataset consisting of 276 CECT images from a private dataset. Their least-performing model achieved an accuracy of 76.00%. Moreover, their study exhibited a sensitivity of 65.00%, specificity of 80.00%, and precision of 54.00%, indicating relatively suboptimal performance in accurately identifying colorectal cancer cases.

(Khazaee Fadafen and Rezaee 2023) [47] conducted a remarkable colorectal cancer detection study by utilizing a substantial dataset (the highest number of images among all) comprising a total of 100,000 medical images sourced from the H&E NCT-CRC-HE-100K dataset. Their preprocessing methodology encompassed the conversion of RGB images to the HSV color space and the utilization of the lightness space. For classification, they harnessed the dResNet architecture in conjunction with DSVM, which resulted in an outstanding accuracy rate of 99.76%. (Jansen-winkeln et al., 2021) [48] conducted a study with a notably smallest dataset, comprising only 54 medical images. Their preprocessing approach included smoothing and normalization. For classification purposes, they employed a combination of MLP, SVM, and RF techniques. This approach yielded commendable results with an accuracy of 94.00%, sensitivity at 86.00%, and specificity reaching 95.00%. Notably, their analysis identified MLP as the most effective model in their study.

Within the corpus of 20 studies dedicated to the realm of colorectal cancer detection, researchers have deployed an array of diverse preprocessing strategies encompassing endocytoscopy, cropping, IPP, stain normalization, CEI, smoothing, normalization, filtering, THN, DRR, augmentation, UM-SN, resizing, BGR2RGB, normalization, scaling, labeling, RGBG, VTI, HOG, RGB to HSV, lightness space, edge preserving, and linear transformation. These sophisticated methodologies collectively served as the linchpin for optimizing machine learning-based colorectal cancer detection, ushering in a new era of precision and accuracy. However, it is captivating to note that, within the comprehensive assessment of 23 studies, a select quartet of research endeavors chose to forgo the utilization of any specific preprocessing techniques. This exceptional cluster includes the works of (Bora et al., 2021) [49], (Fan et al., 2021) [50], and (Lo et al., 2023) [51]. Astonishingly, these studies defied conventional wisdom by attaining commendable accuracies that spanned the spectrum from 94.00% to an impressive 99.44%. Such outcomes suggest that, in cases where the dataset is inherently pristine and impeccably aligned with the demands of the classification task, the impact of preprocessing techniques on the classifier’s performance might indeed exhibit a marginal influence.

In the comprehensive analysis of the research studies under scrutiny, it is noteworthy that only the works of (Grosu et al., 2021) [52] and (Ying et al., 2022) [46] registered accuracy figures falling below the 90% threshold, specifically at 84.7% and 76%, respectively. This observation underscores the intriguing possibility that traditional machine learning models can indeed yield highly accurate cancer detection performance, provided they are meticulously optimized.

Table 3. Performance comparison of traditional ML-based colorectal cancer prediction methods.

Year	References	Pre-Processing	Features	Techniques	Dataset	Data Samples	Train Data	Test Data	Modality	Metrics (%)
2017	[53]	Endocytoscopy	Texture, nuclei	SVM	Private	5843	5643	200	ENI	Acc 94.1 Sen 89.4 Spe 98.9 Pre 98.8 NPV 90.1
2019	[54]	IPP	CSQ, Color histogram	WSVMCS	Private	180	108	72	H&E	Acc 96.0
2019	[55]	Cropping	Biophysical characteristic, WLD,	NB, MLP,	OMIS data	316	237	79	OMIS	Acc 92.6 Sen 96.3 Spe 88.9
2021	[56]	Filtering	HOS, FOS, GLCM, Gabor, WPT, LBP	ANN, RSVM,	KCRC-16	5000	4550	450	H&E	Acc 95.3
2021	[57]	IPP, Augmentation	VGG-16	MLP	KCRC-16	5000	4825	175	H&E	Acc 99.0 Sen 96.0 Spe 99.0 Pre 96.0 NPV 99.0 F1 96.0
2021	[50]	---	AlexNet	EC, SVM, AlexNet,	LC25000	10,000	4-fold cross validation		H&E	Acc 99.4
2021	[58]	THN, DRR	BmzP	NN	MALDI MSI	559	Leave-One-Out cross-validation		H&E	Acc 98.0 Sen 98.2 Spe 98.6
2021	[52]	Filtering	Filters, Texture, GLHS, Shape	RF	Private	287	169	77	CT	Acc 84.7 * Sen 82.0 Spe 85.0 AUC 91.0
2021	[49]	---	GFD, NSCT, Shape	MLP LSSVM,	Private	734	five-fold cross-validation		NBI, WLI	Acc 95.7 Sen 95.3 Spe 95.0 Pre 93.2 F1 90.5
2021	[48]	Normalization, smoothing	Spatial Information	MLP, SVM, RF	Private	54	Leave-One-Out cross-validation		HSI	Acc 94.0 Sen 86.0 Spe 95.0
2022	[59]	VTI	Haralick, VTF	RF	Private	63	cross-validation method		CT	Acc 92.2 Sen 88.4 Spe 96.0 AUC 96.2
2022	[60]	RGBG	GLCM	ANN, RF, KNN	KCRC-16	5000	4500	500	H&E	Acc 98.7 Sen 98.6 Spe 99.0 Pre 98.9
2022	[45]	Resize, BGR2RGB, Normalization,	Deep Features	EC, Hybrid, LR, LGB, MLP, RF, SVM, XGB, Voting	LC25000	2800	10-fold cross-validation		H&E	Acc 100.0
2022	[46]	ROI	FOS, GLCM, GLDM, GLRLM, GLSZM, LoG, NGTDM, Shape, WT	MLR	Private	276	194	82	CECT	Acc 76.0 Sen 65.0 Spe 80.0 Pre 54.0 NPV 86.0
2022	[61]	UM-SN	HIM, GLCM, Statistical	LDA, MLP, RF, SVM, XGB, LGB	LC25000	1000	900	100	H&E	Acc 99.3 Sen 99.5 Pre 99.5 F1 99.5
2022	[26]	---	Color Spaces, Haralick	ANN, DT, KNN, QDA, SVM	KCRC-16	5000	3504	1496	H&E	Acc 97.3 Sen 97.3 Spe 99.6 Pre 97.4
2023	[62]	Filtering, linear Transformation, normalization	Color characteristic, DBCM, SMOTE	CatBoost, DT, GNB, KNN, RF	NCT-CRCHE-7K	12,042	8429	3613	H&E	Acc 90.7 Sen 97.6 Spe 97.4 Pre 90.6 Rec 90.5 F1 90.5
2023	[51]	---	Clinical, FEViT	SEKNN	Private	1729	tenfold cross-validation		ENI	Acc 94.0 Sen 74.0 Spe 98.0 AUC 93.0
2023	[47]	Lightness space, RGB to HSV	dResNet	DSVM	KCRC-16	5000	4000	1000	H&E	Acc 98.8
					NCT-CRC-HE-100 K	100,000	80,003	19,997	H&E	Acc 99.8
2023	[63]	HOG, RGBG, Resizing	Morphological	SVM	Private	540	420	120	ENI	Acc 97.5

* Not given in the paper, calculated from the result table, bold font signifies the best model in the ‘Techniques’ column. Abbreviations: BGR2RGB, Blue-Green-Red to Red-Green-Blue; BmzP, Binning of m/z Points; catBoost, Categorical Boosting; CECT, Contrast-Enhanced CT; CSQ, Color Space Quantization; DBCM, Differential Box Count Method; DSVM, Deep Support Vector Machine; dResNet, Dilated ResNet; DRR, Dynamic Range Reduction; DSVM, Deep Support Vector Machine; ENI, Endomicroscopy Images; FEViT, Feature Ensemble Vision Transformer; FOS, First-Order Statistics; GFD, Generic Fourier Descriptor; GNB, Gaussian Naive Bayes; GLDM, Gray-Level Dependence Matrix; GLHS, Gray Level Histogram Statistics; GLSZM, Gray Level Size Zone Matrix; GNB, Gaussian Naive Bayes; HOG, Histogram of Oriented Gradients; HOS, Higher-Order Statistic; HIM, Hu Invariants Moments; HSI, Hyperspectral Imaging; HSV, Hue-Saturation-Value; LBP, Local Binary Pattern; LDA, Linear Discriminant Analysis; LGB, Light Gradient Boosting; LoG, Laplacian of Gaussian; LSSVM, Least Square Support Vector Machine; MLR, Multivariate Logistic Regression; NGTDM, Neighboring Gray Tone Difference Matrix; NSCT, Non-Subsampled Contourlet Transform; OMIS, Optomagnetic Imaging Spectroscopy; QDA, Quadratic Discriminant Analysis; SEKNN, Subspace Ensemble K-Nearest Neighbor; THN, TopHat and Normalization; UMSN, Unsharp Masking and Stain Normalization; VTF, Vector Texture Features; VTI, Vector Texture Images; WLD, Wavelength Difference; WLI, White Light Imaging; WPT, Wavelet Packet Transform; WSVMCS, Wavelet Kernel SVM with Color Histogram; XGB, Extreme Gradient Boosting.

Table 3. Performance comparison of traditional ML-based colorectal cancer prediction methods.

Year	References	Pre-Processing	Features	Techniques	Dataset	Data Samples	Train Data	Test Data	Modality	Metrics (%)
2017	[53]	Endocytoscopy	Texture, nuclei	SVM	Private	5843	5643	200	ENI	Acc 94.1 Sen 89.4 Spe 98.9 Pre 98.8 NPV 90.1
2019	[54]	IPP	CSQ, Color histogram	WSVMCS	Private	180	108	72	H&E	Acc 96.0
2019	[55]	Cropping	Biophysical characteristic, WLD,	NB, MLP,	OMIS data	316	237	79	OMIS	Acc 92.6 Sen 96.3 Spe 88.9
2021	[56]	Filtering	HOS, FOS, GLCM, Gabor, WPT, LBP	ANN, RSVM,	KCRC-16	5000	4550	450	H&E	Acc 95.3
2021	[57]	IPP, Augmentation	VGG-16	MLP	KCRC-16	5000	4825	175	H&E	Acc 99.0 Sen 96.0 Spe 99.0 Pre 96.0 NPV 99.0 F1 96.0
2021	[50]	---	AlexNet	EC, SVM, AlexNet,	LC25000	10,000	4-fold cross validation		H&E	Acc 99.4
2021	[58]	THN, DRR	BmzP	NN	MALDI MSI	559	Leave-One-Out cross-validation		H&E	Acc 98.0 Sen 98.2 Spe 98.6
2021	[52]	Filtering	Filters, Texture, GLHS, Shape	RF	Private	287	169	77	CT	Acc 84.7 * Sen 82.0 Spe 85.0 AUC 91.0
2021	[49]	---	GFD, NSCT, Shape	MLP LSSVM,	Private	734	five-fold cross-validation		NBI, WLI	Acc 95.7 Sen 95.3 Spe 95.0 Pre 93.2 F1 90.5
2021	[48]	Normalization, smoothing	Spatial Information	MLP, SVM, RF	Private	54	Leave-One-Out cross-validation		HSI	Acc 94.0 Sen 86.0 Spe 95.0
2022	[59]	VTI	Haralick, VTF	RF	Private	63	cross-validation method		CT	Acc 92.2 Sen 88.4 Spe 96.0 AUC 96.2
2022	[60]	RGBG	GLCM	ANN, RF, KNN	KCRC-16	5000	4500	500	H&E	Acc 98.7 Sen 98.6 Spe 99.0 Pre 98.9
2022	[45]	Resize, BGR2RGB, Normalization,	Deep Features	EC, Hybrid, LR, LGB, MLP, RF, SVM, XGB, Voting	LC25000	2800	10-fold cross-validation		H&E	Acc 100.0
2022	[46]	ROI	FOS, GLCM, GLDM, GLRLM, GLSZM, LoG, NGTDM, Shape, WT	MLR	Private	276	194	82	CECT	Acc 76.0 Sen 65.0 Spe 80.0 Pre 54.0 NPV 86.0
2022	[61]	UM-SN	HIM, GLCM, Statistical	LDA, MLP, RF, SVM, XGB, LGB	LC25000	1000	900	100	H&E	Acc 99.3 Sen 99.5 Pre 99.5 F1 99.5
2022	[26]	---	Color Spaces, Haralick	ANN, DT, KNN, QDA, SVM	KCRC-16	5000	3504	1496	H&E	Acc 97.3 Sen 97.3 Spe 99.6 Pre 97.4
2023	[62]	Filtering, linear Transformation, normalization	Color characteristic, DBCM, SMOTE	CatBoost, DT, GNB, KNN, RF	NCT-CRCHE-7K	12,042	8429	3613	H&E	Acc 90.7 Sen 97.6 Spe 97.4 Pre 90.6 Rec 90.5 F1 90.5
2023	[51]	---	Clinical, FEViT	SEKNN	Private	1729	tenfold cross-validation		ENI	Acc 94.0 Sen 74.0 Spe 98.0 AUC 93.0
2023	[47]	Lightness space, RGB to HSV	dResNet	DSVM	KCRC-16	5000	4000	1000	H&E	Acc 98.8
					NCT-CRC-HE-100 K	100,000	80,003	19,997	H&E	Acc 99.8
2023	[63]	HOG, RGBG, Resizing	Morphological	SVM	Private	540	420	120	ENI	Acc 97.5

* Not given in the paper, calculated from the result table, bold font signifies the best model in the ‘Techniques’ column. Abbreviations: BGR2RGB, Blue-Green-Red to Red-Green-Blue; BmzP, Binning of m/z Points; catBoost, Categorical Boosting; CECT, Contrast-Enhanced CT; CSQ, Color Space Quantization; DBCM, Differential Box Count Method; DSVM, Deep Support Vector Machine; dResNet, Dilated ResNet; DRR, Dynamic Range Reduction; DSVM, Deep Support Vector Machine; ENI, Endomicroscopy Images; FEViT, Feature Ensemble Vision Transformer; FOS, First-Order Statistics; GFD, Generic Fourier Descriptor; GNB, Gaussian Naive Bayes; GLDM, Gray-Level Dependence Matrix; GLHS, Gray Level Histogram Statistics; GLSZM, Gray Level Size Zone Matrix; GNB, Gaussian Naive Bayes; HOG, Histogram of Oriented Gradients; HOS, Higher-Order Statistic; HIM, Hu Invariants Moments; HSI, Hyperspectral Imaging; HSV, Hue-Saturation-Value; LBP, Local Binary Pattern; LDA, Linear Discriminant Analysis; LGB, Light Gradient Boosting; LoG, Laplacian of Gaussian; LSSVM, Least Square Support Vector Machine; MLR, Multivariate Logistic Regression; NGTDM, Neighboring Gray Tone Difference Matrix; NSCT, Non-Subsampled Contourlet Transform; OMIS, Optomagnetic Imaging Spectroscopy; QDA, Quadratic Discriminant Analysis; SEKNN, Subspace Ensemble K-Nearest Neighbor; THN, TopHat and Normalization; UMSN, Unsharp Masking and Stain Normalization; VTF, Vector Texture Features; VTI, Vector Texture Images; WLD, Wavelength Difference; WLI, White Light Imaging; WPT, Wavelet Packet Transform; WSVMCS, Wavelet Kernel SVM with Color Histogram; XGB, Extreme Gradient Boosting.

The analysis of colorectal cancer detection using traditional machine learning techniques reveals a notable disparity in model performance across various crucial metrics, showcasing substantial discrepancies between the models with the highest and lowest values as shown in Figure 10. The most proficient model achieved an extraordinary accuracy of 100.0%, whereas the least effective model achieved an accuracy of 76.0%, resulting in a substantial difference of 24.0%. When considering sensitivity, the top-performing model reached an impressive 99.5%, whereas the lowest-performing model registered a mere 65.0%, leading to a remarkable disparity of 34.5%. Similarly, concerning specificity, the superior model attained 99.6%, while the inferior model managed only 80.0%, resulting in a significant difference of 19.6%. In terms of precision, the best model demonstrated 99.5%, while the worst model exhibited a precision of only 54.0%, resulting in a substantial difference of 45.5%. When examining the F1-score, the model with the highest performance achieved 99.5%, whereas the least proficient model attained a score of 63.2%, yielding a notable difference of 36.3%. Lastly, in the case of the area under the curve (AUC), the top model achieved a score of 96.2%, while the bottom model scored 76.0%, marking a significant difference of 20.2%. These conspicuous differences underscore the pivotal role of choosing appropriate machine learning techniques and feature sets in the effectiveness of colorectal cancer detection. Effective cancer detection has far-reaching implications, influencing not only patient outcomes but also the operational efficiency of healthcare systems and the allocation of valuable medical resources.

3.2. Analysis of Gastric Cancer Prediction

Table 4 meticulously encapsulates 16 distinct studies conducted within the temporal frame of 2018 to 2023, each ardently devoted to machine learning-based gastric cancer detection. These investigations collectively underscore the pivotal role of preprocessing in elevating the accuracy of stomach cancer detection models. Notably, the pinnacle of achievement in this realm reached a remarkable 100.0% accuracy, whereas the lowest point stood at 71.2%. This diverse spectrum of performance underscores the profound influence of preprocessing techniques, spanning resizing, filtering, cropping, and color enhancement. These preprocessing strategies, in harmony with segmentation, feature extraction, and the adept utilization of machine learning algorithms encompassing SVM, MLP, RF, and KNN, have collectively converged to engender a triumphant era of stomach cancer detection. This progress extends across diverse modalities such as endoscopy, CT, MRI, and histopathology images. The quantity of images, patients, or slices underpinning these studies spanned a substantial range, from 30 to a staggering 245,196. It is intriguing to note that the enigmatic “Private” dataset emerged as the most recurrently harnessed resource in this insightful analysis.

The research conducted by (Ayyaz et al., 2022) [64] achieved outstanding results in stomach cancer detection, with a remarkable accuracy of 99.80%. They employed various preprocessing techniques, including resizing, contrast enhancement, binarization, and filtering. However, the segmentation method used was not specified in the study. Feature extraction was carried out with deep learning models like VGG19 and AlexNet. For classification, they used multiple techniques such as DT, NB, KNN, SVM, and more. Among these, the cubic SVM model performed the best, achieving an accuracy of 99.80%. This model also had a high sensitivity, precision, F1-score, and an AUC of 100.0%. On the other hand, the study conducted by (Mirniaharikandehei et al., 2021) [65] achieved comparatively lower performance in stomach cancer detection, with an accuracy of 71.20%. Their preprocessing techniques involved filtering and ROI selection, and they utilized the HTS segmentation method. Feature extraction was done using radiomics features such as GLRLM, GLDM, and WT LoG. The classification was carried out using various machine learning models, including SVM, LR, RF, DT, and GBM. The worst-performing model in their analysis was GBM, with an accuracy of 71.20%. This model had lower sensitivity but a higher specificity, precision, and F1-score. (Hu et al., 2022) [66] conducted a stomach cancer detection study with a large dataset of 245,196 medical images. They used various preprocessing techniques, including ROI selection, cropping, filtering, rotation, and disruption. The study extracted features such as color histograms, LBP, and GLCM. For classification, they applied RF and LSVM classifiers, achieving an accuracy of 85.99%. RF was the best-performing model in their analysis. On the other hand, (Naser and Zeki 2021) [67] conducted a stomach cancer detection study with a smaller dataset of only 30 medical images. They applied DIFQ-based preprocessing techniques, and their study used FCM for classification and achieved an accuracy of 85.00%. Table 4 provides an overview of different machine learning-based techniques for stomach (gastric) cancer detection, encompassing 16 reviewed studies. Notably, three of these studies specifically, namely, (Korkmaz and Esmeray 2018) [68], (Nayyar et al., 2021) [69], and (Hu et al., 2022a) [70], opted not to employ any preprocessing techniques. Surprisingly, they achieved noteworthy accuracies of 87.77%, 99.8%, and 85.24%, respectively. This demonstrates the potential for effective stomach cancer detection even in the absence of preprocessing methods. However, it is essential to highlight that a significant portion of the studies examined in the table chose to implement various preprocessing techniques, including CEI, filtering, resizing, Fourier transform, cropping, ROI selection, rotation, disruption, binarization, augmentation, and RSA. These preprocessing steps underscore their pivotal role in enhancing the performance of machine learning models for stomach cancer detection.

Out of the 16 studies focused on gastric cancer detection, 50% of them (8 studies) achieved an accuracy rate of over 90%, indicating highly accurate results. However, the other 50% of the studies received less than 90% accuracy. This discrepancy in performance might be attributed to the utilization of private datasets in these studies. Private datasets may not undergo the same level of processing or standardization as publicly available datasets, potentially leading to variations in data quality and affecting the performance of the machine learning models.

Table 4. Performance comparison of traditional ML-based gastric cancer prediction methods.

Year	References	Preprocessing	Features	Techniques	Dataset	Data Samples	Train Data	Test Data	Modality	Metrics (%)
2018	[71]	Fourier transform	BRISK, SURF, MSER	DT, DA	Private	180	90	90	H&E	Acc 86.7
2018	[72]	Resizing	LBP, HOG	ANN, RF	Private	180	90	90	H&E	Acc 100.0
2018	[68]	---	SURF, DFT	NB	Private	180	90	90	H&E	Acc 87.8
					Private	720	360	360	H&E	Acc 90.3
2018	[73]	CEI, filtering, resizing	GLCM	SVM	Private	207	126	81	NBI	Acc 96.3 Sen 96.7 Spe 95.0 Pre 98.3
2019	[74]	Resizing, cropping	GLCM, Shape, FOF, GLSZM	SVM	Private	490	326	164	CT	Acc 71.3 Sen 72.6 Spe 68.1 Pre 82.0 NPV 50.0
2021	[67]	DIFQ	SMI	FCM, KMC	Private	30	---	---	MRI	Acc 85.0
2021	[75]	Resizing	Extract HOG	RF, MLP	Private	180	90	90	H&E	Acc 98.1
2021	[76]	Resizing	TSS	BP, BPSVM, SVM	Private	78	---	---	MRI	Acc 94.6
2021	[69]	---	Deep Features	CSVM, Bagged Trees, KNNs, SVMs	Private	4000	2800	1200	WCE	Acc 99.8 Sen 99.0 Pre 99.3 F1 99.1 AUC 100
2021	[65]	Filtering, ROI	LoG, WT, GLDM, GLRLM	GBM, DT, RF, LR, SVM.	Private	159	Leave-One-Out cross-validation		CT	Acc 71.2 Sen 43.1 Spe 87.1 Pre 65.8
2022	[77]	Augmentation, resizing, filtering	InceptionNet, VGGNet	SVM, RF, KNN.	HKD	10,662 (47,398 Augmneted)	37,788	9610	Endoscopy	Acc 98.0 Sen 100 Pre 100 F1 100 MCC 97.8
2022	[70]	---	GLCM, LBP, HOG, histogram, luminance, Color histogram	NSVM, LSVM, LR, NB, RF, ANN, KNN	GasHisSDB	245196	196,157	49,039	H&E	Acc 85.2 Sen 84.9 # Pre 84.6 # Spe 84.9 # F1 84.8 #
2022	[64]	Binarization, CEI, filtering, resizing	VGG19 Alexnet	Bagged Tree, Coarse Tree, CSVM, CKNN, DT, Fine Tree, KNN, NB	Private	2590	10-fold cross-validation		EUS	Acc 99.8 Sen 99.8 Pre 99.8 F1 99.8 AUC 100
2022	[66]	Cropping, disruption, filtering, ROI, Rotation	Color histogram, GLCM, LBP	LSVM, RF	GasHisSDB	245,196	196,157	49,039	H&E	Acc 85.9 Sen 86.2 # Spe 86.2 # Pre 85.7 # F1 85.9 #
2023	[78]	Augmentation, CEI	MobileNet-V2	Bayesian, CSVM, LSVM, QSVM, Softmax	KV2D	4854	10-fold cross-validation		Endoscopy	Acc 96.4 Pre 97.6 Sen 93.0 F1 95.2
2023	[79]	RSA	RSF	PLS-DA, LOO, SVM	Private	450	Leave-One-Out cross validation		H&E	Acc 94.8 Sen 91.0 Spe 100 AUC 95.8

# Calculated by averaging the normal and abnormal class, Bold Font techniques represent the best model. Abbreviations: BPSVM, Binary Robust Invariant Scalable Keypoints; BRISK, Binary Robust Invariant Scalable Keypoints; CKNN, Cosine K-Nearest Neighbor; CSVM, Cubic SVM; DA, Discriminant Analysis; DIFQ, Dividing an image into four quarters; FCM, Fuzzy C-Means; GGF, Global Graph Features; HOG, Histogram of Oriented Gradients; HTSS, Hybrid Tumor Segmentation; KMC, K-Means Clustering; LOO, Leave-One-Out; LSVM, Linear Support Vector Machine; MSER, Maximally Stable Extremal Regions; NSVM, Non-Linear Support Vector Machine; OAT, Otsu Adaptive Thresholding; PLS-DA, Partial Least-Squares Discriminant Analysis; QSVM, Quadratic SVM; RSA, Raman Spectral Analysis; RSF, Raman Spectral Feature; SM, Seven Moments Invariants; SMI, Seven Moments Invariants; SURF, Speeded Up Robust Features; TSS, Tumor Scattered Signal.

Table 4. Performance comparison of traditional ML-based gastric cancer prediction methods.

Year	References	Preprocessing	Features	Techniques	Dataset	Data Samples	Train Data	Test Data	Modality	Metrics (%)
2018	[71]	Fourier transform	BRISK, SURF, MSER	DT, DA	Private	180	90	90	H&E	Acc 86.7
2018	[72]	Resizing	LBP, HOG	ANN, RF	Private	180	90	90	H&E	Acc 100.0
2018	[68]	---	SURF, DFT	NB	Private	180	90	90	H&E	Acc 87.8
					Private	720	360	360	H&E	Acc 90.3
2018	[73]	CEI, filtering, resizing	GLCM	SVM	Private	207	126	81	NBI	Acc 96.3 Sen 96.7 Spe 95.0 Pre 98.3
2019	[74]	Resizing, cropping	GLCM, Shape, FOF, GLSZM	SVM	Private	490	326	164	CT	Acc 71.3 Sen 72.6 Spe 68.1 Pre 82.0 NPV 50.0
2021	[67]	DIFQ	SMI	FCM, KMC	Private	30	---	---	MRI	Acc 85.0
2021	[75]	Resizing	Extract HOG	RF, MLP	Private	180	90	90	H&E	Acc 98.1
2021	[76]	Resizing	TSS	BP, BPSVM, SVM	Private	78	---	---	MRI	Acc 94.6
2021	[69]	---	Deep Features	CSVM, Bagged Trees, KNNs, SVMs	Private	4000	2800	1200	WCE	Acc 99.8 Sen 99.0 Pre 99.3 F1 99.1 AUC 100
2021	[65]	Filtering, ROI	LoG, WT, GLDM, GLRLM	GBM, DT, RF, LR, SVM.	Private	159	Leave-One-Out cross-validation		CT	Acc 71.2 Sen 43.1 Spe 87.1 Pre 65.8
2022	[77]	Augmentation, resizing, filtering	InceptionNet, VGGNet	SVM, RF, KNN.	HKD	10,662 (47,398 Augmneted)	37,788	9610	Endoscopy	Acc 98.0 Sen 100 Pre 100 F1 100 MCC 97.8
2022	[70]	---	GLCM, LBP, HOG, histogram, luminance, Color histogram	NSVM, LSVM, LR, NB, RF, ANN, KNN	GasHisSDB	245196	196,157	49,039	H&E	Acc 85.2 Sen 84.9 # Pre 84.6 # Spe 84.9 # F1 84.8 #
2022	[64]	Binarization, CEI, filtering, resizing	VGG19 Alexnet	Bagged Tree, Coarse Tree, CSVM, CKNN, DT, Fine Tree, KNN, NB	Private	2590	10-fold cross-validation		EUS	Acc 99.8 Sen 99.8 Pre 99.8 F1 99.8 AUC 100
2022	[66]	Cropping, disruption, filtering, ROI, Rotation	Color histogram, GLCM, LBP	LSVM, RF	GasHisSDB	245,196	196,157	49,039	H&E	Acc 85.9 Sen 86.2 # Spe 86.2 # Pre 85.7 # F1 85.9 #
2023	[78]	Augmentation, CEI	MobileNet-V2	Bayesian, CSVM, LSVM, QSVM, Softmax	KV2D	4854	10-fold cross-validation		Endoscopy	Acc 96.4 Pre 97.6 Sen 93.0 F1 95.2
2023	[79]	RSA	RSF	PLS-DA, LOO, SVM	Private	450	Leave-One-Out cross validation		H&E	Acc 94.8 Sen 91.0 Spe 100 AUC 95.8

# Calculated by averaging the normal and abnormal class, Bold Font techniques represent the best model. Abbreviations: BPSVM, Binary Robust Invariant Scalable Keypoints; BRISK, Binary Robust Invariant Scalable Keypoints; CKNN, Cosine K-Nearest Neighbor; CSVM, Cubic SVM; DA, Discriminant Analysis; DIFQ, Dividing an image into four quarters; FCM, Fuzzy C-Means; GGF, Global Graph Features; HOG, Histogram of Oriented Gradients; HTSS, Hybrid Tumor Segmentation; KMC, K-Means Clustering; LOO, Leave-One-Out; LSVM, Linear Support Vector Machine; MSER, Maximally Stable Extremal Regions; NSVM, Non-Linear Support Vector Machine; OAT, Otsu Adaptive Thresholding; PLS-DA, Partial Least-Squares Discriminant Analysis; QSVM, Quadratic SVM; RSA, Raman Spectral Analysis; RSF, Raman Spectral Feature; SM, Seven Moments Invariants; SMI, Seven Moments Invariants; SURF, Speeded Up Robust Features; TSS, Tumor Scattered Signal.

The analysis of gastric cancer detection reveals substantial variations in model performance across key metrics, with significant differences observed between the highest and lowest values as shown in Figure 11. Accuracy (Acc) showcased a noteworthy contrast, with the best-performing model achieving a flawless 100.00% and the least effective model scoring 71.20%. This substantial 28.80% difference underscores the pivotal role of model selection in achieving accurate gastric cancer detection. Sensitivity (Sen) displayed a considerable gap, with the top model achieving a perfect 100.00%, while the lowest model only reached 43.10%. This marked difference of 56.90% emphasizes the necessity of sensitive detection techniques in identifying gastric cancer. Similarly, specificity (Spe) followed suit, with the highest model reaching 100.00% and the lowest model achieving 68.10%. The substantial 31.90% difference highlights the importance of correctly identifying non-cancer cases in diagnostic accuracy. Precision (Pre) also exhibited a significant disparity, with the best model achieving 100.00%, and the least effective model achieving 65.80%. The difference of 34.20% underscores the significance of precise identification of gastric cancer cases. It is noteworthy that the negative predictive value (NPV) remained constant at 50.00% for both the highest and lowest models, signifying that neither model excelled in ruling out non-cancer cases. However, since NPV is only used in a single article, its impact on the overall analysis may be limited.

Additionally, the F1-score showed a substantial difference, with the top model achieving a perfect 100.00%, while the lowest model reached 84.80%. The 15.20% difference emphasizes the balance between precision and sensitivity in gastric cancer detection. Lastly, in terms of the area under the curve (AUC), the best model achieved a near-perfect 100.00%, while the lowest model attained a still impressive 95.80%. The modest 4.20% difference indicates that both models performed well in distinguishing between gastric cancer and non-cancer cases. It is also worth noting that the area under the curve (AUC) metric was utilized in only three articles, and the differences in AUC were relatively modest. Therefore, the impact of AUC on the overall analysis may be less generalized. These findings underscore the critical role of model choice and feature selection in the effective detection of gastric cancer. Accurate and sensitive diagnostic tools are crucial for improving patient outcomes and optimizing healthcare resources. While NPV and AUC may have a limited impact in this context due to their restricted usage, the other metrics highlight the significance of selecting appropriate models for reliable gastric cancer detection.

4. Proposed Methodology

In this section, we delineate our proposed methodology for the detection of colorectal and gastric cancer through the application of traditional machine learning techniques. These approaches have been meticulously crafted based on the discerning insights and observations gleaned from the comprehensive review tables. Our primary goal is to introduce a Proposed (optimized) approach, accompanied by the most suitable parameters, in order to attain the most superior results. Our endeavor is to provide an efficient, effective, automated, and highly precise technique for the detection of colorectal and gastric cancer.

4.1. Detection of Colorectal Cancer

Figure 12 is a comprehensive visualization of the architectural framework that underpins our proposed model for the detection of colorectal cancer. This blueprint draws its inspiration from the wealth of insights extracted from Table 3, which provides a foundational understanding of the methodologies that have proven effective in this domain. While we have opted to use the H&E modality as an illustrative example, it is imperative to recognize that our model can seamlessly accommodate other modalities. This flexibility is a testament to the adaptability and robustness of our approach, as it allows for the incorporation of diverse data sources to enrich the depth and scope of our analysis. At the crux of our methodology lies the preprocessing phase, an instrumental step that sets the stage for the rigorous examination of input images. Within this phase, we meticulously execute four pivotal steps: Image Enhancement, Pixel Enhancement, RGB-to-Gray Conversion, and Image Segmentation. These sequential operations are not arbitrary but have been thoughtfully selected and implemented to systematically prepare the input images. Their collective objective is to optimize the images, ensuring they are in a suitable form for efficient feature extraction and subsequent in-depth analysis. The realm of feature engineering is where our approach truly shines. Here, we introduce an innovative and nuanced strategy. Instead of relying solely on one type of feature, we merge two distinct categories: deep learning-based features, which are often referred to as “deep features”, and a varied assortment of other features. This assortment includes Discrete Wavelet Transform (DWT), Gray Level Co-occurrence Matrix (GLCM), Local Binary Pattern (LBP), Texture, and Gray Level Size Zone Matrix (GLSZM). The fusion of these diverse feature sets is not a random choice but a deliberate effort to enhance the robustness and comprehensiveness of our analysis. This fusion is designed to ensure that our model captures both the intricate, high-level representations obtained through deep learning and handcrafted features meticulously tailored to highlight specific aspects of tumor characteristics. By incorporating these different types of features, our model becomes versatile, capable of effectively identifying patterns and characteristics in the data that may not be discernible when using only one type of feature. By executing this innovative approach, we aim to enhance the model’s ability to interpret and understand the complex information contained within medical images. This, in turn, contributes to the accuracy and efficiency of colorectal cancer detection. Furthermore, it enables our model to adapt and excel in different scenarios and datasets, making it a powerful tool for healthcare professionals and researchers working in the field of cancer detection.

The combination of these diverse features enhances the model’s capability to encompass both intricate, high-level representations acquired through deep learning and meticulously tailored handcrafted features that accentuate distinct tumor characteristics. Moving forward in the workflow, we encounter the crucial stages of feature selection and optimization. This pivotal process serves a dual role: it reduces feature redundancy while enhancing the overall model performance by focusing on the most distinctive attributes. Our model evaluation process is underpinned by a rigorous data-partitioning strategy, effectively splitting the dataset into training and testing subsets. The training dataset undergoes additional scrutiny through a k-fold cross-validation approach, fortifying the model’s training and facilitating a robust performance assessment. This approach not only guards against overfitting but also assesses the model’s adaptability to various data scenarios. The test dataset becomes the arena for predicting colorectal cancer, with the cubic support vector machine (SVM) taking the lead in this classification task. The SVM is a formidable presence among traditional machine learning classifiers, known for its prowess in handling high-dimensional data and executing binary classification tasks, making it ideally suited for the intricacies of cancer detection. In summary, our proposed model architecture harmoniously integrates advanced image preprocessing techniques, innovative feature-engineering methodologies, and the proven machinery of a traditional machine learning classifier. This synthesis yields an efficient and accurate framework for colorectal cancer detection. Pending further validation and testing on diverse datasets, this approach has the potential to revolutionize early cancer detection and diagnosis, potentially leading to improved patient outcomes and a transformation in healthcare effectiveness.

4.2. Detection of Gastric Cancer

The system architecture flow diagram, as depicted in Figure 13, outlines our comprehensive and adaptable approach to stomach cancer (Gastric) detection employing traditional machine learning classifiers. Informed by the top-performing models scrutinized in Table 4, our proposed architecture is intentionally crafted to accommodate both endoscopy video datasets, which have gained prominence in recent years, and static image datasets. Initiating with endoscopy video datasets as the primary data source, our architecture seamlessly extends its capabilities to image datasets by extracting individual frames from the video sequences. Subsequently, these extracted frames undergo preprocessing, which encompasses various techniques such as noise reduction, RGB-to-grayscale conversion, or other pertinent methods contingent on the specific application and dataset attributes. Acknowledging the potential constraint of limited video datasets, we introduce data augmentation techniques as part of our solution. This augmentation process generates an ample supply of augmented image datasets, enabling the model to undergo training on a more diverse and representative set of samples. This augmentation strategy empowers the model to generalize better, ultimately leading to enhanced performance outcomes. Moving into the feature extraction phase, we advocate the simultaneous use of deep features and texture-based features. Deep features are sourced from state-of-the-art deep learning models, while texture-based features encompass attributes like GLCM, GLRLM, and GLSZM, harnessed through conventional feature extraction methods. This fusion of diverse feature types ensures that the model possesses the capability to encapsulate both abstract high-level representations and the specific characteristics embedded in the stomach cancer data.

Upon the amalgamation of these features, the subsequent step in our approach involves feature optimization. Here, we employ well-suited algorithms to meticulously select the most pertinent attributes among the fused features. This optimization process serves a dual function: firstly, it mitigates the peril of overfitting, a common pitfall in machine learning endeavors, and secondly, it bolsters the overall efficiency of the model. The carefully curated selection of features enhances the model’s capacity to discriminate between different classes, resulting in improved classification accuracy. Following the optimization phase, the dataset undergoes a deliberate partitioning into two distinct subsets: the training set and the testing set. This partitioning is a strategic maneuver that ensures the robust training and rigorous evaluation of traditional machine learning classifiers. The distribution of the dataset is thoughtfully orchestrated to prevent any data leakage and to create a reliable foundation for our model’s assessment. Depending on the specific nature of the classification task and the unique requirements of the application, we employ a range of classifiers known for their effectiveness in various scenarios. These classifiers include but are not limited to support vector machines (SVM), Random Forest (RF), logistic regression (LR), backpropagation neural networks (BPNN), and artificial neural networks (ANN). Each of these classifiers is chosen judiciously to cater to the specific characteristics of the dataset and the intricacies of the task at hand. These classifiers excel in categorizing stomach cancer into distinct types, thereby providing valuable insights essential for accurate diagnosis and tailored treatment. A standout feature of our proposed system architecture is its inherent adaptability. This architectural flexibility empowers the system to seamlessly accommodate both image and video datasets, thereby rendering it versatile and suitable for a wide spectrum of applications. By harnessing the capabilities of traditional machine learning methods and integrating the novel approaches of feature fusion and optimization, our system architecture exhibits substantial potential for delivering heightened efficiency and heightened accuracy in the realm of stomach cancer detection. Nonetheless, it is imperative to emphasize the essentiality of conducting further validation and in-depth evaluation of our system’s performance.

4.3. Key Observations

The comprehensive assessment of colorectal and gastric cancer detection techniques using traditional machine learning methods and medical image datasets has revealed several key insights:

• Dataset Diversity: Evaluation includes colorectal and gastric cancer datasets, ranging from 30 to 100,000 images. The varied dataset sizes showcase machine learning classifier effectiveness with appropriate tuning.
• Exceptional Model Performances: Models achieve 100% accuracy for both colorectal and gastric cancer, with perfect scores in key metrics like sensitivity, specificity, precision, and F1-score, showcasing the potential of traditional ML classifiers with optimal parameters.
• Preprocessing Techniques: Researchers employ various preprocessing techniques, including image filtering, denoising, wavelet transforms, RGB-to-gray conversion, normalization, cropping (ROI), sampling, and binarization, to optimize model performance and minimize biases during data manipulation.
• Literature Review Significance: This analysis spans 36 literature sources related to colorectal and gastric cancer, underscoring the significant interest in cancer detection through traditional ML classifiers. Researchers have explored an extensive range of cancer types, diverse evaluation metrics, and datasets, collectively advancing the field.
• Dominant Traditional ML Techniques: SVM is a commonly used traditional ML classifier in cancer detection tasks, emphasizing the need to understand each classifier’s strengths and limitations for optimal selection.
• Insightful Dataset and Feature Analysis: Reviewed studies predominantly utilized benchmark medical image datasets, with researchers employing feature extraction techniques like GLCM for informative feature extraction in cancer detection.
• Prudent Model Architecture Design: Optimal results in cancer detection require thoughtful and optimized model architectures, which can enhance accuracy, generalizability, and interpretability, addressing challenges in medical image analysis.

4.4. Key Challenges and Future Scope

Traditional ML classifiers have shown remarkable potential in cancer detection. However, several challenges and the future scope in their application have been identified:

• Variability in Accuracy: Traditional ML classifiers exhibit variable accuracy rates across cancer types, ranging from 76% to 100%. Overcoming these variations poses a challenge, underscoring the need for enhanced models. Future research should prioritize refining models for consistent and accurate performance across diverse cancer types.
• Metric Disparities: Metric variations, especially in sensitivity (43.1% to 100%) for gastric cancer, suggest potential data imbalance challenges. Addressing these issues is crucial for accurate model assessments. Future research should focus on developing strategies to handle imbalanced data and improve model robustness.
• Preprocessing Challenges: Balancing raw and preprocessed data is crucial to ensure input data quality and reliability, contributing to robust cancer detection model performance. Future research should explore advanced preprocessing techniques and optimization methods to further enhance model robustness.
• Limited use of evaluation metrics: Limited use of metrics like NPV, AUC, and MCC in the reviewed literature highlights the challenge of comprehensive model assessment. Addressing this limitation and exploring a broader range of metrics is crucial for future research to enhance understanding and effectiveness in cancer detection tasks.
• Generalizing to novel cancer types: The literature primarily focuses on colorectal and gastric cancers, posing a challenge for extending traditional ML classifiers to less-explored cancer types. Future research should aim to develop versatile ML models with robust feature extraction techniques to adapt to diverse cancer types and domains.
• Addressing overfitting and model selection: The diversity in ML classifiers poses challenges in model selection for specific cancers, emphasizing the need for careful evaluation to avoid overfitting. Future research should focus on refining model selection strategies to enhance the robustness of cancer detection techniques and improve diagnostic accuracy.

5. Conclusions

In this manuscript, a thorough review and analysis of colorectal and gastric cancer detection using traditional machine learning techniques are presented. We have meticulously scrutinized 36 research papers published between 2017 and 2023, specifically focusing on the domain of medical imaging datasets for detecting these types of cancers. Mathematical formulations elucidating frequently employed preprocessing techniques, feature extraction methods, traditional machine learning classifiers, and assessment metrics are provided. These formulations offer valuable guidance to researchers when selecting the most suitable techniques for their cancer detection studies. To conduct this analysis, a range of criteria such as publication year, preprocessing methods, dataset particulars, image quantities, modality, techniques, best models, and metrics (%) were considered. An extensive array of metrics was employed to evaluate model performance comprehensively. Notably, the study delves into the highest and lowest metric values and their disparities, highlighting opportunities for enhancement. Remarkably, we found that the highest achievable value for all metrics reached an astonishing 100%, with gastric cancer detection registering the lowest sensitivity at 43.10%. This underscores the potential of traditional ML classifiers, while indicating areas for further refinement. Drawing from these insights, we present a proposed (optimized) methodology for both colorectal and gastric cancer detection, aiding in the selection of an optimized approach for future cancer detection research. The manuscript concludes by delineating key findings and challenges that offer valuable directions for future research endeavors.

In our future research endeavors, we plan to implement the proposed optimized methodology for the detection of colorectal and gastric cancer within the specified experimental framework. This proactive approach aligns with our commitment to enhancing the effectiveness of cancer detection methodologies. Furthermore, we will conscientiously incorporate and address the challenges and limitations identified in this study, ensuring a comprehensive and iterative improvement in our investigative efforts.