Recent Advances in Machine Learning Applied to Ultrasound Imaging

Abstract

Machine learning (ML) methods are pervading an increasing number of fields of application because of their capacity to effectively solve a wide variety of challenging problems. The employment of ML techniques in ultrasound imaging applications started several years ago but the scientific interest in this issue has increased exponentially in the last few years. The present work reviews the most recent (2019 onwards) implementations of machine learning techniques for two of the most popular ultrasound imaging fields, medical diagnostics and non-destructive evaluation. The former, which covers the major part of the review, was analyzed by classifying studies according to the human organ investigated and the methodology (e.g., detection, segmentation, and/or classification) adopted, while for the latter, some solutions to the detection/classification of material defects or particular patterns are reported. Finally, the main merits of machine learning that emerged from the study analysis are summarized and discussed.

1. Introduction

In recent years, machine learning (ML) techniques, which are based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention, have had a significant impact on industry and science due to their capacity to solve challenging problems. ML applications are innumerable and include computer vision [1,2,3,4], self-driving cars [5,6], virtual person assistants [7], speech recognition [8] and even ultrasound imaging, which is the field of application that will be discussed in detail in this review.

Ultrasound imaging is employed in a wide variety of applications, including sonar [9,10,11], non-destructive evaluation (NDE) [12,13,14], indoor positioning systems (IPS) [15,16] and biometric recognition [17,18,19]. However, its best known application is medical diagnostics, where it has been found to be very attractive and interesting compared to other imaging modalities, such as magnetic resonance (MR), X-rays and computed tomography(CT), because it enables the acquisition of organ images at low cost and in a safe and non-invasive way. At present, it is commonly employed for various analyses, such as fetal monitoring, the anatomical study of blood vessels and flow, lung and liver disease screening and even the diagnosis of tumor pathologies.

The employment of ML techniques in ultrasound imaging has occurred for several years [20,21,22,23,24], but, very recently, there has been a dramatic growth of scientific interest in this issue, where the main fields involved are sonar, NDE and medicine. The present review focuses on the latter two fields, with particular regard to the medical area. Sonar applications are not included here because, due to the very huge quantity of recent studies [25,26,27,28,29,30], they deserve a separate review. Given the very large number of relevant scientific publications, this review is mainly devoted to the analysis of more recent studies, with a particular focus on papers published after 2019 and on journal papers rather than conference papers.

The review is organized as follows: The Section 2 is devoted to a classification of the most used ML algorithms in the analyzed articles; the Section 3 is focused on the basic principles of ultrasound imaging and the most used ultrasound techniques; the Section 4 reviews papers related to applications of ML to ultrasound medical diagnostics, sub-divided by organs; the Section 5 considers the application of ML algorithm in NDE; the Section 6 is devoted to the conclusions, where the benefits and limits of ML applications are highlighted.

2. Machine Learning

Machine learning (ML), one of the most rapidly developing subfields of artificial intelligence (AI) research, is defined as the field of study of computer algorithms capable of learning autonomously to automatically improve the performance of a task on the basis of their own previous experience. The main intention of ML techniques is to allow the system to obtain knowledge with no explicit programming and to learn from the data. A great deal of scientific activity has been dedicated to proposing methods to enable machines to learn by themselves without being explicitly programmed [31]. ML relies on different algorithms to solve data problems, where the kind of algorithm used depends on a series of factors, including the type of problem, the number of variables, and the most suitable model. ML fundamentally includes two types of approaches: supervised and unsupervised learning. Supervised learning is a machine learning method where an algorithm is trained on input data that has been labeled for a particular output. Supervised algorithms, depending on the output to be obtained, can be categorized as classification algorithms, which involve identifying the category to which an object belongs on the basis of the characteristic of the object itself, and regression algorithms, which involve estimating the value of a particular feature of an object. The most used algorithms for regression problems are:

• Linear regression, which establishes a relation between dependent variables (output) and independent variables (input) through a fitting line. Linear regression can be of two types: simple and multiple linear regression, where the first includes one independent variable, while the second includes two or more independent variables.
• Logistic regression, which is a statistical tool used to model a binomial result, i.e., binary problems with one or more explanatory variables. In this way, it is possible to describe the data and the relationship between a binary dependent variable and one or more independent variables.

The most used algorithms for classification problems are listed below.

• Naive Bayes is based on an underlying probabilistic model and enables capture of the uncertainty of a model by determining the probabilities of the outcomes. The Bayesian classification can solve predictive problems by providing practical learning algorithms and combining observed data. This classification provides a useful perspective for understanding and evaluating learning algorithms [32].
• Support vector machine (SVM) is a method for the classification of two groups of data points, which exploits a hyperplane that divides two categories of data points with the largest margin [33]. Linear SVM is the most simple form of an SVM classifier, where examples are represented as points in space and mapped out so that the examples belonging to two different categories are divided by a clear gap that maximizes the difference. The prediction of the category to which examples belong is made on the basis of the side on which they fall. SVM can be performed either linearly or non-linearly. Non-linear SVM results are useful when data are not separable linearly. This approach involves implementation of a kernel trick [31], a non-linear function which replaces the scalar product, thus maximizing the hyperspace. The most used kernels are polynomial and Gaussian.
• Decision tree, the goal of which is to create a model that predicts the value of a certain variable by learning some decision rules obtained from data features. It consists of a tree that classifies instances by sorting them based on feature values; each node of the decision tree is an instance feature to be classified and each branch corresponds to a value that can be assigned to the node. Basically, the procedure consists of classifying the instances and sorting them on their feature values [34].
• Random forest (RF) combines the output of multiple decision trees to reach a single result. The random forest algorithm is an extension of the bagging method because it utilizes both bagging and feature randomness to create an uncorrelated forest of decision trees. RF is employed for classification, regression, and other activity based on the construction of a multitude of decision trees during the training and generates the class that represents the overall prediction of the single trees. Random forest corrects the overfitting problem of decision trees [35].
• K-nearest neighbors (K-NN) aims to predict a new instance by knowing the data points which are separated into different classes. Specifically, within each class there are associated data points or instances, the set of which defines the data set. Its operation is based on the similarity of the characteristics; that is, the closer an instance is to a data point, the more the algorithm considers them similar. To evaluate the similarity, the algorithm uses some distance, such as Euclidean [36], Chebyshev [37], or Minkowski [38]. In addition to the distance, KNN plans to set a parameter k, chosen in an arbitrary way, which identifies the number of minimum distances; the class that obtains the greatest number of these distances is chosen as a prediction [39].
• Linear discriminant analysis (LDA) is a commonly used technique for supervised classification problems, which aims to reduce their dimensions. It is used for modeling differences in groups, i.e., separating two or more classes. It transfers features from higher to lower dimension spaces. LDA, as SVM, computes optimal hyperplanes with respect to their individual objectives. However, LDA hyperplanes are optimal only when the covariance matrices are identical for all of the classes, while SVM computes optimal hyperplanes without making an assumption [40].

With unsupervised learning, it is necessary to identify hidden structures in datasets, without outputs being labeled. The principal unsupervised learning algorithms are:

• K-means clustering, which initially defines k centroids and iteratively selects the closest data points to each centroid and assigns them to the centroid itself [33].
• Principal component analysis (PCA) reduces the dimensionality of a data set consisting of many variables correlated with each other, either heavily or lightly, while retaining the variation present in the dataset, to the maximum extent. The same operation is performed by converting the original variables to a new set of variables, named principal components, which are orthogonal and are ordered in such a way that the retention of variation present in the original variables decreases by moving down the order [41].

Deep learning (DL) is a subfield of machine learning, which attempts to learn high-level abstractions in data automatically using hierarchical architectures [42]. The most popular DL techniques are based on neural networks, which are inspired by the human nervous system and the structure of the brain. It consists of processing units or nodes organized in input, hidden, and output layers where the nodes in each layer are connected to other nodes in adjacent layers. The inputs are multiplied by the respective weights and summed at each node. The sum then undergoes a transformation based on the activation function, which, in most cases, is a sigmoid function, tanh, or rectified linear unit (reLU). The output of the function is then fed as input to the subsequent unit in the next layer. Finally, the result of the final output represents the solution to the problem. The principal type of neural network is the convolutional neural network (CNN). Fundamentally, a CNN consists of a series of convolutional layers, sub-sampling or pooling layers, fully connected layers, and a normalizing layer. In this case, instead of using the activation functions described above, convolution and pooling functions are employed as activation functions. The series of convolutions perform increasingly more refined feature extraction at every layer moving from the input to the output layers. Pooling layers occur between each convolution layer and reduce the feature maps and the size of network and then effectively reduce the feature network’s susceptibility to scale and distortion of the image [43]. Finally, the classification is performed by fully connected layers through a certain number of categories. The most popular CNN types are reported below.

UNet [44] is a CNN developed for biomedical image segmentation. The UNet architecture has a U-shape and is based on two paths, a contraction or encoder path and an expansion or decoder path, where each encoder convolution layer is concatenated to its reciprocal decoder layer. Each concatenation provides a localized feature specific for segmented classes. The basic concept behind the Unet deep learning technique is to use convolutional layers and max-pooling architectures to extract identifying features and patterns from a series of images.

Alexnet [45] is a CNN consisting of eight layers, where the first five are convolutional layers and the last three are fully connected layers. Compared to traditional CNNs, AlexNet identifies more features because it consists of a deep structure and has many parameters in the model.

ResNet [46] is a specific architecture of CNN distinguished by its employment of a skip connection to jump over some layers. The adding of a skip connection is useful because it mitigates the problem of degradation (accuracy saturation) that occurs when a high number of layers are used, leading to training errors. Figure 1 presents a scheme for the algorithms described above.

Traditional ML is based on training data and testing data by considering the same input feature space and same data distribution. However, there are some cases where the training data and testing data show different data distributions and the result of the predictive learner can be degraded. The solution entails transfer learning that allows definition of a high-performance learner for a target domain trained from a related source domain, which copes with the difficulty of obtaining training data that matches the feature space and the predicted data distribution characteristics of the test data.

Transfer learning consists mainly of re-employing a pre-developed model to accomplish a determined activity as a starting point for the development of a model destined to execute another different activity. Transfer learning is widely employed in the majority of deep learning models where neural networks, on which a large data set has already been inserted, are retrained with the purpose of classifying images on a large scale. The intuition behind transfer learning, especially in the image classification activity, is that, if a model is trained on a sufficiently large and general data set, it will effectively act as a generic model of the visual world; it will therefore be possible to exploit the general function maps learned, without having to train a new model of neural networks from scratch, wasting resources and time in the training process of the neural network on data sets large enough to be able to return an optimal result.

3. Ultrasound Imaging

One of the most popular ultrasound imaging techniques is the pulse-echo method, where its basic modality is represented by A(Amplitude)-Mode [47]. In this modality, a single-element transducer is excited by high-intensity short pulses through a signal generator. The waves transmitted by the transducer propagate through the body and echoes are reflected from various tissues because of the large interfaces between organs. The echoes are detected by the same transducer and are amplified because ultrasound energy is attenuated by the tissues as it penetrates deeper into the body, and is processed and displayed.

A two-dimensional image is usually obtained by using linear or convex phased arrays, which allow electronic scanning of the desired volume and performance of beam-forming techniques (e.g., apodization, steering, focusing). A first kind of image, named the B(Brightness)-mode image, is characterized by grayscale pixels whose value is proportional to the amplitude of the returned echo [48]. Figure 2 shows an example of a B-mode image of the liver and kidney.

Another method for generating 2D images is represented by M(Motion)-Mode. In this case, the transducer is placed in front of a moving target, and echo signals are repeatedly acquired along the same A-line orientation. The obtained image represents the distances to the targets as a function of time.

In cardiovascular analysis, the most employed techniques are based on the Doppler effect that exploits the ultrasound capability to measure blood flow, principally to assess the state of blood vessels and functions of an organ. The Doppler effect is the alteration of the frequency of a received wave compared to the transmitted one due to the relative movement of a transmitter and a receiver. It is based on the backscattering of ultrasonic waves by the red blood cells in motion with respect to the probe. Conventionally, ultrasound Doppler flow measurements are based on three main approaches: continuous wave (CW), pulsed wave (PW) Doppler, and color Doppler. CW Doppler systems use two transducers, one for transmission and one for reception, and can obtain information on the velocity along a US beam without any information about the position [49]. PW Doppler, instead, is based on a single transducer that alternatively transmits and receives. It is sensitive to the beam-to-flow angle and enables extraction of the flow velocity at one specific depth [50]. the color Doppler is a technique that allows display of B-mode and Doppler blood flow data simultaneously, where the Doppler information is visualized as color and is superimposed onto the B-mode image. In particular, the color red indicates flow toward the transducer while the color blue indicates flow away from the transducer [51,52]. A color Doppler image of the carotid artery is shown in Figure 3.

Another technique employed in the cardiovascular analysis is intravascular ultrasound (IVUS), a catheter-based technique that provides real-time high-resolution tomographic images of both the lumen and arterial wall of coronary segments [53].

A relatively new imaging technique is represented by elastography. This technique has the aim of imitating palpation, one of the oldest methods to detect tumors and other pathologies, using acoustic waves. Basically, a pathologic region is characterized by lower elasticity in comparison to normal tissues—tissue stiffness is considered an important biomarker for pathological processes. In this case, the reflected echoes are used to map reflectivity properties and geometry by assuming that the examined organ is stationary or moves only because of internal physiological changes. Another kind of contrast that represents the elastic properties of the tissue can be obtained by applying a known external mechanical load to the tissue [54].

An important evolution of ultrasound imaging is represented by the formation of three-dimensional images that provide more information than 2D-images. 3D-images are obtained by acquiring multiple slices of the images; the reconstruction of the 3D-image can be performed offline or in real-time. In the latter case, there is reference to 4D-images, i.e., 3D-images in real-time motion. 3D ultrasound data are achieved by employing a linear array performing a single mechanical scan or a two-dimensional array performing only electronic scans. Basically, 3D-images are displayed through two modalities: a series of multiplanar images orthogonal to one another and/or images showing three-dimensional structures [55].

4. ML in US Medical Diagnostics

In recent years, ML techniques have played a fundamental role in the analysis of US medical images in order to improve the reliability of diagnosis that is often compromised by the relatively poor quality of images due to the presence of noise and acquisition errors. Furthermore, ML techniques reduce operator-dependence, standardize the image interpretation, provide stable results and the capability to make rapid decisions, and relieve the heavy work of radiologists.

The next section is subdivided into several subsections; each subsection is devoted to a particular organ and consists of two parts describing:

• the general issues related to organ diseases;
• the most recent papers on innovative ML techniques organized according to the methodology adopted: detection, segmentation and classification.

4.1. Breast

Breast cancer is a disease that represents one of the principal causes of cancer deaths for women and this number is increasing. The probability that a woman will die from a breast tumor is about 1 in 39. Only 10% of cases are detected at the initial stages. Breast cancer can begin in different parts of the breast which is made up of lobules, ducts, and stromal tissues. Most breast cancers begin in the cells that line the ducts, while some begin in the cells that line the lobules and a small number begin in the other tissues [56]. Breast cancer manifests itself mainly through a breast nodule or thickening that feels different from the surrounding tissue, lymph node enlargement, nipple discharge, a retracted nipple, or persistent tenderness of the breast.

A successful diagnosis in the early stages of breast cancer makes better treatment possible with increase in the probability of the person’s survival [57]. Furthermore, the cost of breast cancer treatment is high. For such reasons, in recent years, several breast diagnostic approaches have been investigated, such as mammography, magnetic resonance imaging, computerized tomography, biopsy, and ultrasound imaging. The latter, in the last few years, has started to become an integral part of the characterization of breast masses because of the advantages previously described. In addition, compared to mammography, ultrasound is the most accessible imaging modality, is age-independent [58] and allows the assessment of breast density that often represents a predictor of breast cancer risk evaluation and prevention. The breast density percentage is defined as the ratio between the area of the fibrograndular tissue and the total area of the breast. Breast ultrasound is also used to distinguish benign from malignant lesions.

For the purposes mentioned above, most of the techniques investigated are based on three principal issues, i.e., detection [59,60,61,62,63,64,65], segmentation [66,67,68,69,70,71,72,73,74,75,76,77], and classification [63,65,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94].

Detection is fundamental in ultrasound analysis because it provides support for segmentation and/or classification between malignant and benign tumors. In a recent study, Gao et al. [60] proposed a method for the recognition of breast ultrasound nodules with low labeled data. Nodule detection was achieved by employing the faster region-based CNN. Benign and malignant nodules were classified through a semi-supervised classifier, based on the mean teacher model, trained on a small amount of labeled data. The results demonstrated that the SSL enabled performances comparable to those obtained with SL trained on a large number of data to be achieved.

Segmentation [66] has an important role in the clinical diagnosis of breast cancer due to the capability to discriminate different functional tissues, providing valuable references for image interpretation, tumor localization, and breast cancer diagnosis. A segmentation approach that combines fuzzy logic and deep learning was suggested by Badawy et al. [67] for automatic semantic segmentation of tumors in breast ultrasound images. The proposed scheme is based on two steps: the first consists of preprocessing based on a fuzzy intensification operator and the second consists of semantic segmentation based on CNN, based on experimenting with eight known models. It is applied using different modes: batch and one-by-one image-processing. The results demonstrated that fuzzy preprocessing was able to enhance the automatic semantic segmentation for each evaluated metric, but only in the case of batch processing. Another automatic semantic segmentation approach was proposed by Huang et al. [69]. In this approach, BUS images are first preprocessed using wavelet features; then, the augmented images are segmented through a fuzzy fully convolutional network, and, finally, an accurately fine-tuning post-processing based on breast anatomy constraints through conditional random fields (CRFs) is performed. The experimental results showed that fuzzy FCN provided better performances than non-fuzzy FCN, both in terms of robustness and accuracy; moreover, its performances were better than all the other methods used for comparison and remained strong when small data sets were used. Ilesanmi et al. [70] used contrast-limited adaptive histogram equalization (CLAHE) to improve image quality. Semantic segmentation was performed through a variant of UNET, named VEU-NET, based on a variant enhanced (VE) block, which encoded the preprocessed image, and concatenated convolutions that produced the segmentation mask. The results indicated that the VEU-Net produced better segmentation than the other classic CNN methods that were tested for comparison. An approach based on the integration of deep learning with visual saliency for breast tumor segmentation was proposed by Vakanski et al. [73]. Attention blocks were introduced into a U-Net architecture and feature representations which prioritized spatial regions with high saliency levels were learned. The results demonstrated that the accuracy of tumor segmentation was better than for models without salient attention layers. An important merit of this investigation was the use of US images collected from different systems, which demonstrated the robustness of the technique.

Image classification is very important in medical diagnostics because it enables distinguishing lesions or benignant tumors from malignant ones and a particular type of tissue from others. Shia et al. [78] presented a method based on a transfer learning algorithm to recognize and classify benign and malignant breast tumors from B-mode images. Specifically, feature extraction was performed by employing a deep residual network model (ResNet-101). The features extracted were classified through the linear SVM with a sequential minimal optimization solver. The experimental results highlighted that the proposed method was able to improve the quality and efficacy of clinical diagnosis. Chen et al. [89] presented a contrast-enhanced ultrasound (CEUS) video classification model for breast cancer into benign and malignant types. The model was based on a 3D CNN with a temporal attention module (DKG-TAM) incorporating temporal domain knowledge and a channel attention module (DKG-CAM) that included feature-based domain knowledge. It was found that the incorporation of domain knowledge led to improvements in sensitivity. A study aimed at testing the capability of AutoML Vision, a highly automatic machine learning model, for breast lesion classification was presented by Wan et al. [91]. The performance of AutoML Vision was compared with traditional ML models with the most used classifiers (RF, KNN, LDA, LR, SVM and NB) and a CNN designed in a Tensorflow environment. The AutoML Vision performances were, on average, comparable to the others, demonstrating its reliability for clinical practice. Finally, Huo et al. [93] experimentally evaluated six machine learning models (LR, RF, extra trees, SWM, multilayer perceptron, and XG Boost) for differentiating between benign and malignant breast lesions using data from different sources. Two examples of the ultrasound depictions of malignant breast lesions are shown in Figure 4. The experimental results demonstrated that the LR model exhibited better diagnostic efficiency than the others and was also better than clinician diagnosis (see

Table 1. Summary of Detection ML algorithms employed in analyzed studies with respect to organ investigated, diagnosis objective, dataset used, and main results achieved.

Ref.	Organ	Objective	Technique	Results	Datasets
[60]	Breast	Recognition of	Faster R-CNN	Mean accuracy: 87%	Public
		breast ultrasound	for detection of	Performances of	6746 and
		nodules with	nodules and SSL	SSL and SL	2220 nodules
		low labeled images	for classification	are comparable
[95]	Arteries	Detection of	Bi-GRU NN	Mean accuracy: 80%	Private
		end-diastolic	trained by a	Better accuracy	20 coronary
		frames in NIRS-IVUS	segment of	than expert analysts	arteries
		images of	64 frames	with Doppler criteria
		coronary arteries
[96]	Heart	Evaluation of	CNN with	Mean ROC AUC	Public
		biomarkers	residual	Anemia: 80%	108521
		from	connections and	BNP: 84%	echocardiogram
		echocardiogram	spatio-temporal	Troponin I: 75%	studies
		videos	convolutions for	BUN: 71.5%
			estimation of
			biomarker values
[97]	Heart	Extract information	Texture-based features	ROC AUC: 80%	Public
		associated	extracted with unsupervised	Sensitivity: 86.4%	392 subjects
		with myocardial	similarity networks	Specificity: 83.3%
		remodeling	ML models (DT,RF,LR,NN)	Prediction of
		from still	for prediction of	myocardial fibrosis
		ultrasound	functional remodeling	only from textures
		images	LR for predicting	of ultrasound images
			presence of fibrosis
[98]	Liver	Detection of	SSD and FPN to	ROC AUC	Public
		gallstones and	classify gallstones with	ResNet-50: 92%	89,000 images
		acute cholecystitis	features extracted	MobileNetV2: 94%
		with still images	by ResNet-50 and	detect cholecystitis
		for preliminary	MobileNetV2	and gallstones with
		diagnosis	to classify	acceptable discrimination
			cholecystitis	and speed
[99]	Fetus	Gestational age and	AlexNet variation	Accuracy %	Private
		automatic estimation	for TC frames	TC plane detection: 99%	5000 TC images
		from TC diameter	extraction	TC segmentation: 97.98%
		as a POCUS solution	FCN for TC	Accurate GA
			localization and	estimation
			measurment
[100]	Fetus	Automatic	LH-SVM	Accuracy: 94.67%	Private
		recognition	SVM for learning of	Average precision: 94.25%	943 standard
		and classification	features extracted	Average recall rate: 93.88%	planes
		of FFUSP	by LBP and HOG	Average F1 score: 94.88%	424 nasolabial
		for diagnosis		Effective prediction	coronal planes
		of cardiac		and classification	50 nonstandard
		conditions		of FFUSP	planes
[101]	Lungs	Assist diagnosis	Pre-trained ResNet50	Average F1-score	Public
		of Covid19	Fully connected layer	Bal dataset: 93.5%	3909 images
		on LUS images	for feature extraction	Unbal dataset: 95.3%
		of FFUSP	Global average pooling	Improves performances
			for features classification.	in radiologists’ diagnosis

).

4.2. Arteries

Another major cause of death in the world is represented by cardiovascular diseases (CVD), caused principally by a pathological condition called atherosclerosis, which is characterized by alterations of artery walls that have lost their elasticity because of the accumulation of calcium, cholesterol, or inflammatory cells. It is the principal cause of ictus and infarct. Early detection of plaques in the arteries has a fundamental role in the prevention of brain strokes. The imaging modality based on ultrasound represents a useful method for the analysis of carotid diseases through visualization and interpretation of carotid plaques because a correct characterization of this disease is fundamental to identifying plaques vulnerable to surgery. A reliable and useful indicator of atherosclerosis is the so-called intima-media (IM) thickness, defined as the distance from the lumen-intima (LI) to the media-adventitia (MA) interface. Most studies have been devoted to the improvement of early atherosclerosis diagnosis; in this respect, three main issues are considered: detection [95,102,103,104,105,106,107], segmentation [108,109,110,111,112,113,114,115], and classification [116,117,118,119,120,121,122,123,124,125,126,127,128].

As far as detection is concerned, Bajaj et al. [95] designed a novel deep-learning methodology for the automated detection of end-diastolic frames in intravascular ultrasound (IVUS) images. Near-infrared spectroscopy(NIRS)-IVUS were collected from 20 coronary arteries and co-registered with the concurrent electrocardiographic (ECG)-signal for identification of end-diastolic frames. A bidirectional-gated recurrent unit (Bi-GRU) neural network was trained by a segment of 64 frames, which incorporated at least one cardiac cycle, and then the test set was processed to identify the end-diastolic frames. The performances of the proposed method demonstrated higher accuracy than expert analysts and conventional image-based (CIB) methodologies.

Two recent segmentation approaches based on DL have been proposed by Blanco et al. [111] and Zhou et al. [112]. The first method [111] employs small datasets for algorithm training. Specifically, plaques from 2D carotid B-mode images, are trained on three small databases, and are segmented through a UNet++ ensemble algorithm, which uses eight individual UNet++ networks with different backbones and architectures in the encoder. Good segmentation accuracy was achieved for different datasets without retraining. The second method [112] involves the concatenation of a multi-frame convolutional neural network (MFCNN), which exploits adjacency information present in longitudinally neighboring IVUS frames to deliver a preliminary segmentation, followed by a Gaussian process (GP) regressor to construct the final lumen and vessel contours by filtering high-dimensional noise. The results obtained with the model developed demonstrated accurate segmentation in terms of image metrics, contour metrics, and clinically relevant variables, potentially enabling its use in clinical routine by reducing the costs involved in the manual management of IVUS datasets. Lo Vercio et al. [128] suggested an automatic detection method fundamentally based on two machine learning algorithms: SVM and RF. The first one is employed to detect lumen, media, and surrounding tissues through SVM algorithms, and the second one to detect different morphological structures and to modify the initial layer classification depending on the detected structures. Successively, the resulting classification maps are inserted into a segmentation method based on deformable contours to detect LI and MA interfaces. The main steps of LI and MA segmentation are described in Figure 5.

With respect to classification, Saba et al. [119] focused on the classification of plaque tissues by employing four ML systems, one transfer learning system, and one deep learning architecture with different layers. Two types of plaque characterization were used: an AI-based mean feature strength and a bispectrum analysis. The results demonstrated that the proposed method was able to accurately characterize symptomatic carotid plaques, clearly discriminating them from symptomatic ones. Another study on carotid diseases was published by Luo et al. [120] that proposed an innovative classification approach based on lower extremity arterial Doppler (LEAD) duplex carotid ultrasound studies. They developed a hierarchical deep learning model for the classification of aortoiliac, femoropopliteal, and trifurcation disease and an RF algorithm for the classification of the quantity of carotid stenosis from duplex carotid ultrasound studies. Then, an automated interpretation of the LEAD and carotid duplex ultrasound studies was developed through artificial intelligence. Successively, a statistical analysis was performed using a confusion matrix and the reliability of novel machine learning models in differentiating normal from diseased arterial systems was evaluated. Good accuracy in classifying the extent of vascular disease was demonstrated (see

Table 2. Summary of segmentation ML algorithms employed in the studies analyzed related to organ investigated, diagnosis objective, dataset used, and main results achieved.

Ref.	Organ	Objective	Technique	Datasets	Results
[67]	Breast	Automatic	Fuzzy preprocessing	GA: 95.45%	Public
		semantic	8 CNN-based	Mean IoU: 78.7%	1200 images
		segmentation	SS models	BF: 68.08%
		of breast		Improvements only
		tumors		with batch processing
[69]	Breast	Automatic	BUS images enhanced	TPR: 90.33%	Private
		semantic	with wavelet features	FPR: 90.00%	325 BUS images
		segmentation	Fuzzy FCN segmentation	IoU: 81.29%
		of breast	Fine-tuning based on	Fuzzy FCN
		tumors	anatomy constraints with	provides better
			conditional random fields	performances
			CRFs performs	than non-fuzzy FCN
[70]	Breast	Automatic	Preprocessing through CLAHE	Mean values	Public
		semantic	UNet variant based on	HD: 77.6%	264 BUS images
		segmentation	VE block for encoding	JM: 80.1%	830 BUS images
		of breast	Concatenated convolutions	DM: 90.7%
		tumors	for segmentation	Better segmentation
				results than
				classic CNN methods
[73]	Breast	Segmentation of	UNet modified	DSC: 90.5%	Private
		tumors by	with attention	Better accuracy	510 images
		incorporating	blocks accounting	with saliency maps
		prior domain	for input saliency	Robustness to
		specific	maps to generate	images from
		knowledge	segmentation	different US scanners
[111]	Arteries	Measurement	8 UNet++ with	Mean DSC: 87.15%	Private
		of total carotid	different backbones	Datasets collected in	144 subjects
		plaque area in	and architectures	different institutions	497 subjects
		B-mode images	Small datasets for
			algorithm training
[112]	Arteries	Automatic	MFCNN for	Median values	Public
		segmentation	preliminary segmentation	JI_lumen: 0.913	160 IVUS
		of lumen and	GP regressor to	JI_vessel: 0.94	pullbacks
		vessel contours	construct lumen	HD_lumen: 0.196 mm
			and vessel contours	HD_vessel: 0.163 mm
[128]	Arteries	Lumen intima and	SVM and RF for	JM	Public
		media-adventia	classification maps	LI: 0.88 ± 0.8 mm	435 images
		fully automatic	LI and MA segmentation	MA: 0.84 ± 0.9 mm
		segmentation of	with deformable contours	Good accuracy
		arterial layers	method	Modular and open-source
[129]	Liver	Automatic	DCNN developed	ICCs	Private
		quantification	with ICNet for	Hepatic: 91.9%	294 liver images
		of the hepatorenal	organ segmentation	Renal: 91.6%
		index (HRI) for	Gaussian texture for	HRI: 73.4%
		evaluation of	HRI quantification	Results comparable to
		fatty liver		those of radiologists
[130]	Fetus	Automatic	CSC: UNet variant	MIOU: 0.55	Private
		segmentation	that calibrate	Better than DeepLabv3+	421 fetal cardiac
		of the ventricular	segmentation with	and U-net	US videos
		septum	time-series information
[131]	Fetus	Automatic	U-net/DeepLabV3+	IoU	Private
		segmentation	segmentation enhanced	DeepLabv3+: 0.47	538 4VC images
		of the thoracic	with MultiFrame and	U-Net: 0.493	in 280 videos
		wall in	Cylinder methods	Improved performances
		ultrasound videos		without altering
				NN structure
[132]	Fetus	Automatic	CPD variation for	Target registration	Private
		fetal head	point cloud segmentation	error: 6.38 ± 3.24 mm	18 fetal brain
		segmentation from	and estimation of
		3D noisy images	probabilistic weights
			obtained with RF
[133]	Lungs	Assessment of	DSA-MIL to combine	Accuracy	Public
		COVID19 from	multiple LUS data	Patient severity: 75%	233 patients
		LUS and	MA-CLR	Binari identification: 87.5 %
		clinical information	for combination	Especially suited for
			of LUS data and	pregnant women
			clinical information	and children
[134]	Thyroid	Efficient and	CNN based on a layer	IoU: 79.5%	Public
		precise semantic	that integrates dense	TPF: 88.5%	3794 images
		segmentation of	connectivity, dilated	FPF: 0.13%
		thyroid nodules	convolutions and	High accuracy/efficiency
			factorized filters	Real time

).

4.3. Heart

Echocardiography is one of the most employed diagnostic tests in cardiology, where heart images are created through Doppler ultrasound. It is routinely employed in the diagnosis, management, and follow-up of patients with any suspected or known heart disease.

The heart is a muscular organ that pumps blood through the body and is fundamentally divided into four different chambers: the upper left and right atria and the lower left and right ventricles. The heart activity can be divided into two principal phases: systole and diastole. During systole, the myocardium contracts, ejecting blood to the lungs (right ventricle) and the body (left ventricle). During diastole, the cardiac muscle dilates expanding the heart’s volume and causing blood to flow in. The heart has four valves, including the mitral valve that collapses the left atria and the left ventricle and plays a fundamental role by regulating the blood transition from atria to ventricle, opening up during the diastole, while during the systole the valve closes and prevents reflux towards the left atria. Echocardiography can provide information about different anatomical heart aspects including position, shape of the atrium and ventricles [135], and even other variables such as cardiac output, ejection fraction and diastolic function. In addition, echocardiography enables detection of a series of heart diseases, including cardiomyopathy, congenital heart diseases, aneurysm, and mitral valve diseases. However, one of the major issues in echocardiography is the difficulty of automatically classifying and identifying large databases of echocardiogram views in order to provide a diagnosis. The classification task is challenging because of several properties of echocardiograms, including the presence of noise, redundant information, acquisition errors, and the variability of different scans due to different acquisition techniques.

Several studies have been devoted to the automation of algorithms for the detection of anomalies and heart anatomy [96,97,136,137,138], and the classification of echocardiogram views to provide a full and reliable assessment of cardiac functionality improving diagnosis accuracy [139,140].

As far as detection is concerned, an advanced method for the evaluation of several biomarkers from echocardiogram videos based on DL has been developed by Hughes et al. [96]. The method, named EchoNet-Labs, is a CNN with residual connections and spatio-temporal convolutions that provides a beat-by-beat estimate for biomarker values. Experimental results have demonstrated high accuracy in detecting abnormal values of hemoglobin, troponin I, and other proteins, and better performance compared to models based on traditional risk factors. A detection method based on radiomics-based texture analysis and supervised learning was proposed by Kagiyama et al. [97], who designed a low-cost texture-based pipeline for the prediction of fibrosis and myocardial tissue remodeling. The first part of the method consists of the extraction of 328 texture-based features of the myocardium from ultrasound images and exploration of the phenotypes of myocardial textures through unsupervised similarity networks. The second part involves the employment of supervised machine learning models (decision trees, RF, logistic regression models, neural network) for the prediction of functional left ventricular remodeling, while, in the third part, supervised models (logistic regression models) for predicting the presence of myocardial fibrosis are employed. Figure 6 shows a comparison of two myocardial fibrosis predictions from ultrasound and magnetic resonance images.

A classification deep learning approach was developed by Vaseli et al. [139]. They defined a method for obtaining a lightweight deep learning model for the classification of 12 standard echography views, by employing a large echography dataset. For this purpose, three different teacher networks are implemented, each of which consists of a CNN module and fully-connected (FC) module, where the first module is based on one of the three advanced deep learning architectures, i.e, VGG-16, DenseNet, and Resnet. A dataset of 16,612 echo cines obtained from 3151 unique patients across several ultrasound imaging machines was employed for the development and evaluation of the networks. The proposed models were shown to be lightweight and faster than state-of-the-art huge deep models, and to be suitable for POCUS diagnosis.

4.4. Liver

Liver disease is one of the principal causes of death worldwide and comprises a wide range of diseases with varied or unknown origins. In 2017, about 1.32 million deaths worldwide were due to cirrhosis. Furthermore, liver cancer represents the fifth most common cancer and the second cause of death for cancer according to the World Health Organization (WHO). Studied pathologies can be summarized as:

• focal liver lesions, solid formations that can be benign or malignant,
• liver fibrosis, excessive accumulation of extracellular matrix proteins, such as collagen,
• fatty liver or liver steatosis, conditions based on the accumulation of excess fat in the liver,
• liver tumors.

A number of studies have sought to develop automated algorithms for detection [98,141,142,143,144], segmentation [143], and classification, [143,145,146,147,148,149,150,151,152,153,154,155,156,157] of the diseases described above.

Yu et al. [98] developed a machine learning system to detect and localize gallstones and to detect acute cholecystitis using still images for preliminary rapid and low cost diagnoses. A single-shot multibox detector (SSD) and a feature pyramid network (FPN) were used to classify and localize objects using image features extracted by ResNet-50 for gallstones and MobileNet V2 to classify cholecystitis. The deep learning models were pretrained using public datasets. The experimental results demonstrated the capability of the proposed system to detect cholecystitis and gallstones with acceptable discrimination and speed and its suitability for point-of-care ultrasound (POCUS).

A recent study by Cha et al. [129] proposed a deep learning model aimed at automatically quantifying the hepatorenal index (HRI) for evaluation of ultrasound fatty liver, in order to overcome limitations due to interobserver and intraobserver variability. They developed an organ segmentation based on a deep convolutional neural network (DCNN) with Gaussian mixture modeling for automated quantification of the hepatorenal index (HRI) by employing B-mode ultrasound abdominal images. Interobserver agreement for the measured brightness of liver, kidney, and calculated HRI were analyzed between two board-certified radiologists and DCNN using intraclass correlation coefficients. The automatic quantification of HRI through DCNN results were found to be similar to those obtained by expert radiologists.

Regarding classification, Wang et al. [146] proposed a method to differentiate malignant from benign focal liver lesions through two-dimensional shear-wave elastography (2D-SWE)-based ultrasomics (ultrasound-based radiomics). The ultrasomics technique was employed to extract from 2D-SWE images features that were used to define an ultrasomics score model, while SWE measurements and ultrasomics features were used to define a combined score model through an SVM algorithm. Good diagnostic accuracy for the combined score in differentiating malignant from benign focal liver lesions was demonstrated. The authors highlighted, however, that, to achieve more reliable results, a higher number of cases would be required to better train the ML model. An alternative approach based on ultrasomics was proposed by Peng et al. [147] who concentrated on the differentiation of infected focal lesions from malignant mimickers. In particular, they defined an ultrasomics model based on machine learning methods with ultrasomics features extracted from grayscale images, and dimensionality reduction methods and classifiers employed to carry out feature selection and predictive modeling. The experimental results demonstrated the usefulness of ultrasomics in differentiating focal liver lesions from malignant mimickers. An alternative approach focusing on ultrasound SWE was proposed by Brattain et al. [149], who developed an automated method for the classification of liver fibrosis stages. This method was based on the integration of three modules for the evaluation of SWE image quality, selection of a region of interest, and use of machine learning-based (SVM, RF, CNN and FCNN) multi-image SWE classification for fibrosis stage ≥ F2. The performance of the system was compared with manual methods, showing that the proposed method improved classification accuracy. A study focused on liver steatosis was published by Neogi et al. [155]. They presented a novel set of features that exploited the anisotropy of liver texture. The features were obtained using a gray level difference histogram, pair correlation function, probabilistic local directionality statistics, and randomness of texture. Three datasets that included anisotropy features were employed for the classification of images using five classifiers: MLP, PNN, LVQ, SVM, Bayesian. The best results were achieved with PNN and anisotropy features.

4.5. Fetus

Ultrasound imaging was introduced into the field of obstetrics by Donald et al. [158], and, since then, it has become the most commonly used imaging modality for investigating several factors related to fetal diagnosis, such as information on fetal biometric measurements, including head and abdominal circumferences, biparietal diameter and information on fetal cardiac activity. Several scientific studies have been devoted to advancement of the quality of prenatal diagnoses by focusing on three main issues: detection of anomalies, fetal measurements, scanning planes and heartbeat [99,100,159,160,161,162,163,164], segmentation of fetal anatomy in ultrasound images and videos [99,130,131,132,164,165,166,167] and classification of fetal standard planes, congenital anomalies, biometric measures, and fetal facial expressions [99,100,163,165,167,168,169,170,171,172,173].

A detection approach based on DL was proposed by Maraci et al. [99]. They designed a method for point-of-care ultrasound estimation of fetal gestational age (GA) from the trans-cerebellar (TC) diameter. In the first step, TC plane frames are extracted from a short ultrasound video using a standard CNN based on a variation of the AlexNet architecture. Then, an FCN is employed to localize TC structure and to perform TC diameter estimation. GA is finally achieved through a standard equation. A good agreement was found between the automatic and manual estimation of GA. A recent ML detection method has been published by Wang et al. [100], who focused on the accurate identification of the fetal facial ultrasound standard plane (FFUSP), which has a significant role in facial deformity detection and disease screening, such as cleft lip and palate detection. The authors proposed an LH-SVM texture feature fusion method for automatic recognition and classification of FFUSP. Texture features were extracted from US images through a local binary pattern (LBP) and a histogram of oriented granted (HOG); successively, features were fused and SVM was employed for predictive classification. The performances obtained demonstrated that the proposed method was able to effectively predict and classify FFUSP.

With respect to segmentation, Dozen et al. [130] proposed a novel segmentation method called cropping-segmentation-calibration (CSC) of the ventricular septum in fetal cardiac ultrasound videos. This method was based on time-series information of videos and specific information for U-Net output calibration, obtained by cropping the original frame. The experimental results demonstrated a clear improvement in performance with respect to general segmentation methods, such as DeepLab v3+ and U-net.

A novel model-agnostic DL method (MFCY) was proposed by Shozu et al. [131] in order to improve the segmentation performance of the thoracic wall in ultrasound videos. Three standard UNet (DeepLabV3+), pre-trained with the original sequence video and labels of thoracic wall (TW), thoracic cavity (TC) and whole thorax (WT), were used to perform a preliminary segmentation of the video sequence. Then a multi-frame method (MF) was used to extract predictions for each labeled target. Finally, a cylinder method (CY) integrated the three prediction labels for final segmentation. The results showed improvement in the segmentation performance of the thoracic wall in fetal ultrasound videos without altering the neural network structure.

Perez-Gonzalez et al. [132] presented a method, named probabilistic learning coherent point drift (PL-CPD), for automatic registration of real 3D ultrasound fetal brain volumes with a significant degree of occlusion artifacts, noise, and missing data. Different acquisition planes of the brain were preprocessed to extract confidence maps and texture features, which were used for segmentation purposes and to estimate probabilistic weights by means of random forest classification. Point clouds were finally registered using a variation of the coherent point drift (CPD) method that basically assigns probabilistic weights to the point cloud. The experimental results, although obtained from a relatively small dataset, demonstrated the high suitability of the proposed method for automatic registration of fetal head volumes.

A recent deep learning classification model was developed by Rasheed et al. [165] for automation of fetal head biometry by employing a live ultrasound feed. Initially, the headframes were classified through the CNN ALEXNET, obtained in this case from the ultrasound videos. The classified headframes were then validated through occipitofrontal diameter (OFD) measurement. Successively, the classified headframes were segmented by a UNET with mask and annotated images. Then, the least square ellipse (LSE) was employed to compute the biparietal diameter (BPD) and head circumference (HC). This approach enabled accurate computation of the gestational age with very reduced interaction of the sonographer with the system (see

Table 3. Summary of classification ML algorithms employed in studies analyzed related to organ investigated, diagnosis objective, dataset used, and main results achieved.

Ref.	Organ	Objective	Technique	Results	Datasets
[78]	Breast	Classification of	Pretrained ResNet-101	Sensitivity: 94.34%	Private
		tumors between	for feature extraction	Specificity: 93.22%	2099 images
		benign and	Linear SVM	PPV: 92.6%
		malignant from	for classification	NPV: 94.8%
		B-mode images		More accurate
				performances
				than radiologists
[89]	Breast	Classification of	3D CNN	Sensitivity: 97.2%	Public
		tumors between	for temporal/spatial	Accuracy: 86.3%	221 lesions
		benign and	extraction	Domain knowledge
		malignant	DKG-TAM for	allows diagnostics
		from CEUS	temporal attention	improvements
			DKG-CAM for
			features concatenation
[91]	Breast	Highly automatic	AutoML Vision	Accuracy: 86%	Public
		classification of	For comparison:	Sensitivity: 84%	895 images
		tumors from	CNN and	Specificity: 88%
		B-mode images	ML classifiers	F1: 0.83
			(RF, KNN, LDA, LR)	AutoML Vision
				comparable with
				other methods
[93]	Breast	Evaluation of ML	6 ML classifiers:	LR	Public
		methods for	LR, RF, Extra Trees,	ROC AUC: 90.6%	1345 patients
		breast cancer	SVM, MLP, XG Boost	Brier score: 0.65
		diagnosis images
[119]	Arteries	Characterization and	7 CNN	Mean accuracy	Public
		classification of	for data optimization	DL: 93.55%	346 patients
		carotid ultrasound	TL for characterizing	TL: 94.55%
		plaque tissues	carotid plaques and	ML: 89%
			4 ML models
			(KNN, SVM,DT,RF)
[120]	Arteries	Automation of the	HNN for classification	Accuracy	Public
		initial interpretation	of aortillac and	Normal: 97%	5761 LEAD
		of lower extremity	trifurcation disease	Aortillac: 82%	studies
		arterial Doppler	RF for	Femoropopliteal: 90.1%	18,659
		and duplex carotid	classification	Trifurcation: 90.5%	duplex
		ultrasound studies	of stenosis	Good performances	carotid
					studies
[139]	Heart	Lightweight and	Three teacher	Accuracy: 89 %	Public
		fast transthoracic	networks (VGG-16,	Six times	16,612
		ecocardiography	DenseNet and Resnet)	faster than	echocines
		classification for	transfer learned	huge models
		diagnosis of	knowledge to
		cardiac conditions	lighweight models
[146]	Liver	Differentiation of	SVM algorithm	AUC: 0.94%	Private
		malignant from	for establishing	Sensitivity: 92.59%	175 focal lesions
		benign focal liver	two predictive	Specificity: 87.5%
		lesions using	models: ultrasomics	PPV: 94.59%
		2D SWE-based	features and	NPV: 82.50%
		ultrasomics	SWE measurements	Good accuracy
				of combined method
[147]	Liver	Differentiation of	LR and PCA to	AUC	Private
		of infected focal	reduce dimension	HCC: 0.836%	104 focal
		liver lesions	of radiomics features	CC: 0.766%	liver lesions
		from malignant	NB, DT,KNN,LR,SVM	cHCC-CC: 0.74	485 hepatic
		mimickers in	to obtain	Liver metastasis: 0.808	tumors
		B-mode images	predictive models	MH tumor: 0.745
[149]	Liver	Liver fibrosis	Image proc pipeline:	AUC	Private
		classification	Quality assessment	Specificity: 71%	5526 SWE
		with SWE images	ROI selection	Sensitivity: 95%	images
			fibrosis classification	Better accuracy
			with SVM, RF, CNN, FCNN	than manual methods
[155]	Liver	Identification of	Five different	Sensitivity: 99%	Private
		liver steatosis	classifiers (MLP,	Accuracy: 100%	340 images
		from anisotropy	PNN,SVM,LVQ,	with anisotropy
		features in B-mode	Bayesian)	features and
		images	Three features	PNN classifiers
			sets including
			anisotropy features
[165]	Fetus	Classification and	CNN AlexNet for	Accuracy: 96 %	Public
		segmentation of	classification of	Almost automatic	10,000 labeled
		fetal head	headframes	gestational age	images
		from ultrasound	CNN Unet for	estimation	1000 ultrasound
		videos	segmentation of		videos
			classified
			headframes
[174]	Lungs	Development of	Deep learning	Accuracy	Private
		a system for	based on Reg-STN	Frame: 92.2%	623 videos
		accurate interpretation	trained with	Video: 91.1%
		of pleural effusion	supervised and	Good agreement
		in LUS images	weakly supervised	with expert
			methods models	clinicians

).

4.6. Lungs

Computed tomography (CT) is considered as the imaging gold standard for pulmonary disease due to its high reliability. However, CT presents a series of disadvantages because it is risky due to the presence of radiation, expensive and non-portable. A valid alternative is represented by lung ultrasound (LU), which is cheap, safe, portable, and is capable of generating medical images in real-time. LU has been used for many years for the evaluation of several lung diseases, including tumors [175,176], interstitial diseases [177,178], post-extubation distress [179], lung edemas [180], and subpleural lesions [181]. In very recent years, research activity into lung ultrasonography has been growing significantly due to the diffusion of the pandemic worldwide. In particular, in COVID-19 evaluation, the use of AI has assumed an increasingly important role concerning the analysis of images in order to make rapid decisions and relieve the heavy workload of radiologists.

AI techniques reduce operator-dependence, standardize the interpretation of images and provide stable results; they have been focused principally on COVID-19 syndrome detection [101,182,183,184,185,186], segmentation of lung regions [133,182,183,184,185,186], classification of lung diseases between COVID-19 positivity and COVID-19 negativity [174,182,183,184,185,186,187,188,189].

With respect to detection, Shang et al. [101] proposed a CAD system that consists of the feature extraction of LUs images through a residual network (ResNet) to assist radiologists in distinguishing COVID-19 syndrome from healthy and non-covid pneumonia. The architecture of the ResNet, pre-trained using ImageNet, was modified by adding a fully connected layer for feature extraction and a global average pooling for features classification. Then, the gradient-weighted class activation mapping (Grad-CAM) method was used to create an activation map that highlights the crucial areas to help radiologist visualization. The CAD system has proved capable of improving radiologists’ performance of COVID-19 diagnosis in experiments carried out.

An interesting segmentation method for accurate COVID-19 diagnosis has been proposed by Xue et al. [133]. The method is based on a dual-level supervised multiple instances learning module (DSA-MIL) and can predict patient severity from heterogeneous LUS data of multiple lung zones. An original modality alignment contrastive learning module (MA-CLR) is proposed for the combination of LUS data and clinical information. Nonlinear mapping was trained through a staged representation transfer (SRT) strategy. This method demonstrated great potential in real clinical practice for COVID-19 patients, particularly for pregnant women and children.

A classification deep learning procedure was proposed by Tsai et al. [174] who defined a standardized protocol combined with a deep learning model based on a spatial transformer network for automatic pleural effusion classification. Then, supervised and weakly supervised approaches, based on frame and video ground truth labels, respectively, were used for training deep learning models. The method was compared with expert clinical image interpretation with similar accuracy obtained for both methods, which brings closer the possibility of achieving the automatic, efficient and reliable diagnosis of lung diseases.

4.7. Other Organs

Machine learning ultrasound is being successfully applied to a number of other organs including:

• Prostate [190,191,192]: research activity has mainly focused on prostate segmentation on ultrasound images, fundamental in biopsy needle placement and radiotherapy treatment planning; it is quite challenging due to the relatively low quality of US images. In recent years, segmentation based on deep learning techniques has been widely developed due to several benefits compared to classical techniques which are difficult to apply in real-time image-guided interventions.
• Thyroid [88,134,193,194,195,196,197,198,199,200,201,202,203,204,205,206]: the risk of malignancy of thyroid nodules can be evaluated on the basis of nodule ultrasonographic characteristics, such as echogenicity and calcification. Much activity has been devoted to automate thyroid detection through CAD systems, mainly based on CNN.
• Kidneys [207,208,209,210,211,212,213,214,215,216,217,218,219,220]: US image-based diagnosis are widely used for the detection of kidney abnormalities including cysts and tumors. For the early diagnosis of kidney diseases, DNN and SVM are very often used as machine learning models for abnormality detection and classification.

Table 1, Table 2 and Table 3 summarize the main features of the analyzed studies subdivided by detection, segmentation, and classification, respectively.

Figure 7 presents a resuming histogram where the frequency of application of the different ML techniques in the analysed period is reported for each analyzed organ. As can be seen, DL techniques based on CNN are clearly the most popular for almost all organs. In particular, for breast and liver, which are the most investigated organs, CNN is employed in about 63 and 50 percent of the cases, respectively. Only for arteries is a slight predominance of SVM methods observed.

5. ML in US Non-Destructive Evaluation

Ultrasound application to the mechanical field is principally focused on material NDE. Ultrasound is implemented in NDE to obtain information about the location and size of subsurface defects in different materials. In general, NDE techniques are employed to extend component life, reduce manufacturing costs and increase safety. The inspections typically involve evaluating a material’s response to a physical stimulus, such as ultrasound. This is often carried out from many angles and positions to build an image of the internal structure of a component and identify the presence of damage. Defect characterization is obtained through inspection of NDE data by a human operator that can create consistency issues, especially when the data is very complex. As data volume increases, inspection by a single operator becomes very slow, and then more operators who work in parallel may be necessary, but the results become more inconsistent. These problems have promoted a push towards the identification of an automated method. Since it is based on a pattern recognition process, machine learning represents the ideal candidate. Machine learning can use all available information and produce a more accurate result, increasing automation, and reducing the possibility that the human operator will create errors. In recent years, several machine learning techniques have been employed in order to improve the reliability of ultrasound non-destructive testing [124,221,222,223,224,225,226,227,228].

Pyle et al. [221] proposed a method based on the use of DL for crack characterization in NDE through ultrasound technology. The principal problem of this method is represented by the scarcity of real defect data to train on. This problem was solved through an efficient hybrid finite element (FE) and ray-based simulation employed in order to train a CNN to characterize real defects. The effectiveness of such a method was demonstrated by sizing surface-breaking cracks in ultrasonic inline pipe inspection obtaining high characterization accuracy of the deep learning approach compared to traditional image-based sizing.

Oliveira et al. [222] proposed the application of several novelty detection methods (identification of novel, or unusual, data from within a dataset), combined with non-destructive ultrasound testing, to identify structural problems in wind turbine blades. In particular, ultrasound signals were preprocessed to extract relevant features through discrete Fourier transform (DFT) and PCA to reduce noise using wavelet decomposition. Several novelty detection algorithms have been applied to detect the presence of defects in wind turbine blades, including k-means, one-class SVM, and distance-based methods. The results of those novelty detection methods were compared with those obtained from multi-class classifiers, such as artificial neural networks, and demonstrated very high discrimination efficiencies.

Li et al. [223] introduced an alternative method of spectral analysis named quantile-frequency analysis (QFA). QFA is based on trigonometric quantile regression and converts a time series into a bivariate function of quantile level and frequency variables. For problems of time series classification, the technique of functional principal component analysis (FPCA) is applied to the quantile periodogram and the resultant projection coefficients are employed as features with reduced dimensions for time-series classification. Various machine learning classifiers were trained and tested by cross-validation using the proposed features. The study case analyzed was on ultrasound signals that have to be automatically classified for NDE of the structural integrity of aircraft panels made with bonded aluminum layers. Three classifiers were employed to evaluate the performance of FPCA: LDA, quadratic discriminant analysis (QDA), and SVM. The QFA method was found to be more effective than its ordinary periodogram–based counterpart in delivering higher out-of-sample classification accuracy.

Arbaoui et al. [229] proposed a methodology for automatic crack detection and monitoring in concrete structures. The method involves three stages. In the first stage, NDE of the specimen is performed and an ultrasonic signal, which contains information on the presence of defects, is obtained. Then a multiresolution analysis based on wavelets is performed to analyze crack sizes at different scales. Finally scalogram features are extracted by CNN in order to identify the type of defect that will be classified as a crack or not. CNN is basically composed of four stages, each composed of four layers (convolution, batch normalization, RELU and pool), and a final FC stage that performs the classification. The method was tested on a public dataset (SDNET2018) using both AlexNet and ResNet50 architectures, achieving good accuracy.

In the archeology field, a study investigating a new pattern recognition application for the ML classification of the provenance of archeological ceramics has recently been published by Salazar et al. [230]. The method is based on non-destructive ultrasonic testing and data analysis through advanced pattern recognition techniques, including feature ranking, sample augmentation, semi-supervised active learning, and optimal late fusion. Ultrasonic characterization of ceramic material from the measured ultrasonic signal is performed for the construction of a material signature that consists of time, frequency, and statistical variables defined on the base of a material reflectivity model. More exactly, the proposed method involves processing of the ultrasound features extracted from a set of pieces by considering classification on the basis of the fusion of the results from three different classifiers: LDA, RF, SVM. A problem of archeological provenance classification of pieces consisting of terra sigillata and non-terra sigillata ceramic shards from the same archeological sites, Iberian ceramic shards from two cities in Spain, and Roman sigillata ceramic shards from two origins, was investigated. The experimental results demonstrated that the fusion-based method achieved the best results in comparison with LDA, RF, and SVM.

6. Conclusions

In the present review, the most recent ML techniques applied to ultrasound imaging have been illustrated with a focus on medical diagnostics and NDE. The review commenced with an overview of the most employed ML and ultrasound imaging techniques in the analyzed papers and subsequently was devoted to the application of such techniques in the field of medical diagnostics and NDE.

From the analysis of the studies examined, in which a great variety of ML algorithms have been tested, we highlighted the noticeable employment of deep learning techniques based on CNN for almost all organs, especially for the breast and liver. The use of SVM methods, in particular for arterial disease diagnosis, was also quite frequent.

The main merits of ML over conventional methods can be summarized as follows. First, ML methods enable enhancing of the quality of US images, which is lower than for other imaging technologies due to the common presence of artifacts and noise. ML approaches also guarantee a more objective evaluation of the data than traditional methods that involve analysis by operators, often based on a heuristic approach, avoiding consistency issues. In addition, they provide a reduction in time and cost for evaluation and analysis. The principal benefits of ML are particularly noticeable in the medical diagnostics field, where ultrasound techniques are extensively used for the diagnosis of several kinds of organ diseases. From the papers analyzed, it is evident that ML enables significant improvements in terms of accuracy in the detection and classification of different tissues and diseases, and the segmentation of several types of organs. Furthermore, it is able to reduce the rate of missing or incorrect diagnoses because the algorithms manage to find some details or particulars that could not be identified by medical operators. The diagnostic results are often better than those provided by clinicians, both in terms of precision and quality. In many cases, the usefulness of ML is to assist radiologists by providing a “second opinion” in clinical preliminary examinations, which should improve their diagnostic capacity, and reduce the time and effort associated with manual analysis of ultrasound images. As an important consequence, it is anticipated that the availability of reliable automatic techniques will democratise effective access to ultrasound diagnostic tools (POCUS), extending them also to people living in rural zones or developing countries. A possible drawback highlighted in many of the studies reviewed is the relatively low number of samples present in databases, which limits the reliability of results. In many cases, the databases from medical diagnosis are generated by a single type of device and/or by a single collection site (e.g. institution or hospital), limiting the generalizability of the ML classification models derived from these databases. Unfortunately, this issue is a peculiar problem of US images because, based on present clinical practice, their quality and information content are highly operator-dependent. This also underlines the almost total absence of ML techniques applied to 3D US images, which are increasingly used in modern US diagnosis methods.