Feature Extraction and Classification of Simulated Monostatic Acoustic Echoes from Spherical Targets of Various Materials Using Convolutional Neural Networks

Abstract

Active sonar target classification remains an ongoing area of research due to the unique challenges associated with the problem (unknown target parameters, dynamic oceanic environment, different scattering mechanisms, etc.). Many feature extraction and classification techniques have been proposed, but there remains a need to relate and explain the classifier results in the physical domain. This work examines convolutional neural networks trained on simulated data with a known ground truth projected onto two time-frequency representations (spectrograms and scalograms). The classifiers were trained to discriminate the target material type, geometry, and internal fluid filling, while the hyperparameters were tuned to the classification task using Bayesian optimization. The trained networks were examined using an explainable artificial intelligence technique, gradient-weighted class activation mapping, to uncover the informative features used in discrimination. This analysis resulted in visual representations that allowed the CNN choices to be related to the physical domain. It was found that the scalogram representation provided a negligible classification accuracy increase compared with the spectrograms. Networks trained to discriminate between target geometries resulted in the highest accuracy, and the networks trained to discriminate the internal fluid of the target resulted in the lowest accuracy.

1. Introduction

There are two types of sonar systems: passive and active. The former is when a hydrophone is recording sound within the ocean, and the latter occurs when a pulse of sound, or a ping, has been sent out to a target of interest in an attempt to determine a target’s information. This research is specific to active sonar. Active sonar target recognition and classification has numerous maritime applications, such as harbor monitoring, autonomous underwater vehicle vision, and seabed characterization. However, classification suffers from feature uncertainties due to unpredictable or unknown environmental (salinity, temperature, sound speed profile, etc.) and target parameters (size, shape, orientation, etc.) [1]. Different forms of obstructions, such as fish or bubbles, or oceanic noise may also be present within a sonar’s return path and can further entangle a received response [2]. These effects combine and degrade the target-specific informative features used for discrimination.

Machine learning algorithms are commonly used to perform classification of sonar data [3,4,5,6,7,8,9,10,11]. Many of these classification pipelines employ convolutional neural networks (CNNs). Williams demonstrated classification of sonar images using a 10 layer CNN [3]. Wang et al. used weights found with a deep belief network and then replaced the randomly initialized CNN weights to perform classification of various sonar images [12]. CNNs were used for feature extraction rather than classification by Zhu et al., who used AlexNet to extract sonar image features prior to classification using a support vector machine [7]. Sonar image feature extraction through edge detection was additionally proposed by Wang et al., who created three different CNNs with skip connections and demonstrated its ability to find continuous edges [13]. However, many of these networks are deep and rely on a large number of samples for training. A large amount of public domain experimental field data for training is challenging and costly to obtain [9]. Many approaches, such as feature engineering or extraction [10], transfer learning [4], pretrained networks [6,7], synthetic data generation [4,5,6], and employing many tiny classifiers [11], have been used to mitigate this challenge. This work is a combination of synthetic data generation, two time-frequency representations, and the employment of moderately sized classifiers that have been optimized for various sonar target recognition tasks. The signals used throughout this work are simulated from known models, giving complete control over the ground truth and simulation options. This allows interpretation of the informative features used by the classifier for discrimination to be related back to the physical domain.

Signal segmentation is performed prior to the projection of simulated target backscattered responses onto two two-dimensional representations. The use of two-dimensional (2D) analysis in conjunction with machine learning techniques is common within the underwater community [10,14,15,16,17,18,19]. An in-depth review of the feature extraction and classification methods is provided in [20]. Choi et al. employed cross-spectral density matrices as inputs to a variety of classifiers trained to discriminate against submerged or surface ships. In most cases, CNNs provided increased classification, with a lowest binary misclassification rate of 0.92% [15]. Power cepstrums were used as CNN inputs trained to perform detection and ranging of vessels in a variety of SNRs [14]. Other reported research describes a novel chirp wavelet [16] using a three-channel spectrogram CNN classfier [17], or using binary features extracted from acoustic spectra [10,18]. An interesting approach was taken by Luo et al. They used multiple spectrograms at various resolutions as a three-channel input in conjunction with a ResNet-inspired network, achieving up to a 96.32% classification accuracy in ship noise classification [19]. This research differs in the use of simulated signals with a known ground truth, a comparison of the short-time Fourier transform and the continuous wavelet transform representations and their impact on classification, and examination of the classifiers post training to explain the classifier choices.

Deep learning techniques are considered to be state of the art and provide increased classification when compared with traditional approaches [21]. However, these networks suffer from a lack of explainability and interpretability of their results due to their complexity [22]. This creates a lack of trust and transparency in a classifier and is sometimes referred to as a `black box’. When incorrect classification may result in harmful real-world outcomes (as is the case in sonar classification), there exists the need for explainable artificial intelligence (XAI) [23]. Explainable artificial intelligence (XAI) is an emerging area of study aimed at increasing the interpretability of machine learning choices. XAI attempts to build transparency and increase trust in a classifier algorithm by making the classifier choices interpretable to humans [24]. There are many different approaches to XAI, and the one employed in this work is gradient-weighted class activation mapping (Grad-CAM), which uses the gradients of test images sent through the trained network to determine which feature is the most important one [25]. This technique can be used to find discriminative, class-specific features and has been used to explore trained networks in the medical field [26,27].

Spectrograms and scalograms are two-dimensional representations that describe how the frequency content of a signal changes over time. These representations have been chosen as they are two standard representations that are commonly used. The goal of this research is not to find the optimum time-frequency distributions for classification but rather to interpret the classifier choices and relate the decisions back to the physical domain. A spectrogram has a fixed window size that forces a constant time-frequency resolution, while scalograms have a scaling parameter that allows the contraction and dilation of the window. This provides a varied resolution. The fixed resolution results in spectrograms being an adequate representation of stationary signals and a poor representation of non-stationary signals. The variable resolution associated with scalograms allows them to be a good representation of non-stationary signals. A comparison between the two representations through classification accuracy is reported within this work. This work is a continuation and expansion of the comparison of spectrogram and scalogram representations of the simulated backscattered responses reported in [28]. This research differs from the previous iteration by updating the signal segmentation, the inclusion of additional simulated targets, the usage of a convolutional neural network that has been optimized for classification, and the examination of the trained classifier to explain its choices and relate them to the physical domain.

A large amount of experimental sonar field data for training is challenging and costly to obtain, but advances in feature extraction and machine learning have been creatively mitigating this problem. However, classifiers lack interpretation to the physical domain. This work uses simulated data that have a known ground truth and a CNN for classification. The hyperparameters are tuned using Bayesian optimization. After training, the networks are examined to determine the important features used by the network for classification. This work is an examination and comparison of two common time-frequency representations, both preceding and following classification. The networks were trained to perform classification for a variety of tasks in order to determine any dependencies between the classification task and optimized hyperparameters. The post-classification examination is performed using an XAI technique to reveal the target-specific features that a CNN uses for discrimination. The key contribution of this work is in the examination of the networks trained on common time-frequency representations and the explanation of the network choices in the physical domain. This post hoc analysis allows for informed decisions to be made for future classification pipelines to intuitively bias networks by forcing them to prioritize influential features. The analysis can additionally be used to examine the failure modes of classifiers and create mitigation strategies.

2. Materials and Methods

In this section, the simulated data generated for this research are described first, followed by the two-dimensional feature representations and classifier. The classification metrics used to compare the performance are reported, and finally an explainable artificial intelligence technique is described. This research was coded and developed using MATLAB 2022a v9.12.0 with the Deep Learning Toolbox [29].

2.1. Data Generation

Elastic targets have both rigid and elastic scattering. The rigid scattering is typically associated with specular or geometric reflections, while the latter is due to the elastic material properties of the target. The common reflections associated with solid elastic spheres are the specular or geometric reflection and the Rayleigh and whispering gallery surface waves [30]. Specular reflection waves are the first component seen in a response, followed by the Rayleigh surface waves. The whispering gallery waves are a smaller contribution. A shell response consists of specular reflections and Lamb surface waves [31]. A Lamb wave propagates around the outside of a shell.

These responses are described by the partial-wave series and have been studied in depth using Sommerfeld–Watson transformation [30,32] and resonant scattering theory (RST) [33,34]. The data used throughout this research have been simulated for a monostatic plane wave incident to elastic spheres and shells using RST. The backscattered response for a solid sphere was generated as described in [33], and the backscattered response for a shell was generated as described in [34].

Backscattered responses were simulated using the following materials: aluminum, stainless steel 347, tungsten carbide, and granite. The longitudinal and shear speed of sound in the materials examined throughout this research were taken from the Engineering Toolbox [35] and are reproduced in

Table 1. Material properties used for simulating backscattered responses. Data from the Engineering Toolbox [35].

Material	Density ($\mathbf{kg}/{\mathbf{m}}^3$)	Longitudinal ($\mathbf{m}/\mathbf{s}$)	Shear ($\mathbf{m}/\mathbf{s}$)
Aluminum	2712	6420	3040
Granite	2700	4500	3500
Stainless Steel 347	7900	5790	3100
Tungsten Carbide	13,800	6860	4185

. In practice, typically a probe pulse such as a linear frequency-modulated (LFM) waveform is transmitted to the target of interest. The measured scattered response is then match filtered. A deconvolution may be performed, but the match-filtering operation itself is approximately equivalent to a finite band deconvolution.

A sampled version of the continuous-time backscattered response was simulated. The solution was sampled based on the dimensionless frequency $ka$. There were 4096 dimensionless frequencies linearly spaced between the real frequency values of 500 Hz and 15.6 kHz for each material. The RST solution resulted in the continuous time solution. Two different fluids filled the shells: air and oil octane. The corresponding material properties were reproduced from the Engineering Toolbox in

Table 2. Interior material properties used for simulating backscattered responses. Data from the Engineering Toolbox [35].

Material	Density ($\mathbf{kg}/{\mathbf{m}}^3$)	Speed of Sound ($\mathbf{m}/\mathbf{s}$)
Air	1.2	343
Octane	702	1171

.

The thickness of the shell greatly impacts the amount of resonance in the response. This parameter is quantified by $h=(r_o-r_i)/r_i$, where $r_o$ is the outer radius and $r_i$ is the inner radius. An example of this impact is shown in Figure 1, where Figure 1a,b shows a thick ($h>0.1$) and thin ($h<0.01$) shell, respectively. There are clear and distinct resonances in the thin shell compared with the thick shell. In Figure 1, the specular reflection is the main component seen in the thick shell, while resonance is the main component in the thin shell. The shell’s outside radii were set to 0.5 m, while the shell thickness was set to 50 linearly spaced values between 0.001 and 0.901. The thickness of the shell was varied for 200 linearly spaced thickness parameter $h=(r_o-r_i)/r_i$ values within the range h = [0.001–0.901]. The radii on the solid spheres were 50 linearly spaced values between 0.4995 m and 0.0495 m, This is denoted in

Table 3. Target properties used for simulating backscattered responses.

Geometry	Interior	Radius	Thickness	Responses Generated
Shell	Air, Octane Oil	0.5 m	0.001–0.901	400
Sphere	Aluminum, Granite, Stainless Steel, or Tungsten Carbide	0.4995–0.0495 m	-	200

.

Independent and identically distributed (i.i.d.) complex Gaussian noise was added to the simulated backscattered signals. Noise was added for two different signal-to-noise-ratio (SNR) cases: 5 dB and 20 dB. The definition of the SNR in this work is $SNR=10{log}_{10}(P_s/\sigma )$ for the signal power $P_s$ and Gaussian standard deviation $\sigma$. A higher SNR will result in increased classification due to less signal contamination. After the inclusion of additive noise, the backscattered responses were normalized to the unit of energy.

2.2. Signal Segmentation

The starting and stopping indices of the signal were estimated using an alternating hypothesis: Page’s test [36]. The estimator was based on the cumulative sum of the logarithm of the ratio of the probability distribution function (pdf) of the signal present to the noise pdf. A brief description is presented here. Estimation was performed for the magnitude of the data squared. In the case of the noise-only signal, this becomes chi-squared distributed as the sum of the squared i.i.d. standard normal random variables ($N_R=N_I\sim \mathcal{N}(0,1)$) is a chi-squared distribution with two degrees of freedom:

$$Y={\left|N\right|}^2=N_R^2+N_I^2\sim {\chi }_2^2,$$

(1)

The distribution takes the reduced form of

$$f_0\left(x\right)=\frac{1}{2}{e}^{-x/2}.$$

(2)

When the signal is present, the distribution can be described by an i.i.d. complex Gaussian with constant means and variances, where $S_R=S_I\sim \mathcal{N}(\mu ,{\sigma }^2)$. The random variable will have a non-central chi-squared distribution with two degrees of freedom and a non-centrality parameter $\delta =2{(\mu /\sigma )}^2$. The density function can be written as

$$\begin{array}{cc}\hfill f(x;\delta )& =\sum _{k=0}^{\infty }\frac{{\delta }^k{e}^{-\delta /2}{x}^k{e}^{-x/2}}{{(k!)}^2{2}^{2k+1}}.\hfill \end{array}$$

(3)

A locally optimal detector was employed as an approximation of the log-likelihood ratio due to the infinite summation. This is a linear function of the data, as only the first two terms of the summation are nonzero [36]. A bias was determined by using Dyson’s method [36]. The signal was segmented using a cumulative summation of the data, which takes the iterative form of

$$T_k=\max [0,T_{k-1}+g\left(x\right)]$$

(4)

where $T_0=0$ and $g\left(x\right)$ is the locally optimal detector with bias. When $T_k$ is greater than some threshold, the signal has been detected. A similar method can be used to find the end of the signal. Due to the cumulative summation, there is a delay associated with the start of the signal. To mitigate loss of the start of the signal, an empirically chosen 1 ms value was selected for the start of the signal index. Figure 2 shows an example of the segmented signals for the granite solid, shell filled with air, and shell filled with oil.

2.3. Time-Frequency Representations

After the data generation, the backscattered responses of the targets were projected into time-frequency representations. Two time-frequency representations were employed: spectrograms generated from STFTs and scalograms generated from the CWT. The STFT and CWT representations were chosen, as they are two standard methods commonly used. The continuous-time STFT is described below to demonstrate the similarities to the continuous wavelet transform. The STFT of a signal $x\left(t\right)$ is defined as follows:

$$STFT(\tau ,\omega )={\int }_{-\infty }^{+\infty }x\left(t\right)w(t-\tau )e^{-j\omega t}dt$$

(5)

where $\tau$ and $\omega$ correspond to the time and frequency, respectively, and $w\left(t\right)$ is a windowing function used to reduce spectral leakage. The Hamming window function of length of 50 samples was used in this application. A 90% overlap was employed with an FFT equivalent to the next power of two of the length of the signal. Typically, this was a 2048 point FFT. The spectrogram is the squared magnitude of the STFT, or

$$S(\tau ,\omega )={\left|STFT(\tau ,\omega )\right|}^2.$$

(6)

The spectrograms of the segmented backscattered responses generated from stainless steel 347 with a 20 dB SNR are displayed in Figure 3. The spectrograms are depicted for a shell filled with air in Figure 3a, a shell filled with oil in Figure 3b—both having an outside radius of 0.50 m and inside radius of 0.49 m—and a solid sphere with a radius of 0.49 m in Figure 3c. Figure 3d shows the spectrogram for a shell with an inner radius of 0.343 m filled with air. This inset was included to demonstrate the morphing of resonances as a function of the target size. Notice how the resonances varied depending on the target geometry (shell or solid), the inside material (air, octane oil, or solid), and the target size. For example, there were distinct and localized resonances between 0 and 10 kHz for the shell filled with air (Figure 3a), while there were complex resonances across the simulated frequency band for the shell filled with oil (Figure 3b). When visually comparing a shell filled with air at different thicknesses (Figure 3a,d), the resonances spread across the simulated frequency band when the shell was thick. The thick shell spectra (Figure 3d) was most similar to the solid sphere (Figure 3c), consisting of a strong specular reflection and wide resonances in this case.

The CWT for the scale parameter $a>0$ and translation parameter b is

$$CWT(a,b)=\frac{1}{\sqrt{\left|a\right|}}{\int }_{-\infty }^{+\infty }x\left(t\right){\psi }^{*}\left(\frac{t-b}{a}\right)dt.$$

(7)

where the previously mentioned notation holds and $\psi (·)$ is the wavelet that can contract and dilate, allowing for a variable time-frequency resolution. The Morlet wavelet was employed in this research and can be described as

$${\psi }_0\left(t\right)={\pi }^{-1/4}{e}^{j{\omega }_{0}t}{e}^{-t^2/2}$$

(8)

for a frequency ${\omega }_0=2\pi f_0$ and time t. The Morlet wavelet belongs to the class of analytic wavelets due to its complex oscillation. This wavelet is sometimes referred to as a Gabor wavelet and is a Gaussian-modulated with a plane wave [37]. The oscillation provides frequency localization, while the Gaussian envelope provides time localization [38]. The scalogram is the magnitude squared of the CWT:

$$C(a,b)={\left|CWT(a,b)\right|}^2.$$

(9)

The scalograms of the segmented, backscattered responses generated from stainless steel 347 with a 20 dB SNR are displayed in Figure 3e–h. Scalograms are depicted for a shell filled with air in Figure 3e, a shell filled with oil in Figure 3f—both having an outside radius of 0.50 m and inside radius of 0.49 m—and a solid sphere with a radius of 0.49 m in Figure 3g. Figure 3h shows the scalograms for a shell with an inner radius of 0.343 m filled with air. This inset was included to demonstrate the morphing of resonances as a function of the target size. Notice how the resonances varied depending on the target geometry (shell or solid), the inside material (air, octane oil, or solid), and the target size. Similar visual patterns described when examining the spectrogram representations held in describing the scalogram representations. The white lines on the scalograms indicate the cone of influence (COI). Features present outside of the COI should be questioned, as they may be caused by edge effect artifacts and boundary effects due to the wavelet being stretched outside the observation interval.

The constant window size force spectrograms have a fixed resolution and may not accurately capture information on non-stationary signals. The scalograms had variable resolutions due to the scaling and translation parameter. This manifested as high-frequency components having good time resolutions but poor frequency resolutions and vice versa. This trade-off can be seen in Figure 3e–h, where the low-frequency component is spread over time and high-frequency component is localized at a time. Typically, scalograms are a more accurate representations of non-stationary signals.

2.4. Multi-Class Classification

A convolutional neural network (CNN) was used for classification. Other deeper network architectures, such as YOLO, ResNet, and iterations of them, have been used for similar tasks (as described in [8,39,40]). However, satisfactory classification results were achieved using the following network structure while explaining the network choices and relating the interpretation into the physical domain. The filter size was set to $5\times 5$. Batch normalization was used after each two-dimensional convolution, and a max pooling layer with a ReLU activation function followed [41]. The max pooling decreased the output size by a factor of two. After the last ReLU activation was a max pooling layer followed by a dense layer with 128 neurons. Lastly, a fully connected layer with the number of neurons equal to the number of classes and a softmax activation were included.

All data were resized to $128\times 128$ via bicubic interpolation along with a low-pass anti-aliasing filter prior to training and testing using MatLab’s imresize function [29]. The data were split as 80% for training and validation and 20% for testing. A common random seed was used. The networks were trained using the Adam optimizer with the default parameters [42]. Early stopping was employed to combat overfitting [43]. The early stopping strategy employed in this work was patience-based for the validation loss; that is, if the validation loss increased across 10 training cycles, then training terminated. The validation loss was calculated for every other training cycle. The cross-entropy loss function was minimized. The multi-class cross-entropy loss function for the true label y and predicted output ${\widehat{y}}_n^c$ at observation n for class c is defined as follows:

$$L(y,\widehat{y})=-\frac{1}{N}\sum _{n=1}^N\sum _{c=1}^C(y_n^{c}ln{\widehat{y}}_n^i)+(1-y_n^c)ln(1-{\widehat{y}}_n^c).$$

(10)

The number of convolutional layers, number of filters, and the initial learning rate were determined through Bayesian optimization by assuming a prior Gaussian process [44]. The acquisition function employed was expected to show improvement and iterated 15 times [44]. The 80% subset of training data was split into 5 cross-validation folds. Each iteration of the Bayesian optimization was performed on the five folds. The average loss across the five folds was minimized. This allowed for the network topology to be optimized for the classification task while ensuring generalizability. More information on the Bayesian optimization method employed in this research is included in Appendix A. Bayesian optimization was chosen over random searches for hyperparameter tuning to minimize the network tuning time [45]. Figure 4 depicts an example of the network structure, where $N_f$ and $N_l$ are the number of filters and the number of layers determined through Bayesian optimization, respectively.

Networks with the goal of classifying material types (aluminum, tungsten carbide, stainless steel 347, or granite), target geometries (solid or shell), and the interior fluid of the target (solid, air, or octane oil) were trained and tested. This was carried out to determine which components of the representations were most influential in the classifier decisions and to uncover any relations between the network structure and classification task.

Table 4. Number of convolution layers and filters and the initial learning rates (LRs) for the CNN chosen through Bayesian optimization for classification of the geometry (sphere or shell). The hyperparameter search space is also reported.

	Spectrogram			Scalogram
SNR	Layers	Filters	LR	Layers	Filters	LR
Search Space	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]
5 dB	2	4	$2.48\times {10}^{-4}$	1	6	$7.36\times {10}^{-4}$
20 dB	4	9	$2.01\times {10}^{-3}$	1	4	$4.24\times {10}^{-4}$

,

Table 5. Number of convolution layers and filters and the initial learning rates (LRs) for the CNN chosen through Bayesian optimization for classification of the interior filling (solid, air, or oil octane). The hyperparameter search space is also reported.

	Spectrogram			Scalogram
SNR	Layers	Filters	LR	Layers	Filters	LR
Search Space	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]
5 dB	4	5	$5.24\times {10}^{-3}$	2	3	$7.68\times {10}^{-3}$
20 dB	1	9	$6.51\times {10}^{-4}$	1	3	$9.80\times {10}^{-4}$

and

Table 6. Number of convolution layers and filters and the initial learning rates (LRs) for the CNN chosen through Bayesian optimization for classification of the material (aluminum, granite, stainless steel 347, or tungsten carbide). The hyperparameter search space is also reported.

	Spectrogram			Scalogram
SNR	Layers	Filters	LR	Layers	Filters	LR
Search Space	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]	[1, 4]	[2, 9]	[$1\times {10}^{-5}$, $1\times {10}^{-1}$]
5 dB	4	9	$9.61\times {10}^{-4}$	2	4	$6.67\times {10}^{-4}$
20 dB	2	6	$1.13\times {10}^{-4}$	2	8	$9.37\times {10}^{-4}$

show the hyperparameter search space and the optimizer results for the networks. The hyperparameters reported in Table 4, Table 5 and Table 6 were used to generate the results discussed.

Upon examination of the selected hyperparameters in Table 4, Table 5 and Table 6, the networks trained on scalograms typically required fewer layers and filters. This may be due to the scalogram representation having better discriminatory features when compared with the spectrogram representation. The spectrogram networks have increased capacity, as the network requires additional parameters in order to decode the spectra. A similar remark can be made for the networks with regard to the SNR, as a lower SNR requires typically requires increased network capacity. This is due to the network needing to filter out the noise prior to finding discriminatory features.

Computations were performed in Matlab 2022a v9.12.0 with the Deep Learning Toolbox [29] using an Intel(R) Core(TM) i9-10900K CPU with 32 GB of RAM and an NVIDIA Quadro P1000 GPU. There were 15 iterations of the optimization process, resulting in a total of 75 trained networks (across 5 folds) per hyperparameter search. Each optimization took roughly 35 min to perform, and these are reported in

Table 7. Hyperparameter optimization time for networks trained on data with 5 dB SNR.

Classification Task	Spectrogram Time (min:s)	Scalogram Time (min:s)
Geometry	38:11	35:47
Interior material	33:11	35:05
Material type	35:44	34:00

and

Table 8. Hyperparameter optimization time for networks trained on data with 20 dB SNR.

Classification Task	Spectrogram Time (min:s)	Scalogram Time (min:s)
Geometry	36:10	33:12
Interior material	35:41	32:29
Material type	42:05	32:39

for the 5 dB and 20 dB SNRs, respectively. While the optimization times do not facilitate real-time applications, the parameters reported within Table 4, Table 5 and Table 6 may aid future researchers by reducing their hyperparameter search space.

2.5. Performance Metrics

The overall accuracy (OA) can be used to determine on average how well a classifier is performing. It is defined as the number of correctly classified responses divided by the total number of attempted classified responses and multiplied by 100%:

$$\mathrm{OA}=\frac{\#\mathrm{correctly}\mathrm{classified}}{\#\mathrm{attempted}\mathrm{classified}}·100\%.$$

(11)

Confusion matrices were used to see the incorrectly classified predictions for each class. On one axis is the true label, while the other has the predicted label. The values indicate the amount of responses classified at that specific instance. A classifier that is 100% accurate will have values only on the diagonal. The confusion matrices in this work were row-normalized so that the value depicted was a percentage of the true class label.

2.6. Gradient-Weighted Class Activation Mapping (Grad-CAM)

Selvaraju et al. developed an explainable technique: gradient-weighted class activation mapping (Grad-CAM) [25]. Other XAI techniques, such as local interpretable model-agnostic explanation (LIME) [46] or CNN feature map examinations [11], have been employed to explain sonar classification choices. This work takes a similar approach in the explanation but uses a model-specific technique to relate the classifier choices to the physical domain. Grad-CAM falls in the general class of gradient-based techniques and uses backpropagation on trained networks to explain network choices. Grad-CAM is a generalization of class activation mapping (CAM), described in [47], and does not require a specific network structure.

Following the notation in [25], Grad-CAM calculates an input relevance score by first sending a test image through the trained network, resulting in an output score $y^c$ for class c. The gradient of the output score $\partial y^c/\partial A^k$, with respect to the kth feature map activation $A^k$ of a specified convolutional layer, is then computed. Typically, the last convolutional layer is chosen. The gradients are then global average-pooled across the channel depth, essentially averaging all pixels in the feature map. These averaged gradients ${\alpha }_k^c$ can be thought of as weights

$${\alpha }_k^c=\frac{1}{Z}\sum _i\sum _j\frac{\partial y^c}{\partial A_{ij}^k}$$

(12)

for the width and height indices i and j, respectively, with Z total pixels. The global average-pooling is apparent through the two summations and the division of Z. The averaged gradient weights are then multiplied by the corresponding feature activation maps. The ReLU activation is used for rectification and ensures only positive influences are considered [25]. The output of the ReLU function is of the same size as the convolutional layer and is upsampled to the input image size.

Intuitively, when a gradient is large, the output score depends on this location. This is essentially what Grad-CAM is depicting. Visualization is performed by using a heat map to highlight class-specific features and provide a relevance value for the input pixels.

3. Results

The overall classification results for the material type, geometry, and interior fill are reported for the different SNRs. Next, the interesting confusion matrices are examined. Lastly, the trained classifiers are examined, using Grad-CAM to explain the network choices and relate the informative features to the physical domain.

3.1. Overall Accuracy

The overall accuracy and standard deviation for the CNNs trained on the simulated data to classify target geometry are in

Table 9. The overall accuracy and standard deviation for classification of geometry of the target (solid or shell). The largest average overall accuracy (OA) and corresponding standard deviation (Std. Dev.) are denoted in bold.

	Spectrogram		Scalogram
SNR|Metric	OA (%)	Std. Dev. (%)	OA (%)	Std. Dev. (%)
5 dB	96.83	1.34	97.75	1.19
20 dB	97.75	0.68	97.91	1.93

, while those for the interior material are shown in

Table 10. The overall accuracy and standard deviation for classification of the inside of the target. The largest average overall accuracy (OA) and corresponding standard deviation (Std. Dev.) are denoted in bold.

	Spectrogram		Scalogram
SNR|Metric	OA (%)	Std. Dev. (%)	OA (%)	Std. Dev. (%)
5 dB	78.50	2.82	76.67	2.18
20 dB	78.58	2.18	78.16	1.29

, and those for the material type are in

Table 11. The overall accuracy and standard deviation for classification of material type. The largest average overall accuracy (OA) and corresponding standard deviation (Std. Dev.) are denoted in bold.

	Spectrogram		Scalogram
SNR|Metric	OA (%)	Std. Dev. (%)	OA (%)	Std. Dev. (%)
5 dB	83.50	1.79	89.25	2.40
20 dB	83.50	1.91	90.91	1.32

. The reported results were generated by training a network using the hyperparameters listed in Table 4, Table 5 and Table 6 on the training and validation data used for the cross-validation Bayesian optimization. Testing was performed using the disjointed 20% data subset. The reported results are from across 10 random seeds to determine a statistical description of the network.

Typically, a higher SNR results in a higher average classification accuracy and lower standard deviation. The scalograms tend to have a higher average classification accuracy. However, there was no discernible trend in the corresponding standard deviations. Both trends were expected, as the increased SNR provided a cleaner representation for the network to classify, and the scalogram representation provided increased localization of the robust features due to the contraction and dilation of the wavelets. There was a negligible difference between the two SNR classification accuracies, accounting for the standard deviation of the random seeds. This demonstrates the CNNs’ ability to perform noise suppression across a difference of 15 dB. In future work, this can be taken to the extreme, and analysis across different SNR levels can be used to determine the failure mode of the classifier. Given these results, and by leveraging the knowledge learned in the hyperparameter optimization, while scalograms provide negligible increased classification accuracy compared with their spectrogram counterpart, the scalogram networks typically have fewer layers and filters, which decreases the number of learned parameters, the complexity of the network, and the network training time.

3.2. Confusion Matrices

The confusion matrices for the highest average accuracies reported Table 9, Table 10 and Table 11 are shown in Figure 5, Figure 6 and Figure 7. The confusion matrices list the classification results across the 10 random seeds. The units of the confusion matrices have been row-normalized to easily determine a percentage per class of correct or incorrect classification. For example, in Figure 5, 3.3% of the shells were misclassified as solid spheres.

The highest classification accuracy occurred when the network was instructed to train based off of the target geometry (solid or shell). The lowest accuracy occurred when the network was instructed to discriminate between the interior materials (air, oil, or solid). These trends were a result of the simulated shells having similar acoustic spectra, regardless of the internal fluid, when the shells were thick. We recall that the radii of the shells were randomly split across the training and testing dataset, as described in Section 2.4. These classification results represent an average across the randomly separated radii. However, the classification result may still be explained through examination of the different feature representations. An example of this similar response can be seen in Figure 8, where the spectrograms were generated with a thick shell ($h\sim 0.9$) made from tungsten carbide. Figure 8a was generated by using air as the internal fluid, while Figure 8b used octane oil. There is little visual difference between the two spectra, which in turn confused the classifier. When the shells had a reduced thickness, as seen when comparing the images in Figure 3a,b, the classifier was able to distinguish between the interior fluid. The thickness where the visually similar acoustic spectra began to differ occurred sooner in the scalogram representation. This gave the network more variability in the spectra providing the increased accuracy when the network was trained on the scalograms. A classifier trained to discriminate between a solid and shell will automatically group the air- and oil octane-filled shells in a class, resulting in the increased classification accuracy seen in Table 9.

3.3. Grad-CAM

Grad-CAM was used to determine what the CNN was selecting as the most influential features for each classification task. The Grad-CAM results are presented as heat maps that show the spatial locations of large gradients. The axes are in units of pixels and representative of the time and frequency axes. The color bar was normalized between 0 and 1. The heat maps are accompanied by the corresponding input image. Note that the input images are not in a decibel scale, as the networks were trained using normalized linear units. The advantage to using Grad-CAM is the location of the class-specific influence on the spectra can be found. These are the locations of the largest gradients. This can provide insight on feature extraction algorithms (i.e., intuitively bias the classifier by forcing it to focus on informative features) and aid in classifier debugging. Examination of the Grad-CAM heat maps on the simulated data allows relations to be drawn between the important features and physical scattering mechanisms.

Grad-CAM was performed on the CNNs for the reported confusion matrices. The resulting heat maps were visually similar to the input images for the classification of the interior material and target geometry. This was due to the initial convolution layers detecting semantically meaningful objects, and both networks were one convolutional layer deep. An example of these heat maps and the corresponding test images are shown in Figure 9. The heat maps depicted are for the correctly classified air-filled shell (Figure 9a), octane oil-filled shell (Figure 9b), and an air-filled shell that was incorrectly classified as an octane oil-filled shell (Figure 9c). The corresponding input test images are depicted in Figure 9d–f, respectively. The corresponding classification scores are listed in the title of the Grad-CAM heat maps.

The CNN depends on the first feature component, typically due to specular or geometric scattering, when choosing the air class and higher-order, more complex resonances when choosing the octane oil-filled shell class. This is evident in the comparison of the heat maps in Figure 9a,b. Insight into the CNN model’s failure in misclassification can be gained through examining Figure 9c. Figure 9d–f depict the corresponding input test image. This test image was incorrectly classified as an octane oil-filled shell and had a classification score of 0.560, which was split between the two classes. The CNN found a similar oil structure within the input image but also relied on features external to the resonances, as is evident from the highlighted features on the exterior of the image. To mitigate this failure mode of the CNN, a segmentation processing step could be included in the classification pipeline. This would likely increase the classification accuracy and decrease the training time, since the network mainly relies on semantic clues within the image.

Figure 10 depicts the heat map for the CNN trained on spectrograms to recognize the target geometry. The input test image was the same as the octane oil input image in Figure 9e. A comparison between Figure 9b and Figure 10 provides insight into the features that are important to different classification tasks. The classifier used to discriminate between interior materials depends on higher-order, more complex resonances (Rayleigh or Lamb surface waves), while the classifier trained to discriminate between the target geometries relies on the shape of the specular scattering and first resonance. This can be leveraged when designing a classification pipeline, as the classification task must be taken into account. If the target geometry is being classified, then the feature representation can focus on the start of the signal and the specular reflection rather than its entirety. This provides an automatic dimensional reduction in the signal truncation, resulting in decreased classifier complexity and training time.

To determine how the varying random seed impacted the classification accuracy, Grad-CAM was performed on the CNNs trained on the scalograms with a 20 dB SNR to classify the material type. The resulting heat maps for the first five random seeds and the test image are shown in Figure 11. Figure 11a–e depicts the Grad-CAM heat maps for various random seeds, and Figure 11f depicts the corresponding test image. The remaining five seeds were omitted due to space constraints. We recall that this network consisted of two convolutional layers with eight feature filters, meaning the network still relied on semantically meaningful objects. In all cases, the CNN depended on the tungsten carbide specular reflection for discrimination. The CNN was additionally using the resonances caused by Rayleigh or Lamb surface waves, but they were not as influential. In some cases, the CNN relied on additional low-frequency features (Figure 11a,c), while others depended on localized resonances (Figure 11b,d,e). An additional processing step could be included in the classification pipeline to highlight the high-frequency resonances and force the CNN to depend on these features, which are known from this analysis to be consistent across random seeds and impact classification.

Throughout examination of the Grad-CAM results, in order to highlight features that were determined to be relevant, various forms of augmentation could be used to increase the dataset and robustness of the network. Additionally, the insight gained from the heat maps allow recommendations to be made when designing different classifiers. For example, if the target geometry is to be classified, then the majority of the information relevant to the network is within the specular scattering, as seen in Figure 10. This places less relevancy on the end of the return, and the signal detector can be adjusted to focus on the initial return. This would automatically decrease the input dimensionality and focus the network beforehand on the relevant features, thereby intuitively positively biasing the results and decreasing the training time.

4. Conclusions

Simulated backscattered responses of various materials, shapes, and sizes were generated prior to projection onto two time-frequency representations: spectrograms generated using the short time Fourier transform and scalograms generated using the continuous wavelet transform. Multiple convolutional neural networks were trained to classify the material type, target geometry, and interior material of the target. Bayesian optimization was used to determine the number of layers, number of feature maps per layer, and the initial learning rate. This resulted in classifiers that were optimized for the specified classification task. The trained networks were examined by using an explainable artificial intelligence technique—gradient-weighted class activation mapping (Grad-CAM)—to determine the post-training features used for classification. The Grad-CAM results were depicted using heat maps, representing the spatial locations of large positive gradients.

The scalogram representation provided a negligible increase in the average classification accuracy over the spectrograms. The networks trained to discriminate between target geometries resulted in the highest classification. The networks trained to discriminate the target’s interior material had the lowest accuracy. The main feature highlighted when examining the CNNs trained to classify the interior of the material was the specular reflection, with a small portion of the resonances being used for classification. This network contained one convolutional layer. Typically, the initial layers of the network will lock onto spatially important features (such as contours), while deeper layers separate out things that are not visually apparent. The CNN used to classify the material type was two convolutional layers. These CNNs highlighted the resonances for discrimination, but the specular component was also still being used. The analysis performed throughout this investigation can be leveraged when designing classification pipelines by amplifying the meaningful scattering mechanisms and suppressing the less-informative features. Possible classifier failure mode mitigation techniques were discussed, and recommendations for how to intuitively and positively bias classifiers were provided.

Future work can further relate the network-determined features through first simulating the modal rigid and soft residuals. The Bayesian optimization topology can have increased classification to follow the best practices in CNN topology, such as by increasing the number of filters with each additional convolutional layer. Additional complexities can be included in the classification pipeline, such as coating the shell and simulating cylinders and investigating different additive noise models at different SNRs. The latter analysis would further investigate the impact of the SNR on the network analysis and explainability technique. The thickness of the shell and relation to the classification results can be further investigated. Deeper networks can be trained and examined to see if separation between the different scattering mechanisms occurs. Lastly, the analysis of the trained networks can be expanded. The results presented were qualitative through visual examination of the Grad-CAM heat maps. An image similarity measurement, such as the structural similarity index measure (SSIM) [48] or feature similarity index (FSIM) [49], could be used to quantify intra-class features and compare them to inter-class features.