Contribution to Uncertainty Propagation Associated with On-Site Calibration of Infrasound Monitoring Systems

Abstract

To improve the confidence and quality of measurements produced by regional and international infrasound monitoring networks, this work investigates a methodology for propagating uncertainty associated with on-site measurement systems. We focus on the propagation of sensor calibration uncertainties. The proposed approach is applied to synthetic infrasound signals with known back azimuth and trace velocity, recorded at the array elements. Relevant input uncertainties are investigated for propagation targeting the incoming signals (noise), instrumentation (microbarometers, calibration system, wind noise reduction system), and the time-delay-of-arrival (TDOA) model (frequency band). Uncertainty propagation is performed using the Monte Carlo method to obtain the corresponding uncertainties of the relevant output quantities of interest, namely back azimuth and trace velocity. The results indicate that, at high frequencies, large sensor uncertainties are acceptable. However, at low frequencies (<0.1 Hz), even a 2${}^{\circ }$ sensor phase uncertainty can lead to errors in the back azimuth of up to 5${}^{\circ }$ and errors in the trace velocity of 20 m/s.

1. Introduction

Atmospheric infrasound is comprised of pressure waves in the atmosphere with frequencies between the acoustic cut-off of approximately 3.3 mHz (a physical limit based on the influence of gravity and a lack of compressibility) and 20 Hz (the lower limit of human hearing) [1,2]. Infrasound can be generated in the atmosphere by many different anthropogenic and natural sources, such as seismic activity [3], atmospheric convection [4], volcanic activity [5,6,7], wind turbines [8], hydroelectric installations [8], etc. A global network of infrasound stations, called the International Monitoring System (IMS), is being developed by the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO) to monitor nuclear testing activity across the globe [8,9], with 53 of 60 planned infrasound stations currently certified [10]. The direction of the infrasound source is determined by triangulating the direction of signal propagation using multiple sensors in an array [11,12,13], also called a sensor array.

The uncertainty evaluation of on-site atmospheric infrasound measurements produced by networks, such as IMS, is a core challenge for improving the confidence and quality of measurements. In practice, the level of confidence associated with a measurement result is based on the instrument’s ability to be stable over time and accurate, which implies assessing its measurement error. Calibration enables rigorous assessment of measurement error and allows for the establishment of corrections to the measurements if necessary. In practice, the calibration process is based on a hierarchy or sequence of calibrations using various standards, from the laboratory to the field (primary, secondary, working), which link the calibrated sensor to the International System of Units (SI). The measurement produced by the sensor, thus calibrated, meets the fundamental concept of the metrological traceability of the measurement. Generalizing this approach to all infrasound sensors spread across a network, as well as across several networks, greatly increases the guarantee of the control of the measurement results over all of the networks, and makes it possible to establish absolute comparisons of the relevant results.

The objective of this paper is to provide guidance for the evaluation of the uncertainty associated with the estimation of the back azimuth and trace velocity of infrasound waves, taking into account the uncertainty associated with the on-site calibration of the infrasound monitoring system, which is a new contribution in this field. This process is equivalently referred to as uncertainty propagation.

There are several different algorithms used for determining the back azimuth and trace velocity from the incoming signals, such as the progressive multi-channel correlation (PMCC) algorithm [12,14], frequency–wavenumber (f–k) technique [15,16], and the multi-channel maximum likelihood (MCML) algorithm [17]. For this paper, we will focus on the time-delay-of-arrival (TDOA) algorithm [18,19], which follows a similar approach to the PMCC algorithm by determining the differences in the arrival times of the signals at different sensors to calculate the back azimuth and trace velocity signals. The difference is that the TDOA algorithm is simpler since it considers all of the sensors equally, and does not progressively add sensors as done in the PMCC algorithm. These approaches will be discussed in more detail in Section 2.1. The main sources of uncertainty affecting the estimation of back azimuth and trace velocity of an incoming infrasound wave are displayed in the flowchart in Figure 1 and described below. A more thorough review of uncertainty sources is provided in Section 2.2.

Previous research studies have extensively investigated the uncertainty of infrasound arrays. For example, Szuberla et al. [20] showed the uncertainties in back azimuth and trace velocity for several different infrasound array designs (including arrays with missing or damaged sensors) as a function of the source back azimuth and trace velocity, with uncertainties as high as 2${}^{\circ }$. There has also been research on the effects of sensor timing errors and defective sensors with high noise levels using various parameterization algorithms [21]. Moreover, De Angelis et al. [22] have shown that uncertainties for real events, such as volcanoes, can be determined using the variances and covariances of the signals, which is useful for better estimation of the source location. In this study, we investigate the effects of sensor calibration to better understand how sensor uncertainties affect the wave parameter estimates and provide more insight into acceptable levels of sensor uncertainty.

Atmospheric conditions can have great effects on the propagation of infrasound sources and, therefore, the detection of these signals [23,24,25]. The formation of waveguides can allow for the observation of signals from great distances or prevent the propagation of sources in certain directions [4,23,26]. Additionally, atmospheric refraction can result in errors in back azimuth measurements relative to the source location [4,27]. Instrumental effects also affect the results, such as loss of sensor elements or increased noise at each sensor [28]. Local environmental conditions, such as the wind, can increase the noise observed by the sensor, increasing the resulting uncertainty on back azimuth and trace velocity of the infrasound sources [9,29] and can cause deviations of the back azimuth from its true value. For this study, only the effects of increased noise are considered, not the atmospheric propagation, as this would be independent of the sensor calibration.

Wind-generated turbulent noise is a challenge for many infrasound stations [30,31,32]. There are several different methods that can be used to reduce this wind noise, while maintaining the signal amplitude, allowing for an increased signal-to-noise ratio (SNR). Some of the more common methods are rosette pipe filters, [30,33,34], microporous houses [29], wind barriers, and domes [32,35]. For this study, we focus on the rosette-style pipe filtering systems used as the default wind noise reduction system (WNRS) for the IMS [9]. It is essential to fully characterize the effects of the WNRS on the signal and, therefore, calibration of these systems is necessary. Gabrielson [31] introduced a method of using a co-located, calibrated reference sensor as a comparison calibration source, which was further refined using across-array coherence to provide low-frequency amplitude estimates [36]. Other studies have used the coupling between infrasound and seismic waves to allow seismometers to be used as a calibration reference for the WNRS [37,38]. In this study, we focus on the Gabrielson approach, which is described in more detail in Section 2.3.2.

Additional uncertainty is caused by the microbarometers. Although there are many different microbarometers that are (or could be) used in infrasound stations, we will focus on the MB2005 and MB3 microbarometers developed by the Commissariat à l’Energie Atomique et aux Energies Alternatives (CEA). MB2005 and MB3 have similar calibration uncertainties, both measuring dynamic pressure up to several Hz; however, the MB3 has much lower self-noise at higher frequencies [9,39,40]. The MB3’s reduced high-frequency self-noise results in self-noise levels below the IMS low noise model for the entire IMS band of interest (0.01 to 4 Hz) [9]. The conversion of the analog signal to a digital one, performed by the digitizer, is an additional source of uncertainty in the measurement chain but is not considered in this paper. Due to the significant variations between in-lab and in situ conditions caused by the WNRS and local environmental conditions, only in situ calibration of the sensor (microbarometer and WNRS) coupled with a calibrated (primary or secondary) reference sensor can provide reliable calibration information for the sensor under test, including the uncertainty. The Gabrielson method [31] performs such an in situ calibration and provides such calibration information but is currently used only to check that the amplitude and phase belong to tolerance bounds ($\pm 5\%$). We discuss this in further detail in Section 3.3.

The mathematical model that describes the relationship between the signal of interest $X\left(t\right)$, which acts as input to the sensor array, and the corresponding observed output signal $Y\left(t\right)$, is denoted as [41]

$$Y\left(t\right)=H\left[X\left(t\right)\right]$$

(1)

where, for this application, H is the transfer function of the sensor array.

The core of this paper assumes that the characteristics of the convolution (1), i.e., the transfer function H that informs on amplitude and phase uncertainty, are known from in-situ calibration of the sensor with the Gabrielson method [31] in the frequency domain. It is important to note that in this paper, sensor array uncertainty is a shorthand notation that covers both amplitude and phase uncertainties in the frequency domain. The quantity of interest (QoI) W, here chosen as the back azimuth or the trace velocity of the signal of interest X, is defined as the following integral

$$W=\int \eta (X)\pi (X)\mathrm{d}X,$$

(2)

where $\pi \left(X\right)$ denotes the uncertainty distribution of X estimated from (1) resulting from the sensor calibration uncertainty and $\eta \left(X\right)$ denotes a run of the TDOA algorithm that produces an estimated value of the QoI as described in Section 2.1.

The process of estimating W and its associated uncertainty is referred to as uncertainty propagation in this paper and is described in Section 2.3. It involves Monte Carlo sampling in the uncertainty distribution of the amplitude and the phase of the sensors in the frequency domain (see Section 2.3.4). The uncertainty associated with the estimation of the QoI can be represented by a probability distribution (see Figure 1). The flowchart of the complete methodology is displayed in Figure 1 and will be referred to throughout the paper.

The performance of the Monte Carlo estimators is assessed in terms of bias (also referred to as error), standard deviation, mean absolute error (MAE), and root mean square error (RMSE). Moreover, this method can be readily extended to any uncertainty source that is quantifiable in the frequency domain and to other propagation models (TDOA in this paper).

The methodology of uncertainty propagation is demonstrated in synthetic signals for existing array designs and experimentally determined calibration information. This approach allows for displaying the probability distributions of bias (error) of the estimated quantities of interest for a more meaningful interpretation of results. A sensitivity study is performed to investigate the effect of sensor phase error (as part of the sensor calibration) and the effects of some hard-to-quantify input uncertainty sources on the estimated quantities of interest.

2. Materials and Methods

In Section 2.1, we recall the principles of back azimuth and trace velocity estimation using the time-delay-of-arrival (TDOA) algorithm. These principles are inversely used to generate the synthetic signals with the known back azimuth and trace velocity in the application, according to the methodology described in Appendix A. In Section 2.2, uncertainty sources relevant to the estimation of back azimuth and trace velocity of an incoming infrasound signal are investigated to justify the focus of the work on sensor calibration uncertainty. In Section 2.3, we present the proposed approach for propagating uncertainty associated with the sensor via the Monte Carlo method.

2.1. Estimation Method with the Time-Delay-Of-Arrival (TDOA) Algorithm

To calculate the back azimuth and trace velocity of the source, TDOA is used [42,43,44]. This algorithm considers the relative arrival time of a signal and, given the known geometry of the array, calculates the slowness vector projected into the plane of the sensor array. This is the same basic algorithm used by PMCC, but it considers all of the sensors at once. The PMCC algorithm starts with a small subnet of sensors, and searches for a signal. If a signal is found, it will progressively add additional sensors for which the signal is observed until all of the sensors are considered. Additionally, to reduce the errors caused by aliasing, as more sensors are added, the time delay of arrival is constrained to match that expected from that calculated using the smaller subnet [12]. This allows for the removal of the aliasing-related ambiguity while allowing for signals, which are coherent over a small sub-array but not coherent over the whole array. The drawback is that this method is more complicated and requires longer computing times. Given the large calculations needed for this study, it was decided that using the simpler TDOA algorithm would be prudent.

At its most basic level, the TDOA is the difference in the signal arrival time between two sensors. This is determined by finding the maximum correlation between the observed signals of the two sensors. The maximum is determined using a parabolic fit of the three points nearest to the maximum. This is repeated for each combination of sensor pairs. By considering three sensors, such that there are three permutations of sensor pairs, one arrives at the closure relationship, such that

$$r_{ijk}=\Delta t_{ij}+\Delta t_{jk}+\Delta t_{ki},$$

(3)

where $\Delta t_{ij}$ is the TDOA between sensors i and j [12]. It should be noted that, for a perfect plane wave with no noise, the closure relation will be 0.

To incorporate all of the sensor pairs, the consistency $C_n$ is used, such that

$$C_n=\sqrt{\frac{6}{n(n-1)(n-2)}\sum _{i>j>k}{r}_{ijk}^2},$$

(4)

where n is the total number of sensors included in the calculation. This consistency can be used as a threshold to determine if a measurement is due to a real signal of interest (SOI) or is caused by noise [11]. Acceptable consistency thresholds have been shown to typically fall between 0.01 s and 1 s [45,46]. For the purposes of this study, a value of 0.1 s was chosen (unless otherwise stated) as an appropriate consistency threshold as it falls in the geometric center of the acceptable consistency threshold range.

The next step is to calculate the slowness vector of the incoming SOI based on the arrival times. The slowness, $\overrightarrow{S}$, can be determined from the TDOA, such that

$$\overrightarrow{S}={\left({\mathbf{XY}}^T\mathbf{XY}\right)}^{-1}{\mathbf{XY}}^T\overrightarrow{\tau },$$

(5)

where

$$\mathbf{XY}=\left[\begin{array}{cc}\Delta x_1& \Delta y_1\\ ⋮& ⋮\\ \Delta x_m& \Delta y_m\end{array}\right]$$

(6)

are the relative positions of the sensor pairs, denoted by the subscript, with a corresponding time delay of arrival of

$$\overrightarrow{\tau }=\left[\begin{array}{c}{\tau }_1\\ ⋮\\ {\tau }_m\end{array}\right]$$

(7)

for m sensor pairs.

From this slowness, the trace velocity $v_h$ is

$$v_h=\frac{1}{\left|\overrightarrow{S}\right|},$$

(8)

and the back azimuth $\theta$ is

$$\theta =arctan\left(\frac{S_x}{S_y}\right).$$

(9)

To determine the values of back azimuth and trace velocity as functions of time and frequency, the time series are first filtered into 26 1/3-octave frequency bands between 0.01 Hz and 4 Hz, and then divided into multiple windows of varying lengths depending on the frequency band. This updated version of frequency bands is based on the proposal by Hupe et al. [10]. A narrowband Chebyshev bandpass filter of order 2 with a ripple factor of 0.01 is used for filtering. The window lengths are chosen such that a minimum of 10 periods will be observable in the time series for the mean frequency of the given frequency band.

TDOA produces output arrays for consistency, back azimuth, and trace velocity as functions of frequency and time as displayed in Figure 2. The TDOA analysis is performed using the same 26 1/3-octave frequency bands described in Hupe et al. [10]. The window sizes are adapted to the frequency, such that they are 10 times the period. Figure 3 shows the third-octave frequency bands and corresponding window lengths used in Figure 2 and for the rest of this study. An overlap of 50% of the window length is used.

For each frequency band, the back azimuth and trace velocity are estimated as the mean of the array values at that frequency for which the corresponding consistency is below a given consistency threshold, set to $0.1$ $\mathrm{s}$ in this application. This estimation process is referred to as $\eta$ in the paper (see, e.g., Figure 1). Note that in the application, the uncertainty on the mean is negligible, and for this purpose is not considered in the paper.

2.2. Analysis of the Measurement Process

As a first step in uncertainty propagation, a comprehensive review of uncertainty sources must be conducted by analyzing the measurement process. For complex measurement processes, such as the estimation of back azimuth and trace velocity of an incoming infrasound wave in this paper, a graphical tool such as an Ishikawa diagram (also called a fishbone diagram) is often necessary. As an example, Figure 4 displays the uncertainty sources considered in this paper that can affect the estimation of the back azimuth and trace velocity. These include uncertainty sources targeting the incoming signals (such as loss of coherence and noise), the instrumentation (such as microbarometers, calibration system, wind noise reduction system, and environmental sensitivity), and the estimation method (TDOA and Monte Carlo method).

A second step in uncertainty propagation involves quantifying the most influential uncertainty sources. Each source of uncertainty is discussed among experts to determine the availability of data or information (such as expertise, calibration results, or tabulated values) needed to quantify the associated uncertainty. For this study, the only quantifiable source of uncertainty is sensor calibration. A deeper investigation into sensor calibration reveals additional sources of uncertainty that should also be included in the Ishikawa diagram, as shown in Figure 4. The quantification and propagation of sensor calibration uncertainty is the main focus of this paper and is addressed in Section 2.3.

All of the other uncertainty sources are either too hard to quantify (at no reasonable cost) or negligible. For example, in the application, we demonstrate in Section 3.1 that the effect of filtering due to TDOA is negligible. A whole set of uncertainty sources regards those defining so-called benchmark conditions, meaning that a continuous description of the effect of the uncertainty source is not of interest, such as the array configuration or the sampling frequency. For these variables, it is more interesting to compare results in different conditions. For this reason, in this paper, we choose to study the effect of the signal/noise ratio (SNR), defective WNRS, and sensor array by treating them as scenario variables.

2.3. Proposed Methodology for Modeling and Propagating Sensor Calibration Uncertainty

The modeling of the microbarometer calibration uncertainty and the sensor calibration uncertainty are extensively described in Section 2.3.1, Section 2.3.2 and Section 2.3.3. Uncertainty propagation is performed with a Monte Carlo simulation method [47,48] described in Section 2.3.4, which applies to quantifiable uncertainty sources, such as sensor calibration, which is the focus of this paper.

2.3.1. Modeling Microbarometer Calibration Uncertainty

The microbarometer transfer function is obtained from the calibration of the microbarometer (see Figure 5), which must be converted from an amplitude and phase into a complex transfer function, given in V/Pa. The amplitude of the response, converted from dB V/Pa, is

$$a_{MB}={10}^{\frac{a_{MB,dB}}{20}},$$

(10)

where $a_{MB,dB}$ is the amplitude given in dB V/Pa. The uncertainty of the amplitude response, also converted from dB V/Pa, is

$$u_{A_{MB}}={10}^{\frac{2a_{MB,dB}}{20}}{\left(\frac{1}{20}ln\left(10\right)u_{A_{MB,dB}}\right)}^2,$$

(11)

where $u_{A_{MB,dB}}$ is the microbarometer uncertainty in dB V/Pa.

The phase of the response, ${\varphi }_{MB}$, and its associated uncertainty, $u_{{\mathsf{\Phi }}_{MB}}$, are directly taken from the calibration curves and converted from degrees to radians. It is important to note that this procedure is applied to all microbarometers used in the study; therefore, $H_0$ or $H_1$ is substituted for $H_{MB}$ in later sections of this paper.

Although this calibration response is for an MB2005, the MB3 has comparable uncertainty since the uncertainty of the calibration is dominated by the calibration apparatus, which is the same for both types of microbarometers.

2.3.2. Determination of the Gabrielson Transfer Function

The determination of the sensor response relative to the reference microbarometer, $H_{cal}$, is obtained using the calibration procedure described by Gabrielson [31]. A reference sensor is placed next to the sensor under test, which includes a pipe-style rosette WNRS. Since the reference sensor does not include a filtering system, the response of this sensor is known via a laboratory calibration (see Figure 5). The relative response of the sensor to the reference ($H_{cal}$) is determined from the ratio of the auto-spectrum of the sensor under test and the cross-power spectrum of the sensor under test and the reference, as follows:

$$H_{cal}=\frac{H_1}{H_0}=\frac{G_{11}}{G_{10}},$$

(12)

where $G_{11}$ is the auto-spectrum of the sensor-under-test signal, $G_{10}$ is the cross-spectrum of the sensor-under-test signal and reference signal, $H_0$ is the response of the reference sensor, and $H_1$ is the response of the sensor under test (microbarometer and WNRS).

The power spectra were determined for short windows of the time series, which were stepped by 50% of the window length. Therefore, many individual measures of the frequency response were determined. The overall response was determined by taking the weighted mean of these windowed responses, and the uncertainty is the standard deviation. This procedure and experiments used to provide the calibration data used in Figure 6, Figure 7 and Figure 8 performed at CEA are described in more detail in Kristoffersen et al. [28].

Figure 6 shows the resulting response of the rosette-style pipe filtering system, including the associated uncertainties. It is worth noting that the dip in amplitude observed between 0.01 Hz and 0.08 Hz is due to partial coherence of the wind noise, as discussed by Gabrielson [31]. Although several methods have been tested to remove this dip (see, e.g., [36]), it does not affect the phase, and the expected impact on the QoI is negligible. Therefore, no additional processing of the WNRS response will be performed in this study.

Testing of WNRS systems with defects was also undertaken in this paper. An example of a defective WNRS, with 8 blocked (out of 32) inlets is shown in Figure 7. This response can be used to replace the response of one of the sensors, in order to see the effects of a defective sensor on the measurement uncertainty. An example of a defective sensor will also be discussed in the results.

2.3.3. Modeling the Uncertainty Associated with the Sensor

$H_{sen}$ is the complex frequency response of the sensor, defined as

$$H_{sen}=H_{cal}{H}_0$$

(13)

where $H_0$ and $H_{cal}$ are the reference microbarometer and the Gabrielson transfer functions, described in Section 2.3.1 and Section 2.3.2, respectively. Hence, by multiplying $H_{cal}$ by the reference microbarometer response, $H_0$, one arrives at the sensor response, given in V/Pa.

Equivalently, the sensor response can be expressed as

$$H_{sen}=A_{sen}{e}^{j{\mathsf{\Phi }}_{sen}},$$

(14)

where $A_{sen}$ and ${\mathsf{\Phi }}_{sen}$ are the sensor’s amplitude and phase distributions obtained from the sensor calibration frequency response, respectively.

The best estimates of the amplitude and phase of the sensor’s complex response are given by $a_{sen}=a_{cal}{a}_0$ and ${\varphi }_{sen}={\varphi }_{cal}+{\varphi }_0$, respectively, where $a_{cal}$, ${\varphi }_{cal}$ are obtained from the Gabrielson calibration responses and $a_0$ and ${\varphi }_0$ are obtained from the laboratory calibration of the reference microbarometer.

For each frequency, these responses are treated as Gaussian distributions:

$$\begin{array}{cc}\hfill A_{sen}& =a_{sen}+{\epsilon }_{A_{sen}},\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{sen}& ={\varphi }_{sen}+{\epsilon }_{{\mathsf{\Phi }}_{sen}},\hfill \end{array}$$

(16)

where ${\epsilon }_{A_{sen}}\sim N\left(0,u_{A_{sen}}\right)$, ${\epsilon }_{{\mathsf{\Phi }}_{sen}}\sim N\left(0,u_{{\mathsf{\Phi }}_{sen}}\right)$, $u_{A_{sen}}$, and $u_{{\mathsf{\Phi }}_{sen}}$ are defined in (17) and (18), respectively.

The final model for the uncertainty assessment associated with the sensor amplitude and phase are, respectively, expressed as

$$u_{A_{sen}}^2=a_{sen}^2\left({\left(\frac{u_{A_{cal}}}{a_{cal}}\right)}^2+{\left(\frac{u_{A_0}}{a_0}\right)}^2\right),$$

(17)

$$u_{{\mathsf{\Phi }}_{sen}}^2=u_{{\mathsf{\Phi }}_{cal}}^2+u_{{\mathsf{\Phi }}_0}^2,$$

(18)

where $u_{A_{cal}}$, $u_{{\mathsf{\Phi }}_{cal}}$ are obtained from the Gabrielson calibration responses and $u_{A_0}$ and $u_{{\mathsf{\Phi }}_0}$ are obtained from the calibration certificate of the reference sensor. The sensor response is shown in Figure 8.

2.3.4. Monte Carlo Method Using Latent Input Signals for Uncertainty Propagation

The Monte Carlo method [47,48] is usually used for the estimation of integrals, as is the case in this paper with the quantity of interest W defined as an integral in (2). For deconvolution problems, such as the estimation of X in (1), the Monte Carlo method is applied in the frequency domain [41]. In this section, we show how to apply the Monte Carlo method in the frequency domain to estimate the QoI W, assuming that the transfer function of the sensor $H_{sen}$ is known.

In the frequency domain, the input signal $\widehat{X}\left(f\right)$ can be obtained from (1) as

$$\widehat{X}\left(f\right)=\frac{\widehat{Y}\left(f\right)}{H_{sen}},$$

(19)

where $\widehat{Y}\left(f\right)$ is the observed output signal in the frequency domain obtained by taking the Fourier transform of $Y\left(t\right)$.

Back in the time domain, the estimated input signal $X\left(t\right)$ is obtained from (19) by taking the inverse Fourier transform

$$X\left(t\right)={\mathcal{F}}^{-1}\left(\widehat{X}\left(f\right)\right).$$

(20)

Algorithm 1 shows how to generate M Monte Carlo samples $X_{\left(1\right)},\dots ,X_{\left(M\right)}$ (called latent signals in the paper, also see Figure 1) from (20) independently with the same distribution, denoted $\pi$ in (2), when the calibration uncertainty is known and modeled, as in Section 2.3.3.

The Monte Carlo estimator $W_{MC}$ of W is given by

$$W_{MC}=\frac{{\sum }_{i=1,\dots ,M}\eta \left(X_{\left(i\right)}\right)}{M},$$

(21)

where $\eta$ is defined in (2) and in Section 2.1.

Algorithm 1: Algorithm for sampling latent signals from (20).
Data: transfer function of the sensor $H_{sen}$ defined in (14), observed output signal
$Y\left(t\right)$, number of Monte Carlo simulations M
Result: M latent signal $X_{\left(1\right)},\dots ,X_{\left(M\right)}$ sampled from (20)
while $i\le M$ do
		generate $a_i\sim N(a_{sen},u_{A_{sen}})$ according to (15); /* sample latent signal
		amplitude /*
		generate ${\varphi }_i\sim N({\varphi }_{sen},u_{{\mathsf{\Phi }}_{sen}})$ according to (16); /* sample latent signal
		phase */
		compute $h_i=a_i{e}^{j{\varphi }_i}$;
		compute ${\widehat{X}}_i\left(f\right)=\frac{\widehat{Y}\left(f\right)}{h_i}$;
		$X_{\left(i\right)}\leftarrow {\mathcal{F}}^{-1}\left({\widehat{X}}_i\left(f\right)\right)$
end

The estimator (21) is a random variable. Denoting $x_{\left(i\right)}$ the actual realization of $X_{\left(i\right)}$, the point estimation of the QoI W, denoted ${\widehat{W}}_{MC}$, is obtained by replacing $X_{\left(i\right)}$ with $x_{\left(i\right)}$ in (21). The performance of the estimator $W_{MC}$ is assessed on the generated Monte Carlo sample $\{x_{\left(1\right)},\dots ,x_{\left(M\right)}\}$ with the empirical bias, standard deviation, MAE, and RMSE, defined as follows.

The bias, also referred to as error throughout the paper, is estimated as

$$bias\left(W_{MC}\right)\approx {\widehat{W}}_{MC}-W.$$

(22)

The distribution of bias (error) is the distribution of the vector of observed biases $(b_1,\dots ,b_M)$ estimated on the Monte Carlo sample $\{x_{\left(1\right)},\dots ,x_{\left(M\right)}\}$ with $b_i$ defined as

$$b_i=\eta \left(x_{\left(i\right)}\right)-W.$$

(23)

The standard deviation is estimated as

$$\sigma \left(W_{MC}\right)\approx \sqrt{\frac{1}{M}\sum _{i=1}^M{\left(\eta \left(x_{\left(i\right)}\right)-{\widehat{W}}_{MC}\right)}^2}.$$

(24)

The MAE, mean absolute error, is estimated as

$$MAE\left(W_{MC}\right)\approx \frac{1}{M}\sum _{i=1}^M|{\eta }_{x_{\left(i\right)}}-W|.$$

(25)

The RMSE, root mean squared error, is estimated as

$$RMSE\left(W_{MC}\right)\approx \sqrt{\frac{1}{M}\sum _{i=1}^M{\left(\eta \left(x_{\left(i\right)}\right)-W\right)}^2}$$

(26)

It can be noticed that these indicators depend on M. A larger number of samples would increase the confidence in the Monte Carlo estimate by decreasing its variability but not its bias. This is known as the variance–bias compromise in statistics.

3. Results

In this section, we illustrate the methodology for calibration uncertainty propagation with the Monte Carlo method to estimate the back azimuth and trace velocity of an incoming infrasound signal at a station, using synthetic output signals from existing array designs (OHP, IS26, …). The array designs that are used in this study are shown in Figure 9. Each array consists of between 4 and 9 sensors, with apertures between about 1 and 3 km.

The calibration uncertainty information is experimentally determined and displayed in Figure 5 for the MB2005 response, in Figure 6 for the WNRS response, and in Figure 8 for the sensor response. In Section 3.1, calibration uncertainty is propagated for infinite SNR and operating WNRS. In Section 3.2, a sensitivity study is performed to show the effect of the SNR, the WNRS, and the station array aperture on the estimates of the back azimuth and trace velocity. A further study on the effects of different amounts of sensor phase uncertainties was performed to determine the acceptable levels of sensor phase uncertainties as functions of the frequencies for infrasound stations. The results are given as functions of the frequencies. The performance of the Monte Carlo estimates is analyzed in terms of bias, standard deviation, MAE, and RMSE, as defined in (22), (24), (25), and (26), respectively. The distribution of bias (error) displayed in Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 and in Appendix B, Appendix C and Appendix D is obtained according to (23).

3.1. Uncertainty Propagation Results for Infinite SNR and Operating WNRS

The methodology of uncertainty propagation is applied to a synthetic signal generated as a potential signal output of the sensor array of the Observatoire de Haute Provence (OHP, see Figure 9 for the array design), according to Appendix A with an infinite SNR. For this signal, back azimuth and trace velocity are known so that the QoI W is known. In this paper, due to the computational time, the number of Monte Carlo simulations is $M={10}^4$.

A slight (non-significant) bias was observed for both back azimuth and trace velocity, see Figure 10 and Figure 11. For the back azimuth, the errors are smaller than the uncertainty; however, for the high-frequency trace velocity results, there are observable biases. Working with a known input signal will allow us in this particular case to investigate the origin of this bias.

We begin by considering a signal generated using ideal microbarometers, with an infinite signal-to-noise ratio (SNR), and no other uncertainty sources except for those present in the signal generation and TDOA analysis. The errors in back azimuth and trace velocity for such a signal are shown in Figure 12 and Figure 13. In this case, the dispersion shown by the boxplots is due to the variability observed from a single run of the TDOA algorithm, such as that shown in Figure 2 for each frequency band. As can be observed, for both the back azimuth and trace velocity, there are statistically significant biases at almost every frequency. It should also be noted that these biases follow the same trend as shown in Figure 10 and Figure 11. This demonstrates that there is a bias introduced by the signal generation and by the TDOA-filtering procedures. However, these values are typically small compared to the observed variability, even for the infinite SNR scenario. Therefore, in practice, these biases can be considered negligible. For the purposes of this study, given that the biases are known a priori, we decided to remove them from the Monte Carlo results.

Further study is needed to identify the exact cause of these biases in the signal analysis routines, but this was determined to be outside the scope of this paper. Additionally, given the small value of the biases relative to the variances of the quantities of interest, particularly once the noise is introduced, the biases are not likely to be observable under true measurement conditions.

3.2. Sensitivity Study

In this section, a sensitivity study is performed to show the effect of the SNR, the WNRS, and the station array aperture, called benchmark conditions in Section 2.2, on the estimates of the back azimuth and trace velocity. For all configurations, the bias due to filtering or signal generation is removed according to Section 3.1 so that no effect due to filtering can be observed in the results.

3.2.1. Results for Different SNR

The objective of this section is to analyze the effects of the SNR (infinite, 20 dB, 0 dB) on the Monte Carlo estimation of the back azimuth and trace velocity when the sensor uncertainty is taken into account.

To begin, we will focus on the microbarometer (see Figure 5 for a typical MB2005/MB3 response). The resulting uncertainties caused by the MB2005/MB3 microbarometers are shown in Figure 14, where the dots represent the MAE and the dashed lines represent the standard deviation. As expected, the uncertainty decreases with increasing frequency and with increasing SNR. As the frequency increases, the uncertainty begins to converge and approaches the no-noise limit. The higher the SNR, the lower the frequency at which the uncertainty converges with the no-noise limit. The errors themselves do not appear to be preferentially positive or negative, suggesting that the errors remain randomly distributed around 0. Additionally, it is clear that the errors are always within the uncertainty bounds, indicating negligible bias at all frequencies.

Next, the resulting errors caused by the sensor, which are the microbarometer and WNRS, are shown in Figure 15. Once again, the uncertainty decreases with both the increasing frequency and SNR. The errors have a negligible deviation from 0, considering the values of the uncertainty. The sensor results are very similar to those of the microbarometer. A comparison between the RMSE of the microbarometer and sensor, shown in Figure 16, shows that the sensor errors at low frequencies (below 0.1 Hz) are typically larger than those of the microbarometer. This is caused by increased uncertainty in the sensor calibration at a low frequency due to a partial coherence of the wind-generated noise [31,36]. At higher frequencies, this increased error is much smaller. The difference in error also decreases with decreasing SNR. Therefore, at a lower SNR, there is no discernible difference between the microbarometer alone and the whole sensor.

Histograms of the errors (after bias removal according to Section 3.1) at each of the 26 frequencies for back azimuth and trace velocity, and SNR equals infinity, 0 dB and 20 dB are displayed in Appendix B, Appendix C and Appendix D, respectively.

3.2.2. Results for Different Sensor Arrays

The design of the sensor array, including its size and shape, has been shown to have an impact on the precision and accuracy of infrasound detections [8,20]. Previous studies have investigated uncertainties associated with the sensor array [20], but they have used the sampling frequency as a measure of time measurement uncertainty. In this study, sensor uncertainty is determined through calibration, including amplitude and phase. Array apertures can range from 10 m for a small high-frequency array to several kilometers (1–3 km for IMS stations). Typically, a larger aperture array will be more precise but may experience aliasing issues at higher frequencies. However, a very dense large aperture array could avoid these issues, but practical limitations such as site availability and cost must also be considered in designing the array. It is, therefore, of interest to study the effects that different arrays may have on the TDOA output parameters. Therefore, several different array designs of existing stations were considered, the Observatoire de haute-provence (OHP), IS22 in New Caledonia, IS26 in Germany, IS27 in Antarctica, IS37 in Norway, and a synthetic array that was the same design as OHP but three times larger, hereinafter referred to as OHP3. These arrays were chosen to provide several different array sizes and shapes, including arrays with different numbers of sensors. However, all of these synthetic data suppose a temperature of 23 ${}^{°}$C and pressure of 1000 hPa. In this manner, only the effects of the sensor array size and shape were considered; for real station data, the different environmental conditions would affect the results.

The results for these six different arrays are shown in Figure 17. Given that IS22 (cyan), a triangle with a four-central element, is largely the same as OHP (blue) but larger, and OHP3 (black), a larger array will return smaller root-mean-squared errors in both back azimuth and trace velocity. There appears to be little correlation between the number of sensors and the errors. However, it should be noted that these analyses do not include decoherence of the infrasound beam form, which would greatly affect the results such that the arrays with small subnets would be better at measuring signals with high decoherence. Decoherence (or loss of coherence) is the result of the finite size of the beam form caused by small variances in the wavenumber. Therefore, as measurements are made further apart, there will be a decrease in coherence until there is no coherence between the measurements at sufficiently large distances. For the purpose of this study, it has been assumed that this decoherence is negligible over the length scales of the arrays that were used.

It is worth noting that stations with approximately the same-sized apertures (IS22, IS26, IS27, IS37) have inverted orders in terms of back azimuth and trace velocity RMSE. However, this is caused by the complementarity of the back azimuth and trace velocity uncertainties as a function of the back azimuth angle. That is to say, at angles where the back azimuth uncertainty is smaller, the trace velocity uncertainty is larger, and vice versa. Therefore, this apparent inversion of the order of the RMSE is due to the choice of back azimuth for the input signal. If all back azimuths were considered, then the order of the mean RMSE would be the same for both back azimuth and trace velocity. In such a case, the RMSE would be, in descending order, OHP, IS22, IS37, IS26, IS27, and OHP3.

3.2.3. Results for a Defective WNRS

Regarding the infrasound station, it is worth considering the effects due to defective WNRS, i.e., a WNRS, which has blockages or leaks. Since there are an enormous number of different combinations of defects and WNRS designs, a single defective system was chosen. Using the same nominal system which has been used throughout the paper (see Figure 6), one of the four sensors was considered to be defective with eight (one-quarter) of the inlets blocked, as shown in Figure 7. The same Monte-Carlo calculations as presented in Section 3.2.1 were performed using this array with one defective WNRS.

For this defective system, there is a noticeable increase in the observed errors for both the back azimuth and trace velocity, as shown in Figure 18. This is quite evident in the no-noise case (blue lines), as, starting around 1 Hz, the errors begin increasing. However, the uncertainties are still trending downward with very small values above 1 Hz. This results in non-negligible biases in both the back azimuth and the trace velocity. As the SNR is decreased, the uncertainties and errors at all frequencies increase, with the exception of the errors near the WNRS resonance peak (2–3 Hz), which approach the same values for all three noise cases (it should be noted that, above 1 Hz, the blue and red dots are hidden behind the green dots). This suggests that a defective WNRS would act as an error floor for a system, and as the noise or other sources of uncertainty increase, the effects of the defect would become less evident. Of course, as the noise and other sources of uncertainty increase, the measurement uncertainty would also increase.

3.3. Sensor Phase Uncertainty

For the IMS stations, the CTBTO has defined an acceptable limit of $\pm 5\%$ for the sensor response [9,36]. Therefore, it is important to investigate the effects of sensor uncertainty on the output parameters and to determine acceptable sensor uncertainty thresholds for these stations. Since phase plays a crucial role in the calculation of back azimuth and trace velocity using TDOA, the focus is on phase uncertainty rather than amplitude uncertainty. Any phase error in a signal measured by a sensor will result in a corresponding error in the time of arrival of the signal, which is inversely proportional to the frequency. Additionally, the sensitivity to phase errors will depend on the array size, with smaller arrays being more sensitive.

To study the effects of different levels of phase errors on the back azimuth and trace velocity calculations, the phase errors were set to a single value over the entire frequency range of interest (0.01–4 Hz), and varied between 0 and 10${}^{\circ }$ to see the effects of different phase errors on the final results. The differences between the expected back azimuth and trace velocity were calculated. Figure 19 and Figure 20 show the RMSE for the back azimuth and trace velocity, respectively. As was previously observed in Section 3.2.1, there is a strong frequency dependence on the errors, with the errors decreasing with increasing frequency. As would be expected, there is also an increase in errors with an increase in sensor uncertainty. At high frequencies (>1 Hz), the observed errors are low, even up to a 10${}^{\circ }$ uncertainty for the sensor. However, at low frequencies (<0.1 Hz), the errors can be quite large (several degrees and 10 m/s) at even small sensor uncertainties.

At a lower SNR of 10 dB, the results are similar, with an increase in the overall errors, as shown in Figure 21 and Figure 22. It should be noted that there are far fewer values that are below the consistency threshold for frequencies lower than 0.025 Hz. This is why there is additional variability at these very low frequencies. Overall, there is a similar trend as with the no-noise case. However, more low-frequency events are missed at an SNR of 10 dB.

Overall, it can be concluded that at high frequencies, sensor errors of less than 5° should not greatly impact results, with other factors such as decoherence and SNR being more important. At low frequencies, sensor errors can have significant impacts on the results, even as little as 2° can result in errors larger than 5° and 20 m/s in back azimuth and trace velocity, respectively. This analysis was performed using the OHP array, with an aperture of approximately 1.2 km. For an array of a significantly different size, a similar analysis with those dimensions would need to be performed. It is, therefore, of use to perform this type of analysis when designing an infrasound array to determine if the sensor is sufficiently precise for the frequency band of interest.

4. Conclusions

In an attempt to determine the effects of the sensor uncertainty on the parametric outputs of the TDOA algorithms for infrasound stations, a propagation of the sensor uncertainty was accomplished. First, the in situ calibration of the sensor was performed using a reference sensor with a known calibration response. Then, after generating a synthetic signal, where we had a synthetic sensor output signal, the signal was deconvolved with the sensor response, including the uncertainty, to provide a latent signal with the measured sensor uncertainties. This latent signal was then used as an input to the TDOA algorithm, providing distributions of the QoI, the back azimuth, and the trace velocity.

Overall, this approach was fruitful, as the distributions of the back azimuth and trace velocity, as functions of frequency, were successfully determined at several different SNR levels for the microbarometers alone, and the sensor (microbarometer and WNRS). The resulting errors in back azimuth and trace velocity were unbiased, with the errors increasing dramatically with decreasing frequency. For example, for the sensor case, at the highest frequency, the back azimuth and trace velocity errors were 0.01° and 0.1 m/s, respectively. However, at the lowest frequency, these errors were 5° and 30 m/s, respectively. This was due to the relative decrease in the phase resolution of the infrasound wave at lower frequencies.

Additionally, the uncertainty was inversely related to the SNR. However, there was also a frequency dependence. In fact, there was no noticeable difference between the uncertainties for the infinite and 20 dB SNR cases above 0.2 Hz, with the 20 dB case having larger uncertainties at lower frequencies. Therefore, it would be possible to measure lower SNR signals at higher frequencies, or with a larger aperture station, assuming that the effects of decoherence are minimal. This Monte Carlo approach could be applied to any station configuration and SNR, and as such, could be used in the identification of ideal station designs in the future.

It was shown that the CTBTO sensor uncertainty threshold of 5% is sufficient at high frequencies, above about 0.1 Hz. However, at low frequencies, sensor phase errors of even 2° can lead to large errors in the back azimuth and trace velocity measurements. Therefore, one must be aware of how the uncertainties vary with frequency when considering the sensor uncertainty.

It is difficult to conclude which parameters have the largest impact on the results as this depends on the frequency. In general, at low frequencies (below ∼$0.1$ Hz), the phase errors in the sensor calibration result in substantial uncertainties in the QoI. At higher frequencies, the sensor phase errors result in almost no noticeable variability in the QoI, with the SNR being a much more important parameter. A full sensitivity analysis would be useful to provide a more complete estimation of the relative importance of the different parameters. An increase in the size of the array is equivalent to a decrease in the frequency.

Future work using this analysis tool would involve additional uncertainty sources, such as the loss of coherence, to consider more input back azimuth angles, or use real infrasound data in the uncertainty analysis. In addition, as previously mentioned, a sensitivity analysis would provide valuable information concerning the relative importance of the different input parameters.