# The Importance of Assay Imprecision near the Screen Cutoff for Newborn Screening of Lysosomal Storage Diseases

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Population Distributions of Assay Measurements and Definition of False Positives and False Negatives

## 3. Lysosomal Storage Diseases as an Example

## 4. Assay Imprecision

_{M}, using the standard formula given below (found in most statistics textbooks):

_{M}= (95% upper confidence limit—mean)/1.96 = (mean—95% lower confidence limit)/1.96

## 5. Analytical Range

## 6. Z-Factor

_{N}+ σ

_{D})/|(µ

_{N}− µ

_{D})|

_{N}and σ

_{N}are the mean and standard deviation, respectively, for the enzymatic activity measured on a collection of DBS from normal patients, respectively, and μ

_{D}and σ

_{D}are measured on DBS from patients with the disease. Z approaches a maximum of 1 when the sum of the standard deviations (σ

_{N}+ σ

_{D}) is much smaller than the difference in the means (µ

_{N}− µ

_{D}). In this case, the enzymatic activities of the disease and normal groups of people are well separated compared to the range of values for each group, and one can assign disease versus normal status to any new enzymatic activity value of a patient with high confidence. When Z = 0, this corresponds to µ

_{N}and µ

_{D}being separated by 3 × (σ

_{N}+ σ

_{D}). Thus, the factor of 3 in equation (1) is an arbitrary constant.

## 7. Assay Imprecision Near the Screen Cutoff

_{M}, in general, depends on the mean enzymatic activity in the DBS. To carry out the convolution, we need these values of σ

_{M}. In Figure 2 we show the imprecision data measured by the CDC on the four types of quality control DBS described above. This set is for the measurement of the enzymatic activity by tandem mass spectrometry due to α-glucosidase that is relevant to NBS of Pompe disease (data is available online at https://www.cdc.gov/labstandards/nsqap_resources.html). In Figure 2 we plot the four values of σ

_{M}versus the mean activity μ

_{M}. Note that σ

_{M}and μ

_{M}are estimates of the true parameters; the estimates are more accurate as the number of measurements in the imprecision analysis is increased. One can thus estimate the errors in σ

_{M}and μ

_{M}, and, according to standard textbooks in statistics, these both have a values of σ

_{M}/N

^{1/2}, where N is the number of punches analyzed. These standard deviations are shown as error bars in Figure 2. It is clear from this figure that σ

_{M}is not constant for all values of enzymatic activity measured in DBS, and thus a single value of σ

_{M}cannot be used for the convolution. To analyze the effect of imprecision properly, we need a continuous mathematical function that relates σ

_{M}to μ

_{M}. One such function is shown by the solid line through the CDC data in Figure 2 (the explicit function is given in the figure legend, its identity is not important).

_{M}, that matters but the relative imprecision. Suppose assay platform 1 gives a value of 2 μmol/h/L for a DBS punch and the other platform gives 4 μmol/h/L for the same sample. This difference is due to the different substrates and buffer conditions used in each enzymatic activity assay. If σ

_{M}for platform 1 is found to be 0.5 μmol/h/L, an equivalent imprecision for platform 2 would be twice this value or σ

_{M}= 0.5 μmol/h/L. In this case, both platforms are equally imprecise. Strictly speaking, adjustment by scaling of this type works only if there are no offsets in the enzymatic activities in comparing one platform to another. For example, for measurement of GAA for NBS of Pompe disease, there is a different degree of interference from an off-target enzyme in fluorimetric versus tandem mass spectrometry assays [4]. This differential offset must be removed from observed enzymatic activity values before a simple proportional adjustment to σ

_{M}is made.

_{M}= 0.5 μmol/h/L for all enzymatic activities. This leads to the black curve in Figure 3A furthest from the origin. The black curve passing close to the origin is for σ

_{M}= 0 μmol/h/L for all enzymatic activities. The red diamonds on each curve corresponds to a cutoff of 2.0 μmol/h/L. Thus, at this fixed cutoff the false positive and false negative rates are higher when assay imprecision is present. In statistical analysis, plots of the type shown in Figure 3 are a form of “receiver operating characteristic” curve (ROC curve, see Supplementary Materials for further definitions).

_{M}= 0.5 μmol/h/L) but relatively low at σ

_{M}= 0.1 μmol/h/L in the 0–3 μmol/h/L region, we obtain the blue curve in Figure 3A. This is almost identical to the no-imprecision curve (black curve) showing that imprecision well above the cutoff has only a small effect to increase the false positive rate and essentially no effect on the false negative rate (based on the position of the red diamonds). On the other hand, with σ

_{M}= 0.5 μmol/h/L well below the cutoff in the 0–1 μmol/h/L range with σ

_{M}= 0.1 μmol/h/L at 1 μmol/h/L or higher enzymatic activity, one obtains the red curve in Figure 3A. In this case the false positive rate changes very little, but the false negative rate increases substantially. This is the expected result since focusing the imprecision well below the cutoff shifts the PDF of the sick newborns more than that of the healthy newborns.

_{M}is centered at 1 μmol/h/L (i.e., covers the range of 0 to 2 μmol/h/L). The green curve has the smaller imprecision centered at the cutoff of 2 μmol/h/L (i.e., 1–3 μmol/h/L). The red curve has the smaller imprecision centered at 3 μmol/h/L (i.e., 2–4 μmol/h/L). Finally, the yellow curve has the smaller imprecision centered at 4 μmol/h/L (i.e., 3–5 μmol/h/L). The diamonds on the curves in Figure 3B correspond to an assay value of 2.0 μmol/h/L, and, at this assay value, which can be considered a cutoff, the green and red curves show the smallest false positive rate. This shows that minimizing the imprecision just above the cutoff is most effective at reducing the false positive rate. Similarly, the green and blue curves show the smallest false negative rate. Note that the green curve shows improvement in both rates, although each to a lesser extent. This analysis leads to the most important conclusion of this study, which is that imprecision near the cutoff affects both the false positive and false negatives rates to the largest extent.

## 8. Comparison of Different NBS Assay Platforms

_{EOI}(t) + P

_{OTE}(t) + P

_{NE}(t) + P

_{IM}] + δ

_{EOI}(t) is the moles of product made by the enzyme of interest, P

_{OTC}(t) is the moles of product made by one or more off-target enzymes in the mixture (not all substrates are completely specific for the enzyme of interest), P

_{NE}(t) is the moles of product generated from substrate by all nonenzymatic processes, and P

_{IM}is the moles of product present in the substrate as a contaminant (which is not a function of time). The constant γ links the total moles of product to AR. The parameter δ does not depend on the amount of product but is a contribution to AR(t) as it includes the assay response from the sample matrix and from the substrate itself and also any quenching of the assay response by the matrix (a negative contribution). For example, fluorimetric assays are subject to quenching by components of the matrix that absorb emitted light from the fluorophore. Mass spectrometry assays display matrix suppression of ionization of the product analyte. The latter is completely removed from consideration by use of a chemically identical, but isotopically substituted internal standard [6], whereas correction for fluorimetric quenching is difficult to achieve.

_{OTE}(t) is ~20% of P

_{EOI}(t)

_{,}whereas for tandem mass spectrometry P

_{OTE}(t) is only ~3% of P

_{EOI}(t) [1]. For fluorimetric substrates using the 4-methylumbelliferone fluorophore, the substrate itself has significant fluorescence in the optical channel where the product is detected (a contribution to δ) [3]. Additionally, P

_{OTE}(t) and δ are likely to be sample-dependent, making it difficult to correct for all of these confounding factors using a single blank sample.

## 9. Postanalysis Tools That Do Not Use Single Cutoffs

## 10. Concluding Remarks

## Supplementary Materials

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Gelb, M.H.; Lukacs, Z.; Ranieri, E.; Schielen, P.C.J.I. Newborn Screening for Lysosomal Storage Disorders: Methodologies for Measurement of Enzymatic Activities in Dried Blood Spots. Int. J. Neonatal Screen.
**2019**, 5, 1. [Google Scholar] [CrossRef] - De Jesus, V.R.; Zhang, X.K.; Keutzer, J.; Bodamer, O.A.; Mühl, A.; Orsini, J.J.; Caggana, M.; Vogt, R.F.; Hannon, W.H. Development and evaluation of quality control dried blood spot materials in newborn screening for lysosomal storage disorders. Clin. Chem.
**2009**, 55, 158–164. [Google Scholar] [CrossRef] [PubMed] - Kumar, A.B.; Masi, S.; Ghomashchi, F.; Chennamaneni, N.K.; Ito, M.; Scott, C.R.; Turecek, F.; Gelb, M.H.; Spacil, Z. Tandem Mass Spectrometry Has a Larger Analytical Range than Fluorescence Assays of Lysosomal Enzymes: Application to Newborn Screening and Diagnosis of Mucopolysaccharidoses Types II, IVA, and VI. Clin. Chem.
**2015**, 61, 1363–1371. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Gelb, M. Newborn Screening for Lysosomal Storage Diseases: Methodologies, Screen Positive Rates, Normalization of Datasets, Second-Tier Tests, and Post-Analysis Tools. Int. J. Neonatal Screen.
**2018**, 4, 23. [Google Scholar] [CrossRef] [PubMed] - Gelb, M.H.; Scott, C.R.; Turecek, F.; Liao, H.-C. Comparison of tandem mass spectrometry to fluorimetry for newborn screening of LSDs. Mol. Genet. Metab. Rep.
**2017**, 12, 80–81. [Google Scholar] [CrossRef] [PubMed] - Li, Y. Direct Multiplex Assay of Lysosomal Enzymes in Dried Blood Spots for Newborn Screening. Clin. Chem.
**2004**, 50, 1785–1796. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Baerg, M.M.M.; Stoway, S.D.; Hart, J.; Mott, L.; Peck, D.S.; Nett, S.L.; Eckerman, J.S.; Lacey, J.S.; Turgeon, C.T.; Gavrilov, D.; et al. Precision newborn screening for lysosomal disorders. Genet. Med.
**2017**, 317, 576. [Google Scholar]

**Figure 1.**(

**A**) Histogram distribution for enzymatic activity values. A collection of dried blood spots (DBS) from newborns is submitted to the enzymatic activity assay. The activity values are parsed into arbitrary bins, in this case of width 0.8 μmol/h/L. The height of each bin (bar in the plot) is the number of newborns displaying enzymatic activity in each bin divided by the total number of newborns. The smooth curve drawn on the histogram is a log-normal mathematical function fitted numerically to the histogram, by adjusting the three parameters, amplitude, mean, and width. (

**B**) Shown are typical probability distribution functions (PDFs) for newborns that are healthy (blue) or confirmed to have the disease (orange). The orange curve intersects the origin and stops (difficult to see). The PDF for the healthy newborns is log-normal with a mean of 10 μmol/h/L and standard deviation 3.5 μmol/h/L. The PDF for the disease newborns is log-normal with mean 0.9 μmol/h/L and standard deviation 0.4 μmol/h/L (see Supplementary Materials for the log-normal function). (

**C**) Expansion of the plot in Figure 1B near the screen cutoff (vertical black line at 3.3 μmol/h/L, chosen as an example). The area under the healthy PDF to the left of the cutoff corresponds to false positives, and the area under the disease PDF to the right of the cutoff corresponds to false negatives. Note that there is no reason to put the cutoff at the point of intersection of the two curves. In this example, we choose a cutoff to the right of where the two curves intersect; this is typical of NBS laboratories where there is more desire to reduce false negatives to a minimum at the expense of an increase in false positives.

**Figure 2.**Plot of standard error (also called the standard deviation) of the measurement (σ

_{M}) versus the mean activity (μ

_{M}). This is for the tandem mass spectrometry assay of the GAA enzyme (relevant to Pompe disease) using the CDC Quality Control DBS. The data was obtained from the CDC (Set 2 2018 LSD FIA Certification) on their NBS web portal (https://www.cdc.gov/labstandards/nsqap_resources.html). The solid line is a model for the CDC data using the function: $c+a({x}^{4}/\left({x}^{4}+{b}^{4}\right)$ where a, b, and c are constants (this is a convenient function that well interpolates through the data points; it has no biological basis that we are aware of). Error bars in σ

_{M}, plotted on the Y-axis, are calculated as σ

_{M}/N

^{1/2}, where N is the number of punches analyzed by the CDC (N = 20). The error bars for μ

_{M}, plotted on the X-axis, are calculated as ${t}_{S}{\sigma}_{M}$/N

^{1/2}, at the 95% confidence interval, here t

_{s}is Student’s t value. These error equations are found in standard textbooks of statistics including the definition of Student’s t value, t

_{s}.

**Figure 3.**Parametric plots (modified received operating characteristic, ROC plots) where the false positive rate (fraction of healthy patients who are screen-positive) is plotted on the Y-axis, and the false negative rate (fraction of disease patients who are screen-negative) is plotted on the X-axis. The diamonds indicate the point on each curve for an assay value of 2.0 μmol/h/L (which could be chosen as the cutoff). The no-imprecision PDF is modeled by a log-normal distribution with a mean of 10.0 and a width of 3.8 μmol/h/L for the healthy population and a mean of 1.0 and width of 0.38 μmol/h/L for the patients with the disease (

**A**). The black curve passing closest to the origin is for no-imprecision, and the other black curve is for uniform and high imprecision of σ

_{M}= 0.5 μmol/h/L for all values of enzymatic activity. The blue curve is for high imprecision of σ

_{M}= 0.5 μmol/h/L for enzymatic activities of 3 μmol/h/L or higher, and a negligible imprecision of σ

_{M}= 0.1 μmol/h/L for enzymatic activities below 3 μmol/h/L. The red curve has the high imprecision in the window of 0 to 1 μmol/h/L and negligible imprecision elsewhere. The green curve has high imprecision in the window of 1 to 3 μmol/h/L (passing through the cutoff) and negligible imprecision elsewhere. (

**B**). Black curves as for Panel A. The blue curve has lowered imprecision (σ

_{M}= 0.25 μmol/h/L) in the window 0–2 μmol/h/L and high imprecision (σ

_{M}= 0.50 μmol/h/L) elsewhere. This region of lowered imprecision is slid to the right to give the green curve (σ

_{M}= 0.25 μmol/h/L in the window of 1–3 μmol/h/L), further to the right to give the red curve (σ

_{M}= 0.25 μmol/h/L in the window 2–4 μmol/h/L), and even further to the right to give the yellow curve (σ

_{M}= 0.25 μmol/h/L in the window 3-5 μmol/h/L).

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Robinson, B.H.; Gelb, M.H.
The Importance of Assay Imprecision near the Screen Cutoff for Newborn Screening of Lysosomal Storage Diseases. *Int. J. Neonatal Screen.* **2019**, *5*, 17.
https://doi.org/10.3390/ijns5020017

**AMA Style**

Robinson BH, Gelb MH.
The Importance of Assay Imprecision near the Screen Cutoff for Newborn Screening of Lysosomal Storage Diseases. *International Journal of Neonatal Screening*. 2019; 5(2):17.
https://doi.org/10.3390/ijns5020017

**Chicago/Turabian Style**

Robinson, Bruce H., and Michael H. Gelb.
2019. "The Importance of Assay Imprecision near the Screen Cutoff for Newborn Screening of Lysosomal Storage Diseases" *International Journal of Neonatal Screening* 5, no. 2: 17.
https://doi.org/10.3390/ijns5020017