Next Article in Journal
The Impact of COVID-19 Pandemic on ESBL-Producing Enterobacterales Infections: A Scoping Review
Previous Article in Journal
Evidence of Antibiotic Resistance and Virulence Factors in Environmental Isolates of Vibrio Species
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluation of Aztreonam and Ceftazidime/Avibactam Synergism against Klebsiella pneumoniae by MALDI-TOF MS

by
Camila Mörschbächer Wilhelm
1,2,*,
Everton Inamine
3,
Andreza Francisco Martins
1,2 and
Afonso Luís Barth
1,2
1
Programa de Pós-Graduação em Ciências Farmacêuticas, Universidade Federal do Rio Grande do Sul, Porto Alegre 90610-000, Brazil
2
Laboratório de Pesquisa em Resistência Bacteriana (LABRESIS), Hospital de Clinicas de Porto Alegre, Porto Alegre 90035-903, Brazil
3
Laboratório Carlos Franco Voegeli, Irmandade Santa Casa de Misericórdia de Porto Alegre, Porto Alegre 90050-170, Brazil
*
Author to whom correspondence should be addressed.
Antibiotics 2023, 12(6), 1063; https://doi.org/10.3390/antibiotics12061063
Submission received: 15 April 2023 / Revised: 30 April 2023 / Accepted: 3 May 2023 / Published: 16 June 2023

Abstract

:
Introduction: Resistance to carbapenems due to the co-production of NDM and ESBL or NDM and KPC is increasing. Therefore, combined therapy with aztreonam (ATM) plus ceftazidime/avibactam (CZA) has been recommended. Then, it is necessary to develop and evaluate fast and simple methods to determine synergism in vitro in microbiology laboratories. Objective: To develop a method to determine the synergism of ATM and CZA by MALDI-TOF MS (SynMALDI). Method: Klebsiella pneumoniae (n = 22) isolates with blaNDM and/or blaKPC genes were tested. The time–kill curve assay was performed for four isolates (three positives for blaNDM and blaKPC and one positive for blaNDM only). For SynMALDI, each isolate was incubated for 3 h in 4 tubes containing brain–heart infusion broth with the following: (1) no antibiotic; (2) ATM at 64 mg/L; (3) CZA at 10/4 mg/L; and (4) ATM at 64 mg/L plus CZA at 10/4 mg/L. After incubation, the bacterial protein extract was analyzed by MALDI-TOF MS, and the relative growth (RG) was determined for each isolate, considering intensities of the peaks of the bacterium incubated with antibiotic (tubes 2, 3, and 4) to the same bacterium incubated without antibiotic (tube 1), as follows: RG = IntensityWith antibiotic/IntensityWithout antibiotic. The combination was determined as synergistic when there was an RG decrease of 0.3 in the antibiotic combination in relation to the RG of the most active antibiotic alone. Results: The combination of ATM plus CZA proved to be synergic by time–kill curve assay. All isolates tested with the SynMALDI method also presented synergism. Conclusions: Detection of synergism for ATM plus CZA combination can be determined by MALDI-TOF MS, providing fast results in order to improve patient treatment.

1. Introduction

Carbapenem-resistant Enterobacterales (CRE) are a major threat to public health worldwide [1,2]. The treatment of infections caused by CRE can be made by the use of new antibiotic combinations of beta-lactam with beta-lactamase inhibitors, such as ceftazidime/avibactam (CZA). Avibactam is a diazabicyclooctane molecule; it is a beta-lactamase inhibitor that protects ceftazidime from being hydrolyzed by serine beta-lactamases, such as extended-spectrum beta-lactamases (ESBL), Klebsiella pneumoniae carbapenemase (KPC) and OXA-48-like. Nonetheless, avibactam is not active against New Delhi Metallo-beta-lactamase (NDM) and other metallo-beta-lactamases (MBL). A therapeutic option for the treatment of infections due to Enterobacterales with NDM carbapenemase is aztreonam (ATM) which, however, is hydrolyzed by serine beta-lactamases [3,4].
The emergence of multidrug-resistant pathogens has been observed around the world [5]. Among Enterobacterales species, K. pneumoniae stands out as the main multidrug-resistant pathogen, as its plasmids carrying resistant genes are easily disseminated, especially in hospital environments [6]. Especially, an increase in the spread of genes encoding NDM among Enterobacterales has brought more concern to the problem of antimicrobial resistance, mainly because the production of NDM has also been reported in co-occurrence with KPC and other serine beta-lactamases [7,8,9]. Although new antibiotics, such as imipenem/relebactam, meropenem/vaborbactam, aztreonam/avibactam, and cefiderocol, have demonstrated effectiveness against multidrug-resistant Enterobacterales [6,10], most of them are not currently available in Brazil and other countries. Therefore, treatment options have become very limited when the infection is caused by CRE-co-producing NDM and KPC carbapenemases. Effective treatment for infections due to CRE-co-producing NDM and KPC can be achieved using the combination of CZA plus ATM, as CZA is active against KPC and ATM is active against NDM, whether no other resistance mechanism is associated [4]. The CZA plus ATM-combined therapy has been clinically debated [11,12,13]. Although the Infectious Diseases Society of America (IDSA) and the European Society of Clinical Microbiology and Infectious Diseases (ESCMID) recommend the empiric use of this therapy to treat infections caused by CRE-co-producing NDM and KPC [14,15], it is preferred that the clinical therapy could be guided by synergism experiments in vitro. However, the methodologies for the detection of synergism in vitro are usually labor-intensive and time-consuming [16,17,18,19,20,21].
Several studies of synergism have been described for ATM and CZA: MIC:MIC ratio technique [18,19,21,22,23]; strip crossing [17,21,22]; double-disk [16,19]; combined disk [16]; and several other variations [20,21,24,25,26]. However, only a few have tested and correlated their results to the in vitro standard assay for synergism detection, the time–kill curve (TKC) assay [16,24,25]. As the TKC assay is laborious and time-consuming, it is necessary to develop methods that could be performed in the microbiology laboratory in a practical way. The TKC alternative methods already proposed, although they may be easy to perform, can take up to 48 h to provide the results as they usually require the MIC of the antibiotics to be previously determined. As these methods are based on antimicrobial disk or gradient strip diffusion methods, we understand that a technique that uses the methodology of matrix-assisted laser desorption ionization—time of flight mass spectrometry (MALDI-TOF MS)—can be used to develop a rapid method to evaluate synergism, since MALDI-TOF has already been used to properly provide antimicrobial susceptibility profile [27].
MALDI-TOF MS has revolutionized the microbiology field regarding microbial identification. Furthermore, this methodology has been used for other applications such as (1) the detection of beta-lactamase activity through hydrolysis assays, (2) the direct detection of beta-lactamase enzymes, and (3) the determination of antimicrobial susceptibility by the evaluation of the relative growth rate [28]. The latter was developed by Lange et al. (2014) and named MALDI Biotyper-antimicrobial susceptibility testing rapid assay (MBT-ASTRA). The MBT-ASTRA compares the spectrum of the bacterium incubated with antibiotic with the spectrum of the same bacterium incubated without antibiotic [1,2]. This methodology has demonstrated satisfactory results for meropenem, ceftazidime, ciprofloxacin, and other antibiotics against mainly Enterobacterales [27,29,30,31].
Therefore, the objective of this study was to develop and evaluate a rapid methodology, termed SynMALDI, to detect ATM plus CZA synergism by MALDI-TOF MS.

2. Materials and Methods

2.1. Bacterial Strains

A total of 22 Klebsiella pneumoniae clinical isolates, obtained from 2019 to 2022, stored at the bacterial bank of Laboratório de Pesquisa em Resistência Bacteriana (LABRESIS) were selected by convenience and included in the study. All isolates were obtained from a hospital in the city of Porto Alegre, Brazil, and were isolated from the urinary tract, rectal swab, and wound. Identification was performed with MALDI-TOF MS (Bruker Daltonics®, Bremen, Germany) according to the manufacturer’s instructions. For 21 meropenem-resistant isolates (Kp01 to Kp21), the presence of the carbapenemase genes was previously evaluated by high-resolution melting real-time PCR (qPCR-HRM), using a multiplex assay with primers for blaKPC, blaNDM, blaOXA-48-like, blaIMP, blaGES, and blaVIM genes, as previously described [32]. One meropenem susceptible isolate (Kp22) was included as a negative control of the SynMALDI method.
In order to avoid testing isolates closely related in the TKC, we performed spectral grouping with data acquired by MALDI-TOF MS using the same parameters for microbial identification. Spectra were grouped using ClinProTools 3.0 (Bruker Daltonics®), as described elsewhere [33], resulting in eight groups (A–H). One NDM producer and ATM resistant and three NDM and KPC co-producers from distant spectral groups were selected for TKC experiments and for whole genome sequencing (WGS) in order to identify beta-lactamase genes (Figure 1).

2.2. Determination of Susceptibility by Disk Diffusion and Minimum Inhibitory Concentration (MIC)

Antimicrobial susceptibility testing by disk diffusion and gradient diffusion strips was performed for ATM and CZA, with disks (Oxoid Thermo Scientific®, Waltham, MA, USA) of 30 µg and 10/4 µg, respectively. The results were interpreted according to EUCAST [34]. Disk diffusion and MIC breakpoints for ATM and CZA are displayed in Table 1. Quality control was performed with Escherichia coli ATCC 25922, according to EUCAST instructions [35]. Antimicrobial susceptibility testing by disk diffusion was performed for all isolates, and MIC was determined by gradient diffusion strip for the isolates selected for TKC assay.

2.3. Time–Kill Curve (TKC) Assay

TKC assay was performed according to CLSI guidelines [36] with some modifications, as previously described [37]. Briefly, fresh cultures grown on blood agar plates were used to prepare an inoculum of approximately 3 × 108 CFU/mL in cation-adjusted Müeller–Hinton (CAMH) broth. This suspension, in the log phase, was diluted 1:5 and added to four tubes containing CAMH to achieve a final concentration of 6 × 105 CFU/mL. The first tube was without antibiotic; the second tube contained ATM at 8 mg/L (BioChimico®, Rio de Janeiro, Brazil); the third tube contained CZA at 8/2 mg/L (ceftazidime at 8 mg/L and avibactam at 2 mg/L; Wyeth/Pfizer®, São Paulo, Brazil), and the fourth tube contained ATM at 8 mg/L and CZA at 8/2 mg/L. All experiments were performed in duplicates. The tubes were incubated at 37 °C, at 125 rpm agitation for 24 h. An aliquot from each tube was removed at times 0, 1, 2, 4, 6, 12, and 24 h and serially (1:10) diluted. A volume of 20 µL of each dilution from the same tube was spotted on a MacConkey agar plate. Plates were incubated at 37 °C for 18 to 24 h. Colonies were counted considering the spots with 10 to 100 colonies, and the results were reported as CFU/mL. Synergism was defined as a decrease ≥ 2 log10 CFU/mL by the antibiotic combination compared to the most active antibiotic alone, and antagonism was defined as an increase ≥ 2 log10 CFU/mL by the combination compared with the most active antibiotic alone after 24 h of incubation [36].

2.4. Whole Genome Sequencing (WGS)

Genomic DNA was extracted from colonies grown on LB broth using a QIAamp DNA Mini Extraction Kit (QIAGEN®, Hilden, Germany). Total DNA concentration was measured using a Qubit dsDNA HS Assay Kit with a Qubit 4 fluorometer (Thermo Fisher Scientific). WGS was performed using the Illumina MiSeq™ platform (Illumina, San Diego, CA, USA). The paired-end library was constructed with the NexteraTM XT DNA Library Prep Kit (MiSeq™ System), and the MiSeqTM Reagent V2 kit (2 × 250 cycles) was used to perform a run with a coverage depth of 100×. The raw reads were quality-trimmed (Q > 30) and assembled using CLC Genomic Workbench 21. Antimicrobial resistance genes were identified (contigs > 200 bp) in silico using ResFinder 4.0 (https://cge.food.dtu.dk/services/ResFinder/, accessed on 26 December 2022) and QIAGEN Microbial Insight-Antimicrobial Resistance database (QMI-AR).

2.5. Antibiotic Solution Preparation for Synergism by MALDI-TOF MS (SynMALDI)

For the synergism determination by MALDI-TOF MS, antibiotic solutions were prepared from commercially available disks of ATM 30 µg and CZA 10/4 µg (Oxoid Thermo Scientific®). Each antibiotic lot of disks was quality-control-checked, as recommended by EUCAST [35]. The disks were eluted for 15 min in distilled water in order to obtain ATM at 64 mg/L and CZA at 10/4 mg/L as final concentrations.

2.6. Synergism by MALDI-TOF MS (SynMALDI)

The method for determination of synergism by MALDI-TOF MS was adapted from MBT-ASTRA [27] and performed for all isolates. Isolates were suspended in brain–heart infusion broth (Kasvi, São José dos Pinhas, Brazil) to obtain turbidity equivalent to 0.5 McFarland. In order to reach a final volume of 200 µL, 50 µL of the bacterial suspension was added to four microtubes as follows: the first without antibiotic; the second tube with ATM; the third tube with CZA; and the fourth tube with ATM and CZA. The microtubes were vortexed and incubated, without agitation, for 3 h at 35 ± 2 °C. Afterward, the suspensions were centrifuged at 16,060× g (13,000 rpm) for 2 min. The supernatant was discarded, and 150 µL of distilled water was added to the microtubes. The microtubes were vortexed and centrifuged at 16,060× g (13,000 rpm) for 2 min. The supernatant was discarded, and 100 µL of ethanol 70% (Merck®, Darmstadt, Germany) was added to each microtube. The microtubes were vortexed, left to rest at room temperature for 5 min, and centrifuged at 16,060× g (13,000 rpm) for 2 min. The supernatant was discarded, and the pellet was left to air dry at room temperature. Subsequently, 10 µL of formic acid 70% (Sigma–Aldrich®/Merck®, Darmstadt, Germany) and 10 µL of acetonitrile 100% (Merck®) containing an internal standard (RNase B 20 g/L; Sigma–Aldrich®) were added, and the microtubes were vortexed and centrifuged at 16,060× g (13,000 rpm) for 2 min. A volume of 1 µL of the supernatant was spotted, in quadruplicates, onto a polished steel target plate and left to air dry. A volume of 1 µL of α-cyano-4-hidroxy-cinnamic acid (HCCA; Bruker) at 10 mg/mL in acetonitrile 50% and trifluoracetic acid 2.5% (Sigma–Aldrich®) was added, and the spots were left to air dry at room temperature.
For spectra acquisition, a Microflex LT (Bruker Daltonics®) mass spectrometer was used with flexControl 3.4 software (Bruker Daltonics®). The parameters applied were the same as used for microbial identification (ion source 1, 20 kV; ion source 2, 18.25 kV; lens, 6 kV; detector gain, 2850 V), with a range from 2.000 to 20.000 Da (method MBT_FC.par, set by the manufacturer). The mass spectrometer was externally calibrated using the bacterial test standard—BTS (Bruker®, Billerica, MA, USA), according to manufacturer instructions. An automated MBT_AutoX programming was used for the acquisition of the spectra.
Spectra were analyzed with flexAnalysis 3.4 software (Bruker Daltonics®). For each spectrum, four peaks were manually selected (two peaks from the bacterium and two peaks from the internal standard), and their intensities were annotated after baseline subtraction and smoothing. Peaks from bacteria were selected between the ranges 6000–6300 and 9000–10,000 m/z, while peaks from the internal standard were approximately 7454 and 14,910 m/z. The peaks from the bacteria were carefully selected after comparison with the spectrum of RNase B alone in order to avoid choosing peaks from the internal standard. For each quadruplicate, the sum of the intensities from the bacterium (IntBac) was normalized with the sum of the intensities from the internal standard (IntRNase B). Then, the RG value was calculated by the ratio of the median of the normalized peaks from bacteria incubated with antibiotic (IntATB) with the median of the normalized peaks from bacteria incubated without antibiotic (IntBHI). RG values were calculated as follows: RG = Median (∑ IntBac+ATB/∑ IntRNase B+ATB)/Median (∑ IntBac+BHI/∑ IntRNase B+BHI). For each isolate, three RG values were calculated, considering ATM and CZA alone and in combination: RG ATM; RG CZA; and RG ATM + CZA. Synergism was defined as a decrease ≥ 0.3 at the RG value of the antibiotic combination compared to the RG value of the most active antibiotic alone, whether RG ATM or RG CZA.

2.7. Double-Disk Synergism Assay

The double-disk method to evaluate synergism was performed according to EUCAST standard susceptibility testing by disk diffusion method [38], with ATM 30 µg and CZA 10/4 µg disks placed at a 20 mm center-to-center distance. Such distance was chosen because it is the same distance between disks when using an antibiotic disk dispenser device. When the presence of an inhibition zone, also known as a “ghost zone”, was observed between the disks, the test was considered positive for synergism. Double-disk synergism testing was performed for all isolates except for Kp20 to Kp22.

3. Results

A total of 19 K. pneumoniae isolates (Kp01 to Kp19) were resistant to both ATM and CZA by disk diffusion standard susceptibility testing, presenting blaNDM and/or blaKPC genes as determined by qPCR-HRM; all isolates were analyzed by MALDI-TOF-MS for spectral grouping (Table 2). The four isolates (Kp04, Kp08, Kp10, and Kp16) that were tested for TKC assay presented high MIC for ATM and CZA, and all were positive for synergism (Table 3), as the combination of ATM and CZA presented a decrease ≥ 2 log10 CFU/mL in comparison to each antibiotic alone (Figure 2). WGS of these four isolates confirmed the presence of carbapenemase genes and demonstrated the presence of ESBL genes (Table 3).
The isolates tested for TKC were tested for SynMALDI and demonstrated a significant decrease at RG of ATM plus CZA in comparison to RG of ATM and of CZA individually. Subsequently, all isolates were tested for SynMALDI (Table 4). ATM-resistant isolates presented RG of ATM ≥ 0.5617 and a mean ± standard deviation of 0.8520 ± 0.1655, while CZA-resistant isolates presented RG of CZA ≥ 0.5841 and a mean of 0.9341 ± 0.2715 (Figure 3). The RG of ATM plus CZA ranged from 0.0486 to 0.5338, with a mean of 0.1335 ± 0.1293. The comparison of the RG of ATM plus CZA with individual results of RG of ATM and CZA indicated that all isolates presented a lower value when both antibiotics were combined, with a minimum difference to RG ATM of 0.3630 and to RG CZA of 0.4709. Therefore, we considered that all isolates, resistant to both ATM and CZA, demonstrated synergism by SynMALDI, as all presented a minimum difference of 0.3 of RG of ATM plus CZA values in relation to the RG of the most active antibiotic alone. Interestingly, the mean of the RG differences considering ATM and CZA, in comparison to ATM plus CZA, were 0.7186 ± 0.1647 and 0.8155 ± 0.2817, respectively. Isolates tested as control (Kp20, Kp21, and Kp22) that were susceptible to ATM and/or CZA demonstrated RG values ≤ 0.3426 and ≤ 0.0647, respectively. These three isolates did not present a minimum difference of 0.3 of the RG of ATM plus CZA compared to the RG of the single most active antibiotic.
Double-disk synergism, which was performed for isolates resistant to both ATM and CZA, demonstrated the “ghost zone”, indicating a positive result of synergism for all isolates (Kp01 to Kp19).

4. Discussion

Although clinical studies demonstrating the efficacy of ATM plus CZA to treat infections of CRE co-producing NDM and KPC are scarce, empiric treatment with this combination of antibiotics is recommended by IDSA and ESCMID [14,15]. However, the synergism of ATM plus CZA is expected if the resistance mechanism is due to serine and MBL co-production only. Resistance to CZA by mechanisms other than MBL production has been reported, such as amino acid substitutions at the active site of beta-lactamases (which is the case of isolates harboring KPC-3 mutations), decreased membrane permeability due to porin and/or PBP mutations, among others [39]. Synergism of ATM plus CZA may not occur if resistance to CZA is due to resistant mechanisms other than MBL production, and in order to better guide the therapy, it is necessary to evaluate in vitro whether ATM and CZA are synergic. Thus, we developed a fast and easy method to determine ATM and CZA synergism with the use of MALDI-TOF MS.
The principle of the TKC assay to evaluate synergism is based on the decrease in the growth rate of the bacterium incubated with combined antibiotics when compared to the most active agent alone. We have used the same principle to consider synergism by the SynMALDI method, i.e., a significant decrease in the RG value of the bacterium with combined antibiotics versus the bacterium with the antibiotic alone. In our study, for the determination of ATM and CZA synergism, we found a minimal difference of around 0.3 to 0.4 and a mean difference of around 0.7 to 0.8. Considering a decrease ≥ 0.3 of RG value of ATM plus CZA, all isolates that were resistant to both ATM and CZA were considered positive for synergism by SynMALDI, as was expected according to TKC and double-disk synergism assays. Conversely, it was also expected that control strains Kp20 to Kp22 would not present synergism since they already presented susceptibility and low RG values for ATM or CZA individually; therefore, the addition of a second antibiotic would not significantly increase the activity of the first antibiotic against the bacterium. Therefore, we suggest performing SynMALDI for ATM with CZA only if the isolate demonstrates to be resistant to both antibiotics individually.
Studies have assessed ATM plus CZA combination by several in vitro methods, attempting to propose a way to detect synergism. Among methodologies using gradient strips, MIC:MIC ratio is one of the most used. Although some studies [18,22] have calculated the fractional inhibitory index, a measure usually determined using the checkerboard assay to indicate synergy, the MIC:MIC ratio technique is laborious, expensive, and may be subjective as, depending on the MIC, an inhibition zone around all strip can be formed [18]. Similar methods that have been used are the direct overlaid of the strips [19] and a disk placed next to the strip [40], which present the same difficulties for interpretation as MIC:MIC ratio method. Another method also used to evaluate ATM plus CZA synergism was strip crossing [17,21,22,41]. All methods using gradient strips require the MIC to be previously determined, which means that it will increase by at least one more day to provide synergy results. Moreover, such methods will present increased costs, as gradient strips are expensive. As an alternative, a double-disk synergy test can be performed in order to reduce costs, although it is a qualitative assay and it requires an overnight incubation [42].
Regardless of the fact that there are several methods to evaluate synergism in vitro, the standard assays are only TKC and checkerboard. Regarding ATM and CZA combination, few studies have performed TKC [16,24,25,43] and checkerboard [21,43,44]. They all have found synergism for the ATM plus CZA combination; however, both assays are laborious and time-consuming, which would not be suitable for a routine microbiology laboratory. The studies mentioned above assessing ATM plus CZA synergism have tested various isolates regarding beta-lactamase production, such as KPC, NDM, other MBL, OXA, and/or ESBL producers; however, only a few isolates were carbapenemase co-producers, such as NDM and OXA [43,44]. In fact, two studies evaluated isolates that were KPC and NDM co-producers [23,43]. We aimed to test carbapenemases-producing isolates according to our local epidemiology, which indicates an increase in NDM dissemination with the co-occurrence of NDM and KPC [9]. Similar to our results, studies that tested NDM and ESBL producers by TKC also found synergism of ATM plus CZA combination [16,24,25]. The first study that evaluated one isolate co-producing KPC and NDM performed a strip crossing method, also demonstrating synergism [23]. On the other hand, Lu et al. (2022) demonstrated synergistic activity of ATM plus CZA combination by TKC assays for one Enterobacter cloacae IMP-producer and one Escherichia coli NDM-producer that were resistant to both antibiotics. Additionally, they showed synergism by checkerboard for other species co-producing KPC and NDM, including K. pneumoniae, by checkerboard method [43]. A limitation of our study and others is that isolates expressing KPC variants that led to CZA resistance, such as KPC-35, KPC-78, and KPC-33 [45], were not tested to assess whether synergistic effects would still occur. Although it was not possible to correlate with clinical outcomes, our results by SynMALDI were confirmed by TKC, which was performed for four representative isolates of our sample, all K. pneumoniae, and by the double-disk synergism method.
The method we presented here, SynMALDI, could be performed and provide synergy results in approximately 4 to 5 h, and once the laboratory already has a MALDI-TOF mass spectrometer, the costs would be very low (less than US$ 2.00). Additionally, it is a simple method to be performed, and it does not require additional software for the interpretation of results. Another advantage is that this methodology can presumably be adapted for other antibiotic combinations. We adjusted conditions for CZA to correspond to the same conditions of ATM in order to perform the assay for these antibiotics combined. This is, so far, the first method proposed to detect synergism by MALDI-TOF MS.

5. Conclusions

As resistance mechanisms against beta-lactam antibiotics tend to evolve faster than new antibiotics are developed, combined therapy with two or more antibiotics is recommended. Therefore, in order to help guide patient therapy for infections due to CRE co-producers of carbapenemase, it is important to develop fast methods, such as SynMALDI, for synergism detection that can be performed in microbiology routine laboratories.

Author Contributions

Writing—original draft: C.M.W.; Writing—review and editing: C.M.W., E.I., A.F.M. and A.L.B.; conceptualization: C.M.W. and E.I.; investigation: C.M.W.; methodology: C.M.W. and E.I.; formal analysis: C.M.W. and A.F.M.; project administration: A.L.B.; funding acquisition: A.L.B. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by Instituto Nacional de Pesquisa em Resistência Antimicrobiana—INPRA (INCT/CNPq 465718/2014-0 and INCT/FAPERGS 17/2551-0000514-7) and Fundo de Incentivo à Pesquisa e Eventos from Hospital de Clínicas de Porto Alegre—FIPE/HCPA (project number 2019-0349).

Institutional Review Board Statement

This research was approved by the Hospital de Clínicas de Porto Alegre from Universidade Federal do Rio Grande do Sul (HCPA UFRGS) ethical committee, under CAAE number 16506819.0.0000.5327.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data is contained within the article and are available from the corresponding author upon reasonable request.

Acknowledgments

We thank the staff of the microbiology laboratory from Hospital de Clínicas de Porto Alegre for the collaboration.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. CDC Antibiotic Resistance Threats in the United States. 2019. Available online: www.cdc.gov (accessed on 9 July 2022).
  2. WHO. Global Priority List of Antibiotic-Resistant Bacteria to Guide Research, Discovery and Development of New Antibiotics; WHO: Rome, Italy, 2017; pp. 1–7. [Google Scholar]
  3. Coleman, K. Diazabicyclooctanes (DBOs): A potent new class of non-β-lactam β-lactamase inhibitors. Curr. Opin. Microbiol. 2011, 14, 550–555. [Google Scholar] [CrossRef]
  4. Bush, K.; Bradford, P.A. B-lactams and B-lactamase inhibitors: An overview. Cold Spring Harb. Perspect. Med. 2016, 6, a025247. [Google Scholar] [CrossRef] [PubMed]
  5. Algammal, A.; Hetta, H.F.; Mabrok, M.; Behzadi, P. Editorial: Emerging multidrug-resistant bacterial pathogens “‘superbugs’”: A rising public health threat. Front. Microbiol. 2023, 14, 1135614. [Google Scholar] [CrossRef]
  6. Karampatakis, T.; Tsergouli, K.; Behzadi, P. Carbapenem-resistant Klebsiella pneumoniae: Virulence factors, molecular mpidemiology and latest updates in treatment options. Antibiotics 2023, 12, 234. [Google Scholar] [CrossRef]
  7. Castanheira, M.; Deshpande, L.M.; Mendes, R.E.; Canton, R.; Sader, H.S.; Jones, R.N. Variations in the occurrence of resistance phenotypes and carbapenemase genes among enterobacteriaceae isolates in 20 years of the SENTRY antimicrobial surveillance program. In Open Forum Infectious Diseases; Oxford University Press: Oxford, UK, 2019; Volume 6, pp. S23–S33. [Google Scholar]
  8. Bush, K.; Bradford, P.A. Epidemiology of β-lactamase-producing pathogens. Clin. Microbiol. Rev. 2020, 33, e00047-19. [Google Scholar] [CrossRef]
  9. Wink, P.L.; Martins, A.S.; Volpato, F.; Zavascki, A.P.; Barth, A.L. Increased frequency of bla NDM in a tertiary care hospital in southern Brazil. Braz. J. Microbiol. 2021, 52, 299–301. [Google Scholar] [CrossRef]
  10. Zhang, B.; Zhu, Z.; Jia, W.; Qu, F.; Huang, B.; Shan, B.; Yu, H.; Tang, Y.; Chen, L.; Du, H. In vitro activity of aztreonam–avibactam against metallo-β-lactamase-producing Enterobacteriaceae—A multicenter study in China. Int. J. Infect. Dis. 2020, 97, 11–18. [Google Scholar] [CrossRef] [PubMed]
  11. Veeraraghavan, B.; Bakthavatchalam, Y.D.; Soman, R.; Swaminathan, S.; Manesh, A.; Nagvekar, V.; Nangia, V. Management of serious infections caused by metallo β-lactamases with or without OXA-48-like expressing Enterobacterales with aztreonam and ceftazidime/avibactam combination: Dosing strategy for better clinical outcome. Indian. J. Med. Microbiol. 2021, 39, 286–288. [Google Scholar] [CrossRef]
  12. Perrotta, F.; Perrini, M.P. Successful Treatment of Klebsiella pneumoniae NDM Sepsis and Intestinal Decolonization with Ceftazidime/Avibactam Plus Aztreonam Combination in a Patient with TTP Complicated by SARS-CoV-2 Nosocomial Infection. Medicina 2021, 57, 424. [Google Scholar] [CrossRef]
  13. Alghoribi, M.F.; Alqurashi, M.; Okdah, L.; Alalwan, B.; AlHebaishi, Y.S.; Almalki, A.; Alzayer, M.A.; Alswaji, A.A.; Doumith, M.; Barry, M. Successful treatment of infective endocarditis due to pandrug-resistant Klebsiella pneumoniae with ceftazidime-avibactam and aztreonam. Sci. Rep. 2021, 11, 9684. [Google Scholar] [CrossRef] [PubMed]
  14. Paul, M.; Carrara, E.; Retamar, P.; Tängdén, T.; Bitterman, R.; Bonomo, R.A.; de Waele, J.; Daikos, G.L.; Akova, M.; Harbarth, S.; et al. European Society of Clinical Microbiology and Infectious Diseases (ESCMID) guidelines for the treatment of infections caused by multidrug-resistant Gram-negative bacilli (endorsed by European society of intensive care medicine). Clin. Microbiol. Infect. 2022, 28, 521–547. [Google Scholar] [CrossRef] [PubMed]
  15. Tamma, P.D.; Aitken, S.L.; Bonomo, R.A.; Mathers, A.J.; van Duin, D.; Clancy, C.J. Infectious Diseases Society of America 2022 Guidance on the Treatment of Extended-Spectrum β-lactamase Producing Enterobacterales (ESBL-E), Carbapenem-Resistant Enterobacterales (CRE), and Pseudomonas aeruginosa with Difficult-to-Treat Resistance (DTR-P. aeruginosa). Clin. Infect. Dis. 2022, 75, 187–212. [Google Scholar] [PubMed]
  16. Marshall, S.; Hujer, A.M.; Rojas, L.J.; Papp-Wallace, K.M.; Humphries, R.M.; Spellberg, B.; Hujer, K.M.; Marshall, E.K.; Rudin, S.D.; Perez, F.; et al. Can ceftazidime-avibactam and aztreonam overcome B-lactam resistance conferred by metallo-B-lactamases in Enterobacteriaceae? Antimicrob. Agents Chemother. 2017, 61, e02243-16. [Google Scholar] [CrossRef] [Green Version]
  17. Monogue, M.L.; Abbo, L.M.; Rosa, R.; Camargo, J.F.; Martinez, O.; Bonomo, R.A.; Nicolau, D.P. In Vitro Discordance with In Vivo Activity: Humanized Exposures of Ceftazidime-Avibactam, Aztreonam, and Tigecycline Alone and in Combination against New Delhi Metallo-β-Lactamase- Producing Klebsiella pneumoniae in a Murine Lung Infection Model. Antimicrob. Agents Chemother. 2017, 61, e00486-17. [Google Scholar] [CrossRef] [Green Version]
  18. Wenzler, E.; Deraedt, M.F.; Harrington, A.T.; Danizger, L.H. Synergistic activity of ceftazidime-avibactam and aztreonam against serine and metallo-β-lactamase-producing gram-negative pathogens. Diagn. Microbiol. Infect. Dis. 2017, 88, 352–354. [Google Scholar] [CrossRef]
  19. Falcone, M.; Daikos, G.L.; Tiseo, G.; Bassoulis, D.; Giordano, C.; Galfo, V.; Leonildi, A.; Tagliaferri, E.; Barnini, S.; Sani, S.; et al. Efficacy of ceftazidime-avibactam plus aztreonam in patients with bloodstream infections caused by MBL-producing Enterobacterales. Clin. Infect. Dis. 2020, 586, 1871–1878. [Google Scholar]
  20. Sahu, C.; Pal, S.; Patel, S.S.; Singh, S.; Gurjar, M.; Ghoshal, U. Phenotypic synergy testing of ceftazidime–avibactam with aztreonam in a university hospital having high number of metallobetalactamase producing bacteria. Infect. Dis. 2020, 52, 801–807. [Google Scholar] [CrossRef]
  21. Khan, A.; Erickson, S.G.; Pettaway, C.; Arias, C.A.; Miller, W.R.; Bhatti, M.M. Evaluation of Susceptibility Testing Methods for Aztreonam and Ceftazidime-Avibactam Combination Therapy on Extensively Drug-Resistant Gram-Negative Organisms. Antimicrob. Agents Chemother. 2021, 65, e00846-21. [Google Scholar] [CrossRef]
  22. Avery, L.M.; Nicolau, D.P. Assessing the in vitro activity of ceftazidime/avibactam and aztreonam among carbapenemase-producing Enterobacteriaceae: Defining the zone of hope. Int. J. Antimicrob. Agents 2018, 52, 688–691. [Google Scholar] [CrossRef] [PubMed]
  23. Maraki, S.; Mavromanolaki, V.E.; Moraitis, P.; Stafylaki, D.; Kasimati, A.; Magkafouraki, E.; Scoulica, E. Ceftazidime-avibactam, meropenen-vaborbactam, and imipenem-relebactam in combination with aztreonam against multidrug-resistant, metallo-β-lactamase-producing Klebsiella pneumoniae. Eur. J. Clin. Microbiol. Infect. Dis. 2021, 40, 1755–1759. [Google Scholar] [CrossRef]
  24. Biagi, M.; Wu, T.; Lee, M.; Patel, S.; Butler, D.; Wenzler, E. Exploring aztreonam in combination with ceftazidime-avibactam or meropenem-vaborbactam as potential treatments for metallo- and serine-B-lactamase-producing Enterobacteriaceae. Antimicrob. Agents Chemother. 2019, 63, e01426-19. [Google Scholar] [CrossRef] [PubMed]
  25. Bocanegra-Ibarias, P.; Camacho-Ortiz, A.; Garza-González, E.; Flores-Treviño, S.; Kim, H.; Perez-Alba, E. Aztreonam plus ceftazidime-avibactam as treatment of NDM-1-producing Klebsiella pneumoniae bacteraemia in a neutropenic patient: Last resort therapy? J. Glob. Antimicrob. Resist. 2020, 23, 417–419. [Google Scholar] [CrossRef]
  26. Liu, Z.; Hang, X.; Yan, T.; Chu, W.; Gong, Z.; Liu, Y.; Dai, Y.; Yang, M.; Li, J.; Zhou, Q. A Simple Disk Stacking Plus Micro-Elution Method for Rapid Detection of the Synergistic Effect of Aztreonam and Ceftazidime/Avibactam Against Metallo-β-Lactamase Producing Enterobacterales. Infect. Drug. Resist. 2023, 16, 1537–1543. [Google Scholar] [CrossRef]
  27. Lange, C.; Schubert, S.; Jung, J.; Kostrzewa, M.; Sparbier, K. Quantitative matrix-assisted laser desorption ionization-time of flight mass spectrometry for rapid resistance detection. J. Clin. Microbiol. 2014, 52, 4155–4162. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Idelevich, E.A.; Becker, K. Matrix-assisted laser desorption ionization-time of flight mass spectrometry for antimicrobial susceptibility testing. J. Clin. Microbiol. 2021, 59, e0181419. [Google Scholar] [CrossRef] [PubMed]
  29. Sparbier, K.; Schubert, S.; Kostrzewa, M. MBT-ASTRA: A suitable tool for fast antibiotic susceptibility testing? Methods 2016, 104, 48–54. [Google Scholar] [CrossRef]
  30. Jung, J.S.; Hamacher, C.; Gross, B.; Sparbier, K.; Lange, C.; Kostrzewa, M.; Schubert, S. Evaluation of a semiquantitative matrix-assisted laser desorption ionization-time of flight mass spectrometry method for rapid antimicrobial susceptibility testing of positive blood cultures. J. Clin. Microbiol. 2016, 54, 2820–2824. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Axelsson, C.; Rehnstam-Holm, A.S.; Nilson, B. Rapid detection of antibiotic resistance in positive blood cultures by MALDI-TOF MS and an automated and optimized MBT-ASTRA protocol for Escherichia coli and Klebsiella pneumoniae. Infect. Dis. 2020, 52, 45–53. [Google Scholar] [CrossRef] [Green Version]
  32. Monteiro, J.; Widen, R.H.; Pignatari, A.C.C.; Kubasek, C.; Silbert, S. Rapid detection of carbapenemase genes by multiplex real-time PCR. J. Antimicrob. Chemother. 2012, 67, 906–909. [Google Scholar] [CrossRef] [Green Version]
  33. Sachse, S.; Bresan, S.; Erhard, M.; Edel, B.; Pfister, W.; Saupe, A.; Rödel, J. Comparison of multilocus sequence typing, RAPD, and MALDI-TOF mass spectrometry for typing of β-lactam-resistant Klebsiella pneumoniae strains. Diagn. Microbiol. Infect. Dis. 2014, 80, 267–271. [Google Scholar] [CrossRef]
  34. European Committee on Antimicrobial Susceptibility Testing (EUCAST). Breakpoint Tables for Interpretation of MICs and Zone Diameters. Available online: https://www.eucast.org/clinical_breakpoints (accessed on 30 June 2022).
  35. European Committee on Antimicrobial Susceptibility Testing (EUCAST). Routine Extended Internal Quality Control for MIC Determination Disk Diffusion as Recommended by EUCAST. Version 12.0. Available online: https://www.eucast.org/ast_of_bacteria/quality_control (accessed on 13 April 2022).
  36. Clinical Laboratory Standards Institute (CLSI). Methods for Determining Bactericidal Activity of Antimicrobial Agents: Approved Guideline; NCCLS Document M26-A; National Committee for Clinical Laboratory Standards: Albany, NY, USA, 1999. [Google Scholar]
  37. Wilhelm, C.M.; Nunes, L.D.S.; Martins, A.F.; Barth, A.L. In vitro antimicrobial activity of imipenem plus amikacin or polymyxin B against carbapenem-resistant Pseudomonas aeruginosa isolates. Diagn. Microbiol. Infect. Dis. 2018, 92, 152–154. [Google Scholar] [CrossRef] [PubMed]
  38. European Committee on Antimicrobial Susceptibility Testing. Antimicrobial Susceptibility Testing EUCAST Disk Diffusion Method Version 10.0. Available online: https://www.eucast.org/ast_of_bacteria/disk_diffusion_methodology (accessed on 12 December 2022).
  39. Wang, Y.; Wang, J.; Wang, R.; Cai, Y. Resistance to ceftazidime-avibactam and underlying mechanisms. J. Glob. Antimicrob. Resist. 2020, 22, 18–27. [Google Scholar] [CrossRef] [PubMed]
  40. Taha, R.; Kader, O.; Shawky, S.; Rezk, S. Ceftazidime-avibactam plus aztreonam synergistic combination tested against carbapenem-resistant Enterobacterales characterized phenotypically and genotypically: A glimmer of hope. Ann. Clin. Microbiol. Antimicrob. 2023, 22, 21. [Google Scholar] [CrossRef] [PubMed]
  41. Bianco, G.; Boattini, M.; Comini, S.; Casale, R.; Iannaccone, M.; Cavallo, R.; Costa, C. Occurrence of multi-carbapenemases producers among carbapenemase-producing Enterobacterales and in vitro activity of combinations including cefiderocol, ceftazidime-avibactam, meropenem-vaborbactam, and aztreonam in the COVID-19 era. Eur. J. Clin. Microbiol. Infect. Dis. 2022, 41, 573–580. [Google Scholar] [CrossRef] [PubMed]
  42. Leber, A.L. Synergism testing: Broth microdilution checkerboard and broth macrodilution methods. In Clinical Microbiology Procedures Handbook, 4th ed.; ASM Press: Washington, DC, USA, 2016; pp. 1–23. [Google Scholar]
  43. Lu, G.; Tang, H.; Xia, Z.; Yang, W.; Xu, H.; Liu, Z.; Ni, S.; Wang, Z.; Shen, J. In vitro and in vivo antimicrobial activities of ceftazidime/avibactam alone or in combination with aztreonam against carbapenem-resistant Enterobacterales. Infect. Drug. Resist. 2022, 15, 7107–7116. [Google Scholar] [CrossRef] [PubMed]
  44. Pragasam, A.K.; Veeraraghavan, B.; Shankar, B.; Bakthavatchalam, Y.; Mathuram, A.; George, B.; Chacko, B.; Korula, P.; Anandan, S. Will ceftazidime/avibactam plus aztreonam be effective for NDM and OXA-48-Like producing organisms: Lessons learnt from in vitro study. Indian J. Med. Microbiol. 2019, 37, 34–41. [Google Scholar] [CrossRef] [PubMed]
  45. Shi, Q.; Han, R.; Guo, Y.; Yang, Y.; Wu, S.; Ding, L.; Zhang, R.; Yin, D.; Hu, F. Multiple novel ceftazidime-avibactam-resistant variants of bla KPC-2-positive Klebsiella pneumoniae in two patients. Microbiol. Spectr. 2022, 10, e01714-21. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Flow chart demonstrating the procedures performed in the study. ATM: aztreonam. CZA: ceftazidime/avibactam. MEM: Meropenem. MIC: Minimum inhibitory concentration. R: Resistant. S: Susceptible.
Figure 1. Flow chart demonstrating the procedures performed in the study. ATM: aztreonam. CZA: ceftazidime/avibactam. MEM: Meropenem. MIC: Minimum inhibitory concentration. R: Resistant. S: Susceptible.
Antibiotics 12 01063 g001
Figure 2. Time–kill curve assay results performed for isolates Kp04 (A), Kp08 (B), Kp10 (C), and Kp16 (D). ATM: aztreonam. CZA: ceftazidime/avibactam.
Figure 2. Time–kill curve assay results performed for isolates Kp04 (A), Kp08 (B), Kp10 (C), and Kp16 (D). ATM: aztreonam. CZA: ceftazidime/avibactam.
Antibiotics 12 01063 g002
Figure 3. Boxplot of mean values of relative growth (RG) for aztreonam (ATM) and ceftazidime/avibactam (CZA) alone and in combination. (A) ATM and CZA resistant isolates positive for blaKPC and blaNDM. (B) ATM and CZA-resistant isolates positive for blaNDM only.
Figure 3. Boxplot of mean values of relative growth (RG) for aztreonam (ATM) and ceftazidime/avibactam (CZA) alone and in combination. (A) ATM and CZA resistant isolates positive for blaKPC and blaNDM. (B) ATM and CZA-resistant isolates positive for blaNDM only.
Antibiotics 12 01063 g003
Table 1. Aztreonam and ceftazidime/avibactam breakpoints for disk diffusion and minimal inhibitory concentration, according to EUCAST.
Table 1. Aztreonam and ceftazidime/avibactam breakpoints for disk diffusion and minimal inhibitory concentration, according to EUCAST.
AntibioticDisk Diffusion (mm)MIC (mg/L)
SRSR
Aztreonam≥26<21≤1>4
Ceftazidime/avibactam≥13<13≤8>8
S: Susceptible, standard dosing regimen. R: Resistant.
Table 2. Carbapenemases, spectral grouping, and susceptibility of all isolates included in this study.
Table 2. Carbapenemases, spectral grouping, and susceptibility of all isolates included in this study.
IsolateCpClusterATM CZA
DD (mm)Cat.DD (mm)Cat.
Kp01KPC, NDMF8R7R
Kp02KPC, NDMH10R6R
Kp03KPC, NDMA6R8R
Kp04KPC, NDMA6R8R
Kp05KPC, NDMH6R6R
Kp06KPC, NDME6R9R
Kp07KPC, NDMC6R6R
Kp08KPC, NDME6R8R
Kp09KPC, NDME6R9R
Kp10KPC, NDMG6R8R
Kp11NDMG6R9R
Kp12NDMG12R7R
Kp13NDMB8R8R
Kp14NDMG9R10R
Kp15NDMD6R6R
Kp16NDMF6R6R
Kp17NDMF6R6R
Kp18NDMC6R6R
Kp19NDMD6R8R
Kp20NDMD30S8R
Kp21KPCF6R20S
Kp22NTF30S24S
ATM: Aztreonam. Cat.: Antimicrobial susceptibility categorization according to EUCAST. Cp: Carbapenemase. CZA: Ceftazidime/avibactam. DD: disk diffusion. R: Resistant. S: Susceptible. NT: Not tested.
Table 3. Minimum inhibitory concentration and time–kill curve assay results of four representative isolates.
Table 3. Minimum inhibitory concentration and time–kill curve assay results of four representative isolates.
IsolateBeta-Lactamase GenesATM CZA ATM + CZA
MIC (mg/L)TKC *MIC (mg/L)TKC *TKC *
Kp04NDM-1, KPC-2, CTX-M-15, SHV-187, TEM-181>102410.42>25610.345.65
Kp08NDM-1, KPC-2, OXA-1, SHV-187, TEM-181>102410.86>25610.833.34
Kp10NDM-1, KPC-2, OXA-1, CTX-M-15, SHV-187, TEM-181>102410.28>25610.420.00
Kp16NDM-1, OXA-1, CTX-M-15, SHV-187, TEM-181>102410.70>25610.466.98
* Log10 CFU/mL at 24 h incubation in time–kill curve (TCK) assay. ATM: Aztreonam. CZA: Ceftazidime/avibactam. MIC: Minimum inhibitory concentration.
Table 4. Relative growth (RG) values obtained for aztreonam (ATM) and ceftazidime (CZA) in combination and alone.
Table 4. Relative growth (RG) values obtained for aztreonam (ATM) and ceftazidime (CZA) in combination and alone.
IsolateATM + CZAATM CZA
RGRGDif. *RGDif. *
Kp010.10821.11961.01141.49971.3915
Kp020.05440.82710.77260.91600.8616
Kp030.11900.79150.67251.19071.0717
Kp040.07930.93600.85660.91230.8330
Kp050.19380.56170.36790.66470.4709
Kp060.07040.70190.63160.74110.6708
Kp070.09660.82820.73150.97100.8744
Kp080.10420.83890.73461.51111.4068
Kp090.04860.64400.59550.58410.5356
Kp100.08410.88800.80400.78330.6993
Kp110.08740.81290.72550.61410.5267
Kp120.11270.71600.60330.72090.6082
Kp130.44111.33130.89020.95920.5181
Kp140.09120.77520.68400.88640.7952
Kp150.08210.76180.67970.77120.6891
Kp160.06960.93950.86991.09621.0266
Kp170.07960.96230.88270.83980.7602
Kp180.08130.85810.77681.23941.1581
Kp190.53380.89670.36301.13050.5967
Kp200.05280.34160.28880.65010.5973
Kp210.05850.85030.79180.06470.0062
Kp220.03110.21950.18840.03140.0003
* Difference of RG value in relation to RG ATM + CZA.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Wilhelm, C.M.; Inamine, E.; Martins, A.F.; Barth, A.L. Evaluation of Aztreonam and Ceftazidime/Avibactam Synergism against Klebsiella pneumoniae by MALDI-TOF MS. Antibiotics 2023, 12, 1063. https://doi.org/10.3390/antibiotics12061063

AMA Style

Wilhelm CM, Inamine E, Martins AF, Barth AL. Evaluation of Aztreonam and Ceftazidime/Avibactam Synergism against Klebsiella pneumoniae by MALDI-TOF MS. Antibiotics. 2023; 12(6):1063. https://doi.org/10.3390/antibiotics12061063

Chicago/Turabian Style

Wilhelm, Camila Mörschbächer, Everton Inamine, Andreza Francisco Martins, and Afonso Luís Barth. 2023. "Evaluation of Aztreonam and Ceftazidime/Avibactam Synergism against Klebsiella pneumoniae by MALDI-TOF MS" Antibiotics 12, no. 6: 1063. https://doi.org/10.3390/antibiotics12061063

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop