Next Article in Journal
Multi-Scale Evaluation on Two Locations and Digital Fruit Imaging Highlight Morpho-Agronomic Performances and Antioxidant Properties in Chilli Pepper Hybrids
Next Article in Special Issue
Hot Pressurized Liquid Extraction of Polyphenols from the Skin and Seeds of Vitis vinifera L. cv. Negra Criolla Pomace a Peruvian Native Pisco Industry Waste
Previous Article in Journal
Physicochemical Characterization and Functional Potential of Phaseolus vulgaris L. and Phaseolus coccineus L. Landrace Green Beans
Previous Article in Special Issue
Phytochemical and Functional Characterization of Phenolic Compounds from Cowpea (Vigna unguiculata (L.) Walp.) Obtained by Green Extraction Technologies
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

An Optimized Extraction Procedure for Determining Acaricide Residues in Foundation Sheets of Beeswax by Using Gas Chromatography-Mass Spectrometry

1
I. U. CINQUIMA, Analytical Chemistry Group, Faculty of Sciences, University of Valladolid, Paseo de Belén 7, 471011 Valladolid, Spain
2
Centro de Investigación Apícola y Agroambiental (CIAPA) de Marchamalo, Instituto Regional de Investigación y Desarrollo Agroalimentario y Forestal de Castilla La Mancha (IRIAF), Camino de San Martín, s/n, 19180 Marchamalo, Spain
*
Author to whom correspondence should be addressed.
Agronomy 2021, 11(4), 804; https://doi.org/10.3390/agronomy11040804
Submission received: 24 March 2021 / Revised: 16 April 2021 / Accepted: 16 April 2021 / Published: 19 April 2021

Abstract

:
Pesticides can be found in beehives for several reasons, including contamination from surrounding cultivars; yet one of the most pertinent is related to the fact that beekeepers employ acaricides to control various types of mites, which may accumulate in beeswax due to their lipophilic nature. In the present study, foundation sheets of different origins, collected over a period of three years, were analyzed to detect the residues of seven acaricides (atrazine, chlorpyrifos, chlorfenvinphos, alpha-endosulfan, bromopropylate, coumaphos, tau-fluvalinate) by gas chromatography with mass spectrometric detection. An efficient sample treatment (recoveries between 90% and 108%) is proposed, involving solvent extraction with 1% acetic acid in acetonitrile mixture followed by dispersive solid-phase extraction (enhanced matrix removal lipid) and a polishing step. An evaluation was made of the analytical performance of the proposed method. It was shown to be selective, linear from a limit of quantification to 5000 µg/kg, precise (relative standard deviation values were below 6%), and with a goo sensitivity (limit of quantification ranging from 5 to 10 µg/kg). Finally, results showed that a large majority of the sheets analyzed (>90%) contained residues of at least one of these compounds. Coumaphos and tau-fluvalinate residues were the most common, with chlorpyrifos and chlorfenvinphos detected to a lesser extent.

1. Introduction

In recent years, many European and North American countries have reported severe losses in honeybee populations [1]; other pollinator populations also appear to be on the decline [2]. The loss of honey production due to the decline in the health of honeybee colonies [3] may be more significant in temperate areas where professional beekeeping is fundamentally dedicated to honey production, the main product of the beekeeping industry. In Europe, where the number of beehives is continuously increasing, with more than 16 million hives [4], Spain is the country with the largest number of honey bee colonies (around 3 million) [5,6]. Therefore, the decrease in honey production in Spain is a major concern for professional beekeepers [7]. Currently, Nosema ceranae and Varroa destructor are major pathogens causing health problems in honeybee colonies in Europe. In particular, the worldwide emergence of the problems created by Varroa destructor has required the use of different mitigating compounds such as acaricides, whose effectiveness is becoming increasingly less due in large part to resistance phenomena [8] and different alternatives for their use. Both are highly conditioned by the method of application and the climatic conditions and the health status of the hive [9]. This situation leads to higher doses than those frequently applied, which implies a strong likelihood of residues appearing in the different hive products, mainly pollen and beeswax. This issue has been investigated in many studies, with pesticide residues being found in beeswax in several cases [10,11,12]. For example, in the study carried out in the US [10], it was reported that more than 60% of the 259 wax and 350 pollen samples contained at least one systemic pesticide, and over 47% had the acaricides fluvalinate and coumaphos.
The situation is quite similar in Europe, as unwanted compounds were confirmed in several studies from different countries. Indeed, the European Commission has since 2018 encouraged investigation into the decline of pollinators, including the causes and consequences [5,13,14,15,16,17,18,19,20,21,22,23], with the aim of preventing it. However, it should be mentioned that pesticide residues in bee products, especially beeswax, were documented in several publications [24,25,26,27,28,29], in which amitraz, coumaphos, chlorfenvinphos, and tau-fluvalinate were detected in many of the analyzed samples. This confirms the significance of acaricide residues as the group of pesticides that are most frequently found in this matrix, including virgin beeswax. These compounds can also resist the wax cleaning and melting temperature; therefore, they can accumulate for a long time. Considering that combs were submitted for wax extraction and the compounds are not eliminated in this procedure, their concentration would increase after recycling the wax in the form of foundations. In fact, it was reported that coumaphos persisted in beeswax for five years [30]. It is, therefore, essential to have some knowledge of the possible residues of miticidal treatments in the wax sheets to be introduced into the hive as a base for the bees’ production of larvae and storage of honey.
Regarding the determination of pesticides in beeswax, there are no official methods for sample preparation and analysis. There are many methods that can be used, and they are usually selected on the basis of the materials and equipment available in the laboratory. As can be expected, the instruments for performing this task have evolved remarkably in the last few years. The current trend is the use of hybrid techniques combining chromatography with mass spectrometry in its different modalities [31]. Different solvents have been employed to extract pesticides from beeswax, such as hexane [10,22,24,25,26,32,33], acetone [34], or mixtures of acetone with hexane [35] or water [36], although in several studies, a water and acetonitrile mixture was selected [10,20,27,37,38,39]. To minimize potential matrix interferences, various clean-up procedures have been proposed, including solid-phase extraction with C18 and/or florisil-based cartridges [10,25,26,32,34], filtration [40], gel permeation chromatography [41], or matrix solid dispersion combined with clean-up on C18 or florisil [42]. However, the current trend in sample preparation techniques focuses on simplifying these procedures to reduce the number/amount of reagents and time spent on this step. Thus, in recent years, the sample preparation is known to be quick, easy, cheap, effective, rugged, and safe (QuEChERS) has generally been used as sample treatment for determining pesticides in beeswax [10,20,27,38,43,44,45]. Different modifications of the QuEChERS methodology have been investigated in these studies, such as using a freezing-out step or a dispersive SPE (dSPE) as a final clean-up.
Our aim was to increase knowledge concerning the presence of the most frequently detected pesticides in beeswax by determining their content in laminated beeswax sheets (natural, purified and decontaminated), usually used by Spanish beekeepers for three years. Field trials in different apiaries were conducted in order to confirm their transfer to the wax comb. To this end, we developed and validated an analytical method that allowed the determination of pesticides in beeswax with good recoveries and decreasing as far as possible the potential matrix effect on MS detection.

2. Materials and Methods

2.1. Chemical and Materials

Analytical-grade standard of pesticides (atrazine, Det. Purity 99.5%; chlorpyrifos, Der. Purity 99.6%; chlorfenvinphos, Der. Purity 98.9%; alpha-endosulfan, Der. Purity 99.0%; bromopropylate, Der. Purity 99.6%; coumaphos, Der. Purity 99.5%; tau-fluvalinate, Der. Purity 99.6%; chlorfenvinphos-d10, Der. Purity 99.1%) were purchased from Dr. Ehrenstorfer (Augsburg, Germany). Isotope-labeled standard (chlorfenvinphos-d10) was chosen as the internal standard (IS) since they have the same physical and chemical properties as the unlabeled analytes. Solvents of Pestinorm grade (acetonitrile, ACN; ethyl acetate, ciclohexane) were obtained from VWR Prolabo Chemicals (Fontenay-sous-Bois, France). Solid reagents and acetic acid were obtained of analytical grade from Sigma-Aldrich (Saint Louis, MO, USA), while QhEChERS reagents were purchased from HPC standards GmbH (Cunnersdorf, Germany). QuEChERS dSPE enhanced matrix removal lipid (EMR-Lipid) and Polish (sodium chloride/magnesium sulfate) tubes were supplied by Agilent Technologies (Folsom, CA, USA). A vibromatic mechanical shaker, a thermostated ultrasound bath and a drying oven, supplied by J.P. Selecta S.A. (Barcelona, Spain), a vortex mechanical mixer from Heidolph (Schwabach, Germany), a 5810 R refrigerated bench-top centrifuge from Eppendorf (Hamburg, Germany), a R-3 rotary evaporator from Buchi (Flawil, Switzerland), a M-20 grinder and an Ultra-Turrax® homogenizer T18, both from IKA (Staufen, Germany) were employed for sample treatment. Nylon syringe filters (17 mm, 0.45 μm) were from Nalgene (Rochester, NY, USA), and ultrapure water was obtained using Milipore Milli-RO plus and Milli-Q systems (Bedford, MA, USA).

2.2. Standards

Standard (matrix-free) stock solutions of each pesticide were prepared by dissolving different amounts of each accurately weighed compound (10 mg) in 10 mL of an ethyl acetate and ciclohexane (20:80, v/v) mixture (~1000 mg/L). These solutions were further diluted with the ethyl acetate and ciclohexane (20:80, v/v) mixture to prepare the intermediate and calibration matrix-free standards. Wax samples (2.0 g) were spiked before (BF samples) or after (AF samples) sample treatment, with different amounts of studied acaracides and with the IS (0.5 µg/kg) to prepare the matrix-matched standards for validation (QC samples calibration curves), matrix effect and sample treatment studies. Purified white wax was used to prepare the standards, and it was necessary to heat the beeswax at 70 °C when spiking with the neonicotinoids to obtain homogenous BF samples. Each QC sample was prepared with 2.0 g of wax spiked with three different concentrations of pesticides within the linear range. These were as follows: low QC, the limit of quantification (LOQ; see Table 1); medium QC, 500 µg/kg; high QC, 5000 µg/kg. The stock solution was stored in glass containers in darkness at −20 °C; working and matrix-matched solutions were stored in glass containers and kept in the dark at +4 °C. All the solutions were stable for over two weeks (data not shown).

2.3. Sample Procurement and Treatment

2.3.1. Samples

Laminated beeswax sheets that were used by Spanish beekeepers between 2018 and 2020 were obtained from commercial suppliers from Spain, Portugal, France and Germany; there were certain differences in quality, including decontaminated ones. In addition, laminated sheets from old, recycled combs were provided by some beekeepers. It should be noted that the appearance and names of the samples were somewhat varied. Thus, those marketed as decontaminated were usually quite white, while those marketed as purified, pure, commercial, raw, etc., ranged from light to dark yellow. Obviously, the darker they were, the greater the number of residues observed (see Section 3.3.1). Moreover, the decontamination procedure was not specified as not all the manufacturers/suppliers provided this information; in some cases, however, the liquid wax was filtrated [46,47]. Bleached beeswax pellets (Fluka Chemie, Steinheim, Switzerland) were employed as blanks in the assays. These underwent preliminary analysis by gas chromatography coupled to a mass spectrometry detector (GC-MS) to check for the presence of pesticides. Once it was confirmed that there was no residual trace of the compounds under study, sub-samples were used as blanks to prepare matrix-matched standards. Subsequently, the pieces of comb from the same apiary were combined and melted, then cleaned by soaking in heated distilled water (70 °C; three times) so as to remove the residual honey. All the samples were kept at −20 °C until analysis.

2.3.2. Sample Preparation

All the samples were previously crushed with dry ice in a mortar and thoroughly homogenized for analysis. Next, 2.0 g of wax was weighed in a 50 mL centrifuge tube, and 10 mL of 1% acetic acid in an acetonitrile mixture was added. The tube was then shaken for 3 min in a vortex device and homogenized for 2.5 min in an Ultra-Turrax®. The extract was then centrifuged (7500 rpm, 5 °C) for 5 min, after which 5 mL of supernatant was transferred to a QhEChERS dSPE EMR cartridge previously activated with 5 mL of ultrapure water. The mixture was shaken for 1 min a vortex device and centrifuged again (7500 rpm, 5 °C, 5 min). The supernatant (5 mL) was then transferred to the polish tube (sodium chloride/magnesium sulfate), shaken for 1 min a vortex device and centrifuged in the same previously mentioned conditions. Two milliliters of the supernatant were evaporated to dryness in a rotary evaporator (60 °C). The dry extract was reconstituted with 1 mL of ethyl acetate ciclohexane (20:80, v/v) mixture in an ultrasound bath; the resulting solution was passed through a nylon filter. Following this, a 1-µL aliquot was injected into the GC-MS system

2.4. GC-MS Conditions

An Agilent Technologies (Palo Alto, CA, USA) 7890A gas chromatograph (GC) coupled to an Agilent Technologies 5975C mass spectrometer (MS), equipped with an ALS 7693B autosampler and an MS ChemStation E 01.00.237 software (Agilent Technologies), was employed. The chromatographic column was an Agilent DB-5MS (30 m × 0.25 mm × 0.2 μm). The GC was operated under programmed temperature conditions, from 60 °C (1 min) to 170 °C (5 min), at 40 °C/min and then increased to 195 °C (10 min) at 9 °C/min. Finally, the temperature was increased to 310 °C (3 min) at 10 °C/min. An injection volume of 1 μL was employed with the autosampler in pulsed splitless mode, the injector temperature set at 280 °C, and helium (Carburos Metálicos, Barcelona, Spain) used as the carrier gas at a flow-rate of 1.2 mL/min. The MS scan parameters included a mass range of 50–400 m/z, operating in electron impact mode with an ionization energy of 70 eV. The ion source and quadrupole temperatures were 230 °C and 150 °C, respectively. Analyses were performed in selected ion monitoring (SIM) mode, with one target/quantification and two qualifier ions for each of the analytes (see Table 2 and Figure 1). The latter were identified and confirmed by comparison of their retention times and mass spectra with a Mass Spectra Library, Wiley 7N edition (Agilent Part No. G1035B) and reference compounds.

2.5. Method Validation

The validation study was performed based on the current European legislation (SANTE guidelines; [48]). The criteria include selectivity, limits of detection (LODs) and quantification (LOQs), matrix effect, linearity, precision and trueness. Basic but efficient chemometric statistical tools from Excel (Microsoft Office 2010, Microsoft Corporation, Redmond, WA, USA), were employed to analyze the data to validate the method.
To determine the selectivity of the proposed method, a set of extracts from non-spiked samples (n = 6) together with spiked samples were injected onto the chromatographic system. The LODs and LOQs were experimentally determined by the injection of a number of blank samples (n = 6), and they were estimated to be, respectively, three and ten times the signal-to-noise (S/N) ratio. In order to ascertain how the matrix influenced MS signal, a comparison was made of the results (analyte peak area/IS peak area) with standard working solutions and blank beeswax samples spiked at three different concentrations (QC levels) following sample treatment (AF samples). The reference standard in the solvent (matrix-free) calibration curves were used to quantify atrazine, chlorpyrifos, chlorfenvinphos, alpha-endosulfan, and bromopropylate, due to the absence of a significant matrix effect. Meanwhile, coumaphos and tau-fluvalinate were quantified with matrix-matched calibration curves as the matrix provoked a significant signal enhancement (see Section 3.2). Calibration curves (n = 6) were constructed by plotting the signal on the y-axis (analyte peak area/IS peak area) against the analyte concentration on the x-axis. Standards were prepared as described in Section 2.2. Precision experiments were performed concurrently by repeated sample analysis using BF samples on the same day (n = 6; intra-day precision), or over three consecutive days (n = 6; inter-day precision). Trueness was evaluated by the mean recoveries (as a measure of trueness), calculated by comparing the responses (analyte peak areas/IS peak area) obtained from BF and AF samples.

3. Results and Discussion

3.1. Sample Treatment

In view of the recent studies determining pesticides in beeswax, it was decided that a QuEChERS extraction should initially be tested for sample treatment because of its simplicity, relatively low cost and promising results. We initially applied familiar QuEChERs protocols [29,31,45,49]. Although the results were good, especially in terms of recovery percentages, we tried to improve certain steps and, above all, reduced the analysis time. Therefore, a series of tests were carried out in order to simplify sample treatment. Firstly, consideration was given to the amount of beeswax to be analyzed, and after several tests (0.5–5 g) 2.0 g beeswax was selected as the maximum amount to be used. Recoveries were adequate with this amount, and good S/N ratios were achieved to obtain the lowest possible limits of detection (LODs) and quantification (LOQs). For the extraction step, tests were performed with 10 mL of different mixtures of acetonitrile, which were chosen by following an optimization procedure (data not shown). Results showed that the best performance in terms of pesticide recoveries was obtained with 1% acetic acid in acetonitrile (>89% see Table 3).
In order to check if the differences among the different solvents were significant or not, it was decided to perform a two-tailed t-test (Microsoft Excel). Results showed that significant differences were observed after performing a two-tailed t-test between each of the mixtures and what theoretically it should be taken in case of maximum performance (100% of recovery, with the exception of chlorfenvinphos and coumaphos in the 1% acetic acid in acetonitrile mixture (data not shown). Thus, it was decided to continue the optimization with the acetic and acetonitrile mixture as it provided the highest recoveries, which were much closer to 100% than the values obtained with the other solvents. This finding could be related to the fact that the use of acetonitrile has the advantage of precipitating proteins and limiting lipid solubility. Moreover, proteins are denatured in pure acetonitrile or aqueous-acetonitrile mixtures, and this increased insecticide extraction efficiency [50,51].
The next step was to ascertain the influence of the shaking/mixing device used to facilitate extraction and thereby evaluate the potential significance of this factor. Two different options were tested, namely, vibromatic and Ultra-Turrax®; the best results (recoveries and precision) were obtained with the latter (data not shown). Next, the influence of the extraction time (1–5 min) was tested to obtain optimal conditions. The best extraction efficiencies (recovery percentages > 90%) were achieved with 2.5 min for extraction. As beeswax is a complex matrix that contains several substances, it is necessary to remove certain matrix components that might affect pesticide ionization. Consequently, a clean-up step was introduced by using a recently commercialized sorbent (enhanced matrix removal-lipid; EMR-lipid). This novel sorbent, which contains C18 and certain special polymers, was chosen for this step since it has displayed promising results when analyzing other pesticides (neonicotinoids) in beeswax [52].
Following this, a polish tube containing sodium chloride and magnesium sulfate was used, as this extra drying step is necessary to remove any water or salt residue remaining in the samples, especially those requiring a concentration step like those of the present study. After this last clean-up step, 2 mL of the supernatant was collected, transferred to a 25-mL conical flask, and gently evaporated to dryness in a rotary evaporator at 60 °C. It is worth mentioning that no loss of pesticides was observed during the evaporation step. This issue was checked by comparing the responses of the analytes in standards and spiked samples of the same concentrations (QCs) before and after the evaporation, which were quite similar (>90%) in all cases (data not shown). Finally, reconstitution was deemed important to improve extraction efficacy. Different volumes (0.5–2.0 mL) of the ethyl acetate and ciclohexane (80:20, v/v) mixture were assayed, and since it was observed that amounts of solvent in excess of 1 mL did not improve the recovery percentages (data not shown), 1 mL of the mixture was considered appropriate to reconstitute the dry residue.

3.2. Method Validation

None of analyzed samples showed chromatographic interference at the retention times of the analytes. In order to identify the compounds, a comparison was made of the mass spectra in standard solutions and spiked beeswax samples; this involved comparable concentrations and measurements under the same conditions. Both mass spectra were quite similar, especially in the case of the ions selected for quantification and confirmation purposes (data not shown). Therefore, it can be concluded that the proposed method is selective. Low LODs and LOQs were obtained in all cases (see Table 1), as the values were between 2 and 10 µg/Kg. In relation to the evaluation of the matrix, it was observed that the responses at the three concentrations (QC levels) assayed ranged from 93% to 105% for five of the compounds studied (atrazine, chlorpyrifos. chlorfenvinphos, alpha-endosulfan and bromopropylate; see Table 4), while values were higher than 120% for the other two compounds (coumaphos and tau-fluvalinate), and subsequently, they did not fulfill the criteria of the European Commission (more than 20% signal suppression or enhancement; [48]).
It must be remarked that no significant differences were observed between the different QC levels for most of the compounds after performing a two-tailed t-test. The different influences of the matrix in the MS signals of the acaricides were also confirmed by the overlapping at the confidence intervals of the slopes from the solvent and matrix-matched standard calibration curves for the five compounds mentioned above (data not shown). Therefore, it can be concluded that the matrix did not significantly affect the MS signal of five of the analytes, which implies that a reference standard in solvents could be used for their quantification. Meanwhile, matrix-matched standard calibration curves should be employed for the other two acaricides. The analytical range was between LOQ and 5000 µg/kg, and the graphs obtained in all the calibration curves were straight lines, with the coefficient of the determination values (R2) above 0.99 in all cases (see Table 1).
Precision results, expressed as relative standard deviation (%RSD), were at all times under or equal to 6% (data not shown). It must be remarked that those values are consistent with the current European legislation (%RSD ≤ 20; [48]). Finally, mean recoveries ranged in all cases from 90% to 108%, with %RSD values lower than 6% in all cases (see Table 4); these are quite satisfactory results and are within the recommended values by the European Commission (recovery percentages between 70% and 120%; % RSD ≤ 20; [48]). It was also checked if the differences among the different QCs were significant or not. The results (two-tailed t-test) showed that significant differences were observed in some cases, especially for the high QC samples (data not shown).

3.3. Application of the Method

3.3.1. Analysis of Foundation Wax

The average values found throughout the three years in which they have been collected are summarized in Table 5. Due to the high concentration values observed in most cases, the samples were diluted. As can be observed, the compounds being studied appeared quite regularly in the laminated waxes, with the exception of the decontaminated waxes, in which the amount/number of residues was significantly lower, and even practically non-existent in some of them. It can also be concluded that tau-fluvalinate was present in most of the samples, followed by coumaphos and chlorfenvinphos. As it was previously stated in the Introduction, the presence of those compounds in beeswax was previously reported in several publications [24,25,26,27,28,29].
In particular, the concentrations of detected coumaphos were quite high in some samples and were also much higher than those reported in 2010 [27]. This may be explained by authorization regarding its use. The detection of chlorfenvinphos in so many samples is another interesting finding, as its use is banned in several Spanish regions. Chlorpyrifos, meanwhile, appeared least frequently and usually in lower concentrations. This could be related to the fact that it is a plant protection agent and not applied to the hive. Finally, it should be mentioned that these results differed from those of certain studies in which it was reported that chlorfenvinphos was the compound most commonly detected and, less frequently, fluvalinate and coumaphos [26]. However, they are in better agreement with other publications that provided a similar order of detection, while also indicating that nearly 70% of the samples contained at least two acaricides [21], or that all the samples were positive in coumaphos and over 80% in fluvalinate [27]. Thus, it seems that the type of hive also affects the residues contained in the wax, and it logically depends to a great extent on the foundation waxes used.

3.3.2. Field Experiments

In 2020, experiments were carried out in various apiaries in Spain (Guadalajara, GU; Valladolid, VA; Palencia, PA) with three frames of different foundations, that is, with a different initial content in miticides, being placed in each hive. The results showed (see Table 6) that in all cases, there was pesticide transfer from the sheets to the wax used by the bees to build cells (new wax operculum). This transfer varied depending on the compound and the initial concentration in the foundation waxes. For example, the transfer observed for chlorpyrifos and chlorfenvinphos was above 40% in all cases, normally ranging from 65% to 90%. Meanwhile, a much lower transfer was generally observed for coumaphos and tau-fluvalinate, especially for those samples containing the highest concentrations. Finally, it was also demonstrated that the sheets used by beekeepers, even within the same province, are very different.

4. Conclusions

The great diversity of foundation sheets used by beekeepers was evident, and as a result, appreciable amounts of pesticide residues were generally present in the hive. The decontaminated foundation sheets that are currently commercially available displayed a small number of residues and in very low concentrations. Therefore, it is suggested that this material should be used whenever possible. Several commercial foundation sheets and those prepared by the beekeepers revealed large amounts of coumaphos. Almost all of the samples analyzed contained tau-fluvalinate residues; in addition, chlorfenvinphos was commonly detected. It was also observed that when any sheet was placed in the hive, there was a clear transfer to the cell building wax, which undoubtedly poses a risk for the hive’s survival.
For this reason, beekeepers should check for the presence of residues on the sheets they usually insert into their hives. It is recommended, therefore, that they ensure the quality of the sheets and that they do not rely on certain procedures that are commonly used for cleaning waxes. Finally, and regarding the analytical methodology, we have proven that homogenization with Ultra-Turrax® was instrumental in obtaining high rates of recovery with good precision. The use of new sorbents for lipid removal was also seen to be very efficient, contributing to a reduction in sample treatment time.

Author Contributions

Conceptualization, M.J.N., J.L.B. and M.H.; methodology, E.I., M.J.N. and J.B.; software, J.L.N.; validation, E.I., J.B. and M.J.N.; resources, M.H. and M.J.N.; data curation, J.L.N. and J.B.; writing—original draft preparation, J.L.B. and M.J.N.; writing—review and editing, J.B.; visualization, J.B. and J.L.B.; supervision, J.B. and M.J.N.; project administration, J.L.B. and M.J.N.; funding acquisition, M.J.N. and M.H. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the National Plan for Scientific and Technical Research and Innovation 2013–2016, National Institute for Agricultural and Food Research and Technology-INIA –FEDER (Spain), grant number RTA2017-00004-C02-01 and 02.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The authors thank David Rixham (White Rose English School, Valladolid, Spain) for the revision of the English language/grammar of the manuscript. In addition, the authors thank M. Martínez for his technical assistance.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Van Der Zee, R.; Pisa, L.; Andonov, S.; Brodschneider, R.; Charriére, J.D.; Chlebo, R.; Coffey, M.F.; Crailsheim, K.; Dahle, B.; Gajda, A.; et al. Managed honey bee colony losses in Canada, China, Europe, Israel and Turkey, for the winters of 2008–2009 and 2009–2010. J. Apic. Res. 2012, 51, 100–114. [Google Scholar] [CrossRef]
  2. Evison, S.E.F.; Roberts, K.E.; Laurenson, L.; Pietravalle, S.; Hui, J.; Biesmeijer, J.C.; Smith, J.E.; Budge, G.; Hughes, W.O.H. Pervasiveness of parasites in pollinators. PLoS ONE 2012, 7, 30641. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Botías, C.; Martín-Hernández, R.; Días, J.; García-Palencia, P.; Matabuena, M.; Juarranz, A.; Barrios, L.; Meana, A.; Nanetti, A.; Higes, M. The effect of induced queen replacement on Nosema spp. infection in honey bee (Apis mellifera iberiensis) colonies. Environ. Microbiol. 2012, 14, 845–859. [Google Scholar] [CrossRef] [PubMed]
  4. Agriculture and Rural Development-European Commission. Available online: http://ec.europa.eu/agriculture/honey.en (accessed on 16 April 2021).
  5. Ministerio de Agricultura, Pesca y Alimentación (Spain). Available online: https://www.mapa.gob.es/es/ganaderia/temas/produccion-y-mercados-ganaderos/indicadoreseconomicossectordelamiel2019_tcm30-419675.pdf (accessed on 16 April 2021).
  6. Chauzat, M.P.; Cauquil, L.; Roy, L.; Franco, S.; Hendrikx, P.; Ribière, M. Demographics of the European apicultural industry. PLoS ONE 2013, 8, 79018. [Google Scholar] [CrossRef]
  7. Cepero, A.; Ravoet, J.; Gómez-Moracho, T.; Bernal, J.L.; Nozal, M.J.; Bartolomé, C.; Maside, X.; Meana, A.; González-Porto, A.V.; De Graaf, D.C.; et al. Holistic screening of collapsing honey bee colonies in Spain: A case study. BMC Res. Notes 2014, 7, 649–659. [Google Scholar] [CrossRef] [Green Version]
  8. Martin, S.J. Acaricide (pyrethroid) resistance in Varroa destructor. Bee World 2004, 85, 67–69. [Google Scholar] [CrossRef]
  9. Gracia, M.J.; Moreno, C.; Ferrer, M.; Sanz, A.; Peribáñez, M.A.; Estrada, R. Field efficacy of acaricides against Varroa destructor. PLoS ONE 2017, 12, 0171633. [Google Scholar] [CrossRef]
  10. Mullin, C.A.; Frazier, M.; Frazier, J.L.; Ashcraft, S.; Simonds, R.; Van Engelsdorp, D.; Pettis, J.S. High levels of miticides and agrochemicals in North American apiaries: Implications for honey bee health. PLoS ONE 2010, 5, 9754. [Google Scholar] [CrossRef] [Green Version]
  11. Berry, J. Pesticides, bees and wax: An unhealthy, untidy mix. Bee Cult. 2009, 137, 33–35. [Google Scholar]
  12. Fulton, C.A.; Huff, K.E.; Reeve, J.A.; Lydy, J. An examination of exposure routes of fluvalinate to larval and adult honey bees (Apis mellifera). Environ. Toxicol. Chem. 2019, 38, 1356–1363. [Google Scholar] [CrossRef]
  13. Agrebi, N.; Willmart, O.; Urbain, B.; Danneels, E.L.; Graaf, D.C.; Saegerman, C. Belgian case study on flumethrin residues in beeswax: Possible impact on honeybee and prediction of the maximum daily intake for consumers. Sci. Total Environ. 2019, 687, 712–719. [Google Scholar] [CrossRef] [Green Version]
  14. Chauzat, M.P.; Faucon, J.P. Pesticide residues in beeswax samples collected from honey bee colonies (Apis mellifera L.) in France. Pest Manag. Sci. 2007, 63, 1100–1106. [Google Scholar] [CrossRef]
  15. Ravoet, J.; Reybroeck, W.; de Graaf, D.C. Pesticides for apicultural and/or agricultural application found in Belgian honey bee wax combs. Bull. Environ. Contam. Toxicol. 2015, 94, 543–548. [Google Scholar] [CrossRef] [Green Version]
  16. Wilmart, O.; Legrève, A.; Scippo, M.-L.; Reybroeck, W.; Urbain, B.; de Graaf, D.C.; Steurbaut, W.; Delahaut, P.; Gustin, P.; Nguyen, B.K.; et al. Residues in beeswax: A health risk for the consumer of honey and beeswax? J. Agric. Food Chem. 2016, 64, 8425–8434. [Google Scholar] [CrossRef]
  17. Boi, M.; Serra, G.; Colombo, R.; Lodesani, M.; Massi, S.; Costa, C. A 10 year survey of acaricide residues in beeswax analysed in Italy. Pest Manag. Sci. 2016, 72, 1366–1372. [Google Scholar] [CrossRef]
  18. Bogdanov, S.; Kilchenmann, V.; Imdorf, A. Acaricide residues in beeswax and honey. In Bee Products, Properties, Applications, Apitherapy; Mizrahi, A., Lensky, Y., Eds.; Plenum Press: New York, NY, USA, 1997; pp. 239–246. [Google Scholar]
  19. Roszko, M.L.; Kaminska, M.; Szymczyk, K.; Jedrzejczak, R. Levels of selected persistent organic pollutants (PCB, PBDE) and pesticides in honey bee pollen sampled in Poland. PLoS ONE 2016, 11, 0167487. [Google Scholar] [CrossRef]
  20. Bommuraj, V.; Chen, Y.; Klein, H.; Sperling, R.; Barel, S.; Shimshoni, J.A. Pesticide and trace elements residues in honey and beeswax combs from Israel in association with human risk assessment and honey adulteration. Food Chem. 2019, 299, 125123. [Google Scholar] [CrossRef]
  21. Perugini, M.; Tulini, S.M.R.; Zezza, D.; Fenucci, S.; Conte, A.; Amorena, M. Occurrence of agrochemical residues in beeswax samples collected in Italy during 2013–2015. Sci. Total Environ. 2018, 625, 470–476. [Google Scholar] [CrossRef]
  22. Bajuk, B.P.; Babnik, K.; Snoj, T.; Milčinski, L.; Ocepek, M.P.; Škof, M.; Jenčič, V.; Filazi, A.; Štajnbaher, S. Coumaphos residues in honey, bee brood, and beeswax after Varroa treatment. Apidologie 2017, 48, 588–598. [Google Scholar] [CrossRef]
  23. Tsigouri, A.; Menkissoglu, U.; Thrasyvoulou, A.; Diamantidis, G. Fluvalinate residues in Greek honey and beeswax. Apiacta 2003, 38, 50–53. [Google Scholar]
  24. Jimenez, J.J.; Bernal, J.L.; Nozal, M.J.; Martín, M.T. Residues of organic contaminants in beeswax. Eur. J. Lipid Sci. Technol. 2005, 107, 896–902. [Google Scholar] [CrossRef]
  25. Adamczyk, S.; Lazaro, R.; Perez, C.; Bayarri, S.; Herrera, A. Impact of the use of fluvalinate on different types of beeswax from Spanish hives. Arch. Environ. Contam. Toxicol. 2010, 58, 733–739. [Google Scholar] [CrossRef]
  26. Serra, J.; Orantes, J. Acaricides and their residues in Spanish commercial beeswax. Pest. Manag. Sci. 2010, 66, 1230–1235. [Google Scholar] [CrossRef]
  27. Calatayud, P.; Calatayud, F.; Simó, E.; Pico, Y. Occurrence of pesticide residues in Spanish beeswax. Sci. Total. Environ. 2017, 605–606, 745–754. [Google Scholar] [CrossRef]
  28. Lozano, A.; Hernando, M.D.; Uclés, S.; Hakme, E.; Fernandez, A.R. Identification and measurement of veterinary drug residues in beehive products. Food Chem. 2019, 274, 61–70. [Google Scholar] [CrossRef]
  29. Morales, M.M.; Gomez, M.J.; Parrilla, P.; Diaz, F.J.; García, M.; Gámiz, V.; Flores, J.M.; Fernández, A.R. Distribution of chemical residues in the beehive compartments and their transfer to the honeybee brood. Sci. Total Environ. 2020, 710, 136288. [Google Scholar] [CrossRef]
  30. Zhu, W.; Schmehl, D.R.; Mullin, C.A.; Frazier, J.L. Four common pesticides, their mixtures and a formulation solvent in the hive environment have high oral toxicity to honey bee larvae. PLoS ONE 2014, 9, 77547. [Google Scholar] [CrossRef]
  31. Svečnjak, L.; Chesson, A.C.; Gallina, A.; Maia, M.; Martinello, M.; Mutinelli, F.; Muz, M.N.; Nunes, F.; Saucy, F.; Tipple, B.J.; et al. Standard methods for Apis mellifera beeswax research. J. Apic. Res. 2019, 58, 1–108. [Google Scholar] [CrossRef] [Green Version]
  32. Bonzini, S.; Tremolada, P.; Bernardinelli, I.; Colombo, M.; Vighi, M. Predicting pesticide fate in the hive (Part 1) experimentally determined tau-fluvalinate residues in bees, honey and wax. Apidologie 2011, 41, 378–390. [Google Scholar] [CrossRef] [Green Version]
  33. Bogdanov, S.; Kilchenmann, V.; Butikofer, U. Determination of acaricide residues in beeswax: Collaborative study. Apiacta 2003, 38, 235–245. [Google Scholar]
  34. Chauzat, M.P.; Faucon, J.P.; Martel, A.C.; Lachaize, J.; Gougoule, N.; Aubert, M. A survey of pesticide residues in pollen loads collected by honey bees in France. J. Econ. Entomol. 2006, 99, 253–262. [Google Scholar] [CrossRef] [PubMed]
  35. Li, Y.; Kelley, R.A.; Anderson, T.D.; Lydy, M.J. Development and comparison of two multi-residue methods for the analysis of selected pesticides in honey bees, pollen and wax by gas chromatography quadrupole mass spectrometry. Talanta 2015, 40, 81–87. [Google Scholar] [CrossRef] [PubMed]
  36. Alkassab, A.T.; Thorbain, D.; Frommberger, M.; Bischoff, G.; Pistorius, J. Effect of contamination and adulteration of wax foundations on the brood development of honeybees. Apidologie 2020, 51, 642–651. [Google Scholar] [CrossRef] [Green Version]
  37. Frison, S.; Breitkreitz, W.; Currie, R.; Nelson, D.; Sporns, P. The analysis of fluvalinate in beeswax using GC/MS. Food Res. Int. 1999, 32, 35–41. [Google Scholar] [CrossRef]
  38. Gil, M.D.; Uclés, S.; Lozano, A.B.; Sosa, A.; Fernández, A.R. Muliresidue method for trace pesticide analysis in honeybee wax comb by GC-QqQ-MS. Talanta 2017, 163, 54–64. [Google Scholar]
  39. Shimshoni, J.A.; Sperling, R.; Massarwa, M.; Chen, Y.; Bommuraj, V.; Borisover, M.; Barel, S. Pesticide distribution and depletion kinetic determination in honey and beeswax: Model for pesticide occurrence and distribution in beehive products. PLoS ONE 2019, 14, 0212631. [Google Scholar] [CrossRef]
  40. Ghini, S.; Fernández, M.; Pico, Y.; Marín, R.; Fini, F.; Mañes, J.; Girotti, S. Occurrence and distribution of pesticides in the province of Bologna, Italy, using honeybees as bioindicators. Arch. Environ. Contam. Toxicol. 2004, 47, 479–488. [Google Scholar] [CrossRef]
  41. Rossi, S.; Dalpero, A.; Ghini, S.; Colombo, R.; Sabatini, A.; Girotti, S. Multiresidual method for the gas chromatographic analysis of pesticides in honeybees cleaned by gel permeation chromatography. J. Chromatogr. A 2001, 905, 223–232. [Google Scholar] [CrossRef]
  42. Łozowicka, B. The development, validation and application of a GC-dual detector (NPD-ECD) multi-pesticide residue method for monitoring bee poisoning incidents. Ecotoxicol. Environ. Safety 2013, 97, 210–222. [Google Scholar] [CrossRef] [PubMed]
  43. Anastassiades, M.; Lehotay, S.J.; Stajnbaher, D.; Schenck, F.J. Fast and easy multiresidue method employing acetonitrile extraction/partitioning and “dispersive solid-phase extraction” for the determination of pesticide residues in produce. J. AOAC Int. 2003, 86, 412–431. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  44. Niell, S.; Jesus, F.; Perez, C.; Mendoza, Y.; Diaz, R.; Franco, J.; Cesio, V.; Heinzen, H. QuEChERS adaptability for the analysis of pesticide residues in beehive products seeking the development of an agroecosystem sustainability monitor. J. Agric. Food Chem. 2015, 63, 4484–4492. [Google Scholar] [CrossRef]
  45. Lambert, O.; Piroux, M.; Puyo, S.; Thorin, C.; L’Hostis, M.; Wiest, L.; Buleté, A.; Delbac, F.; Pouliquen, H. Widespread occurrence of chemical residues in beehive matrices from apiaries located in different landscapes of western France. PLoS ONE 2013, 8, 67007. [Google Scholar] [CrossRef]
  46. Serra, J.; Orantes, F.J. Discoloration and adsorption of acaricides from beeswax. J. Food Process Eng. 2017, 40, 1–10. [Google Scholar]
  47. Navarro, M.D.; Orantes, F.J.; Sánchez, C.; Varela, A.; Giampieri, F.; Torres, C.; Serra, J.; Forbes, T.Y.; Reboredo, P.; Llopis, J.; et al. Industrial-scale decontamination procedure effects on the content of acaricides, heavy metals and antioxidant capacity of beeswax. Molecules 2019, 24, 1518. [Google Scholar]
  48. European Commission Directorate-General for Health and Food Safety, Document SANTE/12682/2019. Available online: https://www.eurl-pesticides.eu/userfiles/file/EurlALL/AqcGuidance_SANTE_2019_12682.pdf (accessed on 16 April 2021).
  49. Lehotay, S.J. Determination of pesticide residues in foods by acetonitrile extraction and partitioning with magnesium sulfate: Collaborative study. J. AOAC Int. 2007, 90, 485–520. [Google Scholar] [CrossRef] [Green Version]
  50. Valverde, S.; Bernal, J.L.; Martín, M.T.; Nozal, M.J.; Bernal, J. Fast determination of neonicotinoid insecticides in bee pollen using QuEChERS and ultra-high performance liquid chromatography coupled to quadrupole mass spectrometry. Electrophoresis 2016, 37, 2470–2477. [Google Scholar] [CrossRef]
  51. Li, H.; Yin, J.; Liu, Y.; Shang, J. Effect of protein on the detection of fluoroquinolone residues in fish meat. J. Agric. Food Chem. 2012, 60, 1722–1727. [Google Scholar] [CrossRef]
  52. Valverde, S.; Ares, A.M.; Bernal, J.L.; Nozal, M.J.; Bernal, J. Fast determination of neonicotinoid insecticides in beeswax by ultra-high performance liquid chromatography-tandem mass spectrometry using an enhanced matrix removal-lipid sorbent for clean-up. Microchem. J. 2018, 142, 70–77. [Google Scholar] [CrossRef]
Figure 1. Representative chromatograms (SIM mode using the quantification/target ions; see Table 2) obtained from a matrix free standard mixture of 1, atrazine; 2, chlorpyrifos; IS, chlorfenvinphos-d10; 3, chlorfenvinphos; 4, alpha-endosulfan; 5, bromopropylate; 6, coumaphos; 7, tau-fluvalinate. The GC-MS conditions are summarized in Section 2.4 and Table 2.
Figure 1. Representative chromatograms (SIM mode using the quantification/target ions; see Table 2) obtained from a matrix free standard mixture of 1, atrazine; 2, chlorpyrifos; IS, chlorfenvinphos-d10; 3, chlorfenvinphos; 4, alpha-endosulfan; 5, bromopropylate; 6, coumaphos; 7, tau-fluvalinate. The GC-MS conditions are summarized in Section 2.4 and Table 2.
Agronomy 11 00804 g001
Table 1. Calibration curve data, LOD and LOQ values.
Table 1. Calibration curve data, LOD and LOQ values.
CompoundAnalytical Range (µg/kg)SlopeR2LOD (µg/kg)LOQ (µg/kg)
Atrazine5–500038.540.999025
Chlorpyrifos5–500017.950.998325
Chlorfenvinphos5–500028.900.999125
Alpha-Endosulfan10–50004.420.9981310
Bromopropylate10–50003.050.9984310
Coumaphos5–500022.160.998225
Tau-Fluvalinate5–500052.660.998225
LOD, limit of detection; LOQ, limit of quantification; R2, coefficient of determination.
Table 2. GC-MS data.
Table 2. GC-MS data.
CompoundRetention Time (min)Target Ions (m/z)Qualifier Ions (m/z)
Atrazine7.81200215, 173
Chlorpyrifos9.72197315, 258
Chlorfenvinphos10.48267329, 270
Alpha-Endosulfan11.20241195, 207
Bromopropylate13.82345185, 183
Coumaphos15.62362226, 109
Tau-Fluvalinate17.39250207, 181
Chlorfenvinphos-d1010.46333-
Table 3. Recoveries A obtained after testing different solvents with spiked blank beeswax at medium QC (500 µg/kg) with the studied pesticides.
Table 3. Recoveries A obtained after testing different solvents with spiked blank beeswax at medium QC (500 µg/kg) with the studied pesticides.
CompoundAcetonitrile:Water (80:20, v/v)Acetonitrile:Water (70:30, v/v)Acetonitrile1% Acetic Acid in Acetonitrile
Atrazine79 ± 773 ± 786 ± 893 ± 7
Chlorpyrifos78 ± 874 ± 885 ± 892 ± 8
Chlorfenvinphos77 ± 773 ± 684 ± 794 ± 6
Alpha-Endosulfan83 ± 576 ± 690 ± 495 ± 4
Bromopropylate76 ± 766 ± 881 ± 891 ± 8
Coumaphos82 ± 575 ± 491 ± 497 ± 3
Tau-Fluvalinate80 ± 472 ± 582 ± 490 ± 4
A mean ± %RSD (three replicates that were injected in triplicate).
Table 4. Evaluation of the efficiency (recoveries) of the sample treatment and the matrix effect.
Table 4. Evaluation of the efficiency (recoveries) of the sample treatment and the matrix effect.
CompoundEvaluation of the Sample TreatmentEvaluation of the Matrix Effect
Mean (%) ± RSD (%)Mean (%) ± RSD (%)
Low QCMedium QCHigh QCLow QCMedium QCHigh QC
Atrazine102 ± 3106 ± 394 ± 3100 ± 4104 ± 299 ± 4
Chlorpyrifos100 ± 3103 ± 392 ± 494 ± 497 ± 499 ± 4
Chlorfenvinphos104 ± 3107 ± 4105 ± 3102 ± 4105 ± 5100 ± 3
Alpha-Endosulfan93 ± 2102 ± 390 ± 393 ± 4102 ± 399 ± 4
Bromopropylate101 ± 3106 ± 293 ± 3103 ± 3106 ± 496 ± 4
Coumaphos108 ± 2105 ± 5107 ± 5131 ± 5140 ± 5128 ± 5
Tau-Fluvalinate101 ± 4103 ± 4107 ± 4122 ± 5132 ± 4127 ± 5
Low QC, LOQs (see Table 2); Medium QC, 500 µg/kg; High QC, 5000 µg/kg.
Table 5. Results A of the investigation of pesticides in beeswax samples from different origins.
Table 5. Results A of the investigation of pesticides in beeswax samples from different origins.
Beeswax OriginChlorpyrifosChlorfenvinphosCoumaphosTau-Fluvalinate
Organic white beeswax pellets (USA)<LOD<LOD<LOD<LOD
Raw beeswax (Álava, Spain)<LOD<LOD2350207
PBS (Guadalajara, Spain)10116243343
PBS from Perfection hive (Palencia, Spain)<LOD75983173
PBS from Dadant hive (Salamanca, Spain)<LOD832217413
PBS from Layens hive (Salamanca, Spain)<LOD1051743569
DBS from Layens hive (León, Spain)<LOD30279258
LBS (Zamora, Spain)<LOD1131339407
LBS (Álava, Spain)37<LOD527182
LBS (Asturias, Spain)8523<LOD179
LBS (Salamanca, Spain)274931513284
LBS (France)<LOD<LOD<LOD451
LBS (Córdoba, Spain)<LOD<LOD<LOD485
LBS (Guadalajara, Spain)<LOD1351764<LOD
DBS (Germany)52<LOD<LOD<LOD
DBS (France)<LOD<LOD<LOD351
DBS (Portugal)<LOD<LOD<LOD210
DBS (Spain)<LOD<LOD<LOD326
A Mean of triplicate analyses (mg/kg), %RSD < 7% in all cases; PBS, purified beeswax sheet; DBS, decontaminated beeswax sheet; LBS, laminated beeswax sheet.
Table 6. Results A of the investigation of pesticides in beeswax samples from different origins.
Table 6. Results A of the investigation of pesticides in beeswax samples from different origins.
ApiarySampleChlorpyrifosChlorfenvinphosCoumaphosTau-Fluvalinate
VA1Foundation140119967191
Collected Beeswax9496148153
VA2Foundation1691181399386
Collected Beeswax110103263218
VA3Foundation9495139180
Collected Beeswax9395120139
PA1Foundation114111910765
Collected Beeswax9395123142
PA2Foundation<LOD1991701265
Collected Beeswax<LOD87332134
GU1Foundation<LOD<LOD<LOD326
Collected Beeswax<LOD<LOD<LOD147
GU2Foundation681622183221
Collected Beeswax5297402121
A Mean of triplicate analyses (mg/kg), %RSD < 7% in all cases.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Nozal, M.J.; Imaz, E.; Bernal, J.L.; Nieto, J.L.; Higes, M.; Bernal, J. An Optimized Extraction Procedure for Determining Acaricide Residues in Foundation Sheets of Beeswax by Using Gas Chromatography-Mass Spectrometry. Agronomy 2021, 11, 804. https://doi.org/10.3390/agronomy11040804

AMA Style

Nozal MJ, Imaz E, Bernal JL, Nieto JL, Higes M, Bernal J. An Optimized Extraction Procedure for Determining Acaricide Residues in Foundation Sheets of Beeswax by Using Gas Chromatography-Mass Spectrometry. Agronomy. 2021; 11(4):804. https://doi.org/10.3390/agronomy11040804

Chicago/Turabian Style

Nozal, María Jesús, Edgar Imaz, José Luis Bernal, José Luis Nieto, Mariano Higes, and José Bernal. 2021. "An Optimized Extraction Procedure for Determining Acaricide Residues in Foundation Sheets of Beeswax by Using Gas Chromatography-Mass Spectrometry" Agronomy 11, no. 4: 804. https://doi.org/10.3390/agronomy11040804

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop