Next Article in Journal
Educational Programme for Cancer Nurses in Genetics, Health Behaviors and Cancer Prevention: A Multidisciplinary Consensus Study
Next Article in Special Issue
The Caliber of Segmental and Subsegmental Vessels in COVID-19 Pneumonia Is Enlarged: A Distinctive Feature in Comparison with Other Forms of Inflammatory and Thromboembolic Diseases
Previous Article in Journal
Patterns and Characteristics of SKYLINE-Lumipoint Feature in the Catheter Ablation of Atypical Atrial Flutter: Insight from a Novel Lumipoint Module of Rhythmia Mapping System
Previous Article in Special Issue
EASIX, Modified EASIX and Simplified EASIX as an Early Predictor for Intensive Care Unit Admission and Mortality in Severe COVID-19 Patients
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JPM
Volume 12
Issue 7
10.3390/jpm12071103
jpm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Lack of Difference in Tocilizumab Efficacy in the Treatment of Severe COVID-19 Caused by Different SARS-CoV-2 Variants

by
Oleksandr Oliynyk
1,2,
Wojciech Barg
3,
Yanina Oliynyk
4,
Serhij Dubrov
1,
Vitaliy Gurianov
5 and
Marta Rorat
6,*
1
Department of Anaesthesiology and Intensive Care, Bogomolets National Medical University, 01601 Kyiv, Ukraine
2
Department of Emergency Medicine, Medical College of Rzeszow University, 35-959 Rzeszów, Poland
3
Department of Human Physiology, Medical College of Rzeszow University, 35-959 Rzeszów, Poland
4
Department of Immunology and Allergology, Bogomolets National Medical University, 01601 Kyiv, Ukraine
5
Department of Medical Statistics, Bogomolets National Medical University, 01601 Kyiv, Ukraine
6
Department of Forensic Medicine, Wroclaw Medical University, 50-367 Wroclaw, Poland
*
Author to whom correspondence should be addressed.
J. Pers. Med. 2022, 12(7), 1103; https://doi.org/10.3390/jpm12071103
Submission received: 5 June 2022 / Revised: 30 June 2022 / Accepted: 2 July 2022 / Published: 4 July 2022
(This article belongs to the Special Issue Clinical Presentation, Treatment and Vaccination in COVID-19)
Browse Figures
Versions Notes

Abstract

:
Tocilizumab (TOC) is presumed to be an effective and safe treatment for severe COVID-19, but its usefulness has not been yet investigated for different SARS-CoV-2 variants. This study aimed to evaluate the influence of TOC on mortality in patients with severe COVID-19 caused by Delta and non-Delta SARS-CoV-2 variants. In a retrospective analysis, we compared the medical records of 78 and 224 patients with severe COVID-19 due to Delta and non-Delta variants, respectively. A total of 30 patients with Delta and 84 with non-Delta variants were treated with TOC in addition to standard therapy. There were no statistically significant differences in mortality rate when comparing Delta vs. non-Delta patients nor when comparing those treated with TOC vs. not treated with TOC in both variants. Using a logistic regression model, in the examined population as a whole, we found an increased (p < 0.05) risk of death as leukocyte and erythrocyte counts decreased and as procalcitonin increased. Increased procalcitonin was significant for mortality in the Delta group, while decreased IL-6, leukocytes, and platelets and increased fibrinogen and procalcitonin were significant in the non-Delta group. Tocilizumab efficacy in severe COVID-19 does not differ between Delta or non-Delta virus variants. The Delta variant of SARS-CoV-2 does not increase mortality when compared to other virus strains.

1. Introduction

The coronavirus disease 2019 (COVID-19) pandemic caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, resulting in over 350 million cases and over 5.5 million deaths by the end of 2021, is a challenge for healthcare systems all over the world [1]. The SARS-CoV-2 infection is biphasic, with an initial phase of virus replication and only mild clinical symptoms, and an inflammatory phase usually with interstitial pneumonia with or without hypoxemia [2]. The majority of patients remain oligosymptomatic or even asymptomatic. However, some of those with severe symptoms develop a life-threatening course of COVID-19 associated with systemic hyperinflammation. The biomarkers of this hyperinflammation are clearly elevated concentrations of inflammatory markers such as D-dimer, ferritin, and C-reactive protein (CRP) and a cytokine storm—an excessive release of pro-inflammatory cytokines, including interleukin-1, interleukin-6 (IL-6), and tumour necrosis factor α [3,4]. It was demonstrated that within the cytokine storm, severe COVID-19 progresses with a significant rise in IL-6 levels [5]. Consequently, anti-Il-6 medications were proposed for the treatment of severe COVID-19, and tocilizumab (TOC) is the one most often used.
Tocilizumab, an IL-6 monoclonal antibody, initially used in the treatment of autoimmune inflammatory diseases, such as rheumatoid arthritis, has been approved in the USA for cytokine release syndrome since 2017 [6]. It has started to be used intensively in the treatment of severe COVID-19. Initial reports on the effectiveness of this therapy in COVID-19 varied significantly, ranging from an evident reduction in the need for mechanical ventilation and death rate, through lack of effect on the course of the disease, to evidence of therapy-related harm. Although, according to most data currently available, TOC is presumed to be beneficial in severe COVID-19, there are no clear indications for its use and no clearly defined biomarkers for the indication of treatment with TOC or precisely identified patients that are most likely to benefit from it [7]. In the very recently updated Infectious Diseases Society of America Guidelines, it is suggested that TOC is added to the standard of care (i.e., steroids), but this recommendation is labelled as conditional, with low certainty of evidence [8].
There is also no data regarding TOC safety and efficacy with respect to SARS-CoV-2 variants. This might be an emerging issue as mutations in the virus could potentially trigger a different inflammatory response. This in turn may alter sensitivity to the treatment used. In the studies conducted so far, the effectiveness of treatment with TOC has not been assessed with regard to the virus variant, which may be of significant clinical importance.
The objective of this research was to assess TOC safety and efficacy in patients with severe COVID-19 and hyperinflammatory syndrome resulting from infection with different SARS-CoV-2 variants.

2. Materials and Methods

In a retrospective study, the medical records of 983 consecutive patients with severe COVID-19 hospitalised from 1 August 2020 to 30 October 2021 in the Department of Anaesthesiology and Intensive Care at Kyiv City Clinical Hospital № 4 were examined. Of them, 302 were selected for further analysis.
Inclusion criteria comprised:
  • laboratory-confirmed SARS-CoV-2 infection (positive reverse transcription-polymerase chain reaction, RT-PCR),
  • presence of interstitial pneumonia on a computed tomography (CT) scan,
  • respiratory failure with oxygenation index (PaO2/FiO2 ratio) <300 mm Hg,
  • hyperinflammatory syndrome—the presence of at least two out of three biomarkers for hyperinflammation (increased C-reactive protein >100 mg/L, ferritin >900 µg/L, D-dimer >1500 µg/L) [9].
Exclusion criteria comprised:
  • symptoms of bacterial sepsis defined as leukocyte count (WBC) >10.0 × 109/L and/or procalcitonin (PCT) >2.0 ng/mL and/or positive blood culture,
  • cardiogenic pulmonary oedema,
  • acute brain stroke,
  • advance chronic pulmonary disease,
  • malignancies or any decompensated chronic diseases,
  • pregnancy,
  • participation in other interventional studies.
Patients were assigned to two groups: those with Delta SARS-CoV-2 variant infections (n = 78) and those with other SARS-CoV-2 variants (n = 224). Both groups were then divided into those who received tocilizumab (30 and 84 patients, respectively) and those who did not (48 and 140, respectively).
Since 1 June 2021, all hospitalised patients were tested for the mutation specific to the SARS-CoV-2 Delta variant. The mutation, characterised as L452R/E484Q/P68IR/E484K, was detected using a Nuclear Acid Extraction-Purification Kit from Sansure Biotech (Sansure Biotech Inc., No 680, Lusong Road, Zuelu District, 410205 Changsha, Hunan Province, China). The determination of the mutation was carried out according to the relevant instructions: Doc.#: S10016E Manual, Doc. Version: V00. Revision Date: 26 August 2020. Patients hospitalized before 1 June 2021 were assigned to the non-Delta group. This criterion was selected based on the analysis of the reports published by the Public Health Center of the Ministry of Health of Ukraine, which conducts epidemiological surveillance in Ukraine by regular molecular testing samples taken from patients from various regions of the country.
Treatment with tocilizumab was started between day 8 and day 14 from the onset of symptoms. Tocilizumab was administered intravenously at a dose of 400 mg for two consecutive days. All patients were treated with methylprednisolone 32 mg or dexamethasone 6 mg daily (ultimately for 10–14 days from admission), low molecular weight heparin administered subcutaneously in prophylactic doses, antibiotics (in case of suspected or confirmed bacterial infection), and balanced fluid therapy. Mortality was assessed over a 28-day period from admission to the intensive care unit (ICU).

Statistical Analysis

MedCalc® Statistical Software version 2014 (MedCalc Software Ltd., Ostend, Belgium; https://www.medcalc.org; accessed on 20 December 2021) was used for the analysis. If the distribution of values differed from normal by the Shapiro–Wilk test, the median (Me) value and interquartile range (QI-QIII) were calculated to characterise the quantitative measures. The Mann–Whitney test was used for comparison between the two groups. Frequency (%) was calculated for qualitative measures and Fisher’s exact test was used to compare the two groups. The logistic regression analysis with a stepwise method was used to quantify the impact of examined variables on mortality risk. The prognostic quality of the models was assessed using a receiver operating characteristic curve (ROC-curve), the area under the ROC-curve (AUC), and its 95% CI were calculated. The impact of the examined variables was assessed by the odds ratio (OR) values, for which a 95% CI was calculated. A two-tailed test was used in the analysis. For all statistical tests, the p value < 0.05 was considered significant.

3. Results

Demographic and medical data for the groups studied is shown in Table 1.
There were statistically significant differences between the groups for age, erythrocyte and lymphocyte counts, and respiratory index.
Sixteen factors were presumed to potentially influence the risk of death in coronavirus variants: treatment with TOC, age, sex, body temperature, platelet, leucocyte, lymphocyte, and red blood cell counts, interleukin-6, procalcitonin, ferritin, fibrinogen, C-reactive protein, D-dimer concentration, and respiratory index PaO2/FiO2. In a multifactor logistic regression model using a stepwise method (entering variable if p < 0.05, removing variable if p > 0.1), three variables were found to influence the risk of death: erythrocytes, leukocytes, and procalcitonin. There was an increased (p < 0.05) risk of death when leukocyte and erythrocyte counts decreased and when procalcitonin levels increased. It is worth noting that neither treatment with TOC nor the virus variant influenced the risk of death. Table 2 presents the results of the multifactor logistic regression analysis.
As demonstrated in Figure 1 using the ROC curve, the model based on the selected variables is adequate with AUC = 0.928 (95% CI 0.89–0.95). This indicates a very strong relationship between the risk of death and the selected variables.
To estimate the influence of treatment with TOC on mortality in the Delta and non-Delta groups, we first compared basic demographic and medical data with respect to TOC treatment in those groups (Table 3, respectively).
As shown in Table 4, there were no statistically significant differences in the number of deaths in the Delta group and the non-Delta group with respect to treatment with TOC.
To determine the set of factors associated with the risk of death separately for the Delta and the non-Delta groups, again multifactor logistic regression analyses were performed using the same set of factors potentially influencing the risk of death as the one used in the previous analysis. Using the stepwise method (entering variable if p < 0.05, removing variable if p > 0.1), only increased procalcitonin was found to be significant for the Delta group, while five variables—decreased IL-6, leukocytes, and platelets, as well as increased fibrinogen and procalcitonin—were found to be significant for the non-Delta group. Table 5 and Table 6 demonstrate the results of those analyses.
The models based on the selected features are adequate. As demonstrated in Figure 2, the AUCs are 0.86 (95% CI 0.76–0.93) and 0.96 (95% CI 0.93 –0.98) for the Delta and non-Delta groups, respectively. This indicates a very strong relationship between the risk of death and the selected variables in both analyses.

4. Discussion

To the best of our knowledge, this is the first study on the efficacy of treatment with tocilizumab in severe COVID-19 focusing on the virus variants. In our retrospective study, we did not find any statistically significant differences in mortality between patients with Delta or non-Delta variants when treated with TOC.
Assessing emerging SARS-CoV-2 mutations for the clinical severity of COVID-19 provides important information for characterising disease burden and risk to the population. The genetic diversity of the virus may affect transmissibility, virulence, and mortality, as well as the efficacy of available therapies [10]. Among the many variants of the SARS-CoV-2 virus to date, five have been recognised by the WHO as variants of concern (VOCs) of public health importance: Alpha (B.1.1.7), Beta (B.1.351), Gamma (P.1), Delta (B.1.617.2), and Omicron (B.1.1.529) [11]. The Delta variant, first detected in India in October 2020, has spread rapidly worldwide and was responsible for the deadly second wave [12]. In Ukraine, the first cases of Delta variant infection were reported in late June 2021 [13]. It was reported that the neutralising activity of vaccine- and plasma-induced neutralising antibodies of convalescents against the Delta variant was significantly reduced, which affects the risk of reinfection [14,15]. Studies on the Delta variant revealed the accumulation of nine amino acid mutations (T19R, G142D, FR156⁃157del, R158G, L452R, T478K, D614G, P681R, D950N) in the spike protein [16].
Data on the course of Delta variant infection is inconclusive. It was found that the incubation period and generation time were shorter, and the basic reproductive number was higher than that of the wild-type strain. In addition, the viral loads of the Delta variant were significantly higher. Thus, the virus showed higher transmissibility [17]. In a Scottish study, the risk of hospital admission was approximately double in the Delta compared to the Alpha variant, especially in people with multimorbidity [18]. In the study by Fisman et al. [19], the risk of hospitalisation, ICU admission, and death relative to non-VOC infections was 108%, 235%, and 133%, respectively. Lin et al. [20] showed a higher overall risk of hospitalisation rather in the Beta than the Delta variant if compared to the wild-type virus (2.16 versus 2.08, respectively). In turn, severity and mortality rates were higher in the Delta group (3.35 and 2.33 versus 2.23 and 1.50, respectively) [20]. In contrast, studies by Gunadi et al., and Tani-Sassa et al., found no significant differences in hospitalisation and mortality rates between patients with Delta and non-Delta variants [21,22].
This was in line with our results. There were no differences in mortality (44.9% versus 44.6%, p > 0.999), despite a significant difference in the severity of respiratory failure (PaO2/FiO2 ratio 111 versus 139 mmHg, p < 0.001), as well as lower lymphocyte and erythrocyte counts and a more advanced age in Delta patients (see Table 1). In a multifactor logistic regression model, three variables were found to influence mortality: erythrocyte and leukocyte counts, and procalcitonin (Table 2). There was an increased risk of death with decreasing leukocyte and erythrocyte counts and with increasing procalcitonin concentration. The analysis showed a very strong relationship between the risk of death and the selected variables with AUC = 0.93, sensitivity = 81.5%, and specificity = 94% (Figure 1).
The primary issue investigated in our paper was the impact of treatment with TOC on mortality in ICU patients with Delta or non-Delta variants. In the literature, we did not find any data on the efficacy of TOC when used for the different SARS-CoV-2 variants. In our study, no statistically significant differences in mortality were found in the Delta and non-Delta groups depending on TOC treatment (Table 4). The stepwise method found that only procalcitonin significantly influenced mortality in the Delta group, while five variables—procalcitonin along with IL-6, fibrinogen, leukocytes, and platelets—were significant for the non-Delta group (Table 5 and Table 6, Figure 2).
Initial clinical studies on the efficacy of TOC in the treatment of COVID-19 have shown inconclusive results. In a cohort study conducted at the University of Michigan, mortality among patients on artificial ventilation almost halved if treated with TOC, despite an increase in infectious complications [23]. A study from three Italian centers compared 179 patients who received TOC and 365 patients on conventional therapy. The use of TOC was associated with a lower risk of artificial ventilation or death [24]. Similarly, in a study by Capra et al., treatment with TOC was associated with improved survival and a favorable clinical course compared with standard treatment [25]. In turn, in a study by Flisiak et al., a significant effect of TOC on death was demonstrated in patients with baseline concentration of IL-6 > 100 pg/mL (OR 0.27, 95% CI 0.10–0.78, p = 0.02), particularly if oxygen saturation was lower than 90% [26].
Some other studies presented opposite conclusions. Campochiaro et al., compared clinical outcomes in 32 patients who received TOC and in 33 controls. Patients who received TOC had a lower mortality rate, but the differences were not statistically significant [27]. Canziani et al., demonstrated that the use of TOC was associated with a decreased probability of artificial ventilation (OR 0.36, 95% CI 0.16–0.83; p = 0.017), but with a higher risk of thrombosis, bleeding, or infection [28]. In addition, TOC did not affect 30-day mortality. In the COVACTA trial, the use of TOC in patients with severe COVID-19 pneumonia did not result in improved clinical status or lower mortality than the placebo at 28 days [29]. Similarly, Stone et al., conducted a randomized, double-blind, placebo-controlled trial involving 243 patients and the authors found no beneficial effect of TOC in preventing intubation or death in moderately ill hospitalized patients with COVID-19 [30].
In contrast, the results of meta-analyses published so far demonstrated the beneficial effects of TOC in the treatment of patients with severe COVID-19. Wei et al. [31] assessed 25 publications with RevMan 5.3. Significantly better clinical outcomes were found in the TOC group when compared to the standard care group with OR (odds ratio) = 0.70, 95% confidential interval (CI) 0.54–0.90, p = 0.007. Treatment with TOC showed a stronger correlation with a good prognosis among COVID-19 patients that needed mechanical ventilation (OR = 0.59, 95% CI 0.37–0.93, p = 0.02). In stratified analyses, overall mortality reduction correlated with TOC treatment in patients less than 65 years old (OR  =  0.68, 95% CI 0.60–0.77, p  <  0.00001), as well as in ICU patients (OR = 0.62, 95% CI 0.55–0.70, p  < 0.00001). In general, TOC treatment predicted better overall survival in COVID-19 patients (HR  =  0.45, 95% CI 0.24–0.84, p =  0.01), especially in severe cases (HR = 0.58, 95% CI 0.49–0.68, p < 0.00001) [31]. Maraolo et al., considered TOC to be one of the few agents that may improve the prognosis of severe COVID-19 [32]. In addition, the results of the meta-analysis by Vela et al., covering 10 RCTs and 6837 patients demonstrated a beneficial effect of TOC in patients with severe COVID-19, hyperinflammatory syndrome, and severe hypoxia. The pooled risk ratio (RR) for all-cause mortality in patients with TOC was RR = 0.88 (95% CI 0.81–0.95, p = 0.0009), RR for mechanical ventilation at 28–30 days was 0.79 (95% CI 0.71–0.88), and RR for serious adverse events (SAE) was 0.91, (95% CI 0.76–1.09). The reduction in mortality was most pronounced in patients who received systemic corticosteroids [33]. Data from a meta-analysis by Kyriakopoulos et al., supports recommendations for using TOC in combination with corticosteroids in patients with severe COVID-19 [34]. Tocilizumab is effective when hyperinflammatory rather than infectious mechanisms dominate the pathogenesis of the disease. The RECOVERY trial has shown that treatment with TOC in patients with severe COVID-19, hypoxemia, and hyperinflammatory syndrome reduced mortality, increased the likelihood of successful discharge from hospital, and reduced the need for invasive mechanical ventilation. These benefits are in addition to the ones provided by systemic corticosteroids [35].
A meta-analysis by Rubio-Rivas et al., comprising 64 studies covering 7668 patients with COVID-19 treated with TOC, indicated hospital-wide pooled mortality of OR 0.73 (95% CI = 0.56–0.93) but with a significant difference in efficacy depending on the severity of the disease. For patients admitted to conventional wards vs. ICU, OR were 1.25 (95% CI 0.74–2.18) and 0.66 (95% CI 0.59–0.76), respectively [36]. Again, this advantage was better seen in patients receiving concomitant corticosteroids. Another assessment covering 13 studies and 2120 patients revealed that TOC treatment resulted in lower mortality (OR = 0.42, 95% CI 0.26) as compared to standard therapy (OR = 0.69, p = 0.0005, I2 = 55%) [37]. Resaei et al., presented the results of 45 comparative studies involving 13,189 patients and 28 single-arm studies involving 1770 patients. Mortality (RR of 0.76 (95% CI 0.65–0.89), p < 0.01) and intubation risks (RR of 0.48 (95% CI 0.24–0.97), p = 0.04) were lower in TOC patients compared to controls, although there were no significant differences in secondary infections, length of hospital stay, hospital discharge before day 14, and ICU admission between groups [38]. A very interesting meta-analysis focused on optimal TOC use in severe and critically ill COVID-19 patients was published by Nugroho et al. [39] The authors demonstrated that TOC reduced all-cause mortality when CRP levels were ≥100 mg/L, and PaO2/FiO2 ratios were 200–300 mmHg or <200 mmHg. It is worth noting that subgroup analysis showed that TOC administration in patients with CRP levels <100 mg/L reduced neither mortality nor length of hospitalisation. Additionally, a very recently published meta-analysis involving 53 papers with 21,656 patients demonstrated a beneficial effect of TOC on mortality and intubation, but also no increase in the risk of secondary infections [40].
We cannot unequivocally justify why TOC treatment did not affect mortality in either the Delta or non-Delta groups. This may support the opinion that the use of TOC does not affect the course or mortality of COVID-19, and a possible beneficial effect can only be noted in more severe cases of the disease [31,36,39]. In our study, patients treated vs. not treated with TOC differed slightly, although those differences were similar in the Delta vs. non-Delta groups. Those treated with TOC in the Delta group had higher D-dimer and ferritin levels and lower erythrocyte and lymphocyte counts while patients with the non-Delta virus variant had higher D-dimer, fibrinogen, and IL-6 levels and lower erythrocyte and lymphocyte counts (Table 3). This may indicate a more severe course of COVID-19 in the TOC-treated groups, but it seems doubtful that these differences are clinically relevant.
Another possible explanation may be COVID-19 heterogeneity. In our previous study, TOC contributed to reduced mortality in patients with macrophage activation syndrome [41]. In contrast, in those with cytokine release syndrome, TOC even increased mortality. The small size of the groups studied did not allow for such a subanalysis. Thus, we believe that TOC efficacy depends on COVID-19 severity and the cytokine storm subtype rather than the virus variant.
It should be also noted that Delta and non-Delta patients were not hospitalised at the same time. Thus, it cannot be excluded that the final outcome could be influenced by the centres experience with COVID-19 treatment, availability of beds, or faster qualification for ICU treatment.
Our study has limitations. Firstly, it is a retrospective, single-centre study. Secondly, a low number of patients in the study groups significantly limits the power of this study and its conclusions. Finally, the study concerned a specific virus variant that is no longer dominant, thus there is a question of whether the results obtained can be extrapolated to other SARS-CoV-2 variants.

5. Conclusions

Tocilizumab did not improve survival in patients with both Delta and non-Delta variants. Despite a more severe clinical course, the Delta variant did not increase mortality in COVID-19.

Author Contributions

Conceptualisation, O.O.; methodology, O.O., V.G., W.B., and M.R.; software V.G.; validation, W.B. and M.R.; formal analysis, O.O., V.G., W.B., and M.R.; investigation, Y.O. and S.D.; resources, Y.O. and S.D.; data curation, O.O.; writing—original draft preparation, O.O.; writing—review and editing, O.O., W.B., and M.R.; visualization, V.G. and M.R.; supervision, M.R.; funding acquisition, O.O. and M.R. All authors have read and agreed to the published version of the manuscript.

Funding

This study is part of the clinical research on the topic “Optimisation of respiratory support methods for patients with severe forms of respiratory insufficiency, including acute respiratory distress syndrome” conducted in the Department of Anaesthesiology and Intensive Care of KNMU and registered in the Clinical Trials Register of the State Expert Centre of the Ministry of Health of Ukraine (Registration number 0112U001413). This research was also carried out according to the subject register in Simple system SUBZ.A120.22.077 and supported by a statutory subsidy granted by the Ministry of Science and Higher Education in Poland. Financial support was also obtained from the University of Rzeszów.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Bioethics Committee of the Kyiv City Clinical Hospital No. 4. (Decision no. 64 dated 2 July 2020).

Informed Consent Statement

All data was fully anonymised before access, and the local ethics committee waived the requirement to obtain informed consent to process data from patients’ medical records used in the research.

Data Availability Statement

All relevant data is provided in the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. COVID-19 Data Explorer. Available online: https://ourworldindata.org/explorers/coronavirus-data-explorer (accessed on 25 December 2021).
  2. Siddiqi, H.K.; Mehra, M.R. COVID-19 illness in native and immunosuppressed states: A clinical-therapeutic staging proposal. J. Heart Lung Transplant. 2020, 39, 405–407. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Copaescu, A.; Smibert, O.; Gibson, A.; Phillips, E.J.; Trubiano, J.A. The role of IL-6 and other mediators in the cytokine storm associated with SARS-CoV-2 infection. J. Allergy Clin. Immunol. 2020, 146, 518–534.e1. [Google Scholar] [CrossRef] [PubMed]
  4. Chukwuma, I.F.; Apeh, V.O.; Chiletugo, O.F. Mechanisms and potential therapeutic targets of hyperinflammatory responses in SARS-CoV-2. Acta Virol. 2021, 65, 3–9. [Google Scholar] [CrossRef] [PubMed]
  5. Coomes, E.A.; Haghbayan, H. Interleukin-6 in COVID-19: A systematic review and meta-analysis. Rev. Med. Virol. 2020, 30, 1–9. [Google Scholar] [CrossRef] [PubMed]
  6. FDA Approves Tisagenlecleucel for B-cell ALL and Tocilizumab for Cytokine Release Syndrome. Available online: https://www.fda.gov/drugs/resources-information-approved-drugs/fda-approves-tisagenlecleucel-b-cell-all-and-tocilizumab-cytokine-release-syndrome?platform=hootsuite (accessed on 10 January 2022).
  7. Campochiaro, C.; Tomelleri, A.; Matucci-Cerinic, M.; Dagna, L. One year later: The case of tocilizumab in COVID-19. Eur. J. Intern. Med. 2022, 95, 5–6. [Google Scholar] [CrossRef] [PubMed]
  8. IDSA Guidelines on the Treatment and Management of Patients with COVID-19. Available online: https://www.idsociety.org/practice-guideline/covid-19-guideline-treatment-and-management (accessed on 10 January 2022).
  9. Ramiro, S.; Mostard, R.; Magro-Checa, C. Historically controlled comparison of glucocorticoids with or without tocilizumab versus supportive care only in patients with COVID-19-associated cytokine storm syndrome: Results of the CHIC study. Ann. Rheum. Dis. 2020, 79, 1143–1151. [Google Scholar] [CrossRef]
  10. SeyedAlinaghi, S.; Mirzapour, P.; Dadras, O.; Pashaei, Z.; Karimi, A.; MohsseniPour, M.; Soleymanzadeh, M.; Barzegary, A.; Afsahi, A.M.; Vahedi, F.; et al. Characterization of SARS-CoV-2 different variants and related morbidity and mortality: A systematic review. Eur. J. Med. Res. 2021, 26, 51. [Google Scholar] [CrossRef]
  11. Aleem, A.; Akbar Samad, A.B.; Slenker, A.K. Emerging Variants of SARS-CoV-2 And Novel Therapeutics against Coronavirus (COVID-19); StatPearls Publishing: Treasure Island, FL, USA, 2022. [Google Scholar]
  12. Ito, K.; Piantham, C.; Nishiura, H. Predicted dominance of variant Delta of SARS-CoV-2 before Tokyo Olympic Games, Japan, July 2021. Euro Surveill. 2021, 26, 2100570. [Google Scholar] [CrossRef]
  13. Ukraine Registers First Cases of COVID-19 Delta Variant. Available online: https://www.reuters.com/business/healthcare-pharmaceuticals/ukraine-registers-first-cases-covid-19-delta-variant-2021-06-23/ (accessed on 15 April 2022).
  14. Liu, C.; Ginn, H.M.; Dejnirattisai, W.; Supasa, P.; Wang, B.; Tuekprakhon, A.; Nutalai, R.; Zhou, D.; Mentzer, A.J.; Zhao, Y.; et al. Reduced neutralization of SARS-CoV-2 B.1.617 by vaccine and convalescent serum. Cell 2021, 184, 4220–4236.e13. [Google Scholar] [CrossRef]
  15. Edara, V.V.; Pinsky, B.A.; Suthar, M.S.; Lai, L.; Davis-Gardner, M.E.; Floyd, K.; Flowers, M.W.; Wrammert, J.; Hussaini, L.; Ciric, C.R.; et al. Infection and Vaccine-Induced Neutralizing-Antibody Responses to the SARS-CoV-2 B.1.617 Variants. N. Engl. J. Med. 2021, 385, 664–666. [Google Scholar] [CrossRef]
  16. Harvey, W.T.; Carabelli, A.M.; Jackson, B.; Gupta, R.K.; Thomson, E.C.; Harrison, E.M.; Ludden, C.; Reeve, R.; Rambaut, A.; COVID-19 Genomics UK (COG-UK) Consortium; et al. SARS-CoV-2 variants, spike mutations and immune escape. Nat. Rev. Microbiol. 2021, 19, 409–424. [Google Scholar] [CrossRef] [PubMed]
  17. Tian, D.; Sun, Y.; Zhou, J.; Ye, Q. The Global Epidemic of the SARS-CoV-2 Delta Variant, Key Spike Mutations and Immune Escape. Front. Immunol. 2021, 12, 751778. [Google Scholar] [CrossRef] [PubMed]
  18. Sheikh, A.; McMenamin, J.; Taylor, B.; Robertson, C.; Public Health Scotland and the EAVE II Collaborators. SARS-CoV-2 Delta VOC in Scotland: Demographics, risk of hospital admission, and vaccine effectiveness. Lancet 2021, 397, 2461–2462. [Google Scholar] [CrossRef]
  19. Fisman, D.N.; Tuite, A.R. Evaluation of the relative virulence of novel SARS-CoV-2 variants: A retrospective cohort study in Ontario, Canada. CMAJ 2021, 193, E1619–E1625. [Google Scholar] [CrossRef]
  20. Lin, L.; Liu, Y.; Tang, X.; He, D. The Disease Severity and Clinical Outcomes of the SARS-CoV-2 Variants of Concern. Front. Public Health 2021, 9, 775224. [Google Scholar] [CrossRef]
  21. Hakim, M.S.; Wibawa, H.; Setiawaty, V.; Trisnawati, I.; Supriyati, E.; El Khair, R.; Iskandar, K.; Anggorowati, N.; Daniwijaya, E.W.; Nugrahaningsih, D.A.A.; et al. Is the Infection of the SARS-CoV-2 Delta Variant Associated with the Outcomes of COVID-19 Patients? Front. Med. 2021, 8, 780611. [Google Scholar]
  22. Tani-Sassa, C.; Iwasaki, Y.; Ichimura, N.; Nagano, K.; Takatsuki, Y.; Yuasa, S.; Takahashi, Y.; Nakajima, J.; Sonobe, K.; Nukui, Y.; et al. Viral loads and profile of the patients infected with SARS-CoV-2 Delta, Alpha, or R.1 variants in Tokyo. J. Med. Virol. 2022, 94, 1707–1710. [Google Scholar] [CrossRef]
  23. Somers, E.C.; Eschenauer, G.A.; Troost, J.P.; Golob, J.L.; Gandhi, T.N.; Wang, L.; Zhou, N.; Petty, L.A.; Baang, J.H.; Dillman, N.O.; et al. Tocilizumab for Treatment of Mechanically Ventilated Patients With COVID-19. Clin. Infect. Dis. 2021, 73, e445–e454. [Google Scholar] [CrossRef]
  24. Guaraldi, G.; Meschiari, M.; Cozzi-Lepri, A.; Milic, J.; Tonelli, R.; Menozzi, M.; Franceschini, E.; Cuomo, G.; Orlando, G.; Borghi, V.; et al. Tocilizumab in patients with severe COVID-19: A retrospective cohort study. Lancet Rheumatol. 2020, 2, e474–e484. [Google Scholar] [CrossRef]
  25. Capra, R.; De Rossi, N.; Mattioli, F.; Romanelli, G.; Scarpazza, C.; Sormani, M.P.; Cossi, S. Impact of low dose tocilizumab on mortality rate in patients with COVID-19 related pneumonia. Eur. J. Intern. Med. 2020, 76, 31–35. [Google Scholar] [CrossRef]
  26. Flisiak, R.; Jaroszewicz, J.; Rogalska, M.; Łapiński, T.; Berkan-Kawińska, A.; Bolewska, B.; Tudrujek-Zdunek, M.; Kozielewicz, D.; Rorat, M.; Leszczyński, P.; et al. Tocilizumab Improves the Prognosis of COVID-19 in Patients with High IL-6. J. Clin. Med. 2021, 10, 1583. [Google Scholar] [CrossRef] [PubMed]
  27. Campochiaro, C.; Della-Torre, E.; Cavalli, G.; De Luca, G.; Ripa, M.; Boffini, N.; Tomelleri, A.; Baldissera, E.; Rovere-Querini, P.; Ruggeri, A.; et al. Efficacy and safety of tocilizumab in severe COVID-19 patients: A single-centre retrospective cohort study. Eur. J. Intern. Med. 2020, 76, 43–49. [Google Scholar] [CrossRef] [PubMed]
  28. Canziani, L.M.; Trovati, S.; Brunetta, E.; Testa, A.; De Santis, M.; Bombardieri, E.; Guidelli, G.; Albano, G.; Folci, M.; Squadroni, M.; et al. Interleukin-6 receptor blocking with intravenous tocilizumab in COVID-19 severe acute respiratory distress syndrome: A retrospective case-control survival analysis of 128 patients. J. Autoimmun. 2020, 114, 102511. [Google Scholar] [CrossRef] [PubMed]
  29. Rosas, I.O.; Bräu, N.; Waters, M.; Go, R.C.; Hunter, B.D.; Bhagani, S.; Skiest, D.; Aziz, M.S.; Cooper, N.; Douglas, I.S.; et al. Tocilizumab in Hospitalized Patients with Severe Covid-19 Pneumonia. N. Engl. J. Med. 2021, 384, 1503–1516. [Google Scholar] [CrossRef]
  30. Stone, J.H.; Frigault, M.J.; Serling-Boyd, N.J.; Fernandes, A.D.; Harvey, L.; Foulkes, A.S.; Horick, N.K.; Healy, B.C.; Shah, R.; Bensaci, A.M.; et al. Efficacy of Tocilizumab in Patients Hospitalized with Covid-19. N. Engl. J. Med. 2020, 383, 2333–2344. [Google Scholar] [CrossRef]
  31. Wei, Q.; Lin, H.; Wei, R.G.; Chen, N.; He, F.; Zou, D.H.; Wei, J.R. Tocilizumab treatment for COVID-19 patients: A systematic review and meta-analysis. Infect. Dis. Poverty 2021, 10, 71. [Google Scholar] [CrossRef]
  32. Maraolo, A.E.; Crispo, A.; Piezzo, M.; Di Gennaro, P.; Vitale, M.G.; Mallardo, D.; Ametrano, L.; Celentano, E.; Cuomo, A.; Ascierto, P.A.; et al. The Use of Tocilizumab in Patients with COVID-19: A Systematic Review, Meta-Analysis and Trial Sequential Analysis of Randomized Controlled Studies. J. Clin. Med. 2021, 10, 4935. [Google Scholar] [CrossRef]
  33. Vela, D.; Vela-Gaxha, Z.; Rexhepi, M.; Olloni, R.; Hyseni, V.; Nallbani, R. Efficacy and safety of tocilizumab versus standard care/placebo in patients with COVID-19; a systematic review and meta-analysis of randomized clinical trials. Br. J. Clin. Pharmacol. 2022, 88, 1955–1963. [Google Scholar] [CrossRef]
  34. Kyriakopoulos, C.; Ntritsos, G.; Gogali, A.; Milionis, H.; Evangelou, E.; Kostikas, K. Tocilizumab administration for the treatment of hospitalized patients with COVID-19: A systematic review and meta-analysis. Respirology 2021, 26, 1027–1040. [Google Scholar] [CrossRef]
  35. RECOVERY Collaborative Group. Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): A randomised, controlled, open-label, platform trial. Lancet 2021, 397, 1637–1645. [Google Scholar] [CrossRef]
  36. Rubio-Rivas, M.; Forero, C.G.; Mora-Luján, J.M.; Montero, A.; Formiga, F.; Homs, N.A.; Albà-Albalate, J.; Sánchez, L.; Rello, J.; Corbella, X. Beneficial and harmful outcomes of tocilizumab in severe COVID-19: A systematic review and meta-analysis. Pharmacotherapy 2021, 41, 884–906. [Google Scholar] [CrossRef] [PubMed]
  37. Sarfraz, A.; Sarfraz, Z.; Sarfraz, M.; Aftab, H.; Pervaiz, Z. Tocilizumab and COVID-19: A meta-analysis of 2120 patients with severe disease and implications for clinical trial methodologies. Turk. J. Med. Sci. 2021, 51, 890–897. [Google Scholar] [CrossRef] [PubMed]
  38. Rezaei, S.; Fatemi, B.; Karimi Majd, Z.; Minaei, H.; Peikanpour, M.; Anjidani, N.; Taheri, A.; Dastan, F.; Mosaed, R. Efficacy and safety of Tocilizumab in severe and critical COVID-19: A Systematic Review and Meta-Analysis. Expert. Rev. Clin. Immunol. 2021, 17, 499–511. [Google Scholar] [CrossRef] [PubMed]
  39. Nugroho, C.W.; Suryantoro, S.D.; Yuliasih, Y.; Rosyid, A.N.; Asmarawati, T.P.; Andrianto, L.; Setiawan, H.W.; Mahdi, B.A.; Windradi, C.; Agustin, E.D.; et al. Optimal use of tocilizumab for severe and critical COVID-19: A systematic review and meta- analysis. F1000Research 2021, 10, 73. [Google Scholar] [CrossRef] [PubMed]
  40. Luo, L.; Luo, T.; Du, M.; Mei, H.; Hu, Y. Efficacy and safety of tocilizumab in hospitalized COVID-19 patients: A systematic review and meta-analysis. J. Infect. 2022, 84, 418–467. [Google Scholar] [CrossRef]
  41. Oliynyk, O.V.; Rorat, M.; Barg, W. Oxygen metabolism markers as predictors of mortality in severe COVID-19. Int. J. Infect. Dis. 2021, 103, 452–456. [Google Scholar] [CrossRef]
Jpm 12 01103 g001 550
Figure 1. ROC curve for the three-factor model for predicting mortality risk in the entire study population comprising both Delta and no-Delta patients. Multivariate logistic regression model using procalcitonin, erythrocytes, and leukocytes.
Figure 1. ROC curve for the three-factor model for predicting mortality risk in the entire study population comprising both Delta and no-Delta patients. Multivariate logistic regression model using procalcitonin, erythrocytes, and leukocytes.
Jpm 12 01103 g001
Jpm 12 01103 g002 550
Figure 2. ROC curves (multivariate logistic regression models) for predicting mortality risk. (A): a model using procalcitonin for the Delta group. (B): a five-factor model using IL-6, fibrinogen, leukocytes, thrombocytes, and procalcitonin for the non-Delta group.
Figure 2. ROC curves (multivariate logistic regression models) for predicting mortality risk. (A): a model using procalcitonin for the Delta group. (B): a five-factor model using IL-6, fibrinogen, leukocytes, thrombocytes, and procalcitonin for the non-Delta group.
Jpm 12 01103 g002
Table
Table 1. Basic demographic and medical data of the groups studied. Median (QI-QIII) for continuous variables, numerical values (%) for categorical variables. The Mann–Whitney test was used for comparison between continuous variables, and Fisher’s exact test was used for categorical variables.
Table 1. Basic demographic and medical data of the groups studied. Median (QI-QIII) for continuous variables, numerical values (%) for categorical variables. The Mann–Whitney test was used for comparison between continuous variables, and Fisher’s exact test was used for categorical variables.
Delta (n = 78)Non-Delta (n = 224)p Value
Age, years70 (68–72)68 (66–71)<0.001
Sex, female (%)37 (47.4)108 (48,2)0.896
Temperature, °C37.5 (37.2–37.8)37.5 (37.3–37.8)0.947
CRP, mg/L113 (109–121)114 (101.5–123.5)0.825
Interleukin-6, pg/mL64 (47–71)60 (47–72)0.537
Procalcitonin, ng/mL0.8 (0.5–1.1)0.8 (0.6–1.0)0.695
Ferritin, ng/mL1211.5 (1011–1478)1216.5 (1033.5–1349)0.457
Fibrinogen, g/L6.7 (6.6–6.9)6.8 (3.5–6.9)0.320
D-dimer, µg/L1422 (1215–1977)1341 (1233–1647)0.081
Leukocytes, × 109/L4.2 (3.5–4.3)4.2 (3.950–4.3)0.934
Lymphocytes, %22 (17–24)24 (22–26)<0.001
Thrombocytes, × 109/L126 (101–138)128 (102–138)0.609
Erythrocytes, × 1012/L2.6 (2.2–3.2)3.2 (2.6–3.7)<0.001
PaO2/FiO2, mm Hg111 (74–130)139 (82–152)<0.001
Treatment with TOC, (%)30 (38.5)84 (37,5)0.893
Mortality, (%)35 (44.9%)100 (44.6%)>0.999
Laboratory test reference ranges: CRP < 5.0 mg/L, interleukin-6 <4.0 pg/mL, procalcitonin < 0.02 ng/mL, ferritin 8–143 ng/mL, fibrinogen 2.0–4.0 g/L, D-dimer <5 00 µg/L, leukocytes 4.0–9.0 × 109/L, lymphocytes 19–37%, thrombocytes 200–400 × 109/L, erythrocytes 3.6–4.2 × 1012/L, PaO2/FiO2 454–495 mm Hg.
Table
Table 2. Analysis of risk of death in a three-factor logistic regression model.
Table 2. Analysis of risk of death in a three-factor logistic regression model.
Independent VariablesThe Model Coefficient,
b ± m		Significance Level of Difference of the Coefficient from 0, p ValueOR (95% CI)
Procalcitonin, ng/mL7.788 ± 0.97<0.0012646 (392–18,000)
Erythrocytes, × 1012/L−0.98 ± 0.310.0020.38 (0.20–0.99)
Leukocytes, × 109/L−0.26 ± 0.130.0380.77 (0.60–1.003)
Table
Table 3. Basic demographic and medical data of the study groups with respect to treatment with tocilizumab. Median (QI–QIII) for continuous variables, numerical values (%) for categorical variables. The Mann–Whitney test was used for comparison between continuous variables, and Fisher’s exact test was used for categorical variables.
Table 3. Basic demographic and medical data of the study groups with respect to treatment with tocilizumab. Median (QI–QIII) for continuous variables, numerical values (%) for categorical variables. The Mann–Whitney test was used for comparison between continuous variables, and Fisher’s exact test was used for categorical variables.
A—Delta Groupno-TOC (n = 48)TOC (n = 30)p Value
Age, years71 (68–72)70 (68–71)0.552
Sex, female (%)22 (45.8)15 (50)0.817
Temperature, °C37.5 (37.2–37.75)37.6 (37.5–37.9)0.111
CRP, mg/L113 (106–121)112.5 (109–123)0.861
Interleukin-6, pg/mL60 (32.5–72)64 (61–68)0.36
Procalcitonin, ng/mL0.8 (0.2–1.1)0.8 (0.7–1.2)0.077
Ferritin, ng/mL1117.5 (998–1313.5)1483 (1143–1789)<0.001
Fibrinogen, g/L6.7 (6.6–6.9)6.7 (6.6–7.6)0.203
D-dimer, µg/L1245 (1121–1423)2108.5 (1657–2513)<0.001
Leukocytes, × 109/L4.2 (3.5–4.3)4.2 (4–4.3)0.36
Lymphocytes, %24 (22–26)17 (16–18)<0.001
Thrombocytes, × 109/L128 (102–138)126 (85–128)0.261
Erythrocytes, × 1012/L3.0 (2.6–3.65)2.2 (2.2–2.3)<0.001
PaO2/FiO2, mm Hg111 (76.5–132.5)98 (71–120)0.051
B—Non-Delta groupno-TOC (n = 140)TOC (n = 84)p value
Age, years68 (66–70)69 (66.5–71)0.062
Sex, female (%)68 (48.6)40 (47.6)>0.999
Temperature, °C37.5 (37.2–37.7)37.5 (37.4–37.95)0.036
CRP, mg/L113 (103.5–122.5)115 (99–125.5)0.391
Interleukin-6, pg/mL60 (34–72)68 (49–76)0.018
Procalcitonin, ng/mL0.8 (0.7–1.1)0.8 (0.5–1.0)0.073
Ferritin, ng/mL1211.5 (1015.5–1323.5)1222.5 (1067–1454.5)0.239
Fibrinogen, g/L6.5 (3.5–6.9)6.8 (6.650–8.6)<0.001
D-dimer, µg/L1305 (1211–1423)1432 (1341–1763)<0.001
Leukocytes, × 109/L4.1 (3.5–4.2)4.2 (4.2–4.35)<0.001
Lymphocytes, %24 (23–26)23 (18–26)0.006
Thrombocytes, × 109/L126 (102–138)128 (85–138)0.96
Erythrocytes, × 1012/L3.2 (2.75–3.7)2.8 (2.6–3.3)0.014
PaO2/FiO2, mm Hg136.5 (81–152)142 (88–149.5)0.070
Table
Table 4. Death n (%) in Delta and non-Delta patients treated or not with tocilizumab (TOC, no-TOC); Fisher’s exact test was used.
Table 4. Death n (%) in Delta and non-Delta patients treated or not with tocilizumab (TOC, no-TOC); Fisher’s exact test was used.
TOCNo-TOCp Value
Delta group
DeathNo16 (53.3%)27 (56.2%)0.819
Yes14 (46.7%)21 (43.7%)
Non-Delta group
DeathNo52 (61.9%)72 (51.4%)0.165
Yes32 (38.1%)68 (48.6%)
Table
Table 5. Analysis of risk of death in a single-factor logistic regression model for the Delta group.
Table 5. Analysis of risk of death in a single-factor logistic regression model for the Delta group.
Independent VariablesThe Model Coefficient,
b ± m		Significance Level of Difference of the Coefficient from 0, p ValueOR (95% CI)
Procalcitonin, ng/mL4.03 ± 0.91<0.00156.5 (9.4–341)
Table
Table 6. Analysis of risk of death in a five-factor logistic regression model for the non-Delta group.
Table 6. Analysis of risk of death in a five-factor logistic regression model for the non-Delta group.
Independent VariablesThe Model Coefficient,
b ± m		Significance Level of Difference of the Coefficient from 0, p ValueOR (95% CI)
Leukocytes, × 109/L–0.541 ± 0.210.0150.60 (0.40–0.91)
Fibrinogen, g/L1.23 ± 0.27<0.0013.43 (2.03–5.80)
IL-6, pg/mL−0.032 ± 0.0160.0400.97 (0.94–0.99)
Thrombocytes, × 109/L−0.063 ± 0.0190.0011.07 (1.03–1.10)
Procalcitonin, ng/mL13.3 ± 2.1<0.0015 × 105 (9900–3.4 × 106)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Oliynyk, O.; Barg, W.; Oliynyk, Y.; Dubrov, S.; Gurianov, V.; Rorat, M. Lack of Difference in Tocilizumab Efficacy in the Treatment of Severe COVID-19 Caused by Different SARS-CoV-2 Variants. J. Pers. Med. 2022, 12, 1103. https://doi.org/10.3390/jpm12071103

AMA Style

Oliynyk O, Barg W, Oliynyk Y, Dubrov S, Gurianov V, Rorat M. Lack of Difference in Tocilizumab Efficacy in the Treatment of Severe COVID-19 Caused by Different SARS-CoV-2 Variants. Journal of Personalized Medicine. 2022; 12(7):1103. https://doi.org/10.3390/jpm12071103

Chicago/Turabian Style

Oliynyk, Oleksandr, Wojciech Barg, Yanina Oliynyk, Serhij Dubrov, Vitaliy Gurianov, and Marta Rorat. 2022. "Lack of Difference in Tocilizumab Efficacy in the Treatment of Severe COVID-19 Caused by Different SARS-CoV-2 Variants" Journal of Personalized Medicine 12, no. 7: 1103. https://doi.org/10.3390/jpm12071103

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop