Absence of “Cytokine Storm” in Hospitalized COVID-19 Patients: A Retrospective Cohort Study

Round 1

Reviewer 1 Report

This is a nice study with high importance for the field and certainly opened my eyes to a possible misunderstanding of "cytokine storm" in COVID-19. 

I think the authors have done an excellent job of showing real cytokine levels of interest and how they differ between patients with COVID-19 and those with other conditions (flu, burns) that have well-characterized cytokine storms.

I think I have smaller comments that don't necessarily require major revisions but would benefit from some discussion and/or additional information for readers:

1. The clinical characteristics of the COVID patients are given, but these are not given for the other cohorts (flu and burns). It would be useful to understand whether there were comparable rates of coinfections, comorbidities, death, etc.
2. Could the authors explain a little more why they assessed 46 cytokines but only reported 5? Was there nothing significant amongst the other 41? I realized these 5 may be the most interesting in cytokine storm biology, but is there any interesting data in the other 41?
3. The methods don't explicitly describe the approach to consent being obtained from the research subjects. Were subjects asked for consent or was a waiver of consent obtained from the IRB? If there was a waiver of consent because the samples were de-identified and remnant samples, how long did samples sit in the lab from time of collection to freezing at -80? At our institution it could take 24-96 hours before remnant samples are released to researchers, which could affect stability of cytokines.
4. Were the other cohorts (burns, flu, control) also remnant samples with same handling times or different?

Otherwise, great study and well explained!

Author Response

We thank the reviewer for constructive feedback. 

1. The clinical characteristics of the COVID patients are given, but these are not given for the other cohorts (flu and burns). It would be useful to understand whether there were comparable rates of coinfections, comorbidities, death, etc.

Respnse:  The influenza (CS) samples were obtained from a biorepository for another study. Thus, we do not have patient characteristics and are unable to include this information.

 

2. Could the authors explain a little more why they assessed 46 cytokines but only reported 5? Was there nothing significant amongst the other 41? I realized these 5 may be the most interesting in cytokine storm biology, but is there any interesting data in the other 41?

Response:  The five cytokines included in the analysis have been associated with cytokine storm and clinical outcomes of COVID-19 (references 5, 6, 7 and 14).  This data is included in the main manuscript and the figures within.  However, based on the reviewers’ suggestion, we now added a supplemental table (Suppl. Table 1) which includes data from additional cytokines and chemokines.  This table further supports our primary conclusion. 

 

3. The methods don't explicitly describe the approach to consent being obtained from the research subjects. Were subjects asked for consent or was a waiver of consent obtained from the IRB? If there was a waiver of consent because the samples were de-identified and remnant samples, how long did samples sit in the lab from time of collection to freezing at -80? At our institution it could take 24-96 hours before remnant samples are released to researchers, which could affect stability of cytokines.

 

Response:  An exemption from consent process was obtained from IRB, as only de-identified remnant samples and selected patient parameters were used.  The COVID samples were in refrigerator for a maximum of 72 hours prior to storage in freezer.  In our experience, this additional storage does not significantly influence concentrations of cytokines/chemokines. 

 

4. Were the other cohorts (burns, flu, control) also remnant samples with same handling times or different?

Response: The control, influenza and burn samples did not go through the same handling as they were obtained as part of other projects.  Some of these samples were thawed once for cytokine analysis (which again did not significantly influence the cytokine measurements: there was a good concordance of cytokine concentrations in the first assay and current study).

Reviewer 2 Report

In my opinion, this work is biased towards the hypothesis of an absence of the cytokine storm in covid-19 patients. The small sample size and the concomitant absence of a post hoc power analysis, the choice of a level of significance lower than the standard one, the absence of information about the recruitment criteria, seem to be in contrast with a balanced judgment on the phenomenon in question. 

Below, you can read some remarks:

1. Is it possible that the low levels of circulating cytokines were due to the fact that, at the moment of the analysis, they had already the receptors? Please, discuss this question.
2. The statistical analyses were performed at a significance level of 0.01, lower than the standard value (0.05) so that some results appearing not significant would be significant at 0.05 level.
3. A sample size calculation would be performed, or, alternatively, a post hoc power analysis would be executed.
4. Regarding data of figure 2 the choice to perform two t-tests is incorrect; the correct approach is an Anova for repeated measures.
5. Both for Anova and for t-test the assumptions of normality and homoscedasticity would be verified.
6. I suggest to specify the enrolment criteria.

Author Response

Although our manuscript might have led the reviewer to believe our bias towards hypothesis of absence of cytokine storm, we did not start with this idea.  In fact, we intended to use cytokine measurement for identifying the patients most likely to benefit from immunosuppressant use.  However, we revised our thinking about the role of cytokines in COVID-19 pathogenesis after careful analysis of data obtained. 

Regardless of our biases, the readers of the manuscript can make independent judgments, as we presented each data point of cytokine measurements for all COVID-19 subjects (Figure 1), in addition to grouped data.  Specific comments are addressed below.

1. Is it possible that the low levels of circulating cytokines were due to the fact that, at the moment of the analysis, they had already the receptors? Please, discuss this question.

Response:  If the reviewer intended to imply that the cytokines bound to their receptors may have contributed to low circulating concentrations of cytokines:  we are not aware of such phenomenon for any biological signal.  Such phenomenon is highly unlikely and would be a first of its kind for cytokines or other circulating signaling molecules.  Furthermore, the cytokine measurements were done over period of two weeks, including early infection periods when the antibody response was not fully developed.

 

2. The statistical analyses were performed at a significance level of 0.01, lower than the standard value (0.05) so that some results appearing not significant would be significant at 0.05 level.

Response:  We agree with the reviewer that the sample size is small and concede the inherent limitations of a retrospective study.  None of the comparisons in (Table 2) had P values between 0.01 and 0.05.  Thus, our primary conclusions would not have been different with 0.05 significance level.  Indeed, the only relevant instance of such P value was in figure 5 (P = 0.011), which was already stated in the legend of original manuscript.

 

3. A sample size calculation would be performed, or, alternatively, a post hoc power analysis would be executed

Response: Utility of a post-hoc power analysis is not universally accepted (For reference: https://www.graphpad.com/guides/prism/latest/statistics/stat_why_it_isnt_helpful_to_compute.htm).  As the reviewer is aware, there is an inverse relationship between observed statistical power and P value.  Hence, we used P < 0.01 to provide sufficient power in our analyses.

 

4. Regarding data of figure 2 the choice to perform two t-tests is incorrect; the correct approach is an Anova for repeated measures.

Response:  We agree with the reviewer that repeated measures ANOVA is the appropriate test to be used in figure 2.  However, for some study subjects we did not have data points for all 3 periods (Before, during and after dexamethasone treatment).  Using an ANOVA would have further decreased our N value.  Thus, we opted to use two paired t-tests.  Lower numerical values in AFTER period (for TNF and IP-10) were not statistically significant.  In fact, these lower values may simply be a reflection of later phase of infection (as seen in figure 3).  Furthermore, there was no difference between DEX and No-DEX groups in the cytokine values further providing support of lack of effect of dexamethasone.

 

5. Both for Anova and for t-test the assumptions of normality and homoscedasticity would be verified.

Response: Kolmogorov-Smirnov normality test was performed on all parametric tests.  As described in methods section, data (for table 1 analysis) was log transformed to achieve normality. 

 

6. I suggest to specify the enrolment criteria.

Response: No specific enrolment criterion was used.  As mentioned in the methods section “Sample inclusion depended on their availability and length of the patient’s hospital stay”.   

 

Reviewer 3 Report

Title: Absence of ‘Cytokine Storm’ in hospitalized COVID-19 patients: a retrospective cohort study

In this paper, the authors study coronavirus-19 and covid-19 disease in relation to the cytokine storm. The authors study 5 cytokines in the cytokine storm and report that the dexamethasone treatment did not significantly alter the concentrations of any of the cytokines analyzed. The authors conclude that an exaggerated cytokine response similar to the "cytokine storm" was not observed in COVID-19 patients during two weeks of hospitalization.

 ABSTRACT  "…if exaggerated cytokine response in the range of a ‘cytokine storm’ ".... This sentence is equivocal. To clarify.

 

Because these hospitalized COVID-19 patients have a fever that is certainly IL-1 mediated, this is not considered in the paper. Add in the discussion.

ABSTRACT "An exaggerated cytokine response similar to the cytokine storm was not observed in COVID-19 patients during two weeks of hospitalization...." This statement must be well explained as it conflicts with the main literature.

The authors should explain why they did not study IL-1 which is the major fever-inducing cytokine and other pro-inflammatory cytokines and chemokines. In addition, the authors should hypothesize an inhibition of pro-inflammatory cytokines. Therefore, to make this paper more interesting for the readers of this important journal, the authors should expand a little the discussion on this subject, in order to give a wider view to the reader.

Below I list 3 interesting articles that should be studied, incorporate the meaning and report them briefly in the discussion, and in the list of references.

 

Coronavirus-19 (SARS-CoV-2) induces acute severe lung inflammation via IL-1 causing cytokine storm in COVID-19: a promising inhibitory strategy.

Conti P, Caraffa A, Gallenga CE, Ross R, Kritas SK, Frydas I, Younes A, Ronconi G.J Biol Regul Homeost Agents. 2020 Oct 5;34(6). 

 

COVID-19 and Multisystem Inflammatory Syndrome, or is it Mast Cell Activation Syndrome?

Theoharides TC, Conti P. J Biol Regul Homeost Agents. 2020 Sep-Oct,;34(5):1633-1636. doi: 10.23812/20-EDIT3.

 

Mast cells contribute to coronavirus-induced inflammation: new anti-inflammatory strategy.

Kritas SK, Ronconi G, Caraffa A, et al. J Biol Regul Homeost Agents. 2020 January-February,;34(1):9-14. 

 

Fig. 3-4-5. The legends lack a clear explanation.

I believe these suggestions are important for improving this paper. Without these corrections the paper cannot be published. So I recommend minor revision.
I'd like to review this article after corrections. 

Author Response

The authors should explain why they did not study IL-1 which is the major fever-inducing cytokine and other pro-inflammatory cytokines and chemokines. In addition, the authors should hypothesize an inhibition of pro-inflammatory cytokines. Therefore, to make this paper more interesting for the readers of this important journal, the authors should expand a little the discussion on this subject, in order to give a wider view to the reader.

Response:  We did measure IL-1 concentrations in our multiplex assay.  These data were not included in the original manuscript for brevity and only the five cytokines implicated in cytokine storm and severity of COVID-19 pathology were included.  Based on the reviewers’ suggestion, we now included IL-1α and IL-1β concentrations in supplemental table 1. 

 

Below I list 3 interesting articles that should be studied, incorporate the meaning and report them briefly in the discussion, and in the list of references.

Response:  We thank the reviewer for directing us to 3 excellent articles on COVID-19 and the role of mast  cells in cytokine release.  At this stage, we are unable to attribute the changes in circulating cytokines to mast cells.  We did not measure IL-37, as it was not a part of the multiplex platform we used.  A different anti-inflammatory cytokine (IL-10) was measured and this data is now part of supplemental table 1. 

Reviewer 4 Report

In this study, the authors retrospectively measured 5 pro-inflammatory cytokines in patients hospitalized for Covid-19 +/- dexamethasone treatment and compared these findings to patients with influenza or burn injury patients. The authors did not find an exaggerated cytokine response in the Covid-19 patients similar to the reported cytokine storms reported in these patients. In addition, the authors were able to demonstrate that dexamethasone did not significantly affect cytokine levels in patients with Covid-19.

Although the study is interesting, the main concern is that the study is significantly underpowered. This is especially true when it comes to treatment comparisons. There are some trends emerging but due to low patient numbers, significant patient data variability, and some time point variability these are not showing as statistically significant. Please see other comments below.

 

Comments:

1. Although it is mentioned that all the Covid-19 patients in this study were hospitalized, were there any differences in regard to severity of symptoms, respiratory symptoms and treatment?
2. Please describe the dexamethasone treatment regimen.
3. Figure 1: The differences between the blue and black in the dexamethasone group are difficult to distinguish.
4. Figure 2: What is considered before, during, and after treatment in regard to time frame? Why were these time frames chosen? The before data points, how do they correlate with start of symptoms or positive test results? For some of the results, there is a trend. However, the low number of patient samples and a significant amount of variability obscures this.
5. Figure 3 demonstrates the significant variability of findings within each cytokine group.
6. Figure 4: Please clarify what was defined as early vs. late infection. It is unclear why a RBD value on the first day of hospitalization was chosen for this. Is there a way to evaluate at what day or time frame of infection the Covid-19 patients were admitted to the hospital and do some more comparisons?

Author Response

1. Although it is mentioned that all the Covid-19 patients in this study were hospitalized, were there any differences in regard to severity of symptoms, respiratory symptoms and treatment?

Response:  Due to retrospective nature of the study, we are unable to obtain additional clinical parameters and correlate the severity of symptoms with cytokine concentrations or dexamethasone treatment. 

 

2. Please describe the dexamethasone treatment regimen.

Response:  Dexamethasone dosage and duration of treatment is added to the methods section. 

 

3. Figure 1: The differences between the blue and black in the dexamethasone group are difficult to distinguish

Response: Higher resolution figure 1 will be used for final submission.  The color of symbols will be changed if needed. 

 

4. Figure 2: What is considered before, during, and after treatment in regard to time frame? Why were these time frames chosen? The before data points, how do they correlate with start of symptoms or positive test results? For some of the results, there is a trend. However, the low number of patient samples and a significant amount of variability obscures this

Response:  Before, during and after simply reflects the relationship to dexamethasone treatment.  The days of dexamethasone treatment (average values of all blue symbols in figure 1) are included in the “during” group.  Average values “before” and “after” this period are included in respective groups.  We agree that there appears to be a trend towards decrease in TNF and IP-10.  Lower numerical values in AFTER period (for TNF and IP-10) were not statistically significant.  In fact, these lower values may simply be a reflection of later phase of infection (as seen in figure 3).  Furthermore, there was no difference between DEX and No-DEX groups in the cytokine values further providing support of lack of effect of dexamethasone.

 

5. Figure 3 demonstrates the significant variability of findings within each cytokine group.

Response:  We agree that there is a large variability in figure 3.  This is simply a reflection of inter-individual variability in cytokine concentrations and a normal Y axis scale (rather than log scale used in figure 1).

 

6. Figure 4: Please clarify what was defined as early vs. late infection. It is unclear why a RBD value on the first day of hospitalization was chosen for this. Is there a way to evaluate at what day or time frame of infection the Covid-19 patients were admitted to the hospital and do some more comparisons?

Response: It is possible to align the data with date of positive RT-PCR test.  However, the samples used for cytokine analysis and subject inclusion are based on the day of hospitalization.  Thus, it was best to present the data with reference to hospitalization day.  A graph made with reference to PCR date would make the data points asynchronous and difficult to read.  We chose an RBD antibody value on first day of hospitalization as a reasonable surrogate for early and late stage infections

Round 2

Reviewer 1 Report

The authors have addressed my concerns.

Author Response

Thanks

Reviewer 2 Report

In my opinion, the lack of a power analysis remains a critical point. I suggest reading all the most important statistical textbooks about this topic (and not a handbook of software. 

In table I both death and intubation have p-value included in the interval 0.01 and 0.05. I'm very interested, however, in significance of Figure 2. The same problem the the authors indicate in missing data, implies that they have used a t-test for unpaired data, that is uncorrect. 

Regarding the assumptions of normality, the K-S test would be used also AFTER  the log-transformation. In addition,which test was used in order to assess omoschedasticity?

 

 

Author Response

In table I both death and intubation have p-value included in the interval 0.01 and 0.05. I'm very interested, however, in significance of Figure 2. The same problem the the authors indicate in missing data, implies that they have used a t-test for unpaired data, that is uncorrect. 

RESPONSE:  Yes.  In table 1 intubation and death numbers have a p-value between 0.01 and 0.05.  In addition, the ARDS rates included in the revised table 1, based on the suggestion of academic editor has a p value of 0.043. 

The exact p-values for figure 2, using a paired t-test were as follows.

IL6:  Before vs During - 0.62, During Vs After - 0.47

IL8:  Before vs During - 0.99, During Vs After - 0.65    

TNF:  Before vs During - 0.25, During Vs After - 0.048  

MCP1:  Before vs During - 0.35, During Vs After - 0.97           

IP10:  Before vs During - 0.041, During Vs After - 0.015

As described in our first response, decrease in TNF in after DEX period and in IP10 in during and after periods are significant at 0.05 level.  However, these are most likely related to time-course of infection rather than the effect of dexamethasone (Please see figure 3).  Similar time-dependent decreases in these cytokines were also observed in SARS1 (reference 16 of manuscript).  In addition, there were no statistical differences in these cytokine values between DEX and NO DEX groups (Table 2).  Thus, based on these multiple lines of evidence, we could not attribute the decreases observed to dexamethasone treatment and did not specifically discuss this aspect of figure 2.    

 

Regarding the assumptions of normality, the K-S test would be used also AFTER  the log-transformation. In addition,which test was used in order to assess omoschedasticity?

RESPONSE:  Yes, we did test K-S test on values after log transformation and the data was confirmed to be normally distributed.  Bartlett’s test was used to ensure homoscedasticity.   

Reviewer 4 Report

No further questions. Thank you

Author Response

Thanks.