Inter-Rater Variability in the Evaluation of Lung Ultrasound in Videos Acquired from COVID-19 Patients

Round 1

Reviewer 1 Report

Good and very interesting job. 

Author Response

Thank you for taking the time to review our study and we greatly appreciate the reviewer’s comment.

Reviewer 2 Report

The authors have presented a well-designed and well-written study evaluating a couple aspects of lung ultrasound image interpretation in patients with COVID: (1) inter-rater reliability in general and (2) severity of sonographic findings in transverse vs longitudinal lung views.

 

Although other authors have published similar small sample size studies on the topic of lung POCUS inter-rater reliability (including in COVID-19) before, this is an important question that merits continued study on more sonographic samples and operators.  Further, the authors comparison of the diagnostic yield of transverse vs longitudinal views is, to my knowledge, novel.

 

I think the study is very worthy of publication.  Minor constructive feedback below.

 

Major comments:

 

Lines 339-347: In this section, I would encourage the authors to add one more point regarding the difference between longitudinal and transverse views of the lung.  Although these two views produced similar diagnostic yield in this study, the study only examined one subtype of lung pathology: increased lung density (ranging from B-lines to consolidations).  However, lung ultrasound also has a major goal to answer whether the patient could have a pneumothorax.   And to determinate that, the longitudinal view is – in my experience – strongly preferrable.  Why?  Because the longitudinal view necessarily contains ribs and these ribs provide a depth landmark: the pleural line is always deep to the ribs.  In contrast, the transverse view typically omits ribs and so makes it difficult to determine whether a visualized horizontal line is the pleural line or not.  If the pleural line is active (e.g., contains lung sliding or lung pulse), then either orientation permits identification of the pleural line.  But if the pleural line is static, only the longitudinal orientation permits consistent visualization of ribs.  And the visualization of ribs helps to determine whether a static horizontal line could be the pleural line or subcutaneous emphysema: a static pleural line would be deep to the ribs whereas subcutaneous emphysema is superficial to the ribs and typically prevents the visualization of ribs entirely.  In other words, if the diagnostic yield of the two orientations is similar per the results of this study, that does NOT mean that the two orientations are similarly useful for lung ultrasound in general.  For busy clinicians who want to do the most efficient exam in the least amount of time and do not wish to perform every view twice (once in transverse and once in longitudinal orientation), the longitudinal view is more diagnostically useful than the transverse view.

 

Minor comments:

 

Lines 96-97: there is subject verb mismatch in the following sentence: “LUS imaging is typically 96 less expensive than conventional chest X-ray or computed tomography (CT), making 97 them convenient”.  To fix this, change “them” to “it”.

Author Response

Comments and Suggestions for Authors

The authors have presented a well-designed and well-written study evaluating a couple aspects of lung ultrasound image interpretation in patients with COVID: (1) inter-rater reliability in general and (2) severity of sonographic findings in transverse vs longitudinal lung views.

 

Although other authors have published similar small sample size studies on the topic of lung POCUS inter-rater reliability (including in COVID-19) before, this is an important question that merits continued study on more sonographic samples and operators.  Further, the authors comparison of the diagnostic yield of transverse vs longitudinal views is, to my knowledge, novel.

 

I think the study is very worthy of publication.  Minor constructive feedback below.

 

Major comments:

 

Lines 339-347: In this section, I would encourage the authors to add one more point regarding the difference between longitudinal and transverse views of the lung.  Although these two views produced similar diagnostic yield in this study, the study only examined one subtype of lung pathology: increased lung density (ranging from B-lines to consolidations).  However, lung ultrasound also has a major goal to answer whether the patient could have a pneumothorax.   And to determinate that, the longitudinal view is – in my experience – strongly preferrable.  Why?  Because the longitudinal view necessarily contains ribs and these ribs provide a depth landmark: the pleural line is always deep to the ribs.  In contrast, the transverse view typically omits ribs and so makes it difficult to determine whether a visualized horizontal line is the pleural line or not.  If the pleural line is active (e.g., contains lung sliding or lung pulse), then either orientation permits identification of the pleural line.  But if the pleural line is static, only the longitudinal orientation permits consistent visualization of ribs.  And the visualization of ribs helps to determine whether a static horizontal line could be the pleural line or subcutaneous emphysema: a static pleural line would be deep to the ribs whereas subcutaneous emphysema is superficial to the ribs and typically prevents the visualization of ribs entirely.  In other words, if the diagnostic yield of the two orientations is similar per the results of this study, that does NOT mean that the two orientations are similarly useful for lung ultrasound in general.  For busy clinicians who want to do the most efficient exam in the least amount of time and do not wish to perform every view twice (once in transverse and once in longitudinal orientation), the longitudinal view is more diagnostically useful than the transverse view.

 

→ Response: This is an interesting point. Following your suggestion, we added: “This does not necessarily mean the two orientations are similarly useful, especially since we only examined a type of interstitial lung disease. In certain pathologies, such as pneumothorax, the visualization of the ribs provides a depth landmark, and helps to better identify the pleural line. In consequence, our group believes that each patient might benefit most from a different approach, adapted to a flexible scanning protocol subject to the clinical scenario.”

 

Minor comments:

 

Lines 96-97: there is subject verb mismatch in the following sentence: “LUS imaging is typically 96 less expensive than conventional chest X-ray or computed tomography (CT), making 97 them convenient”.  To fix this, change “them” to “it”.

 

→ Response: Thank you for pointing out the mistake, it was amended in the revised version.

 

 

Reviewer 3 Report

Summary:

   The authors set out to assess the inter-rater reliability (IRR) of 33 experienced ultrasound physicians with regard to lung ultrasound score in patients with covid.  To my knowledge this is the largest such study in terms of the number of interpreting physicians studied.   The authors fund substantial IRR in clips or normal A-line pattern with only fair IRR in patients with B-line patterns (<50% confluence vs >50% confluence).   They also compared the impact of scanning in a horizontal vs vertical orientation and found that horizontal scanning (avoiding the ribs lead) to slightly higher scoring with similar IRR.   This study highlights one of the limitations with lung ultrasound.

 

Critique:

-        Overall the study was well designed, and assessed the IRR of a large number of physicians experienced with lung ultrasound

-        The study is relevant because lung ultrasound is becoming a more frequently utilized tool but the standardization of both performing the exam and interpretation is incompletely understood at this time

-        There article appears to be well written with little to no grammatical errors. 

 

 

Major points:

-        It was not clear if those reviewing the images were give images in random order, or if they saw all the images from one patient before going onto the next patient.  This point should be clarified as it could bias them toward consistent findings in a given patient.

-        One criticism would be the chosen lung scoring system.  Many exist in the literature. The chosen score probably makes IRR look better than others would as there is typically a better IRR at the extreme findings (a-lines compared to consolidation), but a lot of discrepancy with regard to “the amount of b-lines.”  There are several ways in the literature to score b-lines.    In the presented scoring system a  score of 1 is of unknown significance.  Most references report that a single B-line in an intercostal space can be normal, and that at least 3 B-lines in a single intercostal space is required to definite it as pathologic or “alveolar interstitial syndrome.”  This score does not really account well for the possible difference of 3 small discrete B-lines and b-lines at occupy 49% of the rib space; which are likely quit different severity clinically.

-        A lung score of 3 combines different lung ultrasound patterns more broadly than most scoring systems.  Pleural effusions and consolidations patterns or often excluded from these scoring systems.   Likely comparing only B-lines occupying <50% with only B-lines but occupying >50% of the intercostal space would likely result in weaker IRR.  It would be helpful to report how many of the images scored as a 3 contained effusions or consolidations. 

-        It would also be helpful to know if those scoring the exam score the most severe appearance during the clip, or more of an average (the amount of B-lines often fluctuates within the respiratory cycle.

 

Minor points:

 

-        Figure 3 is very hard to read due to small font, and lacks labeling.  Perhaps it would help to label each physician (A,B,C on each column).    

-        Page 4 line 152 seems like an incomplete sentence.  I would not end a sentence only with a reference number.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

This study describes the inter-rater agreement of LUS experts when evaluating the main LUS findings for COVID-19 and the impact of the transducer orientation in LUS acquisitions in COVID-19 patients on the observed findings. In this study, several LUS findings demonstrated moderate agreement (consolidations and B-lines >50%) and others a fair agreement (individual B-lines and confluent B-lines <50%). At the same time, a substantial IRR was found in the cases of normal ultrasound findings. Longitudinal and transverse scans showed no statistically significant differences between them.

In general, the paper is well written, and it first characterizes the inter-rater agreement of LUS experts when evaluating the main LUS findings for COVID-19.

Some comments which should be addressed.

1. Line 138 please delete a point

2. Please increase the bibliography (Introduction or Discussion section) to underline the advantages of lung ultrasound, in particular in this patient setting, by citing the following works, which show how bedside ultrasound should be encouraged to avoid transporting patients and reduce the risk of contamination of staff and other patients. Please cite: 

Mateos González M, García de Casasola Sánchez G, Muñoz FJT, Proud K, Lourdo D, Sander J-V, Jaimes GEO, Mader M, Canora Lebrato J, Restrepo MI, Soni NJ. Comparison of Lung Ultrasound versus Chest X-ray for Detection of Pulmonary Infiltrates in COVID-19. Diagnostics. 2021; 11(2):373. https://doi.org/10.3390/diagnostics11020373

Pellegrino F, Carnevale A, Bisi R, Cavedagna D, Reverberi R, Uccelli L, Leprotti S, Giganti M. Best Practices on Radiology Department Workflow: Tips from the Impact of the COVID-19 Lockdown on an Italian University Hospital. Healthcare (Basel). 2022 Sep 14;10(9):1771. doi: 10.3390/healthcare10091771. PMID: 36141383; PMCID: PMC9498676.

Wang M, Luo X, Wang L, Estill J, Lv M, Zhu Y, Wang Q, Xiao X, Song Y, Lee MS, Ahn HS, Lei J, Tian J. A Comparison of Lung Ultrasound and Computed Tomography in the Diagnosis of Patients with COVID-19: A Systematic Review and Meta-Analysis. Diagnostics. 2021;11(8):1351.vhttps://doi.org/10.3390/diagnostics11081351

 

3. In line 161” We adopted a 12-zone scanning protocol which has been previously validated and shown to be consistent with higher ICC and a higher degree of concordance with CT”, the reference is missing

 

 

 

 

 

Author Response

Please see the attachment

Author Response File: Author Response.docx