A Thorough Reproducibility Study on Sentiment Classification: Methodology, Experimental Setting, Results

Round 1

Reviewer 1 Report

This work deals with a prominent theme in subjects related to Informatics: the experiments' reproducibility. The main objective is reproducing another paper experiment and results using Sentiment Analysis using BERT architecture. This article reports an important study, reinforcing the value of reproducibility to ensure reliable research and further developments in the related field.

All the methodological procedures of the experiment are well explained, with several code demonstrations in Python, using the Jupyter Notebook (as demonstrated in section 4.1 about the computational environment used).

Despite its value, especially for researchers in the field of Natural Language Processing, I point out below some elements that deserve improvements throughout the text.

1.      Although built on a solid theoretical foundation, with 26 references, the study deserves more current references. I suggest introducing new references from the last three years (2020, 2021, and 2022) with the appropriate comments. It demonstrates the authors' concern with keeping their research in line with the flow of ideas in current research.

2.      I understand that Monya Baker's 2016 article, published in Nature, has opened the door to further studies on the issue of reproducibility in research on informatics subjects. It would be interesting to see a section summarizing these studies. I did a quick search on Scopus with the term "reproducibility crisis" and found 51 results for 2020, 58 for 2021, 54 for 2022, and 5 for 2023. Perhaps the authors can select some of these materials to create a summary table, demonstrating objectives, methodology, and main results.

3.      Although I have no comments on improvements in the results part, as I believe they are at a more than adequate level of depth, I recommend subsections dedicated exclusively to the authors' reflections on what their study implies. The conclusions already contain authors' notes about this, however, due to the reproducibility importance, I recommend a better development. Questions such as the ones I highlight below deserve dedicated subsections:

a.       How does this study reflect on current knowledge about issues of reproducibility of computational experiments? The authors must develop their opinion well, according to the knowledge acquired based on their experiences.

b.      Was the original study by Cheang et al. (2020) able to provide all the necessary information to ensure its reproduction in all respects? Here, I believe it is essential to point out the main points that raised doubts based on the original article, pointing out what the authors of the manuscript under review did to overcome such doubts.

c.       What are the problems and challenges encountered? Here it is important to point out the major limiting points for the study reported in the manuscript under review.

d.      In the authors' opinion, objectively, what could be done in the original study to overcome the problems found? I believe this question can provide ideas that respond to the limitations of the previous question.

As a final comment, I just reinforce that from the technical point of view, the reproduced experiment is very well explained, and its results are adequately represented. However, also thinking about reproducibility and the transparency involved, authors must ensure the fullest possible set of comments on the problems they faced.

Congratulations on the study!

Author Response

Dear reviewer,

thank you so much for this feedback and the appreciation of our work. Please, find hereby the list of the responses to your questions.

1.      Although built on a solid theoretical foundation, with 26 references, the study deserves more current references. I suggest introducing new references from the last three years (2020, 2021, and 2022) with the appropriate comments. It demonstrates the authors' concern with keeping their research in line with the flow of ideas in current research.

We thank the reviewer for this idea. As also suggested in comment 2, we have selected some of the most recent works and prepared a table to summarise the main findings.

2.      I understand that Monya Baker's 2016 article, published in Nature, has opened the door to further studies on the issue of reproducibility in research on informatics subjects. It would be interesting to see a section summarizing these studies. I did a quick search on Scopus with the term "reproducibility crisis" and found 51 results for 2020, 58 for 2021, 54 for 2022, and 5 for 2023. Perhaps the authors can select some of these materials to create a summary table, demonstrating objectives, methodology, and main results.

We agree with the reviewer. As we mentioned in our reply to comment 1, we have selected and organized a subset of the most relevant works and findings in a Table in the introduction section.

3.      Although I have no comments on improvements in the results part, as I believe they are at a more than adequate level of depth, I recommend subsections dedicated exclusively to the authors' reflections on what their study implies. The conclusions already contain authors' notes about this, however, due to the reproducibility importance, I recommend a better development. Questions such as the ones I highlight below deserve dedicated subsections:

We have added the questions and our answers in a dedicated section before the conclusions.

Thank you again and all the best,

Giorgio Di Nunzio

Reviewer 2 Report

Authors describe a reproduction of the experiment from the article “Language Representation Models for Fine-Grained Sentiment Classification”. The article is well written. Authors considered the problem of experiments reproducibility.

I can highlight the following issues:

1. Source code should be presented in text, not images.

2. Environment settings and hardware specifications should be presented as a table.

3. I couldn’t find links to the corrected source code. Where is the source code located?

4. Also, the authors should explain why they chose the article “Language Representation Models for Fine-Grained Sentiment Classification” for analysis.

5. I propose to add a “Discussion” section to the article, in which conclusions should be presented in more details.

Author Response

Dear Reviewer,

thank you very much for your kind feedback. Please, find hereby the list of your questions and our responses.

1. Source code should be presented in text, not images.

We thank you the reviewer for this suggestion. We changed all the figures concerning the source code with the more appropriate latex lstlisting.

2. Environment settings and hardware specifications should be presented as a table.

We thank you the reviewer for this suggestion. We added in Section 4 a table that summarises and compare the main (and known) item of the environment.

3. I couldn’t find links to the corrected source code. Where is the source code located?

The link to the source code is one of the first footnotes on page 2 of the Introduction (footnote 6). We added this information in Section 4 to make sure that the reader can find this link in an easier way.

4. Also, the authors should explain why they chose the article “Language Representation Models for Fine-Grained Sentiment Classification” for analysis.

Thank you for the advice. We added the requirements that led us to this paper at the beginning of Section 3 "Case Study"

5. I propose to add a “Discussion” section to the article, in which conclusions should be presented in more details.

In accordance to the suggestion of reviewer 1, we added a discussion section prior the conclusions.

Thank you again for you feedback and I wish you all the best,

Giorgio Di Nunzio

Round 2

Reviewer 1 Report

The recommendations I made in the previous round were met.

I reiterate that the study has quality and deals with a totally relevant aspect regarding computational experiments.

I am recommending acceptance.

Congratulations to the authors for the interesting study!

Reviewer 2 Report

 Authors considered the problem of experiments reproducibility. The article is well written.

Authors describe a reproduction of the experiment from the article “Language Representation Models for Fine-Grained Sentiment Classification”. 

Can be accepted.