Teaching Challenges in COVID-19 Scenery: Teams Platform-Based Student Satisfaction Approach

Round 1

Reviewer 1 Report

As described in the Introduction section "This research is intended to verify a comparison between online learning and classroom instruction in these circumstances", referring to the new situation occurred by the COVID-19 in so many university campuses around the world.
The text also describes textually: "The objective of this work is not to describe the characteristics of Virtual Learning and Learning Platforms and LMS. It is intended to analyze the level of student’s satisfaction by using the Teams Platform in the REE in the crisis generated by COVID-19. Apply deep learning to compare the results achieved in the descriptive study of satisfaction in the use of Teams by UVM students."

In summary, the three main issues of the paper could be described in relation to the previous paragraph of the Introduction:
1. The paper focuses on assessing the degree of satisfaction with the use of the proprietary platform Teams. What happens is that it seems to focus too much on such a platform, rather than trying to evaluate or distinguish whether the assessment made of it corresponds specifically to specific qualities of the platform, without trying to know whether such an evaluation would have been similar with the use of any other LMS platform. In that sense, the questionnaire is not about discerning assessments that could be more related to an e-learning behavior than to e-learning in Teams.

2. In this regard, a comparison should be made between Teams and other LMS platforms, trying to discern which elements, if any, are unique to this platform and not to the rest. In order to have an adequate approach to the student's perception of the LMS, one should have asked if they know or have used any other LMS before, and if the experience with Teams has been more favorable or not. These types of questions were missed by the researchers.

3. Deep learning is said to be applied to compare results with descriptive analysis. It is important to mention that this is not really done at any time in the paper. This is a very serious shortcoming as it announces something that is not really done at all. According to what can be seen in the appendices it seems that the authors construct several models of deep learning considering part of the questionnaire in each of them, obtaining a regression model with a certain level of precision. This has no greater implication than the fact that it is possible to create such prediction models from the answers, but in itself, it must be clear that it is not really providing any knowledge about such answers, nor does it even make sense to predict by means of regression the answers obtained, at least the authors do not focus on this at all.
The authors do not adequately describe either the purpose of using deep learning nor do they adequately describe the technique used to do so. They limit themselves to describing that they use Rapid Miner when what is relevant is the deep learning architecture implemented and not so much the software with which it is carried out.

Other more specific issues are detailed below:

4. According to the title of the paper, an evaluation using in-depth deep learning techniques is expected from the paper. This does not really happen at any time in the manuscript, as no conclusions are actually described or developed from an analysis carried out with artificial neural networks.

5. The abstract provides data on the analysis carried out with deep learning techniques that are not described or explained in the body of the paper. It only appears in the appendix.

6. Row 27. Missing quote.

7. The numerical notation used in Table 1, of the type "356 thousand 530" is strange and not very understandable.

8. Rows 166-167: It is literally written: "The results generated in RapidMiner allow validation of the findings determined in the descriptive study." This is in no way demonstrated by the study. No real tests or descriptions obtained from the deep learning models are provided.

9. Rows 167-171. It's more like a result or discussion section.

10. Table 3. All calculated coefficients should be shown, clarifying in due time all the questionnaire variables that each one incorporates.

11. The article looks like it's intended to promote Teams. At no time does it compare what Teams offers in relation to other tools, nor does it describe whether what it provides is really innovative or on the contrary can be achieved with open or free source tools, without any corporate interests or benefits from their use. It is almost implied in some way that the success of the experience has been more by Team than by the attitude and effort of the faculty and the students, which does not seem to be founded.

12. Equation (3) is poorly formatted, so it is not possible to review it properly.

13. Although the authors do not describe it, it seems that the deep learning architecture used is "H20" regression. The authors do not describe the variables to be predicted in such a regression model.

14. In relation to deep learning the paper is full of misleading generalities, some very vague, others even incorrect. For example:
-Rows 135-137: "Although there are different techniques for implementing deep learning, it is one of the most common is to simulate an artificial neuronal network (ANN) system within the data analysis software."
-Figure 1. An extremely basic neural network is described, far from a deep learning neural network architecture.
-Rows 343-345: "It is essential to apply Neural Networks, Machine Learning, and deep learning, among other tools, to improve educational processes in higher education institutions in Mexico." This phrase, without supporting it with further evidence, is clearly generic and does not belong in a scientific paper.
-Rows 385-388: "Regarding the application of deep learning, its importance lies in allowing through quantitative techniques, designing, and managing the strategies necessary to transform data into the key, essential, optimal, productive, and scalable information to build (possible futures) to make better decision-making in the present."
-Rows 389-394: "Deep learning is mentioned because of its analytical and predictive ability, allowing the construction of an improved idea of the technology involved in educational processes. Some of its most important variables of connection, access to campus/LMS platform, interactions between stops, and review of teaching content time. But these results will allow defining variables related to teaching work, linked to several specific groups of students, and even covering an entire training project."

15. Row 348: The authors describe that the sample design is adequate, but they do not really evaluate other sample designs that could have been more oriented or intentional, such as stratified sampling.
Nor is it detailed how the surveys were obtained. They were probably obtained voluntarily from students who were effectively well immersed in the new type of learning, and those who were disconnected probably did not have access, opportunity or interest in answering the form. This may indicate a significant bias in the assertive responses to the survey, which is not mentioned or described in the manuscript.

16. The results shown in a quantitative way are quite poor, limited to an analysis of the reliability of the instrument by means of Cronbach's Alpha, but no evidence is shown at the level of statistical significance of the results supposedly obtained. The statistical results that support the discussion are limited to frequencies. Authors are encouraged to explore other statistics of higher statistical power (e.g. non-parametric tests, etc.).

17. The authors are recommended to investigate and deepen the methodologies of survey analysis with the help of artificial intelligence and neural networks, such as:
Abarca-Alvarez, F.J.; Campos-Sánchez, F.S.; Mora-Esteban, R. Survey Assessment for Decision Support Using Self-Organizing Maps Profile Characterization with an Odds and Cluster Heat Map: Application to Children’s Perception of Urban School Environments. Entropy 2019, 21, 916. https://doi.org/10.3390/e21090916

Author Response

Reviewer 1.

1. The paper focuses on assessing the degree of satisfaction with the use of the proprietary platform Teams. What happens is that it seems to focus too much on such a platform, rather than trying to evaluate or distinguish whether the assessment made of it corresponds precisely to specific qualities of the platform, without trying to know whether such an evaluation would have been similar with the use of any other LMS platform. In that sense, the questionnaire is not about discerning assessments that could be more related to an e-learning behavior than to e-learning in Teams.

Thank you very much for the comments.  To improve the manuscript, it was decided to change the Title   From Deep Learning-Based Assessment of a Teaching Remote Platform and Traditional; COVID-19 scenario. UVM Case.

To:  Teaching Challenges in COVID-19 Scenery: Teams platform-based Student Satisfaction Approach    To make it more appropriate with work content. And adjusted the abstract.

Adaptations were made to the objective of the work se lines  79-85.

2. In this regard, a comparison should be made between Teams and other LMS platforms, trying to discern which elements, if any, are unique to this platform and not to the rest. To have an adequate approach to the student's perception of the LMS, one should have asked if they know or have used any other LMS before, and if the experience with Teams has been more favorable or not. The researchers missed these types of questions.

Thank you very much for the comments; conditional logging of the attributes and characteristics of LMS Platforms were included and aspects relevant to the decision of the use of Teams. See Lines 106-120.

3. Deep learning is said to be applied to compare results with descriptive analysis. It is essential to mention that this is not really done at any time in the paper. This is a very serious shortcoming as it announces something that is not really done at all. According to what can be seen in the appendices, it seems that the authors construct several models of deep learning considering part of the questionnaire in each of them, obtaining a regression model with a certain level of precision. This has no greater implication than the fact that it is possible to create such prediction models from the answers, but in itself, it must be clear that it is not really providing any knowledge about such answers, nor does it even make sense to predict by means of regression the answers obtained, at least the authors do not focus on this at all.
   The authors do not adequately describe either the purpose of using deep learning nor do they adequately describe the technique used to do so. They limit themselves to describing that they use RapidMiner when what is relevant is the deep learning architecture implemented and not so much the software with which it is carried out.

Thank you very much for the feedback. The adjusted have been made in the manuscript. See lines 143-156.

4. According to the title of the paper, an evaluation using in-depth deep learning techniques is expected from the paper. This does not really happen at any time in the manuscript, as no conclusions are actually described or developed from an analysis carried out with artificial neural networks.

Thank you very much for the comments.  To improve the manuscript, it was decided to change the Title   From Deep Learning-Based Assessment of a Teaching Remote Platform and Traditional; COVID-19 scenario. UVM Case.

To:  To:  Teaching Challenges in COVID-19 Scenary: Teams platform-based Student Satisfaction Approach    To make it more appropriate with work content.

5. The abstract provides data on the analysis carried out with deep learning techniques that are not described or explained in the body of the paper. It only appears in the appendix.

Thank you very much for the observation, the adjustments have been made in the abstract, based on the change of title so that it is more in accordance with the content.

6. Row 27. Missing quote.

Thank you very much for your observation. The cite was included as footnote one.

7. The numerical notation used in Table 1, of the type "356 thousand 530" is strange and not very understandable.

Thank you very much for the observation, corrections were made in the notation in Table 1.

8. Rows 166-167: It is literally written: "The results generated in RapidMiner allow validation of the findings determined in the descriptive study." This is in no way demonstrated by the study. No real tests or descriptions obtained from the deep learning models are provided.

Thank you very much for the observation,  the word validation was changed by comparison see lines 192-191 reulting in teh following  quote  “the application of data science to compare the results achieved in the descriptive study of satisfaction in the use of Teams by UVM students was obtained by applying machine learning tools such as Deep learning, neural networks, among others. By doing this, the research seeks to establish new procedures for diagnosing and improving educational practices. As a secondary objective, it is intended to apply data science tools to compare the results obtained through statistical analysis (See Lines  80-84)”.  And see lne 250.

9. Rows 167-171. It's more like a result or discussion section.

Thank you very much for the observation. we believe that the observation should continue as part of the results.

• Table 3. All calculated coefficients should be shown, clarifying in due time all the questionnaire variables that each one incorporates.

      Thank you very much for the observation, adjustments were made. See lines 209-212, and lines 276-277.

• The article looks like it's intended to promote Teams. At no time does it compare what Teams offers in relation to other tools, nor does it describe whether what it provides is really innovative or, on the contrary can be achieved with open or free source tools, without any corporate interests or benefits from their use. It is almost implied in some way that the success of the experience has been more by Team than by the attitude and effort of the faculty and the students, which does not seem to be founded.

Thank you very much for the observation adjustments that were made to the manuscript to explain the use of Virtual Platforms.  See lines 93-119.

• Equation (3) is poorly formatted, so it is not possible to review it properly.

 Thank you for the comment; the correction was made in equation 3 (change to equation 2).  See  line 161.

• Although the authors do not describe it, it seems that the deep learning architecture used is "H20" regression. The authors do not describe the variables to be predicted in such a regression model.

Thank you very much for the observation. The RapidMiner app is comparing the usability and resources of Teams.

• In relation to deep learning, the paper is full of misleading generalities, some very vague, others even incorrect. For example: Rows 135-137: "Although there are different techniques for implementing deep learning, it is one of the most common is to simulate an artificial neuronal network (ANN) system within the data analysis software."
  -Figure 1. An extremely basic neural network is described, far from a deep learning neural network architecture.
  -Rows 343-345: "It is essential to apply Neural Networks, Machine Learning, and deep learning, among other tools, to improve educational processes in higher education institutions in Mexico." This phrase, without supporting it with further evidence, is clearly generic and does not belong in a scientific paper. -Rows 385-388: "Regarding the application of deep learning, its importance lies in allowing through quantitative techniques, designing, and managing the strategies necessary to transform data into the key, essential, optimal, productive, and scalable information to build (possible futures) to make better decision-making in the present."
  -Rows 389-394: "Deep learning is mentioned because of its analytical and predictive ability, allowing the construction of an improved idea of the technology involved in educational processes. Some of its most important variables of connection, access to campus/LMS platform, interactions between stops, and review of teaching content time. But these results will allow defining variables related to teaching work, linked to several specific groups of students, and even covering an entire training project."

Thank you very much for the observation adjustments were made. See lines 157-158.

• Row 348: The authors describe that the sample design is adequate, but they do not really evaluate other sample designs that could have been more oriented or intentional, such as stratified sampling. Nor is it detailed how the surveys were obtained. They were probably obtained voluntarily from students who were effectively well immersed in the new type of learning, and those who were disconnected probably did not have access, opportunity, or interest in answering the form. This may indicate a significant bias in the assertive responses to the survey, which is not mentioned or described in the manuscript.

Thank you very much for the observation; changes were made to the manuscript. See line 158 and line 221.

• The results shown in a quantitative way are quite poor, limited to an analysis of the reliability of the instrument by means of Cronbach's Alpha, but no evidence is shown at the level of statistical significance of the results supposedly obtained. The statistical results that support the discussion are limited to frequencies. Authors are encouraged to explore other statistics of higher statistical power (e.g. non-parametric tests, etc.).

     Thank you very much for the observation, changes were made. See lines 211-212 and 269-277.

• The authors are recommended to investigate and deepen the methodologies of survey analysis with the help of artificial intelligence and neural networks, such as:
  Abarca-Alvarez, F.J.; Campos-Sánchez, F.S.; Mora-Esteban, R. Survey Assessment for Decision Support Using Self-Organizing Maps Profile Characterization with an Odds and Cluster Heat Map: Application to Children's Perception of Urban School Environments. Entropy 2019, 21, 916. https://doi.org/10.3390/e21090916

Thank you very much for the observation.  See lines 288-289, and references were added to support.

Author Response File: Author Response.pdf

Reviewer 2 Report

The article talks about the deep learning-based assessment of Microsoft Teams as an online teaching platform in contrast to traditional on-campus teaching. The thing that concerns me most is the machine (deep) learning part, as the whole paper as well as the title is based upon that. 

1. It is not clear as to why authors opted for a deep-learning model. 
2. No details and justification are provided concerning the choice of the model, architecture, and the hyper-parameters.
   1. why 4 layers architecture?
   2. why 10 epochs only?
   3. Why training different models for different aspects (features)? Why not a single model with all features (aspects) combined. 
   4. The validation criteria of the model itself are not explained. i.e., the train-test split, cross-validation, etc. 
   5. Figure 1 is quite general. The authors should replace it with the actual architecture used in their study showing the no. of input/output nodes and the hidden layers. 
   6. What is the difference between Table A4 and A8?
3. A thorough proof-read of the article is needed.
   1. For instance, page 1, line 2 (abstract): "...the classes face to face all educational levels."
   2. Page 3, lines 83-85. 
   3. 'T' in teams is somewhere capitalized and at some places is small. 
   4. Most paragraphs are either right-aligned or not formatted properly. 
   5. Typo on line 304 page 12 - schwool and on
   6. page 13, line 371 - "..than 75% were 75% of a 70..."
   7. page 15, line 413 - "coVID-19"
4. Strange way to show the number of students in Table 1.
5. Page 3, line 93: the link for Microsoft Teams should be a footnote.
6. Table 2 doesn't make much sense to me. "The caption is stages of organization of the virtualized teaching operations" and by looking at the table it's difficult to comprehend the steps followed in each step of the stage. It also refers to the use of Blackboard which is not undertaken in this study. 
7. A lot of studies have been carried out on similar lines, where researchers have evaluated various LMS, MOOCs platforms, and teaching tools. For instance, https://ieeexplore.ieee.org/abstract/document/7403480 . How this study compares to the existing work found in the literature and similar studies. 
8. The paper lacks novelty and does not present state-of-the-art or studies alike in this domain. 
9. Why in Equation 2 are two jj? 
10. Page 6, line 166: authors claim that "the results generated in RapidMinder allow validation of the findings determined in the descriptive study". The question is how? I couldn't find the answer within the document. 
11. Equation 3 on page 6 seems broken.

Author Response

Reviewer 2.

• It is not clear as to why authors opted for a deep-learning model. 
• No details and justification are provided concerning the choice of the model, architecture, and the hyper-parameters.

1. why 4 layers architecture?
2. why 10 epochs only?
3. Why training different models for different aspects (features)? Why not a single model with all features (aspects) combined. 
4. The validation criteria of the model itself are not explained. i.e., the train-test split, cross-validation, etc. 
5. Figure 1 is quite general. The authors should replace it with the actual architecture used in their study showing the no. of input/output nodes and the hidden layers. 
6. What is the difference between Table A4 and A8?

Thank you very much for the observation. The correction was made. See lines 156-158.  And The Tables A4 and A8 were adjusted

• A thorough proof-read of the article is needed.

The correction was made as a footnote

1. For instance, page 1, line 2 (abstract): "...the classes face to face all educational levels."
2. Page 3, lines 83-85. 
3. 'T' in teams is somewhere capitalized and at some places is small. 

Thank you very much for the observation . Formatting correction was made

1. Most paragraphs are either right-aligned or not formatted properly. 
2. Typo on line 304 page 12 - schwool and on

Thank you very much for the observation The correction was made

1. page 13, line 371 - "..than 75% were 75% of a 70..."

Thank you very much for the observation . The correction was made

1. page 15, line 413 - "coVID-19"

Thank you very much for the observation . The correction was made

• Strange way to show the number of students in Table 1.

Thank you very much for the observation . The correction was made in Table 1, See line 27.

• Page 3, line 93: the link for Microsoft Teams should be a footnote.

Thank you very much for the observation . The adjustment was made to the manuscript.

• Table 2 doesn't make much sense to me. "The caption is stages of organization of the virtualized teaching operations" and by looking at the table it's difficult to comprehend the steps followed in each step of the stage. It also refers to the use of Blackboard which is not undertaken in this study. 

Thank you very much for the observation.  Adjustments were made and included comparisons with LMS platforms. See lines 106-110.

• A lot of studies have been carried out on similar lines, where researchers have evaluated various LMS, MOOCs platforms, and teaching tools. For instance, https://ieeexplore.ieee.org/abstract/document/7403480 . How this study compares to the existing work found in the literature and similar studies. 

Thank you very much for the observation Adjustments were made. The suggested reference was included.

• The paper lacks novelty and does not present state-of-the-art or studies alike in this domain. 

Thank you very much for the observation; the topic of Remote Emergency Teaching is a new topic that leads to a virtualization process; there are not many reported cases in the literature. Because of the situation of COVID-19, the use of Virtual Learning Platforms became a topical topic used by most Higher Education Institutions in Mexico. The purpose of the Teams Platform is part of a strategy by the UVM for the virtualization of the educational process; this has been mentioned in lines 100-120.  The use of Data Science tools such as Deep Learning is beginning to be applied in the Higher Education Institutions in Mexico to obtain information that helps in the improvement of educational processes, which is why we consider that the content of this work is innovative. 

• Why in Equation 2 are two jj? 

Thank you very much for the observation. Adjustments were made See line 162.  The number of the equation changed

• Page 6, line 166: authors claim that "the results generated in RapidMinder allow validation of the findings determined in the descriptive study". The question is how? I couldn't find the answer within the document. 

Thank you very much for the observation.  Adjustments were made. Changed the term validation by comparison.

• Equation 3 on page 6 seems broken.

Thank you very much for the observation.  Adjustments were made.

Author Response File: Author Response.pdf

Reviewer 3 Report

I particularly enjoyed reading this paper, since it introduces very important issues most of us faced during the COVID pandemic. The paper follows a postmodern perspective and treats technology not as a neutral medium. It thus explains how the quick shift to online learning during the pandemic had qualitatively changed the educational settings. Personally, I could not agree more with this analysis and I am happy that papers like this emerge. For this reason, I find the objectives of the paper very important.

Line 56: “The main objective in these circumstances is not to recreate 57 a robust educational ecosystem, but rather to provide temporary access and instructional supports in 58 a quick and easy-to-configure way.” I was happy to see this in the paper, since many people do not seem to realize that the pandemic has qualitatively changed the educational needs. People and institutions tried to replace traditional lectures with technology but this fails since technology is not a neutral medium.  I would urge for a little more discussion on this topic at the discussion section, to stress its importance.

The results section provides interesting percentages and frequencies. However, this is a great opportunity to also see possible correlations between different factors (e.g. gender, personality, stress levels, etc). If you have such data, please proceed with correlation analysis. Normally, I would not accept a paper only showing frequencies for a journal publication, but these are exceptional times and the findings raise important questions. 

Author Response

Rewier 3

The results section provides interesting percentages and frequencies. However, this is a great opportunity to also see possible correlations between different factors (e.g. gender, personality, stress levels, etc). If you have such data, please proceed with correlation analysis. Normally, I would not accept a paper only showing frequencies for a journal publication, but these are exceptional times and the findings raise important questions. 

Thank you very much for the observation.  Adjustments were made. The correlation were incluiding. See lines 211-212 and 269-277.

Author Response File: Author Response.pdf

Reviewer 4 Report

This is an informative, interesting and well argued analytical paper, demonstrating a very good knowledge of the topic. However, to improve the level of analysis, minor revisions are suggested:

1. Add theoretical framework/methodology section after the introduction (1 page?)
2. Expand information on assessment
3. Present a better conclusion (Most of it belongs to the discussion section). It is suggested that conclusion could start with 'The most significant result... (line 364, p. 13)and stress the other significant findings.

Author Response

Reviewer 4

This is an informative, interesting and well argued analytical paper, demonstrating a very good knowledge of the topic. However, to improve the level of analysis, minor revisions are suggested:

• Add theoretical framework/methodology section after the introduction (1 page?)
• Expand information on assessment
• Present a better conclusion (Most of it belongs to the discussion section). It is suggested that conclusion could start with 'The most significant result... (line 364, p. 13)and stress the other significant findings.

Thank you very much for the retraction, the adjustments were made in the abstract, the introduction and the conclusions.

The topic of Remote Emergency Teaching is a new topic that leads to a virtualization process; there are not many reported cases in the literature. Because of the situation of COVID-19, the use of Virtual Learning Platforms became a topical topic used by most Higher Education Institutions in Mexico. The purpose of the Teams Platform is part of a strategy by the UVM for the virtualization of the educational process; this has been mentioned in lines 100-120.  The use of Data Science tools such as Deep Learning is beginning to be applied in the Higher Education Institutions in Mexico to obtain information that helps in the improvement of educational processes, which is why we consider that the content of this work is innovative. 

Author Response File: Author Response.pdf