A Case Study on English as a Second Language Speakers for Sustainable MOOC Study
Round 1
Reviewer 1 Report
Dear Authors,
Thank you for the opportunity to review your paper. I found your approach to categorizing learners in a MOOC based on their native language quite interesting. I don't have many comments for your paper because I thought it was overall well designed and well written, and the findings were actually very interesting.
The one question I have is this: on page 15 (lines 352-356) you describe your three categories. Are "completers" people who actually completed the course (i.e., got to the end, and 'completed' all required weeks)? Or is it people from completed at least half of the course. From the writeup it seems that it was people who completed 50% or more of the course. If this is the case, why did you pick the 50% and above to mark someone as a completer? It may be beneficial to separate those who completed the course (were eligible to buy a certificate of completion) from those who completed a good portion of the work of the course, in order to be more precise.
Author Response
Reviewer 1 Comment(s) | Response | Pages and lines affected |
R1.1 | ||
The one question I have is this: on page 15 (lines 352-356) you describe your three categories. Are "completers" people who actually completed the course (i.e., got to the end, and 'completed' all required weeks)? Or is it people from completed at least half of the course. From the writeup it seems that it was people who completed 50% or more of the course. If this is the case, why did you pick the 50% and above to mark someone as a completer? It may be beneficial to separate those who completed the course (were eligible to buy a certificate of completion) from those who completed a good portion of the work of the course, in order to be more precise. | A) Thank you for your review. We have this definition obtained from FutureLearn's website. Please refer to this link: https://about.futurelearn.com/research-insights/learners-learning-know
| |
B) A small explanation is added as "This criteria has been set by FutureLearn. According to FutureLearn, people who completed more than 50% are active users and classified as Completed Learner. People who completed over 90% are qualified to certificate are also classified as Completed Learners by FutureLearn. Since the number of certificate bought students is very low, we have also merged these two group all together as Completers in our study for the sake of the more accurate algorithm performance." | Line 383-388 on Page 15 |
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The manuscript is of potential interest to the readership of this journal, but there are major issues that must be addressed before the article could be published.
1/ * The literature review should be more carefully synthesised and structured. The use of sub-headings and signposting would help the reader to follow the argument being developed through the paper.
2/ * The results section requires far greater organisation and structuring. The analysis is too general, and the reported results are somewhat selective. This section needs to be more carefully and systematically constructed.
3/ * The final discussion and conclusion should make it clear how the findings contribute to new knowledge.
4/* Plagiarism check revelead a 9% similarity.
5/* More recent bibliography is necessary. Furthermore, the reference list is a little bit weak. Before I can make a final decision on the paper, please refer to more references and upload a new version. It is suggested that the author(s) can consider the following paper related to the understanding student engagement as a course progresses and thus can help minimise dropout rates by prompting focused instructor intervention to strengthen the background and conclusions of the study:
Papadakis, S., Kalogiannakis, M., Sifaki, E., & Vidakis, N. (2018). Access Moodle Using Smart Mobile Phones. A Case Study in a Greek University. In A. Brooks, E. Brooks, N. Vidakis (Eds). Interactivity, Game Creation, Design, Learning, and Innovation. ArtsIT 2017, DLI 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 229, (pp. 376-385), Switzerland, Cham: Springer.
6/* The academic writing needs work.
7/* The discussion should be more concise and the outcomes should be discussed in relation to the existing research.
8/* Recommendations should also be given for practice and further research.
9/* In preparing a revised manuscript, please also include a table of how you have responded to each of the issues listed above point by point.
I look forward to receiving your revised manuscript in the near future.
Author Response
Response to Reviewer 2 Comments
Thank you for your time and the constructive reviews on our paper. We have done the corrections and we believe that it has improved our paper accordingly.
Table 2: Reviewer 2 Comment(s) and Responses
Reviewer 2 comment(s) | response | Pages and lines affected |
R1.1 | ||
The literature review should be more carefully synthesised and structured. The use of sub-headings and signposting would help the reader to follow the argument being developed through the paper. | A) We understand the literature review is long and complicated. The related studies to our paper include learning analytics and research on language and MOOCs | |
B)Therefore we have added two sub-titles as 2.1 Learning Analytics in MOOCs and 2.2. Engagement in MOOCs from Language Perspective . | Lines 100 and Lines 120 on Page 3 | |
R1.2 | ||
The results section requires far greater organisation and structuring. The analysis is too general, and the reported results are somewhat selective. This section needs to be more carefully and systematically constructed. | A) The participants in each category have three distinct behaviours on the platform: i) studying and completing learning steps in each week, ii) contributing to discussions by posting comments, and iii) following other participants. We have three subheadings for each of these categories in this section (Section 5). | |
B) As we explained in the methodology section, we have done descriptive analysis on the course engagement of participants in each group. We aimed to see differences between their engagements. Therefore, we selected the behaviours that we can see the differences among the groups. | ||
C) Even though we would like to add more analysis from different perspectives, the generated data by FutureLearn is not always structured for this. For example, we wanted to make analysis on the patterns of comment liking in detail. However, FutureLearn only collects the number of likes each comment received. They do not collect who liked whom, when and so on. Therefore, the analysis we have done sometimes remained too general due to the data availability. | ||
D) To make the design of the section clearer, we have added the following paragraph. “The analysis step of the research has been conducted to answer the second research question: Is there any difference between the behaviours of English as a second language participants and the other participants in completing the course, contributing to the discussions, and interacting with each other? The data generated by FutureLearn allows us to track participants’ engagement with • the course steps i.e. when a learner opened the page of the step and marked as completed; • the discussion forum i.e. what a learner posted to discussion threads; • the followings feature i.e. whom a learner started following another learner. Therefore, we have divided the section into three subsections to analyse the above mentioned behaviours respectively.” |
Lines 269-272 on Page 8
Lines 273-275 on Page 8
Lines 276-277 on Page 8
| |
R.1.3 | ||
The final discussion and conclusion should make it clear how the findings contribute to new knowledge. | To make clear the contribution of our research, the following sentences have been added to A) “Regular expressions have been used in MOOCs to extract hashtags and keywords from text [45,46]. Our study proposed a novel method using regular expressions to identify someone's first language from their comment in discussions.” |
Lines 510-512 on Page 20 |
B) “Differently from the existing literature, our study analysed the distinguish differences among the behaviours of participants diverse in first language.” | Lines 515-516 on Page 20 | |
C) “The overall attendance in the course supports the findings from previous studies [40] as the attendance steadily decreased over the weeks and completion of the course remained low. However, our findings showed that there is a difference in attendance and completion of the course among the language-based groups.” | Lines 520-523 on Page 20 | |
R.1.4 | ||
Plagiarism check revelead a 9% similarity. | A) As far as we are concerned, this is an acceptable rate. We have referred to our previous research and other studies by paraphrasing. We declare that we did not self-plagiarise by exact copying from our previous papers or from any other papers. | |
B) We have re-read the paper and tried to improve the sentences that we have referred to other papers. Please let us know if this is not enough. | ||
R.1.5 | ||
A)Thank you for bringing our attention to this article on students’ perception towards the operation of LMSs. We have added this sentences to the paragraph in sub-Section 2.1 Learning Analytics in MOOCs: “A study conducted by Papadakis et al. [12] shows that the presentation of the content and affordances of the platform is also an important factor which may cause reduced engagement. For example, improving mobile application of the course and gamificiation elements to facilitate instructor intervention could have an impact on use for English as a second language speakers.” |
Lines 108-112 on Page 3 | |
More recent bibliography is necessary. Furthermore, the reference list is a little bit weak. Before I can make a final decision on the paper, please refer to more references and upload a new version. It is suggested that the author(s) can consider the following paper related to the understanding student engagement as a course progresses and thus can help minimise dropout rates by prompting focused instructor intervention to strengthen the background and conclusions of the study:
Papadakis, S., Kalogiannakis, M., Sifaki, E., & Vidakis, N. (2018). Access Moodle Using Smart Mobile Phones. A Case Study in a Greek University. In A. Brooks, E. Brooks, N. Vidakis (Eds). Interactivity, Game Creation, Design, Learning, and Innovation. ArtsIT 2017, DLI 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 229, (pp. 376-385), Switzerland, Cham: Springer | B) In order to improve the literature review we have added a number of recent studies.
[1] Almutairi, F. The Impact of Integrating MOOCs into Campus Courses on Student Engagement. PhD thesis, University of Southampton, 2018.
[2] Liyanagunawardena, T.R.; Williams, S.; Adams, A.A. The impact and reach of MOOCs: a developing countries’ perspective. eLearning Papers 2014, pp. 38–46
[12] Papadakis, S.; Kalogiannakis, M.; Sifaki, E.; Vidakis, N. Access moodle using smart mobile phones. A case study in a Greek University. In Interactivity, Game Creation, Design, Learning, and Innovation; Springer, 2017; pp. 376–385.
[21] Wu, W.; Bai, Q. Why Do the MOOC Learners Drop Out of the School?–Based on the Investigation of MOOC Learners on Some Chinese MOOC Platforms. 2018 1st International Cognitive Cities Conference (IC3). IEEE, 2018, pp. 299–304
[22] Wang, Q.; Chen, B.; Fan, Y.; Zhang, G. MOOCs as an Alternative for Teacher Professional Development. Examining Learner Persistence in One Chinese MOOC 2018.
[23] “Beaven, T.; Codreanu, T.; Creuzé, A. Motivation in a language MOOC: issues for course designers. In Language MOOCs: Providing Learning, Transcending Boundaries., 2014,48-66”
[29] Fuchs, C. The Structural and Dialogic Aspects of Language Massive Open Online Courses (LMOOCs): A Case Study. In Computer-Assisted Language Learning: Concepts, Methodologies, Tools, and Applications; IGI Global, 2019; pp. 1540–1562. [30] “Martín-Monje, E.; Castrillo, M.D.; Mañana-Rodríguez, J. Understanding online interaction in language MOOCs through learning analytics. Computer Assisted Language Learning 2018,31, 251–272.”
[34] “Guo, P.J. Non-native English speakers learning computer programming: Barriers, desires, and design opportunities. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 2018, p. 396.”
[43] Calvo, S.; Morales, A.; Wade, J. The use of MOOCs in social enterprise education: an evaluation of a North–South collaborative FutureLearn program. Journal of Small Business & Entrepreneurship 2019, 31, 201–223.
[44] Castillo, N.M.; Lee, J.; Zahra, F.T.; Wagner, D.A. MOOCS for development: Trends, challenges, and opportunities. International Technologies & International Development 2015,11, 35.
[45] Acosta, E.S.; Otero, J.J.E. Automated assessment of free text questions for MOOC using regular expressions. Information Resources Management Journal (IRMJ) 2014,27, 1–13.
[46] An, Y.H.; Chandresekaran, M.K.; Kan, M.Y.; Fu, Y. The MUIR Framework: Cross-Linking MOOC Resources to Enhance Discussion Forums. International Conference on Theory and Practice of Digital Libraries. Springer, 2018, pp. 208–219.
[47] Jiang, S.; Williams, A.; Schenke, K.; Warschauer, M.; O’Dowd, D. Predicting MOOC performance with Week 1 behavior. Proceedings of the 7th International Conference on Educational Data Mining 2014, 2014. |
Line 23 on Page 1 Line 609 in Ref
Line 27 on Page 1 Line 483 on Page 23 Line 611 in Ref
Line 108 on Page 3 Line 636 in Ref
Line 127 on Page 4 Line 660 in Ref
Line 127 on Page 4 Line 663 in Ref
Line 121 on Page 4 Line 665 in Ref
Line 131 on Page 4 Line 681 in Ref
Line 131 on Page 4 Line 684 in Ref
Line 133 on Page 4 Line 693 in Ref
Line 478 on Page 19 Line 716 in Ref
Line 489 on Page 19 Line 719 in Ref
Line 510 on Page 20 Line 721 in Ref
Line 510 on Page 20 Line 723 in Ref
Line 546 on Page 21 Line 726 in Ref |
R.1.6 | ||
The academic writing needs work | A) Proofreading has been performed. | |
R.1.7 | ||
The discussion should be more concise and the outcomes should be discussed in relation to the existing research. | A) The following sentences are added to Section 7 Discussion and Conclusion. “A more recent study where findings were based on data from international participants in a MOOC focussed on social enterprise education, Calvo et al. [43] specifically identify linguistic and cultural barriers as inhibiting learner’s access to MOOCs. Much of the early speculative literature which promoted the potential for MOOCs highlighted the value of free and open education. Subsequent attitudinal and implementation studies have revealed barriers to accessibility frequently focusing on cultural and linguistic aspects, Whilst it may be evident that a large segment of MOOC participants is drawn from developing countries [2] work remains to be done by MOOC providers to enhance the effective usefulness of this growing set of rich educational resources. Finding ways to automatically identify key features associated with learners (such as their approximate linguistic backgrounds) offers a way for MOOC platform providers and course authoring teams to realistically consider broad brush approaches to personalisation. Furthermore this approach could also potentially be used to provide data to enable effective localisation, the need for which has been identified by Castello et al.[44]. Our reasoning for focusing on English language competencies was based on the observation that a considerable proportion of MOOCs at present are conducted in the English language, coupled with an understanding that socially active learners (those who are involved participants in online discussion based tasks and exercises) are most likely to complete the course [42]. Our research has analysed the social engagement and course completion performance of participants categorised by first language groupings. We find that participants whose first language is English are able to make more active use of the platform and are most likely to complete the course. This inequality can potentially be addressed if we are able to successfully identify learners with other linguistic markers and provide tailored support or customised interventions to narrow this achievement gap. The research presented in this paper identifies some initial steps that could contribute towards such an approach.” |
Lines 477-499 on Pages 19-20 |
B) Line 515-519 “Finding the behaviours in the first week correlated to completion is previously confirmed by the study of Jiang et al. [47]. Differently, their study shows that social integration with a learning community in Week 1 is strongly correlated while the findings of our study shows that the engagement with the course steps in Week 1 is the most predictive.” | Lines 544-548 on Page 21 | |
R.1.8 | ||
Recommendations should also be given for practice and further research | A)In Section 8 Future Work, we have discussed some future research directions. For example on Page 21, “This study was undertaken within a single MOOC (the fourth FutureLearn Understanding Language: Learning and Teaching). It would be particularly interesting to replicate the approach (1) with larger data sets from successive run of the same MOOC, (2) across a range of disciplines. For example, we would like to know whether successful behaviours in different discipline groups conform to the same predictive patterns that we have identified in this study.” | |
B) Recommendations for practice of our findings could be useful for especially MOOC authors and platform. We have added the following paragraph at the end of Section 8 Future Work: “Apart from these future research directions, the findings from our research could be used by the MOOC providers (authors and platform creators) for re-designing their courses and platforms where English as a second language speakers would benefit more efficiently. The findings show that the participants whose first language is English more likely to complete the more numbers of steps than others and they more actively engaged in the discussions. Preparing a MOOC and a platform which is encouraging for English as a second language speakers to more actively attend to discussions may cause for them to complete more numbers of the steps.”. |
Lines 590-596 on Page 21-22
| |
R.1.9 | ||
In preparing a revised manuscript, please also include a table of how you have responded to each of the issues listed above point by point. | A) We have provided a cover letter addressing our changes in general and provided this additional file created for answering your reviews one by one. | |
B) We have also highlighted yellow the changes in the main text. Hope you find it helpful. |
Author Response File: Author Response.pdf
Reviewer 3 Report
Overall, the format of the article looks like a technical report for course evaluation. The manuscript lacks sufficient academic requirement. Some suggestions are:
The Figure 1 is unnecessary. No need to report the structure.
How was the topic related to "Sustainable" issue? No specific rationale can be provided.
What is the research method used in the study? Case study?
Current statical information only report the descriptive results. No advanced findings can be obtained. For example, what is significant difference in terms of gender?
Author Response
Response to Reviewer 3 Comments
Thank you for your time and the constructive reviews on our paper. We have done the corrections and we believe that it has improved our paper accordingly.
Table 3: Reviewer 3 Comment(s) and Responses
Reviewer 3 comment(s) | response | Pages and lines affected |
R1.1 | ||
The Figure 1 is unnecessary. No need to report the structure. | A) Since it is a long paper, we have thought a navigative figure could help the reader for an easy reading. | |
B) Since you pointed out that it is unnecessary, we have removed it. | Page 3 | |
R1.2 | ||
How was the topic related to "Sustainable" issue? No specific rationale can be provided. | A) This paper has been submitted to the Special Issue on Technology-Enhanced Learning: Applications, Architectures and Challenges of the Sustainability. We have now highlighted ways in which this aspect of TEL has important implications with respect to sustainable approaches to education. | |
B) Our research is designed to tackle the challenges in identifying the needs of English as a second language speaker participants in MOOCs. To achieve this, we have proposed an automated method for identifying these English as a second language speakers and identify the pattern of their engagements in this manuscript. According to the statistics, 75% of the MOOC content is authored in English. Considering the large proportion of international participants, we believe that this research is valuable to contribute to the research on sustainable online education and is highly related to the scope of the journal. | ||
C) “MOOCs are increasingly recognised as providing high-value learning resources enabling an accessible route to sustaining the expansion of both formal and informal education. Formally, blended MOOCs provide a means whereby academics can incorporate externally produced resources into their face-to-face teaching. This is being used as a means to rapidly expand and grow educational capacity - for example in women’s universities in Saudi Arabia [1]. Informally, individual learners typically use MOOCs to access educational independently and to update professional skills and gain access to education for little or no cost. However the use of MOOCs in developing Countries is not as straightforward as some might assume with language barriers having been identified as one of the important problem areas [2]. The focus of this paper working on methods to automatically identify different types of learners by their language background, and then looking at means to predict likely learning pathways is particularly relevant to these contexts.” | Lines 19-29 on Page 1 | |
D) MOOCs are a promising application for the future of education. Even though there are many arguments on that, we believe that MOOCs are a good opportunity to reach people from different educational and socio-economic background and could be used for providing equal educational opportunities for everyone around the world. However, MOOCs have different problems needed to be solved for a quality online education. As we have addressed in the first paragraph in Lines 42-43 on Page 2, the large proportion of international participants raises the issue on possible difficulties with communicating in English for those to whom English is their second language. Therefore, detecting the second language English speakers, understanding their behaviours, and meeting their needs are very crucial for improving the MOOC education. | ||
E) We have also stated on Page 19-20, Section 7 Discussion and Conclusion that “A more recent study where findings were based on data from international participants in a MOOC focussed on social enterprise education, Calvo et al. [43] specifically identify linguistic and cultural barriers as inhibiting learner’s access to MOOCs. Much of the early speculative literature which promoted the potential for MOOCs highlighted the value of free and open education. Subsequent attitudinal and implementation studies have revealed barriers to accessibility frequently focusing on cultural and linguistic aspects, Whilst it may be evident that a large segment of MOOC participants is drawn from developing countries [2] work remains to be done by MOOC providers to enhance the effective usefulness of this growing set of rich educational resources. Finding ways to automatically identify key features associated with learners (such as their approximate linguistic backgrounds) offers a way for MOOC platform providers and course authoring teams to realistically consider broad brush approaches to personalisation. Furthermore this approach could also potentially be used to provide data to enable effective localisation, the need for which has been identified by Castello et al.[44]. Our reasoning for focusing on English language competencies was based on the observation that a considerable proportion of MOOCs at present are conducted in the English language, coupled with an understanding that socially active learners (those who are involved participants in online discussion based tasks and exercises) are most likely to complete the course [42]. Our research has analysed the social engagement and course completion performance of participants categorised by first language groupings. We find that participants whose first language is English are able to make more active use of the platform and are most likely to complete the course. This inequality can potentially be addressed if we are able to successfully identify learners with other linguistic markers and provide tailored support or customised interventions to narrow this achievement gap. The research presented in this paper identifies some initial steps that could contribute towards such an approach.” | Lines 477-499 on Pages 19-20 | |
R1.3 What is the research method used in the study? Case study? | A) Yes, this is a case study implemented in Understanding Language: Learning and Teaching from the FutureLearn MOOC platform with 25,598 participants. We explicitly stated it in the title of the paper. | |
B) To further clear this point, we have added the following sentence in Section 3 Research Aim and Methodology on Page 4. “This research has been designed as a case study which performs series of analyses by using the data generated from an Understanding Language: Learning and Teaching MOOC on FutureLearn.” |
Lines 164-165 on Page 4 | |
R1.4 Current statical information only report the descriptive results. No advanced findings can be obtained. For example, what is significant difference in terms of gender? | A) As the steps identified in Figure 1 titled “Four steps to identify effective study strategies”, this paper is organised to share the descriptive statistical analysis and prediction models. Therefore, we have not implemented any inferential statistical analysis for this study. | |
B) Even though we have not included a special section for inferential statistical analysis such as showing the statistical correlation between a social behaviour, a language group, and completion, Table 6 on Page 17 shows that the certain social behaviours are strongly predictive (according to the result of algorithms) to the course completion behaviours. Table 6 also shows that the order of predictive features is different for each language-based group. | ||
C) This research focuses on the study performances of different language groups. We are interested in significant differences among the people whose first language is different, rather than their gender. Therefore, we did not perform any analysis on gender. That might be a topic for future work. |
Author Response File: Author Response.pdf
Round 2
Reviewer 3 Report
Although the previous format of the study did not meet my needs, the current format of the manuscript already went through extensive modification.
