The Use of Educational Process Mining on Dropout and Graduation Data in the Curricula (Re-)Design of Universities

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

I am not certain I agree with the emphasis (both in the abstract and in l. 26) about students dropping out being a "potential loss of money for higher education facilities": 1) if students pay a tuition and then do not make use of the university facilities they could actually contribute positively to the bottom line; 2) if the main source of income of the university is public funding (or the only, as it would be in Austria), this would depend on the rules established for the fundings; 3) even if it were a problem, the most obvious solution to solve the loss of money part would be to lower standards. The cited literature survey [3] writes: "Dropout [rate] negatively affects institutions in the reduction of enrolment and the non-achievement of institutional objectives," which makes much more sense to me, especially if you refer to it to justify the problem being the loss of money.

At no point it is stated if students that transfer to another university (or even another degree programme) count as having dropped out: I suppose they do, but it should be mentioned. It would be also useful if some kind of estimates could be given on the number of students who are actually not dropping out but just trasferring out: at the very least, you should have data on the number of students that transfer into Linz from somewhere else.

In general, the paper does not explain the Austrian system in enough details and many things remain obscure to me:

I really do not understand what is a diploma (l. 181-185): just a 5-year programme? It could also be useful to non European readers to state that the bachelor programme is a 3-year one and the master a 2-year one.

From the description (l. 147-161 is not clear how many times students can sit an exam per year: once as in the US? twice as in France? many times as in Italy?

Is enrolment free, do you have numerus clausus, do students take some sort of orientation test? This might be obvious to any Austrian, but it is not for anyone else, and it makes it a bit more difficult to follow the argument.

l. 24-25: 12% refers to Austrian students or is an OECD average?

l.131: status as dropout/graduate was previously defined only in the abstract; it should be redefined at the first occurrence 

l. 193 you should explain the color coding

l. 240 why only data from 10 academic years should be included? I'm sure there's a valid reason, but I feel it should be at least hinted to.

l. 270-271 (the graphs): you should translate the labels (as you did in some of the accompanying material). I can assume that numbers refer to the number of traces, but it should be stated somewhere (l. 190, perhaps?). In generale, the graphs would benefit from some more thorough explanation of what is going on.

l. 390 Indeed, a lot is missing when trying to understand why students are dropping out: do not have access at the very least to the fact that students failed an exam? Once you have determined that (repeated?) failure in one give exam is the exit point for many students dropping out, it could also be useful, since students have a lot of freedom, to investigate the paths followed by students dropping out. For example, given my experience, I'd expect many students dropping out of a soft science degree to do so after failing calculus many times, and to do so after having taken a lot of other exams (indeed, if I had such a tool at my university, that'd be exactly the first thing I'd look into).

Conclusion

Before the paper could be accepted, I feel two main points have to be addressed:
1) Explain better the Austrian setting
2) Explain better how to read the graphs.

 

Comments on the Quality of English Language

The main issue with the English is the confusion between dropout which is a name, and the verb to drop out: the author(s) makes often use of a non existing verb to dropout. 

l. 175: should be independently not independent

l. 202 does not appear to be in proper English

 

Author Response

 Dear reviewer.

Thank you very much for your review. I tried to address the issues mentioned in your answer. If something is still lacking or too short, please let me know, so that I can adjust the paper accordingly or extend certain points further. Below, I will briefly discuss the major changes and additions based on your comments.

Kind regards

___________

Language issues:

I tried to address the language issues by checking the paper with a language program. I changed what was suggested and what felt odd (in terms of dropout and to drop out). If there is anything left, please let me know.

____________

I am not certain I agree with the emphasis (both in the abstract and in l. 26) about students dropping out being a "potential loss of money for higher education facilities": 1) if students pay a tuition and then do not make use of the university facilities they could actually contribute positively to the bottom line; 2) if the main source of income of the university is public funding (or the only, as it would be in Austria), this would depend on the rules established for the fundings; 3) even if it were a problem, the most obvious solution to solve the loss of money part would be to lower standards. The cited literature survey [3] writes: "Dropout [rate] negatively affects institutions in the reduction of enrolment and the non-achievement of institutional objectives," which makes much more sense to me, especially if you refer to it to justify the problem being the loss of money.

I made a few changes to the abstract as well as to the introduction, making it less of an emphasis on the loss of money and including other factors as well. Additionally, I added more information on the Austrian system in these sections to explain the argument, why the loss of money can still be a problem in some university systems.

At no point it is stated if students that transfer to another university (or even another degree programme) count as having dropped out: I suppose they do, but it should be mentioned. It would be also useful if some kind of estimates could be given on the number of students who are actually not dropping out but just trasferring out: at the very least, you should have data on the number of students that transfer into Linz from somewhere else.

Unfortunately, there is hardly any internal data on this subject. There was a research project on student monitoring addressing this issue, which added to the paper. However, the results were still not really satisfying and numbers of transitions were low. This is probably because the university is the second biggest general university in Austria and other universities in the city are very specialized (Technical University, Medical University, Music and Performing Arts). Therefore, transitions are not that common, except for students keeping their student status valid until they pass an entrance exam at another institution (e.g., for medicine). 

In general, the paper does not explain the Austrian system in enough details and many things remain obscure to me:

I really do not understand what is a diploma (l. 181-185): just a 5-year programme? It could also be useful to non European readers to state that the bachelor programme is a 3-year one and the master a 2-year one.

I explained the different types of degree programs better.

From the description (l. 147-161 is not clear how many times students can sit an exam per year: once as in the US? twice as in France? many times as in Italy?

I added this as well.

Is enrolment free, do you have numerus clausus, do students take some sort of orientation test? This might be obvious to any Austrian, but it is not for anyone else, and it makes it a bit more difficult to follow the argument.

I included a sentence on this point in the introduction. If you need more information, please let me know. For now, I did not want to write an extensive paragraph solely explaining the Austrian university system, if it is not needed for understanding. Your comments are very important since working in this system for years creates a certain kind of blindness in regards to other university systems. Therefore, one can easily misjudge the amount of information international readers need to understand the backgrounds.

l. 24-25: 12% refers to Austrian students or is an OECD average?

This is an OECD average. I made it clear in the according sentence.

l.131: status as dropout/graduate was previously defined only in the abstract; it should be redefined at the first occurrence 

Done.

l. 193 you should explain the color coding

Done.

l. 240 why only data from 10 academic years should be included? I'm sure there's a valid reason, but I feel it should be at least hinted to.

This was about data stability since database systems were introduced in the mid-2000s. I included an explanation.

l. 270-271 (the graphs): you should translate the labels (as you did in some of the accompanying material). I can assume that numbers refer to the number of traces, but it should be stated somewhere (l. 190, perhaps?). In generale, the graphs would benefit from some more thorough explanation of what is going on.

I programmed a translated version of the tool for the screenshots and swapped them out for the English versions. I addressed the issues of this point in the text by extending the results with an explanation of how to read the graphs.

l. 390 Indeed, a lot is missing when trying to understand why students are dropping out: do not have access at the very least to the fact that students failed an exam? Once you have determined that (repeated?) failure in one give exam is the exit point for many students dropping out, it could also be useful, since students have a lot of freedom, to investigate the paths followed by students dropping out. For example, given my experience, I'd expect many students dropping out of a soft science degree to do so after failing calculus many times, and to do so after having taken a lot of other exams (indeed, if I had such a tool at my university, that'd be exactly the first thing I'd look into).

In the limitations, I included a paragraph addressing this issue. This tool is not the best for displaying failed and passed exams. At the university, other analyses are available giving information on this issue.

Conclusion

Before the paper could be accepted, I feel two main points have to be addressed:
1) Explain better the Austrian setting
2) Explain better how to read the graphs.

Both points should have been addressed. If you feel you still need more details and information to better understand the setting and background, extensions and adjustments can be made.

Thank you very much for the helpful comments!

Reviewer 2 Report

Comments and Suggestions for Authors

Attrition of students is an issue that many universities continually struggle to manage. This article outlines the development of a descriptive educational data mining method that utilised student data from both dropouts and graduates with the purpose of presenting curricula developers with information that can be used to increase student retention. This study will appeal to many higher education personnel who need to access data often at a program level to determine the types of changes that may need to be implemented to address attrition rates. The study is innovative in that it describes the development of innovative interactive and animated process maps that depict student pathways based on their course-taking sequence. The maps will enable curriculum developers to access student maps using real time data, providing up-to-date information on students progress through different courses.

Author Response

Dear reviewer.

Thank you very much for your review.

Kind regards

 

 

Reviewer 3 Report

Comments and Suggestions for Authors

The proposed article is well articulated and structured and appropriately cited.

Some minor revisions are needed with regards to explaining the methodology more thoroughly. It is pointed at lines 98, 99: "This study builds upon the existing frameworks, substantially improving the method", but no further detail on which frameworks, what methods used in the respective frameworks, analytic focus etc. 

The figures are still in German, and a translation into English is needed, at least as a legend.

The presented results should be more elaborated, reflecting more of the data in the figures. 

The discussion part should elaborate more on implications for improving managerial processes with a view to reducing the dropout rate of students. 

The references should be more carefully edited, as some of them lack the year of publication with bold numbers, some italics are not appropriate, etc. 

All in all, the article presents an actual topic in a well-structured manner, pointing out an analytic concern from more than 10 years at Uni Graz. The takeaways for an international reader can be more highlighted, with cautions about transferring to different contexts. 

 

Author Response

Dear reviewer.

Thank you very much for your review. I tried to address the issues mentioned in your answer. If something is still lacking or too short, please let me know, so that I can adjust the paper accordingly or extend certain points further. Below, I will briefly discuss the major changes and additions based on your comments.

Kind regards

__________

Some minor revisions are needed with regards to explaining the methodology more thoroughly. It is pointed at lines 98, 99: "This study builds upon the existing frameworks, substantially improving the method", but no further detail on which frameworks, what methods used in the respective frameworks, analytic focus etc. 

I tried to make it clear what the existing paper did and what the intended improvements are, by extending the section.

The figures are still in German, and a translation into English is needed, at least as a legend.

I programmed a translated version specifically for the paper and included new screenshots. In the appendix, animated versions, also with English texts, can be found.

The presented results should be more elaborated, reflecting more of the data in the figures. 

In the results, I added a paragraph on how to read the graphs and referred to Figures 2 and 3 as examples.

The discussion part should elaborate more on implications for improving managerial processes with a view to reducing the dropout rate of students. 

The main results section is now extended by general implications of the method in an international context as well as what this means for university management and dropout rates.

The references should be more carefully edited, as some of them lack the year of publication with bold numbers, some italics are not appropriate, etc.

I had another look at the references and edited them using the guide on the homepage. It should be better now. Previously, I used the automatic function in Word and transferred the citation style from APA to this one manually, which probably has produced some things that were not in line with the citation style.

All in all, the article presents an actual topic in a well-structured manner, pointing out an analytic concern from more than 10 years at Uni Graz. The takeaways for an international reader can be more highlighted, with cautions about transferring to different contexts. 

See above.

 

 

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

l. 165: it could be useful to add "out of XXX," if you have the data

Comments on the Quality of English Language

l. 29: I believe that it should be "drop out rate", but you might want to double check with a mother tongue

Author Response

Thank you for your second review.

I edited both cases. I added the approximate number of 260-300 students per cohort in molecular biology. I also checked for "drop out rate" and your suggestion should be correct. However, I could not locate the phrase in line 29 using the search function, but in the abstract. I changed it accordingly.

Kind regards