# The Accurate Measurement of Students’ Learning in E-Learning Environments

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Background

#### 2.1. The Needs for a Common Metric in MOOCs

#### 2.2. Item Response Theory

#### 2.3. Fixed-Parameter Calibration

## 3. Current Study

- RQ1: Does a common metric make meaningful changes in students’ abilities?
- RQ2: Does a common metric make meaningful changes in students’ gain scores?
- RQ3: Are the IRT scores from FPC more robust to missing data than percentage scores?

## 4. Methods

## 5. Data Analysis

## 6. Results

#### 6.1. Percentage Scores

#### 6.2. Percentage Scores vs. IRT Scores

## 7. Discussion

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Kennedy, J. Characteristics of massive open online courses (MOOCs): A research review, 2009–2012. J. Interact. Online Learn.
**2014**, 13, 1–16. [Google Scholar] - Zhu, M.; Sari, A.; Lee, M.M. A systematic review of research methods and topics of the empirical MOOC literature (2014–2016). Internet High. Educ.
**2018**, 37, 31–39. [Google Scholar] [CrossRef] - Reich, J. Rebooting MOOC Research. Science
**2015**, 347, 34–35. [Google Scholar] [CrossRef] [PubMed] - de Barba, P.; Kennedy, G.; Ainley, M. The role of students’ motivation and participation in predicting performance in a MOOC. J. Comput. Assist. Learn.
**2016**, 32, 218–231. [Google Scholar] [CrossRef] - Kulkarni, C.E.; Bernstein, M.S.; Klemmer, S.R. PeerStudio. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, Vancouver, BC, Canada, 14–18 March 2015. [Google Scholar] [CrossRef] [Green Version]
- Marin, V.J.; Pereira, T.; Sridharan, S.; Rivero, C.R. Automated Personalized Feedback in Introductory Java Programming MOOCs. In Proceedings of the 2017 IEEE 33rd International Conference on Data Engineering (ICDE), San Diego, CA, USA, 19–22 April 2017. [Google Scholar] [CrossRef]
- Meyer, J.P.; Zhu, S. Fair and equitable measurement of student learning in MOOCs: An introduction to item response theory, scale linking, and score equating. Res. Pract. Assess.
**2013**, 8, 26–39. [Google Scholar] - Kim, S. A Comparative Study of IRT Fixed Parameter Calibration Methods. J. Educ. Meas.
**2006**, 43, 355–381. [Google Scholar] [CrossRef] - Huggins, A.C.; Penfield, R.D. An NCME Instructional Module on Population Invariance in Linking and Equating. Educ. Meas. Issues Pract.
**2012**, 31, 27–40. [Google Scholar] [CrossRef] - Kolen, M.J.; Brennan, R.L. Test Equating, Scaling, and Linking; Springer: New York, NY, USA, 2014. [Google Scholar] [CrossRef]
- Dugard, P.; Todman, J. Analysis of Pre-test-Post-test Control Group Designs in Educational Research. Educ. Psychol.
**1995**, 15, 181–198. [Google Scholar] [CrossRef] - Alario-Hoyos, C.; Estévez-Ayres, I.; Pérez-Sanagustín, M.; Kloos, C.D.; Fernández-Panadero, C. Understanding Learners’ Motivation and Learning Strategies in MOOCs. Int. Rev. Res. Open Distrib. Learn.
**2017**, 18. [Google Scholar] [CrossRef] - Seaton, D.; Hansen, J.; Goff, J.; Sellers, P.; Davidson, N.; Houck, A. Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models. Digital Inclusion: Transforming Education Through Technology. In Proceedings of the LINC (Learning International Networks Consortium), Cambridge, MA, USA, 23–25 May 2016; p. 237. [Google Scholar]
- Rogers, C.F. Faculty perceptions about e-cheating during online testing. J. Comput. Sci. Coll.
**2006**, 22, 206–212. [Google Scholar] - Underwood, J.; Szabo, A. Academic offences and e-learning: Individual propensities in cheating. Br. J. Educ. Technol.
**2003**, 34, 467–477. [Google Scholar] [CrossRef] - Rowe, N.C. Cheating in online student assessment: Beyond plagiarism. Online J. Distance Learn. Adm.
**2004**, 7. Available online: https://www.westga.edu/~distance/ojdla/summer72/rowe72.html (accessed on 1 September 2021). - Embretson, S.E.; Reise, S.P. Item Response Theory for Psychologists; Psychology Press: New York, NY, USA, 2013. [Google Scholar]
- Harris, D. Comparison of 1-, 2-, and 3-Parameter IRT Models. Educ. Meas. Issues Pract.
**1989**, 8, 35–41. [Google Scholar] [CrossRef] - De Ayala, R.J. The Theory and Practice of Item Response Theory; Guilford Publications: New York, NY, USA, 2013. [Google Scholar]
- Hambleton, R.K.; Jones, R.W. Comparison of classical test theory and item response theory and their applications to test development. Educ. Meas. Issues Pract.
**1993**, 12, 38–47. [Google Scholar] [CrossRef] - Cook, L.L.; Eignor, D.R. IRT Equating Methods. Educ. Meas. Issues Pract.
**1991**, 10, 37–45. [Google Scholar] [CrossRef] - Jodoin, M.G.; Keller, L.A.; Swaminathan, H. A Comparison of Linear, Fixed Common Item, and Concurrent Parameter Estimation Equating Procedures in Capturing Academic Growth. J. Exp. Educ.
**2003**, 71, 229–250. [Google Scholar] [CrossRef] - Ban, J.C.; Hanson, B.A.; Wang, T.; Yi, Q.; Harris, D.J. A Comparative Study of On-line Pretest Item-Calibration/Scaling Methods in Computerized Adaptive Testing. J. Educ. Meas.
**2001**, 38, 191–212. [Google Scholar] [CrossRef] - Stocking, M.L. Scale drift in on-line calibration. ETS Res. Rep. Ser.
**1988**, 1988, i-122. [Google Scholar] - Wainer, H.; Mislevy, R.J. Item response theory, item calibration, and proficiency estimation. Comput. Adapt. Test. A Primer
**1990**, 4, 65–102. [Google Scholar] - Bergner, Y.; Colvin, K.; Pritchard, D.E. Estimation of ability from homework items when there are missing and/or multiple attempts. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, Poughkeepsie, NY, USA, 16–20 March 2015. [Google Scholar] [CrossRef]
- Mislevy, R.J.; Wu, P.K. Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. ETS Res. Rep. Ser.
**1996**, 1996, i-36. [Google Scholar] [CrossRef] - Koedinger, K.R.; Kim, J.; Jia, J.Z.; McLaughlin, E.A.; Bier, N.L. Learning is Not a Spectator Sport. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, Vancouver, BC, Canada, 14–18 March 2015. [Google Scholar] [CrossRef] [Green Version]
- Koedinger, K.R.; McLaughlin, E.A.; Jia, J.Z.; Bier, N.L. Is the doer effect a causal relationship? In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, Edinburgh, UK, 25–29 April 2016. [CrossRef] [Green Version]
- Robitzsch, A.; Kiefer, T.; Wu, M. TAM: Test Analysis Modules; R Package Version 3.7-16; The R Foundation: Vienna, Austria, 2021. [Google Scholar]

Final-Pre | ||||
---|---|---|---|---|

Negative | Positive | Total | ||

Final-Pre | Positive | 140 | 209 | 349 |

Negative | 66 | 0 | 66 | |

Total | 206 | 209 | 415 |

M(SD) for Pre | M(SD) for Post | Mean Difference | t | p-Value | |
---|---|---|---|---|---|

Percentage scores | 55.204 (17.819) | 77.562 (16.620) | 22.358 | 20.325 | <0.001 |

IRT scores | 0.079 (0.803) | 0.093 (0.942) | 0.013 | 0.246 | 0.805 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Lee, S.; Choi, Y.-J.; Kim, H.-S.
The Accurate Measurement of Students’ Learning in E-Learning Environments. *Appl. Sci.* **2021**, *11*, 9946.
https://doi.org/10.3390/app11219946

**AMA Style**

Lee S, Choi Y-J, Kim H-S.
The Accurate Measurement of Students’ Learning in E-Learning Environments. *Applied Sciences*. 2021; 11(21):9946.
https://doi.org/10.3390/app11219946

**Chicago/Turabian Style**

Lee, Sunbok, Youn-Jeng Choi, and Hyun-Song Kim.
2021. "The Accurate Measurement of Students’ Learning in E-Learning Environments" *Applied Sciences* 11, no. 21: 9946.
https://doi.org/10.3390/app11219946