Has the COVID-19 Pandemic Affected Mathematics Achievement? A Case Study of University Students in Social Sciences

Abstract

This study examines the effects of COVID-19-related measures on the mathematics achievement of university students in social sciences in Slovenia. Our particular concern was to compare two student populations (pre-pandemic and pandemic) in terms of factors affecting student performance in mathematics courses. Data were collected over nine consecutive academic years (from 2013–2014 to 2020–2021) and analyzed using two-stage structural equation modelling (SEM). The analyses confirmed that the conceptual model developed before the pandemic was applicable during the pandemic period. For both populations (pre-pandemic and pandemic), mathematics confidence, perceived level of math anxiety, background knowledge from secondary school, and self-engagement in mathematics courses at university were confirmed as factors influencing mathematics achievement. Moreover, both populations perceived the effects of the factors in the same way, and the magnitude of the effects is comparable. The rather high values of determination coefficient for mathematics achievement (greater than 0.66 for both student populations) indicate that the variables “Perceived Level of Math Anxiety” and “Self-Engagement in Mathematics Course at University” together explain a significant proportion of the total variance before and during the pandemic. Consequently, the results of our case study indicated that pandemic measures did not have a significant impact on our students’ mathematics achievement. Although a more in-depth study of a broader sample of academic courses would be needed to confirm our findings, our experience indicates that mathematics courses at the tertiary level of education can be successfully delivered online.

1. Introduction

Since the spring of 2020, the literature addressing adjustments to and consequences of COVID-19-related school closures at different educational levels has expanded rapidly. The sudden transition to distance learning and the period of its implementation affected all aspects of students’ lives. They faced social, psychological, and educational challenges [1,2,3,4,5].

In this paper, we will look at the impact of the COVID-19 pandemic on educational outcomes, or, more specifically, student achievement in mathematics. A review by Hammerstein et al. [6] provides a selection of studies at the primary and secondary education levels with comparative analyses of student achievement before and after COVID-19-related school closures. Among numerous studies published up to the end of April 2021, the authors chose only those (11 studies) that included applied statistical analyses rather than solely reporting percentages and analyzed them in detail. Most of these studies revealed negative effects of COVID-19-related school closures on students’ achievement, and seven of them reported a negative effect on mathematics achievement. Positive effects on mathematics achievement were less frequent and occurred only in connection with the more intensive use of online learning software already familiar to students.

A similar systematic review of school performance among children and adolescents during the COVID-19 pandemic was prepared by Panagouli [7], in which, in addition to student achievement, parents’ and teachers’ observations were also analyzed. The final selection covered 42 papers published by July 2021. Most of the students involved in the research had a significant decrease in mathematics results compared to previous years. Unfortunately, only a few of them maintained their mathematics performance or improved it.

Recently, another survey on the implications of primary to upper-secondary school closures was prepared by Svaleryd et al. [8]. They provided an overview of virus transmission in schools, student learning, and mental health among students. This survey mostly confirmed learning losses and more pronounced learning inequalities. The available evidence identified the distance learning period as more challenging for younger students.

Since our study is oriented to university students, we focused mainly on research dealing with parallels between populations of students pre-pandemic and during the pandemic and their achievements at the tertiary level. After reviewing the available literature, we found a poorer representation of publications at the tertiary level than the primary and secondary levels. This agrees with the findings of Soesanto et al. [9]. In our opinion, there are several reasons for this. Certainly, one reason is the specificity of the study areas and courses, making the findings more difficult to compare and generalize. University students are perceived as autonomous learners responsible for choosing their own learning techniques and monitoring their learning process [10]. In addition, students entering tertiary education are expected to have much more experience and coping skills than students at lower education levels. Coping strategies and adaptation skills are key to successfully adjusting to university challenges [11]. One might therefore expect that there would be fewer effects on academic achievement at the university level compared to the primary and secondary education levels. However, studies dealing with general COVID-19 confinement on students’ performance in higher education reported both negative [12,13,14] and positive [15,16,17] effects.

Studies analyzing the impact of COVID-19, and consequently distance learning, on mathematics achievement at the tertiary educational level are very rare. Perhaps surprisingly, positive or no effects of distance learning on mathematics achievement at the tertiary level have been reported to a greater extent than at lower education levels [16,17,18].

After reviewing the representative literature (see Section 2), we found that much of the current comparative research on mathematics performance between pre-pandemic and pandemic generations of tertiary education students is mainly based on the final course grade. Hence, it is essential to examine also the extent to which the previously identified primary factors influencing students’ mathematics performance changed during the pandemic [17]. Our study compares the pre-pandemic and pandemic generations of students from this perspective.

The primary goal of the present study was to investigate how COVID-19-related measures affected the mathematical performance of university students in social sciences. For this purpose, we performed a case study at the Faculty of Organizational Sciences, University of Maribor, Slovenia, and compared in detail two populations of students, pre-pandemic and during the pandemic, from the perspective of their achievement in mathematics. Such comparison requires systematic in-depth analysis. To comprehensively understand how student performance has changed due to COVID-19-related school closures, it is important to investigate factors that potentially contributed to the changes [17].

The results of our previous research [19] showed that the mathematics achievement of university students in social sciences before the COVID-19 pandemic depended on the following factors: mathematics confidence, math anxiety, background knowledge from secondary school, and engagement in university mathematics courses. It is thus necessary to determine whether these factors are also relevant to the population of students during the pandemic. In other words, we need to verify whether our conceptual model developed in the pre-pandemic period using structural equation modeling (SEM) [19] is applicable to the pandemic period. Such verification requires the use of advanced statistical methods that include verification of both the measurement and the structural model (see Section 4.2). Only after the applicability of the model has been confirmed can the two student populations be compared. For this purpose multi-group structural equation modeling (MG-SEM) [20,21] is used.

The remainder of the paper is organized as follows: first, the results of a relevant literature review are presented. Then, the methodology of our empirical study is explained, and the results are systematically presented and discussed. Finally, the conclusions based on research implications are presented, and the limitations of the study and future research directions are highlighted.

2. Review of Related Literature

Findings on the academic performance of students before and after COVID-19-related primary and secondary school closures were outlined in the Introduction. The present study compares the academic performance of tertiary students pre-pandemic and during the pandemic. Accordingly, a review of the representative literature on the tertiary educational level was prepared. The results of the recently published bibliometric analysis [22] show that the number of scientific papers on online learning in higher education rises exponentially. Qualitative methodologies, particularly case study analysis, dominate the field. structural equation modelling (SEM) was the most used quantitative method, and questionnaires were the most widely used instruments of data gathering [23]. Partial least squares structural equation modelling (PLS-SEM) was used to analyze a questionnaire related to the performance of mathematics teaching during the COVID-19 era in [24]. According to the topic of the present study, special attention was paid to comparative studies (pre-pandemic vs. pandemic) of mathematical achievements.

Alangari [25] systematically reviewed the current articles of STEM (sustainable science, technology, engineering, and mathematics) courses in higher education during COVID-19. Results indicated that faculty members consider the transition from face-to-face to distance learning as effective even though they faced several challenges. Further investigation showed that the application of STEM online learning has helped to increase students’ creativity and the rate of STEM research by both students and faculty members. The findings of [26] indicate that for many mathematics lecturers in higher education, the move to online teaching has been a beneficial experience that will impact their future teaching.

De Paola et al. [12] applied a difference-in-differences identification strategy to compare the academic performance of students in different cohorts (affected or not affected by the health emergency) in several academic areas (science, humanities, social sciences) offered at an Italian public university and confirmed a negative impact of online teaching, mainly on students with a tendency to procrastinate. The negative effect was reflected in the students’ final examination grades. The findings also showed that the effects were particularly detrimental for first-year students, while almost no effect on master’s degree students was found. Kofoed et al. [13] reported lower final grades in the introductory economics course at the United States Military Academy for students in online classes compared to those in in-person classes, especially students with lower ability. Negative effects of online teaching on student performance were also confirmed by Orlov et al. [14] for students enrolled in economics courses at four R1 PhD granting institutions. At the same time, they found that student characteristics, including gender, race, and first-year status, were not significantly associated with the decline in performance during the pandemic semester. Gonzalez et al. [15] compared assessments of students in three subject areas at Universidad Autónoma de Madrid (Spain) before and after COVID-19 confinement. They found that confinement changed the students’ learning strategies to a more continuous habit, improving their efficiency. Therefore, the significant positive effect of COVID-19 confinement on student performance was not surprising.

Extensive research on virtual mathematics teaching during the COVID-19 pandemic was conducted among mathematics students at California State University, Fullerton [18]. The impact of virtual instruction on student course outcomes, including completion rates, passing rates, and course grades, was studied. A comparison of student outcomes between fall 2019 and fall 2020 indicated that the new modality of virtual classes had no impact on aggregated grades for students completing their courses. The course outcome data for all students enrolled in a mathematics course showed that the overall grade point average and success rates for both semesters were essentially the same. Pócsová et al. [16] found that distance learning did not affect higher-achieving students and reduced the number of students who did not pass the Mathematics 1 course at the Technical University of Košice. Student evaluations from 2020–2021 were compared to overall assessments from the last six academic years using descriptive statistics. The evaluations in 2020–2021 turned out to be significantly better than those in the previous years, which can be attributed to the ample online support for distance education. Specifically, the online study materials for the Mathematics 1 course had been active since 2008 and were completed just before the pandemic outbreak. With when request came to change the teaching method from face-to-face to distance learning, the university was well prepared for immediate transfer to the online space. Tomal et al. [17] empirically measured the impact of online teaching on students’ academic performance in 11 science, technology, engineering, and mathematics courses using a Bayesian linear mixed-effects model fitted to longitudinal data. They observed an increase in overall average marks in courses requiring lower-level cognitive skills (including first-year mathematics courses) according to Bloom’s taxonomy of knowledge [27] and a decrease in marks in courses requiring higher-level cognitive skills (those with hands-on lab, coding, and programming components). Rosillo et al. [28] studied the impact of a new teaching experience called “Escape Room” on academic performance in mathematics of pharmacy and nursing students. The escape room activity has not shown significant differences in the results compared with the two previous years.

In addition to the already described negative influence on mathematical background knowledge from secondary school, recent pandemic literature also reports increased levels of math anxiety among university students [29,30,31] and highlights a general positive attitude toward learning, with information and communications technology (ICT) as a crucial element influencing students’ coping with learning mathematics during the pandemic [32].

3. Research Objectives

The main objective of our research was to compare populations of university social science students pre-pandemic and during the pandemic in terms of their achievement in mathematics and the factors influencing this achievement.

In our previous study [19], conducted before the outbreak of COVID-19, we attempted to identify the factors that influence the mathematics achievement of social science students. For this purpose, we developed a conceptual model with seven interdependent factors selected according to the literature review, which are listed in

Table 1. Constructs of conceptual model developed in [19]: pre-pandemic approach.

Model Construct		No. of Items a	Rating Scale	References
Mathematics Confidence (MC)		4	5-point Likert-type scale: 1 (“I do not agree at all”) to 5 (“I agree completely”)	[33,34,35,36]
Behavioral Engagement (BE)		2	5-point Likert-type scale: 1 (“I do not agree at all”) to 5 (“I agree completely”)	[37]
Perceived Level of Math Anxiety (PLMA)	Mathematics Test Anxiety (MTA)	9	5-point Likert-type scale: 1 (“no anxiety”) to 5 (“high anxiety”)	[36,38,39]
	Numerical Task Anxiety (NTA)	4
	Math. Course Anxiety (MCA)	4
Background Knowledge from Secondary School (BKSS)	Grade in math in final year	1	Achieved grade from 2 (sufficient) to 5 (excellent)	[35,40,41,42,43,44,45,46]
	Grade in math at matura b	1
	Final grade in high school	1
Self-Engagement in Math. Course at Univ. (SEMCU)	e-Activities	1	% of points earned (0–100%)	[46,47,48]
	Additional points	1	Number of points earned (0–13)
Confidence with Technology (CT)		4	5-point Likert-type scale: 1 (“I do not agree at all”) to 5 (“I agree completely”)	[37]
Perceived Usefulness of Technology in Learning Mathematics (PUTLM)		4	5-point Likert-type scale: 1 (“I do not agree at all”) to 5 (“I agree completely”)	[37,49,50,51]
Mathematics Achievement (MA)		1	% of points achieved on final exam

^a Five measured items (BE1, BE2, MTA8, NTA1, and MCA1) were omitted from the original questionnaire due to factor loadings below 0.5 (see [19] for details). ^b Matura is the final national school-leaving exam in Slovenia.

.

To determine which of the seven proposed factors are influential for predicting mathematics achievement in the populations of students under consideration, seven hypotheses were formulated.

Table 2. Summary of hypotheses testing for structural model: pre-pandemic approach [19].

Hypothesis	Path	Expected Sign	Hypothesis Supported?
H1a	MC → PLMA	−	Yes
H1b	BE → PLMA	−	No
H2	PLMA → MA	−	Yes
H3	BKSS → SEMCU	+	Yes
H4	SEMCU → MA	+	Yes
H5	CT → PUTLM	+	Yes
H6	PUTLM → MA	+	No

shows that five of them were supported and two were rejected. The results indicate that the constructs Behavioral Engagement (BE), Perceived Usefulness of Technology in Learning Mathematics (PUTLM) (and consequently Confidence with Technology (CT)) had no significant influence on the mathematical achievement of the student population pre-pandemic. Consequently, the constructs Mathematics Confidence (MC), Perceived Level of Math Anxiety (PLMA), Background Knowledge from Secondary School (BKSS), and Self-Engagement in Mathematics Course at University (SEMCU) were found to have a significant impact and therefore should be considered when planning improvements to educational processes.

The idea of the current study was to apply our pre-pandemic conceptual model to the population of university social science students during the pandemic and try to find parallels or identify possible divergences between the two populations, if any. In order to pursue the research objectives, the following research questions were formulated:

RQ1: is the conceptual model of factors influencing mathematics achievement (developed in [19]) applicable to the population of university social science students during the pandemic?

Answering this question was a basic and preliminary activity of our study. Indeed, the occurrence of the pandemic forced us to face a new reality, requiring us to conduct a comprehensive evaluation of the existing model in terms of its applicability to the new pandemic circumstances. Such an evaluation includes verifying that the understanding of the model constructs remains unchanged in both populations of students. Only if we determine that the model is applicable to the population of students during the pandemic can we proceed with further research questions, such as:

RQ2: which factors of mathematics achievement considered in our pre-pandemic research [19] are important for the population of students during the pandemic, and how strong is their influence? Are there parallels/divergences between the two populations?

In the context of this question, we wanted to examine the extent to which the factors that were found to influence the mathematics achievement of the pre-pandemic population of students also had an impact during the pandemic. We wanted to find out if there were differences in terms of the significance and magnitude of factors between the two populations of students.

4. Materials and Methods

4.1. Measurement Instrument and Data Collection

Our study was aimed at investigating the population of full-time students (bachelor’s degree) at the Faculty of Organizational Sciences, University of Maribor. To collect data, we used the three-part online questionnaire presented in [19].

The first part collected the students’ sociodemographic data (gender, age, year of study, and course of study) and mathematical background knowledge from secondary school (BKSS). The second part focused on the perceived level of math anxiety (PLMA), and the third part was designed to measure students’ attitudes toward mathematics (MC and BE), information technology (CT), and attitudes toward involving technology in learning mathematics (PUTLM).

Data were collected systematically for nine consecutive academic years (from 2013–2014 to 2020–2021). The questionnaire was made available at the beginning of the mathematics course through Moodle (Modular Object-Oriented Dynamic Learning Environment). Participation in the survey was completely voluntary, and the students did not receive any benefits. At the end of a given academic year, the survey data were supplemented with data on students’ self-engagement (SEMCU) and their final achievement of the course. After combining the two data sources, the data were anonymized.

The pre-pandemic population comprised students in the first six academic years (from 2013–2014 to 2018–2019) who took courses through traditional face-to-face sessions. However, due to the COVID-19 pandemic outbreak in the summer of 2020, all class activities for the last two years (2019–2020 and 2020–2021) were moved to distance learning. These students represent the pandemic population. To capture their perceptions of the new reality, students in the pandemic population were asked to rate a series of COVID-19-related stressors in terms of depressive symptoms, anxiety, stress, and loneliness [52]. In order to determine any change in the students’ mental health, they were specifically asked whether they felt more or less stressed by particular factors since the COVID-19 crisis began.

4.2. Statistical Methods

The present study is a continuation of our research [19], in which a conceptual SEM (structural equation modelling) model was developed to investigate the main factors influencing mathematics achievement of selected student populations. SEM is an advanced statistical method widely used in behavioral sciences to test causal and correlational relationships between observed and invisible (latent) variables simultaneously (see e.g., [20,53,54]).

As defined in Section 3, the main objective of the current research was to compare two populations of students, pre-pandemic and during the pandemic, in terms of their achievement in mathematics and the factors influencing this achievement. Therefore, in order to make a comprehensive comparison, the same methodological framework should be used. The data were again analyzed using the two-stage SEM approach. The first stage aimed to evaluate the measurement model (in order to answer RQ1), and while the second stage aimed to analyze the structural relationships among the model constructs (i.e., latent variables) (in order to answer RQ2).

4.2.1. Measurement Model Evaluation

The basis for the analyses was the conceptual model developed in [19] and described in Table 1. In order to answer RQ1, we needed to perform the comprehensive applicability of the research model for the purpose of the current study. This was a relatively complex task that involved the following consecutive steps:

• Step 1: model overall fit evaluation,
• Step 2: confirmatory factor analysis,
• Step 3: multi-group confirmatory factor analysis.

The implementation of a particular step is described in the following subsections. At each step, certain conditions must be met. Otherwise, the research cannot continue, which means that the predefined research goal cannot be achieved.

Step 1: Model Overall Fit Evaluation

The overall fit of the measurement and structural models was assessed with a set of widely used fit indices and compared to the proposed thresholds, as follows:

• comparative fit index (CFI), CFI > 0.9 [55];
• root mean square error of approximation (RMSEA), RMSEA < 0.08 [56];
• standardized root mean square residual (SRMR), SRMR < 0.08 [57];
• ratio of Chi-square to degrees of freedom (χ²/df), χ²/df < 3 [58].

The evaluation of the overall fit of the model was performed in three ways: first, the entire sample was considered, and then each subsample (pre-pandemic and pandemic) was analyzed separately. In all cases, the fit indices should be satisfied. However, Cheung and Rensvold [59] emphasized that some fit indices may be affected by the complexity of the model. Therefore, we can be somewhat more lenient in evaluating the model fit of relatively complex models. The process of overall fit evaluation of the model is illustrated in Figure 1.

Step 2: Confirmatory Factor Analysis

Within this step, the measurement model should be validated on the entire sample of data for convergent and discriminant validity. For this purpose, confirmatory factor analysis (CFA) can be used by checking the following indices and their threshold requirements [55,60]:

• convergent validity:

  ○

  standardized factor loadings (SFL), SFL > 0.5 for all questionnaire items;

• discriminant validity:

  ○

  composite reliability (CR), CR > 0.7

  ○

  average variance extracted (AVE), AVE > 0.5 for all latent variables (i.e., model constructs).

The process of performing CFA is illustrated in Figure 2.

Step 3: Multi-Group Confirmatory Factor Analysis

While CFA examines whether the hypothesized measurement model fits the entire data sample well, within this step, we need to precisely compare the measurement model across both subsamples (pre-pandemic, pandemic). For this purpose, a multi-group confirmatory factor analysis (MG-CFA) (see e.g., [20,21]) can be used. The aim of MG-CFA is to examine whether respondents in different groups understand the conceptual model equally and ascribe similar importance to questionnaire items [61,62].

MG-CFA requires a sequential comparison of nested models that differ in terms of the item parameters constrained equally between groups to identify (non)invariant parameters [63].

In order to assess the psychometric equivalence of a construct between the groups [64], we performed the measurement invariance (MInv) test [65]. MInv is accepted if the difference in model parameters between groups is so small that it is attributable to chance. If the model is statistically invariant between groups, then it can be argued that any differences in factor scores are attributable to characteristics of the groups rather than to any deficiencies of the statistical model or inventory [66], while non-invariance in the measurement indicates that a construct has different structures and/or meanings for respondents in different groups.

MInv is typically tested in the following order: configural, weak, and strong invariance, sometimes followed by strict invariance [21].

Since χ² tends to be oversensitive to small, unimportant deviations from a perfect model in large samples [59,67], the results for each invariance test can be explained by the change in several alternative fit indices (AFI). Therefore, changes in CFI (ΔCFI), SRMR (ΔSRMR), and RMSEA (ΔRMSEA) should be examined. In [67], the author suggested that a criterion of −0.01 for the ΔCFI should be paired with ΔRMSEA of 0.015 and ΔSRMR of 0.03 (for metric invariance) or 0.015 (for scalar or residual invariance).

The process of MG-CFA is illustrated in Figure 3.

4.2.2. Structural Model Analysis

In order to answer RQ2, the structural relationships among the model constructs (i.e., latent variables) were analyzed. In applications involving more than one sample (as is the case in the present study), it is of key importance to determine whether or not the components of the measurement model and/or structural model are equivalent (i.e., invariant) between particular groups of interest. For this purpose, we used multi-group structural equation modeling (MG-SEM) [20,21]. The evaluation of the structural model was performed by analyzing the relationships between the latent variables of the model, reporting the path coefficients (unstandardized b, and standardized β) and corresponding z-values. The path coefficients in both groups were examined in terms of whether they differed or not.

5. Results

All analyses were performed using the R-package lavaan [68] and semTools [69]. The results are presented in the following subsections.

5.1. Sample Characteristics

A total of 477 students participated in the study, 321 (67.30%) from the pre-pandemic period and 156 (32.70%) from the pandemic period; 44.2% were men and 55.8% were women. The average age of the participants was 20.1 years (with a standard deviation of 1.61 years), with a range of 18 to 32 years.

5.2. Descriptive Statistics

Table 3. Descriptive statistics of questionnaire items.

Model Construct	Questionnaire Item	Pre-Pandemic		Pandemic
		M	SD	M	SD
Mathematics Confidence (MC)	I have a mathematical mind. (MC1)	3.92	0.808	3.81	0.878
	I can get good results in mathematics. (MC2)	3.71	0.908	3.58	0.983
	I know I can handle difficulties in mathematics. (MC3)	3.94	0.821	3.78	0.904
	I am confident with mathematics. (MC4)	3.17	1.079	3.10	1.094
Behavioral Engagement (BE)	If I make mistakes, I work until I have corrected them. (BE3)	3.50	0.926	3.74	0.976
	If I cannot do a problem, I keep trying different ideas. (BE4)	3.47	0.939	3.77	0.956
Mathematics Test Anxiety (MTA)	Studying for a math test. (MTA1)	3.13	1.264	3.42	1.186
	Taking the math section of the college entrance exam. (MTA2)	2.70	1.114	2.87	1.131
	Taking an exam (quiz) in a math course. (MTA3)	2.88	1.138	3.12	1.158
	Taking an exam (final) in a math course. (MTA4)	3.36	1.146	3.57	1.192
	Thinking about an upcoming math test one week before. (MTA5)	2.84	1.239	2.89	1.293
	Thinking about an upcoming math test one day before. (MTA6)	3.36	1.238	3.55	1.236
	Thinking about an upcoming math test one hour before. (MTA7)	3.63	1.281	3.83	1.314
	Receiving your final math grade in the mail. (MTA9)	2.88	1.219	3.21	1.223
	Being given a “pop” quiz in a math class. (MTA10)	3.79	1.292	3.93	1.271
Numerical Task Anxiety (NTA)	Being given a set of numerical problems involving addition to solve on paper. (NTA2)	1.56	0.907	1.46	0.740
	Being given a set of s subtraction problems to solve. (NTA3)	1.56	0.882	1.46	0.740
	Being given a set of multiplication problems to solve. (NTA4)	1.60	0.879	1.49	0.749
	Being given a set of division problems to solve. (NTA5)	1.73	0.968	1.63	0.805
Mathematics Course Anxiety (MCA)	Watching a teacher work on an algebraic equation on the blackboard. (MCA2)	1.86	1.037	1.80	1.037
	Signing up for a math course. (MCA3)	2.55	1.209	2.72	1.313
	Listening to another student explain a mathematical formula. (MCA4)	2.07	1.095	2.05	1.088
	Walking into a math class. (MCA5)	1.82	1.100	1.99	1.180
Confidence with Technology (CT)	I am good at using computers. (CT1)	3.92	0.967	4.02	0.898
	I am good at using things like VCRs, DVDs, MP3s, and mobile phones. (CT2)	4.28	0.800	4.26	0.786
	I can fix a lot of computer problems. (CT3)	3.51	1.143	3.58	0.984
	I can master any computer program needed for school. (CT4)	3.57	1.004	3.85	0.928
Perceived Usefulness of Technology in Learning Mathematics (PUTLM)	I like using computers for mathematics. (PUTLM1)	3.47	1.151	3.71	1.029
	Using computers in mathematics is worth the extra effort. (PUTLM2)	3.25	1.120	3.42	1.119
	Mathematics is more interesting when using computers. (PUTLM3)	3.15	1.225	3.37	1.108
	Computers help me learn mathematics better. (PUTLM4)	3.22	1.222	3.40	1.190
Background Knowledge from Secondary School (BKSS)	Grade in mathematics in the final year	3.11	0.882	3.28	0.935
	Grade in mathematics at matura	3.20	0.934	3.25	0.988
	Final grade in high school	3.36	0.749	3.66	0.741
Self-Engagement in Math. Course at Univ. (SEMCU)	e-Activities	73.57	11.988	78.57	9.363
	Additional points	4.28	4.184	7.55	4.010
Mathematics Achievement (MA)	% of points achieved on final exam	65.93	22.801	68.62	24.385

presents descriptive statistics for all model constructs (i.e., latent variables). Results are reported for both groups of respondents, pre-pandemic and during the pandemic. For better transparency, the mean values of questionnaire items that were measured on a 5-point Likert-type scale (items within the constructs MC, BE, MTA, NTA, MCA, CT, and PUTLM) are presented graphically in Figure 4.

5.3. Measurement Model Evaluation

5.3.1. Step 1: Model Overall Fit Evaluation

To evaluate the overall fit of the measurement model we followed the process illustrated in Figure 1. First, the whole sample of respondents was considered. The calculated χ² value was 1392.42 at 570 degrees of freedom, and the χ²/df ratio was 2.44 < 3. Both CFI and SRMR indicate good model fit (CFI = 0.911 > 0.9; SRMR = 0.065 < 0.08). RMSEA is 0.060, and the upper bound of its 90% confidence interval (0.055, 0.065) is less than 0.08, suggesting an appropriate model fit [56].

The same calculations were performed for both subsamples of respondents (pre-pandemic and pandemic). It was found that only CFI for the pandemic group is slightly below the desired threshold of 0.9 (probably due to smaller subsample size), while other fit indices fulfill the criteria (χ²/df = 1.57 < 3; SRMR = 0.079 < 0.08; upper bound of 90% confidence interval of RMSEA = 0.062 < 0.08). Since the recommendations for interpreting fit indices suggest using a set of indices instead of just one value, we conclude that the measurement model fits the overall sample as well as both subsamples. The results of the overall fit evaluation are presented in

Table 4. Results of overall fit of measurement model.

		χ2	df	χ2/df	CFI	SRMR	RMSEA	RMSEA 90% CI
Entire Sample		1392.42	570	2.44	0.911	0.064	0.058	0.054, 0.061
Subsamples	Pre-pandemic	1173.11	570	2.06	0.911	0.065	0.060	0.055, 0.065
	Pandemic	893.88	570	1.57	0.886	0.079	0.062	0.053, 0.068

. It can be seen that all conditions to proceed with Step 2 are fulfilled.

5.3.2. Step 2: Confirmatory Factor Analysis

The analysis followed the process illustrated in Figure 2. As can be seen in

Table A1. Parameter estimates, error terms, and z-values for measurement model.

Construct	Questionnaire Item	Unst. Factor Loading	Error Term	Z-Value	Stand. Factor Loading
Mathematics Confidence (MC)	MC1	1.000	- a	- a	0.563
	MC2	1.607	0.146	11.013	0.806
	MC3	1.374	0.130	10.603	0.757
	MC4	2.002	0.173	11.558	0.866
Behavioral Engagement (BE)	BE3	1.000	- a	- a	0.728
	BE4	1.237	0.113	10.941	0.896
Mathematics Test Anxiety (MTA)	MTA1	1.000	- a	- a	0.735
	MTA2	0.928	0.053	17.359	0.758
	MTA3	1.018	0.047	21.758	0.811
	MTA4	1.031	0.042	24.366	0.811
	MTA5	0.943	0.070	13.516	0.687
	MTA6	1.090	0.070	15.490	0.806
	MTA7	1.084	0.068	15.906	0.767
	MTA9	0.858	0.058	14.831	0.639
	MTA10	0.815	0.068	11.986	0.580
Numerical Task Anxiety (NTA)	NTA2	1.000	- a	- a	0.920
	NTA3	0.991	0.037	26.950	0.931
	NTA4	0.986	0.033	30.322	0.925
	NTA5	1.000	0.050	19.970	0.857
Mathematics Course Anxiety (MCA)	MCA1	1.000	- a	- a	0.660
	MCA2	1.330	0.159	8.365	0.730
	MCA3	1.156	0.089	13.048	0.724
	MCA4	1.146	0.099	11.597	0.695
Perceived Level of Mathematics Anxiety (PLMA)	MTA	1.000	- a	- a	0.814
	NTA	0.401	0.088	4.533	0.379
	MCA	0.735	0.097	7.613	0.801
Background Knowledge from Secondary School (BKSS)	Grade in mathematics in final year	1.000	- a	- a	0.936
	Grade in mathematics at matura	0.727	0.082	8.896	0.649
	Final grade in high school	0.533	0.049	10.954	0.594
Self-Engagement in Mathematics Course at University (SEMCU)	e-Activities	1.000	- a	- a	0.783
	Additional points	0.268	0.039	6.776	0.544
Confidence With Technology (CT)	CT1	1.000	- a	-a	0.880
	CT2	0.689	0.044	15.834	0.721
	CT3	1.115	0.056	19.849	0.848
	CT4	0.782	0.053	14.650	0.658
Perceived Usefulness of Technology in Learning Mathematics (PUTLM)	PUTLM1	1.000	- a	- a	0.880
	PUTLM2	0.689	0.044	15.834	0.721
	PUTLM3	1.115	0.056	19.849	0.848
	PUTLM4	0.782	0.053	14.650	0.658

Note: - ^a Indicates parameter fixed at 1 in original solution. Fit indices: χ² = 1392.42, df = 570, χ²/df = 2.44, SRMR = 0.064, CFI = 0.906, RMSEA = 0.055, 90% CI for RMSEA: (0.052, 0.069).

in Appendix A, the standardized factor loadings (SFLs) for all items exceed the threshold of 0.5, therefore the measurement model achieved overall convergent validity.

Table 5. Composite reliability (CR), average variance extracted (AVE), square root of AVE (on the diagonal), and correlations among latent variables.

Model Construct	CR	AVE	MC	BE	MTA	NTA	MCA	BKSS	SEMCU	CT	PUTLM
MC	0.851	0.602	0.776 a
BE	0.799	0.667	0.550	0.817 a
MTA	0.912	0.538	−0.649	−0.402	0.733 a
NTA	0.949	0.822	−0.227	−0.153	0.205	0.907 a
MCA	0.798	0.498	−0.523	−0.375	0.629	0.497	0.706 a
BKSS	0.787	0.568	0.417	0.324	−0.328	−0.028	−0.248	0.646 a
SEMCU	0.658	0.560	0.527	0.375	−0.384	−0.320	−0.384	0.317	0.748 a
CT	0.864	0.622	0.247	0.197	−0.155	−0.187	−0.144	0.113	0.232	0.789 a
PUTLM	0.917	0.735	0.213	0.205	−0.073	−0.088	−0.071	0.020	0.193	0.455	0.857 a

^a Square root of AVE.

shows the values of composite reliability (CR), average variance extracted (AVE), square root of AVE (on the diagonal), and correlations between model constructs. It can be seen that the AVE value for MCA is slightly below the threshold (0.498 < 0.5), while the value for all other constructs exceeds the threshold of 0.5. Since CR of MCA is above 0.6 (0.798), construct validity is still adequate [60], meaning that all included items are significantly related to the corresponding construct. When we compare the square root of AVE for each construct with the correlations between the other constructs, we can see that the square roots are larger than the correlation coefficients, thus also demonstrating the discriminant validity of the entire measurement model.

All the results obtained within this step prove that we can proceed with Step 3.

5.3.3. Step 3: Multi-Group Confirmatory Factor Analysis

The analysis followed the process illustrated in Figure 3. The results of testing MInv between groups are shown in

Table 6. Testing measurement invariance between groups.

Model (Model Comparison)	χ2 (Δχ2)	df	CFI (ΔCFI)	SRMR (ΔSRMR)	RMSEA (ΔRMSEA)	RMSEA 90% CI	p (Chi-Square Test)
M2 Configural invariance	1996.256	1116	0.910	0.063	0.059	0.055, 0.063	/
M3 Weak invariance (M2)	2020.728 (27.878)	1143 (27)	0.909 (−0.001)	0.064 (0.001)	0.058 (−0.001)	0.054, 0.062	0.4173
M4 Strong invariance (M3)	2080.724 (60.883)	1170 (27)	0.906 (−0.003)	0.066 (0.002)	0.059 (0.001)	0.055, 0.063	0.0002
M4a Partial strong invariance (M3)	2065.358 (44.545)	1169 (26)	0.908 (−0.001)	0.065 (0.001)	0.058 (0.000)	0.054, 0.062	0.0132
M4b Partial strong invariance (M3)	2052.452 (30.985)	1168 (25)	0.909 (0.000)	0.064 (0.000)	0.058 (0.000)	0.054, 0.062	0.1895
M5 Strict invariance (M4b)	2104.861 (56.786)	1204 (36)	0.905 (−0.004)	0.065 (0.001)	0.058 (0.000)	0.054, 0.062	0.0151

Note: In model M4a an intercept of Additional points is set as free between groups; in model M4b intercepts of Additional points and Final grade in high school are set as free between groups.

. MInv tests were used in hierarchical order of the nested models, with configural invariance assessed first, followed by weak, strong, and strict invariance.

It can be seen from Table 6 that the first iteration (M2) indicates good model fit (CFI = 0.910 > 0.9; SRMR = 0.063 < 0.08; RMSEA = 0.059 < 0.08), suggesting that the structure of the constructs is the same between the two groups, thus supporting configural invariance.

In the next iteration, we performed a test for weak invariance, and the factor loadings were constrained to be equal between the groups. Comparing models M3 and M2, we can see that the p-value of the Chi-square test is not significant at the 5% level (p = 0.4173 > 0.05), indicating that weak invariance is supported. In addition, the differences in alternative fit indices (ΔAFI) between the configural and weak models are small (ΔCFI = −0.0031; ΔREMSA = −0.001; ΔSRMR = 0.001), supporting weak invariance.

In testing strong invariance, in addition to factor loadings, the intercepts were also constrained to be equal between the groups. Comparing M4 and M3, we can see that the p-value of the Chi-square test is significant at the 5% level (p = 0.0132 < 0.05), indicating that strong invariance is not supported. Therefore, intercepts are not completely invariant between the two considered groups.

In the next two sequential steps (models M4a and M4b), intercepts of measured items additional points and final grade in high school were freely estimated between the groups, and partial strong variance (M4b) was established since the p-value of the Chi-square test between M4b and M3 is not significant at the 5% level (p = 0.1895 > 0.05).

The first variable (additional points), for which intercepts were set to be freely estimated between groups, belongs to the construct self-engagement in mathematics course at university (SEMCU), while the second one (final grade in high school) belongs to the construct background knowledge from secondary school (BKSS). As can be seen in Table 3, the pre-pandemic students gained on average 4.28 additional points, while the pandemic students gained on average 7.55 additional points. Furthermore, a graphical representation with boxplot and histogram in Figure 5 shows that the fraction of students who did not gain any additional points was much greater pre-pandemic than during the pandemic (29.9% vs. 5.1%). Moreover, 14.2% of the pandemic population gained the maximum number of additional points (13), while 7.2% of the pre-pandemic population did so. Based on the results in Table 3 and Figure 6, we can also conclude that pre-pandemic students, on average, did slightly better according to the final grade in high school. The average final grade for pre-pandemic students was 3.35, while for pandemic students it was 3.66.

To test strict invariance, residual (or error) variance was set as fixed in both groups. There was a significant change (Chi-square p = 0.0151 < 0.05) between the partial strong model (M4b) and the strict model (M5), indicating a lack of fit when also constraining error variance to be invariant in the two groups. According to [64], strict measurement invariance is not obligatory, and we could proceed to evaluating the structural model (analysis of RQ2).

5.4. Structural Model Analysis

5.4.1. Structural Model Evaluation

Since we confirmed RQ1, we proceeded with evaluating the structural model fit. We adopted two variants of the structural model, SM1 and SM2, and performed MInv tests. In SM1, the structural coefficients (paths) between the groups are freely estimated, as are the loadings and intercepts (except for the intercepts of additional points and e-activities), while in SM2, the structural coefficients were constrained to be equal between the groups.

The results of the tests are given in

Table 7. Testing measurement invariance of structural coefficients.

Structural Model (SM)	χ2	df	p	CFI	SRMR	RMSEA	RMSEA
(Model Comparison)	(Δχ2)	(Δdf)		(ΔCFI)	(ΔSRMR)	(ΔRMSEA)	90% CI
SM1 Partial strong invariance	2472.28	1282	/	0.894	0.090	0.061	0.057, 0.065
SM2 Structural coefficients	2365.56	1289	0.2976	0.894	0.094	0.061	0.057, 0.064
(SM1)	−106.72	7		0	0.004	0

. It can be seen that the fit of both models, SM1 and SM2, was good. The fit of SM2 was as follows: χ² = 2481.29; df = 1289; CFI = 0.894; and RMSEA = 0.061 (90% CI: 0.057, 0.064). The χ² test of the two nested models and small or negligible differences in the fit indices show that models SM1 and SM2 are not significantly different at a 5% significance level (p = 0.000). This implies that SM2, the simpler model with equal paths between groups, is preferred. Moreover, it implies that both groups of respondents (pre-pandemic and pandemic) perceived the impact of the factors within the conceptual model from [19] in the same way. Thus, from this point of view, the two populations of students are fully comparable, which provides part of the answer to RQ2.

5.4.2. Hypotheses Testing

The next step is to evaluate the structural paths and test the hypotheses from Table 2. The results are given in

Table 8. Summary of research hypotheses testing.

Hypothesis/Path	Group	b	β	z	p	Hypothesis Supported?
H1a: MC → PLMA	Pre-pandemic	−1.088	−0.678 ***	−6.909	0.000	Yes
Expected sign: −	Pandemic		−0.698 ***
H1b: BE → PLMA	Pre-pandemic	−0.120	−0.107 *	−1.812	0.070	No
Expected sign: −	Pandemic		−0.116 *
H2: PLMA → MA	Pre-pandemic	−6.379	−0.221 ***	−4.134	0.000	Yes
Expected sign: −	Pandemic		−0.199 ***
H3: BKSS → SEMCU	Pre-pandemic	2.936	0.331 ***	3.789	0.000	Yes
Expected sign: +	Pandemic		0.375 ***
H4: SEMC → MA	Pre-pandemic	2.766	0.875 ***	9.700	0.000	Yes
Expected sign: +	Pandemic		0.758 ***
H5: CT → PUTLM	Pre-pandemic	0.512	0.460 ***	8.939	0.000	Yes
Expected sign: +	Pandemic		0.452 ***
H6: PUTLM→ MA	Pre-pandemic	−0.253	−0.011 *	−0.320	0.749	No
Expected sign: +	Pandemic		−0.009 *

Note: Structural coefficient invariance was confirmed, which means unstandardized coefficients and z-values are the same between groups, while there may be small changes in standardized coefficients. *** p < 0.001, * p < 0.05.

. The unstandardized coefficient (b) and the standardized coefficient (β), which reflect the relationships among the model constructs (i.e., latent variables) in terms of magnitude and statistical significance, are listed for both groups (pre-pandemic and pandemic). Corresponding z-values and p-values are also provided.

As can be seen in Table 8, five out of seven hypotheses were confirmed, and two (H_1b and H₆) were rejected at the 5% significance level.

5.4.3. Final Model

To find the most parsimonious final model, the nested models approach was applied, in which insignificant paths were iteratively eliminated from the model.

In the first step, the non-significant path PUTLM → MA (H₆) was eliminated. The new model was compared to SM2, and a non-significant Chi-squared difference (Δχ² = 0.892; Δdf = 1; p = 0.3448) justified the elimination. Consequently, the path CT → PUTLM (H₅) was also eliminated, implying that the constructs CT and PUTLM could be decoupled from the rest of the model.

In the next step, the non-significant path BE → PLMA (H_1b) was eliminated. A non-significant Chi-squared difference (Δχ² = 2.708; ∆df = 1; p = 0.0998) again justified the elimination and indicated acceptance of the simpler model.

Finally, four significant paths (H_1a, H₂, H₃, and H₄) remained in the final structural model. The fit of the final model was good, with index values as follows: χ² = 1584.16, df = 678, CFI = 0.874, and RMSEA = 0.078 (90% CI: 0.073, 0.083).

The structural paths of the final model were than evaluated, and the results are presented in Figure 7.

The results confirmed that all of four hypotheses (H_1a, H₂, H₃, and H₄) could be supported at the 0.1% significance level (H_1a: βpp = −0.736, βp = −0.773, z = −7.332, p = 0.000; H₂: βpp = −0.225, βp = −0.199, z = −4.165, p = 0.000; H3: βpp = 0.331, βp = 0.374, z = 3.793, p = 0.000; H₄: βpp = 0.873, βp = 0.759, z = 9.645, p = 0.000). The coefficient of determination for MA regarding the pre-pandemic population of students is ${\mathrm{R}}_{\mathrm{pp}}^2=0.854$, while the corresponding value for the pandemic population is ${\mathrm{R}}_{\mathrm{p}}^2=0.661$.

6. Discussion and Conclusions

Due to the global crisis caused by the COVID-19 pandemic in 2020, universities all over the world underwent a complete transition to online education. Faculty members with little to no experience in online teaching quickly found themselves teaching all of their courses entirely online [18]. This transition was especially challenging in university mathematics education, where face-to-face lectures combined with tutorials are well established and commonly used forms of instruction (Bergsten (2007), Rach and Heinze (2017) as cited in [32]). Even before the pandemic, it was known that running courses entirely online was more likely to cause difficulties for students in mathematics than in other disciplines [70].

Several studies tried to explain the impact of the rapid transition to online mathematics teaching during the pandemic on students’ academic development [12,13,14,15,16,17,24,26,28]. In April 2021, Soesanto et al. [9] encouraged researchers to bridge the still large research gap regarding the causality of implementing online mathematics learning in times of crisis in order to contribute to developing pedagogical strategies to improve university students’ mathematics performance and attitudes.

In order to provide complementary evidence of the impact of online mathematics teaching at the tertiary level of education, the present research compares populations of university students in social sciences in Slovenia between pre-pandemic and during the pandemic in terms of their mathematics achievement and the factors influencing their achievement. The results of two-stage structural equation modeling confirm that the conceptual model developed in our previous study [19] was also applicable to the pandemic population of students (RQ1). The same factors relevant to mathematics achievement in the pre-pandemic population (mathematics confidence, perceived level of math anxiety, background knowledge from secondary school, and self-engagement in mathematics courses at university) proved to be relevant also for the pandemic population (RQ2). Moreover, both populations perceived the impact of the factors within the conceptual model in the same way, and the magnitude of the effects of given factors is comparable. The structural paths of the final model are listed below:

H_1a: 

Mathematics confidence negatively affects the perceived level of math anxiety.

H₂: 

Perceived level of math anxiety negatively affects the achievement in mathematics.

H₃: 

Background knowledge from secondary school positively affects the self-engagement and motivation to fulfil obligations during mathematics course at university.

H₄: 

Self-engagement in mathematics course at university positively affects the achievement in mathematics.

The nested models approach confirmed that all four hypotheses (H_1a, H₂, H₃, and H₄) could be supported at the 0.1% significance level (see Section 5.4.3). Advanced statistical analysis showed that causal and correlational relationships between mathematical achievement factors remained unchanged. From this point of view, no statistically significant differences between the populations were confirmed. For both populations (pre-pandemic and pandemic), mathematics confidence, perceived level of math anxiety, background knowledge from secondary school, and self-engagement in mathematics courses at university were confirmed as influential. Similar values of standardized path coefficient β prove that the magnitude of the effects of a particular factor are also comparable. Our results therefore proved that no increased negative impact on achievement in mathematics was found for any of the factors. These results support previous studies reporting only minor or no negative effects of distance learning on mathematics achievement [16,17,18].

In addition, the results showed a fairly high value of the coefficient of determination for mathematics achievement for both groups of students (0.854 and 0.661), suggesting that the variables “Perceived Level of Math Anxiety” and “Self-Engagement in Mathematics Course at University” together explain a substantial proportion of the total variance. However, given that the proportion of unexplained influences for the “pandemic” students is 33.9%, this likely indicates that their learning outcomes are also influenced by other factors that were not considered in the present model. We intend to focus on identifying these factors in our future research.

Given the general belief in society and commonly expressed assumptions about the negative effects of COVID-19-related measures on learning outcomes, we were quite surprised to find no significant differences in mathematics achievement between the two groups. The following is a reflection on the possible reasons for such an outcome. We believe that the results reflect students’ and teachers’ extraordinary efforts to successfully implement online mathematics teaching. For the pandemic population, all courses were redesigned for distance education using the Moodle e-learning environment and the MS Teams video conferencing system. We see the most important reasons for successful outcomes of the online learning process in:

• familiarity with the Moodle learning platform. Even before the pandemic, parts of some courses at our faculty were conducted online via Moodle. Therefore, the professors had at least basic knowledge of and some experience with preparing lessons, quizzes, and other e-activities available in this environment;
• well-adjusted study materials. We strived to construct an authentic and engaging online learning experience. A number of online study materials were made available, including recorded classes and tutorials. In the regular annual student survey at the end of the course, these recordings were highlighted as extremely helpful and important in enriching the learning experience. This is consistent with the findings of Busto et al. [71]. Similarly, in [18], recordings were reported as a useful resource for re-watching portions of a lecture, especially in mathematics courses, where lessons are often broken down by concepts using specific examples;
• a positive attitude of students. We agree with Takács et al. [11], who found that students were capable of effectively adapting to the virtual teaching modality. They recently explored the characteristics and changes in coping skills of university students in three age groups. They found that students pre-pandemic and during the pandemic did not differ with regard to coping skills, but confirmed changes over the past 20 years. The younger students were found to be fast at processing information and less socially efficient compared to older students. Students of generation Z (born between 1995 and 2012) generally see new situations as a positive challenge and deal with them creatively. Vergara [72] also found that students had a positive attitude toward learning mathematics and strong persistence despite the challenges they encountered;
• accessibility and responsiveness of professors. The lack of communication between students (most of the first-year students did not know each other) led to an increased volume of messages and questions addressed directly to the professors. Although it was very time-consuming, we tried to respond in a timely manner. Timely feedback on student work is highly encouraged so that they will maintain a positive outlook in terms of their capabilities [73].

In summary, the results of our research show that with proper planning, mathematics lectures and tutorials can be successfully conducted online. At the same time, we are aware that it is impossible to ensure the same conditions for all students, especially in providing adequate study space, sufficient technical accessories, and a reliable internet connection [16,73]. In addition, COVID-19 despair and the social, psychological, and emotional stress that comes with it can significantly determine poor student performance [31]. The impact of the pandemic on stress, mental health, and coping behavior for university students has been well-documented from various perspectives [31,52,74,75,76]. In our case, students’ responses to COVID-19-related stressors as described in [52] show that students during the pandemic felt more socially isolated, felt as if they were missing something, and were more worried about their future career, family, and friends compared to the pre-pandemic time. In order to expand the mathematics achievement model with social, psychological, and emotional stressors, other established psychological scales could be used in future research to more accurately assess the impact of the pandemic on students’ mental health and its correlation with mathematics achievement.

There are also some limitations to our study that should be noted. First, all data were collected from a single faculty of a Slovenian university. Consequently, the extent to which the results presented here are applicable to other types of institutions is unclear. Our sampling technique relied on voluntary participation, which increases selection bias. In addition, student achievement in mathematics could be linked to a modified format of assessment tools or cheating on online exams during the pandemic. However, it should be emphasized that only a small portion of the assessment (around 20%) was undertaken remotely. In this sense, we believe that these aspects are likely to have played a minor role in our setting.