Next Article in Journal
Evaluating Daylighting and Preservation in Heritage Halls: Shading Retrofit Case Study at Chengdu Jiangnan Pavilion Street Site Museum
Next Article in Special Issue
A Multi-Project Evaluation of Engineering Students’ Performance for Online PBL: Taking the Sustainable Decision Analysis Course as an Example
Previous Article in Journal
Influence of the Decision to Select Renovated Elements in a Heritage Building on Further Stages of the Aging Process of the Building—A Case Study
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 15
Issue 24
10.3390/su152416928
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Does ChatGPT Play a Double-Edged Sword Role in the Field of Higher Education? An In-Depth Exploration of the Factors Affecting Student Performance

by
Jiangjie Chen
1,
Ziqing Zhuo
1,* and
Jiacheng Lin
2
1
School of Design, Jiangnan University, Wuxi 214122, China
2
Department of Computer Science and Information Engineering, National Taiwan University, Taipei 106319, Taiwan
*
Author to whom correspondence should be addressed.
Sustainability 2023, 15(24), 16928; https://doi.org/10.3390/su152416928
Submission received: 16 November 2023 / Revised: 9 December 2023 / Accepted: 15 December 2023 / Published: 18 December 2023
(This article belongs to the Special Issue Advanced Technologies in Digital and Sustainable Transformation of Teaching and Learning)
Browse Figures
Versions Notes

Abstract

:
The application of generative artificial intelligence in the field of education has been receiving increasing attention, with the performance of chatbot ChatGPT being particularly prominent. This study aims to explore in depth the performance impact on higher education students utilizing ChatGPT. To this end, we conducted a survey on 448 university students and employed the partial-least squares (PLS) method of structural equation modeling for data analysis. The results indicate that all eight hypothetical paths posited in this study were supported, and surprisingly, the hypothesis that technology characteristics have a direct effect on performance impact was supported. Moreover, the study found that overall quality is a crucial factor determining performance impact. Overall quality indirectly affects performance impact through task-technology fit, technology characteristics, and compatibility, among which the mediating effect of compatibility is most significant, followed by technology characteristics. This study offers practical recommendations for students on the proper use of ChatGPT during the learning process and assists developers in enhancing the services of the ChatGPT system.

1. Introduction

At present, technological advancements represented by artificial intelligence (AI) are reshaping the world in various ways [1,2,3]. Generative artificial intelligence (GAI), an essential branch of AI, focuses on responding to user needs by rapidly generating content such as text, images, videos, and code [4], aiming to develop machines capable of mimicking human reasoning and actions [5]. Currently, GAI has had a profound impact in areas such as tourism [6], medicine [7], and accounting [8], and it has inevitably revolutionized teaching models and learning methods in the field of education [9,10,11,12]. Further research indicates that GAI can reform teaching models in management education [13], language education [9], nursing education [14], etc., thus enhancing students’ learning experience. Concurrently, the study by Yilmaz and Yilmaz [15] suggests that utilizing GAI systems like ChatGPT in education promotes students’ learning processes and outcomes. Furthermore, the GAI system, by analyzing students’ learning data and patterns, can provide individualized learning support for each student, which dynamically adjusts the content and difficulty of learning [12], thereby better addressing the evolving educational needs of students and promoting sustainable educational development. It is evident that GAI, as an auxiliary teaching tool, holds significant application value within the educational domain.
ChatGPT, developed by OpenAI, is one of the highly intelligent and intuitive generative artificial intelligence (GAI) systems. During its pre-training phase, it collects and learns from a vast array of data resources including books, articles, and websites [16], enabling it to nearly naturally comprehend and respond to complex requests [17]. As an AI-driven chatbot, ChatGPT serves as a digital assistant, answering questions, providing explanations, and generating learning materials [18], thereby strongly supporting innovation and development in the field of education [19,20,21]. For instance, Sánchez-Ruiz, Moll-López [4] explored ChatGPT’s potential impact on blended learning methodologies in mathematics education, highlighting its guidance role in students’ problem-solving processes; Bitzenbauer [22] demonstrated how to employ ChatGPT in secondary physics education to enhance students’ performance through practical examples; Keiper, Fried [23] discussed the diverse applications of ChatGPT in physical education, such as thematic debates, information retrieval, and quiz testing, Wollny, Schneider [24] suggest that ChatGPT can play a “guiding” role in education, directing students’ learning growth, aiding in the cultivation of lifelong learning skills, and thereby advancing sustainable educational development; additionally, some studies have affirmed the utility of ChatGPT from an academic writing perspective [25,26,27]. However, like all new technologies, the application of ChatGPT is a double-edged sword [28], with some scholars expressing skepticism and concern regarding its use in educational settings [29,30,31]. Nonetheless, the numerous benefits that the application of ChatGPT brings to education are undeniable [18], and it has the potential to become a reliable auxiliary learning tool for teachers and students alike [32].
Although a substantial number of scholars have explored the potential applications and challenges of ChatGPT in the educational field, there remains limited understanding among students, who are one of the main stakeholders in the educational process, regarding their acceptance and usage of this new technology [20]. Furthermore, there is no definitive conclusion about how this technology might affect students’ academic performance. Scholars such as Foroughi, Senali [18] conducted an in-depth investigation of the motivating factors for students using ChatGPT through the UTAUT2 model, identifying performance expectancy as one of the key factors. Notably, students’ performance impact reflects the completion of their learning tasks, and a higher performance impact can enhance learning efficiency [33], indicating that the tools they are using are positively effective. Research by Butt, Mahmood [34] emphasizes that the higher the overall quality of an online learning system in terms of usability, flexibility, accuracy, and feedback, the more likely students using the system will find it aligned with their values, needs, behaviors, and lifestyles (i.e., having high compatibility), thereby contributing to improved learning performance. Additionally, many scholars have pointed out that the higher the overall quality of a new technology, the more it reflects the users’ willingness and mode of use [35,36], and a technology’s overall quality can influence its performance impact through mediating factors such as task-technology fit and compatibility [34].
From the aforementioned literature, the role of GAI in education is becoming more and more important. It is clear that the adoption of ChatGPT can offer diverse assistance in the field of education. For the sustainable development of higher education, it is necessary to study the application of GAI systems such as ChatGPT in higher education at this stage and consider the performance impact on students during its usage. However, research related to the effect of using ChatGPT on students’ learning performance in education remains notably limited. Therefore, this study emphasizes that defining the specific usage scenarios of ChatGPT by students in higher education, as well as its impact on learning performance, is crucial for understanding the future application trajectory and development prospects of this technology. The aim of this research is to further explore and analyze the various influencing factors when students use ChatGPT as a supplementary learning tool in their educational process. By more accurately assessing the educational value of ChatGPT, this study intends to provide strong empirical support and practical recommendations for its effective utilization and practical implementation within the education field.

2. Theoretical Framework and Research Hypotheses

2.1. ChatGPT in Education

Emerging GAI educational tools have introduced novel opportunities for the digital transformation in the field of education [37] and have become one of the tangible means to enhance the efficacy and sustainability of learning systems [38]. These GAI educational tools seamlessly integrate knowledge from multiple disciplines with various types of technologies. They can learn from data through classification, prediction, and generation, subsequently assisting students in completing tasks, making judgments, and tailoring individualized learning paths based on students’ capabilities [39,40]. During this process, GAI educational tools can evaluate the effectiveness of students’ learning, such as the creativity demonstrated in artistic activities [41] and the depth of understanding of fundamental chemical concepts [42]. They can help students in identifying and rectifying gaps in their knowledge, laying a robust foundation for subsequent in-depth studies, thus promoting the achievement of sustainable educational development objectives [38].
As one of the popular GAI educational tools, ChatGPT is a large language model designed to respond to users’ follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests [43]. The responses ChatGPT generates are almost akin to human thinking [44]. With its strong interactivity, high realism, credibility, and creativity, ChatGPT is capable of offering educational support such as personalized instruction, assistance in optimizing writing, language translation, interactive learning, and adaptive learning [45]. Beyond the students, teachers can leverage ChatGPT to some extent to alleviate their workload.Terwiesch [46] demonstrated in an experiment using ChatGPT for exam question design that the task, which usually takes 30 h to complete, can be done in half the time with the help of ChatGPT. Therefore, ChatGPT can make teachers’ work more efficient, allowing them more time to focus on curriculum innovation, professional development, and to provide personalized tutoring for students [47].Students can enhance their learning experience through ChatGPT’s personalized tutoring features [48]. The emergence and rapid development of ChatGPT have subtly influenced the existing norms in the field of education, with many educators viewing it as a potential game-changer for future learning methods [18]. However, despite its evident advantages in education, ChatGPT has also brought about various negative impacts and potential risks [49]. For example, some research points to the misuse of ChatGPT in the scientific process, potentially leading to academic misconduct such as fabrication and the spread of false information [19,50,51]. As such, ChatGPT, as an emerging auxiliary tool in the field of education, has become one of the widely discussed and controversial hot topics.
Some studies have noted that the majority of students hold a positive attitude towards using ChatGPT, believing that it can effectively solve problems encountered during learning, thereby enhancing learning efficiency, stimulating interest, and boosting motivation [52,53,54]. This study identifies four main reasons for this positive perception: First, ChatGPT, as an online platform, allows students to break spatial and temporal constraints, enabling access to learning information and resources at any time and place [16,55]. Second, with its powerful information gathering and processing capabilities, ChatGPT can comprehend students’ intentions and quickly provide comprehensive and valid information [16,56,57]. Furthermore, ChatGPT assists students in completing academic tasks such as quiz generation, language translation, text writing, code generation [29,58], and simplifies complex scientific theories and concepts into understandable language, thus enhancing students’ comprehension and retention [2]. Lastly, ChatGPT’s interactive question-and-answer format offers users an experience akin to human conversation, making it more engaging and interactive [2,5]. Students can receive immediate, personalized feedback in this ‘conversational’ interaction, enabling them to adapt their learning content on the fly, meet learning needs, and promote the learning process [48].
In summary, this study posits that the benefits of using ChatGPT in the field of education outweigh the drawbacks, significantly enhancing students’ learning performance [59]. The overall quality of ChatGPT impacts the performance of student learning through its technology characteristics, the fit between task and technology, and compatibility with existing systems. Similarly, students’ learning performance is influenced by the system’s technology characteristics, overall quality, task-technology fit, and compatibility. Therefore, this study will propose a series of research hypotheses, and further validation will be conducted by establishing a hypothetical model, as illustrated in Figure 1. The forthcoming segment will delve into these suggested relationships with greater specificity.

2.2. Overall Quality

Undeniably, since its inception, ChatGPT has become one of the hottest information systems in the current market [45]. Overall quality is considered a critical indicator for measuring the success and efficacy of an information system, showing a positive relationship with system usage, and can be utilized to predict actual utilization [34,36,60]. DeLone and McLean [61] initially posited that information quality and system quality were among the dependent variables in the information system success model, individually or collectively impacting system usage and user satisfaction. Later, the model was refined by DeLone and McLean [62], including service quality. Nowadays, overall quality is widely perceived as a second-order structure that encompasses system quality, information quality, and service quality [63,64,65]. System quality aims to describe aspects such as usability, functionality, flexibility, and interactivity of information systems [36,61,66], as Al-Obthani and Ameen [36] pointed out that improving the system quality of smart government can increase the frequency and duration of its use by employees; information quality is defined as the level of the output content of information systems, such as accuracy, timeliness, organization, and completeness [62,67], as Kim, Wang [65] showed through empirical research that consumers (especially Chinese consumers) attach great importance to the information quality of application systems, and the perception of information quality directly or indirectly affects the use experience; service quality refers to the degree of support the information system offers to users [66], including anytime-anywhere access, provision of multimedia content, and real-time interactive feedback [68].
Previous scholarly research has revealed that overall quality has been employed to explain students’ acceptance behavior towards various technologies or information systems in an educational context, exhibiting high explanatory power [63,68,69,70]. Additionally, studies have found that overall quality can indirectly affect task-technology fit through factors such as compatibility [34,35]. Task–technology fit is a composite of technological characteristics, task characteristics, and individual characteristics [71], where technological characteristics are inherent to the information system, and the remaining two vary with individual differences. It is inferred that overall quality influences technological characteristics and task–technology fit. Moreover, research by Isaac, Aldholay [35] has shown that in higher education, the overall quality of online learning significantly impacts compatibility. Wu and Wang [72] defined compatibility as the degree to which a new technology matches user needs. If the new technology can meet user demands and align with their values, i.e., exhibit high compatibility, it will help to fully unleash its potential [73]. Hence, this study believes that ChatGPT’s overall quality has a significant effect on technological characteristics, task–technology fit, and compatibility. Accordingly, the following hypotheses are proposed:
Hypothesis 1 (H1). 
Overall quality has a positive impact on technological characteristics.
Hypothesis 2 (H2). 
Overall quality has a positive impact on task–technology fit.
Hypothesis 3 (H3). 
Overall quality has a positive impact on compatibility.

2.3. Task–Technology Fit

Goodhue and Thompson [71] emphasized the importance of task–technology fit (TTF) in explaining how technology impacts performance. As previously mentioned, task–technology fit represents the alignment between task characteristics, individual characteristics, and technology characteristics. Within this context, technology can be seen as the tool employed by individuals to perform tasks, while the task is broadly defined as the action through which an individual transforms inputs into outputs, with technology aiding in the completion of this task [71]. Al-Emran’s research [74] further points out that if there is a well-matched alignment between a system’s technological features and the students’ learning tasks, it would enhance the students’ acceptance and comfort in using the system.
In the field of education, task–technology fit (TTF) is one of the prevalent theories for assessing online learning performance. Cheng [33], in a study of cloud-based online learning systems based on the TTF model, indicated that technology characteristics typically refer to the ease of access to a technology or system, such as anytime-anywhere access, uninterrupted communication, and convenient sharing and synchronization of materials. Moreover, Cheng’s [33] empirical findings revealed that technology characteristics significantly affect students’ perceived TTF, subsequently governing their learning outcomes.
Additionally, TTF has been found to influence the performance impact of technology or systems across various studies in education, including learning management systems [75], online learning usage within Yemeni higher education [35], and digital libraries [76]. Research by Al-Rahmi, Shamsuddin [77], Alamri, Almaiah [3], Alyoussef [78], and Liu, Yao [79] also demonstrates that TTF has a positive impact on students’ usage behavior. In light of these analyses, this study posits that ChatGPT’s task–technology fit can influence students’ learning performance, and technology characteristics, as an essential variable in the TTF model, will also have an impact on students’ learning outcomes. Therefore, the following hypotheses are proposed:
Hypothesis 4 (H4). 
Technology characteristics have a positive impact on task–technology fit.
Hypothesis 5 (H5). 
Task–technology fit has a positive impact on performance impact.
Hypothesis 6 (H6). 
Technology characteristics have a positive impact on performance impact.

2.4. Compatibility

The original concept of compatibility is defined as “the degree to which an innovation is perceived as being consistent with the existing values, needs, and past experiences of potential adopters” [80], and is often considered one of the fundamental premises for users to adopt new information technology or new information systems [35]. In other words, in the educational field, compatibility refers to learners’ perceptions of the benefits brought by a technology or information system [81], and a high level of compatibility often leads to better adoption [72].
Empirical research by Alamri, Almaiah [3] in the context of social networking applications within higher education has shown a positive correlation between compatibility and task–technology fit. According to a study by Philemon, Chibisa [82], the compatibility of the mathematics application software GeoGebra 5.0 had a significant positive impact on task–technology fit. Isaac, Aldholay [35] suggested that performance impact refers to the extent to which information systems can improve work quality, i.e., enhancements in students’ academic achievements. Previous research also indicates that compatibility can influence students’ performance impact. For example, Akour, Al-Maroof [81] defined compatibility as being aligned with learning objectives, satisfying student expectations, and fitting student culture, stating that the more positive students’ perceived compatibility, the better their learning performance; in Taiwanese mobile learning, compatibility is an essential determinant of learning performance [83]; in electronic learning systems, compatibility positively moderated the relationship between system usage and learning performance [84]; Arkorful, Barfi [85] defined compatibility as students’ belief that using MOOCs system enhances their academic scores, finding it significantly affected their learning performance. In conclusion, this study posits that compatibility between ChatGPT and students influences both task–technology fit and performance impact. Therefore, the following hypotheses are proposed:
Hypothesis 7 (H7). 
Compatibility has a positive impact on task–technology fit.
Hypothesis 8 (H8). 
Compatibility has a positive impact on performance impact.

3. Methodology

3.1. Research Design and Questionnaire Design

In this study, we adopted well-established scales with high reliability and validity from previous research and the source of scales is shown in Table 1. Based on the recommendations of expert scholars, made appropriate revisions to some measurement items to enhance their scientific rigor. Before formally administering the questionnaire, five pre-test participants who met the criteria of this study were randomly invited to review it. These participants were asked to assess whether they could fully understand all the questions, and all the proposed misunderstandings were discussed with the pre-testers, leading to modifications in the phrasing of the questions for better comprehension. The questionnaire items used in this study were modified from scales validated in previous research and were set in Likert’s 7-point style (with 1 as strongly disagree and 7 as strongly agree). Reverse questions were designed in the questionnaire to check the validity of the respondents’ answers, and the respondents’ focus was assessed through their response times to differentiate valid questionnaires.

3.2. Data Collection

In this study, a total of 600 questionnaires were distributed in colleges and universities through the online questionnaire platform, and 448 valid questionnaires were collected, with an effective rate of recovery rate of 74.7%. Moreover, this study was conducted in accordance with the guidelines of the Declaration of Helsinki and received ethical approval from the review committee of the Ministry of Social Science, Changshu Institute of Technology. Informed consent was obtained from all participants, and all methods were carried out in accordance with relevant guidelines and regulations. All the respondents were confirmed have the experience of using ChatGPT before filling out the questionnaire. The respondents’ demographics illustrated in Table 2. Among the respondents, there are 211 males, accounting for 47.1%, slightly less than females. There are 237 female respondents, accounting for 52.9% of the total number of respondents. The age of respondents is mainly concentrated in the range of 18–22 years old, with a total of 352 people, accounting for 78.6%, in line with the age distribution characteristics of college and university student respondents. Most of the respondents are sophomore or above, who have more experience in learning and relatively more clear expectations about the use of ChatGPT.

4. Results and Discussion

In this study, the PLS-SEM algorithm in SmartPLS 4 software (version 4.0.9.2) was used, employing the weighted path scheme with a maximum of 3000 iterations and utilizing default initial weights. Additionally, a non-parametric procedure known as bootstrapping was applied, running 5000 samples, in order to determine the statistical significance of the PLS-SEM results.

4.1. Assessment of Measurement Model

In the assessment of the measurement models, construct validity and reliability were rigorously employed. The reliability of all the core variables within this study’s measurement scheme was ascertained through the evaluation of Cronbach’s alpha coefficients. As shown in Table 3, each individual Cronbach’s alpha coefficient in this research ranged from 0.718 to 0.839, surpassing the recommended threshold of 0.7 [86]. Moreover, the composite reliability (CR) values of constructs ranged from 0.841 to 0.888, exceeding the benchmark of 0.7 [87]. Every item in this study demonstrated factor loadings above the recommended value of 0.7. The value for every Average Variance Extracted (AVE) ranged from 0.608 to 0.697, surpassing the suggested threshold of 0.50 [88]. Consequently, all constructs have satisfactorily met the criteria for convergent validity.

4.2. Assessment of Structural Model

In the present research, the outer loadings derived from various indicators on the construct surpassed those of every cross-loading with additional constructs. Consequently, the cross-loading criterion can be considered to have met the required standards, as illustrated in Table 4. It is noteworthy that cross-loadings are commonly employed as the preliminary step in assessing the discriminant validity of indicators [89].
As depicted in Table 5, discriminant validity has been confirmed in accordance with the Fornell–Larcker criterion. The square roots of the average variance extracted (AVEs) on the diagonals were found to exceed those for the correlations among constructs. This typically signifies robust correlations between the constructs and their corresponding indicators, relative to the other constructs within the model [90]. Moreover, the exogenous constructs demonstrated a correlation of less than 0.85 [91], thereby indicating satisfactory discriminant validity [89].
Additionally, the Fornell–Larcker criterion and cross-loading criteria have been found to be less reliable in detecting problems related to discriminant validity. Researchers have advocated the use of the Heterotrait–Monotrait (HTMT) ratio of correlations, confirming that this testing approach offers sufficient reliability for model processing [92]. In this research, the HTMT ratio was specifically employed to test discriminant validity, as shown in Table 6. According to Gold, Malhotra [93], if the HTMT value is below 0.90, discriminant validity has been duly established, and the model is deemed reliable for further processing.

4.3. Path Analysis

It can be seen from the research results of Figure 2 that all 8 research hypotheses in this study have been supported. Specifically, overall quality and technology characteristics (β = 0.628, p < 0.05), overall quality and task–technology fit (β = 0.149, p < 0.05), and overall quality and compatibility (β = 0.687, p < 0.05) have significant direct effects, meaning that H1, H2, and H3 were supported. Meanwhile, the relationships between technology characteristics, task–technology fit, and performance impact were significant for H4 (β = 0.366, p < 0.05), H5 (β = 0.406, p < 0.05), and H6 (β = 0.189, p < 0.05). Additionally, the direct impact of compatibility on task–technology fit and performance impact was supported, namely H7 (β = 0.312, p < 0.05) and H8 (β = 0.290, p < 0.05) were supported. On the other hand, all 7 indirect effects in this study were significant, with specific coefficients shown in Table 6, and the mediating effects will be further discussed in the discussion section (Section 4.5). Table 6 indicates that the R2 and Q2 values satisfy the relevance of dependent variables, and the R2 values feature acceptable levels of explanatory power, conforming to a significant model [94]. The R2 values between the constructs in this study were between 39.4% and 61%, and an R2 greater than 26% is considered significant. Henseler, Ringle [95] and Hair, Hult [89] emphasized that a Q2 value greater than zero confirms the model’s predictive relevance. The Q2 values between the constructs in this study were all non-zero, hence the hypothesized model in this study has acceptable predictive relevance. The variance inflation factor (VIF) is broadly utilized to determine the degree of multicollinearity present [96]. Table 7 shows that the VIF values in this study are between 1.000 and 2.219, which are smaller than 5 [89], confirming that there are no issues of collinearity in our estimation model.

4.4. Importance–Performance Map Analysis

The importance–performance map analysis (IPMA) in SmartPLS compares the total effects (importance) and average latent variable scores (performance) of exogenous constructs with respect to their impact on a target endogenous construct [89]. After ensuring that all indicators were using a metric scale with the same scale direction, and that their weights were positive [97], this study conducted IPMA under all prerequisite conditions, setting performance impact as a target construct, which was predicted by four predecessors (i.e., overall quality; technology characteristics; task–technology fit, and compatibility). Table 8 depicts the IPMA result for the outcome value of performance impact.
Moreover, based on the IPMA results calculated by SmartPLS, Figure 3 was drawn, representing the performance impact construct priority map. Here, the horizontal axis (on a scale of 1) stands for the average importance score (AIS), while the vertical axis (on a scale of 100) represents the average performance score (APS). As seen from Figure 3, OQ (APS = 70.53), TEC (APS = 69.706), TTF (APS = 66.822), and CO (APS = 68.187) exhibit similar effects in determining performance impact, yet show significant differences in importance (AIS scores of 0.558, 0.337, 0.406, 0.416, respectively). Notably, the overall quality is prominent in both importance and performance regarding performance impact. In other words, overall quality is a crucial factor in determining performance impact. That is to say, factors related to overall quality can enhance students’ performance impact in using ChatGPT for learning. Therefore, the focus for improvement should be placed on the system quality, information quality, and service quality within ChatGPT.

4.5. Discussion

The research results demonstrate that, first and foremost, the overall quality significantly and directly affects technology characteristics, task–technology fit, and compatibility (H1, H2, H3 are supported). Among these, the impact on compatibility is most prominent. This finding indicates that the higher the overall quality of ChatGPT in aspects such as system usability, flexibility, up-to-date information, accuracy, multifunctionality, and timeliness of services, the more likely students are to perceive that ChatGPT aligns with their learning values, methods, and needs. This result is consistent with scholars’ research that overall quality has a positive impact on compatibility [35,98]. Moreover, when ChatGPT possesses a high level of system quality, information quality, and service quality, it can better meet students’ needs for personalized, convenient, and real-time learning, which in turn may enhance students’ learning experience [99], and thereby foster a positive attitude towards learning and the use of ChatGPT for educational purposes [100], laying a solid foundation for the further development of sustainable education. Notably, compared with technology characteristics and compatibility, the influence of overall quality on task–technology fit is weakest. This may be due to the fact that task–technology fit is influenced not only by ChatGPT but also requires thorough consideration of the suitability of the technology provided by ChatGPT for academic tasks [3]. Therefore, as a learning tool assisting students in academic tasks, ChatGPT’s overall quality can affect task–technology fit to a certain extent, but it is not sufficient to fully describe task–technology fit.
Secondly, technology characteristics have significant direct effects on task–technology fit and performance impact, and task–technology fit also has a significant direct effect on performance impact (H4, H5 are supported). As previously stated, technology characteristics, being one of the three influencing factors of task–technology fit, directly affect students’ perceptions of task–technology fit. This result further validates the research findings of scholars Cheng [33] and Al-Rahmi, Shamsuddin [77]. Additionally, by accessing learning knowledge and reasonably applying ChatGPT across various areas at anytime and anywhere, students’ performance impact in using ChatGPT for learning can be effectively enhanced [3], which seamlessly integrates both formal and informal learning contexts, offering technological support for lifelong learning and sustainable education [101].
However, surprisingly, the hypothesis that technology characteristics have a significant direct effect on performance impact was supported (H6 is supported). Previous studies rarely mentioned the validation of this hypothesis, with task–technology fit often mediating the use of technology characteristics, thereby influencing the result of performance impact [76,77,102]. Nevertheless, the rapid emergence and development of new technologies have gradually penetrated the educational field, becoming highly promising learning tools. New technologies, each with unique features for different domains and purposes, allow students to easily access various learning resources and stimulate interest, thereby effectively enhancing the impact on learning performance. Therefore, we should guide students to use new technologies like ChatGPT correctly, fully utilizing their technology characteristics, fully leveraging their technological features and exploring their positive uses in higher education. For example, utilizing its personalization features to act as our growth mind-set recommender [103], using its conversational capabilities to practice speaking or learning foreign languages [29]. In a sense, this means that ChatGPT’s functions such as language editing and translation can break through some of the limitations of material conditions, providing students with a relatively more equitable learning environment and a more sustainable way of learning [104].
Furthermore, compatibility has a positive impact on task–technology fit and performance impact (H7, H8 are supported). Specifically, when students feel that ChatGPT aligns with their learning values, styles, and needs, they consider the use of ChatGPT as essential and beneficial in their learning, enabling them to complete various learning tasks and better integrate into the learning process. This result not only validates the view that compatibility has a positive impact on task–technology fit [3], but also further confirms that if a system is reasonable, flexible, and comprehensive in terms of technology, people usually find it suitable and practical [82]. Moreover, in line with many studies, when ChatGPT has high compatibility, it helps to enhance students’ academic achievements, thus improving the performance impact [81,85], promoting the sustainable development of education.
Finally, this study employed the importance–performance map analysis (IPMA) method, and its analysis revealed that overall quality is the core factor affecting performance impact. In other words, by enhancing ChatGPT’s overall quality, students’ learning performance during use can be significantly improved. Therefore, this study particularly focuses on the indirect relationship between overall quality and performance impact. It is worth emphasizing that overall quality indirectly affects performance impact through task–technology fit, technology characteristics, and compatibility, with compatibility’s mediating effect being the most significant, followed by technology characteristics. Compatibility is often regarded as a determining factor of usage in information systems. Whether students are willing to use ChatGPT in learning depends on its compatibility, which positively modulates the relationship between ChatGPT’s usage and students’ academic achievements [84,85]. This means that ChatGPT, as an auxiliary learning tool, can truly be applied to learning and have a positive impact on learning performance only when it has high overall quality, and students perceive it as highly compatible. Conversely, if ChatGPT does not align with students’ learning values, they will be reluctant to incorporate it into the learning process, making its effect on learning performance negligible. Similarly, when ChatGPT’s technology characteristics match students’ learning tasks, students will recognize its overall quality, considering it a meaningful and valuable learning aid, thus integrating it into the learning process to enhance learning performance [79,105], strengthen learning confidence, stimulate students’ active learning and lifelong learning. Additionally, ChatGPT can offer students instant feedback on their learning outcomes, allowing them to swiftly ascertain their strengths and areas for improvement [106]. This facilitates ongoing advancement and supports the achievement of the sustainable development goal in education.

5. Conclusions and Future Studies

Represented by ChatGPT, General Artificial Intelligence (GAI) systems have instigated a paradigm shift in the field of education and provided strong support for the realization of the sustainable development goal of education. This study aims to investigate the performance impact of students using ChatGPT in higher education and employs Partial Least Squares (PLS) to validate the positive impact of technology characteristics, task–technology fit, and compatibility on performance impact. Simultaneously, overall quality exerts a positive influence on technology characteristics, task–technology fit, and compatibility. Furthermore, the results from the importance–performance map analysis (IPMA) of this study reveal that ChatGPT’s overall quality is a key determinant of performance impact. This research emphasizes the importance of meeting students’ individualized learning needs and advocates for continuous optimization and customization of features such as personalized tutoring, diversified outputs and dynamically adjusted learning content to promote the progress of achieving the sustainable development goals of education; to strengthen the system’s technology characteristics and better adapt to various students’ learning styles and abilities. Additionally, timely learning feedback should be provided to enhance the task–technology fit between ChatGPT and students, as well as students’ perception of compatibility, thereby improving learning performance. The findings of this study aim to provide guidance to students on how to use ChatGPT correctly and offer directions to developers to enhance the ChatGPT system services, optimizing students’ performance impact during their learning process.
However, this study does have certain limitations. Firstly, the frequency with which students use ChatGPT could have a mediating effect on learning performance, a factor that warrants further in-depth investigation. Secondly, differences among students from various academic disciplines and gender disparities may also be influencing factors. Future research could aim to further refine the understanding and appreciation of ChatGPT’s application in higher education. This study believes that the use of ChatGPT in higher education is still an emerging research area and holds the potential to bring about more innovation and transformation in higher education, so as to continuously promote the sustainable development of education, society, and even the economy.

Author Contributions

Conceptualization, J.C.; methodology, J.C.; software, J.C.; validation, J.C. and Z.Z.; formal analysis, J.C.; investigation, J.L.; resources, J.L.; data curation, J.C.; writing—original draft preparation, J.C.; writing—review and editing, Z.Z.; visualization, J.C.; supervision, Z.Z.; project administration, J.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

This study does not involve disease treatment or patients, nor does it involve subjects who can be identified.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgments

We also thank the anonymous reviewers who provided valuable comments on the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Dwivedi, Y.K.; Hughes, L.; Ismagilova, E.; Aarts, G.; Coombs, C.; Crick, T.; Duan, Y.; Dwivedi, R.; Edwards, J.; Eirug, A. Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int. J. Inf. Manag. 2021, 57, 101994. [Google Scholar] [CrossRef]
  2. Zhu, C.; Sun, M.; Luo, J.; Li, T.; Wang, M. How to harness the potential of ChatGPT in education? Knowl. Manag. E-Learn. 2023, 15, 133. [Google Scholar]
  3. Alamri, M.M.; Almaiah, M.A.; Al-Rahmi, W.M. The role of compatibility and task-technology fit (TTF): On social networking applications (SNAs) usage as sustainability in higher education. IEEE Access 2020, 8, 161668–161681. [Google Scholar] [CrossRef]
  4. Sánchez-Ruiz, L.M.; Moll-López, S.; Nuñez-Pérez, A.; Moraño-Fernández, J.A.; Vega-Fleitas, E. ChatGPT Challenges Blended Learning Methodologies in Engineering Education: A Case Study in Mathematics. Appl. Sci. 2023, 13, 6039. [Google Scholar] [CrossRef]
  5. Gimpel, H.; Hall, K.; Decker, S.; Eymann, T.; Lämmermann, L.; Mädche, A.; Röglinger, M.; Ruiner, C.; Schoch, M.; Schoop, M. Unlocking the Power of Generative AI Models and Systems Such as GPT-4 and ChatGPT for Higher Education: A Guide for Students and Lecturers; Hohenheim Discussion Papers in Business, Economics and Social Sciences: Stuttgart, Germany, 2023. [Google Scholar]
  6. Wong, I.A.; Lian, Q.L.; Sun, D. Autonomous travel decision-making: An early glimpse into ChatGPT and generative AI. J. Hosp. Tour. Manag. 2023, 56, 253–263. [Google Scholar] [CrossRef]
  7. Gilson, A.; Safranek, C.W.; Huang, T.; Socrates, V.; Chi, L.; Taylor, R.A.; Chartash, D. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 2023, 9, e45312. [Google Scholar] [CrossRef] [PubMed]
  8. Beerbaum, D.O. Generative Artificial Intelligence (GAI) with Chat GPT for Accounting—A business case. SSRN 2023. [Google Scholar] [CrossRef]
  9. Kohnke, L.; Moorhouse, B.L.; Zou, D. Exploring generative artificial intelligence preparedness among university language instructors: A case study. Comput. Educ. Artif. Intell. 2023, 5, 100156. [Google Scholar] [CrossRef]
  10. Overono, A.L.; Ditta, A.S. The Rise of Artificial Intelligence: A Clarion Call for Higher Education to Redefine Learning and Reimagine Assessment. Coll. Teach. 2023, 1–4. [Google Scholar] [CrossRef]
  11. Zhang, C.; Zhang, C.; Zheng, S.; Qiao, Y.; Li, C.; Zhang, M.; Dam, S.K.; Thwal, C.M.; Tun, Y.L.; Huy, L.L. A complete survey on generative ai (aigc): Is chatgpt from gpt-4 to gpt-5 all you need? arXiv 2023, arXiv:2303.11717. [Google Scholar]
  12. Yu, H.; Guo, Y. Generative artificial intelligence empowers educational reform: Current status, issues, and prospects. Front. Educ. 2023, 8, 1183162. [Google Scholar] [CrossRef]
  13. Ratten, V.; Jones, P. Generative artificial intelligence (ChatGPT): Implications for management educators. Int. J. Manag. Educ. 2023, 21, 100857. [Google Scholar] [CrossRef]
  14. Sallam, M. ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. In Proceedings of the Healthcare, Sydney, Australia, 23–24 March 2023. [Google Scholar]
  15. Yilmaz, R.; Yilmaz, F.G.K. The effect of generative artificial intelligence (AI)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Comput. Educ. Artif. Intell. 2023, 4, 100147. [Google Scholar] [CrossRef]
  16. Aithal, S.; Aithal, P. Effects of AI-Based ChatGPT on Higher Education Libraries. Int. J. Manag. Technol. Soc. Sci. (IJMTS) 2023, 8, 95–108. [Google Scholar]
  17. King, M.R. ChatGPT. A conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell. Mol. Bioeng. 2023, 16, 1–2. [Google Scholar] [CrossRef] [PubMed]
  18. Foroughi, B.; Senali, M.G.; Iranmanesh, M.; Khanfar, A.; Ghobakhloo, M.; Annamalai, N.; Naghmeh-Abbaspour, B. Determinants of Intention to Use ChatGPT for Educational Purposes: Findings from PLS-SEM and fsQCA. Int. J. Hum.-Comput. Interact. 2023, 1–20. [Google Scholar] [CrossRef]
  19. Ifenthaler, D.; Schumacher, C. Reciprocal Issues of Artificial and Human Intelligence in Education; Taylor & Francis: London, UK, 2023; Volume 55, pp. 1–6. [Google Scholar]
  20. Strzelecki, A. To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interact. Learn. Environ. 2023, 1–14. [Google Scholar] [CrossRef]
  21. Elbanna, S.; Armstrong, L. Exploring the integration of ChatGPT in education: Adapting for the future. Manag. Sustain. Arab Rev. 2023. [Google Scholar] [CrossRef]
  22. Bitzenbauer, P. ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemp. Educ. Technol. 2023, 15, ep430. [Google Scholar] [CrossRef]
  23. Keiper, M.C.; Fried, G.; Lupinek, J.; Nordstrom, H. Artificial intelligence in sport management education: Playing the AI game with ChatGPT. J. Hosp. Leis. Sport Tour. Educ. 2023, 33, 100456. [Google Scholar] [CrossRef]
  24. Wollny, S.; Schneider, J.; Di Mitri, D.; Weidlich, J.; Rittberger, M.; Drachsler, H. Are we there yet?—A systematic literature review on chatbots in education. Front. Artif. Intell. 2021, 4, 654924. [Google Scholar] [CrossRef] [PubMed]
  25. Yan, D. Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Educ. Inf. Technol. 2023, 28, 13943–13967. [Google Scholar] [CrossRef]
  26. Su, Y.; Lin, Y.; Lai, C. Collaborating with ChatGPT in argumentative writing classrooms. Assess. Writ. 2023, 57, 100752. [Google Scholar] [CrossRef]
  27. Barrot, J.S. Using ChatGPT for second language writing: Pitfalls and potentials. Assess. Writ. 2023, 57, 100745. [Google Scholar] [CrossRef]
  28. Situmorang, D.D.B.; Salim, R.M.A.; Ifdil, I.; Liza, L.O.; Rusandi, M.A.; Hayati, I.R.; Amalia, R.; Muhandaz, R.; Fitriani, A. The current existence of ChatGPT in education: A double-edged sword? J. Public Health 2023, 45, e799–e800. [Google Scholar] [CrossRef] [PubMed]
  29. Kasneci, E.; Seßler, K.; Küchemann, S.; Bannert, M.; Dementieva, D.; Fischer, F.; Gasser, U.; Groh, G.; Günnemann, S.; Hüllermeier, E. ChatGPT for good? On opportunities and challenges of large language models for education. Learn. Individ. Differ. 2023, 103, 102274. [Google Scholar] [CrossRef]
  30. Tlili, A.; Shehata, B.; Adarkwah, M.A.; Bozkurt, A.; Hickey, D.T.; Huang, R.; Agyemang, B. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn. Environ. 2023, 10, 15. [Google Scholar] [CrossRef]
  31. Qadir, J. Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. In Proceedings of the 2023 IEEE Global Engineering Education Conference (EDUCON), Salmiya, Kuwait, 1–4 May 2023. [Google Scholar]
  32. Opara, E.; Mfon-Ette Theresa, A.; Aduke, T.C. ChatGPT for teaching, learning and research: Prospects and challenges. Glob. Acad. J. Humanit. Soc. Sci. 2023, 5, 33–40. [Google Scholar]
  33. Cheng, Y.-M. How does task-technology fit influence cloud-based e-learning continuance and impact? Educ. Train. 2019, 61, 480–499. [Google Scholar] [CrossRef]
  34. Butt, S.; Mahmood, A.; Saleem, S.; Rashid, T.; Ikram, A. Students’ performance in online learning environment: The role of task technology fit and actual usage of system during COVID-19. Front. Psychol. 2021, 12, 759227. [Google Scholar] [CrossRef]
  35. Isaac, O.; Aldholay, A.; Abdullah, Z.; Ramayah, T. Online learning usage within Yemeni higher education: The role of compatibility and task-technology fit as mediating variables in the IS success model. Comput. Educ. 2019, 136, 113–129. [Google Scholar] [CrossRef]
  36. Al-Obthani, F.S.; Ameen, A. Influence of overall quality and innovativeness on actual usage of smart government: An empirical study on the UAE public sector. Int. J. Emerg. Technol. 2019, 10, 141–146. [Google Scholar]
  37. Liu, M.; Ren, Y.; Nyagoga, L.M.; Stonier, F.; Wu, Z.; Yu, L. Future of education in the era of generative artificial intelligence: Consensus among Chinese scholars on applications of ChatGPT in schools. Future Educ. Res. 2023. [Google Scholar] [CrossRef]
  38. Alshahrani, A. The impact of ChatGPT on blended learning: Current trends and future research directions. Int. J. Data Netw. Sci. 2023, 7, 2029–2040. [Google Scholar] [CrossRef]
  39. Chang, C.-H.; Kidman, G. The rise of generative artificial intelligence (AI) language models-challenges and opportunities for geographical and environmental education. Int. Res. Geogr. Environ. Educ. 2023, 32, 85–89. [Google Scholar] [CrossRef]
  40. Yang, W. Artificial Intelligence education for young children: Why, what, and how in curriculum design and implementation. Comput. Educ. Artif. Intell. 2022, 3, 100061. [Google Scholar] [CrossRef]
  41. Vartiainen, H.; Tedre, M. Using artificial intelligence in craft education: Crafting with text-to-image generative models. Digit. Creat. 2023, 34, 1–21. [Google Scholar] [CrossRef]
  42. Daher, W.; Diab, H.; Rayan, A. Artificial Intelligence Generative Tools and Conceptual Knowledge in Problem Solving in Chemistry. Information 2023, 14, 409. [Google Scholar] [CrossRef]
  43. OpenAI. Introducing ChatGPT. Available online: https://openai.com/blog/chatgpt (accessed on 30 July 2023).
  44. Gill, S.S.; Xu, M.; Patros, P.; Wu, H.; Kaur, R.; Kaur, K.; Fuller, S.; Singh, M.; Arora, P.; Parlikad, A.K. Transformative effects of ChatGPT on modern education: Emerging Era of AI Chatbots. Internet Things Cyber-Phys. Syst. 2024, 4, 19–23. [Google Scholar] [CrossRef]
  45. Baidoo-Anu, D.; Owusu Ansah, L. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. SSRN 2023. [Google Scholar] [CrossRef]
  46. Terwiesch, C. Would chat GPT3 get a Wharton MBA. In A Prediction Based on Its Performance in the Operations Management Course Philadelphia; University of Pennsylvania: Pennsylvania, PA, USA, 2023. [Google Scholar]
  47. Grassini, S. Shaping the Future of Education: Exploring the Potential and Consequences of AI and ChatGPT in Educational Settings. Educ. Sci. 2023, 13, 692. [Google Scholar] [CrossRef]
  48. Berşe, S.; Akça, K.; Dirgar, E.; Kaplan Serin, E. The role and potential contributions of the artificial intelligence language model ChatGPT. Ann. Biomed. Eng. 2023, 1–4. [Google Scholar] [CrossRef] [PubMed]
  49. Yu, H. Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front. Psychol. 2023, 14, 1181712. [Google Scholar] [CrossRef] [PubMed]
  50. Thorp, H.H. ChatGPT Is Fun, But Not an Author; American Association for the Advancement of Science: Washington, DC, USA, 2023; p. 313. [Google Scholar]
  51. Mijwil, M.M.; Hiran, K.K.; Doshi, R.; Dadhich, M.; Al-Mistarehi, A.-H.; Bala, I. ChatGPT and the future of academic integrity in the artificial intelligence era: A new frontier. Al-Salam J. Eng. Technol. 2023, 2, 116–127. [Google Scholar]
  52. Ali, J.K.M.; Shamsan, M.A.A.; Hezam, T.A.; Mohammed, A.A. Impact of ChatGPT on learning motivation: Teachers and students’ voices. J. Engl. Stud. Arab. Felix 2023, 2, 41–49. [Google Scholar] [CrossRef]
  53. Haensch, A.-C.; Ball, S.; Herklotz, M.; Kreuter, F. Seeing ChatGPT through Students’ Eyes: An Analysis of TikTok Data. arXiv 2023, arXiv:2303.05349. [Google Scholar]
  54. Shoufan, A. Exploring Students’ Perceptions of CHATGPT: Thematic Analysis and Follow-Up Survey. IEEE Access 2023. [Google Scholar] [CrossRef]
  55. Thu, C.H.; Bang, H.C.; Cao, L. Integrating ChatGPT into Online Education System in Vietnam: Opportunities and Challenges. 2023. Available online: https://osf.io/preprints/edarxiv/hqyut (accessed on 12 July 2023). [CrossRef]
  56. Aljanabi, M.; Ghazi, M.; Ali, A.H.; Abed, S.A. ChatGPT: Open possibilities. Iraqi J. Comput. Sci. Math. 2023, 4, 62–64. [Google Scholar]
  57. Castillo, A.G.R.; Silva, G.J.S.; Arocutipa, J.P.F.; Berrios, H.Q.; Rodriguez, M.A.M.; Reyes, G.Y.; Lopez, H.R.P.; Teves, R.M.V.; Rivera, H.V.H.; Arias-Gonzáles, J.L. Effect of Chat GPT on the digitized learning process of university students. J. Namib. Stud. Hist. Politics Cult. 2023, 33, 1–15. [Google Scholar]
  58. Gill, S.S.; Kaur, R. ChatGPT: Vision and challenges. Internet Things Cyber-Phys. Syst. 2023, 3, 262–271. [Google Scholar] [CrossRef]
  59. Fauzi, F.; Tuhuteru, L.; Sampe, F.; Ausat, A.M.A.; Hatta, H.R. Analysing the role of ChatGPT in improving student productivity in higher education. J. Educ. 2023, 5, 14886–14891. [Google Scholar] [CrossRef]
  60. Hossain, M.A. Assessing m-Health success in Bangladesh: An empirical investigation using IS success models. J. Enterp. Inf. Manag. 2016, 29, 774–796. [Google Scholar] [CrossRef]
  61. DeLone, W.H.; McLean, E.R. Information systems success: The quest for the dependent variable. Inf. Syst. Res. 1992, 3, 60–95. [Google Scholar] [CrossRef]
  62. DeLone, W.H.; McLean, E.R. The DeLone and McLean model of information systems success: A ten-year update. J. Manag. Inf. Syst. 2003, 19, 9–30. [Google Scholar]
  63. Ho, L.A.; Kuo, T.H.; Lin, B. Influence of online learning skills in cyberspace. Internet Res. 2010, 20, 55–71. [Google Scholar] [CrossRef]
  64. Aldholay, A.; Abdullah, Z.; Isaac, O.; Mutahar, A.M. Perspective of Yemeni students on use of online learning: Extending the information systems success model with transformational leadership and compatibility. Inf. Technol. People 2020, 33, 106–128. [Google Scholar] [CrossRef]
  65. Kim, Y.; Wang, Q.; Roh, T. Do information and service quality affect perceived privacy protection, satisfaction, and loyalty? Evidence from a Chinese O2O-based mobile shopping application. Telemat. Inform. 2021, 56, 101483. [Google Scholar] [CrossRef]
  66. Petter, S.; McLean, E.R. A meta-analytic assessment of the DeLone and McLean IS success model: An examination of IS success at the individual level. Inf. Manag. 2009, 46, 159–166. [Google Scholar] [CrossRef]
  67. Halonen, R.; Thomander, H.; Laukkanen, E. DeLone & McLean IS success model in evaluating knowledge transfer in a virtual learning environment. Int. J. Inf. Syst. Soc. Change (IJISSC) 2010, 1, 36–48. [Google Scholar]
  68. Aldholay, A.H.; Isaac, O.; Abdullah, Z.; Ramayah, T. The role of transformational leadership as a mediating variable in DeLone and McLean information system success model: The context of online learning usage in Yemen. Telemat. Inform. 2018, 35, 1421–1437. [Google Scholar] [CrossRef]
  69. Alyoussef, I.Y. Acceptance of e-learning in higher education: The role of task-technology fit with the information systems success model. Heliyon 2023, 9. [Google Scholar] [CrossRef] [PubMed]
  70. Chopra, G.; Madan, P.; Jaisingh, P.; Bhaskar, P. Effectiveness of e-learning portal from students’ perspective: A structural equation model (SEM) approach. Interact. Technol. Smart Educ. 2019, 16, 94–116. [Google Scholar] [CrossRef]
  71. Goodhue, D.L.; Thompson, R.L. Task-technology fit and individual performance. MIS Q. 1995, 213–236. [Google Scholar] [CrossRef]
  72. Wu, J.-H.; Wang, S.-C. What drives mobile commerce?: An empirical evaluation of the revised technology acceptance model. Inf. Manag. 2005, 42, 719–729. [Google Scholar] [CrossRef]
  73. Almarzouqi, A.; Aburayya, A.; Salloum, S.A. Prediction of user’s intention to use metaverse system in medical education: A hybrid SEM-ML learning approach. IEEE Access 2022, 10, 43421–43434. [Google Scholar] [CrossRef]
  74. Al-Emran, M. Evaluating the use of smartwatches for learning purposes through the integration of the technology acceptance model and task-technology fit. Int. J. Hum.–Comput. Interact. 2021, 37, 1874–1882. [Google Scholar] [CrossRef]
  75. McGill, T.J.; Klobas, J.E. A task–technology fit view of learning management system impact. Comput. Educ. 2009, 52, 496–508. [Google Scholar] [CrossRef]
  76. Omotayo, F.O.; Haliru, A. Perception of task-technology fit of digital library among undergraduates in selected universities in Nigeria. J. Acad. Librariansh. 2020, 46, 102097. [Google Scholar] [CrossRef]
  77. Al-Rahmi, A.M.; Shamsuddin, A.; Wahab, E.; Al-Rahmi, W.M.; Alturki, U.; Aldraiweesh, A.; Almutairy, S. Integrating the role of UTAUT and TTF model to evaluate social media use for teaching and learning in higher education. Front. Public Health 2022, 10, 905968. [Google Scholar] [CrossRef]
  78. Alyoussef, I.Y. Massive open online course (MOOCs) acceptance: The role of task-technology fit (TTF) for higher education sustainability. Sustainability 2021, 13, 7374. [Google Scholar] [CrossRef]
  79. Liu, K.; Yao, J.; Tao, D.; Yang, T. Influence of individual-technology-task-environment fit on university student online learning performance: The mediating role of behavioral, emotional, and cognitive engagement. Educ. Inf. Technol. 2023, 1–20. [Google Scholar] [CrossRef] [PubMed]
  80. Moore, G.C.; Benbasat, I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf. Syst. Res. 1991, 2, 192–222. [Google Scholar] [CrossRef]
  81. Akour, I.A.; Al-Maroof, R.S.; Alfaisal, R.; Salloum, S.A. A conceptual framework for determining metaverse adoption in higher institutions of gulf area: An empirical study using hybrid SEM-ANN approach. Comput. Educ. Artif. Intell. 2022, 3, 100052. [Google Scholar] [CrossRef]
  82. Philemon, N.M.; Chibisa, A.; Mabusela, M.S. Acceptance of the GeoGebra Application in Learning Circle Theorems. Int. J. Learn. Teach. Educ. Res. 2022, 21, 1–20. [Google Scholar] [CrossRef]
  83. Cheng, Y.-M. Towards an understanding of the factors affecting m-learning acceptance: Roles of technological characteristics and compatibility. Asia Pac. Manag. Rev. 2015, 20, 109–119. [Google Scholar] [CrossRef]
  84. Islam, A.N. E-learning system use and its outcomes: Moderating role of perceived compatibility. Telemat. Inform. 2016, 33, 48–55. [Google Scholar] [CrossRef]
  85. Arkorful, V.; Barfi, K.A.; Baffour, N.O. Factors affecting use of massive open online courses by Ghanaian students. Cogent Educ. 2022, 9, 2023281. [Google Scholar] [CrossRef]
  86. Nunnally, J.C.; Bernstein, I.H. Psychometric Theory New York; McGraw-Hill: New York, NY, USA, 1994. [Google Scholar]
  87. Gefen, D.; Straub, D.; Boudreau, M.-C. Structural equation modeling and regression: Guidelines for research practice. Commun. Assoc. Inf. Syst. 2000, 4, 7. [Google Scholar] [CrossRef]
  88. Hair, J.F.; Anderson, R.E.; Babin, B.J.; Black, W.C. Multivariate Data Analysis: A Global Perspective (Vol. 7); Pearson: Upper Saddle River, NJ, USA, 2010. [Google Scholar]
  89. Hair, J.F.; Hult, G.T.M.; Ringle, C.M.; Sarstedt, M. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). 2017. Available online: https://books.google.com.tw/books?id=6z83EAAAQBAJ&dq=A%20Primer%20on%20Partial%20Least%20Squares%20Structural%20Equation%20Modeling%20 (accessed on 12 December 2023).
  90. Fornell, C.; Larcker, D.F. Evaluating structural equation models with unobservable variables and measurement error. J. Mark. Res. 1981, 18, 39–50. [Google Scholar] [CrossRef]
  91. Awang, Z. Structural Equation Modeling Using AMOS Graphic; Penerbit Universiti Teknologi MARA: Selangor, Malaysia, 2012. [Google Scholar]
  92. Carrión, G.C.; Henseler, J.; Ringle, C.M.; Roldán, J.L. Prediction-oriented modeling in business research by means of PLS path modeling: Introduction to a JBR special section. J. Bus. Res. 2016, 69, 4545–4551. [Google Scholar] [CrossRef]
  93. Gold, A.H.; Malhotra, A.; Segars, A.H. Knowledge management: An organizational capabilities perspective. J. Manag. Inf. Syst. 2001, 18, 185–214. [Google Scholar] [CrossRef]
  94. Cohen, J. Statistical Power Analysis for the Behavioral Sciences New York; Academic Press: Cambridge, MA, USA, 1988. [Google Scholar]
  95. Henseler, J.; Ringle, C.M.; Sinkovics, R.R. The use of partial least squares path modeling in international marketing. In New Challenges to International Marketing; Emerald Group Publishing Limited: Leeds, UK, 2009; pp. 277–319. [Google Scholar]
  96. O’brien, R.M. A caution regarding rules of thumb for variance inflation factors. Qual. Quant. 2007, 41, 673–690. [Google Scholar] [CrossRef]
  97. Ringle, C.M.; Sarstedt, M. Gain more insight from your PLS-SEM results: The importance-performance map analysis. Ind. Manag. Data Syst. 2016, 116, 1865–1886. [Google Scholar] [CrossRef]
  98. Deli, D.; Candra, I. Dampak penggunaan media pembelajaran online pada mahasiswa UIB selama pandemi COVID-19. In Proceedings of the CoMBInES-Conference on Management, Business, Innovation, Education and Social Sciences, Online, 17–19 May 2021. [Google Scholar]
  99. Javaid, M.; Haleem, A.; Singh, R.P.; Khan, S.; Khan, I.H. Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system. BenchCouncil Trans. Benchmarks Stand. Eval. 2023, 3, 100115. [Google Scholar] [CrossRef]
  100. Tiwari, C.K.; Bhat, M.A.; Khan, S.T.; Subramaniam, R.; Khan, M.A.I. What drives students toward ChatGPT? An investigation of the factors influencing adoption and usage of ChatGPT. Interact. Technol. Smart Educ. 2023. [Google Scholar] [CrossRef]
  101. Santandreu Calonge, D.; Aman Shah, M.; Riggs, K.; Connor, M. MOOCs and upskilling in Australia: A qualitative literature study. Cogent Educ. 2019, 6, 1687392. [Google Scholar] [CrossRef]
  102. Lin, H.-C.; Han, X.; Lyu, T.; Ho, W.-H.; Xu, Y.; Hsieh, T.-C.; Zhu, L.; Zhang, L. Task-technology fit analysis of social media use for marketing in the tourism and hospitality industry: A systematic literature review. Int. J. Contemp. Hosp. Manag. 2020, 32, 2677–2715. [Google Scholar] [CrossRef]
  103. Abduljabbar, A.; Gupta, N.; Healy, L.; Kumar, Y.; Li, J.; Morreale, P. A Self-Served AI Tutor for Growth Mindset Teaching. In Proceedings of the 2022 5th International Conference on Information and Computer Technologies (ICICT), New York, NY, USA, 4–6 March 2022. [Google Scholar]
  104. Lim, W.M.; Gunasekara, A.; Pallant, J.L.; Pallant, J.I.; Pechenkina, E. Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. Int. J. Manag. Educ. 2023, 21, 100790. [Google Scholar] [CrossRef]
  105. Lin, W.-S.; Wang, C.-H. Antecedences to continued intentions of adopting e-learning system in blended learning instruction: A contingency framework based on models of information system success and task-technology fit. Comput. Educ. 2012, 58, 88–99. [Google Scholar] [CrossRef]
  106. Nikolic, S.; Daniel, S.; Haque, R.; Belkina, M.; Hassan, G.M.; Grundy, S.; Lyden, S.; Neal, P.; Sandison, C. ChatGPT versus engineering education assessment: A multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity. Eur. J. Eng. Educ. 2023, 1–56. [Google Scholar] [CrossRef]
Sustainability 15 16928 g001
Figure 1. Research model.
Figure 1. Research model.
Sustainability 15 16928 g001
Sustainability 15 16928 g002
Figure 2. Analysis results of hypothesized model.
Figure 2. Analysis results of hypothesized model.
Sustainability 15 16928 g002
Sustainability 15 16928 g003
Figure 3. Performance impact construct priority map.
Figure 3. Performance impact construct priority map.
Sustainability 15 16928 g003
Table 1. Questionnaire items.
Table 1. Questionnaire items.
ConstructItemsSource
System qualityI believe that…[35,64]
SYQ1: ChatGPT is easy to use.
SYQ2: ChatGPT is flexible and easy to interact with.
SYQ3: My interaction with Chat GPT is clear and easy to understand.
Information qualityChatGPT provides…
IQ1: The latest knowledge.
IQ2: Accurate knowledge.
IQ3: Comprehensive knowledge.
IQ4: Systematic knowledge.
Service qualityI believe that…
SEQ1: ChatGPT has a good feedback speed.
SEQ2: ChatGPT is a multi-functional and well-trained language model, which can provide code writing, language translation, text generation and other functions.
SEQ3: ChatGPT realizes interactive communication.
CompatibilityI believe that…
CO1: ChatGPT is consistent with my learning values.
CO2: ChatGPT adapts to my learning style.
CO3: ChatGPT can meet my needs.
Technology characteristicsI believe that…[33]
TEC1: ChatGPT enables me to acquire knowledge and complete learning tasks anywhere.
TEC2: ChatGPT is able to access apps on mobile devices and present knowledge to me in an appropriate way.
TEC3: ChatGPT shares the history of PC and mobile phone, so that I can view and learn anytime and anywhere.
Task–technology fitI believe that…[33,35]
TTF1: ChatGPT is suitable for helping me complete learning tasks.
TTF2: ChatGPT is necessary for my learning task.
TTF3: ChatGPT is integrated into all aspects of my learning.
Performance impactI believe that…[35]
PI1: ChatGPT helps me to complete the learning task faster.
PI2: ChatGPT has improved my academic efficiency.
PI3: ChatGPT helps me to review and eliminate errors in learning tasks.
PI4: ChatGPT helps me achieve my future learning goals.
PI5: ChatGPT helps me acquire new skills.
Table 2. Descriptive analysis of respondents.
Table 2. Descriptive analysis of respondents.
SampleCategoryNumberPercentage (%)
GenderMale21147.1
Female23752.9
Age18–2235278.6
23–279621.4
GradeFrosh439.6
Sophomore8017.8
Junior14131.5
Senior15835.3
Postgraduates265.8
Table 3. Descriptive and measurement assessment results.
Table 3. Descriptive and measurement assessment results.
ConstructsItemsLoadingsαCRAVE
(>0.7)(>0.7)(>0.7)(>0.5)
System qualitySYQ10.7960.7590.8610.675
SYQ20.849
SYQ30.819
Information qualityIQ10.8240.8320.8880.666
IQ20.771
IQ30.851
IQ40.816
Service qualitySEQ10.7820.7370.8510.656
SEQ20.845
SEQ30.802
Overall quality
(Second-order)		SYQ0.8260.7830.8740.697
IQ0.839
SEQ0.841
Technology characteristicsTEC10.8380.7180.8410.639
TEC20.804
TEC30.754
Task–technology fitTTF10.8130.7820.8730.696
TTF20.857
TTF30.832
CompatibilityCO10.8550.7710.8680.686
CO20.811
CO30.818
Performance impactPI10.7740.8390.8860.608
PI20.792
PI30.775
PI40.797
PI50.760
Table 4. Discriminant validity: cross loading.
Table 4. Discriminant validity: cross loading.
SYQIQSEQTECTTFCOPI
SYQ10.7960.3700.4550.3880.3420.3830.320
SYQ20.8490.4920.4610.3730.3760.4140.400
SYQ30.8190.4680.4370.3890.4160.3790.405
IQ10.5190.8240.4580.4270.4120.4830.483
IQ20.4230.7710.3950.3830.4070.4600.447
IQ30.4230.8510.4380.4880.4770.5220.478
IQ40.3990.8160.4980.4750.4820.5250.453
SEQ10.4470.3970.7820.4030.3580.4680.384
SEQ20.4390.4940.8450.5110.4180.5730.511
SEQ30.4490.4410.8020.4460.3830.4850.431
TEC10.4340.4870.4800.8380.5950.5950.575
TEC20.3860.3980.4720.8040.4710.4970.526
TEC30.2850.4140.3890.7540.5130.4490.434
TTF10.4340.4700.5180.6130.8130.5820.615
TTF20.3750.4470.3390.5120.8570.5470.601
TTF30.3360.4430.3250.5210.8320.4920.581
CO10.3830.5300.5470.5300.5510.8550.583
CO20.4020.4950.5010.5380.5340.8110.525
CO30.4030.4900.5160.5420.5320.8180.571
PI10.3930.4100.5220.5320.6800.5570.774
PI20.3780.4100.4570.4720.5350.5430.792
PI30.3290.4570.4120.4520.5040.5090.775
PI40.3300.4440.3110.4740.5450.4960.797
PI50.3490.5060.4140.5750.5160.5250.760
Table 5. Discriminant validity: Fornell–Larcker criterion.
Table 5. Discriminant validity: Fornell–Larcker criterion.
OQTECTTFCOPI
OQ0.835
TEC0.628 *0.799
TTF0.593 *0.661 *0.834
CO0.687 *0.648 *0.651 *0.828
PI0.630 *0.645 *0.719 *0.676 *0.780
Note: * The level of significance is 0.05.
Table 6. Discriminant validity: Heterotrait–Monotrait ratio (HTMT).
Table 6. Discriminant validity: Heterotrait–Monotrait ratio (HTMT).
OQTECTTFCOPI
OQ
TEC0.830
TTF0.7510.874
CO0.8810.8640.835
PI0.7720.8210.8790.838
Table 7. Structural assessment result.
Table 7. Structural assessment result.
HypothesisPathStd Betap-ValueResultsR2Q2VIF
H1OQ→TEC0.6280.000Support0.3940.2461.000
H2OQ→TTF0.1490.002Support0.5330.3592.127
H3OQ→CO0.6870.000Support0.4720.3191.000
H4TEC→TTF0.3660.000Support 1.933
OQ→TEC→TTF0.2300.000Support
H5TTF→PI0.4060.000Support0.6100.3622.093
OQ→TTF→PI0.0600.004Support
H6TEC→PI0.1890.000Support 2.079
TEC→TTF→PI0.1480.000Support
OQ→TEC→PI0.1190.000Support
H7CO→TTF0.3120.000Support 2.219
OQ→CO→TTF0.2140.000Support
H8CO→PI0.2900.000Support 2.030
CO→TTF→PI0.1260.000Support
OQ→CO→PI0.1990.000Support
Table 8. Results of IPMA.
Table 8. Results of IPMA.
Latent ConstructsPerformance Impact Total EffectIndex Values
(Importance)(Performance)
Overall quality0.55870.530
Technology characteristics0.33769.706
Task–technology fit0.40666.822
Compatibility0.41668.187
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chen, J.; Zhuo, Z.; Lin, J. Does ChatGPT Play a Double-Edged Sword Role in the Field of Higher Education? An In-Depth Exploration of the Factors Affecting Student Performance. Sustainability 2023, 15, 16928. https://doi.org/10.3390/su152416928

AMA Style

Chen J, Zhuo Z, Lin J. Does ChatGPT Play a Double-Edged Sword Role in the Field of Higher Education? An In-Depth Exploration of the Factors Affecting Student Performance. Sustainability. 2023; 15(24):16928. https://doi.org/10.3390/su152416928

Chicago/Turabian Style

Chen, Jiangjie, Ziqing Zhuo, and Jiacheng Lin. 2023. "Does ChatGPT Play a Double-Edged Sword Role in the Field of Higher Education? An In-Depth Exploration of the Factors Affecting Student Performance" Sustainability 15, no. 24: 16928. https://doi.org/10.3390/su152416928

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop