Implementation of a Mixed Strategy of Gamification and Flipped Learning in Undergraduate Basic Programming Courses

Abstract

The post-pandemic stage has accelerated the search for innovative ways that impact the teaching–learning process. Flipped learning and gamification have been used as active learning strategies to increase motivation and student learning gains. Both strategies have shown positive results when applied alone and when compared to traditional modalities. In this work, we present a quantitative study that was applied to 414 students throughout a complete course of basic programming, divided into four groups: (1) group that applied flipped learning using videos, (2) group that applied outside-class gamification, (3) group that applied both strategies, and (4) control group. A pretest–posttest process, with 96 true or false questions test, was applied to the groups to find out the normalized learning achievements of the students. A statistical analysis found that the students in groups 1 and 2 performed significantly better (+9%) than the students in the control group. In addition, the students of group 3 had a lower performance than the students of groups 1 and 2 (−10%). Our results confirm that active learning in a flipped classroom and the use of gamification can be useful and strategic tools for advancing the new way of educating in the post-pandemic period.

1. Introduction

In recent times, due to a world in constant evolution, with great technological advances [1], and the effects of the recent pandemic [2,3], researchers seek to build novel learning environments to develop lifelong skills in students [4,5]. Additionally, educational institutions seek to implement strategies and methodologies that offer continuity in the students’ learning and motivate them along the teaching–learning process, allowing a balance in the effort of teachers to adapt their courses. Some researchers propose the use of active learning methodologies for the development of such skills, such as in [5]. The active learning process has the student learning at its center. It focuses on how students learn, not just what they learn. Students are encouraged to ‘think hard’, rather than just listen and take notes.

Active learning is important because of its positive impact on learning and because it can be incorporated into a course in numerous ways [6] and there are several works on this. For example, [7] analyzed the impact of the implementation of an active methodology, namely inquiry-based learning. In the academic context, the authors associated self-efficacy with students’ performance, goals, and motivation. They analyzed students’ motivation as a key element in the teaching–learning process. Additionally, according to [8], the existence of active learning helps students improve emotional intelligence. When self-efficacy and emotional intelligence are achieved, student learning outcomes will improve.

On the other hand, the need to continue experimenting with pedagogical strategies has increased as student groups are heterogeneous, with students from different economic, social, and cultural backgrounds, or students who have had negative or frustrating experiences in their learning; more significantly with students who require more motivation and guidance to acquire knowledge. In this sense, an educational approach based on the combination of blended learning and active learning is presented in [9]. Due to the flexibility offered, the authors argue that it is a great prospect for the enhancement of knowledge and skills acquisition.

These differences are accompanied by the challenges that teachers have and that are derived from the pandemic. For example, [10] mention that keeping students motivated and the possibility of applying traditional strategies (and even e-learning for some disciplines) are a complication that increases professors’ workload and creates stress. Additionally, job market demands are changing rapidly, and professors must prepare students for elements that have not yet been discovered. The introduction of technology in the teaching and learning process, and the introduction of new learning methodologies, present important advantages for students, such as enhancing access to educational resources, flexible education, gaining self-learning abilities, increasing learning opportunities, and lifelong learning as presented in [11]. However, technology integration is another challenge to the teaching–learning process as stated by [12]. Technology is not always accessible for every professor or for every student at any moment. As such, learning strategies must be easy to implement given all the variables in this context.

As an alternative to traditional teaching, in [13,14,15], the flipped learning pedagogy is mentioned. The flipped learning approach works by requesting students to do the passive learning activities outside of the classroom while the active learning activities take place inside the classroom [16]. It involves students reviewing learning materials before coming to class; the content attainment is shifted to outside. Class time is used to explore topics in greater depth via problem-solving and peer instruction. This pedagogical approach has gained popularity in the quest to improve student learning outcomes because it presents advantages for students and professors. For example, they can watch a video at their own pace, they can pause, fast forward or rewind it, and watch it as many times as they need. Another advantage is that the professor can use class time to assist the students while working on activities where they may need the professor’s expert help.

Additionally, analyzing the results obtained on flipped learning revealed that not all students consulted the technological resources or the theoretical contents in books, magazines, tutorials, or articles [17]. This led to the establishment of another strategy to support flipped learning, which would guarantee or make it possible to detect those students who did not consult the information outside the classroom. This strategy is known as gamification.

Gamification is a learning technique that has shown to have a positive effect and has been used in different branches of education to encourage student motivation and learning. It consists of incorporating game design elements into the educational context. The most used elements that have generated positive results are the points, badges, and classification tables or leaderboards; this generates commitment and engagement to the activity, and create positive expectations that help in achieving significant learning. For this reason, gamification is frequently used to increase student motivation required to obtain better results in their learning process [18,19,20]. As stated before, gamification and flipped learning strategies have been reported as an efficient methodology combination for learning [21]. For example, in [22], the author argues that the advantages of game-based learning combined with flipped learning improve students’ intrinsic motivation towards learning goals achievement.

The analysis of the application of gamification in online environments when implementing flipped learning is presented in [23]. The study used mixed-methods sequential explanatory design to collect and analyze quantitative and qualitative data. The study seeks to determine if students achieve better learning outcomes by increasing interaction with online resources using gamification. According to the findings reported by the authors, the experimental group got higher scores in terms of interaction data, participation, and achievement compared to the control group.

The methodological contrast between gamification and flipped learning to determine the most influential methodology in teaching–learning processes is described in [24]. The results of the analysis carried out by the authors show that gamification is better valued in participants at an early stage of the process, while more experienced participants value the flipped learning methodology better.

Light teams and gamification are explored in [25] to enhance students’ learning and social experience. In addition, team-based gamification was documented in this study. The experiment consisted of a design and implementation of a Moodle plugin for gamification to reward students for practicing “good” study habits (such as turning in assignments early and retaking quizzes for extra practice). The results documented showed that the students adopted better study habits. However, these good practices did not generate better grades in the final exams.

Another experience of the application of the combination of flipped learning and gamification is described in [26]. Experience documents that the results obtained by students improve when applying both pedagogical strategies.

Therefore, it can be summarized that the combination of these two techniques has been used in various areas of knowledge and at different educational levels. A challenge that has been reported by authors is to define the game elements to be used to scaffold the flipped learning strategy to meet learning objectives [17,23,24,25,26]. Control over the competition environment must be applied to maintain and increase the quality of the teaching and learning process.

In this work, a quantitative study is presented. The objective of the experiment was to test the learning gain of the students through the separated and combined use of the flipped learning and gamification strategies. The study was applied to 414 students in basic programming groups. The students were divided into four groups: in group 1, the pedagogical strategy used was flipped learning, group 2 exclusively applied gamification, group 3 combined both strategies, and group 4 was a control group. Tests were applied to the students at the beginning and at the end of the analyzed courses, deploying a pretest–posttest methodology. Statistical analysis was applied to generate results on the learning gain obtained in each of the groups studied. According to a statistical study, students that went through flipped learning or a gamification strategy perform 9% better than those in a traditional class. Moreover, students subject to a combined strategy of gamification and flipped learning have a lower learning gain (by about 10%) than those that were included in only one of the strategies. Results are presented in subsequent sections and a graphical abstract can be found in Figure 1.

This document is organized as follows:

• Introduction: In this section, a general scene for learning strategies is presented.
• Hypothesis: Objectives of this study regarding the relationship between learning gains for flipped learning, gamification, mixed, and traditional teaching–learning strategies.
• Materials and methods: Description of following the methodologies of flipped learning, gamification, and the combination of both.
• Results: Statistical analysis over normalized learning gains of the pretest–posttest and focus–control groups process.
• Discussion: Analysis of the results under the light of its strengths and limitations and contrast to other studies.
• Conclusions: Summation of the presented work.
• Data Availability Statement, Author Contributions and Acknowledgements appear at the end of the document.

2. Hypothesis

The focus of this work is to compare flipped learning and gamification strategies combined in programming courses and their impact on students’ learning gains. In accordance with the aim, this work presents a research process guided by the following hypotheses:

• Students involved in a class that implements a gamification methodology mixed with a flipped learning strategy have greater learning gains than those students that are enrolled in a traditional class.
• Students involved in a class that implements a gamification methodology mixed with a flipped learning strategy have greater learning gains than those students that are enrolled in a class that implements only flipped learning.
• Students involved in a class that implements a gamification methodology mixed with a flipped learning strategy have greater learning gains than those students that are enrolled in a class that implements only gamification.

3. Materials and Methods

3.1. Flipped Learning Methodology

For this study, the flipped learning methodology recommended by [27] was applied and organized in two phases: preparation and implementation.

The materials that will be used by the students outside the classroom and by the professor inside the classroom are designed and produced during the preparation stage. To better support the students’ learning process, it is also helpful to determine the general characteristics of the students (age, program, semester, etc.) during this phase. The following tasks define the preparation phase.

• Participants definition:
  • Course selection.
  • Topic selection.
  • Group(s) selection.
  • Students’ general characteristics identification and definition (age, semester, study program, etc.).
• Materials design and preparation.
  • Create teaching and evaluation materials and tools (i.e., videos). It is advised to design and produce two distinct types of materials: (a) material to illustrate concepts, and (b) material that demonstrates how to complete exercises step-by-step. Additionally, this material must be organized to be used inside and outside the classroom.
  • Create a pretest/posttest tool to evaluate the learning gain of the students and the effectiveness of the strategy.

The implementation phase, when students engage in learning tasks inside and outside the classroom, is the foundation of this flipped learning methodology. It presents a cycle that must be implemented for each learning element (topic, for example). It has activities to do before, during, and after teaching.

Before teaching:

• Apply the pretest before the students have any interaction with the concepts they will be studying. Implement the exam in class (not as homework). No materials may be consulted by students. Inform students that they are allowed to leave questions blank.

During teaching:

• The second stage of the preparation phase produced material is to be used in this step by students outside the classroom to prepare and explore the contents.
• Apply a test, at the beginning of the class, to explore what students have learned in the last step.
• Start a discussion with all the students to clarify any doubt about the concepts. Avoid the temptation of giving a “lecture”.
• Provide tasks, exercises, or other activities where students can put their newly found information into practice.
• Use an evaluation element. You can use a task that is comparable to the one that is typically assigned as homework.

After teaching:

• In order to measure the effectiveness of the strategy, and after completing the topic in class, apply the posttest. Use the exam in class (not as homework). No materials may be consulted by students. Inform students that they may leave unanswered questions.

Figure 2 shows the preparation phase and Figure 3 shows the implementation phase of the flipped learning methodology.

3.2. Gamification Methodology

The methodology defined by [28] was applied to introduce the gamification strategy and it is organized in two phases: preparation and implementation. The gamification system is organized to show students’ progression in the course and earn rewards through game elements, such as points, badges, or leaderboards (PBL).

This system is designed and produced during the preparation stage. In order to better support the students’ learning process, it is also helpful to determine the general characteristics of the students (age, study program, semester, etc.) during this phase as first step. The following tasks define the preparation phase.

• Participants definition:
  • Course selection.
  • Group(s) selection
  • Topic selection
  • Students’ general characteristics identification and definition (age, semester, program, etc.).
• Materials design and preparation.
  • Create a gamification strategy wherein activity cycles are defined, and game elements are inserted to them by means of mechanics to increase motivation and scaffold learning. These cycles should be designed to update game elements with the objective of maintaining the fun.
  • Define which tools will be used to deploy the gamification system. These elements could be as simple as a spreadsheet or as complex as a learning management system. These elements should be easy to use so not to be an obstacle to motivation or to the teaching–learning process.
  • Create a pretest/posttest tool to evaluate the learning gain of the students and the effectiveness of the strategy.

On the other hand, the implementation phase is the foundation of this gamification learning methodology, in which students explore the course contents while having fun using game elements without putting their grades at risk or the development of their competences. This phase proposes that the duration of the implementation cycle is the same as the duration of the course to have enough time to learn the system and play with it. It has activities to do before, during, and after teaching.

Before teaching:

• Apply the pretest before the students have any interaction with the concepts and before any contact with the gamification strategy. Implement the exam in class (not as homework). No materials may be consulted by students. Inform students that they are allowed to leave questions blank.

During teaching:

• The gamification strategy is presented to the students; they can choose to play the system game and earn rewards or choose not to play without affecting the regular path that a student follows in the class.
• Cycles, game elements and mechanics are displayed on the defined tools to be played by the students and they are updated frequently to support motivation. Elements, such as points, badges, leaderboards, or others can be used to foster the fun.

After teaching:

• In order to measure the effectiveness of the strategy, the same test applied before installing the gamification system must be used. Use the exam as an in-class activity (not as homework). No materials may be consulted by students. Inform students that they may leave unanswered questions.

Figure 4 shows the preparation phase and Figure 5 shows the implementation phase of the gamification learning methodology.

3.3. Flipped Learning and Gamification Combined Methodology

As [17] stated, a pain point that professors must overcome during the implementation of flipped learning is that students do not prepare themselves by doing the readings or by checking materials outside the classroom before the group discussion session. Some authors declare that this is the result of lack of motivation in the students to perform these activities. Taking this into account, in this work, we have created a methodology to implement flipped learning mixed with a gamification strategy in order to increase students’ motivation and to scaffold the flipped learning strategy. This methodology mixes the previous two methodologies presented in this section resulting in the following steps divided into a preparation phase and an implementation phase:

• Preparation phase
  • Course, learning objects, course topics, and group selection to implement the mixed strategy, as well as the identification and definition of the general characteristics of the students.
  • Design pretest and posttest.
  • Create the flipped learning material to be used in the “during the course” phase of the methodology. This includes reference and study material for the students to check before seeing the topics inside the classroom, material for the group session and assessment activities.
  • Define the students target behaviors in the flipped learning strategy, that will be supported by the gamification strategy during the course. These behaviors can be over in-class or out-of-class elements. Flipped learning objectives must complement the gamification system to foster motivation so rewards support the teaching–learning process.
  • Build the gamification system to promote and reward the students’ target behaviors. This system should consider fun elements to create and maintain motivation.
  • Specify the tools that will be used to deploy each element of both strategies and how they will interact to establish a single learning environment inside the classroom to achieve the learning objectives of the course.
• Implementation phase
  • At the beginning of the course
    • Students answer the pretest questions.
    • The gamification system is presented and deployed.
  • During the course, the flipped learning and gamification mixed strategy is deployed and maintained.
    • Students use the material to prepare and explore the concepts of the topic.
    • At the beginning of the class, a test is applied to explore what students have learned.
    • Afterwards, start a group discussion in class to solve doubts. Avoid giving lectures or having regular classes.
    • Help students practice concepts by giving them exercises or other types of activities.
    • Apply an evaluation element to close the topic. This element can be similar to those assigned as homework but applied inside the classroom. This configuration helps students to have the professor at hand to solve any doubt that arises during the process.
    • Update game elements in the gamification system according to the progress of the students. This progress should represent the advance in positive behaviors that will increase the effectiveness of flipped learning methodology. This step must be a cycle that accompanies the actions and elements defined in the flipped learning strategy. For example, give rewards for those that correctly do out-of-class activities for each topic.
  • At the end of the course, students answer the posttest questions.

Figure 6 and Figure 7 show both phases for the mixed flipped learning and gamification strategy.

4. Results

In this study, 414 undergraduate students participated in a basic programming course of engineering programs at Tecnologico de Monterrey. They were divided into 16 groups. The duration of these courses was 16 weeks. 68 students in 3 groups received the flipped learning methodology suggested in [27]. 97 students in 4 groups received the gamification methodology suggested in [28]. 206 students in 7 groups receive a combination of both methodologies. Additionally, 43 students in 2 groups received a traditional methodology and were considered as a control group. A quantitative experimental research methodology with primary data has been used to compare these samples using pretest–posttest and control–focus group processes. In this section, we will describe the variables used to compare pretest and posttest results. Learning methodologies were applied along the whole duration of the course (one semester).

Thirty-four videos were created to cover all the concepts in the course. Some of the videos were focused on step-by-step solved exercises to complement the theory parts. Slides, having the same content as the videos but in text and static images, were published at the same time. These elements were used by the students to explore contents outside the classroom as part of the flipped learning strategy. For the first part of the in-class section and to explore students’ learning, quizzes with one or two questions were prepared. Additionally, exercises, with no grade assigned and like those in the videos, were given to the students to practice and solve doubts with the professors’ help. As an evaluation, a set of programming problems (two to four, depending on the difficulty of the topic) were used in class. A grade was assigned for these activities.

A gamification system based on points, leaderboards and badges was also implemented. A points system was deployed to support the flipped learning methodology and to motivate students to check videos and material before class. Points were awarded based on the grade obtained in quizzes. A leaderboard was also publicly displayed on the learning management system of the class.

Badges were used to provoke different positive behaviors in students for basic activities in the course, such as arriving on time, improving their grades from partial to partial, being the first student to solve homework, among others. A specific badge was integrated into the system to reward students having the highest grade in the group for each quiz.

Additionally, a system with 8 levels was introduced. Students reached a level by accumulating a certain number of points. Students were awarded points in their final grade for the course if they reached levels 6 (1 point over 100), 7 (2 points over 100) or 8 (3 points over 100).

The flipped learning cycle was applied for each topic in the course while the gamification system was active from the beginning of the semester and was closed in the last class, updating and publishing results each week.

Pretest and posttest were the same test. They contained 96 true or false questions, covering all the topics of the course. Pretest was applied in the first session of the semester while the posttest was applied in the last one. Students had one hour to work on them. They were allowed to leave blank questions if they did not know the answer. Support material was not allowed. Students were not previously notified about the application of the tests, so they had not had the opportunity to study before it. The maximum possible grade was 100 points.

To progress with the results analysis, a normalized learning gain (as stated by [29]) was obtained using the following variables:

Pretest result for the student i:

$$-{Pre}_i$$

Posttest result for the student i:

$$-{Post}_i$$

Students’ normalized gain (1) for the student i. This is a measure of the actual gain that the students achieved during the course (Post_i−Pre_i) over the maximum gain that they could have obtained (100−Pre_i), as a percentage.

$$g_i=\frac{{Post}_i-{Pre}_i}{100-{Pre}_i}\times 100$$

(1)

Normalized gain mean for the group (2). N is the number of students.

$$\langle g\rangle =\frac{1}{N}\sum _{i=0}^n\left(g_i\right)$$

(2)

Table 1. Exploratory analysis for samples.

Strategy	N	Min.	1st Qu.	Median	<g>	3rd Qu.	Max	# of Students Having 30 ≤ gi ≤ 70	% of Students Having 30 ≤ gi ≤ 70
Flipped learning	68	15.38	39.86	51.03	50.02	60.90	73.81	57	83.82%
Gamification	97	10.81	41.18	53.09	50.92	61.73	84.44	76	78.35%
Mixed strategy	206	6.494	30.200	41.863	40.339	51.316	81.633	153	74.27%
Control Group	43	7.143	28.902	43.662	41.013	53.516	67.273	32	74.42%

and

Table 2. Standard deviation values for samples.

Strategy	N	Std. Deviation	Std. Error Mean
Flipped learning	68	13.71	1.66
Gamification	97	16.27	1.65
Mixed strategy	206	14.95	1.04
Control Group	43	16.39	2.50

show the summary of these variables for the four samples along with statistical values.

For the control group (N = 43), mean value = 41.013; std. deviation = 16.3918702, and std. error mean = 2.4997368. For the focus group that used only flipped learning (N = 68), mean value = 50.02, std. deviation = 13.7088726, and std. error mean = 1.662445. For the focus group that used only gamification (N = 97), mean value = 50.92, std. deviation = 16.2649151, and std. error mean = 1.6514519. For the focus group that used both methodologies (N = 206), mean value = 40.339, std. deviation = 14.9450952, and std. error mean = 1.0412741. Figure 8 shows these values for the four samples. Figure 9 shows the density charts for flipped learning, gamification, mixed methodology, and control groups’ learning gain variable.

Ref. [29] defined the three ranges to characterize normalized learning gains: low for a value below 30; medium for a value between 30 and 70; and high for a value above 70. Additionally, Ref. [30] say that interactive engagement courses (that use methods for hands-on activities with immediate feedback, such as in gamification or flipped learning) present a normalized learning gain in the medium range.

Table 3. Exploratory analysis for samples.

Strategy	N	Low Gain	% Low Gain	Medium Gain	% Medium Gain	High Gain	% High Gain
Flipped learning	68	8	11.76%	57	83.82%	3	4.42%
Gamification	97	12	12.37%	76	78.35%	9	9.28%
Mixed strategy	206	51	24.76%	153	74.27%	2	0.97%
Control Group	43	11	25.58%	32	74.42%	0	0.0%

shows this categorization for the four samples. As we can see, most of the samples present a medium normalized learning gain value. This agrees with the idea that these groups received an interactive engagement learning methodology.

To check if these results are comparable, we applied the Shapiro–Wilk Normality Test [31] and the Levene’s Test for Homogeneity of Variance [32] to the data.

Table 4. Shapiro–Wilk normality test and Levene’s test for homogeneity of variance results.

Shapiro–Wilk Test								Levene’s Test
Control		Flipped Learning		Gamification		Mixed
W	p-Value	W	p-Value	W	p-Value	W	p-Value	F	Sig.
0.95593	0.09831	0.96568	0.05791	0.97784	0.0999	0.98687	0.05381	0.7348	0.5317

shows the results of these tests.

Given that these results allow us to continue the analysis, a one-way ANOVA test [33] was performed to know if the means of these groups present a difference.

Table 5. One-way ANOVA test.

F	Sig.
14.61	4.65 × 10−9

shows the results of the test. As it can be seen, the results point to the fact that at least one group differs from the others with a 99.9% confidence level.

To analyze the pattern of difference between means, Tukey’s Honestly Significant Difference (HSD) test [34] has been applied to data. Results of the test are shown in

Table 6. HSD test. Significant values are marked with an *.

Comparison	Diff	Lwr	Upr	p
Flipped learning—Control	9.0051422	1.352094	16.658190	0.0135752 *
Gamification—Control	9.9115396	2.715301	17.107778	0.0023937 *
Mixed—Control	−0.6739435	−7.259515	5.911628	0.9935517
Gamification—Flipped learning	0.9063974	−5.306058	7.118853	0.9818105
Mixed—Flipped learning	−9.6790857	−15.172582	−4.185589	0.0000427 *
Mixed—Gamification	−10.5854831	−15.422343	−5.748624	0.0000002 *

.

Figure 10 shows the 95% family-wise confidence level for the test and the learning gain comparisons chart.

As it can be seen from the results of the HSD test, flipped learning (alone) and gamification (alone) groups have a significant difference of mean with the control group. Both present a higher normalized learning gain of around 9% (flipped learning diff = 9.0051422%, p = 0.0135752; and gamification diff = 9.9115396, p = 0.0023937). This means that both learning strategies have a better performance than a traditional course.

On the other hand, the group that applied both methodologies had no significant difference of mean with the control group. Additionally, there is no significant difference between the flipped learning (alone) and gamification (alone) groups. This result means that we cannot agree to hypothesis 1.

In contrast, we can see that there is a negative significant difference of mean between the flipped learning (alone) and gamification (alone) groups compared to the mixed group. The mixed group presents a lower normalized learning gain of −10% (flipped learning diff = −9.6790857%, p = 0.0000427; and gamification diff = −10.5854831, p = 0.0000002). This result means that, alone, each learning strategy increases the impact on students’ learning gains but when mixed, the result is similar (or even a little lower) to the effects of a traditional learning strategy. These results refute hypothesis 2 and 3 and prove that following this combined methodology of flipped learning and gamification strategies have the opposite effect on students’ learning gain.

5. Discussion

As we can observe, results point to the following elements:

• Flipped learning strategy has a greater learning gain compared to a traditional class.
• Flipped learning strategy has a greater learning gain compared to a class that applies the combination of gamification and flipped learning.
• Gamification strategy has a greater learning gain compared to a traditional class.
• Gamification strategy has a greater learning gain compared to a class that applies the combination of gamification and flipped learning.
• The deployment of a flipped learning and gamification mixed strategy does not have a significantly different effect compared to a traditional class.

Results 1 and 3 were expected. There are several studies that confirm this. For example, [35,36,37] found efficacy in the flipped learning strategy. On the other hand, [18,19,38] found that gamification increases the learning gain in students. These findings are important because, given the composition of the used flipped learning and gamification methodologies, their advantages can be extended to other contexts. For example, such a flipped learning strategy can be implemented as a tool for environments with online elements (either completely online courses or in a hybrid learning way) to help professors tackle complicated concepts and to apply authentic assessment by observing students’ behavior and performance directly in the implementation phase. However, it is important to note that this work implemented flipped learning by means of videos and the use of a learning management system to store and publish them. It is possible that these elements are not available for all classes. In these cases, other tools (such as readings) can be used. On the other hand, the positive results obtained from the gamification experience can be extended to other learning environments because the learning strategy is flexible, as it focuses on behaviors instead of concepts or knowledge domains.

Outcomes 2, 4 and 5 of this study were not expected but it agrees with what was found in other studies, such as in [25]. Authors in that study found that the combination of flipped learning and gamification indeed foster motivation on a computer science basic course (domain of this study), but there was no difference regarding the students’ final exam scores when introducing gamification to flipped learning. The present study can be extended to include motivation aspects; however, if this combination of strategies does not show an improvement in the students’ learning, such extension should be aimed to analyze a more effective arrangement of the combination points during the teaching and learning process. For example, the design of the mixed methodology in the present work was focused to engage students to be prepared for the start of the topics in the class by consulting prepared material outside the classroom. Hence, the combination points were directed to reward this behavior and it was not aimed to increase students’ performance (the measured variable). A new design can be created to support and increase learning performance additionally.

A similar experience can be found in [39]. Authors observed that the introduction of gamification to a flipped learning class had a direct positive effect on learning process performance and that this aspect was a significant positive predictor of application-oriented knowledge, thus proving that gamification has an indirect influence on application-oriented knowledge. This result contrasts with the results in the current study where there was no positive effect on learning process performance (according to the type of the pretest–posttest design). It is possible that this difference comes from where the gamification elements were inserted to the flipped learning methodology. Ref. [39] added them to in-class activities while this study placed them around pre-class activities. This contrast could suggest that a mixed approach presents better results when applied to elements of direct work of the professor with the students. Maybe the influence and guidance of a professor is needed in the first stages when learning new contents.

The gamification of pre-class activities and the results presented in this study contribute to increase the analysis of these learning strategies given that, as stated by [21] in their systematic review, in-class elements of flipped learning are more frequently gamified compared to the out-of-class components.

Alternatively, we find other studies that found that the combination of these learning strategies resulted in a greater learning gain. For example, [23] reported that the inclusion of gamification in a flipped learning environment had a small positive effect in the grades of the students compared to those of students in a control group with only a flipped learning experience. Despite this finding, the difference of mean between these groups is approximately 1 or 2 points. Ref. [26] confirm this outcome reporting a difference of means below 1 point over 10 points between focus and control groups. Additionally, [40] have similar results. They find a difference of means around 2 points over 10 points. It is important to note that these studies use pure quizzes grades to measure learning benefits instead of learning gains or normalized learning gains. This could suggest that a mixed strategy has positive results when applied to reinforce short-term learning elements (as for quizzes) instead of being applied to long-term learning elements (as for a complete course).

Another variable to consider for this 5th finding is the complexity added to the course when combining elements from both strategies. Ref. [41] says that a flipped learning and gamification mixed strategy increased motivation on students but presented an increased complexity for the students to be prepared for classes and activities. Additionally, the author states that preparing and deploying the course represented a bigger workload for professors. This is confirmed by the professors that participated in the current study. It is possible that this complexity or difficulty preparing the course and following its journey, and that is applied to both students and professors, interferes with the learning gain results. According to this, we suggest having a separate deployment of strategies initially while students get used to the pace and increase the combined elements towards the end of the course. This demands a long-term course and an increased workload for professors.

Lastly, it is important to say that gamification is a strategy directed to enhance a specific behavior by means of the application of extrinsic motivation. As stated by [42], extrinsic motivation could inhibit players’ autonomy and diminish competence needs satisfaction. This is because, if players do not internalize this type of motivation, there is no meaning for them in performing the activity or changing their behavior. This, in turn, prevents motivation from being sustained. This effect can spill over to the other strategy and result in students losing interest.

On the other hand, an important thing to include as part of the gamification system design is the players’ characteristics and goals, as stated on [22]. The present study, as stated before, used a general leaderboard, points and badges gamification system without any specific element related to aspects outside the learning environment (i.e., students’ age or hobbies). A gamification system with elements as avatars or a game-like story could be designed to explore if this variable can scaffold the teaching–learning process. However, we could suppose in advance that this will not be related to the learning gain even if it could have a positive impact on motivation. Moreover, a design like this could require technological resources to be deployed.

Table 7. Summary of the results of the studies included in this section.

Study	Domain	Methodology	Observed Variable	Student Contact Period with the Strategy	Traditional Course	Flipped Learning	Gamification	Mixed	Results
This one	Undergraduate basic programming courses	Pretest–posttest, focus and control groups	Normalized learning gain	A semester	√	√	√	√	Flipped learning and gamification perform better compared to a traditional course (~9%) Mixed strategy performs lower compared to isolated strategies (~−10%)
[25]	Undergraduate basic computer science courses	Focus and control groups	Final exam scores	A semester	√			√	No difference
[41]	Undergraduate IT course for accounting, finance, marketing, and management programs	Survey	Students’ opinion	6 weeks				√	Combination is appreciated by students, but course complexity increases. Teachers’ preparation workload increased
[39]	Educational sciences	Pretest–posttest, control and focus groups	Learning process performance, Application-oriented knowledge, Intrinsic motivation, Competence need satisfaction, social relatedness need satisfaction	Two lectures		√		√	gamification is a significant positive predictor of learning process performance, learning process performance is a significant positive predictor of application-oriented knowledge
[23]	Higher education	Pretest–posttest, control and focus groups	learning gain			√		√	Focus group has a better but small performance
[26]	Undergraduate energy engineering course	Control and focus group	Grades on tests	89% of the course		√		√	grades and attendance increase when gamification is added to flipped learning
[40]	Undergraduate Physics laboratory courses	Control and focus group	Grades on tests	10 weeks		√		√	Focus group has better performance
[22]	High School English course	Survey	Students’ opinion					√	players’ characteristics and goals must be included in the gamification system design

summarizes the results of the studies analyzed in this section.

6. Conclusions

In this study, we presented a statistical analysis on learning gains between groups of students that received classes with a flipped learning, a gamification, a mixed, and a traditional methodology. A total of 414 undergraduate students in a basic programming course that were part of engineering programs at Tecnologico de Monterrey, participated in this study. A normalized learning gain was obtained by means of a pretest–posttest process based on focus (flipped learning, gamification, and mixed strategy) and control groups (traditional class).

It was found that flipped learning and gamification control groups presented a greater normalized learning gain compared to the control group and the one that received the mixed strategy. Additionally, no statistically significant difference was found between flipped learning and gamification samples, nor between the control group and the mixed strategy group.

It is important to state that these results were obtained by implementing the strategies in a specific area (basic programming). The application of the suggested methodology must be done in other fields to surpass the limitation of generalization.

Future work over motivation elements must be applied to know the impact of the combination of both learning strategies over the scaffolding of students’ positive behaviors. One possible path is to measure changes in both extrinsic and intrinsic motivation. This would help to know the degree of success in internalizing the meaning of the activities carried out by the students. Another possible path would be to use other game elements beyond the basic ones used in this work (points, badges, and leaderboards). For example, the use of avatars or virtual reality could be explored to know if these technologies increase students’ performance.

Additionally, as stated before, the proposed mixed methodology is based on outside-class gamified elements. Research over the points of insertion of gamification elements over the flipped learning structure is required to identify if students’ learning gains depend on how and when the combination of these elements occurs and if there is a change when applying in-class gamification in a mixed methodology. This suggests, for example, the use of gamification elements during the discussion phase of the flipped leaning strategy by assigning roles to students to explain what they have learned by doing the outside-class activities.