Designing and Evaluating a Serious Game for Learning Artificial Intelligence Algorithms: SpAI War as a Case Study

Abstract

Artificial Intelligence (AI) nowadays is the technology of the future, as its applications are constantly expanding in every aspect of human life. The spread of the internet has given a great impetus to technologies that apply AI algorithms and make their presence more and more intense in everyday life. However, despite the ubiquitous presence of AI, few people can comprehend its true meaning and the reason for its existence, especially the way it is applied. The purpose of this work is to present the design and pilot evaluation of SpAI War, which is a serious game that aims to introduce the user–student to the field of AI and its algorithms. Specifically, the paper presents the design of SpAI War in the context of the Educational Games (EG) design framework. This is followed by the results of the pilot evaluation of the game by 58 undergraduates and graduates of Informatics. The study utilized a survey based on the MEEGA+ model, and positive results were recorded in terms of player experience, game scenario, and perceived short-term learning.

1. Introduction

The growth of video games as a popular form of entertainment has generated significant interest in their potential as powerful educational tools. Serious games [1,2], a term used to describe video games with a “serious purpose”, have emerged as a means of converting video games from simple forms of entertainment into purposeful learning experiences. The integration of video games into educational practices has been extensively studied and has been shown to be effective in enhancing learning outcomes. For example, research has demonstrated that video games can foster transformational play and support science education [3]. Additionally, a meta-analysis by Zhonggen [4] showed that the use of serious games has a positive impact on learning outcomes.

Serious games are considered to be a promising means of familiarizing people with various fields of computer science, which is heavily used in everyday life. A particularly vital and constantly advancing field is Artificial Intelligence (AI). As more people and organizations adopt AI, it turns out that learning its concepts is considered quite complex and is accompanied with various difficulties. Serious games might be a promising means for dealing with these difficulties. However, although there are several games for cultivating algorithmic or computational thinking skills [5] and programming [6,7,8,9,10] in various contexts, there are very few games for learning AI algorithms, and this provided the motivation for the research presented in this paper.

Designing an effective game for learning AI algorithms, or generally a serious/educational game, is not considered an easy task. As Van Eck states, “games are not effective because of what they are” [11] (p. 18) but instead because of: the material they incorporate; players’ actions while playing; the right level of difficulty; continuous interaction of the player with the game and appropriate feedback; and a logical balance between the time and effort needed for implementing the game and the learning outcomes achieved by playing it. Several researchers on the other hand conclude that serious games do not achieve their full potential neither in learning nor in entertainment [12,13], while achieving a balance between education and fun is rather challenging [14]. As a matter of fact, Natucci and Borges consider the balance between pedagogy, emotions and game elements as one of the greatest challenges in serious game design [15]. Brisson et al. [16] highlight the importance of AI and personalization toward achieving such a balance.

Taking into account the aforementioned research and aiming at contributing to the field of serious game design, the objective of this research was to create a serious game that not only entertains but also educates players on AI and its algorithms. By employing the interactivity of video games, the proposed serious game intends to make learning about AI more engaging and accessible for all learners. It is designed to provide a thorough introduction to AI and its algorithms. The serious game aims to overcome the difficulties of learning about AI by integrating interactive and appealing features that are typical of video games, making it more appealing to a diverse audience of all ages and backgrounds. The proposed game, called SpAI War, was designed taking into account the Educational Game (EG) design framework by Ibrahim and Jafar [17]. In order to validate the design of the game, a study was carried out based on a survey utilizing the MEEGA+ model [18]. The main research question of the study was the following:

How do undergraduates and graduates of Informatics evaluate the player experience, scenario, and perceived short-term learning of AI algorithms in the first-person serious game SpAI War?

The rest of the article is structured as follows. Section 2 briefly presents related work and Section 3 analyzes SpAI War based on axes and elements of the Educational Game (EG) design framework [17]. This is followed by a materials and methods section (Section 4) describing the context of the study, information about the participants and data collection instruments. In Section 5, the results of the study are presented, and in Section 6, the results are discussed. Limitations and plans for future research are presented in Section 7, and the final conclusions are drawn in Section 8.

2. Related Work

This section provides an overview of the existing research related to the difficulties of learning AI algorithms, methods of teaching AI algorithms, and the role of serious games in teaching algorithms. The information provided in this section will serve as the basis for our proposed serious game.

2.1. Difficulties in Learning AI Algorithms

Mastering AI algorithms can be a formidable challenge for numerous students. This results from the intricate nature of AI and its algorithms coupled with the scarcity of learning materials to simplify these concepts. The hurdles that come with comprehending AI algorithms can be grouped into multiple domains, including mathematical foundations, conceptualization, and visualization.

Mathematical foundations. To comprehend how AI algorithms work, it is necessary for learners to have a deep understanding of complex mathematical concepts such as probability and statistics [19]. Nevertheless, according to Alpaydin, many students struggle to grasp these concepts, which can pose challenges in successfully applying AI algorithms to real-world scenarios.

Conceptualization. The complexity and abstraction of AI algorithms make them challenging to decipher for learners, ascertaining their inability to perceive the interdependencies between diverse algorithms and the concepts they represent. In accordance with a study conducted by Shute et al. [20], learners grapple with the conceptualization of intricate algorithms, often hindering their proficiency in understanding and applying these algorithms.

Visualization. Understanding complex AI algorithms, such as deep neural networks, can be challenging for learners due to their intricate and interconnected nature [21]. Visualization plays a critical role in comprehending these algorithms, according to Goodfellow and his colleagues, as it helps students understand the underlying principles and relationships between different algorithms. Lack of visualization skills can, therefore, hinder a learner’s ability to understand and apply these algorithms in real-world scenarios.

2.2. Methods of Teaching AI Algorithms

To overcome the challenges of learning AI algorithms, several methods have been proposed to help learners understand these concepts. Some of these methods include visualizations, hands-on experience with AI tools and techniques, and serious games.

Visualizations. Visual aids are a useful approach for enhancing the learners’ understanding of complex AI algorithms [22]. Lauscher recommends the use of visualizations to clarify the mathematical foundations of these algorithms and facilitate learners’ comprehension of the intricate processes involved. Specifically, visualizations can be helpful in explaining the structure and function of deep neural networks, which are among the most challenging types of AI algorithms to comprehend visually.

Hands-on experience. Furnishing learners with practical exposure to AI tools and techniques constitutes another effective strategy for augmenting their comprehension of AI algorithms. Shute et al. [20] recommended the provision of hands-on experience with AI algorithms as a means of cultivating a profound understanding of these algorithms, which can prove beneficial in real-world situations. This approach can facilitate learners’ acuity in deciphering how AI algorithms operate in practice.

2.3. The Role of Serious Games in Teaching Algorithms

The utilization of video games as an instructional tool has garnered considerable attention in scholarly research. In particular, the efficacy of serious games in enhancing learning outcomes has been demonstrated by various studies [4]. Serious games offer an immersive and interactive modality of learning intricate concepts that engenders heightened engagement and motivation among learners. However, some players and educators view the use of serious games at a formal context with skepticism [23]. The most important concerns refer to the fact that serious games are far less engaging in comparison with commercial entertainment games and the fact that gaming is a “voluntary and self-driven activity” is somehow incompatible with the structured and formal context of a school. As noted in the literature, although the potential of serious games is promising, more rigorous research is necessary for investigating their actual effectiveness in learning [24]. Serious games are used as supplementary tools for enhancing traditional educational approaches, but they can play an even more important role in education if supported with appropriate resources, theoretical frameworks and clear goals [24].

Serious games have been employed to instruct algorithms in diverse fields, including mathematics and computer science. For example, a serious game was created to teach AVL trees to computer science students, as demonstrated in the work of Wassila and Tahar [25]. The game’s design aimed to furnish a comprehensive introduction to AVL trees and to offer learners practical exposure to them. By leveraging interactive and stimulating gameplay mechanics, the game sought to facilitate learners’ comprehension of the subject matter and foster their proficiency in applying graph algorithms.

Another example is a serious game developed by Hainey et al. [26] which proved to achieve quite good learning results. The game functions as a puzzle-based platformer that requires players to utilize rudimentary programming constructs to advance through each level. It comes with a variety of learning materials, such as tutorials and assessments, which aid students in acquiring programming knowledge and its real-world applications in a game-based environment.

Generally, serious games have been utilized for: cultivating algorithmic and computational thinking skills to primary school students [5]; introducing secondary school students to programming using various programming languages, such as Python [6] and Java [7]; and teaching programming to higher education students [8]. One of the conclusions drawn in a recent review of educational games for primary school students is the need for carrying out empirical studies for investigating the actual impact of such games on learning programming [5]. Moreover, it is highlighted that the implementation of specific features can make the teaching of programming in formal contexts more effective [5] and consequently help in dealing with the skepticism of stakeholders [23]. The features proposed include [5]: learning analytics; support for collaboration between players; enhanced player–game interaction; support for creating customized lesson plans.

Serious games have been used with positive affective and cognitive results in the education of programming in various contexts, such as: programming assignments [9]; lab sessions [8]; and self-learning [10]. CMX, a massive multiplayer online role-playing game for programming in C, was used in the labs of an introductory programming course of an Informatics department for five weeks [8]. The 76 participants evaluated positively their experience and motivation to continue, while the majority of them performed better in the course in comparison with the students that were taught in the labs with the traditional approach (solving programming exercises in a typical integrated development environment for C). SQL Island, a web-based text adventure game for learning the Structured-Query Language (SQL), was utilized in the context of a programming assignment for an undergraduate Web Programming course [9]. Fifty-six students submitted the assignment based on SQL Island and evaluated the game using the MEEGA+ model. Students evaluated positively the player experience and perceived short-term learning, while their performance in the assignment was very good. The authors concluded that a game that is used in the context of programming assignments can have better results if the following features are supported [9]: good examples of new programming constructs; tasks connected to the story of the game; ability to register and save progress; usage of learning analytics for automatic assessment and reporting of class statistics.

Although numerous serious games for programming have been designed, we were able to find only three games for AI concepts and algorithms, namely Obsolescence [27], Maestro [28] and Sokoban Solver [29]. The small number of games for AI and the rather limited research investigating if and how serious games can support students in comprehending AI concepts/algorithms motivated the authors in designing SpAI War. In the paragraphs that follow, the aforementioned AI games are briefly described, while a comparative analysis of the aforementioned games and SpAI War is presented in

Table 1. Comparative analysis of AI serious games.

Game Design	Obsolescence	Maestro	Sokoban Solver	SpAI War
Game type	Digital board game (competing militaries fight for gaining Influence Points)	Attack and defense simulation environment	Puzzle game	First-Person Shooter
Singe/Multiplayer	Single player	Multiplayer	Single player	Single player
Available actions	Move forces, build forces, interact with technology cards (AI-based technologies)	Provide a solution based on the player’s role (attacker/defender)	Navigate the maze, move boxes	Player navigation, Pick up a weapon, Use weapon, Reload weapon, Open–Close doors, Interact with messages
Time pressure	In-game turn timer	No time pressure	No time pressure	No time pressure
Feedback	Post-game scores	Scores/ranks		Feedback is provided for every wrong or correct choice
AI concepts/algorithms	AI impacts in military operations and strategies	Robust AI	Basic search algorithms; heuristics	Nine AI algorithms; Introduction to Machine Learning

.

Obsolescence [27] is a digital board game designed to educate military decision-makers on dealing with the threats imposed by disruptive AI technologies. In the context of the game, the players participate in competing militaries that build and move forces and interact with AI-based technology cards in order to gain influence points. Obsolescence was evaluated by 48 participants recruited from the United States Air Force and Department of Defense. The evaluation utilized a pre–post survey with questions related to the five learning objectives (LOs) of the game and aimed at evaluating short-term learning. The study showed that the game achieved its LOs, while there was a strong correlation between actual learning and perceived learning.

Maestro is an open-source game-based platform under development (https://maestro-ai.github.io/) (accessed on 1 May 2023) that aims to introduce higher education students to robust AI, which refers to countermeasures and the prevention of AI vulnerabilities [28]. Maestro utilizes goal-based scenarios and a competitive programming environment. In this environment, students are assigned the roles of the attacker or the defender and work either alone or in teams for devising their attack or defense according to their role. The solutions provided are evaluated for their quality, and students are ranked in a leaderboard. Maestro was positively evaluated by undergraduate students in two offerings of an AI course though a survey.

Object-oriented Sokoban solver [29] is a game project that can be utilized for teaching both object-oriented analysis and design and AI. When it comes to AI, the Sokoban solver can be used as a case study for: implementing and experimenting on different path-finding algorithms in order to move one or more boxes on a grid with or without obstacles; teaching heuristics in the context of estimating the distance between the player and the goal state. Although this is an interesting approach, no application and/or evaluation results are available.

Based on the comparative analysis of serious games on AI, it turns out that the available games have different goals: Obsolescence aims at educating military decision-makers on dealing with the threats imposed by disruptive AI technologies; Maestro aims at educating future AI workforce on dealing with AI vulnerabilities; and Sokoban solver aims at supporting students experiment with AI algorithms in the context of a Sokoban puzzle game by diving into its code. The proposed game SpAI War aims at introducing anyone interested in AI to nine fundamental AI algorithms and machine learning through the application of the algorithms for advancing in a 3D first-person shooter game.

3. Design of SpAI War

The serious game SpAI War aims to educate the player on the basic algorithms of AI. SpAI War is a 3D first-person shooter game that aims to motivate players through an interesting scenario. Based on the scenario of SPAI War, the country is in danger, as a foreign paramilitary organization called “Paraforce” has managed to invade the National Security Agency’s (NSA) databases, stealing the nuclear weapon codes. Furthermore, Paraforce managed to reveal all the names of the agents NSA has in order to exclude any possible intervention. The danger is great, as everything leads to the conclusion that the organization intends to transfer confidential data to officials of the rival country’s government. In order to prevent a catastrophic war between the countries, NSA decided to send a trainee agent to destroy Paraforce’s servers and databases.

In order to be able to fulfill his/her goal, the player is required to complete a series of missions, with the ultimate goal of finding the building that houses the Paraforce databases and servers and then destroy them. Initially, the player goes through two introductory levels, which aim to familiarize him/her with the game environment in which he/she learns to move, to use the various keys of interaction with the game environment, but also to become familiar with the use of weapons, as he/she tries to avoid and defeat some NPCs. The player is then called upon to complete a series of missions without being defeated. In each mission, he/she defeats all the guards (NPCs) of the level and tries to complete it by applying each time the steps of a specific algorithm that corresponds to the level and is directly connected to game play. For example, correctly applying the underlying AI algorithm can provide the right order of visiting rooms in the building, avoiding paths with enemies and so on. To ensure the correct learning of the concepts of the algorithms, a course system has been created that connects each level with the corresponding algorithm and is directly accessible at the beginning of each level. In addition, in case of incorrect application of the algorithm steps, a message appears explaining the mistake he/she made, and the level is restarted, as it is considered that the algorithm has not been fully understood by the player.

In the following sections, the design of the game is presented in the context of the three axes of the Educational Games (EG) design framework proposed by Ibrahim and Jafaar [11], namely the game design, pedagogy and learning content modeling. The game design axis deals with the following aspects of a game: usability that refers to the elements of satisfaction, efficiency and effectiveness; multimodality that refers to the elements of multimedia and interaction; fun and challenge that refer to the elements of clear goals, uncertain outcomes and raising the player’s self-esteem. The pedagogy axis deals with the following aspects of a game: learning outcomes; motivation theory; support for self-learning; promotion of problem solving. Finally, the learning content modeling axis deals with syllabus and scaffolding. The EG design framework was selected for the following reasons: it is a simple and straightforward framework that incorporates all the important factors [11] that contribute to an effective game; it gives emphasis to both the educational/pedagogical and fun aspects of a game that are considered necessary for an effective game [12,13].

3.1. Game Design

The game design axis deals with usability, multimodality, fun and challenge issues [17]. Usability is an important aspect of every game, and Ibrahim and Jaafar examine it in terms of its effectiveness, efficiency and satisfaction. The player’s sense of satisfaction is fulfilled during the game, as the tests he/she is called to face are of increasing difficulty and go hand in hand with the knowledge that has been acquired up to that point. In addition, the script of the game is divided into sections, each of which raises the player’s interest, posing challenges, the solution of which causes a feeling of satisfaction. In terms of game efficiency, each level provides sufficient information about the algorithm it represents, having simple but targeted activities to which the acquired knowledge should be applied, with the ultimate goal of deep understanding. In order to achieve effectiveness, the game tries to clarify every aspect of the algorithm applied in each case through detailed and clear theory as well as the presentation of the common mistakes that are made. In addition, the levels are designed so that any information derived from theory can be applied in practice.

Being multimodal is another important aspect of a game, and this requires using multimedia and providing easy interaction between the player and the game with immediate feedback [17]. The player’s interaction with the game environment becomes quite easy and fun, as at each level, the player comes in contact with clear and understandable, in terms of their content, graphic elements that provide him/her with either user-friendly interaction windows or informative messages. For instance, one of the graphic elements with which a player can interact is an AI symbol (Figure 1) that transports the player to the learning theory of each algorithm (Figure 2), including detailed examples of its application (Figure 3).

Fun and challenge are also important aspects of game design and are accomplished through clear goals, uncertain outcomes and raising the players’ self-esteem [17]. For the most effective satisfaction of learning objectives, the game at each level ensures clear objectives, uncertain outcomes and finally aims to increase the player’s self-esteem. These are achieved in the following ways:

• Clear goals: At the beginning of each level, the player is presented with the mission that he/she has to complete with clear steps, depictions and photos of the spaces he/she will encounter (Figure 4). In this way, the player is fully focused on learning each algorithm and is not burdened with extra effort to understand what should follow.
• Uncertain outcomes: The game ensures the avoidance of any possible sense of monotony when using it, as almost every level includes different challenges with different approaches to applying the algorithms. In addition, during the crossing of the levels, there are several, unpredictable and unexpected waves of enemies of increasing difficulty, and the player is called upon to repel them. Their existence aims at the constant vigilance of the player but also at the stimulation of his/her interest.
• Raising the player’s self-esteem: The game tries to reward the player and give him/her confidence in fulfilling the goal of the scenario. This is achieved by ensuring that at each level, he/she constantly learns at least one new, simple and understandable piece of information about the final goal which at higher levels are combined to reach a final conclusion regarding the main purpose. In addition, within the levels, when the player makes a right choice, a reward message appears urging the player to continue to the next challenge with more confidence.

3.2. Pedagogy

The pedagogy axis of the model is concerned with the following issues: the learning outcomes of the game and the way they are approached; how the game succeeds in motivating players to learn; how the game supports self-learning; and finally how the game promotes problem solving.

SpAI War is designed to provide players with an immersive and interactive learning experience that helps them develop a solid understanding of basic AI algorithms. Through practical application within the game’s context, players can learn how to use these algorithms and apply them to real-world scenarios. The game aims to improve problem solving, decision making, and strategic thinking skills while also increasing the knowledge and understanding of essential AI concepts such as decision trees, neural networks, and machine learning.

The game’s engaging and enjoyable experience motivates players to learn and fosters their curiosity, creativity, and critical thinking abilities. This approach to learning not only helps players retain information more effectively but also makes the learning process more enjoyable and fulfilling. SpAI War provides an accessible and effective means for anyone interested in AI to gain valuable insights and knowledge in an engaging and entertaining way.

The design of the game is based on the logic that each level is equivalent to a new challenge for the player that must be overcome in order to advance to the next level. In general, the script constantly tries to keep players on their toes by presenting each algorithm as the “ticket” to the next level. Taking as an example level 5 where the player is eventually imprisoned by Paraforce, the game prompts him/her to learn the steps of the Iterative Deepening algorithm so that he/she can “hack” the digital lock of the prison and continue to the next level.

A special effort was made to support the self-learning feature that a serious game should have. This is achieved by providing the necessary training material for each algorithm using examples of correct application, indications of the correct routes within the game areas by providing appropriate feedback when a mistake is made and with positive feedback messages after the completion of a task.

At the start of each level, the player is provided with the necessary educational material. Specifically, the player is informed in detail and with at least one example about the algorithm he/she is about to encounter, as shown in Figure 5.

When the player makes a mistake while applying the algorithm, feedback is provided. The player is presented with a message with the mistake he/she made, and the level is restarted, as it is assumed that the player has not fully understood the algorithm. For example, suppose the player is asked to traverse a series of rooms according to the steps of the Breadth First Search algorithm. In case the player has not crossed all the rooms of the N level of nodes–rooms and passes to N + 1, he/she will see the error message presented in Figure 6. As soon as the player completes a task, a confirmation message appears (Figure 7), which also informs him/her about the continuation of the scenario.

The game supports problem solving by making the player face three different types of tasks that are directly related to the game scenario, thus making the learning process more effective.

Tasks of correctly traversing areas of the map: Here, the player, after studying the theory of each algorithm, enlists his/her spatial intelligence to apply the steps of the algorithm while traversing the specially configured areas. For example, at the level corresponding to the Depth First Search (DFS) algorithm, the player is first informed by the message of Figure 8 about the tree arrangement of the rooms of the space and then asked to apply the steps of DFS in order to find the exit, proceeding then to the next level. In Figure 9, we see a view of room A (node A) which provides access to rooms B and C (nodes B and C).

Tasks of correct selection of objects based on their position in space: Here, the player is asked to apply the steps of the algorithm by selecting an object based on its position in it. For example, in the level corresponding to the Alpha Beta Pruning algorithm, the player is in a forest in which many huts are present and has to choose which one to blow up in order to pass to the next level (Figure 10).

Multiple choice question answering tasks: The player answers multiple choice questions based on one of the game’s algorithms. In Figure 11, we see a multiple-choice question for the K-Nearest Neighbors algorithm.

3.3. Learning Content Modeling

The axis of learning content modeling refers to issues of syllabus matching and possibilities of scaffolding for comprehending the syllabus. In the following, we present for each level of the game the algorithms that the player is trained in, the number and the kind of activities included in it and how they relate to the game scenario, along with a game flow map. The algorithms presented in the game are considered to be fundamental in the field of AI and are essential for developing intelligent systems that can reason, learn, and make decisions. In addition, these algorithms are taught in higher education through various courses and applied by students in many assignments. An additional reason for the selection of this set of algorithms is that their steps can be easily represented graphically and thus become easier to understand.

Level 1 (see Figure 12): The player is in the NSA offices and is briefed on the game’s premise and structure. The player comes into first contact with the course system and is given an introduction to the concept of AI.

Level 2 (see Figure 13): The player is in a mountainous volcanic forest. This is an introductory level, as this is where the player is introduced to what the game’s Graphical User Interface (GUI) symbols represent as well as weapon handling. Moving forward, the player comes across a hidden portal on the opposite side of the mountain, which leads to the first Paraforce lab.

Level 3 (see Figure 14): The player is inside the first chemical laboratory of Paraforce in which he/she must first defeat the enemy NPCs and manage to find the exit. In this level, the player learns the Depth First Search algorithm based on the steps of which he/she has to cross the lab areas.

Level 4 (see Figure 15): As in Level 3, the player finds himself/herself in an even more sophisticated and dangerous nuclear laboratory, in which after defeating the enemy NPCs, he/she must traverse the rooms according to the steps of the Breadth First Search algorithm to find the exit.

Level 5 (see Figure 16): Here, the player has been imprisoned by Paraforce, and in order to escape, he/she will need to breach the prison door. The door is digital and can be breached using the Iterative Deepening (ID) algorithm. After learning the algorithm, the player is asked to “crack” the digital lock by correctly answering questions related to the ID algorithm.

Level 6 (see Figure 17): After the player manages to escape, he/she goes to a hidden NSA office where he/she is sent some documents which include information about some Paraforce buildings that he/she needs to visit. After being trained in the Best First Search algorithm, the player is asked to apply the steps of the algorithm to a series of questions in order to learn which five buildings to visit and in which order to visit them. In the same level, upon reaching the office door, the player learns that the NSA has decided to send a group of marines to invade three of the five buildings he/she found earlier. For security reasons, the message is encrypted, and in order to be able to decrypt it and find out what those buildings are, it applies the steps of the A Star algorithm after first being trained on them.

Level 7 (see Figure 18): The player is in a forest similar to Level 2 but this time filled with huts. Here, the player is trained on the Mini–Max algorithm. As soon as the player learns the steps of the algorithm, he/she is informed that there is no weapon available. To be able to acquire weapons, the player must visit the huts in which, after solving the problem of the Mini–Max algorithm contained in each one, he/she will have the possibility to acquire one of the four available weapons so that he/she can defend himself/herself from the attacking NPCs. As shown in Figure 10, the arrangement of huts in the forest is tree-like, and each hut corresponds to a node from A to G. When the player reaches the last hut, he/she is informed that he/she may have left footprints in one of the previous huts. After first being trained in the steps of the Alpha Beta Pruning algorithm, he/she applies them to find the dangerous hut and blow it up so that his/her footprints are lost.

Level 8 (see Figure 19): In this level, the player is introduced to Machine Learning and its types. He is initially informed that he/she is in the fourth of the buildings he/she found in level 6, which is a small cottage of the Paraforce leader. The player is further prompted to proceed toward the entrance of the house to breach the alarm. After being educated on the concept of Machine Learning and its types, the player proceeds to the door, where answering a Machine Learning-related question disables the alarm. After entering the house and defeating the enemy NPCs, the player proceeds to the library where he/she finds a document containing information about the target building where the Paraforce databases are housed. After first being trained on the K-Nearest Neighbors algorithm, the player applies its steps to find the building with the servers.

Level 9 (see Figure 20): The player reaches the last building in the game where he/she will destroy sensitive government data. Upon entering the building, he/she is trained in the steps of the K Means algorithm, which he/she will then be asked to implement to cause a “short circuit” destroying the servers along with the classified NSA data. The player is presented with a completion message and the game concludes.

In summary, the player at each level is supported in building the knowledge he/she receives by studying the necessary educational material which is equipped with examples and counterexamples for each algorithm. The scenario and the training process are interrelated, making a full understanding of each training material a necessary condition for the completion of the game. Errors are allowed and, most importantly, explained with corresponding messages that indicate where the error occurred and which rule of the algorithm was violated. Accordingly, there are also messages rewarding the player, encouraging him/her to continue and boosting his/her morale.

4. Materials and Methods

4.1. Context of the Study

The study presented in this article was carried out at the Department of Applied Informatics at the University of Macedonia in Greece. In the context of the study, an announcement was made during the summer vacation through the University’s Learning Management System (LMS) and also the social media of the first author. The announcement was made by the second author through an 8th semester elective course on the Didactics of Informatics and was also forwarded to the institutional email accounts of the students that had enrolled in the course in the spring semester. The announcement informed the recipients about a new 3D first-person shooter game called SpAI War that aimed at assisting players in learning basic AI algorithms and kindly asked them to voluntarily play-test the game (a link for downloading it was provided) at the leisure of their home and anonymously evaluate it through an online questionnaire (Google form). The online questionnaire included at the beginning an informative text, publicly available by the Institution’s Committee for Research Ethics, regarding the goal of the questionnaire and the study, data collection methods and storage, as well as the rights of the respondents. Following this informative text, the respondents had to provide their informed consent for their participation in the study in order to have access to the rest of the questionnaire.

4.2. Research Question

The goal of the study was to validate the design of SpAI War. In order to achieve this, we decided to evaluate the game in terms of player experience, the scenario of the game and the perceived short-term leaning of the AI algorithms incorporated in the game. The aforementioned evaluation data would give us the chance to investigate whether the factors that are considered important for serious game design were successfully implemented, and a balance between pedagogy and fun was achieved [11,12,13,14,15]. Consequently, the following research question was investigated:

How do undergraduates and graduates of Informatics evaluate the player experience, scenario, and perceived short-term learning of AI algorithms in the first-person serious game SpAI War?

4.3. Participants

SpAI War was play-tested and anonymously evaluated by fifty-eight persons. The majority of them were undergraduate students (N = 42, 72.4%), and the rest were graduates (N = 16, 27.6%). Half of the graduates (N = 8) were attending a graduate program, while one-third of them (N = 5) had finished a graduate program. The vast majority of the participants had studied or were currently studying Informatics (N = 56, 96.6%). The undergraduate students of the Applied Informatics Department involved can be considered a convenience sample, since they had attended an AI course, while in the context of the Didactics of informatics course, they had been instructed how to evaluate educational/serious games with the MEEGA+ model utilized in the study. Forty-seven of the participants were male (81%), and the rest of them female (19%). The age of the participants was in the range of 19 to 38 years old, while the mean age was 23.5 years (sd = 2.7). Information about the participants’ game play habits, their performance in the game, their perceived difficulties and interest in algorithms in general, AI algorithms and programming are presented in the Section 5.

4.4. Data Collection Instruments and Analysis

An online questionnaire prepared as a Google form was the main data collection instrument. The questionnaire was based on the MEEGA+ model and the corresponding questionnaire for evaluating educational games proposed by Petri et al. [18]. The reliability and validity of the model and the questionnaire have been confirmed in the literature and utilized in several studies [30]. Based on the MEEGA+ model, the questionnaire included questions for two quality factors, namely player experience and perceived short-term learning. The player experience quality factor comprises seven dimensions: usability; confidence; challenge; satisfaction; fun; focused attention; and relevance. Usability is further divided into five subdimensions: aesthetics; learnability; operability; accessibility; error prevention and recovery. The MEEGA+ model was considered ideal for the goals of the study, while it is an instrument that is both complete since it was designed based on an extended literature and validated for its reliability and validity. The questions of the MEEGA+ model are actually 5-point Likert-scale statements, and the respondent expresses his/her level of agreement for each one of them (1 = strongly disagree, 5 = strongly agree). The questionnaire also included demographic questions, questions about the participants’ game-play habits, difficulties and knowledge on algorithms and programming, and finally the scenario of the game and the inclusion of the educational material in it. The data collected were analyzed using descriptive statistics.

5. Results

5.1. Perceived Knowledge and Difficulties on Algorithms and Programming

The results on the participants’ perceived difficulties on algorithms and perceived knowledge and interest on programming are presented in

Table 2. Perceived difficulties with algorithms, knowledge and interest in programming.

Question	Mean	Sd	Median
How difficult is it for you to comprehend algorithms in general (e.g., sorting/searching and graph theory algorithms)? *	2.4	0.85	2
How difficult is it for you to comprehend AI algorithms? *	2.8	0.87	3
How would you describe the level of your programming knowledge? **	3.2	0.98	3
How would you describe your interest in programming? **	4	1.2	4.5

* 1 = not at all, 2 = a little, 3 = averagely, 4 = much, 5 = very much, ** 1 = very low, 2 = low, 3 = satisfactory, 4 = high, 5 = very high.

. The participants do not seem to face difficulties on comprehending algorithms in general (median = 2), while comprehending AI algorithms in specific is of average difficulty for them (median = 3). Finally, although the participants have a high interest in programming (median = 4.5), their perceived level of programming knowledge is considered just satisfactory (median = 3).

5.2. Game-Play Habits and In-Game Performance

In Figure 21, the participants’ game-play habits are presented. All the participants play digital games with the vast majority of them playing digital games daily (36%) or weekly (36%), which is a fact that strengthens the potential of using games for educational purposes as well. Moreover, the familiarity of the participants with entertainment games along with their interest in programming constitutes the sample ideal for evaluating both the player experience and perceived short-term learning in SpAI War. Since almost all the participants were experienced videogame players, we anticipated that they would easily highlight problematic issues regarding player experience and the scenario of the game. Moreover, due to their field of studies and their interest in programming, we anticipated that the participants would be able to locate potential problems with the educational material and tasks and the way they were interleaved with the game mechanics and the scenario of the game.

As can be seen in Figure 22, half of the participants completed all the levels of SpAI War, while the majority of them (69%) played the game for at least one hour (Figure 23: 29% played for 1 h, 35% played for two hours and 5% played for 3 h). This is considered important both in terms of the interest caused by SpAI War but also for the validity of its evaluation by the participants that devoted the necessary time in order to experience all the aspects of the game.

5.3. Perceived Player Experience

In

Table 3. Students’ perceptions on player experience in SpAI War.

Dimension	Subdimension	Statement	Median
Usability	Aesthetics	The game design is attractive (interface, graphics, etc.).	4
		The text font and colors are well blended and consistent.	4
	Learnability	I needed to learn a few things before I could play the game.	2
		Learning to play this game was easy for me.	5
		I think that most people would learn to play this game very quickly.	5
	Operability	I think that the game is easy to play.	5
		The game rules are clear and easy to understand.	5
	Accessibility	The fonts (size and style) used in the game are easy to read.	5
		The colors used in the game are meaningful.	5
	Error prevention and recovery	The game prevents me from making mistakes.	4
		When I make a mistake, it is easy to recover from it quickly.	4
Confidence		When I first looked at the game, I had the impression that it would be easy for me.	4
		The contents and structure helped me to become confident that I would learn with this game.	5
Challenge		This game is appropriately challenging for me.	4
		The game provides new challenges (offers new obstacles. situations or variations) at an appropriate pace.	4
		The game does not become monotonous as it progresses (repetitive or boring tasks).	4
Satisfaction		Completing the game tasks gave me a satisfying feeling of accomplishment.	5
		It is due to my personal effort that I managed to advance in the game.	4
		I feel satisfied with the things that I learned from the game.	5
		I would recommend this game to my colleagues.	5
Fun		I had fun with the game.	4
		Something happened during the game (game elements, competition, etc.) which made me smile.	4
Focused Attention		There was something interesting at the beginning of the game that captured my attention.	4
		I was so involved in my gaming task that I lost track of time.	3
		I forgot about my immediate surroundings while playing this game.	4
Relevance		The game contents are relevant to my interests.	4
		It is clear to me how the contents of the game are related to the course.	5
		This game is an adequate teaching method for this course.	5
		I prefer learning with this game to learning through other ways (e.g., other teaching methods).	5

, the median is presented for all the questions of the seven dimensions comprising the quality factor of player experience. We would like to remind that a five-point agreement scale was used, where 1 = completely disagree and 5 = completely agree. In Figure 24, the corresponding percentages for each level of agreement per question are presented.

As can be seen in Table 3 and Figure 24, the majority of the participants evaluated positively all the dimensions of the player experience quality factor, namely: usability, confidence, challenge, satisfaction, fun, focused attention, and relevance. For twenty-six out of the twenty nine items comprising the player experience quality factor, the level of agreement was in the range of 63% to 96%, and the median value was 4 or 5, definitely showing a positive evaluation of the corresponding aspects of player experience in SpAI War. A median of 2 was recorded for the item “I needed to learn a few things before I could play the game”, since only 19% agreed or totally agreed with this statement. However, this is actually a positive result, since it means that the game has a short learning curve. Finally, half the students stated that they “were so involved in the gaming task that they lost track of time” (50%, median = 3) and “forgot about their immediate surroundings while playing this game” (51%, median = 4). Although this is an indication that an effort should be made to include game mechanics or improve the scenario in order to further immerse players, this result can also be considered quite positive. As Xinogalos and Satratzemi [9] state, achieving a high level of immersion in a serious/educational game is rather challenging, and this is confirmed by the evaluation results of several games evaluated through the MEEGA+ model that usually achieve a median of 2 or 3 in the aforementioned items of the focused attention dimension.

5.4. Game Scenario

The scenario plays an important role in the success of a game whether it is an entertainment game or a serious/educational game [10,31,32,33]. Although the scenario has an impact on various aspects of the player experience quality factor as defined in the MEEGA+ model, such as focused attention, challenge and fun, we decided to directly investigate the participants’ perceptions on the scenario of SpAI War. In

Table 4. Students’ perceptions on the scenario of SpAI War.

Statement	Median
The scenario of the game raised my interest.	4
The elements of the scenario (i.e., attacks, zombies) give a player experience analogous to that of entertainment game.	4.5
The way of integrating the educational material in the game is completely attuned to the scenario.	4
The scenario and the graphics are appropriate for the age of the players that the game is targeted to (students or generally adults interested in the field of AI).	5

, the median is presented for all the statements/questions on students’ perceptions on the scenario of the game, while in Figure 25, the corresponding percentages for each level of agreement per question are presented.

The results showed that the scenario “raised the Interest” of 76% of the participants (median = 4), while “elements like the attacks and the zombies gave a player experience analogous to that of an entertainment game” for 86% of the participants (median = 4.5). The same percentage of participants stated that “the way of integrating the educational material in the game is completely attuned to the scenario” (median = 4), which is rather positive since interleaving successfully the educational material and the scenario and achieving a balance between fun and pedagogy is considered to be challenging [14,15]. Finally, the participants strongly agree that “the scenario and the graphics are appropriate for the age of the players that the game is targeted to” (median = 5).

5.5. Perceived Short-Term Learning

In

Table 5. Participants’ perceptions on short-term learning in SpAI War.

Statement	Median
I learned something new with this game.	4
The game helped me comprehend the basic AI algorithms presented in its context.	5
The activities of the game helped me comprehend my errors and misconceptions in relation to the AI algorithms.	5
During game play, I devoted time to reading the information provided for the algorithms.	5
The game allowed for efficient learning compared with other activities in the course (such as a tutorial or textbook).	5
The knowledge of AI acquired through the game can motivate someone to study further the subject.	5

, the median is presented for all the questions (statements) on the participants’ perceptions on short-term learning, while in Figure 26, the corresponding percentages for each level of agreement per question are presented.

The perceived short-term learning quality factor received the most positive evaluation among the factors investigated. The vast majority of the participants agreed or strongly agreed that they “learned something new” (median = 4), while “the game helped them comprehend basic AI algorithms” (median = 5) and also “their errors and misconceptions in relation to the AI algorithms” (median = 5). Moreover, the participants stated that “the game allowed for efficient learning compared with other activities in the course (such as a tutorial or textbook)” (median = 5), while it “can motivate someone to study further the subject” (median = 5).

6. Discussion

SpAI War is a new 3D first-person shooter game that aims to support players in comprehending basic AI algorithms. SpAI War was designed taking into account the Educational Games (EG) design framework [11] that combines game design, pedagogy and learning content modeling. Special attention was paid to integrating the educational material in the scenario of the game. This was accomplished by requiring players to actually comprehend and apply AI algorithms for finding the right path for accomplishing specific tasks in the game or avoiding enemy attacks and so on. In order to evaluate the design of SpAI War in terms of its player experience, scenario and perceived short-term learning, undergraduates and graduates of Informatics were invited to play-test the game in an informal setting (at their homes) and anonymously evaluate it through an online questionnaire based on the MEEGA+ model [18].

The results showed that SpAI War was positively evaluated by the participants in all the dimensions of the player experience quality factor. The dimensions that were more positively evaluated are: the relevance of the game contents to the players’ interests and AI as a subject, the adequacy of the game as a teaching method and its preference as a learning tool in comparison with other teaching methods (with a median of 5 for three items and a median of 4 for one item); and the satisfaction of the players through the accomplishment of the game tasks, their advancement in the game through their personal effort, and the things learned that persuaded them to recommend the game to their colleagues (with a median of 5 for three items and a median of 4 for one item). Positive results were recorded for the dimensions of usability, confidence, challenge, and fun as well (with a median of 4 or 5 in all the items). SpAI War managed to immerse players at a high degree, and good results were also recorded for the dimension of focused attention (with a median of 4 for two items and 3 for one item), which is not an easy task in serious/educational games [9].

The scenario of the game and the perceived short-term learning were positively evaluated with the vast majority of the participants stating that they agree or strongly agree that all the corresponding items were successfully implemented. The aforementioned results showed that SpAI War achieved a good balance between pedagogy and entertainment/fun, which is considered to be one of the most challenging aspects in designing serious/educational games [12,13,14,15]. In our opinion, this can be attributed to the way that the educational material was integrated into the scenario of the game. Specifically, the fact that the player has to comprehend and apply AI algorithms in order to navigate in the game and advance its scenario (finding the right path or an exit, avoiding enemies, etc.) resulted in keeping players in the virtual world of the game and not moving them away from it when dealing with the educational aspects of the game.

7. Limitations and Future Work

Although the results of the evaluation were rather positive, it is clear that the study presented has some limitations. The main limitation refers to the approach utilized for evaluating the effectiveness of the game. The results regarding the player experience and the scenario of SpAI War can be considered to be solid, since they refer to subjective aspects of a game, and all the participants are experienced game players. Furthermore, it is promising that the participants considered that they learned through game playing, but this has to be confirmed with a different experimental design for an objective assessment of actual learning. Specifically, a control and experimental group study could be designed in the future, while a pre-test and post-test could be utilized for investigating potential statistically significant differences in students’ performance. Moreover, incorporating learning analytics capabilities to the game [8,9] would give us the chance to investigate the interactions of players with the game elements and draw even more interesting conclusions both in terms of player experience and learning. Learning analytics could help us record data: that indicate how often, when and for how long players resorted to the educational material (theory, examples) and if this affected the learning outcomes; how many played some levels more than once, and if they improved; which levels/tasks, and consequently AI algorithms, caused more difficulties to players and so on.

Furthermore, taking into account that the available games for AI [27,28,29] and SpAI War adopt a different game-based learning approach for familiarizing players with AI concepts and algorithms, it would be interesting to carry out further research for: investigating the actual long-term learning in all four games; comparing the learning effects in these games and drawing conclusions on the most effective game-based learning approaches and types of games for AI. This would also give the chance to compare the game-based learning approach with the traditional approach for the field of AI in terms of learning effectiveness, which is currently not possible due to the lack of related research.

8. Conclusions

Designing an effective serious game is not an easy task. Especially in the case of a serious game targeted to learning, there are several factors that have to be taken into account [11], while achieving a good balance between learning and fun is rather challenging [12,13,14,15]. In this paper, relevant literature was taken into account in order to design a serious game for learning AI algorithms called SpAI War. Although several games have been developed for cultivating algorithmic and computational thinking [5] and learning programming [6,7,8,9,10,26], very few games have been developed for learning algorithms [25] and more specifically AI algorithms. This made the design of SpAI War even more demanding. In order to design the game, relevant literature was taken into account [11,12,13,14,15], while the whole process was guided by the Educational Games (EG) design framework [17]. Special attention was paid in interleaving the educational content and the tasks with the scenario of the game with the aim of achieving a balance between playing and learning [12,13,14,15]. In this sense, the player has to comprehend and apply AI algorithms as part of game playing, that is for finding the correct path or avoiding enemies, in an attempt to keep the player engaged. In order to validate the design of the game, it was play-tested and evaluated by undergraduates and graduates of Informatics in terms of player experience, the game scenario and perceived player experience using the MEEGA+ model and the corresponding questionnaire [18,30]. The results were rather positive, validating the design of the game. It is clear that the game has to be further evaluated in terms of its actual learning effectiveness. In spite of this fact the study presented is considered to contribute to: the small number of serious games for AI, including Obsolescence [27], Maestro [28] and Sokoban Solver [29], by proposing a new 3D first-person shooter game for learning fundamental AI algorithms; the limited research on evaluating serious games for AI [27,28] by proposing the use of a serious game design framework [17] as a means of formative evaluation and an established instrument [18,30] for a summative evaluation of player experience and perceived short-term learning; and mainly the field of designing serious games for AI with a focus on interleaving the educational material/tasks with the game scenario, having the ultimate goal of achieving a balance between learning and entertainment [12,13,14,15].