Spectral Analysis of Electroencephalographic Data in Serious Games

Abstract

In this paper, we describe an investigation of brain activity while playing a serious game (SG). A SG is focused on improving logical thinking, specifically on cognitive training of students in the field of basic logic gates, and we summarize SG description, design, and development. A method based on various signal processing techniques for evaluating electroencephalographic (EEG) data was implemented in the MATLAB. This assessment was based on the analysis of the spectrogram of particular brain activity. Changes in brain activity power at a characteristic frequency band during the gameplay were calculated from the spectrogram. The EEG of 21 respondents was measured. Based on the results, the respondents can be divided into three groups according to specific EEG activity changes during the gameplay compared to a relaxed state. The beta/alpha ratio, an indicator of brain employment to a mental task, was increased during gameplay in 18 of the 21 subjects. Our results reflected the sex of respondents, time of the game and the indicator, and whether the game was successfully completed.

1. Introduction

Games have played an important role in human development and evolution [1]. People encounter the term game early on in their childhood, mainly in its practical form. They are an inseparable part of a child’s life [2]. We understand the term game as representing any type of game that a child plays, either in a group or individually, such as sports games, non-digital games, movement games, psychomotor, cognitive or digital games where the goal is to socialize, educate, or enjoy free time.

Such developments brought new possibilities including use in education. We are in line with [3], who states that information and communication technologies (ICT) have become indispensable technologies in our daily lives. The massive development and spread of ICT has brought new possibilities that can be applied to education as well [4,5]. According to [6], ICT can complement, enrich and transform education for the better. Fraszczyk and Piip claim that digital learning could have many equivalents, such as: mLearning, online learning, eLearning or distance learning. All of them mean a similar type of learning, that is, out of the classroom and using technology [7].

E-learning, or electronic education, can be considered a method of teaching and is part of many people’s daily lives, improving knowledge and skills through the use and integration of modern information and communication technology [8,9,10,11,12,13]. One form of E-learning that is increasing in popularity is serious games (SGs) [1,14]. In line with [15], SGs have become an area of intense research interest.

Clark Abt [16] was the first author to use the term SG. He described them as teaching or training devices for students of different categories and for use in many different situations. He said that SGs are motivating and that they communicate the concepts of subjects efficiently. The term “serious” was used to imply a form of study, which requires interest and attention, raising questions not easily explained or decided and having potentially crucial results. In addition, it was claimed that educational training or planning, analysis, research and evaluation are fertile areas for the use of SGs.

SGs and magnification were defined in [17] as the use of games or gaming elements for purposes more serious than mere “entertainment.” In recent years, they have been gaining more attention in many fields and are useful for different groups. It is claimed that SGs have become a viable educational tool that can provide an enjoyable surrounding as well as a space for training, teaching and tutoring pupils, students and employees of all ages [18]. In a previous study [19], we found that SGs can bring people together through location-based experiences. In the workplace [20], an SG is defined as a video game with a purpose other than pure entertainment. Over the last 10 years, an increasing number of SGs have been released, focused on a range of fields. At work [21,22], several authors define SGs as simulations of the real world, with actions developed for training or solving issues. We are in agreement with [23] that SGs represent a substantial possibility in the 21st century for educators, teachers, and coaches to enhance their educational toolkit. According to [24,25,26,27,28,29], corporations, schools, universities, and families can use SGs and gamification elements to improve learning.

The success of SGs is dependent on the individual personality of the user. It has been recommended that pedagogical principles be integrated with the game to achieve learning success [30]. Additionally, we are witnessing the transformation of society into a “digital society,” meaning that the integration of SGs will be a part of everyday life for students and will be become more pervasive.

The remainder of this paper is structured as follows: Section 1 presents a literature review focused on SGs and related works; Section 2 presents game design and a description of SGs; Section 3 presents methods and materials (e.g., hardware, procedures, respondents, and signal processing); Section 4 discusses the data obtained, providing the results and an interpretation of their meaning; and Section 5 concludes the paper and suggests future work.

1.1. Related Work

Mathematics is one of the most important subjects in society [31,32]. It provides us with an understanding in various fields such as technology, engineering, and science. Improving mathematical education via the teaching of mathematical rules and the connections among them plus improving the explanation of the curriculum is part of the didactics of mathematics. It is important for our society and modern civilization to support and develop effective, useful, and practical forms of mathematical education [33].

The level of mathematical skill and competence among students plays a crucial role in our society and is intricately linked to economic performance [34]. In addition, having an understanding of the neural basis of processes in mathematical education could play an important role in improving the quality of education [34].

The teaching of logical thinking should not follow a strict formula. It should be a type of teaching that features some creative thinking, problem-solving, and critical thinking. In addition, it should give students motivation and help them experience the joy of learning. Feelings of pleasure arise when students have an intrinsic motivation to learn [35].

SGs can be defined as an innovative and interesting E-learning tool to improve long- or short-term memory and cognitive abilities. They are beneficial in the training of the cognitive abilities of students [36,37,38], physical training [39,40] and both prevention and physical rehabilitation [41,42]. Recent studies found that some aspects of cognition, namely multitasking and spatial cognition, can be enhanced by gameplay [43,44]. Other recent studies have investigated different aspects of the field using electroencephalographic (EEG) data analysis including the role of opening situations using a set of recommended plans or goals for motivation in mathematics [33], long-term memory of 2D and 3D educational content [15], and learning to solve mathematical tasks by applying divisibility rules [45]. The studies indicated that mental workload involved in mathematical cognition is linearly related to the absolute level of neural activity in the cortex [33,46].

SGs are a growing market and an exciting area for inter- and multi-disciplinary academic research [46]. EEG measurement in education has become more conventional in serious testing [47]. With brain- or neurofeedback, individuals learn to self-regulate their neural activity following operant conditioning. Real-time feedback (visible or auditory) is given based on an individual’s current brain activity and the desired activity could be rewarded while another brain activity could be punished. Respondents receive information on which cognitive states are targeted but not how to reach the state or states [48]. It has been shown that stress or boredom (e.g., activity in the amygdala of the brain) occurs unless some information from the brain is connected with positive emotions that are absorbed by the hippocampus and transferred to the cortex for further processing [49]. This is the reason why content in SGs should be created to activate the hippocampus and not the amygdala—it should evoke interest and positive emotions. Another work [50] analyzed the usability and acceptability of an EEG analysis on the relationship between pre-learning stress, the frontal lobe and long-term memory. These studies provide evidence that SGs have a positive effect on the cognitive load [51,52]. In another study [53] that used a sample of 17 comparisons, it was reported that SGs are helpful to both learn and teach cognitive skills. In a review of 46 research studies, SGs were described as useful for students to train and obtain cognitive skills and had an overall positive affect [54].

1.2. Contributions

In this paper, we focus on the study of SGs from the perspective of logical thinking, specifically on the cognitive training of students in the field of basic logic gates. In SG research, beta waves are the focal point of research. The beta/alpha ratio is a valuable indicator of the brain’s response to specific mental activity. When SGs are well designed, the beta/alpha ratio increases significantly.

The presented research can be seen as a pilot case study for when a teacher with an SG wants to work on teaching and tests whether the SG activates beta waves in students. Put another way, if the teacher wants the beta waves of the students to be activated sufficiently to increase the cognitive process in the classroom, they can proceed in a similar way using an SG with logic gates.

This study aims to investigate and analyze the subject’s brain activity during the SG gameplay, employing time–frequency analysis. If possible, the subjects will be divided into groups according to the specific spectral features. The second aim is to verify if the beta/alpha ratio is a valuable indicator of the brain’s response to SG gaming and investigate the change of beta activity in game levels with increasing difficulty.

This study describes the design and development of an SG focused on cognitive training in the field of basic logic gates. Additionally, signal processing methods were used for the evaluation of EEG data in the MATLAB programming language (ver. R2020b, MathWorks, Natick, MA, USA) and a spectral analysis of the brain activity was carried out.

2. Anatomy of Serious Game Development

Creators of SGs should review the literature to improve designs, development, and/or game testing methods [55]. Sometimes teachers may have a problem determining the right way to explain subject matter to students in their classes and this would be the right moment to integrate an SG as a new and innovative teaching method for improved understanding [56].

Serious Game Description

In this study, we thoroughly describe the design and development stages of an SG for cognitive training. The design and development of the SG was focused on the students’ cognitive training in the field of choosing basic logic gates. In the work of [57,58], the authors explained the following logic gates, which were used in the SG:

• AND: This gate is called a logical product. This logic gate works as follows: 0 is false and 1 is true. The output is 1 (or true) when both inputs are 1, otherwise the output is 0 (or false). The mathematical notation is Y = A × B.
• OR: This logic gate is called a logical sum. This logic gate works as follow: The output is true if either input or both inputs are true. If both inputs are false, the output is false. The mathematical notation is Y = A + B.
• XOR: The exclusive-OR (XOR) gate acts in the same way as the logic either/or. The output is true if either, but not both, of the inputs are true. Another way of looking at this circuit is to observe that the output is 1 if the inputs are different, but 0 if the inputs are the same.
• NOT: A logical inverter, sometimes called a NOT gate to differentiate it from other types of electronic inverter devices, has only one input. It reverses the logic state. If the input is 1, then the output is 0. If the input is 0, then the output is 1.

Cognitive training of logical thinking is realized through logical puzzles, which are placed in different rooms and places in the virtual world of the SG. To avoid the game becoming monotonous or boring, we tried to design and develop an historical magical virtual environment. The attractiveness of the design should help maintain the player’s attention. The puzzles in the game have principles based on chosen logic circuits. These circuits work on the principles of Boolean algebra. The main task of this 3D SG is to configure the inputs of the circuit so that an imaginary flow occurs through the entire circuit up to the circuit’s outputs. The player can advance to the next level when the puzzle is solved correctly. In the SG, logical characters are represented by the following objects:

• Cube: this is the only object in the SG that the player can directly manipulate. In combination with the “Power Generator” (PG) object, together they form a switch that controls whether or not an imaginary electric current flows through a circuit. If the cube is on the PG, electric current enters the circuit; if the cube is outside the PG, no electric current passes through the circuit.
• Logic gates: there are four types of logic gates—namely, and, or, not, and xor. Each logical gateway has one output and two inputs.
• Power generator: represents the input to the circuit. There are up to six types of power generators in the SG as well as the cubes and each has a different color. Several cubes and power generators are used to make the individual circuits cohesive, i.e., there are several logical circuits separated from each other in each level meaning the player cannot use the cube for a red circuit as an input for a different circuit, for example a blue circuit. This means the color of the cube and power generator must always match.
• Cable: representation of the electric current flowing through the logic circuit. When the electric current is flowing, the cable glows with the color of the logic circuit.
• Door: the entrance to the next level of the SG. The door will not open until all logic circuits in the level have been correctly resolved.

The SG was divided into three levels. Each of them represented a particular difficulty level: easy, medium, and hard. The first level, “The Garden,” is an introductory level to the game. This level is situated outside, in the garden. The player gets acquainted with the controls of the game’s character as well as the mechanics of the SG. At the beginning of this level, the first puzzle is introduced to the player, which acquaints them with the possibility of interaction between the cube and the PG (designed for placement of the cube). If the player wants to progress in the game, the player must place the cube on the PG. After that, the second puzzle is presented. This is a variation on the first puzzle where cubes of different colors are introduced to the player. An interaction can only occur between a cube and a PG of the same color. A cube successfully placed on a PG means that current is flowing through the circuit. When the current reaches the closed door, the door opens. In the third puzzle, the first logical gate “OR” is introduced and the player has two PGs and one cube. It does not matter which PG the player places the cube on. Then follows the logical gate “AND” with two PGs and two cubes, where one cube must be placed on each PG. In the last puzzle of the first level, the logical gate “NOT” is introduced. The player must remove the cube from the PG to open the door to the next level. The following two levels are indoors. The second level is named “Four Doorways”, where the player encounters more challenging puzzles, which they have to solve in order to advance to the more complex logic circuits which open the portal or door to the next level. The last level is “The Big Hall”, the most challenging level of the SG. This game level consists of just one logic circuit composed of many logic gates. This level has many more inputs than the previous circuits/levels. After completing it, the door or portal will open for the player. By passing through the portal, the game ends, and a screen congratulating the player on the successful completion of the game is displayed.

When the SG was designed, we divided the development process into the following four stages:

In the first stage, we analyzed the current trends in SGs focused on a similar topic and the development of logic gates. We also analyzed other SGs with a similar theme but with different game genres. After that, we chose the game components that were to be implemented within the game. We also identified the basic parameters and specifications of the game by selecting the game genre, game environment (2D/3D) and identifying the target group, amongst others.

The second stage involved making an SG proposal. After analyzing similar games focused on the same topic, we started working on the game’s design. Discussions in the scholarly community and between teachers during this stage were conducted. Finally, a design approach was formulated. In line with [59], we first created a paper and pencil wireframe and designed the architecture of the game. A pre-prepared game prototype was subjected to an initial review by chosen experts. The SG was created and developed for the PC platform. In a previous study [59,60], the iterative development of user interfaces involved a gradual refinement of the design based on user testing and other evaluation methods. Typically, one would complete a design and note the problems several test users experienced whilst using it. These problems would be fixed in the following iteration, which was again tested to ensure that the fixes had solved the problems and to find any new usability issues introduced by the modifications. The design of the game should be primarily attractive and exciting for students. Methodologists, didactic pedagogists and teachers provided critical feedback and provided suggestions to improve the game’s attractiveness and maintain interest.

The third stage entailed the development of the SG including programming, creating composite 2D and 3D graphic objects, building the environment (Figure 1A–C), animation development, and the graphics for controlling the game (e.g., head-up display, pause menu, position control). The design of the elements was created with the use of external software for editing 2D and 3D graphics.

The fourth stage involved testing the game. After finalizing the game’s development, we tested the software functionality and quality by detecting bugs. White-box and black-box testing was used. White-box testing was done with the knowledge of the program’s internal structure and the source of the game. Tests from this group tried to cover as much of the source code as possible. We included the following test methods in this group: module testing, static and dynamic source code analysis, covering commands, covering branches [61]. Black-box testing was done without knowing the game’s internal structure and did not require programming knowledge. External testers carried this task out and detected some bugs, which were fixed. In the next phase, respondents were tested by electroencephalogram as a non-invasive method. The electroencephalogram test was used to evaluate brain electrical activity and if brain beta waves were recorded while playing the game.

3. Materials and Methods

The following subsections describe hardware used to acquire EEG data, electrodes’ configuration, and signal processing methods used for data analysis. The last subsection deals with the specification of the subjects who participated in the experiment.

3.1. Hardware

As a data acquisition system, the BIOPAC MP36 units (BIOPAC Systems Inc., Goleta, CA, USA) were used (Figure 2A). The precise built-in universal amplifiers and 24-bit A/D converters compose the BIOPAC MP36 unit. It is designed to record a broad spectrum of physiological signals such as electrocardiography (ECG), electromyography (EMG), electroencephalography (EEG), etc. A single BIOPAC unit is capable of measuring a total of four channels. Data synchronization between multiple units is assured by the sync input located on the rear side of the BIOPAC unit. The acquisition unit provides the control input of the electrode, which is reasonable when calculating the impedance between the electrode and the skin. The skin–electrode impedance should be kept under the value of 10 kΩ during the measurement to record data with adequate quality.

The headbox acted as a convenient interconnection between the acquisition units and the EEG electrode cap. We built a headbox (Figure 2B) for up to five BIOPAC acquisition units and a regular EEG cap with 19 electrodes terminated by a 25-pin connector. Each channel was connected to the BIOPAC unit via a shielded cable that was terminated by a cinch connector and a nine-pin connector on the headbox side and the acquisition unit’s side, respectively. In accordance with the international 10–20 scheme, channels (cinch connectors) on the headbox were situated with 19 EEG electrodes. The synchronization button was located on the headbox and enabled all the linked units to start the acquisition at simultaneously. The structure of the 8-channeled EEG system is shown in Figure 3, and the electrode localization is shown in Figure 4. The unipolar EEG measurement configuration was used while the neutral electrode was formed by connected and grounded ears (Figure 4). The sampling rate of the data acquisition was 1 kHz.

3.2. Signal Processing

In a wide range of applications, e.g., epileptic seizure detection or prediction, sleep state classification, motor imagery classification, emotion recognition, and mental task classification, digital signal processing of EEG plays an important role [62]. Most valuable knowledge about the functional stale of human brain information is encoded within six major brain waves, differentiated by their frequency bands. Thus, EEG signal is usually analyzed in the time-frequency domain utilizing the Short-Time Fourier Transform or a Wavelet Transform. We can decompose the EEG signal into different frequency components; delta (1–4 Hz), theta (4–8 Hz), alpha (8–12 Hz), beta (13–30 Hz), gamma (30–80 Hz), and high-frequency gamma waves (>80 Hz). [63] However, the exact frequency ranges associated with these terms differ across studies.

The Power Spectral Density (PSD) refers to the distribution of the signal power over the frequency. The PSD estimation P of the random process is defined as follows

$$P=\frac{1}{N}{\left|X\left[k\right]\right|}^2,$$

(1)

where N is the number of signal samples and X[k] is a Discrete Fourier Transform (DFT) of the discrete input signal x[n]

$$X\left[k\right]={\displaystyle \sum }_{n=0}^{N-1}x\left[n\right]e^{\frac{-2\pi }{N}kn},$$

(2)

where k is the frequency bin number and n is the discrete time. The k index in Equations (1) and (2) can be recalculated to frequency f by using the following formula

$$f=k\frac{f_s}{N}\left[Hz\right],$$

(3)

where fs is the sampling frequency.

The Equation (1) is known as a periodogram (Schuster method), which computes PSD for the entire signal. Ref. [64] The Welch method is an improvement on the periodogram—the signal is split up into K overlapping segments, which are then windowed. After that, the partial periodograms P_i are calculated by using Equation (1) for each segment. The partial periodograms are then averaged, which reduces the variance of the individual power measurements.

$$\mathrm{PSD}=\frac{1}{K}{\displaystyle \sum }_{i=0}^{K-1}{P}_i\left(k\right).$$

(4)

The short-time Fourier transform (STFT) is used to analyze non-stationary signals (EEG, ECG, speech, etc.) when the statistical parameters are change over time. The use of DFT does not allow the investigation of spectrum changes over time. The STFT is able to determine the frequency spectrum of the signal as it changes over time. The STFT can be defined with the help of DFT as follows

$$X\left[m,k\right]={\displaystyle \sum }_{n=0}^{N-1}x\left[n+mR\right]w\left[n\right]e^{-j\frac{2\pi }{N}kn},$$

(5)

where x[n] is the discrete input signal amplitude at discrete time n,

N is the number of discrete-time samples,

[n] is the window function with a length of N,

X(m,k) is the spectrum of windowed signal x at a discrete frequency k,

R is the hop size (in samples) between successive DFTs,

m is the index of a window from 0 to M − 1, where M is a number of windows.

The k index in Equation (5) can be recalculated to frequency f using the Equation (3). Similarly, the m index can be changed to time t in seconds

$$t=m\frac{N}{f_s}\left[s\right].$$

(6)

The spectrogram is a graphical interpretation of STFT. The spectrogram can be defined as

$$\mathrm{spectrogram}=10 {log}_{10}{\left|\mathrm{STFT}\right|}^2\left[\mathrm{dB}\right].$$

(7)

The spectrogram presents the power spectrum of the signal as a function of time. The decadic logarithm is used to decrease the dynamic range of values; therefore, the resulting spectrogram is in decibel (dB) units.

A group of signal features was derived from the power spectra estimates during the resting phase and gameplay. The subjects have been classified by utilizing the k-means clustering algorithm into three groups based on those extracted features. The k-means algorithm iteratively allocates all observations to the nearest cluster while reducing the intra-cluster dimension (e.g., squared Euclidean distance) [65].

3.3. Subjects

In total, 21 university students (nine males and 12 females) with a mean age of 22.7 years (range: 21–26 years) were among the participants. The study sample consisted of students from the University of Zilina. Students voluntarily participated in the research, i.e., they could decide for themselves whether or not they would participate. They received an accompanying letter explaining the study and its purpose. Eight students (seven females and one male) could not finish the game because they could not solve a logical problem in the last level, and they ended a game before a successful finish. No students were excluded after the game finish and from the analysis as well, including eight students who did not complete the SG.

3.4. Procedure

Before the research began, subjects were asked to join the research and were informed of its purpose. Participants were examined individually in a quiet room. The lighting conditions were uniform for all the participants [66]. A SG was presented on a 24-inch LCD monitor. The participant’s eye distance from the screen was approximately 55–65 cm. Subjects played a SG in full-screen mode. Each subject was instructed to sit down and relax with opened eyes for approximately two minutes. Then the subject was asked to close their eyes and relax until he is instructed to open their eyes after approximately one minute. The subject was asked to start the game from the desktop. The game began, and the participant was then concentrated on gameplay. The specific actions are added to the EEG record as flags by pressing keyboard function keys. The flags are added by an examiner and then checked with a synchronized video record. The list and notation of flags in the EEG are introduced in

Table 1. List of flags in EEG record.

Flag	Description
E1	Beginning of Eye Closure
E2	End of Eye Closure
L1	Start of Level 1
L2	Start of Level 2
L3	Start of Level 3
R	Last Puzzle Solved
END	End of Game

. The meaning of flags is explained in more detail in the Results section.

4. Results

The EEG signals were processed in the MATLAB programming language. The movement artifacts were manually identified in the EEG record with the help of a synchronized video record. Ref. [67] The short periods of movement artifacts are denoted in the EEG, and these periods are excluded from spectral EEG analysis. The spectrogram of the EEG signal was computed by using Equations (5) and (7). The length of the window is set to two seconds (N = 2000 samples), windows are overlapped by one second (R = 1000), and the cosine Hamming function was used as a windowing function w[n]. The beta activity was dominant in the frontal and central regions; on the other hand, the alpha activity was dominant in the brain’s posterior regions. To analyze the subject’s brain activity during the gameplay, we used the C4 electrode in the central region as a compromise. Moreover, the C4 electrode was chosen via prior knowledge about all electrodes’ electric impedances during the measurement’s setup. This electrode featured the lowest electric impedance (<5 kΩ) among all electrodes and all subjects. By analyzing all measured subjects, we found out that the results are very individual across the subjects. Beta activity is responsible for active thinking and the stage of high alert. On the other hand, the alpha activity was typical of a relaxed state, increased with closed eyes. Therefore, we focused on analyzing the beta and the alpha activity during relaxation with closed eyes and gameplay. We can divide subjects into three generalized groups based on the type of spectrogram. All three types are discussed further in the following subsections.

4.1. Group of Subjects 1

The spectrogram of the C4 channel of subject 1 is shown at the bottom of Figure 5. This type of spectrogram represents the most examined subjects (10 from 21 subjects). Alpha activity in the frequency range from 8–12 Hz is clearly visible while eyes are closed (between E1 and E2 flags), and the signal spectrum is diffused to the whole brain frequency sub-bands during the gameplay (between L1 and END flags). These two features are typical for this group of subjects. In Figure 5, the alpha activity at a frequency around 10 Hz is clearly seen as a yellow horizontal stripe in the segment of closed eyes. There is an interval of attenuated brain activities at the end of the gameplay. This interval stands for walking to the game exit after the last logical problem is successfully solved, and no more active thinking is necessary. The R flag marks this moment. The artifacts are marked by red in the time domain and blue rectangles in the frequency domain.

The power of brain activity can be computed with the help of PSD estimation, e.g., by the Welch method using Equations (1), (2), and (4). The same parameters used to calculate the spectrogram can be used for this (i.e., Hamming window with a length of two seconds (N = 2000 samples) and one-second window overlap). If the power of specific brain activity (e.g., beta activity) is desirable; then, the signal is filtered by the band-pass filter to extract specific brain activity from the EEG signal before PSD computation. Thanks to the already computed spectrogram, the PSD can be computed in another, and in this case, a simpler manner. The spectrogram shows the power of the signal (PSD) in [dB] per each time segment (one-second resolution on the x-axis in Figure 5) according to Equation (7). The power of the signal is coded into spectrogram colors. Then, the PSD of beta activity at a selected time interval can be taken from the spectrogram in the frequency interval from 13–30 Hz. The average power of beta activity in a specified time interval is then the mean of the values in this frequency band. The progress of average beta activity in time is plotted in the graph shown in Figure 6. The movement artifacts are marked by orange color.

The average power of alpha activity is computed in the same manner. As seen from Figure 7, the alpha activity also raised during the gameplay when subject 1 is examined.

The beta/alpha ratio characterizes the relation between alpha and beta waves. The beta/alpha ratio increases when the brain is activated and decreases during brain inactivation. Therefore, this ratio is a significant parameter in evaluating brain response to some specific mental activity [68,69]. The development of the beta/alpha ratio during the experiment is shown in Figure 8. According to the graph, it is evident that the brain is more activated during the gameplay.

4.2. Group of Subjects 2

A representative spectrogram of the second group is shown in Figure 9B. The alpha activity (in the range from 8–12 Hz) in the closed eyes interval is more evident; it vanishes during the gameplay compared to the previous spectrogram (Figure 9A or Figure 5). In addition, the brain activity is not spread across entire frequency range during the gameplay, as in the previous case.

4.3. Group of Subjects 3

The spectrogram for this group of subjects is quite different from the previous spectrogram types. The alpha activity is missing or significantly attenuated in “closed eyes” (Figure 9C). This can be explained by the disability of the subject to relax because of stress before the gameplay, possibly caused by the fear of an unknown situation. There is only a slight increase in beta activity during the gameplay due to their low concentration during the game.

4.4. Results across Subjects

The three types of spectrograms that characterize each specific group of subjects were discussed in previous subsections. It is desirable to compare each individual’s brain activities in time intervals corresponding to the gameplay and the resting with closed eyes. The average brain activity results in calculating a single number that summarizes the power of specific brain activity in a defined time interval. For instance, let us calculate the average power of beta activity with the help of the graph in Figure 6. The average power of beta activity in some interval, e.g., during the gameplay, can be computed as

$$\overline{\beta }=\frac{{{\displaystyle \sum }}_{k=0}^L{\mathrm{PSD}}_{\beta }\left[k\right]}{L},$$

(8)

where PSD_β is the power of beta activity at discrete time k in the time interval with the length of L.

The $\overline{\beta }$ value is calculated for the three game levels. Levels have increasing difficulty, so it is assumed that beta activity will rise. The change of beta activity in three levels is shown in Figure 10. The power of beta activity is normalized to the maximum value within levels to make the results comparable across subjects. Approximately half of the subjects showed an upward trend. Some of the subjects show a downward or fluctuating trend.

The beta/alpha ratio is a valuable indicator of the brain’s response to specific mental activity. The averaged power of beta/alpha ratio R is computed in the state of relaxation and during the gameplay by the following equation

$$R=\frac{\overline{\beta }}{\overline{\alpha }},$$

(9)

where $\overline{\beta }$ and $\overline{\alpha }$ are calculated by Equation (8). Figure 11 shows the beta/alpha ratio during the closed eyes and gameplay. According to the results, the beta/alpha ratio raised in all subjects except the last three subjects.

4.5. Clustering of Subjects

The subjects are divided into three groups according to the specific features identified in the spectrograms described above. The k-means clustering method is used for clustering based on these signal similarities. Three features are selected as an input to the k-means algorithm.

The first feature (F1) reflects the alpha activity’s presence during the closed eyes and its change during the gameplay. If we calculate average alpha activity during the resting $\overline{{\alpha }_R}$ and average alpha activity during the gameplay $\overline{{\alpha }_G}$, then we can calculate the difference of the alpha activity by the equation

$${\alpha }_{dif}=\overline{{\alpha }_G}-\overline{{\alpha }_R}.$$

(10)

If ${\alpha }_{dif}$ > 0, then the alpha activity increased during the gameplay compared to the state of eyes closed; otherwise, if ${\alpha }_{dif}$ < 0, then the alpha activity decreased during the gameplay. The similar computation as for alpha activity Equation (10) can be used also for beta, delta, and theta activity

$${\beta }_{dif}=\overline{{\beta }_G}-\overline{{\beta }_R},$$

(11)

$${\delta }_{dif}=\overline{{\delta }_G}-\overline{{\delta }_R},$$

(12)

$${\gamma }_{dif}=\overline{{\gamma }_G}-\overline{{\gamma }_R},$$

(13)

The F1 feature vector is normalized to interval <−1; 1> across all subjects.

The second feature (F2) describes the spread of brain activity to frequency sub-bands during the gameplay. At first, the brain activity differences computed by Equations (10)–(13) are normalized to the individual’s maximum value. Then, the mean of brain activity differences A in all frequency sub-bands is calculated for each individual

$$A=\frac{1}{4}\sum {\alpha }_{dif,norm}+{\beta }_{dif,norm}+{\delta }_{dif,norm}+{\gamma }_{dif,norm}.$$

(14)

If the A value is high, then the brain activity is spread to all brain sub-bands, typical for subject in group 1. The F2 features are normalized within the interval −1 to 1 across all subjects.

The third feature (F3) represents an increase in the beta/alpha ratio during the gameplay. The increase of beta/alpha ratio can be defined as the slope of a line connecting eyes closed and gameplay columns in Figure 11. If the slope is more than 1 degree, then the beta/alpha ratio is considered rising, and the F3 feature equals 1. Otherwise, the beta/alpha ratio is static or decreasing, and the F3 value is −1.

The result of k-means clustering in a three-dimensional space is shown in Figure 12. The k-means algorithm splits subjects into three groups with 10, 8, and 3 subjects in group 1, group 2, and group 3, respectively. The order of the subjects presented in this paper is according to the clustering results.

To demonstrate the classification’s correctness, we calculated the increase of brain activity in the delta, theta, alpha, and beta sub-band using Equations (10)–(13) for all examined subjects. The results are depicted in Figure 13. The power of sub-band brain activities is normalized by maximum value within the subject. The absolute values in Figure 13 cannot be compared across individuals because slightly different skin–electrode impedance will raise different signal strength and power. On the other hand, we are able to evaluate the increase or decrease of particular activity during the gameplay across subjects and the mutual ratio of brain activities within the subject. If the value of power is negative, then the brain activity decreased during the gameplay. During the gameplay, increased brain activity in all frequency bands is typical for subjects in group 1. A decrease of alpha activity is typical for group 2, and a negligible increase of beta activity compared to alpha activity is typical for group 3. As we can see from the bar plots, the delta and theta waves were significantly raised during the gameplay [70].

The increase or decrease of beta/alpha ratio during the gameplay compared to relaxed state can be calculated as the difference between ratios

$$R_{dif}=R_G-R_R,$$

(15)

where R_G and R_R are beta/alpha ratios during the gameplay and relaxation, respectively.

The graph of the increase of beta/alpha ratios for all subjects is shown in Figure 14. The negative or low values mean a decrease in the beta/alpha ratio, which is typical for group 3. The graph also includes the sex of subjects, game playtime, and the indicator if the game was successfully finished or not. The average time to successful game finish was 467 s (7 min and 47 s). The decrease of the beta/alpha ratio in subject 19 does not mean decreased attenuation (beta activity) during the gameplay. However, it reflects the highest increase of relaxation (alpha) activity compared to the beta activity across all subjects (see Figure 13). The one possible reason is that subject 19 likes to solve logical puzzles, which she argued after the game was over. This fact is supported by the shortest time of the game walkthrough. It is seen that seven of 12 women and one of nine men could not finish the game’s last level, which is the most difficult. Subjects 20 and 21 were very stressed before the gameplay, which they confirmed after the experiment. They could not relax and focus on the game, so the beta/alpha ratio is decreasing.

5. Conclusions

The EEG activity while playing an SG was investigated. We investigated 21 subjects (nine males and 12 females). The methods which use the standard signal processing techniques to evaluate EEG was implemented in the MATLAB programming language. The evaluation methods are based on spectral analysis of EEG. We investigated the change of beta activity in game levels with arising difficulty (Figure 10). We found that beta activity does not depend on the level of difficulty for all examined subjects. Subsequently, we focused on another important indicator—the beta/alpha ratio. The beta/alpha ratio is a trustworthy indicator of brain concentration or relaxation. During the gameplay, the beta/alpha ratio was raised almost in all examined subjects except three individuals (Figure 11). Based on the spectrograms, we found out that the results are very individual across the respondents. Even though the results are very individual, the subjects can be split into three specific groups based on characteristic changes in brain activities. The subjects were divided into three groups by the k-means clustering method (Figure 12). The rise of brain activity in all frequency sub-bands during the gameplay is typical for group 1. A distinct decrease of alpha activity is characteristic for group 2, and a very low or negative increase of beta/alpha ratio is typical for group 3. Future research will focus on increasing the number of subjects involved in an experiment to confirm the hypotheses described in this paper statistically.

The measured results clearly showed that a SG designed in this way activates beta waves of students solving logic puzzles and increases their brain activation and, consequently, their cognitive abilities. The conclusions of the study can be implemented in the educational process, but also in the creation of SG.

Based on this study, the teacher can be sure during the teaching process that the beta waves of students are activated and their cognitive activities are increased by including a serious game with logical elements in the lesson.

In future work, we plan to realize the research that will be focused on measurement and compare of beta waves of respondents. They will be divided into two groups: a control group (using paper and pencil) and experimental group (using a serious game). Both groups will have same conditions and they will solve same tasks focused on logic gates (AND, OR, NOR, XOR, etc.).