Introducing Sustainable Development Topics into Computer Science Education: Design and Evaluation of the Eco JSity Game

Abstract

With increasing awareness of the advantages of game-based learning, there is a growing number of studies showing its application to both computer science education and sustainable development education. In this paper, we describe, with the example of the Eco JSity application, how both of these areas can be combined into a single tool. The presented evaluation results showed that despite incorporating the sustainable development topics into both the theme and mechanics of the educational game, both the usefulness and ease-of-use of the application were still assessed positively by students. We also identified three clusters of students depending on their attitude towards combining education on computer science and sustainable development. We concluded that game-based learning provided a capable means of incorporating sustainable development topics into teaching of unrelated technical skills.

1. Introduction

Since the times when the seminal works of Huizinga [1] and later Caillois [2] were originally published, the importance of games in both society and the economy has grown considerably [3,4]. Games were adopted to improve motivation in various contexts, from project management [5] to elderly care [6].

Education is one of the fields where the transformational potential of games is well acknowledged. While game design can be an attractive educational subject on its own [7], there is much more interest in applying games to teaching various subjects with the aim of improving learning effectiveness [8,9,10]. There are numerous examples of the successful application of different types of games (board games [11], card games [12], role-playing games [13], video games [14], and mobile games [15]) at various levels of education (beginning with preschool [16], through primary [17], secondary [18], and higher [19], ending with adult education [20]).

Games were found to be convenient tools to support learning and teaching also about sustainable development [21,22,23]. The game-based approach can be used to implement a learner-centered active, collaborative learning environment that enhances student engagement with sustainability concepts [24].

Taking into account the diminishing non-renewable resources of our planet, there is an urgent need to incorporate sustainable development topics in the education of engineers, including computer science engineers [25]. This is even more important in the former Soviet Union and other Eastern Bloc countries, where little attention has been paid to sustainability both during the times of communism, as well as during the economic collapse that followed soon after [26].

There are two basic ways of integrating sustainability and computer science education [27]: one is by introducing new computer science courses whose topics intersect the two areas; the other is by introducing lectures, exercises, and projects themed on sustainability into classic computer science courses. The latter seems more adequate in the case of courses whose subject is only indirectly connected to sustainable development issues, such as algorithms and data structures or introductory computer programming. It can also be seen as a remedy to a recent tendency to remove sustainable development courses from major study programs due to the introduction of specialized sustainable development programs [28].

The facilitation of such integration requires adequate educational tools. In this paper, we advocate the opinion that mobile educational games are a convenient vehicle to incorporate the sustainable development topics in the process of teaching the engineers the core skills of their respective discipline. As a proof of feasibility of this approach, we describe Eco JSity, an interactive educational game of solving classic algorithmic problems with JavaScript code, whose storyline, game space, and rules are all themed around sustainable development. The game app is freely available for web browsers [29] and Android-based mobile devices [30].

In our recent paper [31], we presented the technical internals of the game app and discussed its aspects that are most relevant to computer science education—the coverage of problems to solve and how they were put into the frame of a game, as well as the measured effectiveness of the game in teaching the introductory concepts and the construct of programming logic and thinking. In this paper, we focus on the sustainable development topics featured in the game. The main objective of this study is therefore to answer the following research questions:

RQ1: Has the inclusion of the sustainable development theme resulted in a negative perception of the app’s usefulness with regard to its basic purpose (i.e., computer science education)?

RQ2: What are the students’ stances on the combination of the sustainable development theme and computer science education.

The availability of the prior results of the app’s usefulness evaluation carried out among students in Lithuania and the new results of the evaluation involving Polish students also allow us to discuss to what extent the usefulness evaluation responses are country specific.

2. Related Work

2.1. General State of the Field

Game-based learning has become a significant subject of scientific interest, especially in the areas of business management, marketing, computer science, and engineering [32]. Introducing game-based techniques to education provides various benefits, which may correspond to cognitive (e.g., improved learning), behavioral (e.g., promoting teamwork and team dynamics), and affective (e.g., increased student engagement, motivation, and satisfaction) outcomes (see [32] and the works cited therein).

There is also a vivid scientific community working on the topic of the use of game-based learning in sustainable development education. A study [33] found that simulation and gaming successfully improve cognitive and effective learning outcomes, which in turn positively affect the students’ development of critical thinking skills on sustainability. The latter aspect was supported by the study of Miguel et al., who claimed that in addition to encouraging systemic, critical, and problem-solving thinking, the path to progress in education for sustainability is to improve the development of strategic thinking, collaborative thinking, and self-awareness [34].

Such promising results explain the high number of studies on applying games to sustainable development education: Hallinger et al. identified 376 publications on simulations and serious games used in education on sustainability [22], whereas an earlier study focused on games rather than on publications, identifying 77 serious games on sustainability themes [23].

The widespread use of mobile devices such as tablets and smartphones not only brought break-through accessibility advantages, but also allowed for a more efficient instruction with the advancement of the implementation of the “computing anytime and wherever” paradigm. As a matter of fact, Dos Santos et al., who analyzed only works on digital serious games for sustainability learning, identified 90 relevant publications [35].

2.2. Examples of Games for Sustainable Development Education

For a glimpse of the variety of existing applications of game-based learning to sustainable development education, their selection is presented in

Table 1. Selected games for sustainable development education.

Author	Year	Description
Ašeriškis et al.	2014	Game for supporting sustainable business project management [5]
Capellán-Pérez et al.	2019	Simulation game to educate on the energy and sustainability challenges [36]
Herrera et al.	2019	Game for educating on lean construction principles [37]
Janakiraman et al.	2021	Game-based learning platform to experience energy-related issues in the context of developing an eco-friendly city [38]
Koenigstein et al.	2020	Role-playing game for marine sustainability education [39]
Meinzen-Dick et al.	2018	Action game for groundwater management aimed at water saving [40]
Mellor et al.	2019	Educational game motivating undergraduate and advanced high school students for environmentally-aware chemical design [41]
Miller et al.	2019	Card game to support systems thinking in green chemistry that focuses on the creation of a recycling plant [42]
Orduña Alegría et al.	2020	Educational game on the socio-ecological dynamics of collaboration in agriculture [43]
Phongthanachote et al.	2019	Game to promote sustainable energy conservation [44]
Poole et al.	2020	Digital simulation game for promoting activities and supporting identities and practices oriented toward ecological well-being and sustainability [45]
Saitua-Iribar et al.	2020	Serious game to educate on the Sustainable Development Goals (SDGs) [46]
Su	2018	Strategy game focusing on the effects of human migration on environmental sustainability issues [47]
Vatalis et al.	2017	Role-playing game for teaching higher education students the concept of Sustainable Development Goals [48]

.

Note that all games mentioned above teach either sustainability concepts as such or domain-specific knowledge adhering to sustainable development principles applicable to that domain. The approach reported in this paper—incorporating thematic (urban development) sustainability concepts in the context of teaching technical skills unrelated to that theme (computer programming)—can thus be seen as somewhat original.

3. The Eco JSity Game

3.1. Game Context

While there are several notions of sustainability, such as, e.g., economic sustainability, i.e., the ability to ensure the sufficient and consistent development of goods and services with stable amounts of government and external debt [49], or social sustainability, i.e., the capacity of a democratic system to provide “social well-being”, marked by fair access, equal opportunity, and government transparency, to the delivery of public infrastructure and social services [50], the theme of the presented game addresses environmental sustainability, i.e., the ability to maintain adequate levels of the regeneration of renewable resources, the production of waste, and the limitation of the use of non-renewable resources [51].

The Eco JSity game is based on the concept of “environmental literacy”: the ability to understand the systems and processes of nature and the environment that allow for the creation and functioning of sustainable communities. Environmental literacy, in turn, is one of the main goals of environmental education, which consists of raising awareness of the entire population of the world “about the environment and related problems” and the formation and development of “knowledge, skills, attitudes, motivation and aspirations to act individually and collectively in search of solutions to modern problems and prevention of new ones” [52].

Environmental literacy is at the heart of the Sustainable Development Goals (SDGs) declared by the United Nations, in particular SDG 4, SDG 13, and SDG 14, which represent a structure for fostering environmental literacy through schooling, setting goals for cognitive, socio-emotional, and behavioral learning for each SDG. “Holistic and transformational education” is thus the first imperative to expand on the achievement of “a sustainable, peaceful, prosperous and equitable life for all on earth” [53].

The objectives of the Eco JSity educational game are thus twofold:

• Mastering of computer science knowledge (including techniques for solving algorithmic problems and JavaScript language syntax) and the formation of computer science skills (including solving algorithmic problem, as well as writing and testing JavaScript programs).
• Increasing motivation to protect the environment, helping to understand the nature of the interrelationships of economic and environmental processes, and instigating positive practices of personal and group behavior in relation to the environment, including responsible consumption.

While Eco JSity is obviously rooted in the theory of game-based learning [54], some of its traits are also representative of ubiquitous learning (defined as “an everyday learning environment that is supported by mobile and embedded computers and wireless networks in our everyday life” [55]) and mobile microlearning, which both limits the time required from a student to pass a single unit of learning (a micro-lesson) to a few minutes and emphasizes that the content is specifically prepared to be presented on the small screens of smartphones [56].

The game was designed to function as a supplement to existing teaching techniques, which is in line with the current research consensus (see [32] and the works cited therein).

3.2. Game Content

The direct aim of the player in the Eco JSity game is to prevent city pollution in daily activities. The player acts as a commercial advisor who has to solve programming puzzles to build a quarter of the city to start generating income while avoiding excessive pollution, which decreases the game score. The successful implementation of programming-related tasks allows for increasing city revenue and the game score. The selection of programming tasks follows the list of topics for an undergraduate computer science course suggested by the Association for Computing Machinery Computing Curricula [57]. The game is composed of 19 levels of increasing difficulty, each of them focused on a single algorithmic problem to be solved using JavaScript. The covered range of topics includes common computer science problems, such as: linear algorithms, branching (conditional) algorithms, iterative algorithms, search and sorting algorithms, recursion, tree traversal, and graph algorithms.

In order to allow the student to focus on the problem covered at a specific game level, predefined JavaScript classes and methods specific to the problem context are provided, so that the solution can be kept on an adequately high level of abstraction. However, especially at the later levels, only the minimum necessary information about these predefined elements is provided to the students in the challenge description, so that they usually have to do some experiments on using them before they find out how to make use of them properly to solve the problem. While this was reported as an issue already in early tests of the Eco JSity application, we consider it to be a crucial feature of the game, as it helps the students to learn how to deal with missing code documentation by experimenting directly with the code.

Eco JSity was designed considering all 13 principles of good learning defined by Gee [58].

Table 2. Addressing Gee’s principles of good learning in Eco JSity.

Principle	Implementation in Eco JSity
Co-Design	Students write solution code on their own using their knowledge and creativity.
Customize	Solutions are evaluated on the basis of their effect, not the way they are implemented.
Identity	Students impersonate an advisor responsible for sustainable city development.
Manipulation	Students’ code execution is graphically presented in the city view.
Well-Ordered Problems	The problems are presented in order from the least to the most complex.
Pleasantly Frustrating	Difficulty is fine tuned by providing hints to make the harder problems easier or leaving some details unsaid to make the simple ones more difficult to solve.
Cycles of Expertise	Solving the further problems requires adapting and combining knowledge learned with solving the previous ones.
Information “On Demand” and “Just in Time”	Only minimum necessary information is given initially; students get instant feedback after trying to run their code.
Fish Tanks	Complex real-world problems are simplified to a single dimension and/or aspect so that they can be solved with simple solutions.
Sandboxes	Only a subset of the programming language is available to students so that their code cannot damage the learning environment.
Skills as Strategies	All problems to solve in the game mirror actual real-world problems.
System Thinking	There is a common context for all problems (city management).
Meaning as Action Image	Problems and solutions are visualized in a way emphasizing their key properties.

shows how each of them is addressed in the presented game.

3.3. Game Form

The game was implemented in two versions: as a mobile app for Android-based mobile devices [30] and as a web app for the remaining devices (including mobile devices running other operating systems and personal computers) [29]. Both versions can be downloaded free of charge.

While the reader interested in the detailed description of how the respective problems are presented in the game is referred to the already mentioned paper presenting the application from the technical point of view [31], for a glimpse of how the game looks like, Figure 1 presents a screenshot of an iterative search challenge visualization. To solve it, the student’s program has to find the right waste deposit place depending on the randomly assigned type of waste truck.

What is presented in Figure 1 is the solution visualization view, one of three primary view types available in the Eco JSity app. The second one is the code editor view, where the user can type in the program code solving the current level’s challenge (Figure 2), and the third one is the city planner view (Figure 3), which is the board for the main city-building game played across programming challenges, dedicated only to teaching sustainable development concepts. Its key challenge is to attain a state of reasonable balance between income and pollution, which both increase with the city’s growth. The city-building game is closely connected with solving the programming challenges, as the development opportunities expand with every completed level, and the longer the time taken to solve a challenge, the more pollution is emitted.

4. Materials and Methods

4.1. Evaluation Method

In their seminal work [59], Kirkpatrick and Kirkpatrick defined four levels of training evaluation, covering respectively: reaction, learning, behavior, and results. Only the first level was considered as mandatory by Kirkpatrick and Kirkpatrick, whereas the remaining three should be done if the required resources are available [59]. In this study, we decided to perform only a Level 1 evaluation, because we were not able to meet the requirements of experimental control necessary for Level 2–4 evaluation, as the evaluation was performed in the middle of semester (which is the most appropriate time for a course to make effective use of the game with regard to the scope of its computer science content), and the students participating in it were in the process of learning with other tools (primarily a dedicated massive open online course); hence, the results could not be attributed to the Eco JSity game alone.

The applied approach, while enforced by circumstances, is in line with the research practice of the evaluation of computer science educational games, as only about 1/5th among the 117 studies of that kind analyzed by Petri and von Wangenheim [60] included the evaluation of a level surpassing 1.

4.2. Data Collection Instruments

The research instrument most adequate for Level 1 evaluation is a post-training questionnaire. In the reported study, two questionnaires were actually developed and used, addressing, respectively, Research Questions 1 and 2. Considering the scope of RQ1, we decided to use the questionnaire developed by Davis [61] for the measurement of Perceived Usefulness (PU; users’ belief that using the technology would improve their performance) and Perceived Ease-of-Use (PEU; users’ belief that using the technology would not require significant effort), the fundamental elements of the Technology Acceptance Model (TAM). As its name implies, the purpose of the model was to predict potential acceptance or rejection of the technology. Much later, the TAM-based questionnaire was adopted for the evaluation of applications, providing an alternative to usability measures such as the System Usability Scale (SUS) and the Usability Metric for User Experience (UMUX); for instance, the recently introduced mTAM (modified Technology Acceptance Model) score, essentially the average of PU and PEU scaled up to the 0–100 range, has been reported to be highly correlated with both SUS (0.70–0.90, depending on the case), and UMUX (0.62–0.90) [62].

The TAM-based questionnaire has already been applied to evaluate game-based educational applications, e.g., Giannakoulas and Xinogalos [63] used the model to measure the acceptance of the intervention featuring an educational game for teaching simple programming concepts to primary school children, whereas Onashoga et al. [64] utilized TAM to appraise their 3D game-based learning approach for increasing awareness of phishing attacks.

For the purpose of the evaluation of Eco JSity, the 12 original TAM questionnaire items were translated to Polish, the language of the students (due to involvement of some international exchange students from abroad, the questions had to be back-translated to English in several instances upon the respondent’s request). The translation was meant to keep the original intent as close as possible—usually, the only change made was regarding the name of the evaluated software. Six of the questions referred to Perceived Usefulness (PU1–6) and the other six to Perceived Ease-of-Use (PEU1–6). Each item was assessed with a 7-point Likert-type scale from 1 to 7; note that compared to the original form [61], we reversed the scale to make the interpretation simpler; according to prior research, such a change does not affect the magnitude or structure of the measurements [65]. Following the practice of using TAM-based questionnaire for evaluation [62], we also replaced the original scale labels, from “unlikely/likely” (exacting ex ante perspective) to “disagree/agree” (having no such constraint).

In order to verify how the users perceived the fact of introducing sustainable development topics into the app, a second questionnaire with an additional set of questions had to be prepared. Considering the advantages of forced binary questions shown by Dolnicar et al., i.e., leading to equally reliable results as Likert-scale questions, saving respondents’ time, and being perceived as simpler [66], we decided to use such a form for all questions of the second questionnaire. Below, we list its questions (in English translation, as they were originally stated in Polish), arranged into three groups:

A. My attitude to the problem of sustainable development:

• I believe that sustainable development is a key challenge for governments and researchers.
• I try to follow the principles of sustainable development in my everyday life (e.g., repair instead of throwing away, choose ecological means of transport, etc.).
• I am interested in sustainable development issues (I read articles, watch TV programs, etc.).
• I think that too much is being said about sustainable development nowadays.

S. Sustainability problem in the Eco JSity application:

• I think that it is a good idea to have sustainable development as the game topic.
• I believe that the sustainable development theme should be more prominent in the game.
• I believe that the game can raise awareness of sustainable development problems among some people.
• I believe that the sustainable development theme did not distract me from learning the algorithms.

P. Adding the sustainability theme to the programming course:

• I think that adding the sustainable development theme (in the form of e.g. relevant examples and tasks) will make the programming course more interesting.
• I believe that adding the sustainable development theme will not affect the reception of the course, but will improve the awareness of sustainable development among students.
• I do not think that adding the theme of sustainable development to the programming course will have any positive or negative effect.
• I believe that adding the sustainable development theme to the programming course will unnecessarily distract students from its main topic.

4.3. Survey Administration and Participants

The survey was administered online (as a computer-assisted web interview), using Microsoft Forms, which allowed accepting only answers from invited respondents and avoiding duplicate answers. The survey spanned from late November till December 2020. The students were asked to answer the first questionnaire when they considered they had got to know the app. The second questionnaire was presented two weeks after the first one, in order to give the respondents some perspective.

The respondents were recruited from among the students of two Polish educational institutions: the University of Szczecin (Faculty of Economics, Finances and Management) and the West Pomeranian Business School, participating in three courses: Introduction to Computer Programming, Algorithms and Data Structures, and Programming Languages and Paradigms.

All study participants were informed beforehand that they were taking part in a study; they participated in the evaluation voluntarily; however, there was an incentive for the students, as it was treated as non-compulsory homework, and upon completing it, they received a small bonus of points counted towards their final grade. From the total of 124 invited students, sixty-nine (56%) submitted the first questionnaire, and forty (coincidentally also 56% of those who completed the first one) returned the second questionnaire.

Among the respondents who disclosed their gender, nineteen-point-one percent were female, and eighty-point-nine percent were male, while the mean age was 23.7 years (see Figure 4).

4.4. Data Analysis

In order to address RQ1, the respondents’ answers with regard to PU and PEU were aggregated. As Likert scale is an ordinal scale [67], to facilitate the calculation of means, two ratio scale variables were introduced:

• Agrees, indicating the percent of respondents who gave a positive response (i.e., 5, 6, or 7);
• Net top box, indicating the difference between the percent of respondents who gave an unambiguously positive response (i.e., 6 or 7) and those who gave an unambiguously negative response (i.e., 1 or 2) [68].

For the sake of the further analysis of the correlation between answers to respective questions, Spearman’s rank correlation coefficient (which is appropriate for discrete ordinal variables) was calculated.

With regard to RQ2, in order to give some depth to the results on how the users perceived the fact of introducing sustainable development topics in the app, the results obtained with the second questionnaire were first passed through cluster analysis. The Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH) [69] algorithm was used for this purpose.

The data analysis was performed with Python 3.8.2, using the following libraries: NumPy 1.18.4, Pandas 1.1.4, Scikit-learn 0.23.2, and Matplotlib 3.2.1 for drawing the charts.

5. Evaluation Results

5.1. Perceived Usefulness and Perceived Ease-of-Use

Table 3. Share of positive answers to respective questions on Perceived Usefulness (PU) and Perceived Ease-of-Use (PEU).

Variable	Question	Agrees	Net Top Box
PU1	Using the app would enable me to learn faster	69.57%	26.09%
PU2	Using the app would increase my learning outcomes	55.07%	18.84%
PU3	Using the app would enhance my learning effectiveness	60.87%	23.19%
PU4	Using the app would improve my learning outcomes	65.22%	23.19%
PU5	Using the app would make my learning easier	69.57%	27.54%
PU6	I would find the app useful in my education	73.91%	34.78%
PU	Average	65.70%	25.60%
PEU1	Learning to use the app would be easy for me	66.67%	31.88%
PEU2	I would find it easy to do what I want in app	50.72%	17.39%
PEU3	Learning with the app would be clear and understandable	55.07%	20.29%
PEU4	I would find the app to be flexible to interact with	53.62%	8.70%
PEU5	It would be easy for me to become skillful at using the app	60.87%	23.19%
PEU6	I would find the app easy to use	59.42%	20.29%
PEU	Average	57.73%	20.29%

shows the share of respondents who answered positively (5–7) (in Column 3, “Agrees”) and the difference between the share of respondents who definitely agreed (only answering 6–7) and disagreed (1–2) (in Column 4, “Net Top Box”) to the respective questions regarding perceived usefulness and perceived ease-of-use (Column 2).

The obtained results showed that 65.70% of respondents had a positive perception of the usefulness of the application in learning programming. The group with a clearly positive attitude was 25.6% larger than people who perceived the app in a negative way. Most of the respondents (73.91%) believed that using the application in their education was useful (PU6). Almost 70% of respondents agreed with the statement that the application allowed them to learn faster (PU1) and more easily (PU5). The analysis of differences between highly positive and negative attitudes showed that in each of the six examined statements (PU1–PU6), the predominance of a positive opinions was notable.

The results of the assessment of the perceived ease-of-use of the application (PEU1–PEU6) allowed stating that the respondents evaluated it positively, though not as distinctly as usefulness. The average rating of all variables of the tested construct was 57.73%, and the share of definitely positive responses was 20.29% higher than the negative ones. Two-thirds of the respondents thought that learning using the application would be easy for them (PEU1), and sixty percent expressed the belief that it would be easy for them to become skillful in using the app (PEU5). The results suggested that the application could be improved in terms of the ease of finding what the player wants to do in the app (PEU2), because only slightly more than half of those polled (50.72%) assessed this criterion positively. The flexibility of interaction with the application (PEU4) is also noteworthy, in the case that the difference between strongly positive and negative indications was the smallest of all measures, with only 8.70% in favor of the former. The detailed evaluation results, showing the number of all respective answers for each of the questions concerning the PU and PEU constructs, are presented in Figure 5.

The histograms showing the number of respective answers (remember that a seven-point Likert scale was used) are shown in the left part of Figure 5, whereas the right part shows the correlation between the answers to the respective questions. The analysis of the TAM questionnaire scores allowed noticing a significant advantage of positive indications over negative ones for every measure, the most apparent for PU1, PU5, PU6, and PEU1. For these measures, a large share of maximum indications (seven) was also noted. The most frequently chosen grade for most of the measures was five, except for PU6, where an answer of six predominated, and PEU6 with a similar number of indications of five and seven. The question regarding the respondents’ attitude about the ease of becoming skilled in using the application (PEU5) received the highest number of strong disagreement (one) responses among all questions, though the respective number of strong agreement (seven) answers was still higher.

As expected, a very strong correlation was measured among PU components (the value of $r_s$ ranging from 0.93 down to 0.75), whereas the correlation among PEU components spanned from very strong to strong $(r_s=0.87-0.71)$. The correlation between PU and PEU components was mostly medium ($r_s=0.75-0.48$) (see the right part of Figure 5), which supports the distinction of the PU and PEU constructs. Among PU components, a particularly high correlation was observed between the variables PU4 and PU3 ($r_s=0.93$). This means that the use of applications, increasing the effectiveness of learning, strongly correlated with an increase in learning outcomes. Furthermore, the relationship between PU1 and PU5 ($r_s=0.86$) indicated a strong positive correlation between the possibility of learning faster with the use of the application and the perceived ease of learning. Among the PEU components, the strongest correlation existed between PEU3 and PEU4 ($r_s=0.87$) and between PEU5 and PEU6 $(r_s=0.85)$. This indicated a strong positive correlation between the perception of learning when using the app in a clear and understandable manner and the flexible interaction of the player with the application. Furthermore, there was a strong positive correlation between the perception of the ease to become skillful in the use of the app and the perceived ease-of-use of the app in general. Regarding the correlation of PU and PEU components, the strongest was observed between PEU1 and PU6, as well as PU5 ($r_s=0.75$). This indicated the relationship between the perception of learning using the application as easy and the assessment of learning as faster and easier. The lowest correlation among the examined measures was observed between PU2 and PEU6 ($r_s=0.48$) and between PU3 and PEU6 ($r_s=0.50$).

5.2. Sustainable Development Aspects

Figure 6 shows the results of clustering of the second questionnaire answers. For the sake of chart readability, only two main dimensions of variation are visualized, as obtained with Principal Components Analysis (PCA).

The full survey results are presented in

Table 4. Share of positive answers in the respective clusters.

Cluster	Share	A1	A2	A3	A4	S1	S2	S3	S4	P1	P2	P3	P4
Skeptic	35%	50%	43%	21%	7%	14%	21%	14%	36%	0%	29%	7%	57%
Pragmatic	40%	56%	56%	38%	13%	81%	19%	75%	44%	0%	100%	0%	0%
Keen	25%	90%	60%	10%	0%	70%	40%	50%	50%	100%	0%	0%	0%
All	100%	63%	53%	25%	8%	55%	25%	48%	43%	25%	50%	3%	20%

. Its first column identifies the cluster, the second column the share of respondents who were classified into it, and the remaining columns the share of positive answers for the respective statements (denoted by row; see Section 3.3 for their definitions) obtained from survey participants classified into this cluster (the bottom row gives the share among all respondents).

The cluster marked with red dots was labeled skeptic, as only in it, the opinion that adding the topic of sustainable development to the programming course will unnecessarily distract students from its main topic received positive answers (57%). Despite such an assessment of the sustainability element in the game, every second representative of this group believed that sustainable development was a key challenge for governments and researchers, and forty-three percent of them tried to follow the principles of sustainable development in their everyday lives. The cluster marked with blue dots is labeled pragmatic, as all of its members believed that adding the topic of sustainable development would not affect the reception of the course, but would improve the awareness of sustainable development among students; eighty-one percent of the representatives of this group positively assessed sustainability as a game topic. The last cluster, marked with green dots, was labeled keen as 90% of its members believed that sustainable development was a key challenge for governments and researchers, and all of them thought that adding the topic of sustainability would make the programming course more interesting. Note, though, that even among the keen cluster members, only half of them expressed the opinion that the topic of sustainability had not distracted them from learning the algorithms, which was, however, the highest percentage across the three clusters. This result can be interpreted in two ways, that for more than half of the students:

• the features related to sustainable development did not go unnoticed (good for sustainable development education),
• the features related to sustainable development were a distraction from focusing upon algorithmic problem solving (bad for computer science education).

6. Discussion and Conclusions

The performed study allowed answering both research questions. Regarding RQ1, despite the inclusion of the sustainable development theme, about 2/3 of the respondents declared positive effects of using the Eco JSity app in their learning of computer science. Note that the app’s ease-of-use was evaluated also positively, yet by a smaller share of respondents (about 4/7 agreeing responses), although this, at least to some extent, could be attributed to the chosen didactic approach of providing initially minimum information, forcing the students to find out the rest by experimenting with the app, but at the same time reducing its ease-of-use.

If we compare these results obtained from among Polish students to those obtained earlier from among Lithuanian students [31], the share of respondents who positively evaluated the perceived usefulness of the application in learning programming was at a similar high level in both groups, slightly higher among the Polish students (65.7% vs. 63.6% among Lithuanian students). However, the story changes if we look at the difference between highly positive and negative attitudes (25.6% of Polish vs. 40.7% of Lithuanian students), which suggested that the dissatisfied Polish students were more inclined to pick decisively negative answers than their counterparts from Lithuania. This may stem from the fact that whereas the Lithuanian students hailed from a university of technology, the Polish students hailed from business schools and were thus more accustomed to educational case studies that closely address problems that may probably arise in their future business careers, for which the Eco JSity game story is hardly an example. No such difference was observed by comparing the measurements of the perceived ease-of-use, assessed moderately positively by both Polish (57.7%) and Lithuanian (54.0%) students. Here, the share of definitely positive responses was higher than the negative ones by a similar amount in both countries (20.3% of Polish vs. 19.1% of Lithuanian students). It therefore seems that in the world of globally distributed software, the users in different countries have a similar notion of software that is easy to use.

For the sake of reference, the obtained measurements of the perceived usefulness and perceived ease-of-use of the Eco JSity app (respectively, 66% and 58%) can be compared to others reported in the literature. Giannakoulas and Xinogalos [63] reported, for their game teaching simple programming concepts to primary school children, slightly higher PU (61% of positive answers), yet much higher PEU (83%), whereas Onashoga et al. [64] reported, for their 3D game aimed at increasing awareness of phishing attacks, much higher PU (95%) and moderately higher PEU (68%). Note, however, that each of these studies differed in terms of the game form, the scope of educational content, and the target audience, as well as the form of questionnaires used.

Regarding RQ2, the idea of introducing a sustainable development theme into the app received mostly positive feedback from the users. On the one hand, there was a significant share of respondents (25%) who thought that it would make the programming course more interesting (100% of the cluster members), but on the other hand, there was a number of skeptical users (35% of all respondents) who felt that it would unnecessarily distract students from the main topic of the course (57% of the cluster members). The most numerous, third group (40% of all respondents) believed that adding the sustainable development theme would not affect the reception of the course, but would improve the awareness of sustainable development among students (100% of the cluster members), which we also considered as a positive evaluation of the approach implemented in the app.

The results of the study have obvious limitations: the survey was performed among just two groups of students, in just two higher education institutions located in one Polish city. Students coming from other countries and educational contexts may therefore have very different views of the evaluated app; however, the high similarity of the results presented here with those obtained earlier in Lithuania [31] imply that some level of generalization could be substantiated.

Another limitation stems from the fact that this study covered only the subjective perception of users in the evaluation of the app. The ability of the Eco JSity app to improve the actual learning outcomes was, however, already demonstrated in our previous paper based on data from Lithuania, which was comprised of a comparison between the group playing the game and a control group [31]. That research did not, however, include the measurement of the sustainable development education outcomes.

In conclusion, the proposed approach of incorporating the sustainable development topics into the education of engineers with regard to the core skills of their discipline proved feasible. The presented results of the Eco JSity application evaluation indicated that it achieved the intended objective of supporting both the learning of algorithm design and programming, which belong to the core of computer science education, and the formation of environmental literacy and environmentally responsible behavior, which belong to sustainable development education. The majority of respondents both appreciated the effect the application had on their learning and expected it to improve the awareness of sustainable development needs. We therefore considered the proposed approach as a viable option for providing sustainable development education to engineering students regardless of whether and how it is related to the core contents of their specific course. The presented results added one more case in favor of applying game-based learning to computer science education.

Future research may include two distinct directions, which may however be combined to get a more general view of the outcomes of using educational apps such as Eco JSity. The first is a cross-comparison with other available solutions for game-based learning of computer programming and/or algorithms. The second is examining the actual long-term effect of using the app on the behavior of its users with regard to sustainable development aspects.