A Study on the Design and Application of Fictional Storytelling in Online Learning of Computer Security

Abstract

Computer security is an important part of the computer science (CS) curriculum in all kinds of universities. Although the educational approach tends to be highly technical and practical, it cannot be avoided that some understanding and memorizing of the core concepts is necessary beforehand. One method that has been found to work in favor of human psychology, and of special interest in the case of online learning, where student motivation is at a higher stake, would be to present such content as memorable and relatable situations to the reader, using some form of fictional storytelling. However, this approach must always serve its educational purpose first, following a set of sound design principles. This paper presents a study including 111 participants to evaluate the convenience of using a digital book called the “novelette”, which relies on storytelling, to teach computer security concepts in the CS degree of Universitat Oberta de Catalunya (UOC), a 100% online university. The study involved two questionnaires to gather quantitative and qualitative information about the format, content and students’ perception. This study shows that the validity and usefulness of this approach is on par with traditional ones, but also has a higher impact on student motivation and self-perception of the acquired knowledge.

1. Introduction

Computer security (or cybersecurity) is a very important part of the computer science (CS) curriculum, as its relevance in the successive iterations of the ACM curriculum guidelines for undergraduate degree programs in information technology by [1] testifies. Indeed, in the latest version of the curriculum guidelines, cybersecurity (SCEC) has become a core discipline, added to the body of knowledge “in recognition of the world’s reliance on information technology and its critical role in computer science education”, and “Information Assurance and Security” (CE-SEC) is considered in the list a Computer Science Competencies. Future CS graduates in any sub-field should be aware of the role of computer security and master basic topics, such as how, for instance, they know how to code [2]. As [3] points out, it is very important to address the factors influencing the information security behavior of all IT employees, if not society at large.

On the topic of teaching computer security, even when the subject curriculum is designed as highly practical, focused on the use and application of tools, it is very difficult to fully avoid the situation where it is necessary to justify why this topic is important and to memorize some basic concepts beforehand. On that regard, no matter the course format or subject, reading course materials and defining basic concepts is often one of the main tasks a student must perform during instruction, especially at the beginning. The classic approach in highly technical fields is to present them as a straightforward expository text, that “both describes and explains how something operates” [4], in the appropriate amount of detail. However, it is important that during the first steps of the course students keep their interest in the subject, at the same time that they are actually learning these basics. Another possibly more engaging approach would be to use a fictional story, a “chain of events in cause–effect relationships occurring in time and space”, crafted in a way to interest the audience [5,6].

The use of stories is not new at all to instruction, especially among children [7]. Studies such as the ones by [8] or [9] have shown storytelling to be more powerful in learning, being faster to read and easier to understand and internalize, for both young and adults. According to [10] or [11], human psychology works in favor of this approach, based on presenting memorable situations that the reader can relate to, or even become subject to spontaneous affective reactions. However, as we transition towards course design approaches in higher education, and especially in technical fields, this does not seem to be a very common approach. It is true that, in computer security specifically, there is a broad literature of fiction about “hackers”, where cybersecurity plays an important role in the story. However, and this is very important, in most of them, narrative licences prevail over technical accuracy [12], which goes against any educational purpose.

This paper presents a study on the design and application of a digital book named “the novelette” (or short novel) at Universitat Oberta de Catalunya (UOC), a 100% online university. The design process, and the final product itself, has been crafted from scratch and tested as the introductory educational resource in a computer security subject of a CS Bachelor directed at adult distance learners. However, the “novelette” has a dual nature, depending on the reader. At its core, it is a standalone digital novel, written by a professional of the “thriller” genre and with value as pure entertainment. In fact, it is currently sold in digital book stores to the general public. However, it has been designed by educators as a learning resource from its very inception, first and foremost, with clear learning outcomes. It is worth noting that it is neither expository text wrapped in some story, nor a compilation of examples based in some hypothetical professional situation. The “novelette” is a story of the “invented” type as established by [13], that aims to engage the reader through the mechanisms used by similar fiction (intrigue, a sense of adventure, etc.).

The main contribution of this work is a better understanding of the effectivity of storytelling-based approaches in adult education in distance learning, especially from the student’s point of view, thus testing the hypotheses on this topic by the previously mentioned authors and furthering the research on this topic. This paper also documents the design process of this kind of resource, highlighting the most important aspects that must be taken into account when teaching technical topics through storytelling. Both contributions can be very useful to other educators who want to develop engaging learning resources using this approach, especially in technical fields or at online institutions.

This paper is structured as follows. Section 2 provides an overview of the relevance of engaging students and why resorting to storytelling as an educational approach can be useful. Next, the design methodology and pedagogical background for the “novelette” is presented in Section 3. Section 4 formally defines the research questions and describes the design of the study to evaluate the convenience of employing storytelling in the early stages of learning in computer security with adult students. The results of the study are presented in Section 5 and discussed in Section 6. Finally, Section 8 concludes this paper and provides some insights on present and future work.

2. Background

Difficulties in student motivation are recognized as an important concern in diverse environments. Engagement has become increasingly relevant, since high-motivated people have better odds of achieving better performances and overcoming daily challenges. Indeed, motivation is even at a higher stake in some complex educational contexts, such as fully online institutions, where students may feel isolated, or in adult learning, where students must reconcile their scarce study time with their taxing day-to-day obligations. To achieve this state of engagement, many variables are involved, but a very popular solution in the recent years is the use of ludic or playful approaches. On that regard, the application of techniques such as gamification or the creation of “serious games” as an educational resource [14] has specially proliferated in the scientific literature, as shown by the different reviews on the topic, such as the ones by [15] or [16], among others. Generally speaking about the education of computer security, advances in recent years heavily tend towards the use of games or virtual laboratories, according to [17].

However, inspecting studies on the different existing frameworks to design ludic approaches, such as the one from [18], also shows that this is not trivial. Even though the end results of the experiences tend to be positive, as shown in literature reviews and meta-studies, great effort is required. Instructors report that the costs of design and implementation, the time needed, and the difficulties in the process may be too great in relation to the expected benefits [19]. The risk of incurring a high cost is especially acute when there is an over-reliance in creating complex digital products (i.e., video games or similar), which also may create the burden of additional technological requirements as a side-effect. Moreover, educators are not necessarily professional game designers, much less developers. Maybe a simpler conceptualization of a proven learning resource, still taking into account this ludic experience approach, could be provide a better return of investment in some cases.

When looking for this much simpler approach, one can consider that, at their very core, one of the main motivating factors behind these games are their stories. A key aspect of stories, and what makes them truly memorable, are the emotions and the cognitive aspects they evoke in the audience [20]. On that regard, ref. [21] explicitly considers “narrative”, a story that is being slowly unfolded, as one of its 52 “Gamification Mechanics and Elements”, at the same level as more straightforward, and less nuanced, game elements such as “points” or “leaderboards”. Nevertheless, narrative is not exclusive to games or other interactive media, despite their current popularity. For instance, a simple novel’s ability to engage the reader is purely based on narrative, with the added advantage that it does not require any additional additional competence beforehand, beyond being able to read, and the technological requirements can be very low.

Focusing on a purely narrative approach and its application in learning, it can be found that, similar to the concept of “serious game” exists, the concept of “serious storytelling” is also defined, by [22]: “storytelling outside the context of entertainment, where the narration progresses as a sequence of patterns impressive in quality, relates to a serious context, and is a matter of thoughtful process”. Serious storytelling is concerned with narration in a particular context beyond entertainment. Therefore, in the context of learning, the story is grounded in the pedagogical rules and goals: design learning, learning experiences and outcomes, etc. The concept of serious storytelling is also very close to digital storytelling [23], which takes full advantage of technology, but also becomes absolutely dependent on it in the process. A survey from [24] provides an overview, emphasizing its potential as a knowledge translation tool and relevance in the field of education, training and professional development, providing examples in the field of professional development such as [25] or [26].

On the effectiveness of this approach, according to the work of [27], existing research suggests that it can improve affective and utility reactions to learning. The former represents the emotional response, or enjoyment, whereas the latter is related to the cognitive response, or evaluation of usefulness of the fiction. Different studies also suggest that using storytelling usually has a larger effect than expository text (such as [10] or [28]). However, research is still ongoing, since it is also true that a study by [29] found no significant difference in either kind of reaction (all were positive in both cases).

Taking a look at the application of storytelling in higher education settings, and not aimed at school-age students, several specific examples can be found. However, beyond media studies, which are directly related to interpreting the fiction itself, this approach is more easily found in fields related to social fields, such as history, management, business or law, to name a few ([30,31,32,33]). In fact, ref. [34] studied the acceptance of short novels in the context of a criminology degree, although in complete isolation. The results provided no frame of reference or comparison to the use of direct expository text, nor assessed its impact on learning outcomes. Nevertheless, and despite these few cases, the literature suggests that storytelling is seldom used to teach highly technical concepts, such as those typical of a computer science curriculum.

3. Novelette Design Methodology

A novelette is understood as a short story with a word count below a novella or novel. Currently, there is a vast library of resources based on fictional stories that incorporate aspects of computer security in their plot. As stated by [35], stories about hackers have become part of the popular culture for decades. However, ref. [36] also points out that one must be very careful when taking these kind of existing resources into consideration, since they often contain factually inaccurate content, when not pure fantasy, making them a potent vehicle for learning misinformation about the world. An egregious example, among many, could be the popular fictional situation when passwords are quickly cracked character-by-character, with a progress bar included, in a highly graphical and interactive user interface. Special care must be taken in this regard when designing and incorporating an educational resource based on storytelling, and especially on a topic that has gained relevance in the popular media, but also created strong misconceptions on how things actually work.

Some previous work exists on methods to facilitate the integration of storytelling as a learning resource, with the help of templates [37]. However, this work presents a design methodology that allows starting from scratch, in accordance with the serious storytelling model introduced in Section 2. The novelette at its core is designed according to the model’s four components, applied to the context of education:

• Context of application, a relevant setting for understanding the knowledge.
• Course, as the sum of the plot and how the content evolves in a cause–effect relationship, but constrained by the requirement of the creation of knowledge (i.e., learning). It is important to note that, according to the model, the triggering of emotions is still considered one of its main features. Being able to engage the audience and turning the narrative into an emotional experience becomes the key to mental stimulation, and thus, to the “thoughtful process”.
• Content of the narrative, the elements that can be perceived by the consumer, or the student in this scenario.
• Channel of distribution, a written novel in this case.

The design process was steered to guarantee that these four components would be taken into account.

3.1. Design Process

Three roles collaborated during the development process, helping define components (a)–(c): a professional writer, a computer security instructor, and a pedagogy expert who acted as project leader and mediator between the writer and the instructor. The overall process is summarized in Figure 1.

In the first step of the design process, the instructor and the mediator identified the key concepts of the learning outcomes. Among the different effective strategies to represent and summarize the body of knowledge, a conceptual map was chosen [38]. Starting from a purely pedagogical perspective, and relegating the design of storytelling to a later stage, guaranteed the constraint expressed in component (c). Further, the application of a visual representation eases the visualization of key concepts from an holistic point of view [39].

Then, each key concept was further developed, providing a precise definition by educational standards, akin to a classical expository text representation. During this process, the instructor would add a short description of several fictional situation in which the concept was important, taking into account which actors participated and their roles in the situation. These would help the writer later in the design process, not being a scholar on the topic, to better understand the key concepts as well as providing examples on how to transition the text to a storytelling approach.

The conceptual map and the key concept definition became the starting point for the professional writer to develop the fiction. Even though the educational guidelines had to be followed, the writer had full control of the narrative fiction and how the story should develop to better trigger the emotional response in the reader. On that regard, there are several methodologies to structure the creation of novels that were carefully considered ([40,41,42]).

Starting from a one-sentence summary of the plot, the initial concept was further expanded by the writer to a full paragraph describing the story setup, major setbacks, and the ending. Then, the characters were defined, presenting their names, a short summary of their roles, and their goals and motivations in the story, as well as point of conflict (i.e., what would prevent them from succeeding). This short document was considered the novelette overarching plot, and agreed by all parts. Even though it focused on the narrative fiction, the writer took into account opportunities to include fictional situations similar to the ones defined by the instructor. On that regard, any doubts the writer had about the computer security key concepts would also be clarified by the instructor. Further, any topic in the narrative that could be considered controversial (e.g., depiction of adult-only content) was discussed at this stage.

Once the main components of the story were defined, the chapters of the story would be slowly developed, on by one, in an iterative process. The requisite was that all key concepts should be represented at least once along the full story, using a fictional situation different from the examples, and at least one such situation would appear in each chapter. Each chapter would be validated to ensure that the situations aligned with the educational outcomes before moving forward. The mediator’s role was especially relevant during this stage, since very often there would be clashes between narrative licenses for the sake of a good story (the writer) and how computer security actually works in the real work (the instructor); however, given the story’s purpose, very special care had to be given to the latter.

Once the story was complete, a final revision was performed, as well as a small audience testing with volunteers. This helped fine tune the story and make sure the key concepts were properly explained within the narrative. With the final manuscript, the novelette was edited and published. The whole process amounted to about 3 months, taking into account that the novelette was not the main project of the professional writer.

3.2. The Computer Security Novelette

The final product was a 150-page long novelette, structured in 21 chapters, which required 3 months of work to finish. Despite its page count, its typesetting allowed it to be read at leisure in a few hours, not much longer than a study session using the equivalent expository text resources. The starting conceptual map is shown in Figure 2, defining the reach of the body of knowledge. An example of further development of each key concept also follows, helping exemplify steps 2–3, by presenting its definition and proposing a useful situation where this concept could be used in fiction.

Key concept:

A Denial-of-Service (or DoS) is the inability to access a resource or service by a legitimate user. That is, exclusive appropriation of a resource or service so that third parties will not be able to make use any of it.

Situation 1:

A mafia gang has thousands of computers spread all over the world. Millions of simultaneous connections to an online shop are initiated by means of a DoS method called “IP Flooding”. The shop ceases to be operational and no customer can connect, so every minute that passes means money lost by the owner. The gang demands a ransom in order to stop the attack.

Situation 2:

You have to stop someone from connecting to the public WiFi network and sending an important e-mail. Using DoS, I degrade the connection and nobody is able to connect to the Internet (not even me!).

From this conceptual map and the full key concept development, the writer proposed the following synopsis that would lead to the final manuscript after the thorough iterative process, chapter by chapter.

Linus, an expert in computer security, with problems with the justice, sees helplessly as his life sinks inevitably. One night, by chance, he bumps into a girl who has been the victim of a phishing attack that has turned into attempted assault. Investigating who is behind this deception, Linus will discover a criminal network much larger than it seemed at the beginning, and consequently, a way to help himself and to redeem himself with the judicial system.

The final product included the story and appendixes including the concept map and a glossary of key concepts, with definitions and page numbers were they were presented in the story. Key concepts were also bold-faced the first time they appeared in the text. Including this information helps better anchor the educational objectives of a fully narrative-based approach, as per [43]. The book also concluded with a list of other interesting media related to computer security, such as books, movies, or video games, to invite readers to explore the topic from a purely ludic, not necessarily educational, standpoint (e.g., The Cuckoo’s Egg, War Games, or Uplink).

4. Research Design

This section describes the study that we carried out in an online course on computer security (CompSec) to evaluate the convenience of employing storytelling in the early stages of learning in cybersecurity with adult students. The following sections describe: the context and participants in the study (Section 4.1), the methodological design and research questions (Section 4.2), and the tools used (Section 4.3).

4.1. Context and Participants

The study was conducted in the online CompSec course, starting from February 2019 to July 2019, held at Universitat Oberta de Catalunya (UOC), which is a fully online university offering graduate and postgraduate programs in different languages, currently amounting to about 75.000 students. The typical students are fully autonomous adult learners, 64% of them are more than 30 years old, and 27% for the 40+ age bracket. Most work full-time and are financially independent (95%), married (73%), or have children (58%), according to internal polls performed during the enrollment process. In general, they chose to study online because they cannot attend classes and need to be fully autonomous in their study time. Therefore, there is an important focus on the use of digital self-study resources that can help them engage with the course content and that can be perceived as immediately useful in a professional field.

The CompSec course is an optional subject in the Computer Science and Telecommunication Engineering bachelor degrees. Students can enroll in the CompSec course after having taken courses on computer network theory and management. Both degrees also include separate courses on cryptography or electronic commerce, which are subjects with concepts related to cybersecurity. For this reason, in the study presented in this paper, participants that had passed one of these two courses before enrolling the CompSec course were considered as having previous theoretical knowledge in cybersecurity.

Regarding the CompSec course contents, the course consists of three thematic blocks. The first block provides a general introduction to cybersecurity. The second block studies basic computer attacks, prevention, and detection mechanisms. The third block presents basic concepts of cryptography, authentication mechanisms, and secure communication applications. Assessment is made from three theoretical assignments (one for each block) and three practical assignments (two related to the contents of the second block and one related to the contents of the third one). This study was performed during the teaching period of the first block. We considered that the theoretical basics of cybersecurity is the topic of the course where storytelling could have a greater impact on learning. In the semester of the CompSec course where the study was undertaken, the course had 134 students, out of which 111 voluntarily participated in the study. The student sample was considered as representative of the student profile enrolled in this subject in the past.

4.2. Methodology

In order to test the hypothesis that employing storytelling may assist in the progress of learning over expository text, even when the subject is eminently technical and with adult students, we set the following research questions:

RQ1: 

What is the resulting student attitude towards this format, in respect to a traditional one (i.e., expository text)?

RQ2: 

What is its influence in the students’ self-perception of their knowledge acquisition?

RQ3: 

What is the real impact on the student’s knowledge acquisition?

To address these research questions, the CompSec course was divided into two different virtual classes. The novelette was deployed in one of them (in the following, we will refer to this group as Narrative group or Ng), whereas the standard curriculum format was used in the other (which will be referred as Expository group or Eg).

Following the recommendations by the ethics committee of the university, students in each group had the option to voluntarily participate in this study. Participation consisted basically of answering two questionnaires. To encourage participation, students that answered the two questionnaires were awarded with 1 extra point out of 10 in the first graded assignment of the course. The Eg group included 63 subjects, out of which 49 participated in the study. The Ng included 71 subjects, out of which 62 participated in the study. Figure 3 shows the main demographic variables of the participants in the study for both groups.

The two questionnaires were designed by two instructors to gather quantitative and qualitative data to mainly evaluate the students’ perception of the learning resources format and content, and their perception on the acquired knowledge. The questionnaires and the grades of the first assignment were also used to assess the actual knowledge acquisition. This approach is aligned with [44], which served as a basis to the methodological design of this study.

Table 1. Variables included included in this study.

Variable Type	ID	Question/Description
Format of the learning resources	P1.1	The resource’s format is satisfactory.
	P1.2	The resource is motivating to continue learning about security.
	P1.3	The contents are useful to start working on the graded assignment.
	P1.4	The content is relevant in the real world.
	P1.5	How long did you take to study the learning resources and resolve the graded assignment?
Self-evaluation questions	P2.1	I know the different types of attack on TCP/IP networks.
	P2.2	I know what kind of tools I should use to start exploring a system.
	P2.3	I know what kind of tools I should use to protect my systems.
	P2.4	I would be able to put into practice what I have learned.
Improvement questions	Improvement	Questionnaires 1 and 2 included 10 general questions to evaluate the actual knowledge of the students on cybersecurity. Grades range from 0 to 10.
Course assignment	Activity grade	The first graded assignment of the course included 12 questions related to the content of the learning resources. Grades range from 0 to 10.

contains a summary of the main variables analyzed in this study using both questionnaires. Variables P1.1 to P1.5 and P2.1 to P2.4 were questions posed directly in the second questionnaire. “Improvement” was a variable computed taking the difference on the security knowledge score obtained in both questionnaires. Finally, “Activity grade” was the score obtained in the first assignment of the course.

The first questionnaire was answered by the participants at the beginning of the course, containing two blocks of questions: a block to obtain student profile information, with questions related to their enrollment in the university and their previous experience in security; and a second block with a pre-test of ten questions to assess security knowledge prior to working on the learning resources under study (e.g., questions about security tools, attacks, vulnerabilities, countermeasures following the conceptual map previously shown in Figure 2). The answers to these questions were used to compute a pre-study security knowledge score.

The second questionnaire was answered by the participants after reading the learning resources and working on the first graded assignment of the course, which was self-paced. This questionnaire contained four blocks of questions: the first block with five questions to evaluate the impression of the students about the format of the learning resource under study (questions P1.1 to P1.5 in Table 1); the second block with four questions for the student to self-assess their acquired knowledge (questions P2.1 to P2.4 in Table 1); the third block with ten questions to evaluate the actual knowledge acquired after reading the resources and working on the assignment. These questions were similar to the questions in the second block of the first questionnaire and served to compute a post-study security knowledge score. The “Improvement” variable was calculated taking the difference between the post-study and the pre-study scores. Finally, the second questionnaire contained a fourth block with a single open question, where participants could leave any type of comment. Questions P1.1 to P1.4 and P2.1 to P2.4 were graded using a Likert scale from 1 ( “Strongly disagree”) to 5 (“Strongly agree”). Question P1.5 could be answered selecting among several time interval from 0 to 20 h, in multiples of 5, and an option for those students that needed more than 20 h to finish the course assignment.

4.3. Tools

To carry out this study, we mainly used: Qualtrics [45] and Python. Qualtrics was used to create the questionnaires, collect data, and for an exploratory analysis. Subsequently, data were exported and processed with Python, mainly using the SciPy library [46] to compute the statistical tests mentioned below.

5. Results

In order to answer the research questions listed above, we compared the responses to the questionnaires of the two groups, i.e., Eg and Ng. This section describes the techniques used to extract information from the responses to the questionnaires and presents the main results. The next section discusses the results and answers the research questions.

In order to filter the results and evaluate the suitability of the learning resources, taking into account some important characteristics of the students in our institution, we segmented the subjects into several categories depending on their age and their previous theoretical knowledge in cybersecurity. In the first segmentation, we divided participants into three age categories: a first category with subjects under 30 years old, a second category with subjects older than 29 years and younger than 40, and a third category with the subjects 40 years old and above. In the second segmentation, we divided the subjects into two categories. We labeled subjects as having previous theoretical knowledge in cybersecurity if they had passed the courses Cryptography or Electronic Commerce, or achieved a high score in the first questionnaire pre-test.

In this study, it is highly relevant to evaluate the participants’ responses considering these criteria. Knowing the motivation behind the most common computer attacks is of utmost importance to understand their risks and be able to prevent them. A mere technical description of vulnerabilities and attacks may be useful for experienced subjects, but may not give enough context for more inexperienced ones. On the other hand, narrative texts may not provide the level of technical details required by some students with some cybersecurity experience. It is also important to study if young students, accustomed to a great variety of learning resources, or more senior students have different perceptions towards the narrative text.

For each of these categories, we computed the mean and the standard deviation for all the variables described in Table 1. These results are shown in

Table 2. Results of the Mann–Whitney U test of the variables related to the format of the learning resources.

		Expository Group (Eg)			Narrative Group (Ng)
Filter	Variable	Count	Mean	Std	Count	Mean	Std	Statistic	p-Value	Significance?
No filter	P1.1	49	4.12	0.56	62	3.90	0.74	1301.5	0.067	False
No filter	P1.2	49	4.31	0.68	62	4.21	0.73	1423.0	0.27	False
No filter	P1.3	49	4.04	0.61	62	3.65	0.83	1100.0	0.0035	True
No filter	P1.4	49	4.22	0.62	62	4.27	0.68	1433.5	0.29	False
No filter	P1.5	49	2.96	1.04	62	2.97	1.04	1485.5	0.42	False
No Prev. knowledge	P1.1	38	4.11	0.56	46	3.87	0.75	745.0	0.085	False
No Prev. knowledge	P1.2	38	4.29	0.65	46	4.15	0.76	803.0	0.24	False
No Prev. knowledge	P1.3	38	3.95	0.61	46	3.61	0.91	671.0	0.025	True
No Prev. knowledge	P1.4	38	4.24	0.59	46	4.24	0.71	850.0	0.41	False
No Prev. knowledge	P1.5	38	2.87	1.04	46	2.98	1.00	803.5	0.26	False
Prev. knowledge	P1.1	11	4.18	0.60	16	4.00	0.73	76.0	0.26	False
Prev. knowledge	P1.2	11	4.36	0.81	16	4.38	0.62	85.0	0.45	False
Prev. knowledge	P1.3	11	4.36	0.50	16	3.75	0.58	44.0	0.0063	True
Prev. knowledge	P1.4	11	4.18	0.75	16	4.38	0.62	76.0	0.27	False
Prev. knowledge	P1.5	11	3.27	1.01	16	2.94	1.18	74.5	0.25	False
Age < 30	P1.1	9	4.11	0.60	11	4.09	0.70	49.0	0.5	False
Age < 30	P1.2	9	4.22	0.83	11	4.18	0.87	49.0	0.5	False
Age < 30	P1.3	9	3.89	0.60	11	3.45	0.93	33.0	0.096	False
Age < 30	P1.4	9	4.00	0.71	11	4.45	0.52	32.0	0.073	False
Age < 30	P1.5	9	2.56	0.88	11	2.18	0.87	39.0	0.21	False
Age ≥ 30 and < 40	P1.1	21	4.14	0.65	28	4.00	0.54	258.0	0.2	False
Age ≥ 30 and < 40	P1.2	21	4.29	0.64	28	4.18	0.55	263.5	0.24	False
Age ≥ 30 and < 40	P1.3	21	4.05	0.59	28	3.82	0.82	247.5	0.15	False
Age ≥ 30 and < 40	P1.4	21	4.33	0.58	28	4.25	0.75	286.0	0.43	False
Age ≥ 30 and < 40	P1.5	21	3.19	1.03	28	3.00	0.94	271.5	0.32	False
Age ≥ 40	P1.1	19	4.11	0.46	23	3.70	0.93	166.0	0.065	False
Age ≥ 40	P1.2	19	4.37	0.68	23	4.26	0.86	210.5	0.42	False
Age ≥ 40	P1.3	19	4.11	0.66	23	3.52	0.79	131.5	0.0088	True
Age ≥ 40	P1.4	19	4.21	0.63	23	4.22	0.67	216.0	0.48	False
Age ≥ 40	P1.5	19	2.89	1.10	23	3.30	1.06	172.0	0.11	False

,

Table 3. Results of the Mann–Whitney U test of the variables related to the self-evaluation questions.

		Expository Group (Eg)			Narrative group (Ng)
Filter	Variable	Count	Mean	Std	Count	Mean	Std	Statistic	p-Value	Significance?
No filter	P2.1	49	4.00	0.61	62	4.11	0.52	1398.0	0.18	False
No filter	P2.2	49	3.80	0.79	62	4.08	0.61	1235.0	0.029	True
No filter	P2.3	49	3.43	0.87	62	3.90	0.74	1086.0	0.0025	True
No filter	P2.4	49	3.57	0.82	62	3.74	0.97	1305.5	0.089	False
No Prev. knowledge	P2.1	38	4.00	0.66	46	4.15	0.51	782.0	0.16	False
No Prev. knowledge	P2.2	38	3.71	0.80	46	4.11	0.57	640.0	0.0082	True
No Prev. knowledge	P2.3	38	3.47	0.80	46	3.93	0.71	613.5	0.0051	True
No Prev. knowledge	P2.4	38	3.53	0.80	46	3.85	0.92	659.5	0.02	True
Prev. knowledge	P2.1	11	4.00	0.45	16	4.00	0.52	88.0	0.49	False
Prev. knowledge	P2.2	11	4.09	0.70	16	4.00	0.73	82.0	0.38	False
Prev. knowledge	P2.3	11	3.27	1.10	16	3.81	0.83	64.5	0.11	False
Prev. knowledge	P2.4	11	3.73	0.90	16	3.44	1.09	74.0	0.24	False
Age < 30	P2.1	9	3.89	0.93	11	4.45	0.52	32.0	0.073	False
Age < 30	P2.2	9	3.67	1.00	11	4.18	0.60	36.5	0.13	False
Age < 30	P2.3	9	3.00	1.00	11	4.00	0.89	23.5	0.018	True
Age < 30	P2.4	9	3.44	0.88	11	3.64	1.21	39.5	0.22	False
Age ≥ 30 and < 40	P2.1	21	4.05	0.50	28	4.04	0.58	291.5	0.48	False
Age ≥ 30 and < 40	P2.2	21	4.05	0.67	28	4.07	0.72	288.0	0.45	False
Age ≥ 30 and < 40	P2.3	21	3.62	0.86	28	3.86	0.76	256.5	0.2	False
Age ≥ 30 and < 40	P2.4	21	3.71	0.78	28	3.82	1.02	265.5	0.27	False
Age ≥ 40	P2.1	19	4.00	0.58	23	4.04	0.37	210.5	0.4	False
Age ≥ 40	P2.2	19	3.58	0.77	23	4.04	0.47	139.0	0.01	True
Age ≥ 40	P2.3	19	3.42	0.77	23	3.91	0.67	145.0	0.022	True
Age ≥ 40	P2.4	19	3.47	0.84	23	3.70	0.82	183.0	0.17	False

and

Table 4. Results of the Mann–Whitney U test of the variables related to knowledge acquisition.

		Expository Group (Eg)			Narrative Group (Ng)
Filter	Variable	Count	Mean	Std	Count	Mean	Std	Statistic	p-Value	Significance?
No filter	Improvement	49	3.29	2.10	62	3.23	1.76	1492.0	0.44	False
No filter	Activity Grade	49	9.26	0.78	62	9.24	0.92	1511.0	0.47	False
No Prev. knowledge	Improvement	38	3.05	2.13	46	3.24	1.88	821.0	0.32	False
No Prev. knowledge	Activity Grade	38	9.24	0.83	46	9.15	1.06	852.5	0.37	False
Prev. knowledge	Improvement	11	4.09	1.87	16	3.19	1.42	59.5	0.079	False
Prev. knowledge	Activity Grade	11	9.32	0.60	16	9.50	0.00	80.0	0.13	False
Age < 30	Improvement	9	3.22	1.86	11	4.00	1.90	38.0	0.2	False
Age < 30	Activity Grade	9	8.83	1.00	11	9.32	0.60	37.5	0.1	False
Age ≥ 30 and < 40	Improvement	21	3.05	2.42	28	2.93	1.46	278.5	0.38	False
Age ≥ 30 and < 40	Activity Grade	21	9.21	0.96	28	9.14	1.22	290.5	0.45	False
Age ≥ 40	Improvement	19	3.58	1.89	23	3.22	1.98	202.5	0.35	False
Age ≥ 40	Activity Grade	19	9.50	0.00	23	9.33	0.58	199.5	0.1	False

. In order to compare the results from the Eg and the Ng, first we performed a Shapiro–Wilk test of normality [47]. This test indicated that most of the gathered data did not follow a Gaussian distribution. Hence, taking this into account and, also, that samples are independent, we performed non-parametric Mann–Whitney U tests [48] to evaluate the statistical significance of the differences between the Eg and the Ng. The results of these tests are also shown in Table 2, Table 3 and Table 4. In these tables, the first column indicates the ID of the analyzed variable listed in Table 1; the second column shows the filter applied to the data (i.e., analyzing all data without a filter, segmenting participants according to their previous theoretical cybersecurity knowledge, segmenting participants according to their age); columns 3 to 8 show the number of subjects, the mean and the standard deviation for the Eg and the Ng; and the last three columns show the results of the Mann–Whitney U tests, where the “Significance?” column is set to True if the null hypothesis can be rejected (p-value ≤ 0.05) and, therefore, consider that the samples from the two groups come from different distributions.

6. Discussion

This section addresses the research questions posed above and discusses the results of the study.

6.1. RQ1: What Is the Resulting Student Attitude towards this Format, in Respect to a Traditional One (i.e., Expository)?

To answer this research question we evaluated the subjects’ answers in questions P1.1 to P1.5. In general, the Ng participants value the use of storytelling positively, even though, as it can be seen in Table 2, the differences between the two groups are not statistically significant for most of the variables. The only variable where the difference can be considered statistically significant is P1.3 (“The contents are useful to start working on the graded assignment”). It is possible to observe that the main difference in this case comes from the subject group older than 40, where the subjects that used the novelette think that its content is less useful to start solving the graded activity than the subjects from the Eg (Eg: mean = 4.11, std = 0.66; Ng: mean = 3.52, std = 0.79).

However, if we also consider the answers of P1.5 (“How long did you take to study the learning resources and resolve the graded assignment?”), it can be observed that there is no statistical difference between the two groups, and therefore, it can be concluded that students in the Ng, regardless of their age, did not actually have to invest an extra effort to solve the first mandatory graded activity of the course.

Hence, we can consider that the students’ attitude towards the narrative format is good and comparable to their attitude towards the conventional learning resources, thus validating the novelette and its design methodology as a valid learning resource on equal footing to expository text. On that regard, it is worth noting the high value of P1.4 (“The content is relevant in the real world”) for students both with or without previous knowledge in computer security in the Ng group.

Furthermore, here it is pertinent to analyze the answers given by the participants in the open question included at the end of the second questionnaire. It is important to highlight that the participants belonging to the Eg valued the expository learning resource positively, but, in general, they mentioned that there is a lack of practical examples on how the computer attacks are performed and how to use the detection and prevention mechanisms.

Regarding the comments of the participants belonging to the Ng, the majority of participants responded very positively, expressing that they liked the novelette a lot, that they were delighted, and they highlighted the fact that a novelette is a different and motivating learning method. On the other hand, one participant evaluated the narrative aspects and the used language negatively. Nevertheless, it could be concluded that the novelette elicited more enthusiasm and was considered motivating for most of the students. However, students that do not like reading novels or expect a conventional resource that goes straight to the point may not be satisfied with it.

In broad terms, and especially when taking into consideration the remarks previously cited in Section 2, the conclusions of this research question are in line with the rest of the literature, thus reinforcing the idea that the use of fiction helps spark the interest of students. However, it is difficult to more closely contrast our results, beyond the general idea that using fiction produces a positive attitude. The main reason is that many studies (e.g., [32], [33] or [49]), do not present the results in much detail, basically moving to the discussion straightaway and agreeing with the premise. Further, in the literature in general (including these studies), there is a lack of testing where the use of fiction is contrasted to the use of a classical approach with expository text only. The motivational aspects of fiction in class are often studied in an isolated manner. Further, fiction is mostly used as a complementary resource, a motivational extra, instead of completely replacing expository text resources, such as in our approach. Nevertheless, the general conclusions stand.

Taking a look at studies that provide much more detailed information, the quantitative results in student motivation in [34] are quite similar to ours. This work includes the analysis of an additional dimension, “degree of transportation” [50], or the immersive impact of the fiction. However, again, it lacks any comparison with students that did not use fiction. Nevertheless, a study that actually compares the two different approaches can be found in [30]. Even though the positive reaction of students in this study (measured as the “fondness” quality) could be considered just above average the metrics related to engagement were much higher in study group based on fictional resources than the expository ones. This aspect differs from our study, were the positive reactions of students in the narrative group did not have this strange contrast, especially when considering the open questions.

Therefore, in comparison with the previous literature, we consider that our results provide an additional level of detail, with the addition of specific subquestions (i.e., P1.x) and the use of a control group based on expository text only. They also add an extra facet to the current literature, by providing results on a case where only fiction is used as a study resource, instead of just being complementary.

6.2. RQ2: What Is Its Influence in the Students’ Self-Perception of Their Knowledge Acquisition?

To answer this research question, we evaluated the subjects’ answers to questions P2.1 to P2.4. As it can be seen in Table 3, in general, the subjects using the novelette show a better self-perception of their knowledge acquisition than the Eg, or there is no statistical significance between the results of the two groups.

Analyzing the differences between the two groups globally (i.e., applying no filter), the novelette tends to be graded better overall. More specifically, the differences in P2.2 (“I know what kind of tools I should use to start exploring a system.”) and P2.3 (“I know what kind of tools I should use to protect my systems.”) can be considered as statistically significant. Regarding P2.2, the Eg has a significantly lower mean and higher variance than the Ng (Eg: mean = 3.80, std = 0.79; Ng: mean = 4.08, std = 0.61). These differences are even more remarkable in the category of subjects older than 40 (Eg: mean = 3.58, std = 0.77; Ng: mean = 4.04, std = 0.47). Regarding P2.3, globally, the Ng subjects show a greater self-perceived knowledge on protection tools than the Eg (Eg: mean = 3.43, std = 0.87; Ng: mean = 3.90, std = 0.74). Differences are also particularly meaningful in the category of subjects younger than 30 (Eg: mean = 3.00, std = 1.00; Ng: mean = 4.00, std = 0.89) and the ones older than 40 (Eg: mean = 3.42, std = 0.77; Ng: mean = 3.91, std = 0.67).

It is also noteworthy to see that approaching basic cybersecurity topics in a narrative manner is more effective for inexperienced students than for experienced ones. We can observe significant statistical differences between subjects without previous cybersecurity knowledge; meanwhile, the differences between the Eg and the Ng are not relevant for those that already know the theoretical basis on the area. The differences can be seen for those without previous knowledge in P2.2 (Eg: mean = 3.71, std = 0.80; Ng: mean = 4.11, std = 0.57), P2.3 (Eg: mean = 3.47, std = 0.80; Ng: mean = 3.93, std = 0.71) and P2.4 (“I would be able to put into practice what I have learned.”) (Eg: mean = 3.53, std = 0.80; Ng: mean = 3.85, std = 0.92).

Hence, the self-perception of knowledge acquisition is similar between the two groups. Nonetheless, the results show that the novelette can be a better tool as a starting point in the cybersecurity area. In this regard, the subjects without previous knowledge from the Ng sample answer more positively about being able to put into practice the learned concepts and their understanding on the tools described in the learning resources. This is particularly interesting considering that both learning resources (i.e., the novelette and the conventional expository learning resource) are written focusing on inexperienced technical readers, the general public, and also taking into account that cybersecurity is an eminently practical subject and the CompSec course is mainly taught and graded through practical assignments. From this point of view, storytelling can be considered a better approach to start studying this subject.

Again, the conclusions of this research question can be considered aligned with the previous literature, further justifying the introduction of fiction as study materials, when the educational design is sound. Many studies consider that a study of the motivational aspect also requires an assessment of student perception of acquired knowledge. Nevertheless, some of them stop at the attitudinal aspect, or just treat self-perception in very broad terms, such as “prompting critical reflection” (e.g., [31]). Even when this aspect is taken into account, it must be reminded that only a few make extra effort to compare it with classic learning methods, and qualitative results are seldom fine-grained or grounded on asking about very specific key topics in the curriculum, such as in our case. They mostly rely on a single generic question of the style “Do you think the fiction helped you learn?”.

Nevertheless, comparing our results to some previous studies, ref. [33] found a much stronger positive reaction in students than ours, whereas the results in [30,34] could be considered similar to ours, broadly speaking. Recent work on psychology courses [51] especially focused on reflection in self-knowledge, and again considered it positive, but added a very interesting additional dimension beyond the acquisition of pure information: the understanding of the emotions of others.

6.3. RQ3: What Is the Real Impact on the Student’s Knowledge Acquisition?

In order to answer this research question, we compared the results between the Eg and the Ng in the improvement questions and in the grade of the first course assignment. As Table 4 shows, there is no noticeable statistical difference between the two groups. Thus, we can conclude that the real impact on the student’s knowledge acquisition of using the novelette is the same than of using a conventional learning resource, again, reinforcing the soundness of its instructional design.

Interestingly, most of the literature stops at student self-perception (see RQ2) and do not actually make any effort to assess the students to test whether their knowledge has actually improved (or at least, is on par with an expository text approach). From the aforementioned studies, only [30] does, from a high level qualitative point of view. However, there are other studies that, in fact, only focus on knowledge acquisition, disregarding the motivational analysis, such as [28] (mostly based on infotainment, however, and not pure fiction), and can thus provide us some ability to compare our results. In this case, the subjects of the infotainment content performed significantly better than those using the expository version, whereas in our case performance, was similar. Therefore, even though the instructional design of the novelette may be considered sound, there is room for improvement.

7. Limitations

This section briefly discusses some notable limitations of this study in the use of storytelling as learning resource in a CompSec course.

First and foremost, the scope of the study is limited to its application in a 100% online institution with mostly adult learners (30 years or older), who have their own idiosyncrasies and learning styles [52], and bound to a single academic semester. It is also very important to note that the subject demographics are overwhelmingly skewed towards a male population. For a long time, and still to this day, this has been an endemic circumstance across all technical studies, not only at our institution, but at a cross-national level [53].

Additionally, even though the use of fiction or storytelling is not an uncommon occurrence in the academic literature, previous research in the creation of ad hoc learning resources using a simple fictional linear structure is lacking. Most cases use existing literature resources already created by other authors (e.g., existing novels, movies, etc.) or create more complex structures such as digital games (which is what this paper proposes to avoid in some circumstances). This makes the process of disputing or contrasting previous work on the topic with our study somewhat limited.

Finally, the analysis presented in this paper also shows that some subjects did not have a good perception towards the narrative format. This fact may be due to several reasons that could not be demonstrated from the study that we carried out. Among others, some of the possible reasons for dissatisfaction with this type of resource are: low interest in reading fiction texts, low interest in the plot of the novelette, preference for descriptive texts, lack of depth in certain technical details, perception of storytelling not being appropriate as a university learning resource, and so on. Knowing the exact reasons for the dissatisfaction could help create narrative texts with wider acceptance. Similarly, better understanding the reasons behind the greater dissatisfaction among older subjects would allow the writers to adapt the text to typical student profiles of the different courses. However, the study in its current form cannot provide a conclusive answer to this.

8. Conclusions and Future Work

This research has studied the relevance of using an instructional approach based on fictional storytelling, even when the target subject is highly technical or a context where this kind of approach is seldom used. The study presented the instructional design methodology of a digital book, the novelette, and its application in a computer security course in a CS bachelor degree aimed at adult learners, in a 100% online university. From the results of the study, it can be stated that the design process was sound and the resulting resource is perfectly on par with a classical approach based in expository text only: in format, usefulness, and its ability to achieve the learning outcomes in an appropriate amount of self-study time, as standalone content.

Going back to the Introduction of this work, the goal of a narrative approach is mainly directed at presenting a memorable experience that will help the reader, increasing also student engagement and self-fulfillment from a psychological standpoint. The results show evidence that this initial hypothesis is confirmed up to some point. This is especially shown in students with no previous knowledge in computer security, in their self-perception on how to apply the acquired knowledge. The narrative approach was also valued even by experienced students, who already knew about the topic because of their professional or academic background. After the experiment, students also expressed a high amount of satisfaction and encouraged us to use this kind of resource in other subjects.

The design and implementation also showed some limitations in this approach. For instance, and a result worth further study, would be that older students are more resistant and express their preference for a traditional expository approach. Further, after the whole design and implementation process of the novelette was finished, it was made evident that the cost of this approach is much higher, in time and resources (personnel, cost) than expository materials. Further, given its design process, updating the content is much more complicated, something very important in fields with constant advancements, such as technological ones. Nevertheless, this cost, in time and personnel, is still much lower than other game-based approaches, such as a professionally developed serious game. Still, it should not be used systematically, but surgically in subjects where changes are not expected in the short term, and obvious problems in student engagement and self-perception have been detected; however, not when the goal is exclusively knowledge or competence acquisition.

After this experience, new novelettes have been designed and successfully implemented in additional subjects at our institution, such as psychology, crisis management, social education, or youth crime. However, this study leads us to believe that storytelling could be continued to be applied to content related to other technological fields in engineering subjects. Further research should move in that direction, in order to collect more evidence, and especially focusing on how this approach affects on student perception of self-efficacy, and how or why age impacts its acceptance by adult learners. Moreover, in future evaluations of this type of learning resource, we will explore using the Kruskal–Wallis H test [54], a rank-based non-parametric test extending the Mann–Whitney U test that allows comparing more than two independent groups. It would also be interesting to dedicate further work on assessing the effectiveness of the novelette out of a formal Computer Science curriculum, in readers that acquired it from the digital book store, mostly interested in its narrative aspect as a standalone thriller story.