EduTVA: Metadata Schema for Educational Audiovisual Contents in Digital Television Environments

Abstract

The educational model faced unprecedented changes during the COVID-19 pandemic. In particular, the remote communication between teachers and learners was affected due to either the lack of or a poor Internet connection at home. Hence, television facilitates a more widespread communication technology with easier access than the Internet by playing an educational role for the mass audience through the broadcast of several learning resources (e.g., films, news, documentaries). However, there exists an increase in the educational audiovisual content, which needs to be organized and represented through description structures to be recovered. This paper proposes Educational TV-Anytime (EduTvA), a metadata schema for marking up educational audiovisual content and segments of the content in digital television. EduTVA emerges from an extensive revision of different metadata schemas for digital television and E-learning. Implementations of an EduTVA metadata authoring tool and an educational content search service support the evaluation of the user acceptance and functionality of the EduTVA metadata schema. Results show that EduTVA allows detailed descriptions of educational content and enables an accurate search of educational content.

1. Introduction

During the COVID-19 pandemic, the educational model faced unprecedented changes due to the lockdown restrictions established by different countries and territories around the world. Several educational institutions moved to online learning solutions for enabling remote communication between teachers and learners and continuing the education programs during the pandemic. Moreover, online education platforms (e.g., Coursera, EdX) offered access to most of their courses as Massive Open Online Courses (MOOC) [1]. However, people with fewer financial resources and from rural areas (particularly, in third-world countries) had difficulty accessing these online resources due to either the lack of or a poor Internet connection at home [2].

Television, on the other hand, facilitates a more widespread communication technology with easier access than the Internet. For decades, television has played an educational role for the mass audience through the broadcast of films, documentaries, news, and educational programs [3,4]. Its nature as an entertainment medium has been particularly important for permanent and informal learning [5,6]. In fact, interactive digital television has incorporated educational video content (such as from MOOCs) to promote the communication and interaction for knowledge, entertainment, and recreation in distance education.

Digital television then requires description structures, called metadata schemas, to organize, represent, and recover the audiovisual content [7]. If providers cannot find their own content, it is as if they lack it; if end-users cannot find the provider’s content, it is nonexistent for them [8]. Metadata schemas implement metadata and a neutral language (e.g., E-learning standards). Metadata provides information that describes, identifies, explains, and defines a resource for facilitating its recovery, usage, and management [7]. Neutral language establishes the value of such metadata and describes all the components of a particular resource using a representation that any actor in the television value-chain (i.e., content creator, content aggregator, broadcaster, end-user) understands.

Existing metadata schemas for audiovisual content rely on either educational metadata standards [9,10,11], multimedia metadata standards [12], or both educational and multimedia metadata standards [13,14]. However, these metadata schemas lack different elements for fully describing audiovisual educational content [15], particularly from a pedagogical perspective. Moreover, metadata schemas based on educational standards disregard the segmentation property of the television content, restricting the description of audiovisual content that includes one or more educational segments. Note that audiovisual television content can include different educational topics during the transmission, therefore, segmentation enables learning services to reuse a larger quantity of educational resources.

This paper proposes a metadata schema, called Educational TV-Anytime (EduTVA), which allows describing educational audiovisual content and educational segments of the content for the digital television environment. EduTVA extends TV-Anytime [16], a metadata schema for digital television, by adding educational metadata derived from a revision and mapping of various E-learning metadata schemas. We evaluated both the acceptance and the functionality of the EduTVA metadata schema. The acceptance evaluation involved implementing a metadata authoring tool and surveying a group of users that described educational content with the EduTVA metadata provided by the tool. The functionality assessment validated the educational content obtained from a digital television search service that we also implemented. Results show that EduTVA allows detailed descriptions of educational content through specific metadata and the creation of segments, enabling a more accurate search of educational content than with TV-Anytime.

The remainder of this paper is structured as follows. Section 2 exposes the state of the art that motivated this work. Section 3 introduces the EduTVA metadata schema for marking up educational audiovisual content. Section 4 details the evaluation of the EduTVA metadata schema. Section 5 analyzes the evaluation results. Finally, Section 6 presents the conclusions and future work.

2. Related Work

Digital television is the set of technologies involved in the transmission and reception of high-quality images and sound through digital signals. Its most distinguishing characteristic is the ability to interact between end-user and content providers through interactive applications which receive a request via a return channel.

Research related to educational solutions for digital television have faced problems regarding the description of educational audiovisual content. Metadata schemas for describing television content differ from those describing educational content in other environments (e.g., Web). Therefore, some works have proposed different metadata schemas for describing educational audiovisual content in digital television environments. These works rely on either educational metadata standards, multimedia metadata standards, both educational and multimedia metadata standards, and customized metadata.

Works based only on educational metadata standards [9,10,11,17,18] are restricted either to the Learning Object Metadata (LOM) or to their application profiles for enabling the description of educational audiovisual content. Both IEEE standards 1484.12.1 [19] and 1484.12.3 [20] define LOM, which aims at allowing the structured description of educational content, guiding the process of marking up the content, and enhancing the content search, evaluation, acquisition, and usage. Nevertheless, LOM is not a metadata schema developed for a television environment, so content providers would potentially disregard it in a real educational digital television deployment. Moreover, LOM lacks all the user requirements for different educational contents [15].

On the other hand, few works with educational purposes purely rely on multimedia metadata standards. These works use descriptive features from audiovisual environments, particularly, the segmentation description. However, they lack elements for educational description. For example, Rey-Lopez et al. [12] identify additional content related to the segments of television programs via semantic links coming from the segmentation data of MPEG-7 (multimedia content description interface) [21]. Although this approach considers content segmentation, MPEG-7 lacks specific metadata for educational resources, such as the level of learning. Note this standard would require a controlled vocabulary with educational information for supporting the description of educational content.

Works using both educational and multimedia metadata standards are scarce too. Santos and Iano [13] propose an ontology of interoperable metadata for Web, digital television, and mobile devices by integrating MPEG-21 (multimedia framework) and the Sharable Content Object Reference Model (SCORM) standard. Note this solution enables interoperability between existing standards, television, and learning environments. However, it fails to converge to a single television metadata schema that includes elements for describing content from an educational perspective. In addition, the Brazilian specification for Learning Objects Based on Agents (OBAA; Objetos de Aprendizagem Baseados em Agentes) extended LOM metadata categories to deal with interoperability of digital content across multiple platforms (e.g., Web, digital television, and mobile devices) [14]. OBAA includes metadata elements for supporting educational and segmentation requirements. However, a former evaluation of these elements suggests that further research is required to adapt and adjust these categories into pedagogical planning [22]. Moreover, OBAA lacks specific metadata for achieving a full description of educational content (see Table 1).

Lastly, the customized metadata has been particularly designed for the corresponding project [23,24,25]. For instance, Ye [23] investigates the metadata rules for cataloging educational content regarding the radio and television program classification in China. The author further combines the Chinese E-Learning Technology Standard-42 (CELTS-42) with the cataloging rules to promote teaching and satisfy the actual demand of educational television. However, this metadata schema fails to describe the educational content from a pedagogical perspective.

In summary, the study and analysis of the different related works have allowed finding the following gaps:

• Metadata schemas that consider the features of educational content and television, lack metadata elements for fully describing educational content (e.g., pedagogical planning, segmentation).
• There is no application profile for educational audiovisual television content, developed from existing metadata schemas for television content and optimized for educational content.
• The literature does not discuss an evaluation method for a metadata schema to verify its functionality in the context for which it was designed.

Unlike these works, this paper adds educational metadata to a digital television standard to allow marking up audiovisual educational content as well as describing educational segments from audiovisual content. Furthermore, this paper evaluates the proposed metadata schema by implementing two applications for educational content in digital television: a metadata authoring tool and a search service.

Table 1. Educational metadata from LOM, MLR, LRMI, OBAA, and EduTVA.

Metadata Group	Metadata					Potential Values in EduTVA
	LOM [19]	MLR [26]	LRMI [27]	OBAA [14]	EduTVA
Educational resource	Interactivity type	—	Interactivity type	Interactivity type	Interactivity type	Active, expository, and mixed
	Learning resource type	—	Learning resource type	Learning resource type	Educational resource type	Exercise, animation, exam, narrative text, experiment, lesson, and self-evaluation
	—	—	Educational use	—	Educational use	Established by whoever carries out the marking up
Educational audience	Intended end user role	Audience role	Educational role	Intended end user role	Intended end user role	Student, student with special education needs, student with high intellectual capacities, student with late integration into the education system, student with other specific educational support needs, public in general, teacher, tutor, and family
	Typical age range	Max and min ages	Typical age range	Typical age Range	Typical age range	All ages: children (0-3 years, 4–7 years), preadolescents (8–13 years), adolescents (14–15 years, 16–17 years), and adults (18–24 years, 25–34 years, 35–44 years, 45–54 years, 55–64 years, older than 65 years)
	Language	Audience language	—	Language	Intended end user language	Established by whoever carries out the marking up
Educational context	Context	Audience level	—	Context	Educational context	All the contexts: primary, secondary, university, postgraduate, and permanent learning
Annotation	Annotation	Annotation text	—	—	Annotation	Established by whoever carries out the marking up
Educational results	—	—	—	—	Ability	Those established in the Bloom’s Revised Taxonomy

Table 1. Educational metadata from LOM, MLR, LRMI, OBAA, and EduTVA.

Metadata Group	Metadata					Potential Values in EduTVA
	LOM [19]	MLR [26]	LRMI [27]	OBAA [14]	EduTVA
Educational resource	Interactivity type	—	Interactivity type	Interactivity type	Interactivity type	Active, expository, and mixed
	Learning resource type	—	Learning resource type	Learning resource type	Educational resource type	Exercise, animation, exam, narrative text, experiment, lesson, and self-evaluation
	—	—	Educational use	—	Educational use	Established by whoever carries out the marking up
Educational audience	Intended end user role	Audience role	Educational role	Intended end user role	Intended end user role	Student, student with special education needs, student with high intellectual capacities, student with late integration into the education system, student with other specific educational support needs, public in general, teacher, tutor, and family
	Typical age range	Max and min ages	Typical age range	Typical age Range	Typical age range	All ages: children (0-3 years, 4–7 years), preadolescents (8–13 years), adolescents (14–15 years, 16–17 years), and adults (18–24 years, 25–34 years, 35–44 years, 45–54 years, 55–64 years, older than 65 years)
	Language	Audience language	—	Language	Intended end user language	Established by whoever carries out the marking up
Educational context	Context	Audience level	—	Context	Educational context	All the contexts: primary, secondary, university, postgraduate, and permanent learning
Annotation	Annotation	Annotation text	—	—	Annotation	Established by whoever carries out the marking up
Educational results	—	—	—	—	Ability	Those established in the Bloom’s Revised Taxonomy

3. Overview of the EduTVA Metadata Schema

In this section, the proposal and implementation of a new metadata schema for educational audiovisual contents for digital television environments, named EduTVA, is presented. Because television is the primary environment for which the proposed metadata schema is designed, it is necessary to start from a metadata schema base of that environment. The new schema’s development process has followed the next phases: first, the selection of a metadata schema base, then, the choice of the set of educational metadata to be adapted, and finally, the new proposed metadata schema.

3.1. Base Metadata Schema Selection

For several reasons, it is preferable to use or adapt existing standards (MPEG-7, TV-Anytime, EBUP/Meta, EBU Core, PBCore, and SMPTE) instead of developing our metadata schemas and vocabularies from scratch. Using a current standard can offer, in the first place, a cost-saving because it is unnecessary to create a new schema along with its usage patterns or vocabulary. Second, it can offer access to support and advice because a well-known standard probably has a user community that was built over time. The third is usability because, if users are already familiar with the metadata structure or terminology, they can use the collection of resources described with the standard faster and easier. Fourth, finding the resources make it easier to share the resources. Finally, sustainability, because common standards facilitate the sharing of a collection of resources between different entities (e.g., systems and people) [28].

This work proposes a metadata schema for a digital television scenario that can reuse its productions (such as movies, news, documentaries, and animations) for an educational purpose. Consequently, it is necessary to build on an existing metadata schema to describe television content and adapt it to the characteristics of metadata for educational content. For this reason, TV-Anytime [16] is selected as the base standard for constructing the new metadata schema because it defines most of the metadata set in the H.750 [29] recommendation. In addition, the set of metadata that composes it is simple, so it allows a simple and easy-to-understand markup. Its structure and syntax are not complicated because it provides a temporal segmentation that responds well to the physical and operational constraints of broadcasting. In addition, it defines two metadata oriented to the educational description of the content, which means that the structure already contains a class of metadata intended for educational tagging. Finally, it is the internationally accepted schema in the field of consumption [30].

3.2. Educational Metadata

Considering the results from a previous work [15], we reviewed different metadata schemas that describe educational content and selected the set of educational metadata to build our new metadata schema. We analyzed the elements from the LOM standard since it is the most well-known in E-learning [31,32]. We also examined the Metadata for Learning Resources (MLR) [26] and the Learning Resource Metadata Initiative (LRMI) [27] standards since they add pedagogical elements that older metadata schemas lacked [15]. Therefore, we mapped the educational elements between these standards while applying three selection criteria:

1.

The educational metadata that the Learning Object Metadata for COlombia (LOM-CO) application profile established as obligatory by the National Ministry of Education of Colombia: “interactivity type”, “interactivity level”, “intended end user role” and “context” [33]. Of this metadata, “interactivity level” was eliminated because, in an audiovisual resource, the user plays a passive role, with no interaction.

2.

The metadata that simultaneously met the following two conditions were selected: those defined in at least two of the mapped schemas and apt for this work scenario. Of these metadata, “typical learning time” (“time required” for LRMI) was excluded because it represents the time that it takes to work with the educational resource, so, for a television resource, it translates merely into its duration.

3.

The remaining metadata was analyzed, and that which fit in the digital television scenario (for which the metadata schema is created) was selected, obtaining the “educational use” metadata defined by LRMI.

In addition, an “Ability” metadata is added to describe the competencies that a student can develop through the educational audiovisual resource. However, it should be noted that its values can come from other educational models if required.

We classified the selected metadata into five groups regarding their objectives: educational resource, educational audience, educational context, annotation, and educational results. Table 1 shows the EduTVA metadata in parallel with the educational metadata present in the reference schemas (i.e., LOM, MLR, and LRMI) and in OBAA [14].

Next, a description of the metadata selected with its possible values is presented. The “educational resource” group is focused on the classification of the resource:

• Interactivity type: Learning mode supported by the resource. Possible values are: active (when the viewer actively participates with the resource), expository (if the knowledge is exposed as the content progresses), and blended (if active and expository).
• Educational resource type: The predominant types that characterize the resource from an educational point of view. Possible values: Exercise, Animation, Exam, Narrative text, Experiment, Auto-evaluation, Lesson (class, conference, explanation).
• Educational use: It indicates the educational purpose of the audiovisual resource. It has no selectable values.

The “educational audience” group focuses on the description of the potential user:

• Intended end-user role: Main user(s) for which the resource was created. Possible values: student, student with special educational needs, student with high intellectual capacities, student with late integration into the educational system, student with other specific educational support needs, public in general, teacher, tutor, and family.
• Typical age range: Age groups of the target audience. Possible values: All ages, Children (0–3 years, 4–7 years, 8–13 years, 14–15 years), Adolescents, Preadolescents and Adults (16–17 years, 18–24 years, 25–34 years, 35–44 years, 45–54 years, 55–64 years, older than 65 years).
• Intended end user language: Language of the target audience.

The “educational context” group defines the context of the application of the resource:

• Educational Context: Environments in which the resource can be used. Possible values: All the contexts, Primary, Secondary, University, and Postgraduate/Permanent learning.

The “annotation” group contains the metadata which permits adding descriptions, data or comments:

• Annotation: Comments related to the educational resource. It does not have selectable values.

The “educational results” group describes the resource’s objectives:

• Ability: Thinking competencies achieved through interaction with the resource. The competencies represent the actions provoked in the destination user. That is to say, the cognitive processes involved in the learning process. Possible values: Those established in the Revised Taxonomy of Bloom [34], or skills that you wish to set as the value.

On the other hand, TV-Anytime allows denoting related collections or resources through the ProgrammeGroup element. EduTVA inherits this element, then, the audiovisual educational contents can be grouped such that a group may contain any number of contents, and a content can be a member of any number of groups.

3.3. New Metadata Schema: Adapting the Base Schema

To adequately carry out an adaptation process of TV-Anytime, the base metadata schema selected, it was necessary to follow three steps. First, to choose the set of elements of TV-Anytime suitable for the description of educational audiovisual content. Second, to adapt the educational metadata selected in the previous section to the elements of TV-Anytime. Lastly, to identify the elements of TV-Anytime that describe segments of the content and adapt the educational metadata to them.

As the new schema, EduTVA is the result of the adaptation of the set of educational metadata selected from TV-Anytime. So, it uses the same namespaces established in its in-force XML Schema Definition (XSD) document (i.e., XML document for metadata representation) and inherits all the types of basic data of that standard. Furthermore, it acquires all the attributes of the selected elements and the sets of values for some metadata.

The selected metadata are shown in Figure 1. The process starts at the green point and proceeds counterclockwise. The cardinality of each item selected for EduTVA is specified. The type of data that has had any change is also indicated. If the data type of an element is not detailed, it inherits all the properties defined by TV-Anytime. The attributes of EduTVA elements fully inherit the TV-Anytime standard, and their descriptions can be found in [16,35].

TV-Anytime allows us to define various contexts for the same content. For example, some content used in a travel magazine with an emphasis on entertainment, together with a collection of content with information about a trip, can also be used in an educational geography program. The previous is possible thanks to the element ContextAttributesType, which is a type of abstract data that serves as a base to define the attributes of five contexts: data broadcasting context, interstitial context, the context of an application that contains a set of functions that will be executed in the consumer domain, game context, and educational context.

This work is focused on the description of the educational context of the resource, because of this, it adapts to the element EducationalContextAttributesType of TVAnytime using the educational metadata defined in Section 3.2. The elements that originally composed EducationalContextAttributesType do not cover the whole educational context, which can be defined through the metadata used by the educational schemas. Nevertheless, TV-Anytime has a simple syntax that can be modified easily and adapted according to the set of educational metadata selected for EduTVA.

Three new data types have been created, and the existing elements have been redefined, considering the classification of the new educational metadata defined in Section 3.2. Thus, in the blue point, five new features have been obtained that make up ContextEducativeType, as can be seen in Figure 1.

Metadata created for EduTVA are EducationalResource of the type EducationalResourceType to describe the type of educational resource. EducationalContext of the type ControlledTermType to describe the context of the application of the resource. EducationalAudience of the type EducationalAudienceType to describe the target educational audience. Annotation of the type TextualType to carry out annotations related to the educational resource. Finally, EducationalResults of the type EducationalResultsType to describe the educational results obtained by interacting with the audiovisual resource. Each element added by EduTVA represents one of the metadata groups defined in Section 3.2, and its elements represent the defined metadata in the corresponding group. The new types of data added by EduTVA are EducationalResourceType, EducationalAudienceType, and EducationalResultsType.

It is important to remember that TV-Anytime implements the concept of classification schemes to define a set of values for certain types of metadata. EduTVA inherits this property. Therefore, if the type of data of an element is ControlledTermType, then its values are taken from a classification scheme. EducationalContext, EducationalType, InteractivityType, EducationalRole, TypicalAgeRange, and Ability are EduTVA metadata of this type of data; therefore, their values can be controlled and flexible. The types of data TextualType, LanguageType, and ControlledTermType are defined by TV-Anytime and are inherited by EduTVA.

On the other hand, in the TV-Anytime standard, in Figure 1 from the red dot you can see the selected elements necessary to describe one or more educational segments of the content. Of this structure, the element ExtendedSegmentDescriptionType was conserved to add the element ContextAttributes which permits describing the attributes of the educational context of the content. This last element’s type of data must be implemented employing the element EducationalContextAttributes, as indicated previously. Figure 1 show a segment description example. The red circle in the center describes, also counterclockwise, the elements that are added to a search in the field of Biology with the question “What is a species and what is a hybrid.”

It is important to note that this segmentation method allows descriptions to be made as detailed as wanted. Each segment created has its own basic and educational description, which are independent even of the general description of the whole resource. In the case of existing multiple segments, it is recommended that the basic description and educational metadata of the resource, suitably synthesize all the topics covered in the content. However, this is a conscientious work of the person who performs the making up process. The EduTVA XSD files are available on a public repository [36].

3.4. Adaptability to Knowledge Representation

TV-Anytime is a comprehensive and spread standard in which specifications allow exploiting local persistent storage in consumer electronics platforms. However, the delivered content to the user could not be appropriate and not aligned with its expectations. Hence, approaches based on understanding the user needs and preferences can provide appropriate content through a rich Internet application. To this, the schema metadata proposed in this work could be adapted to a specific markup language to leverage the potential of applications oriented to solving complex tasks based on Knowledge Representation and Reasoning (KRR). In consequence, the in-force XSD specification from the TV-Anytime standard can be mapped to ontology representations, such as the Web Ontology Language (OWL) following the guidelines proposed by Wheeler et al. [37] and using the XSD to OWL tool [38]. Thus, new applications based on artificial intelligence as a query agent [39] can retrieve educational content according to user needs and expectations. The mapper structure and the query agent can be implemented in the back-end application to not overload the user appliances. Finally, the EduTVA schema could further support hybrid recommendation systems that integrate ontology representations [40,41,42,43].

4. Evaluation

The evaluation of the EduTVA metadata schema is broken into two parts. First, we validated the acceptance of the proposed educational metadata by surveying a group of users that marked up content with a metadata authoring tool that implements EduTVA. Second, we evaluated the functionality of EduTVA by inspecting the content obtained from a digital television search service when executing different queries.

4.1. Acceptance of Educational Metadata

For evaluating the acceptance of the educational metadata proposed in EduTVA, first, we require a metadata authoring tool to mark up audiovisual content. Although different metadata authoring tools enable marking up content with TV-Anytime, such as M-Tool [44] and ETRI’s Tool [45], they lack the educational metadata that this evaluation focuses on. Therefore, we implemented EduTVATool, a Web-based metadata authoring tool that implements the EduTVA metadata schema (see Section 3) to enable marking up content with the proposed educational metadata. We resolved implementing a Web tool to promote collaborative marking up audiovisual content in future work.

EduTVATool allows metadata authors interacting with a Web interface to select an audiovisual resource, to visualize the resource, and to describe the resource (or a segment of the resource) by entering the general and educational metadata. Figure 2, for example, depicts the educational metadata form of EduTVATool for marking up the selected resource. We achieved the last version of the tool after conducting two preliminary evaluations with two experts on usability. Each preliminary evaluation followed the concurrent think-aloud technique [46], that is, users interact with EduTVATool while expressing their thoughts about their interaction aloud; if conducting the evaluation remotely, they write their thoughts down [47]. Both evaluations ran independently and at different moments, using different versions of EduTVATool.

Continuing with the evaluation of the EduTVA metadata, a group of 15 participants (different from the preliminary evaluations) used EduTVATool to mark up a given educational audiovisual resource. Then, each participant completed a survey about the educational metadata. This survey also included a questionnaire about the usability of EduTVATool for validating that its Web interface meets the usability criteria. Such a usability assessment allows detecting any bias in the metadata acceptance survey due to usage misunderstanding of the tool. The whole process was under the supervision of an expert. Following the details of the evaluation.

• Purpose: the evaluation aims at measuring the level of acceptance of the EduTVA metadata for marking up educational characteristics of the audiovisual resource. This evaluation also validates if participants understand how to describe the educational resource using the Web interface of EduTVATool.
• Participants: professors and teachers that incorporate technological and pedagogical competences into the education process. Technological competence refers to the capacity of selecting and understanding a variety of technological tools, and combining them into the education process in a proper and responsible way. Pedagogical competence refers to the capability of using technologies for strengthening the education process while recognizing their benefits and limitations in the professional development of the students [48].

  -

  Number of participants: 15, nine university professors and six high school teachers. The number of participants is based on the number of experts and tutors that evaluated SugarTube [49].

  -

  Age range: from 28 to 60 years.

• Scenario: four steps form the evaluation process: introduction, demonstration, task, and data collection. During the introduction step, the participant receives a document that explains the process. Following a summary of the information in the document:

  -

  Participants use EduTVATool, a Web-based metadata authoring tool that allows describing an audiovisual resource from an educational perspective.

  -

  The resource to describe corresponds to the six episode of “Proyecto G”, a science television program of the educational and cultural channel Encuentro from Argentina [50]. All the episodes of this educational program are freely available for academic use [51].

  -

  EduTVATool provides a default mark up for the general basic description of the resource (e.g., title, genre, actors).

  -

  During the test, a supervisor observes the participants’ behavior and registers the time for marking up the resource.

  -

  After marking up the resource, the test redirects the participants to a survey that asks about their perception of the educational metadata as well as their level of satisfaction with EduTVATool.

  The second step of the evaluation provides a demonstration of EduTVATool to the participants by briefly showing the available functions and pointing to the resource to mark up. This step also shares the EduTVATool login credentials with the participants. Third, in the task step, the participants log in EduTVATool for marking up the given educational resource. The evaluation process grants total freedom for the marking-up process, including using help support and marking up any (or none) of the segments.
  Finally, the data collection step presents two questionnaires to the participants and records all the responses. The main questionnaire consists of six items asking about the participants’ perception of the educational metadata for describing the resource. Section 5.1 details the questions and responses for this questionnaire. The second questionnaire is the System Usability Scale (SUS) [52], an industry standard for reliably and effectively measuring the usability of products and services. Section 5.1 summarizes the results for the ten items that form the SUS questionnaire. Note the data collection step goes immediately after the task step.
• Metrics: the responses for the two questionnaires represent subjective metrics that capture the participants’ perception of the EduTVA metadata and the usability of EduTVATool. The EduTVA metadata questionnaire uses three multiple choice questions, two yes/no questions, and a text response question. On the other hand, each question in SUS presents five possible answers that range from strongly agree to strongly disagree.
  Moreover, we established two objective metrics that the supervising expert measures by observing the participants’ behavior. First, the task successful completion, which indicates the percentage of participants that completed the task successfully, either with or without errors. This metric validates if EduTVATool is completely intuitive for the participants. Second, the error-free rate, which refers to the percentage of participants that completed the task without any errors. This metric verifies if EduTVATool is bug-free since the participants could generate exceptions due to improper usage.
• Probing method: the evaluation follows retrospective probing [53], which waits until participants complete the given task to ask questions about their thoughts and actions. This probing method allows monitoring the participants’ behavior during the whole evaluation process.

4.2. Functionality Assessment

For evaluating the functionality of the EduTVA metadata schema, we implemented Crayon, an interactive IPTV application that provides a search service of educational content that has been described with a medatada schema, such as TV-Anytime and EduTVA. Crayon enables users to input a search query, by either text or voice, compares it with the whole metadata of every educational content, and displays the results matching that query. Figure 3 shows the search results graphical interface for a query example. Since no existing IPTV applications can read the metadata structure of EduTVA, we implemented this simple search service for conducting the evaluation. Nevertheless, EduTVA can be adapted to knowledge-based applications, as discussed in Section 3.4. Further research is needed to evaluate the performance of EduTVA in such advanced applications.

Note that the back-end search service in Crayon might operate in a datacasting deployment for assisting the education of people with poor Internet access at home [54,55]. Particularly, such a deployment involves a television broadcasting station with a viable Internet connection for accessing an IPTV service, which offers a search service such as Crayon. The television station can filter educational resources from the IPTV service and broadcast the educational content via digital terrestrial television, such as DVB-T and ATSC, to reach multiple learners over a wide area [56,57]. Furthermore, the television broadcasting station might implement a low-bandwidth return path (e.g., SMS, GSM) to realize interactivity with the IPTV search service [57], as well as broadcast multimedia content from sources other than the IPTV service. The implementation of the datacasting deployment is out of the scope of this paper.

The functionality assessment then consists of validating if the search service in Crayon returns the expected results when executing different search queries to look for educational content described using both TV-Anytime and EduTVA. To measure this functionality, we adapted the OPerational Consistency (OPC) quality attribute from the ISO/IEC 25010 standard [58].

Table 2. Details of the OPerational Consistency (OPC) metric.

Characteristic	Details
Purpose of the metric	OPerational Consistency (OPC) aims at measuring how consistent are the results of the search service
Method of application	Counting and validating the educational content obtained on the search service when executing different queries
Measurement, formula, and data element computations	OPC = 1—(IR / ER); ∘ IR: Inconsistent Results ∘ ER: Expected Results
Interpretation of value	$0\le$ OPC $\le 1$; the closer to 1, the better
Metric scale type	Absolute
Measure type	∘ IR units: count ∘ ER units: count ∘ OPC units: proportion
Target audience	User

details the OPC metric as used in this evaluation.

For the television content to look for in Crayon, we employed 15 programs, most of them educational.

Table 3. Television programs in Crayon.

ID	Program	No. of Contents	No. of Segments
P1	Art Attack	1	0
P2	El Mundo de Beackman	1	0
P3	Física Entretenida	3	0
P4	Brain Games	2	0
P5	La Ciencia de lo Absurdo	6	0
P6	La Pola	2	0
P7	American Genius	3	0
P8	Mythbusters	1	0
P9	Profe en Casa	1	0
P10	Profesor Súper O histórico	11	0
P11	Profesor Súper O idiomático	8	4
P12	Proyecto G	8	0
P13	Hacking the System	1	0
P14	Saber y Ganar	1	0
P15	TVAgro	11	0
TOTAL		60	4

describes each television program, including their number of contents and segments. Note that only the program P11 presents segments; the four segments correspond to two segments of two different pieces of content. In total, Crayon incorporated 60 pieces of television content.

The evaluation involved two participants. Participant A used EduTVATool (see Section 4.1) to mark up all the 60 pieces of television content with both TV-Anytime and EduTVA. The participant entered the same description for the two metadata schemas, except for the educational elements added by EduTVA. Participant B defines and executes on Crayon ten different search queries to look for such marked up content.

Table 4. Search queries for EduTVA functionality assessment.

ID	Search Query
S1	Physics
S2	Physics experiment
S3	Experiment with alternating current
S4	Fertilization of plants
S5	Plants fertilization lesson
S6	History of La Pola
S7	Idiomatic mistakes by Profesor Super O
S8	Identify language errors with Professor Super O
S9	Biology for teachers
S10	Biology

depicts these search queries, consisting of either a simple word or a set of words. The search queries refer to either an academic field (e.g., biology), a type of resource (e.g., physics experiment), or a competence (e.g., plants fertilization lesson).

Both participants watched all the 60 pieces of content and selected the expected contents and segments for each search query.

Table 5. Expected content per program and search query.

		Search Query (see Table 4)
	ID	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10
Program (see Table 3)	P1	0	0	0	0	0	0	0	0	0	0
	P2	1	1	0	0	0	0	0	0	0	0
	P3	3	3	1	0	0	0	0	0	0	0
	P4	2	2	0	0	0	0	0	0	0	0
	P5	6	6	0	0	0	0	0	0	6	6
	P6	0	0	0	0	0	0	0	0	0	0
	P7	3	0	0	0	0	0	0	0	0	0
	P8	1	1	0	0	0	0	0	0	0	0
	P9	1	1	0	0	0	0	0	0	0	0
	P10	0	0	0	0	0	1	11	0	0	0
	P11	0	0	0	0	0	0	8	8	0	0
	P12	8 + 4	7 + 4	0	0	0	0	0	0	0	0
	P13	0	0	0	0	0	0	0	0	0	0
	P14	0	0	0	0	0	0	0	0	0	0
	P15	0	0	0	6	6	0	0	0	0	5
	Total	29	25	1	6	6	1	19	8	6	11

details the number of expected results per program and search query, where $+s$ indicates the number of expected segments. For example, when executing the search query S2, the participants expect 1, 3, 2, 6, 1, 1, and 7 pieces of content from programs P2, P3, P4, P5, P8, P9, and P12, respectively, plus 4 segments from the program P12, for a total of 25 expected results.

5. Results and Discussion

This section presents the discussion of the results obtained during the evaluation. First, we analyze the responses to the EduTVA acceptance questionnaire that evaluates the proposed educational medatada. Second, we describe the OPC results that validate the functionality of the EduTVA metadata schema in the television context.

5.1. Acceptance of Educational Metadata

The EduTVA acceptance questionnaire consists of six items. Following, we analyze the responses given by the 15 participants (see Section 4.1) to each question.

Question 1.

Which of the following metadata do you consider difficult to understand?

Five out of the 15 participants considered some of the educational metadata difficult to understand. As shown in Figure 4, three participants agreed on “annotations” and thee on “educational use”. The “ability”, “intended end user role”, and “educational context” metadata received only one vote each, so we consider them manageable. Note the “annotations” and “educational use” metadata provide no list of values to select but the participants must enter a value as they consider. Therefore, metadata presenting a list of values to select facilitates their familiarization and understanding.

Question 2.

Which of the following metadata do you think is not adequate to detail the educational characteristics of the audiovisual resource?

The responses reflect the participants’ conformity with all of the metadata proposed by EduTVA to mark up educational audiovisual content.

Question 3.

Do you think other data is required that allows describing the educational characteristics of the audiovisual resource? Which?

Two out of the 15 participants proposed extra data for describing educational audiovisual content, in addition to the EduTVA metadata. The proposed data are “methodology”, “required analysis”, “objectives”, “student profile”, and “subtopics or related topics”.

We disregard “methodology”, or educational method, from the EduTVA metadata because educational audiovisual content can be part of any educational method. Therefore, educational content is independent from the method. In addition, it is important to recall that EduTVA aims at marking up educational audiovisual content in the television context, where no curriculum exist. We also ignored “required analysis” because the participant lack providing a clear explanation.

On the other hand, EduTVA supports “objectives”, “student profile”, and “subtopics or related topics”. First, participants can register “objectives” as annotations. Second, the EduTVA element “EducationalAudience” deals with “student profile”. Finally, the TV-Anytime element “genre”, included in the general description of EduTVA, addresses “subtopics or related topics”. TV-Anytime provides a taxonomy of values for “genre” to indicate knowledge areas (i.e., topics), such as applied sciences, physics, literature, history, and philosophy. Moreover, the ”genre” metadata can contain one or more values for supporting “subtopics or related topics”.

Question 4.

For each of the following metadata, indicate whether their selection values were clear and easy to understand.

Figure 5 depicts the percentage of participants per educational metadata reporting that the values to select are clear and easy to understand. The results show that most of the participants ($>86\%$) clearly and easily understand the values to select for these educational metadata.

Question 5.

As a teacher and television viewer, do you think it is important to highlight the competencies that can be achieved by interacting with the audiovisual resource and, with that information, use the resource for a teaching activity?

All the participants agreed on the importance of describing the developed educational competences when interacting with audiovisual content. Moreover, they agreed on using this information to select the content across different teaching activities.

Question 6.

As a teacher and television viewer, do you think it is useful to know if a segment of an audiovisual resource is educational and what educational characteristics it has?

All the participants agreed on the importance of knowing whether some audiovisual content contains an educational segment to use it within the educational process.

In addition to the EduTVA acceptance questionnaire, we employed the SUS questionnaire [52] for evaluating the usability of EduTVATool and detecting any bias in the acceptance survey due to usage misunderstanding of the tool. The responses from the participants indicate a strong agreement with the positive questions and a strong disagreement with the negative questions, resulting in a SUS score of 86.17 for EduTVATool. Note this score is greater than the average SUS score of 68 that experts defined based on multiple studies [59]. Therefore, EduTVATool qualifies as usable, allowing us to infer that no bias affected the EduTVA acceptance questionnaire due to usability issues of the tool.

Finally, recall that an expert supervised the participants during the whole evaluation process to measure and observe their behavior. Regarding the two objective metrics, first, all the participants completed the given task successfully, achieving a task successful completion of 100%. Second, all the participants completed the task without any errors providing an error-free rate of 100%. On the other hand, the supervising expert observed the following behavior from the participants.

1.

Half of the participants filled in the metadata form while watching the audiovisual content, whereas the other half watched the content first and then filled out the metadata form.

2.

All the participants reviewed the tooltip attached to every metadata field but only a few opened the tool’s general support.

3.

A third part of the participants updated the content default description, particularly, the genre taxonomy by selecting the related knowledge areas (e.g., physics, history).

4.

All the participants used the video playback options (e.g., pause, play, forward, rewind) to create the content segments.

5.

46,66% of the participants created segments (two to five segments).

6.

Some participants read the content default description before starting the mark-up.

7.

Some participants watched the audiovisual content on full-screen.

8.

The participants displayed the tool Web interface on diverse screen sizes without having any visualization issues.

9.

Most participants entered similar values for the educational metadata, particularly, for the fields that provide a list of values to select.

Based on these behavior observations, we can conclude that the EduTVA metadata schema enables both simple and detailed descriptions of educational content (items 3 and 5). Moreover, metadata with a list of possible values potentially promotes similar descriptions of educational content with slight differences in the number of segments (items 5 and 9). Finally, EduTVATool is flexible for marking up audiovisual content (items 1, 4, 7, and 8) and provides help support to users (items 2 and 6).

5.2. Functionality Assessment

Table 6. Search results using Crayon when marking up the educational content with both TV-Anytime and EduTVA metadata schemas.

Search Query	ER *	TV-Anytime			EduTVA
		OR ${}^{†}$	IR ${}^{‡}$	OPC ${}^{\mathbf{\S }}$	OR ${}^{†}$	IR ${}^{‡}$	OPC ${}^{\mathbf{\S }}$
S1	29	26	3	0.896	28	1	0.965
S2	25	0	25	0	24	1	0.960
S3	1	0	1	0	1	0	1
S4	6	7	1	0.833	8	2	0.666
S5	6	0	6	0	6	0	1
S6	1	1	0	1	1	0	1
S7	19	19	0	1	19	0	1
S8	8	0	8	0	8	0	1
S9	6	0	6	0	6	0	1
S10	11	8	3	0.727	11	0	1

^* Expected Results (ER) from Table 5. ^† Obtained Results (OR). ^‡ Inconsistent Results (IR). ^§ OPerational Consistency (OPC).

shows the results when executing the search queries on Crayon for looking for educational content (see Table 3) that has been described using both TV-Anytime and EduTVA metadata schemas. This table includes the identifier of the search query (see Table 4), the number of Expected Results (ER) per search query (i.e., see Table 5), the number of Obtained Results (OR) from the search query, the number of Inconsistent Results (IR), and the value of the OPC metric (see Table 2). IR refers to the number of obtained results that were not expected plus the number of expected results that were not obtained.

The results show a poor performance of Crayon when looking for the educational content described with TV-Anytime. The search queries S2, S3, S5, S8, and S9 obtained no content (OR = 0); therefore, IR = ER and OPC = 0. The search queries S1 and S10 missed three pieces of content each (IR = 3), achieving an OPC of 0.896 and 0.727, respectively. The search query S4 obtained a non-expected piece of content besides all the expected ones (IR = 1), for an OPC of 0.833. Only the two remaining search queries (S6 and S7) correctly obtained the expected results (OPC = 1).

Conversely, the performance of Crayon improved when looking for the educational content described with EduTVA. The search queries S1 and S2 missed only one piece of content each (IR = 1), achieving an OPC of 0.965 and 0.96, respectively. The search query S4 obtained two non-expected pieces of content in addition to all the expected ones (IR = 2), for an OPC of 0.666. The remaining seven search queries (S3, S5, S6, S7, S8, S9, S10) correctly obtained the expected results (OPC = 1). Note that only the OPC of S4 performed worse with EduTVA than with TV-Anytime. For the rest of the search queries, the OPC with EduTVA was either better than or equal to the OPC with TV-Anytime.

To summarize, Crayon correctly obtained the expected results (OPC = 1) from 70% of the search queries with EduTVA, in contrast to 20% of the search queries with TV-Anytime. Moreover, 50% of the search queries with TV-Anytime obtained no content (OPC = 0) while all the search queries with EduTVA obtained expected content ((OPC > 0). Lastly, 80% of the search queries with TV-Anytime obtained inconsistent content (IR > 0; OPC < 1) versus 30% of the search queries with EduTVA. Only the search query S4 performed worse with EduTVA (OPC = 0.666) than with TV-Anytime (OPC = 0.833). Therefore, EduTVA provides a more functional metadata schema than TV-Anytime for implementing searching services of educational content in digital television, such as Crayon.

6. Conclusions and Future Work

Digital television must consider the values and principles of a culture, influence knowledge, and promote the viewer’s attitudes and skills in order to be effective in the learning process. In this paper, we proposed the EduTVA metada schema for enabling a complete description of educational audiovisual content and educational segments of the content in digital television. EduTVA extends the TV-Anytime metadata schema by adding educational metadata (e.g., skills) derived from a mapping between different E-learning metadata schemas, including LOM, MLR, and LRMI. EduTVA adds the following metadata elements: interactivity type, educational resource type, educational use, intended end-user role, typical age range, intended end-user language, educational context, annotation, and ability. Note that reusing an existing television standard provides cost savings, access to support and advice, usability, resource discovery, and sustainability.

We evaluated the acceptance of the educational metadata by surveying a group of users that described educational content with the proposed EduTVA metadata schema. These users interacted with EduTVATool, a metadata authoring tool that we implemented for EduTVA. The responses from the users show that the EduTVA metadata schema facilitates both simple and detailed descriptions of educational content by providing specific elements for educational metadata and supporting the creation of educational segments of the content. We also evaluated the functionality of the EduTVA metadata schema by validating the obtained educational content from a digital television search service, named Crayon, when executing different search queries. The content results from the search queries show that Crayon performs more accurate when looking for educational content described with EduTVA than with TV-Anytime.

Finally, although this paper demonstrated that the EduTVA metadata schema facilitates the description and improves the search service of educational content for digital television, there are still research challenges to be addressed.

• Further metadata elements might enrich the EduTVA metadata schema for establishing relationships between educational resources and reaching formal learning processes in digital television. For example, evaluation, pre/post activity, curriculum, and the optional elements from TV-Anytime.
• Applications with more complexity than a search service, such as recommendation systems and content indexing, might leverage the EduTVA metadata schema for improving their results. For example, the artificial intelligence engine of a recommendation system might use the educational metadata of EduTVA for presenting customized content based on the student’s skills.
• Discovering the relationships between segments and resources might support creating semi-automatic links between elements that do not necessarily belong to the same resource. Moreover, measuring the labeling quality of television educational content is a pending task that can contribute to the marking up process.
• Future versions of EduTVA might leverage standards for KRR to improve the content description. KRR abstracts the user’s way of thinking by representing knowledge in a symbolic way. In addition, EduTVA might use the Knowledge Graph (KG) model to generate semantic links and metadata from the relationships between concepts, entities, and content events. Both knowledge-based solutions could improve data integration, analysis, and exchange, while avoiding run-time support issues.
• EduTVATool, as a Web solution, can be extended to support collaborative marking up, even involving the different domain roles in the value chain of digital television. Moreover, a semi-automatic generation of metadata (manually corroborated) might speed up the current manual process of EduTVATool for marking up content.