Ontology and Software Tools for the Formalization of the Visualisation of Cultural Heritage Knowledge Graphs

Abstract

Over the last decade, several projects have been developed to digitise and semanticise cultural heritage data. They have been developed to preserve and maintain this heritage, but also to make it accessible to all types of users and to other sectors such as tourism and education. These developments combine the use of knowledge graphs and interactive visualisation tools with web technologies. Although remarkably interesting projects have been developed, the data visualisation tools in these projects tend to focus on the project context. Consequently, it is difficult to reuse the results of these projects. In addition, there are characteristics of cultural heritage information, such as uncertainty and spatial and temporal granularity, that have not been considered, and how to deal with them has not been described. The same is true for other aspects, such as the relationships between different objects. Considering these problems, this study presents a model that formalises how to visualise this information. The design of an ontology that implements this model, based on other works such as VUMO or VISO, is described. Furthermore, the design and development of a software framework that allows the visualisation of this information through a web application are presented. The evaluation of the application of this framework in projects such as SILKNOW or Arxiu Valencià del Disseny is outlined.

1. Introduction

The importance of protecting and preserving cultural heritage and highlighting the value it deserves is recognised by society at large. In Europe, it is also an asset that fosters the development of a digital economy [1]. The digitisation of museum objects is one way to preserve this content and make it accessible to people and other organisations. European institutions are aware of this issue. At the end of 2011, they issued the Recommendation on Presenting, Digitising and Facilitating Online Access to Cultural Heritage Data [2]. This recommendation promotes initiatives such as Europeana Collections [3]. It also provides more than 70 million euros in funding for six programs within the Horizon 2020 Societal Challenges between 2014 and 2020.

However, much work remains to be done. In 2020, NEMO (the Network of European Museum Organisations) published a report showing that European museums have digitised only 43% of their collections [4]. However, the last decade has witnessed the emergence of many cultural heritage digitisation projects. These projects use knowledge graphs to support the information [5,6,7]. These works offer access to content through web interfaces for the general public and through web services for external applications. In addition, because of the volume of data, they often provide interactive visualisation tools to present it to their users.

Many of these display systems use innovative interfaces to graphically represent information and facilitate user interaction. The spatiotemporal representation of information is one of the main axes of this type of project. Traditionally, this information has been visualised using two-dimensional maps. However, systems in three dimensions [8], two-and-a-half dimensions [9] and even mixed solutions [7] have been used. Screenshots of the results of these projects are shown in Figure 1. Many of these projects are not open source, and the solutions used to visualise the information in these works are usually very project-oriented. Therefore, it is difficult to reuse the system in another data domain. Additionally, two characteristics related to cultural heritage data that have received limited attention in research are uncertainty and granularity variation, both in space and time.

Uncertainty or vagueness refers to the fact that, in the field of cultural heritage, it is common not to know exactly where or when an object was created, who was its creator, etc. Thus, depending on the specialist cataloguing the item, different time and space intervals can be assigned. Moreover, these time intervals, or, in the case of space and geographical references, may vary between different domains. Information about an ancient textile retrieved from the Terrassa Textile Museum [10] is an example of these problems:

• Cronology: Origin: Qing Period (1644–1911 d.C.)/XIX Century
• Origin: Orient/China/France/Holland

Uncertainty is a well-known problem in cultural heritage, and its modelling has been discussed [11] and measured [12]; however, the different ways of visualising this feature and dealing with it in a graphical representation have only recently been explored [13,14].

Another problem is the presence of information with different spatial and temporal granularities. This is because the same information uses different reference units. In the previous example, at the temporal level, a one-time interval defined by years covers three centuries (1644–1911), and another covers one century (19th century). An example at the spatial level could be a reference to an object in France and another in Paris: where is an object in France graphically located? In the middle of the nation? In its capital? Users must be aware of this. However, this information is usually not available. In most studies, the object is displayed without informing the user that there are other possible locations and that the one displayed is not necessarily the correct one [13].

Another aspect that is usually absent or very limited is the relationship between represented objects. Although the objects can have the same materials, techniques and authorship, this similarity is not usually shown graphically. Only a few projects, such as SEMAP [7], which shows textual information, or Jewish Culture, which connects them with lines, show these relationships. In some projects, the semantic distance between objects has been used [15] or the term frequency [16] in the object description to connect with lines those objects that are less than a distance from the central object. In the INDIANA Mas project [15], ring structures were used to graphically represent the number of objects found in archaeological excavations over different periods. These structures make it easy to identify similar excavations, which is essential for recognising patterns and comparing information resulting from the same query.

In most of the projects under study [7,13,17,18], visualisation tools allow the application of temporary filters by century, year, etc., but they do not allow simultaneous viewing of query data at different times. This tool makes it possible to graphically compare disaggregated information at different times and thus analyse the temporal evolution of the data.

This paper proposes the argument that many of the problems identified could be solved, or at least reduced, by formalising the graphical representation of the information through a model implemented with an ontology and by using a software framework capable of representing the information by processing the content of the ontology. Logically, in addition to considering different ways of dealing with the problems mentioned, the model must also consider different visualisation scenarios and be able to adapt to any domain of cultural heritage data. It could also change the way information is visualised by simply defining the instances of the ontology appropriately.

In the next section, previous research on the formalisation and visualisation of the content of knowledge graphs through an ontology is presented. In the Section 3, a model for the visualisation of cultural heritage data is detailed based on this work and the study of data visualisation systems, taking into account the characteristics and problems mentioned above. This section also describes the design of an ontology that represents the model, as well as a software architecture that allows the graphical representation of the contents of a knowledge graph by processing this ontology.

In the Section 4, the application of this tool is shown in two projects with cultural heritage data supported by a knowledge graph, SILKNOW [9] and Arxiu Valencià del Disseny [10]. Finally, conclusions and objectives to be achieved are presented.

2. Previous Work

The first studies and developments [19] were aimed at representing a three-dimensional graphical scene using an ontology. The problem with these works is that they are closely linked to APIs such as Java 3D, and these types of ontologies are scene representation formats. In other words, they were not designed to represent information from a knowledge graph. They were designed for an application to process the ontology with scene information, following the structure and philosophy of visualisation technology and representing it.

These problems are not present in other more advanced works, such as the VISO and VUMO visualisation ontologies.

The VISO ontology [20] is a set of ontologies designed to define how the content of a knowledge graph can be visualised. These ontologies, in particular the GRAPHIC ontology, define low-level and high-level graphical structures to formalise the graphical representation of the scene. However, although most concepts are valid at high levels of abstraction, when examining intermediate or lower levels of abstraction, they are defined to represent two-dimensional scenes. Therefore, it is necessary to extend the ontology to consider three-dimensional scenes. This is not a major problem, but a task that must be considered.

In VISO, part of the visualisation definition is performed using the RDFS/OWL Visualisation Language (RVL) [21]. This language oversees the definition of many aspects of visualisation and is based on OWL schemas. It must be embedded in the definition of the properties and concepts of the domain ontology. Therefore, it is necessary to extend the domain ontology and include the required RVL code. In principle, VISO appears to be in line with the objectives of this study, but there are several important issues:

• RVL is complex, with a low level of readability and significant learning costs. This implies the need for experts in this language to define a visualisation system. The philosophy of this work is to take the graph to be visualised, the VISO ontology and the RVL language as a base. Using this information, the scene is represented using a viewer-software tool. This framework may be acceptable in intermediate or simple graphical scenarios. However, if the goal is to implement a visualisation system with advanced interfaces or complex scenes, the number of RVL scripts required will be large. This results in high development and maintenance costs. In addition, the user interface definition in RVL is limited, and there is no way to connect to external APIs.
• The RVL viewer software tool has been developed and is freely available. However, it requires a lot of work to be able to display many elements of a scene with basic complexity.

VUMO [22] is an ontology designed for visualising urban mobility data. It comprises urban mobility concepts that can be mapped to a domain ontology. Once the domain ontology has been mapped, it provides a recommendation system that populates a visualisation ontology that allows the definition of two types of scenes: a geographic heat map and a calendar heat map. VUMO includes a set of graphical concepts that are not as complex as the VISO ontology but allow the definition of most graphical primitives, thus customising some aspects of the visualisation of the scenes to be represented.

The work performed with VUMO is well designed, but there are several problems with using it fully for the purposes of our project:

• Given its specific domain application a priori, it does not have much in common with cultural heritage, and characteristics such as uncertainty or granularity are not considered. However, at its core, it is still an element located in space, and its position and characteristics can change over time. This means that it can be used to represent cultural heritage data, but extensions will have to be designed to take into account granularity, uncertainty and time. On a temporal level, the main difference is that VUMO is very real-time-oriented, so hours and minutes are particularly important in urban traffic problems. However, this information has never been considered in heritage research.
• Graphic customisation is extremely limited and does not include the use of three-dimensional elements.
• No VUMO Viewer software tool has yet been developed.

After analysing these works, it is ruled out that they should be applied directly to the research objectives. However, several interesting items must be considered for the design of the model and the implementation of the ontology that represents it:

• Map the concepts of the knowledge graph to be represented using the appropriate concepts of a visualisation ontology. This concept is used in VUMO and VISO.
• Delegate to an external system the task of visualising the information in the ontology by means of scene types. This idea is used in the VUMO.
• The GRAPHICS ontology developed by VISO represents different graphical possibilities with a high level of depth.

3. Visualisation Model

This section describes a representation model for cultural heritage data visualisation systems. Its content is based on the analysis presented in this paper on visualising cultural heritage data and formalising the visualisation of knowledge graph content information. In addition, this study is supported by a review of state-of-the-art data visualisation [23,24,25,26]. This model is based on the following two concepts:

• Visual objects are graphic elements that can be visually recognised. These elements are related to the scenario or visual scene in which they are represented. Visual objects are further divided into two concepts:

  ○

  Visual Element: a graphical object representing a set of heritage objects belonging to the same class. It is related to the knowledge graph concept that supports domain data.

  ○

  Scene, which represents the global visual content to be rendered. This concept contains defining properties and relationships with all the visual elements within the scene.

• Interaction Actions: This concept formalises the actions that are allowed to be performed in the scene, either to facilitate interaction with the user, as in other applications, or as events in the visualisation system. These processes have consequences for scene representation.

3.1. Visual Elements

A Visual Element is a concept of the visualisation model that corresponds to a concept of the data domain. The Visual Element is composed of Visual Properties corresponding to attributes or relationships of the concept, accessible directly or through queries. Figure 2 shows an example of the conceptualisation of a Visual Element (X) and Visual Property (M), which refers to the material of the object.

The domain data in the example is based on the CIDOC–CRM model [17], which is one of the most commonly used in cultural heritage, but another reference model could be used. To obtain the value of Visual Property M of an instance of the visual element X, a query must be defined to retrieve the value from the domain data. In this example, where <Element_URI> is the URI of the instance of the data domain concept represented by the Visual Element, the query in SPARQL can be expressed as follows:

• SELECT distinct ?materialLabel
• WHERE {
•   ?s rdf:type <http://erlangen-crm.org/current/E12_Production>;
•     <http://erlangen-crm.org/current/P108_has_produced> Element_URI.;
•     <http://erlangen-crm.org/current/P126_employed> ?q.
•   ?q <http://www.w3.org/2004/02/skos/core#prefLabel> ?materialLabel
• }

Owing to the importance of space and time in the proposed model, the existence of a spatial Visual Property and a temporal Visual Property, which are extensions of the Visual Property, are mandatory for referring to the spatiotemporal data of the object.

A Visual Element also has a marker that serves as a reference for the graphical representation of the element. This representation depends on the scene properties. For example, if the scene is three-dimensional, it is a three-dimensional model. In the case of a 2D scene, the graphical representation of the mark is a polygon with a picture and/or colour. The graphical properties can have a default value or be modified depending on the values of the instance properties.

3.2. Scene

The scene defines the representation of the information at a global level. This implies that spatial and temporal reference systems, as well as navigation capabilities, must be defined in the scene. In addition, other properties must be included in the scene, such as the level of detail, method of grouping objects, possibility of filtering data, visualisation of relationships between objects, temporal selection and management of uncertainty representation. These properties are described as follows.

Reference systems. An adequate graphical representation requires the specification of temporal and spatial reference systems, which must be associated with the scene and the data domain objects to be represented. In addition, the model must specify how the spatiotemporal coordinates of the data domain objects are converted into scene coordinates.

Figure 3 shows a scheme explaining how the reference system works in the model. For example, a coin found at a location defined by its latitude and longitude coordinates must be transformed into different scene coordinate systems.

Navigation. How a user navigates is an important element that can be specified depending on the scene. It is necessary to define whether the user will be able to zoom (approach a point in the scene), pan (move in a plane), rotate (around an axis perpendicular to the plane of vision), as well as move to specific areas of the scene.

Detail level. The scene must be able to manage the different levels of detail. This means that depending on how close the user is to a point in the scene, the way the scene is displayed can change to improve visualisation.

Grouping of objects. When a large amount of data is represented in a scene, it is necessary to use data grouping techniques to understand visualisation. Otherwise, the visualisation scene contains a large number of markers, which can result in overlapping point clouds. The model considers the following grouping strategies:

• Grouping around a representative marker. It replaces groups of objects from a region with a representative marker, which is usually located in the centre of the region defined by the group. This process is repeated at all levels of detail, so that at a maximum level X, all the markers represent objects of the dataset, but at a level X − 1, with less detail, the markers could represent not only objects but also a group of objects defined starting from the representative markers of level X.
• Heat maps. The data groups are created in a manner similar to that in the previous form. Scene regions are assigned to each group based on their densities. Each area is assigned a colour, which is usually more intense depending on the amount of data it contains or even changes colour depending on the amount of data. This type of grouping is static. Once the regions are defined and coloured, they do not change, even if the user navigates to a different level of detail.

Figure 4 depicts a diagram representing the two methods of grouping objects referenced in the visualisation model.

Data filtering. Visualisation systems, especially when managing a large dataset, offer data filtering. This allows better visualisation of information with certain characteristics and also improves the performance of the system at the graphical level, as there is less data to display. However, it also forces the development of user interfaces and the global updating of the scene, modifying groups, relationships, etc. as the dataset in the scene changes. The data filtering proposed in the visualisation model has three options for presenting the possible values on which to filter:

• The values are predefined. Each property that can be filtered has a set of associated values.
• The values are obtained from the dataset for display. The user may select one of the different property values in the dataset.

There are no initial values. The user can enter the value to manually filter the data.

Relations. In addition to spatial and temporal location, users may be interested in knowing which objects are related to others based on their property values and visualising this relationship in some way. The following techniques are proposed to show the relationships between the data in the visualisation model:

• Differentiating related objects using a colour or characteristic graphic mark.
• Connecting the objects related to each other by lines or arcs, whose thickness or colour intensity can vary according to the number of related data points if they are groups.

These techniques, as shown schematically in Figure 5, are useful when the dataset is small. If the dataset is large, the identification of objects related to each other can be costly. This difficulty increases when multiple types of relationships are displayed, as an object may be related to several simultaneously, requiring the use of multiple markers. The same is true for related objects outside the displayed scene.

To solve this problem, based on previous works [27,28], a ring structure is proposed, ideally placed around the object or as close to it as possible. This structure can be divided into as many zones as there are properties related to the object. Each zone is filled with a light colour, and depending on the percentage of objects in the scene with the same value, the same percentage of the zone is filled with a darker colour. Figure 5 illustrates a scheme describing these techniques.

Temporary selection. Filtering techniques can be used to visualise objects associated with a given moment or time interval. However, if the user wants to see how the data fluctuates over time, it is necessary to use special controls. Two methods are proposed in the visualisation model:

• A two-dimensional display where each point on the timeline corresponds to a frame and the objects displayed are those related to the selected point on the timeline. This allows the user to view the passage of time as if it were a video.
• Three-dimensional visualisation using a hypercube. The user visualises the data at any point on the time scale. The hypercube can be simple, display data continuously, or contain sections or slices, displaying only the data for each section.

Uncertainty. It is difficult to use a representation to indicate that there is uncertainty in the data without introducing excessive noise into the scene. According to several studies [13,14], one of the techniques that generates less noise in the scene is based on changing the colour of the marker. For this reason, the visualisation model proposes the application of a mask to objects with uncertainty. This mask consists of applying a lighter colour. An uncertain object can be graphically replicated, in which case it will be necessary to indicate that the object is present in the scene with a technique that duplicates the original marker but with a slight offset that indicates to the user that this is the case. Figure 6 shows the results obtained using this technique.

4. Implementation

This section briefly presents the design and implementation of the STEVO ontology and the implementation of the STEVO framework.

4.1. Ontology

The METHONTOLOGY methodology [29] was used to design and implement an ontology that represents the visualisation model. The ontology is called the spatiotemporal visualisation ontology (STEVO) and can be freely downloaded from the public Github repository [30]. In this repository, extensive documentation of the ontology design can be found.

The content of the previous research, the requirements of the visualisation model, and the decision to use the following ontologies were considered in the design and implementation of the ontology:

• VISO GRAPHICS for the specification of graphic objects, with extensions of three-dimensional scenes.
• TIME [31]. A W3C consortium ontology for representing temporal units and durations.
• IGF. Coordinate systems ontology developed by the Institut National de l’Information Géographique et Forestière (IGF) [32].

The main concepts of the STEVO ontology and their relationships are shown in Figure 7. The Scene Concept represents the scene of the model, and the Visual Concept represents a visual element in the model that is related to the Visual Property. These concepts are the main building blocks of STEVO. From this initial structure, the spatial environment of the scene (coordinate systems, boundaries, etc.) can be defined through its relationship with the Space Environment Concept, and the temporal references through its relationship with the Time Environment Concept. The same information could be obtained from Visual Concept and its relation to the Domain Concept, defining how to link this concept with the domain data concept and its time and space references.

To set the spatiotemporal data of a Visual Concept instance, there are four relationships with the Visual Property concept:

-

Time interval (start and end).

-

Spatial reference.

-

Set of properties displayed in the visualisation system.

The Visual Property refers to the properties of the concepts to be displayed. This concept is related to the Query Concept, which defines how to obtain the value of the displayed property in a domain dataset.

The Marker Concept, designed to graphically represent a Visual Concept, is extended by the Figure_Marker and 3D_Model_Marker concepts to define the properties of the Visual Concept representation in two and three graphical dimensions, respectively. The Marker Concept is also related to the Modifier Concept, which defines how to change some properties of the marker, such as colour, size and texture, depending on the instances of the Visual Property associated with the Visual Concept.

4.2. STEVO Framework

Once the ontology is implemented, the next step is to implement an architecture that allows, through a web application, access to the STEVO ontology and the data domain to visualise the data. The architecture of a system supporting this functionality must consider the following elements:

• Hardware and software design to support and provide access to the content of the STEVO ontology.
• Definition and development of modules for accessing knowledge graph content.
• Web support is used to deploy web applications.
• Development of a visualisation component that allows data to be visualised in two and three dimensions on web platforms.

Figure 8 shows a schematic of the proposed architecture. The developments were carried out entirely within the web application, which is based on JavaScript. The visualisation component is developed in WebGL, which allows viewing and interacting with 2D and 3D graphics and works in most of the main browsers and operating systems on the market. The final implementation was performed using the Node.js framework based on JavaScript, the N3.js library for processing the STEVO ontology, and Unity3D [33] for visualisation.

The web application starts with a request that launches a web page in which a Unity3D plugin is embedded. The plugin receives as parameters a link to the instantiated STEVO ontology and the instances to be displayed by means of a URI list file, or the specification of one or more queries that must be executed over the data domain.

While processing the data to be displayed, the plugin creates the necessary data structures on the client side. Once the basic structures are created, they represent the scene visualisation according to the specifications defined by the STEVO ontology and allow the user to interact according to these specifications.

More details on the design and implementation of the ontology, as well as the visualisation tool, are explained in the doctoral thesis of Javier Sevilla [34].

5. Assessment

After the development of the STEVO ontology and software framework, the results were integrated and evaluated in the SILKNOW and L’Arxiu Valencià del Disseny (AVD) projects.

The SILKNOW project (Silk heritage in the Knowledge Society: from punched cards to big data, deep learning and visual/tangible simulations) was funded by the European Union within the research and innovation program with code 769504. The objective is to develop a computer system where the information is supported by a knowledge graph based on the CIDOC–CRM model, designed to improve the current understanding of European cultural heritage related to fabrics made with silk. Among the project’s developments is a web browser where the search results must be displayed on an interactive map, allowing navigation and space-time filters.

The interactive map was developed with the STEVO Software Framework, which processes an instantiation of the STEVO ontology, also available on the STEVO public Github, within the examples section. Figure 9 shows three screenshots of the results obtained using this tool. In the figure on the left, the relationships between the objects are represented by lines, and it can also be seen that some objects have a lighter colour owing to their uncertainty. The screenshot next to the previous one is another screenshot with the same objects but showing the relationships through the ring structure discussed in the visualisation model. Finally, on the right, the image contains a simultaneous display of data at different times.

A video describing the results obtained with STEVO can be viewed at https://youtu.be/GM5cDWv-P0k (accessed on 22 April 2023).

The usability results of the SILKNOW visualisation tool were evaluated using the Usability Scale System (SUS) [35], with a score above 70 when it was not fully developed. In addition, the tool was evaluated at a technical level through stress testing using different computer systems. It showed technical problems only in sets of more than 30,000 data points and computers with the worst characteristics. This is because the tool does not paginate information, which will be fixed in future versions. More details on this evaluation can be found in deliverable D.7.3 of the SILKNOW project, which is publicly available on the project website [36].

L’Arxiu Valencià del Disseny is a centre for the storage, conservation, cataloguing and digitisation of the documentation produced by designers, design studios and the design departments of the most important Valencian companies. These documents can be consulted, studied and made available to researchers in design and contemporary culture.

The processes of creation, production, dissemination, sale and use of objects can be analysed through the original documents. All this with the aim not only of conservation and study but also of making available to the productive and industrial fabric digitalised and related resources through analytical tools and conceptual and product maps capable of promoting innovation and creativity in the industrial sector.

In this project, an interactive map with information about the fabrics, a product map and a relationship map have to be displayed. All these elements must be developed using the STEVO Ontology and the STEVO Software Framework. At the time of writing, the most developed modules were interactive maps, such as SILKNOW and product maps. A Product Map is a tool for design professionals that displays different products in two and three dimensions according to values that can be modified by the user. The aim is to visually identify common aspects of the selected products to define a basis for generating innovative designs. Figure 10 shows a screenshot of the Product Map that is still being developed as a prototype. This tool was evaluated during workshops of the CREALAB project [37] in Poland, Italy and Spain. The evaluation was carried out with 31 university students using the System Usability Scale (SUS), and the results obtained were similar to those obtained in the SILKNOW project.

6. Discussion

The following conclusions were reached after this study was conducted.

An ontology called STEVO has been designed and implemented to represent the previously described model. This ontology is implemented in OWL and is publicly available.

A proposal for the implementation framework designed in this work was developed after analysing the libraries that allow graphical representation in two and three dimensions on a web application. The chosen platform was WebGL, generated with Unity3D, owing to its power, performance and integration in the most used current browsers and operating systems, making the development a multi-platform solution.

The designs and developments carried out were integrated into two cultural heritage projects, where the data were supported by a knowledge graph. In both projects, different visualisations were developed and tested by several users from different countries, with satisfactory outcomes.

The future lines of action with the designed and developed technologies are related to improving the STEVO Framework and the STEVO Ontology:

• The first relates to the STEVO framework. The existing limitations on the size of the dataset must be removed in the framework design and in the developed version. In the currently implemented version, too much responsibility is delegated to the WebGL display component. There is no data paging management or data request based on the zone or the current level of detail. This situation causes memory problems in load tests. The use of Service Worker technology [38], which allows background processes to run in JavaScript, is essential to solving this problem.
• The definition of new temporal and spatial relationships. The STEVO ontology allows the definition of a temporal, visual and spatial reference property in the scene. This is restrictive because the user of the system may be interested in where and when the data were created, but may also be interested in where the data are now and how long they have been in that location. Therefore, it is necessary that a STEVO scene be able to define several temporal and spatial visual properties associated with different behaviours. These improvements will require changes to the STEVO ontology and STEVO framework.
• Improvement of graphical elements and creation of new base scenes to be able to use the framework in more projects. This has a major impact on the STEVO framework.