Metaverse for Cultural Heritages

Abstract

The metaverse has gained popularity recently in many areas including social media, healthcare, education and manufacturing. This work explores the use of the metaverse concept for cultural heritage applications. The motivation is to develop a systematic approach for the construction of a cultural heritage metaverse and to offer, potentially, more effective solutions for tourism guidance, site maintenance, heritage object conservation, etc. We propose a framework for this cultural heritage metaverse with an emphasis on fundamental elements and on characterization of the mapping between the physical and virtual cultural heritage worlds. Efforts are made to analyze the dimensional structures of the cultural heritage metaverse. Specifically, five different dimensions, linearity, planarity, space, time and context, are discussed to better understand this metaverse. The proposed framework and methodology are novel and can be applied to the digitalization of cultural heritage via its metaverse development. This is followed by a detailed case study to illustrate the tangible procedure, constructing a cultural heritage metaverse with a complex and dynamic nature which can be used for different applications, including heritage conservation.

1. Introduction

1.1. Motivation

Gaining popularity recently, the metaverse is a large-scale three-dimensional networked platform that independently and perennially exists, combining the virtual and physical worlds. This platform spans a 4D space-time continuum, integrates human–computer interaction and generates economic value. The metaverse involves various technologies including Networking and Computing, Internet of Things, Blockchain, Human-computer Interaction, Artificial Intelligence, Virtual & Augmented Reality, Visualization, etc. The cultural heritage metaverse specifically relates to tangible and intangible information regarding cultural heritage which can be used not only for data collection, storing, presentation, sharing and conservation, but also for restoring, monitoring, protecting, researching and inheriting cultural heritage. The definitions of the metaverse, the cultural heritage metaverse and metaverse technology lay the foundation for the framework for the cultural heritage metaverse presented in this paper. Apparently, such a metaverse can also be useful for research and development in archaeology and museology. It is thus imperative to investigate the fundamentals of cultural heritage for its development.

For this purpose, we will examine the metaverse and its digital twin (DT) for its application to support conservation, monitoring, inheritance, research, and development of cultural heritage. The digitalization of cultural heritage using metaverse technology will be able to help in promotion, dissemination, and inheritance of traditional cultural heritage. We will look into the mapping connecting the virtual and physical presence in a time-varying fashion. Such mapping-enabled interaction can enhance the understanding, observation, recognition, monitoring, controlling and transformation of cultural heritage. Cultural heritage can be represented in various forms, with rich historical and cultural connotations. Better understanding of these representational forms will help in a feasible development of a cultural heritage metaverse.

1.2. Objectives and Scope

The objective of this research is to explore the intrinsic features of cultural heritage for the purpose of designing a cultural heritage metaverse. A generic dimensional concept is introduced to classify cultural heritage for development of its metaverse. A framework is proposed to capture the fundamentals for classification and representation of cultural heritage. Specifically, the research is interested in figuring out:

(1)

What is the design dimension of classification and representation for a cultural heritage metaverse?

(2)

What are the key characteristics of cultural heritage to be expressed by this metaverse?

(3)

What new paradigm is desired for the dimensional design of a cultural heritage metaverse?

(4)

What important role can metaverse technology play in supporting cultural heritage?

Cultural heritage can be generally divided into two types: tangible and intangible. The scope of this research includes both types. A literature review covers cultural heritage and its digitalization. This is followed by an investigation on the fundamentals of cultural heritage and on a framework design for the cultural heritage metaverse. A case study is subsequently detailed.

1.3. Challenges

Cultural heritage has different stakeholders, from individuals to organizations, whose interests and responsibilities are significantly different. No doubt, a common digitalization platform, if available, can provide good support for these different stakeholders’ needs. Yet it is not an easy job to understand cultural heritage given its diversity. A good understanding of the fundamentals may be feasible and helpful for the development of a cultural heritage metaverse. Besides the differences in needs or interests from users’ perspectives, cultural heritage can be highly complex in terms of its content, representational forms, and connotations. Typically, the expressions, knowledge, practice and skills related to a specific piece of cultural heritage can be vague, isolated, incomplete and sometimes controversial. We believe that the metaverse, as a digitalization technology, can play a game changing role in helping cultural heritage to enter a new era of service, management and conservation.

1.4. Research Methodology

The main research method used in this work is that of classification and construction. Cultural heritage is classified according to its typical structure and dimensions. This classification method is helpful to better understand the relevant representational forms and connotations for the purpose of developing a cultural heritage metaverse. It is essential to establish a relationship between the virtual and physical presences (or worlds) of the cultural heritage metaverse. In addition, constructionism is used to establish a framework for the design and development of the cultural heritage metaverse.

1.5. Organization of the Paper

The paper consists of five sections. Section 2 is devoted to a literature review of cultural heritage and its digitalization. Section 3 describes digitalization and the cultural heritage metaverse with an emphasis placed on the fundamentals of cultural heritage and the development of a cultural heritage metaverse. Section 4 gives a case study of cultural heritage: Eight Immortals. Section 5 concludes this study, highlighting its contributions, limitations and proposing future work.

2. Literature Review

This literature review covers five parts. Metaverse and DT are first examined from a general perspective. This is followed by the study of prior work on cultural heritage in relation to DT, Building Information Modelling (BIM) and laser scanning.

2.1. Metaverse

Metaverse is in the limelight today since the rename of Facebook as Meta [1]. Microsoft announced its metaverse vision, according to its CEO Satya Nadella [2]: “The metaverse is not just transforming how we see the world. It’s changing how all of us actively participate in it”. NVIDIA [3] has its own metaverse, i.e., Omniverse, built in response to the rapid movement of metaverse research and development. Substantial efforts are being made across the planet, from industry to academia, in relevant research and development.

2.2. Digital Twin

In recent years, the concept of DT has been widely studied with applications found in various areas including engineering, architecture, etc. In 2003, Grieves [4] came up with the concept of virtual digital representation, equivalent to physical products. The National Aeronautics and Space Administration (NASA) [5] described their DT ideas in the Space Technology Roadmap in 2010. The Air Force Research Laboratory [6] discussed a conceptual model for the live prediction of aircraft structures using DT technology in 2011. Maja et al. [7] reported the building of digital twinning on Predix with General Electric. The Centre for Digital Built Britain [8] outlined the principles for guiding development of a “national digital twin”. Zhao [9] illustrated the theory and method of Constructing Interactive Geometry Virtual Twins. Tao et al. [10] shared their DT five-dimensional model characterized by physical entities, virtual models, services, DT data and connections. Tao et al. [11] discussed a Standard System of DT and [12] presented a correlation and comparison of cyber-physical systems (CPS) and DTs for smart manufacturing. Tao et al. [13] also analyzed ten questions relevant to DT. Opoku et al. [14] elaborated the concept-oriented drivers, the production-driven drivers, the operational success drivers and the preservation-driven drivers of DT Adoption in the Construction Industry. Kor et al. [15] investigated the potential integration of deep learning and DTs to facilitate Construction 4.0 through an exploratory analysis. Guray and Burcu [16] provided a significant theoretical reference for the potentials of VR-AR in construction management. Newman et al. [17] mapped the totality of work carried out so far and presented an assessment of Industry 4.0’s progression, potential and degree of uptake within the UK construction industry. Götz et al. [18] explored asset life cycle management use cases, interoperability and integrability enablers of blockchain-based DT and positioned the technological ecosystem within current practice and technological availability. Jwo et al. [19] came up with a concept of DT based deployable service, Data Twin Service (DTS), to support simulation for Industry 4.0. Costantini et al. [20] studied the IoTwins to develop and execute distributed DTs for predictive maintenance purposes. Alizadehsalehi & Yitmen [21] presented a framework for an automated construction progress monitoring system that integrates BIM, various reality capturing technologies, DT and XR technologies (VR, AR and MR), arraying the steps in how these technologies work collaboratively to create, capture, generate, analyze, manage and visualize construction progress data, information and reports. Ogunseiju et al. [22] described a DT framework for improving ergonomic self-management for construction workers.

2.3. Cultural Heritage and Digital Twin

The study of DT for cultural heritage is still in its infant stage. As pointed out by the European Commission [23] in Horizon 2020, digitization of cultural heritage today mainly focuses on capturing the visual appearance of individual objects, collections or sites. There is a real need to establish a holistic comprehension of cultural heritage digitalization to study assets and to capture and re-create not only visual and structural information, but also stories and experiences (via language data), together with their cultural and socio-historical context, as well as their evolution over time. Qin and Zhang [24] proposed the operation mechanism and goals of DT technology in digital development of tangible cultural heritage. Amelio and Zarri [25] looked into a Narrative Knowledge Representation Language (NKRL) to represent specific knowledge and posited an inferential point of view of cultural heritage to make explicit their historic, social and cultural background. They confirmed that NKRL can represent in depth any possible “narrative” environment, for example, the “immaterial” component (including its emotional factors) of the twins associated with cultural heritage’s “iconographic” entities. Marra et al. [26] described a Historical Digital Twin (HDT) paradigm which is conceived as a DT of the main historical, geometric, mechanical, and structural characteristics of a physical piece of cultural heritage. Gabellone [27,28] proposed an online virtual visit of a tangible cultural heritage site and a method for a live-guided tour to explain contextual knowledge related to cultural heritage, using DT and giving a new perspective on cultural heritage management. Fritsch et al. [29] presented a novel workflow for combining voxel representations and colored point clouds to create a much finer DT of physical objects. Zhang et al. [30] proposed an information design based on DT for presentation of the alternation of VR, AR and the real environment. Presently, Cultural Heritage DT is still an area less explored and existing study mostly focus on developing mechanisms in DT for tangible cultural heritage.

2.4. Cultural Heritage and Building Information Modeling

BIM is a well-established field for digitalization of buildings and construction. BIM for heritage, or Heritage BIM (HBIM), has recently gained attention. Darwish and Hassanien [31] suggested that DT is an ideal instrument for monitoring and inferring behavior, deterioration of heritage structures, performance, collection and classification of varied data that can co-exist in the model as an asset for artifact preservation. Jouan and Hallot [32] proposed a Framework for HBIM-based Preventive Conservation of Cultural Heritage. Recognizing the cultural significance of tangible and intangible elements of cultural heritage encoded by DT, and referring to Stephenson’s cultural Values Model, Jouan and Hallot [33] constructed data structuration in a HBIM model of cultural heritage and classified the DT information according to a logic of value. Sang et al. [34] proposed building a digital twin of the Great Wall, including a retrospective twin of the defense system of the Great Wall, twin management of the current situation of the Great Wall and its surrounding environment, and a twin of the future development of the Great Wall. Through sorting and analyzing the tangible and intangible cultural heritage, Sang et al. [35] discussed the construction process, change process and important historical time–space data of the Great Wall for building DT to investigate the potential integration of deep learning and digital twins in order to facilitate Construction 4.0 through an exploratory analysis. Tan et al. [36] proposed a method to merge the digital twin into the chronology of forms to help archaeologists recognize historical information. Dezen-Kempter et al. [37] explained architectural cultural heritage to the audience through a HBIM Model and augmented reality. Bevilacqua et al. [38] analyzed the applications of DT in a case study for tangible, intangible, or no longer existing cultural heritage. Youn et al. [39] developed a HBIM solution of bracket set modelling based on 3D for Korean traditional wooden architecture. Tahmasebinia et al. [40] proposed DT as a monitoring technology for prediction of the structure’s future relating to the Sydney Opera House. Funari et al. [41] presented a parametric Scan-to-FEM procedure of DT, implemented in a visual programming environment, for the generation of historic masonry structures. Selim et al. [42] thought that integrating both human and digital records enriched the virtual twin of cultural heritage sites, as a living and more humane experience that best represents the relevant multi-layered and overlapping history and cultural practices. Sánchez-Sánchez et al. [43] applied 3D modelling and DT for archaeo-seismic damage research at the Roman site of Baelo Claudia. With the application of HBIM, Wang and Wu [44] worked on an efficient solution for online education concerning the risks posed by the ongoing COVID-19 pandemic. In summary, HBIM has been developed to study cultural heritage mostly from the perspective of architectural structure and materials.

2.5. Laser Scanned Cultural Heritage

Laser scanning uses the principle of Light Detection and Ranging (LiDAR) to illuminate a target and then to analyze the reflection. By applying extremely narrow beams, laser scanning is capable of mapping objects to a high degree of resolution with fine details captured at a relatively high speed. The rotational laser of the scanner measures the distance to objects and, together with the device’s rotational angle data, determines its coordinates in space, with integrated. At different measurement speeds and resolution, LiDAR scanners can capture several million points, with their (X, Y, Z) coordinates recorded. Color images are created simultaneously in scanning with the help of the integrated camera. Laser scanning is increasingly used for digitalization of cultural heritage. In the early 2000s, 3D laser scanning was used in the Digital Michelangelo project [45]. A large-scale heritage site was laser scanned for 3D reconstruction by El-Hakim et al. [46]. The archaeological area of Paestum was laser digitized for modelling use by Fiorillo et al. [47].

Building information modelling, digital twinning and laser scanning are increasingly used for digitalization of cultural heritage.

Table 1. Comparison of different methods for heritage digitalization.

Approach	Techniques	Data Format	Advantages	Disadvantages
HBIM	Modeling	IFC 4.0	Compact representation, level of description, suitable for cultural heritage of buildings	Not widely used, not suitable for non-building cultural heritage
Laser scanned cultural heritages	LiDAR laser scanning	Geometry & texture (XYZ & RGB)	Simple representation	Huge amount of Point Cloud data generated
Cultural heritage DT	Mapping	Geometry, texture, & IoT signals	Bi-directional mapping	Data crunching for communication between physical and virtual twins

is a brief comparison of heritage BIM, laser scanned cultural heritage and cultural heritage DT with their advantages and disadvantages highlighted. More importantly, based on prior study, an effort to comprehend the fundamentals of cultural heritage is needed for better digitalization of cultural heritage.

3. Digitalization and Metaverse of Cultural Heritage

3.1. From Virtual-Reality Continuum and Digital Twin to Metaverse

In the special issue on VR for Culture and Heritage of the Journal Presence [48], Zhang & Yang [49] discussed the “Hyper-Presence” of Cultural Heritage in Shaping Collective Memory. Cai et al. [50] described a solution for cultural heritage application based on a virtual-reality continuum (VRC) which spans a whole spectrum from the physical to the virtual world. A case study on the digitalization of Madam Snake White was presented leveraging Virtual Reality, Augmented Reality, and 3D Printing technologies. Apparently, the virtual world and physical world are two extremes within the continuum. Digital twinning, however, focuses on the mapping between the physical twin and the virtual twin. From the application perspective, VRC and DT show synergy when dealing with the physical and virtual presence of cultural heritage. Taking into consideration the time-varying nature of cultural heritage, the metaverse concept, extended from DT and VRC, is introduced in this work for the digitalization of cultural heritage. Figure 1 illustrates the cultural heritage metaverse with a time-varying nature, combining VRC and DT.

3.2. A Framework for the Cultural Heritage Metaverse,

Cultural heritage traditionally has its presence in the physical world delivering stories, values, and histories to users visiting a physical site. Understanding the fundamentals of cultural heritage can, thus, potentially better the design of the cultural heritage metaverse. As a result, the cultural heritage metaverse may help in conservation of the cultural heritage, as well as in improvement of user experience.

Cultural heritage as an entirety is comprised of forms, connotations and time-varying information. These can be represented in different forms, shapes, structures and expressions. Cultural heritage contains connotations and rich knowledge, including historical knowledge, cultural knowledge, content knowledge and scientific and technological knowledge. Evolutionary information can be inherited in a time-varying manner, which may change its form and enriched connotational knowledge. Identifying the fundamentals of cultural heritage is thus essential for the framework design of a cultural heritage metaverse. Good metaverse design may potentially offer cost-effective solutions for cultural heritage conservation and protection. It may enhance user experience and establish sustainable interest through interactions between visitors and cultural heritage.

The framework for a cultural heritage metaverse consists of four major elements: characters, instruments, medium and story (Figure 2). Characters can be human, animal, or other forms of life. While Mona Lisa is a character, Kung Fu Panda and Virus can be characters too. Instruments are typically tools used by characters (e.g., violin, calculator, etc.). A medium can be the surroundings or environment (e.g., water, air, etc.). One or more characters work together to develop a story with or without instruments at specific places and at a given period of time. Generally time-varying, cultural heritage stories deliver messages or values rooted in certain cultures, science or religions. The stories can keep growing with changes of characters, instruments, mediums, and with extended locations. Generation by generation, cultural heritage grows over time. Together, characters, instruments, medium and story within their framework communicate knowledge of history, culture, technology, etc. Characterized by expression and forms, connotations and time-variances, the cultural heritage metaverse can be constructed within the framework proposed.

3.3. Mapping between Physical and Virtual Presence of Culture Heritage

Physical cultural heritage consists of tangible entities passing down from generation to generation, such as folk music, paintings, sculptures, operas, and Atlantic City. However, these physical entities could be partially or completely damaged or destroyed over time. Based on the framework proposed, the cultural heritage metaverse can be established by mapping the physical presence (PP) to the virtual presence (VP). Figure 3 shows a Merlion, constructed and based on a popular myth. The virtual Merlion is a digital copy of the physical Merlion. The mapping between the physical Merlion and the virtual Merlion enables the digitalization, using various technologies such as laser scanning and 3D modelling. In particular, the virtual merlion has interactive features implemented, allowing visitors to interact with it via hand gestures. A metaverse of the Merlion can be developed with the Merlion serving as a character and telling a story of Singapore from past to present.

3.4. Representational Forms and 5-Dimensional Cultural Heritage Metaverse

There are different forms available to represent various types of cultural heritages Typical forms of cultural heritage include folk music, paintings, and sculptures. Folk music can be considered as a type of cultural heritage in sequential or 1-dimensional (1D) form. Paintings (oil, watercolor, Chinese or western) are another form of cultural heritage, normally two-dimensional (2D). Sculptures are a 3-dimensional (3D) form of cultural heritage.

In Section 3.2, characters, instruments, medium and stories are discussed as four major elements within the framework of a cultural heritage metaverse. Characters, instruments, and mediums are typically static, not dynamic presences, within most physical cultural heritage. Cultural heritages contain stories to demonstrate time-varying information, including history and values. The physical presence of many forms of cultural heritage, however, is limited to static forms like paintings, sculptures, etc. A metaverse for cultural heritage, if developed, can demonstrate a virtual presence of the time-varying information using animation, video and other techniques to tell its stories.

Figure 4 shows the classification of the cultural heritage metaverse in five dimensions. Metaverse-1D is for forms of cultural heritage characterized by 1D linear (sequential) form. Metaverse-2D is for forms of cultural heritages characterized by 2D planar form. Metaverse-3D is for types of cultural heritage characterized by 3D solid form. Metaverse-4D is for forms of cultural heritage characterized by 4D time-varying form. Metaverse-5D is for types of cultural heritage characterized by 5D domain-specific applications.

Table 2. The 5-dimensions of cultural heritage metaverse.

# Dimension	Perspective	Metaverse
1-Dimensional	Linear or sequential perspective	Metaverse-1D: sequential form (physical & virtual), such as music notes, song scripts, folk stories, etc.
2-Dimensional	Still planar perspective	Metaverse-2D: still flat form (physical & virtual), such as paintings, drawings, figures, sketches, pictures, patterns, textures, etc.
3-Dimensional	Static 3D shape perspective	Metaverse-3D: static solid shape or environment (physical & virtual), such as sculptures, buildings, museums, antiques, ruins, etc.
4-Dimensional	Time-varying information	Metaverse-4D: time-varying solid shape or environment (physical & virtual), such as opera performance, kinematic installation art, live ancient city, etc.
5-Dimensional	Application context	Metaverse-5D: application domain specific cultural heritages (physical & virtual), such as history, culture, religion, philosophy, technique, etc.

details these five dimensions of cultural heritage. These five dimensions are divided into two categories: noumenon and state of existing. Noumenon includes sequential symbols (metaverse-1D), planar still image (metaverse-2D) and static solid shape (metaverse-3D); State of existing includes time varying information (metaverse-4D) and application context (metaverse-5D). Metaverse-1D/2D/3D represent the geometrical shape information of cultural heritage, metaverse-4D represents the time varying information of cultural heritages and metaverse-5D represents the connotations of cultural heritage.

3.5. Cultural Heritage Metaverse to Enhance User Experience and Conservation

Once a cultural heritage metaverse is developed, users can have their experience enhanced when visiting with the aid of interactive virtual and augmented reality tools. Using the same example, i.e., Merlion cultural heritage (Figure 3), visitors can interact with the virtual 3D Merlion (the metaverse-3D) through their hand gestures while listening to the narration of the Merlion history (metaverse-1D). The ideology and the aesthetics of the Merlion sculptor can help visitors to better understand the motivation, background and significance of this sculptural creation. The Merlion metaverse-5D can improve the user experience and appreciation of contexts and contents classified by history, culture, science, etc., through storytelling with the aid of digital video, animation, audio, etc.

Apart from the enhancement of visiting experience, the metaverse constructed can offer a new avenue for cultural heritage protection, site maintenance, heritage object conservation, monitoring, etc. It will help in promotion, dissemination and inheritance of traditional cultural heritages. The culture heritage metaverse can also be useful for research and development in archaeology, and museology.

4. Eight Immortals Metaverse—A Case Study

4.1. The Cultural Heritage of Eight Immortals

Haw Par Villa (Singapore), also known as Tiger Balm Gardens, was built in 1937 by Mr. Aw Boon Haw (Tiger) and Mr. Aw Boon Par (Leopard), as a venue for teaching traditional values. In the 1970s and 1980s, the villa became a popular site for day trips and school excursions, with a trail of the world’s only eclectic Chinese mythological park of its kind. Among over 1000 tasteful scenes in the villa, Eight Immortals is a group of sculptures designed to promote Taoism.

4.2. Eight Immortals Metaverse—The Four Major Elements

Stories: Eight Immortals Crossing the Sea (Eight Immortals, in short) is a legend of Taoism about a war between Eight Immortals and a Dragon King. While the Eight Immortals represent justice, the Dragon King followed by his army of shrimp soldiers and crab generals represents evil. These immortals cross the eastern sea with the aid of their magic powers, performing miracles with their treasures. The stories reflect the Taoist culture, with an emphasis on the spirit of justice defeating evil. The well-known legend is still widely circulated today through a saying: “Like Eight Immortals crossing the sea, each revealing divine powers”.

Characters: The Eight Immortals are Li Tieguai, Han Zhongli, Zhang Guolao, Lv Dongbin, He Xiangu, Lan Caihe, Han Xiangzi and Cao Guojiu (Figure 5). The Dragon King of the eastern sea and its army are presented as the counterparts of the Eight Immortals. The Dragon King is a God of rain in charge of the weather control. The Dragon King is followed by an army of anthropomorphic monsters including Shrimp Soldiers, Crab Generals, Carp Monsters, Toad Monsters and Turtle Monsters (Figure 6).

Instruments: Eight Immortals each hold one or more instruments to support their magic powers (

Table 3. Instruments of the Eight Immortals.

Immortal	Instrument
He Xiangu	Lotus flower
Cao Guojiu	A pair of jade plates
Li Tieguai	Iron Stick and Calabash bottle
Lan Caihe	Flower basket
Lu Dongbin	Sword
Han Xiangzi	Jade flute
Zhang Guolao	Donkey (also a character)
Zhongli Quan	Fan

). Eight Immortals Crossing the Sea was first recorded in the ancient poetic drama Eight Immortals Crossing the Sea in Competition for Jade Plate, which was performed through traditional artistic forms such as opera singing, spoken parts and dancing. It became a popular Miscellaneous Drama with profound literary and artistic offshoots. Its flexible and diverse form of performance reflects not only the cultural background and aesthetics, but also the leisure interests of the public at that time. The instruments play their unique roles in the dramatic performance. For Dragon King and his army, these anthropomorphic characters are equipped with various weapons, typically swords and knifes (Figure 6).

Medium: Obviously, all Eight Immortals cultural have the sea as their major medium. Yet different forms of cultural heritage may have their own designs of this medium. For instance, the group sculptures Eight Immortals Crossing the Sea in Haw Par Villa are designed based on the Miscellaneous Drama. As such, the environment, including the costumes and scenes from the performance stage of this cultural phenomenon, are created differently.

4.3. Eight Immortals Metaverse—The Five-Dimensions

Singapore’s theme park Haw Par Villa has a cultural heritage site, Eight Immortals Crossing the Sea, in the form of sculptures. As part of Taoism, there are many Eight Immortals-related works presented in different cultural styles. In the following, a metaverse for Eight Immortals is discussed from a dimensional perspective.

Metaverse-1D for Eight Immortals (sequential): Eight Immortals Crossing the Sea appeared first as sung opera. The related musical notes and song scripts are sequentially arranged in one direction to present the story of the war between the Eight Immortals and Dragon King. Han Xiangzi, one of the Eight Immortals, creates sequential musical information when playing the jade flute.

Metaverse-2D for Eight Immortals (still planar): There are a large number of Eight Immortals-related cultural artefacts in still, planar form including paintings, drawings, embroideries, photographs, etc. All these forms a metaverse-2D. Figure 7 is an example of a painting of the Eight Immortals as part of this metaverse-2D.

Metaverse-3D for Eight Immortals (static 3D shape): Similar to still planar cultural forms, there are many Eight Immortals sculptures, statues, etc., available in static solid form. This form of cultural heritage includes group sculptures, theme parks, museums, ruin heritage sites, and so on. Metaverse-3D should have all this included. Figure 8 shows the group sculpture of the Eight Immortals in the Haw Par Villa.

Metaverse-4D for Eight Immortals (time-varying): Unfortunately, static solid sculpture is not able to display time-varying information. For instance, dramatic plays based on the Eight Immortals are typically performed with the characters presenting their stories with different instruments and in different media. VRC solutions can be developed to create a virtual presence of the physical cultural form to overcome the non-time-varying limitation of some of the static solid forms. Storytelling can be used in VR, AR and MR to have time-varying features implemented. Metaverse-4D for Eight Immortals should include these time-varying forms.

Metaverse-5D for Eight Immortals (application context): Generation by generation, cultural representations of the Eight Immortals in linear, planar, solid, and time-varying forms have been developed. This Taoist legend has a long history evolving over various dynasties (Han, Tang, Song, Ming, etc.). It is important to highlight here that the development of the Eight Immortals culturally is sure to continue over the years. Figure 9 shows six of the Eight Immortals developed by the team for this work based on hip-hop culture. Metaverse-5D for Eight Immortals is basically a fusion of all the contexts for Eight Immortals Crossing the Sea.

4.4. Eight Immortals Metaverse–Mapping using Laser Scanning for Digitalization

A LiDAR scanner FARO is used in this work to capture the sculpture of the Eight Immortals and its surroundings in Haw Par Villa. The rotational laser of the scanner measures the distance to objects and, together with the device’s two rotational angle data, determines its coordinates in space, with integrated GPS. At different measurement speed and resolution, FARO can capture several million points with their (X, Y, Z) coordinates recorded. Color scans are created simultaneously during scanning with the help of the integrated camera. A high accuracy scan (~2 mm) requires about 30–40 min while a low one (5–10 mm) requires 5–10 min.

Figure 10 shows the point cloud data and one of the corresponding pictures from the Eight Immortals laser scanned in the Haw Par Villa.

Multi-scan Stitching: Point cloud registration or stitching is an essential task, with multiple scans captured of the Eight Immortals sculptures. Basically, registration is used to estimate a rigid transformation for aligning the two scans, which can be done in two tasks: global registration and local registration. Global registration is to obtain a good initial alignment for arbitrary input poses (scans) of the same environment. Local registration is to refine the correspondences between the two globally registered scans and find the optimal transformations between them.

Cleaning & Repairing: After stitching, the point cloud may retain some features (Figure 8) such as nonuniform sampling, noise, outliers, misalignment, missing data, etc. It is thus required to preprocess the point cloud captured by cleaning and repairing algorithms based on the positions and color information of the point cloud data. After cleaning, the point cloud will also have some holes (parts missed) or ununiform density in some areas. In this work, segmentation is performed using Cloud Compare before position repairing based on a feature-preserving point consolidation method.

Level of Details (LOD): Real-time rendering of large point clouds is often required in VRC, especially on personal mobile devices (e.g., for AR application). To accelerate this process, simplification of the point cloud is desired with acceptable quality of rendering. LOD technique is implemented in this work to achieve real-time rendering for different VRC and metaverse applications.

VRC and Metaverse Development: Figure 11 summarizes the entire process of metaverse building for the Eight Immortals cultural heritage, from laser scanning to VRC modeling and visualization to metaverse development. This process is important for digitalization through the multi-dimensional approach developed in this work (Figure 12).

Revitalization of Cultural Heritage using Metaverse Solutions: Metaverse technology is making rapid advancements recently. The Eight Mortals metaverse can help promote, protect and preserve this form of cultural heritage. With the Eight Immortals metaverse developed, the ability to exhibit and monitor this piece of cultural heritage can be enhanced. A younger generation can learn about traditional culture and values through interactive and immersive play with the Eight Immortals metaverse.

5. Conclusions

This paper proposes a fundamental framework which is significant for the development of a cultural heritage metaverse. A new multi-dimensional paradigm is developed for digitalization of cultural heritage characterized in five-dimensional forms. Connotation and time-varying features are discussed along with forms for better understanding of cultural heritage. Basic elements for understanding of cultural heritage are also elaborated upon for the purpose to metaverse design, with an emphasis on the noumenon and stat of existing for classification purposes. A case study of the Eight Immortals at the Haw Par Villa in Singapore is detailed. Efforts are made in the application of this metaverse for conservation and protection of cultural heritage. The proposed approach is generic, which is important for all cultural heritage in terms of metaverse construction. The study has a potential impact on the digitalization of cultural heritage.

There is still room to improve on this work. For instance, it is interesting to investigate the similarities and differences in metaverse development between tangible and intangible forms of cultural heritage. Gamification and Serious Games [51,52] can be incorporated into the cultural heritage metaverse. Deeper investigation is expected as part of the future work on the applications of the metaverse for conservation and protection of cultural heritage. The effectiveness and efficiency of use of this metaverse technology for education in traditional culture and values with a younger generation in Singapore and Guangdong is a topic of interest for further research. Future work should include the use of IoT [53] and edge/fog computing [54] for cultural heritage study.