CAR-Tourist: An Integrity-Preserved Collaborative Augmented Reality Framework-Tourism as a Use-Case

Abstract

The unprecedented growth in Augmented Reality (AR) has captured the focus of researchers and the industrial sector. The development of AR applications and their implementation in various domains is broadening. One of the advancements in the field of AR is Collaborative AR, which provides ample opportunities for the members of a team to work on a particular project remotely. The various activities carried out remotely, in a collaborative fashion, are based on the active interaction and transmission of data and applications across a communication channel that constitutes a mesh of frequently interacting applications, thus providing a real feeling of working together physically in the purportedly same demographic area. However, in the integration of different roles, remotely working in collaborative AR has a great chance of being intruded upon and manipulated. Consequently, the intrusion may explore novel vulnerabilities to various sensitive collaborative projects. One of the security concerns for collaborative and interconnected remote applications is to have pristine environments, where the participants of the collaborative AR can reliably trust each other during the execution of the various processes. This paper presents an integrity-aware CAR-Tourist (Collaborative Augmented reality for Tourism) framework wherein the unauthorized user’s access is denied and the remote participants of the network are provided with a secure environment through the state-of-the-art Blockchain architecture. This study further provides a use-case implementation of a tourism application. Each tourist has the chance to hire a remote guide for collaborative guidance over a blockchain-trusted network. Moreover, the proposed framework is lightweight, as the only necessary communication between the tourist and guide is recorded in the blockchain network. Each user has to register on a permission blockchain to be allowed to perform certain activities on our proposed CAR-Tourist framework. The decentralized Blockchain approach provides a consensus mechanism based on which not every participant is free to intrude on ongoing communication. Thus, through the proposed framework, all the participants in the collaborative Augmented Reality will have the essential trust of working remotely without external intrusion.

1. Introduction

Industry 4.0’s arsenal accompanies one of the most promising technologies, namely Augmented Reality (AR). It is one of the modern technologies in the current generation that has yielded a rising market for AR application development. The extensive advancements in AR, Virtual Reality (VR), and collaborative AR have considerably changed experiences in different areas of interest. The implications and uses of these technologies are broad and are being actively used to boost the level of experience of clients. In the last decade, the interest of industries, as well as academia in AR, has risen. It was estimated in research in 2009 that mobile-based Augmented Reality (MAR) will reach 732 million dollars by 2014. Also in May 2014, Facebook acquired Oculus for 2 billion dollars, and after that, the AR/VR industry started to rise. During recent years, many AR applications have been launched in the tourism sector and, in addition, the popularity of mobile-based AR games, such as Pokémon Go, has increased [1]. However, in 2015 and onward, AR rose rapidly due to advancements in technology and overall infrastructure. The popularity of AR and VR further increased due to the development of Flash-based AR, AR-based and their utilization in the global marketing campaigns [2].

As later discussed in this paper, AR has a greater impact on the tourism sector. During the COVID-19 pandemic, this sector suffered the most. As this sector contributes massively to the economy of any country, it was the need of the hour to revive this sector. Moreover, in historically rich countries like Saudi Arabia, tourism plays a major role in the country’s economy. To revive and develop the experience of the tourists to a whole new level, AR is used. With the use of AR-based tourism applications, tourists do not need to gather around to hire a guide. They can be guided through this application remotely with the help of virtual objects on the screen as well as a remote guide.

Figure 1 explains the working mechanism of this collaborative AR application and how different tools can be used to develop such kinds of applications. It also explains applications of AR in different fields such as medicine, industry and tourism. It further demonstrates the security of this system, how the data of the users are safe and uncompromisable and how an environment of trust is created among the users.

The rise of smartphones is one of the important factors for the wide adoption of AR and VR [3]. The mobile phone AR experiences were released many years earlier, but these were run on phones that had low graphic power, and low-speed processors, and were difficult to develop. The launch of the iPhone enabled smartphones to develop AR experiences more easily because iPhone had powerful 3D graphics and the fast processing system was enough for real-time computer vision tracking. Similarly, the android platform become the perfect outdoor AR platform, because of its inertial compass sensors, well-developed graphics and with services of Global Positioning System (GPS). In Japan, 66% of consumers demanded AR in off-line stores [4]. In the present era, a widespread adoption of AR and VR technologies is found in health care departments [5,6], education sectors [7,8], manufacturing and logistics [9,10], smart cities [11,12], Museum [13,14], social media [15], etc. In retail, AR-enabled products are visualized in 3D for consumers [16,17] that ultimately leads to customer satisfaction, customer relation, and maximum sales. Figure 2 presents the overview of AR applications in Medical Training, Retail, Entertainment, public safety, etc. The Figure also presents details of various AR development tools [18,19] including ARCore, ARkit, Kudan etc .

The recently emerged Collaborative AR approach further satiates individuals to simultaneously coordinate with each other in the real world surrounding them and a virtual world. Collaborative AR allows multi-users to work in a collaborative manner which is purported to be a physical interaction. The immersing collaborative AR is driving innovation by inspiring many of the aforementioned domains. In collaborative AR leverage, remote participants interact with each other and present a real feeling of existence to one another. Thus, the interacting participants can present, orient, supervise, and monitor each other in a variety of areas including medicine, surgeries, education, retail, marketing, entertainment, and many more sectors [20,21]. In tourism, tourist guides can remotely guide tourists. Similarly, the employer of an organization can supervise and meet with employees within an organization.

This study make use of Tracking Technologies that contribute as building blocks in AR. These technologies create a point of reference for movement and for creating an environment wherein virtual and real objects are mixed together. To achieve a real experience with the augmented objects, several tracking technologies exist, including Sensor-based, markerless, marker-based, and hybrid tracking technologies. This study makes use of vision-based tracking (SLAM) that considers different points taken from camera and adds virtual contents to inertial tracking. After establishing the points of reference during tracking, another step is to have significant accuracy to determine at which particular point the virtual objects have to be mixed to combine virtual objects with real scenes. Display technologies contribute by determining the particular points where the users of augmented reality can have an environment wherein the real and virtual objects are displayed visually. Augmented reality displays can be divided into various categories. Optical see-through has two sub-categories; one is a free-space combiner and the other is a wave-guide combiner. Other categories of AR display include video-based, eye multiplexed, and projected onto a physical surface. Authoring tools are utilized for the creation of highly interactive AR experiences. Available authoring tools include Wizard, Quest3D, and Unity. This study considers AR foundation in Unity to create Colloborative AR. Collaborative AR has two further classifications, namely, Remote Collaboration and Colocated Collaborative AR, which considers same geographical area during collaboration. This study opted for Remote Collaboration in the tourism sector.

1.1. Problem Statement

Collaborative AR, besides facilitating remote interaction, introduces novel vulnerabilities that can expand the surface of cyber attacks over the activities of participation in collaborative AR applications. Ultimately, the technical professionals in security and risk management are assessing the risks and are in dire need of implementing robust security controls over the AR-deploying applications. This is particularly the case in collaborative domains where security-centric infrastructure is required [22,23,24,25]. Collaborative AR is mostly used for remote training. A manager of the company can guide another person remotely from anywhere to manage a problem by using collaborative AR, but in this scenario, both users know each other. While in the use case that is built during this research, most often the tourist is not known to the guide. How can a user trust an unknown person to share data such as the real-time video of the camera and location?

1.2. Motivation

In historically rich countries like Saudi Arabia, tourism plays a major role in the country’s economy. During the COVID-19 pandemic, this sector suffered the most. With the use of AR-based tourism applications, tourists do not need to gather around to hire a guide. However, secure collaboration and communication has always remained a challenge to address.

1.3. Contribution

Security-centric organizations and other sectors, making use of collaborative applications and online interactions, require integrity of the communication and avoid intrusion by non-authorized users or external attackers. This paper proposes a complete security framework wherein different applications working remotely can have a real feeling of trust in each other while no unauthorized transaction is permitted by the consensus mechanism. This study built a framework where two unknown users, a guide and a tourist, can communicate and share data in real-time video and location with each other. In this framework, a blockchain network is integrated on top of collaborative AR and these are inter-operable with each other. The tourist installs the CAR-Tourist application and likewise subscribes to the blockchain network. After subscription, tourists can see the virtual objects on their mobile screens. If the tourist’s credit card is valid and has a certain balance, the visitor can hire a guide to seek information throughout the tour about the history of the visited landmarks. This network is generic, which means the consensus algorithm is pluggable. If another advanced algorithm is launched, it can be utilized in this network. The same is the case with smart contracts and cryptographic algorithms. This study proposes to use the state-of-the-art decentralized blockchain infrastructure to protect communication throughout the collaborative AR. The integrity of the communication is monitored in blockchain through a consensus mechanism, while no nodes within the network can alter the proceedings of the communication without the consent and agreed-upon rules.

1.4. Composition of the Study

The rest of the paper is organized as follows. Section 2 elaborates on the background of the research. Section 1.2 further explains the reason why this application is built and what benefits the tourism sector. Section 3 and Section 5 provide the details of the proposed methodology and results of the study. Section 4 discusses how different mechanisms work and why we chose this specific mechanism called Hyper-ledger fabric for our AR application, while Section 6 concludes the research paper.

2. Literature

This section provides a detailed review of the literature about AR, VR, and collaborative AR. The research work reviewed includes recent publications available on the web of science during the recent past as follows:

Viyanon et al. [26] focused on the development of an AR app known as ‘AR Furniture’ for providing the experience of visualizing the design and decoration to the customers. The customers fit the pieces of furniture in their rooms and were able to decide their experience. This prototype application presents the landmark for the other apps to be developed based on three important factors for designing and tracking AR applications. The evaluation of the user satisfaction with the AR experience, and image analysis with marker and markerless tracking suggested that 3D objects meet the user satisfaction but they faced problems in refreshing the app whenever the problem occurred. Therefore, more rigorous research is required to improve the app. Fang et al. [27] focused on two main aspects of mobile AR. First, a combination of the inertial sensor, 6-DoF motion tracking based on sensor-fusion, and monocular camera for the realization of mobile AR in real time. Secondly, to balance the latency and jitter phenomenon, an adaptive filter design is introduced. This enhances the smooth and real-time 6-DoF motion tracking and alleviates the jitter at the time of visual-inertial fusion. The evaluation of the proposed system suggests its validity and robustness. Roberto et al. [28] proposes a methodology to evaluate depth perception and motion tracking capabilities by using Tango Tablet as a case study. Methodology of Chessboard patterns and graph paper was used as a tool. Depth sensing had two issues: scanned the images with low infrared reflectivity and average errors were compatible with that obtained from desktop cameras. For large and small indoor scenarios, motion tracking errors were around 6 and 14 centimetres. Akccayir et al. [29] This paper systematically reviews the use of AR technology in the educational realm. As AR, can be employed on mobile phones and computers, therefore it is flourishing in the educational field as well. The review suggested the success of AR in education, as it helps teachers and students, but papers had some conflicting views as well. For instance, some papers suggest that AR reduces cognitive loads and others say that it increases the load. These conflicting views could be because of the different experiences of users with different applications; however, AR has the advantage of placing the virtual image on a real object in real time. Therefore, pedagogical and technical issues should be addressed to make the technology more reliable. Boonbrahm et al. [30] aims to develop a design model for remote collaboration. The research introduces the multiple marker technique to develop a very stable that allows users from different locations to collaborate. The multiple marker technique also improves the accuracy of marker-based tracking. This result of the study suggests that the program could be used in real collaboration and other areas like manufacturing, maintenance, and planning. Grandi et al. [31] targets the development of design approaches for collaborative manipulation tasks. The paper aims to develop a design in which a group can easily resolve complex manipulation tasks. For this, purpose fundamental concepts of design interface, human collaboration, and manipulation are discussed. This research the spiral model of research methodology which involves the development, planning, analysis, and evaluation.

AR experiences, at this time, are more evident at the museums or theme parks or in the research laboratory. The technology required high-end computing, tracking hardware, and specialized display to provide a better user experience. For instance, the use of AR technology for showing virtual artefacts in the museum was explored by the ARCO project [32]. At the 2005 America’s Cup, 250,000 people saw the “HIT Lab NZ’s Black Magic AR kiosk”, and the Virtual Showcase project showed that the projection tables could be used to develop high-end AR improved common experiences [33].

AR applications use sensors like camera sensors to sense the environment and then render the virtual objects in that environment. The application gains access to the sensor’s data such as video, audio, location etc. AR applications collect data more than it needs. For example, it doesn’t need audio from the mobile, yet it has access. The application exposes this private data resulting in the creation of a privacy challenge [34]. AR has brought a revolution in different fields and has a lot of benefits. But along with these benefits, some major security and privacy issues also arise. For example, if a user is driving a car with an AR-enabled windshield identify the mark lanes to avoid accidental lane drift. Without proper security, this app can be breached, causing high risks. A buggy or malicious AR app can obscure real-world objects [35]. Pierdicca et al. [36] discusses the privacy and security of AR applications by using AR smart glasses. The proposed solution is linking the cloud-based architecture and the oil extractor along with the combination of Vuzix Blade smart glasses. The results showed that it acts like a guide for a person wearing these glasses. Ahn et al. [37] focuses on the making of policies for AR devices to generate visual output by using deep learning. To protect the system from malicious and disastrous content, simulations are used to automatically understand the situation and create policies. Langfinger et al. [38] focuses on the security and privacy issues regarding AR in industrial sectors. In this paper, the industrial aspect of the AR systems and those applications which required to be secured are being studied. It also discusses the solution to detail the design for security measures for AR devices. Wazir et al. [39] introduces an authentication tool which ensures the privacy and security of AR devices. To achieve this goal, graphical user passwords and AR environments are fused. This authentication tool is considered one of the best to secure AR environments. Zhang et al. [40] focuses on the study of an Android application called “Google Translator” to avoid privacy leaks. This paper also proposes the framework for the detection of unnecessary privacy leaks. Mcpherson et al. [41] discusses the security issues related to AR on the web. It also studies the three AR-based web browsers i.e., Layar, Junaio, and Wikitude that are being operated in more than 300 million smartphones. Lebeck et al. [42] uses an AR headset called Microsoft Hololens for conducting semi-structured interviews for AR immersive headset. The results show to findings. First, Microsoft Hololens has some limitations but the users still find the virtual objects real. Second, multiple security issues were identified while managing the virtual objects.

This study presents a novel idea of security of collaborative AR and issues regarding it which are explained in this section, no one has mentioned a trustful environment where the users can collaborate without any security issues. In such applications, if any business-related transaction is required, it needs to be under a trustful and secure network. To achieve this goal, we used a decentralized blockchain network to secure this collaboration. The tourists can share their private data including mobile location and live camera videos using the blockchain.

3. Methodology and Proposed Architecture

This section presents the methodology of the study about the implementation of collaborative AR through the blockchain infrastructure. The blockchain combines various techniques and presents a novel architecture comprising five elements namely, distribution, encryption, immutability, tokenization, and decentralization. The participants within the blockchain infrastructure are remotely connected to the network having access to a shared ledger. The ledger gets updated with the occurrence of transactions through a mutual consensus mechanism while the ledger cannot be altered except by agreed circumstances. Blockchain uses standard cryptographic and encryption-decryption technologies such as public and private keys to record the data in the blocks securely. Participants within the network control their identity and other information and share only what they need during various transactions. All the completed transactions are signed, time-stamped, and sequentially added to the shared ledger. Within the ledger, transactions and records cannot be corrupted or otherwise changed unless the participants agree on the need to do so. Such an agreement is known as a fork. Since the blockchain is a decentralized network that means no specific node or server is controlling the flow of execution and confirmation of various transactions. All the nodes within the blockchain retain the shared public ledger monitored through a consensus mechanism. This consensus-based approach avoids the requirements of a central controlling authority and it helps against the single point of failure and other fraud if committed. Through this approach, all the communication among the participants of blockchain gets safe and thus the participants are provided to interact with integrity in a digital environment remotely [43,44].

Utilizing the security infrastructure of the blockchain, this study provides a trusted network for tourists and guides. The decentralized network blockchain facilitates various communication concerning latency and enhances the security of communication. The study had choices to adopt among public and private blockchains; however, the public blockchains have latency and suffer from performance issues. Moreover, permission blockchain is more efficient for collaborative AR. It helps in the ease of selecting a consensus algorithm and further creates a smart contract to provide a platform for interaction between tourists and guides.

3.1. Working of Blockchain-Registration of Tourists and Guides

In the proposed architecture, the study outlined four participants which are tourists, guides, blockchain network and CAR module. The tourist and guide interact through the CAR application. The blockchain network is used to ensure trust between the users. All the transactions are captured in the blockchain network to make the interaction trustful. The important transactions of our CAR-Tourist module are formally defined below:

Definition 1.

(Tourist’s request for a guide:) Tourists T can have a guide G request if T and G have a valid registration over Blockchain Network $BN$, and T have a valid status and credit card. Similarly, G should have an eligible status.

Definition 2.

(Blockchain network $BN$ should monitor T’s status to transfer $Tn$ service charges): If $Tn$ cannot be performed T should be disabled. On Low balance, $LB$, Warning W should be generated.

Definition 3.

(Dispute Resolution): Blockchain network $BN$ should verify the Transaction $Tr$ if the Dispute D occurs. D should be generated either from T or G. Dispute D should explain T’s perspective if G did not provide the right service. Or G violates the privacy of T as per the defined rules. Similarly, G can have the dispute D if G didn’t get the specified amount from T.

Definition 4.

(Miscellaneous additional Transaction over BN):

• T can initiate a chat with G without a contract C. Blockchain network $BN$ will only record the request and response.
• T and G can have a voice call without C. $BN$ will only record transaction $Tr$ for record purposes which may be used for dispute resolution.
• T and G can share the $T1$ screen with multiple tourists $Ti$ if and only if $T1$ allows.
• G has given the option by $BN$ to write on V’s screen with CAR module to explain the landmark-specific spot asked by T for the explanation.

According to this study’s use-case requirement, the proposed integrity-aware Collaborative AR (CAR) module will register tourists and individuals with knowledge (known as travel guides) within the Kingdom of Saudi Arabia (KSA). Upon successful registration on the smart contract, the travel guides and visitors will become trusted due to the consensus and underlying trusted network. Moreover, since the consensus and other related nodes on top of the smart contract will be distributed around the KSA, the decentralized peer nodes all over the country can efficiently work from anywhere within the country. The smart contract is used for the Tourists so they can observe the nearby available guides to get their services.

The blockchain configured for this study is composed of essential nodes including the membership nodes, certification authority, orderer nodes, endorsement peers and consensus nodes. All of these nodes are set up on lightweight containers and placed in various locations in the country. After setting up the nodes, a consensus algorithm was opted for. As per present blockchain consensus technologies, the study has to opt among A, RAFT, PBFT, etc., consensus mechanisms. However, observing the effective usage results of the RAFT consensus algorithm and its voting-based approach, this study considered it more appropriate and efficient for the presented use case.

Any registered traveller during visits to different sacred landmarks could have the opportunity to view registered available guides over the blockchain infrastructure and can initiate the CAR module. The proposed framework of the designed integrated blockchain and CAR module can make the guides available to a different blockchain network. This means two blockchain networks can communicate with each other, i.e., one AR-enabled blockchain and another without AR. As such the visitor will not be required to register on the other Blockchain and will just skip the registration process; however, the visitor will need the CAR module to enable guide help. The complete process of requesting a guide and the recording of that transaction is shown in Algorithm 1. The tourist requests to hire a guide for guidance throughout the visitor’s journey. To send this request, the tourist must first be registered to the blockchain network. After registration, a wallet is created for the tourist. If a tourist’s credit card has a certain amount of balance, is registered to the network, and the wallet is created, then the tourist’s request is put forward to be further processed by the endorser, orderer, and consensus nodes. The Endorser node verifies it according to the rules defined. The Orderer node adds this transaction in a block and then adds it to the chain. Then, it is passed to the Raft consensus mechanism for further data verification, to check whether the data is compromised or not. If any of the requirement is missing, then the request is denied. This request is saved as a transaction in the database of the blockchain network for the record.

Algorithm 1 Tourist Request for Guide

Procedure: TouristGuideRequest $TR=TouristRegister\leftarrow Touristshouldberegistered$ $TW=TouristWallet\leftarrow True\leftarrow Touristishavinganactivewalletandregistered$ $TC=TouristCreditCard\leftarrow True\leftarrow Touristisvalidandhasbalance$ $IfVC==empty$ $requestdenied$ $(TW,TC,TS==True)then$ $InitiateRequestforGuide$ $ExecuteRequest\left(\right)$ $Endorsement\left(\right);Ordering\left(\right);Consensus\left(\right);$ $elseRequestDenied$

3.2. Working of Consensus Algorithm

Every transaction is first-endorsed through an endorsement peer. Thus, all transactions generated, or a registration request is sent among the Tourist and guides via the blockchain-enabled CAR module and endorsed through the endorsement peer. After endorsement, the orderer node makes a hash and adds it to the blockchain. In the RAFT consensus algorithm, a leader has to be elected, so, if there are 10 nodes then all of these will vote to elect a leader. When every transaction is created the orderer node hands over the transaction to the leader to distribute it to all other nodes. The benefit of the consensus algorithm is that everyone has one ledger so if one node is compromised, it does not affect the whole network while the compromised node can be replaced with another node, and its ledger can be updated. Thus, the whole process will bring trust to the Tourists to share their live locations and screen travel guides so they can enjoy tourism and be informed about different events.

3.3. CAR-Tourist Module

CAR-Tourist provides information on the landmarks for the users, imposed on the real environment by using AR technology. The user can see detailed information about the landmarks through this app. CAR-Tourist provides two main collaborative ways for CAR-Guide to work collaboratively. The first one is the camera tracking of the Tourist/user to track the position of the place and provide the correct information, and the second one is the location of the user to identify the visited landmark. After the selection of the guide, there is a chat option, voice option, and video option.

3.3.1. Smart Contract Interface

In the interface of the CAR-Tourist app, the user can see multiple guides, reviews and feedback given by other users about the particular guide, the qualification of the guide and most essentially what languages the guide can communicate in. The user can also select the list of guides based on the languages they speak. The rate of the guide per hour is also displayed in the interface and the user can negotiate the price with the guide. The average hourly rate of the guide is displayed in his profile according to his previous record with different users.

3.3.2. Language support

Usually, a user from another country cannot speak and understand Arabic; on the other hand, if the guide cannot speak English, then the user can select a language in the app to filter out the list of guides and show only those guides which can speak the desired language of the user.

3.3.3. Tourist Blockchain Registration Process

To use the CAR-Tourist app, the user must register himself on the blockchain. This registration will be a one-time registration. After registration, the user will now be a member of the blockchain and will be able to see the list of guides it can hire to guide him throughout the journey of the landmarks. The list will consist of many guides with different reviews from different users due to which the user can decide to hire the best guide. If the guide is available online, then it will provide services otherwise the user will go for another guide. The use of blockchain here is to secure the data of the user and the guide as well. The data of the guide and the hashes of transactions are stored in the couch database of the blockchain.

To connect the user with the guide, again the smart contract application is used. The user will send a request to the guide for communication. This request is saved in the blockchain as a transaction; after the creation of this transaction, the endorsement peer will endorse the request as to whether the user can connect with the guide or not. For example, if the user does not have sufficient money in his account to hire a guide then the communication cannot be established. After the endorsement of the request the user will now be able to communicate with the guide for help and can share his screen as well as location. The user can request a video call, audio call or text message as well. Blockchain will save the requests and other data of the user for security.

In Figure 3, it is demonstrated how a tourist initiates AR application in the first step, while in step 2 of Figure Figure 3, Blockchain is initiated to record transactions related to sign up, record audio call, proceed with payment/subscription, etc. In Step 3, when transaction is established, AR is initiated that enables visitor to receive guidance from Travel Guide. The app creates a map of the real environment. The created map and the vision provides a SLAM. i.e., SLAM provides an overall vision and details of different objects in the real world. Inertial tracking controls the movement and direction in the augmented reality application. The virtual objects are then placed after identifying vision and tracking. In a collaborative environment, the guides are provided with an option of annotation that can circle a particular object or spot different locations and landmarks or to point to different incidents.

3.4. CAR-Guide Module

The expert or scholar of the landmark who can provide guidance, the study denotes it as a CAR-Guide module. The CAR-Guide module is used to guide the tourists. The guide can interact with the tourists through this module. The guide can see the shared location and screen of the users. The guide must register himself on the blockchain. After registering himself as a guide, the profile of the guide will be created in which the reviews of the users and the number of users guided will also be displayed. The data of the guide is also saved in the blockchain to avoid any insecurities. The guide also goes through the same process of endorsement to register himself on the blockchain. The complete process of registering the guide via smart contract is explained in Algorithm  2.

Algorithm 2 Guide’s Access to Tourist View

Procedure: Guide View Tourist $GW=GuideWallet\leftarrow Guideisregistered$ $GS=GuideSession\leftarrow True\leftarrow Guidesessioniscurrentlyactive$ $GDS=GuideDisputeStatus\leftarrow False\leftarrow Guidehasgenuinehistory$ $(GW,GS==true,GDS==false)then$ $InitiateaccesstoTouris{t}^{\prime }sView$ $ExecuteRequest\left(\right)$ $Endorsement\left(\right);Ordering\left(\right);Consensus\left(\right);$ $elseAccessDenied$

In this algorithm, the procedure of the guide’s access to the tourist’s view is explained. If the hired guide has registered himself on the blockchain network and has a wallet then he is available to the tourist. Also, the guide’s session must be active at the particular time. The guide must have a clear record without any disputes over payments or privacy breaching complaints in the past. If these requirements are fulfilled, then the guide can access the real-time video of the tourist to guide the visitor properly during the tour and also edit his video by adding or removing annotations on his screen. This transaction is also passed in the same process as the tourist’s request for a guide explained in the first algorithm. If any of this data is missing, the request will be denied.

3.5. Marker-Less Collaborative AR

The main objective of the CAR-Tourist app is to give AR experiences to tourists. Whenever the tourist needs assessment they can use the camera to receive guidance from the guide remotely. The guide receives the live video transmission from the user and the guide can add virtual objects and annotations to describe the landmarks. If the tourist needs guidance to reach a particular spot within the boundary of the visited landmark, the guide can add annotations to help the tourist reach the exact location. After the addition of different virtual objects and annotations. The final view is visible to the tourist through the CAR-Tourist app. The app consists of camera tracking, annotation editing, and the rendering of virtual objects in the real environment. Google ARCore is used to track the position of objects with the help of smartphones. This study used Unity SDK to render the virtual objects in the real environment. After rendering these virtual objects, the guide can modify their position and orientation. The guide can also add annotations in the real-virtual video transmission between the guide and the user. These annotations help the user to better understand and identify the landmarks. The guide can also add other virtual objects or remove them from the screen. As the CAR-Tourist app is a location-based AR application, it uses a camera sensor, gyroscope, and magnetometer to create a local map of the landmark and locate the tourist on that map. This map becomes global when shared with the guide for editing and modifying. If multiple tourists are present at one place, then one guide can assist multiple tourists at once by making a group of tourists visit together. All of them will receive the same information and details by using their own mobile devices, as depicted in Figure 3. This helps the guide to facilitate multiple tourists, saving more time.

CAR-Tourist app based on markerless collaborative AR in other words location-based collaborative AR. In this AR system, the magnetometer is used to detect the orientation of the mobile device. The camera of the device is used to capture the live feed on which the virtual elements are imposed. Magnetometer enables the mobile to always know which direction is north. It is the key element of the proposed framework i.e., location-based AR app CAR-Tourist. GPS (Global Positioning System) is used to provide the correct Geolocation concerning time. The proposed framework’s tracking technique is Inside-Out tracking, in which the camera and other important sensors are built into a device like a smartphone. The most important reason to use Google ARCore in our app is to provide motion tracking with the help of SLAM (Simultaneous Location and Mapping). SLAM gathers the data from the camera and other sensors like magnetometer, gyroscope, and light sensor to understand the real environment and correctly render the virtual objects in the real world [45].

The general idea of SLAM is to build a map of an unknown real environment. This technique is widely used in robots and other AR-based systems. The building of a map and locating itself in it is the purpose of using SLAM. In our CAR-Tourist app, SLAM is used to build a map of the visited landmark so that the AR experiences are rendered accurately. Tourists’ accurate location is dependent on two inputs. The first one is the data on the previous location of the user and the recent movements that occurred. The second dependency is on readings from the camera sensors of the mobile device. SLAM combines these data and readings through general probability distribution as presented below:

$$P(Sx,L|A0-x,H0-x,S0)$$

(1)

$$S0=Startingpointoftheuser$$

(2)

$$A0-x=Setallobservationsofthelandmarks$$

(3)

$$H0-x=Historyofthereadingsofchanginglocationonthemap$$

(4)

$$L=Alllandmarklocations$$

(5)

3.6. Case Study: Tourism

The use case of visiting the Kingdom of Saudi Arabia in the sacred places is elaborated on in this subsection. Figure 3 depicts the proposed study of how the basic components of the blockchain and various interactions among the remote participants take place. A pilgrim during a visit to various sacred places may intend to get particular details about the events that took place during the life span of Prophet Mohammad (SAW). The details such as where Hazrat Hamza R.A was Martyred may be something appealing for the tourists. Such details are normally not known to incoming tourists unless they inquire. A guide can help to identify places and provide such responses with details. Through the use of AR, the tourist at his standing place can use the smart device camera and the application where the available online guide may facilitate the tourist in identifying the appropriate places of events. The identity of the online available guides is mandatory so that the tourist may be provided with actual details. Moreover, since the application and the data of the users are online communicated, it is more likely that the guide may misuse the data of the users. However, since the blockchain infrastructure is proposed, first the online trusted guides are identified over the blockchain, then the identity is checked through the blockchain and an online guide is allocated to the tourist which provides all the required information through AR. The tourist is informed to put the mobile or smart device camera towards particular places and then the online available guide provides augmented details to the tourist on his display screen. All the interaction among the blockchain, the online guide, and the smart devices is carried out through RestFul APIs. Figure 4 presents the proposed solution of the paper. It further elaborates how the tourist uses smart devices to interact with blockchain for the provision of information required to the tourist. It can be seen how the different networks represent the online available guides interacting with the blockchain. This study setups blockchain essential nodes for the execution of various processes and endorsement of the various transactions. The implementation details of setting up the blockchain and establishing a connection with the smart devices are not provided here to avoid complexity; however, they are explained in Figure 4.

4. Implementation

This section provides implementation details of the proposed solution by first providing details of the implementation of the proposed application in Hyper-ledger Fabric, including details of its components and the reason for choosing Hyper-ledger Fabric (other platforms are also available, details of which are provided in the following paragraphs). Later, the study presents the results of the performance analysis of Hyper-ledger Fabric implementation.

IBM Blockchain: A permission blockchain network ensures tighter control over access, which is one of the main reasons for using this platform for business applications. However, there are some issues reported on this platform, which are as follows: there is no support for configuring an external certificate authority (CA), the IBM Kibana interface by default shows the logs of the past 30 days’ activities, chaincode container hangs sometimes, and there are issues related to chaincode installation [46].

R3 Corda: R3 Corda [47] is a distributed ledger platform designed for managing financial records by the company R3. The main purpose of Corda is to replace software used for financial transactions with a shared trusted ledger. Corda was designed to resolve issues such as privacy and scalability of blockchain applications; however, some large companies have observed implementation constraints and thus it is not suitable for our use case.

HydraChain: Hydrachain blockchain platform [48] supports the development of the permission distributed ledger. This technology is extended from Ethereum; therefore, the tools for developing smart contracts and decentralized apps can be exploited; however, it also inherits issues from Ethereum.

Quorum: Quorum [49] is an extension from the Ethereum code base to provide a permission blockchain platform that supports confidentiality and privacy for all transactions using the smart contract. Authors in [49], studied the performance evaluation of Quorum smart contracts and various transactions in terms of latency, throughput, and their effects on blockchain application design. Quorum has lower throughput at higher load and experimental results suggest an increase in transaction latency concerning reading and writing operations.

IOTA: IOTA [50] is a blockchain platform designed specifically for IoT blockchain applications using DAG (Directed Acyclic Graph). In [51], authors have evaluated the performance of various blockchain platforms wherein it is suggested IOTA applications are scalable because of the DAG structure of transactions.

BigChainDB and MultiChain: BigchainDB [51] is a scalable decentralized database used by various blockchain applications including IOTA. For example, BigchainDB is used to create a product database for IoT devices for blockchain applications to maintain the historical record of provenance and ownership of devices. Similarly, another blockchain platform called Multichain [52] was initially developed to maintain ownership and transfer of assets on blockchain applications. Some developers have noticed high latency and response time while testing multichain with Azure VM.

Hyperledger Fabric [53]: provides an open-source platform to implement permission blockchain applications. It is the most used and appealing permission blockchain platform and offers various features, such as an open-source permission blockchain platform, the capability of integration with other platforms (IOTA), generalized, adaptable by different industries, consensus finality, high throughput, and effective smart contracts.

These prominent features motivate us to select Hyper-ledger Fabric for the implementation of our proposed Collaborative AR module. In the following subsections, different components of the Hyper-ledger are elaborated upon along with other blockchain components in terms of tourist guides and the Collaborative AR proposed module. All the transactions are implemented over the smart contract. The hyper-ledger components and practical implementation of this module are explained below in terms of how each transaction can ensure trust among the users.

4.1. Setup of Hyper-Ledger Components

For the proposed system, many Hyper-ledger components are configured, which include orderers, anchors, endorsers, committers, and MSP, i.e., Membership Service Provider for endorsing and carrying out a consensus over transactions, as can be seen in Figure 5. As it is explained earlier, multiple transactions are made for the implementation such as registration, hiring a guide, and allowing collaboration among different tourists and guides. All these transactions are implemented as record-keeping for the blockchain network. All the heavy load actual frames are bypassed by the blockchain but to bring trust among the user. Blockchain is implemented in this module. The role of an orderer node is to deal with incoming transactions and create blocks containing real data that are later added to the shared ledger. The role of various components in the implementation of Hyper-ledger Fabric is discussed in the following subsection, which is also illustrated in Figure 5.

4.1.1. Channel

In the proposed system, channels are created to manage communication among the organizational members. Channels incorporate smart contracts or chaincode, various members of organizations, and orderer and endorsement nodes that are created via different credentials and assigned through MSP.

4.1.2. Smart Contract

The proposed system makes use of an application for the execution of transactions on the blockchain, known as the chaincode or the smart contract. These could be third-party applications connected to blockchain networks through the Restful interface or these applications can be tightly coupled within chaincode. Tourists and guides can interact through the smart contract. The tourist can see the list of guides and their profile. The tourist can also pay for the guide through this system. The sharing of tourists’ real-time videos with the guide and with other tourists is possible due to smart contracts because it provides an interactive interface to the tourists as well as the guide.

4.1.3. Anchor Peer

The peers within the blockchain network are synchronized through the configuration of an anchor peer. The anchor peer gets to know the overall details of all the available peers in the network and, similarly, the peer nodes get details of other peer nodes through the anchor peer.

4.1.4. Endorsement Policy

An endorsement policy is implemented to ensure that before adding details of transactions into the ledger, the details are first signed by predetermined peers. Indeed, through endorsement policy, chaincode is executed for validating the transaction by peers. Different organizations have numerous or more than one orderer. While launching different transactions from the smart contract, the multiple interface interaction is ensured through RestAPI. The endorser node defines certain policies. Thus, with its help, the framework bound the tourist and the guide about the transaction execution. If the transaction is against the endorsement policies, it will not be executed. For example, if the guide is sharing the video of the tourist without seeking permission, the endorser node will not allow the guide to do so and will keep the privacy of the tourist protected.

4.1.5. Certificate Authority

To ensure secure communication among the nodes of the network, establish ownership in the network, and avoid breakdown, a Certificate Authority (CA) is configured. CA supplies administration access for adding peers to the network generates digital certificates to ensure trust, and issues cryptographic public and private keys for communication. The keys are used by peer nodes to encrypt and decrypt messages for maintaining confidentiality and integrity. Each organization is implemented with a separate certificate authority. When the tourist and the guide want to register for the application, a transaction is generated and sent to the membership service provider. It verifies the data of the user and provides a digital certificate from the certification authority. The user also receives a key pair to encrypt the communication between the tourist and guide for a trustful environment.

4.1.6. Validation of System Chaincode

The validation system chaincode is configured so that the Validated System Chain-Code (VSCC) checks three things upon arrival of the transaction to peer. These are 1. Initiating the source of the transaction, 2. Checking if the transaction was signed by predetermined peers and 3. Checking whether the endorsement is sufficient for the transaction or not.

4.1.7. Consensus

The proposed system configures a consensus mechanism to ensure all the blocks added to and removed from the ledger are being supported by nodes of the network. Many consensus mechanisms were opted for and tested among RAFT, Kafka, and Solo. All the transactions made by either the guide or tourist in this application are first received by the orderer node where they are packed in a block. This block of the transaction is then sent to the consensus mechanism.

4.1.8. Practical Byzantine Fault Tolerance (PBFT) and Kafka

BFT provides a practical Byzantine state machine replication that run even if the malicious nodes are running in the system. Nodes in BFT are divided and sequentially ordered to one primary node (the leader) and secondary nodes (backup nodes). Each secondary node could become the primary one in the case of a primary node failure. This aims to give a consensus about the state of system to all honest (correct) nodes through the majority rule. PBFT can function with a condition that the correct nodes have to be greater than 3/2 nodes, which means less than 1/3 fault nodes are allowed. As the number of nodes increases, the system becomes more secure. The PBFT algorithm is apt to Sybil attacks, where one entity (node) controls many identities. As nodes in the network increase, there will be few sybil attacks. However, PBFT is not efficient with big networks. Moreover, PBFT does not scale well since in each step it communicates with all nodes. Moreover, as nodes increase in the network, response time is decreased according to the request.

The Kafka Consensus in blockchain infrastructure also ensures agreement of nodes in terms of order of transactions. Ordering nodes are sent to Kafka transactions, and received from Kafka transactions in the same order, Kafka allows abstraction of a shared queue while all orderers can create blocks when they read enough data from Kafka.

4.1.9. Raft Consensus Algorithm

In the implementation scenario, this study used the default consensus mechanism of Hyper-ledger Fabric which is Raft. Raft works by keeping a replicated log. This log is an append-only data structure where new entries are added, and only a single server, the leader, is responsible for managing this log. Every write request is sent to the leader node, and this node will distribute it to the follower nodes and make sure the client receives a confirmation for this write just when the data is safely stored. The raft has a network of nodes that functions to update the blockchain ledger.

Raft is selected because it is fault-tolerant in the case of a node failure, and efficient compared to Kafka and Solo. In addition, Raft conducts an election among the nodes for electing a leader in the blockchain network. Once Raft opts for the leader, the endorser sends the block to the leader which ensures data state synchronization among members of the blockchain consortium. Ultimately, the ledger state is updated, synchronized, and trusted. In addition, Raft is easy to understand compared with other algorithms. It is relatively easy to implement compared to other alternatives, primarily the Paxos. It can decompose into smaller sub problems that can be tackled relatively independently for better understanding, implementation, and optimizing performance. It supports fault tolerance.

In the proposed collaborated augmented reality solution, when the tourist makes a transaction to initiate connection with the guide or when the guide initiates a transaction, then this information is updated on the ledger through Raft that confirms the transactions are trusted and ultimately the stakehoders receive a trusted network that makes the proposed solution a viable solution to impose security in collaborated augmented reality.

5. Results

To measure and evaluate the prototype implementation concerning latency and throughput of the proposed system, a testbed environment was created. In the testbed setup, the nodes of the blockchain are configured over 2 vCPUs, having a RAM size of 8 GB and a processing speed of an Intel Xeon server. Moreover, appropriate network bandwidth is acquired as well. This study makes use of a fabric-load-gen tool, developed by IBM chaincode, with different test cases, latencies and endorsements. The load-generating scripts configure a mix of write or read transactions as chaincode. The throughput is measured through transactions processed per second $TPS$ and the latency of the framework is measured through the completion of the end-to-end transaction including the latency of the endorsement, the latency of the broadcasting transaction to the orderer node, and the latency associated with commit and validate. Figure 6 depicts Hyper-ledger performance over different block sizes, the latencies and throughput of the transactions for different transaction arrival rates and the effect of various endorsement policies with various transaction arrival rates. The transactions could be the sign up operation, hiring a guide, initiation of an audio call, etc. It is observed that throughput increased linearly with the increase in transaction arrival rate, as expected until it flattened out at around 150 tps. It can also be observed that latency increases when the arrival rate of the transaction becomes 150 tps and it receives a sudden increase and remains stable thereafter. During the validation of the policy, three operations are found that are CPU intensive. The first is de-serialization of identity (i.e., x.509 certificates), the second is the identity validation with organization MSP, and the last is the signature verification over transaction data. It is also observed that the bytes of the blocks increase with the number of endorsements due to the encoding of x.509 certificates with every transaction. Concerning comparing the results of this study with the literature, this study did not find any identical solution in the literature and, hence, a comparison is not presented. It is further elaborated that the study did not implement earlier versions of Hyper-ledger Fabric and Ethereum. Very powerful and computationally efficient hardware are utilized, i.e., the resources (RAM and the Processors) used are three times higher than the resources utilized in our system. It is evident by observing Figure 6 that by utilizing the same computation resources, our proposed system can achieve better throughput for transaction execution. Moreover, this study configured Hyper-ledger Fabric 1.4 which employs Raft consensus. Raft produces more acceptable performance results compared to other versions of the Hyper-ledger Fabric. These results not only support our ideas of blockchain-based collaborative AR but will also encourage blockchain application in other walks of life.

6. Conclusions

Recently, collaborative AR has gained tremendous acceptance in the majority of the areas requiring a remote presence. The strengths of collaborative AR are utilized by surgeons, trainers, engineers, retailers, customers, and many more. Remote collaboration via AR, on one hand, provides an opportunity of working together physically in the same demographic area but on the other hand, it is more prone to intruders and miscreants. The intruders can hijack communications and may pose security threats. To provide protection and integrity to the collaborating applications and essential resources, this study proposes the use of state-of-the-art blockchain infrastructure, wherein each of the participants within the networks is provided with a pristine environment and surety that the communications are safe, while the collaborating applications maintain the integrity of shared resources. This study presents a more secure, trusted, and available online collaborative framework that can provide online guidance to visitors throughout the world appraising visitors on authentic details of various landmarks. The framework is called CAR-Tourist. A use case of collaborative AR is presented for one of the cities of the Kingdom of Saudi Arabia, which has a highly rich historical background. The components of the framework are integrated into pluggable architecture while the results show that the performance of the proposed framework is lightweight.