Virtual Reality and Internet of Things Based Digital Twin for Smart City Cross-Domain Interoperability

Featured Application

Management of Smart City services using Digital Twin concept with the integration of Internet of Things and Virtual Reality. Special focus in interoperability to enable cross-service applications.

Abstract

The fusion of Internet of Things (IoT), Digital Twins, and Virtual Reality (VR) technologies marks a pivotal advancement in urban development, offering new services to citizens and municipalities in urban environments. This integration promises enhanced urban planning, management, and engagement by providing a comprehensive, real-time digital reflection of the city, enriched with immersive experiences and interactive capabilities. It enables smarter decision-making, efficient resource management, and personalized citizen services, transforming the urban landscape into a more sustainable, livable, and responsive environment. The research presented herein focuses on the practical implementation of a DT concept for managing cross-domain smart city services, leveraging VR technology to create a virtual replica of the urban environment and IoT devices. Imperative for cross-domain city services is interoperability, which is crucial not only for the seamless operation of these advanced tools but also for unlocking the potential of cross-service applications. Through the deployment of our model at the IoTMADLab facilities, we showcase the integration of IoT devices within varied urban infrastructures. The outcomes demonstrate the efficacy of VR interfaces in simplifying complex interactions, offering pivotal insights into device functionality, and enabling informed decision-making processes.

1. Introduction

Smart cities represent a paradigm shift in urban living, leveraging advanced technologies to enhance efficiency, sustainability, and quality of life for citizens [1]. However, urban services often operate within isolated silos, each functioning independently with minimal coordination or integration. These silos emerge due to historical development, bureaucratic structures, and specialized expertise, resulting in fragmented delivery of services (e.g., mobility, waste management and street lighting). Each service typically operates within its own department or agency, focusing solely on its specific mandate and objectives, without considering the broader urban context or potential synergies with other services [2]. Breaking down these silos is essential for creating more efficient, resilient, and sustainable urban environments. The integration of Internet of Things (IoT), Digital Twins (DTs), and Virtual Reality (VR) tools plays a pivotal role in shaping the landscape of these futuristic urban environments, enabling not only optimization of municipal services, but also providing valorization of cross-domain applications [3].

The IoT serves as the backbone of smart city infrastructure, connecting devices and systems to gather and analyze data for informed decision-making [4]. IoT facilitates the creation of intelligent transportation systems, environmental monitoring, and smart infrastructure, contributing to enhanced resource management and reduced environmental impact [5]. For example, IoT sensors deployed in traffic lights, public transit vehicles, and roadways can communicate seamlessly to optimize traffic flow, reduce congestion, and enhance the commuter experience. Likewise, IoT-enabled environmental sensors monitor air quality, noise levels, and weather conditions, providing valuable data for urban planners to implement measures to mitigate pollution and improve overall environmental health [6]. On the other hand, DTs, virtual replicas of physical entities or processes, have emerged as powerful tools in smart city planning and management. These digital replicas enable real-time monitoring, simulation, and analysis, fostering a deeper understanding of complex urban systems and facilitating data-driven decision-making and adaptive strategies [7]. One of the key advantages of digital twins in smart cities is their ability to model and simulate various scenarios, allowing stakeholders to anticipate the impact of different interventions or policies before implementation. For example, city officials can use digital twins to simulate the effects of changes in traffic patterns, urban development projects, or environmental policies, enabling them to make informed decisions that optimize outcomes and minimize risks [8]. Furthermore, digital twins enable cities to proactively manage and maintain infrastructure assets, such as roads or public lighting, by providing real-time insights into their condition and performance [9].

Additionally, VR tools bring immersive experiences to smart cities, offering innovative solutions for urban visualization, management interfaces, citizen engagement, and training. Integrating virtual reality technology with digital twins for smart cities and IoT applications profoundly enhances data management and interpretation, promoting wider participation in urban planning and management. VR crafts a dynamic, data-driven environment that is interactive, immersive, and conducive to collaboration, effectively bridging the physical and the digital, abstract worlds [10,11]. Additionally, it affords unique and enriched perspectives that augment contextual awareness of the represented scenarios. By superimposing extra information onto replicated scenarios, VR deepens environmental/operational understanding through pertinent data and graphical representations [12,13]. Moreover, VR boosts collaboration by facilitating interaction among users within a common space, enhancing collective deliberation and decision-making, and thereby fostering participant synergy [14,15,16]. Altogether, the synergy between IoT, DTs, and VR tools amplifies the impact on urban planning, governance, and sustainability [17].

While each of these technologies (IoT, DTs, and VR) individually contributes to the smart city ecosystem, their true potential is achieved through seamless integration, i.e., interoperability [18]. The use of a common data model for IoT devices enables consistent interpretation and utilization of data across various platforms, applications, and smart city services. Current efforts among standardization bodies and industry associations are focused on the importance of metadata to build comprehensive data models that enable interoperability [19]. There seems to be an alignment between many of them, creating an ecosystem definition where oneM2M, FIWARE, ETSI ISG CIM, and OMA LwM2M are co-existing and playing a clear complementary role [20]. There are a few research works focused on erasing urban services silos. Most of them aim at the use of semantic interoperability in different application fields such as energy [21], agriculture [22], or building management [23]. Other approaches concentrate on the integration of BIM and IoT, both for building [24], public facilities [25], and cities [26]. Finally, there are initiatives that work above the IoT platform level, focused on data integration [27], analytics [28], and applications [29]. However, to the best of our knowledge, there is not a previous work that proposes the combined use of IoT, DTs, and VR technologies towards breaking down the barriers between different urban service domains to enable integrated and collaborative approaches to city management.

The objective of this paper is to present the design principles and practical implementation of a DT concept for management of cross-domain smart city services. The DT solution, which has been developed in the framework of the IoTMADLab initiative, consists of a VR tool that creates a virtual replica of the real environment and the main functional features of the IoT devices. This virtual environment offers intuitive interaction to perform typical monitoring tasks through Head Mounted Displays (HMDs) and its controllers.

The rest of the paper is organized as follows: Section 2 introduces the IoTMADLab initiative, explains the methodology followed to develop the work, and provides some insights about the proposed IoT network architecture. Section 3 describes the main components of the DT platform, at both physical and virtual level, the technologies used for its deployment and the interconnections between the different elements. Section 4 presents the services currently implemented for both monitoring and controlling urban assets. Finally, Section 5 summarizes the main achievements and benefits of the proposed work while Section 6 discusses the future research lines to expand the DT capabilities.

2. IoTMADLab and Methodology

2.1. The IoTMADLab

The IoTMADLab is joint initiative between the Madrid city Council (through the Digital Office) and the Universidad Politecnica de Madrid (through the research center CEDINT), aimed at defining a standardized IoT network model that is open, neutral, and interoperable to facilitate the direct connection between devices (equipment, sensors, and actuators) from different manufacturers and services [30]. It serves as a platform for research, development, and innovation in the field of the Internet of Things (IoT) technology, particularly focused on applications for smart cities. Collaborating closely with the technicians from the different municipal services, service awarded companies, and their respective technological providers, the aim is to define a reference architecture to establish a framework that allows cities to implement and use IoT technology efficiently and effectively while ensuring consistency and interoperability among different devices and systems.

The IoTMADLab facilities are divided into three areas: the IoT Laboratory, the Controlled Environment, and Virtual Reality Laboratory.

• IoT Laboratory: an indoor laboratory equipped with measurement and testing equipment and data representation interfaces. It is where the compatibility, interoperability, and cybersecurity IoT devices and sensors are tested. Compatibility certification requires that the devices fulfill the requirements of the specific municipal service and comply with the communication protocols from the IoTMADLab IoT network reference architecture. Interoperability certification, which demands compiling with the data model, may be achieved at different degrees (depending on the communication protocol. For both certifications, minimum cybersecurity requirements must be met.
• Controlled Environment: an outdoor replica of the urban environment, integrating operational city elements such as streetlights, waste containers, parking lots, and green areas. Once the equipment has been validated in the laboratory, it is deployed and tested in a controlled yet real-world environment. In this area, the devices are exposed to variables and conditions they might encounter in actual urban situations, allowing for monitoring and evaluation of their performance in a more genuine setting. This ensures that issues, incompatibilities, or unforeseen challenges can be identified and resolved before the devices are deployed in an operational urban setting. The impact of interoperability across municipal services is closely examined.
• Virtual Reality laboratory: a facility equipped with the most advanced XR hardware and software for the design and development of pioneering solutions that address complex problems and technological challenges across various disciplines. It specializes in creating immersive VR experiences, being the creation of the digital twin for the IoTMADLab. This initiative seeks to construct an interactive virtual counterpart of the laboratory, utilizing Virtual Reality to achieve precise representation, visualization, and interaction within both tangible and digital domains. To achieve its goal, developers undergo a detailed development process encompassing several key stages: 3D modeling, realistic texturing, and integrating communication protocols with IoT platforms. The design of user interactions is carefully considered to provide intuitive and engaging experiences.

2.2. Methodology

Cities are actively engaging in a digital transformation strategy that positions municipal services as both catalysts and beneficiaries. Interconnection among these services is a must to improve management and enable added value applications. In this work, we propose the combined use of IoT, DTs, and VR to foster collaboration among services.

• As a first step, we conducted a detailed review of the city’s current technological infrastructure and services. This assessment aimed to identify areas where enhancements could be made and potential gaps in technological capabilities, particularly in terms of IoT connectivity. This process included mapping out municipal devices, facilities, and equipment that could benefit from IoT technologies, assessing the existing network and communication systems, and evaluating the degree of digital integration within municipal services. The collaboration with the Digital Office of Madrid City Hall was crucial in defining a starting point for strategic planning and the deployment of IoT solutions aimed at enhancing urban life.
• Following this groundwork, the focus shifted to establishing a common and interoperable framework that would allow for seamless communication between IoT devices and other elements within the city’s infrastructure and their respective control centers. An IoT network reference architecture and an open data model was proposed to facilitate clear and effective data interpretation and management across various municipal services, thereby improving service efficiency and quality. The main results from this analysis are described throughout Section 2.3.
• In addition to establishing this framework, the development of a digital twin enhanced with virtual reality technology was initiated. This digital twin is designed to simulate real-world scenarios, test IoT functionalities, and exchange information between physical and virtual components in a controlled environment. The objectives of this model include creating a virtual space for managing and controlling an IoT system equipped with interoperable sensors, enhancing decision-making processes through immersive experiences and data visualization, and conducting thorough tests on IoT communications and services before their widespread implementation in smart urban settings. The main features of such proposal are outlined in Section 3.
• A phase of pilot testing and evaluation was outlined to rigorously assess the practical implementation of IoT technologies and the functionality of the digital twin with virtual reality. This phase took place at the IoTMADLab facilities and is crucial for refining data models, communication protocols, and the synergy between the physical and digital worlds, ensuring that the technological infrastructure and its virtual counterpart operate seamlessly together. This step is vital for confirming the system’s effectiveness and readiness for a larger-scale deployment, aiming to establish a robust foundation for real-time data exchange and analysis in the context of Smart Urban Spaces (SUS). SUS are pilot areas within Madrid, and are designed in collaboration with city technicians and awarded service companies and answer to real needs of municipal services. The aim of SUSs is to show city governors, technicians, and citizens the potentials of collaboration cross municipal services leveraging IoT interoperability.

2.3. IoT Network Reference Architecture

The objective of the IoTMADLab IoT network reference architecture is to establish the requirements in terms of connectivity and semantics for achieving direct interoperability between IoT devices, which may be from different manufacturers, and even from different municipal services. The IoTMADLab focuses on the lower layers of the IoT ecosystem, i.e., at the network level, including the connectivity layer and the data model (see Figure 1).

In IoT applications for Smart City, one of the necessary requirements is a reduced overhead in the information encoding and the use of lightweight communication protocols to reduce the amount of information transmitted and processing needs so that simple, low-cost, low-power devices can be used. It is also important that the data model and protocols used should be based on open, mature standards that are widely implemented, allowing the development of an ecosystem of products and projects that favors competitiveness and innovation.

2.3.1. Data Model

The data model defines the logical structure of the information associated with an asset monitoring, management, and control application such as a city’s public lighting system. It can also define the protocols for accessing and communicating this information. To ensure the interoperability of the systems that make up the application and to avoid dependence on a single vendor lock-in, the data model must be based on open specifications standardized by independent entities not affected by commercial interests.

After analysing different approaches, the IoTMADLab has decided to align with the uCIFI data model [31]. The uCIFI alliance is an open, not-for-profit organization whose goal is to develop open standards for enabling device interoperability of devices in Smart City applications. The uCIFI data model defines the use of four protocols: LwM2M, IPSO, CoAP, and CBOR.

• LwM2M (Lightweight M2M), promoted by the Open Mobile Alliance (OMA), is a protocol designed for efficient management of IoT devices, providing standardized communication and data exchange. It enables remote device management, firmware updates, and monitoring, optimizing IoT deployments. LwM2M’s lightweight nature ensures compatibility with resource-constrained devices, enhancing scalability and interoperability in IoT ecosystems. This protocol streamlines device management processes, facilitating seamless integration and operation of IoT networks [32].
• The IPSO (Internet Protocol for Smart Objects) Alliance provides a framework for interoperability among IoT devices, defining common data models and communication protocols. By standardizing data representation and exchange, IPSO simplifies device integration and application development in IoT ecosystems. It facilitates seamless interoperability between heterogeneous devices, fostering scalability and innovation. IPSO’s approach enhances efficiency and reliability in IoT deployments, promoting the widespread adoption of smart technologies [33].
• CoAP (Constrained Application Protocol) is a specialized web transfer protocol designed for constrained IoT devices, offering lightweight communication with low overhead and high efficiency. It enables devices to exchange data over the internet in a constrained environment, optimizing resource usage. CoAP’s simplicity and flexibility make it ideal for IoT applications, supporting reliable, asynchronous communication and resource discovery [34].
• CBOR (Concise Binary Object Representation) is a compact data serialization format derived from JSON, designed for resource-constrained devices in IoT applications. It efficiently encodes complex data structures into binary form, reducing payload size and transmission overhead. CBOR’s simplicity and efficiency make it suitable for constrained environments, facilitating seamless data exchange between devices and applications [35].

2.3.2. Connectivity Layer Technologies

After analysing the current IoT connectivity technologies, the IoTMADLab has selected those that are (or may be in the future) compatible with the uCIFI data model and that, due to their characteristics, cover all the requirements of the possible use cases from smart city services: Wi-SUN, NB-IoT, and LoRaWAN. For the first phase of the implementation of SUSs, WI-SUN has been chosen as its compatibility with the data model has already been validated.

The Wi-SUN (Wireless Smart Ubiquitous Network) Alliance defines the Wi-SUN FAN (Field Area Network) specification, which has resulted in the IEEE 2857-2021 standard, a mesh network protocol, which works over 6LoWPAN and is based on several IETF, IEEE, and ANSI/TIA standards. 6LoWPAN (IPv6 over Low Power Wireless Personal Area Network) is a communication standard designed to enable connectivity of low-power and resource-limited devices over wireless personal area networks (WPAN) based on IEEE 802.15.4. As it is based on Internet Protocol Version 6 (IPv6), it allows assigning unique IP addresses to each connected device, facilitating its identification and communication in the network.

Hence, communications at the lowest layer of the stack are carried out by means of 6LoWPAN-based technology up to the IoT gateway, where the next step until the server-side platform coordinator is usually conducted via standard Ethernet connectivity, falling back to Wi-Fi or LTE when the lack of wiring becomes a factor. Regardless, this piece within the full-stack communications path is always tunnelled inside a VPN connection so that the actual connectivity infrastructure can be transparent in the deployment process while at the same time ensuring security.

Future IoTMADLab deployments will also be targeting NB-IoT and LoRaWAN at the lowest layer of the stack. This means that the network might even drop the gateways or coordinators when NB-IoT is used. For LoRaWAN use cases, a proper gateway is usually required, so that the stack and logical behaviour of the network shapes that of Wi-SUN’s use cases. Nonetheless, that one hop is avoided in the case of NB-IoT, where network gateways become nil, at least functionally and in terms of provisioning.

Wi-SUN implements a meshed network topology, which enables direct communication between devices (expanding coverage) and uses a hierarchical, self-organizing routing approach, where nodes act as routers to relay packets between devices. On the security side, encryption and authentication mechanisms (128-bit AES) are used to protect the communication between the devices and the network. The following communication parameters within the Wi-SUN FAN specification have been defined, ensuring compatibility with uCIFI reference mesh implementation:

-

Frequency bands: EU1 (863–870 MHz) and EU2 (870–876 MHz).

-

PHY mode of operation: 2a (100 kbps).

3. System Description and Main Components

The foundational elements of the Digital Twin infrastructure for management and operational oversight are illustrated in Figure 2. At the Campus of Montegancedo (Universidad Politecnica de Madrid, Spain), the urban controlled environment encompasses pedestrian zones, green areas, a parking zone, and the building where the IoTMADLab is located. At the IoTMADLab facilities, the following urban elements co-exist: LED streetlights, parking lots, watering systems, garbage bins and containers, a bike sharing station, and building systems (HVAC, lighting, electric panel board, and power plugs). The integration of IoT sensors and actuators within these elements provides the data to feed the virtual/digital world. Within this digital world, simulations and predictive analyses of physical elements are conducted. Acting as a conduit between the tangible and digital worlds, the communication network and IoT platforms facilitate seamless integration. The service layer enriches the Digital Twin experience by offering value-added features such as monitoring and control functionalities, which are accessible through virtual interfaces like Head-Mounted Displays (HMDs).

3.1. The Physical World

The IoTMADLab is a physical space designed to work with various teams, IoT devices, and protocols to develop a standardized IoT network reference architecture. This architecture aims to establish an open, neutral, and interoperable IoT network that facilitates the direct connection of devices from different manufacturers within the same service, as well as direct connectivity with devices from other services and manufacturers. To achieve this, it mainly consists of an integration and testing laboratory and a pilot installation area. In the laboratory, a series of technical tests are conducted to verify the connectivity and interoperability of devices linked to various IoT sensors and actuators. The pilot installation areas comprise a controlled real-world environment where equipment, whose interoperability has been previously verified in the laboratory, is deployed. In this area, their performance under real conditions is tested, monitored, and the impact of interoperability on city services is assessed.

Figure 3 presents the IoT laboratory and two areas of the controlled environment of the IoTMADLab. The controlled environment encompasses diverse urban elements that are equipped with IoT devices enabling the digitalization interface (sensor/actuator) and the communications to the Internet:

• Streetlights: there are 12 LED streetlights from different manufacturers, each one equipped with IoT nodes, illumination, and motion detection sensors (from different vendors). The IoT nodes are connected through a Zhaga interface, allowing plug&play replacement, while sensors use both Zhaga and internal interfaces. The streetlights can work autonomously, being triggered by illumination and motion/presence values, controlled manually (with remote control commands), or depending on values from other sensors following pre-configured rules.
• Parking lots: there are four parking lots equipped with IoT parking sensors, both below and above ground. The sensors, which are battery-powered, detect if a vehicle stands at parking lot by means of radar and earth magnetism sensors. When a change in the parking status is detected, the device sends a message indicating the new status. To save battery, these sensors remain in sleep mode most of the time, waking up periodically to check the parking occupancy or to send a periodic keep alive message if there is no change in the parking status.
• Waste containers: there are two garbage disposal bins and a public paper recycling container equipped with waste IoT sensors. The devices, which are radar-based volumetric sensors, measure the filling level of the bins/containers. These devices also have GPS to generate alarms in case of detecting large movements with respect to the installation location, a temperature sensor with configurable thresholds to generate alarms in the event of fire, and accelerometers to detect emptying operations and alarms due to vandalism. These devices are battery powered and will provide up to eight years of operation depending on how frequent measurements and transmissions are configured.
• Environmental information: different IoT sensors monitor environmental parameters such as outdoor temperature, relative humidity, wind speed, and solar radiation. They are connected to the electrical mains and allow both periodic measurement and data request.
• Watering systems: three irrigation valves regulate water flow for green areas watering, which may be scheduled depending on weather forecast and actual soil moisture values. Soil temperature and pH sensors provide additional information. Both irrigation valves and soil sensors are plugged to IoT nodes to enable communications.
• Building facilities: the IoTMADLab indoor laboratory activities are monitored and optimized using diverse IoT devices. On one side, IoT power meters are connected to the building electric panel boards, measuring the energy consumption of the individual electric lines. On the other side, HVAC, lighting, and other energy consumers (e.g., workstations) are managed by means of IoT smart plugs. Additionally, environmental IoT sensors collect indoor temperature, relative humidity, and presence data.

3.2. The IoT Platform

Synergies between various technologies are essential to conform the backbone of advanced digital twin interactions. The IoTMADLab controlled environment IoT platform is founded on Home Assistant, which serves as the main top-layer application and integrator for the whole system. This thoughtful choice was motivated by several critical factors that align with the goals and requirements of our project, emphasizing interoperability, scalability, and ease of integration with a wide array of IoT devices and protocols. Home Assistant provides an ideal platform to incorporate data from different vendors and third-party platforms, as it is highly expandable, open-source, and modular, while also prioritizing privacy and allowing molding of the local-cloud frontier as one may wish [36].

Some of the decisive key characteristics and benefits of Home Assistant are the following:

• Open-Source and Community-Driven: Home Assistant’s open-source nature fosters a vibrant community contributing to its continuous improvement. This aspect ensures the platform evolves in response to emerging IoT trends and technologies, providing a rich ecosystem of plugins, integrations, and support for a broad spectrum of devices and services, being virtually infinite.
• Privacy-Centric and Local (or not) Control: Unlike many IoT platforms that solely rely on cloud-based logic, operation, and algorithms, Home Assistant pushes privacy and local control. This philosophy ensures that data generated by smart city services remain within Local Government’s boundaries, mitigating privacy concerns, keeping data close, and lowering dependency on third-party cloud services.
• Extensive Device Compatibility, Flexibility, and Customizability: Home Assistant supports a wide range of devices and protocols, making it an ideal choice for smart city applications. The platform’s ability to seamlessly integrate devices across different manufacturers and communication protocols facilitates the development of cohesive and interoperable smart city services. The modular architecture of Home Assistant allows for high degrees of customization and flexibility. Home Assistant can be tailored to meet use cases’ specific needs, ranging from environmental monitoring to urban mobility solutions. This adaptability, along with its rapid growth and constant improvement over the last years, extends the platform’s applicability beyond home automation to more complex smart city infrastructures.
• Scalability: Home Assistant’s basal lightweight and efficient design ensure it can scale to accommodate the growing number of IoT devices and services within smart city ecosystems. This scalability is crucial for sustaining the dynamic expansion of smart city services and their evolving requirements.

The IoT platform makes use of RESTful APIs and MQTT to communicate with IoT network gateways and management systems (CMS), retrieving data and sending commands. In addition, it integrates a state-of-the-art InfluxDB database for time-series data, which allows a very responsive, efficient, and accountable interaction with live events. By integrating Home Assistant with InfluxDB and leveraging the communication capabilities of RESTful APIs and MQTT, the IoT platform bridges the gap between the physical and virtual worlds in real time, retrieving data to mirror the real world (MQTT) and sending commands to control the physical world from the virtual interface (REST). This integration provides a comprehensive solution for monitoring, controlling, and visualizing the events from IoT devices within the digital twin ecosystem.

The IoT platform also integrates Grafana and Telegraf, allowing for the creation of dynamic and interactive dashboards. These dashboards provide real-time insights into the IoT ecosystem, showcasing trends, patterns, and potential anomalies. Grafana’s flexibility in data visualization makes it an indispensable tool for users who seek to understand and analyze the vast amounts of data generated by their IoT devices, while Telegraf provides a rapid and reliable manner of monitoring the performance of the InfluxDB-based system.

3.3. The Virtual World

The Virtual Reality (VR) environment, developed using the Unity 3D engine, is centered on a detailed 3D digital representation of the IoTMADLab where the different devices and sensors are located (as depicted in Figure 4). This virtual model accurately reflects the demonstrator layout and is geographically precise as it utilizes GPS coordinates to position urban elements and associated sensors within the virtual landscape. Moreover, the model boasts visual realism, achieved by texturing the 3D assets with high-resolution images of real materials. To ensure a seamless and immersive user experience, the model has been meticulously optimized to meet the hardware specifications and constraints of popular commercial VR headsets. For instance, when running on standalone VR headsets with limited computational power, such as the Meta Quest 2, the model employs simplified geometry and lower-resolution textures. Various optimization techniques have been implemented to maintain smooth performance and minimize VR-induced discomfort, including object culling to render only visible objects, the use of different levels of detail for distant objects, and reducing computationally intensive visual effects like shadows and transparency effects without compromising overall visual fidelity. The software has been implemented following the OpenXR API [37], which supports interacting with VR system in a platform-agnostic way. Nevertheless, it has been fully tested on some of the most prominent headsets of the market like HTC Vive Pro, HTC Focus 3, and Meta Quest 2.

3.4. The Virtual Interfaces

Navigation within the virtual environment is optimized for an immersive experience, employing a first-person perspective to enhance realism. Users wearing Head-Mounted Displays (HMDs) can freely explore the virtual space, with the virtual camera adjusting its view based on the user’s movements tracked by the HMD’s sensors (see Figure 5). To traverse larger distances, users can employ VR controllers to teleport seamlessly to specific locations, situated near the devices (represented by blue teleport cylinders). Alternatively, users can switch to predefined spaces that agglutinate IoT elements: the laboratory space inside the building, the streetlights path, and the parking lot area. The VR controllers facilitate interaction with operational devices; hovering over a device highlights it in yellow, and enabling interaction is as simple as pulling the controller’s trigger. Active devices are indicated by a green border and offer a 3D user interface displaying status, control options, and historical charts.

3.5. Communication Channels

Figure 6 depicts the interaction between the VR application and the Home Assistant. Utilizing the RESTful API, the login module posts user credentials for authentication and receives an authorization token in return, which must be included in the HTTP header of subsequent requests. The initial state for all devices is acquired through a GET message, comprising GPS coordinates for accurate localization within the virtual environment and the current state values of the IoT devices. Device actuations, such as controlling light dimming, are executed by sending PUT messages to the RESTful API with the new value. Upon receiving a correct response, the device is updated to reflect the real status. Following initialization, synchronization is achieved by receiving state updates through subscription to the MQTT service of the Home Assistant. Lastly, visualization of charts and dashboards for retrieving historical data of the devices is facilitated through an embedded browser communicating with the Grafana dashboard of the Home Assistant.

4. Implementation of Smart City Services

To demonstrate the capabilities of combining VR technology with a Digital Twin infrastructure for effectively managing the IoT devices deployed within the IoTMADLab controlled environment, the following services have been deployed.

4.1. Controlling and Reading Status of IoT Devices

Users have the capability to interact with the urban elements by means of the connected IoT devices. For example, they can adjust the brightness level of individual streetlights by interacting with a slider integrated into the virtual interface. Alternatively, they can switch between different predefined values by pressing the corresponding dimming button. When a virtual streetlight is selected, the VR environment sends a REST PUT message to the IoT platform to update its current dimming value, ranging from 0% (fully off) to 100% (fully on). Upon confirmation, the VR environment is updated to reflect the new status, and a synchronization message confirms the action along with a timestamp. Correspondingly, the physical streetlight’s LEDs adjust accordingly. In the event of communication failure, an error message is displayed, and the dimming value reverts to its last known valid state. This service proves invaluable for targeted maintenance tasks, such as connectivity testing, hardware diagnostics, or selective lighting control. Figure 7 illustrates a user controlling a streetlight’s dimming rate, with the real-world lamp’s status displayed in the inset of the upper left corner.

Users may also consult the status of the urban elements by accessing real time data from connected IoT sensors. Figure 8 shows the interaction with a garbage bin (left) and a parking lot (right), with the real-world status displayed in the inset of the lower left corner. When accessing the IoT waste sensor, the filling level (in both percentage and volumetric amount) can be checked. For the parking lot, the DT tool shows the real status by locating a vehicle in the position and shows the occupancy patterns during the last days.

4.2. Visual Analysis of Sensor Data

The VR application has the capability to access data records from the IoT platform regarding specific variables of interest: e.g., dimming values, parking occupancy, garbage bin filling level, wind speed, soil moisture, or building energy consumption. These data are then visualized using interactive charts within the Grafana IoT platform (Home Assistant). Presented as line charts to depict time series data, each dot represents an average value for a given time range, aiding in trend analysis. Users can interact with the chart similarly to a regular browser, adjusting time ranges, accessing additional information via tooltips, or toggling between available variables. This service facilitates monitoring of sensor operational status across different time scales, identifying regular patterns like day-night alternations, and potential anomalies such as missing data due to communication errors or irregular conditions. Figure 9 exemplifies such monitoring, displaying a temperature sensor within the building facility from night to day, including automatic adjustments based on environmental conditions.

4.3. Cross-Domain Applications

To showcase the potential of the DT solution for the efficient management of cross-domain city applications, the following test concepts have been implemented:

• Lighting control depending on parking occupation: streetlight dimming level can be controlled based on parking occupancy to optimize energy usage and enhance safety in urban environments. When the IoT sensors deployed in parking lots detect occupancy, nearby streetlights may automatically adjust to higher illumination levels to enhance visibility and security for pedestrians and drivers. Conversely, when the parking lot is no longer occupied, streetlights may dim to conserve energy while maintaining safety standards. This cross-domain application (street lighting and mobility services) not only reduces energy consumption and carbon emissions but also ensures that lighting levels are tailored to actual usage, contributing to more efficient and sustainable urban infrastructure.
• Watering management triggered by presence detection: green areas irrigation systems can make use of data coming from the motion detection sensors equipped at streetlights near green spaces. When presence is detected, the irrigation system may temporarily pause or reduce water flow to prevent bothering passers-by and prevent water waste. On the contrary, when no presence is detected, the system resumes or adjusts irrigation schedules to ensure adequate moisture levels for plant growth. This cross-domain application (watering and street lighting services) not only conserves water but also minimizes runoff and contributes to the creation of greener and friendlier urban spaces.
• Garbage container status signaled by streetlights blinking: streetlights can serve as effective indicators for filling levels or fire risk of garbage container, enhancing waste management efficiency in urban areas. When a garbage container reaches a predefined filling threshold or detects an abnormal temperature value, it sends a signal triggering nearby streetlights to start blinking, signaling to waste collection teams the need for emptying to fire risk situation. This cross-domain application (waste management and street lighting services) promotes timely waste removal, preventing overflow and littering, and improving overall cleanliness and hygiene in the city.

5. Conclusions

In this paper, a VR-based Digital Twin solution to optimize the management of smart city services is presented. The system, whose concept test is developed at the IoTMADLab facilities in Madrid, comprises various urban elements equipped with IoT devices, facilitating seamless integration between the physical and digital worlds, enabling real-time data collection, simulation, predictive analysis, and control functionalities. The physical world component of the Digital Twin infrastructure encompasses a diverse array of IoT-enabled urban elements, such as streetlights, parking lots, waste containers and building facilities. These devices provide critical data on various parameters such as illumination, occupancy, filling level, environmental conditions, and energy consumption.

The IoT platform, based on Home Assistant, serves as the backbone of the Digital Twin infrastructure, facilitating communication between IoT network (gateways and management systems) and the virtual world. Leveraging RESTful APIs and MQTT protocols, the platform ensures seamless data retrieval and command transmission, enabling real-time interaction between the physical and virtual worlds. Integration with InfluxDB database and visualization tools like Grafana enhances data analysis and visualization capabilities, empowering users to gain insights into IoT ecosystem trends, patterns, and anomalies.

The virtual world component offers a detailed 3D digital representation of the IoTMADLab environment. Geographically precise and visually realistic, the virtual model enables users to navigate and interact with IoT devices seamlessly using VR headsets and controllers. Optimization techniques ensure smooth performance and immersive user experience across different hardware configurations, while OpenXR API compatibility ensures platform-agnostic support for leading VR headsets.

The smooth operation of such solutions, which integrates different technologies (IoT, DTs, and VR), require the use of an IoT architecture based on open and standard protocols. The IoTMADLab proposed a reference IoT network architecture, which serves as a practical example of how these technologies can be deployed to create cross-domain smart city services. It facilitates direct connectivity between devices from different manufacturers and municipal services, promoting interoperability and efficiency in IoT deployments. Moreover, the DT concept, integrated with VR technology, offers intuitive interaction and visualization for monitoring and controlling urban assets.

The novelty and value of our approach lies not in the individual technologies per se but in their integration at the city-wide level. This endeavor actually poses a multitude of complex challenges that demand innovative solutions.

Firstly, achieving seamless integration of IoT, Digital Twins, and VR across an entire urban ecosystem involves navigating a highly fragmented technological landscape. Cities are complex entities with legacy systems and infrastructure that must be retrofitted with new technologies. Ensuring compatibility between old and new systems, while also maintaining the flexibility to incorporate future advancements, requires an understanding of both the technical and socio-economic dimensions of urban environments.

Another major challenge is the creation of a digital twin or a federated group of digital twins that could accurately represent the working of each urban service. This involves not only the technical difficulty of modeling complex urban elements and behavior, but also the logistical and organizational challenges of coordinating between multiple stakeholders, including government areas, private companies, and citizens. Achieving a comprehensive and coherent digital twin that can be effectively used for simulation, visualization, and decision support in VR requires a concerted effort.

Lastly, ensuring the system’s sustainability and scalability presents another layer of complexity. As cities evolve, the integrated system must be capable of adapting to changing needs and technologies without requiring complete overhauls. This necessitates forward-thinking design principles that prioritize modularity, interoperability, and energy efficiency.

The integration of IoT, Digital Twins, and VR present several advantages, of which we would like to highlight the user interface and the cross-service applications.

On one hand, accessing VR-based digital twins through Head-Mounted Display (HMD) devices, including VR headsets and AR glasses, unlocks a multitude of possibilities and benefits for urban management and planning. These interfaces offer users an immersive, first-person perspective of these digital, 3D replicas, enabling them to navigate and interact with the virtual environment in real-time. This immersive experience facilitates a deeper understanding of complex urban dynamics, enhances decision-making processes by visualizing the outcomes of various scenarios, and improves stakeholder engagement through interactive and engaging presentations of urban projects. Furthermore, the use of HMD devices in accessing digital twins aids in education and training, allowing city officials, engineers, and the general public to simulate and rehearse responses to emergencies, infrastructure developments, and urban planning strategies with unprecedented realism.

On the other hand, the implementation of city cross-service applications, such as controlling streetlights based on parking occupancy, optimizing watering management triggered by presence detection, and signaling garbage container status through streetlight blinking, demonstrates the potential of the Digital Twin infrastructure for efficient urban management. Besides, it lays the groundwork for the invention and deployment of new applications that provide added value to both the urban technicians, the service awarded companies and, of course, the citizens.

In conclusion, the Digital Twin infrastructure deployed at the IoTMADLab represents a pioneering approach to urban management and operational oversight. By bridging the physical and digital worlds through IoT-enabled devices and VR-based interfaces, the system enables real-time monitoring, analysis, and control of urban elements, paving the way for smarter, more sustainable cities of the future.

6. Future Work

Looking ahead, future research should focus on expanding the capabilities of DTs for managing cross-domain smart city services. By further integrating IoT, DTs, and VR tools, cities can unlock new opportunities for innovation, sustainability, and resilience. Collaboration among stakeholders, continued standardization efforts, and investment in technology infrastructure are essential for realizing the full potential of smart cities in improving the quality of life for citizens and fostering sustainable urban development.

While initial tests demonstrate the system’s functionality and rapid synchronization between actions performed in the virtual environment and their real-world counterparts, further experimental validation is required. Specifically, there is a need to characterize the timing behavior, focusing on end-to-end latency and error rates. This endeavor will aid in identifying and addressing potential bottlenecks that may arise within the system involving diverse components and technologies.

To support more informed and intelligent management of the urban assets, advanced visualization functionalities are imperative. Thus, we propose incorporating dashboards embedded with visual analysis capabilities to facilitate tasks such as preemptive anomaly detection and identification of optimal lamp distribution. In this context, the virtual environment is expected to enhance charting expressiveness and implement more intuitive interaction interfaces, such as eye or finger tracking.

Future work may also include technical upgrades such as the integration of NB-IoT and LoRaWAN as IoT network communication technologies, the creation of automated connections between the IoT platform (Home Assistant) and the IoT network management systems.

Deploying digital twins within a Virtual Reality framework, while promising, introduces challenges that extend beyond technical complexity and data integration. The economic and technical hurdles stem not only from the need for sophisticated infrastructure capable of handling real-time data but also from the absence of standardized models and frameworks. This lack of standardization hampers interoperability across diverse systems and IoT devices, affecting the efficiency and effectiveness of digital twin solutions in urban settings.

In this context, Building Information Modeling (BIM) technology emerges as a critical component in addressing some of these challenges. BIM offers a standardized approach to the design, construction, and management of buildings and infrastructure, facilitating the creation of digital representations that are detailed, consistent, and easily shareable among stakeholders. When integrated with digital twins and VR, BIM technology can significantly enhance the modeling accuracy, data management, and interactive capabilities of digital twins. For instance, the use of BIM models as the foundation for digital twins in urban planning can streamline the integration of architectural and engineering data, improving the simulation of real-world scenarios, and enabling more precise decision-making.

To overcome potential drawbacks of using HMD devices, such as user discomfort or limited field of view, continuous technological advancements are essential, like those anticipated from several leading tech companies. These new Extended Reality devices are expected to feature enhanced ergonomics, wider fields of view, and more intuitive user interfaces. These improvements will significantly mitigate user discomfort and increase adoption rates. Moreover, integrating these devices with cutting-edge AR technology will enable the overlay of digital information directly onto the physical world, enriching the user’s interaction with the digital twin. This integration requires robust data processing capabilities and seamless connectivity to ensure real-time updates and interactions. By harnessing the power of next-generation HMD visors, the access to and interaction with VR-based digital twins can be made more intuitive, inclusive, and effective, marking a significant step forward in the digitalization of urban environments.

Finally, incorporating a 5G communication layer and cybersecurity measures into a VR-based digital twin solution for smart cities is essential for enhancing urban infrastructure and services. The integration of 5G provides ultra-reliable, low-latency communication critical for the real-time data transmission required by digital twins and VR applications. This ensures that urban planners and citizens can interact with the digital twin of the city seamlessly, experiencing immersive simulations with minimal delay.

To protect this sophisticated ecosystem, robust cybersecurity measures must be embedded from the outset. This includes the implementation of end-to-end encryption for data in transit, secure authentication protocols for device and user verification, and regular security audits to identify and mitigate potential vulnerabilities. Advanced threat detection systems, powered by AI, should be employed to monitor network traffic for unusual patterns indicative of cyber threats, ensuring the integrity and confidentiality of the digital twin data.