Modeling a Digitally Enhanced Real World Inspired by Agential Realism—Exploring Opportunities and Challenges

Abstract

In this paper, we propose a conceptual-model called the virtualizing/reframing (V/R) twin model to construct a digitally enhanced real world. The V/R twin model simulates the real world, and is an extension of the conventional digital-twin model, which can accurately model the real world and provides a general-purpose method for building digital services that enhance the real world. The major difference between the proposed model and the conventional digital-twin model is its consideration of diverse new information-presentation devices that have been recently developed. The V/R twin model is inspired by agential realism to include the “entanglement of the social and the material”, and the proposed observable-world consists of the social and material that are separate, according to the current context. After explaining the outline of the V/R twin model, where four virtualizing-patterns and reframing-patterns are introduced, the potential opportunities for the V/R twin model are examined, from multiple perspectives.

1. Introduction

The digital transformation (DX) of society has been the driving force of the world’s growth over the past decade [1]. Digital technologies have dramatically changed many aspects of daily life over the last century, including private life, administration, industrial structure, and employment. For example, using the Internet, we can connect to everything, anytime, anywhere, particularly digital information with smartphones. Additionally, just as operating systems virtualize computer systems, DX enables daily life to be more efficient, more enjoyable, and more exciting, by virtualizing the real world. This transformation is also making the world more comfortable and more sustainable [2,3]. Recently, developed new technologies such as computational materials, have enhanced DX one step further by integrating intelligent material into the proposed daily environments [4]. Everything in daily life will soon be described by information and will change form flexibly.

The digital-twin model is a powerful approach for modeling the real world [5]. This approach creates a digitized replica of the “things” that exist in the real world, in a virtual world. Digitized replicas capture a variety of information about the corresponding real-world stuff; they build models, analyze current states, optimize the usage of what exists in the real world, and allow the prediction of future risks [6]. It is a very powerful model for representing and controlling various physical resources in the real world, especially when building future advanced real-world services such as smart-city services [7,8,9].

In this paper, we propose the virtualizing/reframing (V/R) twin model, which is a new conceptual-model that constructs a digitally enhanced world that fuses the real and virtual worlds. The V/R twin model is an extension of the traditional digital-twin model, which can accurately model the real world and provides a general-purpose method to build digital services that enhance the real world. The major difference between the proposed model and the conventional model is its consideration of diverse new information-presentation devices that have been recently developed. However, from a philosophical perspective, the V/R twin model allows us to explicitly investigate the amalgamation of real and digital, inspired by agential realism [10,11], with four virtualizing patterns and reframing patterns that offer powerful ways to model the real world. The model considers the “entanglement of the social and the material”, and the observable world consists of the social and material that are separate, according to the current context. After outlining the V/R twin model, where four virtualizing patterns and reframing patterns are introduced, the potential opportunities of the V/R twin model are examined from multiple perspectives. The opportunities identified are promising for exploring the augmenting of the real world with a variety of recently advanced digital technologies, such as mixed reality, robotic technologies and sensing technologies.

The following characteristics of the proposed model are the major contributions to this study.

(1).

The V/R twin model includes the virtualizing and reframing patterns to consider new services to extend the real world. The real world can be extended by reframing the real world with the patterns. In particular, the reframing patterns enable us to consider augmenting the real world with advanced technologies such as mixed-reality technology and robotics technology.

(2).

The study presents an analysis of potential opportunities and the methods of using the V/R twin model, from multiple perspectives. We extracts diverse opportunities to use the virtualizing and reframing patterns, and present an enhanced conceptual-model which flexibly extends the real world.

The remainder of this paper is organized as follows. In Section 2, we briefly outline a landscape of conventional digital-twins. Section 3 proposes the virtualizing pattern and reframing pattern that are essential components of the V/R twin model. In Section 4, we first show two case-studies, to analyze existing digital services with the V/R twin model. Then, we investigate how the two patterns that are essential components of the V/R twin model offer new opportunities to construct the alternative real world from the analysis of the case-studies. Finally, an enhanced concept model and its use-cases are presented. The section reveals the diverse potential opportunities of the V/R twin model, from multiple perspectives. Section 5 shows related research activities, and Section 6 concludes the paper, and presents some future opportunities and challenges.

2. Digital Twin

Discussions of a real-world model have occurred since the early 1990s [12,13], when Mark Weiser’s famous article on ubiquitous-computing research was published [14]. The latest approach to the world model, called a digital twin, was proposed by Michael Greaves in 2003 [15], and then various studies published new definitions of the digital twin [16,17].

The general and standard architecture of a digital twin consists of three components: a physical object, a digital model, and the data that connect the physical and digital domains [18]. Physical objects provide the basis for building digital models. Digital models support the control, simulation, and decision-making of physical objects. Finally, data are a prerequisite for generating new knowledge.

A digital twin is typically defined as a virtual copy of a physical system [7,16,17,18], but there are some differences in the definition of the data-integration level. For example, Kritzinger et al. define three levels of data integration [19]. Digital models are digital representations of physical objects that do not automatically exchange data between themselves. Digital shadows have an automatic data flow from a physical object to a digital object, but not the other way around. Finally, the digital twin fully integrates the data flow in both directions. A recently published study reports that interactions between physical and virtual objects are primarily offline interactions, and lack continuous online interactions [19,20].

Digital twins fall into the following three primary categories [15]. The first category uses data and virtual worlds to explore system maintenance and operational planning. The second category is a combination of various data mirroring the system of the virtual world from a broader perspective. The final category investigates the product at the time of design through simulation and uses it for better decision-making. The most important thing in all categories is to digitize real-world things and make them available as models.

In a digital twin, various things in the real world (left-hand side of Figure 1) must be converted into digital shadows in the virtual world (right-hand side of the figure), where the numbers represent the ID of each digital shadow in the virtual world, and the ID is used to access detailed information about the digital shadow. One classic example of current approaches is the Virtual Helsinki Project [21]. In the Virtual Helsinki Project, an exact replica of Helsinki is digitally constructed in the virtual world. The digitized replica consists of various virtual objects, such as virtual buildings, virtual streets, and virtual shops, modeled from acquired data. Virtual objects can be used to visualize a virtual world that represents a digitized replica that is a mirror world of the real world. In addition, digitized replicas can be used to predict various aspects of the real world, according to the acquired data. This approach provides an infrastructure that provides promising opportunities for developing innovative services in the future. Digitized replicas within the virtual world are also used to manage invisible information that exists within the virtual world.

The use of digitized replicas increases the chances of expanding the range of use of digital twins. In traditional approaches, the exact model of the real replica is the most important factor in designing a digital twin [22]. Some studies extend real replicas through augmented-reality technology, but most extensions focus only on overlaying invisible supplementary information on the real world.

3. Virtualizing/Reframing Twin Model

As a promising research pathway beyond the conventional digital twin, it is possible to consider embodying a digital replica by its reframing. By reframing digitized replicas, it is possible to extend/change the role of real-world things. Building and visualizing a digitized model using a conventional digital-twin can be considered a type of reframing, but the possibility of reframing should be reconsidered with the latest information-presentation devices, such as robotics displays and projection displays [23,24], and allow the traditional roles of the real-world things to be expanded.

The concept of entanglement was introduced by Orlikowski [25] and Leonardi [26] in a discussion about the notion of sociomateriality that helped analyze how technology use involves a close interplay between social and technical issues: the “entanglement of the social and the material—“a mangling of human and material agencies”, where the entanglement is related to the proposed perceptions and understandings. Sociomateriality builds on Barad’s agential realism, which discusses quantum entanglement based on Niels Bohr’s work, and where she emphasizes that a human observer cannot be separated from the observed phenomenon, but will be entangled with the observed object [10,11].

Within such phenomena, as shown in Figure 2, agential cuts enact local separations that define components. We can then assign these components to be the subject or object of a relation or as cause and effect. The cut is neither arbitrary nor given, and defines how we choose to separate the agencies of the observer and the observed. While interaction assumes separate things (including people) to be given and then looks at actions across this separation, intra-action acknowledges the fundamentally entangled nature of matter, including human bodies, with the environment [11]. The model offers a flexible view of human and material, where observable things in the world can be flexibly virtualized or reframed through agential cuts, as the reframed things appear in the real world according to the world’s diverse contexts.

We reinvestigate conventional digital-twins from the agential-realism perspective by introducing two patterns, as presented in the section. This section presents the two patterns: virtualizing patterns and reframing patterns, to reframe replicas digitized by virtualization as a conceptual-model to build innovative services by reframing.

3.1. An Overview of the Proposed Model

Figure 3 shows an overview of the V/R twin model. The orange circle on the left includes various things that exist in the real world, called real replicas; this group includes artificial things and naturally occurring things. By acquiring information about these real things, digitized things, called digitized replicas, are constructed in the virtual world on the right-hand side, where the virtual world in this study is represented as data on a computer.

We define four operator patterns called virtualizing patterns that offer a guideline for creating digitized replicas. The virtualizing patterns offer diverse opportunities to constructively define digital replicas, according to each situation. Each situation can choose its respective virtualizing patterns by choosing an appropriate pattern from the four virtualizing patterns. Each digital replica must contain a sufficient level of information to be used for building a digital service.

In typical ubiquitous-computing research, determining the state of digitized replicas is used to provide context-aware services [13], or a virtual world represented by 3D graphics, such as the Virtual Helsinki Project, is presented for remote users [21]. However, in the V/R twin model, the digitized replica created by virtualization is reframed to be embodied again as a reframed replica in the real world, as shown in Figure 3. Both real replicas and reframed replicas exist in the real world, but are represented as independent things. The reframed replicas are assigning different meanings from the real replicas; thus, the original real-world things have new roles in the real world. This process makes it possible to have a strong influence on a user’s behavior and cognition/thinking, by providing an outcome which is different from the original outcome. We define four form-patterns called reframing patterns that offer a guideline for creating reframed replicas from digitized replicas. Choosing an appropriate reframing pattern allows us to define a better visual form by reframing a real thing in the real world.

3.2. Virtualizing Patterns

The virtualizing pattern is the first component that constitutes the V/R twin model. This section describes the four operator-patterns that the V/R twin model provides for virtualization to create digitized replicas in the virtual world from real replicas in the real world. In the V/R twin model, digitized replicas even represent invisible things in the real world, and exist independently of human perception. Agential cuts decide how to observe digitized replicas from real replicas. Additionally, digitized replicas may represent fictional things that do not exist in the real world.

The first pattern, the datafication operator, is the most basic operator for data conversion in the real world. The properties related to things that exist in the real world are datafied, and represented as digitized replicas. These data are typically invisible, and in particular, information that cannot be identified by humans visually is the driving force for providing novel services in the virtual world. Typical examples are digitizing various pieces of information on routes within the city [27], or to digitizing the health-status of each individual [3].

The second pattern, the abstraction operator, is an operator that creates models of things that exist in the real world. This operator abstracts important aspects of things that exist in the real world. The abstraction operator makes it possible to systematically manage the behavior of things through computing rhetoric [28,29]. However, this abstraction strips away aspects that are less important from one perspective, which can be inconvenient if the aspects are essential from another perspective. Examples of the abstraction operator include creating meteorological models [30], modeling the perspectives of collective people [31], modeling human work [3], and creating various digitized services. The operator becomes the basic element when building diverse digital-services. Using modeling, it is possible to automate various processes in the real world and predict future situations using past data, in combination with a datafication operator.

The third pattern, the composite operator, is an operator that integrates the divided functions contained in various things so that they can be used as one function. The composite operator enables automation by integrating various functions. It is possible to extract only the functions of the abstract operator from what exists in the real world, and integrate those functions to create a new function. By embedding computation in daily life, it becomes possible to use the functions of various things existing in the real world through computing, and those functions can be effectively used to build advanced novel functions. For example, FedNet [32,33] makes it possible to integrate functions contained in everyday intelligent objects, and define them as one new service.

The final pattern, the decomposition operator, is an operator that divides and makes the functions of the digital-replica created available as a model. Real-world objects, particularly intelligent objects, contain various functions which work in a complex manner by embedding computation. By dividing those functions and making them available to other services, the possibility of building a wider variety of services increases. For example, in Sentient Artifact [32], it is possible to divide various functions of each everyday artificial object, and access them from information services. HAVi is also a standard platform for home appliances [34] but divides the multiple functions of home appliances and provides a standard API for accessing them.

3.3. Reframing Patterns

The reframing pattern is used to visualize a digitized replica in the real world, by creating a reframed replica from a digitized replica. For example, when visualizing the data acquired by the datafication operator, it is necessary to embody multiple reframed-replicas in the real world separately from the same original real-replicas. Additionally, to make a complex function appear as one function, it may be necessary to express it as its own physical form, to offer better affordances. Digital technologies have made it possible to present various situations in the real world with proper expressions, according to the progress of display technology, projection technology, robotic techniques, etc. This section describes the four form-patterns for reframing provided by the V/R twin model. In the V/R twin model, reframed-replicas embody their digitized replicas in the real world. The reframed-replicas offer the alternative affordances of the corresponding digitized-replicas, through alternative agential cuts. To consider reframed-replicas, the alien phenomenological angle becomes a philosophical basis to inspire the use of the reframing-pattern.

The first pattern is the morphing form. It is possible to dynamically combine robotic devices that deform flexibly with multiple devices, to provide complex functions. What is universally deformable offers the possibility of being transformed into various things. In particular, it is desirable that an appropriate form provides a better affordance for the functions it provides; thus, it is desirable for a computer as a general-purpose machine to provide the forms according to the functions it provides. As existing approaches for providing a deformable form, LineFORM and ChainFORM [23] are linear deformable devices that can express various pieces of information, provide tactile feedback, and transform into various objects. In Mobility as a Service (MaaS) [35], it is possible to combine multiple trucks so that they can be handled as one transport machine, and to flexibly change the combination, according to the situation. The morphing form is useful when treating a digitized replica that has been converted into data or a digitized replica that has been combined with functions, as a reframed-replica.

The second pattern is the embodied-form. The embodied-form is a pattern for dealing with things that are invisible in reality. Because information is also considered to be invisible in reality and does not exist in reality, the conventional information display can be considered to be the second pattern. With the advent of high-performance projectors and head-mounted displays (HMDs), it is becoming possible to make things that do not actually exist in the real world, appear. For example, embodying virtual people and buildings in HMDs is becoming common in augmented-reality services [36]. Information display on conventional displays is also classified into this pattern. Digital signage and large-scale vision, which are rapidly becoming popular in modern cities these days [37], are also considered to increase the new possibilities of embodied-forms.

The embodied-form can be roughly divided into two categories. The first is the case of projecting something that does not actually exist, into the real world [38,39], and most conventional information displays are classified as this case. For example, visually displaying information using graphs and visualizing device models using conventional digital-twins are considered to be typical cases. The other is the case of manipulating something that exists in the real world but is not explicitly visible, with gestures or voice [40]. Things that do not actually exist, such as virtual currencies, but which affect the real economy and things such as rules that define the constraints among real things, also have an impact on the real world using embodied-forms.

The third pattern is the layered-form. This pattern was widely used in early augmented-reality services to superimpose and visualize information about the real world, using a handheld computer or head-mounted display [36]. The difference between the layered-form and the embodied-form is that the embodied-form cannot distinguish between the one in the real world and the one made by the embodied-form, but when the layered-form is used, it is used to make a logical thing that does not exist in the real world, appear in the real world. This process is commonly used to superimpose and display textual information that explains the characteristics of things in the real world.

The last pattern is the embedded-form, which represents real-world information by what exists in the real world. For example, immersive projection of the virtual world onto the real world [41] and the presentation of information using ambient displays [24,42], are commonly used. Digitized replicas are rebuilt as a portion of what exists in the real world. In the embodied-form, the thing itself is made to appear by its reframing, but in the embedded-form, it is made to appear as a portion of the physical thing in the real world.

4. Using the Virtualizing/Reframing Twin Model

This section explores opportunities to use the V/R twin model from the following three perspectives. Section 4.1 presents two case-studies, to illustrate how the V/R twin model can be used to analyze existing services. The purpose of this subsection is to present some specific examples of using the V/R twin model. Section 4.2 describes opportunities to use the virtualizing and reframing patterns. In this subsection, we show how a design expert with years of experience in developing various prototypes of real-world services investigated opportunities to use the two patterns. Section 4.3 shows how the V/R twin model can be used to develop a novel conceptual-model for developing advanced smart-city services. This subsection also presents a use-case of this model and discusses further possible use-cases, showing further opportunities from a different angle.

4.1. Analyzing Existing Digital Services with the V/R Twin Model

4.1.1. Overview

Two case-studies are chosen to illustrate the use of the V/R twin model. The first case-study is the Google map service, which is widely used in the real world. Since this service is well known to many people, it provides a useful illustration for understanding the V/R twin model. The second case study is the CollectiveEyes service, a prototype real-world service based on new research. The case-study shows that using the V/R twin model is useful for demonstrating the new opportunities of the service.

4.1.2. Case-Study 1: Google Map

Google Maps [43] uses the abstraction operator to represent the real world as a digital map (Figure 4). In addition, it is possible to acquire information on shops and restaurants in the real world from the datafication operator, and manage it as digitized replicas created from the acquired information on a map. For example, diverse information such as public-transportation information and restaurant information can be added on the digital map. In addition, using the decomposition operator, it is possible to separate the navigation function on the map and use it as a new service integrated with various services.

Google Maps primarily uses the layered-form to create reframed-replicas. Various information is displayed in superimposition on a virtualized map or in a real-world view acquired by a camera. As a new possibility, by displaying a three-dimensional map using the morphing-form, information such as altitude differences can be expressed more easily [44]. In addition, using the embedded-form, it is possible to display the information that was conventionally displayed using the layered-form in a more ambient manner. In particular, the display of information using augmented reality increases risk when used while walking; thus, the display of ambient information in the real world is considered to be an interesting new enhancement [45,46].

Google Map virtualizes various things in the real world, and manages digitized replicas on a map in a virtual world. In the virtual world, the advantages of the digital world can be effectively used, such as easily finding what is required by searching and presenting the shortest path. In addition, by expressing the map in various representational formats, a user is provided with various viewpoints in the real world. This approach allows users to be aware of a variety of real-world things that were previously unnoticed. This advantage will increase as the number of digitized replicas that can be handled in the virtual world and the information about them increases.

4.1.3. Case-Study 2: CollectiveEyes

CollectiveEyes is a distributed platform to collectively collect and share human-eye views [31]. CollectiveEyes makes it possible to develop innovative services, allowing users to access someone else’s seeing capabilities As shown in Figure 5, CollectiveEyes considers the viewpoints of various people as real replicas, and creates digitized replicas by abstracting them using the abstraction operator and storing them in a distributed storage. Treating a portion of human functions as something that can be shared by separating the function from human visual and auditory senses can be considered virtualization using the decomposition operator. In addition, as shown in [47], it is an example of virtualization to use the decomposition operator to enable social watching by combining human viewpoints captured by CollectiveEyes with Citizen Science through Dancing.

CollectiveEyes uses the layered-form to reframe multiple human visual-perspectives. The viewpoints of various other people are superimposed and displayed on a user’s current viewpoint. It is also convenient to present a user’s current viewpoint while looking at the viewpoint of others. Therefore, the use of the embedded-form to show a user’s current viewpoint in the real world, as a portion of what he is currently seeing, is promising.

CollectiveEyes demonstrates the possibility of managing things in the real world that could not be treated as digital replicas in the past. Human vision and hearing are not typically considered to be real-world things but are caused by functions of humans. Making them available as digitized replicas offers a promising opportunity to offer new services. In particular, we think this process offers new opportunities to make it possible to share various things through digital technologies, including those that have not been considered to date.

4.1.4. Discussing More Opportunities

The above case-studies show the opportunities of utilizing the V/R twin model. Various data from the real world can be acquired through the datafication operator, and the data can be used in their respective functions, such as a navigation function. Currently, Google Maps can only search for the shortest route between two points, but by digitizing the amount of landscape and noise on a real-world route, it is possible to manage the search for quiet or beautiful routes by replacing the policies of the datafication operator, such as in Happy Maps [27]. This argument is an example of the advantage of the strong abstraction power of the virtualizing pattern. In this case, the datafication operator hides the details of the sensing mechanism for extracting information from the real world.

As shown in the previous subsection, both case-studies illustrate the power of the reframing-pattern for examining new possibilities of the services. The layered-pattern is already widely used in various existing real-world services, because the approach allows us to superimpose information easily in the real and virtual worlds. However, information shown through the layered-form can disrupt a user’s current perspective. The embedded- and morphing-forms represent information in a more ambient form by allowing information to blend into the real world. The embodied-form can also present information on public displays placed throughout urban cities and explore new possibilities of the services through ubiquitous-information projection. Thus, reframing-patterns are important for revealing new opportunities for existing services.

4.2. Opportunities Analysis of Virtualizing and Reframing-Patterns

In the previous subsection, we extracted several opportunities for the V/R twin model. In particular, we showed how the virtualizing- and reframing-patterns are used by two existing services. However, since the two services use only a portion of the patterns, we were able to partially demonstrate the opportunities of the patterns. In this subsection, we individually analyze the opportunities of each pattern in more detail. We analyze the opportunities by asking a design expert who has developed prototypes of various services, how more opportunities of the services can be extracted by investigating the virtualizing and reframing patterns. After presenting a methodology to extract the opportunities, we summarize the findings on the two patterns obtained from the expert. The findings present further opportunities from a different angle from the previous section.

4.2.1. Methodology

The methodology used in this subsection is based on the approach presented in [48]. The design expert, (male, aged 61), has had extensive experience in developing and analyzing various prototype systems for smart-home services, smart-city services, and crowdsourcing services for over 20 years. He also has experience in building a world modeling platform developed for the Sentient Computing Project [12]. Thus, he has very extensive experience in uncovering opportunities for virtualizing- and reframing-patterns, which could not be revealed in the previous section.

An overview of the V/R twin model and the case-studies presented in Section 4.1 were given to the designer, to deepen his knowledge of the V/R twin model. One author then conducted a focus-group interview, and discussed how investigating the virtualizing- and reframing-patterns could offer further opportunities for the prototype systems that he had developed in the past. From the interview, the findings of each pattern are summarized individually. The following section presents the findings revealed for each pattern and the arguments for further opportunities for the V/R twin model.

4.2.2. Virtualizing-Patterns

Datafication Operator: Various information is added to things that exist in the real world, by human recognition. In particular, the distinction made between various things and their attributes through the human-language function is basic information that can be used to recognize individual things. This information may be recognized by each person’s cognitive functions, such as sight and hearing, but it is difficult to recognize more detailed information. Therefore, by making it possible to acquire such information as an attribute of a digitized replica, it is possible to personalize the service. Past smart-home and smart- city research has increased the ability to digitize things, with the development of various sensing technologies [32,48,49,50]. The development of new sensing-technology will enable the acquisition of diverse information, to make the services more context-aware.

Abstraction Operator: By modeling things abstractly, it is possible to analyze various characteristics of things. For example, it is possible to discover signs of the occurrence of disability, extract the psychological characteristics of each person, grasp the usage status of things, and optimize them. The model can be used to judge various situations of things by simulations, using the current data. For example, it is possible to detect failures in production equipment at an early stage, and predict the trends in the near future of society, etc. Modeling is also a central issue in DX, and it is possible to study efficiency improvements through automation in the real world, through modeling various processes. Advanced digital platforms such as CollectiveEyes enable new services by modeling things such as human perspectives, which were not previously modeled accurately [31]. From this perspective, modeling what was not previously the target of modeling using the abstraction operator will likely produce new opportunities.

Decomposition Operator: Currently, computerized artificial objects generally provide various functions. For example, televisions have a receiving function and a display function, and automobiles provide a moving function and a sensing function to observe both the current position and surrounding conditions [49]. Making these functions available from various other services is effective in developing new services. For example, a car camera can be used as a camera view for city-security monitoring and observing tourist spots. Information technologies offer the possibility of utilizing the functions of various things in the world, not just the functions of computerized ones. For example, new opportunities may arise by allowing people to share human visual and auditory functions, movement functions, and lifting functions, separately.

Composition Operator: The composition operator offers new opportunities by integrating the functions of various digitized replicas and making them available as one function. In cloud computing, the number of computers used dynamically can be increased, according to the amount of calculation [51]. Because multiple computers appear as one service to the user, there is no need to consider the number of computers required. Similarly, by integrating multiple automobile-vehicles, it is possible to prepare a transportation vehicle that automatically adjusts the optimum number of vehicles, regardless of the transportation volume. Increasing the effectiveness of the composite operator depends on how new features can be provided by the decomposition operator. The important issue is to make more things in the real world available as digitized replicas.

4.2.3. Reframing-Patterns

Morphing-Form: The morphing-form creates a reframed replica from a digitized replica, using a generic real replica. The abstraction, decomposition and composition operator may provide real replicas with different roles, and may provide new forms offering alternate affordances to their functionalities. Architect Louis Sullivan argues that the “form follows function” paradigm [52], which became mainstream in the late 19th and early 20th centuries, requires that the form be changed as the function changes. Devices that change their forms using robotics functions and devices that create new forms by combining multiple forms are considered an effective basis for giving new forms to digitized replicas. For example, a toy such as Lego demonstrates that multiple basic blocks can construct various forms. In Minecraft, a player builds various three-dimensional virtual-world spaces, according to the concept [53]. Thus, the advancement of technologies that allow for more flexible forms raises new opportunities for the transformation of the real world.

Embodied-Form: The embodied-form offers a basic approach for expressing something that does not exist in the real world and is considered as the most important in extending conventional digital-twins. As with gamifying services, the provision of points and leaderboards can also be considered a type of embodied-form [54]. Using the embodied-form, it is possible to embody creatures and things that do not exist in reality as reframed replicas in the real world. This process also offers the possibility of an interaction method that goes beyond the physical constraints of reality, as if it actually exists. Similarly, the embodied-form may provide a variety of forms for the real world. For example, it is possible to consolidate multiple scenes in a 360-degree view or to consolidate various viewpoints, including nonhuman viewpoints, to display the scenes in front of a user as scenes that are shown all at once.

Layered-Form: The layered-form has been used in many augmented-reality services. In particular, it has been widely used to display information acquired by digitization in correspondence with a real replica. The layered-form superimposes and displays information in the real world, which may confuse a user’s perception. For example, as demonstrated in Hyper Reality [55], the use of the layered-form excessively increases the cognitive load of a user as the amount of information added increases. Therefore, it is desirable to use other patterns as much as possible, to minimize the use of the layered-form. However, in general, it is often difficult to embed information as a reframed replica in the real world, because the logical level of the superimposed information typically expressed by the layered-form is different from the information provided by the real replica, through the five human senses. The consideration of new approaches to provide better layered-forms offers the potential to provide a more desirable implementation of reframed replicas.

Embedded-Form: The embedded-form constructs the digitized replica as a reframed replica, using a real replica that is different from the original real replica of the digitized replica. Frames for viewing paintings and windows that display the scenery of the outside world of buildings are effective as places to express reframed replicas, and have been widely used in the study of ambient displays in the past [45]. In addition, ambient displays that express information through the movements of wind turbines and mobiles have also been studied [46]. The embedded-form does not cause information overload, because it can reconstruct digitized replicas of what already exists in the real world. However, conversely, it is often difficult for a user to understand the meanings of the represented information, because the form usually reflects the metaphors of the presented information; thus, the meanings are indirectly expressed. As discussed in the opportunities for the morphing-form, “form follows function”, as described above, is an important principle when considering the transformation of reality. When transforming and embodying various digitized replicas, the discussion around giving a suitable form for expressing good affordances, seems to be more important.

4.2.4. Discussing Opportunities of Virtualizing- and Reframing-Patterns

The opportunity analysis of the V/R twin model presented above reveals the roles of virtualizing- and reframing-patterns. Agential realism, as discussed in Section 3, asserts that the world is an entanglement of the social and the material. By abstracting the real world, the virtualizing-patterns untangle its entanglements into multiple digitized replicas, through abstracting the real world. In other words, by analyzing and understanding the entanglements of the real world, the operators of the virtualizing patterns can reveal unexplored opportunities for building advanced real-world services. From the perspective of agential realism, we may say that there are no real replicas, because the reality consists of the real world’s entanglement, and so the replicas are already virtualized from the entanglement. Therefore, there are many ways to investigate digitized replicas for reframing them, coming from the many possibilities of agential cuts.

The reframing-pattern provides a way to exploit new opportunities for representing digitized replicas. The layered-form is currently very popular for developing real-world services. For example, smartphones augment real-world images captured by their cameras, and extend the real world by superimposing information onto the images [36]. Other forms offer a variety of new opportunities for developing novel and innovative real-world services, as shown in Section 4.2.3. The embodied-form allows non-existing virtual objects to be incorporated into the real world. The embedded- and morphing-forms also allow for changing the meaning of real replicas. The power of the patterns allows the creation of a reframing replica, consisting of multiple real replicas. The opportunity will be exploited in the next subsection to introduce an enhanced conceptual-model for developing advanced smart-city services.

4.3. Developing Programmable City-Services Based on the V/R Twin Model

In this subsection, we explore new possibilities to build more advanced smart-city services, based on the results from analyzing the opportunities of the two patterns described in the previous section. As explained in Section 2, digital twins were initially proposed to improve the efficiency of the manufacturing industry, but recently digital twins have become widely used in the context of smart-cities, and promising developments are desired in the future [5,7,50,56].

4.3.1. Overview

From this discussion, to effectively use the V/R twin model, it is important to determine how various things and occurrences in the real world can be used as digitized replicas in the virtual world. In particular, a flexible form to offer the novel expression method of embodying a digitized replica as a reframed replica seems to be indispensable for the development of innovative digital-services using the V/R twin model.

In the following subsections, we propose the in situ programmable composite, a new conceptual-model extended from the original V/R twin model to develop smart-city services. This model was developed from the opportunity described in Section 4.2 to create a new reality in the real world by combining the virtualizing- and reframing-patterns. In this model, a new meaning of a reframed replicas is constructed by dynamically composing and decomposing real replicas, and advanced digital-technologies such as component-based technology, mixed-reality technology, and sensing technology are used to clarify the entanglements of the real world and to integrate these technologies to create advanced smart-city services. Therefore, we present a use-case of the model to demonstrate its power, and argument the possibilities of further use-cases.

4.3.2. In Situ Programmable-Composite

In future smart-cities, various micro-services will be embedded everywhere in the city. Micro-services are single-function services that automate the use of various things that exist in the city. A service actually used by a user is defined as a macro-service that is a composite of multiple micro-services. The in situ programmable-composite proposed in this study allows a macro-service to be spontaneously constructed from embedded micro-services in the city, according to a user’s current situation. In [49], the authors propose an infrastructure to automatically generate user interfaces to access micro-services that are available on demand, where a user currently is. The in situ programmable-composite is similar to the approach, and realizes actual service generation just by specifying the configuration of micro-services, without deploying new specific infrastructure. A programable configuration to use an interactive programming-paradigm similar to Live Coding [57] enabled by an in situ programmable-composite allows various service-combinations to be generated dynamically, so that a user can select one based on their preferences. Therefore, it is possible to spontaneously generate the necessary smart-city services, on the spot.

Figure 6 shows how multiple digitized replicas are treated as one digital-composite through the composite operator in an in situ programmable-composite. There are two digital-composites in this figure, but their combination can be programmable by a user, depending on his or her current situation. The combination pattern is compounded by programming micro-services expressed as a digital-composite, which can be used on the spot. For example, as shown in [58], predefined digital-composites can be used when similar micro-services are available, by programming the micro-service composition to be automatically changed, according to the current situation.

Additionally, as shown in Figure 6, a digital-composite can have a reframed composite that is one-to-one corresponding to a reframed replica embodied in the real world. In the example shown, there is a digital-composite consisting of three digitized replicas. Each digitized replica has a one-to-one relationship with a real replica. Each digital-composite defines its own reframed composite. A reframed composite may consist of multiple real replicas. The composite may be represented as a user interface or a projected virtual object that appears as a single service. The key point in this study is to ensure that a corresponding digital-composite can be manipulated. In particular, when a digital-composite is composed of multiple real replicas, it is important to provide a proper affordance that makes the collection of real replicas look like one integrated reframed replica.

4.3.3. An Actual Use-Case of In Situ Programmable-Composite

Finally, we show a use-case of the in situ programmable-composite that is created using the design-fiction method [59,60]. Figure 7 shows a conceptual diagram of an in situ programmable-composite called a multi-viewpoint aerial projection. It displays the multiple viewpoints stored in CollectiveEyes, by projecting the viewpoints through multiple drones. Each drone projects one viewpoint in the air. Multiple drones are arranged in a spherical shape around a user, and display various viewpoints in the 360-degree space around the user. The user holds a laser beam that emits blue and red in his or her hand, and uses it to express his or her preference for the viewpoint displayed by each drone. When the drone is irradiated with a blue laser, the viewpoint is preferred, but when it is red, the viewpoint is not preferred. The displayed viewpoint is dynamically changed by the user, who presents a preference for each displayed viewpoint. In addition, the distance between the user and the drone is adjusted by the user’s head gesture. As the distance between the user and the drone increases by shaking the head left and right, the diameter of the spherical projection displayed around the user also increases. In this case, the number of drones automatically increases, and the number of viewpoints displayed also increases.

CollectiveEyes creates the viewpoints of multiple people in a virtual world as digitized replicas, and the digitized replicas are constructed as a digital-composite, through the composite operator. In addition, digitized replicas of multiple drones are generated in the virtual world, and are constructed as another digital-composite. These two digital composites are then embodied as a reframed composite, to display multiple viewpoints as a spherical projection display formed by multiple drones, such as in [61], through the morphing-form.

The drone used for the multi-viewpoint aerial projection provides the flight service, display service, and camera service as micro-services. The flight service decides the position of each drone in the air for forming a spherical projection display. The display service draws one viewpoint captured by CollectiveEyes. The combination of the flight service and the display service is essential to enable the morphing-form to construct a reframed replica representing a spherical projection display. The camera service installed in the drone recognizes a user’s laser-beam interaction and their head gesture. Figure 8 shows an example of a simplified configuration. To consider the color of the laser-beam sent to the drone to indicate the preference of the viewpoint as described above, the configuration between a drone group and CollectiveEyes is specified.

In the practical configuration, it is necessary to show the relationship between the digital-composite and the reframed composite; thus, the real configuration tends to be more complex, and a method for easily programming the configuration spontaneously will be investigated in the future.

4.3.4. Investigating Alternative Use-Cases

The situ programmable-composite creates a single reframed replica from a configuration of multiple digitized replicas. This configuration is dynamically changed by replacing some of the digitized replicas, depending on the current situation. This means that since the correspondence between the real replica and the digitized replica is ambiguous, the correspondence represents real-world entanglement. This approach is an extension of the basic V/R-twin-model approach described in Section 3, to better fit the perspective of agential realism. In particular, in the approach, reframed replicas may not have corresponding real replicas, and the reframed replicas enable the real world to be significantly changed from the approach used in the basic V/R twin model.

We have enhanced the V/R twin model to provide more opportunities to investigate alternative use-cases. By adopting new advanced digital-technologies, the following alternative use-cases can be considered. The use-case presented in the previous section can be used in the entertainment sector. The alternative use-cases show that the conceptual mode described in this subsection can also be used in different sectors.

The first alternative use-case is to employ a technology called an ambient-persuasion display [62]. This display extracts various data from the real world by the datafication operator. The extracted data are composed as digitized replicas for representing the current situation in the real world. For example, the display can be applied to the field of health and sustainability by representing the sustainable situation in a city or the health-status of people living in a city on a virtual island.

A second alternative use-case is to employ a technology named a shape-shifting interface [23,63]. Digitized physical forms are now widely used in contemporary architecture [64]. Therefore, a conceptual-model enhanced to represent information as a guideline would provide new opportunities in the fields of architecture and urban design, in designing appropriate meanings for the physical forms that guide people’s actions and thoughts.

The final use-case is to employ a technology to create ambient music, which uses CollectiveEars as a new technology that extracts a variety of real-world sounds [65]. Using this technology, several innovative soundscape-services based on ambient music can be built. Thus, multiple real-world sounds are synthesized as digitized composites, which represent musical sounds in several recent soundscape-services. For example, Music Sonar [66] uses ambient music to represent the distance from objects around the user, in order to represent real-world information in sound. DESI (distributed embedded-sound information) [67] enables the possibility of making ambient music interactive. This use-case can be appropriate for designing soundscapes for public spaces (Brian Eno composed ambient music named Ambient 1: Music for Airports, which is used as the soundscape in an airport [68]. The ambient music asserts that the music is effective for designing the soundscape in public spaces).

5. Related Work

Real-world modeling has been actively discussed since the early days of ubiquitous- computing research. The Sentient Computing Project in the AT&T Laboratories Cambridge proposed real-world modeling using the object-based model [12], where the central concept in the model was the location. The Sentient Computing Project makes it possible to locate various things in the real world, using a highly accurate position-recognition device called Active Bat; thus, its services make it possible to change behaviors according to the current surrounding situation by retrieving the position of things from the real-world model.

Dey et al. suggested new possibilities for managing various pieces of real-world contextual information other than location, which is the case in the traditional real-world model, which have been discussed around location information [13]. Their research suggested that various pieces of real-world contextual information, such as users’ activities and property information of things, makes it possible to customize the behavior of services, depending on the current surrounding situation, through technologies which acquire various pieces of real-world information, using diverse sensing-devices. After their research, more studies of the acquisition of contextual information by analyzing the activities of people and things was conducted.

Digital twins have been developed against the background of advanced analytical-technology, using machine learning. By monitoring the situation of things in the real-world using sensing devices, it is possible to detect failures at an early stage and optimize the use of physical resources such as electric power and water [5]. The V/R twin model does not aim to build an accurate model of the real world, but uses the model as a basis for creating various new services. The virtualizing-pattern ensures that digitized replicas of the virtual world digitally extend various things that exist in the real world. Datalization and abstraction were also considered in traditional digital-twins, but decomposition and composition offer more novel opportunities to create new digitized replicas. The reframing-pattern is an aspect that is not explicitly considered in conventional real-world models and digital twins. By providing different aspects of real replicas as alternative expressions, it is possible to provide users with different perspectives on the real world, enabling people to change their behaviors and think creatively.

The proposed model is similar to the vision called “Mirrorworld”, reported by Keniichi Maeda [69]. He defined Mirrorworlds as alternative dimensions of reality that are layered over the physical world. He said, “Mirrorworlds immerse you without removing you from the space. You are still present, but on a different plane of reality. You will be able to see and engage with other people in your environment, walk around, sit down on a chair. However, you can also shoot fireballs, summon complex 3D models, or tear down your walls to look out on a Martian sunrise. Mirrorworlds recontextualize your space. They change its meaning and purpose, integrating with the proposed daily lives while radically increasing the possibilities for a space.” The direction suggests developing a guideline to use the V/R twin model more practically.

6. Conclusions: Future Opportunities and Challenges

In this paper, we proposed the V/R twin model that extends traditional digital-twin models. The V/R twin model is inspired by agential realism to ensure the “entanglement of the social and the material”, and the proposed observable-world consists of the social and material that are separately based on the current context. The proposed model contains four operator-patterns to create a digitized replica in the virtual world from a real replica in the real world, to make things virtual, and four form-patterns to make a reframed replica from a digitized replica to reframe things in the real world. As shown in this paper, the V/R twin model is effective for exploring new opportunities in existing services and also for investigating potential possibilities of new services.

Reframing digitized replicas has many possibilities other than those described in this paper. For example, persuasive ambient mirrors proposed in [62] reconstruct and display various situations in the real world in a more visually understandable form, to encourage a user to change his or her behavior. Reframing can be thought of as an effective means of altering the meaning of the real world and inducing people’s behavior. The morphing-form also provides the basic element for contextually changing automation processes in the real world. In conventional digital-twins, we only consider how to use the real data of the real world, but using the V/R twin model, considering how to provide the digital twins as services that provide value to users will become easier. In the future, we plan to study other methods of reframing. For example, the model may be used to represent various organisms, using biotechnology and nanotechnology [70].

The V/R twin model was inspired by agential realism to model the proposed real world at the beginning of the paper. Agential realism claims that things in the world are inseparable, but that things can be separated through agential cuts, according to their situations. In the V/R twin model, the virtualizing patterns realize agential cuts. The datafying operator and abstract operator separate from a vertical angle. Things are modeled as abstract virtual replicas in the virtual world. The composition and decomposition operators model things from a horizontal angle. The ambiguous boundaries among things are flexibly identified, according to their situations. Conversely, the reframing patterns offer the diverse implementation of the forms of things, according to their situation. Currently, the most important limitation of the current research is that there are no guidelines for using the patterns. To develop the guidelines, we must investigate the situations surrounding things. This process is very challenging in terms of agential realism, because the situations are also inseparable from the things. To solve the issue, empiricism is the view that all concepts originate in experience, that all concepts are about, or applicable to, things that can be experienced, or that all rationally acceptable beliefs or propositions are justifiable or knowable only through experience. For example, John Dewey’s basic thought, in accordance with empiricism, was that reality is determined by past experience [71]. The proposed modeling-framework is inspired by agential realism, but the proposed guidelines for using the framework can be based on empiricism, based on interpreting multiple case-studies in diverse contexts. Interpretivism, which focuses on understanding the subjective meanings that participants assign to a given phenomenon within a specific, unique context, will also help to develop the guidelines [72].