Assessment of 3D Models Placement Methods in Augmented Reality

Abstract

Augmented reality (AR) is a relevant technology, which has demonstrated to be efficient for several applications, especially in the architecture, engineering, construction and operation (AECO) domain, where the integration of building information modeling (BIM) and AR has proved to be optimal in handling construction projects. However, the main challenge when integrating a virtual 3D model in an AR environment is the lack of precision and accuracy of placement that can occur between the real and the virtual environments. Although methods for placement via AR have been reported in the literature, there is a lack of investigations addressing their evaluation. Therefore, this paper proposes a methodology to perform a quantitative and qualitative assessment of several AR placement methods and a discussion about their usability in the specific context of AECO. We adopt root mean square error (RMSE) to quantify the placement accuracy of a 3D model and standard deviation to examine its stability (jittering). The results revealed that the AR placement error range is extremely wide (from a few centimeters up to meters). In marker-based methods, the results showed centimeter-range in both indoor and outdoor environments, compared to other methods (Inertial, Marker-less, etc.), while marker-less methods have widely varying error range from centimeters to a few meters. Other commercial solutions based on placement-sensors (GNSS and IMU), such as Trimble SiteVision, have proven placement performance in manual mode with centimeter order, while for the automatic mode, the order of placement and stability is metric, due to the low coverage of RTX (real time extended) in the study area.

1. Introduction

Augmented reality (AR) is an immersive technology that allows overlapping information and computer-generated graphics to real-world images or models [1,2]. AR makes it possible to combine a real environment with computer-generated information, developing a three dimensional (3D) space in which generated computational elements are superimposed on the user’s real field of vision [3].

AR has demonstrated great potential in several domains. In particular, the AECO industry is increasingly integrating AR in construction projects, particularly for collaboration and communication purposes [4]. For example, visualization of the construction site with the planned model can improve the detection, processing, and reporting of progress discrepancies, as well as the efficiency of construction site operations [5,6], by providing relevant information for operation work [7]. In this context, building information modeling (BIM) and augmented reality are major innovations that address the main issues about the management of information throughout the life cycle of a construction project. BIM encompasses both 3D geometric representation and a semantic database, which allows early access to information; this fact together with the ubiquity of mobile technology like AR applications that facilitate real-time access to site information, shortens the gap between information availability and response times [8,9]. However, one of the most well-known issues in AR is the placement problem, which relates to the spatial misalignment of the real world and the AR environment. The inaccuracy of sensors on mobile platforms such as smartphones and tablets, calculating the characteristics of the camera position, and the placement method used are the major causes behind this [10].

For many applications where accuracy is critical, determining the degree of misalignment of the objects represented on the screen by calculating distances in the 3D world coordinate system is required. In this context, Ref. [11] presented a reproducible methodology for having the real distances between any real point and the lines of sight of its virtual projections, for any AR application. Ref. [12] proposed a method to assess the accuracy of the outdoor AR for underground utility mapping solutions. The main goal of this study was to create a mobile-based AR-geographic information system (GIS) for mapping and capturing underground utilities. A cloud-based system allows for the immediate sharing of data with other stakeholders. The authors concluded that smartphone accuracy is limited; alternatively, they used external global navigation satellite system (GNSS) devices to reduce positional errors.

Several AR placement methods have been proposed in the literature. Furthermore, many authors [12,13,14] have addressed the AR placement issue. However, we note a lack of studies assessing the existing approaches and methods of 3D model placement in an AR environment. This motivated us to conduct research in this field. The main contributions of our paper are fourfold:

• Propose a workflow for assessing placement methods of a 3D model in an AR environment;
• Study the influencing factors of some placement methods;
• Assess placement methods in terms of accuracy and stability;
• Discuss the applicability of the studied placement methods in the AECO domain.

To achieve these objectives, the reminder of this paper is organized as follows: Section 2 reviews AR placement methods and solutions proposed in the literature. Section 3 describes the assessment approach. Section 4 presents the experimental tests, followed by Section 5, in which we present all results obtained by the qualitative and quantitative assessment. Section 6, we give a discussion of the findings. The paper ends with a conclusion and shows future directions to be investigated in Section 7.

2. Background

The term “placement” or “placing a model” has not a common definition in the literature. This concept was used by some authors [15,16,17], who defined it as the positioning of a model to be enhanced on location. Several references [12,13,14,18] as well as Microsoft have used the word “Alignment” to describe how the augmented reality application “SnapToReality” performed during testing. The placement of virtual items in the real world is also referred to as “overlaying” [19,20]. In the present paper, the term “Placement” is adopted.

According to the literature, placement via AR is the superposition of virtual 3D models and elements at the appropriate, predetermined locations in the real world in a precise and constant way. As a result, a posture estimation method, or more specifically, a camera localization method, is correlated to the solution to this problem [20]. According to Marchand (2009), understanding the location of the camera in reference to the actual scene is necessary for properly positioning virtual items in relation to real-world objects.

The placement of AR information in the real world requires three qualities to achieve an efficient overlay between the virtual and the real environment: placement accuracy; real-time interactivity; and 3D tracking [18,21]. The first step is to establish a link between the physical and virtual worlds. Then, to augment virtual information on real objects, their location must first be identified in 3D space and in real time for the camera. This is known as 3D tracking, and it entails estimating camera poses with six degrees of freedom (6-DoF): three components for position and three components for orientation. 6-DoF refers to the object’s movement in 3D space along the X, Y, and Z axes, as well as rotation along the pitch, yaw and roll axis. As a result, virtual objects in the camera’s field of view can be identified by matching the camera’s position to a previously developed 3D model of the environment when the pose camera is estimated. After that, virtual and real-world objects are superimposed.

One of the most critical issues in AR is to ensure that the placement process is accurate, in real time, and stable [22]. This issue has been addressed by several authors. Ref. [19] studied placement methods by referencing two error types. The first one is the static faults, which include sensor measurement inaccuracies, mechanical misalignments between sensors, and incorrect registration algorithm. The second form of errors is dynamic; it includes time delays between the occurrence of an actual event and its arrival to the host, as well as timing delays. It should be also noted that hardware is critical to the AR system’s advancement because it may enhance the tracking, data storage, and placement of the virtual and real worlds [23].

Several approaches to the placement of virtual objects in a real environment have been proposed in the literature, including vision-based, hybrid, and sensor-based approaches [21,24]. Figure 1 below proposes a taxonomy of these approaches based on the adopted method and the requirements to prepare the environment before usage. Another classification distinguishes two families of methods: marker-based and marker-less methods [25]. In the next sub-sections below, we give an overview of the aforementioned methods by highlighting their advantages and limits as well as their influencing factors.

2.1. Overview of AR Placement Methods

As reported in Figure 1, we can distinguish three types of approaches: marker-based; marker-less and hybrid approaches. An overview of them is given in the next sub sections.

2.1.1. Marker-Based Approaches

Marker-based approaches use a designated marker to activate the AR experience. They can be divided into hyperlink and vision-based methods.

• Hyperlink methods

A hyperlink method connects physical objects to web-based material via graphic tags or automatic identification technologies, such as the radio frequency identification (RFID) system. There are two types of subgroups: direct and indirect URL discovery methods [26]. Direct methods use active emitters of identifiers, whereas indirect methods use passive devices to provide identifiers for consequent active sensors. For further details, we recommend the publications of [26,27].

• Vision-Based methods

A vision-based method is a field of computer vision that allows detecting markers on physical objects. In marker-based AR, a marker must activate an augmentation. Markers can be physical objects in the real world or paper-based patterns that are easily recognized and processed by cameras. Markers are visually independent of their surroundings. Additional data can be calculated, such as object orientation, color, size, and shape [28]. Computer vision-based recognition identifies objects by detecting features in an image. Scale-invariant feature transform (SIFT), speeded up robust features (SURF), oriented FAST, and rotated BRIEF are some of the most commonly used computer vision algorithms for object detection [29]. Important AR libraries in vision-based technologies include ARKit, ARcore, and Vuforia [24]. We note that although vision-based methods are more complicated than sensor-based approaches, they are accurate and reliable [30].

2.1.2. Marker-Less Approaches

Under marker-less approaches, we can cite sensor-based and vision-based methods. The later can be classified into model-based and no model-based methods. The camera pose is the key parameter to connect the real and virtual worlds in both approaches, which can be used for all applications, especially in the AECO and FM (facility management) domains.

• Sensor-based methods

Sensor technologies (e.g., IMU: Initial Measurement Unit and GNSS) are utilized in these methods to determine the camera’s location and orientation. A number of major commercial AR SDKs, including Wikitude, Vuforia, etc, supports sensor technologies for camera position estimation and tracking. In this case, inertial sensor tracking [31], acoustic tracking [32], and magnetic tracking [33] are the most usually utilized approaches in the literature.

• Vision-Based: Model-based methods

Most vision-based methods require a 3D model of the environment for camera pose estimation and tracking, which are called model-based tracking in computer vision [34]. Among these methods, edge-based tracking techniques require projecting a 3D geometric model (GIS or CAD, computer-aid design) on to an image and matching it with the picture’s corresponding edge attributes. The 3D camera motion between frames is then computed using 2D displacement of matching characteristics [35]. The second one refers to interest-point-based or point feature methods. The core idea behind feature-point-based approaches is to extract feature points from a database of pictures and retain their positions as well as display descriptions during an offline training stage. For the template-matching, these methods use texture information in images to estimate the camera pose, but unlike interest-point-based methods, which use features, they only consider a small portion of an image, referred to as a template, to match reference images stored in an image database and a query image in the current frame of the camera at the off process [36,37,38]. Finally, the use of depth pictures encoding the distance of scene objects from the camera view as a pixel value is one of the most recent ways for computing the camera posture. When these depth photos and RGB images are merged, it is possible to estimate camera posture for tracking [39,40].

• Vision-Based: No-Model-based methods

No-model-based methods work by tracking and registering the camera phone without the need for a model or database. Such methods follow the camera’s movement while concurrently constructing a 3D structure of the picture scene [41]. SFM (structure-from-motion) and SLAM (simultaneous localization and mapping) are the two main techniques for estimating the camera’s pose in the AR scene. The SLAM method is developed for use in unknown environments, while the SFM method is used for known environments. We refer the reader to [42,43] for more details about these concepts.

Other challenges in vision-based approaches in monitoring systems exist in addition to the common problems associated with precision and operational time. One disadvantage of some vision-based approaches is their initialization stage. Many of these algorithms require manual [44] or semi-automated [45] initialization. Even if they start up automatically, they should generally start from a known point [46]. Furthermore, when a tracking pane is exposed to a fast movement or a dynamic occlusion, it is necessary to re-initialize it.

2.1.3. Hybrid Placement Approaches

Hybrid approaches try to make a compromise by combining different methods, so overcoming their weak points and challenges. It not only delivers convincingly precise and robust results to AR applications, but it also reduces computational complexity. We cite, as an example, the study carried out by [47] which uses a hybrid approach (marker based and sensor based) to create virtual visits to underwater cultural heritage sites.

2.2. Influencing Factors of Placement Methods

In an AR experience, it is essential to place virtual objects consistently in the real world while insuring stability. Jittering refers to the phenomenon where the augmented model is unstable in the scene and oscillates at high frequencies and small amplitudes [48]. This might be due to the availability of placement points [49]. Furthermore, factors such as weather variations, sunlight, and shadows, which can change lighting, are major problems in outdoor applications [50]. This weakens vision-based placement techniques that rely on image intensity information (pattern matching, points of interest, and pattern matching). The virtual content cannot be increased in the case of marker placement if the markers are blocked by other objects in the environment [51].

Additionally, there are a various source of errors in sensor-based techniques that can lead to a low level of placement accuracy, particularly in sanitary and environmental applications. Although these sensors are calibrated before use, accuracy problems are inevitable. The unavoidable white noise in the gyroscope data causes some rotational angle drift. Over time, this noise builds up and leads to inaccurate placement outcomes [52].

2.3. AR Placement Methods: Related Works

Several developments in AR have been carried out with or without markers (sensor-based or computer vision) to address and evaluate the challenges of an AR system in terms of visualization, portability, placement, etc.

Ref. [53] defined the nature and sensitivity of the errors that cause misregistration in AR displays (in the case of a head-mounted display). They consist of system delay (latency), tracker error, calibration error, optical distortion, model misalignment, etc. However, neither on the screen nor in the world coordinate systems does this research provide a model for estimating the overall error of the AR system.

Ref. [54] studied the camera’s theoretical pinhole model’s ability to accurately represent virtual objects on the screen. The authors measured pixel errors on the screen to assess the impact of the camera in the AR context but did not evaluate the variances of their representation in the real scene in world coordinates (real scale distances).

Ref. [20] addressed the placement problem by showing that the alignment of virtual objects with the real world can be performed by aligning the real and virtual cameras. To obtain a coherent augmented world combining both virtuality and reality, it is necessary to attribute to the virtual camera the same properties (extrinsic and intrinsic) as those of the real camera and to determine in real time for each image the position and the orientation of the camera in the real scene.

An overview of AR application in underground constructions was presented by [55] who addressed both aspects: 3D modeling and AR placement or alignment. The purpose of this study was to give a comprehensive overview of the literature on the application of AR in the building field. The authors identified and examined the challenges, as well as the technical approaches, to solve fundamental barriers to technology adoption in the underground construction industry. They emphasized virtual object placement and alignment issues, as well as related errors. In the same trend, Ref. [19] investigated the placement accuracy of an AR system for underground utility mapping in an outdoor environment. They tested if a smartphone linked to an external GNSS receiver might help to improve and assess horizontal positional inaccuracies in collected data. The data were collected using a phone app called “AR XR-GIS”. The study evaluated four devices with 16 location points using root-mean-square error (RMSE), mean Euclidean error (MEE), and central error (CE). However, a study conducted by Zhang et al. (2018) has shown a smartphone positioning accuracy of around 0.80 m to 1.4 m. Thus, existing smartphones cannot be used in underground construction without the addition of higher-accuracy GNSS devices [56].

Ref. [13] proposed a methodology to evaluate AR placement errors. The main objective of their study was to present a reproducible methodology for calculating the real distances between any real point and the lines of sight of its virtual projections to obtain a quantitative evaluation of these superimposed deviations for any AR application. The authors indicated that there are several possible sources of errors, which do not allow obtaining a perfect superposition of the virtual models on their corresponding real entities. The results were synthesized according to the factors of influence (the AR Scene, geolocation, orientation and camera alteration), the methodology of contrast and evaluation, the partial precision and the corrective actions. For the case of geolocation (placement by GNSS), it was mentioned that it is possible to obtain precise results in X and Y coordinates which do not affect the general precision of the system when the application is not used over very short distances. This was indicated by moving the external GNSS receiver up to 5 cm and checking that the overlay was the same. However, an accuracy of 5 cm horizontal and 10 cm vertical might not be accurate enough to apply AR technologies over short distances or to identify small features on-site.

Ref. [57] presented improvement approaches to increase the accuracy of camera pose estimation for accurate placement. Since the inertial sensors in today’s smartphones are so full of noise, the accuracy of an inertial tracker is often lower than that of a visual tracker. As a result, combining the two results usually only yields a minor improvement in accuracy. As a goal, the authors proposed two different models of visual measurements for use in Kalman-based filters that also integrate inertial/magnetic measurements to estimate tracking and placement by a handheld IMU/camera sensor unit.

Ref. [58] worked on BIM-AR in site inspection using smart glasses. Although the developed BIM-AR system is capable of achieving 1 cm accuracy, this research argues that a more reliable and stable tracking system that could consistently maintain sub-1 cm accuracy would support the adoption of the BIM-AR system.

Ref. [7] used the HoloLens (head-mounted MR device) to develop an AR-based inspection tool that place BIM models onto a physical construction site. The authors used an AR-based interface, allowing inspectors to check off construction elements within a virtual holographic checklist. This study developed a construction inspection system by combining BIM and Mixed Reality (MR), which allows the user of the HoloLens to display the BIM model on the construction project site with the exact size.

Ref. [5] examined the various placement and tracking technologies for AR and concluded that GPS/GIS and fiducial markers are the most often used methods thanks to their low cost and ease of use. However, environmental conditions, such as temperature and sunshine, affect their performance. As a result, they emphasized the need to improve the accuracy and manage the occlusion of AR devices used on construction sites.

According to the state of the art, there have been several research studies addressing the issue of placing 3D models using AR. However, there is currently limited research studying and assessing the various methods and solutions for this placement for indoor and outdoor environments. In this paper, we propose a quantitative and qualitative assessment of several AR placement methods of a 3D model. Our contribution also provides a discussion about the requirements in terms of placement accuracy and stability in some use cases of the AECO domain and so analyzing the fitness of use of each studied placement method.

3. Methodology

In order to compare the accuracy of each placement method, we determined local coordinate deviations between real feature points on site and their correspondents in AR environment, which is expressed by root mean square error parameter (RMSE). The measurement is repeated four times for each characteristic point to deduce the standard deviation of each method, which represents the precision (or the stability of the model in the real environment). We note that precision is a quantification of how close replicated measurements are close to each other, while accuracy is a quantification of how close the measurements are to the true value. In the same stream of research conducted by [11,17], our approach is based on a mathematical evaluation of the placement accuracy and stability of the 3D model in the real environment (Figure 2).

3.1. 3D Modeling

Our assessment approach was tested on the “Capacity Building Center” (designated later by CBC) at IAV Hassan II (Figure 3). A 3D Scan was conducted to generate a 3D model of the building, which is located in an open site to allow testing different AR placement methods, especially sensor-based ones. We used a terrestrial laser scanner for acquiring 3D point clouds of the building and performed the processes of registration, pre-processing and 3D modeling within Revit software (Figure 4). The resulting 3D model is presented in Figure 5.

The 3D survey was performed in two stages: scan of the building outside facades and then its interior. The two 3D scans were aligned using the “Leica Cyclone REGISTER 360” software. We performed manual pre-alignment by matching scans and placing them with approximate relative position and orientation for the best results, followed by an automatic alignment (registration). Finally, we proceeded to scan cleaning by removing noises from the point clouds (due to reflective surfaces, such as windows). After that, the data were exported in two formats: LGS (layered gene scanning) and RCS (random constant scanning) and then imported in Revit for generating the 3D model.

3.2. Development of AR Applications

We implemented our AR applications based on each placement method after creating the 3D model for the site survey. The game engine employed in this study is Unity 3D. It was chosen thanks to its widespread adoption in the AR community and readily available documentation. This section outlines the methodology adopted for creating AR applications for each of the studied placement methods. (Figure 6)

• AR for a sensor-based method

For the development of a sensor-based application (inertial based or location based), we adopted the “AR+GPS Location functionality”. This solution (used under a license) is the most used by the AR community for the development of location-based applications.

• AR for a vision-based application: Model-based

We used the “Model Target Generator” software of Vuforia to prepare and generate the model to be recognized in the field by the smartphone camera in an AR session. The algorithm extracts the corners of the 3D model of the CBC in order to recognize it and overlay it on the ground in the AR scene. The model is then imported into Unity after adding a model target. The last step is to add the model to augment in our scene.

• AR for a vision-based application: No model-based

Once the 3D model is imported into Unity after adding a model target, a basic AR scene is created after installing the required extensions. Three modules are added to the application to make it compatible with Android: “AR Session”, “AR Session Origin” and “AR Camera”.

We proceeded by adding the “AR Plane Manager” component at the level of the “AR Session Origin” to activate the plan detection. Then, we used a script to place the 3D model using the “AR Raycast” function. This script takes the position of object placement, and then the 2D position of the screen (position of the click) is associated with an invisible ray perpendicular to the screen, which intersects a detected plane. Finally, the model appears at this location. To allow rotation and scaling of our model following on-screen interactions, we used the “LeanTouch” asset of Unity.

• AR for a marker-based application using Vuforia SDK and AR Foundation

For Vuforia, after activating a license and importing the SDK on Unity, we downloaded markers and added them as a package format (UnitEditor), which will be imported to the database on Unity. The Vuforia website gives a rating out of five for each added marker. This score depends on the quality of the marker in terms of the number of its characteristic points (feature points) and its resolution. The size of the marker must be well defined at the beginning as well as its orientation.

For the AR Foundation, we start by installing the necessary extensions and adding the components of an AR Foundation scene (AR Session, AR Session Origin, and AR Camera), and then create a basic AR scene. Subsequently, we add the “AR tracked image manager” module, which creates “GameObjects” for each image detected in the environment. Before an image can be detected, the handler must be instructed to find a set of compiled reference images in a reference image library. The next step is to add the markers on a “reference image library”. Then we attach the library of markers as well as our 3D model with the tracked image manager. At this level, we add another scriptable object “tracked image information manager». The mains roles of this script are, firstly, to manage the spatial position of the phone camera in the real environment, and the textures during the AR scene, as well as updating the information relating to the images tracked and finally the object management scale to increase. Finally, we load our application in the Android system (.Apk).

• Commercial solutions

For commercial solutions, we used a license of Gamma AR and Trimble SiteVision with two placement modes. The first is manual, being based on overlaying the 3D model manually on its actual location, and the second is automatic placement, which attaches the georeferenced 3D model to its actual position. We note that the SiteVision system is equipped with a GNSS sensor and an IMU and is based on the functionality of the ARcore.

3.3. Tests of AR Placement Methods

In this research, the study of the performance of AR placement methods includes both indoor and outdoor environments since some methods are suitable only for the outdoor, such as the inertial sensor-based method (GNSS Smartphone) and Trimble SiteVision (for manual and automatic placement by GNSS). The methods assessed in our study are reported in Figure 7.

Among marker-based methods, we tested two free solutions; the first one is a framework API called AR Foundation which brings together two software development kits (SDKs) (ARcore for Android and ARKit for IOS) and the second is Vuforia SDK. We chose these two applications because they are most popular for the development of AR applications. We note that Vuforia SDK is the most used SDK by the AR community thanks to its simplicity in developing AR applications (does not require programming knowledge) and its free trials.

Within marker-less methods, we consider two sub-classes of methods, namely sensor-based methods and vision-based methods. We used the edge-based method available on Vuforia (object recognition) for model-based methods, and the VSLAM (visual SLAM) technology of the ARcore SDK for the no-model-based approach, particularly for the approach based on plane detection.

To properly carry out a comparative study between the different methods of placement of 3D models by AR, we used optimal test conditions for each method, especially the marker-based methods. High-quality markers (rating) with a good distribution on site were used to correctly detect the markers and to avoid the influence of the distance from them.

3.4. Assessment Metrics

Spatial data precision and accuracy can be quantified using two common measures: RMSE and standard deviation [12,59]. RMSE is the square root of the average of the set of squared differences between the data set coordinate values and the coordinate values from the location checkpoints. Given the ($x_{data,i}$;$y_{data,i}$) locational coordinates and the ($x_{base,i}$; $y_{base,i}$) base coordinates of n points; RMSE is defined to be Equation (1).

$$\begin{gathered} RMS{E}_r=\sqrt{{\left(RMS{E}_x\right)}^2+{\left(RMS{E}_y\right)}^2+{\left(RMS{E}_z\right)}^2} \\ RMS{E}_x=\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{(x_{data,i}-x_{base,i})}^2} ; RMS{E}_y=\sqrt{\frac{1}{n}{{\displaystyle \sum }}_{i=1}^n{(y_{data,i}-y_{base,i})}^2} \\ RMS{E}_z=\sqrt{\frac{1}{n}{\displaystyle \sum }_{i=1}^n{(z_{data,i}-z_{base,i})}^2} \end{gathered}$$

(1)

The following relation Equation (2) expresses the standard deviation:

$$\sigma =\sqrt{\frac{1}{n-1}{\displaystyle \sum }_{i=1}^n{(x_i-\overline{\mathrm{x}})}^2}$$

(2)

4. Experimental Tests

In our study, we performed two types of tests about the placement of a 3D model via AR; the first one is for marker-based methods, and the second is for marker-less methods. We first studied how the differences between real and virtual models can be reduced and virtual object stability can be improved by addressing influencing factors for each method, such as the indoor and outdoor environment, the AR SDKs used, the marker type and layout, the size of the model, etc. (Section 4.2).

4.1. Tests Method

To evaluate the accuracy and stability of placement methods, we proceed by testing each application on site and measuring differences of local coordinates between the real (d0) and the virtual environments (d1) by a topographic method (Figure 8a). The accuracy (e) and stability of placement were quantified using RMSE and deviation standards based on metric measurements of 12 quantifiable and significant points on site, such as the corners of the building, stairs, window frames, doors, corners of beams, etc., (indoors and outdoors) by using a robotic total station equipped with a laser system. The process is illustrated in Figure 8b.

The tests were performed in the same conditions (

Table 1. Test parameters.

Number of reference points	24 points (12 indoor and 12 outdoor)
Mobile phone	Mid-range phone (“Huawei P20 Pro”)
Operator	The same operator for topographic surveying of virtual and real points.
Lighting Conditions	Same lighting conditions (in the morning for all tests)

).

For the marker-based techniques, we used several types of markers in terms of forms, colors and the number of characteristic points in the form (QR codes, printed photographs, images by a camera phone, etc.) (Figure 9). We should notice that the markers are kept as AR scene triggers, which implies that they must be detectable in the real world. Figure 9 presents their distribution for outdoor testing (Figure 10).

4.2. Influencing Factors of AR Placement Methods

In an AR scene, the quality of placement is dependent on many factors that were tested using both marker-based and marker-less approaches.

• Marker-based methods

Several scientific works have studied the factors influencing the AR placement in marker-based methods. They identified the following factors (Figure 11): the quality of markers, the maturity of the AR SDKs, the complexity of the virtual models [60], as well as the distance of the virtual object from the marker, the size and the spatial disposition of markers [13]. Practical tests on site were conducted to determine the extent to which these factors influence the AR placement of a 3D model in terms of accuracy and model stability.

• Marker-less methods

Marker-less placement methods are affected by a single factor, which is the quality of localization [60]. Thus, developers can use better location technology and better hardware to improve the stability and placement accuracy of the virtual model. In our case, we used the Trimble SiteVision system, which improves the accuracy value obtained by the method based on the inertial sensors of the smartphone.

5. Results

In this section, we present and analyze the results of the different tests performed for a comparative study between AR placement methods. First, we study the influencing factors of placement methods, and then we compare their quality. As explained in previous sections, the results of our tests are discussed with regard to two parameters: placement accuracy and stability of the model.

5.1. The Influencing Factors of Marker-Based Methods

5.1.1. Distance to Marker

The purpose of this test is to study the relationship between placement accuracy (RMSE) and distance from the marker.

For marker-based methods, tests using the Vuforia SDK show that the further away from the marker, the placement accuracy of the model deteriorates (Figure 12). At a distance of less than 2 m between the physical marker and the device, the placement error is minimal (millimetric order). Figure 13 reports that the RMSE is significantly augmenting with the distance to marker (reaching 17 cm for a distance of 6m from the marker and 20 cm for a distance of more than 8 m).

The distance effect is generally negligible when using the ARcore SDK, which is characterized by the anchor functionality in the augmented model (Figure 14).

According to the diagram above, we notice that the effect of the distance to marker follows a linear function, which can be useful for error prediction with this method.

5.1.2. Complexity of the Model

To study the degree of influence of this factor, we developed two marker-based AR applications. The first application superimposes in AR a single entity of the 3D model (e.g., beams) (Figure 15a), while the second one superimposes in AR a complex object (the whole building) (Figure 15b).

After the quantification of placement deviations, we conclude that the placement accuracy degrades up to 1.5 cm with the complexity of the model (the whole model or an entity of the model) for the two SDKs (Vuforia and ARcore). This degradation remains relatively weak. This confirms the results of the study of [61], which states that the complexity of the augmented model has little effect on the accuracy of the model placement.

5.1.3. The SDK Used: ARcore and Vuforia

Compared to the Vuforia SDK, the ARcore SDK achieves better placement accuracy both indoors and outdoors. This is justified by the fact that ARcore’s algorithms are more reliable than Vuforia’s. However, Vuforia SDK is used by a large AR community and does not require advanced programing skills. This is why we adopted it for performing tests.

The choice of the appropriate SDK depends on the context and users’ requirements in terms of precision and accuracy. Indeed, AR can be recommended for contexts where the quality of placement is required for decision making (work inspection), while Vuforia can be a good alternative in situations where placement is realized for communication purposes (i.e., concertation on site)

5.2. Placement Accuracy

To examine placement accuracy, we chose two indicators that are commonly used in comparison tests; both indicators consider RMSE and standard deviation. We consider four successive measurements for each point during the measurement process in order to calculate the standard deviation (stability) and average and compare them with the actual measurement to obtain the deviation indicating accuracy.

The results in terms of accuracy of AR placement methods are reported in

Table 2. The accuracy of placement (RMSE) for the studied methods.

	Placement Method	RMSE (m)
		Indoor	Outdoor
Marker-based methods	ARcore	0.09	0.16
	Vuforia	0.13	0.20
Marker-less methods	SiteVision (Manuel placement)	0.07	0.07
	GAMMA AR	0.08	0.08
	Marker-less (Model-based)	0.09	0.11
	Marker-less (No model-based)	0.19	0.4
	SiteVision (Automatic placement)	-	3.47

and discussed in the following paragraphs.

• Marker-based methods

Marker-based placement methods can be used to position small virtual objects, such as a beam, with a negligible placement error [60]. According to Table 2, the ARcore method allows achieving an accuracy of 9 cm indoors and 16 cm outdoors. These values are degraded when switching to Vuforia SDK with an accuracy of 13 cm indoors and 20 cm outdoors. Furthermore, we notice that both methods are affected by the change from the indoor environment to the outdoor; this is mainly caused by the influence of the exterior environment and lighting conditions, which can considerably influence the correct detection of markers (Figure 15). Figure 16 illustrates the results obtained by marker-based methods.

It can be concluded that marker-based methods are generally accurate in an indoor environment; however, this accuracy of placement may deteriorate with distance from the marker, particularly in the case of Vuforia, as it has been reported in Figure 12.

• Marker-less methods

Marker-less methods are generally insensitive to the environmental conditions, except for methods not based on a 3D model, where there is a significant difference due to the quality of the manual adjustment method of the model after a first placement based on the detection of the planes by the visual SLAM (V-SLAM) method.

In an indoor environment, marker-less model-based methods give an offset value of 9 cm between the virtual model and the reality. This value deteriorates considerably up to 19 cm for the method not based on a prior 3D model of the building. The accuracy deteriorates for both methods outdoors, reaching 11 cm for model-based methods and 40 cm for no model-based methods.

The “GAMMA AR” method offers a simple and intuitive manual adjustment of the model which gives the same degree of accuracy for every AR experience. However, this solution is commercial and remains inaccessible to all users (Figure 17a).

For SiteVision (Figure 17b,c), we used two methods of placement, namely manual placement and automatic placement. We tested the manual placement in both indoor and outdoor environments which gave the same results (7 cm); this is justified by the fact that the operator readjusts the virtual model without any prior knowledge of the physical environment. For automatic placement, we initiated this mode only outdoors (since it is GNSS based) and we obtained an accuracy of around 3 m. This is explained by the low coverage of the RTX network at the study site.

To summarize, we can conclude that in an indoor environment, “SiteVision (in Manuel placement), GAMMA AR, ARcore, and marker-less methods (model-based) reach an accuracy of less than 10 cm, which degrades to 19 cm using Vuforia or marker-based (no model-based) methods. The two commercial solutions “SiteVision” with its manual placement mode and “GAMMA AR” come out on top as being the most accurate placement methods. This degree of accuracy is explained by the fact that these two methods use quality manual placement to match the virtual model to reality. The ARcore marker-based method is more accurate than the other methods (8 cm).

In outdoor environment, we note three classes of RMSE values. A first class (RMSE less than 10 cm) is obtained by commercial solutions: “SiteVision” with its manual placement mode and “GAMMA AR “. The degree of accuracy achieved by these two methods is explained by the use of an exact manual adjustment like the indoor. The second class of RMSE of 10 cm to 40 cm is obtained from marker-based methods (ARcore and Vuforia) and marker-less methods (based or not on a 3D model). A third class with RMSE less than 10 cm combines two methods: SiteVision with its automatic placement mode and inertial sensors based on consecutive RMSE values of 3.47m and 7.67m. The placement accuracy of 7.67 m obtained by the inertial sensor-based method is due to the low nominal accuracy of the smartphone’s inertial sensors. This RMSE value is somewhat improved with the use of SiteVision with its automatic placement mode. An RMSE of around 3 m with automatic placement by SiteVision is explained by using a medium-sized positioning receiver (Catalyst Trimble) and the low coverage of the RTX network (Figure 18).

According to the results reported in Table 2 and Figure 18, we can conclude that the knowledge of the environment is a determining factor for the accuracy of placement of marker-less methods because computer vision algorithms in this category are easily able to accurately overlay and match pre-existing 3D models with reality.

5.3. Placement Stability (Jittering)

In addition to placement accuracy, the stability of the augmented model is an important qualitative aspect in choosing which placement method to use in a given context.

Table 3. The stability of the AR placement methods.

	Placement Method	σSample (cm)
Marker-based methods	Marker-based: ARcore	2
	Marker-based: Vuforia	7
Marker-less methods	SiteVision (Manual placement)	3
	Marker-less: No Model-based	3
	Marker-less: Model-based	4
	GAMMA AR solution	4
	Inertial-based	6
	SiteVision (Automatic placement)	130

reports the obtained results for the studied methods.

By analyzing the results of Table 3, we notice that the marker-based method using the Vuforia SDK is the most unstable among the other remaining methods; this effect is due to the poor performance of the algorithm in detecting and tracking markers in the real scene. Moreover, the alignment performance in the case of Vuforia depends on the lighting and brightness conditions, the quality of the markers and the camera-to-marker distance (explained in Section 5.1.1). However, the ARcore method, which is based on the anchor method, allows a greater stability than Vuforia. We can so conclude that the SDK used affects the model’s stability. Thus, to develop a stable marker-based AR application, the user needs to choose the best SDK [61].

With the automatic placement mode of SiteVision, an unstable increase in the model is obtained with a standard deviation value of 130 cm. This degree of instability is explained by the low coverage of the RTX network at the study site. This is justified by the fact that the stability of marker-less methods depends on the quality of the positioning method [61]. Figure 19 below summarizes the different results of both marker-based and marker-less methods.

6. Discussion

6.1. Assessment of AR Placement Methods

The present evaluation approach aimed to assess the placement performance of a 3D model using several AR methods and solutions. Several approaches have been tested, discussed and assessed based on two criteria: placement accuracy and model stability. Furthermore, several influencing factors have been studied to identify the optimal conditions to be respected on site.

In case of marker-based methods, the AR scene is influenced by the quality of the markers, the maturity of the AR SDKs and the complexity of the virtual models [60] as well as the distance from the virtual object to the marker, the size and the spatial arrangement of the markers [13]. The results of our tests for the three studied factors (distance to the marker, the SDK and the complexity of the model) have shown that marker-based methods require a preliminary study before deploying the solution, taking into consideration the expected objectives. The accuracy and stability of placement and the time required for development will be the matter of a pre-study based on the requirements. For the case of the marker-based method by Vuforia SDK, this technique may not be usable on site because in order for the model to be augmented and stable, the camera must be close to the markers at all times, which is difficult in the case of a large construction with occlusions.

Tests show that the stability of marker-based AR solutions is relatively low, which is significantly influenced by the quality of the markers and the maturity of the AR SDKs. Thus, to develop a stable marker-based AR application, we need to select an appropriate AR SDK, create high-quality markers, and even optimize existing feature recognition algorithms as needed; this was confirmed by the study of [62].

In the case of marker-less methods, the tests demonstrated that there is a clear distinction between the various methods and solutions used. The algorithm developed is the main difference between these approaches. The ARcore SDK case has demonstrated perfect 3D model stability and placement accuracy. This is the case with the Site-Vision and GAMMA AR solution, which has shown good results in terms of model stability and placement accuracy. However, due to the RTX network’s poor coverage in the study site, the automatic placement mode generates metric stability and accuracy results. It can be stated that marker-less methods remain an unacceptable choice in the construction and 3D model application. The strength of these methods lies in the development of algorithms without limits of applications suitable for the specificities of the construction. Indeed, these methods are not constrained by the presence of elements exogenous to the construction (markers and the accuracy of its location in the real environment).

Consistent with previous studies findings, developers of marker-less AR-3D model applications can either use better localization technology and better hardware to improve stability and accuracy (using Trimble SiteVision) or use vision-based methods with powerful algorithms.

Finally, our research brings an interesting contribution in the context of AR placement methods. It fills in a gap in the literature by proposing an assessment approach for state-art placement methods in AR. Such an evaluation gives a good basis both for the scientific community and the professionals for guiding the choice of the adequate AR placement method for a given context. This last contribution is discussed in the next section.

6.2. Applications in AEC

The results of our study show several placement reliability values in terms of accuracy and stability that should be judged with respect to the context of use. Indeed, the fitness of use of each placement method should be identified to guide the user in choosing the appropriate method. In the following, we highlight the applicability of each method in AECO projects. We note that the need for precise placement depends on several factors: the complexity of the project, its phase (from design to operation), the difficulty of the operation (risk and decision-making), etc.

• Planning and design

In the planning phase of construction, it is crucial to ensure that the project plan aligns with the client’s requirements [63]. By using AR in conjunction with a 3D model, architects and construction companies can truly present working models to their clients before construction begins. For this purpose, high accuracy of the placement of a building model is not required; an error range of a few meters remains tolerable. Marker-less methods or other solutions, such as the SiteVision system, can be used in this context.

• Site supervision

Precise placement in site tracking ensures that virtual objects are properly aligned with the physical world. Marker-based methods are often used for AR applications inside buildings [64] (centimeter accuracy). By equipping the site with beacons, the monitoring of the progress of the work can be the most precise compared to methods based on sensors which lack precision in the covered areas (example of GNSS). However, marker-based AR technology suffers from placement issues, which sometimes dramatically increase setup time. For example, depending on the lighting conditions, the paper marker must be moved so that the model is accurately superimposed on the built environment. In larger sites, the use of several markers may be necessary with an update of the database of markers set up according to the progress of the work. Additionally, the marker installation time could impact the overall value provided by this technology when used in an active construction site with tight time constraints.

For outdoor, the method used will need to meet site requirements such as larger working environments and environmental conditions. The hybrid approach, which combines IMU sensors and a GNSS receiver with computer vision to improve GNSS accuracy, can be one of the methods to use in this context.

• Construction inspection

The use of AR solutions for construction inspection allows identifying potential accidents that could threaten the entire project, such as a pipe collision or a defect on a wall, for example. The exact placement on site is a critical factor for this type of application. To meet these needs, marker-based methods are the most adaptable to this application [65]. As demonstrated by our study, these methods provide placement accuracy down to a few centimeters (example of ARcore marker-based methods).

• Management of public services

AR solutions are recommended to identify buried networks whose location should be determined before any digging [66]. Here, the accuracy of the locations of underground networks is essential. Therefore, the use of a smartphone connected to an external GNSS device (i.e., SiteVision with VRS network) in an open environment is recommended to minimize errors related to positioning by GNSS (multipath, the presence of a power line). In an environment where conditions do not allow precise positioning by GNSS, the use of pre-defined marker placement methods on site can ensure centimeter accuracy.

7. Conclusions

The benefits of AR in the AECO industry have been widely demonstrated in the literature. However, technical limitations about placement accuracy and model stability must be overcome. Our paper fills a gap in the current literature by proposing an assessment approach of main placement methods proposed in the literature in terms of accuracy and model stability. The factors influencing the placement of the 3D model in AR placement was examined by testing and comparing several solutions with or without markers.

The results show that the range of placement errors by AR is highly variable (from a few centimeters to meters). It was demonstrated that the stability of marker-based AR is relatively lower, which is strongly influenced by the quality of the markers, the lighting conditions, the distance between the marker and the camera and the maturity of the SDKs.

An analysis of the usability of the tested AR placement methods in AECO projects was proposed to guide the user in choosing the adequate method depending on the application requirements.

In this context, the study focused on a 3D model for the study of the degree of placement of the different approaches. In a perspective, the results obtained and discussed will be a solid basis for the integration of our BIM-AR projects in the field of AECO. This research concentrated on a 3D model for the analysis of the placement of the various approaches. In a way, the findings and discussions will provide a strong foundation for the integration of our BIM-AR projects in the area of AECO.

We can draw the following conclusion from the tests:

• For a precise and exact placement in a real environment, the BIM-RA system faces challenges that must be overcome. The range of placement error by AR is highly variable, according to test results (from a few centimeters up to meters)
• Prior to use, each technique requires initialization (detection of planes, corners, points of interest, etc.), GNSS planning (for techniques based on GNSS positioning, such as SiteVision), or the placement of physical marks on the site (for techniques based on markers).
• Each method’s actual use in the field has a significant impact on how precisely and accurately it places AR.
• The cost (of the material, the license for the application, and the development), the use, which responds to various cases: precision work or simple visualization, and the time to develop a solution all play a role in the decision of a 3D model-RA placement method (use of an already ready solution or development of an application on a game engine).

However, with all the methods and solutions used, our approach was limited to a single environment. As a future work, we recommend conducting further studies in this field to take into consideration other scenarios, such as underground network, work in progress, etc. Furthermore, future research is recommended to cover how to reduce errors and improve accuracy based on hybrid (sensors and vision) placement approaches [67] and how to perform qualitative assessment and quantity of the placement by smart glasses (example HoloLens 2). Additionally, as it has been stated by [12], the system 3D model -AR placement evaluation methods should be studied in different outdoor environments in order to study the effects of factors, such as clear areas, cluttered areas, etc., on the accurate placement of the 3D model. Additionally, having a larger sample of on-site measurement points would help support the findings of this research. Finally, by comparing the previous mentioned factors in the marker-based approach in order to study their effects and corresponding weights of influence, we suggest that future research be conducted on this issue.