Off-Line Camera-Based Calibration for Optical See-Through Head-Mounted Displays

Featured Application

We here present a completely automated and fast method to calibrate optical see-through head-mounted displays based on the use of a calibrated camera replacing the user’s eye. The method can work with any type of optical see-through display and it is easy to replicate in both laboratory environments and real-world settings.

Abstract

In recent years, the entry into the market of self contained optical see-through headsets with integrated multi-sensor capabilities has led the way to innovative and technology driven augmented reality applications and has encouraged the adoption of these devices also across highly challenging medical and industrial settings. Despite this, the display calibration process of consumer level systems is still sub-optimal, particularly for those applications that require high accuracy in the spatial alignment between computer generated elements and a real-world scene. State-of-the-art manual and automated calibration procedures designed to estimate all the projection parameters are too complex for real application cases outside laboratory environments. This paper describes an off-line fast calibration procedure that only requires a camera to observe a planar pattern displayed on the see-through display. The camera that replaces the user’s eye must be placed within the eye-motion-box of the see-through display. The method exploits standard camera calibration and computer vision techniques to estimate the projection parameters of the display model for a generic position of the camera. At execution time, the projection parameters can then be refined through a planar homography that encapsulates the shift and scaling effect associated with the estimated relative translation from the old camera position to the current user’s eye position. Compared to classical SPAAM techniques that still rely on the human element and to other camera based calibration procedures, the proposed technique is flexible and easy to replicate in both laboratory environments and real-world settings.

1. Introduction

Visual augmented reality (AR) technology aims to enhance the user’s view of the real world by overlaying computer generated elements on it. Currently, optical see-through (OST) head mounted displays (HMDs) are at the leading edge of the AR technology, and they have the potential to become ubiquitous eventually in different fields of applications [1,2,3].

Nevertheless, their profitable usage across medical and industrial settings is still hampered by the complexity of the display calibration procedures required to ensure accurate spatial alignment between a real-world scene and computer generated elements [4,5]. The display calibration procedures of consumer level OST devices are rather simplified to improve usability, and this is achieved at the expense of sub-optimal results that are not tolerable for those applications for which the accurate alignment between virtual content and perceived reality is of the utmost importance [6]. This aspect is pushing research towards the realization of standardized OST calibration procedures that are not only flexible and easy to replicate, but that can also provide reliable and accurate results.

Overall, in visual AR applications, the problem of defining the appropriate spatial location of the digital 3D content with respect to the real scene is the principal factor that provides the user with a sense of perceptual congruity [7]. This problem is particularly challenging in OST HMDs, for the solution of which, knowing of the position of the user’s viewpoint(s) cannot be overlooked. On a first approximation, a single OST display can be modeled as an off-axis pinhole camera whose imaging plane corresponds to the semi-transparent virtual screen of the display and whose projection center corresponds to the nodal point of the user’s eye [8].

The goal of the OST display calibration is therefore to estimate the projection parameters of the combined eye–display system that encapsulates the optical features of the display and whose values vary according to the position of the user’s eye with respect to the imaging plane of the display.

Manual calibration procedures rely on user interaction to collect sets of 3D-2D correspondences by aligning, from different viewpoints, world reference points to image points shown on the HMD virtual screen [9]. These methods, particularly when aimed at estimating all the projection parameters simultaneously, are tedious, highly dependent on operator skill, time consuming, and should be repeated every time the HMD moves on the user’s head.

To lessen the burden on users in terms of time and workload, the calibration process can then be broken down into two phases: a first phase in which all the eye–display projection parameters are determined by performing a sort of factory calibration, ideally in a controlled setup; a second prior-to-use phase in which the calibration is refined by adjusting just a small subset of projection parameters. This is the underlying rationale behind several simplified manual calibration procedures, also referred to as semi-automatic calibration methods [10].

Finally, interaction-free calibration methods that exploit eye tracking techniques are ideally the preferred option for those AR applications that demand for accurate virtual-to-real spatial alignment over an extended period of time [8,11]. Such is the case of OST HMDs used as an aid to high precision tasks (e.g., for surgical or industrial applications). However, and as we illustrate in more detail in the next section, none of these methods is thoroughly automatic, since they also rely upon an off-line calibration phase performed manually or through a calibrated camera that replaces the user’s eye in a controlled setup.

Therefore, irrespective of the approach chosen, the off-line calibration step is paramount to minimize the human element, and for this reason, it ought to be highly accurate and reliable.

This paper presents an approach for performing accurate off-line camera based calibrations of OST HMDs. In our procedure, the projection parameters of the eye–display model are estimated by means of standard camera calibration and computer vision techniques. To this aim, the OST HMD must be mounted over a rigid and adjustable holder equipped with a camera that replaces the user’s eye.

The main contributions of our work are as follows:

• An off-line calibration procedure that is easy to replicate in both laboratory environments and real-world settings.
• A calibration procedure that can work with any type of OST HMD: with finite and infinite focal distances, with complex or simple optical combiners, featuring inside-out or outside-in tracking mechanisms.
• A calibration procedure that is completely automated.
• A calibration procedure that entails a simple on-line refinement step to account for the user’s eye position.
• A detailed formulation of all the mathematical steps involved in the procedure.

The remainder of this paper is structured as follows: Section 2 surveys different two phase approaches for OST HMD calibration. Section 3 provides a general overview of our camera based OST HMD calibration procedure together with its mathematical formulation starting from the standard projection matrix of the off-axis pinhole camera model of the eye–display. Section 4 illustrates the technical implementation of the calibration procedure. Section 5 describes the experiments and discusses the results. Finally, Section 6 concludes with a summary and future work directions.

2. Related Works

Research on how to achieve correct calibration of OST displays has been conducted over many years. In 2017, Grubert et al. [7] presented a comprehensive survey of all the calibration procedures proposed up to that time. In their work, the authors provided also useful insights into the fundamentals of calibration techniques, grouping them into three main categories: manual methods, semi-automatic methods, and automatic methods. We here provide a more focused overview of the different techniques in which the calibration workflow explicitly relies on an off-line phase.

In 2002, Genc et al. [12] proposed a simplified version of the SPAAM (a two phase SPAAM) in which the calibration process was split into two phases: an off-line stage that involves the estimation of the fixed projection parameters and a second on-line stage that aims to update the existing projection matrix partially by applying a 2D image warping of the screen image that includes scaling and shift.

A vision based robust calibration (ViRC) method was proposed in 2013 [13]. The method clearly distinguishes two types of parameters: the ones estimated off-line that are associated with the device dependent parameters and that are based on an approximate position of the user’s eye and those measured on-line that are instead related to the actual position of the user’s eye (4 DoF user dependent parameters). These latter parameters are refined through a perspective-n-point algorithm whose inputs are: the image coordinates of a cross-hair cursor displayed on the see-through screen at different positions and the 3D coordinates of a fiducial marker that the user must visually align to such a cursor. The 3D coordinates of the marker are determined by querying a tracking camera attached to the HMD (i.e., adopting an inside-out tracking approach).

In 2004, Owen et al. [14] presented display relative calibration (DRC), a two phase camera based calibration method. In the first phase of the method, the authors replace the user’s eye with a camera and use a mechanical jig to determine the projection parameters of the display system. In the second phase, the eye–display parameters are optimized on the position of the user’s eye(s) by means of a SPAAM-like procedure.

In 2005, Figl et al. [15] presented a camera based method for calibrating a medical binocular OST HMD, the Varioscope^TM M5. The calibration system uses a precision spindle moving a mobile calibration grid via a stepping motor and a tracker probe to localize the moving grid in space with respect to the HMD (i.e., adopting an outside-in tracking approach). However, the proposed method does not have any calibration refinement step that accounts for the user’s eye position.

A similar camera based procedure was presented by Gilson et al. [16] that also employs an inside-out tracking mechanism. After the calibration, the authors did not include any user centered refinement step to account for the eye position, since in their method: “the camera remains stationary by design, so we have constrained our method to minimize for only one camera pose” (page 6). Overall, the authors declared to have achieved an average virtual-to-real alignment accuracy of about two pixels for three camera positions different from the calibration one.

In 2014, Itoh and Klinker [8,17] proposed an automatic method with inside-out tracking that employed an eye tracker to measure the position of the user’s eye. The relative pose between the external camera and the eye tracking camera was predetermined during an off-line calibration session. The dynamic tracking of the user’s eye was used for continuously refining the eye–display projection parameters, whose initial values once again must be computed off-line through a standard SPAAM-like method or a camera based procedure.

A similar approach was proposed in 2015 by Plopski et al. [11,18]. The method refines on-line the pre-determined calibration parameters through a corneal imaging calibration (CIC) procedure that relies on corneal imaging to obtain correspondence pairs from a pre-calibrated HMD screen with its content reflection on the cornea of the user.

In 2016, Zhang et al. [19] presented an interesting two phase method in which a depth camera refined on-line the eye–display projection parameters performing the optimization of the hand registration through a genetic algorithm. Again, a first off-line phase was needed to provide a baseline for the subsequent optimization.

Finally, in 2017, Klemm et al. [20] presented an off-line automated camera based method that resulted in an optimized rendering frustum together with an arbitrary number of distortion maps determined from different viewpoints. Unfortunately, their triangulation approach only worked for OST HMDs with simple optics for which, as stated by the authors on page 57, the “pixel locations are independent from the eye position”.

As anticipated above, all these methods, require an off-line calibration step performed manually or via a camera replacing the user’s eye. This off-line calibration, to be effective and accurate, should be performed in a controlled setup for the following two main reasons: to reduce the errors due to human interaction and to ease and speed-up the subsequent prior-to-use calibration refinement done by the user.

In line with this, this paper presents a camera based OST calibration method capable of accurately estimating the intrinsic matrix of the eye–display pinhole model, and we provide a detailed and substantiated formulation of each step involved in the calibration procedure.

3. Methods

3.1. Rationale

In a previous work [21], we described a closed-loop calibration method specifically suited for OST displays with the focal plane at infinity. This particular optical feature reduces the complexity of the problem of estimating the projection matrix of the off-axis eye–display pinhole model, whose parameters can be modeled irrespective of the specific eye position with a procedure based on a simple homography based 2D warping of the ideal on-axis imaging plane. In this paper, we extend the scope of such a method, so that it can be applied to any kind of OST display (i.e., with finite focal distances), and we model the contribution of the eye position to the projection parameters.

3.2. Notation

The following notation is used throughout the paper. Uppercase letters denote spatial coordinate systems, such as the eye–display coordinate system $\mathrm{E}$. Lowercase letters denote scalar values, such as the focal length $f_u$. Both 3D and 2D points are represented by homogeneous vectors. Vectors are denoted by lowercase bold letters with a superscript denoting the reference coordinate system (e.g., a 3D point in eye–display coordinates ${\mathbf{v}}^{\mathrm{E}}$ or a 2D image point in virtual screen coordinates ${\mathbf{i}}^{\mathrm{E}}$). Vectors can also be expressed in component form, with a bold subscript indicating the correspondence (e.g., ${\mathbf{v}}^{\mathrm{E}}={(x_{\mathbf{v}},y_{\mathbf{v}},z_{\mathbf{v}},1)}^{\mathrm{E}}$). Matrices are denoted by uppercase typewriter letters (e.g., the intrinsic camera matrix of the eye–display ${}_{}^{\mathrm{E}}\mathtt{K}$).

The 6 DoF transformations from one coordinate system to another are so defined. Given two coordinate systems $\mathrm{A}$ and $\mathrm{B}$, the transformation from $\mathrm{A}$ to $\mathrm{B}$ is defined as ${}_{\mathrm{A}}^{\mathrm{B}}\mathtt{T}=({}_{\mathrm{A}}^{\mathrm{B}}\mathtt{R},{}_{\mathrm{A}}^{\mathrm{B}}\mathbf{t}$ where ${}_{\mathrm{A}}^{\mathrm{B}}\mathtt{R}$ is the rotation matrix and ${}_{\mathrm{A}}^{\mathrm{B}}\mathbf{t}$ is the translation vector. Therefore, we have:

$${\mathbf{v}}^{\mathrm{B}}={}_{\mathrm{A}}^{\mathrm{B}}\mathtt{R}{\mathbf{v}}^{\mathrm{A}}+{}_{\mathrm{A}}^{\mathrm{B}}\mathbf{t}$$

(1)

3.3. The Off-Axis Pinhole Camera Model of the Eye–Display

As mentioned in Section 1, the eye–display system is commonly modeled as an off-axis pinhole camera (i.e., the associated virtual rendering camera) where the nodal point of the user’s eye corresponds to the center of projection and the imaging plane is the semi-transparent virtual screen. This model describes the projection transformation between the coordinates of a point in the 3D space and the associated 2D point displayed on the imaging plane of the see-through display (Figure 1).

The intrinsic matrix of the off-axis eye–display model is:

$${}_{}^{\mathrm{off}\text{-}\mathrm{E}}\mathtt{K}=\left[\begin{array}{ccc}{f}_u& 0& c_u\\ 0& f_v& c_v\\ 0& 0& 1\end{array}\right]\in {\mathbb{R}}^{3\times 3}$$

(2)

where $f_u$ and $f_v$ are the focal lengths in pixels, and they denote the distances between the imaging plane of the display (i.e., the virtual screen) and the pinhole camera projection center (i.e., the nodal point of the user’s eye); $c_u$ and $c_v$ are the coordinates in pixels of the principal point that corresponds to the intersection between the principal axis and the virtual screen.

Using Equation (2), the perspective projection between a 3D point in the eye–display coordinate system ${\mathbf{v}}^{\mathrm{E}}$ and its associated 2D projection ${\mathbf{i}}^{\mathrm{E}}$, both expressed in homogeneous coordinates, is:

$$\lambda {\mathbf{i}}^{\mathrm{E}}={}_{}^{\mathrm{off}\text{-}\mathrm{E}}\mathtt{K}\left[{\mathtt{I}}_{3\times 3}{\mathtt{0}}_{3\times 1}\right]{\mathbf{v}}^{\mathrm{E}}={}_{}^{\mathrm{E}}\mathtt{P}{\mathbf{v}}^{\mathrm{E}}$$

(3)

The above formulation assumes a special choice of the world coordinate system $\mathrm{W}$, with $\mathrm{W}\widehat{=}\mathrm{E}$. In more general terms, by plugging the 6 DoF transformation from $\mathrm{W}$ to $\mathrm{E}$ into Equation (3), we obtain the projection transformation that maps world points onto the imaging plane of the eye–display ${}_{}^{\mathrm{E}}\mathtt{P}\in {\mathbb{R}}^{3\times 4}$:

$$\lambda {\mathbf{i}}^{\mathrm{E}}={}_{}^{\mathrm{off}\text{-}\mathrm{E}}\mathtt{K}\left[{}_{\mathrm{W}}^{\mathrm{E}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{E}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}={}_{}^{\mathrm{E}}\mathtt{P}{\mathbf{v}}^{\mathrm{E}}$$

(4)

The goal of all calibration methods is to provide an accurate estimation of the matrix ${}_{}^{\mathrm{E}}\mathtt{P}$, either by solving it as a whole or by determining each of the four entries of ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}\mathtt{K}$ and each of the six DoF of ${}_{\mathrm{W}}^{\mathrm{E}}\mathtt{T}$ individually.

3.4. Formulation of the Calibration Procedure

As mentioned in the Introduction, calibration procedures of OST displays aim to achieve a perfect spatial alignment between a 3D real-world scene and the computer generated scene perceived through the display. To this end, we need to estimate the perspective projection matrix of the off-axis rendering camera that yields a correct mapping of each 3D vertex ${\mathbf{v}}^{\mathrm{W}}$ of the virtual object onto the imaging plane space ${\mathbf{i}}^{\mathrm{E}}$. This 3D-2D mapping (i.e., process of rasterization) must be done in a way that all 3D vertexes along a ray that originates from the rendering camera center (i.e., the eye center $\mathrm{E}$) is rendered at the same location on the virtual screen and perfectly aligned to their real counterpart. Our calibration procedure measures such a projection matrix by means of a camera that acts as a replacement of the user’s eye. Hereafter, we use the term viewpoint camera to refer to such a camera.

In our formulation, as well as in the implementation of the procedure, we exploit a vision based inside-out tracking technique as done in [14,16,22]. Nonetheless, the same methodological guidelines that we here describe would also apply with an outside-in tracking technique, provided that it measures the pose of the sensor reference system, attached to the HMD, $\mathrm{S}$ with respect to the world scene $\mathrm{W}$.

Figure 2 shows the spatial relationships between the coordinate systems associated with the tracking camera $\mathrm{S}$, the internal viewpoint camera $\mathrm{C}$ (the user’s eye replacement), and the rendering camera $\mathrm{E}$.

The first step in our calibration procedure is the estimation of the viewpoint camera projection matrix that maps world points ${\mathbf{v}}^{\mathrm{W}}$ onto camera image points ${\mathbf{i}}^{\mathrm{C}}$:

$$\begin{array}{c}\lambda {\mathbf{i}}^{\mathrm{C}}={}_{}^{\mathrm{C}}\mathtt{K}\left[{}_{\mathrm{S}}^{\mathrm{C}}\mathtt{R}{}_{\mathrm{S}}^{\mathrm{C}}\mathbf{t}\right]\left[{}_{\mathrm{W}}^{\mathrm{S}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{S}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}=\\ ={}_{}^{\mathrm{C}}\mathtt{K}\left[{}_{\mathrm{W}}^{\mathrm{C}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{C}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}={}_{}^{\mathrm{C}}\mathtt{P}{\mathbf{v}}^{\mathrm{W}}\end{array}$$

(5)

We determine the intrinsic matrix of the viewpoint camera ${}_{}^{\mathrm{C}}\mathtt{K}$ and the relative pose between the viewpoint camera and the tracking camera $\left[{}_{\mathrm{S}}^{\mathrm{C}}\mathtt{R}{}_{\mathrm{S}}^{\mathrm{C}}\mathbf{t}\right]$ with conventional calibration routines [23] described in more detail in Section 4.2, whereas the pose of the world scene $\mathrm{W}$ with respect to $\mathrm{S}$ is measured on-line by querying the tracking camera $\left[{}_{\mathrm{W}}^{\mathrm{S}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{S}}\mathbf{t}\right]$. We used an OpenCV checkerboard as the target object for the inside-out tracking (see Section 4.1).

Next, the key step in the procedure is the computation of the planar homography ${}_{\mathrm{C}}^{\mathrm{E}}\mathtt{H}$ between the two pinhole cameras $\mathrm{C}$ and $\mathrm{E}$ that enables the image points of the viewpoint camera ${\mathbf{i}}^{\mathrm{C}}$ to be mapped onto the imaging plane of the see-through display (i.e., the virtual screen) ${\mathbf{i}}^{\mathrm{E}}$.

$$\lambda {\mathbf{i}}^{\mathrm{E}}={}_{\mathrm{C}}^{\mathrm{E}}\mathtt{H}{\mathbf{i}}^{\mathrm{C}}$$

(6)

where the planar homography ${}_{\mathrm{C}}^{\mathrm{E}}\mathtt{H}$ can be broken down as follows [24]:

$${}_{\mathrm{C}}^{\mathrm{E}}\mathtt{H}={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left({}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}+\frac{{}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}{\left(n^{\mathrm{C}}\right)}^{⊺}}{d^{\mathrm{C}\to \pi }}\right){}_{}^{\mathrm{C}}{\mathtt{K}}^{-1}$$

(7)

In Equation (7), ${}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}$ is the intrinsic matrix of the ideal on-axis camera model of the eye–display, whose entries are established considering the manufacturer’s specifications of the OST display. Notably, we assumed that the focal lengths on both the x-axis and y-axis were equal ($f_x=f_y$), meaning the display pixels were considered as being perfectly square:

$${}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}=\left[\begin{array}{ccc}\frac{w}{2tan(\frac{hfov}{2})}& 0& w/2\\ 0& \frac{h}{2tan(\frac{vfov}{2})}& h/2\\ 0& 0& 1\end{array}\right]$$

(8)

where $\frac{hfov}{2}$ and $\frac{vfov}{2}$ are the horizontal and vertical field-of-view (FOV) of the OST display and w and h are its width and height in pixels.

The other entries of Equation (8) are:

• the rotation matrix between the imaging planes of the viewpoint camera and of the OST display ${}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}$;
• the virtual translation vector between the viewpoint camera and the rendering camera ${}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}$. We also label such a vector as parallax contribution, given that it models the transformation between ideal ${}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}$ and real ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}$ based on a specific eye position (subscript ${\mathrm{p}}_0$). This contribution is described in more detail in Section 3.4.3;
• the normal unit vector to the OST display screen in the viewpoint camera reference system $n^{\mathrm{C}}$;
• the distance between the viewpoint camera center $\mathrm{C}$ and the imaging plane of the OST display $d^{\mathrm{C}\to \pi }$.

In the next two subsections, we explain how we estimate each of the above listed unknown variables.

3.4.1. Estimation of the Imaging Plane Pose with Respect to the Viewpoint Camera

We determine the position ${}_{\mathrm{C}}^{\mathrm{I}}\mathbf{t}$ and orientation ${}_{\mathrm{C}}^{\mathrm{I}}\mathtt{R}\equiv {}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}$ (i.e., the virtual pose) of the imaging plane of the OST display with respect to the viewpoint camera by computing the location of a calibration checkerboard projected onto the imaging plane of the see-through display (see Section 4).

The physical size of the checkerboard square is established by arbitrarily dictating the distance from the projection center to the imaging plane $d^{\mathrm{E}\to \pi }$ and adopting the intrinsic parameters of the ideal on-axis eye–display model (Equation (8)), as also suggested in [25]. By observing such a checkerboard with the calibrated viewpoint camera, we are able to compute both ${}_{\mathrm{C}}^{\mathrm{I}}\mathbf{t}$ and ${}_{\mathrm{C}}^{\mathrm{I}}\mathtt{R}$.

Notably, the translation vector ${}_{\mathrm{C}}^{\mathrm{I}}\mathbf{t}$ is linearly proportional to $d^{\mathrm{E}\to \pi }$ (${}_{\mathrm{C}}^{\mathrm{I}}\mathbf{t}\propto d^{\mathrm{E}\to \pi }$).

3.4.2. Estimation of the Rendering Camera Pose with Respect to the Viewpoint Camera

Given the pose of the imaging plane of the OST display with respect to a particular position of the viewpoint camera (i.e., $\mathrm{C}$), we can retrieve the pose of the rendering camera relative to the viewpoint camera by considering the transformation that brings the imaging plane to the focal plane. This pose encapsulates the rotational contribution caused by the different orientations of the rendering and of the viewpoint camera ${}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}\equiv {}_{\mathrm{C}}^{\mathrm{I}}\mathtt{R}$ and the parallax contribution, which is also proportional to the distance from the projection center to the imaging plane (${}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}\propto d^{\mathrm{E}\to \pi }$).

Geometrically, it is easy to demonstrate the following relation, valid ∀${\mathrm{C}}_{\mathrm{i}}$ position (Figure 3):

$$d^{\mathrm{E}\to \pi }=d^{\mathrm{C}\to \pi }+z_{\mathrm{C}}^{\mathrm{E}}=d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }+z_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{E}}$$

(9)

3.4.3. Meaning of the Parallax Contribution for the Off-Axis Eye–Display Pinhole Model

By plugging Equation (5) into Equation (6), we obtain:

$$\begin{array}{c}\lambda {\mathbf{i}}^{\mathrm{E}}={}_{\mathrm{C}}^{\mathrm{E}}\mathtt{H}{\mathbf{i}}^{\mathrm{C}}={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left({}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}+\frac{{}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}{\left(n^{\mathrm{C}}\right)}^{⊺}}{d^{\mathrm{C}\to \pi }}\right)\left[{}_{\mathrm{W}}^{\mathrm{C}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{C}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}=\\ ={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left({\mathrm{I}}_{3\times 3}+\frac{{}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}{\left(n^{\mathrm{C}}\right)}^{⊺}}{d^{\mathrm{C}\to \pi }}{}_{\mathrm{C}}^{\mathrm{E}}{\mathtt{R}}^{-1}\right){}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}\left[{}_{\mathrm{W}}^{\mathrm{C}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{C}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}=\\ ={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left({\mathrm{I}}_{3\times 3}+\frac{{}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}{\left(n^{\mathrm{C}}\right)}^{⊺}}{d^{\mathrm{C}\to \pi }}{}_{\mathrm{C}}^{\mathrm{E}}{\mathtt{R}}^{-1}\right)\left[{}_{\mathrm{W}}^{\mathrm{E}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{E}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}\end{array}$$

(10)

Since ${\left(n^{\mathrm{C}}\right)}^{⊺}{}_{\mathrm{C}}^{\mathrm{E}}{\mathtt{R}}^{-1}={\left({}_{\mathrm{C}}^{\mathrm{E}}\mathtt{R}{n}^{\mathrm{C}}\right)}^{⊺}={\left(n^{\mathrm{E}}\right)}^{⊺}=(0,0,1)$, which is the unit vector of the principal axis of the display, Equation (10) becomes:

$$\begin{array}{c}\lambda {\mathbf{i}}^{\mathrm{E}}=\stackrel{{}_{}^{\mathrm{E}}\mathtt{P}}{\overbrace{{}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left({\mathrm{I}}_{3\times 3}+\frac{{}_{\mathrm{C}}^{\mathrm{E}}\mathbf{t}{\left(n^{\mathrm{C}}\right)}^{⊺}}{d^{\mathrm{C}\to \pi }}\right)\left[{}_{\mathrm{W}}^{\mathrm{E}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{E}}\mathbf{t}\right]}}{\mathbf{v}}^{\mathrm{W}}\end{array}$$

(11)

This can be rewritten as:

$$\begin{array}{c}\lambda {\mathbf{i}}^{\mathrm{E}}=\underset{{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}}{\underbrace{{}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left[\begin{array}{ccc}1& 0& x_{\mathrm{C}}^{\mathrm{E}}/d^{\mathrm{C}\to \pi }\\ 0& 1& y_{\mathrm{C}}^{\mathrm{E}}/d^{\mathrm{C}\to \pi }\\ 0& 0& 1+z_{\mathrm{C}}^{\mathrm{E}}/d^{\mathrm{C}\to \pi }\end{array}\right]}}\left[{}_{\mathrm{W}}^{\mathrm{E}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{E}}\mathbf{t}\right]{\mathbf{v}}^{\mathrm{W}}\end{array}$$

(12)

${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}$ can be further processed using Equation (9) and imposing ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}(3,3)=1$, given that ${}_{}^{\mathrm{E}}\mathtt{P}$ is defined up to a scale factor $\lambda$:

$$\begin{array}{c}{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left[\begin{array}{ccc}\frac{d^{\mathrm{C}\to \pi }}{d^{\mathrm{E}\to \pi }}& 0& \frac{x_{\mathrm{C}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& \frac{d^{\mathrm{C}\to \pi }}{d^{\mathrm{E}\to \pi }}& \frac{y_{\mathrm{C}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& 0& 1\end{array}\right]\end{array}$$

(13)

Equation (13) tells us that, to achieve an accurate spatial alignment between 3D real-world points and computer generated elements rendered by an OST display and observed by a specific viewpoint (${\mathrm{p}}_0$), each virtual 3D vertex ${\mathbf{v}}^{\mathrm{W}}$ must be observed by an off-axis rendering camera whose projection matrix ${}_{}^{\mathrm{E}}\mathtt{P}$ is modeled as follows:

• The extrinsic parameters of the camera are: $\left[{}_{\mathrm{W}}^{\mathrm{E}}\mathtt{R}{}_{\mathrm{W}}^{\mathrm{E}}\mathbf{t}\right]$. Unsurprisingly, the center of projection of the virtual rendering camera is the user’s eye (in our procedure, the viewpoint camera), whereas its orientation matches the display virtual screen’s orientation.
• The intrinsic parameters of the off-axis camera are obtained by applying a homographic transformation to the ideal on-axis intrinsic matrix:

  $$\begin{array}{c}{}_{\mathrm{on}\text{-}\mathrm{E}}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{H}}_{{\mathrm{p}}_0}={\mathtt{H}}_{{\mathrm{p}}_0}=\left[\begin{array}{ccc}\frac{d^{\mathrm{C}\to \pi }}{d^{\mathrm{E}\to \pi }}& 0& \frac{x_{\mathrm{C}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& \frac{d^{\mathrm{C}\to \pi }}{d^{\mathrm{E}\to \pi }}& \frac{y_{\mathrm{C}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& 0& 1\end{array}\right]\end{array}$$

  (14)

This homography allows computing the intrinsic matrix of the real off-axis eye–display system, and it encapsulates the shift and scaling effect (i.e., the parallax contribution) due to a particular position of the user’s eye.

The intrinsic matrix for a different viewpoint position ${\mathrm{p}}_{\mathrm{i}}$ is:

$$\begin{array}{c}{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}={}_{}^{\mathrm{on}\text{-}\mathrm{E}}\mathtt{K}\left[\begin{array}{ccc}\frac{d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }}{d^{\mathrm{E}\to \pi }}& 0& \frac{x_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& \frac{d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }}{d^{\mathrm{E}\to \pi }}& \frac{y_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{E}}}{d^{\mathrm{E}\to \pi }}\\ 0& 0& 1\end{array}\right]\end{array}$$

(15)

Therefore, from simple algebraic manipulations, we can obtain the transformation matrix that enables us to pass from ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}$ to ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}$:

$$\begin{array}{c}{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}={}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}{\mathtt{H}}_{{\mathrm{p}}_0}^{-1}{\mathtt{H}}_{{\mathrm{p}}_i}=\\ ={}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}\left[\begin{array}{ccc}\frac{d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }}{d^{\mathrm{C}\to \pi }}& 0& \frac{x_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{E}}-x_{\mathrm{C}}^{\mathrm{E}}}{d^{\mathrm{C}\to \pi }}\\ 0& \frac{d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }}{d^{\mathrm{C}\to \pi }}& \frac{y_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{E}}-y_{\mathrm{C}}^{\mathrm{E}}}{d^{ct\mathrm{C}\to \pi }}\\ 0& 0& 1\end{array}\right]\end{array}$$

(16)

From this, we derive the same relation presented in [8] and in [7]:

$$\begin{array}{c}{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}={}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}\left[\begin{array}{ccc}1+\frac{z^{\prime }}{d^{\mathrm{C}\to \pi }}& 0& \frac{-x^{\prime }}{d^{\mathrm{C}\to \pi }}\\ 0& 1+\frac{z^{\prime }}{d^{\mathrm{C}\to \pi }}& \frac{-y^{\prime }}{d^{\mathrm{C}\to \pi }}\\ 0& 0& 1\end{array}\right]\end{array}$$

(17)

where ${}_{\mathrm{C}}^{{\mathrm{C}}_{\mathrm{i}}}\mathbf{t}=[x^{\prime },y^{\prime },z^{\prime }]$ is the translation from the old viewpoint position to the new viewpoint position and where $d^{{\mathrm{C}}_{\mathrm{i}}\to \pi }=d^{\mathrm{C}\to \pi }+z^{\prime }$.

3.4.4. Relation between the Optical Properties of the OST Display and Intrinsic Matrix of the Eye–Display Model

In any HMD, the role of the collimation optics is to display the microdisplay image so that it appears magnified at a comfortable viewing distance [26]. Specifically, the imaging plane of many consumer level OST HMDs is at infinity (i.e., at a very far distance), and this happens if the microdisplay is located in close proximity of the focal point of the collimation optics. In this case, the light rays coming from each pixel of the microdisplay are arranged in a parallel pattern, and the virtual image perceived by the user, to a first approximation, is not subjected to any shift or scaling effect for different positions of the user’s eye within the display eye-motion-box area.

Thereby, for those OST displays, the position of the user’s eye does not influence the intrinsic matrix of the off-axis eye–display pinhole camera model ${}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}$, and this implies that the intrinsic linear parameters of the eye–display can be considered as parallax-free:

$$\begin{array}{c}{}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_i}={}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}\left[\begin{array}{ccc}1+\stackrel{\approx 0}{\overbrace{\frac{z^{\prime }}{d^{\mathrm{C}\to \pi }}}}& 0& \stackrel{\approx 0}{\overbrace{\frac{-x^{\prime }}{d^{\mathrm{C}\to \pi }}}}\\ 0& 1+\stackrel{\approx 0}{\overbrace{\frac{z^{\prime }}{d^{\mathrm{C}\to \pi }}}}& \stackrel{\approx 0}{\overbrace{\frac{-y^{\prime }}{d^{\mathrm{C}\to \pi }}}}\\ 0& 0& 1\end{array}\right]\approx {}_{}^{\mathrm{off}\text{-}\mathrm{E}}{\mathtt{K}}_{{\mathrm{p}}_0}\end{array}$$

(18)

since $x^{\prime }\backslash y^{\prime }\backslash z^{\prime }<<d^{\mathrm{C}\to \pi }$.

In this case, the on-line refinement step accounting for the user’s eye position is not needed. Differently, for those OST displays with shorter focal distances (e.g., Microsoft HoloLens, Magic Leap, Epson Moverio BT-200), the light rays within the eye-motion-box of the display are not parallel. For this reason, the contribution of the position of the user’s eye to the intrinsic matrix cannot be neglected, and the update contribution expressed by Equation (17) must be taken into consideration (i.e., the intrinsic matrix is not parallax-free).

4. Technical Implementation of the Procedure

4.1. Hardware Setup

Our calibration procedure is applicable to any OST HMD, with finite or infinite focal distance, based on waveguides or on large spherical or semispherical optical combiners.

We tested our method on a commercial binocular OST HMD (ARS.30 by Trivisio [27]) appropriately reworked to embody a vision based inside-out tracking mechanism. The visor was provided with dual SXGA OLED panels with $1280\times \mathrm{101,024}$ resolution and with a pair of standard flat optical combiners tilted at about ${45}^{°}$. The two panels were controlled via HDMI. The diagonal FOV of the HMD was of ${30}^{°}$, which corresponded to $\approx 1.11$ arcmin/pixel angular resolution. The eye-relief of the display was 30 mm, and the size of the eye-motion-box was $8\times 10$ mm. The imaging plane of the OST display was projected at 500 mm ($d^{\mathrm{E}\to \pi }=500$ mm).

Following a similar approach to our previous research works [28,29], we embedded the HMD in a 3D printed plastic shell; the plastic shell was designed to house a pair of liquid crystal (LC) shutters and a pair of stereo RGB cameras (Leopard Imaging LI-OV4689) for the inside-out tracking mechanism (Figure 4). The LC panels were placed in front of the optical combiners of the HMD, and they could be electronically controlled so as to modify the transparency of the see-through display and switch from see-through state to occluded state. We used the occluded modality for the second step of the calibration, in which we projected a calibration checkerboard onto the imaging plane of the see-through display.

Figure 5 shows the experimental setup. In our procedure, we used a single camera inside-out tracking mechanism. The tracking cameras had ${109}^{°}$ diagonal FOV, which, was associated with a $1280\times 720$ image resolution, yielding $\approx 4.8$ arcmin/pixel angular resolution. As we will discuss in Section 5, this particular feature enabled us to have a reference limit in the evaluation of the overall calibration accuracy.

In our tests, we performed the calibration on the right display. Therefore, we placed the viewpoint camera behind the right display at approximately the eye-relief distance from the flat combiner (i.e., $\approx 30$ mm).

The viewpoint camera was a SONY FCB-MA130, which had a $1/{2.45}^{″}$ CMOS sensor, a $1280\times 720$ resolution, and a ${59}^{°}$ diagonal FOV, which corresponded to $\approx 2.67$ arcmin/pixel.

Both the HMD and the validation checkerboard were locked to two rigid and adjustable holders. The validation checkerboard was a standard $7\times 4$ OpenCV calibration checkerboard with a square size of 30 mm. The checkerboard was placed at approximately 650 mm of distance from the viewpoint camera. We attached the viewpoint camera to a 3D printed mounting template. The mounting template was equipped with fixing holes for placing the camera in eight pre-set positions radially arranged within the eye-motion-box of the see-through display (Figure 6). The template and the camera were both anchored to the translation bar of the HMD holder.

We calibrated the display for the left position (reference position), and then, we computed the intrinsic matrix of the eye–display for the remaining seven positions using Equation (17), where $d^{\mathrm{C}\to \pi }$ is given by Equation (9) and where the translation vector ${}_{\mathrm{C}}^{{\mathrm{C}}_{\mathrm{i}}}\mathbf{t}$ is computed using two stereo calibrations between the tracking camera (fixed) and the viewpoint camera in the reference and in the new position:

$$\begin{array}{c}{}_{\mathrm{C}}^{{\mathrm{C}}_{\mathrm{i}}}\mathbf{t}=[x^{\prime },y^{\prime },z^{\prime }]={}_{\mathrm{C}}^{\mathrm{S}}\mathbf{t}-{}_{{\mathrm{C}}_{\mathrm{i}}}^{\mathrm{S}}\mathbf{t}\end{array}$$

(19)

4.2. Calibration Software

As outlined in Section 3.4, the calibration workflow was broken down into four main steps, each of which was associated with a different software routine:

• Estimation of the intrinsic camera parameters of both the viewpoint camera and the tracking camera (intrinsic camera calibrations).
• Estimation of the pose of the tracking camera with respect to the viewpoint camera (heterogeneous stereo calibration).
• Estimation of the pose of the rendering camera with respect to the viewpoint camera (parallax contribution estimation).
• Estimation of the final eye–display projection matrix (eye–display matrix calculation).

The calibration workflow is depicted in Figure 7. The details of the calibration steps are presented in the next subsections.

4.2.1. Intrinsic Camera Calibrations

The viewpoint camera and the tracking camera were calibrated with a conventional calibration technique [23] that required storing multiple camera views of a planar pattern (i.e., OpenCV checkerboard). Linear parameters (i.e., intrinsic camera matrix) and non-linearities due to camera lens distortion (i.e., distortion parameters) were computed using non-linear least-squares minimization (i.e., the Levenberg–Marquardt algorithm). This procedure was performed using the MATLAB camera calibration toolbox (R2018b MathWorks, Inc., Natick, MA, USA).

4.2.2. Heterogeneous Stereo Calibration

The relative pose between the viewpoint camera and the tracking camera was estimated through a stereo calibration routine specific for two different camera types. This algorithm was developed in C++ under the Linux Operating System (Ubuntu 16.04) and using the OpenCV API 3.3.1 [30].

4.2.3. Parallax Contribution and Eye–Display Matrix Estimation

The output of the heterogeneous stereo calibration algorithm, together with the intrinsic parameters of the two physical cameras and the manufacturer’s ideal projection parameters of the display were all fed into the final calibration routine. This routine was developed in MATLAB.

As described in Section 3.4, the pose of the OST display imaging plane with respect to the viewpoint camera was determined as follows: a calibration checkerboard was projected onto the imaging plane of the occluded see-through display in full-screen modality and acquired by the viewpoint camera; the virtual pose of the imaging plane was determined through a standard perspective-n-point algorithm [31]. These final data enabled us to compute the projection parameters of the off-axis eye–display model.

5. Experiments and Results

A dedicated software application was developed in MATLAB to validate the accuracy of the calibration technique. In the application, we generated a virtual scene whose virtual viewpoint (i.e., rendering camera) was controlled according to the extrinsic and the intrinsic parameters of the eye–display model determined from the calibration. In this way, we created an AR OST visualization, whose overlay accuracy could be considered as an objective evaluation metric for the calibration procedure. We therefore elaborated the image frames of the viewpoint camera and measured the overlay error, or reprojection error, between real and virtual features.

As ground truth real features, we considered the corners of the validation checkerboard, whereas as virtual features, we used virtual circles. The overlay error was computed as the Euclidean distance ($o_{2\mathrm{D}}$) between the image coordinates of the real and virtual features (Figure 8).

We exploited the hybrid nature of the HMD, which can work both under OST and the video see-through modality [28], to measure the overlay error as follows:

• The image coordinates of the checkerboard corners were extracted by processing a frame of the viewpoint camera acquired with both the display and the optical shutter turned off. In order to do this, we used the MATLAB corner detection algorithm (Figure 9b).
• The image coordinates of the virtual landmarks were extracted by processing the viewpoint camera image without the real-world background, hence with both the display and the optical shutter turned on. The coordinates of the centroids of the virtual circles were retrieved using a semi-automatic centroid detection algorithm (Figure 9c).

5.1. Quantitative Evaluation of Virtual-to-Real Registration Accuracy

Quantitative results are presented in terms of the average value, standard deviation, and max value of the overlay error ($o_{2\mathrm{D}}^{err}$) onto the imaging plane of the viewpoint camera over the eight positions on which the calibration was performed. Similarly, we also measured the values of the subtended angular error (${\alpha }^{err}$). Then, knowing the overlay error in pixels and given the distance $z_{\mathrm{C}}^{\mathsf{ß}}$ and the focal length of the viewpoint camera $f_{\mathrm{C}}$, we also computed the associated absolute error in mm at the validation checkerboard plane by using the following relation: [32,33]:

$$\begin{array}{c}{a}_{3\mathrm{D}}^{err}\approx \frac{o_{2\mathrm{D}}^{err}{z}_{\mathrm{C}}^{\mathsf{ß}}}{f_{\mathrm{C}}}\end{array}$$

(20)

5.2. Results and Discussion

Table 1. Results of the accuracy analysis in pixels (overlay error), arcmin (angular error), and mm (absolute error).

	Overlay Error			Angular Error			Absolute Error
Camera		(Pixel)			(Arcmin)			(mm)
Position	Mean	Std Dev	Max	Mean	Std Dev	Max	Mean	Std Dev	Max
1-L	1.08	0.52	1.98	2.88	1.39	5.3	0.55	0.26	1
2-UL	1.43	0.78	3.23	3.82	2.08	8.66	0.72	0.39	1.64
3-UR	2.03	0.87	3.48	5.43	2.32	9.33	1.03	0.44	1.76
4-T	2.06	0.58	3.08	5.52	1.56	8.25	1.04	0.3	1.56
5-LL	2.74	0.82	4.34	7.35	2.2	11.61	1.39	0.42	2.19
6-LR	3.29	0.6	5.03	8.82	1.6	13.47	1.67	0.3	2.55
7-D	2.61	0.66	4	6.99	1.76	10.72	1.32	0.33	2.03
8-R	2.61	0.67	4.05	6.99	1.8	10.85	1.32	0.34	2.05

shows the results of the eight calibrations in terms of overlay error ($o_{2\mathrm{D}}^{err}$), angular error (${\alpha }^{err}$), and absolute error ($a_{3\mathrm{D}}^{err}$). The overall mean, standard deviation, and max values of $o_{2\mathrm{D}}^{err}$ were $2.23$, $0.96$, and $5.03$ px; the same values for ${\alpha }^{err}$ were $5.98$, $2.58$, and $13.47$ arcmin and for $a_{3\mathrm{D}}^{err}$$1.13$, $0.49$, and $2.55$ mm.

The average overlay error for the reference position was comparable to that obtained by Owen et al. [14] and by Gilson et al. [16], whereas for the remaining positions, the average overlay error was comparable to that obtained in [9] and in [8]. It should be also noted that the results of the angular error are reported considering the rather low angular resolution of the viewpoint camera (≈2.67 arcmin/pixel). This explained the values of the angular errors obtained. If we considered the absolute error at the distance where the validation checkerboard was placed (≈650 mm), our results were comparable to those obtained with the camera based triangulation approach proposed by Klemm et al. [20]. The experiment results suggested that the calibration refinement step accounting for the user’s eye position was paramount to achieve accurate results also for eye positions different from the reference one for all those OST HMDs with finite focal distances (i.e., not parallax-free OST HMDs). This was in accordance with what was suggested by Owen et al. [14] and contrary to what was hypothesized by Gilson et al. and by [34] et al. It should be also stressed that this assumption was clearly different from what was suggested in [20] (page 58), where the classification of the OST HMDs was made based on the level of complexity of the OST optical combiner: “these results show that the triangulation approach only works for OSTG with simple optics and that it is not a universal approach”.

Overall, there were at least two sources of calibration inaccuracies in our method, both of which could be easily addressed in the future. The first one was not due to the calibration procedure per se, but it was due to the intrinsic tracking accuracy of the inside-out mechanism adopted in our study, which, as anticipated, was affected by the angular resolution of the tracking camera (i.e., ≈4.8 arcmin/pixel). This choice was dictated by the need to offer sufficient stereo overlap between the two tracking cameras, albeit at the expense of a reduced pixel density. It should be noted that this value of angular resolution had a detrimental effect on the calibration results as it increased the uncertainty in estimating the position of the validation checkerboard. This measure was used both for calibrating the eye–display model for the reference position of the viewpoint camera and also for computing the translation vector ${}_{\mathrm{C}}^{{\mathrm{C}}_{\mathrm{i}}}\mathbf{t}$. We believe that in the future, the accuracy of the off-line phase of the calibration would benefit from the selection of tracking cameras with a higher value of angular resolution, achievable by selecting cameras with a higher image resolution or with longer focus lenses.

A second source of inaccuracy arose from not considering, in our solution, the contribution of the display optical distortion. Indeed, the collimation optics of any OST HMDs, despite being essential to provide a reasonable FOV and a comfortable viewing distance for the virtual image, is a source of optical distortions such as radial and tangential distortions [35]. The same can be said for the optical combiner, particularly for waveguide based OST displays [20]. These optical distortions not only affect the shapes of the virtual objects perceived by the user, but they also distort the pattern of binocular disparities between the left and right virtual images, and therefore, they alter the stereoscopic perception of the virtual content [36]. To counter this problem, we are currently working on a solution that integrates a non-linear optimization step in the calibration algorithm. This further step can enable us to estimate the optical distortion parameters (radial and tangential). By doing so, we can correctly compensate for the optical distortions through a predistortion technique based on a 2D non-linear mapping of the virtual image on the imaging plane, as done in [25].

In order for the distortion correction, as well as the transformation (17) to be efficient and real time, a GPU based texture mapping approach (i.e., late warping) should be also adopted [37]. This mechanism would require the use of an eye tracking camera. On account of this, our calibration procedure potentially can be integrated in the low level rendering mechanism of commercial OST HMDs to encompass the linear and non-linear 2D warping of the ideal imaging plane yielded by the intrinsic optical properties of the display and by the user’s eye position [38].

6. Conclusions and Future Work

In this paper, we presented a method to easily calibrate OST HMDs based on the use of a calibrated camera replacing the user’s eye. The method, which could work with any type of OST HMD, only required the camera to observe a planar calibration pattern displayed on the imaging plane of the see-through display.

The method exploited standard camera calibration and computer vision techniques for estimating the projection parameters of the off-axis display model for a generic position of the viewpoint camera. In a prior-to-use second phase, the projection parameters were then refined through a planar homography that encapsulated the shift and scaling effect (i.e., the parallax contribution) associated with the relative translation from the old camera position to the current user’s eye position.

To evaluate the efficacy of the calibration technique objectively, we measured the overlay error between real and virtual features observed by the viewpoint camera from different positions within the eye-motion-box of the display. Experimental results indicated that the average overlay error of our calibration was $2.23$ pixel.

Our future work involves the integration of an additional step in the off-line calibration procedure devoted to the estimation of the radial and tangential distortion coefficients due to the lens system of the OST display. This calibration step can be performed through a non-linear optimization method such as the Levenberg–Marquardt algorithm.

We are also working on the integration of an eye tracking camera with our custom made OST HMD to perform user studies to evaluate the efficacy of the run-time refinement of the linear and non-linear calibration parameters and to design possible algorithm improvements that take into consideration the eye model.

Finally, the flexibility of our camera based calibration procedure can pave the way toward the profitable use of consumer level OST HMDs also in those applications that demand high accuracy in the spatial alignment between computer generated elements and the real-world scene (e.g., in surgical or high precision industrial applications).