A Calibration Method for System Parameters in Direct Phase Measuring Deflectometry

Featured Application

The paper proposes a novel method of calibrating system parameters for a PMD (phase measuring deflectometry) based system. This calibration method can be used in the areas of reverse engineering, car industry and aerospace industry.

Abstract

Phase measuring deflectometry has been widely studied as a way of obtaining the three-dimensional shape of specular objects. Recently, a new direct phase measuring deflectometry technique has been developed to measure the three-dimensional shape of specular objects that have discontinuous and/or isolated surfaces. However, accurate calibration of the system parameters is an important step in direct phase measuring deflectometry. This paper proposes a new calibration method that uses phase information to obtain the system parameters. Phase data are used to accurately calibrate the relative orientation of two liquid crystal display screens in a camera coordinate system, by generating and displaying horizontal and vertical sinusoidal fringe patterns on the two screens. The results of the experiments with an artificial specular step and a concave mirror showed that the proposed calibration method can build a highly accurate relationship between the absolute phase map and the depth data.

1. Introduction

The past several decades have seen revolutionary developments in three-dimensional (3D) optical shape measurement techniques [1,2], which are now widely used across many different fields, including reverse engineering, 3D games, digitization of cultural relics, clothing, biometrics and other biological fields [3,4,5,6,7,8,9]. In the area of structured light techniques, phase-calculation-based fringe pattern projection has made rapid progress so far, because it has the advantages of full-field measurement, non-contact operation, automatic data processing, and fast data acquisition [10,11,12,13]. Nevertheless, there are a large number of specular objects that need to be measured [14]. Current fringe pattern projection techniques cannot deal with this demand, especially in the measurement of specular objects that have discontinuous and/or isolated surfaces.

With the increasing demands of industrial development, much relative research has been done to solve the problem of specular object measurement. Shape reconstruction from gradient data [15,16,17,18] based on phase measuring deflectometry (PMD, also called fringe reflection profilometry) [19,20,21,22,23,24,25,26,27,28] has been widely studied because of its high dynamic range, full-field measurement, fast data acquisition, and automatic data processing. Cross path integration techniques, Fourier transform integration, and area wavefront reconstruction methods have been demonstrated [29]. However, existing PMD methods have the drawback of local error accumulation and are incapable of measuring discontinuous specular objects while reconstructing 3D shapes by integrating gradient data. Many improved algorithms have been proposed to avoid this integration procedure. Huang et al. [30] presented an iterative method that uses discrete cosine transforms to deal with the integration problem, with an incomplete gradient dataset in a Southwell configuration. Yuan et al. [31] presented a method of measuring the absolute height and slope of a specular surface with an incident ray, camera pinhole, and dummy assistant surfaces. Song et al. [32] selected a binary shifting strip as a structured light pattern for 3D reconstruction of the specular surface. Recently, Huang et al. [33] have proposed a model phase measuring deflectometry technique that uses a mathematical method to reconstruct 3D shapes of specular objects.

In all these PMD methods, calibration of the system parameters plays a key role in the measurement of the specular surface because it directly influences the accuracy of the measurement system. Ren et al. [34] presented an iterative optimization algorithm to calibrate stereo deflectometry. It includes two steps: a stepwise method and a self-calibration method; however, errors accumulated via the iterative principle is a potential problem. Xiao et al. [35] introduced a calibration method that uses a markerless flat mirror. This method requires a two-dimensional cosine fringe pattern to be reflected three times. It is easy for the images to become out of focus, which influences the accuracy of extraction of the feature points on the screen. System calibration is thus still a challenging problem in measuring 3D shapes by PMD.

A novel DPMD (direct PMD) method that can obtain the full-field 3D shape of specular objects with discontinuous and/or isolated surfaces has been developed to solve the problem of measuring complex specular surfaces [36,37,38]. This method builds a direct relationship between the absolute phase and the depth information. The DPMD method can calculate depth data from one directional fringe patterns at two display screens, instead of two orthogonal fringe patterns. Moreover, 3D shape data of specular objects could be obtained from two unwrapped phase maps directly, without the procedure of integrating gradient data. Calibrating the system parameters is an important step in measuring 3D shape data accurately, including the distance between the reference plane and the liquid crystal device (LCD) screen, and the distance between the reference plane and the charge coupled device (CCD) camera. In order to calibrate the system parameters, the existing methods used a translating stage and a machine vision-based method. Therefore, the calibration accuracy, on one hand, depends on the accuracy of the stage, but on the other hand, it has a low accuracy due to the limited depth of field (DOF) of the lens.

This paper presents a novel parameter calibration method that uses phase data to accurately determine the two system parameters. The phase information is calculated from sinusoidal fringe patterns, which are unaffected by the imaging lens being out of focus, so the calibrated system parameters have high accuracy. The following section briefly describes the DPMD technique. Section 3 explains the principle of using phase data to calibrate the system parameters. Section 4 describes the experiments used to verify the proposed calibration method and the results. Section 5 presents our conclusions.

2. Direct Phase Measuring Deflectometry

Figure 1 is a schematic diagram of the proposed full-field 3D shape measurement system for specular surfaces using the DPMD technique. The system consists of two LCD screens, one mirror, one CCD camera, and one beam splitter plate.

The beam splitter was adjusted to a suitable position that makes a virtual image LCD₁^′ of LCD₁ parallel to LCD₂, as if the two LCD screens were located in two parallel positions. The two screens were parallel to a reference mirror located in the reference plane. The parallelism was established by repeated adjustments of the relative orientations of the three components (the two LCD screens and reference mirror). A checkerboard could be generated and displayed on the LCD₁ and LCD₂ screens, which were both located on the angular displacement platforms. As a result, the screens could change position and direction randomly and accurately. A CCD camera captured the checkerboard images on the two screens. The extrinsic parameters of LCD₁ and LCD₂ were calculated from the captured checkerboard image. Then, based on their differences, the relative orientation between the virtual image of LCD₁ and LCD₂ was adjusted to be parallel, using the angular displacement platforms. This procedure was repeated several times until the three components were nearly parallel. And then there is also Zhang’s method [39] for compensation by generating deformed fringes by software to be equivalent to the exact parallel alignment of two display screens.

In the diagram, h is the depth of the measured point with respect to the reference mirror, d₁ is the distance between LCD₁^′ and the reference mirror, and d₂ is the distance between LCD₂ and the reference mirror.

As illustrated in Figure 1, two rays of light were displayed and reflected into the CCD camera via the measured surface and the mirror at the reference position. The two incident rays corresponded to the same reflection light. The absolute phases of the two incident rays are denoted as ${\varphi }_1$ and ${\varphi }_2$ on the reference plane and ${\varphi }_1^{\prime }$ and ${\varphi }_2^{\prime }$ on the measured specular surface. θ and $2\theta +\beta$ are the angle between the incident ray and the normal vector of the reference and the angle between the incident ray and the normal vector of the measured specular surface, respectively. The period of the displayed fringe pattern on the LCD screen is denoted as q. l is the distance on LCD₁^′ between the two incident rays because of the height and gradient of the measured surface. Parameter h stands for the height of the measured specular surface with respect to the reference plane. d₁ is the distance between LCD₁^′ and the reference mirror, and d₂ is the distance between LCD₂ and the reference mirror.

According to geometric triangle relationships, which are illustrated in Figure 1, Equations (1)–(4) can be acquired respectively.

$$(d_1-d_2)\tan \beta =\frac{({\varphi }_1-{\varphi }_2)\times q}{2\pi }$$

(1)

$$(d_1-d_2)\tan (2\theta +\beta )=\frac{({\varphi }_1^{\prime }-{\varphi }_2^{\prime })\times q}{2\pi }$$

(2)

$$(d_1+h)\tan \theta +l=(d_1-h)\tan (2\theta +\beta )$$

(3)

$$l=\frac{({\varphi }_1-{\varphi }_1^{\prime })\times q}{2\pi }.$$

(4)

Equation (5) can be derived based on the above Equations (1)–(4).

$$h=\frac{d_1({\varphi }_2-{\varphi }_2^{\prime })-d_2({\varphi }_1-{\varphi }_1^{\prime })}{({\varphi }_1-{\varphi }_2)+({\varphi }_1^{\prime }-{\varphi }_2^{\prime })}.$$

(5)

This equation shows clearly that the depth value can be calculated directly from the captured fringe patterns as long as the two parameters d₁ and d₂ and phase information on the reference mirror are known beforehand. The calibration method used to obtain the system parameters is explained in the following section.

3. Calibration of System Parameters

An important step in obtaining 3D shape data is to build the relationship between the absolute phase and the depth, which is known as calibration. In the mathematical model derived in Equation (5), the two parameters d₁ and d₂, need to be determined beforehand. When the extrinsic parameters (orientations) of the mirror, LCD₁, and LCD₂ have been acquired in the same camera coordinate system, d₁ can be calibrated from the relative distance between the mirror and LCD₁, which can be obtained through matrix transformation; d₂ can be calibrated in the same way. A novel parameter calibration method for the DPMD system has been proposed by using a phase target to solve the existing problems of dependency on a translating stage and the limited depth of field (DOF) of the lens.

3.1. Calibration of Internal Parameters

To apply a machine vision-based method to determine distance d₁ and d₂ by a CCD camera, the internal parameters need to be calibrated by using the following equation:

$$\lambda \left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=A\left[\begin{array}{cc}R& T\end{array}\right]\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right],$$

(6)

where R is a matrix representing the three rotation angles and T = [t_x, t_y, t_z] is a vector representing the three linear translations. [X_w, Y_w, Z_w] is the coordinate vector of a point in the world coordinate system, while [u, v] is a coordinate vector of the corresponding point in the pixel coordinate system. λ is an arbitrary scaling factor. A is a matrix of the CCD camera internal parameters, including two focal lengths (F_u and F_v), two principal point coordinates (P_u and P_v), and four image radial and tangential distortion coefficients (K₁, K₂, K₃, and K₄). The eight internal parameters need to be calibrated beforehand and Zhang’s camera calibration method is used to obtain their values by using a checkerboard at different positions [40].

3.2. Orientation of the Mirror

A mirror with a matrix of discrete hollow ring markers on its surface was used to determine the distance between the reference plane and the two LCD screens in orientating the reference plane in the camera coordinate system. After the mirror plane was adjusted to be parallel to the screen surface, the CCD camera captured an image of the hollow ring marker matrix, as illustrated in Figure 2a.

After extracting the inner and outer edges of each ring marker, the center of each of the markers was determined using the principles of cross ratio and the pole–polar relationship [41]. Figure 2b shows the center positions determined for each marker in the matrix, illustrated by red dots. Because the distance between neighboring ring markers is known beforehand, the external parameters of the mirror, rotation matrix R, and translating vector T could be obtained in the camera coordinate system to define the orientation of the mirror [42].

Obviously, the location accuracy of each center position was the most influential factor in the above process, and had an effect on determining the orientation of the mirror. This paper used Chen’s [30] method to calculate the real projection center to reduce the error as much as possible, this method is more accurate and stable than using the ellipse fitting method, as the radius of inner circle increases.

3.3. Orientations of the Two Screens

The luminance nonlinearity caused by the gamma effect of the digital screen and camera is the major error source of digital phase shift technology [43]. As a result, gamma correction is necessary to reduce the influence for the following calibration process. This paper adopted a direct, simple, and effective method by encoding initial fringe pattern based on Wang’s paper [44]. Then, horizontal and vertical encoded sinusoidal fringe patterns with the optimum fringe numbers were subsequently generated and displayed on LCD₁^′ and LCD₂ to determine the orientations of the two LCD screens in the same camera coordinate system. However, the fringe patterns were reflected by the reference mirror and captured by the CCD camera from another position for post-processing. Because phase data were calculated from the captured sinusoidal fringe patterns, they were insensitive to the imaging lens being out of focus. The phase of each pixel on the two LCD screens could be known using certain fringe numbers.

Absolute phase data were calculated using the optimum three-fringe number selection method and a four-step phase shifting algorithm. However, the phase data in the location of the circles could not be obtained correctly due to the low fringe modulation, as illustrated in Figure 3a for the horizontal fringe pattern. Because all phase data were located on the plane mirror, a plane could be fitted by using the obtained correct phase data without circles. This process can eliminate the influence of the inaccurate phase data in the location of the circles, as shown in Figure 3b.

A two-dimensional feature point matrix can be designed for the LCD screen to match the calibration requirements, as illustrated in Figure 4. The world coordinate and corresponding phase data of each feature point can be determined by following equations:

$$\frac{x_w}{{\varphi }_x}=\frac{n_x\times p}{2\pi },$$

(7)

$$\frac{y_w}{{\varphi }_y}=\frac{n_y\times p}{2\pi },$$

(8)

where p is the size of LCD screen pixel pitch, n_x is the number of LCD screen pixels per horizontal fringe, and n_y is the number of LCD screen pixels per vertical fringe. According to Equations (7) and (8), one camera pixel can uniquely locate its corresponding physical position (x_w, y_w) in the world coordinate system based on its phase value (φ_x, φ_y).

As a result, the position of each feature point in the unwrapped phase map was determined using a phase matching algorithm. The absolute phase of every pixel was invariable in the image captured from different camera viewpoints. The phase data can thus be used to build the relationship of the feature points between the world coordinate system and camera coordinate system.

The orientations of the two LCD screens in the camera coordinate system were determined from the parameter matrix of the LCD screen and the calibrated internal parameters of the CCD camera.

3.4. Calculation of the two system parameters

In order to calculate the distance, the orientations of the reference mirror and the two screens LCD₁ and LCD₂ needed to be transformed into the same camera coordinate system. After obtaining the orientations of the reference mirror and screens LCD₁ and LCD₂ in the same camera coordinate system, the coordinate transformations for the LCD screens and the mirror were given by the following equations.

$$R_{MC}\times P_{MW}+T_{MC}=P_{MC}^{},$$

(9)

$$R_{L_{1}C}\times P_{L_{1}W}+T_{L_{1}C}=P_{L_{1}C}^{},$$

(10)

$$R_{L_{2}C}\times P_{L_{2}W}+T_{L_{2}C}=P_{L_{2}C}^{},$$

(11)

where, $R_{MC}$, $R_{L_{1}C}$, and $R_{L_{2}C}$ are the rotation matrices for the mirror, LCD₁, and LCD₂ from the world coordinate system to the camera coordinate system; $P_{MW}$, $P_{L_{1}W}$, and $P_{L_{2}W}$ are the world coordinates of the mirror, LCD₁, and LCD₂; $T_{MC}$, $T_{L_{1}C}$, and $T_{L_{2}C}$ are the translation vectors of the mirror, LCD₁, and LCD₂ from the world coordinate system to the camera coordinate system; and $P_{MC}$, $P_{L_{1}C}$, and $P_{L_{2}C}$ are the position coordinates of the mirror, LCD₁, and LCD₂ in the camera coordinate system.

The distances between the two LCD screens and the mirror were calculated to give the value of the two system parameters d₁ and d₂, and a plane was fitted based on the data for $P_{MC}$. In Figure 5, d₁ and d₂ are the distances between $P_{L_{1}C}$ and the fitted plane and between $P_{L_{2}C}$ and the fitted plane, respectively, allowing the two system parameters to be calibrated in the same camera coordinate system.

In order to describe the proposed calibration method of this paper, a flowchart with the whole calibration procedure is shown in Figure 6.

4. Experiments and Results

4.1. System Hardware

Calibration experiments were carried out to verify the proposed method using the developed DPMD measurement system, as illustrated in Figure 7. The setup consisted of a CCD camera, two LCD screens, a beam splitter plate, and a mirror with a matrix of hollow ring markers. The camera was the model ECO655CVGE from SVS (Bremen, Germany), and had a resolution of 2448 × 2048 pixels. Two small size LCD screens were used in the DPMD system. The two LCD screens were the model LP097QX2 from LG (Seoul, Korea) and had a resolution of 2048 × 1536 pixels. However, they were low quality and the flatness was not so good. The three angular displacement platforms had a minimum readout of 5′ from Micro-nano Optics Automation Equipment Cooperation (Beijing, China). The mirror with the matrix of hollow ring markers and the beam splitter plate were manufactured by Jiaite Photoelectric Corporation (Shenzhen, China) to match the experimental requirements.

4.2. Calibration experiments

In order to determine the internal parameters of the CCD camera, the checkerboard was placed at eighteen random positions and orientations with a large angle between the imaging axis and the board’s normal. The checkerboard size along the rows and columns had the same value of 6 mm, as illustrated in Figure 8.

At each position, the CCD camera captured the checkerboard image. Using the known checkerboard size in the eighteen captured images, the internal parameters of the CCD camera were obtained by the Camera Calibration Toolbox for Matlab [45], the results are shown in

Table 1. The calibrated internal parameters of the camera.

Coefficients	Focal Length (pixel)		Principal Point (pixel)		Radial Distortion		Tangential Distortion
	Fu	Fv	Pu	Pv	K1	K2	K3	K4
Data	10137.452	10137.811	1204.783	1011.174	−0.2712	−1.8758	−0.0002	0.0003

. The radial and tangential distortions of the following captured images were corrected by the obtained internal parameters. The average reprojection error in direction x and y was 0.0856 pixel and 0.0668 pixel, respectively, as illustrated in Figure 9.

After obtaining the internal parameters of the CCD camera, the calibration procedure included the following steps: (1) The camera captured an image of the mirror with the hollow ring marker matrix to calibrate its extrinsic parameters and determine its orientation in the camera coordinate system; (2) Two groups of orthogonal phase-shifted fringe patterns with the optimum three fringe numbers were displayed on screens LCD₁ and LCD₂. The camera captured these fringe pattern images in sequence. The absolute maps were obtained from the captured fringe patterns using a four-step phase shifting algorithm and the optimum three-fringe number selection method [46,47]. A matrix with 12 × 9 feature points was designed on the LCD screens to enable the corresponding points on the absolute map to be searched for. The orientations of the two screens were determined in the camera coordinate system based on the phase matching and fringe order constraint principle; (3) The two system parameters d₁ and d₂ were calculated based on the orientations of the two LCD screens and the reference mirror; (4) Using the known distance between neighboring markers, calibration was achieved by determining the relationship between each pixel position and its XY coordinates.

4.3. Evaluation of Results

A concave mirror, from Micro-nano Optical Corporation (Beijing, China), was measured using the calibrated DPMD system. The center of the concave surface was obtained according to the measured 3D shape data by fitting all the points. The radius could be calculated as a true value. The difference between every point and the obtained true radius was calculated. The average value of all the differences was the calculated systematic error of 26 µm. To evaluate the calibrated system parameters quantitatively, the calibrated DPMD system was also tested using a standard artificial step with specular surfaces, the photos of the objects are illustrated in Figure 10. The specular step was designed and manufactured with known distances between neighboring steps, as shown in the first column in

Table 2. Evaluation results for the artificial specular step (units: mm).

Calibration Distance	Measured Distance	Absolute Error	Root Mean Square Error
3.987	3.965	0.018	0.022
7.025	7.044	0.015	0.019
5.006	5.030	0.022	0.024
6.099	6.079	0.021	0.020

. Twelve sinusoidal fringe patterns with fringe numbers of 81, 80, and 72 were generated and displayed sequentially on the two LCD screens. The fringe patterns reflected by the specular objects being tested were deformed by the slope and shape of the measured surface. The reflected fringe patterns captured by the CCD camera are shown in Figure 11a for the step and Figure 12a for the concave mirror. Figure 11b and Figure 12b illustrate single absolute phase maps for the specular step and the mirror, respectively. The absolute phase map was then converted into depth data, and the calibrated parameters d₁ and d₂ were used to obtain the 3D shape data for the specular step and the concave mirror, as shown in Figure 11c and Figure 12c. Figure 11d and Figure 12d show the profile maps of the specular step and concave mirror being tested.

All the points measured on one step surface were fitted onto a plane to calculate the distance between neighboring steps. The measured distance between neighboring steps was calculated using the average of the distance from all the points obtained on the other step surface to the fitted plane. The measured distance, the absolute error (absolute difference between the average distance measured and the calibration distance), and the standard deviation are listed in the second to fourth columns of Table 2.

The experimental results demonstrate that the proposed calibration method can reconstruct the 3D shape of specular objects with high precision and reliability. The absolute error in the distance between neighboring steps was below 22 µm, while the value is 52 µm by using the existing calibration method [37].

5. Conclusions

This paper proposes a novel method of calibrating system parameters for the PMD technique. The method uses a reference mirror with a matrix of hollow ring markers to determine the orientations of two LCD screens in the same camera coordinate system, by displaying orthogonal sinusoidal fringe patterns on the screens using phase data. This allows the two system parameters d₁ and d₂ to be calibrated in the camera coordinate system. The calibrated DPMD system was tested with an artificial specular mirror with known distances between steps, and a standard concave mirror of known radius. The experimental results validated the effectiveness and reliability of the proposed calibration method for the system parameters.

The proposed calibration method has the following advantages: (1) The parameters of the DPMD system are calibrated using phase information calculated from sinusoidal fringe patterns. The digital phase calibration board has been designed to calibrate the position of LCD screens instead of geometric feature points. Because the plane and screen are different distances from the imaging device, they cannot be clearly captured, given the limited DOF of the lens. Using the phase target can solve this problem efficiently; (2) The calibrated DPMD system can measure specular objects that have discontinuous and/or isolated surfaces; (3) The mirror surface is selected as the reference plane, making it unnecessary to replace components, and ensuring an accurate reference position. Also, the proposed calibration method can obtain the two parameters in the same way, so that it does not need to remove the board in the measuring system after calibration, which is more convenient.

Of course, this paper also considered several factors which would affect the accuracy of whole calibration process. Firstly, Chen’s method was used instead of the traditional ellipse fitting method to calculate the real projection center to reduce the error as much as possible, as it has been proven to have a higher accuracy. Secondly, gamma correction by encoding the initial fringes was applied to reduce the influence of nonlinear distortion. However, the influence of a finite width of the beam splitter, the accuracy evaluation of phase matching principle and other error sources for calibration also need to be studied in further experiments.