Significance of Camera Pixel Error in the Calibration Process of a Robotic Vision System

Abstract

Although robotic vision systems offer a promising technology solution for rapid and reconfigurable in-process 3D inspection of complex and large parts in contemporary manufacturing, measurement accuracy poses a challenge for its wide deployment. One of the key issues in adopting a robotic vision system is to understand the extent of its measurement errors which are directly correlated with the calibration process. In this paper, a possible source of practical and inherent measurement uncertainties involved in the calibration process of a robotic vision system are discussed. The system considered in this work consists of an image sensor mounted on an industrial robot manipulator with six degrees of freedom. Based on a series of experimental tests and computer simulations, the paper gives a comprehensive performance comparison of different calibration approaches and shows the impact of measurement uncertainties on the calibration process. It has been found from the error sensitivity analysis that minor uncertainties in the calibration process can significantly affect the accuracy of the robotic vision system. Further investigations suggest that inducing errors in image calibration patterns can have an adverse effect on the hand–eye calibration process compared to the angular errors in the robot joints.

1. Introduction

Robotic vision systems offer a promising technology solution for rapid inspection of complex scenarios such as industrial assessment, underwater inspection, remote sensing, and more. However, the reliability of system based solutions to real-world problem is substantially limited due to existence of measurement errors, hence creating a challenge for vision systems in taking accurate decisions [1].

The measurement accuracy of an object with the help of an image sensor mounted on a multi-axis-robot substantially depends on precise estimation of three geometrical transformation matrices, namely matrix A, which represents the transformation from the robot wrist (flange) coordinate frame to the robot world coordinate frame (base coordinate), matrix B, which represents the transformation from the object coordinate frame to the camera coordinate frame, and matrix X, which represents the transformation from the flange coordinate frame to the camera coordinate frame.

To estimate matrix A, the kinematics model of the robot needs to be developed first and then its kinematic parameters are to be estimated with the help of robot calibration methods. In order to estimate the kinematic parameters, several calibration methods have been introduced in the literature, examples include, circle point analysis method [2], arm signature identification method [3], screw axis identification method [4], etc. On the other hand, estimating matrix B with the help of camera calibration is a common practice and well documented in the computer vision community [5,6,7,8,9]. In the calibration process, the camera mounted on the robot flange captures images of a calibration board (checkerboard) from a number of different positions and orientations. A simple yet effective algorithm suggested by Zhang [7] is adopted in this work, which is briefly discussed in Section 2.2.

To investigate the major sources of measurement uncertainties and its overall impact on a common robotic vision system, a practical setup constituting of a camera, a six-axis robotic arm and a calibration object is used. The system calibration requires accurate estimation of the relative position and orientation of the camera with respect to the flange, with the help of existing hand–eye calibration algorithms [10,11,12,13,14,15].

The practical setup used in this paper to calibrate the robotic vision system is illustrated in Figure 1. The developed robotic vision system follows the classical approach to estimating all the three transformation matrices required in this experiment. That is, the camera mounted on the flange of the robotic arm is navigated from position i to j to capture images of the calibration object. Note that A and B are transformation matrices that can be easily known by recording the pose of the flange relative to its base for $\eta$ number of positions and capturing images of the calibration object from the same $\eta$ positions, respectively. To calculate unknown matrix X by solving system equation $AX=XB$, the minimum value of $\eta$ should not be less than three [10].

In Figure 1, let P be a transformation matrix that defines the transformation of coordinates from the hand coordinates (flange of the robot) to the robot world coordinates (robot base). The matrix P of dimension $4\times 4$ can be defined as:

$$\begin{array}{c}\hfill P=\left[\begin{array}{c}{R}_P{T}_P\\ 01,\end{array}\right]\end{array}$$

(1)

where $R_P$ is a $3\times 3$ rotation matrix, and $T_P$ is a $3\times 1$ translation vector. Let the matrix P be known for positions i and j, then matrix A can be estimated using the following expression:

$$\begin{array}{c}\hfill A=P_j^{-1}{P}_i.\end{array}$$

(2)

Similarly, let Q be a transformation matrix that defines the transformation of coordinates from the object coordinates to the camera coordinates. The matrix Q is again of dimension $4\times 4$, which is defined below.

$$\begin{array}{c}\hfill Q=\left[\begin{array}{c}{R}_Q{T}_Q\\ 01,\end{array}\right]\end{array}$$

(3)

where $R_Q$ is a $3\times 3$ rotation matrix, and $T_Q$ is a $3\times 1$ translation vector. Let the matrix Q be known for positions i and j by capturing images at both positions and performing extrinsic camera calibration, then matrix B can be estimated using:

$$\begin{array}{c}\hfill B=Q_j{Q}_i^{-1}.\end{array}$$

(4)

Although various robotic vision systems have been proposed for implementation of in-process 3D measurement with the advantages of flexibility and reconfigurability, the level of measurement uncertainties caused by system calibration errors has not been fully investigated. It is worthy to note that developing a robotic vision system for 3D assessment is a challenging task because all the three calibration processes involved in the system are prone to inaccuracies in evaluation of transformation matrices. It is well expected that inaccuracy in estimating matrices A and B can significantly impact the hand–eye calibration process. In this paper, the analyses focuses on propagation of errors from matrices A and B to matrix X. The theoretical and experimental investigations have revealed that due to difference in evaluation of transformation matrices A and B, errors in matrix B predominately impacts the overall calibration process.

Remainder of the paper is arranged as follows: Section 2 gives an overview of the experimental setup and possible sources of measurement uncertainties for hand–eye calibration based on the kinematic model of a six-axis KUKA robot manipulator. Section 3 presents some results on impact of measurement uncertainties on the calibration process. Section 4 demonstrates a case study for measuring the dimension of 3D parts. Finally, the paper is concluded in Section 5.

2. Experimental Setup and Measurement Uncertainties

The robotic vision system illustrated in Figure 1 comprises of a six degrees of freedom (DOF) robot, i.e., KUKA Agilus KR6 R900 sixx robot, a 3CCD Jai camera that is mounted on the flange of the robot, and an calibration object in the form of 63 squares ($7\times 9$) checkerboard. The specifications of Jai camera are given in

Table 1. Specifications of 3CCD Jai camera.

Specification	Quantitative Value
Resolution	1392 × 1040
Pixel size	4.65 $\mathsf{\mu }$m
Frame rate	20.8 fps
Focal length	12 mm
Sensor size	1/2″
Minimum working distance	150 mm

. The accuracy of the robotic vision system depends on the multiple factors, examples include kinematic errors, resolution of the camera, accuracy of parameters determined during camera calibration, accuracy of image processing algorithms used to detect objects etc. [16]. The types of calibrations involved in developing a robotic vision system and possible cause of uncertainties in each step are discussed below. Further, the feasible ways to overcome the effect of such uncertainties are also reported in each sub-section.

2.1. Robot Base to Flange Calibration

Robot calibration has been studied for more than three decades which brings compensation to the inaccuracy of robots. Although the technique of robot making has improved considerately during these years, the theory of robot calibration remains the same. It requires a mathematical model to be tailored to the individual robot which is to be calibrated. This is a software model of the robot and is then applied to robot paths which replaces its nominal physical model.

To calibrate and test a robot, it should be performed using high-end measurement equipment such as a laser tracker. Two types of parameters are commonly used for assessing the performance of an industrial robot: repeatability and accuracy. Modern robots can achieve a reasonably good repeatability, for example, a small robot of 5 kg payload typically has the repeatability of 0.02–0.03 mm. To improve the absolute accuracy of a robot the calibration is carried out in four steps:

1.

Determination of a mathematical model of the robot geometry and motion (kinematics and dynamics);

2.

Measurement of the position and orientation of the robot end-effector (flange);

3.

Identification of a relationship of the measurements and robot joint positions;

4.

Modification of software control commands to compensate for the determined model.

Among the aforementioned four steps, the determination of the robot model (step 1) and the identification process (step 3) are the most challenging. A complete calibration model should contain corrections of the geometric parameters (also referred as kinematic parameters), and necessary dynamics parameters, e.g., stiffness. A correction for the unknown error could be an offset or a linear approximation. In this section, applying the well known Denavit and Hartenberg (D–H) coordinate system to KUKA robot manipulator [17], the offset values associated with the parameters of KUKA robot are described.

The D–H coordinate system is helpful in estimating the kinematics of the robot describing the relative position of the flange with respect to its base. The robot kinematics description is a geometric transformation, where the parameters of kinematics describe the motion of the robot manipulator without the consideration of forces and torques. The generic form of the convention for link l of the robot manipulator illustrated in Figure 2 can be described as follows:

$$T_{l-1}^l=R_{z,{\theta }_l}{t}_{z,d_l}{t}_{x,a_l}{R}_{x,{\alpha }_l}=$$

$$\left[\begin{array}{cccc}cos\left({\theta }_l\right)& -sin\left({\theta }_l\right)cos\left({\alpha }_l\right)& sin\left({\theta }_l\right)sin\left({\alpha }_l\right)& a_{l}cos\left({\theta }_l\right)\\ sin\left({\theta }_l\right)& cos\left({\theta }_l\right)cos\left({\alpha }_l\right)& -cos\left({\theta }_l\right)sin\left({\alpha }_l\right)& a_{l}sin\left({\theta }_l\right)\\ 0& sin\left({\alpha }_l\right)& cos\left({\alpha }_l\right)& d_l\\ 0& 0& 0& 1\end{array}\right].$$

(5)

Note that the link transformation $T_{l-1}^l$ contains two basic rotations and two translations, specifically $R_{z,{\theta }_l}$ is a rotation around z-axis by ${\theta }_l$ degrees; $t_{z,d_l}$ is a translation along z-axis by $d_l$ distance; $t_{x,a_l}$ is a translation along x-axis by $a_l$ distance; and $R_{x,{\alpha }_l}$ is a rotation around x-axis by ${\alpha }_l$ degrees. The KUKA robot manipulator used in this study has six links (joints), therefore its overall kinematics transformation is described using six transformation matrices. Ideally the relationship between each successive joints $l-1$ and l or contiguous link in the form of transformation matrix $T_{l-1}^l$ for KUKA robot is given below.

$$T_0^1=\left[\begin{array}{cccc}cos\left({\theta }_1\right)& 0& sin\left({\theta }_1\right)& a_{1}cos\left({\theta }_1\right)\\ sin\left({\theta }_1\right)& 0& -cos\left({\theta }_1\right)& a_{1}sin\left({\theta }_1\right)\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(6)

$$T_1^2=\left[\begin{array}{cccc}cos\left({\theta }_2\right)& -sin\left({\theta }_2\right)& 0& a_{2}cos\left({\theta }_2\right)\\ sin\left({\theta }_2\right)& cos\left({\theta }_2\right)& 0& a_{2}sin\left({\theta }_2\right)\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(7)

$$T_2^3=\left[\begin{array}{cccc}cos({\theta }_3-\pi /2)& 0& sin({\theta }_3-\pi /2)& a_{3}cos({\theta }_3-\pi /2)\\ sin({\theta }_3-\pi /2)& 0& -cos({\theta }_3-\pi /2)& a_{3}sin({\theta }_3-\pi /2)\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(8)

$$T_3^4=\left[\begin{array}{cccc}cos\left({\theta }_4\right)& 0& -sin\left({\theta }_4\right)& 0\\ sin\left({\theta }_4\right)& 0& cos\left({\theta }_4\right)& 0\\ 0& -1& 0& d_4\\ 0& 0& 0& 1\end{array}\right]$$

(9)

$$T_4^5=\left[\begin{array}{cccc}cos\left({\theta }_5\right)& 0& sin\left({\theta }_5\right)& 0\\ sin\left({\theta }_5\right)& 0& -cos\left({\theta }_5\right)& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(10)

$$T_5^6=\left[\begin{array}{cccc}cos\left({\theta }_6\right)& -sin\left({\theta }_6\right)& 0& 0\\ sin\left({\theta }_6\right)& cos\left({\theta }_6\right)& 0& 0\\ 0& 0& 1& d_6\\ 0& 0& 0& 1\end{array}\right],$$

(11)

where ${\theta }_r\forall r\in [1,6]$ are the six axis specific angles of the KUKA robot, $a_1$ = 25, $a_2$ = 455, $a_3$ = 35, $d_4$ = −420, and $d_6$ = −80. The total transformation matrix $T_{total}$ from base to flange can be estimated by simply multiplying all the transformation matrices, that is,

$$T_{total}=T_0^1\times T_1^2\times T_2^3\times T_3^4\times T_4^5\times T_5^6.$$

(12)

The result of this calibration is a transformation matrix, which describes the movement of the robot’s flange with respects to its base from position i to j in the form of $4\times 4$ matrix comprising of sub-matrix $R_P$ and vector $T_P$. Note that the above kinematics model does not accurately define the movement of industrial robots because such robots are inherently not accurate compared to CNC machines. Robot paths suffer from inaccuracies introduced from the way multiple robot axes linked together. The manipulator arm is also prone to dynamic uncertainties as a result of its speed of travel, payload, backlashes, impact forces applied to the robot end-effector from the part etc.

The identification of a relationship of the measurements and robot joint positions is essentially an inverse displacement problem (IDP). This is the core part of robot calibration and can be solved analytically or numerically. An analytical solution is the direct inverse solution and should be used whenever possible. A numeric solution relies on iterative solutions at a cost of computation time but it generally copes well with complex equations. In this work, we have opted latter technique to estimate the offset values associated with the robot parameters. The offset values ($\mathsf{\Delta }$) associated with our KUKA robot is defined in

Table 2. Parameter values of KUKA robot with offset ($\mathsf{\Delta }$).

Kuka Link	${\mathit{\alpha }}^{\circ }\pm \mathsf{\Delta }{\mathit{\alpha }}^{\circ }$	$\mathit{a}\pm \mathsf{\Delta }\mathit{a}$ (mm)	${\mathit{\theta }}^{\circ }\pm \mathsf{\Delta }{\mathit{\theta }}^{\circ }$	$\mathit{d}\pm \mathsf{\Delta }\mathit{d}$ (mm)
1	90 − 0.013	25 + 0.009	${\theta }_1$ + 0	0
2	0 + 0.005	455 + 0.447	${\theta }_2$ + 0.005	0 + 0.010
3	90 + 0.008	35 + 0.172	${\theta }_3$− 89.990	0
4	−90 + 0.041	0 − 0.104	${\theta }_4$− 0.083	−420 − 0.341
5	90 − 0.001	0 + 0.081	${\theta }_5$ + 0.038	0 + 0.056
6	0	0	${\theta }_6$ + 0	−80 + 0

.

While performing robot base to flange calibration, the overall accuracy of the robot plays an important role. The KUKA robotic arm considered in this experiment has pose repeatability radius of ±0.03 mm [18], which can be assumed as one form of kinematic error. One of the reasons for such kinematic error is due to ageing of mechanical components, which creates a difference in the movement reported by the sensor with respect to the actual arm movement. Some non-kinematic errors that may also impact the calibration process are mechanical backlash, temperature effects on mechanical components etc. In order to assure that the KUKA robotic arm considered in this work is free from both kinematic and non-kinematic errors, the inverse kinematic is used to evaluate the joint angles which achieves a desired position and orientation of the flange relative to the base frame.

2.2. Camera to Object Calibration

Consider a 3D point P, represented by homogeneous coordinates ${(x,y,z,1)}^T$, and its corresponding 2D image projection point p represented by homogeneous coordinates ${(\alpha ,\beta ,1)}^T$. Assuming a pinhole camera model the mapping between 3D point and the corresponding 2D image point can be defined as:

$$\begin{array}{}\hfill \mathrm{(13)}& \begin{array}{cc}\hfill s{[\alpha ,\beta ,1]}^T& =M·{[x,y,z,1]}^T\hfill \end{array}\hfill \hfill \mathrm{(14)}& \begin{array}{cc}& =\left[\begin{array}{cccc}{m}_{11}& m_{12}& m_{13}& m_{14}\\ m_{21}& m_{22}& m_{23}& m_{24}\\ m_{31}& m_{32}& m_{33}& m_{34}\end{array}\right]·{[x,y,z,1]}^T\hfill \end{array}\hfill \hfill \mathrm{(15)}& \begin{array}{cc}& =K\left[R_Q\right|T_Q]·{[x,y,z,1]}^T,\hfill \end{array}\hfill \end{array}$$

where s is a scaling factor, M is the camera projection matrix of dimension $4\times 3$, K is $3\times 3$ camera intrinsic matrix, and $\left[R_Q\right|T_Q]$ is a $3\times 4$ extrinsic camera matrix, with the first three columns $R_Q$ representing rotation and the last column $T_Q$ representing translation, relating object coordinate frame to the camera coordinate frame.

Finally the 2D image point coordinates $\alpha$ and $\beta$ are calculated as:

$$\begin{array}{cc}\hfill \alpha & \hfill =\frac{m_{11}x+m_{12}y+m_{13}z+m_{14}}{m_{31}x+m_{32}y+m_{33}z+m_{34}}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill \beta & \hfill =\frac{m_{21}x+m_{22}y+m_{23}z+m_{24}}{m_{31}x+m_{32}y+m_{33}z+m_{34}}.\hfill \end{array}$$

(17)

To find the matrices K, $R_Q$ and vector $T_Q$, a camera calibration is required [7]. For that process a known 3D points in the object coordinate frame (e.g., checkerboard pattern) and their corresponding 2D image points are often used.

While performing this type of calibration, various factors can influence its accuracy, examples include: camera resolution, sensor noise, inaccurate feature point, instability of camera mounting device, accuracy of the numerical technique used etc. One of the most feasible way to estimate the uncertainty associated with this type of calibration is to evaluate the re-projection errors for each image. A re-projection error is known as a quantitative measure of accuracy, which can be evaluated by estimating the distance between a feature point detected in a calibration object (checkerboard), and a corresponding world point projected into the same image. Ideally the value of re-projection error should be close to zero.

2.3. Flange to Camera Calibration

The estimation of unknown matrix X from equation $AX=XB$ is also known as hand–eye calibration. The system equation can be formulated in two components, namely, orientational and positional components, which are defined below:

$$\begin{array}{cc}\hfill \left[\begin{array}{cc}{R}_A& t_A\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_X& t_X\\ 0& 1\end{array}\right]& =\left[\begin{array}{cc}{R}_X& t_X\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_B& t_B\\ 0& 1\end{array}\right]\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill \left[\begin{array}{cc}{R}_A{R}_X& R_A{t}_X+t_A\\ 0& 1\end{array}\right]& =\left[\begin{array}{cc}{R}_X{R}_B& R_X{t}_B+t_X\\ 0& 1\end{array}\right]\hfill \end{array}$$

(19)

Using the above expression the respective orientational and positional components can be defined as:

$$\begin{array}{cc}\hfill R_A{R}_X& \hfill =R_X{R}_B\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill R_A{t}_X+t_A& \hfill =R_X{t}_B+t_X.\hfill \end{array}$$

(21)

The hand–eye calibration technique is broadly classified in two different categories, that is, either the rotation and translation parts of the matrix X are evaluated either individually or in combined form. In the former category, Shui–Ahmad [10] and Tsai–Lenz [11] have estimated the rotation matrix and translation vector individually using a least squares fitting algorithm. The least square fitting algorithms are fast and simple, but might result in an inaccurate solution [19]. For estimating the rotation matrix of X, the concept of quaternions has been used by Zhuang et al. [20], while Chou–Kamel [21] has simplified the expression by linearising the equation ($XB=0$) before estimating the solution using singular value decomposition. Horaud–Dornaika [12] has formulated the rotation matrix with quaternions and conducted a non-linear optimisation on rotation matrix and translation vector.

In the latter category, Remy et al. [13] have proposed a formulation in which the number of unknown parameters remains constant and constrained this condition over complete data set. Andreff et al. [14] estimated the solution of system equation by using the Kronecker product on reformulated linear system. Zhuang–Shiu [22] have applied non-linear optimisation for both rotation matrix and translation vector and introduced the possibility to disregard the orientation of camera for parameter estimation. A similar approach has been presented by Fassi–Legnani [23]. Park–Martin [24] also conducted a similar nonlinear optimisation with the help of the lie group theory. To estimate the relative position and orientation between the reference frames and robot link, Lu–Chou [25] has combined the kinematic equations of rotation and translation into two simple and well structured linear systems.

The major possible sources of uncertainty in hand–eye calibration is erroneous estimation of matrices A and B. It is well known that the robotic vision systems are not completely free from errors, which make hand–eye calibration process more challenging. The hand–eye calibration algorithms discussed above perform ideally under error-free theoretical setups, but might fail to deliver the same performance for practical setups.

3. Results and Discussion

To estimate the theoretical performance of existing hand–eye calibration algorithms, the matrix B is estimated by moving the camera mounted flange of KUKA robot to a number of different positions, and an image of the calibration object is captured from each position. Further, the theoretical value of X is assumed, and the equation $AX=XB$ is used to estimate matrix A for each flange navigation from position i to j. The purpose of opting for a theoretical robotic vision setup is to estimate the impact of systematic error on hand–eye calibration algorithms. The theoretical setup is also helpful in comparing the role of uncertainties in matrices A and B on estimating matrix X. In this section, the impact of measurement uncertainties on theoretical and practical robotic vision systems are thoroughly discussed.

3.1. Error Sensitivity Analysis of Theoretical Setup

The same setup as illustrated in Figure 1 is used to calibrate the theoretical robotic vision system. Let the camera mounted on the flange of the robotic arm be navigated from position i to j and at each position an image of the calibration object is captured. Then a classical camera calibration method is used to estimate matrix Q for each position. The navigation of the camera mounted robot flange from position i to j (refer Figure 1) can be expressed by the following system equation.

$$\begin{array}{c}\hfill P_{i}X{Q}_i=P_{j}X{Q}_j.\end{array}$$

(22)

In Equation (22), the values of $Q_i$ and $Q_j$ are known with the help of the camera calibration approach suggested by Zhang [7]. Assuming the theoretical value of matrices X (denoted as $X^{\prime }$), matrix $P_j$ can be estimated from $P_i$ by using the system equation defined in Equation (23).

$$\begin{array}{c}\hfill P_j=\left(P_i{X}^{\prime }{Q}_i\right){\left(X^{\prime }{Q}_j\right)}^{-1}.\end{array}$$

(23)

After obtaining all the required matrices, the matrices A and B of the theoretical robotic vision system can be evaluated from Equations (2) and (4). The main objective for theoretical estimation of X is to analyse the performance of hand–eye calibration algorithms under systematic errors. The advantage of opting theoretical setup is that the matrix $X=X^{\prime }$ is accurately known and can be used as the reference matrix while performing uncertainty analyses by adding systematic errors on matrices A and B.

One of the most practical ways to introduce errors in matrix A is to systematically perturb the angles of robotic manipulator. After each additive uncertainty, the erroneous transformation matrix $\widehat{P}$ is evaluated with the help of the kinematics model. Similar approach is to be applied for matrix B, that is, the systematic error is introduced on perfectly detected calibration pattern in the images. After introducing a systematic error each time, the camera calibration algorithm is used to estimate erroneous matrix $\widehat{Q}$. Now the expression defined in Equation (22) is modified to:

$$\begin{array}{c}\hfill {\widehat{P}}_{i}X{\widehat{Q}}_i={\widehat{P}}_{j}X{\widehat{Q}}_j\end{array}$$

(24)

Solving this system equation gives

$$\begin{array}{cc}\hfill {\widehat{P}}_j^{-1}{\widehat{P}}_{i}X& \hfill =X{\widehat{Q}}_j{\widehat{Q}}_i^{-1}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill \widehat{A}X& \hfill =X\widehat{B},\hfill \end{array}$$

(26)

where erroneous matrix $\widehat{A}$ defines a cumulative form of kinematic and non-kinematic errors discussed in Section 2.1, while erroneous matrix $\widehat{B}$ defines the cumulative accuracy error discussed in Section 2.2. Transformation matrix X is unknown and need to be estimated with the help of hand–eye calibration algorithms for different robot positions ($\eta$). Note that to estimate X, the value of $\eta$ should not be less than three, hence we have opted $\eta =4$ and $\eta =5$ in this study.

After introducing systematic errors in transformation matrices P and Q, the system equation is solved using three popular hand–eye calibration methods. The first two methods belong to the first computation category based on the individual estimation of rotation matrix and translation vector [11,12], and the other belonging to the second computation category based on combined estimation [14].

Note that uncertainties in the angles of the robot manipulator are obtained by introducing ${\delta }_A$ error in all the six joint angles ($\theta$’s) of the robot in the form of normally distributed random numbers ($N({\mu }_A,{\sigma }_A)$) simulating noise, where ${\mu }_A$ (in degrees) ranges between $[-1,1]$ with step-size of ${10}^{-1}$ and ${\sigma }_A=0.05{\mu }_A$. Similarly, the uncertainty in the calibration pattern is obtained by introducing ${\delta }_B$ error in all the detected image corner points $({\alpha }_i,{\beta }_i)$ in the form of normally distributed random numbers lying within the window size of $[0,5]$ having unity step-size. The concept of error introduction in one of the detected corner point is illustrated in Figure 3, which can cause due to usage of low resolution camera.

The norms of errors associated with estimation of matrix X using $\widehat{A}$ and $\widehat{B}$ are given by:

$$\begin{array}{c}\hfill U_R=\frac{{\sum }_{i=1}^n{\sum }_{k=1}^o\left|\right|R_{X^{\prime }}-{\widehat{R}}_{X_{i,k}}{\left|\right|}_F}{n\times o}\end{array}$$

(27)

$$\begin{array}{c}\hfill U_T=\frac{{\sum }_{i=1}^n{\sum }_{k=1}^o\left|\right|T_{X^{\prime }}-{\widehat{T}}_{X_{i,k}}{\left|\right|}_E}{n\times o}\end{array}$$

(28)

where n (=56) represents different combinations of flange positions, o (=1000) is simulation runs for normally distributed noise, $\left|\right|·{\left|\right|}_F$ is Frobenius norm and $\left|\right|·{\left|\right|}_E$ is Euclidean norm (in mm) obtained by solving system equation $\widehat{A}X=X\widehat{B}$. $R_{X^{\prime }}$ and $T_{X^{\prime }}$ are the rotation matrix and translation vector of theoretical $X^{\prime }$, while ${\widehat{R}}_X$ and ${\widehat{T}}_X$ are the rotation matrix and translation vector of transformation matrix X obtained after introducing uncertainty in the system. Ideally the value of $U_R$ and $U_T$ are zero, when both errors ${\delta }_A$ and ${\delta }_B$ are zero.

Figure 4 illustrates the impact of measurement uncertainties on estimating matrix X from the system equation using three different hand–eye calibration algorithms, where change in the colour from navy-blue to red represents increasing errors in estimated transformation matrix X accurately. Figure 4a–f show the sensitivity of three hand–eye calibration methods to uncertainty in both matrices A and B. The uncertainties in matrix A is introduced by adding normally distributed noise to all the six axes present in the robot kinematics model, while uncertainty in matrix B is introduced by adding noise to all the perfectly detected calibration points in the image.

Based on three hand–eye calibration algorithms, the observations suggest that inaccuracies in matrices A and B have adverse impact on correct estimation of X, especially for the case of smaller $\eta$ as shown in Figure 4. It is seen that when ${\delta }_A\ne$ 0 and ${\delta }_B=0$, Horaud–Dornaika and Tsai–Lenz methods are able to estimate $R_X$ accurately, while Andreff et al. method is not able to find the true value of $R_X$ and the magnitude of error function $U_R$ increases as ${\delta }_A$ increases. This observation suggests that both methods belonging to the first computation category of hand–eye calibration algorithm are expected to estimate more accurate value of $R_X$.

On the other hand, when ${\delta }_A=0$ and ${\delta }_B\ne 0$, then none of the methods are able to estimate $T_X$ accurately. However, out of the three hand–eye calibration methods, the performance of the Horaud–Dornaika and Tsai–Lenz methods are better than the Andreff et al. method. One of the reasons behind such difference in the performance of two computation categories of the methods is an additive effect of propagated errors from rotation matrix to translation vector in the Andreff et al. method. Further, the Andreff et al. method evaluates matrix X by linearising the system equation. Although linearising the system equation can reduce the complexity of the system and can estimate matrix X in lesser time, such process is expected to reduce the useful constraints of the equation which can result in a wrong estimation of the solution.

A close observation of Figure 4a–f reveals that introducing a minimal error of ${\delta }_B=1$ has significantly impacted the accurate estimation of $R_X$ and $T_X$. Note that ${\delta }_B=1$ resembles an area of $3\times 3$ pixel block, which is coming out to be 194.6 $\mathsf{\mu }$m${}^2$, where each pixel size is 4.65 $\mathsf{\mu }$m (refer Table 1). Hence, it can be asserted that introducing angular errors in the robot joints impacted the hand–eye calibration process, but the magnitude of error is not significantly large. On the other hand, inducing errors in image pixel estimation (such as in the case of low resolution camera setup) has significantly affected the hand–eye calibration process. Nevertheless, such error can be significantly reduced by increasing the number of image captures at different robot positions, as evidenced by the differences between the mesh plot for $\eta =4$ and the solid surface for $\eta =5$ in Figure 4 for all three methods. Note that when the errors in both matrices A and B are zero, then all the three hand–eye calibration algorithms are able to estimate the correct value of X.

3.2. Error Sensitivity Analysis of Practical Setup

The theoretical study of error sensitivity analysis is extended to the practical setup as illustrated in Figure 1. Dissimilar to the theoretical setup, the true value of X is unknown, which needs to be estimated with the help of hand–eye calibration algorithms. The norms of errors associated with estimation of matrix X with the help of practically estimated A and B are given by:

$$\begin{array}{c}\hfill U_R=\frac{{\sum }_{i=1}^n{\sum }_{k=1}^o\left|\right|R_X-{\widehat{R}}_{X_{i,k}}{\left|\right|}_F}{n\times o}\end{array}$$

(29)

$$\begin{array}{c}\hfill U_T=\frac{{\sum }_{i=1}^n{\sum }_{k=1}^o\left|\right|T_X-{\widehat{T}}_{X_{i,k}}{\left|\right|}_E}{n\times o},\end{array}$$

(30)

where n (=56) represents different combinations of flange positions, o (=1000) represents simulation runs for normally distributed noise, $\left|\right|·{\left|\right|}_F$ is Frobenius norm and $\left|\right|·{\left|\right|}_E$ is Euclidean norm obtained by solving system equations $AX=XB$ and $\widehat{A}X=X\widehat{B}$. $R_X$ and $T_X$ are the rotation matrix and translation vector of X when no systematic error is introduced in the system equation i.e., when both ${\delta }_A=0$ and ${\delta }_B=0$, while ${\widehat{R}}_X$ and ${\widehat{T}}_X$ are the rotation matrix and translation vector obtained after solving uncertain system equation $\widehat{A}X=X\widehat{B}$.

Figure 5 shows the uncertainties in rotation and translation of transformation X obtained by solving the system equation using the same three hand–eye calibration algorithms. The change in the colour from navy-blue to yellow represents increasing deviation of $R_X$ and $T_X$ from ${\widehat{R}}_X$ and ${\widehat{T}}_X$ respectively. A quick glance of Figure 5 suggests good similarities in the overall behaviour with respect to that of the theoretical setup (refer Figure 4). A respective comparison of Figure 4a–d and Figure 5a–d for ${\delta }_B=0$ reveals that, for the practical setup, the hand–eye calibration methods are relatively sensitive to uncertainty and resulted into high magnitudes of errors.

A similar trend can be observed for Andreff et al. method from Figure 4e,f and Figure 5e,f, with much higher magnitudes of errors in the practical setup. One of the reasons behind such sensitive behaviour is pre-existence of errors in the image pattern estimation, which has impacted the hand–eye calibration process. It can be asserted that Horaud–Dornaika method performs way better than other methods in over-riding the robotic and camera errors in the calibration process.

4. Case Study: Measuring Dimension of 3D Parts

In this section, a 3D printed car part is considered for estimating its dimension through a robotic vision system under variable camera settings. The objective behind choosing different camera resolutions is to observe the proposed proposition in real world scenario which can replicate error in image pixels by capturing low resolution image.

Figure 6a shows an image of a car door with a planner edge-to-edge dimension of $2.37\times 0.98$ m${}^2$ that has been captured from the top-left position occupying whole frame of a door in the image. Note that in Figure 6a, 42 colourful circular stickers of 1 cm diameter are affixed at the different door locations, which is helpful in estimating the error propagation as the pixel error increases.

To predict the door size and diameter of stickers, initially the image is manually extracted from its background and is then merged with a respective depth image, which is shown in Figure 6c. Note that in the depth image, the gray shaded region indicates the variation in the distance of the camera sensor from the local door regions. The darker the depth image, the farther the camera sensor is; hence more chances for uncertainty in pixel estimation as compared to light gray regions. Further, the variation in the shades of gray in the door region also indicates the existence of the combination of rotational and translational variation. Such variations are very common in industrial settings where large objects are inspected in the production belt.

Figure 7 illustrates the diameter of all stickers captured under three different camera resolution i.e., $1392\times 1040$, $1044\times 780$, and $696\times 520$. A falling trend in Figure 7 indicates that as the resolution of camera decreases the capability of estimating accurate size of object also decreases. It can be noted that as the resolution is dropped to 25% from its original value, the average error in estimating diameter of sticker also dropped by 21%. Further, a relation between sticker diameter estimation and distance of sticker can be established in a similar way under all three resolution settings, i.e., as the distance from sensor to object increases, the error in estimating sticker diameter decreases. Similarly, the dimension of door estimated using three camera settings, i.e., $1392\times 1040$, $1044\times 780$, and $696\times 520$ is coming out to be, $2.36\times 0.98$ m${}^2$, $2.33\times 0.96$ m${}^2$, and $2.31\times 0.95$ m${}^2$, respectively.

5. Conclusions

In this paper, the impact of measurement uncertainties on estimating the exact solution of the robotic vision system has been investigated. The solution of a system equation $AX=XB$ has been evaluated using three different types of approaches, of which two approaches are based on the individual estimation of the rotation matrix and translation vector, whilst the other one is based on combined estimation. Investigations have revealed that introducing angular errors in the robot joints has less significantly impacted the hand–eye calibration process compared to the error in image pixel estimation. The simulation results also suggest that the solution estimated by the Horaud–Dornaika method is best followed by the Tsai–Lenz method, which are comparatively less affected by robot rotation and translation uncertainties than the Andreff et al. method. The reason behind such difference in the performance of two computation categories of the methods is an additive effect of propagated errors from the rotation matrix to the translation vector in the Andreff et al. method. Further, the Andreff et al. method evaluates matrix X by linearising the system equation, which is expected to reduce the useful constraints of the equation and result in poor estimation of the solution. Hence, out of all of them, the Horaud–Dornaika method should be considered for more accurate hand–eye calibration of practical robotic vision systems.