Using an Ensemble of Deep Neural Networks to Detect Human Keypoints in the Workspace of a Collaborative Robotic System ^†

Abstract

It is suggested that the use an ensemble of deep neural networks can determine the spatial position of the operator using keypoints with a multicamera sensor system. The advantage of the algorithm is the use of a multicamera system that allows keypoints to be linked to the local coordinate system of an industrial robotic complex. The testing of this work was made on the basis of modern embedded computing hardware and software. The effectiveness of the proposed approach is demonstrated even when only a subset of key points is found in the frame, as well as when they partially overlap. A software module in Python has been developed for detecting and localizing key points of the operator and industrial manipulator. The proposed approach will make it possible to plan the robot’s trajectories for the safe execution of joint operations in one workspace. The developed algorithm will be used to predict the operator’s actions in the workspace and detect abnormal situations and possible intersections in the trajectories of the collaborative robot.

1. Introduction

The human pose estimation task is one of the most interesting areas of research. It has applications in various fields, such as games, healthcare, augmented reality and sports [1]. The operator position estimation is of great importance in collaborative robotics, since the solution of this task will increase the efficiency of robotics and expand the possibilities of its application. A real-time response is required not only in security applications [2,3], where human actions must be detected in time, but also in industrial applications where human movement is predicted, to prevent collisions with robots in shared workspaces. For these reasons, research on motion-capture systems without markers and wearable sensors has been especially in demand in recent years.

When solving the problem using computer vision methods, it is necessary to find the coordinates of each joint (arm, head, torso, etc.), called keypoints, in the video frame, and form a skeletal representation of the human body. Thus, the detection of keypoints includes the simultaneous detection of people and the localization of their keypoints.

This problem is particularly difficult due to the heterogeneity of objects that have various and potentially complex shapes, as well as the difficulties arising from background noise and partial overlaps between objects (occlusions).

Thus, the most crucial problems can be pointed out, which are [4]:

• variability of lighting conditions;
• partial occlusions and layering of objects on the scene video;
• the complexity of the human skeleton structure;
• loss of three-dimensional information that occurs as a result of observation from one point, etc.

There are several approaches to the human body modeling:

• model based on the skeleton;
• contour model;
• three-dimensional model.

Though the posture estimation has been studied for many years, this task still remains very complex and largely unsolved. There is still no universal approach that can give satisfactory results in general non-laboratory conditions.

The paper [5] considers a linear Tensor-on-Tensor regression model to predict human behavior. However, in this work, only reference points and connections of the human upper limbs are constructed, i.e., only hand movements in the working area are analyzed.

An approach to detect hazardous operator behavior was proposed in [6]. The authors simulate dangerous behavior through time series analysis to detect hazards. Unfortunately, the authors did not give examples of real testing of the proposed algorithms, and all tests were carried out only in simulation.

Industrial cobots must be able to detect the human presence, identify and determine intentions based on hand gestures, actions, etc. There is a promising tendency here towards marker-less recognition [7]. High-level motion planning is usually combined with the ability of a robot to recognize the intentions of a human partner [8,9,10].

In [11] the authors applied the promising technology ViT (Visual Transformers) and obtained the maximum score in the Pose Estimation on MPII Human Pose benchmark [12].

The approaches that use multiple sensors or camera networks appear to be promising, which increase reliability in the case of occlusion and visual field limitations.

The paper [13] proposed an approach to estimate 3D poses of multiple persons in calibrated RGB-Depth camera networks. Each single-view outcome was computed by using a CNN for 2D pose estimation and extending the resulting skeletons to 3D by means of the sensor depth. The authors presented their solution in the form of an open-source library OpenPTrack [14].

The study [15] proposed a convolutional neural network (CNN) approach for estimating human body pose using a small number of cameras, including outdoors scenes. The authors [16] proposed a purely geometric approach to infer a multiview pose from a synchronous set of 2D skeletons.

The paper [17] proposed an approach to estimate a 3D human pose from multi-view video recordings, taken with unsynchronized and uncalibrated cameras. A unique approach to self-calibrate the system using the detected keypoints was employed in the research.

It should be noted that despite the high accuracy of some approaches, they are rather resource intensive and involve the transmission of a video stream and calculations using server graphics processing units (GPUs). However, in industrial systems, the necessity to use embedded computing modules based on GPUs of lower performance as on-board computers for robots arises. The fact imposes significant restrictions on the complexity of algorithms due to their low power consumption and small size.

It is important to emphasize once again that the application possibilities of cobots will be significantly increased thanks to timely recognition, analysis and prediction of human actions based on data from a multimodal sensory system and the development of control actions, taking into account emergency situations and extreme conditions in real time. Thus, the technological solutions, including the development of software and algorithmic support for the implementation of joint work of various types of robots with a person, is a promising task.

From our point of view, the most promising is the approach that combines a multi-camera system, as well as additional sensor devices, thereby providing multimodal data processing based on neural network approaches. For this reason, this article proposes to use an ensemble of deep neural networks (NN) to determine the spatial operator posed using keypoints and to link them to the world coordinate system of the robot.

The scientific novelty of the project is in the proposed set of methods, approaches and algorithms aimed at ensuring effective interaction between the components of the operator-cobot system.

2. Problem Statement

The task of increasing the efficiency of cobot-person interactions is formulated as follows. Judging by the incoming video stream from surveillance cameras, microphones, and other sensors installed in the working area, it is necessary to recognize, localize in space and build a predictive dynamic model of the operator’s behavior, which should be further synchronized and adapted to the collaborative control system of the manipulation robot with a different number of degrees of freedom to perform joint, diverse and previously unknown scenarios.

The research deals with the problem of detecting and localizing human keypoints. Thus, the particular task solved in this article can be formulated as follows.

By the incoming video stream from several surveillance cameras, it is necessary to recognize and localize keypoints of the operator in three-dimensional space. To preliminarily calibrate the system, it is necessary to apply calibration methods “by template” and to apply computer vision methods based on deep neural networks in order to localize keypoints.

One needs to develop software in the Python environment and conduct a full-scale experiment with video scenes which come from surveillance cameras installed at a production facility.

Mathematical Formulation of the Keypoint Detection Task

The task of detecting human keypoints in an image can be formulated as a regression task.

Let there be: a set of images $\omega \in \Omega$, defined using features $x_i$, $i=1,n$, the totality of which for the image $\omega$ is represented by vector descriptions $\mathbf{\Phi }\left(\omega \right)=(x_1\left(\omega \right),...x_n\left(\omega \right))=\mathbf{x}$ and a set of values of the dependent variable $\mathbf{y}$, corresponding to them, each of which is a vector of values $\mathbf{y}=(y_1,y_2,...y_n)$, and ${\mathbf{y}}_i\in \mathbb{R}$.

A priori information is represented with a training set (dataset) $\mathbb{D}=\left(({\mathbf{x}}^j,{\mathbf{y}}^j)\right),j=\overline{1,L}$, defined using a table, each row j, contains a vector image description $\Psi \left(\omega \right)$ and the value of the target variable. Note that the training set characterizes the unknown mapping ${}^{\ast }\mathbf{F}:\Omega \to \mathbb{Y}$.

We specify the regression task to the keypoints detection task. Let there be a video stream frame ${\mathbf{I}}^t$, where t - is the number of the current frame. Judging by the available frames ${\mathbf{I}}^t$ of a continuous video stream $\mathbf{V}=({\mathbf{I}}^1\dots ,{\mathbf{I}}^t,\dots ,{\mathbf{I}}^{\tau })$ and a priori information given by the training set $\mathbb{D}=\left(({\mathbf{x}}^j,{\mathbf{y}}^j)\right),j=\overline{1,L}$ for deep learning of a supervised NN, it is required to solve the problem of image recognition: To detect the key points of an object on a video stream frame. Key points can be represented as a vector $\mathbf{y}=(y_1,y_2,...y_n)$, containing object coordinates.

Generally accepted metrics are used to evaluate the task of detecting the keypoints of a human as a performance criterion [18].

Average precision (AP), which is averaged over various Object Keypoint Similarity (OKS) thresholds is set to $0,50:0,05:0,95$.

OKS is calculated based on the distance between predicted points and ground truth points normalized to the human scale. The scale and constant of the keypoint is needed to equalize the importance of each keypoint: the location of the neck position is more precise than the position of the hips [19]: $\mathrm{OKS}=exp(-\frac{d_i^2}{2s^2{k}_i^2})$, where $d_i$ - Euclidean distance between true keypoint and predicted keypoint; s–scale. The square root of the object segment area; k–is a constant for the keypoint that controls the decline; each keypoint has its own coefficient (the circles for the shoulders and knees can be larger than for the nose or eyes). The OKS metrics only shows how close the predicted keypoint is to the true keypoint (value between 0 and 1). Perfect predictions will have $\mathrm{OKS}=1$, while the predictions for which all keypoints differ by more than a few standard deviations $s·k_i$, will have $\mathrm{OKS}\approx 0$. Mean Average precision is also calculated using OKS with thresholds of 0.50 and 0.75 (AP50 and AP75).

Percentage of Detected Joints–PDJ. A detected joint is considered correct if the distance between a predicted keypoint and a true one is within a certain fraction of the diagonal of the bounding box (diameter of the torso).

The use of the PDJ metrics implies that the accuracy of all joints is estimated using the same error threshold.

$$\mathrm{PDJ}=\frac{{\sum }_{i=1}^{n}bool(d_i<0.05)\ast diagonal}{n},$$

where $d_i$–is the Euclidean distance between the true keypoint and the predicted keypoint; $bool\left(condition\right)$–a function that returns 1 if the condition is true, 0, if it is false; n–the number of keypoints in the image.

Percentage of Correct Key-points PCK [20] considers a body part to be correctly located if the estimated endpoints of the body segments are within 50% of the segment length of their true location. PCKh-0.5 [21]. PCK modification using 50% head segment length matching threshold.

3. Problem Solution

The algorithm which is aimed to detect and localize keypoints in the world coordinate system is divided into a separate subtasks solution (Figure 1):

• Preliminary calibration of the multi-camera system and its binding to the cobot coordinate system is performed.
• Human keypoints are detected on each of the cameras.
• Aggregation of detected keypoints is performed and their coordinates in the world system are compared.

3.1. Calibration of the Multi-Camera System

Let there be a multi-camera system consisting of two cameras observing one scene. The configuration and location of the system is shown in Figure 2. The camera registers a scene containing N reference points. The task is to use the three-dimensional coordinates of the reference points $(X_i^p,Y_i^p,Z_i^p)$ and the coordinates of their projection in the camera image plane $(c_{x_i},c_{y_i})$, where $i=1\dots N$ evaluation of matrix elements $\mathbf{A}$. As a rule, a calibration object in the form of a checkerboard is formed for camera calibration, since the use of alternating black and white squares has a sharp gradient in two directions. The intersections of the checkerboard lines are used as corners.

K images for each camera are generated with a calibration object depending on the number of rotation angles N. To calculate four internal parameters $(f/w,f/h,c_x,c_y)$ where $(c_x,c_y)$ are the natural coordinates of the point, f is the distance from the optical center, $w,h$ are the dimensions along the axes $ox,oy$ and six external parameters such as $(\psi ,\varphi ,\theta )$ rotation angles and $(T_x,T_y,T_z)$ transfer parameters. The number of frames and the number of corners is calculated using the following expression: $2·N·K\ge 6+4$. To obtain the coordinates of reference points $(X_i^p,Y_i^p,Z_i^p)$ and the coordinates of their projection in the camera image plane $(c_x,c_y)$ the findChessboardCorners function for finding corners on the chessboard is used, provided in the OpenCV library (Figure 3). Using the method [22] the matrix element of camera internal parameters is evaluated:

$$\mathbf{A}=\left[\begin{array}{ccc}f/w& 0& c_{x_0}\\ 0& f/h& c_{y_0}\\ 0& 0& 1\end{array}\right].$$

The result of the function is a matrix of the camera internal characteristics $\mathbf{A}$, distortion vector $\mathbf{r}$, rotation vector $\mathbf{m}$ and transfer vectors $\mathbf{q}$.

3.2. Human KeyPoint Detection

The following approach worked out by the authors [23] based on the OpenPifPaf project [24] was used as an algorithm to detect the operator keypoints. The authors [23] proposed a number of modifications that improve the quality of keypoints detection. The underlying OpenPifPaf is a bottom-up detector based on composite fields. The architecture of the model is shown in Figure 4. The input gets an $h\times w$ image with three color channels. The neural network encoder generates PIF and PAF fields $17\times 5$ and $19\times 7$ channels. The decoder converts the PIF and PAF fields into pose estimates containing 17 joints each. Each connection is represented by x and y coordinates and a confidence score. ResNet is used as encoders [25] or ShuffleNetV2 [26].

The Part Intensity Field (PIF) and Part Association Field (PAF) blocks are $1\times 1$ convolutions followed by subpixel convolutions. These blocks are trained to detect and link key points. For architecture training, the COCO dataset was used, which can also be supplemented with synthetic data to adapt to a specific task. The method of the network retraining using only synthetic data with pretrained weights seems more promising.

To obtain synthetic data, the Unity3D engine was used, as well as images obtained from the cameras of the observed scene. This approach allows you to adapt the algorithm to certain conditions.

Figure 5 shows the results of recognition on video surveillance cameras installed on the experimental site (a), as well as on synthetic data (b). As a result, the algorithm detected key points in the COCO format.

3.3. Mapping Keypoints in 3D Space

The result of the second stage was the keypoints of the operator detected on the images of each of the surveillance cameras. It was necessary to map 2D coordinates to the points in 3D space of the world coordinate system.

There are methods to calculate the 3D position of points for stereoscopic systems consisting of two cameras, the optical axes of which are parallel, and the straight line which passes through the optical centers is perpendicular to the optical axes. In this case, to obtain a point in three-dimensional coordinates, it is necessary to calculate the reference points using the following expression:

$$x_1^1=\frac{f(X^w+\frac{b}{2})}{Z^2},x_2^2=\frac{f(X^w-\frac{b}{2})}{Z^2},y_1^1=y_2^2=\frac{f{Y}^w}{Z^w},$$

where $X^w,Y^w,Z^w$ are world coordinates of the point, $x_1,y_1$ are the projection coordinates in the image plane of the first camera, while $x_2,y_2$ for the second camera. Then the point coordinates in three-dimensional space will be as follows:

$$X^{w^c}=b\frac{(x_1^1+x_2^2)}{2(x_1^1-x_2^2)},Y^{w^c}=b\frac{(y_1^1+y_2^2)}{2(x_1^1-x_2^2)},Z^{w^c}=\frac{fb}{x_1^1-x_2^2},$$

where f is the distance from the optical center, b is the length of the straight line segment between the optical centres. In the case when the camera axes are not parallel and the direction of cameras optical center displacement is arbitrary, the calculation of the point coordinates for any camera presupposes that the following parameters should be calculated:

$${\mathbf{v}}_1=\frac{{\mathbf{A}}_{\mathbf{1}}{\mathbf{M}}_{\mathbf{1}}}{Z_1},{\mathbf{v}}_2=\frac{{\mathbf{A}}_{\mathbf{2}}{\mathbf{M}}_{\mathbf{2}}}{Z_2},$$

$$\left[\begin{array}{c}{Z}_1^t\\ Z_2^t\end{array}\right]={\left[\begin{array}{ccc}{\mathbf{v}}_1^T{\mathbf{A}}_1^{-T}{\mathbf{A}}_1^{-1}{\mathbf{v}}_1& & -{\mathbf{v}}_1^T{\mathbf{A}}_1^{-T}{\mathbf{R}}^T{\mathbf{A}}_2^{-1}{\mathbf{v}}_2\\ -{\mathbf{v}}_1^T{\mathbf{A}}_1^{-T}{\mathbf{R}}^T{\mathbf{A}}_2^{-1}{\mathbf{v}}_2& & {\mathbf{v}}_1^T{\mathbf{A}}_1^{-T}{\mathbf{A}}_1^{-1}{\mathbf{v}}_1\end{array}\right]}^{-1}\left[\begin{array}{c}{\mathbf{v}}_1^T{\mathbf{A}}_1^{-T}{\mathbf{R}}^T\\ {\mathbf{v}}_2^T{\mathbf{A}}_2^{-T}\end{array}\right]\mathbf{t},$$

where ${\mathbf{M}}_1,{\mathbf{M}}_2$ characterize the coordinates of a certain point in three-dimensional space in the coordinate system in the system of the first and second cameras, ${\mathbf{A}}_1,{\mathbf{A}}_2$ are the matrix of internal parameters of the first and second cameras. $\mathbf{R}$ is an orthogonal matrix that describes the orientation of the coordinate system of the second camera relative to the first one, $\mathbf{t}$ is the translation vector that determines the position of the optical center of the second camera in the coordinate system of the first one. The obtained parameters can be used to get a vector of 3D point coordinates for any of the cameras: ${\mathbf{M}}_1^p=Z_1^t{\mathbf{A}}_1^{-1}{\mathbf{v}}_1$, ${\mathbf{M}}_2^p=Z_2^t{\mathbf{A}}_2^{-1}{\mathbf{v}}_2$.

4. Semi-Natural Experiment

The proposed approach was implemented in Python using the pytorch library. The following computer configuration was used for testing: Intel Core i5, 8 Gb RAM, Nvidia Geforce 1080 Ti. Figure 6 demonstrates the algorithm in action in the current industrial line of the robotics center.

It has been experimentally proven that this approach allows the pose to be restored when only a subset of keypoints are in the video scene, as well as when they partially overlap.

5. Conclusions

This article solves the problem of determining the spatial position of the operator using keypoints with a multi-camera sensor system. The possibility of the proposed approach application to solve the problems of collaborative robotics has been demonstrated. The advantage of the algorithm is the use of a multi-chamber system that allows you to attach points to the local coordinate system of an industrial robotic complex. The obtained data will further be used in planning the trajectory of the robot to safely perform joint operations in the same workspace. As additional modifications, the possibility of installing a lidar to clarify the coordinates of the edges of the skeleton is being considered.

A research perspective is the development of an algorithm to predict the actions of the operator in the workspace and detect hazardous situations and possible intersections in the trajectories of a collaborative robot.