A Novel, Automated, and Real-Time Method for the Analysis of Non-Human Primate Behavioral Patterns Using a Depth Image Sensor

Abstract

By virtue of their upright locomotion, similar to that of humans, motion analysis of non-human primates has been widely used in order to better understand musculoskeletal biomechanics and neuroscience problems. Given the difficulty of conducting a marker-based infrared optical tracking system for the behavior analysis of primates, a 2-dimensional (D) video analysis has been applied. Distinct from a conventional marker-based optical tracking system, a depth image sensor system provides 3-D information on movement without any skin markers. The specific aim of this study was to develop a novel algorithm to analyze the behavioral patterns of non-human primates in a home cage using a depth image sensor. The behavioral patterns of nine monkeys in their home cage, including sitting, standing, and pacing, were captured using a depth image sensor. Thereafter, these were analyzed by observers’ manual assessment and the newly written automated program. We confirmed that the measurement results from the observers’ manual assessments and the automated program with depth image analysis were statistically identical.

1. Introduction

The motion analysis of non-human primates provides vital data of animal behavior for neuroscience studies. [1,2,3,4,5] Non-human primates serve as the important animal models for behavioral and cognitive studies because of their similarity to humans. They also serve as crucial models of degenerative brain diseases, such as Alzheimer’s disease, Parkinson’s disease, and stroke [6,7,8,9]. In all of these studies, behavioral data were essential to support the hypothesis of the studies. Since behavior and neural activity are related, it is possible to verify what kind of neural activity changes have occurred through behavior observation [10,11]. Therefore, many studies have measured non-human primate behavior through video recording or direct observation by researchers [12,13,14,15]. These conventional methods are limited in that it is difficult to reproduce the measurement results due to human errors and the subjective decisions made by individual researchers, given that the researchers did not complete all of the measurements of all of the behavioral events.

To address these limitations, behavioral research methods have been gaining traction and evolving as such. Computer-based methods have been developed to overcome the limitations of conventional methods [16,17,18,19]. There is a commercially available marker-based motion capture system with a very high accuracy; however, it is not applicable to non-human primates. Flexible skin prevents the tight, correct attachment of markers, and, even if attached, long and dense fur makes it difficult for the machine to detect markers [20]. Animals can also freely move their hands to remove markers attached to their skin as a result of their instinctive curiosity. To address these challenges, advanced markerless research methods have been developed. Markerless measurement methods based on deep neural networks (DNNs), which are calculation algorithms made by simple units consisting of one layer and then stacked in series to form deep networks, have been developed [21,22,23,24,25]. The generated skeleton model and multiple cameras were used to measure the movement of freely moving monkeys without using markers; in a recent study, motion capture was performed in a separate monkey studio with 62 cameras [9,20]. These methods construct 3D images by overlapping 2D images captured by multiple cameras, and then measure behaviors by applying this to a pre-made skeleton model. These methods are limited in that researchers have to intervene during measurements to minimize the free joint movement of monkeys and to avoid one body part covering another. Research intervention as such affects reproducibility of the measurement. In addition, multiple cameras are required for multiple angles, and dissimilar to a home cage, a specific measurement cage is also required, incurring a high cost.

The aim of this study was to develop an automated behavioral pattern analysis program for a non-human primate in a home cage based on a single depth image camera. From the depth images, it is possible to obtain spatial information about the orthogonal direction of the 2-D object plane, which is a very important factor in measuring the behavior of monkeys moving in three dimensions. We designed the program to automatically measure the monkey’s behavior by tracking and calculating the center of the object and height information obtained from the depth images. There are three types of measurable behaviors: sitting, standing, and pacing. The duration of each behavior was measured using the automated program. To verify the reliability of the program, we measured the behaviors of a total of nine monkeys and compared the results with the manual measurements of two observers. In this method, a single depth image camera was used, and measurements were performed in individual home cages in which the monkeys originally resided.

2. Materials and Methods

2.1. Development of an Automated Behavioral Pattern Analysis Program

The automated behavioral pattern analysis program, using a depth-image camera, was developed as follows. The automated analysis program for classifying behavioral patterns was mainly based on background elimination and object tracking algorithms, which include the Gaussian mixture model, morphological filtering, centroid, noise reduction, and classification algorithms. To eliminate unnecessary parts around the cage from the depth image, a 3-D region of interest was defined inside the volumetric area of the cage (width × depth × height = 50 cm × 100 cm × 84 cm).

The automated program consisted of the following procedures (Figure 1).

• Importing real time image frames: a camera (Microsoft Kinect V2 Window) we used for capturing motions of object has 2 types of camera module, one is a RGB images sensor for gathering ColorFrame and the other is depth sensor for acquiring DepthFrame images. Both were utilized as raw images and imported to the analysis computer in real time;
• Image mapping: the ColorFrame, obtained from RGB image sensor module, and Depth Frame from a depth sensor have different image resolutions. The MS Kinect V2 provides 1920 × 1080 resolution by RGB images and 512 × 424 resolution by depth images. For mapping two different resolution images, we merged ColorFrame with DepthFrame as Figure 1a. Both merged images and raw images from a depth camera were utilized in following steps;

3.

Region of interest selection: in order to select the region of interest (ROI) inside a cage in the z direction, image data between the depth camera and the top of cage denoted as “Front Barrier” and image data below the bottom of the cage denoted as “Rear Barrier” were eliminated (Figure 2);

4.

In order to select the ROI inside a cage in the z-y plane, image data outside a cage were eliminated. The removal area is shown as a grey rectangular box in which each side is denoted as “Top,” “Bottom,” “Left,” and “Right” (Figure 2). Figure 1b depicted processed images after applying step 3 and step 4 to raw images;

5.

Noise reduction: after recording 30 frames of image data, the pixels which underwent a small change equal to the specified value were removed (Figure 3a). Then, the “blur” function was used to eliminate minor noise in the image data (Figure 3b). The minor noise is defined as unnecessary pixels for object identification and disregarded for enhancing following image processing steps;

6.

Background elimination: the “BackgroundSubtractorMOG2” in OpenCV was used for separating the targeted object from raw images based on Gaussian mixture method to eliminate background (Figure 3c). We can acquire a clear object image through this processing step;

7.

2nd noise reduction: any group of pixels smaller than 5-pixel size of processing image were considered as a noise and replaced by black (0 value) pixels. Then, we applied a “Morphology method” three times from “dilate” to “erode” for obtaining clear object image. The “dilate” processing converted the positive pixels into the most bright value of the image while the “erode” processing changed negative pixels to the darkest value (Figure 3d);

8.

Creating contour and a center spot: the “findCountours” in OpenCV was used to create the contour of an object. After adopting contouring algorithm to the images, we were able to get a refined shape of the object. Then, we applied “Centroid Based” algorithm in Open CV for finding a center spot of the image;

9.

Find the representing point of an object: rhe centroid of an object was tracked in real time and the pixel of the smallest depth value inside the centroid area as the measuring point of an object was found. “The centroid area” is defined by the rectangular area which has one side measuring 5~10 pixels, and “the smallest depth value” inside the centroid area approximately represents the head of an experimental monkey (Figure 3e).

10.

Calculating behavioral patterns: based on the measurement point of an object obtained (9), we classified sitting, standing, and pacing by measuring the duration of each behavior. The classifying criteria of each pattern were defined based on being maintained for 1 s. The corresponding logical flow chart is shown in Figure 4.

2.2. Behavior Pattern Experiment and Analysis

For the behavioral pattern analysis, nine monkeys (aged approximately nine years) were used in this study. The subjects were considered “young,” according to the age classification standards for humans to macaques. All of the procedures were approved by the K Korea Research Institute of Bioscience and Biotechnology Institutional Animal Care and Use Committee. Behavioral data of monkeys in a stainless-steel-wire-mesh home cage were collected for 15 min using a Kinect camera (Figure 5). Three activities were classified: sitting, standing, and pacing. Each activity pattern is defined as shown in

Table 1. Ethogram (definition of behavior).

	Definition
	Program	Observer
Sitting	When positioned for more than 1 s below the specified height without movement	Sitting without pacing and lasting longer than 1 s
Standing	When positioned for more than 1 s above the specified height without movement	Standing without pacing and lasting longer than 1 s
Pacing	When the measuring point moves more than 1 s	Pacing lasting more than 1 s

. All of the activities were quantified for duration (in seconds). The data were also analyzed by two trained examiners for verification.

2.3. Manual Analysis Using Observer XT

To confirm the accuracy of the program, two observers manually rated the behavioral patterns using “Observer XT” software (Noldus Information Technology, Wageningen, The Netherlands) for the same measurement section as the automated program, and compared the durations of each behavioral pattern with those rated by the automated program. Observer XT is a semi-automated software used for behavioral event coding and data analysis. The user has to manually decide behavioral patterns while watching a video clip. It can perform all processes, such as target behavior setting, event coding, and result analysis (Figure 6).

3. Results

3.1. Measurement Results of Observer #1 and the Automated Program

The observer #1 and the automated program for analyzing depth images measured 535.89 s and 543.78 s for total sitting time, respectively. There were no significant differences between the 2 measurements (

Table 2. Comparison of the measurement analysis between the automated program and the observer #1.

Action	Observer #1	Program	t	p-Value
	Mean ± SE	Mean ± SE
Sitting	535.89 ± 99.92	543.78 ± 98.19	−1.268	0.240
Standing	68.00 ± 14.94	55.67 ± 15.93	1.933	0.089
Pacing	296.00 ± 100.10	304.78 ± 100.38	−1.963	0.085

Mean ± SE: mean ± standard error. t: paired t-test.

. N = 9, t = −1.268, p = 0.240). Observer #1 and the automated program for analyzing depth images measured 68.00 s and 55.67 s for total standing time, respectively. There were no significant differences between the 2 measurements (Table 2. N = 9, t = 1.933, p = 0.089). Observer #1 and the automated program for analyzing depth images measured 296.00 s 304.78 s for the total pacing time, respectively. There were no significant differences between the 2 measurements (Table 2. N = 9, t = −1.963, p = 0.085). As shown above, there was no statistical difference between the results of observer #1 and the program measurement in the action time of the monkeys’ behavioral patterns, including sitting, standing, and pacing. These two measurements were statistically identical.

3.2. Measurement Results of Observer #2 and the Automated Program

Observer #2 and the automated program for analyzing depth images measured 529.67 s and 543.78 s for total sitting time, respectively. There was no significant difference between the 2 measurements (

Table 3. Comparison of the measurement analysis between the automated program and the observer #2.

Action	Observer #2	Program	t/z	p-Value
	Mean ± SE	Mean ± SE
Sitting	529.67 ± 103.99	543.78 ± 98.19	−1.324	0.222
Standing	62.00 ± 16.76	55.67 ± 15.94	1.115	0.297
Pacing †	308.33 ± 108.80	304.78 ± 100.38	0.340	0.743

Mean ± SE: mean ± standard error. t: paired t-test. †(z): Wilcoxon signed ranks test.

, N = 9, t = −1.324, p = 0.222). Observer #2 and the automated program for analyzing depth images measured 62.00 s and 55.67 s for total standing time, respectively. (Table 3, N = 9, t = 1.115, p = 0.297). Observer #2 and the automated program for analyzing depth images measured 308.33 s and 304.78 s for total pacing time, respectively. There were no significant differences between the two measurements (Table 3, N = 9, t = 0.340, p = 0.743). As shown above, there was no statistical difference between the values of observer #2 and the program measurement in the action time of the monkeys’ behavioral patterns, including sitting, standing, or pacing. The two measurements were statistically identical (p > 0.05).

The measurement results of two observers and the automated program for analyzing depth images were revealed to be statistically identical (

Table 4. Comparison of raw data between measurement analysis from the automated program and two observers for each subject.

	Subject #	R1	R2	R3	R4	R5	R6	R7	R8	R9
Sitting	Program	690	102	536	54	751	456	891	719	717
	Observer #1	691	88	531	38	766	414	887	732	671
	Observer #2	667	46	528	14	801	408	889	734	669
Standing	Program	148	19	38	68	42	32	0	117	37
	Observer #1	138	23	41	108	40	66	6	115	74
	Observer #2	164	14	34	57	36	62	5	115	72
Pacing	Program	54	780	326	779	116	432	9	64	184
	Observer #1	71	789	321	752	95	419	7	53	155
	Observer #2	69	844	339	826	61	412	8	52	159

, Figure 7).

3.3. Statistical Analysis

The data collected in this study were statistically analyzed using SPSS Win (ver. 25.0) program. Before data analysis, Kolmogorov-Smirnov and Shapiro-Wilk tests were performed to confirm the normality of measurements. If normality was satisfied, the paired t-test was tested; if not, the Wilcoxon signed-rank test was used. The significance level for all of the statistical analyses was p < 0.005.

4. Discussion

We developed an automated analysis program for behavioral patterns of non-human primates on the basis of depth image video. The program developed in this study was able to measure selected behavioral patterns of non-human primates in a home cage with a single low-cost depth image camera. The program measured “sitting” and “standing” according to the information of the depth image, and measured “pacing” according to the movement of the center of objects.

Non-human primates are important experimental models for a wide range of scientific fields. Therefore, conducting a quantitative analysis of their behavior is of great relevance. Since primates move in three-dimensional space, have joints that move freely, and use hands, motion capture data using conventional methods are consequently very difficult to analyze. Recently, markerless motion capture systems have been developed not only for humans, but also for various experimental animals, such as flies, mice, and rats [9,10,20,21,26,27]. In previous studies, motion capture of non-human primates was achieved using a Kinect camera and skeleton image [9]. Although researcher intervention was necessary, the motion capture of primates was effectively accomplished using this method. However, these studies had the disadvantage of requiring a large number of cameras and a specific measurement space. To provide an alternative method, this study proposed measuring the behavioral patterns of primates in a private cage in which they were housed with a single depth image camera. In this study, the analysis results of the automated program were not statistically different from those measured by the two observers. Therefore, our automated program is highly reliable for measuring and classifying selected behaviors of non-human primate in their home cages.

Behavioral research is also important in many studies of degenerative brain diseases [28]. In many diseases, symptoms related to behavior, such as hyperactivity, bradykinesia, tremor, and gait disorder, progressively emerge. When developing animal models of disease, it is crucial to reproduce behavioral symptoms similar to that of humans, and it is also important to quantitatively evaluate the symptoms. These behavioral symptoms are generally diagnosed through questionnaires or observer evaluations in humans. However, since these methods are not applicable to primates, a method of evaluating behaviors that reflects the emotions and intentions of primates is needed. In many previous studies, behavioral evaluations were performed to assess the motor and cognitive abilities of animal models, and, in some cases, behavioral measurement results were used as an indicator of modeling [4,5,29]. Our automated program correctly classified selected behavior, and may be used as a powerful evaluation tool in disease models, such as Parkinson’s and ataxia, characterized by prominent behavioral symptoms. Moreover, since our cost effective system is a relatively simple and easy to install on each cage, one of applications would be monitoring multiple cages simultaneously in a large animal facility.

The automated program developed in this study could not measure the small movements of body parts, such as the hands or head. The program categorized and measured the selected large movements such as sitting, standing, pacing based on the height information obtained from the depth image and the movement speed of the center of an object. In addition, it is not able to measure the rotation of an object using the automated program in this study since the algorithm in the program tracks and only calculates the center of the object. To classify a rotation in the further study, it is necessary to detect the overall shape of an object and measure the relative movement of the front with regard to the tail.

In addition, this automated program was limited in that only a single object could be measured at a time. Social interactions are essential for species that form social groups. Since primates are animals that form social groups, it is important to identify social interactions; however, our program measured only a single individual. By continuing to address these problems, it will eventually be possible to simultaneously measure the behaviors of several primates using the program. Furthermore, only three major behaviors are automatically classified in the proposed program. Pacing can be specified as translation and circulation. Circulation should also be classified in the further study.

In this study, we proposed an automated behavioral pattern analysis program based on depth images captured by a single low-cost depth image camera. There was no significant difference in the measurement results between the automated program and the observers. The automated program categorized and measured the selected behaviors of primates such as sitting, standing, and pacing well. Therefore, it is expected that the automated program will provide an effective analytical measurement tool for the selected behavior of non-human primate in individual cage for future behavior studies.