A Two-Stream 3D-CNN Network Based on Pressure Sensor Data and Its Application in Gait Recognition

Abstract

Accurate diagnosis of Parkinson’s disease (PD) is challenging in clinical medicine. To reduce the diagnosis time and decrease the diagnosis difficulty, we constructed a two-stream Three-Dimensional Convolutional Neural Network (3D-CNN) based on pressure sensor data. The algorithm considers the stitched surface of the feet as an “image”; the geometric positions of the pressure sensors are considered as the “pixel coordinates” and combines the time dimension to form 3D data. The 3D-CNN is used to extract the spatio-temporal features of the gait. In addition, a twin network of 3D-CNN with shared parameters is used to extract the spatio-temporal features of the left and right foot respectively to further obtain symmetry information, which not only extracts the spatial information between the multiple sensors but also obtains the symmetry features of the left and right feet at different spatio-temporal locations. The results show that the proposed model is superior to other advanced methods. Among them, the average accuracy of Parkinson’s disease diagnosis is 99.07%, and the average accuracy of PD severity assessment is 98.02%.

1. Introduction

Parkinson’s disease (PD) is a typical neurodegenerative disease that is not fatal in itself, but patients often encounter symptoms such as movement disorders that will greatly affect their quality of life [1,2,3]. Currently, the diagnosis of PD depends largely on the patient’s self-description and the clinician’s subjective judgment, which may cause difficulties in the diagnosis of diseases [4]. Because the early symptoms of PD are usually mild and often go unnoticed, which delays diagnosis and intervention. Therefore, it is necessary to find an effective auxiliary diagnostic method [5].

PD is characterized by the degeneration of nerve cells responsible for producing the neurotransmitter dopamine. The loss of neurons in the basal ganglia or the impairment with the other neighbors can lead to a decrease in dopamine concentration. The reduction in dopamine levels directly impacts muscle activity, which leads to the weakening of the patient’s exercise capacity [6,7]. Patients with PD often have classical symptoms such as tremors, postural instability, muscle stiffness, and slow movement, which alter the gait mode of early-stage patients. Gait is a dynamic representation of human walking, intricately involving the synchronized orchestration of numerous muscle groups. In individuals with PD, gait often exhibits distinctive traits such as short steps and inconsistent patterns. Through a comprehensive analysis of a patient’s gait, medical professionals can gain valuable insights into their motor capabilities, balance, and motor coordination. At the same time, gait analysis can also help monitor the disease progression and provide an important reference for making individualized rehabilitation plans. Therefore, it is of great significance to assess the motor capacity of patients with PD by gait analysis [8,9].

With the rapid development of sensing technology, many gait acquisition systems have been studied to provide gait data. First, the inertial sensor system can accurately capture and analyze the patient’s gait data. By recording the information of acceleration and angular velocity, gait parameters such as stride length, stride frequency, and stride period can be obtained [10,11]. The gait parameters can provide doctors with accurate quantitative information to aid in diagnosis and treatment. Secondly, the camera equipment can clearly record the video of the patient walking. The abnormal gait, such as unstable posture and irregular pace, can be identified through computer vision technology and image analysis methods, which can help the early diagnosis of patients [12]. In addition, the development of biological sensors makes the monitoring of physiological signals more accurate and convenient. For example, heart rate variability (HRV) can be used as a non-invasive measurement to assess the state of the patient’s autonomic nervous system [13]. However, camera devices are susceptible to environmental factors, and biosensor devices require professional installation and operation. On the other hand, the inertial sensor system has the advantages of high measurement accuracy, real time, and a wide application range in data acquisition. These advantages make the inertial sensor system play an important role in the early diagnosis and rehabilitation training of diseases. In the future, with the continuous development of technology, the inertial system is expected to make more breakthroughs in the field of gait diagnosis.

With the instant development of neural networks, deep learning has been widely used in various fields. Using machine learning algorithms to model clinical information such as biological signals and image data, it is possible to identify whether the subject has PD and predict the severity of the disease. In the field of classification and recognition of time series signals, 1D-CNN is commonly used to extract temporal features of signals. However, for multiple time series containing spatial relationships, 1D-CNN does not effectively extract the interrelationships between multiple signals [14]. On the other hand, the 3D-CNN network can not only extract the temporal information of data but also effectively extract the spatial information between multiple sensors, so it is adopted in this paper [15]. In addition, the loss of neurons in the basal ganglia or the impairment with the other neighbors can lead to asymmetric muscle control, which alters gait patterns. Considering symmetry as an important gait evaluation index, it can provide valuable information on the recovery status, but existing methods for extracting symmetry-related features are simplistic and have limited application scope [16]. In this regard, this paper makes the following contributions.

• Multiple pressure output is considered as “image” data, and the gait cycle is regarded as 3D “video” data. At the same time, we expand the 1D convolution kernel in space to obtain a 3D convolution kernel, which can extract both spatial and temporal information, providing a new idea for the aided diagnosis of PD patients.
• We propose a method for extracting gait symmetry features. The pressure signals of left and right feet are input into a twin 3D-CNN with the same structure and shared weights, and then the gait symmetry features on different spatio-temporal aspects are extracted, which offers an effective method for the extraction of gait symmetry.

The structure of this paper is as follows: Section 2 provides the related work. Section 3 gives the preprocessing procedure and the evaluation model. Section 4 shows the performance analysis. Finally, the conclusion is given in Section 5.

2. Related Work

In this paper, the works on PD classification are divided into two categories for introduction: manual feature extraction algorithms and deep learning algorithms.

Among the traditional methods, Daliri et al. [17] differenced the gait pressure signal along the timeline and extracted the frequency features using the Short Time Fourier Transform (STFT), obtaining a diagnostic accuracy of 91.2% on the SVM. Sarbaz et al. [18] used Petrosian dimensional features, step-length signal variance, regression error, and the mean and variance of phase signals as features and used the nearest mean scaled classifier to classify the features, which achieved an accuracy of 95.6% in the classification task of PD patients and healthy subjects. Cuzzolin et al. [19] encoded the IMU gait sequence sampled during walking as a hidden Markov model (HMM) to extract its dynamic information, and the distances between the HMMS were classified by a standard nearest neighbor classifier, achieving an accuracy of 85.51%. Ghaderyan et al. [20] proposed a gait signal analysis method based on time-varying singular value decomposition. Specifically, the signal is separated into different components, and the most relevant components are selected to quantify the interlimb symmetry in the singular value space, which has achieved good results in the diagnosis of Parkinson’s disease. Veeraragavan et al. [21] extracted 34 features such as gait period, standing time, swing time, and stride length of the two feet and classified them with an artificial neural network (ANN). The average accuracy of binary classification reached 90.9%, and the accuracy of multi-class classification reached 73.0%. Even though the above-mentioned algorithms obtained some useful information, they did not consider the importance of multi-sensor fusion and gait symmetry features for PD recognition. In addition, most of the existing work has only diagnosed Parkinson’s disease but not assessed the severity of the condition.

With the rapid development of neural networks, many deep-learning methods have been proposed in the field of PD diagnosis and severity rating. For example, Hoang et al. [22] concatenated all the gait signals into a 2D image and used 2D-Convnet to gain the spatial information. Then, the feature maps were flattened to one dimension to gain the temporal information of the gait. Setiawan et al. [23] referenced audio signal processing methods where they converted 16 sensor signals into spectrogram images and then used a pre-trained model (e.g., AlexNet) to diagnose PD patients. Jane et al. [24] proposed a Q-backpropagation time delay neural network classifier that constructed a time classification model to predict the severity of PD subjects by analyzing the instability of their walking patterns. The experimental results show that the classification accuracy of the three subdatasets reaches 91.49%, 92.19%, and 90.91%, respectively. Nguyen et al. [25] first use a transformer network to gain the time series information of each pressure sensor and use a fully connected network to gain the spatial information between each sensor, achieving an accuracy of 95.2% in the Parkinson’s disease diagnosis task. Dong et al. [26] proposed a static-dynamic temporal network to extract the forcing-transfer feature and temporal feature of gait and achieved an accuracy of 96.7% in the binary classification task. Maachi et al. [14] used 1D convolutional neural networks to extract gait features and separately processed 18 one-dimensional signals from multiple pressure sensors, achieving good accuracy. Although deep learning has been widely used in bioinformatics, its application for assessing the severity of PD remains limited, and the extracted gait data tends to be elementary, so it requires further research in this area.

3. Methods

The target of this paper is to classify PD based on gait pressure data. The overall framework of the model is divided into two parts: the data preprocessing and the construction of two-stream 3D-CNN. Among them, the data preprocessing is used to filter and split the original VGRF signal, and the two-stream 3D-CNN is used to extract the gait information; the model structure is shown in Figure 1.

3.1. Experimental Data

In this experiment, we used data from the public dataset PhysioNet, which is composed of three PD gait sub-datasets, namely the Ga dataset [27], Si dataset [28], and Ju dataset [29]. Among them, the Ga dataset contained gait data from 29 PD patients and 18 healthy subjects, the Ju dataset included gait data from 29 PD patients and 26 healthy individuals, and the Si dataset contained gait data from 35 patients with PD and 29 healthy individuals. The three sub-datasets included a total of 93 PD patients with moderate symptoms (H&Y stage 2–3) and 73 healthy controls (CO). Each subject was asked to wear a pair of shoes with eight pressure sensors under each foot, and each subject walked for two minutes at their natural speed on a level surface. The pressure sensors located on the left foot are assigned identification numbers L1 to L8, while the corresponding sensors on the right foot are numbered R1 to R8. Assuming that a person is standing comfortably with feet parallel, the coordinates of these sensors in the plane under the feet are shown in Figure 2. The values measured by the sensors are measured in Newtons, and the outputs of all the sensors are digitized and captured at a sampling rate of 100 hz. The severity of patients is indicated by H&Y scale. Higher levels of H&Y indicate greater disease severity [4,30].

3.2. Data Preprocessing

Firstly, the low-pass filter is used for all VRGF data to remove noise interference. Secondly, this paper divides the gait cycle based on the moment of landing of the left foot, and the specific steps for dividing the cycle are as follows.

When the left foot is in the swing phase, the pressure is minimal and close to zero. However, as soon as the left foot makes contact with the ground, the pressure rapidly increases from zero to a non-zero value, and if the duration of the non-zero value exceeds 30 sampling points, it indicates that it is a sudden change point. These mutation points are taken as segmentation points of the gait cycle, as shown in Figure 3. In this way, each gait cycle can be accurately divided for subsequent analysis and research.

Human walking speed and stride size fluctuate while walking, and the duration of each gait cycle is not always the same. As the network structure requires fixed length input, this paper adopts the zero-padding method to fix the number of sampling points for all gait cycles as N = 150, the value of N is determined by considering the sample rate of sensors and the slowest walking speed. Among them, the maximum duration to complete a gait cycle is approximately 1.5 s. In summary, the sample size for each category is listed in

Table 1. The categorized data based on the healthy condition.

Category	Healthy COs	PD Patients			Merged
	0	2	2.5	3	(2, 2.5, 3)
Sample size	8135	7403	4138	1074	12,615

.

3.3. Two-Stream 3D-CNN

To improve the accuracy of PD patient diagnosis, this paper designs a two-stream 3D-CNN, in which the two-stream network is divided into a spatio-temporal information stream and a symmetry information stream, which are used to extract the spatio-temporal features and symmetry features of the gait, respectively. Then, the concatenation fusion method is used to fuse the two streams.

3.3.1. Spatio-Temporal Stream

In the field of classification and recognition of time series signals, 1D-CNN is commonly used to extract temporal information, while for multiple time series containing spatial relationships, 1D-CNN does not effectively extract the interrelationships between multiple signals.

Since the pressure points at different locations on the soles of the feet are in the same plane, meaning that the pressure values generated by each sensor at a given time are on a common plane. This paper stitched the left and right foot surfaces to form a comprehensive “image” and treat a gait cycle as a three-dimensional “video” data. 3D-CNN uses a three-dimensional convolution kernel to convolve with three-dimensional “video” data to extract spatiotemporal features. The process of 3D convolution is shown in Formula (1).

$${\mathrm{v}}_{\mathrm{i}\mathrm{j}}^{\mathrm{x}\mathrm{y}\mathrm{z}}=\tanh ({\mathrm{b}}_{\mathrm{ij}}+{\displaystyle \sum _{\mathrm{m}}{\displaystyle \sum _{\mathrm{p}=0}^{P_{\mathrm{i}}-1}{\displaystyle \sum _{\mathrm{q}=0}^{Q_{\mathrm{i}}-1}{\displaystyle \sum _{\mathrm{r}=0}^{R_{\mathrm{i}}-1}{\omega }_{\mathrm{ijm}}^{\mathrm{pqr}}{\nu }_{(\mathrm{i}-1)\mathrm{m}}^{(\mathrm{x}+\mathrm{p})(\mathrm{y}+\mathrm{q})(\mathrm{z}+\mathrm{r})}}}}})$$

(1)

where $R_{\mathrm{i}}$ is the size of the 3D kernel along the temporal dimension, $P_{\mathrm{i}}$ and $Q_{\mathrm{i}}$ are the height and width of the kernel, respectively. ${\omega }_{\mathrm{ijm}}^{\mathrm{pqr}}$ is the value of the 3D convolution kernel of the m-channel located at (p,q,r), ${\mathrm{V}}_{\mathrm{ijm}}^{\mathrm{pqr}}$ is the value of the jth channel of the i-th layer located at $(\mathrm{x},\mathrm{y},\mathrm{z})$, and ${\mathrm{b}}_{\mathrm{ij}}$ is the bias of the channel, and $\tanh (\cdot )$ is the activation function.

To increase the receptive field and extract multi-scale features, the 3D-CNN uses multi-channel to extract gait features. In this paper, we introduce the Squeeze-and-Excitation (SE) module to learn the weights of each feature channel, strengthen useful features and suppress useless features, and improve the recognition accuracy of the model [31]. In the SE module, the input feature map per channel is firstly compressed into a real number by global average pooling, which is used to represent the global distribution of responses on the feature channels. The statistical information $\mathrm{z}\in R^{\mathrm{c}}$ is generated by compressing over the spatial dimension $W\times H\times L$, where the feature vector of the c-th channel of z is computed by the following equation.

$${\mathrm{z}}_{\mathrm{c}}=F_{\mathrm{sq}}({\mathrm{u}}_{\mathrm{c}})=\frac{1}{W\times H\times L}{\displaystyle \sum _{\mathrm{i}=1}^H{\displaystyle \sum _{\mathrm{j}=1}^W{\displaystyle \sum _{k=1}^L{\mathrm{u}}_{\mathrm{c}}(\mathrm{i},\mathrm{j},\mathrm{k})}}}$$

(2)

where $F_{\mathrm{sq}}(\cdot )$ is the Squeeze operation and ${\mathrm{u}}_{\mathrm{c}}$ denotes the feature map of the c-th channel.

To assign weights to each channel and limit module complexity, two fully connected (FC) layers and two sigmoid functions are employed. The importance of individual channels is expressed as follows:

$$\mathrm{S}=F_{\mathrm{ex}}(Z,W)=\sigma (\mathrm{g}(Z,W))=\sigma (W_2\delta (W_{1}Z))$$

(3)

where $\sigma (\mathrm{x})$ refers to the Sigmoid function, $\delta (\mathrm{x})$ refers to the ReLU function, $W_1$ and $W_2$ represent two fully connected layers. The final output is obtained by multiplying the input channel with the corresponding weight:

$${\tilde{x}}_c=F_{scale}(u_c,s_c)=s_c\cdot u_c$$

(4)

where $\tilde{X}=[{\tilde{x}}_1,{\tilde{x}}_2,\cdot \cdot \cdot ,{\tilde{x}}_c]$, and the ${\tilde{x}}_c$ is obtained by multiplying the feature graph ${\mathrm{u}}_{\mathrm{c}}\in R^{H\times W}$ and the weight coefficient of the corresponding channel.

3.3.2. Symmetry Information Flow

During walking, the two feet move alternately, so the phase of the two-foot signals will differ by about half a gait cycle. To compare the difference between the two-foot signals at the same phase, the two-foot signals are shifted to make the phase of the two-foot signals uniform. The specific steps of parallel moving of signal are as follows:

When the position of the sole does not make contact with the ground, the pressure is zero, and when it lands on the ground, the pressure value changes from zero to a non-zero value. With the simple threshold method, these mutation points can be detected and used as the start point of the signals, as shown in Figure 4.

The gait pressure signals of the left and right feet are input into two sub-networks with the same structure and shared weights, and the spatio-temporal features of the left and right feet are extracted using 3D-CNN. After the features are flattened, two feature vectors with the same length are obtained. The corresponding positions of the two eigenvectors are subtracted to obtain a symmetry eigenvector, which contains the symmetry feature of the left and right feet at the same phase. The whole process is represented as follows:

$$\mathrm{g}(L,R)=\mathrm{f}(L)-\mathrm{f}(R)$$

(5)

where $L$ refers to the pressure signal of the left foot, $R$ refers to the pressure signal of the right foot, $f(\cdot )$ refers to the convolutional model for extracting features, and $\mathrm{g}(L,R)$ refers to the symmetry features of the left and right feet. The feature extraction layer $f(\cdot )$ adopts a similar structure to C3D [32]. The entire 3D-CNN has three convolutional layers, and each convolutional layer is followed by a ReLU nonlinear activation layer and a pooling layer.

4. Experimental Analysis

4.1. Performance Analysis for the Diagnosis of Patients with PD

To minimize the overfitting problem of the classifier, this paper uses five-fold cross-validation to evaluate the model performance, the dataset is randomly divided into five equal subsets, in each experiment, four subsets are used to train the classifier model, and the other subset is used for testing; the final result is taken as the average of the five experiments. The results show that the model achieves an accuracy of 99.07% in the binary classification problem, which proves that the model can effectively detect PD gait. In addition, the model achieves good performances in terms of sensitivity (Sen), specificity (Spe), as shown

Table 2. Comparison with other techniques for binary classification.

Model	Sen (%)	Spe (%)	Acc (%)
TRAD1 [33]	92.21	91.79	92.08
TRAD2 [34]	97.77	97.02	97.54
1D-CNN [35]	96.34	96.13	96.28
LSTM [35]	98.90	98.46	98.76
CNN-LSTM [36]	98.80	98.05	98.57
TRAD3 [37]	98.92	98.63	98.80
Spatiotemporal stream Convnet	99.32	97.36	98.69
Two-stream model	99.57	98.75	99.07

. The Acc, Sen, and Spe are defined as follows.

$$A\mathrm{cc}=\frac{TP+TN}{TP+TN+FP+FN}$$

(6)

$$S\mathrm{pe}=\frac{TN}{TN+FP}$$

(7)

$$S\mathrm{en}=\frac{TP}{TP+FN}$$

(8)

where True Positive (TP) refers to the number of individuals correctly identified as patients, and True Negative (TN) refers to the number of individuals correctly identified as healthy. False Positive (FP) indicates the number of individuals incorrectly identified as patients, while False Negative (FN) represents the number of individuals incorrectly identified as healthy.

Figure 5 shows that with the increase in iteration number, the average error loss shows an overall decreasing trend. When the number of iterations reaches 20 times, the loss shows a trend of constantly converging to zero, indicating that the model converges quickly. As can be seen in Figure 5b, the recognition accuracy gap between validation and training is small, indicating that the model effectively avoids the overfitting problem.

Finally, to assess the effectiveness of the proposed model in the diagnosis of PD, Table 2 provides a comparison between the proposed approach and other state-of-the-art methods including four deep learning-based models (1D-CNN, DALSTM, LSTM, and CNN-LSTM) as well as two traditional classification schemes (TRAD1, TRAD2 and TRAD3). The main reason for the performance improvement is that this paper extracts the spatio-temporal features of the gait and the gait symmetry features.

TRAD1, designed by Abdulhay et al. [33], utilizes a feature set comprising stance time, stride time, foot strike profile, and swing time for both lower limbs.

TRAD2, designed by Aşuroğlu et al. [34], utilizes a feature set that includes sixteen time-domain features (such as skewness, kurtosis, entropy, and energy) and seven frequency-domain features (such as maximum, minimum, and mean value of the FFT magnitude) for each force signal segment at different sensing positions.

TRAD3, as proposed by Wang et al. [37], extracts the features by using Phase space reconstruction (PSR), Shannon energy envelope (SEE), Tunable Q-factor wavelet transform (TQWT), and dual Q-factor signal decomposition (DQSD). For TRAD3, a combined calculation method from the field of nonlinear methods and machine learning is proposed, which has better performance than TRAD1 and TRAD2.

4.2. Performance Analysis for the Identification of the Severity of PD

Although the two-stream 3D-CNN network provides high accuracy in binary classification, the severity of PD still needs to be evaluated. Accurately identifying Parkinson’s disease patients of different severity levels can contribute to the formulation of more precise personalized treatment plans. Additionally, such a classification system can provide researchers with a deeper understanding of the disease, helping to reveal biological differences between different severity levels. In this paper, the subjects are categorized into four categories according to the H&Y scale: healthy control, H&Y2, H&Y2.5, and H&Y3. To test the performance of the model in the training and validation phases, the loss and accuracy curves of the training process are plotted, as shown in Figure 6. In addition, the confusion matrix is used in this paper to evaluate the performance of the two-stream network. The rows of the confusion matrix represent the predicted categories, and the columns are used to represent the true category labels. The last column on the right side of the confusion matrix shows the precision value for each category, and the last row is the recall value for each category. The confusion matrix for severity assessment is shown in Figure 7.

Table 3. Comparison with other techniques for multi-class classification.

Model	Se (%)	Sp (%)	Ac (%)
TRAD1 [33]	×	×	89.57
TRAD2 [34]	×	×	96.74
1D-CNN [35]	×	×	95.78
LSTM [35]	×	×	96.79
CNN-LSTM [36]	×	×	97.51
TRAD3 [37]	93.37	97.79	96.69
Spatiotemporal stream Convnet	98.49	99.04	97.42
Two-stream model	98.78	99.26	98.02

shows a comparison of the performance of the proposed method with other state-of-the-art methods using the VGRF dataset. The results show that the proposed method provides better classification results, which is due to the spatiotemporal information and gait symmetry information extracted by this model.

5. Conclusions

This paper presents a two-stream 3D-CNN based on VGRF time series signals for diagnosing patients with PD and assessing the severity of their condition. As the placement of the pressure sensors contains the geometric position, the left and right foot surfaces are stitched to form an “image”, combined with the temporal dimension to generate 3D data, which are then used to extract spatio-temporal features using a 3D-CNN. In addition, the left and right foot pressure signals are respectively input into a 3D-CNN network with weight-sharing to extract gait symmetry features in different spatiotemporal locations. After validation, the model achieved an average accuracy of 99.07% for Parkinson’s disease diagnosis and an average accuracy of 98.02% for PD severity assessment. The results show that the proposed model can improve the classification accuracy of PD classification. We hope that this 3D-CNN dual-stream network can also be applied to the identification of other diseases.