Next Article in Journal
Semantic Segmentation of the Malignant Breast Imaging Reporting and Data System Lexicon on Breast Ultrasound Images by Using DeepLab v3+
Previous Article in Journal
Dynamic Calibration Method of Sensor Drift Fault in HVAC System Based on Bayesian Inference
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 22
Issue 14
10.3390/s22145349
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals

by
Dingyi Pei
,
Parthan Olikkal
,
Tülay Adali
and
Ramana Vinjamuri
*
Department of Computer Science and Electrical Engineering, University of Maryland Baltimore County, Baltimore, MD 21250, USA
*
Author to whom correspondence should be addressed.
Sensors 2022, 22(14), 5349; https://doi.org/10.3390/s22145349
Submission received: 13 June 2022 / Revised: 15 July 2022 / Accepted: 16 July 2022 / Published: 18 July 2022
(This article belongs to the Topic Human–Machine Interaction)
Browse Figures
Review Reports Versions Notes

Abstract

:
Brain-machine interfaces (BMIs) have become increasingly popular in restoring the lost motor function in individuals with disabilities. Several research studies suggest that the CNS may employ synergies or movement primitives to reduce the complexity of control rather than controlling each DoF independently, and the synergies can be used as an optimal control mechanism by the CNS in simplifying and achieving complex movements. Our group has previously demonstrated neural decoding of synergy-based hand movements and used synergies effectively in driving hand exoskeletons. In this study, ten healthy right-handed participants were asked to perform six types of hand grasps representative of the activities of daily living while their neural activities were recorded using electroencephalography (EEG). From half of the participants, hand kinematic synergies were derived, and a neural decoder was developed, based on the correlation between hand synergies and corresponding cortical activity, using multivariate linear regression. Using the synergies and the neural decoder derived from the first half of the participants and only cortical activities from the remaining half of the participants, their hand kinematics were reconstructed with an average accuracy above 70%. Potential applications of synergy-based BMIs for controlling assistive devices in individuals with upper limb motor deficits, implications of the results in individuals with stroke and the limitations of the study were discussed.

1. Introduction

Globally, stroke and spinal cord injury are the most common causes of upper limb paralysis. For many of these individuals, the ability to perform simple activities of daily living (ADLs) may be permanently lost. With extensive rehabilitation, gross arm movements are commonly regained. However, hand paralysis is less responsive to therapy, often resulting in permanent disability. As high as 60% of individuals with paralysis return home without functional use of their paretic hand [1]. Limitations in intrinsic recovery from paralysis necessitate the development of neuroprosthetics, wearable exoskeletons and orthotic devices, and the means to control them to attain optimal performance in prehension. Researchers including our group have studied synergy-based brain-machine interfaces (BMIs) and used them in control of hand exoskeletons as a means to offer assistive devices and rehabilitation [2,3,4]. All these studies are based on a hypothesis presented by Nikolai Bernstein [5], where the central nervous system (CNS) simplifies the control and coordination of the musculoskeletal system with high degrees of freedom (DoFs) in a low dimensional space using synergies.
Research in human sensorimotor control supports the idea that the CNS simplifies the control of high-dimensional DoFs and thereby reduces the complexity of motor control by using synergies [6,7,8]. The experimental evidence from human studies suggests that the hand grasp synergies can be decoded from invasive and noninvasive neural recordings [9,10,11]. Transcranial magnetic stimulation (TMS) research has revealed that rather than the development of new patterns, the evoked finger movements were reorganized with pre-existing primitive patterns represented in the corticospinal system [12]. The study on current source (CS) signals derived from electroencephalography (EEG) has shown that the primary motor cortex is the primary area for controlling finger movement, and characteristic differences between various finger movements were reflected in CS synergies patterns [13]. It has been demonstrated that synergies can offer insights in improving motor control and motor rehabilitation [14,15]. Therefore, developing models of synergy-based hand movements and introducing them in BMIs can be significantly useful in machine-assisted motor learning and movement rehabilitation. To allow movement flexibility and to reduce the high complexity of motor control, especially neural control, synergies have been used to provide an optimal control strategy in controlling hand prosthetics and exoskeletons [2,16].
Grasping is one of the fundamental methods of interacting with the environment around us. Non-invasive EEG provides efficient and reliable applications in many fields when paired with advanced signal processing algorithms [17], including EEG-based BMI systems. Over the past years, EEG-based BMIs were able to offer control of external devices and have played a critical role in motor control and motor rehabilitation in individuals with spinal cord injury [18,19] and stroke [20,21,22,23,24]. Most of them are machine-assisted control and learning of hand movements of activities of daily living by decoding human intent from scalp EEG signals. The decoding of EEG correlates of movement intentions has shown the efficiency of BMI-assisted stroke rehabilitation [25]. Agashe et al. decoded reach-to-grasp decoding using low-frequency of EEG [10] and successfully realized robotic hand control by amputees [26]. However, these BMIs based on multiclass intention detection rely on the imagination of basic motor tasks. Thus, researchers started to decode the movement trajectories to realize a more flexible hand grasp by reconstructing the movement trajectory of upper limb movement using time-series EEG [27,28]. It has been confirmed that it is possible to obtain both movement directions [29] as well as kinematic information from low-frequency EEG signals [30]. Mondini et al. successfully transferred offline trajectory reconstruction to continuous online robotic control [31], providing the probability of real-time natural control directly using non-invasive brain activity. Although these studies are encouraging, the high complexity of the human hands has challenged offline and online movement decoding.
One of the current challenges of BMIs is modeling the complex relationships between high-dimensional brain activity, high-dimensional movement kinematics and the underlying movement primitives. Such models can enable dexterous motor control to help restore near natural hand movements for those who lost their hand functions, and also offer new methods of neuromotor rehabilitation. In a related previous study by our group, the hypothesis that the CNS may modulate the combinations of synergies, to realize synergy-based motor control was tested and the results successfully captured the correlations between neural features and the hand kinematic synergies for each individual [32]. However, for individuals with paralysis and other movement disorders who cannot perform physical movements, it is challenging to establish such a correlation model. Therefore, this paper focused on the study of generalization of the correlation between the combination of hand kinematic synergies and corresponding neural activity from the training subject group, and reconstructed the hand kinematics for the testing subject group using only their neural activity based on the correlation model. Potential applications of neural decoding based on the synergies are expected to assist efficient BMIs in machine-assisted motor control and rehabilitation.
The remainder of this paper is organized as follows. Section 2 gives the detailed elaboration of experimental design (Section 2.1 and Section 2.2), and methodologies of synergy derivation and neural feature extraction (Section 2.3, Section 2.4 and Section 2.5). Section 3 presents the results and analysis and Section 4 discusses the implications of the study, potential applications, limitations, challenges and leaves the reader with possible future alternatives and solutions.

2. Materials and Methods

This section described experimental design and the detailed methods used to achieve the research objectives. Section 2.1 and Section 2.2 described experimental protocol and data collection. Section 2.3, Section 2.4 and Section 2.5 described computational methods behind the derivation of kinematic synergies, extraction of neural features and computation of cortical correlates of kinematic synergies.

2.1. Experimental Protocol

Ten healthy, right-handed individuals (4 males, 6 females; mean age 23.0 ± 3.1 years) participated in this study and they were asked to perform the six types of hand grasp movements representative of their activities of daily living as shown in Figure 1A. There were six objects selected to represent different grasps, including a screwdriver (tripod grasp), water bottle (cylindrical grasp), CD (lateral grasp), petri dish (spherical grasp), handle (hook grasp) and bracelet (precision grasp). At the beginning of the experiment, the subject sat and relaxed in front of the experimental table with the dominant hand flat with their palm facing down on the initial place on the table as indicated. The start and stop of the movement were indicated by auditory stimuli. Their hand movements and corresponding EEG were recorded simultaneously. At the ‘start’ beep, the subjects rapidly grasped and held the object until a ‘stop’ beep. The task timeline is provided in Figure 1B. Each object was presented in random order and each grasp was repeated 30 times, for a total of 180 grasping tasks per subject. During the recording, subjects were asked to refrain from blinking or swallowing to restrict the artifacts as far as possible. The trials with eye movements/blinks and muscle artifacts were rejected.

2.2. Data Collection

Data was collected under an approved protocol by the Internal Review Board (IRB) of the Stevens Institute of Technology. During the experiment, the hand kinematics were recorded using a CyberGlove (CyberGlove Systems LLC, San Jose, CA, USA) worn on the subjects’ dominant hand. In this study, 10 of 18 sensors were used to measure the metacarpophalangeal (MCP) joints of the thumb and four fingers, interphalangeal (IP) joint for the thumb and proximal interphalangeal (PIP) joints of the four fingers. Each subject also performed initial postures (flat and fist) to calibrate the sensors of CyberGlove. CyberGlove data was captured at a 125 Hz sampling rate using a custom-built LabVIEW (National Instruments Corporation, Austin, TX, USA) program. This custom-built LabVIEW program also provided the audio cues and sent a synchronizing wave to the EEG recording amplifiers to align hand kinematics and EEG. For neural signal acquisition, EEG was recorded by a high-density EEG cap with 32 active electrodes based on 10/20 system positions with an additional 86 intermediate positions (g.GAMMA cap, g.tec, Schiedlberg, Austria). These electrode locations are F3, Fz, F4, FC3, FC1, FCZ, FC2, FC4, C5, C3, C1, CZ, C2, C4, C6, CP3, CP1, CPZ, CP2, CP4, P3, PZ, P4, OZ covering frontal (F), central (C), parietal (P) and occipital (O) areas. Eight other intermediate locations were also recorded as shown in Figure 1C. A ground electrode was placed at the nasion (yellow) and the reference (Ag, AgCl) electrode was on the right or left ear lobe. A conductive gel (g.GAMMAgel) was used to place each electrode (g.Ladybird). Impedance was kept below 5 kOhms and checked throughout the experiment. Data were continuously captured with two amplifiers (g.USBamps) using BCI2000 [34] with a sampling rate of 256 Hz.

2.3. Derivation of Hand Kinematic Synergies

Ten subjects were randomly divided into a training set (five subjects) and a testing set (five subjects). The kinematic synergies were derived from movements recorded from the training set. The hand kinematics (joint angular velocities) from 10 joints were derived from the differential of the recorded joint angles, and kinematic synergies were extracted from all types of hand grasps according to the synergy-based hand movement model discussed in [35] and as discussed here briefly. The hand kinematics v t can be expressed as a linear combination of kinematic n synergies S t and corresponding synergy weights c (1), where the synergies contain both spatial and temporal characteristics of the hand movement. The movement can also be represented as a matrix and be decomposed using singular value decomposition (SVD, as shown in (2))
v t = ∑ j = 1 n c j S j t
V = U Σ S = C S
where V is the matrix of angular velocities with dimensions m × n, where m is the number of hand movements in training set and n is the concatenated angular velocities of 10 joints. S contains the eigenvectors, or principal components (PCs), which are considered as synergies. C is a matrix that contains weights assigned to synergy S. A total number of six synergies were selected, as provided in Figure 2, depending on the variance at an 85% of threshold. The fraction of the variance of the synergies and reconstruction performance with the selected number of synergies is illustrated in Figure 3.

2.4. Extraction of Neural Features

The baseline of the recorded raw EEG data was corrected by subtracting the mean value of resting EEG signals, and the linear trend of EEG was removed. EEG signals were then filtered to 0.1–56 Hz frequency range. Figure 4 shows the EEG modulation after the audio stimulus. Post-stimulus desynchronization occurred first, followed by post-stimulus synchronization. Neural activity gradually settles down accompanied by the end of movement two seconds post-stimulus. Thus, the dynamic changes from the two-second post-stimulus period were captured by a 500 ms wide sliding window with a 375 ms overlap. The EEG features were extracted by computing the average power of the integral of the power spectral density (PSD) estimate within each sliding window. The 500 ms width was used to obtain a higher spatial resolution in the frequency domain. Since only two seconds were selected from stimulus onset to end of the movement, the 75% overlap was to ensure as much information as possible was included. A total of 13 EEG features were extracted from each electrode. Since 32 EEG electrodes were recording the neural activity simultaneously, the principal component analysis (PCA) was applied to extract top-ranked components as neural features which shared common features across multiple electrodes.

2.5. Cortical Correlates of Kinematic Synergies

Although the synergies vary across different individuals depending on their own unique movement primitives, previous studies have shown [36,37] that spatiotemporal patterns of synergies are not only shared across a different set of motor behaviours but also carry generalized control strategies across individuals. Hence, in this study, kinematic synergies were extracted from half of the subjects, and used for the other half of the subjects, for neural decoding and synergy-based reconstruction of their movements. The decoding paradigm is shown in Figure 5. The brain and behaviour data collected from ten subjects were randomly divided into two halves with five subjects in each called the training set and the testing set. The training set was used to derive the kinematic synergies and model the linear correlation matrix between the synergy weights and corresponded neural features. In the testing phase, by using the extracted kinematic synergies and by applying the trained coefficient matrix from the training set, the synergy weights for the testing group were decoded from their neural features and their hand kinematics were reconstructed. The performance was estimated from the correlations between the recorded kinematics and decoded kinematics.
Previous studies [38,39,40] applied linear regression for modeling the hand movements, where each joint movement is independently regarded as a linear combination of neural activities in the time domain. Studies from our lab [32,34] have successfully correlated the neural features with the coefficients of the synergies. In this paper, the neural decoding model was realized using multivariate linear regression to determine the relationship between neural features and the weights of synergies from the training set:
C = X β
where C is a matrix the weights of synergies, and for each task, C is an n × m matrix, where n is the number of movement trials and m is the number of synergy weights. X represents an n × p neural feature matrix, and p denotes the number of EEG features. β is a p × m coefficient matrix. The weights of synergies from the testing set were calculated using corresponding neural features through this multivariate linear regression model, and the hand kinematics were then reconstructed by linearly combining the decoded synergy weights and the kinematic synergies extracted from the training set. Decoding accuracy between the recorded kinematics and neural decoded kinematics was measured using the Pearson correlation coefficient ( ρ ). The decoding error was defined as 1 − | ρ |. To avoid the bias from certain group of subjects, this model was evaluated with 10-fold cross-validation with shuffled subjects in the training set and testing set each fold.
In order to investigate how each synergy was modulated by a specific cortical area, the independence of the EEG electrodes used for the decoding process was evaluated. The neural features were extracted using the same procedure from each single EEG electrode and the nearest electrodes, followed by the linear projection to corresponding synergy weights. The projection matrix includes the correlation coefficients of each set of neural features with the weights of each synergy. The larger magnitude represents a higher correlation between the synergy weights and neural activity. Thus, the independence of each area was interpreted by the following model:
D n = ∑ k = 1 K ∑ m = 1 M β m k 2
where Dn represents neural independency density of the nth synergy, m is the number of regression parameters β calculated above and k is the number of hand grasp tasks. The variables and acronyms in Section 2 are defined in Table 1.

3. Results

The kinematic synergies and neural decoding model were obtained from the training group of subjects, the weights of the synergies from the testing group of subjects were calculated and their hand kinematics were decoded. Since subjects were randomly grouped, the results were verified with cross validation and the hand kinematics were reconstructed from neural representations with an average decoding accuracy of 70.8 ± 5.5% across all movement tasks and all individuals. Table 2 shows the results of the decoding accuracy represented by the correlations between the recorded kinematics and neural decoded kinematics. This illustrated the performance of each movement task and the accuracy of each individual from one of 10-fold cross-validation. Figure 6 provides the comparison of recorded and decoded kinematics patterns of ten joints (from Subject 2, spherical grasp). Results show the averaged trajectories across 30 repetitions (solid lines), where the deviation of different repetitions varies (shaded areas). The decoding model was able to tune weights differently for different movements and thereby successfully decode the diverse angular velocity patterns across different objects. The reconstruction patterns for six types of grasps are illustrated in Figure 7. The corresponding kinematic trajectories of key digits involved in all types of grasps (thumb, index and middle fingers) were predicted accurately especially in lateral grasp (object: compact disc) and precision grasps (object: bracelet).
Figure 8 provides averaged neural-based decoding errors for each joint and grasp type. Thumb, index and middle are dominant fingers in a variety of grasps, but the extent of IP/PIP joint varies depending on individual grasp natures. Thus, it can be observed in Figure 8A that the MCP joints of the thumb, index and middle are lower in error than other joints, whereas the IP/PIP joints of these three fingers are relatively poor in decoding. Surprisingly, PIP joints of the ring and pinky fingers have relatively lower errors than MCP joints, which may be due to the fact that these two fingers have more freedom in grasping. Figure 8B provides the decoding performance of six grasp types. The lowest decoding error was observed from the lateral grasp, and the tripod grasp (object: screwdriver) and spherical grasp (object: petri dish) have relatively higher errors among all grasp types.
How each synergy potentially correlates to a certain neural activation area is shown in Figure 9. Results were averaged across movement tasks and individuals from the testing set after 10-fold cross-validation. It is observed that each synergy has different topographical representations and shows diverse electrode activations. Since all subjects were right-handed, the hand synergies were highly modulated on the central and left hemisphere, covering the areas of movement intention and motion-related activation generation. Specifically, Synergies 1 to 5 were effective spatially with respect to hand area, and span across the premotor cortex, primary motor cortex and primary sensorimotor cortex. Synergies 1, 4 and 6 were largely activated in the primary motor areas and supplementary motor areas that control the dynamics of movement [41]. When performing simple movements, the primary motor cortex and the primary somatosensory cortex were activated on the contralateral hemisphere, as observed from Synergies 1, 3 and 4.

4. Discussion

The central hypothesis of this paper was that the hand kinematic synergies and a neural decoder derived from a group of healthy individuals with no movement disorders can be generalized and used in reconstructing high dimensional hand movement kinematics solely based on neural activities in individuals who cannot perform hand movements. In this paper, the study participants were first divided into two groups—training and testing. From the training group, hand kinematic synergies were derived and a neural decoder was developed. Using the synergies and neural decoder from the training group and only neural activities from the testing group, their hand kinematics were reconstructed. This supports potential applications of non-invasive synergy-based brain-machine interfaces in neuromotor control and rehabilitation of individuals with upper limb motor deficits due to stroke or spinal cord injury.

4.1. Movement Decoding from EEG

Noninvasive EEG-based BMIs have been developed to decode user’s movement intention based on the markers of active brain involvement in the preparation of the desired movement. Previously, studies have decoded hand kinematics using a linear regression model for individual joints from low-frequency EEG signals (<1 Hz) [38]. Mondini et al. [31] used low-frequency EEG signals to demonstrate offline as well as online continuous closed-loop control of a robotic arm. Global cortical EEG activity was used to predict the shape of hand during grasping [10]. Other studies have also showed that the best performance of motor imagery was identified from mu and beta frequency bands of EEG [28]. Nonlinear models were also used to achieve a robust decoding model for finger movement decoding [13]. These studies strongly support that the neural correlates of fine finger movements can be extracted from EEG signals.

4.2. Neural Representations of Synergies

Initial attempts to detect the neural bases of grasp synergies were conducted in non-human primates [42,43,44,45,46] and humans [6,8,9,47,48]. In direct relevance to this study, Gentner and Classen [8] observed structured similarities between postural synergies derived from the TMS-evoked finger movements and the postural synergies derived from voluntary movements. In a recent study [10], researchers demonstrated that they can achieve up to 60% decoding accuracies in predicting three hand grasping kinematic synergies from scalp EEG signals. In [11,49], neural responses during grasping were measured by functional magnetic resonance imaging (fMRI). The results favored kinematic synergy-based control encoded in motor cortical areas than muscle synergy-based control. Although the question is open whether muscle-synergy models or kinematic-synergy models are encoded in CNS [50], nevertheless, all studies seem to favor the presence of synergies in CNS.
In this study, the synergy-based hand kinematics were successfully decoded with an average accuracy of 70% from EEG spectral features using a multivariate linear regression model. Our model is based on the correlation between the neural features and the weights of kinematic synergies. Our hypothesis emphasizes that the synergies are present in the lower level neural and musculoskeletal systems and the weights of synergies are encoded in the higher-level neural systems. Thus, for achieving any movement, the higher-level neural systems can tweak the weights of the synergies instead of controlling individual joints. The CNS may regulate different combinations of synergies to achieve various hand grasps. The neural features used in this study were from a broad frequency range of EEG signals (0.1–56 Hz). This encompasses compensations from other bands for abnormal mu/beta waves which might be the case in individuals with stroke or other neurodegenerative diseases. Furthermore, the synergies from able-bodied individuals were generalized across individuals among the second group whose kinematics were not considered. This methodology can help in the neural decoding of hand movements in individuals with motor disabilities.

4.3. Relevance to Individuals with Stroke

As BMIs create a strong link between the brain (EEG in this study) and behavior (kinematic synergies in this study), they can be significantly helpful in motor control as well as motor rehabilitation. Thus far, several BMIs have been proposed for neural decoding of movements in individuals with stroke. However, high-dimensional movement decoding using EEG is still an open challenge. To our best knowledge, this is one of the few studies that address high-dimensional movement decoding using low-dimensional and low-resolution non-invasive EEG signals by introducing a synergy-based movement generation model. Furthermore, the performance of EEG-based movement decoders is dependent on the type of decoding algorithms used. In this study, the synergy-based algorithms were restricted to simple linear decoders that can perform well in real-time as well.
In this study, all participants were healthy. Thus, no lesions or abnormalities were present in the recorded EEG that was used for decoding the weights of synergies. Question arises whether these results are applicable in individuals with stroke. Functional reorganization of the motor system after stroke has been documented [51]. It has been shown that stroke lesion location influences the performance of the decoder [52]. Abnormal changes of mu and beta waves were observed in stroke patients [22,23,24], and premovement activity <5 Hz was diminished after acute motor stroke [53]. However, in individuals with stroke, bilateral EEG has been successfully used for movement decoding. After stroke, patients have bilateral or enhanced contralesional cortical activity during motor tasks [54]. It was also demonstrated that patients shifting brain activation towards ipsilesional areas experienced motor recovery [25]. The changes in the alpha/beta rhythms of EEG successfully assisted BMIs to detect the motor intentions in stroke patients [25] and decode upper limb movements [55]. All these studies hold promise that synergy-based movement can be decoded from scalp EEG. In the near future, the current study will be repeated on individuals with stroke.
The suppression of sensorimotor rhythms in individuals with stroke (in the range of mu 8–12 Hz and beta 18–30 Hz bands) before and during the movement could possibly affect the decoding accuracy of EEG-based BCIs. How the synergy modulation is influenced by these changes in rhythms needs further investigation. Based on recordings from healthy subjects in this study, Figure 10 illustrates neuromodulation for each synergy at different EEG rhythms. In the primary motor areas, the synergy-based decoding was strongly affected by mu and beta bands, and in the prefrontal, somatosensory and supplementary areas, the decoding was strongly affected by theta and gamma bands. Thus, considering bihemispheric broad band EEG activity could compensate for suppressed neural signals due to lesions.

4.4. Limitations and Future Research

This study demonstrated that the hand synergies (principal components that captured highest variance) and neural decoder (a multivariate linear regression model) derived from a group of healthy individuals with no movement disorders can be generalized and used in modeling high hand dimensional movements in a new group of individuals solely based on their neural activities. Only ten healthy, right-handed participants were included in the experiment, relatively limiting the diversity and stability of the linear model among a large group of individuals. In the near future, the study will be extended to a larger group of participants and will be repeated in individuals with movement disorders such as paralysis due to stroke or spinal cord injury. Additionally, most of the BMIs control assistive devices by the neural activity derived from imagined movement rather than executed movement, especially for individuals with paralysis who cannot execute a movement. Thus, the practicability and generalizability of hand kinematic reconstruction and generation based on the proposed model will be verified in the future research.

5. Conclusions

In this study, synergy-based hand movements were reconstructed using neural correlates of hand synergies found in spectral features extracted from broad band EEG activity. Although the exact cortical locations of neural representations of synergies could not be pinpointed, which could be due to the poor spatial resolution of scalp EEG, the results nevertheless achieved 70% accuracy in neural decoding of synergy-based movements. By using the kinematic synergies and a linear neural decoder developed within a group of training subjects, the hand grasping movements in a group of testing subjects were successfully decoded solely based on their neural activities. To our best knowledge, this is one of the few studies that attempted to decode high-dimensional hand movements using low-dimensional and low-resolution EEG activities. Based on the neural representations of synergies obtained from multiple frequency bands and multiple cortical areas, it appears that considering bihemispheric EEG activity combined with synergy-based movement decoding can yield functional control of synergy-based prostheses and exoskeletons. Thus, future work will focus on the verification of whether the results presented in this study will transfer to individuals with stroke. The neural decoding model presented in this paper will also be validated in a real-time synergy-based control.

Author Contributions

Conceptualization, R.V.; methodology, R.V. and D.P.; data analysis, software and visualization, D.P.; validation, D.P.; formal analysis and writing—original draft preparation, D.P. and R.V.; writing—review and editing, P.O., R.V. and T.A.; supervision, R.V. and T.A.; project administration, R.V. and T.A.; funding acquisition, R.V. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by National Science Foundation (NSF) CAREER Award, grant number HCC-2053498 and NSF Planning IUCRC Award, grant number 2042203.

Institutional Review Board Statement

The study was conducted and approved by the Institutional Review Board protocol at the Stevens Institute of Technology (Protocol No. 2015-022(20-AR6), Date of Approval: September 2020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patients to publish this paper.

Data Availability Statement

Not applicable.

Acknowledgments

The authors would like to thank the reviewers for their valuable comments that helped significantly in improving the quality of the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Tsao, C.W.; Aday, A.W.; Almarzooq, Z.I.; Alonso, A.; Beaton, A.Z.; Bittencourt, M.S.; Boehme, A.K.; Buxton, A.E.; Carson, A.P.; Commodore-Mensah, Y.; et al. Heart Disease and Stroke Statistics-2022 Update: A Report From the American Heart Association. Circulation 2022, 145, e153–e639. [Google Scholar] [CrossRef]
  2. Burns, M.K.; Pei, D.; Vinjamuri, R. Myoelectric Control of a Soft Hand Exoskeleton Using Kinematic Synergies. IEEE Trans. Biomed. Circuits Syst. 2019, 13, 1351–1361. [Google Scholar] [CrossRef]
  3. Santello, M.; Bianchi, M.; Gabiccini, M.; Ricciardi, E.; Salvietti, G.; Prattichizzo, D.; Ernst, M.; Moscatelli, A.; Jörntell, H.; Kappers, A.M.L.; et al. Hand Synergies: Integration of Robotics and Neuroscience for Understanding the Control of Biological and Artificial Hands. Phys. Life Rev. 2016, 17, 1–23. [Google Scholar] [CrossRef]
  4. Vinjamuri, R.; Weber, D.J.; Mao, Z.; Collinger, J.L.; Degenhart, A.D.; Kelly, J.W.; Boninger, M.L.; Tyler-kabara, E.C.; Wang, W. Toward Synergy-Based Brain-Machine Interfaces. IEEE Trans. Inf. Technol. Biomed. 2011, 15, 726–736. [Google Scholar] [CrossRef] [PubMed]
  5. Bernstein, N.; Bernstein, N.; Bernstein, N. The Co-Ordination and Regulation of Movements; Elsevier: Amsterdam, The Netherlands, 1967. [Google Scholar]
  6. Santello, M.; Baud-bovy, G.; Jörntell, H. Neural Bases of Hand Synergies. Front. Comput. Neurosci. 2013, 7, 23. [Google Scholar] [CrossRef] [PubMed]
  7. Tresch, M.C.; Saltiel, P.; Bizzi, E. The Construction of Movement by the Spinal Cord. Nat. Neurosci. 1999, 2, 162–167. [Google Scholar] [CrossRef] [PubMed]
  8. Gentner, R.; Classen, J. Modular Organization of Finger Movements by the Human Central Nervous System. Neuron 2006, 52, 731–742. [Google Scholar] [CrossRef]
  9. Pistohl, T.; Schulze-bonhage, A.; Aertsen, A.; Mehring, C.; Ball, T. Decoding Natural Grasp Types from Human ECoG. Neuroimage 2012, 59, 248–260. [Google Scholar] [CrossRef]
  10. Agashe, H.A.; Paek, A.Y.; Zhang, Y.; Contreras-Vidal, J.L. Global Cortical Activity Predicts Shape of Hand during Grasping. Front. Neurosci. 2015, 9, 121. [Google Scholar] [CrossRef]
  11. Leo, A.; Handjaras, G.; Bianchi, M.; Marino, H.; Gabiccini, M.; Guidi, A.; Scilingo, E.P.; Pietrini, P.; Bicchi, A.; Santello, M.; et al. A Synergy-Based Hand Control Is Encoded in Human Motor Cortical Areas. eLife 2016, 5, e13420. [Google Scholar] [CrossRef]
  12. Hirano, M.; Funase, K. Reorganization of Finger Covariation Patterns Represented in the Corticospinal System by Learning of a Novel Movement Irrelevant to Common Daily Movements. J. Neurophysiol. 2019, 122, 2458–2467. [Google Scholar] [CrossRef]
  13. Yoshimura, N.; Tsuda, H.; Kawase, T.; Kambara, H. Decoding Finger Movement in Humans Using Synergy of EEG Cortical Current Signals. Sci. Rep. 2017, 7, 11382. [Google Scholar] [CrossRef]
  14. Jarrassé, N.; Proietti, T.; Crocher, V.; Robertson, J.; Sahbani, A.; Morel, G.; Roby-Brami, A.; Ronsse, R.; Sahbani, A.; Morel, G. Robotic Exoskeletons: A Perspective for the Rehabilitation of Arm Coordination in Stroke Patients. Front. Hum. Neurosci 2014, 8, 947. [Google Scholar] [CrossRef]
  15. Roh, J.; Rymer, W.Z.; Perreault, E.J.; Yoo, S.B.; Beer, R.F. Alterations in Upper Limb Muscle Synergy Structure in Chronic Stroke Survivors. J. Neurophysiol. 2013, 109, 768–781. [Google Scholar] [CrossRef]
  16. Furui, A.; Eto, S.; Nakagaki, K.; Shimada, K.; Nakamura, G.; Masuda, A.; Chin, T.; Tsuji, T. A Myoelectric Prosthetic Hand with Muscle Synergy–Based Motion Determination and Impedance Model–Based Biomimetic Control. Sci. Robot. 2019, 4, eaaw6339. [Google Scholar] [CrossRef]
  17. Sanchez-Reyes, L.M.; Rodriguez-Resendiz, J.; Avecilla-Ramirez, G.N.; Garcia-Gomar, M.L.; Robles-Ocampo, J.B. Impact of EEG Parameters Detecting Dementia Diseases: A Systematic Review. IEEE Access 2021, 9, 78060–78074. [Google Scholar] [CrossRef]
  18. Schwarz, A.; Ofner, P.; Pereira, J.; Sburlea, A.I.; Müller-Putz, G.R. Decoding Natural Reach-and-Grasp Actions from Human EEG. J. Neural Eng. 2018, 15, 016005. [Google Scholar] [CrossRef]
  19. Muller-Putz, G.R.; Ofner, P.; Pereira, J.; Pinegger, A.; Schwarz, A.; Zube, M.; Eck, U.; Hessing, B.; Schneiders, M.; Rupp, R. Applying Intuitive EEG-Controlled Grasp Neuroprostheses in Individuals with Spinal Cord Injury: Preliminary Results from the MoreGrasp Clinical Feasibility Study. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 5949–5955. [Google Scholar] [CrossRef]
  20. Frolov, A.A.; Mokienko, O.; Lyukmanov, R.; Biryukova, E.; Kotov, S.; Turbina, L.; Nadareyshvily, G.; Bushkova, Y. Post-Stroke Rehabilitation Training with a Motor-Imagery-Based Brain-Computer Interface (BCI)-Controlled Hand Exoskeleton: A Randomized Controlled Multicenter Trial. Front. Neurosci. 2017, 11, 400. [Google Scholar] [CrossRef]
  21. Mane, R.; Chouhan, T.; Guan, C. BCI for Stroke Rehabilitation: Motor and Beyond. J. Neural Eng. 2020, 17, 041001. [Google Scholar] [CrossRef]
  22. Bartur, G.; Pratt, H.; Soroker, N. Changes in Mu and Beta Amplitude of the EEG during Upper Limb Movement Correlate with Motor Impairment and Structural Damage in Subacute Stroke. Clin. Neurophysiol. 2019, 130, 1644–1651. [Google Scholar] [CrossRef]
  23. Shiner, C.T.; Tang, H.; Johnson, B.W.; McNulty, P.A. Cortical Beta Oscillations and Motor Thresholds Differ across the Spectrum of Post-Stroke Motor Impairment, a Preliminary MEG and TMS Study. Brain Res. 2015, 1629, 26–37. [Google Scholar] [CrossRef]
  24. Giovanni, A.; Capone, F.; di Biase, L.; Ferreri, F.; Florio, L.; Guerra, A.; Marano, M.; Paolucci, M.; Ranieri, F.; Salomone, G.; et al. Oscillatory Activities in Neurological Disorders of Elderly: Biomarkers to Target for Neuromodulation. Front. Aging Neurosci. 2017, 9, 189. [Google Scholar] [CrossRef]
  25. Spüler, M.; López-Larraz, E.; Ramos-Murguialday, A. On the Design of EEG-Based Movement Decoders for Completely Paralyzed Stroke Patients. J. Neuroeng. Rehabil. 2018, 15, 110. [Google Scholar] [CrossRef]
  26. Agashe, H.A.; Paek, A.Y.; Contreras-Vidal, J.L. Multisession, Noninvasive Closed-Loop Neuroprosthetic Control of Grasping by Upper Limb Amputees, 1st ed.; Elsevier B.V.: Amsterdam, The Netherlands, 2016; Volume 228. [Google Scholar]
  27. Korik, A.; Sosnik, R.; Siddique, N.; Coyle, D. Decoding Imagined 3D Hand Movement Trajectories From EEG: Evidence to Support the Use of Mu, Beta, and Low Gamma Oscillations. Front. Neurosci. 2018, 12, 130. [Google Scholar] [CrossRef]
  28. Korik, A.; Sosnik, R.; Siddique, N.; Coyle, D. Imagined 3D Hand Movement Trajectory Decoding from Sensorimotor EEG Rhythms. In Proceedings of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, 9–12 October 2016; pp. 4591–4596. [Google Scholar] [CrossRef]
  29. Kobler, R.J.; Almeida, I.; Sburlea, A.I.; Müller-Putz, G.R. Using Machine Learning to Reveal the Population Vector from EEG Signals. J. Neural Eng. 2020, 17, 026002. [Google Scholar] [CrossRef]
  30. Úbeda, A.; Hortal, E.; Iáñez, E.; Perez-Vidal, C.; Azorín, J.M. Assessing Movement Factors in Upper Limb Kinematics Decoding from EEG Signals. PLoS ONE 2015, 10, e0128456. [Google Scholar] [CrossRef]
  31. Mondini, V.; Kobler, R.J.; Sburlea, A.I.; Müller-Putz, G.R. Continuous Low-Frequency EEG Decoding of Arm Movement for Closed-Loop, Natural Control of a Robotic Arm. J. Neural Eng. 2020, 17, 046031. [Google Scholar] [CrossRef]
  32. Pei, D.; Patel, V.; Burns, M.; Chandramouli, R.; Vinjamuri, R. Neural Decoding of Synergy-Based Hand Movements Using Electroencephalography. IEEE Access 2019, 7, 18155–18163. [Google Scholar] [CrossRef]
  33. Patel, V.; Burns, M.; Pei, D.; Vinjamuri, R. Decoding Synergy-Based Hand Movements Using Electroencephalography. In Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, 18–21 July 2018; pp. 4816–4819. [Google Scholar]
  34. Schalk, G.; McFarland, D.J.; Hinterberger, T.; Birbaumer, N.; Wolpaw, J.R. BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Trans. Biomed. Eng. 2004, 51, 1034–1043. [Google Scholar] [CrossRef]
  35. Vinjamuri, R.; Mao, Z.; Sclabassi, R.J.; Diagnostics, C.; Sun, M. Time-Varying Synergies in Velocity Profiles of Finger Joints of the Hand during Reach and Grasp. In Proceedings of the The 29th Annual International Conference of the IEEE EMBS, Lyon, France, 22–26 August 2007; pp. 4846–4849. [Google Scholar]
  36. Torres-Oviedo, G.; Ting, L.H. Subject-Specific Muscle Synergies in Human Balance Control Are Consistent across Different Biomechanical Contexts. J. Neurophysiol. 2010, 103, 3084–3098. [Google Scholar] [CrossRef] [PubMed]
  37. D’Avella, A.; Bizzi, E. Shared and Specific Muscle Synergies in Natural Motor Behaviors. Proc. Natl. Acad. Sci. USA 2005, 102, 3076–3081. [Google Scholar] [CrossRef] [PubMed]
  38. Agashe, H.; Contreras-Vidal, J.L. Reconstructing Hand Kinematics During Reach to Grasp Movements from Electroencephalographic (EEG) Signals. In Proceedings of the 33rd Annual International Conference of the IEEE EMBS, Boston, MA, USA, 30 August–3 September 2011; pp. 5444–5447. [Google Scholar]
  39. Antelis, J.M.; Montesano, L.; Ramos-murguialday, A.; Birbaumer, N.; Minguez, J. On the Usage of Linear Regression Models to Reconstruct Limb Kinematics from Low Frequency EEG Signals. PLoS ONE 2013, 8, e61976. [Google Scholar] [CrossRef]
  40. Acharya, S.; Fifer, M.S.; Benz, H.L.; Crone, N.E.; Thakor, N. V Electrocorticographic Amplitude Predicts Finger Positions during Slow Grasping Motions of the Hand. J. Neural Eng. 2010, 7, 046002. [Google Scholar] [CrossRef] [PubMed]
  41. Nachev, P.; Kennard, C.; Husain, M. Functional Role of the Supplementary and Pre-Supplementary Motor Areas. Nat. Rev. Neurosci. 2008, 9, 856–869. [Google Scholar] [CrossRef]
  42. Overduin, S.A.; d’Avella, A.; Roh, J.; Bizzi, E. Modulation of Muscle Synergy Recruitment in Primate Grasping. J. Neurosci. 2008, 28, 880–892. [Google Scholar] [CrossRef]
  43. Mason, C.R.; Theverapperuma, L.S.; Hendrix, C.M.; Ebner, T.J. Monkey Hand Postural Synergies during Reach-to-Grasp in the Absence of Vision of the Hand and Object. J. Neurophysiol. 2004, 91, 2826–2837. [Google Scholar] [CrossRef]
  44. Holdefer, R.N.; Miller, L.E. Primary Motor Cortical Neurons Encode Functional Muscle Synergies. Exp. Brain Res. 2002, 146, 233–243. [Google Scholar] [CrossRef]
  45. Mollazadeh, M.; Aggarwal, V.; Thakor, N.V.; Schieber, M.H. Principal Components of Hand Kinematics and Neurophysiological Signals in Motor Cortex during Reach to Grasp Movements. J. Neurophysiol. 2014, 112, 1857–1870. [Google Scholar] [CrossRef]
  46. Townsend, B.R.; Subasi, E.; Scherberger, H. Grasp Movement Decoding from Premotor and Parietal Cortex. J. Neurosci. 2011, 31, 14386–14398. [Google Scholar] [CrossRef]
  47. Desmurget, M.; Richard, N.; Harquel, S.; Baraduc, P.; Szathmari, A.; Mottolese, C.; Sirigu, A. Neural Representations of Ethologically Relevant Hand/Mouth Synergies in the Human Precentral Gyrus. Proc. Natl. Acad. Sci. USA 2014, 111, 5718–5722. [Google Scholar] [CrossRef]
  48. Perez, M.; Rothwell, J.C. Distinct Influence of Hand Posture on Cortical Activity during Human Grasping. J. Neurosci. 2015, 35, 4882–4889. [Google Scholar] [CrossRef]
  49. Rana, M.; Yani, M.S.; Asavasopon, S.; Fisher, B.E.; Kutch, J.J. Brain Connectivity Associated with Muscle Synergies in Humans. J. Neurosci. 2015, 35, 14708–14716. [Google Scholar] [CrossRef]
  50. Tagliabue, M.; Ciancio, A.L.; Brochier, T.; Eskiizmirliler, S.; Maier, M.A. Differences between Kinematic Synergies and Muscle Synergies during Two-Digit Grasping. Front. Hum. Neurosci. 2015, 9, 165. [Google Scholar] [CrossRef]
  51. Yuan, K.; Chen, C.; Wang, X.; Chu, W.C.W.; Tong, R.K.Y. Bci Training Effects on Chronic Stroke Correlate with Functional Reorganization in Motor-Related Regions: A Concurrent Eeg and Fmri Study. Brain Sci. 2021, 11, 1–16. [Google Scholar] [CrossRef]
  52. Lopez-Larraz, E.; Ray, A.M.; Figueiredo, T.C.; Bibian, C.; Birbaumer, N.; Ramos-Murguialday, A. Stroke Lesion Location Influences the Decoding of Movement Intention from EEG. In Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Korea, 11–15 July 2017; 2017; 2017, pp. 3065–3068. [Google Scholar] [CrossRef]
  53. Bönstrup, M.; Krawinkel, L.; Schulz, R.; Cheng, B.; Feldheim, J.; Thomalla, G.; Cohen, L.G.; Gerloff, C. Low-Frequency Brain Oscillations Track Motor Recovery in Human Stroke. Ann. Neurol. 2019, 86, 853–865. [Google Scholar] [CrossRef]
  54. Antelis, J.M.; Montesano, L.; Ramos-Murguialday, A.; Birbaumer, N.; Minguez, J. Decoding Upper Limb Movement Attempt From EEG Measurements of the Contralesional Motor Cortex in Chronic Stroke Patients. IEEE Trans. Biomed. Eng. 2017, 64, 99–111. [Google Scholar] [CrossRef]
  55. Chen, C.C.; Lee, S.H.; Wang, W.J.; Lin, Y.C.; Su, M.C. EEG-Based Motor Network Biomarkers for Identifying Target Patients with Stroke for Upper Limb Rehabilitation and Its Construct Validity. PLoS ONE 2017, 12, e0178822. [Google Scholar] [CrossRef]
Sensors 22 05349 g001 550
Figure 1. Experimental setup, electrode placement and the protocol. (A) The subject was asked to perform six types of hand movements by grasping six different objects including a water bottle, a CD, a petri dish, a screwdriver, a bracelet and a door handle. During the movement, both hand kinematics and EEG activities were recorded. (B) The start of hand movement was instructed by an auditory cue, and the subjects grasped and held the object until the ‘stop’ cue. (C) Neural signals were recorded using 32 electrodes including 24 denominative electrodes (in brown) and 8 intermediate locations (in grey). Adapted from [32,33].
Figure 1. Experimental setup, electrode placement and the protocol. (A) The subject was asked to perform six types of hand movements by grasping six different objects including a water bottle, a CD, a petri dish, a screwdriver, a bracelet and a door handle. During the movement, both hand kinematics and EEG activities were recorded. (B) The start of hand movement was instructed by an auditory cue, and the subjects grasped and held the object until the ‘stop’ cue. (C) Neural signals were recorded using 32 electrodes including 24 denominative electrodes (in brown) and 8 intermediate locations (in grey). Adapted from [32,33].
Sensors 22 05349 g001
Sensors 22 05349 g002 550
Figure 2. Six kinematic synergies of each hand joint derived from the training group of 5 subjects. T: thumb, I: index, M: middle, R: ring, P: pinky, MCP: metacarpophalangeal, IP: interphalangeal, PIP: proximal interphalangeal.
Figure 2. Six kinematic synergies of each hand joint derived from the training group of 5 subjects. T: thumb, I: index, M: middle, R: ring, P: pinky, MCP: metacarpophalangeal, IP: interphalangeal, PIP: proximal interphalangeal.
Sensors 22 05349 g002
Sensors 22 05349 g003 550
Figure 3. Fraction of variance of PCs and reconstruction error using different numbers of PCs.
Figure 3. Fraction of variance of PCs and reconstruction error using different numbers of PCs.
Sensors 22 05349 g003
Sensors 22 05349 g004 550
Figure 4. Dynamic neural activity changes during grasp. The averaged EEG activity amplitude from motor areas in all movement tasks. Post stimulus (t = 0 s) activity in the motor cortex first decreases after post-stimulus and then increases to peak during the movement (the color bar indicates the normalized amplitude of EEG activity).
Figure 4. Dynamic neural activity changes during grasp. The averaged EEG activity amplitude from motor areas in all movement tasks. Post stimulus (t = 0 s) activity in the motor cortex first decreases after post-stimulus and then increases to peak during the movement (the color bar indicates the normalized amplitude of EEG activity).
Sensors 22 05349 g004
Sensors 22 05349 g005 550
Figure 5. The decoding methodology. Ten subjects were randomly divided into two equal groups for training and testing. The training set of five subjects was used to derive the kinematic synergies and model the linear correlation between the synergy weights and corresponding neural features. In the testing phase, by using the coefficients and extracted kinematic synergies from training set, the synergy weights were decoded from neural features and the hand kinematics were reconstructed for testing set. The performance of the neural decoded and reconstructed kinematics was estimated from the correlations between the recorded kinematics and reconstructed kinematics.
Figure 5. The decoding methodology. Ten subjects were randomly divided into two equal groups for training and testing. The training set of five subjects was used to derive the kinematic synergies and model the linear correlation between the synergy weights and corresponding neural features. In the testing phase, by using the coefficients and extracted kinematic synergies from training set, the synergy weights were decoded from neural features and the hand kinematics were reconstructed for testing set. The performance of the neural decoded and reconstructed kinematics was estimated from the correlations between the recorded kinematics and reconstructed kinematics.
Sensors 22 05349 g005
Sensors 22 05349 g006 550
Figure 6. Recorded and neural decoded joint angular velocities of PIP and MCP joints of thumb and four fingers. The recorded kinematics (in grey) and neural decoded kinematics (in green) are averaged across all repetitions for Subject 2, Task 4, and the shaded regions represent standard deviations.
Figure 6. Recorded and neural decoded joint angular velocities of PIP and MCP joints of thumb and four fingers. The recorded kinematics (in grey) and neural decoded kinematics (in green) are averaged across all repetitions for Subject 2, Task 4, and the shaded regions represent standard deviations.
Sensors 22 05349 g006
Sensors 22 05349 g007 550
Figure 7. Hand joint angular velocities of six types of grasps and joints from thumb, index and middle. The recorded kinematics (in grey) and neural decoded kinematics (in green) are averaged across all repetitions from Subject 2, and the shaded regions represent standard deviations. The ring and pinky finger are not shown here for simplicity in illustration.
Figure 7. Hand joint angular velocities of six types of grasps and joints from thumb, index and middle. The recorded kinematics (in grey) and neural decoded kinematics (in green) are averaged across all repetitions from Subject 2, and the shaded regions represent standard deviations. The ring and pinky finger are not shown here for simplicity in illustration.
Sensors 22 05349 g007
Sensors 22 05349 g008 550
Figure 8. Decoding error based on ten joints (A) and six grasp types (B). MCP joints of the thumb, index and middle have lower errors than respective IP/PIP joints, whereas MCP joints of the ring and pinky have relatively higher error.
Figure 8. Decoding error based on ten joints (A) and six grasp types (B). MCP joints of the thumb, index and middle have lower errors than respective IP/PIP joints, whereas MCP joints of the ring and pinky have relatively higher error.
Sensors 22 05349 g008
Sensors 22 05349 g009 550
Figure 9. Spatial correlations of each synergy reflected on the scalp tomograph for Subject 2. The hand synergies are highly modulated on the central and left hemisphere, covering the areas of movement intention and motion-related activation generation. Synergies 1 to 5 were effective spatially with respect to hand area, and span across the premotor cortex, primary motor cortex and primary sensorimotor cortex. Synergies 1, 4 and 6 were mainly activated on the primary motor area and supplementary motor area. The primary motor cortex and the primary somatosensory cortex were activated on the contralateral hemisphere, as observed from Synergies 1, 3 and 4.
Figure 9. Spatial correlations of each synergy reflected on the scalp tomograph for Subject 2. The hand synergies are highly modulated on the central and left hemisphere, covering the areas of movement intention and motion-related activation generation. Synergies 1 to 5 were effective spatially with respect to hand area, and span across the premotor cortex, primary motor cortex and primary sensorimotor cortex. Synergies 1, 4 and 6 were mainly activated on the primary motor area and supplementary motor area. The primary motor cortex and the primary somatosensory cortex were activated on the contralateral hemisphere, as observed from Synergies 1, 3 and 4.
Sensors 22 05349 g009
Sensors 22 05349 g010 550
Figure 10. Synergy modulation in delta, theta, mu, beta and gamma frequency bands of EEG. Synergies activation mainly concentrated on the primary motor area modulated in low-frequency rhythm (delta). In theta rhythm, strong activation was observed on the contralateral premotor cortex, and ipsilateral somatosensory cortex. Mu modulations were observed in the supplementary motor cortex, and beta were observed in the bilateral primary motor region. Activations observed from gamma spanned over all motor-related areas.
Figure 10. Synergy modulation in delta, theta, mu, beta and gamma frequency bands of EEG. Synergies activation mainly concentrated on the primary motor area modulated in low-frequency rhythm (delta). In theta rhythm, strong activation was observed on the contralateral premotor cortex, and ipsilateral somatosensory cortex. Mu modulations were observed in the supplementary motor cortex, and beta were observed in the bilateral primary motor region. Activations observed from gamma spanned over all motor-related areas.
Sensors 22 05349 g010
Table
Table 1. Nomenclature table (symbols and definitions).
Table 1. Nomenclature table (symbols and definitions).
v Joint angular velocities
V Matrix of joint angular velocities
c Synergy weights
C Matrix of synergy weights
S Kinematic synergies
S Matrix of kinematic synergies
X Matrix of neural features
β Regression coefficients
D Neural independency density of the synergy
Table
Table 2. Correlations between the neural decoded kinematics and recorded kinematics (Mean ± Std).
Table 2. Correlations between the neural decoded kinematics and recorded kinematics (Mean ± Std).
Subject 1Subject 2Subject 3Subject 4Subject 5
Task 172.0 ± 8.0%65.1 ± 11.2%56.7 ± 20.5%69.2 ± 8.2%79.4 ± 7.1%
Task 284.4 ± 9.3%74.1 ± 10.0%77.5 ± 6.2%77.7 ± 9.3%74.1 ± 12.8%
Task 389.9 ± 6.3%83.7 ± 14.8%80.9 ± 13.2%84.5 ± 10.4%78.7 ± 9.3%
Task 475.5 ± 6.4%59.3 ± 12.9%75.1 ± 7.5%72.2 ± 9.8%64.6 ± 15.4%
Task 584.3 ± 7.0%82.2 ± 7.2%82.1 ± 8.7%70.5 ± 10.3%79.8 ± 7.1%
Task 677.4 ± 6.6%81.0 ± 6.4%80.1 ± 7.7%73.7 ± 7.4%80.8 ± 8.1%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Pei, D.; Olikkal, P.; Adali, T.; Vinjamuri, R. Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals. Sensors 2022, 22, 5349. https://doi.org/10.3390/s22145349

AMA Style

Pei D, Olikkal P, Adali T, Vinjamuri R. Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals. Sensors. 2022; 22(14):5349. https://doi.org/10.3390/s22145349

Chicago/Turabian Style

Pei, Dingyi, Parthan Olikkal, Tülay Adali, and Ramana Vinjamuri. 2022. "Reconstructing Synergy-Based Hand Grasp Kinematics from Electroencephalographic Signals" Sensors 22, no. 14: 5349. https://doi.org/10.3390/s22145349

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop