Stroke and neurodegenerative diseases can impair speech and nonverbal gestures, limiting natural communication. Brain–computer interfaces (BCIs) aim to restore these functions by translating neural activity into commands for external devices, although prior work has primarily focused on decoding speech or gestures in isolation. Here we show that neural signals recorded with a single high-density electrocorticography implant can support simultaneous decoding of speech and gestures in people with paralysis.
We first show that isolated upper-limb and orofacial movements can be reliably decoded among three participants. Using parallel speech and gesture decoders, we then enabled participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. Training models on both isolated and simultaneous data improved performance across behavioral contexts.
These findings demonstrate that one cortical implant can support multi-effector control and provide a step toward BCIs that enable more natural communication for people with paralysis. Natural communication is often concurrent and multi-effector, involving distinct motor systems such as the vocal tract and upper limbs. For example, spoken language is frequently accompanied by gestures, and these cospeech gestures contribute to how listeners understand and integrate communicated information1,2,3.
Stroke and neurodegenerative diseases, such as amyotrophic lateral sclerosis (ALS), can impair both speech and body movements4,5, limiting agency, participation in social activities and substantially reducing quality of life6,7. Brain–computer interfaces (BCIs) aim to restore lost motor functions by translating neural activity from the sensorimotor cortex (SMC) into commands for external devices. Recent BCI research has demonstrated the decoding of speech8,9,10,11,12,13,14 and other nonspeech movements, such as facial expressions8, cursor control15, manual gestures16,17,18 and walking19.
However, most BCI research to date has focused on decoding these behaviors in isolation. It therefore remains unclear whether a single BCI implant can reliably decode speech and gestures when they are attempted together, as they often are during natural communication. As BCI systems become more capable, developing multi-effector, multifunctional systems that support both speech and body movements—and that remain robust across isolated and concurrent contexts—is important for future real-world clinical utility and for supporting natural communication.
The organization of the SMC has been shown to support multi-effector decoding of isolated movements at multiple spatial scales. At the population level, movements of the legs, arms, hands and face follow a rough somatotopy20,21,22, enabling movements of these whole-body effectors to be decoded from electrocorticography (ECoG) arrays spanning the precentral and postcentral gyri8,23,24,25. While this general somatotopy exists, it does not necessarily equate with strict neural population boundaries.
Extract — continue reading at the source.
