This Brain Implant Allows People With Paralysis to Speak and Gesture at the Same Time Via a Mind-Controlled Virtual Avatar

People with severe paralysis can struggle to communicate with others. While experimental brain-computer interfaces (BCIs) have come a long way in being able to help them convey their thoughts, these technologies have generally been limited to either speech or movement. Now, scientists have developed a tool that can do both simultaneously.

The implant, described in the journal Nature Neuroscience on September 14, allows users to control the speech and movement of a virtual avatar with their mind in real time. While the work is still in its early stages, the innovation could one day help people with paralysis communicate with more nuance or serve as a stepping stone to robotic prosthetics that can be controlled with the mind. Each person had an iPhone-size device with 253 electrodes surgically implanted onto their sensorimotor cortex, a brain region involved in processing incoming sensations and outgoing movements.

The implant can safely cover such a wide area because it lies on the brain’s surface rather than penetrating it like other BCIs, study co-author Edward Chang, a neurosurgeon at the University of California, San Francisco, tells Jennie Erin Smith at Science. The team trained computer models to decode the participants’ thoughts as they attempted to say up to ten phrases like “hello” and “nice to meet you” or attempted to perform up to ten gestures such as clapping and waving. They also practiced up to 100 realistic speech-and-gesture combinations.

Then, the researchers mapped the brain activity linked to the phrases and gestures to a personalized full-body digital avatar, which could make movements and talk with onscreen text. During five rounds of a conversation test with the participant with ALS, the decoders got the individual’s intended gesture correct 85 percent of the time, on average, and their intended phrase right 75 percent of the time, on average. The participant who had a stroke had 100 percent accuracy with the speech and gesture decoders during three rounds of a conversation test.

The performance disparity might have something to do with the participants’ different causes of paralysis, according to the researchers. In March, researchers reported an experimental BCI that helped two people with paralysis type on a virtual keyboard with their minds. One participant could do it as fast as an able-bodied smartphone user can text.

While “the results are of very high quality, the level of speech decoding [in this study] is below what has been reported before,” Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands who wasn’t involved in the work, tells Miryam Naddaf at Nature. What’s more, he notes, the BCI decoded only isolated gestures and sentences, whereas real conversations involve continuous streams of speech and movement. Brosler acknowledges the implant’s current limitations, explaining to the outlet that this study was meant to be a proof of concept.