Researchers have developed a brain-computer interface that can translate the neural signals of paralyzed individuals into words and movements performed by a virtual person, aiming to restore a dimension of human communication that goes beyond simply producing sound.
The new experimental system differs from previous technologies in its ability to simultaneously decode speech and associated gestures. Starting from a single implant that records brain activity, machine learning algorithms then use these signals to animate a digital face and body representing the user.
The initial study involved a team from the University of California, San Francisco. In one participant, the system was able to recognize ten communicative gestures, while in the other, it recognized four. These gestures included nodding in agreement, shaking the head in disagreement, waving, shrugging, and giving a thumbs-up, in addition to the words the participants attempted to speak.
The importance of gestures lies in their ability to add meaning, emotion, and context to speech, and sometimes even replace it. Therefore, the researchers hope to move from interfaces that restore a single function to a unified system that gives the user multiple means of expression.
This technology could one day help people who have lost the ability to speak due to strokes, ALS, or other neurological diseases, with the potential for long-term use in cases of severe difficulty jumping. However, the results do not mean the technology is ready for everyday use.
The implant requires brain surgery, and the system is customized for each participant, as models trained on one person''s brain signals cannot easily transfer to another. Furthermore, the vocabulary and number of movements the system can interpret are still limited.
The team is currently working to expand the vocabulary and gestures, aiming for a more consistent and natural representation of body movement. If these efforts are successful, the "digital face" could transform from a mere technological tool into a means of restoring a patient''s presence and gestures, not just their voice.