In a scientific premiere, Columbia neuro-engineers have created a system that translates thoughts into understandable, recognizable language. By monitoring a person's brain activity, the technology can reconstruct the words a person hears with unprecedented clarity. This breakthrough, harnessing the power of speech synthesizers and artificial intelligence, could open new avenues for the computer's direct communication with the brain. It also lays the foundation for helping people who can not speak, such as people living with amyotrophic lateral sclerosis (ALS) or recovering from a stroke, to be able to communicate with the outside world again.
These findings were published today in Scientific Reports .
"Our voices help us to connect our friends, family and the world around us, which is why the voice is lost through injury or disease is so devastating," Dr. Nima Mesgarani, lead author of the paper and principal investigator at Mortimer B. Zuckerman's Mind Brain Behavior Institute, Columbia University. "With today's study, we have a potential way to restore that power, and we've shown that with the right technology, these people's minds could be decoded and understood for every listener."
Decades of research has shown that people do this ̵
But achieving this feat has been difficult. Early Efforts, Brain Signals by Dr. Ing. To decode Mesgarani and others focused on simple computer models in which spectrograms were analyzed, which are visual representations of sound frequencies.
However, since this approach did not lead to a comparable language, Dr. Instead, Mesgarani's team turned to a vocoder, a computer algorithm that can synthesize speech after being trained on talking people's records.
"This technology was used by Amazon Echo and Apple Siri to provide verbal answers to our questions," Dr. Mesgarani, who is also a professor of electrical engineering at the Fu Foundation School of Engineering and Applied Science in Colombia.
In order to teach the vocoder how to interpret brain activity, Dr. Ing. Mesgarani with Dr. med. Ashesh Dinesh Mehta, a neurosurgeon at Northwell Health Physician, Partners Neuroscience Institute author of today's newspaper. Mehta treats epileptic patients, some of whom need regular surgery.
"In collaboration with Dr. Mehta, we asked epilepsy patients who are already undergoing brain surgery to listen to the sentences spoken by different people." Mesgarani. "These neural patterns trained the vocoder."
Next, the researchers asked the same patients to listen to the speakers, recite the numbers between 0 and 9, while recording brain signals that could then be passed through the vocoder. The vocoder's sound in response to these signals was analyzed and cleared by neural networks, a kind of artificial intelligence that mimics the structure of neurons in the biological brain.
The end result was a robot-sounding voice reciting a sequence of numbers. To test the accuracy of the recording, Dr. med. Mesgarani and his team are the persons to listen to the recordings and report what they have heard.
"We found that people could understand and repeat the sounds 75% of the time, which is actually the case, far beyond any previous attempts," Dr. Mesgarani. The improvement in intelligibility was particularly evident when comparing the new recordings with the earlier spectrogram-based tests. "The sensitive vocoder and powerful neural networks represented the sounds that patients originally heard with surprising accuracy."
Dr. Mesgarani and his team plan to test more complicated words and phrases next, and they want to do the same tests on brain signals that are sent out when a person is talking or making a speech. Ultimately, they hope that their system could be part of an implant, similar to some epilepsy patients, which translates the wearer's thoughts directly into words.
"If the wearer thinks in this scenario, I need a glass of water." "Our system could pick up the brain signals generated by this thought and turn it into synthesized verbal language," Dr. Mesgarani. "This would be a game changer, which would give anyone who has lost his or her ability to speak, injury or disease, a renewed opportunity to connect with the world around them."
This paper is titled "Towards a Reconstructable Language Understanding Language." from the human ear canal. "
Cognitive hearing aid filters out the noise
Engineers translate brain signals directly into speech (2019, January 29)
retrieved on January 29, 2019
This document is subject to copyright. Apart from a fair trade for the purpose of private study or research, no
Part may be reproduced without written permission. The content is provided for informational purposes only.