قالب وردپرس درنا توس
Home / Science / Engineers translate brain signals directly into speech

Engineers translate brain signals directly into speech



  brain
Credit: CC0 Public Domain

In a scientific premiere, Columbia neuro-engineers have created a system that translates thoughts into understandable, recognizable language. By monitoring a person's brain activity, the technology can reconstruct the words a person hears with unprecedented clarity. This breakthrough, harnessing the power of speech synthesizers and artificial intelligence, could open new avenues for the computer's direct communication with the brain. It also lays the foundation for helping people who can not speak, such as people living with amyotrophic lateral sclerosis (ALS) or recovering from a stroke, to be able to communicate with the outside world again.

These findings were published today in Scientific Reports .

"Our voices help us to connect our friends, family and the world around us, which is why the voice is lost through injury or disease is so devastating," Dr. Nima Mesgarani, lead author of the paper and principal investigator at Mortimer B. Zuckerman's Mind Brain Behavior Institute, Columbia University. "With today's study, we have a potential way to restore that power, and we've shown that with the right technology, these people's minds could be decoded and understood for every listener."

Decades of research has shown that people do this ̵

1; or even imagine, talking – telltale activity patterns appear in their brains. A clear (but recognizable) pattern of signals also occurs when we listen to someone or imagine listening. Experts who attempt to record and decode these patterns see a future in which thoughts need not be hidden in the brain, but can be translated at will into verbal language.

But achieving this feat has been difficult. Early Efforts, Brain Signals by Dr. Ing. To decode Mesgarani and others focused on simple computer models in which spectrograms were analyzed, which are visual representations of sound frequencies.

However, since this approach did not lead to a comparable language, Dr. Instead, Mesgarani's team turned to a vocoder, a computer algorithm that can synthesize speech after being trained on talking people's records.

"This technology was used by Amazon Echo and Apple Siri to provide verbal answers to our questions," Dr. Mesgarani, who is also a professor of electrical engineering at the Fu Foundation School of Engineering and Applied Science in Colombia.




A presentation of early approaches to language reconstruction using linear models and spectrograms. Credit: Nima Mesgarani / Zuckerman Institute of Columbia

In order to teach the vocoder how to interpret brain activity, Dr. Ing. Mesgarani with Dr. med. Ashesh Dinesh Mehta, a neurosurgeon at Northwell Health Physician, Partners Neuroscience Institute author of today's newspaper. Mehta treats epileptic patients, some of whom need regular surgery.

"In collaboration with Dr. Mehta, we asked epilepsy patients who are already undergoing brain surgery to listen to the sentences spoken by different people." Mesgarani. "These neural patterns trained the vocoder."

Next, the researchers asked the same patients to listen to the speakers, recite the numbers between 0 and 9, while recording brain signals that could then be passed through the vocoder. The vocoder's sound in response to these signals was analyzed and cleared by neural networks, a kind of artificial intelligence that mimics the structure of neurons in the biological brain.




Presentation of the new approach of Dr. med. Mesgarani, who uses a vocoder and deep neural network to reconstruct speech. Photo credits: Nima Mesgarani / Columbia's Zuckerman Institute

The end result was a robot-sounding voice reciting a sequence of numbers. To test the accuracy of the recording, Dr. med. Mesgarani and his team are the persons to listen to the recordings and report what they have heard.

"We found that people could understand and repeat the sounds 75% of the time, which is actually the case, far beyond any previous attempts," Dr. Mesgarani. The improvement in intelligibility was particularly evident when comparing the new recordings with the earlier spectrogram-based tests. "The sensitive vocoder and powerful neural networks represented the sounds that patients originally heard with surprising accuracy."

Dr. Mesgarani and his team plan to test more complicated words and phrases next, and they want to do the same tests on brain signals that are sent out when a person is talking or making a speech. Ultimately, they hope that their system could be part of an implant, similar to some epilepsy patients, which translates the wearer's thoughts directly into words.

"If the wearer thinks in this scenario, I need a glass of water." "Our system could pick up the brain signals generated by this thought and turn it into synthesized verbal language," Dr. Mesgarani. "This would be a game changer, which would give anyone who has lost his or her ability to speak, injury or disease, a renewed opportunity to connect with the world around them."

This paper is titled "Towards a Reconstructable Language Understanding Language." from the human ear canal. "


Cognitive hearing aid filters out the noise



Provided by
Columbia University

Quote :
Engineers translate brain signals directly into speech (2019, January 29)
retrieved on January 29, 2019
from https://techxplore.com/news/2019-01-brain-speech.html

This document is subject to copyright. Apart from a fair trade for the purpose of private study or research, no
Part may be reproduced without written permission. The content is provided for informational purposes only.


Source link