AI system reads minds and produces text without implants

Researchers at the University of Texas at Austin have developed an artificial intelligence (AI) system that can decode a person’s thoughts and transform them into natural language text. The new brain-computer interface, called a semantic decoder, uses non-invasive recordings of brain activity obtained through functional magnetic resonance imaging (fMRI) to translate imagined or perceived stimuli into continuous streams of text.

The breakthrough technology, which could offer new hope to people who are conscious but unable to speak due to conditions such as stroke or paralysis, differs from other ‘mind-reading’ systems in that it does not need to be implanted into the brain.

The fMRI technology, however, is not perfect, as it measures blood oxygen levels, which are relatively slow to respond to neural activity. As each brain image can be affected by more than 20 words, the researchers developed an encoding model similar to those used by Open AI’s ChatGPT and Google’s Bard to predict how the brain will respond to natural language.

To train the semantic decoder, researchers recorded the brain responses of three people while they listened to 16 hours of spoken stories. The decoder could predict, with considerable accuracy, how the person’s brain would respond to hearing a sequence of words. Although the results do not recreate the stimulus word for word, the decoder can pick up the gist of what is being said.

The technology has shown considerable promise, particularly when it comes to capturing the gist of a story when participants are actively listening to it and ignoring another story being played simultaneously. In addition to stories, the researchers tested the AI on people watching four short, silent videos while their brains were scanned using fMRI. The semantic decoder translated their brain activity into accurate descriptions of certain events from the videos they watched.

The UT Austin team noted that willing participation is key to the process. Those who put up resistance during the encoder training or deliberately thought other thoughts produced unusable results. Similarly, the AI produced unintelligible results when tested on people who hadn’t trained the decoder.

One of the main advantages of the semantic decoder is that it is non-invasive, making it an attractive option for people who might be reluctant to undergo surgery to have an implant inserted into their brains. However, the technology’s current reliance on fMRI means that it is not yet usable outside of the lab environment. The hope is that the technology can be adapted for use with more portable brain imaging systems such as functional near-infrared spectroscopy (fNIRS).

The researchers are also acutely aware of the potential for malicious misuse of inaccurate results and the importance of protecting people’s mental privacy. To this end, the team is working to develop safeguards that will prevent the technology from being used without a person’s consent.

“We take very seriously the concerns that it could be used for bad purposes and have worked to avoid that,” said Jerry Tang, lead author of the study. “We want to make sure people only use these types of technologies when they want to and that it helps them.”

The technology also has potential applications beyond those related to helping those with communication difficulties. For example, it could be used to better understand how people process language, and to develop more sophisticated voice recognition systems. Additionally, the ability to decode a person’s thoughts could also have implications for the criminal justice system.

The study, which was published in the journal Nature Neuroscience, has generated considerable excitement among researchers and those affected by conditions such as stroke or paralysis. While the technology is not yet ready for use outside of the lab environment, it represents a significant step forward in the field of brain-computer interfaces.

“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” said Alexander Huth, corresponding author of the study.