Brain-to-speech decoding will require linguistic and pragmatic data
JournalJournal of Neural Engineering
PublisherInstitute of Physics Publishing
MetadataShow full item record
AbstractObjective. Advances in electrophysiological methods such as electrocorticography (ECoG) have enabled researchers to decode phonemes, syllables, and words from brain activity. The ultimate aspiration underlying these efforts is the development of a brain-machine interface (BMI) that will enable speakers to produce real-time, naturalistic speech. In the effort to create such a device, researchers have typically followed a bottom-up approach whereby low-level units of language (e.g. phonemes, syllables, or letters) are decoded from articulation areas (e.g. premotor cortex) with the aim of assembling these low-level units into words and sentences. Approach. In this paper, we recommend that researchers supplement the existing bottom-up approach with a novel top-down approach. According to the top-down proposal, initial decoding of top-down information may facilitate the subsequent decoding of downstream representations by constraining the hypothesis space from which low-level units are selected. Main results. We identify types and sources of top-down information that may crucially inform BMI decoding ecosystems: communicative intentions (e.g. speech acts), situational pragmatics (e.g. recurrent communicative pressures), and formal linguistic data (e.g. syntactic rules and constructions, lexical collocations, speakers' individual speech histories). Significance. Given the inherently interactive nature of communication, we further propose that BMIs be entrained on neural responses associated with interactive dialogue tasks, as opposed to the typical practice of entraining BMIs with non-interactive presentations of language stimuli. © 2018 IOP Publishing Ltd.
Identifier to cite or link to this itemhttps://www.scopus.com/inward/record.uri?eid=2-s2.0-85056661585&doi=10.1088%2f1741-2552%2faae466&partnerID=40&md5=cddf5c0127230edb506b6dd057eb66dc; http://hdl.handle.net/10713/8775
- Decoding speech using the timing of neural signal modulation.
- Authors: Jiang W, Pailla T, Dichter B, Chang EF, Gilja V
- Issue date: 2016 Aug
- Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids.
- Authors: Ramsey NF, Salari E, Aarnoutse EJ, Vansteensel MJ, Bleichner MG, Freudenburg ZV
- Issue date: 2018 Oct 15
- Phonemes: Lexical access and beyond.
- Authors: Kazanina N, Bowers JS, Idsardi W
- Issue date: 2018 Apr
- Brain2Char: a deep architecture for decoding text from brain recordings.
- Authors: Sun P, Anumanchipalli GK, Chang EF
- Issue date: 2020 Dec 16
- Real-time decoding of question-and-answer speech dialogue using human cortical activity.
- Authors: Moses DA, Leonard MK, Makin JG, Chang EF
- Issue date: 2019 Jul 30