Visual influences on auditory processing of speech

Mikko Sams, Laboratory of Computational Engineering, Helsinki University of Technology

Abstract
Seeing articulation improves speech perception in noise and may also modify auditory percepts, as occurs in the well-known McGurk effect. During the last 15 years, we have started to understand the brain mechanisms underlying audiovisual speech perception. Interactions of auditory and visual speech at posterior superior temporal areas have been found in numerous studies. Evidence has accumulated that visual speech modifies activity in the auditory cortex, even in the primary auditory cortex. Visual influences on cortical auditory processing can be quite early, occurring already about 100 ms from the onset of the auditory stimulus. Moreover, our recent results suggest that seeing speech may influence processing in the auditory brainstem at about 10 ms latency form the onset of the auditory stimulus. When considering such early effects, it is important to remember that in normal speech the onset of articulatory movements start often even 100 ms before the onset of acoustic speech stimulus. The nature of visual influences is far from clear. They might reflect both indirect attention-mediated effects (seeing speech generates expectations), as well as actual processing of visual speech in the auditory system. A summary and synthesis of recent research on effects of visual speech on auditory processing is provided.

Not available

Back to Abstract