Visual influences on auditory processing of speech
Multiple Paper Presentation

Mikko Sams
Laboratory of Computational Engineering, Helsinki University of Technology

     Abstract ID Number: 22

Abstract
Seeing articulation improves speech perception in noise and may also modify auditory percepts, as occurs in the well-known McGurk effect. During the last 15 years, we have started to understand the brain mechanisms underlying audiovisual speech perception. Interactions of auditory and visual speech at posterior superior temporal areas have been found in numerous studies. Evidence has accumulated that visual speech modifies activity in the auditory cortex, even in the primary auditory cortex. Visual influences on cortical auditory processing can be quite early, occurring already about 100 ms from the onset of the auditory stimulus. Moreover, our recent results suggest that seeing speech may influence processing in the auditory brainstem at about 10 ms latency form the onset of the auditory stimulus. When considering such early effects, it is important to remember that in normal speech the onset of articulatory movements start often even 100 ms before the onset of acoustic speech stimulus. The nature of visual influences is far from clear. They might reflect both indirect attention-mediated effects (seeing speech generates expectations), as well as actual processing of visual speech in the auditory system. A summary and synthesis of recent research on effects of visual speech on auditory processing is provided.

To be Presented at the Following Symposium:
Speech as a window to multisensory integration processes
Other papers in this Symposium:

• Salvador Soto-Faraco
  Dept. Psicologia Bàsica - Facultat de Psicologia and Parc Científic de Barcelona, Universitat de Barcelona
  Agnès Alsius
  Dept. Psicologia Bàsica - Facultat de Psicologia and Parc Científic de Barcelona, Universitat de Barcelona
  Jordi Navarra
  Dept. Psicologia Bàsica - Facultat de Psicologia and Parc Científic de Barcelona, Universitat de Barcelona
  Ruth Campbell
  Human Communication Sciences, Univesity College London
  Attention modulates multisensory integration: The case of audiovisual speech
• Virginie van Wassenhove
  Department of Radiology, University of California, San Francisco
  ‘Analysis-by-synthesis’ in auditory-visual speech perception
• Ruth Campbell
  Department of Human Communication Sciences, University College London
  Seeing silent speech - cortical correlates in relation to visual processing requirements