I n t e r n a t i o n a l   M u l t i s e n s o r y    R e s e a r c h    F o r u m
2ND ANNUAL  MULTISENSORY RESEARCH CONFERENCE   

 
Conference Abstracts
 
Abstract Numbers: 1-9   10-19   20-29   30-39   40-49   50-59
Keynote Abstracts: Massaro   Stein

conference program
imrf conference main page

 

 

1 Sensory substitution and mental imagery in early blind humans: Evidences for crossmodal cerebral reorganisation  
Wanet-Defalque, M.C., De Volder, A.G., Vanlierde, A., Arno, P., Sanabria-Boh¤rquez, S.M., Veraart, C.

Since 1986, our group has studied the consequences of early blindness on the functional development of the human brain. Using Positron Emission Tomography (PET) to investigate the neural metabolism, high rates of glucose consumption were observed in the occipital areas of early blind (EB) subjects: they reached the levels observed in the visual areas of sighted controls (SC) studied in normal visual conditions (Wanet-Defalque et al, Brain Research, 1988, 446: 369-373). This elevated metabolism was hypothesised to reflect the synaptic activity and attributed to a lack of synaptic revision in the early deafferented areas during brain development (Veraart et al, Brain Research, 1990, 510 : 115-121). In addition, the cerebral blood flow as well as the oxygen and glucose utilisation at rest were all higher in EB visual cortex as compared to controls‚ a finding that enabled us to exclude an influence of anaerobic glial metabolism (De Volder et al, Brain Research, 1997, 750 : 235-244).

To assess the hypothesis of crossmodal reorganisation in EB occipital cortex, two functional imaging studies were further conducted using regional cerebral blood flow (rCBF) measurements in activation condition. In the first study, devoted to evaluate the effect of sensory substitution, 12 participants (6 EB and 6 SC) were trained to use a Prosthesis for the Substitution of Vision by Audition (PSVA). The PET results showed that, in addition to areas activated in SC, the EB’s occipital cortex was also activated, and this activation was higher when the EB subjects were using PSVA for a pattern recognition task than during auditory control tasks. In a second study (De Volder et al , J Cereb Blood flow metabol., 1999, 19, Suppl. 1, S412), devoted to explore a mental imagery task (in 6 EB and 6 SC), PET results showed that mental imagery of object shape, acquired through tactile experience in EB humans, does recruit the same visual association areas as in SC.

Taken together, these results support the idea of crossmodal brain reorganisations in EB humans. Additional studies should further investigate the exact functional role of the EB occipital cortex during perceptual and mental activities.


2 Neural and cognitive implications of a synaesthete's performance on selective attention tasks
John H. Flowers and Eric C. Odgaard
Recent investigations of selective attention performance by symbol-color synesthetes (individuals who regularly experience involuntary elicitation of color imagery in response to viewing particular letters, words, or numeric digits) indicate that synesthetic percepts share many of the properties of acquired automatic cognitive processes. These include occurrence without intention, and performance decrements in Stroop-like filtering tasks stemming from inability to suppress the synesthetic response (e.g., Wollen & Ruggerio, 1983; Mills, Boteler & Oliver, 1999). Recent research form our laboratory (Odgaard, Flowers & Bradman, 1999) has demonstrated two additional consequences of synesthetic color photisms for selective attention performance that parallel those of acquired automatic responses found in variants of the Stroop task for non-synesthetic subjects. The first of these is negative priming, where a synesthetically elicited response that serves as the distracter on trial N leads to enhanced slowing of color naming when it is the to-be-named ink color on trial N+1.ðThe second of these is the synesthetic interference with manual visual search that is restricted to conditions for which the search set requires a significant working memory load.ðCollectively, these findings suggest a significant lexical component to the synesthetic experience. Additionally, they suggest that the structure of these types of synesthetic associations is acquired, perhaps during the early learning of symbolic and numeric concepts, even though genetic or other influences on early pre-cortical development may make an individual "synesthetically prepared." Converging use of a variety of selective attention and perceptual performance measures in conjunction with imaging techniques may provide a useful future research strategy for increasing our understanding of this unusual perceptual phenomenon.

3 Bias among human intersensory links revealed by individual differences in synesthetic perception
Peter G. Grossenbacher
Synesthetic perception comprises a rare condition in which sensory stimuli evoke additional sensations not regularly experienced by most people. For the relatively few people who regularly experience synesthesia, it takes any of several forms, such as colored sounds or haptically-shaped tastes. In colored-sound synesthesia, auditory stimulation induces a visual percept of color that accompanies an otherwise normal hearing experience. Synesthetic perception is marked by neurophysiological differences from normal perception, consistent with the reportedly rapid onset of synesthetically induced sensations (e.g., Schiltz, 1999). Although synesthetes exhibit abnormal patterns of brain activity, there is no evidence suggesting that the neuroanatomical connections between sensory modalities differ between synesthetes and non-synesthetes. To the extent that neuroanatomical connectivity is indeed normal in synesthetes, the intersensory links evident in synesthesia may inform our understanding of normal human intersensory connectivity. Most recent studies of synesthesia have focussed exclusively on a small number of forms in which color is the synesthetically induced sensation, thereby contributing only limited data relevant to an evaluation of intersensory patterns across the variety of forms that constitute synesthesia.
 
This talk presents data collected with a new instrument designed to screen for many distinct forms of synesthesia in a manner more comprehensive than previous approaches. Verbal report data from a large group of synesthetes provides quantification of endorsed forms of synesthetic perception with regard to frequency of occurrence and vividness of synesthetic phenomena. Within and across individuals, these data reveal asymmetries among both the sensory modalities of stimulation which induce synesthesia and the sense modalities in which synesthetic phenomena are induced. These findings are discussed with respect to normal human intersensory connectivity and a theory of synesthesia that posits abnormal disinhibition of normally existing feed-backward connections in cortical pathways (Grossenbacher, 1997).

4 Increased gene expression in the monkey intraparietal cortex underlying alteration of somatosensory-visual integration accompanying tool-use learning
Hidetoshi Ishibashi, S. Hihara, M. Takahashi and A. Iriki

When skilled to use a tool, we feel it becoming a part of or an extension of the hand. After repeated practice to use the tool, this psychological alteration of body-image becomes perceptible, perhaps resulted from reintegration of somatosensory and visual signals. -- An electrophysiological study in tool-using monkeys showed that visual receptive field of bimodal neurons in the intraparietal cortex, coding the image of the hand, extended to include the tool (ref. 1). The present study was undertaken to investigate if molecular genetic processes in this cortical region are involved in induction of above physiological phenomenon. Monkeys were sacrificed either during or after the training to acquire the skill to use the tool (ref. 2), and the brains were removed for immunohistochemical and quantitative RT-PCR analyses to investigate expression pattern of immediate-early-genes. During the development of the skill, the expression of zif-268 and c-fos, but not of jun-D, were greater in the postcentral gyrus (especially in the forearm region) of the hemisphere contralateral to the hand trained to use the tool, compared with the ipsilateral hemisphere. The increase, however, was less evident after the training was completed.

References

1 A. Iriki, M. Tanaka and Y. Iwamura, Neuroreport 7, 2325-2330. 1996.
2 H. Ishibashi, S. Hihara and A. Iriki, Can. J. Physiol. Pharmacol. in press.


5 Cross-modal Plasticity "Visualizes" Auditory Cortex
Sarah Pallas
Mammalian cortex is subdivided into numerous structurally and functionally unique areas. The relative roles of intrinsic and extrinsic factors in this parcellation process remain unclear. Our work in ferrets supports a strong role for extrinsic, activity-based information in the specification of several aspects of cortical areal identity. Neonatal diversion of retinal axons into auditory thalamus provides visual information to auditory cortex but via the same thalamocortical pathway that normally provides sensory input. As a result, auditory cortex responds to visual stimuli in much the same way as visual cortex, and can mediate visual perception. We have demonstrated that early visual input results in changes in connectivity and neuronal morphology in auditory cortex, which may underlie the visual capability of auditory cortex in cross-modal animals.

6 Cross-modal spatial interactions and their adaptive plasticity 
Andrew J. King, Oliver Kacelnik, Mark E. Walton and Carl H. Parsons

Combining information across different senses can improve our ability to detect, localize, and identify stimuli and results in faster reactions after stimulus presentation. Similar principles apply to neurons in the deeper layers of the superior colliculus (SC), which can exhibit enhanced responses when different modality signals are presented in close spatial and temporal proximity. These multisensory interactions highlight the importance of aligning the sensory maps in this midbrain nucleus, which is achieved during development by using experience to refine the auditory spatial tuning of SC neurons.

We have examined the effects of visual-auditory interactions on the ability of adult ferrets to perform a spatial identification task, in which they were trained to approach the location of a stimulus in the horizontal plane. The accuracy of reflexive head-orienting responses was also measured. Bimodal stimuli led to more accurate behavioural responses than those elicited by either LEDs or noise bursts presented separately, although these effects were less pronounced than those previously reported in cats and much weaker in animals that were highly trained in localizing unimodal stimuli.

Our electrophysiological studies have shown that visual cues play a guiding role in establishing intersensory map registration in the SC. We have now investigated the role of vision in the development of auditory localization behaviour and in the capacity of the auditory system to adapt to abnormal binaural cues produced by plugging one ear. Adult ferrets that were visually deprived from infancy could judge sound source location as accurately as normally sighted animals and exhibited improved auditory spatial acuity at lateral sound locations. In keeping with the SC data, however, these animals made more front-back errors than the controls. Like normally sighted ferrets, visually-deprived animals showed a recovery of performance soon after plugging one ear in adulthood, suggesting that this form of plasticity does not require visual feedback.

Our results provide further evidence that stimulus localization can be enhanced in the presence of matching multisensory cues. The coordination of spatial information across different modalities relies primarily on visual cues, but adaptive changes in auditory localization can take place throughout life in the absence of vision.


7 Compensatory Plasticity and Sensory Substitution in the Cerebral Cortex
Josef Rauschecker
Early visual deprivation in cats and ferrets leads to crossmodal compensation both on the behavioral and neurobiological level (Rauschecker, TINS, 1995). Visually deprived animals can localize sounds in space with greater precision. Correspondingly, neurons in areas of parietal cortex normally activated by visual stimuli are now activated by sound. Thus, parietal areas in blind animals subserving auditory function are expanded significantly; at the same time, neurons in these areas are also more sharply tuned to auditory spatial location.
 In blind humans, strikingly similar results are seen with functional neuroimaging techniques. Using positron emission tomography (PET) it can be shown that activation zones in the inferior parietal lobules associated with auditory spatial processing are vastly expanded into occipital areas. Cross-correlation analysis between areas demonstrates that occipital areas receive auditory input from the superior gyrus via the parietal lobes. The same is found to a much lesser degree for subjects who acquired their blindness later in life. Both sets of data demonstrate the vast capacity of the cerebral cortex to reorganize itself depending on the demands of altered environmental conditions.

8 Functional Neuroimaging of Crossmodal Associative Learning
Desiree Gonzalo, Tim Shallice , Ray Dolan
Functional imaging studies of learning and memory have primarily focused on stimulus material presented within a single modality (see review by Gabrieli, 1998). In the present studies we investigated mechanisms for learning material presented in the visual and auditory modalities, using event-related functional magnetic resonance imaging (fMRI). In one study we evaluated time-dependent learning effects in two conditions involving presentation of consistent (repeatedly paired in the same combination) or inconsistent (items presented randomly paired) pairs. We also evaluated time-dependent changes for bimodal (auditory and visual) presentations relative to a unimodal condition (auditory stimuli alone). Focusing on a time-by-condition analysis to compare paired (bimodal) versus unpaired (unimodal) conditions, time-dependent changes were found in posterior hippocampal for both consistent and inconsistent pairs. This study failed to look at crossmodal learning specifically, since the comparison consisted of paired versus unpaired items. In the second study we evaluated time-dependent effects in three conditions involving crossmodal (auditory-visual) and intramodal (auditory-auditory and visual-visual) pairs of stimuli. We found significant temporal modulation of response in left posterior hippocampus during learning of crossmodal pairs and right-sided activations during learning of intramodal pairs. These results provide evidence that associative learning for stimuli presented in different sensory modalities, is supported by neural mechanisms similar to those described for other kinds of memory processes. The replicated involvement of posterior hippocampus supports a putative function for this region in crossmodal associative learning, but not exclusively, and a possible lateralisation according to material type.

9 Prenatal sensory experience and postnatal intersensory development: Patterns that connect
Robert Lickliter
The link between typical patterns of prenatal sensory stimulation and subsequent postnatal perceptual development has received increasing research attention in recent years. For example ,studies of precocial avian species at both the neuro anatomical and behavioral levels of analysis have highlighted the importance of normal patterns of prenatal sensory experience to the emergence of early perceptual capabilities, including recognition and preference for maternal auditory and visual information, the capability for prenatal perceptual learning, and sensitivity to temporal and spatial features of sensory stimulation. Comparative-based research indicates that the varied effects that prenatal sensory stimulation may have on early perceptual development likely depend on a number of related factors, including the timing of the stimulation relative to the developmental stage of the organism, the amount of sensory stimulation provided or denied the young organism, and the type of sensory stimulation presented. Embryos and fetuses appear to be particularly sensitive to when, how much, and what kinds of stimulus events are made available during the prenatal period and these related factors can facilitate or interfere with normal patterns of postnatal perceptual organization. In general this body of developmental work on intersensory perception (a) highlights the strong link between modalities during the prenatal period, in that changes in the experience of one modality also impacts perceptual responsiveness in other modalities,(b) demonstrates the experience-dependent nature of early perceptual organization, and (c) suggests that the sequence of onset of sensory functioning during prenatal development can provide important limitations or constraints that actively shape subsequent perceptual responsiveness in the postnatal period
 
10 The Role of Experience in the Development of Multisensory Integration
Mark Wallace and Barry Stein
In cat, individual neurons in the superior colliculus (SC) synthesize convergent visual, auditory and somatosensory inputs in order to guide orientation behavior. The product of this cross-modal synthesis is typically large enhancements or depressions in the responses of multisensory neurons; changes collectively referred to as "multisensory integration." In the cat SC, multisensory neurons and multisensory integration appear gradually during normal postnatal development and these events follow a protracted and predictable timetable. Thus, whereas multisensory neurons appear at the end of the first postnatal week, they fail to exhibit multisensory integration until the end of the first postnatal month. We were interested in exploring the role that postnatal experience plays in shaping the development of this multisensory population. We have raised animals in complete darkness from birth until adulthood and have examined the distribution and properties of multisensory neurons. At each age examined (including the adult), multisensory neurons were reduced in number in the dark-reared population. The vast majority of these multisensory neurons had immature receptive fields, often being greater than 200% of adult size. Most dramatically, a significant loss of multisensory integrative capacity was apparent at each age examined. This striking deficit in multisensory integration suggests that the absence of experience with visual-auditory cues precludes the development of the normal ability to synthesize such cues. Future work will examine whether this abnormal experience simply delays, or serves to eliminate, the development of multisensory integration.

11 Development of infants' emotion perception: Implications for an obligatory process?
Arlene Walker-Andrews
The ability to recognize and act in response to another individual's emotions, whether signaled by face, voice, or gesture is an essential social task.ð DeGelder and Vroomen (2000) recently posited the "existence of mandatory bidirectional links between affect detection structures in vision and audition" in human adults (p. 289). In the present paper, experimental evidence from infants will be reviewed. Results from developmental research document an early (chronological) sensitivity to affective information in bimodal displays of emotion. Data from infants ranging in age from 5 to 7 months suggest that by 7 months infants detect the emotion-specific information available in bimodal presentations of affective expressions and look accordingly. Infants participating in preferential looking studies look predominantly at facial expressions that are accompanied by vocal expressions of the same emotion. This is the case for a number of emotion pairs--happy and angry, happy and neutral, happy and sad, happy and interested, angry and interested, sad and angry. Younger infants show somewhat different results. Their ability to detect the congruence between a facial and vocal expression may rely more on accompanying intermodal correspondences (such as synchrony), appears to be influenced by context (familiarity of person, familiarity of situation), and varies with respect to the valence of the emotional displays. Such results from research on infants' emotion perception are relevant to continuing discussions of models for crossmodal integration and to the development of emotion.

12 The missing (or, at least, neglected) link in developmental studies: The relationship between the perception of amodal invariants and of modality specific-relations
Davidð J. Lewkowicz
An infant's sensory/perceptual systems are continually bombarded by a multitude of concurrent multimodal inputs. Some of these inputs are amodal (equivalent across modalities) and some are modality-specific. Normally, infants must be able to perceive both the amodal nature of information and the relations among modality-specific cues in order to have a unified perceptual experience of the world. Thus, the problem for the infant is to select from the continuous flow of information the relevant amodal and modality-specific relations at the same time. This problem is more complicated for a developing infant than for an adult because (a) the detection of these two forms of intersensory relations must occur in the context of continuous structural and functional changes in sensory and perceptual systems, (b) the ability to detect these two forms of intersensory relations emerges at different points in development, and (c) sensory salience hierarchies can alter "veridical" perception. Most developmental research to date has focused primarily on the detection of amodal invariants and thus the interaction between the detection of amodal invariants and the detection of modality-specific relations has not been explicitly examined. The purpose of this talk will be to discuss the theoretical importance of this problem and to consider relevant empirical studies that suggest that this is, indeed, an important but neglected issue.

13 Silent Articulation changes speech perception 
Mikko Sams, Riikka Mñttñnen and Toni Sihvonen
When the auditory syllable /pa/ is dubbed onto the visual presentation of /ka/ articulation, subjects typically hear /ta/ or /ka/. This McGurk effect demonstrates that seeing speech may modify speech perception. We studied if similar modifications may be observed when subjects silently articulate the syllable /ka/ when /pa/ is presented with earphones. Four conditions were compared. 1) The McGurk condition with auditory /pa/ dubbed onto visual /ka/, 2) Articulation condition, where subjects silently articulated /ka/ when /pa/ was presented via earphones, 3) Mirror condition, where subjects saw their own articulation of /ka/ in a mirror when /pa/ was presented via earphones, 4) Auditory control condition, where subjects saw /pa/ syllable on a computer screen and had to recognize an auditory syllable immediately afterwards. Auditory stimuli were imbedded in noise (signal-to-noise ratio 0 dB). In McGurk condition, only 6% of the auditory /pa/ syllables were correctly identified. Interestingly, the results in Mirror condition did nor differ significantly from McGurk condition. A similar modification, albeit smaller, was also obtained in the Articulation condition, where the proportion of correctly identified /pa/ syllables was 32%, significantly (18%, p < 0.01) less that in Auditory control condition. The present results demonstrate that own articulation influences speech perception and that a strong McGurk effect can be produced when the subject sees his/her own articulation in a mirror.

14 Overview of Multimodal Communication in Animal Social Behavior
Sarah Partan
The cacaphony of simultaneous sights, sounds, smells, and vibrations produced by animals during natural social interactions may appear chaotic.ðð In an effort to impose order on these communication signals, I present a framework for categorizing signals based on the logical outcome of combining various components from different sensory systems.ð For simplicity, I will focus on bimodal signals (two sensory channels only).ð Examples will be widely drawn across taxa.ð Data on facial expressions and vocalizations collected from free-ranging rhesus macaques on Cayo Santiago, Puerto Rico, will be used for more in-depth illustration.ð The structure of their visual and vocal signals will be described, as well as the response of the animals to signal combinations in different sensory systems.ðð Both silent and vocal threats, for example, were followed by fearful behavior on the part of the recipients, but responses to silent threats were prolonged, suggesting that the visual channel alone can be as effective as bimodal visual/vocal signals in this population.ð

15 How does the brain combine information from the different senses?
Gemma Calvert
Behavioral reactions to concordant multisensory cues exhibit lower thresholds than their unisensory counterparts and similarly speed reaction times, crossmodal cues that are significantly discordant (e.g. spatially or semantically disparate) can have the opposite effect and depress responses.ðThis pattern of response enhancement and depression is also known to characterise the response of multisensory integrative cells in the superior colliculi of lower mammals to spatially concordant and discordant multisensory inputs (Stein and Meredith, 1993).ðOn the basis of these shared behavioural and physiological features, it is tempting to speculate whether the crossmodal binding of sensory inputs by convergence onto multisensory neurons may form a general physiological basis for intersensory synthesis which extends to humans and for the integration of inputs relating to stimulus identity (such as auditory and visual speech), or remains a feature peculiar to lower mammals and for the integration solely of inputs relating to stimulus location.ðBy modelling as accurately as possible situations in which the audible and visible components of speech would interact to enhance or degrade perception and using fMRI to study the resulting BOLD effects, we identified an area within the left superior temporal sulcus (STS) that displays response properties characteristic of multisensory integration cells, hitherto only demonstrated in non-human mammals.ðIn a further study using non-speech auditory and visual stimuli (8 Hz reversing checkerboard and 100ms white noise bursts) matched or mismatched solely on the basis of their temporal synchrony, similar supra and sub-additive effects were identified in the superior colliculus as well as in a network of cortical regions including the STS, intraparietal sulcus, insula, and lateral and ventro-medial frontal cortex.ðAnatomical studies in monkeys have identified these areas as putative multisensory sites on the basis that they receive afferents from multiple modalities.ðOur findings support a functional role for these areas in crossmodal integration although the precise role of these regions is now the subject of further study.ðSimilar studies examining the integration of spatially congruent and incongruent visuo-tactile cues suggest that whilst supra- and sub-additivity may be hallmarkðfeatures of multisensory integration, the network of brain areas implicated in these processes may be dependent on the precise nature of the signals being integrated and the particular crossmodal task being performed.
 
16 K e y n o t e
Speech Perception by Ear and Eye: A Paradigm for Multisensory Research
Dominic W. Massaro
Speech perception has been studied extensively in the last decades, and we have learned that people use many sources of information in perceiving and understanding speech.ð This talk focuses on the important contribution of visible information given in the talker's face and accompanying body gestures in face-to-face communication, and how it is combined with auditory speech. Speech perception is usually successful because perceivers optimally integrate several sources of information. In addition, audible and visible speech are complementary in that one source of information is most informative when the other source is not. These properties are well-described by the fuzzy logical model of perception (FLMP), a process model mathematically equivalent to Bayes theorem. The FLMP has also proven to provide a good description of performance in a wide variety of other domains of pattern recognition. For example, it describes how cues from both the face and the voice are evaluated and integrated to perceive emotion, and provide a variety of examples. The FLMP is consistent with findings of brain imaging in neuroscience and has remarkable parallels in the activity of multisensory neurons in the mature and developing cat and monkey superior colliculus. Demonstrations of multimodal phenomena will be provided.

17 A visual illusion induced by sound 
Ladan Shams
Traditionally, vision has been considered the dominant modality in our multi-sensory perception of the world. This view is primarily based on prior observations that in presence of a conflict between the information conveyed by the visual system and other modalities, the overall percept is determined by vision, or the quality of the percepts in the conflicting modality is modified by the visual information, and not vice versa.ð We report data that counter these views, by showing that auditory information can change the visual percept qualitatively (causing a visual illusion) and this effect occurs even in the absence of ambiguity in the visual domain. We have found a visual illusion which is induced by sound: when a single flash is accompanied by multiple auditory beeps, the single flash is incorrectly perceived as multiple flashes. We refer to this phenomenon as illusory flashing. Our experiments confirmed that the illusory flashing phenomenon is indeed a perceptual illusion. We next investigated the temporal properties of this illusion by varying the relative timing of visual and auditory stimuli. The illusory flashing effect declined from 70ms separation onwards. However, illusory flashing occurs so long as beep and flash are within approximately 100ms, consistent with the integration time of polysensory neurons in the brain. Others have shown that the perceived direction of motion of an ambiguous visual stimulus is influenced by auditory stimulation. Our work extends this previous finding by showing that the visual perception can be radically altered by sound even when the visual stimulus is not ambiguous. It is noteworthy that the conditions under which we have found this alteration to occur are by no means peculiar. To the contrary, the stimulus configuration as well as the task are very simple. The illusion is also surprisingly robust to variations in many parameters we manipulated (e.g., flashing disk's eccentricity and contrast, spatial disparity between sound and flash, shape and texture of the flashing pattern, flash and beep durations).ð These findings suggest that the illusory flashing phenomenon reflects a fundamental and widespread property of polysensory integration in the brain.

18

Crossmodal interactions in human cortex partly depend on the sensory dominance of the subject

Marie-Helene Giard, Alexandra Fort, C. Delpuech, J. Pernier.

A general observation of behavioral studies is that, whatever their dominant sensory modality, human subjects process more rapidly stimuli characterized by redundant bimodal information than the same stimuli containing unimodal information alone (Miller, 1982). Topographic analysis of event-related potentials has shown that this facilitation effect is associated with multiple crossmodal interactions in cortex that may be induced very early in sensory analysis (from 40-50 msec). Comparison of the neural networks of interactions and their temporal dynamics in three perceptual tasks using the same physical stimuli shows that the organization of these networks depends not only on the nature of the task, but also on the sensory dominance of the subject to perform this task in unimodal condition. Altogether, the data indicate that complex, highly sensitive and flexible neural mechanisms govern multisensory integration in perception.

19 Crossmodal stimulation through the looking glass: cognitive remapping of visual-tactile information in the normal and damaged brain. 
Angelo Maravita
The distance of visual stimuli from the skin surface constrains visual-tactile interactions. Electrophysiological studies in monkeys show that  the response of cross-modal visual-tactile cells is stronger for a visual stimulus presented close to their tactile receptive field. This makes functional sense given that in condition of natural stimulation, related visual-tactile information is normally conveyed with spatial-temporal correspondence. However, there is a common situation where visual stimuli appear far away in space despite being close to the skin surface: That is when we observe ourselves in mirrors, while acting on our own body (combing, etc.). In these cases, visual stimuli near our body (e.g. the comb in our hair) project the image of distant objects, as if placed through the looking glass". How does the brain cope with such paradoxical sensory information? 
We tested normal observers in a visual-tactile interference paradigm. In this paradigm usually tactile judgements are disrupted more by visual distractors close to the stimulated hand, than by far distractors. In one condition (Mirror) we presented visual distractors near the hands while subjects observed their own hands and the distractors reflected in a mirror. In another condition (Box) subjects observed, through the half silvered mirror, the contents of a box, which comprised visual distractors and two stuffed rubber hands, placed far away, at the exact position consistent with the reflections of visual distractors and subjectÃs hands in the Mirror condition. Visual interference was stronger in the Mirror condition than in the Box condition. 
A similar finding was observed when testing visual-tactile extinction in a right brain damaged patient. The effect of a right, ipsilesional visual stimulus in extinguishing awareness of a left, contralesional touch was stronger when the former stimulus was near to the ipsilesional hand, but observed distantly via a mirror, than when it was directly observed in the distance (box condition). These results suggest that visual stimuli near the hands, although optically appearing far away when observed in a mirror (through the looking glass"), are computed by the brain as being in peripersonal space, near to the corresponding body part, thus allowing strong visual-tactile interactions.
 
20 Crossmodal bias in audio-visual perception of emotion  
Bea De Gelder
 

21 A novel form of multisensory neuron resulting from excitatory-inhibitory convergence
Lisa R. Dehner, H.Ruth Clemo, and M. Alex Meredith

Attention to stimuli from one modality can result in reduced responses to stimuli from other modalities.ð These effects cannot be attributed to the activity ofð neurons which are excited by more than one modality.ð Instead, convergence of excitatory inputs from one modality with inhibitory afferents from another would produce a multisensory neuron in which activity evoked by one modality could be modulated (through inhibition and disinhibition) by the other.ð This novel form of multisensory convergence was examined in the anterior ectosylvian sulcal cortices of the cat. Neuroanatomical and antidromic stimulation techniques revealed that auditory Field AES (FAES) reliably projects to somatosensory (SIV), but electrical stimulation of FAES failed to orthodromically excite any SIV neuron.ð However, when FAES activation was paired with effective tactile stimulation, the responses of a majority of SIV neurons were suppressed.ð This suppressive effect was not achieved by stimulation of the adjacent primary auditory cortices, nor were somatosensory responses in the nearby SII regions affected by FAES activation.ð Ultimately, the application of bicuculline methiodide, the antagonist to the inhibitory neurotransmitter GABA- A, blocked the suppressive effects of FAES stimulation on SIV neurons, indicating the presence of active inhibition within the FAES-SIV circuit.These data are consistent with a cross-modal projection between higher-level auditory and somatosensory areas.ð Rather than exhibiting the familiar excitatory bimodal properties, manyð SIV neurons show a here-to-for undescribed form of excitatory-inhibitory convergence that ultimately results in modulation of activity in one modality by inhibition (and disinhibition) from another.ð In this fashion, attention to auditory stimuli could reduce SIV responses to spatially/temporally coincident tactile cues.

Supported byð NS39460 and by the Human Frontiers Science Program RGO174-98.


22 Auditory-visual convergence in the primate prefrontal cortex  
Liz Romanski

Our ability to recognize and integrate auditory and visual stimuli is the basis for many cognitive processes but is especially essential in meaningful communication.  Although many brain regions contribute to recognition and integration of sensory signals, the frontal lobes both receive a multitude of afferents from sensory association areas and have influence over a wide region of the nervous system to govern behavior. Furthermore, the frontal lobes are special in that they have been associated with language processes, working memory, planning, and reasoning, which all depend on the recognition and integration of a vast network of signals.

The dorsolateral prefrontal cortex, including Brodman's areas 8 and 46, has been previously associated with visuo-spatial processing (Goldman-Rakic, 1987).  However, recent studies indicate that it may also be involved in auditory spatial processing (Bushara et al., 1999).  The ventrolateral prefrontal cortex, referred to as the inferior convexity, is most often associated with language processing in the human brain.  Neuroimaging studies indicate that this area is also involved in visual working memory and therefore may be involved in multisensory integration. Recent analysis of the inferior convexity (IFC) of the prefrontal cortex in non-human primates has revealed a non-spatial visual processing region. In these studies, visual neurons with receptive fields that included the fovea were shown to have robust and highly specific responses to pictures of objects and faces (O'Scalaidhe et al., 1997).  Since the macaque inferior convexity is the recipient of auditory afferents (Romanski et al., 1999a, 199b) and the inferior frontal lobe in the human brain is involved in language processing we searched for an auditory processing region in the macaque IFC. We found that there is, in fact, an auditory receptive area with neurons that are responsive to complex sounds including species-specific vocalizations.  The fact that there are juxtaposed and overlapping areas of visual and auditory processing in the IFC indicates a possible role in multimodal sensory integration. Furthermore, the overlap of a visual face-processing area and an auditory area where neurons are activated by vocalizations suggests that this region may integrate multisensory stimuli for the purpose of communication and recognition. Experiments aimed at testing whether single neurons in the IFC are multimodal are currently underway.  Preliminary anatomical and physiological evidence suggests that the IFC of the primate frontal lobe has separate and overlapping domains for the processing of auditory, visual and multimodal information. Further analysis of the IFC of the frontal lobe in non-human primates will provide us with detailed knowledge of the neuronal networks and ensembles that support complex cognitive processes including mulitimodal communication.


23 Visual Imagery in tactile perception
Krish Sathian

Psychophysical studies implicate visual imagery in certain aspects of tactile perception. Functional imaging and event-related potential studies demonstrate that visual cortical areas are active during tactile judgments of certain object properties. Moreover, these visual areas are necessary for optimal tactile performance, as revealed by transient disruption of cortical

activity using transcranial magnetic stimulation. These findings in normally sighted subjects indicate that there are intimate cross-modal links between touch and vision.


24 Crossmodal links in spatial attention
Charles Spence
Most research in cognitive psychology on selective attention has implicitly assumed that each sensory modality can be considered in isolation, and has thus overlooked many issues concerning crossmodal links in attention. In daily life, however, our spatial attention often has to be co-ordinated across several modalities simultaneously. Recent research has revealed the existence of extensive crossmodal links in covert spatial attention between audition, vision, touch, and even olfaction. I will report a number of studies in normal people that highlight some of the constraints inherent in our ability to selectively focus and divide our attention across different modalities and locations simultaneously. Further studies demonstrate that these crossmodal links in spatial attention are maintained under conditions of receptor misalignment, such as when the eyes are deviated with respect to the head, or the hands are crossed over. Finally, I will show how multisensory integration can break down in the split-brainÃ, and in normal people receiving repetitive transcranial magnetic stimulation (rTMS).

25 Computational approaches to multisensory integration: an overview 
Christian Scheier
There has been a significant increase of empirical work on multisensory integration over the past years. Psychophysical and neurophysiological experiments are beginning to reveal some of the principles the brain uses to integrate information across sensory modalities. We can now start to ask which computational mechanisms best describe these principles. This talk will present an overview on the computational approaches to multisensory integration that have recently been proposed. We will focus on the principles that have been suggested for (a) coordinate transformations between sensory channels, (b) development and alignment of multisensory maps, and (c) spatial and temporal integration of multisensory information. We will cover a rather wide range of application areas ranging from integration in the superior colliculus to speech perception and sensor fusion in robots. Approaches discussed include race models, neural network models, Baysian models, and dynamical systems models. We will conclude by pointing out some of the issues that need to be addressed in future computational work on multisensory integration

26 Multiple forms of sensory integration: modeling the fusion process
John Jeka, Kelvon Oie and Tim Kiemel
Different tasks requiring multiple forms of sensory information may invoke fundamentally different processes to solve the fusion problem. For example, reaching to a target specified by more than one sensory source requires estimating position of the target based upon sensory information that is often discontinuous (e.g., flash of light, brief tone). The problem is often posed as where sensory fusion occurs along the processing stream to form a reliable estimate, which is known to be influenced by factors such as relative stimulus intensity, probability, S-R compatibility, etc.ð By contrast, fusion of multisensory information for human postural control requires online updating of estimates of a moving center of mass; a stochastic process based upon sensory information that is continuously available. An important issue for postural control is the nature of the fusion process. Is it inherently linear/nonlinear or does the process of fusion depend upon sensory context? We present results from two experiments which, when compared to the predictions of a 3rd-order model for postural control and its coupling to sensory information, suggest multiple forms of fusion. Subjects stood in front of large-screen visual display while lightly touching a rigid surface with the right index fingertip. In Experiment 1, anti-phase, oscillatory, visual and somatosensory stimuli were presented at a constant frequency (0.2 Hz). The amplitudes of the stimuli were inversely scaled relative to each other in 0.2 cm increments over a range of 0.0-2.0 cm. Across subjects, three forms of fusion were observed; linear averaging, reweighting and switching, indicating that the fusion process is not entirely dependent on sensory context.ð In Experiment 2, we investigated the fusion process when visual and touch stimuli were oscillated at different frequencies and different relative amplitudes. Preliminary analysis of fitted model parameters showed that changes in the gain response across condition were due primarily to changes in the weighting of sensory inputs and not due to changes in control system parameters. Results from these experiments suggest that the nervous system fuses multiple sensory inputs through both linear and nonlinear processes.

27 What are the fundamental, conceptually challenging problems for understanding multi-sensory fusion?
Gregor Schoner
This talk will concentrate on three main points.ð (1) Each individual source of sensory information may carry processes of interpolation or model-based estimation of parameters that characterize the physical source of stimulation. ð(2) Different sources of sensory information may differ with respect to these processes, including, in particular, the reference frames relative to which such interpolation/estimation takes place.ð (3) The fusion process may combine different estimates, but may also involve decision making in which selected sources of sensory specification are neglected or suppressed.ð These issues are discussed from the perspective of a potentially unifying dynamic field theory.ð Other frameworks originating in the engineering discipline of data and decision fusion are also referred to.

28 Modelling Multisensory Enhancement in the Superior Colliculus 
Tom Anastasio
The deep layers of the superior colliculus (DSC) integrate multisensory input and trigger orienting responses toward salient targets. DSC neurons are organized topographically, and the maps representing different sensory modalities are in register. Some individual DSC neurons receive multimodal sensory input, while others are unimodal.Multisensory enhancement (MSE) is a form of multisensory integration in which the neural response to a stimulus of one modality is augmented by a stimulus of another modality. Smaller unimodal responses are associated with larger percentages of multisensory enhancement -- a property called inverse effectiveness. We have developed a probabilistic model of MSE, based on the hypothesis that a deep SC neuron uses its sensory inputs (multimodal or unimodal) to compute the conditional probability that a target is present in its receptive field. We model the sensory inputs to the deep SC as random variables, and cast the computation function in terms of Bayes' rule. The Bayes' rule model reproduces MSE, and suggests that inverse effectiveness results because the increase in target probability, due to the integration of multisensory inputs, is larger when the unimodal responses are smaller. The difference between the input probability distributions when the target is present and when it is absent can be quantified using the Kullback/Leibler difference (D). Consistent with intuition, the amount of target information provided by the inputs increases as D increases, and information gain increases faster for multimodal than for unimodal inputs. However, as D increases further, information gain plateaus at the same level in both the multimodal and unimodal cases. Unimodal deep SC neurons may be those that receive high D input of one modality and have no need for input of another modality. The Bayes' rule model specifies that the amount of MSE exhibited by a deep SC neuron should depend upon the prior probability (unconditioned on sensory input) that a target will appear in its receptive field. This prior is primarily dependent upon the location of the deep SC neuron's receptive field. The model predicts that deep SC neurons with more peripheral receptive fields should exhibit larger amounts of MSE.

29 Information supporting direct multimodal perception 
Thomas Stoffregen and Benoit G. Bardy
In traditional or Helmholtzian theories of perception, the patterns of ambient energy (e. g., light, sound), that are available to perceptual systems are assumed to bear an ambiguous relation to reality. Given that perception generally is accurate, the assumed ambiguity in the stimulus implies that perceptual accuracy depends upon inferential or associative processing in the head. An alternative view is offered by the Ecological Approach to Perception and Action, in which it is argued that patterns of ambient energy bear an unambiguous relation to reality, such that these patterns specify reality. If this is true then, in principle, perception can be accurate without the intervention of inferential or associative processes ; psychological activity may concentrate on interactions with the (accurately perceived) world, rather than on the relation between sensory stimulation and reality. Students of the Ecological Approach have identified (both mathematically and through behavioral research) many patterns of ambient energy that, it is claimed, specify various aspects of reality. This effort has been sufficiently successful that it is now widely accepted that certain patterns are unambiguously related to particular physical events ; for example, that optical flow specifies self-motion. This effort has been concentrated on patterns that exist within individual forms of ambient energy, such as the optic array, the acoustic array, and so on. Such research accepts the traditional assumption that perception takes place in separate modalities (e. g., vision, audition). In the present contribution we present a novel analysis of the information available to perceptual systems, that is, of the stimulus. We point out the existence of patterns that extend across different forms of ambient energy, and we argue that these patterns provide information that is essential for the perception and control of ordinary behavior. These patterns make it possible, in principle, for " multimodal " perception to be accurate without inferential processing. This possibility would require a radical change in our understanding of perceptual systems, including rejection of the assumption that vision, audition, and so on exist as separate sensory channels or perceptual systems.
 
30 Prior Entry: Previous confounds and novel findings using an orthogonal design
David I. Shore, Charles Spence and R.M. Klein
Despite nearly two centuries of research, the question of whether attending to a sensory modality speeds the perception of stimuli in that modality has yet to be resolved. We highlight several confounds inherent in this previous research, and then report a series of four experiments designed to investigate the effects of attending to a particular sensory modality or spatial location on temporal order judgments (TOJs) using a novel methodology. Participants were presented with pairs of visual and tactile stimuli from either the left and/or right of fixation at varying stimulus onset asynchronies (SOAs), and were required to make unspeeded TOJs regarding which stimulus appeared first. Typically, visual stimuli had to be presented before tactile stimuli for them to be judged as simultaneous. Additionally, accuracy was significantly better when the stimuli were presented from different positions rather than from the same position. To be perceived as simultaneous, visual stimuli had to lead by a greater temporal interval when tactile stimuli were attended (Experiment 2A), and by less when visual stimuli were attended (Experiment 2B). When target stimuli were more likely on the left than right, participants judged the left stimulus as having been presented earlier than the right compared to when attention was spatially-divided or when participants attended to the right side (Experiment 3 and 4). These results provide the first conclusive evidence for the existence of crossmodal prior entry; they also provide support for previous claims of an attentional bias toward the visual modality, and also for a bias toward the right side of space.

31 Multisensory integration subserving orienting behavior
Douglas Munoz, Brian Corneil, Andrew Bell, Alex Meredith, John van Opstal
Gaze shifts are coordinated movements of the eyes and head that rapidly reorient the visual axis to a target of interest. To gain insight into the processes of multisensory integration, we studied the influences of competing visual and auditory stimuli on horizontal gaze shifts in humans in a variety of behavioral conditions. Gaze shifts were made to visual or auditory targets in the presence of either an irrelevant visual or auditory cue. Within an experiment, the target and irrelevant cue were either aligned (enhancer condition) or misaligned (distractor condition) in space. We manipulated the temporal asynchrony between onset of visual and auditory stimuli, the spatial location of stimuli, and the state of visual fixation at the time of stimulus onset. We then compared subject performance in enhancer and distractor conditions and measured reaction times and the frequency of incorrect gaze shifts made to the distractors. Our results reveal that, in addition to the spatial and temporal register of the stimuli, the experimental context in which the stimuli are presented influences multisensory integration. First, subjects had more difficulty suppressing incorrect movements to the distractors when the fixation target was extinguished prior to target onset. Second, an irrelevant auditory cue influenced gaze shifts to visual targets differently than an irrelevant visual cue influenced gaze shifts to auditory targets. The above studies have now been expanded upon by examining neural activity in the superior colliculus of monkeys trained to orient to either visual, auditory, or combined stimuli. The activity of saccade-related neurons was modulated by the nature of the saccade target and state of visual fixation. Further, the relative positions and temporal asynchrony of the visual and auditory stimuli also had a strong influence on the activity of these neurons. In terms of behavior, combination of the two stimuli in the enhancer condition resulted in a decrease in saccadic reaction time, similar to what we observed in humans.Our results revealed that the processes of fixation disengagement, multisensory integration, and target selection are interrelated and arise, at least in part, from interactions within the superior colliculus.
 
32 K e y n o t e
Multisensory Research: A summary, integration and future directions
Barry Stein
This talk will provide an overview of issues in Multisensory Science and the proceedings of this year's annual meeting, in addition to perspectives on early questions about Multisensory Integration and the current state of the field.

33 Is binding required to conjoin visual and tactile features?
Christopher T. Lovelace and Peter Grossenbacher

Discrete visual features like color and shape are processed in largely separate, parallel sensory channels, but are integrated ("bound") in perception. Behavioral studies suggest that feature binding occurs only after parallel processing of each feature is complete. Responses to visual targets identified by conjunctions of multiple features (e.g., line orientation and color) are slower than responses to targets identified by a single feature (the implication being that the added time is necessitated by the binding process). The current study sought to determine whether responses to targets identified by a conjunction of features in separate sense modalities are slower than responses to features in each sense. Twelve subjects discriminated low and high frequencies using vision and touch. A brief (151 ms) multisensory stimulus was presented on each trial, and consisted of a fingertip vibration (200 or 425 Hz) and a 5 deg. wide sinusoidal luminance grating (2 or 8 cpd). Frequency in vision and touch were varied independently across trials (ISI = 800 - 1200 ms). Subjects made speeded responses only to targets, which were defined by frequency in either a single modality (Feature) or both modalities (Conjunction). Each frequency and combination of frequencies was a target in separate trial blocks (target probability = 0.25).

If identification of Conjunction targets requires serial binding of the visual and tactual features, then reaction times (RTs) to Conjunction targets should be longer than RTs to Feature targets. Virtually all individual median RTs to Conjunction targets were at least 25 ms larger than to Feature targets, suggesting that identification of multisensory feature conjunctions is a serial process. In addition, there was a robust frequency correspondence effect between the vibrotactile and visuospatial features. Responses to vibrations were faster and more accurate when the vibration frequency matched that of the simultaneous (but irrelevant) luminance grating than when the frequencies did not match. Responses to luminance gratings were unaffected by vibration frequency. These results suggest that a) detection of conjunctions of visual and tactual features may depend on a serial binding process, and b) automatic processing of visual frequency influences tactual frequency discrimination.


34 Visual - tactile spatial interaction in a focused-attention task
Adele Diederich and Hans Colonius
Visual (LED) and tactile stimuli (shaker, vibration to palms) were simultaneously presented at various spatial locations. In a focused attention paradigm with visual targets and tactile distractors, saccadic response times (SRT) were shown to depend on spatial distance between the stimuli. Three experiments are reported. 1) Visual stimuli were presented 10 to the left or right of the central fixation point; tactile stimuli were presented 10, 50, 70, or 110 from central fixation (ipsi- or contra-lateral with respect to the LED). 2) Tactile stimuli were presented 10 to the left or right of central fixation; and visual stimuli were either presented 10, 50, or 70 from central fixation (ipsi- or contra-lateral with respect to the shaker). 3.) Visual and tactile stimuli were presented at the same locations 10, 50, or 70 from central fixation. Cross-modal interaction (SRT facilitation) could be observed as a function of the eccentricity of the stimuli, depending on both spatial distance and hand position.

35 Gating of afferent activity in an Aplysia mechanoafferent neuron
Elizabeth Cropper, C. G. Evans, J. Jing, S. C. Rosen, I. Kupfermann
The Aplysia mechanoafferent neuron B21 is peripherally activated during both phases of ingestive motor programs (i.e., during radula opening/protraction (O/P) and radula closing/retraction/closing (C/R)).ðO/P and C/R are mediated by different groups of cells and are antagonistic. This suggests that afferent transmission in B21 is likely to be regulated so its ability to excite specific follower neurons will be phase-dependent.ðFor example, B21 is not likely to strongly excite a C/R follower throughout both O/P and C/R.ðOur experiments seek to characterize mechanisms that regulate afferent transmission in B21.ðDuring C/R B21 receives excitatory synaptic input from other neurons.ðWhen it is depolarized, peripherally initiated spikes in B21 do produce EPSPs in a follower neuron (B8).ðIn contrast, when B21 is at its resting membrane potential afferent activity is not transmitted to B8.ðThus, central depolarization 'gates in' B21 input to B8.ðB21 is a bipolar neuron.ðOne branch innervates the periphery (i.e., acts as the input branch in the context) and the other branch makes contact with B8 (i.e., acts as the output branch).ðWhen B21 is at its resting potential spikes recorded from the output branch are smaller than those recorded from the input branch or the B21 soma. More specifically, spikes in the output branch are quite small (e.g., 20 mV) thus are unlikely to trigger transmitter release.ðIn contrast, when B21 is centrally depolarized spikes in the output branch become full size.ðThus, we have characterized a mechanism that gates in afferent activity in B21 during C/R.ðTo determine whether afferent activity is actively 'gated out' during O/P we recorded from B21 during ingestive motor programs and found that IPSPs were observed during O/P. In part these IPSPs are due to activity of the O/P interneuron B52.ðB52 makes a monosynaptic connection with B21, and in otherwise quiescent preparations B52 stimulation dramatically decreases the size of spikes in the output branch of B21.ðMoreover B52 stimulation decreases the size of B21-induced EPSPs in B8. In conclusion, we have characterized a mechanism that regulates the transmission of afferent activity in a functionally defined context.

36 The effect of sound intensity and noise level on audiovisual speech perception
Kaisa Tiippana, Mikko Sams and Riikka Mñttñnen

Audiovisual speech perception was studied at various sound intensities and noise levels.ðThe stimuli were meaningless words /aka/, /apa/, /ata/ uttered by a female Finnish talker, presented both audiovisually and unimodally.ðFor congruent audiovisual stimuli, the auditory and visual stimuli were the same.ðFor incongruent audiovisual stimuli, the auditory and visual stimuli were different.ðIncongruent stimuli typically show the so-called McGurk effect, i.e. alteration of the speech percept from that produced by the auditory stimulus alone [1].ðThe signal-to-noise ratio (SNR) was varied either by changing the sound intensity (24 to 72 dB in 6 dB steps), or by adding white noise to the signals (SNRs of -12, -6, 0 and +6 dB).ðThe aims of this study were: (i) To determine whether the two manipulations of the acoustic SNR have a similar effect on audiovisual speech perception.ðIt has been reported that the McGurk effect becomes stronger as the auditory signal becomes louder [2].ðOn the other hand, there is evidence that the number of McGurk responses is greater with more auditory noise [3].ðWe wanted to clarify this discrepancy.ð(ii) To study whether congruent and incongruent stimuli are similarly affected by degradations in acoustic speech quality. (iii) To provide extensive data for testing models of audiovisual speech perception.ðThe consonant recognition results of 22 subjects with normal hearing and vision showed the following trends:ð(i) Both SNR manipulations had a similar effect on the subjects' responses.ðFor example, the McGurk effect became stronger as sound intensity decreased and as noise level increased, i.e. as the SNR decreased.ð(ii) For congruent stimuli, adding visual speech had a greater effect on the number of correct responses at lower SNRs, when compared to unimodal conditions.ðFor incongruent stimuli, the number of visual responses increased as the SNR decreased.ðThese results can be interpreted to indicate that the visual influence becomes greater for both congruent and incongruent stimuli as the SNR decreases.ð(iii) The first step in model testing will be fitting the Fuzzy Logical Model of Perception [4] to our results.

References:

[1]ð McGurk & MacDonald (1976). Hearing lips and seeing voices. Nature, 264, 746-748.

[2]ð Kuhl, Green & Meltzoff (1988). Factors affecting the integration of auditory and visual information in speech: the level effect. Meeting of the Acoustical Society of America, Seattle.

[3]ð Fixmer & Hawkins (1998). The influence of quality of information on the McGurk effect. Auditory-Visual Speech Processing Conference, Terrigal, Australia.

[4]ð Massaro (1998). Perceiving talking faces. MIT Press, Cambridge, Massachusetts.


37 Is there a difference D between the amount of time required for auditory and visual stimuli to reach conscious awareness?
James V. Stone, N.M. Hunkin, J. Porrill, R. Wood, V. Keeler, M. Beanland, M. Port and N.R. Porter
This paper addresses the following question: is there a difference D between the amount of time required for auditory and visual stimuli to reach conscious awareness?ðOn each of 1000 trials, each of 23 observers was presented with a light and a sound, separated by a variable stimulus onset asynchrony (SOA) between -250ms (sound first) and +250ms (light first).ðThe light was a red LED (11cd/square metre), and the sound was a 1kHz square wave (71dB) presented through headphones. An observer indicated if the sound and light came on simultaneously by pressing one of two (yes/no) response keys, at which point both stimuli terminated.ðThe SOA most likely to yield affirmative responses was defined as the point of subjective simultaneity (PSS). The PSS for each observer was estimated as follows. The probability of an affirmative response described an inverted U-shaped curve as the SOA varied between -250ms and +250ms.ðThis inverted U-shaped curve was fitted with a Gaussian function using maximum likelihood estimation (MLE), and the PSS was taken to be the mean (peak) of the fitted Gaussian function.ðThe resultant observer-specific PSS values varied between -21ms (sem=4.5ms) (i.e. sound 21ms before light) and +150ms (sem=17.4ms) (i.e. light 150ms before sound) for different observers.ðAs acontrol, the difference RTd in simple reaction times (RT) to single audio and visual stimuli was also measured.ðThe PSS and RTd values have different means (t=2.10, p=0.052), and are uncorrelated.ðThe stability ofPSS and RTd values was evaluated by testing observers twice, with test sessions at least 24 hours apart.ðThe resultant pairs of PSS values, but not corresponding RTd values, are highly correlated (r=0.954, p=0.003). Together these results suggests that PSS values, but not RTd vales, are stable and observer-specific.ðThe implications of these findings for the perception of multisensory events are discussed.

38 Similar densities of GABA-ergic neurons in visual, auditory and somatosensory 'association' cortices of the cat
H. Ruth Clemo and M. Alex Meredith
It is well known that cortical receptive field properties are shaped byinhibitory processes, regardless of the sensory modality involved. Despite the physiological and perceptual distinctions among the different sensory modalities, the density of identified inhibitory (GABA-ergic) neurons is very similar across the different primary sensory cortices. However, receptive field properties become progressively more complex at higher cortical levels, and it is not known whether the inhibitory contributions to these higher-order processes requires larger , or different, proportions of GABA- ergic neurons.ð This possibility was examined in the cortices surrounding the AnteriorEctosylvian Sulcus (AES) which contains the higher order visual (AEV),somatosensory (SIV) and auditory (Field AES) representations. Coronal sections through the AES cortices were derived from 5 adultcats.ð Alternate sections were reacted for GABA, GAD65 and GAD67 using standard immunocytochemical procedures with silver-gold intensification. Tissue-outlines and immuno-positive neuron locations were plotted using a Nikon Optiphot with a computer-controlled stage; the number of labeled neurons was tabulated and the tissue area calculated. Measurements were taken from across the full thickness of the cortex (all laminae, I-VI)within AEV, SIV,ð and Field AES as well as from the adjacent gyralregions of AI/AII and SII.ð The mean density (neurons/mm2) and standard deviation of GABA-positive neurons was calculated for each cortical field: AEV =ð 92.1 ± 35.1;ð SIV =ð 103 ± 39.3;ð Field AES = 99.2ð ±54.3; AI/AII =ð 88.5 ± 24.9; and SII =ð 89.4 ± 35.1.ðð None of these values differed significantly from the other, nor did they differ dramatically from those obtained from the different primary sensory cortices. These data demonstrate that the concentration of inhibitory, GABA-ergicneurons within a given sensory cortex does not vary according to hierarchical level or sensory modality. This suggests that hierarchicalor modality-specific distinctions among receptive field properties result from other factors, perhaps such as differences in morphology of GABA-ergicneurons,ð their laminar distribution, or the density and distribution of GABA-ergic terminals.Supported byð NS39460 and by the Human Frontiers Science ProgramRGO174-98.

39 Tactile 'Inhibition-of-Return' (IOR) in congenitally blind humans
Brigitte Roeder, Charles Spence, and Frank Roesler
Our attentional systems orient reflexively to novel environmental stimuli. A period of inhibition, known as 'inhibition-of-return' (IOR), typically follows such attentive orienting. IOR has been demonstrated to affect the processing of visual, auditory and tactile stimuli, and also occurs crossmodally. IOR has been linked to the eye movement system and provides a tagging mechanism that prevents perseveration and hence facilitates attentional search. We examined whether the capacity for oculomotor control is a necessary prerequisite for the generation of IOR. We tested a group of 11 congenitally blind people who have either no control of their eye movements or no eyes at all, and 13 blindfolded sighted controls in a tactile IOR paradigm. Tactile targets requiring a speeded detection response were presented to the left or right index fingers, and were preceded by tactile cues occurring at either the same or opposite finger (ipsilateral vs. contralateral trials respectively). Both the sighted and congenitally blind groups demonstrated tactile IOR (i.e., longer target detection latencies on ipsilateral trials than on contralateral cue trials). Tactile IOR was also demonstrated in the blind participant without eyes. Overall mean target detection times were faster in the blind than in the sighted group. These results contradict the view that IOR depends on the control of eye movements, suggesting instead that IOR may reflect, at least in part, a more general inhibition of the orienting response. In addition, the shorter target detection times of the blind (compared to sighted) point to crossmodal compensation due to blindness.
 
40 Crossmodal Dynamic Capture
Salvador Soto-Faraco, Alan Kingstone and Charles Spence
We introduce a new crossmodal illusion in which the perceived direction of movement of a stimulus in one modality is strongly modulated by the direction of movement of a stimulus in a different modality. Participants were asked to determine the direction of movement (leftward vs. rightward) of two successive sounds (pure tones of 50 ms duration, 100 ms ISI). Participants performed at chance when a pair of simultaneous light flashes moved in the opposite direction to the sounds. However, performance was at ceiling when either: 1) No flashes were presented; 2) The lights moved in the same direction as the sounds; or 3) The visual and auditory stimuli were desynchronised, regardless of their respective direction of movement (i.e., same or different). These results show that crossmodal dynamic capture is not produced by cognitive or 'late' response bias factors. Furthermore, ventriloquism (capture) effects were significantly weaker in situations where no motion was perceived (i.e., static displays), demonstrating the crucial role of motion informtion in modulating our effects. These dynamic capture effects are consistent with the view that motion information across different sensory modalities is integrated automatically prior to perception.

41 EEG evidence for early audio-visual integration in the human brain
Klaus Linkenkaer-Hansen, J. Matias Palva, Kai Kaila, and Risto J. Ilmoniemi

A recent EEG-study in humans indicates that the processing streams of auditory and visual information interact already at 40 ms after stimulus onset [1]. Non-linear interaction between the processing streams of different stimuli, however, does not indicate integration of the information. It thus remains unresolved at what latency an integrated audio-visual neural representation emerges from the audio-visual interactions. To dissociate audio-visual (AV) integration from mere AV interaction in MEG/EEG signals, we designed a paradigm in which cross-modal integration was required both for the subject to perform a discrimination task and for the compared brain responses to differ. Four AV stimuli were created using two face images (V1 and V2) and two acoustic syllables /pa/ and /ka/ (A1 and A2). The two standard combinations A1+V1 and A2+V2 occurred with a probability of 0.4 each, while the two deviants, A1+V2 and A2+V1, occurred with a probability of 0.05 each. Additionally, a third image (V3) and the syllable /la/ (A3) occurred at random together either with A1 or A2 and either V1 or V2, respectively, with probability of 0.025 for each combination. In the first, bimodal condition, the deviant AV pairs served as targets in a GO/NO-GO task. In the second condition, the unimodal deviants (A3 and V3) served as targets in a unimodal GO/NO-GO task. Evoked responses were recorded with simultaneous whole-scalp 122-channel MEG and 64-channel EEG. Comparison of the average of the two standards with the average of the two deviants in the bimodal condition revealed a larger negative deflection to AV standards peaking at around 70 ms. The results indicated (p < 0.01) AV integration in the latency range of 60-105 ms (EEG, n = 10). The difference between bimodal standard and deviant responses in the unimodal task was not significant before 360 ms post stimulus, suggesting that strictly bimodal attention is required for early AV feature integration to take place at the cortical level. To our knowledge, this is the first unambiguous electrophysiological demonstration that early AV interactions can reflect true integration of auditory and visual features in the human brain.

[1] Giard MH & Peronnet F, J. Cogn. Neurosci. 11:5, 473-490, 1999.


42 Visual-tactile sensory integration in man: a functional anatomy study using positron emission tomography
Nicholas Stafford, M. Mehta, R. Banati, S. McGowan, J. Aggleton, M. Khwaja, P. Grasby, and C. Bench

Combining information from different sensory streams is important for many cognitive and behavioural functions. We have extended our previous PET study of visual-tactile integration (1) by using (a) a modified version of the arc-circle test incorporating additional control conditions, (b) a more advanced PET camera with wider axial field of view, (c) reducing putative attentional load by including a learning procedure. Eight volunteers, 35-59 years, were tested using the modified arc-circle test, which required subjects to compare touched metal arcs with viewed circles and arcs on a computer screen in three different test conditions for which only instructions differed. Subjects were told to compare the right-hand arc with circles on the screen (touch-vision or TV), or the right-hand arc with the left-hand arc while looking at the screen (TT), or arcs to circles, both on the screen (VV).ð Subject responded using single words. A rest condition was included.ð Regional cerebral blood flow measures were made four times per condition, in a random order, using the H215O bolus method with the Seimens/CTI EXACT 966 PET camera.ð Data were analyzed using SPM99. We used P<0.001 (uncorrected) to signify activations in areas seen in our previous study; P<0.05 (corrected) was used for all other areas.ð The VV-TV comparison replicated our previous findings of bilateral visual cortex activation (BA18/19, P<0.001). TV-VV revealed activations in left BA40 (P<0.001) as reported previously and, in addition, activations in left BA3 (P<0.05) and right cerebellum (P<0.05) in the extended axial view. For TT-TV activations were present in right BA6 (P<0.05) and right BA1/3 (P<0.05). TV-TT revealed bilateral activations in BA18/19 (P<0.05) and left BA40 (P<0.05) more posterior to that in the TV-VV contrast. As previously (1), no amygdala activations were seen. Yet, in contrast, we did not identify activations in the anterior cingulate and DLPFC during cross-modal matching, perhaps due to the pre-scan training shown by improved and stable task performance. These findings suggest modulation of activity in primary sensory areas is specifically important in visual-tactile matching, possibly via selective attention as evidenced by activations seen in the parietal lobes.

[1] Banati et al. (2000) Neuropsychologia 38:115-124.


43 Crossmodal Integration of Sight and Sound by the Human Brain
Ingrid Olson, C.J. Gatenby and J.C. Gore
Previous theories of crossmodal integration propose that sensory-specific areas are involved in representing crossmodal stimuli.ð Another theory proposes that only a central relay point is needed for crossmodal processing. To compare these two hypotheses, we used functional magnetic resonance imaging (fMRI) to identify the neural regions involved in audio-visual crossmodal integration of speech. Ten subjects were imaged while viewing a face in four different conditions: with speech and mouth movements synchronized, with speech and mouth movements desynchronized, during silent speech, or viewing a static face. Subtractions of the different sets of images showed that lip-reading, as defined by the silent speech - static face contrast, primarily activated Broca's Area and STS/STG. Synchronized audio-visual speech and desynchronized audio-visual speech activated similar areas, including Broca's Area, STG, and STS, suggesting that these areas are equally involved in the processing of unimodal and crossmodal stimuli. Regions activated more in the synchronized versus the desynchronized conditions were considered to be those involved in crossmodal integration. One dominant activation focus was found in the left claustrum/putamen. This study extends previous results, using other sensory combinations , and other tasks, indicating involvement of the claustrum in sensory integration. We hypothesize that the claustrum acts as a relay station for different sensory modalities while parts of the basal ganglia register the temporal co-occurrence of different events.

44 Fusing Vision and Touch Through Sensory Reweighting
Kelvin Oie, Tim Kiemel and John J. Jeka
The control of upright stance and postural orientation necessitates the integration of at least visual, somatosensory and vestibular inputs. These channels are held to provide both unique and redundant information relative for posture, enabling the resolution of potential ambiguities, as well as allowing the maintenance of posture when one or more sensory channels is not available. It has been suggested that in optimizing the control of stance these multiple inputs are re-weighted in a dynamic or adaptive manner. However, definitive support for sensory re-weighting is confounded by the possibility that the postural system may simultaneously adapt its control system parameters. By estimating the parameters of a third-order linear model that captures both the postural control system and its coupling to sensory information, we can assess whether sensory coupling and/or control parameters adapt as sensory conditions change. Utilizing a multisensory "moving room paradigm", we presented subjects with simultaneous, oscillatory visual and somatosensory stimuli at 0.2 Hz and 0.28Hz, respectively. Previous observations have shown that the postural response is strong with low-amplitude stimuli, dropping off as stimulus amplitude increases. Therefore, the stimuli used here manipulated relative stimulus amplitude in five conditions: (visual amplitude (mm): somatosensory amplitude (mm)) 4:16, 4:8, 4:4, 8:4 and 16:4. If the postural control system does not re-weight sensory inputs and/or change its control parameters, gain with respect to a given stimulus should not depend on the amplitude of either stimulus. However, if it does adapt to changing sensory conditions, we predict increasing gain with respect to low-amplitude stimuli as the amplitude of the other stimulus increases. Preliminary results showed systematic increases in gain to stimuli at constant amplitude (e.g., vision at 4 mm) when the other stimulus became large (e.g., touch at 8 and 16 mm), and vice versa. Furthermore, fitting model parameters for each condition suggested that the observed changes in gain were due primarily to changes in coupling strength parameters rather than to changes in control system parameters. These results suggest that the postural system may indeed re-weight sensory inputs under different conditions of stimulation. Supported by NIH grant R29 NS35070-01A2 (John J Jeka, PI).

45 Listening to Shapes: The Cross-Modal Equivalence of Vision and Audition in the Presentation of 2-D Shapes
Eric C. Odgaard
Spatial data are usually presented in some visual format, such as line graphs and two-dimensional shapes.ðIf the perceiver is either visually impaired or engaged in an eyes-busy task (such as surgery or piloting an aircraft), visual presentation of such information is impractical.ðThis study investigated the viability of displaying 2-D shapes (the border contours of U.S. states) in the auditory modality via pitch-time displays: auditory displays in which pitch represents vertical position and time represents horizontal position.ðCross-modal matching tasks revealed that perceivers can identify shapes at greater than chance levels; and that when errors in matching are made, they are based on similarities in features of the stimuli (rather than on random chance).ðMultidimensional Scaling Analyses further supported this conclusion, by revealing a stable set of salient features underlying the auditory presentation of the stimuli across several instruction sets. Comparison of the scaling solutions from the auditory presentations to those of a visual presentation revealed that, while the perceptual structure underlying the auditory presentation is stable, it is only partially equivalent to that of the more traditional visual modality.ðIn an applied sense, these results suggest that pitch-time displays may provide a viable alternative for the presentation of complex spatial data.ðIn a theoretical sense, these data would seem to further suggest that performance equivalence can occur between the modalities despite only limited perceptual equivalence between the stimuli.

46 Crossmodal interference between audition and touch
Donna M. Lloyd, Steve Guest, Charles Spence, Natasha Merat and Francis P. McGlone
We report a series of experiments designed to investigate crossmodal links in focused attention between audition and touch. Participants made speeded two-choice discrimination responses concerning the elevation (up vs. down) or roughness (rough vs. smooth) of targets presented in one modality, while trying to ignore irrelevant distractors presented simultaneously in the other sensory modality. The distractors were either congruent (i.e., an upper auditory target together with an upper tactile distractor) or incongruent (i.e., a rough tactile target surface together with the sound of a smooth surface) with the target. Robust compatibility effects were reported (i.e., responding was faster and more accurate when target and distractor were congruent than when they were incongruent) for targets in both modalities under a majority of conditions. These results add to a growing body of research demonstrating extensive crossmodal links between audition and touch in the control of attention.

47 Audio-visual sensory interaction in film perception
Nicola Phillips
Sensory interaction and integration in film perception was explored through a preliminary experimental study of the effects of concurrent musical listening and film viewing on perception of visual 'pulse'* in moving images.ðThis was grounded in two assumptions: that the contribution of audition and vision to integrated audio-visual percepts might be asymmetrical, reflecting the adaptation of these sensory systems for transmission of complementary information, and that the differing transduction times and temporal acuity of audition and vision might result in a processing bias towards auditorily-conveyed temporal information in filmic audio-visual perception. The experiment tested the hypothesis that perception of visual pulse would be modified by concurrent musical listening.ðParticipants viewed seven naturalistic film excerpts, each presented both without music and with various musical accompaniments.ðDuring viewing, participants tapped the perceived underlying visual 'pulse' of the moving images.ðComparison of response inter-onset intervals in 'with music' and 'visuals only' trials showed that the perceived underlying visual pulse of moving images was more rapid in the presence of music than in its absence.ðThe scope for musical listening to influence perception of underlying temporal structure of a moving image appeared to depend on stimulus ambiguity.*'pulse' is used in this sense to mean underlying pace of temporal articulation of a moving image, about moments of change or stasis.

48 Attending to Pain: Nociceptive Attention, Crossmodal Attention & Visual Dominance
Francis P. McGlone, Charles Spence, Deborah E. Bentley, Paula D. Youell and Anthony K. P. Jones
Mechanisms of selective attention serve to facilitate the selection of appropriate objects and events from amongst the cluttered and noisy scenes of everyday life.ð Many researchers have claimed to show that selective attention can also be directed to nociceptiveð (i.e., painful) stimuli? However, a number of alternative interpretations of these findings remain possible in terms of criterion shifts, task-switching, and spatial confounds. We examined whether attention to nociception could still be demonstrated when these alternatives were ruled out. Participants made speeded footpedal discrimination responses to an unpredictable sequence of nociceptive (pulsed CO2 laser) and visual stimuli presented to two locations on the left volar forearm. Attention was directed to one or the other modality by means of a symbolic cue at the start of each trial that predicted the likely modality for the upcoming target on the majority of trials. Significantly larger cuing effects were reported for visual stimuli than for nociceptive stimuli. The implication of these results for the multiple resource view of attention, and for the attentional account of visual dominance (Posner, Nissen & Klein, 1976) are discussed.

49 Both auditory and visual attention modulate activity in area MT+
Rebecca Berman and Carol Colby
The aim of this study was to determine whether attention to nonvisual stimuli can have an impact on activity in early visual cortex.ð We used the motion aftereffect (MAE) to assess visual motion processing in area MT+ during visual and auditory attention conditions. Perception of the MAE occurs after prolonged viewing of a unidirectional moving stimulus: when subjects subsequently view a stationary stimulus, it appears to be moving in the direction opposite to the adapting motion. Psychophysical studies have demonstrated that the MAE is diminished following attention to a central visual stimulus (Chaudhuri 1990), while functional imaging studies indicate that MAE perception is correlated with activity in area MT+ (Tootell et al. 1995). The present study brings together these psychophysical and FMRI findings to ask: Can attention to central visual or to auditory stimuli influence (1) perception of the MAE? and (2) activity in area MT+?We acquired psychophysical and FMRI data at 1.5 Tesla while subjects viewed moving and stationary stimuli in alternating blocks.ð Our FMRI measure of interest was the 'MAE signal': the time for activity in MT+ to decay back to baseline following different conditions of motion adaptation. In the first scan, we established each subject's MAE perception and associated MT+ activity in the absence of attentional demands by comparing reversing motion, which does not induce the MAE, with outward-only motion, which does induce the MAE.ð In the second scan, we asked whether this MAE signal was altered if the adapting stimulus was accompanied by a concurrent central visual task or auditory task.ð Attention to the central visual task caused the MAE signal to decay more rapidly than in the baseline (no-task) condition.ð Attention to the auditory task also caused a similar reduction of the MAE signal.ð Subjects' perception of the MAE paralleled the FMRI results, with diminished MAE perception following both visual and auditory attention conditions.ð These data indicate that attention to auditory stimuli can have an impact on early visual processing, and suggest that attention is not purely modality-specific.
 
50 Multisensory audio-somatosensory neural response interactions in early cortical processing: a high-density electrical mapping and behavioral study in humans
Micah M. Murray, Beth A. Higgins, Deirdre M. Foxe, Kevin H. Knuth, Daniel C. Javitt, Charles E. Schroeder and John J. Foxe

We examined the time course and scalp topography of auditory-somatosensory (A-S) neural response interactions in humans using high-density (128-channel) electrical mapping of somatosensory evoked potentials (SEPs) and auditory evoked potentials (AEPs). Tactile stimuli consisted of 15ms vibratory stimulation (~685Hz) of the fingertips of each hand. Auditory stimuli consisted of 15ms white noise bursts delivered to either of two speakers located next to each hand. Stimuli were presented either unimodally in isolation or bimodally in either conjunctive (same side) or disjunctive (different sides) simultaneous combinations. Subjects completed a simple reaction time task by making foot pedal responses to stimulus detection, regardless of whether stimuli were unimodal or bimodal, while maintaining central fixation. ERPs in response to bimodal stimuli were compared to the summed SEP and AEP from the corresponding unimodal conditions. These responses would be equivalent if neural responses to the unimodal stimuli were independent, whereas divergence is indicative of neural response interactions. In a previous study, we demonstrated A-S neural response interactions at ~50ms post-stimulus onset during passive conditions [1]. Similar interaction effects are seen in the current data set both when the bimodal stimuli were conjunctive combinations as well as when they were disjunctive combinations. Reaction times to bimodal stimuli were faster than those to unimodal stimuli, demonstrating a redundant target effect (RTE) for simultaneously presented A-S stimuli. Further, preliminary results indicate that the RTE is larger in the case of disjunctive versus conjunctive stimulus combinations. Probability summation could fully account for the RTE in each bimodal stimulus combination. Thus, although multisensory A-S neural response interactions occur early in cortical processing, they need not be invoked as a mechanism to describe the reaction time facilitation of the RTE.

[1] Foxe J.J., Morocz I.A., Murray M.M., Higgins B.A., Javitt D.C., and Schroeder C.E. (2000) Multisensory Auditory-Somatosensory Interactions in Early Cortical Processing. Cognitive Brain Research, 10(1-2): 77-83.


51 Multisensory integration in the superior colliculus of the awake behaving primate
Andrew Bell, B.D. Corneil, M.A. Meredith, J. Van Opstal, and D.P. Munoz
The superior colliculus (SC), involved in the generation of saccades, also receives converging sensory inputs of different modalities and is a site of multisensory integration.ð We studied multisensory integration in awake, behaving monkeys.ð Single cell activity was recorded from the SC while monkeys performed a number of behavioral paradigms incorporating both visual and auditory stimuli.ð The paradigms were designed to investigate both the sensory response characteristics to multimodal stimuli and their influence on saccade related activity and resultant behavior.ð The paradigms included both fixation tasks as well as those which required monkeys to generate saccades to the stimuli.ð The influence of state of fixation was examined by presenting stimuli in both the gap condition (fixation point extinguished prior to stimulus presentation) and the overlap condition (fixation point visible during stimulus presentation). SC neurons responding to visual and/or auditory stimuli were examined.ð When the two stimuli were combined, we saw both response enhancement (response magnitude exceeded unimodal responses) and depression (magnitude decreased).ð Neurons which exhibited these interactions were located primarily in the deeper layers of the SC and often exhibited saccade related activity.ð These interactions were modulated by the spatial relationship between the two stimuli, the monkey's state of fixation (i.e., gap vs. overlap condition), as well as the parameters of the behavioral task. This study extends earlier findings on multisensory integration in awake, behaving primates and allows us to examine both its neural aspects as well as how these interactions affect the programming and characteristics of the ensuing saccades.ð Supported by HFSP Grant RG0174/1998-B.

52 Multisensory audio-visual interactions in early cortical processing: A high-density electrical mapping and behavioral study.
Sophie Molholm, John J. Foxe, Micah M. Murray, Beth A. Higgins, Charles E. Schroeder, Daniel C. Javitt, and WalterðRitter
Multisensory integration plays a pivotal role in human performance and perception.ð How and when information from separate sensory modalities comes together in the human brain is not well understood. Information regarding the timing and neural generators of multisensory intereactions is needed as a basis from which to understand how sensory integration operates.ð For example, are interactions driven by stimulus properties or higher order processes?ð Electrophysiological studies that have considered the timing of audio-visual interactions have shown variable onset times for such effects (e.g., de Gelder et al, 1999; Giard & Peronnet, 1999; Pourtois et al, 2000; Sams et al., 1991; Schrñger & Widmann, 1997).ð These differences are likely a result of stimulus, paradigm, and methodological differences.ð For example, while Giard and Peronnet (1999) have shown interaction effects as early as 40 ms over visual areas, Schrñger and Widmann (1997) did not show them until about 190 ms. The surprisingly early onset of Giard and PeronnetÌs (1999) interaction effects appear to belie current understanding ofð the activation time course of visual cortices, while the onset of Schrñger and WidmannÌs earliest effects seem late in light of what has been shown with intracranial investigations in animals.ð The present study re-examined the timing and topography of cortical audio-visual integration effects using high-density electrical mapping (128-channels). Single visual and auditory stimuli were presented either alone or together.ð Subjects were instructed to respond to all stimuli. ERPs to the visual alone and auditory alone stimulus conditions were summed (ÎsumÌ) and compared to the ERPs to the auditory and visual together (ÎpairÌ) condition.ð Divergence between the ÎpairÌ and ÎsumÌ ERPs indicated a neural response interaction.ð In the 200 ms post stimulus onset examined, interaction effects were present over visual areas at about 46-66 ms, corroborating the findings of Giard et al. (1999), and also at about 160-190 ms. Over auditory projection areas, significant interaction effects were observed at about 50-64, 110-130 and 175-200 ms.ðð The topography and timing of the earliest interactions are consistent with a system of areas in both visual and auditory association cortices that subserve multisensory audio-visual interactions early in the cortical processing hierarchy. In the behavioral data, reaction times were faster in the paired condition compared to the alone conditions, indicative of a redundant target effect. However, probability summation could fully account for this reaction time facilitation, providing no indication that neural response interactions facilitated reaction-time performance.

53 Auditory and Visual Objects
David Van Valkenburg and Michael Kubovy
Notions of objecthood have traditionally been cast in visuocentric terminology. As a result, theories of auditory and cross-modal perception have focused more on the differences between modalities than on the similarities. In this talk we will re-examine the concept of an object in a way which overcomes the traditional perspective. We propose a new, cross-modal conception of objecthood which focuses on the similarities between modalities instead of the differences. Further, we propose that the auditory system might consist of two parallel streams of processing (the 'what' and 'where' systems) in a manner analogous to current conceptions of the visual system. We suggest that the 'what' systems in each modality are concerned with objecthood. Finally, we present evidence for the hypothesis that the auditory 'where' system is in the service of the visuo-motor 'where' system.

54 Intersensory Redundancy Guides Perceptual Learning: Discrimination of Tempo in 3-Month-Olds
Lorraine Bahrick, Robert Lickliter and Ross Flom
Bahrick and Lickliter (2000) recently proposed an intersensory redundancy hypothesis, which holds that in early development information presented redundantly and in temporal synchrony across two sensory modalities selectively recruits attention and facilitates perceptual learning more effectively than does the same information presented unimodally.ð This hypothesis was supported by the finding that 5-month-old infants were able to differentiate between two complex, five-element rhythms when the rhythms were presented bimodally, but showed no evidence of differentiating the rhythms when they were presented unimodally (either visually or acoustically).ð The present study extended our test of the intersensory redundancy hypothesis by assessing discrimination of a new amodal property (tempo of action), in a group of younger infants (3-month-olds), using the same stimuli and procedures as the prior study. Thirty-two infants were habituated to films of a hammer tapping out a rhythmic sequence at a given tempo (55 or 120 beats per minute, counterbalanced), half under the bimodal (auditory and visual), and half under unimodal (visual) conditions.ðð Following habituation, infants received a change in tempo under their respective condition.ð Results replicated those of our prior study and demonstrated that infants showed significant discrimination of the change in tempo following bimodal habituation, t (15) = 2.92, p<.01, whereas they showed no evidence of discrimination following unimodal visual habituation.ð These findings support the intersensory redundancy hypothesis and suggest that bimodal stimulation selectively guides infant attention to redundant properties, and in turn, supports perceptual processing of those properties at the expense of others, giving a processing advantage to amodal information in early development.ðð It thus appears that when infants first learn to differentiate an amodal property, discrimination is facilitated by intersensory redundancy and attenuated under conditions of unimodal stimulation.

55 Tactile 'Capture' of Audition
Anne Caclin, Salvador Soto-Faraco, Alan Kingstone and Charles Spence
Previous research has demonstrated that the localization of auditory or tactile stimuli can be biased by the simultaneous presentation of a visual stimulus from a different spatial position. The authors investigated whether sound localization could also be affected by the presentation of spatially-displaced tactile stimuli, using a procedure designed to reveal any crossmodal perceptual interactions. Participants made left-right discrimination responses regarding the apparent location of sounds, which were presented in isolation, or together with tactile stimulation to the fingertips. The results demonstrate that the apparent location of a sound can be biased toward tactile stimulation when it is synchronous, but not when it is asynchronous, with the auditory event. Directing attention to the tactile modality did not increase the bias of sound localization toward synchronous tactile stimulation. These results provide the first unequivocal demonstration of the tactile 'capture' of audition.

56 Auditory-visual integration in human saccades elicited in a noisy environment
Marc van Wanrooij, Brian Corneil, Doug Munoz, John van Opstal

We report on behavioral ; experiments with five human subjects who made saccadic eye movements to a target that could be either visual-only (V), auditory-only (A), or auditory-visual (AV). In each trial, the target was hidden within an audio-visual background, and subjects were allowed up to 4.5 sec to identify the target as quickly as possible. To disambiguate the task in AV conditions, the spatial coordinates of the two stimuli always coincided. The target could be presented at either one of 24 possible locations, distributed over the 2D oculomotor range. To vary the strength of potential auditory-visual interactions, both the timing of the auditory target re. visual target onset, as well as its intensity with respect to the acoustic background noise
(i.e. the S/N ratio), were systematically varied. All 17 different stimulus conditions (one V, 4 A, 12 AV conditions) were randomly interleaved, and for each subject, several recording sessions were performed on different days. In a control experiment, an additional AV condition was included in which the auditory target's location was unrelated to the visual target, thus creating an ambiguous bimodal stimulus condition.

In all subjects tested, response accuracy of V saccades was high on average (as quantified by the response gain), but quite variable, and was typically endowed with long latencies. Often, subjects were unable to locate the target, resulting in idiosyncratic patterns of saccadic search sequences. The A-only saccades at the highest S/N ratios were slightly less accurate than V-saccades, but had much shorter latencies. Searching typically involved scanning along the elevation axis after the first saccade. As the S/N ratio of the A-stimuli was lowered, however, response latencies increased, and response accuracy for saccade elevation, and at the lowest S/N ratios also for saccade azimuth, systematically deteriorated. In the AV-condition, consistent auditory-visual interactions appeared in the first-saccade responses, that affected both the latency and the accuracy of the saccadic responses. At all S/N ratios tested, the saccades were as accurate, or better, as the V-only saccades, but at auditory latencies. Strongest effects were obtained for low S/N ratios, for which A-saccades were very inaccurate, but still have much shorter latencies than V-saccades. Search patterns were very rare in the AV condition, indicating that the auditory system helps the visuomotor system to follow the most likely route towards the target. These effects were absent in the AV control condition, and were less prominent in the condition in which the A-stimulus followed or preceded the V-stimulus by 100 ms. This result supports the notion that the AV interactions require spatial-temporal alignment. 


57 Shifts of Intermodal Attention Induced by Endogenous Auditory Cues
Kaiming G. Fu, Micah M. Murray, Beth A. Higgins, D.C. Javitt, C.E. Schroeder and John J. Foxe

The brainÌs ability to selectively switch attention between modalities is necessary for efficient preferential processing of behaviorally relevant stimuli.ð We examined the mechanisms of this switching phenomenon in humans using high-density electrical mapping (64-channel).ð Endogenous auditory cues instructed subjects to perform either an auditory or visual discrimination (1.2s later) upon a compound auditory-visual stimulus.ð We focused on the 1.2s post-cue, pre-stimulus period to define the difference between auditory and visual attentional switch effects.ð Previous studies [1,2], using visual cues, have shown a difference in parietal-occipitalð ~10Hz activity during this timeframe.ð We proposed that this difference reflects a disengaged visual system in preparation for anticipated auditory input which is attentionally more relevant (see Foxe et al., 1998).ð The current study using endogenous auditory cues illustrated a similar effect during the same period.ð When cued to attend to the auditory modality and ignore the visual, the subjects exhibited significantly higher parietal-occipital ~10Hz amplitude, which began 400ms post-cue and lasted until the 2nd stimulus.ð These findings suggest that gating of the visual system occurs through similar mechanisms regardless of the modality of the instructive cue.

References:

[1] Foxe, J.J., Simpson, G.V., and Ahlfors, S.P. Cued shifts of intermodal attention: Parieto-occipital ~10 Hz activity reflects anticipatory state of visual attention mechanisms. Neuroreport, 1998, 9(17), 3929-3933.
[2] Worden, M.S., Foxe, J.J., Wang, N., and Simpson, G.V. Anticipatory biasing of visuospatial attention indexed by retinotopically specific alpha-band EEG increases over occipital cortex. Journal of Neuroscience, 2000, 20: RC63 (1-6).


58 Somato-auditory convergence in auditory association cortex
Taylor A. Johnston, K.G. Fu, T. McGinnis, J. Smiley, John J. Foxe, D.C. Javitt, C.E. Schroeder
Recently-discovered somatosensory input to the caudo-medial (CM) auditory association cortex in macaques raises the possibility of multisensory integration in an area considered to have a unisensory processing function. Continuing investigation, has examined the qualities of the co-represented auditory and somatosensory inputs, as well as the anatomic substrates of somatosensory inputs into the region. Local patterns of transmembrane current flow and action potentials were indexed by sampling of current source density and multiunit activity profiles with linear array multielectrodes. Auditory stimuli were clicks (65db, 100 ms duration), tones and band-passed noise (65db, 100ms duration with 5ms on/off ramps). Somatosensory stimuli were provided by mild electrical stimulation of the median nerve at the wrist, and by light cutaneous stimuli applied to the hand as well as the rest of the body surface. Anatomical tracing utillized several flourescent compounds injected in small amounts through cannulae attached to multielectrodes; this facilitated precise placement of tracers. As reported earlier, characteristic frequency tuning in CM was broad relative to that in A1, and there was a preference for band-passed noise stimulation. Somatosensory receptive fields were large, and often bilateral. Findings thus far predict a complete body surface representation. Tracer experiments indicate several potential sources of the somatosensory input to CM, including Areas 5, 7 and SII/PV.

59 The Ventriloquist illusion in neglect patients
Paul Bertelson, Francesco Pavani, Elisabetta Ladavas, Jean Vroomen and Béatrice de Gelder
Can visual stimuli that go undetected, because they are presented in the extinguished region of neglect patientsà visual field, nevertheless shift in their direction the apparent location of simultaneous sounds (the well-known ventriloquist effect")? This issue was examined using a situation in which each trial involved the simultaneous presentation of a tone over loudspeakers, together with a bright square area on either the left, the right or both sides of  fixation . Participants were required to report the presence of squares, and indicate by hand pointing the apparent location of the tone. Five patients with left hemineglect consistently failed to detect the left square,  either presented alone or together with another square on the right. Nevertheless, on bimodal trials with a single undetected square to the left, their sound localization was significantly shifted in the direction of that undetected square.  By contrast,  on bimodal trials with either a single square on the right or a square on each side, their sound localization showed only small and non-significant shifts. This particular result might be due to a combination of low discrimination of lateral sound deviations with variable individual strategies triggered by conscious detection of the right square. The important finding is the crossmodal bias produced by the undetected left visual distractors. It provides a new example of implicit processing of inputs affected by unilateral visual neglect, and on the other hand is consistent with earlier demonstrations of the automaticity of crossmodal bias.

 



conference sponsor: Unilever
conference contact: John J. Foxe
website: s p r i n g f e l s .com