Item

Abstract

To interact effectively with our environment, the human
brain integrates information from multiple senses into a
coherent percept. Neurophysiological and functional imaging
studies have revealed multi-sensory interactions in a
widespread neural system encompassing subcortical structures,
putative ’unisensory’ and higher order association cortices.
Combining fMRI and psychophysics, we investigated
where and how different types of sensory features are combined
within the cortical hierarchy. We presented subjects
with object pictures and sounds while factorially manipulating
the relative informativeness of the auditory and visual
modalities. While low level spatio-temporal interactions
were found within Heschl’s gyrus, higher order object
features were integrated within the superior temporal sulci
(STS) bilaterally. Consistent with the law of inverse effectiveness,
the multisensory interactions in STS were primarily
suppressive for intact, but (super)additive for degraded
stimuli. These distinct modes paralleled behavioral indices
of multi-sensory enhancement showing the greatest multisensory
benefit for degraded stimuli. In conclusion, the human
brain integrates information that is abstracted from its
sensory inputs atmultiple levels of the cortical hierarchy. The
operational mode of audio-visual integration is dictated by the informativeness of the auditory and visual modalities.