hide
Free keywords:
-
Abstract:
Introduction:
To form a reliable percept of the multisensory environment, the brain integrates signals across the senses. To estimate for example an object's location from vision and audition, the optimal strategy is to integrate the object's audiovisual signals proportional to their reliability under the assumption that they were caused by a single source (i.e., maximum likelihood estimation, MLE). Behaviorally, it is well-established that humans integrate signals weighted by their reliability in a near-optimal fashion when integrating visual-haptic (Ernst and Banks, 2002) and audiovisual signals (Alais and Burr, 2004). Recently, elegant neurophysiological studies in macaques have shown that single neurons and neuronal populations implement reliability-weighted integration of visual-vestibular signals (Fetsch, et al., 2012; Morgan, et al., 2008). Yet, it is unclear how the human brain accomplishes this feat. Combining psychophysics and multivariate fMRI decoding in a spatial ventriloquist paradigm, we characterized the computational operations underlying audiovisual reliability-weighted integration at several cortical levels along the auditory and visual processing hierarchy.
Methods:
In a spatial ventriloquist paradigm, participants (N = 5) were presented with auditory and visual signals that were independently sampled from four locations along the azimuth (Fig. 1). The signals were presented alone in unisensory conditions or jointly in bisensory conditions. The spatial reliability of the visual signal was high or low. Participants localized either the auditory or the visual spatial signal. The behavioral signal weights were estimated by fitting psychometric functions to participants' localization responses in bisensory conditions without (0°, i.e. congruent) or with a small spatial discrepancy (± 6°). These empirical weights were compared to weights which were predicted according to the MLE model from the signals' sensory reliabilities estimated in unisensory conditions. Similarly, neural signal weights were estimated by fitting 'neurometric' functions to the spatial locations decoded from regional fMRI activation patterns in bisensory conditions and compared to weight predictions from unisensory conditions. For decoding signal locations, a support vector machine was trained on activation patterns from congruent conditions and then generalized to data from discrepant conditions as well as unisensory conditions.
Conclusions:
In summary, the results demonstrate that higher-order multisensory regions perform probabilistic computations such as reliability-weighting. However, despite the small signal discrepancy, the signals were not mandatorily integrated as predicted by the MLE model because task-relevant signals attained larger weights. Thus, probabilistic multisensory computations might involve more complex processes than mandatory reliability-weighted integration, such as inferring whether the signals were caused by a common or independent sources (i.e., causal inference). Only under conditions in which the assumptions of a common source is fostered (e.g., by presenting only correlated signals with a small discrepancy), multisensory signals might be fully integrated weighted by their reliability.
