The multimodal nature of spoken word processing in the visual world: Testing 
the predictions of alternative models of multimodal integration

Smith, Alastair  Charles; Monaghan, Padraic; Huettig, Falk

doi:10.1016/j.jml.2016.08.005

Item

ITEM ACTIONSEXPORT

Add to Basket

Local TagsRelease HistoryDetailsSummary

Released

Journal Article

The multimodal nature of spoken word processing in the visual world: Testing the predictions of alternative models of multimodal integration

MPS-Authors

/persons/resource/persons38015

Smith, Alastair Charles
Psychology of Language Department, MPI for Psycholinguistics, Max Planck Society;

/persons/resource/persons79

Huettig, Falk
Psychology of Language Department, MPI for Psycholinguistics, Max Planck Society;
The Cultural Brain, MPI for Psycholinguistics, Max Planck Society;

External Resource

http://www.sciencedirect.com/science/article/pii/S0749596X16301425
(Supplementary material)

Fulltext (restricted access)

There are currently no full texts shared for your IP range.

Fulltext (public)

Smith_Monaghan_Huettig_JML_2017.pdf
(Publisher version), 2MB

Supplementary Material (public)

There is no public supplementary material available

Citation

Smith, A. C., Monaghan, P., & Huettig, F. (2017). The multimodal nature of spoken word processing in the visual world: Testing the predictions of alternative models of multimodal integration. Journal of Memory and Language, 93, 276-303. doi:10.1016/j.jml.2016.08.005.

Cite as: https://hdl.handle.net/11858/00-001M-0000-002B-3402-0

Abstract

Ambiguity in natural language is ubiquitous, yet spoken communication is effective due to integration of information carried in the speech signal with information available in the surrounding multimodal landscape. Language mediated visual attention requires visual and linguistic information integration and has thus been used to examine properties of the architecture supporting multimodal processing during spoken language comprehension. In this paper we test predictions generated by alternative models of this multimodal system. A model (TRACE) in which multimodal information is combined at the point of the lexical representations of words generated predictions of a stronger effect of phonological rhyme relative to semantic and visual information on gaze behaviour, whereas a model in which sub-lexical information can interact across modalities (MIM) predicted a greater influence of visual and semantic information, compared to phonological rhyme. Two visual world experiments designed to test these predictions offer support for sub-lexical multimodal interaction during online language processing.