Abstract
Virtual reality has the potential to change the way we create and consume
content in our everyday life. Entertainment, training, design and
manufacturing, communication, or advertising are all applications that already
benefit from this new medium reaching consumer level. VR is inherently
different from traditional media: it offers a more immersive experience, and
has the ability to elicit a sense of presence through the place and
plausibility illusions. It also gives the user unprecedented capabilities to
explore their environment, in contrast with traditional media. In VR, like in
the real world, users integrate the multimodal sensory information they receive
to create a unified perception of the virtual world. Therefore, the sensory
cues that are available in a virtual environment can be leveraged to enhance
the final experience. This may include increasing realism, or the sense of
presence; predicting or guiding the attention of the user through the
experience; or increasing their performance if the experience involves the
completion of certain tasks. In this state-of-the-art report, we survey the
body of work addressing multimodality in virtual reality, its role and benefits
in the final user experience. The works here reviewed thus encompass several
fields of research, including computer graphics, human computer interaction, or
psychology and perception. Additionally, we give an overview of different
applications that leverage multimodal input in areas such as medicine, training
and education, or entertainment; we include works in which the integration of
multiple sensory information yields significant improvements, demonstrating how
multimodality can play a fundamental role in the way VR systems are designed,
and VR experiences created and consumed.
