The Multisensory Film Experience, Experiential Film Aesthetics, Cinema and the Senses

However much the spectator may be engaged by plot or genre, subject matter or thematic implication, the texture of the film experience depends centrally upon the moving images and the sound that accompanies them. The audience gains access to story or theme only through that tissue of the sensory materials. [...] However unaware spectators may be of it, style is working at every moment to shape their experience.

(Bordwell, 1997: 7–8)

Sensations of smell, equilibrium, or touch are, of course, never conveyed in a film through direct stimuli, but are suggested indirectly through sight. [...] [The filmmaker] eliminates entire areas of sensory perception, and thereby brings others into higher relief, ingeniously making them take the place of those that are missing.

(Arnheim, 1933: 34)

Let us assume that, unlike the other types of pictures, film images affect primarily the spectator's senses, engaging him physiologically before he is in a position to respond intellectually.

(Kracauer, 1960: 158)

Although this statement may be an audacious way to open this book, I am convinced that there is no such thing as a purely visual — or purely audiovisual — experience of film. What we see, or what we call seeing, is multisensory in its nature and remains multisensory, even in the final stage of what we consider consciousness: that supposed moment when perception lightens and the marvel of awareness comes about. The perceptions resulting from much of the visual and auditory information in the external sensory world are multisensory. They are not visual, auditory or audiovisual, but multisensory.

When this idea first began to germinate in my mind, I had smaller ambitions and aimed only to show that it is possible to have a multisensory experience through an audiovisual medium such as film and that our brains can perceive an audiovisual medium in a multisensory manner. As my research advanced, though, I realized that not only can our brains perceive an audiovisual medium in a multisensory way, but they must do so because there is no other way for perception to occur. Our natural, not exceptional or synaesthetic, way of perceiving is multisensory. This form of perception remains multisensory when we experience a film, whatever the configuration of the apparatus used to watch the film: a dark theatre room, a cosy living room with a television set or even a small tablet. The natural way for the brain to operate is multisensory, and even if we desired purely visual experiences, we would be frustrated and incapable to force our brains to block out and inhibit some of the neural connections between the senses.

The idea of multisensoriality in film is often based on the intellectual and phenomenal capacity of spectators to make associations creatively through imagination and memory (Marks, 2000, 2002; Sobchack, 1992, 2004). I do not wish to refute such a capacity and possibility, but rather wish to show a level of multisensoriality that is perceptual — not intellectual, imagined or remembered — in principle. This multisensoriality presents itself even before our consciousness has the capacity to make any sense or intellectual and phenomenal associations in a synaesthetic manner. To reiterate Siegfried Kracauer's words cited at the beginning of this chapter, a film engages the spectator "[...] physiologically before he is in a position to respond intellectually" (Kracauer, 1997: 158). In point of fact, a multisensory experience is the natural and common way for all of us to perceptually experience film instead of a mere synaesthetic, exceptional capacity to make high-order, intellectual and phenomenal associations among ideas of a multisensory nature.

My analysis of this issue is a matter of perception, not phenomenology. The multisensory film experience derives from hard-wired, low-level mechanisms of perception that are not within our conscious control. These take place in the milliseconds of the time window of sensory integration, that is, when the senses are integrated. We can divide the levels of our perceptual control of a film through the concepts of autonomic and somatic responses, where somatic response refers to a process that does not require conscious and effortful processing but can nevertheless be inhibited or controlled and autonomic response refers to a process on which we cannot exert control. This layout of a multisensory film experience contradicts the idea of a voyeur spectator who leans back comfortably and visually watches and enjoys in a detached way and in control of his private audiovisual cinematic experience.

Because the primary goal of this book is to support the idea that there are no visual, auditory or audiovisual experiences of film — only a multisensory experience — I am much more concerned with the autonomic level of our film perception than with the somatic level. Furthermore, I aim to add an examination of film aesthetics based on multisensoriality to this perceptual investigation of the senses. The resulting corollary of this combination of perception and aesthetics is materialized by what I specifically call the experiential film aesthetics of Gus Van Sant, Ki-Duk Kim and Knut Erik Jensen (see also Antunes, 2015a, 2015b, 2015c). My analysis of the work of these three directors within a multisensory frame of reference is tantamount to stylistic, narrative and emotional elements that together shape the concrete realm of the multisensory film experience using thermoception (perception of temperature), nociception (perception of pain) and the vestibular sense (perception of orientation and balance) as sense modalities not captured by the discussion of the classic five senses or of the traditional senses associated with film perception and film phenomenology.

The multisensory film experience is the conceptual formulation that evolved from my idea — and the apparent contradiction — that film is an audiovisual medium that is perceptually experienced by spectators in a multisensory fashion. This conceptual distinction between a medium source of stimuli and the resulting perceptual experience is paramount to my claims and aims to shift the burden of proof from having to show that it is possible to perceive an audiovisual medium in a multisensory manner to having to corroborate an orthodox view of film as an audiovisual medium and experience. My proposed shift of the burden of proof invites sceptics to investigate the notion that when we experience a film, we can consciously or unconsciously have exclusively visual or, at most, audiovisual experiences of film, not multisensory experiences, without the influence of the other senses.

In my model of the...