Are Audiovisual Correspondences Truly Automatic? The Influence of Top-down Effects

One of the main goals of our perceptual system is to decide when and how to bind information from our different senses to form a single muiltimodal percept. Recently it has been proposed that our knowledge of cross-modal correspondences may be a useful solution to this binding problem (Spence, 2011). If there is a consistent matching between sensory features across modalities, it can guide us to decide when these inputs should be combined. The first goal of my dissertation was to determine the replicability of correspondences between auditory pitch and visual dimensions of size (Size Studies 5-7), height (Height Studies 1a & 1b), spatial frequency (SF Studies 1a & 1b), brightness (Bright Studies 1a & 1b), and sharpness (Sharp Studies 1a & 1b). I failed to replicate seven out of ten correspondences. I conclude that audiovisual correspondences may not be a reliable solution to the binding problem.

Another current debate in visual perception is the extent to which perception is “cognitively impenetrable” to higher-order cognition (Firestone & Scholl, in press). The second goal of my dissertation was to determine whether audiovisual correspondences are subject to top-down cognitive influences. To do so, I asked participants to pair dimensions in a way incongruent with “natural” mappings based on environmental correlations or language knowledge (Size Studies 1-4, Height Studies 2-4, SF Studies 2-3, Bright Study 2, Sharp Study 2). I found that differing instructions changed response speed. I conclude that higher-order cognitive processes do influence basic cross-modal perception.

Together, these results point to the fact that audiovisual correspondences either jointly rely on bottom-up and top-down processing or are solely the result of top-down effects such as task instructions and lexical overlap. My dissertation results therefore strongly question the assumption of automaticity prevalent in the cross-modal correspondence literature. Audiovisual correspondences are a consequence of later decision-level influences rather than being truly automatic perceptual effects.