How an audio-visual instrument can foster the sonic experience.

Abstract

The chapter formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output. Keywords: NIME, Audio-visual relationship, Musical expression, Perception, Multisensory integration