Designing user interfaces with gestures and sound.

A scholarly paper about the benefits of the multimodal feedback combined with non-speech representations of content via sound. The focus is on the benefits of the multimodal feedback combined with non-speech representations of content via sound ...

"This paper explores the use of sound and gestures within a tangible user interface, towards influencing the performance and appeal of browsing through a voice mail collection. Three browsers were developed, in which speech and non-speech sounds were projected in a sound-scape above a tangible interface. For each of the browsers, users were given the task of locating voice mail messages. Search behavior in terms of gestures, task time, and errors, were recorded and categorized. Users’ experience was measured in terms of appeal, consisting of hedonic and ergonomic qualities. Results showed that users’ explorative behavior changed, depending on the specific type of browser. Subjects were observed to utilize non-speech sound cues effectively in searching for target voice mail messages. Subjective results showed that ergonomic and hedonic qualities of appeal were found to interact in an averaging way, such that overall appeal was similar across conditions. Implications towards designing multimodal interfaces are discussed."   continued ...   (Via The Journal of Design Research)