New Interfaces with Auditory Feedback

Research Overview

With the increasing number and popularity of new motion-sensing hardware, and touch screen mobile devices improving the usability of the new interfaces is becoming more important. However, people do not accept the new interfaces as comfortably as traditional ones because they lack in tactile or haptic feedback that traditional input methods such as mice or keyboards provide. However, having tactile feedback in these interfaces is not possible unless users are wearing a device with actuators. Therefore, auditory feedback is an appropriate and unique alternative for assisting visual feedback.

At MARG, we explored various types of novel auditory feedback methods and explore their effects as secondary feedback on complementing visual feedback. We performed a user study for a menu selection task and experimental results show that the proposed auditory feedback is significantly more efficient and more effective than visual-only feedback or conventional auditory feedback in terms of time and accuracy.

Conventional Auditory Feedback on Menu Selection in Gesture Interface

In most gesture interfaces, auditory feedback is used to indicate the transitions between menu items.
Transitional auditory feedback is usually a short and percussive clicking sound which triggers on the boundary or center of a menu item. This auditory feedback has shown to be helpful for separating items on a circular menu.

Auditory Feedback for Browsing Assistance
::Pre-emptive Auditory Feedback

Weaknesses of the transitional auditory feedback lies on the fact that it provides limited information about the position of the gesture in the circular menu. In gesture interfaces, providing single auditory feedback at the transition between menu items is not sufficient to alert the user about the change of the states. Pre-emptive auditory feedback provides additional auditory feedback just before the transitional auditory feedback which triggers sound at the boundary between menu items. It also informs the user that the transition between menu items is imminent. The pre-emptive auditory feedback has additional auditory feedback ten degrees before the triggering point of the transitional auditory feedback. This additional feedback, supplementing conventional transitional auditory feedback, will aid users in noticing the completion of browsing to the desired menu item.

::Continuous Auditory Feedback

The alternative to generating auditory feedback at specific locations in the circular menu like transitional and pre-emptive auditory feedback, is to provide the user with constant auditory feedback. We propose providing continuous feedback using the Shepard’s tone [10] as a new mean of assisting the browsing gesture of the user. The Shepard’s tone is a sum of continuously overlapping sounds whose fundamental frequencies is moving up or down continuously. Using the Shepard’s tone creates an auditory illusion of infinitely descending or ascending pitches to the listener. The illusion of infinitely descending or ascending pitched sounds semantically fits in with circular menus as they also have infinite browsing structures. Clockwise circular gestures generate auditory feedback using the auditory illusion of infinitely increasing pitched sounds, whereas counter-clockwise gestures act the opposite.

Auditory Feedback for Selection Assistance
::Depth Auditory Feedback

Along with browsing gestures, selection gestures are another key operation in circular menus with hand- gesture interfaces. Conventional auditory feedback for menu selection is provided with discrete and simple sounds, which inform users of the completion of the menu item selection. This feedback does not assist the user while the menu selection is taking place and only gives simple notifications about the completion of selecting an item. When it comes to the touching gesture, finding the location of the touch surface is a difficult task for the user. As for the gesture interface, the distance between the gesture interface operating area and the visual display does not have an intuitive relationship. To provide users with auditory clues on the distance between the hand and touch surfaces, we propose using a continuous depth auditory feedback whose frequency and amplitude increases when the hand approaches the touch surface. Using depth auditory feedback, users can tell the distance between their hands and the touch surface. Knowing the distance between the hand and the touch surface makes operating the menu through touch gestures more intuitive.

The continuous feedback and the depth feedback showed significant improvements in selection times in the user study. Moreover, the depth feedback and pre- emptive auditory feedback showed significant improvements in selection accuracy. Overall, menu selections were faster and more accurate when assisted with the proposed auditory feedback. Especially, the depth auditory feedback showed improvement both in selection time and selection accuracy. Menu selection is only one of many possible applications of hand-gesture interfaces.

Video link:


  • Y Park, J Lee, K Lee, Exploring Effects of Auditory Feedback on Menu Selection in Hand-Gesture Interfaces, submitted to IEEE Multimedia, in press
  • Y Park, H Heo, K Lee, Enhanced auditory feedback for Korean touch screen keyboards, International Journal of Human-Computer Studies, Volume 73, January 2015, Pages 1-11, ISSN 1071-5819. pdf
  • Y. Han, J. Na, and K. Lee, “FutureGrab: A wearable synthesizer using vowel formants,”
    Proceedings of the 2012 Conference on New Interfaces for Musical Expression, Univ. of Michigan (2012) pdf
  • Park, Y., Heo, H., and Lee, K. VOICON: An Interactive Gestural Microphone For Vocal Performance, Proceedings of the 2012 Conference on New Interfaces for Musical Expression, Univ. of Michigan (2012) pdf

Project Members

Jae Hun Kim, Jinsu Na, Yongki Park, Yoonchang Han, Hoon Heo