Musical Haptics

The dynamical response of a musical instrument plays a vital role in determining its playability. This is because, for instruments where there is a physical coupling between the sound-producing mechanism of the instrument and the player’s body (as with any acoustic instrument), energy can be exchanged across points of contact. Most instruments are strong enough to push back; they are springy, have inertia, and store and release energy on a scale that is appropriate and well matched to the player’s body. Haptic receptors embedded in skin, muscles, and joints are stimulated to relay force and motion signals to the player. We propose that the performer-instrument interaction is, in practice, a dynamic coupling between a mechanical system and a biomechanical instrumentalist. We take a stand on what is actually under the control of the musician, claiming it is not the instrument that is played, but the dynamic system formed by the instrument coupled to the musician’s body. In this chapter, we suggest that the robustness, immediacy, and potential for virtuosity associated with acoustic instrument performance are derived, in no small measure, from the fact that such interactions engage both the active and passive elements of the sensorimotor system and from the musician’s ability to learn to control and manage the dynamics of this coupled system. This, we suggest, is very different from an interaction with an instrument whose interface only supports information exchange. Finally, we suggest that a musical instrument interface that incorporates dynamic coupling likely supports the development of higher levels of skill and musical expressiveness.

This chapter provides an overview of the human somatosensory system. It is the system that subserves our sense of touch, which is so essential to our awareness of the world and of our own bodies. Without it, we could not hold and manipulate objects dextrously and securely, let alone musical instruments, and we would not have a body that belongs to us. Tactile sensations, conscious or unconscious, arise from the contact of our skin with objects. It follows that the mechanics of the skin and of the hand its interaction with objects is the source of information that our brain uses to dextrously manipulate objects, as in music playing. This information is collected by vast array of mechanoreceptors that are sensitive to the effects of contacting objects, often with the fingers, even far away for the region of contact. This information is processed by neural circuits in numerous regions of the brain to provide us with extraordinary cognitive and manipulative functions that depend so fundamentally on somatosensation.

We suggest that studies on active touch psychophysics are needed to inform the design of haptic musical interfaces and better understand the relevance of haptic cues in musical performance. Following a review of the previous literature on vibrotactile perception in musical performance, two recent experiments are reported. The first experiment investigated how active finger-pressing forces affect vibration perception, finding significant effects of vibration type and force level on perceptual thresholds. Moreover, the measured thresholds were considerably lower than those reported in the literature, possibly due to the concurrent effect of large (unconstrained) finger contact areas, active pressing forces, and long-duration stimuli. The second experiment assessed the validity of these findings in a real musical context by studying the detection of vibrotactile cues at the keyboard of a grand and an upright piano. Sensitivity to key vibrations in fact not only was highest at the lower octaves and gradually decreased toward higher pitches; it was also significant for stimuli having spectral peaks of acceleration similar to those of the first experiment, i.e., below the standard sensitivity thresholds measured for sinusoidal vibrations under passive touch conditions.

We draw from recent research in violin quality evaluation and piano performance to examine whether the vibrotactile sensation felt when playing a musical instrument can have a perceptual effect on its judged quality from the perspective of the musician. Because of their respective sound production mechanisms, the violin and the piano offer unique example cases and diverse scenarios to study tactile aspects of musical interaction. Both violinists and pianists experience rich haptic feedback, but the former experience vibrations at more bodily parts than the latter. We observe that the vibrotactile component of the haptic feedback during playing, both for the violin and the piano, provides an important part of the integrated sensory information that the musician experiences when interacting with the instrument. In particular, the most recent studies illustrate that vibrations felt at the fingertips (left hand only for the violinist) can lead to an increase in perceived sound loudness and richness, suggesting the potential for more research in this direction.

An experiment is presented that measured aspects of functionality, usability and user experience for four distinct types of device feedback. The goal was to analyse the role of haptic feedback in functional digital musical instrument (DMI) interactions. Quantitative and qualitative human–computer interaction analysis techniques were applied in the assessment of prototype DMIs that displayed unique elements of haptic feedback; specifically, full haptic (constant-force and vibrotactile) feedback, constant-force only, vibrotactile only and no feedback. From the analysis, data are presented that comprehensively quantify the effects of feedback in haptic interactions with DMI devices. The investigation revealed that the various types of haptic feedback applied had no significant functional effect upon device performance in pitch selection tasks; however, a number of significant effects were found upon the users’ perception of usability and their experiences with each of the different feedback types.

We listen to music not only with our ears. The whole body is present in a concert hall, during a rock event, or while enjoying music reproduction at home. This chapter discusses the influence of audio-induced vibrations at the skin on musical experience. To this end, sound and body vibrations were controlled separately in several psychophysical experiments. The multimodal perception of the resulting concert quality is evaluated, and the effect of frequency, intensity, and temporal variation of the vibration signal is discussed. It is shown that vibrations play a significant role in the perception of music. Amplifying certain vibrations in a concert venue or music reproduction system can improve the music experience. Knowledge about the psychophysical similarities and differences of the auditory and tactile modality help to develop perceptually optimized algorithms to generate music-related vibrations. These vibrations can be reproduced, e.g., using electrodynamic exciters mounted to the floor or seat. It is discussed that frequency shifting and intensity compression are important approaches for vibration generation.

This chapter presents recent work concerning physically modelled virtual musical instruments and force feedback. Firstly, we discuss fundamental differences in the gesture–sound relationship between acoustic instruments and digital musical instruments, the former being linked by dynamic physical coupling, the latter by transmission and processing of information and control signals. We then present an approach that allows experiencing physical coupling with virtual instruments, using the CORDIS-ANIMA physical modelling formalism, synchronous computation and force-feedback devices. To this end, we introduce a framework for the creation and manipulation of multisensory virtual instruments, called the MSCI platform. In particular, we elaborate on the cohabitation, within a single physical model, of sections simulated at different rates. Finally, we discuss the relevance of creating virtual musical instruments in this manner, and we consider their use in live performance.

Digital musical instruments yielding force feedback were designed and employed in a case study with the Laptop Orchestra of Louisiana. The advantages of force feedback are illuminated through the creation of a series of musical compositions. Based on these and a small number of other prior music compositions, the following compositional approaches are recommended: providing performers with precise, physically intuitive, and reconfigurable controls, using traditional controls alongside force-feedback controls as appropriate, and designing timbres that sound uncannily familiar but are nonetheless novel. Video-recorded performances illustrate these approaches, which are discussed by the composers.

Haptics, and specifically vibrotactile-augmented interfaces, have been the object of much research in the music technology domain: In the last few decades, many musical haptic interfaces have been designed and used to teach, perform, and compose music. The investigation of the design of meaningful ways to convey musical information via the sense of touch is a paramount step toward achieving truly transparent haptic-augmented interfaces for music performance and practice, and in this chapter we present our recent work in this context. We start by defining a model for haptic-augmented interfaces for music, and a taxonomy of vibrotactile feedback and stimulation, which we use to categorize a brief literature review on the topic. We then present the design and evaluation of a haptic language of cues in the form of tactile icons delivered via vibrotactile-equipped wearable garments. This language constitutes the base of a “wearable score” used in music performance and practice. We provide design guidelines for our tactile icons and user-based evaluations to assess their effectiveness in delivering musical information and report on the system’s implementation in a live musical performance.

This chapter considers the use of haptics for learning fundamental rhythm skills, including skills that depend on multi-limb coordination. Different sensory modalities have different strengths and weaknesses for the development of skills related to rhythm. For example, vision has low temporal resolution and performs poorly for tracking rhythms in real time, whereas hearing is highly accurate. However, in the case of multi-limbed rhythms, neither hearing nor sight is particularly well suited to communicating exactly which limb does what and when, or how the limbs coordinate. By contrast, haptics can work especially well in this area, by applying haptic signals independently to each limb. We review relevant theories, including embodied interaction and biological entrainment. We present a range of applications of the Haptic Bracelets, which are computer-controlled wireless vibrotactile devices, one attached to each wrist and ankle. Haptic pulses are used to guide users in playing rhythmic patterns that require multi-limb coordination. One immediate aim of the system is to support the development of practical rhythm skills and multi-limb coordination. A longer-term goal is to aid the development of a wider range of fundamental rhythm skills including recognising, identifying, memorising, retaining, analysing, reproducing, coordinating, modifying and creating rhythms—particularly multi-stream (i.e. polyphonic) rhythmic sequences. Empirical results are presented. We reflect on related work and discuss design issues for using haptics to support rhythm skills. Skills of this kind are essential not just to drummers and percussionists but also to keyboards’ players and more generally to all musicians who need a firm grasp of rhythm.

Touch-sensitive interfaces are more and more used for music production. Virtual musical instruments, such as virtual pianos or drum sets, can be played on mobile devices like phones. Audio tracks can be mixed using a touchscreen in a DJ set-up. Samplers, sequencers or drum machines can be implemented on tablets for use in live performances. The main drawback of traditional touch-sensitive surfaces is the missing haptic feedback. This chapter discusses if adding specifically designed vibrations helps improve the user interaction with touchscreens. An audio mixing application for touchscreens is used to investigate if tactile information is useful for interaction with virtual musical instruments and percussive loops. Additionally, the interaction of auditory and tactile perception is evaluated. The effect of loudness on haptic feedback is discussed using the example of touch-based musical interaction.

While a standard approach is more or less established for rendering basic vibratory cues in consumer electronics, the implementation of advanced vibrotactile feedback still requires designers and engineers to solve a number of technical issues. Several off-the-shelf vibration actuators are currently available, having different characteristics and limitations that should be considered in the design process. We suggest an iterative approach to design in which vibrotactile interfaces are validated by testing their accuracy in rendering vibratory cues and in measuring input gestures. Several examples of prototype interfaces yielding audio-haptic feedback are described, ranging from open-ended devices to musical interfaces, addressing their design and the characterization of their vibratory output.