10. Design of Vibrotactile Feedback and Stimulation for Music Performance

Passive kinaesthetic feedback and stimulation are inherent to the physical characteristics of the controller, and do not require any externally synthesized signal.

O’Modhrain and Essl developed three DMIs that implement passive kinaesthetic feedback. The Pebble Box and the Crumble Bag [41] were used to control an event-based granular synthesizer: the Pebble Box consists of a box filled with different-sized pebble stones and a microphone that picks up the noise produced by the collisions between pebbles. The kinaesthetic feedback offered by the interface comes from the physical properties of the pebbles themselves, and the impact sounds act as triggering events on the granular synthesizer. The Crumble Bag follows the same patter, and it is aimed to take advantage of natural “grabbing gestures.” A fabric bag is filled with different materials that provide haptic feedback, and a small microphone in the bag provides the necessary event triggers to the algorithm. The Scrubber [14] also implemented the same approach: an eraser embedded with a force sensor and two microphones were used to control the synthesis of friction sounds, synthesized by means of granular or wavetable synthesis. The haptic feedback again was directly issued by the manipulation of the device dragged along a surface.

Sinyor and Wanderley [47] developed the Gyroyre, a handheld controller based on a spinning wheel, in which the kinaesthetic feedback comes directly from the dynamic properties of the system. The mapping and synthesis algorithm are designed to take advantage of the haptic feedback, and the interface can be used for different musical applications, sequencing or modifying effects’ parameters.

Active kinaesthetic feedback is the response of the controller to the user’s actions, usually by means of synthesized signals supplied into motors or actuators, which stimulate kinaesthetic receptors. This is most commonly referred to as force feedback.

The earliest example of a force-feedback device specifically developed for musical applications is probably the Transducteur Gestuel Rétroactif (TGR) developed at ACROE, whose development is described in Sect. 8.​3. This device was recently used by Sinclair et al. [46] to investigate velocity estimation methods in the haptic rendering of a bowed string.

Another classical example is the Moose, developed by O’Modhrain and Gillespie [42], consisting of a plastic puck that the user can manipulate in a 2D space, which is attached to flexible metal bars, connected to linear motors. Two encoders sense the movements of the puck, and the motors provide the correspondent force feedback. The device was used in a bowing test, using a virtual string programmed in Synthesis ToolKit (STK) [10], where the presence of friction between the bow and the string was simulated using the haptic device.

The vBow by Nichols [40] is a violin-like controller that uses a series of servomotors and encoders to sense the movement of a rod, acting as the bow, connected to a metallic cable. In its last incarnation, the vBow is capable of sensing moment in 4-DoF and producing haptic feedback accordingly.

More recently, Berdahl and Kontogeorgakopoulos [2] developed the FireFader, a motorized faders using sensors and DC motors to introduce musicians to haptic controllers. Both the software and hardware used for the project are open-source, allowing musicians to customize the mapping of the interface to their specific needs. Applications of the device are described in Chap. 9.

Passive tactile is a form of primary feedback, which leverages the use of different types of materials in a controller for musical expression. The properties of these materials (e.g., stiffness, flexibility) can affect the ergonomics of the instrument and its feel in the user’s hands.

As an example, the Meta-Instrument [11] has the form of a partial exoskeleton embedded with buttons that the performer uses to trigger samples and events in the sound; the performer’s gestures are captured via sensors in the arms and mapped to various effects. The buttons embedded in the controller are covered in a layer of foam, providing the user with immediate passive feedback about the level of pressure applied.

Active tactile feedback and stimulation are the main focus of this chapter, and for this reason we provide a more in-depth analysis of the related literature, as well as an updated taxonomy, based on Giordano and Wanderley [19], which will help categorize examples in this field.

We propose a classification identifying in active tactile feedback and stimulation three different categories according to the function that the tactile effects have in the interface design: tactile notification, tactile translation, and tactile languages.

Tactile Notification

The most straightforward application of tactile stimulation is intended for notifying the users about events taking place in the surrounding environment or about results of their interaction with a system. The effects designed for this kind of applications can be as simple as single, supra-threshold stimuli² aimed at directing users’ attention, but they can also be more complex, implementing temporal envelopes and/or spatial patterns.

Michailidis and Berweck [37] and Michailidis and Bullock [38] have explored solutions to provide haptic feedback in live-electronics performance. The authors developed the Tactile Feedback Tool, a general-purpose interface using small vibrating motors embedded in a glove. The interface gave musicians information about the successful triggering of effects in a live-electronics performance, using an augmented trumpet or a foot pedal switch. This device leverages the capacity of the tactile sense to attract users’ attention, while not requiring them to lose focus on other modalities, which would have been the case with the use of onstage visual displays.

Van der Linden et al. [49] implemented a whole-body general-purpose vibrotactile device. The authors used a motion capture system and a suit embedded with vibrating motors distributed over the body to enhance the learning process of bowing for novice violin players. A set of ideal bowing trajectories was computed using the motion capture system; when practicing, the players’ postures would be compared in real time with the predefined ideal trajectories. If the distance between any two corresponding points in the two trajectories exceeded the threshold value, the motor spatially closer to that point would vibrate, notifying the users to correct their posture. The authors conducted a study in which several players used the suit during their violin lessons. Results showed an improved coordination of the bowing arm, and participants reported an enhancement in their body awareness produced by the feedback.

A similar solution was developed by Grosshauser and Hermann [21], which used a vibrating actuator embedded in a violin bow to correct hand posture. Using accelerometers and gyroscopes, the position of the bow could be compared in real time to a given trajectory, and the tactile feedback would automatically activate to notify the users about their wrong posture.

Tactile Notification

With tactile translation, we refer to two separate classes of applications: One class implements sensory substitution techniques to convey to the sense of touch stimuli which would normally be addressed to other modalities; the other class simulates the haptic behavior of other structures whose vibrational properties have previously been characterized.

The field of sensory substitution has been thoroughly investigated since the beginning of the last century. In 1930, von Békésy started investigating the physiology behind tactile perception by drawing a parallel between the tactile and the auditory channels in terms of the mechanism governing the two perception mechanisms [53]. A thorough review of sensory substitution applications can be found in Visell [52]. In a musical context, several interfaces have been produced with the aim of translating sound into perceivable vibrations delivered via vibrotactile displays. Crossmodal mapping techniques can be utilized to perform the translation, identifying sound descriptors to be mapped to properties of vibrotactile feedback.

Karam et al. [27] developed a general-purpose interface in the form of an augmented chair (the Emoti-Chair) embedded with an array of eight speakers disposed along the back. The authors’ aim was to create a display for deaf people to enjoy music through vibrations. They developed the Model Human Cochlea [26]—a sensory substitution model of the cochlear critical band filter on the back—and mapped different frequency bands of a musical track, rescaled to fit into the frequency range of sensitivity of the skin (see Sect. 4.​2), to each of the speakers on the chair. In a related study, Egloff et al. [12] investigated people’s ability to differentiate between musical intervals delivered via the haptic channel, finding that on the average smallest perceptible difference was a major second (i.e., two semitones). It was also noted that results vary widely due to the sensitivity levels of different receptive fields across the human body. Thus, care must be taken when designing vibrotactile interfaces intended to be used as a means for sensory substitution.

Merchel et al. [36] developed a prototype mixer equipped with a tactile translation system to be used by sound recording technicians. A mixer augmented with an actuator would allow the user to recognize the instrument playing in the selected track only by means of tactile stimulation: A tactile preview mode would be enabled on the mixer, performing a real-time translation of the incoming audio. Preliminary results show that users were able to recognize different instruments only via the sense of touch; better performance was obtained for instruments producing very low-frequency vibrations (bass) or strong rhythmical patterns (drums). A similar touch screen-based system and related test applications are described in Chap. 12.

Tactile Stimulation

In tactile stimulation applications, the vibrational behavior of a vibrating structure is characterized and modeled so as to be able to reproduce it in another interface. Examples in this category include physical modeling of the vibrating behavior of a musical instrument, displayed by means of actuators.

A DMI featuring tactile stimulation capability is the Viblotar by Marshall [35]. The instrument is composed of a long, narrow wooden box equipped with sensors and embedded speakers. Sound is generated from a hybrid physical model of an electric guitar and a flute programmed in the Max/MSP environment. During performance, the instrument rests on the performer’s lap or on a stand. One hand manipulates a long linear position sensor and matching force sensitive resistor (FSR) underneath to “pluck” a virtual string. The location, force, and speed of the motion are mapped to frequency, amplitude, and timbre parameters of the physical model. The other hand operates two small FSRs which control pitch bend up and down. The sound output from the Viblotar can be redirected to external speakers, hence allowing the embedded speakers to function primarily for generating vibrotactile feedback instead of sound output. In this configuration, the sound output is split, with one signal sent directly to the external speakers and another routed through a signal processing module that can produce a variety of customized vibrotactile effects such as compensating for frequency response of loudspeakers, simulating the frequency response of another instrument or amplifying the frequency band to which the skin is most sensitive [34].

Tactile Languages

Tactile languages are an attempt to create compositional languages solely addressed to the sense of touch, in which tactile effects are not just simple notifications, issued from the interaction with a system, but can be units or icons for abstract communication mediated by the skin.

An early example of tactile language is the “vibratese,” proposed by Geldard [16], who aimed at creating a complete new form of tactile communication delivered by voice coil actuators (see Sect. 13.​2). Parameters for defining building blocks for the language would be elements such as frequency, intensity, and waveform. A total of 45 unit blocks representing numbers and letters of the English alphabet were produced, allowing for expert users to read at rates up to 60 words per minute.

More recently, much research on tactile languages has been directed toward the development of tactile icons. Brewster and Brown [6] introduced the notion of tactons, i.e., tactile icons to be used to convey non-visual information by means of abstract or meaningful associations, which have been used to convey information about interaction with mobile phones [8]. Enriquez and MacLean [13] studied the learnability of tactile icons delivered to the fingertips by means of voice coil-like actuators. By modulating frequency, amplitude and rhythm of the vibration, they produced a set of 20 icons, which were tested in a user-based study organized in two sessions, two-weeks apart. Participants recognition rates reached 80% in the first session after 10 min of familiarization with the system and more than 90% during the second session.

In a musical context, attempts to create compositional languages for the sense of touch can be found in the literature. Gunther [22] developed the Skinscape system, a tactile compositional language whose building blocks varied in frequency, intensity, envelope, spectral content of vibrations, and spatial position on the body of the user. The language was at the base of the Cutaneous Grooves project by Gunther and O’Modhrain [23], in which it was used to compose a musical piece to be accompanied by vibrations delivered by a custom-built set of suits embedded with various kinds of actuators.

In terms of tactons, we are not aware of any study evaluating their effectiveness in the context of music performance and practice. This is the object of the remainder of this chapter, where we present the design and evaluation of tactile icons for expert musicians.

We identified three different dimensions defining the tacton design space associated with the six-actuator belt:

For the design of the symbolic tactons, we applied a heuristic approach: We defined several geometric patterns which we hypothesized would feature unique characteristics, making them easily distinguishable from one another; we then implemented these patterns, together with preliminary global and individual temporal properties, on a Max-based tactile sequencer we programmed to control the belt; a music pedagogy doctoral researcher (Audrey-Kristel Barbeau) would then test the icons and provide immediate feedback to allow us to proceed to another iteration of the design process.

This process lasted over several weeks, after which we finalized a set of ten tactons, depicted in Fig. 10.3. Each of the tactile icons consists of two repetitions of the same pattern which are separated by a fixed time interval. The tactons have a total duration which varies from 1.5 to 2.7 s. For the individual temporal properties, we chose a fixed envelope for all the actuations which features 50 ms of attack, 150 ms of sustain at maximum intensity, and no release time (see Fig. 10.4). We decided to keep the vibrotactile envelope parameters fixed for this initial phase of the project to facilitate the tactons’ learning phase. These tactile icons were proposed to undergraduate music students—a saxophone player (performer 1) and a guitar player (performer 2)—who were the participants for the ensuing evaluation sessions.

The symbolic tactons we designed for the belt do not carry any musical or other meaning per se, and need to be learned by the performers to be proficiently used to convey musical information. These icons can be mapped to several musical functions, such as chords or sections of a piece, and these mappings also need to be mastered by musicians to be correctly interpreted.

While the symbolic tactons were designed by first creating geometric and temporal patterns for the vibrotactile stimuli which could later be mapped arbitrarily to musical functions, design of musical tactons for the single-actuator band took the opposite approach. For these, we started by determining the set of musical information this actuator would deliver. From experiences we gathered in our previous work [15], we hypothesized that a single-actuator configuration could be used to provide tempo cues, as well as information about articulation and dynamics.

Using the heuristic approach based on iterative feedback from A.-K. Barbeau, we designed a set of four musical tactons associated with crescendo, decrescendo, staccato, and legato, respectively, which are shown in Fig. 10.5. These tactons contained a musical meaning attached to the temporal properties of the tacton itself and would ideally require a minimal effort to be correctly interpreted.

We conducted a preliminary evaluation of both symbolic and musical tactons’ design with our two musicians, who performed a series of musical tasks we associated with each of the icons. It was important for us to evaluate the learnability and recognition rate of the tactons in the context of music performance in order to establish if musicians actively engaged in a musical task could reliably recognize and respond to the given tactile icons.

We performed two testing sessions, two weeks apart, following a methodology similar to the one reported in [13]. The musicians had 20 min per session to familiarize themselves with the tactons. Subsequently, they were asked to perform two recognition tasks. In task 1, they experienced a series of tactons and verbally reported the name or number of the tacton they thought they had perceived. In task 2, the musicians were given a score, shown in Fig. 10.6, and asked to perform the melody associated with the perceived icons. The melodies were composed to be easy to sight-read and perform. In the first session only symbolic tactons were tested, while in the second session we tested both symbolic and musical tactons. Performances were audio-recorded and subsequently analyzed to determine recognition rates of the tactile icons in both sessions.

Session 1

Two repetitions of task 1 were performed 10 min apart. The results are depicted in Fig. 10.7a and show the average recognition rate of twenty randomly ordered tactons for each of the two repetitions. For the first trial, the two musicians correctly identified 86 and 77% of the tactons, respectively. In the second repetition, both performers achieved 88%.

For task 2, we provided the musicians with the score shown in Fig. 10.6. This time we asked them to play the melody corresponding to the perceived tactile icon. The musicians were free to play at the tempo they desired. Fifteen randomly ordered icons were tested, and a new icon would be delivered via the belt while the musician was playing the half note ending the previous melody. Task 2 was repeated three times, 10 min apart, and the results are depicted in Fig. 10.7b. The performers reached, respectively, a 92 and 79% recognition rate for the first trial, 92 and 86% for the second trial, and 88 and 71% for the last trial. It is notable that the results declined for both performers in the third trial, factors for which we discuss in Sect. 10.3.2.4.

Session 2

A second session took place two weeks after session 1, testing both symbolic and musical tactons. Following the previously described protocol, we performed task 1 first, whose results are depicted in Fig. 10.8a.

For task 2, the musicians wore the belt and the single-actuator elastic band on their left upper arm. A symbolic icon would be delivered via the belt, followed by a musical icon from the single actuator. The musicians were asked to play the corresponding melody following either the articulation or the dynamics indicated by the musical tacton. Results are shown in Fig. 10.8b. For the symbolic icons, the first performer reached a recognition rate of 87% in the first trial, 86% in the second, and 70 and 78% in the third and fourth, respectively. A similar trend can be observed for the musical icons, with a 100% recognition rate in the first repetition, followed by 92, 82, and 88% in the last three trials. The second musician performed less well in this task, reaching a 78% recognition rate for symbolic tactile icons in trial one, 71% for trial two, and 76 and 77% for trials three and four, respectively. For the musical tactons, only 25% of the tactile icons were correctly recognized in trial one, 66% in trial two, and 77 and 57% in trials three and four, respectively.

The two testing sessions with the undergraduate musicians show several patterns: Performers’ recognition rate in both sessions was consistently over 80% for task 1, even after only 20 min of practice with the belt (consistent with findings in Enriquez and MacLean [13]). This suggests that for both the musical and the symbolic tactons, we were able to design learnable and distinguishable tactile icons.

When looking at the data for task 2, in both sessions we can observe important differences between the two performers. Performer 1 consistently achieved better results than performer 2, who afterward reported that the task could become quickly overwhelming, especially in the second session. This suggests that the complexity of the task prevented performer 2 from simultaneously paying attention to both types of tactile icons while reading and playing the melodies on the instrument. Performer 2’s performance nonetheless improved over time, as visible in Fig. 10.8b, going from a 25% recognition rate for the musical icons in trial one to almost 80% in trial three.

Participant 1 scored above 80% in most of the tasks across the two sessions, and two trends can be identified: For both sessions, performer 1’s performance in the musical task decreased in trial three, compared to the first two trials. This might be due to the presence of adaptation effects which would decrease the sensitivity to the tactile icons. The musician stated that the tasks were not too demanding and that the icon design allowed to easily differentiate the tactile effects.

Overall, the variation between the two participants could be caused by different levels of proficiency on their instrument and ability to sight-read, despite their similar self-assessed musical expertise: Participant 1 was very confident in the sight-reading and performance of the melodies we proposed, while for participant 2 this task proved to be quite demanding, as demonstrated by the frequent hesitation in performing the given melodies which can be heard in the audio recording of the testing sessions. The different postures adopted by the two musicians when playing the saxophone and the guitar, respectively, could also be partly responsible for the variation between the two participants, but this aspect would require an investigation conducted on a larger group of musicians. Additionally, the limited number of repetitions and subjects makes it difficult to draw definitive conclusions about significant trends over repetitions, as randomness may have had an impact on the results.

The observations reported above indicate that a satisfying degree of tactile icon recognition can be reached for both musical and symbolic tactons during the performance of a musical task, provided a high degree of confidence and expertise on the performer’s side. While all the musical tactons were equally well recognized during the two testing session, symbolic tactile icons 5 and 6 were the most problematic ones in terms of recognition rates. Tacton 5 would often be confused with tacton 9 since, as reported by performer 1, the vibration coming from the two actuators on the sides would sometimes go unnoticed. This could be due to lower skin sensitivity in the waist area, which, combined with its peculiar geometrical pattern, made tacton 6 also difficult to recognize at times.

Ultimately, our results confirm that the transparency of a tacton [32] is not an absolute property of the tactile icon itself, but is very much influenced by the global context in which tactile information is being transmitted to users and to their available cognitive resources [44].