8. The MSCI Platform: A Framework for the Design and Simulation of Multisensory Virtual Musical Instruments

The distributed (or multi-model) approach to haptic DMIs follows the classic decomposition into gestural controller and sound synthesis sections [33]. The haptic, aural and sometimes visual stimuli are physically decoupled from each other, due to the distributed architecture of the instrument (see Fig. 8.1). Haptic feedback incorporated into the gestural controller enables coupling with certain components of the DMI, for instance, by programming the mechanical behaviour of the gestural control section using a local haptic model. Data extracted from the interaction between the user and this model can then be mapped to chosen sound synthesis parameters.

Some examples of this approach are the Virtual Piano Action by Gillespie [15], or the DIMPLE software [30] in which the user interacts with a rigid dynamics model, and information concerning this interaction (positions, collisions, etc.) is then mapped to an arbitrary sound synthesis process, possibly a physically based simulation.

Vibrotactile feedback inferred from the sound synthesis process itself can be provided to the user by integrating vibration actuators into the gestural controller. Such is the case of Nichols’ vBow friction-driven haptic interface [26] or Marshall’s vibrotactile feedback incorporated directly into DMI controllers [25].

Technical implementations of these systems generally rely on asynchronous computation loops for haptics and sound, employing low- to mid-priced haptic devices such as the Phantom Omni or the Novint Falcon. While these systems tend to bridge the gap between gestural control section and sound synthesis, the sound is still driven by mapping of sensor data, and the user physically interacts only with a local subsection of the instrument.

An alternative approach to implementing haptic DMIs is to model the virtual instrument as a single multisensory physical object that jointly bears mechanical, acoustical and possibly visual properties, inherent to its physical nature. Physical modelling techniques are then the only viable approach. As a result, the gestural controller and sound synthesis sections are tightly interconnected: haptic interaction with one part of the instrument will affect it as a whole, and the player is haptically coupled with a complete single model (see Fig. 8.2 and Chap. 2).

Making use of this approach, one can distinguish:

MSCI fits into the latter category. The platform provides a musician-friendly physical modelling environment in which users can design virtual musical instruments, and allows unified multisensory interaction by simulating those instruments on a dedicated workstation that supplies coherent aural, visual and haptic feedback.

The transducteur gestuel rétroactif (TGR) is a force-feedback device designed by the ACROE-ICA laboratory (Fig. 8.3). The first prototype was proposed by Florens in 1978 [12], conceived specifically for the requirements of artistic creation, in particular for instrumental arts such as music. The first goal of the TGR is to render the dynamic qualities of mechanical interactions with simulated objects with the highest possible fidelity: to this end, it offers both a high mechanical bandwidth (up to 15 kHz) and high peak force feedback (up to 200 N per degree of freedom).

Several slices (1-DoF modular electromechanical systems comprised of a sensor and an actuator) can be combined allowing for any number of force-feedback-enabled degrees of freedom [14]. The device employed in the MSCI platform gathers 12 independent modules that can be combined with various mechanical end-effectors, forming 1D, 2D, 3D or even 6D morphological configurations, adapted to the diverse nature of instrumental gestures such as striking, bowing, plucking, grasping.

CORDIS-ANIMA [5] is a modular formalism for modelling and simulating mass-interaction networks—that is physical models described by Newtonian point-based mechanics. It defines two main module types:

CORDIS-ANIMA incorporates the notion of physical coupling between networks of elementary modules through the interdependence of two dual variables: position, an extensive variable that gives <MAT> modules a position in space, and force, an intensive variable that originates from interactions between <MAT> modules described by <LIA> modules. Computing the network requires a closed-loop calculation: first, of the new positions of <MAT> modules, and second, of all the forces produced by the <LIA> modules according to the new positions of the <MAT> modules that they are connected to (Fig. 8.4).

Several CORDIS-ANIMA implementations are declined for different geometrical spaces: 1D with scalar distances, or 1D, 2D and 3D with Euclidian distances. The 1D scalar distance version is generally used to simulate vibroacoustic deformations in which all <MAT> modules move along a single scalar axis. Models built in this way are topological networks that may represent a first-order approach to vibratory deformations as found in musical instruments—a simplification that works well in most cases.

For sound-producing physical models, networks must be simulated at audio-rate frequency (generally set at 44.1 kHz) in order to faithfully represent acoustical deformations that occur in the audible range (up to 20 kHz). Non-vibrating models, designed to, e.g. produce visual motion or mechanical systems, are often simulated in 1D, 2D or 3D geometrical spaces and at lower frequencies in the range 1–10 kHz, a bandwidth suited to instrumental performance.

The TGR haptic device is represented in CORDIS-ANIMA as a <MAT> module: this reports positions taken from its sensors and receives forces from the connected <LIA> modules which are then sent to the TGR’s actuators.

GENESIS [9] is ACROE-ICA’s modelling and simulation software for musical creation. It allows to model vibrating objects—from elementary oscillators to complex musical scenes—and to simulate them off-line at 44.1 kHz. GENESIS implements the 1D version of CORDIS-ANIMA, meaning that all <MAT> physical modules move along a single scalar axis conventionally labelled x.

The modelling interface consists in a workbench representing the y-z plane, where <MAT> modules can be placed and interconnected through <LIA> modules to form topological networks (Figs. 8.5 and 8.6). Modules are given physical parameters that dictate their physical behaviour and initial conditions (initial position and speed of <MAT> modules).

The vast majority of available haptic devices communicate asynchronously with physical simulations [11, 30]. Generally, the haptic loop runs locally at approximately 1 kHz, whereas other model components are computed with a lower rate and low demanding latency constraints, following a distributed approach. Current general-purpose computer architectures are perfectly suited for these applications. However, when striving for energetically coherent instrumental interaction between the user and the simulated object, the communication between the haptic device and the simulation plays a key role.

As underlined in Sect. 8.3.2, the global physical entity composed of the force-feedback device and virtual object can be defined as a physical, energy-conserving system only if the haptic position and force data streams integrate seamlessly into the CORDIS-ANIMA closed-loop simulation. To this end, the haptic loop must run synchronously at the rate of the physical simulation, with single-sample latency between its position output and force input. For simulations running at several kHz, the time step (approximately 20–100 μs) within which AD/DA conversions, bidirectional communication with the haptic device and a single computation loop for the whole physical model must occur imposes a reactive computing architecture with guaranteed response time, which is not attainable by general-purpose machines [10].

Additionally, the simulation of physical models sufficiently complex for musical purposes is computationally demanding and therefore ill-fitted for calculation on most current embedded systems. A previous simulation architecture at ACROE-ICA [19] was based on the TORO board from Innovative Integration; while it allowed running the haptic loop synchronously at audio rate (44.1 kHz), the available processing power limited the system to small-scale physical models [20].

The hardware and software architecture of the MSCI platform (shown in Fig. 8.7) consequently addresses both the need for high computing power and reactive I/O. It is based on the RedHawk Linux real-time operating system (RTOS), where the physical simulation is computed in two sections: one running at audio rate (44.1 kHz) and the other running at control (gestural) rate (1–10 kHz). The TORO DSP board serves as a front-end for haptic I/O. Sound is handled by an external soundcard. These components are synchronised through a shared master clock (the soundcard’s wordclock). Visualisation data, on the other hand, is processed asynchronously so as to display the physical model during simulation.

This platform can simulate virtual scenes with up to 7000 interacting audio-rate physical modules: an approximate performance gain by a factor of 50 compared to the previous embedded architecture.

The mechanics of traditional instruments present a natural cohabitation of multiple dynamics. In particular, instruments can be generally separated into:

These two sections are coupled by means of nonlinear interactions (percussion, friction, plucking, etc.) that transform low-bandwidth gesture energy into high-bandwidth energy of acoustical vibrations (Fig. 8.8).

Since these two sections of an instrument operate at different frequency rates, it comes naturally to simulate their discrete-time representations at different sampling rates. While this results in computational optimisation, a major issue arises: how to retain coherent physical coupling between the low-rate and high-rate sections of the instrument and at the same time meet the constraints of synchronous simulation?

Creating physical models in MSCI is similar to classic modelling with GENESIS, especially concerning the design of vibrating sections of the instrument. The haptic device is integrated directly into the CORDIS-ANIMA model as a series of TGR <MAT> modules, one for each allocated 1-DoF. However, designing haptic DMIs in this way presents a number of specific concerns:

Details concerning calibration and impedance matching are described in [20], and various instrument designs are discussed in [21].

Since the first release of the MSCI platform in 2015, over 100 virtual instruments have been created by the authors, students and the general public. The computing power of modern systems has allowed for the first time to simulate and interact haptically with large-scale instruments composed of thousands of interacting modules. Figure 8.11 shows an example of such models. Especially for large structures with nonlinear acoustical behaviour—such as membranes or cymbals—exploration through real-time manipulation greatly facilitates the iterative design and fine-tuning process.

One notable feature of MSCI’s models is their rich and complex response to different categories of musical gestures [6]. Indeed, as the entire instrument is modelled physically with CORDIS-ANIMA, the user has access to each single simulated point of physical matter. This is not possible in more encapsulated or global physical modelling techniques such as digital waveguides [32] or modal synthesis [1]. This allows for subtle and complex control of the virtual instrument using various haptic modules for different gestures. In the case of a simple string, the excitation gesture could be, e.g. plucking, striking or bowing, whereas modification gestures could be, e.g. pinning down the string onto the fretboard to change its length and pitch (as shown in Fig. 8.12), gently applying pressure onto specific points of the open string to obtain natural harmonics, applying pressure near the bridge to “palm mute” the string or even dynamically move the bridge or the tuning peg of the string to change its acoustical properties over time.

Hélios is an interactive musical and visual piece that was created for the AST 2015¹ festival. For the first time, an MSCI force-feedback station was used in a public live performance. The entire musical content and the visual scenes are created with GENESIS, associating a vast pre-calculated physical model with a real-time MSCI simulation. Video content is projected onto two screens: a large screen for the calculated visual scenes and a screen for the real-time visuals associated with the MSCI simulation. The sound projection is handled with a sound dome of 24 speakers, placed in a semi-sphere above the audience.

The pre-calculated virtual instrumental scene in Hélios is composed of approximately 200000 GENESIS modules (Fig. 8.13). The off-line simulation of this vast instrumental scene allows:

The MSCI system incorporated into the installation uses a 12 DoF force-feedback device (Fig. 8.3). A model made of approximately 7000 physical modules is loaded onto the MSCI workstation. This model is a subgroup of the entire model, which guarantees coherency between the sound textures produced by the off-line simulation and those produced during the real-time interaction with the MSCI virtual instrument. This fusion blurs the boundaries between off-line and real-time sections and offers rich possibilities for the composition and musical structure in the temporal, spatial and structural dimensions of the piece.

The described configuration illustrates one of the many possible interaction scenarios between real and virtual players, real and virtual instruments, real-time and off-line (“supra-instrumental”) instrumental situations, as previously described by Cadoz [8].