Sonic Design

Sonic design, understood as the activity of intentionally creating sound events, encompasses both musical craftsmanship and analytic reflection. It may include technologies for sound synthesis and processing, as well as traditional methods for sound generation by musical instruments or the human voice, and also principles of orchestration. Common to many instances of sonic design, is having acoustic components that blend with concurrent real or imagined motion sensations. Thus, sonic design can be understood as a multimodal phenomenon, yet we often lack suitable concepts for differentiating and evaluating these multimodal components. This paper aims to present work on developing a scheme to detect, and actively exploit, generic motion components in sonic design, be that as analytic or creative tools.

In the past decades, musicology has been evolving at a pace that matches new developments in technology. Underneath this development, a new theory of music emerged, embracing interaction states as a model for understanding how music can be empowering. In the present chapter, sound design is considered from the viewpoint of interaction states, using caregiver–infant communication as a challenging domain of application. Sound design components of interest are identified, as well as human capacities for dealing with them in terms of empowerment. These are related to the concepts of self-augmented interaction and biofeedback-based sound design.

The perception of musical rhythm includes not only the sonic rhythm but also the endogenous reference structures, such as meter. Musical meter is often described and understood as points in time or durations between such points. In this chapter, I argue that musical meter also has a shape. I propose that we perceive and make sense of musical meter based on our previous musical experiences involving meter-related bodily motion. In other words, the meter-related motion is integral to the perceived meter—they are the same. Meter thus has a shape that relates to the embodied sensations of these movements. Also crucial is the notion that musical meter is conditioned by musical culture. This perspective on meter as shape is highly influenced by Godøy’s motor-mimetic perspective on music perception and musical shape cognition and concurs with the multimodal approach to sonic design that acknowledges motion as intrinsic to music performance and perception.

This chapter draws on prompts from Rolf Inge Godøy, Edmund Husserl, and a range of Indigenous, queer, and decolonial phenomenological thinkers to frame a theory of gestural time for music that rethinks the relationship between experience and perception. It plays with the distinction between Husserl’s “exact” and “descriptive” sciences, putting the latter to work as a productive foil to the drive for empirical exactitude that animates much perception and cognition theory. It does so not to replace exactitude, but to enrich the experiential nexus. Gesture emerges as an at least equally (and perhaps more) plausible first principle for reunderstanding the mechanisms by which perception functions. Focusing on a debate on categorical identity between Rainer Polak and Justin London, it considers the possibility that a turn to affect—understood in Baruch Spinoza’s sense of a pre-personal flow of force relations that condition the very possibility of experience and perception in the first place—can work to elide certain kinds of experimental cleavings to a priori category distinctions and to at least provisionally displace perceptual exactitude as the primary location for understanding musical experience.

In recent years, research on sonification has paid more attention to sound variations induced by expressive gestures. This chapter focuses on conducting gestures, emphasizing expressive gestures performed by the non-dominant hand. It is assumed that these gestures implicitly correspond to musical nuances partially encoded in the scores and convey a meaning based on a grammatical structure specific to gestural languages. We, therefore, propose to analyze these gestures in light of linguistic mechanisms that govern signed languages. In particular, we are interested in the processes of sign formation from the combination of elementary components and the inflection processes that apply to these components to efficiently generate rich and expressive sentences. Based on this grammatical theory underlying sign language and a sound-tracking methodology, we create and evaluate a new dataset of expressive conducting gestures.

In this chapter, we describe a series of studies related to our research on using gestural sonic objects in music analysis. These include developing a method for annotating the qualities of gestural sonic objects on multimodal recordings; ranking which features in a multimodal dataset are good predictors of basic qualities of gestural sonic objects using the Random Forests algorithm; and a supervised learning method for automated spotting designed to assist human annotators. The subject of our analyses is a performance of Fragmente², a choreomusical composition based on the Japanese composer Makoto Shinohara’s solo piece for tenor recorder Fragmente (1968). To obtain the dataset, we carried out a multimodal recording of a full performance of the piece and obtained synchronised audio, video, motion, and electromyogram (EMG) data describing the body movements of the performers. We then added annotations on gestural sonic objects through dedicated qualitative analysis sessions. The task of annotating gestural sonic objects on the recordings of this performance has led to a meticulous examination of related theoretical concepts to establish a method applicable beyond this case study. This process of gestural sonic object annotation—like other qualitative approaches involving manual labelling of data—has proven to be very time-consuming. This motivated the exploration of data-driven, automated approaches to assist expert annotators.

Sound acts as an extension of the body, created by movement and received as vibration. I am focused on the removal of a visual representation of the body as a template; to instead facilitate an embodied experience. As an embodied practitioner, I create immersive sound and media installations derived from recordings of my own moving body. The movement of sound depicts the presence of a body in motion through sensory illusion. Through embodied sonic design, my sound recordings decontextualize, abstract, and reframe the auditory experience. I physically manipulate the recording of sound to perceptually rematerialize the moving physical form during playback with two techniques: sound shadows and embodied binaural spatialization. These techniques encourage the listener to perceive sound and space with the same awareness that situates their body, such as sensation and proprioception. The perceived physical interaction within the reception of this sound is akin to a kinesthetic projection and is an engagement in spatial thinking, activating mirror neurons and kinesthetic empathy. Creating awareness through physical attunement can regulate systems out of balance by offering the embodiment of alternative states: shifting how one thinks and feels in a particular setting. My research seeks to recognize the listener’s unique perspective through their individual body.

The Covid-19 pandemic catalysed disruptions and disturbances in ways of living across the globe. Many of these changes in daily life were felt through stark changes to our soundscapes, particularly those in urban centres. Might we better understand the effects of the Covid-19 lockdowns through sonic analysis? This chapter explores how sound analysis methods, including concepts of the sound-motion object and sonic image, might aid in understanding the environmental soundscapes of the pandemic lockdowns. The discussion focuses on the Sounding Covid-19 project—an initiative involving a series of field recordings carried out during Covid-19 pandemic-related events in the urban environments of Belfast, Northern Ireland (2020–2022) and Montreal, Canada (2020–2021). The project presents the sound archive through various listening experiences, including soundscape compositions, sound mapping and narrative-based radiophonic work. We consider how the pandemic may have invited us to pause and reconsider how we document and archive the present to look back and better understand the future. Sound may be vital in understanding our environment and the socio-cultural shifts over time. This chapter argues that documenting, preserving, and analysing the soundscapes of the pandemic lockdowns may help us reflect on our shared histories in several ways.

Neon Meditations is a collaborative performance work combining visual art and music, where colours are translated into sound in an electronic instrument controlled by two performers. The sound design follows the principle of excitation and resonance. We use exciters attached to resonating objects that colour and distort the sound. The mapping from gesture to sound, and the fact that this is a multi-agent system, tends to cause confusion about the way the performers shape the sound. Godøy's concept of sound–motion objects is well adapted to acoustic instrumental music, but using Neon Meditations as an example, we will see that it faces many challenges when one tries to extend its application to live electronic music.

This article sketches the development of pipe organs in Europe from Roman times to the Baroque era in order to shed light on concepts of sound structure that apparently guided the design and construction of certain types of organs. For a better understanding of empirical measurements presented in Parts 4 and 5 of this study, Part 2 provides some basic organology. In Part 3, the development from the so-called ‘Blockwerk’ to organs comprising several wind chests, manuals, and various types of pipes and stops is outlined. In part 5, relations of sound structure to tunings and temperaments are discussed, including actual measurements.

Algorithms and related technologies are widely used for many musicology-related tasks, such as music analysis or even music composition. The use of algorithms in music analysis may be crucial for a deeper understanding of music theory and history, yet experiential knowledge is proposed here as a more interactive way to take a journey through music history and, more specifically, the evolution of electroacoustic music. From acousmatic music to serialism and from Musique Concrète and Elektronische music to Post-Schaefferian Electronica, numerous techniques have been developed for sound generation and manipulation. In this chapter, SuperCollider is used as a tool to create an interactive composition and to provide a walkthrough of electroacoustic music through live coding. The musicological aspects of the different composition techniques of this music style are explored through their integration into the algorithmic composition.

Analysing electroacoustic music remains challenging, leaving this artistic treasure somewhat out of reach of mainstream musicology and many music lovers. This chapter examines electroacoustic music analysis, covering musicological investigations and desires and technological challenges and potentials. The aim is to develop new technologies to overcome the current limitations. The compositional and musicological foundations of electroacoustic music analysis are based on Pierre Schaeffer’s Traité des objects musicaux. The chapter presents an overview of core analytical principles underpinning more recent musicological approaches, including R. Murray Schafer’s soundscape analysis, Denis Smalley’s spectro-morphology, and Lasse Thoresen’s graphical formalisation. Then the state of the art in computational analysis of electroacoustic music is compiled and organised along broad themes, from detecting sound objects to estimating dynamics, facture and grain, mass, motions, space, timbre and rhythm. Finally, I sketch the principles of what could be a Toolbox des objets sonores.

Mechanical musical instruments have less timbre variability than electronic instruments. Extended playing techniques and more sophisticated acoustic instrument designs have recently appeared. We suggest acoustic metamaterials as a new way to extend the timbre of mechanical instruments beyond their present sound capabilities. In this chapter, we present three examples of acoustic metamaterials: (1) a one-dimensional string, (2) a labyrinth sphere, and (3) a two-dimensional membrane. The string is covered with additional masses, which leads to a dispersion relation of the harmonic overtones in the sound spectrum. The resulting sound still has a detectable pitch but is very different from a regular string on a mechanical instrument. The labyrinth sphere has a clear band-gap damping and can be used in loudspeakers, musical instruments, or room acoustics due to its small size. A circle of masses is attached to the membrane, leading to a cloaking behaviour of vibrations from within the circle to outside and vice versa. Again, the resulting sound is considerably different from a regular drum and leads to increased variability of musical articulations. Using a microphone array, laser interferometry, impedance tube, and high-speed video recordings with subpixel tracking, the vibrations on the string and the membrane are investigated and discussed in relation to new instrument designs.

This chapter presents a retrospective of five interactive systems I have developed focusing on how machines can respond to body movement in music performance. In particular, I have been interested in understanding more about how humans and non-human entities can share musical control and agency. First, I give an overview of my musical and aesthetic background in experimental music practice and a less conventional approach to sound and music control. Then follows a presentation of embodiment and music cognition theories that informed the techniques and methods I employed while developing these systems. Then comes the retrospective section structured around five projects. Biostomp explores the unintentionality of body signals when used for music interaction. Vrengt demonstrates musical possibilities of sonic microinteraction and shared control. RAW seeks unconventional control through chaos and automation. Playing in the “air” employs deep learning to map muscle exertions to the sound of an “air” instrument. The audiovisual instrument CAVI uses generative modeling to automate live sound processing and investigates the varying sense of agency. These projects show how an artistic–scientific approach can diversify artistic repertoires of musical artificial intelligence through embodied cognition.