Skip to main content
Top

Hint

Swipe to navigate through the chapters of this book

2021 | OriginalPaper | Chapter

Gesture-Timbre Space: Multidimensional Feature Mapping Using Machine Learning and Concatenative Synthesis

Authors : Michael Zbyszyński, Balandino Di Donato, Federico Ghelli Visi, Atau Tanaka

Published in: Perception, Representations, Image, Sound, Music

Publisher: Springer International Publishing

share
SHARE

Abstract

This chapter explores three systems for mapping embodied gesture, acquired with electromyography and motion sensing, to sound synthesis. A pilot study using granular synthesis is presented, followed by studies employing corpus-based concatenative synthesis, where small sound units are organized by derived timbral features. We use interactive machine learning in a mapping-by-demonstration paradigm to create regression models that map high-dimensional gestural data to timbral data without dimensionality reduction in three distinct workflows. First, by directly associating individual sound units and static poses (anchor points) in static regression. Second, in whole regression a sound tracing method leverages our intuitive associations between time-varying sound and embodied movement. Third, we extend interactive machine learning through the use of artificial agents and reinforcement learning in an assisted interactive machine learning workflow. We discuss the benefits of organizing the sound corpus using self-organizing maps to address corpus sparseness, and the potential of regression-based mapping at different points in a musical workflow: gesture design, sound design, and mapping design. These systems support expressive performance by creating gesture-timbre spaces that maximize sonic diversity while maintaining coherence, enabling reliable reproduction of target sounds as well as improvisatory exploration of a sonic corpus. They have been made available to the research community, and have been used by the authors in concert performance.
Literature
2.
go back to reference Beller, G.: Gestural control of real time speech synthesis in Luna Park. In: Proceedings of Sound Music Computing Conference, SMC, Padova, Italy (2011) Beller, G.: Gestural control of real time speech synthesis in Luna Park. In: Proceedings of Sound Music Computing Conference, SMC, Padova, Italy (2011)
3.
go back to reference Bernardo, F., Zbyszyński, M., Grierson, M., Fiebrink, R.: Designing and evaluating the usability of a machine learning API for rapid prototyping music technology. Front. Artif. Intell. 3(a13), 1–18 (2020) Bernardo, F., Zbyszyński, M., Grierson, M., Fiebrink, R.: Designing and evaluating the usability of a machine learning API for rapid prototyping music technology. Front. Artif. Intell. 3(a13), 1–18 (2020)
5.
go back to reference Caramiaux, B., Donnarumma, M., Tanaka, A.: Understanding gesture expressivity through muscle sensing. ACM Trans. Comput. Hum. Interact. (TOCHI) 21(6), 31 (2015) CrossRef Caramiaux, B., Donnarumma, M., Tanaka, A.: Understanding gesture expressivity through muscle sensing. ACM Trans. Comput. Hum. Interact. (TOCHI) 21(6), 31 (2015) CrossRef
6.
go back to reference Delle Monache, S., Rocchesso, D.: To embody or not to embody: a sound design dilemma. In: Machine Sounds, Sound Machines. XXII Colloquium of Music Informatics, Venice, Italy (2018) Delle Monache, S., Rocchesso, D.: To embody or not to embody: a sound design dilemma. In: Machine Sounds, Sound Machines. XXII Colloquium of Music Informatics, Venice, Italy (2018)
7.
go back to reference Di Donato, B., Tanaka, A., Zbyszyński, M., Klang, M.: EAVI EMG board. In: Demo of International Conference on New Interfaces for Musical Expression. NIME 2019, Federal University of Rio Grande do Sul, Porto Allegre, Brazil, June 2019 Di Donato, B., Tanaka, A., Zbyszyński, M., Klang, M.: EAVI EMG board. In: Demo of International Conference on New Interfaces for Musical Expression. NIME 2019, Federal University of Rio Grande do Sul, Porto Allegre, Brazil, June 2019
8.
go back to reference Fails, J.A., Olsen Jr, D.R.: Interactive machine learning. In: Proceedings of the 8th International Conference on Intelligent User Interfaces, pp. 39–45 (2003) Fails, J.A., Olsen Jr, D.R.: Interactive machine learning. In: Proceedings of the 8th International Conference on Intelligent User Interfaces, pp. 39–45 (2003)
9.
go back to reference Fiebrink, R., Cook, P.R.: The Wekinator: a system for real-time, interactive machine learning in music. In: Proceedings of the International Society for Music Information Retrieval Conference, ISMIR 2010, Utrecht, Netherlands (2010) Fiebrink, R., Cook, P.R.: The Wekinator: a system for real-time, interactive machine learning in music. In: Proceedings of the International Society for Music Information Retrieval Conference, ISMIR 2010, Utrecht, Netherlands (2010)
11.
go back to reference Françoise, J.: Motion-sound mapping by demonstration. Ph.D. thesis, UPMC Université Pierre et Marie Curie, Paris (2015) Françoise, J.: Motion-sound mapping by demonstration. Ph.D. thesis, UPMC Université Pierre et Marie Curie, Paris (2015)
12.
go back to reference Françoise, J., Caramiaux, B., Bevilacqua, F.: A hierarchical approach for the design of gesture-to-sound mappings. In: 9th Sound and Music Computing Conference, SMC, Copenhagen, Denmark (2012) Françoise, J., Caramiaux, B., Bevilacqua, F.: A hierarchical approach for the design of gesture-to-sound mappings. In: 9th Sound and Music Computing Conference, SMC, Copenhagen, Denmark (2012)
13.
go back to reference Hunt, A., Wanderley, M.M.: Mapping performer parameters to synthesis engines. Organ. Sound 7(2), 97–108 (2002) CrossRef Hunt, A., Wanderley, M.M.: Mapping performer parameters to synthesis engines. Organ. Sound 7(2), 97–108 (2002) CrossRef
17.
go back to reference Parke-Wolfe, S.T., Scurto, H., Fiebrink, R.: Sound control: supporting custom musical interface design for children with disabilities. In: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2019, Porto Alegre, Brazil (2019) Parke-Wolfe, S.T., Scurto, H., Fiebrink, R.: Sound control: supporting custom musical interface design for children with disabilities. In: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2019, Porto Alegre, Brazil (2019)
19.
go back to reference Sanger, T.D.: Bayesian filtering of myoelectric signals. J. Neurophysiol. 97(2), 1839–1845 (2007) CrossRef Sanger, T.D.: Bayesian filtering of myoelectric signals. J. Neurophysiol. 97(2), 1839–1845 (2007) CrossRef
21.
go back to reference Savary, M., Schwarz, D., Pellerin, D., Massin, F., Jacquemin, C., Cahen, R.: Dirty tangible interfaces: Expressive control of computers with true grit. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2013, Paris, France, pp. 2991–2994. ACM (2013). https://​doi.​org/​10.​1145/​2468356.​2479592 Savary, M., Schwarz, D., Pellerin, D., Massin, F., Jacquemin, C., Cahen, R.: Dirty tangible interfaces: Expressive control of computers with true grit. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2013, Paris, France, pp. 2991–2994. ACM (2013). https://​doi.​org/​10.​1145/​2468356.​2479592
22.
go back to reference Schnell, N., Röbel, A., Schwarz, D., Peeters, G., Borghesi, R.: MuBu & friends - assembling tools for content based real-time interactive audio processing in Max/MSP. In: Proceedings of the International Computer Music Conference, ICMC, Montreal, Quebec, QC, pp. 423–426 (2009) Schnell, N., Röbel, A., Schwarz, D., Peeters, G., Borghesi, R.: MuBu & friends - assembling tools for content based real-time interactive audio processing in Max/MSP. In: Proceedings of the International Computer Music Conference, ICMC, Montreal, Quebec, QC, pp. 423–426 (2009)
26.
go back to reference Schwarz, D., Cahen, R., Britton, S.: Principles and applications of interactive corpus-based concatenative synthesis. In: Journées d’Informatique Musicale. JIM, Albi, France (2008) Schwarz, D., Cahen, R., Britton, S.: Principles and applications of interactive corpus-based concatenative synthesis. In: Journées d’Informatique Musicale. JIM, Albi, France (2008)
29.
go back to reference Stowell, D., Pumbley, M.D.: Timbre remapping through a regression-tree technique. In: Proceedings of the Sound Music Computing Conference, SMC (2010) Stowell, D., Pumbley, M.D.: Timbre remapping through a regression-tree technique. In: Proceedings of the Sound Music Computing Conference, SMC (2010)
30.
go back to reference Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018) MATH Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018) MATH
31.
go back to reference Tanaka, A., Di Donato, B., Zbyszyński, M.: Designing gestures for continuous sonic interaction. In: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2019, Porto Alegre, Brazil (2019) Tanaka, A., Di Donato, B., Zbyszyński, M.: Designing gestures for continuous sonic interaction. In: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME 2019, Porto Alegre, Brazil (2019)
32.
go back to reference Tanaka, A., Ortiz, M.: Gestural musical performance with physiological sensors, focusing on the electromyogram. In: Lesaffre, M.L., Maes, P.J., Leman, M. (eds.) The Routledge Companion to Embodied Music Interaction. Routledge, London (2017) Tanaka, A., Ortiz, M.: Gestural musical performance with physiological sensors, focusing on the electromyogram. In: Lesaffre, M.L., Maes, P.J., Leman, M. (eds.) The Routledge Companion to Embodied Music Interaction. Routledge, London (2017)
34.
go back to reference Visi, F.G., AQAXA: “You have a new memory”. In: ICLI 2020 - the Fifth International Conference on Live Interfaces, Trondheim, Norway (2020) Visi, F.G., AQAXA: “You have a new memory”. In: ICLI 2020 - the Fifth International Conference on Live Interfaces, Trondheim, Norway (2020)
35.
go back to reference Visi, F.G., Tanaka, A.: Towards assisted interactive machine learning: exploring gesture-sound mappings using reinforcement learning. In: ICLI 2020 - the Fifth International Conference on Live Interfaces, Trondheim, Norway (2020) Visi, F.G., Tanaka, A.: Towards assisted interactive machine learning: exploring gesture-sound mappings using reinforcement learning. In: ICLI 2020 - the Fifth International Conference on Live Interfaces, Trondheim, Norway (2020)
37.
go back to reference Wessel, D.L.: Timbre space as a musical control structure. Comput. Music J. 45–52 (1979) Wessel, D.L.: Timbre space as a musical control structure. Comput. Music J. 45–52 (1979)
38.
go back to reference Zbyszyński, M., Di Donato, B., Tanaka, A.: The effect of co-adaptive learning & feedback in interactive machine learning. In: ACM CHI: Human-Centered Machine Learning Perspectives Workshop, Glasgow, UK. ACM (2019) Zbyszyński, M., Di Donato, B., Tanaka, A.: The effect of co-adaptive learning & feedback in interactive machine learning. In: ACM CHI: Human-Centered Machine Learning Perspectives Workshop, Glasgow, UK. ACM (2019)
Metadata
Title
Gesture-Timbre Space: Multidimensional Feature Mapping Using Machine Learning and Concatenative Synthesis
Authors
Michael Zbyszyński
Balandino Di Donato
Federico Ghelli Visi
Atau Tanaka
Copyright Year
2021
DOI
https://doi.org/10.1007/978-3-030-70210-6_39