In this paper we present three different experiments designed to explore sound properties associated with fluid movement: (1) an experiment in which participants adjusted parameters of a sonification model developed for a fluid dance movement, (2) a vocal sketching experiment in which participants sketched sounds portraying fluid versus nonfluid movements, and (3) a workshop in which participants discussed and selected fluid versus nonfluid sounds. Consistent findings from the three experiments indicated that sounds expressing fluidity generally occupy a lower register and has less high frequency content, as well as a lower bandwidth, than sounds expressing nonfluidity. The ideal sound to express fluidity is continuous, calm, slow, pitched, reminiscent of wind, water or an acoustic musical instrument. The ideal sound to express nonfluidity is harsh, non-continuous, abrupt, dissonant, conceptually associated with metal or wood, unhuman and robotic. Findings presented in this paper can be used as design guidelines for future applications in which the movement property fluidity is to be conveyed through sonification.
This work was supported by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 645553 (DANCE).
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
1 Introduction
This study is part of the European Union’s H2020 research innovation programme DANCE,1 focusing on how affective and social qualities of human full-body movement can be expressed, represented and analyzed through sound and music. DANCE shares similarities with the previous Embodied Generative Music project (EGM).2 The purpose of the DANCE project is to investigate if it is possible to perceive expressive movement qualities in dance solely through the auditory channel, i.e. to capture expressive qualities of dance movements and convey them through sounds. The ability to translate finer qualities of some information from one modality to another has many use cases and practical implications. For example, communicating movement qualities through sound can be of great use for blind users. The DANCE project primarily focuses on artistic practice, but findings will be relevant also for other areas and domains, such as everyday applications involving movement.
One of the goals of the DANCE project is to identify a collection of expressive qualities that could characterise the expressive content of a dance performance, and to develop computational models for these features. Grounded on literature and theories from humanities, a collection of expressive qualities were included in a multilayer framework building upon the work presented by Camurri et al. in [4]. This conceptual framework is based on physical signals captured using e.g. motion capture systems, IMUs, audio, breath measured with thermistors or EMG. It includes low-level signals (e.g. trajectories and velocities of joints), expressive (mid-level) features (e.g. directness, impulsivity, suddenness and fluidity) and high-level features (e.g. emotional states and social attitudes). The current study focuses on fluidity, which belongs to the mid-level of the conceptual framework. In the context of body movement, a fluid movement is smooth and coordinated, such as a wave-like propagation through body joints [4]. Fluidity has been found to be one of the properties that appears to contribute significantly to perception of emotions in dance [5], suggesting that it could serve as a meaningful parameter that can be mapped to sound in interactive sonification of bodily movement. Apart from fluidity, sonification of other features, ranging from low level features (e.g. velocity) to high level features (e.g. fragility and lightness), have also been done within the scope of the DANCE project.3
Anzeige
In previous studies on sonification of continuous body gestures, researchers have identified certain sound properties that are associated with fluent or irregular movements. For example, in a study on the sonification of handwriting it was found that sound models characterized by low frequency components were more suitable for both aiding and communicating fluency of movements, while sound models characterized by high frequency crackling sounds (impact sounds) were suitable for portraying jerky hand movements lacking fluency [7]. In another study focusing on the use of sounds in learning of movement kinematics, researchers found that sounds that were noisier or had loud high-frequency components could help users identify motion behavior [2]. Moreover, in our previous study [11], a smooth sound model with a continuous sound was rated as significantly more fluid than a sound model characterized by a high level of amplitude modulation and sudden amplitude irregularities. Other related work in this context includes research on sound synthesis of fluid motions, such as physics of liquids in motion and smoothed particle hydrodynamics, as in [8], in which a multi-rate sound synthesis method of liquid phenomena was proposed. There are also examples of more artistic applications, such as e.g. [16], in which materials and techniques for creating a sound installation relying on fluid motion as a source of musical control were explored.
In this paper, we use vocal sketching as a prototyping tool for exploration of sound designs of fluid movements. Vocal sketching involves the use of the voice and body to demonstrate the relationship between action and sonic feedback [19]. It can be effective when describing sounds that don’t have clear agreed upon symbols in language (e.g. when the source of the sound cannot be identified), or when communicating sound characteristics that are ambiguous, such as pitch or temporal qualities (e.g. how low is “low”, how fast is “fast”) [10, 17]. Vocal sketching has emerged as an effective tool in Sonic Interaction Design, and has successfully been applied in a wide range of different projects.4
The current study is a continuation and expansion of a pilot study presented in [12]. In the following sections, we present three different studies aimed at exploring which acoustic properties that are associated with properties of fluid versus nonfluid movements in the context of interactive sonification. We believe that findings from the current study could be useful for researcher in the field of Sonic Interaction Design, especially for those concerned with designing sounds for applications involving movement that requires smooth and wavelike motion trajectories, for example in sonification of writing or in sonification of physiotherapy applications.
2 Method
2.1 Method of Experiment 1
In our previous work [12], we defined five different sound models for sonification of fluidity and evaluated these models based on their ability to express the fluidity property. The model ranked as most fluid in this previous study was used in an interactive experiment in which a gesture from a fluid movement sequence, performed by a dancer, was sonified. Participants were presented with a looping video of this movement and it’s corresponding sonification.5 The movement sequence had previously been found to be perceived as being very fluid (see [18]). The synchronized playback of video, movement features and sound was done using a custom video playback software written in C++ using the OpenFrameworks6 environment. The sonification was based on the following movement features, extracted from the dancer’s movements: fluidity and energy (i.e. kinetic energy). Fluidity was extracted using the method described in [18]. Here, fluidity is estimated by comparing mean jerk values (i.e. third derivative of positional data) of the shoulders, elbows and hands from measurements of a dancer with simulated data of a mass-spring model. For the mass-spring model, each joint of the body is modeled as a mass connected to springs simulating muscle tension. By tuning parameters of the model (e.g. joint masses, spring stiffness and damping coefficients), very fluid movements that generate smooth trajectories can be simulated. This simulated data is then compared to recorded movement data, by calculating the distance between the jerk data of the two datasets, thereby providing an estimate of fluidity for a given trajectory segment. For more details on the energy feature, please refer to our previous paper [12].
Anzeige
The sound model used for sonification was created using the SuperCollider7 programming language. The model exploited harmonic pitch sensitivity, physical interpretation of spectral slope and amplitude, as well as a noise generator that could move from coarse, gravel-like sounds, to smoother sounds reminiscent of a water stream or wind. The sound design was informed by previous research on emotions in music [3, 13, 15], including expressive body motion of musicians [6] and laypersons [14], and by the extensive review on mapping strategies performed by Dubus and Bresin [9]. Results from research on emotions in music show how articulation, pitch height, and spectral content are important parameters in the communication of emotions through bodily motions [6, 14]. In the case of the current study, we assumed that a more regular and dull sound, with narrow spectral content and a low centroid, would be more suitable for the sonification of smooth and regular movements. In music performance, these movement properties are matched to a slower tempo and a more legato articulation, as well as less active emotions (such as tenderness, love and sadness) [6]. In Experiment 1, sounds characterized by the above mentioned characteristics can be produced by setting six sliders on a MIDI controller to low values. On the contrary, we considered a more irregular sound characterized by a broader spectral content and a higher centroid to be suitable for more rigid movements (this can be achieved by setting the six sliders to high values).
An increase in energy was mapped to an increase in amplitude, starting from complete silence, given zero energy. White noise was processed by a bank of parallel band-pass filters, with variable tuning quantized to semitone steps in an equal temperament scale, and variable resonance. An increase in energy increased the cut-off frequency of the filters, maintaining their harmonic relationship. An increase in fluidity increased the Q-value of the filters, making them narrower, with the resulting filtered sounds approaching sinusoidal waves. A decrease in fluidity made the filters wider, resulting in a noisier output. Decreased fluidity also independently added a detuning component to all filters, resulting in less harmonious sounds.
Participants were instructed to adjust 6 sliders (with 8 bit resolution) that controlled the following aspects of the sound model in real-time:
Slider 1: The quantization step of the center frequencies of the band-pass filters, ranging from continuous to steps of a minor third.
Slider 2: The amount of high frequency content in the noise source.
Slider 3: Scaling of the fluidity parameter mapping to the bandwidth of the band-pass filters.
Slider 4: Scaling of the energy parameter mapping to the center frequencies of the band-pass filters.
Slider 6: Manipulation of the center frequencies of the band-pass filters, ranging from harmonic to inharmonic.
Initially, the participants were instructed to perform two tasks; T1: “Adjust sliders so that the sound corresponds well to the movement performed in the video” (fluid condition), and T2: “Adjust sliders so that the sound does not correspond well to the movement performed in the video” (nonfluid condition). Values from the sliders were continuously logged and the audio output was recorded. Each task was completed when the participant stated that (s)he was satisfied with the audible result. Regarding the definition of T1 and T2, it would of course have been possible to perform the tuning in a fluid and nonfluid condition (i.e. to produce a sound that corresponded well to the fluid video and another sound that corresponded well to a nonfluid video). However, adding a second video would introduce another level of complexity to the experiment, as this video segment would have radically different movement trajectory data, and thus would sound rather different from the fluid sonification (even if exactly the same slider settings were used).
In total, 29 participants took part in the experiment (M 18, F 11, Mean \(=\) 26.00 years, SD \(=\) 6.21 years). However, due to software issues that made the sound synthesis engine freeze occasionally, data from only 18 participants (M 12, F 6, Mean \(=\) 26.33 years, SD \(=\) 7.40 years) was included in the analysis.9 Instructions for the experiment were read from a pre-written manuscript. The experiment was carried out on several different occasions with a total of five different instructors (four of which were not aware of the research hypothesis). This approach was used in order to reduce the risk for bias caused by having one single instructor (especially in the subsequent interview described in Sec. 2.2).
2.2 Method of Experiment 2
After Experiment 1, the same participants were asked to complete the following task (T1): “You will now see a video of a movement. After the video, try to describe how you believe that a sound portraying this movement would sound. You are encouraged to use metaphors when describing the sound. If you would use your voice to sketch the sound that would portray this movement, what would it sound like?” Participants were presented with a video of a fluid movement, different from the video used in Experiment 1. Finally, task T1 was repeated, but this time with a video of a nonfluid movement (T2). The stimuli videos used in T1 and T2 were judged as very fluid versus very nonfluid in a previous study [18]. Point light display representations of both videos are available as supplementary material.10 The reason for including vocal sketching in the study was to allow for participants to explore any sounds, as the sound model used in Experiment 1 had limited participants to a set of predefined control parameters.
Participants were allowed to answer the interview questions in either Swedish or English. In total, 23 participants decided to answer in Swedish, and 6 in English. The interviews collected from T1 and T2 were transcribed (Swedish answers were translated after transcription). The obtained results were initially analyzed using word frequency analysis. Content analysis, based on an emergent coding scheme, was subsequently carried out. Content analysis is often an efficient and replicable method for arranging large amounts of transcribed data into well-defined categories [20]. In our previous pilot study [12], we described and analyzed distinctions between the fluid and nonfluid property in terms of categories based on verbal descriptions of both fluid sounds and fluid movements. In this paper, we focus primarily on verbal descriptors of the envisioned fluid versus nonfluid sounds.
Since all participants who took part in Experiment 2 had also taken part in Experiment 1, we also did a follow-up experiment with a group of 13 new participants (8 M, 5 F, Mean \(=\) 28.77 years, SD \(=\) 14.57 years) that had not participated in Experiment 1 prior to taking part in the vocal sketching. The participants were given the same instructions as the other group. This follow-up experiment was carried out for comparative purposes, in order to investigate if there was an effect of being exposed to the sound model used in Experiment 1.
2.3 Method of Experiment 3
Experiment 3 was organized as a workshop at the Interactive Sonification Workshop 2016 (Ison 2016). In total, 5 participants (3 M, 2 F, Mean \(=\) 29.80 years, SD \(=\) 2.86 years), all researchers in fields related to interactive sonification, took part in the experiment. The workshop was divided into two tasks; T1: Identification of fluid sounds, and T2: Production of fluid sonifications. In task T1, participants were shown a longer movement sequence of a dancer performing a fluid movement (also from [18]), and asked to identify and discuss sounds that represented the body motion qualities represented in the video. Participants were instructed to use online resources such as e.g. Freesound Creative Commons database11 to identify particular sounds. Using identified sounds from Freesound database as basis for discussion and understanding of certain qualities of an intended sonic space has previously been successfully done in e.g. [1]. As opposed to the previous experiments described in this paper, in which participants were restricted to existing sound models or the sound-producing capabilities of their own voices, this task enabled participants to identify particular sound properties related to fluidity without any restrictions.
In task T2, features extracted from the dancer’s movement (e.g. energy, smoothness and fluidity) were broadcast to participants through Open Sound Control (OSC).12 This enabled participants to produce their own real-time sonifications of the fluid movement sequence. Participants also got access to log files containing the same data.
2.4 Compliance with ethical standards
All participants gave written consent for participation in the study. For Experiment 1 and 2, participants were recruited from the students and staff at KTH. For Experiment 3, participants were volunteer attendees of the Interactive Sonification Workshop (Ison 2016). Participants did not receive any monetary compensation. All participants consented to their data being collected. At the time that the experiments were conducted, no ethics approval was required from KTH for behavioral studies such as the one reported in this paper. For the management of participants’ personal data, we followed regulations according to the KTH Royal Institute of Technology’s Ethics Officer.
3 Results
3.1 Results of Experiment 1
Boxplots of the final slider settings for all 18 participants are presented in Fig. 1. We computed mean values for all sliders for the fluid and nonfluid condition, thereby obtaining an estimate of slider values for the averaged fluid versus nonfluid sound models. Spectrograms of sounds generated from these averaged settings are presented in Fig. 2.13 A paired-sampled t-test was conducted to investigate if the six mean slider values were significantly different in the two conditions (fluid and nonfluid). Rescaling the values of the sliders to 0-1, the paired-samples t-test indicated that values were significantly lower for the fluid condition (M \(=\) 0.29, SD \(=\) 0.06) than for the nonfluid condition (M \(=\) 0.54, SD \(=\) 0.11), t(5) \(=\) − 4.08, \(p = 0.01\).
For each respective slider controlling a certain aspect of the sound, we carried out pairwise comparisons to investigate if there was a significant difference between the fluid versus nonfluid condition. The data did neither meet the assumption of normality for t-tests nor the assumption of symmetry required for paired Wilcoxon signed-rank tests, and was therefore examined using sign tests for two-sample paired data (using SIGN.test from the BSDA package in R). Statistically significant differences between the conditions were observed for sliders 2, 3 and 4, respectively. For slider 2 the sign test indicated significantly higher values for the nonfluid condition than for the fluid condition (z \(=\) 2, \(p=0.001\)). This suggests more high frequency content in the noise source for the nonfluid condition compared to the fluid one. Also for slider 3 the sign test indicated significantly higher values for the nonfluid condition than for the fluid condition (z \(=\) 3, \(p=0.02\)). Slider 3 controls the scaling of the fluidity parameter mapping to the bandwidth of the band-pass filters. The significance test indicates larger bandwidth of the band-pass filters for the nonfluid condition, with sounds perceived as richer and noisier, compared to the fluid condition. Significantly higher values for the nonfluid condition than for the fluid condition were also observed for slider 4 (z \(=\) 3, \(p=0.02\)). This slider controls the scaling of the energy parameter mapping to the center frequencies of the band-pass filters. The significance test indicates overall higher center frequencies of the band-pass filters as well as larger frequency increase per energy unit for the nonfluid condition, compared to the fluid one.
Fig. 1
Boxplots of slider values in the fluid respective nonfluid condition. Lower and upper hinges correspond to the first and third quartiles, the whiskers extend from the hinge to the largest value no further than 1.5 * interquartile range (IQR) from the hinge. Data beyond the end of the whiskers are plotted individually as outliers
Fig. 2
Spectrograms generated from recordings when sliders were set to mean values for the fluid versus nonfluid condition
×
×
All recordings from Experiment 1 are available as supplementary material.14 When examining these sounds (18 per condition), we observed that the recordings from the fluid condition were generally more homogeneous than the recordings from the nonfluid one. This was to be expected, for two reasons. Firstly, when designing the sonification model, care was taken to make sure that the sound model was capable of producing sounds with characteristics that the authors intuited would be useful for illustrating fluidity. Secondly, the question was of the form “find X, then, find not X”. Taken together, this means that the participants indeed made use of the model to illustrate fluidity in similar ways, but that they also moved away from the consensus of the fluid condition, but in slightly different directions, accounting for the heterogeneity of the nonfluid condition. Still, some trends can be distinguished within the two conditions. Many of the nonfluid recordings were characterized by a high register and/or very prominent noise components. As a whole, the fluid recordings occupied a low- to mid register and were characterized by darker and more muffled timbres.
3.2 Results of Experiment 2
3.2.1 Task 1: Interviews
The most frequently used words associated with fluid versus nonfluid sounds are presented in Table 1. Below, 13 categories are presented together with a few quotes that are emblematic of respective category. The categories also have a brief description that contextualizes how the participants discussed the sounds in respective condition (fluid or nonfluid). Each category has a fraction in parenthesis that represents how many of the participants that stated something that can be sorted into that category: e.g. “(X/X)” implies that all participants used words or expressions from the category.
Table 1
The six most frequently used words associated with a fluid versus nonfluid sound
Fluid
Nonfluid
Wind (8)
Robot (13)
Calm (8)
Harsh (7)
Waves (5)
Screech (5)
Soft (5)
Cracking (5)
Slow (5)
Choppy (5)
Water (4)
Metallic (4)
Categories DescribingFluidSounds
Wind and Air (14/29)
The sound of wind and moving air streams, with metaphors taken from nature or references to swish and whoosh sound effects.
“A wind that blows, but it is still relatively calm.”
“A wind in the forest [...] the sound of wind whispering through the trees.”
“A gentle breeze through a tree crown.”
“Swishing sound, kind of. But also wind.”
Envelope and Timbral Characteristics (13/29)
Temporal characteristics and descriptions of overall timbre of the sound, including keywords such as continuous, soft, muffled, and harmonic.
“It is a very coherent sound.”
“Pitched, not very noisy.”
“Pretty harmonic, in a sense.”
“ [...] sinus-shaped sounds, something like that.”
“Very soft.”
Ocean and Waves (12/29)
Wavelike properties and sounds produced by the ocean.
“Maybe at the sea or something. I can almost hear waves.”
“A sound that is not jerky, but rather flowing.”
“It could be the sound of waves.”
Music (11/29)
Properties associated with certain musical instruments or musical styles.
“Continuous, not drum beats, but flute or Theremin.”
“Pretty soft music, a bit mellow.”
“ [...] pretty much the sound of strings, not so harsh electronic sounds, more like a violin.”
“It feels like a melody that is easy to listen to.”
Speed (11/29)
Qualities of the sound related to perception of speed, with emphasis on non-rhythmic, slow and coherent sounds.
“It feels like it is very soft, pretty calm, not quick.”
“Obviously something slow.”
Frequency or Pitch (9/29)
Pitch variations related to movements and position of the body as well as general descriptions of an overall low pitch or low frequency.
“When her body kind of goes up and her arms are up, the sounds might have higher pitch.”
“It is not a very high pitch.”
“A somewhat low frequency tone.”
Movement in Liquids (8/29)
The sound of movements in liquids or semi-liquids.
“Something thick, gooey, like stirring something gooey [...] Like taking a mud bath or stirring mashed potatoes.”
“Liquid water of some sort.”
“Like mildly shaking water in a vessel.”
Categories DescribingNonfluidSounds
Envelope and Timbral Characteristics (17/29)
Temporal characteristics and descriptions of overall timbre of the sound, including words like harsh, non-continuous, choppy, metallic and dissonant.
“Jerky, pretty harsh sounds.”
“Any sound that has clear stops, actually, so that there are evident abrupt endings and starting points of the sound.”
“I am thinking noise.”
“Absolutely not continuous.”
“[...] like a sawtooth curve, maybe.”
“A bit harsh and angry.”
Screeching and Cracking Sounds (15/29)
References to screeching, cracking and squeaking sounds produced by moving either mechanical or human joints.
“I am thinking about the joints of the body, and cracking and knocking sounds.”
“[...] something like bones cracking.”
“Not as organic, more a screeching sound.”
“When you open a door and there is a creaking sound, but it is staccato sounds, of some sort.”
Materials and Friction (12/29)
Properties related to friction between materials, or general descriptions of materials such as rusty metal, stone and wood.
“Friction and strong interaction between parts that dont really want to interact.”
“[...] kind of wood, or even metal...”
“It sounds like an old sheet metal object that needs to be oiled.”
“[...] it should sound like when there is very high friction against the floor, like skin against a stone or plastic floor [...] like rubbing sounds.
Robots (12/29)
Sounds associated with robots, such as beeping sounds and sounds produced by robotic movement.
“A squeaky robot.”
“C3P0. Perhaps something that sounds like a robot.”
“It is not only mechanical rattling, but more like ’biiip biiip, booop’.”
“Tin Man from The Wizard of Oz.”
“Daft Punk. Or Kraftwerk rather, the song ’We are the robots’.”
Unhuman Characters (10/29)
References to unhuman characters in movies, and sounds used in horror- or zombie films.
“It reminds me of a zombie.”
“Unhuman, jagged, metallic.”
“I am thinking of some horror movie.”
“[...] a bit like Darth Vader.”
Industrial Metaphors (8/29)
Sounds produced by old machines in factories.
“The sound that comes to my mind is a factory, a factory line [...] A long time a go, in the era of industrialism. This is steam engines in the 19th century England.”
“ [...] perhaps like walking around in some factory, there are a lot of mechanical sounds.”
In general, the fluid sounds were mainly described using metaphors from nature, referring to different types of organic movement such as wind through the trees, running water or waves. However, participants also described the sounds by referring to acoustic musical instruments. The nonfluid sounds, on the other hand, evoked a set of metaphors and examples not originating from nature, but rather from old machines or characters in movies, such as robots or zombies. These sounds were associated with mechanical rather than organic movement. There were also fewer references to traditional musical instruments for the nonfluid sounds. However, some participants referred to electronically synthesized or metallic musical sounds.
3.2.2 Task 2: Vocal sketching
The sounds produced during the vocal sketching varied from a few seconds to up to a minute for different participants. A couple of the participants thought that the task was difficult, while others didn’t give it a second thought. Overall, some general characteristics can be observed across participants. In general, the produced sounds go in line with the results from the interviews presented above. Examples of vocal sketches for respective condition are available as supplementary material.15
Spectrograms16 of an example of a fluid versus nonfluid vocal sketch is shown in Fig. 3. In general, the vocal sketches of fluid movements were continuous and somewhat softer than the sketches of nonfluid movements. In addition, they were sometimes characterized by a lower pitch. These sketches used uninterrupted air flows, whistling, whooshing and breathy, whispering sounds, and tended towards darker timbres. The vocal sketches of nonfluid movements were less homogeneous, generally louder, strained, and included vocal creaks and grunts. They often contained bursts of sounds, squeaks, fricatives or short series of completely separated staccato sounds. Overall, they contained much more high frequency energy and noise than vocal sketches of fluid movements. Some participants stated that it was difficult to produce the kind of sounds that they wanted for the nonfluid movement. One participant said: “It feels like I can’t produce the sounds that I am associating with this [movement] using only my mouth”. Moreover, some participants clapped their hands, cracked their fingers or rubbed their hands on the table to produce friction sounds for the nonfluid vocal sketching.
As mentioned in Sect. 2.1, we also reiterated the vocal sketching experiment with 13 participants that had not previously participated in Experiment 1. The vocal sketches produced by subjects that had not participated in Experiment 1 prior to participating in Experiment 2 were similar to the vocal sketches obtained in the group that had participated in both experiments. We could conclude that performing Experiment 1 prior to Experiment 2 did not affect results to any larger extent.
Fig. 3
Spectrograms of excerpts from vocal sketches produced by participant 23
×
3.3 Results Experiment 3
Findings from Experiment 1 and 2 were confirmed in the workshop discussions. Participants mainly discussed the fluid property by referring to continuous sounds, liquid sounds, bubbles, waves and movement in water, and they also attempted to synthesize sounds with such properties. Selected fluid recordings17 included sounds of slowly moving bubbles, ocean waves near a beach, pouring sounds, a looping drone sound of a skipping record on a broken turntable, as well as synthetic bubbles of a boiling fluid. Some participants described that they perceived the dancer’s movement as being “too smooth for a human”, “artificial” or “mysterious”. Selected sounds inspired by this artificial property included for example the sound of detuned guitars.
4 Discussion
In Experiment 1, participants interactively adjusted parameters of a sonification model to correspond well versus not well with the fluid movement of a professional dancer. We observed significant differences between the fluid versus nonfluid condition, with the nonfluid condition having significantly more high frequency content in the noise source, larger bandwidth of the band-pass filters (resulting in richer and more noisy sounds), overall higher center frequencies of the band-pass filters, as well as larger frequency increase per energy unit. It should however be noted that the variance for the control parameters were considerably large for the nonfluid condition, making it difficult to make strong claims regarding general tendencies in this context. Nevertheless, the fluid recordings occupied a low to mid-register and were characterized by darker and more muffled timbres, as opposed to nonfluid recordings that were often characterized by a higher register and more prominent noise.
In Experiment 2, participants used their voices to sketch fluid versus nonfluid sounds and described the acoustic properties of these sounds in interviews. We observed different themes, or categories, that were used to describe sounds in respective condition. Fluid sounds were mainly described using metaphors from nature and acoustic instruments, whereas nonfluid sounds were described using metaphors from fiction and mechanical movement, referring in particular to unhuman characters from movies such as robots and zombies, as well as industrial metaphors. The vocal sketches of fluid movements were continuous, somewhat softer and sometimes lower pitched than the ones of nonfluid movements. Fluid vocal sketches used an uninterrupted air flow, whistling, whooshing and breathy, whispering sounds, and tended towards darker timbres. Nonfluid vocal sketches were generally louder, strained, contained vocal creaks and grunts, bursts of sounds, squeaks, fricatives or short series of completely separated staccato sounds. They generally contained much more high frequency energy and noise than the fluid sketches. To summarize these results, the ideal sound to express fluidity is continuous, calm, soft, slow, pitched, reminds of wind, water, waves or a traditional musical instrument with harmonic properties such as flute or strings, playing melodic sounds at a rather low pitch. The ideal sound to express nonfluidity is harsh, non-continuous, abrupt, choppy, dissonant, metallic or wooden, unhuman and robotic. It is industrial rather than melodic, reminds of friction and includes creaking, screeching, squeaking and cracking sounds.
One interesting finding from the vocal sketching experiment was that it appeared to be difficult for many of the participants to decouple the somewhat theatrical performance of the dancer from the high-level properties of the movement. The association to sounds were therefore very much guided by ideas and metaphors associated with the role that the dancer was enacting.
In Experiment 3, participants took part in a workshop focusing on identification of fluid sounds and production of fluid sonifications. The results from this workshop were in line with previous experiments. Following a full group discussion, participants worked in smaller groups to further explore the topic, and attempted to produce audio illustrations or simple sonification sketches themselves. The majority of the participants discussed the fluid property by referring to and working with sounds associated with water, such as sounds produced by air bubbles in pouring liquids or waves.
5 Conclusions
In this study we aimed to explore the ideal properties of a sound used to express fluid movement. Consistent findings from three different experiments indicated that sounds expressing fluidity generally occupy a lower register and has less high frequency content, as well as a generally lower bandwidth, than sounds expressing nonfluidity. The ideal sound to express fluidity is continuous, calm, slow, pitched, reminding of wind, water or an acoustic musical instrument with harmonic and melodic properties. The ideal sound to express nonfluidity is harsh, non-continuous, abrupt, dissonant, metallic or wooden, unhuman and robotic. These findings can be used as design guidelines in future projects in which the movement property fluidity is to be conveyed through sonification.
Acknowledgements
The authors would like to thank the students at the Media Technology Program at KTH Royal Institute Technology for their participation and contribution to the study.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The EGM project focused on basic research into new means of artistic expression and musical experience through de- and reconstruction of the relation between musical and bodily expression in an interactive performance environment. See https://egm.kug.ac.at/ for more information.
The echo effect was added to provide the option of temporal diffusion or smearing, in order to investigate whether distinct clarity over time could be a relevant feature in sonification of fluidity.
The experiment was completed after a re-set of the synthesis engine, but since this procedure somewhat prolonged the task and may have affected the results, this data was not included in the analysis.
In addition, we also extracted the following features for the fluid versus nonfluid vocal sketches, using MIR Toolbox: RMS, spectral flatness, brightness, spectral centroid, zero-crossing rate (ZCR), spectral roll-off and spectral spread. Due to the large variability in the data, no significant differences between the two conditions could be observed for pairwise comparison tests.
Sound examples are available as supplementary material, see sound files “12193_2018_278_MOESM45_ESM.wav”, “12193_2018_278_MOESM46_ESM.mp3”, “12193_2018_278_MOESM47_ESM.wav” and “12193_2018_278_MOESM48_ESM.wav.
