Top

Journal on Multimodal User Interfaces

Published in:

30-08-2023 | Original Paper

Perceptually congruent sonification of auditory line charts

Authors: Joe Fitzpatrick, Flaithri Neff

Published in: Journal on Multimodal User Interfaces | Issue 4/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In recent years, there has been increasingly more discussion about sonification design, evaluation strategies, and the future of sonification in general. Concerns relate to the efficacy and usability of sonification methodologies and the lack of evidence supporting design choices. In sonification publications to date, there is little discussion on how the principles of psychoacoustics or auditory scene analysis (ASA) might be used to improve design methodologies. Proposed in this article is the perceptually congruent sonification (PerCS) model that highlights the links between sonification mappings and key perceptual factors that are most likely to affect the interpretability of sonified trends. The model explores the perceptual constraints associated with design choices such as presentation rate, tonality, space, and frequency range. The model highlights the dependencies these mappings have on our perception of frequency, timbre, loudness, and space: the core determinants of stream segregation (a key phenomenon of ASA). This model-centred approach is used to identify perceptually congruent design choices for auditory line charts, a popular and accessible form of parameter-mapping sonification. This article focuses predominately on identifying an efficient and perceptually congruent frequency range for auditory line charts, as there are no contemporary studies on this design choice, with auditory graphing tools leaving little guidance on what might be appropriate. Several psychoacoustically justified ranges are proposed by the PerCS research and are contrasted in a user study to determine the performance of narrow vs wide frequency ranges. Results show that a shorter range between 400 and 1400 Hz performs just as well as a wider range between 65 Hz and 1480 Hz.

previous article An interdisciplinary journey towards an aesthetics of sonification experience

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Hermann T, Hunt A, Neuhoff JG (eds) (2011) The sonification handbook. Logos Publishing House, Germany. http://sonification.de/handbook

Ziemer T, Schultheis H (2022) “Both rudimentary visualization and prototypical sonification can serve as a benchmark to evaluate new sonification designs.” In: Proceedings of the 17th international audio mostly conference. New York, USA: association for computing machinery, p. 24-31. [Online]. Available: https://doi.org/10.1145/3561212.3561228

Worrall D (2019) “The sonipy framework: getting started," in sonification design: from data to intelligible soundfields. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-030-01497-1_6CrossRef

Andreopoulou A, Goudarzi V (2021) “Sonification first: the role of icad in the advancement of sonification-related research.” In: Proceedings of the 26th international conference on auditory display, virtual conference June 25–28, [Online]. Available: https://doi.org/10.21785/icad2021.031

Filella NB (2021) Creating and evaluating aesthetics in sonification.” In: Doing research in sound design, M Filimowicz Ed. Focal Press, [Online]. Available: https://doi.org/10.4324/9780429356360

Ferguson J, Brewster SA (2018) Investigating perceptual congruence between data and display dimensions in sonification. ” In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp. 1–9. [Online]. Available: https://doi.org/10.1145/3173574.3174185

Ferguson J, Brewster S (2019) Evaluating the magnitude estimation approach for designing sonification mapping topologies.” In: Proceedings of the 15th international conference on auditory display, newcastle upon tyne, UK, [Online]. Available: https://doi.org/10.21785/icad2019.047

Ziemer T, Schultheis H (2019) Three orthogonal dimensions for psychoacoustic sonification.” arXiv preprint arXiv:1912.00766. [Online]. Available: https://doi.org/10.48550/arXiv.1912.00766

Fitzpatrick J (2023) Percs miro board.” 2023, accessed May 3. [Online]. Available: https://miro.com/app/board/uXjVMPmVeTE=/?share_link_id=686195811894

10.

Nesbitt KV (2006) Modelling human perception to leverage the reuse of concepts across the multi-sensory design space.” In: Proceedings of the 3rd Asia-Pacific conference on conceptual modelling - Volume 53, ser. APCCM ’06. AUS: Australian computer society, Inc., p. 65-74. [Online]. Available: https://doi.org/10.5555/1151855.1151863

11.

Chimir I, Horney M (2001) New visions of old models. In: Beynon M, Nehaniv CL, Dautenhahn K (eds) Cognitive Technology: instruments of mind. Springer, Berlin. https://doi.org/10.1007/3-540-44617-6_16CrossRefMATH

12.

Chimir IA, Abu-Dawwas WA, Horney MA (2005) Object-oriented modeling of attentional systems.” In: Proceedings of the 2nd international refereed conference on information technology “human computer interaction” (ICIT 2005), pp. 3–5. [Online]. Available: http://icit.zuj.edu.jo/icit05/2005/Object%20Oriented/127.pdf

13.

Walker BN, Mauney LM (2010) Universal design of auditory graphs: a comparison of sonification mappings for visually impaired and sighted listeners’’. ACM Trans Access Comput 10(1145/1714458):1714459

14.

Grond F, Berger J (2011) Parameter-mapping sonification.” In: The sonification handbook. Hermann T, Hunt A, and J G Neuhoff, Eds. Berlin: Logos Verlag, pp. 363–397. [Online]. Available: http://sonification.de/handbook

15.

IEC (2006) Medical electrical equipment – part 1-8: general requirements for basic safety and essential performance – collateral standard: general requirements, tests and guidance for alarm systems in medical electrical equipment and medical electrical systems.” ISO Standard, [Online]. Available: https://www.iso.org/standard/41986.html

16.

Cantrell SJ, Walker BN, Moseng Ø (2021) Highcharts sonification studio: an online, open-source, extensible, and accessible data sonification tool.” In: Proceedings of the 26th international conference on auditory display, virtual conference, June 25-28, [Online]. Available: https://doi.org/10.21785/icad2021.005

17.

Nees MA (2018) Auditory graphs are not the “killer app’’ of sonification, but they work’’. Ergon Des 26(4):25–28. https://doi.org/10.1177/1064804618773563CrossRef

18.

Ohshiro K, Hurst A, Dubois L (2021) Making math graphs more accessible in remote learning: using sonification to introduce discontinuity in calculus.” In: Proceedings of ASSETS 2021 - 23rd international ACM SIGACCESS conference on computers and accessibility. Association for Computing Machinery, Inc, [Online]. Available: https://doi.org/10.1145/3441852.3476533

19.

Bregman AS, Campbell J (1971) Primary auditory stream segregation and perception of order in rapid sequences of tones. J Exp Psychol 89:244–249. https://doi.org/10.1037/h0031163CrossRef

20.

Fitzgibbons PJ, Gordon-Salant S (1996) Auditory temporal processing in elderly listeners. J Am Acad Audiol 7:183–189

21.

Deutsch D (2004) The octave illusion revisited again. J Experiment Psychol: Human Percep Perform 30(2):355–364. https://doi.org/10.1037/0096-1523.30.2.355CrossRef

22.

Middlebrooks JC, Onsan ZA (2012) Stream segregation with high spatial acuity. J Acoust Soc Am 132(6):3896–3911CrossRef

23.

Middlebrooks JC, Bremen P (2013) Spatial stream segregation by auditory cortical neurons. J Neurosci 33(25):10986–11001. https://doi.org/10.1523/JNEUROSCI.1065-13.2013CrossRef

24.

Pantev C, Elbert T, Ross B, Eulitz C, Terhardt E (1996) Binaural fusion and the representation of virtual pitch in the human auditory cortex’’. Hearing Res 100(1–2):164–170. https://doi.org/10.1016/0378-5955(96)00124-4CrossRef

25.

ISO, Acoustics - normal equal-loudness-level contours. 2003, 226:2003, ICS:13.140 Noise with respect to human beings. [Online]. Available: https://www.iso.org/standard/34222.html

26.

Glasberg BR, Moore BC (2002) A model of loudness applicable to time-varying sounds. J Audio Eng Soc 50(5):331–342

27.

Walker BN, Neees MA (2011) Theory of sonification. In: the sonification handbook. Hermann T, Hunt A, and Neuhoff J G, Eds. Logos Verlag Berlin, pp. 17–44. [Online]. Available: http://sonification.de/handbook

28.

Walker BN (2002) Magnitude estimation of conceptual data dimensions for use in sonification. J Exp Psychol: Appl 8(4):211. https://doi.org/10.1037//1076-898x.8.4.211CrossRef

29.

Axon L, Goldsmith M, Creese S (2018) Sonification mappings: estimating effectiveness, polarities and scaling in an online experiment. J Audio Eng Soc 66(12):1016–1032CrossRef

30.

Cairo A (2023) Twotone sonification tool.” Available at https://twotone.io/, 2019. accessed: Feb 26

31.

Nees MA, Walker BN (2007) Listener, task, and auditory graph: toward a conceptual model of auditory graph comprehension. In: Proceedings of the 13th international conference on auditory display, Montréal, Canada, June 26-29, [Online]. Available: http://hdl.handle.net/1853/50010

32.

Flowers J, Whitwer L, Grafel D, Kotan C (2001) Sonification of daily weather records: issues of perception, attention and memory in design choices. In: Proceedings of the 2001 international conference on auditory display, espoo, Finland, Jul 29–Aug 1, pp. 222–226. [Online]. Available: https://digitalcommons.unl.edu/psychfacpub/432/

33.

Garner WR (1951) The accuracy of counting repeated short tones. J Exp Psychol 41:310–316. https://doi.org/10.1037/h0059567CrossRef

34.

Cheatham P, White C (1954) Temporal numerosity: iii. auditory perception of number. J Exp Psychol 47:425–428. https://doi.org/10.1037/h0054287CrossRef

35.

Bregman A, Rudnicky A (1975) Auditory segregation: stream or streams? J Exp Psychol: Human Percep Perform 1:263–267. https://doi.org/10.1037//0096-1523.1.3.263CrossRef

36.

Bregman A, Doehring P (1984) Fusion of simultaneous tonal glides: the role of parallelness and simple frequency relations. Percep Psychophys 36(3):251–256. https://doi.org/10.3758/BF03206366CrossRef

37.

Gavin M, Jedir R, Neff F (2016) Sonification playback rates during matching tasks of visualised and sonified eeg data. In: Proceedings of the international audio systems symposium, university of York, [Online]. Available: https://www.york.ac.uk/sadie-project/IASS2016/IASS_Papers/IASS_2016_paper_18.pdf

38.

Muchnik C, Fima B, Kidd G, Sitskoorn M (1985) Minimal time interval in auditory temporal resolution. J Aud Res 25(4):239–246

39.

Krumbholz K, Patterson R, Pressnitzer D (2003) Microsecond temporal resolution in monaural hearing without spectral cues? J Acoust Soc Am 113(5):2790–2800. https://doi.org/10.1121/1.1547438CrossRef

40.

Brown M, Brewster S, Ramloll S, Burton M, Riedel B (2003) Design guidelines for audio presentation of graphs and tables. In Proceedings of the 9th international conference on auditory display, Boston, Massachusetts, Jul 6-9. [Online]. Available: http://hdl.handle.net/1853/50454

41.

Cawthon N, Moere AV (2007) The effect of aesthetic on the usability of data visualization. In: Proceedings of 11th international conference information visualization (IV ’07), pp. 637–648. [Online]. Available: https://doi.org/10.1109/IV.2007.147

42.

Lindborg P, Lenzi S, Chen M (2023) Climate data sonification and visualization: an analysis of topics, aesthetics, and characteristics in 32 recent projects. Front Psychol 13:1020102. https://doi.org/10.3389/fpsyg.2022.1020102CrossRef

43.

Schneider A (2018) Pitch and pitch perception. In: springer handbook of systematic musicology. Bader R, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 630–634. [Online]. Available: https://doi.org/10.1007/978-3-662-55004-5_31

44.

Bregman AS, Liao C, Levitan R (1990) Auditory grouping based on fundamental frequency and formant peak frequency. Can J Psychol Revue can psychol 44(3):400. https://doi.org/10.1037/h0084255CrossRef

45.

Rose MM, Moore BC (2000) Effects of frequency and level on auditory stream segregation. J Acoust Soc Am 108(3):1209–1214. https://doi.org/10.1121/1.1287708CrossRef

46.

Fishman Y, Arezzo J, Steinschneider M (2004) Auditory stream segregation in monkey auditory cortex: effects of frequency separation, presentation rate, and tone duration. J Acoust Soc Am 116(3):1656–1670. https://doi.org/10.1121/1.1778903CrossRef

47.

Elhilali M, Ma L, Micheyl C, Oxenham A, Shamma S (2009) Temporal coherence in the perceptual organization and cortical representation of auditory scenes. Neuron 61(2):317–329CrossRef

48.

Flowers JH (2005) Thirteen years of reflection on auditory graphing: promises, pitfalls, and potential new directions. In: Proceedings of the 11th international conference on auditory display, Limerick, Ireland. http://hdl.handle.net/1853/50188

49.

Barrass S (2009) Developing the practice and theory of stream based sonification. SCAN J Med Arts Cul 6(2). http://scan.net.au/scan/journal/display.php?journal_id=135

50.

Schuett JH, Walker BN (2013) Measuring comprehension in sonification tasks that have multiple data streams.” In: Proceedings of the 8th audio mostly conference, ser. AM ’13. New York, USA: Association for Computing Machinery. [Online]. Available: https://doi.org/10.1145/2544114.2544121

51.

Oxenham AJ (2008) Pitch perception and auditory stream segregation: implications for hearing loss and cochlear implants. Trends Amplific 12(4):316–331CrossRef

52.

Bernstein JG, Oxenham A (2003) Pitch discrimination of diotic and dichotic tone complexes: harmonic resolvability or harmonic number? J Acoust Soc Am 113(6):3323–3334. https://doi.org/10.1121/1.1572146CrossRef

53.

Fitzpatrick J, Neff F (2018) Stream segregation: utilizing harmonic variance in auditory graphs. In: proceedings of SMC 18, Limassol, Cyprus. [Online]. Available: https://doi.org/10.5281/zenodo.1422501

54.

Bonebright TL, Nees MA, Connerley TT, McCain GR (2001) Testing the effectiveness of sonified graphs for education: a programmatic research project. In: Proceedings of the 2001 international conference on auditory display, Espoo, Finland, July 29-August 1. [Online]. Available: http://hdl.handle.net/1853/50654

55.

Song HJ, Beilharz K, Cabrera D (2007) “Evaluation of spatial presentation in sonification for identifying concurrent audio streams. In: Proceedings of the 13th international conference on auditory display, montreal, Canada, Jun 26–29. [Online]. Available: http://hdl.handle.net/1853/49998

56.

Westerhausen R, Moosmann M, Alho K, Medvedev S, Hämäläinen H, Hugdahl K (2009) Top-down and bottom-up interaction: manipulating the dichotic listening ear advantage. Brain Res 1250:183–189. https://doi.org/10.1016/j.brainres.2008.10.070CrossRef

57.

Bless JJ, Westerhausen R, Torkildsen JvK, Gudmundsen M, Kompus K, Hugdahl K (2015) Laterality across languages: Results from a global dichotic listening study using a smartphone application. Laterality: Asymmetries of Body, Brain and Cognition 20(4): 434–452

58.

Morell RJ, Brewer CC, Ge D, Snieder H, Zalewski CK, King KA, Drayna D, Friedman TB (2007) A twin study of auditory processing indicates that dichotic listening ability is a strongly heritable trait. Human Genet 122:103–111. https://doi.org/10.1007/s00439-007-0384-5CrossRef

59.

Deutsch D (1981) The octave illusion and auditory perceptual integration. Hear Res Theor 1(99):142

60.

Tanaka K, Ross B, Kuriki S, Harashima T, Obuchi C, Okamoto H (2021) Neurophysiological evaluation of right-ear advantage during dichotic listening. Front Psychol 12:696263CrossRef

61.

Chambers C, Pressnitzer D (2014) Perceptual hysteresis in the judgment of auditory pitch shift. Attention, Percep Psychophys 76:1271–1279. https://doi.org/10.3758/s13414-014-0676-5CrossRef

62.

Brown M, Brewster S (2003) Drawing by ear: Interpreting sonified line graphs. In: Proceedings of the 2003 international conference on auditory display, Boston, USA, 6–9 . [Online]. Available: http://hdl.handle.net/1853/50453

63.

Walker BN, Kramer G (2004) Ecological psychoacoustics and auditory displays: Hearing, grouping, and meaning making. In: Ecological psychoacoustics, pp. 150–175. [Online]. Available: https://brill.com/display/title/23213

64.

Fastl H, Zwicker E (2007) Pitch and pitch strength. In: Psychoacoustics: facts and models. Springer, Berlin pp. 111–148. [Online]. Available: https://doi.org/10.1007/978-3-540-68888-4_5

65.

Ziemer T (2020) Psychoacoustics. In: Psychoacoustic music sound field synthesis: creating spaciousness for composition, performance, acoustics and perception. Springer, Cham, pp. 65–110. [Online]. Available: https://doi.org/10.1007/978-3-030-23033-3_4

66.

Sek A, Moore BC (1995) Frequency discrimination as a function of frequency, measured in several ways. J Acoust Soc Am 97(4):2479–2486. https://doi.org/10.1121/1.411968CrossRef

67.

Delhommeau K, Micheyl C, Jouvent R (2005) Generalization of frequency discrimination learning across frequencies and ears: implications for underlying neural mechanisms in humans. J Assoc Res Otolaryngol 6:171–179. https://doi.org/10.1007/s10162-005-5055-4CrossRef

68.

Suzuki Y, Takeshima H (2004) Equal-loudness-level contours for pure tones. J Acoust Soc Am 116(2):918–933. https://doi.org/10.1121/1.1763601CrossRef

69.

SuperCollider, Ampcompa supercollider help source 3.12,” Available at https://doc.sccode.org/Classes/AmpCompA.html, 2023, (Accessed: Feb 26, 2023)

70.

Sauro J (2012) 10 things to know about the single ease question (seq),” Available at https://measuringu.com/seq10/, (Accessed: Feb 26, 2023)

71.

Brooke J (1996) Sus-a quick and dirty usability scale. Usabil Eval Indus 189:194

72.

Sauro J, Dumas JS (2009) Comparison of three one-question, post-task usability questionnaires. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp. 1599–1608. [Online]. Available: https://doi.org/10.1145/1518701.1518946

Title: Perceptually congruent sonification of auditory line charts
Authors: Joe Fitzpatrick
Flaithri Neff
Publication date: 30-08-2023
Publisher: Springer International Publishing
Published in: Journal on Multimodal User Interfaces / Issue 4/2023
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI: https://doi.org/10.1007/s12193-023-00413-w

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 4/2023

Introduction to the special issue on design and perception of interactive sonification

Model-based sonification based on the impulse pattern formulation

PepperOSC: enabling interactive sonification of a robot’s expressive movement

In-vehicle air gesture design: impacts of display modality and control orientation

Correction to: PepperOSC: enabling interactive sonification of a robot’s expressive movement

An interdisciplinary journey towards an aesthetics of sonification experience

Premium Partner