Top

Journal on Multimodal User Interfaces

Published in:

19-03-2019 | Original Paper

Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction

Authors: Michael Braun, Nora Broy, Bastian Pfleging, Florian Alt

Published in: Journal on Multimodal User Interfaces | Issue 2/2019

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

In this paper we investigate how natural language interfaces can be integrated with cars in a way such that their influence on driving performance is being minimized. In particular, we focus on how speech-based interaction can be supported through a visualization of the conversation. Our work is motivated by the fact that speech interfaces (like Alexa, Siri, Cortana, etc.) are increasingly finding their way into our everyday life. We expect such interfaces to become commonplace in vehicles in the future. Cars are a challenging environment, since speech interaction here is a secondary task that should not negatively affect the primary task, that is driving. At the outset of our work, we identify the design space for such interfaces. We then compare different visualization concepts in a driving simulator study with 64 participants. Our results yield that (1) text summaries support drivers in recalling information and enhances user experience but can also increase distraction, (2) the use of keywords minimizes cognitive load and influence on driving performance, and (3) the use of icons increases the attractiveness of the interface.

previous article Design and evaluation of auditory-supported air gesture controls in vehicles

next article Investigating the effects of modality switches on driver distraction and interaction efficiency in the car

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

https://www.apple.com/ios/carplay/, last access: 2017-08-12.

https://www.android.com/auto/, last access: 2017-08-12.

https://itunes.apple.com/de/app/remote-s-for-tesla/id991623777, last access: 2017-08-12.

https://www.theguardian.com/technology/2017/feb/07/amazon-alexa-car-logitech-zerotouch-voice-services-assistant, last access: 2017-08-12.

https://azure.microsoft.com/de-de/services/cognitive-services/speech/,last access: 2017-05-14.

https://www.ibm.com/watson/developercloud/text-to-speech.html, last access: 2017-05-14.

http://www.ergoneers.com/eye-tracking/dikablis-glasses/, last access: 2017-05-13.

Asif A, Boll S (2010) Where to turn my car?: comparison of a tactile display and a conventional car navigation system under high load condition. In: Proceedings of the 2nd international conference on automotive user interfaces and interactive vehicular applications, Automotiveui ’10. ACM, New York, pp 64–71. https://doi.org/10.1145/1969773.1969786

Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77. https://doi.org/10.1109/MPRV.2006.18 CrossRef

Beanland V, Fitzharris M, Young KL, Lenné MG (2013) Driver inattention and driver distraction in serious casualty crashes: Data from the australian national crash in-depth study. Accid Anal Prev 54:99–107. https://doi.org/10.1016/j.aap.2012.12.043 http://www.sciencedirect.com/science/article/pii/S000145751300047X

Bellegarda JR (2014) Spoken language understanding for natural interaction: the siri experience. In: Natural interaction with robots, knowbots and smartphones: putting spoken dialog systems into practice. Springer, New York, pp 3–14. https://doi.org/10.1007/978-1-4614-8280-2

Braun M, Pfleging B, Alt F (2018) A survey to understand emotional situations on the road and what they mean for affective automotive uis. Multimodal Technol. Interact 2(4). https://doi.org/10.3390/mti2040075 http://www.mdpi.com/2414-4088/2/4/75

Braun M, Pfleging B, Broy N, Alt F (2017) A design space for conversational in-vehicle information systems. In: Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, MobileHCI ’17. ACM, New York

Braun M, Völkel ST, Wiegand G, Puls T, Steidl D, WeißD, Alt F (2018) The smile is the new like: controlling music with facial expressions to minimize driver distraction. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia, MUM ’18. ACM, New York. https://doi.org/10.1145/3282894.3289729

Bubb H (2003) Fahrerassistenz—primär ein beitrag zum komfort oder fuer die sicherheit? In: Der Fahrer im 21. Jahrhundert: Anforderungen, Anwendungen, Aspekte für Mensch-Maschine-Systeme; Tagung Braunschweig, 2. und 3. Juni 2003, VDI-Berichte 1768. VDI-Verlag, Düsseldorf, pp 25–44

Caird JK, Willness CR, Steel P, Scialfa C (2008) A meta-analysis of the effects of cell phones on driver performance. Accid Anal Prev 40(4):1282–1293CrossRef

10.

Commission of the European Communities (2007) Commission recommendation of 22 december 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface (2007/78/ec). Off J Eur Union 50(L32): 200–241. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2007.032.01.0200.01.ENG

11.

Cooper JM, Strayer DL (2015) Mental workload of voice interactions with 6 real-world driver interfaces. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 2–8. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/002.pdf

12.

Department of Transportation, National Highway Traffic Safety Administration (2013) Visual-manual nhtsa driver distraction guidelines for in-vehicle electronic device. Federal Regist 78(81), pp 24, 817–24, 890. https://federalregister.gov/a/2013-09883

13.

Ehsani J, Li K, Simons-Morton BG (2015) Teenage drivers portable electronic device use while driving. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 219–225. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/034.pdf

14.

Engström J, Johansson E, Östlund J (2005) Effects of visual and cognitive load in real and simulated motorway driving. Transp Res Part F: Traffic Psychol Behav 8(2):97–120. https://doi.org/10.1016/j.trf.2005.04.012 http://www.sciencedirect.com/science/article/pii/S1369847805000185. The relationship between distraction and driving performance: towards a test regime for in-vehicle information systems In-vehicle information systems

15.

Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp 1459–1462

16.

Foley JP, Young R, Angell L, Domeyer JE (2013) Towards operationalizing driver distraction. In: Proceedings of the seventh international driving symposium on human factors in driver assessment, training, and vehicle design, pp 57–63. http://drivingassessment.uiowa.edu/sites/default/files/DA2013/Papers/010_Foley_0.pdf

17.

Geiger M, Zobl M, Bengler K, Lang M (2001) Intermodal differences in distraction effects while controlling automotive user interfaces. In: Proceedings of HCI international 2001 volume 1: usability evaluation and interface design: cognitive engineering, intelligent agents and virtual reality, HCII ’01. Lawrence Erlbaum, Hillsdale

18.

Green P (2013) Standard definitions for driving measures and statistics: overview and status of recommended practice j2944. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 184–191. https://doi.org/10.1145/2516540.2516542

19.

Green PA (2008) Driver interface/hmi standards to minimize driver distraction/overload. In: Convergence 2008 conference proceedings. Society of Automotive Engineers, SAE International, Warrendale (2008)

20.

Group ADFTW (2006) Statement of principles, criteria and verification procedures on driver interactions with advanced in- vehicle information and communication systems. Technical report, Alliance of Automobile Manufacturers

21.

Hackenberg L, Bongartz S, Härtle C, Leiber P, Baumgarten T, Sison JA (2013) International evaluation of nlu benefits in the domain of in-vehicle speech dialog systems. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 114–120. https://doi.org/10.1145/2516540.2516553

22.

Haeuslschmid R, Pfleging B, Alt F (2016) A design space to support the development of windshield applications for the car. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 5076–5091

23.

Harvey C, Stanton NA, Pickering CA, McDonald M, Zheng P (2011) In-vehicle information systems to meet the needs of drivers. Int J Hum Comp Interact 27(6):505–522. https://doi.org/10.1080/10447318.2011.555296 CrossRef

24.

Hassenzahl M, Koller F, Burmester M (2008) Der user experience (ux) auf der spur: zum einsatz von www.attrakdiff.de. In: Brau H, Diefenbach S, Hassenzahl M, Koller F, Peissner M, Röse K (eds) Tagungsband UP08. Fraunhofer Verlag, Stuttgart, pp 78–82

25.

Heenan A, Herdman CM, Brown MS, Robert N (2014) Effects of conversation on situation awareness and working memory in simulated driving. Hum Factors 56(6):1077–1092. https://doi.org/10.1177/0018720813519265 PMID: 25277018CrossRef

26.

Hernandez J, McDuff D, Benavides X, Amores J, Maes P, Picard R (2014) Autoemotive: bringing empathy to the driving experience to manage stress. In: Proceedings of the 2014 companion publication on designing interactive systems, DIS Companion ’14. ACM, New York, pp 53–56. https://doi.org/10.1145/2598784.2602780

27.

ITFG on Driver Distraction (2013) Report on user interface requirements for automotive applications. Technical report, International Telecommunication Union

28.

Jakus G, Dicke C, Sodnik J (2015) A user study of auditory, head-up and multi-modal displays in vehicles. Appl Ergon 46, Part A:184–192. https://doi.org/10.1016/j.apergo.2014.08.008 http://www.sciencedirect.com/science/article/pii/S0003687014001471

29.

Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8):724–736. https://doi.org/10.1016/j.ijhcs.2007.02.003 http://www.sciencedirect.com/science/article/pii/S1071581907000377

30.

Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D (2014) Better driving and recall when in-car information presentation uses situationally-aware incremental speech output generation. In: Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’14. ACM, New York, pp 7:1–7:7. https://doi.org/10.1145/2667317.2667332

31.

Kern D, Marshall P, Schmidt A (2010) Gazemarks: gaze-based visual placeholders to ease attention switching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2093–2102. ACM

32.

Kern D, Schmidt A (2009) Design space for driver-based automotive user interfaces. In: Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’09. ACM, New York, pp 3–10. https://doi.org/10.1145/1620509.1620511

33.

Kim S, Dey AK (2009) Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. ACM, New York, pp 133–142. https://doi.org/10.1145/1518701.1518724

34.

Kountouriotis GK, Wilkie RM, Gardner PH, Merat N (2015) Looking and thinking when driving: the impact of gaze and cognitive load on steering. Transp Res Part F: Traffic Psychol Behav 34:108–121. https://doi.org/10.1016/j.trf.2015.07.012 http://www.sciencedirect.com/science/article/pii/S1369847815001163

35.

Laberge-Nadeau C, Maag U, Bellavance F, Lapierre SD, Desjardins D, Messier S, Saidi A (2003) Wireless telephones and the risk of road crashes. Accid Anal Prev 35(5):649–660CrossRef

36.

Lamble D, Kauranen T, Laakso M, Summala H (1999) Cognitive load and detection thresholds in car following situations: safety implications for using mobile (cellular) telephones while driving. Accid Anal Prev 31(6):617–623CrossRef

37.

Large DR, Burnett G, Anyasodo B, Skrypchuk L (2016) Assessing cognitive demand during natural language interactions with a digital driving assistant. In: Proceedings of the 8th international conference on automotive user interfaces and interactive vehicular applications, Automotive’UI 16. ACM, New York, pp 67–74. https://doi.org/10.1145/3003715.3005408

38.

Lo VEW, Green PA (2013) Development and evaluation of automotive speech interfaces: Useful information from the human factors and the related literature. Int J Veh Technol 2013:924,170:1–924,170:13. https://doi.org/10.1155/2013/924170

39.

Luger E, Sellen A (2016) “like having a really bad pa”: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. ACM, New York, pp 5286–5297. https://doi.org/10.1145/2858036.2858288

40.

McEvoy SP, Stevenson MR, McCartt AT, Woodward M, Haworth C, Palamara P, Cercarelli R (2005) Role of mobile phones in motor vehicle crashes resulting in hospital attendance: a case-crossover study. Bmj 331(7514):428CrossRef

41.

Müller C, Weinberg G (2011) Multimodal input in the car, today and tomorrow. IEEE MultiMed 18(1):98–103. https://doi.org/10.1109/MMUL.2011.14 CrossRef

42.

Nafari M, Weaver C (2013) Augmenting visualization with natural language translation of interaction: a usability study. Comput Graph Forum 32(3pt4):391–400. https://doi.org/10.1111/cgf.12126 CrossRef

43.

Neßelrath R, Feld M (2013) Towards a cognitive load ready multimodal dialogue system for in-vehicle human-machine interaction. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 49–52. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=49

44.

Oh A, Fox H, Van Kleek M, Adler A, Gajos K, Morency LP, Darrell T (2002) Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In: CHI ’02 extended abstracts on human factors in computing systems, CHI EA ’02. ACM, New York, pp 650–651. https://doi.org/10.1145/506443.506528

45.

Paas F, Tuovinen JE, Tabbers H, Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 38(1):63–71. https://doi.org/10.1207/S15326985EP3801_8 CrossRef

46.

Pauzié A (2008) A method to assess the driver mental workload: the driving activity load index (dali). Intell Transp Syst IET 2(4):315–322. https://doi.org/10.1049/iet-its:20080023 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4706518&tag=1

47.

Petzoldt T, Bellem H, Krems JF (2014) The critical tracking task: a potentially useful method to assess driver distraction? Hum Factors 56(4):789–808. https://doi.org/10.1177/0018720813501864 CrossRef

48.

Pieraccini R, Dayanidhi K, Bloom J, Dahan JG, Phillips M, Goodman BR, Prasad KV (2004) Multimodal conversational systems for automobiles. Commun. ACM 47(1):47–49. https://doi.org/10.1145/962081.962104 CrossRef

49.

Ranney TA, Garrott WR, Goodman MJ (2001) Nhtsa driver distraction research: Past, present, and future. In: Proceedings of the 17th international technical conference on the enhanced safety of vehicles. u.s. department of transportation, National Highway Traffic Safety Administration (2001). http://www-nrd.nhtsa.dot.gov/pdf/esv/esv17/proceed/00177.pdf

50.

Redelmeier DA, Tibshirani RJ (1997) Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med 336(7):453–458CrossRef

51.

Reimer B, Mehler B (2013) The effects of a production level “voice-command” interface on driver behavior: summary findings on reported workload, physiology, visual attention, and driving performance. Technical Report White Paper 2013-18A, MIT AgeLab—Massachusetts Institute of Technology (2013). http://agelab.mit.edu/files/MIT_AgeLab_White_Paper_2013-18A_(Voice_Interfaces).pdf

52.

Rümelin S, Siegl P, Butz A (2013) Could you please... investigating cooperation in the car. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 61–64. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=63

53.

SAE International (2013) Operational definitions of driving performance measures and statistics. Technical Report SAE J2944, SAE International. http://standards.sae.org/wip/j2944/. Proposed Draft

54.

Strayer DL, Cooper JM, Turrill J, Coleman JR, Hopman RJ (2015) The smartphone and the driver’s cognitive workload: a comparison of apple, google, and microsoft’s intelligent personal assistants. Technical report, AAA Foundation for Traffic Safety, Washington, DC. https://www.aaafoundation.org/sites/default/files/strayerIIIa_FINALREPORT.pdf

55.

Tsimhoni O, Smith D, Green P (2004) Address entry while driving: speech recognition versus a touch-screen keyboard. Hum Factors 46(4):600–610. https://doi.org/10.1518/hfes.46.4.600.56813 PMID: 15709323CrossRef

56.

Vollrath M, Maciej J, Niederée U (2008) In-car distraction study—final report. Technical report, Kognitions-und Ingenieurpsychologie, Technische Universität Carolo-Wilhelmina zu Braunschweig, Braunschweig

57.

Ward N (2006) Non-lexical conversational sounds in American english. Pragmat Cognit 14(1):129–182CrossRef

58.

Weinberg G, Harsham B, Medenica Z (2011) Investigating huds for the presentation of choice lists in car navigation systems. In: Proceedings of the sixth international driving symposium on human factors in driver assessment, training and vehicle design, DA ’11, pp 195–202

59.

White MP, Eiser JR, Harris PR (2004) Risk perceptions of mobile phone use while driving. Risk Anal 24(2):323–334. https://doi.org/10.1111/j.0272-4332.2004.00434.x CrossRef

60.

Yan B, Weng F, Feng Z, Ratiu F, Raya M, Meng Y, Varges S, Purver M, Lien A, Scheideck T, Raghunathan B, Lin F, Mishra R, Lathrop B, Zhang Z, Bratt H, Peters S (2007) A conversational in-car dialog system. In: Proceedings of human language technologies: the annual conference of the North American chapter of the association for computational linguistics: demonstrations, NAACL-demonstrations ’07. Association for Computational Linguistics, Stroudsburg, pp 23–24. http://dl.acm.org/citation.cfm?id=1614164.1614176

Title: Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction
Authors: Michael Braun
Nora Broy
Bastian Pfleging
Florian Alt
Publication date: 19-03-2019
Publisher: Springer International Publishing
Published in: Journal on Multimodal User Interfaces / Issue 2/2019
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI: https://doi.org/10.1007/s12193-019-00301-2

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 2/2019

Investigating the effects of modality switches on driver distraction and interaction efficiency in the car

Multimodal interaction in automotive applications

Investigating driver gaze behavior during lane changes using two visual cues: ambient light and focal icons

Anthropomorphising driver-truck interaction: a study on the current state of research and the introduction of two innovative concepts

How do angry drivers respond to emotional music? A comprehensive perspective on assessing emotion

Design and evaluation of auditory-supported air gesture controls in vehicles

Premium Partner