Justin Edwards
Christian Janssen
Sandy Gould
Benjamin R. Cowan
ABSTRACT
Current speech agent interactions are typically user-initiated, limiting the interactions they can deliver. Future functionality will require agents to be proactive, sometimes interrupting users. Little is known about how these spoken interruptions should be designed, especially in urgent interruption contexts. We look to inform design of proactive agent interruptions through investigating how people interrupt others engaged in complex tasks. We therefore developed a new technique to elicit human spoken interruptions of people engaged in other tasks. We found that people interrupted sooner when interruptions were urgent. Some participants used access rituals to forewarn interruptions, but most rarely used them. People balanced speed and accuracy in timing interruptions, often using cues from the task they interrupted. People also varied phrasing and delivery of interruptions to reflect urgency. We discuss how our findings can inform speech agent design and how our paradigm can help gain insight into human interruptions in new contexts.
Supplemental Material
Available for Download
pdf
- R. H. Baayen, D. J. Davidson, and D. M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59, 4 (Nov. 2008), 390–412. https://doi.org/10.1016/j.jml.2007.12.005Google Scholar
Cross Ref
- Brian P. Bailey and Shamsi T. Iqbal. 2008. Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Transactions on Computer-Human Interaction 14, 4 (Jan. 2008), 1–28. https://doi.org/10.1145/1314683.1314689Google ScholarDigital Library
- Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68, 3 (April 2013), 255–278. https://doi.org/10.1016/j.jml.2012.11.001Google ScholarCross Ref
- Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1–48. https://doi.org/10.18637/jss.v067.i01Google ScholarCross Ref
- Jelmer P. Borst, Niels A. Taatgen, and Hedderik van Rijn. 2010. The problem state: A cognitive bottleneck in multitasking.Journal of Experimental Psychology: Learning, Memory, and Cognition 36, 2(2010), 363–382. https://doi.org/10.1037/a0018106Google ScholarCross Ref
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (Dec. 2006). https://doi.org/10.1191/1478088706qp063oa Publisher: Taylor & Francis (Routledge).Google Scholar
- Duncan P. Brumby, Anna L. Cox, Jonathan Back, and Sandy J. J. Gould. 2013. Recovering from an interruption: Investigating speed-accuracy trade-offs in task resumption behavior.Journal of Experimental Psychology: Applied 19, 2 (2013), 95–107. https://doi.org/10.1037/a0032696Google Scholar
- Duncan P. Brumby, Samantha C.E. Davies, Christian P. Janssen, and Justin J. Grace. 2011. Fast or safe?: how performance objectives determine modality output choices while interacting on the move. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, Vancouver, BC, Canada, 473. https://doi.org/10.1145/1978942.1979009Google ScholarDigital Library
- Narae Cha, Auk Kim, Cheul Young Park, Soowon Kang, Minkyu Park, Jae-Gil Lee, Sangsu Lee, and Uichin Lee. 2019. “Hello There! Is Now a Good Time to Talk?’’: Opportune Moments for Proactive Interactions with Smart Speakers. 4, 3 (2019), 28.Google Scholar
- Leigh Clark, Cosmin Munteanu, Vincent Wade, Benjamin R. Cowan, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, and Christine Murad. 2019. What Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300705Google ScholarDigital Library
- Laura Dabbish, Gloria Mark, and Víctor M González. 2011. Why do i keep interrupting myself?: environment, habit and self-interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3127–3130.Google ScholarDigital Library
- Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, and Benjamin R. Cowan. 2019. Mapping Perceptions of Humanness in Intelligent Personal Assistant Interaction. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI ’19. ACM Press, Taipei, Taiwan, 1–12. https://doi.org/10.1145/3338286.3340116Google ScholarDigital Library
- Jens Edlund, Julia Bell Hirschberg, and Mattias Heldner. 2009. Pause and gap length in face-to-face interaction. Columbia University (2009). https://doi.org/10.7916/d82f7wt9Google ScholarCross Ref
- Justin Edwards, He Liu, Zhou Tianyu, Gould Gould, Sandy J. J., Leigh Clark, Philip Doyle, and Benjamin R Cowan. 2019. Multitasking with Alexa: How Using Intelligent Personal Assistants Impacts Language-based Primary Task Performance. In Proceedings of the 1st International Conference on Conversational User Interfaces. Dublin, Ireland. Accetped.Google ScholarDigital Library
- Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods 5, 1 (March 2006), 80–92. https://doi.org/10.1177/160940690600500107 Publisher: SAGE Publications Inc.Google ScholarCross Ref
- Emer Gilmartin, Marine Collery, Ketong Su, Yuyun Huang, Christy Elias, Benjamin R. Cowan, and Nick Campbell. 2017. Social talk: making conversation with people and machine. In Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents - ISIAA 2017. ACM Press, Glasgow, UK, 31–32. https://doi.org/10.1145/3139491.3139494Google ScholarDigital Library
- Erving Goffman. 1971. Relations in public: microstudies of the public order. Basic Books, New York, NY, USA. OCLC: 699515377.Google Scholar
- Wayne D. Gray and Deborah A. Boehm-Davis. 2000. Milliseconds matter: An introduction to microstrategies and to their use in describing and predicting interactive behavior.Journal of Experimental Psychology: Applied 6, 4 (2000), 322–335. https://doi.org/10.1037/1076-898X.6.4.322Google Scholar
- Elizabeth Hellier, Judy Edworthy, Ben Weedon, Kathryn Walters, and Austin Adams. 2002. The Perceived Urgency of Speech Warnings: Semantics versus Acoustics. Human Factors: The Journal of the Human Factors and Ergonomics Society 44, 1 (March 2002), 1–17. https://doi.org/10.1518/0018720024494810Google ScholarCross Ref
- William J. Horrey and Mary F. Lesch. 2009. Driver-initiated distractions: Examining strategic adaptation for in-vehicle task initiation. Accident Analysis & Prevention 41, 1 (Jan. 2009), 115–122. https://doi.org/10.1016/j.aap.2008.10.008Google ScholarCross Ref
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI ’99. ACM Press, Pittsburgh, Pennsylvania, United States, 159–166. https://doi.org/10.1145/302979.303030Google ScholarDigital Library
- Shamsi T. Iqbal and Brian P. Bailey. 2005. Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In CHI ’05 extended abstracts on Human factors in computing systems - CHI ’05. ACM Press, Portland, OR, USA, 1489. https://doi.org/10.1145/1056808.1056948Google Scholar
- Christian P. Janssen, Duncan P. Brumby, and Rae Garnett. 2012. Natural Break Points: The Influence of Priorities and Cognitive and Motor Cues on Dual-Task Interleaving. Journal of Cognitive Engineering and Decision Making 6, 1 (March 2012), 5–29. https://doi.org/10.1177/1555343411432339Google ScholarCross Ref
- Christian P. Janssen, Sandy J.J. Gould, Simon Y.W. Li, Duncan P. Brumby, and Anna L. Cox. 2015. Integrating knowledge of multitasking and interruptions across different perspectives and research methods. International Journal of Human-Computer Studies 79 (July 2015), 1–5. https://doi.org/10.1016/j.ijhcs.2015.03.002Google ScholarDigital Library
- David Kieras, David Meyer, James Ballas, and Erick Lauber. 2000. Modern Computational Perspectives on Executive Mental Processes and Cognitive Control: Where to from Here?Google Scholar
- Paul D. Krivonos and Mark L. Knapp. 1975. Initiating communication: What do you say when you say hello?Central States Speech Journal 26, 2 (June 1975), 115–125. https://doi.org/10.1080/10510977509367829Google Scholar
- Tate T. Kubose, Kathryn Bock, Gary S. Dell, Susan M. Garnsey, Arthur F. Kramer, and Jeff Mayhugh. 2006. The effects of speech production and speech comprehension on simulated driving performance. Applied Cognitive Psychology 20, 1 (Jan. 2006), 43–63. https://doi.org/10.1002/acp.1164Google ScholarCross Ref
- Jakob Landesberger, Ute Ehrlich, and Wolfgang Minker. 2020. Do the Urgent Things first! - Detecting Urgency in Spoken Utterances based on Acoustic Features. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization. ACM, Genoa Italy, 53–58. https://doi.org/10.1145/3386392.3397598Google ScholarDigital Library
- Jakob Landesberger, Ute Ehrlich, and Wolfgang Minker. 2020. ”What is it?” How to Collect Urgent Utterances using a Gamification Approach. In 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, Virtual Event DC USA, 19–22. https://doi.org/10.1145/3409251.3411713Google Scholar
- John K. Lindstedt and Wayne D. Gray. 2019. Distinguishing experts from novices by the Mind’s Hand and Mind’s Eye. Cognitive Psychology 109 (March 2019), 1–25. https://doi.org/10.1016/j.cogpsych.2018.11.003Google Scholar
- Ewa Luger and Abigail Sellen. 2016. ”Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM Press, Santa Clara, California, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288Google ScholarDigital Library
- Nikolas Martelaro, Jaime Teevan, and Shamsi T. Iqbal. 2019. An Exploration of Speech-Based Productivity Support in the Car. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300494Google ScholarDigital Library
- Daniel C. McFarlane. 2002. Comparison of Four Primary Methods for Coordinating the Interruption of People in Human-Computer Interaction. Human–Computer Interaction 17, 1 (March 2002), 63–139. https://doi.org/10.1207/S15327051HCI1701_2Google Scholar
- Lotte Meteyard and Robert A.I. Davies. 2020. Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language 112 (June 2020), 104092. https://doi.org/10.1016/j.jml.2020.104092Google ScholarCross Ref
- R Core Team. 2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/Google Scholar
- Dario D. Salvucci. 2005. A Multitasking General Executive for Compound Continuous Tasks. Cognitive Science 29, 3 (2005), 457–492. https://doi.org/10.1207/s15516709cog0000_19Google ScholarCross Ref
- Dario D. Salvucci and Niels A. Taatgen. 2008. Threaded cognition: An integrated theory of concurrent multitasking.Psychological Review 115, 1 (2008), 101–130. https://doi.org/10.1037/0033-295X.115.1.101Google Scholar
- Rob Semmens, Nikolas Martelaro, Pushyami Kaveti, Simon Stent, and Wendy Ju. 2019. Is Now A Good Time?: An Empirical Study of Vehicle-Driver Communication Timing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300867Google ScholarDigital Library
- Sarah M. Simmons, Jeff K. Caird, and Piers Steel. 2017. A meta-analysis of in-vehicle and nomadic voice-recognition system interaction and driving performance. Accident Analysis & Prevention 106 (Sept. 2017), 31–43. https://doi.org/10.1016/j.aap.2017.05.013Google Scholar
- Remo M.A. van der Heiden, Shamsi T. Iqbal, and Christian P. Janssen. 2017. Priming Drivers before Handover in Semi-Autonomous Cars. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM Press, Denver, Colorado, USA, 392–404. https://doi.org/10.1145/3025453.3025507Google ScholarDigital Library
- Martijn H. Vastenburg, David V. Keyson, and Huib de Ridder. 2008. Considerate home notification systems: a field study of acceptability of notifications in the home. Personal and Ubiquitous Computing 12, 8 (Nov. 2008), 555–566. https://doi.org/10.1007/s00779-007-0176-xGoogle ScholarDigital Library
- Priscilla N. Y. Wong, Duncan P. Brumby, Harsha Vardhan Ramesh Babu, and Kota Kobayashi. 2019. Voices in Self-Driving Cars Should be Assertive to More Quickly Grab a Distracted Driver’s Attention. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications(AutomotiveUI ’19). Association for Computing Machinery, Utrecht, Netherlands, 165–176. https://doi.org/10.1145/3342197.3344535Google ScholarDigital Library
- Yunhan Wu, Justin Edwards, Orla Cooney, Anna Bleakley, Philip R. Doyle, Leigh Clark, Daniel Rough, and Benjamin R. Cowan. 2020. Mental Workload and Language Production in Non-Native Speaker IPA Interaction. In Proceedings of the 2nd Conference on Conversational User Interfaces. ACM, Bilbao Spain, 1–8. https://doi.org/10.1145/3405755.3406118Google ScholarDigital Library
- Yunhan Wu, Daniel Rough, Anna Bleakley, Justin Edwards, Orla Cooney, Philip R. Doyle, Leigh Clark, and Benjamin R. Cowan. 2020. See What I’m Saying? Comparing Intelligent Personal Assistant Use for Native and Non-Native Language Speakers. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, Oldenburg Germany, 1–9. https://doi.org/10.1145/3379503.3403563Google ScholarDigital Library
Recommendations
A diary study of task switching and interruptions
CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe report on a diary study of the activities of information workers aimed at characterizing how people interleave multiple tasks amidst interruptions. The week-long study revealed the type and complexity of activities performed, the nature of the ...
Not Merely Deemed as Distraction: Investigating Smartphone Users’ Motivations for Notification-Interaction
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsNotifications are commonly considered a distraction when they arrive during a task, and consequently, prior research has consistently sought effective ways of deferring their arrival until task transitions. However, many smartphone users still interact ...
Investigating interruptions in the context of computerised cognitive testing for older adults
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsInterruptions in the home pose a threat to the validity of self-administered computerised cognitive testing. We report the findings of a laboratory experiment investigating the effects of increased interruption workload demand on older adults' ...
Comments