Q. Vera Liao
Dakuo Wang
N. Sadat Shami
Werner Geyer
ABSTRACT
Many conversational agents (CAs) are developed to answer users' questions in a specialized domain. In everyday use of CAs, user experience may extend beyond satisfying information needs to the enjoyment of conversations with CAs, some of which represent playful interactions. By studying a field deployment of a Human Resource chatbot, we report on users' interest areas in conversational interactions to inform the development of CAs. Through the lens of statistical modeling, we also highlight rich signals in conversational interactions for inferring user satisfaction with the instrumental usage and playful interactions with the agent. These signals can be utilized to develop agents that adapt functionality and interaction styles. By contrasting these signals, we shed light on the varying functions of conversational interactions. We discuss design implications for CAs, and directions for developing adaptive agents based on users' conversational behaviors.
Supplemental Material
- John Langshaw Austin. 1975. How to do things with words. Oxford university press.Google Scholar
- Niels Bernsen and Laila Dybkjær. 2004. Domain-oriented conversation with HC Andersen. Affective Dialogue Systems (2004), 142--153.Google Scholar
- Yulong Bian, Chenglei Yang, Dongdong Guan, Sa Xiao, Fengqiang Gao, Chia Shen, and Xiangxu Meng. 2016. Effects of Pedagogical Agent's Personality and Emotional Feedback Strategy on Chinese Students' Learning Experiences and Performance: A Study Based on Virtual Tai Chi Training Studio. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 433--444. Google ScholarDigital Library
- Timothy Bickmore, Laura Pfeifer, and Daniel Schulman. 2011. Relational agents improve engagement and learning in science museum visitors. In International Workshop on Intelligent Virtual Agents. Springer, 55--67. Google ScholarDigital Library
- Timothy W Bickmore, Laura M Pfeifer, and Brian W Jack. 2009. Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1265--1274. Google ScholarDigital Library
- Dan Bohus and Alexander I Rudnicky. 2003. RavenClaw: Dialog management using hierarchical task decomposition and an expectation agenda. (2003).Google Scholar
- Susan E Brennan. 1990. Conversation as direct manipulation: An iconoclastic view. (1990).Google Scholar
- Hendrik Buschmeier and Stefan Kopp. 2011. Towards conversational agents that attend to and adapt to communicative user feedback. In Intelligent Virtual Agents. Springer, 169--182. Google ScholarDigital Library
- John M Carroll and John C Thomas. 1988. Fun. ACM SIGCHI Bulletin 19, 3 (1988), 21--24. Google ScholarDigital Library
- Justine Cassell. 2001. Embodied conversational agents: representation and intelligence in user interfaces. AI magazine 22, 4 (2001), 67. Google ScholarDigital Library
- Justine Cassell and Timothy Bickmore. 2003. Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. User modeling and user-adapted interaction 13, 1 (2003), 89--132. Google ScholarDigital Library
- Praveen Chandar, Yasaman Khazaeni, Matthew Davis, Michael Muller, Marco Crasso, Q Vera Liao, N Sadat Shami, and Werner Geyer. 2017. Leveraging Conversational Systems to Assists New Hires During Onboarding. In IFIP Conference on Human-Computer Interaction. Springer, 381--391.Google ScholarCross Ref
- Mark Coeckelbergh. 2011. You, robot: on the linguistic construction of artificial others. AI & society 26, 1 (2011), 61--69. Google ScholarDigital Library
- Mark G Core and James Allen. 1997. Coding dialogs with the DAMSL annotation scheme. In AAAI fall symposium on communicative action in humans and machines, Vol. 56. Boston, MA.Google Scholar
- Doris M Dehn and Susanne Van Mulken. 2000. The impact of animated interface agents: a review of empirical research. International journal of human-computer studies 52, 1 (2000), 1--22. Google ScholarDigital Library
- Nicholas Epley, Adam Waytz, and John T Cacioppo. 2007. On seeing human: a three-factor theory of anthropomorphism. Psychological review 114, 4 (2007), 864.Google Scholar
- Dan Fletcher. 2010. The 50 Worst Inventions: Microsoft Bob. TIME. (27 May 2010).Google Scholar
- Eric Gilbert. 2012. Phrases that signal workplace hierarchy. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1037--1046. Google ScholarDigital Library
- Joakim Gustafson and Linda Bell. 2000. Speech technology on trial: Experiences from the August system. Natural Language Engineering 6, 3--4 (2000), 273--286. Google ScholarDigital Library
- John Heritage and John Maxwell Atkinson. 1984. Structures of social action: Studies in conversation analysis. Cambridge University Press.Google Scholar
- Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. 2015. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 506--516. Google ScholarDigital Library
- Diane Kelly and Jaime Teevan. 2003. Implicit feedback for inferring user preference: a bibliography. In ACM SIGIR Forum, Vol. 37. ACM, 18--28. Google ScholarDigital Library
- Alfred Kobsa and Wolfgang Wahlster. 1989. User models in dialog systems. Springer. Google ScholarDigital Library
- Stefan Kopp, Lars Gesellensetter, Nicole C Krämer, and Ipke Wachsmuth. 2005. A conversational agent as museum guide--design and evaluation of a real-world application. In International Workshop on Intelligent Virtual Agents. Springer, 329--343. Google ScholarDigital Library
- Min Kyung Lee, Sara Kiesler, and Jodi Forlizzi. 2010. Receptionist or information kiosk: How do people talk with a robot?. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 31--40. Google ScholarDigital Library
- Namseok Lee, Hochul Shin, and S Shyam Sundar. 2011. Utilitarian vs. hedonic robots: role of parasocial tendency and anthropomorphism in shaping user attitudes. In Human-Robot Interaction (HRI), 2011 6th ACM/IEEE International Conference on. IEEE, 183--184. Google ScholarDigital Library
- Q Vera Liao, Matthew Davis, Werner Geyer, Michael Muller, and N Sadat Shami. 2016. What Can You Do?: Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 264--275. Google ScholarDigital Library
- Ewa Luger and Abigail Sellen. 2016. Like Having a Really Bad PA: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5286--5297. Google ScholarDigital Library
- Chris Matyszczyk. 2012. Apple's Siri wrong 38 percent of the time in test. CNET. (30 June 2012).Google Scholar
- Tanushree Mitra and Eric Gilbert. 2014. The language that gets people to give: Phrases that predict success on kickstarter. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 49--61. Google ScholarDigital Library
- Robert J. Moore, Rafah A. Hosn, and Ashima Arora. 2016. The Machinery of Natural Conversation and the Design of Conversational Machines. In American Sociological Association annual meeting.Google Scholar
- Amy Ogan, Samantha Finkelstein, Elijah Mayfield, Claudia D'Adamo, Noboru Matsuda, and Justine Cassell. 2012a. Oh dear stacy!: social interaction, elaboration, and learning with teachable agents. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 39--48. Google ScholarDigital Library
- Amy Ogan, Samantha L Finkelstein, Erin Walker, Ryan Carlson, and Justine Cassell. 2012b. Rudeness and Rapport: Insults and Learning Gains in Peer Tutoring.. In ITS. Springer, 11--21. Google ScholarDigital Library
- Christopher Peters, Stylianos Asteriadis, and Kostas Karpouzis. 2010. Investigating shared attention with a virtual agent using a gaze-based interface. Journal on Multimodal User Interfaces 3, 1 (2010), 119--130.Google ScholarCross Ref
- Martin Porcheron, Joel E Fischer, and Sarah Sharples. 2017. Do Animals Have A ccents?: Talking with Agents in Multi-Party Conversation. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 207--219. Google ScholarDigital Library
- Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval. ACM, 117--126. Google ScholarDigital Library
- Susan Robinson, Antonio Roque, and David R Traum. 2010. Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue.. In LREC.Google Scholar
- Susan Robinson, David R Traum, Midhun Ittycheriah, and Joe Henderer. 2008. What would you Ask a conversational Agent? Observations of Human-Agent Dialogues in a Museum Setting.. In LREC.Google Scholar
- Maha Salem, Friederike Eyssel, Katharina Rohlfing, Stefan Kopp, and Frank Joublin. 2013. To err is human (-like): Effects of robot gesture on perceived anthropomorphism and likability. International Journal of Social Robotics 5, 3 (2013), 313--323.Google ScholarCross Ref
- Jyotirmay Sanghvi, Ginevra Castellano, Iolanda Leite, André Pereira, Peter W McOwan, and Ana Paiva. 2011. Automatic analysis of affective postures and body motion to detect engagement with a game companion. In Human-Robot Interaction (HRI), 2011 6th ACM/IEEE International Conference on. IEEE, 305--311. Google ScholarDigital Library
- John R Searle. 1976. A classification of illocutionary acts. Language in society 5, 01 (1976), 1--23.Google Scholar
- John R Searle, Ferenc Kiefer, and Manfred Bierwisch. 1980. Speech act theory and pragmatics. Vol. 10. Springer.Google Scholar
- Alexander Serenko. 2008. A model of user adoption of interface agents for email notification. Interacting with Computers 20, 4--5 (2008), 461--472. Google ScholarDigital Library
- Nicole Shechtman and Leonard M Horowitz. 2003. Media inequality in conversation: how people behave differently when interacting with computers and people. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 281--288. Google ScholarDigital Library
- Ben Shneiderman and Pattie Maes. 1997. Direct manipulation vs. interface agents. interactions 4, 6 (1997), 42--61. Google ScholarDigital Library
- Yang Song and Li-wei He. 2010. Optimal rare query suggestion with implicit user feedback. In Proceedings of the 19th international conference on World wide web. ACM, 901--910. Google ScholarDigital Library
- Luke Swartz. 2003. Why people hate the paperclip: Labels, appearance, behavior, and social responses to user interface agents. Ph.D. Dissertation. Stanford University Palo Alto, CA.Google Scholar
- Daniel Szafir and Bilge Mutlu. 2012. Pay attention!: designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 11--20. Google ScholarDigital Library
- Jaime Teevan, Susan T Dumais, and Eric Horvitz. 2005. Personalizing search via automated analysis of interests and activities. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 449--456. Google ScholarDigital Library
- Robert Trappl. 2013. Your Virtual Butler. Springer. Google ScholarDigital Library
- Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 271--280. Google ScholarDigital Library
- William Yang Wang, Samantha Finkelstein, Amy Ogan, Alan W Black, and Justine Cassell. 2012. Love ya, jerkface: using sparse log-linear models to build positive (and impolite) relationships with teens. In Proceedings of the 13th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, 20--29. Google ScholarDigital Library
- Jane Webster. 1988. Making computer tasks at work more playful: Implications for systems analysts and designers. In Proceedings of the ACM SIGCPR conference on Management of information systems personnel. ACM, 78--87. Google ScholarDigital Library
- Jane Webster and Joseph J Martocchio. 1992. Microcomputer playfulness: Development of a measure with workplace implications. MIS quarterly (1992), 201--226. Google ScholarDigital Library
- Joseph Weizenbaum. 1966. ELIZA computer program for the study of natural language communication between man and machine. Commun. ACM 9, 1 (1966), 36--45. Google ScholarDigital Library
- Jun Xiao, John Stasko, and Richard Catrambone. 2004. An empirical study of the effect of agent competence on user performance and perception. In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 1. IEEE Computer Society, 178--185. Google ScholarDigital Library
- Jun Xiao, John Stasko, and Richard Catrambone. 2007. The role of choice and customization on users' interaction with embodied conversational agents: effects on perception and performance. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 1293--1302. Google ScholarDigital Library
- ChengXiang Zhai and John Lafferty. 2006. A risk minimization framework for information retrieval. Information Processing & Management 42, 1 (2006), 31--55. Google ScholarDigital Library
- Ran Zhao, Tanmay Sinha, Alan W Black, and Justine Cassell. 2016. Automatic Recognition of Conversational Strategies in the Service of a Socially-Aware Dialog System.Google Scholar
Index Terms
- All Work and No Play?
Recommendations
Evaluating and Informing the Design of Chatbots
DIS '18: Proceedings of the 2018 Designing Interactive Systems ConferenceText messaging-based conversational agents (CAs), popularly called chatbots, received significant attention in the last two years. However, chatbots are still in their nascent stage: They have a low penetration rate as 84% of the Internet users have not ...
Conversational Agents Replying with a Manzai-style Joke
OzCHI '21: Proceedings of the 33rd Australian Conference on Human-Computer InteractionAutomated conversational agents are becoming popular in various everyday contexts. In order to fulfill a more important role in human society, people would need to feel a sense of familiarity with such agents. To achieve this, we focus on humor, which ...
Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings – Experiences from a Field Study
Chatbot Research and DesignAbstractIn this paper, we analyze the use of small talk conversations based on a dialogue analysis of a long-term field study in which university students regularly interacted with a chatbot during a 3-month period of time in an educational setting. In ...
Comments