Can avatars pass the Turing test? Intelligent agent perception in a 3D virtual environment

doi:10.1016/j.ijhcs.2014.08.001

International Journal of Human-Computer Studies

Volume 73, January 2015, Pages 30-36

https://doi.org/10.1016/j.ijhcs.2014.08.001 Get rights and content

Highlights

•
The first natural language Turing Test was conducted in a 3D virtual environment.
•
Seventy-eight percent of participants incorrectly judged a chatbot to be human.
•
Agency judgments were due to the quality of the AI engine and psychological factors.
•
Passage of the Turing Test involves a complex process of human–computer interaction.

Abstract

The current study involved the first natural language, modified Turing Test in a 3D virtual environment. One hundred participants were given an avatar-guided tour of a virtual clothing store housed in the 3D world of Second Life. In half of the cases, a human research assistant controlled the avatar-guide; in the other half, the avatar-guide was a visually indistinguishable virtual agent or “bot” that employed a chat engine called Discourse, a more robust variant of Artificial Intelligence Markup Language (AIML). Both participants and the human research assistant were blind to variations in the controlling agency of the guide. The results indicated that 78% of participants in the artificial intelligence condition incorrectly judged the bot to be human, significantly exceeding the 50% rate that one would expect by chance alone that is used as the criterion for passage of a modified Turing Test. An analysis of participants׳ decision-making criteria revealed that agency judgments were impacted by both the quality of the AI engine and a number of psychological and contextual factors, including the naivety of participants regarding the possible presence of an intelligent agent, the duration of the trial period, the specificity and structure of the test situation, and the anthropomorphic form and movements of the agent. Thus, passage of the Turing Test is best viewed not as the sole product of advances in artificial intelligence or the operation of psychological and contextual variables, but as a complex process of human–computer interaction.

Section snippets

The Turing Test: definition and significance

In the now classic paper, “Computing Machinery and Intelligence,” Turing (1950) proposed a behavioral approach to determining artificial or machine intelligence. An intelligent agent, or “bot,” is said to have passed the Turing Test when it is mistaken by a judge to be a human intelligence in more than 30% of the cases after five minutes of questioning. Procedurally, in what has become known as the “Standard Turning Test” (Sterrett, 2000), a series of judges engage in chat interactions with two

Participants

One hundred participants were recruited via announcements in the Second Life Events Calendar and in high usage regions of the virtual world, notices sent out by heads of large groups representing major constituencies in Second Life, and word-of-mouth communication. Each method of recruitment offered potential participants the opportunity to go on a guided tour of a clothing store located in Second Life and earn 500 Lindens (virtual currency worth approximately $2 US) for completing a brief

Results and discussion

The current study involved the first natural language, modified Turing Test in a 3D virtual environment. As depicted in Table 1, the results indicate that 78% of participants in the artificial intelligence condition (39 of 50) incorrectly judged a bot to be human, while only 10% of participants in the human condition (5 of 50) mistook the human controlled guide for a machine intelligence, reflecting a highly significant difference in the error rate for the bot and human guide conditions, χ² (1,

References (20)

A.P. Saygin et al.
Pragmatics in human–computer conversation
J. Pragmat.
(2002)
Aron, Jacob, 2011. Software Tricks People Into Thinking it is Human. New Scientist (retrieved...
Burden, D., 2008. Artificial intelligence 2008. In: Paper Presented at the 28th SGAI International Conference on...
Burden, D., 2009. Emotionally responsive robotic avatars as characters in virtual worlds. In: Proceedings of the 2009...
Daden Limited, 2010. The future of virtual worlds. Unpublished...
R. Descartes
Discourse on Method and Meditations (Laurence J. Lafleur, trans.)
(1637)
J. Dionisio et al.
3D virtual worlds and the metaverse: current status and future possibilities
ACM Comput. Surv.
(2013)
N. Epley et al.
On seeing human: a three-factor theory of anthropomorphism
Psychol. Rev.
(2007)
R. French
Subcognition and the limits of the Turing test
Mind.
(1990)
R. Gilbert et al.
The distributed self: virtual worlds and the future of human identity

There are more references available in the full text version of this article.

Cited by (25)

Imagining future digital assistants at work: A study of task management needs
2022, International Journal of Human Computer Studies
Citation Excerpt :
The development of DAs can be traced back to the 1980s and 1990s, when products such as Apple’s Knowledge Navigator and AT&T’s PersonaLinks were developed to support user productivity (Lopatovska and Williams, 2018). Contemporary, or next-generation (Meurisch et al., 2017), DAs such as Cortana, Siri, and Alexa are often referred to as Conversational Agents (CAs) because interaction is generally via speech or text (Motalebi et al., 2019; Feng and Buxmann, 2020; Meyer von Wolff et al., 2020; Xiao et al., 2020; Pradhan et al., 2020; ter Stal et al., 2020; Gilbert and Forney, 2015; Zamani et al., 2022). The CA then responds on the basis of predefined commands (such as “send text message”) (McGregor and Tang, 2017 p. 2208).
Digital Assistants (DAs) can support workers in the workplace and beyond. However, target user needs are not fully understood, and the functions that workers would ideally want a DA to support require further study. A richer understanding of worker needs could help inform the design of future DAs. We investigate user needs of future workplace DAs using data from a user study of 40 workers over a four-week period. Our qualitative analysis confirms existing research and generates new insight on the role of DAs in managing people’s time, tasks, and information. Placing these insights in relation to quantitative analysis of self-reported task data, we highlight how different occupation roles require DAs to take varied approaches to these domains and the effect of task characteristics on the imagined features. Our findings have implications for the design of future DAs in work settings and we offer some recommendations for reduction to practice.
Almost human? A comparative case study on the social media presence of virtual influencers
2021, International Journal of Human Computer Studies
Citation Excerpt :
The affiliation category showed the similar uncanny valley dip, with a slightly significant difference between HVI and HI (d HVI, HI = 0.20*, d HVI, AVI = 0.06, d HI, AVI = 0.13) This work contributes to the UV literature by building on past research comparing peoples’ interactions with human vs. virtual agents (Gilbert and Forney, 2015; Shechtman and Horowitz, 2003; Von der Pütten et al., 2010). However, our work goes a step further by also comparing the similarity of behaviour of virtual and human agents.
As virtual agents become prevalent in many domains, virtual influencers have gone live on social media platforms, integrating human networks and interacting with users. Building on research on human-computer interactions, the Uncanny Valley hypothesis, and Computers Are Social Actors paradigm, this paper aims to investigate (1) virtual agents’ similarity to humans in terms of behaviour in human networks and (2) reactions to human versus virtual agents in human networks where this interaction is publicly visible. We analyse the posting behaviour of and reactions to one human, one human-like virtual, and one anime-like virtual influencer active on a popular social media platform via text and emoji postings over an 11-month period. We find that, despite the general positive atmosphere of the platform, the human-like virtual influencer receives significantly lower positive reactions, providing evidence for the Uncanny Valley. Additional measures of negative reactions show a similar pattern. We discuss these results within the context of authenticity and social identity on social media, providing recommendations for the implementation of virtual influencers in human social networks.
The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots
2021, International Journal of Human Computer Studies
Citation Excerpt :
Although speech-based, Siri brought interaction based on language into the mainstream consumer market in 2010. In 2011, Cleverbot, a chatbot created by Rollo Carpenter, became the first AI program to pass the ‘Modified Turing Test’ (Gilbert & Forney, 2015), while IBM's Watson was on the game show Jeopardy. While in 2013, WeChat launched its chatbot platform.
Over the last ten years there has been a growing interest around text-based chatbots, software applications interacting with humans using natural written language. However, despite the enthusiastic market predictions, ‘conversing’ with this kind of agents seems to raise issues that go beyond their current technological limitations, directly involving the human side of interaction. By adopting a Human-Computer Interaction (HCI) lens, in this article we present a systematic literature review of 83 papers that focus on how users interact with text-based chatbots. We map the relevant themes that are recurrent in the last ten years of research, describing how people experience the chatbot in terms of satisfaction, engagement, and trust, whether and why they accept and use this technology, how they are emotionally involved, what kinds of downsides can be observed in human-chatbot conversations, and how the chatbot is perceived in terms of its humanness. On the basis of these findings, we highlight open issues in current research and propose a number of research opportunities that could be tackled in future years.
Exploring consumers’ response to text-based chatbots in e-commerce: the moderating role of task complexity and chatbot disclosure
2024, arXiv
Impact of the E-brand on the Consumer's E-trust, Reliance and Resistance Towards the Chatbot
2023, Springer Series in Design and Innovation
Imagining Future Digital Assistants at Work: A Study of Task Management Needs
2022, arXiv

View all citing articles on Scopus

^☆: This paper has been recommended for acceptance by J. LaViola.

View full text

Can avatars pass the Turing test? Intelligent agent perception in a 3D virtual environment☆

Highlights

Abstract

Section snippets

The Turing Test: definition and significance

Participants

Results and discussion

J. Pragmat.

Discourse on Method and Meditations (Laurence J. Lafleur, trans.)

3D virtual worlds and the metaverse: current status and future possibilities

ACM Comput. Surv.

On seeing human: a three-factor theory of anthropomorphism

Psychol. Rev.

Subcognition and the limits of the Turing test

Mind.

The distributed self: virtual worlds and the future of human identity