Top

Published in:

2016 | OriginalPaper | Chapter

17. Evaluating the Conversational Interface

Authors : Michael McTear, Zoraida Callejas, David Griol

Published in: The Conversational Interface

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The evaluation of conversational interfaces is a continuously evolving research area that encompasses a rich variety of methodologies, techniques, and tools. As conversational interfaces become more complex, their evaluation has become multifaceted. Furthermore, evaluation involves paying attention not only to the different components in isolation, but also to interrelations between the components and the operation of the system as a whole. This chapter discusses the main measures that are employed for evaluating conversational interfaces from a variety of perspectives.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Implementing Multimodal Conversational Interfaces Using Android Wear

next chapter Future Directions

http://www.ilc.cnr.it/EAGLES/browse.html. Accessed March 2, 2016.

http://www.cocosda.org/. Accessed February 29, 2016.

http://www.elra.info/en/. Accessed February 29, 2016.

http://itl.nist.gov/iad/mig/publications/ASRhistory/index.html. Accessed February 29, 2016.

http://festvox.org/blizzard/. Accessed February 29, 2016.

http://www.speechtechmag.com/Articles/Editorial/Feature/Ten-Criteria-for-Measuring-Effective-Voice-User-Interfaces-29443.aspx. Accessed February 28, 2016.

http://help.voxeo.com/go/help/xml.vxml.bestprac.vui. Accessed February 28, 2016.

https://www.itu.int/rec/T-REC-P.851-200311-I/en. Accessed February 28, 2016.

https://www.mturk.com/mturk/welcome. Accessed on February 28, 2016.

http://www.crowdflower.com/. Accessed on February 28, 2016.

https://microworkers.com/. Accessed on February 28, 2016.

https://www.crowdee.de/en/. Accessed on February 28, 2016.

The papers presented at the special session are accessible here: http://www.isca-speech.org/archive/interspeech_2015/#Sess074. Accessed February 29, 2016.

Bennett C (2005) Large scale evaluation of corpus-based synthesizers: results and lessons from the Blizzard challenge 2005. In: Proceedings of the 9th European conference on speech communication and technology (Interspeech’2005—Eurospeech), Lisbon, Portugal, 4–8 Sept 2005, pp 105–108. http://www.isca-speech.org/archive/interspeech_2005/i05_0105.html

Beringer N, Kartal U, Louka K, Schiel F, Türk U (2002) PROMISE: a procedure for multimodal interactive system evaluation. In: Proceedings of the LREC workshop on multimodal resources and multimodal systems evaluation, Las Palmas, Spain, 1 June 2002, pp 77–80. http://www.lrec-conf.org/proceedings/lrec2002/pdf/50.pdf

Bernsen NO, Dybkjær H, Dybkjær L (1998) Designing interactive speech systems: from first ideas to user testing. Springer, London. doi:10.1007/978-1-4471-0897-9 CrossRef

Black A, Tokuda K (2005) The Blizzard challenge—2005: evaluating corpus-based speech synthesis on common datasets. In: Proceedings of Interspeech’2005—Eurospeech, 9th european conference on speech communication and technology, Lisbon, Portugal, 4–8 Sept 2005, pp 77–80. http://www.isca-speech.org/archive/interspeech_2005/i05_0077.html

Black A, Burger S, Langner B, Parent G, Eskenazi M (2010) Spoken dialog challenge 2010. In: Proceedings of IEEE spoken language technology workshop (SLT), Berkeley, California USA, 12–15 Dec 2010, pp 448-453. doi:10.1109/SLT.2010.5700894

Callejas Z, Griol D, López-Cózar R (2014) A framework for the assessment of synthetic personalities according to user perception. Int J Hum-Comput Stud 72:567–583. doi:10.1016/j.ijhcs.2014.02.002 CrossRef

Carenini G, Moore JD (2006) Generating and evaluating evaluative arguments. Artif Intell 170:925–952. doi:10.1016/j.artint.2006.05.003 CrossRef

Cuayáhuitl H, Renals S, Lemon O, Shimodaira H (2005) Human-computer dialogue simulation using Hidden Markov models. In: Proceedings of the IEEE automatic speech recognition and understanding workshop (ASRU’05), San Juan, Puerto Rico, 27 Nov–1 Dec 2005, pp 290–295. doi:10.1109/ASRU.2005.1566485

Davis FD, Bagozzi RP, Warshaw PR (1989) User acceptance of computer technology: a comparison of two theoretical models. Manage Sci 35:982–1003. doi:10.1287/mnsc.35.8.982 CrossRef

Delogu C, Conte S, Sementina C (1998) Cognitive factors in the evaluation of synthetic speech. Speech Commun 24:153–168. doi:10.1016/S0167-6393(98)00009-0 CrossRef

Dybkjaer L, Bernsen NO (2000) Usability issues in spoken language dialogue systems. Nat Lang Eng 6(3–4):243–271. doi:10.1017/s1351324900002461 CrossRef

Dybkjaer L, Bernsen NO, Carlson R, Chase L, Dahlbäck N, Failenschmid K, Heid U, Heisterkamp P, Jönsson A, Kamp H, Karlsson I, Kuppevelt J, Lamel L, Paroubek P, Williams D (1998) The DISC approach to spoken language systems development and evaluation. In: Proceedings of the first international conference on language resources and evaluation, Granada, Spain, 28–30 May 1998, pp 185–189

Dybkjaer L, Bernsen NO, Minker W (2004) Evaluation and usability of multimodal spoken language dialogue systems. Speech Commun 43(1–2):33–54. doi:10.1016/j.specom.2004.02.001 CrossRef

Eckert W, Levin E, Pieraccini R (1997) User modeling for spoken dialogue system evaluation. In: IEEE workshop on automatic speech recognition and understanding, Santa Barbara, CA, 14–17 Dec 1997, pp 80–87

Engelbrecht K-P (2012) Estimating spoken dialog system quality with user models. Springer Science & Business Media, Berlin. doi:10.1007/978-3-642-31591-6

Eskenazi M, Levow G-A, Meng H, Parent G, Suendermann D (eds) (2013) Crowdsourcing for speech processing: applications to data collection, transcription, and assessment. Wiley, Chichester. doi:10.1002/9781118541241

Fiscus JG, Ajot J, Garofolo JS (2008) The rich transcription 2007 meeting recognition evaluation. In: Stiefelhagen R, Bowers R, Fiscus J (eds) Multimodal technologies for perception of humans. Springer, Berlin, pp 373–389. doi:10.1007/978-3-540-68585-2_36

Gibbon D, Moore R, Winski R (1997) Handbook of standards and resources for spoken language systems. Walter de Gruyter, Berlin

Griol D, Callejas Z, López-Cózar R, Riccardi G (2014) A domain-independent statistical methodology for dialog management in spoken dialog systems. Comput Speech Lang 28:743–768. doi:10.1016/j.csl.2013.09.002 CrossRef

Hassenzahl M (2001) The effect of perceived hedonic quality on product appealingness. Int J Hum-Comput Interact 13:481–499. doi:10.1207/S15327590IJHC1304_07 CrossRef

Henderson J, Jurčíček F (2012) Data-driven methods for spoken language understanding. In: Lemon O, Pietquin O (eds) Data-driven methods for adaptive spoken dialogue systems: computational learning for conversational interfaces. Springer, New York, pp 19–38. doi:10.1007/978-1-4614-4803-7_3

Hone KS, Graham R (2000) Towards a tool for the subjective assessment of speech system interfaces (SASSI). Nat Lang Eng 6:287–303. doi:10.1017/S1351324900002497 CrossRef

Jekosch U (2005) Voice and speech quality perception: assessment and evaluation. Springer, Berlin. doi:10.1007/3-540-28860-0

Jiang J, Awadallah AH, Jones R, Ozertem U, Zitouni I, Kulkarni RG, Khan OZ (2015) Automatic online evaluation of intelligent assistants. In: Proceedings of the 23rd international conference on World Wide Web (WWW ‘15), Florence, Italy, 18–22 May 2015, pp 506–516. http://www.www2015.it/documents/proceedings/proceedings/p506.pdf

Jung S, Lee C, Kim K, Jeong M, Lee GG (2009) Data-driven user simulation for automated evaluation of spoken dialog systems. Comput Speech Lang 23(4):479–509. doi:10.1016/j.csl.2009.03.002 CrossRef

Jurčíček F, Keizer S, Gašić M, Mairesse F, Thomson B, Yu K, Young S (2011) Real user evaluation of spoken dialogue systems using Amazon Mechanical Turk. In: Proceedings of the 12th annual conference of the international speech communication association (Interspeech 2011), Florence, Italy, 27–31 Aug 2011, pp 3061–3064. http://www.isca-speech.org/archive/interspeech_2011/i11_3061.html

Kühnel C (2012) Quantifying quality aspects of multimodal interactive systems. Springer, Berlin. doi:10.1007/978-3-642-29602-4

Lemon O, Pietquin O (eds) (2012) Data-driven methods for adaptive spoken dialog systems: computational learning for conversational interfaces. Springer, New York. doi:10.1007/978-1-4614-4803-7

Lemon O, Janarthanam S, Rieser V (2010) Statistical approaches to adaptive natural language generation. In: Lemon O, Pietquin O (eds) Data-driven methods for adaptive spoken dialogue systems: computational learning for conversational interfaces. Springer, New York. doi:10.1007/978-1-4614-4803-7_6

López Cózar R, Araki M (2005) Spoken, multilingual and multimodal dialog systems: development and assessment. Wiley, Chichester. doi:10.1002/0470021578

López-Cózar R, Callejas Z, McTear M (2007) Testing the performance of spoken dialogue systems by means of an artificially simulated user. Artif Intell Rev 26:291–323. doi:10.1007/s10462-007-9059-9 CrossRef

Möller S (2005) Quality of telephone-based spoken dialogue systems. Springer Sciennce + Business Media, Heidelberg. doi:10.1007/b100796

Möller S, Smeele P, Boland H, Krebber J (2007) Evaluating spoken dialogue systems according to de-facto standards: a case study. Comput Speech Lang 21(1):26–53. doi:10.1016/j.csl.2005.11.003 CrossRef

Möller S, Engelbrecht K-P, Schleicher R (2008) Predicting the quality and usability of spoken dialogue services. Speech Commun 50:730–744. doi:10.1016/j.specom.2008.03.001 CrossRef

Raux A, Langner B, Black A, Eskenazi M (2005) Let’s go public! Taking a spoken dialog system to the real world. In: Proceedings of the 9th European conference on speech communication and technology (Interspeech’2005—Eurospeech), Lisbon, Portugal, 4–8 September 2005, pp 885–888. http://www.isca-speech.org/archive/interspeech_2005/i05_0885.html

Reiter E, Belz A (2009) An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Comput Linguist 35:529–558. doi:10.1162/coli.2009.35.4.35405 CrossRef

Reiter E, Robertson R, Osman LM (2003) Lessons from a failure: generating tailored smoking cessation letters. Artif Intell 144:41–58. doi:10.1016/S0004-3702(02)00370-3 CrossRef

Ruttkay Z, Pelachaud C (eds) (2004) From brows to trust. Evaluating embodied conversational agents. Springer, Netherlands. doi:10.1007/1-4020-2730-3

Schatzmann J, Georgila K, Young S (2005) Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proceedings of the 6th SIGdial workshop on discourse and dialogue, Lisbon, Portugal, 2–3 Sept 2005, pp 45–54. http://www.isca-speech.org/archive_open/sigdial6/sgd6_045.html

Schatzmann J, Weilhammer K, Stuttle M, Young S (2006) A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowl Eng Rev 21:97. doi:10.1017/S0269888906000944 CrossRef

Scheffler K, Young S (2001) Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In: Proceedings of 49th annual meeting of the association for computational linguistics: human language technologies (HLT), Portland, Oregon USA, 19–24 June 2011, pp 12–18. http://dl.acm.org/citation.cfm?id=1289246

Schmitt A, Ultes S (2015) Interaction quality: assessing the quality of ongoing spoken dialog interaction by experts—and how it relates to user satisfaction. Speech Commun 74:12–36. doi:10.1016/j.specom.2015.06.003 CrossRef

Schmitt A, Ultes S, Minker W (2012) A parameterized and annotated spoken dialog corpus of the CMU let’s go bus information system. In: Proceedings of the eight international conference on language resources and evaluation (LREC’12). Istanbul, Turkey. http://www.lrec-conf.org/proceedings/lrec2012/summaries/333.html

Sim DYY, Loo CK (2015) Extensive assessment and evaluation methodologies on assistive social robots for modelling human–robot interaction—a review. Inf Sci 301:305–344. doi:10.1016/j.ins.2014.12.017 CrossRef

Suhm B (2008) IVR Usability engineering using guidelines and analyses of end-to-end calls. In: Human factors and voice interactive systems. Springer US, Boston, MA, pp 1–41. doi:10.1007/978-0-387-68439-0_1

Tur G, de Mori R (eds) (2011) Spoken language understanding: systems for extracting semantic information from speech. Wiley, Chichester, UK. doi:10.1002/9781119992691

Walker MA, Litman DJ, Kamm CA, Abella, A (1997) PARADISE: a framework for evaluating spoken dialogue agents. In: Proceedings of the 8th conference on European chapter of the association for computational linguistics (EACL), Madrid, Spain, 7–12 July 2005, pp 271–280. https://aclweb.org/anthology/P/P97/P97-1035.pdf

Walker MA, Litman DJ, Kamm CA, Abella A (1998) Evaluating spoken dialogue agents with PARADISE: two case studies. Comput Speech Lang 12(4):317–347. doi:10.1006/csla.1998.0110 CrossRef

Walker M, Kamm CA, Litman DJ (2000) Towards developing general models of usability with PARADISE. Nat Lang Eng 6(3–4):363–377. doi:10.1017/s1351324900002503 CrossRef

Walker MA, Rudnicky A, Prasad R, Aberdeen J, Bratt EO, Garofolo J, Hastie H, Le A, Pellom B, Potamianos A, Passonneau R, Roukos S, Sanders G, Seneff S, Stallard D (2002) DARPA Communicator: cross-system results for the 2001 evaluation. In: Proceedings of the 7th international conference on spoken language processing (ICSLP2002), vol 1, Denver, Colorado, pp 273–276. http://www.isca-speech.org/archive/archive_papers/icslp_2002/i02_0269.pdf. Accessed 21 Jan 2016

White RW (2016) Interactions with search systems. Cambridge University Press, CambridgeCrossRef

Williams JD (2008) Evaluating user simulations with the Cramér-von Mises divergence. Speech Commun 50(10):829–846. doi:10.1016/j.specom.2008.05.007 CrossRef

Williams JD, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of the 4th annual SIGdial meeting on discourse and dialogue (SIGDIAL), Metz, France, 22–24 Aug 2013, pp 404–413. http://www.aclweb.org/anthology/W13-4065

Yang Z, Levow G-A, Meng H (2012) Predicting user satisfaction in spoken dialog system evaluation with collaborative filtering. IEEE J Sel Top Signal Process 6:971–981. doi:10.1109/JSTSP.2012.2229965 CrossRef

Young S (2002) The statistical approach to the design of spoken dialogue systems. Tech Report CUED/F-INFENG/TR.433. Cambridge University Engineering Department. http://mi.eng.cam.ac.uk/~sjy/papers/youn02b.ps.gz

Title: Evaluating the Conversational Interface
Authors: Michael McTear
Zoraida Callejas
David Griol
Publisher: Springer International Publishing
Book: The Conversational Interface
Print ISBN: 978-3-319-32965-9

Electronic ISBN: 978-3-319-32967-3

Copyright Year: 2016
DOI: https://doi.org/10.1007/978-3-319-32967-3_17

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"