Top

Published in:

2021 | OriginalPaper | Chapter

Overview of the Dialogue Breakdown Detection Challenge 4

Authors : Ryuichiro Higashinaka, Luis F. D’Haro, Bayan Abu Shawar, Rafael E. Banchs, Kotaro Funakoshi, Michimasa Inaba, Yuiko Tsunomori, Tetsuro Takahashi, João Sedoc

Published in: Increasing Naturalness and Flexibility in Spoken Dialogue Interaction

Publisher: Springer Singapore

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

To promote the research and development of dialogue breakdown detection for dialogue systems, we have been organizing a series of dialogue breakdown detection challenges to detect a system’s inappropriate utterances that lead to dialogue breakdowns in chat-oriented dialogue. In this paper, we overview Dialogue Breakdown Detection Challenge 4 (DBDC4). As in the previous challenges, we used datasets in English and Japanese. Four teams participated in the challenge, in which all four teams worked on English, and two of the four teams worked on Japanese as well. This paper describes the task setting, evaluation metrics, and datasets for the challenge and the results of the submitted runs of the participants.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

previous chapter First Leap Towards Development of Dialogue System for Autonomous Bus

next chapter Dialogue Breakdown Detection Using BERT with Traditional Dialogue Features

https://github.com/DeepPavlov/convai/tree/master/data.

https://requester.mturk.com.

http://crowdworks.jp.

http://crowdsourcing.yahoo.co.jp.

Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 system demonstrations, pp 37–42

Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

D’Haro LF, Shawar BA, Yu Z (2016) RE-WOCHAT 2016—shared task description report. In: Proceedings of the workshop on collecting and generating resources for chatbots and conversational agents-development and evaluation (RE-WOCHAT), pp 39–42

Hendriksen M, Leeuwenberg A, Moens MF (2019) LSTM for dialogue breakdown detection: exploration of different model types and word embeddings. In: Proceedings of the WOCHAT

Higashinaka R, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y, Mizukami M (2015) Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of the SIGDIAL, pp 87–95

Higashinaka R, Mizukami M, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y (2015) Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems. In: Proceedings of the EMNLP, pp 2243–2248

Higashinaka R, Funakoshi K, Kobayashi Y, Inaba M (2016) The dialogue breakdown detection challenge: task description, datasets, and evaluation metrics. In: Proceedings of the LREC, pp 3146–3150

Higashinaka R, Funakoshi K, Inaba M, Tsunomori Y, Takahashi T, Akama R (2019) Dialogue system live competition: identifying problems with dialogue systems through live event. In: Proceedings of the IWSDS

Martinovsky B, Traum D (2003) The error is the clue: breakdown in human-machine interaction. In: Proceedings of the ISCA workshop on error handling in spoken dialogue systems, pp 11–16

10.

Onishi K, Yoshimura T (2014) Casual conversation technology achieving natural dialog with computers. NTT DOCOMO Tech J 15(4):16–21

11.

Ritter A, Cherry C, Dolan WB (2011) Data-driven response generation in social media. In: Proceedings of the EMNLP, pp 583–593

12.

Shin J, Dirafzoon A, Anshu A (2019) Context-enriched attentive memory network with global and local encoding for dialogue breakdown detection. In: Proceedings of the WOCHAT

13.

Sugiyama H (2019) Dialogue breakdown detection using BERT with traditional dialogue features. In: Proceedings of the WOCHAT

14.

Tsukahara H, Uchiumi K (2015) System utterance generation by label propagation over association graph of words and utterance patterns for open-domain dialogue systems. In: Proceedings of the PACLIC, pp 323–331

15.

Tsunomori Y, Higashinaka R, Takahashi T, Inaba M (2018) Evaluating dialogue breakdown detection in chat-oriented dialogue systems. In: Proceedings of the SEMDIAL

16.

Wang C, Kato S, Sakai T (2019) RSL19BD at DBDC4: ensemble of decision tree-based and LSTM-based models. In: Proceedings of the WOCHAT

Title: Overview of the Dialogue Breakdown Detection Challenge 4
Authors: Ryuichiro Higashinaka
Luis F. D’Haro
Bayan Abu Shawar
Rafael E. Banchs
Kotaro Funakoshi
Michimasa Inaba
Yuiko Tsunomori
Tetsuro Takahashi
João Sedoc
Publisher: Springer Singapore
Book: Increasing Naturalness and Flexibility in Spoken Dialogue Interaction
Print ISBN: 978-981-15-9322-2

Electronic ISBN: 978-981-15-9323-9

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-981-15-9323-9_38