Top

Universal Access in the Information Society

Published in:

06-07-2019 | Long Paper

CBET: design and evaluation of a domain-specific chatbot for mobile learning

Authors: Qingtang Liu, Jingxiu Huang, Linjing Wu, Ke Zhu, Shen Ba

Published in: Universal Access in the Information Society | Issue 3/2020

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The popularity of mobile devices and conversational agents in recent years has seen wide use of chatbots in different educational scenarios. In relation to the advances in mobile devices and conversational agents, there are few research works concerning the design and evaluation of domain-specific chatbots to fulfill the demand of mobile learning. To address this issue, we propose an agent-based conceptual architecture to develop a domain-specific chatbot for mobile learning. We extend the open-domain DeepQA agent to make it sensitive to restricted domain questions by building a domain-specific gate, and employ WeChat as user interface. To evaluate our chatbot, subjective and objective criteria are employed to assess its effectiveness. Additionally, its usability evaluation proceeds with system usability scale questionnaire and net promoter score simultaneously. In total, 18 domain experts participated in the evaluation of effectiveness, and 52 participants were involved in the evaluation of usability. Based on the evaluation results, we conclude that our chatbot can serve as an effective information retrieval tool in a specific domain. The perceived usability of our chatbot tends to be moderate and marginal and has positively affected the promotion of our chatbot for mobile learning. This paper contributes to the educative application of chatbots in specific subject fields.

previous article A confirmatory factor analysis of the behavioral intention to use smart wellness wearables in Malaysia

next article Acceptability of notifications delivered to older adults by technology-based assisted living services

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

This domain-specific corpus is available at https://github.com/JimSow/CBET-Corpus.

This dataset is available at https://github.com/JimSow/CBET-Corpus.

The official Web site of ACO is available at http://aco.ccnu.edu.cn.

Chakrabarti, C., Luger, G.F.: Artificial conversations for customer service chatter bots: architecture, algorithms, and evaluation metrics. Expert Syst. Appl. 42(20), 6878–6897 (2015). https://doi.org/10.1016/j.eswa.2015.04.067 CrossRef

Crutzen, R., Peters, G.J.Y., Portugal, S.D., Fisser, E.M., Grolleman, J.J.: An artificially intelligent chat agent that answers adolescents’ questions related to sex, drugs, and alcohol: an exploratory study. J. Adolesc. Health 48(5), 514–519 (2011). https://doi.org/10.1016/j.jadohealth.2010.09.002 CrossRef

Griol, D., Callejas, Z.: Mobile conversational agents for context-aware care applications. Cogn. Comput. 8(2), 336–356 (2016). https://doi.org/10.1007/s12559-015-9352-x CrossRef

Griol, D., Callejas, Z.: An architecture to develop multimodal educative applications with chatbots. Int. J. Adv. Rob. Syst. 10(3), 175–190 (2013). https://doi.org/10.5772/55791 CrossRef

Kerly, A., Hall, P., Bull, S.: Bringing chatbots into education: towards natural language negotiation of open learner models. Knowl. Based Syst. 20(2), 177–185 (2007). https://doi.org/10.1016/j.knosys.2006.11.014 CrossRef

Kerly, A., Ellis, R., Bull, S.: CALMsystem: a conversational agent for learner modelling. In: International Conference on Innovative Techniques and Applications of Artificial Intelligence, pp. 89–102. Springer (2008). https://doi.org/10.1007/978-1-84800-086-5_7

Benotti, L., Martínez, M.C., Schapachnik, F.: Engaging high school students using chatbots. In: Proceedings of the 2014 Conference on Innovation and Technology in Computer Science Education, pp. 63–68. ACM, New York, NY, USA (2014)

Jia, J.: CSIEC: a computer assisted English learning chatbot based on textual knowledge and reasoning. Knowl. Based Syst. 22(4), 249–255 (2009). https://doi.org/10.1016/j.knosys.2008.09.001 CrossRef

Goda, Y., Yamada, M., Matsukawa, H., Hata, K., Yasunami, S.: Conversation with a chatbot before an online EFL group discussion and the effects on critical thinking. J. Inf. Syst. Educ. 13(1), 1–7 (2014). https://doi.org/10.12937/ejsise.13.1 CrossRef

10.

Coniam, D.: The linguistic accuracy of chatbots: usability from an ESL perspective. Text Talk. 34(5), 545–567 (2014). https://doi.org/10.1515/text-2014-0018 CrossRef

11.

Pérez-Marín, D., Boza, A.: A procedure to create a pedagogic conversational agent in secondary physics and chemistry education. Int. J. Inf. Commun. Technol. Educ. 9(4), 94–112 (2013). https://doi.org/10.4018/ijicte.2013100107 CrossRef

12.

Graesser, A.C., Li, H., Forsyth, C.: Learning by communicating in natural language with conversational agents. Curr. Dir. Psychol. Sci. 23(5), 374–380 (2014). https://doi.org/10.1177/0963721414540680 CrossRef

13.

Kumar, R., Rose, C.P.: Architecture for building conversational agents that support collaborative learning. IEEE Trans. Learn. Technol. 4(1), 21–34 (2011). https://doi.org/10.1109/TLT.2010.41 CrossRef

14.

Dyke, G., Adamson, D., Howley, I., Rosé, C.P.: Enhancing scientific reasoning and discussion with conversational agents. IEEE Trans. Learn. Technol. 6(3), 240–247 (2013). https://doi.org/10.1109/TLT.2013.25 CrossRef

15.

Tegos, S., Demetriadis, S., Karakostas, A.: Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Comput. Educ. 87, 309–325 (2015). https://doi.org/10.1016/j.compedu.2015.07.014 CrossRef

16.

Tegos, S., Demetriadis, S., Papadopoulos, P.M., Weinberger, A.: Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions. Int. J. Comput. Support. Collab. Learn. 11(4), 417–440 (2016). https://doi.org/10.1007/s11412-016-9246-2 CrossRef

17.

Preece, A., Webberley, W., Braines, D., Zaroukian, E.G., Bakdash, J.Z.: SHERLOCK: experimental evaluation of a conversational agent for mobile information tasks. IEEE Trans. Hum. Mach. Syst. 47(6), 1017–1028 (2017). https://doi.org/10.1109/THMS.2017.2700625 CrossRef

18.

Wu, W.H., Wu, Y.C.J., Chen, C.Y., Kao, H.Y., Lin, C.H., Huang, S.H.: Review of trends from mobile learning studies: a meta-analysis. Comput. Educ. 59(2), 817–827 (2012). https://doi.org/10.1016/j.compedu.2012.03.016 CrossRef

19.

Becker, S.A., Cummins, M., Davis, A., Freeman, A., Glesinger Hall, C., Ananthanarayanan, V.: The NMC Horizon Report: 2017 Higher Education Edition. The New Media Consortium, Austin (2017)

20.

Liu, Q., Wu, L., Yang, Z., Liu, Y.: Domain phrase identification using atomic word formation in Chinese text. Knowl. Based Syst. 24(8), 1254–1260 (2011). https://doi.org/10.1016/j.knosys.2011.06.001 CrossRef

21.

Wu, L., Liu, Q., Zhao, G., Huang, H., Huang, T.: Thesaurus dataset of educational technology in Chinese. Br. J. Edu. Technol. 46(5), 1118–1122 (2015). https://doi.org/10.1111/bjet.12298 CrossRef

22.

Hwang, G.J., Tsai, C.C.: Research trends in mobile and ubiquitous learning: a review of publications in selected journals from 2001 to 2010. Br. J. Educ. Technol. 42(4), E65–E70 (2011). https://doi.org/10.1111/j.1467-8535.2011.01183.x CrossRef

23.

Hung, J.L., Zhang, K.: Examining mobile learning trends 2003–2008: a categorical meta-trend analysis using text mining techniques. J. Comput. High. Educ. 24(1), 1–17 (2012). https://doi.org/10.1007/s12528-011-9044-9 CrossRef

24.

Su, C.H., Cheng, C.H.: A mobile gamification learning system for improving the learning motivation and achievements. J. Comput. Assist. Learn. 31(3), 268–286 (2015). https://doi.org/10.1111/jcal.12088 CrossRef

25.

Yousafzai, A., Chang, V., Gani, A., Noor, R.M.: Multimedia augmented m-learning: issues, trends and open challenges. Int. J. Inf. Manage. 36(5), 784–792 (2016). https://doi.org/10.1016/j.ijinfomgt.2016.05.010 CrossRef

26.

Lowe, R., Pow, N., Serban, I., Pineau, J.: The Ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In: Proceedings of the SIGDIAL 2015 Conference, pp. 285–294. Association for Computational Linguistics (2015)

27.

Moreno, R., Mayer, R.: Interactive multimodal learning environments. Educ. Psychol. Rev. 19(3), 309–326 (2007). https://doi.org/10.1007/s10648-007-9047-2 CrossRef

28.

Kim, N.Y.: A study on different types of speech acts in voice-chat between EFL students and a chatbot. Stud. Engl. Educ. 22(3), 81–109 (2017)CrossRef

29.

Ayedoun, E., Hayashi, Y., Seta, K.: Web-services based conversational agent to encourage willingness to communicate in the EFL context. J. Inf. Syst. Educ. 15(1), 15–27 (2016). https://doi.org/10.12937/ejsise.15.15 CrossRef

30.

Fadhil, A., Villafiorita, A.: An adaptive learning with gamification & conversational UIs: The rise of CiboPoliBot. In: Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 408–412. ACM, New York, NY, USA (2017)

31.

Tegos, S., Demetriadis, S., Tsiatsos, T.: An investigation of conversational agent interventions supporting historical reasoning in primary education. In: International Conference on Intelligent Tutoring Systems, pp. 260–266. Springer (2016)

32.

Ardito, C., Costabile, M.F., Marsico, M.D., Lanzilotti, R., Levialdi, S., Roselli, T., et al.: An approach to usability evaluation of e-learning applications. Univ. Access Inf. Soc. 4(3), 270–283 (2006). https://doi.org/10.1007/s10209-005-0008-6 CrossRef

33.

Song, D., Oh, E.Y., Rice, M.: Interacting with a conversational agent system for educational purposes in online courses. In: Proceedings of the 10th International Conference on Human System Interactions, pp. 78–82. IEEE (2017)

34.

Shawar, B. A., Atwell, E.: Different measurements metrics to evaluate a chatbot system. In: Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies, pp. 89–96. Association for Computational Linguistics (2007)

35.

Yu, Z., Xu, Z., Black, A.W., Rudnicky, A.: Chatbot evaluation and database expansion via crowdsourcing. In: Proceedings of the chatbot workshop of LREC, pp. 102–107. Association for Computational Linguistics (2016)

36.

Radziwill, N.M., Benton, M.C.: Evaluating quality of chatbots and intelligent conversational agents. CoRR abs/1704.04579 (2017)

37.

Liu, C.W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. CoRR abs/1603.08023 (2016)

38.

Orfanou, K., Tselios, N., Katsanos, C.: Perceived usability evaluation of learning management systems: empirical evaluation of the System Usability Scale. Int. Rev. Res. Open Distrib. Learn. 16(2), 227–246 (2015). https://doi.org/10.19173/irrodl.v16i2.1955 CrossRef

39.

Revythi, A., Tselios, N.: Extension of Technology Acceptance Model by using System Usability Scale to assess behavioral intention to use e-learning. CoRR abs/1704.06127

40.

Apdplat.: deep-qa. Retrieved from https://github.com/ysc/QuestionAnsweringSystem/deep-qa (10 October, 2018)

41.

Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Schlaefer, N.: Building Watson: an overview of the DeepQA project. AI Mag. 31(3), 59–79 (2010). https://doi.org/10.1609/aimag.v31i3.2303 CrossRef

42.

Venkatesh, A., Khatri, C., Ram, A., Guo, F., Gabriel, R., Nagar, A., Goel, R.: On Evaluating and Comparing Conversational Agents. CoRR abs/1801.03625 (2018)

43.

Juszczak, P.: Learning to recognize: A study on one-class classification and active learning. Dissertation, Delft University of Technology (2006)

44.

Zhang, C.: Automatic keyword extraction from documents using conditional random fields. J. Comput. Inf. Syst. 4(3), 1169–1180 (2008)

45.

Désir, C., Bernard, S., Petitjean, C., Heutte, L.: One class random forests. Pattern Recognit. 46(12), 3490–3506 (2013). https://doi.org/10.1016/j.patcog.2013.05.022 CrossRef

46.

Gaudio, R.D., Batista, G., Branco, A.: Coping with highly imbalanced datasets: a case study with definition extraction in a multilingual setting. Nat. Lang. Eng. 20(3), 327–359 (2014). https://doi.org/10.1017/S1351324912000381 CrossRef

47.

Reiplinger, M., Schäfer, U., Wolska, M.: Extracting glossary sentences from scholarly articles: A comparative evaluation of pattern bootstrapping and deep analysis. In: Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pp. 55–65. Association for Computational Linguistics (2012)

48.

Jin, Y., Kan, M.Y., Ng, J.P., He, X.: Mining scientific terms and their definitions: A study of the ACL anthology. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 780–790. Association for Computational Linguistics (2013)

49.

Abacha, A.B., Zweigenbaum, P.: Automatic extraction of semantic relations between medical entities: a rule based approach. J. Biomed. Semant. 2(5), S4 (2011). https://doi.org/10.1186/2041-1480-2-S5-S4 CrossRef

50.

Punuru, J., Chen, J.: Learning non-taxonomical semantic relations from domain texts. J. Intell. Inf. Syst. 38(1), 191–207 (2012). https://doi.org/10.1007/s10844-011-0149-4 CrossRef

51.

Shen, M., Liu, D.R., Huang, Y.S.: Extracting semantic relations to enrich domain ontologies. J. Intell. Inf. Syst. 39(3), 749–761 (2012). https://doi.org/10.1007/s10844-012-0210-y CrossRef

52.

Xu, J., Kang, Q., Song, Z., Clarke, C.P.: Applications of mobile social media: WeChat among academic libraries in China. J. Acad. Librariansh. 41(1), 21–30 (2015). https://doi.org/10.1016/j.acalib.2014.10.012 CrossRef

53.

So, S.: Mobile instant messaging support for teaching and learning in higher education. Internet High. Educ. 31, 32–42 (2016). https://doi.org/10.1016/j.iheduc.2016.06.001 CrossRef

54.

Sun, Z., Liu, R., Luo, L., Wu, M., Shi, C.: Exploring collaborative learning effect in blended learning environments. J. Comput. Assist. Learn. 33(6), 575–587 (2017). https://doi.org/10.1111/jcal.12201 CrossRef

55.

Goh, O.S., Ardil, C., Wong, W., Fung, C.C.: A black-box approach for response quality evaluation of conversational agent systems. Int. J. Comput. Intell. 3(3), 195–203 (2007)

56.

Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. Association for Computational Linguistics (2002)

57.

Murdock, J.W., Fan, J., Lally, A., Shima, H.: Textual evidence gathering and analysis. IBM J. Res. Dev. 56(3.4), 8:1–8:14 (2012). https://doi.org/10.1147/JRD.2012.2187249 CrossRef

58.

Harrati, N., Bouchrika, I., Tari, A., Ladjailia, A.: Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Comput. Hum. Behav. 61, 463–471 (2016). https://doi.org/10.1016/j.chb.2016.03.051 CrossRef

59.

Kortum, P.T.: An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24(6), 574–594 (2008). https://doi.org/10.1080/10447310802205776 CrossRef

60.

Michalco, J., Simonsen, J.G., Hornbæk, K.: An exploration of the relation between expectations and user experience. Int. J. Hum. Comput. Interact. 31(9), 603–617 (2015). https://doi.org/10.1080/10447318.2015.1065696 CrossRef

61.

Gould, D.J., Terrell, M.A., Fleming, J.: A usability study of users’ perceptions toward a multimedia computer-assisted learning tool for neuroanatomy. Anatom. Sci. Educ. 1(4), 175–183 (2008). https://doi.org/10.1002/ase.36 CrossRef

62.

Bringula, R.P., Basa, R.S.: Factors affecting faculty web portal usability. J. Educ. Technol. Soc. 14(4), 253–265 (2011)

63.

Flavián, C., Guinalíu, M., Gurrea, R.: The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inf. Manag. 43(1), 1–14 (2006). https://doi.org/10.1016/j.im.2005.01.002 CrossRef

64.

Sauro, J., Lewis, J.R.: Quantifying the User Experience: Practical Statistics for User Research. Morgan Kaufmann, Burlington (2016). https://doi.org/10.1007/s13398-014-0173-7.2 CrossRef

65.

Wu, Y., Li, Z., Wu, W., Zhou, M.: Response selection with topic clues for retrieval-based chatbots. Neurocomputing 316, 251–261 (2018). https://doi.org/10.1016/j.neucom.2018.07.073 CrossRef

66.

Jin, W., Zhirui, D.: Research on mobile learning model of college English based on WeChat platform. Eurasia J. Math. Sci. Technol. Educ. 13(8), 5847–5853 (2017). https://doi.org/10.12973/eurasia.2017.01034a CrossRef

Title: CBET: design and evaluation of a domain-specific chatbot for mobile learning
Authors: Qingtang Liu
Jingxiu Huang
Linjing Wu
Ke Zhu
Shen Ba
Publication date: 06-07-2019
Publisher: Springer Berlin Heidelberg
Published in: Universal Access in the Information Society / Issue 3/2020
Print ISSN: 1615-5289
Electronic ISSN: 1615-5297
DOI: https://doi.org/10.1007/s10209-019-00666-x

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 3/2020

Special issue on accessibility and software design for all

Understanding the social adoption of smart TVs: the key role of product coolness

Acceptability of notifications delivered to older adults by technology-based assisted living services

A confirmatory factor analysis of the behavioral intention to use smart wellness wearables in Malaysia

Cultural heritage visits supported on visitors’ preferences and mobile devices

On modeling the quality of concept mapping toward more intelligent online learning feedback: a fuzzy logic-based approach

Premium Partner