Top

Journal on Multimodal User Interfaces

Published in:

12-10-2023 | Original Paper

A social robot as your reading companion: exploring the relationships between gaze patterns and knowledge gains

Authors: Xuan Liu, Jiachen Ma, Qiang Wang

Published in: Journal on Multimodal User Interfaces | Issue 1/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Intelligent tutoring systems have been widely used in educational activities over the past 20 years. With significantly less effort put into writing or reading assistance, the majority of intelligent tutoring systems focus on mathematics or problem solving activities. However, with the development of E-reading-centered E-education, how to improve students’ learning performance during reading has become increasingly important. Therefore, in this paper, we take a first step in the direction of an adaptive intelligent tutoring system by investigating how different reading strategies relate to knowledge gain based on gaze features and how an embodied social robot affects gaze patterns and reading strategies. The findings showed that different knowledge gains have significant differences in scanning methods and reading depth, and that the feedback given by social robots significantly affects participants’ gaze patterns during the whole reading process. To automatically differentiate between two levels of knowledge gain, several prediction experiments based on various reading strategy-related gaze features were carried out. The results demonstrate that saccades are the best predictors of knowledge gain, with the best model having an average accuracy of 74.2%. Finally, real-time simulation experiments were conducted with sixty participants using the leave-one-out method, and an accurate prediction of the level of knowledge gain of 71. 5% was achieved.

previous article Pegasos: a framework for the creation of direct mobile coaching feedback systems

next article Hearing loss prevention at loud music events via real-time visuo-haptic feedback

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

https://pupil-labs.com/.

Abidin AZ (2020) Students reading comprehension through scanning technique. J Asian Multicult Res Educ Study 1(1):28–35. https://doi.org/10.47616/jamres.v1i1.13CrossRef

Almasri A, Ahmed A, Almasri N et al (2019) Intelligent tutoring systems survey for the period 2000–2018. Int J Acad Eng Res

Anderson JR, Boyle CF, Reiser BJ (1985) Intelligent tutoring systems. Science 228(4698):456–462. https://doi.org/10.1126/science.228.4698.456ADSCrossRefPubMed

Asmawati A (2015) The effectiveness of skimming–scanning strategy in improving students’ reading comprehension at the second grade of SMK Darussalam Makassar. Engl Teach Learn Res J ETERNAL 1(1):69–83. https://doi.org/10.24252/Eternal.V11.2015.A9CrossRef

Bainbridge WA, Hart JW, Kim ES et al (2011) The benefits of interactions with physically present robots over video-displayed agents. Int J Soc Robot 3(1):41–52. https://doi.org/10.1007/s12369-010-0082-7CrossRef

Bamkin M, Goulding A, Maynard S (2013) The children sat and listened: storytelling on children’s mobile libraries. N Rev Child Lit Librariansh 19(1):47–78. https://doi.org/10.1080/13614541.2013.755023CrossRef

Belpaeme T, Kennedy J, Ramachandran A et al (2018) Social robots for education: a review. Sci Robot 3(21):eaat5954. https://doi.org/10.1126/scirobotics.aat5954CrossRefPubMed

Bickmore T, Kimani E, Shamekhi A et al (2021) Virtual agents as supporting media for scientific presentations. J Multimodal User Interfaces 15:131–146. https://doi.org/10.1007/s12193-020-00350-yCrossRef

Biedert R, Hees J, Dengel A et al (2012) A robust realtime reading-skimming classifier. In: Proceedings of the symposium on eye tracking research and applications, pp 123–130. https://doi.org/10.1145/2168556.2168575

10.

Biswas G, Leelawong K, Schwartz D et al (2005) Learning by teaching: a new agent paradigm for educational software. Appl Artif Intell 19(3–4):363–392. https://doi.org/10.1080/08839510590910200CrossRef

11.

Bixler R, D’Mello S (2016) Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Model User Adapt Int 26(1):33–68. https://doi.org/10.1007/s11257-015-9167-1CrossRef

12.

Block E (1986) The comprehension strategies of second language readers. TESOL Q 20(3):463–494. https://doi.org/10.2307/3586295MathSciNetCrossRef

13.

Blok H, Oostdam R, Otter ME et al (2002) Computer-assisted instruction in support of beginning reading instruction: a review. Rev Educ Res 72(1):101–130. https://doi.org/10.3102/003465430720011CrossRef

14.

Brysbaert M, Mitchell DC (1996) Modifier attachment in sentence parsing: evidence from Dutch. Q J Exp Psychol Sect A 49(3):664–695. https://doi.org/10.1080/713755636CrossRef

15.

Coppi AE, Oertel C, Cattaneo A (2021) Effects of experts’ annotations on fashion designers apprentices’ gaze patterns and verbalisations. Vocat Learn 14(3):511–531. https://doi.org/10.1007/s12186-021-09270-8CrossRef

16.

Crossley SA, Varner LK, Roscoe RD et al (2013) Using automated indices of cohesion to evaluate an intelligent tutoring system and an automated writing evaluation system. In: International conference on artificial intelligence in education. Springer, pp 269–278. https://doi.org/10.1007/978-3-642-39112-5_28

17.

Davison DP, Wijnen FM, Charisi V et al (2021) Words of encouragement: how praise delivered by a social robot changes children’s mindset for learning. J Multimodal User Interfaces 15:61–76. https://doi.org/10.1007/s12193-020-00353-9CrossRef

18.

Bixler ER, D’Mello KS (2021) Crossed eyes: domain adaptation for gaze-based mind wandering models. Association for Computing Machinery, New York, NY, USA, ETRA ’21 full papers. https://doi.org/10.1145/3448017.3457386

19.

Faber M, Bixler R, D’Mello SK (2018) An automated behavioral measure of mind wandering during computerized reading. Behav Res Methods 50(1):134–150. https://doi.org/10.3758/s13428-017-0857-yADSCrossRefPubMed

20.

Feng S, D’Mello S, Graesser AC (2013) Mind wandering while reading easy and difficult texts. Psychon Bull Rev 20(3):586–592. https://doi.org/10.3758/s13423-012-0367-yCrossRefPubMed

21.

Follmer DJ (2018) Executive function and reading comprehension: a meta-analytic review. Educ Psychol 53(1):42–60. https://doi.org/10.1080/00461520.2017.1309295CrossRef

22.

Garner R (1987) Metacognition and reading comprehension. Ablex Publishing, New York

23.

Gernsbacher MA, McKinney VM (1999) Comprehension: a paradigm for cognition. Am Sci 87(6):568

24.

Van der Gijp A, Ravesloot C, Jarodzka H et al (2017) How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Adv Health Sci Educ 22(3):765–787. https://doi.org/10.1007/s10459-016-9698-1CrossRef

25.

Golinkoff RM (1975) A comparison of reading comprehension processes in good and poor comprehenders. Read Res Q 11:623–659. https://doi.org/10.2307/747459CrossRef

26.

Gordon G, Breazeal C (2015) Bayesian active learning-based robot tutor for children’s word-reading skills. In: Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v29i1.9376

27.

Guthrie JT, Wigfield A, Barbosa P et al (2004) Increasing reading comprehension and engagement through concept-oriented reading instruction. J Educ Psychol 96(3):403. https://doi.org/10.1037/0022-0663.96.3.403CrossRef

28.

Harmer J (2001) The practice of English language teaching. London/New York, pp 401–405

29.

Hedge T (2001) Teaching and learning in the language classroom, vol 106. Oxford University Press, Oxford

30.

Holmqvist K, Nyström M, Andersson R et al (2011) Eye tracking: a comprehensive guide to methods and measures. Oxford University Press, Oxford

31.

Huang X, Craig SD, Xie J et al (2016) Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learn Ind Differ 47:258–265. https://doi.org/10.1016/j.lindif.2016.01.012CrossRef

32.

Hutt S, Krasich K, Mills C et al (2019) Automated gaze-based mind wandering detection during computerized learning in classrooms. User Model User Adapt Int 29(4):821–867. https://doi.org/10.1007/s11257-019-09228-5CrossRef

33.

Hyönä J, Lorch Jr RF, Rinck M (2003) Eye movement measures to study global text processing. In: The mind’s eye. Elsevier, pp 313–334. https://doi.org/10.1016/B978-044451020-4/50018-9

34.

Ince G, Yorganci R, Ozkul A et al (2021) An audiovisual interface-based drumming system for multimodal human-robot interaction. J Multimodal User Interfaces 15:413–428. https://doi.org/10.1007/s12193-020-00352-wCrossRef

35.

Kaakinen JK, Hyona J (2005) Perspective effects on expository text comprehension: evidence from think-aloud protocols, eyetracking, and recall. Discourse Process 40(3):239–257. https://doi.org/10.1207/s15326950dp4003_4CrossRef

36.

Kaakinen JK, Hyönä J (2007) Perspective effects in repeated reading: an eye movement study. Mem Cogn 35(6):1323–1336. https://doi.org/10.3758/BF03193604CrossRef

37.

Kaakinen JK, Hyönä J (2010) Task effects on eye movements during reading. J Exp Psychol Learn Mem Cogn 36(6):1561. https://doi.org/10.1037/a0020693CrossRefPubMed

38.

Kaakinen JK, Hyönä J, Keenan JM (2002) Perspective effects on online text processing. Discourse Process 33(2):159–173. https://doi.org/10.1207/S15326950DP3302_03CrossRef

39.

Kaakinen JK, Hyönä J, Keenan JM (2003) How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. J Exp Psychol Learn Mem Cogn 29(3):447–457. https://doi.org/10.1037/0278-7393.29.3.447CrossRefPubMed

40.

Kaakinen JK, Olkoniemi H, Kinnari T et al (2014) Processing of written irony: an eye movement study. Discourse Process 51(4):287–311. https://doi.org/10.1080/0163853X.2013.870024CrossRef

41.

Kassner M, Patera W, Bulling A (2014) Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In: Adjunct proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, New York, NY, USA, UbiComp ’14 Adjunct, pp 1151–1160. https://doi.org/10.1145/2638728.2641695

42.

Kennedy J, Baxter P, Senft E et al (2016) Social robot tutoring for child second language learning. In: 2016 11th ACM/IEEE international conference on human-robot interaction (HRI), pp 231–238. https://doi.org/10.1109/HRI.2016.7451757

43.

Khachatryan GA, Romashov AV, Khachatryan AR et al (2014) Reasoning mind Genie 2: an intelligent tutoring system as a vehicle for international transfer of instructional methods in mathematics. Int J Artif Intell Educ 24(3):333–382. https://doi.org/10.1007/s40593-014-0019-7CrossRef

44.

Kidd CD, Breazeal C (2007) A robotic weight loss coach. In: Proceedings of the national conference on artificial intelligence. London, AAAI Press, MIT Press, Menlo Park, Cambridge, p 1985

45.

Kontogiorgos D, Pereira A, Gustafson J (2021) Grounding behaviours with conversational interfaces: effects of embodiment and failures. J Multimodal User Interfaces 15:239–254. https://doi.org/10.1007/s12193-021-00366-yCrossRef

46.

Lee Y, Chen H, Zhao G et al (2022) Wedar: webcam-based attention analysis via attention regulator behavior recognition with a novel e-reading dataset. In: Proceedings of the 2022 international conference on multimodal interaction. Association for Computing Machinery, New York, NY, USA, ICMI ’22, pp 319–328. https://doi.org/10.1145/3536221.3556619

47.

Leite I, Martinho C, Paiva A (2013) Social robots for long-term interaction: a survey. Int J Soc Robot 5(2):291–308. https://doi.org/10.1007/s12369-013-0178-yCrossRef

48.

Leite I, Pereira A, Mascarenhas S et al (2013) The influence of empathy in human-robot relations. Int J Hum Comput Stud 71(3):250–260. https://doi.org/10.1016/j.ijhcs.2012.09.005CrossRef

49.

Leyzberg D, Spaulding S, Toneva M et al (2012) The physical presence of a robot tutor increases cognitive learning gains. In: Proceedings of the annual meeting of the cognitive science society. https://escholarship.org/uc/item/7ck0p200

50.

Li J (2015) The benefit of being physically present: a survey of experimental works comparing copresent robots, telepresent robots and virtual agents. Int J Hum Comput Stud 77:23–37. https://doi.org/10.1016/j.ijhcs.2015.01.001ADSCrossRef

51.

Lin W, Yueh HP, Wu HY et al (2014) Developing a service robot for a children’s library: a design-based research approach. J Am Soc Inf Sci 65(2):290–301. https://doi.org/10.1002/asi.22975CrossRef

52.

Michaelis JE, Mutlu B (2018) Reading socially: transforming the in-home reading experience with a learning-companion robot. Sci Robot 3(21):eaat5999. https://doi.org/10.1126/scirobotics.aat5999CrossRefPubMed

53.

Mousavinasab E, Zarifsanaiey N, NiakanKalhori RS et al (2018) Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interact Learn Environ 29(1):142–163. https://doi.org/10.1080/10494820.2018.1558257CrossRef

54.

Murray WS (2000) Commentary on section 4. Sentence processing: issues and measures. In: Kennedy A, Radach R, Heller D et al (eds) Reading as a perceptual process. North-Holland/Elsevier Science Publishers, pp 649–664. https://doi.org/10.1016/B978-008043642-5/50030-9

55.

National Reading Panel (US) NIoCH, (US) HD (2000) Teaching children to read: an evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups. National Institute of Child Health and Human Development, National Institutes of Health

56.

Oertel C, Coppi A, Olsen JK, et al (2019) On the use of gaze as a measure for performance in a visual exploration task. In: European conference on technology enhanced learning. Springer, pp 386–395. https://doi.org/10.1007/978-3-030-29736-7_29

57.

Olson E (2011) AprilTag: a robust and flexible visual fiducial system. In: 2011 IEEE international conference on robotics and automation. IEEE, pp 3400–3407. https://doi.org/10.1109/ICRA.2011.5979561

58.

Paris SG, Cross DR, Lipson MY (1984) Informed strategies for learning: a program to improve children’s reading awareness and comprehension. J Educ Psychol 76(6):1239CrossRef

59.

Paris SG, Wasik B, Turner JC (1991) The development of strategic readers. In: Barr R, Kamil ML, Mosenthal PB et al (eds) Handbook of reading research, vol 2. Lawrence Erlbaum Associates Inc, Mahwah, pp 609–640

60.

Pernice K, Whitenton K, Nielsen J et al (2014) How people read online: the eyetracking evidence. Nielsen Norman Group, Fremont

61.

Pollard-Durodola SD, Gonzalez JE, Simmons DC et al (2011) The effects of an intensive shared book-reading intervention for preschool children at risk for vocabulary delay. Except Child 77(2):161–183. https://doi.org/10.1177/001440291107700202CrossRef

62.

Pourhosein Gilakjani A, Sabouri NB (2016) How can students improve their reading comprehension skill. J Stud Educ 6(2):229–240. https://doi.org/10.5296/jse.v6i2.9201CrossRef

63.

Pressley M, El-Dinary PB, Brown R (1992) Skilled and not-so-skilled reading: Good information processing and not-so-good information processing. In: Pressley M, Harris KR, Guthrie JT (eds) Promoting academic competence and literacy in school. Academic Press, pp 91–127

64.

Rajendran R, Kumar A, Carter KE et al (2018) Predicting learning by analyzing eye-gaze data of reading behavior. Int Educ Data Min Soc 16–20. https://api.semanticscholar.org/CorpusID:52173770

65.

Raney GE, Campbell SJ, Bovee JC (2014) Using eye movements to evaluate the cognitive processes involved in text comprehension. J Vis Exp JoVE 83:e50,780. https://doi.org/10.3791/50780CrossRef

66.

Reichle ED, Pollatsek A, Fisher DL et al (1998) Toward a model of eye movement control in reading. Psychol Rev 105(1):125–157. https://doi.org/10.1037/0033-295X.105.1.125CrossRefPubMed

67.

Reynolds RE (2000) Attentional resource emancipation: toward understanding the interaction of word identification and comprehension processes in reading. Sci Stud Read 4(3):169–195. https://doi.org/10.1207/S1532799XSSR0403_1CrossRef

68.

Rumpf C, Boronczyk F, Breuer C (2020) Predicting consumer gaze hits: a simulation model of visual attention to dynamic marketing stimuli. J Bus Res 111:208–217. https://doi.org/10.1016/j.jbusres.2019.03.034CrossRef

69.

Savitzky A, Golay MJ (1964) Smoothing and differentiation of data by simplified least squares procedures. Anal Chem 36(8):1627–1639ADSCrossRef

70.

Sevcenko N, Appel T, Ninaus M et al (2023) Theory-based approach for assessing cognitive load during time-critical resource-managing human-computer interactions: an eye-tracking study. J Multimodal User Interfaces 17(1):1–19. https://doi.org/10.1007/s12193-022-00398-yCrossRef

71.

Shrestha S, Lenz K, Chaparro B et al (2007) “f” pattern scanning of text and images in web pages. In: Proceedings of the human factors and ergonomics society annual meeting. SAGE Publications, Los Angeles, pp 1200–1204. https://doi.org/10.1177/154193120705101831

72.

Steenbergen-Hu S, Cooper H (2013) A meta-analysis of the effectiveness of intelligent tutoring systems on k-12 students’ mathematical learning. J Educ Psychol 105(4):970–987. https://doi.org/10.1037/a0032447CrossRef

73.

Sugimoto M (2011) A mobile mixed-reality environment for children’s storytelling using a handheld projector and a robot. IEEE Trans Learn Technol 4(3):249–260. https://doi.org/10.1109/TLT.2011.13CrossRef

74.

Van Dijk TA, Kintsch W et al (1983) Strategies of discourse comprehension. Psychology 6(6):12

75.

Wang YH, Young SSC, Jang JSR (2013) Using tangible companions for enhancing learning English conversation. J Educ Technol Soc 16(2):296–309

76.

Weinfurt KP (1995) Multivariate analysis of variance. In: Grimm LG, Yarnold PR (eds) Reading and understanding multivariate statistics. American Psychological Association, pp 245–276

77.

Weisstein EW (2004) Bonferroni correction. https://mathworld.wolfram.com/BonferroniCorrection.html

78.

Wijekumar K, Meyer BJ, Lei P et al (2020) Supplementing teacher knowledge using web-based intelligent tutoring system for the text structure strategy to improve content area reading comprehension with fourth-and fifth-grade struggling readers. Dyslexia 26(2):120–136. https://doi.org/10.1002/dys.1634

79.

Yadollahi E, Johal W, Paiva A et al (2018) When deictic gestures in a robot can harm child-robot collaboration. In: Proceedings of the 17th ACM conference on interaction design and children, pp 195–206. https://doi.org/10.1145/3202185.3202743

80.

Yang W, Dai W, Gao L (2012) Intensive reading and necessity to integrate learning strategies. Engl Lang Lit 2(1):55–63. https://doi.org/10.5539/ells.v2n1p112CrossRef

81.

Yueh HP, Lin W, Wang SC et al (2020) Reading with robot and human companions in library literacy activities: a comparison study. Br J Edu Technol 51(5):1884–1900. https://doi.org/10.1111/bjet.13016CrossRef

82.

Zhao Q, Yuan X, Tu D et al (2015) Eye moving behaviors identification for gaze tracking interaction. J Multimodal User Interfaces 9:89–104. https://doi.org/10.1007/s12193-014-0171-2

83.

Zwaan RA, Singer M (2003) Text comprehension. In: Graesser AC, Gernsbacher MA, Goldman SR (eds) Handbook of discourse processes. Lawrence Erlbaum Associates Publishers, Mahwah, pp 83–121

Title: A social robot as your reading companion: exploring the relationships between gaze patterns and knowledge gains
Authors: Xuan Liu
Jiachen Ma
Qiang Wang
Publication date: 12-10-2023
Publisher: Springer International Publishing
Published in: Journal on Multimodal User Interfaces / Issue 1/2024
Print ISSN: 1783-7677
Electronic ISSN: 1783-8738
DOI: https://doi.org/10.1007/s12193-023-00418-5

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 1/2024

Modelling the “transactive memory system” in multimodal multiparty interactions

A study on the attention of people with low vision to accessibility guidance signs

Hearing loss prevention at loud music events via real-time visuo-haptic feedback

Pegasos: a framework for the creation of direct mobile coaching feedback systems

Comparing alternative modalities in the context of multimodal human–robot interaction

Review of substitutive assistive tools and technologies for people with visual impairments: recent advancements and prospects

Premium Partner