Skip to main content
Top
Published in:
Cover of the book

2021 | OriginalPaper | Chapter

Narrative Cognition in Mixed Reality Systems: Towards an Empirical Framework

Authors : Luis Emilio Bruni, Hossein Dini, Aline Simonetti

Published in: Virtual, Augmented and Mixed Reality

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

In this paper, we propose an interdisciplinary theoretical and empirical framework to investigate the particular faculties related to human “narrative cognition”, in general, and in relation to MRT in particular. In order to contextualize our approach, we shortly review the cognitive turn in narratology, as well as state of the art in different domains that have undertaken psychophysiological studies that either characterize aspects that are relevant to narrative cognition, or which investigate mixed reality experiences. The idea is to bring together knowledge and insights from narratology, different branches of semiotics and cognitive sciences, with empirical strategies that bridge the gap between first-person phenomenological approaches and psychophysiological and behavioural methods. We propose a rationale in order to combine tools and techniques from MRT/VR/AR, interactive digital narratives and storytelling, with a suite of integrated psychophysiological methods (such as EEG, HR, GSR and eye tracking) and phenomenological-subjective approaches.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Broniatowski, D.A., Reyna, V.F.: Gist and verbatim in narrative memory. In: Finlayson, M.A., Fisseni, B., Löwe, B., Meister, J.C. (eds.) 2013 Workshop on Computational Models of Narrative, pp. 43–51. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl (2013). https://doi.org/10.4230/OASIcs.CMN.2013.43 Broniatowski, D.A., Reyna, V.F.: Gist and verbatim in narrative memory. In: Finlayson, M.A., Fisseni, B., Löwe, B., Meister, J.C. (eds.) 2013 Workshop on Computational Models of Narrative, pp. 43–51. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl (2013). https://​doi.​org/​10.​4230/​OASIcs.​CMN.​2013.​43
2.
go back to reference Bolioli, A., Casu, M., Lana, M., Roda, R.: Exploring the betrothed lovers. In: Finlayson, M.A., Fisseni, B., Löwe, B., Meister, J.C. (eds.) 2013 Workshop on Computational Models of Narrative, pp. 30–35. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl (2013). https://doi.org/10.4230/OASIcs.CMN.2013.30 Bolioli, A., Casu, M., Lana, M., Roda, R.: Exploring the betrothed lovers. In: Finlayson, M.A., Fisseni, B., Löwe, B., Meister, J.C. (eds.) 2013 Workshop on Computational Models of Narrative, pp. 30–35. Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl (2013). https://​doi.​org/​10.​4230/​OASIcs.​CMN.​2013.​30
3.
go back to reference Bhatt, M., Suchan, J., Schultz, C.: Cognitive interpretation of everyday activities: toward perceptual narrative based visuo-spatial scene interpretation. In: Workshop on Computational Models of Narrative, pp. 24–29. Leibniz-Zentrum für Informatik (2013). https://doi.org/10.4230/OASIcs.CMN.2013.24 Bhatt, M., Suchan, J., Schultz, C.: Cognitive interpretation of everyday activities: toward perceptual narrative based visuo-spatial scene interpretation. In: Workshop on Computational Models of Narrative, pp. 24–29. Leibniz-Zentrum für Informatik (2013). https://​doi.​org/​10.​4230/​OASIcs.​CMN.​2013.​24
6.
go back to reference Szilas, N.: Interactive drama on computer: beyond linear narrative. In: AAAI Fall Symposium on Narrative Intelligence, pp. 150–156 (1999) Szilas, N.: Interactive drama on computer: beyond linear narrative. In: AAAI Fall Symposium on Narrative Intelligence, pp. 150–156 (1999)
7.
go back to reference Schoenau-Fog, H., Bruni, L.E., Khalil, F.F., Faizi, J.: First person victim: developing a 3D interactive dramatic experience. In: Aylett, R., Lim, M.Y., Louchart, S., Petta, P., Riedl, M. (eds) Interactive Storytelling, ICIDS 2010, Lecture Notes in Computer Science, vol. 6432. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16638-9_32 Schoenau-Fog, H., Bruni, L.E., Khalil, F.F., Faizi, J.: First person victim: developing a 3D interactive dramatic experience. In: Aylett, R., Lim, M.Y., Louchart, S., Petta, P., Riedl, M. (eds) Interactive Storytelling, ICIDS 2010, Lecture Notes in Computer Science, vol. 6432. Springer, Heidelberg (2010). https://​doi.​org/​10.​1007/​978-3-642-16638-9_​32
8.
go back to reference Baceviciute, S., Rützou Albæk, K.R., Arsovski, A., Bruni, L.E.: Digital interactive narrative tools for facilitating communication with children during counseling: a case for audiology. In: Oyarzun, D., Peinado, F., Young, R.M., Elizalde, A., Méndez, G. (eds) Interactive Storytelling, ICIDS 2012, Lecture Notes in Computer Science, vol. 7648. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34851-8_5 Baceviciute, S., Rützou Albæk, K.R., Arsovski, A., Bruni, L.E.: Digital interactive narrative tools for facilitating communication with children during counseling: a case for audiology. In: Oyarzun, D., Peinado, F., Young, R.M., Elizalde, A., Méndez, G. (eds) Interactive Storytelling, ICIDS 2012, Lecture Notes in Computer Science, vol. 7648. Springer, Heidelberg (2012). https://​doi.​org/​10.​1007/​978-3-642-34851-8_​5
9.
go back to reference Mulholland, P., Wolff, A., Zdrahal, Z., Li, N., Corneli, J.: Constructing and connecting storylines to tell museum stories. In: Koenitz, H., Sezen, T.I., Ferri, G., Haahr, M., Sezen, D., Çatak, G. (eds) Interactive Storytelling, ICIDS 2013, Lecture Notes in Computer Science, vol. 8230. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-02756-2_14 Mulholland, P., Wolff, A., Zdrahal, Z., Li, N., Corneli, J.: Constructing and connecting storylines to tell museum stories. In: Koenitz, H., Sezen, T.I., Ferri, G., Haahr, M., Sezen, D., Çatak, G. (eds) Interactive Storytelling, ICIDS 2013, Lecture Notes in Computer Science, vol. 8230. Springer, Cham (2013). https://​doi.​org/​10.​1007/​978-3-319-02756-2_​14
10.
go back to reference Kalogeras, S.: Media-education convergence: applying transmedia storytelling edutainment in e-learning environments. Int. J. Inf. Commun. Technol. Educ. 9, 1–11 (2013)CrossRef Kalogeras, S.: Media-education convergence: applying transmedia storytelling edutainment in e-learning environments. Int. J. Inf. Commun. Technol. Educ. 9, 1–11 (2013)CrossRef
11.
go back to reference Marsh, T., Nickole, L.Z., Klopfer, E., Xuejin, C., Osterweil, S., Haas, J.: Fun and learning: blending design and development dimensions in serious games through narrative and characters. In: Ma, M., Oikonomou, A., Jain, L. (eds.) Serious Games and Edutainment Applications. Springer, London (2011). https://doi.org/10.1007/978-1-4471-2161-9_14 Marsh, T., Nickole, L.Z., Klopfer, E., Xuejin, C., Osterweil, S., Haas, J.: Fun and learning: blending design and development dimensions in serious games through narrative and characters. In: Ma, M., Oikonomou, A., Jain, L. (eds.) Serious Games and Edutainment Applications. Springer, London (2011). https://​doi.​org/​10.​1007/​978-1-4471-2161-9_​14
13.
go back to reference Guger, C., Holzner, C., Grönegress, C., Edlinger, G., Slater, M.: Control of a Smart Home with a Brain-Computer Interface (2008) Guger, C., Holzner, C., Grönegress, C., Edlinger, G., Slater, M.: Control of a Smart Home with a Brain-Computer Interface (2008)
17.
go back to reference Bruner, J.: Acts of Meaning. Harvard University Press, Cambridge (1990) Bruner, J.: Acts of Meaning. Harvard University Press, Cambridge (1990)
18.
go back to reference Polkinghorne, D.E.: Narrative Knowing and the Human Sciences. Suny Press, Albany (1988) Polkinghorne, D.E.: Narrative Knowing and the Human Sciences. Suny Press, Albany (1988)
19.
go back to reference Ricoeur, P.: Time and Narrative. University of Chicago Press, Chicago (1984) Ricoeur, P.: Time and Narrative. University of Chicago Press, Chicago (1984)
20.
go back to reference Finlayson, M.A., et al.: Workshop on computational models of narrative. In: CMN 2013, Saarbrücken/WadernSchloss Dagstuhl-Leibniz-Zentrum für Informatik, Hamburg (2013) Finlayson, M.A., et al.: Workshop on computational models of narrative. In: CMN 2013, Saarbrücken/WadernSchloss Dagstuhl-Leibniz-Zentrum für Informatik, Hamburg (2013)
21.
go back to reference Sanford, A.J., Emmott, C.: Mind, Brain and Narrative. Cambridge University Press, Cambridge (2012)CrossRef Sanford, A.J., Emmott, C.: Mind, Brain and Narrative. Cambridge University Press, Cambridge (2012)CrossRef
22.
go back to reference Ryan, M.-L.: Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Johns Hopkins University Press, Baltimore (2001) Ryan, M.-L.: Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Johns Hopkins University Press, Baltimore (2001)
23.
go back to reference Ryan, M.-L.: Avatars of Story. U of Minnesota Press, Minneapolis (2006) Ryan, M.-L.: Avatars of Story. U of Minnesota Press, Minneapolis (2006)
24.
go back to reference Herman, D.: Cognitive narratology. In: Handbook of Narratology, pp. 46–64. de Gruyter, Berlin (2009) Herman, D.: Cognitive narratology. In: Handbook of Narratology, pp. 46–64. de Gruyter, Berlin (2009)
26.
go back to reference Cobley, P.: Narrative. Routledge, London (2014) Cobley, P.: Narrative. Routledge, London (2014)
28.
go back to reference Ranta, M.: Stories in pictures (and non-pictorial objects): a narratological and cognitive psychological approach. Contemp. Aesthetics 9, 6 (2011) Ranta, M.: Stories in pictures (and non-pictorial objects): a narratological and cognitive psychological approach. Contemp. Aesthetics 9, 6 (2011)
29.
go back to reference Matuk, C.F.: Narratives in mind and media: a cognitive semiotic account of novices interpreting visual science media (2010) Matuk, C.F.: Narratives in mind and media: a cognitive semiotic account of novices interpreting visual science media (2010)
33.
go back to reference Ryan, M.L.: Narratology and cognitive science: a problematic relation. Style. 44, 469–495 (2010) Ryan, M.L.: Narratology and cognitive science: a problematic relation. Style. 44, 469–495 (2010)
35.
go back to reference Bruni, L.E., Baceviciute, S., Arief, M.: Narrative cognition in interactive systems: suspense-surprise and the P300 ERP component. In: Mitchell, A., Fernández-Vara, C., Thue, D. (eds.) Lecture Notes in Computer Science, pp. 164–175. Springer International Publishing, Cham (2014). https://doi.org/10.1007/978-3-319-12337-0_17 Bruni, L.E., Baceviciute, S., Arief, M.: Narrative cognition in interactive systems: suspense-surprise and the P300 ERP component. In: Mitchell, A., Fernández-Vara, C., Thue, D. (eds.) Lecture Notes in Computer Science, pp. 164–175. Springer International Publishing, Cham (2014). https://​doi.​org/​10.​1007/​978-3-319-12337-0_​17
49.
go back to reference Bruni, L.E., Baceviciute, S.: Narrative intelligibility and closure in interactive systems. In: Koenitz, H., Sezen, T.I., Ferri, G., Haahr, M., Sezen, D., Çatak, G. (eds.) Interactive Storytelling, pp. 13–24. Springer International Publishing, Cham (2013). https://doi.org/10.1007/978-3-319-02756-2_2 Bruni, L.E., Baceviciute, S.: Narrative intelligibility and closure in interactive systems. In: Koenitz, H., Sezen, T.I., Ferri, G., Haahr, M., Sezen, D., Çatak, G. (eds.) Interactive Storytelling, pp. 13–24. Springer International Publishing, Cham (2013). https://​doi.​org/​10.​1007/​978-3-319-02756-2_​2
60.
go back to reference Cohn, N., Paczynski, M., Holcomb, P., Jackendoff, R., Kuperberg, G.: Comics on the brain: structure and meaning in sequential image comprehension. In: Proceedings of the Annual Meeting of the Cognitive Science Society (2011) Cohn, N., Paczynski, M., Holcomb, P., Jackendoff, R., Kuperberg, G.: Comics on the brain: structure and meaning in sequential image comprehension. In: Proceedings of the Annual Meeting of the Cognitive Science Society (2011)
64.
go back to reference Gilroy, S.W., et al.: A brain-computer interface to a plan-based narrative. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pp. 1997–2005. AAAI Press (2013) Gilroy, S.W., et al.: A brain-computer interface to a plan-based narrative. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pp. 1997–2005. AAAI Press (2013)
65.
go back to reference Ekman, I., et al.: Review on psychophysiological methods in game research. In: Proceedings of the 2010 International DiGRA Nordic Conference: Experiencing Games: Games, Play, and Players, DiGRA Nordic 2010 (2010) Ekman, I., et al.: Review on psychophysiological methods in game research. In: Proceedings of the 2010 International DiGRA Nordic Conference: Experiencing Games: Games, Play, and Players, DiGRA Nordic 2010 (2010)
67.
go back to reference Gjøl, B.A., Jørgensen, N.V., Thomsen, M.R., Bruni, L.E.: Predictability and plausibility in interactive narrative constructs: a case for an ERP study. In: Rouse, R., Koenitz, H., Haahr, M. (eds.) Interactive Storytelling, pp. 121–133. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-04028-4_9 Gjøl, B.A., Jørgensen, N.V., Thomsen, M.R., Bruni, L.E.: Predictability and plausibility in interactive narrative constructs: a case for an ERP study. In: Rouse, R., Koenitz, H., Haahr, M. (eds.) Interactive Storytelling, pp. 121–133. Springer International Publishing, Cham (2018). https://​doi.​org/​10.​1007/​978-3-030-04028-4_​9
81.
go back to reference Marwecki, S., Wilson, A.D., Ofek, E., Gonzalez Franco, M., Holz, C.: Mise-unseen: using eye-tracking to hide virtual reality scene changes in plain sight. In: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, pp. 777–789. ACM, New York (2019). https://doi.org/10.1145/3332165.3347919 Marwecki, S., Wilson, A.D., Ofek, E., Gonzalez Franco, M., Holz, C.: Mise-unseen: using eye-tracking to hide virtual reality scene changes in plain sight. In: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, pp. 777–789. ACM, New York (2019). https://​doi.​org/​10.​1145/​3332165.​3347919
84.
go back to reference Bee, N., et al.: Discovering eye gaze behavior during human-agent conversation in an interactive storytelling application. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction on - ICMI-MLMI 2010, p. 1. ACM Press, New York (2010). https://doi.org/10.1145/1891903.1891915 Bee, N., et al.: Discovering eye gaze behavior during human-agent conversation in an interactive storytelling application. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction on - ICMI-MLMI 2010, p. 1. ACM Press, New York (2010). https://​doi.​org/​10.​1145/​1891903.​1891915
85.
go back to reference Gere, C.: Digital Culture. Reaktion Books, London (2009) Gere, C.: Digital Culture. Reaktion Books, London (2009)
90.
go back to reference Goodman, N.: Twisted tales: story, study and symphony. In: Mitchell, W.J.T. (ed.) Narrative, Chicago (1981) Goodman, N.: Twisted tales: story, study and symphony. In: Mitchell, W.J.T. (ed.) Narrative, Chicago (1981)
93.
94.
go back to reference Goodson, I., Gill, S.R.: Narrative Pedagogy: Life History and Learning. Peter Lang, Bern (2011) Goodson, I., Gill, S.R.: Narrative Pedagogy: Life History and Learning. Peter Lang, Bern (2011)
95.
go back to reference White, M., Wijaya, M., White, M.K., Epston, D.: Narrative Means to Therapeutic Ends. WW Norton & Company, New York (1990) White, M., Wijaya, M., White, M.K., Epston, D.: Narrative Means to Therapeutic Ends. WW Norton & Company, New York (1990)
96.
go back to reference Kleinreesink, E., Moelker, R., Richardson, R.: Books and Bikes. Amsterdam University Press, Amsterdam (2012)CrossRef Kleinreesink, E., Moelker, R., Richardson, R.: Books and Bikes. Amsterdam University Press, Amsterdam (2012)CrossRef
97.
go back to reference Corman, S.R.: Narrating the Exit from Afghanistan. Tempe: Center for Strategic Communication (2013) Corman, S.R.: Narrating the Exit from Afghanistan. Tempe: Center for Strategic Communication (2013)
98.
go back to reference Varela, F.J., Rosch, E., Thompson, E.: The Embodied Mind. MIT Press, Cambridge (1993) Varela, F.J., Rosch, E., Thompson, E.: The Embodied Mind. MIT Press, Cambridge (1993)
Metadata
Title
Narrative Cognition in Mixed Reality Systems: Towards an Empirical Framework
Authors
Luis Emilio Bruni
Hossein Dini
Aline Simonetti
Copyright Year
2021
DOI
https://doi.org/10.1007/978-3-030-77599-5_1