Top

Published in:

2017 | OriginalPaper | Chapter

12. Interaction History in Adaptive Multimodal Interaction

Authors : Nikola Bubalo, Felix Schüssel, Frank Honold, Michael Weber, Anke Huckauf

Published in: Companion Technology

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Modern Companion-Technologies provide multimodal and adaptive interaction possibilities. However, it is still unclear which user characteristics should be used in which manner to optimally support the interaction. An important aspect is that users themselves learn and adapt their behavior and preferences based on their own experiences. In other words, certain characteristics of user behavior are slowly but continuously changed and updated by the users themselves over multiple encounters with the Companion-Technology. Thus, a biological adaptive multimodal system observes and interacts with an electronic one, and vice versa. Consequently, such a user-centered interaction history is essential and should be integrated in the prediction of user behavior. Doing so enables the Companion to achieve more robust predictions of user behavior, which in turn leads to better fusion decisions and more efficient customization of the UI. We present the development of an experimental paradigm based on visual search tasks. The setup allows the induction of various user experiences as well as the testing of their effects on user behavior and preferences during multimodal interaction.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Interaction with Adaptive and Ubiquitous User Interfaces

next chapter LAST MINUTE: An Empirical Experiment in User-Companion Interaction and Its Evaluation

Ailomaa, M., Melichar, M., Rajman, M., Lisowska, A., Armstrong, S.: Archivus: a multimodal system for multimedia meeting browsing and retrieval. In: Proceedings of the COLING/ACL on Interactive Presentation Sessions, pp. 49–52. Association for Computational Linguistics, Stroudsburg, PA (2006)

Ajzen, I.: Attitudes, Personality, and Behavior. McGraw-Hill International, Maidenhead (2005)

Bellik, Y., Rebaï, I., Machrouh, E., Barzaj, Y., Jacquet, C., Pruvost, G., Sansonnet, J.P.: Multimodal interaction within ambient environments: an exploratory study. In: Human-Computer Interaction–INTERACT 2009, pp. 89–92. Springer, Berlin (2009)

Biundo, S., Wendemuth, A.: Companion-technology for cognitive technical systems. Künstl. Intell. 30(1), 71–75 (2016). Special issue on companion technologies

Bolt, R.A.: “Put-That-There”: Voice and Gesture at the Graphics Interface, vol. 14. ACM, New York (1980)

Camp, G., Paas, F., Rikers, R., van Merrienboer, J.: Dynamic problem selection in air traffic control training: a comparison between performance, mental effort and mental efficiency. Comput. Hum. Behav. 17(5), 575–595 (2001)CrossRef

Carter, S., Mankoff, J., Klemmer, S.R., Matthews, T.: Exiting the cleanroom: on ecological validity and ubiquitous computing. Hum. Comput. Interact. 23(1), 47–99 (2008)CrossRef

Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, pp. 31–40. ACM, New York (1997)

De Angeli, A., Gerbino, W., Cassano, G., Petrelli, D.: Visual display, pointing, and natural language: the power of multimodal interaction. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 164–173. ACM, New York (1998)

10.

Dey, P., Madhvanath, S., Ranjan, A., Das, S.: An exploration of gesture-speech multimodal patterns for touch interfaces. In: Proceedings of the 3rd International Conference on Human Computer Interaction, pp. 79–83. ACM, New York (2011)

11.

Domjan, M.: The principles of learning and behavior. Cengage Learning, Stamford, CT (2014)

12.

Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction. Lecture Notes in Computer Science, vol. 5440, pp. 3–26. Springer, Berlin, Heidelberg (2009)CrossRef

13.

Haas, E.C., Pillalamarri, K.S., Stachowiak, C.C., McCullough, G.: Temporal binding of multimodal controls for dynamic map displays: a systems approach. In: Proceedings of the 13th International Conference on Multimodal Interfaces, pp. 409–416. ACM, New York (2011)

14.

Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)CrossRef

15.

Hollender, N., Hofmann, C., Deneke, M., Schmitz, B.: Integrating cognitive load theory and concepts of human–computer interaction. Comput. Hum. Behav. 26(6), 1278–1288 (2010)CrossRef

16.

Jaimes, A., Sebe, N.: Multimodal human–computer interaction: a survey. Comput. Vis. Image Underst. 108(1), 116–134 (2007)CrossRef

17.

Jöst, M., Häußler, J., Merdes, M., Malaka, R.: Multimodal interaction for pedestrians: an evaluation study. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, pp. 59–66. ACM, New York (2005)

18.

Käster, T., Pfeiffer, M., Bauckhage, C.: Combining speech and haptics for intuitive and efficient navigation through image databases. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 180–187. ACM, New York (2003)

19.

Kieffer, S., Carbonell, N.: How really effective are multimodal hints in enhancing visual target spotting? Some evidence from a usability study. J. Multimodal User Interfaces 1(1), 1–5 (2007)CrossRef

20.

Koons, D.B., Sparrell, C.J., Thorisson, K.R.: Integrating Simultaneous Input from Speech, Gaze, and Hand Gestures. MIT, Menlo Park, CA, pp. 257–276 (1993)

21.

Kruijff-Korbayová, I., Blaylock, N., Gerstenberger, C., Rieser, V., Becker, T., Kaisser, M., Poller, P., Schehl, J.: An experiment setup for collecting data for adaptive output planning in a multimodal dialogue system. In: Proceedings of ENLG (2005)

22.

Lalanne, D., Nigay, L., Robinson, P., Vanderdonckt, J., Ladry, J.F., et al.: Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, pp. 153–160. ACM, New York (2009)

23.

Lee, M., Billinghurst, M.: A wizard of oz study for an ar multimodal interface. In: Proceedings of the 10th International Conference on Multimodal Interfaces, pp. 249–256. ACM, New York (2008)

24.

Lee, J.H., Poliakoff, E., Spence, C.: The effect of multimodal feedback presented via a touch screen on the performance of older adults. In: Haptic and Audio Interaction Design, pp. 128–135. Springer, Berlin (2009)

25.

Mignot, C., Valot, C., Carbonell, N.: An experimental study of future “natural” multimodal human-computer interaction. In: INTERACT’93 and CHI’93 Conference Companion on Human Factors in Computing Systems, pp. 67–68. ACM, New York (1993)

26.

Müller, H.J., Krummenacher, J.: Visual search and selective attention. Vis. Cogn. 14(4–8), 389–410 (2006)CrossRef

27.

Neal, J.G., Shapiro, S.C.: Intelligent multi-media interface technology. ACM SIGCHI Bull. 20(1), 75–76 (1988)CrossRef

28.

Ouellette, J.A., Wood, W.: Habit and intention in everyday life: the multiple processes by which past behavior predicts future behavior. Psychol. Bull. 124(1), 54 (1998)CrossRef

29.

Oviatt, S.: Multimodal interactive maps: designing for human performance. Hum. Comput. Interact. 12(1), 93–129 (1997)CrossRef

30.

Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Hum. Comput. Interact. 15(4), 263–322 (2000)CrossRef

31.

Oviatt, S., Coulston, R., Lunsford, R.: When do we interact multimodally?: Cognitive load and multimodal communication patterns. In: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 129–136. ACM, New York (2004)

32.

Oviatt, S., Lunsford, R., Coulston, R.: Individual differences in multimodal integration patterns: what are they and why do they exist? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 241–249. ACM, New York (2005)

33.

Oviatt, S., Arthur, A., Cohen, J.: Quiet interfaces that help students think. In: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, pp. 191–200. ACM, New York (2006)

34.

Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)CrossRef

35.

Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford University Press, Oxford (1990)CrossRef

36.

Ratzka, A.: Explorative studies on multimodal interaction in a PDA- and desktop-based scenario. In: Proceedings of the 10th International Conference on Multimodal Interfaces, pp. 121–128. ACM, New York (2008)

37.

Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.C., McTear, M., Raman, T., Stanney, K.M., Su, H., Wang, Q.Y.: Guidelines for multimodal user interface design. Commun. ACM 47(1), 57–59 (2004)CrossRef

38.

Reis, T., de Sá, M., Carriço, L.: Multimodal interaction: real context studies on mobile digital artefacts. In: Haptic and Audio Interaction Design, pp. 60–69. Springer, Berlin (2008)

39.

Ren, X., Zhang, G., Dai, G.: An experimental study of input modes for multimodal human-computer interaction. In: Advances in Multimodal Interfaces-ICMI 2000, pp. 49–56. Springer, Berlin (2000)

40.

Ruiz, N., Chen, F., Oviatt, S.: Multimodal input. In: Multimodal Signal Processing: Theory and Applications for Human-Computer Interaction, p. 231. Academic, Boston (2009)

41.

Savidis, A., Stephanidis, C.: Unified user interface design: designing universally accessible interactions. Interact. Comput. 16(2), 243–270 (2004)CrossRef

42.

Schüssel, F., Honold, F., Weber, M.: Influencing factors on multimodal interaction during selection tasks. J. Multimodal User Interfaces 7(4), 299–310 (2013)CrossRef

43.

Schüssel, F., Honold, F., Weber, M., Schmidt, M., Bubalo, N., Huckauf, A.: Multimodal interaction history and its use in error detection and recovery. In: Proceedings of the 16th ACM International Conference on Multimodal Interaction, ICMI ’14, pp. 164–171. ACM, New York (2014). doi:10.1145/2663204.2663255

44.

Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cogn. Psychol. 12(1), 97–136 (1980)CrossRef

45.

Van Merrienboer, J.J., Sweller, J.: Cognitive load theory and complex learning: recent developments and future directions. Educ. Psychol. Rev. 17(2), 147–177 (2005)CrossRef

46.

Wasinger, R., Krüger, A.: Modality preferences in mobile and instrumented environments. In: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 336–338. ACM, New York (2006)

47.

Wickens, C.D.: Multiple resources and mental workload. Hum. Factors J. Hum. Factors Ergon. Soc. 50(3), 449–455 (2008)CrossRef

Title: Interaction History in Adaptive Multimodal Interaction
Authors: Nikola Bubalo
Felix Schüssel
Frank Honold
Michael Weber
Anke Huckauf
Publisher: Springer International Publishing
Book: Companion Technology
Print ISBN: 978-3-319-43664-7

Electronic ISBN: 978-3-319-43665-4

Copyright Year: 2017
DOI: https://doi.org/10.1007/978-3-319-43665-4_12

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner