Top

Published in:

2017 | OriginalPaper | Chapter

10. Management of Multimodal User Interaction in Companion-Systems

Authors : Felix Schüssel, Frank Honold, Nikola Bubalo, Michael Weber, Anke Huckauf

Published in: Companion Technology

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

While interacting, human beings continuously adapt their way of communication to their surroundings and their communication partner. Although present context-aware ubiquitous systems gather a lot of information to maximize their functionality, they predominantly offer rather static ways to communicate. In order to fulfill the user’s communication needs and demands, ubiquitous sensors’ varied information could be used to dynamically adapt the user interface. Considering such an adaptive user interface management as a major and relevant component for a Companion-Technology, we also have to cope with emotional and dispositional user input as a source of implicit user requests and demands. In this chapter we demonstrate how multimodal fusion based on evidential reasoning and probabilistic fission with adaptive reasoning can act together to form a highly adaptive and model-driven interactive system component for multimodal interaction. The presented interaction management (IM) can handle uncertain or ambiguous data throughout the complete interaction cycle with a user. In addition, we present the IM’s architecture and its model-driven concept. Finally, we discuss its role within the framework of the other constituents of a Companion-Technology.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Assistive and Adaptive Dialog Management

next chapter Interaction with Adaptive and Ubiquitous User Interfaces

Simplified action cycle process based on [35]: form a goal → plan an action → execute an action → perceive feedback → evaluate the new state of the world → ….

Ameri Ekhtiarabadi, A., Akan, B., Çürüklu, B., Asplund, L.: A general framework for incremental processing of multimodal inputs. In: Proceedings of the 13th International Conference on Multimodal Interface, ICMI ’11, pp. 225–228. ACM, New York (2011). doi:10.1145/2070481.2070521

Atrey, P., Hossain, M.A., El Saddik, A., Kankanhalli, M.: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16, 345–379 (2010)CrossRef

Bertrand, G., Nothdurft, F., Honold, F., Schüssel, F.: CALIGRAPHI - Creation of Adaptive Dialogues Using a Graphical Interface. In: 2011 IEEE 35th Annual Computer Software and Applications Conference (COMPSAC), pp. 393–400. IEEE, New York (2011). doi:10.1109/COMPSAC.2011.58

Biundo, S., Wendemuth, A.: Companion-technology for cognitive technical systems. Künstl. Intell. (2016). doi:10.1007/s13218-015-0414-8

Blumendorf, M., Roscher, D., Albayrak, S.: Dynamic user interface distribution for flexible multimodal interaction. In: ICMI and the Workshop on Machine Learning for Multimodal Interaction. ACM, New York (2010). doi:10.1145/1891903.1891930CrossRef

Brandes, U., Eiglsperger, M., Herman, I., Himsolt, M., Marshall, M.: GraphML progress report: structural layer proposal. In: Mutzel, P., Jünger, M., Leipert, S. (eds.) Graph Drawing. Lecture Notes in Computer Science, vol. 2265, pp. 501–512. Springer, Berlin, Heidelberg (2002). doi:10.1007/3-540-45848-4_59

Calvary, G., Coutaz, J., Thevenin, D., Bouillon, L., Florins, M., Limbourg, Q., Souchon, N., Vanderdonckt, J., Marucci, L., Paternò, F., Santoro, C.: The Cameleon reference framework. Technical Report 1.1. CAMELEON Reference Framework Working Group (2002)

Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, MULTIMEDIA ’97, pp. 31–40. ACM, New York (1997). doi:10.1145/266180.266328

Costa, D., Duarte, C.: Adapting multimodal fission to user’s abilities. In: Proceedings of the 6th International Conference on Universal Access in Human-Computer Interaction: Design for All and EInclusion - Volume Part I, UAHCI’11, pp. 347–356. Springer, Berlin (2011)

10.

Cutugno, F., Leano, V.A., Rinaldi, R., Mignini, G.: Multimodal framework for mobile interaction. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI ’12, pp. 197–203. ACM, New York (2012). doi:10.1145/2254556.2254592

11.

Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: HUC ’99: Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing, pp. 304–307. Springer, Berlin (1999)

12.

Duarte, C., Carriço, L.: A conceptual framework for developing adaptive multimodal applications. In: IUI ’06: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 132–139. ACM, New York (2006). doi:10.1145/1111449.1111481

13.

Dumas, B., Lalanne, D., Ingold, R.: Prototyping multimodal interfaces with the SMUIML modeling language. In: CHI 2008 Workshop on User Interface Description Languages for Next Generation User Interfaces, CHI 2008, Frienze, pp. 63–66 (2008)

14.

Dumas, B., Lalanne, D., Guinard, D., Koenig, R., Ingold, R.: Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces. In: TEI ’08: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, pp. 47–54. ACM, New York (2008). doi:10.1145/1347390.1347403

15.

Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction – Research Results of the MMI Program. Lecture Notes in Computer Science, vol. 5440/2009, chap. 1, pp. 3–26. Springer, Berlin, Heidelberg (2009). doi:10.1007/978-3-642-00437-7_1

16.

Dumas, B., Signer, B., Lalanne, D.: Fusion in multimodal interactive systems: an HMM-based algorithm for user-induced adaptation. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’12, pp. 15–24. ACM, New York (2012). doi:10.1145/2305484.2305490

17.

Foster, M.E.: State of the art review: multimodal fission. Public Deliverable 6.1, University of Edinburgh (2002). COMIC Project

18.

Geier, T., Reuter, S., Dietmayer, K., Biundo, S.: Goal-based person tracking using a first-order probabilistic model. In: Proceedings of the Ninth UAI Bayesian Modeling Applications Workshop (UAI-AW 2012) (2012)

19.

Hina, M.D., Tadj, C., Ramdane-Cherif, A., Levy, N.: A Multi-Agent based Multimodal System Adaptive to the User’s Interaction Context. Multi-Agent Systems – Modeling, Interactions, Simulations and Case Studies, chap. 2, pp. 29–56. InTech (2011)

20.

Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In: Proceedings of the 6th International Conference on Multimodal Interfaces, ICMI ’04, pp. 175–182. ACM, New York (2004). doi:10.1145/1027933.1027964

21.

Honold, F., Schüssel, F., Weber, M.: Adaptive probabilistic fission for multimodal systems. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI ’12, pp. 222–231. ACM, New York (2012). doi:10.1145/2414536.2414575

22.

Honold, F., Schüssel, F., Weber, M., Nothdurft, F., Bertrand, G., Minker, W.: Context models for adaptive dialogs and multimodal interaction. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 57–64. IEEE, New York (2013). doi:10.1109/IE.2013.54

23.

Honold, F., Schüssel, F., Munding, M., Weber, M.: Tangible context modelling for rapid adaptive system testing. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 278–281. IEEE, Athens (2013). doi:10.1109/IE.2013.9

24.

Honold, F., Bercher, P., Richter, F., Nothdurft, F., Geier, T., Barth, R., Hörnle, T., Schüssel, F., Reuter, S., Rau, M., Bertrand, G., Seegebarth, B., Kurzok, P., Schattenberg, B., Minker, W., Weber, M., Biundo, S.: Companion-technology: towards user- and situation-adaptive functionality of technical systems. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 378–381. IEEE, New York (2014). doi:10.1109/IE.2014.60

25.

Honold, F., Schüssel, F., Weber, M.: The automated interplay of multimodal fission and fusion in adaptive HCI. In: IE’14: Proceedings of the 10th International Conference on Intelligent Environments, pp. 170–177. IEEE, Shanghai (2014). doi:10.1109/IE.2014.32

26.

Johnston, M., Bangalore, S.: Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11, 159–187 (2005). doi:10.1017/S1351324904003572CrossRef

27.

Koons, D.B., Sparrell, C.J., Thorisson, K.R.: Integrating simultaneous input from speech, gaze, and hand gestures. In: Maybury, M.T. (ed.) Intelligent Multimedia Interfaces, chap. 11, pp. 257–276. American Association for AI, Menlo Park, CA (1993)

28.

Krahnstoever, N., Kettebekov, S., Yeasin, M., Sharma, R.: A real-time framework for natural multimodal interaction with large screen displays. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, pp. 349–354. IEEE Computer Society, Washington, DC (2002). doi:10.1109/ICMI.2002.1167020

29.

Lalanne, D., Nigay, L., Palanque, P., Robinson, P., Vanderdonckt, J., Ladry, J.F.: Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, ICMI-MLMI ’09, pp. 153–160. ACM, New York (2009). doi:10.1145/1647314.1647343

30.

LaViola, J.J. Jr., Buchanan, S., Pittman, C.: Multimodal Input for Perceptual User Interfaces, chap. 9, pp. 285–312. Wiley, New York (2014). doi:10.1002/9781118706237.ch9

31.

Mansoux, B., Nigay, L., Troccaz, J.: Output multimodal interaction: the case of augmented surgery. In: Bryan-Kinns, N., Blanford, A., Curzon, P., Nigay, L. (eds.) People and Computers XX – Engage. BCS Conference Series, vol. 5, pp. 177–192. Springer/ACM, London/New York (2006). doi:10.1007/978-1-84628-664-3_14

32.

Martin, J.C.: Tycoon: theoretical framework and software tools for multimodal interfaces. In: Lee, J. (ed.) Intelligence and Multimodality in Multimedia Interfaces. AAAI Press, Palo Alto, CA (1998)

33.

Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., Shapiro, S.C.: Natural language with integrated deictic and graphic gestures. In: Proceedings of the Workshop on Speech and Natural Language, HLT ’89, pp. 410–423. Association for Computational Linguistics, Stroudsburg, PA (1989). doi:10.3115/1075434.1075499

34.

Nigay, L., Coutaz, J.: A generic platform for addressing the multimodal challenge. In: CHI ’95: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 98–105. ACM, New York (1995). doi:10.1145/223904.223917

35.

Norman, D.A.: The Design of Everyday Things. iBasic Books, New York (2002)

36.

Nothdurft, F., Honold, F., Zablotskaya, K., Diab, A., Minker, W.: Application of verbal intelligence in dialog systems for multimodal interaction. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 361–364. IEEE, Shanghai (2014). doi:10.1109/IE.2014.59. (Short paper)

37.

Oviatt, S.: Multimodal Interfaces, 2nd edn., chap. 21, pp. 413–432. CRC, Boca Raton (2007). doi:10.1201/9781410615862.ch21

38.

Pfleger, N.: Context based multimodal fusion. In: ICMI ’04: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 265–272. ACM, New York (2004). doi:10.1145/1027933.1027977

39.

Portillo, P.M., García, G.P., Carredano, G.A.: Multimodal fusion: a new hybrid strategy for dialogue systems. In: Proceedings of the 8th International Conference on Multimodal Interfaces, ICMI ’06, pp. 357–363. ACM, New York (2006). doi:10.1145/1180995.1181061

40.

Reddy, B.S., Basir, O.A.: Concept-based evidential reasoning for multimodal fusion in human-computer interaction. Appl. Soft Comput. 10(2), 567–577 (2010). doi:10.1016/j.asoc.2009.08.026CrossRef

41.

Roscher, D., Blumendorf, M., Albayrak, S.: A meta user interface to control multimodal interaction in smart environments. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pp. 481–482. ACM, New York (2009). doi:10.1145/1502650.1502725

42.

Rousseau, C., Bellik, Y., Vernier, F., Bazalgette, D.: A framework for the intelligent multimodal presentation of information. Signal Process. 86(12), 3696–3713 (2006). doi:10.1016/j.sigpro.2006.02.041CrossRefMATH

43.

Schüssel, F., Honold, F., Weber, M.: Influencing factors on multimodal interaction during selection tasks. J. Multimodal User Interfaces 7(4), 299–310 (2013). doi:10.1007/s12193-012-0117-5CrossRef

44.

Schüssel, F., Honold, F., Weber, M.: Using the transferable belief model for multimodal input fusion in companion systems. In: Schwenker, F., Scherer, S., Morency, L.P. (eds.) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. Lecture Notes in Computer Science, vol. 7742, pp. 100–115. Springer, Berlin, Heidelberg (2013). doi:10.1007/978-3-642-37081-6_12CrossRef

45.

Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990). doi:10.1109/34.55104CrossRef

46.

Smets, P.: Data fusion in the transferable belief model. In: Proceedings of the Third International Conference on Information Fusion. FUSION 2000, vol. 1, pp. PS21–PS33. IEEE, New York (2000). doi:10.1109/IFIC.2000.862713

47.

Vanderdonckt, J., Limbourg, Q., Michotte, B., Bouillon, L., Trevisan, D., Florins, M.: USIXML: a user interface description language for specifying multimodal user interfaces. In: Proceedings of W3C Workshop on Multimodal Interaction WMI’2004, pp. 1–7 (2004)

48.

Wahlster, W.: User and discourse models for multimodal communication. In: Sullivan, J.W., Tyler, S.W. (eds.) Intelligent User Interfaces, pp. 45–67. ACM, New York (1991). doi:10.1145/107215.128691

49.

Wahlster, W. (ed.): SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin (2006). doi:10.1007/3-540-36678-4

Title: Management of Multimodal User Interaction in Companion-Systems
Authors: Felix Schüssel
Frank Honold
Nikola Bubalo
Michael Weber
Anke Huckauf
Publisher: Springer International Publishing
Book: Companion Technology
Print ISBN: 978-3-319-43664-7

Electronic ISBN: 978-3-319-43665-4

Copyright Year: 2017
DOI: https://doi.org/10.1007/978-3-319-43665-4_10

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner