Christine Kühnel
Tilo Westermann
ABSTRACT
This paper describes the analysis of established and new questionnaires concerning their applicability for the assessment of quality aspects of multimodal systems. To this purpose, an experiment with 27 participants interacting with a a smart-home system via a voice interface, a smartphone-based interface and a multimodal interface, was conducted. Interaction parameters were assessed and related to constructs measured with these questionnaires. The results indicate that some of the questionnaires are suitable for evaluating multimodal interfaces. On the basis of correlations with interaction parameters subscales of these questionnaires can be mapped to quality aspects, such as effectiveness and efficiency. Recommendations are given how to meet two important evaluation requirements, namely which questionnaire to use for comparing two or more systems or system versions and how to identify factors or components in a system that have to be improved. This is another step forward to establish evaluation methods for multimodal systems.
- N. Beringer, U. Kartal, K. Louka, F. Schiel, and U. Türk. PROMISE: -- A procedure for multimodal interactive system evaluation. In Proc. of the LREC Workshop on Multimodal Resources and Multimodal Systems Evaluation, Las Palmas, pages 77--80, 2002.Google Scholar
- N. Bevan. What is the difference between the purpose of usability and user experience evaluation methods? In Proc. of the UXEM Workshop at INTERACT, 2009.Google Scholar
- J. Brooke. SUS: A quick and dirty usability scale. In P. W. Jordan, B. Weerdmeester, A. Thomas, and I. L. McClelland, editors, Usability evaluation in industry, pages 189--194. Taylor and Francis, London, 1996.Google Scholar
- J. P. Chin, V. A. Diehl, and K. Norman. Development of an instrument measuring user satisfaction of the human-computer interface. In Proc. of ACM CHI, pages 213--218, 1988. Google ScholarDigital Library
- J. Coutaz, L. Nigay, D. Salber, A. Blandford, J. May, and R. Young. Four easy pieces for assessing the usability of multimodal interaction: The CARE properties. In Human-Computer Interaction, Interact, pages 115--120. Chapman & Hall, London, 1995.Google Scholar
- B. Dumas, D. Lalanne, and S. Oviatt. Multimodal interfaces: A survey of principles, models and frameworks. pages 3--26, 2009. Google ScholarDigital Library
- E. Frøkjær, M. Hertzum, and K. Hornbæk. Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In Proc. of CHI, pages 345--352. ACM, 2000. Google ScholarDigital Library
- M. Hassenzahl, M. Burmester, and F. Koller. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität {a questionnaire for measuring perceived hedonic and pracmatic quality}. Mensch & Computer 2003. Interaktion in Bewegung, pages 187--196, 2003.Google Scholar
- M. Hassenzahl, R. Kekez, and M. Burmester. The importance of a software's pragmatic quality depends on usage modes. In H. Luczak, A. E. Cakir, and G. Cakir, editors, Proc. of the 6th international conference on Work With Display Units WWDU 2002, pages 275--276, Berlin, 2002. ERGONOMIC Institut für Arbeits- und Sozialforschung.Google Scholar
- M. Hassenzahl and A. Monk. The inference of perceived usability from beauty. Human-Computer Interaction (accepted), 2010.Google Scholar
- K. S. Hone and R. Graham. Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI). Nat. Lang. Eng., 6(3--4):287--303, 2000. Google ScholarDigital Library
- K. Hornbæk and E. L.-C. Law. Meta-analysis of correlations among usability measures. In Proc. of CHI, pages 617--626. ACM, 2007. Google ScholarDigital Library
- J. Kirakowski and M. Corbett. SUMI: The Software Usability Measurement Inventory. British Journal of Educational Technology, (3):210--212, 1993.Google Scholar
- C. Kühnel, B. Weiss, and S. Möller. Parameters describing multimodal interaction --- definitions and three usage scenarios. In accepted for Interspeech 2010, 2010.Google Scholar
- E. L.-C. Law, V. Roto, M. Hassenzahl, A. P. Vermeeren, and J. Kort. Understanding, scoping and defining user experience: a survey approach. In Proc. of CHI, pages 719--728, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- J. Lewis. Ibm computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57--78, 1995. Google ScholarDigital Library
- R. López-Cózar Delgado and M. Araki. Spoken, multilingual and multimodal dialogue systems: development and assessment. John Wiley & Sons, Chichester, 2005.Google Scholar
- A. M. Lund. Measuring usability with the USE questionnaire. Usability Interface, 8(2), 2001.Google Scholar
- S. Möller, K.-P. Engelbrecht, C. Kühnel, I. Wechsung, and B. Weiss. Evaluation of multimodal interfaces for ambient intelligence. In Human-Centric Interfaces for Ambient Intelligence, pages 347--370. Elsevier, 2010.Google ScholarCross Ref
- S. Möller, K.-P. Engelbrecht, and R. Schleicher. Predicting the quality and usability of spoken dialogue services. Speech Commun., 50(8--9):730--744, 2008. Google ScholarDigital Library
- S. Möller, J. Krebber, A. Raake, P. Smeele, M. Rajman, M. Melichar, V. Pallotta, G. Tsakou, B. Kladis, A. Vovos, J. Hoonhout, D. Schuchardt, N. Fakotakis, T. Ganchev, and I. Potamitis. INSPIRE: Evaluation of a Smart-Home System for Infotainment Management and Device Control. In Proc. 4th Int. Conf. on Language Resources and Evaluation (LREC), volume 5, pages 1603--1606, May 2004.Google Scholar
- A. Naumann and I. Wechsung. Developing usability methods for multimodal systems: The use of subjective and objective measures. In E. L.-C. Law, N. Bevan, G. Christou, M. Springett, and M. Larusdottir, editors, Proc. of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), pages 8--12. Institute of Research in Informatics of Toulouse (IRIT), 2008.Google Scholar
- A. Naumann, I. Wechsung, and J. Hurtienne. Multimodality, inclusive design, and intuitive use. In Proc. of the 23rd British Computer Society Human Computer Interaction Workshop and Conference (HCI 2009), 2009.Google Scholar
- J. Nielsen and J. Levy. Measuring usability: Preference vs. performance. Communications of the ACM, 37(4):66--75, 1994. Google ScholarDigital Library
- M. Nielsen, T. Moeslund, M. Störring, and E. Granum. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Proc. of Gesture-Based Communication in Human-Computer Interaction, 5th International Gesture Workshop, pages 409--420. Springer, 2003.Google Scholar
- L. Nigay and J. Coutaz. A design space for multimodal systems: concurrent processing and data fusion. In Proc. of the INTERACT and CHI, pages 172--178, 1993. Google ScholarDigital Library
- S. Oviatt. Multimodal interfaces. pages 286--304, 2003. Google ScholarDigital Library
- T. Paek. Empirical methods for evaluating dialog systems. In Proc. of the Second SIGdial Workshop on Discourse and Dialogue, pages 1--9, Morristown, NJ, USA, 2001. Association for Computational Linguistics. Google ScholarDigital Library
- L. M. Reeves, J. Lai, J. A. Larson, S. Oviatt, T. S. Balaji, S. Buisine, P. Collings, P. Cohen, B. Kraal, J.-C. Martin, M. McTear, T. Raman, K. M. Stanney, H. Su, and Q. Y. Wang. Guidelines for multimodal user interface design. Communications of ACM, 47:57--59, 2004. Google ScholarDigital Library
- J. Sauro and E. Kindlund. A method to standardize usability metrics into a single score. In Proc. of CHI, pages 401--409. ACM Press, 2005. Google ScholarDigital Library
- N. Tractinsky. Aesthetics and apparent usability: empirically assessing cultural and methodological issues. In CHI '97: Proc. of the SIGCHI conference on Human factors in computing systems, pages 115--122, New York, NY, USA, 1997. ACM. Google ScholarDigital Library
- T. Tullis and J. Stetson. A comparison of questionnaires for assessing website usability. In Proc. of Usability Professionals Association (UPA).Google Scholar
- M. Turunen, J. Hakulinen, A. Melto, T. Heimonen, T. Laivo, and J. Hella. Suxes -- user experience evaluation method for spoken and multimodal interaction. In Proc. of Interspeech, pages 2567--2570, 2009.Google Scholar
- M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella. PARADISE: a framework for evaluating spoken dialogue agents. In Proc. of the ACL/EACL 35th Meeting of the Assoc. for Computational Linguistics, Madrid, pages 271--280, 1997. Google ScholarDigital Library
- I. Wechsung and A. Naumann. Established usability evaluation methods for multimodal systems: A comparison of standardized usability questionnaires. In Proc. of PIT. Heidelberg: Springer, 2008. Google ScholarDigital Library
- J. O. Wobbrock, M. R. Morris, and A. D. Wilson. User-Defined Gestures for Surface Computing. In Proc. of CHI, pages 1083--1092. ACM Press, 2009. Google ScholarDigital Library
Index Terms
- Evaluating multimodal systems: a comparison of established questionnaires and interaction parameters
Recommendations
Multimodal human discourse: gesture and speech
Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the ...
Prediction of Various Backchannel Utterances Based on Multimodal Information
IVA '23: Proceedings of the 23rd ACM International Conference on Intelligent Virtual AgentsThe listener's backchannels are an important part of dialogues. With appropriate backchannels, people are able to smoothly promote dialogues. Thus, backchannels are considered to be important in dialogues between not only humans but also humans and ...
Multimodal interaction systems: information and time features
Multimodal interaction systems combine visual information (involving images, text, sketches and so on) with voice, gestures and other modalities to provide flexible and powerful dialogue approaches, enabling users to choose one or more of the multiple ...
Comments