Skip to main content
SpringerLink
Log in

UXAmI Observer: An Automated User Experience Evaluation Tool for Ambient Intelligence Environments

  • Conference paper
  • First Online:
Intelligent Systems and Applications (IntelliSys 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 868))

Included in the following conference series:

Abstract

Ambient Intelligence constitutes a new human-centered technological paradigm, where environments are oriented towards anticipating and satisfying the needs of their inhabitants. In this context, evaluation becomes of paramount importance. This paper presents UXAmI Observer, an automated user experience evaluation tool for Ambient Intelligence environments, taking advantage of their inherent infrastructure to automatically acquire measurements during user testing experiments. The tool provides powerful data visualizations for the entire experiment, for each system and application evaluated, as well as for each experiment participant individually, ensuring synchronization of data with video recordings, and facilitating manual data input by the evaluators themselves.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Stephanidis, C.: Human factors in ambient intelligence environments. In: Salvendy, G. (ed.) Handbook of Human Factors and Ergonomics, 4th edn., pp. 1354–1373. John Wiley and Sons (2012)

    Google Scholar 

  2. Augusto, J.C., Nakashima, H., Aghajan, H.: Ambient intelligence and smart environments: a state of the art. In: Nakashima, H., Aghajan, H., Augusto, J.C. (eds.) Handbook of Ambient Intelligence and Smart Environments, pp. 3–31. Springer, US (2010)

    Chapter  Google Scholar 

  3. MacDonald, C.M., Atwood, M.E.: Changing perspectives on evaluation in HCI: past, present, and future. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 1969–1978. ACM, New York, NY, USA (2013)

    Google Scholar 

  4. Martins, A.I., Queirós, A., Silva, A.G., Rocha, N.P.: Usability evaluation methods: a systematic review. In: Saeed, S., Bajwa, I.S., Mahmood, Z. (eds.) Human Factors in Software Development and Design, pp. 250–273. IGI Global (2015)

    Google Scholar 

  5. Paz, F., Pow-Sang, J.A.: Current trends in usability evaluation methods: a systematic review. In: 2014 7th International Conference on Advanced Software Engineering and Its Applications (ASEA), pp. 11–15. IEEE (2014)

    Google Scholar 

  6. Nielsen, J.: Usability engineering. Elsevier (1994)

    Google Scholar 

  7. Hornbæk, K.: Current practice in measuring usability: challenges to usability studies and research. Int. J. Hum. Comput. Stud. 64(2), 79–102 (2006)

    Article  Google Scholar 

  8. Hertzum, M., Jacobsen, N.E.: The evaluator effect: a chilling fact about usability evaluation methods. Int. J. Hum. Comput. Interact. 13(4), 421–443 (2001)

    Article  Google Scholar 

  9. Hornbæk, K., Law, E.L.C.: Meta-analysis of correlations among usability measures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘07), pp. 617–626. ACM (2007)

    Google Scholar 

  10. Bevan, N.: What is the difference between the purpose of usability and user experience evaluation methods. In: Proceedings of the Workshop UXEM, vol. 9, pp. 1–4 (2009)

    Google Scholar 

  11. Bargas-Avila, J.A., Hornbæk, K.: Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘11), pp. 2689–2698. ACM (2011)

    Google Scholar 

  12. Maia, C.L.B., Furtado, E.S.: A systematic review about user experience evaluation. In: Marcus, A. (ed.) Design, User Experience, and Usability: Design Thinking and Methods, DUXU 2016. Lecture Notes in Computer Science, vol. 9746, pp. 445–455. Springer, Cham (2016)

    Chapter  Google Scholar 

  13. Gaggioli, A.: Optimal experience in ambient intelligence. In: Riva, G., Vatalaro, F., Davide, F., Alcañiz, M. (eds.) Ambient Intelligence, pp. 35–43. IOS Press (2005)

    Google Scholar 

  14. Poppe, R., Rienks, R., van Dijk, B.: Evaluating the future of HCI: challenges for the evaluation of emerging applications. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds.) Artificial Intelligence for Human Computing. Lecture Notes in Artifical Intelligence, vol. 4451, pp. 234–250. Springer (2007)

    Google Scholar 

  15. Ivory, M.Y., Hearst, M.A.: The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. (CSUR) 33(4), 470–516 (2001)

    Article  Google Scholar 

  16. Hilbert, D.M., Redmiles, D.F.: Extracting usability information from user interface events. ACM Comput. Surv. (CSUR) 32(4), 384–421 (2000)

    Article  Google Scholar 

  17. Macleod, M., Rengger, R.: The development of DRUM: a software tool for video-assisted usability evaluation. In: Alty, J.L., Diaper, D., Guest, S. (eds.) People and Computers, vol. VIII, p. 293. Cambridge University Press (1997)

    Google Scholar 

  18. Sears, A.: Heuristic walkthroughs: finding the problems without the noise. Int. J. Hum. Comput. Inter. 9(3), 213–234 (1997)

    Article  Google Scholar 

  19. Lecerof, A., Paternò, F.: Automatic support for usability evaluation. IEEE Trans. Softw. Eng. 24(10), 863–888 (1998)

    Article  Google Scholar 

  20. Kim, J.H. et al.: Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 443–452. ACM (2008)

    Google Scholar 

  21. Heilbrunn, B., Herzig, P., Schill, A.: Tools for gamification analytics: a survey. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC), pp. 603–608. IEEE (2014)

    Google Scholar 

  22. Bateman, S., Gutwin, C., Osgood, N., McCalla, G.: Interactive usability instrumentation. In: Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 45–54. ACM (2009)

    Google Scholar 

  23. Chang, E., Dillon, T.S.: Automated usability testing. In: Howard, S., Hammond, J., Lindgaard, G. (eds.) Human-Computer Interaction INTERACT’97, pp. 77–84. Springer (1997)

    Google Scholar 

  24. Alexander, J., Cockburn, A., Lobb, R.: AppMonitor: a tool for recording user actions in unmodified Windows applications. Behav. Res. Methods 40(2), 413–421 (2008)

    Article  Google Scholar 

  25. Ma, X., et al.: Design and implementation of a toolkit for usability testing of mobile apps. Mob. Networks Appl. 18(1), 81–97 (2013)

    Article  Google Scholar 

  26. Tran, C.D., Ezzedine, H., Kolski, C.: EISEval, a generic reconfigurable environment for evaluating agent-based interactive systems. Int. J. Hum Comput Stud. 71(6), 725–761 (2013)

    Article  Google Scholar 

  27. Assila, A., de Oliveira, K.M., Ezzedine, H.: An environment for integrating subjective and objective usability findings based on measures. In: 2016 IEEE Tenth International Conference on Research Challenges in Information Science (RCIS), pp. 1–12. IEEE (2016)

    Google Scholar 

  28. Muhi, K., Szőke, G., Fülöp, L.J., Ferenc, R., Berger, Á.: A Semi-automatic usability evaluation framework. In: Murgante, B. et al. (eds.) Computational Science and Its Applications–ICCSA 2013. Lecture Notes in Computer Science, vol. 7972, pp. 529–542. Springer, Heidelberg (2013)

    Google Scholar 

  29. Christensen, L., Frøkjær, E.: Distributed usability evaluation: enabling large-scale usability evaluation with user-controlled instrumentation. In: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI 2010), ACM, pp. 118–127 (2010)

    Google Scholar 

  30. Bao, L., et al.: Extracting and analyzing time-series HCI data from screen-captured task videos. Empirical Softw. Eng. 22(1), 134–174 (2017)

    Article  Google Scholar 

  31. Howarth, J., Smith-Jackson, T., Hartson, R.: Supporting novice usability practitioners with usability engineering tools. Int. J. Hum. Comput. Stud. 67(6), 533–549 (2009)

    Article  Google Scholar 

  32. Morae: Morae 3 Tutorial (2017). Retrieved from: https://www.techsmith.com/tutorial-morae-current.html

  33. Norman, K.L., Panizzi, E.: Levels of automation and user participation in usability testing. Interact. Comput. 18(2), 246–264 (2005)

    Article  Google Scholar 

  34. Alspaugh, S., Ganapathi, A., Hearst, M.A., Katz, R.: Better logging to improve interactive data analysis tools. In: Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics (IDEA 2014), pp. 19–25 (2014)

    Google Scholar 

  35. Au, F.T., Baker, S., Warren, I., Dobbie, G.: Automated usability testing framework. In: Proceedings of the Ninth Conference on Australasian User Interface (AUIC 2008), vol. 76, pp. 55–64. Australian Computer Society, Inc. (2008)

    Google Scholar 

  36. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: Technologies, applications, and opportunities. Pervasive Mob. Comput. 5(4), 277–298 (2009)

    Article  Google Scholar 

  37. Augusto, J.C., McCullagh, P.: Ambient intelligence: Concepts and applications. Comput. Sci. Inf. Syst. 4(1), 1–27 (2007)

    Article  Google Scholar 

  38. Nielsen, J.: Success Rate: The Simplest Usability Metric (2001). Retrieved from: https://www.nngroup.com/articles/success-rate-the-simplest-usability-metric/

  39. Nielsen, J.: Heuristic evaluation. Usability Inspection Methods 17(1), 25–62 (1994)

    Google Scholar 

Download references

Acknowledgment

This work is supported by the FORTH-ICS internal RTD Programme “Ambient Intelligence Environments”.

Author information

Authors and Affiliations

  1. Institute of Computer Science (ICS), Foundation for Research and Technology Hellas (FORTH), Heraklion, Greece

    Stavroula Ntoa, George Margetis, Margherita Antona & Constantine Stephanidis

  2. Computer Science Department, University of Crete, Heraklion, Greece

    Constantine Stephanidis

Authors
  1. Stavroula Ntoa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Margetis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Margherita Antona
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Constantine Stephanidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stavroula Ntoa .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

  2. The Science and Information (SAI) Organization, Bradford, UK

    Supriya Kapoor

  3. The Science and Information (SAI) Organization, Bradford, UK

    Rahul Bhatia

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ntoa, S., Margetis, G., Antona, M., Stephanidis, C. (2019). UXAmI Observer: An Automated User Experience Evaluation Tool for Ambient Intelligence Environments. In: Arai, K., Kapoor, S., Bhatia, R. (eds) Intelligent Systems and Applications. IntelliSys 2018. Advances in Intelligent Systems and Computing, vol 868. Springer, Cham. https://doi.org/10.1007/978-3-030-01054-6_94

Download citation

Publish with us

Policies and ethics

Search

Navigation