Skip to main content
SpringerLink
Log in

A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Controlled psychoperceptual quality evaluation experiments are used to assess the excellence of produced audiovisual quality from fundamental signal processing algorithms to consumer services. When compromising produced quality for consumer services, used in dynamic and heterogeneous mobile contexts, the ecological validity of conventional quality evaluation methods can be questioned. The goal of this paper is to develop a method for evaluating the experienced multimedia quality in the context of use. We conducted three studies where the quality of mobile 2D and 3D television was assessed in three different field contexts, one simulated context and one controlled laboratory situation when audio-video compression and transmission parameters were varied. We propose a hybrid method for the design, data-collection and analysis of the experiments in the contexts of use. Its novelty is to complement conventional quantitative quality evaluation with concrete tools to identify factors that surround the assessment in the context. The methodological framework is part of our long-term aim to measure and understand the user-centered quality of experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Barnard L et al (2007) Capturing the effects of context on human performance in mobile computing systems. Pers Ubiquit Comput 11:81–96

    Article  Google Scholar 

  2. Beresford K, et al. (2006) Contextual effects on sound quality judgements: listening room and automotive environments. AES 120th Convention:6648

  3. Bernhaupt R, Mihalic K, Obrist M (2008) Usability evaluation methods for mobile applications. In: Lumsden J (ed) Handbook Res User Interface Des Evaluation for Mobile Technology XLIV:745–758

  4. Boev A et al (2009) Classification and simulation of stereoscopic artifacts in mobile 3DTV content. Proc SPIE Stereoscopic Displays and Applications XX:7237(72371F)

  5. Bradley NA, Dunlop MD (2005) Toward a multidisciplinary model of context to support context-aware computing. Hum Comput Interact 20(4):403–446

    Article  Google Scholar 

  6. Brewster S (2002) Overcoming the lack of screen space on mobile computers. Pers Ubiq Comput 6:188–205

    Article  Google Scholar 

  7. Buchinger S, Kriglstein S, Hlavacs H (2009) A comprehensive view on user studies: survey and open issues for mobile TV. Proc EuroITV '09:179-188

    Google Scholar 

  8. Chen T, Yesilada Y, Harper S (2008) RIAM D2.6: How do people use their mobile phones while they are walking? A field study of real-world small device usage. Research Report School of Computer Science, Univ of Manchester. http://hcw-eprints.cs.man.ac.uk/98/1/RIAM_D2_6_Field_Study.pdf

  9. Consolvo S et al (2007) Conducting in situ evaluations for and with ubiquitous computing technologies. Int J Hum Comput Interact 22(1):107–122

    Google Scholar 

  10. Cook T, Campbell D (1979) Quasi-experimentation: design & analysis issues for field settings. Houghton Mifflin, New York

    Google Scholar 

  11. Coolican H (2004) Research methods and statistics in psychology, 4th edn. Arrowsmith, London

    Google Scholar 

  12. Cui Y, Chipchase J, Jung Y (2006) Personal television: a qualitative study of mobile TV users. Lect Notes Comput Sci 4471:195–204

    Article  Google Scholar 

  13. DVB-H.org. http://www.dvb-h.org/. Accessed 15 December 2009

  14. European Broadcasting Union (2003) Subjective listening tests on low-bitrate audio codecs. Tech 3296

  15. European Telecommunications Standards Institute (2005) Digital Video Broadcasting (DVB): DVB-H implementation guidelines. ETSI TR 102 377 V1.2.1 (2005-11)

  16. Fiske ST, Taylor SE (1991) Social cognition. McGrow-Hill, Singapore

    Google Scholar 

  17. Flack J, Harrold J, Woodgate GJ (2007) A prototype 3D mobile phone equipped with a next generation autostereoscopic display. Proc SPIE 6490(64900 M)

  18. Fredrickson BL (2000) Extracting meaning from past affective experiences: the importance of peaks, ends and specific emotions. Cogn Emotion 14(4):577–606

    Article  Google Scholar 

  19. Ghinea G, Chen SY (2008) Measuring quality of perception in distributed multimedia: verbalizers vs. imagers. Comput Hum Behav 24(4):1317–1329

    Article  Google Scholar 

  20. Gibson JJ (1979) The ecological approach to visual perception. Houghton Mifflin, Lawrence Eribaum, Boston

    Google Scholar 

  21. Gochev A, et al (2009) Mobile 3D television: development of core technological elements and user-centered evaluation methods toward an optimized system. In: Creutzburg R, Akopian D (ed) Multimedia on Mobile Devices 2009:7256(1)

  22. Goldsmith RE (2001) Using the domain specific innovativeness scale to identify innovative internet consumers. Internet Res Electron Netw Appl Policy 11(2):149–158

    Article  Google Scholar 

  23. Goodman J, Brewster S, Gray P (2004) Using field experiments to evaluate mobile guides. In: Schmidt-Belz B, Cheverst K (ed) Proc HCI 2004

  24. Grill-Spector K, Malach R (2004) The human visual cortex. Ann Rev Neurosci 27:649–677

    Article  Google Scholar 

  25. Gulliver SR, Ghinea G (2006) Defining user perception of distributed multimedia quality. ACM Trans Multimed Comput Commun Appl 2(4):241–257

    Article  Google Scholar 

  26. Hands DS et al (2007) Price-dependent quality: examining the effects of price on multimedia quality requirements. Proc SPIE 6492(62920 N)

  27. Harrison GW, List JA (2004) Field experiments. J Econ Lit 42(4):1009–1055

    Article  Google Scholar 

  28. Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload. North-Holland, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  29. ISO 13407 (1999) Human-centered design processes for interactive systems. Int Org Stand

  30. ITU-R BT.500-11 (2002) Methodology for the subjective assessment of the quality of television pictures. Int Telecommun Union—Radiocommun sector

  31. ITU-T P.911 Recommendation P.911 (1998) Subjective audiovisual quality assessment methods for multimedia application. Int Telecommun Union—Telecommun sector

  32. ITU-T Recommendation P.10 Amendment 1 (2008) Vocabulary for performance and quality of service, New Appendix I Definition of Quality of Experience (QoE). Int Telecommun Union

  33. ITU-T. Recommendation E.800 (1994) Terms and definitions related to quality of service and network performance including dependability. Int Telecommun Union

  34. Jambon F (2009) User evaluation of mobile devices: in-situ versus laboratory experiments. Int J Mobile Comput Hum Interact 1(2):56–71

    Article  Google Scholar 

  35. Jumisko-Pyykkö S (2008) I would like to see the subtitles and the face or at least hear the voice: effects of picture ratio and audio-video bitrate ratio on perception of quality in mobile television. Multimed Tools Appl 36(1–2):167–184

    Article  Google Scholar 

  36. Jumisko-Pyykkö S, Hannuksela MM (2008) Does context matter in quality evaluation of mobile television? Proc MobileHCI'08:63–72

    Google Scholar 

  37. Jumisko-Pyykkö S, Strohmeier D (2008) Report on research methodologies for the experiments. MOBILE3DTV Technical report. http://sp.cs.tut.fi/mobile3dtv/results/tech/D4.2_Mobile3dtv_v2.0.pdf

  38. Jumisko-Pyykkö S, Utriainen T (2010) Experienced audiovisual quality for mobile 3D television. Proc 3DTV Conference 2010

  39. Jumisko-Pyykkö S, Vainio T (2010) Framing the context of use for mobile HCI. Int J Mobile-Human-Computer-Interaction IJMHCI 2(4)

  40. Jumisko-Pyykkö S, Häkkinen J, Nyman G (2007) Experienced quality factors—qualitative evaluation approach to audiovisual quality. Proc SPIE Multimedia on Mobile Devices 2007:6507(65070 M)

  41. Jumisko-Pyykkö S, Malamal Vadakital VK, Hannuksela MM (2008) Acceptance threshold: bidimensional research method for user-oriented quality evaluation studies. Int J Digit Multimed Broadcast 2008:712380

  42. Jumisko-Pyykkö S, Weitzel M, Strohmeier D (2008) Designing for user experience: what to expect from mobile 3D TV and video? Proc 1st Int Conf Designing Interact User Experiences TV and video 2008:183-192

  43. Kaikkonen A et al (2008) Will laboratory test results be valid in mobile contexts? In: Lumsden J (ed) Handbook Res User Interface Design Evaluation for Mobile Tech. Information Science Reference, pp 897–909

  44. Kennedy RS et al (1993) Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviation Psychol 3(3):203–220

    Article  Google Scholar 

  45. Kim H, et al. (2002) An empirical study of the use contexts and usability problems in mobile internet. Proc 35th Ann Hawaii Int Conf System Sciences HICSS'02 5(5):132

  46. Kjeldskov J et al (2004) Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. Proc MobileHCI 2004:61–73

    Google Scholar 

  47. Knoche H, Sasse MA (2009) The big picture on small screens delivering acceptable video quality in mobile TV. ACM Trans Multimed Comput Commun Appl TOMCCAP 5(3):1–27

    Article  Google Scholar 

  48. Knoche H, McCarthy J, Sasse MA (2006) Reading the fine print: the effect of text legibility on perceived video quality in mobile TV. Proc ACM Multimed 2006:727–730

    Google Scholar 

  49. Knoche H, McCarthy J, Sasse MA (2008) How low can you go? The effect of low resolutions on shot types. Multimed Tools Appl 36(1–2):145–166

    Article  Google Scholar 

  50. Lewicki MS (2002) Efficient coding of natural sounds. Nat Neurosci 5(4):292–294

    Article  Google Scholar 

  51. McCarthy JD, Sasse MA, Miras D (2004) Sharp or smooth?: Comparing the effect of quantization vs. framerate for streamed video. Proc CHI 2004:535-542

  52. Mizobuchi S, Chignell M, Newton D (2005) Mobile text entry: relationship between walking speed and text input task difficulty. Proc MobileHCI'05 111:122-128

    Google Scholar 

  53. Mustonen T, Olkkonen M, Häkkinen J (2004) Examining mobile phone text legibility while walking. Ext Abstr Human Factors in Computing Systems CHI '04:1243-1246

  54. Nahrstedt K, Steinmetz R (1995) Resource management in networked multimedia systems. IEEE Comput 28(5):52–63

    Google Scholar 

  55. Neisser U (1976) Cognition and reality, principles and implications of cognitive psychology. Freeman, San Francisco

    Google Scholar 

  56. Oatley K, Jenkins JM (2003) Understanding emotions. Blackwell, Oxford

    Google Scholar 

  57. O'Hara K, Mitchell AS, Vorbau A (2007) Consuming video on mobile devices. Proc CHI '07:857-866

    Google Scholar 

  58. Oksman V, et al (2007) News in mobiles, Comparing text, audio and video. VTT 2007. http://www.vtt.fi/inf/pdf/tiedotteet/2007/T2375.pdf

  59. Oksman V et al (2008) ‘Podracing’: experimenting with mobile TV content consumption and delivery methods. Multimed Syst 14(2):105–114

    Article  Google Scholar 

  60. Oulasvirta A (2009) Field experiments in HCI: promises and challenges. In: Saariluoma P, Isomaki H (eds) Future Interaction Design II, Springer

  61. Oulasvirta A, Nyyssönen T (2009) Flexible hardware configurations for studying mobile usability. J Usability Stud 4(2):93–105

    Google Scholar 

  62. Oulasvirta A, et al. (2005) Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI. Proc SIGCHI Conf Hum Factors Computing Syst CHI '05:919-928

  63. Pirhonen A, Brewster SA, Holguin C (2002) Gestural and audio metaphors as a means of control for mobile devices. Proc CHI’2002:291-298

  64. Reiter U, Jumisko-Pyykkö S (2007) Watch, press and catch—impact of divided attention on requirements of audiovisual quality. Proc 12th HCI Int 2007:943-952

  65. Roto V (2006) Web browsing on mobile phones—characteristics of user experience. Dissertation, Helsinki University of Technology

  66. Sasse MA, Knoche H (2006) Quality in context—an ecological approach to assessing QoS for mobile TV. Proc ISCA/DEGA Tutor & Res Workshop Percept Quality of Systems 2006

  67. Schwarz H, Marpe D, Wiegand T (2007) Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans Circ Syst Video Tech 17(9):1103–1120

    Article  Google Scholar 

  68. Shadish W, Cook T, Campbell D (2002) Experimental and quasi-experimental designs. Houghton Mifflin, Boston

    Google Scholar 

  69. Södergård C (ed) (2003) Mobile television—technology and user experiences. Report on the Mobile TV Project. Espoo: VTT Publications 506

  70. Strauss A, Corbin J (1998) Basics of qualitative research: techniques and procedures for developing grounded theory, 2nd edn. Sage, Thousand Oaks

    Google Scholar 

  71. Strohmeier D, Tech G (2010) Sharp, bright, three-dimensional: open profiling of quality for mobile 3DTV coding methods. Proc SPIE Multimedia on Mobile Devices 2010

  72. Strohmeier D, Jumisko-Pyykkö S, Kunze K (2010) New, lively, and exciting or just artificial, straining, and distracting—a sensory profiling approach to understand mobile 3D audiovisual quality. Proc VPQM 2010

  73. Tamminen S et al (2004) Understanding mobile contexts. Pers Ubiquit Comput 8(2):135–143

    Article  Google Scholar 

  74. Tashakkori A, Teddlie C (2008) Quality of inferences in mixed methods research: calling for an intergrative framework. In: Bergman MM (ed) Advances in Mixed Method Research. Age Publications

  75. Winkler S, Faller C (2005) Audiovisual quality evaluation of low-bitrate video. Proc SPIE IS&T Hum Vis Electron Imaging 5666:139–148

    Google Scholar 

Download references

Acknowledgements

This work is supported by the European Commission within the ICT program of FP7 under Grant 216503 with the acronym MOBILE3DTV (www.mobile3dtv.eu) and the UCIT graduate school. The authors wish to thank Dr. Miska Hannuksela and Dr. Atanas Gotchev for their comments and Cinovent, Red Star Studio, Stereoscape and the Centre of Computer Graphics and Visualization from the University of West Bohemia for providing content for the experiments.

Author information

Authors and Affiliations

  1. Human-Centered Technology, Tampere University of Technology, Korkeakoulunkatu 6, P.O. Box 589, 33101, Tampere, Finland

    Satu Jumisko-Pyykkö & Timo Utriainen

Authors
  1. Satu Jumisko-Pyykkö
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timo Utriainen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Satu Jumisko-Pyykkö.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jumisko-Pyykkö, S., Utriainen, T. A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television. Multimed Tools Appl 55, 185–225 (2011). https://doi.org/10.1007/s11042-010-0573-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0573-4

Keywords

Search

Navigation