Andreas Möller
Matthias Kranz
Barbara Schmid
ABSTRACT
Self-reporting techniques, such as data logging or a diary, are frequently used in long-term studies, but prone to subjects' forgetfulness and other sources of inaccuracy. We conducted a six-week self-reporting study on smartphone usage in order to investigate the accuracy of self-reported information, and used logged data as ground truth to compare the subjects' reports against. Subjects never recorded more than 70% and, depending on the requested reporting interval, down to less than 40% of actual app usages. They significantly overestimated how long they used apps. While subjects forgot self-reports when no automatic reminders were sent, a high reporting frequency was perceived as uncomfortable and burdensome. Most significantly, self-reporting even changed the actual app usage of users and hence can lead to deceptive measures if a study relies on no other data sources.
With this contribution, we provide empirical quantitative long-term data on the reliability of self-reported data collected with mobile devices. We aim to make researchers aware of the caveats of self-reporting and give recommendations for maximizing the reliability of results when conducting large-scale, long-term app usage studies.
- Barkhuus, L., and Polichar, V. E. Empowerment through Seamfulness: Smart Phones in Everyday Life. Personal and Ubiquitous Computing 15, 6 (2010), 629--639. Google Scholar
Digital Library
- Barrett, L. F., and Barrett, D. J. An Introduction to Computerized Experience Sampling in Psychology. Social Science Computer Review 19, 2 (2001), 175--185. Google ScholarDigital Library
- Böhmer, M., Hecht, B., Schöning, J., Krüger, A., and Bauer, G. Falling asleep with Angry Birds, Facebook and Kindle: A Large Scale Study on Mobile Application Usage. In Proc. of the 13th Intl. Conference on Human Computer Interaction with Mobile Devices and Services, ACM (Stockholm, Sweden, 2011), 47--56. Google ScholarDigital Library
- Bolger, N., Davis, A., and Rafaeli, E. Diary Methods: Capturing Life as it is Lived. Annual Review of Psychology 54 (2003), 579--616.Google ScholarCross Ref
- Carter, S., and Mankoff, J. When Participants do the Capturing: the Role of Media in Diary Studies. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM (Portland, OR, USA, 2005), 899--908. Google ScholarDigital Library
- Carter, S., Mankoff, J., and Heer, J. Momento: Support for Situated Ubicomp Experimentation. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM (San Jose, CA, USA, 2007), 125--134. Google ScholarDigital Library
- Consolvo, S., Harrison, B., Smith, I., Chen, M., Everitt, K., Froehlich, J., and Landay, J. A. Conducting In Situ Evaluations for and with Ubiquitous Computing Technologies. Intl. Journal of Human-Computer Interaction 22, 1-2 (2007), 103--118.Google ScholarCross Ref
- Deane, F. P., Podd, J., and Henderson, R. D. Relationship between Self-Report and Log Data Estimates of Information System Usage. Computers in Human Behavior 14, 4 (1998), 621--636.Google ScholarCross Ref
- Demumieux, R., and Losquin, P. Gather Customer's Real Usage on Mobile Phones. In Proc. of the 7th Intl. Conference on Human Computer Interaction with Mobile Devices and Services, ACM (Salzburg, Austria, 2005), 267--270. Google ScholarDigital Library
- Eagle, N., Pentland, A. S., and Lazer, D. Inferring Friendship Network Structure by using Mobile Phone Data. Proc. of the National Academy of Sciences 106, 36 (2009), 15274--15278.Google ScholarCross Ref
- Falaki, H., Mahajan, R., Kandula, S., Lymberopoulos, D., Govindan, R., and Estrin, D. Diversity in Smartphone Usage. In Proc. of the 8th Intl. Conference on Mobile Systems, Applications, and Services, ACM (New York, NY, USA, 2010), 179--194. Google ScholarDigital Library
- Froehlich, J., Chen, M. Y., Consolvo, S., Harrison, B., and Landay, J. A. MyExperience: A System For In Situ Tracing and Capturing of User Feedback on Mobile Phones. In Proc. of the 5th Intl. Conference on Mobile Systems, Applications and Services, ACM (San Juan, Puerto Rico, 2007), 57--70. Google ScholarDigital Library
- Hartley, C., Brecht, M., Pagerey, P., Weeks, G., Chapanis, A., and Hoecker, D. Subjective Time Estimates of Work Tasks by Office Workers. Journal of Occupational Psychology 50, 1 (2011), 23--36.Google Scholar
- Henze, N., Rukzio, E., and Boll, S. Observational and Experimental Investigation of Typing Behaviour Using Virtual Keyboards for Mobile Devices. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM (Austin, TX, USA, 2012), 2659--2668. Google ScholarDigital Library
- Hufford, M. R., and Shields, A. L. Electronic Diaries: Applications and What Works in the Field. Applied Clinical Trials (2002), 46--59.Google Scholar
- Hulkko, S., Mattelmäki, T., Virtanen, K., and Keinonen, T. Mobile Probes. In Proc. of the 3rd Nordic Conference on Human-Computer Interaction, ACM (Tampere, Finland, 2004), 43--51. Google ScholarDigital Library
- Jansen, B. J., Taksa, I., and Spink, A. Handbook of Research on Web Log Analysis. IGI Global, 2009.Google ScholarCross Ref
- Jeon, M. H., Na, D. Y., Ahn, J. H., and Hong, J. Y. User segmentation & UI Optimization Through Mobile Phone Log Analysis. In Proc. of the 10th Intl. Conference on Human Computer Interaction with Mobile Devices and Services, ACM Press (Amsterdam, The Netherlands, 2008), 495--496. Google ScholarDigital Library
- Kärkkäinen, T., Vaittinen, T., and Väänänen-Vainio-Mattila, K. I Don't Mind Being Logged, But Want to Remain in Control: A Field Study of Mobile Activity and Context Logging. In Proc. of the 28th Intl. Conference on Human Factors in Computing Systems, ACM (Atlanta, GA, USA, 2010), 163--172. Google ScholarDigital Library
- Krumm, J. Ubiquitous Computing Fundamentals. Chapman & Hall / CRC, 2009. Google ScholarDigital Library
- Lazar, J., Feng, J. H., and Hochheiser, H. Research Methods in Human-Computer Interaction. John Wiley & Sons Ltd, Chichester, 2010. Google ScholarDigital Library
- Lester, J., Choudhury, T., and Borriello, G. A Practical Approach to Recognizing Physical Activities. Pervasive Computing (2006), 1--16. Google ScholarDigital Library
- Mankoff, J., and Carter, S. Crossing Qualitative and Quantitative Evaluation in the Domain of Ubiquitous Computing. In CHI2005 Workshop "Usage analysis: Combining Logging and Qualitative Methods" (2005).Google Scholar
- Möller, A., Michahelles, F., Diewald, S., Roalter, L., and Kranz, M. Update Behavior in App Markets and Security Implications: A Case Study in Google Play. In Proc. of the 3rd Intl. Workshop on Research in the Large. Held in Conjunction with Mobile HCI (2012), 3--6.Google Scholar
- Nielsen, J. Usability Engineering. Morgan Kaufmann, 1993. Google ScholarDigital Library
- Palen, L., and Salzman, M. Voice-Mail Diary Studies for Naturalistic Data Capture Under Mobile Conditions. In Proc. of the 2002 ACM Conference on Computer Supported Cooperative Work, ACM (New Orleans, LA, USA, 2002), 87--95. Google ScholarDigital Library
- Raento, M., Oulasvirta, A., Petit, R., and Toivonen, H. ContextPhone: A Prototyping Platform for Context-Aware Mobile Applications. Pervasive Computing, IEEE 4, 2 (2005), 51--59. Google ScholarDigital Library
- Robinson, J. P., and Godbey, G. Time For Life: The Surprising Ways Americans Use Their Time. Penn State University Press, 1997.Google Scholar
- Siek, K. A., Connelly, K. H., Rogers, Y., Rohwer, P., Lambert, D., and Welch, J. L. When Do We Eat? An Evaluation of Food Items Input Into an Electronic Food Monitoring Application. In Pervasive Health Conference and Workshops, 2006, IEEE (2006), 1--10.Google ScholarCross Ref
Index Terms
- Investigating self-reporting behavior in long-term studies
Recommendations
ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsWe introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-...
Towards Chatbot-Supported Self-Reporting for Increased Reliability and Richness of Ground Truth for Automatic Pain Recognition: Reflections on Long-Distance Runners and People with Chronic Pain
ICMI '21 Companion: Companion Publication of the 2021 International Conference on Multimodal InteractionPain is a ubiquitous and multifaceted experience, making the gathering of ground truth for training machine learning system particularly difficult. In this paper, we reflect on the use of voice-based Experience Sampling Method (ESM) approaches for ...
What's going on? Age, distraction, and multitasking during online survey taking
Participants in an online survey revealed what other activities they engaged in while taking it.Younger people were more likely than older people to multitask.The relationship between age, multitasking, and sense of distraction was curvilinear.Most ...
Comments