Thao Ngo
Johannes Kunkel
ABSTRACT
While online content is personalized to an increasing degree, eg. using recommender systems (RS), the rationale behind personalization and how users can adjust it typically remains opaque. This was often observed to have negative effects on the user experience and perceived quality of RS. As a result, research increasingly has taken user-centric aspects such as transparency and control of a RS into account, when assessing its quality. However, we argue that too little of this research has investigated the users' perception and understanding of RS in their entirety. In this paper, we explore the users' mental models of RS. More specifically, we followed the qualitative grounded theory methodology and conducted 10 semi-structured face-to-face interviews with typical and regular Netflix users. During interviews participants expressed high levels of uncertainty and confusion about the RS in Netflix. Consequently, we found a broad range of different mental models. Nevertheless, we also identified a general structure underlying all of these models, consisting of four steps: data acquisition, inference of user profile, comparison of user profiles or items, and generation of recommendations. Based on our findings, we discuss implications to design more transparent, controllable, and user friendly RS in the future.
Supplemental Material
Available for Download
- Svetlin Bostandjiev, John O'Donovan, and Tobias Höllerer. 2012. TasteWeights: A Visual Interactive Hybrid Recommender System. In Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys '12). ACM, New York, NY, USA, 35--42. https://doi.org/10.1145/2365952.2365964Google Scholar
Digital Library
- André Calero Valdez, Martina Ziefle, and Katrien Verbert. 2016. HCI for Recommender Systems: The Past, the Present and the Future. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). ACM, New York, NY, USA, 123--126. https://doi.org/10.1145/2959100.2959158Google ScholarDigital Library
- Shuo Chang, F. Maxwell Harper, and Loren Terveen. 2015. Using Groups of Items for Preference Elicitation in Recommender Systems. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1258--1269. https://doi.org/10.1145/2675133.2675210Google ScholarDigital Library
- Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor. 2019. In the shades of the uncanny valley: An experimental study of human--chatbot interaction. Future Generation Computer Systems, Vol. 92 (March 2019), 539--548. https://doi.org/10.1016/j.future.2018.01.055Google ScholarDigital Library
- Juliet Corbin and Anselm Strauss. 2008. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory .SAGE Publications, Thousand Oaks, California. https://doi.org/10.4135/9781452230153Google Scholar
- Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction, Vol. 18, 5 (Aug. 2008), 455. https://doi.org/10.1007/s11257-008--9051--3Google ScholarDigital Library
- Will Dahlgreen. 2016. Streaming wars: the actors Netflix and Amazon customers want to see. https://yougov.co.uk/topics/politics/articles-reports/2016/01/14/streaming-wars-actors-netflix-and-amazon-customers Retrieved January, 15, 2020 fromGoogle Scholar
- Deloitte. 2017. Welchen Video-on-Demand-Anbieter nutzen Sie? https://de.statista.com/statistik/daten/studie/443820/umfrage/genutzte-video-on-demand-anbieter-in-deutschland/ Retrieved April, 24, 2020 fromGoogle Scholar
- Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike Haug, and Heinrich Hussmann. 2018. Bringing Transparency Design into Practice. In 23rd International Conference on Intelligent User Interfaces (IUI '18). ACM, New York, NY, USA, 211--223. https://doi.org/10.1145/3172944.3172961Google ScholarDigital Library
- Muheeb Faizan Ghori, Arman Dehpanah, Jonathan Gemmell, Hamed Qahri-Saremi, and Bamshad Mobasher. 2019. Does the User Have A Theory of the Recommender? A Pilot Study. In Proceedings of Joint Workshop on Interfaces and Human Decision Making for Recommender Systems. CEUR-WS.org, 77--85.Google Scholar
- Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, Vol. 6, 4 (Dec. 2015), 13:1--13:19. https://doi.org/10.1145/2843948Google Scholar
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black Box Models. Comput. Surveys, Vol. 51, 5, Article Article 93 (Aug. 2018), 42 pages. https://doi.org/10.1145/3236009Google ScholarDigital Library
- Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000. Explaining Collaborative Filtering Recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, 241--250. https://doi.org/10.1145/358916.358995Google ScholarDigital Library
- Christie Kodama, Beth St. Jean, Mega Subramaniam, and Natalie Greene Taylor. 2017. There's a creepy guy on the other end at Google!: engaging middle school students in a drawing activity to elicit their mental models of Google. Information Retrieval Journal, Vol. 20, 5 (Oct. 2017), 403--432. https://doi.org/10.1007/s10791-017--9306-xGoogle ScholarDigital Library
- Joseph A. Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User Modeling and User-Adapted Interaction, Vol. 22, 1--2 (March 2012), 101--123. https://doi.org/10.1007/s11257-011--9112-xGoogle ScholarDigital Library
- Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell Me More?: The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1--10. https://doi.org/10.1145/2207676.2207678Google ScholarDigital Library
- Johannes Kunkel, Tim Donkers, Catalin-Mihai Barbu, and Jürgen Ziegler. 2018a. Trust-Related Effects of Expertise and Similarity Cues in Human-Generated Recommendations. In Companion Proceedings of the 23rd International on Intelligent User Interfaces: 2nd Workshop on Theory-Informed User Modeling for Tailoring and Personalizing Interfaces (HUMANIZE). http://ceur-ws.org/Vol-2068/humanize5.pdfGoogle Scholar
- Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300717Google ScholarDigital Library
- Johannes Kunkel, Benedikt Loepp, and Jürgen Ziegler. 2017. A 3D Item Space Visualization for Presenting and Manipulating User Preferences in Collaborative Filtering. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (IUI '17). ACM, New York, NY, USA, 3--15. https://doi.org/10.1145/3025171.3025189Google ScholarDigital Library
- Johannes Kunkel, Benedikt Loepp, and Jürgen Ziegler. 2018b. Understanding Latent Factors Using a GWAP. In Proceedings of the Late-Breaking Results track part of the Twelfth ACM Conference on Recommender Systems (RecSys'18). https://arxiv.org/pdf/1808.10260.pdfGoogle Scholar
- Benedikt Loepp, Tim Donkers, Timm Kleemann, and Jürgen Ziegler. 2019. Interactive recommending with Tag-Enhanced Matrix Factorization (TagMF). International Journal of Human-Computer Studies 121 (Jan. 2019), 21--41. https://doi.org/10.1016/j.ijhcs.2018.05.002Google ScholarCross Ref
- Benedikt Loepp, Tim Hussein, and Jürgen Ziegler. 2014. Choice-based preference elicitation for collaborative filtering recommender systems. In Proceedings of the 32nd International Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3085--3094. https://doi.org/10.1145/2556288.2557069Google ScholarDigital Library
- Julian McAuley and Jure Leskovec. 2013. Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys '13). ACM, New York, NY, USA, 165--172. https://doi.org/10.1145/2507157.2507163Google ScholarDigital Library
- Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 1097--1101. https://doi.org/10.1145/1125451.1125659Google Scholar
- Janice M. Morse. 2015. "Data Were Saturated...". Qualitative Health Research, Vol. 25, 5 (2015), 587--588. https://doi.org/10.1177/1049732315576699Google ScholarCross Ref
- Donald A. Norman. 1983. Some Observations on Mental Models. In Mental Models, Dedre Gentner and Albert L. Stevens (Eds.). Psychology Press, New York, NY, USA, 7--14.Google Scholar
- Donald A. Norman. 1988. The design of everyday things. Basic Books, Inc., New York, NY, USA.Google ScholarDigital Library
- John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: Visual Interactive Recommendation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1085--1088. https://doi.org/10.1145/1357054.1357222Google ScholarDigital Library
- Pearl Pu, Li Chen, and Rong Hu. 2011. A User-centric Evaluation Framework for Recommender Systems. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). ACM, New York, NY, USA, 157--164. https://doi.org/10.1145/2043932.2043962Google ScholarDigital Library
- Lingyun Qiu and Izak Benbasat. 2009. Evaluating Anthropomorphic Product Recommendation Agents: A Social Relationship Perspective to Designing Information Systems. Journal of Management Information Systems, Vol. 25, 4 (Dec. 2009), 145--182. https://doi.org/10.2753/MIS0742--1222250405Google ScholarDigital Library
- Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, and John Riedl. 2002. Getting to Know You: Learning New User Preferences in Recommender Systems. In Proceedings of the 7th International Conference on Intelligent User Interfaces (IUI '02). ACM, New York, NY, USA, 127--134.Google ScholarDigital Library
- Quentin Roy, Futian Zhang, and Daniel Vogel. 2019. Automation Accuracy Is Good, but High Controllability May Be Better. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 520:1--520:8. https://doi.org/10.1145/3290605.3300750Google ScholarDigital Library
- David E. Rumelhart and Donald A. Norman. 1983. Representation in Memory.Google Scholar
- Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents. interactions, Vol. 4, 6 (Nov. 1997), 42--61. https://doi.org/10.1145/267505.267514Google Scholar
- Itamar Simonson. 2005. Determinants of Customers' Responses to Customized Offers: Conceptual Framework and Research Propositions. Journal of Marketing, Vol. 69, 1 (Jan. 2005), 32--45. https://doi.org/10.1509/jmkg.69.1.32.55512Google ScholarCross Ref
- Rashmi Sinha and Kirsten Swearingen. 2002. The role of transparency in recommender systems. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02). ACM, New York, NY, USA, 830--831. https://doi.org/10.1145/506443.506619Google ScholarDigital Library
- E. Isaac Sparling and Shilad Sen. 2011. Rating: how difficult is it?. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). ACM, New York, NY, USA, 149--156.Google ScholarDigital Library
- Statista.com. 2018. Most popular video streaming services in the United States as of July 2018, by monthly average users. https://www.statista.com/statistics/910875/us-most-popular-video-streaming-services-by-monthly-average-users/s Retrieved January, 15, 2020 fromGoogle Scholar
- Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. In Handbook of qualitative research, Norman K. Denzin and Yvonna S. Lincoln (Eds.). SAGE Publications, Thousand Oaks, CA, USA, 273--285.Google Scholar
- Nava Tintarev and Judith Masthoff. 2015. Explaining Recommendations: Design and Evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer US, Boston, MA, USA, 353--382. https://doi.org/10.1007/978--1--4899--7637--6textunderscore10Google Scholar
- Helma Torkamaan, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. How Can They Know That? A Study of Factors Affecting the Creepiness of Recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys '19). ACM, New York, NY, USA, 423--427. https://doi.org/10.1145/3298689.3346982Google ScholarDigital Library
- Chun-Hua Tsai and Peter Brusilovsky. 2019. Explaining Recommendations in an Interactive Hybrid Social Recommender. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI '19). ACM, New York, NY, USA, 391--396. https://doi.org/10.1145/3301275.3302318Google ScholarDigital Library
Index Terms
- Exploring Mental Models for Transparent and Controllable Recommender Systems: A Qualitative Study
Recommendations
User Personality and User Satisfaction with Recommender Systems
In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
A Scalable, Accurate Hybrid Recommender System
WKDD '10: Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data MiningRecommender systems apply machine learning techniques for filtering unseen information and can predict whether a user would like a given resource. There are three main types of recommender systems: collaborative filtering, content-based filtering, and ...
Comments