Benedikt Loepp
Tim Donkers
ABSTRACT
In user studies of recommender systems, participants typically cannot consume the recommended items. Still, they are asked to assess recommendation quality and other aspects related to user experience by means of questionnaires. Without having listened to recommended songs or watched suggested movies, however, this might be an error-prone task, possibly limiting validity of results obtained in these studies. In this paper, we investigate the effect of actually consuming the recommended items. We present two user studies conducted in different domains showing that in some cases, differences in the assessment of recommendations and in questionnaire results occur. Apparently, it is not always possible to adequately measure user experience without allowing users to consume items. On the other hand, depending on domain and provided information, participants sometimes seem to approximate the actual value of recommendations reasonably well.
Supplemental Material
- Gediminas Adomavicius, Jesse C. Bockstedt, Shawn P. Curley, and Jingjing Zhang. 2013. Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects. Information Systems Research 24, 4 (2013), 956--975. Google Scholar
Digital Library
- Mustafa Bilgic and Raymond J. Mooney. 2005. Explaining Recommendations: Satisfaction vs. Promotion. In Proceedings of the Beyond Personalization Workshop.Google Scholar
- Dirk Bollen, Mark P. Graus, and Martijn C. Willemsen. 2012. Remembering the Stars? Effect of Time on Preference Retrieval from Memory. In RecSys '12: Proceedings of the 6th ACM Conference on Recommender Systems. ACM, 217--220. Google ScholarDigital Library
- Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is Seeing Believing? How Recommender Interfaces Affect Users' Opinions. In CHI '03: Proceedings of the 21st ACM Conference on Human Factors in Computing Systems. ACM, 585--592. Google ScholarDigital Library
- Ap Dijksterhuis and Zeger van Olden. 2006. On the Benefits of Thinking Unconsciously: Unconscious Thought can Increase Post-Choice Satisfaction. Journal of Experimental Social Psychology 42, 5 (2006), 627--631.Google ScholarCross Ref
- Mark P. Graus and Martijn C. Willemsen. 2016. Can Trailers Help to Alleviate Popularity Bias in Choice-Based Preference Elicitation?. In IntRS '16: Proceedings of the 3rd Joint Workshop on Interfaces and Human Decision Making for Recommender Systems. 22--27.Google Scholar
- Asela Gunawardana and Guy Shani. 2015. Recommender Systems Handbook. Springer US, Chapter Evaluating Recommender Systems, 265--308.Google Scholar
- Will Hill, Larry Stead, Mark Rosenstein, and George Furnas. 1995. Recommending and Evaluating Choices in a Virtual Community of Use. In CHI '95: Proceedings of the 13th ACM Conference on Human Factors in Computing Systems. ACM, 194--201. Google ScholarDigital Library
- Michael Jugovac and Dietmar Jannach. 2017. Interacting with Recommenders - Overview and Research Directions. ACM Transactions on Interactive Intelligent Systems 7, 3 (2017), 10:1--10:46. Google ScholarDigital Library
- Bart P. Knijnenburg and Martijn C. Willemsen. 2015. Recommender Systems Handbook. Springer US, Chapter Evaluating Recommender Systems with User Experiments, 309--352.Google Scholar
- Bart P. Knijnenburg, Martijn C. Willemsen, and Alfred Kobsa. 2011. A Pragmatic Procedure to Support the User-Centric Evaluation of Recommender Systems. In RecSys '11: Proceedings of the 5th ACM Conference on Recommender Systems. ACM, 321--324. Google ScholarDigital Library
- Joseph A. Konstan and John Riedl. 2012. Recommender Systems: From Algorithms to User Experience. User Modeling and User-Adapted Interaction 22, 1-2 (2012), 101--123. Google ScholarDigital Library
- Benedikt Loepp, Catalin-Mihai Barbu, and Jürgen Ziegler. 2016. Interactive Recommending: Framework, State of Research and Future Challenges. In EnCHIReS '16: Proceedings of the 1st Workshop on Engineering Computer-Human Interaction in Recommender Systems. 3--13.Google Scholar
- Theodora Nanou, George Lekakos, and Konstantinos Fouskas. 2010. The Effects of Recommendations' Presentation on Persuasion and Satisfaction in a Movie Recommender System. Multimedia Systems 16, 4-5 (2010), 219--230. Google ScholarDigital Library
- Pearl Pu, Li Chen, and Rong Hu. 2011. A User-Centric Evaluation Framework for Recommender Systems. In RecSys '11: Proceedings of the 5th ACM Conference on Recommender Systems. ACM, 157--164. Google ScholarDigital Library
- Pearl Pu, Li Chen, and Rong Hu. 2012. Evaluating Recommender Systems from the User's Perspective: Survey of the State of the Art. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 317--355. Google ScholarDigital Library
- Alan Said and Alejandro Bellogin. 2018. Coherence and Inconsistencies in Rating Behavior: Estimating the Magic Barrier of Recommender Systems. User Modeling and User-Adapted Interaction (2018). Google ScholarDigital Library
- Sylvain Senecal and Jacques Nantel. 2004. The Influence of Online Product Recommendations on Consumers' Online Choices. Journal of Retailing 80, 2 (2004), 159--169.Google ScholarCross Ref
- Amit Sharma and Dan Cosley. 2013. Do Social Explanations Work? Studying and Modeling the Effects of Social Explanations in Recommender Systems. In WWW '13: Proceedings of the 22nd International Conference on World Wide Web. ACM, 1133--1144. Google ScholarDigital Library
- Nava Tintarev and Judith Masthoff. 2008. The Effectiveness of Personalized Movie Explanations: An Experiment Using Commercial Meta-Data. In AH '08: Proceedings of the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems. Springer, 204--213. Google ScholarDigital Library
- Nava Tintarev and Judith Masthoff. 2008. Over- and Underestimation in Different Product Domains. In Proceedings of the ECAI Workshop on Recommender Systems. 14--19.Google Scholar
- Nava Tintarev and Judith Masthoff. 2012. Evaluating the Effectiveness of Explanations for Recommender Systems. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 399--439. Google ScholarDigital Library
- Nava Tintarev and Judith Masthoff. 2015. Recommender Systems Handbook. Springer US, Chapter Explaining Recommendations: Design and Evaluation, 353--382.Google Scholar
- Timothy D. Wilson, Douglas J. Lisle, Jonathan W. Schooler, Sara D. Hodges, Kristen J. Klaaren, and Suzanne J. LaFleur. 1993. Introspecting about Reasons can Reduce Post-Choice Satisfaction. Personality and Social Psychology Bulletin 19, 3 (1993), 331--339.Google ScholarCross Ref
- Timothy D. Wilson, Jay Meyers, and Daniel T. Gilbert. 2003. "How Happy was I, Anyway?" A Retrospective Impact Bias. Social Cognition 21, 6 (2003), 421--446.Google ScholarCross Ref
Index Terms
- Impact of item consumption on assessment of recommendations in user studies
Recommendations
Item cold-start recommendations: learning local collective embeddings
RecSys '14: Proceedings of the 8th ACM Conference on Recommender systemsRecommender systems suggest to users items that they might like (e.g., news articles, songs, movies) and, in doing so, they help users deal with information overload and enjoy a personalized experience. One of the main problems of these systems is the ...
Making recommendations from top-N user-item subgroups
Group-aware collaborative filtering (CF) has recently become a hot research topic in recommender systems, which typically divides a large CF task on the entire data (i.e. rating matrix) into some smaller CF tasks on subgroups (i.e., sub-matrices). This ...
Unifying user-based and item-based collaborative filtering approaches by similarity fusion
SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrievalMemory-based methods for collaborative filtering predict new ratings by averaging (weighted) ratings between, respectively, pairs of similar users or items. In practice, a large number of ratings from similar users or similar items are not available, ...
Comments