Bart P. Knijnenburg
Martijn C. Willemsen
Alfred Kobsa
ABSTRACT
As recommender systems are increasingly deployed in the real world, they are not merely tested offline for precision and coverage, but also "online" with test users to ensure good user experience. The user evaluation of recommenders is however complex and resource-consuming. We introduce a pragmatic procedure to evaluate recommender systems for experience products with test users, within industry constraints on time and budget. Researchers and practitioners can employ our approach to gain a comprehensive understanding of the user experience with their systems.
- Bollen, D. et al. 2010. Understanding choice overload in recommender systems. Proc. of the 4th ACM conf. on Rec-ommender systems. RecSys'10. ACM, New York, NY, 63--70. DOI= http://doi.acm.org/10.1145/1864708.1864724. Google Scholar
Digital Library
- Cosley, D. et al. 2003. Is seeing believing? Proc. of the SIGCHI conf. on Human factors in computing systems. ACM, New York, NY, 585--592. DOI= http://doi.acm.org/10.1145/642611.642713 Google ScholarDigital Library
- Davis, F.D. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly. 13, 3 (Sep. 1989), 319--340. DOI= http://dx.doi.org/10.2307/249008 Google ScholarDigital Library
- Knijnenburg, B.P. and Willemsen, M.C. 2010. The effect of preference elicitation methods on the user experience of a recommender system. Extended abstracts on Human factors in computing systems. ACM, New York, NY, 3457--3462. DOI= http://doi.acm.org/10.1145/1753846.1754001 Google ScholarDigital Library
- Knijnenburg, B.P. and Willemsen, M.C. 2009. Understand-ing the effect of adaptive preference elicitation methods on user satisfaction of a recommender system. Proc. of the 3rd ACM conf. on Recommender systems. ACM, New York, NY, 381--384. DOI= http://doi.acm.org/10.1145/1639714.1639793 Google ScholarDigital Library
- Knijnenburg, B.P. et al. Explaining the User Experience of Recommender Systems. Accepted to UMUAI. http://t.co/cC5qPr9 Google ScholarDigital Library
- McNee, S.M. et al. 2002. On the recommending of citations for research papers. Proc. of the 2002 ACM conf. on Com-puter supported cooperative work. ACM, New York, NY, 116--125. DOI= http://doi.acm.org/10.1145/587078.587096 Google ScholarDigital Library
- McNee, S.M. et al. 2006. Being accurate is not enough. Extended abstracts on Human factors in computing systems. ACM, New York, NY, 1097--1101. DOI= http://doi.acm.org/10.1145/1125451.1125659 Google ScholarDigital Library
- Murray, K.B. and Häubl, G. 2008. Interactive Consumer Decision Aids. Handbook of Marketing Decision Models. B. Wierenga, ed. Springer, Heidelberg, Germany, 55--77. DOI= http://dx.doi.org/10.1007/978-0--387--78213--3_3Google Scholar
- Nelson, P. 1970. Information and Consumer Behavior. Journal of Political Economy. 78, 2, 311--329. DOI= http://dx.doi.org/10.1086/259630Google ScholarCross Ref
- Ozok, A.A. et al. 2010. Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behaviour & Information Technology. 29, 1 (Jan. 2010), 57--83. DOI= http://dx.doi.org/10.1080/01449290903004012 Google ScholarDigital Library
- Pu, P. and Chen, L. 2010. User-Centric Evaluation Framework of Recommender Systems. Proc. ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces. CEUR-WS Vol-621. 14--21.Google Scholar
- Pu, P. et al. 2008. Evaluating product search and recom-mender systems for E-commerce environments. Electronic Commerce Research. 8, 1-2 (May. 2008), 1--27. DOI= http://dx.doi.org/10.1007/s10660-008-9015-z Google ScholarDigital Library
- Teltzrow, M. and Kobsa, A. 2004. Impacts of user privacy preferences on personalized systems. Designing personalized user experiences in eCommerce. C.-M. Karat, J. Blom, J. Karat, eds. Springer, Heidelberg, Germany, 315--332. DOI= http://dx.doi.org/10.1007/1-4020-2148-8_17 Google ScholarDigital Library
- Torres, R. et al. 2004. Enhancing digital libraries with Tech-Lens+. Proceedings of the 2004 joint ACM/IEEE conference on Digital libraries. ACM, New York, NY, 228--236. DOI= http://doi.acm.org/10.1145/996350.996402 Google ScholarDigital Library
- van Velsen, L. et al. (2008). User-centered evaluation of adaptive and adaptable systems: a literature review. Knowledge Engineering Review. 23, 3, 261--281. DOI= http://dx.doi.org/10.1017/S0269888908001379 Google ScholarDigital Library
- Ziegler, C.-N. et al. 2005. Improving recommendation lists through topic diversification. Proc. of the 14th intl. conf. on World Wide Web. ACM, New York, NY, 22--32. DOI= http://doi.acm.org/10.1145/1060745.1060754 Google ScholarDigital Library
Index Terms
- A pragmatic procedure to support the user-centric evaluation of recommender systems
Recommendations
CRS-Que: A User-centric Evaluation Framework for Conversational Recommender Systems
An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based ...
Conducting user experiments in recommender systems
RecSys '12: Proceedings of the sixth ACM conference on Recommender systemsThere is an increasing consensus in the field of recommender systems that we should move beyond the offline evaluation of algorithms towards a more user-centric approach. This tutorial teaches the essential skills involved in conducting user experiments,...
Bridging the Gap Between User-centric and Offline Evaluation of Personalized Recommendation Systems
UMAP '18: Adjunct Publication of the 26th Conference on User Modeling, Adaptation and PersonalizationIn this paper, we propose to evaluate recommender systems by conducting both offline and user-centric evaluations, while considering multiple quality aspects in realistic settings. This comprehensive evaluation would provide insight on how to improve ...
Comments