Pearl Pu
Li Chen
ABSTRACT
This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.
- Adomavicius, G. and Tuzhilin, A. 2005. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734--749. Google Scholar
Digital Library
- Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., et al. 2004. Using social psychology to motivate contributions to online communities. In Proceedings of CSCW '04. New York: ACM Press. Google ScholarDigital Library
- Chen, L. and Pu, P. 2006. Trust Building with Explanation Interfaces. In Proceedings of International Conference on Intelligent User Interface, 93--100. Google ScholarDigital Library
- Chen, L. and Pu, P. 2009. Interaction Design Guidelines on Critiquing-based Recommender Systems. User Modeling and User-Adapted Interaction Journal (UMUAI), Springer Netherlands, Volume 19, Issue3, 167--206. Google ScholarDigital Library
- Churchill, G.A. A paradigm for developing better measures of marketing constructs, Journal of Marketing Research 16 (1), 1979, pp. 64--73.Google Scholar
- Davis, F.D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13 319--339. Google ScholarDigital Library
- Grabner-Kräuter, S. and Kaluscha, E.A. 2003. Empirical research in on-line trust: a review and critical assessment Int. J. Hum.-Comput. Stud. (IJMMS) 58(6), 783--812. Google ScholarDigital Library
- Herlocker, J.L., Konstan, J.A., and Riedl, J. 2000. Explaining collaborative filtering recommendations. CSCW'00, 241--250. Google ScholarDigital Library
- Herlocker, J.L., Konstan, J.A., Terveen, L.G., and Riedl, J. 2004. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5--53. Google ScholarDigital Library
- Jones, N., and Pu, P. 2007. User Technology Adoption Issues in Recommender Systems. In Proceedings of Networking and Electronic Commerce Research Conference, 379--394.Google Scholar
- Kirakowski, J. 1993. SUMI: the Software Usability Measurement Inventory. British Journal of Educational Technology, 24 (3) 210--214.Google ScholarCross Ref
- Knijnenburg, B. P., Willemsen, M. C., Gantner, Z. and Soncu, H. 2011. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction Journal (UMUAI), Special Issue on User Interfaces for Recommender Systems. (upcoming) Google ScholarDigital Library
- McNee, S.M., Lam, S.K., Konstan, J.A., Riedal, J. 2003. Interfaces for eliciting new user preferences in recommender systems. User Modeling 2003, 178--187. Google ScholarDigital Library
- McNee, S.M., Riedl, J., and Konstan, J.A. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. CHI Extended Abstracts, 1097--1101. Google ScholarDigital Library
- Nunnally, J. C. 1978. Psychometric Theory.Google Scholar
- Ozok, A.A, Fan, Q., Norcio, A.F. 2010. Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behaviour & Information Technology, Volume 29, Issue 1, 57 -- 83. Google ScholarDigital Library
- Peterson, R.A. 1994. A meta-analysis of Cronbach's coefficient alpha. Journal of Consumer Research, Vol. 21 September, pp. 381--91.Google ScholarCross Ref
- Pu, P., Chen, L., and Kumar, P. 2008. Evaluating Product Search and Recommender Systems for E-Commerce Environments. Electronic Commerce Research Journal, 8(1--2), 1--27. Google ScholarDigital Library
- Pu, P., Zhou, M., and Castagnos, S. 2009. Critiquing Recommenders for Public Taste Products. In proceedings of RecSys'09. New York, ACM Press, 249--252. Google ScholarDigital Library
- Pu, P. and Chen, L. 2010. A User-Centric Evaluation Framework of Recommender Systems. In the 3rd ACM Conference on Recommender Systems, Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces, Barcelona, Spain, Sept. 26-30.Google Scholar
- Simonson, I. 2005. Determinants of customers' responses to customized offers: Conceptual framework and research propositions. Journal of Marketing, 69, 32--45.Google ScholarCross Ref
- Swearingen, K. and Sinha, R. 2002. Interaction design for recommender systems. In Interactive Systems (DIS2002).Google Scholar
- Tintarev, N. and Masthoff, J. 2007. Survey of explanations in recommender systems. ICDE Workshops 2007, 801--810. Google ScholarDigital Library
- Venkatesh,V., Morris, M.G., Davis, G.B. and Davis, F.D. 2003. User acceptance of information technology: Toward a unified view. MIS Quarterly, 2003, 27, 3, 425--478. Google ScholarDigital Library
- Ziegler, C.N., McNee, S.M., Konstan, J.A., and Lausen, G. 2005. Improving Recommendation Lists through Topic Diversification. In Proc. of WWW 2005, ACM Press (2005), 22--32. Google ScholarDigital Library
- Xiao, B. and Benbasat, I. 2007. Ecommerce Product Recommendation Agents: Use, Characteristics, and Impact. Mis Quarterly 31(1), 137--209. Google ScholarDigital Library
Index Terms
- A user-centric evaluation framework for recommender systems
Recommendations
Investigating serendipity in recommender systems based on real user feedback
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingOver the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-...
User Personality and User Satisfaction with Recommender Systems
In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
Comments