Gediminas Adomavicius
Abstract
The article explores stability as a new measure of recommender systems performance. Stability is defined to measure the extent to which a recommendation algorithm provides predictions that are consistent with each other. Specifically, for a stable algorithm, adding some of the algorithm’s own predictions to the algorithm’s training data (for example, if these predictions were confirmed as accurate by users) would not invalidate or change the other predictions. While stability is an interesting theoretical property that can provide additional understanding about recommendation algorithms, we believe stability to be a desired practical property for recommender systems designers as well, because unstable recommendations can potentially decrease users’ trust in recommender systems and, as a result, reduce users’ acceptance of recommendations. In this article, we also provide an extensive empirical evaluation of stability for six popular recommendation algorithms on four real-world datasets. Our results suggest that stability performance of individual recommendation algorithms is consistent across a variety of datasets and settings. In particular, we find that model-based recommendation algorithms consistently demonstrate higher stability than neighborhood-based collaborative filtering techniques. In addition, we perform a comprehensive empirical analysis of many important factors (e.g., the sparsity of original rating data, normalization of input data, the number of new incoming ratings, the distribution of incoming ratings, the distribution of evaluation data, etc.) and report the impact they have on recommendation stability.
- Adomavicius, G. and Tuzhilin, A. 2005. Toward the next generation of recommendation system: A survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Engin. 17, 734--749. Google Scholar
Digital Library
- Adomavicius, G. and Zhang, J. 2010. On the stability of recommendation algorithms. In Proceedings of the ACM Conference on Recommender Systems. 47--54. Google ScholarDigital Library
- Balabanovic, M. and Shoham, Y. 1997. Fab: Content-based, collaborative recommendation. Comm. ACM 40, 3, 66--72. Google ScholarDigital Library
- Bell, R. M. and Koren, Y. 2007a. Improved neighborhood-based collaborative filtering. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 7--14.Google Scholar
- Bell, R. M. and Koren, Y. 2007b. Lessons from the Netflix prize challenge. ACM SIGKDD Explor. Newslet. 9, 75--79. Google ScholarDigital Library
- Bell, R. M. and Koren, Y. 2007c. Scalable collaborative filtering with jointly derived neighborhood interpolation weights. In Proceedings of the 7th IEEE International Conference on Data Mining. Google ScholarDigital Library
- Bennett, J. and Lanning, S. 2007. The Netflix Prize. In Proceedings of the KDD-Cup and Workshop. www.netflixprize.com.Google Scholar
- Billsus, D. and Pazzani, M. 1998. Learning collaborative information Filters. In Proceedings of the 15th International Conference on Machine Learning (ICML’98). 46--54. Google ScholarDigital Library
- Breese, J. S., Heckerman, D., and Kadie, C. 1998. Empirical analysis of predictive algorithms for collaborative filtering. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence. Google ScholarDigital Library
- Burden, R. L. and Faires, D. 2004. Numerical Analysis. Thomson Brooks/Cole.Google Scholar
- D’stous, A. and Touil, N. 1999. Consumer evaluations of movies on the basis of critics’ judgments. Psych. Market. 16, 677--694.Google ScholarCross Ref
- Dias, M. B., Locher, D., Li, M., El-Deredy, W., and Lisboa, P. 2008. The value of personalised recommender systems to e-business: A case study. In Proceedings of the ACM Conference on Recommender Systems (RecSys’08). ACM, New York, NY, 291--294. Google ScholarDigital Library
- Duda, R. O., Hart, P. E., and Stork, D. G. 2000. Pattern Classification. Wiley. Google ScholarDigital Library
- Funk, S. 2006. Netflix update: Try this at home. http://sifter.org/~simon/journal/20061211.html.Google Scholar
- Garfinkel, R., Gopal, R., Pathak, B., Venkatesan, R., and Yin, F. 2006. Empirical analysis of the business value of recommender systems. http://ssrn.com/abstract=958770.Google Scholar
- Gershoff, A., Mukherjee, A., and Mukhopadhyay, A. 2003. Consumer acceptance of online agent advice: Extremity and positivity effects. J. Consumer Psych. 13, 1&2, 161--170.Google ScholarCross Ref
- Goldberg, K., Roeder, T., Gupta, D., and Perkins, C. 2001. Eigentaste: A constant time collaborative filtering algorithm. Inf. Retrieval 4, 133--151. Google ScholarDigital Library
- Grouplens. 2006. Movielens Data Sets. http://www.grouplens.org/.Google Scholar
- Hastie, T., Tibshirani, R., and Friedman, J. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.Google Scholar
- Herlocker, J., Kostan, J., Borchers, A., and Riedl, J. 1999. An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd ACM SIGIR Conference on Information Retrieval. 230--237. Google ScholarDigital Library
- Herlocker, J., Kostan, J., Terveen, K., and Riedl, J. T. 2004. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22, 5--53. Google ScholarDigital Library
- Higham, N. J. 2002. Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia. Google ScholarDigital Library
- Komiak, S. and Benbasat, I. 2006. The effects of personalization and familiarity on trust and adoption of recommendation agents. MIS Quart. 30, 941--960. Google ScholarDigital Library
- Koren, Y. 2010. Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Trans. Knowl. Discov. Data 4, 1--24. Google ScholarDigital Library
- Koren, Y., Bell, R., and Volinsky, C. 2009. Matrix factorization techniques for recommender systems. IEEE Comput. 42, 30--37. Google ScholarDigital Library
- Kostan, J., Miller, B., Maltz, D., Herlocker, J., Gordon, L., and Riedl, J. 1997. GroupLens: Applying collaborative filtering to usenet news. Comm. ACM 40, 77--87. Google ScholarDigital Library
- Lam, S. and Riedl, J. 2004. Shilling recommender systems for fun and profit. In Proceedings of the 13th International Conference on World Wide Web. Google ScholarDigital Library
- Legwinski, T. 2010. Recommendation explanations increase sales & customer confidence. http://blog.strands.com/2010/03/17/recommendations-increase-sales-customer-confidence/.Google Scholar
- Lyapunov, A. M. 1992. The General Problem of the Stability of Motion. CRC Press.Google Scholar
- Massa, P. and Avesani, P. 2006. Trust-aware Bootstrapping of recommender systems. In Proceedings of the ECAI Workshop on Recommender Systems. Google ScholarDigital Library
- Massa, P. and Bhattacharjee, B. 2004. Using trust in recommender systems: An experimental analysis. In Proceedings of the International Conference on Trust Management. Springer, 221--235.Google Scholar
- Mobasher, B., Burke, R., and Sandvig, J. J. 2006a. Model-based collaborative filtering as a defense against profile injection attacks. In Proceedings of the 21st Conference on Artificial Intelligence (AAAI’06). Google ScholarDigital Library
- Mobasher, B., Burke, R., Williams, C., and Bhaumik, R. 2006b. Analysis and detection of segment-focused attacks against collaborative recommendation. In Advances in Web Mining and Web Usage Analysis. Springer, 96--118. Google ScholarDigital Library
- Mobasher, B., Burke, R., Bhaumik, R., and Williams, C. 2007. Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness. ACM Trans. Intern. Techn. 7, 23, 21--38. Google ScholarDigital Library
- O’Donovan, J. and Smyth, B. 2005. Trust in recommender systems. In Proceedings of the 10th International Conference on Intelligent User Interfaces. Google ScholarDigital Library
- O’Donovan, J. and Smyth, B. 2006. Mining trust values from recommendation errors. Int. J. Artif. Intell. Tools 15, 945--962.Google ScholarCross Ref
- O’Mahony, M. P., Hurley, N. J., and Silvestre, G. C. M. 2004. An evaluation of neighbourhood formation on the performance of collaborative filtering. Artif. Intell. Rev. 21, 215--228. Google ScholarDigital Library
- Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the Conference on Computer Supported Cooperative Work. 175--186. Google ScholarDigital Library
- Rigby, C. 2011. Internet Retailing webinars. Review: Boosting email marketing revenue with personalised recommendations by Silverpop, Baynote and MusicRoom.com. http://www.internetretailing.net/2011/10/internet-retailing-webinars-review-boosting-email-marketing-revenue-with-personalised-recommendations-by-silverpop-baynote-and-musicroom-com/.Google Scholar
- Sarwar, B., Konstan, J., Borchers, A., Herlocker, J., Miller, B., and Riedl, J. 1998. Using filtering agents to improve prediction quality in the Grouplens research collaborative filtering system. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 345--354. Google ScholarDigital Library
- Sarwar, B., Karypis, G., Konstan, J. A., and Riedl, J. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conferenceon the World Wide Web. 285--295. Google ScholarDigital Library
- Shani, G. and Gunawardana, A. 2011. Evaluating recommender systems. In Recommender Systems Handbook: A Complete Guide for Research Scientists and Practitioners, P. Kantor, F. Ricci, L. Rokach, and B. Shapira Eds., Springer.Google Scholar
- Simitses, G. and Hodges, D. 2005. Fundamentals of Structural Stability. Butterworth-Heinemann, Burlington, MA.Google Scholar
- Turney, P. 1994. A theory of cross-validation error. J. Exp. Theor. Artif. Intell. 6, 361--391.Google ScholarCross Ref
- Turney, P. 1995. Technical note: Bias and the quantification of stability. Mach. Learn. 20, 23--33. Google ScholarDigital Library
- Van Swol, L. M. and Sniezek, J. A. 2005. Factors affecting the acceptance of expert advice. Brit. J. Social Psych. 44, 3, 443--461.Google ScholarCross Ref
- Wang, W. and Benbasat, I. 2005. Trust in and adoption of online recommendation agents. J. Assoc. Inf. Syst. 6, 72--101.Google ScholarCross Ref
Index Terms
- Stability of Recommendation Algorithms
Recommendations
On the stability of recommendation algorithms
RecSys '10: Proceedings of the fourth ACM conference on Recommender systemsThe paper introduces stability as a new measure of the recommender systems performance. In general, we define a recommendation algorithm to be "stable" if its predictions for the same items are consistent over a period of time, assuming that any new ...
Classification, Ranking, and Top-K Stability of Recommendation Algorithms
Recommendation stability measures the extent to which a recommendation algorithm provides predictions that are consistent with each other. Several approaches have been proposed in prior work to defining, measuring, and improving the stability of ...
A recursive prediction algorithm for collaborative filtering recommender systems
RecSys '07: Proceedings of the 2007 ACM conference on Recommender systemsCollaborative filtering (CF) is a successful approach for building online recommender systems. The fundamental process of the CF approach is to predict how a user would like to rate a given item based on the ratings of some nearest-neighbor users (user-...
Comments