What Recommenders Recommend – An Analysis of Accuracy, Popularity, and Sales Diversity Effects

In academic studies, the evaluation of recommender system (RS) algorithms is often limited to offline experimental designs based on historical data sets and metrics from the fields of Machine Learning or Information Retrieval. In real-world settings, however, other business-oriented metrics such as click-through-rates, customer retention or effects on the sales spectrum might be the true evaluation criteria for RS effectiveness. In this paper, we compare different RS algorithms with respect to their tendency of focusing on certain parts of the product spectrum. Our first analysis on different data sets shows that some algorithms – while able to generate highly accurate predictions – concentrate their top 10 recommendations on a very small fraction of the product catalog or have a strong bias to recommending only relatively popular items than others. We see our work as a further step toward multiple-metric offline evaluation and to help service providers make better-informed decisions when looking for a recommendation strategy that is in line with the overall goals of the recommendation service.