How good your recommender system is? A survey on evaluations in recommendation

Utility has been mentioned in the literature in many names, such as relevance, usefulness, recommendation value and satisfaction. In the Recommender Systems Handbook, Ricci et al. [29] argues that utility represents the value that users receives in being recommended. As their own definition mentions, if the user enjoys the recommended items, he/she received useful recommendations. Moreover, utility has been defined as an order of preference of consumption. If users would only consume what they like most in the first place, therefore, recommending such items would help him/her find them easily, bringing usefulness to the recommendation Herlocker et al. [14]. Moreover, Adamopoulos et al. [1] cites the use of utility theory of economics for improving user satisfaction. Kotkov et al. [20, 21] also mention in a survey that utility or relevance relates to what the user is interested in consuming and therefore related to the tastes of the user.

As it can be seen, most of the definitions associate utility with the desires of consumption of the user and if the user enjoyed the recommendations. On such definition, metrics for assessing utility in recommendation should focus on how the user might react to the predictions made by a recommender. Ricci et al. [29] mention that utility could be measured by evaluating the rating that the user gives to predicted items after consuming them. This method is likely to be correct and capture if the recommendations brought value to the user, however it would involve in a costly online evaluation.

For offline evaluation, Herlocker et al. [14] mention the use of accuracy-based metrics for evaluating utility. The authors discuss the use of predictive accuracy metrics in order to evaluate if the user consumed the recommended items, usually in a train/test experiment. In this paper, we use the notation \(util\left( {{R_u}} \right)~\)for utility, however there are many metrics for this purpose that the following subsections show.

The concept of novelty generally involves the idea of having novel items in the recommendation. Although it seems to be simple at first, novelty has various definitions in the literature. Therefore, in order to make the definition easier, we classify the novelty definitions and metrics into three levels, as it is presented in Table 2. Novelty metrics are called \(nov({R_u})\) in this paper.

Diversity is a concept concerned with the diversity of items in the recommendation list. It has also been widely studied by previous researchers. For diversity metrics, the notation used in this paper is \(div({R_u})\).

According to the Ricci et al. [29], diversity in RSs have the contrary effect of similarity. The authors state that recommendation lists with low variety may not be of interest of the user. Moreover, one of the earliest work concerned with diversification in recommendation is [41]. The authors argue that the RSs usually predict similar items compared to the user’s consumption history. Therefore, diversity means balancing recommendation lists to cover the user’s whole set of interests [41]. In addition, Vargas and Castells [36] state that diversity refers to the variety of the items in the recommendation list. Moreover, Hurley and Zhang [15] and Zhang et al. [38] reinforce the definition of diversity from [41] stating that it is related to the variation of items in predictions of a RS.

Differently from novelty, the definitions of diversity are largely consistent in the literature. All the authors surveyed in this work agree that diversity represents variety of items in recommendation lists.

As a result of this definition, the proposed metrics tend to calculate diversity as a dissimilarity between the items in the recommendation list. Ziegler et al. [41] proposed a metric for intra-list similarity, as Eq. 12 show. The function \(d(i,j)\) calculates the distance between items \(i\) and \(j\) in the recommendation list \({R_u}\). This metric actually captures the similarity of the list; therefore, low values for this metric represent a more similar list, in which the items are similar to one another.

The intra-list similarity metric was also used by other works in diversity. Zhang et al. [38] used the metric proposed by Ziegler et al. [41] and chose the cosine similarity as the distance function. The metric can be seen in Eq. 13. Moreover, Hurley and Zhang [15] used a similar diversity metric as 11, where the distance function (\(d(i,j)\)) is calculated by a Collaborative Filtering memory-based similarity metric.

Individually, Vargas and Castells [36] proposed a distinct metric for calculating similarity. Their metric, as stated in Eq. 14, is a more specific case of the intra-list similarity. The metric takes into consideration a relative rank discount function for the position of each pair of items being analyzed (\(disc(k)\) and \(~disc(l|k)\)). Moreover, the metric also uses a distance function (\(d({i_k},{i_l})\)) between the items, for instance cosine similarity distance. The approach proposed by Vargas and Castells [36] resembles techniques used in Rank Information Retrieval for both the authors’ novelty and diversity metrics.

Unexpectedness is a concept that has been increasingly mentioned in the literature, but it is still involves uncertain definitions. It is usually linked to surprise and avoidance of obviousness in recommendation. The notation used to show the unexpectedness metrics in this paper is \(unexp({R_u})\).

Unexpectedness was firstly stated a component of serendipity by McNee et al. [24] and Ge et al. [12]. In both researches, the authors use the term unexpectedness to define the idea of surprise in recommendation. Moreover, Kaminskas and Bridge [18] also mention that unexpectedness represents surprise in recommendation.

Unexpectedness has also been defined as a divergence from expected recommendations. In Murakami et al. [25], the authors also state unexpectedness as a part of serendipity and describe it as a deviation from items that the user expect to consume, however the authors mostly focus on serendipity. Adamopoulos et al. [1] also explain that serendipity and unexpectedness concepts have been overlapping. The user expectations consist on the set of items the user would like to consume next or the items the user forecast to be recommended [1]. Therefore, unexpectedness would be a deviation from these expected items, evading obvious and uninteresting recommendations, with the possibility of surprising the user [1].

Measuring unexpectedness is not trivial, due to its overlapping definitions. Two set of metrics have been proposed in the literature: metrics based on a primitive recommender and metrics based on principles not involving a primitive method. We present both set of metrics as follows.

Serendipity has been increasingly used in recommender systems, however it has a complicated definition. In this section, metrics for serendipity use the notation \(ser({R_u})\).

The term serendipity means a lucky finding or a satisfying surprise. According to Ricci et al. [29], serendipity represent surprising recommendations. One of the earliest mentions of serendipity in the literature comes from [14]. The authors use the word serendipity as a concept of surprising and interesting item for the user. The same is stated by Iaquinta et al. [17], where they mention serendipity represent items that the users would difficultly find. Moreover, Ge et al. [12] define serendipitous items as surprising and pleasant. As it can be seen, most authors agree that the serendipity concept involve a good and pleasant surprise. However, it is necessary to state that serendipity is a perception of the users with regard to the recommendations they receive [12, 20].

Other definitions of serendipity can be found in the literature. For instance, Zhang et al. [38] and Kotkov et al. [20, 21] say that serendipitous recommendations are unusual and surprising to the users. Furthermore, serendipity can also be seen as good emotional answer to a novel recommendation that the users were not expecting to receive [1]. In this definition, Adamopoulos et al. [1] conclude that serendipitous recommendations are novel, unexpected and useful. Ge et al. [12] and Murakami et al. [25] also associate serendipity and unexpectedness.

Several other work have been studying serendipity problem in recommendation. For example, Lu et al. [11], Onuma et al. [22] and Gemmis [27] are some works that try to propose algorithms for serendipity. Additionally, Kotkov et al. [20, 21] made a large survey on serendipity. In summary, it can be concluded that even though serendipity has a hard to understand definition, most authors agree that it represents a delightful surprise and provide useful and surprising items to the user.

Metrics have been proposed to measure serendipity in recommendation lists and most of them have some relation to the concepts that serendipity is involved to: level 2 of novelty, unexpectedness and utility. Some metrics attempt to use unexpectedness notion of a primitive recommender. Therefore, we divide the metrics into the primitive recommender based metrics and non primitive recommender based.

Coverage is another concept that has been analyzed by previous researches in recommender systems. Although there have been few studies proposing metrics for coverage, it still worthy to mention due to its potential relation with the other explored concepts in this paper. In addition, other concepts such as trust, risk, robustness are far less studied than coverage. The notation of coverage used in this paper is \(cov\). Coverage evaluates the whole RS, not a recommendation list. Moreover, three kinds of coverage are mentioned in the literature: item space coverage and user space coverage, as presented by Ricci et al. [29], and genre space coverage, as proposed by Vargas et al. [35]. In our survey, only metrics for item space coverage were found in the literature.

In order to further enhance the discussion on how good the Recommender Systems are and their evaluation, we present a review on work in the literature that performed evaluations on state-of-art recommenders regarding the aforementioned concepts. This brief review does not focus on comparisons, since different work decide on evaluating their recommender system regarding different concepts. Furthermore, even same concepts are evaluated by different metrics. Therefore, we center our attention on summarizing the main achievements in performance with regard to the six concepts of evaluation. For the sake of summarization, we included Table 4 in the appendix summarizing the performances of the State-of-Art recommenders reviewed in this subsection.

With regard to utility, most articles evaluate their recommender systems with offline accuracy metrics. As an example, some collaborative filtering algorithms were initially evaluated through MAE and RMSE metrics. Wen [37] reviewed Item-based KNN, Item-based EM and Sparse SVD for the recommender system problem. The author evaluated the RMSE metric (Eq. 2) on a Netflix database¹ and obtained results as high as 0.95, 0.91 and 0.8974 for the test set for the Item-KNN, Item-EM and SVD methods, respectively.

Novelty has been evaluated in many ways in the literature, however using different metrics and definitions. For instance, Vargas and Castells [36] evaluated state-of-art methods using level 3 novelty metric stated in Eq. 10. The authors used Matrix Factorization, IA-select and MMR methods in their evaluation of novelty and considered a relevance discount, which resulted values of novelty of 0.058, 0.0639, 0.0620, respectively, in the Movie Lens² dataset. Not considering the relevance discount, the authors achieved results on novelty of 0.76, 0.8080 and 0.7605, respectively, also for the Movie Lens dataset. The authors also evaluate the same methods on Last.fm³ dataset, where they obtained 0.2671, 0.3462 and 0.2439 of novelty ratio, respectively, considering the relevance discount; and the authors obtained 0.8949, 0.8912 and 0.9133, respectively, disregarding the relevance discount. It can be seen that more novelty was achieved in the Last.fm dataset. However, it is noteworthy that, in this case, high novelty may mean low values of accuracy or other utility measurements, because novel items not necessary are highly useful to the users.

Vargas and Castells [36] also evaluated the state-of-art recommenders with regard to their diversity metric. In this case, the authors used the diversity metric described in Eq. 13. For Movie Lens dataset, the recommenders Matrix Factorization, IA-select and MMR achieved 0.0471, 0.0537 and 0.0510, respectively, considering a relevance discount. Ignoring the relevance, the resulting diversity ratio was elevated to 0.7164, 0.8289, 0.7191. Again, similar to the novelty statement above, high diversity in this case may negatively impact the utility. In addition, De Pessemier et al. [8] evaluated Group Recommendation on Movie Lens dataset and diversity was evaluated for User based and Item Based Collaborative Filtering algorithms and an SVD method, although the authors do not further specify which algorithm was used. The metric used for diversity was the intra-list similarity, as Eq. 11 states. When considering the group recommendation size equal to 1, the diversity similarity was 0.7, 0.81 and 0.64 for User-based CF, Item-based CF and SVD methods.

Regarding the concept of unexpectedness, it is not usually assessed by work in the area in the literature. Adamopolous et al. [1] is one of the few work in the literature to evaluate the unexpectedness and compare their own method with state-of-art baselines. The authors, evaluated Item and User KNN and Matrix Factorization methods. However, in this work, the authors evaluate many combinations of parameters and they do not mention which baselines are used in comparison with their recommending methods, instead they calculated the average values of their experimental settings. The authors evaluate unexpectedness through the metric shown in Eq. 15. Two datasets were analyzed in this case: Movie Lens and Book Crossing⁴. For Movie Lens dataset, the unexpectedness ratio for their baseline metrics are 0.71 and 0.75 for recommendation lists size 10 and 100 respectively, using related movies as expected recommendations. For the Book Crossing dataset, the values of unexpectedness are more modest, where it is 0.3 and 0.38 for size 10 and 100, respectively, using related books as expected recommendations.

Although there are work which study the serendipity concept and propose new methods and metrics, few of them effectively use state-of-art baseline as comparison. Lu et al. [22] evaluated serendipity on two datasets: Netflix and Yahoo! Music⁵, using popular methods: SVD, SVD++ and SVDNbr. The metric used for assessing serendipity was analogous to Eq. 21. The authors used different loss functions and optimizing methods, however their best serendipity results for each method for the Netflix dataset was 0.2534, 0.3036 and 0.2978 for SVD, SVD++ and SVDNbr, respectively. For the Yahoo! Music dataset, the best results were: 0.1995, 0.2771 and 0.3834 for the three methods, respectively.

Also, there has been attempts to evaluate a temporally evolving system with regard to the aforementioned concepts. Shi et al. [33] addressed the issue of the performances of recommender systems in a temporally dynamic system. The authors described the datasets (Movie Lens and Netflix) as bipartite networks of users and items divided into a series of subsets which considers the recommendations and the time series. Authors used a recommendation based evolution method to simulate the temporal dynamic of three common collaborative filtering strategies. They evaluate their results at each time step with utility (RMSE Eq. 2), intra-list similarity (similar to Eq. 13), system-level novelty (metric similar to Eq. 9). For Movie Lens dataset for instance, the authors showed to be able to maintain low levels of similarity, around 0.025 with Item-based CF in the temporal evolving process.

Lastly, as mentioned earlier, the coverage concept is also under-explored, both in metrics definitions and work concerning this concept. Nevertheless, one of the few authors that evaluate coverage is Adamopolous et al. [1]. In the authors’ work, catalog coverage is evaluated using metric similar to Eq. 24. For Movie Lens dataset, the authors obtained 0.05 and 0.15 rate of items ever recommended to the user for the size of the prediction list of 10 and 100, respectively. Moreover, for the Book Crossing dataset, the coverage ratio is about 0.2 and 0.5 for 10 and 100 sizes of the prediction list.

It is important to notice that the reviewed work consider different datasets, methods, evaluation metrics, implementations and evaluation methodologies when studying the impact of the user satisfaction concepts in recommendation. We emphasize the need of a novel work which can effectively provide user satisfaction evaluations for the state-of-art recommenders under the same evaluation scenario and datasets considering the mentioned concepts. Moreover, in order to complement such research, such work could also analyse the impacts of attempting to evaluate and optimize all of those concepts simultaneously and attempt to combine recommendations with different objectives, similar to what was performed by Zhang et al. [38] and Ribeiro et al. [28] in a limited level.

The metrics presented before have the main objective to evaluate how good the recommender system is in making predictions for the user under different perspectives, evaluating whether the predictions are novel, diverse, surprising and useful. Although there are many metrics and definitions, they can be classified regarding the property of user dependency. Some metrics are sensitive to users’ information, while others do not depend on the users and simply evaluate the system or recommendation list. Therefore, the metrics can be classified in user-dependent and user-independent.

According to the survey made and the previous discussion, the concepts can be organized into categories that represent their relationship with regard to each other: base and compound concepts. Regardless of the categorization on user dependence, some concepts are related to each other, considering that their definitions take others into consideration. Figure 2 shows the current relationships among the concepts. The arrow represents a relationship between the concepts as it has been mentioned in the literature.

There are five base concepts: utility, novelty, diversity, unexpectedness and coverage. These concepts are grouped together as their definitions on the literature do not depend on any other concept. Moreover, base concepts are used in the definitions of compound concepts. Currently, only utility, novelty and unexpectedness are related with compound concepts. Diversity and coverage are currently isolated in this relationship scheme. So far, no definitions and metrics for diversity and coverage mention other concepts.

Individually for unexpectedness, even though some definitions mention novelty [1], the authors’ usually state that surprising recommendations can be either novel or not. Therefore, since there is no limitation for unexpectedness regarding novelty, as there is for serendipity, it can be currently stated as a base concept.

The only current compound concept is serendipity. The definition of serendipity consists on three components: utility, novelty and unexpectedness, so that serendipitous recommendations can be called a delightful surprise [1]. Herlocker et al. [14] and Ge et al. [12] say that serendipitous recommendations are novel by default, as the user must not have knowledge about them. Moreover, they need to be unexpected as it is the surprise part of the definition [1]. Lastly, since it involves a positive response, there is a direct relation between serendipity and how useful the prediction is for the user [1, 12]. Therefore, in this classification, serendipity is a compound concept.

The currently relationships of the evaluation concepts, as stated by previous work, may be oversimplified. There might exist relationships among the concepts’ definitions and underlying correlations among them. We cover these potential relationships in the next section.

As it was pointed out earlier, there is still much open issues with metrics in recommendation. Metrics for utility concept are an example. One of the reasons researchers started to investigate other concepts rather than predictive accuracy in RS is that it is not known whether accuracy really represents user satisfaction [14]. Currently, a set of concepts is used in evaluating whether the users enjoy the predicted items. Therefore, future researches could further study utility metrics in order to check if evaluating utility is equal to predictive accuracy.

Moreover, we emphasize the classification of novelty in three levels. The existing metrics only explore level 2 and 3 of novelty. Evaluation about the first level of novelty is difficult and so far there are no metrics. Then, more research is required about level 1 of novelty. Future metrics for life level of novelty need to consider information out of the system in order to model the users’ lifetime consumption. It is clearly a challenging task.

Diversity seems to be related to Information Retrieval diversity, by analyzing the metrics, such as the one proposed by Vargas and Castells [36]. Information retrieval techniques may have a lot to contribute to diversity metrics in RSs. Moreover, diversity can be correlated to unexpectedness and serendipity. Attempting to recommend surprising items may enhance diversity in recommendation. Increasing unexpectedness implies recommending items not usually recommended in expected recommendations. Therefore, the variety of items may be enhanced by issuing unexpected recommendation lists. Recommending serendipitous items might result in the same effect, since unexpectedness is one its components.

Serendipity and Unexpectedness seem to be the new directions of evaluations in recommender system. However, as surveyed, definitions and metrics remain overlapping and somewhat fuzzy in the literature. Since there are many metrics to evaluate the two concepts, new researches in the area do not know which metric to use in evaluation. Therefore, more research is necessary to clearly separate the definitions and properly establish metrics for unexpectedness and serendipity. Further, the two definitions for both concepts have limitations. Unexpectedness can reduce tediousness and obviousness and it is not limited to novel recommendations. Serendipity means surprising and useful, but the definition only allows novel items [17]. This is a limiting factor for applying the concept in real-world problems. A scenario requiring evaluation on surprising, useful and not necessarily novel could exist. For instance, forgotten items or items consumed far in the remote past could become serendipitous in some moment in time, but these items are not necessarily novel, because the user knows them, according to the level 2 of novelty.

In this sense, future researchers could discuss relaxing serendipity’s definition, so that it is not limited to novel items. Otherwise, they could enhance unexpectedness’ definition to include utility as well. Alternatively, a new concept could be proposed, which could simply be composed of surprise and usefulness, but no restriction with regard to novelty.

Coverage is an under-explored concept. Few work attempted to evaluate item space coverage in practice, because the metric does not evaluate if the user enjoy the recommendation or not, instead it evaluates the RS. Future works could research the applicability of coverage in real scenarios. Furthermore, item space coverage could be related to other concepts. For instance, coverage could be correlated to recall, as recall is related to items not recommended. Moreover, uncovered items may enhance unexpectedness to the user. Items not effectively recommended in the system could be unexpected, since the user do not reach them. Lastly, there could be a correlation between coverage and diversity. Increasing the system coverage would increase the rate of items that are recommended; therefore, items not commonly recommended would be issued in recommendation lists, potentially decreasing the intra-list similarity among the items. Coverage, as well as its potential relationships with other concepts, is an open under-explored issue, which could be inspected in future research.

According to the ongoing discussion, Fig. 3 shows the discussed changes in the relationships among the concepts. Dashed arrows present the possible correlations among the concepts, which has not been aware by previous work, as previously described in this session. The figure has dashed arrows for representing the possible correlation between diversity and coverage, and between diversity and serendipity. Unexpectedness is transferred to the compound concepts group in this scheme, because it would have relationship with coverage and diversity. Moreover, the hypothetical new concept combining the desired features from serendipity and unexpectedness is also represented in Fig. 3.

In order to better research concepts and metrics for evaluation in recommendation, it is important to consider what are the factors that compose users’ satisfaction. Ultimately, RSs have the objective of satisfying the users’ interest of consumption. In offline and mainly in online experiments, it is important to study what are the interests of the user when using a system, so that concepts and metrics can be designed to evaluate recommenders. Recommenders can also be designed to fulfill the users’ interest.

Users have an intrinsic need for useful recommendations, but sometimes, users also desire to have new experiences and get surprised. Figure 4 shows user satisfaction requirements sorted from the needs to the desires of the user. Needs represent what the recommendations must have, while the desires represent what the recommendations could also have. Utility is the main need the users have, since they want recommender systems to suggest useful items according to their tastes. Even though they are important, serendipity and unexpectedness represent desires for surprise in recommendation. Diversity places in the border of the two constraints, because users desire and need some variety in recommendation. Moreover, novel items are not useful to the users, instead novelty is a concept that users would desire to be suggested from time to time, in order to extend their tastes. Lastly, coverage is not evaluated this image, since it is nor a need neither a desire for the user.

In short, user satisfaction should be considered when modelling new recommenders. Future researches could evaluate prediction lists and their algorithms using all the concepts presented. Offline and online experiments are important, even though the experiments with users can be costly to perform. Moreover, future works could consider designing algorithms that simultaneously balance the surveyed concepts in a single recommendation list. Zhang et al. [38] attempted to balance novelty, diversity and serendipity, while Ribeiro et al. [28] attempted improving accuracy, novelty and diversity, accordingly. Nevertheless, novel attempts in studying and balancing all of the mentioned concepts are necessary for increasing user satisfaction considering the needs and desires of the user.

Most of the work surveyed in this paper refer to offline experiments using the concepts for evaluations of recommendation. Experiments usually elaborate train and tests sets on a dataset and perform evaluations by comparing the recommendations with what was consumed in the test set. Since this approach does not always correctly model the consumption behaviour of the users, as an alternative, evaluation can also be performed by online experiments. In an online experiment, recommendations are presented to a real user and the evaluation is made upon what items the user consumed or liked. In this sense, recommenders can be effectively evaluated regarding the consumption and preferences of the users. However, training new recommenders or researching novel recommending techniques are costlier using online evaluation than offline experiments, because experiments involve real users.

More recently, there has been efforts to reduce experiments with users, incorporating models that simulate users’ participation in research experiments. For instance, using a collection of user browsing data, Zhao et al. [39] developed a model for incorporating eye tracking information with RS without requiring eye tracking technology. The authors developed a new click model for a ranking algorithm, and used as a signal of user satisfaction in Information Retrieval evaluation.

Moreover, there has also been a research in estimating the performance of costly controlled trials. Rosenfeld et al. [30] proposed a framework for predicting counterfactuals using a large historical data and small set of randomized trials. For instance, the authors predicted the Click-Through-Rate of a Search Engine by using a dataset of A/B tests.

The eye tracking and counterfactuals predictions for evaluation has been investigated in few Information Retrieval scenarios. Both researches were not sufficiently observed in Recommender System area. In both cases, online evaluation could be estimated, reducing costs of performing experiments with real users. In this sense, such strategies could be helpful in new algorithm designs focusing on evaluation of user satisfaction and user decision making process.

There are few work in the literature comparing the offline metrics with online evaluation. In Maksai et al. [23], the authors use offline metrics to predict online performance, using concepts such as coverage, diversity and serendipity. Besides this work, currently we see little correlation between the offline metrics and the online evaluation. CTR is a major online evaluation metric and sometimes, the utility based metrics are also used in online evaluation. Besides utility, it is unknown whether users’ notions of novelty, unexpectedness and serendipity are the same to the evaluations performed by the current offline metrics. It is necessary a study to establish the level of consistency that the reviewed offline metrics have with the online scenarios. Such study could perform online experiments with users in order to retrieve their feedback with regard to their perceived novelty, unexpectedness and serendipity towards personalized predictions. In this sense, the users’ perception of these concepts can be correlated to the values obtained by the current metrics. Thus, the most correlated metrics can be used for both offline and online experiments, obtaining the closest values of novelty, unexpectedness and serendipity. Moreover, the users’ feedback can also be used to design metrics for the life-level of novelty, for which there is currently no metric.