A qualitative analysis of knowledge graphs in recommendation scenarios through semantics-aware autoencoders

Nowadays, we are overwhelmed by a large amount of available information we can benefit from. In particular, e-commerce sites and entertainment Web services usually offer thousands of different items among which users are invited to find the ones they need or desire the most. In this direction, recommender systems (RSs) have proven to be a silver bullet in suggesting appropriate items to users according to their past choices and behaviors. A typical RS exploits item ratings available in a system, either implicitly or explicitly, to predict a list of unseen items which are of potential interest to the user (Ricci et al., 2011). Over the years, several strategies have been developed to build efficient recommendation algorithms. They are generally divided into three main categories: collaborative filtering (CF) techniques, content-based (CB) methods and hybrid approaches (Burke, 2002). While CF techniques exclusively rely on the feedbacks (rating, click, watch, listen) from the users on specific items without considering their description (either structured or unstructured), on the other side CB ones exploit the data associated to an item to compute relevant recommendations to a user. One of the main issues to tackle in adopting a CB approach is then getting the right amount of meaningful information/data about items, which in turn results necessary to model content-aware items and users descriptions. In fact, by gathering data about items rated by users, one can infer attributes that can be used to model users’ profile and preferences.

In the last years, the technological wave related to deep learning techniques and approaches hit also the field of RSs. A variety of new approaches based on different configurations of Neural Networks (NNs) have been proposed to compute personalized lists of items to be suggested to end users (Covington et al., 2016; Wang & Wang, 2014; Elkahky et al., 2015). Among them, autoencoders have been proposed as an interesting tool to mimic the user behavior in producing ratings and by exploiting and modeling user preferences on latent item attributes (Wu et al., 2016). Autoencoders are a particular configuration of artificial NNs which turned out to be very effective especially for dimensionality reduction and feature selection tasks (Vincent et al., 2008). However, autoencoders proved useful also in a different configuration. Indeed, in their mirrored structure, neurons of the hidden layers can be interpreted as a projection of the input layer in a different space. In Bellini et al. (2019), the authors presented SEMAUTO¹, a Semantics-Aware autoencoder (SA-autoencoder from now on) which leverages the common graph-based structure to encode the semantics, and the structure of a knowledge graph to enhance the representational power of the underlying NN. One of the main advantages of such a hybrid structure it that of giving an explicit semantics/label to the latent dimensions of the new space (Bellini et al., 2019). In this setting, SA-autoencoders may expand the hidden layer to accommodate all features in the KG relevant for a user.

Among the various and diverse KGs freely available on the Web, for sure DBpedia (Auer et al., 2007) and Wikidata (Vrandecic & Krötzsch, 2014) play a key role due to their encyclopedic nature which makes them the ideal candidates to provide structured descriptions on items in a recommender system (Sacenti et al., 2022; Liu et al., 2023). Although there is a partial overlapping among the information sources to build DBpedia and Wikidata, the data they encode is different under various aspects, such as the amount of data and the way it is organized (Ringler & Paulheim, 2017).

In this paper we test the addition of a KG to a RS through extensive experiments that variate the following dimensions:For all possible RSs built over any combination of the above characteristics, we examine its results in terms of accuracy and diversity, evidencing and discussing how the structure and coverage of information encoded on a KG may affect the accuracy and novelty of the recommendation.

The structure of the paper is as follows: in the next section, we recap how to build SA-autoencoders and their recommendation model. Then, in Sections 2 and 3 we introduce the experimental setting and the different configurations adopted in our investigation, while the results are discussed in Section 4. We then compare our results with related works in Section 5. Conclusions and future work close the paper.

Autoencoders are unsupervised NNs that learn a function capable of reconstructing the network’s input at the output layer. They are built on top of two main components which are the encoder and the decoder. The former is usually responsible for compressing the input data into a lower dimensional representation, while the latter does the opposite job reconstructing the original input data starting from a lower dimensional representation (see Fig. 1). Like every NN configuration, autoencoders are structured in layers which contain neurons. Every neuron in layer i is connected through an edge to all the neurons of the following layer \(i+1\). In other words, in its standard configuration we have a fully connected network. In a recommendation scenario, we may use all the items in a catalog as representative of both the input and the output layer. We may then train the NN by using user ratings as inputs to obtain similar values produced by the output layer.

Autoencoders have been studied also in architectures that augment the input-output dimension. Figure 2 depicts an Overcomplete Autoencoder, a deep learning technique with a hidden layer wider than the input layer, offering greater representation capacity than traditional autoencoders (Ranzato et al., 2006; Rifai et al., 2011). This configuration excels both in handling noisy or missing data and in generating new data (Vincent et al., 2008). However, increased computational complexity and overfitting risks may arise. Mitigation strategies include using specialized algorithms and hardware, regularization, and dropout techniques (Ngiam et al., 2011; Lewicki & Sejnowski, 2000).

Denoising Autoencoders (DAE) also feature a wider middle layer and learn compressed, simplified representations of input data (Vincent et al., 2010; Alain & Bengio, 2014). DAEs can extract meaningful features from user interaction datasets or knowledge graphs, improving recommendation algorithm performance and recommendation quality (Liang et al., 2018).

Starting from the observation that both NNs and KGs are directed graphs, in Bellini et al. (2017) the authors propose to use the topology of the latter to model the former, in an original architecture inspired by overcomplete-autoencoders, whose aim is not dimensionality reduction. The main idea is to keep input- and output-layer nodes as representative of the items in the catalog and substitute anonymous nodes in the hidden layers by labeled resources from a knowledge graph thus inheriting their mutual semantic connections (see Fig. 2). Differently from the generic definition of autoencoder, we see here that the resulting NN is no longer fully connected since the input layer’s nodes are linked to neurons in the hidden layer if and only if a corresponding connection exists in the original knowledge graph. In such a semantics-aware autoencoder, we somehow project the items in a space whose dimensions represent all the entities (features) they are connected to.

As for a generic autoencoder, also a semantics-aware autoencoder is trained with user ratings and it learns how to reconstruct them on the output layer but differently from the original autoencoder, since such a network is not fully connected, user ratings propagate only through those nodes that represent features connected in the KG to items rated by the user. According to NN models, a generic neuron outputs a value that is a non-linear function of the weights summation over incoming edges. Then, in a recommendation scenario it turns out that positively rated items tend to have connected neurons with higher output values than those neurons connected with negatively rated items. As proposed in Bellini et al. (2019), one single autoencoder is trained per each user for 1000 epochs by using the well-known Sigmoid activation function for the hidden layer. The number of neurons in the hidden layer depends on the number of entities (features) encoded in the KG for the user’s rated items. We use the autoencoder to describe a single user’s ratings in a feature space that is tightly coupled with the KG, so this method, even with few rates, is able to weigh items features according to the user’s interest to them. Nevertheless, since the autoencoder is fed with ratings of only one user, the overall approach can be easily parallelized by training each user/autoencoder on an autonomous thread.

We recap the training process of the semantics-aware autoencoder in Fig. 3. At start, the model is initialized with random weights (Fig. 3a). For each epoch it is fed with user ratings and the back-propagation adjusts weights in order to reconstruct input’s user ratings at the output layer (Fig. 3b). The model converges and encodes the relevance for each feature in the hidden layer Fig. (3c).

As a direct consequence of relying on a KG, it stands to reason that its structure, as well as the information it encodes, might affect how user profiles are computed. Thus, it is crucial to investigate how the KGs structure impacts the recommendation accuracy. We may believe that the better the data are engineered and curated, the more accurate a recommendation is. The point is, how to evaluate and measure the recommendation?

Given the recommendation model previously described, we performed an experimental evaluation to verify the influence of a knowledge source in the final recommendation task. In this paper, we chose the two main encyclopedic knowledge graphs freely available on the Web: DBpedia and Wikidata due to the richness of information they encode in different knowledge domains. DBpedia and Wikidata differ from each other not only by their structure but also Ringler and Paulheim (2017) by distinct fields of knowledge are covered in different ways.

We first describe the structure of the datasets used in the experiments, then we move on to the evaluation protocol for the recommendation, and finally we discuss the results.

For the sake of ensuring the reproducibility of our models we provide the link to the public repository from which to retrieve the code and datasets (https://​split.​to/​sa-auto).

For all the recommendation models described, and for both KG, we performed experiments on three datasets: MovieLens 1M, Last.fm and LibraryThing, related to movies, music, and books domains, respectively.

In Table 3, we present the most favorable outcomes we have obtained across three datasets by implementing the methods discussed earlier, with a specific emphasis on the level of sparsity. Regarding the SA-Autoencoder approach, we conducted tests with various numbers of neighbors (k), and we have included only the optimal results in the table. Notably, we have highlighted the overall best-performing approach in bold, while we have underscored the most effective configuration for the SA-autoencoder. We initiate our discussion by evaluating the results in terms of accuracy before delving into the aspect of diversity. Examining the table reveals that the semantics-aware autoencoder surpasses all baseline methods in the Last.fm dataset. Conversely, in the case of the LibraryThing and MovieLens 1M datasets, its performance closely mirrors that of the fully-connected autoencoder. To elucidate this outcome, we can posit that the semantics-aware autoencoder’s performance is contingent upon the quantity of features retrieved from a knowledge graph (KG). A greater number of features corresponds to a larger number of neurons in the hidden layers. According to Hornik’s Universal Approximation Theorem, a neural network with more neurons possesses a superior capacity to approximate any given function. Consequently, as indicated in Table 4, we have computed the ratio of features associated with items, denoted as \(\frac{average \#features}{average \#items}\). Our findings reveal that, considering each pair of KGs (DBPedia-WikiData) for a given setting (S-B-F...), the KG with the higher ratio yields the better results on performance It is imperative to emphasize that this measure also serves as a gauge of connectivity in SA-autoencoders. Specifically, lower values of \(\frac{average \#features}{average \#items}\) indicate a sparse connectivity with fewer connections, while higher values signify a denser connectivity (Pujara et al., 2017).

Nevertheless, when examining the sparsity levels presented in the final column of Table 1, it becomes evident that, despite the LibraryThing and Last.fm datasets having nearly identical sparsity values, they exhibit substantial disparities in their average features per item ratios. Specifically, the SA-Autoencoder demonstrates superior performance on the Last.fm dataset, characterized by a higher ratio, while its performance is notably inferior on the LibraryThing dataset, where the ratio is lower. Consequently, it is reasonable to infer that this method exhibits a relatively lower sensitivity to dataset sparsity in comparison to the quality of data curation within the Knowledge Graph (KG). Consequently, the comparison between the outcomes of the SA-Autoencoder and the Autoencoder can serve as a valuable means of evaluating data quality within KGs, particularly in the context of recommendation tasks.

On the other hand, we can observe that factual information (F) brings more accurate results for a recommendation task than ontological/categorical knowledge when the datasets are very sparse (as for MovieLens 1M and LibraryThing). This is a quite interesting result. A possible explanation is that categorical information introduces fewer connections among item descriptions than factual information. Hence, factual statements result more useful in making denser connections among items (and then users), which are eventually exploited by the latent collaborative part of the semantics-aware autoencoder.

If we just focus on the absolute numbers, one may argue that SA-Autoencoder is not competitive as it is slightly beaten in terms of accuracy by state-of-the-art algorithms such as BPRSLIM and WRMF (although it is the second best performing approach). Nevertheless, we point out that, differently from the other approaches based on matrix factorization (or any deep learning technique) with SA-Autoencoder we compute a meaningful and explicit user profile which contains user preferences on single features. This may result extremely useful in case we want to automatically generate a content-based explanation for the ranking computed with the recommendation list as also shown in Bellini et al. (2018). Then, although we rely on a deep learning approach, we can go beyond the pure black-box method and provide a human-understandable explanation for a recommendation list.

As for diversity, we can observe that when using Wikidata, values for Gini index are in general lower than the DBpedia case. This means that using Wikidata we are able to better diversify the items in a catalog. This result somehow reinforces the one obtained in Nguyen et al. (2015). Lack of diversity in recommendation strongly depends on the so-called “popularity bias”: popular items tend to be recommended more than those in the long tail. This observation leads us to a possible interpretation on diversity results we obtain in our experiments if we consider the popularity of entities in a KG as the number of connections they have. In DBpedia we have that popular resources (e.g. movies) are more connected to other nodes than unpopular ones. This is not the case with Wikidata where there is a less biased distribution of connections among resources in the graph. Hence, when the adopted KG suffers from a popularity bias in terms of connections among resources, this is inherited by the recommendation dataset, thus affecting the final recommended list of results.

Very few works exist about qualitative studies of Linked Open Data (LOD) knowledge graphs. In Ringler and Paulheim (2017) present an analysis of main KGs such as DBpedia, Wikidata, YAGO, underlining their differences in coverage, identifying overlapping and complementary parts of KGs. They assert that KGs are not easily interchangeable and each of them has its strengths and weaknesses for a domain-related task. Thus, using a specific KG that is suitable for the task to accomplish leads to better performances of the overall system. Authors in this work made a category-specific analysis, asserting that even if DBpedia and YAGO come from the same source (Wikipedia) and have a quite similar number of instances, there are notable differences in coverage. YAGO has five times the number of events of DBpedia, while DBpedia has four times as many settlements (i.e., cities and town) as YAGO; but Wikidata contains twice as many persons as DBpedia and YAGO. They conclude their investigation by providing a coverage summarization for some popular classes. A comparative survey of some popular KGs is done in Färber et al. (2018) in which authors propose a method to find the most suitable KG for a given task setting. To achieve this result, authors identify a set of characteristics they found relevant to describe a KG and then compare different KGs accordingly. Furthermore, in Färber et al. (2018) they provide a more detailed analysis of quantitative information stored in KGs by using several statistics such as the number of triples and classes, distribution of classes and corresponding instances, domain and classes coverage. Finally, to select the KG that best fits task requirements, a novel method that takes into account KG’s quantitative assessment is proposed.

In the last few years, thanks to increasing computational resources, new techniques based on deep learning have been successfully adopted in the recommendation scenario (Cheng et al., 2016). These techniques led to the development of different neural network models, each yielding interesting results (Singhal et al., 2017). In particular, some of these models turned out to be more effective for a specific recommendation task than others; e.g. autoencoders have proved their strength in collaborative filtering, outperforming state-of-the-art approaches (Sedhain et al., 2015), while Recurrent Neural Networks seem to be more suitable in session-based recommendation (Hidasi et al., 2016). Among the several autoencoders extensions, denoising autoencoders have been efficiently used to address the recommendation problem by improving users’ profile learning (Wu et al., 2016) or getting a smaller and non-linear representation of the User-Item rating matrix (Strub et al., 2016). Furthermore, Dong et al. (2017) shows how to build a hybrid RS by integrating side information in CF deep learning techniques, alleviating the sparsity problem and improving all system performances (Dacrema et al., 2019).

LOD are increasingly adopted in recommender systems because they provide more complex structured data that leverages relationships among entities in the graph. Moreover, they encode somehow semantics behind the data (Di Noia et al., 2012; de Gemmis et al., 2015). Recent works leveraged the data encoded in KGs to represent items thus achieving interesting results in recommendation scenarios (Oramas et al., 2017; Di Noia et al., 2016, 2012). Other approaches, as the one proposed by Ostuni et al. (2014) uses kernel graphs to compute item similarities by matching their local neighborhood graphs. LODs have been also exploited for measuring semantic distances between resources in order to provide top-N recommendations (Piao & Breslin, 2016). In Bellini et al. (2017) presented a novel method that combines both deep learning techniques and KGs, in which a semantics-aware neural network that explicitly computes user profiles for recommendation tasks is modeled. In particular, they focus on cold-start scenarios using DBpedia as a source of information for both user- and item modeling. In Bellini et al. (2018), authors used the aforementioned method to perform experiments in recommendation scenarios by using DBpedia KG on three different datasets and they compared their approach with state-of-the-art algorithms. Furthermore, they investigated the effectiveness of their method that leverages on a KG to provide an explanation in recommendation scenarios (Bellini et al., 2018). Another approach that uses KGs to represent the relationship between users and items is investigated in Ristoski et al. (2019), in which the authors leveraged on language models to extract features from node sequences in RDF graphs. Recently, there has also been much interest in the topic of KG completion, whose aim is to infer hidden relationships between entities in a KG. Taking advantage of the progress made in this area, the authors of KTUP (Cao et al., 2019) have proposed to exploit KG completion methods with respect to users’ preferences towards items in a catalogue in order to jointly train a model that exploits both the interactions between users and items and the characteristics of the items in a KG.

In this paper, we showed how it is possible to combine the computational predictive power of NNs (in the form of autoencoders) with the representational power of KGs such as DBpedia and Wikidata. We found that the choice of the KG affects the results of a KG-aware approach to recommendation, and that some combinations of KG, recommendation model, and task domain, yield better recommendations than others. In particular, we evaluated an SA-autoencoder on different configurations for DBpedia and Wikidata, and we tested and compared its results in terms of accuracy and diversity with other recommendation models used as baselines, in three different recommendation domains. We showed that the selection of the right information from the right KG may heavily affect recommendations both in terms of accuracy and in terms of diversity of results.

As a prominent example, Wikidata seems to be a better choice than DBpedia if we look for recommendation lists where the popularity bias is mitigated: our results show that Wikidata allows the SA-autoencoder to tackle the popularity bias and to better diversify the items in a catalog, by recommending also less popular items.

As for future work, we are developing a new version of the SA-autoencoder – one that better exploits collaborative information by building a global model encoding preferences from all users at the same time, as done by state-of-the-art CF approaches – with the aim of improving the experiments presented here.