Personalized multi-faceted trust modeling to determine trust links in social media and its potential for misinformation management

Multiagent trust modeling algorithms seek to predict the trustworthiness of another agent based upon first hand experience and reports provided by other agents in the environment, sometimes referred to as advisors [48].

The beta reputation system (BRS) proposed by Jøsang and Ismail [23] estimates reputation of selling agents using a probabilistic model. The beta distributions are a family of statistical distribution functions that are characterized by two parameters \(\alpha \) and \(\beta \). The beta probability density function is defined as follows:where \(\Gamma \) is the gamma function, \(p \in [0, 1]\) is a probability variable, and \(\alpha , \beta > 0\). This function shows the relative likelihood of the values for the parameter p, given the fixed parameters \(\alpha \) and \(\beta \).

Ratings from peers (used to estimate the reputation of a seller) are binary in this model (1 or 0, to represent that the advisor considers the seller to be satisfactory or dissatisfactory in a transaction). Individual ratings received are combined by simply accumulating the number of ratings supporting the conclusion that the seller has good reputation and the number of ratings supporting the conclusion that the seller has bad reputation.

The prior distribution of the parameter p is assumed to be the uniform beta probability density function with \(\alpha = 1\) and \(\beta = 1\). The posteriori distribution of p is the beta probability density function after observing \(\alpha - 1\) ratings of 1 and \(\beta - 1\) ratings of 0.

The reputation of the seller s can then be represented by the probability expectation value of the beta distribution, which is the most likely frequency value, used to predict whether the seller will act honestly in the future. The formalization of this is given as follows:Zhang and Cohen [48] suggest a personalized trust model (PTM) to determine whom to listen to among a network of buyers and sellers in thee-marketplace domain. In particular, they address whether a buyer, b, should purchase a product from a seller, s, based on a combination of global advice from other buyers (i.e., advisors, a), and b’s own local past experiences with s.

The PTM global metric is further broken down to combine private and public trust estimates of advisors. The intuition is that b may have radically different expectations or preferences regarding s’s product than a, and so b should have some notion of how much to trust a. To the extent that b relies on past common experiences to evaluate a’s trustworthiness, b uses a private trust metric to incorporate a’s recommendation. To the extent that b relies on a’s similarity to the global rating of various sellers (i.e., how fair are a’s ratings), b uses a public trust metric to incorporate a’s recommendation.

In particular, b’s private reputation according to a, \(R_{pri}(a,b)\), is modeled by the expectation of a beta distribution where \(\alpha \) is the number of times a and b have agreed in the past about the reputation of other agents, and \(\beta \) corresponds to how many times they have disagreed. The public reputation of b, \(R_{pub}(b)\), is again modeled by the expectation of a beta distribution, where \(\alpha \) corresponds to the number of times b’s advice has agreed with majority opinion, and \(\beta \) the number of times it has not. The final reputation of b fora is then a linear combination of the private and public reputation of b, weighted by a factor w which reflects how much comparable experience a has had with b (i.e., the number of agents commonly rated agents).

Multi-faceted trust modeling (MFTM) is a flexible and data-driven approach to trust modeling. Inspired by work in the social sciences which have outlined the numerous variables which influence the formation of trust relationships [30], MFTM incorporates arbitrarily many indicators of trustworthiness into a single (optionally context-dependent) trustworthiness score. Operationalizing this core idea for trust and social tie prediction has been proposed by multiple researchers (e.g., [9, 13, 22, 26, 29]). As is evident in these works, there is little agreement over whether this technique should be called multi-dimensional, multi-faceted or composite trust modeling, and this confusion has likely led to some difficulty in coordinating efforts in this research direction. We use the term “multi-faceted,” in keeping with the most recent works.

The defining feature of an MFTM is a customizable vector of trust indicators, where each indicator is a real number based on two agents:A “trust indicator” can be thought of as a piece of evidence for or against trusting an agent under a particular context. For example, \(\psi _1(a_1,a_2) = friendCount(a_2)\) may be relevant to assessing the reputation of \(a_2\) in a domain where only popular and reputable agents can accrue large numbers of friends. The indicators \(\psi _i\) must be computable given A and E (the set of agents and their attributes and the history of events). One important feature of MFTM is its flexibility to tune its parameters to different domains of use. The customizability of MFTM is highly attractive for application to social networks, as it is rare to find explicit statements of trust encoded into the feature set of online environments². Instead, an arbitrary number of “imperfect” indicators of trustworthiness, such as popularity, friendship, reputation, interaction history, preference similarity, and institutional credibility can be considered as each contributing to a final tally of trustworthiness. Clearly, the underlying assumption of this model is that the existence of trustworthiness between two agents can be predicted based on a comparison of the attributes and behaviors of those agents.

The consideration of multiple indicators of trust can be viewed as an emulation of the way in which humans consider multiple sources of evidence when deciding to trust or not [30]. For example, consider the problem of choosing an auto mechanic shortly after having moved to a new town. In this case, one has no interaction history with any nearby mechanics and must weigh available evidence in order to choose which mechanic to trust. In a simple case, one might only consider two pieces of evidence toward or against a mechanic: has any colleague recommended them (\(\psi _1\)), and have their prices been posted clearly online (\(\psi _2\)). In this case both indicators are binary, and it seems likely that the mechanic \(a_j\) who satisfies both indicators, \(\Psi (\overset{\rightharpoonup }{a_i}, \overset{\leftharpoonup }{a_j}) = \langle 1, 1 \rangle \), will be a good candidate to trust. (Note that we use \(\overset{\rightharpoonup }{}\) for the trustor and \(\overset{\leftharpoonup }{}\) for the trustee at times in our discussion below, for additional clarity).

In order to predict trustworthiness, the relevance of each indicator can be learned using an off-the-shelf machine learning technique given A and E to train with. To do this, a trust link is chosen as a target of prediction : y (e.g., explicit statements of trust/friendship, high degrees of preference alignment). Then, given the set of existing implicit/explicit trust links, a machine learning model fits a classifier \(\hat{f}\) to the function that determines how trust indicators are related to trust links \(f: \Psi (a_1,a_2) \rightarrow y\). For example, in the case where logistic regression is used, y will be binary and we have:where \(\theta \) is the vector of weights learned through the logistic regression process and \(T_c\) is trustworthiness under context c. (We believe that whether someone is trusted may truly vary according to the context; for the remainder of the paper, we drop the context variable c in the equations). We wish to emphasize that while logistic regression is an elegant and natural choice with some popularity in the literature (e.g., [9]), it is by no means the only choice.

The ability to define custom indicators appropriate to whichever application domain one is pursuing offers a tremendous amount of flexibility. As we will show in Sect. 3, both highly generic as well as application-specific trust indicators can be defined.

Finally, we wish to emphasize how MFTM can be seen as a generalization of a number of existing trust modeling techniques. Primarily this is because many trust modeling techniques do in fact consider multiple sources of evidence, but they weigh or combine this evidence in a non-data-driven manner. For example, the beta reputation system can be configured so that old advice is considered less important than new advice. However, a method for specifying how much more important newer advice should be treated compared to older advice is not specified. A similar situation occurs in the personalized trust model [48], where private and public reputation are weighed against each other. The weighting function chosen has a good statistical justification³, but ultimately does not specify how error bounds should be chosen, and thus how exactly to weigh personal and private reputation. MFTM can consider arbitrarily many sources of information, and learns the weights for them directly from data. For example, PTM could be roughly replicated by treating private and public reputation as trust indicators, and learning an appropriate function for combining them.

Another example of how MFTM is data driven is that it does not specify which distributions should be used to model beliefs. For example, both the foundational trust modeling Beta Reputation System (BRS) [23] and PTM rely heavily on the beta distribution. By allowing arbitrary machine learning methods to combine many forms of trust evidence into prediction, MFTM loses Bayesian rigor, but gains a large degree of flexibility and generalizability.

Trust modeling in the context of recommender systems has been examined by several researchers, dating back to the seminal paper of O’Donovan and Smyth [31]. More recent work has examined such issues as addressing cold start recommendation using trust modeling [17] or examining how to speed up trust-aware recommendation through improvements from matrix factorization [15]. In this paper, trust-aware recommendation arises as a central element of the validation of our proposed framework.

To explain: one of the recurring challenges in the development of trust models is finding grounds for the validation of the accuracy of the models [6]. Trust models aim to predict new trust links but independent agents may choose to follow or ignore these predictions. It is therefore difficult to truly evaluate the effectiveness of models without deploying a system on an active service and measuring the real effects of trust link prediction. As this is expensive and requires the cooperation of an active social network service, many models validate their effectiveness on data generated by an agent simulation instead (e.g., [4, 23, 36, 48]). While this is a useful approach for contrasting the effectiveness of various models and gives the researcher a large amount of control for simulating specific types of agent behavior, it clearly adds a layer of ambiguity between the reported effectiveness of the model and its potential for real-world application. In some cases, merely changing simulation parameters can defeat systems that had performed well on the simulations their creators had designed [24].

A rising trend in this field is to validate models by applying their predictions to a recommendation task (e.g., [9, 29]): that is, using the trust model to predict novel trust links, \(\hat{\Gamma }\), in a multiagent system (MAS), then feeding those predicted links into a trust-aware item recommendation system. These trust-aware recommender systems incorporate both user–item rating behavior and user–user social/trust connections to better recommend items by leveraging the fact that social/trust connections exert influence on the preferences of agents (e.g., you are more likely to watch/enjoy a film a trusted friend recommends). The logic of this two-part process is that when a trust model is able to accurately predict trust links in the context of peer to peer item recommendation, then the resulting accuracy of the recommender system trained with those links will improve.

For the validation of the model, we present in Sect. 3, we implement a task-aware item recommendation task. As will be explained in more detail in Sect. 3.2.3 we introduce two distinct trust-aware recommendation systems, TrustMF [46] and MTR [29]. MTR belongs to a class of recommenders based on k-nearest neighbors [19]. This approach requires a good selection of the value of k and an appropriate distance metric to determine closeness. In Sect. 3.2.3, we provide more insights into how these were chosen for our experimentation with MTR. TrustMF belongs to a class of systems known as latent factor models. To be clearer about how these systems operate, we provide below additional explanation. As will be seen, this method works well with data-driven trust recommendation, in seeking to leverage the most relevant factors of the users.

The rationale for testing the effect of personalization is simple: we expect trust formulation procedures to vary from person to person, therefore learning approximations of trust formulation procedures may be more accurate on a more personalized scale.

The inherent subjectivity of trust has important implication from a machine learning perspective. In particular, it implies that trust predictors trained on large data sets representing the behavior of many individuals are not necessarily more correct than those trained on smaller groups. While the predictor trained in the former case will likely have a higher accuracy across the broad population, it is essentially learning the “average” trust formulation procedure, potentially disadvantaging agent’s whose preferences are not aligned with the population at large.

We note that learning distinct predictors based on the data associated with clusters of users was suggested by Fang et al. [9], but was not pursued. While our approach is useful for capturing the variance of trust formulation in smaller groups, it does not attempt to learn the preferences of individual agents. We will discuss possible avenues for truly individual personalization of trust modeling and other approaches in Sect. 5.2.1.

In this work, we test the effect of personalization or trust link prediction on an item recommendation task. This is the case where agent \(a_i\) encourages agent \(a_j\) to invest resources into accessing or consuming item k based on their own experience with it. In particular, we test whether clustering agents and learning trust link predictors on the basis of clusters of similar agents, as opposed to learning a single trust link predictor for the entire population of agents, can increase the accuracy of trust-aware item recommendation systems. We also test the effect of altering the type of trust link prediction by either (a) attempting to predict the presence of an explicit friend/trust link or (b) predicting positive correlation in item review scores (i.e., two agents having scores that are the same).⁵

Our final analysis will report the recommendation accuracy of 7 configurations, where each configuration uses an identical set of agents and recommendation procedures, but a different procedure for predicting the trust links between agents. Each configuration is given a name reflecting which (if any) type of clustering was performed, and which type of trust link was predicted. The complete list is presented in Table 1.

The entire procedure can be described sequentially, as follows. Each step will be briefly explained, noting its inputs and outputs, then will be more carefully considered in subsections below.

When the first step is skipped, no personalization is performed. When the second step is skipped, no trust link prediction is performed. The rationale of skipping certain steps is to compare and contrast the effect applying these steps has on the final accuracy of the recommendation task.

The overview presented below assists readers in clearly understanding the interplay between the central processes of our approach: clustering, trust link prediction, and recommendation evaluation, before the details of our methodology are provided.

In our original tests, we attempted to set the number of clusters, k, by evaluating the silhouette scores of clusters in a large range for each data set, then simply choosing k based on whichever cluster count had the highest silhouette score. Unfortunately, this method was fickle, as the silhouette evaluation is based on a random sample of the clusters, and a single outlier could achieve a minimal score even if other nearby values of k were not optimal. Further, it is basically a heuristic to use cluster cohesion to choose the number of clusters, when ultimately we are interested in improving the personalized trust links. Therefore, we iterated over a range of cluster values and repeated the entire experiment with each choice of cluster score, using the MTR recommender system. Results are illustrated in Figs. 8, 9, 10, and 11. In general, results show that as the number of clusters searched for (k) increases, the error in the task follows a consistent trend of reduction. Note that when \(k=1\), the situation is equivalent to a non-personalized approach (as only searching for a single cluster is equivalent to doing no clustering), and as k increases the granularity of personalization increases. When predicting whether two users should be friends or not, MAE can be lowered by 0.003 points by adding personalization, while when predicting aligned preferences it is only lowered by 0.0005 points. These results are less impactful than the early results seen using the TrustMF classifier in Fig. 6. That said, the results indicate a consistent trend of improvement as clustering-based personalization is applied: a fairly consistent line of decrease in error can be observed in all lines.

We wished to verify that the results illustrated in these figures are indeed improving because our clustering technique was finding groups of similar users, which allowed the prediction techniques to learn more personalized classifiers for these groups. For instance, it is conceivable that splitting users into groups and learning multiple classifiers is helpful regardless of the groups picked, as this procedure would be similar to bootstrap aggregating [1], which allows simple classifiers to model multiple weak correlations in data. To test this, we repeated the FriendPredict experiment, but clustered agents into k clusters randomly¹⁴. The results illustrated in Fig. 12 compare this random clustering to clustering by social circle overlap. This figure clearly shows that the reduction in error is largely due to the non-random clustering approach. The solid and dashed lines start off identically at \(k=1\) on the x-axis (no clustering) but as the numbers of clusters increases, the error decreases, for the case where social clusters are used. We take this as evidence that the clustering technique is improving accuracy because clustering genuinely enables more personalized predictions, not simply because the number of models being learned has increased.

We also experimented with modifying the \(\kappa \) variable on MTR, effectively simulating sparsity, as this variable controls how many peer advisers can be considered for a recommendation. Results illustrated in Fig. 13 compare an accuracy on the SocialCluster-FriendPredict task, comparing the error rates for a single cluster (unclustered) and for 55 clusters (clustered). Overall, the gap in error rate is most dramatic when a larger \(\kappa \) value is used, but the advantage for a clustered approach applies over the range of values.

Finally, in Table 4 we present the results of a fivefold cross-validation over user ratings for these tasks. Best results are bolded. There are a number of interesting results. First, the conceptually simple MTR system outperforms TrustMF across the board, despite the fact that the TrustMF system was allowed to run for a much greater period of time in order to reach convergence. This gap in performance is often dramatic, for example, in the best cases for each system, MTR has a MAE that is 5% lower than TrustMF, and a RMSE that is 1.6% lower.

On the better performing MTR recommender system, the best results are achieved when predicting friendship links rather than predicting preference correlation. This makes, sense, as the MTR system already considers observable user preference (see Equation 22), thus predicting new instances of aligned preferences is not likely to add much new information. The best performing task, by a small margin, is the SocialCluster-FriendPredict task, which basically reconfirms the findings presented earlier in this section (with the added certainty of being averaged across folds). Clustering did not have an appreciable effect (at least not at the scale of \(10^{-4}\)) for the preference alignment prediction task.

Note that, in the case where an improvement was seen, although the scale of the effect appears small it is clear from the previous graphs that this is not merely due to statistical variance. For example, Fig. 8 clearly shows that the decrease in error between FriendPrediction and SocialCluster-FriendPredict is due to the increasing the number of clusters.

On TrustMF, results are more tightly grouped and there is quite little appreciable difference between experiments. As this system is more conceptually complex than MTR, it is difficult to interpret exactly why this might be the case. Interestingly, the best performing task for MTR is the worst performing task on TrustMF. Clustering does not have a positive effect in these experiments, and in the SocialCluster-FriendPredict task actually seems to harm the performance. Our early experiments with this recommender system (presented in Fig. 6) suggested that there might be more interesting differences between approaches, but this was not the case in these final results. We speculate that because these earlier results were computed using different techniques to select Yelp users (randomly selecting 10000 users versus our final strategy of selecting all 30000 users with more than 20 reviews submitted), the underlying distributions of ratings may have been different enough to cause this change.

In this work, we evaluated the effect that personalization via clustering had on the accuracy of a trust link prediction task. We accomplished this by predicting novel trust links on a data set of Yelp users and measuring accuracy of these predicted trust links via an item recommendation task.

Our results show that the option of predicting novel trust links results in better performance than using the explicitly stated trust links for the recommendation task. Further, our results show a small but consistent improvement in recommendation accuracy when clustering is used to determine groups of similar agents and distinct trust prediction models are learned for each group of agents with the MTR recommender system (e.g., Figs. 8, 9, 10, and 11). We showed that this improvement was not simply the result of the fact that more classifiers were trained, as randomly splitting users into groups does not improve accuracy nearly as much as the clustering technique does (Fig. 12). While our early results with TrustMF inspired confidence, and we hoped to see more dramatic improvement in recommender accuracy from these experiments, the final results show that, while consistent, the improvements in accuracy from the procedures outlined here are small. We will comment on ways these techniques could be improved, and potential avenues for future work, in Sect. 5.2.1.

In addition to the experiments with personalization (which explored multiple approaches for clustering users), we produced a comprehensive MFTM solution, combining techniques from the literature with novel features. We also make clear the applicability of MFTM to social networks. We experimented with predicting two types of trust links: explicit friendship (the FriendPredict experiments) and implicitly stated preference alignment (the PrefPredict experiments) and evaluated the utility of the predicted links derived by our methods, using two distinct trust-aware recommender systems. We found that the preferred target of trust link prediction can vary with the desired use-case: it was not clearly preferable to predict friendship links or preference alignment links. On the MTR system, which already strongly considers user preference alignment, our experiments performed better when predicting friend links, while on the TrustMF system predicting preference alignment between users produces (slightly) better recommendation accuracy.

In order to improve the lives of users of social media, we have presented an approach for predicting trust links between peers within these networks. Our framework makes it possible to assess whether the content created online is a good candidate to display to a user or not (where options may include flagging messages coming from sources that are not established to be well trusted).

Our work aims to improve online experiences by supporting distinct presentation of content to differing users, achieved by reasoning about relationships with peers and the concept of trust. Our concern with trustworthiness of content relates well to companion efforts devoted to detect digital misinformation [7, 21, 47]. There is a spectrum of possible outcomes when messages which are of questionable quality are shown to users, including special attention in contexts such as healthcare where the consequences may be more troubling [32]. Note that there will still be various options for actions to take, once trust modeling has provided some insights into messages of concern.

The methods we present here are designed to be self-contained algorithms which can be provided to any party which has the data at hand, to reason about trustworthiness. It would be possible, for instance, to have platform owners flag less trustworthy posts and individual users have agency to choose what kinds of information should be filtered for them. Our algorithms would be able to indicate, for a particular user, whether the other users in the environment are predicted to be trusted (based on whether a trust link is predicted to exist between them). There is then an interesting dilemma about whether top-down control of the social network (e.g., dictated by government) or bottom-up management of the content (e.g., under the control of individual users) should be launched in order to take actions with respect to the messages of agents with questionable trustworthiness. While our model promotes a solution that is attuned to an individual’s preferences, in cases where these may be in conflict with interests of the public (for instance, promoting hate) a tension may exist in deciding where the control should lie. If a decision is made to consult reputable outside sources to determine acceptability, this could potentially be integrated into the trust models to discourage inappropriate behavior. We do not propose an answer to this challenge of determining appropriate control but merely acknowledge this as a concern for anyone trying to address content recommendation in social media.

As our work has drawn out the value of personalized solutions, the models that we have presented should be flexible enough to support a variety of overall preferences with respect to final outcomes. With respect to the array of concerns for this Special Issue, our work is best viewed as focused on computational approaches grounded in artificial intelligence methods which assist in the detection of misinformation and disinformation. We introduce novel perspectives on this particular agenda for improving online social networks, through techniques for personalizing the analysis and with highlighting of the potential provided by performing trust modeling.

In this paper, we considered the problem of improving the experience of users on social networks, particularly with respect to content overload and the propagation of untrustworthy information. We argued that a trust modeling approach could be appropriate for social networks and could be used to enhance message recommendation systems. We then outlined some of the issues involved in applying these models as they currently exist. The types of trust models that can be applied need to be highly flexible, capable of capturing many different kinds of data, and personalizable.

We argued that a multi-faceted trust model was ideal for application to social networks. This is because the multi-faceted model can incorporate arbitrarily many signals from the agents and their environment into a data-driven model of how trust is apportioned by agents in an environment. We argued that this flexibility was a key feature, as it allows the model to adapt to many different kinds of social networks. In Sect. 3, we designed a comprehensive MFTM and applied it to a large data set, including multiple new features and features proposed in previous works. We experimented with personalizing the predictions generated by a multi-faceted model, by clustering similar users and learning distinct models for each cluster of users. We argued that although this approach is not “truly individualized” personalization, a data-driven model like MFTM imposes a tradeoff between the number of users a model is learned for, as smaller numbers of users will have less data available to train classifiers with. We showed that this approach can lower error rates in a downstream trust-aware recommendation task.

In brief, our primary contributions with this work are to:When misinformation abounds in social media, being able to judge which sources are trusted is a critical step in assisting the users in these networks to navigate the waters. In the sections below, we elaborate on ways to extend the models we have developed, and how to assist users in handling misinformation once our trust link prediction process has been run. We follow this with some suggested steps forward to assist some of the most vulnerable online users, older adults, illustrating how reasoning with clusters of users, the centerpiece of our proposed model, can be quite valuable in allowing unique experiences for this user base.