Federated Constrastive Learning and Visual Transformers for Personal Recommendation

Solutions to personal recommendation in consumer electronics are divided into three categories: Even though these solutions provide high accuracy for non-complex data, they often require access to personal data such as browsing history, purchase history, and social network profiles. This raises privacy concerns and requires careful consideration of data protection regulations. Federated learning is a machine learning technique that allows multiple devices to collaboratively train a model without sharing their raw data with a central server [21, 22]. One possible direction to solve the aforementioned challenges of recommendation systems is to consider federated learning with a trust authority mechanism for improving the accuracy of recommendations while preserving user privacy. Another challenge of the existing solutions is the data quantity and quality. Indeed, these solutions are not able to efficiently learn from massive data. In order to tackle this challenge effectively, we delve into the realms of both clustering techniques and transformer models. By incorporating clustering into our approach, the recommendation system gains the capability to not only formulate a comprehensive global model encompassing all consumers but also to generate localized models tailored to specific clusters of similar consumers. This means that the system can discern nuanced patterns and preferences within different consumer segments, thereby enhancing the precision and relevance of its recommendations on a more granular level. In addition, and motivated by the success of transformers in training from massive data, this research work suggests the use of transformers for the recommendation system. The last challenge that we observe for the existing solutions in solving the personal recommendation problem is that they often face difficulty in capturing the fine-grained relationships between users and items, especially when dealing with sparse data. To address this issue, we propose a novel contrastive learning approach for feature embedding. Our method leverages the power of contrastive learning to learn meaningful representations of users and items in a shared embedding space. Thus, the contrastive learning framework ensures that user-item interactions are embedded in a way that positive pairs have higher similarities while negative pairs have lower similarities [23, 24]. This leads to a more compact and discriminative embedding space, where similar users and items are grouped, enhancing the recommendation performance. Contrastive learning leverages the abundance of negative samples to learn better embeddings, making it well-suited for handling sparse data scenarios in recommender systems.

We suggest federated learning-based transformers for personal recommendations (FLT-PR), a smart hybrid approach that combines federated learning and transformers for use in personal recommendations. The following are the primary achievements of this research work: The rest of this work is structured as follows. The “Personal Recommendation in Consumer Electronics” section shows the basic concepts of the personal recommendation process in consumer electronics. Following a summary of the key relevant works for both the FL-based framework and traffic flow forecasting in the “Related Work” section, the “FLT-PR: Federated Learning-Based Transformers for Personal Recommendation” section provides specifics on the FLT-PR. After that, the “Performance Evaluation” section assesses our approach using extensive data. The research study’s conclusions and the future directions of FLT-PR are discussed in the “Challenges and Future Directions” section.

In the realm of industrial IoT, Fu et al. [25] proposed an innovative approach to address security concerns by introducing federated learning with evidence. Their method centers on verifying the accuracy of the cumulative gradients sent by a server, which is achieved through Lagrange interpolation. However, a potential issue arises with the possibility of the server producing counterfeit aggregation results. To counter this challenge, the authors put forward a solution involving a trustworthy public key generator, which distributes keys following the initialization of the model. Subsequently, each user takes the following steps: first, they locally train their model and then transmit their encrypted gradients to the server. This encryption step is vital to safeguarding privacy, as it ensures that the gradient remains confidential during transmission. The server, in turn, performs accumulation on this encrypted data. Importantly, this process eliminates the need for key exchange between participants, rendering collision attacks unfeasible. Upon receiving the collected ciphertexts from the server, each participating member verifies the integrity of the results before commencing the encryption process. This approach substantially reduces the effectiveness of deciphering counterfeit gradients. When compared to other state-of-the-art techniques, the authors’ solution exhibits a lower vulnerability to potential assaults seeking to access private differentials. Furthermore, the introduced encryption only moderately impacts accuracy in contrast to a federated learning approach without gradient encryption. Lastly, it’s worth noting that this method minimizes overhead costs related to encryption/decryption, verification, communication, and overall operational overhead. The verification overhead remains constant, regardless of the number of participants involved. In the context of fog computing, Zhou et al. [26] have introduced a federated security learning approach. Under this paradigm, each fog node is empowered to carry out learning tasks and gather data from IoT devices. This approach significantly enhances the efficiency of low-level training. To ensure the security of IoT device data and shield the model from a parameter server that may be trustworthy but overly inquisitive, the authors have implemented sophisticated encryption techniques. Specifically, when applied to an individual fog node, this innovative method effectively thwarts the server’s attempts to deduce training set details from hyperparameters. Furthermore, it enables resistance against collusion attacks and safeguards the privacy of the mathematical model, even in situations where adversarial fog nodes or untrustworthy variables collaborate with other nodes. To mitigate vulnerabilities related to data and content disclosure during the training process, Yin et al. [27] have proposed a hybrid privacy-protection approach. Their methodology addresses these issues at multiple levels. First, during the weighted summing process, both the gradient weights of each participant and the characteristics of the information they provide are safeguarded using advanced function encryption techniques. Furthermore, they introduce a local quantification method to account for privacy loss due to Bayesian variability. This approach allows users to adjust their privacy budget in response to the data distributions within the datasets. Notably, Bayesian differential privacy outperforms the traditional differential privacy framework, leading to improved service quality. Lastly, to enhance performance, a sparse distinction matrix is incorporated into the function encryption. This optimization ensures that each training cycle only necessitates modifications to incremental values similar to previous gradients, minimizing the server’s computational load while significantly impacting the data transfer and storage capacities on the client side. Experimental results indicate that the use of sparse variance gradient techniques leads to improved model accuracy compared to traditional differential privacy measures.

Personalized recommendations aim to alleviate the impact of data overload by minimizing ambiguity and the delivery of unwanted information to consumers. Traditional recommendation systems fall into three categories: content-based, collaborative, and a blend of both approaches. In the context of the Massive Open Online Course (MOOC) system, a personalized recommendation framework was developed by integrating deep learning and big data technologies, as proposed by Li et al. [28] in their work. This approach introduces several pertinent strategies leveraging the BERT network to enhance the precision of the recommendation system. Firstly, it outlines the acquisition and preprocessing of open datasets. Secondly, it establishes the groundwork for a recommendation model by employing the BERT model and incorporating a self-attention mechanism. To bolster the model’s recommendation efficacy, a domain feature difference learning technique is implemented to extract profound network insights from the course content. In order to assess users’ attitudes towards connected products, ascertaining whether they hold a positive, negative, or neutral viewpoint, Solairaj and their team conducted a comprehensive analysis of user opinions, beliefs, and sentiments, as documented in their study [29]. This proactive approach enables businesses to adapt and tailor their products to meet customer satisfaction. Many of the existing sentiment analysis methods suffer from inaccuracies and time-consuming processes. As a solution to these challenges, the authors developed EESNN-SA-OPR. In this framework, novel recommendation algorithms were applied, incorporating both Filtering Collaborative (FC) and product-to-product (P-P) similarity techniques. Collaborative filtering aims to “predict the best shops,” while product-to-product similarity focuses on forecasting the best products. In a distinct research context, Swaminathan and colleagues, as documented in their study [30], advocate the utilization of collaborative filtering techniques for determining the optimal quantity of fertilizers required to promote sustainable crop development. Addressing the issue of imbalanced data concerning the unknown fertilizer quantities, they suggest incorporating auxiliary parameters such as soil fertilizer levels, land area, and soil chemical characteristics. Of note, their work introduced an innovative wide matrix factorization technique that derived feature representations by linearly mapping integrated contextual information vectors. To comprehensively comprehend historical fertilizer recommendations for the land, they employed a fully connected multi-layer perceptron, preserving non-linear higher-order interactions among contextual information. In a related vein, the creation of the Knowledge Augmented User Representation (KAUR) network, as presented in the study by Ma et al. [31], is aimed at exploring contrastive learning within collaborative knowledge graphs. This approach enables the discovery of semantic neighbors and the derivation of ambiguous interest sets from such knowledge graphs. The KAUR network leverages a graph neural network to learn the representation of each node in the collaborative knowledge graph. It treats information from nodes and their propagated neighbors as positive contrastive pairs, enhancing node representations through contrastive learning. Users or items with similar profiles are identified as semantic neighbors and included as positive pairs for further exploration of user interests. This user representation is then amalgamated with fuzzy interest sets, enhancing interpretability. In their research, Walek et al. proposed a hybrid recommender system known as the Eshop recommender, as detailed in their study [18]. This innovative system combined a fuzzy expert system with a recommender module comprising three subsystems, each utilizing collaborative filtering and content-based techniques. It serves as an e-commerce recommendation system designed to provide well-suited product recommendations to users. The fuzzy expert system played a pivotal role in generating the list of recommended products. It took into account a wide array of user preferences and their online store behavior. The expert system considered various variables, including the similarity to previously rated products, the coefficient associated with purchased products, and the average product ratings, in order to compile the recommended product list.

Certain major faults exist in the current recommendation systems, where they lack an accurate model from embedding to model training. Moreover, there is a lack of data privacy and model security. Particularly, the model data is distributed less securely and privately throughout the various entities of the recommendation platform. Table 1 offers the most recent research on recommendation systems and federated learning. In this research, we present an end-to-end architecture for a personal recommendation that integrates federated learning with the current transformers to overcome the aforementioned problems. We will show how our solution gives customers more precise product recommendations.

Figure 2 sketches the FTL-PR solution. It starts with consumer clustering, where the consumers that share similar features are merged into one group. Training is performed for each consumer, where the pre-trained model is used to train its proper data. This step includes embedding, which aims to map the consumer data to the feature vector. Additionally, it makes use of transformers to recommend items. By including the trusted authority principle, we create a federated learning framework for protecting model data during the federated learning stage. In this context, we define three types of models (local model for each consumer, local model for each group of consumers, and global model for all consumers). In the remainder of this section, we will go over each component of the FTL-PR approach in detail.

This phase aims to embed the categorical attributes of the consumers to feature vectors to facilitate the training of our transformer. We also consider these feature vector to cluster the consumers into disjoint clusters, each of which contains similar consumers. For embedding, we consider embedding with contrastive learning [32]. Contrastive learning is an unsupervised learning technique that encourages similar instances to be close to each other in the embedding space while pushing dissimilar instances apart. In the context of recommender systems, we adapt this framework to create meaningful user and item embeddings.

To leverage the Angular Similarity for contrastive learning, we propose the “Angular Contrastive Loss” that aims to maximize the angular similarity between positive pairs and minimize it between negative pairs.

Given a batch of anchor instances A, their corresponding positive instances P, and negative instances N, the Angular Contrastive Loss is defined as:The Angular Contrastive Loss encourages the embeddings of positive pairs to have lower angular distances (higher angular similarity) and embeddings of negative pairs to have higher angular distances (lower angular similarity). The Angular Contrastive Loss takes into account the angular distance between embeddings, which provides a more principled way of measuring similarity than the traditional cosine similarity. By maximizing the angular similarity for positive pairs, we ensure that similar user-item interactions are projected closer together in the embedding space. On the other hand, by minimizing the angular similarity for negative pairs, we push dissimilar user-item interactions apart, making the embedding space more discriminative. Angular Contrastive Loss has the advantage of being more robust to variations in magnitude between embeddings, as it focuses on the relative angles between them. This property makes it well-suited for capturing fine-grained relationships in recommender systems, especially when dealing with sparse data.

The goal of this step is to train the model on each network consumer. The k-means algorithm is first used to cluster the customers based on the extracted feature vectors. We then use Transformer to extract the recommendation from the feature vectors of the consumers. A detailed explanation is given in the following:

Three ongoing problems with the federated training process are addressed in this step. One is the validity check of the parameters that each consumer posts to the server. The server’s validity verification of aggregated results is the next issue that needs to be addressed. Furthermore, to safeguard the rights and interests of the consumers, we will offer a plan for assuring the consistency of cumulative results attained by various consumers.It is worth noting that verification takes place during both the uploading and aggregation phases. For backup and recovery validation, we employ one-way hash mechanisms, along with a stratified random sampling technique that generates a unique random number on each iteration. These two stages can be iterated to enhance the model further. Upon receiving the newly generated weight parameters for the encrypted global model, each user in the system proceeds to conduct local retraining using the designed transformer. The updated model can then be forwarded to the cloud server, and this process can be repeated as needed.

We used the two following widely known recommendation system datasets for evaluation: We contrast our suggested recommendation model with the following current state-of-the-art-based models to show its efficacy: We employ Recall@K and NDCG@K, two frequently used metrics, with K = [10, 20, 50], to assess the efficacy of top-k recommendation and preference ranking (Table 2). A measure of the percentage of related things among all items is called recall@K. To explain the position of hits, the ranking quality NDCG@K (also called Normalized Discounted Cumulative Gain) gives higher scores to top-ranked hits.

To train our personalized recommendation model with contrastive learning, we sample positive and negative items for each user from the set of user-item interactions X. The number of negative samples per user, K, is an important hyperparameter that affects the model’s performance. We then optimize the parameters \(\theta \) of the embedding function \(f_{\theta }\) to minimize the contrastive loss \(\mathcal {L}_{\text {contrastive}}\) using stochastic gradient descent or other optimization algorithms. During training, the model learns to map similar user-item pairs close together in the embedding space while pushing dissimilar pairs apart. After training, the model can make recommendations by computing the similarity between a user’s embedding and item embeddings to rank potential recommendations. We compare our feature embedding with dense embedding [36]. It is a mathematical representation of a consumer in a high-dimensional space, created using a dense embedding layer. The embedding captures important features and characteristics of the consumer, such as its genre, popularity, and historical interactions. A dense embedding is considered “dense” because it maps a consumer to a continuous vector space, where each element in the vector has a meaningful value, as opposed to a sparse representation where most elements are zero. This enables our clustering algorithm, and transformer to capture complex relationships between consumers that may be difficult to represent using simple features. Figure 3 presents T-SNE visualizations of both dense embeddings and embeddings generated using contrastive learning. In the dense embedding, the features obtained are dispersed widely across the 2D space without any discernible patterns or groupings. Consequently, T-SNE struggles to identify natural clusters or groupings of data points. Moreover, there is no clear separation between different clusters because this approach distributes points across the entire space without considering any underlying relationships. This results in high uncertainty or entropy in the feature space, where entropy refers to the level of disorder or unpredictability in the data distribution. In this scenario, the high entropy indicates that T-SNE fails to find any meaningful structure or patterns, leading to a lack of information about the underlying organization of the data. Additionally, the absence of structure and clear patterns makes it challenging to differentiate any signal from noise. Random noise in the features may dominate the visualization, making it difficult to interpret or draw meaningful conclusions. In contrast, the contrastive learning-based embedding generates features in a more organized manner, resulting in distinct clusters in the 2D space. The T-SNE algorithm successfully identifies similar features and groups them into clusters. Furthermore, different clusters are well-separated, indicating that T-SNE captures the underlying similarities and differences between groups of features effectively. Our embedding exhibits low entropy, suggesting a more organized and structured representation. The T-SNE algorithm can unveil the inherent organization in the features, providing a clearer view of the feature’s patterns and relationships. The clusters and separation in our embedding are meaningful representations of the features. Features within the same cluster are more similar to each other, while features in different clusters are more dissimilar. This enables better interpretation and analysis of the features.

To rigorously validate the performance of FLT-PR in terms of accuracy, we conducted robust experiments. We varied the top k recommendations retrieved from a set of values, namely \(\{10, 20, 50\}\), and computed recall and NDCG values for two datasets, MoviLens-1M and Amazon-book. Our simulation accounts for the maximum accuracy of both the global model, applicable to all consumers, and the local model within the respective consumer cluster. The results unequivocally demonstrated the superiority of FLT-PR when compared to baseline personal recommendation solutions. In fact, our solution outperformed other alternatives in eight out of twelve cases. Specifically, KAUR and KGAT achieved the best results in two cases each. Moreover, the average accuracy of our solution surpassed that of other solutions, including KAUR and KGAT. FLT-PR boasted an average accuracy of 0.27, while the other solutions failed to attain an average accuracy of 0.25. These achievements can be attributed to the fact that our solution capitalizes on both the global model, encompassing all consumers, and the local models within their respective clusters. Furthermore, the incorporation of the self-attention mechanism from transformer networks, which has demonstrated remarkable success in various applications, including natural language processing, also contributes to these commendable results. We can also explain the promising results of the competitive solutions (KAUR, and KGAT) in some cases by the fact that these solutions consider the knowledge graph in recommending the relevant items to the consumers. However, it is not always straightforward to design a knowledge graph in a real-world scenario. To evaluate the federated learning strategy used in this research work, we measure the stability metrics produced by trust value and information transmission success rate. Because hostile consumers may offer perturbation in terms of recommendation updates, we assign each consumer a trustworthiness score between 0 and 1. High-trust consumers are close to 1, whereas low-trust consumers are near 0. To withstand data manipulation and replay threats, the federated learning system built in this research effort is compared to FLT [37]. With a rigorous evaluation of the trust value of nodes, we can observe that the authority trust-based model employed in this research will lead to this tremendous achievement. Additionally, Figs. 4 and 5 show that their disruptive behaviors — which mainly consist of flooding attacks — are caused by repeatedly adding hostile consumers. The proposed hierarchical confidence model in this paper examines the overall stability of node trust value and is based on federated learning and trusted authority. The fundamental rationale is that the patterned trust scheme ensures that genuine models among the set of consumers receive reliable information even when adversaries are present. Consumers would consume more energy as a result of the flooding attack, and by analyzing edge energy consumption models, we might rapidly spot malicious consumers. The edge paradigm, which can decrease the energy use of the server, is the second concept we implement.