Intrinsically motivated reinforcement learning based recommendation with counterfactual data augmentation

Reinforcement learning-based recommender systems learn from interactions through a Markov Decision Process (MDP). Given a recommendation problem consisting of a set of users \(\mathcal {U} = {u_0,u_1,\cdots ,u_n}\), a set of items \(\mathcal {I} = {i_0,i_1,\cdots ,i_m}\), and users’ demographic information \(\mathcal {D}={d_0,d_1,\cdots ,d_n}\), MDP can be represented as a tuple \((\mathcal {S},\mathcal {A},\mathcal {P},\mathcal {R},\gamma )\), where each represents the following:Given a user \(u_0\) and the state \(s_0\) observed by the agent (or the recommendation system), which includes a subset of the item set \(\mathcal {I}\) and user’s demographic information \(d_0\), a typical recommendation iteration for user \(u_0\) goes as follows: First, the agent makes an action \(a_0\) based on the recommend policy \(\pi _0\) under the observed initial state \(s_0\) and receives the corresponding reward \(r_0\). Then, the agent generates a new policy \(\pi _1\) based on the received reward \(r_0\) and determines the new state \(s_1\) based on the probability distribution \(p(s_{new}\vert s_0,a_0)\in \mathcal {P}\). The cumulative reward after k iterations is as follows (Figure 1):

Structural Causal Models (SCMs) [19] can be represented as a tuple: \(\mathcal {M}_t(V_t,U_t,F)\) with the following components based on the state and action composition form at timestamp t. It is normally represented as a directed acyclic graph (DAG) \(\mathcal {G}\) with the following components:We assume that the dynamic count of states is n and the dynamic count of actions is m. The state observed at timestamp t is written as \(s_t\), which is the composition of \({s_t^1, s_t^2, \cdots , s_t^n}\) Local causal models are an extension of SCMs that only consider the local causal effect [20]. A local causal model can be represented as \(\mathcal {M}_t^\mathcal {L}(V_t^\mathcal {L},U_t^\mathcal {L},F^\mathcal {L})\), with the DAG \(\mathcal {G}^\mathcal {L}\) derived from the global causal model \(\mathcal {M}_t\) in the subspace \(\mathcal {L}\), having the same components with the following additional constraints:Moreover, the local causal model requires the set of edges in \(\mathcal {G}\) to be structurally minimal [21].

Formally, given an arbitrary trajectory \(\tau :(s,a,r,s')\) sampled from the replay buffer, where r is the reward signal received by the agent when action a is executed at state s, most of the trajectories in the large candidate item set situation are not informative, resulting in zero rewards. As a result, it is challenging to sample informative trajectories since the number of non-informative trajectories is significantly larger. Augmenting these informative trajectories to increase the likelihood of sampling them is a straightforward solution. We assume that the state \(s_{t+1}\) satisfy the SCM:\(f(\cdot )\) represents the causal mechanism, \(a_t\) is the action taken at timestamp t, and \(U_{t+1}\) is the noise term that is independent of \((s_t,a_t)\). Our main objective is to estimate the causal mechanism \(f(\cdot )\) and generate more data that is unseen but causally valid. However, estimating the global \(f(\cdot )\) is challenging [22]. To overcome this challenge, we draw inspiration from recent advances in local causal models [20, 23] and focus on estimating the local causal mechanism \(f_l(\cdot )\). The local causal model assumes the existence of a local directed acyclic graph (DAG) in a subspace such that where \(\mathcal {L}: \mathcal {S}\times \mathcal {A}\). It satisfies the following condition:where is used to represent the independence. In recommender systems, there is a large subspace of states in which users’ previous interests will not affect the final recommendation, as users’ interests are dynamic. By focusing on the subspace , we can formulate a local causal model such that the local DAG contains no edge from \(V_t^i\) to \(s{t+1}^j\). This implies that the local DAG is strictly sparser than the global DAG \(\mathcal {G}\). With this property, we can use the local causal model to conduct data augmentation to alleviate the sparsity problem in DRL RS.

Consider the counterfactual question “What if user u had been interested in item j instead of item i at timestamp t?” This question can be described in causal form as What if component \(s_t^i\) had the value x instead of y at timestamp t?” It can be solved by using Pearl’s do-calculus to the causal model \(\mathcal {M}\) to obtain a sub-model,Moreover, the incoming edge to \(s_t^i\) will be removed from \(\mathcal {G}{\text {do}(s_t^i=x)}\). Now, we utilize the local causal model to generate data that is unseen by the agent but causally valid. In order to achieve this, we will augment the data based on the counterfactual modification with the subset of causal factors at timestamp t and keep the remaining factors unchanged. Such an augmentation process can use the counterfactual model \(\mathcal {M}{\text {do}(s_t^i=x)}^\mathcal {L}\) to modify the causal factors \(s_t^{i\cdots j}\) and regenerate the corresponding children in the DAG.

However, such a process is computationally expensive in our recommendation scenario as it requires re-sampling for the children in the DAG. Inspired by the idea of collaborative filtering and the state composition form which was mentioned previously, we can simplify the process by omitting the sampling step. Specifically, the core of the augmentation is to estimate \(\mathcal {M}{\text {do}(s_t^i=x)}^\mathcal {L}\), which can be obtained easily by assuming that similar users will have similar interests, an idea inherent in collaborative filtering. Under such an assumption, we can obtain \(\mathcal {M}{\text {do}(s_t^i=x)}^\mathcal {L}\) by replacing the causally independent component of \(s_t\) using the local causal model. For example, we can identify those interaction histories that do not affect the current recommendation, and are thus causally independent of the current action \(a_t\). The overall algorithm¹ can be found in Algorithm 1.

The second aspect we use to address sparsity is intrinsically motivated exploration strategies. We propose to use empowerment to represent intrinsic motivation, which can boost the agent’s exploration capability, allowing it to reach more states and produce corresponding potentially informative interaction trajectories.

Empowerment is an information-theoretic method in which an agent executes a sequence of k actions \({\textbf {a}}^k \in \mathcal {A}\) while in state \(s \in \mathcal {S}\), according to an exploration policy \(\pi _{\textit{empower}}(s,{\textbf {a}}^k)\) (which we will shorten to \(\pi _e(s,{\textbf {a}}^k)\)). This exploration policy is a conditional probability distribution: \(\pi _e:\mathcal {S}\times \mathcal {A}\rightarrow [0,1]\). The agent’s goal is to identify an optimal policy \(\pi _e\) that maximizes the mutual information \(I[{\textbf {a}}^k, s'\vert s]\) between the action sequence \({\textbf {a}}^k\) and the state \(s'\) to which the environment transitions after executing the sequence \({\textbf {a}}\) in the current state s. This can be formulated as follows:The aim is to maximize the expectation of the logarithmic ratio between the joint probability distribution of the action sequence \({\textbf {a}}^k\), the state \(s'\), and the state transition probability distribution \(\mathcal {P}(s,{\textbf {a}}^k,s')\), and the exploration policy \(\pi _e({\textbf {a}}^k,s)\). By maximizing this quantity, the agent can boost its exploration capability and reach more states, which can in turn produce potentially informative interaction trajectories.

Here, \(\overline{\mathbb {E}}(s)\) refers to the optimal empowerment value, and \(\mathcal {P}(s,{\textbf {a}}^k,s')\) refers to the probability of transitioning to \(s'\) after executing the action sequence \({\textbf {a}}^k\) in state s, where \(\mathcal {P}:\mathcal {S}\times \mathcal {A}\times \mathcal {S} \rightarrow [0,1]\). Importantly,is the inverse dynamics model of \(\pi _e\). The optimal empowerment values are obtained by the policy \(\pi ^*\) that maximizes \(\mathbb {E}^{\pi ^*}(s)\).

However, the above definition of empowerment is more general than the RL setting since it considers a k-step policy, while RL usually considers a single-step policy. Moreover, estimating the k-step empowerment is challenging. Therefore, in this study, we use \(k=1\) to narrow down the empowerment into the RL setting, which only considers one step ahead. The Blahut-Arimoto algorithm [24, 25] shows that empowerment can be solved in low-dimensional discrete settings. Additionally [26], uses parametric function approximators to estimate empowerment in high-dimensional and continuous state-action spaces. It provides theoretical guarantees for using empowerment in recommender systems since state-action spaces are high-dimensional [1]. There are two possibilities for utilizing empowerment in RL:Both approaches are feasible for the normal reinforcement learning setting, which can encourage the agent to take an action that can result in the maximum number of future states. However, there is some conceptual difference between them. The second approach seeks states with a large number of reachable next states [27, 28], while the first approach aims to find high mutual information between actions and subsequent states, which is not necessarily the same as seeking highly empowered states [26]. The first approach can be achieved by transforming the state and its subsequent states’ representations into KL divergence and minimizing it [29]. However, this transformation introduces extra complexity and information loss, which may affect performance. The second approach, which uses the behavioral policy to explore highly empowered states, would be more suitable and simple for our setting. The main reason is that we are using a model-free approach to solve the problem. Model-free RL methods maintain two policies, which are the target policy \(\pi \) and the behavior policy \(\pi _e\). The second approach is more suitable for model-free approaches as it does not require extra computational cost to traverse all subsequent states and calculate the KL divergence. It can be easily adopted into existing RL frameworks.

Hence, the goal of the MDP process with the empowerment can be rewritten as,\(\pi _b\) is the behavior policy, and \(\alpha \) and \(\beta \) are constants used to balance the weight of instant reward and empowerment. We include the empowerment term as an additional component to the reward signal \(R(s_t,a_t)\) in order to encourage exploration by the agent.

From an information-theoretic perspective, optimizing for empowerment is equivalent to optimizing the inverse dynamics [28, 30] based on the distribution \(\pi _e(s,a)\). Therefore, we introduce the inverse dynamics into the objective function to calculate the empowerment. Our method is built on Soft Actor-Critic (SAC) [31] with temperature tuning and deterministic policy. The overall training algorithm can be found in Algorithm 2.

We follow the same training strategy as the standard SAC algorithm. However, since the empowerment is introduced, we modify the objective function to ensure that the empowerment term can be optimized. We use several function approximators to learn different components in the proposed method. The value function V is parameterized by \(\psi \), the Q-function is parameterized by \(\theta \), the target policy is parameterized by \(\phi \), and the inverse dynamics is parameterized by \(\xi \). Since we are using an off-policy algorithm where the transition probability is not learned, we use \(\mathcal {P}\) to represent the state transition probability in the environment. The soft Q-function can be trained by minimizing the following objective function:The target function \(V_\psi \) can be optimized by minimizing:where \(\beta \) is a constant used to balance the empowerment. Different from the origin SAC algorithm, we replace the policy term from \(-\log \pi _\phi (s_t,a_t)\) to \(g(s_t,a_t)\) to consider the empowerment where \(g(s_t,a_t)\) is defined as:Note that, different from \(s_t\), the \(s'\) represent all the possible subsequent states where \(a_t\) is executed in state \(s_t\) at timestamp t.

Similarly, the optimization of the policy \(\pi (\phi )\) can be written as:where apply the same substitution. The inverse dynamic \(p(\xi )\) will be updated based on:Lastly, the temperature parameter will be adjusted automatically by using the following entropy method [32]:Note that, SAC uses exponentially averaged value \(\psi '\) to stabilize the training process [33]. The update rule can be written as: \(\psi ' \leftarrow \lambda _{\psi '} \psi + (1-\lambda _{\psi '})\psi '\).

Moreover, we perform data augmentation on the replay buffer after each interaction to generate causally valid, unseen trajectories. This augmentation can provide more trajectories at the early stage to increase the number of samples. Specifically, most of the model parameters are learned by sampling from the replay buffer \(\mathcal {D}\). The training process can be described as searching for states or state-action pairs in \(\mathcal {D}\) to update the target policy such that the received reward is maximized. As the augmentation introduces more samples into the replay buffer, the gradient update process has a higher chance of achieving a better policy.

It is worth mentioning that we only augment informative trajectories. However, the definition of informative trajectories highly depends on the learning progress. We believe that every trajectory with non-zero reward is informative in the early stages but harmful when the final target policy is close to optimal. Hence, we selectively conduct augmentation with the replay buffer to ensure that zero-reward trajectories are not augmented to increase sparsity. However, as the interaction progresses, the way we determine informative trajectories changes. Some trajectories are informative in the early stages as the agent needs to explore all possibilities. In the later stages, the agent will pursue higher rewarding trajectories, making those low-rewarding trajectories less useful. In such situations, we design an adaptive threshold to evaluate whether the trajectory is worth augmenting or not. The designed adaptive threshold is intuitive, where a moving average is used. It can be represented as:\(\sigma \) is a custom constant used to determine the initial value of the threshold, and the decay rate \(\lambda _d \in (0,1]\) decreases as the number of episodes increases. By setting \((\sigma , \lambda _d)\) to appropriate values, we can achieve a monotonically increasing threshold. Ideally, we start with the initial values \(\sigma = 1\) and \(\lambda _d = 1.1\). \(T_{max}\) is a constant specific to the environment that represents the maximum reward that the agent can achieve at each step.

In order to demonstrate the superiority of IMRL, we conducted experiments in both offline and online simulation settings.

The complete results can be found in Table 2. We found that our method IMRL generally outperforms all existing state-of-the-art methods, including both non-reinforcement learning-based methods and reinforcement learning-based methods. It should be noted that IMRL does not outperform CauseRec in two datasets, but it still performs better than all other methods. Although IMRL has lower precision than CauseRec in Book-Crossing, we observe that the recall and nDCG are better than CauseRec. A similar situation occurs in Netflix, where the nDCG of IMRL is lower than CauseRec, but precision and recall are better than CauseRec.

We also report the performance of the selected reinforcement learning-based baselines in three online simulation environments. The results can be found in Figure 2. As we can see, IMRL outperforms all the others in all of the selected three simulation platforms. The performance in RecoGym and RecSim is quite close as those two environments are very small and do not require a complex exploration policy. Hence, we will focus on the later discussion in VirtualTB as it has a more complex environment which is more similar to the real-world situation.

The simplest way to evaluate the sparsity is to measure the speed of the model that tends to converge. In reinforcement learning, the way we use to measure sparsity is the number of useful samples that are fed into the agent via the replay buffer or sampled from the environment. Hence, dense environments can boost the model to converge at an early stage. In Figure 2a, we can see that IMRL has an outstanding speed of convergence than other methods in VirtualTB, which shows that it can overcome the sparse environment. In RecoGym and RecSim, IMRL also demonstrates considerable improvement when compared with those baselines. The main reason is that RecoGym and RecSim are small environments that contain only a few items and users. The sparsity is not serious and can be handled by random exploration.

To answer RQ3, we conducted experiments with the two major components of IMRL: empowerment and augmentation. The results of this study can be found in Figure 3, where IMRL-E denotes IMRL without empowerment, and IMRL-A denotes IMRL without augmentation. Additionally, we investigated the effect of the different strategies of empowerment in IMRL, including the KL-divergence approach mentioned in Section 3.2. We use IMRL-KL to represent this method.

We observed that both components play an important role in IMRL and contribute jointly to its final performance. Furthermore, we noticed that IMRL-KL did not perform as well as the other methods. One possible reason for this is that information is lost during the transformation in the calculation of KL-divergence. Hence, we can infer that our approach of using empowerment is better than KL-Divergence. In the next part, we will investigate the effect of the adaptive threshold.

We observed that removing empowerment-based exploration resulted in a drop in the model’s performance. As we mentioned earlier, empowerment-based exploration has a higher probability of reaching or producing informative states, which can enhance the model’s performance. This indicates that random exploration has limitations and may harm the model’s performance in recommendation tasks. However, we did not observe a significant improvement with the augmentation component, only a slight one. One possible reason for this is that VirtualTB is a simulation platform that can simulate real-world situations, but the number of states is limited due to computational resource constraints. A larger simulator may reveal a more pronounced difference in performance between augmented and non-augmented models. However, further study is required, and it is not the goal of this paper.

An important difference between our previous work [17] and this study is the role played by the adaptive augmentation threshold in early-stage discovery and reducing the number of uninformative trajectories. In this section, we investigate how the threshold affects early-stage performance. We start with \(\sigma =10\) and \(\lambda _d=1.1\). Note that the decay function of \(\lambda _d\) varies depending on the environment. We use VirtualTB as the primary evaluation platform and the following decay function:with \(T_{max} = 10\). We report the CTR at different stages of IMRL with adaptive threshold compared to IMRL without adaptive threshold (referred to as IMRL-T for short) in Table 3. We repeated the experiments five times with five different random seeds and report the average value. We found that with the adaptive threshold, IMRL can reach peak performance around 70, 000 episodes, whereas, without the adaptive augmentation threshold, it takes until 90, 000 episodes to reach peak performance. However, when the number of episodes reaches 100, 000, the performance of both methods is similar, which supports our theory that adaptive augmentation can improve the early-stage performance of the model.