Self-learning Agents for Recommerce Markets

A market is most commonly referred to as a circular economy if it includes the three activities of reduce, reuse, and recycle, e.g., see Kirchherr et al. (2017). This means that while in a classical linear economy market each product is sold once at its new price and after use thrown away, in a circular economy a focus is put on recycling and thereby waste reduction. An entry point to study the concepts of circular economy and sustainable markets is given in, e.g., Stahel (2016) and Bocken et al. (2016). The overall idea is to save resources, reduce the use of resources and to avoid waste, which is also closely related to closed-loop supply chains, see, e.g., Savaskan et al. (2004), Gönsch (2014). Further, related aspects are environmental costs (Commoner 1972) or recycling investments (Schlosser et al. 2021).

Selling products on online marketplaces is a classical revenue management application, see, e.g., Talluri and Van Ryzin (2006). A comprehensive overview of the literature in dynamic pricing research is provided by the surveys by Chen and Chen (2015), den Boer (2015), Strauss et al. (2018), and Klein et al. (2020). The recent work of Gerpott and Berends (2022) particularly discusses dynamic pricing models under competition on online marketplaces.

The combined problem of (i) updating prices, (ii) learning demand behaviors, and (iii) identifying strategy equilibria in competitive markets is challenging as information is incomplete and merchants may constantly adapt their strategies. For analyzing and evaluating the complex interplay and long-term behavior of mutual self-adaptive pricing strategies, market simulations for classical e-commerce applications have been used, see Kephart et al. (2000), DiMicco et al. (2003), van de Geer et al. (2019), and Schlosser and Richly (2019). The latter put a focus on the dynamic interaction of two market participants and their global effects, focusing on each agent’s profit as the main performance indicator.

While the previously mentioned research projects focus mostly on online retail markets, many other use cases of dynamic pricing are known. A magnitude of publications considers pricing under competition in very different scenarios, potentially including other constraints, optimization goals, or mechanics which have to be taken into account, ranging from inventory management to fairness goals. Some of those publications rely on reinforcement learning (RL), for example, Maestre et al. (2018). This is one of the examples relying on learning from a simulation, a requirement that is demanded by all model-free RL methods.

RL is usually considered in those use cases in which the state representation becomes too complex to apply otherwise optimal dynamic programming-based solution methods. This can occur in online markets with many competitors or relevant demand features, but in other specialized use cases as well. For example, if the underlying optimization goal does not only aim for revenue optimization, but also for optimal usage of a limited resource, as described in Turan et al. (2020). In this use case, an RL-based policy is charged with the task of managing not only the price of rides but also the supply of electric vehicles in a dynamic market.

The previously mentioned publications vary widely in their choice of the solution method. The one presented here, relying on RL and neural networks, is only rarely chosen. To learn specific market dynamics in dynamic pricing, neural networks are used, for example in Yang et al. (2022). In this example, a recurrent neural network is used to represent the market dynamics in a market with perishable products. The combination of RL and recurrent networks has been shown in the past, but is not applied here. Another project considering dynamic pricing using RL and perishable products is provided by Shihab and Wei (2022). Other examples of specialized use cases incorporating competition and other relevant features include cloud resource pricing, see Chen et al. (2022). Markets in which customers also provide products to the market and thus influence the market prices by adjusting their own sales, are, e.g., small-scale energy markets and smart grid systems. Pricing in such markets has been studied by Tsao et al. (2022).

One example of research that considers retail products and which also incorporates selling used versions of those, is provided in Wen et al. (2022). In this use case, the pricing is not dynamic regarding competition, which distinguishes it from the scenario described here.

However, the additional buyback option of recommerce markets and associated circular resource flows, all while keeping the demand dependent on competitors’ choices, have not been studied in the mentioned frameworks.

In this section, we give a brief overview of existing RL methods. RL agents are trained through a process of trial-and-error. They iteratively interact with an unknown environment by means of an observable state and an action and observe feedback signals as well as the following state according to an underlying Markov process. Simulation-based RL algorithms enable the heuristically solving of large Markov decision problems with incomplete information.

The books by Sutton and Barto (2018) and Bertsekas (2019) summarize the state of the art in the field of RL. Examples of established RL algorithms are, e.g., Deep Q-learning Networks (DQN), cf. Mnih et al. (2015), Proximal Policy Optimization (PPO), cf. Schulman et al. (2017), or Soft Actor Critic (SAC), cf. Haarnoja et al. (2018).

Applications of DQN and SAC to dynamic pricing under competition can, e.g., be found in Kastius and Schlosser (2022). Unfortunately, RL algorithms typically require a lot of training data which makes them less attractive to use in practice. To overcome this issue, current approaches such as transfer learning, cf. Zhu et al. (2020), or multitask RL, cf. Teh et al. (2017), seek to use observable data more efficiently. Alternatively, synthetic market environments that mimic the use case under consideration can be used to pretrain RL agents.

In this context, we aim at closing a research gap by studying the applicability and effectiveness of different state-of-the-art RL methods to recommerce problems with additional buyback options using a rich open-source simulation and evaluation framework.

To mimic real-life recommerce markets, in our market simulation framework, we consider the following main components: (i) supplier(s), (ii) firm(s) including a private data/event store, (iii) a (shared) marketplace, (iv) consumer, (v) resource in use, and (vi) waste (cf. garbage), see Fig. 1. The components are connected as follows. Firms set prices for new and used products on their (or a central) marketplace. Arriving consumers decide (whether and) which product to buy from which firm. Bought products are considered as a resource in use on the consumer side. They may be disposed of as garbage after a while or sold back to one of the firms which offer a corresponding buyback price. Firms can also order new resources from their (or a central) supplier at a certain cost (cf. virgin cost). To be able to easily integrate various RL libraries, we follow a standard stationary discrete-time setup with an infinite horizon.

Note that this basic model sketched above also allows to describe the following special cases: (i) monopoly settings, (ii) scenarios with just one product type, and (iii) classical e-commerce scenarios with no buyback option (cf. linear economy), as well as combinations of those cases (i)-(iii).

Active decisions are made by the firms and the consumers. The pricing decisions of a firm can be organized by a certain rule-based strategy as well as an RL agent exploiting a firm’s gathered partially observable market data. Consumer behavior can be arbitrarily defined, e.g., by generating random numbers of interested customers with heterogeneous preferences and a diversified willingness-to-pay. Besides consumers of a myopic type, also certain shares of strategic or loyal customers can be defined and considered.

In the following, we describe the setup, a firm’s controls, the consumers’ behavior, competitors’ reactions, and a firm’s objective.

In this experiment, we consider a duopoly against the rule-based benchmark competitor (RBB). We compare the performance of the three RL algorithms: A2C, PPO, and SAC.

First, we discuss the results for the on-policy algorithms A2C and PPO. Figure 2 shows the learning curves of both algorithms. Each of these experiments was run four times independently for one million steps (two thousand episodes of 500 steps each), cf. Table 1. The learning curves depict the running averages of the episode returns (cf. profits) over a window of 100 episodes. The range between maximum and minimum episode returns of these four runs is colored. The bold line represents their average. In this graph, the loss region (i.e., values below zero) is hidden to allow a more accurate comparison in the upper-performance region. We observe that all agents reach the profit zone. However, with average scores of about 8000 per episode, we find that PPO clearly outperforms A2C. Further, we observe that PPO’s learning stability is significantly higher compared to A2C. Figure 3 illustrates a detailed view of a typical PPO training run. The performance develops in a narrow band, and catastrophic forgetting does not occur. Its average performance increases to 8160 by the end and shows a consistently stable trend in return and price selection. Results for single runs of A2C are given in the Appendix, see Fig. 13.

One observation in Fig. 3 deserves special attention because it seems paradoxical at first: Although with regard to profit the RL agent outperforms the rule-based agent after some time, it is later outperformed again. This gives the impression that the agent gets worse during training. However, the opposite is true. The PPO agent first learns to price new and used goods higher. For new sale prices greater than four, it experiences that the rule-based competitor always underbids its price by 1. Through reciprocal undercutting, a downward price spiral leads to the price settling just above the purchase price. However, the extremely low rate of return allows only low profits. While the agent gains experience through exploration, it learns that it can increase its overall profit by moving the market to a higher price band. The competitor then continues to undercut the agent, but only by the value of one. In doing so, the agent accepts that more customers will buy from the competitor due to the low price and that the competitor will also earn more from each customer as prices increase. However, it can still increase its profits compared to the low-priced market. The effect is, therefore, that the RL agent, in reaction to the competitor who is always undercutting him, allows the latter an increase in profits in order to be able to achieve higher profits for himself as well. The existence of this effect is due to the fact that the only optimization criterion for the agent is its own profit. A reward formulation that includes outperforming the competitor as an objective is discussed in Sect. 4.3.

On the same duopoly market, we also evaluated SAC. Figure 2 shows the learning curve of four SAC agents in a training run of 2000 episodes as in the experiments for PPO and A2C. Within the 2000 episodes, SAC achieves slightly less mean rewards compared to PPO.

SAC was developed as an off-policy algorithm for two main goals besides achieving competitive results: First, it is intended to have high sample efficiency, meaning that it requires significantly fewer steps during training. Second, SAC aims at a high training stability. The learning curve confirms that SAC achieves these two main goals. At about 250 to 300 episodes, agents are already close to their peak performance. The rest of the training yields only small performance gains.

Similar to Figs. 3 and 4 shows more details during training. The average prices start – apparently due to a different parameter initialization – at higher levels than for PPO. They then drop and end at 6.4 and 5.4, similar to PPO.

With the implementation and hardware used for these experiments, training per step with SAC takes about 3.7 times as long as with PPO. Collecting the examples from the market takes about 25% of the training time with PPO, and only about 6% with SAC. The speed of training is very similar for A2C and PPO. If we calculate the actual training time, the higher sampling requirement for PPO is put into perspective. A2C is superior to the other algorithms in pure time requirements.

Note, because the SAC training is time-consuming and the learning progress is slow later on, the number of training steps is reduced in some of the subsequent experiments. The same is done with A2C since its maximum performance is achieved very early on.

In Experiment A, the RL agents achieved overall good profits but were still outperformed by the rule-based competitor. In Sect. 4.2, we explained that this should not be interpreted as a weakness of the algorithms, but can be attributed to the reward function, which only evaluates its profit. Now, maximizing one’s own profit is not an inappropriate metric, yet firms will be reluctant to leave more profit to their competitors than to themselves in a symmetric market. Therefore, it is obvious to evaluate not only one’s own profits in the reward function but also whether the competitor is outperformed.

Therefore, a mixed reward function was used for the following series of experiments. It simply calculates the sum of (i) profit and (ii) the difference to the competitor’s profit. For these experiments, both summands were weighted equally, but a hyperparameter could be inserted to balance the two goals of maximizing profit and outperforming the competitor.

The results, see Fig. 5, show that the competitor can be clearly outperformed, but naturally, the agents’ profits are lower than compared to the original implementation, but this difference turns out to vary in its strength among the algorithms.

In Experiment A and B, the RL agents all trained against the undercutting rule-based competitor RBB. This satisfies the theoretical requirements of an MDP with fixed and known dynamics but poses a number of problems for practical applications. First, the competitor’s policy must be known for this to work. However, because it can be assumed that the competitor will not reveal its pricing strategy, it would have to be estimated from historical data, accepting inaccuracies. Second, the strategy trained using RL is only reliable against that particular rule-based strategy. If the competitor suddenly changes its pricing strategy, this weakens the performance of the RL strategy and necessitates further expensive training.

Therefore, the user wants a policy that can hold up against various different competitor strategies rather than being overfitted to a specific one. DeepMind had a similar challenge in training go and chess strategies, which was solved by self-play, see, e.g., Silver et al. (2017).

For our market, we developed a self-play variant in which an RL agent continues to train on a duopoly market, but the competitor’s policy is its own policy. In the programming implementation, a pointer to the RL agent is passed to the opponent, which eventually also implements the policy function. Naturally, as the agent is playing against itself and continuously updates its strategy, the Markov property is violated. Yet, the training still leads to relatively stable rewards in most cases. The motivation behind self-play is that the agent constantly develops strategies against its own, which are then, in turn, challenged. This is to prepare the agent for a variety of opponents’ policies.

Our experiments with self-play were again conducted with the algorithms A2C, PPO, and SAC. In this context, the entire series of experiments was performed with the normal reward function and the mixed reward function from Experiment B, cf. Sect. 4.3.

Figure 6 shows the learning curves of the three algorithms when trained against themselves. The learning curves under mixed reward look similar and can be found in the Appendix, see Fig. 14. These curves initially show that learning success is achieved for all of these three agents. However, they alone cannot tell us whether the agents can actually compete against an opponent with an arbitrary pricing strategy. Therefore, the returns of these learning curves should not be directly compared to those from the previous sections. To establish comparability, a model was saved every 50 episodes during self-play and each of these models was subsequently tested for 25 episodes. This created accurate learning curves comparing the model trained during self-play to the rule-based agent RBB. A mean run was selected for the three algorithms and the two reward functions and illustrated in Fig. 7.

For both reward functions, the agents have successfully learned to cope with the market and show that they are also successful against the rule-based benchmark competitor. Note, the peak rewards of all agents in the evaluation are at 8000 or just below. A2C suffers from strong fluctuations, some of the A2C agents do not reach a level of 5000, see Fig. 6.

The effects of the mixed reward function can also be seen in the training runs. During training, the difference between the agent’s and the opponent’s profit is also tracked as an optimization criterion. When evaluated against RBB, the agent trained by self-play via PPO and SAC beats the competitor. While the benchmark results do not achieve the same results as by training directly against the rule-based agent, we observe that they are close. This is noteworthy because these successes were achieved without ever having observed the rule-based competitor before. Thus, these experiments can be considered successful, particularly for PPO and SAC.

Figure 8a–c shows the learning curves of the three algorithms when trained against the undercutting rule-based strategy RBB under different observable state spaces. The result here is surprising. The expectation that less information would lead to worse results is not fulfilled. For PPO, except for a slight deterioration in stability, performance is similar for the three scenarios (cf. blue, orange, and green plots). This slight drop in stability can probably be explained directly by the lack of information. For A2C, performance improves without additional information, and also for SAC, mean rewards improve significantly. The SAC agents that are only deprived of the competitor’s stock level perform quite similarly (slightly better) than those with complete information; the agents that additionally lack the information about the number of products in circulation perform significantly better. Not only does it lower the time to peak performance to below 100 episodes, but it actually improves the peak performance. This brings the maximum closer to that of PPO. This is an important finding (cf. practical applicability), and yet, it raises the question of how to explain unexpectedly good performance under incomplete information. The first explanation is that the omitted information plays only a subordinate role. They do explain to some extent the future action of the competitor and the number of owners willing to sell, but the effects are so indirect that they are difficult to exploit for improving a policy.² Further, a higher-dimensional observation space also poses a challenge in principle for machine learning methods. The need for samples increases and patterns in the data are harder to detect.

However, the higher dimension is not sufficient to explain the specific phenomenon in SAC. Another theory might serve as the underlying reason here. Figure 9 (middle) displays that an exemplary SAC agent has a strong sensitivity with regard to the two arguments, number of resources in circulation and the stock level of the competitor. Thus, the mean of the SAC policy for used prices differs between 2 and 10 for similar situations, a jump through the entire action space. That a near-optimal policy should behave this way can be ruled out given the low importance of the two arguments, confirming the policy of the clearly more successful PPO agent. As one would expect, the policy of the PPO agent hardly depends on these two arguments. The lower performance maxima of SAC can be explained by this weakness.

An explanation can be found in the internal replay buffer Soft Actor-Critic relies on. It is based on the fact that more products are in circulation when policies are evolved in the market. This is shown in Fig. 9 (left) and is due to the fact that little-trained policies often buy back too much (and spend too much money in the process) in contrast to the more evolved ones. This means that, especially in early episodes, the experience buffer is filled only with state transitions with low in-circulation values. These samples remain in the experience-buffer, but are no longer useful for later training and can explain the anomalies in the policy. For example, a strong anomaly in the policy is in the range between 50 and 150 products in circulation, exactly the range from which the early samples come. Thus, omitting the in-circulation entry arguably improves the performance of SAC because it then no longer has the opportunity to overfit early to relationships that are not sampled from the market reliably in the long term. It is thus noted as a weakness of SAC compared to the on-policy method PPO that it is less able to ignore this bias in the early generated data.

In the previous experiments A-D, the algorithms were tested in duopoly scenarios. Naturally, in real-world applications, there are also different scenarios to master. In Experiment E, we investigate how the RL algorithms perform on market scenarios such as monopoly setups (cf. Sect. 4.6.1) and, in particular, oligopoly scenarios (cf. Sect. 4.6.2).

In this section, we provide an ablation study with respect to various model parameters in order to verify the general applicability of the proposed model framework as well as the robustness of the market results.

We summarize the results of the ablation experiments, cf. Table 2, in the following remark.

Naturally, the question arises of how to integrate the proposed model and solution framework into real-world information systems to solve companies’ individual revenue management challenges. In this regard, it would be necessary to calibrate the environment of our model, cf. Section 3, to the use case such that it mimics the market under consideration. One way to do this is based on domain knowledge and corresponding experts. Alternatively, a suitable environment could also be defined by using historical data. First, besides known or easy-to-estimate market and cost parameters, one would have to estimate consumers’ demand probabilities under competition, see, e.g., Schlosser and Boissier (2018a, b). For this purpose, established methods and sophisticated forecasting tools are available, see, e.g., Salinas et al. (2020) and references therein. Second, the underlying competitors’ price reactions need to be imitated. Based on suitable data regarding own price adjustments and competitors’ price reactions, standard methods can be used to compute and predict data-driven price anticipations, see, e.g., Schlosser and Richly (2019).

These two dynamics can then be used within an artificial model environment to generate price reactions, sales, and inventory levels for multiple parties, i.e., the agent and the competitors. Finally, the calibrated artificial environment can be used to train an RL agent without being forced to do that in practice. Finally, if obtained pricing strategies are plausible, they can be applied and trained further in the specific real-life application.

To test the applicability of the sketched approach, we distinguish between an original test environment A (which is used to produce realized market data) and a fitted auxiliary environment B (which is used supposed to be used for training). We consider the following example.

Example 1 As environment A, we consider the setup of the Base Case, cf. Section 4.1, where the competing firm 2 plays again the RBB strategy. (Our) firm 1 initially plays the two-bound RSS strategy, see (14)–(16). To ensure a sufficiently diversified dataset, we add an exploration rate of \(\varepsilon =0.1\) (for i.i.d. uniform prices within set A) to firm 1’s policy. For this setup, we simulate test data for \(D=20\) episodes (with \(E=500\) periods each) representing an available set of historic market data.

Our main insights can be summarized as follows:

The lack of more or better benchmark strategies and alternative RL algorithms, cf. Section 3.3.2, is a limitation. Also, results are to some degree stochastic and a larger number of runs would have to be used to quantify mean values and their standard deviations accurately. Further, the successful calibration of auxiliary training environments deserves further analysis. However, in the existing framework, alternative competitor strategies, calibration techniques, and other RL algorithms can be easily added and tested in greater detail. Moreover, the basic model could also be extended to capture more complex settings. For instance, for each firm, we may additionally consider a technology state serving as a sustainability image (cf. greenness, signaling, etc.), which increases demand. Further, this state could be stimulated via corresponding investment efforts and otherwise depreciates over time. Another research direction is to include strategic consumer behavior.