Introspection dynamics: a simple model of counterfactual learning in asymmetric games

Social behaviors often follow an evolutionary process. Behaviors that yield a high payoff proliferate, whereas inferior strategies go extinct [1, 2]. Providing a quantitative description of how individuals learn, make choices, and interact, however, is non-trivial. Actions of one group member may affect the behaviors of others; and, in a complex chain of interdependencies, this may, in turn, affect the group's surrounding environment [3–5]. Researchers aim to capture this complex dynamics with models of learning [6–13] and the tools of evolutionary game theory [14–16]. One key application of this theory is the study of cooperation in social dilemmas, where an individual's self-interest is at odds with the common good [17, 18]. For example, to be physically isolated during a pandemic is an action that is simultaneously costly for the individual and valuable for public health. Game theory is a standard mathematical framework to model decision-making in this context [19]. A game—defined by the players, their possible strategies, and their payoffs—captures how individuals interact and what the resulting consequences are.

One of the more subtle aspects of such a model is how it specifies the strategy updating process [20, 21]. In evolutionary game theory, individuals do not act optimally from the outset. Rather they dynamically adjust their strategies, based on their current payoffs. To capture this dynamics, the model needs to specify how individuals adopt new strategies, which is governed by the players' strategy update rule or selection rule. One example of such an update rule is imitation by pairwise comparison [22–24]: occasionally, individuals compare their own payoff to the payoff of a co-player. The larger the co-player's payoff, the more likely the focal individual imitates the co-player's strategy. Alternatively, there is also work that considers conformity-driven imitation [25–27]. Here, individuals tend to copy those strategies that are abundant among their neighbors, independent of the strategies' payoff consequences.

Yet 'not all learning is social learning' (i.e., learning from others) [2]. People can also adjust their strategies via introvert reasoning. For example, some models suggest that individuals may experiment with new strategies when their current payoff falls below a predefined aspiration level [28–30]. Other models assume that individuals adopt new strategies by figuring out the optimal response to their co-players' strategies, as in fictitious play or different variants of best reply dynamics [31–33]. Additionally, there is a body of work that studies the interplay and competition among different strategy updating rules [34–36].

Models of introvert reasoning are particularly relevant in asymmetric games [37, 38], where individuals are inherently unequal. Such inequalities naturally arise when individuals take different roles in an interaction (as in the ultimatum game [39, 40]), or when they differ in their available strategies [41, 42]. But even if individuals have access to the same set of strategies, they may differ in other relevant aspects, such as their initial endowments [43, 44], costs [45], productivities [46], or their location in a social network or hierarchy [47–49]. In the presence of such heterogeneities, strategies that yield a high payoff for one player may no longer be optimal for another. In this case, at least some degree of introvert reasoning seems to be required for effective learning.

In the following, we consider a learning model based on introvert reasoning that applies to all pairwise games. We assume that, from time to time, one individual is given the opportunity to change their strategy. In doing so, the individual thinks counterfactually: 'how much better off a random alternative strategy would have made me in the last interaction?' The larger the difference between the counterfactual payoff and the realized payoff, the more likely the individual is to abandon the current strategy in favor of the random alternative. We refer to this strategy update rule as introspection dynamics. Similar to previous studies [50–53], introspection dynamics assumes that players are 'myopic'. That is, when considering switching to a different strategy, players assume their co-player's strategy to be fixed. In particular, they do not anticipate that their new strategy may motivate their co-player to adapt later on. The rule implemented by introspection dynamics was previously used to simulate learning processes in (repeated) social dilemmas [52–55]. Here, we introduce an easy-to-implement model in the context of one-shot games, and we systematically analyze its mathematical properties. In particular, we derive an explicit formula for how likely players are to use any given strategy in the long run. This formula takes a particularly simple form when selection is weak (i.e., when realized payoffs only marginally affect a player's learning behavior).

Although symmetric games are more commonly studied, researchers have been interested in the evolutionary dynamics of asymmetric games for decades [56–66]. In comparison to these previous models, introspection dynamics has several properties that make it particularly comfortable to work with. For example, in models based on best reply dynamics and fictitious play [7, 31], players need to be aware of the entire payoff matrix. Moreover, they need to be able to compute best responses at all updating steps. Instead, in introspection dynamics, individuals only compare two given strategies at any point in time. In this way, introspection dynamics makes weaker cognitive assumptions on the players, and it allows a more immediate characterization of the eventual strategy distribution. This advantage seems to be particularly valuable in games with many strategies [67]. Similarly, there are natural connections between introspection dynamics and classical evolutionary models of asymmetric games [56–64]. For instance, a common approach is to assume that each participant in an asymmetric game is a member of a distinct subpopulation. Evolution then only occurs within each subpopulation. This modeling approach, however, is difficult to apply when all players are unique (for example, when all players in a population differ in how much they benefit from a public good [54]), or when there are no clear boundaries between different subpopulations. In such cases, introspection dynamics may provide a more natural way to explore the resulting strategy dynamics.

The remainder of this paper is organized as follows. In section 2, we formalize introspection dynamics for 2-player, 1-shot games with finitely many strategies. Section 3 contains our main analytical results. Here, we compute the unique stationary distribution of strategies. In addition, we provide explicit formulas for two cases: when selection is weak, and when players can choose among two strategies only. To illustrate these results, section 4 discusses the effect of asymmetry in several well-known social dilemmas. In particular, for the volunteer's timing dilemma, we show that the player with the lower cooperation cost tends to cooperate without delay. We also look at symmetric games and compare introspection dynamics with an evolutionary model of imitation [24]. Finally, section 5 discusses the results and puts them into a broader context. All proofs are provided in the appendices.

To model asymmetric interactions, we consider two individuals, player 1 and player 2, who interact in a normal-form game. Player 1 can choose among m pure strategies, ${\mathbf{S}}_{1}$ ,..., ${\mathbf{S}}_{m}$, and player 2 can choose among n strategies, ${\mathbf{S}}_{1}^{\prime }$, ..., ${\mathbf{S}}_{n}^{\prime }$. If player 1 chooses strategy ${\mathbf{S}}_{i}$ and player 2 chooses strategy ${\mathbf{S}}_{j}^{\prime }$, the resulting payoffs are π_ij and ${\pi }_{ij}^{\prime }$, respectively. We represent such a game by a bi-matrix,

In this representation, player 1 can choose one of the m rows, whereas player 2 chooses one of the n columns. Thus, we can also refer to player 1 as the 'row player', and to player 2 as the 'column player'. The first entry in the bi-matrix is the payoff to player 1, whereas the second entry is the payoff to player 2. The game is called symmetric if the two players have the same set of strategies (that is, if m = n and ${\mathbf{S}}_{i}={\mathbf{S}}_{i}^{\prime }$ for all i), and if the corresponding strategies yield the same payoffs (if ${\pi }_{ij}={\pi }_{ji}^{\prime }$ for all i, j). In what follows we mainly focus on asymmetric games.

We assume that the two players interact over many time steps, engaging each time in the game defined by the above bi-matrix. In each time step, they can dynamically adjust their strategies. A possible learning dynamics may posit that players compare their current strategy with another one, assuming that the co-player's strategy remains fixed. The process, which we refer to as introspection dynamics, goes as in figure 1. At each time step, one player is randomly chosen to reconsider their strategy. To this end, the player randomly draws an alternative strategy from the set of all its possible strategies. The player then compares its realized payoff π with the payoff $\tilde{\pi }$ the player could have obtained by playing the alternative strategy instead (keeping the co-player's strategy fixed). Let ${\Delta}\pi {:=} \tilde{\pi }\!-\!\pi$ be the difference between the counterfactual payoff and the realized payoff. The player switches to the alternative strategy with probability φ_β (Δπ) given by the Fermi function [23, 24, 68],

$$\begin{equation}{\varphi }_{\beta }({\Delta}\pi )=\frac{1}{1+{\mathrm{e}}^{-\beta \,{\Delta}\pi }}.\end{equation} \tag{ 2 }$$

Here, β ⩾ 0 is a parameter that measures the intensity of selection or selection strength. In one limiting case, β → 0, payoffs are irrelevant for the learning process and any alternative strategy is adopted with probability 1/2. In the other limiting case β → ∞, players only adopt the alternative strategy if the resulting payoff matches at least the current payoff. We refer to these two limits as the case of weak selection and strong selection, respectively.

Zoom In Zoom Out Reset image size

As we iterate this elementary updating step over time, we obtain a stochastic process on the space of all strategy profiles $({\mathbf{S}}_{1},{\mathbf{S}}_{1}^{\prime })$, $({\mathbf{S}}_{1},{\mathbf{S}}_{2}^{\prime })$, ..., $({\mathbf{S}}_{m},{\mathbf{S}}_{n}^{\prime })$. Each state corresponds to the strategies that the two players currently adopt. Simulations of this process are straightforward [54, 55]. Here, our goal is to analyze the mathematical properties of this process.

3.1. An explicit formula for the stationary strategy distribution

To derive explicit results, we note that each player's updating behavior only depends on the players' current strategies (whereas it is independent of the previous strategies of the players). As a result, we can represent the dynamics by a Markov chain. Given the current state $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$, we can compute the probability that the state changes to $({\mathbf{S}}_{k},{\mathbf{S}}_{l}^{\prime })$ in one time step. This transition probability is

$$\begin{equation}{T}_{ij,kl}=\begin{cases}\frac{1}{2(m-1)}\cdot {\varphi }_{\beta }({\pi }_{kj}-{\pi }_{ij})\quad \hfill & \text{if}\;i\!\ne \!k\wedge j=l\enspace \;(\text{player}\;\text{1}\;\text{changes})\hfill \\ \frac{1}{2(n-1)}\cdot {\varphi }_{\beta }({\pi }_{il}^{\prime }-{\pi }_{ij}^{\prime })\quad \hfill & \text{if}\;i=k\wedge j\!\ne \!l\enspace \;(\text{player}\;\text{2}\;\text{changes})\hfill \\ 0\quad \hfill & \text{if}\;i\ne k\wedge j\ne l\;(\text{both}\;\text{players}\;\text{change})\hfill \\ 1-\frac{1}{2}\left(\sum\limits _{p\ne i}^{m}\frac{{\varphi }_{\beta }({\pi }_{pj}-{\pi }_{ij})}{m\!-\!1}+\sum\limits _{q\ne j}^{n}\frac{{\varphi }_{\beta }({\pi }_{iq}^{\prime }-{\pi }_{ij}^{\prime })}{n\!-\!1}\right)\enspace \enspace \quad \hfill & \text{if}\;i=k\wedge j=l\;(\text{no}\;\text{player}\;\text{changes})\text{.}\hfill \end{cases}\end{equation} \tag{ 3 }$$

In these expressions, the factor $\frac{1}{2}$ corresponds to randomly drawing one of the two players. Similarly, the factors 1/(m − 1) and 1/(n − 1) correspond to randomly drawing an alternative strategy. The expressions φ_β (Δπ) then give the probability that the alternative strategy is adopted Eq. (2). We can collect these transition probabilities in an mn × mn matrix T = (T_ij,kl ). Here, the first double index denotes the row that corresponds to the previous state $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$ of the Markov chain, whereas the second double index corresponds to the next state $({\mathbf{S}}_{k},{\mathbf{S}}_{l}^{\prime })$. By construction, T is nonnegative and row stochastic (see appendix A).

We can use the matrix T to describe how likely we are to find the process in a given state at some time t given the initial distribution of the process. Let v_ij (t) denote the probability that at time t, the process is in state $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$. The mn-dimensional row-vector v(t) collects these probabilities in the same order of states as in the transition matrix. By the theory of Markov chains, the strategy distribution at time t is

$$\begin{equation}\mathbf{v}(t)=\mathbf{v}(0)\,{T}^{t},\end{equation} \tag{ 4 }$$

where v(0) is the initial strategy distribution.

To describe the long-run dynamics, we assume in the following that the selection strength β is finite (even if it may be arbitrarily large). In that case, the transition matrix T is primitive (see proposition 1 in appendix A). Thus, it follows from the Perron–Frobenius theorem [69] that v(t) converges in time to a unique and positive stationary distribution u, which is independent of the initial distribution v(0). The stationary distribution u solves the eigenvector problem

$$\begin{align}\hfill \mathbf{u}& =\mathbf{u}\,T\hfill \\ \hfill \mathbf{u}\,{\mathbf{e}}^{\top }& =1.\hfill \end{align} \tag{ 5 }$$

Here, e denotes the mn-dimensional row-vector where each entry is equal to 1 and the superscript ⊤ indicates transposition. Hence, the second equation in (5) is the usual normalization for a probability vector (requiring that the sum of all components of u is equal to 1). While (5) defines the stationary distribution implicitly, we can also obtain an explicit formula. To this end, we multiply the second equation in (5) by the row-vector e from the right, and add the result to the first equation. After some rearranging, this yields the relationship u (I + U − T) = e. Here, U = e^⊤ e is the mn × mn matrix with all entries being equal to 1, and I is the identity matrix. The matrix (I + U − T) is invertible (see proposition 2 in appendix A, with (I + U − T)⁻¹ being a fundamental matrix of the ergodic chain [70]). Thus, we obtain the explicit representation

$$\begin{equation}\mathbf{u}=\mathbf{e}\,{(I+U-T)}^{-1}.\end{equation} \tag{ 6 }$$

This equation allows us to numerically compute the stationary distribution u = (u_ij ) of introspection dynamics for any finite normal-form game.

To derive expressions that more immediately relate u to the payoffs of the game, we consider two important special cases in the following. First, for any number of strategies, we derive an approximation in the limit of weak selection (β ≪ 1). After that, we provide exact formulas in the case where each player can choose among two strategies (m = n = 2).

3.2. Weak selection

To directly relate the long-run abundance of strategies with the game's payoffs, we consider the limit of weak selection. For this, we first note that both the transition matrix (3) and its stationary distribution (6) depend on β; to make this dependence more explicit, we write T(β) for the transition matrix and u(β) for the stationary distribution. We then expand u(β) and T(β) as a Taylor series around β = 0,

$$\begin{align}\hfill T(\beta )& ={T}_{0}+{T}_{1}\beta +\mathcal{O}({\beta }^{2})\hfill \\ \hfill \mathbf{u}(\beta )& ={\mathbf{u}}_{0}+{\mathbf{u}}_{1}\beta +\mathcal{O}({\beta }^{2}).\hfill \end{align} \tag{ 7 }$$

Here, T₀ := T|_β=0 and T₁ := ∂T/∂β|_β=0 are the constant and the linear term of the Taylor expansion of the transition matrix. Both terms can be computed explicitly, see appendix B. Similarly, u₀ := u|_β=0 and u₁ := ∂u/∂β|_β=0 are the constant and the linear term of the stationary distribution; those are the terms we wish to compute. To this end, we take the expressions (7) and plug them into the eigenvector problem (5). This yields the relation

$$\begin{align}\hfill {\mathbf{u}}_{0}+{\mathbf{u}}_{1}\beta +\mathcal{O}({\beta }^{2})& ={\mathbf{u}}_{0}{T}_{0}+({\mathbf{u}}_{0}{T}_{1}\!+\!{\mathbf{u}}_{1}{T}_{0})\beta +\mathcal{O}({\beta }^{2}),\hfill \\ \hfill {\mathbf{u}}_{0}{\mathbf{e}}^{\top }+{\mathbf{u}}_{1}{\mathbf{e}}^{\top }\beta +\mathcal{O}({\beta }^{2})& =1.\hfill \end{align} \tag{ 8 }$$

By setting β = 0 in (8), we note that u₀ needs to satisfy the linear system

$$\begin{align}\hfill {\mathbf{u}}_{0}& ={\mathbf{u}}_{0}\,{T}_{0}\hfill \\ \hfill {\mathbf{u}}_{0}\,{\mathbf{e}}^{\top }& =1.\hfill \end{align} \tag{ 9 }$$

Similarly, by taking the first derivative of both sides of (8) with respect to β, and then setting β = 0, it follows that u₁ needs to satisfy

$$\begin{align}\hfill {\mathbf{u}}_{1}\,(I-{T}_{0})& ={\mathbf{u}}_{0}\,{T}_{1}.\hfill \\ \hfill {\mathbf{u}}_{1}\,{\mathbf{e}}^{\top }& =0.\hfill \end{align} \tag{ 10 }$$

As we show in proposition 3 in appendix B, both systems (9) and (10) can be solved explicitly.

In the special case that both players have the same number of strategies, m = n, the solution becomes particularly simple. In that case, we can approximate the abundance of strategy profile $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$ in the stationary strategy distribution by

$$\begin{equation}{(\mathbf{u})}_{ij}=\frac{1}{{n}^{2}}+\frac{1}{2{n}^{4}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{ij}\!+\!{\pi }_{iq}\!-\!{\pi }_{pj}\!-\!{\pi }_{pq})+({\pi }_{ij}^{\prime }\!+\!{\pi }_{pj}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\beta +\mathcal{O}({\beta }^{2}).\end{equation} \tag{ 11 }$$

To interpret this formula, let us introduce the following shortcut notation for some payoff averages for player 1,

$$\begin{equation}{\pi }_{{\bullet}j}=\frac{1}{n}\sum\limits _{i=1}^{n}{\pi }_{ij},\qquad {\pi }_{i{\circ}}=\frac{1}{n}\sum\limits _{j=1}^{n}{\pi }_{ij},\qquad {\pi }_{{\bullet}{\circ}}=\frac{1}{{n}^{2}}\sum\limits _{i,j=1}^{n}{\pi }_{ij}.\end{equation} \tag{ 12 }$$

Here, π_•j is the average payoff player 1 obtains when randomly sampling a strategy against a co-player with strategy ${\mathbf{S}}_{j}^{\prime }$. The next expression, π_i◦ is player 1's average payoff when using strategy ${\mathbf{S}}_{i}$ against a randomly sampling co-player. Finally, π_•◦ is player 1's average payoff if both players sample randomly. Analogous averages can be defined with respect to player 2's payoffs π'. Using this notation, we can rewrite the weak-selection formula (11) as

$$\begin{equation}{(\mathbf{u})}_{ij}\approx \frac{1}{{n}^{2}}+\frac{\beta }{2{n}^{2}}\left(({\pi }_{ij}\!-\!{\pi }_{{\bullet}j})+({\pi }_{i{\circ}}\!-\!{\pi }_{{\bullet}{\circ}})+({\pi }_{ij}^{\prime }\!-\!{\pi }_{i{\circ}}^{\prime })+({\pi }_{{\bullet}j}^{\prime }\!-\!{\pi }_{{\bullet}{\circ}}^{\prime })\right).\end{equation} \tag{ 13 }$$

We say the strategy profile $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$ is favored by selection if its abundance is larger than neutral, (u)_ij > 1/n². For weak selection, expression (13) suggests the following two mechanisms for a strategy profile to be favored: either (i) player 1's payoff from using strategy ${\mathbf{S}}_{i}$ against ${\mathbf{S}}_{j}^{\prime }$ is better than average, π_ij > π_•j ; or (ii) player 1's payoff from using strategy ${\mathbf{S}}_{i}$ against a random strategy of the co-player is better than average, π_i◦ > π_•◦. Two analogous mechanisms apply to player 2.

Interestingly, similar results have been previously derived for a birth–death model [60]. In that model, members of two separate populations engage in a bi-matrix game. The members of the first population act in the role of the row player, whereas the members of the second population act as the column player. In the special case that mutations are rare and selection is weak, the results of that model coincide with ours. In that case, the likelihood that population 1 uses strategy ${\mathbf{S}}_{i}$ and that population 2 uses strategy ${\mathbf{S}}_{j}^{\prime }$ simplifies to our Eq. (13). This agreement can be regarded as another instance of a more general observation: in the limit of weak selection, different selection rules often turn out to be equivalent [71, 72].

The previous expressions (11) and (13) tell us how often we are to observe a strategy profile $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$ on average. In many applications, it is also relevant to know how often player 1 adopts the given strategy ${\mathbf{S}}_{i}$, irrespective of the co-player's strategy. Based on (11), we can approximate the corresponding marginal probability ${\xi }_{i} {:=} \sum _{j=1}^{n}{u}_{ij}$ as

$$\begin{equation}{\xi }_{i}=\frac{1}{n}+\frac{\beta }{{n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{iq}\!-\!{\pi }_{pq})+\mathcal{O}({\beta }^{2}).\end{equation} \tag{ 14 }$$

Using the notation (12), this expression simplifies to ${\xi }_{i}=1/n+\beta ({\pi }_{i{\circ}}-{\pi }_{{\bullet}{\circ}})/n+\mathcal{O}({\beta }^{2})$. In particular, we can use this formula to predict which of two strategies S_i and S_k is more likely to be played over time. We obtain that ξ_i > ξ_k if and only if π_i◦ > π_k◦ (i.e., if and only if S_i performs better than S_k against a uniform sample of the co-player's strategies). Interestingly, this condition naturally depends on player 1's own payoffs π_pq , but it is independent of the co-player's payoffs ${\pi }_{pq}^{\prime }$. While such a result may appear intuitive, we show that it only holds under weak selection. Once selection becomes stronger, the co-player's payoffs can have a major impact on how likely the focal player is to play a certain strategy (for an explicit example, see appendix B).

In this section, we have derived the first-order approximation of the stationary distribution as a function of the selection strength β. Using similar methods, we can recursively compute all higher-order terms in the Taylor expansion of u. The respective expressions are derived in appendix C.

3.3. Games with two strategies

We now apply our model of introspection dynamics to general asymmetric 2-player, 2-strategy games. In contrast to the previous section, the results here are valid for any intensity of selection. For m = n = 2, the payoff matrix (1) simplifies to

Since the transitions in introspection dynamics only depend on payoff differences between strategies, we can further simplify this payoff matrix. Appendix A shows that the transition probabilities in (3) remain unchanged if we add a constant to all payoffs π_ij in a given column j or to all payoffs ${\pi }_{ij}^{\prime }$ in a given row i. We can therefore assume without loss of generality that the payoff matrix (15) takes the form

where A := π₁₁ − π₂₁, ${A}^{\prime } {:=} {\pi }_{11}^{\prime }\!-\!{\pi }_{12}^{\prime }$, B := π₂₂ − π₁₂, and ${B}^{\prime } {:=} {\pi }_{22}^{\prime }\!-\!{\pi }_{21}^{\prime }$. Thus, the number of free parameters reduces from eight in (15) to four in (16). For general m × n payoff matrices, this approach reduces the number of free parameters from 2mn to 2mn − m − n.

Using the payoff matrix (16), we can write the transition matrix T according to (3) explicitly,

$$\begin{equation*}\left(\begin{matrix}\hfill 1\!-\!\frac{1}{2(1+{\mathrm{e}}^{\beta A})}\!-\!\frac{1}{2(1+{\mathrm{e}}^{\beta {A}^{\prime }})}\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{\beta {A}^{\prime }})}\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{\beta A})}\hfill & \hfill 0\hfill \\ \hfill \frac{1}{2(1+{\mathrm{e}}^{-\beta {A}^{\prime }})}\hfill & \hfill 1\!-\!\frac{1}{2(1+{\mathrm{e}}^{-\beta {A}^{\prime }})}\!-\!\frac{1}{2(1+{\mathrm{e}}^{-\beta B})}\hfill & \hfill 0\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{-\beta B})}\hfill \\ \hfill \frac{1}{2(1+{\mathrm{e}}^{-\beta A})}\hfill & \hfill 0\hfill & \hfill 1\!-\!\frac{1}{2(1+{\mathrm{e}}^{-\beta A})}\!-\!\frac{1}{2(1+{\mathrm{e}}^{-\beta {B}^{\prime }})}\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{-\beta {B}^{\prime }})}\hfill \\ \hfill 0\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{\beta B})}\hfill & \hfill \frac{1}{2(1+{\mathrm{e}}^{\beta {B}^{\prime }})}\hfill & \hfill 1\!-\!\frac{1}{2(1+{\mathrm{e}}^{\beta B})}\!-\!\frac{1}{2(1+{\mathrm{e}}^{\beta {B}^{\prime }})}\hfill \end{matrix}\right).\end{equation*}$$

We then obtain the stationary strategy distribution u = (u₁₁, u₁₂, u₂₁, u₂₂) by either solving (5) or using (6),

$$\begin{equation}\begin{aligned}\hfill {u}_{11}& =\frac{1}{C}\left({\mathrm{e}}^{(A+B)\beta }+{\mathrm{e}}^{({A}^{\prime }+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{(A+{A}^{\prime })\beta }+2\,{\mathrm{e}}^{(A+{A}^{\prime }+B)\beta }+2\,{\mathrm{e}}^{(A+{A}^{\prime }+{B}^{\prime })\beta }\right)\hfill \\ \hfill {u}_{12}& =\frac{1}{C}\left(2\,{\mathrm{e}}^{A\beta }+{\mathrm{e}}^{(A+B)\beta }+2\,{\mathrm{e}}^{{B}^{\prime }\beta }+{\mathrm{e}}^{({A}^{\prime }+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{(A+{B}^{\prime })\beta }\right)\hfill \\ \hfill {u}_{21}& =\frac{1}{C}\left(2\,{\mathrm{e}}^{B\beta }+{\mathrm{e}}^{(A+B)\beta }+2\,{\mathrm{e}}^{{A}^{\prime }\beta }+{\mathrm{e}}^{({A}^{\prime }+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{({A}^{\prime }+B)\beta }\right)\hfill \\ \hfill {u}_{22}& =\frac{1}{C}\left({\mathrm{e}}^{(A+B)\beta }+{\mathrm{e}}^{({A}^{\prime }+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{(B+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{(A+B+{B}^{\prime })\beta }+2\,{\mathrm{e}}^{({A}^{\prime }+B+{B}^{\prime })\beta }\right).\hfill \end{aligned}\end{equation} \tag{ 17 }$$

Here, C is a normalization constant ensuring that components of u add up to 1. Accordingly, the respective marginal probabilities ξ₁ = u₁₁ + u₁₂ and ${\xi }_{1}^{\prime }={u}_{11}\!+\!{u}_{21}$ are

$$\begin{align*}\hfill {\xi }_{1}& =\frac{2}{C}\left(1+{\mathrm{e}}^{{A}^{\prime }\beta }\right)\left({\mathrm{e}}^{A\beta }+{\mathrm{e}}^{{B}^{\prime }\beta }+{\mathrm{e}}^{(A+B)\beta }+{\mathrm{e}}^{(A+{B}^{\prime })\beta }\right)\hfill \\ \hfill {\xi }_{1}^{\prime }& =\frac{2}{C}\left(1+{\mathrm{e}}^{A\beta }\right)\left({\mathrm{e}}^{{A}^{\prime }\beta }+{\mathrm{e}}^{B\beta }+{\mathrm{e}}^{({A}^{\prime }+{B}^{\prime })\beta }+{\mathrm{e}}^{({A}^{\prime }+B)\beta }\right).\hfill \end{align*}$$

The remaining marginal probabilities are ξ₂ = 1 − ξ₁ and ${\xi }_{2}^{\prime }=1\!-\!{\xi }_{1}^{\prime }$. We note that, in general, the stationary distribution u does not factorize over these marginal distributions. That is, in general, ${u}_{ij}\ne {\xi }_{i}\cdot {\xi }_{j}^{\prime }$ for i, j ∈ {1, 2} and β > 0. However, one can verify that such a factorization holds when the game's payoffs satisfy A = −B and A' = −B'. These latter two conditions imply that each player's incentive to choose one strategy (rather than the other) is independent of the opponent's strategy. In such a case, this independence is also reflected in the game's stationary distribution.

The previous results hold for arbitrary selection strength. We can also look at the weak selection approximation. Using (13), we obtain

$$\begin{equation}\begin{aligned}\hfill {u}_{11}& \approx \frac{1}{4}+\frac{\beta }{32}(3A+3{A}^{\prime }-B-{B}^{\prime })\hfill \\ \hfill {u}_{12}& \approx \frac{1}{4}+\frac{\beta }{32}(A-3{A}^{\prime }-3B+{B}^{\prime })\hfill \\ \hfill {u}_{21}& \approx \frac{1}{4}+\frac{\beta }{32}(-3A+{A}^{\prime }+B-3{B}^{\prime })\hfill \\ \hfill {u}_{22}& \approx \frac{1}{4}+\frac{\beta }{32}(-A-{A}^{\prime }+3B+3{B}^{\prime }).\hfill \end{aligned}\end{equation} \tag{ 18 }$$

In this case, the resulting marginal probabilities become particularly simple, ξ₁ = 1/2 + β(A − B)/8 and ${\xi }_{1}^{\prime }=1/2+\beta ({A}^{\prime }\!-\!{B}^{\prime })/8$. Player 1 favors strategy ${\mathbf{S}}_{1}$ if and only if A > B, and the analogous result holds for player 2. In the following, we apply these formulas to discuss the introspection dynamics of some classical 2-strategy games.

4.1. Social dilemmas with two strategies

As a first application of our model, we explore the introspection dynamics of asymmetric social dilemmas. In social dilemmas, players can choose whether to cooperate (C) or to defect (D). When players choose the same action, they prefer mutual cooperation to mutual defection; yet when they choose different actions, a defector gets a higher payoff than a cooperator [17].

4.1.1. Prisoner's dilemma

To start with, we consider the most stringent form of a social dilemma, the prisoner's dilemma [18]. For a simple instantiation of this game, we assume that a cooperating player pays a cost in order for the co-player to get a benefit. We incorporate asymmetry by assuming that the cooperation costs c_i > 0 may differ between players (whereas the benefit b of cooperation is the same for both). Therefore, the payoff matrix takes the form

The unique Nash equilibrium of this game is for both players to defect, independent of the players' exact costs. For an easier interpretation, however, we assume without loss of generality that cooperation tends to be more costly for the first player, c₁ ⩾ c₂.

In the following, we ask how likely players are to cooperate when they update their strategies according to introspection dynamics. To this end, we first consider the case of a fixed benefit of b = 1. Moreover, we assume that the first player faces considerable cooperation costs, c₁ = 0.6, whereas the second player's costs are negligible, c₂ = 0.1. To illustrate the workings of introspection dynamics, we start by simulating the basic process described in section 2. Figure 2(a) shows a representative realization. Over time, the two players independently switch between cooperation and defection. As a result, they experience all possible outcomes: there are times in which both players defect, but also instances in which either one or both players cooperate. Overall, however, mutual defection appears to be most abundant, as one may expect.

Zoom In Zoom Out Reset image size

To obtain a more quantitative understanding, we compute how likely we are to observe each of the four possible outcomes over time. To this end, we assume that initially both players defect, such that $\mathbf{v}(0)=\left({v}_{\mathbf{CC}}(0),{v}_{\mathbf{CD}}(0),{v}_{\mathbf{DC}}(0),{v}_{\mathbf{DD}}(0)\right)=(0,0,0,1)$. Then we use (4) to compute v(t) for all future time steps. In figure 2(b), we show the resulting cooperation probability for each player, as defined by ξ_C (t) = v_CC (t) + v_CD (t) for the first player and ${\xi }_{C}^{\prime }(t)={v}_{\mathbf{CC}}(t)\!+\!{v}_{\mathbf{DC}}(t)$ for the second player. As before, we observe that both players are most likely to defect. However, while player 1's cooperation probability remains low (approximately 5%), player 2's cooperation probability quickly reaches a stable value of about 38%.

To further explore this limiting behavior, we compute the game's stationary distribution. Setting the four game parameters in the payoff matrix (16) as A = −c₁, A' = −c₂, B = c₁, B' = c₂, we can use (17) to compute the stationary distribution $\mathbf{u}=\left({u}_{\mathbf{CC}},{u}_{\mathbf{CD}},{u}_{\mathbf{DC}},{u}_{\mathbf{DD}}\right)$ as

$$\begin{equation}\mathbf{u}=\frac{1}{(1\!+\!{\mathrm{e}}^{\beta {c}_{1}})(1\!+\!{\mathrm{e}}^{\beta {c}_{2}})}\cdot \left(1,{\mathrm{e}}^{\beta {c}_{2}},{\mathrm{e}}^{\beta {c}_{1}},{\mathrm{e}}^{\beta ({c}_{1}+{c}_{2})}\right).\end{equation} \tag{ 20 }$$

In particular, we note that the invariant distribution does not depend on the benefit of cooperation (since b does not enter the payoff differences A, A', B, B'). Moreover, because β ⩾ 0 and c₁ ⩾ c₂ > 0, the abundances of the four possible states always obey the same relationship as in figure 2(c),

$$\begin{equation}{u}_{\mathbf{CC}}\leqslant {u}_{\mathbf{CD}}\leqslant {u}_{\mathbf{DC}}\leqslant {u}_{\mathbf{DD}}.\end{equation} \tag{ 21 }$$

For β > 0 and c₁ > c₂ all inequalities in (21) are strict. That is, among the two players, the player with the lower cooperation cost can be expected to cooperate more often. However, mutual defection is always the most abundant state.

We can also use the stationary distribution (20) to study how parameter changes affect cooperation. In figure 2(d), we explore the effect of asymmetry. To this end, we increase the difference in the players' costs c₁ − c₂ while keeping the average cost (c₁ + c₂)/2 fixed. Interestingly, we observe a weakly positive effect. As we increase c₁ and decrease c₂, the first player's reduced cooperation is more than compensated by the second player.

Similarly, in figure 2(e) we explore the impact of selection strength. When selection is weak, such that payoff differences have a negligible impact, the stationary distribution simplifies to

$$\begin{equation}\mathbf{u}\approx \left(\frac{1}{4}\!-\frac{{c}_{1}\!+\!{c}_{2}}{8}\!\cdot \!\beta ,\frac{1}{4}\!-\!\frac{{c}_{1}\!-\!{c}_{2}}{8}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{{c}_{1}\!-\!{c}_{2}}{8}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{{c}_{1}\!+\!{c}_{2}}{8}\!\cdot \!\beta \right).\end{equation} \tag{ 22 }$$

In particular, the last two states (D, C) and (D, D) are favored by selection, whereas the other two states are disfavored.

In contrast, for the strong selection regime, we take the limit of u as β becomes arbitrarily large. We obtain

$$\begin{equation}\underset{\beta \to \infty }{\mathrm{lim}}\,\mathbf{u}=(0,0,0,1).\end{equation} \tag{ 23 }$$

Thus, we recover the classical prediction that both players learn to defect at all times.

4.1.2. Stag-hunt game

As another instance of a social dilemma, we explore a version of the stag-hunt game [73]. In contrast to the prisoner's dilemma, we now assume that players only derive a benefit if they both cooperate (which could reflect the advantage of collective action in hunting expeditions). Under this assumption, the payoff matrix becomes

As before, we assume the payoff parameters satisfy b > c₁ ⩾ c₂ > 0. Under this assumption, the stag-hunt game becomes a coordination game. There are two pure equilibria, according to which either both players cooperate, or both players defect. Mutual cooperation is always payoff-dominant (it is the equilibrium that gives a higher payoff to both players). However, when the benefit of cooperation is comparably small, b < c₁ + c₂, mutual defection is risk-dominant (loosely meaning that mutual defection is the safer option when there is uncertainty about the co-player's actions [74]).

We can analyze the introspection dynamics of the stag-hunt game in the same way as the prisoner's dilemma. This time, we set A = b − c₁, A' = b − c₂, B = c₁, B' = c₂. As a result, the stationary distribution according to (17) becomes

$$\begin{equation}\mathbf{u}=\frac{1}{{\mathrm{e}}^{\beta b}+{\mathrm{e}}^{\beta {c}_{1}}+{\mathrm{e}}^{\beta {c}_{2}}+{\mathrm{e}}^{\beta ({c}_{1}+{c}_{2})}}\cdot \left({\mathrm{e}}^{\beta b},{\mathrm{e}}^{\beta {c}_{2}},{\mathrm{e}}^{\beta {c}_{1}},{\mathrm{e}}^{\beta ({c}_{1}+{c}_{2})}\right).\end{equation} \tag{ 25 }$$

It follows that there are two possible orderings of the abundance of the four states. Which ordering applies depends on how the benefit b relates to the total costs of cooperation c₁ + c₂,

$$\begin{equation*}\begin{aligned}\hfill {u}_{\mathbf{CD}}\leqslant {u}_{\mathbf{DC}}\leqslant {u}_{\mathbf{DD}}\leqslant {u}_{\mathbf{CC}}\enspace \enspace \enspace \enspace \enspace \text{if}\enspace b\geqslant {c}_{1}+{c}_{2}\enspace \\ \hfill {u}_{\mathbf{CD}}\leqslant {u}_{\mathbf{DC}}\leqslant {u}_{\mathbf{CC}}\leqslant {u}_{\mathbf{DD}}\enspace \enspace \enspace \enspace \enspace \text{if}\enspace b\leqslant {c}_{1}+{c}_{2}.\end{aligned}\end{equation*}$$

Irrespective of the precise ordering, players are always most likely to settle at one of the two equilibrium outcomes. Moreover, introspection dynamics lends further support to the static notion of risk-dominance: players only coordinate on mutual cooperation if the benefit exceeds the sum of the costs. In figure 3, we illustrate this result for our baseline parameters, b = 1, c₁ = 0.6, and c₂ = 0.1. In particular, since c₁ + c₂ < b, mutual cooperation is by far the most abundant outcome.

Zoom In Zoom Out Reset image size

As in the prisoner's dilemma, we can also use the stationary distribution to discuss the impact of selection strength on cooperation, see figure 3. If selection is weak, we can approximate the stationary distribution by (18), yielding

$$\begin{equation}\mathbf{u} \approx \!\left(\frac{1}{4}\!+\frac{3b\!-\!2{c}_{1}\!-\!2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{-b\!-\!2{c}_{1}+2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{-b\!+\!2{c}_{1}\!-\!2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{-b\!+\!2{c}_{1}\!+\!2{c}_{2}}{8}\!\cdot \!\beta \right).\end{equation} \tag{ 26 }$$

In particular, the second state (C, D) in which only the high-cost player cooperates is always disfavored by selection. The other three states may be favored or disfavored, depending on the exact values of the benefits and costs. For the strong selection limit, we obtain

$$\begin{equation}\underset{\beta \to \infty }{\mathrm{lim}}\,\mathbf{u}=\begin{cases}(1,0,0,0)\quad \hfill & \text{if}\enspace b\! > \!{c}_{1}\!+\!{c}_{2}\hfill \\ (1/2,0,0,1/2)\quad \hfill & \text{if}\enspace b={c}_{1}\!+\!{c}_{2}\hfill \\ (0,0,0,1)\quad \hfill & \text{if}\enspace b\!< \!{c}_{1}\!+\!{c}_{2}\;\text{.}\hfill \end{cases}\end{equation} \tag{ 27 }$$

That is, in the limit of strong selection, introspection dynamics always selects the risk-dominant equilibrium (even though mutual cooperation is payoff-dominant for all parameter values).

4.1.3. Volunteer's dilemma

As a final example of a social dilemma, we consider the volunteer's dilemma [75]. Here, already one cooperating player is sufficient for both players to get a benefit. The payoff matrix is

where b > c₁ ⩾ c₂ > 0. The game has two pure equilibria in which either only the first player or only the second player cooperates, (C, D) and (D, C). In both cases, the dilemma arises because players prefer the other player to volunteer as a cooperator.

For the volunteer's dilemma, we set A = −c₁, A' = −c₂, B = −(b − c₁), B' = −(b − c₂). By (17), the stationary distribution is

$$\begin{equation}\mathbf{u}=\frac{1}{{\mathrm{e}}^{\beta b}+{\mathrm{e}}^{\beta (b+{c}_{1})}+{\mathrm{e}}^{\beta (b+{c}_{2})}+{\mathrm{e}}^{\beta ({c}_{1}+{c}_{2})}}\cdot \left({\mathrm{e}}^{\beta b},{\mathrm{e}}^{\beta (b+{c}_{2})},{\mathrm{e}}^{\beta (b+{c}_{1})},{\mathrm{e}}^{\beta ({c}_{1}+{c}_{2})}\right).\end{equation} \tag{ 29 }$$

Similar to the stag-hunt game, there are two possible orderings of the four states, depending on how the benefit b relates to the total costs c₁ + c₂ of cooperation,

$$\begin{equation}\begin{aligned}\hfill {u}_{\mathbf{DD}}\leqslant {u}_{\mathbf{CC}}\leqslant {u}_{\mathbf{CD}}\leqslant {u}_{\mathbf{DC}}\enspace \enspace \enspace \text{if}\enspace b\geqslant {c}_{1}\!+\!{c}_{2}\\ \hfill {u}_{\mathbf{CC}}\leqslant {u}_{\mathbf{DD}}\leqslant {u}_{\mathbf{CD}}\leqslant {u}_{\mathbf{DC}}\enspace \enspace \enspace \text{if}\enspace b\leqslant {c}_{1}\!+\!{c}_{2}.\end{aligned}\end{equation} \tag{ 30 }$$

In either case, the most likely outcome is that the low-cost player 2 cooperates, whereas the high-cost player 1 defects, as illustrated in figure 3.

The impact of selection strength can be discussed analogously to the previous cases (figure 3). When selection is weak, the stationary distribution simplifies to

$$\begin{equation}\mathbf{u}\!\approx \!\left(\frac{1}{4}\!+\frac{b\!-\!2{c}_{1}\!-\!2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{b\!-\!2{c}_{1}+2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{b\!+\!2{c}_{1}\!-\!2{c}_{2}}{16}\!\cdot \!\beta ,\frac{1}{4}\!+\!\frac{-3b\!+\!2{c}_{1}\!+\!2{c}_{2}}{8}\!\cdot \!\beta \right).\end{equation} \tag{ 31 }$$

Here, the third state (D, C) is always favored by selection. The other three states may be favored or disfavored, depending on the magnitudes of b, c₁, c₂. For strong selection and c₁ > c₂,

$$\begin{equation}\underset{\beta \to \infty }{\mathrm{lim}}\,\mathbf{u}=(0,0,1,0).\end{equation} \tag{ 32 }$$

That is, the low-cost player volunteers with certainty.

We briefly summarize the key results of this section in table 1. For each of the three social dilemmas we considered, we describe (i) the respective game parameters, (ii) the resulting stationary distribution, (iii) the respective weak selection limit, (iv) the strong selection limit, and (v) the game's Nash equilibria.

Table 1. Summary of our results on asymmetric social dilemmas with two strategies.

Game	Prisoner's dilemma	Stag-hunt game	Volunteer's dilemma
Parameters (A, A', B, B')	(−c1, −c2, c1, c2)	(b − c1, b − c2, c1, c2)	$\left(\!-\!{c}_{1},\!-{c}_{2},\!-(b\!-\!{c}_{1}),\!-(b\!-\!{c}_{2})\right)$
Stationary distribution (up to a constant)	$\left(1,{\mathrm{e}}^{{c}_{2}\beta },{\mathrm{e}}^{{c}_{1}\beta },{\mathrm{e}}^{({c}_{1}+{c}_{2})\beta }\right)$	$\left({\mathrm{e}}^{b\beta },{\mathrm{e}}^{{c}_{2}\beta },{\mathrm{e}}^{{c}_{1}\beta },{\mathrm{e}}^{({c}_{1}+{c}_{2})\beta }\right)$	$\left({\mathrm{e}}^{b\beta }\!\!,{\mathrm{e}}^{(b+{c}_{2})\beta }\!\!,{\mathrm{e}}^{(b+{c}_{1})\beta }\!\!,{\mathrm{e}}^{({c}_{1}+{c}_{2})\beta }\right)$
Weak selection (β ≪ 1)$\frac{1}{4}(1,1,1,1)+\frac{\beta }{16}(\star )$	$2(\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{c}_{1}\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{c}_{2},\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{c}_{1}\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}{c}_{2}$, $\left.{c}_{1}\!-\!{c}_{2},{c}_{1}\!+\!{c}_{2}\right)$	$\left(3b\!-\!2({c}_{1}\!+\!{c}_{2}),-b\!-\!2({c}_{1}\!-\!{c}_{2})\right.$, $\left.-b\!+\!2({c}_{1}\!-\!{c}_{2}),-b\!+\!2({c}_{1}\!+\!{c}_{2})\right)$	$\left(b\!-\!2({c}_{1}\!+\!{c}_{2}),b\!-\!2({c}_{1}\!-\!{c}_{2})\right.$, $\left.b\!+\!2({c}_{1}\!-\!{c}_{2}),-3b\!+\!2({c}_{1}\!+\!{c}_{2})\right)$
Strong selection (β → ∞)	(0, 0, 0, 1)	$\begin{array}{cccc}(1,0,0,0)& \text{if}\enspace b\phantom{\rule{0ex}{0ex}} > \phantom{\rule{0ex}{0ex}}{c}_{1}\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}{c}_{2}\\ (\frac{1}{2},\enspace 0,\enspace 0,\enspace \frac{1}{2})& \text{if}\enspace b\phantom{\rule{0ex}{0ex}}=\phantom{\rule{0ex}{0ex}}{c}_{1}\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}{c}_{2}\\ (0,0,0,1)& \text{if}\enspace b\phantom{\rule{0ex}{0ex}}< \phantom{\rule{0ex}{0ex}}{c}_{1}\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}{c}_{2}\end{array}$	$\begin{array}{cccc}(0,0,1,0)& \text{if}\enspace {c}_{1}\phantom{\rule{0ex}{0ex}} > \phantom{\rule{0ex}{0ex}}{c}_{2}\\ (0,\frac{1}{2},\frac{1}{2},0)& \text{if}\enspace {c}_{1}={c}_{2}\\ (0,1,0,0)& \text{if}\enspace {c}_{1}\phantom{\rule{0ex}{0ex}}< \phantom{\rule{0ex}{0ex}}{c}_{2}\end{array}$
Pure Nash equilibria	(D, D)	(C, C) and (D, D)	(C, D) and (D, C)

4.2. Comparing introspection and imitation in symmetric games

While the previous section focused on asymmetric social dilemmas, introspection dynamics is equally applicable to symmetric games. In the special case that the game has two strategies only, the assumption of symmetry implies A = A' and B = B', and the payoff matrix (16) simplifies to

Similarly, the stationary distribution u = (u_CC , u_CD , u_DC , u_DD ) reduces to

$$\begin{equation}\mathbf{u}=\frac{1}{2+{\mathrm{e}}^{A\beta }+{\mathrm{e}}^{B\beta }}\cdot ({\mathrm{e}}^{A\beta },1,1,{\mathrm{e}}^{B\beta }).\end{equation} \tag{ 34 }$$

In particular, we can immediately see that u_CD = u_DC for any A and B, as one would expect from a symmetric game. Moreover, the average cooperation probability of player 1 (and therefore also of player 2) becomes

$$\begin{equation}{\xi }_{C}={u}_{\mathbf{CC}}+{u}_{\mathbf{CD}}=\frac{1+{\mathrm{e}}^{A\beta }}{2+{\mathrm{e}}^{A\beta }+{\mathrm{e}}^{B\beta }}.\end{equation} \tag{ 35 }$$

For a given selection strength, this formula has only two free parameters. Hence, we can use this formula to explore the expected cooperation rate across all symmetric 2 × 2 games, by simultaneously varying both A and B as in figure 4(a). Depending on the signs of A and B, we recover the four classical symmetric games [76]: the prisoner's dilemma (A < 0 and B > 0), the stag-hunt game (A > 0 and B > 0), the snowdrift game (A < 0 and B < 0), and the harmony game (A > 0 and B < 0). For each possible combination of (A, B), we plot the resulting cooperation probability according to (35). We observe that there are three qualitative regions: (i) for B > 0 and A < B, defection is either a dominant strategy, or it is risk-dominant. In this parameter region, we, therefore, observe comparably little cooperation. (ii) Conversely, for A > 0 and B < A, cooperation is either dominant or risk-dominant. As a consequence, introspection dynamics leads to almost full cooperation. (iii) If both A < 0 and B < 0 (i.e., in the snowdrift game), players have an incentive to choose the opposite strategy of the opponent. In this parameter region, we, therefore, observe an approximately equal share of cooperators and defectors.

Zoom In Zoom Out Reset image size

For symmetric games, we can compare introspection dynamics to the classical pairwise imitation rule [24], see figure 4(b) (for details of the implemented imitation model, see appendix D). In most parameter regions, the corresponding results are strikingly similar. Only in the snowdrift game regime, imitation leads to a more gradual change from full cooperation (when B < 0 and A ≈ 0) to full defection (when A < 0 and B ≈ 0). To explain this difference, we note that the imitation process takes place in an entire population of players. As a result, individuals do not adapt their strategy to a specific opponent, but rather to the population average. The resulting imitation dynamics can lead to a mixed equilibrium in which cooperators and defectors coexist. The exact position of this equilibrium changes gradually in the game parameters, as observed in figure 4(b). Overall, these results suggest that the two dynamics are comparable when the game tends to converge toward a homogeneous population. However, if the game has at least one stable mixed equilibrium (as in the snowdrift game), the predictions of the two models may diverge.

4.3. The volunteer's timing dilemma

In the previous examples, we have considered social dilemmas with two strategies only. In this section, we illustrate how introspection dynamics can be applied to a game with arbitrarily many strategies. To this end, we take the volunteer's dilemma and turn it into a timing dilemma. Here, players no longer only determine whether or not to volunteer. Instead, the game takes place over time, and players determine how long they wait for the co-player to volunteer before they volunteer themselves. This kind of game has been first studied by Weesie [78] as a model to explore the emergence of 'wait and see' behaviors in the context of voluntary action. Here we explore the game's introspection dynamics.

To formalize the game, we assume that within a given time interval [t₀, t_max] := [0, 1], players need to make a decision whether to volunteer. For simplicity, we assume that time is discretized, such that there are n + 1 evenly spaced time points {t₀, t₁, ..., t_n } = {0, 1/n, ..., 1}, at which players may volunteer. As before, we assume that players may have different costs to volunteer and that player 1's cost tends to be larger, c₁ ⩾ c₂. If at least one player volunteers during the time interval, both players derive some benefit. However, to add a component of time pressure, we assume that the benefit of cooperation is linearly degrading in time. That is, if one of the players cooperates immediately at time t₀, both players get a benefit of b > c_i . However, if the first player to volunteer cooperates at time t_max, the resulting benefit is zero.

A strategy for the volunteer's timing dilemma is now a rule that tells the player at which point to volunteer. To this end, we associate each strategy {${\mathbf{S}}_{0},\,{\mathbf{S}}_{1}$, ..., ${\mathbf{S}}_{n}$} with a waiting time {t₀, t₁, ..., t_n }. For i < n, a player with strategy S_i volunteers at time t_i , unless the co-player has already volunteered earlier (in which case the focal player's cooperation is no longer required). For i = n, we associate the respective strategy S_n with not cooperating at all. If player 1 and player 2 adopt the strategies S_i and ${\mathbf{S}}_{j}^{\prime }$, respectively, the resulting payoffs are

$$\begin{equation}\begin{aligned}\hfill {\pi }_{ij}=\begin{cases}\left(1\!-\!\mathrm{min}[{t}_{i},{t}_{j}]\right)\!\cdot \!b-{c}_{1}\quad \hfill & \text{if}\enspace i\!\leqslant \!j\enspace \text{and}\enspace i\!< \!n\hfill \\ \left(1\!-\!\mathrm{min}[{t}_{i},{t}_{j}]\right)\!\cdot \!b\quad \hfill & \text{otherwise}\hfill \end{cases}\\ \hfill {\pi }_{ij}^{\prime }=\begin{cases}\left(1\!-\!\mathrm{min}[{t}_{i},{t}_{j}]\right)\!\cdot \!b-{c}_{2}\quad \hfill & \text{if}\enspace j\!\leqslant \!i\enspace \text{and}\enspace j\!< \!n\hfill \\ \left(1\!-\!\mathrm{min}[{t}_{i},{t}_{j}]\right)\!\cdot \!b\quad \hfill & \text{otherwise.}\hfill \end{cases}\end{aligned}\end{equation} \tag{ 36 }$$

Equivalently, the game can be represented by the payoff matrix

Note that for n = 1, we recover the payoffs of the original volunteer's dilemma.

We explore the introspection dynamics of the volunteer's timing dilemma numerically, by computing the stationary distribution with (6). To this end, we first consider a case in which n = 4, such that the players' possible waiting times are $\!t\!\in \left\{0,\frac{1}{4},\frac{1}{2},\frac{3}{4},1\right\}$. Moreover, we consider a normalized benefit of b = 1, and the player's cooperation costs are c₁ = 0.7 and c₂ = 0.3, respectively. The resulting stationary distribution is displayed in figure 5(a). As one may expect, we observe that the low-cost player is more cooperative; however, more surprisingly, this player typically cooperates without any delay.

Zoom In Zoom Out Reset image size

A natural question, then, is if the cost asymmetry helps to solve the timing dilemma. In figure 5(b), we show that the higher the difference in costs, the faster one of the players volunteers. This positive effect of asymmetry is particularly pronounced when selection is strong. To explore how these results depend on our discretization of the time interval, we have repeated the above analysis for different n ∈ {1, ..., 15}. In addition, we have run simulations in which players are able to volunteer at any time in [0, 1]. The results of this analysis are displayed in figure 5(c). Interestingly, the low-cost player is most likely to volunteer when n = 1 (the original volunteer's dilemma), in which case a 'wait and see' approach is ruled out by the design of the game. However, across all values of n considered, the low-cost player always cooperates with a probability of at least 80%, implying that this player remains the most reliable volunteer.

Herein, we present a simple model of learning in social interactions. The model considers individuals who can choose among several strategies using introspection, that is, by reasoning about their strategies' prospective consequences. Compared to imitation models [24], this approach has the advantage that it can be applied to symmetric and asymmetric games alike. As another advantage, we can derive explicit formulas that describe how the system evolves in time, and what the long-run abundance of each strategy is. These formulas become particularly simple when players can only choose among two strategies, or when selection is weak [79–81].

Mathematically, the model takes the form of a Markov chain, whose states are the possible combinations of the strategies of the two players. In particular, if the players can choose among m and n strategies, respectively, there are mn states, arguably the minimum number of states that any learning model for such asymmetric games must have. While a similar Markov approach can also be used to analyze imitation processes in populations of players, the two approaches differ in their computational complexity. Population models need to record how many players apply any given strategy at any point in time. As a result, if a symmetric game with n strategies is played in a population of size Z, there are $\left(\genfrac{}{}{0.0pt}{}{Z+n-1}{n-1}\right)$ possible states [82]. Since this number of states increases exponentially in n, numerically exact results are only feasible in games with a few strategies [83], or when mutations are rare [84–87]. In contrast, the computational complexity of introspection dynamics only depends polynomially on the number of strategies. As a result, the stationary distribution can be easily computed even in complex games with many possible moves [67].

Despite these differences in terms of computational complexity, the results of introspection dynamics are often in remarkable agreement with other evolutionary processes. For example, in the limit of weak selection, the introspection dynamics of two adapting learners becomes equivalent to a birth–death model of two co-evolving populations [60] (section 3.2). Similarly, across all symmetric 2 × 2 games, we find that introspection dynamics often recovers the results of pairwise imitation [24] (figure 4). The only notable exception occurs for the snowdrift game (as in [10]). In the snowdrift game, pairwise imitation typically selects for the mixed symmetric equilibrium, in which cooperators and defectors coexist. In contrast, introspection dynamics selects pure but asymmetric equilibria in which one player cooperates and the other defects.

To illustrate our analytical results, we apply our framework to a number of asymmetric games. As one particular example, we study the dynamics of the volunteer's timing dilemma [78]. In this game, one player is required to volunteer as quickly as possible to create a benefit for the whole group; yet each player may be tempted to wait, hoping the other player would give in first. When players differ in their costs to volunteer, we observe not only that the player with the lower cost is more likely to volunteer; we also note that this player usually volunteers without any delay (figure 5). This game thus illustrates how players may sometimes benefit from asymmetry because it helps them to coordinate more efficiently. Herein, we have studied this advantage of asymmetry by assuming that players have different costs. However, similar results could be obtained for other sources of asymmetry, such as when people differ in their endowments [43, 44], their productivities [54], or their strategic options more generally [41].

Here, we consider a comparably simple setup of introspective learning: there are two individuals who continually interact with each other. However, one could also imagine an entire population of introspective learners who interact with one another in such pairwise games. The results of such a process are likely to depend on the population's network topology [51, 77, 88–90]. Alternatively, one may also imagine that introspective learners engage in interactions that involve more than two players at a time [91, 92]. The introspection dynamics of such multiplayer games can easily be studied with simulations [54], similar to the 2-player games considered in this paper. Especially for asymmetric games with many unequal players, introspection dynamics can serve as a simple model to study the resulting learning processes.

This work was supported by the European Research Council Starting Grant 850529 (E-DIRECT) and by the Max Planck Society. We would like to thank the members of the Research Group Dynamics of Social Behavior and the Thesis Advisory Committee of MC for valuable feedback.

The data that support the findings of this study are available upon reasonable request from the authors.

Here, we derive some properties of the transition matrix T defined in (3) in the main text and its stationary distribution u.

Proposition 1 (Properties of the transition matrix T).

• (a)  
  For a fixed j, T is unchanged by adding an arbitrary constant d_j to all payoffs π_ij .
• (b)  
  For a fixed i, T is unchanged by adding an arbitrary constant ${d}_{i}^{\prime }$ to all payoffs ${\pi }_{ij}^{\prime }$.
• (c)  
  If β is finite, T is primitive.

Proof. 

• (a)  
  Transition probabilities in (3) in the main text only depend on payoff differences. If we fix j and replace all π_ij with ${\tilde{\pi }}_{ij}{:=}{\pi }_{ij}\!+\!{d}_{j}$, the transition probabilities are unaltered since, for all p ∈ {1, ..., m}, ${\varphi }_{\beta }({\tilde{\pi }}_{pj}-{\tilde{\pi }}_{ij})={\varphi }_{\beta }\left(({\pi }_{pj}\!+\!{d}_{j})-({\pi }_{ij}\!+\!{d}_{j})\right)={\varphi }_{\beta }({\pi }_{pj}-{\pi }_{ij}).$
• (b)  
  A similar argument as in point (a) applies by fixing i and replacing all ${\pi }_{ij}^{\prime }$ with ${\pi }_{ij}^{\prime }+{d}_{i}^{\prime }$.
• (c)  
  T is primitive if there is $a\in \mathbb{N}$ such that T^a is positive. Let a = 2. For arbitrary i, j, k, l,

  $$\begin{equation*}{({T}^{2})}_{ij,kl}=\sum\limits _{p,q}{T}_{ij,pq}{T}_{pq,kl}\geqslant {T}_{ij,il}{T}_{il,kl}+{T}_{ij,kj}{T}_{kj,kl}.\end{equation*}$$

  For finite β, the entries of T as defined in (3) in the main text imply that both terms on the right-hand side of the above inequality are positive. Therefore, T² is positive and T is primitive under finite selection intensity.

  □

The first two properties described in proposition 1 are useful as they allow us to simplify the payoff matrices we need to consider (as illustrated in section 3.3). The last property helps us to analyze the long-term abundances of each strategy. Because T is primitive for finite β, the Perron–Frobenius theorem implies that the likelihood u_ij to observe the two players in a given state (S _i , ${\mathbf{S}}_{j}^{\prime }$) converges as a function of time. Moreover, this likelihood can be determined by finding the unique solution of (5) in the main text

$$\begin{align}\hfill \mathbf{u}& =\mathbf{u}\,T\hfill \\ \hfill \mathbf{u}\,{\mathbf{e}}^{\top }& =1.\hfill \end{align} \tag{ A1 }$$

While (A1) provides an implicit characterization of the stationary distribution u, we can also provide an explicit representation.

Proposition 2 (An explicit representation of the stationary distribution). For $p\!\in \!\mathbb{N}$, let T be a row-stochastic and primitive p × p matrix, I denote the p × p identity matrix, and U = e^⊤ e denote the p × p matrix whose entries are all equal to 1. Then (I + U − T) is invertible, and the unique solution of (A1) can be given by u = e (I + U − T)⁻¹.

Proof. One proof of this result can be found in [70], where (I + U − T)⁻¹ is shown to be a fundamental matrix of the ergodic chain. In the following, we provide an independent proof.

Suppose u satisfies (A1). If we multiply the second equation in (A1) by e from the right, and subtract the result from the first equation, we obtain, after rearranging some of the terms

$$\begin{equation}\mathbf{u}\,(I+U-T)=\mathbf{e}.\end{equation} \tag{ A2 }$$

Thus, for the proposition to hold, we only need to verify that (I + U − T) is invertible. To this end, let λ₁, ..., λ_p be the eigenvalues of T and w₁, ..., w_p the corresponding right eigenvectors. Since T is row-stochastic it has an eigenvalue λ₁ = 1 with corresponding right eigenvector w₁ = (1, 1, ..., 1)^⊤. By T's primitivity, it follows that I − T has a unique eigenvalue equal to 0 with corresponding right eigenvector w₁, while 1 − λ_k ≠ 0 for k = 2, ..., p. Because $U={\mathbf{w}}_{1}{\mathbf{w}}_{1}^{\top }$, it follows by Brauer's theorem (example 1.2.8, p 51 of [93]), that the eigenvalues of (U + I − T) are ${\mathbf{w}}_{1}^{\top }{\mathbf{w}}_{1},1-{\lambda }_{2},\dots ,1-{\lambda }_{p}$. That is, in general the eigenvalues are the same as for the matrix I − T; only the eigenvalue corresponding to w₁ gets replaced by ${\mathbf{w}}_{1}^{\top }{\mathbf{w}}_{1}$. Since ${\mathbf{w}}_{1}^{\top }{\mathbf{w}}_{1}=p\! > \!0$ and 1 − λ_k ≠ 0 for k = 2, ..., p, the matrix (U + I − T) has no eigenvalue equal to 0. Therefore, it is invertible. □

While the stationary distribution u is well-defined for any finite β, in section 4.1 we also study which strategies are played as β approaches infinity. To this end, we first compute an expression for u(β) that is valid for finite β. Thereafter, we take the limit lim_β→∞  u(β) in ${\mathbb{R}}^{mn}$. For all considered 2 × 2 games, this yields a unique prediction of the strong selection limit—even for those games for which the respective limiting transition matrix lim_β→∞  T(β) allows for several absorbing states. One example is the stag-hunt game (see table 1). Here, introspection dynamics predicts that if (C, C) is risk-dominant, players coordinate on mutual cooperation. Similarly, if (D, D) is risk-dominant, players coordinate on mutual defection. In comparison, according to the limiting transition matrix lim_β→∞  T(β), both (C, C) and (D, D) are absorbing, irrespective of which one is risk-dominant. These results suggest that the strong selection limit of introspection dynamics can serve as an equilibrium selection device for 2 × 2 games, similar to other evolutionary dynamics [94, 95].

Here, we derive an explicit expression for the linear approximation of the stationary distribution when selection is weak, u ≈ u₀ + u₁ β. As we show in the main text, in (9) and (10), the constant term u₀ of the stationary distribution obeys

$$\begin{align}\hfill {\mathbf{u}}_{0}& ={\mathbf{u}}_{0}\,{T}_{0}\hfill \\ \hfill {\mathbf{u}}_{0}\,{\mathbf{e}}^{\top }& =1,\hfill \end{align} \tag{ B1 }$$

whereas the linear u₁ term satisfies

$$\begin{align}\hfill {\mathbf{u}}_{1}\,(I-{T}_{0})& ={\mathbf{u}}_{0}\,{T}_{1}\hfill \\ \hfill {\mathbf{u}}_{1}\,{\mathbf{e}}^{\top }& =0.\hfill \end{align} \tag{ B2 }$$

Both systems can be solved explicitly. For that, we first compute T₀ := T|_β=0 and T₁ := ∂T/∂β|_β=0, which are the constant and the linear term of the transition matrix. Based on (3), we obtain

$$\begin{equation}{({T}_{0})}_{ij,kl}=\!\begin{cases}\frac{1}{4(m\!-\!1)}\quad \hfill & \text{if}\,\,i\!\ne \!k\wedge j=l\hfill \\ \frac{1}{4(n\!-\!1)}\quad \hfill & \text{if}\,\,i=k\wedge j\!\ne \!l\hfill \\ 0\quad \hfill & \;\text{if}\;i\!\ne \!k\wedge j\!\ne \!l\hfill \\ \frac{1}{2}\quad \hfill & \;\text{if}\;i=k\wedge j=l\hfill \end{cases}\quad {({T}_{1})}_{ij,kl}=\!\begin{cases}\frac{{\pi }_{kj}-{\pi }_{ij}}{8(m-1)}\quad \hfill & \text{if}\,\,i\!\ne \!k\wedge j=l\hfill \\ \frac{{\pi }_{il}^{\prime }-{\pi }_{ij}^{\prime }}{8(n-1)}\quad \hfill & \text{if}\,\,i=k\wedge j\!\ne \!l\hfill \\ 0\quad \hfill & \;\text{if}\;i\!\ne \!k\wedge j\!\ne \!l\hfill \\ \sum\limits _{p=1}^{m}\frac{{\pi }_{ij}-{\pi }_{pj}}{8(m-1)}\!+\!\!\sum\limits _{q=1}^{n}\frac{{\pi }_{ij}^{\prime }-{\pi }_{iq}^{\prime }}{8(n-1)}\quad \hfill & \;\text{if}\;i=k\wedge j=l.\hfill \end{cases}\end{equation} \tag{ B3 }$$

In the special case that both players have the same number of strategies, the stationary distribution takes a particularly simple form, as the following result shows.

Proposition 3 (Weak selection approximation of the invariant distribution).

• (a)  
  The system of equations (B1) has a unique solution given by

  $$\begin{equation}{({\hat{\mathbf{u}}}_{0})}_{ij}=\frac{1}{mn}.\end{equation} \tag{ B4 }$$
• (b)  
  The system of equations (B2) has a unique solution. For m = n, it takes the simple form

  $$\begin{equation}{({\hat{\mathbf{u}}}_{1})}_{ij}=\frac{1}{2{n}^{4}}\,\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{ij}\!+\!{\pi }_{iq}\!-\!{\pi }_{pj}\!-\!{\pi }_{pq})+({\pi }_{ij}^{\prime }\!+\!{\pi }_{pj}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right).\end{equation} \tag{ B5 }$$

Proof. 

• (a)  
  To prove the first part, we note that T₀ is symmetric; indeed, swapping i with k and j with l in the expression for T₀ in (B3) shows that ${({T}_{0})}_{ij,kl}={({T}_{0})}_{kl,ij}$. The stationary distribution of a Markov chain with symmetric transition matrix is uniform [70]. Since T is of size mn, the result in (B4) immediately follows.
• (b)  
  To show that any solution to (B2) must be unique, we multiply the second equation in (B2) by the row-vector e from the right, and add the result to the first equation. This yields

  $$\begin{equation}{\mathbf{u}}_{1}\,(I+U-{T}_{0})={\mathbf{u}}_{0}\,{T}_{1},\end{equation} \tag{ B6 }$$

  where again U = e^⊤ e. By proposition 2, the matrix (I + U − T₀) is invertible, and hence any solution to (B2) is uniquely determined by ${\mathbf{u}}_{1}={\mathbf{u}}_{0}\,{T}_{1}{(I+U-{T}_{0})}^{-1}$.

It remains to show that for m = n, the vector ${\hat{\mathbf{u}}}_{1}$ as defined by (B5) is in fact a solution of (B2). To this end, we first compute the right-hand side of the first condition in (B2) by plugging in the definitions of ${\hat{\mathbf{u}}}_{0}$ according to (B4) and of T₁ according to (B3),

$$\begin{align}\hfill {({\hat{\mathbf{u}}}_{0}{T}_{1})}_{kl}& =\sum\limits _{i=1}^{n}\sum\limits _{j=1}^{n}{({\hat{\mathbf{u}}}_{0})}_{ij}{({T}_{1})}_{ij,kl}\stackrel{(\text{B}4)}{=}\sum\limits _{i=1}^{n}\sum\limits _{j=1}^{n}\frac{1}{{n}^{2}}{({T}_{1})}_{ij,kl}\hfill \\ \hfill & \stackrel{(\mathrm{B}\mathrm{3})}{=}\frac{1}{{n}^{2}}\left(\sum\limits _{i\ne k}\frac{{\pi }_{kl}-{\pi }_{il}}{8(n-1)}+\sum\limits _{j\ne l}\frac{{\pi }_{kl}^{\prime }-{\pi }_{kj}^{\prime }}{8(n-1)}+\sum\limits _{p=1}^{n}\frac{{\pi }_{kl}-{\pi }_{pl}}{8(n-1)}+\sum\limits _{q=1}^{n}\frac{{\pi }_{kl}^{\prime }-{\pi }_{kq}^{\prime }}{8(n-1)}\right)\hfill \\ \hfill & =\frac{1}{{n}^{2}}\left(\sum\limits _{p=1}^{n}\frac{{\pi }_{kl}-{\pi }_{pl}}{8(n-1)}+\sum\limits _{p=1}^{n}\frac{{\pi }_{kl}^{\prime }-{\pi }_{kp}^{\prime }}{8(n-1)}+\sum\limits _{p=1}^{n}\frac{{\pi }_{kl}-{\pi }_{pl}}{8(n-1)}+\sum\limits _{p=1}^{n}\frac{{\pi }_{kl}^{\prime }-{\pi }_{kp}^{\prime }}{8(n-1)}\right)\hfill \\ \hfill & =\frac{1}{4(n\!-\!1){n}^{2}}\,\sum\limits _{p=1}^{n}\!\left(({\pi }_{kl}\!-\!{\pi }_{pl})+({\pi }_{kl}^{\prime }\!-\!{\pi }_{kp}^{\prime })\right).\hfill \end{align} \tag{ B7 }$$

Similarly, we can compute the left-hand side of the first condition in (B2). By setting m = n and using the definition of T₀ according to (B3), we obtain

$$\begin{align}\hfill {\left({\hat{\mathbf{u}}}_{1}(I\!-\!{T}_{0})\right)}_{\!\,kl}& =\sum\limits _{i=1}^{n}\sum\limits _{j=1}^{n}{({\hat{\mathbf{u}}}_{1})}_{ij}{(I-{T}_{0})}_{ij,kl}\hfill \\ \hfill & \stackrel{(\mathrm{B}\mathrm{3})}{=}\frac{{({\hat{\mathbf{u}}}_{1})}_{kl}}{2}-\sum\limits _{i\ne k}\frac{{({\hat{\mathbf{u}}}_{1})}_{il}}{4(n\!-\!1)}-\sum\limits _{j\ne l}\frac{{({\hat{\mathbf{u}}}_{1})}_{kj}}{4(n\!-\!1)}\hfill \\ \hfill & =\frac{{({\hat{\mathbf{u}}}_{1})}_{kl}}{2}-\left(\sum\limits _{i=1}^{n}\frac{{({\hat{\mathbf{u}}}_{1})}_{il}}{4(n\!-\!1)}-\frac{{({\hat{\mathbf{u}}}_{1})}_{kl}}{4(n\!-\!1)}\right)-\left(\sum\limits _{j=1}^{n}\frac{{({\hat{\mathbf{u}}}_{1})}_{kj}}{4(n\!-\!1)}-\frac{{({\hat{\mathbf{u}}}_{1})}_{kl}}{4(n\!-\!1)}\right)\hfill \\ \hfill & =\frac{n}{2(n\!-1)}{({\hat{\mathbf{u}}}_{1})}_{kl}-\sum\limits _{i=1}^{n}\frac{{({\hat{\mathbf{u}}}_{1})}_{il}}{4(n\!-\!1)}-\sum\limits _{j=1}^{n}\frac{{({\hat{\mathbf{u}}}_{1})}_{kj}}{4(n\!-\!1)}.\hfill \end{align} \tag{ B8 }$$

In a next step, we plug in the definition of ${\hat{\mathbf{u}}}_{1}$ according to (B5), yielding

$$\begin{align}\hfill {\left({\hat{\mathbf{u}}}_{1}(I\!-\!{T}_{0})\right)}_{\!\,kl}& =\frac{1}{4(n\!-1){n}^{3}}\,\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kl}\!+\!{\pi }_{kq}\!-\!{\pi }_{pl}\!-\!{\pi }_{pq})+({\pi }_{kl}^{\prime }\!+\!{\pi }_{pl}^{\prime }\!-\!{\pi }_{kq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & \quad -\frac{1}{8(n\!-\!1){n}^{4}}\sum\limits _{i=1}^{n}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{il}\!+\!{\pi }_{iq}\!-\!{\pi }_{pl}\!-\!{\pi }_{pq})+({\pi }_{il}^{\prime }\!+\!{\pi }_{pl}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & \quad -\frac{1}{8(n\!-\!1){n}^{4}}\sum\limits _{j=1}^{n}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kj}\!+\!{\pi }_{kq}\!-\!{\pi }_{pj}\!-\!{\pi }_{pq})+({\pi }_{kj}^{\prime }\!+\!{\pi }_{pj}^{\prime }\!-\!{\pi }_{kq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right).\hfill \end{align} \tag{ B9 }$$

We can further simplify this expression by rewriting the sums in the second and third line,

$$\begin{align}\hfill {\left({\hat{\mathbf{u}}}_{1}(I\!-\!{T}_{0})\right)}_{\!\,kl}& =\frac{1}{4(n\!-1){n}^{3}}\,\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kl}\!+\!{\pi }_{kq}\!-\!{\pi }_{pl}\!-\!{\pi }_{pq})+({\pi }_{kl}^{\prime }\!+\!{\pi }_{pl}^{\prime }\!-\!{\pi }_{kq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & \quad -\frac{1}{8(n\!-\!1){n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{pl}\!+\!{\pi }_{pq}\!-\!{\pi }_{pl}\!-\!{\pi }_{pq})+({\pi }_{pl}^{\prime }\!+\!{\pi }_{pl}^{\prime }\!-\!{\pi }_{pq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & \quad -\frac{1}{8(n\!-\!1){n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kq}\!+\!{\pi }_{kq}\!-\!{\pi }_{pq}\!-\!{\pi }_{pq})+({\pi }_{kq}^{\prime }\!+\!{\pi }_{pq}^{\prime }\!-\!{\pi }_{kq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right).\hfill \end{align} \tag{ B10 }$$

Now, the second and the third sum can be further simplified naturally. Then, by collecting like terms across the three sums, we eventually obtain a simple result,

$$\begin{align}\hfill {\left({\hat{\mathbf{u}}}_{1}(I\!-\!{T}_{0})\right)}_{\!\,kl}& =\frac{1}{4(n\!-1){n}^{3}}\,\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kl}\!+\!{\pi }_{kq}\!-\!{\pi }_{pl}\!-\!{\pi }_{pq})+({\pi }_{kl}^{\prime }\!+\!{\pi }_{pl}^{\prime }\!-\!{\pi }_{kq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & \quad -\frac{1}{4(n\!-\!1){n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{pl}^{\prime }\!-\!{\pi }_{pq}^{\prime })\enspace -\enspace \frac{1}{4(n\!-\!1){n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{kq}\!-\!{\pi }_{pq})\hfill \\ \hfill & =\frac{1}{4(n\!-1){n}^{3}}\,\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{kl}\!-\!{\pi }_{pl})+({\pi }_{kl}^{\prime }\!-\!{\pi }_{kq}^{\prime })\right)\hfill \\ \hfill & =\frac{1}{4(n\!-1){n}^{2}}\,\sum\limits _{p=1}^{n}({\pi }_{kl}\!-\!{\pi }_{pl})+({\pi }_{kl}^{\prime }\!-\!{\pi }_{kp}^{\prime }).\hfill \end{align} \tag{ B11 }$$

This expression for ${\left({\hat{\mathbf{u}}}_{1}(I\!-\!{T}_{0})\right)}_{\!kl}$ coincides with the expression we obtained for ${\left({\hat{\mathbf{u}}}_{0}{T}_{1}\right)}_{\!kl}$ in (B7). Thus, the first condition in (B2) is satisfied. To verify the second condition in (B2), we compute

$$\begin{align}\hfill ({\hat{\mathbf{u}}}_{1})\!\cdot \!{\mathbf{e}}^{\top }& =\frac{1}{2{n}^{4}}\sum\limits _{i=1}^{n}\sum\limits _{j=1}^{n}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{ij}\!+\!{\pi }_{iq}\!-\!{\pi }_{pj}\!-\!{\pi }_{pq})+({\pi }_{ij}^{\prime }\!+\!{\pi }_{pj}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\hfill \\ \hfill & =\frac{1}{2{n}^{2}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{pq}\!+\!{\pi }_{pq}\!-\!{\pi }_{pq}\!-\!{\pi }_{pq})+({\pi }_{pq}^{\prime }\!+\!{\pi }_{pq}^{\prime }\!-\!{\pi }_{pq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)=0.\hfill \end{align} \tag{ B12 }$$

□

Next, we derive (14) in the main text. For a given stationary distribution u, we define the respective marginal distributions by ${\xi }_{i}{:=}{\sum }_{j=1}^{n}{(\mathbf{u})}_{ij}$ and ${\xi }_{j}^{\prime }{:=}{\sum }_{i=1}^{m}{(\mathbf{u})}_{ij}$. Using proposition 3 we can derive the following weak selection formulas for ξ_i and ${\xi }_{j}^{\prime }$ when the two players have the same number of strategies (m = n).

Corollary 1 (Marginal distributions for weak selection). For m = n and small β, the abundance of the strategies S_i and ${\mathbf{S}}_{j}^{\prime }$ is

$$\begin{align}\hfill {\xi }_{i}=\frac{1}{n}+\frac{\beta }{{n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{iq}\!-\!{\pi }_{pq})+\mathcal{O}({\beta }^{2}),\\ \hfill {\xi }_{j}^{\prime }=\frac{1}{n}+\frac{\beta }{{n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{pj}^{\prime }\!-\!{\pi }_{pq}^{\prime })+\mathcal{O}({\beta }^{2}).\end{align} \tag{ B13 }$$

Proof. To obtain the first equation, we use the weak selection formula in proposition 3 and sum up over all co-player's strategies ${\mathbf{S}}_{j}^{\prime }$,

$$\begin{align}\hfill {\xi }_{i}& =\sum\limits _{j=1}^{n}{({\mathbf{u}}_{0})}_{ij}+\left(\sum\limits _{j=1}^{n}{({\mathbf{u}}_{1})}_{ij}\right)\beta +\mathcal{O}({\beta }^{2})\hfill \\ \hfill & =\sum\limits _{j=1}^{n}\frac{1}{{n}^{2}}+\frac{1}{2{n}^{4}}\sum\limits _{j=1}^{n}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{ij}\!+\!{\pi }_{iq}\!-\!{\pi }_{pj}\!-\!{\pi }_{pq})+({\pi }_{ij}^{\prime }\!+\!{\pi }_{pj}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\beta +\mathcal{O}({\beta }^{2})\hfill \\ \hfill & =\frac{1}{n}+\frac{1}{2{n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}\left(({\pi }_{iq}\!+\!{\pi }_{iq}\!-\!{\pi }_{pq}\!-\!{\pi }_{pq})+({\pi }_{iq}^{\prime }\!+\!{\pi }_{pq}^{\prime }\!-\!{\pi }_{iq}^{\prime }\!-\!{\pi }_{pq}^{\prime })\right)\beta +\mathcal{O}({\beta }^{2})\hfill \\ \hfill & =\frac{1}{n}+\frac{\beta }{{n}^{3}}\sum\limits _{p=1}^{n}\sum\limits _{q=1}^{n}({\pi }_{iq}\!-\!{\pi }_{pq})+\mathcal{O}({\beta }^{2}).\hfill \end{align} \tag{ B14 }$$

The analogous formula for ${\xi }_{j}^{\prime }$ follows by symmetry. □

Three remarks are in order.

• (a)  
  Independence of the co-player's payoffs. According to corollary 1, the abundance of a player's strategy in the limit of weak selection only depends on that player's payoffs. For instance, in (B13), the abundance ξ_i depends on the first player's payoffs π_pq , but it is independent of the second player's payoffs ${\pi }_{pq}^{\prime }$. While such a result may appear intuitive, it is important to note that this result only holds in the limit of weak selection. As an example, consider the game with 2 × 2 payoff matrix

  

  For this game, (B13) implies that for sufficiently small β, the marginal abundance of ${\mathbf{S}}_{1}$ is approximately given by ${\xi }_{1}\!=\!\frac{1}{2}\!+\!\beta \frac{3}{8}$, irrespective of the value of x. However, in the limit of strong selection, β → ∞, one can show that ξ₁ = 1 if x is positive, but ξ₁ = 0 if x is negative. Therefore, the co-player's payoffs do in general affect how likely a player is to adopt a certain strategy. Only when selection is weak (such that each co-player's strategy is played with approximately equal frequency), this dependency disappears.
• (b)  
  Comparing the abundance of different strategies. Corollary 1 also allows us to rank the different strategies of a player according to how often they are played in the stationary distribution. To this end, we say strategy S_i is favored to S_k , and we write S_i ≻ S_k , if ξ_i > ξ_k . An analogous notation can be defined for the column player. By corollary 1 we find that when selection is weak, then S_i ≻ S_k if and only if $\sum _{q=1}^{n}{\pi }_{iq}\! > \!\sum _{q=1}^{n}{\pi }_{kq}$. Again, the respective condition only depends on the focal player's payoffs π_pq and not on the co-player's payoffs ${\pi }_{pq}^{\prime }$. However, as before, this independence vanishes for stronger selection (which can be shown with the same example (B15)).
• (c)  
  Comparative statics. Finally, we can use corollary 1 to compute how changes in the players' payoffs affect the resulting strategy abundances in the limit of weak selection. By (B13), we note that ∂ξ_i /∂π_pq is positive if p = i, while it is negative if p ≠ i. Hence, increasing any one of the payoffs π_iq has a positive effect on the long-run abundance of strategy S_i , whereas increasing any one of the other payoffs π_pq with p ≠ i has a negative effect.

In the previous section, we have derived a linear approximation for the long run abundance of each strategy profile $({\mathbf{S}}_{i},{\mathbf{S}}_{j}^{\prime })$ under weak selection. Using similar techniques, we can obtain arbitrary higher-order approximations. To this end, we define the kth derivative of v and T, respectively, evaluated at β = 0,

$$\begin{equation}{\mathbf{u}}_{k}{:=}{\left.\frac{{\partial }^{k}\mathbf{u}}{\partial {\beta }^{k}}\right\vert }_{\beta =0}\enspace \enspace \enspace \text{and}\enspace \enspace \enspace {\left.{T}_{k}{:=}\frac{{\partial }^{k}T}{\partial {\beta }^{k}}\right\vert }_{\beta =0}.\end{equation} \tag{ C1 }$$

By computing the respective higher-order expansion of (7) and comparing coefficients with respect to the power β^k , we obtain for k ⩾ 1

$$\begin{align}\hfill {\mathbf{u}}_{k}& =\sum\limits _{j=0}^{k}\left(\genfrac{}{}{0.0pt}{}{k}{j}\right){\mathbf{u}}_{j}\,{T}_{k-j}\hfill \\ \hfill {\mathbf{u}}_{k}\,{\mathbf{e}}^{\top }& =0.\hfill \end{align} \tag{ C2 }$$

By the first equation, it follows that we can compute u_k recursively from all lower terms u_j with j < k as

$$\begin{equation}{\mathbf{u}}_{k}(I-{T}_{0})=\sum\limits _{j=0}^{k-1}\left(\genfrac{}{}{0.0pt}{}{k}{j}\right){\mathbf{u}}_{j}\,{T}_{k-j}.\end{equation} \tag{ C3 }$$

By multiplying the second equation in (C2) by the row-vector e from the right, and adding the result to (C3), we obtain

$$\begin{equation}{\mathbf{u}}_{k}(I+U-{T}_{0})=\sum\limits _{j=0}^{k-1}\left(\genfrac{}{}{0.0pt}{}{k}{j}\right){\mathbf{u}}_{j}{T}_{k-j}.\end{equation} \tag{ C4 }$$

Since (I + U − T₀) is invertible by proposition 2, we obtain the explicit solution

$$\begin{equation}{\mathbf{u}}_{k}=\left(\sum\limits _{j=0}^{k-1}\left(\genfrac{}{}{0.0pt}{}{k}{j}\right){\mathbf{u}}_{j}{T}_{k-j}\right)\cdot {(U\!+\!I\!-\!{T}_{0})}^{-1}.\end{equation} \tag{ C5 }$$

For introspection dynamics, the first factor on the right-hand side can be further simplified. We note that T_i is the zero matrix for all even i ⩾ 2, as the following result shows.

Proposition 4. For T_i defined as in (C1), T_i = 0 for any even integer i ⩾ 2.

Proof. By its definition (3) in the main text, we can write all entries of the transition matrix T as a linear combination of constant terms and functions of the form ${\varphi }_{\beta }({\Delta}\pi )={(1+{\mathrm{e}}^{-\beta {\Delta}\pi })}^{-1}$. e^x Since, by taking the kth derivative of (2) in the main text,

$$\begin{equation}\frac{{\partial }^{k}{\varphi }_{\beta }}{\partial {\beta }^{k}}=\frac{{(-{\Delta}\pi )}^{k}\!\cdot \!{\mathrm{e}}^{-\beta {\Delta}\pi }}{{(1+{\mathrm{e}}^{-\beta {\Delta}\pi })}^{k+1}}\sum\limits _{l=0}^{k}{(-1)}^{l+1}\!\cdot \!E(k,l)\cdot {\mathrm{e}}^{-l\beta \cdot {\Delta}\pi },\quad k\!\geqslant \!1\end{equation} \tag{ C6 }$$

the entries of T_k are proportional to

$$\begin{equation}{\left.\frac{{\partial }^{k}{\varphi }_{\beta }}{\partial {\beta }^{k}}\right\vert }_{\beta =0}=\frac{-{({\Delta}\pi )}^{k}}{{2}^{k+1}}\sum\limits _{l=0}^{k}{(-1)}^{l+1}\!\cdot \!E(k,l),\end{equation} \tag{ C7 }$$

where E(k, l) are the Eulerian numbers defined by

$$\begin{equation*}E(k,l)=\sum\limits _{r=0}^{l}{(-1)}^{r}\left(\genfrac{}{}{0.0pt}{}{k+1}{r}\right){(l+1-r)}^{k}.\end{equation*}$$

For 1 ⩽ k ⩽ 5 and 0 ⩽ l ⩽ 5, these numbers are tabulated in table 2. A useful property of Eulerian numbers is the symmetry E(k, l) = E(k, k − l − 1). The sum in (C7) translates into adding the numbers of the kth line of the table with alternating signs. For even k, that sum is zero because the different Eulerian numbers exactly cancel each other out due to their symmetry. □

As a result, the first seven terms of the Taylor expansion of the stationary distribution u are

$$\begin{equation}\begin{array}{cccc}{\mathbf{u}}_{0}=& \;\mathbf{e}\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{1}=& \;({\mathbf{u}}_{0}{T}_{1})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{2}=& \;(2{\mathbf{u}}_{1}{T}_{1})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{3}=& \;(3{\mathbf{u}}_{2}{T}_{1}+{\mathbf{u}}_{0}{T}_{3})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{4}=& \;(4{\mathbf{u}}_{3}{T}_{1}+4{\mathbf{u}}_{1}{T}_{3})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{5}=& \;(5{\mathbf{u}}_{4}{T}_{1}+10{\mathbf{u}}_{2}{T}_{3}+{\mathbf{u}}_{0}{T}_{5})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}\\ {\mathbf{u}}_{6}=& \;(6{\mathbf{u}}_{5}{T}_{1}+20{\mathbf{u}}_{3}{T}_{3}+6{\mathbf{u}}_{1}{T}_{5})\;{(I\phantom{\rule{0ex}{0ex}}+\phantom{\rule{0ex}{0ex}}U\phantom{\rule{0ex}{0ex}}-\phantom{\rule{0ex}{0ex}}{T}_{0})}^{-1}.\end{array}\end{equation} \tag{ C8 }$$

Pairwise imitation dynamics is a frequency-dependent update rule where players can imitate other players' strategies [24]. The dynamics takes place in an entire population of size Z. For symmetric games with two strategies, as considered in section 4.2, the state of the population can be defined by the number of individuals i who use strategy C (the number of individuals who use strategy D is then Z − i). Given the current population state i, one can calculate the expected payoffs of cooperators and defectors, respectively, as

$$\begin{align}& {f}_{\mathbf{C}}=\frac{i-1}{Z-1}{\pi }_{\mathbf{C}\mathbf{C}}+\frac{Z-i}{Z-1}{\pi }_{\mathbf{C}\mathbf{D}}\\ & {f}_{\mathbf{D}}=\frac{i}{Z-1}{\pi }_{\mathbf{D}\mathbf{C}}+\frac{Z-i-1}{Z-1}{\pi }_{\mathbf{D}\mathbf{D}}.\end{align} \tag{ D1 }$$

Similar to introspection dynamics, pairwise imitation dynamics assumes that at regular time intervals, a randomly chosen player is given the opportunity to revise their strategy. With probability μ (the mutation rate), this player simply adopts a randomly chosen strategy. With probability 1 − μ, the player instead picks a randomly chosen role model from the population. If the focal player's payoff is π and the role model's payoff is $\tilde{\pi }$, the focal player adopts the role model's strategy with probability ${\varphi }_{\beta }(\tilde{\pi }-\pi )$, where φ_β is again the Fermi function defined by (2).

This process can also be described by a Markov chain. Given that the current population state is i, the probability that a D-player changes to C in the next time step is

$$\begin{equation}{T}_{i,i+1}=(1-\mu )\left[\frac{Z-i}{Z}\;\frac{i}{Z-1}\;\frac{1}{1+{\mathrm{e}}^{-\beta ({f}_{\mathbf{C}}-{f}_{\mathbf{D}})}}\right]+\mu \frac{Z-i}{Z}.\end{equation} \tag{ D2a }$$

Similarly, the probability that a C-player changes to D is

$$\begin{equation}{T}_{i,i-1}=(1-\mu )\left[\frac{i}{Z}\;\frac{Z-i}{Z-1}\;\frac{1}{1+{\mathrm{e}}^{-\beta ({f}_{\mathbf{D}}-{f}_{\mathbf{C}})}}\right]+\mu \frac{i}{Z}.\end{equation} \tag{ D2b }$$

Since only one player is allowed to update at each time step,

$$\begin{equation}{T}_{i,j}=0\quad \text{for}\,\text{all}\enspace j\notin \left\{i-1,i,i+1\right\}.\end{equation} \tag{ D2c }$$

Finally, for the normalization condition to hold, the probability of remaining in the same state is

$$\begin{equation}{T}_{i,i}=1-\sum\limits _{{i}^{\mathrm{\prime }}\ne i}{T}_{i,{i}^{\mathrm{\prime }}}.\end{equation} \tag{ D2d }$$

As a result, the transition matrix is

$$\begin{equation*}\left[\begin{matrix} {T}_{0,0} & {T}_{0,1} & 0 & 0 & 0 & \dots & 0 & 0 \\ {T}_{1,0} & {T}_{1,1} & {T}_{1,2} & 0 & 0 & \dots & 0 & 0 \\ 0 & {T}_{2,1} & {T}_{2,2} & {T}_{2,3} & 0 & \dots & 0 & 0 \\ {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & \ddots & {\vdots} & {\vdots} \\ 0 & 0 & 0 & 0 & 0 & \dots & {T}_{Z,Z-1} & {T}_{Z,Z} \end{matrix}\right].\end{equation*}$$

As before, we can characterize the long-run dynamics of this process by computing the stationary distribution u = (u₀, u₁, ..., u_Z ). In this case, however, u_i denotes the prevalence of a population state with i cooperators and Z − i defectors.

To construct a metric that is comparable to the one used for introspection dynamics, we compute the average fraction of cooperation,

$$\begin{equation}{p}_{\mathbf{C}}=\sum\limits _{i=0}^{Z}{u}_{i}\frac{i}{Z}\end{equation} \tag{ D3 }$$

which we plot in figure 4(b). Additionally, one can also obtain the average probability of drawing CC, CD, DC, and DD pairs from the population

$$\begin{equation}{p}_{\mathbf{CC}}=\sum\limits _{i=0}^{Z}{u}_{i}\,\frac{i}{Z}\,\frac{i-1}{Z-1}\end{equation} \tag{ D4a }$$

$$\begin{equation}{p}_{\mathbf{CD}}={p}_{\mathbf{DC}}=\sum\limits _{i=0}^{Z}{u}_{i}\,\frac{i}{Z}\,\frac{Z-i}{Z-1}\end{equation} \tag{ D4b }$$

$$\begin{equation}{p}_{\mathbf{DD}}=\sum\limits _{i=0}^{Z}{u}_{i}\,\frac{Z-i}{Z}\,\frac{Z-i-1}{Z-1}.\end{equation} \tag{ D4c }$$

Again, p_CD = p_DC because players are indistinguishable in the symmetric case. These quantities are comparable to the state frequencies of our stationary distribution, u = (u_CC , u_CD , u_DC , u_DD ).