Exploitation by asymmetry of information reference in coevolutionary learning in prisoner's dilemma game

Before formulating the learning process, we mathematically define the strategy and class. Recall that a single game can have one of four results: CC, CD, DC, and DD. First, a player using a memory-one strategy can refer to their own and the opponent's previous actions and respond with a different action to each result of the previous game. Thus, the player has four independent stochastic variables x₁, x₂, x₃, and x₄ as the probabilities of choosing C regardless of the outcome of the previous game. Thus, the memory-one class is defined as the possible set of such memory-one strategies, which is denoted as {x₁, x₂, x₃, x₄} ∈ [0,1]⁴. The TFT and WSLS [18, 44–46] strategies are examples of the emergence of cooperation as x₁ = x₃ = 1, x₂ = x₄ = 0, and x₁ = x₄ = 1, x₂ = x₃ = 0, respectively.

Second, a player using the reactive strategy can only refer to the opponent's action and therefore cannot distinguish between CC and DC (CD and DD). Thus, the strategy is given by two independent variables x₁₃ and x₂₄, where the former (latter) is the probability of choosing C where the previous result was either CC or DC (CD or DD). Therefore, the reactive class is defined as {x₁₃, x₂₄} ∈ [0,1]². Here, the notation x₁₃ (variable for CC and DC) clearly indicates the integration of x₁ (CC) and x₃ (DC) from the memory-one class. Indeed, all the strategies in the reactive class are included in the memory-one class, as one can set x₁ = x₃ = x₁₃ and x₂ = x₄ = x₂₄ for all x₁₃ and x₂₄. Thus, the above TFT strategy can be represented as x₁₃ = 1 and x₂₄ = 0, whereas the WSLS strategy cannot be represented. This makes it clear that the memory-one class is more complex than the reactive one, because the former can include all the strategies of the latter. The ordering of complexity can be defined as a class that includes all the strategies of the other is more complex.

Third, in the classical mixed strategy [47], a player stochastically chooses their action without referencing any actions from the previous game. Thus, the strategy controls only one variable x₁₂₃₄ ∈ [0, 1], which is the probability of choosing C. This class is the least complex of the three classes.

In this section, we analyze a infinitely repeated game under the condition that the strategy of both players are fixed. We define $\mathbf{p}{:=}{({p}_{\text{CC}},{p}_{\text{CD}},{p}_{\text{DC}},{p}_{\text{DD}})}^{\text{T}}$ as the probabilities that (CC, CD, DC, DD) are played in the present round, and the result of the next round is calculated by p' = Mp with

$$\begin{equation}\mathbf{M}{:=}\left(\begin{matrix}\hfill {x}_{1}{y}_{1}\hfill & \hfill {x}_{2}{y}_{3}\hfill & \hfill {x}_{3}{y}_{2}\hfill & \hfill {x}_{4}{y}_{4}\hfill \\ \hfill {x}_{1}\bar{{y}_{1}}\hfill & \hfill {x}_{2}\bar{{y}_{3}}\hfill & \hfill {x}_{3}\bar{{y}_{2}}\hfill & \hfill {x}_{4}\bar{{y}_{4}}\hfill \\ \hfill \bar{{x}_{1}}{y}_{1}\hfill & \hfill \bar{{x}_{2}}{y}_{3}\hfill & \hfill \bar{{x}_{3}}{y}_{2}\hfill & \hfill \bar{{x}_{4}}{y}_{4}\hfill \\ \hfill \bar{{x}_{1}}\bar{{y}_{1}}\hfill & \hfill \bar{{x}_{2}}\bar{{y}_{3}}\hfill & \hfill \bar{{x}_{3}}\bar{{y}_{2}}\hfill & \hfill \bar{{x}_{4}}\bar{{y}_{4}}\hfill \end{matrix}\right).\end{equation} \tag{ 1 }$$

When none of the strategy variables, x_n and y_n for n ∈ {1, ..., 4} are 0 or 1, the infinitely repeated game has only one equilibrium state ${\mathbf{p}}_{\text{e}}{:=}{({p}_{\text{CCe}},{p}_{\text{CDe}},{p}_{\text{DCe}},{p}_{\text{DDe}})}^{\text{T}}$. Here, we can directly compute p_e as

$$\begin{equation}\begin{matrix}\hfill {p}_{\text{CCe}}=k\left\{({x}_{4}+({x}_{3}-{x}_{4}){y}_{2})({y}_{4}+({y}_{3}-{y}_{4}){x}_{2})-{x}_{3}{y}_{3}({x}_{2}-{x}_{4})({y}_{2}-{y}_{4})\right\},\hfill \\ \hfill {p}_{\text{CDe}}=k\left\{({x}_{4}+({x}_{3}-{x}_{4}){y}_{4})(\bar{{y}_{2}}-({y}_{1}-{y}_{2}){x}_{1})-{x}_{4}\bar{{y}_{1}}({x}_{1}-{x}_{3})({y}_{2}-{y}_{4})\right\},\hfill \\ \hfill {p}_{\text{DCe}}=k\left\{(\bar{{x}_{2}}-({x}_{1}-{x}_{2}){y}_{1})({y}_{4}+({y}_{3}-{y}_{4}){x}_{4})-\bar{{x}_{1}}{y}_{4}({x}_{2}-{x}_{4})({y}_{1}-{y}_{3})\right\},\hfill \\ \hfill {p}_{\text{DDe}}=k\left\{(\bar{{x}_{2}}-({x}_{1}-{x}_{2}){y}_{3})(\bar{{y}_{2}}-({y}_{1}-{y}_{2}){x}_{3})-\bar{{x}_{2}}\bar{{y}_{2}}({x}_{1}-{x}_{3})({y}_{1}-{y}_{3})\right\}.\hfill \end{matrix}\end{equation} \tag{ 2 }$$

Here the coefficient k is determined by the normalization of the probabilities p_CCe + p_CDe + p_DCe + p_DDe = 1.

We next consider adaptive learning from past experiences of repeated games. For instance, we assume that the probability of CC being the result of the previous round is p_CCe. Then, the next action being C (D) would have the probability of x₁ ($\bar{{x}_{1}}$). Here, we define u_CC(C) (u_CC(D)) as the benefit that the player gains by performing action C (D). First, the time evolution of x₁ is assumed to depend on the amount of experience: the previous game's result and the action in the present one must be CC and C, respectively. Thus, $\dot{{x}_{1}}$ is proportional to p_CCe x₁. Second, $\dot{{x}_{1}}$ also depends on the benefit of action C and thus is proportional to ${u}_{\text{CC}(\text{C})}-({x}_{1}{u}_{\text{CC}(\text{C})}+\bar{{x}_{1}}{u}_{\text{CC}(\text{D})})$. To summarize, we get

$$\begin{equation}\dot{{x}_{1}}={x}_{1}\bar{{x}_{1}}{p}_{\text{CCe}}({u}_{\text{CC}(\text{C})}-{u}_{\text{CC}(\text{D})}).\end{equation} \tag{ 3 }$$

Next, we compute u_CC(C) and u_CC(D). When the previous game's result and the present self-action are CC and C, respectively, the present state is given by $\mathbf{p}={\mathbf{p}}_{\text{CC}(\text{C})}{:=}{({y}_{1},\bar{{y}_{1}},0,0)}^{\text{T}}$. If p_CC(C) ≠ p_e holds, then the state gradually relaxes to equilibrium with the repetition of the game. Thus, u_CC(C) is the total payoff generated by p_CC(C) until equilibrium is reached, which is given by

$$\begin{equation}{u}_{\text{CC}(\text{C})}=\sum\limits _{t=0}^{\infty }{\mathbf{M}}^{t}({\mathbf{p}}_{\text{CC}(\text{C})}-{\mathbf{p}}_{\text{e}})\cdot \mathbf{u}.\end{equation} \tag{ 4 }$$

Here, we define u := (R,S,T,P)^T as the vector for the payoff matrix. By contrast, when the previous game's result and the present self-action are CC and D, respectively, the present state is given by $\mathbf{p}={\mathbf{p}}_{\text{CC}(\text{D})}{:=}{(0,0,{y}_{1},\bar{{y}_{1}})}^{\text{T}}$. Then, u_CC(D) is computed in the same way using

$$\begin{equation}{u}_{\text{CC}(\text{D})}=\sum\limits _{t=0}^{\infty }{\mathbf{M}}^{t}({\mathbf{p}}_{\text{CC}(\text{D})}-{\mathbf{p}}_{\text{e}})\cdot \mathbf{u}.\end{equation} \tag{ 5 }$$

By substituting equations (4) and (5) into equation (3), we can write the learning dynamics of x₁ as

$$\begin{equation}\dot{{x}_{1}}={x}_{1}\bar{{x}_{1}}{p}_{\text{CCe}}\sum\limits _{t=0}^{\infty }{\mathbf{M}}^{t}({\mathbf{p}}_{\text{CC}(\text{C})}-{\mathbf{p}}_{\text{CC}(\text{D})})\cdot \mathbf{u},\end{equation} \tag{ 6 }$$

using only strategy variables, x_n and y_n for n ∈ {1, ..., 4}, and the payoff variables (T, R, P, S). Similarly, we can derive the time evolution of the other strategy variables x₂, x₃, and x₄.

Equation (6) appears complicated at first glance, but it can be simplified as

$$\begin{equation}\dot{{x}_{1}}={x}_{1}\bar{{x}_{1}}\frac{\partial {\mathbf{p}}_{\text{e}}}{\partial {x}_{1}}\cdot \mathbf{u},\end{equation} \tag{ 7 }$$

(see the proof in appendix A). The same equations hold for the learning of x₂, x₃, and x₄, and for opponent player. Notably, this equation reproduces the original coupled replicator model [40, 41].

In the previous sections, we formulated the learning dynamics of memory-one class strategies against another within the same class. In this section, we consider other cases in which both learned and learning players adopt the reactive class.

First, we consider a case in which a learned player uses the reactive class, and the learning player uses the memory-one class. In this case, the learning is easily given by

$$\begin{equation}\dot{{x}_{n}}(\left\{{x}_{1},\dots ,{x}_{4}\right\},\left\{{y}_{13},{y}_{24}\right\})=\dot{{x}_{i}}{\vert }_{{y}_{1}={y}_{3}={y}_{13},{y}_{2}={y}_{4}={y}_{24}},\end{equation} \tag{ 8 }$$

for n ∈ {1, ..., 4} because the learned reactive player's strategy is constrained by y₁ = y₃ = y₁₃ and y₂ = y₄ = y₂₄.

Second, we consider a case in which the learning player uses the reactive class, and the learned player uses the memory-one class. In this case, the learning player's strategy is given by (x₁₃, x₂₄). Recall that the learning speed of our model depends on the amount of experience. Because the frequency of observing the opponent's previous action, C, is the total of both CC and DC, the time evolution of x₁₃ is the sum of x₁ and x₃. Thus, we get

$$\begin{equation}\begin{matrix}\hfill \dot{{x}_{13}}(\left\{{x}_{13},{x}_{24}\right\},\left\{{y}_{1},\dots ,{y}_{4}\right\})=(\dot{{x}_{1}}+\dot{{x}_{3}}){\vert }_{{x}_{1}={x}_{3}={x}_{13},{x}_{2}={x}_{4}={x}_{24}},\hfill \\ \hfill \dot{{x}_{24}}(\left\{{x}_{13},{x}_{24}\right\},\left\{{y}_{1},\dots ,{y}_{4}\right\})=(\dot{{x}_{2}}+\dot{{x}_{4}}){\vert }_{{x}_{1}={x}_{3}={x}_{13},{x}_{2}={x}_{4}={x}_{24}}.\hfill \end{matrix}\end{equation} \tag{ 9 }$$

Before investigating the game between the memory-one and reactive classes, we first study the learning of the memory-one and reactive classes against the other fixed strategy. Figure 2 shows the time series of each strategy's payoff based on the learning dynamics. The payoffs of both classes monotonically increase their payoffs over time because the opponent's strategy is fixed. However, there are two major differences between the two classes in the way the payoff increases.

Zoom In Zoom Out Reset image size

First, the reactive class learns faster than the memory-one class. This is because the reactive class is a compressed version of the memory-one model, as the constraints x₁ = x₃ and x₂ = x₄ are postulated, that is, the learning in the cases of CC and DC (CD and DD) are integrated. Recall that the change in strategy is optimized based on the empirical data sampled through the played games. In the reactive class, the number of strategy variables is fewer; therefore, quick optimization can be achieved, as shown in equation (9).

Second, the memory-one strategy gains a larger payoff in equilibrium than the reactive one. This is simply because the memory-one strategy contains a reactive strategy. Accordingly,

$$\begin{equation}\underset{{x}_{1},{x}_{2},{x}_{3},{x}_{4}}{\mathrm{max}}\enspace {\mathbf{p}}_{\text{e}}(\left\{{x}_{1},{x}_{2},{x}_{3},{x}_{4}\right\},\mathbf{y})\cdot \mathbf{u}\geqslant \underset{{x}_{13},{x}_{24}}{\mathrm{max}}\enspace {\mathbf{p}}_{\text{e}}(\left\{{x}_{13},{x}_{24}\right\},\mathbf{y})\cdot \mathbf{u},\end{equation} \tag{ 10 }$$

is derived for all the opponent's fixed strategies y.

In section 3.1, we considered one-sided learning, where a player dynamically optimizes the strategy against their opponent's fixed strategy. In this section, we consider mutual learning, where both players optimize their strategies as the opponent's strategy continues to change.

We excluded the mixed class in this study because the results of matches with the mixed class are trivial. A player with a mixed class must use the same action independently of the opponent's previous actions. Thus, the opponent always receives a higher payoff by choosing D according to the payoff matrix of the PD. When the opponent's choice is always D, the best choice of the mixed class is also D. Thus, only the pure DD will result in equilibrium. Therefore, the mixed class can establish neither cooperative nor exploitative relationships.

Therefore, we consider only the game between players using the memory-one and reactive classes. In our model, the dynamics of players' strategies are deterministic. Thus, the equilibrium state is uniquely determined by the initial values of strategies x and y. Here, we take sampling over the initial conditions. A match between each pair of classes was evaluated using a sufficiently large number of initial conditions of x and y. In each sample, the initial values of the strategy were assumed to be given randomly. In other words, when the former player takes a memory-one (reactive) class, the strategy is randomly chosen from {x₁, x₂, x₃, x₄} ∈ [0, 1]⁴ ({x₁₃, x₂₄} ∈ [0, 1]²).

Figure 3 shows the final state of mutual learning for three matches: (A) between two memory-one classes, (B) between the memory-one and reactive classes, and (C) between two reactive classes. (Note that the last case represented in (C) was already studied in [36].) Here, recall that mutual cooperation satisfies p_CCe = 1, mutual defection satisfies p_DDe = 1, and exploitation satisfies p_CDe ≠ p_DCe.

Zoom In Zoom Out Reset image size

First, we studied the matches between the same classes, represented in (A) and (C). In these matches, exploitation with p_CDe ≠ p_DCe can be in equilibrium. In other words, asymmetry is permanently established between the players depending on their initial strategies, even though both deterministically improve their own strategies to receive a larger payoff. Notably, this asymmetry emerges symmetrically between the players when using the same class. In this case, the number of samples that satisfied p_CDe > p_DCe was equal to those that satisfied p_CDe < p_DCe. In (C), i.e., the match between the reactive classes, the equilibrium exists as multiple fixed points with p_CDe ≠ p_DCe (see [36] for the detailed analysis). By contrast, each exploitative state in (A) permanently oscillated to form a limit cycle, whereas the temporal averages of p_CDe and p_DCe are not equal. There is an infinite number of limit cycles, one of which is achieved depending on the initial conditions. A detailed analysis of these limit cycles will be explained in the next section 4.

The heterogeneous match (B) between the memory-one and reactive classes has the same exploitative states as match (A). However, the most remarkable difference here is that in this exploitation, the reactive class can receive a larger payoff in the match with the memory-one class, and the reverse never occurs. In other words, only the one-sided exploitation from the reactive class to the memory-one class emerges, regardless of the initial conditions. This result appears paradoxical, when one notes that the memory-one class has more information for the strategy choices and is indeed in a more advantageous position than the reactive one in equilibrium when the other player's strategy is fixed, as already confirmed in section 3.1. We will discuss the origin of this unintuitive result in section 4.

We first analyzed the exploitation between the memory-one classes, but the analysis is also applicable to the case between memory-one and reactive classes. An example of the trajectory of strategies x_i and y_i during exploitation is shown in figure 4. For all cases, the exploiting player's strategy satisfies

$$\begin{equation}\begin{aligned}{x}_{1}:\mathrm{fi}\mathrm{x}\mathrm{e}\mathrm{d}\enspace \mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t},\;& {x}_{2}\to 0,\\ {x}_{3}:\mathrm{o}\mathrm{s}\mathrm{c}\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s},\;& {x}_{4}\to 0.\end{aligned}\end{equation} \tag{ 11 }$$

On the other hand, the exploited opponent's strategy satisfies

$$\begin{equation}\begin{aligned}& {y}_{1}:\mathrm{fi}\mathrm{x}\mathrm{e}\mathrm{d}\enspace \mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t},\enspace {y}_{2}\to 0,\\ & {y}_{3}\to 1,\enspace {y}_{4}:\mathrm{o}\mathrm{s}\mathrm{c}\mathrm{i}\mathrm{l}\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s}.\end{aligned}\end{equation} \tag{ 12 }$$

Here, note that x₁ and y₁ are neutrally stable, and the asymptotic value continuously varies with the initial condition. Assuming that x₂ = x₄ = y₂ = 0 and y₃ = 1 and inserting them into equation (2), the possibility vector p satisfies

$$\begin{equation}\mathbf{p}\propto {(0,{y}_{4}{x}_{3},{y}_{4},\bar{{x}_{3}})}^{\text{T}}.\end{equation} \tag{ 13 }$$

This equation leads to p_DCe > p_CDe, which proves that player 1 always receives a larger payoff than player 2. By inserting this into the learning dynamics in equation (7), we obtain

$$\begin{equation}\begin{aligned}& \dot{{x}_{1}}=\dot{{y}_{1}}=0,\\ & \dot{{x}_{3}}={x}_{3}((T-2P+S)-(T-S){y}_{4})\frac{{y}_{4}\bar{{x}_{3}}}{{(1+{y}_{4}-{x}_{3}+{y}_{4}{x}_{3})}^{2}},\\ & \dot{{y}_{4}}=\bar{{x}_{4}}((T-P){x}_{3}-(P-S))\frac{{y}_{4}\bar{{x}_{3}}}{{(1+{y}_{4}-{x}_{3}+{y}_{4}{x}_{3})}^{2}}.\end{aligned}\end{equation} \tag{ 14 }$$

These equations indicate that x₁ and y₁ are neutral, as expected. This is because the players do not experience CC in this oscillatory equilibrium of exploitation and do not have the chance to change x₁ or y₁ by learning. Now, the two-variable dynamics x₃ and y₄ are obtained, which leads to oscillation.

Zoom In Zoom Out Reset image size

The oscillatory dynamics for (x₃, y₄) follow the Lotka–Volterra type equation. Equation (14) has an infinite number of periodic solutions (cycles), and the cycle that is reached depends on the initial strategy. An example of a trajectory is presented in figure 4(B). In equation (14), there is only one fixed point, ${x}_{3}^{{\ast}}$ and ${y}_{4}^{{\ast}}$, which is given by

$$\begin{equation}({x}_{3}^{{\ast}},{y}_{4}^{{\ast}})=\left(\frac{P-S}{T-P},\frac{T-2P+S}{T-S}\right).\end{equation} \tag{ 15 }$$

Then, the players' expected payoffs, ${u}_{\text{e}}^{{\ast}}$ and ${v}_{\text{e}}^{{\ast}}$, are given by

$$\begin{equation}({u}_{\text{e}}^{{\ast}},{v}_{\text{e}}^{{\ast}})=\left(\frac{T+S}{2},P\right).\end{equation} \tag{ 16 }$$

This linear stability analysis shows that this fixed point is neutrally stable, that is, the fixed point is not a focus but a center, as in the original Lotka–Volterra equation. Indeed, the time evolution of (x₃(t), y₄(t)) has a conserved quantity, given by

$$\begin{equation}H({x}_{3},{y}_{4})=-(P-S)(\mathrm{log}\enspace {x}_{3}+2\enspace \mathrm{log}\enspace \bar{{y}_{4}})+(T-P){x}_{3}-(T-S){y}_{4},\end{equation} \tag{ 17 }$$

which is determined by the initial condition and preserved.

Furthermore, this Lotka–Volterra type oscillation provides an explanation of the exploitation we observed. The original Lotka–Volterra equation shows the prey–predator relationship, where the predator increases its own population by sacrificing the prey population. Here $\bar{{x}_{3}}$ (y₄) represents the exploiter's defection (cooperation on the exploited side), which is a selfish (altruistic) action in the PD. Figure 4(B) shows that $\bar{{x}_{3}}$ is larger when x₄ is larger. In other words, the exploiting side learns to use the selfish action with the altruistic action of the exploited. This result means that the exploiting one increases its own payoff at the expense of the exploited side. Thus, the oscillation of the exploitative relationship is interpreted as a prey–predator relationship.

In the previous sections, we mathematically showed how the exploitative relationship is maintained. Next, we intuitively interpret the strategies in equations (11) and (12), which implies that exploitation emerges between the narrow-minded and generous players. Here, we also focus on why the memory-one class is exploited one-sidedly by the reactive one.

Before analyzing equations (11) and (12), we present the well-known TFT and two related strategies in table 1. In the TFT strategy, the player deterministically responds with C to the opponent's previous C, and with D to the opponent's D. A more generous strategy [14] is that the player accepts the opponent's D and probabilistically responds with C. In contrast, in a more narrow-minded strategy [36], the player betrays the opponent's C probabilistically and responds with D. In contrast to the TFT strategy that was adopted to represent the emergence of symmetric cooperation, the generous and narrow-minded TFT strategies represent asymmetric exploitation. However, these strategies do not refer to previous self-actions. Here, we analyze the exploiting and exploited strategies with the generous and narrow-minded TFT, under the constraint that the previous actions of the self are C and D.

As seen in equation (11), the exploiting player uses the narrow-minded TFT strategy in both cases where their previous action was C and D. In other words, the strategy is characterized by x₁ > 0 and x₂ = 0 for a self C, and x₃ > 0 and x₄ = 0. Therefore, CC never occurs in exploitative equilibrium, where x₁ > 0 can be arbitrarily chosen. Thus, a player has the potential to use the exploiting strategy without referring to their own action, as x₁₃ = x₁ = x₃ > 0 and x₂₄ = x₂ = x₄ = 0. Similarly, the exploited player also uses the narrow-minded TFT for a previous self C, that is, y₁ > 0 and y₂ = 0. However, for a self D, the player uses the generous TFT, that is, y₃ = 1 and y₄ > 0. Thus, the exploited player refers to their own action: the player cannot take x₁₃ = x₁ = x₃ and x₂₄ = x₂ = x₄. Although this additional reference to self-action enriches the player's choice of strategy, the player instead tends to be more generous to the opponent's defection and accepts the opponent's exploitation.

In the above, we consider only the equilibrium states given by equations (11) and (12). However, the question remains whether a memory-one class acquires generosity and accepts exploitation during the transient learning process. We attempt to answer this question in the following three steps: first, we classify three equilibrium states in the game between reactive classes: mutual defection, cooperation, and exploitation. Second, by assuming that one of the players adopt a memory-one class instead of the reactive one under these three equilibrium states, we discuss whether the player changes the strategy for each of the three equilibrium states. Third, we consider the one-sided learning process by the above memory-one class under the equilibrium states and examine if the memory-one side's learning increases the opponent's payoff by acquiring generosity.

Step 1. As shown in figure 3(C), there are three cases of equilibria in a match between reactive classes; mutual defection (yellow dots), mutual cooperation (blue dots), exploitation (purple and orange dots). Here, all possible equilibria for exploitation are shown in figure 5.

Zoom In Zoom Out Reset image size

Step 2. Next, we assume that in each of the above states, one player adopts a memory-one class instead of a reactive one. Here, the player can refer to their own actions, after which the state can be unstable. First, for cases of mutual defection and cooperation, the equilibrium states are stable even if one player alternatively adopts a memory-one class. This is easily explained by the analytical result that all the equilibrium states between reactive classes are completely included in those among the memory-one and reactive classes. By contrast, in case of exploitation, the memory-one class receives a larger payoff by releasing constraints x₁ = x₃ = x₁₃ and x₂ = x₄ = x₂₄.

Step 3. Finally, we consider one-sided learning by a memory-one class under the exploitation case. Since we assume that the strategy of the reactive opponent is fixed, the memory-one side receives a larger payoff. For all states, the memory-one class releases the constraints x₁ = x₃ and x₂ = x₄ and learns to be x₁ = x₂ = 0 and x₃ = x₄ = 1, which represents an asymptotic relationship to the exploited strategy of equation (12). Then, the learning of the memory-one side increases the opponent's benefit much more than the increase in one's own benefit (see figure 5). This result shows that the memory-one class becomes generous toward the opponent's defection by also referring to their own actions.

Thus, we have shown that memory-one class becomes generous for the opponent's defection in equilibrium. If the learning feedback from the opponent reactive class is considered, the one-sided exploitation is generated as seen in figure 3(B).

We have demonstrated that the reference to own actions lead to generosity toward the opponent's defection when comparing memory-one and reactive classes. Recall that the reactive class identifies both CC and DC, and CD and DD as the same. Therefore, we can consider two intermediate classes between the memory-one and reactive classes: the player only compresses either CC with DC or CD with DD. These strategies refer to the opponent's action completely but refer to the self-action only when the opponent's action is C or D. In this section, we study the learning dynamics of such extended classes.

Before analyzing the matches, we labeled these new classes according to figure 6. Here, we renamed the memory-one class as the '1234' class because the class distinguishes all of CC (1), CD (2), DC (3), and DD (4), and has four strategy variables x₁, x₂, x₃, and x₄. Further, the reactive class uses two variables x₁₃ and x₂₄, as DC (3) and CC (1) represent one class, and DD (4) and CD (2) represent another. Thus, we renamed it as the '1212' class. In the same way, the newly defined classes are renamed as '1214' and '1232'; the former (latter) combines CC and DC (CD and DD). Among these four classes, complexity can be introduced as the degree to which the self-action is referred to. Figure 6(A) shows the ordering of this complexity. 1234 and 1212 are the most complex and simple of the four classes, respectively, whereas 1214 and 1232 lie between the two.

Zoom In Zoom Out Reset image size

The outcomes of all possible 10 matches between any pairs of the 1234, 1232, 1214, and 1212 classes are shown in figure 6(B). As shown in these figures, 1212, which is the simplest class, can exploit 1212, 1232, and 1234. Class 1214 exploits 1232 and 1234. Class 1232 can exploit 1234. These exploitative relationships are summarized in the bottom right panel. Interestingly, the results show that the simpler classes generally exploit the more complex classes, but the reverse never occurs. Thus, the reference to self-action generally leads to generosity toward the opponent's defection and accepts exploitation.

Theoretically, there are 15 types of classes for choosing strategies that use the information of the self and the opponent's actions up to the previous turn (see appendix B for the formulation and appendix C for the results of matches). The ordering of complexity can be defined over these 15 classes. Although, the complexity among these 15 classes does not always indicate the degree of reference to self-action. Interestingly, the one-sided exploitation by a complex class of a simple one is not observed, except for a single case, i.e., the match between the 1232 and 1131 classes.

So far, we have studied the PD game with the standard score (T, R, P, S) = (5, 3, 1, 0). The above results shows that one-sided exploitation by simple classes of complex classes is valid if the payoff satisfies T − R − P + S > 0. This condition is called the 'submodular' PD [48, 49]; the summation of asymmetric payoffs, T + S, is larger than the summation of symmetric payoffs, R + P.

When the payoff matrix does not satisfy the submodularity, mutual learning does not generate an exploitative relationship. In all the matches among 1234, 1214, 1232, and 1212 classes, the players achieve either mutual cooperation or mutual defection, depending on the initial condition.