EFTA00810742

Theoretical Population Biology 12S (2019) 38-55 Contents lists available at ScienceDirect Theoretical Population Biology ITSEVIER journal home pa go: ..vww.elscvier.comflocateApb A two-player iterated survival game John Wakeley a.*, Martin Nowak a'b'c 'Department of Organismic and Evolutionary Biology. Harvard University. Cambridge. MA, 02138. USA b Program for Evolutionary Dynamics. Harvard University. Cambridge. MA 02138. USA ' Department of Mathematics, Harvard University. Cambridge, hfA 02138, USA ARTICLE INFO ABSTRACT Ankle history: We describe an iterated game between two players, in which the payoff is to survive a number of steps. Received 22 May 2018 Expected payoffs are probabilities of survival. A key feature of the game is that individuals have to survive Available online 12 December 2018 on their own if their partner dies. We consider individuals with hardwired, unconditional behaviors Keywords: or strategies. When both players are present. each step is a symmetric two-player game. The overall Prisoner's Dilemma survival of the two individuals forms a Markov chain. As the number of iterations tends to infinity, all Survival game probabilities of survival decrease to Zero. We obtain general, analytical results for n-step payoffs and use Iterated game these to describe how the game changes as it increases. In order to predict changes in the frequency of Replicator equation a cooperative strategy over time, we embed the survival game in three different models of a large, well- Moran model mixed population. Two of these models are deterministic and one is stochastic. Offspring receive their parent's type without modification and (finesses are determined by the game. Increasing the number of iterations changes the prospects for cooperation. All models become neutral in the limit (ri co). Further, if pairs of cooperative individuals survive together with high probability, specifically higher than for any other pair and for either type when it is alone, then cooperation becomes favored if the number of iterations is large enough. This holds regardless of the structure of pairwise interactions in a single step. Even if the single-step interaction is a Prisoner's Dilemma, the cooperative type becomes favored. Enhanced survival is crucial in these iterated evolutionary games: if players in pairs start the game with a fitness deficit relative to lone individuals, the prospects for cooperation can become even worse than in the case of a single-step game. 02018 Elsevier Inc. All rights reserved. I. Introduction the temptation, to be the one who comes out ahead even if it is at the expense of other individuals. More than a century ago, Kropotkin (1902) argued that what Indeed, game theory and evolutionary theory have uncovered he called mutual aid should be ranked among the main factors of numerous situations in which cooperative or otherwise helping evolution. aneven more important driver ofevolution than within- behaviors are detrimental t0 the individual or selectively disad- species competition. The idea ofmutual aid is similar to that ofmu- vantageous (Hofbauer and Sigmund, 1998; Mesterton-Gibbons. tualism: partnerships may be beneficial to both partners and thus 2000; Cressman, 2005; Schecter and Gintis, 2016). It has been unconditionally favored. Kropotkin had been deeply impressed shown using a variety of models that such behaviors can be disfa- by the abilities of animals to endure harsh winters and perilous vored even if there are obvious advantages to mutual cooperation. migrations in northern Eurasia by living together and supporting For example, in the classic two-player game called the Prisoner's each other in situations where a lone animal had little chance Dilemma (Tucker, 1950; Rapoport and Chammah, 1965; Axelrod. 1984). cooperation results in a "reward payoff' if one's partner of surviving. Thus, he inferred a connection between cooperative behaviors and survival under adverse conditions. But Kropotkin did also cooperates but a "suckers payoff' if one's partner defects. Defection yields a "temptation payoff" if one's partner cooperates not spell out the ways in which adversity might favor cooperative but a "punishment payoff" if one's partner also defects. The reward behaviors, nor did he consider that cooperative behaviors could be is of course better than the punishment, but it is further assumed disadvantageous. He took the very fact that animals seem to do that the temptation payoff is greater than the reward, and the better in groups than alone as evidence for mutual aid, apparently sucker's payoff is even worse than the punishment This leads to a unaware that any interaction presents the opportunity, or at least paradox: an individual who defects always receives a higher payoff but it is clearly illogical for everyone to defect. * Corresponding author. The Prisoner's Dilemma, in which the payoff for defection is E-mail address: wakeleyefas.harvard.edu U. Wakeley). higher than for cooperation regardless of one's partner behavior, hups://doLorg/10.1016b.tpb.2018.12.001 0040-S809/0 2018 Elsevier Inc. All rights reserved. EFTA00810742 J. Wakeley and M. Nowak/Theoretical Population Biology 125(2019)38-55 39 Table 1 we will call PD for short.Ifa(n) > c(n)and b(n) a d(n), cooperation The payoff, a(n). J(n), c(n) or d(n). that an individual receives in a symmetric has the higher payoff only when one's partner cooperates. This is two-player game depends both on the individual and on the individual's paltrier. In a survival game, these payoffs are probabilities of survival A and B denote the Stag-Hunt class of games, or SH for short. If a(n) e c(n) and two possible strategies or types of individuals (e.g. Dove and Hawk). Payoffs are b(n) > d(n), cooperation has the higher payoff only when one's simultaneously awarded to both players, i.e. each is considered the Individual (and partner does not cooperate. This is the Hawk-Dove class, or HD for the other the Partner) and awards a payoff according to the table. short. Finally, if a(n) > c(n) and b(n) > d(n), cooperation has the Partner higher payoff regardless of the partner. We follow De Jaegher and A Hoyer (2016) and call this a Harmony Game. or HG for short. A o(n) b(n) We refer to the n-step survival game as a repeated or iterated Individual B c(n) d(n) game because the payoff structure in each of then steps is identical and equivalent to the single-step version of the game. The notion of repeated or iterated games is generally predicated on the idea that individuals can act differently in different steps and can react represents one of four possible types of two-player games. It is the polar opposite of what Kropotkin (1902) imagined for mutual to their partner's behavior. When this is true, the relatively sim- aid, that instead it would cooperation that would always yield ple conclusions about which strategy may be favorable based on the higher payoff. Between these extremes lie two other kinds of Table 1 do not necessarily hold. For example, if two individuals games. In the Stag Hunt (Skyrms, 2004) and related games, the play an iterated Prisoner's Dilemma under these conditions, then payoff to an individual is higher ifit matches the partner's behavior reactive strategies like Tit-for-Tat (Axelrod, 1984) or win-stay, regardless of what that behavior is. Cooperators in the Stag Hunt lose-shift (Nowak and Sigmund, 1993) can be favored over the go for the big game, a stag, which two such individuals can catch single-iteration strategy of always-defect. Such repeated games but one alone cannot. Non-cooperators opt out of the big-game allow for the phenomena ofdirect and indirect reciprocity (Trivers. hunt and accept a middling payoff, a hare, which a single individual 1971; Nowak and Sigmund, 1998) and trigger strategies which can catch. In this case, cooperation yields the higher payoff only punish non-cooperation (Osborne and Rubinstein, 1994). These when one's partner also cooperates. The Hawk-Dove game (May- are examples of a general phenomenon of behavioral responsive- nard Smith and Price, 1973; Maynard Smith, 1978) represents the ness (Van Cleve and Alccay, 2014) which is one of a small number fourth kind, in which havinga different behavior than one's partner of mechanisms known to promote the evolution of costly cooper- produces a higher payoff. Doves cooperate by sharing resources but ation (reviewed in Nowak, 20066; Van Cleve and Alccay, 2014). retreat when challenged. Hawks do not cooperate. They are ready We do not consider reactive strategies or behavioral respon- to fight to avoid leaving an interaction empty-handed. A Hawk gets siveness in this work. Individuals have one of two possible simple the entire resource when facing a Dove but suffers badly when fac- strategies: A or B as in Table 1. When both individuals are present, ing another Hawk Hawk and Dove, ofcourse, refer to stereotypical which is always the case initially, their survival probabilities in personalities not animals. In this last case, cooperation yields the each iteration are given by the single-step version of Table 1. i.e. higher payoff only when one's partner does not cooperate. Besides with a( 1), b(1), c(1) and d( 1). We will refer to these single-step Hawk-Dove, other well-studied games of this type are the game survival probabilities as a, b, c and d. The n-step survival proba- of chicken and the snowdrift game (Doebeli and liauert. 2005: bilities, a(n), b(n), c(n) and d(n), will depend on these as well as on Nowak, 2006a). the probabilities of survival when an individual is alone. Although We introduce a two-player survival game which can fall into the game always begins with two individuals, if one dies the other any of these four classes. It may also change, for example from a must continue. In the remaining steps, a loner plays a game against Hawk-Dove game into a Prisoner's Dilemma, depending on one Nature. Specifically, a loner survives a single step with probability key feature, the length of the game. In this game, an initially as if it has strategy A and probability do if it has strategy B. We are sampled pair of individuals confronts a hazardous situation which especially interested in cases in which the single-step, two-player is repeated n times. With reference to Kropotkin (1902). we might game defined by a, b. c and d is of a different type than the n-step imagine that each day of a long journey presents a similar set of game defined by a(n), b(n), c(n) and d(n). challenges which threaten survival. They might, for example, be Several previous works have considered games in which the trying to survive a number of very cold nights or attempting to random survival of individuals is an important factor and envi- defend themselves repeatedly against a predator. How they fare in ronmental conditions may be harsh. Eshel and Weinshall (1988) each step depends on their behavior and their partner's behavior. introduced a model in which payoffs are probabilities of survival. The payoff for an individual is all or nothing: either survive to Payoffs were drawn randomly from a distribution, and the game the end of the game or not. Crucially, an individual must face the was repeated with a fixed probability. As in the model we pro- perilous situation alone for the remainder of the game ifits partner pose, Eshel and Weinshall (1988) allowed that the game may con- dies. Survival payoffs are thus meted out at each step, and this will tinue even if one individual dies. They considered optimal strategy be important for determining total payoffs in the game. Denoting choice by individuals under the assumption that individuals have survival as I and death as 0, the expected total payoff to an perfect knowledge of the game's structure, including the distribu- individual in a given situation(i.e. witha specified partner initially) tion of the payoffs. Eshel and Shaked (2001)described a similar sur- is its probability of surviving to the end of the game. The n-step vivaI game but included a general probability that both members survival game is a symmetric two-player game with total payoffs, of a pair survive, thus allowing for arbitrary synergistic effects of a(n), d(n), c(n) and d(n) as Table 1. Using Table 1 to represent an partnership (Hauert et al.. 2006; Kun et al.. 2006) within each step n-step game that is a Prisoner's Dilemma, with A for cooperation of the game. Eshel and Weinshall (1988) had assumed that pairwise and B for defection, the reward would be a(n), the sucker's payoff survival probabilities were the products of two individual survival b(n), the temptation payoff c(n) and the punishment d(n). probabilities. More generally, depending on a(n), b(n), c(n) and d(n), any Garay (2009) combined the idea of a survival game with the n-step survival game falls into one of the four classes described "selfish herd" theory of Hamilton (1971) to produce a model of above. Ignoring the detail that some payoffs might be identical, cooperation between pairs of individuals in defense against re- these are defined as follows.Ifa(n) < c(n) and b(n) < d(n). cooper- peated attacks by a predator. Individuals were of two possible ation has the lower payoff regardless of the partner. Then the game types, characterized by a probability of helping their partner in falls into the class exemplified by the Prisoner's Dilemma. which defense against attack. Garay (2009) derived the total survival EFTA00810743 40 J. Wakefey and M. Nowak/ Theoretical Population Biology 125 (2019)38-55 probabilities of individuals in different kinds of partnerships given lower payoff given partners of type A and B, respectively. In game a faced number ofattacks, and used these to describe evolutionarily theory and much of evolutionary game theory which treats pop- stable strategies (Maynard Smith and Price, 1973) in an infinite ulations, these two comparisons are relevant because individuals population. Using the example contrast of one attack versus eight choose or modify their strategies based on their knowledge of attacks, this revealed cases in which helping could not invade a the game and its outcomes (Sandholm, 2010). When individuals completely selfish population given a one-attack game but could cannot alter or choose their behaviors but payoffs affect fitness. invade if the number of attacks was larger (Garay. 2009). the same sorts of evolutionary models are used to predict changes Kropotkin's notion that high levels of adversity could favor in the frequency of hardwired behaviors in a population. In this cooperation has also been studied using simulations of structured section, we briefly summarize the evolutionary game dynamics of populations. Harms (2001) found that in a Prisoner's Dilemma co- infinite populations. operators could gain an advantage at the inhospitable margins of a The replicator equation, in which general payoffs are inter- population by colonizing patches cleared by the local extinction of preted as contributions to individual fitness (Taylor and Jonker, defectors. More recent simulations of another model by Smaldino 1978; Hofbauer et al., 1979; Zeeman, 1980; Schuster and Sigmund, et al. (2013) also found cases in which cooperation can be favored 1983; Hofbauer and Sigmund, 1988, 1998) provides an evolution- despite the Prisoner's Dilemma. Specifically, if there is a high cost ary setting for the classification of two-player games. In a survival of living for all individuals which, if unchecked, would lead to the game payoffs are fitnesses, specifically viabilities.If x is the relative extinction of the population and if occasional interactions with frequency of A in an infinite population, the replicator equation cooperators are required for survival, then cooperation can be gives the instantaneous rate of change favored. As in Harms (2001). the success ofcooperators in this case relied on their ability to form clusters (Smaldino et al., 2013). = — x)(la(n)— c(n)Ix lb(n)— d(n)j(1 — x)). (1) De Jaegher and Hoyer (2016) considered two game-theoretic Eq. (I) illustrates the evolutionary consequences of partner- models of the behavior and ecology of a pair of individuals, in dependent payoffs when pain are formed at random in proportion which higher levels of adversity can change the type of the game. to the frequencies ofA and B. When x x 1. so thatA is very common Their Model I includes a degree ofcomplementarity which rescales and nearly all partners are A, it is the sign of a(n) — c(n) that payoffs for mixed-strategy pairs compared to same-strategy pain. determines whether A is favored in the population. On the other and which could represent an environmental challenge for mixed hand, when x x 0, so that nearly all partners are B, the sign of pairs. Their Model 2 includes a number of attacks by a predator b(n) — d(n) is what matters. Beyond this. Eq. ( I ) shows that for any on individuals protecting a common resource. Cooperators are value of x, it is the sign of (a(n) — c(n)ix lb(n) — d(n)1(1 — x) immune to attacks while defectors can sustain at most one at- that determines whether A is favored. Thus. in a population and in tack before the common resource is lost. As in Garay (2009). the evolution the relative magnitudesof both a(n)—c(n) and b(n)—d(n) number of attacks indicates the level of environmental challenge. are important. such that one may dominate the other at a given Cooperation may be disfavored when the number of attacks is value of x. This is not something that can be gleaned directly from small and yet become favored when the number of attacks is large. Table 1. Depending on the other parameters in the model, however, a range Without mutation, the fates ofA and B in this infinite population ofother switches between types of games may occur as the number are entirely determined by Eq. (1). From any starting point, x will of attacks increases. De Jaegher (2017) extended these results to move deterministically toward one of three possible equilibria multi-player games. which are the solutions of k = 0 and which may correspond to Our concerns here are similar to those of Garay (2009) and the evolutionarily stable strategies mentioned in Section 1. Two De Jaegher and Hoyer (2016). We study how the structure of the monomorphic equilibria always exist. z = 0 and 1 = 1. The two-player iterated survival game changes as a function of its polymorphic equilibrium parameters, in particular the number of steps it. We compare the conclusions for single-step games with those for n-step games. — b(n) — d(n) (2) Because the payoffs which are probabilities ofsurvival in this game b(n) — d(n) — a(n) c(n) decrease as n increases. n is a measure of adversity. We present might also exist, depending on the payoffs.It exists, which is to say general results for any level ofadversity, then focus on the possible that Eq. (2) gives a biologically meaningful value, when a(n), b(n), structures of large-n games. We find conditions under which any c(n) and d(n) are such that 0 < it < 1. type of single-step game, be it a Prisoner's Dilemma, a Stag Hunt or Eq. ( I ) defines the same four types of symmetric two-player a Hawk-Dove game. will become a Harmony Game as n increases. games. In the PD case, a(n) — c(n) < 0 and b(n) — d(n) < 0. We identify three other large-n results, one of which shows the and Eq. (1) shows that z < 0 for all values of x E (0, 1). Starting opposite: single-step advantages of cooperation may disappear at any value x < 1, the frequency of A will decrease to zero. The as n grows. To facilitate comparisons, we define A throughout as polymorphic equilibrium given by Eq. (2) does not exist. In the SH the more cooperative type based on the single-step game, so that case, a(n)—c(n) > 0 and b(n)—d(n) < O. ThenA isdisfavored when a > d. Thus, AA pairs survive better than BB pairs. We consider x is below the polymorphic equilibrium Si in Eq. (2) and favored both infinite and finite populations but in neither case is there when x > 2. The polymorphic equilibrium exists and is unstable. spatial or any other kind of structure. We do not develop a detailed In the HD case, a(n) — c(n) < 0 and b(n) — d(n) > 0. Then A is biological. ecological or behavioral model. Akin to what is done in favored when x <2. and disfavored when x > z. The polymorphic models of diploid viability selection, we describe the game and its equilibrium exists and is stable. In the HG case. a(n)— c(n) > 0 and results directly in terms of survival probabilities. Our results are b(n) — d(n) > O. Then k > 0 and A is favored for all x e (0. 1). The applicable to a wide range of specific scenarios in which payoffs polymorphic equilibrium does not exist. Fig. I shows examples of a affect viability (as opposed to fertility or fecundity) and in which Prisoner's Dilemma, a Stag Hunt and a Hawk-Dove survival game. partnerships may either enhance survival or detract from it. depicting payoffs in their contexts (upper panels) and associated shapes of k (lower panels). A Harmony Game is not shown but It Evolutionary dynamics would be another case of directional selection, like Fig. ID but with 1 > 0 for all x e (0, 1). Comparing a(n) to c(n) and b(n) to d(n) in Table I indicates These four types of games defined by the shape of ic over the whether the more cooperative strategy A yields the higher or the interval 10, 1] are identical to what is observed in classical models EFTA00810744 J. Wakeley and M. Nowak/Theoretical Population Biology 125(2019)38-55 41 A 1. Prisoners Dilemma c(n) B 1. Stag Hunt C 1. Hawk-Dove cm) 1 098 096 a(n) a(n) = a 7 o sa ow n) n) k 0.98 0.98 ..................., v'z U^) a(n) 0.94 E -0 094 :E 0.94 2 M^> c • n) d(n) BB M BA Ark - BB MBA AA BB MBA AA Individual-Partner Pair Individual-Partner Pair Individual-Paitner Pair D E Stag Hunt F Hawk-Dove o. 0.001 0. -0.CO1 0. X -0.002 A -0.001 A -0.001 -0.003 -0.002 -0.003 -0.002 0 0.25 0.5 0.75 0 0.25 0.5 0.75 0 0.25 0.5 0.75 1 x x Flg. 1. A and D: Prisoner's Dilemma with payoffs a = 0.97. b = 0.94, c = 0.99, d = 0.95 (a linear transformation of the classic R = 3, 5 = 0, 7 = 5, P = 1 of Axelrod 0984e.g.a = 0.94 + R/100).B and E: Stag Hunt, with a = 0.99.0 = 0.94, c = d = 0.97 (corresponding to a stag value of0.05 and a hare value of0.03 added to a baseline survival probability of 0.94). C and F: Hawk-Dove, with a = 0.97, b = 0.95, c = 0.99, d = 0.94 (corresponding to a cost of fighting of 0.03 and a resource value of 0.04, with a baseline survival probability of 0.95). Line segments in A, B and C connect the survival probabilities for B (left) versus A (right) when each occurs with given type of partner. The curves in D. E and F show k from Eq. (1). In diploid population genetics. these three cases fora are called directional selection (D), underdominance (E) and overdominance (F). of diploid viability selection (e.g. see section 4.2 in Nagylaki, 1992). and the state in which both individuals have died which we denote The difference is that, barring atypical phenomena such as meiotic 0. It is convenient to represent a single iteration using the matrix drive (Dunn, 1953; Sandler and Novitsky, 1957) or segregation- AA AB BB A B 0 distortion (Sandler and Hiraizumi, 1960; Hard. 1974), standard IA a2 0 0 2a(1 - a) O (1 — a)2 1 diploid models always have b(n) = c(n). It is by allowing b(n) # AB 0 be 0 b(1 — r) r(1 — b) (1 — b)( 1 — c) c(n) that two-player games introduce the paradox of the Prisoner's BB 0 0 d2 0 2d(1 — d) (1 — d)2 Dilemma, in which c(n) > a(n) and d(n) > b(n) but a(n) > d(n). In A 0 0 0 a0 O 1 — ao (3) order to have c(n) > a(n) and d(n) > b(n) in a standard diploid B 0 0 0 0 1 — do do model it would be necessary to have a(n) e d(n). Accordingly. O 0 0 0 0 O 1 the Prisoner's Dilemma is the most stringent form of a cooperative dilemma (Doebeli and Hauert, 2005; Hauert et al., 2006; Nowak, with entries equal to the transition probabilities among he six 2012). A cooperative dilemma exists when (i) mutual cooperation states. For example, the transition from state AB to state B means results in a higher payoff than mutual non-cooperation, so that that the B individual survives and the A individual dies. In a single a(n) > d(n) when A represents cooperation, but (ii) there is incen- step, this occurs with probability c(1— b). Note that the transitions tive to be non-cooperative in at least one of three ways: (iia) c(n) > in Eq. (3) include the fates of both individuals but the individuals a(n), (iib) d(n) > b(n) or (iic) c(n) > b(n) (Nowak, 2012). The are not labeled Individual and Partner as they are in Table I. The process described by Eq. (3) is depicted in Fig. 2. State Prisoner's Dilemma includes all three of these incentives. Games O is an absorbing state. There is no possibility of transition be- with fewer barriers to cooperation, such as the Stag Hunt and the tween the paired states M. AB, and BB. Instead each of these Hawk-Dove game, represent relaxed cooperative dilemmas. The feeds either straight into state 0, from which there is no escape. Harmony Game involves no cooperative dilemma and presents no or into one of the loner states. A or B. and from there into 0. barrier to cooperation. Therefore, transitions from any of the starting, paired states to absorption in state 0 involve either one or two changes of state. 2. An n-step survival game between two players Further, all of the transitions that cannot occur in a single iteration (the 0 entries in the matrix) cannot ever occur regardless of the Our survival game always includes n iterations and begins with number of iterations. Because of this simple structure, the n-step a pair of individuals. In each step, both might survive or one, transition probabilities can be calculated directly by conditioning the other or both might die. These same outcomes hold for the on the times these transitions take place, i.e. on their positions in the sequence of n iterations. It is also possible to compute the complete game, only the probabilities of surviving will be smaller. n-step transition probabilities using standard techniques for We consider two unconditional strategies. A and B. When both Markov chains, and this provides a useful framework for decom- players are present, their individual survival probabilities are given posing the process and describing its behavior when n is large. by the single-step version of Table I with a(1) • a, b(1) • b, Details of the matrix approach are given in the Appendix. but c(1) • c and d(1) • d. However, the next iteration must be two key features of it inform our presentation. The first is the fact faced by whoever has survived so far, until all n iterations are done. of the absorbing state (0) in which both individuals have died. It Thus, an individual might have to play alone. Then the survival will be reached eventually, meaning in the limit n oo. For large probabilities become ao forA without a partner and do for B without n the game process will be just the approach to this state. Second, a partner. In all of what follows. A will be the more cooperative when n is large, the rate of approach to state 0 will be given by the strategy, meaning that a > d. largest non-unit eigenvalue of the matrix in Eq. (3). Because it is an The n-step survival game is a stochastic process with six possi- upper triangular matrix, the eigenvalues are simply the entries on ble states: the paired states AA, AB and BB, the loner states A and B, the diagonal. By convention, we call the largest of these 2. = land EFTA00810745 42 J. Wakefey and M. Nowak/Theoretical Population Biology 12S (2019)38-55 Prob(BB 8) = E 2d(1- (14) I 0 = (d2n A„ 2d( 1 d) ) d2 (15) ® The only other possibility is that neither individual survives, so we also have Prob(AA 0) = 1 —Prob(AA AA) — Prob(AA —* A) (16) Prob(AB —a. 0) = 1 — Prob(AB AB) —Prob(AB A) .® Prob(AB B) (17) 0 Prob(BB —* 0) = 1 — Prob(BB —a. BB) — Prob(BB B). (18) a In the Appendix we show how these probabilities are obtained using techniques for Markov chains. Eqs. (8) through (15) make it clear that this paired survival process is one in which the fortunes of individuals may change. rv R 0 perhaps drastically depending on the values of ao and do relative to a, b, c and d. Consider starting state AB. Eq. (12) shows how the probability of beingin state Bat the end ofn iterations is computed. 0 In words, both individuals survive for some time (i-1 steps), then A dies, and B survives the rest of time (n - i steps). If A dies, B trades its individual survival probability ofc. which B enjoys in the Flg. 2. Flow diagram ofthe stochastic process givenby the matrix of Eq. (3).Arrows show all possible transitions among the six states (A4. At BB. A. B. 0).States AA, AB, presence and A, fora loner survival probability ofdo. It could be that BB. A and B are transient. State 0 is absorbing. The process always begins in one of do c c, making B worse off after A dies. In fact, B's fate is closely the three states on the left, M. AB or BB. After n iterations. it may be in any of the tied with A's because this switch could occur quickly if A's survival six possible states. probability, b, is small. Of course. while there is a cost to B when A dies (assuming 4 c c), the death of A in this partnership also represents a rather direct disadvantage toA.In order to understand note that this corresponds to the eventual absorption in state 0. In whether A or B will prevail in evolution, it is necessary to account all, we have for the full dynamics of reproduction in a population, with fitnesses that are determined by this game. We will take this up in Section 3. = 1, a2, bc, d2, ao, . (4) In the n-step game, the differences a(n) - c(n) and b(n) — d(n) give the conditions under which A is favored. The n-step payoffs. Following the discussion of Table 1 and Eq. ( I), we expect the or survival probabilities, are computed by accounting for the two fates of A and 8 in this iterated game to depend on the relative ways an individual may survive the game. An individual survives magnitudes of a versus c and b versus d. Eq. (4) suggests that if both it and its partner survive or if it survives but its partner their fates will also depend on the relative magnitudes of the dies. The total probabilities of individual survival in each kind of pair survival probabilities. a2. be and d2. and on the loner survival partnership are probabilities. ao and 4.and that this dependence may be especially a(n) = Prob(AA AA) + Prob(AA -o A)/2 strong when n is large. We compute the n-step pair survival probabilities directly as _ a2" a—ao + a(1 — a) (19) a2 — ao ao — a2 Prob(AA —* AA) = a2" (5) b(n) = Prob(AB AB) + Prob(AB A) Prob(AB —a. AB) = (bcr (6) b ao b(1 — c) = (kr +4 (20) be - ao ao — be Prob(BB BB) = dm. (7) c(n) = Prob(AB AB) Prob(AB B) Then, by considering the possibility that one of the individuals c — do c(1 — b) = (bcr +4 (21) might die in step 1 < i < ri and the other individual survives to be -d o do — bc the end of the game, we have d(n) = Prob(BB BB) + Prob(BB B)/2 re 2 d dO d — d) —d" + 4 (22) Prob(AA -* A) = E (a2)1-1 2a(1 - a)arl d2 — 4 4 - d2 The transitions AA A and BB B are adjusted by a factor = 02. 4) 2a(1 — a) of 1/2 because they are equally likely to happen by the death 2 — an of the partner as by the death of the focal individual. Eqs. (19) through (22) are our general results, namely the n-step survival Prob(AB -> A) = E(bc)i-1b(1 - c)arl probabilities for individuals of types A and B given each kind of a-t initial partnership. They are exact for any values of a, b, c, d, ao and do greater than zero and less than one, and for any number ((kr d°) 1_ c) of iterations n 1. As expected, when n = 1, they reduce to the It single-step survival probabilities a(1) = a, b( 1) = b, c(1) = c and Prob(AB B) = DbCr I CO — LOCIrj d(1) = d. i-t The two key differences in payoff are then CO — b) a ao (ocr - d rf c do a(n) - c(n)- am ao + 4 Go ——a2 a) a EFTA00810746 J. Waketey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 43 — c(I — b) iterations increases. We present results both for a(n) — c(n) and _ (krc__ (23) be —do ° do — bc b(n) — d(n) and in terms of the unified predictions of Eq. ( I ). bt c) Eq. (1) is a standard replicator equation for a symmetric two- b(n)— d(n)= (bon + aiot player game. In the Appendix, we show how it may be derived bc —ao ao aol — bc for the n-step survival game. This has two notable features: the _ d2nd_ do d(1 — d) (24) game works by removing individuals from the population instead d2 — do ° do — d2 of affecting their rates of reproduction and the derivation follows Each term in Eqs. (23) and (24) includes a factor A? for one of the pairs of individuals rather than single individuals. Also in the non-unit eigenvalues (a2, bc, d2. a0. do). These factors are positive. Appendix, we take a discrete-time, population-genetic approach The denominator of each term is the difference between two to show that the change in frequency of A over one generation is eigenvalues.If we know the eigenvalues, we know whether the de- nominators are positive or negative. In some cases the numerator Ax = x(1 — x)0a(n)— c(n)1x (b(n)— d(n)1(1 — x))/th, (27) might be negative, but by the definition of a, b, c, d, ao and do as which is identical in form to ic in Eq. ( I ) but scaled by the average probabilities, the numerator will be positive if the denominator is fitness, tb. Eq. (27) is directly comparable to classical results for positive. Thus Eqs.(23) and (24) are written so that the sign ofeach diploid viability selection. Importantly, tun 0. Therefore, all of the term indicates whether it favors A (+ ) or disfavors A (—) when conclusions concerning the ultimate fate of the cooperative type the denominator is positive. We do this to facilitate the analysis of A in the population, based on the sign of the change in frequency. large n, in which case a(n) — c(n) and b(n) — d(n) will come to be will be the same whether one appeals to Eq. ( I ) or Eq. (27). dominated by the terms involving the largest non-unit eigenvalue. Each of the n-step payoffs a(n), b(n), c(n) and d(n) depends on Another way to write Eqs. (23) and (24) is two of the non-unit eigenvalues, and each eigenvalue is raised to a2" — an on) — c(n) = (a — ao)° (c do) (bcr cag - the power n. Thus a(n), b(n), c(n) and d(n) all decrease to zero as n tends to infinity. Surviving more steps is less likely than surviving a2 — ao be —do fewer steps. As a consequence, the two key fitness differences. + a; — (25) a(n) — c(n) and b(n) — d(n), and the instantaneous rate of change. _4 2* — 4 also approach zero in the limit 17 -. co. We study the approach b(n)— d(n)= (b — ao)(kr (d do) to this neutral limit in order to understand whether increasing n bc ao d2 — do fundamentally alters the prospects for cooperation. For example. + ao — d; (26) if ic < 0 when n = 1 but the neutral limit z = 0 is approached from above, then there exists a value of n beyond which k > 0. In which brings attention to the possible survival value of having a this case. the single-step game favors the non-cooperative type B partner and of being A versus B without a partner. Thus, a — ao and but large-n games favor the cooperative type A. c — do in Eq. (25) are the increments to the survival probabilities In the following subsections, we describe four main categories of A and B when each has partner A compared to when they are of large-n. or prolonged, survival games. These are characterized alone. Likewise. b — ao and d — 4 in Eq. (26) are the corresponding both in the usual way by the relative values of a, b, c and d, that increments for A and B with partner B versus alone. Note that the is by the structure of the single-step game. and by the rank order fractions multiplying these increments are always positive. All else of non-unit eigenvalues a2, bc, d2 ao and do. We have defined the beingequal, a(n)—c(n)increases witha—ao > 0 but decreaseswith cooperative strategy A throughout by a > d, which guarantees that c — do > 0, and b(n)—d(n)increases with b — ao > 0 but decreases a2 > d2. Thus d2 will never be the largest non-unit eigenvalue. The with d-4 > O. Negative increments have the opposite effects. For first two types of prolonged survival games we describe are those example,ifB suffers by having aB partner compared to being alone, for which a2 is the largest and those for which bc is the largest. then d—do < 0. which in turn favors A via the difference b(n)—d(n). The third and fourth types of prolonged survival games are ones The last two terms of both Eqs. (25) and (26) quantify the n-step in which lone individuals survive with high probability. Simplifi- effects of differences in the loner survival probabilities, ao and do. cations emerge when 17 is large because the n-step payoffs become If ao > do the contribution to both a(n) — c(n) and b(n) — d(n) is dominated by their largest terms. We present approximations for positive, and if 4 < do it is negative. large n, showing just the leading terms ofa(n)—c(n)and b(n)—d(n), These five possible increments to individual survival, namely and ofit. a —ao, b—ao, c —do, d—do and ao—do.in Eqs.(25)and (26 ) are helpful for understanding some of our results and the structure of survival 11. Games in which cooperation will prevail games generally. However, it is also clear from the additional presence of the pair-state eigenvalues in Eqs. (25) and (26) as well The first case we consider is when having a partner significantly as in Eqs. (23) and (24). that these five increments do not entirely enhances survival and it is theAA type not theAB type that survives determine the prospects for cooperation. The relative magnitudes best This is case in which a2 is the largest non-unit eigenvalue: of the five eigenvalues will also be important. This may be seen in a2 > bc, ao, do. Thus, the probability that both members of an the denominators in Eqs. (25), (26), (23) and (24), which contrast AA pair survive a single iteration is greater than the corresponding the survival of pairs to the survival of loners: a2 —ao, be —ao. bc —do, probability for any other pair and for either type of lone individual. d2-4. The precedingexpressions, Eqs.(23)and(24). are especially The condition a2 > bc may be true of any kind of two-player useful for predicting the structure of the game when n is large. game, but the chance it is depends on the game. One way to quantify this is toconsider the total parameter space of two-player. 3. Four types of prolonged survival games single-step survival games. Because a, b. c and d are probabilities and here it is assumed that a > d, this space is equal to half of a Here, we present analytical results for large n and use our four-dimensional hypercube. Further, since a > d implies a2 > d2, exact results to illustrate how the prospects for A depend on n for there are just three possible relations of the pair-state eigenvalues, intermediate values. The analysis of large n allows us to answer a2, bc and d2. Either bc > a2, a2 > bc > d2 or d2 > bc. The latter questions such as: Is there a number of iterations beyond which A two satisfy the condition a2 > bc. Table 2 lists the proportions is unambiguously favored? More generally, we ask how the game of the total parameter space satisfying the criteria for each type might change for A, for better or for worse, as the number of of single-step game. In addition, Table 2 shows the percentages EFTA00810747 44 J. WaRefry and M. Nowak/ Theoretical Population Many 125 (2019)38-55 Table 2 Partitioning of the four-dimensional hypercube into fractions meeting the game conditions in the first column. The types of single-step games are abbreviated PD, SH. HD and HG for Prisoners Dilemma, Stag Hunt, Hawk-Dove game, and Harmony Game. Exact results were obtained by integration in Mathematica (Wolfram Research. Inc., 2018). Percentages are rounded to the nearest 0.1%. Note that in all cases a > el by definition, so it is always true that 02 > d2. Order of pair-state eigenvalues Game type Proportion be >a, a2 > bc > d2 el? > bc PD: a <c,b<d 1/12 10% 40% SO% with a > (b + c)/2 (3/4 of 1/12) 0% 40% 60% with a > (b + c)12.d < (b+ c)/2 (1/2 of 1/12) 0% 60% 40% SH: a>c,b<d 1/4 0% 11.1% 88.9% HD: a < b > d 1/4 80% 20% 0% HG: a>c,b>d S/12 10% 663% 23.3% of the game-specific parameter regions corresponding to each of In this case, the large-n game becomes a Harmony Game in whichA the three possible relations of the pair-state eigenvalues. So, for is unambiguously favored. On the other hand, if the second largest example, 90% of PD-type games have az > bc, compared to 20% of eigenvalue is dz, then from Eq. (24) we have HD-type games. — d° d When the single-step game is a canonical Prisoner's Dilemma,it b(n)— d(n) dea e0. (31) crz — do is always true that az > bc. This is due to the assumption that the reward payoff must be larger than the average of the temptation In this case, the large-n game becomes or remains a Stag Hunt payoff and the suckers payoff (see, e.g. Rapoport and Chammah. game, so A does not become unambiguously favored. The chances 1965; Axelrod, 1984), in which case we have of being in either of these two sub-cases are given in Table 2. Similar arguments can be applied when ao or do is the second- b c +(b c) 2 largest non-unit eigenvalue. Then Eq. (24) shows that for large iv, 02 > (- )2- bc (- > bc. (28) b(n) — d(n) > 0 if ao is the second-largest, and b(n) — d(n) < 0 if 2 do is the second-largest. In what follows, it will be important to know which is the second- Regardless of which is the second-largest non-unit eigenvalue, largest non-unit eigenvalue, especially in the consideration of fi- if az is the largest then a(n) — c(n) will be much greater than nite populations in Section 4. Thus, the two cases az > bc > dz b(n) — d(n) when n is large. This may be expressed as and d2 > bc are distinguished in Table 2. We find that canonical Prisoner's Dilemmas are 40%az > bc > dz and 60%d2 > bc. If b(n) — d(n) lim (32) we also require that the punishment payoff must be less than the 1?- .00 a(n) — c(n) average of the temptation payoff and the sucker's payoff (e.g. see Thus, when n is very large, the advantage in survival that A has p. 219 of Cressman. 2005). this 40 : 60 split is reversed. over B when the partner is A will be much greater than whatever It would be possible to expand Table 2 by including ao and do difference there is in survival between A and B when the partner is and integrating over the six-dimensional space of all parameters. B. One may argue from an evolutionary standpoint that the large-n There would be more than three relations among the five non-unit Stag Hunt game, with a(n) — c(n) in Eq. (29) and b(n) — d(n) in eigenvalues. We do not pursue this here. For the prolonged games Eq. (31), will be a very relaxed cooperative dilemma. Specifically. considered in this section and in Section 3.2, it is taken for granted owing to Eq. (32), a large-n Stag Hunt of this sort will have its that neither ao nor do is the largest non-unit eigenvalue.In contrast. unstable polymorphic equilibrium x in Eq. (2) close to zero, with in Sections 3.3 and 3.4 it is taken for granted that one of ao or do is the result that A will be favored over most of the range of x. the largest non-unit eigenvalue or they both are and they are equal. Finally, when n is very large the replicator equation, Eq. (1), If we were to include a0 and do in Table 2, we expect they would becomes often be larger than az and bc due to the fact that these pair-state a — a° k rr x2(1 — x)a2"— >0 for all x e(0, 1). (33) eigenvalues are products of probabilities. 0 There is just one term in Eqs. (19) through (22) which includes We conclude that, if n is large enough in the case of this first type the factor a2..corresponding to what is assumed here to be the of prolonged survival game, the more cooperative type A will be largest non-unit eigenvalue, az.It is in the expression for a(n).Thus. uniformly favored in an infinite population. This result requires we have only that az is the largest non-unit eigenvalue. It is robust to a(n) — c(n) r a2„ a2 differences in the loner survival probabilities 00 and do. As long as ao > 0 (29) az is large enough, the non-cooperative type B cannot be rescued when n is very large. Eq. (29) gives the approximate value of a(n)— by an advantage in loner survivability. Table 2 shows that az > bc c(n) as it approaches zero in the limit n co. This result shows for the great majority of possible single-step survival games, the exception being games of the Hawk-Dove type, of which only 20% that whatever value or sign a(1) — c(1) = a — c might have in a will have this property. single iteration, there exists a value of n above which a(n) — c(n) Fig. 3 illustrates how the two key differences in payoff. a(n) — will be greater than zero. c(n) and b(n) — d(n), change as n increases for two examples based The leading term in the other key difference. b(n) — d(n), will on the Prisoner's Dilemma. In Fig. 3A, the survival probabilities for depend on which of bc, dz. 00 or do is the next-largest eigenvalue. individuals with partners are a = 0.8. b = 0.6, c = 0.9. d = 0.7. Consider, for example, a game which strongly enhances survival, Thus. the cooperative type A incurs an 11% loss of fitness compared such that az, bc, d2 > ao, do. If bc is the next largest eigenvalue to B when the partner is A and a 14% loss when the partner is B.The after az, then from Eq. (24) we have survival probabilities for loners are much smaller and identical: ao = do = 0.3. The eigenvalues are az = 0.64. bc = 0.54. b(n)— d(n) e- (bcr bc — ao > 0. (30) = 0.49, ao = 0.3 and do = 0.3. This game starts as a Prisoners b ao EFTA00810748 J. Wakeley and M. Nowak/Theoretical Population Biology 125 (2019)38-55 45 A B Payoff Difference Number of iterations. n Number of Iterations, n fig. 3. Values of the two key differences.a(n) — c(n)and b(n) — d(n).as a function of the number of iterations. for two different survival games which are Prisoner's Dilemmas when n = 1. Panel A: a = 0.8. b =0.6,c =0.9.d= 0.7 and ao = do = 0.3. Panel 6: a= 0.9.6 =0.8,c =0.95. d = 0.85 and ao =do = 0.65. A 0.04 PO HD HG B 0.04 PD • SH HG 8 0.02 8 0.02 0 0. ,go z -0.02 • a(n)-c(n) -0.02 • a(n)-c(n) -0.04 • b(n)- d(n) I -0.04 • b(n)-d(n) 0 . - 0.06 (5 -0.06 - 0.08 -0.08 1 10 20 30 40 SO 60 70 80 1 10 20 30 40 60 60 70 80 Number of Iterations, n Number of Iterations, n Fag. 0. Values °IG(n) — c(n) and b(n) — d(n) over n = 1...80 iterations for two games which begin as Prisoner's Dilemmas at n = I. In both cases, the loner survival probabilities are ck, = 4 = 0.9. PanelA: a = 0.97.6 = 0.94. c = 0.99,d = 0.95; these are the same as in Fig. IA and D. PanelIt a = 0.9733.6 = 0.94.c = 0.99, d = 0.9567: this differs from panel A only in that a and d are spaced evenly between c and b. Dilemma (PD) when n = 1. For intermediate numbers of iterations. how an increase in this sort of environmental challenge affects the specifically n = 3 and n = 4, the game changes to one in which A is outcome of the game. Single-step survival probabilities are smaller disfavored when rare (b(n) — d(n) < 0) but favored when common in Fig. 3A, so we would say that the environment is harsher in (a(n) — c(n) > 0), as in the Stag Hunt (SH). Then as n grows it Fig. 3A than in Fig. 3B. Comparing the two shows that increasing becomes a game in which A is uniformly favored, as in the Harmony this kind of adversity causes the cooperative type A to become Game (HG). unambiguously favored (HG) sooner, that is for smaller n. Fig. 3B displays results for a very similar game, but one in which Fig. 4 shows how a(n) — c(n) and b(n) — d(n) change as the the single-iteration probabilities of individual survival are shifted number of iterations increases for two Prisoner's Dilemmas with closer to 1 by a factor of 1/2. That is, a = 0.9, b = 0.8. c = 0.95. survival probabilities even closer to 1. In Fig. 4A. the game at n = 1 d = 0.85 and ao = do = 0.65. Now the eigenvalues are a2 = 0.81. is identical to the example in Fig. IA and ID. Again, the payoffs in be = 0.76, d2 = 0.7225. ao = 0.65 and do = 0.65. Overall, the this game follow the classical spacing (R = 3, S = 0, T = 5. P = 1) picture is similar to Fig. 3A, with the three phases PD then SH then of Axelrod (1984). For comparison, the game in Fig. 4B has the same HG. But now in moving from n = I ton = 2. the selection against value of the temptation payoff (c = 0.99) and the sucker payoff A becomes stronger rather than weaker. This is more in line with (b = 0.94) but has reward a and punishment d spaced evenly the usual notion of iterated games, in which payoffs are assumed between these, as was the case in Fig. 3. 1n both Fig. 4A and 4B, to accrue additively. In survival games, however, this decrease the loner survival probabilities are ao = 4 = 0.9. Both show cannot continue because all games become neutral as n approaches the phenomenon of a worsening dilemma (greater disadvantage infinity. for A) at small n. then a switch to a relaxed dilemma and finally to Fig. 3 may be compared to Fig. 4 (Model 2AIII) of De Jaegher a Harmony Game. In Fig. 4B, the intermediate, relaxed dilemma is and Hoyer (2016). with their number of attacks being analogous a Stag Hunt game, as in Fig. 3. In Fig. 4A, the intermediate, relaxed to n. A point of contrast between our model based on multiplying dilemma is instead a Hawk-Dove game. Eqs. (25) and (26) are single-step survival probabilities and the behavioral and ecological helpful in understanding the difference between Fig. 4Aand Fig. 48. scenario, Model 2 of De Jaegher and Hoyer (2016) is that in our In moving from Fig. 4A to Fig. 48, the increment a — ao in Eq. (25) is model the payoff differences a(n) — c(n) and b(n) — d(n) may be boosted by 0.0033 and the increment d — 4 in Eq. (26) is boosted non-monotonic. as they are in Fig. 3, whereas these are always by 0.0067. This increases a(n) — c(n) and decreases b(n) — d(n). monotonic in Model 2 of De Jaegher and Hoyer (2016). We will Fig. 5 shows how a(n) — c(n) and b(n) — d(n) change as n return to Model 2 ofDeJaegher and Hoyer (2016) and make further increases for examples of the Stag Hunt and the Hawk-Dove game. comparisons in the Discussion. Parameters are as in Fig. I with the addition of ao = 4 = 0.9 Because survival probabilities decrease with each iteration. n is for both models. As with the examples in Fig. 4, these are cases a measure of the adversity faced by individuals, or the harshness in which a2 is the largest non-unit eigenvalue. so Eqs. (29), (30) of the environment Mother way to measure adversity is directly and (31) apply. In contrast to what is seen in Fig. 4, one or the in terms of single-step survival probabilities. Adversity is greater other of a(n) — c(n) and b(n) — d(n) is still negative in Fig. 5 even when a, b, c, d, ao and do are smaller. Fig. 3 provides an example of at n = 100. From n = 1 to beyond n = 100, these remain a EFTA00810749 46 J. Wake!ey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 A Stag Hunt B Hawk-Dove 0.15 • a(n)-c(n) 0.4 ▪ b(n)-din) 0.3 Od 0.2 OM 0. -0.05 -0.2 -0.1 20 40 80 80 100 20 40 60 80 100 Number of Iterations. n Number of Iterations, n Fig. 5. Values of o(n) — c(n) and b(n) — d(n) over n = 1... 100 iterations for two games whkh begin as a Stag Hunt and as a Hawk-Dove game at n = I. In both cases, as in Fig. 4, the loner survival probabilities are cio = do = 0.9. Panel A: a = 0.99. b = 0.94. r = d = 0.97; these are the same as in Fig. IB and E. Panel El: a = 0.97.6 = 0.95. c = 0.99. d = 0.94: these are the same as in Fig. IC and F. A B 0.005 0. s h. 0 -0.005 C 4 1 4, -0.005 4.) -0.01 -001 60 O. 005 o. 0.2 0.2 0.4 0 OA 0.8 x 0.8 15 n x Fig. 6. Picas of k for the models in Fig. 4. that is for two diferent Prisoner's Dilemmas at n = I. Panel A: a = 0.97,8 = 0.94.4 = 0.99. = 0.95 and ao = do = 0.9. Panel It a = 0.9733.6 = 0.94, c = 0.99. d = 0.9567 and ao = do = 0.9. In both panels. the thick contour line is drawn at it = 0, which corresponds to the polymorphic equilibrium of Eq. (2) for each value of rt. For simplicity of presentation, we plot the surface as a smooth function of n, whereas in reality it is discrete. Stag Hunt and a Hawk-Dove game. In the Stag Hunt of Fig. 5A, the in the picture for the largest values of ri in Fig. 7, in both cases A next-largest eigenvalue is d2, so b(n) — d(n) will remain negative. will become favored if n is large enough. In the Stag Hunt of Fig. 7A However, as can already be seen in the figure. it will be much or 5A, ic 0 as ri co. In the Hawk-Dove game of Fig. 76 or 5B. it smaller in magnitude than a(n) — c(n). In the Hawk-Dove game of will exit the biologically relevant interval and become larger than Fig. 56. the next-largest eigenvalue is bc, so b(n) — d(n)will remain one when a(n) — c(n) first becomes positive, when n = 615. positive. But since a2 is the largest non-unit eigenvalue, a(n) — c(n) will become positive (when n = 615. from numerical solution) and 12. Games which preserve noncooperative behaviors eventually will become much larger than b(n) — d(n). Another way to view the results presented in Figs. 3-5 is in Here we consider the case that the game enhances survival terms of * rather than a(n) — c(n) and b(n) — d(n). Fig. 6 shows this significantly and it is AB that fares best. Thus. bc is the largest non- for the examples of Fig. 4. Now the overall shape of * can be seen unit eigenvalue. Table 2 shows that this will occur most readily as it moves from ri = 1, where it is a Prisoner's Dilemma identical when the single-step game is a Hawk-Dove game. It might also to what is depicted in Fig. ID. to ri = 60, where it is a Harmony occur for Prisoner's-Dilemma type games, but only if the canonical Game. In both panels, the thick contour line traces z = 0. showing assumption a > (b t c)/2 is violated. It will not occur when the the polymorphic equilibrium in Eq. (2) when it exists. In Fig. 6A, single-step game is a Stag Hunt. Here, again, we assume that having this equilibrium enters at x = 0 and exits at x = 1. In between, a partner significantly enhances survival, so bc > a0, do. the shape of z is like that of Fig. 1F, namely similar to the Hawk- There are two terms in Eqs. ( 19) through (22) which include the Dove game in which A is favored when rare and disfavored when common. In Fig. 6B, the equilibrium enters at x = 1 and exits at factor (bc)". They are in the expressions for b(n) and c(n). For very x = 0. Here the shape of z for intermediate n is like that of Fig. IE. large n we have namely similar to the Stag Hunt in which A is disfavored when c— rare and favored when common. Note that although a(n) — c(n) a(n)— c(n) —Cory — do 0 (34) bc — do and b(n) — d(n) vary non-monotonically in these cases (Fig. 4), the — equilibrium A is monotonic inn. b(n)— d(n) (bc)" > 0. (35) bc b —aoao Fig. 7 shows how the shape of * changes as the number of iterations increases for the Stag Hunt and the Hawk-Dove game Thus, survival games of this sort either stay or become Hawk-Dove depicted in Fig. 5. Note that to aid in visualizing these surfaces. games as n increases. In addition. a(n) — c(n) and b(n) — d(n) will the viewpoints are different in the two panels and the ranges of be of the same order of magnitude as n tends to infinity. In the n are different than in Fig. 5. As with the Prisoner's Dilemmas in previous section. where AA survived best, increasing ri converted Figs. 3 and 6, these are both cases in which a2 is the largest non- single-step Prisoner's Dilemmas. Stag Hunts and some Hawk- unit eigenvalue. Following the discussion of Fig. 5, although the Dove games into progressively more relaxed cooperative dilemmas polymorphic equilibria traced by the thick contour lines are still until, eventually, A became favored. Here. Eqs. (34) and (35) show EFTA00810750 J. Wakeley and M. Nowak/Theoretical Population Biology 125 (2019)38-55 47 A B 0.015 0.01 0.005 5( 0. -0.005 0 1. Fig. 7. Plots of* for the models in Fig. 5. For the n = 1, Stag Hunt in panel A: a = 0.99, b = 0.94.c = d = 0.97 and no = do = 0.9. For the n = 1. Hawk-Dove game in panel B: a = 0.97, b = 0.95, c = 0.99, d = 0.94 and ao = do = 0.9. In both panels, the thick contour line is drawn at i = 0, which corresponds to the polymorphic equilibrium it of Eq. (2) for each value of n. Note that the viewpoints and the ranges °Info; the two panels are different. A 0.4 B 0.2 a 0. -0.4 20 40 60 80 100 120 1. Number of Iterations, n Hg. B. Hawk-Dove type game in which be is the largest eigenvalue: payoffs a = 0.95, b = 0.97.c = 0.99, d = 0.94 and ao = do = 0.9. The thick contour line in B shows k = 0, which stabilizes to ilk = 0.5625 as n increases. that single-iteration Hawk-Dove games in which AB survives best thick contour line) as n increases, even as the game first intensifies are robust to such alteration. Eqs. (34) and (35) also show that if for intermediate n then approaches neutrality for large n. a survival game is a non-canonical Prisoner's Dilemma at n = 1, such that bc rather than a is largest non-unit eigenvalue, the 13. Games ruled by loner survivability n-step game will eventually become a Hawk-Dove game rather than a Harmony Game. In both of the previous sections. it was assumed that having When bc is the largest non-unit eigenvalue, the large-n approx- a partner provided a substantial boost to survival. In this section imation for the change in frequency is and the next, we consider cases in which having a partner does not enhance survival. If either a lone A or a lone B has the highest is Az x(1 — x)(bc)" (— c — do x+ —ao b (1 x)). (36) probability of surviving each iteration of the game, then either a0 be —do bc — ao or do is the largest non-unit eigenvalue. Two terms in Eqs. (19) We deduce that, for large n, is > 0 if x < it& and it <Olin. icon through (22) include the factor ag—they are in the expressions where for a(n) and b(n)—and two terms include the factor dg—they are in the expressions for c(n) and d(n). If ao is the largest non-unit (b — 00)/(bc — Go) eigenvalue, then for very large n we have itbr = (37) (b — ao)/(bc — ao) + (c — do)/(bc — do) This frequency cutoff may be interpreted as the probability, when n a(n) — c(n)-- ao 0(1 a ) > 0 (38) 00 — 02 is large, that if AB plays the game and only one individual survives. b(1 — c) it is the A individual. For reference, if a0 and d0 are both very small. b(n)— d(n)-- ao >0 (39) then Eq. (37) reduces to b/(b + c). If the population frequency of a0 —bc A is smaller than this probability in Eq. (37), then A is favored. and if do is the largest non-unit eigenvalue, we have otherwise it is disfavored. This cutoff is of course equal to the cr c(1 — b) limiting (ri oo) value of Eq. (2) when bc is the largest non-unit a(n) — c(n) (40) ° do — be < ° eigenvalue, which is stable in this case. d"nd(1 — d) Fig. 8 depicts both how a(n) — c(n) and b(n) — d(n) change and b(n)— d(n)-- ., — do _ d2 e0. (41) how ic changes as n increases fora Hawk-Dove example in this case. It may be more easily seen in Fig. 8B that this example remains Thus. if single individuals of one type have a higher probability a Hawk-Dove game with a stable polymorphic equilibrium (the of survival than single individuals of the other type and than any EFTA00810751 48 J. Wake!ey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 A 0.12 HO SH PD B 0.025 S 0.09 Payoff Difference C • a(n)-c(n) 0. 0.06 b(n)-d(n) -0.025 15 0.03 -0.05 a 0. -0.075 -0.03 -OA 5 10 15 20 Number of Iterations. n Number of Iterations, n flg.9. Values of o(n) — c(n) and b(n) — d(n) as n increases for two games which begin as a Harmony Came and as a Hawk-Dove game at n= 1. Panel A: a= 0.8, b= 0.5, c= 0.7, cl= 0.4 and ao =do = 0.9. Panel B: Hawk-Dove a = 0.93.6 = 0.91.c = 0.954 = 0.90 and ao = = 0.95. pair of individuals, that type will become favored in the population enhance survival but does cause payoff differences - and is the as the number of iterations grows, regardless of the structure of only source of these differences - there is no expectation that the pairwise interactions in the single-step game. The corresponding prospects for cooperation will improve as the number of iterations approximations for k can be inferred from Eqs. (38) through (41). increases. These cooperative dilemmas may intensify rather than becoming more relaxed. 3.4. Games with a pairwise survival deficit 4 Evolutionary dynamics in a finite population A related but probably more interesting case is when single individuals have higher survival probabilities than any pair of All populations are finite, so changes in the frequency of A will individuals but the loner survival probabilities are identical. Then be stochastic rather than deterministic. Here we describe these having a partner does not enhance survival but the two-player changes for a particular population model and check the validity game may still provide an advantage to one strategy or the other. of our previous conclusions based on the deterministic predictions In this case ao = do is the largest non-unit eigenvalue. When n is for k (or .dx). In a finite population, xis discrete, not continuous. If very large, using Eqs. (23) and (24) and substituting ao for 4, we N is the population size and K is the number of A individuals. then have K E (0, 1 N - 1, N) and x E (0, 1/N (N - 1)/N, 1).We a(n) - c(n) a(1 - a) c(1 — b)] study the process ofjumps inK orx when both death and reproduc- 4 [ (42) tion are stochastic. We continue to assume that the survival game at, ao _ bc is the only source of fitness differences and that the population is b(1 — c) d(1 — d)i • well-mixed in the sense that partners are chosen at random. d(n) 4 [— be (43) b(n) — ao — ao — d2 Due to the relative simplicity of its transitions and the ease Whether these differences are greater or less than zero will depend with which the survival game may be embedded within it, we on which of the two terms in the bra kets is larger in absolute base our model on the haploid population-genetic model of Moran value. This situation, in which the game is played starting in the (1958).The standard version ofthe Moran model pairs single births pair state with a survival deficit, is similar in spirit to the way the and deaths such that the population size remains constant. In the Prisoner's Dilemma was originally formulated, with two individu- absence of selection, each individual has probability 1/N of being als in custody, each wanting to get out. Here, as in Section 3.3, we chosen to reproduce in a given time step, and a I/N chance to do not show the corresponding result for k, which can be inferred die. The individual chosen to reproduce makes an offspring which from Eqs. (42) and (43). replaces the individual chosen to die. When selection is included. Consider the case of a strong deficit, specifically with ao = do it usually acts by modifying the probability an individual is chosen. not too small and all of a, b, c and d close to zero. Then, neglecting now based on its fitness, either to reproduce or to die. terms of a2, be and d2 which will be even smaller, Eqs. (42) and (43) We. instead, consider a model in which two randomly chosen become individuals play the survival game. Players are chosen without replacement from the population. This precludes self-interaction. a(n) — c(n) 4 -1(a — (44) which is appropriate because we wish to model two individuals b(n)— (12 — 6). (45) being sequestered from the rest of the population for a period of time. When the n iterations of the game are finished. both Thus, with a strongdeficit, the results for large n will beof the same individuals may be alive, one may be alive or both may be dead. type as the results for n = 1. This is the only source of death in the population and the only When the deficit is milder, no simple approximations for Eqs. place in the life cycle where selection acts. In each time step, these (42) and (43) are available, but numerical analysis shows that the 0, 1 or 2 deaths are compensated by the same number of births. conclusions of Section 3.1 can be reversed. Fig. 9 shows two exam- All individuals have an equal chance to reproduce, including the ples ofhow a(n) - c(n) and b(n) - d(n) may change as n increases, two who play the game. Thus, parents are sampled at random with converting a single-step Harmony Came and a single-step Hawk- replacement from the population. Finally, offspring inherit their Dove game into Prisoner's Dilemmas. In both examples, ao = 4 parent's type without modification, i.e. there is no mutation. is the largest non-unit eigenvalue. In Fig. 9A, the game becomes a Let K be the current number of A individuals. After one time Stag Hunt for intermediate values ofmin Fig. 9B. the game changes step, the number is K', which will differ from K by at most 2 and directly from a Hawk-Dove game into a Prisoner's Dilemma as is bounded by 0 and N. The numbers of A individuals in a series n increases. Thus, when having a partner does not significantly of time steps forms a Markov chain with an (N + 1) x + 1), EFTA00810752 J. Wakeley and M. Nowakl Theoretical Population Biology ITS (2019)38-55 49 pentadiagonal transition probability matrix. Because there is no of N the deterministic prediction is incorrect The error comes mutation. the states K = 0 and K = N are absorbing.The transition from overestimating the importance of AA and BB pairings in the probabilities are computed as products of three terms: one for the infinite-population model by implicitly allowing self-interaction. sampling of a pair of individuals to play the game, one for the Our aim here is to reassess the conclusions of the previous section outcome of the game and one for the sampling of parents. In all, in light of such differences and especially in light of the fact that we have A can be fixed or lost from a finite population regardless of the N—KN—K—I KK value of nein Criteria for A being favored in a finite population PK.K — Prob(BB 0)— (46) N—1 NN are generally based on the probability it reaches fixation starting N—KN—K-1 at K = 1 versus the probability it is lost starting at K = N - 1 PK.K-E1 — (Nowak et al.. 2004; Lessard. 2005). If the probability of fixation of N N—1 K — Ki A is greater than the neutral probability I/N and the probability x [Prob(BB B) + Prob(BB 0)2N — of loss is less than 1/N, we might conclude that A is favored. N N When selection is weak, a number of analytical results for fixation KN —K probabilities and times in evolutionary games are available (Wu -I- 2 NN— I et al.. 2010). Alternatively, in models with mutation the decision KK as to whether A is favored can be made by comparing the expected x [Prob(AB A)— + Prob(AB —t• — (47) 0)—N N equilibrium frequency ofA under neutrality versus selection(Rous- K N —K set and Billiard. 2000). When the mutation rate is small, these two PICK — 2 approaches become identical. NN—1 In Section 3, we used ic > 0 for all x E (0, I) as a criterion for A x [Prob(AB B)N — K Prob(AB -o 0)N - N —K N — — K] being unambiguously favored. The analogue here is Wiz] > 0 for all x E (I/N (N - 1)/N). We take this to be a more stringent KK- 1 criterion than the one based on fixation probabilities. However. + NM - 1 from Eqs. (52) and (53), it is clear that even when AA survives best N K NKi 0)2- and a2 is the largest non-unit eigenvalue, it will not necessarily be x [Prob(AA A)- K + Prob(AA N true that 6[44> 0 for all x e (1/N (N -1)/N).In particular. for the two terminal frequencies, we have (48) - - c(n)— d(n) PICK-2 — Prout.AA -0 0)NNKNNK (49) s(n: K = 1) = b(n) — d(n) (54) =N N - I N—1 PX.K = I - PK .X+2 PX - PK.K -1 — PK .K-2- (50) and Although this process is not as simple as the standard Moran On) — fa) s(n; K = N — 1) = a(n)— c(n) (55) process, for which a number of exact results are available (Moran, N—1 1962), as a pentadiagonal matrix it may still be amenable to study. When there is only a single A individual, the possible benefit of the Following our previous analysis ofic, we focus on the change in the AA pair is unrealized. At the other extreme. when there is only one number of cooperators, AK = K' - K. in particular on the sign and B individual. it is the BB pair that does not matter. magnitude ofElidiK) which measures the direction and strength of Therefore, even when a2 is the largest non-unit eigenvalue, it selection. will always be possible to find a population size small enough that The expected value of AK is given by A remains disfavored even as n tends to infinity. In particular, if EIAKI =2PK.K+2 + Pg.g PX.K - I - 2PX.K-2- (51) the population consists of just two individuals, there is only one polymorphic frequency, K = N - K = 1, and we have Using Eqs. (46) through (50) and simplifying yields KN —K s(n; K = 1) = b(n) — c(n) ifN = 2. (56) nAKI — 2 From Eqs. (20) and (21) we have x (a(n)K — 1 „ ‘ DI K c(n) + DM ll boo coo = bb — ao (bcy, b(1 — c),.„ c — do (bcy, N—I N—1 N-1 N —1 d(n)N-K-1/ ro ao - bc bc —do (52) c(1 — b)d,, (57) with a(n), b(n), c(n), d(n) as before in Eqs. (19) through (22). — do — bc ° Because x = K/N, we also have Ekix] = EfAKI/N. Aside from which will not in general be favorable to A.To illustrate, if we make the factor of 2, which arises because two individual are chosen to play the game and possibly die, Eq. (52) shows nearly the same the simplifying assumption that the loner fitnesses are the same dependence on a(n), b(n), c(n) and d(n) as the deterministic result (a0 = do), then Eq. (57) reduces to for z in Eq.(I). We define the selection coefficient (bcr - a" b(n)- c(n) = (b - c) ° if ao = do. (58) —1 K + N—K bc — Co s(n; K) = c(n) — bi) °(n)N-1 N 1 N—I When n = 1, this is simply b - c. Then as n becomes large. N — K —1 regardless of which eigenvalue (bc or ao = do) is larger, the sign d(n) (53) N—1 ' of b(n) - c(n) will be the same as the sign of b — c. One of the which captures the dependence of E(AK] on a(n), b(n), c(n) and ways in which a game may be considered a cooperative dilemma d(n) in its entirety. Of course, the sign of s(n; K) determines the is that b e c (Nowak, 2012). Note that b e c in all ofour numerical sign of ElAK and E(Ax]. examples. When N = 2, none of the conclusions about relaxed Comparing s(n; K) in Eq. (53) to its deterministic counterpart in cooperative dilemmas based on z and increasing n will hold. Just Eq. (I) reveals that when N is small or when K is small regardless the opposite: A will be disfavored. EFTA00810753 SO J. Wakefey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 While there is no expectation that conclusions from an infinite- Case II, reconsidered. This is the case in which bc is the largest non- population model will be accurate for small populations, we ex- unit eigenvalue and the game tends to preserve the less coopera- pect their heuristic value to hold for moderately large N. In what tive strategy. The finite-population result corresponding to Eq. (36) follows, we focus on N >> 1. We begin as before by noting that is our finite-population model collapses to a neutral Moran model so; ic) c — do K b — ao — K 1 ocr I in the limit n co. although it would be one in which pairs of bc — do N — 1 bc — ao N - 1 individuals rather than single individuals are chosen to die and to reproduce. Again, we answer the question aboutA being favored by for all K = I N - 1. (62) studying the approach to this neutral limit In our finite-population Thus, the terminal frequencies x = 1/N and x = 1 — I/N model, this neutral limit is E[dx] = 0. Then if. for example. do not require a separate treatment as they did under Case I. Wax] < 0 when n = 1 but the neutral limit E(Ax) = 0 Here, just as in the infinite-population model, increasing n will not is approached from above, then there exists a value of n above fundamentally alter the game. Selection pressure will tend to keep which E[dx] > 0. We briefly reconsider each type of prolonged both A and B in the population. game. Case Bl, reconsidered. This is the case in which ao or do is the largest Case 1, reconsidered This is the case in which a2 is the largest non- non-unit eigenvalue. For these games ruled by loner survivability, unit eigenvalue and cooperation is expected to prevail. The finite- we have population result corresponding to Eq. (33) is /a(I —a)K — 1 b(I — c)D1 —K\ a—ao K-1 >0 s(n; K) a-- — > 0 for K = 2 N - 1. (59) s(n; K) bc a2 — ao N 1 for all K = I, N- I (63) Thus, for n large enough. EWA > 0 for every frequency x = K/N except for one. We must investigate K = 1 because, for a given if ao is the largest non-unit eigenvalue, and large value of n, even though a(n) may dominate all other terms (c(1 — b) K d(1 — d)N — K — 1 when K = 2 N — 1. it is only b(n), c(n) and d(n) that appear in s(n; K) s < 0 s(n; K = 1) in Eq. (54). When n is large the sign and magnitude 4 — brig — I 4 — d2 N-1 of s(n; K = 1) will be determined by the second largest non- for all K = 1 N- 1 (64) unit eigenvalue. We consider the same two possibilities as before. namely when either bc or d2 is next-largest eigenvalue, and we if do is the largest non-unit eigenvalue. The conclusions are un- continue to assume that the loner survival probabilities, ao and do changed in the modified Moran model relative to the infinite- are small by comparison. population model. Whichever type survives best alone becomes If the next largest eigenvalue is bc, then for K = 1 we have favored when n is large. b— o \ Case IV, reconsidered. This is the case in which ao = do is the so ; K = 1) (kr largest non-unit eigenvalue. For these games with a pairwise sur- bc — aGo N , 1 bc — do (60) vival deficit we have For very small N it could be that Eq. (60) is negative, but it will ta(I — a) K — 1 c(1 — b) K be positive for any moderate to large N, making E[dx] > 0 for x= 1/N. Combining this with Eq. (59), we conclude that here, with An; K) — a2 N — 1 ao - bc — 1 a2 > bc > d2 do, if n is large enough, then Eldx] > 0 for all b(1—c)N —K d(1 — d)N —K — 1\ 4 K=1 N - 1. Previous Figs. 3 and 4 are examples of this case (65) ao — bc N — 1 ao — d2 N — 1 ) in which A becomes unambiguously favored. If the second-largest non-unit eigenvalue is '12, we have in general, and d — do N — 2 lC — 1 N —K dN — K — 1) I s(n; K = 1) —a- c ici 6 1 <0, (61) An; K) (a c +b ao N-1 N-1 N— 1 N—I and EkIxi < 0 when x = 1/N. But. again, for large n it will (66) also be true that d2" <C al", so the magnitude of the selection coefficient against A when K = 1 will be small compared to when the deficit is strong. If K = 1, the first terms in the paren- selection coefficients in favor of A for every other value of K. We theses in Eqs. (65) and (66) disappear, making A worse off when it state without proof that in this case, if n is large enough, the is very rare. because those terms are positive. When K = N - I, probability of fixation of A starting from a single copy should be the last terms in the parentheses in Eqs. (65) and (66) disappear, greater than the neutral probability I/N, and the probability of making A better off when B is very rare, because those terms are loss of A starting from N — 1 copies should be less than I/N. negative. It is not expected that this will improve the already dim We conclude that A should be considered favored in this case as prospects for the evolution of cooperative behaviors in this case in well. which there is no survival advantage to having a partner. Situations in which one of the loner survival probabilities. ao or do, is the second largest eigenvalue can be treated similarly. We S. Discussion do not pursue this in detail, but in Case III below we consider the possibility that one of these is the largest non-unit eigenvalue, and We have proposed a simple but general, biologically motivated from that we can infer that nothing beyond the two possibilities survival game. It is a repeated game which always starts with a above for s(n: K = 1) will arise. That is. as n grows. s(n; K = I) will pair of individuals and includes a fixed number of iterations. Prob- be either positive or negative but will be very small compared to abilities of survival are its expected payoffs. We have assumed that s(n; K) which is positive for K = 2 N- 1.Thus, the conclusion multi-step survival is a Markov process, which has consequences that A becomes favored when a2 is the largest non-unit eigenvalue such as that the game may change stochastically to a loner game still holds. against Nature if one's partner dies and that all payoffs tend to EFTA00810754 J. Wakefey and M. Nowak/Theoretical Population Biology 125 (2019)38-SS SI zero as the number of iterations tends to infinity. Results provid- Table 3 ing some insight into the evolution of cooperation are obtained Two models in which individuals of type A and B have the same single-step probability of survival given the type of their partner, but in which the survival when the number of iterations is large, including the conversion probability of any individual depends on the partner's type. The format is as in of cooperative dilemmas into situations in which cooperation is Table I but with an additional column for the case in which an individual no longer unambiguously favored. has a partner. In the first model having a partner of type A boosts the probability of Survival is an appropriate, direct measure of payoff (or utility) survival of both types of individuals by an amount t above background (do). In the in evolution. Our analysis is similar to the analysis of diploid second model having a partner of type B lowers the probability of survival of both types of individuals by an amount e. population-genetic models in that we have described interactions Equal-payoff model with A-partner increment: between individuals only in terms of fitness. What we have called strategies correspond to the alleles in a diploid population-genetic A B No partner model. We have assumed that individuals' strategies are fixed. A a = do + e b = do ao =4 B e=do+e d = do do meaning the same in every step of the game. For this sort of survival game, our results are general. They hold for any pattern Equal-payoff model with 8-partner decrement: of payoffs in the single-step matrix in Eq. (3). They can be applied A B No partner to any biological, behavioral or ecological model for which the A a=do = do — • ao = do assumption of Markov survival is reasonable. They need not be B c = do d= 4 —• .4 pure strategies, for example. They could be mixed, as in Garay (2009), so long as they are the same in every step. Our Markov assumption is twofold: (i) when an individual has a equations and their equivalents, Eqs. (23) and (24). Pairwise sur- partner, its single-step survival probability depends on its strategy vival probabilities are crucial to understanding the structures of and on its partner's strategy, and (ii) when an individual is alone. prolonged (large-n) games. which depend strongly on the largest its single-step survival probability depends on its strategy. An non-unit eigenvalue of the Markov survival process. individual's multi-step survival probability is the product of its In the first example, having a partner of one particular strategy single-step survival probabilities given its situation in each step. The model of repeated predator attacks in Garay (2009) is precisely either increases or decreases the survival probability of the indi- such a Markov survival model.In constructing the matrix in Eq. (3), vidual. but by an amount which does not depend on the strategy of we have further assumed (iii) that conditional on both of their the individual. We consider the two possibilities shown in Table 3. strategies, which determine their individual survival probabilities. that either having an A partner increases survival or having a B the two members of a pair survive each step independently of partner decreases survival. In both cases, the change in survival one another. If synergistic effects of partnership (Hauert et al.. (+4 or -6)is the same for A individuals and B individuals. All other 2006; Kun et al., 2006) are defined as non-multiplicative, then survival probabilities are equal to a baseline, do. Thus a — c = 0 assumption (iii) precludes synergy within a single step in our and b — d = 0. The single-step game is neutral. Fig. 10 shows how model. Any synergies in an n-step game result from the Markov a(n)— c(n)and b(n)— d(n)depend on n when do = 0.5 ande = 0.1. process of survival for a given set of parameters (a, b, c, d, ao. do). In this case, ao = 4 is the largest non-unit eigenvalue. In Fig. 10k We began with the motivation of Kropotkin (1902), whose a(n)— c(n) > 0 for n ≥ 2 while b(n)— d(n) = 0 for all n. In Fig. 10B. work focused on vertebrates, but we note here that our model a(n) — c(n) = 0 for all n while b(n) — d(n) e 0 for n > 2. These is not designed with any particular species in mind. None of our results may be understood with reference to Eqs. (25) and (26). results rely on individuals possessing the intelligence or decision- with ao = do. In the case of Fig. 10A, a — ao = c — do = f but, owing making capability of vertebrates. It could be argued that many to the eigenvalues, the positive increment a — 4 is weighed more fundamental steps in the evolution of cooperation happened long heavily than the negative increment c — 4 when n > 2. so Eq. (25) before, perhaps many hundreds of millions of years before this gives a(n) — c(n) > 0 for n > 2. Eq. (26) gives b(n) — b(n) = 0 kind of intelligence. The iterated survival game described in Sec- because b — ao = d — 4 = 0. Analogous arguments explain tion 2 may be applied in a variety of settings, but the evolutionary the case of Fig. 10B. In sum, the more-cooperative A is favored if it models of Sections 3 and 4 may be better suited to populations unilaterally increases the survival of its partner (Fig. 10A) and the of single-celled organisms than complicated metazoans, because less-cooperative B is favored ifit unilaterally decreases the survival they assume haploid genetic transmission and individuals with of its partner (Fig. 10B). fixed strategies. For the second example, we design an iterated survival game Kropotkin inferred a connection between cooperation and sur- which expresses key ideas behind Model 2 of De Jaegher and Hoyer vival under adverse conditions, and our work draws a distinc- (2016) directly in terms of survival. Again, this is a model of a tion between games in which partnerships enhance survival and number of attacks on a pair of individuals. The individuals are those in which having a partner is a liability. If there is a fitness identical in most respects. There is a public good which benefits all advantage of having a partner, then less-cooperative behaviors individuals equally, ifit is preserved against attack. Each individual which are advantageous in each single step of a game may become disadvantageous as the number of iterations increases. This can be also has a private good which may impart a by-product benefit on true even if cooperative behaviors are strongly disfavored in the its partner, again if it is preserved against attack The difference is single-step game, as in the Prisoner's Dilemma (e.g. Fig. 3A). But that cooperators (A) pay a cost and are immune to attacks whereas if partnerships do not significantly enhance survival, the opposite defectors (B) pay no cost and are susceptible to attacks. Thus, may occur. In this case, even cooperative behaviors which are defectors may benefit when they are paired with cooperators. favored in the single-step game can become disfavored as the This is not a survival game because individuals do not die in the number of iterations increases (e.g. Fig. 9A). attacks. It is Markovian only in that each attack hits one member We close with a discussion of two examples which illustrate of the pair randomly with probability 1/2. Nonetheless. this behav- the effects of enhanced survival. We have identified two kinds ioral and ecological model has roughly similar outcomes to those of enhanced survival: for individuals and for pairs of individuals. we have described here. when n corresponds to the number of These different boosts to survival are captured by the individual- attacks. survival increments of Eqs. (25) and (26), and by the presence of Consider a survival game in which the probability an individual the pair-state eigenvalues of the Markov process in these same survives each attack. or iteration. depends on its type and its EFTA00810755 52 J. Wakeley and M. Nowak/ Theoretical Population Biology 125 (2019)38-55 A B 0.008 0.008 • a(n)-c(n) Payoff Difference a b(n)-d(n) 0.004 0 004 0 11 -0.004 -0 004 a. -0.008 -0.008 Number of Iterations, n Number of Iterations, n Fly 10. Values of the two key differences, a(n)- c(n)and LIN - d(n). as a function of the number of iterations. for the two versions of the equal-payoff model depicted in Table 3, with baseline survival probability do = 0.5 and increment/decrement t = 0.1. Panel A: a = 0.6, b = OS, r = 0.6, d = OS. Panel B: a = 0.5, b = 0.4. r = 0.5, d= 0.4. Both panels: ao = do = 0.5. partner's type as follows. PD SH HG A with A: (67) 0. I -0.01 A with B: b= do g - u (68) co aa -0.02 e • a(n)-c(n) B with A: c= do -Fg-l-u (69) pi • • b(n)-d(n) t -0.03 • B with B: d=d o tg (70) 0. -0.04 • • A alone: ao = d,3 — u (71) ep . -0.05 B alone: do (72) 1 10 20 30 40 Here u is the cost of being a cooperator. v is the benefit an in- dividual receives when its partner is a cooperator and g is the Number of Iterations, n added benefit of simply having a partner. The parameters do and Fig. 11. Values of the two key differences, a(n)— c(n)and b(n) — d(n)as a function g might represent the private and public goods in De Jaegher and of the number of iterations. for the game with payoffs shown in Table 4. Hoyer (2016). The general versions of our model and Model 2 of De Jaegher and Hoyer (2016) each have seven free parameters. Table 4 whereas the model above has five. Numerical example of the model defined in Eqs. (67) through (72), assuming that For this model a —c = b-d = -u < 0, so the cooperative type A do =0.8.0 = 0.02. = 0.03andwithg = (I + Nir o — O/2 — do x 0.B. is disfavored in the single-step game.In addition. ao —do = —u < 0 A B No partner which suggests that the cooperative type A might be disfavored for A a = 0.94 = 0.91 00 = 0.78 larger n. However, considering the terms in Eqs. (25) and (26) we 8 c=0.96 d=0.93 do = 0.80 havea—ao =c —do =g+v> 0 andb—ao =d - do = g > 0. If a2 is the largest non-unit eigenvalue then a(n) — c(n) will eventually become positive, and if bc is the next-largest eigenvalue then b(n) - d(n) will also eventually become positive. are immune to attack in Model 2 of De Jaegher and Hoyer (2016). A strong criterion for partnerships enhancing survival would be Here, instead, cooperation becomes favored as n increases due to that a2, bc and d2 are all greater than do. A weak criterion would the subtle advantages the more cooperative type A accrues because be that only a2 > do. If we adopt the strong criterion and further AA pairs survive best and AB second best among the three possible assume that a2. bc > d2. then the game would enhance survival if pairings, despite the direct disadvantage of A compared to B in AB d2 > do. This implies that pairs as well as in the loner state. Rather than considering that individuals might change their g> Vito - do. (73) strategies between steps of the game in response to their part- ners, we have focused on the evolution of hardwired, non-reactive We have assumed throughout that a, b,c and dare all less than one. strategies within populations. Among other results, we find the so it must also be true that conditions under which cooperation will increase in frequency g < 1 — do — u. (74) over time in both finite and infinite populations despite being disfavored in every step of the game. In the case of a finite pop- For the sake of illustration, set g equal to the midpoint between ulation, this conclusion is based on a strong criterion for selective these two extremes, so that g = (1 + — u)/2 — do. Assume advantage, that the expected change in frequency of cooperation further that v > u, which means that the single-step game is a is positive over the entire range of possible frequencies. Prisoner's Dilemma. Using the particular values do = 0.8. u = 0.02 and v = 0.03 gives the survival game shown in Table 4. One would not immediately guess that the more cooperative Acknowledgments type A would ever be favored in such a game. But as Fig. 11 illustrates, this does become true as the number of attacks in- We thank Sabin Lessard and two anonymous reviewers for creases. For n > 32 both a(n) — c(n) and b(n) - d(n) are greater comments which greatly improved the work We also thank Jo- than zero, though clearly the prolonged game yields rather weak hannes Wirtz for posing a question which led to the model pre- selection in this particular example. Note that we have not granted sented in Table 3 and Fig. 10. This work was supported in part by A any obvious advantage. As previously mentioned, cooperators grants N00014-16-1-2914 from the US Office of Naval Research. EFTA00810756 J. Waketey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 53 55832 from the John Templeton Foundation and W911NF-18-2- 2d(1 — d) 2d(1 — d)1T 0265 from the US Defense Advanced Research Projects Agency to I4 [0, 0, 1, 0, (88) = 4 — d2 , 1 4 — d2 _I Martin Nowak. 0,0,0,1,0, 1]T 15 = (89) Appendix /6 = (0, 0, 0, 0, 1, —1jr. (90) Analysis of the Markov chain In erms of the notation here, the probabilities in the main text are Corresponding to the matrix in Eq. (3), we define the stochastic Prob(AA AA) = (M")11 (91) matrix Prob(AB AB) = (M922 (92) a2 0 0 2a(1 — a) 0 (1 — a? 0 be 0 a(1 — b) b(1 — a) (1 — a)(1 — b) Prob(BB BB) = (M")33 (93) M = 0 0 d2 0 2d(1 — d) (1 — d)2 (75) Prob(AA A1= (M")14 (94) 0 0 0 ao 0 1— ao Prob(AB (95) A) = (M")24 0 0 0 0 do 1— do 0 0 0 0 Prob(AB B) = (M"hs (96) whose entries are the transition probabilities between the six Prob(BB B) = (97) possible states (M. AB, BB, A, B. 0). Thus, M describes a Markov Prob(AA —0. 0) = (M")16 (98) process with a single absorbing state (0). When this is iterated n times, the probabilities of each of the six possible outcomes will be Prob(AB 0) = (M"126 (99) the entries of the n-step transition probability matrix Al". Because Prob(BB 0) = (Mn)34. (100) M is a triangular matrix, its eigenvalues are given by its diagonal elements. These are just the probabilities ofremainingin each state We can define the limiting matrix over a single iteration. For reference, we reproduce Eq. (4) A4(' ) = lim Mn = rig (101) 111-.DO A= (l,a2 „bc.d2,a0. do) (76) which has all six rows equal to [0. 0, 0. 0. 0, 1] and thus captures and further note that 12, 13 and A4 are the probabilities that the fact that absorption in state 0 is guaranteed, ifn is large enough. both individuals survive the iteration. Under the assumption that regardless of the starting state. Then we may write neither surviving nor perishing of individuals is guaranteed over a 6 single iteration, which is to say 1 > a, b. c, d, ao. do > 0, xl = I is M" =m(-)i-E4,17. (102) the largest eigenvalue and 1 > 12,13, 13, A6 > 0. i-2 From the standard theory of finite Markov chains (cf. section 2.12 in Ewens, 2004), if ri and fi are the corresponding right and Because 1 > Ai > 0 for i E (2, 3, 4, 5. 6). the eventual rate of left eigenvectors of M, normalized so that decay toward AO') will be depend on which Ai is the leading non- unit eigenvalue of M. This, in turn, will depend on the values of a, 6 b, c, d, ao and do. Ifn is very large, the sum in Eq. (102) will become ErA=1, (77) dominated by a single term, namely the one involving the largest i- non-unit eigenvalue. for each i E (1. 2, 3, 4. 5, 6). then M" may be expressed in terms of For example, if12 = a2 is the largest eigenvalue and n is very its spectral decomposition large. Prob(AA M) = (M")11 x ay' (103) mn=EA.frilj (78) Prob(AA —* A) = 04'114 — 2a(1 — )a2" (104) a2 — ao in which T denotes the transpose. With A defined as above, the a( — a) 2a21 Prob(AA 0) = (ien16 I a2" ao (105) corresponding right and left eigenvectors of the matrix M are = [1, 1. 1, 1, 1. lir (79) Prob(AB 0) = (M")26 as- 1 (106) r2= [1, O. 0, 0, O. OIT (80) Prob(BB 0) = (Ain)3o rr 1. (107) r3 = [0, 1. 0, 0, 0, Of (81) Then the effects of selection via the game reduce to r4 = [0, O. 1, 0, 0, OIT (82) a(n) — c(n)*.: a(n) = Prob(AA —. AA) Prob(AA A)/2 2a(1 — a) b(1 — c) a — ao a2n (108) rs — 0,1, 0. 01 (83) a2 — ao [ — e 'ap —bc . c(1 — b) 2d(1 — d) b(n)— d(n) O. (109) ro — [ be 0, I. (di. (84) — do d2 Our general results, Eqs. (23) and (24), are written to facilitate the direct inference of such conclusions. fl = o, O. 0, 0, O. 11T (85) 2a(1 — a) Derivation oldie replicator equation f2 = [1, 0,0, ao _ 02 .- 0, 1 2a( an I_—a2a) (86) We assume a well-mixed population of infinite size. Well- b(1 — c) c(1 — b) b(1 — c) c(1 — b)1T mixed means specifically that initial partnerships in the game are f3= [0, 1,0, 1 be—ao ' be —do ' be—ao be —do ] formed at random in proportion to the current frequencies of A 87) and B. The game is the only cause of death in the population EFTA00810757 54 J. Wakefey and M. Nowak/Theoretical Population Biology 125 (2019)38-55 and the only source of differences in fitness. Here we consider a Loners must receive half the weight of intact pain in this account- continuous-time model. The replicator equation for the iterated ing. So. what remains of the total population is represented as survival game can be developed as follows. Changes in the frequen- cies (x and y) ofA and B are given by a pair of differential equations x2(Prob(AA AA) + Prob(AA A)/2) + 2x(1 — x)(Prob(AB —. AB) + Prob(AB A)/2 = —x2 [2Prob(AA 0) + Prob(AA A)] + Prob(AB B)/2) — 2xy[Prob(AB 0) + Prob(AB B)] + xo (110) + (1 — x)2(Prob(BB -* BB) Prob(BB B)/2). (121) = —y2[2Prob(BB —* 0) Prob(BB B)] — 2xy[Prob(AB 0) + Prob(AB (111) By definition, this is equivalent to the average fitness of the popu- A)] + yO lation. Using Eqs. (19) through (22), it may be written in which the factor of 2 inside the brackets in Eqs. (110) and (111) comes from the fact that both individuals die in the transition to tip = x2a(n) + x(1 — x)13(n)+ x(1 — x)c(n)+ (1 — x)2d(n). (122) state 0. Individuals perish in the game, which causes x and y to Note that is proportional to 4, in Eqs. (110) and (I11). Finally. decrease. This is balanced by reproduction, by the positive terms accounting as in Eq.( 114) for the fact that only one member ofeach xib and yO. As there are only two types, y = 1 - x. sox + y = 1 intact AB pair is A and El in Eq. (122) for the average fitness, the and +j, = 0. Then using these to solve for ib and substituting into frequency of A after the game is Eq. (110) gives aC = (x2a(n) + x(1 — x)b(n))/Cb. (123) =2x(1 — x)([Prob(AB 0) + Prob(AB A) Then the change in the frequency of A over one generation, Ax = — Prob(AA 0) — Prob(AA A)/2]x — x, is given by + [Prob(BB 0) + Prob(BB B)/2 Ax =x(1 — x)((a(n)— c(n)fx + [b(n)— d(n)1(1 — x))/ii.• (124) — Prob(AB 0) — Prob(AB B)](1 — x)). (112) which is presented in the main text as Eq. (27). Note that pairs of terms in the brackets in Eq. (112) are related References individual survival probabilities. For example, Prob(AB 0) + Prob(AB —. A) = 1 — c(n). In addition, because the choice of Axelrod. K.. 1954. the Evolution of Cooperation. Basic Books. New York. NY. Revised time scale here is arbitrary, we may drop the leading factor of 2 edition published in 2006. in Eq. (112). Making these changes yields Cressman. R. 2005. Evolutionary Dynamics and Extensive Form Games. MIT Press. Cambridge. Massachusetts. if = - x)(fa(n)— c(n)ix + fb(n) — d(n)K1 — x)) (113) De Jaegher. K.. 2017. Harsh environments and the evolution of multi-player coop- eration. Theor. Popul. Biol. 113. 1-12. which is identical to Eq. (1) but now with payoffs from the n-step De Jaegher. K.. Moyer. 8..2016. By-product mutualism and the ambiguous effects of game. harsher environments - A game-theoretic model. J. Theoret. Biol. 393.82-97. Doebeli. M.. Haunt. C. 2005. Models of cooperation based on the Prisoners Dilemma and the Snowdrift game. Ecol. Lett. 8.748-766. Classical population-genetic model Dunn. LC. 1953. Variations in the segregation ratio as causes of variations of gene frequency. Acta Genet. Statist. Med. 4. 139-147. The replicator equation is a model of instantaneous, infinitesi- Eshel. Shaked. A..2001. Partnership. J. Theoret. Biol. 208.457-474. mal change in a population. Alternatively, we may follow the fates Eshel.I.. Weinshall. D.. 1988. Cooperation in a repeated game with random payment function. J. Appl. Probab. 25.478-491. of pairs of individuals when generations are non-overlapping, as Ewer's. W.J.. 2004. Mathematical Population Genetics. Volume I: Theoretical Foun- in classical models of diploid viability selection. The population dations. Springer-Verlag. Berlin. is still assumed to be infinite but time is now measured in units Fisher. RA. 1930. The Genetical Theory of Natural Selection. Clarendon. Oxford. of discrete generations. As a straightforward extension of classical Garay. J.. 2009. Cooperation in defence against a predator. J. Theoret. BioL 257. models (Fisher. 1930; Haldane. 1932; Wright, 1931), we assume 45-51. Haldane. J.B. S., 1932. The Causes of Natural Selection. Longmans Green & Co.. that the population is well-mixed and that at the start of each London. generation it is composed of pain AA, AB and BB in frequencies x2. Hamilton. W.D.. 1971. Geometry for the selfish herd. J.Theoret. Biol. 31.295-311. 2x(1 — x) and (1 — 4 2. The frequency of A is computed from the Harms. W.. 2001. Cooperative boundary populations: the evolution of cooperation pair frequencies as on mortality risk gradients. J. Theoret. Biol. 213.299-313. Haul. D.L. 1974. Genetic dissection of segregation distortion. I. Suicide combina- 1 tions of 50 genes. Genetics 76. 477-486. x = s.2 + 2x(1 — x)-2 (114) Hauer. C.. Michor. F.. Nowak. M.. Doebeli. M.. 2006. Synergy and discounting in social dilemmas. J. Theoret. Biol. 76.477-486. because both members ofAA are A but only one member of AB is A. Hofbauer.J..Schuster. P.. Sigmund. K.. 1979. A note on evolutionary stable strategies We emphasize that the accounting is of pain. The total population and game dynamics. J. Theoret. Biol. 81.609-612. is represented by Hofbauer. J.. Sigmund. K.. 1988. The Theory of Evolution and Dynamical Systems. Cambridge University Press. Cambridge UK. + 2x( 1 — x) + (1 — x)2 = 1. (115) Hofbauer. J.. Sigmund. K.. 1998. Evolutionary Games and Population Dynamics. Cambridge University Press. Cambridge UK. After selection, which is to say after the game, the population is Kropotkin. P.. 1902. Mutual Aid: A Factor of Evolution. Heinemann. London. composed of some intact pain with type AA. AB or BB and some Kun. A.. Boza. G..Scheuring. L.2006. Asynchronous snowdrift game with synergistic loners with type A or B. Their relative frequencies are effect as a model of cooperation. Behay. Ecol. 17.642-650. Lessard. S.. 2005. Long-term stability from fixation probabilities in finite popula- AA : iProb(AA AA) tions: New perspectives fro ESS theory. Theoret. Pop. Biol. 68. 19-27. Maynard Smith. J.. 1978. The evolution of behavior. Sci. Am. 239. 176-192. AB : 2.2c(1 — x)Prob(AB AB) Maynard Smith. J.. Price. G.R.. 1973. The logic of animal conflict. Nature 246. 15-18. BB : (1 — x)2Prob(BB BB) Mesterton-Gibbons. M..2000. An Introduction to Came-Theoretic Modelling. Amer- ican Mathematical Society. Providence. Rhode Island. A : x2Prob(AA A) + 2x(1 — x)Prob(AB A) Moran. PAP.. 1958. Random processes in genetics. Proc. Ca mb. Phil. Soc. 54, B : 2x( 1 — x)Prob(AB B)i- (1 — x)2Prob(BB B). 60-71. EFTA00810758 J. Wakeley and AI. Nowak/Theoretical Population Biology OS (2019)39-SS 55 Moran. P.A.P.. 1962. Statistical Processes of Evolutionary Theory. Clarendon Press. Sandler. L. Novitsky, E.. 1957. Meiotic drive as an evolutionary force. Am. Nat. 91. Oxford. 105-110. Nagylaki.T.. 1992. Introduction to 'Theoretical Population Genetics. Springer-Verlag. Schecter. S.. Gintis. H.. 2016. Game Theory in Action: An Introduction to Classical Berlin. and Evolutionary Models. Princeton University Press. Princeton. New Jersey. Nowak. M.A. 2006a. Evolutionary Dynamics. Harvard University Press. Cambridge. Schuster. P.. Sigmund. IC. 1983. Replicator dynamics. J. Theor. Biology 100.533-538. Massachusetts. Slcyrms. B.. 2004. The Stag Hunt and Evolution of Social Structure. Cambridge Nowak. MA. 2006b. Five rules for the evolution of cooperation. Science 314. University Press. Cambridge. UK. 1560-1563. Smaldino. P.L.Schank.J.C.. McElreath. R..2013. Increased costs of cooperation help Nowak. MA. 2012. Evolving cooperation. J. Theoret. Biol. 299. 1-8. cooperators in the long run. Am. Nat. 181.451-463. Nowak. M.A.. Sasaki. A. Taylor. C.. Fudenberg. D.. 2004. Emergence of cooperation Taylor. P.D.. Jon ker. L. 1978. Evolutionarily stable strategies and game dynamics. and evolutionary stability in finite populations. Nature 428.646-650. Math. Biosa. 40. 145-156. Nowak. MA. Sigmund. IC. 1993. A strategy of win-stay. lose-shift that outperforms Trivers. R.L. 1971. The evolution of reciprocal altruism. Quart. Rev. Biol. 46. tit for tat in the Prisoners Dilemma game. Nature 364. 56-58. 35-57. Nowak. M.A. Sigmund. K.. 1998. Evolution of indirect reciprocity by image scoring. Tucker.A.. 1950.A tyro-person dilemma. In: Rasmussen. E. (Ed.). Readings in Games Nature 393.573-577. and Information. Blackwell. Oxford. pp. 7-8. Osborne. M.J.. Rubinstein. A.. 1994. A Course in Game Theory. MIT Press. Cambridge. Van Cleve. J.. Akcay.L 2014. Pathways to social evolution: reciprocity. relatedness. Massachusetts. and synergy. Evolution 68.2245-2258. Rapoport. A.. Chammah. A.M.. 1965. Prisoners Dilemma: A Study in Conflict and Wolfram Research. Inc.. 2018. Mathematica. Version 11.2. Champaign. IL Cooperation. University of Michigan Press. Ann Arbor. Michigan. Wright. S.. 1931. Evolution in Mendelian populations. Genetics 16.97-159. Roussel. F.. Billiard. S.. 2000. A theoretical basis for measures of kin selection in Wu. B.. Altrock. P.M.. Wang. L Traulsen. A.. 2010. Universality of weak selection. subdivided populations: finite populations and localized dispersal. J. Evol. Biol. Physical Rev. E 82. 046106. 13.814-825. Zeeman. E.C. 1980. Population dynamics from game theory. In: Nitecki. 2.. Robin- Sandholm. W.H.. 2010. Population Games and Evolutionary Dynamics. MIT Press. son. R.0 (Eds.). Global Theory of Dynamical Systems. In: Lecture Notes in Cambridge. Massachusetts. Mathematics. voL 819. Springer-Verlag. Berlin. pp. 471-497.(Proceedings of an Sandler. L. Hiraizumi. Y.. 1960. Meiotic drive in natural populations of Drosophila International Conference Held at Northwestern University. Evanston. Illinois. melanogaster. V. On the nature of the SD region. Genetics 45. 1617-1689. June 18-22. 1979). EFTA00810759