Cooperative Stochastic Games with Mean-Variance Preferences

Abstract

In stochastic games, the player’s payoff is a stochastic variable. In most papers, expected payoff is considered as a payoff, which means the risk neutrality of the players. However, there may exist risk-sensitive players who would take into account “risk” measuring their stochastic payoffs. In the paper, we propose a model of stochastic games with mean-variance payoff functions, which is the sum of expectation and standard deviation multiplied by a coefficient characterizing a player’s attention to risk. We construct a cooperative version of a stochastic game with mean-variance preferences by defining characteristic function using a maxmin approach. The imputation in a cooperative stochastic game with mean-variance preferences is supposed to be a random vector. We construct the core of a cooperative stochastic game with mean-variance preferences. The paper extends existing models of discrete-time stochastic games and approaches to find cooperative solutions in these games.

1. Introduction

The class of stochastic games was initially introduced by L. Shapley [1]. He considered two-player zero-sum stochastic games with finite state space and finite strategy spaces and proved the existence of optimal stationary strategies when players’ payoff function is discounted mathematical expected payoff. This result has been extended to the case of n-player games by Fink [2], Takahashi [3], Sobel [4]. The results in stochastic games of the recent 60 years, mostly for non-cooperative case, are discussed by Neyman and Sorin [5], Solan [6], and Solan and Vieille [7].

Players’ payoffs in stochastic games are stochastic variables. In most papers, the expected discounted payoff is considered as a measure of a player’s payoff in a stochastic game. In this case, it is supposed that the players are risk neutral. Hence, the risk averse and risk loving attitudes cannot be modeled considering only expectation as a criterion. We propose a model of a stochastic game in which the players’ attentions to risk are taken into account. We use a mean-variance preference popular in decision making theory. In particular, this type of preference for random payoffs is used in financial risk management, in a portfolio decision theory proposed by Markowitz [8]. The risk allocation problem in portfolio management based on cooperative game theory approach is also examined by Csóka et al. [9].

We consider n-person stochastic games with finite sets of actions in any state of the game. A mean-variance stochastic game in discrete time with the payoff function, which is a linear combination of expectation and standard deviation of the payoff, is proposed. The goal of the paper is to construct the cooperative version of a mean-variance stochastic game and define the core as a cooperative solution of such a cooperative game. In the paper, we first define the players’ payoffs according to mean-variance preferences in a non-cooperative setting, and second, we define a cooperative game in the form of characteristic function. Then, we determine a solution of a cooperative stochastic game using the method proposed by Suijs and Borm [10], and Suijs et al. in [11]. The contribution of the paper is in proposing the theory of cooperative stochastic games with mean-variances preferences. The research question about the stability of the cooperative solution over time is open and briefly discussed in Section 5.

First, we consider a non-cooperative stochastic game with mean-variance utility function. Second, we define the cooperative version of the game constructing characteristic function using maxmin approach (see Neumann and Morgenstern [12]). This approach supposes that the value of characteristic function for any coalition is the maxmin payoff in zero-sum stochastic game between the coalition and anti-coalition. As the players may have different coefficients before standard deviations in the utility functions, the summation of the utilities does not make sense. When a coalition is formed, the maximal utility of the coalition payoff over the coalition members is used to measure the utility of the coalition payoff. There are other approaches of defining characteristic function in dynamic games proposed by Gromova and Petrosyan [13], Petrosjan and Zaccour [14] and examined by Reddy and Zaccour [15]. However, applications of these approaches need corrections, considering the stochastic nature of payoffs and mean-variance utility functions. Cooperative stochastic games with expected payoff as a utility function are introduced by Petrosjan [16] and Petrosjan and Baranova [17], and then extended by Parilina [18] and Parilina and Tampieri [19] by examining the stability of cooperative solutions in stochastic dynamic settings. As proposed by Suijs et al. [11], we consider the core as a solution of a cooperative game. The conditions of node-consistency of the core in dynamic games played over event trees with stochastic nature are obtained by Parilina and Zaccour [20]. Recently, the sufficient conditions of strong subgame consistency of the core are obtained by Parilina and Petrosyan [21].

Then, we define a solution of a cooperative stochastic game. The idea is briefly described below. Suijs et al. introduce a new class of cooperative games arising from cooperative decision making problems in the stochastic environment [10,11]. They introduce some types of preferences to order stochastic payoffs and model the games with risk neutral, as well as risk averse and risk loving players. Suijs et al. in [11] define the core of the cooperative game with stochastic payoffs and obtain the necessary and sufficient conditions for the non-emptiness of the core. The deterministic equivalent of a stochastic payoff with some properties is introduced by Suijs and Borm [10]. The nucleolus for this type of games is introduced by Suijs et al. [11]. The Shapley-like solutions for the cooperative game with stochastic payoffs are proposed by Timmer et al. [22]. The model of transferable utility game with uncertainty is introduced by Habis and Herings [23]. They propose a solution concept of a weak sequential core for this class of games and show that it is non-empty if all ex post TU-games are convex.

The goal of the paper is to propose a scheme of construction of a cooperative version of a stochastic game, in which not only expectation but also standard deviation of players’ payoffs is taken into account. The idea of considering risk-sensitive players recently appears in different variations in some papers on dynamic and differential games. The risk sensitive nonzero-sum continuous time stochastic games are considered by Wei [24], where the existence of Nash equilibrium is proved. The risk-sensitive average payoff criterion for zero-sum (see Bäuerle and Rieder [25]) and for the nonzero-sum discrete-time stochastic games with a denumerable state space (see Wei and Chen [26]), in which an existence of the Nash equilibrium is examined. The risk sensitive Nash equilibria in stochastic games are discussed by Nowak [27]. The class of risk sensitive stochastic games with overlapping generations is considered by Jaśkiewicz and Nowak [28]. Recently, the Nash equilibria for stochastic games of resource extraction in which risk coefficient is included into utility function are found by Asienkiewicz and Balbus [29]. In mean-field games, the model with both expectation and variance in payoff criteria is proposed by Saldi et al. [30].

The paper is organized as follows. The class of mean-variance stochastic games in discrete time is introduced in Section 2. The propositions providing formulae to calculate a variance and covariance of players’ payoffs in stochastic games are given in Section 3. In Section 4, we define a cooperative mean-variance stochastic game and an imputation as a cooperative solution of the game. We briefly conclude in Section 5. The proofs of all propositions are given in the appendix.

2. Stochastic Games with Mean-Variance Preferences

2.1. Stochastic Game

Define a finite stochastic game G using the following notations. The finite set of players is $N=\{1,\dots ,n\}$, and the finite set of states is $S=\{1,2,\dots ,t\}$. In any state $s\in S$, an n-person normal-form game $\langle N,{\{A_i^s\}}_{i\in N},{\{K_i^s\}}_{i\in N}\rangle$ is defined, where $A_i^s$ is the finite set of Player i’s actions in state s, $K_i^s\left(a_1^s,\dots ,a_n^s\right)=K_i^s(a^s)\in \mathbb{R}$ is a payoff function of Player i in state s, $i\in N$, $s\in S$. The action profile $a^s=(a_1^s,\dots ,a_n^s)$ belongs to the set $A^s={\prod }_{i\in N}{A}_i^s$.

The transition map $\pi$ is $T\to \Delta (S)$ function, where $T=\{(s,a^s)|s\in S;a^s\in A^s\}$ and $\Delta (S)$ is a probability distribution on S. For each pair $(s,a^s)$, mapping $\pi$ identifies the vector of probabilities $(\pi (1|s;a^s)$, …, $\pi (s^{\prime }|s;a^s)$, …, $\pi (t|s;a^s))$, where $\pi (s^{\prime }|s;a^s)$ is the probability that state $s^{\prime }$ will be realized at the next stage if at the present stage (state s) action-profile $a^s$ has been realized, $\pi (s^{\prime }|s;a^s)⩾0$ and ${\sum }_{s^{\prime }\in S}\pi (s^{\prime }|s;a^s)=1$ for any $a^s\in A^s$ and $s\in S$.

Let vector $p^0=(p_1^0,\dots ,p_t^0)$ be a vector of initial distribution on S, and $p_s^0$ is the probability that state s will be realized at the first stage of the game, $p_s^0⩾0$ for any $s\in S$, and ${\sum }_{s\in S}{p}_s^0=1$.

Let all players have the same discount factor $\delta \in (0,1)$ for their payoffs.

Definition 1.

The stochastic game G is defined by

$$〈N,S,\pi ,p^0,\delta 〉.$$

(1)

Definition 2.

Stochastic subgame $G^s$ is a finite stochastic game $(1)$ with initial distribution vector $p^0$, such that $p_s^0=1$. Subgame $G^s$ is a stochastic game starting from state $s\in S$.

The implementation of a stochastic game can be briefly described in the following way. First, the chance moves and chooses the initial state of the game according to vector $p^0$. Let the initial state be $s\in S$. Second, the first stage of game G is realised, i.e., the normal-form game $\langle N,{\{A_i^s\}}_{i\in N}$, ${\{K_i^s\}}_{i\in N}\rangle$ is played. Players simultaneously choose their actions from the action sets ${\{A_i^s\}}_{i\in N}$. Let Player i’s action in state s be $a_i^s\in A_i^s$, $s\in S$. Therefore, an action profile $a^s=(a_1^s,\dots ,a_n^s)\in {\prod }_{i\in N}{A}_i^s$ is implemented. Player i obtains her payoff $K_i^s(a^s)$, $i\in N$. Then, the transition to the next stage is realized. The probability that state $s^{\prime }$ will appear at the next stage is equal to $\pi (s^{\prime }|s;a^s)$, $s^{\prime }\in S$. After that, the game proceeds in the described way.

Assume that players know all the parameters of the game. We consider a class of stationary strategies. Let Player i’s strategy in game G be a mapping ${\eta }_i(·)$, such that ${\eta }_i(s)=a_i^s\in A_i^s$, $s\in S$. Denote the set of stationary strategies of Player i as $H_i$. The stationary strategy defines an action at any stage depending only on which state is realized at the stage, i.e., ${\eta }_i:s⟼a_i^s\in A_i^s$, $s\in S$, and not depending on the stage number and the history of the stage. A vector of stationary strategies is a strategy profile $\eta =({\eta }_1,\dots ,{\eta }_n)\in H_1\times \dots \times H_n$ in stochastic game G. Obviously, $\eta$ is a strategy profile in any subgame $G^s$.

2.2. Mean-Variance Payoff Functions

Players’ payoffs in stochastic games are random variables and they are functions of stationary strategy profile $\eta$. Denote Player i’s payoff in stochastic game G as ${\overline{\xi }}_i={\overline{\xi }}_i(\eta )$ and Player i’s payoff in stochastic subgame $G^s$ as ${\xi }_i^s={\xi }_i^s(\eta )$ for any $s\in S$. Random variables ${\overline{\xi }}_i$ and ${\xi }_i={({\xi }_i^1,\dots ,{\xi }_i^t)}^{\prime }$ satisfy the following equation:

$${\overline{\xi }}_i(\eta )=p^0{\xi }_i(\eta ).$$

To find a player’s “optimal” strategy, it is necessary to determine how to compare random payoffs. The most common way is to measure a random payoff by its mathematical expectation, assuming the risk neutrality of the players. Strictly speaking, ${\xi }_i$ is preferred to ${\psi }_i$ or ${\xi }_i{\succ }_E{\psi }_i$, if and only if $E({\xi }_i)>E({\psi }_i)$, where $E(\xi )$ is the mathematical expectation of random variable $\xi$. This preference is complete and transitive but it does not allow one to model different attentions to risk.

Example 1.

One may compare random payoffs using the principle of stochastic domination. Let ${\xi }_i$, ${\psi }_i\in L^1(\mathbb{R})$ be random payoffs of Player i and $F_{{\xi }_i}(·)$ and $F_{{\psi }_i}(·)$ be cumulative distribution functions of ${\xi }_i$ and ${\psi }_i$, respectively. It is said that ${\xi }_i$ stochastically dominates ${\psi }_i$ and write ${\xi }_i{\succsim }_F{\psi }_i$, if and only if, for all $x\in \mathbb{R}$ inequality $F_{{\xi }_i}(x)⩽F_{{\psi }_i}(x)$ holds. This preference is incomplete, therefore, random payoffs can be incomparable with this preference. Another disadvantage of this preference is its computational complexity for a class of stochastic games. Thus, its application to the theory of stochastic games is questionable.

We use the method of ordering random payoffs proposed in [10,11]. Let every player i have a utility function $u_i$ of a random payoff such that $u_i:L^1(\mathbb{R})\to \mathbb{R}$. For random payoffs $\xi ,\psi \in L^1(\mathbb{R})$, payoff $\xi$ is preferred or equivalent to payoff $\psi$ by Player i if and only if $u_i(\xi )⩾u_i(\psi )$. In [10] the preferences ${({\succsim }_i)}_{i\in N}$ are considered for the cooperative games with stochastic payoffs and they are such that for any player $i\in N$ there exists a utility function $u_i:L^1(\mathbb{R})\to \mathbb{R}$ satisfying properties:

• For all $\xi \in L^1(\mathbb{R})$: $\xi {\sim }_i{u}_i(\xi )$.
• For all $\xi ,\psi \in L^1(\mathbb{R})$: $\xi {\succsim }_i\psi$ if and only if $u_i(\xi )⩾u_i(\psi )$.
• For all $d\in \mathbb{R}$: $u_i(d)=d$.
• For all $\xi \in L^1(\mathbb{R})$: $u_i(\xi -u_i(\xi ))=0$.
• For all $\xi \in L^1(\mathbb{R})$ and all d, $d^{\prime }\in \mathbb{R}$ with $d<d^{\prime }$: $u_i(d+\xi )<u_i(d^{\prime }+\xi )$.

Expression $\xi {\sim }_i{u}_i(\xi )$ means that the random payoff $\xi$ is equivalent to $u_i(\xi )$, and $\xi {\succsim }_i\psi$ means that random payoff $\xi$ is preferred or equivalent to random payoff $\psi$ for Player i. In [10], the function $u_i(·)$ is called a deterministic equivalent of the preference. In our work, we call function $u_i(\xi )$ the utility function of a random payoff $\xi$. Obviously, the utility function $u_i({\xi }_i)=E({\xi }_i)$ satisfies properties 1–5 given above.

Example 2.

One may consider a preference ${\succsim }_{\alpha }$ such that for any ξ, $\psi \in L^1(\mathbb{R})$ the relation $\xi {\succsim }_{\alpha }\psi$ holds if α-quantile of c.d.f. $F_{\xi }$ of a random variable ξ is not less than the α-quantile of c.d.f. $F_{\psi }$, i.e., $u(\xi )=u_{\alpha }^{\xi }=sup\{t|F_{\xi }(t)<\alpha \}$. This utility function also satisfies properties 1–5 given above.

Let Player i’s utility function of random payoff ${\xi }_i$ be

$$u_i({\xi }_i)=E({\xi }_i)+b_i{\left(Var({\xi }_i)\right)}^{\frac{1}{2}},$$

(2)

where $Var({\xi }_i)$ is the variance of ${\xi }_i$, parameter $b_i\in \mathbb{R}$ shows player’s attention to risk. Note that if $b_i=0$, it is understood as Player i is risk neutral. If $b_i>0$, then Player i is risk loving, and if $b_i<0$, then Player i is a risk averse person. Denote the preference corresponding to utility function (2) as ${\succsim }_{b_i}$. The utility function (2) satisfies properties 1–5.

Preference ${\succsim }_{b_i}$ is complete but not implied by ${\succsim }_F$. The preference ${\succsim }_{b_i}$ is attractive for applications because of its computation simplicity and possibility of modeling attention to risk. If $b_i=0$ for any $i\in N$, utility function (2) becomes $u_i({\xi }_i)=E({\xi }_i)$, which coincides with preference ${\succsim }_E$.

2.3. Stochastic Games with Mean-Variance Preferences

We define a stochastic game in which any player $i\in N$ has a utility function $u_i$:$L^1(\mathbb{R})\to \mathbb{R}$ of form (2).

Definition 3.

Stochastic game $G_b$ with mean-variance preferences is defined by tuple

$$G_b=〈N,S,\pi ,p^0,\delta ,b〉,$$

(3)

where $b=(b_1,\dots ,b_n)\in {\mathbb{R}}^n$ is a profile of parameters determining players’ utility functions (2) for any $i\in N$.

By $G_b^s$, $s\in S$, we define the subgame of stochastic game (3) with $p^0=(p_1^0,\dots ,p_t^0)$ such that $p_s^0=1$.

In the next section, we present the formulae to calculate the utilities of random payoffs in stochastic game (3) and any subgame $G_b^s$. We also consider the case when players form a coalition and calculate its summarized stochastic payoff.

3. Expectation and Variance of Random Payoffs in Stochastic Games

Consider a strategy profile $\eta =({\eta }_1,\dots ,{\eta }_n)$, where ${\eta }_i$ is a stationary strategy of Player i, such that ${\eta }_i(s)=a_i^s\in A_i^s$ for all $s\in S$, $i\in N$. First, find expectation $E({\xi }_i^s)$ of Player i’s payoff in subgame $G^s$:

$$E({\xi }_i^s(\eta ))=K_i^s(a^s)+\delta \sum _{s^{\prime }\in S}\pi (s^{\prime }|s;a^s)E({\xi }_i^{s^{\prime }}(\eta ))$$

or in a vector form:

$$E({\xi }_i(\eta ))=K_i(a)+\delta \mathsf{\Pi }(\eta )E({\xi }_i(\eta )),$$

(4)

where $E({\xi }_i(\eta ))={(E({\xi }_i^1(\eta )),\dots ,E({\xi }_i^t(\eta )))}^{\prime }$, $K_i(a)={(K_i^1(a^1),\dots ,K_i^t(a^t))}^{\prime }$. Equation $(4)$ is equivalent to the following one:

$$E({\xi }_i(\eta ))={(\mathbb{I}-\delta \mathsf{\Pi }(\eta ))}^{-1}{K}_i(a),$$

(5)

where $\mathbb{I}$ is $t\times t$ identity matrix and $\mathsf{\Pi }(\eta )$ is a matrix of transition probabilities in stochastic game (1), which is a function of strategy profile $\eta$ taking a form of

$$\mathsf{\Pi }(\eta )=\left(\begin{array}{ccccc}\pi (1|1;a^1)& \dots & \pi (s|1;a^1)& \dots & \pi (t|1;a^1)\\ ⋮& & \ddots & & ⋮\\ \pi (1|t;a^t)& \dots & \pi (s|t;a^t)& \dots & \pi (t|t;a^t)\end{array}\right).$$

Matrix ${(\mathbb{I}-\delta \mathsf{\Pi }(\eta ))}^{-1}$ is denoted as $\widehat{\mathsf{\Pi }}(\eta )$ and it always exists for $\delta \in (0,1)$.

Remark 1.

All the eigenvalues of stochastic matrix $\mathsf{\Pi }(\eta )$ are from the interval $[-1,1]$. For the existence of matrix $\widehat{\mathsf{\Pi }}(\eta )$, it is necessary and sufficient that the determinant of matrix $(\mathsf{\Pi }(\eta )-\mathbb{I}/\delta )$ be not equal to zero. Therefore, matrix $(\mathsf{\Pi }(\eta )-\mathbb{I}/\delta )$ must not have an eigenvalue equal to $1/\delta$. The last condition is satisfied because $1/\delta >1$. Therefore, it cannot be an eigenvalue of stochastic matrix $\mathsf{\Pi }(\eta )$.

An expectation of Player i’s payoff in stochastic game G equals

$$E({\overline{\xi }}_i(\eta ))=p^{0}E({\xi }_i(\eta ))=p^0\widehat{\mathsf{\Pi }}(\eta )K_i(a).$$

(6)

The variance $Var({\overline{\xi }}_i(\eta ))$ of Player i’s payoff ${\overline{\xi }}_i(\eta )$ in game G can be found using the following proposition.

Proposition 1.

Let in stochastic game G or $G_b$, Player i’s strategy be ${\eta }_i\in H_i$. The variance of Player i’s payoff is

$$\begin{array}{cc}\hfill Var({\overline{\xi }}_i(\eta ))=& (p^0\tilde{\mathsf{\Pi }}(\eta )\circ K_i(a))(2\widehat{\mathsf{\Pi }}(\eta )-\mathbb{I})K_i-{(p^0\widehat{\mathsf{\Pi }}(\eta )K_i(a))}^2,\hfill \end{array}$$

(7)

where operation $A\circ B$ is the Hadamard product (Let A and B be $m\times n$ matrices with entries in $\mathbb{R}$. The Hadamard product of A and B is an $m\times n$ matrix defined by ${[A\circ B]}_{i,j}={\left[A\right]}_{i,j}{\left[B\right]}_{i,j}$ for all $1⩽i⩽m$, $1⩽j⩽n$.) of matrices A and B of the same size, and $\tilde{\mathsf{\Pi }}(\eta )={(\mathbb{I}-{\delta }^2\mathsf{\Pi }(\eta ))}^{-1}$.

Proof. 

See Appendix A.1. □

Corollary 1.

The variance $Var({\xi }_i^s(\eta ))$ of Player i’s payoff in subgame $G^s$, $s\in S$, can be found by formula $(7)$ substituting $p^0=(p_1^0,\dots ,p_t^0)$ s. t. $p_s^0=1$ and $p_{s^{\prime }}^0=0$ for any $s^{\prime }\ne s$, $s^{\prime }\in S$.

For further calculations, in particular, for the construction of a cooperative form of stochastic game $G_b$ and subgames $G_b^s$, it is required to determine the random payoff of any coalition $C\subset N$, which is denoted as ${\overline{\xi }}_C$ and equals

$${\overline{\xi }}_C(\eta )=\sum _{i\in C}{\overline{\xi }}_i(\eta ),$$

and for subgame $G^s$:

$${\xi }_C^s(\eta )=\sum _{i\in C}{\xi }_i^s(\eta ).$$

We should notice that players’ payoffs ${\xi }_i^s$ and ${\xi }_j^s$ are not independent variables for $i\ne j$, $i,j\in C$. The following proposition gives the formulae to calculate an expectation and variance of random payoff ${\overline{\xi }}_C(\eta )$ of any coalition C.

Proposition 2.

The expectation of payoff ${\overline{\xi }}_C(\eta )$ of coalition C in stochastic game G or $G_b$ is

$$E({\overline{\xi }}_C(\eta ))=p^0\widehat{\mathsf{\Pi }}(\eta )\sum _{i\in C}{K}_i(a).$$

(8)

The variance of payoff ${\overline{\xi }}_C(\eta )$ of coalition C in stochastic game G or $G_b$ is

$$Var({\overline{\xi }}_C(\eta ))=\sum _{i\in C}Var({\overline{\xi }}_i(\eta ))+2\sum _{\begin{array}{c}l<m,\\ l,m\in C\end{array}}Cov({\overline{\xi }}_l(\eta ),{\overline{\xi }}_m(\eta )),$$

(9)

where variance $Var({\overline{\xi }}_i(\eta ))$ of Player i’s payoff is defined by formula (7), and $Cov({\overline{\xi }}_l(\eta ),{\overline{\xi }}_m(\eta ))$ is a covariance of Players l and m’ payoffs, which is equal to

$$\begin{array}{cc}\hfill Cov({\overline{\xi }}_l(\eta ),{\overline{\xi }}_m(\eta ))& =(p^0\tilde{\mathsf{\Pi }}(\eta )\circ K_l(a))(\widehat{\mathsf{\Pi }}(\eta )-0.5\mathbb{I})K_m(a)\hfill \\ \hfill & +(p^0\tilde{\mathsf{\Pi }}(\eta )\circ K_m(a))(\widehat{\mathsf{\Pi }}(\eta )-0.5\mathbb{I})K_l(a)-(p^0\widehat{\mathsf{\Pi }}(\eta )K_l(a))(p^0\widehat{\mathsf{\Pi }}(\eta )K_m(a)).\hfill \end{array}$$

(10)

Proof. 

See Appendix A.2. □

Corollary 2.

The variance of payoff ${\xi }_C^s(\eta )$ of coalition C in subgame $G^s$ or $G_b^s$, $s\in S$, can be found by formula $(9)$ substituting $p^0=(p_1^0,\dots ,p_t^0)$ s. t. $p_s^0=1$ and $p_{s^{\prime }}^0=0$ for any $s^{\prime }\in S$, $s^{\prime }\ne s$.

Equation (6) and Proposition 1 provide formulae to calculate the utilities of players’ payoffs in stochastic games with mean-variance preferences for any strategy profile $\eta$. Proposition 2 provides formulae to calculate the utilities of coalition payoffs in stochastic games with mean-variance preferences.

4. Cooperative Stochastic Games with Mean-Variance Preferences

In this section, we construct a cooperative version of a stochastic game $G_b$ and subgame $G_b^s$, starting from state $s\in S$. Therefore, we determine characteristic function $v(·):2^N\to \mathbb{R}$ satisfying two properties:

• $v(⌀)=0$,
• (superadditivity) For any disjoint coalitions $C,C^{{}^{\prime }}\subset N$, $C\cap C^{{}^{\prime }}=⌀$, inequality $v(C)+v(C^{{}^{\prime }})⩽v(C\cup C^{{}^{\prime }})$ holds.

The value $v(C)$ can be interpreted as a “worth” or “power” of coalition C when its members play together as a unit. We use “maxmin” approach to determine superadditive characteristic function [12].

The characteristic function for subgame $G_b^s$ is denoted as $v^s(·)$ and defined as

$$v^s(C)=\underset{{\eta }_C}{max}\underset{{\eta }_{N\setminus C}}{min}\underset{i\in C}{max}{u}_i({\xi }_C^s({\eta }_C,{\eta }_{N\setminus C})),$$

(11)

for any coalition $C\subset N$. Here, ${\eta }_C\in {\prod }_{i\in C}{H}_i$ is a stationary strategy of coalition C and ${\eta }_{N\setminus C}\in {\prod }_{i\in N\setminus C}{H}_i$ is a stationary strategy of coalition $N\setminus C$. Strategies $({\eta }_C,{\eta }_{N\setminus C})$ form a strategy profile in any subgame $G_b^s$ and game $G_b$. In Equation (11) and further calculations, the utility function of any player i is of form (2), as required in the definition of stochastic game $G_b$. An expression in the right-handed side of (11); we find maxmin of the maximal value of utility among players from C. Maxmin value is such that coalition C can guarantee by strategy ${\eta }_C$ when coalition $N\setminus C$ plays against coalition C using strategy ${\eta }_{N\setminus C}$.

Characteristic function for game $G_b$ is denoted by $\overline{v}(·)$ and it can be defined for coalition C using the following expression:

$$\overline{v}(C)=\underset{{\eta }_C}{max}\underset{{\eta }_{N\setminus C}}{min}\underset{i\in C}{max}{u}_i({\overline{\xi }}_C({\eta }_C,{\eta }_{N\setminus C})).$$

(12)

This function is equal to the maximal value of utility function of random payoff ${\overline{\xi }}_C({\eta }_C,{\eta }_{N\setminus C})$ (among the players from coalition C), which coalition C can guarantee by itself using stationary strategy ${\eta }_C\in {\prod }_{i\in C}{H}_i$ if coalition $N\setminus C$ plays against C in zero-sum game using strategy ${\eta }_{N\setminus C}\in {\prod }_{i\in N\setminus C}{H}_i$.

Cooperation in stochastic game $G_b$ assumes that players form the grand coalition N and play together to maximize the maximal utility of their total payoff ${\overline{\xi }}_N$. We call the stationary strategy profile ${\eta }^{*}$ in game $G_b$ a cooperative one if

$${\eta }^{*}=arg\underset{\eta \in \prod _{i\in N}{H}_i}{max}\underset{i\in N}{max}{u}_i({\overline{\xi }}_N(\eta )).$$

(13)

Therefore, the value of characteristic function $\overline{v}(N)$ can be rewritten as

$$\overline{v}(N)=\underset{i\in N}{max}{u}_i({\overline{\xi }}_N({\eta }^{*})).$$

(14)

The value $v^s(N)$ of the characteristic function $v^s(N)$ in cooperative subgame $G_b^s$ is

$$v^s(N)=\underset{\eta \in {\displaystyle \prod _{i\in N}}{H}_i}{max}\underset{i\in N}{max}{u}_i({\xi }_N^s(\eta )).$$

(15)

Definition 4.

Cooperative stochastic game with mean-variance preferences is a tuple $\langle N,\overline{v}\rangle$ with characteristic function $\overline{v}:2^N\to \mathbb{R}$ defined by $(12)$ and $(14)$, and $\overline{v}(⌀)=0$.

Definition 5.

Cooperative stochastic subgame with mean-variance preferences is a tuple $\langle N,v^s\rangle$ with characteristic function $v^s:2^N\to \mathbb{R}$ defined by $(11)$ and $(15)$, and $v(⌀)=0$.

In cooperation, players with mean-variance preferences realize a cooperative stationary strategy profile ${\eta }^{*}$ given by (13).

The main problem of cooperative game theory is to find a ”fair” imputation of coalition N’s total payoff. In a classical cooperative game theory, values of characteristic function are non-random variables and there is a variety of solution concepts to determine, in some sense, ”fair” imputation.

The concept of a random payoff imputation is proposed in [11], in which it is supposed that the components of the imputation are random variables. We give a definition of an imputation of a random payoff ${\overline{\xi }}_N({\eta }^{*})$.

Definition 6

(see [11]). An imputation of random payoff ${\overline{\xi }}_N({\eta }^{*})$ is a vector $\phi =({\phi }_1,\dots ,{\phi }_n)$, such that ${\phi }_i=d_i+r_i\left({\overline{\xi }}_N({\eta }^{*})-E({\overline{\xi }}_N({\eta }^{*}))\right)$ and $d\in {\mathbb{R}}^n$, $r\in {\mathbb{R}}^n$ satisfy the following conditions:

1. 

${\displaystyle \sum _{i\in N}}{d}_i=E({\overline{\xi }}_N({\eta }^{*}))$,

2. 

${\displaystyle \sum _{i\in N}}{r}_i=1$ and $r_i⩾0$ for any $i\in N$.

According to Definition 6, the payoff ${\overline{\xi }}_N({\eta }^{*})$ of coalition N in game $G_b$ is a random vector characterized by an allocation of expected payoff $E({\overline{\xi }}_N({\eta }^{*}))$ and allocation of random variable $({\overline{\xi }}_N({\eta }^{*})-E({\overline{\xi }}_N({\eta }^{*}))$, which can be interpreted as a risk of random payoff ${\overline{\xi }}_N({\eta }^{*})$. Denote the set of imputations in cooperative game $G_b$ as $\mathsf{\Phi }(N)$.

We use the statement from [11] about necessary and sufficient conditions that an imputation belongs to the core of cooperative stochastic game $G_b$ with mean-variance preferences as a definition of the core.

Definition 7.

An imputation φ belongs to the core of the cooperative game $G_b$, if and only if the inequality

$$\sum _{i\in C}\left\{d_i+r_i\left(u_i({\overline{\xi }}_N({\eta }^{*}))-E({\overline{\xi }}_N({\eta }^{*}))\right)\right\}⩾\overline{v}(C)$$

holds for any coalition $C\subset N$.

To illustrate the idea of defining the core in cooperative stochastic game with mean-variance preferences, we provide a numerical example demonstrating calculations step by step.

Example 3.

Consider two-person stochastic game with two states $〈N,A_1^s,A_2^s,$ $K_1^s,K_2^s〉$, $s=1,2$, with players’ payoffs:

$$s=1:\left(\begin{array}{cc}(3,5)& (3,10)\\ (10,4)& (8,6)\end{array}\right),s=2:\left(\begin{array}{cc}(2,4)& (2,10)\\ (7,3)& (5,4)\end{array}\right)$$

Transition probabilities are defined in matrices:

$$s=1:\left(\begin{array}{cc}(0.7,0.3)& (0.4,0.6)\\ (0.4,0.6)& (0.3,0.7)\end{array}\right),s=2:\left(\begin{array}{cc}(0.9,0.1)& (0.4,0.6)\\ (0.2,0.8)& (0.3,0.7)\end{array}\right),$$

where element $(i,j)$ of the first (second) matrix consists of transition probabilities from states 1 (2) to the states 1, 2 respectively under condition that Player 1 chooses strategy i and Player 2 chooses strategy j in state 1 (2).

Let discount factor δ be $0.9$ and vector of initial distribution $p^0$ on state set $\{1,2\}$ be $(0.5,0.5)$. Suppose that both players are risk averse and $b_1=-2$, $b_2=-1$.

Using condition $(13)$, we obtain $\overline{\eta }=({\overline{\eta }}_1,{\overline{\eta }}_2)$, where ${\overline{\eta }}_1=(0,0)$, ${\overline{\eta }}_2=(1,1)$, i.e., player 1 chooses action 2 in both states, and player 2 chooses action 1 in both states. The maximal value of the maximal utility function is equal to the following one:

$$\overline{v}(N)=\underset{i\in N}{max}{u}_i({\overline{\xi }}_N(\overline{\eta }))=94.1709.$$

Therefore, imputation defined by $(d,r)$ is in the core of game $G_b$ according to Definition 7 if and only if the following inequality takes place for any $C\subset N$:

$$\sum _{i\in C}(d_i+r_i(u_i({\overline{\xi }}_N(\overline{\eta }))-E({\overline{\xi }}_N(\overline{\eta })))⩾\overline{v}(C).$$

If $C=\{1\}$, the inequality is

$$d_1+r_1(u_1({\overline{\xi }}_N(\overline{\eta }))-E({\overline{\xi }}_N(\overline{\eta })))⩾\overline{v}(\{1\}),$$

and for $C=\{2\}$:

$$d_2+r_2(u_2({\overline{\xi }}_N(\overline{\eta }))-E({\overline{\xi }}_N(\overline{\eta })))⩾\overline{v}(\{2\}).$$

Taking into account Definition 6 of the imputation, the following conditions hold:

$$\begin{array}{cc}\hfill d_1+d_2& =E({\overline{\xi }}_N(\overline{\eta }))=128.2,\hfill \\ \hfill r_1+r_2& =1.\hfill \end{array}$$

The following system defines the core of the cooperative stochastic game with mean-variance preferences:

$$\left\{\begin{array}{cc}& d_1+r_1(60.1419-128.2000)⩾10.9955,\hfill \\ & d_2+r_2(94.1709-128.2000)⩾25.4971,\hfill \\ & d_1+d_2=128.2000,\hfill \\ & r_1+r_2=1,\hfill \end{array}\right.$$

or in equivalent form:

$$\left\{\begin{array}{cc}\hfill & d_1\in [10.9955+68.0581r_1,68.6739+34.0291r_1],\hfill \\ \hfill & d_2=128.2000-d_1,\hfill \\ \hfill & r_1\in [0,1],r_2=1-r_1.\hfill \end{array}\right.$$

(16)

The region of $(r_1,d_1)$, $r_1\in [0,1]$, satisfying the first inequality of the latter system, is depicted in Figure 1.

We can easily notice that the core is non-empty and contains an infinite number of imputations. The imputation of the core is a vector

$$\left(d_1+r_1({\overline{\xi }}_N({\eta }^{*})-128.2),d_2+r_2({\overline{\xi }}_N({\eta }^{*})-128.2)\right),$$

where $d_1,d_2,r_1,r_2\in \mathbb{R}$ and satisfy (16).

5. Conclusions

We have constructed a model of cooperative stochastic games with mean-variance preferences. The utility function for mean-variance preference takes into account players’ attention to risk. The risk sensitivity of a player is modeled by a linear combination of mathematical expectation and standard deviation of the payoff. We have provided formulas to calculate player’s and coalition utilities for any stationary strategy profile. We have defined a cooperative stochastic game with mean-variance preferences in the form of characteristic function determined by a maxmin approach. A particular cooperative solution proposed for the game is the core. The natural research question arising in the area of dynamic games is the sustainability and time-consistency of the cooperative solution over time. The problem of stability of a cooperative agreement in stochastic games with mean preferences is discussed by Parilina and Tampieri [19], Parilina [18], Parilina and Petrosyan [21], where conditions of stable cooperation are examined. The other research question is how to determine another cooperative solution like the Shapley value, nucleolus, etc., for stochastic games with mean-variance preferences and how to sustain them over time.