You are on page 1of 16

396

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

International Journal of Physical and Mathematical Sciences


journal homepage: http://icoci.org/ijpms

A Brief Introduction to Differential Games


L. Gmez Esparza, G. Mendoza Torres, L. M. Saynes Torres.
Facultad de Ciencias de la Electrnica. Benemrita Universidad Autnoma de Puebla.

1. Introduction
The theory of dynamics games is concerned with multi-person decision making. The
principal characteristic of a dynamic game is that involves a dynamic decision process evolving
in time (continuous or discrete), with more than on decision maker, each with its own cost
function and possibly having access to different information. Dynamic game theory adopts
characteristics from game theory and optimal control theory, although it is much more versatile
than each of.
Differential games belong to a subclass of dynamic games called games in the state space.
In a game in the state space, the modeler introduces a set of variables to describe the state of a
dynamic system, at any particular instant of time in which the game takes place. The systematic
study of the problems of differential games was initiated by Isaacs in 1954.
After development of the maximum principle of Pontryagin's maximum principle, it became
clear that there was a connection between differential games and optimal control theory. In fact,
the differential game problems are a generalization of the optimal control problems in cases
where more than one driver or player. However, differential games are conceptually much more
complex than optimal control problems in that it is not as what constitutes a solution. There are
different kinds optimal solutions for problems such as differential games minimax solution,
Nash equilibrium, Pareto equilibrium, depending on the characteristics of the games (see e.g.,
Tolwinski (1982) and Haurie, Tolwinski, and Leitman (1983)).
We present some results on differential games cooperative and non-cooperative differential
games, and theirs "optimal" solutions. In particular we will study those that relate Pareto
equilibrium and Nash equilibrium (non-cooperative games), although other types of cooperative
and non-cooperative games, for example, commitment games, Stackelberg games, to name a
few.
2. Preliminary in optimal control theory
As mentioned above, optimal control problems are a special class of differential games
played and a cost criterion. In this section we study some basic results on optimal control theory:
dynamic programming and the maximum principle, since these results are determining in
dynamic game theory.

397

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

2.1. Statement of optimal control problem (OCP)


In general, the optimal control problem (continuous time) can be defined as follows

(1)

where

is called the state equation and

is called the objective function or cost criteria. This is, in own words, the problem is find the
admissible control , which Maximizes the objective function, subject to the state equation and the
control constraints
(2)
Usually the set
is determined by constraints (physical, economic, biological, etc.) on the
values of the control variables
at time . The control is called the optimal control
and , determined by means of state equation with
, is called the optimal trajectory or an
optimal path.
2.2. Dynamic Programming and the Maximum Principle.
Dynamic programming is based on Bellman's principle of optimality (Richard Bellman in
1957 stated this principle in his book on dynamic programming)
An optimal policy has the property that, whatever the initial
state and initial conditions are, the remaining decision must
constitute an optimal policy with regard to the outcome
resulting from the first decision.
Let us consider the optimal control problem (1). The principle of maximum can be derived from
Bellman's principle of optimality (see [45]). We state the principle of maximum as follows

Theorem 1. Let us assume, that exists an optimal couple


for the optimal control
problem (1), and we assume that
and are continuously differentiable in and continuous in
and . Then, exists an adjoint variable
that satisfies

398

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

(3)

(4)

(5)
where the so-called Hamiltonian is defined as
(6)

The maximum principle states that under certain assumptions there exists for every optimal
control path
a trajectory
such that the maximum condition, the adjoin equation, and
transversality condition (eq. 4) are satisfied. To obtain a sufficiency theorem we augment these
conditions by convexity assumptions. This yields the following theorem.
Theorem 2. Consider the optimal control problem given by the equation (1), (2), (3) and define
the Hamiltonian function like in (7), and the maximized Hamiltonian function
(7)

Assume that the state space is a convex set and that is continuously differentiable and
concave. Let
be a feasible control path with corresponding state trajectory
. If there exists
an absolutely continuous function
such that the maximum condition
(8)
the adjoint equation
(9)

and the transversality condition


(10)

are satisfied, and such that the function


is concave and continuously
differentiable with respect to for all
, then
is an optimal path. If the set of feasible
controls,
does not depend on , this result remains true if equation (10) is replaced by

399

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

(11)

3.

Differential games: basic concepts

The general -player (deterministic) differential game time is described by the state equation
(12)
and the cost functional for each player

is given by the equation


(13)

for

, where the index set

is called the players' set.

In this formulation we consider a fixed interval of time


that is the prescribed duration of
the game,
is the initial state known by all players. Let
called
trajectory space of the game. The controls
are chosen by player for all
, here
is named an admissible strategy set for player . Then the problem can be stated as follows
For each
, player
the cost functional (profits)

wants to choose his control


such as to minimize (or maximize)
subject to the state equation (13).

It is assumed that all players know the state equation as well as the cost functionals.
Example 1. In a two-firm differential game with one state variable
time according to the differential equation

, the state evolves over

in which
are scalar control variables of firm 1 and 2, respectively. The state variable
represents the number of customers that firm 1 has at time and
is the constant size of the
total market. Hence
is the number of customers of firm 2. The control variables
are the firm`s respective advertising effort rates at time . The differential equation, in this case, can
be interpreted in the following way: the number of customers of firm 1 tends to increase by the
advertising efforts of firm 1 since these efforts attract customers from firm 2. On the other hand, the
advertising efforts of firm 2 tend to draw away customers from firm 1.
Payoffs are given by

400

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

in which
represent firm i's unit revenues. The second term in the integrand of is a convex
advertising cost function of firm . Feasibility requires that
and
are not negative. Each
firm wishes to choose its advertising strategy over
so as to maximize its payoff. The payoff is
simply the present value of a firm's profit on the horizon.
Remark. In this game, the rival firm's actions do not influence a firm's payoff directly but only
indirectly through the state dynamics.

3.1. The information structure


In many problems the control function
information structure, which is denoted by

where

, for each , should be specified by means of an


, and is defined as

is nondecreasing in .

Depending on the type of information available, we can define a strategy space


of all suitable mappings as follows

We also require that

belongs to

for

of player

Some types of standard information structures that arise in deterministic differential games are
stated in following definition.

Definition 1. In -player continuous time deterministic dynamic game of prescribed duration


we say that s information is

401

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

(i) open-loop if
(ii) closed-loop if
(iii)
memory less perfect state if
(iv) feedback if
, for

ISSN: 2010-1791

,
for

The following theorem provides a set of conditions under which the problem given by
equations (12) and (13), admits a unique solution for every -tuple
Theorem 3. Let the information structure for each pair be any one of the information patterns of
the definition above. Furthermore, let
then if
(i)

is continuous in

for each

,
(ii)
(iii)
for
uniformly Lipschitz in

is uniformly Lipschitz in
is continuous in for each

the differential equation (12) admits a single state trajectory for every
, and furthermore this unique trajectory is continuous.

,
and

, so that

3.2. Cooperative Games


In this section we fix the initial state

, and hence it will be omitted from the notation.

As mentioned above, the differential games can be classified in two classes: Cooperative
games and non-cooperative games. In a cooperative game the players wish to cooperate to reach a
result that will be beneficial to all.
3.2.1 Pareto Equilibrium
Definition 2. Let us consider a game with players. Let
) be the player cost function,
given the initial state
and that the players follow the multi-strategy
. Let
be the set of admissible strategies for the player
and
and
(14)
where

. An admissible strategy
is called Pareto-optimal if there is if it does not exist another

such that
(15)

This concept can be illustrated in the following figure

402

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

Figure 1.
We can to see in figure 1, that the pair ( , ) with the cost vector V=V( , ) is not a Pareto
equilibrium, since there exist other points in that are "below" V.
Let

be the set of Pareto equilibrium (which supposed to be not empty). The set of vectors

is called the Pareto front of the game.


The following theorem provides one method to study the existence of Pareto equilibrium.

Theorem 4. Let we consider

and for each

consider the scalar function


(16)

If for some vector

there exists a strategy

that minimizes

, i.e.
(17)

then

is a Pareto equilibrium.
3.3. Non-cooperative Games

In a non-cooperative game the players act independently and each one wishes to optimize his
own objective function, i.e. players are rivals and all players act in their own best interest, paying no
attention whatsoever to the fortunes of the other players.

403

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

An important example of non-cooperative games is the problem


3.3.1. Two person Zero-Sum Differential Games
Consider the state equation
(18)
We may assume all variables to be scalar for the time being. In this equation, we let
denote the controls applied by players
and , respectively. We assume that

where

and

are convex sets in

and

. Consider the cost functional

which player
wants to maximize and player
wants to minimize. Since the gain of player
represents a loss to player , such games are named zero-sum games (because the sum of their cost
functional is identically zero). Thus, we are looking for admissible control trajectories
and
such that
(20)
The solution
is known as a saddle point but some authors call it the minimax solution.
Here
and
stand for
and ,
, respectively.
3.3.2. Nash equilibrium
First, we consider the case
Definition 3. Let
to the strategy
as

(two players)

be a strategy of player 2. We define the set of optimal responses of player 1

(21)
Similarly, the set of the optimal responses of player 2 to a strategy

of the player 1 is defined

as
(22)
A multi-strategy

is said to be a Nash equilibrium if

404

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

Equivalently,

ISSN: 2010-1791

is a Nash equilibrium if
(23)

and
(24)
In our words, in Nash equilibrium, a player can't improve his situation if he alters his strategy
unilaterally.
Generalizing to any finite number
Definition 4. A multi-strategy
the following inequalities hold for all

of players we have the following


in

constitutes a Nash-equilibrium solution if

The interpretation of a Nash equilibrium solution is as follows: If one player tries to alter his
strategy unilaterally, he can't improve his performance by such a change. In this sort of situation
each player is just interested in his own performance, that is, the game is played non-cooperatively.
Definition 5. Let we consider a dynamical game with players and let it be
objective function for the player
since the initial condition of the game is
in the time
. Let it be
, a markovian multi-strategy, that is to say, each
markovian (or feedback). It's said that is a perfect Nash Equilibrium if, for each
any initial condition
, is hold that

the
it is
and

(25)
where the infimum is calculated above all the marcovians strategies

of the player .

In other words, a perfect Nash equilibrium is a marcovian multi-strategy that it is a Nash


equilibrium for anyone that it will be the initial condition of the game. In this case, some authors
say that the Nash equilibrium is perfect in the sub games (sub game perfect). Observe that to solve
(23) or (24) substantially we would to solve an OCP for each . This suggests that, in principle, we
can use technical like the principle of the maximum or dynamic programming to find Nash
equilibrium.
We will formulate the maximum principle for the case
. We will to consider a
differential game with
players, state space
, and actions set
, for
.
The dynamical model is
(26)
for all

405

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

The admissible controls are of open loop,


, where
measurable function of
to . The players wish "to maximize" the objective functions

Let

be the

is a

matrix of adjoints variables which -th file is

In this case is defined the Hamiltonian as follows

Let us suppose that


is a Nash equilibrium and that
is the corresponding path, then are
hold the following necessary conditions for each
the adjoints equations are

the terminal condition is

and the maximization of the Hamiltonian is

We can note that this reduce the problem to one border problem with two border conditions
that in some instances it can be solved explicitly. For example, Clemhout and Wan (1974) consider
games three-linear, called thus because the Hamiltonian is linear in the state, in the controls, and in
the variable attaches. Also, Dockner et al. (1985) identify several types of differential games that
they are soluble, in the sense of the fact that they can be determined balances of Nash of open loop,
either explicitly.
Example of perfect Nash equilibrium
Let us consider a differential game with
players and finite horizon . To save on
notation, denote the control variables of the two players by and instead of and . The state
space is
, the initial state is a fixed number
, and the set of feasible controls is
for player 1 and
for player 2. The objective functionals are

406

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

and

The system dynamics are given by

Let us first try to find open-loop Nash equilibrium of the above game, that is, a pair
where
and
are the strategies for player 1 and player 2,
respectively. If player 2 chooses to play
then player 1's problem can be written as
Maximize
Subject to
Since

(27)

is assumed by player 1 to be a fixed function, of the integral in (27) is equivalent

to the maximization of

, so that problem above is equivalent to

the problem
Maximize
Subject to
with

(28)

and this problem has the optimal open-loop strategy


, where

is the unique solution of

and

We also have that


(29)

407

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

where

ISSN: 2010-1791

. Finally, the state trajectory generated by this solution is given by

Note that the formula for

now depends on player 2's control path

still to be determined.

Now consider player 2's control problem. If player 1 chooses


can be written as

, player 2's problem

Maximize
Subject to

(30)

Denoting by the costate variable of player 2, the Hamiltonian function for this problem is
given by
. Maximization with respect to
yields
if
and
if
. If
then
, independently of . These properties imply that the maximized Hamiltonian function is
given by

The adjoin equation and transversality condition for player 2's problem are

Using

this can be written as


(31)

The boundary value problem consisting of (28) and (31) has a unique solution which is given
by
with
from (29). The function
is easily seen to be non-positive and
strictly increasing on
It depends on the parameters
and whether
for all
or whether
) can be smaller than
for some
. Because of the monotonicity
of
, however, we know that in the latter case there exists a number
such that
for
and
for all
Careful analysis of (31) reveals that such a
number exists if and only if

and

, in which case is given by

In all other cases let us formally set


. We summarize our results as follows. There exists
a candidate for an open-loop Nash equilibrium, given by

408

where

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

and

ISSN: 2010-1791

are specified as above.

To verify that the above candidate is an open-loop Nash equilibrium it suffices therefore to
prove that
is an optimal control path in player 2's problem. This, however, follows from
theorem 2 by noting that
for all , which shows that the maximized Hamiltonian
function
) of player 2's problem is a concave function with respect to . This concludes
the derivation of an open-loop Nash equilibrium for this example.
4. References

[1]
Amir, R. (2003). Stochastic games in economics and related fields: an
overview. In Neyman and Sorin (2003), Chapter 30.
[2]
Arkin, V.L, Evstigneev, LV. (1987). Stochastic Models of Control and
Economic Dynamics. Academic Press, London.
[3]
Balbus, L., Nowak, A.S. (2004). Construction of Nash equilibrium in
symmetric stochastic games of capital accumulation. Math. Meth. Opero Res. 60, pp.
267-277.
[4]
Basar, T. Editor (1986). Dynamic Games and Applications in Economics.
Lecture Notes in Economics and Mathematical Systems 265, Springer-Verlag, Berlin.
[5]
Basar, T, Olsder, G.J. (1999). Dynamic Non-cooperative Game Theory,
Second Edition. SIAM, Philadelphia. (The first edition was published by Academic
Press, in 1982.)
[6]
Bellman, R. (1956). Dynamic Programming. Princeton University Press,
Princeton, N.J.
[7]
Bernhard, P. (2005). Robust control approach to option pricing, including
transaction costs. In Nowak and Szajowski (2005), pp. 391-416.
[8]
Dockner, E., Feichtinger, G., Jorgensen, S. (1985). Tractable classes of
nonzero sum open-loop Nash differential games: theory and examples. J. Optim. Theory
Appl. 45, pp. 179-197.
[9]
Dockner, E.J., Jorgensen, S., Long, N.V., Sorger, G. (2000). Differential
Games in Economics and Management Science. Cambridge University Press,
Cambridge, U.K.
[10]
Feichtinger, G., Jorgensen, S. (1983). Differential game models in
management. Euro. J. Opero Res. 14, pp. 137-155.
[11]
Fershtman, c., Mullar, E. (1986). Tumpike properties of capital accumulation
games. J. Econ.' Th. 38, pp. 167-177.

409

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

[12]
Filar, J.A., Petrosjan, L.A. (2000). Dynamic cooperative games. Internatl.
Game Theory Rev. 2, pp. 47-65.
[13]
Filar, J., Vrieze, K. (1997). Competitive Markov Decision Processes.
Springer Verlag, New York.
[14]
Fleming, W.H., Rishel, R.W. (1975). Deterministic and Stochastic Optimal
Control. Springer-Verlag, NewYork.
[15]
Fleming, W.H., Soner, H.M. (1992). Controlled Markov Processes and
Viscosity Solutions. Springer-Verlag, New York.
[16]
Fudenberg, D., Tirole, J. (1991). Game Theory. MIT Press, Cambridge,
MA.Gaidov, S.D. (1986). Pareto-optimality in stochastic differential games. Problems of
Control and Information Theory 15, 439-450.
[17]
Gaidov, S.D. (1990). On the Nash-bargaining solution in stochastic
differential games. Serdica 16, pp. 120-125.
[18]
Isaacs, R. (1965). Differential Games. Wiley, New York.
[19]
Jorgensen, S., Sorger, G. (1990). Feedback Nash equilibrium in a problem of
optimal fishery management. J. Optim. Theory Appl. 64, pp. 293-310.
[20]
Jorgensen, S., Yeung, D.W.K. (1996). Stochastic differential game model of a
common property fishery. J. Optim. Theory Appl. 90 pp. 381-403.
[21]
Kaitala, V., Hamalainen, RP., Ruusunen, J. (1985). On the analysis of
equilibrium and bargaining in a fishery game. In Feichtinger (1985), pp. 593-606.
[22]
Kalai, E., Smorodinsky, M. (1975). Other solutions to Nash's bargaining
problem. Econometrica 43, pp. 513-518.
[23]
Kannan, D., Lakshmikantham, v., editors (2002). Handbook of Stochastic
Analysis and Applications. Dekker, New York.
[24]
Karatzas, L., Shreve, S.E. (1998). Methods of Mathematical Finance.
Springer-Verlag, New York.
[25]
Kirman, A.P., Sobel, M.J. (1974). Dynamic oligopoly with inventories.
Econometrica 42, pp. 279-287.
[26]
Kuhn, H.W. Szego, G.P., editors (1971). Differential Games and Related
Topics. North-Holland, Amsterdam.
[27]
Leitmann, G. (1974). Cooperative and Non-cooperative Many Players
Differential Games. Springer-Verlag, New York.
[28]
Le Van, E., Dana, R-A. (2003). Dynamic Programming in Economics.
Kluwer, Boston.
[29]
Nash, J. (1950a). Equilibrium points in N-person games. Proc. Natl. Acad.
Sci. 36, pp. 48-49.
[30]
Nash, J. (1950b). The bargaining problem. Econometrica 18, pp. 155-162.
[31]
Nash, J. (1951). Non-cooperative games. Ann. Math. 54, pp. 286-295.
[32]
Nash, J. (1953). Two-person cooperative games. Econometrica 21, pp. 128140.

410

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

[33]
Neck, R (1982). Dynamic systems with several decision makers. In
Operations Research in Progress, ed. by G. Feichtinger and P. Kall, Reidel, New York,
pp. 261-284.
[34]
Neyman, A, Sorin, S., editors (2003). Stochastic Games and Applications.
Kluwer, Dordrecht.
[35]
Nowak, AS., Szajowski, P. (2003). On Nash equilibrium in stochastic games
of capital accumulation. In Stochastic Games and Applications, Volume 9, edited by L.A
Petrosjan and Y.Y. Mazalov, Nova Science, pp. 118-129.
[36]
Nowak, AS., Szajowski, K., editors (2005). Advances in Dynamic Games.
(Annals of the International Society of Dynamic Games, vol. 7) Birkhauser, Boston.
[37]
Petrosyan, L.A (2003). Bargaining in dynamic games. In Petrosyan and
Yeung (2003), pp. 139-143.
[38]
Petrosyan, L.A. (2005). Cooperative differential games. In Nowak and
Szajowski (2005), pp. 183-200.
[39]
Petrosyan, L.A. Zenkevich, N.A (1996). Carne Theory. World Scientific,
Singapore.
[40]
Petrosyan, L.A. Yeung, D.W.K, Editor s (2003). ICM Millennium Lectures
on Games. Springer-Verlag, Berlin.
[41]
Pohjola, M. (1983). Nash and Stackelberg solutions in a differential game
model of capitalism J. Economic Dynamics and Control 6, pp. 173-186.
[42]
Ricci, G., Editor (1991). Decision Processes in Economics. Lecture Notes in
Economics and Mathematical Systems 353, Springer-Verlag, Berlin.
[43]
Roth, AE. (1979). Axiomatic Models of Bargaining. Springer-Verlag, Berlin.
[44]
Roth, AE. (1985). Game-Theoretic Models of Bargaining. Cambridge
University, Cambridge, U.K.
[45]
Sethi, S.P., Thompson, G.L. (2000). Optimal Control Theory: Applications to
Management Science and Economics, 2nd Edition. Kluwer, Boston.
[46]
Shapley, L. (1953). Stochastic games. Proc. Natl. Acad. Sci. 39, pp. 10951100.
[47]
Stokey, N.L., Lucas, R.E. (1989). Recursive Methods in Economic Dynamics.
Harvard University Press, Cambridge, MA.
[48]
Tabak, D., Kuo, B.C. (1971). Optimal Control by Mathematical Programming.
Prentice-Hall, Englewood Cliffs, N.J.
[49]
Tolwinski, B., Haurie, A., Leitmann, G. (1986). Cooperative equilibrium in
differential games. J. Math. Anal. Appl. 119, pp. 182-202.
[50]
Toussaint, B., (1985). The transversality condition at infinity applied to a
problem of optimal resource depletion. In Feichtinger (1985), pp. 429-440.
[51]
Vaisbord, E.M., Zhukovskii, V.I. (1988). Introduction to Multi-Player
Differential Games and Their Applications. Gordon and Breach, New York.
[52]
Vega-Amaya, O. (2003). Zero-sum average semi-Markov games: fixed-point
solutions of the Shapley equation. SIAM J. Control Optimal. 42, pp. 1876

411

International Journal of Physical and Mathematical Sciences Vol 4, No 1 (2013)

ISSN: 2010-1791

[53]
Von Newmann, J., Morgenstern, O. (1944). The Theory of Games and
Economic Behavior. Princeton University Press, Princeton, N.J.
[54]
Young, J., Zhou, X.Y. (1999). Stochastic Controls: Hamiltonian Systems and
HJB Equations. Springer-Verlag, New York.
[55]
Yu, P.L., Leitmann, G. (1974). Compromise solutions, domination structures
Salukvadze's solution. J. Optim. Theory Appl. 3, pp. 362-378.
[56]
Zhukavskiy. V.L, Salukvadze, M.E. (1994). The Vector-Valued Maximin.
Academic Press, Boston.