This action might not be possible to undo. Are you sure you want to continue?

# Copyright c 2009 by Karl Sigman

**1 IEOR 6711: Continuous-Time Markov Chains
**

A Markov chain in discrete time, {X

n

: n ≥ 0}, remains in any state for exactly one

unit of time before making a transition (change of state). We proceed now to relax this

restriction by allowing a chain to spend a continuous amount of time in any state, but

in such a way as to retain the Markov property. As motivation, suppose we consider the

rat in the open maze. Clearly it is more realistic to be able to keep track of where the

rat is at any continuous-time t ≥ 0 as oppposed to only where the rat is after n “steps”.

Assume throughout that our state space is S = Z = {· · · , −2, −1, 0, 1, 2, · · · } (or some

subset thereof). Suppose now that whenever a chain enters state i ∈ S, independent of

the past, the length of time spent in state i is a continuous, strictly positive (and proper)

random variable H

i

called the holding time in state i. When the holding time ends,

the process then makes a transition into state j according to transition probability P

ij

,

independent of the past, and so on.

1

Letting X(t) denote the state at time t, we end up

with a continuous-time stochastic process {X(t) : t ≥ 0} with state space S.

Our objective is to place conditions on the holding times to ensure that the continuous-

time process satisﬁes the Markov property: The future, {X(s + t) : t ≥ 0}, given the

present state, X(s), is independent of the past, {X(u) : 0 ≤ u < s}. Such a process will

be called a continuous-time Markvov chain (CTMC), and as we will conclude shortly,

the holding times will have to be exponentially distributed.

The formal deﬁnition is given by

Deﬁnition 1.1 A stochastic process {X(t) : t ≥ 0} with discrete state space S is called

a continuous-time Markvov chain (CTMC) if for all t ≥ 0, s ≥ 0, i ∈ S, j ∈ S,

P(X(s +t) = j|X(s) = i, {X(u) : 0 ≤ u < s}) = P(X(s +t) = j|X(s) = i) = P

ij

(t).

P

ij

(t) is the probability that the chain will be in state j, t time units from now, given

it is in state i now.

For each t ≥ 0 there is a transition matrix

P(t) = (P

ij

(t)),

and P(0) = I, the identity matrix.

As for discrete-time Markov chains, we are assuming here that the distribution of the

future, given the present state X(s), does not depend on the present time s, but only on

the present state X(s) = i, whatever it is, and the amount of time that has elapsed, t,

since time s. In particular, P

ij

(t) = P(X(t) = j|X(0) = i).

1

P

ii

> 0 is allowed, meaning that a transition back into state i from state i can ocurr. Each time this

happens though, a new H

i

, independent of the past, determines the new length of time spent in state i.

See Section 1.14 for details.

1

But unlike the discrete-time case, there is no smallest “next time” until the next

transition, there is a continuum of such possible times t. For each ﬁxed i and j, P

ij

(t), t ≥

0 deﬁnes a function which in principle can be studied by use of calculus and diﬀerential

equations. Although this makes the analysis of CTMC’s more diﬃcult/technical than for

discrete-time chains, we will, non-the-less, ﬁnd that many similarities with discrete-time

chains follow, and many useful results can be obtained.

A little thought reveals that the holding times must have the memoryless property

and thus are exponentially distributed. To see this, suppose that X(t) = i. Time t lies

somewhere in the middle of the holding time H

i

for state i. The future after time t tells

us, in particular, the remaining holding time in state i, whereas the past before time t,

tells us, in particular, the age of the holding time (how long the process has been in state

i). In order for the future to be independent of the past given X(t) = i, we deduce that

the remaining holding time must only depend (in distribution) on i and be independent of

its age; the memoryless property follows. Since an exponential distribution is completely

determined by its rate we conclude that for each i ∈ S, there exists a constant (rate)

a

i

> 0, such that the chain, when entering state i, remains there, independent of the

past, for an amount of time H

i

∼ exp(a

i

):

A CTMC makes transitions from state to state, independent of the past, ac-

cording to a discrete-time Markov chain, but once entering a state remains in

that state, independent of the past, for an exponentially distributed amount of

time before changing state again.

Thus a CTMC can simply be described by a transition matrix P = (P

ij

), describing

how the chain changes state step-by-step at transition epochs, together with a set of rates

{a

i

: i ∈ S}, the holding time rates. Each time state i is visited, the chain spends, on

average, E(H

i

) = 1/a

i

units of time there before moving on.

1.1 Interpretation of the {a

i

} as transition rates; the transition

rate matrix (inﬁnitesimal generator) Q

Assume here that P

i,i

= 0 for all i ∈ S. a

i

can be interpreted as the transition rate

out of state i given that X(t) = i; the intuitive idea being that the exponential holding

time will end, independent of the past, in the next dt units of time with probability a

i

dt.

This can be made rigorous recalling the little o(t) results for the Poisson process: Let

{N

i

(t)} denote a Poisson counting process at rate a

i

. Then P(N

i

(h) > 0) = a

i

h + o(h)

and P(N

i

(h) = 1) = a

i

h +o(h) (and P(N

i

(h) > 1) = o(h)). Thus

lim

h↓0

P(X(h) = i | X(0) = i)/h = lim

h↓0

P(N

i

(h) = 1)/h = lim

h↓0

(a

i

h +o(h))/h = a

i

. (1)

More generally, for j = i, we additionally use the Markov property asserting that once

leaving state i the chain will, independent of the past, next go to state j with probability

2

P

i,j

to obtain

P

i,j

(0) = lim

h↓0

P

i,j

(h)/h = lim

h↓0

P(N

i

(h) = 1)P

i,j

/h = a

i

P

i,j

. (2)

a

i

P

i,j

can thus be interpreted as the transition rate from state i to state j given that

the chain is currently in state i.

When i = j, P

i,i

(h) = 1 −P(X(h) = i | X(0) = i) and so

P

i,i

(0) = lim

h↓0

(P

i,i

(h) −1)/h = lim

h↓0

−P(N

i

(h) = 1)/h = −a

i

. (3)

Deﬁnition 1.2 The matrix Q = P

**(0) given explicitly by (2) and (3) is called the tran-
**

sition rate matrix, or inﬁnitesimal generator, of the Markov chain

For example, if S = {0, 1, 2, 3, 4}, then

Q =

¸

¸

¸

¸

¸

−a

0

a

0

P

0,1

a

0

P

0,2

a

0

P

0,3

a

0

P

0,4

a

1

P

1,0

−a

1

a

1

P

1,2

a

1

P

1,3

a

1

P

1,4

a

2

P

2,0

a

2

P

2,1

−a

2

a

2

P

2,3

a

2

P

2,4

a

3

P

3,0

a

3

P

3,1

a

3

P

3,2

−a

3

a

3

P

3,4

a

4

P

4,0

a

4

P

4,3

a

4

P

4,3

a

4

P

4,3

−a

4

**Note in passing that since we assume that P
**

i,i

= 0, i ∈ S, we conclude that each row

of Q sums to 0.

1.2 The embedded discrete-time Markov chain

Letting τ

n

denote the time at which the n

th

change of state (transition) occurs, we see that

X

n

= X(τ

n

+), the state right after the n

th

transition, deﬁnes the underlying discrete-time

Markov chain, called the embedded Markov chain. {X

n

} keeps track, consecutively, of

the states visited right after each transition, and moves from state to state according to

the one-step transition probabilities P

ij

= P(X

n+1

= j|X

n

= i). This transition matrix

(P

ij

), together with the holding-time rates {a

i

}, completely determines the CTMC.

1.3 Chapman-Kolmogorov equations

The Chapman-Kolmogorov equations for discrete-time Markov chains generalizes to

Lemma 1.1 (Chapman-Kolmogorov equations for CTMC’s) For all t ≥ 0, s ≥

0,

P(t +s) = P(s)P(t),

that is, for all t ≥ 0, s ≥ 0, i ∈ S, j ∈ S

P

ij

(t +s) =

¸

k∈S

P

ik

(s)P

kj

(t).

3

As for discrete-time chains, the (easy) proof involves ﬁrst conditioning on what state

k the chain is in at time s given that X(0) = i, yielding P

ik

(s), and then using the

Markov property to conclude that the probability that the chain, now in state k, would

then be in state j after an additional t time units is, independent of the past, P

kj

(t).

1.4 Examples of CTMC’s

1. Poisson counting process: Let {N(t) : t ≥ 0} be the counting process for a Poisson

process ψ = {t

n

} at rate λ. Then {N(t)} forms a CTMC with S = {0, 1, 2, . . .},

P

i,i+1

= 1, a

i

= λ, i ≥ 0: If N(t) = i then, by the memoryless property, the next

arrival, arrival i + 1, will, independent of the past, occur after an exponentially

distributed amount of time at rate λ. The holding time in state i is simply the

interarrival time, t

i+1

−t

i

, and τ

n

= t

n

since N(t) only changes state at an arrival

time. Assuming that N(0) = 0 we conclude that X

n

= N(t

n

+) = n, n ≥ 0; the

embedded chain is deterministic. This is a very special kind of CTMC for several

reasons. (1) all holding times H

i

have the same rate a

i

= λ, and (2) N(t) is a non-

decreasing process; it increases by one at each arrival time, and remains constant

otherwise. As t → ∞, N(t) ↑ ∞ step by step.

2. Consider the rat in the closed maze, in which at each transition, the rat is equally

likely to move to one of the neighboring two cells, but where now we assume that

the holding time, H

i

, in cell i is exponential at rate a

i

= i, i = 1, 2, 3, 4. Time is in

minutes (say). Let X(t) denote the cell that the rat is in at time t. Given the rat is

now in cell 2 (say), he will remain there, independent of the past, for an exponential

amount of time with mean 1/2, and then move, independent of the past, to either

cell 1 or 4 w.p.=1/2. The other transitions are similarly explained. {X(t)} forms

a CTMC. Note how cell 4 has the shortest holding time (mean 1/4 minutes), and

cell 1 has the longest (mean 1 minute). Of intrinisic interest is to calculate the

long-run proportion of time (continuous time now) that the rat spends in each cell;

P

i

def

= lim

t→∞

1

t

t

0

I{X(s) = i}ds, i = 1, 2, 3, 4.

We will learn how to compute these later; they serve as the continuous-time analog

to the discrete-time stationary probabilities π

i

for discrete-time Markov chains.

P = (P

1

, P

2

, P

3

, P

4

) is called the limiting (stationary) distribution for the CTMC.

3. FIFO M/M/1 queue: Arrivals to a single-server queue are Poisson at rate λ. There

is one line (queue) to wait in, and customers independently (and independent of the

Poisson arrival process) have service times {S

n

} that are exponentially distributed

at rate µ. We assume that customers join the tail of the queue, and hence begin

service in the order that they arrive ﬁrst-in-queue-ﬁrst-out-of-queue (FIFO). Let

X(t) denote the number of customers in the system at time t, where “system”

4

means the line plus the service area. So (for example), X(t) = 2 means that there

is one customer in service and one waiting in line. Note that a transition can only

occur at customer arrival or departure times, and that departures occur whenever

a service completion occurs. At an arrival time X(t) jumps up by the amount 1,

whereas at a departure time X(t) jumps down by the amount 1.

Determining the rates a

i

: If X(t) = 0 then only an arrival can occur next, so the

holding time is given by H

0

∼ exp(λ) the time until the next arrival; a

0

= λ, the

arrival rate. If X(t) = i ≥ 1, then the holding time is given by H

i

= min{S

r

, X}

where S

r

is the remaining service time of the customer in service, and X is the

time until the next arrival. The memoryless property for both service times and

interarrival times implies that S

r

∼ exp(µ) and X ∼ exp(λ) independent of the

past. Also, they are independent r.v.s. because the service times are assumed

independent of the Poisson arrival process. Thus H

i

∼ exp(λ + µ) and a

i

=

λ + µ, i ≥ 1. The point here is that each of the two independent events “service

completion will ocurr”, “new arrival will ocurr” is competing to be the next event

so as to end the holding time.

The transition probabilities P

ij

for the embedded discrete-time chain are derived

as follows: X

n

denotes the number of customers in the system right after the n

th

transition. Transitions are caused only by arrivals and departures.

If X

n

= 0, then the system is empty and we are waiting for the next arrival;

P(X

n+1

= 1|X

n

= 0) = 1. But if X

n

= i ≥ 1, then X

n+1

= i + 1 w.p. P(X <

S

r

) = λ/(λ + µ), and X

n+1

= i − 1 w.p. P(S

r

< X) = µ/(λ + µ), depending on

whether an arrival or a departure is the ﬁrst event to occur next. So, P

0,1

= 1, and

for i ≥ 1, P

i,i+1

= p = λ/(λ+µ), and P

i,i−1

= 1 −p = µ/(λ+µ). We conclude that

The embedded Markov chain for a FIFO M/M/1 queue is a simple ran-

dom walk (“up” probability p = λ/(λ + µ), “down” probability 1 − p =

µ/(λ +µ)) that is restricted to be non-negative (P

0,1

= 1).

4. M/M/c multi-server queue: This is the same as the FIFO M/M/1 queue except

there are now c servers working in parallel. As in a USA postoﬃce, arrivals wait

in one FIFO line (queue) and enter service at the ﬁrst available free server. X(t)

denotes the number of customers in the system at time t. For illustration, let’s

assume c = 2. Then, for example, X(t) = 4 means that two customers are in

service (each with their own server) and two others are waiting in line. When

X(t) = i ∈ {0, 1}, the holding times are the same as for the M/M/1 model; a

0

=

λ, a

1

= λ + µ. But when X(t) = i ≥ 2, both remaining service times, denoted by

S

r

1

and S

r

2

, compete to determine the next departure. Since they are independent

exponentials at rate µ, we deduce that the time until the next departure is given by

min{S

r

1

, S

r

2

} ∼ exp(2µ). The time until the next arrival is given by X ∼ exp(λ)

and is independent of both remaining service times. We conclude that the holding

time in any state i ≥ 2 is given by H

i

= min{X, S

r

1

, S

r

2

} ∼ exp(λ + 2µ).

5

For the general case of c ≥ 2, the rates are determined analogously: a

i

= λ+iµ, 0 ≤

i ≤ c, a

i

= λ + cµ, i > c.

For the embedded chain: P

0,1

= 1 and for 0 ≤ i ≤ c−1, P

i,i+1

= λ/(λ +iµ), P

i,i−1

=

iµ/(λ +iµ). Then for i ≥ c, P

i,i+1

= λ/(λ +cµ), P

i,i−1

= cµ/(λ +cµ). This is an

example of a simple random walk with state-dependent “up”, “down” probabilities:

at each step, the probabilities for the next increment depend on i, the current state,

until i = c at which point the probabilities remain constant.

5. M/M/∞ inﬁnite-server queue: Here we have a M/M/c queue with c = ∞; a special

case of the M/G/∞ queue. Letting X(t) denote the number of customers in the

system at time t, we see that a

i

= λ + iµ, i ≥ 0 since there is no limit on the

number of busy servers.

For the embedded chain: P

0,1

= 1 and P

i,i+1

= λ/(λ +iµ), P

i,i−1

= iµ/(λ + iµ), i ≥

1. This simple random walk thus has state-dependent “up”, “down” probabilities

that continue to depend on each state i, the current state. Note how, as i in-

creases, the down probability, P

i,i−1

, increases, and approaches 1 as i → ∞: when

the system is heavily congested, departures occur rapidly; this model is always

stable.

1.5 Birth and Death processes

Except for Example 2 (rat in the closed maze) all of the CTMC examples in the previous

section were Birth and Death (B&D) processes, CTMC’s that can only change state by

increasing by one, or decreasing by one; P

i,i+1

+ P

i,i−1

= 1, i ∈ S. (In Example 2,

P

1,3

> 0, for example, so it is not B&D.) Here we study B&D processes more formally,

since they tend to be a very useful type of CTMC. Whenever the state increases by one,

we say there is a birth, and whenever it decreases by one we say there is a death. We

shall focus on the case when S = {0, 1, 2, . . .}, in which case X(t) can be thought of as

the population size at time t.

For each state i ≥ 0 we have a birth rate λ

i

and a death rate µ

i

: Whenever X(t) = i,

independent of the past, the time until the next birth is a r.v. X ∼ exp(λ

i

) and,

independently, the time until the next death is a r.v. Y ∼ exp(µ

i

). Thus the holding

time rates are given by a

i

= λ

i

+ µ

i

because the time until the next transition (change

of state) in given by the holding time H

i

= min{X, Y } ∼ exp(λ

i

+ µ

i

). The idea here is

that at any given time the next birth is competing with the next death to be the next

transition. (We always assume here that µ

0

= 0 since there can be no deaths without a

population.)

This means that whenever X(t) = i ≥ 1, the next transition will be a birth w.p.

P

i,i+1

= P(X < Y ) = λ

i

/(λ

i

+ µ

i

), and a death w.p. P

i,i−1

= P(Y < X) = µ

i

/(λ

i

+ µ

i

).

Thus the embedded chain for a B&D process is a simple random walk with state dependent

“up”, “down” probabilities.

6

When µ

i

= 0, i ≥ 0, and λ

i

> 0, i ≥ 0, we call the process a pure birth process;

the population continues to increase by one at each transition. The main example is the

Poisson counting process (Example 1 in the previous Section), but this can be generalized

by allowing each λ

i

to be diﬀerent. The reader is encouraged at this point to go back

over the B&D Examples in the previous Section.

1.6 Explosive CTMC’s

Consider a pure birth process {X(t)}, P

i,i+1

= 1, i ≥ 0, in which a

i

= λ

i

= 2

i

, i ≥ 0.

This process spends, on average, E(H

i

) = 1/λ

i

= 2

−i

units of time in state i and then

changes to state i + 1; . Thus it spends less and less time in each state, consequently

jumping to the next state faster and faster as time goes on. Since X(t) → ∞ as t → ∞,

we now explore how fast this happens. Note that the chain will visit state i at time

H

0

+ H

1

+ · · · + H

i−1

, the sum of the ﬁrst i holding times. Thus the chain will visit all

of the states by time

T =

∞

¸

i=0

H

i

.

Taking expected value yields

E(T) =

∞

¸

i=0

2

−i

= 2 < ∞,

and we conclude that on average all states i ≥ 0 have been visited by time t = 2, a ﬁnite

amount of time! But this implies that w.p.1., all states will be visited in a ﬁnite amount

of time; P(T < ∞) = 1. Consequently, w.p.1., X(T +t) = ∞, t ≥ 0. This is an example

of an explosive Markov chain: The number of transitions in a ﬁnite interval of time is

inﬁnite.

We shall rule out this kind of behavior in the rest of our study, and assume from now

on that all CTMC’s considered are non-explosive, by which we mean that the number of

transitions in any ﬁnite interval of time is ﬁnite. This will always hold for any CTMC

with a ﬁnite state space, or any CTMC for which there are only a ﬁnite number of

distinct values for the rates a

i

, and more generally whenever sup{a

i

: i ∈ S} < ∞. Every

Example given in the previous Section was non-explosive. Only the M/M/∞queue needs

some clariﬁcation since a

i

= λ + iµ → ∞ as i → ∞. But only arrivals and departures

determine transitions, and the arrivals come from the Poisson process at ﬁxed rate λ,

so the arrivals can not cause an explosion; N(t) < ∞, t ≥ 0. Now observe that during

any interval of time, (s, t], the number of departures can be no larger than N(t), the

total number of arrivals thus far, so they too can not cause an explosion. In short, the

number of transitions in any interval (s, t] is bounded from above by 2N(t) < ∞; the

non-explosive condition is satisﬁed. This method of bounding the number of transitions

by the underlying Poisson arrival process will hold for essentially any CTMC queueing

model.

7

1.7 Communication classes, irreducibility and recurrence

State j is said to be reachable from state i for a CTMC if P(X(s) = j|X(0) = i) =

P

ij

(s) > 0 for some s ≥ 0. As with discrete-time chains, i and j are said to communicate

if state j is reachable from state i, and state i is reachable from state j.

It is immediate that i and j communicate in continuous time if and only if they do

so for the embedded discrete-time chain {X

n

}: They communicate in continuous-time if

and only if they do so at transition epochs. Thus once again, we can partition the state

space up into disjoint communication classes, S = C

1

∪C

2

∪· · · , and an irreducible chain

is a chain for which all states communicate (S = C

1

, one communication class). We state

in passing

A CTMC is irreducible if and only if its embedded chain is irreducible.

Notions of recurrence, transience and positive recurence are similar as for discrete-

time chains: Let T

i,i

denote the amount of (continuous) time until the chain re-visits state

i (at a later transition) given that X(0) = i (deﬁned to be ∞ if it never does return);

the return time to state i. The chain will make its ﬁrst transition at time H

i

(holding

time in state i), so T

ii

≥ H

i

. State i is called recurrent if, w.p.1., the chain will re-visit

state i with certainty, that is, if P(T

ii

< ∞) = 1. The state is called transient otherwise.

This (with a little thought) is seen to be the same property as for the embedded chain

(because X(t) returns to state i for some t if and only if X

n

does so for some n):

A state i is recurrent/transient for a CTMC if and only if it is recurrent/transient

for the embedded discrete-time chain.

Thus communication classes all have the same type of states: all together they are

transient or all together they are recurrent.

1.8 Positive recurrence and the existence a limiting distribution

P = {P

j

}

State i is called positive recurrent if, in addition to being recurrent, E(T

ii

) < ∞; the

expected amount of time to return is ﬁnite. State i is called null recurrent if, in addition

to being recurrent, E(T

ii

) = ∞; the expected amount of time to return is inﬁnite. Unlike

recurrence, positive (or null) recurrence is not equivalent to that for the embedded chain:

It is possible for a CTMC to be positive recurrent while its embedded chain is null

recurrent (and vice versa). But positive and null recurrence are still class properties, so

in particular:

For an irreducible CTMC, all states together are transient, positive recurrent,

or null recurrent.

8

A CTMC is called positive recurrent if it is irreducible and all states are positive

recurrent. We deﬁne (when they exist, independent of initial condition X(0) = i) the

limiting probabilities {P

j

} for the CTMC as the long-run proportion of time the chain

spends in each state j ∈ S:

P

j

= lim

t→∞

1

t

t

0

I{X(s) = j|X(0) = i}ds, w.p.1., (4)

which after taking expected values yields

P

j

= lim

t→∞

1

t

t

0

P

ij

(s)ds. (5)

When each P

j

exists and

¸

j

P

j

= 1, then

P = {P

j

} (as a row vector) is called the

limiting (or stationary) distribution for the Markov chain. Letting

P

∗

=

¸

¸

P

P

.

.

.

(6)

denote the matrix in which each row is the limiting probability distribution

P, (5) can

be expressed nicely in matrix form as

lim

t→∞

1

t

t

0

P(s)ds = P

∗

. (7)

As for discrete-time Markov chains, positive recurrence implies the existence of lim-

iting probabilities by use of the SLLN. The basic idea is that for ﬁxed state j, we can

break up the evolution of the CTMC into i.i.d. cycles, where a cycle begins every time

the chain makes a transition into state j. This yields an example of what is called a

regenerative process because we say it regenerates every time a cycle begins. The cycle

lengths are i.i.d. distributed as T

jj

, and during a cycle, the chain spends an amount of

time in state j equal in distribution to the holding time H

j

. This leads to

Proposition 1.1 If {X(t)} is a positive recurrent CTMC, then the limiting probability

distribution

P = (P

i,j

) as deﬁned by Equation (4) exists, is unique, and is given by

P

j

=

E(H

j

)

E(T

jj

)

=

1/a

j

E(T

jj

)

> 0, j ∈ S.

In words: “The long-run proportion of time the chain spends in state j equals the expected

amount of time spent in state j during a cycle divided by the expected cycle length (between

visits to state j)”.

Moreover, the stronger mode of convergence (weak convergence) holds:

P

j

= lim

t→∞

P

ij

(t), i, j ∈ S. (8)

Finally, if the chain is either null recurrent or transient, then P

j

= 0, j ∈ S; no

limiting distribution exists.

9

Proof : Fixing state j, we can break up the evolution of the CTMC into i.i.d. cycles,

where a cycle begins every time the chain makes a transition into state j. This follows by

the (strong) Markov property, since every time the chain enters state j, the chain starts

over again from scratch stochastically, and is independent of the past. Letting τ

n

(j)

denote the n

th

time at which the chain makes a transition into state j, with τ

0

(j) = 0,

the cycle lengths, T

n

(j) = τ

n

(j) −τ

n−1

(j), n ≥ 1, are i.i.d., distributed as the return time

T

jj

. {τ

n

(j) : n ≥ 1} forms a renewal point process because of the assumed recurrence,

of the chain, and we let N

j

(t) denote the number of such points during (0, t]. From the

Elementary Renewal Theorem, wp1,

lim

t→∞

N

j

(t)

t

=

1

E(T

jj

)

. (9)

Letting

J

n

=

τn(j)

τ

n−1

(j)

I{X(s) = j}ds,

(the amount of time spent in state j during the n

th

cycle) we conclude that {J

n

} forms

an i.i.d. sequence of r.v.s. distributed as the holding time H

j

; E(J) = E(H

j

). Thus

t

0

I{X(s) = j}ds ≈

N

j

(t)

¸

n=1

J

n

,

from which we obtain

1

t

t

0

I{X(s) = j}ds ≈

N

j

(t)

t

×

1

N

j

(t)

N

j

(t)

¸

n=1

J

n

.

Letting t → ∞ yields

P

j

=

E(H

j

)

E(T

jj

)

,

where the denominator is from (9) and the numerator is from the SLLN applied to

{J

n

}. P

j

> 0 if E(T

jj

) < ∞ (positive recurrence), whereas P

j

= 0 if E(T

jj

) = ∞ (null

recurrence). And if transient, then I{X(s) = j|X(0) = i} → 0 as s → ∞, wp1, yielding

P

j

= 0 as well from (4).

Uniqueness of the P

j

follows by the unique representation, P

j

=

1/a

j

E(T

jj

)

.

The weak convergence in (8) holds in addition to the already established time-average

convergence because the cycle-length distribution (the distribution of T

jj

for any ﬁxed

j) is non-lattice.

2

T

jj

has a non-lattice distribution because it is of phase type hence

a continuous distribution. In general, a positive recurrent regenerative process with a

non-lattice cycle-length distribution converges weakly. The details of this will be dealt

with later when we return to a more rigorous study of renewal theory.

2

The distribution of a non-negative rv X is said to be non-lattice if there does not exists a d > 0 such

that P(X ∈ {nd : n ≥ 0}) = 1. Any continuous distribution, in particular, is non-lattice.

10

1.9 Allowing an arbitrary random initial value for X(0)

If ν is a probability distribution on S, and if X(0) ∼ ν, then the distribution of X(t) is

given by the vector-matrix product νP(t); X(t) ∼ νP(t), t ≥ 0. Recalling the deﬁnition

of P

∗

in (6), note that νP

∗

=

P for any probability distribution ν. Thus for a positive

recurrent chain it holds more generally from (7) (by multiplying each left side by ν) that

lim

t→∞

1

t

t

0

νP(s)ds =

P. (10)

This merely means that the initial condition (e.g., the value of X(0)) can be random

as opposed to only deterministic (e.g., X(0) = i) without eﬀecting the limit; the same

limiting distribution

P is obtained regardless.

1.10 The limiting distribution yields a stationary distribution

and hence a stationary version of the Markov chain

As in discrete-time, the limiting distribution

P is also called a stationary distribution

because it yields a stationary version of the chain if the chain is initially distributed as

P at time t = 0:

Proposition 1.2 For a positive recurrent Markov chain with limiting distribution

P: If

X(0) ∼

P, then X(t) ∼

P for all t ≥ 0; that is,

PP(t) =

P, t ≥ 0. This means that

¸

i∈S

P

i

P

i,j

(t) = P

j

, j ∈ S, t ≥ 0.

In fact this limiting distribution

P is the only distribution (it is unique) that is stationary,

that is, for which

PP(t) =

P, t ≥ 0. Moreover, letting {X

∗

(t) : t ≥ 0} denote the chain

when X(0) ∼

P, it forms a stationary stochastic process: {X

∗

(s + t) : t ≥ 0} has the

same distribution for all s ≥ 0.

Proof : From the deﬁnition of P

∗

(each row is

P) we must equivalently show that

P

∗

P(t) = P

∗

for any t ≥ 0. (Intuitively we are simply asserting that P(∞)P(t) = P(∞)

because ∞+t = ∞.)

11

Recalling the Chapman-Kolmogorov equations, P(s + t) = P(s)P(t), and using (7),

we get

P

∗

P(t) =

lim

u→∞

1

u

u

0

P(s)ds

P(t)

= lim

u→∞

1

u

u

0

P(s)P(t)ds

= lim

u→∞

1

u

u

0

P(s + t)ds

= lim

u→∞

1

u

u

0

P(s)ds

= P

∗

.

The second to last equality follows due to fact that adding the ﬁxed t is asymptotically

negligible:

u

0

P(s +t)du =

u

0

P(s)ds +

u+t

u

P(s)ds −

t

0

P(s). All elements of P(s) are

bounded by 1, and so the last two integrals when divided by u tend to 0 as u → ∞.

If a probability distribution ν satisﬁes νP(t) = ν, t ≥ 0, then on the one hand, since

the chain is assumed positive recurrent, we have Equation (10) and hence

lim

t→∞

1

t

t

0

νP(s)ds = νP

∗

=

P. (11)

But on the other hand νP(t) = ν implies that

1

t

t

0

νP(s)ds =

1

t

t

0

νds = ν,

and we conclude that ν =

P; the stationary distribution is unique.

By the Markov property, a Markov process is completely determined (in distribution)

by its initial state. Thus {X

∗

(s + t) : t ≥ 0} has the same distribution for all s ≥ 0

because for all s, its initial state has the same distribution, X

∗

(s) ∼

P.

1.11 Computing P

ij

(t): Kolmogorov backward equations

We have yet to show how to compute the transition probabilities for a CTMC, P

ij

(t) =

P(X(t) = j|X(0) = i), t ≥ 0. For discrete-time Markov chains this was not a problem

since P

(n)

ij

= P(X

n

= j|X

0

= i), n ≥ 1 could be computed by using the fact that the

matrix (P

(n)

ij

) was simply the transition matrix P multiplied together n times, P

n

. In

continuous time however, the problem is a bit more complex; it involves setting up linear

diﬀerential equations for P

ij

(t) known as the Kolmogorov backward equations and then

solving. We present this derivation now.

12

Proposition 1.3 (Kolmogorov Backward Equations) For a (non-explosive) CTMC

with transition rate matrix Q = P

**(0) as in Deﬁnition 1.2, the following set of linear dif-
**

ferential equations is satisﬁed by {P(t)}:

P

(t) = QP(t), t ≥ 0, P(0) = I, (12)

that is,

P

ij

(t) = −a

i

P

ij

(t) +

¸

k=i

a

i

P

ik

P

kj

(t), i, j ∈ S, t ≥ 0. (13)

The unique solution is thus of the exponential form;

P(t) = e

Qt

, t ≥ 0, (14)

where for any square matrix M,

e

M

def

=

∞

¸

n=0

M

n

n!

.

Proof : The Chapman-Kolmogorov equations, P(t +h) = P(h)P(t), yield

P(t +h) −P(t) = (P(h) −I)P(t) (15)

= (P(h) −P(0))P(t). (16)

dividing by h and letting h → 0 then yields P

(t) = P

(0)P(t) = QP(t). (Technically,

this involves justifying the interchange of a limit and an inﬁnite sum, which indeed can

be justiﬁed here even when the state space is inﬁnite.)

The word backward refers to the fact that in our use of the Chapman-Kolmogorov

equations, we chose to place the h on the right-hand side in back, P(t + h) = P(h)P(t)

as opposed to in front, P(t + h) = P(t)P(h). The derivation above can be rigorously

justiﬁed for any non-explosive CTMC.

It turns out, however, that the derivation of the analogous forward equations P

(t) =

P(t)Q, t ≥ 0, P(0) = I, that one would expect to get by using P(t + h) = P(t)P(h)

can not be rigorously justiﬁed for all non-explosive CTMCs; there are examples (inﬁnite

state space) that cause trouble; the interchange of a limit and an inﬁnite sum can not be

justiﬁed.

But it does not matter, since the unique solution P(t) = e

Qt

to the backward equations

is the unique solution to the forward equations, and thus both equations are valid.

For a (non-explosive) CTMC, the transition probabilities P

i,j

(t) are the unique

solution to both the Kolmogorov backward and forward equations.

13

Remark 1.1 It is rare that we can explicitly compute the inﬁnite sum in the solution

P(t) = e

Qt

=

∞

¸

n=0

(Qt)

n

n!

.

But there are various numerical recipes for estimating e

Qt

to any desired level of accuracy.

For example, since

e

M

= lim

n→∞

(1 + M/n)

n

, for any square matrix M, one can choose n large and use

e

Qt

≈ (1 + (Qt)/n)

n

.

1.12 Balance equations and positive recurrence

Consider any deterministic function x(t), t ≥ 0 with values in S. Clearly, every time

x(t) enters a state j, it must ﬁrst leave that state in order to enter it again. Thus the

number of times during the interval (0, t] that it enters state j diﬀers by at most one,

from the number of times during the interval (0, t] that it leaves state j. We conclude

(by dividing by t and letting t → ∞) that the long-run rate at which the function leaves

state j equals the long-run rate at which the function enters state j. In words, “the rate

out of state j is equal to the rate into state j, for each state j”. We can apply this kind

of result to each sample-path of a stochastic process. For a positive recurrent CTMC

with limiting distribution

P = {P

j

}, the rate out of state j is given by a

j

P

j

, while the

rate into state j is given by

¸

i=j

P

i

a

i

P

ij

, j ∈ S, by interpreting the limiting probability

P

j

as a proportion of time and recalling Section 1.1 on transition rates.

Deﬁnition 1.3 The balance equations for a positive recurrent CTMC are given by

a

j

P

j

=

¸

i=j

P

i

a

i

P

ij

, j ∈ S, (17)

which in matrix form are given by

PQ =

0, (18)

where Q = P

**(0) is the transition rate matrix from Section 1.1.
**

Theorem 1.1 An irreducible (and non-explosive) CTMC is positive recurrent if and only

if there is a (necessarily unique) probability solution

P to the balance equations

PQ =

0.

The solution satisﬁes P

j

> 0, j ∈ S and is the limiting (stationary) distribution as

deﬁned in Equations (4)-(7).

Proof : Suppose that the chain is positive recurrent. Then from Proposition 1.1 and

Proposition 1.2, there is a unique limiting probability distribution

P and it is a stationary

distribution;

PP(t) =

P, t ≥ 0. Taking the derivative at t = 0 on both sides of

PP(t) =

P yields

PQ =

0,

14

the balance equations.

Conversely, suppose that

P is a probability solution to the balance equations. We

will ﬁrst show that any such solution must also satisfy

PP(t) =

P, t ≥ 0, that is, it

is a stationary distribution. We then will show that if an irreducible chain has such a

stationary distribution, then the chain must be positive recurrent. To this end: Suppose

that

PQ =

**0. Multiplying both right sides by P(t) yields
**

PQP(t) =

**0, which due to the
**

Kolmogorov Backward equations, P

(t) = QP(t), is equivalent to

PP

(t) =

0 which is

equivalent to

d(

PP(t))

dt

=

0.

But this implies that

PP(t) is a constant in t and hence that

PP(t) =

PP(0) =

PI =

P;

P is indeed a stationary distribution. Now suppose that the chain is not positive

recurrent. For an irreducible CTMC, all states together are transient, positive recurrent,

or null recurrent, so the chain must be either null recurrent or transient and hence by

Proposition 1.1, we have

lim

t→∞

1

t

t

0

P(s)ds = 0. (19)

Multiplying both sides on the left by

P yields

lim

t→∞

1

t

t

0

PP(s)ds = 0. (20)

But using the already established

PP(t) =

P we have

1

t

t

0

PP(s)ds =

1

t

t

0

Pds =

P

and we end with a contradiction

P = 0 (

**P is a probability distribution by assumption).
**

Finally, from Proposition 1.2 we know that there can only be one stationary distribution

for a positive recurrent chain, the limiting distribution as deﬁned in Equations (4)-(7),

so we conclude that

P here is indeed the limiting distribution.

As for discrete-time Markov chains, when the state space is ﬁnite, we obtain a useful

and simple special case:

Theorem 1.2 An irreducible CTMC with a ﬁnite state space is positive recurrent; there

is always a unique probability solution to the balance equations.

Proof : Suppose (without loss of generality) the state space is S = {1, 2, . . . , b} for some

integer b ≥ 1. We already know that the chain must be recurrent because the embedded

chain is so. We also know that the embedded chain is positive recurrent because for ﬁnite

state discrete-time chains irreducibility implies positive recurrence. Let τ

1,1

denote the

discrete return time to state 1, and let T

1,1

denote the corresponding continuous return

time. We know that E(τ

1,1

) < ∞. Also, T

1,1

is a random sum of τ

1,1

holding times,

starting with H

1

. Let a

∗

= min{a

1

, . . . , a

b

}. Then a

∗

> 0 and every holding time H

i

15

satisﬁes E(H

i

) ≤ 1/a

∗

< ∞, i ∈ {1, 2, . . . , b}. Letting {Y

n

} denote iid exponential rvs

at rate a

∗

, independent of τ

1,1

, we conclude that

E(T

1,1

) ≤ E

τ

1,1

¸

n=1

Y

n

= E(τ

1,1

)(1/a

∗

) < ∞.

1.13 Examples of setting up and solving balance equations

Here we apply Theorems 1.1 and 1.2 to a variety of models. In most cases, solving

the resulting balance equations involves recursively expressing all the P

j

in terms of one

particular one, P

0

(say), then solving for P

0

by using the fact that

¸

j∈S

P

j

= 1. In the

case when the state space is inﬁnite, the sum is an inﬁnite sum that might diverge unless

further restrictions on the system parameters (rates) are enforced.

1. FIFO M/M/1 queue: X(t) denotes the number of customers in the system at time

t. Here, irreducibility is immediate since as pointed out earlier, the embedded chain

is a simple random walk (hence irreducible), so, from Theorem 1.1, we will have

positive recurrence if and only if we can solve the balance equations (17):

λP

0

= µP

1

(λ +µ)P

1

= λP

0

+µP

2

(λ +µ)P

2

= λP

1

+µP

3

.

.

.

(λ + µ)P

j

= λP

j−1

+µP

j+1

, j ≥ 1.

These equations can also be derived from scratch as follows: Given X(t) = 0, the

rate out of state 0 is the arrival rate a

0

= λ, and the only way to enter state 0 is

from state i = 1, from which a departure must occur (rate µ). This yields the ﬁrst

equation. Given X(t) = j ≥ 1, the rate out of state j is a

j

= λ + µ (either an

arrival or a departure can occur), but there are two ways to enter such a state j:

either from state i = j − 1 (an arrival occurs (rate λ) when X(t) = j − 1 causing

the transition j −1 → j), or from state i = j +1 (a departure ocurrs (rate µ) when

X(t) = j causing the transition j + 1 → j). This yields the other equations.

Note that since λP

0

= µP

1

(ﬁrst equation), the second equation reduces to λP

1

=

µP

2

which in turn causes the third equation to reduce to λP

2

= µP

3

, and in general

the balance equations reduce to

λP

j

= µP

j+1

, j ≥ 0, (21)

which asserts that

16

for each j, the rate from j to j + 1 equals the rate from j + 1 to j,

or

P

j+1

= ρP

j

, j ≥ 0,

from which we recursivly obtain P

1

= ρP

0

, P

2

= ρP

1

= ρ

2

P

0

and in general

P

j

= ρ

j

P

0

. Using the fact that the probabilities must sum to one yields

1 = P

0

∞

¸

j=0

ρ

j

,

from which we conclude that there is a solution if and only if the geometric series

converges, that is, if and only if ρ < 1, equivalently λ < µ, “the arrival rate is less

than the service rate”, in which case 1 = P

0

(1 −ρ)

−1

, or P

0

= 1 −ρ.

Thus P

j

= ρ

j

(1 −ρ), j ≥ 0 and we obtain a geometric stationary distribution.

Summarizing:

The FIFO M/M/1 queue is positive recurrent if and only if ρ < 1 in

which case its stationary distribution is geometric with paramater ρ; P

j

=

ρ

j

(1−ρ), j ≥ 0. (If ρ = 1 it can be shown that the chain is null recurrent,

and transient if ρ > 1.)

When ρ < 1 we say that the M/M/1 queue is stable, unstable otherwise. Stability

intuitively means that the queue length doesn’t grow without bound over time.

When the queue is stable, we can take the mean of the stationary distribution to

obtain the average number of customers in the system

l = lim

t→∞

1

t

t

0

X(s)ds (22)

=

∞

¸

j=0

jP

j

(23)

=

∞

¸

j=0

j(1 −ρ)ρ

j

(24)

=

ρ

1 −ρ

. (25)

2. Birth and Death processes: The fact that the balance equations for the FIFO

M/M/1 queue reduced to “for each state j, the rate from j to j + 1 equals the

rate from j + 1 to j” is not a coincidence, and in fact this reduction holds for any

Birth and Death process. For in a Birth and Death process, the balance equations

are:

17

λ

0

P

0

= µ

1

P

1

(λ

1

+ µ

1

)P

1

= λ

0

P

0

+µ

2

P

2

(λ

2

+ µ

2

)P

2

= λ

1

P

1

+µ

3

P

3

.

.

.

(λ

j

+µ

j

)P

j

= λ

j−1

P

j−1

+µ

j+1

P

j+1

, j ≥ 1.

Plugging the ﬁrst equation into the second yields λ

1

P

1

= µ

2

P

2

which in turn can

be plugged into the third yielding λ

2

P

2

= µ

3

P

3

and so on. We conclude that for

any Birth and Death process, the balance equations reduce to

λ

j

P

j

= µ

j+1

P

j+1

, j ≥ 0, the Birth and Death balance equations. (26)

Solving recursively, we see that

P

j

= P

0

λ

0

×· · · ×λ

j−1

µ

1

×· · · ×µ

j

= P

0

j

¸

i=1

λ

i−1

µ

i

, j ≥ 1.

Using the fact that the probabilities must sum to one then yields:

An irreducible Birth and Death process is positive recurrent if and only if

∞

¸

j=1

j

¸

i=1

λ

i−1

µ

i

< ∞,

in which case

P

0

=

1

1 +

¸

∞

j=1

¸

j

i=1

λ

i−1

µ

i

,

and

P

j

=

¸

j

i=1

λ

i−1

µ

i

1 +

¸

∞

j=1

¸

j

i=1

λ

i−1

µ

i

, j ≥ 1. (27)

For example, in the M/M/1 model,

1 +

∞

¸

j=1

j

¸

i=1

λ

i−1

µ

i

=

∞

¸

j=0

ρ

j

,

which agrees with our previous analysis.

We note in passing that the statement “for each state j, the rate from j to j + 1

equals the rate from j + 1 to j” holds for any deterministic function x(t), t ≥ 0,

18

in which changes of state are only of magnitude 1; up by 1 or down by 1. Arguing

along the same lines as when we introduced the balance equations, every time this

kind of function goes up from j to j +1, the only way it can do so again is by ﬁrst

going back down from j +1 to j. Thus the number of times during the interval (0, t]

that it makes an “up” transition from j to j + 1 diﬀers by at most one, from the

number of times during the interval (0, t] that it makes a “down” transition from

j + 1 to j. We conclude (by dividing by t and letting t → ∞) that the long-run

rate at which the function goes from j to j + 1 equals the long-run rate at which

the function goes from j + 1 to j. Of course, as for the balance equations, being

able to write this statement simply as λ

j

P

j

= µ

j+1

P

j+1

crucially depends on the

Markov property

3. M/M/1 loss system: This is the M/M/1 queueing model, except there is no waiting

room; any customer arriving when the server is busy is “lost”, that is, departs

without being served. In this case S = {0, 1} and X(t) = 1 if the server is busy

and X(t) = 0 if the server is free. P

01

= 1 = P

10

; the chain is irreducible. Since

the state space is ﬁnite we conclude from Theorem 1.2 that the chain is positive

recurrent for any λ > 0 and µ > 0. We next solve for P

0

and P

1

. We let ρ = λ/µ.

There is only one balance equation, λP

0

= µP

1

. So P

1

= ρP

0

and since P

0

+P

1

= 1,

we conclude that P

0

= 1/(1 + ρ), P

1

= ρ/(1 + ρ). So the long-run proportion of

time that the server is busy is ρ/(1 + ρ) and the long-run proportion of time that

the server is free (idle) is 1/(1 + ρ).

4. M/M/∞ queue: X(t) denotes the number of customers (busy servers) in the system

at time t. Being a Birth and Death process we need only consider the Birth and

Death balance equations (26) which take the form

λP

j

= (j + 1)µP

j+1

, j ≥ 0.

Irreducibility follows from the fact that the embedded chain is an irreducible simple

random walk, so positive recurrence will follow if we can solve the above equations.

As is easily seen by recursion, P

j

= ρ

j

/j!P

0

. Forcing these to sum to one (via

using the Taylor’s series expansion for e

x

), we obtain 1 = e

ρ

P

0

, or P

0

= e

−ρ

. Thus

P

j

= e

−ρ

ρ

j

/j! and we end up with the Poisson distribution with mean ρ:

The M/M/∞ queue is always positive recurrent for any λ > 0, µ > 0; its

stationary distribution is Poisson with mean ρ = λ/µ.

The above result should not be surprising, for we already studied (earlier in this

course) the more general M/G/∞ queue, and obtained the same stationary dis-

tribution. But because we now assume exponential service times, we are able to

obtain the result using CTMC methods. (For a general service time distribution

we could not do so because then X(t) does not form a CTMC; so we had to use

other, more general, methods.)

19

5. M/M/c loss queue: This is the M/M/c model except there is no waiting room; any

arrival ﬁnding all c servers busy is lost. This is the c−server analog of Example 3.

With X(t) denoting the number of busy servers at time t, we have, for any λ > 0

and µ > 0, an irreducible B&D process with a ﬁnite state space S = {0, . . . , c},

so positive recurrence follows from Theorem 1.2. The B&D balance equations (26)

are

λP

j

= (j + 1)µP

j+1

, 0 ≤ j ≤ c −1,

or P

j+1

= P

j

ρ/(j + 1), 0 ≤ j ≤ c −1; the ﬁrst c equations for the FIFO M/M/∞

queue. Solving we get P

j

= ρ

j

/j!P

0

, 0 ≤ j ≤ c, and summing to one yields

1 = P

0

1 +

c

¸

j=1

ρ

j

j!

,

yielding

P

0

=

1 +

c

¸

j=1

ρ

j

j!

−1

.

Thus

P

j

=

ρ

j

j!

1 +

c

¸

n=1

ρ

n

n!

−1

, 0 ≤ j ≤ c. (28)

In particular

P

c

=

ρ

c

c!

1 +

c

¸

n=1

ρ

n

n!

−1

, (29)

the proportion of time that all servers are busy. Later we will see from a result called

PASTA, that P

c

is also the proportion of lost customers, that is, the proportion of

arrivals who ﬁnd all c servers busy. This turns out to be a very famous/celebrated

queueing theory result because the solution in (28), in particular the formula for

P

c

in (29), turns out to hold even if the service times are not exponential (the

M/G/c-loss queue), a result called Erlang’s Loss Formula.

6. Population model with family immigration: Here we start with a general B&D pro-

cess (birth rates λ

i

, death rates µ

i

), but allow another source of population growth,

in addition to the births. Suppose that at each of the times from a Poisson process

at rate γ, independently, a family of random size B joins the population (immi-

grates). Let b

i

= P(B = i), i ≥ 1 denote corresponding family size probabilities.

Letting X(t) denote the population size at time t, we no longer have a B&D process

now since the arrival of a family can cause a jump larger than size one. The balance

20

equations (“the rate out of state j equals the rate into state j”) are:

(λ

0

+γ)P

0

= µ

1

P

1

(λ

1

+µ

1

+γ)P

1

= (λ

0

+γb

1

)P

0

+µ

2

P

2

(λ

2

+µ

2

+γ)P

2

= γb

2

P

0

+ (λ

1

+ γb

1

)P

1

+µ

3

P

3

.

.

.

(λ

j

+µ

j

+γ)P

j

= λ

j

P

j−1

+µP

j+1

+

j−1

¸

i=0

γb

j−i

P

i

, j ≥ 1.

The derivation is as follows: When X(t) = j, any one of three events can happen

next: A death (rate µ

j

), a birth (rate λ

j

) or a family immigration (rate γ). This

yields the rate out of state j. There are j additional ways to enter state j, besides

a birth from state j −1 or a death from state j +1, namely from each state i < j a

family of size j −i could immigrate (rate γb

j−i

). This yields the rate into state j.

1.14 Transitions back into the same state; P

i,i

> 0.

In our study of CTMC’s we have inherently been assuming that P

i,i

= 0 for each i ∈ S,

but this is not necessary as we illustrate here.

Suppose that 0 < P

i,i

< 0. Assume X

0

= i and let K denote the total number

of transitions (visits) to state i before making a transition out to another state. Since

X

0

= i, we count this initial visit as one such visit. Then P(K = n) = (1−p)

n−1

p, n ≥ 1,

where p = 1 − P

i,i

. Letting Y

n

denote iid exponential rvs at rate a

i

(the holding time

rate), we can represent the total holding time H

T

in state i as an independent geometric

sum

H

T

=

K

¸

n=1

Y

n

.

In particular E(H

T

) = E(K)/a

i

= 1/pa

i

. In fact H

T

∼ exp(pa

i

) as is easily seen by

deriving its Laplace transform:

E(e

sH

T

) =

pa

j

pa

i

+s

, s ≥ 0.

(Condition on K ﬁrst.)

Thus, we can reset a

i

= pa

i

, reset P

i,i

= 0 and reset P

i,j

= P

i,j

/p for j = i. This yields

the same CTMC {X(t)} (e.g., it has the same distribution), but for which P

i,i

= 0.

In any case, even if we keep P

i,i

> 0, as long as one is consistent (on both sides of the

balance equations), then the same balance equations arise in the end. We illustrate with

a simple example: A CTMC with two states, 0, 1, and embedded chain transition matrix

P =

0.25 0.75

.20 0.80

.

21

a

0

> 0 and a

1

> 0 are given non-zero holding-time rates. By deﬁnition, a

i

is the

holding time rate when in state i, meaning that after the holding time H

i

∼ exp(a

i

) is

completed, the chain will make a transition according to the transition matrix P = (P

ij

).

If we interpret a transition j → j as both a transition out of and into state j, then the

balance equations are

a

0

P

0

= (0.25)a

0

P

0

+ (0.20)a

1

P

1

a

1

P

1

= (0.75)a

0

P

0

+ (0.80)a

1

P

1

.

As the reader can check, these equations reduce to the one equation

(0.75)a

0

P

0

= (0.20)a

1

P

1

,

which is what we get if we were to instead interpret a transition j → j as neither a

transition into or out of state j. Resetting the parameters as explained above means

resetting a

0

= (0.75)a

0

, a

1

= (0.20)a

1

and P to

P =

0 1

1 0

.

So, it makes no diﬀerence as far as {X(t)} is concerned

3

. This is how it works out for

any CTMC.

1.15 Poisson Arrivals See Time Averages (PASTA)

For a stable M/M/1 queue, let π

a

j

denote the long-run proportion of arrivals who, upon

arrival, ﬁnd j customers already in the system. If X(t) denotes the number in system at

time t, and t

n

denotes the time of the n

th

Poisson arrival, then

π

a

j

def

= lim

N→∞

1

N

N

¸

n=1

I{X(t

n

−) = j},

where X(t

n

−) denotes the number in system found by the n

th

arrival.

On the one hand, λπ

a

j

is the long-run rate (number of times per unit time) that X(t)

makes a transition j → j +1. After all, arrivals occur at rate λ, and such transitions can

only happen when arrivals ﬁnd j customers in the system. On the other hand, from the

B&D balance equations (21), λP

j

is also the same rate in question. Thus λπ

a

j

= λP

j

, or

π

a

j

= P

j

, j ≥ 0,

which asserts that

3

But there might be other associated stochastic processes that will become diﬀerent by making this

change. For example, in queueing models, allowing P

i,i

> 0 might refer to allowing customers to return

to the end of the queue for another round after completing service. By resetting P

i,i

= 0, we are forcing

the customer to re-enter service immediately for the extra round instead of waiting at the end of the

queue. This of course would eﬀect quantities of interest such as average waiting time.

22

the proportion of Poisson arrivals who ﬁnd j customers in the system is equal

to the proportion of time there are j customers in the system.

This is an example of Poisson Arrivals See Time Averages (PASTA), and it turns out

that PASTA holds for any queueing model in which arrivals are Poisson, no matter how

complex, as long as a certain (easy to verify) condition, called LAC, holds. (Service

times do not need to have an exponential distribution, they can be general, as in the

M/G/∞ queue.) Moreover, PASTA holds for more general quantities of interest besides

number in system. For example, the proportion of Poisson arrivals to a queue who, upon

arrival, ﬁnd a particular server busy serving a customer with a remaining service time

exceeding x (time units) is equal to the proportion of time that this server is busy serving

a customer with a remaining service time exceeding x. In general, PASTA will not hold

if the arrival process is not Poisson.

To state PASTA more precisely, let {X(t) : t ≥ 0} be any stochastic process, and

ψ = {t

n

: n ≥ 0} a Poisson process. Both processes are assumed on the same probability

space. We have in mind that X(t) denote the state of some “queueing” process with which

the Poisson arriving “customers” are interacting/participating. The state space S can

be general such as multi-dimensional Euclidean space. We assume that the sample-paths

of X(t) are right-continuous with left-hand limits.

4

The lack of anticipation condition (LAC) that we will need to place on the Poisson

process asserts that for each ﬁxed t > 0, the future increments of the Poisson process

after time t, {N(t +s) −N(t) : s ≥ 0}, be independent of the joint past, {(N(u), X(u)) :

0 ≤ u ≤ t}. This condition is stronger than the independent increments property of

the Poisson process, for it requires that any future increment be independent not only

of its own past but of the past of the queueing process as well. If the Poisson process

is completely independent of the queueing process, then LAC holds, but we have in

mind the case when the two processes are dependent via the arrivals being part of and

participating in the queueing system.

Let f(x) be any bounded real-valued function on S, and consider the real-valued

process f(X(t)). We are now ready to state PASTA. (The proof, ommitted, is beyond

the scope of this course.)

Theorem 1.3 (PASTA) If the Poisson process satisﬁes LAC, then w.p.1.,

lim

N→∞

1

N

N

¸

n=1

f(X(t

n

−)) = lim

t→∞

1

t

t

0

f(X(s))ds,

in the sense that if either limit exists, then so does the other and they are equal.

4

A function x(t), t ≥ 0, is right-continuous if for each t ≥ 0, x(t+)

def

= lim

h↓0

X(t + h) = x(t).

It has left-hand limits if for each t > 0, x(t−)

def

= lim

h↓0

x(t − h) exists (but need not equal x(t)). If

x(t−) = x(t+), then the function is said to be discontinuous at t, or have a jump at t. Queueing processes

typically have jumps at arrval times and departure times.

23

A standard example when X(t) is the number of customers in a queue, would be to

let f denote an indicator function; f(x) = I{x = j}, so that f(X(t)) = I{X(t) = j}, and

f(X(t

n

−)) = I{X(t

n

−) = j}. This would, for example, yield π

a

j

= P

j

for the M/M/1

queue.

The reader should now go back to Example 5 in Section 1.13, the M/M/c-loss queue,

where we ﬁrst mentioned PASTA in the context of Erlang’s Loss Formula.

1.16 Multi-dimensional CTMC’s

So far we have assumed that a CTMC is a one-dimensional process, but that is not

necessary. All of the CTMC theory we have developed in one-dimension applies here as

well (except for the Birth and Death theory). We illustrate with some two-dimensional

examples, higher dimensions being analogous.

1. Tandem queue: Consider a queueing model with two servers in tandem: Each

customer, after waiting in line and completing service at the ﬁrst single-server

facility, immediately waits in line at a second single-server facility. Upon completion

of the second service, the customer ﬁnally departs. in what follows we assume that

the ﬁrst facility is a FIFO M/M/1, and the second server has exponential service

times and also serves under FIFO, in which case this system is denoted by

FIFO M/M/1/ −→ /M/1.

Besides the Poisson arrival rate λ, we now have two service times rates (one for each

server), µ

1

and µ

2

. Service times at each server are assumed i.i.d. and independent

of each other and of the arrival process.

Letting X(t) = (X

1

(t), X

2

(t)), where X

i

(t) denotes the number of customers in

the i

th

facility, i = 1, 2, it is easily seen that {X(t)} satisﬁes the Markov property.

This is an example of an irreducible two-dimensional CTMC. Balance equations

(rate out of a state equals rate into the state) can be set up and used to solve for

stationary probabilities. Letting P

n,m

denote the long-run proportion of time there

are n customers at the ﬁrst facility and m at the second (a joint probability),

λP

0,0

= µ

2

P

0,1

,

because the only way the chain can make a transion into state (0, 0) is from (0, 1)

(no one is at the ﬁrst facility, exactly one customer is at the second facility, and

this one customer departs (rate µ

2

)). Similarly when n ≥ 1, m ≥ 1,

(λ + µ

1

+µ

2

)P

n,m

= λP

n−1,m

+µ

1

P

n+1,m−1

+µ

2

P

n,m+1

,

because either a customer arrives, a customer completes service at the ﬁrst facility

and thus goes to the second, or a customer completes service at the second facility

24

and leaves the system. The remaining balance equations are also easily derived.

Letting ρ

i

= λ/µ

i

, i = 1, 2, it turns out that the solution is

P

n,m

= (1 −ρ

1

)ρ

n

1

×(1 −ρ

2

)ρ

m

2

, n ≥ 0, m ≥ 0,

provided that ρ

i

< 1, i = 1, 2. This means that as t → ∞, X

1

(t) and X

2

(t)

become independent r.v.s. each with a geometric distribution. This result is quite

surprising because, after all, the two facilities are certainly dependent at any time t,

and why should the second facility have a stationary distribution as if it were itself

an M/M/1 queue? (For example, why should departures from the ﬁrst facility be

treated as a Poisson process at rate λ?) The proof is merely a “plug in and check”

proof using Theorem 1.2: Plug in the given solution (e.g., treat it as a “guess”)

into the balance equations and verify that they work. Since they do work, they are

the unique probability solution, and the chain is positive recurrent.

It turns out that there is a nice way of understanding part of this result. The

ﬁrst facilty is an M/M/1 queue so we know that X

1

(t) by itself is a CTMC with

stationary distribution P

n

= (1 − ρ

1

)ρ

n

1

, n ≥ 0. If we start oﬀ X

1

(0) with this

stationary distribution (P(X

1

(0) = n) = P

n

, n ≥ 0), then we know that X

1

(t) will

have this same distribution for all t ≥ 0, that is, {X

1

(t)} is stationary. It turns out

that when stationary, the departure process is itself a Poisson process at rate λ,

and so the second facility (in isolation) can be treated itself as an M/M/ 1 queue

when {X

1

(t)} is stationary. This at least explains why X

2

(t) has the geometric

stationary distribution, (1 −ρ

2

)ρ

m

2

, m ≥ 0, but more analysis is required to prove

the independence part.

2. Jackson network:

Consider two FIFO single-server facilities (indexed by 1 and 2), each with exponen-

tial service at rates µ

1

and µ

2

respectively. For simplicity we refer to each facility as

a “node”. Each node has its own queue with its own independent Poisson arrival

process at rates λ

1

and λ

2

respectively. Whenever a customer completes service

at node i = 1, 2, they next go to the queue at node j = 1, 2 with probability Q

ij

,

independent of the past, or depart the system with probability Q

i,0

, where the state

0 refers to departing the system, and we require that Q

0,0

= 1, an absorbing state.

We always assume that states 1 and 2 are transient, and state 0 is absorbing. So

typically, a customer gets served a couple of times, back and forth between the

two nodes before ﬁnally departing. In general, we allow feedback, which means

that a customer can return to a given node (perhaps many times) before departing

the system. The tandem queue does not have feedback; it is the special case when

Q

1,2

= 1 and Q

2,0

= 1 and λ

2

= 0, an example of a feedforward network. In general,

Q = (Q

ij

) is called the routing transition matrix, because it represents the transi-

tion matrix of a Markov chain. Letting X(t) = (X

1

(t), X

2

(t)), where X

i

(t) denotes

the number of customers in the i

th

node, i = 1, 2, {X(t)} yields an irreducible

25

CTMC. Like the tandem queue, it turns out that the stationary distribution for

the Jackson network is of the product form

P

n,m

= (1 −ρ

1

)ρ

n

1

×(1 −ρ

2

)ρ

m

2

, n ≥ 0, m ≥ 0,

provided that ρ

i

< 1, i = 1, 2. Here

ρ

i

=

λ

i

µ

i

E(N

i

),

where E(N

i

) is the expected number of times that a customer attends the i

th

facility. E(N

i

) is completely determined by the routing matrix Q: Each customer,

independently, is routed according to the discrete-time Markov chain with transition

matrix Q, and since 0 is absorbing (and states 1 and 2 transient), the chain will

visit each state i = 1, 2 only a ﬁnite number of times before getting absorbed.

Notice that α

i

= λ

i

E(N

i

) represents the total arrival rate to the i

th

node. So

ρ

i

< 1, i = 1, 2, just means that the total arrival rate must be smaller than the

service rate at each node. As with the tandem queue, the proof can be carried out

by the “plug in and check” method. The α

i

can be computed equivalently as the

solution to the ﬂow equations:

α

i

= λ

i

+

¸

j

α

j

Q

j,i

, i, j ∈ {1, 2}.

Letting Q

T

= (Q

j,i

), i, j ∈ {1, 2}, denote the 2 × 2 matrix without the absorbing

state 0 included, the ﬂow equations in matrix form are

α =

λ + αQ

T

,

with solution

α =

λ(I −Q

T

)

−1

.

We recognize that (I −Q

T

)

−1

= S = (s

i,j

), where s

i,j

denotes the expected number

of times the discrete-time chain visits transient state j given it started in transient

state i, i = 1, 2.

26

But unlike the discrete-time case, there is no smallest “next time” until the next transition, there is a continuum of such possible times t. For each ﬁxed i and j, Pij (t), t ≥ 0 deﬁnes a function which in principle can be studied by use of calculus and diﬀerential equations. Although this makes the analysis of CTMC’s more diﬃcult/technical than for discrete-time chains, we will, non-the-less, ﬁnd that many similarities with discrete-time chains follow, and many useful results can be obtained. A little thought reveals that the holding times must have the memoryless property and thus are exponentially distributed. To see this, suppose that X(t) = i. Time t lies somewhere in the middle of the holding time Hi for state i. The future after time t tells us, in particular, the remaining holding time in state i, whereas the past before time t, tells us, in particular, the age of the holding time (how long the process has been in state i). In order for the future to be independent of the past given X(t) = i, we deduce that the remaining holding time must only depend (in distribution) on i and be independent of its age; the memoryless property follows. Since an exponential distribution is completely determined by its rate we conclude that for each i ∈ S, there exists a constant (rate) ai > 0, such that the chain, when entering state i, remains there, independent of the past, for an amount of time Hi ∼ exp(ai ): A CTMC makes transitions from state to state, independent of the past, according to a discrete-time Markov chain, but once entering a state remains in that state, independent of the past, for an exponentially distributed amount of time before changing state again. Thus a CTMC can simply be described by a transition matrix P = (Pij ), describing how the chain changes state step-by-step at transition epochs, together with a set of rates {ai : i ∈ S}, the holding time rates. Each time state i is visited, the chain spends, on average, E(Hi ) = 1/ai units of time there before moving on.

1.1

Interpretation of the {ai } as transition rates; the transition rate matrix (inﬁnitesimal generator) Q

Assume here that Pi,i = 0 for all i ∈ S. ai can be interpreted as the transition rate out of state i given that X(t) = i; the intuitive idea being that the exponential holding time will end, independent of the past, in the next dt units of time with probability ai dt. This can be made rigorous recalling the little o(t) results for the Poisson process: Let {Ni (t)} denote a Poisson counting process at rate ai . Then P (Ni (h) > 0) = ai h + o(h) and P (Ni (h) = 1) = ai h + o(h) (and P (Ni (h) > 1) = o(h)). Thus lim P (X(h) = i | X(0) = i)/h = lim P (Ni (h) = 1)/h = lim(ai h + o(h))/h = ai .

h↓0 h↓0 h↓0

(1)

More generally, for j = i, we additionally use the Markov property asserting that once leaving state i the chain will, independent of the past, next go to state j with probability 2

3 Chapman-Kolmogorov equations The Chapman-Kolmogorov equations for discrete-time Markov chains generalizes to Lemma 1.1 a1 P1. called the embedded Markov chain. 3. of the states visited right after each transition. deﬁnes the underlying discrete-time Markov chain.Pi.3 a0 P0.3 a2 P2.j can thus be interpreted as the transition rate from state i to state j given that the chain is currently in state i.2 a1 P1. for all t ≥ 0. When i = j. 1.j (h)/h = lim P (Ni (h) = 1)Pi.1 a3 P3.i (h) = 1 − P (X(h) = i | X(0) = i) and so Pi.4 a1 P1. {Xn } keeps track. we see that Xn = X(τn +).i (0) = lim(Pi.3 −a3 a4 P4. h↓0 h↓0 (3) Deﬁnition 1.0 a4 P4. s ≥ 0. that is. consecutively. 4}.j (0) = lim Pi.j /h = ai Pi. together with the holding-time rates {ai }. P (t + s) = P (s)P (t).2 −a2 a3 P3.i = 0.1 (Chapman-Kolmogorov equations for CTMC’s) For all t ≥ 0.4 a2 P2. and moves from state to state according to the one-step transition probabilities Pij = P (Xn+1 = j|Xn = i).0 −a1 Q = a2 P2. i ∈ S. we conclude that each row of Q sums to 0.3 a0 P0. of the Markov chain For example. completely determines the CTMC.2 The matrix Q = P (0) given explicitly by (2) and (3) is called the transition rate matrix. 1. or inﬁnitesimal generator. j ∈ S Pij (t + s) = k∈S Pik (s)Pkj (t).j .0 a2 P2. Pi. 3 . then −a0 a0 P0.0 a3 P3.1 a4 P4.4 a3 P3.3 a0 P0.j to obtain Pi. the state right after the nth transition.4 −a4 Note in passing that since we assume that Pi.2 a4 P4. s ≥ 0.i (h) − 1)/h = lim −P (Ni (h) = 1)/h = −ai . 1.3 a1 P1. i ∈ S. This transition matrix (Pij ). if S = {0. h↓0 h↓0 (2) ai Pi. 2.2 The embedded discrete-time Markov chain Letting τn denote the time at which the nth change of state (transition) occurs.

Consider the rat in the closed maze. Note how cell 4 has the shortest holding time (mean 1/4 minutes). . There is one line (queue) to wait in. and cell 1 has the longest (mean 1 minute). ti+1 − ti . the (easy) proof involves ﬁrst conditioning on what state k the chain is in at time s given that X(0) = i.=1/2. Pi. P2 . i = 1. and then using the Markov property to conclude that the probability that the chain. Time is in minutes (say). and (2) N (t) is a nondecreasing process. Given the rat is now in cell 2 (say). and then move. they serve as the continuous-time analog to the discrete-time stationary probabilities πi for discrete-time Markov chains. independent of the past.p. . independent of the past. Poisson counting process: Let {N (t) : t ≥ 0} be the counting process for a Poisson process ψ = {tn } at rate λ. Then {N (t)} forms a CTMC with S = {0. where “system” 4 . As t → ∞. 2. the embedded chain is deterministic. in cell i is exponential at rate ai = i.i+1 = 1. 1. 1. 2. Let X(t) denote the cell that the rat is in at time t.4 Examples of CTMC’s 1. independent of the past. P3 .}. i = 1. Of intrinisic interest is to calculate the long-run proportion of time (continuous time now) that the rat spends in each cell. Let X(t) denote the number of customers in the system at time t. the rat is equally likely to move to one of the neighboring two cells. The other transitions are similarly explained. by the memoryless property. he will remain there. n ≥ 0. Hi . occur after an exponentially distributed amount of time at rate λ. FIFO M/M/1 queue: Arrivals to a single-server queue are Poisson at rate λ. to either cell 1 or 4 w. 3. yielding Pik (s). 3. This is a very special kind of CTMC for several reasons. P = (P1 . i ≥ 0: If N (t) = i then. now in state k. {X(t)} forms a CTMC. P4 ) is called the limiting (stationary) distribution for the CTMC. for an exponential amount of time with mean 1/2. the next arrival. (1) all holding times Hi have the same rate ai = λ. it increases by one at each arrival time. would then be in state j after an additional t time units is. Pi = lim def 1 t→∞ t t I{X(s) = i}ds. but where now we assume that the holding time. and customers independently (and independent of the Poisson arrival process) have service times {Sn } that are exponentially distributed at rate µ. ai = λ. will. Pkj (t). The holding time in state i is simply the interarrival time.As for discrete-time chains. 2. 4. . 0 We will learn how to compute these later. and remains constant otherwise. We assume that customers join the tail of the queue. arrival i + 1. 3. 4. and hence begin service in the order that they arrive ﬁrst-in-queue-ﬁrst-out-of-queue (FIFO). independent of the past. in which at each transition. and τn = tn since N (t) only changes state at an arrival time. Assuming that N (0) = 0 we conclude that Xn = N (tn +) = n. N (t) ↑ ∞ step by step. 2.

4. X(t) = 4 means that two customers are in service (each with their own server) and two others are waiting in line. X(t) denotes the number of customers in the system at time t. and Pi. If X(t) = i ≥ 1. X} where Sr is the remaining service time of the customer in service. Note that a transition can only occur at customer arrival or departure times. “down” probability 1 − p = µ/(λ + µ)) that is restricted to be non-negative (P0. let’s assume c = 2. a0 = λ. the arrival rate. and that departures occur whenever a service completion occurs. Then. so the holding time is given by H0 ∼ exp(λ) the time until the next arrival. P (Xn+1 = 1|Xn = 0) = 1. We conclude that the holding time in any state i ≥ 2 is given by Hi = min{X. both remaining service times. Sr2 } ∼ exp(2µ). i ≥ 1. As in a USA postoﬃce.i−1 = 1 − p = µ/(λ + µ).1 = 1). 5 . for example. then the holding time is given by Hi = min{Sr . Since they are independent exponentials at rate µ. and Xn+1 = i − 1 w. Sr2 } ∼ exp(λ + 2µ). Pi.p.p. whereas at a departure time X(t) jumps down by the amount 1. X(t) = 2 means that there is one customer in service and one waiting in line. a0 = λ. Transitions are caused only by arrivals and departures. and for i ≥ 1. arrivals wait in one FIFO line (queue) and enter service at the ﬁrst available free server. 1}. So. Determining the rates ai : If X(t) = 0 then only an arrival can occur next. Also.v. But if Xn = i ≥ 1. So (for example).means the line plus the service area. Sr1 . they are independent r. If Xn = 0. compete to determine the next departure. P (X < Sr ) = λ/(λ + µ). M/M/c multi-server queue: This is the same as the FIFO M/M/1 queue except there are now c servers working in parallel. When X(t) = i ∈ {0.1 = 1. and X is the time until the next arrival. But when X(t) = i ≥ 2. because the service times are assumed independent of the Poisson arrival process. we deduce that the time until the next departure is given by min{Sr1 . denoted by Sr1 and Sr2 . For illustration. The transition probabilities Pij for the embedded discrete-time chain are derived as follows: Xn denotes the number of customers in the system right after the nth transition.s.i+1 = p = λ/(λ + µ). then the system is empty and we are waiting for the next arrival. P0. The point here is that each of the two independent events “service completion will ocurr”. a1 = λ + µ. “new arrival will ocurr” is competing to be the next event so as to end the holding time. The time until the next arrival is given by X ∼ exp(λ) and is independent of both remaining service times. We conclude that The embedded Markov chain for a FIFO M/M/1 queue is a simple random walk (“up” probability p = λ/(λ + µ). P (Sr < X) = µ/(λ + µ). Thus Hi ∼ exp(λ + µ) and ai = λ + µ. At an arrival time X(t) jumps up by the amount 1. then Xn+1 = i + 1 w. The memoryless property for both service times and interarrival times implies that Sr ∼ exp(µ) and X ∼ exp(λ) independent of the past. depending on whether an arrival or a departure is the ﬁrst event to occur next. the holding times are the same as for the M/M/1 model.

) This means that whenever X(t) = i ≥ 1. Then for i ≥ c.p. This simple random walk thus has state-dependent “up”. This is an example of a simple random walk with state-dependent “up”.i+1 = λ/(λ + cµ). in which case X(t) can be thought of as the population size at time t.i+1 = λ/(λ + iµ). Y } ∼ exp(λi + µi ).p. (In Example 2. M/M/∞ inﬁnite-server queue: Here we have a M/M/c queue with c = ∞. the current state. Pi.For the general case of c ≥ 2. For each state i ≥ 0 we have a birth rate λi and a death rate µi : Whenever X(t) = i.i+1 = λ/(λ + iµ).1 = 1 and Pi. Pi.i−1 = cµ/(λ + cµ).) Here we study B&D processes more formally.5 Birth and Death processes Except for Example 2 (rat in the closed maze) all of the CTMC examples in the previous section were Birth and Death (B&D) processes. the rates are determined analogously: ai = λ+iµ. i > c. Pi. i ≥ 1.v. since they tend to be a very useful type of CTMC. Y ∼ exp(µi ). we say there is a birth. independent of the past. the next transition will be a birth w.i+1 + Pi.i−1 . We shall focus on the case when S = {0. 1. Letting X(t) denote the number of customers in the system at time t. Note how. 5. ai = λ + cµ.i−1 = iµ/(λ + iµ). or decreasing by one. Thus the holding time rates are given by ai = λi + µi because the time until the next transition (change of state) in given by the holding time Hi = min{X. and whenever it decreases by one we say there is a death. i ≥ 0 since there is no limit on the number of busy servers. “down” probabilities. Pi. the probabilities for the next increment depend on i. Pi. the current state. the time until the next death is a r. 0 ≤ i ≤ c. and approaches 1 as i → ∞: when the system is heavily congested. departures occur rapidly. 6 .i−1 = 1. “down” probabilities: at each step. .i−1 = P (Y < X) = µi /(λi + µi ). independently. for example. 2. CTMC’s that can only change state by increasing by one.v. and a death w. . Pi. The idea here is that at any given time the next birth is competing with the next death to be the next transition. “down” probabilities that continue to depend on each state i. until i = c at which point the probabilities remain constant.}.3 > 0. a special case of the M/G/∞ queue. . Whenever the state increases by one. i ∈ S. Pi. X ∼ exp(λi ) and. P1. (We always assume here that µ0 = 0 since there can be no deaths without a population. as i increases. For the embedded chain: P0. 1. For the embedded chain: P0. Thus the embedded chain for a B&D process is a simple random walk with state dependent “up”.1 = 1 and for 0 ≤ i ≤ c−1.i−1 = iµ/(λ + iµ). this model is always stable. so it is not B&D. Pi. Pi. increases. the down probability. the time until the next birth is a r.i+1 = P (X < Y ) = λi /(λi + µi ). we see that ai = λ + iµ.

the non-explosive condition is satisﬁed. so the arrivals can not cause an explosion. w. Consequently. This is an example of an explosive Markov chain: The number of transitions in a ﬁnite interval of time is inﬁnite.1. and assume from now on that all CTMC’s considered are non-explosive. Now observe that during any interval of time. but this can be generalized by allowing each λi to be diﬀerent. We shall rule out this kind of behavior in the rest of our study. and λi > 0. the number of departures can be no larger than N (t).p. we now explore how fast this happens. the total number of arrivals thus far. Thus the chain will visit all of the states by time ∞ T = i=0 Hi . Only the M/M/∞ queue needs some clariﬁcation since ai = λ + iµ → ∞ as i → ∞. Note that the chain will visit state i at time H0 + H1 + · · · + Hi−1 . Pi. Taking expected value yields ∞ E(T ) = i=0 2−i = 2 < ∞. (s. the population continues to increase by one at each transition. Since X(t) → ∞ as t → ∞. and the arrivals come from the Poisson process at ﬁxed rate λ. so they too can not cause an explosion. the sum of the ﬁrst i holding times. 1. In short.When µi = 0. consequently jumping to the next state faster and faster as time goes on. 7 . The reader is encouraged at this point to go back over the B&D Examples in the previous Section. t]. i ≥ 0. all states will be visited in a ﬁnite amount of time. E(Hi ) = 1/λi = 2−i units of time in state i and then changes to state i + 1. a ﬁnite amount of time! But this implies that w.. we call the process a pure birth process.6 Explosive CTMC’s Consider a pure birth process {X(t)}. Thus it spends less and less time in each state. or any CTMC for which there are only a ﬁnite number of distinct values for the rates ai . N (t) < ∞. This will always hold for any CTMC with a ﬁnite state space. i ≥ 0. The main example is the Poisson counting process (Example 1 in the previous Section).. i ≥ 0. t ≥ 0. in which ai = λi = 2i .i+1 = 1. the number of transitions in any interval (s. This method of bounding the number of transitions by the underlying Poisson arrival process will hold for essentially any CTMC queueing model. X(T + t) = ∞. by which we mean that the number of transitions in any ﬁnite interval of time is ﬁnite. and more generally whenever sup{ai : i ∈ S} < ∞. on average.p. Every Example given in the previous Section was non-explosive. P (T < ∞) = 1. t] is bounded from above by 2N (t) < ∞. . t ≥ 0.1. But only arrivals and departures determine transitions. and we conclude that on average all states i ≥ 0 have been visited by time t = 2. i ≥ 0. This process spends.

E(Tii ) < ∞. Thus communication classes all have the same type of states: all together they are transient or all together they are recurrent. so Tii ≥ Hi . The chain will make its ﬁrst transition at time Hi (holding time in state i).i denote the amount of (continuous) time until the chain re-visits state i (at a later transition) given that X(0) = i (deﬁned to be ∞ if it never does return). The state is called transient otherwise. As with discrete-time chains. the return time to state i. We state in passing A CTMC is irreducible if and only if its embedded chain is irreducible. one communication class). i and j are said to communicate if state j is reachable from state i. all states together are transient. we can partition the state space up into disjoint communication classes. the expected amount of time to return is ﬁnite. in addition to being recurrent. positive recurrent.. It is immediate that i and j communicate in continuous time if and only if they do so for the embedded discrete-time chain {Xn }: They communicate in continuous-time if and only if they do so at transition epochs. the expected amount of time to return is inﬁnite. State i is called recurrent if. and state i is reachable from state j. the chain will re-visit state i with certainty.1. 8 . Thus once again. S = C1 ∪ C2 ∪ · · · . State i is called null recurrent if. so in particular: For an irreducible CTMC. transience and positive recurence are similar as for discretetime chains: Let Ti. in addition to being recurrent. This (with a little thought) is seen to be the same property as for the embedded chain (because X(t) returns to state i for some t if and only if Xn does so for some n): A state i is recurrent/transient for a CTMC if and only if it is recurrent/transient for the embedded discrete-time chain.8 Positive recurrence and the existence a limiting distribution P = {Pj } State i is called positive recurrent if. Unlike recurrence. or null recurrent. But positive and null recurrence are still class properties.p.1. E(Tii ) = ∞. irreducibility and recurrence State j is said to be reachable from state i for a CTMC if P (X(s) = j|X(0) = i) = Pij (s) > 0 for some s ≥ 0. that is. and an irreducible chain is a chain for which all states communicate (S = C1 . if P (Tii < ∞) = 1. positive (or null) recurrence is not equivalent to that for the embedded chain: It is possible for a CTMC to be positive recurrent while its embedded chain is null recurrent (and vice versa). 1. w. Notions of recurrence.7 Communication classes.

we can break up the evolution of the CTMC into i.1 If {X(t)} is a positive recurrent CTMC.1. Letting P P∗ = P (6) . The basic idea is that for ﬁxed state j.A CTMC is called positive recurrent if it is irreducible and all states are positive recurrent. 0 (4) which after taking expected values yields Pj = lim 1 t→∞ t t Pij (s)ds. This leads to lim Proposition 1. Moreover. E(Tjj ) E(Tjj ) In words: “The long-run proportion of time the chain spends in state j equals the expected amount of time spent in state j during a cycle divided by the expected cycle length (between visits to state j)”. distributed as Tjj . t→∞ (8) Finally.i. the chain spends an amount of time in state j equal in distribution to the holding time Hj .i. w.d. and is given by Pj = 1/aj E(Hj ) = > 0. i. . then P = {Pj } (as a row vector) is called the limiting (or stationary) distribution for the Markov chain. . j ∈ S.p.j ) as deﬁned by Equation (4) exists. then the limiting probability distribution P = (Pi.. where a cycle begins every time the chain makes a transition into state j. then Pj = 0. This yields an example of what is called a regenerative process because we say it regenerates every time a cycle begins. j ∈ S. denote the matrix in which each row is the limiting probability distribution P . the stronger mode of convergence (weak convergence) holds: Pj = lim Pij (t). 9 . 0 (5) When each Pj exists and j Pj = 1. and during a cycle. no limiting distribution exists.d. cycles. (7) t→∞ t 0 As for discrete-time Markov chains. j ∈ S. We deﬁne (when they exist. independent of initial condition X(0) = i) the limiting probabilities {Pj } for the CTMC as the long-run proportion of time the chain spends in each state j ∈ S: Pj = lim 1 t→∞ t t I{X(s) = j|X(0) = i}ds. (5) can be expressed nicely in matrix form as 1 t P (s)ds = P∗ . if the chain is either null recurrent or transient. The cycle lengths are i. is unique. positive recurrence implies the existence of limiting probabilities by use of the SLLN.

sequence of r. t]. Pj = E(Tjj ) . Thus t Nj (t) I{X(s) = j}ds ≈ 0 n=1 Jn .. distributed as the return time Tjj . (the amount of time spent in state j during the nth cycle) we conclude that {Jn } forms an i. yielding Pj = 0 as well from (4).d. and we let Nj (t) denote the number of such points during (0.d. then I{X(s) = j|X(0) = i} → 0 as s → ∞. Pj > 0 if E(Tjj ) < ∞ (positive recurrence). the cycle lengths. where a cycle begins every time the chain makes a transition into state j. and is independent of the past. E(J) = E(Hj ).Proof : Fixing state j. in particular. 2 10 .d. is non-lattice. wp1. whereas Pj = 0 if E(Tjj ) = ∞ (null recurrence). In general. Tn (j) = τn (j) − τn−1 (j). t→∞ lim 1 Nj (t) = . Any continuous distribution. 1/aj Uniqueness of the Pj follows by the unique representation. cycles. The distribution of a non-negative rv X is said to be non-lattice if there does not exists a d > 0 such that P (X ∈ {nd : n ≥ 0}) = 1. And if transient.i.s. {τn (j) : n ≥ 1} forms a renewal point process because of the assumed recurrence. from which we obtain 1 t Letting t → ∞ yields Pj = E(Hj ) . since every time the chain enters state j.i.2 Tjj has a non-lattice distribution because it is of phase type hence a continuous distribution. the chain starts over again from scratch stochastically. From the Elementary Renewal Theorem. The weak convergence in (8) holds in addition to the already established time-average convergence because the cycle-length distribution (the distribution of Tjj for any ﬁxed j) is non-lattice. The details of this will be dealt with later when we return to a more rigorous study of renewal theory.i. n=1 where the denominator is from (9) and the numerator is from the SLLN applied to {Jn }. are i. distributed as the holding time Hj . This follows by the (strong) Markov property.v. n ≥ 1. t E(Tjj ) τn (j) (9) Letting Jn = τn−1 (j) I{X(s) = j}ds. E(Tjj ) t 0 Nj (t) 1 I{X(s) = j}ds ≈ × t Nj (t) Nj (t) Jn . Letting τn (j) denote the nth time at which the chain makes a transition into state j. a positive recurrent regenerative process with a non-lattice cycle-length distribution converges weakly. wp1. we can break up the evolution of the CTMC into i. of the chain. with τ0 (j) = 0.

P P (t) = P . and if X(0) ∼ ν. note that νP∗ = P for any probability distribution ν.2 For a positive recurrent Markov chain with limiting distribution P : If X(0) ∼ P . t ≥ 0. 1. the same limiting distribution P is obtained regardless..) 11 .g.j (t) = Pj . 0 (10) This merely means that the initial condition (e.g. X(t) ∼ νP (t). letting {X ∗ (t) : t ≥ 0} denote the chain when X(0) ∼ P . that is. Recalling the deﬁnition of P∗ in (6).9 Allowing an arbitrary random initial value for X(0) If ν is a probability distribution on S.1. X(0) = i) without eﬀecting the limit. Moreover. for which P P (t) = P . it forms a stationary stochastic process: {X ∗ (s + t) : t ≥ 0} has the same distribution for all s ≥ 0. then the distribution of X(t) is given by the vector-matrix product νP (t). (Intuitively we are simply asserting that P (∞)P (t) = P (∞) because ∞ + t = ∞. j ∈ S. t ≥ 0. then X(t) ∼ P for all t ≥ 0. t ≥ 0. Proof : From the deﬁnition of P∗ (each row is P ) we must equivalently show that P∗ P (t) = P∗ for any t ≥ 0.. the value of X(0)) can be random as opposed to only deterministic (e. the limiting distribution P is also called a stationary distribution because it yields a stationary version of the chain if the chain is initially distributed as P at time t = 0: Proposition 1. i∈S In fact this limiting distribution P is the only distribution (it is unique) that is stationary. that is. This means that Pi Pi. t ≥ 0. Thus for a positive recurrent chain it holds more generally from (7) (by multiplying each left side by ν) that 1 t→∞ t lim t νP (s)ds = P .10 The limiting distribution yields a stationary distribution and hence a stationary version of the Markov chain As in discrete-time.

0 and we conclude that ν = P . X ∗ (s) ∼ P . All elements of P (s) are bounded by 1. 1. and using (7). we get P∗ P (t) = = = = = 1 u P (s)ds P (t) u→∞ u 0 1 u P (s)P (t)ds lim u→∞ u 0 1 u lim P (s + t)ds u→∞ u 0 1 u lim P (s)ds u→∞ u 0 P∗ . We present this derivation now. Pij (t) = P (X(t) = j|X(0) = i). 12 . we have Equation (10) and hence 1 t→∞ t lim t νP (s)ds = νP∗ = P . lim The second to last equality follows due to fact that adding the ﬁxed t is asymptotically u u u+t t negligible: 0 P (s + t)du = 0 P (s)ds + u P (s)ds − 0 P (s). 0 (11) But on the other hand νP (t) = ν implies that 1 t t νP (s)ds = 0 1 t t νds = ν. a Markov process is completely determined (in distribution) by its initial state. n ≥ 1 could be computed by using the fact that the (n) matrix (Pij ) was simply the transition matrix P multiplied together n times. then on the one hand. By the Markov property. and so the last two integrals when divided by u tend to 0 as u → ∞. t ≥ 0. its initial state has the same distribution.11 Computing Pij (t): Kolmogorov backward equations We have yet to show how to compute the transition probabilities for a CTMC. P (s + t) = P (s)P (t). the stationary distribution is unique. t ≥ 0. In continuous time however. Thus {X ∗ (s + t) : t ≥ 0} has the same distribution for all s ≥ 0 because for all s. P n . If a probability distribution ν satisﬁes νP (t) = ν.Recalling the Chapman-Kolmogorov equations. since the chain is assumed positive recurrent. For discrete-time Markov chains this was not a problem (n) since Pij = P (Xn = j|X0 = i). the problem is a bit more complex. it involves setting up linear diﬀerential equations for Pij (t) known as the Kolmogorov backward equations and then solving.

Proposition 1. (Technically. P (0) = I. that is. It turns out. The derivation above can be rigorously justiﬁed for any non-explosive CTMC. P (t + h) = P (h)P (t) as opposed to in front. n! Proof : The Chapman-Kolmogorov equations. j ∈ S. t ≥ 0. however. For a (non-explosive) CTMC. yield P (t + h) − P (t) = (P (h) − I)P (t) = (P (h) − P (0))P (t). that the derivation of the analogous forward equations P (t) = P (t)Q. (13) The unique solution is thus of the exponential form. P (t + h) = P (t)P (h). P (t + h) = P (h)P (t). this involves justifying the interchange of a limit and an inﬁnite sum. that one would expect to get by using P (t + h) = P (t)P (h) can not be rigorously justiﬁed for all non-explosive CTMCs. since the unique solution P (t) = eQt to the backward equations is the unique solution to the forward equations. which indeed can be justiﬁed here even when the state space is inﬁnite. 13 . t ≥ 0. we chose to place the h on the right-hand side in back. t ≥ 0. and thus both equations are valid. P (0) = I. the following set of linear differential equations is satisﬁed by {P (t)}: P (t) = QP (t). the interchange of a limit and an inﬁnite sum can not be justiﬁed.3 (Kolmogorov Backward Equations) For a (non-explosive) CTMC with transition rate matrix Q = P (0) as in Deﬁnition 1.2.j (t) are the unique solution to both the Kolmogorov backward and forward equations. ∞ (14) e M def = n=0 Mn . where for any square matrix M . there are examples (inﬁnite state space) that cause trouble. (15) (16) dividing by h and letting h → 0 then yields P (t) = P (0)P (t) = QP (t). the transition probabilities Pi. Pij (t) = −ai Pij (t) + k=i (12) ai Pik Pkj (t). t ≥ 0.) The word backward refers to the fact that in our use of the Chapman-Kolmogorov equations. But it does not matter. i. P (t) = eQt .

Clearly. there is a unique limiting probability distribution P and it is a stationary distribution. since eM = limn→∞ (1 + M/n)n .12 Balance equations and positive recurrence Consider any deterministic function x(t). 14 (18) . by interpreting the limiting probability Pj as a proportion of time and recalling Section 1. j ∈ S. j ∈ S and is the limiting (stationary) distribution as deﬁned in Equations (4)-(7). it must ﬁrst leave that state in order to enter it again.3 The balance equations for a positive recurrent CTMC are given by aj Pj = i=j Pi ai Pij . For example. t ≥ 0 with values in S. Deﬁnition 1. For a positive recurrent CTMC with limiting distribution P = {Pj }. every time x(t) enters a state j.1. t ≥ 0. P P (t) = P . t] that it enters state j diﬀers by at most one. We conclude (by dividing by t and letting t → ∞) that the long-run rate at which the function leaves state j equals the long-run rate at which the function enters state j. Proof : Suppose that the chain is positive recurrent.1 on transition rates. Theorem 1. “the rate out of state j is equal to the rate into state j. The solution satisﬁes Pj > 0. for each state j”. j ∈ S. for any square matrix M . In words.1 An irreducible (and non-explosive) CTMC is positive recurrent if and only if there is a (necessarily unique) probability solution P to the balance equations P Q = 0.2. where Q = P (0) is the transition rate matrix from Section 1. t] that it leaves state j.Remark 1.1 It is rare that we can explicitly compute the inﬁnite sum in the solution ∞ P (t) = eQt = n=0 (Qt)n . one can choose n large and use eQt ≈ (1 + (Qt)/n)n . 1. the rate out of state j is given by aj Pj . Then from Proposition 1. Thus the number of times during the interval (0. n! But there are various numerical recipes for estimating eQt to any desired level of accuracy.1 and Proposition 1. We can apply this kind of result to each sample-path of a stochastic process. while the rate into state j is given by i=j Pi ai Pij . from the number of times during the interval (0. (17) which in matrix form are given by P Q = 0. Taking the derivative at t = 0 on both sides of P P (t) = P yields P Q = 0.

positive recurrent.the balance equations. Let τ1. T1. Conversely. For an irreducible CTMC. suppose that P is a probability solution to the balance equations. . Also.1 is a random sum of τ1.1 holding times. and let T1. 0 t t (20) But using the already established P P (t) = P we have 1 0 P P (s)ds = 1 0 P ds = P t t and we end with a contradiction P = 0 (P is a probability distribution by assumption). so the chain must be either null recurrent or transient and hence by Proposition 1. starting with H1 . we have 1 t P (s)ds = 0. all states together are transient.1 denote the corresponding continuous return time. dt But this implies that P P (t) is a constant in t and hence that P P (t) = P P (0) = P I = P . To this end: Suppose that P Q = 0. . ab }.1 denote the discrete return time to state 1. As for discrete-time Markov chains. which due to the Kolmogorov Backward equations. We then will show that if an irreducible chain has such a stationary distribution. We know that E(τ1. so we conclude that P here is indeed the limiting distribution. . from Proposition 1. or null recurrent. Then a∗ > 0 and every holding time Hi 15 . Multiplying both right sides by P (t) yields P QP (t) = 0. then the chain must be positive recurrent. P is indeed a stationary distribution. 2. (19) lim t→∞ t 0 Multiplying both sides on the left by P yields 1 lim t→∞ t t P P (s)ds = 0. is equivalent to P P (t) = 0 which is equivalent to d(P P (t)) = 0. . we obtain a useful and simple special case: Theorem 1.2 we know that there can only be one stationary distribution for a positive recurrent chain. Proof : Suppose (without loss of generality) the state space is S = {1. . when the state space is ﬁnite. b} for some integer b ≥ 1. Let a∗ = min{a1 . Finally. We will ﬁrst show that any such solution must also satisfy P P (t) = P . . it is a stationary distribution. t ≥ 0. that is.1. Now suppose that the chain is not positive recurrent. . the limiting distribution as deﬁned in Equations (4)-(7). there is always a unique probability solution to the balance equations.1 ) < ∞. We also know that the embedded chain is positive recurrent because for ﬁnite state discrete-time chains irreducibility implies positive recurrence. . P (t) = QP (t). We already know that the chain must be recurrent because the embedded chain is so.2 An irreducible CTMC with a ﬁnite state space is positive recurrent.

Given X(t) = j ≥ 1. . 1. irreducibility is immediate since as pointed out earlier. Letting {Yn } denote iid exponential rvs at rate a∗ . j ≥ 1. j ≥ 0.1 . or from state i = j + 1 (a departure ocurrs (rate µ) when X(t) = j causing the transition j + 1 → j). . the embedded chain is a simple random walk (hence irreducible). . This yields the other equations. but there are two ways to enter such a state j: either from state i = j − 1 (an arrival occurs (rate λ) when X(t) = j − 1 causing the transition j − 1 → j). we conclude that τ1. which asserts that 16 (21) . then solving for P0 by using the fact that j∈S Pj = 1. Here.1.satisﬁes E(Hi ) ≤ 1/a∗ < ∞. the rate out of state j is aj = λ + µ (either an arrival or a departure can occur).1 and 1. . (λ + µ)Pj = λPj−1 + µPj+1 . Note that since λP0 = µP1 (ﬁrst equation). . from Theorem 1. solving the resulting balance equations involves recursively expressing all the Pj in terms of one particular one. and in general the balance equations reduce to λPj = µPj+1 . independent of τ1. and the only way to enter state 0 is from state i = 1. FIFO M/M/1 queue: X(t) denotes the number of customers in the system at time t. the rate out of state 0 is the arrival rate a0 = λ. i ∈ {1. we will have positive recurrence if and only if we can solve the balance equations (17): λP0 = µP1 (λ + µ)P1 = λP0 + µP2 (λ + µ)P2 = λP1 + µP3 .1 )(1/a∗ ) < ∞. so. 1.1 ) ≤ E n=1 Yn = E(τ1. This yields the ﬁrst equation. In the case when the state space is inﬁnite.13 Examples of setting up and solving balance equations Here we apply Theorems 1. the sum is an inﬁnite sum that might diverge unless further restrictions on the system parameters (rates) are enforced.2 to a variety of models. P0 (say). the second equation reduces to λP1 = µP2 which in turn causes the third equation to reduce to λP2 = µP3 . .1 E(T1. 2. from which a departure must occur (rate µ). In most cases. b}. These equations can also be derived from scratch as follows: Given X(t) = 0.

or Pj+1 = ρPj . and transient if ρ > 1. 1−ρ 2.for each j. j ≥ 0 and we obtain a geometric stationary distribution. P2 = ρP1 = ρ2 P0 and in general Pj = ρj P0 . and in fact this reduction holds for any Birth and Death process. j ≥ 0. in which case 1 = P0 (1 − ρ)−1 . “the arrival rate is less than the service rate”.) When ρ < 1 we say that the M/M/1 queue is stable. the rate from j to j + 1 equals the rate from j + 1 to j. j ≥ 0. Using the fact that the probabilities must sum to one yields ∞ 1 = P0 j=0 ρj . from which we conclude that there is a solution if and only if the geometric series converges. Thus Pj = ρj (1 − ρ). from which we recursivly obtain P1 = ρP0 . When the queue is stable. Stability intuitively means that the queue length doesn’t grow without bound over time. or P0 = 1 − ρ. For in a Birth and Death process. unstable otherwise. (If ρ = 1 it can be shown that the chain is null recurrent. that is. Summarizing: The FIFO M/M/1 queue is positive recurrent if and only if ρ < 1 in which case its stationary distribution is geometric with paramater ρ. we can take the mean of the stationary distribution to obtain the average number of customers in the system 1 l = lim t→∞ t ∞ t X(s)ds 0 (22) (23) (24) (25) = j=0 ∞ jPj j(1 − ρ)ρj j=0 = = ρ . the rate from j to j + 1 equals the rate from j + 1 to j” is not a coincidence. equivalently λ < µ. the balance equations are: 17 . if and only if ρ < 1. Birth and Death processes: The fact that the balance equations for the FIFO M/M/1 queue reduced to “for each state j. Pj = ρj (1−ρ).

µi 1 in which case P0 = and Pj = 1+ 1+ ∞ j=1 j λi−1 i=1 µi . j ≥ 1. the rate from j to j + 1 equals the rate from j + 1 to j” holds for any deterministic function x(t). . . the Birth and Death balance equations. We note in passing that the statement “for each state j.λ 0 P0 = µ 1 P1 (λ1 + µ1 )P1 = λ0 P0 + µ2 P2 (λ2 + µ2 )P2 = λ1 P1 + µ3 P3 . j ≥ 1. ∞ j 1+ j=1 i=1 λi−1 = µi ∞ ρj . we see that λ0 × · · · × λj−1 λi−1 Pj = P0 = P0 . We conclude that for any Birth and Death process. t ≥ 0. j ≥ 1. j ≥ 0. (λj + µj )Pj = λj−1 Pj−1 + µj+1 Pj+1 . µ1 × · · · × µj µi i=1 Using the fact that the probabilities must sum to one then yields: An irreducible Birth and Death process is positive recurrent if and only if ∞ j j (26) j=1 i=1 λi−1 < ∞. Plugging the ﬁrst equation into the second yields λ1 P1 = µ2 P2 which in turn can be plugged into the third yielding λ2 P2 = µ3 P3 and so on. Solving recursively. j=0 which agrees with our previous analysis. in the M/M/1 model. the balance equations reduce to λj Pj = µj+1 Pj+1 . j λi−1 i=1 µi ∞ j λi−1 j=1 i=1 µi . 18 . (27) For example.

Thus the number of times during the interval (0. We next solve for P0 and P1 . So the long-run proportion of time that the server is busy is ρ/(1 + ρ) and the long-run proportion of time that the server is free (idle) is 1/(1 + ρ). t] that it makes a “down” transition from j + 1 to j. The above result should not be surprising.in which changes of state are only of magnitude 1. But because we now assume exponential service times. departs without being served. 1} and X(t) = 1 if the server is busy and X(t) = 0 if the server is free. from the number of times during the interval (0. Being a Birth and Death process we need only consider the Birth and Death balance equations (26) which take the form λPj = (j + 1)µPj+1 . we are able to obtain the result using CTMC methods. the only way it can do so again is by ﬁrst going back down from j +1 to j. µ > 0. as for the balance equations. for we already studied (earlier in this course) the more general M/G/∞ queue. Pj = ρj /j!P0 . being able to write this statement simply as λj Pj = µj+1 Pj+1 crucially depends on the Markov property 3. that is. Since the state space is ﬁnite we conclude from Theorem 1. any customer arriving when the server is busy is “lost”. except there is no waiting room. M/M/1 loss system: This is the M/M/1 queueing model. the chain is irreducible. we conclude that P0 = 1/(1 + ρ). So P1 = ρP0 and since P0 + P1 = 1. Forcing these to sum to one (via using the Taylor’s series expansion for ex ). we obtain 1 = eρ P0 . 4. methods. M/M/∞ queue: X(t) denotes the number of customers (busy servers) in the system at time t. We let ρ = λ/µ. more general. P01 = 1 = P10 . (For a general service time distribution we could not do so because then X(t) does not form a CTMC. In this case S = {0. Arguing along the same lines as when we introduced the balance equations.) 19 . We conclude (by dividing by t and letting t → ∞) that the long-run rate at which the function goes from j to j + 1 equals the long-run rate at which the function goes from j + 1 to j. Thus Pj = e−ρ ρj /j! and we end up with the Poisson distribution with mean ρ: The M/M/∞ queue is always positive recurrent for any λ > 0. every time this kind of function goes up from j to j + 1. its stationary distribution is Poisson with mean ρ = λ/µ.2 that the chain is positive recurrent for any λ > 0 and µ > 0. Of course. or P0 = e−ρ . so positive recurrence will follow if we can solve the above equations. λP0 = µP1 . Irreducibility follows from the fact that the embedded chain is an irreducible simple random walk. so we had to use other. There is only one balance equation. and obtained the same stationary distribution. As is easily seen by recursion. P1 = ρ/(1 + ρ). j ≥ 0. up by 1 or down by 1. t] that it makes an “up” transition from j to j + 1 diﬀers by at most one.

The B&D balance equations (26) are λPj = (j + 1)µPj+1 . for any λ > 0 and µ > 0. the proportion of arrivals who ﬁnd all c servers busy. Suppose that at each of the times from a Poisson process at rate γ. 0 ≤ j ≤ c − 1. Letting X(t) denote the population size at time t. c}. in particular the formula for Pc in (29). 0 ≤ j ≤ c. Later we will see from a result called PASTA. we no longer have a B&D process now since the arrival of a family can cause a jump larger than size one. an irreducible B&D process with a ﬁnite state space S = {0. 0 ≤ j ≤ c − 1. a result called Erlang’s Loss Formula. or Pj+1 = Pj ρ/(j + 1). M/M/c loss queue: This is the M/M/c model except there is no waiting room. The balance 20 . that is. i ≥ 1 denote corresponding family size probabilities. Solving we get Pj = ρj /j!P0 . Thus ρj ρn Pj = 1+ j! n! n=1 In particular c c . With X(t) denoting the number of busy servers at time t. and summing to one yields c 1 = P0 1 + j=1 ρj . that Pc is also the proportion of lost customers. death rates µi ). (29) the proportion of time that all servers are busy. the ﬁrst c equations for the FIFO M/M/∞ queue.5. Let bi = P (B = i). any arrival ﬁnding all c servers busy is lost. (28) ρn ρc 1+ Pc = c! n! n=1 −1 . j! yielding c P0 = 1 + j=1 ρj j! −1 −1 . . so positive recurrence follows from Theorem 1. independently. Population model with family immigration: Here we start with a general B&D process (birth rates λi . 6. but allow another source of population growth. .2. This is the c−server analog of Example 3. a family of random size B joins the population (immigrates). This turns out to be a very famous/celebrated queueing theory result because the solution in (28). . 0 ≤ j ≤ c. turns out to hold even if the service times are not exponential (the M/G/c-loss queue). in addition to the births. . we have.

In particular E(HT ) = E(K)/ai = 1/pai .20 0. n ≥ 1. besides a birth from state j − 1 or a death from state j + 1.j /p for j = i.25 0. Suppose that 0 < Pi. Letting Yn denote iid exponential rvs at rate ai (the holding time rate). as long as one is consistent (on both sides of the balance equations). . even if we keep Pi. j−1 (λj + µj + γ)Pj = λj Pj−1 + µPj+1 + i=0 γbj−i Pi .i = 0 and reset Pi. then the same balance equations arise in the end. Then P (K = n) = (1−p)n−1 p. j ≥ 1. 1. 1. Assume X0 = i and let K denote the total number of transitions (visits) to state i before making a transition out to another state.i < 0. but for which Pi. . In our study of CTMC’s we have inherently been assuming that Pi. it has the same distribution). reset Pi.) Thus.g. we count this initial visit as one such visit.14 Transitions back into the same state. 0. we can reset ai = pai . .i > 0. we can represent the total holding time HT in state i as an independent geometric sum K HT = n=1 Yn . where p = 1 − Pi. This yields the same CTMC {X(t)} (e. a birth (rate λj ) or a family immigration (rate γ). and embedded chain transition matrix P = 0.i = 0 for each i ∈ S. This yields the rate out of state j.i .i = 0. s ≥ 0. any one of three events can happen next: A death (rate µj ).. Since X0 = i.equations (“the rate out of state j equals the rate into state j”) are: (λ0 + γ)P0 = µ1 P1 (λ1 + µ1 + γ)P1 = (λ0 + γb1 )P0 + µ2 P2 (λ2 + µ2 + γ)P2 = γb2 P0 + (λ1 + γb1 )P1 + µ3 P3 .j = Pi. This yields the rate into state j. pai + s (Condition on K ﬁrst.75 . Pi. There are j additional ways to enter state j. We illustrate with a simple example: A CTMC with two states. In any case. The derivation is as follows: When X(t) = j.80 21 . but this is not necessary as we illustrate here. namely from each state i < j a family of size j − i could immigrate (rate γbj−i ). In fact HT ∼ exp(pai ) as is easily seen by deriving its Laplace transform: E(esHT ) = paj .i > 0.

80)a1 P1 . This of course would eﬀect quantities of interest such as average waiting time. If X(t) denotes the number in system at time t. ai is the holding time rate when in state i. Thus λπj = λPj . let πj denote the long-run proportion of arrivals who. and such transitions can only happen when arrivals ﬁnd j customers in the system. the chain will make a transition according to the transition matrix P = (Pij ). meaning that after the holding time Hi ∼ exp(ai ) is completed. a1 = (0. in queueing models. which is what we get if we were to instead interpret a transition j → j as neither a transition into or out of state j. allowing Pi. then the balance equations are a0 P0 = (0. λPj is also the same rate in question. j ≥ 0. 1 0 So. 3 22 . n=1 where X(tn −) denotes the number in system found by the nth arrival. λπj is the long-run rate (number of times per unit time) that X(t) makes a transition j → j + 1. which asserts that But there might be other associated stochastic processes that will become diﬀerent by making this change. these equations reduce to the one equation (0. For example. By resetting Pi. As the reader can check.20)a1 P1 .75)a0 . a On the one hand.20)a1 and P to P = 0 1 . upon arrival. On the other hand. then def a πj = lim 1 N →∞ N N I{X(tn −) = j}. it makes no diﬀerence as far as {X(t)} is concerned3 . from the a B&D balance equations (21).20)a1 P1 a1 P1 = (0.i > 0 might refer to allowing customers to return to the end of the queue for another round after completing service.i = 0.75)a0 P0 + (0. By deﬁnition. we are forcing the customer to re-enter service immediately for the extra round instead of waiting at the end of the queue. and tn denotes the time of the nth Poisson arrival. or a πj = Pj .75)a0 P0 = (0.15 Poisson Arrivals See Time Averages (PASTA) a For a stable M/M/1 queue. If we interpret a transition j → j as both a transition out of and into state j.a0 > 0 and a1 > 0 are given non-zero holding-time rates. This is how it works out for any CTMC. ﬁnd j customers already in the system. Resetting the parameters as explained above means resetting a0 = (0. After all. 1. arrivals occur at rate λ.25)a0 P0 + (0.

Both processes are assumed on the same probability space. or have a jump at t. and it turns out that PASTA holds for any queueing model in which arrivals are Poisson. def def It has left-hand limits if for each t > 0. is beyond the scope of this course. is right-continuous if for each t ≥ 0. This is an example of Poisson Arrivals See Time Averages (PASTA). (The proof. then w.3 (PASTA) If the Poisson process satisﬁes LAC. then the function is said to be discontinuous at t. To state PASTA more precisely.) Moreover. 23 . If x(t−) = x(t+). let {X(t) : t ≥ 0} be any stochastic process. for it requires that any future increment be independent not only of its own past but of the past of the queueing process as well.. as in the M/G/∞ queue.1. We assume that the sample-paths of X(t) are right-continuous with left-hand limits. then so does the other and they are equal. PASTA holds for more general quantities of interest besides number in system. {(N (u). The state space S can be general such as multi-dimensional Euclidean space. 4 The lack of anticipation condition (LAC) that we will need to place on the Poisson process asserts that for each ﬁxed t > 0. t ≥ 0. (Service times do not need to have an exponential distribution. the future increments of the Poisson process after time t. but we have in mind the case when the two processes are dependent via the arrivals being part of and participating in the queueing system. the proportion of Poisson arrivals to a queue who. 1 N →∞ N lim N f (X(tn −)) = lim n=1 1 t→∞ t t f (X(s))ds. In general. Let f (x) be any bounded real-valued function on S. {N (t + s) − N (t) : s ≥ 0}.) Theorem 1. 0 in the sense that if either limit exists. called LAC. x(t+) = limh↓0 X(t + h) = x(t). and ψ = {tn : n ≥ 0} a Poisson process. upon arrival.the proportion of Poisson arrivals who ﬁnd j customers in the system is equal to the proportion of time there are j customers in the system. and consider the real-valued process f (X(t)). no matter how complex. holds. be independent of the joint past. as long as a certain (easy to verify) condition. This condition is stronger than the independent increments property of the Poisson process. then LAC holds. PASTA will not hold if the arrival process is not Poisson. ommitted. Queueing processes typically have jumps at arrval times and departure times. If the Poisson process is completely independent of the queueing process. x(t−) = limh↓0 x(t − h) exists (but need not equal x(t)). 4 A function x(t). they can be general. ﬁnd a particular server busy serving a customer with a remaining service time exceeding x (time units) is equal to the proportion of time that this server is busy serving a customer with a remaining service time exceeding x.p. We have in mind that X(t) denote the state of some “queueing” process with which the Poisson arriving “customers” are interacting/participating. We are now ready to state PASTA. X(u)) : 0 ≤ u ≤ t}. For example.

m ≥ 1. and this one customer departs (rate µ2 )). Letting X(t) = (X1 (t).16 Multi-dimensional CTMC’s So far we have assumed that a CTMC is a one-dimensional process. the M/M/c-loss queue. Upon completion of the second service.m denote the long-run proportion of time there are n customers at the ﬁrst facility and m at the second (a joint probability). it is easily seen that {X(t)} satisﬁes the Markov property. where we ﬁrst mentioned PASTA in the context of Erlang’s Loss Formula. All of the CTMC theory we have developed in one-dimension applies here as well (except for the Birth and Death theory). we now have two service times rates (one for each server).m + µ1 Pn+1. 1. so that f (X(t)) = I{X(t) = j}.m+1 . exactly one customer is at the second facility. Besides the Poisson arrival rate λ. a customer completes service at the ﬁrst facility and thus goes to the second. 1) (no one is at the ﬁrst facility. This would. and independent of each other and of the arrival process. in which case this system is denoted by F IF O M/M/1/ −→ /M/1. immediately waits in line at a second single-server facility.i. Balance equations (rate out of a state equals rate into the state) can be set up and used to solve for stationary probabilities. or a customer completes service at the second facility 24 . the customer ﬁnally departs. where Xi (t) denotes the number of customers in the ith facility. 0) is from (0. Service times at each server are assumed i. f (x) = I{x = j}. µ1 and µ2 . after waiting in line and completing service at the ﬁrst single-server facility. and the second server has exponential service times and also serves under FIFO. (λ + µ1 + µ2 )Pn. 1.m = λPn−1.13.0 = µ2 P0. but that is not necessary.A standard example when X(t) is the number of customers in a queue. in what follows we assume that the ﬁrst facility is a FIFO M/M/1. because either a customer arrives.m−1 + µ2 Pn. for example. Tandem queue: Consider a queueing model with two servers in tandem: Each customer.d.1 . i = 1. X2 (t)). Similarly when n ≥ 1. yield πj = Pj for the M/M/1 queue. higher dimensions being analogous. Letting Pn. because the only way the chain can make a transion into state (0. 2. This is an example of an irreducible two-dimensional CTMC. The reader should now go back to Example 5 in Section 1. We illustrate with some two-dimensional examples. λP0. would be to let f denote an indicator function. and a f (X(tn −)) = I{X(tn −) = j}.

The ﬁrst facilty is an M/M/1 queue so we know that X1 (t) by itself is a CTMC with stationary distribution Pn = (1 − ρ1 )ρn . and so the second facility (in isolation) can be treated itself as an M/M/ 1 queue when {X1 (t)} is stationary.. Q = (Qij ) is called the routing transition matrix. 2. The remaining balance equations are also easily derived. Whenever a customer completes service at node i = 1. independent of the past. each with a geometric distribution. i = 1. and the chain is positive recurrent. 2.2 = 1 and Q2. We always assume that states 1 and 2 are transient. but more analysis is required to prove 2 the independence part. they next go to the queue at node j = 1.0 = 1 and λ2 = 0.s. Since they do work. i = 1. or depart the system with probability Qi.and leaves the system. it turns out that the solution is Pn. that is. n ≥ 0.0 . In general. and state 0 is absorbing. For simplicity we refer to each facility as a “node”. This means that as t → ∞. X2 (t)). m ≥ 0. the two facilities are certainly dependent at any time t. a customer gets served a couple of times. The tandem queue does not have feedback. where the state 0 refers to departing the system. This at least explains why X2 (t) has the geometric stationary distribution. i = 1. an example of a feedforward network. If we start oﬀ X1 (0) with this 1 stationary distribution (P (X1 (0) = n) = Pn . we allow feedback. So typically. Each node has its own queue with its own independent Poisson arrival process at rates λ1 and λ2 respectively. X1 (t) and X2 (t) become independent r.g. the departure process is itself a Poisson process at rate λ. {X1 (t)} is stationary. n ≥ 0). m ≥ 0.0 = 1. (1 − ρ2 )ρm . which means that a customer can return to a given node (perhaps many times) before departing the system. 2. each with exponential service at rates µ1 and µ2 respectively.m = (1 − ρ1 )ρn × (1 − ρ2 )ρm . {X(t)} yields an irreducible 25 . and we require that Q0. where Xi (t) denotes the number of customers in the ith node. This result is quite surprising because.v. an absorbing state.2: Plug in the given solution (e. It turns out that when stationary. It turns out that there is a nice way of understanding part of this result. Letting ρi = λ/µi . 2. Letting X(t) = (X1 (t). back and forth between the two nodes before ﬁnally departing. after all. because it represents the transition matrix of a Markov chain. they are the unique probability solution. and why should the second facility have a stationary distribution as if it were itself an M/M/1 queue? (For example. 2 with probability Qij . Jackson network: Consider two FIFO single-server facilities (indexed by 1 and 2). then we know that X1 (t) will have this same distribution for all t ≥ 0. 2 1 provided that ρi < 1. 2. it is the special case when Q1. n ≥ 0. treat it as a “guess”) into the balance equations and verify that they work. why should departures from the ﬁrst facility be treated as a Poisson process at rate λ?) The proof is merely a “plug in and check” proof using Theorem 1. In general.

the chain will visit each state i = 1. i. 2}. i = 1. 2. So ρi < 1. j ∈ {1. independently.m = (1 − ρ1 )ρn × (1 − ρ2 )ρm . µi where E(Ni ) is the expected number of times that a customer attends the ith facility. i. where si. 26 . The αi can be computed equivalently as the solution to the ﬂow equations: αi = λi + j αj Qj. and since 0 is absorbing (and states 1 and 2 transient). Notice that αi = λi E(Ni ) represents the total arrival rate to the ith node. the ﬂow equations in matrix form are α = λ + αQT . As with the tandem queue. it turns out that the stationary distribution for the Jackson network is of the product form Pn. denote the 2 × 2 matrix without the absorbing state 0 included. 2.j ). i = 1.CTMC. the proof can be carried out by the “plug in and check” method. Letting QT = (Qj. m ≥ 0.i ). 2 only a ﬁnite number of times before getting absorbed. 2}. with solution α = λ(I − QT )−1 . n ≥ 0. just means that the total arrival rate must be smaller than the service rate at each node.j denotes the expected number of times the discrete-time chain visits transient state j given it started in transient state i. We recognize that (I − QT )−1 = S = (si. i = 1. j ∈ {1. 2 1 provided that ρi < 1.i . is routed according to the discrete-time Markov chain with transition matrix Q. E(Ni ) is completely determined by the routing matrix Q: Each customer. Here ρi = λi E(Ni ). 2. Like the tandem queue.