Stochastic dynamical models, Mathematical Engineering 2019/2020 - SDM 8

Exercise 1.
1) Let (Nt )t≥0 be a Poisson process with parameter λ. Prove that the process Mt = Nt − λt is a
martingale with respect to the natural filtration of (Nt )t≥0 .
2) Let B = (Bt )t≥0 be a Brownian motion. Prove that Xt = Bt2 − t is a martingale with respect to
the natural filtration of B.
Exercise 2. Let I = N and (Xn )n≥0 be a Markov chain with state space E and let P = (pjk )j,k∈E its
transition matrix. Suppose that λ ∈ (0, 1] is an eigenvalue of P with eigenvector a bounded function
f :→ R, i.e. X
(P f )(j) = pjk f (k) = λf (j), ∀j ∈ E
k∈E

Prove that the process

Mn = λ−n f (Xn )
is a martingale with respect to the natural filtration (Fn )n≥0 of the Markov chain (Xn )n≥0 (Fn =
σ{Xj |j ≤ n}.
Exercise 3. A car washing system does the job in two stages. When a car enters in the system, it first
goes through stage 1, then stage 2, and finally it goes away. Service times in both stages are independent
(and independent of the car) with exponential distribution with parameters a and b respectively. Potential
customers arrive according to a Poisson process with parameter λ (independent of all other exponential
random variables) but new cars enter in the system only if there are no other cars (at stage 1 and at
stage 2).
1) Write the transition rate matrix Q of a three-state 0,1,2 continuous time Markov chain model.

2) Find the frequency of visit in each state in the stationary regime.

Suppose now that λ = 1/24 (i.e. the interarrival time of two cars is an exponential random variable with
average 24 minutes , to fix the ideas), a = 1/4, b = 1/2 and denote by (Xt )t≥0 the three-state Markov
chain.

3) Let f : {0, 1, 2} → R be the function f (0) = 24, f (1) = 0, f (2) = 16. Show that the stochastic
process (Mt )t≥0 defined by
Z t
Mt = f (Xt ) − (4 1{Xs >0} − 1{Xs =0} )ds
0

is a martingale with respect to the natural filtration of (Xt )t≥0 .

4) Find the average service time (stage 1 + stage 2) applying the stopping theorem.
5) Find the matrix Pt of the transition probabilities at time t of the Markov chain model
3 5 3 5 3 5
4
+ e− 8 t − 45 e− 12 t 2
− 13 e− 8 t + 15 e− 12 t 1
− 23 e− 8 t + 35 e− 12 t
 
5 3 5
15 3 5
15 3 5
Pt =  4
5 − 8e− 8 t + 365 e
− 12 t 2
15 + 83 e− 8 t − 95 e− 12 t 1
15 + 163 e
−8t
− 275 e
− 12 t 

? ? ?

Exercise 4. A family has three children. When one of them catches a cold he remains infectious, that
is he can transmit the disease to another child, for an exponentially distributed random time with mean
6 (days). Luckily, his illness heals in an exponentially distributed random time with mean 3 (days).
1) Construct a Markov chain representing the number Nt of infected children at time t.
2) Classify states and determine invariant probability distributions.

3) Let f : {0, 1, 2, 3} → R be the function f (0) = 0, f (1) = 4, f (2) = 6, f (3) = 7. Show that the
stochastic process (Mt )t≥0 defined by
Z t
Mt = f (Nt ) + 1{Ns >0} ds
0

is a martingale.
4) Applying the martingale stopping theorem, compute the mean time for the extinction of the epi-
demic starting from N0 = 1.
5) Compute the probability that, starting from N0 = 1, the ill child recovers before infecting other
children.