Click on the title of the lesson to go to the recording.

• 1st lesson Sept 14, 2021 (Markov Chains, Introduction),

Definition, transition (stochastic) matrices, joint probability
distributions. Examples: random walks, queues.
• 2nd lesson Sept 15, 2021 (Classes of states, transient and recur-
rent states, Gambler’s ruin problem),
Accessibility, classes of states, period. Irreducible Markov chains.
Transient and recurrent states. Renewal equation.
• 3rd lesson Sept 16, 2021 (Recurrence critieria and applications.),
Recurrence condition based on divergence of a series. States of
the same class are all recurrent or transient.
• 4th lesson Sept 21, 2021 (Random walks on Zd.),
The d = 1 random walk is recurrent for p = q = 1/2 (i.e.
when it is symmetric), transient otherwise. Symmetric random
walks are recurrent for d = 1, 2, non-symmetric are transient.
Random walks on Zd with d ≥ 3 are transient.
• 5th lesson Sept 23, 2021 (Invariant densities)
Existence for finite Markov chains, examples with infinite Mar-
kov chains
• 6th lesson Sept 28, 2021 (Invariant densities, Birth-and-death
chains),
Birth-and-death chains, reversibility, asymptotic behaviour of
(n)
pii for a recurrent aperiodic i
• 7th lesson Sept 29, 2021 (Ergodic Theorem),
Ergodic theorem for fast recurrent aperiodic Markov chain. Re-
1
 versible Markov chains. Remarks on invariant densities.

• 8th lesson Oct 5, 2021 (Strong Markov Property I),

Natural filtration of a Markov chain. Stopping times. Chain
restarting from a stopping time T .

• 9th lesson Oct 6, 2021 (Strong Markov property II),

Continuation. Infinitely many returns in a recurrent state.
Probability of wandering forever in transient states.
• 10th lesson Oct 12, 2021 (Absorption probabilities, mean ab-
sorption times),

• 11th lesson Oct 13, 2021 (Mean absorption times. Criteria for
transience or recurrence),
Coupon collector and gambler’s ruin. Criteria for transience
or recurrence of infinite Markov chains. Application to the
discrete time one counter queueing model.
• 12th lesson Oct 26, 2021 (Criteria for recurrence and existence
of invariant densities. Branching models),
Criteria for recurrence and existence of invariant densities. Ap-
plication to the discrete time one counter queueing model.
Branching models, computation of the extinction probability.
• 13th lesson Oct 27, 2021 (Sojourn times in transient states.
Continuous time Markov chains.),
Sojourn times in subsets of the set transient states; moment ge-

2
 nerating function. Continuous time Markov chains, definition,
transition semigroup, Chapman-Kolmogorov equations.
• 14th lesson November 2, 2021 (Transition matrices and transi-
tion rates.),
Continuous time Markov chains. Transition matrices and tran-
sition rates. Forward and backward Kolmogorov equations.
• 15th lesson November 3, 2021 (Transition rates. Discrete skele-
ton),
Exponential distribution of residual visiting times, Pi{ XTi =
j } = qij /(−qii) (j 6= i). Discrete skeleton
• 16th lesson November 11, 2021 (Hitting probabilities. Poisson
process.),
Hitting probabilities: from discrete to continuous time. Poisson
process.
• 17th lesson November 16, 2021 (Recurrence, transience. Birth-
and-death processes.),
Continuous time Markov chains: recurrence, transience, inva-
riant densities. Birth-and-death processes.
• 18th lesson November 17, 2021 (Continuous time Markov chains:
invariant densities),
Invariant densities, M/M/k queue model.
• 19th lesson Nov 23, 2021 (Queueing models and epidemic mo-
dels. ),
M/M/k and SIS models.

3
• 20th lesson Nov 24, 2021 (Conditional expectaction),
Conditional expectaction of an integrable random variable with
respect to an arbitrary σ-algebra: introduction.
• 21st lesson Nov 30, 2021 (Martingales),
Conditional expectation of square integrable random variables.
Martingales, examples.
• 22nd lesson Dec 01, 2021 (Stopping theorem),
Stopping theorem with applications
• 23rd lesson Dec 15, 2021 (Martingales of a Markov Chain),
Proof of the topping theorem. Martingales of a Markov Chain
with applications,
• 24th lesson Dec 21, 2021 (Martingale convergence theorem, sub-
martingales and supermartingales),
Martingale convergence theorem, submartingales and super-
martingales, examples