Practical: Markov chains and Poisson processes

The first few questions consider the following maze:

2 3

1 4 5

As you can see, the maze contains a mouse, which is free to wander around as it
chooses. Whenever the mouse enters a room, it stays for one time unit, then chooses
to leave by going through one of the doors out, picking among available doors at
random. We will consider the sequence of rooms that the mouse passes through
(numbered as above).

PART I: Exploring mouse moves

To explore the movement of the mouse around the maze, we can use MATLAB to
simulate the path of the mouse.

The function “mousemove” takes a single argument, the current position of the
mouse, and returns a randomly chosen new position. The function is stored in the
usual WebLearn resource directory in the file “mousemove.m” which you can
download into MATLAB.

Questions:

1. Is the sequence of rooms the mouse passes through a Markov chain?

2. Use “mousemove” to simulate the path of a mouse, starting in room 3 as shown, for
10 moves. Repeat a few times to get a feel for the movement. Try different starting
points for the mouse.
3. Simulate just 2 moves of the mouse, 100 different times, starting each time in room
three. In what proportion of the 100 simulations does the mouse finish in room 4?

4. If the mouse starts in room 3, what are the possible routes the mouse could take to
be in room four after exactly two steps?
a. What are the probabilities of taking each path?
b. Sum over these possibilities, to obtain the exact probability of moving from
room 3 to room 4 in two steps.
c. Compare this solution with the result obtained by simulation in question 3.

5. If the mouse now starts in room 2, simulate 100 different random paths for the
mouse, for 10 moves in total. Store each of the final elements, to produce a vector of
length 100 giving the 100 different positions for the mouse after the different
realisations of 10 mouse moves.
a. Calculate the proportion of times the mouse finishes in room 1.
b. Repeat part (a) for the probability of finishing in one of the other rooms.
c. Could you calculate this probability analytically using the method of question
4?

6. Write down the transition matrix for the Markov chain defined by movement of the
mouse from room to room. (Make sure you check this against the transition matrix
you will be given in the following lecture)

END OF PART I

PART II: Using the transition matrix

The transition matrix provides a powerful tool to analytically calculate probabilities in

a Markov chain.

Questions:

7. Input the transition matrix into MATLAB, and store this as P. Make sure this agrees
with the matrix from the lectures, as it will be important for the other questions.

8. The probability of question 3 can be calculated using the matrix P (and letting the
matrix algebra “do the work” of summing the possibilities). The matrix P2 gives the
two-step transition probabilities.
a. Calculate P2 using MATLAB.
b. Which element of P2 gives the probability of moving from room 3 to room
4 in two steps? Verify that consulting this element gives the same answer
as the direct addition performed for question 4.

9. Use MATLAB to calculate the transition probability matrix for 10 moves of the
mouse.
a. If the mouse starts in room 2, look up the appropriate row of this matrix to
give the probabilities of finishing in each of the five possible rooms.
 b. Use this row to obtain the analytic answers for the probabilities you calculated
via simulation in question 5.
c. What would the answers be, if instead the mouse had begun in room 1?

10. Suppose now the mouse starts in room 3.

a. Use MATLAB to calculate powers of P and thus the probability of the mouse
being in room 3 again after 3, 5, 9 moves.
b. What possible routes can the mouse take to return to room 3 after 5 moves?
c. Verify the result for 5 moves by direct calculation, by summing over the
probabilities for these route(s).
d. What would happen if you tried to think about 9 moves this way?

11. Suppose now that into the maze are brought two things; a mouse of the opposite
sex, dropped in room 1, and a piece of cheese, in room 2. The mouse of the opposite
sex finds our mouse very attractive, but nevertheless decides that the best strategy to
win its heart is to play it cool and wait in room 1 until the original mouse finds his/her
way to them. The original mouse is keen for romance, but also hungry, and simply
randomly roams the maze as before until they find either the cheese or the ideal mate.
a. Assuming the original mouse starts in room 3, use the function “mousemove”
to simulate a path through the maze for the mouse, until it reaches either the
cheese, or the other mouse. Which happens first?
b. Repeat (a) 100 times using a loop, and for each simulation record whether the
cheese, or the mate, is located first. What is the proportion of times the mouse
eats instead of meets?

END OF PART II

PART III: Long term moves

The long term behaviour of Markov chains has some interesting properties. Next, we
explore these properties for our particular mouse maze.

Questions:

12. We can use MATLAB to explore the long term movement of the mouse.
a. Suppose the mouse starts in room 1, and use MATLAB to calculate the
probability of finishing in room 5 after 10, 50, and 500 moves.
b. Repeat (a), to obtain the probability of finishing in room 2 after 10, 50, 500
moves starting from room 1.
c. What seems to happen to the probability of finishing in a given room as the
number of moves gets large?
d. Now pick a different room for the mouse to start in, and repeat parts (a) and
(b). Compare your answers to those for the first two parts.
e. What does P500 tell us about the probability of a given final room, for different
initial rooms of the mouse? Use this matrix to write down long term
probabilities for the mouse being found in each of the five rooms.

13. Suppose that the mouse starts in room 3.

 a. Use “mousemove” to simulate 1000 moves of the mouse, counting the number
of times the mouse visits room 1. Divide this number by 1000 to get the
proportion of times the mouse is in room 1.
b. Repeat part (a) five times, writing down the proportion in each case. Does the
proportion vary much between runs?
c. Still starting the mouse from room 3, choose a different room and repeat parts
(a) and (b), counting the proportion of occasions the mouse is in this second
room.
d. Compare the long term proportions you obtain to the long term probabilities
you found in part e of question 12, and suggest long term proportions for the
other three rooms. Why might you expect this result intuitively?

END OF PART III

PART IV: Poisson processes

The final practical is based around the Poisson process defined in the last lecture.
Consider the situation at a hypothetical vet’s surgery. The times at which sick or
injured animals are brought in might be modelled as a Poisson process. Suppose that
the surgery is open 9 hours a day for 5 days a week (45 hours per week), and that the
rate at which animals arrive is on average 2 per hour. Throughout, let’s measure time
(t) in units of hours.

Questions:

14. Under the Poisson process model, the times between animal arrivals are
independent and exponentially distributed with mean ½ (rate 2 per hour).

(Throughout this question it may be useful to note that if U is a uniform(0,1) random

variable, then the transformed variable Y = -m * ln(U) has the exponential distribution
with mean m. )

a. Use MATLAB to simulate the arrival time (in hours) of the first sick animal,
by simulating a single exponential time.
b. Now simulate the arrival time of a second animal, by simulating a second
exponential waiting time, and adding this to the time of arrival of the first
animal.
c. Use MATLAB to simulate a collection of arrival times of animals over the
course of one complete week (45 hours), by successively simulating
exponential gaps (starting at time zero) while the total combined gap time is
below 45 hours; then the jth arrival time is the sum of the first j gaps. How
many animals arrive in the course of the week?
d. Repeat part (c) 100 times, and for each of these simulations record the number
of animals arriving during the week. Store the 100 numbers as a vector
“arrivals”
e. What distribution does theory say the number of arrivals should have?
i What is the appropriate parameter (the mean) of this distribution here?
ii Draw a QQ-plot for the vector “arrivals” against the correct distribution,
to check that this is indeed what you see.
15. Suppose now that one in every three pets that arrives is a rodent. Simulate a
collection of arrival times of rodents, by simulating a full set of animal arrivals
(ie repeat question 14 part c, this time storing all the arrival times for each animal) and
then thinning to just the times of rodent arrivals by selecting each arrival as a rodent
with probability 1/3.
a. Plot a histogram of the time intervals between rodent arrivals (including the
time to the first arrival).
i. Suggest a continuous distribution that this histogram resembles.
ii. What is the mean time between arrivals?
b. Repeat the simulation of rodent arrivals 100 times, and for each simulation
store just the number of rodents, producing a vector “rodents”.
i. Plot a histogram of “rodents”.
ii. Can you suggest a discrete distribution that this histogram resembles?
iii. What is the mean number of rodents that arrive in a week?
c. In view of (a) and (b), what kind of process do you think the arrival of rodents
process is? What do you think is the appropriate rate for this arrivals process?

16. Simulate a full set of animal arrivals, storing all the arrival times.
a. Use this vector of arrival times to calculate the length of the gap between
animal arrivals that the time point 15 lies in. That is, what is the time interval
between the last animal arrival before t=15, and the first animal arrival after
t=15?
b. Repeat part (a) 100 times, and for each simulation, store the length of the gap
between animal arrivals that the time point 15 lies in, forming a vector “gaps”.
c. Calculate the average of “gaps”. How does this compare to the mean time
between arrivals of ½ hour? Can you explain why this is?