You are on page 1of 4

# Stochastic Processes, Markov Chains, and Markov Models

Stochastic Process

Stochastic Processes, Markov Chains, and Markov Models

Stochastic Process Markov Chain

Stochastic Process

Stochastic Processes, Markov Chains, and Markov Models
L645 Dept. of Linguistics, Indiana University Fall 2009

Markov Models

A stochastic or random process is a sequence ξ1 , ξ2 , . . . of random variables based on the same sample space Ω. The possible outcomes of the random variables are called the set of possible states of the process.

Markov Chain Markov Models

The process will be said to be in state ξt at time t.

Note that the random variables are in general not independent.

In fact, the interesting thing about stochastic process is the dependence between the random variables.

1 / 22

2 / 22

Stochastic Process: Example 1

Stochastic Processes, Markov Chains, and Markov Models

Stochastic Process: Example 2

Stochastic Processes, Markov Chains, and Markov Models

Stochastic Process Markov Chain Markov Models

Stochastic Process Markov Chain

The sequence of outcomes when repeatedly casting the die is a stochastic process with discrete random variables and a discrete time parameter.

There are three telephone lines, and at any given moment 0, 1, 2 or 3 of them can be busy. Once every minute we will observe how many of them are busy. This will be a random variable with Ωξ = {0, 1, 2, 3}. Let ξ1 be the number of busy lines at the ﬁrst observation time, ξ2 the number of busy lines at the second observation time, etc. The sequence of the number of the busy lines then forms a random process with discrete random variables and a discrete time parameter.

Markov Models

3 / 22

4 / 22

Characterizing a random process

Stochastic Processes, Markov Chains, and Markov Models

Markov Chains and the Markov Property (1)

Stochastic Processes, Markov Chains, and Markov Models

Stochastic Process

Stochastic Process

To fully characterize a random process with time parameter t, we specify: 1. the probability P(ξ1 = xj ) of each outcome xj for the ﬁrst observation, i.e., the initial state ξ1 2. for each subsequent observation/state ξt+1 : t = 1, 2, ... the conditional probabilities P(ξt+1 = xit+1 |ξ1 = xi , ..., ξt = xit ) Terminate after some ﬁnite number of steps T

Markov Chain Markov Models

A Markov chain is a special type of stochastic process where the probability of the next state conditional on the entire sequence of previous states up to the current state is in fact only dependent on the current state. This is called the Markov property and can be stated as:

Markov Chain Markov Models

P(ξt+1 = xit+1 |ξ1 = xi1 , . . . , ξt = xit ) =

P(ξt+1 = xit+1 |ξt = xit )

5 / 22

6 / 22

and Markov Models Stochastic Process Markov Chain Markov Models Stochastic Process Markov Chain In the telephone example. pij ≥ 0. (Note that we have numbered the rows and columns 0 through 3.2 0. Markov Chains.2 0. a matrix with these properties is called a stochastic matrix. . a possible transition matrix could for example be: s0 0. . . Markov Chains.1 0. Markov Chains. . i. . . . and Markov Models Transition Matrix: Example Stochastic Processes.2  0.1 0. . .1. A ﬁnite Markov chain must at each time be in one of a ﬁnite number of states.. n j=1 s0 s1 s2 s3 In general.5 0. Markov Chains. .1  s1 0. 7 / 22 8 / 22 Transition Matrix for a Markov Process Consider a Markov chain with n states s1 . p1n P =  . pn1 . .3 s3 0. . and Markov Models Stochastic Process Markov Chain Stochastic Process (2) pij Assume that currently. ξ2 . More generally. We can turn the telephone example with 0. Column 1 (p31 ) is 0. What is then the probability that at the next point in time exactly one line is busy? The element in Row 3.2 0.Markov Chains and the Markov Property (2) Stochastic Processes. . ξt = xit ) = = P(ξ1 = xi1 ) · P(ξ2 = xi2 |ξ1 = xi1 ) · . and not on the previous history.4 0. .1 s2 0.) Markov Models = P(ξt+2 = sj |ξt = si ) n Markov Chain Markov Models = r =1 n P(ξt+1 = sr |ξt = si ) · P(ξt+2 = sj |ξt+1 = sr ) pir · prj = r =1 The element pij can be determined by matrix multiplication as the value in row i and column j of the transition matrix P 2 = PP. .4 0. .  .3   0. sn . pnn  n Stochastic Processes. .e. 2 or 3 busy lines into a (ﬁnite) Markov chain by assuming that the number of busy lines will depend only on the number of lines that were busy the last time we observed them. i = 1. . P(ξt+1 = sj |ξt = si ).1. . The transition matrix for this Markov process is then deﬁned as  p11 . .1 0. and Markov Models Transition Matrix after Two Steps Stochastic Processes. . 9 / 22 10 / 22 Transition Matrix: Example (2) Stochastic Processes. .5     Markov Models P= pij = 1. . Markov Chains. 11 / 22 12 / 22 (2) . the transition matrix for t steps is P t . can be calculated as: Stochastic Process Markov Chain Markov Models Stochastic Process Markov Chain Markov Models P(ξ1 = xi1 . and thus p(1|3) = 0. all three lines are busy. 1. . . · P(ξt = xit |ξt−1 = xit−1 ) The conditional probabilities P(ξt+1 = xit+1 |ξt = xit ) are called the transition probabilities of the Markov chain. and Markov Models The probability of a Markov chain ξ1 . . Markov Chains. . Let pij denote the transition probability from state si to state sj . and Markov Models Markov Chains: Example Stochastic Processes. .3 0. .

24 Stochastic Process Markov Chain Markov Models Let’s go to http://people. . Markov Chains. .2 0. and Markov Models Stochastic Process Markov Chain Markov Models For the telephone example: Assuming that currently all three lines are busy.5 0.12 ] Markov Chain Markov Models Markov models add the following to a Markov chain: ◮ ◮ a sequence of random variables ηt for t = 1. and Markov Models A vector v = [v1 . ..31 0. and Markov Models Stochastic Process Stochastic Process Markov Chain Markov Models Let v = [0.. P 2 = PP = s0 s1 s2 s3     ⇒ p31 = 0.15 0. T representing the signal emitted at time t ◮ We will use σj to refer to a particular signal ⇒ p(ξ2 = s2 ) = 0.html and practice. Markov Chains. then [p t (s1 ). is called an initial probability vector.22 s2 0.21   0.e.edu/Stefan Waner/realWorld/tutorialsf1/frames3 2. .22 (2) 13 / 22 14 / 22 Initial Probability Vector Stochastic Processes. n i=1 vi Stochastic Process Stochastic Process Markov Chain Markov Models = 1 is Markov Chain Markov Models The probability vector that determines the state probabilities of the observations of the ﬁrst element (state) in a Markov chain.12 s1 0. i.15 0.38 0.24 17 / 22 18 / 22 . .. . What is then the probability that after two steps exactly two lines are busy? vP 2 = vPP = [ s0 0. Markov Chains. . Markov Chains. vn ] with vi ≥ 0 and called a probability vector.22  0. .24 s3 0. and Markov Models Initial Probability Vector Stochastic Processes.25 0.20 0.hofstra.30 0. what is the probability of exactly one line being busy after two steps in time?  s0 0. i. p t (sn )] = vP t−1 15 / 22 16 / 22 Initial Probability Vector: Example Stochastic Processes.0] be the initial probability vector for the telephone example of a Markov chain.37 0..21 s1 0. The initial probability vector and the transition matrix together determine the probability of the chain in a particular state at a particular point in time. and Markov Models Matrix after Two Steps: Example Stochastic Processes.Matrix Multiplication Stochastic Processes.17 0. Markov Chains. If p t (si ) is the probability of a Markov chain being in state si at time t.25 0. Markov Chains. where vi = P(ξ1 = si ).28 s3 0.3 0.. and Markov Models Markov Models Stochastic Processes.e. after t − 1 steps.43 s2 0. .

and Markov Models Markov Models (3) Stochastic Processes. .7 ColaPref. with emission probabilities: ◮ CP IP cola ice-t lem 0. after t − 1 steps. .Markov Models (1) A Markov Model consists of: ◮ ◮ ◮ ◮ Stochastic Processes. p.3 Iced Tea Pref.e.6 0.1 0. . and Markov Models Stochastic Process Markov Chain Markov Models Stochastic Process a ﬁnite set of states Ω = {s1 . Markov Chains. . and an initial vector v = [v1 . Markov Chain Markov Models 0. . p (t) (σm )] = vPt−1 A 21 / 22 22 / 22 . and Markov Models Stochastic Process Markov Chain Markov Models The probability that signal σj will be emitted at time t is then: n n Stochastic Process Markov Chain Markov Models p (si . .5 an n × m signal matrix A = [aij ].5 0.2 from: Manning/Schutze. . vn ] where vi = P(ξ1 = si ). σm }. an n × n state transition matrix P = [pij ] where pij = P(ξt+1 = sj | ξt = si ). . Markov Chains. then [p (t) (σ1 ). i. Markov Chains. . . . . Markov Chains. .7 0. an signal alphabet Σ = {σ1 .. σj ) = p (si ) · p(ηt = σj | ξt = si ) where ◮ (t) (t) p (t) (si ) is the ith element of the vector vPt−1 p (t) (σj ) = i=1 p (t) (si . . and Markov Models HMM Example 1: Crazy Softdrink Machine Stochastic Processes. 321 ¨ 19 / 22 20 / 22 Markov Models (2) Stochastic Processes. . σj ) = i=1 This means that the probability of emitting a particular signal depends only on the current state ◮ p (t) (si ) · p(ηt = σj | ξt = si ) . . 0.1 0. sn }.3 0. which for each state-signal pair determines the probability aij = p(ηt = σj | ξt = si ) that signal σj will be emitted given that the current state is si . 0. and not on the previous states Thus if p (t) (σj ) is the probability of the model emitting signal σj at time t. . .