4 views

Uploaded by Harsh Arya

dtnc

- pertemuan 7
- Propability
- ch2
- Simio asd Simulation PREVIEW 3e
- Robust Control and Model Misspecification
- Levy 1215 Bfs Theme Carr
- RandomWalk
- polya note pdf
- stochastic processess
- How to Gamble if You Must
- Prob&StatsBook.pdf
- Rr210403 Probability Theory Stochastic Process 2005 Supply
- Notes for math for computer science
- Review
- FEbook1MAIN
- احصاء شبتر 2
- IPC2012-90139
- stochastic process
- Simulation
- 2011+ +STA2020F+Test+2+Solutions

You are on page 1of 185

and classification

Bo Friis Nielsen1

1 Applied

Bo Friis Nielsen

Today:

I

I

I

I

Markov chain introduction (Markov property)

Chapmann-Kolmogorov equations

First step analysis

Next week

I

I

I

I

Random walks

First step analysis revisited

Branching processes

Generating functions

I

I

I

Classification of states

Classification of chains

Discrete time Markov chains - invariant probability

distribution

Bo Friis Nielsen

Bo Friis Nielsen

Sample space

Bo Friis Nielsen

Sample space

Bo Friis Nielsen

Sample space

Outcome

Bo Friis Nielsen

Sample space

Outcome

Bo Friis Nielsen

Sample space

Outcome

Event

Bo Friis Nielsen

Sample space

Outcome

Event

A, B

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac =

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Union

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

Union

A, B

Ac = \A

AB

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

Union

A, B

Ac = \A

AB

A or B

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

Union

A, B

Ac = \A

AB

A or B

Intersection

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Union

AB

A or B

Intersection

AB

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Union

AB

Intersection

AB

A or B

Outcome is in

both A and B

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Union

AB

Intersection

AB

A or B

Outcome is in

both A and B

Bo Friis Nielsen

Sample space

Outcome

Event

Complementary event

A, B

Ac = \A

Union

AB

Intersection

AB

A or B

Outcome is in

both A and B

Bo Friis Nielsen

0 P(A) 1,

Bo Friis Nielsen

P() = 1

0 P(A) 1,

P() = 1

Bo Friis Nielsen

for A B =

0 P(A) 1,

P() = 1

for A B =

Leading to

Bo Friis Nielsen

0 P(A) 1,

P() = 1

for A B =

Leading to

P() = 0,

P(Ac ) = 1 P(A)

Bo Friis Nielsen

0 P(A) 1,

P() = 1

for A B =

Leading to

P() = 0,

P(Ac ) = 1 P(A)

(inclusion- exlusion)

Bo Friis Nielsen

P(A|B) =

P(A B)

P(B)

Bo Friis Nielsen

P(A|B) =

P(A B)

P(A B) = P(A|B)P(B)

P(B)

Bo Friis Nielsen

P(A|B) =

P(A B)

P(A B) = P(A|B)P(B)

P(B)

(multiplication rule)

i Bi =

Bo Friis Nielsen

P(A|B) =

P(A B)

P(A B) = P(A|B)P(B)

P(B)

(multiplication rule)

i Bi =

Bi Bj =

Bo Friis Nielsen

i 6= j

P(A|B) =

P(A B)

P(A B) = P(A|B)P(B)

P(B)

(multiplication rule)

i Bi =

P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

Independence:

P(A|B) = P(A|B c ) = P(A)

Bo Friis Nielsen

P(A|B) =

P(A B)

P(A B) = P(A|B)P(B)

P(B)

(multiplication rule)

i Bi =

P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

Independence:

P(A|B) = P(A|B c ) = P(A) P(A B) = P(A)P(B)

Bo Friis Nielsen

Mapping from sample space to metric space

(Read: Real space)

Probability mass function

f (x) = P(X = x) = P ({|X () = x})

Distribution function

F (x) = P(X x) = P ({|X () x}) =

f (t)

tx

Expectation

X

X

X

xP(X = x), E(g(X )) =

g(x)P(X = x) =

g(x)f (x)

E(X ) =

x

Bo Friis Nielsen

Joint distribution

f (x1 , x2 ) = P(X1 = x1 , X2 = x2 ),

F (x1 , x2 ) = P(X1 x1 , X2 x2 )

X

X

fX1 (x1 ) = P(X1 = x1 ) =

P(X1 = x1 , X2 = x2 ) =

f (x1 , x2 )

x2

FX1 (x1 ) =

x2

P(X1 = t1 , X2 = x2 ) = F (x1 , )

tx1 ,x2

Bo Friis Nielsen

P(X0 = x0 , X1 = x1 ) = P(X0 = x0 )P(X1 = x1 |X0 = x0 ) = P(X0 = x0 )Px0 ,x1

Suppose now some stationarity in addition that X2 conditioned

on X1 is independent on X0

P(X0 = x0 , X1 = x1 , X2 = x2 ) =

P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X0 = x0 , X1 = x1 ) =

P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X1 = x1 ) =

px0 Px0 ,x1 Px1 ,x2

which generalizes to arbitrary n.

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

= P (Xn = xn |Xn1 = xn1 )

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Markov property

P (Xn = xn |H)

= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

I

Generally the next state depends on the current state and

the time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Bo Friis Nielsen

Storage/inventory models

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I

I

The number of days since last rainfall

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I

I

I

The number of days since last rainfall

The number of passengers booked for a flight

Bo Friis Nielsen

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I

I

I

I

The number of days since last rainfall

The number of passengers booked for a flight

See textbook for further examples

Bo Friis Nielsen

reflecting barriers 0 and N

Bo Friis Nielsen

reflecting barriers 0 and N

P=

Bo Friis Nielsen

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

...

reflecting barriers 0 and N

1 p

q

P =

Bo Friis Nielsen

...

reflecting barriers 0 and N

1 p

p

q

1

p

q

P =

Bo Friis Nielsen

...

reflecting barriers 0 and N

1 p

p

q

1

p

q

P =

0

p

Bo Friis Nielsen

...

reflecting barriers 0 and N

1 p

p

q

1

p

q

P =

0

p

Bo Friis Nielsen

...

...

0

0

0

0

0

0

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

Bo Friis Nielsen

...

...

...

..

.

0

0

0

..

.

0

0

0

..

.

0

0

0

..

.

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

0

0

0

Bo Friis Nielsen

...

...

...

..

.

0

0

0

..

.

0

0

0

..

.

0

0

0

..

.

...

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

0

0

0

Bo Friis Nielsen

...

...

...

..

.

0

0

0

..

.

0

0

0

..

.

...

q 1pq

0

0

0

..

.

p

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

0

0

0

0

0

0

Bo Friis Nielsen

...

...

...

..

.

0

0

0

..

.

0

0

0

..

.

...

...

q 1pq

0

0

0

0

..

.

p

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

0

0

0

0

0

0

Bo Friis Nielsen

...

...

...

..

.

0

0

0

..

.

0

0

0

..

.

...

...

q 1pq

0

q

0

0

0

..

.

p

reflecting barriers 0 and N

1 p

p

0

q

1

q

p

0

q

1pq

P = .

..

..

.

.

.

.

0

0

0

0

0

0

Bo Friis Nielsen

...

...

...

..

.

...

...

q 1pq

p

0

q

1 q

0

0

0

..

.

0

0

0

..

.

0

0

0

..

.

reflecting barrier at 0

Bo Friis Nielsen

reflecting barrier at 0

P=

Bo Friis Nielsen

reflecting barrier at 0

1 p

P =

Bo Friis Nielsen

0 ...

reflecting barrier at 0

1 p

p

q

1

p

q

P =

Bo Friis Nielsen

0

p

0

0

0 ...

0 ...

reflecting barrier at 0

1 p

p

0

0

q

1

q

p

0

0

q

1pq

p

P =

0

0

q

1

p

q

Bo Friis Nielsen

0

0

0

p

...

...

...

...

reflecting barrier at 0

1 p

p

0

0

q

1

q

p

0

0

q

1pq

p

P =

0

0

q

1

p

q

.

.

.

.

..

..

..

..

Bo Friis Nielsen

0

0

0

p

..

.

...

...

...

...

...

absorbing barriers

P=

Bo Friis Nielsen

absorbing barriers

1

P =

Bo Friis Nielsen

absorbing barriers

1

P =

Bo Friis Nielsen

0 ...

absorbing barriers

1

0

q 1 p q

P =

0

p

Bo Friis Nielsen

0

0

0 ...

0 ...

0

0

0

0

0

0

absorbing barriers

P =

0

0

0

1

q 1pq

p

0

0

q

1pq

p

0

0

q

1pq

..

..

..

..

.

.

.

.

Bo Friis Nielsen

0

0

0

p

..

.

...

...

...

...

..

.

0

0

0

0

..

.

0

0

0

0

..

.

0

0

0

0

..

.

absorbing barriers

P =

0

0

0

1

q 1pq

p

0

0

q

1pq

p

0

0

q

1pq

..

..

..

..

.

.

.

.

0

0

0

p

..

.

0 ...

Bo Friis Nielsen

...

...

...

...

..

.

0

0

0

0

..

.

0

0

0

0

..

.

0

0

0

0

..

.

q 1pq p

absorbing barriers

P =

0

0

0

1

q 1pq

p

0

0

q

1pq

p

0

0

q

1pq

..

..

..

..

.

.

.

.

0

0

0

p

..

.

0

0

0 ...

0 ...

0

0

0

0

Bo Friis Nielsen

0

0

...

...

...

...

..

.

q 1 p q p

0

0

1

0

0

0

0

..

.

0

0

0

0

..

.

0

0

0

0

..

.

Bo Friis Nielsen

Bo Friis Nielsen

p1,1 p1,2 p1,3 . . . p1,n

p2,1 p2,2 p2,3 . . . p2,n

P = p3,1 p3,2 p3,3 . . . p3,n

..

..

..

..

..

.

.

.

.

.

pn,1 pn,2 pn,3 . . . pn,n

Bo Friis Nielsen

p1,1 p1,2 p1,3 . . . p1,n

p2,1 p2,2 p2,3 . . . p2,n

P = p3,1 p3,2 p3,3 . . . p3,n

..

..

..

..

..

.

.

.

.

.

pn,1 pn,2 pn,3 . . . pn,n

Two reflecting/absorbing barriers

Bo Friis Nielsen

or infinite

Bo Friis Nielsen

Bo Friis Nielsen

p1,1 p1,2 p1,3 . . .

p2,1 p2,2 p2,3 . . .

p3,1 p3,2 p3,3 . . .

..

..

..

P = ..

.

.

.

.

pn,1 pn,2 pn,3 . . .

.

..

..

..

..

.

.

.

Bo Friis Nielsen

p1,n . . .

p2,n . . .

p3,n . . .

..

..

.

.

pn,n . . .

..

..

.

.

p1,1 p1,2 p1,3 . . .

p2,1 p2,2 p2,3 . . .

p3,1 p3,2 p3,3 . . .

..

..

..

P = ..

.

.

.

.

pn,1 pn,2 pn,3 . . .

.

..

..

..

..

.

.

.

p1,n . . .

p2,n . . .

p3,n . . .

..

..

.

.

pn,n . . .

..

..

.

.

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

a collection of discrete probability distributions

Bo Friis Nielsen

a collection of discrete probability distributions

Bo Friis Nielsen

a collection of discrete probability distributions

I

I

What is the probability of the next state being j knowing that

the current state is i

Bo Friis Nielsen

a collection of discrete probability distributions

I

I

What is the probability of the next state being j knowing that

the current state is i pij = P (Xn = j|Xn1 = i)

Bo Friis Nielsen

a collection of discrete probability distributions

I

I

What is the probability of the next state being j knowing that

the

P current state is i pij = P (Xn = j|Xn1 = i)

j pij = 1

Bo Friis Nielsen

a collection of discrete probability distributions

I

I

I

I

What is the probability of the next state being j knowing that

the

P current state is i pij = P (Xn = j|Xn1 = i)

j pij = 1

We say that P is a stochastic matrix

Bo Friis Nielsen

Bo Friis Nielsen

and onwards

Bo Friis Nielsen

and onwards

Boundary conditions

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

I

X0 could be certain

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

I

X0 could be certain X0 = a

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

I

I

X0 could be certain X0 = a

or random

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

I

I

X0 could be certain X0 = a

or random P (X0 = i) = pi

Bo Friis Nielsen

and onwards

Boundary conditions specify e.g. behaviour of X0

I

I

I

X0 could be certain X0 = a

or random P (X0 = i) = pi

Once again we collect the possibly infinite many

parameters in a vector p

Bo Friis Nielsen

Bo Friis Nielsen

P (Xn = j|X0 = i)

Bo Friis Nielsen

(n)

Bo Friis Nielsen

(n)

I

started in i

Bo Friis Nielsen

(n)

I

started in i

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(P n )

(n)

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

p

0

q

0

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

p

0

q

0

0

0

p

0

0

p

q 1q

P (2) =

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

P (2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2

=

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

P (2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq

=

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

P (2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

Small example

1 p

q

P =

0

0

P (2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

p2

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

p2

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

q(1 p)

=

Bo Friis Nielsen

p2

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

q(1 p)

2qp

=

Bo Friis Nielsen

p2

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

q(1 p)

2qp

=

Bo Friis Nielsen

p2

0

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

q(1 p)

2qp

=

Bo Friis Nielsen

p2

0

0

p2

Small example

1 p

q

P =

0

0

(2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

q(1 p)

2qp

=

2

q

0

Bo Friis Nielsen

p2

0

2qp

0

p2

p(1 q)

Small example

1 p

q

P =

0

0

P (2)

p

0

q

0

0

0

p

0

0

p

q 1q

(1 p)2 + pq (1 p)p

p2

0

2

q(1

p)

2qp

0

p

=

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

P (B) =

Bo Friis Nielsen

n+m

We applyP

the law of total probability

P (B) = k P (B|Ak )

Bo Friis Nielsen

n+m

We applyP

the law of total probability

P (B) = k P (B|Ak ) P (Ak )

P (Xn+m = j|X0 = i)

Bo Friis Nielsen

n+m

We applyP

the law of total probability

P (B) = k P (B|Ak ) P (Ak )

P (Xn+m = j|X0 = i)

=

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

n+m

We applyP

the law of total probability

P (B) = k P (B|Ak ) P (Ak )

P (Xn+m = j|X0 = i)

=

Bo Friis Nielsen

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k )

k

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

(n)

(m)

Pik Pkj

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

P (n+m) = P (n) P (m)

Bo Friis Nielsen

X

P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)

k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

P (n+m) = P (n) P (m) = P n+m

Bo Friis Nielsen

The probability of Xn

Bo Friis Nielsen

The probability of Xn

Bo Friis Nielsen

The probability of Xn

(n)

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j)

(n)

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j) = pj

(n)

(n)

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j) = pj

(n)

(n)

(n)

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j) = pj

(n)

(n)

(n)

p (n)

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j) = pj

(n)

(n)

(n)

p (n) = p

Bo Friis Nielsen

The probability of Xn

Define P (Xn = j) = pj

(n)

(n)

(n)

p (n) = pP (n) = pP n

Bo Friis Nielsen

1 p

q

P =

0

0

p

0

q

0

Bo Friis Nielsen

0

0

p

0

0

p

q 1q

1 p

q

P =

0

0

with p = 13 , 0, 0, 23

p

0

q

0

Bo Friis Nielsen

0

0

p

0

0

p

q 1q

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

0

0

p

0

0

p

q 1q

p (1)

Bo Friis Nielsen

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

(1)

=

1

2

, 0, 0,

3

3

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

p (1) =

1 p

1

2 q

, 0, 0,

3

3 0

0

p

0

q

0

0

0

p

0

0

p

q 1q

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

p (1) =

1 p

1

2 q

, 0, 0,

3

3 0

0

p

0

q

0

0

0

p

0

0

p

q 1q

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

= 1 p ,

3

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

p (1) =

1 p

1

2 q

, 0, 0,

3

3 0

0

p

0

q

0

0

0

p

0

0

p

q 1q

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

= 1 p , p ,

3

3

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

p (1) =

1 p

1

2 q

, 0, 0,

3

3 0

0

p

0

q

0

0

0

p

0

0

p

q 1q

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

= 1 p , p , 2q ,

3

3 3

1 p p

q

0

P =

q

0

0

0

with p = 13 , 0, 0, 23 we get

p (1) =

1 p

1

2 q

, 0, 0,

3

3 0

0

p

0

q

0

0

0

p

0

0

p

q 1q

0

0

p

0

0

p

q 1q

Bo Friis Nielsen

= 1 p , p , 2q , 2(1 q)

3

3 3

3

p=

2

1

, 0, 0,

3

3

,

(1 p)2 + pq (1 p)p

p2

0

2

q(1

p)

2qp

0

p

P 2 =

q2

0

2qp

p(1 q)

0

q2

(1 q)q (1 q)2 + qp

Bo Friis Nielsen

p (2)

Bo Friis Nielsen

(2)

=

Bo Friis Nielsen

1

2

, 0, 0,

3

3

(2)

=

1

2

, 0, 0,

3

3

(1 p)2 + pq (1 p)p

p2

0

q(1 p)

2qp

0

p2

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

Bo Friis Nielsen

(2)

=

1

2

, 0, 0,

3

3

(1 p)2 + pq (1 p)p

p2

0

q(1 p)

2qp

0

p2

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

=

(1 p)2 + pq

,

3

Bo Friis Nielsen

(2)

=

1

2

, 0, 0,

3

3

(1 p)2 + pq (1 p)p

p2

0

q(1 p)

2qp

0

p2

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

=

(1 p)2 + pq (1 p)p

,

,

3

3

Bo Friis Nielsen

(2)

=

1

2

, 0, 0,

3

3

(1 p)2 + pq (1 p)p

p2

0

q(1 p)

2qp

0

p2

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

=

,

,

,

3

3

3

Bo Friis Nielsen

(2)

=

1

2

, 0, 0,

3

3

(1 p)2 + pq (1 p)p

p2

0

q(1 p)

2qp

0

p2

2

q

0

2qp

p(1 q)

2

0

q

(1 q)q (1 q)2 + qp

=

,

,

,

3

3

3

3

Bo Friis Nielsen

1 0 0

P =

0 0 1

Define

T = min {n 0 : Xn = 0 or Xn = 2}

and

u = P(XT = 0|X0 = 1)

Bo Friis Nielsen

v = E(T |X0 = 1)

u = P(XT = 0|X0 = 1)

=

2

X

k =0

2

X

k =0

And we find

u=

P1,0

=

1 P1,1

+

Bo Friis Nielsen

= E(T |X0 = 1)

=

2

X

k =0

= 1+

2

X

k =0

And we find

v=

1

1

=

1 P1,1

1

Bo Friis Nielsen

1

0

0

0

P1,0 P1,1 P1,2 P1,3

P =

P2,0 P2,1 P2,2 P2,3

0

0

0

1

ui = P(XT = 0|X0 = i)

Bo Friis Nielsen

Leading to

u2 = P2,0 + P2,1 u1 + P2,2 u2

and

v1 = 1 + P1,1 v1 + P1,2 v2

v2 = 1 + P2,1 v1 + P2,2 v2

Bo Friis Nielsen

Q R

P =

0 I

Bo Friis Nielsen

T = min {n 0, Xn r }

In state j we accumulate reward g(j), wi is expected total

reward conditioned on start in state i

!

T

1

X

wi = E

g(Xn )|X0 = i

n=0

leading to

wi = g(i) +

Pi,j wj

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

- pertemuan 7Uploaded bybudiabuy
- PropabilityUploaded byArjun Kumar
- ch2Uploaded byPedro Sant'Anna
- Simio asd Simulation PREVIEW 3eUploaded byecho24lock
- Robust Control and Model MisspecificationUploaded byElcan Diogenes
- Levy 1215 Bfs Theme CarrUploaded bymsgwriting
- RandomWalkUploaded byhellogoodbye1234561
- polya note pdfUploaded byNashitah Abd Kadir
- stochastic processessUploaded byAlbert Ingwani
- How to Gamble if You MustUploaded bysamuel_santa
- Prob&StatsBook.pdfUploaded bybrandon_medina2011
- Rr210403 Probability Theory Stochastic Process 2005 SupplyUploaded byapi-3818050
- Notes for math for computer scienceUploaded byCleisaxolocolvtwo Anderson
- ReviewUploaded byhannybag123
- FEbook1MAINUploaded bygwydior
- احصاء شبتر 2Uploaded byapi-3720556
- IPC2012-90139Uploaded byMarcelo Varejão Casarin
- stochastic processUploaded byAnil Kumar
- SimulationUploaded byAnonymous 0qR2NHz9D
- 2011+ +STA2020F+Test+2+SolutionsUploaded byAnn Kim
- ProbabilityUploaded byNaba Raj Shrestha
- handout2Uploaded byfuckscribdlol
- 2 Probability 1Uploaded bypravinthombre
- MIT15_075JF11_chpt14Uploaded byMohammad Jibran Changi
- Queueing AnalysisUploaded byMaria Victoria Blancaflor-Agoncillo
- Stats HW1Uploaded bykobeadjordan
- Notes Ch 4.1 (Student)Uploaded byagralepois
- 2010 ExcelUploaded bySaad Ali
- Discrete & Continuous VariablesUploaded byTeena Thapar
- Basic Probabilty Questions.pdfUploaded byNeil Laquian

- StatisticsUploaded byLuvyy Sugar
- ZtableUploaded bybatmanbittu
- Lecture Notes-March06 (1)Uploaded byrukma
- Decision Tree Analysis for Legal Risk ManagementUploaded byDeona Aucamp
- Ch.8 Discrete Random VariablesUploaded byAmber Michaels
- Proposed Guidelines for the Internal Quality Control of Analytical ResultsUploaded byLaura Marcela
- iso2859-1Uploaded byAlejandra Loaiza
- EE230_Spr2012_hw4Uploaded bygfcalavera
- Mat2001 Cat1 ModelUploaded byPhillip Jebakumar
- edma310Uploaded byapi-284333267
- Lecture 8 nptelUploaded byNARENDRA
- Irjet-v2i3318.pdfUploaded byTeju Ashu
- Class Problems - Basics of Probability.pdfUploaded byAdhitya Kamakshidasan
- cse4syll (1).pdfUploaded byPavankumar Gop
- Problem Set VUploaded byR Gandhimathi Rajamani
- Mathematics ProbabilityUploaded byeduardo acunia
- 14695_Chapter06mgt159Uploaded byNeeraj Bhagat
- probabilityUploaded byqyh58
- Markov Process to forecast the demand of the motor policies in Insurance Industry in the marketUploaded byJournal of Computing
- An advanced approach to earthquake risk scenarios with applications to different European townsUploaded bycuahtli
- Statistics drillsUploaded byIDoBite
- rp_nd07Uploaded byvsalaiselvam
- ESI RiskAssessmentModelUploaded byMilagros R. Annicchiarico
- Handout12Uploaded byBinasxx
- ass5Uploaded byLowry Guetta
- Nilsen-2000.pdfUploaded byAyala Ru
- Probability FundamentalsUploaded bySalman Ahmed
- ChiSquare Df1 Df2 Df3Uploaded byJerome Alexander Ramirez
- 7Accelerated.math.YLPUploaded bycalvinballing
- Ezstrobe TutorialUploaded byPaul Christopher Charlesraj V