0 Up votes0 Down votes

10 views8 pagesX

Jul 05, 2013

© Attribution Non-Commercial (BY-NC)

PDF, TXT or read online from Scribd

X

Attribution Non-Commercial (BY-NC)

10 views

X

Attribution Non-Commercial (BY-NC)

- Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race
- Hidden Figures Young Readers' Edition
- The Law of Explosive Growth: Lesson 20 from The 21 Irrefutable Laws of Leadership
- The Art of Thinking Clearly
- The E-Myth Revisited: Why Most Small Businesses Don't Work and
- The Wright Brothers
- The Other Einstein: A Novel
- State of Fear
- State of Fear
- The Power of Discipline: 7 Ways it Can Change Your Life
- The Kiss Quotient: A Novel
- The 10X Rule: The Only Difference Between Success and Failure
- Being Wrong: Adventures in the Margin of Error
- Algorithms to Live By: The Computer Science of Human Decisions

You are on page 1of 8

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Stephen Bianchi <>

(These notes also include material from the subsequent guest lecture given by Ani Adhikari.) Consider a continuous time stochastic process (Xt , t 0) taking on values in the nite state space S = {0, 1, 2, . . . , N }. Recall that in discrete time, given a transition matrix P = p(i, j ) with i, j S , the n-step transition matrix is simply P n = pn (i, j ), and that the following relationship holds: P n P m = P n+m . Where pn (i, j ) is the probability that the process moves from state i to state j in n transitions. That is, pn (i, j ) = Pi (Xn = j ), pn (i, j ) = P(Xn = j |X0 = i), pn (i, j ) = P(Xm+n = j |Xm = i). Now moving to continuous time, we say that the process (Xt , t 0) is a continuous time markov chain if the following properties hold for all i, j S , t, s 0: Pt (i, j ) 0,

N j =0

Pt (i, j ) = 1,

N k =0 N k =0

This last property can be written in matrix form as Ps+t = Ps Pt . This is known as the Chapman-Kolmogorov equation (the semi-group property). Example 21.1 (Poissonize a Discrete Time Markov Chain) Consider a pois , which son process (Nt , t 0) with rate , and a jump chain with transition matrix P 21-1

21-2

makes jumps at times (Nt , t 0). Let Xt = YNt (YNt is the value of the jump chain at time Nt ), where Xt takes values in the discrete state space S . Assume (Y0 , Y1 , . . .) and Nt are independent. Find Pt for the process Xt . By denition Pt (i, j ) = P(Xt = j |X0 = i). Condition on Nt to give,

Pt (i, j ) =

n=0

=

n=0

t n=0

)n (i, j ) (tP n! .

t tP

Pt = et(P I ) . Pt is a paradigm example of a continuous time semi-group of transition matrices. It is easy to check that Pt Ps = Pt+s , since eA+B = eA eB .

21.1

Generalization

Let A be a matrix of transition rates, where qij = A(i, j ) is the rate of transitions from state i to state j (i = j ), qij 0, and qi = A(i, i) is the total rate of transitions out of state i. Hence, qi = N j =0,j =i qij and each row of A sums to 0. Also, since Pt is continuous and dierentiable we have qij = lim qi Ph (i, j ) h0 h 1 Ph (i, i) . = lim h0 h

In words, qij is the probability per unit time of transitioning from state i to state j in a small time increment (h). The limit relations above can be expressed in matrix form, Pt+h Pt Pt Ph P t Pt [Ph I ] d Pt = lim = lim = lim h0+ h0+ h0+ dt h h h

21-3

hence

Ph I d d Pt = Pt lim = Pt P0 = Pt A = APt , h 0+ dt h dt

d P. dt 0

where A =

d d is Pt = eAt . dt Pt = Pt A is known as the Kolmogorov forward equations and dt Pt = APt is known as the Kolmogorov backward equations. We claim that (Pt , t 0) is a collection of transition matrices of some continuous time markov chain.

In the example given above we have, I) A = (P (i, i) 1) A(i, i) = (P (i, j ). A(i, j ) = P Example 21.2 Consider the two state markov chain with states {0, 1} and A= .

The sojourn times in state 0 and 1 are independent and exponentially distributed with parameters and , respectively. Find Pt for this chain. First note that d Pt = APt = dt Then Pt (0, 0) + Pt (1, 0) Pt (0, 1) + Pt (1, 1) Pt (0, 0) Pt (1, 0) Pt (0, 1) Pt (1, 1) .

d (Pt (0, 0) Pt (1, 0)) = ( + )(Pt (0, 0) Pt (1, 0)). dt This equation has the form f (t) = cf (t), and a solution is Pt (0, 0) Pt (1, 0) = ce(+)t . Since c evaluated at t = 0 is one (i.e., P0 (0, 0) P0 (1, 0) = 1)), we have Pt (0, 0) Pt (1, 0) = e(+)t , which implies d Pt (0, 0) = e(+)t . dt

21-4

Then

t

Pt (0, 0) = P0 (0, 0) +

0

e(+)s ds

(+)t Pt (0, 0) = 1 + e + + (+)t + e . Pt (0, 0) = + + Likewise (+)t e + + (+)t e Pt (0, 1) = + + (+)t Pt (1, 1) = + e . + + Pt (1, 0) = Notice that as t Pt (0, 0) = Pt (1, 0) = + Pt (0, 1) = Pt (1, 1) = . +

These limits are independent of the initial state. The stationary distribution of the chain is 0 = + . 1 = +

21.2

Let (Xt , t 0) be a markov chain with transition rate matrix A, such that A(i, j ) 0 for i = j and A(i, i) = i=j A(i, j ). Let Hi be the holding time in state i (i.e., the process holds in state i for an amount of time Hi and then jumps to another state). The markov property implies Pi (Hi > s + t|Hi > s) = Pi (Hi > t). We also recognize this as the memoryless property of the exponential distribution (fact: the only continuous distribution with the memoryless property is the exponential).

21-5

By the idea of competing risks we will see that Pi (Hi > t) = eA(i,i)t . Imagine that each state j = i has a random variable j exp(A(i, i)). Think of a bell that rings in state j at time j , and from state i the process moves to the state where the bell rings rst. Construct Hi = mini=j j , then Pi (Hi > t) = P(

i=j

j > t)

= i=j eA(i,j )t P = e( i=j A(i,j ))t = eA(i,i)t , where A(i, i) < 0. What is the probability is that the rst transition out of state i is to state k ? This is given by

P(min j = k ) =

i=j 0

P(min dt, min = k )dt P(k dt, all the others are > t)dt

0

= =

0

0 P

i=j

i=k,j

A(i,k ))t

dt

= =

A(i,j ))t

dt

Observe from this calculation that the following mechanisms are equivalent: 1. (a) competing random variables j exp(A(i, j )) for state j = i (b) Hi = minj =i j (c) jump to the state j which attains the minimum 2. (a) Hi exp(A(i, i) =

i=j

A(i, j ))

A(i,k ) A(i,i)

Example 21.3 (Hold Rates for a Poissonized Chain) Consider a general tran , where some of the diagonal entries maybe greater than zero. So we sition matrix P have Pt A A(i, i) A(i, j ) = = = = et(P I ) I) (P (i, i) 1) = (1 P (i, i)) (P (i, j )). (P

21-6

(i, i)) = Hence A(i, i) = (1 P i=j P (i, j ) = i=j A(i, j ) is the total rate of (i, j )) is the rate of i to j transitions. Let transitions out of state i, and A(i, j ) = (P Hi = S 1 + S 2 + + S G , where S1 , S2 , . . . exp() are independent. Dene G to be the number of steps of the chain until the process leaves state i. Then P (i, i)n1 (1 P (i, i)). P(G = n) = P (i, i) and Hi has an exponential So G has a geometric distribution with parameter 1 P distribution with parameter (1 P (i, i)). It is easy to check the expectations, since (i, i)), and E[Hi ] = 1/(1 P (i, i)). Then by Walds E[Si ] = 1/, E[G] = 1/(1 P identity 1 1 1 . = E[Hi ] = E[Si ]E[G] = (i, i) (i, i)) 1P (1 P

21.3

Theorem

Theorem 21.4 (for nite state irreducible chain) There is a unique probability distribution so that 1. limt Pt (i, j ) = j for all i. 2. solves the balance equations A = 0 (equivalently, Pt = for all t). 3. j is the long-run proportion of time spent in state j 4. the expected return time is Ej [Tj ] = Proof: [for part (2)] Assume (1) is true. Ps+t (i, j ) =

k

1 . q j j

Letting s we get j =

k

k Pt (k, j )

21-7

Look at the j th element of A (which we get by multiplying by the j th column of A). We see that i qij j qj = 0

i=j

i=j

This is the reason these equations are referred to as balance equations. Computations can often be simplied if there is also detailed balance, which simply means that for all pairs i, j (i = j ) we have i qij = j qji . This is the condition of reversibility of the markov chain. Example 21.5 (Birth-Death Process) Consider a process that, from state i, can only move to state i + 1 or state i 1. That is qi,i+1 = i qi,i1 = i qi,j = 0, j {i + 1, i 1}. So i is the birth rate and i is the death rate. We try to nd so that 0 q0,1 = 1 q1,0 1 q1,2 = 2 q2,1 . . . i1 qi1,i = i qi,i1 . Then 0 q0,1 = 1 q1,0 0 0 = 1 1 0 1 = 0 . 1 And 1 q1,2 = 2 q2,1 1 1 = 2 2 1 0 1 1 = 0 . 2 = 2 1 2

21-8

Hence k = and 0 =

0 1 k1 0 , 1 2 k 1

0 1 k1 k =1 1 2 k

1+

- markov chain chapter 6Uploaded bydurga03
- Class 05 - Failure Analysis TechniquesUploaded byb_shadid8399
- Or Ch5 Queing ZaferUploaded byMackee
- Intro to MarkocChain Chutes-And-LaddersUploaded byMahewish Jamali
- Complexity ParondoUploaded byNattapat Bunyuenyongsakul
- 2010 Time Series - An Introduction to Nonlinear Empirical Modeling Bezruchko SmirnovUploaded byHeraldo Souza
- profit or pauper shott piner stem cafe feb17Uploaded byapi-264326218
- Notes on Markov ChainUploaded byCynthia Ng
- tmp4533.tmpUploaded byFrontiers
- Operating Characteristic Analysis on the M[x]G1 System With a Variant Vacation Policy and BalkingUploaded byebenesarb
- Lecture on econometrics of income inequalityUploaded byRjnbk
- Extraordinary Bonus Home TasksUploaded byk_zen865412
- InTech-Spatial Dynamic Modelling of Deforestation in the AmazonUploaded byTony Bu
- Stochastic Differential Equation ModelingUploaded byJournal of Computer Applications
- Queueing Kap 5Uploaded byRahayuKristiyanti
- Mat TinglyUploaded byF_habibi
- UT Dallas Syllabus for opre7310.001.10f taught by Shun-Chen Niu (scniu)Uploaded byUT Dallas Provost's Technology Group
- UT Dallas Syllabus for opre6330.501.08f taught by Shun-chen Niu (scniu)Uploaded byUT Dallas Provost's Technology Group
- Selcuk Et Al 1Uploaded byAntobilang Antobilang
- dualsubmit2Uploaded bypostscript
- Papers i Max RevUploaded byLuis Carlos Gonzales Rengifo
- KWY04Uploaded byAnonymous BkmsKXzwyK
- help-section1.pdfUploaded byVrund
- Monte Carlo SimulationsUploaded byDaniel Casas-Orozco
- psol-ch6Uploaded byAl Farabi
- Clayton RevisedUploaded byannglove
- Towards the Analysis of Extreme ProgrammingUploaded bydoug_disposable
- Markov Chain Sequencial Negotiation - Weingart-prietula-seqUploaded bycarmo-neto
- veewUploaded byMuhammadRusdin
- A fractional PDE for first passage time of time-changed Brownian motion and its numerical solutionUploaded byFrancesca Carfora

- ps08Uploaded byspitzersglare
- sol05.pdfUploaded byspitzersglare
- sol04Uploaded byspitzersglare
- sol03Uploaded byspitzersglare
- sol01Uploaded byspitzersglare
- Polling News and Notes 04-26-12Uploaded byspitzersglare
- sol02Uploaded byspitzersglare
- ps13Uploaded byspitzersglare
- ps11Uploaded byspitzersglare
- Lec25Uploaded byspitzersglare
- ps12Uploaded byspitzersglare
- ps10Uploaded byspitzersglare
- ps09Uploaded bysniprip
- ps07Uploaded byspitzersglare
- ps06Uploaded byspitzersglare
- ps05Uploaded byspitzersglare
- ps04Uploaded bysniprip
- ps03Uploaded byspitzersglare
- ps02Uploaded byspitzersglare
- ps01Uploaded byspitzersglare
- UntitledUploaded byapi-61200414
- Proofs SolutionsUploaded byspitzersglare
- Probability Jan 12Uploaded byspitzersglare
- Parking Placards Reg 3164Uploaded byspitzersglare
- Mqf Top 10 RanksUploaded byspitzersglare
- Midterm2 ConceptsUploaded byspitzersglare

- LG_42LD650_lcdUploaded byUlrih VonStachziert Swansfidensogoth Perez
- Actual Costing/ Material LedgerUploaded bymanish_fico
- Vdocuments.net Extreme Exos Concepts Guide 153Uploaded byErnesto Casariego
- multibridgemacmanual.pdfUploaded bygpn
- RF low noise ampUploaded byGiannis Gkouzionis
- Online RTO ReportUploaded byArchie Srivastava
- AF MicroAdjustGuide DesktopUploaded bysm3in
- CCNA 1 Final Exam 2012Uploaded bypimp267
- GATE-2018-Computer-Science-Information-Technology-Answer-Key.pdfUploaded byGurudev
- Euro Code Soft wUploaded byzrilek1
- ImplantableUploaded byFlorea Marius
- wsdl newUploaded byPradeep Gurunathan
- CM215fw UserGuide en 424bUploaded byjagad boemi
- ADOC 2010 Workshop PeñalolenUploaded byBenjamin David Coderch Soto
- Liferay, Jira & Crowd IntegrationUploaded bymustufarahi
- 60434956 OracleappstechDocument PDFUploaded byDeepak Khosla
- Toll Free Numbers (India) _ Customer Care Service ListUploaded byvkrm14
- 1999 WeaponSystemsHandbookUploaded byAnn Caroline McMichael
- Sample TestsUploaded bykarafile
- MIKE213 FM HD Short DescriptionUploaded byDicky Farista
- Westok Software User Guide_v5aUploaded byChris Evans
- An Analysis of the Broadband Noise ScenarioUploaded byCristiano Marques
- MAI Custom is MetricsUploaded byELisaRull
- Red Hat Enterprise Linux-5-5.3 Release Notes-En-USUploaded byyudha_mulya
- Technology Forecasting and Assessment.docxUploaded bysolo_sudhan
- Lecture 3 - SDLCUploaded byMartin Obretenov
- Numerical Simulation and Physical Modeling as Educational Tools to Teach Metal Forming ProcessesUploaded byjcabarcav7953
- Domonial TCU Operating InstructionsUploaded byargenteo1
- siemens_igbt_caractUploaded bykaran007_m
- Create your ADempiere customization environment - AdempiereWikiUploaded bywarx11

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.