In which Pij is the conditional probability of the system being in state j at tn+1 , given that it was in

state i at tn . Here Pij is the transitional probability : the probability thatthe system changes from state
i to the state j in the discrete time step from tn to tn+1.

4.3 THE MARKOV CHAIN

In its simplest classical from the first-order, discrete-state, discrete time Markov chain can be
visualized as representing a system with a finite number of discrete states, A, B, C, ..., behaving in
such a way that a transition occurs from state i to j ( where j may be either the same state or a
different state) at each tick of a conceptual Markovian Clock . The model is expressed as a
transition probability matrix with rows represented by i and columns by j. A three-state system can
be shown as :

To state j

A B C

A PAA PAB PAC

From state i B PBA PBB PBC

C PCA PCB PCC

Here, PAA, PBB, PCC,, commonly designated as Pii, represent transitions from a given state to itself,
whereas the offdiagonal entries, designated as Pij where j i, represent transitions to other states.
The notation in the matrix is such that, for state A, PAA is the probability that the system will remain
in the same state, PAB is the probability that the system will move to state B at a given clock tick, and
PAC is the probability that it will move to state C. These three probabilities sum to 1.0. Note that
when transitions occur from a given state to itself, no change in the system is apparent to an on-
looker until the system, on some given clock tick, does change to a different state. The length of time
that the system remains in a given state after having entered it at a particular click, is called the

4.4. GEOMETRIC DISTRIBUTION

This distribution conveniently examined in terms of indenpendent-events model of the kind shown
in equation(4.2.2) .Consider a single six-sided die, in which the sample space has six elements,
representingthe six faces, each with its pattern of dots.Each face has probability 1/6 of occuring face
up on any toss, and probability 5/6 that some other facewill appear on top.If we consider the die
inthese terms , we have a simple system of two states. Ifa given face, say 4, comes up on a given
toss, we ca translatethis into a waiting time by asking how long the system will remain in this
statei.e., What is the liklikhood that a 4 will occur once, twice, or more times in successive throws
before a non-4 shows up.

Wecan set up an equation for this as follows : let P (R= K}be probbability that the number of the
times, R, that a 4 comes faceup isexactly k, where k = 1,2,3 .... the probability that a 4-face apears
initiallyis p =1/16, and the probability that it will notapearonthe next trial is 1-p =5/6. Thus once a 4-
face occurs, the probability that it will occur exactly ktimes means that it must be repeated exactlyk-
1 times, so that on the k-th trial some face other than a 4 will occur. The leads to the gepmetric
density with parameter (1-p) :
P [R =K] = (1-p) (pk-1 ) = (5/6)(1/6)k-1

In this expression asuccess occur when on the k-th trial a non-4 appear face up.

The successive probabilitu are aesily calculated. By setting k= 1, we obtain p{R=1} =(5/6) (1/6)0 = 5/6
= 0,8333