Mean time spent in transient states

Some formulas:

For any event A, the indicator function is defined as follows,

(
1, ω ∈ A;
IA (ω) =
0, ω∈/ A.

Then P (A) = E(IA ). The conditional expectation E[X|A] of a random variable X given that the
event A happens is defined as, assuming that P (A) > 0,

E[XIA ]
E[X|A] = .
P (A)

Note that if X = IB for some event B, then the above formula coincides with the definition of
P (B|A). Furthermore,

E[XIA∩B ] EXIA IB E[XIA |B]

E[X|A ∩ B] = = = ,
P (A ∩ B) P (A|B)P (B) P (A|B)

and hence
E[XIA |B] = E[X|A ∩ B]P (A|B).

Derivations for Sij and fij :

Recall that
Sij = E[ the number of n : Xn = j|X0 = i],

and
fij = P ( the chain X ever enters the state j|X0 = i).

1
2

Sij =E[ the number of n : Xn = j|X0 = i]

X X
= E[( the number of n : Xn = j)I[X1 =k] |X0 = i] (because I[X1 =k] = 1)
k k
X
= E[( the number of n : Xn = j)|X1 = k, X0 = i] pik
k
X
=δij + E[( the number of n : Xn = j)|X1 = k] pik
k
X
=δij + Skj pik .
k

In the fourth equality, δij appears because if X0 = i = j, 1 needs to be added since the information
that X0 = j is missing, by the Markov property, in the conditional expectation given the condition
X1 = k. Therefore,
S = I + PT S ⇐⇒ S = (I − PT )−1 .

Now we derive the formula for fij

Sij =E[ the number of n : Xn = j|X0 = i]

=E[(the number of n : Xn = j)I[X ever enters j] |X0 = i]
+ E[(the number of n : Xn = j)I[X never enters j] |X0 = i]
=E[(the number of n : Xn = j)|X ever enters j, X0 = i]fij
+ E[(the number of n : Xn = j)|X never enters j, X0 = i](1 − fij )
=(δij + Sjj )fij + δij (1 − fij ) = δij + fij Sjj .

Therefore,
Sij − δij
fij = .
Sjj