You are on page 1of 6

# Continuous-time continuous-state Markov process

Suppose { X (t ), t  } is a Markov Process with the state space V   . Here  is continuous and
the state transition can take at any instant of time t. Suppose the process is at state x0 at time
t  t0 . The state transition probability density function at t is given by f X (t )/ X (t0 ) ( x / x0 ) . For
notational simplicity, let us denote this pdf by f ( x, t / x0 , t0 ) . Further assume that the process is
homogeneous.

Consider the random variable X (t1 ) ) at a time. Given X (t0 )  x0 , the joint PDF of X (t1 ) and
X (t ) is f ( x1 , t1 ; x, t / x0 , t0 ) .Then the marginal density f ( x1 , t1 / x0 , t0 ) can be obtained from as

f ( x1 , t1 / x0 , t0 ) 

 f ( x, t ; x1 , t1 / x0 , t0 )dx

Using the chain rule and subsequently the Markov property, we get

f ( x1 , t1 / x0 , t0 ) 

 f ( x, t / x0 , t0 ) f ( x1 , t1 / x, t , x0 , t0 ) dx


 f ( x, t / x0 , t0 ) f ( x1 , t1 / x, t ) dx

## The result is the Chapman-Kolmogorov equation for a continuous-time continuous-state Markov

process and given by

f ( x1 , t1 / x0 , t0 )  

f ( x, t / x0 , t0 ) f ( x1 , t1 / x, t )dx

We have to know how the process evolves. Similar to the Kolmogorov forward and backward
equations for the evolution of the CTMC, we can get those equations for a continuous time Markov
process. Particularly, the corresponding forward Kolmogorv equation is known as the Fokker
Planck (FP) equations. We omit the derivation of the FP equation here.

## The FP equation is given by

f ( x, t / x0 , t0 ) f ( x, t / x0 , t0 ) 1 2  2 f ( x, t / x0 , t0 )
   ( x, t )   ( x, t )
t x 2 x 2
where
E (( X (t  t )  X (t )) / X (t )  x)
 ( x, t )  lim and
t 0 t
E (( X (t  t )  X (t )) 2 / X (t )  x )
 2 ( x, t )  lim are the time-varying and space-varying
t  0 t
parameters of the process. Note that they are the mean and the variance of the infinitesimal
transition of the process and finite. It is further assumed that
E (( X (t  t )  X (t )) n / X (t )  x )
lim  0 for n  3.
t  0 t
Note that the FP equations are linear partial differential equation (PDE) with the time and space
varying coefficients. The solution is is generally difficult.
When  ( x, t ) and  2 ( x, t ) are constants, the FP equation simplifies to the diffusion equation
given by
f ( x, t / x0 , t0 ) f ( x, t / x0 , t0 ) 1 2  2 f ( x, t / x0 , t0 )
   
t x 2 x 2
with  and  respectively known as the drift and the diffusion coefficient. For the Wiener
2

## process, the transition pdf follows the above PDE.

The FP equation has diverse applications as in the dispersion of suspended particles, the
dynamics of electrons in a semiconductor, aeronautics, image processing and stochastic
finance.

## Solution of the diffusion equation for   0

The diffusion equation in this case is given by
f  x , t  1 2  2 f  x , t 
 
t 2 x 2
Considering t0  0 and x0  0 the solution to the diffusion equation gives
1  x2 
  2 
1 2  t 
f ( x, t / x0  0, t0  0)  e
2 2t

Thus the the transition PDF is Gaussian with time-varying mean and variance. With partial
differentiations of f ( x, t / x0  0, t0  0) with respect to t and x it is easy to show that the above
Gaussian PDF satisfies the FP equations.

We can solve the above PDE using the Fourier transform method.

## Consider the diffusion equation

f  x , t  1 2  2 f  x , t 
  with initial condition X  0   0 with probability 1.
t 2 x 2
 f  x, 0     x, 0 
Let
Y  , t   FT ( f ( x, t ))

 

f ( x, t )e j x dx

Then,
 f  x, t    f  x, t   j x
FT    e dx
 t   t

 
t 
f ( x, t )e  j x dx

 Y , t 
t
Similarly,
 2   2
FT  2 f ( x, t )    2 f ( x, t )e  j x dx
 x   x
 
  f ( x, t )
 f ( x, t )e  j x      j  e j x dx
x    x

## Note that lim F ( x, t )  1 and lim F ( x, t )  0 , As both are constants,

x  x 

f
 0 as x   and x   .
x
 2   f ( x, t )  j x
 FT  2 f ( x, t )     j  e dx
 x   x

  j  f ( x, t )     f  x, t  e
 2  j x
dx


 f  x, t  e
 j x
  2 dx


  Y  , t 
2

Taking the Fourier transform of both sides of the initial condition f  x, 0     x  , we get
Y  , 0   1 .

## The differential equation in the Fourier transform domain is given by

Y  , t  1
   2 2Y  , t 
t 2
with the initial condition Y  , 0   1 .
The above equation can be solved for t as
1
  2 2 t
Y  , t   e 2

## Taking the inverse Fourier transform, we get

1  x2 
  2 
1 2  t 
f ( x, t )  e
2 2t

Note that X (t ) is symmetric about horizontal axis and the variance increases linearly with
time.
If   x, t     0, then
1  ( x  t )2 
  
1 2   2 t 
f  x, t   e
2 2t

## Wiener process or Brownian motion process

One example of the CTMP is the Wiener process or the Brownian motion process .

 
Definition: The random process X  t  , t  0 is called a Wiener process or the Brownian motion
process if it satisfies the following conditions:

## (2) X  t  is an independent increment process.

(3) For each t0  0, t  0 X  t  t0   X (t0 ) has the normal distribution with mean 0
and variance  2 t .
1 x2
1 
f X  t  t0   X  t 0  ( x ) 
2
e 2 t
2 t
Remarks
1 x2
1 
We have f X  t  ( x ) 
2
 e 2 t
2 t
 The conditional CDF
F ( x, t / x0 , t0 )  P ( X (t )  x / X (t0 )  x0 )
 P ( X (t )  X (t0 )  x  x0 / X (t0 )  x0 )
 P ( X (t )  X (t0 )  x  x0 )
 f ( x, t / x0 , t0 )
1 ( x  x0 ) 2

1 2  2 ( t t0 )
 e
2 (t  t0 )
 Wiener process was used to model the Brownian motion – microscopic particles suspended in a
fluid are subject to continuous molecular impacts resulting in the zigzag motion of the particle
named Brownian motion after the British botanist Robert Brown. (1773-1858)
 The Wiener process is characterized by the parameter  . When   1 , the process is called the
standard Wiener process.
A realization of the Wiener process is shown in Figure below

X(t)

Figure

The Wiener process is Markov because of the independent-increment property. It is easy to show that

## Autocorrelation and autocovariance function of the Wiener process

RX  t1 , t2   EX  t1  X  t2 
 EX  t1   X  t2   X  t1   X  t1  Assuming t2  t1
 EX  t1  E  X  t2   X  t1   EX 2
 t1 
 EX 2  t1 
  2t1

Similarly if t1  t 2 RX  t1 , t2    t2
2

 RX  t1 , t2    2 min  t1 , t2 

Thus the Wiener process is not stationary. Since the process is zero-mean,

C X  t1 , t2    2 min  t1 , t2 

## Continuity of the Differentiability of the Wiener process:

For a Wiener process X  t  , 
RX  t1 , t2    2 min  t1 , t2 

 RX  t , t    2 min  t , t    2t  lim RX  t1 , t2 
t1 t ,t2  t

Thus the autocorrelation function of the Wiener process is continuous everywhere implying that it is
m.s. continuous everywhere.

RX  t1 , t2    2 min  t1 , t2 
 2t if t2  0
 RX  0, t2    2
0 other wise
RX  0, t2   2 if t2  0
 
t2 0 if t2  0
RX  0, t2 
 does not exist at t2  0
t2
 2 RX  t1 , t2 
 does not exist at (t1  0, t2  0)
t1t2

## Thus a Wiener process is m.s. differentiable nowhere.

Remark: The Wiener process is not only m.s. continuous but it is continuous with probability 1.
Moreover, each realization of this process is nowhere differentiable. Such a process is difficult to
visualize but has many applications,