Computers and Chemical Engineering 27 (2003) 235
/249

www.elsevier.com/locate/compchemeng

Simulation of population dynamics using continuous-time finite-state

Markov chains
K. Karen Yin a,*, Hongchuan Yang b, Prodromos Daoutidis c, G. George Yin d
a
Department of Wood and Paper Science, University of Minnesota, St. Paul, MN 55108, USA
b
Department of Electrical Engineering, University of Minnesota, Minneapolis, MN 55455, USA
c
Department of Chemical Engineering and Material Science, University of Minnesota, Minneapolis, MN 55455, USA
d
Department of Mathematics, Wayne State University, Detroit, MI 48202, USA

Received 16 November 2001; accepted 23 August 2002

Abstract

This paper is concerned with the simulation of certain types of population dynamics using Markov chains. A brief review of the
Markovian property, the Chapman
/Kolmogorov equation, and the forward equation together with its solution is given. The
essentials in simulating continuous-time Markov chains are provided and two case studies are presented for demonstration. The first
is a process of drug delivery for which a closed-form solution of the forward equation can also be obtained. The second one deals
with cell population dynamics, for which a Markovian model capturing birth, death, and growth is developed and simulated. The
key to such simulation studies is to specify the generator Q , which in turn requires a thorough understanding of the properties of the
underlying system. In return, one obtains not only the means and variances, but also the entire probability distributions, and the
evolution of the random processes of interest.
# 2002 Elsevier Science Ltd. All rights reserved.

Keywords: Markov chain; Forward equation; Infinitesimal generator; Population dynamics; Particulate processes

1. Introduction Markov chains, a well-known subject introduced by

Population dynamics are concerned with the evolu-
tion of size and/or other properties of a population with
time, and its dependence on the initial and environ-
mental conditions. The study of population dynamics
has been attracting growing attention in the biological,
ecological, and medical sciences, in chemical and
biochemical engineering, and in other fields (see, e.g.
Ramkrishna, 2000 for a recent broad treatment of
population balance modeling and analysis for particu-
late processes). In this paper, we illustrate how Markov
chains can be used as the basis for modeling and
simulating certain types of population dynamics of
interest to engineers.
* Corresponding author. Tel.: /1-612-624-1761; fax: /1-612-625-
6286
E-mail addresses: kyin@umn.edu (K.K. Yin), hyang@ece.umn.edu
(H. Yang), daoutidi@cems.umn.edu (P. Daoutidis), gyin@math.
wayne.edu (G.G. Yin).
0098-1354/02/$ - see front matter # 2002 Elsevier Science Ltd. All rights reserved.
PII: S 0 0 9 8 - 1 3 5 4 ( 0 2 ) 0 0 1 7 9 - 5
Markov in 1906, has been studied by a host of
researchers for many years (see the classical books of
Chung, 1960; Doob, 1953; see also Khasminskii & Yin,
1996; Yin & Zhang, 1998; Yin, Zhang, Yang & Yin,
2001 for some recent results of related works). Stochas-
tic models and Markovian formulations have been
applied to numerous real-world systems under uncer-
tainties (see, e.g. Berthiaux, 2000; Burge & Karlin, 1997;
Chiang, 1980; Durbin, Eddy, Krogh & Mitchison, 1998;
Fan, Nassar, Hwang & Chou, 1985; Nassar, Schmidt &
Luebbert, 1992; Taylor & Karlin, 1998; Yin & Zhang,
1998; Yin, Liu & Johnson, 2002 and the references
therein). Their applications in genetics for finding
genotype distribution of certain generation given the
parent genotype distribution, in the research of relapse,
recovery and death for cancer patients by modeling
birth, death, and illness processes of cancer cells, in
computational molecular biology for gene identification
and prediction, and in the studies of discontinuous
media such as solid mixing, multiphase flow, bubble
236 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
behavior in the gas
/solids fluidized bed, particle diffu-
sivity and mixing coefficient in a liquid
/solid fluidized
bed are a few such examples. Once a Markovian model
is established, It can be used to characterize the mean
behavior of the underlying system, or more importantly
to fully explore its dynamics (this is important especially
for non-Gaussian stochastic processes, where the mean
and variance alone do not contain enough information).
The latter task requires in principle the solution of the
so-called forward equation which provides the evolution
of the probability distribution, This solution is a matrix
exponential, involving the so-called generator, which,
however, can be obtained in closed form only for simple
special cases. Alternatively, for continuous-time Markov
chains one can numerically generate sample paths of the
Markov chain (from which probability distributions and
various statistics can be obtained) based only on the
specification of the generator and the initial conditions
(see Taylor & Karlin, 1998 for more detailed descrip-
tions).
In this paper, we consider two case studies for which
we illustrate the formulation of continuous-time finite-
state Markov chains and their numerical simulation.
The first example concerns a process of drug delivery via
the gastric and intestinal tract, for which a stochastic
multi-compartmental model (Chen, Chen, Hikal, Shen
& Fan, 1998) was developed and the master equation
was derived, yielding the means and covariances of the
random variables of interest. Because of its low dimen-
sion and the special form of its generator, a closed-form
solution of the forward equation of this process is easily
obtainable, and will be presented for comparison. Our
emphasis though is on simulating the piecewise constant
sample paths of the corresponding Markov chains. Such
an approach can provide us with single realizations of
the random processes as well as important statistics, and
yield the probability distributions of the random vari-
ables of interest at any given time. It is particularly
useful in cases where there are no closed-form solutions
in the forward equation, which is typical in high-
dimensional random processes. The second case study
is such an example that concerns cell populations in an
environment of changing substrate concentration. De-
terministic formulations of corpuscular models of such
systems (i.e. cell population balance models) lead to
systems of partial-integro-differential equations whose
solution is a challenging task (see Mantzaris, Daoutidis
& Srienc, 2001a,b,c for recent results in this direction).
In this work, we use Markov chains to model birth,
death, and growth of cells, and simulate the cell
population dynamics.
The paper is organized as follows. Section 2 briefly
reviews the notion of Markov chain and Markovian
property, the Chapman
/Kolmogorov equation, and the
forward equation and its solution. The need of simula-
tion in the study of many stochastic systems is eluci-
dated; the essentials in simulating a continuous-time
Markov chain are also provided. Two illustrative
examples are presented in the next two sections. The
problem formulation, numerical scheme, and simulation
results are presented in Section 3 for the drug delivery
example, and in Section 4 for the cell population
example. Summary and further discussion are given in
Section 5.
2. Markovian property and Kolmogorov equations
A stochastic process (see e.g. Chiang, 1980; Chung,
1960; Doob, 1953; Karlin & Taylor, 1975) can be
considered as a collection of random variables {a (t,
v )} (or {a (t)}) defined on a common probability space
and indexed by the parameter t
/T , where T is a
suitable set and the index t often represents time. For
a fixed t, a (t , ×/) is a random variable. For each v , a ( ×/,
v ) is called a sample path or a realization of the process.
Note that the parameter v is often omitted for brevity.
The state space of a (t) is the collection of all values it
may take.
Markov chains are stochastic processes which satisfy
the so-called Markovian property, i.e. for which given
the current state, the probability of the chain’s future
behavior is not altered by any additional knowledge of
its past behavior. Suppose that a (t) is a continuous-time
stochastic process taking values in the set M/{1, . . .,
m } (i.e. a finite state space). Taking the index set T to be
T /[0, ), the Markov property states that for each i
/
M, the following conditional probability relation holds:
P(a(t)i½a(r):r5s)P(a(t)i½a(s)): (1)
Let a (t) be a Markov chain with state space M and the
transition probabilities Pij (t) /P {a (t/s) /j ja (s) /i}
for all s ]/0 be independent of s; then we say that the
Markov chain has stationary transition probabilities,
the corresponding Markov chain is often referred to as
being stationary or homogeneous, and the transition
matrix P (t)
/Rm m can be written as:
0 1
P1;1 (t) P1;2 (t) . . . P1;m1 (t) P1;m (t)
BP2;1 (t) P2;2 (t) . . . P2;m1 (t) P2;m (t) C
P(t) B@n :: C: (2)
A
n : n n
Pm;1 (t) Pm;2 (t) . . . Pm;m1 (t) Pm;m (t)
In a continuous-time and finite-state Markov chain,
we deal with a family of random variables {a (t):0 5/t B/
} whose states may be indexed by integers. For a
finite-state Markov chain, Pij (t) ]/0, am
j1 Pij  1 for
each i
/M, and the well-known Chapman
/Kolmogorov
equation (see e.g. Chiang, 1980; Chung, 1960; Karlin &
Taylor, 1975; Taylor & Karlin, 1998) holds:
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249 237

X
m Eqs. (5)
/(7) are applicable to any Markov chains with
Pij (th) Pik (t)Pkj (h); (3) finite state space once the generators are identified. A
k0
thorough understanding of the physical process under
or in a more compact form study is needed to prescribe the generator Q , and to
formulate Eqs. (5) and (6). Note that Eq. (7) is merely a
P (t/h )/P (t )P (h ). representation of the solution. To solve Eq. (6) requires
That is, to move from state i to state j in an interval finding the eigenvalues of Q first and then obtaining
t/h, the chain a (t ) moves to some intermediate state k exp(Qt ). A key to this is to find the solutions of det(lI/
in the duration [0, t], and then moves from k to j in the Q )/0, where det(A ) denotes the determinant of A ,
remaining time h . If in addition we assume that which is a polynomial equation of order m . Although
limi 00 P (t)/dij where dij /1 for i/j and dij /0 this polynomial equation can be solved in some special
otherwise, then P (t) is continuous at t /0. Further- cases, there is no closed-form solution available in
more: general for m ]/5.
In this paper, we follow an alternative approach
P(t)  I (Taylor & Karlin, 1998) which numerically generates
lim Q (qij )
Rmm (4)
t00 t the sample paths of the Markov chain on the basis of the
generator Q . Note that for a continuous-time homo-
where I is the identity matrix exists, and the qij ’s satisfy:
geneous Markov chain, in view of Eqs. (5)
/(7), the Q
qij (t)]/0 for j "/i and qii (t)aj"i qij (t) for t ]/0. The
‘generates’ the probabilities, so only the generator Q
matrix Q is termed an infinitesimal generator, or in
and not the transition probability matrix is needed for
short, a generator, and represents the rates of change of
the simulation (see also Chung, 1960; Taylor & Karlin,
the transitions. The transition matrix satisfies the
1998; Yin & Zhang, 1998). Specifically, suppose that
differential equation:
a (t) is a continuous-time Markov chain with state space
dP(t) M /{1, . . ., m } and generator Q /(qij ). To simulate
P(t)Q; P(0)I: (5) a (t) requires designating the state from which it starts,
dt
determining the duration of its stay and the state of its
To find the probability distribution of the Markov chain next entry, and repeating this procedure to determine
at time t, let p(t)(P(a(t)1); . . . ;/. the subsequent sojourn times and resident states. To this
def end, we need to specify the chain’s initial conditions,
P(a(t)m)) (pi (t))
R1m prescribe the parameters of interest, and to identify the
Then p (t) satisfies the following equation: infinitesimal generator Q , upon which the path structure
of the chain can be built.
dp(t)
p(t)Q; p(0)p0 ; (6)
dt The sojourn time

where p0 is the initial distribution. Both Eqs. (5) and (6) Assume that at time t, the chain enters certain state i.
are known as forward equations. It is straightforward Let Si be the sojourn time of a (t) in state i, which is a
that the solutions of Eqs. (5) and (6) are: random variable. To determine the residence time Si
P(t)exp(Qt)
Rmm ; and until the chain first leaves this state requires its prob-
1m
(7) ability distribution. Using Q and the Markov property,
p(t)p0 exp(Qt)
R ;
it can be shown that P {Si ]/t} /exp(qii t). Namely, Si
which are the transition probabilities and the probability follows an exponential distribution with mean /qii1.
distribution at time t , respectively. Its piecewise con-
stant behavior enables us to write the Markov chain as The transition
a(t)am i1 iIfa(t)ig ; where IA(x ) is the indicator func-
tion of the set A given by IA (x) /1 if i
/A and 0 During a period of h (with h sufficiently small) a
otherwise. It is well-known that the mean and higher transition will occur. The chain moves from i to j with
moments can be easily computed by: probability qij /o (h ). Due to the randomness of Si ,
however, the transitions occur at random times rather
X
m X
m
than at fixed time period.
Ea(t) ipi (t); and E[a(t)]l  il pi (t)
i1 i1
(8)
The path structure
for any positive integer l;
where pi (t ) is the ith component of p (t ). The variance To construct the sample path of a (t) amounts to
and other statistics of interest can also be obtained determining its sojourn time at each state and its
readily. subsequent moves. The chain sojourns in any given
238 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
state i for a random length of time, Si , which has an
exponential distribution with parameter (/qii ). Subse-
quently, it will enter another state. Each state j (j/1,
. . ., m , j "/i) has a probability of qij /(/qii ) being the
chain’s next residence. To simulate this process we first
sample an exponential random number Si with para-
meter (/qii ) to fix the residence time in compartment i;
then define a discrete random variable Xi that takes
values in {1, 2, . . ., i/1, i/1, . . ., m }. The chain’s post-
jump location is determined by the value of Xi as
follows:
8 liquid environment. A portion of the microspheres is
>
> 1; if U Bqi1 =(qii );
<2; if q =(q
i1 ii )B U B(qi1 qi2 )=(qii );
Xi 
>
> n
:m; if P q =(q )BU:
k"i;kBm ik ii
where U is a random variable uniformly distributed in
(0, 1). The sample path is thus constructed by sampling
from exponential and U (0, 1) random variables alter-
nately.
In what follows, we illustrate this approach by using
two case studies.
3. The drug delivery process
The first example concerns modeling and simulation
of a controlled-drug delivery by the oral route. This is a
typical random process subject to constant fluctuation.
A number of factors such as physicochemical properties
of the drug and biological conditions of the patient will
affect the drug delivery and release in the gastric and
intestinal regions.
In an earlier work, Chen et al. (1998) developed a
stochastic multi-compartmental model for the con-
trolled-drug release process to describe the drug dis-
solution in four compartments, the stomach, duodenum,
jejunum, and ileum. They derived the master equation
(van Kampen, 1992) from probabilistic population
balances, and obtained the solution by using probability
generating function, which resulted in a first-order
partial differential equation. This partial differential
equation was solved by means of the Lagrange method
of auxiliary equations. The solution further led to the
means and covariances of the microspheres inside the
system. An alternative approach to obtain the means
and covariances by generating the first and second
moments of the master equation was also presented.
Observing the randomness and regularity involved in
the drug release process, we choose to model it with a
family of continuous-time finite-state Markov chains.
Their piecewise constant sample paths constructed by
the numerical scheme simulate the evolution of the
dynamic system. The flexibility in various choices of the
transition and dissolution parameters allows us to
explore a host of phenomena. The major challenge is
to find connections linking the elements of the physical
process to the Markov chains.
3.1. Problem formulation
When a capsule containing chemically-active micro-
spheres/pellets is administered into the GI tract, it
immediately moves to the stomach that is considered
as the first compartment, in which it ruptures and
discharges the microspheres into the slightly acidic
dissolved in the stomach; the undissolved portion exits
from it and moves to the consecutive compartments of
(9)
small intestine, where the dissolution and transition
processes continue. Eventually, the undissolved drug
enters the colon. The drug dissolved in the GI tract is
absorbed into the blood stream for subsequent circula-
tion and distribution into various organs. Given the fact
that the fraction of drug entering the blood from the
colon is small (low bioavailability), we will follow the
conventional treatment (Chen et al., 1998; Gibaldi &
Perrier, 1982 and the references therein) and only
consider dissolutions in the first four compartments
(stomach, duodenum, jejunum, and ileum) that lead to
high bioavailability.
To establish a stochastic model and to design the
simulation procedure for the system described above, we
make the following assumptions:
(A1) the drug transition and dissolution in the system
are Markovian. The process is considered stationary.
(A2) The transition and dissolution of each individual
microsphere are independent; i.e. they are not
affected by the presence of other microsperes.
(A3) Drug degradation is negligible.
3.2. Associated Markov chains
Given the total number of the chemically-active
microspheres/pellets and its initial distribution in each
compartment, we are interested in changes in the
distribution with respect to time and the possible
fluctuations in the random process. We use a family of
continuous-time Markov chains, ak (t) (k /1, 2, . . ., k ).
For each Markov chain, the state space M includes all
m/1 compartments along the GI tract, starting from
the stomach and ending with colon, coded 1, 2, . . ., m/
1, and the blood stream as the mth state. For easy
comparison with results of the referenced work, we
divide the intestine into three sections and, therefore,
m /6 hereafter. Representing k /6 random processes,
these six chains have the same state space M/{l, . . ., 6}
and the same transition probabilities, but start from
different compartments/states, i.e. a (0) /k . Since we are
interested in the control in the first four compartments,
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249 239

only k/2 /4 chains are monitored in the following
simulation examples.
Let y (t )/[y1(t), y2(t), . . ., yk (t)]?, where the symbol
‘?’ denotes transpose. We define yi (t) to be the total
number of microspheres at time t that originated from
division i (at t/0). Note that y(0) /[y1(0), y2(0), . . .,
yk (0)]?, where yi (0) are the number of microspheres in
division i at t/0. Since yi (t) are the number of those at
i originated from the ith division, the dynamics of this
group is governed by the Markov chain starting from
division i . This relates the number of microspheres in
each division i to the Markov chains ai (t).
Let N (t) be the total number of microspheres in the
system at t :
X
k
N(t) yi (t)N(0)N0 :
i1
Thus N (t ) is also related to the Markov chains.
7) mm1 /0; mi /0 for i /1, . . ., m/2.
Note that m /6 in our simulation and that the fifth
and the sixth states correspond to the colon and the
blood stream, respectively. Assumptions 1
/3 and 6
describe the probabilities of the microspheres’ transiting
from one compartment to the next, dissolving and
entering the blood stream from each of the first m/2
compartment, and retaining in the current compartment
during a very short period of time h . Assumption 4 is the
initial condition. In this work, we are not concerned
with the subsequent circulation and distribution from
the blood stream that is the m th division in the model.
Therefore, it is treated as an absorbing state in
Assumption 5. The drug’s dissolution in the colon is
considered negligible and hence mm1 /0 as stated in
Assumption 7. Consequently, colon is also an absorbing
state. Organizing all the assumptions above yield the
infinitesimal generator Q /(qij ).

0 1
(l1 m1 ) l1 0 0  0 m1
B0 (l2 m2 ) l2 0  0 m2 C
B C
B0 0 (l3 m3 ) l3  0 m3 C
B :: C
QB
Bn n n : n n n C
C (10)
B0 0 0  (lm2 mm2 ) lm2 mm2 C
B C
@0 0 0  0 0 0 A
0 0 0  0 0 0

Consequently, simulating the dynamics of the con-
trolled-drug release process amounts to keeping track
of each Markov chain ai (t) as explained in the following
sections.
3.3. Infinitesimal generator
Observe that qij (t) ]/0 for j "/i and qii (t)aj"i qij (t)/
for t ]/0. The last column in the Q matrix indicates the
dissolution in the blood stream from each compartment.
Note that there are two absorbing states in this
problem */the blood stream and the colon. Rather
than a 6/6 matrix, an m /m matrix Q is used for
generalization.

Our approach requires the infinitesimal generator Q .

3.4. Markovian drug delivery process and the closed-form
Suppose all k chains have the same transition prob-
solution
abilities. For notation simplicity, we drop the designator
k of the chain ak (t). Assume that for any fixed a (t )/i The transition and dissolution rates were computed
(i/1, . . ., m ), Pij (h ) for small h satisfies the following
based on the transition and dissolution intensity func-
conditions:
tions provided by Chen et al. Using their estimated
1) Pi ,i1(h )/li h/o(h ) as h 0/0, m /i ]/1 where li is quantities, we calculate the qij ’s and construct the
called the infinitesimal transition rate at division i.
2) Pi ,m (h) /mi h/o (h) as h 0/0, m/2 ]/i ]/1 where mi Table 1
Elements of the generator
is the infinitesimal dissolution rate at division i .
3) Pi ,i (h) /1/(li /mi )h/o (h ) as h 0/0, m /i ]/1. Compartment Index qii //(li /mi ) qi ,i1 /li qi 6 /mi
4) Pi ,j (0) /dij ; the Kronecker delta dij /0 for i "/j,
Stomach 1 /2.249 1.114 1.114
and dij /1 for i/j . Duodenum 2 /0.154 0.149 0.005
5) Pm ,j (t)/0 and Pm ,m (t) /1 for m/1 ]/j ]/1, i.e. m Jejunum 3 /1.059 0.807 0.252
is an absorbing state. Ileum 4 /1.419 0.452 0.967
6) li /0 for i/1, 2, . . ., m/1.
240 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249

generator Q defined in Eq. (10), whose elements are
tabulated in Table 1.
Note that the Q matrix has a simple upper triangular
form. It is readily seen that the eigenvalues of Q are qii
for i /1, . . ., 4 and 0. From elementary linear algebra, it
is also easy to obtain that V 1QV /D (i.e. Q is
diagonalizable), where:
conditions as that in the work of Chen et al., where
y (0) /[100 0 0 0 0 0]?. Namely, at t/0, there were a
total of 100 chemically-active pellets, all in the stomach.
Fig. 1 presents the result of a typical simulation run,
in which (a)
/(d) demonstrate the changes of number of
microspheres in the stomach, duodenum, jejunum, and
ileum along time. The spread of the pellets in the system

0 1
1:0000 0:4695 0:1503 0:1421 0:1049 0:4750
B0 0:8829 0:1606 0:1059 02118 0:4157C
B C
B0 0 0:9755 0:8988 0:2189 0:4114C
V B B0
C;
B 0 0 0:4010 02873 0:3702C C
@0 0 0 0 0:9020 0 A
0 0 0 0 0 0:5433
0 1
1:0000 0:05317 0:0666 0:0647 0:0130 0:4738
B0 1:1326 0:1865 0:1190 0:3491 1:0889C
B C
B 0 0 1:0251 2:2980 0:9808 2:3423C
1
V B B C
B0 0 0 2:4941 0:7944 1:6996C C
@0 0 0 0 1:1086 0 A
0 0 0 0 0 1:8406

and D is a diagonal matrix with diagonal entries /
2.249, /0.154, /1.059, /1.419, 0, and 0. Thus, in
view of Eq. (7), the transition matrix P (t) and the
probability distribution p (t ) (with initial distribution p0)
are given by:
P(t)V exp(Dt)V 1 ; and
1
(11)
p(t)p0 V exp(Dt)V ; distribution of the number of the pellets in each
respectively. A closed-form solution Eq. (11) is thus
obtained.
Due to the low dimension and the special form of the
generator of this process, solving the forward equation
Eqs. (5) and (6) is straightforward. With P (t ) and p (t)
available, statistics such as mean, covariance, and higher
moments can be computed easily from Eq. (8). How-
ever, it should be emphasized that although formulas
are available for solving second-, third- and fourth-oder
polynomial equations, there is no closed-form solution
available in general, while the simulation approach
provides a viable alternative.
3.5. Simulation results
The simulation is relatively simple owing to its low
dimension. It includes three components: generating
sample paths of the Markov chains, examining the
dynamic behaviors of each individual chain and the
combined effects of all chains, and studying the prob-
ability distribution in each state/division at a fixed time.
For easy comparison, we have adopted the same initial
at any given time and its progress along time can also be
easily obtained. Each simulation run provides us with a
single realization of this random process. Repeating the
same simulation procedure a finite number of times and
collecting the results for a fixed time t will yield
probability distribution of the said random variable
a (t, v) at that particular time. Fig. 2 shows the
compartment at the end of the second hour. It shows
that we can expect that, with above 95% confidence,
there will be about 30
/50% drug left in the stomach, and
about 2
/18% in the jejunum, etc.
Repeating the numerical experiment a number of
times, collecting the results at each time instant and then
taking the average give us the sample mean of the pellets
in each compartment. Similar procedure can be used for
the sample variance or standard deviation (S.D.). Fig. 3
displays the means and S.D. obtained from 200 simula-
tion runs. They are in good agreement with those
obtained from the solutions of the master equation as
well as their governing equations (see Figs. 6
/9 in Chen
et al., 1998). Also observe the similarity of the trends
between the same compartments of Figs. 1 and 3.
Our results, e.g. the mean values of 41, 1.5, 11 and 7 at
t /2 h are consistent with those obtained in the work of
Chen et al. Indicative of the fluctuations around the
means, the variances or S.D. in all compartments can
also be computed easily. It has been observed that
compartments retaining larger numbers of particles tend
to have larger variances. Nevertheless, their relative
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249 241

Fig. 1. A single realization of the controlled drug release process-changes of the numbers of pellets in (a) the stomach; (b) the deuodenum; (c) the
jejunum; and (d) the ileum.

Fig. 2. Population distribution at t /2 h in (a) the stomach; (b) the deuodenum; (c) the jejunum; and (d) the ileum.
242 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249

Fig. 3. Calculated temporal variation of the mean number of pellets and its deviations for (a) the stomach; (b) the deuodenum; (c) the jejunum; and
(d) the ileum, ( */) mean; (  ) /S.D.; (
/) /S.D.

Fig. 4. A single realization of the controlled-drug release process when the second dose is taken 4 h after the first */changes of the numbers of pellets
in (a) the stomach; (b) the deuodenum; (c) the jejunum; and (b) the ileum.
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
fluctuations with respect to the means tend to be
smaller. On the contrary, retention in the smallest
compartment, duodenum, is usually the smallest among
the four. However, it experiences the largest relative
fluctuation as compared with other compartments.
We have also calculated the mean residence time of
the drug pellets in the four compartments. They are
2.43, 0.13, 1.16 and 1.38 h, respectively. They are close
to the mean sojourn times /qi ,i with
q1;1  2:25; q2;2  0:15; q3;3  1:06;
(12)
q4;4 1:42
as expected. The results are consistent with those in
Chen et al. (1998) and are useful in determining the
dosage etc. Fig. 4 presents the situation when another
dose is taken 4 h after the first one. Results similar to
those presented in Fig. 2 can also be readily obtained.
3.6. Remarks
In this work, we modeled the controlled-drug release
process using continuous-time and m -state Markov
chains, which simulate the rnicrospheres’ move along
time. All the chains have the same state space {1, . . .,
m }. The generator Q used is similar to a birth
/death
process. Nevertheless, in a birth
/death process, if the
birth and death rates li /0 and mi /0 for all i ]/1, then
it is irreducible and all states are recurrent. Conse-
quently, we can obtain its stationary distribution (Karlin
& Taylor, 1975). On the contrary, in the drug delivery
model there are two absorbing states (colon and blood
stream) while all other states are transient; thus even-
tually there will be no pellets left in the first four
compartments.
4. Cell population dynamics
Numerous investigations have been made of the
continuous propagation of cells with various modeling
methods. Among the earliest works, Tsuchiya, Fredrick-
son and Aris (1966) provided a comprehensive review of
the mathematical modeling of population dynamics.
They considered ‘pure birth processes’ of indistinguish-
able individual cells. Based on the assumption that the
proliferation of cells is a Markov process, the differ-
ential equation of the transition probability was estab-
lished; its solution under a constant growth rate was
obtained; subsequently, the generating function was
obtained and used to estimate the mean and variance
of the distribution. It was pointed out in that work that
if we choose to include cellular structure in the model,
the distribution of structures in the cells must be
considered.
243
Contrary to the traditional stochastic approach which
assumes that cells in a bio-system are all the same, we
take growth into consideration and distinguish indivi-
dual cells according to their mass, Rather than a birth
/
death process, we treat birth
/growth
/death processes.
Observing the randomness and regularity in cells’ birth,
death, and growth process, we model it with continuous-
time and finite-state-space Markov chains. Our formu-
lation allows both constant and variable birth, growth,
and death rates. In addition, it enables us to describe
population dynamics at different ‘stages’ by incorporat-
ing age and other related factors into the model.
The major challenge we face herein is that the cells are
deemed different. As a result, a number of Markov
processes are involved rather than a single one as in the
traditional model. To relate elements of the bioprocess
to the Markov chains, we define the state space of the
Markov chains on the basis of the mass of individual
cells, and formulate the infinitesimal generators of the
underlying Markov processes with the birth
/growth
/
death rates of the bio-system. The developed stochastic
model is used as the basis for simulating the cell
population dynamics in representative case studies.
Issues including specification of the initial conditions,
construction of the sample path structure, and determi-
nation of the sojourn times are addressed.
4.1. Problem formulation
To establish a stochastic model for the growth, split/
birth and death process, we make the following assump-
tions:
(B1) The population is well mixed; each cell acts
independently.
(B2) The cells of a population differ in size, and
individual cells of different sizes have different
growth, death, and reproduction rates, respectively.
This is the major difference between the current
model and the traditional birth and death process
model.
(B3) We assume equal split; namely, one mother cell
splits into two identical daughter cells.
Fig. 5. State space of the Markov chains for the mass structured cell
birth
/growth
/death process.
244 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249

(B4) There are sufficient nutrition supplies, therefore,
the effects of substrate concentration can be ne-
glected.
(B5) The age differences can be neglected.
(B6) The properties of growth, birth, and death do
not vary with time.
We assume equal split for simplicity. In fact, unequal
division can be treated by a slight modification of the
model presented in the following sections. The substrate
condition specified in assumption (B4) can also be
relaxed by including the nutrition effect into the split
and growth parameters. We assume the process of
interest is stationary, therefore (B6) is required. In order
to be able to distinguish the age difference we may
consider the bio-process as a multistage process and
apply the model to different stages using different model
parameters.
4.2. Associated Markov chains
First, we need to decide what the states of the Markov
chains in the model correspond to and how many states
there should be in the model. Let w (t) denote the cell
mass at time t, w(t)
/(0, wmax], where wmax is the
maximum cell mass. To construct a mass-structured
model we divide the entire population into m sections
according to the mass of the individual cells; i.e.
partition (0, wmax] into m subintervals [wi1, wi ]
where:
wmax
wi wi1  for i 1; 2; . . . ; m:
m 8) mi /0 for i/1, 2; mi /0 for i/3, . . ., m .
Assume that cells in the same division are all the same;
i.e. they have the same growth, split, and death rates.
The cells in division i may grow and enter the next
division, i/1; they may split and become two identical
cells leaving division i then entering the division /wi /2 /,
where /A / denotes the integer part of A . The cells in
any division 1B/i 5/m/1 may die and enter the m th
Assume that there are k independent, continuous-
time finite-state Markov chains having the same state
space M /{1, . . ., m }, however, each starting from
different states, i.e. ak (0) /k , (1 B/k B/k ). Assume the
transition probabilities of each chain ak (t) are station-
ary, i.e. Pkij (t)/P {ak (t/s)/jjak (s)/i} for all s /0
for the k th chain. Thus we can model the situation with
a family of k members continuous-time, m -state Mar-
kov chains where each state corresponds to the cell
mass. This model is illustrated in Fig. 5.
4.3. Infinitesimal generator
To specify the generator Q in the structured cell
population problem, we consider all of the k chains
have the same transition probabilities. For notation
simplicity, we drop the subscript k . Assume that for
each fixed i and j/1, . . ., m , Pij (h ) for sufficiently small
h satisfies the following conditions:
1) Pi ,i1(h )/li h/o (h ) as h 0/0, m /i ]/1 where li is
the growth rate of cells in division i .
2) Pi ,m (h )/mi h/o(h) as h 0/0, m /i ]/2 where mi is
the death rate of cells in division i.
3) Pi,l (h )/gi h/o (h ) as h 0/0, m /i ]/3; where l is
the division satisfying wl1 5/wi /2 B/wl and gi , is the
split rate of cells in division i .
4) Pi ,i (h )/1/(li /mi /gi )h/o (h) as h 0/0, m /i ]/
1.
5) Pi ,j (0) /dij , dij /0 for i "/j and dij /1 for i/j.
6) Pm ,j (t) /0, Pm ,m (t) /1; i.e. m is an absorbing state.
7) li /0 for i/1, 2, . . ., m/2; lm1 /lm /0.
9) gi /0 for i/1, m ; gi /0 for i/2, 3, . . ., m/1.
Assumptions 1
/3 describe the probabilities of the
chain’s growth, death, and split in a very short period of
time h if the chain starts in state i . Assumption 6
indicates that the division m is the so-called absorbing
state. Organizing all the assumptions above gives us the
infinitesimal generator Q /(qij ) of a (t).

0 1
l1 l1 0   0 0 0
B0 (l2 m2 ) l2 0   0 m2 C
B C
Bg3 0 (l3 m3 g3 ) l3 0  0 m3 C
B C
QB
B 0 g4 0 (l4 m4 g4 ) l4 0  m4 C C (13)
Bn n n n n n n n C
B C
@0 0  0 gm1 0 (mm1 gm1 ) mm1 A
0 0 0 0 0 0 0 0

division. All the cells in division m are dead cells that Observe that qij (t) ]/0 for j "/i and qii (t)aj"i qij (t)
will not grow or split and will stay there forever. for t ]/0. Note the position of gi , in the matrix is
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
Fig. 6. Simulation example of the Markov model for the mass structured cell birth
/growth
/death process-comparison of different linear growth
rates: l/1.0w in (a and b) vs. l /5.0w in (c and d). Population distributions shown in (c and d) are for t/0 (solid lines) and t /200 (dashed lines),
respectively.
symbolic and it should be determined by the cell mass in
the ith division.
4.4. Markovian cell population dynamics
Given the initial population size and its distribution
with respect to cell mass, the population dynamics can
be modeled by a family of continuous time Markov
chains ai (t ), for i /1, 2, . . ., m/1. These m/1 chains
have the same state space, M/{1, . . ., m }, and the
same transition probabilities, but starting from different
initial states, ai (0) /i. Note that k /m/1 in this case.
Let y(t)/[y1(t ), y2(t), . . ., ym1(0)]? where the
symbol ‘?’ denotes transpose. We define yi (t ) to be the
total number of offsprings at time t that are originated
from division i (at t /0). Note that y(0) /[y1(0), y2(0),
. . ., ym1(0)]?, where yi (0) are the number of cells in
division i at t/0. Since yi (t) are the number of
offsprings at t originated from the ith division, the
dynamics of this group is governed by the Markov chain
starting from division i. Equivalently, the number of
cells in each division i is related to the Markov chains
ai (t).
245
Let N (t) be the size of the entire population at t.
X
m1
N(t) yi (t):
i1
Thus N (t) is also related to the Markov chains.
Consequently, simulating the dynamics of the popula-
tion amounts to keeping track of each Markov chain
ai (t) and the offsprings resulting from its move as
explained in the following sections.
4.5. Simulation results
Several simulation examples performed according to
the above scheme are presented in this section. Each
simulation started from the same initial mass distribu-
tion and initial population size. We have tried different
numbers of divisions in our study. As can be expected,
more divisions can provide smoother profile, however,
require more computation time. In the examples to
follow, the entire population was divided into m /20
sections according to the mass of individual cells. Under
conditions of sufficient substrate concentration as well
as equal partitioning, we studied population dynamics
for constant, linear, and quadratic growth rates with or
246 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249

Fig. 7. Simulation example of the Markov model for the mass structured cell birth
/growth
/death process-comparison of linear (a and b) vs.
quadratic (c and d) growth rates. Population distributions shown in (c and d) are for t/0 (solid lines) and t /200 (dashed lines), respectively.

Fig. 8. Simulation example of the Markov model for the mass structured cell birth
/growth
/death process-comparison of constant (a and b) and
linear split rates (c and d). Population distributions shown in (c and d) are for t/0 (solid lines) and t /200 (dashed lines), respectively.
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249 247

Fig. 9. Simulation example of the Markov model for the mass structured cell birth
/growth
/death process-comparison of cases with (a and b) and
without (c and d) death. Population distributions shown in (c and d) arc for t /0 (solid lines) and t/200 (dashed lines), respectively.

Fig. 10. Simulation example of the Markov model for the mass structured cell birth
/growth
/death process-changes in biomass composition for t /
0
/500.
248 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
without death. The paths of the Markov chains were
constructed by keeping track of their sojourn times and
residence states as illustrated in Section 2.
Figs. 6 and 7 compare two situations where the cells’
split rates in each mass division are the same but their
growth rates are different. Both figures show the
population dynamics along with population distribu-
tions at t/0 and t/200. Fig. 6(c) and (d) present the
case of which the growth rates in each mass division,
li /kli wi (for i/1, . . ., m/1), are five times those
represented in (a)
/(b) of Fig. 6. Fig. 7 displays a
comparison of the case of linear growth rates (a)
/(b)
versus the case where the growth rates are quadratic
functions of mass (c)
/(d), namely. li /kqi w2i .
Fig. 8 compares two situations that have different
split rates for the same mass division but their growth
rates are the same. It was assumed in the above
examples that the cell death is negligible. Fig. 9
demonstrates the death effect. In Fig. 9(a) and (b) the
death rates were assumed to be zero whereas a constant
death rate was assumed for each division and presented
in Fig. 9(c) and (d).
Fig. 10 shows the change in mass composition along
time, where the abscissae are the time axes, covering the
periods of 0 B/t B/100 in (a), 200 B/t B/300 in (b), and
400 B/t B/500 in (c), and the ordinates denote the 20
divisions of this system. Different shades represent
different compositions. It can be seen that the popula-
tion distribution moves from lower mass to higher mass
as time goes on.
It should be noted that the meaning of linear
/
quadratic growth rates used herein are different from
those used in the deterministic models or those used in
classical birth and death processes. The linear or
quadratic in the latter cases refer to the functional
relationship between rates and the number of cells or the
population size. In our case, however, we relate the
growth rates to the cell mass rather than the population
size.
Birth, death, and growth processes can be generalized
to birth, growth, illness, and death processes in which
illness is potentially concurrent, repetitive, and reversi-
ble. Further complexity can also be introduced by
competition of various causes. Models similar to the
one above can be established and all the health, illness,
and death transition probabilities as well as the expected
duration of stay in each state can be calculated. These
quantities, in turn, are useful in estimating other
important parameters of interest such as the population
size in each state and the duration of survival time.
These can be further expanded to the study of various
factors and their impacts on the transition probabilities
and the time of staying in each state, hence their effects
on the survival time, and population size in different
stages etc. More importantly, we will be able to develop
effective measures to create a favorable environment for
cell growth. In this work, we have partitioned the entire
population into a number of groups according to cell
mass, and constructed the model that consists of a
family of continuous-time finite-state Markov chains. It
should be noted that such an approach is not restrictive
to cell mass. It can be generalized to other cases where
groups in a bio-system are characterized by certain
attributes.
5. Summary and discussion
This paper is concerned with stochastic modeling and
simulation using continuous-time and finite-state Mar-
kov chains. Some of the theoretical aspects involved in
Markov chains, the Chapman
/Kolmogorov equation
and the forward equations were reviewed briefly, along
with the necessity of numerical simulation in certain
cases. Two illustrative case studies, a drug delivery
process and a cell growth process, were modeled and
their dynamics were simulated within the Markovian
framework. The approach followed requires integrating
the information of the randomness with the inherent
regularity of the underlying physical process, expressing
them in a probability framework, and relating them to
the corresponding Markov chains. The key is to identify
the generator, which requires a thorough understanding
of the basic properties of the underlying systems. Both
application examples belong to biological systems.
Nevertheless, such an approach is not restricted to
bio-systems. Rather, it is applicable to other participate
processes that may be described via finite-state Markov
chains. Stochastic modeling and simulation have be-
come frequently used and powerful tools in quantifying
dynamic relations of sequences of random events and
uncertainties. Considering the complex and random
natures of many chemical processes, it is conceivable
that these methods will play an increasingly important
role in chemical engineering and related fields.
References
Berthiaux, H. (2000). Analysis of grinding processes by Markov
chains. Chemical Engineering and Science 55 , 4117
/4127.
Burge, C., & Karlin, S. (1997). Prediction of complete gene structures
in human genomic DNA. Journal of Molecular Biology 268 , 78
/
84.
Chen, X., Chen, W. Y., Hikal, A. H., Shen, B. C., & Fan, L. T. (1998).
Stochastic modeling of controlled-drug release. Biochemical and
Engineering Journal 2 , 161
/177.
Chiang, C. L. (1980). In E. Robert (Ed.), An introduction to stochastic
processes and their applications . New York: Krieger Publishing.
Chung, K. L. (1960). Markov chain with stationary transition prob-
abilities . Berlin: Springer.
Doob, J. L. (1953). Stochastic processes . New York: Wiley.
 K.K. Yin et al. / Computers and Chemical Engineering 27 (2003) 235
/249
Durbin, R., Eddy, S., Krogh, A., & Mitchison, G. (1998). Biological
sequence analysis */probabilistic models of proteins and nucleic
acids . Cambridge: Cambridge University Press.
Fan, L. T., Nassar, R., Hwang, S. H., & Chou, S. T. (1985). Analysis
of deep bed filtration data: modeling as a birth
/death process.
American Institute of Chemical and Engineering Journal 31 (11),
1781
/1790.
Gibaldi, M., & Perrier, D. (1982). Pharmacokinetics (2nd ed.). New
York: Marcel Dekker.
van Kampen, N. G. (1992). Stochastic process in physics and chemistry
(2nd ed.). Amsterdam: North-Holland.
Karlin, S., & Taylor, H. M. (1975). A first course in stochastic
processes . Boston: Academic Press.
Khasminskii, R. Z., & Yin, G. (1996). Asymptotic series for singularly
perturbed Kolmogorov Fokker Planck equations. SIAM Journal of
Applied Mathematics 56 , 1766
/1793.
Mantzaris, N. V., Daoutidis, P., & Srienc, F. (2001a). Numerical
solution of multi-variable cell population balance models I. Finite
difference methods. Computers and Chemical Engineering 25 (11
/
12), 1411
/1440.
Mantzaris, N. V., Daoutidis, P., & Srienc, F. (2001b). Numerical
solution of multi-variable cell population balance models II.
Spectral methods. Computers and Chemical Engineering 25 (11
/
12), 1441
/1462.
249
Mantzaris, N. V., Daoutidis, P., & Srienc, F. (2001c). Numerical
solution of multi-variable cell population balance models III. Finite
element methods. Computers and Chemical Engineering 25 (11
/
12), 1463
/1481.
Nassar, R., Schmidt, J., & Luebbert, A. (1992). A stochastic dispersion
model in gas
/liquid flow systems. Chemical and Engineering
Science 47 , 3657
/3664.
Ramkrishna, D. (2000). Population balances. Theory and applications
to particulate systems in engineering . New York: Academic Press.
Taylor, H. M., & Karlin, S. (1998). An introduction to stochastic
modeling . Boston: Academic Press.
Tsuchiya, H. M., Fredrickson, A. G., & Aris, R. (1966). Dynamics of
microbial cell populations. Advnace in Chemical Engineering 6 ,
125
/206.
Yin, G., & Zhang, Q. (1998). Continuous-time Markov chains and
applications. A singular perturbation approach . New York:
Springer.
Yin, G., Zhang, Q., Yang, H., & Yin, K. (2001). Discrete
/time
dynamic systems arising from singularly perturbed Markov chains.
Nonlinear Analysis of Theory Methods and Applications 47 , 4763
/
4774.
Yin, K., Liu, H., & Johnson, N. (2002). Markovian inventory policy
with application to the paper industry. Computers and Chemical
Engineering 26 (10), 1399
/1413.