You are on page 1of 17

# Markov Models in Medical Decision Making

:
A Practical Guide

FRANK A. SONNENBERG, MD, J. ROBERT BECK, MD

Markov models are useful when a decision problem involves risk that is continuous over
time, when the timing of events is important, and when important events may happen more
than once. Representing such clinical settings with conventional decision trees is difficult
and may require unrealistic simplifying assumptions. Markov models assume that a patient
is always in one of a finite number of discrete health states, called Markov states. All events
are represented as transitions from one state to another. A Markov model may be evaluated

by matrix algebra, as a cohort simulation, or as a Monte Carlo simulation. A newer repre-
sentation of Markov models, the Markov-cycle tree, uses a tree representation of clinical
events and may be evaluated either as a cohort simulation or as a Monte Carlo simulation.
The ability of the Markov model to represent repetitive events and the time dependence of
both probabilities and utilities allows for more accurate representation of clinical settings that
involve these issues. Key words: Markov models; Markov-cycle decision tree; decision mak-
ing. (Med Decis Making 1993;13:322-338)

A decision tree models the prognosis of a patient sub- of survival’ or from standard life tables.’ This paper
sequent to the choice of a management strategy. For explores another method for estimating life expec-
example, a strategy involving surgery may model the tancy, the Markov model.
events of surgical death, surgical complications, and In 1983, Beck and Pauker described the use of Mar-
various outcomes of the surgical treatment itself. For kov models for determining prognosis in medical ap-
practical reasons, the analysis must be restricted to a plications.’ Since that introduction, Markov models
finite time frame, often referred to as the time horizon have been applied with increasing frequency in pub-
of the analysis. This means that, aside from death, the lished decision analyses.’-9 Microcomputer software
outcomes chosen to be represented by terminal nodes has been developed to permit constructing and eval-
of the tree may not be final outcomes, but may simply uating Markov models more easily. For these reasons,
represent convenient stopping points for the scope of a revisit of the Markov model is timely. This paper
the analysis. Thus, every tree contains terminal nodes serves both as a review of the theory behind the Mar-
that represent &dquo;subsequent prognosis&dquo; for a particular kov model of prognosis and as a practical guide for
combination of patient characteristics and events. the construction of Markov models using microcom-
There are various ways in which a decision analyst puter decision-analytic software.
can assign values to these terminal nodes of the de- Markov models are particularly useful when a de-
cision tree. In some cases the outcome measure is a cision problem involves a risk that is ongoing over time.
crude life expectancy; in others it is a quality-adjusted Some clinical examples are the risk of hemorrhage
life expectancy.’ One method for estimating life ex- while on anticoagulant therapy, the risk of rupture of
pectancy is the declining exponential approximation an abdominal aortic aneurysm, and the risk of mor-
of life expectancy (DEALE),2 which calculates a patient- tality in any person, whether sick or healthy. There
specific mortality rate for a given combination of pa- are two important consequences of events that have
tient characteristics and comorbid diseases. Life ex- ongoing risk. First, the times at which the events will
pectancies may also be obtained from Gompertz models occur are uncertain. This has important
implications
because the utility of an outcome often depends on
when it occurs. For example, a stroke that occurs im-
Received February 23, 1993, from the Division of General Internal
mediately may have a different impact on the patient
Medicine, Department of Medicine, UMDNJ Robert Wood Johnson than one that occurs ten years later. For economic
Medical School, New Brunswick, New Jersey (FAS) and the Infor- analyses, both costs and utilities are discounted&dquo;,&dquo;
mation Technology Program, Baylor College of Medicine, Houston, such that later events have less impact than earlier
Texas (JRB). Supported in part by Grant LM05266 from the National ones. The second consequence is that a
given event
Library of Medicine and Grant HS06396 from the Agency for Health than As the
Care Policy and Research. may occur more once.
following example
Address correspondence and reprint requests to Dr. Sonnenberg: shows, representing events that are repetitive or that
occur with uncertain timing is difficult
Division of General Internal Medicine, UMDNJ Robert Wood John- using a simple
son Medical School, 97 Paterson Street, New Brunswick, NJ 08903. tree model.

322
Downloaded from http://mdm.sagepub.com at National Institutes of Health Library on January 21, 2009

patient is &dquo. For a model that ample.given credit&dquo. the average amount perioperative myocardial infarction (MI) following pre. there spans the entire life history of a patient and relatively is little advantage to using a monthly cycle length. Recursive tree mod- eling complications of antico- agulant therapy. if only yearly probabilities are available. The length of the cycle is chosen to represent a determined by the available probability data. On the other hand. rare events the cycle length can be one year. For ex- clinically meaningful time interval. of time) spent in each state. for example monthly or even weekly. An example is the risk of number of cycles (or analogously. for the time spent in each The rapidity of this change in risk dictates a monthly state. 2009 .&dquo. The cycle time also must be shorter if a rate Evaluation of a Markov process yields the average changes rapidly over time. the vious MI that declines to a stable value over six months.324 FIGURE 2. Often the choice of a cycle time will be survival.com at National Institutes of Health Library on January 21. if the time frame is shorter and models events that may occur much more frequently. the cycle INCREMENTAL UTILITY time must be shorter. Seen another way.sagepub. then one need only add together the average Downloaded from http://mdm. If the only attribute of interest is duration of cycle time.

The DEALE with spending one cycle in a particular state is referred can be used to derive the constant mortality rates to as the incremental utility. as shown in cremental utility of 1. * For medical the incremental utility of the absorbing examples. If it has an absorbing state. they can be represented by an n x n matrix.sagepub. However. the transition probability for the FIGURE 3.5 solution of Markov chains described in detail by Beck cycles in the WELL state and 1. of course.7. In general. If the patient spends. a separate incremental utility may be specified for each state. be zero.5 X 1) + (1. there will be n2 transition probabilities. The model is evaluated separately for cost and survival. as discussed below.0. Markov-state diagram. the utility assigned would be (2. expected the probability of dying from unrelated causes will increase continuously.7 quality-adjusted cycles specific mortality rates have resulted in greater reli- to the expected utility. Arrows indicate allowed transitions.7). 325 When performing cost-effectiveness analyses. each multiplied by the incremental utility The net probability of making a transition from one for that state. as the patient gets older. The second component is the n probability of suffering a fatal hemorrhage or embolus Expected utility = ~ tss=l i during the cycle. called the P quantity of utility equal to the duration of a single matrix. For example. state to another during a single cycle is called a tran- sition probability. or 3. the cess depicted in figure 3. Each circle represents a Markov state. the net utility for the Markov process would be infinite. This means that for every cycle table 1. Downloaded from http://mdm. This may or may not be constant over time. 2.9 quality-ad- justed cycles.25 cycles in the DISABLED and Pauker. P Matrix expectancy of the patient. needed to implement a Markov chain.* and that the WELL state has an in. each state. its behavior factor representing the quality of life in that state rel. The Markov process is completely n defined by the probability distribution among the Expected utility = ~ s=l i ts X Us starting states and the probabilities for the individual allowed transitions. representing the financial cost of being in that state for one cycle. Consider the Markov pro. For a Markov model of n states. This number is the quality-adjusted life TaMe 1 . Utility accrued for the entire ance on Markov processes with time-variant proba- Markov process is the total number of cycles spent in bilities. amount of time in the DEAD state and if the incremental utility were non-zero.10. If the incremental utility of availability of specialized software to evaluate Markov the DISABLED state is 0. 2009 . on average. Each state is associated with a quality Markov chain. where ts is the time in state A of Markov process in which the tran- special type spent s. The utility that is associated simple matrix algebra.25 X 0. Cost-effectiveness ratios are cal- culated as for a standard decision tree. changes over time because.’ state before entering the DEAD state. the transition probabilities may change over time. quality of survival is consid- however. forms the basis for the fundamental matrix Markov cycle.com at National Institutes of Health Library on January 21. The first component is the probability of dying from unrelated causes. This matrix. In the most general type of Markov process. When these Let us assume that the DEAD state has an incremen- tal utility of zero. then spending the cycle in processes and the greater accuracy afforded by age- the DISABLED state contributes 0. sition are constant over time is called a probabilities Usually. this probability times spent in the individual states to arrive at an survival for the process. over time can be determined as an exact solution by ative to perfect health. Probabilities representing disallowed transi- spent in the WELL state the patient is credited with a tions will. probabilities are constant with respect to time. the ered important. DEAD state must be because the patient will spend an infinite zero. transition from WELL to DEAD consists of two compo- nents.11 TYPES OF MARKOV PROCESSES Markov processes are categorized according to whether the state-transition probabilities are constant over time or not.

are modelled by tree structure &dquo. The Markov process is being used simply to calculate survival for a terminal node of the tree. At least for medical ap- plications. the Markov process is used to represent all events. The second disadvantage is the restriction to constant transition probabilities. The third disadvan- tage is the need to represent all the possible ways of making a transition from one state to another as a single transition probability. the Markov process incorporates all events of interest and the decision analysis is reduced simply to comparing the values of two Markov processes. if prognosis depends in In any way on past history.exact&dquo. general. the expected utility may be calculated by matrix algebra to yield the fundamental matrix. For more details of the matrix algebra solution the reader is referred to Beck and Pauker. In this case. In panel A below) permits representing all relevant events within (top). CHOLECYSTECTOMY. Tunnel states: the three shaded circles represent tem- porary statesthat can be visited only in a fixed sequence. it requires that there be one distinct state to represent the different histories. However. solution that is not affected by the cycle length. USE OF THE MARKOV PROCESS IN DECISION ANALYSIS The Markov process models prognosis for a given patient and thus is analogous to a utility in an ordinaiy decision tree. the Markov process. if we are trying to choose between surgeiy and medical therapy. 327 1#nufllati’is of Markov Models THE FUNDAMENTAL MATRIX SOLUTION When the Markov process has constant transition probabilities (and constant incremental utilities) for all states. events of interest. A far more efficient structure is shown in figure 6B.’ FIGURE 5. the Markov process. 2009 . which shows.outside&dquo. the matrix algebra solution has been largely relegated to the history books. the expected length of time spent in the state. In panel B (bottom). Use of Markov processes in a decision model. because it requires that an entire Markov process be run for each terminal node. Downloaded from http://mdm. such as operative death and cure. the Markov process is used only as a utility.sagepub.com at National Institutes of Health Library on January 21. we may con- struct a decision tree like that shown in figure 6A. for each starting state. For example. There are three main disadvantages of the matrix formation. This structure is inefficient. In this case. this is less of a problem than when Beck and Pauker’ described the technique. because many commonly available micro- computer spreadsheet programs now perform matrix algebra. The first is the difficulty in per- forming matrix inversion. of which there may be dozens or even hundreds. The use of the cycle tree representation (discussed in detail FIGURE 6. The matrix solution is fast and provides an &dquo.

state is 0. Panel C 2. does not contribute to the cycle sum because its in- ing it into blocks and summing their areas versus cal- culating the area by solving the integral of the function Table 2 . as shown in table 2.000 X 1) + (2.sagepub. The simulation is run for enough cycles so that the entire cohort is in the DEAD state (fig. and U.328 simulation in the DEAD state as a result of operative mortality. In ac- cordance with the transition probabilities specified in FIGURE 7. fs is the fraction of the cohort in state s. This process is repeated in subsequent cycles. because the incremental utility of the DISABLED between a cohort simulation and the matrix formu. is the incremental utility of state s. The first row of the table represents the start- ing distribution. Markov cohort simulation. PanelA (top). which is the sum of MARKOV COHORT SIMULATION the number of cohort members in each state multi- The Markov cohort simulation is the most intuitive plied by the incremental utility for that state. the fraction of the cohort initially in each state is par- titioned among all states according to the transition probabilities specified by the P matrix. For each cycle. The cohort simulation can be represented in tabular form. For example. with the entire cohort in the DEAD state. This results in a new distribution of the cohort among the various states for the subsequent cycle.000 X 0. The simulation considers a hy- pothetical cohort of patients beginning the process with some distribution among the starting states.400. The cycle sum is added to a running total that is referred to as the cumulative utility.000 pa- tients begins in the WELL state. Figure 7B shows the distribution of the cohort after a few cycles.000 patients (20% of the original distribution with all patients in the WELL state.com at National Institutes of Health Library on January 21. The fifth column in table 2 shows the calculation of the cycle sum. For ex- representation of a Markov process. This method may be im- plemented easily using a microcomputer spreadsheet program. The simulation is &dquo. 2009 . The difference ample.7) 7. it is not necessary to have all patients in the same state at the beginning of the simulation. if the strategy repre- sents surgery. The second row shows the distribution at the end of the first cycle. represented by the Markov-state dia- gram in figure 3. shows the initial the P-matrix (table 1). a fraction of the cohort may begin the Downloaded from http://mdm. 7C). In this example. Markov Cohort Simulation describing the curve. However. 2.000 patients to the DEAD state. as follows. Fifty percent of the cohort remains in the WELL state. Thirty percent of the cohort is in the SICK state and 20% in the DEAD state. Con- sider again the prognosis of a patient who has a pros- thetic heart valve. remaining in the WELL state. A hypothetical cohort of 10.run&dquo. the cycle sum during cycle 1 is equal to lation may be thought of as analogous to the difference (6. The utility accrued for the cycle is referred to as the cycle sum and is cal- culated by the formula: where n is the number of states. all pa- tients are in the WELL state.7.000 (60%) (bottom) shows the final distribution. Figure 7A illustrates the cohort at the beginning of the simulation. This leaves 6. Panel B (middle) cohort) have moved to the DISABLED state and another shows the distribution partway through the simulation. The DEAD state = between determining the area under a curve by divid.

com at National Institutes of Health Library on January 21. The features of the three representations are summarized in table 3. where f is the fraction surviving at time t and r is the constant transition rate. on the other hand. and cycle tree) per- mit the analyst to specify transition probabilities and incremental utilities that vary with time. Monte Carlo. Rates may be converted to probabilities if their proper relationship is considered. solution that is not sensitive to cycle time (as in the cohort simulation) or number of trials (as in . . Tpmsiflon Probabilides CONVERSION OF RATES TO PROBABILITIES The tendency of a patient to make a transition from one state to another is described by the rate of tran- sition. The major disadvantages * Adapted from Beck et al. - the Monte Carlo simulation). the fraction COMPARING THE DIFFERENT REPRESENTATIONS Each has Table 3 9 Characteristics of Markov Approaches* representation specific advantages and disadvantages for particular purposes. if these are desired. describes the like- lihood that an event will occur in a given length of time. and Monte Carlo) is the necessity for repetitive and time-consuming calculations.. A probability. Monte Carlo simulation. At any given time.5 Downloaded from http://mdm. The Monte Carlo method and the matrix solutions provide measures of variability.332 of the matrix formulation are the restriction to prob- lems with constant transition probabilities. Such variation is necessary to model certain clinical realities. The matrix manipulations required for a Mar- kov process with a large number of states may require special computational resources.sagepub. The simula- tions (Markov cohort.--. 2009 . However. Such measures are not possible with a cohort simulation. The rate describes the number of occurrences of an event (such as death) for a given number of patients per unit of time and is analogous to an in- stantaneous velocity. -. such as the increase in baseline mortality rate with age.exact&dquo. Probabilities range from zero to 1. The figure shows the state tran- sitions of a single person until death occurs during cycle 6. The fundamental matrix solution is very fast because it involves only matrix algebra and provides an &dquo. the need to express each composite transition probability as a single number. and the difficulty of performing matrix algebra. Rates range from zero to infinity. A dis- advantage common to all simulations (cohort. the avail- ability of specialized microcomputer software to per- form these simulations has made this much less of an issue. The probability of an event that occurs at a constant rate (r) in a specified time (t) is given by the equation: This equation can be easily understood by examining the survival curve for a process defined by a constant rate. cycle tree.. The equation describing this survival curve is: FIGURE 13.