Ch.

21 - Markov Decision Processes

Contents:
Prototype Example for Markov Decision Processes (LP Model for Average Cost Case)
Prototype Example for Markov Decision Processes (LP Model for Discounted Cost Case)

Click on the tab below to select the worksheet of your choice.

 A B C D E F G H I J K L M N O P Q
1 Prototype Example for Markov Decision Processes (LP Model for Average Cost Case)
2
3 Transition Probabilities:
4
5 State 0 State 1 State 2 State 3
6 Decision Decision Decision Decision
7 1 2 3 1 2 3 1 2 3 1 2 3
8 0 0 0.875 0.0625 0.0625
9 State 1 0 1 0.75 0 0.125 0 0.125 0
10 2 0 0 1 0 1 0 0.5 0 0 0.5 0 0
11 3 1 0 0 0
12
13
14 Costs:
15 Decision
16 1 2 3
17 0 $0
18 State 1 $1000 $6000
19 2 $3000 $4000 $6000
20 3 $6000 Average Cost per Period = $1,667
21
22
23 Decision Variables:
24 Decision
25 1 2 3
Sy jk -
Syikpij(k) Constraints
26 0 0.0952 0.0952 - 0.0952 = 0 = 0
27 State 1 0.7143 0 0.7143 - 0.7143 = 0 = 0
28 2 0 0.0952 0 0.0952 - 0.0952 = 0 = 0
29 3 0.0952 0.0952 - 0.0952 = 0 = 0
30 Total Sum: 1 = 1
31
32 Optimal Policy:
33 Decision
34 1 2 3
35 0 1
36 State 1 1 0
37 2 0 1 0
38 3 1
 A B C D E F G H I J K L M N O P Q R
1 Prototype Example for Markov Decision Processes (LP Model for Discounted Cost Case)
2
3 Transition Probabilities:
4
5 State 0 State 1 State 2 State 3
6 Decision Decision Decision Decision
7 1 2 3 1 2 3 1 2 3 1 2 3
8 0 0 0.875 0.0625 0.0625
9 State 1 0 1 0.75 0 0.125 0 0.125 0
10 2 0 0 1 0 1 0 0.5 0 0 0.5 0 0
11 3 1 0 0 0
12
13
14 Costs:
15 Decision
16 1 2 3
17 0 $0
18 State 1 $1000 $6000 Discount Factor (a) = 0.9
19 2 $3000 $4000 $6000
20 3 $6000 Expected Total Discounted Cost = $17,325
21
22
23 Decision Variables:
24 Decision
25 1 2 3
Sy jk -
aSyikpij(k) Constraints
26 0 1.2102 1.2102 - 0.9602 = 0.25 = 0.25
27 State 1 6.6561 0 6.6561 - 6.4061 = 0.25 = 0.25
28 2 0 1.0669 0 1.0669 - 0.8169 = 0.25 = 0.25
29 3 1.0669 1.0669 - 0.8169 = 0.25 = 0.25
30
31
32 Optimal Policy:
33 Decision
34 1 2 3
35 0 1
36 State 1 1 0
37 2 0 1 0
38 3 1