THEORY OF DECISIONS

UNIT 3. MARKOV DECISION PROCESSES

VICTOR ALFONSO RODRIGUEZ

COD. 1053585556

GRUPO: 212066_27

Tutor
RICARDO JAVIER PINEDA

UNIVERSIDAD NACIONAL ABIERTA Y A DISTANCIA UNAD

SCHOOL OF BASIC SCIENCES, TECHNOLOGY AND ENGINEERING
INDUSTRAIL ENGINEERING
2020
INTRODUCTION
Activities to develop

Next, you will find 6 exercises that make up task 3, these should be developed
individually. As a collaborative activity in addition to the discussion, correction,
consolidation of the final work, is the use of the online tool that is available in the
practical learning environment to check results of manual and online methods of
the Exercises raised.

Exercise 1. Markov chains (steady state):

XYZ insurance company charges its customers according to their accident history.
If you have not had accidents the last two years will be charged for the new policy
$ 530,000 (state 0); if you have had an accident in each of the last two years you
will be charged $ 719,000 (State 1); If you had accidents the first of the last two
years you will be charged $ 517.000 (state 2) and if you had an accident the
second of the last two years will be charged $ 778.000 (State 3). The historical
behavior of each state is given by the following cases of accident, taken in four
different events.

According to Table 1 by applying the Markovian processes, ie finding the transition

matrix and solving the respective equations of p * q, where p is the transition matrix
and q the vector [W X Y Z]. Answer:
a. What is the transition matrix resulting from proportionality according to
the accident history?

State    
If you have not had accidents the last two years the last two
E0: years will be charged for the new policy $ 530.000,00
if you have had an accident in each of the last two years you will
E1: be charged $ 719.000,00
If you had accidents the first of the last two years you will be
E2: charged $ 517.000,00
if you had an accident the second of the last two years will be
E3: charged $ 778.000,00

ACCIDENTS IN THE YEAR

STATE E0 E1 E2 E3 TOTAL
E0 1230 820 1025 1025 4100
E1 990 1155 495 660 3300
E2 1125 1575 1800 0 4500
E3 780 780 1300 2340 5200

TRANSITION MATRIX
STATE E0 E1 E2 E3 TOTAL
E0 0.30 0.20 0.25 0.25 1.00
E1 0.30 0.35 0.15 0.20 1.00
E2 0.25 0.35 0.40 0.00 1.00
E3 0.15 0.15 0.25 0.45 1.00

∑
W 0,30 0.20 0.25 0.25 = 1,0
X 0,30 0.35 0.15 0.20 = 1,0
Y q= 0,25 0.35 0.40 0.00 = 1,0
Z 0,15 0.15 0.25 0.45 = 1,0

q*p
EC1 0,30W + 0,20X + 0,25Y + 0,25Z =w W
p= W X Y Z
EC2 0,30W + 0,35X + 0,15Y + 0,20Z = X

EC3 0,25W + 0,35X + 0,40Y + 0,00Z = Y

EC4 0,15W + 0,15X + 0,25Y + 0,45Z = Z

EC5 w+x+y +z= 1

aw+bx+cy+dz+e=0 where w, x, y, z are the variables and e the independent term
equaled to 0

EC1 0,30W-W + 0,20X + 0,25Y + 0,25Z =W 0

EC2 0,30W + 0,35X-X + 0,15Y + 0,20Z = 0
EC3 0,25W + 0,35X + 0,40Y-Y + 0,00Z = 0
EC4 0,15W + 0,15X + 0,25Y + 0,45Z-Z = 0
EC5 w + x + y + z -1 = 0

EC1 -0,70W + 0,20X + 0,25Y + 0,25Z = 0

EC2 0,30W - 0,65X+ 0,15Y + 0,20Z = 0
EC3 0,25W + 0,35X - 0,60Y + 0,00Z = 0
EC4 0,15W + 0,15X + 0,25Y - 0,55Z = 0
EC5 w + x + y + z -1 = 0

TRANSITION MATRIX
STATE E0 E1 E2 E3 TOTAL
E1 -0.70 0.20 0.25 0.25 0
E2 0.30 -0.65 0.15 0.20 0
E3 0.25 0.35 -0.60 0.00 0
E4 0.15 0.15 0.25 -0.55 0
E5 1 1 1 1 -1

b. What is the average premium paid by a customer in Payoff, according to

historical accident rate?
 TRANSITION MATRIX  
STATES EO E1 E2 E3 TOTAL
EO 0,3000 0,2000 0,2500 0,2500 1
E1
E1 0,3000 0,3500 0,1500 0,2000 1
E2 0,2500 0,3500 0,4000 0,0000 1
E3 0,1500 0,1500 0,2500 0,4500 1

STATES EO E1 E2 E3 TOTAL
EO 0,2500 0,2550 0,2675 0,2275 1
E2 E1 0,2625 0,2650 0,2375 0,2350 1
E2 0,2800 0,3125 0,2750 0,1325 1
E3 0,2200 0,2375 0,2725 0,2700 1

STATES EO E1 E2 E3 TOTAL
EO 0,2544 0,2690 0,2630 0,2137 1
E3 E1 0,2534 0,2672 0,2625 0,2169 1
E2 0,2582 0,2716 0,2609 0,2094 1
E3 0,2530 0,2683 0,2638 0,2149 1

STATES EO E1 E2 E3 TOTAL
EO 0,2548 0,2690 0,2625 0,2137 1
E4 E1 0,2548 0,2690 0,2625 0,2137 1
E2 0,2548 0,2690 0,2625 0,2137 1
E3 0,2548 0,2690 0,2625 0,2137 1

AVERAGE PREMIUM= (0,2548∗530,000)¿+(0,2690∗710,000)+¿

( 0,2625∗517,000 ) + ( 0,2137∗778,000 )=$ 630.426
Exercise 2. Markov chains (Initial state multiplication):

In Colombia there are 5 main mobile operators such as Tigo, Comcel, Movistar,
ETB and Uff, which we will call states. The following chart summarizes the odds
that each client has to stay in their current operator or make a change of company.

STATE TIGO COMCEL MOVISTAR ETB UFF

TIGO 0,25 0,15 0,35 0,1 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05
ETB 0,15 0,25 0,05 0,25 0,3
UFF 0,15 0,25 0,3 0,15 0,15

The current percentages of each operator in the current market are for Tigo 0.2 for
Comcel 0.3, for Movistar 0.3, for ETB 0.1 and 0.1 for Uff (initial state).

According to Table 2 by applying the Markovian criteria, solve the multiplication of

the initial state vector (market share) by the probability matrix (transition matrix).
Answer:

a. Find the probability that each user stays with the mobile company for the
3 next periods.
 TRANSITION MATRIX
STATES TIGO COMCEL MOVISTAR ETB UFF
TIGO 0,25 0,15 0,35 0,1 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05
ETB 0,15 0,25 0,05 0,25 0,3
UFF 0,15 0,25 0,3 0,15 0,15

PROBABILITY 0
INITIAL
STATE TIGO COMCEL MOVISTAR ETB UFF
0,2 0,3 0,3 0,1 0,1

0,25 0,15 0,35 0,1 0,15 1

0,2 0,35 0,15 0,1 0,2 1
Po = 0,35 0,2 0,2 0,2 0,05 1
0,15 0,25 0,05 0,25 0,3 1
0,15 0,25 0,3 0,15 0,15 1

Xo = 0,2 0,3 0,3 0,1 0,1 1

0,25 0,15 0,35 0,1 0,15 1

0,2 0,35 0,15 0,1 0,2 1
P1 = 0,35 0,2 0,2 0,2 0,05 1
0,15 0,25 0,05 0,25 0,3 1
0,15 0,25 0,3 0,15 0,15 1

PROBABILITY 1
STATE 1 TIGO COMCEL MOVISTAR ETB UFF
0,245 0,245 0,21 0,15 0,15
 TRANSITION MATRIX
STATES TIGO COMCEL MOVISTAR ETB UFF
TIGO 0,25 0,15 0,35 0,1 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05
ETB 0,15 0,25 0,05 0,25 0,3
UFF 0,15 0,25 0,3 0,15 0,15

PROBABILITY 2
STATE 2 TIGO COMCEL MOVISTAR ETB UFF
0,22875 0,2395 0,217 0,151 0,16375

TRANSITION MATRIX
STATES TIGO COMCEL MOVISTAR ETB UFF
TIGO 0,25 0,15 0,35 0,1 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05
ETB 0,15 0,25 0,05 0,25 0,3
UFF 0,15 0,25 0,3 0,15 0,15

PROBABILITY 3
STATE 3 TIGO COMCEL MOVISTAR ETB UFF
0,22825 0,240225 0,216063 0,152538 0,162925

TIGO 0,22825
COMCEL 0,24023
MOVISTAR 0,21606
ETB 0,15254
UFF 0,16293

ES 1,00000
Exercise 3. Markov chains (Initial state multiplication):

In Colombia there are 6 main mobile operators such as Avantel, Tigo, Comcel,
Movistar, ETB and Uff, which we will call states. The following chart summarizes
the odds that each client has to stay in their current operator or make a change of
company.

STATE TIGO COMCEL MOVISTAR ETB AVANTEL UFF

TIGO 0,1 0,2 0,4 0,1 0,1 0,1
COMCEL 0,1 0,2 0,1 0,2 0,3 0,1
MOVISTAR 0,1 0,3 0,2 0,2 0,2 0
ETB 0,1 0,3 0,2 0,1 0,1 0,2
AVANTEL 0,1 0,15 0,35 0,1 0,1 0,2
UFF 0,1 0,2 0,2 0,3 0 0,2

The current percentages of each operator in the current market are for Tigo 0.1 for
Comcel 0.2, for Movistar 0.3, for ETB 0.1, Avantel 0.1 and 0.2 for Uff (initial state).

According to Table 3 by applying the Markovian criteria, solve the multiplication of

the initial state vector (market share) by the probability matrix (transition matrix).
Answer:
a. Find the probability that each user stays with the mobile company for the
nexts 4 periods.

  TRANSITION MATRIX
STATE TIGO COMCEL MOVISTAR ETB AVANTEL UFF
TIGO 0,1 0,2 0,4 0,1 0,1 0,1
COMCEL 0,1 0,2 0,1 0,2 0,3 0,1
MOVISTA
0,1 0,3 0,2 0,2 0,2 0
R
ETB 0,1 0,3 0,2 0,1 0,1 0,2
AVANTEL 0,1 0,15 0,35 0,1 0,1 0,2
UFF 0,1 0,2 0,2 0,3 0 0,2

PROBABILITY 0
INITIAL
STATE TIGO COMCEL MOVISTAR ETB AVANTEL UFF
0,1 0,2 0,3 0,1 0,2 0,1

0,1 0,2 0,4 0,1 0,1 0,1 1

0,1 0,2 0,1 0,2 0,3 0,1 1
Po = 0,1 0,3 0,2 0,2 0,2 0,0 1
0,1 0,3 0,2 0,1 0,1 0,2 1
0,1 0,15 0,35 0,1 0,1 0,2 1
0,1 0,2 0,2 0,3 0 0,2

Xo = 0,1 0,2 0,3 0,1 0,2 0,1 1

0,1 0,2 0,4 0,1 0,1 0,1 1

0,1 0,2 0,1 0,2 0,3 0,1 1
P1 = 0,1 0,3 0,2 0,2 0,2 0,0 1
0,1 0,3 0,2 0,1 0,1 0,2 1
0,1 0,2 0,4 0,1 0,1 0,2 1
0,1 0,2 0,2 0,3 0,0 0,2 1
 PROBABILITY 1
STATE 1 TIGO COMCEL MOVISTAR ETB AVANTEL UFF
0,1000 0,2350 0,2150 0,1900 0,1500 0,1100 1

MATRIZ DE TRANSICION
STATES TIGO COMCEL MOVISTAR ETB AVANTEL UFF
TIGO 0,25 0,15 0,35 0,1 0,15 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05 0,05
ETB 0,15 0,25 0,05 0,25 0,3 0,3
UFF 0,15 0,25 0,3 0,15 0,15 0,15

PROBABILITY 2
STATE 2 TIGO COMCEL MOVISTAR ETB AVANTEL UFF
0,1000 0,2330 0,2190 0,1670 0,1575 0,1235 1

MATRIZ DE TRANSICION
STATES TIGO COMCEL MOVISTAR ETB AVANTEL UFF
TIGO 0,25 0,15 0,35 0,1 0,15 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05 0,05
ETB 0,15 0,25 0,05 0,25 0,3 0,3
UFF 0,15 0,25 0,3 0,15 0,15 0,15

STATE 3 PROBABILITY 3
TIGO COMCEL MOVISTAR ETB UFF UFF
0,1000 0,23725 0,22033 0,1699 0,1562 0,1229 1

MATRIZ DE TRANSICION
STATES TIGO COMCEL MOVISTAR ETB AVANTEL UFF
TIGO 0,25 0,15 0,35 0,1 0,15 0,15
COMCEL 0,2 0,35 0,15 0,1 0,2 0,2
MOVISTAR 0,35 0,2 0,2 0,2 0,05 0,05
ETB 0,15 0,25 0,05 0,25 0,3 0,3
UFF 0,15 0,25 0,3 0,15 0,15 0,15

PROBABILITY 3
STATE 4 TIGO COMCEL MOVISTAR ETB AVANTEL UFF
0,10000 0,23122 0,22035 0,16969 0,15589 0,12286 1
 TIGO 0,10000
COMCEL 0,23122
MOVISTAR 0,22035
ETB 0,16969
AVANTEL 0,15589
UFF 0,12286

ES 1,00000
 Exercise 4. Markov chains (Initial state multiplication):

Suppose that 4 types of soft drinks are obtained in the market: Colombian, Pepsi
Cola, Fanta and Coca Cola when a person has bought Colombian there is a
probability that they will continue to consume 40%, 20% of which will buy Pepsi
Cola, 10% that Fanta buys and 30% that Coca Cola consumes; when the buyer
currently consumes Pepsi Cola there is a probability that he will continue to buy
30%, 20% buy Colombian, 20% that Fanta consumes and 30% Coca Cola; if Fanta
is currently consumed, the likelihood of it continuing to be consumed is 20%, 40%
buy Colombian, 20% consume Pepsi Cola and 20% go to Coca Cola. If you
currently consume Coca Cola the probability that it will continue to consume is
50%, 20% buy Colombian, 20% that consumes Pepsi Cola and 10% that is passed
to Fanta.

At present, each Colombian brand, Pepsi Cola, Fanta and Coca Cola have the
following percentages in market share respectively (30%, 20%, 10% and 40%)
during week 3.

According to the data by applying the Markovian criteria, solve the multiplication of
the initial state vector (market share) by the probability matrix (transition matrix).
Answer:
a. Find the transition matrix.

  COLOMBIANA PEPSI COLA FANTA COCACOLA

COLOMBIANA 0,4 0,2 0,1 0,3
PEPSI COLA 0,2 0,3 0,2 0,3
FANTA 0,4 0,2 0,2 0,2
COCACOLA 0,2 0,2 0,1 0,5

PROBABILITY 0
INITIAL STATE COLOMBIANA PEPSI COLA FANTA COCACOLA
0,3 0,2 0,1 0,4
 MATRIZ DE TRANSICION
  COLOMBIANA PEPSI COLA FANTA COCACOLA
COLOMBIANA 0,3 0,22 0,13 0,35
PEPSI COLA 0,28 0,23 0,15 0,34
FANTA 0,32 0,22 0,14 0,32
COCACOLA 0,26 0,22 0,13 0,39

b. Find the probability that each user stays with the mark or change to
another for periods 4, 5, 6 and period 7.
0,4 0,2 0,1 0,3 1
0,2 0,3 0,2 0,3 1
Po = 0,4 0,2 0,2 0,2 1
0,2 0,2 0,1 0,5 1

Xo = 0,3 0,2 0,1 0,4 1

0,4 0,2 0,1 0,3 1

0,2 0,3 0,2 0,3 1
P1 = 0,4 0,2 0,2 0,2 1
0,2 0,2 0,1 0,5 1

PROBABILITY 1
STATE 1 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,2800 0,2200 0,1300 0,3700 1

PROBABILITY 2
STATE 2 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,2820 0,2220 0,1350 0,3610 1

PROBABILITY 3
STATE 3 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,2834 0,2222 0,1357 0,3587 1

PROBABILITY 4
STATE 4 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,28382 0,22222 0,13579 0,35817 1

STATE 5 PROBABILITY 5
 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,283922 0,222222 0,135801 0,358055 1

PROBABILITY 6
STATE 6 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,283945 0,222222 0,135802 0,358031 1

PROBABILITY 7
STATE 7 COLOMBIANA PEPSI COLA FANTA COCACOLA
0,283949 0,222222 0,135802 0,358027 1

COLOMBIANA 0,28395
PEPSI COLA 0,22222
FANTA 0,13580
COCA COLA 0,35803

ES 1,00000
Exercise 5. Markov chains (Initial state multiplication):

Suppose you get 6 types of Jeans brands in the Colombian market: Brand 1, Brand
2, Brand 3, Brand 4, Brand 5 and Brand 6. The following table shows the odds that
you continue to use the same brand or change it.

STATE BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6

BRAND 1 0,18 0,18 0,15 0,21 0,18 0,1
BRAND 2 0,14 0,18 0,2 0,19 0,15 0,14
BRAND 3 0,13 0,16 0,15 0,21 0,2 0,15
BRAND 4 0,22 0,16 0,18 0,2 0,18 0,06
BRAND 5 0,15 0,17 0,15 0,17 0,15 0,21
BRAND 6 0,17 0,15 0,17 0,19 0,19 0,13

At present, brand, have the following percentages in market share respectively

(20%, 15%, 17%, 15%, 13% y 20%) during week 4.

According to the data by applying the Markovian criteria, solve the multiplication of
the initial state vector (market share) by the probability matrix (transition matrix).
Answer:

a. Find the transition matrix.

STATE BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6

BRAND 1 0,18 0,18 0,15 0,21 0,18 0,1
 BRAND 2 0,14 0,18 0,2 0,19 0,15 0,14
BRAND 3 0,13 0,16 0,15 0,21 0,2 0,15
BRAND 4 0,22 0,16 0,18 0,2 0,18 0,06
BRAND 5 0,15 0,17 0,15 0,17 0,15 0,21
BRAND 6 0,17 0,15 0,17 0,19 0,19 0,13

PROBABILITY 0
INITIAL
BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
STATE
0,2 0,15 0,17 0,15 0,13 0,2

MATRIZ DE TRANSICION
STATE BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
BRAND 1 0,1673 0,168 0,1673 0,1951 0,1732 0,1291
BRAND 2 0,1645 0,1665 0,1675 0,1957 0,1755 0,1303
BRAND 3 0,167 0,1663 0,1673 0,1937 0,1737 0,132
BRAND 4 0,1666 0,1688 0,1652 0,1964 0,174 0,129
BRAND 5 0,1959 0,1658 0,1678 0,1947 0,1755 0,1303
BRAND 6 0,1661 0,167 0,1658 0,1949 0,1745 0,1317

b. Find the probability that each user stays with the mark or change to
another for periods 4, 5, 6 and period 7.

0,18 0,18 0,15 0,21 0,18 0,1 1

0,14 0,18 0,2 0,19 0,15 0,14 1
0,13 0,16 0,15 0,21 0,2 0,15 1
Po = 0,22 0,16 0,18 0,2 0,18 0,06 1
0,15 0,17 0,15 0,17 0,15 0,21 1
0,17 0,15 0,17 0,19 0,19 0,13 1

Xo = 0,2 0,15 0,17 0,15 0,13 0,2 1

PROBABILITY 1
STATE 1 BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1656 0,1663 0,1660 0,1963 0,1770 0,1288 1

PROBABILITY 2
STATE 2
BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
 0,1663 0,1671 0,1668 0,1951 0,1743 0,1304 1

PROBABILITY 3
STATE 3
BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1662 0,1671 0,1668 0,1951 0,1744 0,1303 1

PROBABILITY 4
STATE 4 BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1662 0,1671 0,1668 0,1951 0,1744 0,1303 1

PROBABILITY 5
STATE 5 BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1662 0,1671 0,1668 0,1951 0,1744 0,1303 1

PROBABILITY 6
STATE 6 BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1662 0,1671 0,1668 0,1951 0,1744 0,1303 1

PROBABILITY 7
STATE 7 BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
0,1662 0,1671 0,1668 0,1951 0,1744 0,1303 1

BRAND 1 0,16625
BRAND 2 0,16711
BRAND 3 0,16682
BRAND 4 0,19513
BRAND 5 0,17439
BRAND 6 0,13031

ES 1
Exercise 6. Use of the practical learning environment. Collaborative activity

Enter the Practical Environment, in this space videos are presented for the use of
the Excel Solver Plug-in and practical tutorials to develop the proposed activities,
remember to attach screenshots to your final collaborative work, the income and
results table for the Exercises raised. In this same space you can carefully review
the Guide for the use of educational resources.
CONCLUSIONS
 BIBLIOGRAPHY

Sharma, J. (2016). Operations Research : Theory and Applications. (pp. 347-378),

New Delhi: Laxmi Publications Pvt Ltd, v. Sixth edition. Retrieved from
http://bibliotecavirtual.unad.edu.co/login?
url=http://search.ebscohost.com/login.aspx?
direct=true&db=nlebk&AN=1950625&lang=es&site=eds-
live&scope=site&ebv=EB&ppid=pp_C

Pineda, R. (2017). Virtual learning object Unit 1. Decisions in an environment of

risk. [Virtual learning object]. Retrieved from
http://hdl.handle.net/10596/13263