Deep Reinforcement Learning for

Cloud-Edge: Example

Dr. Rajiv Misra

Professor
Dept. of Computer Science & Engg.
Indian Institute of Technology Patna
rajivm@iitp.ac.in
Deep Reinforcement Learning for Cloud-Edge
 Preface
Content of this Lecture:
• In this lecture, we will discuss how Collaborative cloud-edge
approaches can provide better performance and efficiency
than traditional cloud or edge approaches with the help of
some examples

Deep Reinforcement Learning for Cloud-Edge

Example
Consider the resource allocation problem where a client submits the following demands in
three consecutive time slots:

Time-Slot (𝑡) Demand 𝐷! = (𝑑! , 𝑙! )

1 (30, 2)

2 (10, 1)

3 (20, 2)

where (𝑑! ) represents the number of VMs requested and (𝑙! ) represents the duration of
service request. Assume that time slot (1) is the starting slot such that no VMs have been
allocated a priori. There are 80 VMs available at the edge node.

Deep Reinforcement Learning for Cloud-Edge

Example
(a) Resource allocation using private cloud: Suppose that we have our own private cloud and
a policy has been deployed to allocate VMs as per client demands which outputs the following
actions at each timeslot:
Time-Slot (𝑡) Policy Action (𝑥!" )

1 0.4

2 0.7

3 0.8

The action (𝑥)* ∈ [0, 1]) represents the ratio of VMs allocated from the private cloud to the total VMs
requested by client at time slot t. The remaining VMs (1 − 𝑥)* ) are allocated from the edge node.
+,-
Calculate the cost of collaborative cloud side computing (𝐶) ) in the given private cloud setting at each of
the three time slots. Also, find out the number of VMs that will be available at the edge node at the
beginning of fourth time slot
Given Constants:

Constant Value

Stand-by cost of a VM at the edge node (𝑝# ) 0.03

Computing cost of a VM at the edge node (𝑝$ ) 0.20

Computing cost of a private cloud (𝑝% ) 3.00

Deep Reinforcement Learning for Cloud-Edge

Example
(b) Resource allocation using public cloud: Assume that we have replaced the private cloud
with a public cloud setting with a new policy that outputs the following actions at each
timeslot:
Time-Slot (𝑡) Policy Action (𝑘! , 𝑥!" )

1 (1,0.4)

2 (0,0.7)

3 (2,0.8)

where (𝑘) ∈ {0=on_demand,1=reserved,2=spot}) represents the type of public cloud instance that was
+01
allocated. Calculate the cost of collaborative cloud side computing (𝐶) ) in the given public cloud setting at
each of the three time slot. Assume that the same demands were made by client as in part (a) and that no
customization is performed on reserved instances.
Additional Constants:
Constant Value

Unit price of on-demand instance in public cloud (𝑝&' ) 3.0

Unit price of reserved instance in public cloud (𝑝(# ) 1.5

Customization price of reserved instance (𝑝)*$(&+! ) 800

Unit price of spot instance in public cloud (𝑝! ) 1.0

Deep Reinforcement Learning for Cloud-Edge

Example : Solution
Let (𝑒! ) represent the number of VMs available at the edge node after allocation at time slot (t).
Assume 𝑒" = 𝐸 = 80
At time slot (t = 1):
Demand: 𝐷" = (𝑑" , 𝑙" ) = (30, 2)
Action: 𝑥"# = 0.4
No of VMs allocated from cloud: 𝑑"$ = 𝑥"# ∗ 𝑑" = 0.4 ∗ 30 = 12
No of VMs allocated from edge node: 𝑑"% = 𝑑" − 𝑑"$ = 30 − 12 = 18
No of VMs remaining at the edge node: 𝑒" = 𝑒" − 𝑑"% = 80 − 18 = 62
Resources can be successfully allocated from edge node; hence, allocation record will be generated:
Allocation record: ℎ" = 𝑑"% , 𝑙" = (18, 2)
Allocation Record List 𝐻: < ℎ" > ∶ < 18, 2 >
Updated Allocation Record List 𝐻: < ℎ" > ∶ < 18, 1 >
Number of VMs waiting to be released: 𝑛" = 0
Number of VMs available at next time slot: 𝑒& = 𝑒" + 𝑛" = 62 + 0 = 62
Cost at the edge node: 𝐶"% = 𝑒" 𝑝% + 𝐸 − 𝑒" 𝑝' = 62 ∗ 0.03 + (80 − 62) ∗ 0.2 = 1.86 + 3.6 = 5.46
()*
Cost at the private cloud: 𝐶" = 𝑑"$ 𝑝$ + 𝐶"% = 12 ∗ 3.0 + 5.46 = 41.46
(+,
Cost at the public cloud: 𝐶" = 𝑑"$ 𝑝)% + 𝐶"% = 12 ∗ 1.5 + 5.46 = 23.46
Deep Reinforcement Learning for Cloud-Edge
Example : Solution
At time slot (t = 2):
Demand: 𝐷& = (𝑑& , 𝑙& ) = (10, 1)
Action: 𝑥&# = 0.7
No of VMs allocated from cloud: 𝑑&$ = 𝑥&# ∗ 𝑑& = 0.7 ∗ 10 = 7
No of VMs allocated from edge node: 𝑑&% = 𝑑& − 𝑑&$ = 10 − 7 = 3
No of VMs remaining at the edge node: 𝑒& = 𝑒& − 𝑑&% = 62 − 3 = 59
Resources can be successfully allocated from edge node; hence, allocation record will be generated:
Allocation record: ℎ& = 𝑑&% , 𝑙& = (3, 1)
Allocation Record List 𝐻: < ℎ" , ℎ& > ∶ < 18, 1 , (3, 1) >
Updated Allocation Record List 𝐻: < ℎ" , ℎ& > ∶ < 18, 0 , (3, 0) >
Number of VMs waiting to be released: 𝑛& = 18 + 3 = 21
Number of VMs available at next time slot: 𝑒- = 𝑒& + 𝑛& = 59 + 21 = 80
Cost at the edge node: 𝐶&% = 𝑒& 𝑝% + 𝐸 − 𝑒& 𝑝' = 59 ∗ 0.03 + (80 − 59) ∗ 0.2 = 1.77 + 4.2 = 5.97

Cost at the private cloud: 𝐶&()* = 𝑑&$ 𝑝$ + 𝐶&% = 7 ∗ 3.0 + 5.97 = 26.97
(+,
Cost at the public cloud: 𝐶& = 𝑑&$ 𝑝./ + 𝐶&% = 7 ∗ 3.0 + 5.97 = 26.97

Deep Reinforcement Learning for Cloud-Edge

Example : Solution
At time slot (t = 3):
Demand: 𝐷- = (𝑑- , 𝑙- ) = (20, 2)
Action: 𝑥-# = 0.8
No of VMs allocated from cloud: 𝑑-$ = 𝑥-# ∗ 𝑑- = 0.8 ∗ 20 = 16
No of VMs allocated from edge node: 𝑑-% = 𝑑- − 𝑑-$ = 20 − 16 = 4
No of VMs remaining at the edge node: 𝑒- = 𝑒- − 𝑑-% = 80 − 4 = 76
Resources can be successfully allocated from edge node; hence, allocation record will be generated:
Allocation record: ℎ& = 𝑑-% , 𝑙- = (4, 2)
Allocation Record List 𝐻: < ℎ- > ∶ < (4, 2) >
Updated Allocation Record List 𝐻: < ℎ- > ∶ < (4, 1) >
Number of VMs waiting to be released: 𝑛- = 0
Number of VMs available at next time slot: 𝑒0 = 𝑒- + 𝑛& = 76 + 0 = 76
Cost at the edge node: 𝐶-% = 𝑒- 𝑝% + 𝐸 − 𝑒- 𝑝' = 76 ∗ 0.03 + (80 − 76) ∗ 0.2 = 2.28 + 0.8 = 3.08

Cost at the private cloud: 𝐶-()* = 𝑑-$ 𝑝$ + 𝐶-% = 16 ∗ 3.0 + 3.08 = 51.08

Cost at the public cloud: 𝐶-()* = 𝑑-$ 𝑝! + 𝐶-% = 16 ∗ 1.0 + 3.08 = 19.08

Deep Reinforcement Learning for Cloud-Edge

Thank You