Future Generation Computer Systems 78 (2018) 191–210

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

An autonomic resource provisioning approach for service-based cloud

applications: A hybrid approach
Mostafa Ghobaei-Arani a , Sam Jabbehdari b,∗ , Mohammad Ali Pourmina a
a
Department of computer engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
b
Department of computer engineering, North Tehran Branch, Islamic Azad University, Tehran, Iran

highlights
• We designed a framework for autonomic resource provisioning to cloud services.
• We customized an autonomic resource provisioning approach based on the control MAPE loop.
• We enhanced the performance of the planning phase by using the RL-based agent.
• We conducted a series of experiments under real-world workload traces for different metrics.

article info abstract

Article history: In cloud computing environment, resources can be dynamically provisioned on deman for cloud services
Received 24 March 2016 The amount of the resources to be provisioned is determined during runtime according to the workload
Received in revised form changes. Deciding the right amount of resources required to run the cloud services is not trivial, and it
22 November 2016
depends on the current workload of the cloud services. Therefore, it is necessary to predict the future
Accepted 12 February 2017
Available online 15 February 2017
demands to automatically provision resources in order to deal with fluctuating demands of the cloud
services. In this paper, we propose a hybrid resource provisioning approach for cloud services that is
Keywords:
based on a combination of the concept of the autonomic computing and the reinforcement learning (RL).
Cloud computing Also, we present a framework for autonomic resource provisioning which is inspired by the cloud layer
Cloud services model. Finally, we evaluate the effectiveness of our approach under two real world workload traces. The
Resource provisioning experimental results show that the proposed approach reduces the total cost by up to 50%, and increases
Autonomic computing the resource utilization by up to 12% compared with the other approaches.
Reinforcement learning © 2017 Elsevier B.V. All rights reserved.

1. Introduction an enabling technology for cloud computing, and this architectural

Cloud computing is one of the most popular technologies in
the businesses, educational institutions, governments, and the
research community that has become an integral part of many
users are heavily dependent on cloud-based applications for their
day-to-day activities in both professional and personal life [1,2]. In
the simplest terms, cloud computing means storing and accessing
data and programs that are delivered as services to the end users
over the internet [3,4]. Service-oriented architecture (SOA) [5] is
∗ basic service models includes: infrastructure as a service (IaaS),
Corresponding author.
E-mail addresses: m.ghobaei@srbiau.ac.ir (M. Ghobaei-Arani),
s_jabbehdari@iau-tnb.ac.ir (S. Jabbehdari), pourmina@srbiau.ac.ir
(M.A. Pourmina).
http://dx.doi.org/10.1016/j.future.2017.02.022
0167-739X/© 2017 Elsevier B.V. All rights reserved.
style helps to deliver separately application functionalities in
form of a service as the main building blocks of applications
and system development. The cloud services are the result of
integrating cloud computing and SOA, and are typically comprise
one or more functionalities that are offered by cloud providers
and are can be delivered to the end users over the internet [6,
7]. A cloud application is a software product running on the cloud
environment, and it can be used with a web browser connected
to the internet. Typically, each cloud application is composed of
one or more cloud services that together perform the function of
an application. Usually, cloud services are categorized into three
platform as a service (PaaS), and software as a service (SaaS). In
this paper, we use the term cloud service instead of the phrase cloud
software service offered by SaaS providers. In cloud ecosystem,
192 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210
the end users submit the requests for utilizing the cloud services
offered by a SaaS provider that owns a cloud application, and to
host its cloud application, it rents resources from an IaaS provider
such as Amazon EC2 [8].
One of the unique characteristics of cloud computing is
its elasticity, which enables SaaS providers to adapt workload
changes of their cloud services by provisioning and deprovisioning
resources automatically such that at each point in time, the
available resources match the current demand as closely as
possible [9]. The SaaS providers can acquire and release resources
for running their cloud services on demand and only pays for the
resources that are actually used based on a pay-per-use model [10,
11].
Deciding the right amount of resources for the cloud service
during its execution time is not trivial, and it depends on the
current workload of the cloud service. As users have irregular
access to cloud services offered by a SaaS provider, the cloud
services will experience workload fluctuations. These fluctuating
workloads may lead to undesirable states that are referred
to as over-provisioning or under-provisioning states. The over-
provisioning state can exist when the more resources than
demands of a cloud application are provisioned. This is correct from
the point of view of service level agreements (SLAs); however, it
incurs an unnecessary cost to the user and SaaS provider. On the
other hand, the under-provisioning state can exist when the fewer
resources than demands of a cloud application are provisioned.
This problem causes SLA violations, which lead to lose of revenue
and users [12,13]. Therefore, an effective elasticity mechanism
must be able to estimate the needed resources properly to satisfy
a given SLA based on the current workload of a cloud application.
To deal with the mentioned above resource provisioning prob-
lems, dynamic resource provisioning is utilized. The dynamic re-
source provisioning is an effective approach which its fundamental
idea is to provision the resources based on the workload changes of
the cloud application. Its objective is to automate the dynamic pro-
visioning of resources by minimizing the cost of renting resources
from an IaaS provider and meeting the SLA of the cloud applica-
tion. The main objective of the SaaS provider is to maximize its
profit during the execution of its cloud application, and this can be
achieved by minimizing the payment of using resources from the
IaaS provider, as well as the penalties cost caused by SLA violations
that have to be paid to users.
In this paper, we propose a hybrid resource provisioning ap-
proach for cloud applications based on a combination of the con-
cept of the autonomic computing and the reinforcement learning
(RL). To achieve autonomic computing, IBM has proposed a refer-
ence model for autonomic control loops [14,15], which is called the
control MAPE (Monitor, Analysis, Plan, Execute) loop. The control
MAPE loop is similar to the general agent model proposed by Rus-
sel and Norvig [16], in which an intelligent agent perceives its en-
vironment using sensors, and uses these perceptions to determine
actions to be executed in the environment. The proposed approach
follows the control MAPE loop, which consists of four phases: mon-
itoring (M), analysis (A), planning (P), and execution (E). First, in
the monitoring phase, a monitoring component gathers the infor-
mation about the resources and cloud application state, and this in-
formation is processed to estimate future resource utilization and
demands at the analysis phase. In the planning phase, a suitable re-
source modification action (e.g., scale in or scale out) is determined,
and finally, the modification actions determined in the planning
phase are performed in the execution phase. The control MAPE
loop is regularly executed and manages the virtual machines (VMs)
that are allocated to each cloud service at specific time intervals.
We apply RL [17,18] as a decision-maker that uses the predicted
results of an analysis phase in order to obtain the optimal action to
remove or add VMs in the planning phase. RL is an adaptive self-
learning system that improves its performance through repeated
interactions with the cloud environment. The main contributions
of this research can be summarized as follows:
• We designed a framework for autonomic resource provisioning
which is inspired by the cloud layer model and PaaS layer, it is
responsible for resource provisioning to cloud services based on
the control MAPE loop.
• We customized an autonomic resource provisioning approach
for cloud services offered by a SaaS provider, which is covered
all the phases defined within the control MAPE loop.
• We enhanced the performance of the planning phase of the
control MAPE loop by using the RL-based method as a decision-
maker.
• We conducted a series of experiments to evaluate the
performance of proposed approach under real-world workload
traces for different metrics.
The rest of this paper is organized as follows: In Section 2, we
focus on a survey of related work. Section 3 provides the necessary
background. In Section 4, we formulate the problem and describe
the proposed solution. In Section 5, we present an evaluation and
discuss the experimental results, and in Section 6, we conclude the
paper and present future works.
2. Related works
The dynamic resource provisioning mechanisms are achieved
by scaling in/out the resources (i.e., removing or adding a
VM) through a set of rules to match as closely as possible
the available resources with the current workload. Since the
resource provisioning proposed approach is a combination of
the autonomic computing and the reinforcement learning (as a
machine learning technique), we will focus on dynamic resource
provisioning techniques into the following two major categories:
(i) resource provisioning based on the autonomic computing
techniques [19–27], (ii) resource provisioning based on machine
learning techniques [28–37].
2.1. Resource provisioning based on autonomic computing techniques
Huebscher et al. [19] presented a survey on autonomic
computing and the IBM’s MAPE-Knowledge (MAPE-K) reference
model. They introduced the motivation and concepts of autonomic
computing, degrees, models, and applications. In [20] proposed a
framework dynamic cloud provisioning of system topologies for
common two-tier application scenarios based on a MAPE loop
concept, while our framework applies to cloud services offered
by SaaS providers. Pop et al. [21] reviewed advanced topics
in resource management for ubiquitous cloud computing, and
proposed an adaptive approach that maximizes the profit for
service providers, while it minimizes the total cost to customers.
Maurer et al. [22] proposed adaptive resource provisioning
techniques based on the autonomic control MAPE loop for cloud
infrastructural management. These techniques are the case-based
reasoning and a rule-based approach, while our approach employs
time series analysis and machine-learning techniques. In [23]
designed an adaptive framework based on the control MAPE loop
for optimizing the configuration of scientific applications in three
layers, i.e., the application layer, execution environment layer, and
the resource layer. In [24], the LoM2HiS framework is presented,
which is used for managing the mappings of the low-level resource
metrics into high-level SLA parameters. The LoM2HiS framework
is embedded into the FoSII [25] infrastructure, which facilitates
autonomic SLA management and enforcement. In [26], the authors
developed a dynamic resources provisioning and monitoring
(DRPM) system. Moreover, they proposed a multi-agent system
to manage the cloud provider’s resources, where the customers’
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 193

QoS requirements that are determined by the SLA is considered.

The DRPM system has three main phases: monitoring, analysis,
and execution. However, our approach considers the planning and
analysis phases separately. In [27] proposed mechanisms for SLA
provisioning to application service provider (ASP) that uses a cloud
infrastructure to achieve scalable provisioning of its services. They
designed four autonomic service management architectures that
differ from each other based on the degree of control of an ASP.
These architectures include extreme ASP control, full ASP control,
partial ASP control, and limited ASP control. Several components
of the PaaS layer of our framework are inspired from the proposed
full ASP control.

2.2. Resource provisioning based on machine learning techniques

Bahrpeyma et al. [28] proposed a novel adaptive control

approach that is based on continuous reinforcement learning
and provides dynamic resource provisioning while dealing with
uncertainty in the cloud’s spot market. Their approach considers
the interaction between the IaaS provider and SaaS provider,
while our proposed method considers the interaction between the
user and SaaS provider. Islam et al. [29] developed prediction-
based resource measurement and provisioning strategies using a
combination of neural networks and linear regression to satisfy
future resource demands, while our work considers only the
time series prediction models. In [30] applied reinforcement
learning towards automating resource allocation and application
scalability in the cloud. They proposed a novel parallel Q -learning
approach which consists of multiple RL agents that aimed at
reducing the time taken to determine optimal policies, while
our proposed method uses only single RL agent. Liu et al. [31]
presented a generic reinforcement learning-based aggressive
virtualized resource management system for IaaS clouds. It utilizes
reinforcement learning with an aggressive provisioning strategy
to overcome the rapidly increasing workloads. In [32] proposed
a dynamic resource provisioning approach-based discrete-time
Markov chain (DTMC) model. The proposed approach focuses
on resource allocation at the application level, which allocates
an appropriate number of VMs to an application, whereas our
proposed method also considers the interaction between the
user and SaaS layer. Muppala et al. [33] proposed a coordinated
admission control and adaptive resource provisioning approach for
multi-tier service differentiation. They develop a new model based
on reinforcement learning techniques for VM auto-configuration.
Our approach utilized simple Q -learning algorithm, which is a kind
of reinforcement learning that does not have a high computational
overhead. Roy et al. [34], developed a predictive-aware model for
the workload prediction in which a second-order autoregressive
moving-average method filter is used. Their algorithm utilized
a second-order auto-regressive moving-average (ARMA) method
filter, while our algorithm utilized a combination of the time
series prediction models based on error metrics for workload
prediction. In [35] presented an learning automata-based QoS
(LAQ) framework that ensures the computing resources are used
in an efficient manner, and are not over- or under-utilized by
the consumer applications. The learning system into the LAQ
framework, which is based on LA, is responsible for selecting
suitable target resources for the user requests with a maximum
job success rate. In our approach, the RL acts as a decision-maker
in the planning phase when selecting adaptation actions. In [36]
presented a unified reinforcement learning methodology for auto-
configuration of VMs and appliances in cloud environment. Their
method relies on an RL-based App-Agent to tune the application
parameter settings and an RL-based VM-Agent to adjust VM
configurations whereas our approach focuses only RL-based agent
for tuning of VM configurations. Yang et al. [37] proposed a cost-
aware auto-scaling approach based on workload prediction using
Fig. 1. The control MAPE-K loop.
the linear regression model for dynamic resource provisioning.
Their approach considers only workload prediction techniques for
dynamic resource provisioning, even though our approach utilized
the workload prediction technique for the analysis phase of the
control MAPE loop concept.
Table 1 sums up some of the most relevant works related
to resource provisioning techniques, highlighting the elasticity
solutions based on four characteristics: (1) objective metrics, (2)
policy, (3) scope and (4) method to make provisioning decisions.
3. Background
This section exposes a brief overview of the autonomic
computing and reinforcement learning technique.
3.1. Autonomic computing
To effectively manage cloud deployed applications it is essential
that reactions to regularly occurring or anticipated events are
built into the system. The autonomic computing can be applied to
implement a cloud application system which knows its state and
reacts to changes in it. The term autonomic computing was first
used by IBM in 2001 to describe computing systems that are said
to be self-managing [14]. To achieve autonomic computing, IBM
has proposed a reference model for autonomic control loops [38],
which is called the control MAPE-K loop, as shown Fig. 1. In
the control MAPE-K loop, the managed element represents any
software or hardware resource such as a cloud application, a
cloud service, the operating system, a CPU, a storage, a VM, etc.
The sensors, collect information about the managed element. The
information can be the response time to cloud services, CPU
and memory utilization. The effectors carry out changes to the
managed element. The change can be adding or removing VMs
to a cloud application (i.e., replication) or changing configuration
parameters in a VM (i.e., resizing). The data collected by the sensors
allows the autonomic manager to monitor the managed element
and execute changes through effectors. The autonomic manager is
a software component for monitor the system, analyze the metrics,
plan corrective actions and execute them.
The knowledge in the control MAPE-K loops is shared
data between the actual MAPE phases. The cloud application
deployment is monitored and the configuration is adjusted based
194 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Table 1
Survey of works related to resource provisioning techniques.
Ref. Category Objective metrics Policy Scope Method

Ritter [20] Autonomic computing CPU utilization Proactive SaaS Replication

Maurer [22] Autonomic computing CPU utilization Proactive IaaS Replication/Resizing
Emeakaroha [24] Autonomic computing Response time Reactive IaaS Replication/Resizing
Ayyoub [26] Autonomic computing CPU utilization Proactive IaaS/SaaS Replication
Casalicchio [27] Autonomic computing CPU utilization/response time Reactive SaaS Replication
Bahrpeyma [28] Machine learning Response time Proactive IaaS Replication
Islam [29] Machine learning CPU utilization Proactive SaaS Replication
Barrett [30] Machine learning Response time Reactive IaaS Replication
Liu [31] Machine learning Response time Proactive IaaS Replication
Qavami [32] Machine learning CPU utilization Reactive SaaS Replication
Muppala [33] Machine learning Response time Proactive SaaS Replication/Resizing
Misra [35] Machine learning Response time Proactive IaaS Replication
Xu [36] Machine learning CPU utilization/response time Proactive SaaS Replication/Resizing
Proposed Hybrid CPU utilization/response time Proactive SaaS Replication

on metrics reported by sensors attached to the application or

its environment. Monitoring data is analyzed by the autonomic
manager in the corresponding phase of the control MAPE-K loop.
The raw sensor data is turned into knowledge in this phase.
Analysis of the system model may indicate that one or more criteria
of acceptable system behavior are no longer met (reactive trigger)
or some metric is about to exit its tolerated range (proactive
trigger). Given such a situation, the autonomic manager will enter
the planning phase with the purpose of creating a plan of action to
bring the metric values back to or keep them in the tolerance zone.
This plan can be based on a set of rules that govern the operation
of the autonomic manager. The execution phase is where the
autonomic manager or its delegate effector components interface
with the application and the cloud environment to carry out Fig. 2. The relationship between reinforcement learning agent and its environ-
the actions decided in the planning phase. This phase relies on ment.
automation APIs available for the environment and the runtime
configurability of the application. The executed actions will cause
where γ is a discount factor (0 ≤ γ ≤ 1), a value close to 1 for γ
changes in the behavior of the system which are then reported back
assigns a greater weight to future rewards, whereas a value close to
to the autonomic manager in subsequent control loops.
0 considers only the most recent rewards. The steps of Q -learning
are shown in Algorithm 1.
3.2. Reinforcement learning RL is appropriate for learning where there is incomplete
information regarding the environment and in dynamic, complex,
RL [17,18,28,30,31] is a process of learning which generates
or random environments with a large number of uncertainties
optimal action policies on a given environment state through trial-
such as cloud environment, computer networks, etc. RL has been
and-error interactions within a dynamic environment. Formally,
applied successfully across a range of domains include cloud
the RL problem can be modeled as a Markov Decision Process (MDP)
computing [28,30,31,36,39], computer networks [40,41], multi-
that consists of a discrete set of states s, a discrete set of actions
agent systems [42,43] and robotics [44,45].
a and a reinforcement signal (i.e. reward). For a set of states S
and a set of actions A, we can define MDP as a reward function
R : S × A → R and a state transition function T : S × A → π (S ). 4. Proposed approach
π(S ) is a probability distribution over S. The objective of the agent
is to find an optimal policy π to choose the action that maximizes In this section, we explain our proposed approach in more
the long-term reward. As illustrated in Fig. 2, for a system at time t detail. The proposed approach matches the control MAPE loop
with state st ∈ S, the agent selects an action at ∈ A. After the action concept and regularly executes regarding specified time intervals
at is applied to the environment, the state transits from st to st +1 for each cloud service offered by the SaaS provider. For the rest
and an immediate reward rt +1 is returned from the environment. of this section, we first introduce a framework based on the
The planning phase of the control MAPE loop in proposed ap- control MAPE loop, and we then present a problem formulation for
proach is implemented as an RL-based agent, which continuously resource provisioning. Finally, we describe the autonomic resource
takes actions to change the amount of provisioned resources ac- provisioning algorithm for cloud services.
cording to the current performance status of the system. Among RL
methods, we use a Q -learning algorithm for the implementation of 4.1. The resource provisioning framework
RL, because it is a model-free learning algorithm that does not re-
quire an interactive model of the environment. The Q -learning al- In this section, we describe a resource provisioning framework
gorithm is defined by the Q function. Q (s, a) is the value function that supports the control MAPE loop for which the proposed
that the agent at time t performing action at at the current state approach is intended. As depicted in Fig. 3, the overall framework
st . The Q function can be updated each time an action is applied to consists of three layers in the cloud model, namely the SaaS layer,
the environment and is defined by Eq. (1): PaaS layer, and IaaS layer. The SaaS layer includes cloud services
offered by SaaS providers to users. The PaaS layer is responsible
Q (st , at ) ← r (s, a) + γ max Q (st +1 , at +1 ) (1) for resource provisioning to cloud services offered by the SaaS
at +1
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210
Fig. 3. Resource provisioning framework based on the control MAPE loop.
layer. The role of this layer is to handle the resource provisioning
algorithm based on control MAPE loop. The IaaS layer includes
data centers at which VMs are hosted, and it offers VMs to SaaS
providers; Moreover, it is responsible for dispatching VMs to
run on their physical resources. Also, we consider three entities,
namely users, SaaS provider, and IaaS provider. The users (i.e., end
users) submit requests for the use of cloud services that are
offered by a SaaS provider. The SaaS provider is the owner of a
cloud application, and to host the cloud application, it utilizes the
internal resources of its data centers (in-house hosting) or leases
resources from the third party IaaS provider such as Amazon EC2.
The latter case is the subject of this research. Each SaaS provider
offers the different types of cloud services. The IaaS provider offers
virtually unlimited resources in the form of VMs. The cloud services
are loosely coupled, and each of them runs on one VM. It can be
extended to run more than one cloud services on a VM, but we
will not discuss that scenario in this paper. Furthermore, there
are two types of SLA, namely the user SLA and the resource SLA.
The user SLA is a contract between the SaaS provider and users,
and includes items such as the deadline, budget, penalty rate, and
request length. The resource SLA is a contract between the SaaS
provider and IaaS provider, and includes items such as the VM
195
price, VM type, processing speed, service time, and data transfer
speed. In this work, we focus on the interaction between users
and the SaaS Provider (i.e., the user SLA) and calculating the SLA
violation. The SaaS provider has a dual role (i.e., buyer/seller). In
other words, the SaaS provider leases resources from IaaS provider
(buyer), and in turn leases cloud services to users (seller). The main
objective of SaaS providers is to maximize profit. To do this, they
need to minimize the infrastructure cost as well as the penalty cost
caused by SLA violations. To achieve this goal, SaaS providers aim
to minimize the payment for using resources from IaaS provider,
and also to guarantee the QoS requirements for users.
As shown in Fig. 3, the main components of the resource
provisioning framework based on the control MAPE loop are:
A. Monitor
The monitor component is responsible for collecting metrics
related to the resources and users. The monitor component con-
sists of two subcomponents, namely the resource monitor and user
monitor. The resource monitor component is responsible for col-
lecting information about the metrics of the computational, stor-
age, and network resources (e.g., CPU utilization, memory usage,
and network traffic). The user monitor component is responsible
for collecting information about the workload submitted by users
196 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Table 2
Notations and definitions.
Notation Definition

U Number of users
User u The uth user
C Total number of requests for all users
Ru The number of requests belongs to uth user
Reqru rth request of the uth user
BDru Budget of rth request belongs to uth user
DLru Deadline of rth request belongs to uth user
AT ru The time at which the rth request belongs to uth user arrives on the system
λru Penalty rate of rth request belongs to uth user
Si ith cloud service offered by SaaS provider
Numi (1t ) Number of VMs allocated to cloud service Si at the 1tth interval
U i ( 1t ) The CPU Utilization of VMs of allocated to cloud service Si at the 1tth interval
Wi (1t ) Number of requests for cloud service Si at the 1tth interval
N Number of initiated VMs
I The total number of cloud services offered by the SaaS provider
L The number of VM types (e.g., L = 3, small, medium, large)
VM Pricel Price of VM type l
initVM Pricel The initiation VM price of type l
VM hour n The duration of time for the nth VM is on
TS n The time spent in setting up the VM n
FT ru The finishing time for user request Reqru
TDru The delay time for request Reqru

(e.g., request rate, type of requests, size of requests, processed
requests, pending requests, and dropped requests). These differ-
ent sets of monitoring information are collected, aggregated, and
stored in a knowledge base for use by other components.
B. Analyzer
The analyzer component is responsible for processing the
information gathered directly from the monitor component. Data
obtained by the monitors (average response time, and request
arrival rate, etc.) are examined by this component to determine
whether adaptive actions are required to guarantee the requested
QoS level. Moreover, the workload analyzer uses the linear
regression model (LRM) to predict future demands in order to deal
with fluctuating demands. If any change is needed, it should trigger
the planner component.
C. Planner
The planner component is the core component of the frame-
work. It determines when and how many VMs should be allocated
to different cloud services in order to find a satisfactory trade-off
between the guaranteed SLA and optimized costs. In this paper, we
use the RL-based method as a decision-maker that uses the pre-
dicted results of an analyzer component to remove or add VMs.
D. Executor
The executor component consists of two subcomponents,
i.e., the load balancer and VM manager. The load balancer will
receive all of the incoming requests from users, and distribute
them to a suitable VM according to the load-balancing policies
(e.g., round robin, and random, etc.). Because the VM Manager
deals directly with virtual resources in the IaaS provider, it is
responsible for actually executing the actions decided (e.g., scale in,
scale out) by the planner. Finally, it allocates or de-allocates VMs.
4.2. Problem formulation
In this section, we describe equations and notations used in the
proposed approach. As shown in Table 2, let U be the number of
users (User = {User 1 , User 2 , User 3 , . . . , User U }), and let each user
u have Ru requests (Requ = {Req1u , Req2u , Req3u , . . . , ReqRuu }) for the
use of cloud services offered by the SaaS Provider (i.e., the total
U r
number of requests is equal to C = u=1 Ru ). Let Requ denote the
rth request belonging to the uth user. Every request Reqru includes
items having the maximum cost budget offered by user u to the
SaaS provider for the requested cloud services (BDru ), the maximum
time (deadline) that the user would prefer to wait for the result
(DLru ), the time at which the request arrives on the system (AT ru ) and
the penalty rate for the requested cloud service, which depends
on the request type (λru ) (Reqru = {BDru , DLru , AT ru , λru }). Let I be the
total number of cloud services offered by the SaaS provider, and Si
indicates the cloud service with ID i (i.e., S = {S1 , S2 , S3 , . . . , SI }).
Let Numi (1t ) denote the number of VMs needed for executing the
cloud service Si at the 1tth interval, Ui (1t ) is the CPU utilization
of VMs that are allocated to cloud service Si at the 1tth interval,
and Wi (1t ) is the number of requests for cloud service Si at the
1tth interval. Let N be the number of initiated VMs (VM =
{VM 1 , VM 2 , VM 3 , . . . , VM N }) offered by IaaS provider to a SaaS
provider for executing cloud services. Every IaaS provider offers L
types of VM (e.g., L = 3, small, medium, large Amazon EC2), where
each VM type l has VM Price1 a price. Let Total Cost be the total cost
incurred by the SaaS provider to serve all requested cloud services,
and as described in Eq. (2), it includes the VM Costs and the Penalty
Costs:
Total Cost = VM Costs + Penalty Costs. (2)
The VM Costs is the total cost for all VMs, and is expressed by Eq.
(3):
N

VM Costs = VM Cost n . (3)
n=1
For each nth VM, the VM Cost n depends on the VM price of type
l (VM Price1 that is costs for the SaaS provider to use a VM for
the customer request per hour and it measured according to $/h),
the duration of time for which the VM is on (VM hour n ), the start-
up time of the nth VM (TS n ), and the initiation VM price of type
1 (initVM Price1 that is costs for initiating a VM of type l and it
measured according to $/h) and it is expressed by Eq. (4):
VM Cost n = (VM Price1 × VM hour n )
+ (initVM Price1 × TSn ); ∀ n ∈ N ; l ∈ L. (4)
The Penalty Costs is the total penalty cost for all user requests, and
is expressed by Eq. (5):
Ru
U 

Penalty Costs = Penalty Cost (Reqru ). (5)
u =1 r =1
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210
Table 3
Decision table corresponding to proposed MDP.
U (t ) > Uupper-threshold Ulower-threshold < U (t ) < Uupper-threshold
State (t) Over-Utilization Normal-Utilization
Action (t + 1) INC VM (Scale out) No Operation
Fig. 4. The proposed Markov decision process with three states.
An SLA violation occurs when a SaaS provider fails to predefine
criteria (i.e., service level objectives (SLO)) that meet in the user
SLA. In this paper, we calculate the difference between the value
of the response time defined in the user SLA (i.e., deadline) and
the actually experienced response time (i.e. finish time) for user
request Reqru , which is expressed by Eq. (6):
FT ru − DLru FT ru > DLru

TDru = (6)
0 otherwise
TDru indicates the delay time, FT ru and DLru are the finishing time
and deadline time for the user request Reqru , respectively. If the
delay time for the user request Reqru is greater than 0, then an SLA
violation occurs, which is expressed by Eq. (7):
TDru > 0

Yes
SLAV (Reqru ) = (7)
No otherwise.
The Penalty Cost for each user request Reqru is expressed by Eq. (8),
where λru is the penalty rate for the requested cloud service that
depend on request type.
Penalty Cost (Reqru ) = λru × TDru ; ∀ ∈ Ru , u ∈ U .
(8)
The objective function is used to minimize the Total Cost using Eq.
(9):
Minimize(Total Cost = VM Costs + Penalty Costs).
(9)
4.3. Proposed algorithm
In this section, we explain in more detail our autonomic
resource provisioning algorithm for cloud services. After deploying
the SaaS provider with various cloud services in the cloud platform,
the proposed algorithm (refer to algorithm 2) runs until there are
no cloud services running in the SaaS provider. This algorithm
follows the control MAPE loop concept and manages the VMs that
are allocated to each cloud service Si in each time interval. First, the
algorithm boots an appropriate number of VMs based on historical
experience using prediction techniques for cloud services from
a pool of VMs offered by IaaS provider (line 1). For every cloud
service Si offered by SaaS provider at the interval 1t (line 4), the
control MAPE loop is regularly executed at specific time intervals,
and adapts the optimal amount of resources corresponding to the
workload of cloud services (lines 5–10).
197
U (t ) < Ulower-threshold
Under-Utilization
DEC VM (Scale in)
4.3.1. Monitoring phase
In this phase, metrics that are related to both the users and
resources parts are monitored by the user monitor and the
resource monitor, respectively (refer to Algorithm 3). In the user
part, for every cloud service Si offered by a SaaS provider, the user
monitor collects metrics of the number of VMs that are leased from
an IaaS provider by the SaaS provider to execute cloud service Si
at the 1tth interval (Numi (1t )), the number of requests to use
the cloud service Si at the 1tth interval (Wi (1t )) (lines 2–3).
In the resources part, the resource monitor observes the CPU
utilization of VMs that are allocated to cloud service Si at the 1tth
interval (Ui (1t )) (lines 4–5). Finally, this information is stored in
a knowledge base for using by other phases.
4.3.2. Analysis phase
The objective of this phase is to process the metrics obtained
directly from the monitoring phase (i.e., Numi (1t ) , Wi (1t ) , Ui
(1t )) in order to predict the number of requests for cloud service Si
in the next interval (∆(t + 1)) by utilizing forecasting techniques.
In this paper, the workload analyzer uses a linear regression model
(LRM) to predict future demands for cloud service Si in order to deal
with fluctuating demands, as shown in Algorithm 4. The generic
form of the linear regression model for cloud service Si at the 1tth
interval expressed by Eq. (10):
Yt +1 = β1 + β2 × Xt (10)
where t is indexes the sample observation and Yt is the workload
(i.e., the number of requests for cloud service Si ) and Xt is actually
the value of the time stamp at the moment that each sample was
taken. Also, we assume n is the total number of observed samples.
Therefore, the β1 , β2 determined by solving a linear regression
equation based on ordinary least squares, which is expressed by
Eqs. (12) and (13):
Xt2
   
Yt − Xt Xt Y t
β1 =  2 (11)
Xt2

n − Xt
n

Xt Yt −

Xt

Yt
β2 = 2 . (12)

Xt2

n − Xt
4.3.3. Planning phase
In this phase, we propose an RL-based method in order to
obtain the optimal action. RL-based methods can typically be
modeled using a MDP. A MDP can usually be represented with the
number of finite states and transitions between them. We define
the system states as a MDP to simply model the system, as shown in
Fig. 4. It consists of three states: normal-utilization, under-utilization
and over-utilization. These states indicate that the resource
provisioning fits the performance requirement, resources are over-
provisioned, and resources are under-provisioned, respectively.
The state can be inferred from the CPU utilization, if the CPU
utilization is less than the lower threshold, the system state is
considered under-utilization and if the CPU utilization is greater
than the upper threshold, the system state is considered over-
utilization. Otherwise, the system state is considered normal-
utilization. Furthermore, the decision table corresponding to
proposed MDP to choose one of the actions (i.e., scale in, scale out,
No-op) based on current system state is shown in Table 3.
198 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Table 4 Table 6
Decision table corresponding to proposed MDP. Decision table corresponding to proposed MDP.
Q (State/Action) Scale-in No-action Scale-out Q updated (State/Action) Scale-in No-action Scale-out
Over-Utilization 0 0 0 Over-Utilization 0.723 0.217 0.060
Normal-Utilization 0 0 0 Normal-Utilization 0.385 0.205 0.410
Under-Utilization 0 0 0 Under-Utilization 0.115 0.131 0.754

Table 5 next time interval is determined based on look up the next state
Decision table corresponding to proposed MDP. into the updated Q -values table (lines 17–20) as shown in Table 6.
R (State/Action) Scale-in No-action Scale-out

Over-Utilization 0.7 0.2 0.1 4.3.4. Execution phase

Normal-Utilization 0.1 0.8 0.1
Under-Utilization 0.1 0.2 0.7
According to Algorithm 5, Firstly, the Q -values table consist of
the value of Q (s, a) which is initialized by zero (line 4), as shown
in Table 4.
The CPU utilization of the cloud service Si at the time t (i.e. Ui (t ))
is calculated (line 8), and then based on it is determined the current
state (i.e. st ) (lines 9–11). Based on the current state is selected an
action using Table 3 (i.e. at ) and applied it to environment, then, RL-
agent receives a response from the environment which expresses
the value of the (state, action) pair (i.e. r (st , at )) and may be results
in a state transition to the new state (i.e., st +1 ) (lines 12–13). We
define the reward function (R(s, a)) as the transition matrix of the
proposed MDP, as shown in Table 5.
The Q -values table can be updated each time an action is
applied to the system based on Eq. (1) (line 14) and RL-agent
learns through the effect of applying the actions on the cloud
environment. Finally, the optimal action for cloud service Si in the
In this phase, the adoption actions (e.g., scale in/out) are
determined in the planning phase, and are executed in the
execution phase by the VM manager component in order to realize
efficient resource provisioning and achieve a trade-off between
meeting the SLA and minimizing the cost. The VM manager create
a new VM for cloud service Si based on Table 8 (e.g., the small VM
to cloud service1 and the large VM to cloud service3) and release
latest initiated VM (i.e., the VM with lowest CPU utilization) for
scale out and scale in actions, respectively (line 3–5), as shown
in Algorithm 6. Also, the load balancer dispatch requests for cloud
service Si at the 1tth interval to suitable VMs according to the best-
fit strategy by sorting requests based on deadline in ascending and
VMs based on CPU utilization in descending (line 6).
5. Performance evaluation
In this section, we present an evaluation of the proposed
approach discussed in the previous section. First, we describe the
simulation settings and performance metrics, and then we present
and discuss experimental results.
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 199

Table 7 Table 8
All types of virtual machines. Different settings for cloud services.
VM type Extra large Large Medium Small Type service Edition VM type Max user

Core 8 4 2 1 Cloud service1 Standard Small 100

CPU (MIPS) 4000 2000 1000 500 Cloud service2 Standard, enterprise Medium 200
RAM (GB) 15 7.50 3.75 1.7 Cloud service3 Enterprise, professional Large 350
Disk (GB) 1690 850 410 160 Cloud service4 Professional Extra large 500
VM price ($/h) 2.560 1.280 0.640 0.320
VM initiation price ($/h) 3.200 1.550 0.720 0.450
VM initiation time (min) 9 7 5 2
Space Center WWW server in Florida, from August 1 to 31, 1995.
The NASA traces repeats patterns in certain periods. These traces
represent realistic load variations over time observed from well-
5.1. Experimental setup
known websites. The two real-world workload patterns are shown
in Fig. 5(a), (b), respectively. Furthermore, we used four cloud
The experiments presented in this section were developed services in the experiments. Table 8 shows the different settings
using Cloudsim toolkit [46], which is a framework for modeling and for the cloud services.
simulating the cloud computing infrastructures and services. We According to Table 8, ‘‘Edition’’ is defined as the version of
can assume four types of VMs that are offered by an IaaS provider: cloud service that is offered to users (e.g., standard, professional,
small, medium, large, and extra-large. The configuration details of and enterprise). ‘‘Max User’’ is defined as the maximum number
four different types of VMs where they have different capacities of users who can use the cloud service and restricted by the
and costs, as shown in Table 7. type of the cloud service edition. Furthermore, the VMs have
To evaluate our approach, we used two real world workload different capability to serve different numbers of users. The VM
traces: the ClarkNet traces [47], the NASA traces [48]. These with higher capabilities (e.g., ‘‘Extra large’’) is suitable for higher
workload traces have different characteristics which make the edition with more users (e.g., professional). Also, the time intervals
results more realistic and reliable to be used in real cloud are considered in 5-min intervals. Thus, each day includes 288 time
applications. The ClarkNet traces was extracted from a web server intervals. In this paper, the time simulation lasts 6 h and includes
for the Metro Baltimore–Washington DC area, from August 1 to 31, 72 time intervals. Also, we assume that the general scheduling
1995. The ClarkNet traces presents a behavior that has similarity policy is time-shared scheduling.
over time, and indicate periods with higher demand. The NASA We compare our approach with three baseline strategies. The
traces was contained all HTTP requests to the NASA Kennedy first strategy is referred to as ‘‘Cost-aware (LRM)’’, which is an auto-
200 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Fig. 5. Workload patterns: (a) ClarkNet workload, (b) NASA workload.

scaling approach based on workload prediction and it uses the
linear regression model for dynamic resource provisioning [37].
The second strategy is referred to as ‘‘Cost-aware (ARMA)’’, which
uses a second-order ARMA method filter [34] to predict the
workload and adjusts resources that are allocated to users ahead
of time. The third strategy is referred to as ‘‘DRPM’’ [26], which is
a multi-agent based strategy for dynamic resources provisioning
and monitoring. The DRPM strategy includes a global utility agent
for the management of all of the system resources and a set of
local utility agents that are assigned to each user. Finally, our
approach is referred to as a ‘‘Proposed’’ approach. The reason of
choose of these strategies for comparison this is that the firstly,
these follow the control MAPE loop and the secondly, these are
proactive, i.e., attempt to predict the amount of resources at any
given time, to anticipate unwanted events.
5.2. Performance metrics
We applied the following metrics for a comparison of our
approach with other strategies:
Utilization: The CPU utilization of the cloud service Si at the 1tth
interval is defined as the ratio between the allocated Machine
Instruction Per Second (MIPS) of VMs for requests of cloud service
Si and the total MIPS that is potentially offered by VMs at the 1tth
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210
interval, and is expressed by Eq. (13):
Allocated MIPS (1t )
USi (1t ) = . (13)
Total MIPS (1t )
VMs allocated: This metric is defined as the number of VMs
allocated to a cloud service Si at the 1tth interval.
SLA Violation: The percentage SLA violation of the cloud service Si
at the 1tth interval for user request Reqru of cloud service Si at the
1tth interval is expressed by Eq. (14):
C
1
SLAV Si (1t ) = SLAV (Reqru )
C c =1
 r r during demand surges. This problem will lead to SLA violations,
 FT u − DLu FT ru > DLru
SLAV (Requ ) = DLu − AT ru
r r

0 otherwise
where AT ru indicates the time at which the user request c arrives on
the system, FT ru and DLru are the finishing time and deadline time
for the user request Reqru , respectively.
Total Cost: This metric is defined as the total cost incurred by the
SaaS provider to serve all of the requests for cloud service Si at the
1tth interval, and is expressed by Eq. (16):
Total Cost Si (1t ) = VM CostsSi (1t ) + Penalty CostsSi (1t ).
Profit: This metric is defined as the profit gained by the SaaS
provider to serve all requests for cloud services, and is expressed
by Eq. (18):
Profit Si (1t ) = Budget Si (1t ) − Total Cost Si (1t )
U K

Budget Si (1t ) = BDku
u=1 k=1
where k is the number of requests that are finished and BDku is the
budget of kth requests belonging to the uth user at the 1tth time
interval.
5.3. Experimental results
We evaluated the performance of the proposed approach under
two real world workload traces for metrics that were discussed in
the previous subsection. For the sake of accuracy, we performed
each experiment 20 times using different cloud services, and we
report the average of the results.
5.3.1. Evaluation based on Metrics
Fig. 6 shows the CPU utilizations of the four approaches for
both ClarkNet and NASA workloads at each interval. The average
CPU utilizations under both workloads for the proposed, cost-
aware (LRM), cost-aware (ARMA), and the DRPM are 95%, 83%,
78%, and 71%, respectively, as shown in Table 9. From the results,
we observe that the proposed, cost-aware (ARMA) and cost-aware
(LRM) approaches are able to utilize resources more fully, and
that the DRPM wastes more resources in both workloads for most
intervals. This is because of the conservative and static allocation
of VMs by the DRPM approach. Also, cost-aware (LRM) due to
accuracy prediction compared with cost-aware (ARMA), it has
outperforms in term of the CPU utilization. In some time intervals,
the CPU utilizations are more than 100%; this is because the
SaaS Provider does not have enough resources to process all the
incoming requests for the cloud services at those time intervals and
will lead to under-provisioning state (i.e., over-utilization state)
which causes SLA violations.
Fig. 7 shows the trends of the allocated VMs of the four
approaches for both ClarkNet and NASA workloads at each interval.
201
Because of statically provisioning of VMs by the DRPM, cost-
aware (LRM) and the cost-aware (ARMA) approaches compared
with the proposed approach, there will be more idle resources
for most intervals, and therefore, will lead to under-utilization
state (i.e., over-provisioning state). Although this will reduce SLA
violations however, the will incurs an unnecessary cost to the SaaS
provider under light workloads and the CPU utilization will be
reduced Also, the DRPM consumes the most resources compared
to the other approaches, as shown in Table 10.
Fig. 8 shows the percentage of SLA violations for four
approaches at each interval. As mentioned above, the under-
(14) provisioning of resources (i.e., over-utilization state) cause cloud
services end-users to experience excessive delays, especially
and subsequently, lead to lower profit and fewer users. Therefore,
(15) the DRPM compared with the cost-aware (LRM) and the cost-
aware (ARMA) approaches, because of having more resources for
most intervals, it reduces the SLA violations in both workloads
for all of the intervals. In general, compared with the other
approaches, the proposed approach leads to a 65% reduction in the
number of SLA violations, as shown in Table 11. Although the cost-
aware (LRM) outperforms cost-aware (ARMA) in terms of the CPU
utilization, it is worse than cost-aware (ARMA) in terms of the SLA
violations.
(16) Fig. 9 shows the total cost of the four approaches for both
ClarkNet and NASA workloads at each interval. As mentioned
above, the total cost depends on the cost of SLA violation (i.e., the
Penalty Cost, Fig. 10) and the cost of allocated VMs (i.e., the VM
Cost, Fig. 11). Therefore, the proposed approach outperforms other
(17) approaches by saving about 50% of the cost in the both ClarkNet
and NASA workloads, as shown in Table 12. Moreover, as shown in
Fig. 8, when the workload is NASA, all approaches having less SLA
(18)
violations compared to the ClarkNet workload, because of there
are regular patterns and predictable inNASA workload at certain
intervals. Therefore, the total cost generated in the case of a NASA
workload is less than that for the ClarkNet workload ones. Also,
the DRPM compared with the cost-aware (LRM) and the cost-
aware (ARMA) approaches, because of having less SLA violations
for most intervals, it reduces the total cost in both workloads for
most intervals.
Fig. 12 shows the profit for the four approaches at each interval.
A SaaS provider can maximize its profit by minimizing the total
cost, which depends on the allocated VMs and number of SLA
violations. Because the proposed approach outperforms other
approaches by minimizing the total cost, the proposed approach
therefore outperforms other approaches by saving about 50% of the
cost in the case of both ClarkNet and NASA workloads. Also, the
cost-aware (LRM) compared with the other approaches, because
of having more resources and the higher SLA violations, it reduces
the profit of SaaS provider in both workloads for most intervals, as
shown in Table 13.
Based on the above results, we compared the CPU utilization,
allocated VMs, SLA violations, total cost and profit of the proposed
and three approaches under both ClarkNet and NASA workloads.
We observed that the proposed approach reduces the total cost by
up to 50%, the number of SLA violations by up to 60%, the allocated
VMs by up to 16% and increases the CPU utilization by up to 12%,
the profit by up to 11% compared with the other approaches, as
shown in Table 14 and Table 15.
5.3.2. Overhead of proposed approach
Although elasticity is one of the main characteristics of cloud
computing environment to resource provisioning dynamically ac-
cording to demand, but on the other hand, this dynamic provision-
ing process is that it takes time. The Start-up time represents the
length of time between requesting scaling up/down actions and
202 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Fig. 6. CPU utilization at different intervals for (a) ClarkNet workload, (b) NASA workload.

Table 9
The average CPU utilization for ClarkNet and NASA workloads.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 97.21% 88.73% 82.57% 77.30%

NASA workload 93.91% 77.88% 75.17% 66.26%

Table 10
Average allocated VMs for ClarkNet and NASA workloads.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 14.6875 16.5590 17.6493 18.7291

NASA workload 10.9583 13.6527 13.7430 15.6527
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 203

Fig. 7. Allocated VMs at different intervals for (a) ClarkNet workload, (b) NASA workload.

Table 11
Average percentage of SLA violations for ClarkNet and NASA workloads.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 0.94% 5.78% 5.08% 4.47%

NASA y workload 0.88% 3.20% 3.03% 2.17%

Table 12
Average total cost for ClarkNet and NASA workloads.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 3.6202 16.9930 15.3400 13.9914

NASA workload 2.4835 7.0499 6.7725 5.2774
204 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Fig. 8. Percentage of SLA violation at different intervals for (a) ClarkNet workload, (b) NASA workload.

Table 13
Average profit of SaaS provider for ClarkNet and NASA workloads.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 70.7221 57.3494 59.0023 60.3509

NASA workload 50.9831 46.4167 46.6940 48.1891

actual resource provisioning/de-provisioning time [49]. The start- period for start-up time because new machines generally take be-
up time is including allocating IP address, configuring the OS, and tween 5 and 10 min to be operational. In this paper, we consider
booting the OS, and it depends on data center location, VM type, the start-up time between 2 and 9 min based on VM type, as shown
and the number of VMs. The RightScale [50] recommends a 15 min- in Table 4. Since the goal of this paper is to minimize the total cost,
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 205

Fig. 9. Total cost at different intervals for (a) ClarkNet, (b) NASA workloads.

Table 14
Comparison of four approaches for ClarkNet workload.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

Utilization (%) 97.21% 88.73% 82.57% 77.30%

SLA violation (%) 0.94% 5.78% 5.08% 4.47%
Total cost ($) 3.6202 16.9930 15.3400 13.9914
Profit ($) 70.7221 57.3494 59.0023 60.3509
Allocated VMs (#VMs) 14.6875 16.5590 17.6493 18.7291
206 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Fig. 10. Penalty cost at different intervals for (a) ClarkNet, (b) NASA workloads.

Table 15
Comparison of four approaches for NASA workload.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

CPU utilization (%) 93.91% 77.88% 75.17% 66.26%

SLA violation (%) 0.88% 3.20% 3.03% 2.17%
Total cost ($) 2.4835 7.0499 6.7725 5.2774
Profit ($) 50.9831 46.4167 46.6940 48.1891
Allocated VMs (#VMs) 10.9583 13.6527 13.7430 15.6527

the VM initiation cost as overhead of proposed algorithm is con- on the start-up time of the VM (initVM Time), and the VM initiation
sidered. For each VM, the VM initiation cost (initVM Cost) depends price of type 1 (initVM Pricel that is costs for initiating a VM of type
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 207

Fig. 11. VM cost at different intervals for (a) ClarkNet, (b) NASA workloads.

1 and it measured according to $/h) and it is expressed by Eq. (19): the dynamic allocation of VMs according to the workload changes
needs a greater number of scaling.
initVM Cost = initVM Pricel × initVM Time (19)

Fig. 13 shows the cumulative scaling cost (i.e., the VM initiation
cost) of the four approaches for both ClarkNet and NASA workloads
at each interval. From the results, it was observed that the DRPM
approach outperforms the others for both workloads at each
interval, as shown in Table 16; however, the proposed approach
fails to satisfy the VM initiation cost. This outperformance is
because of the static allocation of VMs by the DRPM approach, as
well as having a lower number of scaling (i.e., the numbers of VMs
allocated/deallocated); while the proposed approach because of
6. Conclusion and future work
In this paper, we investigate the problem of dynamic resource
provisioning for cloud services, and we present a hybrid resource
provisioning approach based on a combination of the autonomic
computing and the reinforcement learning. In order to implement
the proposed approach, we present a resource provisioning
framework that supports the control MAPE loop. The proposed
approach should dynamically adapt to uncertainties, workload
208 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210

Fig. 12. Profit of SaaS provider at different intervals for (a) ClarkNet (b) NASA workloads.

Table 16
Average Scaling Cost for ClarkNet and NASA at each interval.
Proposed Cost-aware (LRM) [37] Cost-aware (ARMA) [34] DRPM [26]

ClarkNet workload 0.5684 0.2862 0.3717 0.1651

NASA workload 0.5263 0.1517 0.0931 0.0931

spikes and should deal with the undesirable states of over-
provisioning and under-provisioning. Our approach is regularly
executed at specified time intervals and we vary the optimal
amount of resources corresponding to the workload of cloud
services. We evaluated the performance of the proposed approach
under real world workload traces for different metrics using the
cloudsim toolkit, and compared the results obtained with those
of the other approaches. The results show that the proposed
approach increases the resource utilization and decreases the total
cost, while avoiding SLA violations. In future work, we plan to
explore: integration of the proposed approach with admission
control strategies, using the proposed approach for multi-tier
 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210 209

Fig. 13. The overhead of cumulative scaling costs at different intervals for (a) ClarkNet (b) NASA workloads.

cloud applications and extension of planning phase of the control
MAPE loop by using of the fuzzy logic.
References
[1] R. Buyya, C. Vecchiola, S.T. Selvi, Mastering Cloud Computing: Foundations and
Applications Programming, Newnes, 2013.
[2] K. Chandrasekaran, Essentials of Cloud Computing, CRC Press, 2014.
[3] S.S. Manvi, G.K. Shyam, Resource management for Infrastructure as a Service
(IaaS) in cloud computing: A survey, J. Netw. Comput. Appl. 41 (2014) 424–440.
[4] S. Mustafa, B. Nazir, A. Hayat, S.A. Madani, Resource management in cloud
computing: Taxonomy, prospects, and challenges, Comput. Electr. Eng. (2015).
[5] M.P. Papazoglou, W.J. Van Den Heuvel, Service oriented architectures:
approaches, technologies and research issues, VLDB J. 16 (3) (2007) 389–415.
[6] S. Rodríguez, D.I. Tapia, E. Sanz, C. Zato, F. de la Prieta, O. Gil, Cloud computing
integrated into service-oriented multi-agent architecture, in: Balanced
Automation Systems for Future Manufacturing Networks, Springer, Berlin,
Heidelberg, 2010, pp. 251-–259.
[7] B. Piprani, D. Sheppard, A. Barbir, Comparative analysis of SOA and cloud
computing architectures using fact based modeling, in: On the Move
to Meaningful Internet Systems: OTM 2013 Workshops, Springer, Berlin,
Heidelberg, 2013, pp. 524–533.
[8] J. Varia, Architecting applications for the Amazon cloud, in: Cloud Computing:
Principles and Paradigms, Wiley Press, New York, USA, 2011, pp. 249–274.
[9] N.R. Herbst, S. Kounev, R. Reussner, Elasticity in cloud computing: What it is,
and what it is not, in: Proceedings of the 10th International Conference on
Autonomic Computing, ICAC 2013, San Jose, CA, 2013.
[10] E.F. Coutinho, F.R. de Carvalho Sousa, P.A.L. Rego, D.G. Gomes, J.N. de
Souza, Elasticity in cloud computing: a survey, Ann. Telecommun.—Ann.
Télécommun. (2015) 1–21.
210 M. Ghobaei-Arani et al. / Future Generation Computer Systems 78 (2018) 191–210
[11] T. Lorido-Botran, J. Miguel-Alonso, J.A. Lozano, A review of auto- scaling
techniques for elastic applications in cloud environments, J. Grid Comput. 12
(4) (2014) 559–592.
[12] M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R. Katz, A. Konwinski, G. Lee, D.
Patterson, A. Rabkin, I. Stoica, M. Zaharia, A view of cloud computing, Commun.
ACM 53 (4) (2010) 50–58.
[13] S. Singh, I. Chana, Q-aware: Quality of service based cloud resource
provisioning, Comput. Electr. Eng. 47 (2015) 138–160.
[14] B. Jacob, R. Lanyon-Hogg, D.K. Nadgir, A.F. Yassin, A Practical Guide to the IBM
Autonomic Computing Toolkit, IBM Redbooks, 2004.
[15] M. Maurer, I. Breskovic, V.C. Emeakaroha, I. Brandic, Revealing the MAPE
loop for the autonomic management of Cloud infrastructures, in: 2011
IEEE Symposium on Computers and Communications, (ISCC), IEEE, 2011,
pp. 147–152.
[16] S. Russel, P. Norvig, Artificial Intelligence: A Modern Approach, EUA: Prentice
Hall, 2003.
[17] L.P. Kaelbling, M.L. Littman, A.W. Moore, Reinforcement learning: A survey, J.
Artificial Intelligence Res. (1996) 237–285.
[18] R. Rajavel, M. Thangarathanam, Adaptive Probabilistic Behavioural Learning
System for the effective behavioural decision in cloud trading negotiation
market, Future Gener. Comput. Syst. 58 (2016) 29–41.
[19] M.C. Huebscher, J.A. McCann, A survey of autonomic computing— degrees,
models, and applications, ACM Comput. Surv. (CSUR) 40 (3) (2008) 7.
[20] T. Ritter, B. Mitschang, C. Mega, Dynamic provisioning of system topologies
in the cloud, in: Enterprise Interoperability V, Springer, London, 2012,
pp. 391–401.
[21] F. Pop, M. Potop-Butucaru, ARMCO: Advanced topics in resource management
for ubiquitous cloud computing: An adaptive approach, Future Gener. Comput.
Syst. 54 (2016) 79–81.
[22] M. Maurer, I. Brandic, R. Sakellariou, Adaptive resource configuration for
Cloud infrastructure management, Future Gener. Comput. Syst. 29 (2) (2013)
472–487.
[23] M. Koehler, An adaptive framework for utility-based optimization of scientific
applications in the cloud, J. Cloud Comput. 3 (1) (2014) 1–12.
[24] V.C. Emeakaroha, I. Brandic, M. Maurer, S. Dustdar, Cloud resource provision-
ing and SLA enforcement via LoM2HiS framework, Concurr. Comput.: Pract.
Exper. 25 (10) (2013) 1462–1481.
[25] Foundation of Self-governing ICT Infrastructures (FoSII), Available
http://www.infosys.tuwien.ac.at/linksites/FOSII/index.htm.
[26] M. Al-Ayyoub, Y. Jararweh, M. Daraghmeh, Q. Althebyan, Multi- agent based
dynamic resource provisioning and monitoring for cloud computing systems
infrastructure, Cluster Comput. 18 (2) (2015) 919–932.
[27] E. Casalicchio, L. Silvestri, Mechanisms for SLA provisioning in cloud-based
service providers, Comput. Netw. 57 (3) (2013) 795–810.
[28] F. Bahrpeyma, H. Haghighi, A. Zakerolhosseini, An adaptive RL based
approach for dynamic resource provisioning in Cloud virtualized data centers,
Computing (2015) 1–26.
[29] S. Islam, J. Keung, K. Lee, A. Liu, Empirical prediction models for adaptive
0020resource provisioning in the cloud, Future Gener. Comput. Syst. 28 (1)
(2012) 155–162.
[30] E. Barrett, E. Howley, J. Duggan, Applying reinforcement learning towards
automating resource allocation and application scalability in the cloud,
Concurr. Comput.: Pract. Exper. 25 (12) (2013) 1656–1674.
[31] J. Liu, Y. Zhang, Y. Zhou, D. Zhang, H. Liu, Aggressive resource provisioning for
ensuring QoS in virtualized environments, IEEE Trans. Cloud Comput. 3 (2)
(2015) 119–131.
[32] H.R. Qavami, S. Jamali, M.K. Akbari, B. Javadi, Dynamic resource provisioning
in cloud computing: A Heuristic Markovian approach, in: Cloud Computing,
Springer International Publishing, 2014, pp. 102–111.
[33] S. Muppala, G. Chen, X. Zhou, Multi-tier service differentiation by coordinated
learning-based resource provisioning and admission control, J. Parallel Distrib.
Comput. 74 (5) (2014) 2351–2364.
[34] N. Roy, A. Dubey, A. Gokhale, Efficient auto-scaling in the cloud using
predictive models for workload forecasting, in: 2011 IEEE International
Conference on Cloud Computing, (CLOUD), IEEE, 2011, pp. 500–507.
[35] S. Misra, P.V. Krishna, K. Kalaiselvan, V. Saritha, M.S. Obaidat, Learning
automata-based QoS framework for cloud IaaS, IEEE Trans. Netw. Serv.
Manage. 11 (1) (2014) 15–24.
[36] C.Z. Xu, J. Rao, X. Bu, URL: A unified reinforcement learning approach for
autonomic cloud management, J. Parallel Distrib. Comput. 72 (2) (2012)
95–105.
[37] J. Yang, C. Liu, Y. Shang, B. Cheng, Z. Mao, C. Liu, L. Niu, J. Chen, A cost-aware
auto-scaling approach using the workload prediction in service clouds, Inf.
Syst. Front. 16 (1) (2014) 7–18.
[38] J.O. Kephart, D.M. Chess, The vision of autonomic computing, Computer 36 (1)
(2003) 41–50.
[39] M. Verma, G.R. Gangadharan, N.C. Narendra, R. Vadlamani, V. Inamdar, L.
Ramachandran, R.N. Calheiros, R. Buyya, Dynamic resource demand prediction
and allocation in multi-tenant service clouds, Concurr. Comput.: Pract. Exper.
(2016).
[40] W. Li, F.C. Delicato, P.F. Pires, A.Y. Zomaya, Energy-efficient task allocation with
quality of service provisioning for concurrent applications in multi-functional
wireless sensor network systems, Concurr. Comput.: Pract. Exper. 26 (11)
(2014) 1869–1888.
[41] K.L.A. Yau, P. Komisarczuk, P.D. Teal, Reinforcement learning for context
awareness and intelligence in wireless networks: review, new features and
open issues, J. Netw. Comput. Appl. 35 (1) (2012) 253–267.
[42] N. Verstaevel, C. Régis, M.P. Gleizes, F. Robert, Principles and experimentations
of self-organizing embedded agents allowing learning from demonstration in
ambient robotics, Future Gener. Comput. Syst. (2016).
[43] J. Wu, X. Xu, P. Zhang, C. Liu, A novel multi-agent reinforcement learning
approach for job scheduling in Grid computing, Future Gener. Comput. Syst.
27 (5) (2011) 430–439.
[44] J. Kober, J. Peters, Reinforcement learning in robotics: A survey, in: Reinforce-
ment Learning, Springer, Berlin, Heidelberg, 2012, pp. 579–610.
[45] S. Cho, S. Fong, Y.W. Park, K. Cho, Simulation framework of ubiquitous network
environments for designing diverse network robots, Future Gener. Comput.
Syst. (2016).
[46] R.N. Calheiros, R. Ranjan, A. BelogRLzov, C.A. De Rose, R. Buyya, CloudSim: a
toolkit for modeling and simulation of cloud computing environments and
evaluation of resource provisioning algorithms, Softw. - Pract. Exp. 41 (1)
(2011) 23–50.
[47] Clarknet-http-two weeks of http logs from the clarknet www server.
http://ita.ee.lbl.gov/html/contrib/ClarkNet-HTTP.html (Accessed 15.10.14).
[48] Nasa-http- two months of http logs from the kscnasa www server.
http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html (Accessed 15.10.14).
[49] M.A.N. Bikas, A. Alourani, M. Grechanik, How elasticity property plays an
important role in the cloud: A survey, Adv. Comput. (2016).
[50] Rightscale. set up autoscaling using voting tags.
https://support.rightscale.com/ (Accessed 01.10.14).
Mostafa Ghobaei-Arani received the B.Sc. degree in Soft-
ware Engineering from Kashan University, Iran in 2009,
and M.Sc. degree from Azad University of Tehran, Iran in
2011, respectively. He was honored Ph.D. degree Software
Engineering from Islamic Azad University, Science and Re-
search Branch, Tehran, Iran in 2016. His current research
interests are Distributed Systems, Cloud Computing, Per-
vasive Computing, Big Data, SDN, and IoT.
Sam Jabbehdari currently working as an associated
professor at the department of Computer Engineering in
IAU (Islamic Azad University), North Tehran Branch, in
Tehran, since 1993. He received his both B.Sc. and M.S.
degrees in Electrical Engineering Telecommunication from
Khajeh Nasir Toosi University of Technology, and IAU,
South Tehran branch in Tehran, Iran, respectively. He was
honored Ph.D. degree in Computer Engineering from IAU,
Science and Research Branch, Tehran, Iran in 2005. His
current research interests are Scheduling, QoS, MANETs,
Wireless Sensor Networks and Cloud Computing.
Mohammad Ali Pourmina is associated professor in elec-
trical engineering (Telecommunication) in Science and re-
search Branch, Islamic Azad University, Tehran, Iran and
top professor of this university in 2016. He received his
Ph.D. degree in electrical engineering (Telecommunica-
tion) at Science and research Branch, Islamic Azad Uni-
versity, Tehran, Iran, in 1996 and joined to this university
1996. He has been a member of Iran Telecommunication
research center since 1992. He has performed research in
the areas of packet radio networks and digital signal pro-
cessing systems since 1991. His current research interests
include spread-spectrum systems, cellular mobile communications, indoor wireless
communications, DSP processors and wireless multimedia networks.