Dynamic CPU Resource Provisioning in Virtualized Servers
using Maximum Correntropy Criterion Kalman Filters
Evagoras Makridis∗ , Kyriakos M. Deliparaschos∗ , Evangelia Kalyvianaki† and Themistoklis Charalambous‡
∗ Cyprus University of Technology, Limassol, Cyprus, Email: ep.makridis@edu.cut.ac.cy, k.deliparaschos@cut.ac.cy
† City
University of London, London, United Kingdom, Email: evangelia.kalyvianaki.1@city.ac.uk
‡ Aalto University, Espoo, Finland, Email: themistoklis.charalambous@aalto.fi
Abstract—Virtualized servers have been the key for the ef-
ficient deployment of cloud applications. As the application
demand increases, it is important to dynamically adjust the CPU
allocation of each component in order to save resources for
other applications and keep performance high, e.g., the client
mean response time (mRT) should be kept below a Quality
of Service (QoS) target. In this work, a new form of Kalman
filter, called the Maximum Correntropy Criterion Kalman Filter
(MCC-KF), has been used in order to predict, and hence,
adjust the CPU allocations of each component while the RUBiS
auction site workload changes randomly as the number of clients
varies. MCC-KF has shown high performance when the noise is
non-Gaussian, as it is the case in the CPU usage. Numerical
evaluations compare our designed framework with other current
state-of-the-art using real-data via the RUBiS benchmark website
deployed on a prototype Xen-virtualized cluster.
Index Terms—Resource provisioning, virtualized servers, CPU
allocation, CPU usage, RUBiS, Kalman filter.
I. I NTRODUCTION
Popular applications, such as the Instagram and Dropbox,
use the cloud computing paradigm for their services. Virtu-
alization [1] is one of the fundamental technologies used for
server consolidation in cloud computing. When a physical ma-
chine is virtualized, it is transformed into one or more virtual
execution environments, called virtual machines (VMs), and
their resource allocations must be adjusted online in order to
match their workload needs. Applications can run in isolation
on each VM. Infrastructure as a Service (IaaS) constitutes a
form of cloud computing in which virtual computing resources
are provisioned and utilized over the Internet [2]. IaaS plat-
forms offer automated resource provisioning (AutoScaling) for
adjusting resource allocation based on demand. This makes
IaaS well-suited for enterprises that experience workload vari-
ations. Unexpected workload variations should also be handled
seamlessly without any performance degradation.
Server consolidation is the efficient usage of server re-
sources in order to reduce the total number of physical
machines required. Consolidated applications share server
resources, such as CPU time, memory, network bandwidth
and disk space. To operate efficiently, each application is
allocated with enough resources from the hosting server in
order to meet its performance requirements of Service Level
Objectives (SLOs), as measured, for example, by the high
throughput, availability, and requests mean Response Time
(mRT). However, adjusting the shares of resources of consol-
idated applications while their demands change over time is
978-1-5090-6505-9/17/$31.00
2017
challenging, because (a) the workloads may vary significantly
over time, (b) the workload of some applications may change
tremendously over a very short period of time, and (c) resource
allocation should be optimized for better performance (see, for
example, [3]–[6] and references therein).
While performing virtualization, it is important to ensure
that the demands meet their SLOs. Towards this end, auto-
nomic resource management methods are needed to dynam-
ically allocate resources across virtualized applications with
diverse workload and highly fluctuating workload demands.
Autonomic resource management in a virtualized environment
using control-based techniques has recently gained significant
attention; see [7] for a survey. One of the most common
approaches to control the application performance is by con-
trolling its CPU utilization within the VM; see, for example,
[8] and references therein.
The Kalman filter, adopted in [8], [9], is one of the most
widely-used adaptive filters due to its simplicity, optimality,
and versatility. The traditional Kalman filter is derived under
the minimum mean square error (MMSE) criterion and is
the optimal estimator if the measurement and process noises
are Gaussian. However, in non-Gaussian noise environments,
the Kalman filter uses only second-order signal information,
and hence it is suboptimal. In general, the use of MSE
is desirable if the signals follow a Gaussian distribution.
Otherwise, if the distribution is non-Gaussian, the performance
of the system may degrade considerably and in these cases, a
non-quadratic cost is more desirable than MSE [10]. In order
to handle environments with non-Gaussian disturbances, such
as impulsive noise and Gaussian mixture noise, and improve
the robustness of state estimation, a Maximum Correntropy
Criterion Kalman filter (MCC-KF) was introduced in [11] and
[12]. The MCC-KF adopts the robust Maximum Correntropy1
Criterion (MCC) instead of the MMSE criterion which is
sensitive to large outliers and results in robustness decay of
the Kalman filter in non-Gaussian environments.
Dynamic resource provisioning (DRP) is characterized by
workloads that have sudden and irregular fluctuations. In this
work, motivated by the workload fluctuations in such an
environment, we use the MCC-KF for better state estimation,
1 The correntropy is a similarity measure related to the probability of how
similar two random variables are [13]. Since correntropy is insensitive to
outliers, it is a natural robust adaptation cost in presence of heavy-tailed
impulsive noises.
c IEEE
and hence, a more efficient resource provisioning control
mechanism. More specifically, a new controller, based on the
MCC-KF, is proposed. The proposed controller allocates CPU
resources in virtualized applications and aims at meeting its
SLO, which is to maintain the requests’ mRT below a certain
threshold. Experimental evaluation shows that the proposed
controller dynamically allocates the resources in order to meet
its SLO. Compared to current-state-of-the-art in the literature,
the proposed controller reduces the overall mRT and the
instances for which the SLO is not met, without providing
more resources than necessary. This is achieved due to the
fact that better state prediction is obtained, because the new
controller takes into account the fact that the process noise may
be non-Gaussian and it is characterized by sudden workload
changes.
The remainder of the paper is organized as follows. In
Section II, we provide a simple motivating example and
present related work in the field. In Section III, we provide the
notation, we describe the performance metric being considered
and the model adopted for capturing the dynamics of the
CPU utilization. In Section IV, we introduced the controller
developed for our system. The performance of the controller
is evaluated and compared with other state-of-the-art in Sec-
tion V. Finally, in Section VI we conclude this work and draw
directions for future research.
II. BACKGROUND
A. Motivating example
In this subsection, we provide a simple example with three
single-component applications that can be hosted on a single
physical server machine using DRP, in order to show the
concept of server consolidation.
Traditionally, each application is hosted on a dedicated
server, as shown in the left diagram of Fig. 1. Assuming that
two of the applications have workload requirements such that
the sum of resources for both applications does not exceed the
total available physical resources for a server machine, two
VMs are created, each one hosting an application with fixed
resources allocated as required (as in the middle diagram in
Fig. 1). In this way, both applications are served adequately
and the total resource utilization of the physical machine is
now increased simply by co-locating two running servers.
Consider now the case where the workload in both applications
changes considerably. For example, consider the case in which
resource utilization in VM A increases, while in VM B
it decreases so fewer resources are needed. If there is no
DRP, the application in VM A experiences under-provisioning
resulting in performance degradation. At the same time, the
application in VM B experiences over-provisioning, which
does not affect the performance of the system, but resources
are under-utilized. In the right diagram of Fig. 1, we see
how with DRP, one can dynamically allocate the resources
and even include a third application on the same server. As
a result, applications that would traditionally require three
dedicated physical servers, now they are all served by a
single physical server, thus reducing the resources needed
for the same number and type of requests. Virtualization
further facilitates moving applications from dedicated servers
to dynamically provisioned servers, thus reducing the number
of physical machines, which results in significant energy and
cost savings.
Fig. 1: Motivating example with three single-component appli-
cations that can be hosted on a single physical server machine
using DRP.
In this paper, we use dynamic resource adaptation for
capturing the workload demands change. Our approach adopts
modern virtualization platforms which export a user-level
interface to bound the maximum resource allocation per VM
at runtime.
B. Related work
The application performance can be controlled, by control-
ling its CPU utilization. It has been observed that the appli-
cation response times stay low [14] as long as the utilization
remains below the allocation by a certain threshold. When the
threshold is exceeded, the response times increase dramatically
and, as a result, the performance of the application drops.
In [15] and [16], the authors directly control application
response times through runtime resource CPU allocation using
an offline system identification approach with which they tried
to model the relationship between the response times and the
CPU allocations in regions where it is measured to be linear.
However, as this relationship is application-specific and relies
on offline identification performance models, it cannot be
applied when multiple applications are running concurrently
and it is not possible/easy to adjust to new conditions. This
triggered the need for searching for other approaches.
The authors in [16] and [17] were among the first to connect
the control of the application CPU utilization within the VM
with the response times. The use of control-based techniques
has emerged as a natural approach for resource provisioning in
a virtualized environment. Control-based approaches have de-
signed controllers to continuously update the maximum CPU
allocated to each VM based on CPU utilization measurements.
For example, Padala et al. [18] present a two-layer non-
linear controller to regulate the utilization of the virtualized
components of multi-tier applications. Kalyvianaki et al. [8],
[9] formulate the CPU allocation problem as a tracking one
and propose adaptive Kalman-based controllers to track and
maintain the CPU utilization to a user-defined threshold. Even
though, the Kalman filter provides an optimal estimate if
the noise is Gaussian, it may perform poorly if the noise
characteristics are different. To account for uncertainties in
the system model and noise statistics, Charalambous et al. [19]
propose the use of an H∞ controller in order to minimize the
maximum error caused by the uncertainties in the model. This
type of controller showed improved performance in saturation
periods and sudden workload changes, but it requires tuning
an extra parameter for achieving the desired performance.
Multi-Input-Multi-Output (MIMO) feedback controllers
have also been considered; see, for example, [4] and [9]. These
controllers make global decisions by coupling the resource
usage of all components of multi-tier server applications. In
addition, the resource allocation problem across consolidated
virtualized applications under conditions of contention have
been considered in [18], [20]: when some applications demand
more resources than physically available, then the controllers
share the resources among them, while respecting the user-
given priorities.
In another line of research, researchers used neuro-fuzzy
control for controlling CPU utilization decisions of virtual
machines level controllers. For example, Sithu et al. [21] use
CPU load profiles to train the neuro-fuzzy controller to only
predict the usage with 100% allocation, without having any
CPU allocation scheme to train on. Deliparaschos et al. [22],
instead, use data from established controllers, such as [8] and
[19], to train their neuro-fuzzy controller.
III. N OTATION AND P RELIMINARIES
A. Notation
R and R+ represent the real and the nonnegative real
numbers, respectively. Vectors are denoted by small letters,
matrices are denoted by capital letters, and sets by calligraphic
capital letters. AT and A−1 denote the transpose and inverse
of matrix A respectively. By I we denote the identity matrix.
âk|k−1 and âk|k denote the a priori and a posteriori estimates
of random vector ak for time instant k. Pk denotes the matrix
P at time instant k. E{·} represents the expectation of its
argument.
B. Performance metric
One of the most widely used metrics for measuring server
performance is the client mean request response times (mRT).
It is difficult to predict the values of the mRT of server
applications across operating regions and different applications
and workloads. However, it is known to have certain charac-
teristics [14]. More specifically, its values can be divided into
three regions (see, e.g., [8], [9], [19]):
(a) when the application has abundant resources and, there-
fore, all requests are served as they arrive and the
response times are kept low;
(b) when the utilization approaches 100% (e.g. around 70-
80% on average) the mRT increases above the low values
from the previous region, due to the fact that there
are instances in which the requests increase abruptly,
approaching 90-100%;
(c) when resources are scarce and very close to 100%, since
requests compete for limited resources, they wait in the
input queues for long and, as a result, their response times
increase dramatically to relatively high values.
In this work, the response time of every type of request was
captured calculating the time difference between the request
and its response, as Fig. 2 shows. All requests were issued to
our RUBiS cluster and specifically to the Web Server, through
the Client Emulator that was deployed on a seperate physical
machine. When all requests were completed, a mean value
of response times, in a time interval of 1s, was calculated in
order to have an image of mRT over time.
Fig. 2: Request-to-response path.
To maintain a good server performance, the operators try
to keep the CPU utilization below 100% of the machine
capacity by a certain value, which is usually called headroom.
Headroom values are chosen such that they form the boundary
between the second and the third mRT regions. At such
values the server is well provisioned and response times
are kept low. If the utilization exceeds the boundary due to
increased workload demands, operators should increase the
server resources.
Firstly, we measure the server’s performance when 100%
of resources is provisioned, without any controller adjusting
the allocation of resources, in order to extract what is the
required headroom. In this work, we consider a Browsing Mix
workload type, in order to specify the server’s performance
while the number of clients varies. Fig. 3 shows the mean
response times (mRT) when the number of clients increases
in steps of 200, until the mRT grows above 1s.
2500
Client mRT
2000
1500
mRT (ms)
1000
500
0
200 400 600 800 1000 1200 1400 1600 1800 2000
Number of Clients
Fig. 3: Mean Response Times (mRT) for different workloads.
It is evident that for more than 1200 clients, the mRT grows
fast and beyond 1700 the SLO is violated.
Initially, as the number of clients increases, the mRT stays
low. However, when the number of 1200 clients is exceeded,
the mRT starts to grow above the low values. Specifically, the
QoS target of 1s is reached when the number of clients that
simultaneously issuing requests to the server is 1700.
 Fig. 4 shows the average CPU usage per component while
the number of clients increases. As shown in this figure,
the database server demand is lower than the web server’s
one with the same number of clients. When, the number of
clients exceeds 1400, the web server’s CPU usage becomes
the bottleneck and even though the database server does not
use 100% of its resources, it remains (almost) constant. Hence,
it is important to establish the required resources for all the
involved components comprising the requests.
100
Web Server
Database Server
80
Average CPU usage %
60
40
20
0 ak+1 = max 0, min{(1 + h)x̂k+1|k , amax } .
200 400 600 800 1000 1200
Number of Clients
Fig. 4: Average CPU usages per component for different
workloads.
C. System model
For this paper, we assume a two-tier server application
composed of 2 components. Each component runs on a differ-
ent VM. However, the allocations are predicted and adapted
on one component while the other component has 100%
of its resources available. The time-varying CPU utilization
per component is modeled as a random walk given by the
following linear stochastic difference equation [8], [9], [19],
[22]:
xk+1 = xk + wk , (1)
where xk ∈ R+ is the percentage of the total CPU capacity ac-
tually used by the application component during time-interval
k. The independent random process wk ∈ R+ represents
the process noise. The process noise models the utilization
between successive intervals caused by workload changes, e.g.,
requests being added, doing work from previous intervals, or
leaving the server.
By yk ∈ R+ we denote the total CPU utilization actually
observed in the VM, i.e., yk models the observed utilization in
addition to any usage noise coming from other sources, such
as the operating system, to support the application, i.e.,
y k = x k + vk ,
where vk ∈ R+ denotes the utilization measurement noise.
By ak ∈ R+ we denote the CPU capacity of a physical
machine allocated to the VM, i.e., the maximum amount of
resources a VM can use. The purpose of a designed controller
is to control the allocation of the VM running a server appli-
cation while observing its utilization in the VM, maintaining
good server performance in the presence of workload changes.
This is achieved by adjusting the allocation to values above the
utilization. For each time-interval k, the desired relationship
between the two quantities is given by:
ak = min{(1 + h)xk , amax }, (3)
where h ∈ (0, 1) represents the headroom (i.e., how much
extra resources are provided above the actual CPU utilization)
and amax is the maximum CPU that can be allocated. To
maintain good server performance, the allocation ak should
adapt to the utilization xk . In [8], [9] and in [19] Kalman
and H∞ filters have been designed, respectively, to predict
the CPU utilization per component of the server and thus
allocate the correct amount of resources. Let Yk represent
the set of all observations up to time k. Let the a posteriori
and a priori state estimates be denoted by x̂k|k = E {xk |Yk }
and x̂k+1|k = E {xk+1 |Yk }, respectively; hence, x̂k+1|k is the
predicted CPU utilization for time-step k + 1. As a result, the
CPU allocation controller is given by

(4)
1400 1600 1800 2000
IV. C ONTROLLER DESIGN
In this work a new Kalman filter approach is used that
uses the Maximum Correntropy Criterion (MCC) for state
estimation, referred in literature as MCC Kalman filter (MCC-
KF) in [11] and [12]. The correntropy criterion measures the
similarity of two random variables using information from
high-order signal statistics [13], [23]–[25]. Since the Kalman
filter uses only second-order signal information is not optimal
if the process and measurement noises are non-Gaussian noise
disturbances, such as shot noise or mixture of Gaussian noise.
Consider the system with dynamics
xk+1 = Axk + wk , (5a)
yk = Cxk + vk , (5b)
where xk ∈ Rnx is the state of the system, uk ∈ Rnu is
the control input, wk ∈ Rnx is a stochastic disturbance with
zero mean and finite second order matrix Wk , yk ∈ Rny
is the observation of the state of the system xk , vk ∈ Rny
is a stochastic disturbance with zero mean and finite second
order matrix Vk , and A and C are matrices of appropriate
dimensions. Let, also, the a posteriori (updated) and a priori
(predicted) error covariances be given by
Pk|k = E (xk − x̂k|k )(xk − x̂k|k )T |Yk ,

(2)
Pk+1|k = E (xk − x̂k+1|k )(xk − x̂k+1|k )T |Yk .

The equations for the MCC-KF are summarized below [12].
For the prediction phase:
x̂k|k−1 = Ax̂k−1|k−1 , (6a)
T
Pk|k−1 = APk−1|k−1 A + Wk , (6b)
and for the update phase:
 
Gσ k yk − C x̂k|k−1 kV −1
k
Lk = ,

Gσ k x̂k|k−1 − Ax̂k−1|k−1 kP −1
k|k−1
−1
Kk = (Pk|k−1 + Lk C T Vk−1 C)−1 Lk C T Vk−1 ,
x̂k|k = x̂k|k−1 + Kk (yk − C x̂k|k−1 ),
T
Pk|k = (I − Kk C)Pk|k−1 (I − Kk C) +
where Gσ is the Gaussian kernel, i.e.,
k xi − yi k2
 
Gσ (k xi − yi k) = exp −
2σ 2
with kernel size σ. Note that Lk is called the minimized
correntropy estimation cost function and Kk is the Kalman
gain.
Since in this work we are using a SISO controller in our
model, given by (1) and (2), the original MCC-KF equations
(6a)-(6f) are simplified to:
x̂k|k−1 = x̂k−1|k−1 ,
Pk|k−1 = Pk−1|k−1 + Wk ,
 
Gσ k yk − x̂k|k−1 kV −1
k
Lk =  ,
Gσ k x̂k|k−1 − x̂k−1|k−1 kP −1
Lk
Kk = , (7d)
(Pk|k−1 + Lk Vk−1 )Vk
−1
x̂k|k = x̂k|k−1 + Kk (yk − x̂k|k−1 ),
2 2
Pk|k = (1 − Kk ) Pk|k−1 + Kk Vk ,
As it can be observed in (7a)-(7f), MCC-KF has the same
structure as the Kalman filter, but in addition, it uses high-
order statistics to improve state estimation.
V. P ERFORMANCE E VALUATION
A. Experimental Setup
A two-tier prototype cluster of Rice University Bidding
System (RUBiS)2 was deployed in order to evaluate the
of our prototype RUBiS server application, as the performance
metric. The goal of the control system is to adapt the CPU
resource allocations on a single component (e.g., Web Server
(6c)
VM) in order to save resources for other applications that
can be hosted on the same physical machine. There are two
(6d) parameters by which a server’s performance can be affected:
(6e) (i) the number of clients that send requests simultaneously
to the server and (ii) the workload type. An overview of the
Kk Vk KkT , (6f) system’s architecture is shown in Fig. 5 below.
Fig. 5: System architecture.
B. Headroom
(7a) Let parameter c denote the desired CPU utilization to CPU
(7b) allocation ratio, i.e., c = 1/(1 + h). The mRT with respect
to parameter c for Kalman filter, H∞ filter and MCC-KF, is
shown in Fig. 6. In this evaluation, we set a stable workload
(7c)
of 500 clients sending requests simultaneously to the RUBiS
k|k−1 auction site. Each measurement is derived from an experiment,
where c differs every time in steps of 0.1. While c approaches
1, more resources are saved for other applications to run, but
(7e) the mRT of requests increases and thus the performance of
the RUBiS benchmark drops. This occurs because while the
(7f) headroom approaches 0, there are not enough resources left
to be used, which means that the requests have to wait in a
queue.
500
Kalman
H∞
400 MCC−KF
300
mRT (ms)

performance of our system. The cluster is consisted from a

web server and a database server which are deployed over 200
Xen Hypervisor VMs. Another physical server emulates the
client’s behavior, with sending requests to the RUBiS auction 100
site, via the RUBiS Client Emulator. All controllers presented
in this work were added on the base project code called
0
ViResA3 , which was first deployed for the synthetic data 0.5 0.6 0.7 0.8 0.9 1.0
generation and the performance evaluation of the controllers c parameter
in [22]. The performance of the dynamic CPU allocation is Fig. 6: mRT with respect to c: the closer parameter c is to
evaluated using the mRT, which is measured at the Client-side 1, the largest the mRT becomes. It is observed that MCC-KF
2 RUBiS is an auction site benchmark that implements the core functionality
can have a larger value of c (i.e., smaller headroom) than the
of an auction site. It is modeled after ebay.com with a web front end and a
Kalman filter for the same mRT.
database. It has provisions for selling, browsing and bidding items, allowing As can be observed, all filters can allocate resources without
for different sessions for different type of users in the form of visitor, buyer a big increase of mRT when c is less than 0.8. However,
and seller to be implemented. when the c parameter increases above 0.8, the Kalman filter
3 ViResA (Virtualized [server] Resource Allocation) is a base project code
hosted in Atlassian Bitbucket (https://bitbucket.org) as a private Git repository. performs worse than H∞ and MCC-KF. This happens because
For download requests, please contact the authors. H∞ and MCC-KF can predict the next allocation when CPU
usage changes in a random way which brings out the non- Evidently, the Kalman filter performs well as the fluctuation
Gaussian noise environments of the demand. Based on these of workload is relatively small. During time interval 0 − 30s,
observations, we can see that while the c parameter is set 400 clients issue requests to the RUBiS auction site, while
at 0.8, the mRT does not increase. However, we set the c at 30s, 600 more clients are added to the RUBiS application,
parameter at 0.7 for future experiments, in order to evaluate which effectively increases the demand in CPU resources. As
the controller’s performance while the number of clients and a result, the CPU usage changes abruptly and the allocation
hence the workload varies randomly. does not adjust fine which leads to a sudden mRT increase,
In all experiments shown in the following figures, c is set reaching a value of almost 2200ms (see lower graph of Fig. 7).
to 0.7. For all the schemes, the initial value of the error After this sudden CPU demand increase, the mRT is kept low
covariance matrix, P0 , is set to 10, the variance of the process at its normal values.
noise, W , is set to 4, and the variance of the measurement
noise, V , is set to 1. All CPU measurements for utilization and D. H∞ filter
allocations are exported from the Web Server’s VM, via Xen Next, we evaluate the performance of the H∞ filter, pro-
Hypervisor. In order to evaluate the performance of our system posed in [19] with the same parameters as before. Fig. 8
we set up experiments of 100s total duration, which is enough (upper and lower graphs) depicts the CPU usages-allocations
to evaluate the system performance. At the start of each and mRT for H∞ filter, respectively.
experiment and for the first couple of seconds the CPU usage
100
remains relatively stable at a mean value of 25% as a result of allocation
the constant workload of 400 clients that issue requests to the 90
usage
RUBiS auction site. During time interval 30 − 70s the RUBiS 80
application is loaded with 1000 clients in total, which then 70
after 70s drops to 400 clients. 60
CPU (%) 50
C. Kalman filter 40
30
The upper graph of Fig. 7 shows the predicted Kalman
20
filter allocations that are used to adjust the Web Server
10
tier allocations. The lower graph of Fig. 7 depicts RUBiS
0
performance in respect to mRT during Kalman filtering. 0 10 20 30 40 50 60 70 80 90 100
time interval (s)
3000
100 H∞
90 allocation 2500
usage
80
2000
70
mRT (ms)

60 1500
CPU (%)

50
40 1000
30
500
20
10 0
0 10 20 30 40 50 60 70 80 90 100
0 time interval (s)
0 10 20 30 40 50 60 70 80 90 100
time interval (s) Fig. 8: Upper: H∞ filter CPU usage and allocation. Lower:
3000
Kalman mRT for H∞ filter.
2500
In this experiment the H∞ filter predicts and adjusts the
2000 CPU allocations for a period of 100s. During time interval
mRT (ms)

0 − 30s, 400 clients issue requests to the RUBiS auction

1500
site, while at 30s, 600 more clients are added to the RUBiS
1000 application, which effectively increases the demand in CPU
resources. After 40s, the workload of the application remains
500 relatively stable with no big fluctuations, hence the mRT is
kept low. It is evident that the H∞ filter performs better
0
0 10 20 30 40 50 60 70 80 90 100 than the Kalman filter as the workload increases. As an
time interval (s)
effect when the CPU usage changes abruptly, the allocation is
Fig. 7: Upper: Kalman filter CPU usage and allocation. Lower: adjusted satisfactorily while maintaining a lower mRT value
mRT for Kalman filter. than the Kalman. Note that the proposed H∞ filter in [19]
was originally evaluated only via simulations using synthetic performance of each controller is obtained through our per-
data, while in this work a real-time testbed is used. formance metric, which is mRT and thus MCC-KF performs
slightly better than Kalman and H∞ filters.
E. MMC-Kalman filter (MCC-KF)
The CPU usages and allocations that MCC-KF predicts 100
and adjusts are shown in the upper graph Fig. 9. RUBiS 90
performance in respect to mRT is shown in the lower graph 80
of Fig. 9. 70
60

CPU (%)
100 50
90 allocation
usage 40
80 30 Kalman
70 20 H∞
60 MCC−KF
10 usage
CPU (%)

50 0
0 10 20 30 40 50
40 time interval (s)
30 Fig. 10: Allocation Predictions with the same CPU usage per
20 Controller.
10 Overall, we observe that all controllers perform well while
0
0 10 20 30 40 50 60 70 80 90 100 the workload is relatively stable, and thus the mRT is kept at
time interval (s) the low mRT region. However, when there is a big workload
3000
MCC−KF change because of an increase in the number of clients issuing
2500 requests to the RUBiS, MCC-KF manages to keep the mRT
at lower values than the Kalman filter and slightly lower than
2000 H∞ filter. The H∞ filter, as it is shown in Fig. 10, manages to
mRT (ms)

1500
1000
500
0 A. Conclusions
0 10 20 30 40 50 60 70 80
time interval (s)
Fig. 9: Upper: MCC-KF filter CPU usage and allocation.
Lower: mRT for MCC-KF filter.
This experiment shows the way that MCC-KF responds
when the number of clients issuing requests to the server
increases from 400 to 1000 like previous experiments. As
for this experiment, the sudden workload change was done
at time period 30 − 40s. MCC-KF, operates well enough at
sudden CPU usage changes while the mRT is kept at 1800ms.
During all other regions, the mRT stays at low levels because
the remaining amount of CPU resources is enough for serving
the stable workload.
In order to observe how each controller operates, we ran an
experiment with exactly the same demand for all controllers,
with a big workload change at time interval 20 − 30s. During
this experiment, the allocations that are predicted from the
controllers were not applied, since only one controller can
handle the CPU allocations, for each experiment, using the
Xen Hypervisor. Fig. 10 shows that, while the workload is
relatively stable, the predictions by each controller are almost
the same. However when the demand changes suddenly, the
predictions of H∞ filter allocates more resources for the
RUBiS application than Kalman and MCC-KF. However, the
keep the mRT low by allocating more CPU resources, while
MCC-KF allocates less, and as a result, more resources are
available for use by other applications on the same physical
machine.
VI. C ONCLUSIONS AND F UTURE D IRECTIONS
90 100
Several available virtualization technologies allow for dy-
namic CPU allocation adjustment of each component in
virtualized server environment in such a manner so as to
prevent resource saturation. In order to adjust the allocation,
suitable controllers must be developed to take on the role of
predicting and hence adjusting the CPU allocations of each
server component. In this work, a Maximum Correntropy
Criterion Kalman Filter (MCC-KF) has been used in order to
adjust the CPU allocations of each component when random
workload changes are introduced from a RUBiS auction site
for a varying number of clients deployed on a Xen-virtualized
cluster. The MCC-KF allows for state estimation in the pres-
ence of non-Gaussian noise environments, as it is the case of
the CPU usage.
B. Future Directions
In this work, all the controllers were implemented assuming
prior knowledge of the distribution pertaining the workload
changes. However, the system dynamics change with time,
since the volume and fluctuation of requests might vary
significantly due to increased/decreased popularity of a site,
advertisement, etc. For this reason, it is important that self-
adaptation techniques are adopted in order to adjust the
proposed controllers to these fluctuations [26]–[28].
 Resource needs across multiple dimensions, such as com-
pute, storage, and network bandwidth, are coupled; hence,
workload consolidation should be performed while taking
care of the resource coupling in multi-tier virtualized appli-
cations in order to provide timely allocations during workload
changes. For this reason, part of ongoing research concentrates
on using system identification/learning to extract the coupling
between the resource needs for more efficient workload con-
solidation.
Data centers distributed in different geo-location can work
collaboratively as a whole. They are connected with high-
speed internet or dedicated high-bandwidth communication
links. VMs can be migrated, if necessary, within a data
center, or even across data centers [29]. The combination of
dynamically changing the size of VMs and their migration,
while a long standing issue, constitutes a very challenging
and open problem.
ACKNOWLEDGEMENTS
The authors would like to thank Ms. Kika Christou from the
Information Systems & Technology Service of Cyprus Univer-
sity of Technology, for her continued support on computing
and networking infrastructure related issues.
R EFERENCES
[1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neuge-
bauer, I. Pratt, and A. Warfield, “Xen and the Art of Virtualization,” in
Proceedings of the ACM Symposium on Operating Systems Principles
(SOSP), 2003, pp. 164–177.
[2] L. Rodero-Merino, L. M. Vaquero, V. Gil, F. Galán, J. Fontán, R. S.
Montero, and I. M. Llorente, “From infrastructure delivery to service
management in clouds,” Future Generation Computer Systems, vol. 26,
no. 8, pp. 1226–1240, 2010.
[3] G. Jung, K. R. Joshi, M. A. Hiltunen, R. D. Schlichting, and C. Pu, “A
cost-sensitive adaptation engine for server consolidation of multitier ap-
plications,” in Proceedings of the 10th ACM/IFIP/USENIX International
Conference on Middleware. New York, NY, USA: Springer-Verlag New
York, Inc., 2009, pp. 9:1–9:20.
[4] E. Kalyvianaki, T. Charalambous, and S. Hand, “Resource Provisioning
for Multi-Tier Virtualized Server Applications,” Computer Measurement
Group (CMG) Journal, vol. 126, pp. 6–17, 2010.
[5] D. Ardagna, B. Panicucci, M. Trubian, and L. Zhang, “Energy-aware
autonomic resource allocation in multitier virtualized environments,”
IEEE Transactions on Services Computing, vol. 5, no. 1, pp. 2–19,
March 2012.
[6] C. Mastroianni, M. Meo, and G. Papuzzo, “Probabilistic consolidation
of virtual machines in self-organizing cloud data centers,” IEEE Trans-
actions on Cloud Computing, vol. 1, no. 2, pp. 215–228, July 2013.
[7] J. Zhang, H. Huang, and X. Wang, “Resource provision algorithms
in cloud computing: A survey,” Journal of Network and Computer
Applications, vol. 64, pp. 23–42, 2016.
[8] E. Kalyvianaki, T. Charalambous, and S. Hand, “Adaptive resource pro-
visioning for virtualized servers using Kalman filters,” ACM Transaction
on Autonomous and Adaptive Systems, vol. 9, no. 2, pp. 10:1–10:35, July
2014.
[9] ——, “Self-Adaptive and Self-Configured CPU Resource Provisioning
for Virtualized Servers using Kalman Filters,” in Proceedings of the 6th
International Conference on Autonomic Computing (ICAC). New York,
NY, USA: ACM, 2009, pp. 117–126.
[10] B. Chen, Y. Zhu, J. Hu, and J. C. Principe, System Parameter Identifi-
cation: Information Criteria and Algorithms, 1st ed. Amsterdam, The
Netherlands, The Netherlands: Elsevier Science Publishers B. V., 2013.
[11] B. Chen, X. Liu, H. Zhao, and J. C. Príncipe, “Maximum Correntropy
Kalman Filter,” arXiv:1509.04580 [cs, stat], Sep. 2015, arXiv:
1509.04580. [Online]. Available: http://arxiv.org/abs/1509.04580
[12] R. Izanloo, S. A. Fakoorian, H. S. Yazdi, and D. Simon, “Kalman
filtering based on the maximum correntropy criterion in the presence
of non-Gaussian noise,” in Annual Conference on Information Science
and Systems (CISS), Mar. 2016, pp. 500–505.
[13] W. Liu, P. P. Pokharel, and J. C. Príncipe, “Correntropy: Properties and
Applications in Non-Gaussian Signal Processing,” IEEE Transactions
on Signal Processing, vol. 55, no. 11, Nov 2007.
[14] L. Kleinrock, Queueing Systems, Volume 1, Theory. Wiley-Interscience,
1975.
[15] Z. Wang, X. Zhu, and S. Singhal, “Utilization and SLO-Based Control
for Dynamic Sizing of Resource Partitions,” in Proceedings of the
IFIP/IEEE International Workshop on Distributed Systems: Operations
and Management (DSOM), October 2005, pp. 133–144.
[16] X. Zhu, Z. Wang, and S. Singhal, “Utility-Driven Workload Management
using Nested Control Design,” in Proceedings of the American Control
Conference (ACC), 2006, pp. 6033–6038.
[17] Z. Wang, X. Liu, A. Zhang, C. Stewart, X. Zhu, T. Kelly, and S. Singhal,
“AutoParam: Automated Control of Application-Level Performance in
Virtualized Server Environments,” in Proceedings of the IEEE Interna-
tional Workshop on Feedback Control Implementation and Design in
Computing Systems and Networks (FeBID), 2007.
[18] P. Padala, K. Shin, X. Zhu, M. Uysal, Z. Wang, S. Singhal, A. Merchant,
and K. Salem, “Adaptive Control of Virtualized Resources in Utility
Computing Environments,” in Proceedings of the European Conference
on Computer Systems (EuroSys), 2007, pp. 289–302.
[19] T. Charalambous and E. Kalyviannaki, “A min-max framework for CPU
resource provisioning in virtualized servers using H∞ Filters,” in 49th
IEEE Conference on Decision and Control (CDC), Dec. 2010, pp. 3778–
3783.
[20] P. Padala, K.-Y. Hou, K. G. Shin, X. Zhu, M. Uysal, Z. Wang, S. Sing-
hal, and A. Merchant, “Automated Control of Multiple Virtualized
Resources,” in Proceedings of the 4th ACM European Conference on
Computer Systems (EuroSys ’09). New York, NY, USA: ACM, 2009,
pp. 13–26.
[21] M. Sithu and N. L. Thein, “A resource provisioning model for virtual
machine controller based on neuro-fuzzy system,” in 2011 The 2nd
International Conference on Next Generation Information Technology
(ICNIT), Jun. 2011, pp. 109–114.
[22] K. M. Deliparaschos, T. Charalambous, E. Kalyvianaki, and
C. Makarounas, “On the use of fuzzy logic controllers to comply with
virtualized application demands in the cloud,” in European Control
Conference (ECC), June 2016, pp. 649–654.
[23] W. Liu, P. P. Pokharel, and J. C. Príncipe, “Correntropy: A Localized
Similarity Measure,” in The 2006 IEEE International Joint Conference
on Neural Network Proceedings, 2006, pp. 4919–4924.
[24] R. He, W. S. Zheng, and B. G. Hu, “Maximum Correntropy Criterion
for Robust Face Recognition,” IEEE Transactions on Pattern Analysis
and Machine Intelligence, vol. 33, no. 8, pp. 1561–1576, Aug. 2011.
[25] A. Singh and J. C. Principe, “Using Correntropy As a Cost Function
in Linear Adaptive Filters,” in Proceedings of the 2009 International
Joint Conference on Neural Networks, ser. IJCNN’09. Piscataway,
NJ, USA: IEEE Press, 2009, pp. 1699–1704. [Online]. Available:
http://dl.acm.org/citation.cfm?id=1704175.1704421
[26] A. Filieri, C. Ghezzi, A. Leva, and M. Maggio, “Self-adaptive software
meets control theory: A preliminary approach supporting reliability
requirements,” in Proceedings of the 2011 26th IEEE/ACM Interna-
tional Conference on Automated Software Engineering, ser. ASE ’11.
Washington, DC, USA: IEEE Computer Society, 2011, pp. 283–292.
[27] A. Filieri, H. Hoffmann, and M. Maggio, “Automated design of self-
adaptive software with control-theoretical formal guarantees,” in Pro-
ceedings of the 36th International Conference on Software Engineering,
ser. ICSE 2014. New York, NY, USA: ACM, 2014, pp. 299–310.
[28] A. Filieri, M. Maggio, K. Angelopoulos, N. D’Ippolito,
I. Gerostathopoulos, A. B. Hempel, H. Hoffmann, P. Jamshidi,
E. Kalyvianaki, C. Klein, F. Krikava, S. Misailovic, A. V. Papadopoulos,
S. Ray, A. M. Sharifloo, S. Shevtsov, M. Ujma, and T. Vogel, “Software
engineering meets control theory,” in Proceedings of the 10th
International Symposium on Software Engineering for Adaptive and
Self-Managing Systems, ser. SEAMS ’15. Piscataway, NJ, USA: IEEE
Press, 2015, pp. 71–82.
[29] P. T. Endo, A. V. de Almeida Palhares, N. N. Pereira, G. E. Goncalves,
D. Sadok, J. Kelner, B. Melander, and J. E. Mangs, “Resource allocation
for distributed cloud: concepts and research challenges,” IEEE Network,
vol. 25, no. 4, pp. 42–46, July 2011.