Professional Documents
Culture Documents
On A Catalogue of Metrics For Evaluating Commercia
On A Catalogue of Metrics For Evaluating Commercia
net/publication/235427265
CITATIONS READS
90 83
4 authors, including:
He Zhang
Nanjing University
113 PUBLICATIONS 1,343 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Zheng (Eddie) Li on 25 August 2014.
Abstract— Given the continually increasing amount of [31], we proposed to perform a comprehensive investigation
commercial Cloud services in the market, evaluation of into evaluation metrics in the Cloud Computing domain.
different services plays a significant role in cost-benefit Unfortunately, in contrast with traditional computing
analysis or decision making for choosing Cloud Computing. In systems, the Cloud nowadays is still chaos [56]. The most
particular, employing suitable metrics is essential in evaluation outstanding issue is that there is a lack of consensus of
implementations. However, to the best of our knowledge, there standard definition of Cloud Computing, which inevitably
is not any systematic discussion about metrics for evaluating leads to market hype and also skepticism and confusion [28].
Cloud services. By using the method of Systematic Literature As a result, it is hard to point out the range of Cloud
Review (SLR), we have collected the de facto metrics adopted
Computing and a full scope of metrics for evaluating
in the existing Cloud services evaluation work. The collected
metrics were arranged following different Cloud service
different commercial Cloud services. Therefore, we decided
features to be evaluated, which essentially constructed an to unfold the investigation along a regression manner. In
evaluation metrics catalogue, as shown in this paper. This other words, we tried to isolate the de facto evaluation
metrics catalogue can be used to facilitate the future practice metrics from the existing evaluation work to help understand
and research in the area of Cloud services evaluation. the state-of-the-practice of the metrics used in Cloud services
Moreover, considering metrics selection is a prerequisite of evaluation. When it comes to exploring the existing
benchmark selection in evaluation implementations, this work evaluation practices of Cloud services, we employed three
also supplements the existing research in benchmarking the constraints:
commercial Cloud services. This study focused on the evaluation of only
commercial Cloud services, rather than that of
Keywords- Cloud Computing; Commercial Cloud Service; private or academic Cloud services, to make our
Cloud Services Evaluation; Evaluation Metrics; Catalogue effort closer to industry’s needs.
This study concerned Infrastructure as a Service
I. INTRODUCTION (IaaS) and Platform as a Service (PaaS) without
Cloud Computing, as one of the most promising considering Software as a Service (SaaS). Since
computing paradigms [1], has become increasingly accepted SaaS with special functionalities is not used to
in industry. Correspondingly, more and more commercial further build individual business applications [21],
Cloud services offered by an increasing number of providers the evaluation of various SaaS instances could
are available in the market [2, 5]. Considering that customers require infinite and exclusive metrics that would be
have little knowledge and control over the precise nature of out of the scope of this investigation.
commercial Cloud services even in the “locked down” This study only explored empirical evaluation
environment [3], evaluation of those services would be practices in academic publications. There is no doubt
crucial for many purposes ranging from cost-benefit analysis that informal descriptions of Cloud services
for Cloud Computing adoption to decision making for Cloud evaluation in blogs and technical websites can also
provider selection. provide highly relevant information. However, on
When evaluating Cloud services, a set of suitable the one hand, it is impossible to explore and collect
measurement criteria or metrics must be chosen. In fact, useful data from different study sources all at once.
according to the rich research in the evaluation of traditional On the other hand, the published evaluation reports
computer systems, the selection of metrics plays an essential can be viewed as typical and peer-reviewed
role in evaluation implementations [32]. However, compared representatives of the existing ad hoc evaluation
to the large amount of research effort into benchmarks for practices.
the Cloud [3, 4, 16, 21, 34, 45], to the best of our knowledge, Considering that the Systematic Literature Review (SLR)
there is not any systematic discussion about metrics for has been widely accepted as a standard and rigorous
evaluating Cloud services yet. Considering that the metrics approach to evidence collection for investigating specific
selection is one of the prerequisites of benchmark selection research questions [26, 27], we adopted the SLR method to
identify, assess and synthesize the published primary studies
of Cloud services evaluation. Due to the limit of space, the because they are inevitably reflected by the changes in the
detailed SLR process is not elaborated in this paper 1 . index of normal performance features.
Overall, we have identified 46 relevant primary studies Naturally, here we display the performance evaluation
covering six commercial Cloud providers, such as Amazon, metrics mainly following the sequence of these performance
GoGrid, Google, IBM, Microsoft, and Rackspace, from a set elements. In addition, the evaluation metrics for overall
of popular digital publication databases (all the identified performance of Cloud services are particularly listed. The
primary studies have been listed online for reference: metrics for evaluating Scalability and Variability are also
http://www.mendeley.com/groups/1104801/slr4cloud/papers separated respectively.
/). More than 500 evaluation metrics including duplications
were finally extracted from the identified Cloud services
evaluation studies. Physical Capacity Part
This paper reports our investigation result. After Property Part
removing duplications and differentiating metric types, the Transaction
evaluation metrics were arranged according to different Speed
Cloud service features covering the following aspects: Communication
Performance, Economics, and Security. The arranged result Availability
essentially constructed a catalogue of metrics for evaluating
commercial Cloud services. In turn, we can use this metrics Computation Scalability
catalogue to facilitate the Cloud services evaluation work,
Latency (Time)
such as quickly looking up suitable evaluation metrics,
identifying current research gap and future research Memory
(Cache) Variability
opportunities, and developing sophisticated metrics based on
the existing metrics. Reliability
The remainder of the paper is organized as follows.
Storage
Section II arranges all the identified evaluation metrics under
different Cloud service features. Section III introduces three Data Throughput
(Bandwidth)
scenarios of applying this metrics catalogue. Conclusions
and some future work are discussed in Section IV.
Figure 1. Performance features of Cloud services for evaluation.
II. THE METRICS FOR CLOUD SERVICES EVALUATION
It is clear that the choice of appropriate metrics depends 1) Communication Evaluation Metrics (cf. Table I):
on the service features to be evaluated [31]. Therefore, we Communication refers to the data/message transfer between
naturally organized the identified evaluation metrics internal service instances (or different Cloud services), or
according to their corresponding Cloud service features. In between external client and the Cloud. In particular, given
detail, the evaluated features in the reviewed primary studies the separate discussions about IP-level and MPI-message-
can be found scattered over three aspects of Cloud services level networking among public Clouds [e.g. 8], we also
(namely Performance, Economics [35], and Security) and distinguished evaluation metrics between TCP/UDP/IP and
their properties. Thus, we use the following three subsections MPI communications.
to respectively introduce those identified metrics. Brief descriptions of particular metrics in Table I:
Packet Loss Frequency vs. Probe Loss Rate: Here
A. Performance Evaluation Metrics
we directly copied the names of these two metrics
In practice, an evaluated performance feature is usually from [43]. Packet Loss Frequency is defined as the
represented by a combination of a physical property of Cloud rate between loss_time_slot and total_time_slot, and
services and its capacity, for example Communication Probe Lost Rate is defined as the rate between
Latency, or Storage Reliability. Therefore, we divide a lost_probes and total_probes. Considering that the
performance feature into two parts: Physical Property part concept Availability is driven by the time lost while
and Capacity part. Thus, all the elements of performance Reliability is driven by the number of failures [10],
features identified from the aforementioned primary studies we can find that the former metric is for
can be summarized as shown in Figure 1. The detailed Communication Availability evaluation while the
explanations and descriptions of different performance latter is for Communication Reliability.
feature elements have been clarified in our previous Correlation between Total Runtime and
taxonomy work [57]. In particular, Scalability and Communication Time: This metric is to observe a set
Variability are also regarded as two elements in the Capacity of applications about their runtime and the amount
part, while further distinguished from the other capacities, of time they spend communicating in the Cloud. The
trend of the correlation can be used to qualitatively
discuss the influence of Communication on the
1
The SLR report can be found online: applications running in the Cloud.
https://docs.google.com/open?id=0B9KzcoAAmi43LV9IaEgtNnVUenVX
Sy1FWTJKSzRsdw
TABLE I. COMMUNICATION EVALUATION METRICS TABLE II. COMPUTATION EVALUATION METRICS