An Early Comparison of Commercial and Open-Source

Cloud Platforms for Scientific Environments

Ivan Voras, Marin Orli, Branko Mihaljevi
University of Zagreb Faculty of Electrical Engineering an !o"#uting, Zagreb, !roatia
{ivan.voras, marin.orlic, branko.mihaljevic}@fer.hr
Abstract. !lou co"#uting #ro"ises efficient use of har$are resources
through virtuali%ation an elastic co"#uting facilities& Various clou co"#uting
solutions have e"erge on the "arket fro" o#en'source co""unities an
co""ercial venors& In this #a#er $e iscuss criteria for feature co"#arison of
#rivate clou #latfor"s an co"#are several o#en'source an co""ercial
#roucts& (e test #erfor"ance of hy#ervisors use in these clous $ith a set of
bench"ark suites containing tests for various as#ects of the syste"& (e iscuss
the results in the conte)t of $hat is co""only escribe as a scientific $ork'
loa& *he escribe feature an #erfor"ance ifferences can hel# "ake $iser
#latfor" choices&
Keywords. !lou'co"#uting, o#en'source, co""ercial+close, hy#ervisors
1 Introduction
!lou co"#uting is a co"bination of technologies for resource "anage"ent an
#rovisioning $ith the goals of achieving cost control, elasticity, ease of use an "ass
e#loy"ent& In the $ake of the clou tren, nu"erous o#en'source an co""ercial
clou #roucts have e"erge& In this $ork, $e focus on #rivate'clou infrastructure'
as'a'service ,Iaa-. #roucts, an #resent a co"#arison of features an #erfor"ance of
o#en'source an co""ercial solutions& (e look at #rivate clous as #ri"ary #lat'
for"s in scientific environ"ents, $ith full local control over the #latfor"&
O#en'source solutions are often the first choice in scientific environ"ents for their
lo$er initial invest"ents an su##ort for scientific libraries an tools, as $ell as
o#enness to custo"i%ations& /ggressive #ricing an licensing o#tions fro" venors of
co""ercial #roucts leas us to e)#ect a variety of clou environ"ents "anaging
"i)e virtuali%ations in the future& 0revious #erfor"ance co"#arisons of clou ser'
vices for scientific co"#uting focuse on o#en'source an "oifie o#en'source
solutions 123& In this #a#er, $e co"#are #erfor"ance of o#en'source an co""ercial
hy#ervisors, the key ele"ents an enablers of clou #latfor"s&
*he rest of this #a#er is organi%e as follo$s& -ections 4 an 5 escribe select
o#en'source an co""ercial clou #latfor"s, res#ectively& -ection 6 escribes the
evaluation criteria use to co"#are #rouct features an #resents the co"#arison
results& -ection 7 co"#ares the #erfor"ance of o#en'source an co""ercial hy#ervi'
sors, an -ection 8 conclues the #a#er an iscusses the future $ork in this area&
Open-Source Cloud Platforms
O#en'source clou #latfor"s e"#loy o#en'source hy#ervisors ,9VM an :en.,
but so"e of the" also su##ort co""ercial+close hy#ervisors $ith e)#ose interfaces
,VM$are.& !lou #latfor"s co"bine various tools of the unerlying O- an virtual'
i%ation layer $ith their o$n co"#onents in a "ore or less sea"less clou interface&
Eucalyptus Community Cloud. Eucaly#tus is one of the "ost ao#te clou co"'
#uting architectures, $ith o#en'source ,!o""unity !lou. an co""ercial versions
,Enter#rise Eition.& E!! is a fle)ible an highly "oular syste" $ith co"#onents
e)#ose in the for" of $eb services intero#erable $ith the /"a%on /(- /0I $hich
allo$s sea"less integration $ith e)isting /"a%on #ublic clou services& E!! cur'
rently su##orts :en an 9VM virtuali%ations an can be e#loye on all "ajor ;inu)
istributions ,Ubuntu, !entO-, <ebian, =>E;, o#en-U-E, -;E- etc&.&
Open!ebula. O#en?ebula is a "anage"ent toolkit for #rivate an #ublic clous,
$hich orchestrates e)isting syste"s an services, relying on ;inu) an e)ternal #ro'
ucts for virtuali%ation, net$ork, storage or security technologies& Infrastructure ab'
straction an "oular a##roach su##ort stanari%ation an intero#erability $ith "ost
co""on virtuali%ations ,:en, VM$are, an 9VM., interfaces ,/"a%on /(-,
VM$are v!lou, an O@F O!!I., an /0Is& Its features inclue secure "anage"ent
of virtual i"ages, "achines, net$orks an storage, authentication, "ulti'tenancy,
Auota "anage"ent, an clou'bursting $ith /"a%on E!4&
" Commercial Cloud Platforms
(e have chosen the follo$ing co""ercial #latfor"s for this iscussionB
VM$areCs ,E-:i hy#ervisor., MicrosoftCs ,>y#er'V hy#ervisor., an !itri)Cs ,:en'
-erver hy#ervisor, a "oifie version of the o#en'source #rouct :en.& *hese hy#er'
visors are a basis for other co""ercial offerings, es#ecially those co"ing fro"
OEMs
2
, an bench"ark results also a##ly for those #roucts&
#$ware vCloud. VM$areDs offering is the olest an the "ost feature'co"#lete,
relying on the #o#ularity an #roven track recor of its virtuali%ation #roucts ,E-:i,
v-#here.& On to# of the virtuali%ation #latfor", a co"#rehensive "anage"ent #lat'
for" $as built an subseAuently e)tene for use in clou'like environ"ents $ith the

2
>0 !lou-yste" Matri), htt#B++h2EFF6&$$$2&h#&co"+#roucts+blaes+co"#onents+"atri)+
"ain&ht"l
v!lou <irector #rouct& *hough VM$areCs offer is truly co"#lete, it is also so"e'
$hat frag"ente, $ith services such as security "anage"ent ,v-hiel., billing
,!hargeback Manager., high availability ,-erver >eartbeat., an others elivere an
charge se#arately, $hich "ay significantly increase initially #rojecte costs&
/s an ine#enent venor not tie to s#ecific har$are #latfor"s or o#erating sys'
te"s, VM$are has create the "ost #latfor"'agnostic #rouct a"ong the evaluate
solutions, $hich has been ao#te by har$are venors an $hich fully su##orts the
$iest nu"ber of guest o#erating syste"s a"ong the evaluate solutions&
$icrosoft Private Cloud. MicrosoftDs #resence as a server virtuali%ation an clou
co"#uting venor is very recent, beginning in 4FFE $ith the introuction of its >y'
#er'V hy#ervisor #rouct& *he features an abilities of >y#er'V are noticeably sub'
#ar $hen co"#are to others $e have evaluate, lacking i"#ortant features such as
live "igration, su##ort for guest O- booting fro" -!-I rives, "e"ory over'co""it,
su##ort for "ore than 6 virtual !0Us, an official su##ort for #o#ular non'(ino$s
o#erating syste"s& On the other han, >y#er'V is tightly integrate $ith other Micro'
softDs #roucts an elivers an environ"ent $hich is easy to use an "anage, es#e'
cially $hen virtuali%ing (ino$s $orkloas& *he G0rivate clouH solution integrates
several #roucts $hich auto"ate resource "anage"ent ,"ost notably, the -yste"
!enter Virtual Machine Manager.&
Citri% CloudStac&. !itri)Ds clou solution is centere on its :en-erver #rouct, an
su##orte by the infrastructure an the co""unity of the !lou-tack o#en source
#roject, $hose #arent co"#any ,!lou&co". it recently acAuire& *he relationshi#
bet$een !itri) an !lou-tack see"s to be so"e$hat co"#le), $ith !itri) su##ort'
ing an enorsing it for its clou co"#uting efforts, as $ell as selling GEnter#riseH
an G-ervice 0rovierH eitions& Of all the evaluate solutions, !itri)Ds is #erha#s the
"ost on the borer bet$een being a truly o#en an a co""ercial+close solution&
/long $ith VM$areCs #roucts, :en-erver has the #roven track recor an #res'
ence in the business sector, but its o#enness an e)cellent ;inu) su##ort has also
"ae it attractive in the scientific an acae"ic environ"ents&
' Evaluation Criteria
Evaluating co"#le) #roucts such as clou #latfor"s reAuires a co""on set of cri'
teria for co"#arison 143, 153& *he #ri"ary goal for our criteria $as to establish a co"'
#arison baseline for o#en'source Iaa- #roucts, $hich $e later e)tene to inclue
co""ercial+close #roucts& -o"e criteria, es#ecially high'level ones, can be a##lie
to other clou "oels ,0aa- an -aa-.& *he criteria are ivie over si) "ain grou#sB
storage, virtuali%ation, net$ork, "anage"ent, security an venor su##ort& *he crite'
ria a"ittely favor #latfor"s aa#te to ;inu) $orkloas an hosts, an functional'
ities over usability an integration, $hich can be correcte using $eights&
(e have grae iniviual criteria in the range of F to 5, $here F esignates no
su##ort an 5 esignates full su##ort& In cases $here su##ort for a feature relies on
the unerlying o#erating syste" rivers or tools, $e assigne the grae of 4 if the
o#erating syste" su##ort is "ature& (e escribe each grou# in short, as their full
escri#tion is #rovie in our earlier $ork an is out of sco#e of this #a#er&
Stora(e criteria. Manage"ent an functionalities of virtual "achine storage are criti'
cal to achieve fle)ible an scalable i"#le"entations& -torage'relate criteria focus on
"ain technologies $hich i"#le"ent clou storageB irect'attache storage ,</-.,
storage area net$orks ,-/?., an net$ork'attache storage ,?/-., as $ell as backu#
features an technologies& Main technology grou#s are further ivie into s#ecific
technologies, e&g& file syste" an re#lication su##ort&
#irtuali)ation Criteria. Virtuali%ation technologies are the enablers of clou co"'
#uting& !o"#are to co""ercial #roucts, o#en'source #latfor"s su##ort a $ier
range of virtuali%ation technologies, often three or "ore o#en'source an co""ercial
hy#ervisors& Virtuali%ation criteria focus features such as VM ty#e an technology,
Auotas, "igration an cloning, resource #rioriti%ation, hot configuration an #rovi'
sioning& 0latfor"s su##orting a $ie range of hy#ervisors fare better than #latfor"s
that focus on a single hy#ervisor, regarless of the level of su##orte functionalities,
$hich $e trie to a"en by graing the level of su##ort&
!etwor& Criteria. ?et$ork connectivity bins the clou co"#onents together, an
connects it $ith its a"inistrators an en'users& ?et$ork su##ort is covere $ith
criteria escribing su##ort for V;/? services such as tagging ,IEEE EF4&2A., net'
$ork "anage"ent an isolation, fire$all su##ort via net$ork filtering, Ethernet Aual'
ity of service ,IEEE EF4&2#., an integration ca#abilities such as I0v8 an virtual
#rivate net$orks for access an "anage"ent&
$ana(ement Criteria. !o"#rehensive an usable "anage"ent is irectly relate to
the $ay a virtual "achine clou is hanle& !riteria escribe "anage"ent facilities of
a #latfor" an focus on host'guest O- integration, iniviual an "ass "anage"ent
of both hosts an guests, e)#ose an consu"e /0Is, ata collection for billing an
re#orting, auto"atic recovery, high availability an alerting features&
Security Criteria. !lou installations can #ose a significant security risk& (e e'
scribe security'relate ca#abilities of a #latfor" such as ata encry#tion, irectory
services integration, authori%ation levels an auiting events for s#ecific resources
,e&g& VM server or storage servers.& -ecurity re#ort ty#es available to clou a"inis'
trators, thir'#arty #rouct evaluations an certifications of co"#liance $ith security
stanars, an secure "anage"ent access are also looke into&
Community and #endor Support Criteria. 0rouct su##ort is often ivie be'
t$een a user co""unity an a venor& !o""unity'relate criteria eal $ith freely
available su##ort channels of the user or evelo#er co""unities gathere aroun the
#rouct, an their Auality& Venor'relate criteria concern $ith irect venor su##ort
channels an the #ossibility of a -;/ contract& (e also cover "ethos of custo"er
relations such as #ublic issue tracking, #roactive u#ates an !=M'like a##roaches,
co"#leteness of o#en'source or free #rouct versions an relate ocu"entation,
track recor, future viability, an the #ossibility of thir'#arty auiting&
'.1 *eature Comparison
(e have co"#are the features of Eucaly#tus !o""unity !lou 4&F&5 ,E!!.,
O#en?ebula 5&4 ,O#en?ebula., VM$are v!lou 7 ,v!lou., Microsoft 0rivate
!lou 4F24 ,M-0!., an !itri) !lou-tack 4&4 ,!lou-tack.& *able 2 #resents a su"
of graes by criteria grou#, nor"ali%e to the "a)i"u" #ossible score #er grou#& *he
full evaluation table contains IJ graes #er #rouct, 6E7 in total&
+able 1. Feature co"#arison $ith evaluation criteria
Criteria (roup ECC Open!ebula vCloud $SPC CloudStac&
-torage 44 ,6IK. 46 ,75K. 2I ,64K. 42 ,6JK. 48 ,7EK.
Virtuali%ation 2E ,4JK. 5F ,67K. 76 ,E4K. 42 ,54K. 72 ,JJK.
Manage"ent 55 ,6EK. 66 ,86K. 7E ,E6K. 5J ,76K. 67 ,87K.
?et$ork 22 ,62K. 8 ,44K. 26 ,74K. 2F ,5JK. 26 ,74K.
-ecurity 25 ,5IK. 28 ,6EK. 48 ,JIK. 26 ,64K. 26 ,64K.
Venor -u##ort 5F ,7IK. 5J ,J5K. 5I ,J8K. 45 ,67K. 5J ,J5K.
+otal score 1, -''./ 10, -0'./ 11 -,./ 12 -'"./ 13, -2'./

/s e)#ecte, v!lou is the "ost feature'co"#lete solution, $ith !lou-tack an
O#en?ebula follo$ing suit& !lou-tack has a ca#able hy#ervisor, $hile Microsoft
has the ege $ith its (ino$s ecosyste" har$are su##ort, "anage"ent an integra'
tion, as $ell as venor su##ort, but "ost of the avance features it offers a##ly only
for (ino$s $orkloas& O#en'source #roucts e)hibit a lo$er level of co"#onent
integration, an e)cel in so"e areas $hile lacking in other&
0 4ypervisor Performance Comparison of Commercial5Closed
and Open-Source Cloud Solutions
(hile virtuali%ation is not a necessary #art of clou co"#uting solutions, it is al'
"ost universally #resent for the convenience it brings to the i"#le"entation of elas'
ticity an auto"ation 163& Virtuali%ation is co""only achieve by e"ulating co"'
#lete virtual "achines uner the control of hy#ervisor soft$are ,as o##ose to "ore
light'$eight virtuali%ation environ"ents 173., "aking such soft$are essential for sta'
bility, security an #erfor"ance of the $hole clou infrastructure& *he scientific co"'
#uting environ"ent is so"e$hat s#ecial as it co""only uses #rivate or se"i'#rivate
infrastructure ,i&e& $here both the users an the infrastructure are kno$n an con'
trolle by a single entity., $hich "akes both security an soft$are stability less risky&
-yste" #erfor"ance of virtuali%e environ"ents is of i"#ortance to "any #otential
users, $hich rove us to investigate this as#ect of clou co"#uting& (e have set our
goal to evaluate co""on co"#uting $orkloas an #rovie a co"#rehensive ans$er
on the influence of "oern hy#ervisor soft$are on the overall syste" #erfor"ance&
Our evaluation $as centere on four of the "ost co""on virtuali%ation #rouctsB
VM$are E-:i 7&F, !itri) :en-erver 8&F, Microsoft >y#er'V 4FFE =4 -02, an
9VM fro" ;inu) 4&8&54& *he baseline $as an un'virtuali%e syste" on an ientical
har$are #latfor" ,4) :eon 768F !0Us, 28 @B of "e"ory, 4) 27 k=0M -/- isk
rives.& (e have use the !entO- 8&F o#erating syste" for the virtual "achines, $ith
the sa"e virtual "achine isk i"age use on all hy#ervisors& *he latter $as necessary
to reuce the nu"ber of variables an outsie influences on the "easure syste", but
has lea us to conclue that, saly, there is no stanar for "igrating virtual "achines
bet$een virtuali%ation environ"ents ao#te by all the "ajor venors an #rojects&
(e ho#e that the O#en Virtuali%ation For"at 183 ratifie as an /?-I stanar $oul
have been $iely ao#te, but its su##ort is notably absent fro" MicrosoftDs #roucts&

0.1 !otes on t6e 4ypervisors
Of the evaluate hy#ervisors ,su""ari%e in *able 4., E-:i, :en-erver an >y'
#er'V can be escribe as G*y#e 2H hy#ervisor 1J3, running on Gbare'"etalH $ithout a
su##orting o#erating syste", $hile 9VM is eciely G*y#e 4G, "aking full use of
regular ;inu) kernel features an subsyste"s in its o#eration& *he "ost i"#ortant
assu"#tion $hich can be erive fro" this istinction is that the *y#e 2 hy#ervisors
are "ore light'$eight an introuce less latency& One of our goals $as to verify the
valiity of this assu"#tion for the chosen hy#ervisors an the guest o#erating syste"&
/nother goal $as to check the i"#act of #aravirtuali%ation su##ort for certain hy'
#ervisors, "ost notably :en ,use in :en-erver. an 9VM& :enDs #aravirtuali%ation
ca#abilities are "uch "ore e)tensive than 9VMDs, $hich le us to e)#ect significant
#erfor"ance i"#rove"ents& ?either VM$are not 9VM #rovie #aravirtuali%ation at
the level of virtual "e"ory "anage"ent or other o#erating syste" co"#onents $hich
$oul influence our bench"ark&
+able . >y#ervisors teste in this $ork
4ypervisor 7icense +ype Cloud product
VM$are E-:i 7&F !o""ercial *y#e 2 v!lou
!itri) :en-erver 8&F !o""ercial+@0; *y#e 2 !lou-tack
Microsoft >y#er'V 4FFE =4 -02 !o""ercial *y#e 2 Microsoft 0rivate !lou
!entO- 8&F 9VM ,;inu) 4&8&54. @0; *y#e 4 O#en?ebula, Eucaly#tus&&&

/ccoring to inustry surveys 1E3, 1I3, the three co""ercial virtuali%ation solu'
tions o"inate the "arket, $ith a cu"ulative share u#$ars of EFK in #rouct e'
#loy"ents& Of these, VM$areDs #roucts are "ore #o#ular by a significant "argin&
-cientific an acae"ic environ"ents are traitionally very strongly tie to o#en
source solutions for reasons of cost'efficiency an historically better su##ort for s#e'
cific $orkloas& (e have observe ever stronger inustry lobbying e"#hasi%ing Gen'
ter#rise'graeH features an "ore refine "anage"ent tools, $hich coul "ean a
$ier ao#tion of co""ercial hy#ervisors in scientific co"#uting environ"ents&
0. Simulatin( Scientific 8or&loads
*here is of course no single efinition of a Gscientific $orkloaH, but there are
regular atte"#ts to characteri%e ty#ical $orkloas of certain ty#es of scientific envi'
ron"ents 12F'253& (e ecie on an a##roach of escribing ty#ical #erfor"ance by
using a nu"ber of bench"ark suites, $ith the intent that the overall results sho$ the
relative ifferences bet$een the teste syste"s, $ith res#ect to !0U, "e"ory, an
O- interaction #erfor"ance& *able 5 su""ari%es the selecte bench"ark suites&
+able ". Bench"ark suites use in this $ork
9enc6mar& Platform5environment +6readin( :esult interpretation
<a!a#o I&24 Lava ,O#enL<9 8 b2J. Mi)e *i"e ,lo$er is better.
0ysysbench F&I 0ython+! ,0ython 4&8. Multi'#rocess
*hrough#ut
,higher is better.
Bullet !ache 2&F&2 !, !MM ,gcc 6&6&8. Multi'threae
*ransactions #er secon
,higher is better.
-ciMark4+! ! ,gcc 6&6&8. -ingle'threae
*hrough#ut
,higher is better.

*he <a!a#o bench"ark suite 1223 evaluates real'$orl #erfor"ance by running a
set of o#en source a##lications of a varie behavior& Fro" this set of bench"arks $e
have use the follo$ingB avrora ,/V= "icrocontroller si"ulation, single'threae.,
batik ,-calable Vector @ra#hics renering, single'threae., ecli#se ,Ecli#se evel'
o#"ent environ"ent, "ulti'threae., h4 ,in'"e"ory atabase, "ulti'threae.,
jython ,0ython inter#reter in Lava, "ulti'threae., luine) ,;ucene full'te)t search'
ing library, single'threae., lusearch ,;ucene, "ulti'threae., #" ,source coe
analy%er, "ulti'threae., sunflo$ ,ray'tracing gra#hics, "ulti'threae., an )alan
,:M; transfor"er, "ulti'threae.& *he <a!a#o results are re#orte as average e)e'
cution ti"e for each bench"ark, in "illisecons&
*he 0ysysbench bench"ark "easures basic through#ut of co""on lo$'level
!0U'intensive tasks i"#le"ente in ! libraries calle fro" 0ythonB >ash'->/478,
>ash'->/724, Zlib'!o"#ress, Zlib'<eco"#ress, -ocket'-yscalls ,a loo# e)ecuting
the socket,. an close,. syste" calls., -ocket'One0i#e ,a loo# transferring 2 byte of
ata over a #air of sockets.& /ll tasks are e)ecute in #arallel #rocesses an the results
inicate through#ut ,a"ount of ata #rocesse or the nu"ber of o#erations #er sec'
on., as a co"#osite i"ensionless score&
*he Bullet !ache bench"ark runs the "ulti'threae Bullet !ache
4
1243 "e"ory
atabase $ith 7 threas on the server an on the client sie, $ith a varie nu"ber of
client connections, an re#orts the nu"ber of transactions #er secon, in thousans&
*he -ciMark4+! bench"ark is the ! i"#le"entation of the -ciMark4 bench"ark,
i"#le"enting a set of scientific nu"erical algorith"sB FF*, -O=, Monte !arlo inte'
gration, s#arse "atri) "ulti#lication, an ense ;U "atri) factori%ation& *he bench'
"ark $as use in the GlargeH configuration an the re#orte results #resent an a#'
#ro)i"ate MF;O0- #erfor"ance&
/ll bench"arks $ere installe an ran fro" a single virtual "achine i"age, con'
verte as necessary for various hy#ervisors& *he hy#ervisors the"selves $ere in'
stalle on ientical har$are syste"s eAui##e $ith t$o :eon 768F !0Us an 28 @B
of "e"ory& *he virtual "achines $ere configure $ith E virtual !0Us an E @B of
"e"ory, e)ce#t in the case of >y#er'V $here, ue to its li"itations, the virtual "a'
chine $as configure $ith 6 virtual !0Us&
0." 9enc6mar& :esults
*he chosen bench"ark suites #rouce etaile out#ut for each iniviual bench'
"ark, $hich can be use to inter#ret s#ecific as#ects of the teste syste"s ,iscusse
in follo$ing sections.& *he bench"ark results are the averages of five consecutive
runs, an the overall results are #resente su""ari%e #er bench"ark an #er hy#er'
visor in *able 6 an Figure 2&
+able '. >y#ervisor bench"ark results ,co"#are to non'virtuali%e baseline.
ES;i ;enServer 4yper-# K#$ !on-virt.
<a!a#o 775F ,M47K. 8F2F ,M5JK. 846F ,M64K. 8F4F ,M5JK. 65IF
0ysysbench 2F25 ,'7K. 2F65 ,'4K. 754 ,'7FK. 2F4F ,'6K. 2F8E
Bullet!ache 42J ,'68K. 42E ,'68K. 287 ,'7IK. 2E6 ,'76K. 6F2
-ciMark4+! 744 ,'4K. 74F ,'5K. 7FJ ,'7K. 722 ,'6K. 755

*he results sho$ that the #erfor"ance of current generation of hy#ervisors is rea'
sonable an there are, at first sight, no significant outstaning #roble"s& *he one
e)ce#tion is MicrosoftDs >y#er'V $hose results are ske$e in aggressively "ulti'
threae bench"arks ,0ysys"ark an Bullet !ache. because it oes not su##ort "ore
than 6 virtual !0Us& (e ra$ the follo$ing conclusions fro" the resultsB
2& 0erfor"ance ifferences a"ong the latest versions of the hy#ervisors, teste uner
the s#ecifie o#erating syste",s. an har$are, are "ostly $ithin 2FK& *he #er'
for"ance ifference to the non'virtuali%e case, ho$ever, can be larger than 7FK
,in the <a!a#o an Bullet !ache bench"arks.N
4& -ingle'threae, #urely co"#utational $orkloas ,-ciMark4+!. e)hibit the s"all'
est #erfor"ance i"#act, bet$een 4K ,E-:i an :en-erver. an 7K ,>y#er'V.N
5& *he largest ifferences are in e)tensively "ultithreae $orkloas ,Bullet !ache.N

4
Bullet !ache + Multi<o"ain !ache <ae"on, htt#B++"cache&sourceforge&net+
6& Fro" the etaile results $e can also conclue that si"#le syste" calls o not e)'
hibit a significant #erfor"ance i"#act& !o"bine $ith the #revious #oint it is
likely that e)ecution conte)t s$itching co"es $ith huge #enalties in virtuali%e
environ"ents& >o$ever, "ore testing is reAuire on ne$er eAui#"ent ,!0Us i"'
#le"enting ?este 0age *ables. to verify that the i"#act is still significant&
*he 9VM hy#ervisor e)hibite goo overall #erfor"ance, but it $as never at the
to#& *he s"allest virtuali%ation i"#act ,6K. $as "easure for -ciMark4 an 0ysys'
bench but this result is still far fro" the best ,4K, "easure for the :en-erver.&
VMWare ESXi 5.0
Citrix XenServer 6.0
Microsoft Hyper-V 2008 R2 S!
Cent"S 6 #VM
0$
20$
%0$
60$
80$
!00$
!20$
!%0$
!60$
!80$
&aCapo pysys'enc( )*++et Cac(e SciMar,2-C
C
*
.
*
+
a
t
i
v
e

p
e
r
c
e
n
t
a
/
e

p
o
i
n
t
s

*i(. 1. =elative #erfor"ance i"#act of virtuali%e environ"ents, co"#are to non'virtual envi'
ron"ents, by cu"ulative #ercentage #oints
*he absolute #erfor"ance i"#act of 9VM co"#are to the non'virtuali%e case is
large ,figure 2.& *he relative i"#act $hen co"#are to the leaing hy#ervisor ,E-:i.
is "uch s"aller, 27K in the $orst case ,Bullet !ache. but only F&JK in the best case
,0ysysbench.& *he ifference for the single'threae co"#ute'intensive $orkloa
,-ciMark4. is 4K an nearly IK for the "i)e a##lication $orkloa ,<a!a#o.&
(hether this is a #roble" in #ractice e#ens on a s#ecific e#loy"ent, but $e are
confient that the #erfor"ance is suitable for "ost uses& :en-erver an :en hy#ervi'
sors share the sa"e coe base an $e e)#ect the" to have ientical #erfor"ance&
2 Conclusion and *uture 8or&
(e have given a brief overvie$ of several o#en'source an co""ercial+close
#rivate'clou #latfor"s& 0resent #o#ularity of clou co"#uting #araig" an hea'
roo" for further evelo#"ent guarantee continuous gro$th an i"#rove"ent of
clou #latfor"s& Using a set of criteria evelo#e for evaluating o#en'source #lat'
for"s, $e have evaluate the features of leaing co""ercial+close solutions, an
confir"e that o#en'source an co""ercial solutions have co"#arable features&
(e also co"#are #erfor"ance of hy#ervisors running at the core of evaluate
clou #roucts& In an atte"#t to si"ulate scientific $orkloas, $e use a set of
bench"ark suites testing s#ecific syste" as#ects O !0U, "e"ory, an o#erating sys'
te" interaction #erfor"ance& Bench"ark results le us to conclue that "oern hy'
#ervisors #erfor"ance is close to non'virtuali%e case $ith single'threae $ork'
loas& ;arge #erfor"ance ifferences $ere observe $ith "ulti'threae $orkloas,
both bet$een the hy#ervisors an co"#are to non'virtuali%e har$are& *he tests
$ere "ae on an oler generation of har$are, an $e #lan to continue the tests once
$e have state'of'the'art eAui#"ent available&
:eferences
2& Iosu#, /&, Oster"ann, -&, Pigitbasi, ?& et&al&B 0erfor"ance /nalysis of !lou !o"#uting
-er'vices for Many'*asks -cientific !o"#uting& IEEE *ransactions on 0arallel an <istrib'
ute -yste"s& 44, I52'I67 ,4F22.&
4& Voras, I&, Mihaljevi, B&, Orli, M& et&al&B Evaluating O#en'-ource !lou !o"#uting -olu'
tions& 0roc& of the 56th International !onvention MI0=O 4F22& 4FI'426, Zagreb ,4F22.&
5& Voras, I&, Mihaljevi, B&, Orli, M&B !riteria for Evaluation of O#en -ource !lou !o"#ut'
ing -olutions& 0roc& of the I*I 4F22 55r International !onference on Infor"ation *echnol'
ogy Interfaces ,I*I.& ##& 25J'264&, <ubrovnik, !roatia ,4F22.&
6& /r"brust, B&, @riffith, =&, Lose#h, /&<& et&al&B / vie$ of clou co"#uting& !o""unications
of the /!M& 75, 7F'7E ,4F2F.&
7& 9a"#, 0&'>&, (atson, =&?&M&B LailsB !onfining the O"ni#otent =oot& 0roc& of the 4n In'
ternational -yste" /"inistration an ?et$orking !onference -/?E 4FFF ,4FFF.&
8& VM$are Inc&B *he O#en Virtual Machine For"at (hite#a#er for OVF -#ecification, ver'
sion F&I, htt#B++$$$&v"$are&co"+#f+ovfQ$hite#a#erQs#ecification&#f&
J& @olberg, =&0&B /rchitectural 0rinci#les for Virtual !o"#uter -yste"s, octoral isserta'
tion, >arvar University ,2IJ4.&
E& Bigelo$, -&L&B -tate of virtuali%ation an clou co"#utingB 4F22, htt#B++searchatacenter&
techtarget&co"+feature+-tate'of'virtuali%ation'an'clou'co"#uting'4F22&
I& !itri) -yste"s, Inc&B !lou&co" 4F22 !lou !o"#uting Outlook -urvey =esults,
htt#B++$$$&clou&co"+clou'co"#uting'outlook+survey&#f&
2F& Braley, <&, !ybenko II, @&, ;arson, L& et&al&B -u#erco"#uter $orkloa eco"#osition an
analysis& 0roc& of the 7th international conference on -u#erco"#uting& ##& 67E'68J& /!M
,2II2.&
22& !irne, (&, Ber"an, F&B / co"#rehensive "oel of the su#er'co"#uter $orkloa& 0roc& of
the Fourth /nnual IEEE International (orksho# on (orkloa !haracteri%ation ((!6 !at
?oF2E:75E& 26F'26E ,4FF2.&
24& ?u"rich, =&B / note on scaling the ;in#ack bench"ark& Lournal of 0arallel an <istribute
!o"#uting& 8J, 6I2'6IE ,4FFJ.&
25& @ray, ;&, 9u"ar, /&, ;i, >&B (orkloa !haracteri%ation of the -0E!#o$erQssj4FFE
Bench"ark& InB 9ounev, -&, @orton, I&, an -achs, 9& ,es&. 0erfor"ance Evaluation Metrics
Moels an Bench"arks& ##& 484'4E4& -#ringer Berlin >eielberg ,4FFE.&
26& Blackburn, -&M&, @uyer, -&Z&, >ir%el, M& et&al&B *he <a!a#o bench"arks& In Object Ori'
ente 0rogra""ing -yste"s ;anguages an /##lications OO0-;/& 62, 28I ,4FF8.&
27& Voras, I&B !ache server for istribute a##lications aa#te to "ulticore syste"s, octoral
issertation, Faculty of Electrical Engineering an !o"#uting, University of Zagreb ,4F22.&