You are on page 1of 57

Reliable Data Centres

Guide Version 2

Impressum
Publishedby: BITKOM BundesverbandInformationswirtschaft, TelekommunikationundneueMediene.V. Albrechtstrae10A 10117Berlin-Mitte Tel.:+4930.27576-0 Fax:+49.30.27576-400 bitkom@bitkom.org www.bitkom.org Contact: HolgerSkurk Tel.:+49.30.27576-250 h.skurk@bitkom.org Responsible BITKOM-committee: Editorialteam: Assistanteditor: Design/Layout: Copyright: Titelbild: HolgerSkurk BilianaSchnberg DesignBureaukokliko/AnnaMller-Rosenberger(BITKOM) BITKOM2010 AlejandroMendoza,istockphoto.com AKDataCentre&IT-Infrastructure

Thecontentsofthisguidehavebeencarefullyresearched,andreflectBITKOMsunderstandingatthetime ofpublication.Wedonotmakeanyclaimsregardingthecompletenessofthispublication,however.Despite takingthegreatestpossiblecare,wedonotassumeanyliabilityforthecontents. Thelatestversionoftheguidecanbeobtainedfreeofchargefromwww.bitkom.org/publikationen.BITKOM reservesallrights,includingthosegoverningthereproductionofpartsofthisdocument.

Reliable Data Centres

Reliable Data Centres


Guide Version 2

Content
1 Introduction 2 Theavailabilityofadatacentre 3 Theinfluenceofsafetystandardsonthedesignofdatacentres 3.1 ISO27001/ISO27002:2008 3.2 ITIL 3.3 Sarbanes-OxleyActandSAS70 3.4 Assessmentofstandards 4 ThebasisofITinfrastructure:therack 4.1 Secureservercabinets 4.2 Networktechnology 4.3 Reliabledatacentre 5 Energy,airconditioningandcooling 5.1 Powersupplycompanies(PSCs)electricityfeed-inanddistributioninthecompany 5.2 Powerdistributioninthecompany 5.3 Uninterruptiblepowersupply(UPS) 5.4 Backuppower 5.5 Service/maintenance 5.6 Airconditioning 6 Firesafety 6.1 Technicalfireprotection 6.2 Structuralfiresafetymeasures 7 Designofpremisesandsafetyzonesfordatacentres 8 Wiring 8.1 Currentsituation 8.2 Basicstandards 8.4 Structure 8.5 Redundancyandsecurity 8.6 Installation 8.7 Documentationandlabelling 9 Certificationofareliabledatacentre 9.1 Themanagementsystem 9.2 Certificationofamanagementsystem 10 Annex 11 Glossary 12 Acknowledgements 3 4 7 7 8 8 8 9 9 10 11 12 12 14 16 22 27 28 34 34 38 41 43 43 43 4 4 45 4 6 4 6 47 47 47 50 53 54

Reliable Data Centres

1 Introduction
Planning,developingandoperatingITinfrastructuresfor companiesvitalapplicationsinreliabledatacentresisa realchallenge.NotonlydoyouhavetoselecttherightIT equipment,thedesignofthedatacentreandtheresultingrequirementsforthesizeandtypeofconstruction, theelectricalpower,heatdissipation,wiring,safetyand availabilitynottomentionpurchaseandoperatingcosts allthesearedecisivefactors. Thisguideoffersassistancewithplanningandimplementingadatacentre.Itsupplementsexistingstandardsand specifications,whichcanalsobereferredtoforsupport. Theseareoftenextremelygeneral,however,whereasthis guideismorein-depthandprovidesconcreteinstructions onhowtodesignadatacentre.Italsosupplementsthe PlanningGuideforaReliableDataCentrematrixwhich, liketheguide,isavailabletodownloadfreeofchargeon theBITKOMwebsite.Thecontentsofthematrixarecontainedinextractsinthesectionsoftheguide.Theguide andtheplanningaidarenosubstituteforexpertadvice andsupportfromexperiencedadvisorsandspecialist planners,however.1

ThisguidewillbecontinuallyrevisedbytheBITKOMReliableDataCentre&ITInfrastructureworkinggroup.Aspartofthisprocess,itwillbeadaptedin linewiththelatesttechnicaldevelopmentsandotherboundaryconditions,andextendedtoincludetopicssuchastherequirementsoftheinsurance industry,riskmanagement,processesandservices,forexample.

2 The availability of a data centre


Therapidpaceofdevelopmentandtheintegrationof informationtechnologyinallareasofbusinessmeans thattoday,nocompanycanaffordthistechnologytofail. Justafewyearsago,manycompaniescouldsurvivethe failureoftheirITinfrastructure,evenforseveralhours. Today,thenumberofthoseforwhomcontinuousavailabilityoftheirITisindispensableisgrowingdramatically. AccordingtoastudybytheMetaGroup,a10-dayfailure ofkeyITsystemscanpermanentlydamageacompanyto suchanextentthatthereisa50%probabilitythatitwill disappearfromthemarketwithinthenextthreetofive years. AvailabilitytiersUSUptimeInstitute AsthedemandforavailabilityoftheITinfrastructure grows,therequirementsbecomeincreasinglystringent notjustfortheITsystemsthemselvesbut,aboveall, forcontinuallyensuringtheambientconditionsandthe powersupply.Asaresult,redundancyintheairconditioningandelectricitysupply,dualfeed-ins,andinterruption-freemaintenanceofsystemshaveestablishedthemselvesasthenormforhigh-availabilityITinfrastructures. Butbeforeyoustartworkontheplanningandlayoutof technicalcomponentsforachievingtheavailabilityyou desire,additionalreflectionbasedonriskanalysisand thechoiceofsiteisimperative.Thisshouldcoverpossible risksassociatedwiththearea,whichcouldinfluencethe probabilityofpotentialfailuregeographically(airtraffic, flooding,etc.),politically(war,conflict,terrorism,etc.)orin theformofneighbouringsites(premisesatparticularrisk offiresuchasservicestations,chemicalstoragefacilities, etc.).Furthermore,potentialwilfuldamagebyonesown employeesandfromoutsidethecompanymustalsobe takenintooverallconsideration. However,thedemandforhighavailabilitydoesnotonly entailgettingtogripswithpossibletechnicalsolutions, butrequirestheownertodesignandsetupacomprehensiveorganisationalstructure.Thisincludestheprovisionoftrainedservicepersonnel,forexample,ofspare partsoraserviceagreement.Preciseinstructionsonthe procedureintheeventoffailureoremergencymustalso bedefined.Inaddition,thisstructuremustenablerapid, preciseandtargetedcommunication,withtraceable documentationofevents. WhatmaximumITdowntimescanthecompany tolerate? Thesedays,whencreatingandexpandinganITconcept, orexamininganexistingone,itisvitaltofindouthow importanttheavailabilityofacompanysITinfrastructure is.Consequently,thequestiontobeaskedisthis:

Tier IntroExplanation classification duction


TierI 1960s Singlepowersupply channel,single air-conditioning supply,noredundant components99.671% availability Singlepowersupply, singleair-conditioning supply,redundant components99.741% availability Severalpowersupply channelsavailablebut onlyoneactive,redundantcomponents, maintenancepossible withoutinterruption 99.982%availability Severalactiveelectricityandchilledwater supplychannels, redundantcomponents,error-tolerant 99.995%availability

TierII

1970s

TierIII

Endof 1980s

TierIV

1994

Source:USUptimeInstitute:IndustryStandardsTierClassification

Reliable Data Centres

Thetermavailabilityreferstotheprobabilitythatasystemcanactuallybeusedasplannedatagivenmoment. Availabilityisthereforeameasurethatcanberecorded anddeterminedquantitatively.Ontheotherhand,there existqualitativeavailabilityclasses,asshowninthetable below,AvailabilityclassesaccordingtotheHV-Kompendium(HighAvailabilityCompendium)oftheBSI.Here, theavailabilityclassofaserviceisameasureofitsquality intermsofavailabilitybasedonhoursperyear. Asystemisregardedasavailableifitiscapableof fulfillingthetasksforwhichitisintended.Availabilityis measuredastheratiooffailure-induceddowntimetothe overalltimeofasystem.

Availability = (in percent)

1-

Downtime Uptime+ Downtime

100

Ifwecalculateavailabilityovertheperiodofayear,using theaboveformula,anavailabilityof99.99%,forexample, wouldcorrespondtoadowntimeof52.6minutes. 99%*87.66hoursperyear 99.9%*8.76hoursperyear 99.99%*52.6minutesperyear 99.999%*5.26minutesperyear 99.9999%*0.5265minutesperyear TheGermanFederalOfficeforInformationSecurity(BSI) hasdefinedthefollowingavailabilityclasses:

Availability class
AC0~95%

Norequirements approx.23weeks foravailability Normal availability Highavailability Lessthan90hrs. Lessthan9hrs.

Effects Description Cumulated, probable downtime per year

AC199,0% AC299.9%

AC399.99%

Veryhigh availability

Lessthan1hr.

AC499.999%

Maximum availability

Approx.5min.

AC5100%

Disaster-tolerant

Nomeasuresarenecessaryasregardsavailability. ImplementingtheITGrundschutz(formerlyknownas theITBaselineProtectionManual)fortheotherbasic valueswillhaveabeneficialeffectonavailability. Availabilityrequirementsaresatisfiedbythesimple applicationoftheITGrundschutz(BSI100-1andBSI 100-2) ThesimpleapplicationoftheITGrundschutzhasto besupplementedbytheimplementationofmodules recommendedforhighavailabilityrequirements,e.g. modulesB1.3ContingencyPlanningConceptandB1.8 HandlingSecurityIncidents,andariskanalysisonthe basisoftheITGrundschutz(BSI100-3). Implementationofthemeasuresrecommendedfor selectedobjectsinaccordancewiththeITGrundschutz,withparticularemphasisonbasicavailability, e.g.measureM1.28UPSintheserverroomorM1.56 SecondaryPowerSupplyinthedatacentre,supplementedbyHA(highavailability)measuresfromthe HACompendium ITGrundschutzwithadditionalmodellingonthebasis oftheHACompendium. TheITGrundschutzasthebasisisincreasinglysupplementedandreplacedbyHAmeasures. ModellingaccordingtotheHACompendium. TheITGrundschutzcontinuestoserveasabasisfor theaboveareasandothersafety/securityvaluessuch asintegrityandconfidentiality.

Table1:BSIavailabilityclasses

TheFederalOfficeforInformationSecurity(BSI)hasdevelopedaratingsystemfordatacentres:VAIR(VerfgbarkeitsanalysederInfrastrukturinRechenzentrenAvailabilityAnalysisofDataCentreInfrastructure).Atwww. vair-check.de,datacentreoperatorscanenterthedetails oftheirdatacentreinfrastructureandcheckthereliability oftheirdatacentreanonymouslyandfreeofcharge.

Reliable Data Centres

3 The influence of safety standards on the design of data centres


Theplanninganddesignofdatacentresisgovernedbya largenumberofsafetystandards.Theyprovideplanners anddesignerswithassistanceontheonehand,butalso setoutvariousrequirements. Here,wewillpresentthemostimportantstandardsin thefieldofISMS(InformationSecurityManagement Systems)andITIL(ITInfrastructureLibrary),aswellasthe Sarbanes-OxleyAct. ThePLANphaseentailstheplanningoftheISMS.Above all,theareaofapplicationandboundariesoftheISMS aredefinedhere,andthenapprovedbythemanagement. Ariskanalysisisalsocarriedout.Thisascertainswhich systemsandapplicationsareimportantformaintaining acompanysbusinessoperations,andhowgreatdependencyonthemis.Basedontheresultsofthisanalysis, TheISO/IEC27001standardsthatcameintoforcein October2005covertheprotectionofconfidentialbusinessinformation.Itisbecomingincreasinglyimportant, asitprovidesafoundationonwhichcompaniesarein apositiontosatisfytherequirementsofthirdparties. Thesecanbelegalrequirements(suchasKonTraG,HGB andGoB,GoBS,GDPdU,BDSG,TMG,TKGandStGB), contractualrequirements(e.g.fromcustomers)orother requirements.ThisstandardreplacesthepreviousBritish standardBS7799-2,whichwasannulledwithoutreplacementinFebruary2006. Thespecialistbusinesstermusediscompliance,and denotesbothconformitytolawsandguidelinesand voluntarycodesofpracticebybusiness. TheISO/IEC27001assiststhecreationofaprocessfor buildingandoperatinganInformationSecurityManagementSystem.Thisprocessofcontinuousimprovement takesplaceinfourknownsteps:Plan,Do,Check,Act, withwhichyouwillbefamiliarfromISO9001(Quality Management). Inthefourthphase(ACT),measuresdefinedasimproveMajorassistanceisalsoprovidedbytheGrundschutz manuals(catalogues),whichhavebeendevelopedbythe mentsareputintopractice. TheDOphaseincorporatesconcretemeasurestominimiseanddetectriskswiththeaidofariskhandlingplan. TheISO27002:2008(formerly17799)isaCodeofPractice forInformationSecurityManagementand,assuch, providesvaluabletipsforcomplyingwiththeControls/ MeasurescontainedinISO27001.Itismoreorlessa guideforputtingtheISO27001intopractice.Section9 PhysicalandEnvironmentalProtectionalsosetsout measuresandsuggestionsforroomsandinfrastructures. CertificationisonlygrantedonthebasisofISO27001or BSIISO27001,basedontheITGrundschutz. DuringtheCHECKphase,regularmonitoringandperiodic auditsensurethatimplementedmeasuresarecheckedon aregularbasis,inorderthatpotentialimprovementscan beflaggedup(e.g.firesafetymonitoringmechanisms, firesafetytests). conclusionsaredrawnabouttherequiredlevelofprotection,andthedesiredavailabilityofsystemsandapplicationsisdetermined. BSI(FederalOfficeforInformationSecurity)overmany years,andconformstoISO27001,basedonITGrundschutz.Themodulesinthecataloguesareextremely valuableintheimplementationofanInformationSecurityManagementSystem.

3.1 ISO27001/ISO27002:2008

3.2 ITIL
ITServiceManagementisanimportantconsideration duringtheplanningandoperationofareliabledata centre.BestpracticerecommendationsforITservice managementhavebeeninexistencesincetheendofthe 1980s,whentheBritishgovernmentsCentralComputer andTelecommunicationsAgency(CCTA,nowpartofthe OfficeofGovernmentCommerceOGC),publishedthe firstelementsoftheITInfrastructureLibrary(ITIL).These guidelines,whichhavebeensetoutinwriting,rangefrom detailedadvicetoindividualprocesseswithintheITIL, fromrulesofproceduretothenewlypublishedstandard ISO20000(formerlyBS15000). Forexistingdatacentres,customersalsouseaservice managementsysteminaccordancewithITILasorientation.Computerservicecentresareoftenconfrontedby invitationstotenderthatrequiretheparticipatingcompaniestohaveITIL.Twokeyareasarealwaysincluded: Service-Support Service-Delivery ThissetofrulesappliestoallITorganisationsinallcompaniesnomatterwhattheirsize.

incorrectfinancialstatements.Companiesaremonitored byapprovedauditorsinaccordancewiththeSAS70 audit.ThisinturnislargelybasedontheCobit4.1rules fromtheISACA(USA). IfacompanythatmustcomplywithSOXhasoutsourced individualsystemsoritsentireIT,forexample,theSAS70 auditmustalsobeappliedtotheproviderinquestion, althoughtheresponsibilityalwaysremainswiththe contractor.Inthiscase,eitherthecustomersauditorscan carryoutcheckstoSAS70inthecomputerservicecentre, orthecentreitselfcanarrangetohavecontrolscarried out.Theauditorsreportmustnotexceedthetimeof thecustomersannualfinancialstatementbymorethan sixmonths.Forthisreason,SOXcontrolsmustbasically beconductedtwiceayear,whichentailsconsiderable expenditure. Atinternationallevel,possibleconflictsbetweenthe Sarbanes-OxleyActandnationalregulationshavebeen discussed.Anysolutiontotheseconflictsisstillfarfrom clear.AEuroSOXisinprogress,however.Inaddition,the IDW(InstitutderWirtschaftsprferGermanInstitute ofPublicAuditors)iscurrentlybasingitsinstructionsfor compliancewithtestrequirementsonCobit4.1.

3.3 Sarbanes-OxleyActandSAS70
TheSarbanes-OxleyAct(SOX)isaUSlawintendedto improvecompanyreporting,andwasissuedinresponse tothebalancesheetscandalsinvolvingcompaniessuch asEnronandWorldcom.Thislaw,whichhasbeeninforce since30July2002,doesnotonlyapplytofinancialdata,it alsodemandssecurityinIT. Firstly,thelawapplestoallcompanieslistedonthe Americanstockexchange.Thenitalsoappliestonon-US companiesthathaveaparentcompanyorsubsidiary listedontheAmericanstockexchange. TheSarbanes-OxleyActstipulatesthatcompanyprocessesmustbedescribedanddefinedandmonitoringproceduresestablished,withtheaimofminimisingtheriskof

3.4 Assessmentofstandards
Thestandardsdescribedabovearefrequentlychecked bycustomers,certificationsocieties,auditorsandother institutions.WhethertheSarbanes-OxleyActandSAS 70makeadatacentremorereliablemaybedebatable however,thegeneralrequirementscontainedinISO/IEC 27002:2008andISO/IEC27001:2005governingmeasures toimprovesecurityarethoroughlyjustifiedandsensible. ITILandISO20000demonstrablysafeguardandimprove processesinadatacentre.Publicsectorcontractorsoften requirecertificationtoBSIhere,however,theexpenditurefordocumentingandoperatingtheISMSisvery high.ThebestcombinationisISO27001withreferenceto theITGrundschutz(whereapplicable),notcertificationby theBSIinBonn.

Reliable Data Centres

4 The basis of IT infrastructure: the rack


Whetheryouhaveaseparatedatacentreoranindividual servercabinet,thebasisforsecureaccommodationof ITsystemsisalwaystherack.Sinceinmostcompanies ITsystemsconsistofstandardised19-inchcomponents, scalable,flexibleracksystemsinthisdesignofferthebest choicewhenputtingtogetherastableandresilientIT infrastructure.Itensuresperfectinteractionbetweensystemandsupportcomponents,suchasthepowersupply andairconditioning.Acompanysdecisionastowhether tohouseitsITsystemsinadedicateddatacentreorto haveastandalonesolutioninindividualservercabinets dependsontheITandthestructuralrequirements.In bothcases,however,thesamefiresafetyandothersecuritystandardsapply,fortheyareintendedtoprotectICT systemsanddatafromtheinside. Anotherimportantaspectistheroutingofcables.As increasinglyfastnetworksareusedincombinationwith copperwiring,itisessentialthatpoweranddatacables areroutedseparately,topreventthemfromcausingone anotherinterference.Whenselectingarack,itisalso Asecureservercabinetshould,ifpossible,bemodular indesign.Itprovidesthecompanywithreasonable securityatamanageablecost.Amodularcabinetcanbe dismantled,convertedormovedtoanotherlocationif necessary.Thisflexibilityalsooffersadvantagesinterms oftransportandre-erectionifacompanyrelocates. Modularityisalsoadvantageouswhenwishingtoexpand housingsolutionsorair-conditioningconcepts. Whenplanningasecureservercabinetaswithareliable datacentrethefollowingcharacteristicsarenecessary forensuringthatsystemshavecomprehensivesecurity andavailability: Ensuringconstanttemperatureandairhumidity throughprecisionairconditioning Safeguardingtheelectricitysupplybymeansofan uninterruptiblepowersupply(UPS)andanadditional backuppowersupplyifnecessary Protectionagainstaccessbythirdparties(locking systems,network-monitoredrackaccess,biometric dataacquisition) Asservercapacityandthepackingdensityintherack increase,theimportanceofventilationconceptssuchas perforateddoorsandpartitionsbetweenhotandcold areasoftherackhasgrownenormously.Otherenergyoptimised,performance-enhancingsolutionscanbe achievedbyhotorcoldhousingconceptsthatformpart oftherack.Inthecaseofextremepowerlossesinthe rack,liquid-cooledsolutionsintheformofair/waterheat exchangersareindispensable. vitaltoensurethatthepowerdistributionsystemcanbe easilyintegrated,forintheendareliablepowersupply isaprerequisiteforavailableIT.Fuse-protectedlowvoltagesub-distributionmustalsobeavailable,aswell asaflexiblepowerdistributionsystemintherack,which canbeconnectedbothtothemainspowersupplyand anuninterruptiblepowersupply(UPS).Modernsolutions haveupto88kWinarack.Thisismadepossiblebyfour independent,three-phaseincomingfeeders,whichguaranteeareliablepowersupplyeveninthefaceofmore exactingrequirements. Thesubjectofstabilityisavitalaspectinallracksolutions.Duetothehighpackingdensityofmodernserver systemsandstoragesolutions,aswellasthepower supplyunitstheycontain,serverrackswithaload-bearing capacityofupto1,000kgmayberequired,depending ontheapplication.Consequently,thebasesofunitsand slidingrailsmustalsobedesignedforhighloads.Bases andrailscancurrentlywithstandaweightofupto150kg each. Firepreventionandreaction Modulescanbeintegratedinacentralmonitoring andmanagementsystem

4.1 Secureservercabinets

Bothelements,safeguardingthepowersupplyandtheair conditioning,canbemonitoredbysensorsintegratedin theinfrastructure.Whetherwiredorwireless,thesesensorsrecordtheservershumidityandtemperature,and alsotheirpowerinput.Amodern,sensor-basedmonitoringsystemmayalsotakeonthetaskofaccesscontrol.

4.2 Networktechnology
Acompleteexaminationofdatacentres,withspecial considerationofsecurityaspects,cannotfailtoinclude thesubjectofnetworktechnology.Manycompanieshave alreadyswitchedtheirtelephonesystemstoVoiceoverIP (VoIP).Virtualisedclientsarethenextstep.Thisenables anincreasingnumberofbasicservicescriticaltobusiness totakeplaceviadatalineswhich,withPoweroverEthernet(PoE),alsosupplypowertotheterminaldevices.As networktechnologybecomesincreasinglyimportantfor problem-freebusinessoperations,securityrequirements aretighteningupinthisarea,too. Aswiththeservers,fornetworktechnologytherack providesthebasichousing.Sincetheseactivecomponentsarealsostandardisedto19inches,networkcabinets aregenerallybasedonthesameplatform.Requirements arealsocomparableintermsofstability,firesafetyand accesscontrol.However,sincethenetworkinfrastructure installedinthebuildingtendstobedesignedtolastmore than10years,long-termplanningisrecommendedwhen purchasingnetworkcabinets,andflexibilityofaccessoriesisdesirable.Thisway,futuredevelopmentswillalso betakenintoaccount.Itmustbeborneinmindthatthe interiorstructureofracksdiffersgreatlyfromonetothe next. Thefrequentswitchingbetweenthevariousconnections ofthenetworkcomponentsmeansthatcablesinnetwork cabinetshavetobereroutedconsiderablymoreoften thanisthecaseinservercabinets.Thesemovements,also knownasMACs(Move,Add,Change),aswellasincreasingportdensity,lendspecialimportancetothetask ofcablemanagement.Thisstartswiththeroofpanels andbases.Simplecableentryatthesepointsfacilitates retrofittingandkeepscablerunsshort.Routingductsand guidepanelsensureorderlyfinedistributionintherack. Atthesametime,cablemanagement,inparticular,must placegreatimportanceonthestabilityofcomponents. Formoderncurrent-carryingnetworkcablesaremuch moreexpensiveandrigidthantheirCat-5predecessors.

4.1.1 Makinganinventoryoftheserver cabinet


Indatacentresespeciallythoseofacertainsizeitis hardtomaintainanoverviewofallinstalledhardware components.Althoughitispossibletocommunicate withanyintelligentITdevicetoday,theequipments physicalconnectiontotherackandtherelevantheight unitisanothermatter.Thestructureofequipmentin theindividualcabinets,withservers,fans,UPS,etc.is oftennottransparent.Asaresult,makinginventories andcontinuallyupdatingdataaboutthearrangementof componentsinthedatacentreisexpensiveand,inmost cases,time-consuming.Inmanycases,existingmanuallywrittendocumentationisnotcheckedforaccuracy. However,thisinformationisvitalformakingdecisionsin theeventoffailure. Anotherproblemisthehalf-lifeoftheacquiredinformation,fortheserecordsorupdatesstillonlyeverrepresent asnapshotofthedatacentreinventory.Yetefficientrack assignmentandtransparentcomponentmanagement requirecontinual,up-to-dateandthereforereliabledata. Toensureconstantaccesstoup-to-dateinventorydata, moderninventorysystemsexistdirectlyintherack,which registertheequipmentofthe19levelcompletelyand withoutcontact. Ontheonehand,therackconfigurationsarevisuallydisplayedonawebsitebelongingtothemonitoringsystem inquestion;ontheotherhand,theusercanopenalistor import/exportdatausingtheXMLfileformat.Thesedata recordscanthenbefurtherprocessedinexternaldatabases,forexample,andconsiderablyfacilitateeveryday workduringdatacentreoptimisation.

10

Reliable Data Centres

Onesubjectthatiscurrentlygaininginimportancewhen itcomestonetworkcabinetsisairconditioning.Switchesandroutersarebecomingevermorepowerful,and producemoreandmorewasteheat.Forthisreason,here, too,possibilitiesforexpansionhavetobeconsidered.The spectrumrangesfrompassiveairconditioningviaroof panels,ventilationattachmentsanddual-walledhousings tofans,rightuptoroofcoolingunits.

Amodular(andhenceextendible/modifiable)fireproof securityroom,certifiedasfaraspossible,mayformpart oftheequipmentofthedatacentre.Theuseofastable, multilayeredfiresafetydoorwiththesameprotection valuesasthesecurityroomisalsocompulsory.Other constructionssuchasahermeticallysealedceiling,wall andfloorsystemtoprotectagainsttheingressofsmoke orwater,forexample,oramultistageveryearlyfiredetectionsystemwithmultiplesamplingpoints,including intheraisefloor,arenowthestateoftheart.Theseare joinedbyanappropriatelydimensionedautonomousfire extinguishingsystemwithpressurereliefandventilation ductdampers,personalaccesscontrolusingcardreaders orbiometrictechnology,anddatacentreperipherymonitoringviaLANvideotechnology. Toensuretheflexibleexpansionofdatacentres,itisa goodideatoworkwithplannersandsupplierswhocan ensurethelong-termavailabilityoftherequiredproducts.

4.3 Reliabledatacentre
Inadditiontothebasicrequirementsforareliabledata centrementionedabove,numerousprojectdetailshave tobeclarifiedinrelationtoconstruction. Firstofall,apreciseanalysisofrisksandweakpointsin thecompanymustbecompiled,flagginguppossiblerisks forITsystems.Responsibilityforplanningandbuilding adatacentre,personswithaccessauthorisationand regularsafetychecksbyindependentauditorsmustbe covered. Manypeopleareresponsiblefortheplanning,constructionandoperationofadatacentre.InadditiontoITspecialists,theseincludebuildingexpertssuchasarchitects andconstructionengineers,specialistplannersforair conditioning,energyandactivedefence,thedepartment inchargeoforganisationand,lastbutnotleast,the managementboard. Thephysicalrequirementsfacingadatacentredonot consistonlyofpureIT-relatedmatters,suchasthenumberandtypeofserversandnetworkandstoragedevices, butalsoactiveandpassivedefence.

11

5 Energy, air conditioning and cooling


5.1 Powersupplycompanies(PSCs) electricityfeed-inanddistributioninthe company 5.1.1 Currentsituation
Theelectricitysupplyisofmajorimportancefortheoperationofservercabinetsorentiredatacentres. Whenbuildingadatacentre,ready-madepowersupply Thepowersupplychainbeginsatthepowersupply companies,whichgenerateandsupplyprimaryenergy. Theprimaryenergygeneratorstransporttheelectricity throughcablesviahigh-voltagemaststomedium-voltage substations.Fromthemedium-voltagesubstations,the electricityisconveyedviaundergroundcablestotransformerstations.Transformerstationsaremostlysituated inlargebuildingsorattheedgeoftheroadonspecially designedpremises. Largedatacentreswithanareaofseveralthousand squaremetresoftenfeatureadditionalfeed-insviaa secondmedium-voltagesubstation,ensuringfullredundancyi.e.severalchannelstoincreaseavailabilityeven asfarasthepowerplantsthemselves. Possiblecausesforaninterruptiontothepowersupply couldbe: Technicalfaultsinequipment(e.g.serversandtheir components) Technicalfaultsinthepowerdistribution(e.g.cables, sub-distributionboards) Faultsinalternativepowersupplysolutions(e.g. standbygeneratingsystems,alsoknownasemergencydieselgenerators,battery-backeduninterruptiblepowersupply(UPS)systems) Faultsduetotheprocess(e.g.designfaultsinthepower supply,logisticalerrors) TheoutputfromtheUPSsystemsisroutedviathe UPSsub-distributors,andfromtheretotheindividual servercabinets.Junctionboxesordistributionboxesare installedintheraisedfloorforthispurpose.Fromthese branchesorjunctions,morecablesfeedtheelectricity tothepowersupplyunitsoftheserversinthecabinet. IfonlyoneUPSsystemisinstalled,powersupplyunitsA andBhaveajointsupply;iftherearetwoUPSsystems, theirsupplyisseparate.A2xNsupplythereforeincreases availability. solutionscannotsimplybepulledoutofahat.However,basicprincipleshavebeensetout,whichmustbe adaptedtoindividualcircumstances.Thechallengefacing theplannerliesinputtingtheseprinciplesintopractice inaccordancewiththecustomer,hisrequirementsand wishesand,lastbutnotleast,hisbudget. Pastexamplesshowhowdramaticallysituationscan escalateifthepowersupplyfailsforlongerperiodsand nobackupsolutionisavailable.Thegeneralpowersupply cansuccumbtofailureforseveraldaysinlargeareas. Fromreportsofthedamagecausedinsuchcases,itis easytoseehowessentialanautonomouspowersupply is,particularlyinareasofcriticalimportancetocompanies,suchasIT.

5.1.2 Howtheinfrastructurefunctions
Thepowersupplyfunctionsbymeansofringmains,the electricityfromwhichistransformedto400Vinthe transformerstationsandwhichisconveyedtothedata centrethroughcablesorbusbarsviathelow-voltage maindistributionandutilitypowersupplynetwork.The normalmainssub-distributionalsosuppliestheuninterruptiblepowersupply(UPS)systemswithelectricity.

12

Reliable Data Centres

5.1.3 Recommendedequipmentfordifferentdowntimes Data centre category Permitted data centre downtime* Electricity feed-in from PSC Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW
Standard Standard Redundantfeed-ins Redundantfeed-ins Redundantfeed-insfromdifferenttransformerstations

Data centre / server room 500 to 2500 W/m2

A B C D E

72h 24h 1h 10min 0min

Table2fromtheBITKOMmatrixPlanningGuideforaReliableData CentreElectricityfeed-infromPSC

GreatersecurityisofferedbyCategoryC.Here,thepower supplyisredundantevenfromthelow-voltagemain distribution.Ifasupplychannelupstreamofthelowvoltagemaindistributionfails,electricityisautomatically suppliedviathesecond,redundantchannel.Iftheprimary energysuppliersuffersfailure,thepowersupplyisstill assuredbythemobilestandbygenerator. UnlikeCategoriesAandB,CategoryCfeaturesasecond UPSwithtwodedicatedUPSsub-distributors.This ensuresaredundantsupplyfromtheUPSsystemstothe serverpowersupplyunits. CategoryDdiffersfromCategoryCinthatthesupplyis ensuredbytwodifferenttransformerstations,eachwith theirowndownstream,redundantinfrastructure.Moreover,intheeventofapowercutattheenergysuppliers end,electricityissuppliedbythestationarystandbygenerator.Thiscategorypromisesaveryhighlevelofsecurity. CategoryErepresentsthegoldstandard.Notonlydoes itofferadditionalredundancyviaasecondstandbygenerator,italsoprovidesanadditionalfeed-infromanother independentmedium-voltagesubstation.However,in almostallcasesthesecondcablefeedfromanother medium-voltagestationfirsthastobeproducedbythe responsibleprimaryenergysupplier.Thiscanmeanthat severalkilometresofnewcablemustberoutedtothe

CategoriesAandBarecurrentlyinplaceinmanysmallto medium-sizedbusinesses,oftenevenwithoutanypossibilityoffeed-inforamobilestandbygenerator.When examinedmoreclosely,however,thisversiondoesnot provideanyrealsecurity,andplacesitstrustsolelyinthe energysuppliers.WearealwayshearingtheoldsayingIt wonthappentome.Anditmaynothavehappenedyet. Butifasinglelinkinthesupplychainfails,thisinterrupts theentiresupplyfromthePSC,andelectricityhastobe suppliedbytheUPSsystem.Moreover,thebackuptime ofaUPSsystemisgenerallyseverelylimited.Itdepends onthenumberofbatteriesinstalled,andthepowerthat hastobeprovided.AUPSsystemisgenerallyunableto bridgeapowercutlastingmorethan30minutes.Inthis case,afunctioningcomputershutdownroutineshould automaticallybeinitiated,messagestransmitted,data saved,applicationsclosedand,finally,computersproperly shutdown. Plannersmustthereforetakeparticularcaretoensure thatthebackuptimeoftheUPSsystemislongerthanthe timerequiredtotransportandactivateamobilestandby generator.Asarule,batteriesareusedfortheabove constellations.

13

datacentrepremises,whichisextremelycost-intensive andshouldbetakenintoaccountduringcosting. Regularinspectionandmaintenanceoftheentireinfrastructurebyqualifiedpersonnelandcompliancewith regulationsandguidelinesonoperationofthesystems areallindispensableformaintainingavailability.

cut-outsmaybeused.Acabinettypicallycontainstwo separatepowerstrips,whichpermitredundantoperation ofITsystems. Overviewofpoweroutputclasses:

Phases
One One Three Three

Maximum amperage
16A 32A 16A 32A

Maximum power output


3.6kW 7.2kW 11kW 22kW

5.2 Powerdistributioninthecompany 5.2.1 Currentsituation


Powerdistributorsconveythepoweroftheutilitypower supplynetwork,thegeneratorandtheUPStotheequipment,systemsandlighting.Twodistributionsystemscan beemployedtoensurehigheravailability.

(Furthercombinationswithtwophasesarealsopossible butareuncommoninGermany) Modernpowerdistributionunits(PDUs)alsofeature measuringorswitchingfunctions,aswellasanetwork connectionforextendedenergymanagement.Inaddition,variousmodelsofferenvironmentalmonitoringwith arangeofsensorsformeasuringthetemperatureandair humidity,forexample. SincemostITunitsindatacentresareinstalledin19cabinets,thequestionarisesastowherethepowerdistributorsshouldbesituatedandhowthepowercablesshould beroutedtothe19cabinets.PDUsareeitherinstalledin thewallorsurface-mounted,inseparatecabinetsorintegratedina19version.Thecablesareoftenroutedunder theraisedfloor,whichisalsousedasaconduitforcoldair. Thepassageofaircanbeimpaired,whileaccesstocables isdifficult.Thisproblemcanberemediedbyrouting cablesintothe19cabinetfrombelowviaaraisedfloor plate.Alternatively,cablescanberoutedalongtheceiling orwallsincablerunsorviapowerdistributionsystems,so thattheyhavetoenterthe19cabinetfromabove.IntegratedPDUshaveanadvantageinthattheyarelocatedin thevicinityofthepointofuse,andarethereforewithin easyreachofthe19cabinets.Cablescanberoutedon theroofofthe19cabinets,providedthatpoweranddata cablesareroutedseparately.

5.2.2 Howtheinfrastructurefunctions
Inthepowerdistributionsystem,thenormalmainssuppliesthebuildinginfrastructure,includinglifts,lighting butnotemergencylightingsystemstoVDE0108compressorsinDXairconditioners(DX=directexpansion), chillersandotherinstallations.Intheeventofpower failure,thispowersupplyisinterrupteduntilanavailable generatorstartsupandanautomaticswitchcausesitto restoretheelectricitysupply. Allpowerdistributionsystemsmustbeequippedwithan inputfuse.Thesizeandtypeofpowerdistributionsystem dependsuponthepowertobedistributed,thedesired numberofcircuitsandthepowerpercircuit.Pleaserefer tothetablebelow. Ideally,fuseprotectioniseffectedselectivelywithinthe powerstrip,i.e.theoutputsaremonitorednotbyan overallfusebutbyseveralfuses,eitherseparatelyorasa group.Asaresult,intheeventoffailureonlytheaffected outputorgroupisdisconnectedfromthemains,instead oftheentirepowerstrip.Eitherfusesorautomatic

14

Reliable Data Centres

SpecialattentionmustbepaidtothePDUstripsinthe cabinets.Thankstotheunitsmodern,compactdesign, thesedaysmanysystemscanbeinstalledinonecabinet. Inextremecases,acabinetmayhave42heightunitswith 42serversperheightunitandtwopowersupplyunitsper server,forexample.Atotalof84socketswouldhavetobe availableforthiscabinet.

toassignspecialistpersonneltoconverttheirsystems. Todaysmultipleoutletstripsallowmodulestobereplacedduringongoingoperation.High-endsystemssuch asthisgenerallyalsofeatureHTTPorSNMPmonitoring andmanagementoptionsanduseradministration,which guaranteesthatonlyauthorisedpersonnelcanconfigure theoutletstrip.Thesemodularsystemspermitbasic equipmentoftheracksbymeansofaverticalmounting railwiththree-phasefeed-in.Thevariousplug-inmodules cansimplybeinsertedinthisrail,considerablyreducing wiringandassemblytimeandexpenditure.

5.2.3.Intelligentmultipleoutletstrips
Formanagementatracklevel,transparency,orderand easeofhandlingarevital.Ideally,themultipleoutlet stripsusedinadatacentrewillhavedifferentplug-in modulesforcountry-specificsystems,forexample,which canbeusedbyuntrainedstaff.Inthiscase,international organisationshavetheoptionofusingthesametypeof multipleoutletstripinalltheirbranches,withouthaving

Finally,hostingcompanies,forexample,whichmust demonstrateahighdegreeofaccuracyinthedistribution ofenergycostsperserver(inonerack),cannowbenefit fromofficiallycalibratedsocketmodules.Calibrating measuringinstrumentsofthiskindarealsoavailablefor thePDU.

5.2.4 Recommendedequipmentfordifferentdowntimes Data centre category Permitted data centre downtime* Distribution Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW Data centre / server room 500 to 2500 W/m2

A B C D E

Standard, connectionofserversviaUPSandutilitypowersupplynetworkisrecommended Standard, connectionofserversviaUPSandutilitypowersupplynetworkisrecommended Redundantsetup(AandB) Redundantsetup(AandB) Redundantsetup

72h 24h 1h 10min 0min

Table3fromtheBITKOMmatrixPlanningGuideforaReliableDataCentreDistribution

15

Theformredundancytakesdependsonthenumberof powersupplyunitsintheITdevices.Agoodbasisforhigh availabilitywouldbetwopowersupplyunitsperdevice, whichareredundant.Then,intheeventoffailureofa powersupplyunit,theremainingpowersupplyunitis abletocontinuesupplyingtheITdevicenormally.These twopowersupplyunitsperdeviceshouldbesupplied bythePDUviatwoseparatePDUstripstotwoseparate powercircuits.Availabilitycanbefurtherincreasedby usingtwoseparatePDUs,whichreceivepowerfromtwo separateUPSsystemsviatwoseparatetransformersand twoseparategenerators.

besafelyisolatedwithoutanyilleffectsonothercabinets andITunits. Wherepersonalprotectionisconcerned,newrequirementsexistforadditionalprotectionforfinalcircuitswith sockets.Since01.06.2007,DINVDE0100-410:2007-06, ProtectionAgainstElectricShockappliestonewlyconstructedsystems.Modificationsandextensionstoexisting systemsmustalsobecarriedoutinaccordancewiththis standard. Theabovestandardstipulatesadditionalprotectionby meansofresidualcurrentdevices(RCDs)forallsockets ina.c.voltagesystems,ifthesysteminquestionisto beusedbylaypeopleorisforgeneraluse.Caremust betakentoensurethatfaults/damagesareeliminated immediatelybyanelectricalengineer,includinginconnectedelectricaldevices/consumables/equipment.This requiresapermanentmonitoringsystem,plusorganisationalmeasurestoensurerapidtroubleshooting. Acontinuousresidualcurrentmonitorsatisfiesthe currentstandardgoverningprotectivemeasuresandalso offersincreasedfireprotection,evenwithoutswitch-off byanRCD.

5.2.5 Protectivemeasuresandhigh availability


Datacentresaresubjecttomaximumrequirements regardingavailability.Theenergysupplymustbeassured permanentlytoreflectthis.Itthereforegoeswithout sayingthatthepowersupplytothedatacentreitselfand allareasinthesamebuildingtowhichdatacablesare routedmusttaketheformofaTN-Ssystem.Permanent monitoringofacleanTN-Ssystem(e.g.withRCMs) andforwardingmessagestoaconstantlymannedpoint, e.g.thecontrolcentre,isabsolutelyessentialforreliable operation.Messagesinformtheelectricalengineerabout whatactionneedstobetaken,sothathecantakeappropriateservicemeasurestopreventdamage. AllPDUsmustbeequippedwithaninputfusetoprotect thelines.Thesizeandtypeofpowerdistributionsystem dependsuponthepowertobedistributed,thedesired numberofcircuitsandthepowerpercircuit.Power circuitshaveeitheroneorthreephases.Typicalpower supplysizesare16A,single-phase(approx.3.5kW),32A, single-phase(approx.7kW)or32A,three-phase(22kW) forhigh-performancecabinets.Aparticularlydifficult subjectisthatofselectivefuseco-ordination,which enablesashortcircuitorshorttogroundinacabinetto

5.3 Uninterruptiblepowersupply(UPS) 5.3.1 Currentsituation


Thecausesofpowerfailureareoftenquitebanal:even simplevoltagefluctuationsortransientfailuresinthe electricitygridcanbeenoughtodamagehardwareor softwareortointerferewithittosuchanextentthat severeerrorsoccurinITprocesses.Irregularitiesinthe gridmayberare,buttheyaremorefrequentthanmany peopleassume.

16

Reliable Data Centres

UPSsystemsareemployedtopreventthepossible negativeconsequencesofbriefpowerfailures.Theyfilter outdisturbances,suchassurgesanddipsinvoltage,and bridgeinterruptionsinthemainssupply.Thisreduces transmissionerrors,computercrashes,bugsandlossof data.

5.3.2 UPSsystemtechnology
UPSsystemsdifferintheirdifferenttechnology.Themost frequentlyusedtechnologyisthestaticUPSsystem. Rechargeable(secondary)cells(accumulators)areused tostoreenergy.Whentwoormoreconnectedcellsare linkedinacircuit,thisisknownasasecondarybattery,or justarechargeablebattery.Intheeventofapowerfailure, theenergyfromtheaccumulatoriskeptreadyforcritical loadsbyastatictransformer(inverter)attheoutputof theUPSsystem.Thebackuptimeisdeterminedbythe loadandthecapacityoftheaccumulatorbatteries,butis typicallywithintherangeof10tomax.30minutes. ThesecondtypeoftechnologyusedisthedynamicUPS systemeitherwithorwithoutpistonengine.Dependingonthedesign,theenergyisstoredbyakineticbulk storagedeviceor,asabove,arechargeablebatterysystem. ThedynamicUPSsystemmakesthebatterysenergy availabletocriticalloadsbymeansofarotatingtransformer(generator)attheoutputoftheUPSsystem.With kineticstorage,thebackuptimedependsontheloadof theITunitsandthekineticenergyofthestoragedevice

Number of power failures and brief power losses p.a 60 50 40 30 20 10 0 0-10 ms 10-20 ms 20 ms-1s Most dangerous lurks here: in the range up to 1 second 1 s-1h > 1h

Duration of power failures

Figure1:Frequencyofmainsfaultsinrelationtotheiraverageduration

(massandspeed),andismeasuredinseconds. ThedynamicUPSsystemwithcombustionengine

Sincethe1980s,investmentinGermanpowersupply systemshasfallenbyapproximately40%.Inrecentyears, averagedowntimehasrisenfrom15minutesto23minutesperyear.Andthegrowinguseofregenerativeenergy meansthatelectricitygridsareincreasinglyreachingthe limitsoftheircapacity.Theriskofbottlenecksandblackoutsisrising.ThesearetheconclusionsofthenewVDE studyonpowersupplyquality.

combinesaUPSsystemwithastandbygenerator,andis thereforecapableofbridgingpowerfailuresoveralonger period.

5.3.3 Methodofoperation
StaticUPSversionsaredividedinto3categories.The classificationandassociatedmethodsfordetermining staticUPSsystemsaredefinedanddescribedinEuropean standardEN62040-3.

17

Mains failures
1. Powerfailure 2. Voltage fluctuations 3. Voltagepeaks 4. Undervoltage 5. Overvoltage 6. Surge 7. Effectsof lightning 8. Voltagedistortion (burst) 9. Voltageharmonics 10.Frequency fluctuations

Time
>10ms <16ms

EN 62040-3
VFD Voltage+Frequency Dependent

UPS-solution
Classification3 Passivestand-by-operation(offline)

Arresto solution
............................. .............................

4...16ms ............................. continuous continuous ............................. <4ms sporadic VFI Voltage+Frequency Independent Classification3 Doubleconversion operation(online) Deltaconverter ............................. Lightningandover voltageprotection IEC60364-5-534 ............................. continuous ............................. sporadic ............................. VI*) VoltageIndependent Classification2 Lineinteractive operation .............................

periodic

*)Alternativetechnologiesarecapableofovercomingmainsdisturbances1to9.

Table4:TypesofmainsfaultsandsuitableUPSsolutionsaccordingtoEN62040-3(ref.:UninterruptiblePowerSupplies,EuropeanGuide;Hsgr.ZVEI2004)

DynamicUPSsystemswithandwithoutcombustion enginearesubjecttoDIN6280-12. Indatacentres,staticUPSsystemswiththeclassification VFItoEN64040-3ordieselUPSsystemstoDIN6280-12 shouldalwaysbeused. StaticUPSsystemswiththisclassificationareavailable withanoutputrangeof10kVAto1600kVA,andcanbe

connectedinparalleluptoanoutputof4800kVA,dependingonthemake. DieselUPSsystemsareavailablewithanoutputof200to 1750kVA.Theyareabletocoverthelowandmedium-voltagerange.Numerousunitscanbeconnectedinparallel.

18

Reliable Data Centres

5.3.4 UPSredundancy
RedundancyistakesthefollowingformwhenUPSsystemsareemployed.

presentintheinputandoutputterminalsoftheUPSsystemeveninbypassmode.Thesystemcannotbereplaced withoutcuttingofftheloads.However,whenanexternal manualbypassisused,theUPSsystemcanbereplaced withouttheloadshavingtobeswitchedoff.Ifthemanual bypasstakestheformofmodularblocksconnectedin

100%

2N

100% 100%

parallel,orparallelblocks,itmustbedesignedformaximumconsumerload.

N+1

100% 100% 50% 50% 50% 2(N+1) 50% 50% 50% 50%

5.3.6 Energystorageunits
Kineticenergystorageunitsaredesignedordimensioned almostexclusivelybythemanufacturersofUPSsystems. Theachievablebackuptimesarerestrictedtoseconds,so thattheiruseislimitedtodieselUPSsystemsorcombinationswithfast-startstandbygenerators. Theelectrochemicalstorageunitsusedinconnection withUPSsystemsincludeleadandnickelcadmium batteries.Useoflithiumionbatterieshasnotbecome widespread.Nickelcadmiumrechargeablebatteriesare relativelyinsensitivetoincreasedambienttemperatures, buttheiruseiscontentiousbecauseofenvironmental pollution. TheenergystorageunitmostcommonlyusedinUPSsystemsistheleadbattery.Leadbatteriesareverysensitive totemperature.Lowtemperaturesreducethebattery capacityandthereforethebackuptimeorpower,while hightemperaturesreducethebatterysusefullife.The optimumambienttemperatureis20C. Theusefullifeofbatterysystemsvariesdependingonthe technology,thematerialsusedandotherfactors.AccordingtoEurobat,usefullifeisbasedonanambienttemperatureof20Candlaboratoryconditions.Thefollowing usefullifefiguresarespecified: 35yearsstandardcommercial 69yearsgeneralpurpose 1012yearshighperformance 12yearsormorelonglife

N+1

50% 50%

Figure2:RedundancyinUPSsolutions

5.3.5 Electronic/manualbypass
Theelectronicbypasshasthetaskofswitchingtheloads withoutinterruptionfromthemainstotheinverterof theUPSsystem(safebusbar)andback.Intheeventof faultsduringinverteroperationorwithlargeamounts ofoverload,theelectronicbypassswitchestheloadback tothemainswithoutabreak.Dependingondesign,the electronicbypasscaneitherbeintegratedintheUPSsystem(singleblockandmodularblock)ortaketheformof anexternalcomponent(parallelblockwithexternalelectronicbypass).Anotherelectronicbypasscanbeconnected inparalleltocreateredundancy(N+1). EveryUPSsystemshouldbeequippedwithamanual bypass.ThisswitchesthevoltageoftheUPSsystemoff forthepurposeofserviceandmaintenancework.Ifthe manualbypassisintegratedinthesystem,voltageis

19

Toensureareliablepowersupply,thebatterysystem mustbeinspectedregularlyandreplacedbeforetheservicelifehaselapsed.Inaddition,itmustbeborneinmind thatthebatterylosescapacityoveritslife.Settingthe systemupforveryshortbackuptimescarriestheriskthat thealreadyagingsystemmaynolongerbeabletoprovidetherequiredpowerandmaycutofftheUPSsystem. Inareasofimportancetosafety,overdimensioning(factor 1.25)isrequired,toensurethatsufficientlyhighcapacityis stillavailableattheendofthebatterysservicelife. Iftheoperatordecidestodispensewithredundancyin hisUPSsystem,thebatterysystemshouldatleastbe dividedintotwosections.Theachievablebackuptimeof asectionisjustonepartoftheplannedbackuptime.This ensuresthatpoweroutages,atleast,arebackedupfor afewseconds.Thisisnotasuitableapproachforhighly availabledatacentres,however.

Furthermore,redundancyarrangementsandthepossibilitiesofinputandoutputsupplyareotherimportant considerations. Awholerangeofdifferentconceptsareavailablefor selectionforconstructingtheUPSsystem.Smaller,individualUPSdevicesarepopularforsafeguardingthesupply ofafewserversandITstoragesystems.Differentversions aretheUPScabinetortowerunitswithintegratedbattery,theexternalbatterypack,andtherackversionfor installationina19cabinet.LargerUPSsystemsassingle blockorparallelsystems,mostlywithexternalbattery cabinets,batteryracksorflywheelaccumulatorsystems, aremostlysetupandoperatedindedicatedoperating rooms.Here,amodern,liquid-cooledUPSsystemoffers low-cost,efficientanddirectUPSairconditioningwithout anyspecialroomairconditioning.Furtheradvantagesof operatingroomsdedicatedtoUPSarethatthickpower cablesincomputerroomsareavoided,asarebatteries, whichalsorepresentafirerisk.ModularUPSsystems combineservicefriendlinesswithrapidadaptabilityto frequentlychangingmaximumpowerrequirements. However,thenumberofmodulesusedhastobeconsidered,asavailabilitydecreasesthemorecomplexthe systembecomes.WhenUPSsystemsareinstalledin servercabinetsortaketheformofadedicatedUPSrack inroomssharedwithITequipment,theadditionalfire riskposedbytherechargeablebatteriesmustbetaken intoconsiderationinthedesignofalarmandfiresafety devices.

5.3.7 Recommendedequipmentfor differentdowntimes


ThemostimportantfactorstobearinmindwhendesigningaUPSsystemistherequiredelectricalpowerof connected,criticalloadsandtheinstallationconditions.To providebackuppowerduringoutages,anenergystorage unit,suchasabatterysystem(cabinetorrackwithdisconnectorsandfuses),oraflywheelstoragedevicemust beplannedthatsuitsthepowersupplyenvironment.

20

Reliable Data Centres

Data centre category

Permitted data centre downtime*

UPS Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW Data centre/ server room 500 to 2500 W/m2

Standard, min.backuptime10minutes,maximumdurationdependentoncontrolledshutdowntime ofservers Standard, min.backuptime(incl.ventilation)10minutes, maximumdurationdependentoncontrolled shutdowntimeofservers

Standard, min.backuptime1h,maximumduration dependentoncontrolledshutdowntimeof servers Standard, min.backuptime1h,maximumduration dependentoncontrolledshutdowntimeof servers

72h

24h

C D E

Redundant(N+1), backuptime1020min Redundant(N+1), backuptime1020min Redundant, backuptime1030min

1h 10min 0min

Table5:FromtheBITKOMmatrixPlanningGuideforaReliableDataCentreUPS

Ventilationunits,chilledwaterpumpsandcoolingunits/ compressorsmayneedtobesuppliedviaaUPSsystem, dependingontheenergydensityandselectedbackup time.Theamountofenergyrequiredforcoolingcanalso bemadeavailableviaastorageunit,insteadofcooling units/compressors.Nocoolingathighpowerdensities resultsinoverheating,causingITdevicestoshutdown withoutusebeingmadeoftheestablishedbackuptime foranyplannedshutdown.

Efficiencyandpowerlossforthevariousloadratios duringtypicaloperatingcycles(e.g.day/night,weekday/weekend),examinationofenergyefficiency DetailsonUPSfuseprotectionforthevariousoperatingmodes Effectsonthemainsinputandinputpowerfactor. However,theeffectsoftheconnectedloadwhen theUPSisinbypassmodemustalsobetakeninto consideration Availablebackuptimeofabatterysystemorflywheel accumulatoratactualload Maximumavailablebackuptimeofabatterysystem orflywheelaccumulatoratactualload Informationontheenergystorageunitandcharge/ dischargebehaviour Permittedambientparameters,suchasoperating temperatureandhumidity;implementeddegree ofprotection,firesafetyandair-conditioning requirements Noise Electromagneticcompatibility(EMC) Dimensionsandweights

5.3.8 Specialfeatures
ImportantplanningfeaturesfordimensioningandinstallingaUPSsystemare: Outputpowerratingattherequiredloadpowerfactor (thesedaysmin.0.95) Connectionvaluessuchasinput/outputvoltageand frequency Current,wirecrosssectionsandpossibilitiesforconnectingUPSinputsandoutputs

21

Thisguidecannotofferapreciseanalysisoftheseindividualfeatures,becausethecircumstancesofthedata centrepowersupplywillalwaysrequiredetailedplanning. Someinterdependenciestobeconsideredarementioned hereasexamples: Theimportanceoftheconnectedbattery/flywheel accumulatorforthebackuptimeonpowerfailure,if anemergencygeneratorisavailable. Considerationoftheinputpowerfactorwhendimensioninganemergencygenerator.OperationviaUPS powerelectronicsandoperationviathebypassmust alsobetakenintoaccounthere. TheinfluenceoftheUPSoutputpowerfactoronthe possibilityofsupplyingmodernswitchingpower supplyunitswithelectricity,evenatfullload Thelimitationofpowerduringoperationathigh altitudes. Theimportanceofconsideringefficiencyoveratypical operatingcycle(fluctuationsincapacityutilisation),in ordertoobtainrealisticestimatesofrunningcosts ThepriceofaUPSdependsonequipmentdetailssuch asfilters,transformers,fans,electronicbypass,integratedorexternalmanualbypassanddifferentswitching concepts.Calculatingthepriceofbest-practicesolutions isextremelycomplexforUPSsystems,andrequirestimeconsuminganalysisofthecircumstances,boundaryconditionsanddependenciesandconsiderationofnumerous individualparameters.

inordertomaintaintheoperationofadatacentreandits technicalsystems,suchasairconditioning,electricityand safety. Theamountofpermitteddowntimehasmaximumprioritywhenplanningemergencypowersystems.Standby generatorsaredividedintovariousgroups,accordingly: Generatorswithoutarequiredtimefortakingonthe load.Systemsareputintooperationmanually.These systemsareunsuitableforautomaticoperationin datacentres. Generatorswitharequiredtimefortakingonthe load.Here,aninterruptionmustbelessthan15 secondslongbeforethegeneratorassumesthetask ofsupplyingpowerafterautomaticallybeingbrought intooperation.ADINstandardsetsouttherequirementsforgeneratingsetswithcombustionengines forsafetypowersuppliesinhospitalsandinbuildings andpremisesintendedforgatheringsofpeople.This standardshouldalsoberegardedasaminimum requirementforgeneratingsetsinthedatacentre sector. Generatorswithauto-reclosingintheformofstandby powersupplyunits.Here,theinterruptiontimemust belessthanonesecond.Thesesystemsarenolonger usedindatacentres,asaninterruptiontimeofless thanonesecondisnotnecessary. Generatorsforuninterruptiblepowersupplyinthe formofdieselUPSsystems.Here,onpowerfailurethe loadistakenonwithoutinterruption.

5.4 Backuppower 5.3.1 Generatingsetsforsupplying backuppower(emergencypower)inthe eventofpowerfailure


Electricitysupplierscannotguaranteeanundisturbed powersupplyatalltimesandallplaces,andpowersupply companies(PSCs)alwayswaiveanyliabilityintheirstandardcontracts.Therefore,briefinterruptionsorsustained powerfailuresmustbebridgedbybackuppowersystems,

5.4.2 Emergencypowersupplies
Inthetwolastcases,specialversionsofgeneratingsets arerequired,whicharestandbyunitswithanenergy storageunit.Thelattermustbecontinuouslyfed.The resultingoperatingcostsmeanthattheconsumerpays forhismorereliablesupply. Variousversionsofstandbyunitsexist,ascombinations madeupofdieselengine,flywheel,electricmachineand thenecessarycouplings.

22

Reliable Data Centres

Standbypowersupplyunitsarerequiredwheneveran interruptiontime,suchasthatcausedbytheuseof simpleemergencygenerators,wouldnotbeafeasible optionforreliablycontinuingongoingoperationatthe consumersbusiness. Themostcommonlyusedsystemsinthedatacentreare thosementionedsecond.Theinformationbelowrefersto thesesystems

Switch-on behaviour
Thestartingandswitch-onbehaviourofelectricmotors, transformersandlargelightingsystemswithbulbshas aninfluenceongeneratoroutputpower.Whereasynchronousmotorsareused,theapparentpowercanreach 6times,therealpower23timesnominalpower.The requiredgeneratoroutputpowercanbeconsiderably reducedbyphasedswitch-on.Allavailablemeasuresto limitstartingpowershouldbeusedtothefull.

5.4.3 Designingtheemergencypower system


Thefactorsbelowaredecisivewhendesigningtheoutput powerofthegenerator: Sumtotalofconnectedloads Demandfactor Startingcurrentsandstartingcosphiofloads Circuitfeedbackofloads(rectifiertechnologyofUPS systemsorfrequencyconverters) Permitteddynamicbehaviour Reserveforexpansions Supplementfordifferentambientconditions

Dynamic behaviour
Thedynamicbehaviourofthegeneratorwhenthefull loadisswitchedonandduringexpectedloadchanges duringoperationmustbeadaptedtothepermitted valuesoftheconsumers. Themotor,generatororbothmayhavetobeoverdimensionedinordertosatisfytherequiredvalues.

Ambient conditions
AccordingtoDIN6271,themotorreferencetemperature is27C.Ifoperatingtemperaturesexceedthis,themotor mustbeoflargerdimensions.Themotorsreduction factorsmustbeestablished.

Consumer power
Whencalculatingtheconsumerpower,bearinmindthat theapparentpowerandrealpowerhavetobestated.

Demand factor
Indatacentres,thegeneratoroutputpowermustbeset upwithademandfactorof1,asoperationofthedata centremustbemaintainedinbothsummerandwinter.

23

5.4.4Recommendedemergencypowersupplyasafunctionofthepermitteddowntimes: Data centre category Permitted data centre downtime* Emergency power Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW
Optional Optional Redundant, availabilityin15s,fuelreserve:24hours Redundant, availabilityin15s,fuelreserve:72hours Redundant, availabilityin15s,fuelreserve:72hours

Data centre/ server room 500 to 2500 W/m2

A B C D E

72h 24h 1h 10min 0min

Table6:FromtheBITKOMmatrixPlanningGuideforaReliableDataCentreEmergencyPower

Itispossibletohiregeneratingsetsfromyourpower supplycompany,whichguaranteetheemergencypower supplyduringmaintenanceandrepairsviaanexternal connection.Buthiredgeneratorsarenoanswertounforeseenpoweroutages,asyoucannotbecertainwhetheror nothiredgeneratorsareactuallyavailableatthetimeof occurrence.

Basic room requirements


Theroominwhichanemergencygeneratoriserectedis anelectrotechnicaloperatingroom.Itmustbeprotected toF90quality,andisafirezoneinitsownright.Ventilationaperturesmustbeprovidedtosupplythecooling andcombustionair,andforremovingheatedcoolingair. Theseaperturesmustleaddirectlytotheoutside.Rooms withoutexteriorwallsareunsuitablebecauseofthecross sectionsrequiredforventilation.Ifnecessary,ventilation ductsinF90qualitymustbebuilt,whichleaddirectlyto theoutside.Inordertoavoidshortcircuitsofair,supply andextractionaperturesmustnotbesituatedimmediatelynexttooneanother.Thegeneratorroommusttake theformofacatchbasin,withacircumferentialbarrier of10cmwith3coatsofoil-resistantpaint,forprotection againstfloodingandforenvironmentalprotection.This basinmustbemonitoredforleaks.Thesizeoftheroom mustallowanescaperoute1mwide.Thedoorsofthe roommustbeofatleastT30qualitywithapaniclock.

Room planning/detailed planning for emergency generators


Thefollowingdetailsmustbeconsideredduringroom planning/detailedplanning: Compliancewiththefollowingregulations(DINVDE, VDS,WHG,TANoise,TAAir,VAws,TRbF,VDN) Basicconstruction/versionofthegenerator(stationaryinstallation,containerorhood-typegenerator) Designofthetanksystem(daytankandstoragetank) Designoftheexhaustsystem Enginecooling(front-mountedradiator,tablecooler andtheuseofheatexchangers) Backuppowercontrol/switchingsystems Immissioncontrol

24

Reliable Data Centres

Applicable regulations
Theregulationsandlawslistedhereensurethecorrectfunctionofthesystem,operationalreliabilityand protectionoftheenvironment.Approvalauthoritiesmay imposefurtherrestrictionsandrequirements.Inallcases, dialoguewiththeauthoritiesshouldbeinitiatedatan earlystageoftheplanningphase. Noiseprotectionisanespeciallyimportantconsideration. Belowisalistofcontinuousemissionreferencevaluesfor emissionssitesoutsidebuildings. Industrialestate Businesspark Coreregions, villagesandmixed regions Residentialareasand smallhousingestates Purelyresidentialareas Spa/rehabilitationareas forhospitals/carehomes 70dB(A) daytime 65dB(A) daytime 60dB(A) daytime 55dB(A) daytime 50dB(A) daytime 45dB(A) night-time 50dB(A) night-time 45dB(A) night-time 40dB(A) night-time 35dB(A) night-time 35dB(A)

availableinthebuilding,orinstallationinthebuildingis unsuitableforotherreasons.Aswithastationarybuilt-in generator,powercanrangefromafewkVAtomanyMVA. Thethirdtypeisthehood-typegenerator.Thisismostly employedwhenpowerofafewkVAtoseveralhundred kVAisrequired.Itsadvantageliesinitsspace-saving design.Adisadvantageisthedifficultyofaccesstoall systemcomponentsintheeventofserviceorrepair.The diagramsbelowshowstandbygeneratorsinabuilding andinacontainer.

Exhaust gas system

Switchgear

Discharge air section

Machine unit

Batteries

Supply air section

Theresidualnoiselevelisassessedatanappropriate distance,notatthesiteofemission.
Figure3:Standbygeneratorinabuilding

Basic generator construction/version


Therearethreepossiblegeneratorconstructions/versions.Withabuilt-ingenerator,theentiresystemis installedinthebuilding.Interfacestotheoutsidearethe airsupplyandextractionapertures,theexhaustsystem and,ifnecessary,anexternaltablecooler.Inthisversion, powercanrangefromafewkVAtomanyMVA.Container generatorsarefrequentlyusedwheninsufficientspaceis

25

entiretankcontents.Iffuellinesareinplacebetween thedaytankandthestoragetank,theentirelengthof whichcannotbeseen,theymusthavedoublewalls.These double-walledpipes,thecatchbasinsandthejacketof double-walledtanksmustbemonitoredforleaks.The componentsofthetanksystemaredescribedbelow:

Design of the exhaust system


Thenominalwidthoftheexhaustsystemisbasedonthe nominalpoweroftheemergencygenerator,theplanned pipelength,thenumberandtypeofchangesindirection andtherequiredsoundabsorption.Exhaustsystemsof emergencygeneratorsarepressuresystems,andreach temperaturesofupto500C.Theymustberestrictedin suchawayastoexcludeanydangertopeopleandvaluables.Theexhaustsystemisdesignedasdescribedbelow:

Design of the engine cooling


Enginecoolingusingafront-mountedradiatorispossible uptoanoutputrangeofapprox.1150kVA.Thismeansthat allthecoolingairhastobeconveyedthroughthegeneratorroom.Atanoutputofapprox.800kVAandabove, partoftheheatfromtheenginecanbedissipatedviaa tablecooler.Thisreducestheamountofcoolingairthat hastobeconveyedthroughthegeneratorroom.Ifthereis adifferenceinheightbetweenthedieselengineandthe
Figure4:Standbygeneratorinacontainer

tablecoolerinexcessof10m,aheatexchangermustbe usedtoreducethepressureontheenginescoolingcircuit. Thedesignoftheenginecoolingandexternalcoolingsys-

Design of the tank system


Thekeyfactorthatdeterminestanksizeisthenecessary operatingtimeandthesystemoutputpower.Aquantityoffueloflessthan5000litrescanbestoredinthe generatorroom.Ifmorethan5000litresarerequired,a separatestorageroominF90quality,atankforabovegroundstorageoutsidethebuildingoranunderground tankmustbeprovided.Thedaytankisasingle-walled tankwithcatchbasin.Itmustbefittedinsuchaway thatstaticpressureisappliedtotheenginesinjection system.Thestoragetankmusthavedoublewalls,orthe storageroommustbedesignedasacatchbasinforthe

temsbasedonthedifferenceinheightbetweenthediesel engineandthecoolingcircuitisdescribedbelow:

Design of backup power control/switching systems


Eachgeneratingunitmustatleastbeequippedwitha backuppowercontroller,whichassumesthefollowing tasks: Monitoringthemainspowerforcompliancewiththe permittedtolerances Communicationwiththeenginemanagement system Startingandstoppingthedieselengine

26

Reliable Data Centres

Monitoringthegeneratornetworkforcompliance withthepermittedtolerances Monitoringtheengineparametersandcontrolling thenecessaryparameters Managingandcontrollingtherequiredauxiliary drives(motor-drivenventilationflaps,supplyand extractorfans,fuelpumps,solenoidvalves,leakage sensors,pipeheaters,coolantpreheating,starterbatterycharging,controlbatterycharging,etc. Managingthenecessarymainsandgeneratortie switchesforautomaticmode Chargingandmonitoringthebattery

5.5 Service/maintenance 5.5.1 Service/maintenanceofUPSsystems


Annualmaintenanceinaccordancewiththe manufacturersinstructionsbyspecialistpersonnel authorisedbythemanufacturerisaprerequisitefor maintainingproperfunction.Wearingpartsmustbe replacedbeforetheirservicelifeexpires,inaccordance withthemanufacturersinstructions. Lessemphasisisplacedonmaintenanceofcommonly

Thefollowingpossibilitiesexistforthepowerunit: Themainsandgeneratorswitchesarelocatedinthe backuppowercontroller Themainsswitchislocatedinthelow-voltagemain distributor,thegeneratorswitchinthebackuppower controller. Themainsandgeneratorswitchesarelocatedinthe low-voltagemaindistributor,thegeneratorpower ismonitoredbymeansofexternalvoltagetaps,and thegeneratorisprotectedbyastar-pointcurrent transformer. Hereisanexamplepowersupplydiagram:

usedsealedleadbatteries,asthesearemaintenance-free. Thetermmaintenance-freereferstotheinteriorofthe battery,however,andmeansthatdistilledwaterdoesnot havetobetoppedup.Ontheotherhand,allconnections andbatteryconnectionscrewshavetobecheckedto ensuretheyhavethecorrecttorque.Thevoltagesofthe individualbatteriesmustberecordedandloggedinthe chargeholdinganddischargingphases.Thecondition ofthebatterycanonlybeevaluatedonthebasisofthis data.Justasimportantisregularcleaningofthebattery system,topreventleakagecurrentorshortcircuits. Theavailabilityofappropriatespecialistpersonnelto remedyproblemsisanothersafetyaspectthatshouldnot beignored.

Switchgear room

Generator room G
U<

5.5.2 Service/maintenance/testrunsof theemergencygenerator


Annualmaintenanceinaccordancewiththe manufacturersinstructionsbyspecialistpersonnel

E Generatorswitch

Mains switch

authorisedbythemanufacturer,andmonthlytestruns, areprerequisitesformaintainingproperfunction.To ensurecorrectfunction,thesemonthlytestrunsmustlast atleastonehourandoperateat50%ofratedload.They maybeperformedbytheplantoperatorhimselfifhe

60%

Load

40%

hasreceivedadequateinstructiononthesystem.During atestrun,thesystemmustreachitsoperatingtemperature.Afixed,installedresistor,theconsumerstobe suppliedbybackuppower,ortheexistingpowersystem,

Figure5:Powersystemmonitoring/switchover

27

viamainsparalleloperation,maybeemployedasaload. Thelatterrequiresagreementandacceptancetestingon thepartofthepowersupplycompany,however.

ofheatwhich,justafewyearsago,wouldhavebeen unimaginableinsucharestrictedspace. Arangeofair-conditioningsystemsisavailableonthe

AswiththeUPSsystem,theavailabilityofappropriate specialistpersonneltoremedyproblemsmustalsobe considered.

market,tosuittheoutputandpowerlossi.e.wasteheat oftheICTcomponents.Withmorethan130W/cmper CPUequivalenttotwostandardlightbulbspersquare metrethehugetaskofdatacentreairconditioningis becomingclear:today,thispowerdensityresultsinheat loadsfarinexcessof1kWpersquaremetre. Theairconditioningofdatacentresgivesrisetofurther

5.5.3 Maintenance/testingofthe electricalinstallation


Electricalsystemsmustbeinspectedandmaintainedat regularintervalsinaccordancewiththeapplicableregulations(VDE0105)andstipulationsoftheprofessional association.Tothisaim,thesystemsmustbeoff-load,and appropriaterepeatmeasurementsandtestsconducted.If necessary,anA/Bsupplymustalreadybeincorporatedin planningoftheinfrastructure.Thisenablesthenecessary isolationandtestingtotakeplace.

challenges.Measurementsandexperiencefromthefield showthatpowerlossesofupto8kWinarackorhousing canstillbetackledwithtraditionalairconditioningusing coolingairintheraisedfloor,stillcommonlyusedinmany datacentres.However,airconductionthroughtheraised floorintheclassicmainframedatacentrecannotalways keeppacewiththesometimeshighdemandsthatprevail today. Afterdecadesduringwhichacoolingcapacityof1to3 kWper19cabinetwasfullyadequate,todayithastobe possibletoconsiderablyincreasethecoolingcapacityper rack.ModernITdevicesina19cabinetwith42height unitscanconsumeover30kWofelectricityandemitover 30kWofheat.Afurtherincreasecanbeanticipateddue todemandsforeverincreasingperformanceinasmaller installationspace.

5.6 Airconditioning 5.6.1 Currentsituation


Eachkilowatt(kW)ofelectricalpowerthatisconsumed byICTdevicesisreleasedasheat.Thisheatmustbe conductedoutthedevice,thecabinetand theroom,inordertokeeptheoperating temperatureconstant.Air-conditioning systemswithdifferentmethodsofoperation andperformanceratingsareemployedto dissipatetheheat.

5.6.2 Thechallengeofair conditioning


TheairconditioningofICTsystemsisacrucialfactorintheiravailabilityandsecurity. Theincreasingintegrationandpackingdensityofprocessorsandcomputer/serversystemsproducesquantities
Figure6:Basicsketchofacoldaisle/hotaislesolution

28

Reliable Data Centres

Inordertoimprovetheperformanceofexistingairconditioningsolutionswitharaisedfloor,todayICT componentsareequippedwithapartitionandarranged accordingtotheso-calledcoldaisle/hotaisleprinciple. Sectionsofthecoldorhotaislesareenclosed,toenable moreheatperracktobeemitted. Someofthekeyfactorsinfluencingthechoiceofan air-conditioningsolutionaretheexpectedmaximum powerloss,therunningcosts,purchasecost,installation conditions,costsofexpansion,futureproofness,costof downtimesandphysicalsafety.

Optimumconditionsintermsoftemperatureandrelative humiditycanonlybeachievedwithclosed-circuitair-conditioningunitsso-calledprecisionair-conditioningunits. Inthesesystemstheenergyusedisputtobetteruse,i.e. loweringthetemperatureofthereturnairisofprimary importance.Incontrasttothese,comfortair-conditioning unitsforhomesandofficesexist,suchassplitair-conditioningunits,forexample,whichpermanentlyusealarge proportionoftheenergytodehumidifythecirculating air.Thisgivesrisetocriticalroomconditions,butalsoto considerablyhigherrunningcosts,whichiswhytheuseof thesesystemsindatacentresisuneconomical.

5.6.3 Methodofoperationofclosedcircuitairconditioning
Inthepast,theroomconditionsfordatacentreairconditioningwerearoomtemperatureofapprox.22Cand roughly50%relativehumidity(r.h.).Today,duetothe prevalenceofthecoldaisle/hotaisleprinciple,wetendto referinsteadtosupplyairandexhaustairconditions,as theroomconditionsassuchnolongerapplytoallareas oftheroominquestion. Dependingontheapplication,thesupplyairconditionsin thecoldaisleshouldliebetween20andupto27Cthe relativehumidityofthesupplyairbetween40and60% r.h.Lowerhumiditywouldresultinelectrostaticcharging, highhumiditytocorrosionofelectricalandelectronic components.Itisabasicprerequisitethatoperating conditionswithverycoldtemperaturesbelow18Canda highhumiditythatresultsintheformationofcondensationonITdevicesmustbeavoided. Air-conditioningsystemsinICTroomsoperatewithavery highproportionofcirculatingair.Intheclosed-circuitairconditioningsystem,supplyaircooledbytheair-conditioningsystemcirculatestotheICTcomponentsandabsorbs theheat.Thisheatedairreturnstotheair-conditioning systemforrenewedcooling.Onlyasmallproportionof ambientairisbroughtintotheroom,andthisisusedfor airexchange.

5.6.4Designofclosed-circuitairconditioningsystems
Closed-circuitair-conditioningsystemsdifferconsiderably intheirdesign.Veryroughly,theycanbeclassedasDX systems(DirectExpansionsystems)andCWsystems (ChilledWatersystems).Bothsystemscanadditionallybe equippedwithIndirectFreeCooling.

DX systems
Coldairisgeneratedintheclosed-circuitair-conditioning cabinet,andassuchissituatedinthevicinityoftheICT room.Thecirculatingairiscooleddirectlybyexpanding refrigerant,andtheabsorbedheatisconveyedtoanoutdoorunit,theair-cooledcondenser. WhenIndirectFreeCoolingisused,anadditionalheat exchangerforFreeCoolingisinstalledintheclosed-circuit air-conditioningcabinet.Atlowambienttemperatures,onlyawater/glycolmixturecirculatesbetweenthe closed-circuitair-conditioningunitandtheoutdoorunit, ordrycooler.Thismixtureisheatedbythecirculatingair andcooledagainoutsidebythedrycooler.Thissystemis knownasIndirectFreeCoolingbecausethewater/glycol mixtureisinterconnectedbetweenthecirculatingaircircuitintheICTroomandtheambientair.Whenambient temperaturesarehigh,thesystemswitchestoDXmode, andtheheatofthecirculatingairisfirsttransferredtoa liquid-cooledcondenserintheair-conditioningcabinet,

29

beforetheheatisemittedintotheambientairbythedry cooler. DXsystemswithorwithoutIndirectFreeCoolingare employedforsmalltomedium-sizedinstallationswitha heatloaduptoapprox.500kW.

theflowofairismoreorlessforcedtoabsorbheatfrom theICTcomponentsonitswayfromtheraisedfloorback totheair-conditioningunit. Performanceisincreasedstillfurtherbyenclosingthe coldaisleorthehotaisle.Theheightoftheraisedfloor, viawhichthedatacentrecanbesuppliedwithcoldsupplyair,isalsoanextremelyimportantfactor.Thesupply airisguidedintothecoldaislethroughlouveredpanelsor grilles.AfterithasbeenheatedbytheICTequipment,the returnairisconveyedbacktotheair-conditioningsystem forrenewedcooling.

CW systems
Here,coldisgeneratedinchillers,whicharegenerally installedoutdoors.Awater/glycolmixturecirculatesin thebuilding.Theheatfromthereturnairistransferred tothecoldwater/glycolmixtureintheclosed-circuitairconditioningcabinet,whichiscooledbychilledwater.The heatedwater/glycolmixtureiscooledoncemoreinthe chiller,andreturnstotheclosed-circuitair-conditioning cabinet. Inthissystem,too,IndirectFreeCoolingispossible.To thisaim,anadditionalfreecoolingcoilmustbeinstalled outsideonthechiller.Atlowambienttemperatures, thewater/glycolmixturecirculatesbetweenthechilled water-cooledair-conditioningunitsandthefreecooling coil.Duringthisprocess,theheatofthecirculatingairis absorbedintheair-conditioningunitandemittedoutside bythefreecoolingcoil.Athighambienttemperatures, thewater/glycolmixtureiscooledbythechiller. CWsystemswithorwithoutIndirectFreeCoolingare employedforlargerinstallationswithaheatloadof approx.500kWormore.

5.6.6Directcoolingofracks
Whenheatloadsexceed1015kWperrack,directcooling oftheracksisnecessary.Directrackcoolingisachievedby heatexchangersintheimmediatevicinityoftheservers. Asarule,theseareliquid-cooled,andaresituatedeither belowornexttothe19racks.Upto40kWperrackcanbe dissipatedwiththismethod.

5.6.5 Rackarrangementandcirculatingair conduction


Therackarrangementandairconductionarefurther factorsinfluencingtheperformanceofclosed-circuitairconditioningsystems. Thesedays,19racks,inparticular,arearrangedaccording totheso-calledhotaisle/coldaisleprinciple,inorderto permitaswellaspossibletherequiredhorizontalflowof airaroundtheICTcomponents.Insuchanarrangement,
Figure7:Basicsketchofaliquid-cooledserverrack

30

Reliable Data Centres

Achilledwaterinfrastructuremustbeestablishedaround theracksforthispurpose.Liquid-cooledracksensurethe rightclimaticconditionsforeachservercabinet,andare thereforeautonomousintermsofroomairconditioning.

Inexistingbuildingswithalowceilingheight,liquidcooledserverracksrepresentagoodmethodofreliably dissipatinghighheatloadswithouttheuseofaraised floor.

5.6.7 Recommendedequipmentfordifferentdowntimes Data centre category Permitted data centre downtime* Air conditioning Servers cabinet up to 5 kW
Precisioncooling

Servers cabinet from 5 kW to 30 kW


High-performancecooling orliquidcooling Completecold/hotseparation High-performanceorliquidcooling,on-callservice,redundancy requiredforcabinetswithhigh powerdensity. UPSbackupforventilation, Completecold/hotseparation High-performanceorliquidcooling,Redundantarrangement, UPSbackupforventilation, Completecold/hotseparation High-performanceorliquidcooling,Redundantarrangement, UPSbackupforventilation, Completecold/hotseparation

Data centre/ server room 500 to 2500 W/m2


Precisioncooling

72h

B Precisioncooling

24h Precisioncooling, on-callservice

Precisioncoolinginredundant arrangement

Precisioncoolinginredundant arrangement Precisioncoolinginredundant arrangement, UPSbackupforventilation Precisioncoolinginredundant arrangement, UPSbackupforventilation, Emergencycoolingfunctionsviaanadditionalairconditioningsystem(e.g.well water,mainswater,ventilation system)

1h

Precisioncoolinginredundant arrangement,UPSbackup

10min

E Precisioncoolinginredundant arrangement,UPSbackup High-performanceorliquidcooling,Redundantarrangement, UPSbackupforventilation, Completecold/hotseparation

0min

Table7:FromtheBITKOMmatrixPlanningGuideforaReliableDataCentreAirConditioning

31

Thehigh-performancecoolingrequiredathigherpower outputsalsoincorporatestheneedforcoldorhotaisle enclosures.

Asthecoolingcapacityincreases,theair-conditioning systemsmustbeadaptedaccordingly.Thediagrambelow showsthepowerdensitiesthatcancurrentlybehandled bythedifferentair-conditioningsystems.

Airconditioningviatheraisedfloor,racksnotarranged toensurebestpossibleventilation

Airconditioningliquid-cooled withenclosedwarmaisles

Airconditioningviatheraisedfloor,racksarrangedin cold/hotaisles

Airconditioningliquid-cooled withenclosedcoldaisles

Airconditioningviatheraisedfloor, withenclosedcoldaisles

Airconditioningwithliquid-cooledracks (closedsystem)

Figure8:Air-conditioningsolutionsfordifferentcoolingloads

32

Reliable Data Centres

5.6.8Furtherrecommendations
Energy efficiency
Asenergycostscontinuetorise,thequestionofenergy efficiencyisofparticularimportanceduringtheplanning phaseofanair-conditioningsystem.Here,theoverall costsi.e.capitalinvestmentinthenewsystemandanticipatedrunningcoststhroughouttherunningtime,plus maintenancecostsmustbecalculatedandevaluated.A systemwithfreecoolinggenerallyentailshighercapital investment,buttheadditionalexpenseisoffsetovera shorttomediumperiodoftimeduetotheconsiderably reducedrunningandmaintenancecosts.

5.6.9Serviceconcept
Inanair-conditioningsystem,wearingparts,e.g.filter matsandsteamcylinders,areused,butsoarenumerous mechanicallymovedcomponents.Therefore,preventive maintenanceatregularintervalsisamust.Thetaskstobe undertakenaredescribedinDIN31051andVDMA24186, amongothers. Suitabletypesofservicecontractsareavailable,dependingonindividualrequirementsforair-conditioning availability.Thesecontractsdifferintermsoftheservices offered: Repaircontract Comesintoforceafterafailure orfault.Systemoperabilityis restoredbydownstreamcorrectiveservicemeasures Regularworkthatensures systemavailabilitythroughpreventiveservicemeasures. Combinationofrepairandservice,whichcombinespreventive andcorrectiveservicework Completemaintenance,with constantcoststhroughoutthe termofthecontractoffering budgetsecurity

Scalability
Inmanydatacentres,ICTsystemsreachtheirmaximum finalstageonlyafterafewyears.Air-conditioningsystemsmustthereforebescalabletoreflectthis.Consequently,achilledwatersystemoperatingatpartialload mayrunwithpoorefficiencyformanyyears,untilthe plannedefficiencyisatleastreachedinthefinalstage. Here,solutionsthatgrowtomatchrequirements,i.e. modularsystems,arealwaysadvantageous. Maintenance contract Fullmaintenance contract Servicecontract

Redundancy
Thesecontractscansometimesalsobecombinedwith Afailureprobabilitymustalwaysbetakenintoaccount, duetothenumerousmechanicalcomponentsinthe air-conditioningsystem.Redundantair-conditioning unitsassumethetaskofgeneratingtherequiredcooling powerintheeventoffailure,andofferalmost100% operationalreliability.Duringfailure,redundancyinthe air-conditioningsystemisgenerallynolongeravailable, andcorrectiveactionmustbeinitiatedwithoutdelay. otherwise,thereisariskthatitwillnolongerbepossible toairconditiontheroomintheeventoffurtherfailure. a24/7emergencyservice,andoffercontractuallyfixed onsitetime.Thisensuresthatcorrectivemeasuresare initiatedimmediately,andtheavailabilityofthesystemis restoredasquicklyaspossible.

33

6 Fire safety
Experienceshowsthatwemustexpecttheoutbreak offireatpracticallyanytime.Thefactthatnofireshave occurredinmanybuildingsfordecadesdoesnotprove thatnoriskexists,butisratherapieceofluckforthose concerned,whichcanbeexpectedtocometoanendat anytime. Eventoday,nothingneedstobeaddedtothispronouncementbyaHigherAdministrativeCourtfrom1987.This iswhyareliable,fastfiredetectionandextinguishing systemorfirepreventionsystemisanindispensableprerequisiteforthereliableoperationofadatacentre. Waterasanextinguishingagentisinappropriateinthe datacentre.Today,specialistfirmsintheindustryoffer suitableextinguishingsystemsforeveryrequirement. Whenconstructinganewdatacentreorretrofittingfire safetymeasures,however,preciseplanninganddesignof thesesystemsisvital. Thechoiceofthemostsuitablefiredetectordepends ontherequirementsoftheintendedsiteofuse.Inareas withoutparticulardetectionrequirements,fixedpoint detectorsaregenerallyused.Indatacentres,ontheother hand,acushionofheatedairupto1metrethickcanform undertheceiling.Thisisproducedashotairfromair-conFire,smoke,aggressivefumesandextinguishingwater representalatentdangerfordatacentres.Toensure safety,atechnicallyhigh-qualityveryearlyfiredetectionsystemcombinedwithveryhigh-qualityextinguishingtechnologyisnecessary.Onealternativeisfire preventionbyloweringtheamountofoxygenintheair. Here,nitrogenisintroduced,whichreducestheoxygen concentrationpreciselytoapresetvalue.Despitethis, roomsprotectedinthiswayremainaccessibletopersons withoutjeopardisingtheirhealth. Extinguishingfoam,ontheotherhand,cannotbeused asitwoulddamageITsystemsandtheirpowersupply units.Powderextinguishingsystemsarealsounsuitable fordatacentreswithverysensitiveequipment,asthe extinguishingprocessmaycausemoredamagethanthe fireitself.Today,therefore,gasextinguishingagentisused almostexclusively. FortheprotectionofventilatedITequipment,theVdS (VerbandderSachversichererGermanAssociationof PropertyInsurers)pointsoutthatearlyfiredetection usingfixedpointsmokedetectorsisdifficult,tosaythe least,orevenimpossible.Here,too,theuseofahighly sensitivesmokeaspiratingsystemisrecommended,as thisoffersthepossibilityofanearly,localisedreaction. ditioningsystems,powerpackfansandrackventilation systemsisdrivenupwards,andmaypreventsmokefrom reachingfixedpointdetectorsinsufficientquantitiesfor earlyfiredetection.Inareassuchasthis,inroomswitha lotofairconditioningandinbuildingswithhighceilings, fixedpointdetectorsquicklycomeupagainsttheirlimits. Wethereforerecommendinsteadtheuseofhighlysensitiveaspiratingsystems. Smokealarmsthatworkontheprincipleofscattered lightareprimarilyusedforfiredetectioninITrooms.Here, thescatterofarayoflightonanysmokeparticlespresent providesameasureofthesmokedensity.Thistechnique isemployedbothinconventional,pointsmokedetectors (opticalsmokedetectors,fixedpointdetectors)andin highlysensitivesmokeextractionsystems(aspirating detectors,activedetectors).Incontrast,theradioactive ionisationdetectorusedyearsagohasnowalmostcompletelydisappearedfromtheEuropeanmarket.

6.1.1 Methodofoperationofthe infrastructure


Fire alarms

6.1 Technicalfireprotection

34

Reliable Data Centres

Modernserverracks,whichhavetheirowncoolingsystem andareencapsulatedfromtheenvironment,representa newchallengeforfiresafetyinITrooms.Smokeproduced insidetheracksonlyescapesafteralongdelay,which rendersnormalroommonitoringwithsmokedetectors totallyineffectual.Furthermore,extinguishingagent releasedintheeventofafirecanonlypenetratethe servercabinetafteradelay.Forrackssuchasthis,manufacturersofferfiredetectionandextinguishingsystems forequipmentprotectionintheformof19plug-inunits. Oneofthesedevicescanprotectupto5racks.Forthis application,too,inwhichmajordilutionofthesmokecan beexpectedbecauseofthehighlevelofairconditioning, highlysensitive19aspiratingsystemsarenowusedto anevergreaterextent.Thelatterarealsoavailablewith integratedextinguishingsystem,externalextinguisher activationandintegrateddualdetectorinterdependency. Wheretheactivationofautomaticextinguishingsystems isconcerned,falsealarmsmustberuledoutasfaras possible.Tothisaim,firedetectorsforroommonitoring mustbeinstalledaccordingtotheprincipleofdualdetectorinterdependency;inotherwords,twodetectorsmust soundthealarmbeforeitisescalated.Singledetectors triggeronlytechnical,internalalarms.

AswithveryearlyfiredetectionforITrooms,here,too, veryearlyfiredetectionsystemswithaspiratingdetectorsofferclearadvantages,becauseinthiscasetheair samplescanbetakendirectlyinthecabinetandtheinlets ofthecirculatingairorairconditioning.Thesedays,the extensivemodularityofmodernsystemsmeanstheyare bestequippedforenhancingallstagesoffiredetection andfirefighting.Theyareideallysuitedfornew-generationEDP,serverandcontrolcabinets,whicharesealed withintegratedcoolingsystems.Theytaketheformof 19plug-inunitsforhighlysensitivefiredetection,emit controlsignalsforsoftshutdownandcanindependently extinguishafireonanobject.Anextinguishingunitcan eitherbeintegratedintheunitoractivatedexternally. PreferredextinguishingagentsareNovec1230andnitrogen.Nitrogenhasanadvantageinthatitdistributesitself extremelywellandhasalongerlife,whileNovec1230is verycompactandspace-savingtokeepinstock.Devices in19designareavailableonthemarketthathavebeen approvedbytheVdSfortheirfiredetectionability,and alreadysatisfythelatestAandBsensitivityclassesof DINstandardEN54-20.

Fire extinguishing systems


Theeffectivenessandreliabilityofafiresafetysystem

Forsomeyearsnow,manufacturersoffiredetectionand extinguishingsystemshavebeenproducingdevicesthat arecapableofindependentlymonitoringindividualracks orentirecabinetunitsforfireaerosols,andextinguishing them.Anoutbreakoffireinlivecabinetscanbetriggered byscorching,smoulderingorglowing,oftencausedby overloadedcomponentsandfaultycontacts.Afireona printedcircuitboard,forexample,coverslargeareasof hardwarewithsootandcauseswidespreadcorrosion.A furtherdifficultyarisesthroughthefactthatmodern,airconditionedracksworkwithextremelyhighairchange rates,whichimmediatelydiluteanysmoke,renderingit virtuallyundetectablebyconventionalsmokedetectors whilethefireisinitsinitialstages.

dependsuponplanningthattakesaccountofalltherisks, dimensioningandevaluationoftherequiredvolume. Gasextinguishingsystemsaredividedintoinertgases andchemicalgases.

Inert gases:
Extinguishingfiresbyoxygenremoval Thegasextinguishingmethodissuitablefordatacentres andtheirequipment.Itusesthetechniqueofoxygen removal.Here,theextinguishingagentreducestheproportionofoxygenintheroomairtosuchanextentthat thecombustionprocessissuppressed.Theextinguishing gasusedisaninertorchemicalgas.

35

Carbondioxide(CO2) Carbondioxideispresentintheatmosphere,butisno longerusedinnewsystemsbecauseoftheriskitposesto people. Argon(Ar) Argonisaninertgasandcanbeextractedfromtheambientair.Argonitselfisnottoxic,butcanleadtooxygen deprivationorariskoffireeffluentattheconcentration requiredtoextinguishafire. Nitrogen(N2) Nitrogenisalsocontainedintheatmosphere.Itiscolourless,odourlessandtasteless.Nitrogenisnottoxic,butcan poseathreatduetofireeffluentandoxygendeprivation.

Whatisofoverridingimportanceisthatthegasextinguishingtechnologyoffersthetypeofobjectandroom protectionneeded.Thereisnowaterdamage,andpowder andfoamresiduesarealsoavoided.Floodingtakesplace within120secondsinthecaseofinertgases,andwithin 10secondsinthecaseofchemicalgases.Fireextinguishinggasesarenotelectricallyconductive,andshort circuitsduringorafterextinctionareavoided. Withgasextinguishingsystems,itisimportanttobearin mindthatsecureandreliablepressurereliefviapressure reliefdampersisrequiredtodealwiththeoverpressure thatresultswhenthegasesaretriggered.Theapplicable regulationsstipulatethatthecalculatedmaintained concentrationmustbeupheldfor10minutes.Thepower supplymustbeswitchedoffthroughoutthedatacentre inordertopreventre-ignition.

Fire prevention system


ExtinguishinggasFM-200(HFC227ea),Novec1230 Thisversiondoesnotjustreactintheeventofafire,but Theseextinguishinggasestakeeffectbyabsorbing theheatintheflame.Aphysicalcomponentand,toa lesserextent,achemicalcomponent,areinvolvedinthis process.Thesegasesareapprovedforstorageinextinguishingagentcontainersinadedicatedfireextinguishingzone,undercertainpreconditions,whichmaybea decisivefactorwhenretrofittinganexistingdatacentre withoutafireextinguishingsystem. preventsit.Theoxygenreductioniskeptatanappropriate levelbyveryprecisecontrols.Atthesametime,nitrogen isintroducedintotherooms.Insuchanatmosphere,no openflamescandevelop.Theprotectedareasremain accessibletopeople.

36

Reliable Data Centres

6.1.2 Recommendedequipmentfordifferentdowntimes Data centre category Technical fire protection Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW Data centre/ server room 500 to 2500 W/ m2

Permitted data centre downtime*

A B C D E

Monitoringunitwithfiredetection Monitoringunitwithearlyfiredetectionandfire extinguishingtechnology

Monitoringunitwithearlyfiredetection Firealarmsystem,monitoringunit withveryearlyfiredetectionandautonomous fireextinguishingtechnology

72h 24h 1h 10min 0min

Firealarmsystem,monitoringunitwithveryearlyfiredetection andautonomousfireextinguishingtechnology Firealarmsystem,monitoringunit withveryearlyfiredetectionandautonomous fireextinguishingtechnology Firealarmsystem,monitoringunitwithvery earlyfiredetectionandautonomousfireextinguishingtechnology/oxygenreducingsystem

Firealarmsystem,monitoringunitwithveryearlyfiredetectionandautonomousfireextinguishing technology/oxygenreducingsystem,eachinredundantarrangement

Table8fromtheBITKOMmatrixPlanningGuideforaReliableDataCentreTechnicalfireprotection

Data centres
Ifthetolerateddowntimesare24hoursmaximum,detectionandmonitoringunitswithoutdownstreamextinguishingsystemsaresufficient.Ifmaximumdowntimesof lessthanonehourarerequired,adownstreamfireextinguishingsystemwithgasastheextinguishingagent isindispensable.Ifevenshorterornodowntimesdueto firecanbetolerated,firepreventionusingthetechnique ofoxygenreduction,combinedwithactiveveryearlyfire detectionandagasextinguishingsystemwithNovec1230 isapossiblealternative. Thedecisioninfavourofgasextinguishingsystemsorfire preventionsystemscanonlybetakenfollowingthorough analysisbyexperts.Aspectstobetakenintoconsideration whendimensioningandinstallingafireextinguishing systemwithcontrolare: Requirementsforthesystem(s),tolerabledowntimes, etc. Typeofdetectors(fixedpointdetectors,aspirating detectors)

Typeoffireextinguishingsystemandextinguishing agent Possibleplanningofafirepreventionsystem

Server cabinets
Amonitoringunitwithfiredetection(veryearlyfire detection)detectsextremelytinyquantitiesofsmoke aerosolsthatarereleasedintheearliestphaseofemergenceoffire.Thisgainsvaluabletime,whichcanbe utilisedfororganisationalmeasures(automaticphone callers,paging,etc.)andtheinitiationofcountermeasuressuchasasoftshutdown,externaldatabackup, selectiveswitch-offorpossiblytargetedfireextinctionof particularareasorobjects. Intheeventofafire,unitswitch-offisthesafestalternativetopreventfurtherspreadingofthefireorofcorrosive fumes.Modern,softshutdowninnowayimmediately shutsoffthepowersupply,however.Ifsoftshutdown occurs,theveryearlyfiredetectionsystemactivatesan intelligentservermanager,whichquicklydivertsvaluable

37

datatoneighbouringdataorservercabinets.However, thiscanonlybeachievediftheappropriatesoftware/ hardwareenvironmentiscombinedwithahigh-performanceveryearlyfiredetectionsystem.Thedefinitive cut-offofthepowersupplythentakesplacewhenthe datatransferiscomplete.

planningphaseyoumustclarifywhethertheregulations, guidelinesandfiresafetyobjectivesthemselvescan beputintopractice.Theuseofexperiencedplanners isrecommended,becausestructuralandtechnicalfire safetymeasuresmustbebroughtintobalancewith interruption-freedatacentreoperation.Subsequent installationsandconversionsdevourhugesumsofmoney orleadtoaspectacularriseinpremiumsforinsurance relatingtofireandelectronicsystems.

6.2 Structuralfiresafetymeasures
Theaimofstructuralfiresafetymeasuresistosave humanlives.Thisdemandsmaximumqualityofmaterials andworkmanship,andstrictadherencetoregulations andprocedures.

6.2.2 Methodofoperationandroom requirements


Structuralelementsarecategorisedinfireresistanceclassesaccordingtotheirfirebehaviour.Thefireresistance stagesaremostlysetat30,60,90and120minutes.So,F 30means,forexample,thatduringafiretest,atleast30 minutesmustpassbeforethewallgivesway.Thebuilding authoritiesdenotetheseclassesasF60forfire-retardantandF90forfire-resistant. Walls,floorsandceilingsmustbeconstructedwithat leastfireresistanceclassF90.DoorsmustconformtoT90 qualityasaminimum,i.e.doorsmustberesistanttofire for90minutes.Protectionagainstfumesandspraywater isalsoindispensable. Cableandinstallationductsfromandtothedatacentre mustbeeffectivelyprotected.Flame-retardantcable ductscanhaveprotectionwithE30orevenE90quality.InstallationductsmustbeinI30orI90quality,and independentventilationductsinL90.Ifelectriccablesare routedthroughfire-resistantceilingsandwalls,thecable bushingsmustalsobefireandfumeresistantinother words,sealedoff.Thisisolationmayinsomecasesbe achievedbymeansofintumescentfirepillows. Cablerunsconstituteaveryhighriskintheeventofa

Thebasicsofstructuralfiresafetyaresetoutinnational buildingregulations,andprovisionsgoverningtechnicalfiresafetyequipment,firesafetyplans,firewalls andemergencyescaperoutes.Thefirecharacteristics ofbuildingmaterialsandcomponentsisregulatedby DIN4102,whichdoesnot,however,coverthenecessary objectivesoffiresafetymeasures,suchasarevitalforIT datacentres. Furtheraspectstobeconsideredcarefullyarethefire resistanceratingsofload-bearingstructuresandfire protectioninelectricalinstallationsandpowersupply systems.Whenplanningadatacentre,thefireresistance ratingandescaperoutesmustalsobeconsideredwitha viewtofirefighteraccessandsafety.Firefightingliftsand safetystairwellsmustbeincluded.Moreover,datacentres aregovernedbyfiresafetyregulationsaimedspecifically atbusinesses,suchastheregulationsbytheVdS. Firefighting,extinguishingagentsandsmokeextraction mustalsobeincorporatedinplanning.Thiswouldinclude mobilefireextinguishers,thepossibilityofextinguishing agentretentionfeatures,etc.

6.2.1 Firesafetyobjectives
Whenplanningadatacentre,itisoftheutmostimportancetodefinethefiresafetyobjectives.Duringthe

fire,andshouldbecoatedordesignedtobewaterand moistureresistant.Thiswillmakethemintumescent, capableofreliablypreventingfirefromspreadingalong thecables.Thecablesthemselvesshouldconsistof

38

Reliable Data Centres

flame-retardantmaterial,whichalsodoesnotproduce aggressivefumes. Firespreadsrapidlyanduncontrollablythrough(flammable)pipes,whicharelaidonandinceilingsandwalls.Pipe shieldingprovidesafire-resistantandfume-proofbarrier. However,simpletestingofstructuralelementsisinno wayadequateforcomplex,high-availabilitydatacentres. Whenhighavailabilityisrequired,theroomsormodular

safetycellstobeconstructedmustundergoastandardisedsystemtesttoEN1047-2,asmustthestructural elementsoftheceiling-wallandwall-floorconnections, thecableentries,overpressurereliefandthedoorsand surroundingareas.TheEuropeanstandardforthedata centreinfrastructurespecifiesboththeintensityandthe durationofpreciselydefinedloads.Theuseristhussafein theknowledgethathisentiresystemnotjustonewall orthedoorisfire-resistant.

6.2.3 Recommendedequipmentfordifferentdowntimes Data centre category Structural fire safety Servers cabinet up to 5 kW Servers cabinet from 5 kW to 30 kW Data centre/ server room 500 to 2500 W/m2
Walls,floors,ceiling:Fireresistanceclassmin.F90, protectionagainstfumesandwater,doorsmin. T90,protectionofequalqualityforcablewalls Walls,floors,ceiling:Fireresistanceclassmin.F90, protectionagainstfumesandwater,doorsmin. T90,protectionofequalqualityforcablewalls Systemtestofstructuralfiresafety Walls,floors,ceiling,doors:asperEuropean StandardEN1047-2,protectionagainstfumesand waterfor60min,protectionofequalqualityfor cablewalls Systemtestofstructuralfiresafety Walls,floors,ceiling,doors:asperEuropean StandardEN1047-2,protectionagainstfumesand waterfor60min,protectionofequalqualityfor cablewalls

Permitted data centre downtime*

Walls,floors,ceiling:Fireresistanceclassmin. F90,protectionagainstfumesandspraywater, doorsmin.T90,protectionofequalqualityfor cablewalls Walls,floors,ceiling:Fireresistanceclassmin. F90,protectionagainstfumesandspraywater, doorsmin.T90,protectionofequalqualityfor cablewalls Systemtestofstructuralfiresafety Walls,floors,ceiling,doors:asperEuropean StandardEN1047-2,protectionagainstfumes andspraywaterfor60min,protectionofequal qualityforcablewalls Systemtestofstructuralfiresafety Walls,floors,ceiling,doors:asperEuropean StandardEN1047-2,protectionagainstfumes andspraywaterfor60min,protectionofequal qualityforcablewalls

72h

24h

1h

10min

Systemtestofstructuralfiresafety Walls,floors,ceiling,doors:asperEuropeanStandardEN1047-2,protectionagainstfumes andspraywaterfor60min,protectionofequalqualityforcablewalls

0min

Table9:FromtheBITKOMmatrixPlanningGuideforaReliableDataCentreStructuralfiresafetymeasures

39

6.2.4Specialfeatures
Aspectstobetakenintoconsiderationduringproject planningare: Determinationoffiresafetyobjectives,under considerationofthespecialrequirementsoftheIT infrastructure Determinationofstructuralcharacteristics Planningtheconstructionbyprofessionalplanners, ifpossible Compilationofspecificationsfortheindividualelementstobeofferedfortender Collectionofincomingquotations,comparison, evaluation Drawingupacontract-awardingproposalforthe decision-makers

40

Reliable Data Centres

7 Design of premises and safety zones for data centres


Securityofinformationtechnologyisasweepingterm thatcoverslogisticaldatasecurity,thephysicalsafetyof systemsandtheorganisationalreliabilityoftheprocesses.Theobjectiveofacomprehensivesecurityconcept istotakeallareasintoconsideration,torecogniseand assessrisksearlyonandtoinitiatemeasuresinsucha waythatacompanyscompetitivenessonthemarketis notjeopardised. ThroughanoverallexaminationoftheITinfrastructure andvariousfunctionalareasoftheIT,awellthought-out plancanreduceoreveneliminatemajorriskstophysical safety.Adecisiveroleisplayedbythepremisesinwhich theITislocatedontheonehand,andthespatialarrangementofthevariousfunctionsinrelationtooneanother ontheotherhand. Whendesigningandplanningadatacentre,thevariousfunctionalareasarearrangedinaccordancewith therequirementsfortheirsafetyandsecurity,andtheir importanceformaintainingthefunctionoftheinformationtechnology. Ifallriskfactorsandtheconstraintsspecifictothe companyaretakenintoconsideration,potentialdanger andtheresultingtimeandexpensecanbeeradicatedin advanceduringthedesignoftheITinfrastructure. Aprotectedlocationtoensureprotectionagainst sabotage Assessmentofpotentialthreatsonthegroundsofthe companyssocialpolicy

Structure of a data centre

Location of IT rooms
Thevariousfunctionalareascanbesetoutasfollows: ThedesignofanITinfrastructureandthustheselection ofasiteforadatacentremustbebasedonthecompanys datasecurityprinciples,whichreflectavailabilityrequirementsandthecompanysstrategicdirection. Thefollowingcriteriashouldbetakenintoconsideration whenexaminingthephysicalsafetyofasite: Lowriskpotentialfromneighbouringusage,adjacent partsofthebuildingorfunctions Avoidanceofrisksfrommediaandpowersupplylines, tremorsandchemicals,whichwouldcompromisethe physicalsafetyoftheITsystems Avoidanceofpossibledangersposedbyelementary risks(water,gales,lightningstrikes,earthquakes),risk assessmentofaspectspeculiartotheregion Thedatacentreasaseparate,independentfunctional zone
Table10:Functionalareasofadatacentre

Safety zones
1 2 3 4 5

Function
Plot Semi-publicarea, neighbouringoffices

Identification (example)
White Green

Operatingareas,rooms Yellow adjoiningIT Technicalsystemsfor operatingIT ITandnetwork infrastructure Blue Red

41

Layout of safety zones


Aschematicrepresentationofthevarioussafetyzones producessomethingliketheexampleillustratedhere:the ITarea(red)issituatedintheinteriorandisprotectedby theadjacentzones3and4(yellow/blue).Safetyzones1 and2(white/green)formtheouterlayers.Theindividual safetyzonesaredividedbysafetylines. Thesafetylinesrepresentthemonitored,safetransition fromonezonetoanother,andaresetupinconformity withthecompanyssafetyrequirements. Thelocationsofthedifferentfunctionalareasandthe Afittingsolutionforavoidingpossiblesabotageisto separatethefunctionalareasbyensuringlimitedaccess tosensitiveareas.Forexample,amaintenancetechnician fortheair-conditioningsystemsorUPShasaccessonlyto thetechnicalzones(blue),nottothecompanysITzone (red). divisionofthesafetyzonesorsafetylinesareimportantforguaranteeingthesafetyoftheITinfrastructure. However,continuousITavailabilitycanonlybeachievedthroughacomprehensivesafetyplan,whichtakes accountofallaspectsofITsecurityandsafety.

42

Reliable Data Centres

8 Wiring
8.1 Currentsituation
Theprimary,originaltaskofdatacentresistorunITapplicationsonmainframesandservers,tomaintaindataand saveitonstoragesystems. FromanITpointofview,theprincipalrequirementis availability,i.e.theabilityofITapplicationsthataregenerallycriticalforcompaniestoremainoperationalpossibly withoutanyinterruption.ThesetypicallyincludeERP systems,productionapplicationsinindustrialcompanies, databases,officeapplicationsandtheiroperatingsystems,andalsoaccesstoprovidernetworks(MAN,WAN) andtotheinternet. TheISO-OSI7LayerReferenceModelappliestotheIT, andthisdefinestheapplicationastheuppermostlayer, andthephysicalinfrastructurerequiredfortransporting thedata,theITwiringanddatatransportdevicesasthe lowest,orfirst,layer,e.g.layer1switches. Requirementsformaximumavailabilityandeverfaster Therefore,theITwiringisoffundamentalimportancefor theavailabilityofITapplicationsinadatacentre:without functioningITwiring,ITdevicessuchasservers,switches andstoragedevicescannotcommunicatewitheachother orexchange,process,holdorsavedata. Itisfrequentlythecase,however,thatITwiringhassimply grownovertime,andonlysatisfieswithdifficultycurrent requirementssuchas highductdensities hightransmissionspeeds interruption-freehardwaremodifications servicesupport ventilationissues Whetherfibre-opticorcoppercablesareused,theIT ThestructuringofITwiringandmeticulous,anticipatory planningarethereforekeytasksforadatacentreoperator.LegalprinciplessuchasBaselIIorSOXalsodemand rigorousandcomprehensivetransparency. Preassembledsystemshavethebestpossible,reproduciblequality,andthereforepromiseverygoodtransmission wiringcomponentsmaytaketheformoffactory-assembled,turnkeysystemsforplug&playinstallations. datatransmissionratesmeanthatthequalityrequirementsfacingITwiringcomponentsfordatacentresare manytimesmoreexactingthanthoseforproductsused inLANs.Whenitcomestoselectingsystems,quality principlesshouldalreadybeincorporatedintheveryearly planningphase,withaviewtosatisfyingperformance requirementssuchas Cabledesignforcopperandfibre-optics Bandwidthsforcoppersystemsandfibre-opticcables Insertionandreturnlossbudgetsforfibre-optics EMCimmunityforcoppersystems Updatecapabilitytohigherspeedclasses 19cabinetdesign Theplanning,installationandacceptancetestingofIT wiringindatacentresisdescribedintheDINEN50174 seriesofstandards.Importantcontentsincludethequalityplan,potentialequalisation,safetydistancesbetween copperITwiringandotherelectricalsources,andthe documentationandacceptancetestingoftheentiredata centre.

8.2 Basicstandards
WhatallstandardstoISO/IEC,ENandTIAhaveincommonisthattheydemandorstipulatestructured,application-neutralITwiring.Inaddition,theyexpressclear recommendationsforsettingupredundantITwiring,for ensuringhigh-levelreliabilityofthedatacentre.

8.3 Quality/selectionofcomponents/ systems

43

characteristicsandhighreliability.Onlyscreenedsystems maybeusedincoppersystems,duetothedemanding availabilityrequirements.Globalstandardsrequirecopper wiringwithatleastclassEA.

ITwiringstructurethatisclearlyarranged,transparent andisolatedfromthesurroundingdevicefleetishighly recommended. Itshouldconnecttherespectivedevicelocationstoa

Sufficientprioritymustalsobelenttotheselectionof suppliersofITwiring.Besidesthequalityofthewiring components,thechiefrequirementsforareliablesupplier arespecialistdatacentreexpertise,experiencewithdata centreITwiringandlastingabilitytosupply.Ideally,the suppliershouldalsooffercomprehensiveplanning,installationandmaintenanceservices.

uniform,consistentITwiringstructure. StandardsDINEN50173-5andISO/IEC24764dividethis equipmentwiringintothesegmentsareaanddevice connectionwiring,withthedeviceconnectioninterfacein between.Theactivedevicesareconnectedtothedeviceneutralareawiringviathedeviceconnectioninterface bymeansofdevice-specificconnectingcablesthatareas shortaspossible.Consequently,whenadeviceisreplaced whichoftenrequiresreplacingtheconnectionfaceon thedeviceonlythecablespecifictothisconnectionhas tobereplacedwithnoneedtomodifyorstripthearea wiring. Areaswithahighpackingdensityareworthyofparticular attentioninthiscontext.

8.4 Structure
Datacentresarethecompanyscentralnervoussystem. Theyarethereforesubjecttocontinuouschange,driven bytheshortlifecyclesoftheactivecomponents.Inorder toavoidfundamentalorfar-reachingchangestotheIT wiringwitheverynewpieceofequipment,aphysical

Office

Interface to external netwrok

Telecommunication room

Main distributor

Area distributor Local distributor Device connection

Area distributor

Area distributor

Device connection

Device connection Data Center...

backbone wiring

horizontal wiring

Figure9:SchematicdiagramtoEN50173-5/TIA942,wiringlayout

44

Reliable Data Centres

Theabovetechniqueensuresthatanyrewiringinvolved duringequipmentreplacementsarereducedtoaminimumbothfinanciallyandintermsoftimewhilethe definedstructureisfullypreserved. Wherenecessary,thehorizontalwiringmustbemadeof copperandfibre-opticcables,inorderthatvariousdevices canbeconnected.Thebackbonewiringshouldbeinfibreopticandcoppercables,andofferredundancy.

Suitableplug-insystemsmustbeselectedforthedevice connectioninterfaces,inaccordancewiththerequired packingdensityoftheconnecteddevices.Suitableplug-in systemsarenamedinstandardsDINEN50173-5andISO/ IEC24764.

8.5 Redundancyandsecurity
Therequirementforhighavailabilitynecessitatesredundancyofconnectionsandcomponents:Thus,itmustbe possibletoreplacehardwareduringongoingoperation, andanalternativecomponentmusttakeontherunning oftheapplicationwithoutinterruptionintheeventof cablefailure. Itisthereforeobviousthatanappropriate,all-inclusive ITwiringplatformmustbeprovidedthatensuresthe correctbendingradii,safeguardsperformanceandcanbe fittedquicklyandreliablyduringoperation.

Figure10:Representationofhorizontalwiring(Cuandfibre-optic)with areadistributorandserver/storagecabinetswithdeviceconnection

Theavailabilityofapplicationscanbeincreasedthrough theuseofpreassembledITwiringsystems.Thisreduces thetimethatinstallationpersonnelhavetospendinthe safetyzoneofthedatacentretoaminimum,bothduring initialinstallationandeventualhardwaremodifications, andalsopromisesadditionaloperationalreliability. Inaddition,caremustbetakentoensurethatallproductsareinspectedanddocumentedaspartofquality management. Forthemutualconnectionofdatacentres,e.g.redundant datacentres,backupdatacentres,orsimplyforbacking upandsavingdataatadifferentsite,theincorporation andsecurityofMANandWANprovidernetworks(data transportservicesorso-calleddarkfibres)ordedicated fibre-opticcablerunsisofenormousimportancewhere

Figure11:Representationofbackbonewiring(fibre-optic)withmain distributorandconnectiontohorizontalwiring(Cuandfibre-optic)with areadistributorandserver/storagecabinetswithdeviceconnection

reliabilityandavailabilityareconcernedand,liketheIT wiringinsidethedatacentre,mustberedundant.

45

8.6 Installation
Engineersandtechniciansmustbetrainedinthespecificationsofthesystemsinordertoensurethesafe,reliable operationoffibre-opticITwiringinthedatacentre, particularlyinthematterofinstallationandworkwith patches.Duetotherequiredwiring,whenselectingthe 19serverorITwiringcabinetandwithreferencetoitem 4.1Secureservercabinets,theuseofcabinetsystems atleast800mmwideisrecommended.Theseenablethe installationofanintegralcablemanagementsystemin theverticalandhorizontaldirection.Asarule,thedepth ofthecabinetisdeterminedbythepassiveandactive componentstobeinstalledinside.Cabinetsystemswitha minimumdepthof800mmhavealsoproventheirworth forpassivedistributors.Adepthof1000to1200mmis recommendedforcabinetsystemsdestinedtoaccommodateactivecomponents. Thepossibleadvantageoffactory-preassembledITwiring systemsalreadymentionedinthesectiononsafety principlescomestotheforeduringinstallationinthe formoftimesavings.Whenthesesystemsareemployed, itisworthmentioningthatwhendatacentrecapacity isexpandedduetoanincreaseinITequipment,these devicesandhencetheITapplicationsthemselvescanbe wiredtooneanotherandbroughtintooperationatthe greatestpossiblespeedandthesameappliestohardwaremodifications.

8.7 Documentationandlabelling
Painstaking,continuallyupdateddocumentationisan importantmeansofensuringthesimplemanagement ofITwiringandreliableplanningofconversionsand upgrades.Alargerangeofpossibilitiesexistinthis respect,fromindividualExcelworksheetstoproven, software-baseddocumentationtools.Whatisimportant isthatthedocumentationisalwayskeptup-to-date,and reflectstheactual,installedITwiring.Thechoiceoftoolis attheusersdiscretion. Closelyconnectedtodocumentationistheclearand eveninlimitedlighteasilylegiblelabellingofcables. Here,too,numerousidentificationmethodsareavailable, e.g.fromcabletagswithreplaceablelabelstobarcode labels.Thetypeselectedwilldependonindividual requirements.Itisimportanttoensurethatidentical nomenclatureisusedthroughoutthecompany.Central managementofthedataisrecommendedtoensureclear, unambiguouslabellingofcables.

46

Reliable Data Centres

9 Certification of a reliable data centre


9.1 Themanagementsystem 9.1.1 Thedesignofamanagementsystem
Forallcompanies,customerorientation,costefficiency andcompetitivenessareintheforeground.Decisivefactorsareresourceplanning,faultpreventionandaprocessorientedapproachtotheperformanceofservices. InternationalstandardssuchasISO/IEC27001:2005,etc. provideconcreteassistanceandastructureforimplementingamanagementsystem,andcanbecertifiedby neutralassociations. Amanagementsystemformsthebasisfor processorientation continuousimprovementofbusinessprocesses(CIP) greatercustomerandsupplierorientation interpretationofinteractionsbetweenbusiness processes faultprevention(preventiveandcorrectivemeasures), and compliancewithlegalrequirements Examplesofsynergiesarethemergingofthefollowing elements Corporatepolicyandgoals Managementreview Existingdocumentation(assignmentofnames,revision,approval,etc.) Internalaudits Evaluationoferrorfrequencyandsources Evaluationofcustomerandemployeesatisfaction Evaluationofsuppliers Procurementprocess Asuccessfulmanagementsystembringsthecompany decisiveadvantages: Businessprocessesbecometraceable,productsand servicesareimproved Weakpointsinthesystemareflaggedup Risksarerecognised,analysed,minimisedoraverted Thecompanyispreparedforemergencies Continuousmonitoringlowerstheerrorrateand thereforeimprovestheservicesprovided.Thisresults inreducedcostsandhigherquality Oncethequalitymanagementsystemhasbeendocumentedandeffectivelyintroducedinthecompany,itcan becertifiedbyanindependent,neutralcompanythatis authorisedtoperformcertification.Firstofall,thecertificationcompanyinspectsthedocumentationandthen thesystemonsite.Theauditorpossessesthenecessary TheuniformstructureofISOmanagementsystemsbasicallypermitsseveralstandardstobecombined.Ifsucha combinationispossible,itwillbeeasytoincorporatenew standardsandexploitsynergisticeffectsindifferentareas ofthecompany. Employeesaremotivatedbythecleardelineationof responsibilities Thedocumentationofprocessescreatestransparency Customerstrustinthecompanysabilitytodeliver qualityisboosted Thecompanysimageisenhanced Competitivenessisstrengthened Liabilityisminimised

9.1.3 Combinationofdifferentstandards

9.1.2 Advantagesofamanagement system

9.2 Certificationofamanagement system

47

qualificationsandprofessionalexperience.Apositive resultleadstocertification,whichisgenerallyvalidfor3 years. Afterthecertificationaudit,themanagementsystemis checkedforitseffectivenessinannualmonitoringaudits.

Performing a preliminary audit


Theaimofthepreliminaryauditistoascertainwhether thebasicprerequisitesforcertificationofthequality managementsystemaremet.Itdetermineswhetherthe certificationauditcanbecarriedoutontheplanneddate withsuccessasthelikelyoutcome.

9.2.1 Theadvantagesofcertification
Certificationisneutralproofoftheeffectivenessof themanagementsystem,andcanofferthefollowing advantages: Acquisitionofnewcustomers,openingthedoorto newmarkets Strengtheningcompetitiveness Fostersthecustomerstrustinthecompany Improvesthecompanysrankingandcreditworthiness Reducesthetimeandcostofdemonstratingquality Internationalrecognitionandacceptance

Alternatively,thecheckperformedduringthepreliminary auditmayincorporatetheinspectionoftheQMmanual andassociateddocuments.Thepreliminaryauditis basicallyarandomsampletest,anddoesnotclaimtobe complete.

The certification process


Duringthecertificationaudit,theauditorscheckwhether thedocumentedproceduresandroutinessatisfythe requirementsoftherelevantrules,andwhethertheprocessesandagreementsdefinedbythecompanymatch theQMdocumentation.

9.2.1 Atypicalcertificationprocedure
Aninformativediscussionshouldbeheldbeforeacertificationcompanyisselected.

Thecertificationauditconsistsofacomprehensiveassessment.Indetail,itcoversinspectionofthefollowing: QMsystemdocumentation Structuralorganisation Implementationofthedocumentedprocessesand theireffectsinthecompany First-timecertificationtakestheformofa2-stageprocess. Instage1,importantkeyelementsarecheckedandan assessmentascertainstowhatextentthecompanyis preparedforcertification.ThecompanysQMdocumentationisalsocheckedforconformitytostandardsatthis point. Instage2,theimplementationandeffectivenessof themanagementsystemthroughoutthecompanyis checked.Ifdeviationsaredisclosedduringtheassessment ofthequalitymanagementsystem,apostauditmay

The informative discussion


Thesubjectsunderdiscussionbasicallyincludequestions aboutcertificationandauditing,organisationalprocedure (e.g.scopeandschedule)andcosts.

The certification order


Byplacinganorderforcertification,thecontracting companyplacesitselfunderobligationtoprovidethe certificationcompanywiththenecessaryQMdocumentation.Alternatively,theQMdocumentationmayalsobe inspectedonsite.Apreliminaryauditmaybeconducted, ifthecompanysodesires.

48

Reliable Data Centres

needtobeperformed.Thecertificateisthenonlygranted afterallsuchdeviationshavebeeneliminatedandwhen thenegativefindingsfallbelowthemaximumnumber recommendedbytheauditor. Theresultisdocumentedinanauditreport.Thisisthe basisuponwhichthecertificationcommitteeofthe certificationcompanydecideswhetherornottogranta certificate.

anotherthreeyears.Therecertificationauditmustbe carriedoutbeforethecertificatesvalidityexpires.The requirementsoftherecertificationauditarevirtually identicaltothoseofthecertificationaudit.Here,however, theauditmaybeperformedinasinglestage.

9.2.3 Selectingtherightcertification partner


Theselectionoftherightcertificationpartnerisdecisive forthesuccessoftheprocess.Aswithanyservice,aprice marginapplies.Itisthereforeadvisabletoobtainseveral quotations,ortoextendtheexistingsystemwithatrustedcertificationpartnerwhoisalreadyfamiliarwiththe system. Incertaincases,thecertificationcompanysinternational organisationmayconstituteamajorcostfactorif,for example,companysitesinothercountriesaretobeincludedinthecertificationprocess. Auditorswithdifferentcertificationbodiesarequalified toasimilarlyhighlevel,astheapprovalofauditsisregulatedcentrallybyaccreditationbodiesandISOstandards.

The monitoring audit


Annualmonitoringauditstakeplaceduringthethreeyearperiodofvalidityofthecertificate. The1stand2ndmonitoringauditsconstitutearandom samplecheckastowhether: Negativefinding(s)fromthepreviousaudithas/have beenremedied Organisationalchangeshavetakenplaceinthe company TheQMsystemhaschanged Thecertificateandcertificationlogoarebeingused correctly Recentchangestorelevantstandards,lawsandregulationshavebeentakenintoconsideration Themanagementsystemcontinuestosatisfythe requirements,and Thequalitymanagementsystemhasbeenadheredto andcontinuestobeeffective. Iftheresultofthemonitoringauditsissuccessful,a completeinspectionoftheQMsystemwilltakeplace inanewprocessinthreeyearstime,beginningwitha recertificationaudit.

The recertification audit


Arecertificationauditisnecessaryforensuringaseamlessextensionoftheperiodofvalidityofacertificateby

49

10 Annex
Selectionofimportantrulesandregulations DIN6280,Part1-15 Part1 Part2 Part3 Part4 Part5 Generatingsetswithreciprocatinginternalcombustionengines Generaldefinitions Powerratingsandpowerratingplates Limitingvaluesfortheoperatingbehaviouroftheengine,thegeneratorandthegeneratingset Speedgoverningandspeedbehaviourofreciprocatinginternalcombustionengines;definitions Operationalbehaviourrelatingtosynchronousalternators forgeneratingsets Part6 Operationalbehaviourrelatingtoasynchronousalternators forgeneratingsets Part7 Part8 Part9 Part10 Part11 Controllingandgoverningequipmentforgeneratoroperation Operationalbehaviourrelatingtogeneratingsets;definitions Acceptancetest Smallpowergeneratingsets;requirementsandtests Measurementandassessmentofmechanicalvibrationsingeneratingsetswithreciprocatinginternal combustionengines Part12 GeneratingsetsUninterruptiblepowersupplyDynamicUPSsystemswithandwithoutreciprocatinginternalcombustionengine Part13 GeneratingsetsGeneratingsetswithreciprocatinginternalcombustionengineforemergency powersupplyinhospitalsandpublicbuildings Part14 Combinedheatandpowersystem(CHPS)withreciprocatinginternalcombustionengineBasics, requirements,componentsandapplication Part15 Combinedheatandpowersystem(CHPS)withreciprocatinginternalcombustionengineTests

50

Reliable Data Centres

ISO8528

Reciprocatinginternalcombustionenginedrivenalternatingcurrentgeneratingsets.

GermanFederalControlofPollutionAct 4. Regulationontheimplementation oftheFederalControlofPollutionAct 9. Regulationontheimplementation oftheFederalControlofPollutionActTA TA DIN/VDE0107 Supplement1 Supplement2 DIN/VDE0108 Supplement1 Part2 Part3 Part4 Part5 Part6 Part7 Part8 DIN/VDE0100 Part728 AirGermanTechnicalGuidelinesonAirQualityControl NoiseGermanTechnicalGuidelinesonNoisePrevention Electricalinstallationsinhospitalsandlocationsformedicaluseoutsidehospitals Extractsfrombuildingandoccupationalhealthandsafetyregulations Interpretation,explanation Powerinstallationsandsafetypowersupplyincommunalfacilities Buildingregulations Placesofassembly Stores,shopsandexhibitionrooms Multi-storeybuildings Restaurants Closedcarparks Workingpremises Temporarystructures,etc. Standbypowerinstallations Basicprinciplesoftheapprovalprocedure Regulationonplantsandsystemsrequiringapproval.

51

PSC VDEW VDEW EltBauVO VDS WHG

ConnectionconditionsofPSCs GuidelinesforemergencygeneratingsetsbytheGermanelectricityassociation(VDEW) Paralleloperationwiththelow-voltagenetworkbytheGermanelectricityassociation(VDEW) Ordinancegoverningtheconstructionofoperatingroomsforelectricalinstallations RegulationsfromtheGermanAssociationofPropertyInsurers GermanWaterResourcesAct

FuelOilTaxationAct (operationofstationarysystemswithfueloil) DIN31051 Maintenance

52

Reliable Data Centres

11

Glossary
19cabinet Rackwithapproximately40heightunits,overallheightapproximately2metres,installation width483mm,installationheightismeasuredinheightunits(HU),1HU=44.45mm CW Datacentre DX Emission EMC Immission IT Modular Paralleloperation PDU ChilledWater;air-conditioningsystemswithchilledwater Serverroomand/ordatacentre DirecteXpansion;air-conditioningsystemswithrefrigerant Influencesemittedbyadevicethataffecttheenvironment ElectroMagneticCompatibility Influencesoriginatingfromtheenvironmentthataffectaparticularlocation InformationTechnology(formerlyEDPElectronicDataProcessing) Describesasystemthatisconstructedofseveralmodules(assemblies) Twoormoreinstallationsjointlysupplyconnectedconsumers PowerDistributionUnitorlow-voltagedistributionsystem

Precisionair-conditioning Air-conditioningsystemthatisabletokeepboththetemperatureandtheairhumidityconssystem tant.TheparametersoftheairattheinletopeningsoftheITunitsshouldbebetween22and 27Cand40and60%r.h. PSC Redundant Scalable SG UPS PowerSupplyCompany Duplicatearrangementtoincreaseavailability(errortolerance) Step-by-stepadaptabilitytorequirements Standbygenerator(mostlyanemergencydieselgenerator) UninterruptiblePowerSupply

53

12 Acknowledgements
ThisReliableDataCentreguidewascompiledinconsultationwiththeBITKOMDataCentre&ITInfrastructure workinggroup. Wewishtoexpressoursincerethankstoallmembers oftheworkinggroupforthevaluablediscussionswe hadwiththem,andourparticularthanksgotothe followingfortheirinvolvement: SilviaBader(DEKRAcertificationGmbH) HaraldBecker(Rosenberger-OSIGmbH&Co.OHG) Dr.GeraldBerg(Rosenberger-OSIGmbH&Co.OHG) KlausClasen(NotstromtechnikClasenGmbH) PeterClauss(WagnerGroupGmbH) AykutGven(DEKRAcertificationGmbH) FrankHauser(ServerTechnologyInternational) DieterHenze(RittalGmbH&Co.KG) Dr.SiegbertHopf(MasterguardGmbH) KnutKrabbes(QMKIT-Security+Quality) StephanLang(WeissKlimatechnikGmbH) HelmutMuhm(Dipl.-Ing.W.BenderGmbH&Co.KG) Hans-JrgenNiethammer(TycoElectronicsAMP GmbH) TorstenPed(NotstromtechnikClasenGmbH) AchimPfleiderer(StulzGmbH) ThorstenPunke(TycoElectronicsAMPGmbH) ZeynepSakalli(euromicronsolutionsGmbH) MichaelSchumacher(APCDeutschlandGmbH) KarlheinzVolkert(OrangeBusinessGermanyGmbH) PeterWsch,SCHFERAusstattungs-SystemeGmbH RalphWlpert(RittalGmbH&Co.KG) ThefollowingpeopleassistedwithVersion1fromNovember2006: DieterHenze(RittalGmbH&Co.KG) SiegbertHopf(MasterguardGmbH) PeterKoch(KnrrAG),HelmutGhl(O2GmbH) KnutKrabbes(QMKIT-Security+Quality) MatthiasLohmann(TVSecure) IngoLojewski(EmersonNetworkPowerGmbH) AchimPfleiderer(StulzGmbH) JrgRichter(I.T.E.N.O.SGmbH) HarrySchnabel(HarrySchnabelConsult) MichaelSchumacher(APCDeutschlandGmbH) JrgenStrate(IBMDeutschlandGmbH) JudithWagener(BullGmbH) RalphWlpert(LampertzGmbH&Co.KG) EckhardWolf(AEGPowerSolutionsGmbH) SandraSchulz(Giesecke&DevrientGmbH)

54

TheFederalAssociationforInformationTechnology,TelecommunicationsandNewMedia(BITKOM)represents morethan1,300companiesinGermany.Its950directmembersgenerateasalesvolumeof135billioneuros annuallyandemploy700,000people.Theyincludeprovidersofsoftware,ITandtelecommunicationservices, manufacturersofhardwareandconsumerelectronicsaswellasdigitalmediabusinesses.BITKOMisworking, inparticular,toimprovetheregulatoryframeworkinGermany,formodernizationoftheeducationsystemand foraneconomicpolicywhichencouragesinnovation.

Thistranslationwaskindlysponsoredbythefollowingcompanies:

BundesverbandInformationswirtschaft, TelekommunikationundneueMediene.V. Albrechtstrae10A 10117Berlin-Mitte Tel.:03o.27576-0 Fax:030.27576-400 bitkom@bitkom.org www.bitkom.org