IntroductiontoDataCentres

Grant Sauls GrantSauls

CCDA CDNIDSCDCSNDSCDCDDCCAJNCIAERJNCISEProject+DCECCTT(FIA)CDS
CertifiedDataCenterDesignSpecialist
AGENDA AGENDA
9 5 9am 5pm
WhatandwhywehaveDataCentres
GeneralissuesfacingDataCentres
10CriticalcomponentsofaDataCentre
Datacentredesignguidelines
What is a Data Centre? WhatisaDataCentre?
ABuildingorportionofabuildingwhose
i f i i h primaryfunctionistohouseacomputerroom
anditssupportareas
.....Simplicity,Flexibility,scalability,
modularity
Th f f ti l i t f d t Therearefourfunctionalrequirementsofadata
center
1 Location i e A place to locate computer storage 1. Locationi.e.Aplacetolocatecomputer,storage
andnetworkingdevices.
2 Power i e Power needed to maintain the 2. Poweri.e.Powerneededtomaintainthe
devices
3 HVAC i e Temperature controlled environment 3. HVACi.e.Temperaturecontrolledenvironment
withintheparametersneed.
4 Structured cabling i e Connectivity provided to 4. Structuredcablingi.e.Connectivityprovidedto
otherdevicesbothinsideandout.
Data Centre Global Market DataCentreGlobalMarket
$8billiondollarmarket2007
$15.6 billion expected in 2008 $15.6billionexpectedin2008
50%spendingofI.TandFacilitiesManagers
b d ill hi budgetswillgotothisarea
DataCentreexpectgrowthfor2009/2010will p g /
be40%
Vi t li ti ? Virtualisation?
Yes/No
Market drivers for Data Centres MarketdriversforDataCentres
DisasterBackup
Data Protection SarbanesOxley DataProtection Sarbanes Oxley
GrowingFinancialsectorsi.e.Internetbanking
Distancelearningforuniversitystudents
Corporates requiring security storage Corporatesrequiringsecurity,storage,
resilience
Growthforonlinebusiness
Issues facing Data Centres IssuesfacingDataCentres
ReliabilityandAvailability
Power Power
RackSpace
SiteLocation
Heat Heat
Cooling
Reliability Reliability
Temperature
55%
Electronics
Failuredue
to
Vibration
20%
Dust 6%
to...
High/Low
Humidity Humidity
19%
POWER POWER
DX Cooling DXCooling
Systems
50%
Critical
L d
About9%ofa
power stations
Power
Consumption
Loads
36%
powerstations
powermakesit
toaServer
p
Issues...
UPS/
Battery Lightning y
Charging
11%
g g
3%
Rack Space RackSpace
S i 3 2 b i ili i Serversrequire3+2basicutilities
Power
Data
Control/KVM
Cooling
Racking
18%ofRackspacegoesunusedduetopower
consumptionandcooling.
Site Location SiteLocation
S i i bl SeismicProblems
GroundSubsidenceproblems
Security/criminalproblems
Availability of Electricity, water, sewage and AvailabilityofElectricity,water,sewageand
telecoms.
Proximity to railway lines airports chemical Proximitytorailwaylines,airports,chemical
storageandmilitary
EMC Problems i e Mobile masts radar EMCProblemsi.e.Mobilemasts,radar
transmitters
HEAT HEAT
H t t f 55% f l t i f il Heataccountsfor55%ofelectronicfailure.
Theremustbe1kwofaircon for1kwofheatinsideyourrack.
Therearethreemethodsofheat
Conduction transferringheatthroughmetal
Convection transferringheatthroughliquidsorgas
Radiation transferring heat through electromagnetic waves Radiation transferringheatthroughelectromagneticwaves
Itsimportantnottooverfillyourrack.
For every eight server racks you should have one comms and one Foreveryeightserverracks,youshouldhaveonecomms andone
storagerack.
RememberheatgenerationoccursattheChip,Server,Rackand
R l l Roomlevel
A4kwrackisideali.e.Averageserveris370W
10criticalcomponentsofa
DataCentres
k d l RackFundamentals
CoolingFundamentals
PowerDistributionFundamentals
Generator Fundamentals GeneratorFundamentals
AvailabilityandReliabilityFundamentals
Ph i l I f t t M t PhysicalInfrastructureManagement
Fundamentals
i i h d FireProtectionMethods
GeneralDesign
Rack Location Unit Concept RackLocationUnitConcept
Anonapplicationspecificworkingspace
An area of the computer room floor i.e. 600 x Anareaofthecomputerroomfloori.e.600x
1000mm
Ad d i di i i h Adequatepowerandairconditioningtothat
locationisessential
RackLocation
withinDataCenters
TIA942andASHRAE
recommendlayingoutequipmentracks
accordingtotheHOTAISLE COLDAISLE, g ,
sevenpitchtilemodel
Rack Location cond RackLocationcon d
Thi M d l th t i ThisModelmeansthatair
conditioningunitspump
coldairintothefloor
void.
Ventsinthefloordeliver
cold air to the front of the coldairtothefrontofthe
racksfacingeachother
ColdAisle
HotAirescapesthrough
thebackoftherackinto
the Hot Aisle where it theHotAislewhereit
returnstheairtotheair
conditioningunit
Rack Location Cond RackLocationCon d
ThereareninedifferentAirFlowmodels
Three different types of rack location designs Threedifferenttypesofracklocationdesigns
7tilepitch
8 il i h 8tilepitch
9tilepitch
The 7 tile pitch is the biggest space saver The7tilepitchisthebiggestspacesaver
whenusinga4kwrack
Rack Location Safety RackLocationSafety
EnsuretheFloorisstrongenough
Ensure the floor is level and in good order Ensurethefloorislevelandingoodorder
Putheavyequipmentatthebottomofthe
k rack.
Correctlyearthallthemetalwork y
EIA310Disthestandardforracks,cabinets,
Panelsandassociatedequipment.
Rack Location Clearance RackLocationClearance
Aminimumof1mfrontclearancemustbe
providedforinstallationofequipment. p q p
1.2mfrontclearanceisrecommended.
A i i f 0 6 l b Aminimumof0.6mrearclearancemustbe
providedforserviceaccess.
A1mRearclearanceisrecommended.
RackLocationDesign
IssuestoAddress
SizeofRack
Cableentry,toporbottomincludingcablesealing y, p g g
CoolingMethod
P S li (C t li d R k M t UPS) PowerSupplies(CentralisedorRackMountUPS)
Earth/Grounding
CableManagement
Rack Monitoring RackMonitoring
FireSuppression
Data Center Cooling DataCenterCooling
Th li t f D t C t i t f CRAC ThecoolingsystemforaDataCenterconsistofCRAC
(computerroomairconditioning)andforlargerData
CentresCRAH(computerroomairhandlingunit) ( p g )
DataCenterhaveHOTSPOTSthattraditionalcooling
methodshavenotbeenabletoaddress.
Thereare3typesofCoolingArchitectures Room,
RackandRow.
The primary distinctions that affect the capability of Theprimarydistinctionsthataffectthecapabilityof
coolingsystemsarerootedinthedistributionsystems.
It is the configuration of these distributions systems Itistheconfigurationofthesedistributionssystems
thatdistinguishesthedifferenttypesofcooling
systems.
Data Center Cooling Cond DataCenterCoolingCon d
Thereare3basicdistributionsystems
Flooded notrecommended1 3kW
Th CRAC d h l d d i b lk i f h i h TheCRACandtheloadsdrawinbulkairfromtheroom,without
anyspecialductinginbetweenthem.
LocallyDucted 4 8kW y
Airisprovidedorreturnedviaductswhenhaveventslocated
neartotheloads
Fully Ducted 5 15kW FullyDucted 5 15kW
Supplyorreturnairisdirectlyductedintooroutoftheloads
Basedonthisinformationthereare9combinationsof
Coolingavailable.
Data Center Cooling Cond DataCenterCoolingCon d
Design Cooling Considerations DesignCoolingConsiderations
Layoutracksinalternatingrows
LocationofCRACUnits Crucial
Quantityandlocationofvents Crucial
SizingofDuctwork
Proper internal configuration of racks Properinternalconfigurationofracks
Ensureplumbingroutesareavailable
bl h d l l h Establishandplanpowersuppliestotheunits
Decideonlevelsofredundancy
Data Center Cooling DataCenterCooling
T pical Challenges for Rack Room and Ro oriented Cooling TypicalChallengesforRack,RoomandRoworientedCooling
Architectures
Agility abilityofsystemtoadapttochange
P D it PowerDensity
ExtensiveEngineeringinstallations
Adapt businessrequirements
System Availability SystemAvailability
Eliminatehotspots
redundancy
Eliminate vertical temperature Eliminateverticaltemperature
Liquidleaks
HumanError
LifecycleCost Capitalinvestmentvs.Operationalefficiency
Serviceability Meantimetorecover
Manageability predictivefailureanalysis g y p y
Data Center Power Distribution DataCenterPowerDistribution
P i k t i t i il bilit i d t Poweriskeytomaintainavailabilityinadata
center
US claim to lose $164 Billion a year due to power USclaimtolose$164Billionayearduetopower
disruptions
Each Data Center must have a separate and EachDataCentermusthaveaseparateand
dedicatedpowersourceandpowerinfrastructure
Power from the utility to the DC is either stepped PowerfromtheutilitytotheDCiseitherstepped
downorsteppedupatleast6timesbeforeit
reachestheDCorUser.
Nominalvs.NormalVoltage
Data Center Rack Power Distribution DataCenterRackPowerDistribution
R k P S t t b d t bl t th RackPowerSystemsmustbeadaptabletothe
following
Equipment Equipment
OnDemand
SafetyHazards
Adverselyeffectthesystem
P tt h th t k i t Patternsshowthatrackpowerrequirementsareonan
increasefromyeartoyear.i.e.2004 2kWrack
2008 patterns have shown that the average rack power 2008patternshaveshownthattheaveragerackpower
requirementisat6kWperrack.
Advancementarebeingmadetowardsthe15kWrack g
atarapidrate.
DataCenterRackPowerDistribution
HereisabreakdownofEquipmentvs.kiloWatts q p
VeryLowloads
PatchPanels
Switches
Loadsin1kWrange16Amp
Entry Level 1u severs EntryLevel1usevers
Loadsin23kWrange1620Ampcircuitsonly
Mediumlevel1uservers
Switches
Majorityofrackopen
Loadsin5kWrange 20 30Ampsdependingonwhatsconnected
Halffilledwith1uhighendServers
Routers
Mix of technologies Mixoftechnologies
7kw+Range 30AmpCircuitsOnly
BladeServers HPC3000
StorageDevices
SwitchingGear
ExtremelyRare
Data Center Power Distribution DataCenterPowerDistribution
VoltagecomingintotheDCmustbecheckto
verifyifitistoohighortoolowasthiscan y g
impacttheequipment.
Therefore Delta transformers must be ThereforeDeltatransformersmustbe
installed.
Steppingupordown3phasepowermustuse
a delta transformer. adeltatransformer.
Data Center Power Distribution DataCenterPowerDistribution
TherearetwotypesofTransformers
Delta Transformers use 3 phase power with DeltaTransformers use3phasepowerwith
Ground
W T f 3 h i h WyeTransformers use3phasepowerwitha
Neutral
Themostcommontransformerconfiguration
is a Delta to Wye isaDeltatoWye.
Data Center Power Requirements DataCenterPowerRequirements
l l i i l l d Calculatecriticalload
NamePlaterating worsecaserating
Voltagerequirement
SingleorThreePhase g
UPSRequirements
multiplying the incoming current in amps by the multiplyingtheincomingcurrentinampsbythe
voltageofthedevicewithgiveyourVArating,
then multiply the anticipated VA number by 0.67 thenmultiplytheanticipatedVAnumberby0.67
toestimatetheactualpower,inWattsandthen
divideby1000todetermineforkWrating y g
Generator Stats GeneratorStats
AccordingtoContingencyPlanningResearch
powerrelatedeventssuchas;
bl k t d t f 31% f blackoutsandsurgesaccountfor31%of
computerdowntimeepisodeslastingmorethan12hours,
power failure and surges account for 45 3% of data loss and powerfailureandsurgesaccountfor45.3%ofdataloss,and
accordingtoIDCpowerdisturbancesaccountforabout33%ofall
f l serverfailures.
Thereforestandbygeneratorsareaessential
componentinaDataCenter
Data Center Standby Generators DataCenterStandbyGenerators
d f h f ll i Generatorsaremadeupofthefollowing:
PrimeMover 4stroke
Alternator
Governor
MaintainRPM/ACoutputQuality
IsochronousDesign
SuperiorElectronicDesign
DistributionSystems
Automatictransferswitch,
SwitchingGear
PhysicalInfrastructureManagement
Basics
C I CoreIssues
MaintainingSystemAvailability;and
Managing problems and changes Managingproblemsandchanges
E ti l C t i f I f t t EssentialCategoriesforInfrastructure
Managementare
Incident management Incidentmanagement
Changemanagement
Availability management Availabilitymanagement
Capacitymanagement
PhysicalInfrastructureManagementBasics
Availability Management AvailabilityManagement
Systemicallyidentifying
availabilityrequirements
andReliability
requirementsagainst
actualperformance
Physical Change
I id t t
Infrastructure
Management
Basics
Management
Focusesonthemethods
andproceduresfor
makingchangesintheDC
Infrastructure
Incidentmanagement
ReturntonormalASAP
withlittleimpacton
Business
CapacityManagement
Providing IT Resources at ProvidingITResourcesat
therighttime;rightcost
andmustbealignedwith
currentandfuture
requirements.
Physicalinfrastructuremanagement
Basics
Ph i l i f t t t i ll k d i Physicalinfrastructuremanagementrequiresallkeydevices
anddatapointstobemonitored.Theseincludeallthe
devicesinthephysicalinfrastructurelayerandthe
surroundingenvironment.
Best practices dictate that the following list of devices be Bestpracticesdictatethatthefollowinglistofdevicesbe
monitoredattheracklevel:
Aminimumoftwotemperaturedatapoints p p
Individualbranchcircuits
Transferswitches
Coolingdevices,and
UPSsystems
DataCenter
FireProtection,preventionandsuppression
43%ofbusinessthatcloseduetofire,never
reopen p
29%whodoreopenfailwithin3years
Th f i i i l d Thereforeitisessentialasadatacenterto
prepareforunseendisasters.
ThebestFireprotectionmethodisFire
prevention prevention
DataCenter
FireProtection,preventionandsuppression
NFPA 75 i th t d d f t ti f t d t NFPA75isthestandardforprotectionofcomputerordata
processingequipment.OnenotableadditiontoNFPA75
thattookplacein1999,allowsdatacenters tocontinueto
f powerelectronicequipmentuponactivationofaGaseous
AgentTotalFloodingSystem,whichwewilldiscusslaterin
detail.Thisexceptionwasmadefordatacenters thatmeet p
thefollowingriskconsiderations:
Economiclossthatcouldresultfrom:
L f f ti l f d Lossoffunctionorlossofrecords
Lossofequipmentvalue
Loss of life Lossoflife
andtheriskoffirethreattotheinstallation,tooccupants
orexposedpropertywithinthatinstallation
DataCenter
FirePrevention
Itisimportanttomaintainthefollowing
guidelinesinDataCenterforFirePrevention g
DonotbuildaDCclosetoourbuilding
All furniture must be constructed in Metals except AllfurnituremustbeconstructedinMetalsexcept
forchairs
D t ll ki i d Donotallowanysmokinginoraroundyour
communicationandITfacilitieswithintheDC.
DCsshouldbevoidofanytrashreceptacles
Noacousticalmaterialsuchasfoamorfabric
DataCenter
FireProtection
Thereare3systemobjectivestoFire
Protection
FireDetection
Detecting the presence of fire Detectingthepresenceoffire
Communication
C i t th i t f fi Communicatetheexistenceofafire
Suppression
Containandextinguishthefire
DataCenter
FireProtection
Thefourstagesofcombustionare:
1.Theincipientstageorprecombustionstage, p g p g ,
2.Thevisiblesmokestage,
3 The flaming fire stage and lastly 3.Theflamingfirestage,andlastly
4.Theintenseheatstage.
DataCenter
Detectors
Th t t f S k d t t TherearetwotypesofSmokedetectors
Airsamplingsmokedetectors
VESDA
Highpoweredphotoelectric
200 7000sqm
Usesalaserbeam
Intelligentspottypedetectors
More sensitive Moresensitive
Theyusealasertoscanparticles
Individualaddressable
EPO EmergencyPowerOff
DataCenter
FireSuppression
f i f 2typesofSuppressionagentsforDCs
FireExtinguishers
Totalfloodingfireextinguishingsystems
2commonlyusedagentsinDCs
Inert gas Inergen InertgasInergen
Fluorinebasedcompounds
FE 200 FE200
FE227ea
DataCenter
AvailabilityandReliabilityFundamentals
93% f i h h l il bili f 10d 93%ofcompaniesthathavelostavailabilityfor10days
ormorehavefiledforbankruptcyin1year
A il bilit d li bilit b ilt NCPI AvailabilityandreliabilityarebuiltonNCPI
NCPIarebuilton3corebusinessobjectives
I i Increaseinturnover
Reductionincost
Better utilization of assets Betterutilizationofassets
Availabilityisdeterminedbyasystemsreliabilityand
its recovery time itsrecoverytime.
Five9sofAvailability
DataCenter
AvailabilityandReliabilityFundamentals
F t ff ti A il bilit d R li bilit FactorsaffectingAvailabilityandReliability
ACPowerConditions
Sag / under voltage Sag/undervoltage
Swell/overvoltage
transients
Cooling Cooling
HotSpots
EquipmentFailure
TemperatureSwings
NaturalDisasters
Blackouts Blackouts
HumanError
PoortrainingandDocumentationmanagement
Data Center DataCenter
GeneralDesignGuidelines
Data Centre Standards DataCentreStandards
TIA942TelecommunicationsInfrastructure
standardforDataCentres,April2005 p
EN501735InformationTechnology data
centres Europe centresEurope
ANSIBICSI002DataCenterStandard
complementarytoTIA942
ISO/IEC NP 24764 Information Technology ISO/IECNP24764 InformationTechnology
GenericcablingforDataCentres
Others Standards within Data Centres OthersStandardswithinDataCentres
TR / ER TR/ER
l i i i l d ATelecommunicationsRoomisanenclosed
architecturalspaceforhousing
t l i ti i t bl telecommunicationsequipmentcable
terminationsandcrossconnectcabling.
AnEquipmentRoomisanenvironmentally
controlledcentralizedspacefortelecom
i equipment
DifferencebetweenTRandERsarethatERs
serveabuildingorcampuswhileTRsserveafloor
inbuilding.
TR / ER Sizing TR/ERSizing
f 60 l b 3 2 TRsof460sqmorlessmustbe3mx2.4m
TRsof929sqmandnotlessthan740sqmmust
be3mx3.4m
ERminimumsizemustbe3x5m
Generally allow 9 sq m per work area Generallyallow9sqmperworkarea
Multiplynoofworkareasby0.07sqm
000 5000sqm
5000/9=555x0.07=39sqmofERspace.
Ideal Computer Room Size IdealComputerRoomSize
Theidealroomsizewouldbenotlargerthan
600sqm q
Theyrequirelargeamountsofairconditioning
CRAC / CRAH CRAC/CRAH(ComputerRoomAirConditioning)
Calculationsbecomedifficultwhentheroom
isbiggerthan600sqm
Gas Fire Suppression quantities will be huge GasFireSuppressionquantitieswillbehuge.
Requirements for a Data Center RequirementsforaDataCenter
Space Required Function SpaceRequired Function
ComputerRoom/ServerRoom HousecomputerRacksandCommunicationsEquipment
ControlRoom allcontrolandmonitoringfunctionsareconcenrated
GeneralOfficeArea OfficeareawhereITstaffcanwork
EntranceFacility wherealltheexternalcommunicationscablingenterthebuilding
FireGasSuppressionStore StoragespacedependsonwhichgasisusedInertgase orHalocarbons
ElectricalSwitchRoom Externalpowercablesenterthebuildingandformsademarcationpoint
UPSandBatteryRoom Loadsof100kVAitisrecommendedtohaveaseperateUPSroom
GeneratorRoom houseastandbyDieselGenerator
Requirement for Data center cond RequirementforDatacentercon d
S i i SpaceRequirement Function
OilStore House Dieselfuelforgenerator
Storage and Build Area To store and unpack equipment to build StorageandBuildArea Tostoreandunpackequipment tobuild
DeliveryandLoadingArea Adjacent areatoallowheavyequipment
tobeshippedintobuilding
PlanningandMeetingRoom Toholdmeetings
Internalstafffacilities Male/female/disabled toiletsandshowers
Electrical Substation Due to power needs a separate substation ElectricalSubstation Duetopowerneeds aseparatesubstation
mayberequired
AirConditioningCondensers Asecureareaisrequireddependingon
th i i t theaircon requirements
Externalstafffacilities Parkingspace,bicycle andsmokeshelter
Facilities Requirements FacilitiesRequirements
i i d i h RoomDimensionsandHeight
FloorStrength g
ConnectionofServices
External Services ExternalServices
Access,LoadandDDA
Decor
Lighting Lighting
FireRegulations
Data Center Room Height DataCenterRoomHeight
MinimumCeilingheightmustbe2.6mfrom
thefinishedfloortoanyobstructionsuchas y
sprinklers
SlabtoSlabmustbe2.9mi.e.
400mmunderFloor
2 1m racks 2.1mracks
400mmairreturnpathforCRAC
Data Center Floor Strength DataCenterFloorStrength
i i di ib d fl l di i Minimumdistributedfloorloadingcapacity
mustbe7.2kPA
Therecommendedloadingcapacityis12kPA
The floor must have minimum of 1 2kPA Thefloormusthaveminimumof1.2kPA
hangingcapacityforsupportingloadsthatare
suspended from the bottom of the floor suspendedfromthebottomofthefloor.
Therecommendedhangingcapacityofthe
fl i 2 4kPA flooris2.4kPA
KPA/1000=N/persqmeter
Data Center Lighting DataCenterLighting
Aminimumof500lux inthehoriziontal plane
and200lux intheverticalplan. p
Thismustbemeasured1mabovethefinished
floor in the middle of all aisles between floorinthemiddleofallaislesbetween
cabinets.
Data Center Access DataCenterAccess
Doorsmustbeaminimumof1mwideand
2.13mhighwithoutdoorsills,hingedtoopen g g p
outward
Door must be fitted with locks and have either Doormustbefittedwithlocksandhaveeither
centerpostsorremovablecenterpoststo
f l f l facilitateaccessforlargeequipment
DDA Ramps to be not less than 1.12 and all DDA Rampstobenotlessthan1.12andall
pathwaystobe900wide
Data Center Decor DataCenterDecor
Floors,wallsandceilingmustbe
Sealed
Paintedand,
Constructed out of material to minimize dust Constructedoutofmaterialtominimizedust.
Floorsmusthaveantistaticpropertiesasper
IEC6100042
Data Center Fire Regulations DataCenterFireRegulations
FirePlanandRiskassessment
Emergency Lighting and Signage EmergencyLightingandSignage
DoorandEmergencyexits
MUST ALL BE CLEARLY MARKED MUSTALLBECLEARLY MARKED
Data Center Connection Services DataCenterConnectionServices
El t i it S l ElectricitySupply
Whofrom?
WhatCapacity? p y
Wheredoesitenterthebuilding?
Istheremorethanone?
T l i ti Telecommunications
Whofrom?
Whatcapacity/type p y / yp
Wheredoesitenterthebuilding
Istheremorethanone.
G t d i d Gas,water,drainageandsewage
Nootherserviceshouldcrossthecomputerroomspace
Data Center External services DataCenterExternalservices
S it bl l ti f t db t Suitablelocationforstandbygenerators
Whatnoise?
Whatweight?
WhatSecurity
WhatFuelStorage
Whatproximity
Suitablelocationforexternalairconditioningcomponents
What DX condenser units? WhatDXcondenserunits?
Whatcentralchiller system?
Whatweight?
What security? Whatsecurity?
Whatpowersupplies?
Whatproximity
END END