Professional Documents
Culture Documents
SAN Design PDF
SAN Design PDF
VMware Infrastructure 3
VMwareESXServercanbeusedinconjunctionwithaSAN(storageareanetwork),a
specializedhighspeednetworkthatconnectscomputersystemstohighperformance
storagesubsystems.UsingESXServertogetherwithaSANprovidesextrastoragefor
consolidation,improvesreliability,andhelpswithdisasterrecovery.
TouseESXServereffectivelywithaSAN,youreexpectedtobefamiliarwiththeSAN
technology.ThiswhitepaperoffersabriefintroductiontosomebasicSANconcepts,
butdoesntaimtobeanexhaustivesourceofinformationonSANs.IfyouareanESX
ServeradministratorplanningtosetupESXServerhoststoworkwithSANs,you
shouldalsoconsultotherresourcesavailableinprintandontheInternettoachievea
workingknowledgeofSANconcepts.AdditionalinformationonhowESXServer
interactswithSANmaybefoundinSANConfigurationGuide.
Thewhitepaperdiscussesthesetopics:
!
SANBasicsonpage 1
SANComponentsonpage 4
UnderstandingSANInteractionsonpage 7
SANInstallationConsiderationsonpage 10
SANDesignBasicsonpage 11
SAN Basics
ASANisaspecializedhighspeednetworkofstoragedevicesandswitchesconnected
tocomputersystems.Thiswhitepaperreferstothecomputersystemsasserversor
hosts.
ASANpresentssharedpoolsofstoragedevicestomultipleservers.Eachservercan
accessthestorageasifitweredirectlyattachedtothatserver.ASANsupports
centralizedstoragemanagement.SANsmakeitpossibletomovedatabetweenvarious
storagedevices,sharedatabetweenmultipleservers,andbackupandrestoredata
rapidlyandefficiently.Inaddition,aproperlyconfiguredSANfacilitatesbothdisaster
recoveryandhighavailability.
ThephysicalcomponentsofaSANcanbegroupedinasinglerackordatacenteror
connectedoverlongdistances.ThismakesaSANafeasiblesolutionforbusinessesof
anysize:theSANcangroweasilywiththebusinessitsupports.
SANSwitches(3)SANswitchesconnectvariouselementsoftheSAN.In
particular,theymightconnecthoststostoragearrays.SANswitchesalsoallow
administratorstosetuppathredundancyintheeventofapathfailurefromhost
servertoswitchorfromstoragearraytoswitch.
Fabric(4)TheSANfabricistheactualnetworkportionoftheSAN.Whenone
ormoreSANswitchesareconnected,afabriciscreated.TheFCprotocolisusedto
communicateovertheentirenetwork.ASANcanconsistofmultiple
interconnectedfabrics.EvenasimpleSANoftenconsistsoftwofabricsfor
redundancy.
VMware, Inc.
Connections:HostBusAdapters(5)andStorageProcessors(6)Hostservers
andstoragesystemsareconnectedtotheSANfabricthroughportsinthefabric.
!
AhostconnectstoafabricportthroughanHBA.
Storagedevicesconnecttofabricportsthroughtheirstorageprocessors.
ESX
Server
1
HBA 2
HBA 1
5
4 fabric A
fabric B
SAN switch
SAN switch
6
SP
SP
storage array
WhenahostwantstoaccessastoragedeviceontheSAN,itsendsouta
blockbasedaccessrequestforthestoragedevice.
SCSIcommandsareencapsulatedintoFCpackets.Therequestisacceptedbythe
HBAforthathostandisconvertedfromitsbinarydataformtotheopticalform
requiredfortransmissiononthefiberopticcable.
Atthesametime,therequestispackagedaccordingtotherulesoftheFCprotocol.
TheHBAtransmitstherequesttotheSAN.
DependingonwhichportisusedbytheHBAtoconnecttothefabric,oneofthe
SANswitchesreceivestherequestandsendsittothestorageprocessor,which
sendsitontothestoragedevice.
Theremainingsectionsofthiswhitepaperprovideadditionalinformationaboutthe
componentsoftheSANandhowtheyinteroperate.Thesesectionsalsopresentgeneral
informationonconfigurationoptionsanddesignconsiderations.
VMware, Inc.
SAN Components
ThecomponentsofanFCSANcanbegroupedasfollowsandarediscussedbelow:
!
HostComponentsonpage 4
FabricComponentsonpage 5
StorageComponentsonpage 5
Figure 2showstheSANcomponentlayers.
ESX
Server
ESX
Server
host
components
HBA
HBA
HBA
HBA
fabric A
fabric B
fabric
components
SAN switch A
SAN switch B
SP
SP
SP
SP
storage
components
storage array
storage array
Host Components
ThehostcomponentsofaSANconsistoftheserversthemselvesandthecomponents
thatenabletheserverstobephysicallyconnectedtotheSAN.
!
HBAsarelocatedintheservers,alongwithacomponentthatperforms
digitaltoopticalsignalconversion.Eachhostconnectstothefabricportsthrough
itsHBAs.
HBAdriversrunningontheserversenabletheserversoperatingsystemsto
communicatewiththeHBA.
VMware, Inc.
Fabric Components
AllhostsconnecttothestoragedevicesontheSANthroughtheSANfabric.The
networkportionoftheSANconsistsofthefollowingfabriccomponents:
!
SANSwitchesSANswitchescanconnecttoservers,storagedevices,andother
switches,andthusprovidetheconnectionpointsfortheSANfabric.Thetypeof
SANswitch,itsdesignfeatures,anditsportcapacityallcontributetoitsoverall
capacity,performance,andfaulttolerance.Thenumberofswitches,typesof
switches,andmannerinwhichtheswitchesareinterconnecteddefinethefabric
topology.
!
ForsmallerSANs,thestandardSANswitches(calledmodularswitches)can
typicallysupport16or24ports(thoughsome32portmodularswitchesare
becomingavailable).Sometimesmodularswitchesareinterconnectedto
createafaulttolerantfabric.
ForlargerSANfabrics,directorclassswitchesprovidealargerportcapacity
(64to128portsperswitch)andbuiltinfaulttolerance.
DataRoutersDataroutersareintelligentbridgesbetweenSCSIdevicesandFC
devicesintheSAN.ServersintheSANcanaccessSCSIdiskortapedevicesinthe
SANthroughthedataroutersinthefabriclayer.
CablesSANcablesareusuallyspecialfiberopticcablesthatareusedtoconnect
allofthefabriccomponents.ThetypeofSANcableandthefiberopticsignal
determinethemaximumdistancesbetweenSANcomponentsandcontributeto
thetotalbandwidthratingoftheSAN.
CommunicationsProtocolFabriccomponentscommunicateusingtheFC
communicationsprotocol.FCisthestorageinterfaceprotocolusedformostof
todaysSANs.FCwasdevelopedasaprotocolfortransferringdatabetweentwo
portsonaserialI/Obuscableathighspeeds.FCsupportspointtopoint,
arbitratedloop,andswitchedfabrictopologies.Switchedfabrictopologyisthe
basisformostcurrentSANs.
Storage Components
ThestoragecomponentsofaSANarethestoragearrays.Storagearraysincludestorage
processors(SPs).TheSPsarethefrontendofthestoragearray.SPscommunicatewith
thediskarray(whichincludesallthedisksinthestoragearray)andprovidethe
RAID/LUNfunctionality.
Storage Processors
SPsprovidefrontsidehostattachmentstothestoragedevicesfromtheservers,either
directlyorthroughaswitch.TheserverHBAsmustconformtotheprotocolsupported
bythestorageprocessor.Inmostcases,thisistheFCprotocol.
VMware, Inc.
Storageprocessorsprovideinternalaccesstothedrives,whichcanbeusingaswitchor
busarchitecture.Inhighendstoragesystems,drivesarenormallyconnectedinloops.
ThisbackendlooptechnologyemployedbytheSPprovidesseveralbenefits:
!
Highspeedaccesstothedrives
Abilitytoaddmoredrivestotheloop
Redundantaccesstoasingledrivefrommultipleloops(whendrivesare
dualportedandattachedtotwoloops)
Storage Devices
Dataisstoredondiskarraysortapedevices(orboth).
DiskarraysaregroupsofmultiplediskdevicesandarethetypicalSANdiskstorage
device.Theycanvarygreatlyindesign,capacity,performance,andotherfeatures.
Storagearraysrarelyprovidehostsdirectaccesstoindividualdrives.Instead,the
storagearrayusesRAID(RedundantArrayofIndependentDrives)technologyto
groupasetofdrives.RAIDusesindependentdrivestoprovidecapacity,performance,
andredundancy.Usingspecializedalgorithms,severaldrivesaregroupedtoprovide
commonpooledstorage.TheseRAIDalgorithms,commonlyknownasRAIDlevels,
definethecharacteristicsoftheparticulargrouping.
InsimplesystemsthatprovideRAIDcapability,aRAIDgroupisequivalenttoasingle
LUN.ALUNisasingleunitofstorage.Dependingonthehostsystemenvironment,a
LUNisalsoknownasavolumeoralogicaldrive.FromaVIClient,aLUNlookslike
anyotherstorageunitavailableforaccess.
Inadvancedstoragearrays,RAIDgroupscanhaveoneormoreLUNscreatedfor
accessbyoneormoreservers.TheabilitytocreatemorethanoneLUNfromasingle
RAIDgroupprovidesfinegranularitytothestoragecreationprocess.Youarenot
limitedtothetotalcapacityoftheentireRAIDgroupforasingleLUN.
Note
ASANadministratormustbefamiliarwiththedifferentRAIDlevelsandunderstand
howtomanagethem.Discussionofthosetopicsisbeyondthescopeofthisdocument.
Moststoragearraysprovideadditionaldataprotectionandreplicationfeaturessuchas
snapshots,internalcopies,andremotemirroring.
!
AsnapshotisapointintimecopyofaLUN.Snapshotsareusedasbackupsources
fortheoverallbackupproceduresdefinedforthestoragearray.
InternalcopiesallowdatamovementfromoneLUNtoanotherforanadditional
copyfortesting.
RemotemirroringprovidesconstantsynchronizationbetweenLUNsonone
storagearrayandasecond,independent(usuallyremote)storagearrayfor
disasterrecovery.
VMware, Inc.
SmallerSANsmightusehighcapacitytapedrives.Thesetapedrivesvaryintheir
transferratesandstoragecapacities.Ahighcapacitytapedrivemightexistasa
standalonedrive,oritmightbepartofatapelibrary.
Typically,alargeSAN,oraSANwithcriticalbackuprequirements,isconfigured
withoneormoretapelibraries.Atapelibraryconsolidatesoneormoretapedrives
intoasingleenclosure.Tapescanbeinsertedandremovedfromthetapedrivesin
thelibraryautomaticallywitharoboticarm.Manytapelibrariesofferlargestorage
capacitiessometimesintothepetabyte(PB)range.
SANPortsandPortNamingonpage 7
MultipathingandPathFailoveronpage 7
Active/ActiveandActive/PassiveDiskArraysonpage 8
Zoningonpage 9
Inthecontextofthisdocument,aportistheconnectionfromadeviceintothe
SAN.EachnodeintheSANeachhost,storagedevice,andfabriccomponent
(routerorswitch)hasoneormoreportsthatconnectittotheSAN.Portscanbe
identifiedinanumberofways:
WWPNWorldWidePortName.Agloballyuniqueidentifierforaportwhich
allowscertainapplicationstoaccesstheport.TheFCswitchesdiscovertheWWPN
ofadeviceorhostandassignaportaddresstothedevice.
Port_ID(orportaddress)WithintheSAN,eachporthasauniqueportIDthat
servesastheFCaddressfortheport.ThisenablesroutingofdatathroughtheSAN
tothatport.TheFCswitchesassigntheportIDwhenthedevicelogsintothe
fabric.TheportIDisvalidonlywhilethedeviceisloggedon.
IndepthinformationonSANportscanbefoundathttp://www.snia.org,theWebsite
oftheStorageNetworkingIndustryAssociation.
FromaspecificHBAportinthehost,
Throughtheswitchesinthefabric,and
Intoaspecificstorageportonthestoragearray.
VMware, Inc.
AgivenhostmightbeabletoaccessaLUNonastoragearraythroughmorethanone
path.HavingmorethanonepathfromahosttoaLUNiscalledmultipathing.
Bydefault,VMwareESXServersystemsuseonlyonepathfromthehosttoagiven
LUNatanytime.IfthepathactivelybeingusedbytheVMwareESXServersystem
fails,theserverselectsanotheroftheavailablepaths.Theprocessofdetectingafailed
pathandswitchingtoanotheriscalledpathfailover.Apathfailsifanyofthe
componentsalongthepathHBA,cable,switchport,orstorageprocessorfails.
Anactive/activediskarrayallowsaccesstotheLUNssimultaneouslythroughall
thestorageprocessorsthatareavailablewithoutsignificantperformance
degradation.Allthepathsareactiveatalltimes(unlessapathfails).
Inanactive/passivediskarray,oneSPisactivelyservicingagivenLUN.Theother
SPactsasbackupfortheLUNandmaybeactivelyservicingotherLUNI/O.I/O
canbesentonlytoanactiveprocessor.Iftheprimarystorageprocessorfails,one
ofthesecondarystorageprocessorsbecomesactive,eitherautomaticallyor
throughadministratorintervention.
Note
Usingactive/passivearrayswithpathpolicyFixedcanpotentiallyleadtopath
thrashing.SeeESXServerSANConfigurationGuideformoreinformationonresolving
paththrashing.
InFigure 3,onestorageprocessorisactive,theotherispassive.Dataarrivesthrough
theactivearrayonly.
storage
processor
storage
processor
active
passive
physical
disks
storage array
VMware, Inc.
Zoning
ZoningprovidesaccesscontrolintheSANtopology;itdefineswhichHBAscan
connecttowhichSPs.YoucanhavemultipleportstothesameSPindifferentzonesto
reducethenumberofpresentedpaths.
WhenaSANisconfiguredusingzoning,thedevicesoutsideazonearenotvisibleto
thedevicesinsidethezone.Inaddition,SANtrafficwithineachzoneisisolatedfrom
theotherzones.
WithinacomplexSANenvironment,SANswitchesprovidezoning.Zoningdefines
andconfiguresthenecessarysecurityandaccessrightsfortheentireSAN.
Typically,zonesarecreatedforeachgroupofserversthataccessasharedgroupof
storagedevicesandLUNs.Youcanusezoninginseveralways.Herearesome
examples:
!
ZoningforsecurityandisolationYoucanmanagezonesdefinedfortesting
independentlywithintheSANsotheydontinterferewiththeactivitygoingonin
theproductionzones.Similarly,youcouldsetupdifferentzonesfordifferent
departments.
ZoningforsharedservicesAnotheruseofzonesistoallowcommonserver
accessforbackups.SANdesignsoftenhaveabackupserverwithtapeservicesthat
requireSANwideaccesstohostserversindividuallyforbackupandrecovery
processes.Thesebackupserversneedtobeabletoaccesstheserverstheybackup.
ASANzonemightbedefinedforthebackupservertoaccessaparticularhostto
performabackuporrecoveryprocess.Thezoneisthenredefinedforaccessto
anotherhostwhenthebackupserverisreadytoperformbackuporrecovery
processesonthathost.
MultiplestoragearraysZonesarealsousefulwhentherearemultiplestorage
arrays.Throughtheuseofseparatezones,eachstoragearrayismanaged
separatelyfromtheothers,withnoconcernforaccessconflictsbetweenservers.
LUN Masking
LUNmaskingiscommonlyusedforpermissionmanagement.LUNmaskingisalso
referredtoasselectivestoragepresentation,accesscontrol,andpartitioning,
dependingonthevendor.
LUNmaskingisperformedattheSPorserverlevel;itmakesaLUNinvisiblewhena
targetisscanned.Theadministratorconfiguresthediskarraysoeachserverorgroup
VMware, Inc.
ofserverscanseeonlycertainLUNs.Maskingcapabilitiesforeachdiskarrayare
vendorspecific,asarethetoolsformanagingLUNmasking.
ESX
Server
ESX
Server
Mask LUN 11
21:00:00:E0:8B:19:AB:31
zoning:
done at switch level;
used to segment
the fabric
Mask LUN 12
HBA
HBA
21:00:00:E0:8B:19:B2:33
LUN masking:
done at SP or server level;
makes a LUN invisible
when a target is scanned
FC switch
50:05:01:60:10:20:AD:87
SP
LUNs
0
11
12
WWN
(world-wide name)
unique, 64-bit
address assigned
to Fibre Channel node
storage array
Requirements
TointegrateallcomponentsoftheSAN,youmustmeetthevendorshardwareand
softwarecompatibilityrequirements,includingthefollowing:
!
HBA(firmwareversion,driverversion,andpatchlist)
Switch(firmware)
Storage(firmware,hostpersonalityfirmware,andpatchlist)
VMware, Inc.
10
SAN Setup
WhenyourereadytosetuptheSAN,completethesetasks.
To prepare the SAN
1
Assembleandcabletogetherallhardwarecomponentsandinstallthe
correspondingsoftware.
a
Checktheversions.
SetuptheHBA.
Setupthestoragearray.
Changeanyconfigurationsettingsthatmightberequired.
Testtheintegration.
Duringintegrationtesting,testalltheoperationalprocessesfortheSAN
environment.Theseincludenormalproductionprocessing,failuremodetesting,
backupfunctions,andsoforth.
EstablishabaselineofperformanceforeachcomponentandfortheentireSAN.
Eachbaselineprovidesameasurementmetricforfuturechangesandtuning.See
ESXServerSANConfigurationGuideforadditionalinformation.
DocumenttheSANinstallationandalloperationalprocedures.
DefiningApplicationNeedsonpage 11
ConfiguringtheStorageArrayonpage 12
ConsideringHighAvailabilityonpage 13
PlanningforDisasterRecoveryonpage 13
VMware, Inc.
11
simplenorquick.ThefirststepindesigninganoptimalSANistodefinethestorage
requirementsforeachapplicationintermsof:
!
I/Operformance(I/Opersecond)
Bandwidth(megabytespersecond)
Capacity(numberofLUNsandcapacityofeachLUN)
Redundancylevel(RAIDlevel)
Responsetimes(averagetimeperI/O)
Overallprocessingpriority
EachRAIDgroupprovidesaspecificlevelofI/Operformance,capacity,and
redundancy.LUNsareassignedtoRAIDgroupsbasedontheserequirements.
IfaparticularRAIDgroupcannotprovidetherequiredI/Operformance,capacity,
andresponsetimes,youmustdefineanadditionalRAIDgroupforthenextsetof
LUNs.YoumustprovidesufficientRAIDgroupresourcesforeachsetofLUNs.
ThestoragearraysneedtodistributetheRAIDgroupsacrossallinternalchannels
andaccesspaths.ThisresultsinloadbalancingofallI/Orequeststomeet
performancerequirementsofI/Ooperationspersecondandresponsetimes.
Caching
ThoughESXServersystemsbenefitfromwritecache,thecachecouldbesaturatedwith
sufficientlyintenseI/O.Saturationreducesthecacheseffectiveness.
Becausethecacheisoftenallocatedfromaglobalpool,itshouldbeallocatedonlyifit
willbeeffective.
!
AreadaheadcachemaybeeffectiveforsequentialI/O,suchasduringcertain
typesofbackupactivities,andfortemplaterepositories.
AreadcacheisoftenineffectivewhenappliedtoaVMFSbasedLUNbecause
multiplevirtualmachinesareaccessedconcurrently.Becausedataaccessis
random,thereadcachehitrateisoftentoolowtojustifyallocatingareadcache.
VMware, Inc.
12
Areadcacheisoftenunnecessarywhentheapplicationandoperatingsystem
cachedataarewithinthevirtualmachinesmemory.Inthatcase,thereadcache
cachesdataobjectsthattheapplicationoroperatingsystemalreadycache.
RedundantSANComponentsRedundantSANhardwarecomponents
includingHBAs,SANswitches,andstoragearrayaccessports,arerequired.In
somecases,multiplestoragearraysarepartofafaulttolerantSANdesign.
RedundantI/OPathsI/Opathsfromtheservertothestoragearraymustbe
redundantanddynamicallyswitchableintheeventofaport,device,cable,orpath
failure.
I/OConfigurationThekeytoprovidingfaulttoleranceiswithinthe
configurationofeachserversI/Osystem.
WithmultipleHBAs,theI/OsystemcanissueI/OacrossalloftheHBAstothe
assignedLUNs.Failurescanhavethefollowingresults:
IfanHBA,cable,orSANswitchportfails,thepathisnolongeravailableand
analternatepathisrequired.
IfafailureoccursintheprimarypathbetweentheSANswitchandthestorage
array,thenanalternatepathatthatlevelisrequired.
IfaSANswitchfails,theentirepathfromservertostoragearrayisdisabled,
soasecondfabricwithacompletealternatepathisrequired.
MirroringProtectionagainstLUNfailureallowsapplicationstosurvivestorage
accessfaults.Mirroringcanaccomplishthatprotection.
MirroringdesignatesasecondnonaddressableLUNthatcapturesallwrite
operationstotheprimaryLUN.MirroringprovidesfaulttoleranceattheLUN
level.LUNmirroringcanbeimplementedattheserver,SANswitch,orstorage
arraylevel.
Note
UsuallyRAIDSdontmirror.
DuplicationofSANEnvironmentForextremelyhighavailability
requirements,SANenvironmentsmaybeduplicatedtoprovidedisasterrecovery
onapersitebasis.TheSANenvironmentmustbeduplicatedatdifferentphysical
locations.ThetworesultantSANenvironmentsmayshareoperationalworkloads
orthesecondSANenvironmentmaybeafailoveronlysite.
VMware, Inc.
13
analternateservertostartthedatarecoveryprocess.TheSANmayhandlethesitedata
synchronization.
ESXServermakesdisasterrecoveryeasierbecauseyoudonothavetoreinstallan
operatingsystemonadifferentphysicalmachine.Justrestorethevirtualmachine
imageandcontinuewhatyouweredoing.
14