Professional Documents
Culture Documents
San Conceptual and Design Basics: Vmware Infrastructure 3
San Conceptual and Design Basics: Vmware Infrastructure 3
VMware Infrastructure 3
VMwareESXServercanbeusedinconjunctionwithaSAN(storageareanetwork),a
specializedhighspeednetworkthatconnectscomputersystemstohighperformance
storagesubsystems.UsingESXServertogetherwithaSANprovidesextrastoragefor
consolidation,improvesreliability,andhelpswithdisasterrecovery.
TouseESXServereffectivelywithaSAN,youreexpectedtobefamiliarwiththeSAN
technology.ThiswhitepaperoffersabriefintroductiontosomebasicSANconcepts,
butdoesntaimtobeanexhaustivesourceofinformationonSANs.IfyouareanESX
ServeradministratorplanningtosetupESXServerhoststoworkwithSANs,you
shouldalsoconsultotherresourcesavailableinprintandontheInternettoachievea
workingknowledgeofSANconcepts.AdditionalinformationonhowESXServer
interactswithSANmaybefoundinSANConfigurationGuide.
Thewhitepaperdiscussesthesetopics:
! SANBasicsonpage 1
! SANComponentsonpage 4
! UnderstandingSANInteractionsonpage 7
! SANInstallationConsiderationsonpage 10
! SANDesignBasicsonpage 11
SAN Basics
ASANisaspecializedhighspeednetworkofstoragedevicesandswitchesconnected
tocomputersystems.Thiswhitepaperreferstothecomputersystemsasserversor
hosts.
ASANpresentssharedpoolsofstoragedevicestomultipleservers.Eachservercan
accessthestorageasifitweredirectlyattachedtothatserver.ASANsupports
centralizedstoragemanagement.SANsmakeitpossibletomovedatabetweenvarious
storagedevices,sharedatabetweenmultipleservers,andbackupandrestoredata
rapidlyandefficiently.Inaddition,aproperlyconfiguredSANfacilitatesbothdisaster
recoveryandhighavailability.
1
SAN Conceptual and Design Basics
ThephysicalcomponentsofaSANcanbegroupedinasinglerackordatacenteror
connectedoverlongdistances.ThismakesaSANafeasiblesolutionforbusinessesof
anysize:theSANcangroweasilywiththebusinessitsupports.
Initssimplestform,aSANconsistsofoneormoreservers(1)attachedtoastorage
array(2)usingoneormoreSANswitches.Eachservermighthostnumerous
applicationsthatrequirededicatedstorageforapplicationsprocessing.
Thefollowingcomponents,discussedinmoredetailinSANComponentsonpage 4
areinvolved:
! SANSwitches(3)SANswitchesconnectvariouselementsoftheSAN.In
particular,theymightconnecthoststostoragearrays.SANswitchesalsoallow
administratorstosetuppathredundancyintheeventofapathfailurefromhost
servertoswitchorfromstoragearraytoswitch.
! Fabric(4)TheSANfabricistheactualnetworkportionoftheSAN.Whenone
ormoreSANswitchesareconnected,afabriciscreated.TheFCprotocolisusedto
communicateovertheentirenetwork.ASANcanconsistofmultiple
interconnectedfabrics.EvenasimpleSANoftenconsistsoftwofabricsfor
redundancy.
VMware, Inc. 2
SAN Conceptual and Design Basics
! Connections:HostBusAdapters(5)andStorageProcessors(6)Hostservers
andstoragesystemsareconnectedtotheSANfabricthroughportsinthefabric.
! AhostconnectstoafabricportthroughanHBA.
! Storagedevicesconnecttofabricportsthroughtheirstorageprocessors.
ESX
Server
1
HBA 2 HBA 1
5
4 fabric A 4 fabric B
3 3
6
SP SP
2 storage array
1 WhenahostwantstoaccessastoragedeviceontheSAN,itsendsouta
blockbasedaccessrequestforthestoragedevice.
2 SCSIcommandsareencapsulatedintoFCpackets.Therequestisacceptedbythe
HBAforthathostandisconvertedfromitsbinarydataformtotheopticalform
requiredfortransmissiononthefiberopticcable.
3 Atthesametime,therequestispackagedaccordingtotherulesoftheFCprotocol.
4 TheHBAtransmitstherequesttotheSAN.
5 DependingonwhichportisusedbytheHBAtoconnecttothefabric,oneofthe
SANswitchesreceivestherequestandsendsittothestorageprocessor,which
sendsitontothestoragedevice.
Theremainingsectionsofthiswhitepaperprovideadditionalinformationaboutthe
componentsoftheSANandhowtheyinteroperate.Thesesectionsalsopresentgeneral
informationonconfigurationoptionsanddesignconsiderations.
VMware, Inc. 3
SAN Conceptual and Design Basics
SAN Components
ThecomponentsofanFCSANcanbegroupedasfollowsandarediscussedbelow:
! HostComponentsonpage 4
! FabricComponentsonpage 5
! StorageComponentsonpage 5
Figure 2showstheSANcomponentlayers.
ESX ESX
Server Server
host
components
fabric A fabric B
fabric
components
SAN switch A SAN switch B
SP SP SP SP
storage
components
Host Components
ThehostcomponentsofaSANconsistoftheserversthemselvesandthecomponents
thatenabletheserverstobephysicallyconnectedtotheSAN.
! HBAsarelocatedintheservers,alongwithacomponentthatperforms
digitaltoopticalsignalconversion.Eachhostconnectstothefabricportsthrough
itsHBAs.
! HBAdriversrunningontheserversenabletheserversoperatingsystemsto
communicatewiththeHBA.
VMware, Inc. 4
SAN Conceptual and Design Basics
Fabric Components
AllhostsconnecttothestoragedevicesontheSANthroughtheSANfabric.The
networkportionoftheSANconsistsofthefollowingfabriccomponents:
! SANSwitchesSANswitchescanconnecttoservers,storagedevices,andother
switches,andthusprovidetheconnectionpointsfortheSANfabric.Thetypeof
SANswitch,itsdesignfeatures,anditsportcapacityallcontributetoitsoverall
capacity,performance,andfaulttolerance.Thenumberofswitches,typesof
switches,andmannerinwhichtheswitchesareinterconnecteddefinethefabric
topology.
! ForsmallerSANs,thestandardSANswitches(calledmodularswitches)can
typicallysupport16or24ports(thoughsome32portmodularswitchesare
becomingavailable).Sometimesmodularswitchesareinterconnectedto
createafaulttolerantfabric.
! ForlargerSANfabrics,directorclassswitchesprovidealargerportcapacity
(64to128portsperswitch)andbuiltinfaulttolerance.
! DataRoutersDataroutersareintelligentbridgesbetweenSCSIdevicesandFC
devicesintheSAN.ServersintheSANcanaccessSCSIdiskortapedevicesinthe
SANthroughthedataroutersinthefabriclayer.
! CablesSANcablesareusuallyspecialfiberopticcablesthatareusedtoconnect
allofthefabriccomponents.ThetypeofSANcableandthefiberopticsignal
determinethemaximumdistancesbetweenSANcomponentsandcontributeto
thetotalbandwidthratingoftheSAN.
! CommunicationsProtocolFabriccomponentscommunicateusingtheFC
communicationsprotocol.FCisthestorageinterfaceprotocolusedformostof
todaysSANs.FCwasdevelopedasaprotocolfortransferringdatabetweentwo
portsonaserialI/Obuscableathighspeeds.FCsupportspointtopoint,
arbitratedloop,andswitchedfabrictopologies.Switchedfabrictopologyisthe
basisformostcurrentSANs.
Storage Components
ThestoragecomponentsofaSANarethestoragearrays.Storagearraysincludestorage
processors(SPs).TheSPsarethefrontendofthestoragearray.SPscommunicatewith
thediskarray(whichincludesallthedisksinthestoragearray)andprovidethe
RAID/LUNfunctionality.
Storage Processors
SPsprovidefrontsidehostattachmentstothestoragedevicesfromtheservers,either
directlyorthroughaswitch.TheserverHBAsmustconformtotheprotocolsupported
bythestorageprocessor.Inmostcases,thisistheFCprotocol.
VMware, Inc. 5
SAN Conceptual and Design Basics
Storageprocessorsprovideinternalaccesstothedrives,whichcanbeusingaswitchor
busarchitecture.Inhighendstoragesystems,drivesarenormallyconnectedinloops.
ThisbackendlooptechnologyemployedbytheSPprovidesseveralbenefits:
! Highspeedaccesstothedrives
! Abilitytoaddmoredrivestotheloop
! Redundantaccesstoasingledrivefrommultipleloops(whendrivesare
dualportedandattachedtotwoloops)
Storage Devices
Dataisstoredondiskarraysortapedevices(orboth).
DiskarraysaregroupsofmultiplediskdevicesandarethetypicalSANdiskstorage
device.Theycanvarygreatlyindesign,capacity,performance,andotherfeatures.
Storagearraysrarelyprovidehostsdirectaccesstoindividualdrives.Instead,the
storagearrayusesRAID(RedundantArrayofIndependentDrives)technologyto
groupasetofdrives.RAIDusesindependentdrivestoprovidecapacity,performance,
andredundancy.Usingspecializedalgorithms,severaldrivesaregroupedtoprovide
commonpooledstorage.TheseRAIDalgorithms,commonlyknownasRAIDlevels,
definethecharacteristicsoftheparticulargrouping.
InsimplesystemsthatprovideRAIDcapability,aRAIDgroupisequivalenttoasingle
LUN.ALUNisasingleunitofstorage.Dependingonthehostsystemenvironment,a
LUNisalsoknownasavolumeoralogicaldrive.FromaVIClient,aLUNlookslike
anyotherstorageunitavailableforaccess.
Inadvancedstoragearrays,RAIDgroupscanhaveoneormoreLUNscreatedfor
accessbyoneormoreservers.TheabilitytocreatemorethanoneLUNfromasingle
RAIDgroupprovidesfinegranularitytothestoragecreationprocess.Youarenot
limitedtothetotalcapacityoftheentireRAIDgroupforasingleLUN.
Note ASANadministratormustbefamiliarwiththedifferentRAIDlevelsandunderstand
howtomanagethem.Discussionofthosetopicsisbeyondthescopeofthisdocument.
Moststoragearraysprovideadditionaldataprotectionandreplicationfeaturessuchas
snapshots,internalcopies,andremotemirroring.
! AsnapshotisapointintimecopyofaLUN.Snapshotsareusedasbackupsources
fortheoverallbackupproceduresdefinedforthestoragearray.
! InternalcopiesallowdatamovementfromoneLUNtoanotherforanadditional
copyfortesting.
! RemotemirroringprovidesconstantsynchronizationbetweenLUNsonone
storagearrayandasecond,independent(usuallyremote)storagearrayfor
disasterrecovery.
VMware, Inc. 6
SAN Conceptual and Design Basics
! SmallerSANsmightusehighcapacitytapedrives.Thesetapedrivesvaryintheir
transferratesandstoragecapacities.Ahighcapacitytapedrivemightexistasa
standalonedrive,oritmightbepartofatapelibrary.
! Typically,alargeSAN,oraSANwithcriticalbackuprequirements,isconfigured
withoneormoretapelibraries.Atapelibraryconsolidatesoneormoretapedrives
intoasingleenclosure.Tapescanbeinsertedandremovedfromthetapedrivesin
thelibraryautomaticallywitharoboticarm.Manytapelibrariesofferlargestorage
capacitiessometimesintothepetabyte(PB)range.
! SANPortsandPortNamingonpage 7
! MultipathingandPathFailoveronpage 7
! Active/ActiveandActive/PassiveDiskArraysonpage 8
! Zoningonpage 9
! WWPNWorldWidePortName.Agloballyuniqueidentifierforaportwhich
allowscertainapplicationstoaccesstheport.TheFCswitchesdiscovertheWWPN
ofadeviceorhostandassignaportaddresstothedevice.
! Port_ID(orportaddress)WithintheSAN,eachporthasauniqueportIDthat
servesastheFCaddressfortheport.ThisenablesroutingofdatathroughtheSAN
tothatport.TheFCswitchesassigntheportIDwhenthedevicelogsintothe
fabric.TheportIDisvalidonlywhilethedeviceisloggedon.
IndepthinformationonSANportscanbefoundathttp://www.snia.org,theWebsite
oftheStorageNetworkingIndustryAssociation.
! Throughtheswitchesinthefabric,and
! Intoaspecificstorageportonthestoragearray.
VMware, Inc. 7
SAN Conceptual and Design Basics
AgivenhostmightbeabletoaccessaLUNonastoragearraythroughmorethanone
path.HavingmorethanonepathfromahosttoaLUNiscalledmultipathing.
Bydefault,VMwareESXServersystemsuseonlyonepathfromthehosttoagiven
LUNatanytime.IfthepathactivelybeingusedbytheVMwareESXServersystem
fails,theserverselectsanotheroftheavailablepaths.Theprocessofdetectingafailed
pathandswitchingtoanotheriscalledpathfailover.Apathfailsifanyofthe
componentsalongthepathHBA,cable,switchport,orstorageprocessorfails.
! Anactive/activediskarrayallowsaccesstotheLUNssimultaneouslythroughall
thestorageprocessorsthatareavailablewithoutsignificantperformance
degradation.Allthepathsareactiveatalltimes(unlessapathfails).
! Inanactive/passivediskarray,oneSPisactivelyservicingagivenLUN.Theother
SPactsasbackupfortheLUNandmaybeactivelyservicingotherLUNI/O.I/O
canbesentonlytoanactiveprocessor.Iftheprimarystorageprocessorfails,one
ofthesecondarystorageprocessorsbecomesactive,eitherautomaticallyor
throughadministratorintervention.
Note Usingactive/passivearrayswithpathpolicyFixedcanpotentiallyleadtopath
thrashing.SeeESXServerSANConfigurationGuideformoreinformationonresolving
paththrashing.
InFigure 3,onestorageprocessorisactive,theotherispassive.Dataarrivesthrough
theactivearrayonly.
storage storage
processor processor
active passive
physical
disks
storage array
Figure 3. Active/passive Storage Array
VMware, Inc. 8
SAN Conceptual and Design Basics
Zoning
ZoningprovidesaccesscontrolintheSANtopology;itdefineswhichHBAscan
connecttowhichSPs.YoucanhavemultipleportstothesameSPindifferentzonesto
reducethenumberofpresentedpaths.
WhenaSANisconfiguredusingzoning,thedevicesoutsideazonearenotvisibleto
thedevicesinsidethezone.Inaddition,SANtrafficwithineachzoneisisolatedfrom
theotherzones.
WithinacomplexSANenvironment,SANswitchesprovidezoning.Zoningdefines
andconfiguresthenecessarysecurityandaccessrightsfortheentireSAN.
Typically,zonesarecreatedforeachgroupofserversthataccessasharedgroupof
storagedevicesandLUNs.Youcanusezoninginseveralways.Herearesome
examples:
! ZoningforsecurityandisolationYoucanmanagezonesdefinedfortesting
independentlywithintheSANsotheydontinterferewiththeactivitygoingonin
theproductionzones.Similarly,youcouldsetupdifferentzonesfordifferent
departments.
! ZoningforsharedservicesAnotheruseofzonesistoallowcommonserver
accessforbackups.SANdesignsoftenhaveabackupserverwithtapeservicesthat
requireSANwideaccesstohostserversindividuallyforbackupandrecovery
processes.Thesebackupserversneedtobeabletoaccesstheserverstheybackup.
ASANzonemightbedefinedforthebackupservertoaccessaparticularhostto
performabackuporrecoveryprocess.Thezoneisthenredefinedforaccessto
anotherhostwhenthebackupserverisreadytoperformbackuporrecovery
processesonthathost.
! MultiplestoragearraysZonesarealsousefulwhentherearemultiplestorage
arrays.Throughtheuseofseparatezones,eachstoragearrayismanaged
separatelyfromtheothers,withnoconcernforaccessconflictsbetweenservers.
LUN Masking
LUNmaskingiscommonlyusedforpermissionmanagement.LUNmaskingisalso
referredtoasselectivestoragepresentation,accesscontrol,andpartitioning,
dependingonthevendor.
LUNmaskingisperformedattheSPorserverlevel;itmakesaLUNinvisiblewhena
targetisscanned.Theadministratorconfiguresthediskarraysoeachserverorgroup
VMware, Inc. 9
SAN Conceptual and Design Basics
ofserverscanseeonlycertainLUNs.Maskingcapabilitiesforeachdiskarrayare
vendorspecific,asarethetoolsformanagingLUNmasking.
ESX ESX
Server Server
Mask LUN 11 Mask LUN 12
LUN masking:
zoning: done at SP or server level;
done at switch level; makes a LUN invisible
used to segment FC switch
when a target is scanned
the fabric
50:05:01:60:10:20:AD:87
SP WWN
(world-wide name)
LUNs unique, 64-bit
address assigned
0 11 12 to Fibre Channel node
storage array
Requirements
TointegrateallcomponentsoftheSAN,youmustmeetthevendorshardwareand
softwarecompatibilityrequirements,includingthefollowing:
! HBA(firmwareversion,driverversion,andpatchlist)
! Switch(firmware)
! Storage(firmware,hostpersonalityfirmware,andpatchlist)
VMware, Inc. 10
SAN Conceptual and Design Basics
SAN Setup
WhenyourereadytosetuptheSAN,completethesetasks.
1 Assembleandcabletogetherallhardwarecomponentsandinstallthe
correspondingsoftware.
a Checktheversions.
b SetuptheHBA.
c Setupthestoragearray.
2 Changeanyconfigurationsettingsthatmightberequired.
3 Testtheintegration.
Duringintegrationtesting,testalltheoperationalprocessesfortheSAN
environment.Theseincludenormalproductionprocessing,failuremodetesting,
backupfunctions,andsoforth.
4 EstablishabaselineofperformanceforeachcomponentandfortheentireSAN.
Eachbaselineprovidesameasurementmetricforfuturechangesandtuning.See
ESXServerSANConfigurationGuideforadditionalinformation.
5 DocumenttheSANinstallationandalloperationalprocedures.
ThissectiondiscussessomeSANdesignbasics.ItdoesnotfocusonSANdesignforESX
Serverhosts,butonthefollowinggeneraltopicsofinterest:
! DefiningApplicationNeedsonpage 11
! ConfiguringtheStorageArrayonpage 12
! ConsideringHighAvailabilityonpage 13
! PlanningforDisasterRecoveryonpage 13
AproperlydesignedSANmustprovidesufficientresourcestoprocessallI/Orequests
fromallapplications.DesigninganoptimalSANenvironmentisthereforeneither
VMware, Inc. 11
SAN Conceptual and Design Basics
simplenorquick.ThefirststepindesigninganoptimalSANistodefinethestorage
requirementsforeachapplicationintermsof:
! I/Operformance(I/Opersecond)
! Bandwidth(megabytespersecond)
! Capacity(numberofLUNsandcapacityofeachLUN)
! Redundancylevel(RAIDlevel)
! Responsetimes(averagetimeperI/O)
! Overallprocessingpriority
! EachRAIDgroupprovidesaspecificlevelofI/Operformance,capacity,and
redundancy.LUNsareassignedtoRAIDgroupsbasedontheserequirements.
! IfaparticularRAIDgroupcannotprovidetherequiredI/Operformance,capacity,
andresponsetimes,youmustdefineanadditionalRAIDgroupforthenextsetof
LUNs.YoumustprovidesufficientRAIDgroupresourcesforeachsetofLUNs.
! ThestoragearraysneedtodistributetheRAIDgroupsacrossallinternalchannels
andaccesspaths.ThisresultsinloadbalancingofallI/Orequeststomeet
performancerequirementsofI/Ooperationspersecondandresponsetimes.
Forexample,apeakperiodmayoccurduringnoontimeprocessing,characterizedby
severalpeakingI/Osessionsrequiringtwiceorevenfourtimestheaverageforthe
entirepeakperiod.Withoutadditionalresources,I/Odemandsthatexceedthecapacity
ofastoragearrayresultindelayedresponsetimes.
Caching
ThoughESXServersystemsbenefitfromwritecache,thecachecouldbesaturatedwith
sufficientlyintenseI/O.Saturationreducesthecacheseffectiveness.
Becausethecacheisoftenallocatedfromaglobalpool,itshouldbeallocatedonlyifit
willbeeffective.
! AreadaheadcachemaybeeffectiveforsequentialI/O,suchasduringcertain
typesofbackupactivities,andfortemplaterepositories.
! AreadcacheisoftenineffectivewhenappliedtoaVMFSbasedLUNbecause
multiplevirtualmachinesareaccessedconcurrently.Becausedataaccessis
random,thereadcachehitrateisoftentoolowtojustifyallocatingareadcache.
VMware, Inc. 12
SAN Conceptual and Design Basics
! Areadcacheisoftenunnecessarywhentheapplicationandoperatingsystem
cachedataarewithinthevirtualmachinesmemory.Inthatcase,thereadcache
cachesdataobjectsthattheapplicationoroperatingsystemalreadycache.
! RedundantSANComponentsRedundantSANhardwarecomponents
includingHBAs,SANswitches,andstoragearrayaccessports,arerequired.In
somecases,multiplestoragearraysarepartofafaulttolerantSANdesign.
! RedundantI/OPathsI/Opathsfromtheservertothestoragearraymustbe
redundantanddynamicallyswitchableintheeventofaport,device,cable,orpath
failure.
! I/OConfigurationThekeytoprovidingfaulttoleranceiswithinthe
configurationofeachserversI/Osystem.
WithmultipleHBAs,theI/OsystemcanissueI/OacrossalloftheHBAstothe
assignedLUNs.Failurescanhavethefollowingresults:
! IfanHBA,cable,orSANswitchportfails,thepathisnolongeravailableand
analternatepathisrequired.
! IfafailureoccursintheprimarypathbetweentheSANswitchandthestorage
array,thenanalternatepathatthatlevelisrequired.
! IfaSANswitchfails,theentirepathfromservertostoragearrayisdisabled,
soasecondfabricwithacompletealternatepathisrequired.
! MirroringProtectionagainstLUNfailureallowsapplicationstosurvivestorage
accessfaults.Mirroringcanaccomplishthatprotection.
MirroringdesignatesasecondnonaddressableLUNthatcapturesallwrite
operationstotheprimaryLUN.MirroringprovidesfaulttoleranceattheLUN
level.LUNmirroringcanbeimplementedattheserver,SANswitch,orstorage
arraylevel.
Note UsuallyRAIDSdontmirror.
! DuplicationofSANEnvironmentForextremelyhighavailability
requirements,SANenvironmentsmaybeduplicatedtoprovidedisasterrecovery
onapersitebasis.TheSANenvironmentmustbeduplicatedatdifferentphysical
locations.ThetworesultantSANenvironmentsmayshareoperationalworkloads
orthesecondSANenvironmentmaybeafailoveronlysite.
VMware, Inc. 13
SAN Conceptual and Design Basics
analternateservertostartthedatarecoveryprocess.TheSANmayhandlethesitedata
synchronization.
ESXServermakesdisasterrecoveryeasierbecauseyoudonothavetoreinstallan
operatingsystemonadifferentphysicalmachine.Justrestorethevirtualmachine
imageandcontinuewhatyouweredoing.
14