You are on page 1of 18

IT-Symposium 2007 17.04.

2007

Storage Systems Division

Storage Virtualization with
IBM SAN Volume Controller

Torsten Rothenwaldt
IBM European Storage Center

DECUS IT-Symposium 2007 | 1G04 © 2007 IBM C orporation

Storage Systems Division

Agenda

 Review different architectures of disk block virtualization.

 Explain implementation details of the IBM SAN Volume Controller
as example of in-band appliance.

 Discuss current enterprise usage of disk/file virtualization.

2 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation

www.hp-user-society.de 1

IT-Symposium 2007 17.04.de 2 .2007 Storage Systems Division Table of contents Disk block virtualization IBM SAN Volume Controller Storage virtualization in enterprises today 3 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Disk (Drive) Virtualization 0000 0001 0002 0003 …. nnnn  Physical data layout  Virtual data layout Cylinder/Head/Sector addresses Logical Block Addresses (LBA)  Media defects  “Defect-free” 4 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.hp-user-society.

2007 Storage Systems Division Block-Level Virtualization  Bound to a physical machine  Anywhere (can be moved or replaced)  Fixed size and (pseudo-) geometry  Size as needed (can grow.de 3 .IT-Symposium 2007 17.Management Performance Device-specific Stable & reliable “per-box” Perceived security 6 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www. shrink. or morph)  Limited performance  Performance scaling  Do break (occasionally)  Reliable as needed Where does block-level virtualization reside? 5 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Virtualization in the Storage Subsystem Provisioning Per-host Virtualization RAID subsystem +Heterogeneous hosts .Homogeneous storage +Mature industry & products .04.hp-user-society.

hp-user-society.2007 Storage Systems Division Virtualization in the Host Provisioning Per-host…still Virtualization Host RAID subsystem +Flexibility (any storage) .04. of volumes movement.…) . re-layout. snapshots.IT-Symposium 2007 17.May be host-invasive management Host-specific +Shorter I/O path .de 4 .Appliance availability +Light-weight compared to .Complexity shared data clusters 7 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division “Out-of-Band” Virtualization Provisioning Data center-wide Virtualization Host “SAN appliance” separate box Infrastructure In-host RAID subsystem +Data center-wide .Server-centric management +File system coupling of RAID subsystems (online growth.Secure protocols full volume manager 8 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.

hp-user-society. Standard) 10 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www. One pool.de 5 .Performance perceptions 9 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Virtualization Devices for In-Band  Server-based device (appliance)  Switch-based device (fabric application) Virtualizes a variety of physical storage using different HBAs.2007 Storage Systems Division Virtualization in the Network: “In-Band” Provisioning Data center-wide SAN Virtualization Host access Device access Host Infrastructure RAID subsystem +Data center-wide .Complexity management User (un)familiarity Heterogeneous storage Integration needs Heterogeneous hosts Needs clustering .IT-Symposium 2007 17.04. Network optimized Implements complex storage solutions High port counts inexpensively Expensive Adds another layer (managed Elaborated functions not available separately) Fabric integration critical Less interoperability issues (no fabric integration) Standardization just beginning (ANSI T11 Fabric Application Interface Appears as standard device.

F. 42 (2003) Nr. 2. IBM Systems Journal.2007 Storage Systems Division Table of contents Disk block virtualization IBM SAN Volume Controller Storage virtualization in enterprises today 11 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Example of a Block-Level In-Band Appliance: IBM SAN Volume Controller  J. Glider.ibm. S.research. pp.ibm.IT-Symposium 2007 17. W.com/journal/ )  IBM Redbooks: SG24-6423 IBM System Storage SAN Volume Controller SG24-7275 Implementing the SVC in an OEM Environment SG24-7371 Using the SVC for Business Continuity ( http://www. 232-249 ( http://www. J. C.04.redbooks.de 6 . Fuente.hp-user-society. Scales: The software architecture of a SAN storage control system. Vol.com/ ) 12 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.

04.IT-Symposium 2007 17.de 7 .hp-user-society.2007 Storage Systems Division IBM SVC: Architecture Storage Storage Netw Network ork Virtual LUNs Virt ual LUNs Virtu al LUNs Virt ual LUNs N ode Node Node N ode Node Node Node N ode Node Node N ode Node M anaged Disks 13 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division IBM SVC: Zoning Host Zone: -Hosts zoned only to SVC -See only Virtual Disks that SVC allows to see Device Zone: -Devices zoned only to SV -See only SVC nodes as connected hosts 14 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.

. 512-MB extents = 2 PB Cannot migrate vdisks between MDGs with different extent sizes Can migrate extents from mdisk to mdisk R1 Pool Ex te nt 1 R1 Ex te nt 2 R1 R1 Ex te nt 3 R5 Pool Ex te nt 4 R10 Pool Ex te nt 5 R5 R5 R5 .IT-Symposium 2007 17. default 16 MB. R10 R10 R10 Ex te nt n 16 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.04.de 8 . the user assigns them to one or more pools called Managed Disk Groups  These MDGs are addressed by the SVC in terms of extents: Extent size is determined at MDG creation time. or RAID-1 recommended  Normally LUNs "surfaced" from storage systems are what the hosts on the SAN see as physical disks  Disks surfaced by the storage systems are discovered by SVC and referred to as Managed Disks (mdisks)  Spare capacity on mdisks can be reallocated transparently and dynamically R5 R 10 R5 R1 R5 R1 MD MD MD MD MD MD SA NVC R5 R5 R1 R5 R1 R 10 VLUN VLUN VLUN VLUN VLUN VLUN R AID C ontrolle r R AID C ontrolle r 15 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division IBM SVC: Managed Disk Groups  Once the mdisks are available to SVC.2007 Storage Systems Division IBM SVC: Managed Disks  SVC does not perform RAID functions  Utilizes RAID capability of backend storage server  RAID-5.hp-user-society. max 512 MB Extent size determines maximum amount of storage SVC can manage 16-MB extents = 64 TB.. RAID-10.

de 9 . the user can build "virtual disks"  Various policies can be used to build them: Striped — taking an extent in turn from each disk in the pool or a subset of the disks in the pool (Virtual Disk striped across multiple disks) Sequential — using a single disk in the pool (Virtual Disk mapped sequentially to managed disk) Image Mode — Virtual Disk = Physical LUN SVC Cache — Enabled or Disabled  Real physical capacity must be available to create a vdisk  Virtual disks can be expanded.04.512 MB 17 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Cache Disabled Virtual Disks  Can be used with Image Mode VDisks only to enable use of copy services of underlying disk subsystem  Can be used with striped mode VDisks May be useful to preserve cache resources Example: Use for TSM/HSM disk pools Example: Use for test/dev VDisks in mixed production environment  Cache Disabled Virtual Disks are basically normal VDisks and can utilize SVC capabilities like migration and copy services Can use SVC copy services on the source VDisk Cannot use SVC copy services on the target VDisk  Provide a way for parallel direct access to the underlying disk subsystem 18 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.IT-Symposium 2007 17.hp-user-society. reduced. or deleted  IO governing can be enabled to limit IO/s or MB/s Extent 1a Extent 2a Extent 1a Extent 2a Extent 3a Extent 3a Extent 1b Extent 2b Extent 3b Create a Extent 1b Extent 1c Extent 2c Extent 3c striped Extent 2b Extent 1d Extent 2d Extent 3d Extent 1e Extent 2e Extent 3e virtual disk Extent 3b Extent 1c Extent 1f Extent 2f Extent 3f Extent 2c Extent 1g Extent 2g Extent 3g Extent 3c A host vdisk is a collection of Extents — each 16 MB .2007 Storage Systems Division IBM SVC: Virtual Disks  From these extents.

IT-Symposium 2007 17.2007 Storage Systems Division IBM SVC: IO Groups and Hosts  Each virtual disk is assigned to a particular IO Group (node pair).  It is these virtual disks that SVC presents to hosts on the SAN as targets of IO. Managed Disk mdiskgrp0 [FA StT Group] . The hosts see these as physical disks (in terms of the OS). IO WWPN1 Group WWPN2 VDisk Host WWPN3 WWPN4 19 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division SVC Combined Physical & Logical View Virtual Disks Mapped to Hosts Virtual vdisk0 vdisk1 vdisk2 vdisk3 vdisk4 Disk 20GB 20GB 20GB 20GB 20GB Virtual Disks are associated with particular I/O Groups SVC I/O Group 1 I/O Group 2 Cluster Managed Disk Groups are accessible by all I/O Groups in the Cluster. Can be mapped to multiple hosts for use with clustering software.40GB mdiskgrp1 [ESS Group] . SVC knows hosts as groups of HBA WWPNs. Every node in the cluster is aware of the virtual disk.de 10 .hp-user-society.60GB Groups Managed mdisk0 mdisk1 mdisk2 mdisk3 mdisk4 mdisk5 mdisk6 Disk 10GB 10GB 10GB 10GB 20GB 20GB 20GB FA StT FA StT FA StT FA StT ESS ESS ESS LUN 10GB 10GB 10GB 10GB 20GB 20GB 20GB 20 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www. The virtual disks are mapped to hosts (SDD for mutli-path operation).04. All IO is targeted at either of the nodes in the IO Group for purposes of caching and load balancing. but only owning IO Group services requests.

failover/failback is function of SVC code One node automatically designated config/boss node for cluster  Assigned cluster IP address  Responsible for coordination of node transitions  Automatic failover of config node and IP address Auto-restart of a node on failure and re-admission to cluster Cluster requires majority of nodes remain operating to ensure quorum Two nodes is a special case  Quorum disk is elected as a tie-breaker  Quorum disks are existing mdisks under SVC and use 1 extent A node stores a write in its own write cache and the write cache of its partner node before acknowledging to the host application On node failure.IT-Symposium 2007 17.de 11 .04. surviving node empties write cache and proceeds in write-through mode 21 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division IBM SVC: Copy Services VD 1 VD 2 3 4 5 VD 6 VD 7 VD 1 VD 2 3 4 5 VD 6 VD 7 MD 1 VD 2 VD 3 VD 4 MD 5 MD 6 MD 7 MD 8 MD 1 VD 2 VD 3 VD 4 MD 5 MD 6 MD 7 MD 8 IO Group IO Group MD MD MD MD MD MD LUN 1 LUN 2 LUN 3 LUN 4 LUN 1 LUN 2 LUN 3 LUN 4 LUN 1 LUN 2 LUN 3 LUN 4 LUN 1 LUN 2 LUN 3 LUN 4 RAID RAID RAID RAID controller 1 controller 2 controller 3 controller 4 SANMigration SAN Data Peer-to-Peer Remote SANCopy FlashCopy “outside the(PPRC) box” “outside the “outside box”the box” (Metro / Global Mirror) 22 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.2007 Storage Systems Division IBM SVC: Clustering Cluster comprised of 2…8 nodes but administered as single image No Linux clustering software.hp-user-society.

de 12 .IT-Symposium 2007 17.04.2007 Storage Systems Division IBM SVC Global Mirror  Long distance asynchronous remote mirroring function  Practically unlimited distances for business continuity  Does not wait for secondary I/O before completing host I/O Minimizes performance impact to applications  Designed to maintain consistent secondary copy at all times Once initial copy has completed  Built on Metro Mirror code base  Metro and Global Mirror delivered as single feature Offers great implementation flexibility No additional license charge for existing MM users 23 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Global Mirror: Overview  Single remote copy partnership per cluster  Supports consistency groups that span I/O groups within cluster  Consistency group consists of MM or GM relationships not both Cluster 1 Partnership Cluster 2 Vdisk Vdisk Relationship A C Consist- ency Group Vdisk Vdisk Relationship B D 24 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.hp-user-society.

2007 Storage Systems Division Global Mirror: Algorithm  I/O complete acknowledged to host at primary once following occurs: Data is committed to cache in both nodes in I/O group Log of I/O and its sequence number committed on both nodes of I/O group  Remote I/O is transmitted concurrently with processing of local I/O Local host I/O normally completes before remote I/O completes  Extra messages passed between nodes in cluster Used to detect concurrent host I/O among VDisks in same consistency group  Concurrent I/O is assigned a shared “sequence number” During high loads many I/Os will share sequence number to minimize overhead  Remote I/Os are applied at secondary in “sequence number” order Writes within a “sequence number” are non-dependant and written in any order once received at secondary Many sequence number “batches” can be outstanding on the long distance link At secondary site a single sequence number is applied at a time 25 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Global Mirror: Link Tolerance  Existing redundancy used to continue operation if node or link fails: Recovery begins by re-driving writes that were in flight.hp-user-society.de 13 .04.IT-Symposium 2007 17. Log of I/O established during initial host write used to support recovery. Recovery uses read to primary VDisk to capture data for replay of write  Autonomic Link Tolerance feature to prevent performance problems leading to application outage: Monitors write latency impact of GM and considers stopping a GM relationship or consistency group if impact exceeds a threshold Link tolerance threshold allows ‘bursts’ of work to be tolerated Defines period of time poor performance can be tolerated before GM stops Settable from 1 minute to 1 day in 10 second increments (default 5 minutes) Disabled by setting to ‘0’ 26 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.

operating systems.hp-user-society. … Too complex policies Too large migration projects  Availability requirements  Storage resource utilization  Storage performance 28 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www. storage systems.2007 Storage Systems Division Table of contents Disk block virtualization IBM SAN Volume Controller Storage virtualization in enterprises today 27 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Why To Use Storage Virtualization  The management nightmare Too many servers.IT-Symposium 2007 17.04.de 14 . management consoles.

storageperformance. NSC55 DS4800 9910/9960 9520V 8000 16000 280.hp-user-society. 3050 240. 280 DS8000 9970V 95xxV 4000/6000 12000 512.2007 Storage Systems Division SVC Version 4.. 380 29 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Performance: SVC SPC-1 Benchmark http://www. McData iSCSI to hosts Via Cisco IPS SAN with 4Gbps fabric SAN Continuous Copy Point-in-time Copy Full volume Metro Mirror Copy on write Global Mirror SAN SAN Volume Controller Volume Controller IBM IBM IBM Hitachi Hitachi Hitachi HP HP HP EMC EMC/Dell Sun NetApp STK ESS. 128 8000 FC4700 9910/9960 3020.. 1024 DMX CX2/3/4/5/6/700 9970/9980 FLX210. VSS. DS N series Lightning Thunder Tagma EVA MA/EMA XP Symm CLARiiON FAS D173.1 Supported Environments HP/UX 11iV2 Linux TRU64 (Intel/Power/zLinux) Microsoft IBM Sun Novell VMWare OpenVMS RHEL/SUSE IBM MSCS AIX Solaris 10 ServiceGuard W / LVM NetWare Win / NW MPIO. 240.04. System p Win/Linux/VMWare/AIX Hosts OPM/FCS/IBS Cisco .org/results/a00043-r1 _IBM_SPC1_executive-summary. GDS BladeCenter 1024 HACMP/XD VCS Clustering Clustering RHEL AS 4 Clustering guests x64 GPFS / VIO SUN Cluster Ia32.IT-Symposium 2007 17.pdf 30 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.de 15 . FAStT DS4K / 6K 5200. 178. 5500 9980V 9200 Store 3000/5000 8000 48. x64. 220. USP.

517 GB/s) + DS8300 (3.217 GB/s) lead the industry in this new benchmark 31 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Enterprise Usage Today  Widely accepted by customers: Tape drive and library virtualization Disk block level virtualization in LVM and storage subsystem In-band file system virtualization (NAS) Aggregated networks (iSCSI.hp-user-society. long distance connections)  Beginning of the lifecycle: Disk block fabric-layer virtualization (in-band and out-of-band)  Niche markets (this may change): Virtual SANs File and record virtualization 32 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.de 16 .IT-Symposium 2007 17.2007 Storage Systems Division Performance: SPC-2 Benchmark  Newest addition to the Storage Performance Council Benchmarks  Composite of three workloads to measure sequential performance Large file processing – scientific and large-scale financial processing Large database queries – data mining and business intelligence Video on demand – streaming movies to end users  SVC posted the highest results in this industry-recognized storage performance test SVC (3.04.

 Market situation Religious wars about architectures. IBM Custo mer Agree me nt. 33 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation Storage Systems Division Disclaimer Information Copyright © 2007 by Inte rnatio na l B us iness M achines Corporation. IBM EXP RESSLY DISCLAIMS AN Y WAR RANTIES O F MERC HAN TABILIT Y.) unde r whic h they are provided.IT-Symposium 2007 17. Too many different solutions. progra ms. Statement of Limited Warra nty. and does not.04. All stateme nts regarding IBM 's future dire ctio n and intent are subje ct to cha nge or withdra wa l without notice.de 17 . gra nt any right or lice nse under any IBM pate nts or copyrights.S. in writing. and represent goals and objective s only.  Complexity Lack of standards. Products difficult to understand and evaluate. THE INFORMATION PRO VIDE D IN THIS DO CUM EN T IS DISTRIBUT ED "AS IS" WITHOUT ANY WAR RAN TY. FITNESS FO R A PARTICU LAR PURPOS E OR IN FRIN GEM EN T. IBM shall ha ve no responsib ility to upda te this info rmatio n. T he provisio n o f the informatio n contained herein is no t intended to. Refere nce s in this document to curre nt or future IBM produc ts.2007 Storage Systems Division Why Do Customers Hesitate?  Reliability SAN and storage failures may affect the whole datacenter. NY 10504-1785 U. Inquiries regarding patent or copyright licenses sho uld be made.A.g. Security implications not clear. programs or services availab le in a ll countries in whic h IBM operates or does bus iness. EIT HER EXPR ESS O R IMPLIED. etc.hp-user-society. SAN and storage failures may destroy data. IBM products are warra nted accord ing to the terms and cond itions of the agree me nts (e. to : IBM D irector of L ice nsing IBM C orporatio n North Castle Drive Armonk. Interna tional Progra m License Agreement. No part o f this document ma y be reprod uced or trans mitted in any form w ithout writte n permissio n from IBM Corporation. 34 Storage Virtualization with IBM SA N Volume Controller | 1G04 © 2007 IBM C orporation www.. or services does not imp ly tha t IBM intend s to make suc h s uch products. Multi-vendor support.

hp-user-society.2007 Storage Systems Division Closing slide Thank you DECUS IT-Symposium 2007 | 1G04 © 2007 IBM C orporation www.IT-Symposium 2007 17.de 18 .04.