Advanced SAN Design - Virtualization Technologies and Intelligent Applications Design Considerations

Advanced SAN Design—
Virtualization
Technologies and
Intelligent Applications
Design Considerations
BRKSAN-3707
BRKSAN-3707
14694_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
Agenda
Brief Review of Virtual Fabrics
Virtual Fabrics (VSANs)
Port-Channels, Trunking and IVR
Virtualization Technologies
SAN Device Virtualization (SDV)
N-Port ID Virtualization (NPIV)
N-Port Virtualizer (NPV)
FlexAttach
Intelligent Application
Data Mobility Manager (DMM)
Storage Media Encryption (SME)
SANTap
Storage Virtualization
BRKSAN-3707
Virtual Fabrics
BRKSAN-3707

Presentation_ID.scr
Virtual Fabric: Three Key Concepts
Virtual Fabric
Provide independent (‘virtual’) fabric services on a single
physical switch
Virtual Fabric Trunking and Port-Channels

Ability to transport multiple virtual fabrics over a single ISL or
common group of ISLs
Fabric Routing (IVR)

Ability to provide selected connectivity between virtual fabrics
without merging them
Trunk BRCD = Port Channel Cisco

Group of ISLs = Port Channel
BRKSAN-3707
Cisco’s Approach to Virtual Fabric:

Virtual SAN (VSANs)
A VSAN provides a method to
allocate ports within a physical
fabric to create virtual fabrics
Virtual fabrics created from larger
cost-effective physical fabric
Reduces wasted ports with islands Cisco MDS 9000 Physical SAN
Family with Islands Are
Fabric events are isolated per VSAN Service Virtualized onto
VSAN—maintains HA Common SAN
Infrastructure
Hardware-based isolation—traffic
is explicitly tagged across ISLs
with VSAN membership info
Statistics gathered per VSAN
BRKSAN-3707

Presentation_ID.scr
VSAN—MDS Family
Each port on the MDS
family exists in a VSAN
VSAN
Up to 256 VSANs in a single ‘A’
switch (hardware can
support up to 4095)
VSAN
VSAN
‘B’
Logical configuration to ‘B’
move a port from one

fabric to another VSAN
‘C’
WWN-based VSANs can
provide automated VSAN
VSAN
membership ‘D’
Basis for Virtual Fabric

Trunking (VFT) Extended
Header (ANSI T11 FC-FS-2
section 10)
BRKSAN-3707
VSAN Numbering Rules

Configured
VSANs
VSAN 1 is the default VSAN VSAN 10
All ports are originally in VSAN1 VSAN 20
VSAN 2 through 4093 can be VSAN 30
assigned to ‘user’ VSANs—VSAN Trunking E_Port

0, 4094, 4095 are reserved (TE_Port)
Currently 256 VSANs is supported from Enhanced ISL VSAN 30 Is Not

the range of 2–4093 (EISL) Trunk Propagated Across EISL
Carries Tagged Due to Nonexistence on
Remote Switch
VSAN 4094 is a reserved as Traffic from
Multiple VSANs
‘special’ VSAN Trunking E_Port
(TE_Port)
Called the ‘isolated VSAN’ Port Is In VSAN 4094
(Isolated VSAN) VSAN 10
Used to isolate ports whose port-VSAN
VSAN 20
has been deleted
Host Is Isolated VSAN 30
Not propagated across switches From the Fabric
Configured
Always present, can’t be deleted VSANS
Always in suspended state
BRKSAN-3707

Presentation_ID.scr
Standard Fibre Channel Frame Fields
4B SOF (Start of Frame)
ANSI T11.3 task group is the standard
8B VSAN Header
committee working on Virtual Fabrics
T11.3 FC-FS-2 fabric services includes
virtual fabrics specification 24B FC Header
Defines “Extended-Headers”
In FC-FS-2 Section 10.2
Defines frame tagging mechanism
Applicable to N_Ports, F_Ports and

E_Ports 0 ->
2112B Payload
Enables Inter-Switch Link to support trunking
virtual interfaces
Define the trunking virtual interfaces for end
devices (hosts, storages)
The ANSI T11 FS-SP group has

4B CRC
accepted Cisco VSAN as standard
(FC-FS-2 Section 10) 4B EOF (End of Frame)
BRKSAN-3707
VSAN Header Field

Frame MPLS More User VSAN CDL # PAD Msg
R_CTL Ver TTL P_VL Rsvd OAM
Type Present Header Priority Number Present Bytes Info
8 2 4 1 1 3 12 bits 1 8 2 4 2 8 8
Each frame on a VSAN trunk carries an extra 8 bytes of header:

User priority—3 bits—used for QoS functionality to designate
priority of frame
VSAN ID—12 bits—used to mark the frame as part of a particular
VSAN—supports up to 4096 VSANs
MPLS flag—1 bit—used to designate whether this frame is subject
to Multi-Protocol Label Switching processing—future use
Time-to-live (TTL)—8 bits—used to help avoid routing loops—
standard part of an IP frame
Other misc. fields including version, frame type, and other
reserved fields
BRKSAN-3707

Presentation_ID.scr
Trunking and
Port-Channels
BRKSAN-3707
EISLs and TE-Port

1. The Trunking E_Port (TE_Port)
Negotiated between MDS switches—default Cisco MDS 9513
Carries tagged frames from multiple VSANs Director with
VSAN Service
Can be optionally disabled to yield E_Port
Only understood by MDS switches
Also has a native VSAN assignment Trunking
(for E_Port) E_Port
(TE_Port)
Trunk all VSANs (1-4093) by default Enhanced ISL
Not to be confused with Brocade ISL (EISL) Trunk
aggregation (trunking) Carries
Tagged Traffic
Trunking
2. The EISL link from Multiple
E_Port Cisco MDS
VSANs
The resultant link created by two (TE_Port) 9216
connected TE_Ports Trunking Fabric with
E_Port VSAN Service
Superset of ISL functionality (TE_Port)
Carry individual control protocol information
per VSAN (e.g. zoning updates)
Can be extended over distance (DWDM, Enhanced
ISL (EISL)
FCIP, etc.) Trunk Carries
Tagged Traffic
Notice: Blue VSAN Doesn’t Have to Reside From Multiple
on Switch for it to Traverse Switch VSANs
BRKSAN-3707

Presentation_ID.scr
VSAN—EISL Establishment
(Negotiation Protocol)
Two interconnected switch ports conduct an ELP (Exchange ELP
Link Parameters) exchange—forms two E_Ports and ISL links Exchange
(Standard-based negotiation) E E
Two switches then conduct an ESC (Exchange Switch ESC

Exchange
Capabilities) exchange—determines whether Cisco switch on Two Cisco Switches
other end or not capable of EISL E E
(Standard-based negotiation) EPP
Exchange
If yes—then proceed to negotiate EISL/ISL
E E
If Cisco switches, two switches then conduct an EPP
Normal ISL
(Exchange Peer Parameters: Cisco prop protocol)
exchange—determines whether to stay as ISL, move to Or
TE TE
EISL (VSAN-enabled), or isolate in case of mismatched
port VSANs EISL
Formed
These modes are negotiated based on the configuration of Or
the switches and the parameters of the ports; isolation can
occur if VSANs are mismatched Isolated
E E
Done
* Provided ELP Parameters Match Such as Timers
and Switches in Interoperability Mode if Required
BRKSAN-3707
Port Channels
Port Aggregation Feature Used to Create a
Single Logical ISL from 1–16 Physical ISLs
Increases bandwidth and
availability
Very granular load balancing -
per exchange/src/dst or per
src/dst (policy on a per VSAN
basis)
Interfaces can both be added
and removed in a non-
disruptive manner in
production environments
4 Link Port Channel
Preserved FC guarantee of in-
EISL
order delivery (IOD)
BRKSAN-3707

Presentation_ID.scr
Port Channel Protocol (PCP)
Exchange-based in-order load
balancing
Mode 1: based on src/dst
FC_ID/OX_ID/RX_ID
Mode 2: based on src/dst FC_ID
Consistently detect mis-

Up to 160 Gbps
configuration Port Channel
with HA
Transition mis-configured ports to
isolated state so as to be able to
correct the misconfiguration
Synchronize bring up of ports in a
channel across peer switches
Provide the ability for the system
to automatically create port
channels among compatible ports
BRKSAN-3707
VSANs, EISLs, TE_ports, Port Channels—How

All These Work Together
VSAN METRIC
Hierarchical relationship— 10
20
100
50
VSAN Metric
10 50
20 100
Port Channels provide link aggregation to
yield virtual ISL (E_Port)
Single-link ISL or Port Channel ISL can be
configured to become EISL—(TE_Port)
VSANs can be selective grafted or pruned
from EISL trunks
All member links of a Port Channel 8 Gbps

p
PortChannel
must have same configuration prior 20 cku
VS On
a
AN B E_Port
AN ly
Trunking S
V 10
to creating channel (e.g., TE_Port or
10
E_Port AN
E_Port, VSANs enabled, etc.) (TE_Port) VS
Port Channel technology provides E_Port

Trunking
high availability and fast recovery for E_Port
VSAN trunk (EISL) (TE_Port)
Multiple Port Channels yield multiple 4 Link (8 Gbps)

PortChannel
paths for custom traffic engineering Configured as EISL
BRKSAN-3707

Presentation_ID.scr
VSANs and Non-Cisco Switches
The VSANs feature involves a frame
tagging mechanism which is not
understood by 3rd party fabrics
MDS Family switches support
heterogeneous switch
interoperability—non-VSAN aware
EISL Trunks
Cisco “Interoperability Mode” is Carrying
Simple ISL
Links
configured per-VSAN—no loss of Numerous
VSANs E_Ports
functionality in MDS 9000 switches
MDS switches negotiate a standard Non-Cisco
Fabric
E_Port with non-Cisco switches Switches
MDS 9000 E_Ports also have a
port VSAN
Therefore, the entire non-Cisco switch,
including all its ports, will reside in the Each Non-Cisco Switch
port VSAN of the connecting E_Port Belongs to Only One VSAN
BRKSAN-3707
Fabric Routing
BRKSAN-3707

Presentation_ID.scr
Fabric Routing: Cisco Inter-VSAN
Routing
We use fabric as an extension of virtual Physical Physical
SAN SAN
fabrics to enable cross-fabric connectivity
Physical
Done without merging the routed fabrics Physical Islands
SAN
Without propagation of irrelevant
fabric events
Without concern for overlapping domain IDs
VSAN
Without concern for fabric interoperability VSAN
differences VSAN
Without fabric services interference across Virtual

Fabric
multiple fabrics
Enable end devices from different virtual

fabrics to access one another VSAN
VSAN
VSAN
Routed
Virtual
Fabric
BRKSAN-3707
Fabric Routing Applications—

Sharing Common Resource
Overlay data replication fabrics

on common physical fabric Common Physical Fabric
No need for separate pair of

switches for each replication MS Sales
connection Marketing
SAN MS
SAN
Use one virtual fabric per
replication connection TAPE
MS SAN
HR
Being able to share common SAN
SAN Extension circuits

amongst multiple virtual fabrics
Fabric routing adds resiliency MS
Tape Media Server
to the solution
BRKSAN-3707

Presentation_ID.scr
Fabric Routing Applications—
SAN Extension Solutions
Minimize the impact of change in fabric services across geographically
distributed sites
Limit fabric control traffic such as RSCNs and Build/Reconfigure Fabric
(BF/RCF) to local VSANs
Augments the high availability of the solution
Filters unnecessary events, Isolates from remote faults, Enables selective visibility
Works with any transport service (FC, SONET/SDH, DWDM/CWDM, FCIP)
Inter-VSAN Connection Between Completely Isolated Fabrics
IVR Isolation Minimizes PortChannel Protects
Impact if Transit VSAN Lost Against Loss of
Member Links/Paths
Replication Replication
VSAN_1 EISL#1 in VSAN_4
Port Channel
IVR EISL#2 in
IVR
Transit
VSAN_3 (IVR) Port Channel
Local IP WAN Local

VSAN_2 VSAN_5
BRKSAN-3707
Virtualization
Technologies
BRKSAN-3707

Presentation_ID.scr
SAN Device Virtualization (SDV)
Allows provisioning with Server Storage Arrays
virtualized servers and storage X Y

devices
Physical to Virtual Mapping
Significantly reduces time to
replace HBAs and Storage Virtual
Initiator
Virtual
Target
devices
No reconfiguration of zoning,
VSANs, etc. required on MDS
Presents virtual WWN to
No need to reconfigure storage array servers and storage device
LUN masking after replacing HBAs
Eliminates re-building driver files on
AIX and HP-UX after replacing
storage
BRKSAN-3707
NPIV (N_Port Identifier Virtualization)

Designed for virtual server environments — Linux on zSeries, VMware
Assigning multiple port IDs to a single N_Port
Multiple applications on the same port can use different IDs in the same VSAN
Zoning and port security can be implemented at the application level
Data Center Session on Server Virtualization: DCT-2868
33 Name
Name Server
Server entries
entries
Virtual Servers 33 Virtual
Virtual Devices
Devices
Email All
All share
share 11 FC
FC Port
Port but
but
Web maintain
maintain individual
individual identity
identity
Print
3 Logins
LUN 1 N_Port ID=1
FC F_Port
LUN 2 N_Port ID=2
LUN 3 N_Port ID=3

3 FCIDs
N_Port Controller
HBA
MDS 9000
BRKSAN-3707

Presentation_ID.scr
NPIV FLOGI/FDISC Login Process
NPIV Enabled Switch
When host physical port
comes up, it first does a
FLOGI and PLOGI into the
switch to register into the FC
Name Server F
NPIV capable devices

(typically HBAs) will continue P1 NPIV Capable HBA
login process using FDISC
(Fabric Discovery) to register
vP1
virtual PWWN into the FC vP2
Name Server using the same vP3
physical interface
BRKSAN-3707
Blade Switch Explosion Issues

Scalability
Each Blade Switch uses a single
Domain ID
Theoretical maximum number of
Domain IDs is 239 per VSAN
Supported number of domains is
quite smaller (depends on OSM)
EMC: 40 domains
Cisco Tested: 75
HP: 40 domains
Other OSM Do Not Post
Manageability
More switches to manage
Shared management of blade
switches between storage and
BRKSAN-3707
server administrators

Presentation_ID.scr
Cisco MDS’ N-Port Virtualizer (NPV)
MDS NPV
NPV enables the switch to act as a NPIV host
NPV mode is no longer a “switch”
Changing from switching mode to NPV mode is disruptive
Upgrading SAN OS code is non-disruptive
NPV switch uplink is no longer an ISL (NP-port)
NPV switch DOES NOT use a Domain ID
No longer limited to Domain ID boundaries
Manageability
Less amount of switches to manage
NPV enable switch is now managed like a NPIV enabled host
Eliminates the need for server administrators to manage the SAN
BRKSAN-3707
Differences Between NPIV and NPV

NPIV (N-Port ID Virtualization) NPV (N-Port Virtualizer)
Functionality geared towards Functionality geared towards
server’s host bus adapters MDS fabric switches (MDS
(HBA) 9124, MDS 9134, Nexus 5000
and blade switches)
NPIV provides a means to
assign multiple Server Logins NPV provides the FC switch’s
to a single physical interface connections (uplink) to act as
server connections – instead
The use of different virtual
of acting like a standard ISL
pWWN allows access control
(zoning) and port security to be Utilizes NPIV type functionality
implemented at the application to allow multiple server logins
level from other switch ports to use
NP-port uplink
Usage applies to applications
such as VMWare, MS Virtual
Server and Linux Xen
BRKSAN-3707

Presentation_ID.scr
NPV FLOGI/FDISC Login Process NPV Core Switch
When NP port comes up on a NPV edge switch, it first

FLOGI and PLOGI into the core to register into the FC
Name Server
End Devices connected on NPV edge switch does FLOGI
F F
but NPV switch converts FLOGI to FDISC command,
creating a virtual PWWN for the end device and allowing to
login using the physical NP port. NP P1 NP P2
All I/O of end device will always flow through same NP port
NPV Edge Switch
F F
P4 = vP2 P5 = vP3
BRKSAN-3707
Nested NPIV FLOGI/FDISC

Login Process
NPV-Core Switch
When NP port comes up on a NPV edge
switch, it first FLOGI and PLOGI into the
core to register into the FC Name Server
End Devices connected on NPV edge
F F
switch does FLOGI but NPV switch
converts FLOGI to FDISC command,
creating a virtual PWWN for the end NP P1 NP P2
device and allowing to login using the NPV Edge Switch

physical NP port.
F F
NPIV capable devices connected on NPV
switch will continue FDISC login process P3 = vP1 P4 = vP5
for all virtual PWWN which will go through
same NP port as physical end device
vP2 vP6
vP3 vP7
vP4 vP8
BRKSAN-3707

Presentation_ID.scr
NPV Supported Switches
NPV Edge Switches
MDS 9124, MDS 9134 and
NX5K
IBM and HP Blade Switches
NPV Core Switches
MDS 9500 Family of Directors
MDS 9216A, MDS 9216i and
MDS 9222i
3rd Party Switches
Needs to support NPIV
Needs Testing/Qualification
BRKSAN-3707
MDS 9124—NPV Architecture

NPV Architecture
Total of 6 Port-Groups – every 4 ports
By default, first port in each Port-Group (ports 1, 5, 9, 13, 17 and
21) is set to “NP” mode for uplink to NPV Core Switch (Can be
changed)
All other ports are set to “F” for device connectivity (DOES NOT
SUPPORT FL-Ports)
Port-Group1: Ports 1 – 4
Port-Group6: Ports 21 - 24
BRKSAN-3707

Presentation_ID.scr
MDS 9134—NPV Architecture
NPV Architecture
Total of 10 Port-Groups
Port-Group consists of 4 ports for 1/2/4Gig ports grouping
Each 10Gig port is its own Port-Group
By default, first port in each Port-Group (ports 1, 5, 9, 13, 17, 21, 25 and
29) is set to “NP” mode for uplink to NPV Core Switch (Can be changed)
Both 10Gig port is set to “NP” mode
All other ports are set to “F” for device connectivity (DOES NOT
SUPPORT FL-Ports)
Port-Group1: Ports 1 – 4 Port-Group2: Ports 5 – 8

Port-Group7: Ports 25-28 Port-Group8: Ports 29-32
Port-Group9: Port 1 (10G) Port-Group10: Port 2 (10G)
BRKSAN-3707
Port Mapping for HP Blade Switches

HP Blade Switch Port-Group Mapping
External Links (All links set to
NP-port only)
PG 4
PG 2
PG 1
PG 3
PG 3
PG 6
PG 4
PG 5
PG 1 -> EXT Port 1

PG 2 -> EXT Port 2
PG 3 -> EXT Port 3 and EXT Port 4
EXT 3
EXT 2
EXT 1
EXT 4
EXT 6
EXT 8
EXT 5
EXT 7
PG 4 -> EXT Port 5 and EXT Port 6

PG 5 -> EXT Port 7
PG 6 -> EXT Port 8
Internal Links (All links set to F-

port only)
PG 6
PG 5
PG 2
PG 3
PG 2
PG 1
PG 1
PG 4
PG 4
PG 3
PG 1
PG 2
PG 6
PG 5
PG 5
PG 6
PG 1 -> Bays 3,4 and 11

PG 2 -> Bays 1,2 and 12
PG 3 -> Bays 9 and 10
Bay 11
Bay 10
Bay 12
Bay 14
Bay 16
Bay 13
Bay 15
Bay 3
Bay 2
Bay 6
Bay 1
Bay 4
Bay 8
Bay 5
Bay 9
Bay 7

PG 5 -> Bays 7, 14 and 15
PG 6 -> Bays 6, 7 and 13
BRKSAN-3707

Presentation_ID.scr
Port Mapping for IBM Blade Switches
IBM Blade Switch Port-Group Mapping
External Links (All links set to NP-
PG 1
PG 1
PG 2
PG 3
PG 5
PG 4
port only)
PG 1 -> Port 0 and Port 15
PG 2 -> Port 16
Port 16
Port 15
Port 17
Port 19
Port 18
Port 0
PG 3 -> Port 17
PG 4 -> Port 18
PG 5 -> Port 19
Internal Links (All links set to F-
PG 2
PG 4
PG 1
PG 1
PG 2
PG 3
PG 3
PG 5
PG 5
PG 4
PG 3
PG 2
PG 5
PG 4
port only)
PG 2 -> Bays 2, 4 and 7
Bay 10
Bay 11
Bay 13
Bay 12
Bay 14
Bay 3
Bay 2
Bay 9
Bay 1
Bay 4
Bay 6
Bay 8
Bay 5
Bay 7
PG 3 -> Bays 5, 6 and 8
PG 4 -> Bays 9, 13 and 14
PG 5 -> Bays 10, 11 and 12
BRKSAN-3707
Number of NPIV Logins: MDS 9200/9500

Type of Logins Number of Logins
Logins per Port 126
Logins per Line Card 400
Logins per Switch 2,000
Logins per physical fabric 10,000
These are the number of logins allowed on all Gen1 and Gen2 line cards.
The limits applied to on a per switch will also apply to all MDS 9200 and
MDS 9500. MDS 9124/9134 and Blade switches will have different limits
and will be shown later.
BRKSAN-3707

Presentation_ID.scr
Number of NPIV Logins:
MDS 9124/9134 and Blade Switches
Switching Mode NPV Mode
Logins per Port 42 114
Logins per Port-Group 168 114
Logins per MDS 9124 1,008 684
Logins per MDS 9134 1,680 1,140
Logins per MDS 9124e 1,008 684
Logins per IBM Blade Switch 840 570
Logins per Nexus 5000 2,048 2,048
The stated numbers are verified logins and are the supported number of logins.
BRKSAN-3707
Intelligent Fabric
Applications
Data Mobility Manager
BRKSAN-3707

Presentation_ID.scr
Data Migration Solutions—Host Base
Host Base Migration
Benefits Application
Server
Server Data Flow
Uses existing host base volume
management Mirrored Data Flow
Non-disruptive to application server Host Volume

9G RAID 1
Heterogeneous array migration
No added cost (other than cost of
volume manager with mirroring
capability) Fabric A Fabric B
Draw Backs
CPU intensive when migrating
Affects application performance 9G 9G
Existing New
Storage Storage
Vendor Vendor
X Y
BRKSAN-3707
Data Migration Solutions—Storage Base

Host Base Migration
Benefits Applicatio
n Server
Server Data Flow
Offloads application server CPU
Migration Data Flow
Migrates multiple servers at a time
9G Host Volume
Draw Backs
Uses Proprietary replication
technology from array
Requires separate port for specific Fabric A Fabric B
replication (migration) on array
Migration within same vendor’s
family of storage and may have to
be within same tier 9G 9G
Very costly $$$
Existing New
Storage Storage
Vendor X Vendor
X
BRKSAN-3707

Presentation_ID.scr
Data Migration Solutions—Network Base
Network Base Migration Application
Benefits Server Data Flow
Server
Offloads Application CPU Migration Data Flow

Lower cost tool
9G Host Volume
Heterogeneous across array vendors
More scalable
No single point of failure
Draw Backs Fabric A Fabric B

Single disruption to application
server during cut-over
9G 9G
Existing New
Storage Storage
Vendor Vendor
X Y
BRKSAN-3707
Supported Hardware for DMM

32-port FC Storage Services Module
Fully distributed architecture provides huge
aggregate performance
Embedded ASICs for inline SCSI
processing
Integrated 32 Fibre Channel port
Number of SSMs Required

A minimum of 1 SSM
A minimum of 2 SSMs is supported for
Dual Fabric
Advanced Feature Support in SAN-

OS 3.2(1)
FC-Redirect
DMM utilizes FC-Redirect
BRKSAN-3707

Presentation_ID.scr
What Is FC-Redirect (FCR)?
Is a Target centric Transport infra structure feature on

the MDS supervisor, does the FC DID/SID re-write only.
Seamless integration of one or more intelligent services
in a fabric for a specific Host & Disk (I_T) pair.
No re-wiring or re-configuring existing Host’s & Disk’s.
No Splitting of fabrics into multiple VSAN's.
Operate in a heterogeneous switch environment
Disk must be attached to a FC-Redirect aware MDS,
Host & SSM can be located anywhere in the fabric.
BRKSAN-3707
Life of Packet from Host to Disk
VT < H VT < H
2
DPP FWD
VI > T VI > T
SSM
2
[H => VT]
1 [VI => T]
3
[H =>T] FC
Target
FCID: H Switch
[H => T]
4
MAC FWD FCID: T
H>T H > VT
1
Link Between Target SW & Host
T
MAC FWD MAC
VI > T H>T
3
Trunk Link Between Target SW & SSM SW
BRKSAN-3707

Presentation_ID.scr
Server I/O Handling—Synchronous Mode
Server
Dealing with Server IOs
Writes to “Migrated”
Area are Mirrored
Writes to “Being
Migrated” Area are
queued temporarily
(till region has been
Migrated migrated)
Writes to “To be
Being Migrated Migrated” Area are
written to Existing
Storage only
To be Migrated
Server Reads are read
from Existing Storage
Existing New only
Storage Storage
LUN LUN
BRKSAN-3707

Server Dealing with Server IOs
Area are Mirrored
Writes to “Being
queued temporarily
Migrated migrated)
Writes to “To be
written to Existing
Storage only
To be Migrated
Existing New only
Storage Storage
LUN LUN
BRKSAN-3707

Presentation_ID.scr
Area are Mirrored
Writes to “Being
queued temporarily
Migrated migrated)
Writes to “To be
written to Existing
Storage only
To be Migrated
Existing New only
Storage Storage
LUN LUN
BRKSAN-3707

Area are Mirrored
Writes to “Being
queued temporarily
Migrated migrated)
Writes to “To be
written to Existing
Storage only
To be Migrated
Existing New only
Storage Storage
LUN LUN
BRKSAN-3707

Presentation_ID.scr
Server I/O Handling—Asynchronous Mode
Server
Mark all regions in MRL dirty

Modified Region Log [MRL]
While (MRL regions left) {
Select a Region;
Copy Region;
Clear MRL Region
}
Existing New
Storage Storage
LUN LUN
BRKSAN-3707

Modified Region Log [MRL] • Writes are written to

Existing Storage only
• MRL entry is updated
for each Write issued
Multiple passes of
MRL done until all
regions are clear
For cut-over last MRL
pass done with the
LUN in the offline
Existing New mode
Storage Storage
LUN LUN
BRKSAN-3707

Presentation_ID.scr
Modified Region Log [MRL]

from Existing Storage only
Existing New
Storage Storage
LUN LUN
BRKSAN-3707
DMM for Core-Edge

Environment Configuration Existing Storage New Storage
Place SSM at the Core switches for both Fabric
A and Fabric B
Existing Storage and New Storage should be
on the same switch where SSM resides
SSM SSM
Storage SHOULD NOT be connected to
the SSM Core
Switches
Storage can be connected on
16-port
MPS
12-port Edge
24-port Switches
Storage Services Module

Install DMM license
Enable DMM feature Server
Recommended that SSM ports not to be used
for any devices
BRKSAN-3707

Presentation_ID.scr
Intelligent Fabric
Applications
Storage Media
Encryption
BRKSAN-3707
Cisco SME Overview

Application
Server Encrypts storage media (data at rest)
Strong IEEE compliant AES-256
Name: XYZ
SSN: 1234567890 encryption
Amount: $123,456
Status: Gold
Integrated as transparent fabric service
Key Management
Center Supports heterogeneous tape
IP
Encrypt devices, and VTLs
Offers secure, comprehensive key
management
Name: XYZ
@!$%!%!%!%%^&
SSN: 1234567890
*&^%$#&%$#$%*!^
Amount: $123,456
@*%$*^^^^%$@*)
Status: Gold
%#*@(*$%%%%#@
Compresses tape data
Allows offline media recovery
Tape Built upon FIPS level-3 system
Library
architecture
Networkers Session BRKSAN-2893
BRKSAN-3707

Presentation_ID.scr
Transparent Fabric Service
Integrates seamlessly with existing
Application Servers Cisco MDS fabrics
Non-disruptive deployment (FC-R)
No appliances to insert in data path
No SAN re-wiring or re-configuration
Redirects traffic flows after enabling

MPS-18/4 MPS-18/4 encryption
Highly saleable performance
Load balances automatically
Reliable, highly available service
Tape Routes traffic to another MPS when
Library one fails
BRKSAN-3707
Cisco SME Enabled Platforms

HIGH-PERFORMANCE INTEGRATED SOLUTION WITH
MULTI-GIGABIT THROUGHPUT
18 4-Gbps ports for FC,
4 GigE for IP Services
MDS 9222i
MDS 9216A
MDS 9216i MDS 9506 MDS 9509 MDS 9513
18/4-Port Multiservice Module (MSM)
Cisco Fabric Manager w/Key Management Center
BRKSAN-3707

Presentation_ID.scr
SME Cluster
Consists of up to four SME enable
Application Servers switches (nodes) in the same
physical fabric
Node-to-node communication via IPFC
through management interface
Quorum based cluster
Provides scalability, reliability, availability
MSM-18/4 MSM-18/4
and automatic load balancing
Scalability is achieved by adding additional line
card in the fabric
Target based load balancing
Re-routes traffic when failure occurs
Single point of management with
Tape
Cisco FM
Library Can provide services across multiple
VSANs
One cluster per physical fabric
BRKSAN-3707
Cisco Key Management Center (KMC)

Cisco Key
Management Center Essential key lifecycle management
FMS Key Archives, recovers, distributes, and shreds
Catalog DB media keys
Application Servers Transports keys and management

traffic securely (SSH, HTTPS)
Integrates with Cisco FM server
No additional software to install
Intuitive provisioning and management with
Cisco FM Web client
MSM-18/4
MSM-18/4 MSM-18/4
MSM-18/4
Fabric ’A’ Fabric ’B’
May use the local data base or the
enterprise data base for the desired
level of reliability and availability. Key
Tape Catalog data base options:
Library PostgreSQL
Oracle 10g Express
Third party key manager (ex: EMC’s RSA)
BRKSAN-3707

Presentation_ID.scr
Intelligent Fabric
Applications
SANTap
BRKSAN-3707
SANTap Intelligent Write Splitting

Initiators
Initiator’s VSAN
(SANTap)
Initiator target I/O Not in
primary
data path
SANTap
SAN
Copy of Appliance
primary
I/O Targets and
Appliance
VSAN
Target
Appliance Partners leverages SANTap services

Part of the Cisco Storage Services Module (SSM)
Out-of-band architecture
SANTap redirects I/O and eliminates need for host splitter
Virtual SAN configurations

BRKSAN-3707

Presentation_ID.scr
SANTap Partner Solutions
Appliance Appliance
Network-Based Data Protection CDP/CRR Recovery at Local or

Support heterogeneous storage and servers Remote Site
Integrated with Cisco MDS9000 SANTap Tracks all data changes to every protected LUN
Supports VMWare Virtual Machines (RDM) Utilizes bookmarks for application-aware recovery
Heterogeneous Replication Enables Read/Write processing of replicated LUNs
Works with any supported storage CRR Advanced WAN functionality
True Any to Any Volume Replication WAN data reduction and compression
FC to TCP/IP conversion
TCP Optimization
BRKSAN-3707
SANTap Deployment—Before/After
BEFORE AFTER
Application Server Application Server
Front-End VSAN
DVTLUNs 1
Appliances
2 Cluster
DVT
PRODUCTION VSAN
SSM
SSM Back-End VSAN
9 Virtual
Initiators
AVT
CVT 0
0
1
1
CVTLUNs 2
2 AVTLUNs
Storage Array
Storage Array
BRKSAN-3707

Presentation_ID.scr
CRR—Asynchronous Flow
Main Data Center Remote Data Center
Application Server
Application Server
EMC RecoverPoint Asynchronous Replication
1. Write I/O is sent to SSM module
1 4 2. Write I/O is then forward to both local Storage Array
and local Appliance
3. Both local Storage Array and local Appliance
acknowledge Write I/O back to the SSM
4. Once SSM receives both acknowledgements, then
sends acknowledgment to Application Server
3 4
2 1 SSM
SAN SSM SAN
WAN
2 3 Appliances Appliances 2 3
1. I/O is sent through the WAN to remote Appliance

2. I/O is then sent to replication LUN(s) through the SSM
3. I/O is then acknowledged back to the Remote Appliance
4. Remote Appliance then sends acknowledgement back to
Primary Data Center Appliance through the WAN
Storage Array Storage Array

BRKSAN-3707
X-Bar
SSM Line Card
2 Gbps each 2 Gbps each
Forwarding Engine
2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps
DPP2 DPP3 DPP6 DPP7 DPP1 DPP4 DPP5 DPP8

DVT
Ports 1– 4 Ports 5– 8 Ports 9-12 Ports 13-16 Ports 17-20 Ports 21-24 Ports 25-28 Ports 29-32
Host1
BRKSAN-3707

Presentation_ID.scr
X-Bar
SSM Line Card
2 Gbps each 2 Gbps each
Forwarding Engine
2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps 2 Gbps
DPP2 DPP3 DPP6 DPP7 DPP1 DPP4 DPP5 DPP8

DVT
Ports 1– 4 Ports 5– 8 Ports 9-12 Ports 13-16 Ports 17-20 Ports 21-24 Ports 25-28 Ports 29-32
Host1
BRKSAN-3707
Front-End VSAN—Zoning
Only physical host initiators and DVTs reside in Front-End VSANs
Normal zoning applies where “Host Initiator” is zoned with DVT
NOTE: A single host initiator zoned with 2 or more separate DVTs,
must make sure that all of those DVTs reside on the same DPP
Fabric-A Fabric-B
RecoverPoint Front-End VSAN 30 RecoverPoint Front-End VSAN 40
Host1 Zone Host1 HBA1 Host1 Zone Host1 HBA2

DVT1, DVT2 DVT1, DVT2
Host2 Zone Host2 HBA1 Host2 Zone Host2 HBA2

DVT3, DVT4 DVT3, DVT4
BRKSAN-3707

Presentation_ID.scr
Back-End VSAN—Zoning
Fabric-A Fabric-B
SANTap Back-End VSAN SANTap Back-End VSAN
Appliance- Appliance-
Targets SSM 9VIs APP1-P0 Targets SSM 9VIs APP1-P2
CVT APP2-P0 CVT APP2-P2
Appliance- Appliance-
Initiators APP1-P1 Initiators APP1-P3
Storage Ports Storage Ports
APP2-P1 APP2-P3
Appliance VT- Appliance VT-

Initiators AVT Initiators Initiators AVT Initiators
APP1-P1 APP2-P1 APP1-P3 APP2-P3
Appliance VT- Appliance VT-

Target AVT Targets Target AVT Targets
BE-Host Zone BE-Host Zone

Host VI Host VI
Storage Ports Storage Ports
Appliance Local Appliance Local

Storage Storage Port Storage Storage Port
BRKSAN-3707
SANTap Limits
Table 1: SANTap Limits
SSI Images 3.0(2j) 3.1(2m) 3.1(3) 3.2(3i)

Max # of ITL per DPP 1,024 1,024 1,024 3,096
Max # of ITL per SSM 1,024 2,048 4,080 24,576
Max # of Sessions per SSM 1,024 2,048 2,048 2,048
Max # of LUNs per Initiator per DVT 256 for all SSI images
Max # of LUNs per DVT 1,024 1,024 1,024 3,096
Max # of host (initiators) per DVT 16 16 16 64
Max # of DVTs per SSM 16 16 32 64
Max # of DVTLUNs per SSM 1,024 2,048 4,096 16,384
LUN ID Addressing size 16 16 16 32
BRKSAN-3707

Presentation_ID.scr
Intelligent Fabric
Applications
Storage Virtualization
BRKSAN-3707
SAN-Based Storage Virtualization

Performance architecture
Leverages next-generation “intelligent”
SAN switches
Scalable architecture
Virtual volumes Split-path architecture for high performance
A “stateless” virtualization architecture does
not store any information written by the
application.
Meta-Data Meta-Data High speed, high throughput data mapping
Purpose-built ASICs (DPP) that handle
and redirect I/O at line speed, with almost
no additional latency
Based on instructions provided by the Meta-
Data Appliances
Provides advanced functionality

Multi-vendor arrays
Supports heterogeneous environments
BRKSAN-3707

Presentation_ID.scr
Storage Virtualization Logical Topology
Front-End Pooled Back-End

VSAN resources VSAN
Virtual Virtual
targets initiators
BRKSAN-3707
Data Flows
Control Frame
Data Frame
Meta-Data
Appliance
IP
BRKSAN-3707

Presentation_ID.scr
Network-Based Volume Management
Applications
Simplify volume presentation and

management
Create, delete, change storage volumes
Provides front-end LUN Masking and
mapping of storage volume to hosts
Virtual
Centralize management and control volumes
Single Invista console to manage virtual

volumes, clones, and mobility jobs
Reduce management complexity of

a heterogeneous storage
Single management interface to allocate
and reallocate storage resources
BRKSAN-3707 Physical storage
Dynamic Volume Mobility Explained

Virtualization
Hosts see Storage Virtualization
as an array
Presents virtual volumes to hosts
Maps virtual volumes to physical
Virtual Volumes Virtual LUN: 10 volumes
To Move a Volume:
Data Path Data Path
Controlle
r
Controlle
r Select source and target volumes
Virtual initiators
Network synchronizes the
volumes, then changes the virtual-
physical mapping
Array: 1 Array: 2 No I/O disruption to host
LUN: 20 LUN: 30
EMC HDS
BRKSAN-3707

Presentation_ID.scr
Heterogeneous Point-in-Time Copies
Applications
Create point-in-time copies
Source and clone can be on different,
heterogeneous storage arrays
Enable replication across SAN

heterogeneous storage
Leverage existing storage investments
Reduce replication storage capacity and Virtual volume Active
management costs volume
Maximize replication benefits to support

service levels
Backup and recovery
Clone Clone Clone Data
Testing, development, and training
Parallel processing, reporting, and queries
Physical storage
BRKSAN-3707
VSAN Considerations
Back-End VSAN
Multiple Back-End VSAN
supported by some partners
HR VSAN 20
Zone all 9 VIs to
storage ports FC
Storage
FC
FC VSAN 10
VT1
Best practice to create
fcalias for all 9 VIs MDS 9xxx
DEV VSAN 30 VI 1- 9
Front-End VSAN FC
VT2
FC
Invista
FC
Up to 32 Virtual Targets
per SSM
VT3
Zone server HBA to one
Virtual Target
FC
FC
Control VSAN
FC
ERP VSAN 40
ERP Admin
Communication to
external CPC
Zone up IP interfaces for
VSAN and SSM’s CPP
BRKSAN-3707

Presentation_ID.scr
Q and A
BRKSAN-3707
Recommended Reading
Continue your Cisco Live

learning experience with further
reading from Cisco Press
Check the Recommended
Reading flyer for suggested
books
Available Onsite at the Cisco Company Store

BRKSAN-3707

Presentation_ID.scr
Complete Your Online
Session Evaluation
Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.
BRKSAN-3707
BRKSAN-3707

Presentation_ID.scr

Advanced SAN Design - Virtualization Technologies and Intelligent Applications Design Considerations

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced SAN Design - Virtualization Technologies and Intelligent Applications Design Considerations

Uploaded by

Copyright:

Available Formats

Advanced SAN Design—

© 2006, Cisco Systems, Inc. All rights reserved. 1

© 2006, Cisco Systems, Inc. All rights reserved. 2

 Virtual Fabric Trunking and Port-Channels

 Fabric Routing (IVR)

Trunk BRCD = Port Channel Cisco

Cisco’s Approach to Virtual Fabric:

© 2006, Cisco Systems, Inc. All rights reserved. 3

move a port from one

 Basis for Virtual Fabric

VSAN Numbering Rules

All ports are originally in VSAN1 VSAN 20

 VSAN 2 through 4093 can be VSAN 30

assigned to ‘user’ VSANs—VSAN Trunking E_Port

Currently 256 VSANs is supported from Enhanced ISL VSAN 30 Is Not

© 2006, Cisco Systems, Inc. All rights reserved. 4

 Applicable to N_Ports, F_Ports and

 The ANSI T11 FS-SP group has

VSAN Header Field

 Each frame on a VSAN trunk carries an extra 8 bytes of header:

© 2006, Cisco Systems, Inc. All rights reserved. 5

EISLs and TE-Port

© 2006, Cisco Systems, Inc. All rights reserved. 6

 Two switches then conduct an ESC (Exchange Switch ESC

© 2006, Cisco Systems, Inc. All rights reserved. 7

 Consistently detect mis-

VSANs, EISLs, TE_ports, Port Channels—How

 All member links of a Port Channel 8 Gbps

 Port Channel technology provides E_Port

 Multiple Port Channels yield multiple 4 Link (8 Gbps)

© 2006, Cisco Systems, Inc. All rights reserved. 8

© 2006, Cisco Systems, Inc. All rights reserved. 9

Without fabric services interference across Virtual

 Enable end devices from different virtual

Fabric Routing Applications—

 Overlay data replication fabrics

No need for separate pair of

 Being able to share common SAN

SAN Extension circuits

© 2006, Cisco Systems, Inc. All rights reserved. 10

Local IP WAN Local

© 2006, Cisco Systems, Inc. All rights reserved. 11

 Allows provisioning with Server Storage Arrays

virtualized servers and storage X Y

NPIV (N_Port Identifier Virtualization)

LUN 3 N_Port ID=3

© 2006, Cisco Systems, Inc. All rights reserved. 12

 NPIV capable devices

Blade Switch Explosion Issues

© 2006, Cisco Systems, Inc. All rights reserved. 13

Differences Between NPIV and NPV

© 2006, Cisco Systems, Inc. All rights reserved. 14

 When NP port comes up on a NPV edge switch, it first

Nested NPIV FLOGI/FDISC

device and allowing to login using the NPV Edge Switch

© 2006, Cisco Systems, Inc. All rights reserved. 15

MDS 9124—NPV Architecture

© 2006, Cisco Systems, Inc. All rights reserved. 16

Port-Group1: Ports 1 – 4 Port-Group2: Ports 5 – 8

Port Mapping for HP Blade Switches

PG 1 -> EXT Port 1

PG 4 -> EXT Port 5 and EXT Port 6

Virtual Fabric Trunking and Port-Channels

Fabric Routing (IVR)

Basis for Virtual Fabric

VSAN 2 through 4093 can be VSAN 30

Applicable to N_Ports, F_Ports and

The ANSI T11 FS-SP group has

Each frame on a VSAN trunk carries an extra 8 bytes of header:

Two switches then conduct an ESC (Exchange Switch ESC

Consistently detect mis-

All member links of a Port Channel 8 Gbps

Port Channel technology provides E_Port

Multiple Port Channels yield multiple 4 Link (8 Gbps)

Enable end devices from different virtual

Overlay data replication fabrics

Being able to share common SAN

Allows provisioning with Server Storage Arrays

NPIV capable devices

When NP port comes up on a NPV edge switch, it first

Internal Links (All links set to F-

Internal Links (All links set to F-

Draw Backs Fabric A Fabric B

Number of SSMs Required

Advanced Feature Support in SAN-

Is a Target centric Transport infra structure feature on

Storage Services Module

Redirects traffic flows after enabling

Application Servers Transports keys and management

Appliance Partners leverages SANTap services

Virtual SAN configurations

Network-Based Data Protection CDP/CRR Recovery at Local or