Professional Documents
Culture Documents
BRKSAN-1701
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3
Storage Basics
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4
SCSI Primer
Storage topologies
DAS, NAS, and SAN
SCSI Transport
Fibre Channel (FC) , FCIP, iSCSI
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5
SCSI Primer
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7
SAN
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10
SCSI
Initiator
SCSI
Target
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12
I/O Channel
Channel
SCSI bus which can be internal or external Controller
to a host
Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth up to 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
Target 1 Target 2
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13
Target
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14
Adapter (HBA)
SCSI
SCSI
Fibre Channel overcomes many
shortcomings of Parallel I/O including:
Addressing for up to 16 million nodes Fibre Channel
HBA
Loop (shared) and fabric (switched) transport
Host speeds of 100 to 400 MBps (1–4 Gbps)
Fibre Channel
Segments of up to 10km (without extenders) Fabric
Target
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16
Storage Topologies
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18
IP
Network SAN
Tape
Direct Attached Storage
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21
Host/ File System File System File System File System File System File System
SAN IP IP IP IP
Storage
Transport NIC
TCP/IP Stack
NIC
TCP/IP Stack
NIC
TCP/IP Stack
NIC
TCP/IP Stack
iSCSI Layer iSCSI Layer File System File System
SCSI FC Bus Adapter FC HBA Device Driver FC HBA
Media
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24
Parallel SCSI
Fibre Channel (FC), FCoE, FCIP, iSCSI
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26
I/O Channel
Channel
parallel SCSI bus which can be internal Controller
or external to a host
Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth up to 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation
Target 1 Target 2
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29
Link FC Single
FC-4 SNMP Mapping SCSI-3 Mapping
Encapsulation Byte Command
(FC-SNMP) (SCSI-FCP)
(FC-LE) Sets
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30
FCP Goes
Here (FC-4)
FC-3
FC-2
FC-AL
FC-1
FC-0
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32
Frame Content
Frame
Idles* SOF Data Field CRC EOF Idles*
Header
(4) (24) (0-2112) (4) (4)
*Six idle words (24 bytes) required by TX; two idle words (8 bytes) guaranteed to RX
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33
OX_ID &
RX_ID Exchange
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34
Sequence #6
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36
Ethernet
Fibre
Channel
Traffic
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37
Storage Protocols—FCoE
FCoE Frame Format
From a Fibre Channel standpoint it’s FC connectivity
over a new type of cable called an Ethernet cloud
From an Ethernet standpoints it’s yet another ULP
(Upper Layer Protocol) to be transported
FC frame not changed
FCoE
FC
SCSI
Data
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39
Storage Protocols—FCIP
FCIP Frame Format
Primary use is for Storage-to-Storage connectivity via an IP
WAN/MAN
The FCIP Link carries encapsulated fibre channel traffic between
Link End Points over an IP network by using TCP on port 3225
The result is a virtual Inter Switch Link (ISL) between FC Fabrics
FC frame not changed
IP
TCP
FCIP
FC
SCSI
Data
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41
IP
TCP
iSCSI
SCSI
Data
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42
FC FC FC
FCIP
TCP TCP
IP IP FCoE
PHYSICAL WIRE
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45
Physical
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46
Channel switches
‘GL’ port: A generic port capable of operating
as either an ‘E’ or ‘F’ port; it’s also capable of E E
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47
Fabric Switch
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48
Device
Host
Host/ Device
Interface
FC
N_Port
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49
E E
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50
A link consists of
Two unidirectional “fibers” transmitting in opposite directions
May be either:
Optical fiber or Copper
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51
Node
The equipment which contains one or more
N_Port or NL_Port (topology dependent)
May be:
Computer (HBA)
Controller (Port on Disk Subsystem)
Device (SCSI FC to Parallel converter)
Is not a switch fabric device
Controller
Point to point
N_Port to N_Port
Flow controlled
Acknowledged Node Node
Host Host
TX RX
N_Port
FC FC
RX TX
Link
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54
Link
NL_Node
“B”
Switched Fabric
N_Port N_Port N_Port N_Port N_Port N_Port
A B
Switch
Fabric
Communications Model—Source to Destination Based on
Address Routing through the Fabric; Still a Point to Point
Connection; FSPF routing required when more then two
switches make up the Fabric
A B
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57
Operational
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59
Fabric Addressing
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61
Fabric Controller
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63
Fabric Configuration
Disruptive/Non-Disruptive
One of the following three conditions can trigger BF
(non-disruptive) or RCF (disruptive)
Two disjoint fabrics are combined together
A principal ISL fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66
Benefits ZoneC
Zoning—Enforcement
Zoning is used to control access in a SAN
Soft zoning
Enforced by name server query responses
Name server sends membership list to N_Port
N-port accesses members only
Hard zoning
Enforced by hardware (forwarding ASIC) at wire speed pWWN, fWWN, FC_ID, FC_Alias
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69
ISL PortChanneling
A PortChannel Is a Logical Bundling of Identical Links
Criteria for forming a PortChannel
Same speed links
Same modes (auto, E, etc.) and states
Between same two switches
Same VSAN membership
E.g. 4 Gbps
Treated as one logical ISL by upper layer PortChannel
protocols (FSPF) (Two x
2Gbps)
Can use up to 16 links in a PortChannel (32Gbps max)
Can be formed from any ports on any modules— E.g. 8 Gbps
PortChannel
HA enabled (Four x
Exchange-based in-order load balancing 2Gbps)
Mode 1: Based on src/dst FC_IDs
Mode 2: Based on src/dst FC_ID/OX_ID
Much faster recovery than FSPF-based balancing
Given logical interface name with aggregated bandwidth
and derived routing metric
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76
A B
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77
Collapsed Core-Edge
Oversubscription primarily in
chassis and linecards A B
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78
Q and A
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80
BRKSAN-1701
14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82