Introduction to Storage Area Networks (SAN

)

BRKSAN-1701

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Session Agenda
Main Topics for Today Storage Basics
What is it?

Fibre Channel Basics
What makes it tick?

Introduction to SAN Design
… and a little bit of the how-to

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3

Storage Basics

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

4

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Storage Basics
SCSI Primer Storage topologies
DAS, NAS, and SAN

SCSI Transport
Fibre Channel (FC) , FCIP, iSCSI

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

SCSI Primer

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Standards
SCSI has evolved since it was introduced as SASI in 1979 by Shugart Associates—it was approved as a standard by ANSI in 1986 and is now referred to as SCSI-1 SCSI-2 was approved by X3 in 1990 and by ANSI in 1994 SCSI-3 refers to a collection of standards, each of which defines a very specific part of SCSI: physical interface, transport interface, command interface, architecture model, programming interface, etc.
Also known as SCSI Architecture Model 3 (SAM-3)

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

7

SCSI Architecture Model

SAN

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

8

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Small Computer System Interface—SCSI
SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk) Interface refers to connectors, cables, electrical signals, optical signals and the command protocol that allow initiators and targets to communicate Logical Unit Number (LUN): A 64-bit field within SCSI that identifies the Logically addressable Unit within a target SCSI device

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

9

SCSI Command Protocol
SCSI command protocol is the de facto standard that is used extensively in high-performance storage applications The command part of SCSI can be transported over various types of interconnecting technologies A basic understanding of SCSI communications is beneficial in understanding storage networks.

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

The SCSI I/O Transaction
The SCSI protocol forms the basis of an I/O transaction The channel provides connectivity between communicating devices in a SCSI transaction
The following shows two sample SCSI exchanges:
Disk (Target)
READ

Host (Initiator)
DATA STATUS DATA DATA

SCSI READ OPERATION
SCSI I/O Channel

Host (Initiator)
READY STATUS

SCSI WRITE OPERATION
SCSI I/O Channel

Disk (Target)
DATA WRITE DATA DATA

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

11

The SCSI I/O Channel—Starting Point
SCSI I/O Channel provides half-duplex pipe for SCSI command, data, and status SCSI I/O channel can be internal or external to host Multiple SCSI I/O channels can exist within host A network approach can scale the I/O channel in many areas (length, devices, throughput)
Applications File System Block Device SCSI Generic

Half-Duplex I/O Channel
SC SI

TCP/IP Stack

SCSI

NIC Driver NIC Adapter

Adapter Driver

SCSI Adapter

SI SC

SCSI Initiator SCSI Target
BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

12

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

The Parallel SCSI I/O Channel
The SCSI Channel is used to transmit SCSI commands, data, and status Multiple devices can exist on the channel each playing the role of SCSI initiator or SCSI target
I/O Channel Host (Initiator)

SCSI SCSI

Most common channel is the basic parallel SCSI bus which can be internal or external to a host Parallel SCSI Specifications:
Up to 25m bus length Shared channel bandwidth up to 320MBps Up to 16 devices per SCSI bus Half-duplex operation

Channel Controller

Target 1
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Target 2
13

Networking the I/O Channel
Host (Initiator)

Same SCSI protocol carried over a network transport via serial implementation Transport must not jeopardize SCSI payload (security, integrity, latency)

Initiator SCSI SCSI

Networked I/O Channel

Two primary transports to choose from today, namely IP and Fibre Channel A networked I/O channel allows for multiple improvements:
Distance limitations greatly increased Dedicated bandwidth (not shared) High # of addressable devices Bandwidth increase (including link bundling)

Channel Controller

Network

Target
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel I/O Networking
Host System

Fibre Channel provides high speed transport for SCSI payload via Host Buss Adapter (HBA) Fibre Channel overcomes many shortcomings of Parallel I/O including:
Addressing for up to 16 million nodes Loop (shared) and fabric (switched) transport Host speeds of 100 to 400 MBps (1–4 Gbps) Segments of up to 10km (without extenders) Support for multiple protocols

Initiator SCSI SCSI

Fibre Channel HBA

Fibre Channel Fabric

Combines best attributes of a channel and a network together

Target
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

15

Consolidated I/O Networking
Traditional Ethernet with Fibre Channel over Ethernet Consolidated Network Adapter – (CAN)
Native Ethernet NIC Native FC HBA
FCoE (Target) FCoE Host (Initiator) Ethernet Network

FCoE FCoE

FCoE

CNA

Relies on Lossless Ethernet
PAUSE per IEEE 802.1p

Able to build Ethernet-based SANs using FCoE arrays

SCSI SCSI

FC Fabric

FC HBA Attached Host (Initiator)

Storage (Target)

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

TCP/IP I/O Networking
IP access to open systems iSCSI and Fibre Channel storage iSCSI driver is loaded onto hosts on Ethernet network Able to consolidate servers via iSCSI onto existing storage arrays Able to build Ethernet-based SANs using iSCSI arrays Storage can be mapped from iSCSI to Fibre Channel
iSCSI (Target) IP Network iSCSI Gateway
SCSI SCSI

iSCSI-Enabled Host (Initiator)

iSCSI iSCSI Ethernet NIC

FC Fabric

FC HBA Attached Host (Initiator)

Storage (Target)
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

17

Storage Topologies

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Storage Topologies

IP Network

SAN

IP Front-End Network
Host to Host Application to file system Client to Server NFS, SMB, CIFS, NCP NAS, WAFS
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

SAN Back-End Network
Host to Storage Block I/O to Storage Storage to Storage SCSI, IDE, NTFS, FAT SAN
19

Direct Attached Storage (DAS)
Hosts directly access storage as block-level devices Storage is captive ‘behind’ the server, limited mobility Limited scalability due to limited devices No efficient storage sharing possible Costly to scale; complex to manage

Clients

LAN

Application Servers

Win2k Linux Unix

Win2k Linux Unix

SCSI

FC

FC

Tape

Direct Attached Storage
BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Storage Area Network (SAN)
Storage is accessed at a block-level via SCSI High performance interconnect providing high I/O throughput Lower TCO relative to direct attached storage, storage can be shared Limited vendor interoperability Complex management
Fibre Channel SAN Block Storage Devices Clients

LAN
Servers

Separation of Storage from the Server

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

Network Attached Storage (NAS)
Storage is accessed at a file level via NFS or CIFS Storage is accessed over an IP network Storage devices can be shared between servers Files can be shared between users TCP can be tuned to optimize for file transport
LAN

Win2k Linux Unix

Generic

Generic

Application Servers

NAS = Optimized for File I/O
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

NAS Appliances or NAS Head Ends

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

iSCSI Storage Network
Storage is accessed at a block-level via iSCSI Good performance via standard Ethernet NIC Enhanced performance with TCP Offload Engine (TOE) Lower TCO relative to direct Fibre Channel HBA/Fabric Standards based
iSCSI
iSCSI-Enabled Hosts (Initiators)

iSCSI iSCSI

iSCSI Appliance iSCSI (Target) IP Network iSCSI Gateway FC Fabric FC HBA Attached Host (Initiator) Storage Pool (Target)

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

DAS, SAN, iSCSI, NAS Comparison
DAS
Computer System Application File System Volume Manager SCSI Device Driver SCSI Bus Adapter

SAN
Computer System Application File System Volume Manager SCSI Device Driver

iSCSI Appliance
Computer System Application File System Volume Manager
SCSI Device Driver iSCSI Driver TCP/IP stack NIC

iSCSI NAS Gateway Appliance
Computer System Application File System Volume Manager
SCSI Device Driver iSCSI Driver TCP/IP stack NIC

NAS Gateway
Computer System Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC

Computer System Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC

Host/ Server

FC HBA

Block I/O

File I/O

Storage Transport
SCSI

SAN

IP
NIC TCP/IP Stack iSCSI Layer Bus Adapter

IP
NIC TCP/IP Stack iSCSI Layer FC HBA

IP
NIC TCP/IP Stack File System Device Driver

IP
NIC TCP/IP Stack File System FC HBA

FC

Storage Media
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

FC

Block I/O

FC

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

SCSI Transport

Parallel SCSI Fibre Channel (FC), FCoE, FCIP, iSCSI

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

Network stack comparison
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

The Parallel SCSI I/O Channel
The SCSI Channel is used to transmit SCSI commands, data, and status Multiple devices can exist on the channel each playing the role of SCSI initiator or SCSI target
I/O Channel Host (Initiator)

SCSI SCSI

Most common channel is the basic parallel SCSI bus which can be internal or external to a host Parallel SCSI Specifications:
Up to 25m bus length Shared channel bandwidth up to 320MBps Up to 16 devices per SCSI bus Half-duplex operation

Channel Controller

Target 1
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Target 2
27

Fibre Channel Stack
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

28

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

What Is It? Fibre Channel
Channels
Connection service Physical circuits Reliable transfers High speed Low latency Short distance Hardware intense

Networks
Connectionless Logical circuits Unreliable transfers High connectivity Higher latency Longer distance Software intense

Fibre Channel Circuit and packet switched Reliable transfers High data integrity High data rates Low latency High connectivity Long distance

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Fibre Channel Protocol Architecture
ULP (Upper Level Protocol)
(sample – there are more)

SCSI-3

IP

FICON

FC-4

SNMP Mapping (FC-SNMP)

SCSI-3 Mapping (SCSI-FCP)

Link Encapsulation (FC-LE)

FC Single Byte Command Sets

FC-3 FC-2 FC-1 FC-0
BRKSAN-1701 14693_05_2008_X1

Common Services
Framing Protocol Encode / Decode Physical Interface Fibre Channel Physical & Signaling Interface (FC-PH, FC-PH2, FC-PH3) FC-AL FC-AL-2 FC-SW-2

8B/10B Encoding Copper Optical

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Mapping
SCSI Architecture to Fibre Channel Mapping

FCP Goes Here (FC-4)

FC-3 FC-2 FC-AL FC-1 FC-0

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Fibre Channel Functions
Structure Is Divided into Five Levels of Functionality
FC-0
Defines the physical interface characteristics Signaling rates, cables, connectors, distance capabilities, etc.

FC-1
Defines how characters are encoded/decoded for transmission Transmission characters are given desirable characters

FC-2
Defines how information is transported Frames, sequences, exchanges, login sessions

FC-3
Place holder for future functions

FC-4
Defines how different protocols are mapped to use Fibre Channel SCSI, IP, Virtual Interface architecture, FICON, others
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Frame Format
All FC-2 frames follow the general frame format as shown below Idles are ‘Ordered Sets’ used for synchronization and basic signaling SOF—Start-of-Frame, EOF—End-of-Frame

General FC-2 Frame Format Frame Content Idles* SOF (4) Frame Header (24) Data Field (0-2112)
0-528 Transmission Word *Six idle words (24 bytes) required by TX; two idle words (8 bytes) guaranteed to RX
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

CRC (4)

EOF (4)

Idles*

33

Fibre Channel FC-2 Hierarchy
Multiple exchanges are initiated between initiators (hosts) and targets (disks) Each exchange consists of one or more bi-directional sequences Each sequence consists of one or more frames For the SCSI3 ULP, each exchange maps to a SCSI command
OX_ID & RX_ID

Exchange Sequence Frame Sequence Frame
ULP Information Unit
34

SEQ_ID

Sequence Frame

SEQ_CNT

Frame Fields
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Example: Fibre Channel Exchange
One SCSI command is issued—a single exchange is created consisting of multiple sequences Each exchange is identified with unique OX_ID/RX_ID Each sequence is identified with unique SEQ_ID
Initiator Fibre Channel Fabric Data Returned in Multiple Sequences Target

Sequence

#1
SEQ#1 , CN T#1

Read Command Issued New Exchange Issued New Sequence Issued Sequence #2 Sequence #3 Sequence #4 Sequence #5 Sequence #6
BRKSAN-1701 14693_05_2008_X1

CNT#1 SEQ#2, CNT#1 SEQ#3, CNT#1 SEQ#4, CNT#1 SEQ#5,

CNT#2 SEQ#2, CNT#2 SEQ#3, CNT#2 SEQ#4, CNT#2 SEQ#5,

Exchange

CNT#1 SEQ#6,

Status Returned in Additional Sequence

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

35

Fibre Channel over Ethernet Stack
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FCoE: FC over Ethernet
FCoE is I/O consolidation of FC storage traffic over Ethernet
FC traffic shares Ethernet links with other traffics Requires a lossless Ethernet fabric

Ethernet Fibre Channel Traffic

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

Storage Protocols—FCoE
FCoE Frame Format From a Fibre Channel standpoint it’s FC connectivity over a new type of cable called an Ethernet cloud From an Ethernet standpoints it’s yet another ULP (Upper Layer Protocol) to be transported FC frame not changed

FCoE FC SCSI Data
BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel over IP Stack
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

39

Storage Protocols—FCIP
FCIP Frame Format
Primary use is for Storage-to-Storage connectivity via an IP WAN/MAN The FCIP Link carries encapsulated fibre channel traffic between Link End Points over an IP network by using TCP on port 3225 The result is a virtual Inter Switch Link (ISL) between FC Fabrics FC frame not changed

IP TCP FCIP FC SCSI Data
BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Internet SCSI Stack
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Storage Protocols - iSCSI
iSCSI Frame Format
Primary use is for Host-to-Storage connectivity via an IP LAN SCSI command and data are encapsulated into iSCSI by adding a special header iSCSI data is encapsulated into a TCP packet IP is the transport protocol

IP TCP iSCSI SCSI Data

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Network stack comparison
SCSI SCSI iSCSI SCSI FCP FC FCIP TCP IP Ethernet TCP IP Ethernet PHYSICAL WIRE
BRKSAN-1701 14693_05_2008_X1

SCSI FCP FC

SCSI FCP FC

FCoE Ethernet

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

SCSI Transport Summary
Parallel SCSI—Legacy host to storage bus topology Fibre Channel—Robust serial transport network for SCSI
Host-to-Storage Connectivity Primary transport for Storage Area Networks

FCoE – FC-2 encapsulation in Ethernet
Host-to-SAN Connectivity Typically used for unified server I/O

FCIP—FC-2 encapsulation in TCP/IP
SAN-to-SAN Connectivity Typically used in the WAN/MAN

iSCSI—TCP/IP transport for SCSI protocol
Host-to-SAN Connectivity Typically used in the LAN
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Basics

Fibre Channel Physical Attributes Fibre Channel Operational Characteristics

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

Physical
The main physical objects in Fibre Channel are:
Ports Links Nodes Fabric

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

46

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Port Types
‘N’ port: Node ports used to connect devices to switched fabric or point to point configurations ‘F’ port: Fabric ports residing on switches connecting ‘N’ port devices ‘L’ port: Loop ports are used in arbitrated loop configurations to build networks without FC switches; these ports often also have ‘N’ port capabilities and are called ‘NL’ ports ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches ‘GL’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port; it’s also capable of acting in an ‘L’ port capacity; Auto Discovery
N N

N

F

NL

FL

L

L

E

E

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

Port Type Examples

Fabric Switch Node NL_Port
FL_Port E_Port E_Port

Node NL_Port

G_Port

F_Port

N_Port

Node

Node NL_Port

G_Port

F_Port

N_Port

Node

G_Port

F_Port

N_Port

Node

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

48

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

N_Port

Device

Host
Host/ Device Interface

FC

N_Port
Serial Data Out Serial Data In

Can Be More than One N_Port on the Device

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

Inter-Switch Link (ISL)
The interconnection between switches is called the Inter-Switch Link (ISL)
E_Port to E_Port

Supports all classes of service
Class 1, 2, 3, and a special Class F

FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput E E

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Link
A link consists of
Two unidirectional “fibers” transmitting in opposite directions May be either: Optical fiber or Copper

Transmitters may be:
Long wave laser Short wave laser LED Electrical

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

Node
The equipment which contains one or more N_Port or NL_Port (topology dependent) May be:
Computer (HBA) Controller (Port on Disk Subsystem) Device (SCSI FC to Parallel converter)

Is not a switch fabric device

Controller

N_Port

N_Port

N_Port

N_Port

Link
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Link

Link

Link
52

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FC Communications Model
Point to point N_Port to N_Port Flow controlled Acknowledged
Node Node

Host
FC N_Port

Host
TX RX RX TX
FC

Link
BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Fibre Channel Fabric
The entity which interconnects N_Ports Provides routing based on destination address Fabric may be:
Point to point: Arbitrated loop: Switched: No routing required Routing is distributed across ports Routing provided by switch

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Arbitrated Loop
NL_Node “A”

Link

NL_Node “B”

Communications Model—Source to Destination Based on Address Routing Distributed in the NL_Ports on the Loop
A
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

B

Still Pt. to Pt. Communication
55

Switched Fabric
N_Port N_Port N_Port N_Port N_Port N_Port

A

B

Switch
Fabric Communications Model—Source to Destination Based on Address Routing through the Fabric; Still a Point to Point Connection; FSPF routing required when more then two switches make up the Fabric
A
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

B
56

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

FC Operational Characteristics

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Operational
The Main Operational Subsystems in Fibre Channel are:
Worldwide Names Fabric Addressing Directory Server Fabric Controller Principle Switch Fabric Routing Zoning Virtual SANs

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Worldwide Names
Each switch element is assigned a WWN at time of manufacture Each switch port is assigned a WWN at the time of manufacture During FLOGI the switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID) These address assignments can then correlate each fabric port with the switch element

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Fabric Addressing
The 24-bit FCID address is partitioned into three fields
Device Area Domain

This partitioning helps speed up routing Switch element assigns the address to N_Ports Address portioning is transparent to N_Ports
8 bits
Switch Topology

8 bits

8 bits

Switch Domain

Area

Device

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Directory Server
Repository of information regarding the components that make up the Fibre Channel network Located at address ‘FF FF FC’ (some readings call this the name server) Components can register their characteristics with the directory server An N_Port can query the directory server for specific information
Query can be the address identifier, WWN and volume names for all SCSI targets

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

Fabric Controller
Each switch has a fabric controller Assigned address ‘FF FF FD’
Every fabric controller in the fabric has the same address It is the N_Port within the switch Responsible for managing fabric, initialization, routing, setup and teardown of Class-1 connections

Responsible to receive request and generate responses for the switch fabric
Information must be consistent independent of which fabric controller responds to a request

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fabric Configuration: PS Selection
A principal switch shall be selected whenever at least one inter-switch link (a link between two E_Port) is established The selection process chooses a principal switch, which is then designated to assign domain identifier to all the switches in the fabric, and any who join later the fabric later on The principal switch selection can be triggered by anyone of the following events
Switch Boot and Exchange Fabric Parameters (EFP) Build Fabric (BF) Reconfigure Fabric (RCF)

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Fabric Configuration Disruptive/Non-Disruptive
One of the following three conditions can trigger BF (non-disruptive) or RCF (disruptive)
Two disjoint fabrics are combined together A principal ISL fails (upstream or downstream) A switch with Domain_ID request for another Domain_ID

Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; it forwards BF/RCF out of all E_ports except the incoming port (only once) and wait for the timer to expire When the timer expires, BF/RCF propagation state is left and principal switch selection begins BF is not a disruptive process RCF is a disruptive process
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fabric Routing: FSPF
FSPF routes traffic based on destination domain ID For FSPF a domain ID identifies a single switch
This limits the max number of switches that can support in the Fabric to 239 when FSPF is supported

FSPF performs hop-by-hop routing FSPF uses total cost as the metric to determine most efficient path FSPF supports hierarchical path selection
Provides the scalable routing tables in large topologies

Static routes can be applied
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

65

Fabric Routing: FSPF
Everyone says HELLO to their neighbor, on all initialized ISLs The neighbors say HELLO back, unless they are dead When the HELLO packet is received with both originator and recipient domain id, the two way communication is done and:
The ISL is active The ISL may be available as a two-way path for frames

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Fibre Channel Fabric Zoning
Zoning operation
Zone members “see” only other members of the zone Zones can be configured dynamically based on WWN Devices can be members of more than one zone FC-AL zoning allows the creation of private loops on a single hub Switched fabric zoning can take place at the port or device level Based on physical switch port Based on device WWN Based on LUN ID
SAN
Disk2 Disk3 Disk1

Benefits
Secured device access Allows operating system co-existence
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

ZoneA

Host1

ZoneC
Disk4 Host2

ZoneB

67

Zoning—Enforcement
Zoning is used to control access in a SAN Soft zoning
Enforced by name server query responses Name server sends membership list to N_Port N-port accesses members only

Hard zoning
Enforced by hardware (forwarding ASIC) at wire speed pWWN, fWWN, FC_ID, FC_Alias

Soft Zone
Host FC Zone-1 Host FC Zone-1

Hard Zone

Array MDS MDS Host FC Zone-2

Array MDS MDS Host FC Zone-2

Array
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved.

Array

Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Introducing Virtual SANs (VSANs)
A Virtual SAN (VSAN) Provides a Method to Allocate Ports within a Physical Fabric to Create Virtual Fabrics
Analogous to VLANs in Ethernet Virtual fabrics created from larger costeffective redundant physical fabric Reduces wasted ports of island approach Fabric events are isolated per VSAN— maintains isolation for HA (i.e. RSCNs) Hardware-based isolation—traffic is explicitly tagged across interswitch links with VSAN membership info Statistics can be gathered per VSAN
Physical SAN Islands Are Virtualized onto Common SAN Infrastructure Cisco MDS 9000 Family with VSAN Service

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

Virtualizing the Fabric—The Full Solution
To Build a Cost-Saving Fabric Virtualization Solution, Seven Key Services Are Required:
1. Virtual fabric attachment—the ability to assign virtual fabric membership at the port level 2. Multiprotocol extensions—the ability to extend virtual fabric service to iSCSI, FCIP, FICON, etc. 3. Virtual fabric services—the ability to create fabric services per virtual fabric (routing, zones, RSCNs, QoS, etc.) 4. Virtual fabric diagnostics—the ability to troubleshoot per virtual fabric problems 5. Virtual fabric security—the ability to define separate security policies per virtual fabric 6. Virtual fabric management—the ability to map and manage virtual fabrics independently 7. Interfabric routing—the ability to provide connectivity across virtual fabrics—without merging the fabrics ISL MDS MDS 9000 9000 Family Family Full Service End-to-End Virtual Fabric Implementation
70

Virtual Fabric Service Model Intervirtual Fabric Routing Virtualized Fabric Management Virtualized Fabric Security Policies Virtualized Fabric Diagnostics Virtualized Fabric Services Multiprotocol Transport Extensions Virtualized Fabric Attachment

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

VSANs and Zones—Complimentary
Virtual SANs and Fabric Zoning Are Very Complimentary
Hierarchical relationship—
First assign physical ports to VSANs Then configure independent zones per VSAN

Relationship of VSANs to Zones Physical Topology
VSAN 2 Disk2 ZoneA Disk3 Host1 Disk1 Disk4 Host2

VSANs divide the physical infrastructure Zones provide added security and allow sharing of device ports VSANs provide traffic statistics VSANs only changed when ports needed per virtual fabric Zones can change frequently (e.g. backup) Ports are added/removed nondisruptively to VSANs
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

ZoneC

ZoneB VSAN 3

ZoneD Host4

ZoneA Disk5 Host3 Disk6

71

ISL PortChanneling
A PortChannel Is a Logical Bundling of Identical Links
Criteria for forming a PortChannel
Same speed links Same modes (auto, E, etc.) and states Between same two switches Same VSAN membership

Treated as one logical ISL by upper layer protocols (FSPF) Can use up to 16 links in a PortChannel (32Gbps max) Can be formed from any ports on any modules— HA enabled Exchange-based in-order load balancing
Mode 1: Based on src/dst FC_IDs Mode 2: Based on src/dst FC_ID/OX_ID

E.g. 4 Gbps PortChannel (Two x 2Gbps) E.g. 8 Gbps PortChannel (Four x 2Gbps)

Much faster recovery than FSPF-based balancing Given logical interface name with aggregated bandwidth and derived routing metric
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

72

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

PortChannel vs. Trunking
ISL = inter-switch link PortChannel = E_Ports and ISLs Trunk = ISLs that support VSANs Trunking = TE_Ports and EISLs

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

73

Fibre Channel Basics Summary
Fibre Channel is a very robust, hierarchical standard Fibre Channel utilizes a Point-to-Point communications model irrespective of the topology Fibre Channel includes a full set of services for naming, addressing, building, and managing fabrics Fibre Channel utilizes FSPF, an OSPF like routing protocol to route traffic Fibre Channel Zoning is a method of logically grouping devices within a given fabric We Have Only Scratched the Surface of FC Protocol Here!
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

74

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Introduction to SAN Design

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

75

Type of SAN Networks
Two primary designs for SANs; Core-Edge and Collapsed Core-Edge Each design provides different advantages SAN still driven by oversubscription; host to disk, ISL, chassis As SANs increase in size, some modification to each model may be required SAN network services need to be considered in SAN design

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Core-Edge
Traditional SAN design for growing SANs High density directors in core and fabric switches, directors or blade switches on edge Predictable performance Scalable growth up to core and ISL capacity
A B A B

A A B

B

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

Collapsed Core-Edge
SAN design to take advantage of high density directors Most traffic localized, reducing number of ISLs Oversubscription primarily in chassis and linecards Potential to scale further than traditional Core-Edge design
A B

A

B

A

B

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Additional Information
Cisco Storage Networking
http://www.cisco.com/go/storagenetworking

Cisco Data Center Networking
http://www.cisco.com/go/datacenter

Storage Network Industry Association (SNIA)
http://www.snia.org

Internet Engineering Task Force—IP Storage
http://www.ietf.org/html.charters/ips-charter.html

ANSI T11—Fibre Channel
http://www.t11.org/index.htm
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

79

Q and A

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press® Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store
BRKSAN-1701 14693_05_2008_X1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

81

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes; winners announced daily Receive 20 Passport points for each session evaluation you complete Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008 Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

82

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

BRKSAN-1701 14693_05_2008_X1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr