SAN and FC Fundamentals

SAN and FC Fundamentals
Fibre Channel Technologies

Fibre Channel
Fibre Channel is currently the technique for the realisation of storage networks. Interestingly, Fibre Channel was originally developed as a backbone technology for the connection of LANs.
A high-performance interconnect and serial I/O protocol that delivers:

Reliability Throughput Distance flexibility Low rate of transmission errors Low delay (latency) of the transmitted data
19 March 2014

Brief Overview on SCSI
The Small Computer System Interface (SCSI) was for a long time the technology for I/O buses in Unix and PC servers. The first version of the SCSI standard was released in 1986. The SCSI protocol introduces SCSI IDs (sometimes also called target ID) and Logical Unit Numbers (LUN) for the addressing of devices. Each device in the SCSI bus must have an unambiguous ID, with the host bus adapter in the server requiring its own ID. Storage devices such as RAID disk subsystems or tape libraries can include several subdevices such as Tape Drives, Disks or Media Changer. which means that the ID's would be used up very quickly. Therefore, so LUN's were introduced in order to address sub-devices within larger devices. The OS must note three things Controller ID, SCSI ID and LUN.
3
19 March 2014

SCSI Addressing
19 March 2014

Brief Overview on SCSI
SCSI ID Priority: Originally, the SCSI protocol permitted only eight IDs, with the ID 7 having the highest priority. More recent versions of the SCSI protocol permit 16 different IDs. For reasons of compatibility the IDs 7 to 0 should retain the highest priority, so that the IDs 15 to 8 have a lower priority SCSI is only suitable for the realisation of storage networks to a limited degree. First, a SCSI daisy chain can only connect a very few devices with each other. The maximum lengths of SCSI buses greatly limit the construction of storage networks Techniques such as Fibre Channel SAN and iSCSI merely replace the SCSI bus by a network; the SCSI protocol is still used for communication over this network.
19 March 2014

Fibre Channel Layers
The function levels of the Fibre Channel standard define:
FC-0: Physical interface, transmission, signalling, cables/connectors FC-1: 8b/10b encode/decode, link control, ordered set specifications FC-2: Framing, flow control, classes, exchange/sequence management FC-3: Application-specific layer for encryption, compression, raid striping FC-4: Protocol mapping of existing protocols and native Fibre Channel protocols
19 March 2014

Fibre Channel Layers
19 March 2014

Fibre Channel Layers - FC-0 Physical Level
FC-0 deals with the physics of transmitting and receiving a signal at different transfer rates. Defines the physical link in the Fibre Channel system
Media type Connection Transceivers Electrical and optical characteristics
It also specifies the process of converting the parallel bus signals used by some hosts and targets to the serial signals used by Fibre Channel. This function is called serializerdeserializer, or SERDES. When there is a problem with a GBIC or fibre link, it is an FC-0 problem.
19 March 2014

Fibre Channel Layers - FC-1 Encoding Level
FC-1 specifies a standard for encoding data at the bit and byte levels: FC-1 defines how data is encoded before it is transmitted via a Fibre Channel cable (8b/10b encoding) 8b/10b encoding converts an eight-bit byte to be transmitted into a ten-bit character, which is sent via the medium instead of the eight-bit byte.
19 March 2014

Fibre Channel Layers - FC-1 Encoding Level
Ordered Sets
The Fibre Channel standard differentiates between two types of transmission word: data words and ordered sets Data words represent a sequence of four eight-bit data bytes. Data words may only stand between a Start-of-Frame delimiter (SOF delimiter) and an Endof-Frame delimiter (EOF delimiter). Ordered sets may only stand between an EOF delimiter and a SOF delimiter All ordered sets have in common that they begin with a certain transmission character, the so-called K28.5 character. The K28.5 character includes a special bit sequence that does not occur elsewhere in the data stream. Fibre Channel aggregates four ten-bit transmission characters to form a 40-bit transmission word.
10
19 March 2014

Fibre Channel Layers - FC-2 Signalling Protocol Level
FC-2 is the most comprehensive layer in the Fibre Channel protocol stack. It regulates the flow control that ensures that the transmitter only sends the data at a speed that the receiver can process it.
FC-2 specifies a structure for the data

The FC-2 layer controls the flow of data by initiating or preventing transfer of data frames (flow control), preventing collisions or overruns on the FC network. Information is passed down from the FC-4 layer and through FC-3 with a request for it to be transported Information is packaged into frames and passed down through layer FC-1 where it is converted into 10 bit characters and passed out on the link by FC-0.
Frames carry identification information:
Address information in frame header and information identifying how the network should serve, deliver and respond to this type of frame
19 March 2014 11

Fibre Channel Frame
The building blocks of Fibre Channel connection are the frames. The primary function of the fabric is to receive the frames from the source port and route them to the destination port. It is the responsibility of the FC-2 layer to break the data to be transmitted into frame size, and reassemble the frames.
19 March 2014
12

Fibre Channel Frame
Frames contain: The payload (the information to be transmitted) Addresses of the source and destination ports Link control information
Frames are broadly categorised as Data frames and Link control frames. Data frames May be used as Link_Data frames and Device_Data frames Link control frames Are classified as Acknowledge (ACK) and Link_Response (Busy and Reject) frames
19 March 2014
13

Fibre Channel Frame
Frames Contents: Each frame begins and ends with a frame delimiter (the ordered sets SOF and EOF) The frame header immediately follows the SOF delimiter. The frame header is used to control link applications, control device protocol transfers, and detect missing or out of order frames. An optional header may contain further link control information. A maximum-2112-byte-long data field contains the information to be transferred (payload) from a source N_Port to a destination N_Port. The 4-byte CRC precedes the EOF delimiter. The CRC is used to detect transmission errors.
19 March 2014
14

Fibre Channel Frame
Frames Header: The header of the frame contains: CTL Defines the type of frame, either data or control Source address Where the frame is starting from (PID) Destination address The destination (PID) of the frame Type FC-4 types; most common are SCSI (8) and IP (5) Seq_Cnt Each frame within a sequence is uniquely numbered with a sequence count Seq_ID Each sequence has its own unique identifier Exchange_ID An exchange is composed of one or more non-concurrent sequences for a single operation
19 March 2014
15

Fibre Channel Layers - FC-3 Common Services level
The FC-3 level of the Fibre Channel standard is intended to provide the common services required for advanced features such as: Services are functions that can span multiple ports, and can be applied to multiple protocols. Examples include data compression, encryption services, or multiplexing multiple links to form one virtual high-bandwidth link. The FC-3 standard is under definition at this time. In the meantime, the types of services that FC-3 was designed to implement have been implemented by vendors using proprietary protocols.
19 March 2014
16

Fibre Channel Layers - FC-4 Mapping
The highest level in the Fibre Channel structure, FC-4 defines the application interfaces that can execute over Fibre Channel. Allows multiple protocols to be transported over the same physical interface It specifies the mapping rules of upper layer protocols using the Fibre Channel levels below. Fibre Channel is equally adept at transporting both network and channel information and allows both protocol types to be concurrently transported over the same physical interface. FC-4 defines the rules these standard protocols must use to map to the lower layers: The mapping of the SCSI protocol to FC is called FCP The mapping of the IP protocol to FC is called IPFC (note that this is different than FCIP!) FC-4 makes sure that the ULP data or commands get broken down appropriately and packaged correctly into FC frames.
19 March 2014
17

Fibre Channel Classes of Services
To ensure efficient transmission of different traffic types, Fibre Channel defines specific service classes. Users select service classes based on the characteristics of their applications Class-1 is a connection-oriented circuit that dedicates 100% of the bandwidth between the sending and receiving ports with acknowledgement. Class-2 is a connectionless class with an acknowledgement. No bandwidth is allocated or guaranteed. IP uses this class. Class-3 is a connectionless class with out an acknowledgement. No bandwidth is allocated of guaranteed. FCP uses this class. Class-4 is a connection-oriented class that uses virtual circuits and confirmation of delivery. Unlike Class-1 that reserves the entire bandwidth, Class-4 can allocate a requested amount of bandwidth. Class-6 is a variation of Class-1 that provides a one-to-many multicast service with a confirmation of delivery. Avionics uses this class.
2014 Class-F is a connectionless class with acknowledgements between two switches. 19 March
18

Fibre Channel Cables
Fibre Channel signals can run over both copper and glass fibre media. Longer distances can be achieved with glass fibre than with copper; however, copper is less expensive. Copper
Copper
Video cable
Miniature cable Shielded twisted pair (the most common)
Optical (Glass Fibre
Optical (glass fibre) 62.5 micron multi-mode 50 micron multi-mode 9 micron single-mode
19 March 2014
19

Fibre Channel Cables - Single-Mode Fibre
Highest bandwidth and lowest performance loss One coherent stream of light travels a single path Long-wave lasers Single-mode, Step-index fibre
9 Micron Diameter Cladding 125 Micron
Core
Single-Mode
Single-mode fibre supports distances up to 10Km Single-mode fibre supports speeds of tens of gigabits per second and can carry many gigabit channels simultaneously. Each channel carries a different wavelength of light without any interference. The preferred medium for long-distance telecommunications is Single-Mode Step-Index Fibre
20
19 March 2014

Fibre Channel Cables - Multi-Mode Fibre
50/62.5 Micron Diameter
Multi-mode uses a short wave laser to emit many different light modes. These reflect off the cable cladding at different angles, which causes dispersion. This dispersion reduces the total distance from which the original signal can be reclaimed. Multi-mode has a larger core than single-mode.
Core
Cladding 125 Micron
Multimode
The larger the core, the greater the dispersion factor and thus the reduction in distance. Two types of Multi-Mode Fibre are Multi-mode step-index fibre and Multi-mode graded-index fibre. Multi-mode step-index fibre: Is Inexpensive and Decreases bandwidth and distance. Is seldom used in networking and data communications.
Multi-mode graded-index fibre: Increases bandwidth and distance and is expensive than stepindex fibre. Is frequently used in networking and data communications. 19 March 2014 21

Optical core: Single mode fibre = 9 microns Multimode fibre = 50 or 62.5 microns
Fibre cladding: 125 microns diameter
Fibre coating: 250 microns diameter
External jacket:
2.5 mm diameter
19 March 2014
22

Fibre Channel Connectors
SC connector: Most common optical fibre connector used for 1Gb/s Fibre Channel Comes in both simplex (one connector that contains both TX and RX fibres) and duplex (one connector for each fibre)
LC connector: Lucent Connector (LC), manufactured by Lucent Technologies Commonly used for 2Gb/s Fibre Channel with SFP transceivers
Smaller form factor allows greater density of ports and has two alignment pins to more securely align the fibre 19 March 2014
23

Fibre Channel Transceivers
Gigabit Interface Converters (GBIC'S): Inserted between the physical port (such as one found on an HBA or a switch) and the media Convert between optical and electrical signalling, or, if electrical cabling is used, they simply allow multiple connector types to be used with the same device IBM was one of the first to manufacture the GBIC Internal interface is electrical serial External interface is optical or electrical serial
The serial-to-parallel signalling (serializer-deserializer, or SERDES) function is performed by the device (the HBA, array controller, hub, or switch).
19 March 2014
24

Fibre Channel Transceivers
Small form-factor pluggable (SFP) transceivers: Used with 2Gb Fibre Channel components Architecturally similar to GBIC'S Smaller form factor supports higher port density SFF is same form factor, but fixed and not pluggable
19 March 2014
25

Attenuation It is the loss of power as a signal travels over a distance and is specified in decibels per kilometre (dB/km). Attenuation is lessened with higher-quality, more expensive, single mode fibres and is greater with lower quality, less expensive, multimode fibres Attenuation can result from: Light absorption caused by material impurities Light scattering caused by material impurities or by the defects at the core/cladding interface, and by the scattering of the molecules of the medium (silica) Macro bends (cable bends beyond the specified radius) Micro bends (cable wrapping or squeezing) Scattering and reflection at cable splices
19 March 2014
26

Dispersion The degree of scattering of the light beam as the light beam travels along the fibre optic cable. Types of dispersion are:
Scattering Loss of a light signal caused by microscopic impurities of the material

Chromatic dispersion Loss of light signal caused by different wavelengths travelling at different speeds Modal dispersion Loss of light signal caused by different light rays travelling different path lengths within the fibre
19 March 2014
27

Micro-bending Micro-bending means microscopic curvatures in Fibre Channel cable, caused by bending the cable beyond the minimum bend radius. Micro-bends create spatial wavelength displacements of a few millimetres This causes scattering and transmission loss. The minimum bend radius is of particular importance in the handling of fibre optic cables. The minimum bend radius for a Fibre Channel cable should be at least 3cm. If the cabling is bent beyond 3cm, the cladding can crack or break.
19 March 2014
28

Macro-bending Macro-bending is the physical bending of the fibre cable past the specified radius. The specified radius is the radius of the drum or mandrel that the optical fibre or cable ships wrapped around. As the fibre exceeds the specified radius, the light loses some particles and attenuation increases.
19 March 2014
29

Fibre Channel over IP
FCIP encapsulates Fibre Channel frames into IP packets and tunnels them through an existing IP network infrastructure to transparently connect two or more SAN fabrics together. FCIP gateways perform the Fibre Channel encapsulation process into IP packets and reverse that process at the other end. Fibre Channel switches connect to the FCIP gateways through an E_port for SAN fabric extension to remote locations. A tunnel connection is set up through the existing IP network routers and switches across a LAN/WAN/MAN.
19 March 2014
30

SAN Topologies
Point-to-point topology Point-to-point topology involves two nodes (N_port's) with a dedicated connection between them. The transmitter of each N_Port is linked to the receiver of the other N_Port. The two N_port's perform an initial port login to assign N_Port addresses. Then a persistent connection is established for transactions. This topology can effectively transfer data at full bandwidth Users cannot add devices to an existing point-to-point configuration. Separate point-to-point configurations must be created for each new storage device, requiring a separate host bus adapter (HBA) for each one.
19 March 2014
31

SAN Topologies
Fibre Channel Arbitrated Loop Fibre Channel arbitrated loop (FC-AL) topology is a serial, full-duplex data transfer architecture for high-performance storage systems. A server is connected to the I/O devices through a Fibre Channel loop. Each port on the loop has a transmit line (TX) and a receive line (RX) Only one port at a time can transmit data, so the bandwidth is divided among all devices on the loop. Before transmitting data, a port must participate with all other ports in the loop in an arbitration. The arbitration logic is distributed among all ports of a loop. Arbitrated loop is not a simple token-passing scheme. When a device is ready to transmit data, it first must arbitrate and gain control of the loop
19 March 2014
32

SAN Topologies
Switched Fabric A network of switches in a Fibre Channel environment is referred to as a fabric A Fibre Channel fabric is designed as a generic interface between a node and the physical layer. Fibre Channel nodes can communicate over the fabric with other nodes.
Frames are routed through various switches, which can be connected to multiple ports. When a frame arrives at each fabric element, the fabric element reads the destination address identifier in the frame. Ports on one node communicate with ports on other nodes connected to the same fabric.
Within a fabric topology, many connections can be active at the same time. The any-toany connection service and peer-to-peer communication service provided by a fabric are fundamental to Fibre Channel architecture.
19 March 2014
33

Fibre Channel Terminology
Nodes: Servers and storage devices are called nodes Nodes present the data to the Fibre Channel network. The node does not know whether it is attached to a SCSI bus or a Fibre Channel infrastructure.
Switches and hubs are not nodes but are interconnect devices through which the data passes.
Ports: Ports connect devices to the Fibre Channel network Ports transmit and receive data to and from the network.
Ports are found in nodes, switches and bridges/routers.
Links: Fibre Channel links Carry Fibre Channel transmissions from the transmitting port to the receiving port Include cables, connectors, and optical-electrical transceivers
34
19 March 2014

World-Wide Name (WWN)
Each HBA or Fibre Channel interface has a fixed 64-bit World-Wide Name (WWN) assigned by a recognised naming authority, and regulated by the Institute of Electrical and Electronics Engineers (IEEE) Not used for routing traffic across network The WWN is unique worldwide. It is assigned for the life of a connection device. World-Wide Node Name (WWNN): Uniquely identifies entire device (node) or may be unique per Port on each nodal device.
World-Wide Port Name (WWPN): Uniquely identifies each port in that device Used to facilitate services such as routing and zoning
19 March 2014
35

Fabric Addressing
When a node attaches to the fabric it must receive a unique 24-bit address. The network address is a three byte address based upon the Domain ID, the Area ID and, if a loop device, its AL_PA. This address is the source address and is used for routing data thru the fabric from one device to another.
The 24-bit address scheme also removes the overhead of manual administration of addresses by allowing the topology itself to assign addresses.
19 March 2014
36

Fabric Addressing
Port Address: A 24-bit port address consists of three parts: Domain (bits from 23 to 16) This is the address of the switch itself and is the most significant byte of the port address. One byte allows up to 256 possible addresses. Because some of these are reserved (such as the one for broadcast), there are only 239 addresses actually available. You can have as many as 239 interconnected switches in your SAN environment, and the domain number allows each switch to have a unique identifier. Area (bits from 15 to 08) Identifies the individual FL_Port's supporting loops, or it can be used as the identifier for a group of F_port's. The area field provides 256 addresses. Port or arbitrated loop physical address (bits from 07 to 00) The final part of the address provides 126 possible values (+ 00 for the switch). A simple calculation (domain x area x ports) determines the number of available addresses. This means that there are 239 x 256 x 127 = 7,770,368 addresses available, including the switch FL_port or 239 x 256 x 126 for actual devices.
19 March 2014
37

Fibre Channel Port Types
Device Ports (Nx_Ports) N_Port - Node Port, a Fabric device directly attached NL_Port - Node Loop Port, a device attached to a loop
Switch Ports U_Port - Universal Port, a port waiting to become another port type F_Port - Fabric Port, a port to which an N_Port attaches FL_Port - Fabric Loop Port, a port to which a loop attaches
E_Port - Expansion Port, a port used for inter-switch links (ISL's)

G_Port - Generic Port, a port waiting to be an E_Port or F_Port
19 March 2014
38

NL_Port: NL_Port is the node connection pertaining to hosts or storage devices in an arbitrated loop topology. Also known as Node Loop port. FL_port: FL_port is the fabric connection in a public loop for an arbitrated loop topology. It is connected to a switch. Also known as Fabric Loop port. N_Port: N_Port is the node connection pertaining to hosts or storage devices in a point-topoint or switched fabric topology. Also known as Node port. F_port: F_port is the fabric connection on the switch side of a switched fabric topology. Also known as Fabric port. E_port: E_port is the connection between 2 fibre channel switches. Also known as Expansion port. When 2 E_port's between 2 switches form a link, that link is referred to as ISL. TE_Port: TE_Port is the term used for multiple E_port's trunked together to create high bandwidth between switches. Also known as Trunking Expansion port.
19 March 2014
39

G_Port: G_Port is the connection that can be classified or emulated as F port or E port. It has no function until it has logged into a fabric. Also known as Generic port. U_Port: Universal port (U_Port) is one that has not yet assumed a specific function in the fabric. It is a generic switch port that can operate as an E_Port, F_Port, or as a FL_Port.
Node Types and Port Types
Nodes that attach to the fabric can either be an N (Node) or NL (Node Loop). NL nodes can be either private or public Private NL nodes do not do a fabric login and require additional assistance to be seen in the fabric. Public NL nodes can communicate with any member of the same loop and have the ability to send a frame to the fabric. Fabric Nodes (N) can communicate with any other Fabric Node and can communicate with private and public NL nodes on a loop.
40
19 March 2014

Fabric Initialisation Process
19 March 2014
41

Fabric Login Sequence
19 March 2014
42

Fabric Login Sequence: Fabric Login (FLOGI)
A node (device which can be an Initiator or Target) performs a fabric login (FLOGI) request to join a fabric and receive its 24-bit address A device does a Fabric Login (FLOGI) to determine if a fabric is present and, if so, exchange service parameters with the fabric. A successful Fabric Login sends back the 24-bit address for the device in the fabric. The device must do a FLOGI before communicating with other devices in the fabric. Since the device does not know its 24-bit address until after it does the Fabric Login, the SID (Source ID) in header of the frame making the FLOGI request will be zeros (0x000000)
19 March 2014
43

N Port Login Sequence
19 March 2014
44

Fabric Login Sequence: N Port Login Sequence (PLOGI)
Before a node port can communicate with another node port, it must first perform N_Port login (PLOGI) with that node port. Similar to FLOGI, the process entails transmitting a PLOGI frame to the destination node port. Again, the normal response is an ACC frame. N_Port Login accomplishes the following: Provides a specific set of operating characteristics associated with the destination N_Port, including the classes of service that are supported Both Fabric Login and N_Port Login are intended to be long-lived. After it is logged in, a device can stay logged in indefinitely, even if it has no further data to transmit at that time.
19 March 2014
45

Loop Addressing
The AL_PA is assigned during the loop initialisation In a public loop, the switch has ALPA 00 and so becomes master; in a private loop the node with the lowest ALPA becomes master. The loop master then transmits frames to identify existing nodes and allow new nodes to obtain an ALPA/LOOP_ID. Maximum of 126 AL_PA's attached to FL_Port. AL_PA 00 reserved for FL_Port The lower the AL_PA, the higher the priority. Arbitrated loop topology can have a maximum of 127 devices per loop.
19 March 2014
46

Fabric Addressing
In a point-to-point connection, the port address is assigned by the port with the higher WWN Only the Port segment is used. One device takes the address 00000000 and the other device is assigned 00000001.
In a private loop (not connected to a switch) a public device will only use the 8-bit AL_PA A private loop is not connected to a switched fabric, so all N_port's can only communicate with each other based upon the ALPA which is assigned to each port during loop initialisation The Domain and Area segments are not used.
In a public loop (connected to a switch) a private device will only use the 8 bit ALPA:
A public loop is connected to a switched fabric, so ports which login with the Name Server are assigned a full 24 bit fabric address. 126 ALPA addresses are available to N_port's in the arbitrated loop.
19 March 2014
47

Fabric Addressing
239 Domains are available to the fabric Domains 01EF are available Domains 00 and F0FF are reserved
24-bit addressing Provides for 16,777,216 potential addresses per fabric Several blocks of addresses are reserved for Fibre Channel functions, so the number of addresses available to connected ports is reduced to 15,663,104 (239 x 256 x 256)
19 March 2014
48

Fibre Channel Routing
The Fabric Shortest Path First (FSPF) protocol was developed to address routing path issues in Fibre Channel fabrics: The FSPF algorithm is a cost-based routing algorithm. It computes the shortest path between two connected nodes by assigning a cost to each path. Every path has a cost equal to the number of hops between switches
This algorithm avoids looping of frames. In addition, all frames in an exchange follow the same path, ensuring in-order delivery.
The FSPF protocol supports load sharing, in which multiple paths through the fabric can be used, but it does not support true load balancing:
Load sharing simply means that multiple paths can be used. Load balancing means that actual traffic load is balanced across multiple paths. FSPF supports only load sharing.
FSPF chooses the least-cost path based only on the number of hops between nodes. In other words, an unused path with 2 hops will be disregarded in favour of an overutilized path with 1 hop.
19 March 2014 49

Fibre Channel Routing
Paths are recomputed only when the physical ISL configuration changes. FSPF does not recompute paths based on traffic load. All of this is according to the FC specification. However, some switch vendors do implement load-balancing using proprietary protocols. FSPF default link costs for Brocade SAN Switches 1000 at 1 Gbit/sec; 500 at 2 Gbit/sec; 500 at 4 Gbit/sec (Deliberately set to 500 under FOS 4.4)
19 March 2014
50

Fabric Topologies
There are a number of different configurations we can use to create a fabric. Here we consider the choices available, and which to choose for a particular application. Cascade Best suited for fabrics with low switch counts, or where there are islands of connectivity with little need for symmetrical or any-any bandwidth between islands.
Full Mesh (or Partial Mesh) Best suited to fabrics with low switch counts where there is a need for high inter-switch connectivity. However, this topology does not scale well beyond a few switches.
Core-Edge Best topology for very large fabrics, with high switch count.
In each of the topologies we usually assume a single or multiple ISL between switches.
51
19 March 2014

Fabric Topologies
There are a number of different configurations we can use to create a fabric. Here we consider the choices available, and which to choose for a particular application. Cascade Best suited for fabrics with low switch counts, or where there are islands of connectivity with little need for symmetrical or any-any bandwidth between islands.
Full Mesh (or Partial Mesh) Best suited to fabrics with low switch counts where there is a need for high inter-switch connectivity. However, this topology does not scale well beyond a few switches.
Core-Edge Best topology for very large fabrics, with high switch count.
In each of the topologies we usually assume a single or multiple ISL between switches.
52
19 March 2014

Fabric Topologies
Cascade Each switch is connected to its neighbour through a single ISL Each ISL becomes a single point of failure Any ISL failure will segment the fabric Can use multiple ISL'S for increased reliability
19 March 2014
53

Fabric Topologies
Cascade Ring Each switch is now connected to two adjacent switches If an ISL fails, the switch can use an alternate path An ISL failure will not segment the fabric Can use multiple ISL'S for increased performance
19 March 2014
54

Fabric Topologies
Full Mesh Every switch is connected to every other switch No single point of failure Large number of E_port's used to connect ISL'S (n-1) Smaller number of ports available for nodes Single hop count between any two nodes
19 March 2014
55

Fabric Topologies
Core-Edge fabric In a core-edge design, one or more core switches are used to connect to other edge switches, while the nodes are attached to the edge switches. This allows the edge switches to accommodate higher numbers of devices. Typically multiple core or cross-connect switches are used for redundancy.
Devices are connected to the edge switches

Core switches are used to interconnect E_port's Ensures a four-hop count from any device to any other device
19 March 2014
56

Fabric Topologies
Core-Edge Topology Benefits Easy scalability and enhanced performance If more devices need to be connected to the fabric, more edge switches can be added If more performance is needed, more cross-connect core switches can be added
Redundant core switches mean no single point of failure

Maintains a 1 or 2-hop count from any device to any other device. Efficient 19 March 2014 use of E_port's and ISL'S
57

VSAN's (Virtual SAN's)
Virtual SAN's (VSAN's) are a proprietary feature of Cisco MDS 9000 switches that allows a single SAN fabric to be partitioned into multiple logical SAN's. VSAN's allow redundant fabrics to be configured on a single physical infrastructure, eliminating the need to physically partition the SAN. VSAN's enable more robust security, availability, and scalability in FC fabrics.
VSAN's can be used to isolate traffic based on the unique requirements

Production and test environments in an enterprise network Low and high security requirements Backup and data replication traffic on separate VSAN's Dedicated resources and bandwidth for priority applications
19 March 2014
58
VSAN's provide the following features: Traffic isolation: All traffic, including multicast and broadcast traffic, is contained within VSAN boundaries. Each device resides in only one VSAN, ensuring absolute separation between user groups. Scalability: VSAN's are overlaid on top of a single physical SAN. The ability to create several logical VSAN layers increases the scalability of the SAN.
Per-VSAN fabric services: Each instance of a fabric service within each VSAN runs as a separate thread. Replication of fabric services on a per-VSAN basis provides increased scalability and availability.
Redundancy: Several VSAN's created on the same physical SAN ensure redundancy. If one VSAN fails, redundant protection is provided by a pre-configured backup path between the host and the switch. Ease of configuration: Nodes can be added, moved, or changed between VSAN's without changing the physical structure of a SAN. Moving a device from one VSAN to another only requires configuration at the port level, not at a physical level.
19 March 2014 59

VSAN's
The primary architectural difference between VSAN's and zones is that VSAN's are completely isolated from each other, while zones share some common traffic (such as fabric service messages) and share fabric services (such as the Domain Manager and Name Server). Each VSAN effectively functions like a separate SAN, with its own Domain Manager, Name Server, and so on. This means that a reconfiguration or fault in one VSAN will not affect traffic in any other VSAN. A port can belong to only one VSAN, but Cisco also offers a licensed feature called InterVSAN Routing (IVR) that allows ports to communicate with ports in other VSAN's. This provides the flexibility to isolate applications while still allowing resource sharing, such as multiple applications in individual VSAN's accessing a common tape library port.
19 March 2014
60

How VSAN works
Special Enhanced ISL (EISL) header is attached to every frame by ingress port Currently, only Cisco switches support EISL headers. EISL header contains 12-bit VSAN ID
To recognise and isolate the traffic travelling in a VSAN, each frame must be uniquely labelled with the VSAN ID of the VSAN to which it belongs. The labelling of frames is called tagging. Inside MDS switches, each frame is tagged with a VSAN ID that is stored in a special header called the Extended Inter-Switch Link (EISL) header. The same Fibre Channel IDs (FCID's) can be assigned to a host in another VSAN, thus increasing VSAN scalability.
Frames passed between MDS switches and end devices do not carry the EISL header, and neither do frames passing between MDS switches and non-MDS switches.
The 8-byte EISL header includes the following fields:
R_CTL (8 bits); Version (2 bits); Frame type (2 bits); User Priority (3 bits); VSAN-ID (12 bits); MPLS (1 bit); Reserved (4 bits); TTL (8 bits); Reserved (24 bits) 19 March 2014 61

Trunking and ISL in VSAN
Trunking E_port's (TE_Port's) have the following characteristics: TE_Port's can pass tagged frames belonging to multiple VSAN's. TE_Port's are only supported by Cisco MDS 9000 switches. By default, TE_Port's can pass all VSAN traffic (1-4093). The passing of traffic for specific VSAN's can be disabled. By default, E_port's are assigned as part of VSAN 1. TE_Port's allow for the segregation of SAN traffic and should not be confused with port aggregation (referred to by some vendors as trunking).
Enhanced ISL'S (EISL's) are ISL's that connect two TE_Port's:

An EISL is created when two TE_Port's are connected. EISL's offer a superset of ISL functionality, EISL's carry per-VSAN control protocol information.
19 March 2014
62

VSAN Configuration
VSAN membership is assigned on a port-by-port basis. Up to 1024 VSAN's can be configured on a physical SAN. VSAN's are identified by the VSAN ID, which can be one of the following: VSAN 1 is the default VSAN. VSAN 4094 is an isolated VSAN.
User-defined VSAN IDs range from 2 to 4093.
The default VSAN has the following characteristics: It is the VSAN to which ports are assigned until they are assigned to another VSAN. If no VSAN's are configured, all devices in the fabric are considered part of the default VSAN. It is the only VSAN enabled by default for switches in the Cisco MDS 9000 Family.
19 March 2014
63

VSAN Configuration
The isolated VSAN has the following characteristics: All ports in the isolated VSAN are disabled and cannot communicate with other ports. When a VSAN is deleted, all its member ports are transferred to the isolated VSAN. This avoids an implicit transfer of ports to the default VSAN or to another configured VSAN.
The geographic location of the switches and the attached devices is independent of their segmentation into logical VSAN's.
Within each VSAN, all members can talk to one another. Between VSAN's no communication is possible.
19 March 2014
64

LSAN's (Logical SAN's)
Brocade offers a proprietary solution called Logical SAN's, or LSAN's This feature allows traffic between devices that would otherwise be isolated in separate SAN's LSAN implementation requires purchase of at least one proprietary multi-protocol router. LSAN's allow customers to: Make efficient use of high-cost tape devices Derive value from legacy SAN equipment and configurations Retain the reliability and security of separate SAN's
Maintain separate address spaces and scalability of individual SAN's
19 March 2014
65

The LSAN feature can be used to: Connect SAN islands, similar to a VPN Allows specific devices to communicate across SAN fabrics Does not allow all devices in the two fabrics to communicate Does not merge the two fabrics Extend SAN's over long distances FCIP tunneling over IP WAN
Technology is implemented similarly to iFCP

Share SAN resources with iSCSI devices Multi-protocol router bridges FC and iSCSI protocols
19 March 2014 66

LSAN's use a Brocade multi-protocol router to: Join fabrics, in the sense of allowing devices in separate fabrics to communicate, without merging the fabrics. Create phantom domains (NR_Port's - Proprietary Brocade phantom port) to allow each fabric to retain its namespace and Domain ID. Perform Network Address Translation (NAT) to join the separate address spaces for each edge fabric or separate SAN fabric. Create an Inter-Fabric Link (IFL), an E_Port-to-EX_Port connection. Perform functions similar to an iFCP gateway when tunneling FC over IP. EX_Port:The type of E_Port used to connect a Multi-protocol Router to an edge fabric.An
EX_Port follows standard E_Port protocols and supports FC-NAT but does not allow fabric merging across EX_Port's
19 March 2014 67

LSAN vs VSAN
VSAN's work well in enterprise environments, because they can be used to: Partition large SAN's Isolate traffic per port, on a large, multi-port switch
Create separate address spaces to enhance scalability and reliability
LSAN's work well in small and medium sized businesses, because they can be used to: Connect SAN islands Share costly SAN storage devices Provide a connection to a remote disaster-recovery facility Preserve value of legacy switches
19 March 2014
68

Multipathing
Multipathing drivers provide three basic features: Failover. This includes detection of link failure and automatic rerouting to alternate path. Different HBA drivers provide different levels of transparency to the OS and application. Load-balancing. Some multipathing drivers allow you to manually determine which LUN's are accessed over which HBA ports, while others provide automatic path selection capability. More advanced multipathing software provides actual load-balancing-dynamic levelling of traffic flows. Administration. Multipathing software can provide intelligent path selection capability, dynamic configuration, and centralised administration of multiple hosts from one console.
19 March 2014
69

Microsoft Multipath I/O (MPIO)
New MPIO features include a Device Specific Module (DSM) designed to work with storage arrays. The Microsoft DSM provides the following load balancing policies. Load balance policies are generally dependent on the controller model of the storage array. Features of the included DSM are: Failover - No load balancing is performed. The application specifies a primary path and a set of standby paths. The primary path is used for processing device requests. If the primary path fails, one of the standby paths is used. Standby paths must be listed in decreasing order of preference (the most preferred path first). Failback - Failback is the ability to dedicate I/O to a preferred path whenever it is functioning. If the preferred path fails, I/O is directed to an alternate path until function is restored to the preferred path, but I/O automatically switches back to the preferred path when function is restored. Round-robin - The DSM uses all available paths for I/O in a balanced, round-robin fashion.
19 March 2014
70

Microsoft Multipath I/O (MPIO)
Features of the included DSM are: Round-robin with a subset of paths - The application specifies a set of paths to be used in a round-robin fashion, and a set of standby paths. The DSM uses paths from primary pool of paths for processing requests, as long as at least one of the paths is available. The DSM uses a standby path only when all primary paths fail. Standby paths must be listed in decreasing order of preference (most preferred path first). If one or more of the primary paths become available, DSM uses the standby paths in their order of preference. Dynamic Least Queue Depth - The DSM routes I/O to the path with the least number of outstanding requests.
Weighted Path - The application assigns weights to each path; the weight indicates the relative priority of a given path. The larger the number, the lower the priority. The DSM chooses the path that has the least weight from among the available paths.
19 March 2014
71

LUN Mapping and LUN Masking
LUN Mapping - LUN mapping allows administrators to bind LUN's to hosts: On the array, each LUN is assigned a unique WWN. LUN mapping is typically managed and enforced either by the HBA firmware or by the HBA driver.
LUN mapping controls that storage resources appear consistently to the host.
LUN Masking - LUN masking controls the visibility of LUN's to hosts: The SAN administrator determines which hosts are allowed to see which LUN's Prevents hosts from accessing volumes that do not belong to them Particularly important when multiple HBA's are configured in one host: Prevents the host from seeing multiple copies of each LUN
19 March 2014
72

SAN and FC Fundamentals

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SAN and FC Fundamentals

Uploaded by

Copyright:

Available Formats

SAN and FC Fundamentals

Fibre Channel Technologies

A high-performance interconnect and serial I/O protocol that delivers:

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

The function levels of the Fibre Channel standard define:

Fibre Channel Technologies

Fibre Channel Technologies

Media type Connection Transceivers Electrical and optical characteristics

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

FC-2 specifies a structure for the data

Frames carry identification information:

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Optical (Glass Fibre

Fibre Channel Technologies

9 Micron Diameter Cladding 125 Micron

Fibre Channel Technologies

50/62.5 Micron Diameter

Cladding 125 Micron

Fibre Channel Technologies

Fibre cladding: 125 microns diameter

Fibre coating: 250 microns diameter

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Scattering Loss of a light signal caused by microscopic impurities of the material

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Ports are found in nodes, switches and bridges/routers.

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

E_Port - Expansion Port, a port used for inter-switch links (ISL's)

Fibre Channel Technologies

Fibre Channel Technologies

Node Types and Port Types

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies

Fibre Channel Technologies