Brocade Product Training

Fibre Channel Layers and Related Components

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Course Objectives
After completing this module, attendees should be able to:
Identify Link Link

FC protocol layers

components to FC layers functions to FC layers

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Topics
Ethernet FC

and FC layer relationships

protocol layers
FC 0 Speeds and Feeds FC 1 8b/10b Encoding FC 2 Data Delivery FC 3 Common Services FC 4 Upper Level Protocols (ULPs)

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Ethernet and FC Layer Comparison

TCP/ IP

Ether

net

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

A look at network protocol layers Ethernet is the most widely used LAN technology today. Originally conceived and developed by Xerox Corporation, it is specified in the IEEE 802.3 standard. Since its inception, initial speeds of 10 Mbit/sec have evolved into Fast Ethernet (100BASET), providing transmission speeds up to 100 Mbit/sec. Gigabit Ethernet provides an even higher level of backbone support at 1000 Mbit/sec primarily on fiber optic cable. In the diagram, we illustrate how Ethernet utilizes the OSI model. Although it looks like Ethernet provides solutions for layers one through four of the OSI model, in fact, Ethernet by itself only covers the two bottom layers. Ethernet is, in general, accessed through TCP/IP (or UDP/IP) protocol stack, where the SAN is accessed through a simple SCSI protocol stack with fewer overheads on the server processor. Most Ethernet adapters work at the packet level. All higher-level segmentation and reassembly into IP datagrams, or TCP-level sockets are software driven and require server CPU intervention. Be aware that TCP/IP often drops packets of data when the network becomes congested. When this happens, the packet must then be retransmitted using more bandwidth. Footnote 1: FC-2 has elements of OSI layer four, so it is not a perfect one-to-one correlation between the OSI and the Fibre Channel layers. Overall, it is a modular architecture, 5 layers (FC-0 to FC-4) and each level does not define physical or programming interfaces between the levels.

Multiple Protocols
On Common Fibre Channel Transport

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC - 0 and 1 layers specify physical and data link functions needed to physically send from one port to another. FC 0 specifications include information about media connections and cables, it is sometimes referred to at the feeds and speeds layer. 10 Mbaud speed, although not depicted, is also in development. FC 1 layer contains specifications for 8b/10b encoding, ordered set and link control communication functions. FC 2 specify content and structure of information along with how to control and manage information delivery. This layer contains basic rules needed for sending data across network. This includes: (1) how to divide the data into smaller frames, (2) how much data should be sent at one time before sending more (flow control), and (3) where the frame should go. It also includes Classes of Services, which define different implementations that can be selected depending on the application. FC 3 defines advanced features such as striping (to transmit one data unit across multiple links) and multicast (to transmit a single transmission to multiple destinations) and hunt group (mapping multiple ports to a single node). So, while FC-2 level concerns itself with the definition of functions with a single port. The FC-3 level deals with functions that span multiple ports. There are parallel link service functions that many think reside at this layer because they logically fit. These link service functions utilize all FC-2 and lower services like an ULP but are not ULPs neither but neither do they reside at FC-3. I think of them as performing FC-4 like functions in a category parallel to FC-3. FC 4 provides mapping of Fibre Channel capabilities to pre-existing protocols, such as IP or SCSI, or ATM, etc.

FC-0 Layer: Media Connectors and Cables

Serial interface (separate one-bit transmit and receive lines) Media types: Cables Fiber Optic Cables Single Mode Multi Mode Connectors Fiber and Electrical Cable specifications

FC-0 and FC-1 combined are referred to as FC signaling interface

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-0: Physical Interface The lowest level, FC-0, specifies the physical link. One big purpose of Fibre Channel is to have the selected protocol operate over various physical media and data rates. This approach ensures maximum flexibility, allowing existing cable plants and a number of different technologies to be used to meet a wide variety of system requirements. Common cables are either copper or optics. Single Mode optical fiber is used between buildings or sites because of its long distance data transmission capabilities. Multi Mode Fiber (MMF) is used within a building between nodes and switches or between floors. Optical connectors are used for interconnection between devices such as nodes, fibre channel switches and hubs. These connectors include SFP (Small Form Pluggable), GBIC (Gigabit Interface Connector) and MIA (Media Interface Adapter). Serial data received from FC-1 is converted to a signal type associated with the transmission media and sent out transmission port. Transmission continues as long as link remains in OFC Active state also called Open-Fiber State.

FC-0 Fiber Optic Cables

Glass Core Glass Cladding Coating Glass Core Glass Cladding Coating

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Fibre Channel links are driven optically or electronically. The optical and electrical links can be combined in a single system when there is a Fabric or other media-type converter available. The active part of the optical cable is constructed out of the core (the optical conduit), surrounded by cladding (to keep the light in the core), and fiber coating wrapped around the cladding. The optical fiber is very thin and could easily become damaged. The fiber core can be either 9 micron, 50 micron or 62.5 micron. For comparison, a human hair is about 75 micron thick.

A ferrule

Optical FC links consists of two fibers; one for transmitting information, the other for fiber for receiving information. Each side TX connects to the other sides RX consisting of a single point-to-point connection. Using a switched fabric a nodes TX is attached to the Brocade ports RX and the Brocade TX is connected to the nodes RX.

FC-0 Optical Cable Links Supported

50/62.5 micron Multi Mode Fiber

780 nm Shortwave Lasers 850 nm Shortwave Lasers

Cladding (125 m)

Core (50/62.5 m)

9 micron Single Mode Fiber

1300 nm Longwave Laser

Cladding (125 m)

Core (9 m)

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Fiber types are available in two types: single mode or multimode. The smaller the glass core the greater the distance. Multimode offers a core of 50 or 62.5 micron in shorter distances and have a smaller cost base. Single-mode is used for high speed, long distance links, while Multimode is utilized in lower cost, intermediate links. (MMF) is used for short wavelength. It comes in 50 & 62.5 micron cores, with a cladding of 125 microns. The core is much larger (50/62.5 um) allowing for multiple modes and paths the light can follow. This propagation method is referred as Modal Dispersion. The distance under this propagation method is significantly reduced. Single-mode fiber (SMF) is used for long wavelength. It comes with a 9 micron core, and a cladding of 125 microns. 9 micron goes 2m 10km distances. The light travels along the same path since the diameter of the core is reduced (9 um) to such a degree it constrains the light.
In Single-mode fiber, the idea is to reduce the core size until the possibility of modal dispersion is reduced to such a degree that only one mode of photon flight is exhibited. By eliminating the chromatic dispersion (by utilizing monochromatic laser light sources), great distance and bandwidths become possible in single-mode fiber. Multi-mode propagation refers to the fact that the pulses of light can travel down the optical wave guide taking different reflective paths. Following are the prefix and measurement for the small numbers: Deci-meter (dm) = 10-1 of a meter Centi-meter (cm) = 10-2 of a meter Milli-meter (mm) = 10-3 (One thousandth) of a meter Micro-meter (m) = 10-6 (One millionth) of a meter ( - Greek word) Nano-meter (nm) = 10-9 (One billionth) of a meter Pico-meter (pm) = 10-12 (One trillionth) of a meter Femto-meter (fm) = 10-15 of a meter Atto-meter (am) = 10-18 of a meter

FC-0 Cables and Connector Distances

Fibre Channel does not necessarily mean just fiber cable Optical and Copper cables are the most common
Media Type Speed 100 MB/s 200 MB/s 100 MB/s 200 MB/s 100 MB/s 200 MB/s 100 MB/s .5 m 500 m .5 m 300 m .5 m 175 m .5 m 90 m 0 m 33 m Distance (m=meter) 10 km

9 micron Single-Mode Fibre (Long Distance) 50 micron Multi-Mode Fibre (Short Distance) 62.5 micron Multi-Mode Fibre (Short Distance) Electrical (Copper)

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Fibre Channel can be implemented using either fiber optic or copper cabling. Each has its own advantages and disadvantages. Fiber optic cables are more expensive but they give reliability, distance (up to 10 kilometers), and ease connectivity. They come in two kinds: single-mode (yielding greater distance) and multi-mode. Multi-mode is by far the most common today, typically providing 500 meters distance at 1Gbit/sec. Each optical patch in a fiber is called a mode. Fiber pairs come with easyto-use push-pull SC (Sieman Connector) or LC (Lucent Connector) connectors at the cable ends.

2Gbit/sec switches typically use the LC connectors. The 1Gbit/sec switches typically use an SC connector. The 2Gbit/sec switches use an LC connector. Both cables have white tips, called ferrules, that stick into the SFP (GBIC).

Copper cables are less expensive but suffer from more reliability problems and limited distance. They are available as Coaxial cable and Twisted Pair with connectors in DB-9 or High Speed Serial Data Connectors (HSSDCs). The HSSDC is a new connector designed for Fibre Channel, providing for a screw-less, easy to plug/unplug connection.

FC-0 Fiber Optical Cable Connections

The two most common optical connections used in FC involve LC SFPs and SC GBIC pairs LC (Lucent Connectors) plug into SFPs (Small Form Factor Pluggable) receptors

10

LC
SC (Siemens Connectors) plug into GBICs (Gigabit Interface Connector) receptors

SFP

SC

GBIC
Typically - 2Gbit/sec cables use LC connectors to SFPs while 1Gbit/sec fiber cables use SC connectors to GBICs
2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

The SC connector features a molded body and a push-pull locking system and was designed as a low cost alternative to the ST connector. It is utilized in both Multimode and Single mode hi-bandwidth applications. The LC connector, a small-form factor connector, features a ceramic ferrule and looks like a mini SC connector. SWL Fiber Optic GBIC/SFP Modules The SWL fiber optic GBIC/SFP modules, with an SC or LC connector (respectively), are based on short-wavelength 850 nm lasers supporting 1 or 2 Gbit/sec link speeds. This GBIC/SFP modules support 50/62.5 micron, multimode fiber optic cables 50/125 microns will go up to 500 meters in length @ 1 Gbit/sec and 300 meters in length @ 2 Gbit/sec. Fiber optic cables 62.5/125 microns will reach 175m @ 1 Gbit/sec and 90m @ 2 Gbit/sec. LWL Fiber Optic GBIC/ SFP Modules The LWL fiber optic GBIC/SFP modules, with SC or LC connector are based on long-wavelength 1300nm lasers supporting 1 or 2 Gbit/sec link speeds. This GBIC module supports 9 micron single-mode fiber optic cables up to 10 kilometers in length with a maximum of five splices.

FC-0 GBICs and SPFs

Copper GBIC Fiber SFP Fiber GBIC

11

Cable connection:

LC Connector

SC Connector

Passive Copper (Cu) - Up to 13M (1Gbit/sec) Active Copper (Cu) - Up to 33M (1Gbit/sec) Short wavelength (SWL) GBIC / SFP Long wavelength (LWL) GBIC / SFP Extended Fabrics - Up to 100K
2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Check OEM Compatibility matrices for a list of supported GBICs and SFPs. Brocade supported GBICs / current vendors include: Agilent, Finisar and IBM. Passive/Active Copper GBIC Module The Copper (Cu) GBIC module is based on the High-Speed Serial Data Connection (HSSDC) interface standards. The GBIC provides a female HSSDC connector. Passive Copper cables up to 13 meters and active up to 33 meters have currently been qualified, thereby supporting ANSI X3.230 FC-PH intracabinet requirements. Standard cables with HSSDC-to-DB9 male connectors are also available. SWL Fiber Optic GBIC/SFP Modules The SWL fiber optic GBIC/SFP modules, with an SC or LC connector (respectively), are based on short-wavelength 850 nm lasers supporting 1 or 2 Gbit/sec link speeds. This GBIC/SFP modules support 50/62.5 micron, multimode fiber optic cables 50/125 microns will go up to 500 meters in length @ 1 Gbit/sec and 300 meters in length @ 2 Gbit/sec. Fiber optic cables 62.5/125 microns will reach 175m @ 1 Gbit/sec and 90m @ 2 Gbit/sec. LWL Fiber Optic GBIC/ SFP Modules The LWL fiber optic GBIC/SFP modules, with SC or LC connector are based on long-wavelength 1300nm lasers supporting 1 or 2 Gbit/sec link speeds. This GBIC module supports 9 micron single-mode fiber optic cables up to 10 kilometers in length with a maximum of five splices.

Cables and Related Components

12

Types of Optical Cable(s) LC to LC SFP to SFP LC to SC SFP to GBIC SC to SC GBIC to GBIC Ethernet Port: RJ45 Serial Port: Straight 9-pin female-female D-sub cable Pins 2,3,5 are required

MIA

Media Interface Adapters (MIAs) are used to convert electrical signals to optical signals (DB9 to SC)

SC Coupler

LC Coupler

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-1 Layer:

13

8b/10B encode/ decode Link Control Ordered Sets

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-1: Byte Encoding This layer describes the 8-bit/10-bit transmission code that is used to provide balance of the transmitted bit stream. In addition, the coding provides a mechanism for detection of the transmission and reception errors. 8-bit/10-bit encoding scheme is selected for its superior transmission characteristics. This well-balanced code allows for low-cost component design and provides good transition density for easier clock recovery. Note: 8-bit/10-bit scheme is used in IBM ESCON as well. FC-1 level defines the method to encode data prior to transmission. As data is passed down in octet form, FC1 layer encodes it into a ten bit code group (20% overhead). Each octet is given a code group name according to the bit arrangement. This will guarantee certain characteristics about the information that gets sent across the serial link.

Syncing up summary -- Serial data comes into a receiver port with 7 bit comma patterns at the front of each ordered set character. These ordered set characters act as frame delimiters, primitive signals and primitive sequences. Receiver ports first bit sync (using these commas) and then count to 10 thus character syncing. They also count to 40 to word sync. Now frames can flow. If a loss of sync occurs a Loss-of-Sync procedure takes place (5 invalid transmission words receiver attempts to re-sync).

FC-1 Ordered Sets

Fibre Channel uses transmission word-ordered sets (four transmission characters) to perform control and signaling functions:
Transmission Word

14

Ordered Set (K28.5,Dxx.y,Dxx.y,Dxx.y)

Data Word (Dxx.y,Dxx.y,Dxx.y,Dxx.y)

Frame Delimiter SOF- Start of Frame EOF- End of Frame

Primitive Signal Fill Word IDLE ARB Non-Fill Word R_RDY VC_RDY CLS OPN

Primitive Sequence NOS - Not Operational OLS - Offline LR - Link Reset LRR - Link Reset Resp. LIP - Loop Initialization LPB - Loop Port Bypass LPE - Loop Port Enable

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Transmission words can be classified into two categories: 1. Words that begin with a special character are called ordered sets. Any character that starts with a K (ex: K28.5) is called a special character to indicate its usage as an ordered set. Ordered sets occur outside of the FC frame content and include frame delimiters, primitive sequences and signals. 2. Words that begin with an encoded data character are called data words. Data Words occur within the FC Frame content. Each transmission character in both the Ordered Sets and Data Words is 10 bits, so transmission words are 40-bit words. Ordered Sets (K28.5, Dxx.y, Dxx.y, Dxx.y) include: Frame Delimiters (SOF, EOF) to delimit, or identify start and end of frames Primitive Signals to indicate events at the sending port. Primitive Signals are used to indicate events or actions and are normally transmitted once.
Fill Word (Idle, ARB(X), ARB(F0), ARB(FF) transmitted on a link whenever a port is operational and has no other specific information to send Non-Fill Word (R_RDY, VC_RDY, CLS, OPN, DHD, MRK, SYNx,y,z)- signal the events

Undefined but valid ordered sets are treated as fill words (no explicit action in necessary. Primitive Sequences (NOS, OLS, LR, LRR, LIP) and (LPB, LPE) to control an optional port bypass circuit (for loop). Primitive Sequences are used to indicate states or conditions and are normally transmitted continuously until something causes the current state to change. At least 3 primitive sequences must be received before the appropriate response can be generated.

FC-1 - Link Control Protocols

Link control protocols use ordered sets for low-level link initialization and control link or arbitrated loop (AL) initialization, reset or failure:

15

Port State Machine (PSM) provides control signal specifications for Fabric connections Loop Port State Machine (LPSM) provides control signal specifications for arbitrated loop connections

Rx

Rx

Nx_Port A
Tx

Nx_Port B
Tx

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

There are four states of the Port State Machine (PSM) used for Fabric (G_Port) connections: 1. Active State (AC) 2. Link Reset State (LR) 3. Offline State (OLS) 4. Link Failure State (LF) Active state (AC) in the Active state, the port is able to transmit and receive frames and primitive signals. Certain conditions encountered within the port may cause the port to exit the active state and perform one of the following primitive sequence protocols: Link Reset, Offline state, or Link Failure. Link Reset state consists of: LR1 (Link Reset Transmit) LR2 (Link Reset Receive) LR3 (Link Reset Response) Offline state consists of : OL1 (Offline Transmit) OL2 (Offline Receive) OL3 (Wait for Offline) Link Failure state consists of: LF1 (No Operational Receive) LF2 (No Operational Transmit)

FC-2 Layer: Framing Protocol / Flow Control

16

Exchange and Sequence Management Frame Structure Class of Service Flow Control Buffer to Buffer End to End

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-2 defines the structure and organization of the information being delivered and how that delivery is controlled and managed. Exchange management is the mechanism that two fibre channel ports use to identify and assign an exchange ID number for a set of related information units. When the entire stream of data will fit in a single frame (2112 bytes) a single exchange id is created and a sequence number is assigned. However, when a stream of data will not fit into a single frame (2112 bytes), data is put into sequences of frames. Within the exchange ID sequence management is used to number the sequence segments in the stream of data. Sequence numbers associated with the exchange will be used at the recipient to re-order the sequence segments, to re-assemble as a contiguous stream of data. In other protocols, this is commonly known as fragmentation and re-assembly. Frame Structure has a start-of-frame delimiter ordered set and ends with an end-of-frame delimiter set. Flow control is the process to deliver a frame. When a frame is ready for transmission, it is sent thru the encoder (8b/10b), to the serializer (sfp/gbic) and transmitted to the receiver port where it is deserialized, decoded and stored in a receive buffer. The receiving port sends to the transmitting port a credit to send another frame and decrements a credit from the credit value established during the login session (buffer to buffer credit). When the receiving port moves the buffer to the next port, the debit is restored. Buffer credits regulate the flow of frames into and out of the fabric. When a N_port and a destination N_port communicate, an end-to-end credit is established. End-to-end credit is established between the pairs and is used to manage the flow of frames between a specific pair of N_ports and allows the receiving port to control which source N_ports are allowed to send frames to the receiver.

FC 2 Exchange, Sequence, Frame Management

Command 1 ULPs FC-4 Mapping Single Exchange Multiple Sequences Frames
CMD (Sequence)

17

Command 2
DATA IN (Sequence)

Command 3
STATUS (Sequence)

Information Unit

Information Unit

Information Unit

CMD

DATA IN

STATUS

CMD

DATA IN

STATUS

1
FC- 0 & 1

4 3

Port 2 1

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

With Fibre Channel, there are almost no limits on the size of transfers between applications. Whereas with Ethernet the software is sensitive to a maximum packet size that can be transmitted (1518 bytes). With Fibre Channel, frame sizes are transparent to the software because of a logical construct called a "sequence." A frame is not a unit of transfer but sequence is and it always maps with the Upper Layer Protocol (ULP) command instruction. Lets see how it works: First, all the commands coming from a ULP are mapped into logical constructs called "information units." An Individual information unit is generally mapped to a sequence. Related information units, such as those required in an I/O operation, are mapped as a single exchange. The sequence and exchange structures are general enough and contain tunable options concerning flow control and error recovery policy. Sequence - A sequence is a set of one or more related data frames transmitted for a single operation, flowing in the same direction on the link (unidirectional from one N_Port to another N_Port). The N_Port that transmits a sequence is referred to as the "sequence initiator," and the N_Port that receives the sequence is referred to as the "sequence recipient". A sequence is also the recovery boundary in Fibre Channel. When an error is detected, Fibre Channel identifies the sequence in error and allows that sequence to be retransmitted. Exchange - An exchange is composed of one or more non-concurrent, related sequences for a single higher-level operation. For example, an operation may consist of several phases: a command to read some data, followed by the data, then followed by the completion status of the operation. Each phase of the command, data, and status is a separate sequence, but they can form a single exchange. Within the single exchange, only one sequence direction can be active at a time, although sequences for different Exchanges may be concurrently active. (Fibre Channel multiplexing support). Sequence initiative is handed over before sequences flow in the other direction. Sequences going in one direction can be streamed. This just means that a 2nd sequence can be sent in that direction before waiting for delivery confirmation. Note: The four Exchange error policies are: (1) Abort, discard multiple sequences; (2) Abort, discard a single sequence; (3) Process with infinite buffering (this policy is a special design for specific transfers, such as video data); (4) Discard multiple sequences with immediate retransmission.

FC 2 Fibre Channel Frame Format

18

FRAME
S O F 4 HEADER PAYLOAD C E R O C F 4 4

24

Up to 2112 2148 Bytes

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

A frame has a header and may have a payload. The header contains control and addressing information associated with the frame. The payload contains the information being transported by the frame on behalf of the higher level service or FC-4 upper level protocol. There are many different payload formats, based on the protocol. The TYPE field (Word 2, bit 31- 24) tells which format to use. The routing control INFO bit (bit 27-24) determines how to interpret the payload.

Field Definitions Routing Control bits (R_CTL) are the first 8 bits of the header. They define the type of frame and its content or function. The first 4 bits (Bit 31-28) identifies the frame type. The 2nd four bits INFO bit (Bit 27-24) defines the contents of the frame or identify the function of the frame. Destination_ID (D_ID)- Port Identifier (PID) or 24 bit address of the recipient. It could also be a a well-known address like the Name Server FFFFFC more on this later. Class specific Control Field (CS_CTL) The control necessary for different classes of service. This field is always zero for classes 2 and 3 per the standards. Classes 1 and 4 use it. Brocade switches currently only use Classes 2,3, and F. If CS_CTL is something other than zero in a Brocade port log (a running log extracted from portions of the FC frame displayed with the portLogDump command), then it is a Brocade internal code called IU_Status Values. Source_ID (S_ID) - Port identifier (PID) or 24 bit address of the source. It could be a a wellknown address like the Name Server FFFFFC. Type identifies the protocol of the frame content for Data Frames (i.e FC_CT, FCP, IPFC) Frame Control (F_CTL) -This field contains miscellaneous control information regarding the frame such as who owns initiative, first frame of the Exchange, last frame of the Exchange, etc. Sequence ID (SEQ_ID) used to identify and track all of the frames within a sequence between a source and destination port pair. Data Field Control (DF_CTL) this field indicates if any optional headers are present at the beginning of the data field of the frame. Optional headers are used for information that may be required by some applications or protocol mappings. Sequence Count (SEQ_CNT) used to indicate the sequential order of frame transmission within a sequence or multiple consecutive sequences within the same exchange. This is a counter that increments as sequences of frames are transmitted. Originator_ID (OX_ID) Exchange ID assigned by the originator port Responder_ID (RX_ID) - Responder_ID, optionally assigned by the responder to the Exchange. Data Field/Payload The standards limit the size. The maximum size is 2112 bytes.

FC-2 Class of Service (COS)

Associates a set of attributes with sequences and the frames within the sequence to provide different delivery characteristics.

19

There are six classes of Fibre Channel service:

Class 1 connection oriented: port-to-port, with ack (acknowledge), guaranteed bandwidth, in-order delivery (IOD) A circuit-switching COS One One connectionless: end-to-end(EE_Credit)/ ack and buffer to buffer / receiver ready (BB_Credit/r_rdy)flow control connectionless: no ack, BB_Credit/r_rdy flow control only Errors are handled at higher level connection oriented with ack: Creates Virtual Circuits with dedicated fractional bandwidths. Bandwidth and latency handled by a Quality of Service parameter that is IOD guaranteed behaves like Class 1 with one many capabilities

Class 2* Class 3* Class 4

Class 6

Class F* connectionless: switch-to-switch, with ack and BB_Credit ready

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

The ability to differentiate Classes of Service characteristics will be needed when discussing Link and Flow control of frames The Classes of service with an asterisk (*) are the ones that Brocade switches currently support. Note 1 - There are other link control signals associated with Classes 1, 2, 4 and 6 such as ack, f_bsy (fabric busy), p_bsy (port busy), f_rjt (frame reject), lcr (link credit reset), nty (destination port engaged in a class-1 connection), and other control signals. Note 2 - Classes of service 2, 3 and F have no bandwidth, deterministic latency, or IOD guarantees. Other Fabric services can provide this capability. Additional Notes: Class 1 Circuit-switching means that when Fabric receives Start of Frame connect Class One (SOFc1) signal it starts establishing a connecting to destination port. There are variations of COS 1: Exclusive connection, Intermix (allows class 2 and 3 frame transmission when class 1 not in use) , Intermix Bandwidth Recovery (allows Fabric to hold class 1 frame one frame cycle and make up that frame cycle in idle cycles). Class F Based on a simplified Class 2 model It is used to communicate Fabric-related traffic on inter-switch links (ISLs).

FC-2 Common Classes of Service

20

Class 2/F - Connectionless with ACK Class 3 - Connectionless without ACK

Bidirectional multiplexed frame streams without dedicated connections

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Class 2 Service Multiplexes frames at the frame boundary Adaptive routing 9Each frame routed separately by Fabric 9If multiple routes supported, frames might be delivered out of order Confirmation of delivered frames Login with both N_Port and Fabric required Connectionless service with no turnaround delay to establish connection N_Port to N_Port flow control End-to-End (EE) delivery confirmation with ACK Buffer-to-Buffer (BB) Link level flow control Notification of frame delivery failure Class 3 Service Multiplexes frames at the frame boundary Adaptive routing 9Each frame routed separately by Fabric 9If multiple routes supported, frames might be delivered out of order Unconfirmed delivery of frames 9Datagram service Login with both N_Port and Fabric required Connectionless service with no turnaround delay to establish connection Buffer-to-Buffer (BB) Link level flow control ULP recovery from frame delivery failures Brocade supports Class 2 and 3.

Flow Control

21

Flow Control mechanisms

ACK

Acknowledgement

R_RDY Receiver_Ready

Flow control is related to COS Brocade switches support COS 2, 3 and F

COS 3 uses R_RDY (buffer to buffer or BB_Credit) flow control, each R_RDY received increments BB_Credit value COS 2 and F use R_RDY and ACK (end to end or EE_Credit) flow control, each ACK received increments EE_Credit value

Brocade switches use cut through routing read S_ID / D_ID pairs and then shoot out the wire

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Flow Control also includes mechanisms to communication busyness: F_BSY Fabric Busy F_RJT Fabric_Reject P_BSY N_PORT_BUSY P_RJT N_PORT_REJECT
Flow control definitions: Acknowledgement (ACK) is used to confirm successful frame delivery and used by class 2, F and not Class 3. ACK times are established with the RA_TOV value. The default time used by Brocade is 10 seconds. Receiver Ready (R_RDY) is used by the receiving port to signal it is ready to receive a new frame. Fabric Busy (F_BSY) occurs when the fabric is unable to deliver a frame. Fabric busy conditions are an unusual event. The time for frame delivery is determined by the Error Error Detection Time Out Value (E_D_TOV). The default time used by Brocade is 2 seconds Fabric_Reject (F_RJT) Used to signal that a frame could not be delivered. Typical reason is for an invalid S_ID N_Port_Busy (P_BSY) When a N_PORT is unable to accept a valid frame due to a busy condition, the receiving N_PORT will send a P_BSY to the sender of the frame. N_PORT_REJECT (P_RJT) Used to signal that a frame could not be delivered. Typical reason is for an ACK

FC-2 Buffer-to-Buffer (BB) Flow Control

22

Frames are moved from one buffer to another using Receiver Ready (R_Rdy) primitive signals Frame flow is always from the source buffer to the destination buffer Multiple intermediate buffers may be involved

Tx

data

Rx

Tx

data

Rx

N_Port A
Rx Tx

Fabric
Rx

N_Port B
Tx

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-2 BB and End to End (EE) Flow Control

23

Frame flow is controlled by the receiver as a back-pressure mechanism Flow control is dependent on class of service (COS), but most use BB_Credits

BB_Credit and length established during Fabric login BB_Credits are also exchanged during port login but ignored unless in a point-to- point topology (two devices, no switches)

End to end (EE_Credit) flow control is a originator device port to destination device port flow control mechanism

EE_Credits are establish during node port login ACKs increments EE_Credit value

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-2 Flow Control Cont.

24

Flow control is a mechanism to establish the maximum amount of data that can be sent at any one time A throttle-back mechanism to handle congestion without the need to discard frames Flow control considerations include:

Bandwidth Internal Host Buses Extended Distance

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Bandwidth Considerations - At high speeds more data buffers are required to achieve the same link utilization for a given distance. Internal Host Buses - Internal host Buses (like PCI) can become congested and thus throttle back receiving data from the Fabric Extended Distance - Over extended distances, more data buffers are required to maintain a high throughput

FC-2 Flow Control Cont.

Increasing E_Port BB_Credits (points 3 and 4) allows full bandwidth utilization of extended Fabrics at specified distance

25

Switch 1 ISL
Point 2 (F_Port)

Switch 2

Points 3&4 (E_Ports)

BB_credits are exchanged between points 1 and 2 during Fabric login and are used as throttle-back mechanisms BB_credits are also exchanged across E_Ports (points 3 and 4) and are used as both throttle-back and bandwidth utilization (performance) mechanisms
2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Point 1 (N_Port)

Increase E_Port buffer credits to optimize performance at required distance The latency of a laser beam going through a fiber is 5nsec/meter (light wave propagation) or 5sec/km
10km link round trip latency is 100sec

A 2KB frame propagation delay at 100MB/sec is 20sec. So to keep a 10km pipe full, the switch would need at least 5 E_Port buffers (credits) Proportionally, a 2KB frame at 50km needs 25 buffer credits
At high speeds more data buffers are required to achieve the same link utilization for a given distance

A 2KB frame propagation delay at 200MB/sec is 40sec and would need at least 10 E_Port buffers (credits) Proportionally, a 2KB frame at 50km needs 50 buffer credits

FC-2 Flow Control Summary

FC flow control mechanisms associated with the Classes of Service (COS) that Brocade switches support:
Feature
Flow control based on Busy/ reject response Seq_id assignment Source sequence complete Destination sequence complete

26

Class 2/ F
BB and EE (AWK) credit Yes, by Fabric or destination 0 to 255 Final ACK received w/ EOFt All data frames received and all ACKs sent

Class 3
BB credit No, frame discarded 0 to 255 Acc to ABTS; or ULP All data frames received and EOFt in last frame

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Note Due to limited space and non-use of COS 1 and 4 in Brocade SANs at this time, they will not be discussed in relation to flow control

FC-3 Layer Common Services

27

Largely unused: mostly theoretic Functions span multiple ports Common service advanced features: Striping Hunt groups

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

FC-3: Common Services This layer defines advanced features such as striping (to transmit one data unit across multiple links) and multicast (to transmit a single transmission to multiple destinations) and hunt group (mapping multiple ports to a single node). So, while FC-2 level concerns itself with the definition of functions with a single port. The FC-3 level deals with functions that span multiple ports. Fibre Channel provides internal protocols called Basic Link Services and Extended Link Services. These services would logically reside at FC-3 but are NOT part of the FC-3 standards. Basic link services include low-level functions that are transported as a single data frame within a sequence. Basic link service examples include: Abort sequence (ABTS) used to abort a sequence or a frame, BA_ACC or BA_RJT are both ABTS responses and respectively mean Basic link service accept or reject. Another basic link frame is called No operation (NOP) which is used to initiate or terminate connections and sequences. There is also a remove connection (RMC) class of service (COS) 1 frame that is used to request a COS 1 connections removal. Extended link services (ELS) perform upper level protocol like functions between two fabric ports, one of them is often a wellknown address like the fabric F_Port, fabric controller or name/directory server. Common ELS commands include FLOGI (Fabric login), PLOGI (port login), SCR (State Change Registration), RSCN (Registered State Change Notification), PRLI (Process login) and more.

FC-4 Upper Level Protocol (ULP) Mapping

28

SCSI IP WAN tunneling FICON

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

SCSI (SCSI-FCP or FCSI-201) - Transport of SCSI commands and data over the Fibre Channel protocol hierarchy is such a major part of the Fibre Channel usage that a particular acronym, FCP, is used to denote the SCSI over Fibre Channel, it is like the native protocol for Fibre Channel, so we called it Fibre Channel Protocol. IP(IETF Draft FC_IP or FCSI 202)- Maps the IP protocol over Fibre Channel WAN Tunneling (FC-BB)- defines mapping for ATM or LAN over FC Audio Visual (FC-AV)- Mapping for digital TV standard and MPEG (For example) over FC VIA (FC-VI)- Virtual Interface Architecture mapping, (i.e. clustering) FICON (FC-SB-2) - Mapping FICON over FC. The standard is called FC-SB-2 for Fibre Channel Single Byte Command 2.

Fibre Channel Protocol Mappings

Upper Level Protocols
(ULPs) Support existing protocols Transparent to OS No changes + New capabilities IPI - 3 Command Set Mapping (IPI-3 STD)

29

SCSI-3

IP

ATM

FC-4

SCSI - 3 Command Set Mapping

FC Link Encapsulation FC - LE

FC - ATM

FC-3 FC-2 FC-1 FC-0

Framing Protocol Encode / Decode Physical Variant

Common Services

Fibre Channel Physical & Signaling Interface ( FC- PH, FC-PH2, FC-PH3 )

FC - AL

FC - AL -2

8b/10b Encoding Copper, Optical

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

A large part of the work done in developing the Fibre Channel architecture has been concentrated in assuring that the architecture could efficiently and naturally operate as a transport or delivery mechanism for a wide variety of well-established Upper Layer Protocols, or ULPs. Since much of the investment in current Operating Systems is at the device driver level, the incremental cost in transferring systems over to a Fibre Channel data communication level decreases if the interfaces can be made as similar to previously-existing interfaces as possible. This allows new capabilities to be added with minimal changes to currently-available interface. The following protocols are currently support: Storage protocols - Small Computer System Interface (SCSI) command set - Single Byte command code set (SBCCS) - High Performance Parallel Interface (HIPPI) - Intelligent Peripheral Interface (IPI) Network protocols - IP (Internet Protocol) - ATM Adaptation Layer for Computer Data (AAL5) - IEEE 802.2 - Link Encapsulation (FC-LW) Cluster Server protocols - VI (Virtual Interface) FC-4, the ULP mappings, some of which are listed, isolate the ULPs from the underlying Fibre Channel fabric and define interoperability of processes within a ULP. This layer maps the ULP onto the Fibre Channel transport layers. Fibre Channel is equally adept at transporting both network and channel information and allows both protocol types to be concurrently transported over the same physical interface.

Summary

30

FC - 0 and 1 layers specify physical and data link functions needed to physically send from one port to another

FC-0 defines media connections and cables (speeds and feeds) FC-1 defines 8b/10b encoding, link control and ordered sets

FC 2 specify content and structure of information along with how to control and manage information delivery

FC-2 defines exchange and sequence management; frame structure, COS and flow control mechanisms

While FC-2 level concerns itself with the definition of functions with a single port, FC-3 level deals with functions that span multiple ports FC 4 provides mapping of Fibre Channel capabilities to pre-existing protocols, such as IP or SCSI, or ATM, etc.

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003

Additional Information

31

From the resources tab

Use

Internet Links to access SAN ED 101 SAN Fabric Foundation chapter 2 (Fibre Channel Essentials)

Also use the Fibre Channel Theory Fundamentals Resource and Reference Material selection under course syllabus SAN Glossary and FC Recommended Reading and Resource List

2003 Brocade Communications Systems, Incorporated. Revision0.1_FC101_2003