You are on page 1of 15

Fibre Channel Classes of Service

for Data Transport

Kumar Malavalli

kumar@brocadecomm.com
www.brocadecomm.com

Abstract
The Fibre Channel ANSI Standard defines multiple classes of service to provide different
modes of data transport within a Fibre Channel system. These Classes are independent of
topology. In this document, only Fibre Channel Class-1,-2, and -3 service are discussed. Each
service meets different data transfer and delivery requirements and suits various user
application environments. The choice of a particular class of service depends on user traffic
patterns and loads, the transport protocol used by the application, and performance and
configuration requirements. Brocade Communications Systems, Inc. (Brocade), designed
the SilkWorm switch based on Fibre Channel Class-2 and -3 service to build Fibre Channel
Fabrics. Influential factors in selecting these classes are the requirements for applications and
the data transport protocols used in today’s server-storage network.
 1997 Brocade Communications Systems, Inc.
ALL RIGHTS RESERVED

Order number: 0015-WP02-1

The Brocade logotype design is registered with the U.S. Patent and Trademark Office as a trademark of Brocade
Communications Systems, Incorporated. Brocade and SilkWorm are registered with the U.S. Patent and
Trademark Office as trademarks of Brocade Communications Systems, Incorporated.
All other brands, products, or service names are or may be trademarks or service marks of, and are used to
identify, products or services of their respective owners.
Notice: This document is for informational purposes only, and does not set forth any warranty, express or implied,
concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the
right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This
informational document describes features that may not be currently available. Contact a Brocade sales office for
information on feature and product availability.
Export of technical data contained in this document may require an export license from the United States
Government.

Brocade Communications Systems, Incorporated


1901 Guadalupe Parkway
San Jose, CA 95131
T 408.487.8000 info@brocadecomm.com
F 408.487.8101 http://www.brocadecomm.com
Fibre Channel Service Classes
In Fibre Channel the information interchange is structured hierarchically and common to all classes of
service. This hierarchical constructs are Frames, Sequences and Exchanges. A frame is the atomic unit
of data transmission between the communicating devices (N_Ports). A Sequence is a set of one or more
related data frames transmitted unidirectionally from one device to another device within an Exchange.
An Exchange is the basic construct consisting of a set of one or more related Sequences for
coordinating the information interchange between the communicating devices during upper layer
protocol such as SCSI and TCP/IP.
The Fibre Channel FC-2 construct, which serves as the data transport mechanism, defines the following
key mechanisms to achieve efficient information transfer, in different classes of service:
• A means to provide confirmation of delivery of valid frames at the destination device through
“Acknowledgment” frames. Depending on the application, one of the following methods of
“Acknowledgment” can be deployed:
1. ACK_1: Acknowledgment of every frame received
2. ACK_N: Acknowledgment of a group of ‘n’consecutive frames received
3. ACK_0: Acknowledgment of every sequence received
• A flow control handle to pace the flow of frames (i) end-to-end between devices (N_Ports) and (ii)
buffer-to-buffer (link level), between a device (N_Port) and the intervening Fabric (F_Port) to prevent
data overrun at the receiver. A “credit” based flow control is deployed. The “credit” is the number of
receive buffers allocated to a transmitting port (N_Port or F_Port). The following are the frame
messages to implement flow control for different classes of service
1. End-to-End:
ACK_1: For every frame delivered
ACK_N: For a group of ‘n’ consecutive frames delivered
2. Buffer-to-Buffer: R_RDY (Receiver Ready)
• An error handling means both between the device and the Fabric and end-to-end between the
communicating devices. This includes frame messages to indicate whether a frame has been busied or
rejected, with reason, either by the Fabric or the destination device as follows:
1. Link level:
F_BSY: Busied by the Fabric
F_RJT: Rejected by the Fabric with reason
2. End-to-End:
P_BSY: Busied by the destination device
P_RJT: Rejected by the destination device

Brocade Communications Systems, Inc. 1 Fibre Channel Classes of Service for Data Transport
Class-1 Service
Fibre Channel Class-1 service is connection-oriented and based on a circuit-switched mode of
operation, where switching is performed between connections. Class-1 provides a dedicated connection
with the full bandwidth guaranteed between two communicating devices (Figure 1).

A B C D E F
I G H
Device 1
Device 3

D E F A B C
G H I
Device 2 Device 4

Bidirectional frame streams on dedicated


connections in a Class-1 Fabric

Figure 1. A Class-1 Fibre Channel system

Before communication can begin, a bidirectional connection between the source and the destination
devices must be established by the intervening switch. Each frame or group of frames is acknowledged
for delivery or nondelivery to the destination device (Figure 2a). This delivery acknowledgment is also
used to manage end-to-end flow control from the source device to the destination device. In addition,
there is support for messaging to indicate whether a frame has been busied or rejected, with reason, by
the destination device once the connection is established
(Figure 2b). However, no flow control, error checking, or busy indication exist at link level between the
device and the switch.

DATA DATA

FABRIC FABRIC

Acknowledge Reject or
Source
Busy
Source Destination

Acknowledge: Flow
control between the devices
Figure 2a. Normal Class-1Transaction Figure 2b. Class-1 Error Handling

In Class-1 service, connection management within the switch must prevent a deadlock that might be
caused by a third device requesting connection with one of the two devices already in a connection. A
three-way unidirectional connection among three devices is thereby created. This is achieved by
ensuring that the connection is not confirmed until the originating device that requested the connection
receives an acknowledgement from the destination device on the return path, signifying connection
acceptance. Thus, every time a Class-1 connection is established, there is a penalty of round-trip
connect latency.

Brocade Communications Systems, Inc. 2 Fibre Channel Classes of Service for Data Transport
When a device sends a connection request to the Fabric, there is no guarantee that the connection will
be granted if the destination device is busy communicating with another device. The connection request
may be stacked (queued) within the Fabric until the destination device has terminated the connection
with the previous device and is free for a new connection. There may be other devices waiting in the
queue ahead of this device. Once a connection is established between two devices, no other device can
communicate with either of them.
In Class-1 there is a possibility of one device monopolizing a connection, thereby reducing the
bandwidth available to other devices. Hence, this class of service is not suitable if multiple devices
need to communicate with multiple other devices, such as in client-server configurations or when
storage is shared by multiple servers.
Class-1 builds a bidirectional circuit between two ports. Data frames and acknowledgement frames can
flow in both directions simultaneously. The ability to transmit multiple frames is controlled by the end-
to-end flow control rules. The higher the end-to-end credit, the more work can be done and better
utilization of the dedicated bandwidth. However with credit of one, normally each frame must be
acknowledged before sending the next frame.
The main problem with the Class-1 service is that a source device, with no Class-1 data ready wastes
the bandwidth on the dedicated connection with the destination device. To overcome this deficiency,
Class-1 service needs the assistance of the Intermix option. Intermix is a special feature of Class-1
service that allows interleaving of possible Class-2 or -3 data frames or high priority Class-2 and -3
control frames on an established Class-1 connection, one frame at a time, when the connection is idle.
To take advantage of this function, both source and destination devices engaged in the dedicated
connection, and the intervening Fabric must support the Intermix feature. However, implementation of
Intermix requires extra switch resources. In a Fabric of cascaded switches, tracking and managing the
Intermix frames at different switch stages is very difficult.

Brocade Communications Systems, Inc. 3 Fibre Channel Classes of Service for Data Transport
Class-2 Service
This service is connectionless and based on a frame-switched mode of operation where switching is
performed for each frame. In Class-2, frames can be multiplexed between a source device and one or
more destination devices (Figure 3).

1 2 3 A B
FABRIC

6
1 # 2
A 5 2

B
Device 1 A 3

A
Device 3

2
5 A

#
4
5
* *
#

4
6 4B 1
3 4 5
#
* 1 3
B 4
*
Device 4
Device 2
Bidirectional multiplexed frame streams with no dedicated
connection in Class-2 and -3 Fabric

Figure 3. A Class-2 and -3 Fibre Channel system

The Fabric examines the destination address of each frame before forwarding the frame on a path
toward the destination device. Flow-control support is provided for both end-to-end, between the
communicating devices, and at link level between the Fabric and communicating devices. Individual
frames or group of frames are acknowledged for successful delivery by the destination devices (Figure
4a). Also, to facilitate error handling, there is support for messaging to indicate whether a frame has
been busied or rejected, with reason, either by the Fabric or the destination device (Figure 4b). This
low-level flow-control and error-handling tools of Class-2 service offer high data reliability and
integrity, which is mandatory in business-critical applications.

DATA DATA

Ready FABRIC Ready Ready Ready


FABRIC

Source
Reject or
Source Acknowledge Destination Busy Reject or
Busy

Ready Ready Ready


Ready

Ready: Flow control between


the Fabric and the device
Acknowledge: Flow control
between devices
Figure 4a. Normal Class-2 transaction Figure 4b. Class-2 error handling

Brocade Communications Systems, Inc. 4 Fibre Channel Classes of Service for Data Transport
Multiplexing data to several destinations allows very efficient use of the link and Fabric bandwidth. In
server-storage network environments, the existing applications require multiplexing large amounts of
data with many small control (handshake) frames. The frame-switched communication that is offered
by Class-2 service facilitates this multiplexing very efficiently. The ability to multiplex a large number
of frames between the devices through the Fabric yields high bandwidth utilization. As a result, large
systems need fewer components, such as servers, to run the user applications leading to lower system
cost.
In a connectionless Fabric with a mesh topology of cascaded switches, each frame may be forwarded
via a different path. Hence, frame delivery to the destination device may not be in the same order as
transmitted by the source device (Figure 5a). The Brocade Fabric, on the other hand, guarantees in-
order delivery of frames by assigning a single path between the communicating devices during the path-
selection phase of the Fabric initialization (Figure 5b). In many large systems, the benefits of
multiplexing in realizing increased bandwidth utilization can occur only if in-order delivery by the
Fabric is guaranteed. This is because many large systems use virtual memory and a scatter-and-gather
mechanism to find out where the data payload can be placed in main storage. Without in-order delivery,
the scatter-and-gather list must be scanned for every frame, which is detrimental to performance.

FABRIC
FABRIC

Switch Switch Switch Switch


1 2 3 2 1 2 3

1
2
1

2
3

3
Switch
Sitch
1 2
1 3
3
2

Figure 5a. Out-of-order frames in a generic Class-2 and -3 Fabric Figure 5b. In-order frames in a Brocade Class-2 and -3 Fabric

Another advantage of the connectionless communication mode is the support for speed matching
between devices with different transmission speeds. By utilizing switch buffers, communication is
possible between today’s 1-gigabit-per-sec (Gb/s) device ports and future 2-Gb/s and 4-Gb/s device
ports. This feature protects today’s investment by establishing a good migration path.

Brocade Communications Systems, Inc. 5 Fibre Channel Classes of Service for Data Transport
Class-3 Service
Class-3 service is also a connectionless service based on frame switching. Unlike Class-2, it is a
datagram service, which does not provide confirmation of delivery or nondelivery of the frames at the
destination devices. The confirmation of delivery is delegated to the upper-level protocol, such as TCP
(Transmission Control Protocol), which manages its own end-to-end flow control and data integrity.
Only link-level flow control exists between the Fabric and each connected device (Figure 6). There is
no acknowledgment of a frame or notification that a frame has been rejected or busied by the Fabric or
by the destination device.

DATA
FABRIC
Ready
Source Ready Destination

Ready: Flow control


between the Fabric
and the device

Figure 6. Class-3 transaction

During heavy congestion in Class 3, the Fabric may occasionally discard some frames. However, the
lost frames can be recovered by retransmission of the sequence in which one or more frames have been
lost. The upper-level protocol (such as TCP) is responsible for this type of recovery. The prime
advantage of Class 3 for Multicast Service is precisely the absence of frame-delivery
acknowledgements, rejects, and busy indications from the destination devices. When the source device
sends frames to the members of a multicast group during a multicast transmission, confusion would
result if all the members were to send back acknowledgment frames. In this scenario, the bandwidth
available for transmission of user data would also decrease, resulting in reduction of the Fabric’s
aggregate throughput. The absence of acknowledgement frames in Class-3 also eliminates round-trip
delays during information transfers between devices.

Brocade Communications Systems, Inc. 6 Fibre Channel Classes of Service for Data Transport
Fabric Delays In Different Classes of Service
In Class-1, the connection set-up delay consists of a connection set-up time in each switch in the end-
to-end data path between the communicating devices and the time it takes for the return confirmation
message to indicate that the connection has been accepted by the destination device (Figure 7). For
transmission of small-to-medium sized data blocks, this delay constitutes a significant proportion of the
overhead. The delays associated with the queuing of “Connection Request” in the switches are not
considered.

Class-1 service is ideal for transferring very large amounts of data between two devices for a sustained
period of time. Class-1 is also suitable when a constant transfer latency is required, and when the
connection-establishment and tear-down latencies are significantly small in relation to the data transfer
time. For transfers of data blocks among multiple devices, as in storage and transaction applications, a
single device cannot afford connection latency. This leads to very poor link- and Fabric-bandwidth
utilization.

Connection
Source set-up time within Information Block
Device the switch
CR
Source
Switch

CR 1 2 3 4 CR: Connection
Frames
Requests
FC CC: Connection
Intermediate CC
Confirmation (Ack)
Switch
CA: Connection
Acceptance
CR
FC: Final
Confirmation (Ack)
Destination
Switch CA

Destination
Device TIME
Connection Set Up Delay Transmission Delay
Total Delay

Figure 7. Class-1 Fabric delays

In the absence of connection set up in Class-2 and -3, each frame is examined within each switch in the
data path between the communicating devices, for the address information for routing between the in-
bound port and the out-bound port of each switch. (Figure 8). The latency incurred due to this routing
decision time can be small if the cut-through routing method is employed, such as in the Brocade
Fabric. “Cut-through” routing allows routing of a frame to occur from the in-bound port of a switch to
the out-bound port of a switch as soon as the frame header with the address information becomes
available. The absence of connections in Class-2 and -3 Fabrics will eliminate the extra overhead for
connection set-up time thus increasing the bandwidth utilization and decreasing the total delay.

Brocade Communications Systems, Inc. 7 Fibre Channel Classes of Service for Data Transport
Source
Information Block
Device

Source
Switch Frames
1 2 3 4

Intermediate
Switch
1 2 3 4 FC: Final
FC
Confirmation (ACK)

Destination
Switch
Cut-Through 1 2 3 4
Frame Routing
Destination
TIME
Device
Total Delay (Class 3)

Total Delay (Class 2)

Figure 8. Class-2 and-3 Fabric delays

The Effect of Classes of Service on Cascading Switches in a Mesh Fabric


A Class-1 switch is not suitable for a mesh Fabric with cascaded switches. For example, Figure 9
illustrates the bottleneck created by an Inter-Switch Link (ISL) between two switches, engaged in a
connection between Devices 1 and 3. In this set-up, device 2 on one switch is blocked from
communicating with Device 4 on the other switch, even though both devices are free to communicate.

Ded Connectio
icat nless

Figure 9. Class-1 blocking in


Figure 10. Class-2 or -3
cascaded Fabric
non-blocking in

When a Class-2 or -3 switch is used with a mesh Fabric of cascaded switches, the ISLs are not a part of
any dedicated connection between the devices. Although the ISLs may experience some congestion, as
shown in Figure 10, there is no blockage of information flow to the free devices. Information from
Devices 1 and 2 can be multiplexed on the ISLs for delivery to Devices 3 and respectively 4, and vice
versa.

Brocade Communications Systems, Inc. 8 Fibre Channel Classes of Service for Data Transport
Brocade further reduces congestion on the ISLs by deploying a technique where the frames can be
segregated into separate frame flows or virtual channels (VCs) with a priority scheme. On a single ISL,
there are eight virtual channels available, with four priority levels and individual queuing mechanisms
for each VC. These virtual channels can be used to tune the Fabric for different applications by
prioritizing frame traffic and reducing congestion on the ISLs. Virtual Channels can separate data flow
between the following types of frames:
• Data or interswitch control and signaling frames

• Class-2 or -3 frames

• Unicast or multicast frames

The frames on the ISLs are subject to independent flow control between the adjacent switches per VC.

ISL
VC 1
VC 2

VC 3

VC 4
Brocade
Brocade
VC 5
Brocade Switch
VC 6 Brocade Switch

VC 7

VC 8

8 Virtual Channels (VC’s) with 4 priority levels

Figure 11. Congestion management between Brocade Switches

Brocade Communications Systems, Inc. 9 Fibre Channel Classes of Service for Data Transport
Fibre Channel Service Classes Summary
Table 1 summarizes the characteristics of Fibre Channel Class-1, 2, and 3 services. Table 2 summarizes
the advantages and disadvantages of each of these services for server-storage network applications.

Table 1. Characteristics of Classes of Services 1, 2 and 3


Service Characteristics Class-1 Service Class-2 Service Class 3 Service
Communication mode Connection-oriented, Connectionless, frame- Connectionless, frame-
circuit-switched switched (reliable) switched (datagram)

Flow control between the Acknowledge: Acknowledge: Not provided


communicating devices (end-to- ACK-1 each frame ACK-1 each frame
end)
ACK-N each nth frame ACK-N each nth frame
Flow control between the Fabric Not provided (except for Receiver Ready: Receiver Ready:
and the device the connection request)
R_RDY R_RDY

Confirmation of delivery from the ACK_1: Each frame ACK_1: Each frame Not provided
destination device to the source th th
ACK_N: Each n frame ACK_N: Each n frame
device
ACK_0: Each sequence ACK_0: Each sequence
Confirmation of nondelivery from Frame Rejected: P_RJT Frame Rejected: F_RJT Not provided
the destination device to the source
Frame Busied: P_BSY Frame Busied: F_BSY
device

Confirmation of nondelivery from Not provided (except for Frame Rejected: P_RJT Not provided
the Fabric to the device the connection request)
Frame Busied: P_BSY

Broadcast and multicast support Not supported Not supported Supported

In-order delivery of frames Guaranteed Fabric-implementation Fabric-implementation


dependent dependent

Brocade Communications Systems, Inc. 10 Fibre Channel Classes of Service for Data Transport
Table 2. Advantages and Disadvantages of Class-1 Service
Class of Service Advantage Disadvantage
Class-1 Users and applications on a dedicated connection Connection between the devices must be
can interact and engage without others interfering. established before information transfer can begin,
thus increasing the latency.
The connection is not terminated until both parties The path for end-to-end connection, consisting of
are satisfied that all information has been correctly individual links and switches, must be available at
transmitted. full bandwidth for the full duration of the entire
transfer.
The data transport is ideal for information that During connections, bandwidth is wasted when
needs to flow at a constant rate for extended the links are idle while devices are performing
periods of time. other tasks.
Guaranteed in-order delivery of frames in any The time to set up and terminate a connection may
Fabric configuration take much longer than the amount of time required
to transfer an information block, which is
inappropriate for some applications (e.g, storage
using a SCSI protocol). In circuit switching, the
connection set-up time is a greater proportion of
the overhead for short- to medium-length
information transfers.
The time to set up a connection through the
network increases as network traffic increases,
thus increasing the possibility of bandwidth
starvation to the devices wanting to communicate.
Since the applications are based on an information
transfer rate, with a high peak to average ratio,
circuit switching is inefficient for the traffic bursts
that are prevalent in today’s data communication
applications.

Brocade Communications Systems, Inc. 11 Fibre Channel Classes of Service for Data Transport
Table 3. Advantages and Disadvantages of Class-2 and Class 3 Services
Class of Service Advantage Disadvantage
Class-2 and 3 The transmission resources (such as Due to the lack of a dedicated connection, the
bandwidth and buffers) can be shared among transfer of very large blocks of raw
many users on a link (between a switch and information (as in visualization and number
device) and on an ISL (Inter-Switch Link) crunching applications) can cause long fabric
between the switches. delays in a congested Fabric
Due to frame switching, the links are No error control at the link level or end-to-
occupied only for the duration of frame end in Class 3 results in dropping frames
transmission, making the path available for without notification when the Fabric becomes
frames generated by other sources and highly congested. An upper-level protocol
increasing both the link and Fabric bandwidth must get involved to handle retransmission,
utilization. thus resulting in degradation of device and
Fabric throughput.
Frames in the same information block can be In transactions between many devices and
in transit simultaneously on different paths in one device, improper buffer allocation in the
the Fabric, thus fully utilizing expensive link devices may result in long queuing delays
and Fabric resources. within the Fabric.
Although Fabric delay can vary based on Out-of-order frame delivery in the generic
traffic, frame switching has a lower Fabric Fabric case. (The Brocade Fabric guarantees
delay (as compared to circuit-switching) for in-order delivery of frames)
short- and medium-size information transfers
in server-to-server and server-to-storage
communications.
The frame switching capability enables the
forwarding of data frames on available paths
in a Fabric of mesh-connected switches,
which optimizes the utilization of system
bandwidth and provides alternate routes when
link or switch failures occur.
Error handling in Class-2 is performed both
at the link level and on an end-to-end basis,
thereby providing the highest level of data
reliability and integrity.
Due to no acknowledgement frames from the
multicast group members, Class 3 is the ideal
service to achieve high throughput for
multicast mode operation.
In a controlled and robust environment,
Class-3 can be deployed to reduce round-trip
delays and to increase the Fabric throughput.
Data transfer between the devices takes place
without having to wait for the end-to-end
acknowledgement frames.

Brocade Communications Systems, Inc. 12 Fibre Channel Classes of Service for Data Transport