SAN SOLUTIONS

INTRODUCING
BROCADE MULTIPROTOCOL
ROUTING SERVICES

New software capabilities support cost-effective

SAN island connectivity and SAN extension.

To help organizations maximize the value of their Storage Area Networks (SANs)
Brocade continues to develop innovative solutions that extend SAN benefits throughout
the enterprise. As part of this effort, Brocade has developed a unique set of multiprotocol routing services that increase the functionality, scalability, and versatility of todays
SANs. These new services include:
Brocade FC-FC Routing Service for SAN connectivity
Brocade FCIP Tunneling Service for SAN extension over distance
Brocade iSCSI Gateway Service for sharing Fibre Channel resources with
iSCSI devices
Delivered on the Brocade SilkWorm Multiprotocol Router, these services provide
new options for connecting SAN islands and extending SAN benefits over multiple
networks, to larger SAN sizes, and across longer distances. A key aspect of this
approach is the unprecedented capability to configure SAN protocols on a portby-port basis within the Multiprotocol Router.

Multiprotocol Routing Services Overview

The Brocade multiprotocol routing services include three types of services: the
FC-FC Routing Service for SAN island connectivity, the FCIP Tunneling Service
for distance extension, and the iSCSI Gateway Service for sharing Fibre Channel
resources with iSCSI devices. This document describes FC-FC routing in detail
and provides an overview of the FCIP and iSCSI capabilities.
To better understand these services, it helps to know the general terminology
(a glossary of terms also appears at the end of the paper). The primary purpose of
the services is to provide device connectivity between two or more fabrics without
merging those fabrics. The Multiprotocol Router enables the creation of Logical
Storage Area Networks (LSANs) that connect multiple fabrics without merging
them, thereby providing strategic advantages for change management, network
management, scalability, reliability, availability, and serviceability.
An LSAN is similar to a Virtual Private Network (VPN): it connects different networks
but allows only specific devices on those networks to communicate rather than enabling
unrestricted communication. However, it is most useful to think of an LSAN in terms
of zoning: an LSAN is a zone that spans multiple SAN fabrics.

SAN SOLUTIONS

The basic requirement for this type of LSAN solution is to create large, flat Fibre
Channel SANs that can continue to grow in a scalable, cost-effective manner. For
example, organizations with many Fibre Channel SAN islands might not want to
merge them, because they would have to contend with domain ID conflicts, zoning
inconsistencies, fabric-wide parameter mismatches, and other challenges. It might
simply be too much administrative work and risk to production services to justify the
benefit of enhanced connectivity. By using FC-to-FC routing, however, organizations
could interconnect devices without having to solve any of those problems, or face
any of those risks. Each SAN island would remain its own Fibre Channel fabric,
known as an edge fabric in this context.
It is important to note that edge fabrics retain their own separate fabric services:
nameservers, zoning databases, routing tables, domain ID spaces, and so on.This means,
for example, that one fabric could have a domain ID 1 switch, and another fabric could
also have a domain ID 1 switch.Without the Multiprotocol Router, these fabrics could
not merge until such conflicts were resolved, which could be a time-consuming
and risky process in a production environment. With the Multiprotocol Router,
these conflicts are irrelevant. Moreover, FC-to-FC routing provides additional strategic
advantages (none of which would be true with merged fabrics):
The scalability of one edge fabric does not affect another.
Fabric reconfigurations do not propagate across edge fabrics.
Faults in fabric services are contained.
The resulting routed network would consist of multiple individual fabrics that
nevertheless form one storage network connectivity model. This kind of network is
a level above the traditional definition of a SAN, in which a SAN equals a fabric
and a dual-redundant SAN equals a pair of fabrics. This higher-level network is
therefore called a Meta SAN. Figure 1 illustrates a Meta SAN comprised of two
Fibre Channel routers and n edge fabrics.
Meta SAN
Fabric 1

Figure 1.

A Meta SAN that includes

multiple edge fabrics

Fabric n

Edge Fabrics

Standard E_Port
connections from installbase from Brocade switches

EX_Port
connections from
FC Routers

Multiprotocol Routers

In the Brocade LSAN model, Multiprotocol Routers connect to the edge fabrics and
export devices between them by using EX_Ports.These ports look just like normal
Brocade E_Ports to the edge fabrics but limit what each edge fabric sees by using Fibre
Channel Network Address Translation (FC-NAT).This is similar to the hide-behind
NAT used by most IP firewalls. FC-to-FC routing can use FC-NAT to hide an entire
n-domain remote edge fabric behind one translation phantom domain.
An EX_Port can be thought of as being an E_Port lite since it appears like a standard
E_Port to the edge fabric. An E_Port-to-EX_Port connection is called an Inter-Fabric
Link (IFL). Unlike an E_Port, an EX_Port terminates at the router and does not propagate fabric services or routing topology information to other edge fabrics. Moreover,
EX_Ports do not switch between themselves except when crossing between different
edge fabrics. Figure 2 shows a pair of devices exported between two fabrics.
Exporting Devices

Figure 2.

A pair of devices exported

between two edge fabrics
Fabric 1

A host exported from

Fabric 2 now also
appears in the name
server on Fabric 1

Fabric 2

A storage array port

exported from Fabric 1
appears in Fabric 2

Each EX_Port has a user-assigned Fabric Identifier (FID) that specifies the fabric to
which it is attached. Any number of EX_Ports can have the same FID if they attach
to the same edge fabric. In Figure 2, all EX_Ports on all routers connected to Fabric 1
would have FID=1.
In a Meta SAN, a fabric reconfiguration in one edge fabric is not propagated to the
others. Nor is zoning database and fabric topology data propagated, so scalability is
not limited by the sum of all the fabrics zoning, routing, or convergence timing limits.
Even nameserver entries are exported between fabrics only for devices that have
been explicitly added to a relevant LSAN for sharing. For example, if Fabric 1 has
a scalability limit of 1024 nameserver entries and currently has 768 devices, and so
does Fabric 2, an administrator could not merge the fabrics. However, the fabrics
could be connected by Multiprotocol Routers and some devices could be shared
between fabrics without reaching the Simple Name Server (SNS) scalability limit.
When a set of devices on different edge fabrics is allowed to communicate through the
Multiprotocol Router, the resulting connectivity group is known as an LSAN, as shown in
Figure 3. Many different LSANs can exist in the Meta SAN, and many different LSANs
can exist between a given pair of edge fabrics. Likewise, devices can be members of multiple LSANs, and LSANs can overlap with traditional zoning mechanisms on local fabrics.
3

LSAN
Fabric 2

LSANs spanning multiple

fabrics to share devices

LSAN

An LSAN allows connectivity

that spans two or more fabrics

SAN SOLUTIONS

Fabric 1

Figure 3.

Multiprotocol Router Installation

Installing the Multiprotocol Router is a relatively simple process that mirrors the
installation steps of any other Brocade SilkWorm switch, such as configuring an IP
address for management.The appropriate ports must be configured as EX_Ports and
set to the appropriate FIDs using the same tools for traditional switch administrative
tasks:WEB TOOLS, Fabric Manager, the Fabric OS command line interface, and so
on. Configuring each port in this manner is simple enough that relatively junior
SAN administrators can easily install and configure an a Multiprotocol Router.
After the installation, day-to-day administration is performed through zoning on each
edge fabric.This practice enables existing tools from Brocade and many third-party
vendors to work as usual, thereby eliminating the need to extensively retrain SAN
administrators. If a specially named zone (known as an LSAN zone) is created on each
of two fabrics that the Multiprotocol Router can accessand the devices in that zone
are onlinethe Multiprotocol Router would automatically create FC-NAT entries
between those fabrics.This is all that must be done to create an LSAN, as shown in
Figure 4.
LSAN Zones
Fabric 1

Figure 4.

Creation of LSAN zones

Fabric 2

An LSAN is created by
building LSAN zones, which
are indistinguishable from
regular zones to edge fabrics

The Multiprotocol Router obtains the zoning database from each edge fabric and
parses the database for zones with LSAN_ in the name. (The prefix is case-insensitive,
so LSAN_ and lsan_ would be equivalent.) The Multiprotocol Router then compares the WWNs from LSAN zones in each fabric with entries from the other fabrics.
Matching entries define which devices can communicate through the Multiprotocol
Routers across fabrics.As a result, these devices are considered to be in the same LSAN.
The user-defined portion of the LSAN zone name from one fabric does not have to
match the user-defined portion of the LSAN zone name from another fabric for devices
to reside in the same LSAN.
It is important to note that these are real zones in the edge fabrics, and the devices that
exist in these zones are subject to normal zoning enforcement by the switches in each
edge fabric. If the administrator of Fabric 1 does not zone a host with a storage array
from Fabric 2, it doesnt matter if the Fabric 2 administrator did so.The devices will
be able to communicate only when the zoning policy on both fabrics allows it.
LSAN Administration Models

There are two primary administration models for LSANs: the model in which one
administrator owns the routers and all edge fabrics involved, and the model in which
different administrators own each component.
An administrator who wants to allow connectivity between two fabrics and who has
administrative access to both can accomplish this task by using traditional zoning tools or
by using a single operation through Fabric Manager. Because Fabric Manager can access
both fabrics, it can simultaneously create both LSAN zones.When instructed to create
an LSAN, Fabric Manager determines which fabrics the devices reside in and creates the
appropriate LSAN zones in each fabric.This is known as the Super Admin model.
In contrast, if different administrators have access to each fabric, they can create LSAN
zones containing the devices they want to export.This is the Multi Admin model.
If a large number of SAN islands will be combined into a very large Meta SAN,
administrators could network Multiprotocol Routers over a centralized backbone
fabric, as shown in Figure 5.
Backbone Fabric Meta SAN

Figure 5.

Networking Multiprotocol
Routers over a centralized
backbone fabric

Fabrics 1 through 8

Fabrics 9 through 16

NR_Ports are virtual

N_Port devices attached
to the router

additional edge fabrics

Routers connect to
the backbone with
standard E_Ports
Router domains talk
across the Backbone
fabric with FCRP

The Backbone Fabric

uses standard E_Ports
on standard switches

additional backbone fabrics

SAN SOLUTIONS

A host in Fabric 1 could be exported into Fabric 16 across the backbone fabric, even
though those fabrics are not attached to the same Multiprotocol Router.To accomplish
this, Brocade uses a standards-track Fibre Channel Router Protocol (FCRP) that operates on backbone-attached E_Ports.The Multiprotocol Router does not use a special
E_Port type (such as an EX_Port) for a router-to-backbone connection. Instead, it uses
a standard Brocade E_Port. Each Multiprotocol Router on the backbone fabric can
see that other routers have entered that fabric, and can send FCRP messages from its
domain address to the other routers domain addresses. FCRP also operates between
domains projected by EX_Ports into an edge fabric. In addition to these routing tasks,
FCRP handles coordination between domains, exchanging LSAN zone information as
well as device and fabric state information.
Each Multiprotocol Router also projects special virtual N_Ports (known as NR_Ports)
onto the backbone fabric. NR_Ports serve as sources and destinations for data frames
sent across the backbone.They are similar to router ports in IP networks and can be
thought of as the back side of an EX_Port.They are discovered via FCRP and do
not exist in the nameserver of the backbone fabric.
Data frames sent between NR_Ports use an encapsulated global header that contains
items such as the source and destination fabric IDs, so that a receiving router knows the
true origin and destination of a frame.This process is transparent to switches in the
backbone fabric.
Deployment Examples

To help illustrate the Brocade multiprotocol routing services, this section contains
deployment examples of possible solutions. However, this section does not represent
a comprehensive list of all possible applications of the Multiprotocol Router.
Basic Resilient Meta SAN

The defining characteristic of a resilient SAN is that there is no single point of failure
within the core-to-edge connectivity model. Each Inter-Switch Link (ISL) has one
or more alternate paths, and each core switch is likewise duplicated. It is possible to lose
an edge switch, but critical nodes are always connected to at least two edge switches.
In a resilient Meta SAN, each edge fabric that exports devices must have at least two
Multiprotocol Routers providing paths to every other relevant edge fabric. Nodes
would generally be connected to A/B fabrics within this model, as shown in Figure 6.
Figure 6.

Resilient Meta SAN

Fabric 1 (A)

Fabric 2 (B)

Fabric 3 (A)

A resilient Meta SAN with

redundant paths between devices

Fabric 4 (B)

Basic Dual-Redundant Meta SAN

A fully redundant SAN duplicates the entire connectivity model: a dual-redundant SAN
fabric has two completely separate fabrics. Similarly, a dual-redundant Meta SAN has
two completely separate (A/B) Meta SANs (see Figure 7). Hosts and storage arrays are
dual-attached, with at least one connection to each Meta SAN.This provides the greatest possible fault isolation, such as preventing a misbehaving Host Bus Adapter (HBA)
on one Meta SAN from interfering with nodes on the other.
Dual-Redundant Meta SAN

Figure 7.

A dual-redundant Meta SAN

with enhanced fault isolation

Meta SAN A

Meta SAN B

Fabric 1A

Fabric 2A

Fabric 1B

Fabric 2B

Tape Consolidation Meta SAN

One key FC-to-FC routing application is centralizing backup and restore resources
across SAN islands. In this example, 30 isolated fabrics can share a central tape pool
consisting of two large backup and restore fabrics, as shown in Figure 8.

Figure 8.

Tape Consolidation Meta SAN

A centralized tape pool

shared across SAN islands
Fabric 1

Fabric 30

Distributed Host and

Storage Edge Fabrics

Centralized Backup and

Restore Edge Fabrics

Fabric 31

Fabric 32

In many environments, even if SAN islands could be merged without crossing

fabric scalability limits, the process of doing so would be too difficult and risky to
be considered. In contrast, FC-to-FC routing enables the connection of SAN islands
without needing to resolve domain conflicts, rework zoning configurations, or resolve
fabric-wide parameter conflicts such as Timeout Value (TOV) and Port ID (PID)
formats (see Figure 9).
Figure 9.

Island Consolidation Meta SAN

Fabric ID = 1
PID format = 1
Domain IDs = 1, 2, 3, 4

Fabric ID = 4
PID format = 0
Domain IDs = 1, 2

SAN island consolidation

in a Meta SAN

Fabric ID = 15
PID format = 0
Domain IDs = 1, 6, 8, 9

Each fabric can have its

own set of domain IDs, its
own zoning, its own PID format,
its own TOVs, etc.

SAN SOLUTIONS

SAN Island Consolidation Meta SAN

Fabric ID = 19
PID format = 1
Domain IDs = 1, 2

This example shows many small fabrics

which could be merged from a scalability
standpoint, but not without making many
changes to resolve conflicts.

Distance Extension Meta SAN (FC-to-FC Routing and FCIP)

In another example, an organization might have separate A/B fabric pairs for disastertolerant production environments in distinct geographical locations, and a separate fabric
for pre-production.This implementation does not provide optimal sharing of resources
since there is no connectivity between fabrics unless hosts and storage arrays have five
network connections each.This implementation also requires the use and management
of expensive external FCIP gateways (see Figure 10).
Disaster-Tolerant Without FC-to-FC routing

Figure 10.

SAN distance extension over

FCIP without FC-to-FC routing

Production A
Disaster-Tolerant A

Production B
IP WAN

Pre-Production

Disaster-Tolerant
Nodes
Disaster-Tolerant B
Hosts and storage involved in
Disaster-Tolerant and pre-production
operaions in any way need five HBAs

Unreliable WAN links can create

instability for all Disaster-Tolerant fabrics

Dedicated Pre-Production
Nodes

In contrast, the implementation of a dual-redundant Meta SAN enables fault isolation

between each fabric, retaining the fully redundant model but allowing connectivity
as needed. Hosts and storage can communicate across all fabrics but need only two
HBAs for fully redundant operation.There are still five fabrics, but the LSAN switches
allow cross-fabric communication (see Figure 11).
Disaster-Tolerant With FC-to-FC routing

Figure 11.

SAN distance extension

with FC-to-FC routing

Production A

Backbone A
Disaster-Tolerant A
Production B

IP WAN

Pre-Production
Disaster-Tolerant B

Disaster-Tolerant
Nodes

Backbone B

Integrated FCIP capabilities simplify management and eliminate cost and potential failures caused by external WAN tunneling equipment. Because the Multiprotocol Routers
are running FC-to-FC routing, the WAN forms a completely redundant and isolated
pair of backbone fabrics.This capability increases fault isolation in the WAN: IP instability is no longer translated into edge fabric reconfigurations. A similar design would also
work with external gateways for SONET, ATM, or other third-party solutions.
Service Provider Meta SAN

In the example of a service provider model, the provider might have a central set of
resources that are projected as needed into fabrics owned and managed by its customers. No customer could access any other customers fabric without permission.
Nor could any customer access resources on the service providers fabric without
permission. As a result, the appropriate group could autonomously manage each
fabric without faults in one section impacting any other section (see Figure 12).
Figure 12.

Service Provider Meta SAN

A Meta SAN in a service

provider environment

Fabric 31

The IP WAN is really one or more tunnelled

backbone fabrics. It could also be a native
Fibre Channel network, or use other gateways
such as SONET.

Customer A

Customer B

IP WAN

Customer C

Customer D

The Brocade iSCSI Gateway Service enables organizations to integrate low-cost

Ethernet-connected servers into their Brocade Fibre Channel SANs via the iSCSI
protocol.This practice facilitates storage consolidation and improves management of
applications such as centralized backup in cases where hosts do not need the performance or reliability of Fibre Channel but could benefit from the management simplicity
and white space optimization of a SAN infrastructure.This type of integration reduces
the cost of connecting a low-tier server to centrally managed storage within a SAN.

SAN SOLUTIONS

iSCSI Gateway Service Overview

Organizations that use this service would not typically attach iSCSI hosts directly to
gateway ports. Direct attachment of an iSCSI TCP Offload Engine (TOE) adapter to
an iSCSI gateway is less cost-effective than using a Fibre Channel HBA. Instead, there
would be an external fan-out using existing Ethernet infrastructure.
When an iSCSI host accesses the service, it is projected onto the backbone fabric of
the router and can then communicate with any storage device in that fabric.
FCIP Tunneling Service Overview

The Brocade FCIP Tunneling Service enables organizations to extend their Fibre
Channel SANs over longer distances that would be impractical with native Fibre
Channel links, or situations where dark fiber links would be impractical but in
which IP WANs already exist.
This service offers two important advantages.The first advantage is full integration with
the switch. It is easier and more cost-effective to deploy and manage an FCIP link integrated into the switch as opposed to one that requires an external gateway. In addition
to easier management, tighter integration means lower cost and a smaller rack footprint.
The second advantage is integration with FC-to-FC routing. It is possible to have a
Multiprotocol Router in which a port is both an E_Port into the backbone fabric and
an FCIP-to-FCIP port.This prevents faults on the WAN link from turning into Meta
SAN-wide events.This is a key advantage, because IP networks in general and WANs in
particular are less reliable than Fibre Channel networks. A flappingWAN link might
disturb the backbone fabric, but these disturbances are isolated from all edge fabrics so
no host/storage conversations would be affected other than those that actually crossed
the unreliable WAN.
This service will initially be most useful for campus networks and WANs that have
full-bandwidth links. In addition, Brocade plans to continue partnering with CNT
for enterprise-class FCIP-to-FCIP distance extension solutions, and with other leading
vendors such as Alcatel and Nortel for other WAN protocols.
For More Information

For more information about the Brocade SilkWorm Multiprotocol Router and
Brocade Multiprotocol SAN Routing Services, visit www.brocade.com.

10

Glossary of Terms

Backbone Fabric: A capability that enables scalable Meta SANs by allowing the networking of multiple Multiprotocol Routers that connect to the backbone fabric via
E_Port interfaces. Devices attached to Multiprotocol Routers via F_Port or FL_Port,
or imported via the iSCSI Gateway Service, are also considered part of the backbone.
E_Port: A standard Fibre Channel mechanism that enables switches to network with
each other.
Edge Fabric: A Fibre Channel fabric connected to a Multiprotocol Router via one or
more EX_Ports.This is where hosts and storage are typically attached in a Meta SAN.
EX_Port: The type of E_Port used to connect a Multiprotocol Router to an edge
fabric. An EX_Port follows standard E_Port protocols and supports FC-NAT but
does not allow fabric merging across EX_Ports.
Exported Device: A device that has been mapped between fabrics. A host or storage
port in one edge fabric can be exported to any other fabric through LSAN zoning.
Fabric: A collection of Fibre Channel switches and devices, such as hosts and storage.
Fabric ID (FID): Unique identifier of a fabric in a Meta SAN.
FCIP Tunneling Service: A service that enables SANs to span longer distances than
could be supported with native Fibre Channel links. FCIP is a TCP/IP-based tunneling
protocol that allows the transparent interconnection of geographically distributed SAN
islands through an IP-based network.
Fibre Channel: The primary protocol for building SANs. Unlike IP and Ethernet,
Fibre Channel is designed to support the needs of storage devices of all types.
Fibre Channel Network Address Translation (FC-NAT): A capability that allows
devices in different fabrics to communicate when those fabrics have addressing conflicts.
This is similar to the hide-behind NAT used in firewalls.
Fibre Channel Router Protocol (FCRP): A Brocade-authored standards-track
protocol that enables LSAN switches to perform routing between different edge
fabrics, optionally across a backbone fabric.
FC-FC Routing Service: A service that extends hierarchical networking capabilities
to Fibre Channel fabrics. It enables devices located on separate fabrics to communicate
without merging the fabrics. It also enables the creation of LSANs.
Inter-Fabric Link (IFL): A connection between a router and an edge fabric.
Architecturally, these can be of type EX_Port-to-E_Port or EX_Port-to-EX_Port.
The former method is supported in the first release.
iSCSI Gateway Service: A service that maps the SCSI protocol to the IP transport.
This service projects iSCSI hosts onto the backbone fabric of a gateway switch.
Logical Storage Area Network (LSAN): A logical network that spans multiple fabrics.
The path between devices in an LSAN can be local to an edge fabric or cross one or
more Multiprotocol Routers and up to one intermediate backbone fabric. LSANs are
administered through LSAN zones in each edge fabric.
11

Meta SAN: The collection of all devices, switches, edge and backbone fabrics, LSANs,
and Multiprotocol Routers that make up a physically connected but logically partitioned storage network. In a data network, this would simply be called the network.
However, an additional term is required to specify the difference between a single-fabric
network (SAN), a multifabric network without cross-fabric connectivity (for example,
a dual-redundant fabric SAN), and a multifabric network with connectivity
(Meta SAN).

SAN SOLUTIONS

LSAN Zone: The mechanism by which LSANs are administered. A Multiprotocol

Router attached to two fabrics will listen for the creation of matching LSAN zones
on both fabrics. If this occurs, it will create phantom domains and FC-NAT entries
as appropriate, and insert entries for them into the nameservers on the fabrics. LSAN
zones are compatible with standard zoning mechanisms.

Multiprotocol Router: A device that enables the Brocade multiprotocol routing services.
Multiprotocol routing services: An optionally licensed software bundle available on
certain Brocade platforms, such as the Multiprotocol Router, that includes the FC-FC
Routing Service, the iSCSI Gateway Service, and the FCIP Tunneling Service.
NR_Port: A port used as a source and destination address for frames traversing a backbone fabric. A normal E_Port (not an EX_Port) is used to connect a Multiprotocol
Router to a backbone. An NR_Port appears to the rest of the backbone as a standard
N_Port connected to the Multiprotocol Router domain.

12

Corporate Headquarters
San Jose, CA USA
T: (408) 333-8000
info@brocade.com

Asia Pacific Headquarters

Tokyo, Japan
T: +81 3 5402 5300
apac-info@brocade.com

European Headquarters
Geneva, Switzerland
T: +41 22 799 56 40
europe-info@brocade.com

Latin America Headquarters

Miami, FL USA
T: (305) 716-4165
latinam-sales@brocade.com

2004 Brocade Communications Systems, Inc. All Rights Reserved. 02/04 GA-WP-642-02
Brocade, the Brocade B weave logo, Secure Fabric OS, and SilkWorm are registered trademarks of Brocade Communications Systems, Inc., in the United States and/or
in other countries. FICON is a registered trademark of IBM Corporation in the U.S. and other countries. All other brands, products, or service names are or may be
trademarks or service marks of, and are used to identify, products or services of their respective owners.
Notice:This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or
service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for
its use.This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.