Introduction to

Storage Area Networks (SAN)

SAN-1501

SAN-1501
12686_05_2006_c2 © 2006 Cisco Systems, Inc. All rights reserved. Cisco Public 1
 Session Agenda

Main Topics for Today

• Storage Basics
The What …

• Fibre Channel Basics

… and the How.

• Where to go from here

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 2
 Storage Basics

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 3
 Storage Basics

• Storage components
Host bus adapter
Disks, JBODs, RAID, storage arrays

• Storage topologies
DAS, NAS, and SAN

• Storage protocols
SCSI
Storage transport protocols
Fibre Channel (FC) , FCIP, iSCSI

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 4
 Storage Components

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 5
 Host Storage Components

• Host bus adapter (HBA)

• Hardware resident on host server
• Connection interface to disk subsystem
• Connection method
Copper
Optical

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 6
 Data Storage Components
I/O Devices—Disk Drives
• Fundamental unit for data storage
• Disk drive types
Parallel SCSI
Fibre channel
Advanced Technology
Attachment ATA) or Integrated
Drive Electronics (IDE)
Serial ATA (SATA)
Other

• Tape drives

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 7
 Connecting Hosts to Storage
Storage Interconnects
• Parallel SCSI copper interconnects
• Optical direct connect
• Fibre Channel switch

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 8
 The SCSI I/O Channel—Starting Point

• SCSI I/O Channel provides

Applications
half-duplex pipe for SCSI Half-Duplex
command, data, and status File System I/O Channel

SI
• SCSI I/O channel can be Block Device
S C

internal or external to host SCSI Generic

• Multiple SCSI I/O channels

TCP/IP Stack
can exist within host

SCSI
NIC Driver Adapter Driver
• A network approach can
scale the I/O channel in many NIC Adapter SCSI Adapter
areas (length, devices, SI
SC
throughput)
SCSI
Initiator
SCSI
Target

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 9
 The Parallel SCSI I/O Channel
Host (Initiator)
• The SCSI Channel is used to transmit
SCSI commands, data, and status
• Multiple devices can exist on the channel SCSI
SCSI
each playing the role of SCSI initiator or
SCSI target
• Most common channel is the basic

I/O Channel
Channel
parallel SCSI bus which can be internal Controller

or external to a host
• Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth upto 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation Target 1 Target 2
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 10
 Networking the I/O Channel
Host (Initiator)

• Same SCSI protocol carried over a network Initiator

transport via serial implementation
• Transport must not jeopardize SCSI payload SCSI
SCSI
(security, integrity, latency)
• Two primary transports to choose from

I/O Channel
Networked
Channel
today, namely IP and Fibre Channel Controller

• A networked I/O channel allows for multiple

improvements: Network
Distance limitations greatly increased
Dedicated bandwidth (not shared)
High # of addressable devices
Bandwidth increase (including link bundling)

Target
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 11
 Fibre Channel I/O Networking
Host System

• Very common method for networking SCSI Initiator

• Fibre Channel provides high speed transport
SCSI
SCSI
for SCSI payload
• Fibre Channel overcomes many shortcomings
of Parallel I/O including: Ethernet
NIC
Addressing for up to 16 million nodes
Loop (shared) and fabric (switched) transport
Fibre Channel
Host speeds of 100 to 400 MBps (1-4 Gbps) Fabric

Segments of up to 10km (without extenders)

Support for multiple protocols
• Combines best attributes of a channel and a
network together
Target
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 12
 IP I/O Channel Networking

• IP access to open systems iSCSI iSCSI-Enabled

Host (Initiator)
and Fibre Channel storage iSCSI
iSCSI

• iSCSI driver is loaded onto hosts

on Ethernet network Ethernet
iSCSI NIC
iSCSI
• Able to consolidate servers Appliance
(Target)
via iSCSI onto existing IP
storage arrays Network

• Able to build Ethernet-based iSCSI

Gateway
SANs using iSCSI arrays
• Storage can be mapped from FC
SCSI
SCSI

iSCSI to Fibre Channel Fabric FC HBA

Attached
Host
(Initiator)
Storage
Pool (Target)
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 13
 Storage Subsystems

• Just a bunch of disks (JBOD)

• Storage arrays
• Redundant array of independent
disks (RAID)

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 14
 Simple Storage Systems
I/O Devices—JBODs

TRGT 1 TRGT 3
SCSI CABLE
SCSI BUS
Server
With SCSI TRGT 0 TRGT 2 TRGT 4
SCSI JBOD

TRGT 5 TRGT 4
TRGT 3

FC CABLE FC-AL

Server
With Fibre Channel
Arbitrated Loop TRGT 0 TRGT 1 TRGT 2
(FC-AL)
FC-AL JBOD
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 15
 Large Storage Systems – Bus Type
I/O Devices—Intelligent Storage Arrays

Host Bus Adapters Host Bus Adapters

Host Channel
Connections

Dual-Processor CD CD CD CD
Channel Director Cache
Cache
X BUS
System Bus
Y BUS

Dual-Processor DD DD DD DD
Disk Director

Disks Disk Drives Disk Drives

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 16
 Large Storage Systems – Switched Type
I/O Devices—Intelligent Storage Arrays

Host Bus Adapters Host Bus Adapters

Host Channel
Connections
Dual-Processor
Channel CD CD CD CD
Connectors Cache

Switched
Backplane

Dual-Processor DD DD DD DD
Disk Connectors

Disks Disk Drives Disk Drives

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 17
 Logical Arrays of Storage
RAID
• Redundant Array of Independent Disks (RAID)
Word coined by researchers at University of California,
Berkeley in 1987
A method to inexpensively put together a set of physical
hard drives into an array
Provides fault tolerance by mirroring or parity operations
RAID can be performed using hardware or using host
based software

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 18
 Types of RAID

• RAID 0: Stripping
For performance and size, two or more hard disks are
concatenated together to form a larger volume

• RAID 1: Mirroring
For reliability and availability

• RAID 3: Error Detection

For reliability, availability of data using using a dedicated
parity drive

• RAID 5: Error Correction

For reliability, availability of both data and parity

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 19
 RAID 0: Striping

• Data is segmented and

split across multiple spindles
• I/O benefits
Short reads and writes,
easily handles multiple
simultaneous reads
Long reads and writes,
single operations split and
processed in parallel
• Redundancy—none
• Cost—good
(no extra hardware)

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 20
 RAID 1: Mirroring

• Data is duplicated on
multiple spindles
• I/O benefits
Short and long reads
with shorter latency
Short and long writes
slower due to multiple
writes

• Redundancy—good
• Cost—need double the
amount of disks

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 21
 RAID 0+1:

• Data is striped and

duplicated on multiple
spindles
• I/O benefits
Short and long reads
with shorter latency
Short and long writes
faster as writes are
spread across multiple
spindles
• Redundancy—good
• Cost—need double the
amount of disks

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 22
 RAID 1+0:

• Data is duplicated
and striped on
multiple spindles
• I/O benefits
Less downtime than
RAID 0+1
• Redundancy—good
• Cost—need double
the amount of disks

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 23
 RAID 5: Error Correcting

• Data protection using ECC

(Error Correction Control)
parity spread over all drives
• I/O benefits
Short and long reads at
normal speeds
Short and long writes
slower due to parity
calculations
• Redundancy—better than
RAID3
• Cost—needs only one extra
disk for an entire logical
disk
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 24
 Storage Topologies

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 25
 Storage Topologies

IP
SAN
Network

IP Front-End Network SAN Back-End Network

• Host to Host • Host to Storage
• Application to file system • File system to Device
• Client to Server • Program to Device
• NFS, SMB, CIFS, NCP • SCSI, IDE, NTFS, FAT
• NAS, WAFS • SAN

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 26
 Direct Attached Storage (DAS)

• Storage is captive ‘behind’ the

server, limited mobility
Clients
• Limited scalability due to
limited devices
LAN Application
• No efficient storage sharing
Servers
possible
• Costly to scale; complex to Win2k Linux Unix Win2k Linux Unix
manage
SCSI FC FC

Tape
Direct Attached Storage

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 27
 Network Attached Storage (NAS)

• Storage is accessed at a
file level via NFS or CIFS
• Storage is accessed over
an IP network
• Storage devices can be LAN
shared between servers
• Files can be shared Win2k Linux Unix
between users Generic Generic

• TCP can be tuned to

optimize for file transport Application
Servers
NAS Appliances
or
NAS Head Ends
NAS = Optimized for File I/O
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 28
 Storage Area Network (SAN)

• Storage is accessed at a Clients

block-level via SCSI
• High performance interconnect
providing high I/O throughput LAN

• Lower TCO relative to direct

Servers
attached storage, storage can
be shared
Fibre
• Limited vendor interoperability Channel
SAN
• Complex management
Block
Storage
Devices
Separation of Storage from the Server

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 29
 iSCSI Storage Network
iSCSI-Enabled
• Storage is accessed at a iSCSI Hosts (Initiators)
iSCSI
block-level via iSCSI iSCSI

• Good performance via

standard Ethernet NIC iSCSI
Appliance iSCSI
(Target)
• Enhanced performance with
TCP Offload Engine (TOE) IP
Network
• Lower TCO relative to direct
Fibre Channel HBA/Fabric iSCSI
Gateway
• Standards based
FC
• Complex management Fabric
FC HBA
Attached
Host
(Initiator)
Storage
Pool (Target)
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 30
 DAS, SAN, iSCSI, NAS Comparison
iSCSI iSCSI NAS NAS
DAS SAN
Appliance Gateway Appliance Gateway
Computer System Computer System Computer System Computer System Computer System Computer System

Application Application Application Application Application Application

Host/ File System File System File System File System File System File System

Server Volume Manager Volume Manager Volume Manager

SCSI Device Driver
iSCSI Driver
Volume Manager
SCSI Device Driver
iSCSI Driver
I/O Redirector
NFS/CIFS
I/O Redirector
NFS/CIFS
SCSI Device Driver SCSI Device Driver TCP/IP Stack TCP/IP Stack
TCP/IP stack TCP/IP stack
SCSI Bus Adapter FC HBA NIC NIC NIC NIC

Block I/O File I/O

SAN IP IP IP IP
Storage
NIC NIC NIC NIC
Transport TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack
iSCSI Layer iSCSI Layer File System File System
SCSI FC Bus Adapter FC HBA Device Driver FC HBA

Storage FC Block I/O FC

Media
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 31
 Wide Area File Services (WAFS)

WAFS Provides File Services Between a

Remote Client and a Centralized Server
Protocol proxy to handle non-critical messages
Intelligent Operation batching and message bundling
Message Suppression
Decouple the user experience from the WAN

Validate and serve information locally

Data Distribution Prepopulate via preposition and on-demand
and Caching
Asynchronous writes to mask large transfers

Compress all messages

Link and Throughput Pipeline multiple segments into a common window
Optimization
Multiplex over many sockets if necessary

Maintain data integrity at all costs

Data Integrity Never compromise security of the data
Guarantee support for all protocol semantics

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 32
 Storage Protocols

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 33
 Small Computer System Interface—SCSI

• SCSI is a standard that defines an interface

between an initiator (usually a computer) and
a target (usually a storage device such as
a hard disk)
• Interface refers to connectors, cables, electrical
signals, optical signals and the command protocol
that allow initiators and targets to communicate
• Logical Unit Number (LUN): A 64-bit field within
SCSI that identifies the Logically addressable Unit
within a target SCSI device

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 34
 SCSI Command Protocol

• SCSI command protocol is the de facto standard

that is used extensively in high-performance
storage applications
• The command part of SCSI can be:
Transported over a Fibre Channel storage area network
Encapsulated in IP and carried across IP networks

• To understand the finer points involved with

transporting SCSI across a network with FC or
Ethernet, the basics of SCSI must be well
understood

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 35
 Standards

• SCSI has evolved since it was introduced as SASI

in 1979 by Shugart Associates – it was approved as
a standard by ANSI in 1986 and is now referred to
as SCSI-1
• SCSI-2 was approved by X3 in 1990 and by ANSI
in 1994
• SCSI-3 refers to a collection of standards, each of
which defines a very specific part of SCSI: physical
interface, transport interface, command interface,
architecture model, programming interface, etc.
Also known as SCSI Architecture Model – 3 (SAM-3)

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 36
 SCSI Architecture Model

SAN

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 37
 The SCSI I/O Transaction

• The SCSI protocol forms the basis of an I/O transaction

• The channel provides connectivity between communicating
devices in a SCSI transaction
The following shows two sample SCSI exchanges:

Host (Initiator) Disk (Target)

SCSI READ OPERATION
DATA STATUS
DATA DATA READ

SCSI I/O Channel

Host (Initiator) Disk (Target)

SCSI WRITE OPERATION
STATUS DATA WRITE
DATA DATA

SCSI I/O Channel

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 38
 SCSI Commands

• Data transfer
READ, WRITE

• Commands used in boot/discovery:

REPORT LUNS
INQUIRY
TEST UNIT READY

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 39
 SCSI Read

1. Send SCSI Cmd issued by

initiator – the command sent
is READ;
2. SCSI Command Received by
target;
…
Data transfers occur
during the ‘working’ phase
between initiator and target;
3. Send Command Complete is
returned by the target;
4. Command Complete Received
by target.

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 40
 SCSI Write

1. Send SCSI Cmd issued by

initiator—the command sent
is WRITE;
2. SCSI command received
by target;
3. Target returns TRGT-RDY;
4. Initiator receives TRGT-RDY;
Data transfers occur during
the ‘working’ phase between
initiator and target;
5. Status complete is returned
by the target;
6. Status complete received by
initiator

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 41
 Storage Transport Protocols

• Protocols used to transport SCSI Command and Data

Parallel SCSI
Fibre Channel
FCIP
iSCSI

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 42
 Storage Protocols
Protocol and Transport Stack

SCSI Applications (File Systems, Databases)

SCSI Block SCSI Stream Other SCSI

Commands Commands Commands

SCSI Commands, Data, and Status

Parallel SCSI Fibre Channel

Fibre Channel iSCSI
Interfaces FCIP

TCP

IP

Parallel SCSI
Fibre Channel Ethernet
Interfaces

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 43
 The Parallel SCSI I/O Channel
Host (Initiator)
• The SCSI Channel is used to transmit SCSI
commands, data, and status
• Multiple devices can exist on the channel each SCSI
SCSI
playing the role of SCSI initiator or SCSI target
• Most common channel is the basic parallel SCSI

I/O Channel
bus which can be internal or external to a host Channel
Controller
• Parallel SCSI Specifications:
Up to 25m bus length
Shared channel bandwidth upto 320MBps
Up to 16 devices per SCSI bus
Half-duplex operation

Target 1 Target 2
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 44
 Storage Protocols

Fibre Channel Protocol and Transport Stack

SCSI Applications (File Systems, Databases)

SCSI Block SCSI Stream Other SCSI

Commands Commands Commands

SCSI Commands, Data, and Status

Parallel SCSI Fibre Channel

Fibre Channel iSCSI
Interfaces FCIP

TCP

IP

Parallel SCSI
Fibre Channel Ethernet
Interfaces

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 45
 What Is It?
Fibre Channel
Channels Networks
• Connection service • Connectionless
• Physical circuits • Logical circuits
• Reliable transfers • Unreliable transfers
• High speed • High connectivity
• Low latency Fibre Channel • Higher latency
• Short distance • Longer distance
• Hardware intense • Circuit and packet • Software intense
switched
• Reliable transfers
• High data integrity
• High data rates
• Low latency
• High connectivity
• Long distance

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 46
 Fibre Channel Protocol Architecture

ULP (Upper Level Protocol)

SCSI-3 IP FICON
(sample – there are more)

Link FC Single
FC-4 SNMP Mapping SCSI-3 Mapping
Encapsulation Byte Command
(FC-SNMP) (SCSI-FCP)
(FC-LE) Sets

FC-3 Common Services

FC-2 Framing Protocol Fibre Channel FC-AL FC-AL-2 FC-SW-2

Physical & Signaling
FC-1 Encode / Decode Interface 8B/10B Encoding
(FC-PH, FC-PH2,
FC-0 Physical Interface FC-PH3) Copper Optical

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 47
 Fibre Channel Functions
Structure Is Divided into 5 Levels of Functionality
• FC-0
• Defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.
• FC-1
• Defines how characters are encoded/decoded for transmission
Transmission characters are given desirable characters
• FC-2
• Defines how information is transported
Frames, sequences, exchanges, login sessions
• FC-3
• Place holder for future functions
• FC-4
• Defines how different protocols are mapped to use Fibre Channel
SCSI, IP, Virtual Interface architecture, FICON, others

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 48
 Fibre Channel FC-2 Hierarchy

• Multiple exchanges are initiated between initiators (hosts) and

targets (disks)
• Each exchange consists of one or more bi-directional
sequences
• Each sequence consists of one or more frames
• For the SCSI3 ULP, each exchange maps to a SCSI command

OX_ID &
RX_ID Exchange

SEQ_ID Sequence Sequence Sequence

SEQ_CNT Frame Frame Frame

Frame
Fields ULP Information Unit
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 49
 Fibre Channel Frame Format

• All FC-2 frames follow the general frame format as

shown below
• Idles are ‘Ordered Sets’ used for synchronization and
basic signaling
• SOF – Start-of-Frame, EOF – End-of-Frame

General FC-2 Frame Format

Frame Content

Frame
Idles* SOF Data Field CRC EOF Idles*
Header
(4) (24) (0-2112) (4) (4)

* 6 Idle words (24 bytes) requires by TX 0-528 Transmission Word

2 Idle words (8 bytes) guaranteed to RX
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 50
 Fibre Channel Storage Area Network (SAN)

FC = Fibre Channel
Storage F = F_Port
Host Switch A

F FC

FC F
E

FC
Storage
Switch B

F FC

FC E
F

FC
Host

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 51
 Storage Protocols

FCIP Protocol and Transport Stack

SCSI Applications (File Systems, Databases)

SCSI Block SCSI Stream Other SCSI

Commands Commands Commands

SCSI Commands, Data, and Status

Parallel SCSI Fibre Channel

Fibre Channel iSCSI
Interfaces FCIP

TCP

IP

Parallel SCSI
Fibre Channel Ethernet
Interfaces

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 52
 Storage Protocols - FCIP
FCIP Frame Format
• Primary use is for Storage-to-Storage connectivity via an
IP WAN/MAN
• The FCIP Link carries encapsulated fibre channel traffic
between Link End Points over an IP network by using TCP
on port 3225
• The result is a virtual Inter Switch Link (ISL) between FC
Fabrics
• FC frame not changed

IP
TCP
FCIP
FC
SCSI
Data

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 53
 FCIP SAN Interconnect

FC = Fibre Channel
F = F_Port
E = E_Port
Storage VE = Virtual E_Port
Host Switch A GE = Gigabit Ethernet
Storage
FC Switch C
F
F F GE
FC IP
E
GE VE
E
FC
FC
Switch B

VE GE FCIP Link
FC E
F
Virtual ISL
FC Host

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 54
 Storage Protocols

iSCSI Protocol and Transport Stack

SCSI Applications (File Systems, Databases)

SCSI Block SCSI Stream Other SCSI

Commands Commands Commands

SCSI Commands, Data, and Status

Parallel SCSI Fibre Channel

Fibre Channel iSCSI
Interfaces FCIP

TCP

IP

Parallel SCSI
Fibre Channel Ethernet
Interfaces

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 55
 Storage Protocols
Storage Protocols—iSCSI
• Primary use is for Host-to-Storage connectivity via an IP LAN
• SCSI command and data are encapsulated into iSCSI by
adding a special header
• iSCSI data is encapsulated into a TCP packet
• IP is the transport protocol

IP
TCP
iSCSI
SCSI
Data

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 56
 iSCSI Architecture
iSCSI—SCSI Relationship
• SCSI device
IP host
iSCSI node initiator SCSI Device (iSCSI Node – Initiator)

FC storage iSCSI Initiator Port

iSCSI node target Network Portal Network Portal

HBA/NIC HBA/NIC
• SCSI port
iSCSI initiator port IP Network
iSCSI target port
Network Portal Network Portal Network Portal
• Network portal IP Address IP Address IP Address
TCP Port TCP Port TCP Port
Any network interface
with TCP/IP iSCSI Target Port iSCSI Target Port

Provides physical IP SCSI Device (iSCSI Node – Target)

network access`
Logical Units

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 57
 iSCSI Host Attachment

FC = Fibre Channel
F = F_Port
E = E_Port
Storage VT = Virtual Target
Host Switch A GE = Gigabit Ethernet iSCSI
Host
F FC

FC F
E IP
FC

Switch B

VT GE TCP/IP Link
FC E
F
iSCSI
FC

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 58
 Storage Protocols Summary
Storage Transport Protocols
• Protocols used to transport SCSI
Parallel SCSI – Legacy host to storage bus topology
Fibre Channel – Robust serial transport network for SCSI
Primary transport for Storage Area Networks
FCIP – FC-2 encapsulation in TCP/IP
SAN-to-SAN Connectivity
Typically used in the WAN/MAN
iSCSI – TCP/IP transport for SCSI protocol
Host-to-SAN Connectivity
Typically used in the LAN

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 59
 Fibre Channel Basics

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 60
 Fibre Channel Basics

• Fibre Channel Physical Attributes

Ports
Links
Nodes
Fabrics
• Fibre Channel Operational Characteristics
Naming
Addressing
Fabric Services
Routing
Zoning
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 61
 FC Physical Attributes

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 62
 Physical

• The main physical objects in Fibre Channel are:

Ports
Links
Nodes
Fabric

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 63
 Port Type Examples

Fabric Switch

Node NL_Port FL_Port E_Port E_Port

G_Port F_Port N_Port Node

Node NL_Port

G_Port F_Port N_Port Node

Node NL_Port

G_Port F_Port N_Port Node

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 64
 Fibre Channel Port Types

• ‘N’ port: Node ports used to connect devices to

N N

switched fabric or point to point configurations.

• ‘F’ port: Fabric ports residing on switches N F

connecting ‘N’ port devices

NL FL

• ‘L’ port: Loop ports are used in arbitrated loop

configurations to build networks without FC L L

switches. These ports often also have ‘N’ port

capabilities and are called ‘NL’ ports.
E E

• ‘E’ port: Expansion ports are essentially trunk ports

used to connect two Fibre Channel switches
• ‘GL’ port: A generic port capable of operating as
either an ‘E’ or ‘F’ port. Its also capable of acting in
an ‘L’ port capacity. Auto Discovery.
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 65
 Inter-Switch Link (ISL)

• The interconnection between switches is called the

Inter-Switch Link (ISL)
E_Port to E_Port

• Supports all classes of service

Class 1, 2, 3, and a special Class F

• FC-PH permits consecutive frames of a sequence to

be routed over different ISL links for maximum
throughput
E E

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 66
 Other standard and Cisco Specific
Port Types

• TE Trunking E-Port that to carry Virtual SAN’s

• Fx Ports found on Over-subscription Port
• SD Span Destination port for Analyzer
• ST Span Tunnel port used in Remote Span
function
• TF Trunking F-Port
• TL Translation Loop (Private to Public address
Translation)

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 67
 N_Port

Device

Host/ Device
Host Interface

FC
N_Port

Can be more then Serial Data Out Serial Data In

one N_Port on the
device

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 68
 Fibre Channel Link

• A link consists of
2 unidirectional “fibers” transmitting in opposite directions
May be either:
Optical fiber or Copper

• Transmitters may be:

Long wave laser
Short wave laser
LED
Electrical

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 69
 Node

• The equipment which contains one or more

• N_Port or NL_Port (topology dependent)
• May be:
Computer (HBA)
Controller (Port on Disk Subsystem)
Device (SCSI FC to Parallel converter)
• Is NOT a switch fabric device

Controller

N_Port N_Port N_Port N_Port

Link Link Link Link

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 70
 FC Communications Model

• Point to point
• N_Port to N_Port
• Flow controlled
• Acknowledged Node Node

Host Host
Node
RX
N_Port
FC TX N_Port
FC

TX
RX

Link

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 71
 Fibre Channel Fabric
Fabric
• The entity which interconnects N_Ports
• Provides routing based on destination address
• Fabric may be:
Point to point: No routing required
Arbitrated loop: Routing is distributed throughout
attached L_Ports
Switched: Routing provided by switch

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 72
 Point to Point

Node Node
N_Port N_Port
Transmitter
Receiver

Receiver Transmitter

Communications Model
• Source to destination
• Host Bus Adaptor (HBA) keeps in memory a Port Login Table
of each N_Port known to it on the network

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 73
 Arbitrated Loop
NL_Node
“A”

Link

NL_Node
“B”

Communications Model—Source to Destination Based on

Address Routing Distributed in the NL_Ports on the Loop

A B Still Pt. to Pt. Communication

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 74
 Switched Fabric

N_Port N_Port N_Port N_Port N_Port N_Port

A B

Switch

Fabric

Communications Model—Source to Destination Based on

Address Routing through the Fabric. Still a Pt. to Pt.
connection. FSPF routing required when more then 2
switches make up the Fabric
A B
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 75
 FC Operational
Characteristics

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 76
 Worldwide Names

• Each switch element is assigned a WWN at time of

manufacture
• Each switch port is assigned a WWN at the
time of manufacture
• During FLOGI the switch identifies the WWN
in the service parameters of the accept frame and
assigns a Fibre Channel ID (FCID)
• These address assignments can then correlate
each fabric port with the switch element

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 77
 Fabric Addressing

• The 24 bit FCID address is partitioned into 3 fields

Device
Area
Domain

• This partitioning helps speed up routing

• Switch element assigns the address to N_Ports
• Address portioning is transparent to N_Ports

8 bits 8 bits 8 bits

Switch Topology
Switch Domain Area Device

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 78
 Directory Server

• Repository of information regarding the

components that make up the Fibre
Channel network
• Located at address ‘FF FF FC’ (Some readings
call this the name server)
• Components can register their characteristics
with the directory server
• An N_Port can query the directory server for
specific information
Query can be the address identifier, WWN and volume
names for all SCSI targets

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 79
 Fabric Controller

• Each switch has a fabric controller

• Assigned address ‘FF FF FD’
Every fabric controller in the fabric has the same address
It is the N_Port within the switch
Responsible for managing fabric, initialization, routing,
setup and teardown of Class-1 connections

• Responsible to receive request and generate

responses for the switch fabric
Information must be consistent independent
of which fabric controller responds to a request

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 80
 Extended Link Services

• Extended link services provide a set of protocol

functions used by the port to specify a function or
service at another port
Usually sent from N_Port to F_port to perform
needed request
The R_CTL field of the first word will be set to 0x22 to
indicate an extend link service request
Many ELS services will return a payload in response some
have no reply

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 81
 Extended Link Services

Some of the more important and most used ELS

commands are:
• FLOGI F_Port Login
• PLOGI N_Port Login
• FAN Fabric Address Notification
• PRLI Process Login
• PRLO Process Logout
• SCN State Change Notification
• SCR State Change Registration
• RSCN Registered State Change Notification

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 82
 Fabric Configuration: PS Selection

• A principal switch shall be selected whenever at

least one inter-switch link (A link between two
E_Port) is established
• The selection process chooses a principal switch,
which is then designated to assign domain
identifier to all the switches in the fabric, and any
who join later the fabric later on
• The principal switch selection can be triggered by
anyone of the following events
Switch boot and Exchange Fabric Parameters (EFP)
Build Fabric (BF)
Reconfigure Fabric (RCF)
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 83
 Fabric Configuration
Disruptive/Non-Disruptive

• One of the following three conditions can trigger BF (non-

disruptive) or RCF (disruptive)
Two disjoint fabrics are combined together
A principal ISL fails (upstream or downstream)
A switch with Domain_ID request for another Domain_ID
• Whenever a switch receives a BF/RCF, the switch starts
F_S_TOV timer and enters the BF/RCF state; It forwards
BF/RCF out of all E_ports except the incoming port (only
once) and wait for the timer to expire
• When the timer expires, BF/RCF propagation state
is left and principal switch selection begins
• BF is not a disruptive process
• RCF is a disruptive process
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 84
 Fabric Routing: FSPF

• For FSPF a domain ID identifies a single switch

This limits the max number of switches that can support
in the Fabric to 239 when FSPF is supported
Treat each Cisco MDS VSAN as separate Fabric each
potentially supporting 239 switches

• FSPF performs hop-by-hop routing

Each Cisco MDS VSAN runs it’s only FSPF process
Routing between VSAN’s is done with Inter VSAN
Routing (IVR)

• FSPF supports hierarchical path selection

Provides the scalable routing tables in large topologies

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 85
 Fabric Routing: FSPF

• Everyone says HELLO to their neighbor, on all

initialized ISLs
• The neighbors say HELLO back, unless they
are dead
• When the HELLO packet is received with both
originator and recipient domain id, the two way
communication is done and:
The ISL is active
The ISL may be available as a two-way path for frames

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 86
 FSPF Characteristics

• Uses FSPF as the routing algorithm

• FSPF routes traffic based on destination domain ID
• FSPF uses total cost as the metric to determine
most efficient path
• Static routes can be applied

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 87
 Fibre Channel Fabric Zoning

• Fibre Channel zoning is used to restrict access

within the fabric to certain nodes/resources
• Zones may overlap devices

Fabric Switch

Node NL_Port FL_Port E_Port E_Port

Zone_A

F_Port N_Port Node

Node NL_Port
Zone_B

F_Port N_Port Node

Node NL_Port

F_Port N_Port Node

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 88
 Fibre Channel Fabric Zoning

• Zoning operation
Zone members “see” only other members of the zone
Zones can be configured dynamically based on WWN
Devices can be members of more than one zone
FC-AL zoning allows the creation of private loops on a single hub
Switched fabric zoning can take place at the port or device level
Based on physical switch port
SAN
Based on device WWN Disk2 Disk3
Disk1
Based on LUN ID ZoneA Host1
ZoneC
• Benefits
Disk4 Host2
ZoneB
Secured device access
Allows operating system co-existence
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 89
 Fibre Channel Basics Summary

• Fibre Channel is a very robust, hierarchical standard

• Fibre Channel utilizes a Point-to-Point communications model
irrespective of the topology
• Fibre Channel includes a full set of services for naming,
addressing, building, and managing fabrics
• Fibre Channel utilizes FSPF, an OSPF like routing protocol to
route traffic
• Fibre Channel Zoning is a method of logically grouping
devices within a given fabric

We have only scratched the surface of

FC Protocol here!
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 90
 Where Do I Go from here ?
√ SAN-2501: Fibre Channel Storage Area Network Design
√ SAN-2602: SAN Migration and Interoperability
√ SAN-2604: SAN Extension Design and Operation
√ SAN-2605: iSCSI SAN Design and Operation
√ SAN-2606: SAN Virtualization
√ SAN-3607: Advanced SAN Design, Virtual Fabric and Fabric Routing
√ SAN-3608: Advanced SAN Troubleshooting
√ LABSAN-2251, 2252, 2253, 2254, and 2265: Cisco MDS 9000 Labs
√ TECSAN-2601 SAN Fundamentals, Protocols, and Architecture

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 91
 Q and A

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 92
 Recommended Reading

• Continue your Cisco Networkers

learning experience with further
reading from Cisco Press
• Check the Recommended
Reading flyer for suggested
books

Available Onsite at the

Cisco Company Store

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 93
 Complete Your Online Session Evaluation

• Win fabulous prizes; Give us your feedback

• Receive ten Passport Points for each
session evaluation you complete
• Go to the Internet stations located
throughout the Convention Center to
complete your session evaluation
• Drawings will be held in the
World of Solutions
Tuesday, June 20 at 12:15 p.m.
Wednesday, June 21 at 12:15 p.m.
Thursday, June 22 at 12:15 p.m. and 2:00 p.m.

SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 94
SAN-1501
12686_05_2006_c2 ©
© 2005
2006 Cisco
Cisco Systems,
Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 95