SAN PHYSICAL DESIGN: CABLING AND CONNECTIONS

Cable Types

An Optical Fiber consists of a very thin glass core surrounded by a cladding. The cladding has a lower
refractive index than the core, so the boundary between core and cladding acts like a mirror and
reflects light down the core. The light source is usually infra-red and might not be visible to the human
eye. Optical Fibers come in two modes; single mode and multi mode with the thicknes if the light pipe
measured in microns. Popular sizes are 9 (singe mode), 50 and 62.5 (multi-mode) microns. In general
the thinner the cable, the further the light goes. In laymans terms, single mode fiber is thin enough to
restrict the light, so that it does not refract, or bounce off the fiber walls. This means that only one light
wavelength can go down the pipe, but it suffers less interference and so can travel further without
losing effective signal strength. Single mode also uses light emitters with a longer wavelength than
multi-mode. In multi-mode fibres, the light can refract off the walls, so several wavelengths and paths
are available. Single mode is more expensive than multi-mode and is typically used where
transmission speed is essential, for example for multiplexed remote syncronous data mirroring.

If you are cabling up new kit to existing cables, then be aware that while you can join 50 micron multi-
mode to 62.5 micron multi-mode, you will get some light loss when going from 62.5 to 50. Some of the
light will leak away around the edges. If at all possible, you should keep a consistent cable size
throughout a path. 62.5 micron cables are generally not used for new installations and are becoming
obsolete.

The transmission speed and acceptable distance of a fiber depends on a number of factors, including
light source and cable characteristics. In general, a single-mode 9 micron cable can trasmit up to
50km and a multi-mode 50 micron cable can transmit up to 900m.
One fiber term you will come across is 'dark fiber', usually used as if it is some special kind of fibre.
When utility companies lay fiber in the ground, they always install more than is required for current
needs, as the installation is by far the most expensive cost. This spare fiber has no light going down it,
so it is 'dark'. And one other thing, I'm English and I like to use correct English spelling. Industry
standards are that the architecture is called Fibre Channel and the cables are called fibers, which
should explain why I spell Fibre two different ways.

Most machine halls use structured cabling these days, where individual fiber pairs are aggregated
together into a cable, and terminated on a patch panel. Jumper cables are then used to connect
appliances to the patch panel. Every connection will result in some light loss, so there are limits on
how many patch panels can be in a circuit. It's best to check with individual vendors for details.

Connections

The most common types of fiber optic connectors that are available now are SC, LC, ST, and MT-RJ.
Most of these types of connectors can be used with either multimode or single mode fiber.
There is some dispute as to what SC stands for. I think it was originally 'Siemens Connector', but the
official term now is 'Subscriber Connector'. Other variants are 'Square', 'Small', 'Set and Click' or 'Stab
and Click'. SC fits into a GBIC (Gigabyte interface connector) and has a square molded plastic body.
To connect it you push it into the GBIC until it locks. It is larger than LC and is generally used for
slower speed fibers.
LC stands for 'Lucent Connector' and plugs into SFPs (Small Form Factor Pluggable). LC for are used
for faster speed fibers where space is important. SPF+ is an extension of this technology and
originally supported speeds up to 10 Gbit/s.
ST stands for Straight Tip and is a quick release bayonet style connector with a twist lock coupling.
They used to be common, but are being replaced by SC connectors.
MT-RJ stands for Mechanical Transfer Registered Jack and is very popular for small form factor
devices due to its small size. Housing two fibers and mating together with locating pins on the plug,
the MT-RJ comes from the MT connector, which can contain up to 12 fibers.
As mentioned above, trunk cables are used to connect patch panels within data centres, and there is
a specification for the connectors used by these cables. Trunk cables are terminated with MPO or
Multi-fiber Push On connectors and are typically available with 8, 12 or 24 fibe cores.
MTP or 'Multi-fiber Termination Push-on' connector is a brand name for an MPO connector
manufactured by US Conec.

Other types of connector include HSSDC (High-Speed Serial Data Connection) and DBm/DBf.
When copper cables and copper-based Fibre Channel devices are used in a SAN, they are attached
to other Fibre Channel devices using two types of connectors, Copper Gigabit Interface Connectors
(GBICs) and Media Interface Adapters (MIAs). Copper GBICs are hot pluggable connectors that
attach to Fibre Channel devices using either a DB-9 or the High Speed Serial Data Connector
(HSSDC). Media Interface Adaptors have a standard DB9 serial plug on one end and an SC socket
on the other end. They are used to convert an electrical signal to optical.

Fiber cables come as a pair of fibers, and each end will have a send and a receive fiber, known as TX
and RX. An SC plug will only fit one way into a GBIC socket, and the most common error I've come
across when recabling SAN switches is that the fiber pair is crossed. This means that the LED on one
side is shining at the LED on the other side, and nothing is working. You will also see that the
transmission light is orange instead of green. In this case, it is reasonably easy to unclip the two fibers
from the plug and swap them over. However take care not to break the glass fiber itself.
The picture below shows the two common fiber connectors

Another parameter you might come across is polish type. There are three types of polishes, 'PC' or
Physical contact, 'UPC' or Ultra Physical contact and 'APC' or Angled Physical contact. These are
important as each polish type reflects light off the end of a fiber connector in a different way. If the
types of polish are not matched correctly then a lot of the light signal can be lost.
APC connectors come with the fiber end face polished at an angle to prevent light that reflects from
the interface from traveling back up the fiber. Angle-polished connectors should only be mated to
other angle-polished connectors. You can usually identify angle-polished connections as they have
either a green strain relief boot or a green connector body. Also, the name of the connector might
have '/APC' (angled physical contact) added to the end. Two different versions of FC/APC exist,
FC/APC-N (NTT) and FC/APC-R (Reduced). An FC/APC-N connector key will not fit into a FC/APC-R
adapter key slot.

Port types
There are several types of device ports:

 N_PORTs are end-points in a FC network and are used to connect servers to storage devices
to fibers.
 F_PORTs or Fabric Port, used to connect fibers to switches, so typically a fiber will be
connected to a device using an N_PORT on one end, then to a switch using an F_PORT on
the other end. An F-PORT can only connect to one device.
 An L-PORT is a loop capable node or switch port.
 An NL-PORT or Node Loop Port is used to attach devices to a FC-AL arbitrated Loop.
 An Fx_Port is a switch port capable of operating as either an F_Port or FL_Port, that is, it can
support both loop and non-loop functionality.
 An FL_Port is a fabric port used for connection to a loop. This enables loop nodes to link to
fabric nodes.
 An E-PORT or Extension Port is used for ISL links, so the other end will be an E-PORT on
another switch
 Ports can also be in unassigned status, when they are G-PORTS or U-PORTS. A G-Port is
assigned but the link initialisation is not complete. A U-Port is unassigned and not connected.
 An EX-Port connects an FC router to an edge fabric but does not allow the fabrics to merge.
 A VE-Port is a virtual port which emulates an E-Port over FCIP
 A VEX-Port is an EX_Port, except the underlying transport is IP rather than FC.
 A TE-Port is a trunking E-Port which allows routing of multiple VSANs.
 U_Port (Universal port) is port waiting to become another port type

The following types of ports are also used in Fibre Channel:

Node and Port Addresses

A Node is an appliance that is connected to a fabric SAN and every Node has a unique 64-bit address
called 'World Wide Node Name'. Every Port also has a unique 64-bit address called the 'World Wide
Port Name'. These addresses are usually writen as a sequence of 8 hex bytes separated by colons
like this 10:00:00:60:69:50:60:02. Bytes 3-5 are assigned to each vendor by the IEE naming
standards body. Every Node as a unique WWNN node name, and this can mean individual HBAs,
switches, storage units, storage arrays, tape libraries and tape drives. Every port on every node has a
unique WWWPN port name.
This unique set of names means that it is possible to specify exactly which port in which switch or
HBA that you need to address, pretty much in the same way you can use a telephone to reach a
person.

Buffer Credits

When a data record is transmitted down a Fibre Channel record it is split up into a number of varying
size blocks or frames. These blocks are transmitted through the fabric, then re-assembled at the far
end to create the data record again. Switches are store-and-forward devices, they do not just pass the
data straight through. To make sure the ISL fibers are used efficiently, every switch port has a number
of buffers associated with it, called buffer credits or BB-Credits. The switch can then store several
blocks of incoming data, while waiting to pass it on to the next node. When a receiver is ready to take
information, it signals to its sender, then decrements the BB-Credit. When the data block is passed on
to the next receiver the BB-Credit is incremented again. This means that BB-Credits are also used to
throttle back the data transmission flow when devices or links get too busy.
Buffer credits are consumed by frames, not data. A large frame and a small frame will both use one
buffer, so it's frame count, not the amount of data that matters.
EE-Credits or End to End credits are established between two communicating N-Ports and control the
overall flow of the data stream through the fabric.
On a Brocade switch, you can check status of these buffers with the command 'portbuffershow'. This
will tell you, among other things, how many buffers are allocated to each port, how many are in use,
and how many are needed for efficient channel usage.
Record Keeping

Record keeping is the boring, but absolutely essential part of installing and managing a SAN. A good
set of records makes it much easier to fix things when they go wrong.
At a minimum, you should have a diagram that shows every port on every switch, what type of port it
is and what it is connected to. The good news is that most SAN management packages will produce
this documentation for you, but it is important that you extract a fresh copy every time you make a
change. Export it off onto a laptop, or even print it out, so you can take it into the machine hall with
you.

Storage Area Networks

 SAN Theory
 Swiches and Fabrics
 Cables and Connectors
 SAN zoning design
 Hints and Tips
 Virtual SANs
 IBM SVC
 SVC config.
 SVC zoning
 EMC VPLEX
 VPLEX Configuration
 VMware virtual SAN

Disk Protocols

 NVMe protocol
 Parallel and Serial ATA
 Serial ATA 2
 SCSI

Lascon updTES

I retired 2 years ago, and so I'm out of touch with the latest in the data storage world. The Lascon site
has not been updated since July 2021, and probably will not get updated very much again. The site
hosting is paid up until early 2023 when it will almost certainly disappear.
Lascon Storage was conceived in 2000, and technology has changed massively over those 22 years.
It's been fun, but I guess it's time to call it a day. Thanks to all my readers in that time. I hope you
managed to find something useful in there.
All the best

back to top
DISCLAIMER - By entering and using this site, you accept the conditions and limitations of use

Click here to see the Full Site Index       Click here to see the Cookie Policy       Click here to see
the Privacy Policy                             ©2019 and later, A.J.Armstrong