ShamirSharma Final Best Practices and Deployment Network iSCSI

Best Practice and Deployment of the Network
for iSCSI, NAS and DAS in the Data Center

Samir Sharma, Juniper Networks
Author: Samir Sharma, Juniper Networks

SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise
noted.
Member companies and individual members may use this material in presentations
and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without
modification
The SNIA must be acknowledged as the source of any material used in the
body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Neither the author nor the presenter is an attorney and nothing in this
presentation is intended to be, or should be construed as legal advice or an opinion
of counsel. If you need legal advice or a legal opinion please contact your attorney.
The information presented herein represents the author's personal opinion and
current understanding of the relevant issues involved. The author, the presenter,
and the SNIA do not assume any responsibility or liability for damages arising out of
any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved. 2
Abstract
Best Practice and Deployment of the Network for

iSCSI, NAS and DAS in the Data Center
Low Latency, high bandwidth, lossless, spanning tree, IO
Convergence, Layer 2, Layer 3, Storage Resource
Management ,Two Tier, Three Tier, and Flat… when
deploying iSCSI, NAS and DAS what should the network
topology be? How can network topology affect Large,
Medium and Small Scale network deployments? In this
presentation we will discuss best practice and
deployments of the network for iSCSI , NAS and DAS in
the Data Center.
Agenda
1 Scope of this presentation?
2 Definitions – The Storage Puzzle
3 Benefits and where are they being deployed?
4 Lossless or Lossy … That is the Question…
5 Ideal and Non Ideal Ethernet Storage Network Deployments
6 Mapping the Deployment to what is Ideal?
Agenda
1 Scope of this presentation?
Scope of this presentation
Background
?
The pieces of the puzzle that make up Ethernet network
storage deployments.
Topologies
Ethernet Network topologies that are conducive
to storage deployments.
Summary and final thoughts.
Agenda
2 Definitions – The Storage Puzzle
6
Definitions – The Storage Puzzle
File Level Storage
File Level storage : is the most common storage system that we find with
our hard-drives, NAS systems, etc. In this type of storage, the storage disk
is configured with a particular protocol (Like NFS, etc) and files are stored
and accessed from it as such, in bulk.
Advantages of File Level Storage System:

• File level storage system is simple to implement and simple to use.
• It stores files and folders and is visible as such, to both the systems storing the files and
the systems accessing it.
• File level storage systems are generally inexpensive, when compared to block level
storage systems.
• File level storage systems are more popular with NAS based storage systems –
Network Attached Storage.
• They can be configured with common file level protocols like NTFS (Windows), NFS
(Linux), etc.
• File level storage systems are well suited for bulk file storage.
• The file level storage device itself can generally handle operations like access control,
integration with corporate directories, etc.
Block Level Storage
Block level storage : is raw blocks (storage volumes) are created and each
block can be controlled like an individual hard drive. Generally, these
blocks are controlled by the Server based Operating Systems. Each block/
storage volume can be individually formatted with the required file system.
Advantages of Block level storage systems:
• Block level storage systems offer a better performance/ speed than file level storage
systems.
• Each block / storage volume can be treated as an independent disk drive and are
controlled by external Server OS.
• Each block / storage volume can be formatted with the file system required by the
application (NFS / NTFS / SMB , etc).
• Block level storage systems are very popular with SAN – Storage Area Networks.
• Block level storage systems are more reliable, and their transport systems are very
efficient.
• Block level storage can be used to store files and also provide the storage required for
special applications like Databases, VMFS (Virtual Machine File Systems), etc.
• They can support external boot-up of the systems connected to them.
Direct-attached Storage (DAS)
Direct-attached storage (DAS) refers to a digital storage
system directly attached to a server or workstation, without a storage
network in between.
Protocols used in DAS: ATA, SATA, eSATA, SCSI, SAS, and Fibre
Channel.
Pros: Cheep, Compact great for very small business.

Con: It is hard to share data or unused resources
with other servers.
Network-attached storage (NAS)
Network-attached storage (NAS) is file-level computer data

storage connected to a computer network providing data access
to heterogeneous clients. It provides file-based storage.
Protocols used in NAS: uses file-based protocols NFS, SMB/CIFS, or

AFP. NAS units rarely limit clients to a single protocol.
Pros: Often a striped down OS and hardware is needed.

Cons: Typically not conducive for large scale deployments.
Internet Small Computer System Interface (iSCSI)
Internet Small Computer System Interface (iSCSI)- is a transport for iSCSI

using TCP/IP (typically native 1GbE, 10GbE, 40GbE and 100GbE)
Pros:
• Cost effect components, hard drives, servers, network cards, cables, switches, optics..
• Leverages TCP/IP to guarantee lossless traffic and in order frame delivery.
• Built in security, authentication with RADIUS servers and can leverage IPSec, MACSec.
Cons:
• Rely on overlay protocols to get lossless and in order frame delivery. Causing
complexity and overhead in the network.
Fibre Channel over Ethernet (FCoE)
Fibre Channel over Ethernet (FCoE) is an encapsulation of Fiber

Channel frames over Ethernet networks.
Why is this important?

FCoE will carry FC frames (eg SCSI commands) directly over Ethernet.
This is one of the key drivers that enables SAN LAN convergence.
Ethernet
Ethernet HD FC Frame
FCS
FCoE Frame = Ethernet Frame that has an FC frame inside it.
Agenda
3 Benefits and where are they being deployed?
Benefits of Ethernet Storage
COST: In-order/ Lossless:
Transport ultimately more TCP/IP = In-order,

economical than other guaranteed delivery
storage transports. DCB = Lossless, traffic
seperation
SPEED: EQUIPMENT
Higher bandwidth More cost effective

• 10GbE • Storage equipment
• 40GbE • Servers
• 100GbE • Networking equipment
Where is it being deployed?
FCoE Converged Access Large
End to end FCoE &

Medium Business
iSCSI
iSCSI Small Business
Agenda
4 Lossless or Lossy … That is the Question…
Lossless or Lossy.. That is the Question….
Very Important Rule:

Thy shall never loose or drop a storage frame.
Lossless Network Topology:

No Frame drop between initiator and target.
Lossy Network Topology:

The network allows for frame drop within the
network and rely on overlay protocols to handle
frame drop (eg TCP/IP).
Realities of Traditional Ethernet in the Data Center
Possible Problem Areas
• Traditionally not meant to transport lossless traffic
Frame loss
• Forced to rely on protocols like TCP/IP to achieve a lossless topology
-> Resulting in protocol overhead/congestion due to retransmission.
(iSCSI used TCP/IP)
• Multi-tier
Network • Multiple device to manage

• Spanning Tree adds inefficiencies 50% of links blocked
• Heavily Oversubscribed 10:1, 5:1, 4:1 take your pick……
Topologies
Network • Mix Traffic Types

• Video Running on Ethernet
• Voice Running on Ethernet
Congestion • Traditional Applications Running on Ethernet

• Now add Storage and Server Virtualization to the mix……
© 2012 Storage Networking Industry Association. All Rights Reserved.
Storage over Ethernet
What do you need?
Relying TCP/IP for the Solution….
Lossless • Solution : If a frame is dropped

TCP/IP will retransmit
• Need: No frame drops
Network between initiator and target
• Result: This adds latency and can
Topology exacerbate congestion
• Need: Frames must be in • Solution : TCP/IP guarantees in-

In order frame order between initiator to order delivery per session
target
delivery
Fibre Channel’s Solution….
• Uses a buffer credit mechanism to

Lossless Network • Need: No frame drops create a lossless network.
Topology/ In order between initiator and target • Eliminates frame drop and frame
frame delivery retransmission
Solution : Map storage flows to PFC
Creates a lossless network PFC and

DCB Lossless
Ethernet No need for retransmission
(dcbx allows the creation of arbitrary (Closest way to emulate FC Credit
application tlv’s based on higher level Storage Mechanism)
protocol to easily capture and
configure for cifs,nfs etc …) Fabric
Mapping Different Flows on Separate Priority…

Result True Traffic Separation
Agenda
5 Ideal and Non Ideal Ethernet Storage Network Deployments
Realities of Ethernet in the Data Center
The challenges Multi-tier legacy network
 Too slow Unnecessary

N layers add hops
 Too expensive and latency
 Too complex Up to 50% of the

ports interconnect
switches,
Network Complexity not servers or
TCP/IP can’t scale storage
W Up to 75% of traffic E
Spanning Tree
disables up to 50%
S of bandwidth
Complexity
Scale
Non-Ideal Network Topology
Ethernet L3
Core Switch/Router
Ethernet L2/L3
Aggregation Switch
Ethernet L2/L3
Access Switch
POD 1 POD 2 POD 3 POD 4
Non-Ideal Network Topology
Ethernet L3
Core Switch/Router
Ethernet L2/L3
Aggregation Switch
Ethernet L2/L3
Access Switch
POD 1 POD 2 POD 3 POD 4
The Network gets in the way…
• 7 switches
• 10 chances to drop the frame
• 7 devices to manage….
Summary of issues..
Three Tier design (access, aggregation and core)

design flaws:
To many hops add latency.
Spanning Tree removes ~50% of the links thus increasing

congestion.
Lossy fabric rely on overlay protocols like TCP/IP this adding

to congestion.
Simple solutions..
Three Tier design (access, aggregation and core)

design flaws:
To many hops add latency.

Solution Flatten the Network Topology
Spanning Tree removes ~50% of the links thus increasing
congestion.
Use overlay protocols like MC-LAG
Lossy fabric rely on overlay protocols like TCP/IP this adding
to congestion.
Create a lossless fabric by mapping all storage
flows to a lossless queue (use DCB)
Traditional Network Topology
Storage Array
Aggregation
Layer
DCB Enabled
Access Layer
DCB Enabled
Servers
Ideal Network Topology
One way……..
Storage Array
Spine
40GbE/100GbE
MC-LAG
Leaf
10GbE/40GbE
Servers Connected to Leaf/Access Layer
Use MC-LAG to remove the ills of Spanning Tree….
Ideal Network Topology
Desired way..
Flat Fabric
• Everything is connected to everything

(Pooling all resources together)
• Access is always 1 hop away.
• Removing the ills of Spanning Tree….
Agenda
6 Mapping the Deployment to what is Ideal?
Hadoop Network Topology
One way
Spine
40GbE/100GbE
MC-LAG
Leaf
10GbE/40GbE
Hadoop Network Topology
Desired Way….
Flat Fabric
Note
• No need for STP everything
is connect to everything else
one hop away.
• Best Latency, least amounts of hops.
Mixing DAS, NAS, iSCSI and FCoE
FC SAN
DCB Enabled Fabric FC SAN
Example: DCB Enabled Fabric/Network
Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric
Note
• Keep Traffic separated on different priorities
MC-LAG on all network devices.
Eg) Priority 1 = Tape Traffic (on all switches)
Priority 2 = iSCSI Traffic (on all switches)
Priority 3 = FCoE Traffic (on all switches)
Priority 4 = NAS Traffic (on all switches)
Example: DCB Enabled Fabric/Network
Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric
DCB Enabled Fabric
Note
• Keep Traffic separated on different priorities
on network device.
Eg) Priority 1 = Tape Traffic (on all switches)
Priority 2 = iSCSI Traffic (on all switches)
Priority 3 = FCoE Traffic (on all switches)
Priority 4 = NAS Traffic (on all switches)
Flat Topology:
Easy to do operationally when there is only
one device to manage……
Remote office Data Replication
LOSSLESS LOSSLESS LOSSLESS
Backup
Data Center
Data Center
Frames in-fight
PAUSE
Things to think about…
• Does your WAN devices have enough buffer to handle frame

that are in-flight after pause is sent.
• Distance correlates directly to the amount of

Campus buffer needed.
LOSSLESS LOSSY with LOSSLESS

Protocol Overlay
Backup
Data Center
Data Center
TCP/IP
Note
• WAN Routers should be able buffer several flows
to account for frame drop and ensure retransmission.
Campus
LOSSLESS LOSSY with LOSSLESS

Protocol Overlay
Backup
Data Center
Data Center
MPLS/VPLS
Note
• WAN links are dedicated to just storage traffic
and no other traffic.
• Speed is pre-negotiated to ensure no packet drop.
Campus
Final Thoughts
Loop in the Network (Spanning Tree)
Flatten your fabric to remove loops, this allows better pooling of Ethernet
storage.
Or Use protocols like MC-LAG (Spin-Leaf) to remove loop.
Lossless Fabric/Networks
Map flows on the same priority on all network devices.
For long distances be mindful of the amount of buffer on the network
device. Else frame loss will be introduced.
Lossy Fabric/Networks
Use a protocol like TCP/IP to ensure proper frame delivery.
If you are brave and just want to rely on Ethernet. (Not suggested)
Dedicated to just storage traffic and no other traffic.
Pre-negotiated speed to ensure no packet drop.
Q&A / Feedback
Many thanks to the following individuals

for their contributions to this tutorial.
- SNIA Education Committee
Joseph White Kishore Inampudi

Simon Gordon Haruki Sonehara
Andy Ingram
Send any questions or comments on this

presentation to SNIA: tracktutorials@snia.org

ShamirSharma Final Best Practices and Deployment Network iSCSI

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ShamirSharma Final Best Practices and Deployment Network iSCSI

Uploaded by

Copyright:

Available Formats

Best Practice and Deployment of the Network

for iSCSI, NAS and DAS in the Data Center

Author: Samir Sharma, Juniper Networks

Best Practice and Deployment of the Network for

1 Scope of this presentation?

2 Definitions – The Storage Puzzle

3 Benefits and where are they being deployed?

4 Lossless or Lossy … That is the Question…

5 Ideal and Non Ideal Ethernet Storage Network Deployments

6 Mapping the Deployment to what is Ideal?

1 Scope of this presentation?

Summary and final thoughts.

2 Definitions – The Storage Puzzle

Advantages of File Level Storage System:

Pros: Cheep, Compact great for very small business.

Network-attached storage (NAS) is file-level computer data

Protocols used in NAS: uses file-based protocols NFS, SMB/CIFS, or

Pros: Often a striped down OS and hardware is needed.

Internet Small Computer System Interface (iSCSI)- is a transport for iSCSI

Fibre Channel over Ethernet (FCoE) is an encapsulation of Fiber

Why is this important?

FCoE Frame = Ethernet Frame that has an FC frame inside it.

3 Benefits and where are they being deployed?

COST: In-order/ Lossless:

Transport ultimately more TCP/IP = In-order,

Higher bandwidth More cost effective

FCoE Converged Access Large

End to end FCoE &

iSCSI Small Business

4 Lossless or Lossy … That is the Question…

Very Important Rule:

Lossless Network Topology:

Lossy Network Topology:

Possible Problem Areas

• Traditionally not meant to transport lossless traffic

Network • Multiple device to manage

Network • Mix Traffic Types

Congestion • Traditional Applications Running on Ethernet

Relying TCP/IP for the Solution….

Lossless • Solution : If a frame is dropped

• Need: Frames must be in • Solution : TCP/IP guarantees in-

Fibre Channel’s Solution….

• Uses a buffer credit mechanism to

Creates a lossless network PFC and

Mapping Different Flows on Separate Priority…

5 Ideal and Non Ideal Ethernet Storage Network Deployments

The challenges Multi-tier legacy network

 Too slow Unnecessary

 Too complex Up to 50% of the

POD 1 POD 2 POD 3 POD 4

POD 1 POD 2 POD 3 POD 4

The Network gets in the way…

Three Tier design (access, aggregation and core)

To many hops add latency.

Spanning Tree removes ~50% of the links thus increasing

Lossy fabric rely on overlay protocols like TCP/IP this adding

Three Tier design (access, aggregation and core)

To many hops add latency.

Servers Connected to Leaf/Access Layer

Use MC-LAG to remove the ills of Spanning Tree….

• Everything is connected to everything

6 Mapping the Deployment to what is Ideal?

DCB Enabled Fabric FC SAN