Elements of SAN capacity planning

Mark Friedman
VP, Storage Technology markf@demandtech.com (941) 261-8945

DataCore Software Corporation

Founded 1998 - Storage networking Software 170+ employees, private - Over $45M raised
Top

Venture firms - NEA, OneLiberty Funds VanWagoner, Bank of America, etc Intel Business and Technical collaboration agreement

Exec. Team
Proven

Storage expertise Proven Software company experience Operating systems, high-availability, Caching, networking Enterprise level support and training

Worldwide: Ft. Lauderdale HQ, Silicon Valley, Canada France, Germany, U.K., Japan

Overview
How do we take what we know about storage processor performance and apply it to emerging SAN technology? What is a SAN? Planning for SANs:
SAN

performance characteristics Backup and replication performance

08/28/11

Evolution Of Disk Storage Subsystems

Cached Disk Spindles

Strings & Farms

Storage Processors

See: Dr. Alexandre Brandwajn, A study of cached RAID 5 I/O CMG Proceedings, 1994.

Write-thru Cached subsystems

What Is A SAN?
Storage Area Networks are designed to exploit Fibre Channel plumbing Approaches to simplified networked storage:
SAN

appliances SAN Metadata Controllers (out of band) SAN storage managers (in band)

08/28/11

The Difference Between NAS and SAN

Storage Area Network (SAN) designed to exploit Fibre Channel plumbing require a new infrastructure. Network Attached Storage (NAS) devices plug into the existing networking infrastructure.
Networked

file access protocols (NFS, SMB, CIFS) TCP/IP stack

Application: HTTP, RPC Host-to-Host: TCP, UDP Internet Protocol: IP Media Access: Ethernet, FDDI
Packet Packet Packet Packet

08/28/11

NAS devices plug into existing TCP/IP networking support. Performance considerations:
1500

The Difference Between NAS and SAN

Application Interfaces
RPC DCOM Winsock NetBIOS
User Mode Kernel

Named Pipes Redirector NetBT TDI TCP ICMP IGMP IP Filtering UDP Server

MTU TCP requires acknowledgement of each packet, limiting performance.

byte Ethernet

IP

ARP IP Forwarding NDIS

Packet Scheduler NDIS Wrapper

NDIS Miniport NIC Device Driver

08/28/11

Performance considerations:

The Difference Between NAS and SAN

e.g., 1.5 KB Ethernet MTU
Requires processing 80,000 Host interrupts/sec @ 1 Gb/sec or Jumbo frames, which also requires installing a new infrastructure
Which

is why Fibre Channel was designed the way it is!

Source: Alteon Computers, 1999.

08/28/11

Competing Network File System Protocols

Universal data sharing is developing ad hoc on top of de facto industry standards designed for network access.
Sun

NFS HTTP, FTP Microsoft CIFS (and DFS)

also known as SMB CIFS-compatible is the the largest and fastest growing category of data

08/28/11

CIFS Data Flow

Session-oriented: e.g., call backs
Client
MS Word
Redirector Network Interface
SMB Request
System Cache

File Server Server

Network Interface

System Cache

SMB Request

08/28/11

What About Performance?

NFS Client NFS Server

User Process
Remote Procedure Call (RPC)

Client Process

NFSD Daemon
Application: HTTP, RPC Host-to-Host: TCP, UDP

TCP/IP Driver

Response Data

TCP/IP Driver

Internet Protocol: IP
Media Access: Ethernet, FDDI

TCP/IP Network

08/28/11

What About Performance?

Network-attached yields fraction of the performance of direct-attached drives when block size does not match frame size. System Client System File
MS Word
Redirector Network Interface
SMB Request
Cache Cache

Server Server
Network Interface

Application: HTTP, RPC Host-to-Host: TCP, UDP Internet Protocol: IP Media Access: Ethernet, FDDI

SMB Request

See ftp://ftp.research.microsoft.com/pub/tr/tr-2000-55.pdf
08/28/11

What about modeling?

Add a network delay component to interconnect two Central Server models and iterate.

08/28/11

The Holy Grail!

Storage Area Networks
Uses low latency, high performance Fibre Channel switching technology (plumbing) 100 MB/sec Full duplex serial protocol over copper or fiber Extended distance using fiber Three topologies:
Point-to-Point Arbitrated

bridged Fabric: 16 MB addresses

Loop: 127 addresses, but can be

08/28/11

The Holy Grail!

Storage Area Networks FC delivers SCSI commands, but Fibre Channel exploitation requires new infrastructure and driver support Objectives:
Extended

pools Dynamic, hot-plugable interfaces Redundancy, replication & failover Security administration Storage resource virtualization

addressing of shared storage

08/28/11

Distributed Storage & Centralized Administration

Traditional tethered vs untethered SAN storage
Untethered storage can (hopefully) be pooled for centralized administration Disk space pooling (virtualization)
Currently,

using LUN virtualization In the future, implementing dynamic virtual:real address mapping (e.g., the IBM Storage Tank)

Centralized back-up
SAN

LAN-free backup

08/28/11

Storage Area Networks

FC is packet-oriented (designed for routing). FC pushes many networking functions into the hardware layer.
e.g., Packet

fragmentation Routing

Upper Level Protocol SCSI IPI-3 HIPPI IP

Fc4 Fc3

Common Services 8B/10B Encode/Decode 100MB/s Physical Layer

Framing Protocol/Flow ControlFc2

Fc1 Fc0

08/28/11

Storage Area Networks

FC is designed to work with optical fiber and lasers consistent with Gigabit Ethernet hardware
100

MB/sec interfaces 200 MB/sec interfaces

This creates a new class of hardware that you must budget for: FC hubs and switches.

08/28/11

Storage Area Networks

Performance characteristics of FC switches:
low latency ( 1 sec), except when cascaded switches require frame routing Deliver dedicated 100 MB/sec point-to-point virtual circuit bandwidth Measured 80 MB/sec effective data transfer rates per 100 MB/sec Port
Extremely

08/28/11

Storage Area Networks

When will IP and SCSI co-exist on the same network fabric?
iSCSI Nishan Others?

Upper Level Protocol SCSI IPI-3 HIPPI IP

Fc4 Fc3

Common Services 8B/10B Encode/Decode 100MB/s Physical Layer

Framing Protocol/Flow ControlFc2

Fc1 Fc0

08/28/11

Storage Area Networks

FC zoning is used to control access to resources (security) Two approaches to SAN management:
Management

functions must migrate to the switch, storage processor, or. OS must be extended to support FC topologies.

08/28/11

Approaches to building SANs

Fibre Channel-based Storage Area Networks (SANs)
SAN

appliances SAN Metadata Controllers SAN Storage Managers

Architecture (and performance) considerations

08/28/11

Approaches to building SANs

Where does the logical device:physical device mapping run?
Out-of-band:

on the client In-band: inside the SAN appliance, transparent to the client

Many industry analysts have focused on this relatively unimportant distinction.

08/28/11

SAN appliances
Conventional storage processors with Fibre Channel interfaces Fibre Channel support
FC

Fabric Zoning LUN virtualization

08/28/11

SAN Appliance Performance

Same as before, except faster Fibre Channel interfaces
Commodity
Multiple Processors

processors, internal buses, disks, frontend and back-end interfaces Proprietary storage processor architecture considerations

Internal Bus

Host Interfaces

FC Interfaces

Cache Memory

08/28/11

FC Disks

SAN appliances
SAN and NAS convergence?
Adding

Fibre Channel interfaces and Fibre Channel support to a NAS box SAN-NAS hybrids when SAN appliances are connected via TCP/IP.

Current Issues:
Managing

multiple boxes Proprietary management platforms

08/28/11

SAN Metadata Controller

SAN clients acquire an access token from the Metadata Controller (out-ofband)
1 2
SAN SAN clients then access disks directly Metadata Controller using proprietary distributed file system

SAN Clients SAN Clients

Token

3
Fibre Channel

Pooled Storage Pooled Storage Resources Resources

08/28/11

SAN Metadata Controller

Performance considerations:
MDC

latency (low access rate assumed) Additional latency to map client file system request to the distributed file system

Other administrative considerations:

Requirement

for client-side software is a burden!

08/28/11

SAN Storage Manager

Requires all access to pooled disks through the SAN Storage Manager (in-band)!
Storage Domain Servers

SAN Clients SAN Clients

Fibre Channel

Pooled Storage Pooled Storage Resources Resources

08/28/11

SAN Storage Manager

SAN Storage Manager adds latency to every I/O request How much latency is involved?
SAN Clients SAN Clients

Fibre Channel Storag e Domai n Server s

Can this latency be reduced using traditional disk caching strategies?

Pooled Storage Pooled Storage Resources Resources

08/28/11

Architecture of a Storage Domain Server

Runs on an ordinary Win2K Intel server The SDS intercepts SAN I/O requests, impersonating a SCSI disk Leverages:
Native

Client I/O

SANsymphony Storage Domain Server

Initiator/Target Emulation FC Adaptor Polling Threads Security Fault Tolerance Data Cache

Natives W2K I/O Manager

Disk Driver Diskperf (measurement) Fault Tolerance (Optional) SCSI miniport Driver Fibre Channel HBA Driver

Device drivers Disk management Security Native CIFS support

08/28/11

Sizing the SAN Storage Manager server

In-band latency is a function of Intel server front-end bandwidth:
Processor

speed Number of processors PCI bus bandwidth Number of HBAs

and performance of the back-end Disk configuration

08/28/11

SAN Storage Manager

Can SAN Storage Manager in-band latency be reduced using traditional disk caching strategies?
Read

hits Read misses

Disk I/O + (2 * data transfer)
Fast

Writes to cache (with mirrored caches)

2 * data transfer Write performance ultimately determined by the disk configuration

08/28/11

SAN Storage Manager

Read hits (16 KB block): Timings from an FC hardware monitor 1Gbit/s Interfaces
SCSI Read Command Length = 4000 140 sec

16x1024 Byte Data Frames

27 s ec

Status Frame

No bus arbitration delays!

08/28/11

Read vs. Write hits (16 KB block)

Fibre Channel Latency (16KB Blocks)

SCSI Command

Write Setup

Data Frames

SCSI Status

08/28/11

Decomposing SAN in-band Latency

How is time being spent inside the server?
PCI bus? Host Bus adaptor? Device polling? Software stack?

SCSI Command Write Setup Data Frames SCSI Status

08/28/11

Benchmark Configuration
4-way 550 MHz PC
Maximum

of three

64bit/33MHz PCI

32bit/33MHz PCI

08/28/11

32bit/33MHz PCI

FC interface polling threads 3 PCI buses (528MB/s Total) 1, 4, or 8 QLogic 2200 HBAs

4x550MHz XEON Processors

Memory Bus

Decomposing SAN in-band Latency

How is time being spent inside the SDS? PCI bus? Host Bus adaptor? Device polling:
1

CPU is capable of 375,000 unproductive polls/sec

2.66 secs per poll

Software stack:
3

CPUs are capable of fielding 40,000 Read I/Os per second from cache

73 secs per 512-byte I/O

08/28/11

Decomposing SAN in-band Latency

SANsymphony in-band Latency (16KB Blocks)

SDS

FC Interface

Data Transfer

08/28/11

Impact Of New Technologies

Front-end bandwidth:
Different

speed Processors Different number of processors Faster PCI Bus Faster HBAs

e.g. Next Generation Server

2GHz

GHz Processors (4x Benchmark System) 200MB/sec FC interfaces (2x Benchmark System) 4x800MB/s PCI bus (6x Benchmark System)

...

08/28/11

Impact Of New Technologies

2GHz CPU, New HBAs, 2Gbit Switching 2GHz CPU & New HBAs

Today

08/28/11

Sizing the SAN Storage Manager

Scalability
Processor

speed Number of processors PCI bus bandwidth

32bit/33MHz 132MB/sec 64bit/33MHz 267MB/sec 64bit/66MHz 528MB/sec 64bit/100MHz 800MB/s (PCI-X)

Infiniband

technology??? Number of HBAs

200 MB/sec FC interfaces feature faster internal processors

08/28/11

Sizing the SAN Storage Manager

Entry level system:
Dual

Processor, single PCI bus, 1 GB RAM Processor, dual PCI bus, 2 GB RAM Processor, triple PCI bus, 4 GB RAM

Mid-level departmental system:

Dual

Enterprise-class system:
Quad

08/28/11

SAN Storage Manager PC scalability

Max R ead I/ Os
50 ,0 0 0

Max W r it e I/ Os Max W r it e Thr oughput 30 0 250 200

Max R ead Thr oughput

30 ,0 0 0

150
20 ,0 0 0

10 0
1 ,0 0 0 0

50 0 0 2 4

FC H BA s

10

08/28/11

Max MB/ sec (@ 16 K B)

4 0 ,0 0 0

Max IOs per second (@ 512 byt es)

SAN Storage Manager PC scalability

Max R ead I/ Os
50 ,0 0 0

Max W r it e I/ Os Max W r it e Thr oughput 30 0 250 200

Max R ead Thr oughput

30 ,0 0 0

Entry level
20 ,0 0 0

150

Enterprise class 10 0
50 0 0 2 4

1 ,0 0 0 0

FC H BA s

10

08/28/11

Max MB/ sec (@ 16 K B)

4 0 ,0 0 0

Max IOs per second (@ 512 byt es)

Departmental SAN

SANsymphony Performance
Conclusions
FC

switches provide virtually unlimited bandwidth with exceptionally low latency so long as you do not cascade switches General purpose Intel PCs are a great source of inexpensive MIPS. In-band SAN management is not a CPU-bound process. PCI bandwidth is the most significant bottleneck in the Intel architecture. FC Interface cards speeds and feeds are also very significant

08/28/11

SAN Storage Manager Next Steps

Cacheability of Unix and NT workloads
Domino,

MS Exchange Oracle, SQL Server, Apache, IIS

Given mirrored writes, what is the effect of different physical disk configurations?
JBOD RAID

0 disk striping RAID 5 write penalty

Asynchronous disk mirroring over long distances Backup and Replication (snapshot)

08/28/11

Questions

08/28/11

www.datacore.com