DS4000 Installation & Tuning

Terry Murray
Midrange Storage Services
Warwick

|IBM Internal Use Only © 2005 IBM Corporation

Cabling – DS4100/4300

F
• Single Expansion Grouping
• Exp In/Out ports are
multifunctional, BUT labelling
should be followed for clarity of
diagnostics, ie Out Port -> In Port

| |IBM Internal Use Only © 2005 IBM Corporation

Cabling – DS4400/4500

F
• Two expansion groups ( with 4
drive side mini hubs)
• Add expansion drawers to
alternate groups (if available)
• Exp In/Out ports are
multifunctional, BUT labelling
should be followed for clarity of
diagnostics, ie Out Port -> In Port

| |IBM Internal Use Only © 2005 IBM Corporation

Adding An Expansion Drawer Dynamically

Steps to add drawer:

1. Set the tray ID – Must be unique

2. First, add a new expansion (EXP3) on drive loop A by connecting a new cable labelled 1.
3. Move the cable on loop B from EXP2 to EXP3, as indicated by the cable labelled 2.
4. Add a cable from EXP2 to EXP3 (as indicated by the cable labelled 3) to complete loop B.

| |IBM Internal Use Only © 2005 IBM Corporation

Moving An Expansion Drawer Dynamically

Steps to move drawer

1. Remove the drive loop B cable between the second mini hub and EXP2 (cable labelled a). Move the cable from EXP2
going to EXP1 (cable labelled b, from loop B) and connect to second mini hub from EXP1 (cable labelled 1).

2. Connect a cable from the fourth mini hub to EXP2, establishing drive loop D (represented by cable labelled 2).

3. Remove the drive loop A cable between EXP1 and EXP2 (cable labelled c) and connect a cable from the third mini
hub to EXP2, establishing drive loop C (represented by the cable labelled 3).

| |IBM Internal Use Only © 2005 IBM Corporation

Cabling Tips
 Balance drawers across multiple expansion groups if possible, ie
DS4400/4500
 Stick to the expansion Input/Out port assignments for clarity
 Set drawer ID’s Logically, ie 00 01 02 03 etc
 If drawers are across multiple expansion loops consider setting ID’s as
<group><drawer> eg 00 – 0x and 10 – 1x
 Don’t change a drawer ID dynamically.
 Maintenance slots are safer but less fun ;)

| |IBM Internal Use Only © 2005 IBM Corporation

Host Attachment Options
 Hosts can be direct attached to DS4000 without a fabric (dependant on
number of installed host side ports) – FC-AL protocol

 DS4000 can be fabric attached via SAN Switches – P-P Protocol (slight
performance improvement)

 Multiple host attachment requires a “partition licence”, at least one

partition per host, otherwise LUN’s will be available to all hosts!

| |IBM Internal Use Only © 2005 IBM Corporation

Host Attachment Ports – DS4100/DS4300
 Up to 2 host attachment ports per controller

| |IBM Internal Use Only © 2005 IBM Corporation

Host Attachment Ports – DS4400/DS4500
 Up to 4 host attachment ports per controller
 Do not use the In-Ports for fabric attachment (bottom ports)

| |IBM Internal Use Only © 2005 IBM Corporation

Direct Host Attachment – DS4400/DS4500

| |IBM Internal Use Only © 2005 IBM Corporation

Fabric Attachment
 Two switches are required to eliminate SPOF
 A host HBA should be zoned so that it can only see a LUN on ONLY one path
(compare with ESS) – Excludes non clustered windows with RDAC load
balancing drivers.
 Zones should be created with only one initiator (HBA) per zone for resilience
against faults disrupting other devices.
 Dual host HBA attachment is always recommended to eliminate SPOF
 Single HBA attachment is not supported for AIX. (may have changed)
 If a single HBA is used, the HBA must be zoned to see both controllers.

 An HBA/HBA pair is required per DS4000 storage partition per host.

| |IBM Internal Use Only © 2005 IBM Corporation

Fabric Attachment – Examples
 Single Partition

| |IBM Internal Use Only © 2005 IBM Corporation

Fabric Attachment – Examples
 Two Partitions

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Preparation
 Get the manuals from http://www-1.ibm.com/servers/storage/support/disk/
 Get the red books from http://www.redbooks.ibm.com
SG247010 - IBM TotalStorageDS4000 Series and Storage Manager 9.10
SG246363 - IBM TotalStorage FAStT Best Practices
 Understand the restrictions of the SM code level and hosts you will be
attaching, eg LUN’s per partition 32, 256 SM8.4 onwards, LUN’s per host.
 Plan your storage allocations, RAID Levels + LUN sizes
 Plan your host spares – one spare per three populated drawers. I would be
inclined to put two in from day one. Always have at least 1!
 Obtain the latest Storage Manager, ESM, Firmware, NVRAM, HDD and HBA
firmware & Drivers code levels from:
 http://www-1.ibm.com/servers/storage/support/disk/
 Useful Internal PE Site:
 http://ssgtech1.sanjose.ibm.com/Raleigh/FastT%20PE%20Support.nsf/Home
?OpenPage

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Set IP addresses – Method 1 (SMClient 9.1)

 Install the DS4000 Storage Manager Client package (SMclient) on your workstation.
 Attach both DS controllers and your the workstation to a network switch or hub.
 Set you workstation IP to 192.168.128.254/255.255.255.0 (You may need to disable
socks if locally connecting)
 Using SMClient define the DS4000 to the Storage Manager, the default IP address are
192.168.128.101 & 102.

 From the controller properties tab set

the both controller IP address, network
masks and gateways, and apply the
change.
 You will get disconnect when you apply
the changes, and will need to
rediscover the DS unit.

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Set IP addresses – Method 2 (Serial Port)
 Connect to the serial port on a controller in the DS4000 Storage Server with a null modem cable. For
the serial connection, choose the correct port and use the following settings:
• 57600 Baud
• 8 Data Bits
• 1 Stop Bit
• No Parity
• Xon/Xoff Flow Control
 Send a break signal to the controller. The exact command varies with different terminal emulation
software. For most terminal emulations, press Ctrl+Break.
 If you only receive unreadable characters, send another break signal until the following message
appears:
Press <SPACE> for baud rate within 5 seconds.
 Press the Space bar to ensure the correct baud rate setting. If the baud rate is set correctly, a
confirmation message appears.
Send a break signal to log onto the controller. The following message appears:
Press within 5 seconds: <ESC> for SHELL, <BREAK> for baud rate.
 Press the Esc key to access the controller shell. The default password for the DS4000 Storage
Server is infiniti.
 Run the netCfgShow command to see the current network configuration. The controller outputs a list
similar to :

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Set IP addresses – Method 2 (Serial Port)

| |IBM Internal Use Only © 2005 IBM Corporation

Installation – Set IP addresses – Method 2 (Serial Port)

 To change any of the values, enter the netCfgSet command. This command
will prompt you for each line in the network configuration list.
 Press carriage return to skip the current entry.
 Press “.” to reset the current value to factory default.
 After you assign a fixed IP address to Controller A, disconnect from Controller
A and repeat the procedure for Controller B. Remember to assign a different IP
address.
 Because the configuration changed, the network driver is reset and uses the
new network configuration.

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Basic Setup
 Upgrade the Firmware, NVRAM, ESM and HDD code levels to the latest
levels.
 Note: Read the release notes to check dependencies and order of upgrade.
 Both controller must be available and contactable via the LAN to perform a
firmware upgrade.
 Install the DS4000 Storage Manager package on a designated workstation.
 To manage storage subsystems through a firewall, configure the firewall to open
port 2463 for TCP data.
 Start SMclient and define the DS4000 to it
 Add a comment to the device to indicate its model and serial number, etc
 Configure the SNMP and SMTP alerting if required.
 Monitoring and alerting is done by the system running the Storage Manager Client
and not by the actual DS4000/FAStT – so don’t set up on your laptop!
 Manage the device and rename the it to its hostname.
 Synchronise the controller clocks
 Assign a storage manager client password to the DS4000
 Activate any purchased licenses such as partitions, etc

| |IBM Internal Use Only © 2005 IBM Corporation

Installation - Configuration
 Assign hot spare – two across expansion groups and drawers.
 Standard recommendation is one per three drawers
 Create Arrays and LUN’s
 Spread arrays across expansion groups and drawers for maximum performance
and resilience.
 An even number of LUN’s per array, split across controllers A and B is the easiest
way to achieve a balanced performance (but not always!)
 Define Host Groups/Hosts/Ports/Partitions
 One host per partition
 Or host group per partition for clusters
 Allocate LUN’s to Hosts
 If allocating to a clustered host, LUN’s should be allocated at the “host group” level
if they are shared, and non-shared LUN’s to the “host” level
 Assign the “Access” LUN to hosts requiring in-band management of the
DS4000

| |IBM Internal Use Only © 2005 IBM Corporation

Installation – Host Attachment
 Upgrade the switches to the latest firmware level and apply any
purchased licenses.
 Note: Brocade based switches at v4.4.0 MUST be upgraded – reboots every 56
days.

 Install latest host HBA firmware and drivers

 Attach the DS4000 controllers and host HBA to the SAN
 One to each switch for resilience
 Zone the SAN
 Discover devices on host system – cfgmgr, devmgr, etc
 Take a snapshot of the “Storage System Profile” for reference

| |IBM Internal Use Only © 2005 IBM Corporation

General Tips for Performance
 Spread arrays across array loop groups and expansion drawers

 Allocate an even number of LUNS per array and balance across each
DS4000 controller, allocating A/B pairs to hosts. – You may need to
rebalance later depending on the I/O characteristics of the using host.

 Consider stripping volumes across multiple arrays/luns at the host level.

 Note: You cannot cluster windows dynamic disks.

| |IBM Internal Use Only © 2005 IBM Corporation

Global Cache Block Size
 Battery Backed RAM - size dependant on
model
 Shared for read/write
 4k or 16K block size – Setting depends
on application workload characteristics
 Setting 4k with large I/O’s will add cache
management overheads
 Setting 16K with small I/O’s may leave
cache with up to 50% wastage
 Good idea to set to 16K when using
large segment sizes with sequential I/O
 Appears to perform better with high/low
flushing set same (cached writes)

| |IBM Internal Use Only © 2005 IBM Corporation

Array RAID Levels – A reminder!

| |IBM Internal Use Only © 2005 IBM Corporation

Setting LUN Parameters – Segment Size
 Segment Sizes – 8K, 16K, 32K, 64K, 128K, 256K and 512K (in 9.1 GUI , but available
at command line)
 Can dramatically effect IOP’s and throughput – Benchmark settings
 Small segment sizes increase IOP’s by allowing more disks to serve more I/O’s
independently – Good for multiuser, random I/O environments
 Large segment sizes increase throughput by allowing more disks to server a
single I/O request. – Good for single user / large file apps.
 Dynamically changeable – but one step at a time

In general :
 Storage Manager sets a default block size of 64 KB.
 For database application block sizes between 8-16 KB have been shown to be more effective.
 In a large file environment, such as on media streaming or CAD, 128 KB and above is
recommended.
 For Web servers and file and print servers, the range should be between 16-64 KB.

| |IBM Internal Use Only © 2005 IBM Corporation

Setting LUN Parameters - Segment Size = I/O Request

 Fewer disks used per request

 Higher IOPS
 Ideal for random I/O requests such as
database file system

| |IBM Internal Use Only © 2005 IBM Corporation

Setting LUN Parameters - Segment Size < I/O Request

 More disks used/requested

 Higher throughput (Mbps)
 Ideal for sequential writes such as large
multimedia application

| |IBM Internal Use Only © 2005 IBM Corporation

Setting LUN Parameters – Read ahead
 Read ahead – Determines the amount of blocks read in to the cache, normally
sequential disk blocks

 Improves cache hit rate for sequential I/O reads

 Set to 1-4 and see if the cache hit ratio increases, if so the workload is
sequential and will benefit from it. If not set to 0 to avoid populating the cache
with unnecessary data, forcing other data out!

 In SM9.1 firmware – either on(>0) or off (0). New algorithm determines amount of
read ahead by I/O behaviour. Maybe best to set on, unless you specifically want
to stop cache being used.

| |IBM Internal Use Only © 2005 IBM Corporation

Partitioning Example

| |IBM Internal Use Only © 2005 IBM Corporation

Dynamic Changes
 Dynamic RAID Migration (DRM) – Change array type
 RAID 1 or 10 requires an even number of disk drives.
 RAID 3 and 5 require at least three drives.
 There is a limit of 30 drives per array.
 Dynamic Capacity Expansion – Add additional drives to an array
 Dynamic Volume Expansion – Increase the capacity of a LUN
 Requires host support
 Dynamic Segment Size Changes and Array Defgramentation
 Notes :
 DCE/DVE can be combined into one operation
 A maximum of two drives can be added at a time, up to a total of 30 drives per
array.
 Single threaded. Only one modification in progress at once
 Operations can generally not be cancelled once started
 Performance impact of dynamic operations

| |IBM Internal Use Only © 2005 IBM Corporation

Dynamic Modification Priorities

 Lower modification priorities will favour better I/O performance

 Array sizes are a trade off between rebuild times and performance
 Large arrays – provide better performance but longer rebuild times with a larger
window of exposure
 Small arrays – provide lower performance, but quicker rebuild times and a small
window of exposure

| |IBM Internal Use Only © 2005 IBM Corporation

Firmware Upgrades

 Always ensure you have upgraded to the latest DS Storage Manager

Client before performing a firmware upgrade
 Check the order of the firmware upgrade – don’t assume anything!
 It is possible to load then activate later, dependant on current firmware
level (not DS4400 – Fastt700)
 Parallel drive firmware upgrade requires SM9.1 firmware

| |IBM Internal Use Only © 2005 IBM Corporation

Collecting DS4000 Performance Data
 Create a script file called perfmon.scr containing the following:
--- perfmon.scr ---
on error stop;
set performanceMonitor interval=5 iterations=360;
upload storageSubsystem file="/tmp/perf01.txt” content=performanceStats;
--- perfmon.scr ---
 Interval; is the period between samples being taken in seconds.
 Iterations; is the number of samples that are taken.
 File; is where the data will be uploaded to.
 Then use the following command to invoke the performance monitoring.
 smcli [IP-Addr. Ctr.A] [IP-Addr. Ctrl.B] -f perfmon.scr
 Once the total number of iterations has been completed, the file
perf01.txt can be loaded into a spreadsheet for analysis.
 Scripts can also be executed directly from the DS Storage Manager GUI.

| |IBM Internal Use Only © 2005 IBM Corporation

Sample Performance Data:
Performance Monitor Statistics for Storage Subsystem: NXPFASTT1
Date/Time: 12/05/05 12:57:40
Polling interval in seconds: 5

Devices,Total,Read,Cache Hit,Current,Maximum,Current,Maximum
,IOs,Percentage,Percentage,KB/second,KB/second,IO/second,IO/second
Capture Iteration: 1
Date/Time: 12/05/05 12:57:42
CONTROLLER IN SLOT A,761.0,78.2,27.7,1434.7,1434.7,126.8,126.8,
Logical Drive next_lpar01_iw_R01_D02_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar01_iw_R10_D08_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar01_iw_R10_D08_L3,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R0_D03_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D12_L3A,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D12_L3C,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D14_L1A,295.0,73.2,15.1,326.0,326.0,49.2,49.2,
Logical Drive next_lpar02_nps_R10_D14_L1C,466.0,81.3,36.4,1108.7,1108.7,77.7,77.7,
Logical Drive next_nim02_nim_R00_D02_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_nxcstrain_nps_R10_D06_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_nxpnim02_test_lun,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
CONTROLLER IN SLOT B,790.0,65.7,14.1,714.0,714.0,131.7,131.7,
Logical Drive next_lpar01_iw_R10_D08_L2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar01_iw_R10_D08_L4,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D12_L3B,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D12_L3D,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
Logical Drive next_lpar02_nps_R10_D14_L1B,443.0,61.2,14.8,416.7,416.7,73.8,73.8,
Logical Drive next_lpar02_nps_R10_D14_L1D,347.0,71.5,13.3,297.3,297.3,57.8,57.8,
Logical Drive next_nxcstrain_nps_R10_D02_L1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
STORAGE SUBSYSTEM TOTALS,1551.0,71.8,20.9,2148.7,2148.7,258.5,258.5,

| |IBM Internal Use Only © 2005 IBM Corporation

Recovering from a lost password
 Login to the controller via the serial port using the same procedure for
setting the IP addresses.

 Issue the following command on only ONE controller:

 clearSYMbolPassword

 Logout of the controller and hey presto!

| |IBM Internal Use Only © 2005 IBM Corporation

AIX Tuning Tips
 FC adapter attribute changes: (chdev –l fcsX –a )
 lg_term_dma to 0x1000000 (16 MB)
 num_cmd_elems to 2048 (i.e., a queue depth for the adapter)
 link_init to pt2pt (switched) or al ( direct)
 FC SCSI Protocol changes: (chdev –l fscsiX –a )
 fc_err_recov to fast_fail
 dar attribute changes: (chedev –l darX –a )
 load_balancing set to yes and balance_freq to xxxx (dedicated attachment only)
 autorecovery set to yes
 hdisk attribute changes: (chedev –l hdiskX –a )
 queue_depth to >= 16

Note :Total queue_depth for ALL fastt attached host hdisks should not exceed
1024(firmware 6.1x, 5.4x) or 512 (firmware 5.3x)

chdev comands may require –P (apply to database only) and a system

reboot to take effect if devices are in use.

| |IBM Internal Use Only © 2005 IBM Corporation

Tips for TSM
 Split arrays into even LUNS across controllers.
 Mirror database and log if storage allows for the best performance
 Multiple db volumes across arrays – TSM threads each db volume, aim
for 1 per backup thread.
 Database drives have write cache enabled, write cache with mirroring enabled,
and 0 for cache read ahead multiplier
 Log volume need not be spread, given its sequential nature. RAID-0 with
TSM mirroring if possible or RAID-1.
 If db/log is not mirrored at TSM level, enable page shadowing in
dsmserv.opt to protect against partial page writes
 MIRRORWRITE DB PARALLEL , MIRRORWRITE LOG PARALLEL
 DBPAGEShadow Yes , DBPAGEShadowFile /…../dbpgshdw.bdt
 RAID-5 storage pool volumes for protection.
 Storage pool drives have segment size of 128K,read/write cache enabled and write
cache with mirroring, and cache read ahead multiplier of 4 set (or just 1 in SM 9.1)

| |IBM Internal Use Only © 2005 IBM Corporation