You are on page 1of 4

3/22/2011 A Guide to Running Amber Molecular …

A Guide to Running AMBER at SDSC


Supermicro 1U Ads by Google Amber Amber Insect Jew elry Baltic Amber Beads David & Amber Dave and Amber
Twin™
2x Intel® Xeon®
based Systems in 1U This page provides an end user guide to running the AMBER molecular
High-efficienc y, dynamics simulation software on the various High Performance Computing Fastest Supercomputer www.Appro.com
Double Performance (HPC) resources available at San Diego Supercomputer Center (SDSC). To High Performance At Lower Costs w/ Intel®
www.supe rm icro.com Xeon® technology. Learn More
skip directly to instructions for the various machines click here.
Laptops Sold for
$33.33 This guide is designed to help you with both running simulations on the
Today: All HP Laptops Energy Benchmarking www.myBenchmarx.com
are Sold for up to various machines and give you a series of suitable benchmark numbers that Automated Energy Benchmarking Utility
98% Off. Buy Yours you can use for preparing allocation requests. If you have any specific Monitoring & Disclosure
Today? questions relating to running AMBER at SDSC please contact
Q uiBids.com
consulting@sdsc.edu. General questions concerning AMBER should be
It Benchmark www.TNS-Infratest.com/It+Benchmark
Mobile blog directed to the AMBER mailing list (amber@scripps.edu).
Get Your Internet Comparison of 15 Top IT locations Download
Marketing Degree. Full At present SDSC provides the following supercomputer resource machines your free digital copy!
Sail's Online Degree on which the AMBER software can be run:
Programs.
www.FullSa il.e du
Datastar
Event Driven DataStar has 272 (8-way) P655+ and 7 (32-way) P690 compute nodes. The 1.5 GHz 8-way nodes (176 in number) have 16 GB, the 1.7 GHz 8-way
Architecture nodes (96 in number) have 32 GB, while the 32-way nodes have 128 GB of memory. There is also one 32-way node with 256 GB of memory for
Free 24-page Guide applications requiring unusually large memory space. DataStar has a nominal theoretical peak performance of 15.6 TFlops.
Shows How Event
Driven Architecture
Gets Results! Teragrid
www.progress.com /apa m a SDSC's TeraGrid cluster system is composed of 262 IBM Itaninum 2 (1.5GHz) nodes with two processors per each node. Each node is built with
SuSe Linux and interconnected with Myricom's Myrinet network. The system has a peak performance of 3.1 Teraflops, a total memory of 1
Point Of Sale
Computers Terabyte, and a total of 50 Terabytes GPFS disk through the fiber optic SAN network.
First Data Retail
Solution. Free Setup. Intimidata (Bluegene/L)
Get Pricing Today!
IntlC a rdSe rvice .com /Fre e … SDSC was the first academic institution to install and offer allocated time on the IBM BlueGene eServer. Nicknamed Intimidata, the single-rack
server has 2048 processors offering 5.7 teraflops of compute power and a total memory of 512 possible gigabytes. The system also offers the
maximum possible ratio of input/output nodes to compute nodes to support data-intensive computing in a compact footprint.

The choice of which machine to use is very much a function of the size and type of job you plan to run and the allocations you have
available. Note, on Datastar and Teragrid all of AMBER's MD codes are available (sander.MPI, sander.PIMD.MPI and PMEMD)
and so any supported MD job can be run. On Intimidata (Bluegene/L) the architecture is such that only PMEMD is available and so
only explicit solvent PME calculations and Generalized Born implicit solvent simulations can be run.

The following tables provide benchmarks for PMEMD on the various machines. Click on the machine name in each table to go to a
page with instructions on how to run AMBER simulations on the machine.

All benchmarks are for AMBER 9. All timings are in ps/day and so large numbers are better. The benchmarks that were run were
Factor IX, JAC, Cellulose and GB_MB. If you are running Amber 8 at present it is highly recommended that you switch to Amber 9
as the performance has been considerably improved. Bluegene/L needs large scaling for efficient usage and so only Factor/IX and
Cellulose benchmarks are shown.

Factor IX
Large size explicit water (PME) system. There are 90,906 total atoms, and PME used with a direct space cutoff of 8.0 Ang. This
benchmark is in amber9/benchmarks/factor_ix. This uses dt=0.0015, so the conversion is ps-per-day = 129.6/(time-per-step) -
Timings are for PMEMD v9.0. All Bluegene data is for VN mode.
Datastar P655+ Datastar P655+ Datastar P690 1.7GHz Intimidata
TeraGrid (IA64
1.5GHz 1.7GHz (high, (high32, normal32, highL, (Bluegene/L
1.5GHz)
(high,normal) normal) normalL) 0.7GHz)
NCPUS Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup
1 96.67 1.00 110.71 1.00 103.65 1.00 176.11 1.00 28.73 1.00
2 186.15 1.93 212.71 1.92 200.83 1.94 275.74 1.57 57.32 1.99
4 354.11 3.66 398.11 3.60 389.63 3.76 518.00 2.94 110.17 3.83
8 634.73 6.57 719.46 6.50 775.96 7.49 935.50 5.31 210.81 7.34
16 1218.56 12.61 1380.19 12.47 1411.46 13.62 1606.55 9.12 407.18 14.17
32 2292.59 23.72 2596.15 23.45 2279.28 21.99 2652.21 15.06 739.73 25.74
64 4120.17 42.62 4677.01 42.25 4041.79 38.99 3887.81 22.08 1353.101 47.09

www.rosswalker.co.uk/amber_sdsc/ 1/4
3/22/2011 A Guide to Running Amber Molecular …
128 6822.85 70.58 7772.11 70.20 4489.09 43.31 5737.05 32.58 2123.55 73.91
192 8022.28 82.99 9784.82 88.38 6476.14 36.78 2495.19 86.84
256 9243.94 95.63 10464.27 94.52 2966.01 103.23
384 9123.55 94.38 10160.72 91.78 3139.16 109.25
512 8415.58 87.06 9452.95 85.39 2997.92 104.34
768 6758.80 69.92 2525.09 87.88
1024 6106.01 63.17 1893.63 65.90

click image for larger view

JAC:
Medium size explicit water (PME) system. Joint Amber/Charmm DHFR benchmark. This is the protein DHFR, solvated with TIP3
water, in a periodic box. There are 23,558 total atoms, and PME used with a direct space cutoff of 9 Ang. This benchmark is in
amber9/benchmarks/jac. Timings are for PMEMD v9.0.
Datastar P655+ 1.5GHz Datastar P655+ 1.7GHz Datastar P690 1.7GHz (high32, TeraGrid (IA64
(high,normal) (high, normal) normal32, highL, normalL) 1.5GHz)
NCPUS Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup
1 197.30 1.00 227.47 1.00 220.00 1.00 395.37 1.00
2 392.68 1.99 454.22 2.00 435.44 1.98 667.00 1.69
4 744.69 3.77 862.76 3.79 831.30 3.78 1200.97 3.04
8 1397.29 7.08 1600.18 7.03 1682.44 7.65 2102.80 5.32
16 2616.44 13.26 3008.57 13.23 3098.11 14.08 3882.10 9.82
32 4851.21 24.59 5608.93 24.65 5330.70 24.23 6115.52 15.47
64 8368.85 42.42 9602.13 42.21 8803.75 40.02 7948.48 20.10
128 12087.3 61.26 14154.65 62.23 11647.34 52.94 11131.15 28.15
192 12910.94 65.44 14285.71 62.80 10611.64 26.84
256 13688.21 69.38 15428.57 67.83 10359.71 26.20
384 10775.75 54.62 12206.84 53.66
512 9538.53 48.35 6501.13 28.58
768 7116.97 36.07
1024 5566.29 28.21

www.rosswalker.co.uk/amber_sdsc/ 2/4
3/22/2011 A Guide to Running Amber Molecular …

click image for larger view

Cellulose:
Very Large size explicit water (PME) system. This us a cellulose fibre of 408K atoms solvated in TIP3 water, in a periodic box.
PME is used with a direct space cutoff of 9 Ang. The large size of this system means that efficient scaling to large numbers of cpus is
possible. Timings are for PMEMD v9.0.
Datastar P655+ 1.5GHz (high,normal) TeraGrid (IA64 1.5GHz) Intimidata (Bluegene/L 0.7GHz)
NCPUS Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup
1 14.44 1.00 28.41 1.00 4.55 1.00
2 28.01 1.94 45.90 1.62 9.10 2.00
4 53.96 3.74 87.52 3.08 17.71 3.89
8 103.03 7.14 162.61 5.72 34.09 7.49
16 197.27 13.66 298.53 10.51 64.44 14.15
32 371.40 25.73 530.92 18.69 124.16 27.27
64 633.87 43.91 792.22 27.89 215.23 47.28
128 945.14 65.47 915.74 32.34 283.97 62.38
192 1201.50 83.23 1153.31 40.60 391.71 86.04
256 1342.24 92.97 1427.86 50.26 486.79 106.93
384 1521.53 105.39 1545.76 54.41 660.02 144.98
512 1664.42 115.29 739.69 162.48
768 2622.16 181.63 914.77 200.94
1024 2558.48 177.22 891.59 195.85

click image for larger view

GB_MB:
Medium size Generalized Born Implicit Solvent system. Generalized Born myoglobin simulation. This protein has 2492
atoms, and is run with a 20 Ang. cutoff and a salt concentration of 0.2 M, with nrespa=4 (long range forces computed every 4 steps.)
This benchmark is in amber9/benchmarks/gb_mb. Timings are for PMEMD v9.0.
Datastar P690 1.7GHz (high32, normal32
Datastar P655+ 1.5GHz (high,normal) TeraGrid (IA64 1.5GHz)
,highL, normalL)
NCPUS Ps/Day Speedup Ps/Day Speedup Ps/Day Speedup
www.rosswalker.co.uk/amber_sdsc/ 3/4
3/22/2011 A Guide to Running Amber Molecular …
1 220.14 1.00 249.93 1.00 191.51 1.00
2 430.69 1.96 488.04 1.95 358.97 1.87
4 855.55 3.89 955.14 3.82 789.85 4.12
8 1676.76 7.62 1916.85 7.67 1524.46 7.96
16 3243.97 14.74 3696.42 14.79 2839.12 14.82
32 5992.51 27.22 6858.23 27.44 4830.59 25.22
64 9869.77 44.83 10701.02 42.82 7651.44 39.95
128 14257.43 64.77 15882.35 63.55 10695.72 55.85

click image for larger view

Instructions for Running Amber Simulations at San Diego Supercomputer Center


Each of the links below will take you to a page that gives details on how to run amber9 simulations on that machine at SDSC. This
includes a re-cap of the benchmarks, available queues, example Load Leveler and PBS job scripts and general tips on maximizing the
performance. Executables are pre-installed on all of the machines.

| Datastar Instructions | Teragrid Instructions | Intimidata (Bluegene/L) Instructions |

www.rosswalker.co.uk/amber_sdsc/ 4/4