QuakeCoRE OpenSees

Training Workshop 2016

OpenSees in Parallel and on
HPC Resources
Parallel Processing/HPC Resources
Even though the computational speed on desktop and laptop computers has
increased significantly over the past few decades, it is still easy to push the limits
in regard to what is reasonable to run on a local device or a single processor.

In general, the computational demand associated with a particular modeling

effort comes from one of two places:
o Very large and complex models. Many degrees-of-freedom = very large
systems of equations to solve = large computational demand
o A parameter study where many small to moderate sized models will need
to be run in a large variety of configurations (e.g. multiple ground motions
and/or multiple material properties)

We can take advantage of parallel processing tools and/or high-performance

computing resources that can better manage large computational demands to
greatly reduce the time involved in running large models or parameter studies.
Parallel Processing and OpenSees

The OpenSees framework has been written such that nonlinear finite element
models can be developed for both sequential and parallel environments
o Sequential is what we have been doing so far. A single processor has been
handling everything from running the interpreter to solving the systems
of equations in our models.
o It is also possible to run OpenSees on a parallel computer

There are two parallel OpenSees interpreters.

o OpenSeesSP – for running very large models
o OpenSeesMP – for running parameter studies AND very large models
Parallel Processing and OpenSees
The OpenSees framework has been written such that nonlinear finite element
models can be developed for both sequential and parallel environments
o Sequential is what we have been doing so far. A single processor has been
handling everything from running the interpreter to solving the systems
of equations in our models.
o It is also possible to run OpenSees on a parallel computer

There are two parallel OpenSees interpreters.

o OpenSeesSP – for running very large models
o OpenSeesMP – for running parameter studies AND very large models
Parallel Processing and OpenSees
OpenSeesSP – for very large models

McKenna (2010)
Parallel Processing and OpenSees
OpenSeesSP – for very large models

Additional domain and analysis classes have been added for partitioning the
domain and solving the global system across a network of processes

The system class has been modified to accept two additional solvers:
o Mumps
o Diagonal
Running OpenSeesSP in parallel requires virtually no additional steps compared
to running the sequential OpenSees interpreter.
o Simply change the system command to Mumps or Diagonal (depending
on the system you have)
o Run using MPI (allows nodes in network to talk to each other)
o mpirun –np 4 OpenSeesSP myModel.tcl
Parallel Processing and OpenSees
OpenSeesSP – Example Usage

McKenna (2010)
Parallel Processing and OpenSees
OpenSeesMP – for parameter studies and very large models

McKenna (2010)
Parallel Processing and OpenSees
OpenSeesMP – for parameter studies and very large models

Additional commands have been added to the interpreters (all three)

o getNP – returns number of processes in computation
o getPID – returns unique process ID {0, 1, …, NP-1}
o send –pid pid? Data pid = {0, 1, …, NP-1}
o recv –pid pid? variableName pid = {0, 1, …, NP-1, ANY}
o barrier
o domainChange
Other commands have been modified to allow for the analysis of large models in
parallel
o numberer ParallelPlain and ParallelRCM
o system Mumps
o integrator ParallelDisplacementControl $node $dof $dU

o This last set of modified commands will only work for a parallel model
Parallel Processing and OpenSees
OpenSeesMP – for parameter studies and very large models

McKenna (2010)
Parallel Processing and OpenSees
OpenSeesMP – for parameter studies and very large models

McKenna (2010)
Parallel Processing and OpenSees
This example file shows how to
use OpenSeesMP to run a large
parameter study.

A simple model contained in

the file model.tcl is subjected
to a large list of ground
motions following the set of
analysis commands contained
in analysis.tcl

The file at left sends the

analysis to each of the available
processes called when running
the model. When each process
has completed its task, a new
ground motion analysis will be
sent to be run
OpenSees on HPC Resources

There are a number of high-performance computing options available to

QuakeCoRE researchers, both in New Zealand and internationally
o These resources can be used to run OpenSees, OpenSeesSP, and
OpenSeesMP on either very fast sequential machines, or very large (and
very fast) parallel machines with many more nodes than would be
available on a local desktop or laptop
o The general steps involved in running the OpenSees interpreters on such
systems are not all that different than what we’ve already been doing
running things sequentially on our local machines, but there are a few
extra steps that may be required

The Natural Hazards Engineering Research Infrastructure (NHERI) DesignSafe-CI

is a cloud-based environment for natural hazards engineering research hosted at
the Texas Advanced Computing Center (TACC) in Austin and sponsored by the
United States National Science Foundation (NSF)
o QuakeCoRE researchers are able to get a login ID for this system to take
advantage of their vast computing resources
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
OpenSees on NHERI Designsafe-CI
Thank you!

www.quakecore.nz