Professional Documents
Culture Documents
McKenna - 2011 - OpenSees A Framework For Earthquake Engineering Simulation
McKenna - 2011 - OpenSees A Framework For Earthquake Engineering Simulation
A
ccording to Wikipedia,1 earthquake earthquake occur, many structures might not be
engineering studies how buildings resilient.
and structures behave subject to seis As a consequence, though loss of life in a ma
mic loading, with the main objects of jor earthquake event occurring in an urban re
gion of the US might be minimal, the economic
• understanding the interaction between the losses could be devastating. In recognition of this,
ground and structures; new design procedures and guidelines founded on
• designing, constructing, and maintaining struc performance-based design are being developed by
tures that will perform during earthquakes in the Applied Technology Council (ATC) under the
compliance with building codes and expecta ATC-58 project for inclusion in the building codes.
tions; and Here, I present an overview of the complexi
• foreseeing the potential consequences of ties facing structural engineers in analyzing and
strong earthquakes on urban areas and civil designing structures to withstand earthquakes
infrastructure. using performance-based design. I also introduce
OpenSees, an open source framework that the
In the past, structural engineers typically earthquake engineering research community uses
designed structures to comply with local and to develop finite-element (FE) and FE reliability
regional building codes—codes that attempted applications.
to ensure that structures would suffer no struc
tural damage in moderate events and wouldn’t Tools for Earthquake
collapse in major events. There’s a major mis Engineering Simulation
conception among building owners, insurers, Currently, more than 75 million Americans in
lending institutions, and some government 39 states live in towns and cities at risk of earth
agencies that code-conforming buildings are quake devastation. Researchers have estimated
“earthquake proof.” They’re not; they’re sim that should a major earthquake event occur in a
ply designed not to collapse. Should a major US metropolitan area, loss of life could be mini
mal but economic losses would be crippling—
1521-9615/11/$26.00 © 2011 IEEE upwards of $200 billion.2 Many structures in
Copublished by the IEEE CS and the AIP such an event, even though they will not collapse,
will undergo significant structural and non
Frank McKenna
structural damage. The large economic losses are
Pacific Earthquake Engineering Research Center a consequence of structures that would have to be
July/August 2011 59
800
5 700 Measured
Engineer prediction
Maximum horizontal
600
displacement (mm)
4 500
Floor 400
3
300
Measured 200
2
Researcher
Practicing engineer 100
1 0
0 100 200 300 400 1 2 3 4 5 6
(a) Maximum relative displacement (mm) (b) Earthquake
Figure 2. Blind prediction test results. (a) E-defense 2007 three-story building test (best of 47 entries). (b) The Pacific
Earthquake Engineering Research Center (PEER) and the George E. Brown Jr. Network for Earthquake Engineering Simulation
(NEES) 2010 single-column prediction test (41 entries). In the tests, engineers attempted to predict the response of known
structures to prescribed single or multiple earthquake-input motions for a four-story steel building and a single concrete
bridge column.
As researchers have pointed out for situations with GPU processors accessing remote data
in which computer predictions can affect our wel bases. If cloud computing indeed fulfills the
fare and security—whether in civil engineering promise of becoming the next computing plat
or other fields such as climate modeling, energy, form, or if the petascale machines with hundreds
defense, biology, and medicine—“we must insist of thousands of processors currently available to
that the predictions include concrete, quantifiable researchers through the US National Science
measures of uncertainty.”5 Foundation (NSF) Teragrid resources—or the
Structural engineers can use FE reliability upcoming exascale machines with millions of
methods to characterize uncertainty.6 These me processors—become available for structural engi
thods let analysts define the input parameters as neers, today’s FE software will need to change to
random variables to represent the uncertainties take advantage of these changing resources.
and employ approximate probabilistic methods For research-oriented FE developers, the ca
to obtain estimates of the response variance/ pability to experiment with the FE application’s
covariances, sensitivities, and probabilities of lower-level building blocks (data structures, solv
exceedi ng performance thresholds. Although ers, and algorithms), which affect the numerical
there are many FEM analysis packages used by solution’s performance and accuracy, is almost as
structural engineers today, I’m not aware of any important as being able to add new elements and
with nonl inear capabilities that perform these FE materials. None of the commercial packages offer
reliability analysis methods directly. the ability to experiment with these basic building
To perform reliability analysis, structural engi blocks and none will provide the source code to
neers must use other external software packages, researchers wishing to experiment on new archi
such as CalREL from the University of California, tectures or numerical solution strategies.
Berkeley, PredictionProbe’s Unipass, and Sandia To encourage the testing of new routines
National Laboratories’ Dakota. Because external needed in nonlinear earthquake engineering
software requires additional expertise, practicing analysis and the communication of new research
engineers rarely engage in reliability analysis. among researchers and practicing engineers,
the NSF has sponsored two multi-institutional
OpenSees research centers, the Pacific Earthquake Engi
None of the commercial software is obviously neering Research Center (PEER) and the George
made available for researchers wishing to imple E. Brown Jr. Network for Earthquake Engineer
ment these reliability methods in the application. ing Simulation (NEES). The PEER mission is to
Of course, it’s not just the changing of the reli develop, validate, and disseminate performance-
ability analysis applications that researchers wish based seismic design technologies for structures.
to explore; all FE software runs on computer The vision of NEES is to shift earthquake en
hardware that’s also changing rapidly under our gineering research’s emphasis from its current
feet. In a few short years, structural engineers reliance on physical testing to integrated experi
have gone from the single-processor machines mentation, computation, theory, databases, and
using a local file system to multicore machines model-based simulation.
Analysis
Material
ElementalLoad NodalLoad SP_Constraint
Uniaxial nD Section
Reliability
Figure 3. OpenSees software packages for FEM and reliability analysis. Classes in italics are abstract classes.
Developers can provide their own subclasses to personalize applications.
Both PEER and NEES have supported the The OpenSees Framework
development and maintenance of a new open OpenSees is an open source software framework
source, object-oriented FEM software frame for creating applications for the nonlinear analy
work, the Open System for Earthquake Engi sis of structural and soil systems using either a
neering Simulation (OpenSees). Although it has standard FEM or an FE reliability analysis. It
many industrial users, OpenSees is not intended is object-oriented by design and—in addition to
as a commercial application and doesn’t have achieving computationally efficiency—it’s de
some of the sophisticated element and material signed to be flexible, extensible, and portable.
routines found in the most powerful commercial It’s written primarily in C++, but uses consider
applications. The purpose of OpenSees is to pro able legacy Fortran code; many C++ wrappers are
vide a common open source framework in which provided to interface with external numerical li
researchers can quickly test, experiment with, braries written in C or Fortran, such as Lapack,
and share not only new material and element ScaLapack, Petsc, Mumps, SuperLU, and Arpack.
routines (a facility provided by most commer OpenSees is highly modular. The package dia
cial applications) but also new solvers, integra grams in Figure 3 show for the framework’s FEM
tion schemes, nonlinear solution algorithms, and and reliability parts. Unlike traditional commer
even lower-level data structures and communica cial and research applications, this modularity al
tion protocols. Such features are necessary as re lows developers to extend the code in the analysis,
searchers both improve the modeling capabilities reliability, and output specification parts of the
of the software and as new structural systems, software (in Figure 3, every class in italics is an ab
space-age materials, and mechanical devices stract class and developers can provide their own
are tested for use in structures to improve their subclass to personalize their applications). This is
resilience. particularly important in earthquake engineering,
July/August 2011 61
where researchers are studying new algorithms framework maintainers, it also has the advantage
and integration schemes, such as for collapse anal of limiting the branching of the code.
ysis and hybrid testing. Using existing interpreters allowed us to
In addition to being highly modular, Open quickly provide applications that let users test the
Sees makes extensive use of object composition, framework’s classes. It also gave engineers—who
as shown in the use of aggregation (diamond sym are used to either fixed file format input programs
bols) in Figure 3’s package diagrams. This offers or GUIs—a new programmable interface for
a superior software design compared to the more building models and performing analysis. Ulti
common use of class inheritance for the FEM’s mately, it’s proven to be a highly useful format
analysis part, as it lets users combine components for researchers exploring the nuances of nonlin
in ways that are simply not available in commer ear analysis and performance-based design (once
cial or other research codes.7 some of them overcame the initial shock of having
In designing the classes and their interfaces, to learn a programming language).
heavy use was made software design patterns to The disadvantage of providing only interpreted
identify the relationships between classes and applications to users is that it has limited the
organize communication among system compo framework’s usage, particularly among practic
nents. This has reduced communication overhead ing engineers. The most requested feature in our
among the objects and resulted in only minor surveys (40 percent) is always a GUI. The effort
changes to the class interfaces as new developers to produce one that would satisfy the user base,
used the framework and extended it in areas not however, greatly exceeds our allocated resources;
originally envisioned, such as in hybrid simulation it would also offer limited research opportunities
and reliability analysis. for academics in our field, who are the primary
There are currently more than 1,000 classes in contributors to the framework’s development.
the framework, with contributions from more than However, as our user base swells, commercial
100 developers in the engineering community. GUIs that use OpenSees are now becoming avail
able from several small software companies, such
Applications as nvStructural from Novel CAE Solutions and
Developers use software frameworks to cre CDS Win from Software Tecnico Scientifico.
ate applications. Because the core OpenSees
development team’s domain is structural and OpenSees and HPC
geotechnical engineering, we decided to release At the outset of OpenSees, we recognized that
several interpreted applications that utilized the users in the performance-based analysis field
framework and thereby encourage the use of might need to analyze large models with more
OpenSees. 100,000 degrees-of-freedom (dof ) and that they
Currently, these applications are all extensions might need to perform many thousands of analyses
of the Tcl interpreters, tclsh and wish. In addition on these large models with varying inputs and
to providing users access to the features of a com model parameters.
plete programming language, these interpreted Today, on a single processor or dual-core
applications provide low-level access to the frame machine, the computation required of a compre
work. By low-level, we mean that the user at the hensive performance-based design or a regional
command level can create new objects from the simulation could take weeks, months, even years,
framework’s available classes and invoke methods to complete. This is, of course, unacceptable.
on certain objects that they’ve constructed. However, by the time performance-based engi
In addition to the low-level access to the frame neering does come to fruition, the HPC resources
work’s existing code, the interpreters also let available to academics today will be available to
users introduce new classes at runtime using practicing engineers, either directly on their
shared objects or dynamic link libraries. The abi local machines or through some cloud service
lity to add new classes dynamically to the inter accessible through their desktop. The frame
preters has proven to be an important ingredient work contains many classes available for paral
for these research-oriented applications. This is lel processing, which has allowed researchers to
because it reduces compile time for developers build specific applications for large models8 and
and also lets them test and share their ideas with regional simulation.9 To encourage and promote
others without having to share code, which is the use of HPC among engineers, we provide two
useful in our highly competitive and funding- parallel, interpreted applications, OpenSeesSP
depleted research area. From our viewpoint as and OpenSeesMP.
MovableObject
sendSelf(Channel, int)
TCP UDP MPI Database
recvSelf(Channel, int,..)
Figure 4. Movable object and channel class hierarchy. Because every instantiable class is a descendent of
the MovableObject class, users can move the objects in the domain from one partition to another during
partitioning.
350
Total execution time (minute)
300
250
200
150
100
50
0
0 2 4 6 8 10 12 14 16 18
Number of processors
Figure 5. OpenSeesSP model of the Humboldt Bay Middle Channel Bridge. The 100,000-plus degree-of-
freedom model uses implicit integration and the Mumps direct solver.
OpenSeesSP facilitates the analysis of large application’s state to and from a database for back
models. In it, we’ve followed the traditional up and restore purposes by including the Data-
approach for parallelizing the FEM—that of base class as a subclass of the Channel class.
using domain-decomposition, load-balancing To solve equations, users can choose from
techniques, and parallel equation solvers. In four parallel equation solvers: Mumps, SuperLU,
OpenSeesSP, domain-decomposition is achieved Petsc, and Diagonal. OpenSeesSP’s scalability is
by partitioning the domain at analysis time using highly dependent on the choice of solver. Struc
Metis. To move the objects in the domain from tural engineers tend to use direct solvers and
one partition to another during partitioning, implicit integration techniques, which typically
we make use of the fact that every instantiable limits the application’s scalability to 16 or 32 pro
class is a descendent of the MovableObject class cessors, depending on the model’s size and non
(see Figure 4). As a result, each class must provide linearity (see Figure 5, which shows a model of the
two methods, sendSelf() and recvSelf(), Humboldt Bay Middle Channel Bridge, which has
so that the object and any contained objects can more than 100,000 dof10). For engineers willing to
move from one process’s address space to another use either explicit integration schemes and direct
using the Channel class interface. Adding an extra solvers or iterative solvers, the scalability is much
integer tag in these two methods lets us use better. We’ve seen perfect scalability up to 6,000
these methods for sending and receiving the processors using a diagonal solver and a central
July/August 2011 63
engineers to modify their scripts, so that when
scripts run on a parallel machine, the computation
can be done in parallel using the message-passing
paradigm. The commands we added— send,
recv, barrier, getNP, and getPID—are similar
to the basic ones found in any message-passing
library. They simply allow running processes to
communicate at the scripting level. To date, this
has proven the more popular parallel application
due to the amount of time it’s saving researchers.
For example, with as few as six additional lines of
code in the script, a number of developers have
obtained months of computational work in a few
hours; Figure 6 shows an example for a concrete
building study (one of many such studies carried
out within PEER and NEES).
Software
databse
that, from a practical viewpoint, can’t be placed es
tin
on shaking tables and studied in the laboratory. r ou del
st o s
te m er
The advantage of using OpenSees for these ap La and met
ra
plications is that the numerical analysis can be pa
performed using high-performance computers. Drawings and Location
Thus, the specimen (or multiple experiments
Federal database
soil data FEM software EQ motions
using the geographically distributed NEES adjacent soil data
facilities) can be shaken in real-time using special Results Adjacent
integrators (which aren’t available in commercial structures
applications). Also, finely tuned TCP/IP settings Su
bm ed
et
ensure optimal communication costs for these it f pl s
or m ng
ap Co awi
sometimes time-sensitive experiments. pr
ov Local d r
al building
department
A
w
s performance-based seismic design FEM vie
Re
replaces existing prescriptive-based software
design methods, new workflows built
upon sophisticated analytical tools
Figure 8. A possible workflow for earthquake engineering in the future.
such as OpenSees are becoming possible. Starting with digitized engineering drawings and soil borehole data,
Figure 8 shows one possible design scenario. the application builds a finite-element model of the structure. Based
The process would start with digitized engineer on the location and the performance requirements, the application
ing drawings and soil borehole data. The applica chooses an appropriate loading scenario and augments the FEM
tion would build the FEM model of the structure model to include a soil model and (possibly) models for adjacent
buildings. Results from a probabilistic structural analysis are then
from the drawings. Based on the structure loca visually presented to the engineer for evaluations and improvements.
tion and performance requirements, the appli
cation would select from a database of loading
scenarios and augment the FEM model to include Realizing such a future requires application–
a soil model, and possibly models for adjacent database interfaces, automatic model generation,
buildings. Results from a probabilistic structural reliability analysis, large-scale analysis, better el
analysis, which would include the earthquake ement and material modeling capabilities, better
scenarios but also distributions for material and algorithms and integration schemes, and federal
element properties, would be visually presented action. There’s much to be done. Can we hope
to the engineer, who could then evaluate and im to make the necessary advances? If we don’t, the
prove the structure. Once accepted and submit earthquake engineering community will continue
ted for full design review, the structural plans simply reacting to major events as they unearth
and borehole data would be submitted to a central hidden flaws in our designs and software, which
database so that adjacent property owners could our limited experimentation and prior earthquake
review the design for the impact on their build experience failed to illuminate. We’ll also conti
ings. Before final approval, as-built structural nue to hunt down buildings whose details—later
drawings, material tests, and borehole data would shown experimentally or in earthquake response
be submitted to a federally maintained database. to be poor—are buried in drawings in the archives
Such a scenario would also allow existing build and hidden behind partition walls and plaster.
ings to be reevaluated. This could occur using We hope to continue the advancement of
new analysis techniques or instruments for struc OpenSees through contributions from those in
tural system identification or if existing modeling our own community as well as through the direct
assumptions or analysis programs are discovered or indirect help of those in the mathematical and
to be incorrect or inadequate. In addition, at the computer science disciplines.
federal level, this scenario would allow earthquake
scenarios (simulated forecast scenarios based on Acknowledgments
either a future event’s likelihood or, if processing OpenSees development and maintenance has been
capabilities exist, based on an actual occurring supported by the Pacific Earthquake Engineering Re
event) to be used to determine system weaknesses search Center under US National Science Foundation
and vulnerabilities using regional simulations. grant no. EEC-9701568 and the George E. Brown Jr.
July/August 2011 65
Network for Earthquake Engineering Simulation under Object Composition,” J. Computing in Civil Eng.,
NSF grant no. NEES-4101-35704. My colleagues and vol. 24, no. 1, 2010, pp. 95–107.
I are grateful to the visionaries, particularly Dean 8. T. Oden, R. Moser, and O. Ghattas, Computer Predic
Gregory L. Fenves, and the many developers who tions with Quantified Uncertainty, tech. report 10-39,
have contributed to the OpenSees framework. Inst. Computational Eng. and Sciences, Univ. Texas,
Austin, Oct. 2010.
References 9. A. Pavan, Blind Prediction of a Full-Scale 3D Steel Frame
1. J. Bielak et al., “Simulation for Determining the Tested Under Dynamic Conditions, master’s disserta-
Seismic Performance of Urban Regions,” Proc. 6th tion, Inst. Advanced Studies, Univ. Pavia, May 2008.
Euro. Conf. Structural Dynamics, C. Soize and G. I. 10. G. Petropoulos and G.L. Fenves, “Interprocessor
Schuëller, eds., Millpress, 2005, pp. 1479–1485. Communication for High Performance, Explicit Time
2. A. Elgamal et al., “Three Dimensional Seismic Integration,” Eng. with Computers, vol. 26, no. 2,
Response of Humboldt Bay Fridge-Foundation- 2010, pp. 149–157.
Ground System,” J. Structural Eng., vol. 134, no. 7, 11. Y. Takahashi and G.L. Fenves, “Software Frame-
2008, pp. 165–1176. work for Distributed Experimental-Computational
3. T. Haukass and M.H. Scott, “Shape Sensitivities in Simulation of Structural Systems,” Earthquake Eng.
the Reliability Analysis of Nonlinear Frame Struc- and Structural Dynamics, vol. 35, no. 3, 2006,
tures,” Computers and Structures, vol. 84, no. 15, pp. 267–291.
2006, pp. 964–977. 12. US Congressional Senate Committee on Research,
4. Y. Huang et al., “Coupling FE Software The National Earthquake Reductions Program: Past,
through Adapter Elements: A Novel Use of Present and Future, 108th Congress, session 1, 8 May
User-Defined Elements,” Proc. 10th Int’l LS-DYNA 2003, US Gov. Printing Office, 2010.
Users Conf., DynaLook, 2008; www.dynalook.com/ 13. Wikipedia, “Earthquake Engineering,” Wikimedia,
international-conf-2008/SimulationTechnology1-4.pdf. 25 Oct. 2010; http://en.wikipedia.org/wiki/
5. B. Jeremic’ and G. Jie, “Parallel Soil–Foundation– Earthquake_engineering.
Structure Interaction Computations,” Progress in
Computational Structural Dynamics and Earthquake Frank McKenna is a research engineer in the Pacific
Eng., M. Papadrakakis et al., eds., 2008, CRC Press Earthquake Engineering Research Center. His re
6. P.L. Liu and A. Der Kiureghian, “Finite Element search interests include earthquake engineering and
Reliability of Geometrically Nonlinear Uncertain computational engineering. McKenna has a PhD in
Structures,” ASCE J. Eng. Mechanics, vol. 117, no. 8, civil engineering from the University of California,
1991, pp. 1806–1825. Berkeley. He is a member of ACM and the American
7. F. McKenna, M.H. Scott, and G.L. Fenves, “Nonlinear Society of Civil Engineers. Contact him at fmckenna@
Finite Element Analysis Software Architecture Using ce.berkeley.edu.
http://computingnow.computer.org