You are on page 1of 279

The libMesh Finite Element Library

Object-Oriented High-Performance Computing
Benjamin S. Kirk†
benjamin.kirk@nasa.gov
John W. Peterson‡
peterson@cfdlab.ae.utexas.edu
Roy H. Stogner?
roystgnr@ices.utexas.edu
† NASA

Lyndon B. Johnson Space Center, USA
‡ Idaho National Labs, USA
? The University of Texas at Austin, USA

June 17, 2013
Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

1 / 215

Outline

1

Introduction

2

Motivation

3

Software Installation & Ecosystem

4

A Generic Boundary Value Problem

5

Key Data Structures

6

Weighted Residuals

7

Poisson Equation

8

Other Examples

9

Some Extensions

10

Adaptive Mesh Refinement

11

Parallelism on Adaptive Unstructured Meshes

12

Going Further

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

2 / 215

Introduction

1

Introduction

2

Motivation

3

Software Installation & Ecosystem

4

A Generic Boundary Value Problem

5

Key Data Structures

6

Weighted Residuals

7

Poisson Equation

8

Other Examples

9

Some Extensions

10

Adaptive Mesh Refinement

11

Parallelism on Adaptive Unstructured Meshes

12

Going Further

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

3 / 215

Introduction

1

Introduction

2

Motivation

3

Software Installation & Ecosystem

4

A Generic Boundary Value Problem

5

Key Data Structures

6

Weighted Residuals

7

Poisson Equation

8

Other Examples

9

Some Extensions

10

Adaptive Mesh Refinement

11

Parallelism on Adaptive Unstructured Meshes

12

Going Further

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

4 / 215

Introduction

Acknowledgments

Code Contributors
Benjamin S. Kirk
Bill Barth
Cody Permann
Daniel Dreyer
David Andrs
David Knezevic
Derek Gaston
Dmitry Karpeev
Florian Prill
Jason Hales
John W. Peterson
Paul T. Bauman
Roy H. Stogner
Steffen Petersen
Sylvain Vallaghe
Tim Kroeger
Truman Ellis
Wout Ruijter


benkirk
bbarth
permcody
ddreyer
andrsd
knezed01
friedmud
karpeev
fprill
jasondhales
jwpeterson
pbauman
roystgnr
spetersen
svallagh
sheep tk
trumanellis
woutruijter

Thanks to Wolfgang Bangerth and the deal.II team for initial technical inspiration.
Also, thanks to Jed Brown, Robert McLay, & many others for discussions over the years.

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

5 / 215

Carey. Carey taught them to us. despite his continued insistence that “no one ever got a PhD from here for writing a code. we went back to the lab. 2013 6 / 215 . INL. Graham F.” Kirk. professor of aerospace engineering and engineering mechanics at The University of Texas at Austin. Stogner (NASA. director of the ICES Computational Fluid Dynamics Laboratory.Introduction Acknowledgments Thanks to Dr. Peterson. and immediately began coding. he was ultimately responsible for this library that we hope you may find useful. and holder of the Richard B. Carey The original development team was heavily influenced by Professor Graham F. In a very real way. Many of the technologies employed in libMesh were implemented because Dr. Curran Chair in Engineering. UT) The libMesh Finite Element Library June 17.

• Leveraging existing software whenever possible is the most efficient way to manage this complexity.Introduction Background Background • Modern simulation software is complex: • Implicit numerical methods • Massively parallel computers • Adaptive methods • Multiple. Stogner (NASA. UT) The libMesh Finite Element Library June 17. Kirk. 2013 7 / 215 . INL. Peterson. coupled physical processes • There are a host of existing software libraries that excel at treating various aspects of this complexity.

. • Teaming is a prerequisite for success. • It is not reasonable to expect a single person to have all the necessary skills for developing & implementing high-performance numerical algorithms on modern computing architectures. 2013 8 / 215 . UT) The libMesh Finite Element Library June 17.Introduction Background Background • Modern simulation software is multidisciplinary: • Physical Sciences • Engineering • Computer Science • Applied Mathematics • . INL. Stogner (NASA. Peterson.. Kirk.

INL.Introduction Background Background • A large class of problems are amenable to mesh based simulation techniques. • Consider some of the major components such a simulation: Kirk. Peterson. 2013 9 / 215 . UT) The libMesh Finite Element Library June 17. Stogner (NASA.

Introduction Background Background • A large class of problems are amenable to mesh based simulation techniques. INL. 2013 9 / 215 . Peterson. • Consider some of the major components such a simulation: 1 Read the mesh from file 2 Initialize data structures 3 Construct a discrete representation of the governing equations 4 Solve the discrete system 5 Write out results 6 Optionally estimate error. and repeat Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. refine the mesh.

and repeat • With the exception of step 3. Kirk. Peterson. the rest is independent of the class of problems being solved. 2013 9 / 215 . • Consider some of the major components such a simulation: 1 Read the mesh from file 2 Initialize data structures 3 Construct a discrete representation of the governing equations 4 Solve the discrete system 5 Write out results 6 Optionally estimate error.Introduction Background Background • A large class of problems are amenable to mesh based simulation techniques. Stogner (NASA. refine the mesh. INL. UT) The libMesh Finite Element Library June 17.

refine the mesh. Kirk. • Consider some of the major components such a simulation: 1 Read the mesh from file 2 Initialize data structures 3 Construct a discrete representation of the governing equations 4 Solve the discrete system 5 Write out results 6 Optionally estimate error.Introduction Background Background • A large class of problems are amenable to mesh based simulation techniques. 2013 9 / 215 . Peterson. INL. and repeat • With the exception of step 3. UT) The libMesh Finite Element Library June 17. Stogner (NASA. • This allows the major components of such a simulation to be abstracted & implemented in a reusable software library. the rest is independent of the class of problems being solved.

INL. Stogner (NASA. • Primary goal is to provide data structures and algorithms that can be shared by disparate physical applications. the libMesh library began with these ideas in mind. that may need some combination of • Implicit numerical methods • Adaptive mesh refinement techniques • Parallel computing • Unifying theme: mesh-based simulation of partial differential equations (PDEs). UT) The libMesh Finite Element Library June 17. Peterson. 2013 10 / 215 .Introduction The libMesh Software Library The libMesh Software Library • In 2002. Kirk.

• libMesh is not an application code. UT) The libMesh Finite Element Library June 17. 2013 11 / 215 . Peterson. Stogner (NASA. and to do so quickly with advanced numerical algorithms on high-performance computing platforms.” • It can be used to help you develop an application to solve problem XYZ. INL. • It does not “solve problem XYZ. scientists. and engineers as a tool for developing simulation codes or as a tool for rapidly implementing a numerical method. researchers.Introduction Software Reusability The libMesh Software Library Key Point • The libMesh library is designed to be used by students. Kirk.

. • We implement the “glue” to these pieces..Introduction Software Reusability Software Reusability • At the inception of libMesh in 2002. Peterson. Stogner (NASA. 2013 12 / 215 . there were many high-quality software libraries that implemented some aspect of the end-to-end PDE simulation process: • Parallel linear algebra • Partitioning algorithms for domain decomposition • Visualization formats • . INL. Kirk. as well as what we viewed as the missing infrastructure: • Flexible data structures for the discretization of spatial domains and systems of PDEs posed on these domains. UT) The libMesh Finite Element Library June 17. • A design goal of libMesh has always been to provide flexible & extensible interfaces to existing software whenever possible.

2013 13 / 215 . INL.Introduction Software Reusability Tumor Angiogenesis Compressible RBM NS Library Structure DD • Basic libraries are libMesh’s “roots” LibMesh • Application “branches” built off the library “trunk” Petsc Laspack Metis STL Mpich Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson.

UT) The libMesh Finite Element Library June 17. Stogner (NASA.Introduction Library Trivia Trivia – Downloads Kirk. Peterson. INL. 2013 14 / 215 .

Introduction Library Trivia Trivia – Mailing List Membership libmesh-users@lists. 2013 15 / 215 .net libmesh-devel@lists. UT) The libMesh Finite Element Library June 17.sourceforge. Peterson. Stogner (NASA. INL.sourceforge.net Kirk.

Introduction Library Trivia Trivia – Citations Kirk. Peterson. 2013 16 / 215 . INL. Stogner (NASA. UT) The libMesh Finite Element Library June 17.

INL. 2013 17 / 215 . UT) The libMesh Finite Element Library June 17. Peterson. • The notion of Base Classes defining an abstract interface and Derived Classes implementing the interface is key to this programming model. Stogner (NASA.Introduction Library Design The “Glue” • The C++ programming language provides a powerful abstraction mechanism for separating a software interface from its implementation. Kirk.

2013 17 / 215 . 5.. INL. } Kirk.. UT) The libMesh Finite Element Library June 17. Peterson. .Introduction Library Design The “Glue” • The C++ programming language provides a powerful abstraction mechanism for separating a software interface from its implementation. shapes[0] = new Rectangle (10. i<2. 20. 25. shapes[1] = new Circle (15. 8). 6). for (int i=0. • The classic C++ example: Shapes. ++i) shapes[i]->Draw(). int main () { /* using shapes polymorphically */ Shape * shapes[2]. Stogner (NASA. • The notion of Base Classes defining an abstract interface and Derived Classes implementing the interface is key to this programming model.

2013 18 / 215 . int newy) = 0. int dy) = 0. Kirk. INL.Introduction Library Design Abstract Shape /* abstract interface declaration */ class Shape { public: virtual void Draw () = 0. virtual void MoveTo (int newx. UT) The libMesh Finite Element Library June 17. Stogner (NASA. }. Peterson. virtual void RMoveTo (int dx.

virtual void SetHeight (int newHeight). virtual void MoveTo (int newx.Introduction Library Design Specific Shape: Rectangle /* Class Rectangle */ class Rectangle : public Shape { public: Rectangle (int x. virtual void Draw (). Peterson. 2013 19 / 215 . Stogner (NASA. }. virtual void RMoveTo (int dx. INL. int h). private: int x. int height. int y. int width. int w. Kirk. UT) The libMesh Finite Element Library June 17. int newy). y. virtual void SetWidth (int newWidth). int dy).

int radius. Stogner (NASA.Introduction Library Design Specific Shape: Circle /* Class Circle */ class Circle : public Shape { public: Circle (int initx. int dy). UT) The libMesh Finite Element Library June 17. 2013 20 / 215 . int newy). virtual void MoveTo (int newx. y. }. virtual void RMoveTo (int dx. INL. Kirk. virtual void SetRadius (int newRadius). int inity. Peterson. private: int x. int initr). virtual void Draw ().

shapes[1] = new Circle (15. 6). ++i) { shapes[i]->Draw(). Stogner (NASA. 2013 21 / 215 . Peterson. 5.Introduction Library Design Object Polymorphism int main () { /* using shapes polymorphically */ Shape * shapes[2]. i<2.. 20.SetWidth (30). for (int i=0. 8). 25. 15). rect. } Kirk. INL. DoSomethingWithShape (shapes[i]). UT) The libMesh Finite Element Library June 17. } /* access a rectangle specific function */ Rectangle rect(0. 15..Draw (). . shapes[0] = new Rectangle (10. 0. rect.

Stogner (NASA.Introduction Library Design Examples of Polymorphism in libMesh Kirk. INL. Peterson. 2013 22 / 215 . UT) The libMesh Finite Element Library June 17.

UT) The libMesh Finite Element Library June 17. INL. Stogner (NASA. 2013 23 / 215 . Peterson.> Introduction Library Design The “Glue:” Linear Algebra libMesh::EigenSparseMat rix< T > libMesh::Epet raMat rix< T > libMesh::SparseMat rix< T > libMesh::LaspackMat rix< T > libMesh::Pet scMat rix< T > Kirk.

UT) The libMesh Finite Element Library June 17. INL. Stogner (NASA. Peterson. 2013 24 / 215 .Introduction Library Design The “Glue:” I/O formats libMesh::AbaqusIO libMesh::Mat labIO libMesh::OFFIO libMesh::Nemesis_IO libMesh::Tet GenIO libMesh::UCDIO libMesh::MeshInput < MeshBase > libMesh::UNVIO libMesh::VTKIO libMesh::XdrIO libMesh::ExodusII_IO libMesh::GmshIO libMesh::MeshOut put < MeshBase > libMesh::GMVIO libMesh::LegacyXdrIO libMesh::GnuPlot IO libMesh::MEDITIO libMesh::Post script IO libMesh::Tecplot IO libMesh::DivaIO libMesh::Ensight IO libMesh::FroIO Kirk.

UT) The libMesh Finite Element Library June 17. INL.Introduction Library Design Disretization: The Mesh libMesh::ParallelMesh libMesh::MeshBase libMesh::Mesh libMesh::BoundaryMesh libMesh::Unst ruct uredMesh libMesh::SerialMesh Kirk. Peterson. 2013 25 / 215 . Stogner (NASA.

2013 26 / 215 .Introduction Library Design Disretization: Geometric Elements libMesh::Hex20 libMesh::Hex27 libMesh::Hex libMesh::Hex8 libMesh::Prism15 libMesh::Prism18 libMesh::Prism libMesh::Prism6 libMesh::Cell libMesh::Pyramid libMesh::Pyramid5 libMesh::Tet libMesh::Tet 10 libMesh::Edge2 libMesh::Tet 4 libMesh::Edge3 libMesh::Quad4 libMesh::Edge libMesh::Edge4 libMesh::Quad8 libMesh::InfEdge2 libMesh::Quad9 libMesh::Quad libMesh::Tri3 libMesh::Face libMesh::Tri libMesh::ReferenceCount edObject < Elem > libMesh::Elem libMesh::ReferenceCount er libMesh::ReferenceCount edObject < DofObject > libMesh::Tri6 libMesh::InfHex16 libMesh::InfCell libMesh::InfHex libMesh::InfQuad libMesh::InfPrism libMesh::NodeElem libMesh::InfQuad4 libMesh::DofObject libMesh::InfHex18 libMesh::InfHex8 libMesh::InfPrism12 libMesh::Remot eElem libMesh::InfQuad6 libMesh::InfPrism6 Kirk. Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17. Peterson.

2013 27 / 215 . UT) libMesh::Edge3 libMesh::Edge4 The libMesh Finite Element Library libMesh::InfEdge2 June 17. INL. Peterson. Stogner (NASA.Introduction Library Design Disretization: Geometric Elements libMesh::ReferenceCount er libMesh::ReferenceCount edObject < DofObject > libMesh::ReferenceCount edObject < Elem > libMesh::DofObject libMesh::Elem libMesh::Edge libMesh::Edge2 Kirk.

2013 28 / 215 . UT) libMesh::Quad9 libMesh::Tri libMesh::Tri3 The libMesh Finite Element Library libMesh::Tri6 June 17.Introduction Library Design Disretization: Geometric Elements libMesh::ReferenceCount er libMesh::ReferenceCount edObject < DofObject > libMesh::ReferenceCount edObject < Elem > libMesh::DofObject libMesh::Elem libMesh::Face libMesh::Quad libMesh::Quad4 libMesh::Quad8 Kirk. INL. Peterson. Stogner (NASA.

INL. UT) The libMesh Finite Element Library June 17. 2013 29 / 215 .Introduction Library Design Disretization: Geometric Elements libMesh::Hex20 libMesh::Hex27 libMesh::Hex libMesh::Hex8 libMesh::Prism15 libMesh::Prism18 libMesh::Prism > libMesh::Elem libMesh::Prism6 libMesh::Cell libMesh::Pyramid libMesh::Tet libMesh::Pyramid5 libMesh::Tet 10 libMesh::Tet 4 Kirk. Peterson. Stogner (NASA.

UT) The libMesh Finite Element Library June 17. LAGRANGE_VEC > libMesh::FELagrangeVec< Dim > libMesh::FE< Dim. Stogner (NASA. L2_LAGRANGE > libMesh::FEL2Lagrange< Dim > libMesh::FE< Dim. INL.Introduction Library Design Disretization: Finite Elements Abst ract > libMesh::FEAbst ract libMesh::FE< Dim. SCALAR > libMesh::FEScalar< Dim > libMesh::FE< Dim. CLOUGH > libMesh::FEClough< Dim > libMesh::FE< Dim. Peterson. XYZ > libMesh::FEXYZ< Dim > libMesh::FEGenericBase< Out put Type > libMesh::FE< Dim. LAGRANGE > libMesh::FELagrange< Dim > libMesh::FEGenericBase< FEOut put Type< T >::t ype > libMesh::FE< Dim. L2_HIERARCHIC > libMesh::FEL2Hierarchic< Dim > libMesh::FE< Dim. T > libMesh::InfFE< Real > Kirk. NEDELEC_ONE > libMesh::FENedelecOne< Dim > libMesh::FE< Dim. HERMITE > libMesh::FEHermit e< Dim > libMesh::FE< Dim. HIERARCHIC > libMesh::FEHierarchic< Dim > libMesh::FE< Dim. MONOMIAL > libMesh::FEMonomial< Dim > libMesh::FEGenericBase< Real > libMesh::FE< Dim. 2013 30 / 215 .

2013 31 / 215 .Introduction Library Design Algorithms: Domain Partitioning Kirk. Stogner (NASA. Peterson. INL. UT) The libMesh Finite Element Library June 17.

INL. 2013 32 / 215 .Introduction Library Design Algorithms: Domain Partitioning libMesh::Cent roidPart it ioner libMesh::LinearPart it ioner libMesh::Part it ioner libMesh::Met isPart it ioner libMesh::Parmet isPart it ioner libMesh::Hilbert SFCPart it ioner libMesh::SFCPart it ioner libMesh::Mort onSFCPart it ioner Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson.

Introduction Library Design Algorithms: Error Estimation libMesh::Adjoint Refinement Est imat or libMesh::Adjoint ResidualErrorEst imat or libMesh::Discont inuit yMeasure libMesh::Exact ErrorEst imat or libMesh::ErrorEst imat or libMesh::KellyErrorEst imat or libMesh::JumpErrorEst imat or libMesh::LaplacianErrorEst imat or libMesh::Pat chRecoveryErrorEst imat or libMesh::Weight edPat chRecoveryErrorEst imat or libMesh::UniformRefinement Est imat or Kirk. Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL. 2013 33 / 215 . Peterson.

UT) The libMesh Finite Element Library June 17. Peterson. Stogner (NASA.Motivation 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. INL. 2013 34 / 215 .

UT) The libMesh Finite Element Library June 17. Stogner (NASA. INL. 2013 35 / 215 .Motivation Application Results Results from Physics Applications built on top of libMesh Kirk. Peterson.

Peterson.Motivation Application Results Compressible Navier-Stokes Kirk. Stogner (NASA. 2013 36 / 215 . UT) The libMesh Finite Element Library June 17. INL.

2013 36 / 215 . Peterson. INL. UT) The libMesh Finite Element Library June 17.Motivation Application Results Compressible Navier-Stokes Kirk. Stogner (NASA.

Peterson.Motivation Application Results Compressible Navier-Stokes Kirk. 2013 36 / 215 . INL. Stogner (NASA. UT) The libMesh Finite Element Library June 17.

2013 36 / 215 . Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL.Motivation Application Results Compressible Navier-Stokes Kirk. Peterson.

INL.Motivation Application Results Arcjet Nozzle Calculation Kirk. 2013 37 / 215 . Stogner (NASA. Peterson. UT) The libMesh Finite Element Library June 17.

Peterson. Stogner (NASA. INL. 2013 37 / 215 .Motivation Application Results Arcjet Nozzle Calculation Kirk. UT) The libMesh Finite Element Library June 17.

Motivation

Application Results

Arcjet Nozzle Calculation

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

38 / 215

Motivation

Application Results

Arcjet Nozzle Calculation

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

38 / 215

Motivation

Application Results

Coupled Pyrolysis, Temperature

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

39 / 215

Motivation

Application Results

Coupled Pyrolysis, Temperature

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

39 / 215

Motivation

Application Results

˙
Coupled Pyrolysis, Pyrolysis gas mass flux, m

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

40 / 215

UT) The libMesh Finite Element Library June 17. Pyrolysis gas mass flux. Stogner (NASA. Peterson. m Kirk.Motivation Application Results ˙ Coupled Pyrolysis. 2013 40 / 215 . INL.

Motivation Application Results Coupled Thermal/Solid Mechanics Kirk. INL. UT) The libMesh Finite Element Library June 17. 2013 41 / 215 . Stogner (NASA. Peterson.

Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL. Peterson. 2013 41 / 215 .Motivation Application Results Coupled Thermal/Solid Mechanics Kirk.

INL Hai Huang. LANL Ryosuke Park. INL The libMesh Finite Element Library June 17. 2013 42 / 215 .. INL Michael Tonks. Stogner (NASA. INL Chris Newman. INL David Andrs.Motivation Application Results The MOOSE Framework . INL Robert Podgorney. INL Cody Permann. Peterson.Gaston et al. INL MOOSE www. INL. Idaho National Laboratory Kirk. LANL Glen Hansen. UT) Collaborators: Dana Knoll.inl.gov Derek Gaston.

Motivation

Application Results

The MOOSE Framework - Gaston et al., INL

MOOSE – Multiphysics Object Oriented
Simulation Environment
•  A framework for solving computational nuclear engineering
problems in a well planned, managed, and coordinated way
–  Leveraged across multiple programs

•  Designed to significantly reduce the expense and time
required to develop new applications
•  Designed to develop analysis tools
–  Uses very robust solution methods
–  Designed to be easily extended and maintained
–  Efficient on both a few and many processors

•  Currently supports ~7 applications which are developed and
used by ~20 scientists.

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

43 / 215

Motivation

Application Results

The MOOSE Framework - Gaston et al., INL
MOOSE Ecosystem
Application

Physics

Start

Time To
Results

Lines of Code

BISON

Thermo-mechanics,
Chemical Diffusion,
coupled mesoscale

June 2008

4 Months

931

PRONGHORN

Neutronics, Porous
Flow, Eigenvalue

September
2008

3 Months

2,883

SALMON

Multiphase Porous
Flow

June 2009

3 Months

800

MARMOT

4th Order Phasefield
Mesoscale

August 2009

1 Month

838

RAT

Porous ReActive
Transport

August 2009

1 Month

439

FALCON

Geo-mechanics,
coupled mesoscale

September
2009

3 Months

810

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

44 / 215

Motivation

Application Results

The MOOSE Framework - Gaston et al., INL

BISON fuel performance
• 
• 
• 
• 
• 
• 

LWR, Triso, and TRU fuel performance code
Parallel 1D-3D thermomechanics code
Thermal, mechanical, and chemical models for FCI
Constituent redistribution
Material, fission product swelling, fission gas release models
Mesoscale-informed material models

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

45 / 215

Software Installation & Ecosystem

1

Introduction

2

Motivation

3

Software Installation & Ecosystem

4

A Generic Boundary Value Problem

5

Key Data Structures

6

Weighted Residuals

7

Poisson Equation

8

Other Examples

9

Some Extensions

10

Adaptive Mesh Refinement

11

Parallelism on Adaptive Unstructured Meshes

12

Going Further

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

46 / 215

Software Installation & Ecosystem

Important Websites

Important Websites

Primary website

Revision Control & Collaboration with GitHub

Continuous Integration with Buildbot

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

47 / 215

Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17.sourceforge. 2013 48 / 215 .Software Installation & Ecosystem Important Websites http://libmesh. INL.net Kirk.

UT) The libMesh Finite Element Library June 17.Software Installation & Ecosystem Important Websites http://github. Stogner (NASA.com/libMesh/libmesh Kirk. INL. Peterson. 2013 49 / 215 .

INL.utexas. Peterson.ices.edu:8010/waterfall Kirk. Stogner (NASA.Software Installation & Ecosystem Important Websites http://buildbot. 2013 50 / 215 . UT) The libMesh Finite Element Library June 17.

Peterson. Stogner (NASA.Software Installation & Ecosystem Compiling the library Building libMesh Kirk. 2013 51 / 215 . UT) The libMesh Finite Element Library June 17. INL.

git Kirk. INL.com/libMesh/libmesh.Software Installation & Ecosystem Compiling the library Getting the libMesh Source • Blessed. Peterson. 2013 52 / 215 .net/projects/libmesh/files/libmesh • Development tree: Grab the latest source tree from GitHub: $ git clone git://github. UT) The libMesh Finite Element Library June 17. Stable releases: Download prepackaged releases from http://sourceforge. Stogner (NASA.

2013 53 / 215 . UT) The libMesh Finite Element Library June 17. • Other recommended. • Highly recommended you first install PETSc and/or Trilinos. Stogner (NASA. • Intel’s Threading Building Blocks for shared-memory multithreading. Peterson.Software Installation & Ecosystem Compiling the library libMesh Suggested Dependencies • MPI is of course required for shared-memory parallelism. optional packages are: • SLEPc: eigenvalue support on top of PETSc. libMesh will build with support for serial linear systems. Kirk. INL. which libMesh uses for solving linear systems in parallel. • Out of the box.

Install.tar. Stogner (NASA. Configure.9.9. UT) The libMesh Finite Element Library June 17.bz2 && cd libmesh-0.1. 2013 54 / 215 .1 configure.Software Installation & Ecosystem Compiling the library Building libMesh from source Unpack. Peterson./configure --prefix=$PWD/install build & install make -j 4 && make -j 4 install run all the examples. Build. & Test # $ # $ # $ # $ unpack the distribution tar jxf libmesh-0. but only the optimized flavor make -j 4 check METHODS=opt Kirk. INL. install into the current directory .

/configure --prefix=$PWD/install \ --with-glpk-include=/opt/local/include \ --with-glpk-lib=/opt/local/lib \ --with-vtk-include=/opt/local/include/vtk-5. INL. Stogner (NASA.9.10 \ --with-eigen-include=/opt/local/include/eigen3 \ --with-cxx=clang++-mp-3.Software Installation & Ecosystem Compiling the library Building libMesh from source Advanced Configurations # $ # $ # $ unpack the distribution tar jxf libmesh-0.3 \ --disable-fortran # build & install $ make -j 4 && make -j 4 install Kirk.1 build in a subdirectory. allows multiple simultaneous builds mkdir -p clang && cd clang configure specifing optional packages & compilers .tar. Peterson.bz2 && cd libmesh-0.3 --with-cc=clang-mp-3.1.10 \ --with-vtk-lib=/opt/local/lib/vtk-5. 2013 55 / 215 . UT) The libMesh Finite Element Library June 17.9..

.. example installation adaptivity/ex2 ... Peterson.h .. Kirk..h ..... Stogner (NASA. Testing Testing Testing Testing Testing Testing .h .h . UT) The libMesh Finite Element Library June 17.. base/dof_map. base/auto_ptr.1/_inst/examples example installation adaptivity/ex1 ...... example installation adaptivity/ex4 .. example installation adaptivity/ex3 .. [ [ [ [ [ OK OK OK OK OK ] ] ] ] ] [ [ [ [ [ OK OK OK OK OK ] ] ] ] ] Checking for self-sufficient examples. INL. Testing Testing Testing Testing Testing .9.Software Installation & Ecosystem Compiling the library Building libMesh from source Testing the Installation $ make -j 4 installcheck Making installcheck in include Making installcheck in libmesh Checking for standalone headers in installed tree .. base/dirichlet_boundaries. example installation adaptivity/ex5 .. 2013 56 / 215 ..... Header Header Header Header Header libmesh/libmesh_config. examples in /tmp/libmesh-0. base/dof_object....h ..

UT) The libMesh Finite Element Library June 17. INL. 2013 57 / 215 . Peterson. Stogner (NASA.Software Installation & Ecosystem Compiling Applications Getting Started with libMesh Applications Kirk.

dylib libmesh_dbg. 2013 58 / 215 .common contrib examples lib bin etc include share $ ls lib | grep mesh libmesh_dbg.pc libmesh. INL. Stogner (NASA.la libmesh_opt. UT) The libMesh Finite Element Library June 17.pc libmesh-prof.Software Installation & Ecosystem Compiling Applications The libMesh installation Installation Tree # henceforth we assume LIBMESH_DIR points to the installation path $ cd $LIBMESH_DIR $ ls Make.pc Kirk.common libmesh-devel. Peterson.pc netcdf.dylib libmesh_dbg.0.dylib libmesh_opt.la libmesh_devel.dylib libmesh_opt.la $ ls lib/pkgconfig Make.pc libmesh-oprof.pc libmesh-dbg.0.0.dylib libmesh_devel.pc libmesh-opt.dylib libmesh_devel.

Software Installation & Ecosystem Compiling Applications The libMesh installation Compiling Simple Applications with pkg-config # make sure pkg-config can find the libMesh configuration $ export PKG_CONFIG_PATH=$LIBMESH_DIR/lib/pkgconfig:$PKG_CONFIG_PATH # copy the first example program $ cp -r $LIBMESH_DIR/examples/introduction/ex1 .C run. UT) The libMesh Finite Element Library June 17.sh # compile against the full debug version of libMesh $ mpicxx -o introduction_ex1 introduction_ex1.C \ ‘pkg-config --cflags --libs libmesh-dbg‘ # compile against the default.C \ ‘pkg-config --cflags --libs libmesh‘ Kirk. Stogner (NASA. Peterson. 2013 59 / 215 . && cd ex1 # see what we’ve got $ ls Makefile introduction_ex1. INL. optimized version of libMesh $ mpicxx -o introduction_ex1 introduction_ex1.

this is similar. Stogner (NASA. Peterson. INL. UT) The libMesh Finite Element Library June 17.Software Installation & Ecosystem Compiling Applications The libMesh installation Compiling Simple Applications with libmesh-config # # # $ we support a similar utility./configure $ libmesh-config --cxx mpicxx Kirk. but also can report the compiler used. PATH=$LIBMESH_DIR/bin:$PATH $ libmesh-config usage: libmesh-config libmesh-config libmesh-config libmesh-config libmesh-config libmesh-config libmesh-config --cppflags --cxxflags --include --libs --cxx --cc --fc --fflags --version --host # get the name of the compiler used. 2013 60 / 215 . which predates pkg-config support. as passed to . libmesh-config.

Linking example-dbg.Software Installation & Ecosystem Compiling Applications The libMesh installation Compiling Applications with make # copy the first example program $ cp -r $LIBMESH_DIR/examples/introduction/ex1 . && cd ex1 # compile against the default. UT) The libMesh Finite Element Library June 17.. Stogner (NASA. # compile against the full debug version of libMesh $ make METHOD=dbg Compiling C++ (in debug mode) introduction_ex1..C. Kirk..C... 2013 61 / 215 . Peterson. optimized version of libMesh $ make Compiling C++ (in optimized mode) introduction_ex1. Linking example-opt.... INL.

if (argc < 4) { if (libMesh::processor_id() == 0) std::cerr << "Usage: " << argv[0] << " -d 2 in.mesh [-o out.comm()). 2013 62 / 215 . argv). } Mesh mesh(init.print_info(). int main (int argc. libmesh_error(). Peterson. Stogner (NASA.Software Installation & Ecosystem Compiling Applications A first libMesh application #include <iostream> #include "libmesh/libmesh. mesh. if (argc >= 6 && std::string("-o") == argv[4]) mesh.h" using namespace libMesh. char** argv) { LibMeshInit init (argc.h" #include "libmesh/mesh. UT) The libMesh Finite Element Library June 17. mesh. INL.write (argv[5]). } Kirk.mesh]" << std::endl.read (argv[3]). return 0.

# run the example./example-opt -d 3 \ $LIBMESH_DIR/share/reference_elements/3D/one_hex27.Software Installation & Ecosystem Compiling Applications A first libMesh application # copy & build the example $ cp -r $LIBMESH_TUTORIAL/basic .. INL..xda -o out. UT) The libMesh Finite Element Library June 17. $ cd basic $ make Compiling C++ (in optimized mode) introduction_ex1..exo Mesh Information: mesh_dimension()=3 spatial_dimension()=3 n_nodes()=27 n_local_nodes()=27 n_elem()=1 n_local_elem()=1 n_active_elem()=1 n_subdomains()=1 n_partitions()=1 n_processors()=1 n_threads()=1 processor_id()=0 Kirk.. Stogner (NASA. Linking example-opt. Peterson. 2013 63 / 215 . reading a trivial mesh and writing output $ .C.

Stogner (NASA.A Generic Boundary Value Problem 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. Peterson. UT) The libMesh Finite Element Library June 17. 2013 64 / 215 . INL.

INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson.A Generic Boundary Value Problem Solving Problems the libMesh way Discretizing a Generic Boundary Value Problem Kirk. 2013 65 / 215 .

2013 66 / 215 .A Generic Boundary Value Problem A Generic BVP A general class of PDE ∂ΩD • We assume there is a Boundary Value Problem of the form to be approximated in an FE function space ∂u = F(u) M ∂t G(u) = 0 u = uD N(u) = 0 Ω ∈Ω ∈Ω ∈ ∂ΩD ∂ΩN ∈ ∂ΩN u(x. Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL. 0) = u0 (x) Kirk.

Peterson. INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA. 2013 67 / 215 .A Generic Boundary Value Problem A Generic BVP • Associated to the problem domain Ω is a libMesh data structure called a Mesh Ω h • A Mesh is essentially a collection of finite elements Ωh := [ Ωe e Kirk.

Kirk. UT) The libMesh Finite Element Library June 17. 2013 67 / 215 .A Generic Boundary Value Problem A Generic BVP • Associated to the problem domain Ω is a libMesh data structure called a Mesh Ω h • A Mesh is essentially a collection of finite elements Ωh := [ Ωe e • libMesh provides some simple structured mesh generation routines. and supports a rich set of input file formats. interfaces to Triangle and TetGen. Stogner (NASA. Peterson. INL.

Peterson.Key Data Structures 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. Stogner (NASA. INL. 2013 68 / 215 . UT) The libMesh Finite Element Library June 17.

INL./configure --enable-complex # turns on complex number support • The underlying linear algebra libraries must support the requested type. $ . Peterson./configure Gradient a tuple of type Number. depending on . Stogner (NASA. UT) The libMesh Finite Element Library June 17. whose size is the spatial dimension • libMesh can be compiled to support either real or complex-valued systems. Kirk./configure Number a Real or std::complex<Real>. 2013 69 / 215 . a double.Key Data Structures Data Types Data Types Real generally. depending on .

mesh.Key Data Structures The Mesh Class The Mesh int main (int argc. LibMeshInit init (argc. INL. Peterson.print_info().read ("in. mesh. UT) The libMesh Finite Element Library June 17. 2013 70 / 215 . other libraries and resources are // finalized. } Kirk. char** argv) { // Initialize the library.dat"). // Print information about the mesh to the screen..write ("out. argv). MPI and PETSc) // that require initialization before use. // Create a mesh Mesh mesh. // Read the input mesh. This is necessary because the library // may depend on a number of other libraries (i.e.exo"). When the LibMeshInit // object goes out of scope.. Stogner (NASA. // Write the output mesh mesh. .

xda * *************************************************************** Mesh Information: mesh_dimension()=3 spatial_dimension()=3 n_nodes()=27 n_local_nodes()=27 n_elem()=1 n_local_elem()=1 n_active_elem()=1 n_subdomains()=1 n_partitions()=1 n_processors()=1 n_threads()=1 processor_id()=0 Kirk.xda -o output. UT) The libMesh Finite Element Library June 17.Key Data Structures The Mesh Class The Mesh *************************************************************** * Running Example introduction_ex1: example-opt -d 3 reference_elements/3D/one_hex27. Stogner (NASA. Peterson. INL. 2013 71 / 215 .

Key Data Structures The Mesh Class Operations on Objects in the Mesh • From a Mesh it is trivial to access ranges of objects of interest through iterators. Peterson. UT) The libMesh Finite Element Library June 17. 2013 72 / 215 . • The nodes in the mesh. INL. for example. • Iterators are simply a mechanism for accessing a range of objects. • The “active” elements in the mesh assigned to the local processor in a parallel simulation. Stogner (NASA. Kirk. • All elements in the mesh. • libMesh makes extensive use of predictated iterators to access.

const Elem* elem = *el. // hence we use a variant of the \p active_elem_iterator.Key Data Structures The Mesh Class Mesh Iterators void foo (const MeshBase &mesh) { // Now we will loop over all the elements in the mesh that // live on the local processor. for ( .. el != end_el. } Kirk... MeshBase::const_element_iterator el = mesh.active_local_elements_begin(). Stogner (NASA. This allows for nicer syntax later. 2013 73 / 215 . Since the mesh // may be refined we want to only consider the ACTIVE elements. } .active_local_elements_end().. ++el) { // Store a pointer to the element we are currently // working on. We will compute the element // matrix and right-hand-side contribution. Peterson. const MeshBase::const_element_iterator end_el = mesh. UT) The libMesh Finite Element Library June 17. . INL.

Stogner (NASA. } Kirk. const MeshBase::const_node_iterator node_end = mesh. node_it != node_end.nodes_end(). UT) The libMesh Finite Element Library June 17. MeshBase::const_node_iterator node_it = mesh. Peterson.Key Data Structures The Mesh Class Mesh Iterators void foo (const MeshBase &mesh) { // We will now loop over all nodes. 2013 74 / 215 . } .nodes_begin(). ++node_it) { // the current node pointer const Node* node = *node_it.... .. for ( . INL.

which represents one or more coupled system of equations posed on the Mesh. • High-level operations such as solution input/output is usually handled at the EquationSystems level. INL. Kirk. UT) The libMesh Finite Element Library June 17. Peterson. each representing a logical system of equations. Stogner (NASA. • For a given Mesh. there can be an EquationSystems object. • The EquationSystems object can hold many System objects. 2013 75 / 215 .Key Data Structures The EquationSystems Class EquationSystems • The Mesh is a discrete representation of the geometry for a problem. • There is only one EquationSystems object per Mesh object.

SECOND.print_info(). // Create an equation systems object. EquationSystems equation_systems (mesh). FIRST).add_variable("u". equation_systems. FIRST). equation_systems.get_system("Complex System"). // Now we declare the system and its variables. FIRST). Variables "c" and "T" will use first-order // Lagrange approximation. and we give it the name "Complex System". Each system can contain multiple variables of different // approximation orders...add_variable("c". // Initialize the data structures for the equation system. // and we give it the name "Simple System". while variable "dv" will use a // second-order discontinuous approximation space. equation_systems. INL.get_system("Complex System"). MONOMIAL).Key Data Structures The EquationSystems Class EquationSystems .each with a different // approximation order. // Adds the variable "u" to "Simple System". equation_systems. Peterson. so the order of construction here // is important.init(). // Give "Complex System" three variables -.add_variable("T".add_variable("dv". Kirk. equation_systems.add_system<TransientLinearImplicitSystem> ("Simple System").get_system("Complex System").. // Next we’ll by add an "ExplicitSystem" to the EquationSystems // object. "u" will be // approximated using first-order approximation. equation_systems. equation_systems. UT) The libMesh Finite Element Library June 17. Stogner (NASA. // Prints information about the system to the screen. The EquationSystems object needs a // reference to the mesh object. equation_systems. We begin by adding // a "TransientLinearImplicitSystem" to the EquationSystems object.. 2013 76 / 215 . . This object can contain // multiple systems of different flavors for solving loosely coupled // physics.get_system("Simple System").add_system<ExplicitSystem> ("Complex System").

88889 Maximum On-Processor Bandwidth <= 10 Maximum Off-Processor Bandwidth <= 8 Kirk. 2013 77 / 215 . "Complex System" Type "Explicit" Variables={ "c" "T" } "dv" Finite Element Types="LAGRANGE" "MONOMIAL" Approximation Orders="FIRST" "SECOND" n_dofs()=222 n_local_dofs()=60 n_constrained_dofs()=0 n_local_constrained_dofs()=0 n_vectors()=1 n_matrices()=0 System #0. "Simple System" Type "TransientLinearImplicit" Variables="u" Finite Element Types="LAGRANGE" Approximation Orders="FIRST" n_dofs()=36 n_local_dofs()=12 n_constrained_dofs()=0 n_local_constrained_dofs()=0 n_vectors()=3 n_matrices()=1 DofMap Sparsity Average On-Processor Bandwidth <= 6.Key Data Structures The EquationSystems Class EquationSystems EquationSystems n_systems()=2 System #1. UT) The libMesh Finite Element Library June 17. Stogner (NASA.22222 Average Off-Processor Bandwidth <= 1. Peterson. INL.

INL. • An Elem is defined by Nodes.3D) and Faces (3D).Key Data Structures The Elem Class Elements • The Elem base class defines a geometric element in libMesh. Peterson. Stogner (NASA. 2013 78 / 215 . Edges (2D. UT) The libMesh Finite Element Library June 17. Kirk. • An Elem is sufficiently rich that in many cases it is the only argument required to provide to a function.

UT) The libMesh Finite Element Library June 17. . Peterson. INL.. side<elem->n_sides(). Stogner (NASA. } ... } Kirk. n<elem->n_nodes()... . If it has no neighbor. // then it lies on a physical boundary.. if (elem->neighbor(side) == NULL) { // Construct the side as a lower dimensional element AutoPtr<Elem> elem_side (elem->build_side(side)). } // Perform some operation for elements on the boundary for (unsigned int side=0. // get a user-specified material property. n++) { const Node *node = elem->get_node(n). side++) { // Every element knows its neigbor. based on // the subdomain the element belongs to const Real k_diff = my_matprop_func (elem->subdomain_id(). 2013 79 / 215 . *node).Key Data Structures The Elem Class Elements // access each Node on the element for (unsigned int n=0.

Stogner (NASA.Key Data Structures The Node Class Nodes • Nodes define spatial locations in arbitrary dimensions. Kirk. • Processor ownership. INL. UT) The libMesh Finite Element Library June 17. • Logically. Peterson. a Node is a point in N-space plus metadata: • Global ID. 2013 80 / 215 . • Degree of freedom indexing data.

Stogner (NASA. Peterson.Key Data Structures The Node Class Nodes libMesh::TypeVect or< Real > libMesh::Point libMesh::ReferenceCount edObject < DofObject > libMesh::DofObject libMesh::Node libMesh::ReferenceCount er libMesh::ReferenceCount edObject < Node > Kirk. INL. 2013 81 / 215 . UT) The libMesh Finite Element Library June 17.

i++) { _vmin[i] = std::min(_vmin[i].Key Data Structures The Node Class Nodes // loop over a range and determine the bounding box void bounding_box(const ConstNodeRange &range) { . } } ..end(). UT) The libMesh Finite Element Library June 17. i<LIBMESH_DIM. (*node)(i)). } .. INL.begin(). for (unsigned int i=0. // Query the number of DOFs for a particular node in a system const unsigned int n_dofs_per_node = node->n_dofs(sys_num).. Stogner (NASA.. _vmax[i] = std::max(_vmax[i]. it != range. 2013 82 / 215 .. Peterson. for (ConstNodeRange::const_iterator it = range.. (*node)(i)). Kirk. ++it) { const Node *node = *it.

INL. Stogner (NASA.Weighted Residuals 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. Peterson. 2013 83 / 215 . UT) The libMesh Finite Element Library June 17.

Stogner (NASA. Peterson.Weighted Residuals 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. UT) The libMesh Finite Element Library June 17. 2013 84 / 215 . INL.

INL. 2013 85 / 215 . Stogner (NASA. UT) ∀v ∈ V The libMesh Finite Element Library June 17. Peterson.Weighted Residuals • The point of departure in any FE analysis which uses libMesh is the weighted residual statement (F(u). v) = 0 Kirk.

2013 85 / 215 . INL. Peterson. the weighted residual statement associated with the finite-dimensional space V h ⊂ V (F(uh ). vh ) = 0 Kirk. UT) ∀vh ∈ V h The libMesh Finite Element Library June 17.Weighted Residuals • The point of departure in any FE analysis which uses libMesh is the weighted residual statement (F(u). more precisely. v) = 0 ∀v ∈ V • Or. Stogner (NASA.

Peterson.Weighted Residuals Some Examples Poisson Equation −∆u = f Kirk. INL. UT) ∈ Ω The libMesh Finite Element Library June 17. Stogner (NASA. 2013 86 / 215 .

Weighted Residuals Some Examples Poisson Equation −∆u = f ∈ Ω Weighted Residual Statement Z [∇u · ∇v − fv] dx (F(u). 2013 86 / 215 . INL. UT) The libMesh Finite Element Library June 17. Peterson. Stogner (NASA. v) := ΩZ (∇u · n) v ds + ∂ΩN Kirk.

2013 86 / 215 . INL. Peterson. UT) ∈ Ω The libMesh Finite Element Library June 17. Stogner (NASA.Weighted Residuals Some Examples Linear Convection-Diffusion −k∆u + b · ∇u = f Kirk.

Peterson. 2013 86 / 215 . UT) The libMesh Finite Element Library June 17. v) := Ω [k∇u · ∇v + (b · ∇u)v − fv] dx Z + k (∇u · n) v ds ∂ΩN Kirk. Stogner (NASA.Weighted Residuals Some Examples Linear Convection-Diffusion −k∆u + b · ∇u = f ∈ Ω Weighted Residual Statement Z (F(u). INL.

Peterson. 2013 86 / 215 .Weighted Residuals Some Examples Stokes Flow ∇p − ν∆u = f ∇·u = 0 Kirk. Stogner (NASA. INL. UT) ∈ Ω The libMesh Finite Element Library June 17.

p] . Peterson. Stogner (NASA. v := [v. UT) The libMesh Finite Element Library June 17. INL. q] Z [−p (∇ · v) + ν∇u :∇v − f · v Z + (∇ · u) q] dx + (ν∇u − pI) n · v ds (F(u).Weighted Residuals Some Examples Stokes Flow ∇p − ν∆u = f ∇·u = 0 ∈ Ω Weighted Residual Statement u := [u. 2013 86 / 215 . v) := Ω ∂ΩN Kirk.

and Ω ← Ωh in the weighted residual statement. Kirk. v ← vh . we simply replace u ← uh . Stogner (NASA. Peterson. UT) The libMesh Finite Element Library June 17.Weighted Residuals Some Examples • To obtain the approximate problem. INL. 2013 87 / 215 .

INL. Stogner (NASA. UT) The libMesh Finite Element Library June 17. Peterson.Poisson Equation 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. 2013 88 / 215 .

UT) The libMesh Finite Element Library June 17. Stogner (NASA. 2013 89 / 215 . INL. Peterson.Poisson Equation 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk.

Stogner (NASA. with ∂ΩN = ∅. UT) The libMesh Finite Element Library June 17. INL.Poisson Equation Weighted Residual Statement • For simplicity we start with the weighted residual statement arising from the Poisson equation. (F(uh ). 2013 90 / 215 . vh ) := Z  h  ∇u · ∇vh − fvh dx = 0 ∀vh ∈ V h Ωh Kirk. Peterson.

. Stogner (NASA.Poisson Equation Element Integrals • The integral over Ωh . UT) The libMesh Finite Element Library June 17. Z  0 =  ∇uh · ∇vh − fvh dx ∀vh ∈ V h Ωh Kirk. 2013 91 / 215 . . INL. Peterson.

UT)  Ωe The libMesh Finite Element Library June 17.Poisson Equation Element Integrals • The integral over Ωh . is written as a sum of integrals over the Ne finite elements: Z  ∇uh · ∇vh − fvh dx ∀vh ∈ V h  ∇uh · ∇vh − fvh dx ∀vh ∈ V h  0 = Ωh = Ne Z X e=1 Kirk. . Stogner (NASA. Peterson. INL. . 2013 91 / 215 .

0 ≤ i ≤ Ns .Poisson Equation Finite Element Basis Functions • An element integral will have contributions only from the global basis functions corresponding to its nodes. vh . Ωe • We call these local basis functions φi .

Ωe .

INL. 2013 92 / 215 . Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17. = Ns X 11111 00000 00000 11111 00000 11111 00000 11111 00000 11111 00000 11111 ci φi i=1 Kirk.

Poisson Equation Finite Element Basis Functions • An element integral will have contributions only from the global basis functions corresponding to its nodes. Ωe • We call these local basis functions φi . vh . 0 ≤ i ≤ Ns .

Ωe .

INL. = Ns X 11111 00000 00000 11111 00000 11111 00000 11111 00000 11111 00000 11111 ci φi i=1 Z Ωe vh dx = Ns X Z φi dx ci i=1 Kirk. Peterson. 2013 92 / 215 . UT) Ωe The libMesh Finite Element Library June 17. Stogner (NASA.

Z   ∇uh · ∇vh − fvh dx Ωe Kirk.Poisson Equation Element Matrix and Load Vector • The element integrals . . UT) The libMesh Finite Element Library June 17. . Peterson. Stogner (NASA. INL. 2013 93 / 215 .

. i = 1. . Stogner (NASA. 2013 93 / 215 . UT) f φi dx . . . Peterson. INL.Poisson Equation Element Matrix and Load Vector • The element integrals . . Z   ∇uh · ∇vh − fvh dx Ωe • are written in terms of the local “φi ” basis functions Ns X j=1 Z Z ∇φj · ∇φi dx − uj Ωe Kirk. Ns Ωe The libMesh Finite Element Library June 17. .

Poisson Equation

Element Matrix and Load Vector

• The element integrals . . .

Z  

∇uh · ∇vh − fvh dx

Ωe

• are written in terms of the local “φi ” basis functions
Ns
X
j=1

Z

Z
∇φj · ∇φi dx −

uj
Ωe

f φi dx ,

i = 1, . . . , Ns

Ωe

• This can be expressed naturally in matrix notation as

K e Ue − Fe

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

93 / 215

Poisson Equation

Global Linear System

• The entries of the element stiffness matrix are the integrals

K eij

Z
∇φj · ∇φi dx

:=
Ωe

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

94 / 215

Poisson Equation

Global Linear System

• The entries of the element stiffness matrix are the integrals

K eij

Z
∇φj · ∇φi dx

:=
Ωe

• While for the element right-hand side we have

Fei

Z
f φi dx

:=
Ωe

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

94 / 215

Poisson Equation

Global Linear System

• The entries of the element stiffness matrix are the integrals

K eij

Z
∇φj · ∇φi dx

:=
Ωe

• While for the element right-hand side we have

Fei

Z
f φi dx

:=
Ωe

• The element stiffness matrices and right-hand sides can be
“assembled” to obtain the global system of equations

KU = F

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

94 / 215

Poisson Equation

Reference Element Map

ˆe
• The integrals are performed on a “reference” element Ω

x

Ωe

Kirk, Peterson, Stogner (NASA, INL, UT)

x (ξ )

The libMesh Finite Element Library

ξ

Ωe

June 17, 2013

95 / 215

Poisson Equation Reference Element Map ˆe • The integrals are performed on a “reference” element Ω x x (ξ ) Ωe ξ Ωe • The Jacobian of the map x(ξ) is J . Stogner (NASA. 2013 95 / 215 . Peterson. Fei = Z Z f φi dx = Ωe Kirk. UT) ˆe Ω f (x(ξ))φi |J|dξ The libMesh Finite Element Library June 17. INL.

2013 95 / 215 . INL. Peterson. Stogner (NASA. UT) ˆe Ω ˆ ξ φj · ∇ ˆ ξ φi |J|dξ ∇ The libMesh Finite Element Library June 17.Poisson Equation Reference Element Map ˆe • The integrals are performed on a “reference” element Ω x x (ξ ) Ωe ξ Ωe ˆξ • Chain rule: ∇ = J −1 ∇ξ := ∇ K eij Z Z ∇φj · ∇φi dx = = Ωe Kirk.

INL. Kirk. UT) The libMesh Finite Element Library June 17. 2013 96 / 215 . Stogner (NASA.Poisson Equation Element Quadrature • The integrals on the “reference” element are approximated via numerical quadrature. Peterson.

Peterson. Kirk.Poisson Equation Element Quadrature • The integrals on the “reference” element are approximated via numerical quadrature. INL. • The quadrature rule has Nq points “ξq ” and weights “wq ”. Stogner (NASA. 2013 96 / 215 . UT) The libMesh Finite Element Library June 17.

Poisson Equation Element Quadrature • The integrals on the “reference” element are approximated via numerical quadrature. • The quadrature rule has Nq points “ξq ” and weights “wq ”. Fei Z = ˆe Ω Nq ≈ X f φi |J|dξ f (x(ξq ))φi (ξq )|J(ξq )|wq q=1 Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson. 2013 96 / 215 . INL.

2013 96 / 215 . Peterson. INL. Stogner (NASA.Poisson Equation Element Quadrature • The integrals on the “reference” element are approximated via numerical quadrature. K eij Z = ˆe Ω Nq ≈ X ˆ ξ φj · ∇ ˆ ξ φi |J|dξ ∇ ˆ ξ φj (ξq ) · ∇ ˆ ξ φi (ξq )|J(ξq )|wq ∇ q=1 Kirk. • The quadrature rule has Nq points “ξq ” and weights “wq ”. UT) The libMesh Finite Element Library June 17.

UT) value of ith shape fn. gradient The libMesh Finite Element Library June 17. 2013 97 / 215 .Poisson Equation libMesh Quadrature Point Data • libMesh provides the following variables at each quadrature point q Code Math Description JxW[q] |J(ξq )|wq Jacobian times weight phi[i][q] value of ith shape fn. dphi[i][q] φi (ξq ) ˆ ξ φi (ξq ) ∇ xyz[q] x(ξq ) location of ξq in physical space Kirk. Stogner (NASA. Peterson. INL.

2013 98 / 215 . for (q=0.j) += JxW[q]*(dphi[j][q]*dphi[i][q]). UT) The libMesh Finite Element Library June 17. Peterson. } Kirk. Stogner (NASA. ++q) for (i=0. i<Ns. for (j=0. INL. ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. j<Ns. ++j) Ke(i.Poisson Equation Matrix Assembly Loops • The libMesh representation of the matrix and rhs assembly is similar to the mathematical statements. q<Nq.

Poisson Equation Matrix Assembly Loops • The libMesh representation of the matrix and rhs assembly is similar to the mathematical statements. for (j=0. j<Ns. for (q=0.j) += JxW[q]*(dphi[j][q]*dphi[i][q]). Stogner (NASA. 2013 98 / 215 . ++j) Ke(i. Peterson. ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. } Fei = Nq X f (x(ξq ))φi (ξq )|J(ξq )|wq q=1 Kirk. q<Nq. INL. ++q) for (i=0. UT) The libMesh Finite Element Library June 17. i<Ns.

Poisson Equation Matrix Assembly Loops • The libMesh representation of the matrix and rhs assembly is similar to the mathematical statements. Peterson. ++j) Ke(i. i<Ns. q<Nq. } Fei = Nq X f (x(ξq ))φi (ξq )|J(ξq )|wq q=1 Kirk. for (q=0. j<Ns. 2013 98 / 215 .j) += JxW[q]*(dphi[j][q]*dphi[i][q]). ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA. for (j=0. ++q) for (i=0.

q<Nq. UT) The libMesh Finite Element Library June 17. Stogner (NASA. 2013 98 / 215 .Poisson Equation Matrix Assembly Loops • The libMesh representation of the matrix and rhs assembly is similar to the mathematical statements. for (j=0. i<Ns. Peterson. INL. ++j) Ke(i.j) += JxW[q]*(dphi[j][q]*dphi[i][q]). } Fei = Nq X f (x(ξq ))φi (ξq )|J(ξq )|wq q=1 Kirk. ++q) for (i=0. j<Ns. ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. for (q=0.

++q) for (i=0. UT) The libMesh Finite Element Library June 17. ++j) Ke(i. Peterson.Poisson Equation Matrix Assembly Loops • The libMesh representation of the matrix and rhs assembly is similar to the mathematical statements. q<Nq.j) += JxW[q]*(dphi[j][q]*dphi[i][q]). i<Ns. j<Ns. for (j=0. } Fei = Nq X f (x(ξq ))φi (ξq )|J(ξq )|wq q=1 Kirk. 2013 98 / 215 . Stogner (NASA. for (q=0. ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. INL.

Poisson Equation

Matrix Assembly Loops

• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i) {
Fe(i)
+= JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
}
K eij =

Nq
X

ˆ ξ φj (ξq ) · ∇
ˆ ξ φi (ξq )|J(ξq )|wq

q=1

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

98 / 215

Poisson Equation

Matrix Assembly Loops

• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i) {
Fe(i)
+= JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
}
K eij =

Nq
X

ˆ ξ φj (ξq ) · ∇
ˆ ξ φi (ξq )|J(ξq )|wq

q=1

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

98 / 215

Poisson Equation

Matrix Assembly Loops

• The libMesh representation of the matrix and rhs assembly is
similar to the mathematical statements.
for (q=0; q<Nq; ++q)
for (i=0; i<Ns; ++i) {
Fe(i)
+= JxW[q]*f(xyz[q])*phi[i][q];
for (j=0; j<Ns; ++j)
Ke(i,j) += JxW[q]*(dphi[j][q]*dphi[i][q]);
}
K eij =

Nq
X

ˆ ξ φj (ξq ) · ∇
ˆ ξ φi (ξq )|J(ξq )|wq

q=1

Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

98 / 215

Poisson Equation

Matrix Assembly Loops

// We now define the matrix assembly function for the Poisson system
// by computing the element matrices and right-hand sides. We are
// omitting BCs for brevity.
void assemble_poisson(EquationSystems& es,
const std::string& system_name)
{
// Get a constant reference to the mesh object.
const MeshBase& mesh = es.get_mesh();
// The dimension that we are running
const unsigned int dim = mesh.mesh_dimension();
// Get a reference to the LinearImplicitSystem we are solving
LinearImplicitSystem& system = es.get_system<LinearImplicitSystem>("Poisson");
// A reference to the DofMap object for this system. The DofMap
// object handles the index translation from node and element
// numbers to degree of freedom numbers. We will talk more about
// the DofMap in future examples.
const DofMap& dof_map = system.get_dof_map();
// Get a constant reference to the Finite Element type for the first
// (and only) variable in the system.
FEType fe_type = dof_map.variable_type(0);
// Build a Finite Element object of the specified type. Since the
// FEBase::build() member dynamically creates memory we will store
// the object as an AutoPtr<FEBase>. This can be thought of as a
// pointer that will clean up after itself.
AutoPtr<FEBase> fe (FEBase::build(dim, fe_type));
// A 5th order Gauss quadrature rule for numerical integration.
QGauss qrule (dim, FIFTH);
Kirk, Peterson, Stogner (NASA, INL, UT)

The libMesh Finite Element Library

June 17, 2013

99 / 215

Poisson Equation

Matrix Assembly Loops

// Tell the finite element object to use our quadrature rule.
fe->attach_quadrature_rule (&qrule);
// Here we define some references to cell-specific data that will be
// used to assemble the linear system. We begin with the element
// Jacobian * quadrature weight at each integration point.
const std::vector<Real>& JxW = fe->get_JxW();
// The physical XY locations of the quadrature points on the
// element. These might be useful for evaluating spatially varying
// material properties at the quadrature points.
const std::vector<Point>& q_point = fe->get_xyz();
// The element shape functions evaluated at the quadrature points.
const std::vector<std::vector<Real> >& phi = fe->get_phi();
// The element shape function gradients evaluated at the quadrature
// points.
const std::vector<std::vector<RealGradient> >& dphi = fe->get_dphi();
// Define data structures to contain the element matrix and
// right-hand-side vector contribution. Following basic finite
// element terminology we will denote these "Ke" and "Fe".
DenseMatrix<Number> Ke;
DenseVector<Number> Fe;
// This vector will hold the degree of freedom indices for the
// element. These define where in the global system the element
// degrees of freedom get mapped.
std::vector<dof_id_type> dof_indices;
// Now we will loop over all the elements in the mesh.

Kirk, Peterson, Stogner (NASA, INL, UT)

We will

The libMesh Finite Element Library

June 17, 2013

100 / 215

dof_map. // This allows for nicer syntax later. Peterson. MeshBase::const_element_iterator el = mesh.size()). // Compute the element-specific data for the current element. // Note that this will be the case if the element type is // different (i. the last element was a triangle. Stogner (NASA. const MeshBase::const_element_iterator end_el = mesh.e. dphi) for the current // element. 2013 101 / 215 . // Zero the element matrix and right-hand side before summing // them.Poisson Equation Matrix Assembly Loops // compute the element matrix and right-hand-side contribution.dof_indices (elem. INL. const Elem* elem = *el. Kirk. // These define where in the global matrix and right-hand-side // this element will contribute to. See // example 3 for a discussion of the element iterators. We use the resize member here because the number of // degrees of freedom might have changed from the last element. Ke.size(). el != end_el. // This involves computing the location of the quadrature points // (q_point) and the shape functions (phi.active_local_elements_begin(). fe->reinit (elem).active_local_elements_end(). UT) The libMesh Finite Element Library June 17. dof_indices.resize (dof_indices. for ( . dof_indices).size()). ++el) { // Store a pointer to the element we are currently working on.resize (dof_indices. Fe. // Get the degree of freedom indices for the current element. now we are // on a quadrilateral).

// If we are using an adaptive mesh this will apply any hanging // node constraint equations dof_map.*phi[i][qp]. i<phi. system. i<phi. i++) Fe(i) += JxW[qp]*10. INL. Stogner (NASA.heterogenously_constrain_element_matrix_and_vector (Ke. j<phi. // Now we build the element right-hand-side contribution. qp<qrule. // The element matrix and right-hand-side are now built for this // element.size(). qp++) for (unsigned int i=0.matrix->add_matrix (Ke. This involves a double // loop to integrate the test funcions (i) against the trial // functions (j).rhs->add_vector (Fe. 2013 102 / 215 . for (unsigned int qp=0. qp<qrule. The SparseMatrix::add_matrix() and // NumericVector::add_vector() members do this for us. dof_indices). j++) Ke(i.n_points().size(). This // involves a single loop in which we integrate the "forcing // function" in the PDE against the test functions.Poisson Equation Matrix Assembly Loops // Now we will build the element matrix. dof_indices). UT) The libMesh Finite Element Library June 17. i++) for (unsigned int j=0. for (unsigned int qp=0.size(). Fe. qp++) for (unsigned int i=0.n_points(). Peterson. dof_indices).j) += JxW[qp]*(dphi[i][qp]*dphi[j][qp]). Add them to the global matrix and right-hand-side // vector. } } Kirk. system.

Peterson.Poisson Equation Matrix Assembly Loops A Complete Program: poisson Kirk. 2013 103 / 215 . INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA.

const Real z = 0.5*pi*z).).5*pi*x)*sin(. INL. return cos(. Stogner (NASA. } private: EquationSystems &es. }.5*pi*y)*cos(. Real exact_solution (const Real x.) const { static const Real pi = acos(-1.Poisson Equation Matrix Assembly Loops Poisson class definition // headers omitted for brevity class Poisson : public System::Assembly { public: Poisson (EquationSystems &es_in) : es (es_in) {} void assemble (). Peterson. const Real y. UT) The libMesh Finite Element Library June 17. 2013 104 / 215 . Kirk.

h" "libmesh/equation_systems. #include "libmesh/quadrature_gauss.h" // Define useful datatypes for finite element // matrix and vector components. UT) The libMesh Finite Element Library June 17.h" #include "libmesh/numeric_vector.h" "libmesh/mesh_generation. INL.h" // Define Gauss quadrature rules. which handles degree of freedom // indexing.h" Kirk.Poisson Equation Matrix Assembly Loops Poisson main() I // C++ include files that we need #include <iostream> #include <algorithm> #include <math. #include "libmesh/dof_map.h" "libmesh/gnuplot_io.h" "libmesh/exodusII_io.h" "libmesh/linear_implicit_system.h" // Define the Finite Element object. Stogner (NASA.h" // Define the DofMap. #include "libmesh/sparse_matrix. Peterson.h" #include "libmesh/dense_matrix.h> #include <set> // Basic #include #include #include #include #include #include #include include file needed for the mesh functionality.h" "libmesh/mesh. 2013 105 / 215 . "libmesh/libmesh. #include "libmesh/fe.

Stogner (NASA.h" #include "libmesh/string_to_enum.h" // Bring in everything from the libMesh namespace using namespace libMesh. UT) The libMesh Finite Element Library June 17.Poisson Equation Matrix Assembly Loops Poisson main() II #include "libmesh/dense_vector.h" #include "poisson_problem. a performance logging utility.h" // Define the PerfLog.h" #include "libmesh/getpot.h" // The definition of a geometric element #include "libmesh/elem. // Define a wrapper for exact_solution that will be needed below Kirk. // It is useful for timing events in a code and giving // you an idea where bottlenecks lie. Real y. // Exact solution function Real exact_solution (const const const prototype. INL. Real x.). Real z = 0.h" #include "libmesh/analytic_function. 2013 106 / 215 . Peterson.h" // To impose Dirichlet boundary conditions #include "libmesh/dirichlet_boundaries. #include "libmesh/perf_log.

Poisson Equation Matrix Assembly Loops Poisson main() III void exact_solution_wrapper (DenseVector<Number>& output. char** argv) { // Initialize libMesh and any dependent libaries. if (argc < 3) { if (libMesh::processor_id() == 0) std::cerr << "Usage:\n" <<"\t " << argv[0] << " -d 2(3)" << " -n 15" << std::endl. 2013 107 / 215 . argv). Stogner (NASA. LibMeshInit init (argc. line number. // This handy function will print the file name. like in example 2. // Create a GetPot object to parse the command line GetPot command_line (argc. argv). (LIBMESH_DIM>1)?p(1):0. Peterson. const Real) { output(0) = exact_solution(p(0). // Declare a performance log for the main program // PerfLog perf_main("Main Program"). } // Begin the main program. (LIBMESH_DIM>2)?p(2):0). Kirk. UT) The libMesh Finite Element Library June 17. // Check for proper calling arguments. INL. int main (int argc. const Point& p.

search(1. } // Brief message to the user regarding the program name // and command line arguments. Use int // instead of unsigned since the GetPot overload is ambiguous // otherwise. if ( command_line. 2013 108 / 215 . // Read number of elements from command line Kirk. // Create a mesh with user-defined dimension. for (int i=1.next(dim). std::cout << std::endl << std::endl. Currrently the library does not use C++ // exception handling. Peterson. "2D/3D support"). Stogner (NASA. INL.Poisson Equation Matrix Assembly Loops Poisson main() IV // and then abort. } // Read problem dimension from command line. UT) The libMesh Finite Element Library June 17. "-d") ) dim = command_line. i++) std::cout << " " << argv[i]. libmesh_error(). i<argc. int dim = 2. // Skip higher-dimensional examples on a lower-dimensional libMesh build libmesh_example_assert(dim <= LIBMESH_DIM. else { std::cout << "Running " << argv[0].

// Cannot use discontinuous basis. 2013 109 / 215 . or \p Hex27 elements in 3D. } // Create a mesh. Building these higher-order elements allows Kirk. // Read FE order from command line std::string order = "SECOND".next(ps). // Read FE Family from command line std::string family = "LAGRANGE". // // // // Use the MeshTools::Generation mesh generator to create a uniform grid on the square [-1. We instruct the mesh generator to build a mesh of 8x8 \p Quad9 elements in 2D. if ( command_line.comm()).search(2. "-f") ) family = command_line. Mesh mesh(init.search(2. INL. "-FEFamily". "-Order". Stogner (NASA." << std::endl. if ( command_line. if ( command_line. Peterson.Poisson Equation Matrix Assembly Loops Poisson main() V int ps = 15.next(family). "-n") ) ps = command_line. distributed // across the default MPI communicator. "-o") ) order = command_line. libmesh_error().next(order).search(1. with dimension to be overridden later. UT) The libMesh Finite Element Library June 17. if ((family == "MONOMIAL") || (family == "XYZ")) { if (libMesh::processor_id() == 0) std::cerr << "ex4 currently requires a Cˆ0 (or higher) FE basis.1]ˆD.

(dim>2) ? ps : 0. Real halfheight = dim > 2 ? 1. ps.. -1.. MeshTools::Generation::build_cube (mesh.. : 0. UT) The libMesh Finite Element Library June 17. -1. -halfheight. (dim>2) ? ps : 0. -halfwidth.Poisson Equation Matrix Assembly Loops Poisson main() VI // us to use higher-order approximation. halfheight. halfwidth. (dim>1) ? ps : 0. 1. (dim==1) ? EDGE3 : Kirk. ps. as in example 3.. : 0. -halfheight. halfwidth. Real halfwidth = dim > 1 ? 1. (dim>1) ? ps : 0. if ((family == "LAGRANGE") && (order == "FIRST")) { // No reason to use high-order geometric elements if we are // solving with low-order finite elements.. Stogner (NASA. INL. 1. -halfwidth. } else { MeshTools::Generation::build_cube (mesh. 2013 110 / 215 . halfheight. Peterson.. (dim==1) ? EDGE2 : ((dim == 2) ? QUAD4 : HEX8)).

attach_assemble_object (poisson). mesh. // Construct a Dirichlet boundary condition object Kirk. Stogner (NASA. // Add the variable "u" to "Poisson".print_info(). // Declare the system and its variables. system. "u" // will be approximated using second-order approximation. Utility::string_to_enum<Order> (order). // Give the system a pointer to the matrix assembly // function. Peterson. INL.Poisson Equation Matrix Assembly Loops Poisson main() VII ((dim == 2) ? QUAD9 : HEX27)). // Create a system named "Poisson" LinearImplicitSystem& system = equation_systems. Utility::string_to_enum<FEFamily>(family)). } // Print information about the mesh to the screen. EquationSystems equation_systems (mesh). unsigned int u_var = system. UT) The libMesh Finite Element Library June 17. // Create an equation systems object. 2013 111 / 215 .add_variable("u". Poisson poisson(equation_systems).add_system<LinearImplicitSystem> ("Poisson").

2 and 3 if(dim>=2) { boundary_ids.Poisson Equation Matrix Assembly Loops Poisson main() VIII // Indicate which boundary IDs we impose the BC on // We either build a line.insert(5). 2013 112 / 215 . // Create an AnalyticFunction object that we use to project the BC // This function just calls the function exact_solution via exact_solution_wrapper AnalyticFunction<> exact_solution_object(exact_solution_wrapper). 4 and 5 if(dim==3) { boundary_ids. INL.insert(0). boundary_ids. boundary_ids. a square or a cube. 3.insert(1).insert(4). UT) The libMesh Finite Element Library June 17. boundary_ids. 1.insert(3). and // here we indicate the boundaries IDs in each case std::set<boundary_id_type> boundary_ids. DirichletBoundary dirichlet_bc(boundary_ids. 1. variables[0] = u_var.insert(2). // the dim==2 mesh has four boundaries with IDs 0. Stogner (NASA. Peterson. Kirk. 2. // the dim==1 mesh has two boundaries with IDs 0 and 1 boundary_ids. } // Create a vector storing the variable numbers which the BC applies to std::vector<unsigned int> variables(1). } // the dim==3 mesh has four boundaries with IDs 0.

equation_systems).e" : "out_2. UT) The libMesh Finite Element Library June 17.equation_systems). // We must add the Dirichlet boundary condition _before_ // we call equation_systems.print_info(). just like example 2. equation_systems. plot.Poisson Equation Matrix Assembly Loops Poisson main() IX variables. mesh.write_equation_systems("gnuplot_script".write_equation_systems ((dim == 3) ? "out_3. Stogner (NASA. // Solve the system "Poisson". } #ifdef LIBMESH_HAVE_EXODUS_API else { ExodusII_IO (mesh).add_dirichlet_boundary(dirichlet_bc). INL. 2013 113 / 215 . system. Kirk.e". &exact_solution_object).get_dof_map().print_info(). // After solving the system write the solution // to a GMV-formatted plot file.GnuPlotIO::GRID_ON). // Initialize the data structures for the equation system. Peterson. equation_systems. 1D".solve(). if(dim == 1) { GnuPlotIO plot(mesh."Introduction Example 4.init(). // Print information about the system to the screen.init() system.

Peterson. INL. return 0. 2013 114 / 215 . Stogner (NASA. } Kirk. UT) The libMesh Finite Element Library June 17.Poisson Equation Matrix Assembly Loops Poisson main() X } #endif // #ifdef LIBMESH_HAVE_EXODUS_API // All done.

The \p DofMap // object handles the index translation from node and element numbers // to degree of freedom numbers. FEType fe_type = dof_map.get_dof_map(). // Get a constant reference to the Finite Element type // for the first (and only) variable in the system.get_system<LinearImplicitSystem>("Poisson").get_mesh(). // A reference to the \p DofMap object for this system. Peterson. We will talk more about the \p DofMap // in future examples. const DofMap& dof_map = system.h" void Poisson::assemble () { // Declare a performance log. since there may be many PerfLogs in an // application.mesh_dimension(). // Get a constant reference to the mesh object. INL. // Get a reference to the LinearImplicitSystem we are solving LinearImplicitSystem& system = es.variable_type(0). Stogner (NASA. const MeshBase& mesh = es. // The dimension that we are running const unsigned int dim = mesh. 2013 115 / 215 .Poisson Equation Matrix Assembly Loops Poisson assembly() I #include "poisson_problem. Kirk. UT) The libMesh Finite Element Library June 17. Give it a descriptive // string to identify what part of the code we are // logging. PerfLog perf_log ("Matrix Assembly").

UT) The libMesh Finite Element Library June 17. fe_type)).Poisson Equation Matrix Assembly Loops Poisson assembly() II // Build a Finite Element object of the specified type. QGauss qface(dim-1. Peterson. Since the // \p FEBase::build() member dynamically creates memory we will // store the object as an \p AutoPtr<FEBase>. Kirk. fe_face->attach_quadrature_rule (&qface). fe->attach_quadrature_rule (&qrule). // A 5th order Gauss quadrature rule for numerical integration. 2013 116 / 215 . AutoPtr<FEBase> fe_face (FEBase::build(dim. QGauss qrule (dim. FIFTH). // Boundary integration requires one quadraure rule. // with dimensionality one less than the dimensionality // of the element. fe_type)). INL. // Declare a special finite element object for // boundary integration. FIFTH). // Tell the finte element object to use our // quadrature rule. AutoPtr<FEBase> fe (FEBase::build(dim. This can be thought // of as a pointer that will clean up after itself. We begin with the element Jacobian * quadrature weight at each integration point. // // // // Here we define some references to cell-specific data that will be used to assemble the linear system. Stogner (NASA. // Tell the finite element object to use our quadrature rule.

const std::vector<std::vector<RealGradient> >& dphi = fe->get_dphi(). Peterson. // // // // Now we will loop over all the elements in the mesh.Poisson Equation Matrix Assembly Loops Poisson assembly() III const std::vector<Real>& JxW = fe->get_JxW(). // The element shape functions evaluated at the quadrature points. 2013 117 / 215 . Kirk. const std::vector<Point>& q_point = fe->get_xyz(). See example 3 for a discussion of the element iterators. std::vector<dof_id_type> dof_indices. We will compute the element matrix and right-hand-side contribution. DenseMatrix<Number> Ke. These define where in the global system // the element degrees of freedom get mapped. Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17. // The physical XY locations of the quadrature points on the element. const std::vector<std::vector<Real> >& phi = fe->get_phi(). // The element shape function gradients evaluated at the quadrature // points. // These might be useful for evaluating spatially varying material // properties at the quadrature points. More detail is in example 3. // Define data structures to contain the element matrix // and right-hand-side vector contribution. Following // basic finite element terminology we will denote these // "Ke" and "Fe". // This vector will hold the degree of freedom indices for // the element. DenseVector<Number> Fe.

UT) The libMesh Finite Element Library June 17. const Elem* elem = *el. const MeshBase::const_element_iterator end_el = mesh. 2013 118 / 215 . Stogner (NASA. Note that this will be the case if the Kirk. perf_log.active_local_elements_end(). ++el) { // Start logging the shape function initialization. // Store a pointer to the element we are currently // working on.dof_indices (elem. dof_map. This involves computing the location of the // quadrature points (q_point) and the shape functions // (phi. dof_indices).Poisson Equation Matrix Assembly Loops Poisson assembly() IV MeshBase::const_element_iterator el = mesh. for ( . This allows for nicer syntax later.push("elem init"). // Compute the element-specific data for the current // element. fe->reinit (elem). el != end_el. // // // // Zero the element matrix and right-hand side before summing them. // This is done through a simple function call with // the name of the event to log. dphi) for the current element. // Get the degree of freedom indices for the // current element. These define where in the global // matrix and right-hand-side this element will // contribute to. Peterson. We use the resize member here because the number of degrees of freedom might have changed from the last element. INL.active_local_elements_begin().

// If you forget to stop logging an event the PerfLog // object will probably catch the error and abort.pop("elem init"). perf_log. now we are on a quadrilateral). Fe. i++) for (unsigned int j=0. // Now we will build the element matrix. i<phi. qp++) for (unsigned int i=0. j++) Ke(i. // Stop logging the shape function initialization.size().resize (dof_indices.size()).e. INL.size()). // Stop logging the matrix computation Kirk. UT) The libMesh Finite Element Library June 17. Ke. for (unsigned int qp=0. qp<qrule.n_points(). j<phi.size().push ("Ke"). 2013 119 / 215 .Poisson Equation Matrix Assembly Loops Poisson assembly() V // element type is different (i. dof_indices. the last element was a // triangle. Stogner (NASA.size().resize (dof_indices. // // Now start logging the element matrix computation perf_log.j) += JxW[qp]*(dphi[i][qp]*dphi[j][qp]). // // We have split the numeric integration into two loops // so that we can log the matrix and right-hand-side // computation seperately. This involves // a double loop to integrate the test funcions (i) against // the trial functions (j). Peterson.

Stogner (NASA.j) + u(i+1. // // Start logging the right-hand-side computation perf_log. 2013 120 / 215 . which in 2D on a structured grid is // // u_xx + u_yy = (u(i-1. UT) The libMesh Finite Element Library June 17. // In this case we set fxy to be a finite difference // Laplacian approximation to the (known) exact solution. for (unsigned int qp=0.pop ("Ke").push ("Fe"). // Now we build the element right-hand-side contribution.Poisson Equation Matrix Assembly Loops Poisson assembly() VI perf_log. #if LIBMESH_DIM > 1 const Real y = q_point[qp](1). outside of the i-loop const Real x = q_point[qp](0). // This involves a single loop in which we integrate the // "forcing function" in the PDE against the test functions. qp<qrule.n_points().. qp++) { // fxy is the forcing function for the Poisson equation. // // We will use the second-order accurate FD Laplacian // approximation.j) + // u(i.j))/hˆ2 // // Since the value of the forcing function depends only // on the location of the quadrature point (q_point[qp]) // we will compute it here.j+1) + // -4*u(i. INL.j-1) + u(i. Peterson. Kirk. #else const Real y = 0.

2013 121 / 215 . const Real uyy = (exact_solution(x.y.z))/eps/eps.y-eps.*exact_solution(x. UT) The libMesh Finite Element Library June 17.z) + -2.25*pi*pi)*sin(.z) + exact_solution(x+eps.z+eps) + -2.z) + -2.z) + exact_solution(x.z-eps) + exact_solution(x. const Real pi = libMesh::pi.e-3. Real fxy.5*pi*x). const Real uzz = (exact_solution(x.y. Real z = 0.y.y.y. const Real uxx = (exact_solution(x-eps.y+eps.y.*exact_solution(x. fxy = (0. } else { Kirk. Real eps = 1.z))/eps/eps. Peterson. if(dim==1) { // In 1D.Poisson Equation Matrix Assembly Loops Poisson assembly() VII #endif #if LIBMESH_DIM const #else const #endif const > 2 Real z = q_point[qp](2).y. Stogner (NASA. INL.z))/eps/eps..*exact_solution(x. compute the rhs by differentiating the // exact solution twice.

note that here we call heterogenously_constrain_element_matrix_and_vector // to impose a inhomogeneous Dirichlet boundary conditions. dof_indices). The \p SparseMatrix::add_matrix() // and \p NumericVector::add_vector() members do this for us. // If this assembly program were to be used on an adaptive mesh. dof_indices). Stogner (NASA. UT) The libMesh Finite Element Library June 17.push ("matrix insertion"). i<phi.matrix->add_matrix (Ke. } // Add the RHS contribution for (unsigned int i=0. // Start logging the insertion of the local (element) // matrix and vector into the global matrix and vector Kirk. dof_indices). // The element matrix and right-hand-side are now built // for this element. Peterson.size(). } // Stop logging the right-hand-side computation perf_log. // Start logging the insertion of the local (element) // matrix and vector into the global matrix and vector perf_log.pop ("Fe"). i++) Fe(i) += JxW[qp]*fxy*phi[i][qp].Poisson Equation Matrix Assembly Loops Poisson assembly() VIII fxy = . system.rhs->add_vector (Fe.heterogenously_constrain_element_matrix_and_vector (Ke. system. : uzz)). // we would have to apply any hanging node constraint equations // Also.(uxx + uyy + ((dim==2) ? 0. Fe. dof_map. INL. 2013 122 / 215 . Add them to the global matrix and // right-hand-side vector.

huh? } Kirk. 2013 123 / 215 . } // That’s it.Poisson Equation Matrix Assembly Loops Poisson assembly() IX perf_log. UT) The libMesh Finite Element Library June 17. We don’t need to do anything else to the // PerfLog. Peterson. When it goes out of scope (at this function return) // it will print its log to the screen. Stogner (NASA. INL.pop ("matrix insertion"). Pretty easy.

Peterson. 2013 124 / 215 .Poisson Equation Matrix Assembly Loops Running the program Running the program # $ $ $ copy & build the example cp -r $LIBMESH_TUTORIAL/poisson ./example-opt -d 2 -n 20 # run the example in 3D with 20 elements in each direction $ . cd poisson make # run the example in 2D with 20 elements in each direction $ . UT) The libMesh Finite Element Library June 17. Stogner (NASA./example-opt -d 3 -n 20 --use-trilinos Kirk./example-opt -d 3 -n 20 # ’’ ’’ except using the Trilinos linear solvers $ . INL.

UT) The libMesh Finite Element Library June 17. INL. 2013 125 / 215 . Stogner (NASA. Peterson.Poisson Equation Matrix Assembly Loops Output Kirk.

Peterson. INL.Poisson Equation Matrix Assembly Loops Extension: Multithreaded Assembly: poisson threaded Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. 2013 126 / 215 .

2013 127 / 215 . Peterson.Poisson Equation Matrix Assembly Loops Multithreading in libMesh • The libMesh::Threads::() namespace provides a transparent wrapper for Intel’s Threading Building Blocks. • To get the most benefit from threads you’ll want to write a threaded assembly routine as well. this is easy. • Fortunately. make sure you are compiled with --enable-tbb and add --n threads=# to the command line. Kirk. • to use it. Stogner (NASA. • This is used extensively inside the library to speed up compute-heavy operations. UT) The libMesh Finite Element Library June 17. INL.

5*pi*y)*cos(. UT) The libMesh Finite Element Library June 17. } private: EquationSystems &es.) const { static const Real pi = acos(-1.). Kirk. INL. Peterson. 2013 128 / 215 .5*pi*x)*sin(. return cos(. Stogner (NASA.Poisson Equation Matrix Assembly Loops Poisson class definition // headers omitted for brevity class Poisson : public System::Assembly { public: Poisson (EquationSystems &es_in) : es (es_in) {} void assemble (). Real exact_solution (const Real x. mutable Threads::spin_mutex assembly_mutex. const Real z = 0. void operator()(const ConstElemRange &range) const. }. const Real y.5*pi*z).

active_local_elements_begin(). } void Poisson::operator()(const ConstElemRange &range) const { .get_mesh(). dof_indices). Threads::parallel_for (// the range over which we will perform threaded operations assembly_elem_range. INL.rhs->add_vector (Fe. { Threads::spin_mutex::scoped_lock lock(assembly_mutex).active_local_elements_end()). so we // must be careful to lock for exclusive access.. 2013 129 / 215 . // the function object to apply to each element in the range *this). UT) The libMesh Finite Element Library June 17.matrix->add_matrix (Ke. system. // insert the local (per-thread) element matrix/vector into // the global matrix/vector. This is a shared object. mesh. dof_indices). Stogner (NASA. ConstElemRange assembly_elem_range (mesh. system.Poisson Equation Matrix Assembly Loops Threaded Poisson assembly #include "poisson_problem. Peterson. } } Kirk.h" void Poisson::assemble () { const MeshBase& mesh = es..

Stogner (NASA. Peterson./example-opt -d 3 -n .Poisson Equation Matrix Assembly Loops Running the program Running the program # $ $ $ copy & build the example cp -r $LIBMESH_TUTORIAL/poisson_threaded ./example-opt -d 3 -n ./example-opt -d 3 -n Kirk. UT) with 20 elements in each direction of threads 20 --n_threads=1 20 --n_threads=2 20 --n_threads=4 The libMesh Finite Element Library June 17./example-opt -d 2 -n 20 # # $ $ $ run the example in 3D using various numbers . INL. 2013 130 / 215 . cd poisson_threaded make # run the example in 2D with 20 elements in each direction $ .

UT) The libMesh Finite Element Library June 17.Other Examples 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. 2013 131 / 215 . Peterson. Stogner (NASA. INL.

2013 132 / 215 .Other Examples 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. Stogner (NASA. INL. Peterson. UT) The libMesh Finite Element Library June 17.

} Kirk. ++q) for (i=0.j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q]) +(b*dphi[j][q])*phi[i][q]). ++j) Ke(i. −k∆u + b · ∇u = f for (q=0. j<Ns. UT) The libMesh Finite Element Library June 17. INL. i<Ns. for (j=0. 2013 133 / 215 . ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. q<Nq. Peterson.Other Examples Convection-Diffusion Equation • The matrix assembly routine for the linear convection-diffusion equation. Stogner (NASA.

j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q]) +(b*dphi[j][q])*phi[i][q]). UT) The libMesh Finite Element Library June 17. } Kirk.Other Examples Convection-Diffusion Equation • The matrix assembly routine for the linear convection-diffusion equation. −k∆u + b · ∇u = f for (q=0. for (j=0. Peterson. ++j) Ke(i. j<Ns. q<Nq. ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q]. i<Ns. ++q) for (i=0. INL. Stogner (NASA. 2013 133 / 215 .

j<Ns. for (j=0. ++q) for (i=0. INL. q<Nq. Stogner (NASA. ++j) Ke(i. UT) The libMesh Finite Element Library June 17. } Kirk. i<Ns. 2013 133 / 215 .Other Examples Convection-Diffusion Equation • The matrix assembly routine for the linear convection-diffusion equation. −k∆u + b · ∇u = f for (q=0. Peterson.j) += JxW[q]*(k*(dphi[j][q]*dphi[i][q]) +(b*dphi[j][q])*phi[i][q]). ++i) { Fe(i) += JxW[q]*f(xyz[q])*phi[i][q].

Peterson. 2013 134 / 215 . ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • We have an array of submatrices: Ke[ ][ ] Kirk. INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA.Other Examples Stokes Flow • For multi-variable systems like Stokes flow.

Stogner (NASA.Other Examples Stokes Flow • For multi-variable systems like Stokes flow. INL. Peterson. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • We have an array of submatrices: Ke[0][0] Kirk. UT) The libMesh Finite Element Library June 17. 2013 134 / 215 .

INL. 2013 134 / 215 . Peterson. Stogner (NASA.Other Examples Stokes Flow • For multi-variable systems like Stokes flow. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • We have an array of submatrices: Ke[1][1] Kirk. UT) The libMesh Finite Element Library June 17.

2013 134 / 215 . UT) The libMesh Finite Element Library June 17. INL.Other Examples Stokes Flow • For multi-variable systems like Stokes flow. Peterson. Stogner (NASA. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • We have an array of submatrices: Ke[2][1] Kirk.

Stogner (NASA. 2013 134 / 215 . Kirk.Other Examples Stokes Flow • For multi-variable systems like Stokes flow. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • And an array of right-hand sides: Fe[]. UT) The libMesh Finite Element Library June 17. Peterson. INL.

Peterson. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • And an array of right-hand sides: Fe[0].Other Examples Stokes Flow • For multi-variable systems like Stokes flow. Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL. 2013 134 / 215 . Kirk.

Other Examples Stokes Flow • For multi-variable systems like Stokes flow. Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17. Kirk. ∇p − ν∆u = f ∇·u = 0 ∈ Ω ⊂ R2 • The element stiffness matrix concept can extended to include sub-matrices Kue1 u1  Kue u 2 1 e Kpu 1  Kue1 u2 Kue2 u2 e Kpu 2  e   e  Kue1 p Uu1 Fu1 Kue2 p   Uue2  −  Fue2  e Upe Fpe Kpp • And an array of right-hand sides: Fe[1]. INL. 2013 134 / 215 .

Stogner (NASA. Peterson. i<Ns. UT) The libMesh Finite Element Library June 17. ++q) for (d=0.d)*phi[i][q]. 2013 135 / 215 .Other Examples Stokes Flow • The matrix assembly can proceed in essentially the same way. • For the momentum equations: for (q=0. for (j=0. INL. d<2. } Kirk. q<Nq. j<Ns.j) += JxW[q]*nu*(dphi[j][q]*dphi[i][q]). ++i) { Fe[d](i) += JxW[q]*f(xyz[q]. ++j) Ke[d][d](i. ++d) for (i=0.

UT) The libMesh Finite Element Library June 17.Some Extensions 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. 2013 136 / 215 . Stogner (NASA. INL. Peterson.

INL. 2013 137 / 215 . UT) The libMesh Finite Element Library June 17. Stogner (NASA.Some Extensions 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. Peterson.

Stogner (NASA. UT) The libMesh Finite Element Library June 17.Some Extensions Time-Dependent Problems • For linear problems. we have already seen how the weighted residual statement leads directly to a sparse linear system of equations KU = F Kirk. INL. 2013 138 / 215 . Peterson.

e. 2013 138 / 215 . Stogner (NASA. ∂u = F(u) ∂t • we also need a way to advance the solution in time.v ∆t  = F(uθ ).g. UT) The libMesh Finite Element Library June 17.Some Extensions Time-Dependent Problems • For time-dependent problems. vh  ∀vh ∈ V h uθ := θun+1 + (1 − θ)un • Leads to KU = F at each timestep. Kirk. Peterson. INL. a θ -method  un+1 − un h .

e. typically a sequence of linear problems must be solved. Peterson. 2013 139 / 215 .g. v) where F 0 (uk ) is the linearized (Jacobian) operator associated with the PDE. ∂F • Must solve ∂U δU = −F (Inexact Newton method) at each iteration step. for Newton’s method (F 0 (uk )δuk+1 . v) = −(F(uk ).Some Extensions Nonlinear Problems • For nonlinear problems. Kirk. Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17.

INL. Stogner (NASA. Peterson.Some Extensions Nonlinear Problems Examples: Nonlinear & Transient Problems laplace young transient convection diffusion navier stokes Kirk. UT) The libMesh Finite Element Library June 17. 2013 140 / 215 .

2013 141 / 215 . UT) The libMesh Finite Element Library June 17.Some Extensions Nonlinear Problems Lapace-Young “minimal surface” problem The Laplace-Young equation governs the behavior of films. which seek to form a minimal surface:  −∇ · ∇u √ 1 + ∇u · ∇u  + κu = 0 or equivalently −∇ · (K (u) ∇u) + κu = 0 This problem behaves like a Helmholtz problem with nonlinear diffusion coefficient. K(u). Peterson. Stogner (NASA. INL. Kirk.

}. NonlinearImplicitSystem &system). NumericVector<Number> &resid. private: EquationSystems &es. Peterson. SparseMatrix<Number> &jacobian. UT) The libMesh Finite Element Library June 17. Stogner (NASA. NonlinearImplicitSystem &system). virtual void residual (const NumericVector<Number> &soln.Some Extensions Nonlinear Problems Laplace-Young Assembly // headers omitted for brevity class LaplaceYoung : public NonlinearImplicitSystem::ComputeJacobian. public NonlinearImplicitSystem::ComputeResidual { public: LaplaceYoung (EquationSystems &es_in) : es(es_in) {} virtual void jacobian (const NumericVector<Number> &soln. 2013 142 / 215 . Kirk. INL.

/std::sqrt(1. for (unsigned int i=0.. Peterson. This involves // a double loop to integrate the test funcions (i) against // the trial functions (j).size(). Stogner (NASA.. const Number K = 1. } Kirk. // for (unsigned int qp=0. . i++) grad_u += dphi[i][qp]*soln(dof_indices[i]).n_points(). 2013 143 / 215 . dof_indices). // Now we will build the element Jacobian. INL. Note that the Jacobian depends // on the current solution x. i<phi.Some Extensions Nonlinear Problems Laplace-Young Assembly // Get the degree of freedom indices for the // current element. + grad_u*grad_u). qp<qrule. qp++) { Gradient grad_u. which we access using the soln // vector. UT) The libMesh Finite Element Library June 17.dof_indices (elem. dof_map.

Some Extensions Nonlinear Problems Running the laplace young program Running the program # $ $ $ copy the example cp -r $LIBMESH_TUTORIAL/laplace_young . 2013 144 / 215 ./example-opt -r 3 -o SECOND Kirk. Peterson. cd laplace_young make # run the example with 3 uniform refinement steps./example-opt -r 3 -o FIRST # run the example with 3 uniform refinement steps. using first # order Lagrange elements $ . Stogner (NASA. using first # order Lagrange elements $ . INL. UT) The libMesh Finite Element Library June 17.

Some Extensions Nonlinear Problems Output Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. INL. 2013 145 / 215 . Peterson.

cd transient_convection_diffusion make # run the example $ ./example-opt Kirk. Peterson.Some Extensions Nonlinear Problems Running the transient convection diffusion program Running the program # $ $ $ copy the example cp -r $LIBMESH_TUTORIAL/transient_convection_diffusion . UT) The libMesh Finite Element Library June 17. INL. Stogner (NASA. 2013 146 / 215 .

INL.Some Extensions Kirk. UT) Nonlinear Problems The libMesh Finite Element Library June 17. Peterson. Stogner (NASA. 2013 147 / 215 .

UT) The libMesh Finite Element Library June 17./example-opt Kirk./example-opt $ mpirun -np 2 . Peterson.Some Extensions Nonlinear Problems Running the navier stokes program Running the program # $ $ $ copy the example cp -r $LIBMESH_TUTORIAL/navier_stokes . 2013 148 / 215 . cd navier_stokes make # run the example $ . INL. Stogner (NASA.

Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17. 2013 149 / 215 . Peterson.Some Extensions Nonlinear Problems Output Kirk.

UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. INL. 2013 150 / 215 . Peterson. Stogner (NASA.

2013 151 / 215 .Adaptive Mesh Refinement Adaptive Mesh Refinement Kirk. INL. Peterson. UT) The libMesh Finite Element Library June 17. Stogner (NASA.

Peterson. Kirk. Stogner (NASA.Adaptive Mesh Refinement Model Problem • Consider the 1D model ODE   −u00 + bu0 + cu = f u(0) = u0  u(L) = uL ∈ Ω = (0. INL. UT) The libMesh Finite Element Library June 17. L) (1) • with weak form Z 0 0 0  Z u v + bu v + cuv dx = Ω (2) fv dx Ω for every v ∈ H01 (Ω). 2013 152 / 215 .

2013 153 / 215 . UT) fv dx (4) Ω The libMesh Finite Element Library June 17. Peterson.Adaptive Mesh Refinement Model Problem (cont.) • The analogous d-dimensional problem with Ω ⊂ Rd and boundary ∂Ω is  −∆u + b · ∇u + cu = f u = g ∈ Ω ∈ ∂Ω (3) • with weak form Z Z (∇u · ∇v + (b · ∇u)v + cuv) dx = Ω Kirk. Stogner (NASA. INL.

and summing the contributions of the element integrals Ne Z X e=1  ∇uh · ∇vh + (b · ∇uh )vh + cuh vh − fvh dx = 0 (5) Ωe • Remark: We considered here a standard piecewise continuous finite element basis. INL.) • The finite element method works with the weak form. vh . Stogner (NASA. 2013 154 / 215 . In general. UT) The libMesh Finite Element Library June 17. ∇uh will have a jump discontinuity across element boundaries. Peterson. replacing the trial and test functions u. Kirk.Adaptive Mesh Refinement Model Problem (cont. v with their approximations uh .

UT) The libMesh Finite Element Library (8) June 17. Peterson. 2013 155 / 215 . N . Ke U = Fe Kirk. . INL. .Adaptive Mesh Refinement Galerkin FE Method • Expressing uh and vh in our chosen piecewise continuous polynomial basis uh = N X vh = u j ϕj j=1 N X ci ϕi (6) i=1 we obtain on each element Ωe N X Z Z (∇ϕj · ∇ϕi + (b · ∇ϕj )ϕi + cϕj ϕi ) dx = uj j=1  Ωe f ϕi dx (7) Ωe for i = 1 . Stogner (NASA. • In the standard element-stiffness matrix form.

Stogner (NASA. the user must provide the routine which computes Ke and Fe for each element. q = 1 . INL.Adaptive Mesh Refinement LibMesh Representation • To code the model problem in LibMesh. . UT) ˆe Ω f ϕi |J|dξ ≈ Nq X wq |J(ξ q )|f (ξ q )ϕi (ξ q ) (9) q=1 The libMesh Finite Element Library June 17. 2013 156 / 215 . . Nq . Z Z f ϕi dx = Ωe Kirk. Peterson. • The integrals are computed over the reference element using an appropriate numerical quadrature rule with weights wq and points ξ q .

Peterson. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement LibMesh Representation (cont. this is ∂φ ∂ξ ∂x (ξq )) Kirk. Stogner (NASA.g. in 1D. 2013 157 / 215 .) • LibMesh provides the following variables for constructing Ke and Fe at quadrature point q: • JxW[q] = the scalar value of the element Jacobian map times the quadrature rule weight • phi[i][q] = ϕi (ξ q ) i ∂ξ • dphi[i][q] = (J −1 · ∇ξ ϕi )(ξ q ) (e. INL.

Stogner (NASA.) for (q=0.Adaptive Mesh Refinement LibMesh Representation (cont. c. ++j) Ke(i. f at this quadrature point // . for (j=0. q<Nq. ++q) { // Compute b. for (i=0.. 2013 158 / 215 .. j<N. INL.j) += JxW[q]*( (dphi[i][q]*dphi[j][q]) + (b*dphi[j][q])*phi[i][q] + c*phi[j][q]*phi[i][q] ). i<N. } } Kirk. ++i) { Fe(i) += JxW[q]*f*phi[i][q]. Peterson. UT) The libMesh Finite Element Library June 17.

f3 0 k32 k33 . Peterson. Stogner (NASA. .  . Kirk. The libMesh Finite Element Library June 17.Adaptive Mesh Refinement Boundary Conditions • Dirichlet boundary conditions are typically enforced after the global stiffness matrix K has been assembled • This usually involves 1 placing a “1” on the main diagonal of the global stiffness matrix 2 zeroing out the row entries 3 placing the Dirichlet value in the rhs vector 4 subtracting off the column entries from the rhs  k11 k12 k13 k21 k22 k23  k31 k32 k33 .  . . . . INL. UT)        . f2 − k21 g1        . 2013 159 / 215 . 0 . f1 1 0 0 0 g1       . . . f3 − k31 g1  . f2  → 0 k22 k23 .

it is inefficient to change individual entries once the global matrix is assembled. INL. Peterson. Stogner (NASA. Kirk. penalty boundary conditions were the preferred approach in libMesh.) • This approach works for an interpolary finite element basis but not in the general case. • A Solution: “Penalty” Boundary Conditions • For many years. • For large problems with parallel sparse matrices. so you may encounter them. • What is required is a way to enforce boundary conditions for a generic finite element basis at the element stiffness matrix level.Adaptive Mesh Refinement Boundary Conditions (cont. 2013 160 / 215 . UT) The libMesh Finite Element Library June 17.

Stogner (NASA. INL. Kirk. and works for general finite element bases. This approach only impacts elements who have a face on the boundary of the domain. 1 + 1 = 1 . in floating point arithmetic. 2013 161 / 215 . Peterson.Adaptive Mesh Refinement Penalty Boundary Conditions • An additional “penalty” term is added to the standard weak form Z (∇u · ∇v + (b · ∇u)v + cuv) dx + Ω 1  | Z Z (u − g)v dx = ∂Ω {z } fv dx Ω penalty term • Here   1 is chosen so that. UT) The libMesh Finite Element Library June 17. • This weakly enforces u = g on the boundary at the element level.

Peterson. j<Nf. i++) { Fe(i) += JxW_f[qf]*penalty*g*phi_face[i][qf]. j++) Ke(i. qf<Nqf.Adaptive Mesh Refinement LibMesh Representation LibMesh provides: • A quadrature rule with Nqf points and JxW f[] • A finite element coincident with the boundary face that has Nf shape function values phi f[][] for (qf=0. Stogner (NASA. for (j=0.j) += JxW_f[qf]*penalty*phi_f[i][qf]*phi_f[j][qf]. } } Kirk. 2013 162 / 215 . i<Nf. UT) The libMesh Finite Element Library June 17. INL. ++qf) { // Compute g at this face quadrature point for (i=0.

we’ll discuss • “Natural” refinement patterns • A flux-jump error indicator Kirk. 2013 163 / 215 . Stogner (NASA. Peterson. physics-independent error indicators • In this context. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement Adaptivity And Error Indicators • A major goal of the LibMesh library is to provide: • Adaptive mesh refinement support for standard geometric elements • Generic. INL.

Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL.Adaptive Mesh Refinement Natural Refinement Patterns Triangle Quadrilateral Tetrahedron Prism Kirk. 2013 164 / 215 . Peterson.

INL.Adaptive Mesh Refinement Flux-Jump Error Indicator • The flux-jump error indicator is derived starting from the element integrals Ne Z X e=1  ∇uh · ∇vh + (b · ∇uh )vh + cuh vh − fvh dx = 0 (5) Ωe • Applying the divergence theorem “in reverse” obtains Ne Z X e=1  −∆uh + (b · ∇uh ) + cuh − f vh dx + (10) Ωe X Z ∂Ωe 6⊂∂Ω ∂Ωe Kirk. UT) s { ∂uh h v dx = 0 ∂n The libMesh Finite Element Library June 17. 2013 165 / 215 . Stogner (NASA. Peterson.

INL. Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement Flux-Jump Error Indicator (cont. knowledge of the “physics”). • The second sum leads to a physics-independent method for estimating the error in the approximate solution uh .) • Defining the cell residual r(uh ) = −∆uh + (b · ∇uh ) + cuh − f (11) we have Ne Z X e=1 h h r(u )v dx + Ωe X Z ∂Ωe 6⊂∂Ω ∂Ωe s { ∂uh h v dx = 0 ∂n (12) • Clearly. Kirk.e. 2013 166 / 215 . the exact solution u satisfies (12) identically. • Computing r(uh ) requires knowledge of the differential operator (i.

etc. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement Flux-Jump Error Indicator (cont. the interior residual term may dominate • Relatively expensive to compute • Makes no sense for discontinuous and C1 FE bases Kirk.) • Pros • Ideal for low-order (piecewise linear) elements • Easily extensible to adaptivity with hanging nodes • Works well in practice for nonlinear. Peterson. layers. Stogner (NASA. time-dependent problems. 2013 167 / 215 . • Cons • For higher-order elements. and problems with shocks. discontinuities. INL.

the error indicator η for a particular element Ωe = (xe . UT) The libMesh Finite Element Library June 17. Stogner (NASA. Kirk. xe+1 ) is defined as η2 = Nint he X Ju0 (yi )K2 Nint (13) i=1 where he = xe+1 − xe is the element length.Adaptive Mesh Refinement 1D Example • In 1 dimension. 2013 168 / 215 . Peterson. • For linear elements. the jump integrals reduce to point-wise evaluation of the derivatives at the element boundaries. INL. and Nint ≤ 2 is the number of interior nodes yi the element has.

5 0.2 0.7 June 17. Kirk.4 0. and compute the error indicator for a sequence of uniformly refined grids. Peterson.9 0.4 0.7 0.8 which is a solution of the classic 1D advection-diffusion boundary layer equation.6 0.2 • We assume here that the finite element solution is the linear interpolant of u.Adaptive Mesh Refinement 1D Example (cont. UT) The libMesh Finite Element Library 0.9 1 169 / 215 .1 0. Stogner (NASA.) • Consider the function u= 1 − exp(10x) 1 − exp(10) 1 0.1 0 0 0. INL.8 0.3 0. 0. 2013 0.3 0.5 0.6 0.

5 6 0.8 0.1 0 0 0.4 0. 2013 170 / 215 .4 0.8 ||e||L2 = 0.4 0. Peterson.6 0.8 1 x x 4 elements 1.6 0.2 1 0.2 0 1 2 3 Element Number Kirk.9 12 uexact u’exact u 10 0.6 0.Adaptive Mesh Refinement 1 0.4 4 0.8 0.8 1 0 0 0.2 0.6 1.3 0. Stogner (NASA.09 η 1.6 0. UT) 4 The libMesh Finite Element Library June 17.2 2 0.4 1.7 8 0. INL.2 0.

6 0.8 1 0 0 0. 2013 170 / 215 .1 0 0 0.4 0.027 η 1 0.3 0.6 0.8 0. Stogner (NASA.8 1 x x 8 elements 1.5 ||e||L2 = 0.5 0 1 2 3 4 5 Element Number 6 Kirk. Peterson. INL. UT) 7 8 The libMesh Finite Element Library June 17.2 0.2 0.7 8 0.2 2 0.4 0.5 6 0.Adaptive Mesh Refinement 1 12 uexact u’exact u 0.4 4 0.9 10 0.6 0.

5 0. 2013 170 / 215 .3 0.4 0.7 8 0.6 0.4 0.Adaptive Mesh Refinement 1 12 uexact u’exact u 0.8 0. UT) The libMesh Finite Element Library June 17.4 0.2 0.1 0 0 0.3 0.2 2 0.4 4 0.5 6 0.1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Element Number Kirk.0071 η 0.2 0.7 0.2 0.9 ||e||L2 = 0. INL.6 0.9 10 0. Stogner (NASA.6 0.8 1 0 0 0.8 0.6 0.8 1 x x 16 elements 0. Peterson.

2013 171 / 215 . Stogner (NASA. INL. while (r < r_max) Compute the FE solution (linear interpolant) Estimate the error (using flux-jump indicator) Refine the elements with error in top 10% Increment r end Kirk. Peterson. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement A Simple Refinement Strategy • A simple adaptive refinement strategy with r max refinement steps for this 1D example problem is: r=0.

INL.8 0.8 1 x Kirk.4 0.6 0.3 0.Adaptive Mesh Refinement 1 u exact 0. 2013 172 / 215 .2 0. Peterson.1 0 0 0.7 0. UT) The libMesh Finite Element Library June 17.9 u 0.5 0.6 0.2 0.4 0. Stogner (NASA.

6 0. Peterson.6 0.4 0.5 0.8 0.7 0. UT) The libMesh Finite Element Library June 17.1 0 0 0.2 0.9 u 0. 2013 172 / 215 . INL.8 1 x Kirk. Stogner (NASA.3 0.Adaptive Mesh Refinement 1 u exact 0.2 0.4 0.

1 0 0 0.4 0.9 u 0.Adaptive Mesh Refinement 1 u exact 0.7 0. Stogner (NASA.5 0.6 0.2 0.2 0. INL. 2013 172 / 215 .8 0. Peterson.3 0.6 0.8 1 x Kirk. UT) The libMesh Finite Element Library June 17.4 0.

2 0.1 0 0 0. 2013 172 / 215 . UT) The libMesh Finite Element Library June 17.8 0.6 0.9 u 0.5 0.6 0. Peterson.7 0. INL.Adaptive Mesh Refinement 1 u exact 0.3 0.4 0.4 0. Stogner (NASA.2 0.8 1 x Kirk.

Peterson. INL.4 0.5 0.2 0.7 0.4 0.Adaptive Mesh Refinement 1 u exact 0.8 1 x Kirk. 2013 172 / 215 .6 0.1 0 0 0.3 0. Stogner (NASA.6 0.8 0.9 u 0.2 0. UT) The libMesh Finite Element Library June 17.

INL.7 0.2 0.1 0 0 0.2 0.8 0.4 0.4 0. Stogner (NASA.6 0.9 u 0. Peterson. 2013 172 / 215 .5 0.6 0.3 0.8 1 x Kirk. UT) The libMesh Finite Element Library June 17.Adaptive Mesh Refinement 1 u exact 0.

UT) The libMesh Finite Element Library June 17. Peterson.5 0.8 0.6 0.2 0.9 u 0.6 0.Adaptive Mesh Refinement 1 u exact 0.4 0.2 0. Stogner (NASA.4 0.1 0 0 0. 2013 172 / 215 .8 1 x Kirk.7 0.3 0. INL.

2 0.8 1 x Kirk.2 0.3 0.8 0.5 0.Adaptive Mesh Refinement 1 u exact 0.9 u 0. INL.6 0. 2013 172 / 215 .4 0.1 0 0 0.7 0. Stogner (NASA.4 0.6 0. UT) The libMesh Finite Element Library June 17. Peterson.

3 0.6 0. 2013 172 / 215 .Adaptive Mesh Refinement 1 u exact 0. UT) The libMesh Finite Element Library June 17.1 0 0 0. Peterson.5 0.4 0.8 1 x Kirk.7 0.4 0.9 u 0. INL.6 0.2 0. Stogner (NASA.8 0.2 0.

6 0. Peterson.2 0. 2013 172 / 215 .7 0.Adaptive Mesh Refinement 1 u exact 0.8 0. INL.1 0 0 0.4 0.2 0. Stogner (NASA. UT) The libMesh Finite Element Library June 17.8 1 x Kirk.6 0.9 u 0.4 0.5 0.3 0.

Adaptive Mesh Refinement 1 u exact 0.4 0.6 0.2 0.1 0 0 0.8 1 x Kirk.3 0.9 u 0.2 0. Peterson. 2013 172 / 215 .6 0.4 0. Stogner (NASA.7 0. INL.8 0.5 0. UT) The libMesh Finite Element Library June 17.

5 −2 −2.5 log10 ||e|| −1 −1. 2013 173 / 215 .) 0 Uniform Adaptive −0. UT) 0.5 log10 Nnodes 1 The libMesh Finite Element Library 1.5 June 17. INL. Peterson.5 −3 0 Kirk. Stogner (NASA.Adaptive Mesh Refinement A Simple Refinement Strategy (cont.

INL. Stogner (NASA. 2013 174 / 215 .Adaptive Mesh Refinement Examples: Transient Problem with AMR transient convection diffusion AMR Kirk. Peterson. UT) The libMesh Finite Element Library June 17.

// Print out the H1 norm..solution.. UT) The libMesh Finite Element Library June 17. Stogner (NASA. r_step++) { // Assemble & solve the linear system system. // Possibly refine the mesh if (r_step+1 != max_r_steps) { std::cout << " Refining the mesh. ErrorVector error. Peterson. r_step<max_r_steps." << std::endl. for (unsigned int r_step=0.solve().. std::cout << "H1 norm = " << H1norm << std::endl. SystemNorm(H1))..Adaptive Mesh Refinement The transient convection diffusion AMR program I MeshRefinement mesh_refinement (mesh). // The ErrorVector is a particular StatisticsVector // for computing error information on a finite element mesh. .calculate_norm(*system. 2013 175 / 215 . INL. Kirk. for verification purposes: Real H1norm = system.

and any element within 7% of the minimum error on any element might be coarsened. error). Note that the elements flagged for refinement will be refined. Stogner (NASA.Adaptive Mesh Refinement The transient convection diffusion AMR program II // The ErrorEstimator class interrogates a finite element // solution and assigns to each element a positive error value. KellyErrorEstimator error_estimator. error_estimator. 2013 176 / 215 . Peterson. Note in general you will need to // provide an error estimator specifically designed for your // application. // // // // // // // This takes the error in error and decides which elements will be coarsened or refined. but those flagged for coarsening _might_ be coarsened. Kirk.estimate_error (system. Any element within 20% of the maximum error on any element will be refined. UT) The libMesh Finite Element Library June 17. // This value is used for deciding which elements to refine // and which to coarsen. // Compute the error for each active element using the provided // flux_jump indicator. INL.

equation_systems. // This call reinitializes the EquationSystems object for // the newly refined mesh.max_h_level() = 5. UT) The libMesh Finite Element Library June 17. mesh_refinement.refine_and_coarsen_elements().80. 2013 177 / 215 . // old_solution.flag_elements_by_error_fraction (error). mesh_refinement.reinit (). Peterson. One of the steps in the // reinitialization is projecting the solution. mesh_refinement.coarsen_fraction() = 0.refine_fraction() = 0. Stogner (NASA. etc.Adaptive Mesh Refinement The transient convection diffusion AMR program III mesh_refinement. vectors from the old mesh to // the current one.07. } } Kirk. mesh_refinement. INL...

UT) The libMesh Finite Element Library June 17. Peterson. INL. 2013 178 / 215 ./example-opt -read_solution -n_timesteps 25 \ -output_freq 10 -init_timestep 25 Kirk./example-opt -n_timesteps 25 -n_refinements 5 \ -output_freq 10 -init_timestep 0 # restart the example.Adaptive Mesh Refinement Running the transient convection diffusion AMR program Running the program # $ $ $ copy the example cp -r $LIBMESH_TUTORIAL/transient_convection_diffusion_AMR . cd transient_convection_diffusion_AMR make # run the example for 25 timesteps from an initially refined mesh $ . Stogner (NASA. reading the refined mesh and solution $ .

Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17. Peterson. 2013 179 / 215 .Adaptive Mesh Refinement Output Kirk.

INL. Peterson. 2013 180 / 215 .Parallelism on Adaptive Unstructured Meshes 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA.

2013 181 / 215 . UT) The libMesh Finite Element Library June 17.Parallelism on Adaptive Unstructured Meshes Parallel Data Structures in libMesh ParallelMesh Kirk. INL. Peterson. Stogner (NASA.

Stogner (NASA. Peterson.Parallelism on Adaptive Unstructured Meshes Finite Element Computation FEM Computational Costs • Discrete system solves • Discrete Jacobian assembly • Discrete residual assembly • Sparse Jacobian allocation • I/O • Mesh generation • Mesh movement Kirk. 2013 182 / 215 . UT) The libMesh Finite Element Library June 17. INL.

UT) The libMesh Finite Element Library June 17. INL. Peterson. 2013 183 / 215 . Stogner (NASA.Parallelism on Adaptive Unstructured Meshes Finite Element Computation Adaptive FEM Costs • Error estimator evaluation • Adaptive refinement/coarsening • Inter-mesh projections • Adaptivity flagging • Adaptive constraint calculations Kirk.

2013 184 / 215 . UT) The libMesh Finite Element Library June 17. Stogner (NASA.Parallelism on Adaptive Unstructured Meshes Parallelism Parallelism Goals Reduced CPU time • Distributed CPU usage • Asynchronous I/O Reduced memory requirements • Larger attainable problem size Kirk. Peterson. INL.

Stogner (NASA. INL. MatVec operations Shared-memory Threads • Mesh Partitioning Kirk.Parallelism on Adaptive Unstructured Meshes Parallelism Parallelism Levels SIMD Instructions • Assemblies. UT) The libMesh Finite Element Library June 17. 2013 185 / 215 . Peterson.

INL. 2013 186 / 215 . Peterson. UT) The libMesh Finite Element Library June 17.Parallelism on Adaptive Unstructured Meshes Parallelism Parallelism Levels Separate Simulations • Parametric studies • Uncertainty analysis Distributed-memory Processes • Asynchronous I/O • Mesh Partitioning Kirk. Stogner (NASA.

Parallelism on Adaptive Unstructured Meshes ParallelMesh Mesh Classes MeshBase +boundary_info: AutoPtr<BoundaryInfo> +elem(unsigned int) +node(unsigned int) +elements_begin/end() +nodes_begin/end() +active_local_elements_begin/end() +read() +write() CartesianMesh • Abstract iterator interface hides mesh type from most applications • UnstructuredMesh ”branch” for most library code UnstructuredMesh +find_neighbors() +read/write() +add/delete_elem() SerialMesh • ParallelMesh implements data storage. Peterson. 2013 187 / 215 . synchronization ParallelMesh #_elements: vector<Elem*> #_elements: mapvector<Elem*> +elements_begin() +allgather() +delete_remote_elements() Kirk. UT) The libMesh Finite Element Library June 17. INL. Stogner (NASA.

2013 188 / 215 . Stogner (NASA. UT) The libMesh Finite Element Library June 17. Peterson.Parallelism on Adaptive Unstructured Meshes ParallelMesh SerialMesh Partitioning • Each element. INL. node is “local” to one processor • Each processor has an identical Mesh copy • Mesh stays in sync through redundant work • FEM data synced on “ghost” elements only Kirk.

UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson. 2013 189 / 215 . INL.Parallelism on Adaptive Unstructured Meshes ParallelMesh ParallelMesh Partitioning • Processors store only local and ghost objects • Each processor has a small Mesh subset • Mesh stays in sync through MPI communication Kirk.

Peterson.Parallelism on Adaptive Unstructured Meshes ParallelMesh ParallelMesh Partitioning Pros • Reduced memory use • O(NE /NP ) CPU costs Kirk. 2013 190 / 215 . INL. UT) The libMesh Finite Element Library June 17. Stogner (NASA.

Peterson. 2013 191 / 215 . INL. Stogner (NASA.Parallelism on Adaptive Unstructured Meshes ParallelMesh ParallelMesh Partitioning Cons • Increased code complexity • Increased synchronization “bookkeeping” • Greater debugging difficulty Kirk. UT) The libMesh Finite Element Library June 17.

Modification. Generation. UT) The libMesh Finite Element Library June 17. reconstruct non-semilocal objects • Parallelized DofMap methods • Parallelized MeshRefinement methods • Parallel Mesh I/O support • Load balancing support Also working on parallel support in Boundary. Function. Generation. Peterson. INL.Parallelism on Adaptive Unstructured Meshes ParallelMesh Gradual Parallelization Starting from SerialMesh behavior • New internal data structure • Methods to delete. Stogner (NASA. Tools classes Kirk. 2013 192 / 215 .

Peterson. Stogner (NASA. UT) The libMesh Finite Element Library June 17. INL. jump error estimators • Refinement-based estimators • Adjoint-based estimators • Recovery estimators Kirk. 2013 193 / 215 .Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Error Estimation • Local residual.

Stogner (NASA. INL. Peterson. UT) The libMesh Finite Element Library June 17. 2013 193 / 215 .Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Error Estimation • Local residual. jump error estimators: embarrassingly parallel • Refinement-based estimators: use solver parallelism • Adjoint-based estimators: use solver parallelism • Recovery estimators: require partitioning-aware patch generation Kirk.

UT) The libMesh Finite Element Library June 17. 2013 193 / 215 . Stogner (NASA. Peterson.Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Error Estimation • Local residual. INL. jump error estimators: embarrassingly parallel • Refinement-based estimators: use solver parallelism • Adjoint-based estimators: use solver parallelism • Recovery estimators: require partitioning-aware patch generation Refinement Flagging 2 /N • Flagging by error tolerance: ηK2 < ηtol E • Flagging by error fraction: ηK < r maxK ηK • Flagging by element fraction or target NE Kirk.

Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Error Estimation • Local residual. INL. Peterson. jump error estimators: embarrassingly parallel • Refinement-based estimators: use solver parallelism • Adjoint-based estimators: use solver parallelism • Recovery estimators: require partitioning-aware patch generation Refinement Flagging 2 /N • Flagging by error tolerance: ηK2 < ηtol E • Embarrassingly parallel • Flagging by error fraction: ηK < r maxK ηK • One global Parallel::max to find maximum error • Flagging by element fraction or target NE • Parallel::Sort to find percentile levels? • Binary search in parallel? • TBD Kirk. UT) The libMesh Finite Element Library June 17. Stogner (NASA. 2013 193 / 215 .

Stogner (NASA. Node creation Kirk.Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Elem. UT) The libMesh Finite Element Library June 17. 2013 194 / 215 . Peterson. INL.

INL. Stogner (NASA.Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Elem. Peterson. UT) The libMesh Finite Element Library June 17. Node creation • Ids {i : i mod (NP + 1) = p} are owned by processor p Kirk. 2013 194 / 215 .

INL. Node creation • Ids {i : i mod (NP + 1) = p} are owned by processor p Synchronization • Refinement Flags • New ghost child elements. UT) The libMesh Finite Element Library June 17. Stogner (NASA. nodes • Hanging node constraint equations Kirk.Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Elem. 2013 194 / 215 . Peterson.

Stogner (NASA. Node creation • Ids {i : i mod (NP + 1) = p} are owned by processor p Synchronization • Refinement Flags • Data requested by id • Iteratively to enforce smoothing • New ghost child elements..Parallelism on Adaptive Unstructured Meshes ParallelMesh Distributed Mesh Refinement Elem. UT) The libMesh Finite Element Library June 17. Peterson. nodes • Id requested by data • Hanging node constraint equations • Iteratively through subconstraints. Kirk. 2013 194 / 215 . INL. subconstraints-of-subconstraints..

INL. Bogner-Fox-Schmidt quads or hexes 2D SerialMesh vs.Parallelism on Adaptive Unstructured Meshes Performance “Typical” PDE example Transient Cahn-Hilliard. Peterson. 2013 195 / 215 . distributed sparse matrix costs are unchanged • Serialized mesh. indexing once dominated RAM use Kirk. Stogner (NASA. ParallelMesh Per-Node Memory Usage 104 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 2D 103 RAM (MB) 104 102 101 3 10 1 Node Serial 1 Node Parallel 2 Node Serial 2 Node Parallel 4 Node Serial 4 Node Parallel 8 Node Serial 8 Node Parallel 16 Node Serial 16 Node Parallel 32 Node Serial 32 Node Parallel 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 102 4 10 5 6 10 10 Mesh Size (quad elements) 7 10 101 3 10 4 5 10 10 Mesh Size (hex elements) 6 10 Results • Parallel codes using SerialMesh are unchanged for ParallelMesh • Overhead. UT) The libMesh Finite Element Library June 17. ParallelMesh Per-Node Memory Usage RAM (MB) 103 1 Node Serial 1 Node Parallel 2 Node Serial 2 Node Parallel 4 Node Serial 4 Node Parallel 8 Node Serial 8 Node Parallel 16 Node Serial 16 Node Parallel 32 Node Serial 32 Node Parallel 3D SerialMesh vs.

send. more legible Example: std::vector<Real> send. . Kirk. INL. recv.. 2013 196 / 215 .. recv). source_processor_id.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel:: API Encapsulating MPI • Improvement over MPI C++ interface • Makes code shorter. Stogner (NASA. send_receive(dest_processor_id. UT) The libMesh Finite Element Library June 17. Peterson.

size(). Stogner (NASA. recvsize. MPI_Sendrecv(&sendsize. recvsize.resize(recvsize). source_processor_id. MPI_Sendrecv(sendsize ? &send[0] : NULL. MPI_DOUBLE. dest_processor_id. 0. 2013 197 / 215 . 0. 0. &status). recv. MPI_DOUBLE. #ifdef HAVE_MPI else { unsigned int sendsize = send. INL. Peterson. datatype<unsigned int>(). 1. MPI_Status status. 0. sendsize. datatype<unsigned int>(). libMesh::COMM_WORLD. libMesh::COMM_WORLD. &recvsize. source_processor_id. dest_processor_id. 1. UT) The libMesh Finite Element Library June 17.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel:: API Instead of: if (dest_processor_id == libMesh::processor_id() && source_processor_id == libMesh::processor_id()) recv = send. &status). } #endif // HAVE_MPI Kirk. recvsize ? &recv[0] : NULL.

UT) The libMesh Finite Element Library June 17. 2013 198 / 215 . Stogner (NASA. INL.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms ParallelMesh Data Structure std::vector fails • Not sparse • O(NE ) storage cost std::map • “mapvector” interface provides iterators • O(log(NE /NP )) lookup time without std::hash map • O(1) lookup time with std::hash map Hybrid data structure? • Dense vector for most elements • Sparse data structure for new elements Kirk. Peterson.

2013 199 / 215 .Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Round Robin Communications • Processor P sends to processor P + K while receiving from P − K • New data is operated on and old data discarded • K is incremented “round robin” from 1 to NP − 1 Kirk. Peterson. UT) The libMesh Finite Element Library June 17. Stogner (NASA. INL.

Stogner (NASA. Peterson. 2013 200 / 215 .only one data exchange at a time • Straightforward to code • Reliable Kirk.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Round Robin Communications Pros • O(NG /NP ) memory usage . INL. UT) The libMesh Finite Element Library June 17.

synchronous communications Kirk. 2013 201 / 215 .Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Round Robin Communications Cons • Communications loop over non-neighboring processors • O(NG ) execution time • Multiple. UT) The libMesh Finite Element Library June 17. INL. Stogner (NASA. Peterson.

DoFs • Synchronization of ghost objects • Load balancing. INL.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Adaptivity and ParallelMesh Challenges • Reindexing elements. nodes. 2013 202 / 215 . Repartitioning Kirk. Stogner (NASA. Peterson. UT) The libMesh Finite Element Library June 17.

Stogner (NASA. INL. UT) The libMesh Finite Element Library June 17. 2013 203 / 215 . Peterson.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel Global Indexing One-pass indexing • Index processor P from • Pass PP 1 PP−1 1 NEp NEp to processor P + 1 • O(NE ) work • O(NE ) execution time Kirk.

UT) The libMesh Finite Element Library June 17. 2013 204 / 215 . INL. Peterson.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel Global Indexing Two-pass indexing • Count processor P indices from 0 • Gather NEp on all processors • Re-index processor P from PP−1 1 NEp • Double the work • O(NE /NP ) execution time Kirk. Stogner (NASA.

Peterson. nodes Kirk. nodes • New degrees of freedom • Hanging node constraint equations • Repartitioned elements. INL. Stogner (NASA.Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel Synchronization What can lose sync? • Refinement flags • New child elements. 2013 205 / 215 . UT) The libMesh Finite Element Library June 17.

Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Parallel Synchronization Round-Robin Complications • Refinement flags must obey consistency rules • New ghost nodes may have unknown processor ids • Constraint equations may be recursive • Hanging node constraint equations • Repartitioned elements. Stogner (NASA. 2013 206 / 215 . Peterson. INL. nodes Kirk. UT) The libMesh Finite Element Library June 17.

UT) The libMesh Finite Element Library June 17. Stogner (NASA. Peterson. INL. 2013 207 / 215 .Parallelism on Adaptive Unstructured Meshes Parallel Algorithms Debugging • Regression tests • Precondition. postcondition tests • Unit testing • Parallel debuggers • Low NP test cases Kirk.

ease of programming/debugging • Latency vs. UT) The libMesh Finite Element Library June 17. bad assumptions and guidelines • Reuse existing code/algorithms • Build incrementally • Test extensively Kirk. 2013 208 / 215 . INL.Parallelism on Adaptive Unstructured Meshes Parallelism Summary Summary Parallelism tradeoffs • Efficiency vs. redundant work • “Premature optimization” mistakes vs. Peterson. Stogner (NASA.

Stogner (NASA. 2013 209 / 215 . INL. UT) The libMesh Finite Element Library June 17. Peterson.Going Further 1 Introduction 2 Motivation 3 Software Installation & Ecosystem 4 A Generic Boundary Value Problem 5 Key Data Structures 6 Weighted Residuals 7 Poisson Equation 8 Other Examples 9 Some Extensions 10 Adaptive Mesh Refinement 11 Parallelism on Adaptive Unstructured Meshes 12 Going Further Kirk.

• gmsh: open-source mesh generator. • Recommended mesh generation packages: • gridgen/pointwise: export mesh in ExodusII format.net Kirk. Peterson. 2013 210 / 215 . assigning boundary conditions. Stogner (NASA. • Need another mesh format? libmesh-users@lists. UT) The libMesh Finite Element Library June 17.Going Further Mesh Generation & Importing Mesh Generation • You will need access to a complete mesh generation package to create complex meshes for use inside libMesh. and exporting the mesh.sourceforge. creating a volume mesh. INL. • Cubit: unstructured mesh generator from Sandia National Labs. • General process involves importing geometry. libMesh supports gmsh format.

2013 211 / 215 . INL. see $LIBMESH DIR/examples/miscellaneous/ex5 Kirk. Peterson. • For an example using interior penalty DG.Going Further Discontinuous Galerkin FEM Discontinuous Galerkin Support • libMesh supports a rich set of discontinous finite element bases. UT) The libMesh Finite Element Library June 17. Stogner (NASA.

Going Further Multiphysics Applications Multiphysics Support • Tightly coupled multiphysics: • All variables should be placed in the same System. Stogner (NASA.f. Peterson. Kirk. • You can restrict variables to subdomains: c. UT) The libMesh Finite Element Library June 17. use different Systems • On disjoint Meshes. 2013 212 / 215 . $LIBMESH DIR/examples/subdomains/ex1 • Loosely coupled multiphysics: • On the same Mesh. INL. pass data via MeshfreeInterpolation objects.

• See $LIBMESH DIR/examples/fem system Kirk. UT) The libMesh Finite Element Library June 17. • Allows more direct interaction between solution algorithms and finite element approximation. Stogner (NASA. 2013 213 / 215 .Going Further FEMSystem Framework FEMSystem • FEMSystem provides an alternative programming interface specifically for finite element applications. Peterson. INL.

INL. • This functionality allows fine-scale solutions to be approximated in an optimal basis with an associated error estimate. • See $LIBMESH DIR/examples/reduced basis Kirk. Peterson.Going Further Reduced Basis Modeling Reduced Basis Modeling • Recent effort led by David Knezevic to add certified reduced basis support to libMesh. Stogner (NASA. 2013 214 / 215 . UT) The libMesh Finite Element Library June 17.

• Public site. 3–4. “libMesh: a C++ library for parallel adaptive mesh refinement/coarsening simulations”. 2006. Peterson. R.net/ Kirk. Carey. examples. 22. vol. INL. p. mailing lists.sf. Peterson. 2013 215 / 215 . etc.: http://libmesh.Reference • B. Kirk. UT) The libMesh Finite Element Library June 17. Stogner and G. 237–254. no. Engineering with Computers. Stogner (NASA. J. SVN tree.