Computer methods

in applied
mechanics and
engineering
ELSEVIER Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417
www.elsevier.com/Iocate/cma

Multiphysics simulation of flow-induced vibrations and

aeroelasticity on parallel computing platforms
S t e v e n M. R i f a i a'*, Z d e n ~ k J o h a n a, W e n - P i n g W a n g a, J e a n - P i e r r e G r i s v a l b,
T h o m a s J.R. H u g h e s a, R o b e r t M. F e r e n c z ~'
~'Centric Engineering Systems, Inc., 624 East Evelyn Avenue, Sunnyvale, CA 94086, USA
~OJfice National d'Etudes et de Recherches Aerospatiales', BP 72, 92322 Chatillon, France

Received 27 January 1998: revised 20 March 1998

Abstract

This article describes the application of multiphysics simulation on parallel computing platforms to model aeroelastic instabilities and
flow-induced vibrations. Multiphysics simulation is based on a single computational framework for the modeling of multiple interacting
physical phenomena. Within the multiphysics framework, the finite element treatment of fluids is based on the Galerkin-Least-Squares
(GLS) method with discontinuity capturing operators. The arbitrary-Lagrangian-Eulerian (ALE) method is utilized to account for
deformable fluid domains. The finite element treatment of solids and structures is based on a three-field variational principle. Fully-coupled
interaction constraints are enforced using the augmented-Lagrangian method. The multiphysics architecture lends itself naturally to
high-pefl\)rmance parallel computing. Several applications are presented. The importance of capturing the nonlinear eftects and accounting
fur mesh-movement is highlighted and the scalability of the software is illustrated. © 1999 Elsevier Science S.A. All rights reserved.

1. Introduction

Many of the current directions in product design and analysis are driven by competitive and regulatory
constraints, such as the need to shorten design cycles, reduce cost, meet increasingly stringent government
regulations, improve quality and safety, and reduce environmental impact. These directions have increased the
need for accurate product and component simulation and pressed analysts for simulations of unprecedented scale
and complexity. For example, aeroelastic modeling, involving strong coupling of fluids and structures, is an
essential element in the design process of aircraft. In general, flow-induced vibrations are a primary concern in
the aerospace, automotive and defense industries.
For many wing configurations, flow separations and shock wave oscillations often lead to buffeting. Excessive
flutter has been responsible for numerous fighter plane crashes. In High Speed Civil Transport and Advanced
Subsonic Transport studies, vortex-induced aeroelastic oscillations and transonic buffet associated with
structural oscillations are a strong concern. These phenomena can severely impair the performance of an aircraft.
Whirl flutter limits cruise speeds of tiltrotor aircraft. Blade-vortex interactions are a major source of noise in
helicopters. Flow-induced vibrations in submarine power generation components are a significant noise source.
In advanced aeropropulsion systems, critical problems such as aerodynamics-induced vibration, turbofan/blade
flutter or stall can lead to reduced life-cycle or even premature component failure resulting in potential disaster.
High-cycle fatigue in engine fans and compressors can result from aeroelastic instabilities and limit-cycle
oscillations. Another example is the next generation fan blades for ducted propellers.

* Corresponding author.

0045-7825 / 99 / $ see front matter © 1999 Elsevier Science S.A. All rights reserved.
Pll: S0045-7825 (98)00306-5
394 S.M. Rif~li el al. / Comlmt. Mcthod.~ Appl. Mech. EH~,r~,. 174 (1999)_793-417

Current industrial modeling of aeroelasticity and flow-induced vibrations employs closed-lbrm solutions and
linear models coupling aerodynamics and modal structural analysis. Although these models are well advanced,
their utility is limited because of the complex interaction between nonlinear ttuid flow and large-deformation
structures ~n many applications. In order to address the full range of design problems, high-fidelity simulation of
the nonlinear fluid-structure interactions is necessary. The Spectrum TM software [ 1,2] described in this work is
designed specifically for mulfiphysics simulation on high-performance parallel computing platforms. Fig. 1
illustrates the application of multiphysics simulation for the high-fidelity modeling of aeroelastic behavior of the
ARW2 wing. This configuration is a test drone derived from experimental studies of aeroelasticity at NASA
Langley 131. The simulation en-lploys a tetrahedral mesh for modeling inviscid flow about the wing at Math 0.8.
The structural mesh consists of shell, beam and truss elements to model the skin, spars and stifteners of the
wing. To simulate the aeroelastic properties of the wing, transient, nonlinear, fully-coupled fluid-structure
interaction computations are carried out for each set of operational parameters (e.g. free-stream flow speed,
angle of attack, etc,). Instantaneous wing deformations and pressure contours at several stations along the wing
span are depicted in the figure.
Much research attention has been paid towards the development of aeroelastic analysis on aircraft wings using
coupled time-marching methods. Specifically, the fluid physics is solved using either the Euler or Navier Stokes
equations, typically with finile-difference or finite-volume methods. For example, Farhat and co-workers couple
an explicit finite-volume code with implicit finite element structural analysis for aeroelastic simulations [41.
Rigorous investigations into geometric conservation laws, sub-cycling and transient accuracy are presented.
Guruswamy and co-workers employ a modular architecture to accommodate different numerical schemes from
different disciplines [5,6]. Finite-difference Euler or Navier-Stokes solvers ll)r fluids, and modal or finite-
element analyses for solids are used. hee-Rausch, Batina and co-workers employ an implicit finite-w~lume
algorithm for flow and modal analysis for wing structures 17,81. Three-dimensional cases such as the A G A R D
Wing 445.6 are demonstrated. The Euler flutter characteristics showed close agreement on low-Mach number
conditions, and Navier-Stokes calculations showed improved agreement at higher speeds. Other works to
enhance the flow solvers using unstructured grids or to speed-up the turn-around time by running in parallel
mode have also been reported 18,6]. Boschitsch and Quackenbush [91 have perlknmed direct ltuid-solid
interactions on a two-dimensional cascade transonic flow for turbomachinery applications. An unstructured mesh
finite-volume Euler solver was used together with finite-element structural analysis. Edwards and Mahme
present an overview of recent aeroelasticity research [101.
A key feature of most research on aeroelasticity is that two separate computational approaches are merged to
solve a coupled problem. In our multiphysics approach, a single computational fl'amework (software architecture
and the finite element method) is used. The software architecture naturally and cleanly supports independent and
parallel computation.

(a) (b)

Fig. 1. Aeroelastic simulation of the ARW2 wing configuration: mesh discretization, deformed view of the wing structure and instantaneous
pressure contours at several stations on the wing.
 S.M. Rifai et al. / Comput.MethodsAppl. Mech. Engrg. 174 (1999) 393-417 395

2. The Multiphysics Problem Model

Multiphysics simulation is based on a single computational framework for the modeling of multiple
interacting physical phenomena [1,2]. In this model, the problem domain is decomposed into spatial regions,
each simulating a different physical discipline. By using the finite element formulation, (automatically
generated) unstructured meshes are admitted. The different disciplines interact at the shared region boundaries
through general purpose interfaces. Because of the varying discretization requirements of the different physical
phenomena, this approach is designed to allow variable mesh densities and element topologies at the region
interfaces (e.g., Fig. 1). The generality of the interface treatment permits a variety of interaction constraints to be
used independently on the mechanical, thermal and mesh field variables. The slave-master algorithm is used to
impose continuity relations between two sides of an interface.
The multiphysics architecture supports multiple physical interactions through a data model defined as a
hierarchical tree of regions and interfaces [1]. Regions of a problem are used to separate the different physics
being analyzed over the spatial domain. For example, in a fluid-structure interaction (FSI) problem, the fluid
domain is one region and the solid structure is another (Fig. 2). Interfaces are used to enforce the coupling
constraints between the different regions.
Because of the varying discretization requirements of the different physical phenomena, this approach is
designed to allow variable mesh densities and element topologies at the region interfaces. For example, in a
typical FSI problem, a highly refined tetrahedral mesh may be used to model the fluid domain with a relatively
coarse hexahedral discretization representing the solid region. These discretizations do not, in general, coincide
at the shared region boundaries.
To support various element types, each region contains multiple element sets. Each element set is uniform in
material model, element topology and element formulation. As described in Section 6 below, this decomposition
organization is not only useful for user model management, but is also utilized to leverage data-parallel, vector
and cache-sensitive hardware architectures.

3. Fluid regions

The finite element treatment of fluid regions within this framework is based on the Galerkin-Least-Squares
(GLS) method with discontinuity capturing operators [1 1-13]. Reynolds-averaged and large-eddy-simulation
models are utilized for turbulence simulation [14]. In this treatment, the compressible flow formulation makes
use of the physical entropy variables

fi,-- lul2 / 2 ) '

(1)

where the superscript c denotes compressible flow variables, T is the fluid temperature and u is the fluid velocity
vector with components u~, u 2 and u 3. The chemical potential /2 is defined as

Fluid Region Interface Solid Region

Fig. 2. Multiphysics problem domain.

396 S.M. Rifid et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) ,393-417

~t = e + p i p - Ts, (2)

with e, s, p and p being the specific internal energy, specific entropy, density and pressure, respectively.
With these variables, the fluid conservation laws are expressed in symmetric form which intrinsically
expresses the mathematical and physical stability provided by the second law of thermodynamics. In turn, the
finite element techniques employed herein inherit this fundamental stability and convergence proofs are
available [15].
Entropy variables, however, do not yield the most efficient form of the incompressible equations. In practice,
the advantages of the entropy variable formulation are not significant without the presence of the shocks and
discontinuities of compressible flows [13]. Hence, the state variables used in the incompressible flow
formulation are defined as

rip0\'
Vi= u~ , (3)

where the superscript i denotes incompressible flow variables.

With the definitions above, the Navier-Stokes equation system can be expressed as follows: A solution V is
sought for the symmetric convective-diffusive system

~()Vt -{- ~ iV,,i = (l~ijV..,),i ~- i~ src (4)

where

a,, = Uv, (5)

,,v , (6)

/~,iV, = F di,f
i , (7)

and

F~ =F~(U(V)). (8)

Here U is the vector of conservation variables, V is the vector of entropy variables, and U = U ( V ) is the
appropriate transformation of variables leading to the symmetric system. The matrices d o, A, and /~,j and the
l~src
vector are nonlinear functions of V. For convenience, the vector C is introduced such that the source vector
may be written as / ~ " - - - - C V .

3. I. The Galerkin-Least Squares formulation

The finite element weighted residual formulation is as follows: given the definition of the above vectors and
matrices, find solution vector V ~' such that for all admissible weighting functions, W h, the vector V h satisfies the
variational equation

~ ( W " . A o l )h -- W e',, " F .....

, (V " ) + W h,i" I~iiV,jh _ W h . l ~ s r C ( V h ) ) d ~
2
nelement f~
+ ~'~ ( L W h + (~Wh) • r ( L y h - F ~ ) d.Q
e=l 2e
~telem f .
e=l 2e b'VV,~: "~0V,~.

= l W h ' ( - F , cony ( V tt) + r ~~diff~Trh~,

(v)~n~dF (9)
al"
 S.M. Rifai et al. / Cornput. Methods Appl. Mech. Engrg. 174 (1999) 393 417 397

where ~(~ is the computational domain with boundary F ; at2" is the domain of element e, e = 1. . . . . neJem; and
n~j.... is the total number of elements. The vectors V t' and W h are the usual f n i t e element functions; they are
piecewise polynomials within each element domain S'Y and C o continuous over the entire domain S2. The vector
l )h is the time derivative of Vh; se is a generalized vector of local coordinates for each element domain ~2"; and
n is the outward normal to the boundary F
In Eq. (9), the first integral on the left-hand side and the integral on the right-hand side constitute the usual
Galerkin formulation as applied to the symmetric convective-diffusive system. The convective and diffusive
fluxes, F~ ..... and F ~ ~", are formulated in an integrated-by-parts form. This results in the conservation of fluxes
under all quadrature rules. In addition, the boundary integrals (right-hand side of Eq. (9)) lead to a set of natural
(or Neumann) boundary conditions.
In Eq. (9), the second integral on the left-hand side is the least-squares operator. The time dependent
quasilinear differential operator for the symmetric convective-diffusive system is

L,v" ---~oV~ + Lv" (10)

where

0 h 0 0 Vt '
(ll)

The metric ~- is the least-squares matrix and is a symmetric positive-semidefinite matrix of intrinsic time scales.
The design of ~- crucially influences the behavior of the numerical solutions; therefore, considerable care is
required here.
In Eq. (9), the third integral on the left-hand side is the discontinuity-capturing operator. The scalar
discontinuity-capturing factor ~ has dimension of reciprocal time and is a function of the residual of the
differential equation, L,V h - ~ r c . The operator satisfies three fundamental properties:
• It acts in the direction of the gradient to control oscillations.
• It is proportional to the residual L,V h - I ~rc for consistency.
• It vanishes quickly in smooth regions of the solution to ensure accuracy.
These properties are achieved through the proper choice of

3.2. The arbitrary Lagrangian-Eulerian formulation

Multiphysics problems often require the movement of the computational fluid domain in response to the
deformation of the common solid region boundaries. The arbitrary-Lagrangian-Eulerian (ALE) method is
utilized to account for the deformations in fluid domains [16,171.
ALE boundary conditions ensure that the deforming mesh conforms to both the stationary and moving
boundaries. Within the interior of the fluid domain, mesh movement is modeled by the equations of large
deformation elasticity. In effect, this model computes the position of the interior nodes as if all nearest nodal
neighbors were coupled by an elastic medium and sets the positions and velocities of the boundary nodes to
exactly match the boundary motions. Note that this model is a purely mathematical construct; it is a method for
updating the mesh in a way that has a reasonable chance of maintaining mesh integrity. Thus, terms such as
Cauchy stresses, elastic moduli, and so forth do not have the usual physical interpretation within this context.
To take mesh movement into account, the convective or Euler flux in equations (4),(6) is replaced by the
ALE convective flux. The ALE convective flux is related to the Euler flux by the equation
FALE conv ALERT
i =El --U i O (12)

where U ALE is the velocity field of the mesh. The Euler Jacobians of Eq. (4) become
-ALE ALE~
Ai = A i --Ui o (13)

where t h e / [ o and A~i are defined in Eqs. (5) and (6), respectively. This substitution is equivalent to transforming
the material derivatives of field variables using
398 S.M. Ri)~d et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7

DW ALE~
Dt -- w~ + (u i -- tl t )W i (14)

where w represents any o f the field variables.

Note the following special cases:

Eulerian Lf ~ = Lf,, u "~ = 0

(15)
F At-E = F con"
i i

ALE
Lagrangian ~Alm = ~ , _ u r i c , ,
(16)
F ~ cE Fi -- t t i U

Recently, Farhat et al. have linked (violating) geometric conservation laws for ALE displacements with weak
instabilities in the solution of fluid-structure interaction problems [4]. This observation leads to constraints on
the choice of time-integration algorithms for computing ALE displacements. Consistent with this observation,
we chose the mid-point rule for ALE velocities
ALE: ALE UdALE--d,, AIA'~ ) / A t (17)
Un+ I --lln q- \~n+ I

where d,,ALE is the A L E displacement at time step n and At is the time increment.

4. Structural regions

The finite element treatment of solid regions within this framework employs a three-field formulation based
on the H u - W a s h i z u variational principle. This method is well documented in the literature to address numerical
locking phenomena [18,19]. The kinematic description admits small and finite deformations and strains. Linear
and nonlinear material models are used for the constitutive relations with thermo-mechanical coupling.

4.1. C o n t i n u u m elements

The balance of linear momentum in a solid continuum can be expressed for the current and reference
configurations in terms of the Cauchy stress, o- and the first P i o l a - K i r c h h o f f stress, P, respectively,

d i v c r + p b m = pip
D i v P + pob,,, = p o i , (18)

where p is the mass density, b m is the body force, p is the particle velocity and the 0 subscript denotes a
quantity in the reference configuration.
The balance of angular momentum leads to the symmetry requirement on the Cauchy stress tensor, cr = o-T.
This result also leads to a requirement on the first P i o l a - K i r c h h o f f stress tensor, F P v = P F v, and subsequently
to the symmetry of the second P i o l a - K i r c h h o f f stress tensor, S = S v.
In order to address incompressibility locking, a mixed method which modifies the definition of the
deformation gradient is used [18]. The modified deformation gradient is based on a separation of the
deformation gradient F into volumetric and deviatoric parts

F = Fvo.Fde v (19)
The determinant, J, of the deformation gradient measures the volumetric part of the deformation

J = det F = det F,,o~ det F d.... (20)

This leads to
detFvo , = J and detFde , , = 1 . (21)
In constructing the modified deformation gradient, a mixed treatment replaces the volumetric part. The modified
 S.M. Rff'ai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7 399

deformation tensor, F, is defined using a mixed representation, 0, for the determinant of the deformation
gradient
= (O/J) I/3F (22)

The virtual modified deformation gradient, ~ff, is

1
~/~ = [ ~ 1, + ( V ~ / - ~ div ~ f l , ) ]ff (23)

where d is the solid displacement. Here, the rank two identity tensor, 1, with respect to the current configuration
basis vectors e,, is given by

1, = ~ie, ® ej. (24)

Adding the mixed pressure, p, to the motion, qS, and the mixed determinant, 0, of the modified deformation
gradient completes a three-field variational statement of the problem. Variational equations can be written for the
linear momentum, the relationship between the mixed pressure and the trace of the stress, and the relationship
between the mixed pressure and the determinant of the deformation gradient

f~.(,oi,)d.O+f~tr[Vaa(a-,,o,+pl,)[d..O=f,, ,, d*, + f (25)

/tr8 p)
,80k 30 dO = 0 (26)

£ 8p(1-O) dg2=O (27)

In this variational statement, the modified Cauchy stress, 6; in Eq. (26) is related to the modified Kirchhoff
stress, ~, and to the modified second Piola-Kirchhoff stress, S, by J o = r = / ~ S f f . The spherical part of the stress
is given by the mixed pressure, p, not by the trace of the modified Cauchy stress, tr 6-. The mixed pressure, p, is
computed from tr 6- using variational relation (26). Thus, the stress in this approach is computed using
o-= p l , + ~'do~ (28)

4.2. Structural elements

The structural formulations beams and shells) are expressed in resultant form [20-22]. For example, the shell
kinematic assumption defines the placement of points qb in the shell in terms of the midsurface position,
q~((l, (2), and a distance ( = (3 along the director, t ( ( I, (2)

q,(~:,, ~:2, () = ~(~:,, ~:~) + ~t(~:l, ~2) (29)

The shell deformation gradient, F, is described using the gradients of the current and initial (reference) shell
placements, V@ and V@o
F ~ VqO(V~,)-' (30)

with
Vq~= ( ~ + ~t ~ ) ® E ~ + t ® E (31)

and the fixed inertial frame, E ~, E : and E 3 = E spanning the reference configuration (Fig. 3).
The Jacobian determinant of the deformation gradient is defined as
J -~ det[F] =J/Jo (32)
The resultant shell equations are derived using the convected basis, g;, and the dual (reciprocal) convected basis
g;, defined as
400 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7

~2

g3

,/- T E2

E1

Pig. 3. Illustration of shell coordinate frames.

045
gt = ;~sct = V45EI (33)

g l = (745) r E / . (34)

Particularizing (33) to the midsurface gives the midsurface convected frame, {at} ,

a~, = ~ a3 = g3 = t. (35)
The stress resultants n ~, the stress couples in" and the director stress couples th ", are defined in terms of the
three-dimensional Cauchy stress tensor o-,
1 ( h+
n ~7 Jh- °'g~j'd(' (36)

m ~ --=7-
J , (45 - ~) × o-g"j d s C = =J t Xj~, scog~" dsc , (37)

1
m ~- (~+ s~o-g'j• ds~ (38)
J ah
where f is the Jacobian determinant at the midsurface.
Based on these definitions, the structural elements are coupled to general material models using numerical
integration of the constitutive relations through the thickness direction.
The resultant linear momentum, p, angular momentum, rr, and director momentum, #, are defined in terms of
the three-dimensional density, p, as

p =-- ~-1
J
f[+ p45j dsc , (39)

1 fh h +
7r ~- -7- P(45 - q~) × q6j dsC, (40)
J
"h'--= rr × t . (41)

With these definitions, the resultant form of the balance of linear and angular momentum is
1 7" c~
_ (in)~, + a = ,6 = , 6 ¢ , , (42)
J
1
_ ( j m7- cr
).+ qL x nor
+rh = ~=hXi" (43)
J
 S.M. R(fai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417 401

where l~ and ni are the applied resultant force and stress couple per unit length, respectively. The symmetry of
the Cauchy stress tensor, and Eq. (43) give the balance of director momentum
I
_ (j=ni~),~ - 1 + ~ = [i" (44)
J
In order to address shear and membrane locking, a mixed method based on the H u - W a s h i z u variational
principle and employing independent interpolations for the resultant membrane and shear strains is used
[19-221.
Configuration updates for the structural elements respect the inextensibility constraints of the director fields.
The exponential map is employed for rotational updates which are geometrically exact and singularity free [21].

5. Multiphysics interfaces

Interactions between regions are enforced through unstructured mesh interfaces (Fig. 4). The discretizations
on either side of these interfaces need not coincide or share the same topology. A s l a v e - m a s t e r algorithm is
used to define the discrete interface constraints of multiphysics problems [23]. These constraints are enforced
with the augmented Lagrangian formulation.
Full exposition of these algorithms is beyond the scope of this presentation, however, the basic concepts can
be illustrated with simple examples.

5.1. Slave-master algorithm

Interfaces are defined on surfaces between two regions labeled as a master and a slave surface. In FSI
applications, the solid surface is chosen as the master surface. A search/projection algorithm identifies the
master element facet which contains each slave node. This algorithm supports contact with separation, tied and
sliding interfaces.
The interface constraints which couple the physics between two regions are enforced as a relationship
between state variables at the slave nodes and the master element facets. This is illustrated in two dimensions in
Fig. 5, where slave nodes 1 and 2 are tied to master facet B with nodes 3 and 4.

5.2. Interface constraints

At the fluid-structure interface, constraints coupling the fluid particle velocities, A L E mesh movement and
solid displacements are prescribed. These constraints can be expressed as 'strong' conditions on the solution
variables, or ' w e a k ' conditions on the gradients. Strong conditions are enforced with the augmented-Lagrangian
method described below. Weak conditions are applied as external loads.

Fig. 4. Interface illustration on a wing aeroelasticity problem. Fig. 5. Slave master interface illustration.
4O2 S.M. Rifai el al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417

Table 1
Interface constraint options

Constraint type Strong condition Weak condition

Mechanical Slip u s -n = u m-n p~ = p,,,

No-Slip u~ = U m t = t,~,

Mesh Slip u AIE- n = u m "n N/A

No-Slip /,/ALE = Um

Table 1 depicts the constraint options for fluid-structure interaction coupling. Here, the subscripts s and m
refer to the (fluid) slave node and the corresponding point on the (solid) master surface of the interface,
respectively. We denote by u, p, t, u ALE, and n the fluid or solid particle velocity, pressure, traction, ALE
velocity, and the normal to the interface surface, respectively.

Remark. In many aeroelastic applications, the tangential tractions on the structure are negligible. Under this
assumption, the no-slip weak mechanical condition is approximated by the slip case.

5.3. Augmented-Lagrangian formulation

To illustrate the augmented Lagrangian formulation, consider the case of mechanically coupled flow at a fluid
slave node, s, with a solid master facet, m. The coupling constraints Z for the slave node s in this case are

['u~l~- u~].,1
= 0, s = 1,n~n, (45)
b,31 .31 J
where the subscripts s and m denote a quantity at the slave and master surfaces, respectively, and n,, is the
number of slave nodes on the interface. Recall that u~]~ and u,l,, are the velocity components at the slave and
master side of the interface, respectively. These constraints are enibrced through an augmented Lagrangian
formulation with penalty regularization. In the Uzawa implementation of this method, two sets of decoupled
equation systems are solved in an iteration to determine the physical simulation variables, V, and a set of
Lagrange multipliers, A, associated with the interface constraints. These equation systems can be represented as

dr dy _ dy]
6v f ,
(46)
KAA=y

where r is the equation system associated with the physical simulation variables, f is the residual force
associated with that system, y is the set of interface constraints, AV and AA are solution increments, and the
matrix K is a diagonal matrix of penalties for regularizing the interface constraints. The derivatives dy/dV in
Eq. (46) are obtained from Eq. (45) using the chain rule.

REMARK. For computational efficiency, Eq. (46) contains some approximations to the rigorous derivation of
the augmented Lagrangian formulation. In Eq. (46)~, second derivative terms of the interface constraints are
omitted from the linearization. In practice, these terms have shown a destabilizing effect on nonlinear iterations
for large time steps. In equation (46) b, the matrix K is an approximation of the linearization of the Lagrangian
equations. This approximation is valid for large values of the penalty terms. Both of these approximations are
typical in augmented Lagrangian implementations.
 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7 403

6. Parallel processing

The high-fidelity simulation of aeroelasticity and flow induced vibrations is computationally expensive. While
single processor computers improve every year, the quantum increase in computational needs requires a
quantum increase in computational power. The hardware industry is answering this increased need with
affordable high-performance parallel architectures which are built on commodity parts and compatible with
workstation systems. These architectures offer the level of computational performance and increased memory
capacity needed for large-scale computing. In the context of these architectures, two parallel programming
models have emerged as popular and supported approaches: data-parallel and Same-Program Multiple Data
(SPMD). Because the underlying architecture is different than traditional uniprocessor or vector computers, the
increased performance in both models is most fully available to codes designed to take advantage of the new
architectures.
The same software architecture that supports multiple physics simulation naturally and cleanly supports
independent and parallel computation. For each problem, we maintain the concept of multiple subdomains. A
subdomain is a collection of regions which are uniform with respect to linear solution technology (e.g. iterative,
direct, etc.). This model provides the support and underlying data structures for efficient parallel processing (as
illustrated in Fig. 6). At the higher-level, the coarse-grained subdomains map well to the SPMD programming
model [1]. At the lower-level, the fine-grained element sets map well to the data-parallel model [24]. Therefore,
the multiphysics architecture maps well to either (or both) parallel programming models.
The data-parallel approach has been investigated within the context of the present method [24-26].
Data-parallel programming requires significant re-coding in a data-parallel language such as Fortran-90 (F90) or
High-Performance Fortran (HPF). Along with these dialects, proprietary languages also exist. Thus, no
data-parallel language standard exists. In addition, the programmer is expected to specify explicit, and often
system dependent, data layout. It was also observed that the unified program view simplifies debugging and the
investigation showed good speedup in many cases. However, the program development and support costs are
significant and the resulting source code is generally not portable.
By comparison, the alternate approach of SPMD is implemented via message-processing. In this case, there is
support for a wide range of hardware platforms with either PVM or MPI. This programming model has the
advantage of allowing the reuse of large portions of code from the uni-processor version. Excellent performance
has been demonstrated on a number of different parallel systems for large, multiphysics simulations. The
relative ease of programming and support as well as the portability, scalability and wide availability of systems
supporting SPMD make it the preferred model for our applications.

I Top Driver ) Time Stepping

Nonlinear Solver
~ - I Gl°bal C h ~ a p h e ~ ) { Linear SOlver
. .-" ,, ".. "- -. control
Subdomains
Coarse grain
SPMD
Message Passing
Linear I MatrixForm I ~ _1~-- I
0
Element sets
Fine grain
Data-parallel
F90/HPF

Inter-subdomain ',........ -,-.......... -'......... Data Exchange

Communication

Fig. 6. Parallel processing models under the multiphysics ar- Fig. 7. Multi-subdomain architecture for coarse-grain parallel
chitecture, processing.
404 S.M. R(fai el al. / Col~qmt. Methodv Appl. Mech. l-5ngr,q. 174 (1999) .:/9_:/-417

6.1. C o a r s e - G r a i n P a r a l l e l i s m

In the coarse-grain model, each processor of the parallel machine runs a copy of the application to solve one
subdomain of the partitioned grid. A schematic description of the coarse-grain parallel architecture is presented
in Fig. 7.
Conceptually, this approach attempts to parallelize computation at the newly created subdomain level (i,e. at
the outermost loop level). For example, the pseudo-code below shows the outer subdomain loop implicit in the
computation.
doacross (subdomains)
p e r f o r m local s u b d o m a i n c o m p u t a t i o n s
enddo
c o m m u n i c a t e n o n - l o c a l data
doacross (subdomains)

enddo

6.2. D o m a i n d e c o m p o s i t i o n

The decomposition of the computational domain (e.g. Fig. 8) is an active area of research which has produced
many algorithms and general purpose tools [29-35]. Many decomposition methods commonly use the recursive
spectral bisection (RSB) approach [29]. A significant cost of the RSB method is associated with the computation
of eigenvectors of a Laplacian matrix constructed from the adjacency structure of the mesh. Hendrickson and
Leland introduced a multi-level implementation for the construction of the Laplacian matrix, resulting in
significant CPU perlkwmance improvement [31]. Karypis and Kumar present a rigorous analysis of multilevel
methods and demonstrate analytically their effectiveness [351.
A common feature of most domain decomposition research is the focus on single homogeneous grid
applications. In order to support multiphysics simulations, algorithms which extend the multi-level method of
Karypis and Kumar to heterogeneous interlaced discretizations are used in the present approach [36].
A simple extension to a domain decomposition algorithm is illustrated here which ensures that all elements on
a single interface remain in a single subdomain. This extension is carried out in the following steps:
• Identification of all elements associated with an interface. In this step, one subdomain is created containing
the elements involved in the fluid-solid interface.
• Partitioning of the remaining elements into subdomains using a single-physics decomposition algorithm.
• Identifying the boundaries of these divisions so that the proper message-passing information can be tagged
at each location where the mesh is partitioned for parallel processing.

Fig. g. Domain decomposition illuslration.

 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393 417 405

The motivation of this approach is twofold: Firstly, it simplifies the implementation of a multi-physics domain
decomposition algorithm and secondly, this approach is ideal for a class of multi-physics applications where the
number of elements on an interface is much smaller than the rest of the computational domain. In this context,
the integrity of interface computations is preserved and inter-subdomain communication overhead is reduced.
The main drawback of this algorithm is that the number of elements on an interface places an upper-bound on
the parallel efficiency. Once a certain level of decomposition is exceeded, the workload of the interface
subdomain remains unchanged.

7. Multi-subdomain equation solver

To take full advantage of the architecture described above, a multi-subdomain solver is employed to solve the
matrix set of equations which result from the discrete finite element problem. In this context, the solver is
composed of one global solver and a set of local subdomain solvers. At the subdomain level, the local solver
may be explicit, implicit iterative or implicit direct. The local subdomain solver may vary from subdomain to
subdomain. The global solver is implicit iterative.
Two iterative solvers are used for multiphysics problems: the preconditioned conjugate gradient (CG) and
generalized minimum residual ( G M R E S ) methods [37-391. CG is used for symmetric systems which arise, e.g.
from solid linear momentum, heat transfer and mesh movement equations. GMRES is used for non-symmetric
systems which arise, e.g. from the fluid linear momentum, thermal, scalar transport, and turbulence equations.
The example shown in Fig. 9 can be used to demonstrate how the interior nodes of certain subdomains are
"removed" from the global solver. The general structure of the equation systems of domain decomposition is also
illustrated by the example.
Two sets of nodes are distinguished: subdomain boundary nodes and subdomain interior nodes. Boundary
nodes are shared by elements belonging to different subdomains. All other nodes are interior nodes. Arrays
associated with interior nodes and boundary nodes are denoted by subscripts 1 and 2, respectively. In solving the
global system A x = b, contributions to A are assembled from A "xp of the explicit subdoinain, A d~ of the implicit
direct subdomain, and A '~' of the implicit iterative subdomain.

Ac'e= j 0
cxp
]
These contributions can be written as

'
[d,,
A d i , = All A]2
dir]
eli[" • and
A,~,
[,z~,ter iicr]
IJ A]2
(47)
A22 J _adi'21 A 2 2 ] 2liter A22ilcr]

where A ' ' e is a block diagonal matrix and both A a~r and A ~'~r are potentially nonsymmetric matrices with
symmetric profiles.
Fig. 10 graphically represents the structure of these matrices. Note the symmetric skyline structure o f A "~ and
the fact that A :t~' is stored in the form of unassembled element matrix files.

Implicit Direct Implicit Iterative

Subdomain n

O Subdomain-interior node
• Subdomain-boundary noc

Explicit Subdomain
Fig. 9. Subdomain partitioning example.
406 S.M. R{fai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7

A exp --

Fig. I0. Explicit, implicit direct, and implicit iterative matrix structures.

With these representations, the global system A x = b is

**11 "*12 xdir _ -i (48)

"'011 0 ailt~' ail ter // X itcr / - biltcr

A2~dir A2lit. . . . . (A22P -}- at d Xi r 4- - iler

- - --zl 22
2 )2J bndy J (beXP+b~ir+b~ e')
where x ~xp, x a~ and x i'~ are the solution subvectors for the interior nodes of the three subdomains and x u'ay is
the solution subvector for the shared boundary nodes. The right-hand side vectors b exp =J and b*t~
r , lbdir r , are stored
with their respective element subdomain data; whereas b~×P+ b~ ~ + b ~ ~~ is stored with the global data.
Direct solution techniques are used to eliminate x ''~p and x a~. The solution of
A ~ P x exp = b~ xp (49)

involves the triangularization of uncoupled, symmetric, positive-definite matrices. A static condensation is used
to eliminate x d~. The reduced system has the form

b'l'" } (50)
LA21
"" (Aexp
'e-22 + a ~~dir
,. . +. a. ~ ile, ) [xbndy J (b~xp + b ~dir
2 +b~'%
where, using the familiar Schur complement,
dir : A d i r __Adir( a dir] IAdir
A2, --''22 ~21 ',** II / *~12 (51)
and
b ~ ir : ~ d i r "zt dir/z[ dir ] - Ih dir (52)
--~2 **2 \ ' * 1 1 / Vl

The reduced system contains only the interior nodes of the implicit-iterative elements and the boundary nodes.
This is the desired result. Once Eq. (50) is solved, x d~r is determined from
dir dir -- I dir dir bndy
X =(All ) (b 1 - A . 2 x )

The global system has been replaced by Eqs. (49), (50) and (53).

8. Applications

The work described herein has been applied to various problems from industry. In this section, we illustrate
applications of aeroelastic instability and flow-induced vibrations. In general, the process for modeling
instability applications is divided into the following four steps for each set of flow parameters:
(1) Simulate the steady-state flow assuming rigid solid boundaries.
(2) Simulate the steady-state fluid-structure interaction accounting for solid deformations and using the
solution of step 1 as an initial condition.
(3) Simulate the transient fluid-structure interaction applying an initial perturbation to the solid and using the
solution of step 2 as an initial condition. The initial perturbation usually takes the form of a small force
applied for a short period of time or an initial velocity. It is chosen to excite a few of the (first) modes of
the structure.
 S.M. Rijkli et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417 407

(4) Post-process the time-history results of step 3 to obtain frequency and damping response of the
fluid-structure system to the initial perturbation.
The following applications will illustrate the application of this process to aeroelastic and flow-induced
vibration problems. We will highlight the importance of capturing the nonlinear effects and accounting for
mesh-movement. The scalability of the software is also depicted.

8. 1. P a n e l f l u t t e r

A 2-D panel flutter problem is used to validate the methods developed in this work and illustrate the
difference between linear and nonlinear aeroelastic computations. The problem consists of supersonic flow over
a flat plate which is clamped on both ends (Fig. l 1). The panel is initially flat, with a constant freestream
pressure on both sides. An analytical solution exists for the linear aeroelasticity problem: with the panel
characteristics listed in Table 2, the stability limit (i.e. the flow speed above which flutter occurs) for the panel is
at Mach 2.
To perturb the instability, the lower side pressure is dropped by 0.1% for 4 milliseconds. Thereafter, the
pressure is brought back to 28 kPa. The flow is assumed inviscid (compressible Euler equations). Within each
time-step, the FSI equations are solved in a staggered order. The multisubdomain solver described in section 7 is
used for this analysis. The fluid region is one subdomain with the solid and interface regions combined in a
second subdomain. In the fluid subdomain, G M R E S is used for the flow equations, and conjugate gradients are
used for the mesh movement equations. A direct solver is used on the second subdomain.
Fig. 12 plots the time history of the displacement of the panel for several flow speeds using linear elasticity.
As seen in the plots, the oscillations of the plate for Mach numbers 1.90 and 1.95 are damped, while the
oscillations for Mach numbers 2.00 and 2.05 grow unbounded. The case of Mach 1.98 appears critically
damped.

Fig. I 1. Mesh discretization of the panel flutter problem. A triangular mesh is used for the Euler flow domain and a quadrilateral mesh is
used for the plate,

Table 2
Panel flutter properties.
Property Value Property Value
Young's modulus 77.28 GPa Length 0.5 m
Poisson's ratio 0.33 Thickness 1.35 mm
Density 2,710 kg/m ' Freestream pressure 28 kPa
408 S,M. Rifai et al. I Comput. Methods Appl. Meeh. Engrg. 174 (1999) 39.3-417

0.002

0.0015
I I I I I t i i

i¸ !
r j
0.001

E
0.0005
o
II
x

-0.0005

"1o M = 1.90 - -
-0.001 M= 1.95 --- ! i , -i
M = 1.98 ...... r

M = 2.00 .........
-0.0015 M = 2.05 -

I i
~j
I I I I I I I I I
-O.O02
0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
time ( s )

Fig, 12. Time histories of the plate deformation at different flow speeds.

The difference between the linear and nonlinear aeroelastic solutions is depicted in Fig. 13 for a Mach number
of 2.3. The linear solution (consistent with the analytical solution) predicts an unbounded exponential growth of
the oscillations of the panel at this unstable speed. The nonlinear solution, however, shows a limit cycle (a result
often observed in experiments). The difference between the two solutions can be attributed to the coupling

0.01

0.005
M ach = 2.3 i/ ~tl
E

c5 ",.~/" ',:~L/~ ",..

II
x

r-

E -0.005

linear
nonlinear
-0.01

I
-0.015 ~ L , ,
0 0.02 0.04 0.06 0.08 0.1 0.12
time (s)

Fig. 13. Comparison of the linear and nonlinear aeroelastic response of the panel at Mach 2.3. The linear theory predicts exponential growth
of the displacements at this unstable velocity, whereas the nonlinear theory captures the limit cycle oscillations usually observed.
 S.M. Rijai et al. I Comput. Methods Appl. Mech. Engrg. 174 (1999) 393 417 409

between membrane and bending stresses in the panel with nonlinear elasticity. This phenomenon increases the
effective stiffness of the panel with its deformation, which changes the behavior of the system.
Many aerospace components (particularly in military applications) are used beyond the stability limits
predicted by the linear theory. The methods developed in this work offer a tool which takes advantage of
high-performance computing to predict limit-cycle behavior with nonlinear aeroelasticity.

8.2. NACA 64A010 airfoil

The NACA 64A010 airfoil configuration was used by Grisval et al. to perform 2D flutter analysis [40]. Two
applications were considered: a viscous transient flow simulation with prescribed boundary movement, and an
inviscid transient fluid-structure interaction simulation with a rigid solid domain. The mesh configurations for
both applications are shown in Fig. 14. The fluid flow free-stream conditions for these simulations are listed in
Table 3.
The first application (viscous transient flow) was subjected to a prescribed oscillation of the wing boundary at
a frequency f = 34.4 Hz and an amplitude cr = 1.01 °. In order to demonstrate the importance of accounting for
mesh movement in aeroelastic applications, this application was solved with two approaches:
(1) Transpiration Approximation (TRSP): Simulate the airfoil motion without mesh movement, using only
prescribed particle velocities at the boundaries.
(2) Mesh Movement Approach (ALE): Simulate the airfoil motion by prescribing both mesh movement and
particle velocities at the boundaries.
The real and imaginary parts of the pressure coefficients for both cases are shown in Fig. 15. Note the close
agreement between the ALE results and experimental data and the significant errors with the TRSP
approximation. This example illustrates the importance of taking mesh movement into account in aeroelastic
applications, even for small deformations.

~ ( . ~ ~ - Z ~ ,,,%,"x/~'~ # 'v_,x/,,,v~,x - ~ . s 4
,, -, ,,% ,_~ ,, ,

(a) (b)
Fig. 14. Fluid and solid discretizations of the NACA 64A010 airfoil.

Table 3
Freestream flow conditions for NACA 64A010 airfoil
Property Value Property Value

Velocity 267.5 m / s Mach number 0.796

Pressure 1.339 × 105 Pa Reynolds number 12.6 × 10~
Temperature 281.0 K
410 S . M . Rifizi et al. I C o m p u l . Methods Appl. Mech. Engrg. 174 ( 1 9 9 9 ) 393-417

15 I0

6S~li
I
10 :' ~,,

5 ?"~, , __y'
I

+ ~ °
, I
w
' i: + i ....~ ',;o_ ,,oo
0 ' /

-5

upper surf (ale) upper sud'(ale) ~ I

10 lower surf (ale) lower surf (alel ',, /
upper surf (trsp) upper sud (trsp)
lower surf (trsp) lower surf (trsp)
15 ' -10
0.05 0.1 0.15 0.2 025 0.3 0.35 0.4 0.45 0.5 0 0.05 0. l {}.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5
x (m) x (m)

(a) (b)
F i g . 15. Real and imaginary parts o f the unsteady pressure coefficient. Isolated points represent e x p e r i m e n t a l data. N o t e the loss o f a c c u r a c y
w h e n m e s h m o v e m e n t e f f e c t s are ignored.

2c -05 {).005

0.004
15e-05
0.003

I e-05 -

5e-{16
0.{X)2 HI

o.ooi : ' ,,, ,

',/,, ',~:

,
' ,' ,' ; : ' :

," , ,/ ':'i, :'/ :,,:,~ :~, ,J, ~:r~:, i:" i:, :iI :'i

°I -0.001 i~

0.002 ~ "
5e 06 Case 1 Case I ';,"
Case IX -O.{X)3 I Case I1
|
-le-05 -O.(X)4 L
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 ().2
time (s) time (s)

F i g . 16. Vertical d i s p l a c e m e n t and rotation o f the N A C A 64A010 a i r f o i l in t w o rigid configurations.

The second application (inviscid fluid-solid interaction) employs a solid with high stiffness to model the rigid
airfoil. The airfoil is free to rotate about an elastic axis which moves only in the vertical direction. The rigid
airfoil is governed by a 2-degree of freedom (vertical displacement and rotation) dynamic system• The properties
of the system are chosen so that the translational and rotational frequencies are 60 and 65 Hz, respectively. To
perturb the system, the solid is given initial translational and rotational velocities of l0 4 m/s and 10 4 rad/s,
respectively. Two cases are used to simulate the airfoil motion under stable and unstable conditions• The results
are shown in Fig. 16. The difference in the results between the two cases is highly influenced by the location of
the airfoil's center of gravity.

8.3. DYVAS wing flutter

The aeroelastic properties for the DYVAS wing configuration have been reported by Grisval et al. using the
methods described in this work [41]. The structural mesh and surface mesh of the flow domain are depicted in
Fig. 17 for this computation. The structural model is designed to accurately replicate the dynamics of the
experimental wing for the first two modes of vibration. A system of beams and concentrated masses is used at
the root of the wing. Vertical stiffeners between the upper and lower wing surfaces provide additional rigidity for
the wing panels. Shell elements model the skin of the wing.
 S.M. Rifai et at. I Comput. Methods AppI. Mech. Engrg. 174 (1999) 393-417 411

Fig. 17. Surface mesh of the flow domain and mesh discretizatinn of the wing structure for the DYVAS wing.

Fig. t8, Deformed and undeformed views of the DYVAS wing structure at steadystate prior to the aeroelastic simulation,

I P ~ = 0.9 bar I Pi~ = 2.0 bar

0.002 0.015
0.01
0.001 /'11 /~, ~ 0.005
~1 i / / , , ;
"~ 0

-0.001
I/ I~I!'~ t,/ 'v' 0
-0.005
-0.01
"~ -0.002
V
-0.015
0 0.05 0.1 0.15 0.2 0.05 0.1 0.15 0.2
time (s) Ume (s)

Fig. 19. Oscillations of the DYVAS wing tip at Mach 0.60 for a stable freestream pressure (0.9 bar) and a critically stable case (2.0 bar).

Pioo = 0.9 bar ] Pi~ = 1,5 bar ]

0.015
~" 0.002

0.001
0.01
,~,//~11/,/~
",,,u#/,lj/~ / 1/, ,,,,
0.005
0

-0.001
I/'~"Jlj 1j bl 0
-0.005
-0,01
"~ -0.002
-0,015
0 0.05 0.1 0.15 0.2 0.25 0.05 0.1 0.15 0.2
time (s) time (s)

Fig. 20. Oscillations of the DYVAS wing tip at Mach 0.78 for a stable freestream pressure (0.9 bar) and an unstable case ( 1.5 bar).
412 S.M. Ri/}ti et al. / Comput. Method,s Appl, Mech, Engrg. 174 (1999) 393 417

Table 4
DYVAS Wing Results
Mach number Pressure Compuled Measured
Frequency, Damping factor Frequency, Dampingfactor
0.60 0.9 bar 26.4 - 0.051 26.1 +0.056
30.3 0.010 30.9 +-0.014
2.0 bar 29.0 +0.176 N/A
30.0 +0.000
(}.78 0.9 bar 29.4 +0.063 28.2 +(}.057
29.6 + 0.000 30.0 + 0.0 [9
1.5 bar 29.7 0.018 N/A
30.6 +0.113

To excite the wing, a tip load is applied for 4 i n s and then removed. The flow is assumed inviscid
(compressible Euler equations). Fourier transform identification is used to capture the frequency and damping
factors of the oscillations. Within each time-step, the FSI equations are solved in a staggered order. The
multi-subdomain solver is used for this analysis. The fluid region is divided into multiple subdomains with the
solid and interface regions combined in an additional subdomain. In the fluid subdomains, GMRES is used for
the flow equations, and conjugate gradients are used for the mesh movement equations. A direct solver is used
on the remaining (solid) subdomain.
Two flow speeds were used in this study: Mach 0.60 and Mach 0.78. Fig. 18 shows the steady state
defl)rmation of the wing at Mach 0.60. For each speed, the oscillations of the wing at a low fleestream pressure
case in the stable wing region and a high pressure case in the unstable regime are simulated. Figs. 19 and 20
depict the oscillations of the wing for the Mach 0.60 and 0.78 cases, respectively. Table 4 summarizes the
frequency and damping results. Good agreement with experimental results is observed I411.

8.4. S o c k e t i n s t a b i l i O'

A socket instability problem is used to illustrate this work on a flow-induced vibration application as shown in
Fig. 21. A cylindrical rod is submerged inside of a fluid-filled pipe, with fluid flow induced along the axial
direction. Although the geometry of the problem is axisymmetric, the socket instability admits no symmetries.
Typical dimensions of this problem are on the order of 4 inch diameters for the rod, with less than 0.01 inch
clearance between the rod and the pipe wall. Due to the relatively small clearance, slight perturbations of the rod
geometry cause 3-dimensional disturbances in the flow phenomena. Typical flow rates may vary between 1 and
20 GPM in this setup.

Fluid Outflow

I
Clamp

Fluid Inflow

Fig. 21. Geometry and coarse 3D mesh of the socket instability model problem.
 S.M. RifkU et al. I Comput. Methods Appl. Mech. Engrg 174 (1999) 393 417 413

2e 0 5 I I 8e-05 i i i
v = 0.000031 - v = 0.01524 -
v = 0.000061 v = 003048 --
1.5e-05 t ;';; v = 0.000152 -- 6e-05 v = 0.04572 ....
;; v - 0.000305 v = 006096
le 05 ,',, ;
,
v
v
=
-
0.000610
0.001524
-
- 4e-05
v - 0.003048 -
5e-06 ....~, , tlli kk~l
3e-05

0 ~ ~ ~ s ,,'~ ~~ " ,',,':~ ,

0
-5e-06

-2e-05
I /
le-05
ii",j
-1.5e-05 -4e-05

2e-05 I. I I -6e-05 l I
0.1 0.2 0.3 0.4 05 0.6 0,7 0 006 01 01, 02 02° 03 035
Time Time

(a) (b)
Fig. 22. Rod tip displacement histories at differenl flow rates.

The flow is assumed viscous (incompressible Navier-Stokes equations). The 3-dimensional nature of the flow
and the complex flow patterns, including turbulence modeling, require very large mesh discretizations for
accurate simulation of the fluid region of this problem. With the vibration frequencies expected to be in the
range of 5 - 5 0 Hz, time steps on the order of 10 ~ s spanning several oscillations of the entire frequency range
are needed, resulting in over a thousand time steps. Within each time-step, the FSI equations for fluid and solid
particles are solved in a fully-coupled single stagger. The ALE equations are solved in a second stagger.
Multiple iterations are used for nonlinear convergence. The multi-subdomain solver is used for this analysis. The
fluid region is divided into multiple subdomains with the solid and interface regions combined in an additional
subdomain. The global solver is GMRES for the fluid and solid particle equations and conjugate gradients for
the mesh movement equations. A direct solver is used within all subdomains.
A 2D model was used to evaluate the damping characteristics of the socket configuration and the results were
compared with the 3D model. The simulations all used a steady state fluid flow computation to create the fluid
initial conditions in the multiphysics problem. The solid was then perturbed by a small lateral force held tk~r a
short period of time and released to allow the rod to oscillate within the flow. Experimental data for this problem
indicate that:
• The rod oscillations are damped at low flow rates, but as the flow speed increases, the fluid-solid dynamic
system becomes unstable and the damping factor decreases exponentially.
• The frequencies of oscillation are nearly constant for all the low flow speeds, however, as the flow rate
increases, the frequencies begin to increase exponentially.
Fig. 22 depicts the time-histories of the lateral tip-displacements of the 2D model. The characteristics of this

15 160 ...... I . . . . . . . . I . . . .

i/
10 150
/4/;
j4 ~ 140
5
130
¢ 0
t 20 /
5 !
{3 110 Deformable Fluid Domain ~
c~
c
"5_
-10
100
Fixed Fluid Domain -+
/
£3 -15 Deformable Fluid Domain ~ "~\
Fixed Fluid Domain -~ ~, 90

-20

-25
\ 80

70

-30 ...... I ,,i . . . . . . . . I ,

0001
.... i
0.01
. . . . .
00001 0.001 001 0.0001
Velocity Velocily

Fig. 23. Socket instability damping and frequency response to wu'ying ttow rates (inflow velocities).
414 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 3 9 3 - 4 1 7

1<~ I l l i i i
1.2e-05 3D-- 0.9 ! ~ - Theoretical -
2D 0.8 Spectrum -~*-
1e-05 0,7
0.6
8e-O0
6e-08 o.51

4e~36 O
0.4

2e-06 0.3
0

-2e-06

-4e-06 0.2 ~11%

-6e-06
0 '
0.06 o,I g' 02! 8' 1'2 1~ ,o 20 24 28 32
Time Number of Subdomains

Fig. 24, Comparison of the 2D vs. 3D results. Fig. 25. Scalability of the 250K element socket instability flow
simulation. Near-theoretical seating is seen in this case with
superlinearity at 12 and 16 subdomains.

figure are in good qualitative agreement with the experimental data. The damping and frequency characteristics
are further illustrated in Fig. 23.
Due to the small displacements of the solid prior to the onset of instability phenomena, it has been suggested
that the deformation of the fluid domain need not be taken into account for this class of problems. We tested this
hypothesis by comparing the damping and frequency characteristics of the socket configuration with and without
moving fluid boundaries. Our investigation revealed that the motion of the fluid boundaries is indeed crucial to
replicating the instability phenomena (Fig. 23). With the fluid domain held fixed, the fluid-solid system
damping increased with the flow rate rather than decreasing as the experiments indicate. Furthermore, the
frequency of oscillation remained constant in sharp contrast to the experimental data. These results indicate that
the modeling of fluid boundary motions with an arbitrary-Lagrangian-Eulerian formulation is critical to
capturing the physics of the problem.
Fig. 24 also compares the 3D results with the 2D results for this problem at a baseline flow rate. It is clear that
the 3D effects are quite important in this case. The frequency and damping are both higher in the 3D case than
in the 2D case.

50000
Theoretical - -
450OO
0.875 Spectrum ~'-
40000 50K mesh - -
250K mesh
0.75 35000 1M mesh ....
6hr Threshold .....
30000
0.625 25000
i.= . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [ ........

o
20000
15000 n
i
i

10000 ; ' i '

r L, iI , n p
J
,o00 '

0.375 / I I I 0 iIL, i i d
24 32 40 48 56 64 10000 2 0 0 0 0 3 0 0 0 0 4 0 0 0 0 50000 60000 70000
Number of Subdomains Average Nodes per Subdomains

Fig. 26. Scalability of the 1M element socket instability flow Fig. 27. Alternative, industrial view of scalability: In a given
simulation. Excellent overall scaling with superlinearity at 32 time-frame (dictated by design cycles) a solution can be obtained
subdomains is observed. for any level of mesh discretization by adding more parallel
processors.
 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 292 417 415

8.5. Scalability illustration

An internal flow computation from the socket instability analysis is used to illustrate the scalability of our
work on the IBM/SP2. The simulation involves the computation of steady state flow around a solid cylinder
inserted in a cooling socket (see Fig. 21). The computations were repeated for a 250 thousand and 1 million
element mesh discretizations using various domain decompositions in the range of 4-64.
The scalability of fluid flow computations is illustrated in Figs. 25 and 26. The 250K element mesh exhibits
near-theoretical scaling between 4 and 32 partitions with superlinearity at 12 and 16 subdomains. In general, this
phenomenon is likely due to the subdomains becoming small enough to fit into the individual processor cache
memory. Finally, the 1M element mesh shows excellent overall scaling between 24 and 64 partitions with
superlinearity at 32 subdomains.
Fig. 27 presents an alternative view of scalability for the fluid flow problem. In industrial applications of
high-performance computing, the focus is on obtaining solutions in a timely manner which can fit into product
design cycles. Hence, the industrial concern is to obtain a solution for any level of mesh discretization in a given
time-frame by distributing the larger meshes onto more processors. The threshold at which computations fit into
the design cycle varies by industry, product, application, and individual user. We refer to it as the 'engineering
reasonable time'.

9. Conclusions

A method for the multiphysics simulation of aeroelastic instabilities and flow-induced vibrations on parallel
computing platforms has been described. The method is based on a single computational framework for the
modeling of multiple interacting physical phenomena. The software architecture lends itself naturally to
high-performance parallel computing. Several applications demonstrated the importance of capturing the
nonlinear effects, accounting for mesh-movement and the scalability of the software.

Acknowledgments

Portions of this work were supported by NASA Ames under Cooperative Research Agreement NCC2-9000
for HPCCPT-I. Additional support was provided by the Department of Energy under grant DE-FG03-
96ER82139. Centric gratefully acknowledges this support as well as that provided by our commercial
customers.

References

Ill M. Eldredge, T.J.R. Hughes, R. Raefsky, R. Ferencz, S. Rffai and B. Herndon, High performance parallel computing in industry, in: T.
Tezduyar and T.J.R. Hughes, eds., Parallel Computing, Special issue on High Performance Computing in Flov,' Simulations, 23 (1997)
1217-1233.
[21 S.M. Rifai, Multiphysics smmlation fur coupled fluids, thermal and structural analysis on high-performance computing platforms, Proc.
8th Annual Thermal and Fluids Analysis Workshop (TFAWS 97), NASA Johnson Space Center, Houston, Texas, September 1997.
[3] C.V. Eckstrom, D.A. Seidel and M.C. Sandford, Measurements of unsteady pressure and structural response for an elastic supercritical
wing, NASA Technical Paper 3443, November 1994.
[4] C. Farhat, M. Lesoinne and N. Maman, Mixed explicit/implicit time integration of coupled aeroelastic problems: three-field
formulation, geometric conservation and distributed solution, Int. J. Numer. Methods Fluids, 21 (1995) 807-835.
[5] G.R Guruswamy and C. Byun, Fluid-structural interactions using Navier-Stokes flow equations coupled with shell finite element
structures, AIAA Paper 93-3087, 1993.
[6] C. Byun and G.R Guruswamy, An efficient procedure for flutter boundary prediction on parallel computers, AIAA Paper 95-1484,
1995.
17] J.T. Batina, Unsteady Euler airfoil solutions using unstructured dynamic meshes, AIAA Paper 89-0115, 1989.
416 S.M. Rifai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417

[8] R.D. Rausch, J.T. Batina and H.T. Yang, Three-dimensional time-marching aeroelastic analyses using an unstructured grid Euler
method, AIAA Paper 92-2506, 1992.
[9] A.H. Boschitsch and T.R. Quackenbush, High accuracy computation of fluid-structure interaction in transonic cascades, AIAA paper
93-0485, 1993.
~t6~ I:w F_d~.~,:d~ :,,rid I.t~. Malone, Current status of computational methods for transonic unsteady aerodynamics and aeroelastic
applications, AGARD Structures and Materials Panel Specialists on Transonic Unsteady Aerodynamics and Aeroelasticity, Paper No.
1, 1991.
[11] T.J.R. Hughes, Recent progress in the development and understanding of SUPG methods with special reference to the compressible
Euler and Navier-Stokes equations, Int. J. Numer. Methods Fluids, 7 (1987) 1261-1275.
[12] F. Shakib, T.J.R. Hughes and Z. Johan, A new finite element formulation for computational fluid dynamics: X. The compressible Euler
and Navier-Stokes equations, Comput. Methods Appl. Mech. Engrg. 89 (1991) 141-219.
[13] G. Hauke and T.J.R. Hughes, A unified approach to compressible and incompressible flows, Comput. Methods. Appl. Mech. Engrg.
113 (1994) 384-396.
[14] K. Jansen, Z. Johan and T.J.R. Hughes, Implementation of a one-equation turbulence model within a stabilized finite element
formulation of a symmetric advecfive-diffusive system, Comput. Methods Appl. Mech. Engrg. 105 (1993) 405-433.
[15] C. Johnson, A. Szepessy and E Hansbo, On the convergence of shock-capturing streamline diffusion finite element methods for
hyperbolic conservation laws, Math. Comput. 54 (1990) 107 129.
[16] T.J.R. Hughes, W.K. Liu and T.K. Zimmerman, Lagrangian-Eulerian finite element formulation for incompressible viscous flows,
Comput. Methods Appl. Mech. Engrg. 29 (1981) 329-349.
[17] T. Tezduyar, M. Behr and J. Liou, A new strategy for finite element computations involving moving boundaries and interfacesIthe
deforming spatial domain/space-time procedure: I. The concept and the preliminary numerical tests, Comput. Methods Appl. Mech.
Engrg, 94 (1992) 339-351.
[18] J.C. Simo, R.L. Taylor and K.S. Pister, Variational and projection methods for the volume constraint in finite deformation
elastoplasticity, Comput. Methods Appl. Mech. Engrg. 51 (1985) 177-28.
[19] J.C. Simo and M.S. Rifai, A class of mixed assumed-strain methods and the method of incompatible modes, Int. J. Numer. Methods
Engrg. 29 (1990) 1595-1638.
[20] J.C. Simo and L. Vu-Quoc, A 3-dimensional finite strain rod model. Part lI: Computational aspects, Comput. Methods Appl. Mech.
Engrg. 58 (1986) 79-116.
[21] J.C. Simo, D.D. Fox and M.S. Rifai, On a stress resultant geometrically exact shell model. Part lII: computational aspects of the
Nonlinear Theory, Comput. Methods Appl. Mech. Engrg. 79 (1990) 21 70.
[22] J.C. Simo, M.S. Rifai and D.D. Fox, On a stress resultant geometrically exact shell model. Part VI: Dynamic analysis, Int. J. Numer.
Methods Engrg. 34 (1992) 117 164.
[23] I. Doghri, A. Muller and R.L. Taylor, A general three-dimensional contact procedure for implicit finite element codes, Engrg. Comput.
15(2) (1998) 233-259.
[24] Z. Johan, Data-parallel finite element techniques fbr large-scale computational fluid dynamics, Ph.D. Thesis, Stanford University,
Stanford, 1992.
J25] Z. Johan, T.J.R. Hughes, K.K. Mathur and S.L. Johnsson, A data parallel finite element method for computational fluid dynamics on
the connection machine system, Comput. Metht)ds Appl. Mech. Engrg. 99 (1992) 113-134.
[26] Z. Johan, K.K. Mathur, S.L. Johnsson and T.J.R. Hughes, An efficient communications strategy for finite element methods on the
connection machine CM5 system, Comput. Methods Appl. Mech. Engrg. 113 (1994) 363-387.
[27] M.S. Shephard and M.K. Georges, Automatic three-dimensional mesh generation by the finite octree technique, Int. J. Numer. Methods
Engrg. 32 (1991) 709-7449.
[28] B. Nour-Omid, R. Raefsky and G. Lyzenga, Solving finite element equations on concurrent computers, Procedures of the Symposium
on Parallel Computations and their Impact on Mechanics, Boston, ASME, 1988.
[29] H.D. Simon, Partitioning of unstructured problems for parallel processing, Comput. Syst. Engrg. 2 (1991) 135-148.
[30] C. Farhat and M. Lesionne, Automatic partitioning of unstructured meshes for the parallel solution of problems in computational
mechanics, Int. J. Numer. Methods Engrg. 36 (1993) 745-764.
[31] B. Hendrickson and R. Leland, A multi-level algorithm for partitioning graphs, Sandia National Labs Rep., Albuquerque, NM
87185-1110.
[32] J.G. Malone, Automated mesh decomposition and concurrent finite element analysis for hypercube computers, Comput. Methods Appl.
Mech. Engrg. 70 (1988) 27-58.
[33] A. George and J. Lui, Evolution of the minimum degree ordering algorithm, SIAM Rev. 31 (1989) 1-19.
[34] M.T. Heath and P. Raghavan, A Cartesian nested dissection algorithm, Technical Report UIUCDCS-R-92-1772, Department of
Computer Science, University of Illinois, Urbana, IL, 1992.
[35] G. Karypis and V. Kumar, Analysis of multilevel graph partitioning, Report 95-(137, University of Minnesota, Department of Computer
Science, Minneapolis, MN 55455, 1995.
[36] G. Karypis and V. Kumar, METIS: Unstructured graph partitioning and sparse matrix ordering system, Department of Computer
Science, University of Minnesota, Minneapolis, MN, 1995.
[37] Y. Saad and M.H. Schultz, GMRES: A generalized minimum residual algorithm for solving nonsymmetric linear systems, SIAM J. Sci.
Statist. Comput. 7 (1986) 856-869.
[38] J.M. Winget and T.J.R. Hughes, Solution algorithms for nonlinear transient heat conduction analysis employing element-by-element
iterative strategies, Comput. Methods Appl. Mech. Engrg. 52 (1985) 711-815.
[39] R.M. Ferencz, Element-by-element preconditioning techniques for large scale vectorized finite element analysis in nonlinear solid and
structural mechanics, Ph.D. Thesis, Stanford University, Stanford, 1989.
 S.M. RiJai et al. / Comput. Methods Appl. Mech. Engrg. 174 (1999) 393-417 417

[40J J.P. Grisval, C. Sauvignet and Z. Johan, Application of the finite element method to aeroelasticity, Euromech Colloquium 349,
DLR-Gottingen, Germany, 1996.
[41] J.P. Grisval, C. Sauvignet and Z. Johan, Aeroelasticity calculation using a finite element method, International Forum on Aeroelasticity
and Structural Dynamics, Rome, Italy, 1997.
[42] Centric Engineering Systems, Spectrum Theory Manual, Sunnyvale, CA, 1993.