15th ASCE Engineering Mechanics Conference

June 2-5, 2002, Columbia University, New York, NY

EM
2002

AN ALGORITHM FOR THE NONLINEAR DYNAMIC ANALYSIS OF

STRUCTURES IN A DISTRIBUTED COMPUTING ENVIRONMENT
Yunus Dere1
and Elisa D. Sotelino2

ABSTRACT
Several concurrent algorithms have been developed for transient analysis of structures in the past
fifteen years. In this work, the Modified Iterative Group Implicit Algorithm (MIGI), a domain-by-domain
parallel solution method is extended for nonlinear applications. For the nonlinear case, while MIGI iterates
to find compatible and equilibrating interface forces between the subdomains, a global nonlinear iteration is
also performed. The integration of these two iterations is investigated. More specifically, the two iterations
may be combined and performed in a single process loop or they may be isolated. It is found that the
efficiency of the procedure resulting from the isolation of the iterations is superior than the alternative. The
developed procedure has been tested via numerical examples. A 20-story model building considering
material and geometric nonlinearities under earthquake loading has been simulated. Excellent accuracy in
the solution and significant speedup values have been obtained from the simulations, which were
performed on an IBM-SP parallel machine.
Keywords: Iterative, Group, Implicit, Parallel, Concurrent, Dynamics

INTRODUCTION
Dynamic loads such as those produced by earthquakes and blast loading can cause dramatic
and tragic effects on society by causing loss of lives, property damage, and societal disruption.
Response to these types of loads can be found in a number of ways. The most versatile and
informative method to accomplish this consists of directly integrating the equations of motion
arising from a finite element analysis. In general, such a transient finite element analysis of real
structures is computationally intensive, especially when nonlinearities are present. Parallel
processing provides an excellent venue for achieving the additional computational power
necessary to successfully perform these simulations.
Several concurrent algorithms have been developed for transient analysis of structures in the
past fifteen years. Amongst these methods, the Iterative Group Implicit (IGI) algorithm (Modak
and Sotelino, 2000) has been considered as one of the most promising methods that take
advantage of parallel computing architectures. In this method, the solution is found on a
domain-by-domain basis and compatibility and equilibrium at the interface degrees of freedom is

1
School of Civil Engineering, Purdue University, West Lafayette, IN 47907, E-mail: ydere@purdue.edu
2
School of Civil Engineering, Purdue University, West Lafayette, IN 47907, E-mail: sotelino@purdue.edu
restored by means of an iterative procedure. The IGI algorithm has been implemented and tested
for linear structural dynamics applications. In the work by Dere and Sotelino (2001), it is shown
that for some applications the IGI algorithm has convergence and accuracy problems. The source
of these problems is found to be the approximation made in the distribution of the corrective
interface forces during the iterative procedure. In the original method, only the diagonal terms of
the effective stiffness matrix were used in this distribution. In addition, they found that the
mass-averaging rule required for the consistency of the GI algorithm is redundant in the IGI
algorithm, and they proposed a straightforward averaging of the displacements. The resulting
algorithm is called MIGI (Modified Iterative Group Implicit algorithm). The MIGI algorithm
provides accurate and reliable solutions. Furthermore, the performance of the method in
distributed computing environments has been found to be excellent producing near linear
speed-ups in large applications.
Nonlinear implicit transient analysis of structures is an even more challenging computational
task than its linear counterpart. This is because nonlinear schemes require frequent updating of
the structural stiffness matrix and of the solution of linear systems of algebraic equations.
Consequently, the utilization of a low cost distributed network or a dedicated parallel machine
becomes highly desirable, especially for structures with numerous degrees of freedom, such as
three-dimensional structural systems. In the present work, the MIGI algorithm is extended for
nonlinear applications. For nonlinear case, while MIGI iterates to find compatible and
equilibrating interface forces between the subdomains, a global nonlinear iteration must also be
performed. The integration of these two iterations has been investigated. These iterations may be
combined and performed in a single process loop or they may be isolated. Convergence studies
have been carried out to select the iteration scheme with the best performance. It is found that the
procedure developed by separating the two iterations performs much better than the alternative.

MIGI ALGORITHM
The first step of the MIGI algorithm consists of the decomposition of the structure into
subdomains, so that each element in the mesh belongs to a single subdomain. This partitioning
can be accomplished manually or using automatic decomposition algorithms. The nodes
connecting the elements at the subdomain borders are called interface nodes, and the degrees of
freedom (DOF) associated with these nodes are the interface DOF. The remaining DOF are
referred to as internal DOF. The internal DOF should be numbered first. This is because, in
present implementation of the MIGI, the internal DOF in the stiffness matrix are condensed out to
allow for faster interface iterations. The solution for the internal DOF is recovered from the
computed interface DOF results.
In a linear dynamic analysis step, MIGI obtains the solution for the interface degrees of
freedom by iterating until the force equilibrium and displacement compatibility are satisfied
simultaneously. When geometric and/or material nonlinearities exist in the structure, the solution
for each time step may require dynamic equilibrium iterations in which the structural elements
must update their state using the computed nodal displacements. As a result, internal
displacement recovery becomes necessary at each iteration. Due to the change in the internal
forces, effective interface force vector F̂H 0 (which is similar to ‘fixed-end-moment’ concept in the
Moment Distribution method) needs to be updated.
In addition to this, the subdomain stiffness matrix may require reassembling according to the
preferred iteration strategy (ex. Newton Raphson or Modified Newton Raphson). Due to the
stiffness reassembly, the subdomain effective stiffness matrix must be recondensed and the

2
subdomain distribution factor must be updated.
The issue of integrating the MIGI iterations into a nonlinear iteration has been addressed in
this work. Two different strategies have been developed for this purpose: the first is referred to as
Separated Iteration Procedure and the second is referred to as Combined Iteration Procedure.

Separated Iteration Procedure

In this strategy, the MIGI iterations are totally separated from the nonlinearity iterations. At
the beginning of each nonlinearity iteration, the correct interface displacements are found using
MIGI iterations, then the internal displacements are recovered from that.
The solution for a time step can be summarized as the following:

1. Estimate displacement, velocity and acceleration for all DOF.

2. Calculate the residual forces as the difference between the external and the internal
forces.
3. Start nonlinear iterations
a. Update the subdomain effective stiffness matrix if necessary. If updated,
recondense it and recompute the subdomain distribution factor.
b. Calculate effective interface force vector F̂H 0 from the calculated residual
~
forces assuming that the corrective interface force vector Fh is zero.
~
c. Assume Fh = − F̂H 0 for the subdomain and distribute the ‘global’ unbalanced
interface forces.
i. Start MIGI iterations
ii. Solve for interface displacements
iii. Enforce compatibility at the interfaces
iv. Compute incremental residual interface forces
v. Check interface displacement and force convergence, stop iterating if
convergence is achieved
d. Recover internal displacements from the computed interface displacements
e. Update the responses using the estimated responses and the incremental
displacements calculated from the residual forces. Then update the state of nodes
and elements.
f. Calculate the residual forces as the difference between the external and the
internal forces
g. Check for convergence of forces and displacements for internal DOF only, stop
iterating if the internal convergence is achieved.

Combined Iteration Procedure

In this strategy, a single iteration is performed that combines the MIGI iteration with the
nonlinear iteration. In this case, convergence is achieved when both the solutions for the interface
DOF and for the internal DOF converge. This combined iteration consists of executing the MIGI
iteration in the beginning of a nonlinear iteration. The procedure can be explained by referring to
the steps provided for the Separated Iteration Procedure. The combined procedure starts
performing the steps from 1 to 3.c. However, steps 3.c.i and 3.c.v are not performed in this case.
In step 3.g, the condition for exiting the nonlinear iteration loop must change to reflect that
internal and interface convergence must be reached simultaneously.

3
 When there are two subdomains, the case in which there is no need for a MIGI iteration, the
combined procedure and separated procedure are identical. However, in cases where there is no
need to iterate for internal convergence, but for interface convergence only, the computation
related to internal DOF is unnecessary. Through a number of numerical analyses of simple
structures using more than two subdomains, the combined iteration procedure displayed
insignificant speed-ups. As it will be discussed later in this paper, the separate iteration procedure
produces almost linear speed-ups.

NUMERICAL STUDY
The MIGI algorithm for nonlinear dynamic analysis has been implemented using an object
oriented approach and the C++ programming language. It has been integrated to an existing
object oriented framework, referred to Parallel Transient Finite Element (PTFE++) framework
(Sotelino and Modak, 1998). PTFE++ is one of the components of the Structural Engineering
Software Development Environment (SECSDE), which is being developed in the School of Civil
Engineering at Purdue University (Sotelino et al. 1992). The algorithm has been implemented
such that only one processor may be assigned per subdomain. The algorithm has been developed
and tested in a network of UNIX workstations and successfully ported to IBM-SP multi-computer
with no changes.

Number of elements: 347

Number of nodes: 243 (3 DOF per node)

4
5 6
19@13' 2
3
4

2
2 3
1
18'
1
1
2@12'

5@20'

FIG 1. The elevation of the 20-story frame and partitioning into subdomains

The 20-story benchmark-building frame whose elevation is shown in Fig. 1 has been analyzed
using the developed separated iteration procedure. The detailed properties (ex. section properties,
loadings, etc.) of the building can be found in Gupta et al. (1999).
Two elements per each column and beam have been assumed in the construction of the finite
element mesh of the frame. Both geometric and material nonlinearities are considered in the
analyses. Since the nonlinear elements used in the analysis (Rajagopala, 1997) are based on

4
distributed plasticity at the gauss integration points, each element was divided into 6 segments
internally to capture any possible occurrence of material nonlinearity close to the beam-column
connections. A bi-linear isotropic hardening model with a very low plastic modulus (103 psi) has
been assumed for all elements.
The frame is partitioned into subdomains using MPE++ (Hsieh et al., 1998), which is an
object-oriented automatic mesh-partitioning framework. The mesh is partitioned into 2, 4 and 6
sub-domains for the simulations. The partitioning results are illustrated in Fig. 1. The time
elapsed during the partitioning was trivial (~2 to 3 seconds).
The analyses have been carried out on the IBM SP multi-computer located in the Purdue
University Computer Center. A time step size of 0.01 seconds is used in all of the analyses. This
time step is selected based on preliminary testing of convergence for a sequential nonlinear
analysis. The Modified Newton Raphson iteration strategy has been adopted. The dynamic
nonlinear analysis of the frame has been performed using the East-West component of the
Northridge-Newhall earthquake acceleration record. The structural masses are assumed to be
lumped at the nodes. The force and moment convergence tolerances are selected to be 10-3.
For comparison, a linear-elastic analysis of the same example frame has been performed. The
horizontal displacement time history for a node located in the middle of the top story has been
shown in the Fig. 2. Irrespective of the number of subdomains, the results for the nonlinear
response are found to be almost the same (max error 10-4%) in all cases
Horizontal
displacement (in)
30

20

10

0
0 2 4 6 8 10 12 14 16 18 20
-10

-20
MIGI-Nonlinear
Linear-elastic
-30
Time(secs)

FIG. 2. The top story horizontal displacement time history responses

The iteration statistics for the selected subdomain configurations are given in Table 1. As it is
seen from this table, the number of iterations for the interior DOFs are the same or very close to
each other irrespective of the number of subdomains. Thus, it can be concluded that the interface
problem did not really affect the convergence of the interior DOF. The fast convergence

5
capability of the algorithm can be seen from the number of interface iterations. However, it
should be noted that the number of interface iterations depends on the time step size adopted.
Smaller time step sizes would tend to cause less interface iterations.
Table 1. Iteration* statistics

Interface iterations
Processors Internal Maximum Average
iterations (per internal iter.)
1 2564 − −
2 2564 0 0
4 2570 4 1.62
6 2564 6 4.2
*Iteration, here, is defined as the number of non-convergent solution trials.

The performance of the MIGI is evaluated using the speed-up and efficiency measurements,
which are defined by
t sequential
Speed − up =
t parallel
(1)

Speed − up
Efficiency = x100% (2)
Np

In Fig. 3, the speed-up graph is shown. As it is seen from this figure, the achieved speed-up’s are
very close to the ideal linear speed-up line. The efficiency for 6-subdomain solution was ~84%.
This efficiency is expected to deteriorate as the number of subdomains increases, due to the likely
increase of number of iterations as well as to communication overhead. The comparison between
communication time and computational time for different number of processors is shown in Fig.4.
The communication time usually increases as the number of processors increases, and the ratio of
computation/communication time decreases. Therefore if the problem size is not large enough,
the communication overhead may deteriorate the performance when large number of processors
are adopted.
The solutions obtained using the MIGI algorithm with varying numbers of subdomains are
found to be practically identical. For the 20-story building, the maximum error measured in the
displacement results between the solutions obtained with 6-subdomains using MIGI and the
sequential solution was 10-4%, which is insignificant.

CONCLUSION
The MIGI algorithm has been extended for transient dynamic nonlinear analysis and its
efficiency has been investigated through numerical analyses. The integration of the MIGI
iteration with nonlinear equilibrium iteration has been successfully achieved by separating the
MIGI iteration and the nonlinear iteration.

6
 Speed-up
6
Linear speed-up

Actual speed-up
5

1
1 2 3 4 5 6
Number of processors

FIG. 3. The speed-up obtained from different processor configurations

Communication time

Computation time

1 2 4 6
Number of processors

FIG. 4. Distribution of communication and computation times

7
 The performance of the MIGI algorithm has been tested for a real-world example, a 20-story
steel model building frame which was designed for the city of Los Angeles. The MIGI algorithm
converged to accurate (compatible, in equilibrium) interface solution in a few number of
iterations and the convergence of the internal solution almost unaffected by that. The accuracy of
the results obtained with different number of subdomains is found to be very satisfactory for the
problem analyzed. Significant speed-up’s have been achieved when compared to the sequential
version. The algorithm is stable as long as the integration scheme adopted at the subdomains is
stable.

REFERENCES
Dere, Y. and Sotelino, E.D. (2001), “Convergence of the Iterative Group-Implicit Algorithm for
Parallel Transient Finite Element Analysis”, Proceedings of the Eighth International
Conference on Civil and Structural Engineering Computing, Vienna, Austria, September 19-21,
2001.
Gupta, A. and Krawinkler, H. (1999), “Seismic Demands for Performance Evaluation of Steel
Moment Resisting Frame Structures”, SAC Task 5.4.3, Report No. 132, The John A. Blume
Earthquake Eng. Center, Dept. of Civil and Env. Eng., Stanford University.
Hsieh, S.H., Yang, Y.S., Cheng, W.C., Lu, M.D., Sotelino, E.D. (1998), “MPE++: An
Object-Oriented Mesh Partitioning Environment in C++”, Proceedings of the Sixth East
Asia-Pacific Conference on Structural Engineering and Construction, Taipei, Taiwan, January
14-16, 1998, 313-318.
Modak, S., Sotelino, E.D. (2000), “The Iterative Group Implicit Algorithm for Nonlinear
Structural Analysis", International Journal for Numerical Methods in Engineering 47(4),
869-885.
Rajagopala, M.G. (1997), “Object-Oriented Parallel Second-Order Inelastic Analysis of Frames”,
Ph.D. thesis, Purdue University, West Lafayette, Indiana, USA.
Sotelino, E.D., White, D.W., and Chen, W.F. (1992), “Domain-Specific Object-Oriented
Environment for Parallel Computing", Journal of Singapore Structures Steel Society, 3(1),
47-60.
Sotelino, E.D., Modak, S. (1998), “A Programming Framework for Parallel Transient Analysis of
Building Structures”, Proceedings of the 1998 World Structural Congress (in CD-ROM), San
Francisco, CA, USA, July 18-23, 1998.