Proceedings of the ASME 2015 International Design Engineering Technical Conferences &

Computers and Information in Engineering Conference

IDETC/CIE 2015
August 2-5, 2015, Boston, MA, USA

DETC/CIE2015-46476

TOWARDS REAL-TIME COMPOSITE MATERIAL CHARACTERIZATION USING

SURROGATE MODELS AND GPGPU COMPUTING
John Steuben Athanasios Iliopoulos
John G. Michopoulos Computational Materials Science Center
Computational Multiphysics Systems Lab George Mason University, Fairfax VA 22030, USA
Center of Computational Material Science Resident at Center of Computational Material Science
Naval Research Laboratory Naval Research Laboratory
Washington DC 20375, USA Washington DC 20375, USA

Cameron Turner
College of Engineering and Computer Science
Colorado School of Mines
Golden CO 80401, USA

ABSTRACT
In this paper we address the particular need for high-speed or
”real-time” characterization of realistic anisotropic material sys-
tems such as laminated composites. This is driven by the desire
to dynamically alter the loading paths applied by a multiaxial
robotic test frame during the testing of a specimen, so that strain
states are developed in the specimen in a manner that activates
the maximum excitation of the specimen’s constitutive proper-
ties. In order to achieve this goal, we present an evolutionary
adaptation of earlier work into computationally efficient material
characterization using response-surface surrogate models. This
approach is enhanced by the adoption of highly-parallel Gen-
eral Purpose Graphics Processing (GPGPU) computing. We dis-
cuss the challenges of adapting the characterization problem for
GPGPU computing, particularly in terms of parallelization, syn-
chronization, and approximation. Two parallelized algorithms
for characterization are developed, and the merits of each are
discussed. We then demonstrate validation results on a simple
linear-elastic material system, and present statistical data which
demonstrate the robustness of the approach in the presence of ex-
perimental noise. We conclude with remarks regarding the per-
formance of the GPGPU-enabled characterization algorithm, and
its applicability to more complex material systems.
INTRODUCTION
Composite materials play an important role in many engineer-
ing design and manufacturing disciplines. As the usage of these
materials continues to grow, the development of computational
tools that allow deeper understanding of these materials and their
responses is necessary. The problem of characterization is par-
ticularly important, both to analyze the properties of novel ma-
terial systems and the as-built properties of more conventional
analogs. One of our current research needs is the development of
“real-time” characterization algorithms which can compute the
properties of test specimens at least several times per second.
The development of real-time characterization is desirable as it
will allow multiaxial robotic test frames, such as the “NRL66.3”
that was custom-designed at the Naval Research Laboratory [1,
2] shown in Fig. 1, to apply dynamically varying direction load
paths in order to properly excite all of the constitutive parame-
ters of the specimen undergoing testing. In this context we use
the term “excite” to refer to the development of strains states in
the test specimen which produce high sensitivity in the mate-
rial’s response relative to its constitutive properties. This feature
will essentially remove the requirement to use one specimen per
linear loading path as it will enable a non-linear but optimized
loading path and therefore will lead to lower numbers of ma-
terial specimens than the ones used under the current practices,

1
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.

FIGURE 1. The NRL 66.3 six degree-of-freedom test frame. This
fully automated test frame may apply any combination of forces and
torques to the test specimen.
and therefore will further reduce the cost of automatic material
characterization.
Furthermore, in order to address the challenge of real-time
characterization, we extend our previous research [3] on the
subject of fast characterization using surrogate (or approxima-
tion) models. In order to realize increased computational per-
formance we explore the use of massively-parallel General Pur-
pose Graphics Processing Unit (GPGPU) computing to acceler-
ate the surrogate-enabled characterization approach. This com-
putational technology was selected for its comparatively high
performance at relatively low expense. The nature of GPGPU
computing requires the careful consideration of parallelization,
or the decomposition of the characterization task in to work units
which can be completed simultaneously, along with the asso-
ciated requirements of synchronization and communication be-
tween these parallel processor units. Additionally, the approx-
imation methods employed for characterization may also pose
different parallelization or synchronization challenges and must
be wisely chosen.
In the following section, we begin by introducing the char-
acterization problem, and related prior research on this subject.
We proceed, after that, to discuss surrogate models, and previous
2
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
research on their use in materials characterization. In the sub-
sequent section, we discuss the opportunities for parallelization
of the characterization problem. Expanding on this theme, we
then discuss the topics of parallelization, synchronization, and
approximation in-depth, and develop a pair of parallel algorithms
for characterization. Following this, we give some sample results
from the characterization of a linear-elastic specimen in real-
time. Statistical analyses of these results are presented to demon-
strate the robustness of the approach in the presence of experi-
mental noise. In a final section, we present our conclusions re-
garding GPGPU-accelerated material characterization, and focus
in particular in the immediate extension of the proposed method-
ology to examine more complex physical systems.
THE CHARACTERIZATION PROBLEM
When the properties of a material system are unknown, but ex-
periments can be made to measure the response of the material
system under controlled loading, then one faces the challenge of
solving the continuous or discrete form of the partial differen-
tial equations (PDEs) equations describing the behavior of the
system inversely, aiming into the determination of the material
parameters. This is the reason that the mathematical activity for
achieving this is called inverse solution of the elastostatic prob-
lem, or constitutive properties characterization.
When the form of the constitutive model is assumed to
be known (in our case the constitutive model is that of linear
anisotropic elasticity), the goal of material characterization is
to calculate the values of the unknown constitutive parameters,
which in the materials community are known as the constitu-
tive properties. Decades of advances in force, displacement, and
strain measurement techniques in general, and full-field strain
measurement techniques in particular, have enabled the collec-
tion of very rich data sets from mechanical tests [4-14]. The
availability of such data both inspires and enables an integrated
approach that combines experimental with numerical methods
to recover the elastic parameters. With this strategy, the elas-
tic constants can be recovered by minimizing an objective func-
tion, expressing some vectorial measure of the distance (usually
the L2 norm) between the full experimental measurements and
the corresponding analytical model predictions. This is usually
achieved via an iterative optimization procedure [15-23]. Since
the numerical model that represents the elastostatic PDEs usually
takes the discretized form of a Finite-Element Analysis (FEA)
that exploits the constitutive response of the model, this opti-
mization procedure has been found to be computationally costly
[2, 23].
In order to improve the computational efficiency of ma-
terial characterization, alternative approaches have been devel-
oped. Virtual field methods [24, 25] avoid the use of FEA sim-
ulation by finding virtual fields and inversely solving for param-
eters by substitution of full-field measurements. The weakness
 ConductfPhysical Experimental
Experiment Dataf
InversefCharacterization
Constitutive AdjustfConstitutive
properties Properties No
EvaluatefSystem Model
Converged?
Modelf Data
Yes
IdentifiedfConstitutive AcquirefConstitutive
Properties Properties
FIGURE 2. Illustration of the FEA-in-the-loop numeri-
cal/experimental method for material characterization.
of this method is that it is limited to cases where such fields
are derivable. Another recent technique reduces the problem
to that of finding the pseudoinverse of a matrix formed using
the forward material model by taking advantage of the full-field
strain measurements achieved via the Meshless Random Grid
Method (MRGM) [26]. Energy-based characterization methods
using total and dissipated strain energy density determination
[27,28], that take advantage of the total potential energy [29]
have also been demonstrated. Nonparametric constitutive mod-
els of anisotropic materials have also been used to identify elas-
tic constants. The use of stochastic corrections to parameter esti-
mates using Kalman filtering is shown in [30]. Another approach
based on the use of artificial neural networks [31] is presented in
[32] and further demonstrated in [33].
In the present work we focus on the acceleration of the FEA-
in-the-loop characterization approach (as opposed to defining a
new method for characterization) as it is conceptually straight-
forward. Figure 2 illustrates this technique: a physical experi-
ment produces an experimental dataset and is associated with a
material system model (finite element or otherwise) of the tested
specimen. Initial constitutive parameters for the material system
are estimated using any a priori knowledge available. From this
initial estimate, an optimization algorithm iteratively adjusts this
model until the model response matches the experimental data.
Once this has been achieved, the model parameters are extracted
and reported.
CHARACTERIZATION USING SURROGATE MODELS
One approach to accelerating inverse material characterization
is to remove the computationally expensive combination of the
FEM model with the constitutive model from the optimization
loop and replace it with an efficient surrogate, as seen in Fig.
3. Generally speaking, surrogate models are computationally
3
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
efficient but physics-agnostic representations of more expensive
physics-specific models which are generated by sampling the un-
derlying model and subsequently fitting some functional form to
the sampled data. They can also be conceptualized as a model
of another model. This is the reason the term metamodel is fre-
quently found in the literature, especially in the field of design
engineering. We avoid the use of this term however, as it conflicts
with an existing definition in the field of mathematical logic.
A wide variety of surrogate models exists. They can be
broadly grouped into three families [3, 34]. Rigorous compar-
isons of the different aspects of these surrogate modeling tech-
niques are presented in [34, 35]. Generally, there are tradeoffs
associated with the selection of a surrogate model type; for in-
stance, a technique capable of approximating highly non-linear
data may be more computationally expensive than other alterna-
tives. In this work we use Response Surface Model (RSM) sur-
rogates of material constitutive models composed with the shape
of structural components. This choice is driven by the relatively
compact and computationally efficient nature of these models. In
the next section, we turn to the discussion of how these surrogate
models are used to characterize materials.
Surrogate Characterization Methodology
Our characterization approach employs a mixed experimental-
numerical methodology. We assume a known model of the ma-
terial system that operates over a physical domain Ω ⊂ R3 (i.e.
a test specimen such as that shown in Fig. 4), and we approach
characterizing this system as the task of determining a set of in-
dependent constitutive parameters in a such that,
tε (xt , ft ) + ε e + ε r = cε (a, xc , fc ) + ε m , (1)
where tε is a vector field function that collects experimen-
tally observed strain tensor components at a set of points
xt ∈ So ⊆ Ω ⊂ R3 and cε is likewise a system model vector
function of strain tensor components, which are functions of
a constitutive model. The specimen’s response is sampled at
locations xc ∈ So0 ⊆ Ω0 . The domain Ω0 is the computational do-
main the system model evaluates into. The subdomains So and
So0 serve the purpose of providing an optimization space for the
inverse problem that is compatible with the capabilities of exper-
imental measurement techniques. The parameters fc dictate the
specific loading of the test specimen . Given a well-designed
experimental/numerical setup, it is reasonable to assume that xt
can be taken to coincide with xc . Similarly, the loading of the
experimental and numerical forms can be considered to be ap-
proximately equal ft ≈ fc . The terms ε e , ε r , and ε m represent
respectively systematic experimental errors, random experimen-
tal errors, and errors due to system model formulation. In a well
designed experiment and a properly constructed computational
 Conduct?Physical Experimental
Experiment Data?

Preprocessing? Inverse?Characterization
Evaluate?System
Constitutive Adjust?Constitutive
Model?
Properties Properties No
Sampled
Data Evaluate?Surrogate Model
Converged?
Model? Data
Construct?Surrogate
model Yes

Surrogate?Model Identified?Constitutive Acquire?Constitutive

Description Properties Properties

FIGURE 3. Illustration of the FEA-in-the-loop numerical/experimental method for material characterization.

Material domain (Ω) 4. does not require a priori assumptions on the values of ā or
Domain surface (S) xi the structure of cε .
y
Observable surface
region (So) If the physical experiment, underlying model, and surrogate
Observable x model are all well designed, then all of the error terms will be
z
region boundary small. Combined with the choice of common measurement loca-
Direction of major tions, geometry and loadings between the experiment and model,
orthotropic axis Multiple-ply layup Eq. (4.9) can be expressed compactly as

FIGURE 4. Illustration of the test specimen domain. mε (ā, x) u tε (x) (3)

system model, these errors should be relatively small in magni-
tude.
We note at the outset that cε is typically computationally ex-
pensive to formulate and to evaluate. In some simple cases, ana-
lytical constitutive models are available, but cε is more typically
formulated via the solution of a FEA model. In order to achieve
a more computationally efficient characterization, we introduce
a surrogate model, mε , of the original strain response model cε
that involves the original constitutive model. Therefore, Eq. (1)
becomes,
tε (xt , ft ) + ε e + ε r = mε (ā, xc , fc ) + ε m + ε s
where ā are the approximated material parameters of the surro-
gate model and ε s denotes additional error introduced by the sur-
rogate model. Successful characterization depends on construct-
ing mε such that it
1. represents cε sufficiently well, i.e. ε s is small;
2. can be evaluated with minimum computational expense;
3. is robust with respect to experimental noise in the sampling
of tε , i.e. increases in ε e or ε r have little effect on the results
produced;
Definition of the Surrogate Model
The development and definition of the surrogate model are
largely driven by experimental considerations. We assume that
the test specimen is represented by a domain Ω ∈ R3 with a
surface boundary S. Since mechanical elements of physical test
apparatus (e.g. the grips) typically do not permit experimental
measurements on the entirety of S, we define the observable sub-
set of So ⊂ S as shown in Fig. 4. The experimental measurement
locations are thus restricted to x ∈ So .
Because FEA based system models evaluate to a set of do-
main nodes we also assume that our sampling occurs on a dis-
(2) crete number of such points represented as vectors xi ∈ So , i =
1 . . . nm . Consequently, our data consists of a set of nm individual
surrogate models, mεi , each of which approximates cε at a single
point i represented by a vector xi .
mεi (ā) = mε (ā, xi ) , i = 1 . . . nm (4)
The purpose of each of these surrogates is to generate a vector
of scalars measurable at a point on the material specimen, that
depend on the constitutive parameters. We have decided to use
a full field strain measurement methodology and therefore, con-
struct these surrogates to consist of the measured surface strain

4
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
tensor components of the form,
T
i
i i
mεi (ā) = m εxx , m εyy , m εxy , i = 1 . . . nm
Similarly, we define a set of experimental observations at corre-
sponding locations,
ti = t (xi ) , i = 1 . . . nm ,
where the individual observations ti have the same form as mi ,
i i i T
ti = t εxx , t εyy , t εxy , i = 1 . . . nm
Construction of the Surrogate Model
The primary challenge of constructing each of the mεi in Eq.
7 is the construction of each of the individual surrogate mod-
els mε such that they reproduce the behavior of cε for a fixed
load/geometry at a predetermined point. In order to do so, it is
necessary to sample or evaluate cε at combinations of values for
the elements of a. We use a space filling grid technique to sample
cε : each element of a is varied in even increments between some
minimum and maximum value. This technique does require a
priori determination of upper and lower bounds of the constitu-
tive parameters, but in practice these bounds are very loose and
can span several orders of magnitude.
For each of the nodes xi ∈ So the system model c is sampled
for l combinations of the constitutive parameters aq , q = 1 . . . l.
Depending on the dimensionality of the vector space of a this
sampling can be performed on an 1D line a 2D planar grid, a
3D cubic lattice or a higher dimensional hyperrectangle lattice.
This sampling results in a set of vector valued elements s j (ak )
that correspond to the response of cε at a particular location on
So with a predetermined combination of constitutive material pa-
rameters and loading conditions.
Once c has been sampled, there are several techniques which
could be employed to fit the surrogate models to this data. We
use a direct method which produces an exact interpolation of the
acquired data. In the case where aq ∈ R1 (hence aq = aq ) this can
be formulated as the solution to a set of simultaneous linear equa-
tions. Once this linear system has been formed, it may be solved
using any technique. Typically these systems are small; involv-
ing no more than a few thousand equations. In this research we
use routines built into the Mathematica computer algebra system
to invert the linear system and fit the surrogates.
Characterization by Optimization
As indicated earlier, the characterization process can be formed
as an inverse problem that seeks to minimize the difference be-
tween the experimentally observed and computationally calcu-
lated strain tensor components on the surface of a specimen.
5
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
Based on Eq. 3, such a problem can be formulated as,
Z
(5) min h (ā) = kmε (ā, x) − tε (x)k dS (8)
So
However, we recall that mε decomposes into nm surrogate mod-
els at particular locations i . If cε is built by a finite-element
model, then we take xi to be the vectors that represent the nodal
(6)
locations of that model that lie within So . If cε is built by a
closed-form analytic solution, then xi can be chosen arbitrarily,
e.g. a radial pattern of samples on a plate-with-hole test speci-
men. Recalling the recasting of t from Eq. 6 and m from Eq. 4,
(7) the minimization problem of Eq. 8 can be expressed in a discrete
form as,
nm
min. f (ā) = ∑ kpi (ā) − ti k
i=1 (9)
s.t. amin j ≤ ā j ≤ amax j ∀ j ∈ 1 . . . na
where na is the number of constitutive parameters (i.e. ā ∈ Rna ).
Figure 5 demonstrates this algorithm in a visual fashion. The
presence of experimental noise complicates efforts to analyze
the properties of f . In practice, any reasonably powerful non-
linear optimization routine can be employed. In the previous re-
search we have elected to use either simulated annealing [36,37]
or Nelder-Mead downhill simplex algorithms [38-39] which do
not require derivative information.
OPPORTUNITIES FOR PARALLELIZATION
Because the material characterization algorithm defined previ-
ously is a relatively complex multi-stage computation, there are
many avenues by which the computational effort may be di-
vided into simultaneously executable sub-units. Our focus was
on the development of strategies appropriate for GPGPU com-
puting, which ideally employs a large number of parallel “com-
pute threads” or simply “threads.” This decision was driven by
the attractive price to performance ratio offered by this technol-
ogy as well as its increasing ubiquity. Given the choice to focus
on GPGPU technology, there are five key areas where paralleliza-
tion may occur:
1. The computation of each evaluation of the system mode.
2. The computation of batches of system model evaluations si-
multaneously.
3. The fitting of each surrogate mode.
4. The fitting of groups of surrogate models simultaneously.
5. The computation of the objective function for the optimiza-
tion process used to achieve the inverse solution, as given in
Eq. 9.
Each of these opportunities is discussed below.
 Constitutive1Model1 Experimental1Full-Field
(e.g.1FEA1mesh) Strain1Measurements
Sampling1Locations εxy
xi1(e.g.1FEA1nodes)
εyy
εxx

nm
min.11∑||mi(a1,1a2)1-1ti||1
Surrogate1Models1(mi) i=11

εxx εyy εxy

a2 a2 a2
a1 a1 a1 a2
a1

FIGURE 5. Visual demonstration of the characterization algorithm.

Evaluating the System Model Each sample of the system
model presents an opportunity for parallelization. At the present
time, and for most material systems, a forward model evaluation
is equivalent to a finite element analysis. This requires the inver-
sion of linear system which is potentially very large. The possi-
bilities for GPU computing for this application have already been
well-explored (e.g. [40,41]), and are already offered as features
in most commercial FEA packages. For this reason, we chose
not to pursue further research in this direction.
Computing Evaluation Batches As discussed previously, the
system model is evaluated many times for different combinations
of constitutive material parameters. Because these system model
evaluations are independent, it is possible to conduct them in
parallel. However, because each evaluation is computationally
complex, most GPU architectures will be constrained by mem-
ory capacity to conducting only a few simultaneous evaluations.
For this reason, it seems that this opportunity for paralleliza-
tion is better exploited by traditional multi-computer clusters in
a distributed memory setting. Many commercial FEA solvers al-
ready offer this feature, and allow “parametric sweeps” to be dis-
tributed across a cluster of workstation or server computers [42]
where each node is responsible for the evaluation of the system
model for a particular subset of constitutive parameter values.
For these reasons we chose to pursue the rest of the areas of op-
portunity as enumerated earlier. It should be noted, however, that
the latest generation of GPU hardware features greatly increased
memory capacity. As these devices decrease in cost their use for
computing evaluation batches will become competitive.
Fitting Surrogate Models As outlined in the previous section
the surrogate models must be fit to the data produced by evalu-
ating the system model by solving a system of linear equations.
The solution of linear systems on the GPU is an extensively re-
searched application of GPU computing [43,44], and suitable al-
gorithms have reached a high level of sophistication. As before,
this encouraged us to study other avenues of parallelization.
Batch-fitting Surrogate Models Just as the possibility for
computing system model samples in parallel exists, so does the
possibility of fitting the surrogates in parallel. In this case the
results of all evaluations of the system model must be collected
at every point in So corresponding to a surrogate model. As be-
fore, the limited memory resources on the GPU do not encourage
parallelization in this fashion.
Objective Function Computation From the perspective of the
broader goal of this research, the four methods of parallelization
mentioned earlier do not address the goal of achieving real-time
characterization. These four steps are effectively preprocessing
operations which can be computed “off-line” before physical ex-
perimentation begins. While GPU computing may be used to ac-
celerate this preprocessing, especially in the system model eval-
uation, we elected to focus upon the fifth opportunity enumer-
ated previously: the computation of the objective function for
optimization in order to achieve the inverse solution. This led
us to the development of a fast parallel implementation of the
optimization algorithm used to recover the material constitutive
parameters.

6
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
 CPUNPreprocessing

CPUNPreprocessing
SampleNNMaterialN SampleNNMaterialN
START START
SystemNModel SystemNModel

EstimateNItitial EstimateNItitial
MaterialN ConstructN/NFit ConstructN/NFit
MaterialN SurrogateNModels
PropertiesN SurrogateNModels
PropertiesN

GPUNParallelNOptimization
GPUNParallelNEvaluationN

ComputeN
ComputeN
ComputeN ComputeN
ComputeN
ComputeN
ComputeN ComputeN
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
atNnodeNi atNnodeNi

ComputeN
ComputeN
ComputeN ComputeN
ComputeN
ComputeN
ComputeN ComputeN
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
atNnodeNi atNnodeNi

ComputeN
ComputeN
ComputeN ComputeN
ComputeN
ComputeN
ComputeN ComputeN
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N
Obj.NFxn.N Obj.NFxn.N
atNnodeNi atNnodeNi

Σ Σ
CPUNPostprocess CPUNOptimization

CPUNPostprocess

Adjust Store
Converged? N MaterialN Material
PropertiesN PropertiesN

END
Store
Material END
PropertiesN

FIGURE 6. Two possible options for the implementation of the GPGPU-accelerated characterization algorithm. At left, a concept termed “GPU
Parallel Evaluation” (GPE) is shown, while at right another concept termed “GPU Parallel Optimization” (GPO) is given.

PARALLELIZATION, SYNCHRONIZATION & APPROXI-
MATION
Upon consideration of the topics above, it is clear that a focus
on the inversion problem itself will be of the greatest utility. As
expressed in Eq. 9, the inversion of the forward material sys-
tem model is achieved by the use of a non-linear optimization
algorithm which is used to discover the global minimum of an
objective function. This objective function is itself the summa-
tion of many terms, each corresponding to a sampling location
xi within the domain So as illustrated in Fig. 5. The develop-
ment of the GPGPU-accelerated characterization algorithm was
conducted with great care given to the relationships between par-
allelizaton, synchronization, and approximation. The algorithm
developed is termed the GPU-Accelerated Material Mechanics
Analysis (GAMMA). It is outlined in Fig. 6, and explored in
further detail herein.
In terms of implementation, GAMMA is principally writ-
ten in C++. The CUDA framework [45-47] was used to imple-
ment this algorithm in order to achieve good computational per-
formance on the GPU. However, this choice sacrifices platform
interoperability, as CUDA-enabled codes will only run on graph-
ics hardware manufactured by NVIDIA.

7
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
Parallelization The parallelization concept used to implement
GAMMA is based on spatial discretization. A paradigm of one
thread per term of the objective function is used. This also corre-
sponds to a concept of one thread per sample location, as each
sample location contributes one distinct term to the objective
function. Given typical formulations of the forward problem, the
number of threads is typically no greater than 105 . This paradigm
does not enforce a specification for the grouping of threads into
thread blocks. As a result, the computational grid is initialized
in the simplest fashion: a one-dimensional grid of thread blocks
each consisting of a one-dimensional grid of 64 threads. The
thread block size is selected in a semi-arbitrary fashion, as the
computational performance does not appear to be particularly FIGURE 7. A pair of objective functions with two independent vari-
sensitive to this parameter. While this parallelization paradigm ables corresponding to the horizontal axes, and one dependent variable
is conceptually straight-forward, and indeed seems to be the only corresponding to the vertical direction. At left, a poorly conditioned ob-
option readily at hand, it has serious consequences in terms of jective function is shown, while at right a well conditioned example is
synchronization needs. given.
It should also be noted that GAMMA utilizes previously-
developed parallel evaluation techniques when possible. If a
scheme was implemented and tested. It was immediately found
GPU-enabled finite element solver is available, for instance, it
to produce inaccurate or incorrect results during synthetic exper-
is employed to collect the initial samples of the system model.
imental benchmarking. This aberrant behavior was explained by
Additionally, surrogate models are fit to the sample data using a
plotting the objective functions on which each of the optimiz-
traditional Message Passing Interface (MPI) framework for dis-
ers were operating. Examples of these are shown in Fig. 7. It
tributing the workload across all available CPU cores.
was found that a large number of the objective functions in the
GPO formulation were poorly conditioned, and did not exhibit
Synchronization The one thread per term parallelization
a single minimum corresponding to the true material properties.
paradigm clearly creates a situation where synchronization is re-
Therefore, the numerical optimization routine produced incorrect
quired. In particular the summation of the terms to compute the
results, which when averaged resulted in poor performance.
entire objective function is problematic, and has the potential to
As a result of the problems found in the GPO formulation,
collapse the parallelization of the problem by forcing threads to
the GPE algorithm was ultimately selected for use. In order to
serially access an accumulator variable. In order to avert a pos-
mitigate the effects of repeated synchronizations a parallel re-
sible loss of performance, a pair of schemes were investigated.
duction algorithm was implemented. The parallel reduction al-
These were termed GPU-Parallel Evaluation (GPE) and GPU-
gorithm implemented is outlined in Fig. 8, and discussed in much
Parallel Optimization (GPO). Both possibilities for addressing
greater detail in [48]. All possible options for optimizing this al-
synchronization needs are shown in Fig 6.
gorithm were employed, including sequential addressing of data
The GPE algorithm is based on the concept of evaluating
in fast shared memory and full loop unrolling. Further optimiza-
the terms of the objective function in parallel, accumulating their
tion of the GPE algorithm was conducted by the careful consid-
sum, and then passing the objective function value back to the
eration of the approximation systems in use.
Central Processing Unit (CPU). The CPU is then responsible for
computing the adjustments to the material parameters based on
Approximation GAMMA involves several layers of approxi-
a predetermined optimization scheme. This is the most conven-
mation. The first has already been mentioned, and consists of
tional and cognitively straightforward option, but it does require
the sampling of the system model response at a finite set of
a synchronization at every iteration of the optimization solver.
points. This approximation is tied very tightly to the paralleliza-
In order to avoid repeated synchronization, an alternate GPO
tion scheme outlined previously. There is another explicit ap-
scheme was investigated. In this algorithm, the optimization loop
proximation employed, which consists of the set of surrogate
for each point in the domain So is conducted separately in paral-
models employed to interpolate the system model response. In
lel. In essence, each optimization loop is recovering the material
the previous research described earlier, NURBs-based surrogates
properties at one point in the domain, and the bulk properties
were employed. However, in the process of visualizing the ob-
are thus computed by averaging all of these point computations.
jective functions mentioned in the previous section, it was seen
This seems promising from a synchronization standpoint, as each
that the design spaces were not strongly nonlinear. This implies
optimizer loop can run completely independently with only one
that the choice of NURBs as an interpolator is non-ideal.
synchronization operation required in total. As a result, the GPO

8
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
DatalValues 5 1 1 2 1 0 2 5 0 9 8 1 5 1 2 4 ... InputlDatalVector to all other memories on the GPU. Because only a few thousand
ThreadlIDs 0 1 2 3 4 5 6 7 Stridel=l8 threads are running on the GPU, it is important that the occur-
rence of stalls when waiting for memory access be minimized.
5 10 9 3 6 1 4 9 0 9 8 1 5 1 2 4 ReductionlStepl1
Since the RSM can be stored in a very fast memory, this choice
0 1 2 3 Stridel=l4 of approximation has a large impact on the performance of the
11 11 13 12 6 1 4 9 0 9 8 1 5 1 2 4 ReductionlStepl2
characterization algorithm.

0 1 Stridel=l2

24 23 13 12 6 1 4 9 0 9 8 1 5 1 2 4 ReductionlStepl3 PRELIMINARY RESULTS

0 Stridel=l8
In order to demonstrate the GAMMA algorithm’s correct imple-
mentation and computational performance, a set of synthetic ex-
47 23 13 12 6 1 4 9 0 9 8 1 5 1 2 4 ReductionlStepl4 periments was conducted. The primary goal was to establish the
Result accurate recovery of constitutive parameters using GAMMA, es-
pecially in the presence of experimental noise.
FIGURE 8. Illustration of the optimized parallel reduce algorithm Towards this end, the following procedure was employed:

1. Begin: build the surrogate model mεi of the constitutive

The great strength of NURBs-based surrogate models is
their ability to represent extremely non-linear data in a reason-
ably compact recursive form. However, an initial observation
of the objective functions computed for the material character-
ization problem revealed functions which resembled low-order
quadratic polynomial functions. Based on this observation, the
choice was made to switch from NURBs to RSMs in order to
formulate the surrogate models. RSMs have a simple compact
representation
n
y (x) = ∑ βi xi
i=0
which is equivalent to a multidimensional interpolating polyno-
mial. The order of the polynomial is typically limited to n = 3
to avoid the oscillatory behavior associated with Runge’s phe-
nomenon. As before, the coefficients of the surrogate model are
computed using preexisting routines in the Mathematica com-
puter algebra tool. However, because of the more compact form
of Eq. 10, relatively few parameters are required to describe each
surrogate model. This has two important effects on the GPU im-
plementation.
1. It reduces the overhead associated with transferring the sur-
rogate models to the GPU, and also reduces total memory
requirements
2. It allows the surrogate model coefficients to be stored in
“register” memory, which is very fast compared to the larger
GPU memories
The second of these effects is the most important, with re-
spect to the goal of achieving real-time characterization. The
register memory usually holds the kernel program instruction
codes, and features very low latency and access times compared
model cεi as described previously.
2. Generate a synthetic experimental data set tεi by running the
full FEA or closed-form analytical solution model using ref-
erence values of the constitutive parameters as inputs.
3. Add noise to the synthetic experimental data set. The noise
should be applied to the different responses (εxx , εyy , and
εxy ) independently, in a normally distributed fashion, with a
mean µε = 0 and a standard deviation sε .
4. Recover the parameters from the noisy experimental dataset
using the optimization procedure outlined previously.
5. Repeat steps 2-4 20 times in order to collect statistical data
on the effect of experimental noise at a level of sε on the
(10)
accuracy of the recovered constitutive parameters.
6. Increase sε . Repeat steps 2-5 until sε has reached a maxi-
mum value.
7. Terminate: plot the mean accuracy of each parameter’s re-
covery as a function of sε .
Based on the existing physical apparatus available for auto-
mated multiaxial testing [2] we set an accuracy goal of parameter
error not exceeding 2.5% in the presence of sε = 100µε (micros-
train) noise. To verify that this goal was met we ran the synthetic
experiments for sε = 0 to 200µε, in increments of 10µε.
A well-known model for the strain fields of an infinite
isotropic material specimen with a hole is given by [26]. Fig.
9 illustrates the domain, the coordinates and parameters used, as
well as the sampling locations xi . Given some far-field stress σ∞
the stress field is calculated in polar coordinates by,

9
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
 TABLE 1. Sampling data for the first numerical experiment.

Parameter True Minimum Maximum Sampling

y
r Value value Value Increment
ϴ

E 69 GPa 34.5 GPa 103.5 GPa 17.25 GPa

σ∞ 2a x σ∞ ν 0.333 0.2 0.8 0.15

xT (subset)
which for the isotropic case takes the matrix form

    
εxx 1 −ν −ν 0 0 0 σxx
 εyy   1 −ν 0 0 0   σyy 
FIGURE 9. An infinite isotropic domain with a hole of radius a sub-
    
 εzz  1  1 0 0 0   σzz 
 2εyz  = E
    
jected to in-plane loading σ∞ . Only a subset of the actual sampling  2(1 + ν) 0 0   σyz 
    
locations xT are shown.  2εxz   (sym) 2(1 + ν) 0   σxz 
2εxy 2(1 + ν) σxy
(14)
This strain model was evaluated on 1600 locations xi in a uni-
form radial pattern around the hole (r = 1), a subset of which are
a2 3a4
   
σ∞ seen in Fig. 9. The samples were taken with values of θ = 0
σθ θ = 1 + 2 − 1 + 4 cos (2θ )
2 r r to 2π in increments 40 π
and r = 1 to 3 in increments of 0.1. A
2 4a2 3a4 single frame of data was used, with σ∞ = 50MPa. Table 1 shows
   
σ∞ a
σrr = 1 − 2 + 1 − 2 + 4 cos (2θ ) (11) data regarding the sampling of the constitutive parameters E and
2 r r r
 2 4
 ν (Young’s modulus and Poisson’s ratio, respectively) The true
σ∞ 2a 3a
σrθ = − 1 + 2 − 4 cos (2θ ) . parameter values in Table 1 reflect the properties of Al-6061-T6 .
2 r r With 1600 values of xi , at each of which the model of Equations
(30-33) was evaluated at 25 combinations of constitutive param-
The frame reference transformation from the polar to the eter values, the constitutive model was evaluated 40,000 times in
Cartesian coordinate system is implemented by using the well total.
known transform, GAMMA was executed on a workstation computer featur-
ing a quad-core processor operating at 4.8 GHz, 16 GB of sys-
tem memory and a NVIDIA GeForce GTX 650 GPU. It should
1 be noted that this is a very inexpensive GPU, costing less than
σxx = (σrr + σθ θ + (σrr − σθ θ ) cos (2θ ) − 2σrθ sin (2θ )) $100 at the present time. GAMMA was run with the following
2
1 parameters:
σyy = (σrr + σθ θ + (−σrr + σθ θ ) cos (2θ ) + 2σrθ sin (2θ ))
2 1. RSM polynomial order: 3
σxy = σrθ cos (2θ ) + (σrr − σθ θ ) cos (θ ) sin (θ ) 2. Optimization methodology: Newton / gradient descent
(12) 3. CUDA block size: 64 threads
The strains are then computed from the stresses using the inverse
form of Hooke’s constitutive law, Results from the application of GAMMA to the test problem
are shown in Fig. 10, below. The effects of synthetic experimen-
tal noise up to 200 microstrain in magnitude are investigated.
On average, 0.035 seconds were required to recover the material
ε = C −1 : σ (13)
properties.
The results show that the error in E is roughly constant at
a magnitude of 0.5% irrespective of experimental noise. This
implies that the error in the RSM approximation is dominating
errors caused by other factors. However, given that the error is
small, and well within the pre-established goal of 2.5%, this ap-
pears to be an acceptable compromise. The error in ν rises pro-

10
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
 Error in Parameters vs Noise Magnitude
parallelization strategy forces a choice between synchronization
2.5 paradigms (GPO vs GPE). Unfortunately, the approximation of
the material response at a set of points invalidates the GPO syn-
2.0 E chronization paradigm, and promotes the choice of GPE. The
ν GPE strategy requires repeated synchronizations and vector re-
ductions which reduces the performance the algorithm. For-
% Error

1.5
tunately, by making judicious choices regarding the surrogate
model approximation basis and implementing a highly optimized
1.0
parallel-reduce algorithm, this disruption is mitigated.
Because the concepts of parallelization, synchronization,
0.5 and approximation were considered from the start, the GAMMA
implementation effectively moderates the problematic interac-
0.0 tions of these concepts. Because of this, high computational per-
0 50 100 150 200
Noise Level (με) formance is achieved, which enables real-time materials charac-
terization.
FIGURE 10. Mean error in the recover of constitutive parameters E The preliminary results given in this paper are for a linear-
and ν (plot markers) and the associated standard deviation (whiskers) isotropic continuum. However, the underlying surrogate-based
as a function of noise level for the first numeric experiment, when using methodology has been validated on more complex material sys-
the GAMMA GPE algorithm. tem models, including those relevant to engineering composites.
The most immediate future work is thus updating the GAMMA
tool to accommodate these higher-dimensional surrogates. This
portionally to experimental noise, indicating that this noise is the
will only require the implementation of a higher-order RSM ker-
primary determinant of the accuracy of the recovery of ν.
nel, and it is anticipated that the evaluation of higher-dimensional
Compared to the timing results given in previous work [3],
polynomials will not significantly impact the computational per-
GAMMA exhibits a significant performance advantage, compute
formance of GAMMA.
time is reduced by roughly a factor of 10. However, it is impor-
Expanding GAMMA into the domain of composite mate-
tant to note that the results given in that section were developed
rial systems will enable the testing of composite materials. Live
using a different surrogate model formulation, a different opti-
feedback from the characterization algorithm will be available
mization algorithm, and an optimized commercial solver. Com-
during the testing of material specimens, which will allow for
pared to an efficient multi-core CPU implementation using the
the test frame’s load path to be dynamically adjusted to achieve
same algorithms as GAMMA, the performance margin shrinks
maximum excitation of the material’s constitutive parameters. In
slightly to a factor of 8. Given the small number of threads em-
the future, GAMMA may also be used to detect time or load-
ployed, along with the very inexpensive GPU processor used,
dependent changes in the properties of materials as well. Finally,
this is evidence that the GAMMA implementation is reasonably
with the ability to rapidly achieve characterizations, it may be
efficient.
feasible to partition the test data into sub-regions and capture the
variations in material properties throughout the spatial domain.
CONCLUSIONS AND PLANS
In this paper, we have presented the background, theory, and
Acknowledgments
methodology associated with a surrogate model-based technique
The authors acknowledge the Office of Naval Research for sup-
for materials characterization. Following this, the discussion
porting this research through both the Naval Research Enterprise
and details of a GPGPU-accelerated implementation of this al-
Internship Program (NREIP) and the Naval Research Laboratory
gorithm is given. The characterization algorithm was developed
(NRL) core funding. This research was performed while one au-
from the ground up with an awareness of massively parallel na-
thor held a National Research Council Research Associateship
ture of this computing technology. As a result, the close inter-
Award at NRL.
relationships between parallelization, synchronization, and ap-
proximation have been carefully considered and are readily ob-
servable.
In the materials characterization algorithm, the choice to
parallelize the problem by sampling the material system response
at a finite set of discrete points essentially defines an approx-
imation of the underlying material physics. Furthermore, this

11
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
 References
[1] J. Michopoulos, J. Hermanson, and A. Iliopoulos, 2010,
“Towards a Recursive Hexapod for the Multidimensional
Mechanical Testing of Composites,” Proceedings of the
ASME 2010 IDETC/CIE Conferences, Montreal, Canada.
DETC2010-13221
[2] Michopoulos JG, Hermanson JC, Iliopoulos AP, Lambrakos
SG, Furukawa T. Data-driven design optimization for com-
posite material characterization. Journal of Computing and
Information Science in Engineering 2011; 11(2):1-10.
[3] J. Steuben, J. Michopoulos, A. Iliopoulos, and C. Turner,
2012, “Inverse Characterization of Composite Materials Us-
ing Surrogate Models,” Proceedings of the ASME 2013
IDETC/CIE Conferences, Portland, OR. DETC2013-13221
[4] Sevenhuijsen J. Two simple methods for deformation
demonstration and measurement. Strain 1981; 17(1):20-24.
[5] Bruck H, McNeill S, Sutton M, Peters W. Digital image cor-
relation using newton-raphson method of partial differential
correction. Experimental Mechanics 1989; 29:261-267.
[6] Andrianopoulos NP, Iliopoulos AP. Strain measurements by
a hybrid experimental-numerical method using a mesh-free
field function, Honorary Volume for Professor P. S. Theo-
caris Armenian Academy of Sciences, 2005; 31-41.
[7] Michopoulos JG, Iliopoulos AP. A Computational Work-
bench for Remote Full Field 2D Displacement and Strain
Measurements. ASME Conference Proceedings, 2009; 55-
63.
[8] Iliopoulos A, Michopoulos JG. Effects of Anisotropy on
the Performance Sensitivity of the Mesh-Free Random Grid
Method for Whole Field Strain Measurement. ASME Con-
ference Proceedings 2009; 65-74.
[9] Michopoulos JG, Iliopoulos AP. A Computational Work-
bench for Remote Full Field 2D Displacement and Strain
Measurements. ASME Conference Proceedings 2009; 55-
63.
[10] Michopoulos JG, Iliopoulos A. A Computational Work-
bench for Remote Full Field 3D Displacement and Strain
Measurements. ASME Conference Proceedings, 2011; 489-
498.
[11] Iliopoulos A, Michopoulos J. Direct Strain Imaging For Full
Field Measurements. Proceedings of the ASME 2012 In-
ternational Design Engineering Technical Conferences &
Computers and Information in Engineering Conference Au-
gust 13-15, 2012, Chicago, IL, USA.
[12] Iliopoulos AP, Michopoulos JG, Andrianopoulos NP. Per-
formance Analysis of the Mesh-Free Random Grid Method
for Full-Field Synthetic Strain Measurements Strain, Black-
well Publishing Ltd, 2012; 48:1-15.
[13] Iliopoulos A, Michopoulos JG, Hermanson JC. Performance
analysis and experimental validation of the direct strain
imaging method. ASME 2013 International Design Engi-
neering Technical Conferences & Computers and Informa-
12
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
tion in Engineering Conference August 4-7, 2013, Portland,
OR, USA.
[14] Iliopoulos A, Michopoulos JG. Direct strain tensor approx-
imation for full-field strain measurement methods. Interna-
tional Journal for Numerical Methods in Engineering 2013;
95:313-330.
[15] Michopoulos J, Hermanson JC, Furukawa T, Iliopoulos A.
A Framework For The Automated Data-Driven Constitutive
Characterization Of Composites. Proceedings of the 17th
International Conference on Composite Materials, ICCM-
17 2009.
[16] Michopoulos J, Hermanson JC, Iliopoulos AP, Lambrakos S,
Furukawa T. On the Constitutive Response Characterization
for Composite Materials via Data-Driven Design Optimiza-
tion, Proceedings of the 2011 ASME International Design
Engineering Technical Conferences & Computers and In-
formation in Engineering Conference August 28 - 31, 2011,
Washington, DC, USA, DETC2011-47740.
[17] Michopoulos J, Iliopoulos A, Hermanson JC, Orifici AC,
Thomson RS. Preliminary Validation Of Composite Mate-
rial Constitutive Characterization. Proceedings of the ASME
2012 International Design Engineering Technical Confer-
ences & Computers and Information in Engineering Con-
ference August 13-15, 2012, Chicago, IL, USA.
[18] Michopoulos J, Hermanson JC, Iliopoulos A. First Industrial
Strength Multi-Axial Robotic Testing Campaign For Com-
posite Material Characterization. Proceedings of the ASME
2012 International Design Engineering Technical Confer-
ences & Computers and Information in Engineering Con-
ference August 13-15, 2012, Chicago, IL, USA.
[19] Iliopoulos A, Michopoulos J, Hermanson, JC. Composite
Material Testing Data Reduction To Adjust For The Sys-
tematic 6-Dof Testing Machine Aberrations. Proceedings of
the ASME 2012 International Design Engineering Technical
Conferences & Computers and Information in Engineering
Conference August 13-15, 2012, Chicago, IL, USA.
[20] Meuwissen M, Oomens C, Baaijens F, Petterson R,
Janssen J. Determination of the elasto-plastic proper-
ties of aluminum using a mixed numerical-experimental
method. Journal of Materials Processing Technology 1998;
75(1):204 -211.
[21] Kajberg J, Lindkvist G. Characterisation of materials sub-
jected to large strains by inverse modeling based on in-
plane displacement elds. International Journal of Solids and
Structures 2004; 41(13):3439-3459.
[22] Schmidt T, Tyson J, Galanulis. Full-eld dynamic displace-
ment and strain measurement using advanced 3d image cor-
relation photogrammetry: Part 1. Experimental Techniques
2003; 27(3):47-50.
[23] Molimard J, Le Riche R, Vautrin A, Lee J. Identication of
the four orthotropic plate stiffnesses using a single open-hole
tensile test. Experimental Mechanics 2005; 45:404-411.
[24] Grediac M, Toussaint E, Pierron F. Special virtual fields for
the direct determination of material parameters with the vir-
tual fields method. 1principle and definition. International
Journal of Solids and Structures 2002; 39(10):2691-2705.
[25] Pierron F, Vert G, Burguete R, Avril S, Rotinat R, Wis-
nom MR. Identication of the orthotropic elastic stiffnesses of
composites with the virtual fields method: Sensitivity study
and experimental validation. Strain 2007; 43(3):250-259.
[26] Michopoulos JG, Iliopoulos AP, Furukawa T. Accuracy of
inverse composite laminate characterization via the mesh
free random grid method. ASME International Design En-
gineering Technical Conferences and Computers and Infor-
mation in Engineering Conference 2009; 2:367-374.
[27] Furukawa T, Michopoulos JG,Kelly DW. Elastic character-
ization of laminated composites based on multiaxial tests.
Composite Structures 2008; 86(1 - 3):269-278.
[28] Furukawa T, Michopoulos JG. Online planning of multiaxial
loading path for elastic material identification. Computer
Methods in Applied Mechanics and Engineering 2008; 197(
9 - 12): 885-901.
[29] Michopoulos JG, Furukawa T, Lambrakos SG. Data-driven
characterization of composites based on virtual determinis-
tic and noisy multiaxial data. ASME International Design
Engineering Technical Conferences and Computers and In-
formation in Engineering Conference 2008; 2:1095-1106.
[30] Furukawa T, Pan JW. Stochastic identification of elastic con-
stants for anisotropic materials. International Journal for
Numerical Methods in Engineering 2010; 81(4):429-452.
[31] Samarasinghe S. Neural networks for applied sciences and
engineering, in From Fundamentals to Complex Pattern
Recognition. Auerbach Publications: Massachusetts, 1988.
[32] Michopolous J. Constitutive characterization of composites
via artificial neural nets. Tech. Rep. Code 6383 Internal
Report. Naval Research Laboratory: Washington DC 1992.
[33] Furukawa T, Yagawa G. Implicit constitutive modeling for
viscoplasticity using neural networks. International Jour-
nal for Numerical Methods in Engineering 1998; 43(2):195-
219.
[34] Turner CJ. HyPerModels: Hyperdimensional Performance
Models for engineering Design. PhD thesis, The University
of Texas at Austin 2005.
[35] Turner CJ, Crawford RH. Selecting an appropriate meta-
model: The case for NURBs metamodels. ASME Interna-
tional Design Engineering Technical Conferences and Com-
puters and Information in Engineering Conference 2005;
2:759-771.
[36] Aarts EHL, Korst J. Simulated annealing and Boltzmann
machines: a stochastic approach to combinatorial optimiza-
tion and neural computing, Wiley, 1989.
[37] Kirkpatrick S. Optimization by Simulated Annealing: Quan-
titative Studies. Journal of Statistical Physics 1984; 34(5-
6):975-986.
13
This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States.
Approved for public release; distribution is unlimited.
[38] Nelder JA, Mead R. A Simplex Method for Function Mini-
mization. The Computer Journal 1965; 7(4):308-313.
[39] Lagarias JC, Reeds JA, Wright MH, Wright PE. Conver-
gence Properties of the Nelder-Mead Simplex Method in
Low Dimensions. SIAM Journal of Optimization 1998;
9(1):112-147.
[40] Goddeke, D., 2011, Fast and Accurate Finite-Element
Multigrid Solvers for PDE Simulations on GPU Clusters,
Logos Verlag, Berlin, Germany.
[41] Krawezik, G.P., Poole, G., 2010, “Accelerating the AN-
SYS direct sparse solver with GPUs,”Proceedings of the
2010 Symposium on Application Accelerators in High Per-
formance Computing, Available: http://saahpc.ncsa. illi-
nois.edu/09/papers/Krawezik paper.pdf.
[42] Tabatabaian M, 2013, Comsol for Engineers, Mercury
Learning and Information.
[43] Kruger, J., Westermann, R., 2003, “Linear algebra operators
for GPU implementation of numerical algorithms,” ACM
Transactons on Graphics, 22(3):908-916.
[44] Li, R., Saad, Y., 2013, “GPU-accelerated preconditioned it-
erative linear solvers,” Journal of Supercomputing, 63(2):
443-466.
[45] Courtier, R., 2013, Designing Scientific Applications on
GPUs, CRC Press, Boca Raton, FL.
[46] NVIDIA, 2010, “Introduction to CUDA C”,
GPU Technology Conference 2010, Available at
http://www.nvidia.com/content/GTC-2010/
pdfs/2131_GTC2010.pdf.
[47] NVIDIA, 2012, “NVIDIA CUDA C Programming
Guide”, Available at http://developer.download.
nvidia.com/compute/DevZone/docs/html/C/
doc/CUDA_C_Programming_Guide.pdf.
[48] Harris M, “Optimizing Parallel Reduction in CUDA,”
NVIDIA Developer Technology, Presentation available
online at http://developer.download.nvidia.
com/compute/cuda/1.1-Beta/x86_website/
projects/reduction/doc/reduction.pdf.