Surrogate model

A surrogate model is an engineering method used when an outcome of interest cannot be easily measured
or computed, so an approximate mathematical model of the outcome is used instead. Most engineering
design problems require experiments and/or simulations to evaluate design objective and constraint
functions as a function of design variables. For example, in order to find the optimal airfoil shape for an
aircraft wing, an engineer simulates the airflow around the wing for different shape variables (length,
curvature, material, ..). For many real-world problems, however, a single simulation can take many minutes,
hours, or even days to complete. As a result, routine tasks such as design optimization, design space
exploration, sensitivity analysis and what-if analysis become impossible since they require thousands or
even millions of simulation evaluations.

One way of alleviating this burden is by constructing approximation models, known as surrogate models,
metamodels or emulators, that mimic the behavior of the simulation model as closely as possible while
being computationally cheaper to evaluate. Surrogate models are constructed using a data-driven, bottom-
up approach. The exact, inner working of the simulation code is not assumed to be known (or even
understood), solely the input-output behavior is important. A model is constructed based on modeling the
response of the simulator to a limited number of intelligently chosen data points. This approach is also
known as behavioral modeling or black-box modeling, though the terminology is not always consistent.
When only a single design variable is involved, the process is known as curve fitting.

Though using surrogate models in lieu of experiments and simulations in engineering design is more
common, surrogate modeling may be used in many other areas of science where there are expensive
experiments and/or function evaluations.

Goals
The scientific challenge of surrogate modeling is the generation of a surrogate that is as accurate as
possible, using as few simulation evaluations as possible. The process comprises three major steps which
may be interleaved iteratively:

Sample selection (also known as sequential design, optimal experimental design (OED) or
active learning)
Construction of the surrogate model and optimizing the model parameters (bias-variance
trade-off)
Appraisal of the accuracy of the surrogate.

The accuracy of the surrogate depends on the number and location of samples (expensive experiments or
simulations) in the design space. Various design of experiments (DOE) techniques cater to different sources
of errors, in particular, errors due to noise in the data or errors due to an improper surrogate model.

Types of surrogate models

Popular surrogate modeling approaches are: polynomial response surfaces; kriging; more generalized
Bayesian approaches,[1] gradient-enhanced kriging (GEK); radial basis function; support vector machines;
space mapping;[2] artificial neural networks and Bayesian networks.[3] Further methods recently explored
are Fourier surrogate modeling [4][5] and random forests .[6]
For some problems, the nature of true function is not known a priori, and therefore it is not clear which
surrogate model will be the most accurate one. In addition, there is no consensus on how to obtain the most
reliable estimates of the accuracy of a given surrogate. Many other problems have known physics
properties. In these cases, physics-based surrogates such as space-mapping based models are commonly
used.[2][7]

Invariance properties
Recently proposed comparison-based surrogate models (e.g. ranking support vector machine) for
evolutionary algorithms, such as CMA-ES, allow to preserve some invariance properties of surrogate-
assisted optimizers: [8]

1. Invariance with respect to monotonous transformations of the function (scaling)

2. Invariance with respect to orthogonal transformations of the search space (rotation).

Applications
An important distinction can be made between two different applications of surrogate models: design
optimization and design space approximation (also known as emulation).

In surrogate model based optimization, an initial surrogate is constructed using some of the available
budgets of expensive experiments and/or simulations. The remaining experiments/simulations are run for
designs which the surrogate model predicts may have promising performance. The process usually takes
the form of the following search/update procedure.

1. Initial sample selection (the experiments and/or simulations to be run)

2. Construct surrogate model
3. Search surrogate model (the model can be searched extensively, e.g. using a genetic
algorithm, as it is cheap to evaluate)
4. Run and update experiment/simulation at a new location(s) found by search and add to
sample
5. Iterate steps 2 to 4 until out of time or design 'good enough'

Depending on the type of surrogate used and the complexity of the problem, the process may converge on
a local or global optimum, or perhaps none at all.[9]

In design space approximation, one is not interested in finding the optimal parameter vector but rather in the
global behavior of the system. Here the surrogate is tuned to mimic the underlying model as closely as
needed over the complete design space. Such surrogates are a useful, cheap way to gain insight into the
global behavior of the system. Optimization can still occur as a post-processing step, although with no
update procedure (see above) the optimum found cannot be validated.

Surrogate modeling software

Surrogate Modeling Toolbox (SMT: https://github.com/SMTorg/smt): is a Python package that
contains a collection of surrogate modeling methods, sampling techniques, and
benchmarking functions. This package provides a library of surrogate models that is simple
to use and facilitates the implementation of additional methods. SMT is different from
existing surrogate modeling libraries because of its emphasis on derivatives, including
training derivatives used for gradient-enhanced modeling, prediction derivatives, and
 derivatives with respect to the training data. It also includes new surrogate models that are
not available elsewhere: kriging by partial-least squares reduction and energy-minimizing
spline interpolation.[10]
Surrogates.jl (https://surrogates.sciml.ai/latest/): is a Julia packages which offers tools like
random forests, radial basis methods and kriging.

See also
Linear approximation
Response surface methodology
Kriging
Radial basis functions
Gradient-enhanced kriging (GEK)
OptiY
Space mapping
Surrogate endpoint
Surrogate data
Fitness approximation
Computer experiment
Conceptual model
Bayesian regression
Bayesian model selection

References
1. Ranftl, Sascha; von der Linden, Wolfgang (2021-11-13). "Bayesian Surrogate Analysis and
Uncertainty Propagation" (https://doi.org/10.3390%2Fpsf2021003006). Physical Sciences
Forum. 3 (1): 6. doi:10.3390/psf2021003006 (https://doi.org/10.3390%2Fpsf2021003006).
ISSN 2673-9984 (https://www.worldcat.org/issn/2673-9984).
2. J.W. Bandler, Q. Cheng, S.A. Dakroury, A.S. Mohamed, M.H. Bakr, K. Madsen and J.
Søndergaard, "Space mapping: the state of the art (https://ieeexplore.ieee.org/document/126
2727)," IEEE Trans. Microwave Theory Tech., vol. 52, no. 1, pp. 337-361, Jan. 2004.
3. Cardenas, IC (2019). "On the use of Bayesian networks as a meta-modeling approach to
analyse uncertainties in slope stability analysis". Georisk: Assessment and Management of
Risk for Engineered Systems and Geohazards. 13 (1): 53–65.
doi:10.1080/17499518.2018.1498524 (https://doi.org/10.1080%2F17499518.2018.149852
4). S2CID 216590427 (https://api.semanticscholar.org/CorpusID:216590427).
4. Manzoni, L.; Papetti, D. M.; Cazzaniga, P.; Spolaor, S.; Mauri, G.; Besozzi, D.; Nobile, M. S.
Surfing on Fitness Landscapes: A Boost on Optimization by Fourier Surrogate Modeling.
Entropy 2020, 22, 285.
5. Bliek, L.; Verstraete, H. R.; Verhaegen, M.; Wahls, S. Online optimization with costly and
noisy measurements using random Fourier expansions. IEEE transactions on neural
networks and learning systems 2016, 29(1), 167-182.
6. Dasari, S.K.; P. Andersson; A. Cheddad (2019). "Random Forest Surrogate Models to
Support Design Space Exploration in Aerospace Use-Case" (https://www.springerprofession
al.de/en/random-forest-surrogate-models-to-support-design-space-explorati/16724106).
Artificial Intelligence Applications and Innovations (AIAI 2019). Springer. pp. 532–544.
Retrieved 2019-06-02.
 7. J.E. Rayas-Sanchez,"Power in simplicity with ASM: tracing the aggressive space mapping
algorithm over two decades of development and engineering applications" (http://ieeexplore.
ieee.org/xpl/articleDetails.jsp?arnumber=7423860&action=search&sortType=&rowsPerPag
e=&searchField=Search_All&matchBoolean=true&queryText=(%22Document%20Title%22:
simplicity%20in%20asm)), IEEE Microwave Magazine, vol. 17, no. 4, pp. 64-76, April 2016.
8. Loshchilov, I.; M. Schoenauer; M. Sebag (2010). "Comparison-Based Optimizers Need
Comparison-Based Surrogates" (https://hal.inria.fr/file/index/docid/493921/filename/ACM-E
S.pdf) (PDF). Parallel Problem Solving from Nature (PPSN XI). Springer. pp. 364–1373.
9. Jones, D.R (2001), "A taxonomy of global optimization methods based on response surfaces
(http://www.ressources-actuarielles.net/EXT/ISFA/1226.nsf/9c8e3fd4d8874d60c125705200
3eced6/e7dc33e4da12c5a9c12576d8002e442b/$FILE/Jones01.pdf)," Journal of Global
Optimization, 21:345–383.
10. Bouhlel, M.A.; Hwang, J.H.; Bartoli, Nathalie; Lafage, R.; Morlier, J.; Martins, J.R.R.A. (2019).
"A Python surrogate modeling framework with derivatives" (http://mdolab.engin.umich.edu/c
ontent/python-surrogate-modeling-framework-derivatives). Advances in Engineering
Software. 135: 102662. doi:10.1016/j.advengsoft.2019.03.005 (https://doi.org/10.1016%2Fj.a
dvengsoft.2019.03.005). S2CID 128324330 (https://api.semanticscholar.org/CorpusID:1283
24330).

Further reading
Queipo, N.V., Haftka, R.T., Shyy, W., Goel, T., Vaidyanathan, R., Tucker, P.K. (2005),
“Surrogate-based analysis and optimization (https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nas
a.gov/20050186653.pdf),” Progress in Aerospace Sciences, 41, 1–28.
D. Gorissen, I. Couckuyt, P. Demeester, T. Dhaene, K. Crombecq, (2010), “A Surrogate
Modeling and Adaptive Sampling Toolbox for Computer Based Design (http://jmlr.csail.mit.e
du/papers/volume11/gorissen10a/gorissen10a.pdf)," Journal of Machine Learning
Research, Vol. 11, pp. 2051−2055, July 2010.
T-Q. Pham, A. Kamusella, H. Neubert, “Auto-Extraction of Modelica Code from Finite
Element Analysis or Measurement Data (http://www.ep.liu.se/ecp/063/074/ecp11063074.pd
f)," 8th International Modelica Conference, 20–22 March 2011 in Dresden.
Forrester, Alexander, Andras Sobester, and Andy Keane, Engineering design via surrogate
modelling: a practical guide (https://books.google.com/books?id=ulMHmeMnRCcC&dq=%2
2Engineering+design+via+surrogate+modelling%3A+a+practical+guide%22&pg=PR5),
John Wiley & Sons, 2008.
Bouhlel, M. A. and Bartoli, N. and Otsmane, A. and Morlier, J. (2016) "Improving kriging
surrogates of high-dimensional design models by Partial Least Squares dimension
reduction (https://hal.archives-ouvertes.fr/hal-01232938/file/KPLS_paper2015.pdf)",
Structural and Multidisciplinary Optimization 53 (5), 935-952
Bouhlel, M. A. and Bartoli, N. and Otsmane, A. and Morlier, J. (2016) "An improved approach
for estimating the hyperparameters of the kriging model for high-dimensional problems
through the partial least squares method (http://downloads.hindawi.com/journals/mpe/2016/
6723410.pdf)", Mathematical Problems in Engineering

External links
Matlab code for surrogate modelling (http://www.wiley.com//legacy/wileychi/forrester/terms.ht
ml)
Matlab SUrrogate MOdeling Toolbox – Matlab SUMO Toolbox (http://sumowiki.intec.ugent.b
e)
Surrogate Modeling Toolbox -- Python (https://github.com/SMTorg/SMT)
Retrieved from "https://en.wikipedia.org/w/index.php?title=Surrogate_model&oldid=1160704396"