RESEARCH ARTICLE | JUNE 01 2020

Exploration and prediction of fluid dynamical systems using

auto-encoder technology 
Lionel Agostini 

Physics of Fluids 32, 067103 (2020)

https://doi.org/10.1063/5.0012906

CrossMark

 
View Export
Online Citation

11 December 2023 21:54:35

 Physics of Fluids ARTICLE scitation.org/journal/phf

Exploration and prediction of fluid dynamical

systems using auto-encoder technology
Cite as: Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Submitted: 7 May 2020 • Accepted: 8 May 2020 •
Published Online: 1 June 2020

Lionel Agostinia)

AFFILIATIONS
Department of Aeronautics, Imperial College London, South Kensington, London SW7 2AZ, United Kingdom

a)
Author to whom correspondence should be addressed: l.agostini@imperial.ac.uk

ABSTRACT
Machine-learning (ML) algorithms offer a new path for investigating high-dimensional, nonlinear problems, such as flow-dynamical systems.
The development of ML methods, associated with the abundance of data and combined with fluid-dynamics knowledge, offers a unique
opportunity for achieving significant breakthroughs in terms of advances in flow prediction and its control. The objective of this paper is to
discuss some possibilities offered by ML algorithms for exploring and predicting flow-dynamical systems. First, an overview of basic concepts
underpinning artificial neural networks, deep neural networks, and convolutional neural networks is given. Building upon this overview, the

11 December 2023 21:54:35

concept of Auto-Encoders (AEs) is introduced. An AE constitutes an unsupervised learning technique in which a neural-network architecture
is leveraged for determining a data structure that results from reducing the dimensionality of the native system. For the particular test case
of flow behind a cylinder, it is shown that combinations of an AE with other ML algorithms can be used (i) to provide a low-dimensional
dynamical model (a probabilistic flow prediction), (ii) to give a deterministic flow prediction, and (iii) to retrieve high-resolution data in the
spatio-temporal domain from contaminated and/or under-sampled data.
Published under license by AIP Publishing. https://doi.org/10.1063/5.0012906., s

I. INTRODUCTION
Fluid mechanics is in the enviable position of having a firmly
established, mathematical foundation, in the form of the Navier–
Stokes (NS) equations, which describe all single-phase, non-reacting
fluid-flow scenarios.
For an incompressible flow, the NS equations can be written as
∂u 1
+ (u ⋅ ∇)u = − ∇p + g + ν∇2 u ,
∂t ´¹¹ ¹ ¹ ¹ ¹ ¹¸¹ ¹ ¹ ¹ ¹ ¹ ¶ ρ ²
° convection ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ diffusion
variation sources
where u is the velocity field, ρ is the density, ν is the dynamic vis-
cosity, p is the pressure, and g represents body acceleration. The
motion of any viscous fluid can be accurately predicted, in principle,
by solving these equations if the boundary conditions are fully pre-
scribed (Laplace’s determinism65 ). However, the non-linear terms
associated with the convective acceleration are the source of instabil-
ity that manifests itself, at more than very modest speeds, by a wide
range of interacting eddies that evolve chaotically in time and space.
The non-linear terms are the origin of the turbulence—a chaotic (but
not random) phenomenon that is characterized by a broad range of
spatio-temporal scales, some of which are associated with coher-
ent structures and feature certain aspects of order and organiza-
tion.15,55,64 This makes most fluid flows complex dynamical sys-
tems that contain numerous spatio-temporal degrees of freedom—
characterizing the system’s “dimensionality.”
As the dimensionality becomes ever higher at the increas-
(1) ing speed (Reynolds number), solving the NS equations becomes
more and more resource-intensive, exceeding present computa-
tional capabilities in most practically relevant conditions. A route
often taken for predicting the flow is, therefore, to simplify the sys-
tem by replacing it with an appropriate model, thus reducing the
system’s dimensionality. The real system is then represented by a
simplified reduced-order model, composed of a minimum num-
ber of modes able to represent the principal physical and opera-
tional characteristics of the flow system. As non-trivial fluid-flow
systems are inherently complex—characterized by extremely high
dimensionality, a wide range of scales, and strong non-linearity—
the identification of the phenomena driving the flow dynamics and

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-1

Published under license by AIP Publishing
 Physics of Fluids
their associated modes remains a major challenge in present-day
fluid-flow physics and engineering.
Thanks to recent advances in experimental and computational
methods, and the drastic reduction in computational costs due to
advances in hardware, both the number and the size of databases
that contain the output of measurements and simulations have
increased dramatically over the past decade or two. This abundance
of data holds great promise for deepening the knowledge of fluid-
dynamic systems but poses new challenges for processing the data
that requires novel tools. Candidates for such tools and related tech-
niques are progressively available within the rapidly growing field
“Machine Learning (ML),” and present methods are increasingly
made applicable to the analysis of spatio-temporal fluid-flow data,
aided by recent advances in hardware (Graphics/Tensor Process-
ing Unit—GPU/TPU—platforms, in particular), algorithms, and
open-source libraries.
By definition, ML is the field of study that gives computers the
ability to learn without being explicitly programmed.61 ML algo-
rithms are categorized as supervised, semi-supervised, and unsuper-
vised (see Fig. 1 and the recent review of Brunton et al.7 ). The task
of supervised learning is to learn a function that maps input to out-
put based on labeled examples. In contrast, unsupervised learning
consists of learning unknown patterns in a set of data without a
priori information. Finally, semi-supervised learning operates on a
large amount of unlabeled data, or with corrective input from an
appropriate environment, for training a small amount of labeled
data. The specific potential of some ML algorithms lies in their abil-
ity to recognize structure in the data and define alternative flow-
representing models at much-reduced effort and cost than conven-
tional human-driven approaches. By leveraging large data sets that
describe the evolution of associated flows, ML algorithms thus offer
a new path for deriving flow representations, which are more com-
pact and efficient than the native data sets in the original domains.
Several recent reviews7,20,33 show the potential of ML algorithms
to radically transform the way data mining is performed in fluid
mechanics.
FIG. 1. Schematic representation of a deep neural network.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
ML is also becoming increasingly pertinent to the construc-
tion of efficient control algorithms for non-linear fluid-flow sys-
tems. Within a closed-loop-control framework, the fluid flow as a
whole may be regarded as a dynamical system, which is steered
toward a desired state by control laws. Inputs are disturbances and
actuation signals applied by the controller, whereas the dynami-
cal system’s outputs are the cost function—the quantity that has
to be optimized—and sensor signals corresponding to the observ-
able quantities. The controller’s objective is to determine from the
sensors the control laws that must be imposed onto the system for
maximizing or minimizing the cost function.
Rapid developments of ML algorithms in fluid mechanics (see
the recent reviews of Brunton et al.7 and Brenner et al.6 ) and modern
computational hardware provide the rationale for pursuing the use
of data-driven methods and ML algorithms for active flow control.
The following two ML-based routes can be taken for implementing
flow-control strategies, one being conventional and the other more
progressive:
● Model-driven approach. The real system is represented
by a simplified, reduced-order model. This model can be
described by a set of equations—e.g., linearized Navier–
Stokes equations,2,18 resolvent analysis,41 exact coherent
structures,25 or by equations coupled with data—e.g., POD-
Galerkin14 or data assimilation.16,68 Whichever method
is used, the model-driven approach poses majors chal-
11 December 2023 21:54:35
lenges for complex flows: deriving the modes is, generally,
extremely resource-intensive. This is especially so for non-
linear multi-scale flows. In this case, the flow dynamics can
change dramatically in an unpredictable manner due to the
complex non-linear characteristics of the flows, and this
makes it extremely hard for the model to represent suffi-
ciently well the real phenomena being provoked by control
laws.
● Data-driven approach. By taking advantage of recent devel-
opments in ML algorithms and modern computational
32, 067103-2
 Physics of Fluids
hardware, data-driven algorithms hold great potential for
identifying control strategies by focusing directly on the
controller without any knowledge of the dynamics and
without recourse to any model—the so-called model-free
approach. ML optimizes the system based on data the sys-
tem generates—thus unlike model-driven control, which
optimizes the constrained model.
A whole range of ML methods have been successfully used in
fluid mechanics for a variety of desired outcomes; among meth-
ods used specifically for “Machine Learning Control (MLC),” evo-
lutionary algorithms and genetic algorithms have been shown to
be very effective for designing well behaved control laws. Evolu-
tionary algorithms are based on the biological principle of natural
selection. Starting with an initial population of control laws, which
are all competing against one another, only the most effective one
for optimizing the objective function is selected. Control laws are
improved from generation to generation by applying genetic oper-
ations (elitism, mutation, replication, crossover). By “breeding” the
most effective control laws, these are selectively improved from gen-
eration to generation. Unlike genetic algorithms, which search for
the best parameters to apply to the control law, genetic program-
ming control learns the structure of control laws and parameters.17
Genetic programming has been used for turbulence control in vari-
ous applications, for example, the mixing-layer,52 flow-separation,21
turbulent separated shear flow,4 and flow around bluff bodies (e.g.,
the Ahmed body39,40 ). Another approach for identifying an opti-
mal control strategy is Reinforcement Learning63 (RL). This is an
autonomous, self-learning strategy that, essentially, acquires knowl-
edge by interacting with its environment. In this approach, an agent
interacts with its environment through actions. With every action,
the agent modifies its environment and gets a reward. The agent’s
goal is to determine those actions that maximize its long-term
rewards. To accelerate convergence, a model reward, combining
state and action, is learned from the data. In some applications,
computer memory is also attached to the agent to remember the
best past actions/states. The landscape of the cost function can
be learned by a neural network. This method is referred to as
Deep Reinforcement Learning (DRL), and it has been success-
fully used by Rabault et al.57 for reducing drag behind a rotating
cylinder.
As any non-trivial fluid-flow system is inherently complex and
of high dimensionality, determining control laws directly from its
data is hardly feasible because of resource constraints. However, this
becomes tenable if the raw data are replaced by low-dimensional
flow representations that can be “handled” by MLC algorithms. A
possible route is to simplify the system by replacing it with a “coarse”
model that reduces the dimensionality of the flow by identifying a
small number of modes, which are dominant and thus represent the
principal physical and operational characteristics of the flow system.
The model is then used as “a bridge” between the high-dimensional
complex raw data and the MLC algorithms. Based on the objec-
tive defined by a cost function, MLC algorithms select modes that
have to be enhanced and/or modes that have to be depressed
or suppressed and define a strategy for achieving the purpose of
the algorithm. This approach is a hybrid between model-driven
and data-driven methods, correcting the deficiencies of the latter
mutually.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
Auto-Encoder (AE) technology is a “good candidate” for reduc-
ing efficiently the problem dimensionality, as shown by the online
article “A look at deep learning for science” by Prabhat,49 which pro-
vides an excellent overview of some uses of the deep learning tech-
nology in science applications. Indeed, several studies presented in
this overview were only possible by the use of AEs. This method
aims at extracting essential information and representing it in opti-
mal space, by reducing the dimensionality of the data and conserv-
ing only the most essential information. By identifying new coor-
dinate systems, built on non-linear embedding, the AE combined
with other ML algorithms allows for the approximation of strongly
non-linear dynamics by a sparse, weakly non-linear model,11,43,51
enabling MLC methods to design the most efficient control laws.
After a short introduction to Artificial Neural Networks
(ANNs) in Sec. II, the concept of the AE is discussed and com-
pared to the Proper Orthogonal Decomposition (POD) in Sec. III,
a conventional technique used to represent flow-structure informa-
tion. In Secs. V–VII, the low-dimensional dynamical representation
obtained by the AE is then used, in conjunction with other machine
learning algorithms, to
(i) provide a low-dimensional dynamical model (a probabilistic
flow prediction)—in Sec. V,
(ii) give a deterministic flow prediction—in Sec. VI, and
(iii) retrieve high-resolution data in the spatio-temporal domain
from contaminated and/or under-sampled data—in Sec. VII.
11 December 2023 21:54:35
II. ARTIFICIAL NEURAL NETWORK (ANN) AND DEEP
LEARNING
ANNs are computing systems composed of functional nodes,
entities transforming and transferring information among one
another. The role of each of these entities, called “perceptrons,” is
similar to that of a neuron in biological neural networks. The notion
of a perceptron was first introduced by Rosenblatt59 to define a
simple binary network.
A deep neural network (DNN) is an ANN with several layers
between the input and output layers. An example of operations per-
formed by a DNN is given in Fig. 1. In this example, a single obser-
vation x provides inputs into the first layer of nodes. Each of these
inputs is multiplied by a connection weight W1 , and a bias (b1 ) is
added for shifting the activation function f1 up or down. This deter-
mines which perceptron should be activated (“fired”) and decides
which amount and nature of information be transferred to the fol-
lowing layer. As is shown in Fig. 1, the output of a hidden layer l is
given by hl = fl (Wl hl-1 + bl ), where hl−1 is the previous-layer out-
put. If a DNN with L layers is considered, the final output y is then
given by the relation
y = fL (WL fL−1 (⋯(f 1 (W1 x + b1 ))⋯ + bL−1 ) + bL ). (2)
A DNN is a multi-layer stack of simple modules that can be
non-linear, transforming the input data successively with increas-
ing levels of abstraction. By adding and combining enough of these
modules, any function can be learned, and a DNN can then emu-
late this function. Results obtained by Hecht-Nielsen,26 based on
Kolmogorov’s theorem,31 show that a DNN is a “universal approxi-
mator,” able to approximate any function.
32, 067103-3
 Physics of Fluids
To demonstrate the effect of the use of non-linear activation
functions, the functions in Eq. (2) are initially prescribed to be linear
[e.g., f (x) = ax]. The transformation operated by the network is then
given by
L ⎛ L
bi ⎞ A backpropagation algorithm allows the DNN to adjust its inter-
y = (∏ ai Wi ) x + ∑ i j j . (3)
i=1 ⎝ i=1 ∏j=1 a W ⎠
This formula shows clearly that their combination is still a lin-
ear function—i.e., the final activation function of the last layer is a
linear function of the input into the first layer. Therefore, a DNN
with L layers can be replaced by an ANN with a single layer model-
ing only a linear regression between the input and the output. This is
contrasted with the relation given in Eq. (2) for which the DNN cor-
responds to a non-linear regression model if the activation functions
are non-linear.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
Training the network entails finding the parameters θ
L
= {Wl , bl }l=1 that minimize the (expected) loss of meaningful infor-
mation between the output and the target value. Modern neural
networks use a technique called backpropagation9,37,60,67 to train the
model and discover intricate structures in high-dimensional data.
nal parameters used to compute the representation in each layer
from the representation in the previous layer. This approach places
an increased computational strain on the activation function and
its derivative function. To properly adjust the weight vector Wl ,
a gradient vector is computed by the learning algorithm for each
weight, providing information on the error variation as the weight is
slightly modified. The weight vector is then adjusted in the opposite
direction to the gradient vector.
The concept of backpropagation by which ANNs learn to tune
their parameters is illustrated in Fig. 2 by using an electric circuit
11 December 2023 21:54:35
FIG. 2. Simple representation of how
a single layer neural network tunes its
weights (learns) using the backpropaga-
tion method and gradient descent.
32, 067103-4
 Physics of Fluids
as a surrogate for a DNN. The electric current flows from a socket
to a bulb light, and its intensity is regulated by two rotary switches.
The intensity of the input and output currents is represented by x
and y, respectively, and h is the voltage after the first switch (hidden
layer). The objective here is to find the configuration of both switches
(weights) for which the brightness is optimal (not necessarily maxi-
mal), y′ being the optimal intensity. A cost function J can be defined
and represented by the relation J = ∥y − y′ ∥. The optimal bright-
ness can be obtained by minimizing the cost function J by adjusting
w1 and w2 . The variation of the cost function for small changes in
w1 and w2 has to be determined. Although these derivatives can-
not be defined directly, they can be estimated by backpropagating
the sensitivity of J to variations in the electric current. The route
to extracting these derivatives and linking them to ∂y ∂J
is referred
to as the “chain of sensitivity” or “chain of rules,”9,35,67 and this is
shown in the second frame in Fig. 2. The role of the backpropaga-
tion is to define the amounts by which w1 and w2 have to be adjusted,
the adjustments being weighted by the sensitivity. Finding the opti-
mal configuration is an iterative process of making small changes
in w1 and w2 and updating the sensitivity estimates until J cannot
decrease further. Small weight changes Δw1 and Δw2 are imposed at
each iteration in order to perform a step in the downhill direction
of the sensitivity curve. This approach is referred to as the “gradient
descent” method. Weight increments are proportional to the sen-
sitivity through the constant multiplier lr, called the “learning rate.”
This constant has to be chosen with care. If it is too large, the descent
learning process will be unstable, while an excessively small value
will result in the learning process taking an unreasonably amount of
time to converge.
Updating weights by backpropagating gradients of loss defined
from the cost function can be an unstable process, and two well-
known problems altering the learning process are “vanishing gra-
dients” and “exploding gradients.” Vanishing gradients was a sig-
nificant problem when sigmoid and tanh were the only activation
functions available, and only shallow DNNs could be trained. As
previously explained and illustrated in Fig. 2, derivatives of activa-
tion functions must be computed for propagating information in the
network. If the values of the latter are inferior to 1, gradients tend to
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
get smaller and smaller as information is pushed backward in the
network, resulting in that the neurons embedded in the first layer
are learning very slowly as compared to the neurons in the later lay-
ers in the hierarchy. The problem of vanishing gradients implies that
DNNs take a long time to train and thus degrade the model’s accu-
racy. On the other hand, exploding gradients is the exact opposite
problem to vanishing gradients: large gradient errors accumulate,
as the magnitude of weights is high, resulting in excessively large
updates to the model’s weights. The model oscillates around the
minimum or even overshoots the optimum, and the model is thus
unable to learn.
Generally, the corruption of the learning process by vanish-
ing gradients can be avoided by using the REctified Linear Unit
(RELU)48 function as the activation function, as the gradient is 0
for negative inputs and 1 for positive ones. The RELU is relatively
robust toward the vanishing/exploding gradient problem, and for
DNNs, the initial values of the weights can be set to particular values,
depending on the non-linear activation function used, as defined by
Glorot and Bengio.22
III. DIMENSIONALITY REDUCTION WITH
AUTO-ENCODER
An AE is a neural-network architecture that has three parts:
an encoder, a bottleneck (latent space), and a decoder. An AE is an
11 December 2023 21:54:35
unsupervised ML algorithm that is trained to reconstruct its inputs,
considered high-dimensional data. As a consequence, the algorithm
forces the hidden layer to try to learn good compressed representa-
tions of the inputs. A conceptual representation of the AE training
process and examples of applications are conveyed in Fig. 3. By
reducing the dimensionality of the data and conserving only the
most essential information, AE technology offers, in the context
of fluid dynamics, a new path for deriving a representation of the
flow in question that is more compact and efficient than the native
data sets in the original description. Unlike Principal Component
Analysis (PCA, closely connected to POD), which is intrinsically a
linear regression method, an AE is capable of modeling complex
FIG. 3. Conceptual representation of
auto-encoder training and objectives.
32, 067103-5
 Physics of Fluids
non-linear functions and consequently is more effective for describ-
ing the underlying manifold structure of the data.3,5 The AE can
be regarded as a generalization of the POD54 or Dynamic Mode
Decomposition (DMD).51,66 The automated extraction of flow fea-
tures by unsupervised learning algorithms,35 such as the AE com-
bined with other ML algorithms—for instance, clustering methods
(to be discussed later)—offers innovative methods for flow modeling
and control using low-dimensional dynamical models.
A. Auto-encoder technology: Principles and purpose
The AE’s principles were first introduced in the ML commu-
nity more than 30 years ago.5,34 However, it is only the development
of new powerful computational hardware and Python libraries that
have enabled its widespread utilization. As previously mentioned,
an AE is a multi-layered neural network, composed of an encoder
and a decoder, separated by a “bottleneck.” As a first step, the input
data are compressed into a lower-dimensional space by the encoder.
Then, the decoder reconstitutes the original input by using the com-
pressed information embedded in the “latent space,” as illustrated
in Fig. 3. If some structure or order exists in the input data, in
whatever sense (e.g., correlations between features), this structure
is “learned” and consequently leveraged when the input is forced
through the AE’s bottleneck. By reducing the latent-space dimen-
sion, the AE has to exploit the natural data structure for identi-
fying the most efficient representation, in order to compress the
data without any loss of important information. The size of the
bottleneck constrains the amount of information allowed to tra-
verse the full network by imposing a bottleneck dimension (much)
smaller than the dimension required to express the original data.
As the decoder has imperfect information, following the compres-
sion, penalizing the AE according to the reconstruction error, the
encoder and decoder are forced to cooperate to identify the essen-
tial attributes of the input data and determine how to best recon-
struct the original input from the “encoded” state. The AE thus
aims to learn and describe latent attributes of the input data, and
dimensionality reduction can be considered an information-filtering
process.
B. AE as a non-linear generalization of the proper
orthogonal decomposition
Proper Orthogonal Decomposition (POD) is a conventional
method, which is most frequently used in fluid mechanics to analyze
the modal structure of flow fields. POD was introduced by Lumley42
and is also known as the Principal Component Analysis (PCA) pre-
viously defined by Pearson.53 It is interesting to contrast the basic
properties of POD and AE in order to clarify the differences in
performance.
By its very nature and construction, POD modes are orthog-
onal to one another. The original field is thus reconstructed by a
combination of linearly uncorrelated modes. The projection basis is
defined in such a way that most of the variance (energy) is embed-
ded in the first mode, and additional contributions decrease within
the following modes. Thus, most of the signal energy is captured by
the first few modes, unless the fluctuation field is populated with a
wide spectrum of energetic scales, as is the case in highly turbulent
flows at a high Reynolds number.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
POD is intrinsically a linear regression method, and it essen-
tially learns a linear transformation that projects the data into
another space, where vectors of projections are defined by the
variance of the data. The AE methodology is not subject to such
linearity constraints, as data dimensionality is reduced by stack-
ing multiple non-linear transformations. The AE is capable of
modeling complex non-linear functions and consequently is more
effective for describing the underlying manifold structure of the
data.3,5
The manner in which a single-layer AE operates on input data
to give an output that optimally represents the input by way of a
low-order representation of the input data is given by Eq. (2) and
illustrated in Fig. 4. For maximizing the likeness between input and
reconstruction, the AE has to adjust its weights and biases. If h and
g are linear activation functions and no bias is used, then the output
is y = WW ′ x [see also Eq. (3)]. Plaut54 showed that the AE learns to
span the same subspace as POD, where y = WW T x, if the mean-
squared error defined the loss function (J = (x − y)2 ). By using
non-linear activation functions, non-linear trends in the data set
can be identified. Hence, the AE is capable of modeling complex
non-linear functions, and it can be seen as a generalization of the
POD.3,5,12,45,54
To illustrate the difference between PCA and ANN with non-
linear activation functions, an imaginary set of data is prescribed
in Fig. 5. A set of points are organized, broadly, along a curved
path with two lines of different slopes in quadrants II and IV,
11 December 2023 21:54:35
respectively, characterizing the behavior of the data. The objective
is to represent the data as best as possible with minimum effort.
The first POD mode Φ1 captures the maximum variance. In this
two-dimensional example, the first mode is represented by the red
line, and the second mode—the blue line—is perpendicular to it.
Although a large proportion of the information is captured by the
first mode, the second mode is still required for adequate data repre-
sentation. The first mode represents the average of the left and right
“dispersion branches,” revealing the inability of the POD to describe
the non-linear trend, and making the interpretation of this mode
contentious. While a neural network attempts to discover a lower-
dimensional hyperplane, which describes the original data, the AE is
able to learn non-linear manifolds (a manifold is defined, in simple
terms, as a continuous, non-intersecting surface).
The compression performance operated by an AE can be dras-
tically increased with respect to POD by using non-linear activa-
tion functions with bias, and by increasing the number of layers
(depth). However, the more elaborate the AE becomes, the harder
it is to extract the modes, and ultimately to interpret them. This
FIG. 4. Data transformation process operated by a vanilla AE.
32, 067103-6
 Physics of Fluids
means that, by adding layers with a large number of neurons or fil-
ters with non-linear activation functions, the potency of the AE for
function approximation can be efficiently increased, but the nega-
tive aspect is that interpretability of the data structure in the latent
space decreases. Even if the dynamics in the latent space mimics
the fluid dynamics in the original space, data in the latent space
cannot be directly easily related to the physics of the flow. Nev-
ertheless, the motivation of using the AE with a sufficiently high-
level of complexity is the opportunity to work within a space of low
dimension.
C. Definition of a convolutional auto-encoder (CAE)
In the early stages of the development of computer vision, neu-
ral networks were composed of stacked layers of perceptrons, fully
inter-connected to one another. Aside from the vast numbers of
parameters to be determined and the risk of learning information
not connected to relevant input data, such as noise (over-fitting),
the main disadvantage of this technique is that input images or
volume data are flattened, and consequently, shared information
in small neighborhoods cannot be exploited. Convolutional Neural
Networks (CNNs) are much more sophisticated techniques,24,32,36
and they have proven extremely effective in areas such as image
recognition and classification. They comprise convolution layers
with activation functions and pooling layers. The primary purpose
of convolutional layers is to extract “features” from the input image.
Convolution preserves the spatial relationship between pixels by
learning image features using short lines, or small squares or vol-
umes of input data. By considering small portions and using the
shared information between the different points within these areas,
the CNN learns to recognize patterns. To do this, small matrices,
called “filters” or “kernels” in CNN terminology, are spanned across
the whole input, identifying and matching patterns within the whole
input domain. They act as feature detectors from the original input
image. Matrices obtained by sliding filters over the data and com-
puting the dot-product (convolution) are often referred to as “acti-
vation maps” or “feature maps.” The role of convolutional layers is
to identify patterns by tuning the filters and the bias, while the role of
pooling layers is to select only the most important of them by reduc-
ing the dimensionality of each feature map (pooling is also referred
to as sub-sampling or down-sampling).
It can be argued that the CNN pertains to image recognition
and cannot be related to, or exploited for, fluid dynamics. However,
fluid flows, even if turbulent and thus chaotic, contain influential
coherent structures and feature certain aspects of order and orga-
nization. By defining “libraries” of filters able to describe coherent
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
FIG. 5. Conceptual representation of the
projection basis using the PCA method
against a neural network.
scales, the CNN can thus be applied to the identification of patterns
featuring in such flows.10,38
D. Autoencoder cost function: Measure of the
“likeness”
The primary goal of most ML algorithms is to construct a
model able to map input variables by identifying their features to
output variables, referred to as a target. Given a training set, the
objective is to learn a function f, such that f (x) is a “good” pre-
dictor for the corresponding value of y. Whether the predictor is
considered “good” or “bad” is subjective, and it is tied to the prob-
lem hypothesis. In fact, a single data set can be used for different
purposes, and the cost function measuring the model accuracy dur-
ing the training process must be adapted to the objectives in order
11 December 2023 21:54:35
for the ML algorithm to identify which features are relevant to the
objective.
As the AE objective is to reproduce the input data, the cost
function must provide a good measure of the “likeness” between
the input data and their reconstruction. One option is to use the
mean-squared error, as for the PCA. However, by minimizing the
cost function J = (y − x)2 (averaged over the total number of obser-
vations, or data points, by batch), the AE will first learn features
responsible for large fluctuations. This approach may be adequate
for most cases. However, as the fluid-dynamic system is non-linear,
large-amplitude phenomena may have their roots in weak phenom-
ena. Hence, a cost function that is not weighted by the amplitude
fluctuation should be more appropriate. Against this background,
the similarity between the output y and the target value x is quanti-
fied by the parameter α given by the following equation (values being
centered, i.e., xi = xi′ + xi with xi′ = 0):
2x′ y′
α= . (4)
(x′ )2 + (y′ )2
The profile of α against y′ /x′ is shown in Fig. 6(a). Possible val-
ues taken by α span within the limit −1 to 1, obtained when y′ = −x′
and y′ = x′ , respectively. The mean value of α is qualitatively in a
x ′ y′
manner akin to the correlation coefficient γ = √ √ . How-
(x )2 (y′ )2
′
ever, while the latter measures the linear link between fluctuations,
α is more restrictive, as the magnitude is also taken into considera-
tion. For example, if y is linearly correlated with x by the function y′
′
√
= ax , then γ2 is equal to 1 whatever the value of a, while α equals
1 if and only if a = 1; thus, y′ = x′ .
32, 067103-7
 Physics of Fluids ARTICLE scitation.org/journal/phf

In order to effectively use an algorithm that is based on the gra-

dient descent method during the training process for converging to
the solution y′ = x′ , a function j may be derived, using α as follows:
j(α, ε) = α+11+ε − 1+ε
1
. Figure 6(b) shows that j tends to 0 as y′ tends
to x′ . The upper limit of j = 1 can be imposed by using ε = 0.618. As
2

mentioned in Sec. II, during the backpropagation process, the values

of the weight increments are proportional to the sensitivity. The cost
function J is defined such that the gradient can be increased during
the training process, namely,

1 1
J= − . (5)
[ α+1
2
] 2n + ε 1 + ε

Profiles of J for different values of n are shown in Fig. 6(c).

For the present study, there is virtually no difference between fields
reconstructed by an AE trained using this cost function or if the
mean-squared error is used instead. This finding is not surprising, as
the loss of information is almost zero. The cost function introduced
is designed for flows populated by a broad spectrum of eddy sizes;
the advantage of using this method over the standard MSE requires
further investigations.

IV. APPLICATION OF AE TO A TWO-DIMENSIONAL

FLOW AROUND A CYLINDER
A. AE architecture and parameters

11 December 2023 21:54:35

A simulated 2D unsteady flow around a cylinder is used here
as a test case. The flow was computed by the lattice Boltzmann
technique, using a Python code.50 The simulation was performed at
Re = 220, with a two-dimensional nine-velocity square lattice
(D2Q9) implementation of the code. The computational domain is
covered by a mesh of 256 × 88 nodes in the streamwise and spanwise
directions, respectively. A snapshot of the streamwise velocity field
is conveyed in Fig. 8(a).
A summary of the AE architecture and parameters used for the
current study is shown, schematically, in Fig. 7. As the flow struc-
tures are coherent in space and time, volumes of 12 stacked snap-
shots are used as an input tensor, corresponding approximately to
one period of vortex shedding. Both the encoder and the decoder
are built using three 3D convolutional layers [with filter dimensions
given in Fig. 7(b)]. The initialization of the distribution of weights
for each layer relies on a uniform distribution proposed by Glorot
and Bangio.22 By using 3D convolutional layers, the CNN exploits
both spatial and temporal correlations. As shown in Fig. 7, the size of
the input is successively reduced when traversing through the differ-
ent layers of the encoder, and then the input is reflated to its original
size by the decoder. The information propagates from the encoder to
the decoder through a layer composed by three-perceptrons forming
3
the bottleneck. Only 12×256×88 × 100 ≈ 0.001% of the original infor-
mation is conserved in the latent space, and only this information is
used to reconstruct the input.
The ability of the AE to perform non-linear regression is con-
veyed by the “Exponential Linear Unit (ELU)” as the activation func-
FIG. 6. Construction of the cost function J used to quantify the reconstruction error tion for all layers, except for the final layer in which the sigmoid
2x′ y′
between x and y, Eq. (5), where ε = 0.618. Profiles of (a) α = (x′ )2 +(y′ )2 , (b) function is preferred. The ELU is very similar to the well-known
j= 1
α+1
+ε
− 1
1+ε
, and (c) J = 1
[ α+1 ]2 +ε
n − 1
1+ε
. RELU function.48 In fact, for non-negative inputs, both are identi-
2 2
cal. The main difference between the two functions is the way the

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-8

Published under license by AIP Publishing
 Physics of Fluids
FIG. 7. AE architecture and parameters: (a) table and (b) graphical representation.
negative inputs are processed: the ELU varies slowly until its output
equals a negative constant, whereas the RELU becomes sharply null.
In the case of ELU, the neurons never contain zero ordered gradient
even for negative inputs and the training can continue, whatever the
sign of the input values, unlike ANNs using the RELU. This modifi-
cation allows the model using the ELU to give more accurate results
with faster convergence.13
B. Autoencoder: Learning process and reconstruction
The database used for the present study contained around 43
shedding periods, which are sampled over 500 temporal snapshots.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
11 December 2023 21:54:35
A subset that is composed of 128 snapshots (corresponding approx-
imately to ten shedding cycles) is used for the AE training. This
number has been chosen to take full advantage of the TPU hard-
ware, developed by Google. All codes used to obtain the results in
this paper were written using the COLAB platform with version 14
of the Tensorflow library.1
Once the training has been performed, a volume of streamwise
velocity fields is randomly chosen from the validation subset and fed
into the AE. One of the 12 snapshots is shown in Fig. 8(a). The corre-
sponding reconstructed snapshot from the AE is shown in Fig. 8(b).
For measuring the reconstruction error, the parameter ϕ given by
the following equation is introduced:
32, 067103-9
 Physics of Fluids ARTICLE scitation.org/journal/phf

11 December 2023 21:54:35

FIG. 8. Snapshot of the streamwise velocity field for a 2D flow around a cylinder: (a) original field; (b) and (c) reconstructed snapshots from the AE and three POD modes,
respectively; (d) and (e) reconstruction error defined by ϕ2 [Eq. (6)]; (f) probability-weighted distribution of ϕ2 .

√
n n
(Urec (x, z) − Uori (x, z))2 Fig. 8(d). A visual comparison of the input image with the recon-
ϕn (x, z) = 100 × n , (6) structed one demonstrates the high-fidelity of the reconstruction
Uori (x, z)
process, with errors occurring mainly in the region just behind the
where U ori and U rec are the original and reconstructed fields, respec- cylinder. The error margin increases up to ϕ2 = 15. However, the
tively. The reconstruction error defined by ϕ2 (x, z) is conveyed in probability-weighted distribution of ϕ2 , plotted in blue in Fig. 8(f),

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-10

Published under license by AIP Publishing
 Physics of Fluids
FIG. 9. Temporal fluctuations of the space-averaged reconstruction errors ⟨ϕ2 ⟩x,z
[Eq. (6)]: blue line, POD reconstruction (three first modes); red and green lines, AE
reconstructions from training and validation subsets.
reveals that the error magnitudes are mainly below 4, the mean
value of ϕ2 (x, z) can be estimated by integrating the curve (⟨ϕ2 ⟩x,z
= ∫ ϕ2 Pdf (ϕ2 )dϕ2 ), and it is in the vicinity of 0.7. The information
loss is, therefore, insignificant. This result is impressive, especially
since the input data have been compressed by more than 99.998%;
this demonstrates the ability of the AE to extract the most impor-
tant features of the dynamical system and to provide an extremely
FIG. 10. Identification process of a low-dimensional dynamical model by using the AE combined with the clustering algorithm.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
low-dimensional representation. In what follows next, the AE recon-
struction is contrasted with that achieved with Proper Orthogo-
nal Decomposition (POD). The application of this method results
in a modal decomposition of the input data, with modes ordered
according to their energy content. Once the POD modes have been
obtained, these can be recombined to give an approximate repre-
sentation of the original flow. The more the modes used for the
reconstruction, the lower the reconstruction error. Hence, the effi-
ciency of the POD may be compared to that of the corresponding
reconstruction obtained with the AE. For the same flow analyzed
above with the AE, the streamwise velocity field reconstructed from
the modes (eigenvectors) and the associated error fields are shown in
Figs. 8(c) and 8(e), respectively. As the POD’s temporal coefficients
describe the fluid dynamics, and because the AE with a bottleneck
dimension of 3 yields only three temporal coefficients, only three
POD modes will be used for the reconstruction. The principal fea-
tures of the original field, mainly associated with the large vortices
being shed, are evidently reproduced by the POD. However, the
error is much larger than the one obtained with the AE, especially
behind the cylinder where the structures, although being relatively
small, may play an essential role in the dynamics of the shedding
process. In Fig. 8(f), the probability-weighted distribution function
of ϕ2 for the POD is given by the blue line, and the error level is
seen to be much greater than the maximum level obtained with the
AE reconstruction. This set of figures thus brings into focus how the
11 December 2023 21:54:35
32, 067103-11
 Physics of Fluids ARTICLE scitation.org/journal/phf

POD reconstruction is outperformed by the AE. Most importantly,

the AE can extract the large-scale and small-scale features, despite
the massive compression in the latent space.
The temporal evolution of the spatial average of the error ϕ2
obtained with the AE (red and green lines) and the POD (blue line)
is plotted in Fig. 9. The red portion of line for the AE corresponds
to the training data. The green line represents the reconstruction
errors obtained when a random field is fed into the AE after the
learning process. In both cases, the error margins are low and very
similar in level. This result is important, as it signifies that the AE
has not learned undue details and noise only present in the train-
ing subset, i.e., there is no “overfitting”—a condition that would be
indicated by a lower error in the reconstruction of the training data.
In comparison with the AE result, the average and the variance of
the reconstruction error for the POD are almost ten times higher,
thus demonstrating the superiority of the AE for finding an optimal
representation.

V. FROM FLOW FEATURES TO LOW-DIMENSIONAL

DYNAMICAL MODELS
Fluid-mechanics problems of engineering interest are gener-
ally characterized by dynamical systems of very high dimension
and strong non-linearity. Approximating the flow dynamics by low-
dimensional models is then extremely challenging in most practi-
cally relevant cases. Whatever the approach used for modeling the
flow, the first step is to determine which features are the most impor-

11 December 2023 21:54:35

tant and have to be conserved for predicting the flow. The POD,
or its variants such as spectral POD, offers a path for discover-
ing features based on the energy distribution. Indeed, as previously
mentioned, the POD identifies the best-fitting hyperplanes though
the input variables using linear regression, POD modes are by con-
struction orthogonal. The AE offers more “flexibility” for extract-
ing the most important features relevant to the dynamics, as the
cost function can be adapted to the problem. The AE also offers a
superior route for modeling non-linear dynamic systems as it can
identify non-linear relationships. By leveraging the AE capacity to
identify and represent in a compact manner the important flow fea-
tures, the objective of this section is to determine a probabilistic
low-dimensional dynamical system. In the present study, the dimen-
sionality reduction is performed in five steps (see the illustration in
Fig. 10):
● In the first step, a low-dimensional projection subspace is
identified by using the AE.
● In the second step, this low-dimensional representation is
exploited to efficiently cluster the data set by similarity
principles.
● In the third step, a dynamical model describing the transi-
tion probabilities from one cluster to another is derived.
● Finally, in steps 4 and 5, the decoder layer is used to deter-
mine a low-dimensional dynamical system in the original
data space.
These approaches are detailed in the following paragraphs.

A. Low-dimensionality projection using AE
The requirement that the latent-space dimensions should be
substantially lower than the input tensor implies that the encoder
FIG. 11. (a)–(c) Plots showing a variable in the latent space from different perspec-
tives. Each input volume is mapped to a single black dot point. The first 11 data
volumes are represented by the colored dots, ranging from blue to red.

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-12

Published under license by AIP Publishing
 Physics of Fluids ARTICLE scitation.org/journal/phf

FIG. 12. Conceptual representation of clustering methods

based on “distance” against “connectivity.”

and decoder must be able to identify and keep the most meaningful
features. As the flow structures are coherent in space and time, a vol-
ume of 12 stacked snapshots is used as an input tensor, 12 snapshots
corresponding to the vortex-shedding period. The encoder converts
the input volume of 12 × 256 × 88 points to encoded features embed-
ded within a three-component vector. Each data-volume input is
thus reduced to a single point in the 3D latent-space representa-
tion. The projection of the volumes is conveyed by the black dots
in Fig. 11, and the first 11 volumes are represented by the colored
dots, sequentially arranged from blue to red. The encoded features
form a loop, where a revolution is equivalent to a shedding cycle.
Although the physical flow features cannot be interpreted directly
from the components in the latent space, the compact representa-
tion nevertheless reflects a spatio-temporal organization. The low-
dimensionality of the latent space can be leveraged for represent-
ing the underlying temporal dynamics by processing elements or
groups of elements, which will then be used by the decoder to yield
a physical representation of the original field.

11 December 2023 21:54:35

B. Spectral clustering: Grouping similar events
One path to reducing the problem dimensionality is to group
similar elements or events, and to approximate them by a “mean”
representation. This approach raises several questions: Which flow
features are the most relevant for describing the fluid dynamics?
How can these features be related to a criterion upon which the simi-
larity is based? How can similarity be measured? How should a mean
representation be determined? There are no unambiguous answers
to these questions, and it is only by an iterative process, based on
knowledge and discovery, that answers arise.
Several methods exist for grouping elements (i.e., “cluster-
ing”). They can be arranged, broadly, in two groups: (i) based on
their “distance” to neighbors—the “K-means” clustering method
being probably the most popular algorithm, and (ii) based on their
connectivity—in which case points widely separated may lie in the
same cluster and points even extremely close can be split into differ-
ent clusters. Spectral clustering is one method, among others, based
on the principle of connectivity. To increase clarity, one may con-
sider points distributed in such a way that they form two parallel data
segments (Fig. 12), with all data being clustered in two ways, each in
one of the two above categories. With the first method, based on the
distance, points from both lines are contained in each cluster; their
inclusion is in cluster 1 or cluster 2 depending upon the position of
the points relative to the centers of the segments: points on the left
are in cluster 1 and points on the right are in cluster 2. In the sec-
ond method, based on the principle of connectivity between points,
items that share a common attribute—in this case, position within
an identifiable coherent region, belong to the same cluster. FIG. 13. Spectral clustering of data projection into the latent space. Data are dis-
tributed into eight different clusters. The three plots (a)–(c) show the same latent
The most significant flow features are automatically selected
variables from different perspectives.
by the AE. The encoder layer converts the input data to encoded

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-13

Published under license by AIP Publishing
 Physics of Fluids ARTICLE scitation.org/journal/phf

features embedded within an optimal subspace. By combining this
low-dimensional representation of the data and the spectral cluster-
ing algorithm, a coarse-grained description of the data is defined.
Results from spectral clustering are shown in Fig. 13, each cluster
being represented by a different color. The number of clusters is
constrained to be 8, and as shown in the histogram in Fig. 14(a),
the events are equally distributed over all the states (clusters).
cluster is associated with a flow state, forming the “skeleton” of the
flow-dynamical system. This system can then be represented as a
probabilistic Markov chain, corresponding to a stochastic model,
which describes a sequence of possible events in which the probabil-
ity of each event depends only on the state attained in the previous
event, following the cluster-based reduced-order modeling (CROM)

C. Low-dimensional dynamical model construction

By exploiting the ability of the AE to represent the most rele-
vant flow-dynamic features in a low-dimensional space, in combina-
tion with a spectral clustering algorithm for regrouping events based
on their connectivity, a low-dimensional system—its dimension
being fixed by the number of clusters, can be determined. Each

11 December 2023 21:54:35

FIG. 14. Probabilistic clustered dynamical model. Probability values are estimated
in percentage terms and given by the integers. (a) Probabilities for the flow to FIG. 15. Low-dimensional dynamical model: (a) map of migration probabilities
stay in its current configuration (same cluster) and (b) probabilities for the flow to between modes and (b) modes reconstructed from clusters by the decoder.
navigate from one state to another. Integers in figure (a) correspond to modes shown in figure (b).

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-14

Published under license by AIP Publishing
 Physics of Fluids
framework introduced in Kaiser et al.29 Thus, it is necessary to deter-
mine the probability of a flow configuration remaining in its present
state, Fig. 14(a), and the probability of the flow migrating toward
another state, the map of the motions between clusters being shown
in Fig. 14(b).
The flow configuration used in this study is relatively simple,
and the low-dimensional dynamical system is depicted in Fig. 15.
The skeleton of the low-dimensional dynamical system is obtained
using the decoder for reconstructing elements from clusters in the
latent space and is shown in Fig. 15(b). The dynamics of the sys-
tem are illustrated by the orientation and thickness of the arrows in
Fig. 15(a).
In this section, it was shown that a model, making probabilis-
tic prediction possible, can be derived by clustering elements from
the latent space obtained by the AE. This approach can be useful
for providing insight into which control strategy to adopt, by either
promoting some modes rather than others or undermining modes
leading the flow to the disadvantageous configuration (such as drag
increase47 ).
VI. PREDICTION OF TEMPORAL EVOLUTION:
DETERMINISTIC APPROACH
In Sec. V, a low-dimensional model of the flow dynamical sys-
tem was obtained using an AE in combination with spectral clus-
tering. Importantly, the flow prediction is probabilistic—providing
an approximation of the flow state. However, the AE can also be
FIG. 16. Illustration of the temporal prediction process using the AE: upper row, training; lower row, prediction.
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
used to make “conventional,” deterministic, predictions of the flow
evolution, based on a preceding learning process.
Intuitively, one approach is to use as input a snapshot at a spe-
cific time step and “ask” the AE to construct the following time-step
snapshot, using the latter as output—an approach similar to the
dynamic mode decomposition. In fact, if the activation functions
are linear and MSE is used as the cost function, the AE is a linear
operator, which should be similar to the DMD.66 However, the AE
is also capable of performing non-linear transformation if the activa-
tion functions are non-linear. Therefore, the “time-lagged AE” can
be seen as the generalization of the DMD.
Another possible approach, illustrated in Fig. 16, is to per-
form the prediction in the latent space, taking advantage of the low
dimensionality, in which case only the dynamics of the most relevant
features are conserved. Modeling the evolution of latent variables is
easier and more cost-effective, as with this approach, the evaluation
of new steps scales with the size of the low-dimensional represen-
tation and not with the size of the full-dimensional data. The first
step is to train the AE to learn how to extract and compact the most
valuable information associated with the flow dynamics in the latent
space. Once this task is completed, the following step is to predict the
temporal evolution of the low-dimensional system by using a Con-
volutional Neural Network (CNN) in the latent space, which aims
to extract the temporal evolution. As is indicated in Fig. 16, once
the training phase is completed, the combination of the AE with the
CNN can be used to predict the flow evolution from an initial con-
11 December 2023 21:54:35
dition. A data volume at t r represents the initial condition, and this
is fed into the AE. The second CNN uses its projection in the latent
32, 067103-15
 Physics of Fluids ARTICLE scitation.org/journal/phf

space to predict the following time step (at t r +1 ). The vector obtained
is fed into the CNN in order to predict the time step t r +2 , and this is
repeated within a closed loop. The vectors obtained are decoded “on
the fly” for constructing the time-lagged data volumes. The CNN
used comprises five convolutional layers with mono-dimensional fil-
ters, the input being time-series data covering 12 time steps in the
latent space.
One of the most useful ways to uncover the structure in high-
dimensional data is to project it down to a subspace, such as a 2D
plane, where hidden features may become visible. Of course, the
challenge is to find the plane that best illuminates the data structure.
In the present test application, the above process is imple-
mented in order to predict up to 20 shedding cycles. The low-
dimensional data predicted in the latent space are compared with the
original encoded data in Fig. 17. The left-hand side plots show the
time evolution for the three latent space dimensions. The predicted
variations (blue lines) virtually collapse onto the variations result-
ing from feeding the actual fields into the latent space (red lines).
However, closer examination reveals differences in the peaks of the
latent-space vector and also reveals a lag between the prediction and
the actual values [see Fig. 17(b)]. In the phase plot in Fig. 17(d), every
projection of the original event over the 20 cycles is represented by
a black dot, forming a near-continuous loop. Prediction results are
compared to original projection at four selected time steps: ◯ t/T 0
≈ 4, △ t/T 0 ≈ 7, ◽ t/T 0 ≈ 15, and ◇ t/T 0 ≈ 19, with T 0 being the period
for one shedding cycle. At t/T 0 ≈ 4, i.e., after four vortex-shedding

11 December 2023 21:54:35

periods, the predicted and the encoded data are indicated by the blue
and red symbols, respectively, showing that the lag between the pre-
dicted and the actual set of latent-space values increases. However,
the figure shows that, notwithstanding the lag, the predicted points
are still part of “the correct orbit,” implying that the predicted flow
field should essentially be accurate, except for a slight lag.
The accuracy of the prediction is demonstrated in Fig. 18,
which compares the predicted with the actual flow fields at four dif-
ferent time steps. For each of these values, the predicted velocity field
is reconstructed by the AE/CNN, and a visual comparison with the
original field highlights the prediction accuracy. As seen, even after
19 cycles, the differences between original and predicted fields are
visually insignificant.
Using Eq. (6), the prediction error ϕ2 [Eq. (6)] can be quanti-
fied, and this is represented by the blue line in Fig. 19(a), in com-
parison with the AE-reconstruction from Fig. 9 (red line). Over the
first predicted cycle, the error is contained within the range between
9% and 16%, the mean value being around 12%. The error mar-
gin steadily increases with the number of cycles, the error level
varying between 10% and 50% during the cycle. This increase in
error appears to be at odds with the visually insignificant differ-
ences between the flow fields in Fig. 18, but this is a consequence
of the increasing lag indicated in Fig. 17. In order to evaluate the
effect of the time lag on the spatial prediction accuracy, the error in
Fig. 19(b) is estimated between the predicted field at t/T 0 ≈ 16.71
and time-lagged fields; a massive drop from 50% to less than 10%
is observed when the original field is shifted to the earlier time
step.
The use of the AE, for extracting and compacting the informa- FIG. 17. Comparison between the original data (red) and the AE prediction (blue)
tion to a low-dimensional representation, in combination with the within the embedded space; (a)–(c) temporal evolution along the three latent com-
ponents; (d) projection in the 3D latent space at four different time steps t/T 0 : ◯
CNN for the temporal prediction in the latent space, can be claimed ≈ 4, △ ≈ 7, ◽ ≈ 15, and ◇ ≈ 19.
to provide an effective method for predicting the flow. However,

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-16

Published under license by AIP Publishing
 Physics of Fluids
while the general spatial features are well predicted, the precise tem-
poral evolution suffers from the phase lag discussed above. The accu-
racy of the CNN can probably be improved by tuning its architecture
and the training parameters.
To conclude, the implicit assumption behind dimensional-
ity reduction is the existence of relationships between features of
high-dimensional data, i.e., data points lying on, or being close to,
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
FIG. 18. Velocity fields predicted by the
AE combined with a CNN in the latent
space alongside of the original plots after
(a) 4 cycles, (b) 7.6 cycles, (c) 14.8
cycles, and (d) 19.3 cycles.
11 December 2023 21:54:35
a smooth manifold in the high-dimensional input space. By identi-
fying dependencies between data in a high-dimensional space, man-
ifolds on which data live on are learned by the AE and then lever-
aged for computing a low-dimensional embedding of the underlying
manifold. Input data are represented by the AE in a more com-
pact coordinate system in which manifolds are “unfolded.” As only
essential information is conserved and represented in lower-space
32, 067103-17
 Physics of Fluids
FIG. 19. Measure of the error between the predicted snapshot and the ground-
truth field: (a) time evolution of the prediction error and (b) error between the
reconstructed field at t/T 0 ≈ 16.71 and time-shifted snapshots.
dimensions, the dynamics of the latent variables can be more easily
discovered and then leveraged to cost-effectively predict the flow. In
the present work, the CNN provides the evolution of the dynamical
system in the low-dimensional coordinate system. However, other
data-driven methods can also be used, for example, long short-term
memory networks27 (LSTMs), which are capable of learning long-
term dependencies, connecting previous time-delayed information
to the current time step. Several recent works23,30,44 combining the
AE with LSTMs prove to be an excellent approach for flow predic-
tion. By combining an AE with another data-driven method called
SINDy, the method introduced by Brunton et al.,8 Champion et al.11
were able to unveil governing equations from raw data.
VII. SPARSE RECONSTRUCTION
Gathering high-resolution data, in space and time, requires a
significant amount of resource. In recent years, several methods have
been developed for reconstructing “high-quality” data from con-
taminated and/or under-sampled data. Particle Image Velocimetry
(PIV) is an example that illustrates resolution limitations when seek-
ing high spatio-temporal resolution. An interesting study carried
out by Rabault et al.56 shows how a simple ANN can carry out an
end-to-end PIV, and it could become a convincing alternative to the
traditional PIV algorithm.
The AE method offers a new way for reconstructing high-
resolution fields from data collected from a few sensors. The sparse-
reconstruction process is shown schematically in Fig. 20. In the
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
FIG. 20. Illustration of the process of sparse reconstruction using the AE and
CNN.
present application, this approach is exemplified by data collected
with sensors at three locations relatively far behind the cylinder,
as shown in Fig. 21. The first step is a two-stage training process.
In the first stage, the AE learns to compress information in a low-
dimensional space, and in the second phase, the CNN learns how
11 December 2023 21:54:35
to predict the latent-space vector by taking the sensed data as input.
Once both the AE and the CNN are trained, data collected by sen-
sors can be fed into the CNN and converted to data “intelligible” by
the decoder, which can then reconstruct the full-volume data.
Figure 22(a) shows three velocity fields: the left-hand side plot
represents the raw field taken for the last time step available, i.e.,
more than 30 cycles after the last training-set snapshot, the field in
the middle is obtained by feeding the raw field into the AE only, and
the right-hand side plot shows the velocity field created by the CNN
and the decoder from the sensor measurements. The reconstructed
field is seen to be close to the raw field, demonstrating the CNN’s
capacity to provide accurate prediction in the latent space.
The error Φ2 between the raw field and the sparse reconstruc-
tion is indicated by the blue line in Fig. 22(b). The average error
is below 1%, which is comparable to the error obtained with the
AE-reconstructed field in the training subset represented by the red
line.
For the present periodic dynamical system, the AE may also
be an effective means for recovering high-frequency information.
FIG. 21. Flow visualization with the location of sensors used for the sparse
reconstruction represented by the red points.
32, 067103-18
 Physics of Fluids
FIG. 22. Visualization of the sparse-reconstruction accuracy: (a) velocity fields for
the vortex-shedding problem; from left to right: raw field, AE reconstruction, and
AE/CNN sparse reconstruction; (b) error estimate between the reconstructed and
original fields, defined by Φ2 [see Eq. (6)]; red line, field reconstructed with the AE;
blue line, sparse reconstruction with AE/CNN/sensors.
For the present flow configuration, the sampling frequency is
around 11/T 0 , where T 0 represents one shedding cycle. The low-
dimensional representation in the latent space constitutes a loop
formed by the colored circles in Fig. 11, for which a full revolution
is completed over one shedding cycle. As more and more volume
snapshots are projected into the latent space, points add up suc-
cessively (represented by the tiny black dots in Fig. 11), reducing
the gap between them for ultimately forming a continuous line; this
gap can be associated with the time delay between two snapshots.
Thus, as the time gap between successive points in the latent space
reduces, the high-frequency resolution of one shedding cycle can be
recovered by successively decoding all points forming the loop.
VIII. CONCLUSIONS
Present-day experimental and numerical-simulation tech-
niques provide an unprecedented volume of extremely detailed data
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
on many practically relevant flows. As a consequence, the num-
ber and the size of databases have experienced phenomenal growth,
and this abundance of data poses serious challenges to data min-
ing and post-processing that requires appropriate tools. This paper
has exposed some recent developments in machine learning that
are beginning to contribute thoughtfully to a solution of these
challenges.
Deep learning methodologies in fluid dynamics form a rapidly
growing field of research, the Auto-Encoder (AE) technology being
an important segment of the ML area. AE architectures are used to
compress and decompress high-dimensional data. By reducing the
dimensionality of the data and conserving only the most essential
information, AE systems offer a new path to deriving a representa-
tion of the flows that is more compact and efficient than the native
data sets in the original domains. Unlike the principal component
analysis algorithm, the AE is capable of modeling complex non-
linear functions and consequently is more effective for unveiling
manifold structures in the data. The automated extraction of flow
features by unsupervised learning algorithms, such as the AE com-
bined with clustering, offers innovative methods for flow modeling
and control using low-dimensional dynamical models.
The main messages of this study are as follows:
1. A probabilistic low-dimensional dynamical model can be built
by applying a clustering algorithm in the latent space, allowing
for a probabilistic flow prediction at low cost, which can also
provide insight into flow-control strategies to be implemented.
11 December 2023 21:54:35
2. An accurate prediction of the flow evolution can be performed
by using a second neural network in the latent space.
3. High-resolution spatio-temporal information can be retrieved,
using a second neural network as a transfer function from sen-
sors to corresponding data representation in the latent space,
and by accumulating snapshots.
By choosing a relatively simple test case, this study aims to show
that ML algorithms do not necessarily have to be “black boxes.”
They can be used for extracting the physical principles and struc-
tures underlying the data to obtain interpretable models, as shown
by other studies undertaken in a broad range of scientific fields
(biomolecular dynamics,62 astronomy,28 economical and social sci-
ences,19 biology,12 and quantum mechanics58 among others). For
more complex flows, the task of identifying interpretable and robust
models is more challenging. This will require more sophisticated AE
architectures, such as a variational autoencoder, and more impor-
tantly, ML algorithms will have to be physics-informed, for example,
by embedding the mass-conservation law (see Ref. 46) directly into
the AE’s architecture. AEs are more challenging to implement and
use than other data-driven methods (such as POD and DMD). How-
ever, AEs combined with other ML algorithms hold great promise
for mapping high-dimensional strongly non-linear problems into
low-dimensional (approximately) linear ones. A low-dimensional
dynamical system can then be leveraged to predict the flow and to
design control laws more effectively.
ACKNOWLEDGMENTS
The author would like to thank Michael Leschziner for prag-
matic and fruitful discussions and is deeply grateful for his support
32, 067103-19
 Physics of Fluids
in allowing him to conduct this work under favorable working con-
ditions. The author is also grateful for the insightful comments
offered by Laurent Cordier, Nicholas Hutchins, and the anonymous
peer reviewers. The generosity and expertise of one and all have
improved this study in innumerable ways.
DATA AVAILABILITY
Data and codes that support the findings of this study are avail-
able on Github: github.com/LionelAgo/Vortex_AE. All codes used
to obtain the results in this paper were written using the COLAB
platform with version 14 of the Tensorflow library.1
REFERENCES
1
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghe-
mawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G.
Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu,
and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in 12th
USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)
(USENIX Association, Savannah, GA, 2016), pp. 265–283.
2 New York, 1987), Vol. 3, pp. 11–14.
S. Bagheri, L. Brandt, and D. S. Henningson, “Input-output analysis, model
reduction and control of the flat-plate boundary layer,” J. Fluid Mech. 620,
263–298 (2009).
3 S. Jamal and J. S. Bloom, “On neural architectures for astronomical time-series
P. Baldi and K. Hornik, “Neural networks and principal component analysis:
Learning from examples without local minima,” Neural Networks 2(1), 53–58
(1989).
4 J. Östh, S. Krajnović, and R. K. Niven, “Cluster-based reduced-order modelling of
N. Benard, J. Pons-Prats, J. Periaux, G. Bugeda, P. Braud, J. P. Bonnet, and
E. Moreau, “Turbulent separated shear flow control by surface plasma actuator:
Experimental optimization by genetic algorithm approach,” Exp. Fluids 57(2), 22
(2016).
5
H. Bourlard and Y. Kamp, “Auto-association by multilayer perceptrons and
singular value decomposition,” Biol. Cybernetics 59(4-5), 291–294 (1988).
6
M. P. Brenner, J. D. Eldredge, and J. B. Freund, “Perspective on machine learning
for advancing fluid mechanics,” Phys. Rev. Fluids 4(10), 100501 (2019).
7
S. L. Brunton, B. R. Noack, and P. Koumoutsakos, “Machine learning for fluid
mechanics,” Annu. Rev. Fluid Mech. 52, 477 (2019).
8
S. L. Brunton, J. L. Proctor, and J. N. Kutz, “Discovering governing equations
from data by sparse identification of nonlinear dynamical systems,” Proc. Natl.
Acad. Sci. U. S. A. 113(15), 3932–3937 (2016).
9
A. E. Bryson, Applied Optimal Control: Optimization, Estimation and Control
(CRC Press, 1975).
10
S. Cai, S. Zhou, C. Xu, and Qi Gao, “Dense motion estimation of particle images
via a convolutional neural network,” Exp. Fluids 60(4), 73 (2019).
11
K. Champion, B. Lusch, J. N. Kutz, and S. L. Brunton, “Data-driven discovery
of coordinates and governing equations,” Proc. Natl. Acad. Sci. U. S. A. 116(45),
22445–22451 (2019).
12
D. Chicco, P. Sadowski, and P. Baldi, “Deep autoencoder neural networks for
gene ontology annotation predictions,” in Proceedings of the 5th ACM Conference
on Bioinformatics, Computational Biology, and Health Informatics (ACM, 2014),
pp. 533–540.
13
D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate
deep network learning by exponential linear units (ELUs),” arXiv:1511.07289
(2015).
14
M. Couplet, C. Basdevant, and P. Sagaut, “Calibrated reduced-order pod-
galerkin system for fluid flow modelling,” J. Comput. Phys. 207(1), 192–220
(2005).
15
P. A. Davidson, Turbulence: An Introduction for Scientists and Engineers
(Oxford University Press, 2003).
16
J. Derber and A. Rosati, “A global oceanic data assimilation system,” J. Phys.
Oceanogr. 19(9), 1333–1347 (1989).
17
T. Duriez, S. L. Brunton, and B. R. Noack, Machine Learning Control-Taming
Nonlinear Dynamics and Turbulence (Springer, 2017).
Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906
Published under license by AIP Publishing
ARTICLE scitation.org/journal/phf
18
B. F. Farrell and P. J. Ioannou, “Stochastic forcing of the linearized Navier-
Stokes equations,” Phys. Fluids A 5(11), 2600–2609 (1993).
19
I. Foster, R. Ghani, R. S. Jarmin, F. Kreuter, and J. Lane, Big Data and Social
Science: A Practical Guide to Methods and Tools (CRC Press, 2016).
20
P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem, “A
review on deep reinforcement learning for fluid mechanics,” arXiv:1908.04127
(2019).
21
N. Gautier, J.-L. Aider, T. Duriez, B. R. Noack, M. Segond, and M. Abel,
“Closed-loop separation control using machine learning,” J. Fluid Mech. 770,
442–457 (2015).
22
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedfor-
ward neural networks,” in Proceedings of the Thirteenth International Conference
on Artificial Intelligence and Statistics (AISTATS, 2010), pp. 249–256.
23
F. J. Gonzalez and M. Balajewicz, “Deep convolutional recurrent autoen-
coders for learning low-dimensional feature dynamics of fluid systems,”
arXiv:1808.01346 (2018).
24
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning (MIT Press, 2016).
25
P. Hall and S. Sherwin, “Streamwise vortices in shear flows: Harbingers of
transition and the skeleton of coherent structures,” J. Fluid Mech. 661, 178–205
(2010).
26
R. Hecht-Nielsen, “Kolmogorov’s mapping neural network existence theorem,”
in Proceedings of the International Conference on Neural Networks (IEEE Press,
27
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput.
9(8), 1735–1780 (1997).
28
classification,” arXiv:2003.08618 (2020).
29
E. Kaiser, B. R. Noack, L. Cordier, A. Spohn, M. Segond, M. Abel, G. Daviller,
11 December 2023 21:54:35
a mixing layer,” J. Fluid Mech. 754, 365–414 (2014).
30
R. King, O. Hennigh, A. Mohan, and M. Chertkov, “From deep to physics-
informed learning of turbulence: Diagnostics,” arXiv:1810.07785 (2018).
31
A. N. Kolmogorov, “On the representation of continuous functions of many
variables by superposition of continuous functions of one variable and addition,”
in Doklady Akademii Nauk (Russian Academy of Sciences, 1957), Vol. 114, pp.
953–956.
32
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep
convolutional neural networks,” Commun. ACM 60(6), 84–90 (2017).
33
J. N. Kutz, “Deep learning in fluid dynamics,” J. Fluid Mech. 814, 1–4 (2017).
34
Y. Le Cun, “Learning process in an asymmetric threshold network,” in Disor-
dered Systems and Biological Organization (Springer, 1986), pp. 233–240.
35
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature 521(7553),
436–444 (2015).
36
Y. LeCun et al., “Generalization and network design strategies,” Connectionism
in Perspective (Citeseer, 1989), Vol. 19.
37
Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski, “A theoretical framework
for back-propagation,” in Proceedings of the 1988 Connectionist Models Summer
School (Morgan Kaufmann, CMU, Pittsburgh, Pa, 1988), Vol. 1, pp. 21–28.
38
Y. Lee, H. Yang, and Z. Yin, “PIV-DCNN: Cascaded deep convolutional
neural networks for particle image velocimetry,” Exp. Fluids 58(12), 171
(2017).
39
R. Li, B. R. Noack, L. Cordier, J. Borée, and F. Harambat, “Drag reduction
of a car model by linear genetic programming control,” Exp. Fluids 58(8), 103
(2017).
40
R. Li, B. R. Noack, L. Cordier, J. Borée, E. Kaiser, and F. Harambat, “Linear
genetic programming control for strongly nonlinear dynamics with frequency
crosstalk,” Arch. Mech. 70(6), 505–534 (2018).
41
M. Luhar, A. S. Sharma, and B. J. McKeon, “Opposition control within the
resolvent analysis framework,” J. Fluid Mech. 749, 597–626 (2014).
42
J. Lumley, “The structure of inhomogeneous turbulent flows,” in Army Turbu-
lence Radio Wave Propagation, edited by A. M. Yaglom and V. I. Tatarsky (Nauka,
1967), pp. 166–178.
43
B. Lusch, J. N. Kutz, and S. L. Brunton, “Deep learning for universal linear
embeddings of nonlinear dynamics,” Nat. Commun. 9(1), 4950 (2018).
32, 067103-20
 Physics of Fluids ARTICLE scitation.org/journal/phf

44
R. Maulik, B. Lusch, and P. Balaprakash, “Reduced-order modeling of
advection-dominated systems with recurrent neural networks and convolutional
autoencoders,” arXiv:2002.00470 (2020).
45
M. Milano and P. Koumoutsakos, “Neural network modeling for near wall
turbulent flow,” J. Comput. Phys. 182(1), 1–26 (2002).
46
A. T. Mohan, N. Lubbers, D. Livescu, and M. Chertkov, “Embedding hard
physical constraints in neural network coarse-graining of 3d turbulence,”
arXiv:2002.00021 (2020).
47
A. G. Nair, C.-A. Yeh, E. Kaiser, B. R. Noack, S. L. Brunton, and K. Taira,
“Cluster-based feedback control of turbulent post-stall separated flows,” J. Fluid
Mech. 875, 345–375 (2019).
48
V. Nair and G. E. Hinton, “Rectified linear units improve restricted Boltz-
mann machines,” in Proceedings of the 27th International Conference on Interna-
tional Conference on Machine Learning, ICML’10 (Omnipress, Madison, WI, USA,
2010), pp. 807–814.
49
See https://www.oreilly.com/content/a-look-at-deep-learning-for-science/ for
information about deep learning technologies apply to science.
50
See https://palabos.unige.ch/lattice-boltzmann/lattice-boltzmann-sample-codes-
various-other-programming-languages for information about the code.
51
S. E. Otto and C. W. Rowley, “Linearly recurrent autoencoder networks for
learning dynamics,” SIAM J. Appl. Dyn. Syst. 18(1), 558–593 (2019).
52
V. Parezanović, J.-C. Laurentie, C. Fourment, J. Delville, J.-P. Bonnet, A. Spohn,
T. Duriez, L. Cordier, B. R. Noack, M. Abel et al., “Mixing layer manipulation
experiment,” Flow, Turbul. Combust. 94(1), 155–173 (2015).
53
K. Pearson, “LIII. On lines and planes of closest fit to systems of points in space,”
London, Edinburgh Dublin Philos. Mag. J. Sci. 2(11), 559–572 (1901).
54
E. Plaut, “From principal subspaces to principal components with linear autoen-
coders,” arXiv:1804.10253 (2018).
55
S. B. Pope, Turbulent Flows (IOP Publishing, 2001).
56
J. Rabault, J. Kolaas, and A. Jensen, “Performing particle image velocimetry
using artificial neural networks: A proof-of-concept,” Meas. Sci. Technol. 28(12),
125301 (2017).
57
J. Rabault, U. Reglade, N. Cerardi, M. Kuchta, and A. Jensen, “Deep rein-
forcement learning achieves flow control of the 2d Karman vortex street,”
arXiv:1808.10754 (2018).
58
A. Rocchetto, E. Grant, S. Strelchuk, G. Carleo, and S. Severini, “Learning hard
quantum distributions with variational autoencoders,” npj Quantum Inf. 4(1), 1–7
(2018).
59
F. Rosenblatt, “The perceptron: A probabilistic model for information storage
and organization in the brain,” Psychol. Rev. 65(6), 386 (1958).
60
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by
back-propagating errors,” Nature 323(6088), 533–536 (1986).
61
A. L. Samuel, “Some studies in machine learning using the game of checkers,”
IBM J. Res. Dev. 3(3), 210–229 (1959).
62
H. Sidky, W. Chen, and A. L. Ferguson, “Machine learning for collective variable
discovery and enhanced sampling in biomolecular simulation,” Mol. Phys. 118,
e1737742 (2020).
63
R. Sutton and A. Barto, Reinforcement Learning: An Introduction (MIT Press,
2011).
64
H. Tennekes and J. L. Lumley, A First Course in Turbulence (MIT Press, 1972).
65
N. G. Van Kampen, “Determinism and predictability,” Synthese 89(2), 273–281
(1991).
66
C. Wehmeyer and F. Noé, “Time-lagged autoencoders: Deep learning of slow
collective variables for molecular kinetics,” J. Chem. Phys. 148(24), 241703 (2018).
67
P. Werbos, “Beyond regression: New tools for prediction and analysis in the
behavioral sciences,” Ph.D. dissertation (Harvard University, 1974).
68
X. Zou, I. M. Navon, and J. Sela, “Control of gravitational oscillations in
variational data assimilation,” Mon. Weather Rev. 121(1), 272–289 (1993).

11 December 2023 21:54:35

Phys. Fluids 32, 067103 (2020); doi: 10.1063/5.0012906 32, 067103-21

Published under license by AIP Publishing