Use of GPU to simulate dispersion in a porous media with

Smoothed Particle Hydrodynamics

Carlos Alvarado-Rodríguez1, Jaime Klapp2,3, Estela Mayoral3

1
Universidad of Guanajuato, División de Ciencias Naturales y Exactas, Noria Alta s/n
Guanajuato México, q_l_o@hotmail.com
2
ABACUS, Cinvestav, Departamento de Matemáticas, Cinvestav Del I.P.N., 07360 México,
D.F., México.
3
Instituto Nacional de Investigaciones Nucleares, Departamento de Física, La Marquesa
Ocoyoacac s/n, Edo. de México, México.

Abstract. Dispersion in porous media play a relevant role in some industrial

processes, the proper characterization of the porous media, and the dynamics of
the involved fluids, are key in these processes and a challenge for the numerical
methods community. The SPH open-source code named DualSPHysics is
presented as a new tool for evaluating the fluid dynamics in porous media. The
code was made using CUDA and its able to compute equations on GPU devices.
In particular, the obtained performance and a comparison between the efficient
implementations for CPU and GPU are compared using the DualSPHysics code.
The use of GPU made possible get simulations with a high number of particles in
a practical time.

1 Introduction

Multiphase flow in porous media has a relevant role on the analysis of many
industrial activities. The velocity, dispersion, and displacement of fluids in a porous
media are key parameters for processes such as pollutant transport, mobile reactive
beds, oil recovery, etc. The proper characterization of the porous media properties,
such as absolute and relative permeability, is crucial for this type of analysis. In order
to predict the behavior of these processes, the numerical simulations (at proper scales)
provide information of the flow almost impossible to obtain from other sources. This
topic is therefore an open challenge in the numerical methods community.
Smooth Particle Hydrodynamics (SPH) is a meshfree Lagrangian numerical method.
This method was first proposed for astrophysical problems [1] but nowadays it is
widely used in computational fluid dynamics [2]. SPH is a Lagrangian and meshless
method in which the fluid is discretized into a set of particles. Each of these particles
is a nodal point for which physical quantities (such as position, velocity, density, and
pressure) are computed as an interpolation of the values of the neighboring particles.
One of the drawbacks of this method is its computational time cost. This issue has
been overcome thanks to the parallelized SPH code named DualSPHysics [3] that
takes advantage of the rising new technology of the Graphic Processor Units (GPUs).
DualSPHysics allows the user to implement a whole new set of cases with the proper
resolution to obtain a good accuracy in its results.
The fluid transport in porous media is a cutting edge topic in the CFD community, so
a new set of equations should be implemented to satisfy all the peculiarities of these
simulations. In order to represent properly these cases, the implementation of two
phases with different properties (viscosity, density, wet ability) is needed.

2 DualSPHysics

DualSPHysics is open-source code developed by the University of Manchester (UK)

and the University of Vigo (Spain) in collaboration with experts form all around the
glove that can be freely downloaded from www.dual.sphysics.org. This project starts
with the need of improving the performance in the SPH simulations, the use of High
Performance Computing (HPC), and more precisely GPU devices can enhance the
performance of a SPH code. The GPU computing accelerates the serial codes up to
two orders of magnitude. DualSPHysics has proven its outstanding performance,
reaching limits like being able to simulate more than 10^9 particles using 64 GPUs
with an efficiency close to 100%.
For the aim of this work, changes were implemented in the source code. Firstly, a
multiphase treatment is now possible with DualSPHysics. Secondly, the
implementation of the wettability of the phases [4, 5] allows the determination of the
surfactant with highest permeability in the porous media and offers better results
when evaluating the capability of the surfactant to efficiently clean the pollutant.
These treatments are coupled with the diffusion equation to evaluate dispersion in
porous media.

3 Implementation on GPU

Big simulation can be achieved with the promising GPU technology for problems that
previously required high performance computing (HPC). Moreover, the coming out of
languages such as the Compute Unified Device Architecture (CUDA) made possible
programming GPUs for general purpose applications. In the particular case of SPH,
the first implementation of the method on GPUs were carried out by [6, 7].
The diffusion equation (5) was implemented in the DualSPHysics code which
governing equations are presented in (1-4):

Momentum
(1)

Continuity
(2)

Estate Equation
(3)
Displacement
(4)

Diffusion
(5)

The equation (5) was implemented using both C++ and CUDA programming
languages. The program can then be executed either on the CPU or on the GPU. The
code is organized in three main stages that are repeated each time step: 1 creating a
neighbor list; 2 computing particle interactions for momentum, concentration and
continuity conservation equations and 3 time integration. Following to [8], the
equation (5) was implemented in the most efficient option where all data is keeping in
the memory of the GPU where all process are parallelized and only output data
requires transfers from GPU to CPU. The diagram flow is showing in the Figure 1.

CPU-GPU GPU-CPU
Initial Data Neighbour System Save Data
Memory Memory
List Update (occasionally)
Transfer Transfer

Particle
Interaction

Fig. 1. Protocol used to simulate on GPU.

The system update can be parallelized esaly on GPU. Example pseudocode where the
cocentration value are update is shown below.

template<…> __global__ void KerComputeStepVerlet (…)

{
Unsigned p=blockIdx.y*gridDim.x*blockDim.x +
blockIdx.x*blockDim.x + threadIdx.x;
if(p<n){
conc1[p] +=(0.5f*dt)*(dcdtm11[p] +dcdt1[p]);
conc2[p] +=(0.5f*dt)*(dcdtm12[p]+dcdtr2[p]);
conc3[p] +=(0.5f*dt)*(dcdtm13[p]+dcdtr3[p]);
}
}
4 Validations and Simulation of Dispersion in Porous Media

Different test with analytical solution were used to validate the implementation of
diffusion equation.
The first numerical simulation refers to a finite vertical strip of tracer within a
rectangular water tank. The width of the strip is equal to the size of the pollutant
particles. The tank is b = 0.4 m and h = 1 m and the length of the strip is equal to 0.11
m, located at X0 = 0.2 m and ranging from Z1 = 0.44 m to Z2 = 0.55 m.The pollutant
has an initial constant concentration C0 = 1 kg/m3 and the initial water concentration
is set equal to 0. The diffusion coefficient D is equal to 10-4 m2/s.
Referring to the z-position of the pollutant, the time evolution of the concentration
field along z coordinate using SPH models is compared with the following analytical
solution (Equation 7, 8) derived from the classical diffusion equation (Equation 6) as
a function of the length and the initial constant distribution of the contaminant
concentration:

(6)

(7)

(8)

Figure 2 shows the comparison between numerical and analytical results for the
proposed case. Numerical results show good agreement with the analytical solution.

1 1
0.9 0.9
0.8 0.8
0.7 0.7
C (kg/m^3)

C (kg/m^3)

0.6 0.6
0.5 0.5
0.4 0.4
0.3 0.3
0.2 0.2
0.1 0.1
0 0
0.4 0.45 0.5 0.55 0.6 0.4 0.45 0.5 0.55 0.6
Z(m) Z(m)
1 second 2 seconds
Fig. 2. Numerical and analytical solution comparison to 1 and 2 seconds.
The second test case is Poiseuille flow between stationary infinite plates at
and . The fluid is initially at rest and is driven by an applied body force F
parallel to the x-axis for . The series solution for the transient behavior is

(9)

DualSPHysics was used to simulate Poiseuille flow for ,

, , and 12500 particles with an initial
separation of . This corresponds to a peak fluid velocity .
A comparison between velocity profiles obtained using (9) and SPH appears in Fig. 3.
The results are again in close agreement.

0.000012

0.00001

0.000008
Vx (m/s)

0.000006

0.000004

0.000002

0
0 0.0002 0.0004 0.0006 0.0008 0.001
Y (m)
DualSPHysics t=0.01 DualSPHysics t=0.1
DualSPHysics t=0.5 Anlitycal Solution t=0.01
Anlitycal Solution t=0.1 Anlitycal Solution t=0.5

Fig. 3. Numerical and analytical solution comparison at 0.01, 0.1 and 0.5 seconds.

A case of dispersion in porous media was simulated using the validated

implementations in DualSPHysics. In this case the porous media with a porosity of
0.5 was represented with circles of in a diagonal configuration. Particles
are moving by a piston with a constant velocity of . Particles with a
miscible pollutant are set at the middle of inlet flow to the porous media. The
diffusion coefficient is . The initial condition is showing in the
Figure 4.
Fig. 4. Initial conditions of simulation of dispersion in a porous media.

Results illustrate the pollutant dispersion in the porous media. These types of
simulations allow evaluate process where the flow and dispersion in porous media
take an important role. Numerical solutions at different times are showing in the
Figure 5.

Fig. 5. Numerical solution of dispersion in porous media at 1, 2 and 3 seconds.

5 Run Time Comparisons

The first possible step in incorporating some form of parallelism is to fully use the
cores and the memory available in a system. Most computer processors today possess
some degree of parallelism with the system memory available to each core
simultaneously, so with parallel computing the processing power of each core can be
assigned to creating multiple threads executing the code in parallel. The data are then
uploaded in the shared memory so they can be accessed and modified by each thread.
However, while using a massively parallel CPU system will significantly decrease the
computational time, its cost can be prohibitive. Obtaining the processors needed, as
well as sufficient space to store them is extremely expensive. The cost of connecting
the machines and their maintenance costs, especially the energy and the cooling
required are also quite significant [9].
GPUs are specialized hardware designed to process large quantities of data to be
displayed on computer screens in real time. The unique architecture of GPUs makes
them particularly suitable for computationally intensive simulations using Lagrangian
methods such as SPH. The large number of multi-processors on a GPU enables speed-
ups close to two orders of magnitude compared to a single CPU code [10].
Simulations were conducted using the CPU-GPU approach and were compared to a
CPU-only approach. The GPU was in general about 2 times faster than the CPU. A
small number of particles were however used for their simulations and a larger speed-
up was expected when the particle resolution is increased. Figure 6 shows the runtime
comparison for the simulation of dispersion in porous media using CPU and GPU
processors.

50

40
Time (min)

30

20

10

0
0.00E+00 4.00E+04 8.00E+04 1.20E+05 1.60E+05
Particles

GPU Geforce GTX 660M CPU Intel Core i7

Fig. 6. Computational time for different numbers of particles using different units of processing

6 Conclusions

A new tool called DualSPHysics has been developed to evaluate the transport of fluid
in porous media. DualSPHysics has shown good accuracy at a reasonable
computational time using GPU computing and provides reliable results according to
the validation. This new implementation allows the study of different options in order
to choose the most optimal approach in fields like pollutant transport, mobile reactive
beds, oil recovery, etc.

References

1. Gingold, R. A., Monaghan, J. J.: Smoothed particle hydrodynamics: theory and

application to non- spherical stars. Mon Not R Astr Soc (1977) 181: 375–389
2. Gómez-Gesteira M., Rogers, B. D., Crespo, A. J. C., Dalrymple, R. A.,
Narayanaswamy, M., Domínguez, J. M.: SPHysics - development of a free-surface
fluid solver- Part 1: Theory and Formulations. Computers & Geosciences, (2012) 48:
289-299
3. Crespo, A. J. C., Domínguez, J. M., Rogers, B. D., Gómez-Gesteira, M., Longshaw,
S., Canelas, R., Vacondio, R., Barreiro, A., García-Feal, O.: DualSPHysics: open-
source parallel CFD solver on Smoothed Particle Hydrodynamics (SPH). Computer
Physics Communications (2014)
4. Bandara, D. M. U. C., Tartakovsky, A. M., Oostrom, M., Palmer, B. J., Grate, J. W.,
Zhang C.: Smoothed Particle Hydrodynamics pore-scale simulations of unstable
immiscible flow in porous media. Advances in Water Resources (2013) 62(Part
C):356-369
5. Holmes, D. W., Williams, J. R., Tilke, P., Leonardi, C. R.: Characterizing flow in oil
reservoir rock using SPH: Absolute permeability, International Journal for Numerical
and Analytical Methods in Geomechanics (2011)
6. Kolb, A., Cuntz, N.: Dynamic particle coupling for GPU-based fluid simulation. 18th
Symposium on Simulation Technique. Erlangen, Germany (2005)
7. Harada, T., Koshizuka, s., Kawaguchi, Y.: Smoothed Particle Hydrodynamics on
GPUs. Proc. of Computer Graphics International (2007b) pp. 63-70.
8. Tartakovsky, A. M., Meakin, P.: Pore-scale modeling of immiscible and miscible
fluid flows using smoothed particle hydrodynamics. Advances in Water Resources
(2006) 29 (10):1464-1478
9. Mcintosh-smith, S., Wilson, T., Ibarra, A. A., Crisp, J., Sessions, R. B.:
Benchmarking Energy Efficiency, Power Costs and Carbon Emissions on
Heterogeneous Systems. Computer Journal, (2012) 55, 192-205
10. Crespo, A. C., Dominguez, J. M., Barreiro, A., Gomez-Gesteira, M., Rogers, B. D.:
GPUs, a New Tool of Acceleration in CFD: Efficiency and Reliability on Smoothed
Particle Hydrodynamics Methods. Plos One, 6 (2011a)