Distinguished Author Series
Megacell Reservoir Simulation
Ali H. Dogru, SPE, Saudi Arabian Oil Co.
Summary
Reservoir simulation has moved into a new era with the
rapid advancements in parallel-computer hardware and soft-
ware technologies. In the past decade, the total number of
gridblocks used in a typical reservoir study increased from
thousands to millions. Mega (million) -cell simulation capa-
bility allows accurate representation of reservoirs because of
its ability to include detailed information provided by logs,
3D seismic, horizontal wells, and stratigraphic studies. One
of the main advantages of megacell simulation is that it min-
imizes or eliminates upscaling effects, making more-accu-
rate reservoir-performance predictions possible. Further-
more, the speed of parallel computers significantly reduces
the manpower-intensive history-matching process. Megacell
simulators typically are developed on parallel computers.
With the current low-cost parallel computers, a million-cell
simulation run covering 20 years of production history takes
less than 1 hour. State-of-the-art computer graphics provide
an effective means of visualizing large data sets covering mil-
lion-cell descriptions of a reservoir and its performance. This
paper presents an overview of megacell simulation tech-
nologies in the oil industry and academia.
Introduction
The development of parallel computers allowed faster reser-
voir simulation, placing heavy demand on number crunch-
ing and data storage. Parallel computers are classified in two
categories: shared- and distributed-memory computers.
Shared-memory computers, such as Cray vector super-
computers, are composed of multiple central processing
units (CPU’s) sharing a common memory. Each CPU of this
type of machine is usually very fast and uses vector process-
ing. In contrast, in distributed-memory machines, each CPU
has its own memory and communicates with the other
nodes (CPU’s) through a high-speed network. The number
of CPU’s in shared-memory machines cannot be increased
beyond a specific number (usually 4, 8, or 16), but distrib-
uted-memory machines have no such limitations. Distrib-
uted-memory machines [sometimes called massively paral-
lel processors (MPP’s)] can be made up of hundreds or thou-
sands of CPU’s. Each CPU can be very low cost (the cost of
a PC); however, the network communication and program-
ming language become major challenges in this type of
architecture. Both major and small hardware companies
manufacture massively parallel computers. The collection
(cluster) of workstations and, more recently, the collection
of PC’s with a high-speed switch box provide both signifi-
cant cost savings and attractive performance. Ref. 1 gives a
Copyright 2000 Society of Petroleum Engineers
This is paper SPE 57907. Distinguished Author Series articles are general, descrip-
tive representations that summarize the state of the art in an area of technology by
describing recent developments for readers who are not specialists in the topics dis-
cussed. Written by individuals recognized as experts in the area, these articles provide key
references to more definitive work and present specific details only to illustrate the tech-
nology. Purpose: to inform the general readership of recent advances in various areas
of petroleum engineering.
54
comprehensive description of parallel computers and the
associated programming languages, and Abate et al.* discuss
the latest efforts and provide an example of a reservoir sim-
ulator running on cost-effective PC clusters.
Early Oil Industry Efforts
As the developments described were taking place in the
computer industry, researchers in the oil industry began
investigating the benefits of the parallelization in reservoir
simulation. The research effort on parallel reservoir simula-
tion during the 1980’s is widely discussed in the literature.
Shared Memory. Refs. 2 through 4 present early work on
parallelization of reservoir simulators for shared-memo-
ry/vector computers. The results presented in these works
demonstrate the potential benefits of parallel processing.
Distributed Memory. Wheeler5 developed a black-oil sim-
ulator on a hypercube, Killough and Bhogeswara6 present-
ed a compositional simulator on an Intel iPSC/860, and
Rutledge et al.7 constructed a black-oil simulator for the
CM-2 and showed that reservoir models with more than 2
million gridblocks could be run on this type of machine
with 65,536 processors.
The mid-1990’s brought further distributed-memory pub-
lications. Kaarstad et al.8 presented a 2D oil/water simulator
for a 16,384-processor MasPar MP-2. He showed that a
model problem with 1 million gridblocks could be solved in
a few minutes. Rame and Delshad9 parallelized a chemical-
flooding code and tested scalability on a variety of systems.
Distributed-memory simulators began to move from
research to production in 1997 with the presentation of real
field applications with massively parallel simulators. Shi-
ralkar et al.10 presented FALCON, a distributed-memory
simulator that uses several programming languages to handle
data distribution and interprocessor communication. These
languages include high-performance FORTRAN and FOR-
TRAN 90 coupled with message passing [parallel virtual
machine or message-passing interface (MPI)].1 The simula-
tor was tested on various computing platforms, including
TMC CM-5, IBM SP-2, SGI Power Challenge, Cray T3D and
T3E, and Origin 2000.
Chien et al.11 presented a distributed-memory simulator
based on an existing FORTRAN 77 simulator. They used
domain decomposition and MPI message-passing libraries
on an IBM SP-2. Their largest reported field was more than
1 million gridblocks running on a 16- or 32-node SP-2 sys-
tem. Killough et al.12 presented locally refined grids based
on domain decomposition for a distributed-memory simula-
tor. The largest field model featured implicit pressure/explic-
it saturations (black oil and compositional) for more than 1
million gridblocks and up to 32 SP-2 processors.
*J. Abate, P. Wang, and K. Sephernoori: “Parallel Compositional Reservoir Simulation on a Clus-
ter of PCs,” Dept. of Petroleum Engineering, U. of Texas, Austin, Texas (December 1999).
MAY 2000

Fig. 1—Conventional simulation process.
Hwang and Xu1 provide useful information on parallel
computers and programming languages, and Gropp et al.13
describe the MPI for parallel programming in detail.
Upscaling Concerns
As previously stated, parallel reservoir simulation technol-
ogy offers the potential to include detailed reservoir
description available from the geological models. Geologi-
cal models usually cover millions of gridblocks that contain
the core, log, seismic, and geologically interpolated data. If
we consider only the vertical direction, we see that logs
alone provide foot-by-foot vertical heterogeneity (layering)
information. Owing to the limitations in the current simu-
lation process, describing all the layers seen in the logs in a
simulation model is impossible. Instead, flow layers are
defined by lumping the log information. For example, for a
200-ft-thick reservoir, defining 200 vertical simulation lay-
ers is not practical; therefore, the information is lumped
into 10 to 20 flow layers. This process, called upscaling,
carries the risk that vertical barriers and heterogeneity are
not properly defined. Upscaling also changes the locations
of perforations, which can be moved up or down 10 to 20
ft, depending on the layering. This can significantly affect
calculation of the arrival of water or gas at the wells. Simu-
lation engineers then must change gridblock permeabilities
and introduce pseudorelative permeability curves (well
functions to match the water or gas breakthrough times
and behavior). This often results in unrealistic property val-
ues and causes disagreements between the simulation engi-
neers and reservoir management.
Another adverse effect of areal upscaling is more pro-
nounced when simulating large oil fields, such as those in the
Arabian Gulf where reservoirs usually contain billions of bar-
rels of oil. In such reservoirs, wells generally are located 0.62
miles apart and geological models have a grid spacing of 820
ft. If a simulation engineer wants to place at least a few grid-
blocks between the wells and wishes to use 20 to 30 layers to
describe thick vertical pay, the simulation model will have a
few million cells. Because of this, current simulation prac-
tices force the engineer to use coarser areal grid sizes and 20
or less vertical layers to fit the reservoir into a conventional
simulator running on a mainframe or high-speed worksta-
56
Fig. 2—Megacell simulation process.
tion. These limitations also force engineers to study a section
of the reservoir with fine-grid definition rather than the full-
field representation. Full-field representations are more rep-
resentative, especially for the large, high-permeability reser-
voirs found in the Middle East. Unlike sector models, full-
field models do not use boundary-condition assumptions
and, therefore, represent total reservoir behavior better.
Impact of Parallel Simulation
Fig. 1 illustrates the conventional process of upscaling with
sequential computer technology. As the figure shows, geo-
logical models containing more than 1 million gridblocks
typically are reduced 100,000 or fewer cells for reservoir
simulation. Depending on the procedures used in the
upscaling process, it is possible to end up with different
representation of the reservoir for the simulation process.
The process may change from one company to another and
from one individual to another, depending on the style and
approach and computer hardware availability. As a result,
different reservoir behavior could be generated.
In contrast, megacell simulation carried out on various
platforms, such as MPP’s, a cluster of workstations (PC’s),
or distributed/shared-memory machines, can overcome or
minimize upscaling effects (Fig. 2). As Fig. 2 shows, the
detailed description of the reservoir contained in the geo-
logical models can be incorporated directly into the simu-
lation model or critical features of the geology can be trans-
ferred to a simulation model with minimum upscaling. The
size of the simulation model generated from the geological
model depends on the hardware configuration. Assuming
that sufficient nodes are available in the parallel computer,
it is possible to include the entire geological model as a
reservoir simulation model.
Industrial Applications
Several oil companies and universities currently are devel-
oping and implementing parallel simulation technologies.
Among these are Saudi Arabian Oil Co. (Saudi Aramco),14
Amoco,10 and Chevron.11 Consulting companies (e.g.,
Landmark12 and Geoquest) also provide technologies in
parallel simulation. Here, we present Saudi Aramco’s data to
highlight the megacell simulation experience and results.
Saudi Aramco is a dedicated user of reservoir simulation
technologies for its giant hydrocarbon reservoirs. Current-
ly, we use three well-known commercial simulators. All the
simulators use conventional finite-difference schemes and
have options for black-oil, compositional, and dual-poros-
ity features. The simulators are operated on IBM SP-2
high-end superworkstations. The company previously
used Cray supercomputers for reservoir simulation.
Parallel with the development of computer technologies,
the number of models used increased from 25 in 1988 to
MAY 2000
 Fig. 3—Impact of parallel reservoir simulator.
100 in 1998. In all these applications, we used conven-
tional simulators running on supercomputers and, more
recently, high-end workstations.
To examine reservoir complexities in the giant oil reser-
voirs better, Saudi Aramco developed its own MPP parallel
simulator called POWERS.14 Fig. 3 illustrates the average
number of gridblocks used in the simulation models in the
past 14 years. As the figure shows, the average number of
gridblocks increased from 30,000 in 1988 to 100,000 in 1996
and, with the development of the parallel simulator, jumped
an order of magnitude to more than 1 million cells in 1998.
Note that the megacell simulation process includes an effi-
cient pre- and post-processing environment. This means
users can easily generate million-cell models from the geo-
logical models, run them in a practical time frame, and ana-
lyze the results using efficient visualization packages, which
makes it comparable with building, running, and analyzing
the results of a 100,000-cell model on a workstation. This is
considered a major achievement during the past decade.
Academic Applications/PC Clusters
The U. of Texas (UT), Austin, is active in parallel reservoir
simulation.15 The Center for Subsurface Modeling of the
Texas Inst. of Computational & Applied Mathematics
(TICAM) developed a parallel reservoir simulator called
IPARS that is heavily used for R&D of new mathematical
algorithms in reservoir simulation. Similarly, the Center for
Petroleum & Geosystems Engineering (CPGE) is active in
developing parallel compositional simulation models.
TICAM researchers built a cluster of PC’s as a parallel
computer and successfully ran their parallel simulator on
the same machine. The PC cluster is called a Beowulf sys-
tem and is composed of 16 nodes. Another such system
called Wonderland was easily built in a student office by an
undergraduate student (Fig. 4). This system is composed of
16 PC’s connected by an ethernet network, one switch box,
and a monitor. Its primary advantage is cost. The system
costs at least an order of magnitude less than a commercial
parallel computer. TICAM also built a 64-node cluster called
Longhorn with a fast Myrinet network, and they currently
run their parallel simulator on this new system.
CPGE also built a 16-node PC cluster. They developed a
fully implicit, parallel compositional simulator and studied
its performance on both a cluster of PC’s and massively par-
allel supercomputers (Fig. 5). By using a fast Myrinet net-
work (1.28 gigabits/sec) rather than the 100-megabit/sec
MAY 2000
Fig. 4—TICAM’s Wonderland 16-CPU cluster.
ethernet network, they reached a speed comparable with a
16-node IBM SP-2. Fig. 5 shows the CPGE PC cluster and
also the results of a cluster of 16 PC’s (each PC is a 400-
cycle/sec Pentium II with 512 Mbytes of memory).* They
compared this system with a 16-node IBM SP-2 of the Maui
High Performance Computing Center. The SP-2 system had
thin nodes (160-cycle/sec P2SC, 512 Mbytes with a high-
performance switch box). The reservoir simulator problem
used for the comparison was the fifth SPE comparative prob-
lem with 30,720 gridblocks, three hydrocarbon components,
and 100 days of gas injection.16 The compositional simula-
tor was run on two different systems with the same data set.
The scaleup runs consisted of running the same problem
on 4, then 8, then 12, then 16 processors. Total time for the
16-processor run was 82 seconds for the cluster and 77 sec-
onds for the SP-2. Fig. 5 shows that the PC clusters and the
SP-2 system gave very similar scaleup results. The figure also
shows that the scaleup was linear, making it potentially very
attractive. The main comparison, however, is cost. PC clus-
ters cost only a small fraction of what a commercial parallel
computer costs. CPGE has also run more-realistic problems
consisting of 200,000 gridblocks and six components.17
Benefits of Megacell Simulation
The following are the major benefits of megacell simula-
tion on parallel computers.
1. Adequate description of reservoir heterogeneities in
both areal and vertical directions.
2. Full-field description with sufficient resolution in
areal and vertical directions.
3. Speed of simulation.
4. Integrated reservoir simulation (coupled reservoir
and surface facility simulation).
These factors all can reduce the time-consuming history-
matching process.14 Fig. 6 shows the pressure and water-
cut performance match for a giant offshore oil reservoir.
These results were from the initial run and indicate close
agreement. This allows less effort to go into obtaining a
final history match.
Fig. 7 illustrates one major effect of use of fine-grid mega-
cell simulation. A bypassed oil zone is detected by the mega-
cell simulation with 67 vertical layers (a total of 2.5 million
*J. Abate, P. Wang, and K. Sephernoori: “Parallel Compositional Reservoir Simulation on a Clus-
ter of PCs,” Dept. of Petroleum Engineering, U. of Texas, Austin, Texas (December 1999).
57
 Fig. 5—PC clusters parallel computer, CPGE Beowulf system.
cells), while the conventional simulator with five vertical
layers (40,500 cells) misses the oil pockets left behind the
flood front. This is in spite of the fact that both models
matched the water-cut behavior for Well A (Fig. 8).
Table 1 provides an example of the speed of the megacell
simulation. The table shows execution times with the fully
implicit black-oil option of Saudi Aramco’s parallel simula-
tor for four different full-field models representing large
reservoirs. To demonstrate the effect of the number of
processors on speed, the total number of processors was
increased from 64 to 128 nodes of a CM5E parallel com-
puter and resulting CPU times are reported. Each node has
a modest processing speed of 128 MFLOPS and 128
megabytes of memory. The speed of each model is nearly
doubled by doubling the number of nodes. In each case,
megacell simulators with 20 to 50 years of production/injec-
tion history involving hundreds of horizontal and vertical
wells can be run within hours (like conventional simula-
tors) with an order of magnitude more gridblocks.
Fig. 6—Early runs of megacell simulation of offshore oil
reservoir showing pressure and water-cut match.
58
Computational speed achieved in megacell simulation
offers great promise for coupling surface-facility simulators
(from sandface to separator) with reservoir simulators. Such
capability would produce more-accurate reservoir-perform-
ance computations and eliminate the need to use hydraulic
flow tables.
Conclusions
The megacell simulation process is becoming a feasible
process with the development of fast, cost-effective parallel
computers. Megacell simulation allows detailed description
of the reservoir in a simulation mode, leading to a close his-
tory match in a short time. Having millions of cells mini-
mizes and, in some cases, eliminates the upscaling effects and
Fig. 7—Comparison of conventional simulation (40,500
cells, 5 layers) and megacell simulation (2.45 million
cells, 67 layers).
MAY 2000
 TABLE 1—SPEED OF MEGACELL MODELS ON A
CM5E PARALLEL COMPUTER, POWERS SIMULATOR

Model Size History CPU Hours

(millions of Length On 64 On 128
Reservoir gridblocks) (years) Nodes Nodes
Carbonate 1.12 27 3 1.7
Sandstone 1.3 49 4.5 2.5
Carbonate with 3.9 10 — 2.0
gas cap
Carbonate 2.5 25 — 4.0

10. Shiralkar, G.S. et al.: “FALCON: A Production Quality Dis-

Fig. 8—Water-cut match for Well A.
process. Technology offers the potential to couple reservoir
and surface-facility simulators. On parallel computers, mega-
cell simulation models can be run in a matter of hours and
graphical tools are available to analyze and construct these
large models in a very practical time frame. The megacell
description is a better reservoir management tool. JPT
Acknowledgments
We thank the Saudi Aramco management for permitting
the publication of this paper and the POWERS team mem-
bers, L.S. Fung and W.T. Dreiman at Saudi Aramco,
TICAM, and the UT Dept. of Petroleum Engineering for
providing data.
References
11. Hwang, K. and Xu, Z.: Scalable Parallel Computing: Technolo-
gy, Architecture, Programming, WCB, McGraw-Hill Book Co.
Inc., Boston (1998).
12. Scott, S.L. et al.: “Application of Parallel (MIMD) Computers
to Reservoir Simulation,” paper SPE 16020 presented at the
1987 SPE Reservoir Simulation Symposium, San Antonio,
Texas, 1–4 February.
13. Chien, M.C.H. and Northrup, E.J.: “Vectorization and Paral-
lel Processing of Local Grid Refinement and Adaptive Implic-
it Schemes in a General Purpose Reservoir Simulator,” paper
SPE 25258 presented at the 1993 SPE Reservoir Simulation
Symposium, New Orleans, 28 February–3 March.
14. Li, K.G. et al.: “Improving the Performance of MARS Reser-
voir Simulator on Cray-2 Supercomputer,” paper SPE 29856
presented at the 1995 SPE Middle East Oil Show, Bahrain,
11–14 March.
15. Wheeler, J.A. and Smith, R.A.: “Reservoir Simulation on a
Hypercube,” SPERE (November 1990) 544.
tributed Memory Reservoir Simulator,” SPEREE (October
1998) 400.
11. Chien, M.C.H. et al.: “A Scalable Parallel Multipurpose Reser-
voir Simulator,” paper SPE 37976 presented at the 1997 SPE
Reservoir Simulation Symposium, Dallas, 8–11 June.
12. Killough, J.E., Camilleri, D., and Darlow, B.: “A Parallel Sim-
ulator on Local Grid Refinement,” paper SPE 37978 present-
ed at the 1997 SPE Reservoir Simulation Symposium, Dallas,
8–11 June.
13. Gropp, W., Lusk, E., and Skjellum, A.: Using MPI—Portable
Parallel Programming with the Message-Passing Interface, MIT
Publications, Cambridge, Massachusetts (1994).
14. Dogru, A.H. et al.: “A Massively Parallel Reservoir Simulator
for Large Scale Reservoir Simulation” paper SPE 51886 pre-
sented at the 1999 SPE Reservoir Simulation Symposium,
Houston, 14–17 February.
15. Parashar, M. et al.: “A New Generation EOS Compositional
Reservoir Simulator: Part II—Framework and Multiprocess-
ing,” paper SPE 37977 presented at the 1997 SPE Reservoir
Simulation Symposium, Dallas, 8–11 June.
16. Killough, J. and Kossack, C.: “Fifth SPE Comparative Solu-
tion Project: Evaluation of Miscible Flood Simulators,” paper
SPE 16000 presented at the 1987 SPE Reservoir Simulation
Symposium, San Antonio, Texas, 1–4 February.
17. Wang, P. et al.: “A Fully Implicit Parallel EOS Compositional
Simulator for Large Scale Reservoir Simulation,” paper SPE
51885 presented at the 1999 SPE Reservoir Simulation Sym-
posium, Houston, 14–17 February.
SI Metric Conversion Factors
bbl ×1.589 873 E−01= m3
cycle/sec ×1.0* E+00= Hz
ft ×3.048* E−01= m
mile ×1.609 344* E+00= km
psi ×6.894 757 E+00= kPa
*Conversion factor is exact.

16. Killough, J.E. and Bhogeswara, R.: “Simulation of Composi-

tional Reservoir Phenomena on a Distributed Memory Parallel
Computer,” JPT (November 1991) 1368; Trans., AIME, 291.
17. Rutledge, J.M. et al.: “The Use of Massively Parallel SIMD
Computer for Reservoir Simulation,” paper SPE 21213 pre-
sented at the 1991 SPE Reservoir Simulation Symposium,
Anaheim, California, 17–20 February.
18. Kaarstad, T. et al.: “A Massively Parallel Reservoir Simulator,”
paper SPE 29139 presented at the 1995 SPE Reservoir Simu-
lation Symposium, San Antonio, Texas, 12–15 February.
19. Rame, M. and Delshad, M.: “A Compositional Reservoir Sim-
ulator on Distributed Memory Parallel Computers,” paper
SPE 29103 presented at the 1995 SPE Reservoir Simulation
Symposium, San Antonio, Texas, 12–15 February.
Ali H. Dogru is General Supervisor of the Technology Devel-
opment Div. of Saudi Arabian Oil Co. in Dhahran, Saudi Ara-
bia, with responsibility for development of new reservoir and
production technologies. His areas of interest are numerical
reservoir simulation, fluid mechanics, heat transfer, and
applied mathematics. He previously worked at Mobil R&D
Co. and Core Laboratories and held teaching positions at UT,
Austin; California Inst. of Technology; Norwegian Inst. of
Technology; and Technical U. of Istanbul (TUI). Dogru holds
an MS degree from TUI and a PhD degree from UT, both in
petroleum engineering. He served as a 1984–90 member of
the Editorial Review Committee, a 1994–95 member of the
Forum Series in the Middle East Steering Committee, and on
Annual Meeting technical committees.

60
MAY 2000