Professional Documents
Culture Documents
t
yS
p
e
e
d
CPU
Leve 2 Cache
Leve 1 Cache
Mechanca Dsk
Leve 3 Cache
Disk access is more complicated than memory access
OOC libraries should provide an easier interface
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Easing Disk Access Using the NumPy OO Paradigm
array[index]
(array1**3 / array2) - sin(array3)
numpy.dot(array1, array2)
Many existing OOC libraries are already mimicking parts of this
abstraction.
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Some OOC Libraries Mimicking NumPy Model
Interfaces to binary formats (HDF5, NetCDF4):
Interfaces to HDF5:
h5py
PyTables
Interfaces to NetCDF4:
netcdf4-python
Scientic.IO.NetCDF
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Using NumPy As Default Container for OOC
All the previous libraries are using NumPy as default container (and
they can also use compression lters for improved I/O).
Interfaces for RDBMS in Python lacks support for direct NumPy
containers (very inecient!).
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
PyTables: Retrieving a Portion of a Dataset
array[index], where index can be one of the following:
scalar: array[1]
slice: array[3:1000, ..., :10]
list (or array) of indices (fancy indexing): array[[3,10,30,1000]]
array of booleans: array[array2 > 0]
All these selection modes are supported by PyTables.
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
PyTables: Operating With Disk-Based Arrays
tables.Expr is an optimized evaluator for expressions of
disk-based arrays.
It is a combination of the Numexpr advanced computing
capabilities with the high I/O performance of PyTables.
Similarly to Numexpr, disk-temporaries are avoided, and
multi-threaded operation is preserved.
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
tables.Expr in Action
Evaluating .25*x**3 + .75*x**2 - 1.5*x - 2
import tables as tb
f = tb.openFile(h5fname, "a")
x = f.root.x # get the x input
r = f.createCArray(f.root, "r", atom=x.atom, shape=x.shape)
ex = tb.Expr(.25*x**3 + .75*x**2 - 1.5*x - 2)
ex.setOutput(r) # output will got to the CArray on disk
ex.eval() # evaluate!
f.close()
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
tables.Expr Performance (In-Core Operation)
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Other Features of PyTables
Allows organizing datasets on a hierarchical structure
Each dataset or group can be complemented with user
metadata
Powerful query engine allowing ultra-fast queries (based on
Numexpr and OPSI)
Advanced compression capabilities (Blosc)
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
PyTables Pro Query Performance
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Summary
These days, you should understand that there is a CPU
starvation problem if you want to get decent performance.
Make sure that you use NumPy as the basic building block for
your computations.
Leverage existing memory-ecient libraries for performing your
computations optimally.
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
More Info
Francesc Alted
Why Modern CPUs Are Starving and What Can Be Done
about It
Computing in Science and Engineering, IEEE, March 2010
http://www.pytables.org/docs/CISE-March2010.pdf
NumPy crew
NumPy manual
http://docs.scipy.org/doc/numpy
PyTables site
http://www.pytables.org
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Questions?
Contact:
faltet@pytables.org
Francesc Alted Large Data Analysis with Python
The Starving CPU Problem
High Performance Libraries
Why Should You Use Them?
In-Core High Performance Libraries
Out-of-Core High Performance Libraries
Acknowledgments
Thanks to Stfan van der Walt for giving permission to use his cool
multidimensional container picture:
This was made using a Ruby plugin for Google SketchUp.
Francesc Alted Large Data Analysis with Python