Linear Algebra on Massively

Parallel Systems
USING LAPACK, SCALAPCK, PETSc
 Linear Algebra Libraries
• BLAS (The father)
• Children
1) LAPACK
2) ScaLAPACK
3) PETSc
BLAS routines in vendor supplied
libraries are tuned for architectures.

Even FORTRAN intrinsics like matmul

use BLAS for most compilers.
An example: Parallel matrix-
matrix and matrix vector
multiplication
Intel MKL Library ships with
LAPACK (and of course
BLAS)

• All references, uses can be found in

the Intel MKL library developers’
manual.

 
Symmetric: , Cholesky
Decomposition
An example: Linear solver
with LU factorization
•  Our very familiar
problem! Linear solver
for .
The method to use LAPACK within MKL
• Determine what package and solver you need. Any linear algebra
problem you can think of, LAPACK has an implementation.
• Use link line advisor to gather an idea about what libraries are required.
• Decide if you want static or dynamic linking (i.e. whether the libraries
need to be included in the executable or looked up at runtime). Include
them in a Makefile.
• Write a wrapper for your use (note that this is opposite to the
philosophy in Numerical Recipes!).
• Now we will see a couple examples of BLAS and LAPACK wrappers for
Fortran.
How to build code with Intel MKL
• First decision – how to link libraries?

• Once you know how you want to link, what specific codes to use, what
platform to use it for, platform architecture – USE LINK LINE ADVISOR (
https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor)
Intel MKL Link Line
Advisor
Intel MKL Version

OS

Compiler

Architecture

Linking choices

Cluster library for

distributed memory
computing

Let’s look at a practical example: The dike diffusion problem

Now the Makefile
# Compiler
FC = ifort
# Get flags from compiler options in LLA
CFLAGS = -O2 ?
# Directories
OBJDIR = /HPC/obj
SRCDIR = /HPC/src
# Libraries: ifortvars.sh sourced in .bashrc, dynamic linking at runtime, get from link line in LLA
LIBS = ?
# Files and folders • Note that you need to use the precompiled
TARGET = diff modules:
_OBJ = util_mkl.o diffusion_dike_mkl.o 1) lapack95
OBJ = $(patsubst %,$(OBJDIR)/%,$(_OBJ)) # Here is the object path 2) f95_precision
# Compiling the .o files already packaged within Intel MKL in order to use
$(OBJDIR)/%.o: $(SRCDIR)/%.f90 the modern, shorter, LAPACK F95 calls.
$(FC) -c $(CFLAGS) $< -o $@
# Finally linking for the executable • Do not forget this point when using the LLA!
$(TARGET): $(OBJ)
$(FC) -o $@ $^ $(CFLAGS) $(LIBS) -module $(OBJDIR)
.PHONY: clean
clean:
rm -f $(OBJDIR)/*.o *.mod
ScaLAPACK and PETSc

General advice on choosing linear algebra packages

 Example 2: Calculate EOF to express the
spatial variability of climate data
• Suppose you see a spatial pattern in
climatology over a region of the earth.
• You might want to ask what processes
lead to the pattern – a similar Figure 1: The climatology of summer precipitation (May to September) during 1979–2013.  (Figure courtesy:
question is what dominant spatial http://english.iap.cas.cn/RE/201511/t20151106_155361.html)

patterns lead to the pattern that we

are observing?

Figure 2: The first two leading EOF modes of the Pan-Asian Monsoon summer precipitation and their
corresponding principal component over the period of 1979–2009. The first mode explained 15.92% of total
variance; the second mode explained 9.11% of total variance (Gao and Wang, 2015) 
 How to calculate EOF’s
• Then
  if matrix has rows , matrix has columns as , then .

• We want to find the s such that and .

• It can be shown, that to find such that its columns are is identical to solving
the problem:

where is the data covariance matrix.

This is of course a spectral decomposition problem (or an eigenvalue problem).
The question is how expensive is the calculation? We immediately have a
matrix-matrix multiplication [ complexity] followed by an eigenvalue problem
[ complexity]. Extremely expensive!
Singular value Decomposition
•  It can be instead shown, that if: is
the singular value decomposition
of , then and .
• Also, .
• This is an order of operation
implemented in LAPACK.
• We will see an example of the
implementation of the SVD of a
dense matrix in LAPACK.
The LAPACK SVD routine