Distributed 


Graph Processing
Immanuel Trummer

itrummer@cornell.edu

www.itrummer.org
 Outlook: 

Beyond Relational Data

• Graph data

• Data streams

• Spatial data

Slides by Immanuel Trummer, Cornell University

 Reading List

• "Pregel: a system for large-scale graph processing",

SIGMOD 2010, G. Malewicz et al. [Google]

• "One trillion edges: graph processing at Facebook-scale",

VLDB 2015, A. Ching et al. [Facebook]

Slides by Immanuel Trummer, Cornell University

Motivation: Large Graphs

• Graphs may exceed resource limits of single machines

• Graphs representing the entire Web (Google)

• Graphs representing large social networks (FB)

• ...

• This motivates graph processing in clusters

Slides by Immanuel Trummer, Cornell University

 Example: PageRank

• Google ranks search results via the PageRank algorithm

• Operates on a graph representation of the Web

• Nodes represent Web sites

• Edges represent links

• Pages with higher PageRank are preferable

Slides by Immanuel Trummer, Cornell University

 Random Surfer
• PageRank is based on the random surfer model

• Random surfer starts from random Web site

• Randomly selects outgoing links to follow

• May select random page with probability 𝛂

• Selects random page if no outgoing links

• PageRank: probability to visit site at specific instant

Slides by Immanuel Trummer, Cornell University

 PageRank Example

3.3 38.4 34.3

3.9 3.9

8.1

1.6
1.6
1.6
1.6
1.6
Slides by Immanuel Trummer, Cornell University
 PageRank Example
Do We Sometimes Select Random Pages?

3.3 38.4 34.3

3.9 3.9

8.1

1.6
1.6
1.6
1.6
1.6
Slides by Immanuel Trummer, Cornell University
 Calculating PageRank

• We can calculate PageRank via an iterative algorithm

• We initialize each node's PageRank to 1/NrNodes

• In each iteration, we redistribute PageRank over links

• Each node partitions PageRank among outgoing links

• PageRank in next iteration is sum over incoming links

Slides by Immanuel Trummer, Cornell University

PageRank Iterative Updates
Iteration I Iteration I+1
30
90 35
30
30
5
10 30

5
...

Slides by Immanuel Trummer, Cornell University

...
 Pregel Overview

• Pregel is a system for distributed graph processing

• Proposed in 2010 (Google), PageRank is use case

• Pregel distributes graph partitions over cluster nodes

• Worker nodes process their partition in parallel

Slides by Immanuel Trummer, Cornell University

Pregel Computation Model
• Computation is divided into iterations ("supersteps")

• In each iteration, we invoke Compute for each node

• Compute function can be customized by user

• Input: messages sent to this vertex in prior iteration

• Can message other nodes, delivered in next iteration

• Computation ends once all nodes vote to halt

Slides by Immanuel Trummer, Cornell University

 Illustration of Computation
Iteration I-1 Iteration I Iteration I+1

... t e ! ...
m pu
C o
e s ge s
s a g ss a
s e
d M...e Node
i v e ...
M
Se n ce
Re
... ...

Slides by Immanuel Trummer, Cornell University

 Illustration of Computation
Iteration I-1 Iteration I Iteration I+1

... t e ! ...
m pu
C o
e s ge s
s a g ss a
s e
d M...e Node
i v e ...
M
Se n ce
Re
... o t e ...
a y V
M

Halt

Slides by Immanuel Trummer, Cornell University

 Parallel Processing
3.3 38.4 34.3

3.9 3.9

8.1

1.6
1.6
1.6
1.6
1.6

Slides by Immanuel Trummer, Cornell University

 Parallel Processing
3.3 38.4 34.3

3.9 3.9

8.1

1.6
1.6
1.6
1.6
1.6

Slides by Immanuel Trummer, Cornell University

Graph Partitions
 Parallel Processing
Worker 1

Wo
rk
3.3 38.4 34.3

er
2
3.9 3.9

8.1

1.6
1.6
1.6
1.6
1.6

Slides by Immanuel Trummer, Cornell University

Graph Partitions
 Parallel Processing
Worker 1

Wo
rk
3.3 38.4 34.3

er
o d e

2
na to r N
oo rd i
e d b y C
A s si
3.9 g n 3.9

8.1

1.6
1.6
1.6
1.6
1.6

Slides by Immanuel Trummer, Cornell University

Graph Partitions
 Fault Tolerance
• Workers persist input and state at iteration start

• Coordinator detects worker failures via pings

• Recovery may start several supersteps earlier

• Re-partition graph to replace failed workers

• "Confined recovery" restricted to failed partitions

• Requires persisting outgoing messages as well

Slides by Immanuel Trummer, Cornell University

 PageRank in Pregel

Compute(ReceivedPR : int[]):

NewPR = sum(ReceivedPR)

For o in OutgoingLinks:

Send(o.target, NewPR/|OutgoingLinks|)

(Extensions required for random jumps and handling "dead ends")

Slides by Immanuel Trummer, Cornell University
 Better Performance 

with Combiners

• Basic version sends lots of page rank values

• Can aggregate messages via custom "Combiners"

• Here: can combine page rank for same target as sum

Slides by Immanuel Trummer, Cornell University