Techniques and Systems for Large Dynamic Graphs

[Supervisor: M. Tamer Özsu]

Khaled Ammar
Cheriton School of Computer Science
University of Waterloo, Ontario, Canada
khaled.ammar@uwaterloo.ca

ABSTRACT
Many applications regularly generate large graph data. Many
of these graphs change dynamically, and analysis techniques
for static graphs are not suitable in these cases. This thesis
proposes an architecture to process and analyze dynamic
graphs. It is based on a new computation model called
Grab’n Fix. The architecture includes a novel distributed
graph storage layer to support dynamic graph processing.
These proposals were inspired by an extensive quantitative
and qualitative analysis of existing graph analytics plat-
form.

1. INTRODUCTION
Large volumes of data generated from our interaction with
software systems that support daily applications in areas
such as commerce, entertainment and social networking have
given rise to what is commonly referred to as the “Big Data
Problem”. Scale is key to all aspects of big data manage-
ment, and new scalable techniques are required for large-
scale data management as traditional techniques are no longer
effective.
Graph data are of growing importance in this context. Ap-
plications that rely on graph data include the semantic web
(i.e., RDF), bioinformatics, finance and trading, and social
networks, among others. Graphs naturally model compli-
cated structures, such as protein interaction networks, prod-
uct purchasing, business transactions, relationships and in-
teractions in social or computer networks, and web page
connections. The size and complexity of these graphs raise
significant data management and data analysis challenges.
Graph analytics can be performed on a static or a dynamic
graph. Existing graph analytics work mostly focus on static
graphs. However, since graphs are used to represent rela-
tionships between entities, in many cases they change and
evolve - most graph applications are dynamic in nature:
• Twitter’s users write 500 million tweets per day. Every
tweet can influence graphs that model relationships
between users, products, interests, locations, etc.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-
publish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
SIGMOD’16 PhD Symp, June 25-July 01 2016, San Francisco, CA, USA
c 2016 ACM. ISBN 978-1-4503-4192-9/16/06. . . $15.00
DOI: http://dx.doi.org/10.1145/2926693.2929897
Figure 1: System Architecture
• Facebook had about 829 million daily active users on
average in June 2014. It has been reported that Face-
book sees activity from half a billion users to generate
130TB of logs every day.
• The web graph has more than 40 billion web pages and
several million websites are created every year.
• Business transaction graphs for several online or retail
stores have multiple billion transactions per year.
My research explores both the algorithmic and system as-
pects of building new and scalable solutions for efficient and
effective management, querying and analysis of large and dy-
namic graphs. My experiments include real graph datasets
that require more than 3TB of main memory. The main
objective is to create a framework that is capable of main-
taining query answers for very large and dynamic graphs in
real time. Figure 1 shows the proposed framework and its
main components. Components in red boxes are going to be
the focus of my thesis.
I briefly introduce two use cases that would benefit from
dynamic graph analysis:
Dynamic Clustering: The weakly connected compo-
nent workload is essential for several clustering approaches.
When the clustering dimensions are change over time, the
clustering model needs to be updated frequently to reflect
new clusters. In several applications, such as the analysis
of stock market, election, social network, and health infor-
matics, the dimensions often represent a time series. A suc-
cessful application should keep track of all dimensions and
update the cluster accordingly.
Dynamic Network-aware Search: Shortest path is a
fundamental graph problem that has been used in many ap-
plications for network-aware search. In a social network, web

7
 Figure 2: A taxonomy for graph processing techniques.
graph, or wireless network routing, the closeness between
two vertices in the graph need to be computed frequently
as the graph changes. It is essential to build an index that
can efficiently change rapidly as the original graph changes.
An efficient index would enhance user experience in social
networks to find new friends quickly, and enhance commu-
nication quality in wireless networks.
The rest of this paper is organized as follows. Section 2
formalizes the problem this research tackles. Section 3 out-
lines the methodology that I follow in this research. Sec-
tion 4 introduces a new computation model designed for dy-
namic graphs. Section 5 discusses existing data structures
in graph systems, and describes the success criteria for an
efficient dynamic graph processing. Finally, section 6 sum-
marizes my progress so far, and my future work.
2. PROBLEM DEFINITION
A graph, G is defined as G = (V, E), where V is a set of
vertices and E is a set of edges, such that |V | = n, |E| = m.
For an edge e = (u, v, Pe ), e ∈ E if v, u ∈ V . Pe is a set
of properties associated with edge e. A vertex v ∈ V , also,
has a set of properties, Pv . Pe and Pv can be empty. Every
graph G may receive multiple updates ∆G. These updates
could be, insert, delete, or change. An ad-hoc query, such as
shortest path, or an analytical query, such as PageRank, can
be abstracted as a function F . The answer for this function
on the input graph G is F (G).
Existing graph processing systems have limited capabili-
ties in utilizing previously computed answers F (G) to com-
pute the answer of F (G + ∆G) where ∆G represents the
change to the graph. For example, systems such as GraphLab [9,
14], Giraph [1], and Blogel [20] need to repartition the graph
after every update, which is expensive. Moreover, some sys-
tems (e.g., GraphLab) does not support edge deletions.
In this thesis, I am building algorithms and data struc-
tures implemented in a prototype system to efficiently com-
pute F (G+∆G). The objective is to compute the new result
by only considering the graph changes, i.e.:
F (G + ∆G) = F (G) + F (∆G) (1)
Ideally, F (G + ∆G) should be computed by only evalu-
ating F (∆G) when a change occurs. However this is hard
to achieve, since typically there is an overhead. Thus, the
formula is:
F (G + ∆G) = F (G) + F (∆G) +  (2)
where  is the overhead.
8
The main objective of this thesis is to propose algorithms
for typical graph workloads that minimize the amount of
work represented by  in a parallel environment.
2.1 Graph Processing Techniques
There are several dimensions to categorize graph process-
ing techniques, such as: graph models, workload types, dy-
namicity, algorithm types and computing models. Figure 2
summarizes them. In the interest of space, I will not discuss
all dimensions. A detailed survey that describes these di-
mensions in detail is being prepared for publication1 . This
thesis focuses on dynamic algorithms for online and an-
alytical workloads on property graph(PG) models that
represent fully dynamic [17] graphs. Grab’n Fix is orthog-
onal to existing computation models in the literature.
3. METHODOLOGY
I intend to approach the research both algorithmically and
using a system-building approach. I develop dynamic algo-
rithms over large, dynamic graphs and incorporate these
algorithms into a prototype system. Algorithmically, I con-
sider optimization opportunities for future updates in a dy-
namic graph (such as incremental processing). These al-
gorithms will be incorporated into a parallel system over a
cluster environment to store and process large graph data.
My initial focus is on dynamic algorithms for three differ-
ent analytical queries, Pagerank , APSP (All Pair Shortest
Path) [6], and Connected Component [18]. The common-
alities between these algorithms contributed in my design
for the proposed computation model. These workloads are
chosen, because each has a different challenge. For example,
Pagerank can use precomputed ranks (before graph updates)
as a heuristic during the re-computation phase. On the
other hand, APSP is a classic graph problem which builds
an N × N matrix to store the shortest distance between
each pair of vertices. The main objective is to answer short-
est path queries in constant time using this matrix. There
have been significant effort made towards reducing the typi-
cal O(N 3 ) computation complexity of APSP; however, there
has not been much focus to reduce its O(N 2 ) memory re-
quirements. For large graphs, it is not feasible to build an
O(N 2 ) matrix in memory. Existing approaches to solve this
problem focus on incremental updates using native graph
algorithms on a single machine [2, 6, 17]; they either cannot
scale-out or cannot handle edge removals.
1
www.slideshare.net/MTamerOzsu/graph-lecture-58376147
 Function Grab (vertex v, Adj-list graph G, result
F (G), updates ∆G)) is
Result: Intermediate state I
1 I ← G + ∆G // Include the whole graph in the
Intermediate state
2 return I
Function Fix (Intermediate I, Adj-list graph G, result
F (G), updates ∆G)) is
Result: G0 , F (G0 )
3 G0 ← G + ∆G
4 F (G0 ) ← CC(I) // Run the connected component
algorithm CC
5 return G0 , F (G0 )
Algorithm 1: Static algorithm for Connected Component
considering vertex v ∈ G.
Recent research from Google [13, 18] proposed efficient
algorithms to find connected component. The implementa-
tion was presented for MapReduce and they show that in
shared-environment this could be more efficient than other
graph processing systems, such as Pregel [16]. The algo-
rithms work with incremental growing dynamic graphs only,
ignoring 40% of the updates in the Twitter following graph
that are edge removals. Note that deleting an edge is signif-
icantly harder than inserting an edge. For example, adding
an edge should always reduce the shortest path distance, or
merge two components. On the other hand, edge removal
can split a graph or isolate a vertex.
An essential part of my research is building a controlled
experimental study for parallel graph systems. The main
objectives are two-fold: prepare a universal benchmark for
future systematic evaluations, and evaluate existing systems
in analyzing different types of graphs to find their shortcom-
ings and address them in my system. Therefore, I built a
universal graph benchmark, Waterloo Graph Benchmark [3].
This is now being used in an extensive and systematic exper-
imental study of graph analysis systems. This study includes
systems from different computation models, such as vertex-
centric, block-centric, and map-reduce. I will also use the
same benchmark, and experimental framework to evaluate
the techniques that I develop as part of my PhD research.
4. COMPUTATION MODEL
The main requirement in dynamic algorithms is efficiently
computing changes to a previously computed answer. In or-
der to help developers write efficient algorithms that mini-
mize  in Equation 2, I propose a new graph computation
model specialized for dynamic graphs, which is orthogonal
to existing computing paradigms, such as Map-Reduce [5],
vertex-centric [1, 16], and block-centric [20].
The “Grab’n Fix” approach allows every entity in the
graph, such as a vertex, an edge, or a block, decide inde-
pendently whether or not it should be part of the overhead
(). Typically, to facilitate efficient dynamic computation,
it is required to store some information about the current
solution; I call this the provenance information. This is any
information used by the algorithm to compute F (G), but
may help reduce the overhead in computing F (G+∆G). For
example, during the computation of the connected compo-
nent, every vertex may store the shortest path to the graph
representative. Note that a common distributed connected
component algorithm starts identifying the component of
9
Function Grab (vertex v, Adj-list graph G, result
F (G), updates ∆G)) is
Result: Intermediate state I
1 if v ∈ ∆G then
2 if v ∈ add(u, v) then
3 if F (v)! = F (u) then
4 if F (v) > F (u) then
5 I←v
6 else
7 I←u
8 if v ∈ del(u, v) or (v ∈ add(u, v) &
F (v) == F (u)) then
9 if v.dist > u.dist then
10 I←v
11 else
12 I←u
13 return I
Algorithm 2: Dynamic algorithm for Connected Compo-
nent considering vertex v ∈ G. The Fix function has been
removed for the interest of the space. A vertex’s dist is the
maintained provenance information. It stores the distance
to the vertex with minimum id in the graph.
every vertex by the vertex id. As the algorithm iterates,
vertices communicate with their neighbors their component
id. The component id of every vertex is the smallest vertex
id that may reach this vertex. Therefore, the vertex with
minimum id is called the graph representative.
In the Grab’n Fix model developers need to implement
two functions: Grab and Fix. The Grab stage receives the
original graph G, its answer F (G) (along with any main-
tained provenance information), and the updates ∆G. From
each graph entity’s perspective (for example from vertex per-
spective) the Grab function describes how this entity can de-
cide if it should be part of the answer update process in the
Fix phase.
Algorithms 1 and 2 are both implemented using the Grab
and Fix functions. The former acts as a static algorithm; for
every update to the graph, the algorithm discards the pre-
vious answer and recomputes the connected component on
the updated graph. The second algorithm reduces the inter-
mediate state (I) to vertices that might be influenced by the
graph update(∆G). If a vertex is involved in add-edge oper-
ation, this means it might join a new component. Therefore,
the Grab function checks if the component for two vertices is
different, the vertex with larger component should be added
to I. If the components are identical, the vertex with larger
distance may have a chance to be closer, therefore it is also
added to I. Note that Algorithm 2 is not complete. It does
not include the Fix function for the interest of the space.
It also does not consider traversing the graph from a vertex
that was added to I to find potential vertices that should be
considered because one of their neighbors was added to I.
5. GRAPH STORAGE
It is common practice to keep all data in a “data lake’.
This frequently means a distributed file system that can ac-
commodate a large dataset created by different data sources.
A typical example for such a file system is Hadoop Dis-
tributed File System (HDFS). HDFS creates replicas of data
blocks (typically of size 64MB) across the cluster for back
up and recovery purposes. Graph processing systems typi-
cally process graphs in memory, and assume they are stored
using text file formats on HDFS [1, 14, 19, 20]. For dynamic
graph processing, my system (Figure 1) challenges these as-
sumptions from three dimensions: the distributed storage,
in memory restriction, and data structures.
5.1 Distributed Storage
Although HDFS has emerged as an industry standard for
distributed storage, it has limitations:
• It is not efficient in random reads.
• Incremental updates to datasets are efficient, but decre-
mental updates are not.
• The storage space and computation power are associ-
ated with each other.
There have been several attempts/alternatives to avoid
these limitations. For example, key-value stores, inspired by
Google BigTable [4], are distributed and fault tolerant, in
addition to being efficient with random reads and incremen-
tal/decremental update operation (using the key). Recently
SnowFlake proposed an elastic data warehouse that replaces
HDFS by Amazon Simple Storage Service (S3)2 . S3 is a dis-
tributed file system service. However, it is possible to use
one machine or thousands of machines to process the same
amount of storage at S3. HDFS does not have this flexibility
because the storage is distributed on the computing power,
the machines. However, similar to key-value stores, S3 offers
reading and writing data using their key, which is often a
file name. File updates are not permitted. Finally, Cloudera
recently proposed Apache Kudu3 , a new distributed storage
for fast analytics. It is trying to fill the gap between HDFS
efficiency in reading/writing large batch data, and key-value
store efficiency in processing small ad-hoc queries. Kudu
was designed to support large structured data; it has tables,
primary keys, and column types.
For dynamic graphs, HDFS cannot be the most efficient
option, because of the overhead of graph updates. Other
alternatives such as key-value stores, distributed column
stores (Vertica [12]), or a hybrid approach (Kudu) might
be interesting. However, since Kudu was designed for struc-
tured data and inspired by relational systems, we need to
introduce multiple modifications to efficiently support large
graphs. My experiments show that column store systems,
such as Vertica carry more overhead as the cluster scales out.
In clusters larger than 16 machines, Vertica under-performs
several existing graph processing systems.
Existing systems consistently assume a text file format
(edge or adjacency list) because they read the graph data
once, then write the output after the computation finishes.
However, text file format is inefficient. HDFS already has
different file formats, such as Sequence, Avro, and Parquet
file format. Every format has different advantages and disad-
vantages based on the use case. Since column-store formats
were shown to be efficient for graph processing [12], Parquet
format has a great potential on top of HDFS.
The proposed system will utilize the column-store features
in Kudu and Apache Arrow4 to ensure fast processing for
2
This work is accepted in SIGMOD 2016.
3 7
http://getkudu.io/overview.html
4 8
http://arrow.apache.org/
10
Dataset (size in GB) 16 32 64 128
Twitter (12.5) 191.5 323.6 606.4 923.5
UK0705 (31.9) 264.0 411.8 717.6 1322.6
Table 1: Giraph Memory consumption. All numbers are in
GB; first row shows cluster size.
graph updates. Arrow is an in-memory column store that
is compatible with multiple data stores (Parquet/HDFS,
Kudu, Cassandra) and processing frameworks (Spark).
5.2 Out-of-memory Computation
Many distributed graph processing systems keep the graph
in the cluster memory during computation. However, for
large real graphs, such as the web graph, this assumption is
hard to satisfy. My experiments show that most graph pro-
cessing systems cannot store the ClueWeb5 , in the aggregate
memory of 3TB6 .
There have been a few attempts to utilize out-of-memory
computation in Giraph and GraphlabCreate. I advocate for
out-of-memory computation for multiple reasons:
1. The in-memory restriction was based on using HDFS
as a distributed storage to avoid its inefficiency in pro-
cessing random read/write requests.
2. The overhead of storing a graph in memory signifi-
cantly increase as number of machines increases. This
overhead may overcome the benefit of in memory effi-
ciency. Table 1 shows the total memory consumption
by Giraph to process two real datasets: Twitter7 and
UK07058 .
3. For large graphs, changes do not impact the whole
dataset.
The proposed system utilizes local disks in the computa-
tion process, if necessary. Caching is essential for this set-
ting; however, proposing algorithms for caching graph enti-
ties (vertices, edges, etc) is beyond the scope of this thesis.
5.3 Data Structure
Graph systems adopt simple data structures for graph
processing, such as hash-map or Compressed Sparse Row
(CSR). These standard representations are not efficient for
dynamic graph processing. Recently, LLAMA [15] proposed
a modified version of CSR to support graph updates and
out-of-memory computation; however it is not distributed.
The proposed system utilizes the unique capabilities of
DiStinger [8], distributed data structure extension to Stinger [7].
Stinger is an efficient graph data structure built for dynamic
graphs and designed for a supercomputer (Cray XMT). In
a nutshell, DiStinger’s master machine has summary infor-
mation about the graph, which helps directing queries to
the right slave. Slaves have two main data structures: ver-
tex array and edge array. Vertex array stores statistical
information about each vertex, such as its degree. It has a
pointer to relevant edge block headers. Edge block headers
represent a linked list of edge blocks. Each edge block is as-
5
http://law.di.unimi.it/webdata/clueweb12/
6
My experiment cluster has 128 r3.xlarge amazon ma-
chines, each has 30 GB. Giraph [1], GraphLab [9], and
GraphX [10] could not accommodate the graph in memory.
http://law.di.unimi.it/webdata/twitter-2010
http://law.di.unimi.it/webdata/uk-2007-05/
sociated with group of edges, and stores information about
them, such as smallest and largest time stamp in the block.
DiStinger is a step towards an efficient distributed data
structure for dynamic graph processing. However, it needs
several modifications, such as augmenting DiStinger with
out-of-memory capabilities. This is going to reduce its mem-
ory demand, and hopefully increase its efficiency.
6. PROGRESS AND FUTURE WORK
I started my research by a comprehensive quantitative and
qualitative analysis for the distributed graph processing lit-
erature. Both studies are in the final stages. The quantita-
tive analysis is the first of its scope; it considers systems that
use different computation models to efficiently process static
graphs. These systems were evaluated using different clus-
ter sizes (up to 128 machines), and multiple graph datasets
including the largest public web graph and road network
graph. This wide-ranging experience grants me the ability
to understand the existing shortcomings and empower my
design decisions for the Grap’n Fix model and the proposed
architecture.
This thesis extends three main components in my archi-
tecture (Figure 1) to efficiently support dynamic graphs:
• Build a new graph processing system based on the
Grab’n Fix computation model.
• Extend DiStinger to interact with Apache Arrow and
to optimize the amount of data in memory/local disk.
• Extend Kudu to support graph structures instead of
relational data.
7. ACKNOWLEDGMENTS
I am indebted to my supervisor technical and personal
support. I am thankful for every faculty member and every
graduate student; the weekly meetings in the Data Systems
Group at University of Waterloo are invaluable. In par-
ticular, I appreciate the feedback from Dr. Ihab Ilyas, Dr.
Khuzaima Daudjee, and Dr. Semih Salihoglu. This research
is partially supported by Natural Sciences and Engineering
Research Council (NSERC) of Canada, and IBM Center of
Advanced Research (CAS).
8. REFERENCES
[1] Giraph. http://giraph.apache.org.
[2] T. Akiba, Y. Iwata, and Y. Yoshida. Dynamic and
historical shortest-path distance queries on large
evolving networks by pruned landmark labeling. pages
237–248, 2014.
[3] K. Ammar and M. Özsu. WGB: Towards a universal
graph benchmark. In e. a. Rabl, Tilmann, editor,
Advancing Big Data Benchmarks, Lecture Notes in
Computer Science, pages 58–72. Springer, 2014.
[4] F. Chang, J. Dean, S. Ghemawat, W. Hsieh,
D. Wallach, M. Burrows, T. Chandra, A. Fikes, and
R. Gruber. Bigtable: A distributed storage system for
structured data. ACM Trans. Comp. Syst.,
26(2):Article 4, 2008.
[5] J. Dean and S. Ghemawat. MapReduce: Simplified
data processing on large clusters. In Proc. 6th
USENIX Symp. on Operating System Design and
Implementation, pages 137–149, 2004.
11
[6] C. Demetrescu and G. F. Italiano. A new approach to
dynamic all pairs shortest paths. J. ACM,
51(6):968–992, 2004.
[7] D. Ediger, R. McColl, J. Riedy, and D. Bader. Stinger:
High performance data structure for streaming graphs.
In IEEE Conference on High Performance Extreme
Computing, pages 1–5, 2012.
[8] G. Feng, X. Meng, and K. Ammar. Distinger: A
distributed graph data structure for massive dynamic
graph processing. In IEEE International Conference
on Big Data, pages 1814–1822, 2015.
[9] J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and
C. Guestrin. Powergraph: Distributed graph-parallel
computation on natural graphs. In Proc. 10th
USENIX Symp. on Operating System Design and
Implementation, pages 17–30, 2012.
[10] J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw,
M. J. Franklin, and I. Stoica. Graphx: Graph
processing in a distributed dataflow framework. In
Proc. 11th USENIX Symp. on Operating System
Design and Implementation, pages 599–613, 2014.
[11] M. Han, K. Daudjee, K. Ammar, M. T. Ozsu,
X. Wang, and T. Jin. An experimental comparison of
pregel-like graph processing systems. Proc. VLDB
Endowment, 7(12):1047–1058, 2014.
[12] A. Jindal, S. Madden, M. Castellanos, and M. Hsu.
Graph analytics using vertica relational database. In
IEEE International Conference on Big Data, pages
1191–1200, 2015.
[13] R. Kiveris, S. Lattanzi, V. Mirrokni, V. Rastogi, and
S. Vassilvitskii. Connected components in mapreduce
and beyond. In Proc. 5nd ACM Symp. on Cloud
Computing, pages 18:1–18:13, 2014.
[14] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson,
C. Guestrin, and J. M. Hellerstein. Distributed
GraphLab: A framework for machine learning in the
cloud. Proc. VLDB Endowment, 5(8):716–727, 2012.
[15] P. Macko, V. Marathe, D. Margo, and M. Seltzer.
LLAMA: Efficient graph analytics using large
multiversioned arrays. In Proc. of ICDE, pages
363–374, 2015.
[16] G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert,
I. Horn, N. Leiser, and G. Czajkowski. Pregel: a
system for large-scale graph processing. In Proc. ACM
SIGMOD Int. Conf. on Management of Data, pages
135–146, 2010.
[17] J. Mondal and A. Deshpande. Managing large
dynamic graphs efficiently. In Proc. ACM SIGMOD
Int. Conf. on Management of Data, 2012.
[18] V. Rastogi, A. Machanavajjhala, L. Chitnis, and
A. Das Sarma. Finding connected components in
map-reduce in logarithmic rounds. In Proc. 29th Int.
Conf. on Data Engineering, pages 50–61, April 2013.
[19] Y. Tian, A. Balmin, S. A. Corsten, S. Tatikonda, and
J. McPherson. From “think like a vertex” to “think like
a graph”. Proc. VLDB Endowment, 7(3):193–204,
2013.
[20] D. Yan, J. Cheng, Y. Lu, and W. Ng. Blogel: A
block-centric framework for distributed computation
on real-world graphs. Proc. VLDB Endowment,
7(14):1981–1992, 2014.