Large Scale Distributed Data Science using Apache Spark
James G. Shanahan
NativeX and
University of California, Berkeley
875 Howard Street
San Francisco, CA, USA
James.Shanahan@NativeX.com
ABSTRACT
Apache Spark is an open-source cluster computing framework for
big data processing. It has emerged as the next generation big data
processing engine, overtaking Hadoop MapReduce which helped
ignite the big data revolution. Spark maintains MapReduce’s
linear scalability and fault tolerance, but extends it in a few
important ways: it is much faster (100 times faster for certain
applications), much easier to program in due to its rich APIs in
Python, Java, Scala (and shortly R), and its core data abstraction,
the distributed data frame, and it goes far beyond batch
applications to support a variety of compute-intensive tasks,
including interactive queries, streaming, machine learning, and
graph processing.
This tutorial will provide an accessible introduction to Spark and
its potential to revolutionize academic and commercial data
science practices.
Categories and Subject Descriptors
D.3.3 [Concurrent Programming]: • Distributed programming.
G.3 [PROBABILITY AND STATISTICS] • Statistical
computing.
General Terms
Algorithms, Measurement, Performance, Design, Reliability,
Experimentation, Security, Languages.
Keywords
Distributed Systems, Hadoop, HDFS, Map Reduce, Spark, Large
Scale Machine Learning, Data Science.
1. INTRODUCTION
Big data is a broad term for data sets so large or complex that
traditional data processing applications are inadequate.
Concretely, imagine an average size laptop (4-8 gig of memory,
dual-core, with one terabyte (TB) of disk space) that gets
overwhelmed with a machine learning task involving a 3-4 gig
dataset; imagine the same laptop taking 3 hours to read one TB of
data. Clearly, this type of single node sequential computing is
inadequate for processing terabytes/petabytes of data, which are
commonplace in modern day society where both machines
(internet of Things (IoT)) and humans generate petabytes of data
every day (Figure 1).
Permission to make digital or hard copies of part or all of this work for personal
or classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice
and the full citation on the first page. Copyrights for third-party components of
this work must be honored. For all other uses, contact the Owner/Author(s).
Copyright is held by the owner/author(s).
KDD '15, August 10-13, 2015, Sydney, NSW, Australia.
ACM 978-1-4503-3664-2/15/08..
DOI: http://dx.doi.org/10.1145/2783258.2789993
2323
Liang Dai
NativeX
875 Howard Street
San Francisco, CA, USA
Liang.Dai@NativeX.com
Until recently “big data” was very much the purvey of database
management and summary statistics systems such as Hadoop
(HDFS and MapReduce) and was largely underleveraged by
machine learning. These systems though useful suffered from
their limited utility (challenging to code up; lack of purpose built
tools and libraries). This course builds on and goes beyond this
collect-and-analyze phase of big data by focusing on how
machine learning algorithms can be rewritten and in some cases
extended to scale to work on petabytes of data, both structured
and unstructured, to generate sophisticated models that can be
used for real-time predictions. Predictive modeling at this scale
can lead to huge boosts in performance (typically in the order of
10-20%) over small scale models running on standalone
computers that require one to significantly down-sample, and
necessarily simplify big data.
Figure 1. Core concepts of data science at scale
Concretely, this tutorial focuses on how the Map-Reduce design
pattern from parallel computing can be extended and more
faithfully leveraged to tackle the somewhat “embarrassingly
parallel” task of machine learning (a lot of machine learning
algorithms fit this mold).
In this tutorial, this is accomplished via the Apache Spark project
and it many related sub projects. Spark has emerged as the next
generation big data processing engine, overtaking Hadoop
MapReduce which helped ignite the big data revolution. Spark
maintains MapReduce’s linear scalability and fault tolerance, but
extends it in a few important ways: it is much faster (100 times
faster for certain applications), much easier to program in due to
its rich APIs in Python, Java, Scala (and R), and its core data
abstraction, the distributed data frame, and it goes far beyond
batch applications to support a variety of compute-intensive tasks,
including interactive queries, streaming, machine learning, and
graph processing
2. Tutorial Outline
This tutorial provides an accessible introduction to Spark and its
potential to revolutionize academic and commercial data science
practices. It is divided into two parts:
 the first part will cover fundamental Spark concepts,
including Spark Core, data frames, the Spark Shell,
Spark Streaming, Spark SQL and vertical libraries such
as MLlib and GraphX;
 the second part will focus on hands-on algorithmic
design and development with Spark (developing
algorithms from scratch such as decision tree learning,
graph processing algorithms such as pagerank/shortest
path, gradient descent algorithms such as support
vectors machines and matrix factorization.
Industrial applications and deployments of Spark will also be
presented. Example code will be made available in python
(pySpark) notebooks. We also review some of the limitations of
Spark in its current form. See Table 2 for a list of the subject
matter that will be covered in this tutorial.
3. BIBLIOGRAPHY
[1] Karau, H., Konwinski. A., Wendell, and P., Zaharia, M..
2015. Learning Spark: Lightning-Fast Big Data Analysis.
O'Reilly Media.
[2] Owen, S., Ryza, S., Laserson, U., and Wills, J.. 2015.
Advanced Analytics with Apache Spark. O’Reilly Media.
Table 2: Tutorial subject matter outline
2324
[3] Zaharia, M.. 2011. Spark: In-Memory Cluster Computing for
Iterative and Interactive Applications. Invited Talk. NIPS Big
Learning Workshop: Algorithms, Systems, and Tools for
Learning at Scale (Granada, Spain. December 12-17, 2011)
[4] Apache Foundation. 2014. Cluster Mode Overview - Spark
1.2.0 Documentation - Cluster Manager Types. Apache Org.
[5] Databricks. 2015. Databricks Spark Reference Applications.
https://www.gitbook.com/book/databricks/databricks-spark-
reference-applications/details
[6] Shanahan, J. G., and Kurra, G., 2010. Web Advertising:
Business Models, Technologies and Issues in Information
Retrieval, Edited by Melucci, M., and Baeza-Yates, R..
[7] Panda, B., Herbach, J. S., Basu, S., and Bayardo, R. J.. 2009.
PLANET: massively parallel learning of tree ensembles with
MapReduce. Proc. VLDB Endow. 2, 2 (August 2009), 1426-
1437. DOI=10.14778/1687553.1687569
http://dx.doi.org/10.14778/1687553.1687569
[8] Takács, G. et al. 2008. Matrix factorization and neighbor
based algorithms for the Netflix prize problem. In:
Proceedings of the 2008 ACM Conference on Recommender
Systems, Lausanne, Switzerland, October 23 - 25, 267-274.
[9] Ott, P. 2008. Incremental Matrix Factorization for
Collaborative Filtering. Science, Technology and Design,
Anhalt University of Applied Sciences.