2015 Seventh International Symposium on Parallel Architectures, Algorithms and Programming

Spark㧦A Big Data Processing Platform Based On

Memory Computing
Zhijie Han Yujie Zhang
Institute of Data and Knowledge Engineering Institute of Data and Knowledge Engineering
Nanjing University of Posts and Telecommunications Henan University
Nanjing, Jiangsu, China Kaifeng, Henan, China

Abstract—Spark is a memory-based computing framework Spark is better suited for iterative applications, such as Data
which has a better ability of computing and fault tolerance, Mining and Machine Learning [5].
supports batch, interactive, iterative and flow calculations. In this
paper, we analyze the Spark's primary framework, core Spark revolves around the concept of RDD (Resilient
technologies, and point out the advantages and disadvantages of Distributed Dataset) [6]: RDD is a fault-tolerant collection of
the Spark. In the end, we make a discussion for the future trends elements that can be operated in parallel. Spark provides a new
of the Spark technologies. way in data computing with fault-tolerant, which effectively
reduces the disk and network I/O overhead. RDD is a fault-
Keywords—Spark; Memory Computing; Spark SQL; MLlib; tolerant, parallel data computing structure, which allows users
GraphX; Spark Streaming; to explicitly store the data in disk and memory, and it can
control data partitions. RDD is a read-only data collection,
I. INTRODUCTION natural supports fault tolerance. Because, RDD is an immutable
With the rapid developing of the Big Data over the past few data sets, and it is able to remember the graph of operations. At
years, more and more applications need to be extended to large the same time, RDD provides a rich set of operations to
Clusters [1]. Programmable Clusters environment has brought manipulate the data [6]. The following operations of RDD
several challenges: Firstly, many applications need to be implement the monad mode, which are very suitable for the
rewritten in a parallel manner, and the programmable Clusters Scala, such as map, flatMap, and filter. In addition, RDD
need to process more types of data computing; Secondly, the provides more convenient operations such as join, groupBy,
fault tolerance of the Clusters is more important and difficult; reduceByKey, etc.
Thirdly, Clusters dynamically configure the computing
Currently, Spark has developed into a data computing
resources between shared users ,which increases the
platform with many components. The ecosystem of Spark is
interference of the applications. With the rapid increase of
called BDAS (The Berkeley Data Analytics Stack) [7]. The
applications, Clusters computing requires a working solution to
core framework of BDAS is the Spark, which is supports for
suit different calculations. Spark is a concurrent framework for
Spark SQL (a query and analyze engine), MLlib (an underlying
common Clusters computing and data analyzing which is
distributed Machine Learning repository), GraphX (a parallel
developed by the UC Berkeley AMP Lab Laboratory. Spark
graph calculation framework), Spark Streaming (a flow
offers a very good platform to integrate MapReduce [2],
calculation framework), distributed memory file systems, and
Streaming [3], SQL, Machine Learning, Graph Processing and
resource management framework. This ecosystem is described
other applications. Also, Spark provides a consistent API and
as Fig 1.
the unified deployment, which offers a perfect solution for the
data computing on the Big Data.
This section describes the issues and advantages of Spark.
Section 2 describes what the Spark is and its core technology.
Section 3 describes the four sub-frames of Spark. Section 4
discusses trends of the Spark technologies. Section 5
summarizes the work in the paper and introduces the further
research work.
II. SPARK AND ITS CORE TECHNOLOGY
INTRODUCTION Fig.1. The ecosystem of the Spark
Spark is a general distributed computing framework which
In general, the core technology and the foundation
is based on Hadoop MapReduce algorithms [4]. It absorbs the
architecture of Spark is RDD. And the Spark SQL, MLlib,
advantages of Hadoop MapReduce, but unlike MapReduce, the
GraphX, Spark Streaming are the core members of the Spark
intermediate and output results of the Spark jobs can be stored
ecosystem.
in memory, which is called Memory Computing. Memory
Computing improves the efficiency of data computing. So,

2168-3042/15 $31.00 © 2015 IEEE 172

DOI 10.1109/PAAP.2015.41

Authorized licensed use limited to: Nitte Meenakshi Institute of Technology. Downloaded on October 27,2022 at 14:51:48 UTC from IEEE Xplore. Restrictions apply.
 III. THE CORE MEMBERES OF SPARK ECOSYSTEM
A. Spark SQL
As a component of Spark, Spark SQL [8] supports the SQL
implementation in Spark: Spark SQL is evolved by the Shark
[9], and reducing the dependence on the Hive. Spark SQL
absorbs the memory storage (In-Memory Columnar Storage) of
Shark, compatibility of Hive; it has a great development in
terms of data compatibility, performance optimization,
components extension. Generally, Spark SQL consists of four
modules: Core, Catalyst, Hive, and Hive-Thriftserver:
z The Core processes input and output data , getting the
data from different sources (RDD, Parquet, JSON,
etc.), putting the query results output as schemaRDD
[10];
z The Catalyst processes the query statement during the
entire process, including parsing, binding,
optimization, physical plans;
z Hive provides CLI and JDBC / ODBC interfaces for
Hive data processing. In these four modules, Catalyst
is the core part of the merits and its performance will
affect the overall performance.
Generally speaking, Spark SQL has three major advantages:
z Data compatibility: Spark SQL is not only compatible
with Hive, can also be obtained from the RDD,
parquet, JSON files.
z Performance optimization: in addition to use In-
Memory Columnar Storage, byte-code generation and
other optimization techniques, Spark SQL introduces
Cost Model for dynamic evaluation query, obtains the
best physical plans.
z Component extension: redefine and expand the parser,
analyzer and optimizer of SQL.
Spark SQL is the core project of the Spark, the optimization
of Spark SQL has an important significance, which mainly is
full utilized the hardware resources and distributed parallel
computing performance of the system.
B. MLlib
MLlib [11] (Machine Learning library) is a scalable
Machine Learning library of Spark, it includes relevant tests
and data generators. The performance of Machine Learning
algorithms it has a 100 times increase than MapReduce [2].
MLlib supports the main Machine Learning algorithms, such as
classification, regression, clustering, collaborative filtering,
dimensionality reduction, and supports Sparse Matrix [12]. So,
MLlib has brought a great convenience for the users. Among
the algorithms, the classification is the most basic work of
Machine Learning. Typical application scenario is to get the
profits resource of Internet companies. Most Machine Learning
algorithms include two parts training and predicting [13],
which Spark uses in Machine Learning. MLlib provides three
programming languages API: Python, Java and Scala. In
addition, MLlib also provides some sample code to facilitate
users with different backgrounds.
173
Authorized licensed use limited to: Nitte Meenakshi Institute of Technology. Downloaded on October 27,2022 at 14:51:48 UTC from IEEE Xplore. Restrictions apply.
Machine Learning is divided into classification and
regression analysis based on the results [14]. MLlib supports
many classification algorithms, such as SVM (Support Vector
Mschine), LR (Logistic Regression), DT (Decision Tree), NB
(Naive Bayes); MLlib supports regression analysis algorithms
such as Linear Least Squares, Lasso and Ridge Regression.
Many standard machine learning methods can be
formulated as a convex optimization problem; we can write
this as an optimization problem݉݅݊௪‫א‬ோ೏ ݂ሺ‫ݓ‬ሻ, where the
objective is equation (1).
ͳ
݂ሺ‫ݓ‬ሻ ؔߣܴሺ‫ݓ‬ሻ ൅ ݊ σ݊݅െͳ ‫ܮ‬൫‫ݓ‬Ǣ ‫ ݅ݔ‬ǡ ‫ ݅ݕ‬൯ (1)
In this equation, w is a variable vector which is called
weights in the code. Vectors ‫ݔ‬௜ ‫ܴ א‬ௗ are the training data
examples, forͳ ൑ ‹ ൑ , and the corresponding label which we
want to predict is‫ݕ‬௜ ‫ܴ א‬ௗ . We call the method is linear if
ሺ‫ݓ‬Ǣ ‫ݔ‬ǡ ‫ݕ‬ሻ can be expressed as an equation by the ‫ š ் ݓ‬and ›.
Some MLlib’s classification and regression algorithms are the
linear type.
MLlib can be used to make recommendations . The process
of movie recommendation is expressed as follows:
z Get data set: load dataset in the HDFS which contains
1 million ratings from 5000 users on 3000 movies;
z Collaborative filtering: MLlib supports model-based
collaborative filtering, which is described by a small
set of latent factors to predict missing entries. In the
process, we use the alternating least squares (ALS)
algorithm to make movies recommendation;
z Setup: In this process, we use a standalone project
template for this exercise. The first, we configure the
path and file , then we create a SparkConf object and
use it to create a Spark context object;
z Running the program: compiling the MovieLensALS
class and creating a JAR file, the output can be seen
on the screen;
z Rating elicitation: counting ratings which are
received from each movie and sort movies by rating
counts. Then, obtaining the highest rating of 50
movies and sampling a small subset to rating
elicitation;
z Splitting training data: MLlib will train different
models based on the training set, and select the best
model on the validation set based on RMSE (Root
Mean Squared Error). Finally, add the users ratings in
the training set and make recommendations;
z Training using ALS: The movies recommendations
use the method of ALS.train to train a bunch of
models, select and evaluate the best one;
z Movie recommendation: Accord to the training
model we can list the top 50 movies on the screen.
In general, MLlib is very outstanding between general
Machine Learning algorithms and tools.. Its evaluation index
contains Accuracy, Recovery, F-Value, ROC (Receiver
Operating Characteristic Curve) [15], Precision Recovery
 Curve, AUC (Area Under The Curve) [16], AUC is used to
compare model accuracy, Recovery, F-Value, ROC is used to
determine the threshold.
C. GraphX
GraphX [17] is a parallel computation API used for Spark
charts and graphs processing. GraphX is developed based on
Bagel, and has a great improvement on the performance and
the memory overhead reducing. GraphX extends the Spark’s
RDD by introducing Resilient Distributed Property Graph; a
multiple graph can be attached to each vertex and edge
attribute [18]. In order to support graph computing, GraphX
discloses a basic set of operators, such as: Subgraph,
JoinVertices, MapReduceTriplets, optimal transition Pregel
API. Moreover, GraphX contains a graphical algorithms and
builders package to simplify graphical analysis tasks.
As a graph processing framework of Spark, GraphX
supports the following algorithms: PageRank algorithm,
ConnectedComponents algorithm, TriangleCounting algorithm.
z PageRank algorithm is Google's proprietary algorithm
[19], is used to measure the relative importance index
of specific pages with other pages in SE (Search
Engine). As shown in equation (2). The importance of
a page can be determined by "votes". The votes of a
page are determined by all the chains. P1, P2... Pn are
studied the pages, M (Pi) is the number of linked in
page Pi, L (Pj) is the number of linked out page Pj, and
N is the total number of the pages.
ଵି௤ ௉௔௚௘ோ௔௡௞ሺ௣ೕሻ
ܴܲܽ݃݁ܽ݊݇ሺܲ௜ ሻ ൌ ൅ ‫ ݍ‬σ௣ ೕ
ே ௅ሺ௣ೕ ሻ
z Connected Components algorithm is used to find the
related users for a theme [20]. In GraphX, it is means
that in an undirected graph, if vertex vi to vertex vj has
a path, called vi and vj is communication. If the map
between any two vertices is communicating, called the
map is a Connected Graph [21], On the contrary, the
map is Unconnected Graph. One of the great
communication sub-graphs is called Connected
Component. Connected Graph has only oneself
Connected Component; Unconnected Graph has
multiple Connected Components. As shown in Fig 2.
Finding Connected Graph is a core application of
graph calculation such as identifying clusters by
keywords.
Fig.2. Unconnected Graph G and its two connected component H1, H2
Authorized licensed use limited to: Nitte Meenakshi Institute of Technology. Downloaded on October 27,2022 at 14:51:48 UTC from IEEE Xplore. Restrictions apply.
z Triangle Counting [22] algorithm is used to count the
number of sub-graphs in a large graph. In the GraphX,
it can be used for community discovery. It is used on
the Microblogging applications to express the mutual
concern objects. The relationship between the
concerns objects will form many triangles, the
triangles represent that there has a stable and intimate
relationship between the two objects.
D. Spark Streaming
Real-time processing of data streams motivation is to obtain
real-time data information, and get the most valuable data at
the arrival time [23]. Spark Streaming is a stream computing
framework based on Spark [24], it provides a rich API, and
integrates streaming, batch and interactive query applications.
Spark Streaming disperses stream computing tasks into a series
of short DStream (Discretized Stream) data [25], each piece of
data is converted into RDD. The DStream’s transformation
operation turns it into the Transformation of RDD. The
Transformation of RDD intermediate results can be stored in
memory. Fig.3 shows the entire process of Spark Streaming.
(2)
Fig.3. Spark Streaming processing flow
Spark Streaming becomes a hot topic on real-time
streaming data computing due to several advantages:
z Fault Tolerance: Fault tolerance is very important in
the stream computing. Spark Streaming uses the Fault
tolerance of RDD, each RDD is connected by lineage
[7], if input data is fault-tolerant, you can re-calculate
the error or unavailable partitions by transforming the
original input data. Otherwise, Spark Streaming can
copy input data streams when they are loading from
the disk and network. So, the Fault tolerance of Spark
Stream is more efficient than the continuous model,
such as Storm.
z Expansibility and Throughput: Spark Stream is able to
linearly extend to 100 nodes (each node 4Core) on
EC2 (Elastic Compute Cloud) [26], and it can process
data by the 60M records / s in a few seconds delay, In
the GREP (Globally search a Regular Expression and
Print) [24] test, the throughput of Spark Streaming
each node is 670k records / s, is far greater than Storm.
IV. DIISCUSSION
Spark specializes in a variety of business scenarios, such
as Machine Learning, graph computing, stream processing. The
174
 applications of Spark can be divided into the following three
categories: The first is Machine Learning and the computation
of complexity graph, such as relation analysis, similarity
measure, community discovery and data mining; The second is
the migration of Complex ǃ Interactive ǃ OLAP(On-Line
Analytical Processing)[27]; The third is Stream Processing.
Spark provides a rich application programming models and
many data processing applications. It has a high performance
and convenience. Thus, Spark has a great advantage in
becoming to the next nuclear platform for Big Data analyzing.
Meanwhile, the Spark also needs a further improve in
robustness, availability, stability and performance. And it can
be improved by the following aspects:
z Memory Management. The core advantage of the
Spark is memory computing, so how to manage
memory is very important for the Spark, such as we
can optimize the code to free up initial memory space;
we can formulate a hierarchical storage strategy to
rationally allocate the disk, hard disk, and memory.
By this methods we will make a full use of the cache,
get a greater space to analyze the data, reduce GC
(Garbage Collection) [28] overhead and improve
efficiency.
z Improving I/O throughput. Such as data local storage,
by this method we can reduce network bandwidth
consumption, improve network utilization. Otherwise,
in some sophisticated Machine Learning algorithms
we can use algebraic operations, such as matrix
calculation, if we take advantage of instruction-level
optimization, it will be greatly improved on
performance.
z CPU Optimizing. Taking account of the optimization
of the whole software stack can improve the
efficiency of the data computing. so, improving and
optimizing the CPU performance plays an important
role for the balance of a system.
In general, Spark will play an important role in the
ecosystem in data computing. If we can efficiently use the
memory, network, disk, and CPU, Spark will be better.
V. CONCLUSIONS
In this paper, we provide a general introduction of the core
technologies of the Spark. Based on the analysis of the present
core technologies, we propose the future trends of Spark
technologies, and make a preparation work for the next step
research work on memory optimization of the Spark.
ACKNOWLEDGMENT
The subject is sponsored by the National Natural Science
Foundation of P. R. China (No.61572260, No.61373017,
No.61572261, No.61300215, No.61170243), Scientific &
Technological Support Project of Jiangsu Province (No.
BE2015702), Postdoctoral Research Funding Plan of Jiangsu
Province (No. 1302084B), China Postdoctoral Science
Foundation (No. 2014M560439).
175
Authorized licensed use limited to: Nitte Meenakshi Institute of Technology. Downloaded on October 27,2022 at 14:51:48 UTC from IEEE Xplore. Restrictions apply.
REFERENCES
[1] Chongchitnan S, Silk J. A Study of High-order Non-Gaussianity with
Applications to Massive Clusters and Large Voids[J]. Astrophysical
Journal, 2010, 724(1):285-295.
[2] Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large
Clusters[J]. Operating Systems Design & Implementation, 2004,
51(1):147–152.
[3] Lee Y, Song S. Distributed Indexing Methods for Moving Objects based
on Spark Stream[J]. International Journal of Contents, 2015, 11(1):69-
72.
[4] Solovyev A, Mikheev M, Zhou L, et al. SPARK: A Framework for
Multi-scale Agent-based Biomedical Modeling[J]. International Journal
of Agent Technologies & Systems, 2010, 2(3):págs. 1-7.
[5] Zaharia M, Chowdhury M, Franklin M J, et al. Spark: cluster computing
with working sets[J]. Book of Extremes, 2010, 15(1):10-10.
[6] Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: A
fault-tolerant abstraction for in-memory cluster
computing[C]//Proceedings of the 9th USENIX conference on
Networked Systems Design and Implementation. USENIX Association,
2012: 2-2.
[7] Franklin M J. Making sense of big data with the Berkeley data analytics
stack[C]//SSDBM. 2013: 1.
[8] Armbrust M, Xin R S, Lian C, et al. Spark SQL: Relational data
processing in Spark[C]//Proceedings of the 2015 ACM SIGMOD
International Conference on Management of Data. ACM, 2015: 1383-
1394.
[9] Armbrust M, Xin R S, Lian C, et al. Spark SQL: Relational data
processing in Spark[C]//Proceedings of the 2015 ACM SIGMOD
International Conference on Management of Data. ACM, 2015: 1383-
1394.
[10] Conese A. Inferring latent user attributes in streams on multimodal
social data using spark[J]. 2015.
[11] Franklin M. Mllib: A distributed machine learning library[J]. NIPS
Machine Learning Open Source Software, 2013.
[12] Biswas S, Poddar S, Dasgupta S, et al. Subspace based low rank and
joint sparse matrix recovery[J]. arXiv preprint arXiv:1412.2700, 2014.
[13] Alpaydin E. Introduction to machine learning[M]. MIT press, 2014.
[14] Gronlund S D, Wixted J T, Mickes L. Evaluating eyewitness
identification procedures using receiver operating characteristic
analysis[J]. Current Directions in Psychological Science, 2014, 23(1): 3-
10.
[15] Demler O V, Pencina M J, D'Agostino R B. Equivalence of
improvement in area under ROC curve and linear discriminant analysis
coefficient under assumption of normality[J]. Statistics in medicine,
2011, 30(12): 1410-1418.
[16] Pereira F, Mitchell T, Botvinick M. Machine learning classifiers and
fMRI: a tutorial overview[J]. Neuroimage, 2009, 45(1): S199-S209.
[17] Xin R S, Gonzalez J E, Franklin M J, et al. Graphx: A resilient
distributed graph system on spark[C]//First International Workshop on
Graph Data Management Experiences and Systems. ACM, 2013: 2.
[18] Langville A N, Meyer C D. Google's PageRank and beyond: The
science of search engine rankings[M]. Princeton University Press, 2011.
[19] Blei D M. Probabilistic topic models[J]. Communications of the ACM,
2012, 55(4): 77-84.
[20] Gonzalez J E, Xin R S, Dave A, et al. Graphx: Graph processing in a
distributed dataflow framework[C]//Proceedings of OSDI. 2014: 599-
613.
[21] Chakrabarti A. CS85: Data Stream Algorithms Lecture Notes, Fall
2009[J].
[22] Sakaki T, Okazaki M, Matsuo Y. Earthquake shakes Twitter users: real-
time event detection by social sensors[C]//Proceedings of the 19th
international conference on World wide web. ACM, 2010: 851-860.
[23] Zaharia M, Das T, Li H, et al. Discretized streams: an efficient and fault-
tolerant model for stream processing on large clusters[C]//Proceedings
of the 4th USENIX conference on Hot Topics in Cloud Ccomputing.
USENIX Association, 2012: 10-10.
 [24] Possanzini C, Van Liere P, Roeven H, et al. Scalability and channel
independency of the digital broadband dStream architecture[C]//Proc
Intl Soc Magn Reson Med. 2011, 1863.
[25] Zhao H, Canny J F. High Performance Machine Learning through
Codesign and Rooflining[D]. PhD thesis, EECS Department, University
of California, Berkeley, 2014.
[26] Biggs J, Ebmeier S K, Aspinall W P, et al. Global link between
deformation and volcanic eruption quantified by satellite imagery[J].
Nature communications, 2014, 5.
[27] Hu M H, Tu S T, Xuan F Z, et al. On-Line Structural Damage Feature
Extraction Based on Autoregressive Statistical Pattern of Time
Series[C]//ASME 2014 Pressure Vessels and Piping Conference.
American Society of Mechanical Engineers, 2014: V005T10A009-
V005T10A009.
[28] Boyle W B, Fallone R M. System and method for optimizing garbage
collection in data storage: U.S. Patent 8,706,985[P]. 2014-4-22.

176

Authorized licensed use limited to: Nitte Meenakshi Institute of Technology. Downloaded on October 27,2022 at 14:51:48 UTC from IEEE Xplore. Restrictions apply.