4th International Conference on Advanced Technologies
For Signal and Image Processing – ATSIP’ 2018
March 21-24, 2018 – Sousse, Tunisia
A Comparison of Big Remote Sensing Data
Processing with Hadoop MapReduce and Spark
I. Chebbi
RIADI Laboratory, LIASD Laboratory
University of Manouba Tunisia, University of Paris 8 France University of Manouba, Tunisia University of Paris 8, France
ichebbi88@gmail.com wadii.boulila@riadi.rnu.tn
M. Lamolle
LIASD Laboratory
University of Paris 8, France
m.lamolle@iut.univ-paris8.fr
Abstract—The continuous generation of huge amount of re-
mote sensing (RS) data is becoming a challenging task for
researchers due to the 4 Vs characterizing this type of data
(volume, variety, velocity and veracity). Many platforms have
been proposed to deal with big data in RS field. This paper
focus on the comparison of two well-known platforms of big RS
data namely Hadoop and Spark. We start by describing the two
platforms Hadoop and Spark. The first platform is designed for
processing enormous unstructured data in a distributed comput-
ing environment. It is composed of two basic elements : 1) Hadoop
Distributed file system for storage, and 2) Mapreduce and Yarn
for parallel processing, scheduling the jobs and analyzing big
RS data. The second platform, Spark, is composed by a set of
libraries and uses the resilient distributed data set to overcome
the computational complexity. The last part of this paper is
devoted to a comparison between the two platforms.
Index Terms—Big Data, Architectures, Hadoop, Spark, Remote
Sensing Image
I. I NTRODUCTION
Nowadays, a huge number of earth observation data is
delivered from RS sensors. This data is used in various
applications in several domains. Many researchers refer to data
coming RS as big data [1]. When coupling the complexity
of huge amount of data (big data) with the nature of RS
data, the problem becomes more challenging to researchers.
The complexity of big data is coming from the its 4Vs
(Volume, Velocity, Value and Veracity) and even many others
characteristics of this data [2]. Whereas, the complexity of
RS data refers to the multiplicity of image resolution (spatial,
spectral and temporal). Besides, this later type of complexity
can be increased specially when RS metadata is organized into
complex data structures [3]. Thus, the main challenging is how
to deal with these 4Vs in the RS big data.
Let us start with the first V: Volume. The volume of RS data
is increasing in terms of hours and minutes. Last years, we
are rapidly moving from terabytes to exabytes. For example,
in 2013, EOSDIS (NASA’s Earth Observation System Data
and Information System) recorded an RS data volume of
978-1-5386-5239-8/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: ULAKBIM UASL - Atilim Universitesi. Downloaded on November 21,2022 at 19:18:33 UTC from IEEE Xplore. Restrictions apply.
RSA-147
W. Boulila N. Mellouli
RIADI Laboratory LIASD Laboratory
n.mellouli@iut.univ-paris8.fr
I.R. Farah
RIADI Laboratory
University of Manouba, Tunisia
riadh.farah@ensi.rnu.tn
7.5 petabytes with 1.5 million of users. The second V is
Velocity. It refers to generating, analysing and interpreting a
rapid growth of RS data [4]. For instance, RS data archives
of EOSDIS reach 4 TB per day distributed on more than
600 millions data files. The third V concerns the Value. The
RS data can be multisource, multitemporal, multispectral or
even multi resolution. Multisource means RS images can be
taken from multiple sources such as RADAR, LiDAR, optical,
etc. Multitemporal means that RS images can be scenes of
multi dates images. Multiresolution means different resolution
(spatial or spectral) [5].
Furthermore, understanding big RS data [6] consists on know-
ing the three facets: quality, methodologies and applications.
• ”Quality”: RS data are acquired from various sensors.
The delivered data are complex, heterogeneous, and high-
dimensional. Therefore, the remote sensing data should
be analyzed, classified, preprocessed, and may be com-
bined with other traditional data in order to obtain better
knowledge.
• ”Methodologies”: are set of techniques, principles and
methods allowing to process and extract information from
big RS data. Methodologies can be composed by a set of
tasks such as preprocessing, processing, dimensionality
reduction, deployment, visualization, analysis and inter-
pretation. Methodology is important in RS big data since
many constraints interfere like uncertainty, processing
scalability, computational times, etc.
• ”Applications”: play an important role on the choice
of the two previous facets. Since RS data has many
applications fields. For example, a problem of classifi-
cation of low image resolution doesn’t require the same
quality of data and the same methodologies as a problem
of prediction in case of very high image resolution.
Moreover, processing multispectral images is not similar
to processing hyperspectral images.
The difficulty in processing big RS data comes from not
only the big volume of this data but also from many tasks
that characterize the RS data which are data acquisition,
preprocessing, storage, processing, analysis and interpreting
[7]. In literature, many frameworks are proposed to deal with
these problems. Among these framework, we list two popular
tools named Hadoop and Spark.
The rest of the paper is organized as follows. we begin by
presenting the framework hadoop with its component and we
focus on Map and reduce phases, section 2 is about Apache
Spark and its component and finally, we end up with summary
and a conclusion.

II. H ADOOP M AP R EDUCE

Hadoop MapReduce [8] is an open source programming
framework used for the processing of large structured and Fig. 1: Map and Reduce Phases
unstructured data sets that are stored in HDFS (Hadoop Dis-
tributed File System) in a distributed computing environment.
The current Apache Hadoop ecosystem is based on several • Jobtracker
modules which are Hadoop Kernel, MapReduce, HDFS, Yarn, • Datanode
Apache Hive, Zookeeper and many others. • Tasktracker
Apache Hadoop is basically composed of two core compo-
nents: Mapreduce and HDFS [9]. A brief description of these
two components is given below.
• MapReduce is a programming model that was built from
models found in the field of functional programming
and distributed computing [10]. Tasks in MapReduce are
broken down into three parts: map, reduce and driver.
They are organized in pairs of key and value. Mappers
emit key and value pairs and the reducers receive them,
work on them, and produce the final result. Applications
that use MapReduce read the data using map function (in
the form of key and value pairs) and produced an output
also in form of (key and value) pairs. This type of data
fit very well to distributed environment. Fig. 2: Master-Slave Architecture in Hadoop
Then, the reduce function takes the generated pairs and
produces the final results. Pairs are, generally, sorted
according to keys when entered to the reduce function. III. S PARK
The reducer logic would then work on each key group; Apache spark is a powerful open source platform for big
in this case, it would sum up the values for each key and data processing. It is considered as multipurpose platform that
would produce the final result. is characterized by its flexibility, scalability and speed. It is,
Figure 1 describes the functional representation of the also, considered as an open source common parallel computing
map and reduce functions. framework which has the advantages of MapReduce. Spark is
• HDFS is responsible for the storage of files. It was intended to run on top of Hadoop and can be an alternative to
designed and developed to handle large files efficiently the traditional batch map and reduce models [10]. Moreover,
[11]. It is a distributed filesystem designed to work on a Spark is well adapted for iterative process, fast queries and
cluster. The main goal of using HDFS is to facilitate the real time data processing [12].
storage of large files by splitting them into blocks and Spark integrates four main libraries :
distributing them across multiple nodes redundantly [9]. • SQL for querying in large and structured data.
The Hadoop MapReduce stores and processes data in a • MLlib which contains main learning algorithms and sta-
distributed architecture, to achieve this goal, Hadoop imple- tistical methods.
ments a master and slave model. The namenode and jobtracker • GraphX for processing graphs and networks.
daemons are master daemons, whereas the datanode and task- • Spark Streaming to process streaming data.
tracker daemons are slave daemons (Fig.2). Apache Hadoop Apache spark is characterized by a set of efficient machine
consists of the following daemons: learning algorithms and enhanced linear algebra Libraries
• Namenode [12] [13]. An optimized engine is added in Spark to support
• Secondary namenode general execution graphs [13]. A wide range of workloads

Authorized licensed use limited to: ULAKBIM UASL - Atilim Universitesi. Downloaded on November 21,2022 at 19:18:33 UTC from IEEE Xplore. Restrictions apply.
is covered by Spark including interactive queries, batch
applications, iterative algorithms and streaming. Also, by
using Spark framework the management burden of maintaining
separate tools is reduced. Indeed, it integrates the concept
of RDD (Resilient Distributed Dataset). Schematically, each
data partition remains in memory on its computing server
between two iterations while managing the principles of
fault-tolerance. Spark specific commands run in Java, Scala,
and some for Python.
Figure 3 depicts the architecture of Apache Spark.
Fig. 3: Apache Spark
RDDs are the core of Spark’s functionalities. They are set of
records or subjects of a specific types, partitioned or distributed
across multiple nodes in the cluster. If a node is affected by
a hardware or network failure. The resilient table is rebuilt
automatically on other nodes and the task is completed. The
main property of Spark RDDs is the ability to store them in
the memory of each node [14]. This saves a lot of disk access
which is the main lock, in terms of computing time, when
running iterative algorithms. Another specificity of Spark is the
broadcast variables. These variables are read-only and defined
from the master node, and they are known and kept in memory
of all the other nodes. An accumulator is a special case of
broadcast variables. However, is not a read-only varaiable.
Each local version can be incremented by each node while
the master node has access to global accumulation.
In the rest of this part we will detail the MLlib which is a
component of Spark, that has been developed into a tool that
supports many common machine learning algorithms including
deep learning [15]. Our description will be focused on the type
of data, labeled point, rating,
• Types of data: the use of RDD at the core of Spark’s
computation efficiency makes MLlib dependent on very
compelling data structures. A map functions used to
create objects of these types.
• Labeled point: this type is specific to learning algorithms
and associates a ”label” ; a real and a vector. This ”
label ” is either the value of the quantitative variable Y
Authorized licensed use limited to: ULAKBIM UASL - Atilim Universitesi. Downloaded on November 21,2022 at 19:18:33 UTC from IEEE Xplore. Restrictions apply.
to be modeled in regression, or a class code 0.0,1.0, ...
in supervised classification or discrimination.
• Rating: The type of data(note of an article by a customer)
specific to the recommendation systems and therefore to
the variable factorization algorithm (ALS).
• Model: The model class is the result of a learning
algorithm that has the predict() function to apply the
model to a new or resilient data observation of new
observations.
• Supported Methods and utilities: MLlib supplies fast,
distributed implementations of common Learning algo-
rithms, it provides low-level primitive and basic utilities
for convex optimization, distributed linear algebra, sta-
tistical analysis and feature extraction. It also supports
several I/O formats.
• Algorithm optimization: MLlib involves many optimiza-
tion to support efficient distributed learning and predic-
tion. For example, ALS algorithm for recommendation
makes careful use of blocking to reduce JVM garbage
collection overhead and leverage high-level linear algebra
operations.
• Pipeline API: MLlib provide native support for the di-
verse set of functionality required for pipeline construc-
tion such as a sequence of data pre-processing , feature
extraction, model-fitting and validation.
• Spark integration: MLlib benefits from the several com-
ponents of Spark integration. At the lowest level, Spark
core provides an engine with over than 80 operators for
transforming data, MLlib also leverage the other high-
level libraries packeded with Spark. SparkSQL , GraphX
are also integrated in Spark and their improvements lead
to MLlib improvements.
• Documentation: MLlib has an user guide which provides
a good documentation for users, describing its methods
and utilities with detailed code examples.
IV. D IFFERENCE BETWEEN H ADOOP M APREDUCE AND
A PACHE S PARK
This comparison (TABLE I) describes many differences
between Apache spark and Hadoop MapReduce. However, we
must specify that both of them are compatible with each other,
they integrate with all the data sources and file formats that
are supported by each of them and the duo are scalable.
V. C ONCLUSION
In this paper we described two frameworks for processing
RS bid data. These two frameworks are Hadoop and Spark. A
comparison between these two frameworks is made focusing
on their architectures and especially the libraries, the phases
and the tasks who are specialized in the development of
machine learning algorithms. This will be very useful for our
future work consisting in integrating deep Learning libraries
with Apache Spark and TensorFlow. The later is a framework
developed by Google based on neural networks for numerical
computations.
TABLE I: A comparison between Spark and Hadoop MapRe-
duce
Spark
Storage In memory storing and
processig
Data Processing Hybrid processing:batch
processing and stream
processing
Real-Time Processing It can process real-time
data from real-time event
streams
Code Writing Code is complex and
lengthy
Fault Tolerance Spark uses RDD which
rebuilds the lost partition
through the information it
already has
Execution Time It is 100 time faster than
MapReduce
Security Sparks security is cur-
rently in its infancy, of-
fering only authentication
support through shared
password authentication
Cost Spark uses large amounts
of RAM to run everything
in-memory, and RAM is
more expensive than hard
disks
Machine learning It comes with a Mllib
library to make things
simple and allows the
algorithm to perform
much better than
traditional MapReduce
programs
[9] I Chebbi, W Boulila, IR Farah, Improvement of satellite image clas-
sification: Approach based on Hadoop/MapReduce, 2nd International
Conference on Advanced Technologies for Signal and Image Processing,
Hadoop Mapreduce 2016.
On disk storing data [10] V. A. Ayma; R. S. Ferreira; P. N. Happ; D. A. B. Oliveira; G. A.
O. P.Costa; R. Q. Feitosa; A. Plaza; P. Gamba On the architecture
Batch processing of a big data classification tool based on a map reduce approach for
hyperspectral image analysis IEEE International Geoscience and Remote
Sensing Symposium (IGARSS), 1508 - 1511;2015
It fails when it comes [11] Yanfeng Lyu; Xunli Fan; Kun Liu ”An Optimized Strategy for Small
to real-time data Files Storing and Accessing in HDFS” IEEE International Conference
processing on Computational Science and Engineering (CSE) , Volume: 1, 611 -
614,2017.
Code is Compact [12] W Huang, L. Meng, D. Zhang and W. Zhang, ”In-memory Parallel
with the API of Processing of Massive Remotely Snesed Data Using an Apache Spark
Scala, Python, Java on Hadoop YARN Model”, IEEE Journal of Selected Topics in Applied
and Sparksql Earth Observations and Remote Sensing, pp. 1-17, 2016.
It achieves fault [13] Akhmedov Khumoyun; Yun Cui; Lee Hanku, ”Spark based distributed
tolerance through Deep Learning framework for Big Data applications” International
replication by using Conference on Information Science and Communications Technologies
TaskTracker and (ICISCT),Pages: 1 - 5, 2016
JobTracker [14] Shyam R. and Bharathi Ganesh H.B. and Sachin Kumar S. and Praba-
haran Poornachandran, Apache Spark a Big Data Analytics Platform for
It is slower than spark Smart Grid, Procedia Technology (21), 171 - 178, 2015.
[15] Hameeza Ahmed and Muhammad Ali Ismail and Muhammad Faraz
Hadoop MapReduce Hyder and Syed Muhammad Sheraz and Nida Fouq, Performance
has better security Comparison of Spark Clusters Configured Conventionally and a Cloud
features than Spark Service, Procedia Computer Science (82), 99 - 106, 2016.
it supports Kerberos
authentication, which
is a good security
feature but difficult to
manage
Hadoop is disk-
bound, so saves
the costs of buying
expensive RAM
It uses iterations
for the same data
,each iterative step
involves a map-
reduce sequences and
it does not support
all the algorithms

R EFERENCES
[1] Yan Ma, Haiping Wu, Lizhe Wang, Bormin Huang, Rajiv Ranjan, Albert
Y. Zomaya, Wei Jie: ”Remote sensing big data computing: Challenges
and opportunities”,Future Generation Comp. Syst. 51: 47-60,2015
[2] Bhosale, H. S, Gadekar D. P, ”A Review Paper on Big Data and
Hadoop”, International Journal of Scientific and Research Publications,
volume.4, issue 10, October 2014
[3] Michael R. Evans, Dev Oliver,XunZhou, and Shashi Shekhar,” Spatial
Big Data: Case Studies on Volume, Velocity, and Variety , in Big
Data: Techniques and Technologies in Geoinformatics” ,isbn 978-1-46-
658651- 2, CRC Press, 2014.
[4] Ahmed Eldawy, Mohamed F. Mokbel,” The Era of Big Spatial Data:
Challenges and Opportunities” ,MDM (2) 7-10,2015.
[5] Wadii Boulila, Imed Riadh Farah, Amir Hussain, A novel decision
support system for the interpretation of remote sensing big data, Earth
Science Informatics, vol. 11, Issue 1, pp 31-45, 2018
[6] I Chebbi, W Boulila, IR Farah, Big Data: Concepts, Challenges and
Applications Computational Collective Intelligence ICCCI(2), pp. 638-
647, 2015.
[7] Ahmed Oussous, Fatima-AzhraBenjelloun, AyoubAitLahcen, Samir
Belfkih, ”Big data Technologies: A survey”,Journal of King Saud
University-Computer and information Sciences,4 June 2017.
[8] Manika Manwal, Amit Gupta ”Big data and hadoop A technological
survey”, Emerging Trends in Computing and Communication Technolo-
gies (ICETCCT), February 2018

Authorized licensed use limited to: ULAKBIM UASL - Atilim Universitesi. Downloaded on November 21,2022 at 19:18:33 UTC from IEEE Xplore. Restrictions apply.