Hashem2018 Article MapReduceSchedulingAlgorithmsA

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/329541245
MapReduce scheduling algorithms: a review
Article in The Journal of Supercomputing · December 2018

DOI: 10.1007/s11227-018-2719-5
CITATIONS READS
3 544
11 authors, including:
Ibrahim Abaker Targio Hashem Nor Badrul Anuar

University of Malaya University of Malaya
44 PUBLICATIONS 2,546 CITATIONS 169 PUBLICATIONS 4,347 CITATIONS
SEE PROFILE SEE PROFILE
Mohsen Marjani
11 PUBLICATIONS 261 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Multi-objective Software Defined Network Measurement View project
Looking forwards to explore the capability of data mining in big data. View project
All content following this page was uploaded by Ibrar Yaqoob on 09 April 2019.
The user has requested enhancement of the downloaded file.

The Journal of Supercomputing
https://doi.org/10.1007/s11227-018-2719-5
Ibrahim Abaker Targio Hashem, et al. [full author details at the end of the
article]
© Springer Science+Business Media, LLC, part of Springer Nature 2018
Abstract
Recent trends in big data have shown that the amount of data continues to increase at
an exponential rate. This trend has inspired many researchers over the past few years
to explore new research direction of studies related to multiple areas of big data. The
widespread popularity of big data processing platforms using MapReduce framework
is the growing demand to further optimize their performance for various purposes.
In particular, enhancing resources and jobs scheduling are becoming critical since
they fundamentally determine whether the applications can achieve the performance
goals in different use cases. Scheduling plays an important role in big data, mainly
in reducing the execution time and cost of processing. This paper aims to survey
the research undertaken in the field of scheduling in big data platforms. Moreover,
this paper analyzed scheduling in MapReduce on two aspects: taxonomy and perfor-
mance evaluation. The research progress in MapReduce scheduling algorithms is also
discussed. The limitations of existing MapReduce scheduling algorithms and exploit
future research opportunities are pointed out in the paper for easy identification by
researchers. Our study can serve as the benchmark to expert researchers for proposing
a novel MapReduce scheduling algorithm. However, for novice researchers, the study
can be used as a starting point.
Keywords Big data · Hadoop · MapReduce · Cloud computing · Scheduling

algorithms
1 Introduction
Recent trends in big data have shown that the amount of data continues to increase
at an exponential rate [1]. This trend has inspired many researchers over the past few
years to explore new research direction of studies related to multiple areas of big data
B Ibrahim Abaker Targio Hashem

ibrahimabaker.targiohashem@taylors.edu.my
Extended author information available on the last page of the article
123
I. A. T. Hashem et al.
[2–4]. Meanwhile, the widespread popularity of big data processing platforms using
MapReduce framework is the growing demand to further optimize their performance
for various purposes [5]. In particular, enhancing resources and jobs scheduling are
becoming critical since they fundamentally determine whether the applications can
achieve the performance goals in different use cases [6]. Scheduling plays an impor-
tant role in big data, mainly in reducing the execution time and cost of processing [7,
8]. Moreover, scheduling is definitely not a new problem; it has been widely studied
in distributed computing literature. The main process that the MapReduce platform
depends on significantly is tasks scheduling [9]. In MapReduce tasks, scheduling is
given access to the resources (e.g., process time, CPU, communication, bandwidth)
in order to perform the execution and achieve the optimum quality of service [10].
Moreover, solving the problem of scheduling may require making a discrete choice
in order to obtain a desirable solution among different alternatives [7]. For example,
Althebyan et al. [9] proposed a new scheduling algorithm to overcome the challenge
in tasks scheduling. The algorithm is based on utilized multi-threading principles that
consider the time and energy consumption factors. Likewise, Jayasena et al. [11] pro-
posed the implementation of multimedia big data analytics including data distribution.
The architecture of the proposed multimedia implementation system contains three
different layers: service, platform, and infrastructure. The MapReduce that runs on
Hadoop distributed file system and the media processing libraries Xuggler is used to
design and implement the platform layers of the system. This reduces the transcod-
ing time of large-scale data into certain formats. In addition, ant colony optimization
algorithm (ACO) is used to efficiently allocate resources in the infrastructure layer.
The result indicated that the ACO can allocate virtual machine with minimal response
time compared to particle swarm optimization and artificial bee colony.
MapReduce scheduling is an emerging technology and has been explored from a
various angles by many researchers over the past few years. More specifically, opti-
mization, building new algorithms, and framework in the heterogeneous environment
have been the major focused in big data research [12]. Beside, Hadoop cluster(s) has
also gained its popularity through storing and analyzing a large amount of unstructured
data in a distributed cluster environment. However, processing the massive amount of
data in a distributed cluster requires effective scheduling mechanism to deal with them.
As such, this work aims to provide a comprehensive and performance study of MapRe-
duce scheduling on the large-scale distributed environment, where jobs/tasks are in a
distributed fashion but under resource management control. Our focus is motivated,
first, by the current availability of resource scheduling frameworks and secondly, by
the widespread of different MapReduce scheduling algorithms. Moreover, this paper
aims to serve as a useful guide of research progress in MapReduce scheduling, as
well as a point of reference for future work in improving the scheduling MapReduce
algorithms or introducing novel scheduling algorithms and frameworks for big data
analytics.
In order to achieve the above aims, we first offer a comparison of Hadoop, Mesos,
Corona frameworks. Then, we provide a comprehensive study of the existing schedul-
ing systems and classify them into different categories based on their implementation
requirement. Our identified category that is based on the related MapReduce schedul-
ing is as follows: scheduling strategy, resource, requirement, optimization approach,
123
workload, and speculative execution. Subsequently, with each category, we highlight

the aspects of scheduling in a distributed environment and classify each according to
their basic characteristics. Moreover, a summary and comparative table of each cate-
gory are presented. In addition, the paper provides a performance evaluation for some
of the scheduling algorithms that have been used by the framework. Finally, discus-
sion and the future direction for MapReduce scheduling are presented. The limitations
associated with previous approaches of scheduling systems are pointed out for easy
identification by researchers.
Lee et al. [6], Rao and Reddy [13], Tiwari et al. [14], Doulkeridis and Nørvåg
[15] and Arora et al. [16] provided recent surveys on MapReduce framework and
its scheduling algorithms. These surveys covered a discussion on various MapRe-
duce scheduling algorithms as well as their weaknesses and limitations. Moreover,
Tiwari et al. [14] offered a MapReduce scheduling algorithm classification based on
the quality requirement, entity, and the environment. Although the literature provides
a useful guidelines, it does not address the actual development of each algorithm. This
paper offers a comprehensive survey by discussing the recent development of vari-
ous algorithms, the issues, and limitations of the existing approaches. Other related
works include, “MapReduce scheduling for deadline-constrained jobs in heteroge-
neous cloud computing systems” [17], “Malleable scheduling for flows of jobs and
applications to MapReduce. Journal of Scheduling” [18], and “Scheduling MapRe-
duce tasks based on estimated workload distribution” [19]. Moreover, Tang et al. [20]
proposed the modification of Hadoop called OEHadoop to improve convergence time
of the classical Hadoop. The pipeline replication process of the classical Hadoop
MapReduce is redesigned. A reconfigured optical multicast is redesigned to adapt to
the architecture of the data center network for multicasting traffic. The adjustment of
the data center network architecture and the exchange of information with the applica-
tion layer are achieved by implementing a software-defined network controller in the
data center. The experiment result shows that the proposed multicast request schedul-
ing algorithm performs better than the classical results and the OEHadoop converges
faster than the classical Hadoop.
The paper is structured as follows: Sect. 2 offers the background information on
the current resource scheduling frameworks and presents a comparison of the popu-
lar resource scheduling frameworks. Section 3 provides a taxonomy of MapReduce
scheduling. We also provide a performance analysis of scheduling in MapReduce in
Sect. 4. Section 5 provides future research directions before concluding remarks in
Sect. 6.
2 Background
Before we begin our discussion on the taxonomy of MapReduce scheduling, we briefly

recall some of the resource scheduling frameworks that are being used by the Hadoop
framework. This is to provide readers with basic operations of the resource scheduling
framework.
123
Table 1 A comparison of resources scheduling frameworks
Features MapReduce Yarn [22] Mesos [23] Corona [24]

default [21]
Resources Request based Request based Offer based Push based

Scheduling Memory Memory Memory/CPU Memory/CPU/Disk
Cluster utilization Low High High High
Fairness No Yes Yes Yes
Job latency High Low Low Low
Scalability Medium High High High
Computation Job/task based Cluster based Cluster based Slot based
model
Language Java Java C++ –
Platform Apache Hadoop Apache Hadoop Cross-platform Cross-platform
Open source Yes Yes Yes Yes
Developer ASF ASF ASF Facebook
2.1 Resource scheduling frameworks
The need for resource scheduling is most evident in big data storage and complex
processing engine. The growing demand over the past last decade has resulted in a
lot of scheduling frameworks have emerged and also become popular, including Yarn,
Mesos, and Corona. These frameworks are used for parallel data-intensive applications
such as Hadoop, Spark, Kafka. Table 1 shows a comparison of MapReduce default,
Yarn, Mesos, and Corona scheduling frameworks.
2.1.1 Yarn
Hadoop Yarn is a framework, which provides a management solution for big data
in distributed environments. The main idea is to separate the resource management
and job scheduling from the data processing which allows Hadoop to supports vari-
ous big data computing paradigms such as interactive analysis and stream processing
[22]. Moreover, Yarn offers a Hadoop framework, a giant flexibility in terms of job
completions, which offer an effective management and monitoring of the workloads.
Moreover, such features can assist in providing support for the maintenance of the
multi-tenant environment, cluster utilization, high scalability and the implementation
of the security controls for the Hadoop framework. Yarn consists of two main com-
ponents which are: ResourceManager and ApplicationMaster as shown in Fig. 1. The
ResourceManager is used as the master to manage the available resource in the cluster,
whereas ApplicationMaster is working with the NodeManagers to start the containers
and used for monitor and execute resources.
123
NodeManager
ApplicationMaster
Scheduler
Client NodeManager
ResourceManager
NodeManager
Fig. 1 Illustration of Yarn architecture
2.1.2 Apache Mesos
Apache Mesos [23] is a cluster manager, originally developed at Berkeley, University

of California, to offer effective heterogeneous resources isolation and allocation across
distributed applications. Apache Mesos is an open source and is being used by many
companies such as Twitter, MediaCrossing, Xogito, Airbnb, and Apple. It offers an
abstraction of computing resources (CPU, storage, network, memory, and file system)
from nodes aiming to efficiently manage and deploy various big data applications
in distributed clustered environments. Mesos adapting features of the modern ker-
nel—“cgroups” in Linux, “zones” in Solaris and reside between the operating system
layer and the applications layer which offers isolation for rack locality, I/O, CPU, and
memory. This can allow applications like Hadoop, Spark, Kafka to run dynamically
across a shared pool of nodes. Mesos determines which resources are available, and
it makes offers back to an application scheduler [25]. Those offers can be accepted
or rejected by the framework. The Hadoop framework uses Mesos-aware scheduler
in order to register against the Mesos master. The Mesos master sends offers with
available resources to the registered Hadoop. When MapReduce requires launching
tasks, it replies to the master that it is taking part in the offers to launch the tasks.
Hadoop often uses a pre-defined set of resources to task for one of its specific tasks.
Figure 2 illustrates the basic architecture of the Apache Mesos framework.
2.1.3 Corona
Corona is an extension of the MapReduce framework; it provides high scalability and

cluster utilization for small tasks. This extension was designed to overcome some of
the important Facebook challenges, such as scalability, low latency for small jobs, and
processing needs [24]. Facebook has rewritten its scheduling framework in Corona,
which is based on a task resource requirement rather than a count of the map and
reduce tasks. Cluster manager is also introduced in Corona to monitor nodes in the
123
Mesos Master
Mesos Agent Mesos Agent Mesos Agent Mesos Agent Mesos Agent Mesos Agent
Fig. 2 Illustration of Apache Mesos architecture
cluster and report their available resources. For each job, a dedicated job tracker is
initialized in either small or large job with a separate process.
2.1.4 Summary
A comparison of the discussed frameworks is shown in Table 1. It outlines the type

of resource scheduling mechanism used in each framework. In contrast with Corona
and Mesos, where push based and offer based are employed to produces less delay for
jobs in the data center; default MapReduce and Yarn employ request-based scheduling
model to manage Hadoop jobs. The default MapReduce and Yarn are primarily written
in Java, while Mesos is primarily written in C++. Moreover, Yarn, Mesos, and Corona
frameworks have high cluster utilization and scalability compares with the default
MapReduce framework. Mesos and Corona can be used to manage the entire resources
in the data center, whereas default MapReduce and Yarn are not capable of managing
the entire resources in the data center, but rather managing Hadoop jobs at the cluster
level [25].
3 Taxonomy of MapReduce scheduling
This section focuses on the use of scheduling algorithms on big data platforms like
MapReduce. These scheduling algorithms can be classified into the following cate-
gories
123
(a) MapReduce framework based on scheduling strategy: adaptive and non-

adaptive.
(b) Resources in order to execute the workload such as CPU, Memory, Disk.
(c) Scheduling requirement determines the data locality, SLA, and the granularity
or the level of detail considered when making a scheduling decision.
(d) The optimization approach is to find an optimal solution that is corresponding
to the minimization and/or maximization such as response time, throughput,
resources usage, deadline.
(e) The workload defines the consumers of the resources.
(f) Speculative execution when a particular node in the cluster is taking a long time
to complete a task, the duplicate task is created on another node.
Figure 3 provides a classification of MapReduce scheduling according to the used

algorithms in which we base our discussion. We also provide a list of existing resource
scheduling framework and MapReduce scheduling algorithms that have been modified
into on Hadoop framework in “Appendix A” (Table 9).
3.1 Scheduling strategy
There have been numerous studies that attempt to solve tasks scheduling problems
in MapReduce with the aim of making use of the framework efficiently. MapReduce
scheduling algorithm is classified using two strategies to manage workload according
to the way they schedule the tasks as follows: (1) adaptive algorithm which consider
data, physical resources and workload while taking scheduling decisions [14], (2)
non-adaptive where each task are assigned a fixed number of resources at runtime
as shown in Fig. 4. Homogeneous virtual machines (VM) in cloud data processing
platforms can clearly affect the time needed to complete the tasks influencing the
overall performance. The proposed algorithm in [26] developed to allocate VM with
the minimum data transfer delay and faster completion time based on the VM location
and the triangular equality and inequality. The MapReduce job assigned to VMs with
adjacent juxtaposition will execute faster. Pulgar-Rubio et al. [27] proposed a novel
algorithm based on evolutionary fuzzy systems for subgroup discovery in big data.
The development of the proposed algorithm was done in the Apache Spark based
on MapReduce. To evaluate the effectiveness and efficiency of the algorithm, the
experiment is conducted with the high-dimensional dataset. The result shows that
the algorithm significantly reduces the runtime, while values in the standard measure
quality in the subgroup were maintained.
3.1.1 Adaptive scheduling strategy
The adaptive scheduling algorithm is the process of prioritizing and coordinating the
time necessary for tasks to be executed in order to meet their desired objectives [28].
For example, the adaptive scheduler would be one which takes many parameters to
allocate a task to a node based on the availability, capacity, and load on nodes in the
cluster at runtime [29].
123
123
Scheduling in MapReduce
Scheduling strategy Resource Requirement Optimization approach Workload Speculative execution
Node
Adaptive Heterogeneity Data locality Single objective Source
Rack
Non-Adaptive Sharing SLA-Based Multiobjective Job structure
Jobs
Level Real time

Resource Aware
Map/Reduce tasks
Fig. 3 Taxonomy of scheduling in MapReduce

Fair scheduler
Capacity scheduler Adaptive Scheduling Strategy FIFO

Non-Adaptive
Delay scheduling
Others
Fig. 4 Taxonomy of scheduling strategy in MapReduce
Many researchers have attempted to propose adaptive scheduling strategies. These

efforts are discussed as follows:
Fair scheduler [30] is a method of assigning equal resources for jobs in Hadoop
cluster this way small tasks which require more CPU time can run in parallel with
other tasks. The fair scheduler is an open source Apache Hadoop project, originally
developed by Facebook before released to Apache Hadoop community. Unlike the
default Hadoop scheduler which forms a queue of jobs, fair scheduler allows an iden-
tical share of the cluster capacity between pools over time. A minimum number of
map and reduce slots are allocated for each pool. The submissions of the new jobs in
MapReduce normally are assigned to unused resource from the pool. As the result,
each job submitted for processing in MapReduce may eventually receive almost an
equal amount of resources. In order to manage the resource efficiently, fair scheduler
uses priorities for each job assigned based on the weights to deal with the portion
of total resources. Moreover, fair scheduler supports preemption where jobs are dis-
missed from pools for some time. For instance, the submitted job will be divided into
multiple tasks and assigned to the available slots for processing. Thus, the scheduler
checks the time deficit against the ideal fair allocation for that job [13]. Once the
tasks have completed and the slot becomes free for next scheduling, the tasks with
high priority are assigned to that slot. Gradually, such a process will have an effect of
guaranteeing that the jobs get roughly equal amounts of resources.
The jobs with a small number of tasks are allocated sufficient resources in order to
finish the tasks very fast. However, the problem with such a scheduler is the resource
congestion when dealing with a lot of a number of tasks.
Capacity scheduler [31] is originally developed at Yahoo to run Hadoop applica-
tions as shared, where the fair allocation of computation resource is critical to the user.
The main idea is that Hadoop MapReduce cluster is partitioned using the available
resources between users who collectively use the cluster based on computing needs.
Besides, the capacity scheduler is similar to the fair scheduler, it uses a queue to allo-
cate jobs to users, but with the major difference of using prioritized queue jobs. Each
queue is given a configured capacity, which contents a scheduling that operates on a
modified priority queue basis with specific user limits. The queue with less number
of jobs is selected if the slot becomes available in order to scheduler the task for that
job. Generally, this may have an impact on the cluster capacity sharing among users
than among jobs, as was the case in the Fair Scheduler.
Delay scheduling [32] is proposed to improve data locality in MapReduce, in a
situation where there is a conflict between fairness in scheduling and data locality.
The idea is that if the next jobs can be scheduled according to fairness, then that
123
job cannot launch local tasks. Such a solution is suitable if the jobs to be scheduled
are less and the task is not been scheduled locally. Thus, delaying its scheduling
time can significantly improve data locality. The result shows that delay scheduling
can achieve nearly optimal data locality. Moreover, it outperformed fair sharing by its
simplicity under a wide variety of scheduling policies. Also, Tan et al. [33] suggested an
analytically tractable model under different schedulers such as default FIFO Scheduler
and the popular Fair Scheduler for job processing delay distribution in MapReduce.
3.1.2 Non-adaptive scheduling strategy
The non-adaptive scheduler is the process of scheduling in which the basic control
mechanism is not modified on the basis of the system activity. FIFO [34] is the default
Hadoop scheduler and the most popular algorithm in the non-adaptive scheduling of
Hadoop MapReduce. Possibly the most straightforward approach to schedule task is
to maintain a FIFO run queue based on policies or the solution of some optimization
problems.
Casas et al. [35] proposed a genetic algorithm to solve the challenges of scheduling
scientific workflows in the cloud computing environment. This is because a sched-
uler that manages both computing and data-intensive applications is scared. The
genetic algorithm crossover and mutation operators were modified to allow the addi-
tion/removal of a virtual machine from a given chromosome. The experiment indicated
that the proposed genetic algorithm scheduler was found to be better than the state-
of-the-art schedulers in terms of makespan and monetary cost.
3.1.3 Summary
According to the scheduling strategy discussion on the current research improvement,

we summarize the implementation approach and limitations for MapReduce schedul-
ing algorithms in Table 2. For each work, we present the objective of the algorithm, the
implementation approach and the limitation of each algorithm. We observe that most
of the algorithms are a focus on improving the performance of jobs or tasks scheduling.
On the basis of our analysis, we anticipate, gradually the challenges can be reduced to
achieve accurate scheduling algorithm by exploring the underlying research progress
on schedule.
3.2 Resources
Set of CPUs/vCPUs, memory, I/O, networking connectivity resources that can be

consumed by nodes in the cluster to accomplish or support tasks or activities. These
resources can be either heterogeneous or homogeneous in any cluster.
3.2.1 Heterogeneity
Generally, resource heterogeneity in the cluster may compress of either heteroge-

neous or homogeneous [44]. Homogeneous clusters mean that the nodes in the cluster
123
Table 2 Summary of MapReduce scheduling algorithms
References Algorithm Objectives Implementation approach Limitation
[21] FCFS To minimize the completion Run a queue Does not handle multiple jobs at
time the same time
Each task’s progress is Low performance when running
compared with the average multiple jobs
progress.
Poor response times for short
jobs compared to large jobs
Fair scheduling Share resources equally among Fair sharing The inefficient resource can be
the jobs assigned selected for a job
Pool weights Resource and job mismatch

Capacity scheduling It provides capacity guarantees Queue
for queues. Provide simplicity
for the organizations to share
clusters
Capacity guarantees
[36] LATE To improve response time for Progress rate is calculated by Deal with slow tasks only in the
short jobs dividing the progress score cluster
Processing time of the tasks not
accurate and finding the real
slow tasks is difficult
Poor performance in terms of
dynamically
[37] Quincy Reduce the scheduling problem Fairness and data locality Resource sharing
to the min-cost flow problem
123
Table 2 continued
References Algorithm Objectives Implementation approach Limitation
123
Predictability and consequently
fairness
[38] MCP To solve the problem of the Based on the bandwidth and Poor performance due to the
speculative execution progress rate static manner in computing
the progress of the tasks
[39] SHadoop To improve the performance of Optimize the setup and cleanup Hadoop cluster can only be
the job completion tasks statically configured
Messaging communication It does not work well in
mechanism heterogeneous
[40] MRA++ Provide a set of algorithms for Grouping Running multiple jobs slow the
heavy workload application performance of the processing
when dealing with a large
amount of data
Data distribution
Task scheduling
[41] Maestro Improve the overall Using a replica-aware execution Maestro algorithm can only be
performance of MapReduce of map tasks statically configured
computation
[42] ARIA To provides scheduler Uses job profiles and a soft It does not consider node
mechanism for job completion deadline failures
deadline
[43] Flex To optimize any of the variety Flexible scheduling allocation Schemes are very metric
of standard scheduling theory scheme dependent
metrics
have similar resources such as CPU, memory, storage and networking capabilities,
whereas heterogeneous clusters consist of different resources with the nodes in terms
of CPU, memory, storage, or communication speeds [45, 36]. Hadoop assumed that
the resources are similar or often the only scheduling constraint is homogenous and
the data locality [46, 47]. However, the popularity of virtual machines in many data
centers has brought about heterogeneous in Hadoop cluster. As the result, a virtual-
ized environment like cloud computing can contain more than one cluster with various
characteristics [47, 48]. Many of the scheduling algorithms are not aware of the various
characteristics of the virtual machines, which may affect the selection of the data local-
ity in the Hadoop cluster. Krish et al. [47] adopted a novel method of tackling this issue
by studying different applications running on various hardware configurations. Then
incorporate the information into a hardware-aware scheduler in order to improve the
resource-application match. Moreover, Ahmad et al. [46] introduced an optimization
approach named Tarazu in order to improve the performance of the MapReduce. The
approach is based on communication-aware scheduling, communication-ware load
balancing of map computation and the predictive load balancing.
An adaptive task tuning (Ant) approach to improve MapReduce clusters per-
formance in heterogeneous environments was proposed by Cheng et al. [49]. The
proposed technique handles practical issues and challenges that are related to auto-
matic configurations of MapReduce workloads (i.e., in large scale) in heterogeneous
environments. In Ant, jobs are modified with different settings to equal the capacities
of the available heterogeneous nodes by providing the ideal settings for each job. In
addition, it used in several clusters because of its flexible and adaptive nature.
3.2.2 Sharing
Resource sharing in Hadoop MapReduce is shown to be challenging tasks, more impor-

tantly allocating resources in the cluster efficiently and adopted various applications
dynamically for achieving its performance goals [50, 51]. It shows how jobs can be
influenced by other jobs to improve the performance. When a job runs on a physical
cluster in a distributed fashion, each node processed its tasks independently, which
may not influence the performance of other jobs residing in the same cluster, whereas
allowing virtual machines to participle in the cluster can enable resources sharing
between different jobs with minimum interference [52]. Moreover, in a cloud environ-
ment, virtual machines shared the physical resources among them running different
jobs. Chen et al. [53] focused on a practical heuristics for scheduling the tasks for the
problem scheduling all three phases conjointly during the MapReduce process with a
view to understanding the theoretical complexity of the joint scheduling.
3.2.3 Resource aware
In MapReduce with multi-job workloads, resource aware is important for enhancing

resource utilization across nodes [54]. It provides fast reconfigurable architectures
capable of adapting at runtime according to changing requirements and constraints
[55]. For example, to minimize contention for CPU and I/O Yong et al. [56] proposed
resource aware schemes on the slave’s nodes to improve cluster performance. Such
123
a solution can offer a learning mechanism in which tasks can be classified based on
their resources and assign jobs as appropriate. Furthermore, Guo et al. [57] offered
a new way of resource aware, named, resource stealing to allow tasks running in the
cluster. The idea is, some resources from other idle slots are used until new tasks are
assigned to those slots. Such a method can help to prevent wasting resources of the
cluster. The results show that resource stealing may enhance the performance of the
compute-intensive and network intensive applications.
3.2.4 Summary
Based on our discussion above on the latest research progress, we summarize the
resource based on three subcategories: heterogeneity, sharing, and resource aware of
MapReduce schedule in Table 3. There are the different mechanism used to develop the
scheduling algorithms, the most common one is the queue-based mechanism such as
FIFO, Fair scheduler, and Maestro. In regard to heterogeneity, most of the MapReduce
scheduling algorithms are implemented to work on the homogenous environment,
except the few that are supportive of the heterogeneous environment and the resource
can be sharable among the nodes. However, these algorithms can only deal with the
resources in statically.
Table 3 A summary of the resources in MapReduce scheduling
Algorithm Mechanism Heterogeneity Sharing Resource aware
FIFO [21] Queuing Homogeneous No Static

Fair scheduler [30] Queuing/sharing Homogeneous Yes Static
Capacity scheduler Sharing/priority Homogeneous Yes Static
[31]
Delay scheduler Relax queuing policy Heterogeneous Yes Static
[32]
Maestro [41] Queue utilization Heterogeneous Yes Static
LATE [36] Priority based Heterogeneous Yes Static
COSHH [58] Queue utilization Heterogeneous Yes Static
MCP [38] Progress rate and the Heterogeneous/ Yes Static
processing bandwidth homogeneous
SHadoop [39] Instant messaging Homogeneous Yes Static
communication
MRA++ [40] Establishes a training task Heterogeneous Yes Static
to gather information
prior to the data
distribution
Quincy [37] Flow-based scheduling Heterogeneous Yes Semi-static
ARIA [42] SLO based Heterogeneous Yes Dynamic
FLEX [43] Flexible scheduling Homogeneous Yes Semi-static
allocation scheme
123
3.3 Requirement
In this subsection, we have provided a discussion of the resources and its effect in
determining the scheduling requirements such as data locality and SLA. Scheduling
level is also one of the important requirements which determine the granularity or the
level of detail considered when making a scheduling decision. We discuss three levels
of scheduling decisions: job, task and speculative.
3.3.1 Data locality
Data locality is a major part of the MapReduce framework during the assignment of
the tasks for data processing in data parallel systems. Data locality is the assigning
of the tasks locally or close to the data. Data locality consists of many levels such
as node and rack level. Hadoop MapReduce determines whether cluster/rack is being
scheduled based on the availability of the data locally. It assumes that nodes within
the same rack have higher bandwidth than those that are not resident in the same
cluster. Knowing this, the scheduler can simply increase data locality for tasks. One
of the main objectives of using data locality is to overcome the problem of network
traffic in data-intensive computing [59]. Some scheduling policies in Hadoop consider
the effect of data locality, which can be classified as a cluster and rack [60, 61]. For
instance, to assign map tasks to a node, the Hadoop default FIFO chooses the job from
the queue and schedule its local map tasks. When the job does not have any map task
locally, it will be assigned to the non-local map in the cluster [62]. Furthermore, Abad
et al. [63] proposed a new algorithm based on distributed adaptive data replication that
helps the scheduler to achieve better data locality. The advantages of this approach
are to allow many replicas to be allocated for each file and make use of probabilistic
sampling. Zhang et al. [64] emphasized on data locality problem of MapReduce by
introducing a next-k-node scheduling method, which has implemented in Hadoop
framework. Also, Jin et al. [65] suggested that initial task allocation is produced first
before the job completion time can be reduced gradually. The author introduced a
heuristic task scheduling algorithm called BAlance-Reduce (BAR), which adjust data
locality dynamically according to network state and cluster workload by tuning the
initial task allocation using a global view. The experimental result of the algorithm
shows that the BAR is able to outperform previously related algorithms in terms of the
job completion time and deal with large problem instances in a few seconds. Wang
et al. [66] demonstrated a resilient proof through practical experimentations to illustrate
that the conventional concept of data locality is generally not constantly suitable for
MapReduce in a virtual environment. This shows that there is to differentiate between
the physical local nodes and the virtual nodes. To solve this issue, a task scheduling
algorithm mainly for the purpose of locating VMs and provide priorities between
them was developed by the authors, which lead them to the development of vLocality.
The vLocality is a robust solution for data locality in virtual environments and it is
implemented in Hadoop 1.2.1.
123
3.3.2 SLA based
Scheduling Hadoop tasks in virtual machines in the cloud demand resources of the
cloud, typically, users are aware of the deadline of when the job will be completed.
However, in the cloud computing environment, all machines compete for resources to
execute the jobs. These resources are controlled by batch queue systems, which may
not offer guarantee deadline during the task execution, only if the priority used for
resource reservation which is a restricted level of service.
Lim et al. [67] proposed a real technique to manage cloud resources using SLAs
for processing MapReduce end-to-end jobs. It is a novel constraint programming (CP)
based and ResourceManager (MRCP-RM) approach that is capable of matching and
scheduling different MapReduce jobs through SLAs on a system that is liable to open
stream of arrival jobs. Java programming is used for the implementation of MRCP-RM
and CP uses to IBM CPLEX to solve matching and scheduling problem.
3.3.3 Level
In map and reduce phases, the allocation and decisions are made based on the job and
task levels. For example, many tasks may not be scheduled instantaneously during
the execution. Unlike resource (node) level, job scheduling level can enable various
machines with different requirements to provide numbers of slots. Furthermore, the
completion time is assumed by the scheduling algorithm to be the same for each slot
[68]. There are two types of level in Hadoop scheduling, which are: job level and the
task level. The job level means the job scheduler has to choose a ready job while the
task level means the task scheduler has to select a ready task.
(1) Job level Scheduler determines which job run next from the job queue at the
job level. Normal jobs in a cluster are selected to run, may be put in the queue or
rejected to be resubmitted [69]. For instances, queued MapReduce jobs and that
are ready for execution and resources in a distributed environment managed and
monitored by the job trackers.
(2) Map/Reduce task level Scheduler decides which task of a given job will run
and in which node. Two types of tasks are created by The MapReduce—map and
reduce—that need to be scheduled in respective slots. The reason is, the resource
requirements for each task are different during the execution which complicates the
process of allowing the resources to be available for task-level schedulers to reduce
job execution time [70].
3.4 Optimization approach
In a distributed environment, the optimizer must decide, among others, where each
node of the plan will be executed [71]. Most of the scheduling problems mainly
fall under two approaches, single-objective, and multi-objective approach [72] as we
discuss in the next subsection (Fig. 5).
123
MTSD
Flex
Maestro
MRA++
Fair scheduler COSHH
Capacity scheduler Single-objectives Optimization Approaches Multi-objectives
Delay scheduling
MOMTH
FIFO
Quincy
MCP
SHadoop
ARIA
Fig. 5 Taxonomy of optimization approach in MapReduce
3.4.1 Single-objective scheduling approach
A single-objective optimization in MapReduce scheduling is presented by finding an

optimal solution that is corresponding to the minimization or maximization such as
response time, throughput, resources usage, deadline. Considering the nature of the
study, a single-objective offer decision makers with insights into the problem. How-
ever, such a solution may not be able to handle an alternative result that has conflicting
objectives [73]. For example, task scheduling based on deadline constraints is pro-
posed by Tang et al. [10] for the Hadoop platform. In order to measure the processing
capacity, the proposed task scheduling using node classification is implemented. The
classification is divided into many levels into heterogeneous clusters. Moreover, the
algorithm provides a scheduling mechanism to calculate the requirement of map and
reduce tasks to make the efficient decision. With such an approach, the deadline of
the jobs can be specified; it reduces the execution time and using a capacity level to
distributed data across multiple nodes. The result of the experiments demonstrates that
data locality and the average completion time have improved dramatically compared
to Hadoop default. Table 4 shows the comparison of scheduling algorithms that are
based on the single-objective optimization approach.
3.4.2 Multi-objective scheduling approach
Scheduling often requires considering several objectives in an optimization pro-

cess. For example, a trade-off between resource consumption and performance while
scheduling a speculative task. This optimization procedure manages two or more
objectives and is called multi-objective optimization.
COSHH [58] focused on the application and cluster levels to develop new Hadoop
scheduling system named “COSHH.” The system considers the heterogeneity of the
cluster when executing multiple tasks. The main idea is to offer an improvement in
terms of completion time. A typical Hadoop scheduler receives two main messages
123
Table 4 Comparison of scheduling algorithms based on a single objective
123
Algorithm Objectives Constraints Minimum schedule length Global optimality
FIFO [21] Execution time Deadline Yes No

Data dependency
Fair [30] Response time and data locality Data dependency Yes No
Capacity [31] Response time and data locality Data dependency Yes No
Delay [32] Locality and Fairness, resource usage Yes No
Fairness
LATE [36] Response time The cost to run speculative task Yes No
scheduling is expensive
Quincy [37] Min-cost flow Resource usage Yes No
MCP [74] Response time The cost of running speculative Yes No
task scheduling is expensive
SHadoop [39] Response time Throughput performance Yes No
MRA++ [40] Perform data-intensive Delay Yes No
computing in heterogeneous
environments
Maestro [41] Improving the locality execution – Yes No
of map task efficiency
ARIA [42] Completion time SLO Yes No
Flex [43] Response time, stretch, Deadline, fairness Yes No
makespan, and service level
agreements
MTSD [10] Data locality Deadline Yes No
from the Hadoop system: a message signaling a new job arrival from a user to store the
incoming job in an appropriate queue, and a heartbeat message from a free resource
by triggering the routing process to assign a job.
MOMTH [75] proposed Hadoop scheduling algorithm for many tasks based on
multi-constrained and multi-objective approaches to improve the process of the big
data. Thus, two objective functions are considered associated with users and resources
with constraints such as deadline and budget. With the purpose of evaluating the algo-
rithm in the scheduling load simulator, a collaboration platform known as MobiWay
is used for performance analysis to provide the ability to deal with a large amount of
sensor mobile and various applications The algorithm is compared with FIFO and fair
schedulers and it obtained similar performance for the same approach.
MORM [76] trying to solve the problem of data management in the large-
scale distributed environment. The author proposed a multi-objective optimization
approach called Multi-objective Optimized Replication Management based on offline
for replication management. The proposed solution focused on mean service time, file
unavailability, energy consumption, load variance, and latency. The artificial immune
algorithm is used to provide improvement for replication using a set of solution can-
didates via mutation, clone, and selection processes. The experimental result shows
the effectiveness of the proposed solution. Moreover, the approach outperforms the
existing default replication management in terms of load balancing. Table 5 shows
the comparison of scheduling algorithms based on a single objective. It shows the
objectives of the proposed scheduling algorithms with different constraints. These
scheduling algorithms are dealing with more than one objectives. Furthermore, it out-
lines the minimization length and the schedule of global optimality supported by the
optimization model.
Jiang et al. [77] introduced the makespan minimization technique for MapRe-
duce systems to process data from different servers at different speeds. Though, it is
the first time to study how to minimize MapReduce running on servers at different
speeds, which makes it difficult to develop a model and also analyzing the algorithm
for real MapReduce is complex. However, the makespan minimization scheduling
in this technique is classified into two, namely; offline and online scheduling. The
Table 5 Comparison of scheduling algorithms based on multi-objective
Algorithm Objectives Constraints Minimum Global

schedule length optimality
COSHH [58] Fairness, locality, and Heterogeneity Yes No

minimum share
satisfaction
MOMTH [75] Avoiding resource Deadline and budget Yes Yes
contention and having
an optimal workload of
the cluster
MORM [76] Optimized replication Load variance, energy Yes Yes
management consumption, and
Latency
123
offline scheduling is basically design for non-preemptive reduced jobs, while the online
scheduling can work for both preemptive non-preemptive reduced jobs.
In [78] multi-objective task scheduling is developed to improve energy-efficient in a
green data center. The purpose is to overcome the problem of energy in the data center
through green renewable energy. Improved multi-objective evolutionary algorithm
technique is used to develop the solution by applying generalized opposition-based
learning to find appropriate node computing, an efficient time scheduling strategy, and
clock frequency and supply voltage for the assigned jobs. Moreover, scheduling time
is utilized to determine when to run the task to guarantee that tasks wait in the queue
until the renewable energy is enough and the task can be completed in minimum time
needed. Experimental results show that the proposed solution confirm the superiority
and effectiveness.
3.4.3 Summary
Based on the recent research progress discussed in this subsection, we summarize the
objective functions, constraints, minimum schedule length and global optimality for
MapReduce scheduling optimization problem in Table 6.
3.5 Speculative execution
The jobs are divided across many virtual nodes in the cloud to be executed in parallel.
However, it is possible that a few nodes slow down the overall execution of a task. The
Table 6 Summary of MapReduce scheduling optimization approach
Optimization approach Algorithm Advantages Disadvantages
Single-objectives FIFO [21] Avoid complexity May is not able to handle an

alternative result that has
conflicting objectives
Fair [30] Only one optimal solution for
Capacity [31] a single run is detected in
Delay [32] Single-objective
LATE [36] optimization
Quincy [37]
MCP [38]
SHadoop [39]
MRA++ [40]
Maestro [41]
ARIA [42]
Flex [43]
MTSD [10]
Multi-objectives COSHH [58] Based on trade-offs of Identifying the true benefits
MOMTH [75] solutions for multiple and constraints of the
objectives to be optimized Hadoop scheduling is
difficult
The previous convergence to
local optima can be avoided
123
execution of the task may be slow due to various reasons such as software miscon-
figuration and hardware degradation. When the client submits the jobs to the master
node, the jobs will be broken down into tasks. These tasks will be executed by the
DataNode in which the total execution is dominated by the slowest DataNode in the
cluster. Thus, to overcome such a challenge, Hadoop is designed in such a way that, it
detects the slow running tasks and runs backup. For example, if one node has a slow
disk controller, then it may be reading its input at only 10% the speed of all the other
nodes.
LATE [36] proposed a new scheduling algorithm called Longest Approximate Time
to End (LATE) to improve the performance of the Hadoop scheduler in terms of
degradation in the heterogeneous environment. Late has built upon three important
concepts prioritizing task to speculate, selecting fast nodes to run on, and capping
speculative tasks to prevent thrashing. The idea is based on the assumptions that
homogeneous cluster makes progress linearly and has many speculative executions.
As the result, many nodes may slow down the overall performance of the Hadoop
cluster. The proposed algorithm has improved Hadoop performance by reducing the
response time and offer high robust to heterogeneity.
MCP [38] proposed novel speculative execution strategies. The main goal of the
scheduler is to choice struggle tasks accurately and then moves them to an available
node. This process is to guarantee fairness when the submitted jobs are divided into
multiple tasks for particular slots. Unlike Late, the proposed solution uses bandwidth
and rate progress in order to choose the slowest task. However, for the remaining time
and the progress speed of the tasks are calculated using averages weighted. Moreover,
the scheduler can decide on the backup of the task based on the cluster load using a
different method such as cost–benefit model, the backups of map tasks is optimized
locally, and for each works nodes that are slower is known by the process speed of
its map tasks to be executed on the node. The result of the experiment has shown that
MCP is 39% faster when running jobs in the cluster and also offer better throughput
that is up to 44% compared to default Hadoop.
Yang and Chen [79] provide new scheduling solution for speculative execution
named adaptive task allocation in order to improve the Hadoop framework. The algo-
rithm is designed to deal with the selection of the response time and backup in an
accurate manner to improve the Hadoop framework. The idea is to make the success
rate higher in terms of backup of the tasks and the response time. The experimental
results indicated that the proposed solution can decrease the average task throughput
by 33% and the task latency by 36% when to compare with the traditional Hadoop
Speculative.
Xu and Lau [80] consider various loading conditions in their design when dealing
with speculative execution. They have designed new schemes for parallel processing
cluster. These proposed schemes are based on two algorithms which are the straggler
detection algorithm and the smart cloning algorithm. Straggler detection algorithm
focuses on minimizing the overall resource consumption of the job. While Smart
cloning algorithm based on the maximization of the job utility.
You et al. [81] offered a new scheduling method called Load-Aware scheduler to
improve the performance of the MapReduce. The objective is to reduce the number of
speculative tasks in the heterogeneous cluster. The proposed method can improve the
123
overall performance of Hadoop clusters with dynamic loading. Experimental results

illustrate that the proposed scheduler can reduce the average response time by 20%.
Similarly, a novel method is introduced by [82] called “CREST (Combination Re-
Execution Scheduling Technology)” for speculative map tasks in order to achieve
the optimal running time while executing MapReduce jobs. The main purpose is to
perform re-execution of a combination of tasks on a set of computing nodes to decrease
the execution time.
Fu et al. [83] introduced Failure-Aware, Retrospective and Multiplicative (FARM)
scheme for efficient failure recovery in short jobs using MapReduce speculation (Yarn).
The Yarn and its speculation mechanisms were used by the authors as a case study
to demonstrate how speculation failure causes significant performance degradation
among MapReduce jobs, particularly for that job with shorter turnaround time. An
experimental result shows that, if a single node fails, it will affect the entire job by
slowing it down approximately 9.2 times. However, to overcome the issue of specula-
tion breakdown, a novel failure-aware speculation scheme called FARMS is proposed,
to handle node failure and to also improve the performance of Yarn to run well in an
unstable environment.
Brahmwar et al. [84] presented a scheme to aid scheduler in the identification
of nodes with potentials for executing stragglers. The scheme proposes in the study
uses the utilization of resource and network information of cluster nodes in finding the
most optimal node for scheduling the speculative copy of a slow task. The performance
evaluation of the proposed scheme has been done by series of experiments. From the
performance analysis, 27% improvement in terms of the overall execution time has
been observed over Hadoop Fair Scheduler (HFS).
In a distributed system operational status of processes monitored by failure detectors
device. Failures in MapReduce depend on the heartbeat to detected failure by enabling
dynamic or static timeout service. In [85] the algorithms developed high, medium
and low relax failure detector in a MapReduce environment depend on the timeout
adjustment. The high relax Flier detector kept the default MapReduce timeout, while
the medium and low dynamically changing the timeout according to the score of the
task and the failure detecting devices in the system.
3.5.1 Summary
In MapReduce, any available nodes that marked as straggler may poorly perform task
execution, which may pose some challenges to the overall execution time. However,
to overcome this challenges of straggler slowing the overall job execution, Hadoop
has implemented a speculative execution mechanism by running the copy of the task
on a different node. Thus, it avoids the misbehavior of some nodes to slow down the
whole job. Hadoop assumes that all the machines are homogeneous, but as a matter
of fact, in most cases, machines are not homogeneous especially in cloud computing
environment where the hardware could be in different generations and virtualized
data center as the uncontrollable several of virtualized resources. Table 7 depicts the
summary of speculative execution algorithms.
123
Table 7 Summary of speculative algorithms
Scheduling algorithm Solution Speculative execution Environment
FIFO [21] Backup Yes Homogeneous

Late scheduler [36] Priority based Yes Cloud
Quincy [37] Queue based Yes Heterogeneous
MCP [74] Yes Cloud
4 Performance analysis of scheduling in MapReduce
Since the birth of the Hadoop paradigm, the MapReduce programming model has been
one of its main components as highlighted in Sect. 1. The traditional implementation
of MapReduce has revealed high latencies during execution of Hadoop MapReduce
jobs. Basically, the submitted jobs are performed based on the steps structure in which
the data will be split, Map, shuffle, sort and then reduce [15]. This problem is exac-
erbated for more complexes processing involving statistical MapReduce jobs which
require time on the order of minutes, hours, or longer—even with fairly small data
volumes. In order to describe and compare the performance characteristics of the
MapReduce in the cluster, define a series of performance indicators is required. This
section mainly focuses on measuring the working capability of the MapReduce jobs,
including the measurement of throughput and execution time of each MapReduce job,
and the processing duration, CPU utilization of the node.
4.1 Summary
MapReduce scheduling is an emerging technology and has been explored from a vari-
ous angles by many researchers over the past few years. More specifically, optimization
and building new algorithms and framework in the homogeneous environment have
been the major focus on big data research. For example, performance issues like het-
erogeneity, hardware and software failure, and datasets have brought new challenges
when processing large amount of data in the distributed environment. Thus, tuning
MapReduce scheduling is one of the important factors to be considered to improve the
performance of the framework for a different environment. Table 8 shows the compar-
ison of different performance measurement. Gouasmi et al. [86] proposed federated
and geographically distributed MapReduce frameworks for federated cloud platforms.
Exact MapReduce scheduling algorithm is also proposed in the study. The federated
distributed MapReduce is responsible for reducing the cost of the job while maintaining
a deadline. Performance analysis suggested that the federated distributed MapReduce
has the potential of enhancing the utilization of resource while maintaining minimal
cost, the response time of a job and deadline. The Exact MapReduce scheduling algo-
rithm can serve as a point of reference in case of benchmarking. Zhao et al. [87]
motivated by the challenge of scheduling subtasks in a cloud computing environment
for minimizing transcoding convergence time. As such, the authors propose the par-
allelizing of video transcoding on the heterogeneous MapReduce clusters based on
prediction and locality-aware task scheduling technique (PLATT). The decoding and
123
Table 8 Comparison of different performance measurement
Algorithms Throughput Resource CPU time Total response Total response

utilization spent time by Map time by reducing
FIFO [21] High Low High Low High

Fair scheduler [30] Medium Medium Medium High High
Capacity scheduler [31] Medium High Medium High High
Resource aware Low High High Medium High
encoding video technologies analyzed. Subsequently, the segment transcoding com-

plexity is predicted to determine the next scheduling. The subtask with data locality
was scheduled. The scheduling is formulated as job scheduling problem. Experimental
results indicated that the PLATT is able to balance load in the clusters and reduces the
video transcoding time. Comparison with state-of-the-methods: PLTS-DVFS, PLTS-
Length, PLTS-Size, and PLTS-Cheng prove the effectiveness and efficiency of the
PLATT.
5 Future research directions and recommendations
While existing research on MapReduce scheduling has shown great improvement,

numerous challenging problems yet to be solved. Hadoop composed of one master
node and several data nodes dependents on the size of the cluster. Users can make their
choices according to the availability of the resources and the nodes. This research offers
insights into the future research opportunities as follows:
(1) QoS-Based Scheduling In the last few years, cloud computing has become a
fully service-oriented paradigm, which effectively allows users to consume based
on their QoS requirement [88]. Generally, in order for scheduling to make an effec-
tive decision, it assumed that workflow tasks dependencies are clearly defined [89].
Thus, it is significant for the scheduler to monitor task execution based on the SLA
including resource capability and service time with service providers. Moreover,
there is a lack of algorithms that consider the trade-off between multiple quality
requirements [14]. MapReduce scheduling algorithm adapts an alternative service
and execution time demanded an SLA as regards to its currently accepted set of
SLAs and expected the return of unscheduled tasks. Moreover, most of the previous
scheduling is based on focuses on deterministic Directed Acyclic Graphs (DAGs).
Moreover, using Hadoop on the cloud require a good QoS-based workflow execu-
tion. It is important that new benchmarks put in place to ensure QoS-based workflow
scheduling algorithms by comparing and evaluating different workflow applications
that are suitable for the Hadoop framework.
(2) Multi-dimensional resource scheduling Traditional scheduling systems based
on a single-resource optimization, like processors, fail to provide near-optimal
solutions [90]. A Multi-dimensional resource may be considered using a series
of resources such as CPU, memory, network bandwidth. For example, the slot
management scheme [91] is proposed in order to enable dynamic slot configu-
123
ration in Hadoop. The idea behind the slot management scheme is to improve
resource utilization and reduce the makespan of multiple jobs. Thus, there is a
lack of related metrics for scheduling algorithms considering multi-dimensional
resource. For future research in, the main idea of the scheduling algorithm with
a multi-dimensional resource such as CPU, memory, and network bandwidth. is
to obtain a minimal execution schedule through efficient management of available
cloud resources [92, 93]. Moreover, the scheduling algorithms should be able to
tasks into account the different resource requirements of different tasks and shown
to obtain a minimal execution schedule through efficient management of available
cloud computing resources.
(3) Event-based scheduling Previously, the offline problem has been the main focus
in scheduling research in order to minimize execution time for a single workflow
with known task runtimes [94]. Heterogeneity brings a new challenging issue to
the Hadoop scheduling the low support for complex requirements in current queue-
based scheduling algorithms and arising problems of the schedule-based solution
when applied in a dynamic environment with uncertainties. Optimization of these
scheduling algorithms for Hadoop demanded to schedule many jobs in the queue.
Thus, it is essential to overcome the traditional method and develop automatic
synchronization activity execution for enabling workflows to exchange data with
other workflows or other applications [95, 96].
(4) Energy consumption and efficiency models for MapReduce jobs: Although a
new power-aware MapReduce application model has introduced in [97] to be used
for power-aware computing with consideration of users’ requirements. There is a
need for detailed energy efficiency model for MapReduce environments to predict
the energy consumed for mix workload scenarios [98]. It should also consider the
background HDFS activities carried out for availability checks. It should be able
to incorporate the idle nodes energy as well. The performance of the MapReduce
framework can possibly use for prediction of the map and reduce task timings
depending on the data volume, their distribution, underlying hardware, etc. Energy
and performance models can be combined to evaluate various scheduling algorithms
for predicting the energy consumptions and thus decide which one maximizes the
performance and energy efficiency.
6 Conclusion
MapReduce has offered a new alternative way to efficiently mining petabytes of

unstructured information across multi-machines with lower cost commodity hard-
ware. As an important technology, MapReduce has increasing drawn an attention to
the necessity of big data processing in recent years. Meanwhile, MapReduce schedul-
ing is an emerging technology and has been explored from a various angles by many
researchers over the past few years. More specifically, optimization, building new algo-
rithms and framework in the homogeneous environment have been the major focused
in big data research. For example, performance issues like heterogeneity, hardware
and software failure, and datasets have brought new challenges when processing large
123
amount of data in the distributed environment. Thus, tuning MapReduce scheduling

is one of the important factors to be considered to improve the performance of the
framework for a different environment. This paper provided an overview of schedul-
ing in MapReduce and discussed the current scheduling algorithms. A comparison
of Hadoop, Mesos, Corona frameworks is provided. It outlines the type of resource
scheduling mechanism used in each framework. We provided analysis on MapReduce
scheduling in two aspects: taxonomy and performance evaluation. The scheduling
algorithms are based on strategies, resources, workload, optimization approaches,
requirement, and speculative execution. The main objective is to assist researchers to
understand the current research improvement and to address future research directions
in the area of MapReduce scheduling.
Acknowledgements This paper is financially supported by University Malaya Research Grant Programme
(Equitable Society) under Grant RP032B-16SBS.
Appendix A
See Table 9.
Table 9 Modifications induced by existing scheduling approach to MapReduce
System Implemented in Hadoop systems
Yarn [22] Yes, build on top of Hadoop to provide resource management

Mesos [23] No, uses for resource management and scheduling of the tasks by
providing applications like Hadoop with API
Corona [24] No, separates cluster resource management from job coordination
FIFO [21] Yes, integrated into Hadoop
Fair scheduler [30] Yes, integrated into Hadoop
Capacity scheduler [31] Yes, integrated into Hadoop
Late [36] No, add-on algorithm uses estimated finish times to speculatively
execute the tasks that hurt the response time the most
Quincy [37] No, uses for scheduling multiple jobs on distributed environment
and more flexible with fine-grain resource sharing
MCP [38] No, add-on algorithm improved to select suitable worker nodes for
backup tasks
SHadoop [39] Yes, integrated into the Intel Distributed Hadoop (IDH)
MRA++ [40] No, during data distribution, task scheduling, and job control the
heterogeneity of nodes is considered
Maestro [41] No, extension based on replica-aware Map Scheduling
ARIA [42] No, implemented on top of Hadoop to determine job ordering and a
number of resources to allocate for meeting the job deadlines
Flex [43] No, an add-on module
MTSD [10] No, extensional MapReduce Task Scheduling Algorithm for
Deadline constraints in Hadoop platform
COSHH [58] No, add-on algorithm considers heterogeneity
MOMTH [75] No, a tool integrated into Hadoop
MORM [76] Different system
123
References
1. Chen M et al (2014) Big data: a survey. Mob Netw Appl 19(2):171–209
2. Maass W et al (2017) Big data and theory. In: Schintler LA, McNeely CL (eds) Encyclopedia of big
data, Springer International Publishing, Cham, pp 1–5
3. Wang Y et al (2018) Big data analytics: understanding its capabilities and potential benefits for health-
care organizations. Technol Forecast Soc Change 126:3–13
4. Tahmassebi A et al (2018) Deep learning in medical imaging: fMRI big data analysis via convolutional
neural networks. In: Proceedings of the Practice and Experience on Advanced Research Computing.
ACM
5. Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM
51(1):107–113
6. Lee K-H et al (2012) Parallel data processing with MapReduce: a survey. AcM sIGMoD Rec
40(4):11–20
7. Chang H et al (2011) Scheduling in MapReduce-like systems for fast completion time. In: 2011
Proceedings IEEE INFOCOM. IEEE
8. Yoo D, Sim KM (2011) A comparative review of job scheduling for MapReduce. In: 2011 IEEE
International Conference on Cloud Computing and Intelligence Systems (CCIS). Citeseer
9. Althebyan Q et al (2017) A scalable MapReduce tasks scheduling: a threading-based approach. Int J
Comput Sci Eng 14(1):44–54
10. Tang Z et al (2012) MTSD: a task scheduling algorithm for MapReduce base on deadline constraints.
In: 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & Ph.D.
Forum (IPDPSW). IEEE
11. Jayasena K, Li L, Xie Q (2017) Multi-modal multimedia big data analyzing architecture and resource
allocation on cloud platform. Neurocomputing 253:135
12. Page AJ, Naughton TJ (2005) Framework for task scheduling in heterogeneous distributed computing
using genetic algorithms. Artif Intell Rev 24(3–4):415–429
13. Rao BT, Reddy L (2012) Survey on improved scheduling in Hadoop MapReduce in cloud environments.
arXiv preprint arXiv:1207.0780
14. Tiwari N et al (2015) Classification framework of MapReduce scheduling algorithms. ACM Comput
Surv (CSUR) 47(3):49
15. Doulkeridis C, Nørvåg K (2014) A survey of large-scale analytical query processing in MapReduce.
VLDB J 23(3):355–380
16. Arora S, Goel DM (2014) Survey paper on scheduling in Hadoop. Int J Adv Res Comput Sci Softw
Eng 4(5):4886
17. Chen C-H, Lin J-W, Kuo S-Y (2018) MapReduce scheduling for deadline-constrained jobs in hetero-
geneous cloud computing systems. IEEE Trans Cloud Comput 6(1):127–140
18. Nagarajan V et al. (2018) Malleable scheduling for flows of jobs and applications to MapReduce. J
Sched 752:1–19
19. Duan N et al (2018) Scheduling MapReduce tasks based on estimated workload distribution. Google
Patents
20. Tang Y et al (2018) OEHadoop: accelerate Hadoop applications by co-designing Hadoop with data
center network. IEEE Access 6:25849–25860
21. Hadoop A (2011) Apache Hadoop. https://hadoop.apache.org/. Accessed 3 May 2017
22. Vavilapalli VK et al (2013) Apache Hadoop YARN: yet another resource negotiator. In: Proceedings
of the 4th Annual Symposium on Cloud Computing. ACM
23. Hindman B et al (2011) Mesos: a platform for fine-grained resource sharing in the data center. In:
NSDI
24. Facebook (2012) Facebook engineering. Under the hood: scheduling MapReduce jobs more efficiently
with Corona. 2012 [cited 2015 5 March]. https://www.facebook.com/notes/facebook-engineering/
under-the-hood-scheduling-mapreduce-jobs-more-efficiently-with-corona/10151142560538920
25. Scott J (2015) A tale of two clusters: Mesos and YARN. [cited 2016 1/6/2016]. http://radar.oreilly.
com/2015/02/a-tale-of-two-clusters-mesos-and-yarn.html
26. Shabeera T, Kumar SM, Chandran P (2016) Curtailing job completion time in MapReduce clouds
through improved Virtual Machine allocation. Comput Electr Eng 58:190–202
27. Pulgar-Rubio F et al (2017) MEFASD-BD: multi-objective evolutionary fuzzy algorithm for subgroup
discovery in big data environments-a MapReduce solution. Knowl-Based Syst 117:70–78
123
28. Casavant TL, Kuhl JG (1988) A taxonomy of scheduling in general-purpose distributed computing
systems. IEEE Trans Softw Eng 14(2):141–154
29. Gao Y, Rong H, Huang JZ (2005) Adaptive grid job scheduling with genetic algorithms. Future Gener
Comput Syst 21(1):151–161
30. Hadoop A (2009) Fair scheduler. https://hadoop.apache.org/docs/stable1/fair_scheduler.html.
Accessed 13 June 2017
31. Hadoop A Capacity scheduler guide. https://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html.
Accessed 13 June 2017
32. Zaharia M et al (2010) Delay scheduling: a simple technique for achieving locality and fairness in
cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems. ACM
33. Tan J, Meng X, Zhang L (2012) Delay tails in MapReduce scheduling. ACM SIGMETRICS Perform
Eval Rev 40(1):5–16
34. Hadoop A Apache Hadoop. https://hadoop.apache.org/. Accessed 3 May 2017
35. Casas I et al (2016) GA-ETI: an enhanced genetic algorithm for the scheduling of scientific workflows
in cloud environments. J Comput Sci 26:318–331
36. Zaharia M et al (2008) Improving MapReduce performance in heterogeneous environments. In: OSDI
37. Isard M et al (2009) Quincy: fair scheduling for distributed computing clusters. In: Proceedings of the
ACM SIGOPS 22nd Symposium on Operating Systems Principles. ACM
38. Qi C, Cheng L, Zhen X (2014) Improving MapReduce performance using smart speculative execution
strategy. IEEE Trans Comput 63(4):954–967
39. Gu R et al (2014) SHadoop: improving MapReduce performance by optimizing job execution mech-
anism in Hadoop clusters. J Parallel Distrib Comput 74(3):2166–2179
40. Anjos JC et al (2015) MRA++: scheduling and data placement on MapReduce for heterogeneous
environments. Future Gener Comput Syst 42:22–35
41. Ibrahim S et al (2012) Maestro: Replica-aware map scheduling for MapReduce. In: 2012 12th
IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). IEEE
42. Verma A, Cherkasova L, Campbell RH (2011) ARIA: automatic resource inference and allocation for
MapReduce environments. In: Proceedings of the 8th ACM International Conference on Autonomic
Computing. ACM
43. Wolf J et al (2010) Flex: a slot allocation scheduling optimizer for MapReduce workloads. In: Mid-
dleware 2010. Springer, pp 1–20
44. Polo J et al (2010) Performance management of accelerated MapReduce workloads in heterogeneous
clusters. In: 2010 39th International Conference on Parallel Processing (ICPP). IEEE
45. Lopes R, Menascé D (2015) A taxonomy of job scheduling on distributed computing systems. http://
cs.gmu.edu. Accessed 3 Sept 2017
46. Ahmad F et al (2012) Tarazu: optimizing MapReduce on heterogeneous clusters. In: ACM SIGARCH
Computer Architecture News. ACM
47. Krish K, Anwar A, Butt AR (2014) [phi] Sched: a heterogeneity-aware Hadoop workflow scheduler.
In: 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and
Telecommunication Systems (MASCOTS). IEEE
48. Dong F, Akl SG (2007) PFAS: a resource-performance-fluctuation-aware workflow scheduling algo-
rithm for grid computing. In: IEEE International Parallel and Distributed Processing Symposium.
IPDPS 2007. IEEE
49. Cheng D, Rao J, Guo Y, Jiang C, Zhou X (2017) Improving performance of heterogeneous mapreduce
clusters with adaptive task tuning. IEEE Trans Parallel Distrib Syst 28(3):774–786
50. Murthy AC et al (2011) Architecture of next generation Apache Hadoop MapReduce framework.
Technical report, Apache Hadoop
51. Ghit B et al (2014) Balanced resource allocations across multiple dynamic MapReduce clusters. In:
ACM SIGMETRICS
52. Barham P et al (2003) Xen and the art of virtualization. ACM SIGOPS Oper Syst Rev 37(5):164–177
53. Chen F, Kodialam M, Lakshman T (2012) Joint scheduling of processing and shuffle phases in MapRe-
duce systems. In: Proceedings IEEE INFOCOM. IEEE
54. Polo J et al (2011) Resource-aware adaptive scheduling for MapReduce clusters. In: Middleware 2011.
Springer, pp 187–207
55. Sousa E et al (2014) Resource-aware computer vision application on heterogeneous multi-tile archi-
tecture. In: Proceedings of the Hardware and Software Demo at the University Booth at Design,
Automation and Test in Europe (DATE), Dresden
123
56. Yong M, Garegrat N, Mohan S (2009) Towards a resource aware scheduler in Hadoop. In: Proceedings
of the 2009 IEEE International Conference on Web Services, Los Angeles, CA, USA
57. Guo Z et al (2012) Improving resource utilization in MapReduce. In: 2012 IEEE International Con-
ference on Cluster Computing (CLUSTER). IEEE
58. Rasooli A, Down DG (2014) COSHH: a classification and optimization based scheduler for heteroge-
neous Hadoop systems. Future Gener Comput Syst 36:1–15
59. Guo Z, Fox G, Zhou M (2012) Investigation of data locality in MapReduce. In: Proceedings of the
2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID
2012). IEEE Computer Society
60. Park J et al (2012) Locality-aware dynamic VM reconfiguration on MapReduce clouds. In: Proceedings
of the 21st International Symposium on High-Performance Parallel and Distributed Computing. ACM
61. Li J-J et al (2011) Survey of MapReduce parallel programming model. Dianzi Xuebao (Acta Electron
Sin) 39(11):2635–2642
62. He C, Lu Y, Swanson D (2011) Matchmaking: a new MapReduce scheduling technique. In: 2011 IEEE
Third International Conference on Cloud Computing Technology and Science (CloudCom). IEEE
63. Abad CL, Lu Y, Campbell RH (2011) DARE: adaptive data replication for efficient cluster scheduling.
In: 2011 IEEE International Conference on Cluster Computing (CLUSTER). IEEE
64. Zhang X et al (2011) Improving data locality of MapReduce by scheduling in homogeneous computing
environments. In: 2011 IEEE 9th International Symposium on Parallel and Distributed Processing with
Applications (ISPA). IEEE
65. Jin J et al (2011) Bar: an efficient data locality driven task scheduling algorithm for cloud computing.
In: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid
Computing. IEEE Computer Society
66. Wang W, Zhu K, Ying L, Tan J, Zhang L (2016) Maptask scheduling in mapreduce with data locality:
Throughput and heavy-traffic optimality. IEEE/ACM Trans Networking (TON) 24(1):190–203
67. Lim N, Majumdar S, Ashwood-Smith P (2014) Engineering resource management middleware for
optimizing the performance of clouds processing MapReduce jobs with deadlines. In: Proceedings of
the 5th ACM/SPEC International Conference on Performance Engineering. ACM
68. Sandholm T, Lai K (2010) Dynamic proportional share scheduling in hadoop. In: Workshop on Job
Scheduling Strategies for Parallel Processing, Springer, Berlin, Heidelberg, pp 110–131
69. Nanduri R et al (2011) Job aware scheduling algorithm for MapReduce framework. In: 2011 IEEE
Third International Conference on Cloud Computing Technology and Science (CloudCom). IEEE
70. Zhang Q et al (2015) PRISM: fine-grained resource-aware scheduling for MapReduce. IEEE Trans
Cloud Comput 1:1
71. Kllapi H et al (2011) Schedule optimization for data processing flows on the cloud. In: Proceedings of
the 2011 ACM SIGMOD International Conference on Management of data. ACM
72. Ponnambalam S, Jawahar N, Chandrasekaran S (2009) Discrete particle swarm optimization algorithm
for flowshop scheduling. INTECH Open Access Publisher
73. Savic D (2002) Single-objective vs. multiobjective optimisation for integrated decision support. Integr
Assess Decision Support 1:7–12
74. Chen Q, Liu C, Xiao Z (2013) Improving MapReduce performance using smart speculative execution
strategy. Parallel Distrib Syst 24:1107
75. Nita M-C et al (2015) MOMTH: multi-objective scheduling algorithm of many tasks in Hadoop. Clust
Comput 18:1–14
76. Long S-Q, Zhao Y-L, Chen W (2014) MORM: a multi-objective optimized replication management
strategy for cloud storage cluster. J Syst Archit 60(2):234–244
77. Jiang Y et al (2017) Makespan minimization for MapReduce systems with different servers. Future
Gener Comput Syst 67:13–21
78. Lei H et al (2016) A multi-objective co-evolutionary algorithm for energy-efficient scheduling on a
green data center. Comput Oper Res 75:103–117
79. Yang S-J, Chen Y-R (2015) Design adaptive task allocation scheduler to improve MapReduce perfor-
mance in heterogeneous clouds. J Netw Comput Appl 57:61–70
80. Xu H, Lau WC (2014) Optimization for speculative execution of multiple jobs in a MapReduce-like
cluster. arXiv preprint arXiv:1406.0609
81. You H-H, Yang C-C, Huang J-L (2011) A load-aware scheduler for MapReduce framework in hetero-
geneous cloud environments. In: Proceedings of the 2011 ACM Symposium on Applied Computing.
ACM
123
82. Lei L, Wo T, Hu C (2011) CREST: towards fast speculation of straggler tasks in MapReduce. In: 2011
IEEE 8th International Conference on e-Business Engineering (ICEBE). IEEE
83. Fu H et al (2017) FARMS: efficient MapReduce speculation for failure recovery in short jobs. Parallel
Comput 61:68–82
84. Brahmwar M, Kumar M, Sikka G (2016) Tolhit—a scheduling algorithm for Hadoop cluster. Proc
Comput Sci 89:203–208
85. Memishi B, Pérez MS, Antoniu G (2017) Failure detector abstractions for MapReduce-based systems.
Inf Sci 379:112–127
86. Gouasmi T et al (2018) Exact and heuristic MapReduce scheduling algorithms for cloud federation.
Comput Electr Eng 69:274
87. Zhao H et al (2018) Prediction-based and locality-aware task scheduling for parallelizing video
transcoding over heterogeneous MapReduce cluster. IEEE Trans Circuits Syst Video Technol
28(4):1009–1020
88. Singh S, Chana I (2015) QoS-aware autonomic resource management in cloud computing: a systematic
review. ACM Comput Surv (CSUR) 48(3):42
89. Yu J (2007) QoS-based scheduling of workflows on global grids
90. Sheikhalishahi M et al (2016) A multi-dimensional job scheduling. Future Gener Comput Syst
54:123–131
91. Yao Y et al (2015) Self-adjusting slot configurations for homogeneous and heterogeneous Hadoop
clusters. IEEE Trans Cloud Comput 5:344
92. Khoo BB et al (2007) A multi-dimensional scheduling scheme in a Grid computing environment. J
Parallel Distrib Comput 67(6):659–673
93. Yao Z, Papapanagiotou I, Callaway RD (2015) Multi-dimensional scheduling in cloud storage systems.
In: International Communications Conference (ICC)
94. Dong X, Wang Y, Liao H (2011) Scheduling mixed real-time and non-real-time applications in MapRe-
duce environment. In: 2011 IEEE 17th International Conference on Parallel and Distributed Systems
(ICPADS). IEEE
95. Casati F, Shan M-C (2007) Event-based scheduling method and system for workflow activities. Google
Patents
96. Ilyushkin A, Ghit B, Epema D (2015) Scheduling workloads of workflows with unknown task runtimes.
In: 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid).
IEEE
97. Li Y, Zhang H, Kim KH (2011) A power-aware scheduling of MapReduce applications in the cloud.
In: 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing
(DASC). IEEE
98. Goiri Í et al (2012) GreenHadoop: leveraging green energy in data-processing frameworks. In: Pro-
ceedings of the 7th ACM European Conference on Computer Systems. ACM
Aﬃliations
Ibrahim Abaker Targio Hashem1,2 · Nor Badrul Anuar2 · Mohsen Marjani1 ·

Ejaz Ahmed3 · Haruna Chiroma4 · Ahmad Firdaus5 ·
Muhamad Taufik Abdullah6 · Faiz Alotaibi6 ·
Waleed Kamaleldin Mahmoud Ali2 · Ibrar Yaqoob2 · Abdullah Gani1
Nor Badrul Anuar

badrul@um.edu.my
Mohsen Marjani
mohsen.Marjani@taylors.edu.my
Ejaz Ahmed
ejazahmed@ieee.org
Haruna Chiroma
freedonchi@yahoo.com
123
Ahmad Firdaus
firdausza@ump.edu.my
Muhamad Taufik Abdullah
mta@upm.edu.my
Faiz Alotaibi
faiz.eid@hotmail.com
Waleed Kamaleldin Mahmoud Ali
waleed.k.ali@gmail.com
Ibrar Yaqoob
ibraryaqoob@ieee.org
Abdullah Gani
abdullah.gani@taylors.edu.my
1 School of Computing and Information Technology, Taylor’s University, 47500 Subang Jaya,
Malaysia
2 Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur,
Malaysia
3 Centre for Mobile Cloud Computing Research, University of Malaya, Kuala Lumpur, Malaysia
4 Department of Computer Science, Federal College of Education (Technical), Gombe, Nigeria
5 Faculty of Computer Systems and Software Engineering, Universiti Malaysia
Pahang, 26300 Gambang, Kuantan, Pahang, Malaysia
6 Faculty of Computer Science and Information Technology, Universiti Putra Malaysia,
43400 Serdang, Selangor, Malaysia
123
View publication stats

Hashem2018 Article MapReduceSchedulingAlgorithmsA

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hashem2018 Article MapReduceSchedulingAlgorithmsA

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

MapReduce scheduling algorithms: a review

Article in The Journal of Supercomputing · December 2018

Ibrahim Abaker Targio Hashem Nor Badrul Anuar

SEE PROFILE SEE PROFILE

Multi-objective Software Defined Network Measurement View project

The user has requested enhancement of the downloaded file.

MapReduce scheduling algorithms: a review

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Keywords Big data · Hadoop · MapReduce · Cloud computing · Scheduling

B Ibrahim Abaker Targio Hashem

workload, and speculative execution. Subsequently, with each category, we highlight

Before we begin our discussion on the taxonomy of MapReduce scheduling, we briefly

Table 1 A comparison of resources scheduling frameworks

Features MapReduce Yarn [22] Mesos [23] Corona [24]

Resources Request based Request based Offer based Push based

2.1 Resource scheduling frameworks

Fig. 1 Illustration of Yarn architecture

2.1.2 Apache Mesos

Apache Mesos [23] is a cluster manager, originally developed at Berkeley, University

Corona is an extension of the MapReduce framework; it provides high scalability and

Fig. 2 Illustration of Apache Mesos architecture

A comparison of the discussed frameworks is shown in Table 1. It outlines the type

3 Taxonomy of MapReduce scheduling

(a) MapReduce framework based on scheduling strategy: adaptive and non-

Figure 3 provides a classification of MapReduce scheduling according to the used

3.1 Scheduling strategy

3.1.1 Adaptive scheduling strategy

Scheduling strategy Resource Requirement Optimization approach Workload Speculative execution

Level Real time

Fig. 3 Taxonomy of scheduling in MapReduce

Capacity scheduler Adaptive Scheduling Strategy FIFO

Fig. 4 Taxonomy of scheduling strategy in MapReduce

Many researchers have attempted to propose adaptive scheduling strategies. These

3.1.2 Non-adaptive scheduling strategy

According to the scheduling strategy discussion on the current research improvement,

Set of CPUs/vCPUs, memory, I/O, networking connectivity resources that can be

Generally, resource heterogeneity in the cluster may compress of either heteroge-

References Algorithm Objectives Implementation approach Limitation

Pool weights Resource and job mismatch

References Algorithm Objectives Implementation approach Limitation

Resource sharing in Hadoop MapReduce is shown to be challenging tasks, more impor-

3.2.3 Resource aware

In MapReduce with multi-job workloads, resource aware is important for enhancing

Table 3 A summary of the resources in MapReduce scheduling

Algorithm Mechanism Heterogeneity Sharing Resource aware

FIFO [21] Queuing Homogeneous No Static

3.3.1 Data locality

3.3.2 SLA based

3.4 Optimization approach

Fair scheduler COSHH

Capacity scheduler Single-objectives Optimization Approaches Multi-objectives

Fig. 5 Taxonomy of optimization approach in MapReduce

3.4.1 Single-objective scheduling approach

A single-objective optimization in MapReduce scheduling is presented by finding an

3.4.2 Multi-objective scheduling approach

Scheduling often requires considering several objectives in an optimization pro-

FIFO [21] Execution time Deadline Yes No

Table 5 Comparison of scheduling algorithms based on multi-objective

Algorithm Objectives Constraints Minimum Global

COSHH [58] Fairness, locality, and Heterogeneity Yes No

3.5 Speculative execution

Table 6 Summary of MapReduce scheduling optimization approach