Big Data Huawei Course

MAP REDUCE
YARN
AVISO
Este documento foi gerado a partir de um material de estudo
da Huawei. Considere as informações nesse documento
como material de apoio.

Centro de Inovação EDGE - Big Data Course

Table of Contents
1. MapReduce Framework ..............................................................................1

2. YARN – Yet Another Resource Negotiator ..................................................1

3. MapReduce and YARN Working Process ...................................................1

3.1. MapReduce Working Process ............................................................ 2

3.2. YARN Architecture.............................................................................. 2

3.3. YARN Working Process...................................................................... 3

3.4. MapReduce and YARN Interaction..................................................... 4

3.5. Shuffle Mechanism ............................................................................. 5

4. YARN Features............................................................................................5

4.1. YARN High Availability (HA) Solution ................................................. 5

4.2. YARN AppMaster Fault Tolerant Mechanism ..................................... 5

5. Resource Management and Task Scheduling of YARN ..............................6

5.1. Resource Management ...................................................................... 6

5.2. Resource Allocation Model ................................................................. 6

5.3. Capacity Scheduler ............................................................................ 6

5.4. Task Selection by Capacity Scheduler ............................................... 7

5.5. Resource limitation ............................................................................. 7

6. Enhanced features of YARN in FusionInsight HD .......................................8

Centro de Inovação EDGE - Big Data Course

 Technical Principles of MapReduce and YARN
1. MapReduce Framework

� It is a processing technique and a programming model for distributed parallel

computing of a massive data set
� Introduced by Google in 2004
� Advantages
o Easy to program - programmers only need to describe what to do and
the execution framework will do the job accordingly
o Outstanding scalability - it is easy to scale data processing over multiple
computing nodes. We can just simply add nodes to improve cluster ca-
pabilities
o High fault tolerance - if one of the nodes in the cluster fails, tasks running
on this node (the faulty node) can be performed on another node using
computer migration or data migration to ensure successful execution of
tasks
� HDFS is the storage infrastructure. Thus, input and output files are read from/
written to HDFS
� Process large files, normally greater than 1TB
� Processing is broken into multiple smaller tasks (Map and Reduce tasks) and
distributed over the cluster

2. YARN – Yet Another Resource Negotiator

� It is the Hadoop cluster resource manager, providing resource management and

scheduling for upper-layer applications
o Computing engines like Hive, MapReduce, Spark, need to apply for re-
sources from YARN
o No need to maintain one resource manager for each computing engine
o Only need to maintain YARN, which reduces the operation and mainte-
nance cost
o Cluster shares the resources so that they can be highly used

3. MapReduce and YARN Working Process

Centro de Inovação EDGE - Big Data Course 1

3.1. MapReduce Working Process

� Map and Reduce tasks are user defined. Combine and Partition can be defined
by user too. Combine is optional
� Each Map task takes one split of the input file and generates an output as key-
value pairs
o WordCount example:
�<hello, 1>

�<world, 1>

�<bye, 1>

�<world, 1>

� Outputs from all Map tasks are merged, combined, and a MapOutputFile (MOF)
is generated. Then, it is sorted and copied to input of Reduce tasks
� Input to Reduce tasks is a file with pairs with key-<list of values>
o WordCount example:
�<bye, 1 1 1>

�<hadoop, 2 2>

�<hello, 1 1 1>

�<world, 2>

� Reduce tasks takes MOF and generates output of all MapReduce processing

3.2. YARN Architecture

� Client
o Submit jobs to Resource Manager
� Resource Manager
o Manage the use of resources across the cluster
o Only one (active) Resource Manager in a Hadoop cluster
� Node Manager
o Run on every node in the cluster
o Launch and monitor Containers
o Report node status to Resource Manager periodically
� ApplicationMaster
o Negotiate resources with Resource Manager
o Work with NodeManager to execute and monitor tasks

Centro de Inovação EDGE - Big Data Course 2

 � Container
o Abstraction of YARN resources (memory and CPU)
o It executes applications with a set of resource limitations
o Report status of tasks to ApplicationMaster

3.3. YARN Working Process

1. Client submits a job to Resource Manager

2. Resource Manager receives the job and allocates the first Container for the ap-
plication program and asks NodeManager to start Application Master in the
Container
3. Application Master registers with Resource Manager, so users can directly view
the operating status of the application programs
4. Application Master applies for and obtains resources from Resource Manager
5. Application Master asks NodeManager to start the tasks
6. NodeManager starts the tasks
7. Each task uses remote procedure call (RPC) to report its status and progress to
Application Master. So that Application Master can restart the task if one fails
� Users can obtain the operating status of the application program from Applica-
tion Master at any time
� After application program ends, Application Master cancels its registration with
Resource Manager and closes itself

Centro de Inovação EDGE - Big Data Course 3

3.4. MapReduce and YARN Interaction

1. Files to be processed must be stored in HDFS

2. MapReduce submits request to Resource Manager
3. File is split.
o By default, the number of splits is determined by the number of blocks
that the file is already stored in HDFS
o Client applications can also redefine the mapping relation between
blocks and splits
o The number of Map tasks is defined by the number of splits, since each
Map task processes one split
4. Jobs are submitted to Resource Manager
5. Map tasks start
o Each map task takes a set of data and converts it into another set of data,
where individual elements are broken down into key-value pairs
o Temporary output (key-value pairs) are written to buffer of 100MB by de-
fault
o When buffer is 80% full, data is written to local disks
6. Data is divided into partitions
o The number of partitions is determined by the number of Reduce tasks
o The number of Reduce tasks is determined by the client

Centro de Inovação EDGE - Big Data Course 4

 o Data with the same key-value are sent to the same Reduce tasks
7. Data is sorted and combined
o Combine operation is optional
8. Temporary files (spill files) are merged and MapOutputFile (MOF) is created
9. When the output progress of MOF reaches 3%, the Reduce tasks are started and
copy MOF files from each Map task
10. Multiple MapOutputFiles are merged into one
11. Reduce function is invoked for each key
12. The output of Reduce is written directly to the output file system, typically HDFS

3.5. Shuffle Mechanism

� It is the data transfer process between the Map task and Reduce task
� It involves that Reduce tasks copy MOF files from the Map tasks and sort and
merge MOF files

4. YARN Features

4.1. YARN High Availability (HA) Solution

� Active/Standby architecture
� Failover can be triggered automatically or manually to switch between active
and standby state
o Manually - administrator must run a command to manually switch one of
the Resource Manager nodes to the active state
o Automatically - when the active Resource Manager goes down or be-
comes unresponsive, another Resource Manager is automatically elected
through Zookeeper to be the active node; then, services are reestab-
lished

4.2. YARN AppMaster Fault Tolerant Mechanism

� If an Application Master goes down, its Containers will be closed, including the
ones where tasks are running
� Resource manager will start a new Application Master on another computing
node

Centro de Inovação EDGE - Big Data Course 5

 � YARN supports the feature of preserving the Container status when starting a
new Application Master, so that the tasks in these Containers can run continu-
ously without any failure

5. Resource Management and Task Scheduling of YARN

5.1. Resource Management

� Allocation of two types of resources: memory and CPU

� Settings of NodeManager
o (yarn.nodemanager.resource.memory-mb) - indicates the physical mem-
ory size of the container running on NodeManager expressed in
megabytes
o (yarn.nodemanager.vmem-pmem-ratio) - indicates the ratio of the maxi-
mum available virtual memory. It is specified based on physical memory
o (yarn.nodemanager.resource.cpu-vcore) - indicates the number of CPU
cores (virtual cores) that can be allocated to NodeManager. It is recom-
mended that this parameter value to be set to 1.5 to 2 times the number
of CPU cores

5.2. Resource Allocation Model

� YARN organizes resources through hierarchical queues

� Users can submit applications to one or more queues
� Queues are a subset of the whole cluster resources
� Queues are classified into Parent queue and Child/Leaf Queues
o Parent queue can have multiple Leaf queues
o Tasks run on Leaf queues
� If a queue is specified when the task is submitted, the scheduler allocates the
task to the queue
� If no queue is specified, the task is allocated to the default queue
� If resources fail to be allocated to an application due to the limit of some param-
eters, the scheduler will select the next application
� The Scheduler first satisfies the request for local resources and then for re-
sources on the same rack. And finally, for resources from any machine

5.3. Capacity Scheduler

Centro de Inovação EDGE - Big Data Course 6

 � Designed to maximize the throughput and utilization of the cluster
� It provides a strict set of limits, to ensure that a single application or a user or a
queue cannot consume disproportionate amount of resources in the cluster
� Administrators can restrict resources used by a queue, user or a job
� Features: Capacity assurance, Flexibility, Priority, Multi-leasing or Multi-tenant,
Dynamic update of configuration files

5.4. Task Selection by Capacity Scheduler

� First, selects an appropriate queue

o The queue with lower resource usage is allocated first
o The queue with the minimum hierarchy level is allocated first
o The queue that reclaims their resources is allocated first
� Then, a task is selected based on
o Task priority
o Submission sequence (FIFO)
o Resource limits (memory and CPU)

5.5. Resource limitation

� By queue
o Resources used by a queue may exceed its capacity due to resource shar-
ing
o The maximum resource usage can be limited by parameter, which is set
to 100 by default
� When maximum capacity of a queue is set to 100, it means that
tasks running on this queue can use all the cluster resources if
available
� By user
o minimum resource assurance of a user
� multiple tasks of a different users running at the same time in a
queue
� available resources for each user are between the minimum value
and the maximum value
� maximum value is determined by the number of running tasks

Centro de Inovação EDGE - Big Data Course 7

 � minimum value is determined by this parameter: yarn.scheduler
.capacity.root.QueueA.minimum-user-limit-percent
o maximum resource usage of the user, specified as multiples of queue ca-
pacity
� 1 by default
� resources obtained by a user can exceed the resource capacity, but
not its maximum capacity.
� By task
o Maximum number of active tasks
� Default value is 10000
� When limit is reached, new tasks will be rejected
o Maximum number of tasks in a queue
� Default value is 1000
� When limit is reached, new tasks will also be rejected
o Maximum number of tasks submitted by a user
� Depends on 3 parameters
� Maximum number of tasks in a queue
� yarn.scheduler.capacity.root.<queue-name>.minimum-
user-limit-percent
� yarn.scheduler.capacity.root.<queue-name>.user- limit-
factor

6. Enhanced features of YARN in FusionInsight HD

� Dynamic memory management

o Optimize the memory usage of Containers in NodeManager
o Kill a Container only when the overall memory usage of all Containers in
NodeManager reaches a certain threshold value

Centro de Inovação EDGE - Big Data Course 8

� Label-based scheduling
o Control which node (NodeManager) a task is submitted to
o Users can label each NodeManager
o Users can label each queue
o Jobs submitted to a labeled queue use resources on the corresponding
labeled node

Centro de Inovação EDGE - Big Data Course 9

Centro de Inovação EDGE - Big Data Course 10