UNIT

Undarsturding Hadotop Ecosyskem : Hadoop Eo sqstem,

Hadoop aictribeded File ss tem, Mapkeduce, tHadoop
Pig ard Piglakin, Sqcop,
yAPN, Hbase) Hive
Zcokeeper , Fkime, Oozie.
HBae; The
Uncers landing MapRecuce Furcdamen tals curd
MapRedeuce Frameork, Techniques to Optimi2e
Role of HPse
MapRecluee Jobs ) Uses oP Mpkeduce,
in Big Dala Processing.

Hadoop Ecosysicm.

+ Hadocp ecbsqsterm is a pkafform or framecvork cohich hel

solving the big dta problem.
Compises of different Components cd services
maintaining) inside
Cinges ting, storing, analgzirg and
OF t .
de finedas a Comprefensive
Hadoop ecosyctem Can be
took cnd technologies, thit can be
collection of
effectively implemen led nd depleged to provIde
Cost eFfecive manner,
By pata Solutions.

Most of the services avgilable in the Hadocp

to Supplement the main fouY
ecosqstem re

of Hadoop wbich is ixude

Core Components.
HDFS, YARN, MapReduce asd oromon.

y
Hadoop ecosystem iocludes both Apache opeb
Sorce brgjecks nd other eoicle vaniety
Corgmeual loo Ls and Solutiens.
Cpen soUAL
Some of the cwell kn oon
e9camplks inc ludl Sponk, Hive, Pr9, sqep
 Apache Hadocp Ecosstem.
Data
Exchasge Previsicni nq, Managirgeunt
NachineleanMonilo ig ibtepcloa
Comectes
Re |Colomnar
Dta
Stoe
Ialor
kflos iphng Sfah'shès
oop bozie Maboul
Pig
Coordiahon
dcor
colle
Log
Zoo
keepe? Hbase

Elume SARN Napfek1ce V2.

pisibcutkd Poscessing framesbtk
HDFS
Hadosp Pist ibued Fik Systmn.

Figcrve: Hdop Ecosysk).

Oozle chukwa flume bata mangeest

Cwo»Plbs CMonitoing) Zookepe Y
CMonitorirg)(Manageno)
Monitoi')

Mahout Ayio
(akatou) CHochipe Sgoop
CPDENS
Data Aceess
Lennng) Sentalaat Connecto)

MapRedeuce yARN Dato Paocessing

(c fus tes &
tusk» Managemen) Reseuree Managme)

HOPS HBase Data Stoage.

(bislnuted File ystern) Ccolumn DBstorage)
igune: lacloop Fcosystem element Q vaious
shges of Data Processing.
 Au tyctot) (HDES )
Hadep Aistacbutd
HDES ane desired o mangeS oF bgelalq.
HDFS
stoAage SGstem ahich

largage
Hadocy epplicatins
pmay
steady
Alo buikling is oealy twthot a sofid bae
the ccse of Hadcop -the bae consist o HD[S
and Mapkeduce

which are
HDS Consis t o two componerit
Nane Node and atanode

These applicatons used to stose lage ata

mltiple nodes on the clceste9
Hactop
() Nane Node

which operate and taintwn all' Data Node s

(slaweg)
Recodey of metaeata Por al blocks, iH cogtaing
infomotion like sizey locattn )SOQe ete.
It vecoÝds
changer -that happen to
toetada to.
If file gets -deleted -tnthe'HDES, the
Namenode cuilI artomatically ecord it to

Name Noce freguestty kaeves heat beat nd

blck eport from the dala hocdes tn the cla rler
to an ye.
ensue
they woking
(i) Data Node
slavenooe cohrch on cach saue
bmachir.
acts Storage kuice
Taker ecponsbility to Serve Deacd and wrte
Kequert foog the uses
It icts -to -lbe
Name bde
aroading
cuh ieh lele ting. Blocks,
ncucbg
adling blocks and -ho blo ckr
plocing
 to the Nbe Nocle
J sends hearl beat ep actual
ts Ume ts
and he
reglaly
3 seconcds

HOFS Aachikcure

4 HDfS has a msk- slave ochileclcdse

Num be or bata Nedes,
y 1t Comnprlses Naroe Node and a
the varlos
Narne Node the mstr that nanges
DgaNpdes HDES AAchitect Metoatu
op scponley

Motalata cps Nmelocd

Ruplicas =3
Name -Iheme /fealahata
Client Raplcas- 3

Block Ops
Raad ataNes
Datantecli
Keplitabtin

Roek 2

Client
Figuu ; Diplaying the-hehikekuut ot NDES.
Hanae hocl manages DES clugles me taclata o heve qs Date hlod
Stohe the ale
Re cord! and dielories ae puscnte d by eleent, to the
Wamenode
These e o ds and direclonie managed on th Alamonhde
Openc lions Such as moditicaton fpenung ond closng
pexßosrmeod by the Nam hde
cCun be divtdod iito One od
Intenally
MOe bloeks stoed a goy o6
Datawedes
Data wck Acac and tuiles the Aequst oom lhe client
 Can also oceeto opealins iko

the catalion deletion and

the nstauetons ron the
bloekr depending
Nami ntole .
BlockS HDFS Anehitectne.
Concept
4 dtk ha cea lein % 512 byteg.
helbr tha minimize the seelk cferaln
Which
aultiple. block reed
y IF the block size & legs

cohich leac's o lhe cheAeasce

huge eco Ad
s seek time

HOFS clstes NDES Cluu lee

SeRve Seue
OataNek Alametat

Heartpéat Hea}beat Wotam

Henutheat
Sve |secve Suval seves
JQatat

Fig' INustkaton Heoloep tteat beat Messge

A hcalt beat mesagc seatPpeaus OA rew featial
Aecetved the espeetive Datadoda
seNding tre mesrege is adoled othe cluites
HOPS pefoane is ealculated
Fault tolerance
and quikly Aecuiny the cate.
 cnd
Thus
Atcovety conpletid thaeegh aeplicotien
Kcliable

Facili tale numbel

tothen HDES
Some

commncale
’ Monitodin Kata nocle and Alameecle

thteugh ContihuLas sinalt (ien beat ).

is con sdejec)
not node
hoated the
availabu .
and Cuoeelo be Corges
jauled
The, node s teplaca) by the aoplito
chene also changed.
he blocks shiftod 6Dm one
-’ Rebalancing
to ano thez locatiog
the faee spece
owaibble.
-the
’ Meledate eplicotin H maintaun thu neplica
the Same ADES
eorresponcig Rl-Uhifopg
Resoue
The Comhaand-kios Ineascue,. Idesife/

iriterfaces fox HDES.

WumeUUs ditPerent
'machine We need to
eNe ete HOFS 0n
Neec to nale p<oo y.
in a. dttacbatd
Set up fast Hacdoop
hdfs :|
The prncpal as ß. defacut . rame
Lshuch Îs used to
set default Hacdec
local host /
File syste nd hee cwe have
URI,
ik ytems ave tagged by9
utiluxed an HDES to fesgo hdoop.
Using HDFS Eles .
ppica tioys
The HDES File sys tey is aCeessed bey
culth the help . of the HDs chent:
cquied
-lhe metackkta the filesystern
whcathed
 cfonage
have tmeltipe oplics
7he ohject ctass ir Chcalad fos

auessin9
Yhe ile syeters
HDES.
abrlaoct base tlaor
c locs
geneaic file syctem.
An instante clorr (an bo
Fikyetem
cAta te d by passing a ne) contguacton
Chyect nto Conslacc los

Asseeme that Pladecp contigunalon

SUch as and hadop srlo m
hadecpe Faut. zm)
paesent the clafath.

Configcaa tion corbg neo conpyuna tion ();

cong'igenatg objet contg.

Fkiyclem Psgs = Fie ystem get (untig);

CAeales a ile Sycte cyject fr

kitt : Cieating a Fileystey Chjeet

Patt s ano-th Apostart HOES ebjet,

whieh the haS Che
peities
dinectoieg the hile systen.
Path objecks alloo you to
and
HDES Fs
pvgramalie opeuatony n
pafon
and diecloey,
Path Cile name),e Cealing
n
path fps Me prtß c lasr
pa
iP(fsys. exists (p)) e Checkrng the file
Nstetenent 1

Boolean
fsyst Ceat eufik efp).
Bolean teult : Iys. dele te Cfp):
 ES Data 1npt staeam
hle
FSData OetputStaen) tNnting
he file

Hadop Spetic ik Slem ype

HDES provides Sovea specific file syctem types thaB provlc
richer Funelionality ad Simplty the proce gsim
client
The loal fe gstem Hadop berfcoms the
sde checki) cpera lion.
clicnt oß the
When (pos UKle a a fle, -the

tile dvectty cvealer hyddern doceimant

sycteg cantaining ctheckscng
ncdex
.Pikrame . crc in the Same

chank o6
the document
each
For
managre by -lh io byes pr,
Si2 byles &y tofauk
the chunk is
eohuh
chect sem. pacpeaty bits eD a
Refeys to he no. 6

enit inckudec ewcth the cenit to enable

tiansussICn
the humben
the ocerea o See cwhethe
bits has aAied.

ile yeten URI Jae Iropkmenation DePinition.

Schene
Corg. apoche. tadop)
Loeal File Is. LoealFile Syelery Afik sstem fea a kraly
eonneckee dsk oth eheo
side check surns.
Rowkocalfie.Syeten tera
Joal File sskn twjth no
checkcurns .
HDES bdfs hEs. DislibutcdFie
Hedecp iaviblad le

HETP
bftp hdfs. Htpfilegskn Afie sysken pivicing
Iend.
HDES
only aCtess to

HSETP hifs. HsPp fle gity ile syetem proviting

hsftp Veoc. only accese to
oer HITPS

HAR
I. tlhufieSyetem
on anomer file ystey

Tabre: Macop speorte file yilari Bres.

 HDPS Corohande.
Comnand intedect wits
Vaious shell ike
dikect ly with othet file sys tm
Such as Local s, HFTP
siypoctid by Hacoep
FS, S3 FS,
poiclod by the File yslon(Fs)
Commancds ae
shell.
Fs Shell can be covo ke by chu follorong Comnard
bin hadcp As Ca49s7
[Commarck Descatn
aypor) ToFile Lised tor ppondug a shge hek d
the leca Fale sy'tem to the <oealsrc>.-..Kdy
destiato tie syso
Cat
Csed foa copyihg 4sage : hdßs dr -cat
patß t. sdlot. URI [URI..]
ehmod tlsed for changing thu
panissions %gles kmooEL, MODEJ...
|0CTAL MDDE > URI[URI.
chown Used Po changing the ocuna ls dPs -cho
of Pile tR]
CowNER]c: TaRO UP]]
URI LUR1]
Count Csed Far Cocunbng ¬he nber
oP dieties, Pile and
beytes cnde the paths
match the mentined
Ple pattem.

ised Fas copying, Pils 6on hd Ps d

the sounce to the dstintin CIEpl-d
get tied oy copying files to the URI [URI.1<4es/
tocal file stom. hfs dfe-get
kdii Ced Boy Cieating diascoaes -igroreereILol
{src <localdsty
by taking -the kath UR) as hdB ds - trkd
HDES an argnont.
G <paths 7
Tabtei, Commans d theh Desuçtoy
 EeatosS of HDFS
4 Data Aeplicalitn, data xesilience nd dale. cntegity
thce key featu0C
en sue that
ata sblcatun
the clate availa6te and prevent
js always
datu lofs.
an onganizatuon,
Aota kesiliencc Ability -the face
mainain itt clate is
tetoVeA, and
e% cenexprcke d e veils disrptis.
+hat ehsini
Conept and paocess
Rala antcgnty i aud
completenes Consisteny
accnaly OAgeizhoy'z data
veliclity HbES have fu ability
FaLlt tolesance and eliabiiy i
file blocks auc <toc them
kepliat Saut toleeunce
ncles in

and aeliability a(tcTS hitl,

Beccuue s aeplicely
Hagh Aveuile;labilihg the Nametode
avaiable QUen

Jata Node Pails dali

scalehilety in th clus tot
Aegluncmerd1
Vaiocur noels
can sCal to fenclecols of noce
incaease,
HDES sloses dala
tigh thuougbput Can be beocesse in paAallel
tyannes the cla ta
clustes hocdes

iHh HDES
Tompataton heppens
Data hecah ty i where he deta es/ des
Data Nodes
-the cohete
the cate
tathes han having miniý/ng
conputa liosaf cnit By
Ihe the data and he
the doc ases
helutk
ths prsach
boists
congesten
thvgpul
 Mapledue. Refea eom Chik I.

Hacleop YARN

MpRedue Hodop kosysterg

Act as cOmyorent
1B Facilitale logi e 7 prucessing
ohich enabks
SoFtwanl data seks
laige
writing applications -hak proc ecs in
distbakd and
using
environrent.
Hodoop plays a
9 Auallel feature. of MapReduce
piocess/ng
Chucial role in hdocp ecocyck
perfoming Big Dat Analysc sing
Help Sane
ces ter.
machines in the
toeltiple
How cdos MapReduce WeAk
have uo Functens
we
the mapReckuce pogran is Redue.
Map and the other
data nto
II coverts ohe set o
Map Fceretiop i individual alements
ansther, where

byoken duen. into tydee

CIe

Ckeyl vake pais)

takes clata Frorn e the Map
Redce Fencion:
C) înpet. Recduce
Funefion and surmmeh zes
Feunc ficn ggregales
precluecd by NapFoctn.
the tesls

yARN
(Yet Anotber
Negoialer)
Resocrce
dets

yARN
bran of -tfe tHodocp eco sys km.
the ompukcthnad
Roponeibility in proviling oppkatkn escecuhb.
peccdecl fer Be
re soerce
 YARN Consist o two essenial ebnoonenks
-’ Pcsor Ce Mager
Node Manager
Rescuce Manciger.

Resocrce
Manager

Nade Node
Node
Manager Manager
Manager
cluser level and takes reponabilty
* tworks the
acter machine .
for runring9 the
heavtbeats from be Nede
It sores the track

oanager . negtiaBes -the

the job
brgssKHS a
takes
pphicahn
Frst Coptainer For exechng
Componenb: Appl1cahicn.
Consist
and cchecaler,
tmanager
Alode Manager.

Secondory
Name Node
Namelocde

Dakhde
 and uns
korko

On verg save hachine .

vepnoibk or mntomg resaut te

each
cont atner managny
Cntupets
managernest
Ihock of log
hode beal th.
CommOICaton
ContinuocLs
majntaun
tecorC manager
o giue pdates

HBase
because it i
Hacoop dalatase
Sca lable, distibutcd,
No SQL atabICe reuns lop
II iS
to stoae
the truchuoed
Apoche HBase i's designed
data Fable Fomat. oilljcns o
billions oF rows and
Table Constss of

erlumns
data to read
HBase gives Qcces to get real -tme.
orile on HDFS.

Chent

HMaster

8 |Zookeepes
(Ragi-n (Pegjio)
(Regon ) Rogi)

HDPS

Fiq pach #Bcse chbikctuse

 HBase featies
NosQA latabace
HBase an Cpen sOCre

has a featike all typer G

tinique to'suppot
data .
crucial tole in hanclin
his featcere it play
variocus tgpes of datu. in tkecocp.
its
torrHen in Java
The HBose originally cnd
be eoriHen in Ayo, REST
pplicatons
ThaiFH APl's.
Cbmponeols oF HBose
HBase .
Componeok
There gjosly
HBase Mastey

-’ Regrona/ sever.
HBase Mastek: IL s not a pot os actual dla stokege
activihes
but it manages foeLd balning
Region SeveAs
Contols the Pailove!
cadhoinis/Aaliog actiuities Lohieh
Retorms and deletirg
ntefate fon cAeating epdating
tables.
Hanclles
epeAatins the Hetiop clesten.
aintains and tnonilors
It
wueke. node.
Regional Servek
clint
eueites and eleles requet tiom cttA
hoele c6 Hacloop
Region SeveA
40S dala mclos.
clestes. gerveL

Hcatalogue
fos Hadocp.
Table end stooge management available
Coiponnts
ve and Mapteclile
An Hadep Sueh c!
Pg
 qnd eile cate, rom th clus tu
read
queickly tohich a lows
ike
hae the teatne
data. ahy
format and ciuct
thoil

Benepils ef Calelee
wite
Read data >om Haleep
ckueler
datu into a Hadoep
cwcth oth tadeep
-the integaatuon
tods
and cwebsevAS,. to
H enables API s
metastoAe
the mefadata hive
claba
,acchiving and
It gies 'visibiliy for
data cleanig toos

Hive
pen sowace obtoe
data tUoaLhoure

Apoebe
bor pekomirg
olata quey nd anolye
data suneede
Mainly doee ehace genetion
24eay .an analyes
language dalled Hie QL CHQL)., gmilan
Hwe ses
tianskla
tiancilalon
welke
Hive QA maphldee Jots
queies nto
-the sÇL
Cwll be execeled Hadoop.
Huc Qae.
Man Componet
olevie tor
’ Meta s Cor- Ie SeeSe cis teauge
holds the intonmatir
Thi hyetcda
the
metaclata.
locateÝn cnd thernee
each table Such as
ata ass
backeup stoe
Lts also keep
 -the nstauetin and
Centollea cbseaves the paget
Cxeu ons by
and li7e cycle
Ckeating sessions.
cor6 the task
’ (cnypileA: he conyplea cullocetocl
the thve (2
into MopPeduca
6 conveeting quey
Co ex ecute
input, is decigned to paoces the
heeded lo enable
the st
HiveQL outpcet Aequitd by the MapRedkue
Apache Pig.
language þlat7m
IS a Piyh level da tareli that
and gueaying lange
analy ing
ane sloted to Jcwa
altecnative ang cuage
Pig wnkr and geneAate
prgaamwg
tcunctins acitomabcally.
Maplelee wbich
Pig katib !
includes
Pig nto
langcage. latin seipt
tAansate the
YARN and paCef
Pag Can
which
MapRedece
chustes .

s0 soing compux cue catcs

Pig s best suitable
data operatins
that negtuas tmltiple
Houy oloes Pag. cwoAk ?
the data eo
load' commad co eovd
lale

on sech a gecping date

Perfom Vaaious pesaten
fitteaing, joinng, sentíng et.
Can
cleemp the data
the aesclt back i HDES aetoAçlig
you Cn <toe
gyeua Aequitement.
 Apachi Sqoep

Sqep enaBlos
that
Iont end m
teaface
Ke lo tronal dalabaseg
Bculk Ha doop
slauctuaed data raute
and indo vatioacly
tre twin called develo png sapt'
data
and eport
amport
to oveng data a
mainly help to
ct eapause data base Lo Hadoop clutes
the
CExtaect, Tranc7ouy
þeatomng
ane koad (E7).
How doDeI Sgoop woAk/

RDMS
Hadoep
DB2)
CH#DFS, Hie
HBase)

Ilhat SÌcp cloes

the to tlocorng tasks to integ aati belk data
moverment Hacloop and slauctune dale kaisl.
b/w
data
qccp telfills oe 9owing hecd to
-the masn foe to HDES

Helpu achlvng bmprva

Iight weght indering tor advanced guey
beßonante
foatuu lo -laantfeorn he la'li, pena llally
pe omence
 an
Sqrop c«cates fast data copies om
(xtesna/ SOCIACe tHadeop
load b lancen by integ ratng entag
act
ofhu devices
Stoaage and paocesing
O0zie.
Can be
* Tt is a tool n cohich
hanne
cwot
pipelined teqeuined
distaibead envitonment
schecleelen Sys tem lo Ar and
Oozie wOrk as
jebs.
anage Hadocp

Oozie chent
1. cL)
2.Davq
3. REST APM

Hacdccp clutte
Oozre Webapp Lauche bb Actual Prcgr
Cliard /6uted Single tnap tact 1MR,Hive, Pig,
Savecontainer Wo rede lask pakeh.

Rg: fpahe Ooe.

Oozie combineng maltiple Coplke Job be d n

seguent1al dea, to Qehieve desied tutpul

cuith Hadoop ctauk
skrongly integratdl and systro
ve . Sq oop
Stypor tng jobs ike Pig, shell.
Specfic jobs ik Java and
Java
web applicatons
pen- SocALe
Oozie
Consist two gobs
athons
) D0ze thu job
*asanged to peafom
Ghe aler anothe.
cuorkf low jobs
i) Oo2je Cooncinalus
baed
avaibbitty
data
pAccle7incd schedules
and
 Cen dnale multiple #adecp

and antahng
rganizng
dittecbated

evith simple Apls

Zeckeepe
and Achitatne

alloevs dve lope«s

Yokeper Concentia ting
a

applca lions instead pplicalton.

diteibca ted

Zokeepe aett cncagh Com0

euhete te data
than becquuc
- Zerkeepea acts disciplincd
antans

Apache Flure.
sek
collects aggregales and moves aige
back
data Oigin and sert it
HDFS.
echanisn
gault- to le ant
tiansmitting
Hadeep ehubaonment
wnto
in getng data tm
erabls
setveU iomedia te y nly hadecp
 The MapRockice famubk
and sachcall y
hae MapReduee, s
tho two capabhtes of
a combinalin %
cxs ting conyjen largags.
Map ad kecluce.
7he capabilites ae
ng
bo-th as conmAc0al patuls
laileble
mogols

-the
tealues 5 MapRekua
Exploxing
MepRecdice inLoves
Schetu liny creciled by
cvhich
ap and redule chenks. 7hese
Snalle
achems to
diuclng (onge ky defßeant compcäng
paalel
chunk. are pealon
hke gpeAd lons. The maPpng -the no.
proatixation besed
hodes aRe tewee
Io cCKe
an th cfus tei,
ef ocles ate xeceto
a

then tasks
than tagks ,
tak basis.

) SyochAorization : Eec utron of concurent ocess reguies

Syochorizatiin. The MapReclce progan exCCatioy
FramewOk awane the. mapping and reducirng
Cperatrens, The trarmeeork Hacks al he facks abng
Lwrth thejr timinings an the

poces the Cenpktin

boethod knoon shuptle and
the intermediate da to, ewb1eh

tbe nl. mecha`ho colkching

the most e PFecbiue prcering Cutilme.
n) Honding
Grors/fautts . Mapketuce engns ustally
fofeance anf oBustner
prvide high leyel of factt
robustnr
in handling eror8. The reson tor poovehg
lo mak erPS
to fhe se thel
Or Faul Is. 7here re high chnes