0% found this document useful (0 votes)
111 views14 pagesBDA - Unit-3
Big Data
Uploaded by
Aishwarya RayasamCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
0% found this document useful (0 votes)
111 views14 pagesBDA - Unit-3
Big Data
Uploaded by
Aishwarya RayasamCopyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF or read online on Scribd
ratory of MapReduge:
c re ‘
At the bighest level, there are)
¥ The ctent — which submits
¥ The jobtracker — which
15 0 java © Subriasion 1 ™ olcahe oan
hy tbTmitey method on Job creates on imlemal Joesie
Ofoe and calls SubmitdsbFntemall ) on ik Gepy
J°b submission process fmplemented “by Jobsubmitter 7 |
the foticsir ia
X Asks the jobtracker for anew jobID (step 4 ; ide
¥ Computer the input splits for the job Copies the esuacsy pe
te run the job> inclucling. the job’ TAR file > the icon! 7
ard the computed input éptits, bo the jobtracker's filesy %
™ a. directory named after the job ap, (step 8). if 0 |
* Tells the jobtiacker. that the job ts veady for execution (EP
® then the jobTiacker geceives a call tp its submitJob() methods
it puts it into a internat queue from where. the job Scheduler
will pick it ap and initiatize it- ; a iF
sxe Initialization involves creating an object to 4
Job berng wm. (Step 9) {
# To Create the list of tasks to un7 the job scheduler Tivet ve trie
the ‘pet Splits computed by the chek’ tiem the Shared “file”
System (step 6) iy
e It then creates ene map task fevsach -split
asics neciquinen i aaa } Raa
yp Tasktackels, awn. a “simple loop’ that actically sends
heortbert rnettod call to the jobttacker, fy coped |
oe Heartbeats’ tell the jobtracker that a tasktacker is alive,
yo 7S a part of the heartbeat, a tasktracker will incticate whether
it fs ready to mn a new, task, ond if it 184 the, Gobtiackey wil)
qallocale “it & tak, which it commanicaleristathe task tracker
usm the, heart beat Value (step).
Task Brecation i= Z borat \
we Now the tasktracker hai beén® awighed ebtask y the next step is
Abi it i trun the task. | icy : a
yp First» it locdtizes ‘the job JAR by copying it from the shared
file system fo ‘the tasktracker's file systern i a
# Tt algo copes apy filer needed fom the dishibuted &
Syste) cache by the application te the Local disktep s)
F Tadsk Rarer’ Wumnches a ew Java Virtual A
to wun tayeach task in (step 19)
j
seperti the PER ONE) Otaus y Opole &
¥* Values Tt 1s
| 3 key Values Text Empat Format = Hh ee
|p Tt 8 sirotilan: be / TextTnpt Formak. This “nput-format also
treats each Ife ‘oF Tnput- 03 oO Seperate record while the
difference 35 ttat TextLnputformat treats) entire Iinevas the:
value, but the key Value Text Tnput- format breaks the line
Itself into ey and value by a tab character (Yt!) :
* Key — every thing: up to the tab chaactex
* Value— Te is the remaining port of; the line after tab
abst) charactes
(4) Seqnens ti) ee
Tb is an Inputformet- cohich reacls sequence Files s
| 96 Sequence. Files ave binary. -files these files also store
Sequence of binary Key~Value pais:
> These are block compressed and provide cliect serialization
and desevialication of seuerak anbitany - i are byte ofa
ine S05 each
Tnpue with Te
tas " Compression is conbollecl by the Static |
14 ae Format +
? Se quenc file As Binary Output Format = a
Te is another form oF sequence FiledsyOut pal ae
Uorites Keys & Value to ne file. ™ iat Format~
4) Mapfile output Form
4 Tt is another form of | ile outpubtonmae 0) aleaadiam 7
Formats
SE Tt is sect to tbe. write output as) maptiles-
5) Multiple outputs it
* TE altows writing clata to Files tohose Tarney are ey e
M4
2H
‘fom the output teys @ valves.
6) dazy zy Output: Format a
Bi 1S O. cura ppey per “output: formt- ich Ser (eae
Cutput file will be created onl when the aa js emitter
for a given Partitions. iF
3) DB output Formats od
DBoutput-Fermat- im Hacleop is an | Output formpltitorl
voriting- to relational databases ard H~Base -
File Tinput Format oF f _Mapreduce & ‘
The input to a job is specified a a collection Uf J
paths which offers great Frenette fn constre.binng the
Fnpur fo ajob- Pd
+E File Tnput Format offers “four Static convenience * :
for setting cc Job conf's tmput paths: has oO
pablic Static void add Tnput paths( Job Gfnf cone Fathpatt)
public. static uid add Tepur FE ee :
1) 8ting co
public Gtatic. wie set Trpur ths Job Cont cenf, pac ea
public static void setTnpu Raths (Job Ontconks Sting.
Seperated Faths)
The ~ add pat Path )-and, ackd-boput paths) resin
pet ov paths to the fist of Fnpats yor can call th
ds repel to build the ee § ae
5] (a
E 4 é|
ef | 5
key Vali Tnpat eat <5 hemp |
| Bie
Li
le aa if is
BPs e602 1 i
eee eee
foc et cif
f
Formate koV>
<< nkerfaces>
mpul}
Ls
"2 th. vepresenting a. divectony vincludes gH the
2 ae! rotttea «i. 9 bell None call
* e i is done sith Uthich dec’
the properties that Hive exposes and there default values
you can ovenide | the configuration divecloy: that %
teoks fox in pive-site-xm) by passing the ~confiq optior
the hive eonmand- og
7 bi ~-config. [Users i /
The bive-site-xml is a matitral place to put the cluste
Commection cetails:
¥* You can Specify. the Filesystem and dobtracker using the
useral hadeop Properties » fs default «name ard mapred «job-
# tive also Permits you to set Properties on @ fer-session b
by passing- the hive conf option tp the bive command.
F To see the current value of any property » use SET wil
the Property name +
bive > ser hive: enforce - bucketing +
bive- enforce - bucketing = true:
F By itself seT will list abt the
Properties set-by hive
Note that the list will “oct include | Pactoye eerie
they have been explicit. overviden in one ofthe ‘A
covered in this Section. “ d
# Use SET-V fo list all the hes PANG
ocludting! “Cidfaults- Te Apen
jis cute hierarchy, bp ;
*” “There ‘is @ proce cs t sett is
lowes numbers take preedence over re Properties Tn
1» The Hive SET Command
2. The Command line-hive confoplion 3
» 3 hives Sitersernbins > i> od Shaclood —defautt.rxn)
zbyie e3 spiel
%
3
t
i
E
2 3) Hiss
y The “Hive Shet!
using) the five
using. “sewi
¥ Type hive sewiel
the most useftl ”
¥ The command | Th
/
sexvice © /
hive Server
fluns bive asa server La Gra
fiom vange of clients uovitten im
5 * Applications ustnq the thrift, J
creed to wm a HWve serves fo comm
2 set the tive PoRT Enviopment van
sewer will listen on-
“The Hive web interface
; Jar 4
| “The Hive equivalent to hadeop Jarry a ce é
|
:
:
java. applications that includes both hedleops t
classpalb-
meta store +
By defaulty the metastme is oun im th
the Hive sewice+ Using the service 5 it is pos:
amelastore as O Stardalone process
x jet the MIETASTORE-PORT envionment va
the po the server will listen on-
Hive clients «
TF you wn Hive @3 a sever, then there ayo
mechanism for comecting - -to it from applications . k
