Big Data: Hadoop 2.

0
Map Reduce / HDFS 2.0
@diego_pacheco
Software Architect | Agile Coach
Big Data
Hadoop - Cases
Hadoop
H
D
F
S
adoop
distributed
ile
ystem
4000 nodes: 14PB storage
HDFS Assumptions and Goals
Hardware Failure: Houndred or thousands machines, expect to fail.
Streaming Data Access: Batch processing, high throughtput not low latency.
Large Data Sets: Terrabytes, works on cluster, scale, milions of files single instance.
Simple Coherency Model: Write-once-read-many(create, read, close, no
changes) maximize coherency and high throughtput, perfect for Map/Reduce.
Moving Computation instead of Moving Data: Is way
more cheaper, huge data, minimize network. HDFS moves the computation close to the data.
Sofware and hardware Portability: Easily Portable.
HDFS
Very large distributed FS
10k nodes, 100M files, 10PB
Works with comodity hardware
File replication
Detect and recover from failures
Optimized for batch processing
Files break by blocks 128mb
blocks: replicated in N dataNodes
Data Coherency
Write Once, Read Many
Only Append to existent files

HDFS - Architecture
HDFS 2.0 - Federation
Hadoop
Map Reduce
Today: Parallelism per file
Single LARGE File
Single Thread
No
Parallelism
Map/Reduce: Unit of data
Task 0 Task 1 Task 2 Task 3
0..64 mb
64..128mb 128..192mb 192..256mb
Each task process a unit of data
Today: Network issue
Task 0 Task 1 Task 2 Task 3
Node 0
0..64 mb
64..128mb 128..192mb
192..256mb
Node 1 Node 2 Node 3
Map/Reduce: Local Read
Local Read, no need for network copy
Data is read from many disks in parallel
Map/Reduce: The Magic!
Single Hard Drive: Reads 75mb/second
12 hard drive
Per machine
12 * 75mb/second * 4k = 3.4 TB/ second
Map
Reduce
Big Data: Hadoop 2.0
Map Reduce / HDFS 2.0
@diego_pacheco
Software Architect | Agile Coach
Obrigado!
Thank You!