Professional Documents
Culture Documents
Big-Data Computing: Hadoop Distributed File System: B. Ramamurthy
Big-Data Computing: Hadoop Distributed File System: B. Ramamurthy
B. RAMAMURTHY
cse4/587 10/27/2021
Reference
2
cse4/587 10/27/2021
Examples
4
cse4/587 10/27/2021
Traditional Storage Solutions
5
Off system/online
File system
storage/ Offline/ tertiary
abstraction/
secondary memory/ DFS
Databases
memory
RAID: Redundant
NAS: Network SAN: Storage area
Array of
Accessible Storage networks
Inexpensive Disks
cse4/587 10/27/2021
Solution Space
6
cse4/587 10/27/2021
Google File System
7
cse4/587 10/27/2021
Hadoop
9
Highly fault-tolerant
High throughput
Suitable for applications with large data sets
Streaming access to file system data
Can be built out of commodity hardware
HDFS initially providesd Java API for applications
to use.
(See MR in python here)
A HTTP browser can be used to browse the files of a
HDFS instance.
cse4/587 10/27/2021
How to study a new concept?
11
Architecture
Protocol
Data organization (s)
Non-functional aspects (Robustness/failure
managment)
API to access services
Software (MapReduce in this case)
cse4/587 10/27/2021
Architecture
12
cse4/587 10/27/2021
Namenode and Datanodes
13
Master/slave architecture
HDFS cluster consists of a single Namenode, a master server that
manages the file system namespace and regulates access to files by
clients.
There are a number of DataNodes usually one per node in a cluster.
The DataNodes manage storage attached to the nodes that they run
on.
HDFS exposes a file system namespace and allows user data to be
stored in files.
A file is split into one or more blocks and set of blocks are stored in
DataNodes.
DataNodes: serves read, write requests, performs block creation,
deletion, and replication upon instruction from Namenode.
cse4/587 10/27/2021
HDFS Architecture
14
Metadata(Name, replicas..)
Metadata ops Namenode (/home/foo/data,6. ..
Client
Block ops
Read Datanodes Datanodes
replication
B
Blocks
Client
cse4/587 10/27/2021
File system Namespace
15
cse4/587 10/27/2021
Data Replication
16
cse4/587 10/27/2021
Replica Placement
17
The placement of the replicas is critical to HDFS reliability and performance.
Optimizing replica placement distinguishes HDFS from other distributed file systems.
Rack-aware replica placement:
Goal: improve reliability, availability and network bandwidth utilization
Many racks, communication between racks are through switches.
Network bandwidth between machines on the same rack is greater than those in different
racks.
Namenode determines the rack id for each DataNode.
Replicas are typically placed on unique racks
Simple but non-optimal
Writes are expensive
Replication factor is 3
Replicas are placed: one on a node in a local rack, one on a different node in the local rack
and one on a node in a different rack.
1/3 of the replica on a node, 2/3 on a rack and 1/3 distributed evenly across remaining
racks.
cse4/587 10/27/2021
Replica Selection
18
cse4/587 10/27/2021
Safemode Startup
19
cse4/587 10/27/2021
Filesystem Metadata
20
cse4/587 10/27/2021
Namenode
21
cse4/587 10/27/2021
Datanode
22
cse4/587 10/27/2021
Protocol
23
cse4/587 10/27/2021
The Communication Protocol
24
cse4/587 10/27/2021
Robustness
25
cse4/587 10/27/2021
Possible Failures
26
cse4/587 10/27/2021
DataNode failure and heartbeat
27
cse4/587 10/27/2021
Cluster Rebalancing
29
HDFS architecture is compatible with data
rebalancing schemes.
A scheme might move data from one Datanode to
another if the free space on a Datanode falls below a
certain threshold.
In the event of a sudden high demand for a
particular file, a scheme might dynamically create
additional replicas and rebalance other data in the
cluster.
These types of data rebalancing are not yet
implemented: research issue.
cse4/587 10/27/2021
Data Integrity
30
cse4/587 10/27/2021
Data Organization
32
cse4/587 10/27/2021
Data Blocks
33
cse4/587 10/27/2021
Staging
34
cse4/587 10/27/2021
Replication Pipelining
36
cse4/587 10/27/2021
API (Accessibility)
37
cse4/587 10/27/2021
FS Shell, Admin and Browser Interface
38
HDFS organizes its data in files and directories.
It provides a command line interface called the FS
shell that lets the user interact with data in the
HDFS.
The syntax of the commands is similar to bash and
csh.
Example: to create a directory /foodir
/bin/hadoop dfs –mkdir /foodir
There is also DFSAdmin interface available
Browser interface is also available to view the
namespace.
cse4/587 10/27/2021
Space Reclamation
39
When a file is deleted by a client, HDFS renames file to a
file in be the /trash directory for a configurable amount of
time.
A client can request for an undelete in this allowed time.
After the specified time the file is deleted and the space is
reclaimed.
When the replication factor is reduced, the Namenode
selects excess replicas that can be deleted.
Next heartbeat transfers this information to the Datanode
that clears the blocks for use.
cse4/587 10/27/2021
MapReduce Engine
40
cse4/587 10/27/2021
Large scale data splits Map <key, 1>
<key, value>pair Reducers (say, Count)
Parse-hash
Count
P-0000
, count1
Parse-hash
Count
P-0001
, count2
Parse-hash
Count
P-0002
Parse-hash ,count3
cse4/587 41 10/27/2021
MapReduce Engine
42
cse4/587 10/27/2021
Summary
45
cse4/587 10/27/2021