HDFS Harnessing Big Data: OLTP: Online Transaction Processing (DBMSs) • OLAP: The per-application ApplicationMaster has the
The per-application ApplicationMaster has the responsibility of negotiating
Characteristics of Big Data: Scale(Volume), Complexity(Variety) &
Speed(Velocity)
HDFS should have the following properties: Distributed - Fault and Error
tolerant - Scalable - High performance - High availability - Able to store and
manipulate large files - Low (or medium) cost.
Distributed FS: Mounted on multiple servers (transparent to client) – Consistent
namespace across all (clients can think of it as one big system, without worrying
about location)
-Concurrency – Failure tolerant - Should allow remote access.
HDFC Concepts: Latency (measures speed of response. e.g. If you hit a key, how
fast do you get the data back) - Throughput (measures how much data you are able
to process in a unit of time.) - HDFS is designed for large batch jobs, not smaller
interactive jobs.
NameNode: NameNode does not directly read/write data. (This is one of the
reasons of scalability of HDFS) - Client interacts with NN only to update
namespace (file names) or get block locations. - Client interacts with DN to get
data. (HDFS is single-write, multiple-read) - Checks periodically on the DataNode
state
How does NN persist metadata?: Two different constructs:
Fsimage - point-in-time snapshot of file system’s metadata Think of
it like Mac’s time machine - It’s good for writing bulk changes, but
not very suitable for incremental changes e.g. if you rename a file, you
don’t want to go and write it to the fsimage.
edits file - smaller incremental changes are recorded. Thus, the major
changes (checkpoints) are recorded in the fsimage file and incremental
updates are in the edits file.
NN Failure: Namenode is a single point of failure. What happens if it crashes?? 2
solutions: - Backup the files that make up the persistent state to multiple filesystems
- Run a secondary namenode. It periodically merges the latest namespace image to
the edit log. Creates checkpoints.
Conversions:
Design Decisions: What would happen if a cluster needs more processing power?
- HDFS is based on concept of “scale-out” and not “scale-up”. - That means, you
can add more commodity hardware easily. - You don’t need to upgrade to expensive
hardware - Another advantage- there is no single point of failure.
HDFS Federation • In Hadoop 2.x, concept of HDFS federation is introduced. •
Allows a cluster to scale by adding namenodes, each of which manages a portion
of the filesystem namespace.
Diff B/W RDBMS & HDFS: • RDBMS is for structured data, HDFS can work
with unstructured, larger datasets. • RDBMS is transactional, has ACID properties;
HDFS is batch oriented, failure tolerant. • RDBMS supports read-write operations,
HDFS has write-once, read-many property.
What is high throughput? How is it achieved? - Throughput measures the
amount of processing done in a unit of time. - By performing computation in a
distributed, parallel way, HDFS achieves high throughput. - HDFS provides
streaming access to data, which ensures the entire data stream can be accessed and
processed in the most efficient manner
What is commodity hardware? Would you use it for the namenode also?
Commodity hardware is a non-expensive systems which is not of high quality or
high-availability. Hadoop can be installed in any average commodity hardware. We
don’t need super computers or high-end hardware to work on Hadoop. No.
Namenode can never be a commodity hardware because the entire HDFS rely on
it. It is the single point of failure in HDFS. Namenode has to be a high-availability
machine.
What is secondary namenode? Is it a backup node to the main namenode?
Secondary namenode keeps the checkpoint on the namenode, It reads the edit logs
from the namenode continuously after a specific interval and applies it to the
fsimage copy of secondary namenode. In this way the fsimage file will have the
most recent state of HDFS.
Online Analytical Processing (Data Warehousing) • RTAP: Real-Time Analytics
Processing (Big Data Architecture & technology)
Key points:-
Features of HDFS: multi-node system - fault-tolerant and uses redundance for
storage - best suited for batch jobs and can provide high throughput of data
processing - uses inexpensive, commodity hardware.
Block size in HDFS much larger as compared to other operating systems To
minimize disk seek time as compared to data transfer time
namenode data storage and recovery operations: Secondary namenode
periodically merges the fsimage and edits file into a new fsimage file - NN stores
all meta-data about block-node mapping in memory - fsimage is a point-in-time
snapshot of the metadata - Incremental edits after the point-in-time snapshot are
written to the edits file
Properties of HDFS as a storage medium: Distributed, Partitioned, Fault-
Tolerant by using replication, Write-once and read-many, Commodity Hardware,
File stored as blocks, Designed for high latency, high throughput batch processing.
MapReduce
MapReduce Design Consideration: process vast amounts of data (multi-terabyte
data-sets) - parallel processing - large clusters (thousands of nodes) of commodity
hardware - reliable - fault-tolerant - should be able to increase processing power by
adding more nodes ("scale-out" and not "scale-up") - sharing data or processing
between nodes is bad (ideally want "shared-nothing" architecture.) - want batch
processing (process entire dataset and not random seeks.)
Key-Value Pairs • Mappers and Reducers are users’ code (provided functions) •
Just need to obey the Key-Value pairs interface • Mappers: Consume <key, value>
pairs and Produce <key, value> pairs • Reducers: Consume <key, <list of values>>
and Produce <key, value> • Shuffling and Sorting: • Hidden phase between
mappers and reducers • Groups all similar keys from all mappers, sorts and passes
them to a certain reducer in the form of <key, <list of values>>
Hadoop MapReduce Architecture
The concept of YARN was started in Hadoop 2.
YARN allows multiple application engines and frameworks such as graph
processing, stream processing, batch processing, etc
Functionalities of resource management and job scheduling/monitoring into
separate daemons
The ResourceManager is the ultimate authority that arbitrates resources among
all the applications in the system.
The NodeManager is the per-machine framework agent responsible for
containers, monitoring their resource usage (cpu, memory, disk, network) and
reporting the same to the ResourceManager/Scheduler.
NodeManager runs on the worker node.
The ResourceManager has two main components: Scheduler and
ApplicationsManager.
The Scheduler is responsible for allocating resources to the various running
applications subject to familiar constraints of capacities, queues etc.
 Scheduler allocates resources based on the idea of container which incorporates
elements such as memory, cpu, disk, network etc.
Scheduling policies can be FIFO scheduler, CapacityScheduler and the
FairScheduler.
The ApplicationsManager is responsible for accepting job-submissions, and
provides the service for restarting the ApplicationMaster container on failure.
Application Manager is responsible for starting application masters in the worker
nodes.
The ApplicationManager is responsible for negotiating the first container of an
application from the resource manager.
appropriate resource containers from the Scheduler, tracking their status and
monitoring for progress.
There is one ApplicationMaster for every application running in the cluster.
ApplicationMaster can request more containers by contacting the Scheduler in
the ResourceManager
One ApplicationMaster can manage multiple application containers.
Task Tracker: There is a process called TaskTracker that runs on each DataNode.
It monitors the tasks and communicates results with a JobTracker that runs on
NameNode
Speculative Execution
A MapReduce job is dominated by the slowest task.
MapReduce attempts to locate slow tasks (stragglers) and run redundant
(speculative) tasks that will optimistically commit before the corresponding
stragglers.
This process is known as speculative execution.
Only one copy of a straggler is allowed to be speculated.
Whichever copy (among the two copies) of a task commits first, it becomes the
definitive copy, and the other copy is killed by JT
Spark
In MapReduce, the only way to share data across jobs is stable storage which is
slow!
Slow due to replication and disk I/O, but necessary for fault tolerance.
Goal is the In memory sharing - 10-100× faster than network/disk, but how to
get FT?
Resilient Distributed Datasets (RDDs):-
They are fault-tolerant in-memory collection of elements that can be operated on
in parallel.
Restricted form of distributed shared memory
- Immutable(read-only), partitioned collections of records
- Can only be built through coarse-grained deterministic transformations
(map, filter, join, …)
Efficient fault recovery using lineage
- Log one operation to apply to many elements
- Recompute lost partitions on failure – No cost if nothing fails
Despite their restrictions, RDDs can express surprisingly many parallel
algorithms.
2 ways of creating RDDs?
- parallelize an existing collection.
- reference an existing dataset in external storage system e.g. HDFS, Hbase,
etc
RDDs support two types of operations:
1. Transformations: which create a new dataset from existing one. (map, filter,
sample, groupByKey, reduceByKey, sortByKey, flatMap, union, join, cogroup,
cross, mapValues)
2. Action: returns a value to the driver program after running a computation on the
dataset. (collect, reduce, count, first, save, lookupKey)
All transformations in Spark are lazy
They do not compute their results right away. Instead, they just remember the
transformations applied to some base dataset (e.g. a file).
The transformations are only computed when an action requires a result to be
returned to the driver program.
Key-Value Pair Operations:-
1) reduceByKey(func): Combine values with the same key
2). groupByKey(): Group values with the same key
3) substractByKey(rdd): Remove element with a key present in the right RDD.
4). Join(rdd): Perform an inner join between two RDDs
5). rightOuterJoin(rdd): Perform a join between two RDDs where the key must be
present in the first RDD.
6). leftOuterJoin(rdd): Perform a join between two RDDs where the key must be
present in the right RDD.
7). cogroup(rdd): Group data from both RDDs sharing the same key.
BroadCast Variables: BroadCast Variables let programmer keep a read-only
variable cached on each machine rather than shipping a copy of it with tasks. For
example, to give every node a copy of a large input dataset efficiently.
Accumulators: Accumulators are variables that can only be added to through an
associative operation. Used to implement counters and sums, efficiently in parallel.
Spark provides three options for persist RDDs:
(1) in-memory storage as deserialized Java Objects (fastest, JVM can access RDD
natively)
(2) in-memory storage as serialized data (space limited, choose another efficient
representation, lower performance cost)
(3) on-disk storage (RDD too large to keep in memory, and costly to recompute)
RDD Fault tolerance & Recovery:
RDDs maintain lineage information that can be used to reconstruct lost
partitions.
RDDs track the graph of transformations that built them (their lineage) to rebuild
lost data.
Benefits of RDD Model • Consistency is easy due to immutability • Inexpensive
fault tolerance (log lineage rather than replicating/checkpointing data) • Locality-
aware scheduling of tasks on partitions • Despite being restricted, model seems
applicable to a broad variety of applications
RDD Dependencies:-
1) Narrow Dependencies: map, filter, join with input co-partitioned
2) Wide Dependencies: groupByKey, join with inputs not co-partitioned
Important features of Apache Spark:-
 Open-source cluster computing framework
 Developed to provide real-time, low latency queries on data that is stored in a
cluster, such as Hadoop
Uses partitioned, and distributed in-memory datasets, known as Resilient
Distributed Datasets (RDD) to speed up computation.
Disk I/O, which is the limiting factor in case of traditional MapReduce
algorithms, is avoided by using RDDs
Runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x
faster on disk.
 Uses lazy evaluation for efficient processing
RDDs are immutable i.e. they cannot be updated once created
Spark core is the base engine for computation.
ML - In the next iteration of the training phase, the decision tree will help correct
feature extractors available in Spark MLlib for converting text to numerical for previous iteration's mistakes.
vectors: CountVectorizer, Word2Vec, FeatureHasher, TF-IDF - The Log Loss is an appropriate loss function for a classification task.
Feature Tranformation Algorithms in ML Lib applied on text data: - GBTs iteratively train an ensemble of decision trees in order to minimize a loss
Tokenizer, StopWordsRemover, n -gram, Binarizer, PCA, PolynomialExpansion, function.
Discrete Cosine Transform (DCT), StringIndexer, IndexToString, OneHotEncoder, - GBT can handle non-linearities and feature interactions in the data.
VectorIndexer, Interaction, Normalizer, StandardScaler, RobustScaler,
MinMaxScaler, MaxAbsScaler, Bucketizer, ElementwiseProduct,
SQLTransformer, VectorAssembler, VectorSizeHint, QuantileDiscretizer, Imputer.
Feature Hasher: work on categorical or numerical variables, Output lower MapReduce Structured & Unstructured Data
dimensionality feature vector, "MurmurHash 3" hash algorithm used. Structured data (Processing relational data with MapReduce)
TF-IDF Feature Extractor: Unstructured data (Basics of indexing and retrieval, Inverted indexing in
- For computing TF, either HashingTF or CountVectorizer can be used. MapReduce)
- The size of the output of the HashingTF transformer is determined by the
numFeatures parameter. Boolean Retrieval
- Two separate transformers need to be used to compute the final value. Users express queries as a Boolean expression (AND, OR, NOT)(Can be
- Inverse document frequency (IDF) is a numerical measure of how much arbitrarily nested)
information a term provides. Retrieval is based on the notion of sets (Any given query divides the collection
Tokenizer: Tokenization is the process of taking text (such as a sentence) and into two sets: retrieved, not-retrieved)(Pure Boolean systems do not define an
breaking it into individual terms (usually words). ordering of the results)
StopWordsRemover takes as input a sequence of strings (e.g. the output of a
Tokenizer) and drops all the stop words from the input sequences. Ranked Retrieval
n-gram is a sequence of n tokens (typically words) for some integer n . The Order documents by how likely they are to be relevant (Estimate
NGram class can be used to transform input features into n -grams. relevance(q, di))(Sort documents by relevance)(Display sorted results)
Binarizer: Binarization is the process of thresholding numerical features to
binary (0/1) features.
IndexToString maps a column of label indices to a column containing the
original labels as strings.
StringIndexer can encode a column containing categorical values that are of
string datatype into another column of label indices.
StandardScaler transforms a dataset of Vector rows, normalizing each feature
to have unit standard deviation and/or zero mean.
OneHotEncoder converts categorical variables into binary vectors with a single
one in each element.
VectorIndexer can automatically identify the categorical features in a dataset of
vectors.
RobustScaler transforms a dataset of Vector rows, removing the median and Term Weighting
scaling the data according to a specific quantile range (by default the IQR: Term weights consist of two components
Interquartile Range, quantile range between the 1st quartile and the 3rd quartile). - Local: how important is the term in this document?
VectorAssembler is a transformer that combines a given list of columns into a - Global: how important is the term in the collection?
single vector column. It is useful for combining raw features and features generated Terms that appear often in a document should get high weights
by different feature transformers into a single feature vector, in order to train ML Terms that appear in many documents should get low weights
models like logistic regression and decision trees. How do we capture this mathematically?
Normalizer is a Transformer which transforms a dataset of Vector rows, - Term frequency (local)
normalizing each Vector to have unit norm. It takes parameter p, which specifies - Inverse document frequency (global)
the p-norm used for normalization. (p=2 by default.).
- If the input vector is [2.0, 1.0, 1.0] and value of p = 1, the output vector would
[0.5,0.25,0.25].
- It takes a parameter p which specifies the value of the norm.
- It transforms a dataset of Vector rows into another column of vectors having
unit norm.
logistic regression model:
regParam decreases  Overfitting increases
regParam increases  Underfitting increases
DataFrame
LinearRegression Algorithm: DataFrames are part of Spark SQL.
- By default the regParam = 0 i.e. there is no regularization. Like RDDs, DataFrames (DF) are immutable, distributed, partitioned
- There are two steps involved in model training: collection of data
1. Defining the algorithm and specifying hyperparameters They have all the properties of RDDs, such as lazy evaluation, recovery
2. fitting the algorithm on the training data. through lineage graphs, etc.
- It can work with various types of regularizers, such as L1 and L2. They contain specialized APIs for working with tabular data, and have
- The aim of the algorithm is to minimize the specified loss function, with named columns.
regularization.
Gradient-Boosted Trees (GBTs) algorithm: