Saad Khan

MSCS
2nd semester
Content

 Introduction
 What is Big Data.
 Characteristic of Big Data.
 Storing, Selecting and processing of Big Data
 Why Big Data
 How it is Different
 Hive
 Pig
 Flume
Introduction

 Big Data may well be the Next Big Thing in the IT world

 The first organizations to embrace it were online and startup firms. Firms like
Google, eBay, LinkedIn, and Facebook were built around big data from the
beginning.

 Big data burst upon the scene in the first decade of the 21st century
 What is BIG DATA?
 ‘Big Data’ is similar to ‘small data’, but bigger in size.

 An aim to solve new problems or old problems in a

better way
What is BIG DATA (Cont..)

 Walmart handles more than 1 million customer transactions every hour.

 Facebook handles 40 billion photos from its user base.

Characteristic of Big DATA
Volume

 A typical PC might have had 10 gigabytes of storage in 2000

 Today, Facebook ingests 500 terabytes of new data every day

 Boeing 737 will generate 240 terabytes of flight data during a single
flight across the US.
Velocity

 Clickstreams and ad impressions capture user behavior at millions of events

per second.

 High-frequency stock trading algorithms reflect market changes within

microseconds

 Machine to machine processes exchange data between billions of devices

Variety

 Big Data isn't just numbers, dates, and strings. Big Data is also geospatial
data, 3D data, audio and video, and unstructured text, including log files
and social media.

 Traditional database systems were designed to address smaller volumes of

structured data, fewer updates or a predictable, consistent data structure
Storing Big Data

 Selecting data source for analysis

 Eliminating redundant data

 Establishing the role of NoSQL

Selecting Big Data Stores

 Choosing the correct data stores based on your data characteristics.

 Moving code to data.

 Implementing polyglot data store solutions

Processing Big Data

 Mapping data to the programming framework

 Connecting and extracting data from storage.

 Transforming data for processing.

Why Big Data

 Increase of Storage capacities.

 Increase of processing.

 Availability of data(different data types).

How is big data different?

 Automatically generated by a machine

(e.g. Sensor embedded in an engine)

 Typically an entirely new source of data

(e.g. Use of the internet)

 Not designed to be friendly

(e.g. Text streams)
Hive
What is Hive?

 Hive is a data warehouse infrastructure tool to process structure data in

Hadoop. It resides on top of Hadoop to summarize Big Data, and makes
querying and analyzing easy

 Initially Hive was developed by Facebook, later the Apache Software

Foundation took it up and developed it further as an open source under
the name Apache Hive.
Feature of Hive

 It stores Schema in a database and processed data into HDFS(Hadoop

Distributed File System).

 It is designed for OLAP.

 It provides SQL type language for querying called HiveQL or HQL.

Architecture Of Hive

 User Interface - Hive is a data warehouse infrastructure software that can

create interaction between user and HDFS. The user interfaces that Hive
supports are Hive Web UI, Hive command line, and Hive HD.

 Meta Store -Hive chooses respective database servers to store the schema
or Metadata of tables, databases, columns in a table, their data types and
HDFS mapping.
Architecture Of Hive(Cont..)
Architecture Of Hive(Cont..)

 HiveQL Process Engine- HiveQL is similar to SQL for querying on schema info
on the Megastore. It is one of the replacements of traditional approach for
MapReduce program

 HDFS or HBASE - Hadoop distributed file system or HBASE are the data
storage techniques to store data into the file system.
Working of Hive

 Get Plan- The driver takes the help of query complier that parses
the query to check the syntax and query plan or the requirement of
query.

 Get Metadata- The compiler sends metadata request to Megastorez

 Send Metadata- Metastore sends metadata as a response to the

compiler.
Working of Hive(Cont..)

 Send Plan- The compiler checks the requirement and resends the plan to
the driver. Up to here, the parsing and compiling of a query is complete.

 Execute Plan- the driver sends the execute plan to the

execution engine.
Pig
What is Pig?

 A platform for analyzing large data sets that consists of a high-level

language for expressing data analysis programs

 Compiles down to MapReduce jobs

 Developed by Yahoo!
Pig Component

 Two Main Components.

 High Level Language (Pig Latin)
 Set of Commands
 Two Execution Modes
 Local: Read/Write to local file system
 MapReduce: connects to Hadoop cluster and reads/write to HDFS
Why Pig?

 Common design patterns as key word (joins, distinct, counts)

 Data flow analysis

 Avoid java level errors

Language Feature Pig

 Keywords
 Load, Filter, For each Generate, Group By, Store, Join, Distinct, Order by,…

 Aggregations
 Count, Avg, Sum, Max, Min

 Schema
 Defines at query-times not when files are loaded
Flume
What is flume?

 Apache Flume is a tool/service/data ingestion mechanism for collecting

aggregating and transporting large amounts of streaming data such as log
files, events (etc...) from various sources to a centralized data store

 Flume is a highly reliable, distributed, and configurable tool. It is principally

designed to copy streaming data (log data) from various web servers to
HDFS.
Flume Architecture

 Flume Event
 An event is the basic unit of the data transported inside Flume.

 Flume Agent.
 Take a look at the following illustration. It shows the internal components of an
agent and how they collaborate with each other.
Application of Flume

 Assume an e-commerce web application wants to analyze the customer

behavior from a particular region.

 To do so, they would need to move the available log data in to Hadoop for
analysis. Here, Apache Flume comes to our rescue.

 Flume is used to move the log data generated by application servers into
HDFS at a higher speed
Feature of flume

 Flume ingests long data from multiple web serves into a centralized store

 Using flume, we can get the data from multiple servers immediately into
Hadoop.

 Flume supports a large set of sources and destinations types

 Flume can be scaled horizontally.

Advantages of flume

 Using apache flume we can store the data in to any of the centralized
stores (Hbase, HDFS).

 Flume provides the feature of contextual routing.

 Flume is reliable, fault tolerant, scalable, manageable, and customizable

Any Question