Big Data & Hadoop Class Syllabus

 BigData Introduction and Hadoop o Mapper

o Reducer
Fundamentals
o Driver program
o Data Storage and Analysis
o How to package the job
o Comparison with RDBMS
o MapReduce WebUI
 Hadoop – A Brief History o How MapReduce Job run?
 MapReduce – Part1 o Shuffle & Sort
o Map and Reduce o Speculative Execution
o Sample Program  InputFormats
o Combiner o Input Splits and Record Reader
o Practitioners and Custom Partitioned o Default Input Formats
 Hadoop Streaming & Pipes o Implement Custom Input Format
 HDFS  OutputFormats
o Blocks o Default Output formats
o NN & DN o Output Record Reader
o HDFS Federation & High Availability  Compression
 HDFS Clients o Map Output
o HDFS Command Line o Final Output
o HDFS CLI – File System
S Operations Lab o Splittable vs Non Splittable
o HDFS Web UI o Compression Codecs
o HDFS Java Client  Serialization
o HDFS Java Client – File System o Data types –default
Operations Lab o Writable vs Writable Comparable
o CRUD Operations using Java Client
o Custom Data types – Custom
o Anatomy of File Read and File Write
Writable/Comparable
o DistCp
 File Based Data structures
o Cluster balancing
o Sequence file
 YARN – Cluster Management o Reading and Writing into Sequence file
(Hadoop 2.x) o Map File
o How Yarn Applications run?  Tuning MapReduce Jobs
o YARN vs MapReduce
 Advanced MapReduce
o YARN Scheduling
o Counters
 Capacity Scheduler
 Built-In Counters Classification
 Fair Scheduler
 User Defined Counters
 FIFO Scheduler
o Sorting
 Map Reduce – Part2  Partial Sort
o Env Setup
 Total Sort
o Tool and ToolRunner  Secondary Sort

Page 1
 Big Data & Hadoop Class Syllabus
o Joins
 Map-side joins
 Reduce-side joins
 Distributed Cache
 Hive
o Comparison with RDBMS
o HQL
o Data types
o Tables
o Importing and Exporting
o Partitioning and Bucketing – Advanced.
o Joins and Join Optimization.
o Functions- Built in & user defined
o Advanced Optimization of HQL
o Storage File Formats – Advanced
o Loading and Storing Data
o SerDes – Advanced
 Sqoop
o Important basics
o Import – Deep dive
o Export – Deep dive
o Sqoop Optimization – Incremental Load
o Many more
 PIG
o Important basics
o Pig Latin
o Data types
o Functions – Built-in, User Defined
o Loading and Storing Data
 Flume
o Configure Flume and Import data
o Architecture and LAB
 Oozie
o Different workflow jobs
o Ooze scheduler.
o LAB – covers advanced topics
 HBase
o NoSQL databases Introduction
Page 2
o CAP theorem
o HBase Architecture
o HBase Clients – Java Client
o Loadling Data
o UDF,UDAF,UDTFs
 Zookeeper
o Zookeeper in HBase
o How Zookeeper is used in Production
 Ambari
o Real time Cluster deployment Using
Ambari
o Monitoring the Cluster
 REST API
o Introduction
o Real time Use cases of How REST is used
with Hadoop
 Labs:
o Real Time use cases and Data sets
covered (10+ Real Time datasets)
o Word count, Sensors(Weather
s
Sensors)Dataset, Social Media data sets
like YouTube, Twitter data analysis,
o Jav and Unix Basics Lab
o Hadoop, Hive, Sqoop, Oozie, HBase,
Flume Installations –Pseudo&Cluster
 Master Project:
o Real-time DataWarehouse migration:
o Real-time concepts covered are
 Hive - Advanced topics
 Sqoop import/export
 Oozie Scheduling
 How Hadoop MR used in DW
 RDBMS concepts
 ETL tool concepts
 Integration with Reporting tools