MINI PROJECT ON BIGDATA

Contents
 Prerequisites before Initiate HDFS file operation.
 $start-dfs.sh (To Start all Daemons)
 Jps (To check for all Daemons working)

1)Basic HDFS File Operation.

 put Command (File import from Local file system to Hdfs)
 get Command (File import from Hdfs to Local file system)
 cp Command (Copy file from one directory to other Hdfs
directory)
 mv command (move file from one directory to destination
Hdfs directory )

2)Sqoop Commands
 Sqoop import command.
 Sqoop import with Where clause command.
 Sqoop export command
 Sqoop Incremental append.
 3) Hive Commands
 Internal/Managed table Creation in Hive
 External table Creation in Hive
 Loading data from Local file system to Hive
 Static partitioning in Hive
 Dynamic partitioning in Hive
 Bucketing in Hive

HDFS File Operation:-

 put Command
Loading file from Local file system to specific directory in HDFS

 get Command
-getcommand is used to copy data from hadoop system to local file
system, it will copy the data from hdfs stored directories to local file
system we can do the same by using copyToLocal command .

 cp Command
It will copy file from one directory of HDFS file system to destination
directory in HDFS file system itself.
 mv command

By using move command the File1.txt in wep directory will be moved to the new directory
/user/sumit .
 Sqoop Command
 Sqoop import command.
RDBMS-HDFS

The above Sqoop command will copy the file from local database which is in
localhost and in table student will be copied into Sqoop data directories in hdfs,
here first we create connections with jdbc and mysql and then data will be copied
to hdfs part file in directories.
Sqoop import with Where clause command.
 Sqoop export command
HDFS-RDBMS

 Sqoop Incremental append.

The command is used to load data from local database to hdfs in incremental
manner that means by looking at the last value of check column data will be
 loaded to hdfs , the data after the specified value of column will be loaded

Hive Commads:
 Internal table creation
  External table creation

By creating the external table it helps in the way as if we will drop the
external table then the table will be deleted (Metadata) but the data
associated with the table (Actual data) will remain there in the
warehouse directories of hive .

 Loading data from Local file system to Hive

STEPS TO DO PARTITIONING
Data can be stored either two ways one with
Internal/managed table or External table
Step -1: Create non partition table( Internal/External)
Step-2: Loading data into Created table
Step-3: Create Partition table
Step -4: For Dynamic Partioning Set Property
For Static its not needed
Step-5 : Loading data into Partition table

 Static partitioning in Hive

static partitioning, where you explicitly specify partition column and that
column corresponding directory will be created in hive/warehouse
directory.
 Dynamic partitioning in Hive
Unlike static partitioning, where you explicitly specify partition values,
dynamic partitioning lets Hive determine these values automatically based
 on the data itself. And separate directory will be created implicitly in
hive/warehouse directory

 Bucketing in Hive

Bucketing is based on the hashing technique.

For a given column value, calculate the modulo of that value with the
number of required buckets (let’s say, F(x) % 3).

Based on the resulting value, store the data into the corresponding bucket.
Data is distributed evenly between corresponding buckets.