Mohamed Aziz Zouaghia SR-A

TP SPARK
Start Hadoop containers:

Enter Master Container Start YARN and HDFS Daemons:

Check Daemon Status for the master container:

Check Daemon Status for the slave container:

Create a text file file1.txt on the master node:

Load the file into HDFS:

Mohamed Aziz Zouaghia SR-A

Check Spark installation:

Test Spark with a Scala code snippet:

Download the result directory file1.count from HDFS:

Check the result:

Mohamed Aziz Zouaghia SR-A

Spark Batch in Java:

Project Setup:

Add dependencies to the pom.xml file:

Mohamed Aziz Zouaghia SR-A

Create a package tn.isetcom.tp21 under the java directory with a class named
WordCountTask inside of it:

Insert loremipsum.txt in the src/main/resources directory:

Create a run configuration:

Mohamed Aziz Zouaghia SR-A

After running the configuration a directory out created under resources containing
two files: part-00000 and part-00001:

Part-00000:

Part-00001:
Mohamed Aziz Zouaghia SR-A

Cluster Execution:
Modify the WordCountTask class for cluster execution:

Run the commands package install:

The file wordcount-1.0-SNAPSHOT.jar created under target directory:

Copy the JAR file to the Hadoop master container:

At the master container:

Mohamed Aziz Zouaghia SR-A

Submit the Spark job in local mode:

After the job completes the output directory created and contains result file:

Part-00000:

Part-00001:

Launch the Spark job on YARN in cluster mode:

Mohamed Aziz Zouaghia SR-A

After the job completes the output2 directory created and contains result file:

Spark Streaming:
Create a new Maven project:

Pom.xml configuration:

Create a class Stream in the package tn.isetcom.tp22:

Mohamed Aziz Zouaghia SR-A

Local Testing:
Execute the Stream class:

Cluster Execution:
Modify the Stream class for cluster streaming:

Run mvn package install to create the JAR file:

Copy the JAR file to the Hadoop container:

Mohamed Aziz Zouaghia SR-A

Launch the Spark streaming job on the cluster:

Observe the Result: