INCEPTEZ TECHNOLOGIES Installation guide

1. Install spark

cd /home/hduser/install

tar xvzf spark-1.6.0-bin-hadoop2.6.tgz

sudo mv spark-1.6.0-bin-hadoop2.6 /usr/local/spark

2. Create a log directory

mkdir -p /home/hduser/sparkdata/logs

3. Create tmp directory for spark

mkdir -p /home/hduser/sparkdata/tmp

4. Configure Spark Copy spark/conf/spark-env.sh.template:

cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh

Now edit spark/conf/spark-env.sh and specify the location of Hadoop configuration directory and a
YARN job queue where you have permissions to submit jobs:

vi spark-env.sh

export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export YARN_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_LOG_DIR=/home/hduser/sparkdata/logs
export SPARK_WORKER_DIR=/home/hduser/sparkdata/tmp
export SPARK_WORKER_INSTANCES=1
export SPARK_MASTER_MEMORY=400m
export SPARK_EXECUTOR_MEMORY=400m
export SPARK_WORKER_MEMORY=400m
export SPARK_WORKER_CORES=2
export SPARK_EXECUTOR_CORES=1

Inceptez Technologies| No.27 A, Brahmin street, Velachery, Chennai -600042

1
Contact No. +91 7871299810 | Email : Info@inceptez.com | www.inceptez.com
 5. To Start Spark Services

start-master.sh

6. Spark Workers/Executors

start-slaves.sh

7. Web UI:
localhost:8080

8. Run Spark

To start Pyspark
pyspark

To Start Scala Spark cli

Spark shell

To start in YARN mode

cd /usr/local/spark/conf/
cp -p spark-defaults.conf.template spark-defaults.conf

spark-shell --master yarn-client

Inceptez Technologies| No.27 A, Brahmin street, Velachery, Chennai -600042

2
Contact No. +91 7871299810 | Email : Info@inceptez.com | www.inceptez.com