SPARK TASKS :

1) CONNECT SPARK WITH JUPYTER NOTEBOOK :

Using Jupyter Notebook Kernel called Sparkmagic to send the code to remote cluster after installing the
Livy on the remote spark clusters. Livy is an open-sou

rce Apache licensed REST Web Service for managing long running Spark Contexts and submitting Spark
Jobs.

In house Spark Clusters can be installed using Apache Ambari.

Issues Faced

1): Installation ,Connection and Authentication

The installation issues can be resolved by using Docker Containers provided by SparkMagic.

2)