Single Node Cluster

Set up

Copyright 2021@Learn-Spark.info - All Rights Reserved 1

Subscribe at Learn-Spark.info
Learn-Spark.info
Sibaram Kumar
Sr Data Engineer | Spark | Big Data
Linkedin: in/SibaramKumar/
Copyright 2021@Learn-Spark.info - All Rights Reserved
Subscribe at Learn-Spark.info 3
3 Modes to Set up
Hive Metastore

Copyright 2021@Learn-Spark.info - All Rights Reserved 4

Subscribe at Learn-Spark.info
 Mode #1

Embedded Derby Metastore :

Localhost

Hive Service JVM

Drive Metastore Derby

Copyright 2021@Learn-Spark.info - All Rights Reserved 5

Subscribe at Learn-Spark.info
Local Metastore : Mode # 2

Localhost

Hive Service JVM Postgress Service JVM

Drive Metastore

Postgress
Drive Metastore
//localhost:6432/metastore

Copyright 2021@Learn-Spark.info - All Rights Reserved 6

Subscribe at Learn-Spark.info
Remote Metastore: Mode #3

Localhost Different Machine(1.2.3.4)

Hive Service JVM

Metastore
Drive
Server JVM
Postgress
Metastore
Drive
Server JVM
/432/metastore //1.2.3.4:6432/metastore

Copyright 2021@Learn-Spark.info - All Rights Reserved 7

Subscribe at Learn-Spark.info
 Localhost

Hive
Metastore

PostgreSQL
Image

PostgreSQL Client

Copyright 2021@Learn-Spark.info - All Rights Reserved 8

Subscribe at Learn-Spark.info
Set up Web UI:
1. Run a job using spark-submit Command.
2. Get the Port when the applications are submitted. <8088>
3. Set up the Firewall to allow port 8088 at our instance.
4. Open a Browser and test. <PublicIP>:8088
5. At this time, we are still unable to access the Application History.
6. Go to History URL and check the Port for Application History. <18080>
7. Also Initiate the Service : /opt/spark3/sbin/start-history-server.sh
8. Allow port 18080 in the Firewall at our instance.
9. Access the Application History. <PublicIP:18080>

Copyright 2021@Learn-Spark.info - All Rights Reserved 9

Subscribe at Learn-Spark.info