--Practica Clase 8

sudo apt update

sudo apt upgrade

sudo apt install unrar

git clone https://github.com/KumarRoshandot/AirFlow_Kafka_Spark_Docker
cd Project_Flight_Docker2
cd data
unrar e Extract_2_json_Files.rar
cd ..
sudo apt upgrade pip
sudo docker-compose build
sudo docker-compose up

1) When the Airflow server is up and running (localhost:8080)

2) Open COmmand Prompt , enter following commands:-
-> sudo docker container ls
-> Look for Image and name of which has name airflow_container , let say
image_airflow
-> sudo docker exec -ti airflow_container bash ( by this you have entered
into airflow_container
-> echo $JAVA_HOME
-> if it shows java version then well and good else export java_home with
below command
-> export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
-> source ~/.bashrc

3) Hive setups after point 1 and 2.( Hive meta store db needs to be started ,so )
a) Before you run hive for the first time, run

-> schematool -initSchema -dbType derby

b) If you already ran hive and then tried to initSchema and it's failing:

-> cd /data/hive/
-> mv metastore_db metastore_db.tmp

c) Re run
-> schematool -initSchema -dbType derby

d) Run /opt/apache-hive-2.0.1-bin/bin/hiveserver2 --service metastore

e) NOw trigger the DAG from Airflow UI

f) OPen another command prompt

-> docker exec -ti airflow_container bash
-> cd /opt/apache-hive-2.0.1-bin/bin
-> chmod 777 hive
-> hive -
-> Now you are in hive shell

sudo docker-compose down