1.
Installing and Configuring Tools
1.1 Install Apache NiFi & Airflow:
1. Download Apache NiFi and Airflow from official websites.
2. Install dependencies (Java for NiFi, Python for Airflow).
3. Configure NiFi by editing nifi.properties.
4. Configure Airflow by initializing the database (airflow db init).
5. Start NiFi (nifi.sh start) and Airflow Scheduler/Webserver (airflow scheduler
& airflow webserver).
6. Verify installations by accessing NiFi at http://localhost:8080 and Airflow at
http://localhost:8081.
1.2 Install Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4:
1. Download and install Elasticsearch and Kibana.
2. Start Elasticsearch and Kibana services.
3. Verify Elasticsearch (http://localhost:9200) and Kibana
(http://localhost:5601).
4. Install PostgreSQL and pgAdmin 4.
5. Create databases and users in PostgreSQL using pgAdmin.
Here's a step-by-step guide for installing and configuring Apache NiFi, Airflow,
Elasticsearch, Kibana, PostgreSQL, and pgAdmin 4 on Windows 11:
✅ 1. Installing and Configuring Tools on
Windows 11
⚡ 1.1 Install Apache NiFi & Airflow
Step 1: Install Apache NiFi
✅ 1.1.1. Install Java (NiFi Dependency):
1. Download Java JDK 11 or later from Oracle or AdoptOpenJDK.
2. Install Java and set JAVA_HOME:
o Open System Properties → Environment Variables.
o Add new variable:
ini
CopyEdit
JAVA_HOME = C:\Program Files\Java\jdk-<version>
o Add %JAVA_HOME%\bin to the Path variable.
3. Verify installation:
bash
CopyEdit
java -version
✅ 1.1.2. Install Apache NiFi:
1. Download NiFi from Apache NiFi Downloads.
2. Extract the ZIP file to C:\nifi.
3. Configure NiFi:
o Navigate to:
makefile
CopyEdit
C:\nifi\conf\nifi.properties
o Set the following:
properties
CopyEdit
nifi.web.http.host=localhost
nifi.web.http.port=8080
4. Start NiFi:
bash
CopyEdit
C:\nifi\bin\nifi.bat start
5. Verify:
o Visit: http://localhost:8080
Step 2: Install Apache Airflow
✅ 1.1.3. Install Python (Airflow Dependency):
1. Download and install Python 3.8+ from Python.org.
2. Ensure pip is installed and added to Path.
3. Verify:
bash
CopyEdit
python --version
pip --version
✅ 1.1.4. Install Airflow Using pip:
1. Open Command Prompt or PowerShell.
2. Create a virtual environment (optional but recommended):
bash
CopyEdit
python -m venv airflow_env
airflow_env\Scripts\activate
3. Set Airflow constraints and install:
bash
CopyEdit
set AIRFLOW_VERSION=2.8.1
set PYTHON_VERSION=3.8
set
CONSTRAINT_URL=https://raw.githubusercontent.com/apache/airflow/const
raints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt
pip install "apache-airflow==%AIRFLOW_VERSION%" --constraint
%CONSTRAINT_URL%
✅ 1.1.5. Initialize Airflow Database:
bash
CopyEdit
airflow db init
✅ 1.1.6. Start Airflow Services:
1. Start Scheduler:
bash
CopyEdit
airflow scheduler
2. Open New Terminal and Start Webserver:
bash
CopyEdit
airflow webserver -p 8081
3. Verify:
o Visit: http://localhost:8081
⚡ 1.2 Install Elasticsearch, Kibana, PostgreSQL, and
pgAdmin 4
Step 3: Install Elasticsearch & Kibana
✅ 1.2.1. Install Elasticsearch:
1. Download Elasticsearch from Elastic Downloads.
2. Extract ZIP to C:\elasticsearch.
3. Configure Elasticsearch (C:\elasticsearch\config\elasticsearch.yml):
yaml
CopyEdit
network.host: localhost
http.port: 9200
4. Start Elasticsearch:
bash
CopyEdit
C:\elasticsearch\bin\elasticsearch.bat
5. Verify:
o Visit: http://localhost:9200
✅ 1.2.2. Install Kibana:
1. Download Kibana from Elastic Downloads.
2. Extract ZIP to C:\kibana.
3. Configure Kibana (C:\kibana\config\kibana.yml):
yaml
CopyEdit
server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]
4. Start Kibana:
bash
CopyEdit
C:\kibana\bin\kibana.bat
5. Verify:
o Visit: http://localhost:5601
Step 4: Install PostgreSQL & pgAdmin 4
✅ 1.2.3. Install PostgreSQL:
1. Download PostgreSQL from PostgreSQL Downloads.
2. Run the installer and follow steps:
o Set a password for the postgres user.
o Default port: 5432
3. Verify installation:
bash
CopyEdit
psql -U postgres
✅ 1.2.4. Install pgAdmin 4:
1. Download from pgAdmin Downloads.
2. Install and launch pgAdmin.
3. Connect to PostgreSQL using:
o Host: localhost
o Port: 5432
o Username: postgres
o Password: (set during installation)
✅ 1.2.5. Create Databases & Users in pgAdmin:
1. In pgAdmin, right-click on Databases → Create → Database.
2. Create a new user:
o Navigate to Login/Group Roles → Create.
o Set permissions and assign a password.
