ELK SETUP

CONTENTS

1.) What is ELK Stack

 Overview of all components
2.) Architecture
 Architectural overview for testing purpose.
3.) Kafka
 Kafka overview and installation
 Kafka configuration
 Starting Kafka services
 Testing Kafka for failover

4.) Filebeat

 Filebeat installation
 Filebeat configuration
 Starting filebeat services

5.) Elasticsearch

 Elasticsearch installation
 Elasticsearch configuration
 Starting Elasticsearch services

6.) Logstash

 Logstash installation
 Logstash configuration (Defining pipeline)
 Starting Logstash services
7.) Datadog Setup to monitor ELK setup
 Datadog overview
 Agent installation on all nodes
 Setting-up monitoring for Elasticsearch
 Setting -up monitoring for Logstash
 Setting-up monitoring for Kafka
What is ELK setup:

The ELK Stack is a collection of three open-source products — Elasticsearch, Logstash, and Kibana —
all developed, managed and maintained by Elastic.
Elasticsearch is a NoSQL database that is based on the Lucene search engine.
Logstash is a log pipeline tool that accepts inputs from various sources, executes different
transformations, and exports the data to various targets.
Kibana is a visualization layer that works on top of Elasticsearch.

Architecture:

2
Kafka
Prerequisites:

 Linux machine (Ubuntu16.04)

 Java- JDK 1.8..0

Install java using below cmd.

# sudo apt-get install openjdk-8-jdk

Check java version:

# Java -version

) Download kafka binaries

# wget http://mirrors.fibergrid.in/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz

) Untar it

Zookeeper configuration:

) On each node create a zookeeper directory and a file ‚myid‘ with a unique number

) Change the myid value according to your node.

) On all three Server go to Kafka home folder /home/sigmoid/kafka_2.11-2.0.0 and setup zookeeper
like this.

3
) Start zookeeper on all three server:

# bin/zookeeper-server-start.sh config/zookeeper.properties

) Change the Kafka server. Properties on all three servers (set a unique broker id on each server).

4
) Start kafka on all three nodes.

# bin/kafka-server-start.sh config/server.properties

) Verify also all brokers are registered to zookeeper:

# bin/zookeeper-shell.sh 167.99.221.43:2181 ls /brokers/ids

) Create a test_lab Topic with three partitions and replicationfactor 3

# bin/kafka-topics.sh --create --zookeeper 167.99.221.43:2181 --topic lab_test --partitions 3 --

replication-factor 3

) Start a Producer on one node:

# bin/kafka-console-producer.sh --broker-list
167.99.221.43:9093,174.138.9.151:9093,167.99.221.40:9093 --topic lab_test

) Start also a Consumer on a different node:

# bin/kafka-console-consumer.sh --zookeeper 167.99.221.43:2181 --topic example-topic --from-

beginning

) Write some text in the producer console. You should then see the Text on the Consumer Console

) Stop a node and write again some messages in the producer console to verify the high availability
is working.

5
Filebeat
Install File (Here we are going with binaries)

# wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz

) Untar it in home directory of user or in other directory.

) Change the configuration file to start collecting intended logs. Say /var/sys/log. We can add as
many directory to monitor logs file for example: app log, web server logs,etc.

) Go to the location of binary file where filebeat.yml is stored.

)Change the configuration as below:

) Scroll down and configure output section. Here we want to send output to Kafka. Please mention
the Kafka server details.

6
) Start filebeat service on each node to start sending data to kafka cluster

# ./filebeat -e -c filebeat.yml -d "publish"

Elasticsearch
) Download binaries from wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-
6.4.2.tar.gz

)Untar it on all three nodes

) Before starting the services and making changes in config file. Please change few OS properties.

) In ubuntu higher than 14.04 ,there is no need to change the Limit in security.conf file.However
below version of ubuntu will throw error to change this limit.

) Increase the size of virtualmemory to 262144

) To make this setting permanent edit the file /etc/sysctl.conf and append the parameter
vm.max_map_count=262144 to the end of this file.

) Reload the configuration by issuing command # sudo sysctl -p

) Now start configuring cluster by making changes to config file

) Go to elastic search home directory and under config dir start making changes as below.

) Change the node and cluster properties. Give a cluster name

) Change this property on all nodes

7
) Add the list of server in cluster to start discover all nodes.

) Exit from the file and start the Elasticsearch on all three nodes.

#bin/elasticsearch

) Test whether Elasticsearch has started properly or not by firing the below command and you will
get response as below:

8
Logstash
Logstash is an open source, server-side data processing pipeline that injest data from a multitude of
sources simultaneously, transform it and then send it to specific output

It’s a data collection engine. At its core, Logstash is a form of Extract-Transform-Load (ETL) pipeline.
Unstructured log data is extracted, filters transform it, and the results are loaded into some form of
data store.

) Download binary from # wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.tar.gz

) Go to Logstash home directory and then to config directory.

) Copy logstash-sample.conf to logstash.conf and edit the file as below.

) Feel free to make changes as per your need.

) Start the logstash service

# bin/logstash -f config/logstash.conf

) Now start the kafka consumer and provide the topic name to check the whether the logs has
started consumed and getting directed to the topic.

# bin/kafka-console-consumer.sh --bootstrap-server <server Ip>:9092 --from-beginning –topic <name>

9
Kibana
) Download binaries for kibana from wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-
linux-x86_64.tar.gz

) Change config file kibana.yml and edit parameter to point to your elastic search cluster.

) Start the kibana.

# bin/kibana

) Go to kibana and create index as mentioned in logstash.conf pipeline.

10
) Go to Discover tab and logs will start populating there.Use search and filters to get the desired log
output

DATADOG
The Datadog Agent is lightweight piece of software that runs on your hosts. Its job is to faithfully collect
events and metrics and bring them to Datadog on your behalf so that you can do something useful with
your monitoring and performance data.

Agent Architecture:

The agent is composed of 4 major components, each written in Python running as a separate process:

 Collector (agent.py) - The collector runs checks on the current machine for whatever integrations
you have, it captures system metrics like memory and CPU.
 Dogstatsd (dogstatsd.py) - This is a StatsD backend server, it's responsible for aggregating local
metrics sent from your code
 Forwarder (ddagent.py) - The forwarder is pushed data from both dogstatsd and the collector and
queues it up to be sent to Datadog.
 SupervisorD This is all controlled by a single supervisor process. We keep this separate so you don’t
have to have the overhead of each application if you don’t want to run all parts (though we generally
recommend you do).

11
Installation of Data-dog step by step on Ubuntu Machine:
) Set up apt so that it can download through https:

# sudo apt-get update

# sudo apt-get install apt-transport-https

) Set up the Datadog deb repo on your system and import Datadog's apt key:

# sudo sh -c "echo 'deb https://apt.datadoghq.com/ stable 6' > /etc/apt/sources.list.d/datadog.list"

# sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 382E94DE

) Update your local apt repo and install the Agent:

# sudo apt-get update

# sudo apt-get install datadog-agent

) Copy the example config into place and plug in your API key

# sudo sh -c "sed 's/api_key:.*/api_key: efd1da1a3cf743d76b141a0365c79794/' /etc/datadog-

agent/datadog.yaml.example > /etc/datadog-agent/datadog.yaml"

) Start the agent on all the nodes

# sudo datadog-agent run

) Go to datadog agent dashboard and you will see all metrics related to your infrastructure only
server metrics.

12
Monitor Elastic-search with data dog
) Datadog agent comes with elastic.d config.Go to /etc/datadog-agent/conf.d

) Edit the elastic.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration
directory to start collecting your Elasticsearch metrics and logs.

) Add this configuration block to your elastic.yaml file to start gathering your ElasticSearch
metrics:

) Restart the agent

Note: If you're collecting Elasticsearch metrics from just one Datadog Agent running outside the cluster -
e.g. if you use a hosted Elasticsearch - set cluster_stats to true.

13
) Go to data-dog dashboard and there will be preset elasticsearch dashboard.Click on it and you
will be presented with all metrics.

14
15
16