Grafana and AWS - Implementation and Usage

Table of Contents
Grafana and AWS - Implementation and Usage ....................................................................... 1
What is Grafana?......................................................................................................................... 2
Exploring metrics and logs ......................................................................................................... 2
Grafana Implementation ............................................................................................................. 2
Adding AWS CloudWatch as Datasource in Grafana ................................................................ 4
Grafana Dashboard for Amazon AWS ECS ............................................................................... 7
Importing the Amazon AWS ECS Dashboard in Grafana.......................................................... 7
Adding load on SampleApp-Web service................................................................................. 10
Grafana Dashboard for AWS Cloudwatch Data ....................................................................... 11
Grafana CloudWatch Logs Insights Custom Queries ............................................................... 14
Grafana in Production ............................................................................................................... 16
Conclusion ................................................................................................................................ 17
References ................................................................................................................................. 17
What is Grafana?
Grafana is an open source visualization and analytics software. It allows you to query, alert,
visualize, and explore your metrics no matter where they are stored. It provides you with the
tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations.

Exploring metrics and logs

Explore your data through ad-hoc queries and dynamic drilldown. Split view and compare
different time ranges, queries and data sources side by side.

Grafana Implementation
Though there are several methods of implementing Grafana, we shall broadly look into its
installation and usage on:

 Linux
 AWS
 Docker

Implementing Grafana is relatively simple and involves few steps in the installation. Its usage is
rather complex and can involve several steps to accomplish stages from data collection, data
sources, database connections and finally querying and visualizations.

For Linux, visit Install Grafana webpage covers the most popular Linux distributions like
CentOS, Fedora, OpenSuse, RedHat and others.

For AWS, one can follow Amazon Managed Service for Grafana and Grafana Integration with
CloudWatch sections.

For Docker, it is also given on the link above in “Install Grafana” for Linux.

Grafana is only for visualizations and requires a database connection to fetch data to display
several types of graphs. There are several exercises available for monitoring and visualization.
Following are a few images taken from such exercises carried out for Grafana implementation.

|
Figure: Grafana dashboard connected to influxDB

Figure: Prometheus dashboard

Adding AWS CloudWatch as Datasource in Grafana
Run the following docker-compose.yml to start the Grafana container:

version: '3'
services:
grafana:
image: grafana/grafana
container_name: grafana
networks:
- grafana
environment:
- GF_PATHS_DATA=/var/lib/grafana
ports:
- 3000:3000
volumes:
- ./data:/var/lib/grafana

networks:
grafana: {}

# docker-compose up -d
Creating grafana ... done

This should show output as here:

[root@ns3078348 cloudwatch1]# docker container ls

CONTAINER ID IMAGE COMMAND CREATED STATUS
PORTS NAMES
a49b3eee6e74 grafana/grafana "/run.sh" About an hour ago Up About an
hour 0.0.0.0:3000->3000/tcp grafana

Access Grafana in Web UI

Server Link : http://188.165.220.196:3000/login

Username : admin
Password : admin
 Figure: Grafana Dashboard Login Screen

Now let’s add AWS CloudWatch as Datasource in Grafana.

Figure: Click on Add data source and select CloudWatch

 Figure: Provide AWS IAM Access key ID, Secret access key, and Region

Figure: Click on Save & Test to check if Cloudwatch datasource is working

After adding the Cloudwatch datasource, we can now get data from Amazon AWS ECS as has
been configured in the last few steps.
Grafana Dashboard for Amazon AWS ECS
Many Grafana Dashboards are readily available here https://grafana.com/grafana/dashboards that
can be imported inside the Grafana user interface to perform specific analytics operations and
visualizations.

Figure: Dashboard for Amazon AWS ECS

Source: https://grafana.com/grafana/dashboards/551

Importing the Amazon AWS ECS Dashboard in Grafana

To import the above given dashboard in Grafana, ensure that the AWS Cloudwatch data source
has been added already.
 Figure: On the left side, click on the + icon and import

Figure: Provide the Dashboard number, here which is 551 for AWS ECS
 Figure: Verify the dashboard being imported

Figure: Dashboard fetches ECS clusters and services

Adding load on SampleApp-Web service
Create the script given below.

[root@ns3078348 cloudwatch1]# vi ab.sh

ab -n 100000 -c 1000 http://example.com/
ab -n 100000 -c 1000 http://example.com/
ab -n 100000 -c 1000 http://example.com/
ab -n 100000 -c 1000 http://example.com/
ab -n 100000 -c 1000 http://example.com/
ab -n 100000 -c 1000 http://example.com/

Run the script. It will run the commands in batch mode, just to make the test run continuously for
long enough.

################# OUTPUT #################

[root@ns3078348 cloudwatch1]# sh ab.sh
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking example.com (be patient)

Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software: nginx/1.16.0

Server Hostname: example.com
Server Port: 80

Document Path: /
Document Length: 3032 bytes

Concurrency Level: 1000

Time taken for tests: 21.355 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 326600000 bytes
HTML transferred: 303200000 bytes
Requests per second: 4682.65 [#/sec] (mean)
Time per request: 213.554 [ms] (mean)
Time per request: 0.214 [ms] (mean, across all concurrent requests)
Transfer rate: 14935.08 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 98 104 40.1 102 1136
Processing: 99 106 22.7 103 959
Waiting: 99 106 20.2 103 728
Total: 197 210 46.4 205 1245

Percentage of the requests served within a certain time (ms)

50% 205
66% 207
75% 209
80% 210
90% 214
95% 219
98% 246
99% 271
100% 1245 (longest request)

Grafana Dashboard for AWS Cloudwatch Data

After running the script, here is what we see on the Grafana dashboard.

Figure: On the top right, change interval to Last 5 minutes and refresh rate to 5 seconds
 Figure: The load started at 15:44 as per this graph

Figure: CPU utilisation starts linearly, becomes constant and decreases in 3 minutes
Figure: Graph after Changing interval to Last 1 hour on top right

Figure: SampleApp-Web service Logs on CloudWatch

 Figure: Dashboard view of CloudWatch ECS Container Insights

After we are done with the Grafana’s testing of metrics and visualisation, it may be desirable to
stop the container service.

Run the following command:

[root@ns3078348 cloudwatch1]# docker-compose down

Stopping grafana ... done
Removing grafana ... done
Removing network cloudwatch1_grafana

This will also close the ports open for public access.

Grafana CloudWatch Logs Insights Custom Queries

We require monitoring several metrics from Example applications. Writing custom queries is the
starting point to reach several such scenarios, like:

 monitoring the number of successful events per minute

 monitoring the number of unsuccessful events per minute
 monitoring the time taken by an event’s iteration
 monitoring the total time taken by an API call

Using the same Cloudwatch Data Source in Grafana UI open the explore window on the left side
tool bar, and browse Cloud Watch logs.
Run the Grafana CloudWatch Logs Insights Query as given below for SampleApp-API Logs
Group :

fields @timestamp, @message | filter @message like /WARN/ | sort @timestamp

desc | limit 25

Figure: Custom CloudWatch Logs Insights Query for SampleApp-API Logs Group

The query here is to extract all lines in the logs with a “WARN“ message and display the time on
the graph when the log message was generated.
Grafana in Production
Grafana can be setup in production by following any of the methods of implementation as given
in the beginning of this article. Once you set it up, considering that we are using the
example.com domain with SSL, Grafana will be available on the following URL:

Web URL : https://grafana.example.com:3000/login

Login user : admin
Password : your-password

Figure: Grafana in Production

 Figure: Grafana Dashboards

Conclusion
Grafana is flexible to provide a wide variety of visualizations using data from several types of
sources. Activities on Grafana is a continuous process and we are likely to see more action with
this tool in future.

References
Grafana documentation
https://grafana.com/docs/grafana/latest

AWS CloudWatch data source

https://grafana.com/docs/grafana/latest/datasources/cloudwatch

Analyzing Log Data with CloudWatch Logs Insights

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html

CloudWatch Logs Insights Query Syntax

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html