Kibana Fundamentals

Kibana Search

Kibana Visualizations

Kibana Time Series Visual Builder

Kibana Geovisualizations

Kibana Dashboards

Tools for Advanced Analysis

Module 3
Machine Learning Fundamentals
Kibana Kibana Interface

Visualizations
Topics
• Behind the Visualizations
• More Types of Visualizations
• Improving Visualizations

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !105
 Kibana Visualizations

Lesson 1
Behind the Visualizations
Overview
• Data is often complex and involves many dimensions
• Often, we want summarized insights:
‒ slices based on specific attributes
‒ calculations based on specific attributes
‒ ...
• Spreadsheets might fulfill this using a "pivot table"
• In the Elastic Stack we call the equivalent functionality an
aggregation
• All aggregations are performed in elasticsearch
‒ Kibana just displays the results

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !107
A Simple Example: Spreadsheet

id user age country category

1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !108
A Simple Example: Elasticsearch

Elasticsearch

users

{
 {
 {
"User": "Bill", "User": "Marie", "User": "Claire",
"Age": 30, "Age": 32, "Age": 32,
"Country": "FR", "Country": "US", "Country": "US",
"Category": "A" "Category": "A" "Category": "A"
} } }

{ { {

"User": "Tom", "User": "John", "User": "Emma",
"Age": 44, "Age": 40, "Age": 26,
"Country": "DE", "Country": "US", "Country": "US",
"Category": "B" "Category": "B"
 "Category": "B"

} } }

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !109
Metrics Aggregations
• Metrics aggregations
‒ calculate numerical values over a set of documents
‒ similar to how values are summarized in a pivot table for a
specific column
‒ mathematical operation that output:
‒ a single value (eg., avg, sum, min, max, unique count)
‒ or multiple values (eg., percentiles, percentile_ranks)

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !110
A Simple Average Using Pivot Table

id user age country category

1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

Pivot table definition Pivot table

Rows Values AVG of age
AVG of age 34

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !111
A Simple Average Using Aggregations

Elasticsearch
{
"aggregations": {
"avg_of_age": {
"avg": {
{
 {
 { "field": "age"
"User": "Bill", "User": "Marie", "User": "Claire", }
"Age": 30, "Age": 32, "Age": 32, }
"Country": "FR", "Country": "US", "Country": "US", }
"Category": "A" "Category": "A" "Category": "A" }
} } }

{ { {

"User": "Tom", "User": "John", "User": "Emma",
"Age": 44, "Age": 40, "Age": 26,
"Country": "DE", "Country": "US", "Country": "US", "aggregations" : {
"Category": "B" "Category": "B"
 "Category": "B"
 "avg_of_age" : {
} } } "value" : 34.0
}
}

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !112
Bucket Aggregations
• Bucket aggregations allow you to:
‒ slice the data
‒ similar to grouping by values in rows or columns in a pivot
table
‒ create buckets
‒ collection of documents that share a common criterion
‒ can have one or more metrics associated with it
‒ number of documents (doc count) per bucket is the default metric

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !113
Simple Bucket Using a Pivot Table

id user age country category

1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

Pivot table definition Pivot table

Rows Values category COUNT of id
Order ASC by COUNT of id A 3
category B 3

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !114
Simple Bucket Aggregation
{
Elasticsearch "aggregations": {
"categories": {
"terms": {
"field": "category"
{
 {
 { }
"User": "Bill", "User": "Marie", "User": "Claire", }
"Age": 30, "Age": 32, "Age": 32, }
"Country": "FR", "Country": "US", "Country": "US", }
"Category": "A" "Category": "A" "Category": "A"
} } } "aggregations": {
"categories": {
Bucket: A "buckets": [
Count: 3 {
"key": "A",
"doc_count": 3
},
{ { {
 {
"User": "Tom", "User": "John", "User": "Emma", "key": "B",
"Age": 44, "Age": 40, "Age": 26,
"Country": "DE", "Country": "US", "Country": "US", "doc_count": 3
"Category": "B" "Category": "B"
 "Category": "B"
 }
} } } ]
}
Bucket: B }
Count: 3

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !115
Adding Metrics

id user age country category

1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

COUNT AVG of
Rows Values category
of age age
Order ASC by COUNT of age A 3 31.33
category AVG of age B 3 36.66

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !116
Adding Metrics "aggregations": {
"categories": {
"terms": {
"field": "category"
Elasticsearch },
"aggregations": {
"avg_age_per_category": {
"avg": {
{
 {
 { "field": "age"
}
"User": "Bill", "User": "Marie", "User": "Claire",
} } } }
"Age": 30, "Age": 32, "Age": 32,
"Country": "FR", "Country": "US", "Country": "US",
"Category": "A" "Category": "A" "Category": "A"
} } }
Bucket: A "aggregations": {
Count: 3 "categories": {
Avg of age: 31.33 "buckets": [
{
"key": "A",
"doc_count": 3,
"avg_age_per_category": {
{ { {
 "value": 31.33
"User": "Tom", "User": "John", "User": "Emma", }
"Age": 44, "Age": 40, "Age": 26, },
"Country": "DE", "Country": "US", "Country": "US", {
"Category": "B" "Category": "B"
 "Category": "B"
 "key": "B",
} } } "doc_count": 3,
Bucket: B "avg_age_per_category": {
Count: 3 "value": 36.66
Avg of age: 36.66 }
} ] } }
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !117
Nesting Rows/Columns in a Pivot Table
id user age country category
1 Bill 30 FR A
2 Marie 32 US A
3 Claire 32 US A
4 Tom 44 DE B
5 John 40 US B
6 Emma 26 US B

COUNT AVG of
category country
Rows Values of age age

Order ASC by COUNT of age

A FR 1 30
category US 2 32
Order ASC by AVG of age
country B DE 1 44
US 2 33

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !118
Adding Sub-Bucket Aggregation
Elasticsearch
Bucket: A
Count: 3
Bucket: FR Bucket: US
Count: 1 Count: 2
Avg of age: 30 Avg of age: 32
{
 {
 {
"User": "Bill", "User": "Marie", "User": "Claire",
"Age": 30, "Age": 32, "Age": 32,
"Country": "FR", "Country": "US", "Country": "US",
"Category": "A" "Category": "A" "Category": "A"
} } }

Bucket: B
Count: 3
Bucket: DE Bucket: US
Count: 1 Count: 2
Avg of age: 44 Avg of age: 33
{ { {

"User": "Tom", "User": "John", "User": "Emma",
"Age": 44, "Age": 40, "Age": 26,
"Country": "DE", "Country": "US", "Country": "US",
"Category": "B" "Category": "B"
 "Category": "B"

} } }

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !119
Metrics Aggregations

Metrics Aggregation 6
Count of Documents

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !120
Bucket Aggregations

Metrics
2
Aggregation
1

0
US FR DE

Bucket Aggregation
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !121
Sub-bucket Aggregations

Sub-Bucket Aggregation
4
A

3 B

Metrics
2
Aggregation
1

0
US FR DE

Bucket Aggregation
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !122
 Kibana Visualizations

Lesson 1
Review - Behind the
Visualizations
Summary
• Kibana renders visualizations using the results of
Elasticsearch aggregations
• There are two main types of aggregations:
‒ metric
‒ bucket
• Metric aggregations are used to compute numeric values
• Bucket aggregations are used to group data together

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !124
Quiz
1. What are the two main types of aggregation?
2. True or False: Aggregations are used by Kibana to render
visualizations.
3. Explain which aggregations are used to build the following
visualization.

US

FR

DE

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !125
 Kibana Visualizations

Lesson 1
Lab - Behind the Visualizations
 Kibana Visualizations

Lesson 2
More Types of Visualizations
Pivot Table in Kibana
• Aggregations and pivot tables are two concepts that are
very similar
• By using the data table visualization it is possible to have a
"pivot table" like structure in Kibana
• What aggregations are needed in order to generate the
following table?
Category Country Count Average(age)

Category A US 2 32

Category A FR 1 30
Category B US 2 33
Category B DE 1 44
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !128
Pivot Table
• We need the following to create the table:
‒ a bucket aggregation of type terms on the category field first
‒ a sub-bucket aggregation of type terms on the country field
‒ a metric aggregation of type count (default metric aggregation)
on all generated buckets
‒ another metrics aggregation of type average on the age field
on all the generated buckets

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !129
Table Visualization
• By default data table in Kibana will have only one row
‒ Showing a count of all documents

Count

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !130
Table visualization
• The row can be split by first applying a terms
aggregation on the category field

Category Count

Category A 3

Category B 3

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !131
Table visualization
• Resulting rows can then be split again by applying
another terms aggregation on the country field

Category Country Count

Category A US 2

Category A FR 1
Category B US 2
Category B DE 1

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !132
Table visualization
• A column can then be added to the table by adding the
average metrics aggregation on the age field:

Category Country Count Average(age)

Category A US 2 32
Category A FR 1 30
Category B US 2 33
Category B DE 1 44
Total 6 139

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !133
Customizing Data Table
• Every row in a table is the result of a bucket aggregation
• The number of rows displayed in the table can be
customized using the options tab
• Tables can span multiple pages and you can paginate to
subsequent pages using the link at the bottom of the table
• It is possible to apply the metric aggregations at the parent
level using the options tab
• A grand total of all the metrics can also be computed

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !134
Heat Map Overview
Metrics Aggregation

Category A

Category B

Bucket Aggregations DE US FR

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !135
Heat Map Example

Countries and download size by response time

0k-10k

10k-20k

20k-30k

30k-40k

40k-50k

50k-60k

60k-70k

70k-80k

Haiti Jordan Mayotte Venezuela Lebanon Botswana

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !136
Tag Cloud Overview
• Tag cloud is way to visually represent words in a dataset
• It supports only two bucket aggregations:
‒ terms aggregation
‒ significant term aggregation

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !137
Tag Cloud
• Every word is the result of the bucket aggregation
• The size of a word is a function of the result of the metric
aggregation (defaults to count)

Marie
John
Bill
Tom
Emma

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !138
 Kibana Visualizations

Lesson 2
Review - More Types of
Visualizations
Summary
• Kibana provides a lot of ways to visually represent data.
• Elasticsearch aggregation concepts are central to building
Kibana visualization
• In Heat Maps, bucket aggregations on X and Y axis
produce the cells of the matrix and the result of a metrics
aggregation determines color or intensity of individual cells

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !140
Quiz
1. True or False: The colors in a heat map visualization are
determined by a metrics aggregation
2. How would you create the following table:

Name Count Average Like

Smith 321 25.4

Goodwill 219 193.4

De Bourraine 200 149.4
Schwartz 143 123.8

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !141
 Kibana Visualizations

Lesson 2
Lab - More Types of
Visualizations
 Kibana Visualizations

Lesson 3
Improving Visualizations
 Comparing Multiple Metrics

Bucket Aggregation

?
4 40

3 30
Document count

Average age
Metrics
2 20
Aggregation

1 10

0 0
US FR DE
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !144
Visualizing Multiple Metrics
• Multiple metrics can be displayed side by side in the same
visualization
• Having multiple metrics can make the visualization hard to
read for the following reasons:
- the metrics have different scale
- the metrics are stacked (or not, depending of the purpose of the
visualization)
- the metrics have same representations when comparing different
variables (bars, lines, area)
- poor color choice

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !145
Visualizing Multiple Metrics
• To overcome these challenges, every metrics aggregation
can be customized with its own style:
- color
- axis
- properties
• Each metric can be displayed differently:
- lines
- bars
- areas

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !146
Visualizing Multiple Metrics
Two different Y axes

Different chart

types

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !147
Bubble Chart
• Bubble chart can be built by defining a new metric:
1. define X-axis
2. define Y-axis
3. define dot size
4. change visualization to display a line
5. uncheck "show line"

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !148
Bubble Chart

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !149
Multi Charts
• Sometimes, multiple buckets or sub-buckets on a single
visualization may not be suitable:
‒ buckets or sub-buckets may not be related
‒ too many metrics on the same visualization can make it busy
• In Kibana it is possible to display multiple charts in the
same visualization by using split chart

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !150
Single Chart
Count Average

4 40

3 30
Document count

Average age
2 20

1 10

0 0
US FR DE
Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or
distributing without written permission is strictly prohibited !151
Multi Charts

2 40

Document count

Average age
Category A 1 20

0 0
US FR

2 40
Document count

Average age
Category B
1 20

0 0
US DE

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !152
Demo

Instructor Demo

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !153
 Kibana Visualizations

Lesson 3
Review - Improving Visualizations
Summary
• Multiple metrics can be displayed in a single visualization
• Every metric can be customized with a dedicated axis,
different color, style, ...
• It is possible to create multiple charts inside a single
visualization by using a bucket aggregation to divide the
data

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !155
Quiz
1. What is the minimum number of metrics required for a bubble
chart?
2. True or False: Splitting a chart relies on the same concept as
a bucket aggregation
3. True or False: It is possible to display 2 metrics that have
very different scales on the same visualization

Copyright Elasticsearch BV 2015-2019 Copying, publishing and/or

distributing without written permission is strictly prohibited !156