Kibana Data Analyst An Elastic Tra Training ining Course
6.5.0
elastic.co/training
Kibana Data Analyst Course: Kibana Data Analyst Version 6.5.0 © 2015-2019 Elasticsearch BV. All rights reserved. Decompiling, copying, publishing and/or distribution without written consent of E lasticsearch BV is strictly prohibited.
2
Agenda and Introductions
About This Training • Environment • Introductions • Agenda...
4
Course Agenda 1
Kibana Fundamentals
2
Kibana Search
3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
5
Lesson 1
Kibana Fundamentals
1
Kibana Fundamentals
2
Kibana Search
3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
Introduction to Kibana
The Elastic Stack
INGEST
INDEX | QUERY | AGGREGATE
8
VISUALIZE
Ingest: Logstash and Beats • Logstash ! Server-side data processing ! Ingests data from multiple sources simultaneously (MongoDB, PostgreSQL, Elasticsearch, ...)
! Parse, transform and prepare your data for ingestion
• Beats ! Single purpose data shippers ! Many flavors: Filebeat, Metricbeat, Packetbeat, Winlogbeat, ... ! Lightweight agents that send data from a machine to Elasticsearch or Logstash
9
Index Query and Aggregations • Elasticsearch is the heart of the Elastic Stack • Elasticsearch is ! distributed: easy to scale ! RESTful: easy to communicate with using APIs
• It can be used to: ! search ! analyze ! and store data
10
Visualize • Kibana Kibana is is a graphical user interface for the Elastic Stack • Kibana Kibana can can be used to: ! manage the stack ! interact with the data ! store data
11
Data Journey 1. Genesis Beats
2. Ingest
3. Store
Kibana
Elasticsearch
Discovery Visualize Dashboard Graph
Logstash
4. Search & Analyze
12
Definition of a Document • The same data can be represented differently: title
category
Fighting Ebola with Elastic
date
User Stories
author_first_name au a uthor_last_name author_company Emily
Mosher
A row in a table table { "title": title": "Fighting Ebola with Elastic", "category": category ": "User Stories", "author": author ": { "first_name": first_name ": "Emily", "last_name": last_name": "Mosher" } }
JSON
Emily Emily Mosher Mosher User User Stories Stories Fighting <title>Fighting Ebola with Elastic
XML
• A document is is a serialized JSON object that is stored in Elasticsearch under a unique ID 13
A Simple Example: Spreadsheet
id
user
age
country
1
Bill
30
FR
A
2 3 4 5 6
Marie Claire Tom John Emma
32 32 44 40 26
US US DE US US
A A B B B
14
category
A Simple Example: Elasticsearch Elasticsearch
{
{ "User": "Bill", "Age": 30, "Country": "FR", "Category": "A"
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "B"
"User": "Claire", "Age": 32, "Country": "US", "Category": "A"
}
}
}
{
{
{
"User": "Tom", "Age": 44, "Country": "DE", "Category": "B" }
"User": "John", "Age": 40, "Country": "US", "Category": "B" }
"User": "Emma", "Age": 26, "Country": "US", "Category": "A" }
15
Static Data vs. Time Series Data • Data can be categorized into two groups: ! (relatively) static data: a large (or small) dataset that may grow or change slowly
! time series data: event data associated with a moment in time that typically grows rapidly
{ "cuisine": "French", "ingredients": "ingredients": "Cheese, "Cheese, flour, flour, butter, eggs, milk, nutmeg", "time_in_min": "time_in_min": 50, "level": "easy" }
Which category do these documents belong to? { "tweet": "Wow Elasticsearch Elasticsearc h 7.0 seems awesome!", "hashtags": ["elasticsearch", ["elasticsearch", "kibana"] "timestamp timestamp": ": September 1st 2017, 07:15:40.035 }
16
Documents Are Stored in an Index • Indices Indices are are a way of grouping similar data together • One common way to index data index data into different indices is to use the timestamp value
Elasticsearch
Beats
tweets-2018-12-24 tweets-2018-12-24 Logstash
cooking_recipes
17
tweets-2018-12-23 tweets-2018-12-23
tweets-2018-12-22 tweets-2018-12-22
Index Patterns • An index pattern identifies pattern identifies one or more Elasticsearch indices that indices that you want to explore with Kibana
Elasticsearch
Kibana
tweets* tweets-2018-12-24 tweets-2018-12-24
tweets-2018-12-23 tweets-2018-12-23
tweets-2018-12-22 tweets-2018-12-22
cooking_recipes cooking_recipes
Indices
18
Index Patterns
Datasets
Messages #vacation #dream
Elasticsearch
{
John Smith Germany Berlin 130 Followers
"message_id": "message_id": 1, "user.first_name": "user.first_name": "John", "user.last_name": "user.last_name": "Smith", "user.geo.country": "user.geo.country": "Germany", "user.geo.city": "user.geo.city": "Berlin", "user.nb_of_followers": "user.nb_of_followers": 130, "subjects": "#vacation #dream", "number_of_subjects": "number_of_subjects": 2, "likes": 32, "geo.country": "geo.countr y": "United Kingdom", "geo.city": "London" }
20
user_messages
Users
John Smith ..... 32 likes
Elasticsearch
John Smith ..... 123 likes
John Smith ..... 18 likes user_messages
{ "message_id": "message_id" : 41, "first_name": "first_name" : "John", "last_name": "last_name": "Smith", "geo.country": "geo.country": "Germany", "geo.city": "Berlin", "nb_of_followers": "nb_of_followers": 130, "average_like": "average_lik e": 87.45, "salary": 120000, "occupation": "occupation" : "Sales"
users
}
21
Lab Environment
Lab Environment • Using the link that was shared with you, login to Strigo • Click on "My "My Lab" Lab" on the left
23
Lab Environment • Click on the gear icon next to "My "My Lab" Lab" and select "Machine Info" Info"
24
Lab Environment • Copy the hostname that is shown under "Public "Public DNS" DNS"
25
Lab Environment • From here you can access lab instructions and guides ! You also have them in your .zip file, but it is easier to access and use the lab instructions from here:
26
Accessing your Cluster • Click on the Kibana link: • Log in ! username: training ! password: kibana_management
27
Lesson Review
Summary • Elasticsearch, Kibana, Logstash, and Beats are components of the Elastic Stack • Kibana can be used to analyze, search, interact with and visualize the data in Elasticsearch • Kibana can be used to manage the Elastic Stack • Data is sent as JSON objects into Elasticsearch • In Kibana, an index pattern can be created to target a specific set of indices
29
Quiz 1. What are the four four main component components s of the Elastic Stack? Stack? 2. True or False: False: Data is stored inside Kibana. 3. What would would be a suitable index index pattern pattern for accessing accessing both cooking_recipes and cooking_recipes and cooking_user indices? 4. What kind of dataset dataset the two following following document documents s belong to? { "heartbeat": "heartbeat": 123, "timestamp": "Mon, 24 Dec 2018 00:23:28 GMT" } { "first_name": "first_name" : "Bill", "last_name": "last_name": "Smith", "age": 27, "country": "Mongolia" }
30
Lab 1
Discover Interface
Overview • Data in Elasticsearch can have different types: ! numeric ! text ! date ! keywords ! ...
• The discover interface allows you to explore documents in Elasticsearch ! and better understand the different aspects of the data
33
The Discover Interface Side navigation
Tool bar
34
Time picker
The Discover Interface Query bar
Index pattern
Histogram
35
Document table
Search is Everywhere • Elasticsearch is a search engine ! Kibana can be used to search documents in Elasticsearch
• A search is executed by sending a query to Elasticsearch query to ! A query can answer answer many different types of questions: ! who are the users that are called Melissa? ! what are the names of the people living in France? Fr ance? ! are there any messages about Netflix?
• In Kibana, a search can be executed from the query bar ! Kibana supports multiple query languages *
36
Designing a Query • Before using the search bar, the first thing to do is to formalize the question: What are the messages messages from from John John in in the US US? ?
• Then, the query can be written and is actually fairly simple: john us
• Elasticsearch scans through all the dimensions of the data for the queried values id 1 2 3 4 5 6
user
a e
Bill
30
FR
A
Marie
32
US
A
Claire
32
US
A
John
40
DE
B
John
44
US
B
Emma
44
US
B
37
countr
cate or
Search a Specific Field • By default, the query below will search all fields for all values john AND us
! but often you want to be more specific What are the messages messages in in which the user is is John in the US country country? ?
• If that is the case, you can rewrite the query above
! so that Elasticsearch will only query the specified fields user:john country:us
38
Boolean Operators • By default, Kibana uses the OR OR logic logic ! so it matches any documents containing john OR us
• Kibana allows you to use the following boolean operators: operators: ! AND, OR, and NOT
• Now Now,, you can rewrite the query with the AND AND logic logic user:john AND country:us id
user
age
country
1 2 3
Bill
30
FR
A
Marie
32
US
A
Claire
32
US
A
John
40
DE
B
John
44
US
B
Emma
44
US
B
4 5 6
39
category
Querying Numeric Fields • Let's add some complexity to the question: What are the messages in messages in which the user is John in the US country whose age is over 40? 40 ?
• Numbers are different than text ! instead of exact matches you often have relations: ! less than (<) ! less than or equal (<=) ! greater than (>) ! greater than or equal (>=)
• Now Now,, you can rewrite the query as: user:john AND country:us AND age:>40
40
Query "Context" • There are two important factors to take into consideration: ! where are we looking for those documents?
Index Pattern
! when are we looking for those documents?
Time Picker
• Make sure to set the correct index pattern and timeframe:
41
Demo
42
Lesson Review
Summary • The discover interface allows you to explore the different data dimensions • The most common mistake in the discover interface is not checking the index pattern and pattern and time picker • The search bar can be used to search all the data inside Elasticsearch • The document table can be customized to display only some of the fields
44
Quiz 1. What are the first first two settings settings someone someone should check check when using the discover interface? 2. What are the three three different different boolean boolean operators? operators? 3. Build the query: query: "Find the messages messages from from Claire Claire younger younger than 30 years old that belong to the category A?"
45
Lab 2
Aggregations Aggregatio ns
Overview • Data is often complex and involves many dimensions • Often, we want immediate insights about a dataset, such as: ! the average of a specific attribute ! slicing the data based on specific attributes ! ...
• Spreadsheets might fulfill this using a "pivot "pivot table" table" • In the Elastic Stack we call the equivalent functionality a search aggregation
48
A Simple Example: Spreadsheet
id
user
age
country
1
Bill
30
FR
A
2 3 4 5 6
Marie Claire Tom John Emma
32 32 44 40 26
US US DE US US
A A B B B
49
category
A Simple Example: Elasticsearch Elasticsearch
users {
{ "User": "Bill", "Age": 30, "Country": "FR", "Category": "A"
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "A"
"User": "Claire", "Age": 32, "Country": "US", "Category": "A"
}
}
}
{
{
{
"User": "Tom", "Age": 44, "Country": "DE", "Category": "B" }
"User": "John", "Age": 40, "Country": "US", "Category": "B" }
"User": "Emma", "Age": 26, "Country": "US", "Category": "B" }
50
Metrics Aggregation • In Elasticsearch metrics are calculated using aggregations ! similar to how values are summarized in a pivot table for a specific field or column
• Metrics aggregations compute numeric values based on your dataset ! they are mathematical operations that output a single value: avg,, sum sum,, min min,, max max,, unique count ! avg
! or multiple values: ! percentiles percentiles,, percentile_ranks
51
A Simple Average Using Pivot Table id
user
age
country
1 2 3 4 5 6
Bill Marie Claire Tom John Emma
30 32 32 44 40 26
FR US US DE US US
Pivot table definition
Rows
category A A A B B B
Pivot table
Values
AVG A VG of age
AVG AVG of age
34
52
A Simple Average Average Using Aggregations Elasticsearch
{
{ "User": "Bill", "Age": 30, "Country": "FR", "Category": "A"
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "A"
"User": "Claire", "Age": 32, "Country": "US", "Category": "A"
}
}
}
{
{
{
"User": "Tom", "Age": 44, "Country": "DE", "Category": "B" }
"User": "John", "Age": 40, "Country": "US", "Category": "B" }
"User": "Emma", "Age": 26, "Country": "US", "Category": "B" }
53
{ "aggregations "aggregations": ": { avg_of_age": "avg_of_age ": { avg": "avg ": { "field field": ": "age "age" " } } } }
"aggregations" : { "avg_of_age" "avg_of_age " : { "value" : 34.0 } }
Buckets • A bucket aggregation is a way of slicing slicing the the data ! in a pivot table it could be compared to rows or columns
• A bucket is is simply a collection of documents that share a common criterion • Every bucket can have one or more metrics associated to it ! the default metric associated to a bucket is the number of documents inside the bucket
54
Simple Bucket Using a Pivot Table id
user
age
country
1 2 3 4 5 6
Bill Marie Claire Tom John Emma
30 32 32 44 40 26
FR US US DE US US
A A A B B B
Pivot table definition
Rows
Values
Order ASC by category
COUNT of id
category
Pivot table
55
category
COUNT of id
A
3
B
3
Simple Bucket Aggregation Elasticsearch
{
{ "User": "Bill", "Age": 30, "Country": "FR", "Category": "A"
}
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "A"
}
"User": "Claire", "Age": 32, "Country": "US", "Category": "A" }
Bucket : A Bucket: Count: 3
{
{ "User": "Tom", "Age": 44, "Country": "DE", "Category": "B"
}
{ "User": "John", "Age": 40, "Country": "US", "Category": "B"
}
"User": "Emma", "Age": 26, "Country": "US", "Category": "B" }
Bucket: B Count: 3 56
{ "aggregations": { "categories": "categories" : { "terms terms": ": { "field field": ": "category "category" " } } } } "aggregations": "aggregations": { "categories categories": ": { "buckets buckets": ": [ { "key": "A", "doc_count": 3 }, { "key": "B", "doc_count": 3 } ] } }
Adding Metrics id
user
age
country
1 2 3 4 5 6
Bill Marie Claire Tom John Emma
30 32 32 44 40 26
FR US US DE US US
category
category A A A B B B
COUNT of age
AVG A VG of age
Rows
Values
Order ASC by category
COUNT of age
A
3
31.33
AVG AVG of age
B
3
36.66
57
Adding Metrics Elasticsearch
{
{ "User": "Bill", "Age": 30, "Country": "FR", "Category": "A"
}
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "A"
}
"User": "Claire", "Age": 32, "Country": "US", "Category": "A" }
Bucke t: A Bucket: Count: 3 Avg of age: 31.33
{
{ "User": "Tom", "Age": 44, "Country": "DE", "Category": "B"
}
"aggregations": { "categories": { terms": "terms ": { "field field": ": "category "category" " }, aggregations": "aggregations ": { "avg_age_per_category": { "avg avg": ": { "field field": ": "age "age" " } } } } }
{ "User": "John", "Age": 40, "Country": "US", "Category": "B"
}
"User": "Emma", "Age": 26, "Country": "US", "Category": "B" }
Bucket: B Count: 3 Avg of age: 36.66 58
"aggregations": { "categories": { "buckets": [ { "key": "A", "doc_count": 3, "avg_age_per_category": { "value": 31.33 } }, { "key": "B", "doc_count": 3, "avg_age_per_category": { "value": 36.66 } } ] } }
Nesting Rows/Columns in a Pivot Table Table id
user
age
country
1 2 3 4 5 6
Bill Marie Claire Tom John Emma
30 32 32 44 40 26
FR US US DE US US
category country Rows
Values
Order ASC by category
COUNT of age
Order ASC by country
AVG AVG of age
A
B
59
category A A A B B B COUNT AVG AVG of of age age
FR
1
30
US
2
32
DE
1
44
US
2
33
Adding Sub-Bucket Aggregation Elasticsearch
{
Bucket: FR Count: 1 Avg of age: 30
{
"User": "Bill", "Age": 30, "Country": "FR", "Category": "A" }
Bucket : A Bucket: Count: 3
{ "User": "Marie", "Age": 32, "Country": "US", "Category": "A"
}
Bucket: US Count: 2 Avg of age: 32 "User": "Claire", "Age": 32, "Country": "US", "Category": "A"
}
Bucket: B Count: 3
{
Bucket: DE Count: 1 Avg of age: 44
{
"User": "Tom", "Age": 44, "Country": "DE", "Category": "B" }
{ "User": "John", "Age": 40, "Country": "US", "Category": "B"
}
"User": "Emma", "Age": 26, "Country": "US", "Category": "B" }
60
Bucket: US Count: 2 Avg of age: 33
Metrics Aggregation
6
Metrics Aggregation
Count of Documents
61
Bucket Aggregation
4
3
Metrics Aggregation
2 1 0 US
FR
Bucket Aggregation
DE
Sub-bucket Aggregation Sub-Bucket Aggregation 4
A
3
Metrics Aggregation
B
2 1 0 US
FR
Bucket Aggregation
DE
Lesson Review
Summary • Kibana renders visualizations using the results of Elasticsearch aggregations • There are two main types of aggregations: ! metrics ! bucket
• Metrics aggregations are used to compute numeric values • Bucket aggregations are used to group data together
65
Quiz 1. What are are the two main types types of aggrega aggregations? tions? 2. True or False: Aggregations are used by Kibana to render visualizations. 3. Explai Explain n which aggregation aggregations s are used to build the following following visualization. A B US FR DE 66
Lab 3
Lesson 2
Kibana Search
1
Kibana Fundamentals
2
Kibana Search
3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
The Query Bar
Improving Search • Search is extremely important ! almost every page in Kibana will have a search bar
• But, search is not limited to searching for a specific word ! or searching over a specific interval
• Search may involve: ! fuzzy data ! using regex or wildcard to search over specific patterns ! boosting specific fields over others
70
Fuzzy Search • In today’s world, we expect a search application to grant us some leniency in terms of our spelling skills • A known approach is to use edit distance: distance: ! edit distance are functions that count the number of edits to go from one word to another
! the Damereau-Levenshtein distance is one of the most used ones Edit distance = 1
Edit distance = 2
“Mario”
“Eifele”
o
e
f e
Marie
“Eiffel”
71
Fuzzy Search • The edit distance can be defined per query term ! the allowed values are 0 (default), 1, 2, and auto ! auto will define the fuzziness based on the length of the word Searched word
Fuzziness
Edit distance value
Marie
Mario~1
... Eiffel
Eifele~2
... Marie
Mario~auto
... 72
Fuzzy Search • Be aware that fuzzy search is expensive ! it will not be as fast as regular searches ! and it can crash your cluster
• Great for incident analysis • Not so great for common queries
73
Wildcard Overview • Wildcard can be used to enhance searches • In a lot of cases some data follow a certain pattern: ! var/lib/elasticsearch/conf/ elasticsearch.yml var/lib/elasticsearch/conf/ jvm.options jvm.options var/lib/elasticsearch/conf/ log4j2.properties
! jump, jumps, jumping, jumped
• There are two options when it comes to wildcard: ! *, which matches any character sequence (including empty ones) ! ?, which matches any single character
74
Wildcard Overview • Both * and ? can be used at part of the string • Let's search for every word that start with Mari Mari Marie Mari*
Maria Marion Marine
• Let's search for every word that starts with Mar?a Mar?a,, in which ? is any single character Marea Mar?a
Maria Marya
75
Regexp Query • The regexp regexp query query offers even more power to match patterns ! syntax is based on the Lucene regular expression engine https://www.elastic.co/guide/en/elasticsearch/reference/cur https://www .elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query rent/query-dsl-regexp-query.html#regexp-syntax .html#regexp-syntax
• For example, you can use it to find all the documents that contain a phone number from France: ! starts with +33 followed by 9 numbers
my phone number: +33235151 +33235151151 151 /.*+33[0-9]{9}.*/
Here: +33235151151 call me later ...
76
Searching for Patterns is Expensive • Both wildcard queries and regexp queries are expensive ! they will not be as fast as regular searches ! and they can crash your cluster
• Great for incident analysis • Not so great for common queries ! If you need it, prefer trailing patterns ! Always avoid leading patterns
77
Boosting • When searching on multiple fields, some fields can be more important than others: ! for example, an email title is probably more relevant than the body
• In the search bar, it is possible to reflect this by boosting specific fields: ! which impacts the order of the returned documents subjects:instafood subjects:instafood OR first_name:john^2 first_name:john^2
Searching on the subjects and first_name fields
78
Boosting the score by 2 on the field first_name
Kibana Query Language Enhancement • In Kibana version 6.0 a new Kibana query language has been introduced, Kuery ! it simplifies the Lucene syntax ! it is enabled when you turn on the autocomplete feature ! when enabled, the Lucene syntax cannot be use anymore
• The autocomplete feature suggests: ! field names ! operators ! values
79
Lesson Review
Summary • Fuzzy, Fuzzy, Wildcard, and Regular Expression queries qu eries are powerful and can help you find documents with partial information • However, they are expensive and should be used with care • Boosting allows you to influence the order that documents are returned • Kuery is a new Kibana query language introduced in 6.0 that simplifies the Lucene query syntax • Kuery has autocomplete
81
Quiz 1. True True or or False False:: Fuzzy, Fuzzy, Wildcard, and Regular Expression queries are powerful and cheap. 2. True True or or False False:: Kuery is enabled when you turn t urn on the autocomplete feature. 3. Explai Explain n the followi following ng query: query: subjects:insta* AND user:maria~auto
82
Lab 4
Search on Text
Text Subtleties • Text is very interesting when it comes to search ! different people will always expect different behavior:
• Let's take a comment and a user ID as examples: Wow! I love this city, Paris is beautiful!
B6IAWRLOLb
• Both examples are text, but ! a search for "paris" should still find documents that contain " Paris" B6IAWRLOLB B" should not find documents that ! a search for "B6IAWRLOL B6IAWRLOLb b" contain "B6IAWRLOL
• Why?
85
Full Text Search vs Exact Match • There are different use cases for text: ! full text search (where the text will be normalized) ! exact match (where the text will be kept as is)
• Let's dive more into those two use cases
86
Document Analysis • When there is a need for full text search, the data needs to be normalized: ! this process is called text analysis and is done by an analyzer ! text analysis is customizable and can have operations like lower casing, punctuation removal, tokenization, ...
• Let's index documents in Elasticsearch: Tokens Wow! I love this city, 1 Paris is beautifu beautiful! l! 2
3
I love Paris!
Analysis
Paris
87
ID
wow
1
i
1, 2
love
1, 2
this
1
city
1
paris
1, 2, 3
is so beautiful
1 1 1
Query Analysis • When querying an analyzed field, by default, the query will also be analyzed ! and then compared to the tokens of the indexed documents
Query Tokens The city of Paris
Analysis
the
city
of
paris
1
2
Response 88
ID
w ow
1
i
1, 2
love
1, 2
t h is
1
city
1
paris
1, 2
is so beautiful
1 1 1
Exact Match • For exact matches, the data is not going to be analyzed ! no normalization will be done
• Let's index documents in Elasticsearch:
1
2
Wow! I love this city, Paris is beautiful!
Tokens
ID
Wow! I love this city, Paris is beautiful!
1
I love Paris! I love Paris!
3
2
Paris Paris
89
3
Exact Match Query • When querying a not analyzed field, by default, the query will also not be analyzed ! but directly compared to the tokens of the indexed documents
• It should be used for strict values instead of sentences
Paris 3
Tokens
ID
Wow! I love this city, Paris is beautiful!
1
I love Paris!
paris
Paris
90
2
3
Index Patterns • Index patterns summarize well what a field is designed for • In the index pattern view, you will be able to see ! which fields you can use to build visualizations ! which fields you can user to search
• When the field is both aggregatable and searchable it means that the field is designed for exact match • When the field is only searchable it means that the field is designed for full text search
91
Lesson Review
Summary • Text data has two main use cases: full-text search and exact match • In full-text search, the data needs to be normalized • In exact match, the data is not normalized • When querying, by default, the query will be analyzed or not respectively • In Kibana, the index patterns view shows this information for each of the fields
93
Quiz 1. What are are the two main use use cases cases for text data? data? 2. True or False True or False:: When querying, by default, the query will be analyzed if the field is setup as an exact match. 3. In Kibana, Kibana, how do you know know if a field should should be used used for search or for aggregations?
94
Lab 5
Search with Query DSL
Search with Query DSL • So far, we discussed two methods to search data in Kibana: ! Lucene syntax ! Kuery (has auto-completion)
• Another way is based on the Query DSL syntax: ! it is mostly used by developers to query Elasticsearch directly ! it offers a lot of flexibility and fine tuning
97
The Query DSL Syntax • The Query DSL syntax relies on a simple JSON syntax to write complex queries: ! JSON makes it easy to read and understand the query ! JSON works with a system of key/value "Index Pattern" GET users* users*/_search /_search { "query": { "TYPE_OF_QUERY": "TYPE_OF_QUERY": { ... } } }
GET users*/_search Values { "query query": ": { "match": { "subjects subjects": ": { "query": "#dogs" } } } }
Keys
Type of query
98
Match Query • Imagine you want to search for "food", "dogs" or "chocolate" on the subjects subjects field field • Using the Lucene syntax, you would write the following: subjects:(food dogs chocolate)
• Using the query DSL syntax, you would write the following: GET user_messages* /_search { "query":{
"match": { "subjects": { "query": "food dogs chocolate" } } } }
99
Match Query Operator • By default the match query is applying the OR OR operator operator ! food OR dogs OR chocolate
• You can change the behavior of the match query by defining the operator GET user_messages*/_search { "query":{ "match": {
"subjects": { "query": "food dogs chocolate", "operator": "and" } } } }
100
Minimum Should Match • Searching for "food", "dogs" or "chocolate" "chocolate" may be too permissive and give too many results • Elasticsearch allows you to specify the minimum number of terms that should match ! for example, at least 2 words out of 3 should match ! (food AND dogs) OR (food AND chocolate) OR (dogs AND chocolate) GET user_messages*/_search { "query":{ "match": { "subjects": { "query": "food dogs chocolate", "minimum_should_match": "minimum_sho uld_match": 2 } } } }
101
Going Deeper • The query DSL syntax may seem complex at first but is very powerful • there are many types of queries: ! match ! match_phrase ! range ! ...
• The documentation will help you build queries: ! https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
• Engineer I and I and Engineer II trainings II trainings cover it in depth
102
Demo
103
Lesson Review
Summary • The Query DSL is more complex to write ! but gives you full control to all query features in Elasticsearch • The match query is the main query to match documents • The match query uses an OR as the default operator, but it can be easily changed to an AND • If you have multiple terms in a match query, you can define the minimum number of terms that should match
105
Quiz 1. True True or or False False:: The Query DSL allows you to use every search feature in Elasticsearch. 2. What is the default default operator operator of the match query? query? 3. True True or or False False:: By default, the match query should match a minimum of two terms.
106
Lab 6
Filters
Query Bar Limitations • Imagine you have multiple search criteria: ! age:>30 ! subjects:"art" ! subjects:"fashion" ! subjects:"summer" ! first_name:"Claire"
• Often times, you don't want to apply all of them together ! but, to analyze the data by enabling and disabling those searches
• It will require some manual work to do such a thing using the query bar ! this is where filters can be used... 109
Filters Overview • A filter is equivalent to a search criteria and they work in a very similar fashion • Once defined a filter can be: ! enabled/disabled ! pinned ! negated/positivized ! dropped ! edited
110
Filters Customization • Internally filters are transformed in a query DSL • There are two ways of customizing a filter: ! adding a label to the filter to quickly identify it ! redefine the way the filter behave by editing the query DSL
• When editing the generated query DSL it will be possible to change the behavior of the filter
111 111
Discover Interface Navigation • Filters can be used to navigate and explore the data: 1. create multiple relevant filters 2. type a query 3. use the filters for quick navigation
112
Kibana Interface Navigation • Filters can be used to navigate through multiple Kibana interface 1. create filters in discover 2. pin the interesting filters 3. go to visualize 4. create a visualization
113
Filter and Query Bar • The filters and the query bar are complementary • It is possible to have multiple search criteria in the query bar but they will be all applied in the same time • Filters allows for a better granularity, granularity, with some filters being enable or disabled • They will have similar performance
114
Internals • Internally the query inside the query bar will be transformed into a query_string query_string,, which will parse the query: ! https://www.elastic.co/guide/en/elasticsearch/reference/current/ query-dsl-query-string-query.html
• Depending of the created filter, different kind of queries will be generated: ! range: https://www.elastic.co/guide/en/elasticsearch/reference/ current/query-dsl-range-query.html
! match_phrase: https://www.elastic.co/guide/en/elasticsearch/ reference/current/query-dsl reference/curre nt/query-dsl-match-query-p -match-query-phrase.html hrase.html
! exists: https://www.elastic.co/guide/en/elasticsearch/reference/ current/query-dsl-exists-query.html
! ... 115
Demo
116
Lesson Review
Summary • Kibana filters give you a lot of flexibility when exploring the data, as they allow you to ! easily enable and disable different filters ! navigate through different pages with pinned filters
• You can customize filters using: ! pre-defined settings ! the query DSL
• Kibana filters and the query bar are complementary
118
Quiz 1. True True or or False False:: Kibana only allows a single filter at a time. 2. Cite three actions actions that you you can apply to a filter. filter. 3. True or False You should use either eit her kibana filters or the True or False:: You query bar.
119
Lab 7
1
Kibana Fundamentals
2
Kibana Search
3 Kibana Visualization
Lesson 3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
Lab 8
Pipeline Aggregations
A Simple Problem • New users are referenced every months: ! How would you build the cumulative sum of this chart?
400
300
s r e s u f o r 200 e b m u N
100
0
l i r p A
i a M
e n u J
y l u J
t s u g u A
r e b m e t p e
r e b o t c O
r e b m e v o N
r e b m e c e D
y r a u n a J
A Simple Solution • Stacking the different buckets will solve the issue...
400
2000
s r e s u f 1500 o m u s e v 1000 i t a l u m u C 500
300
s r e s u f o r 200 e b m u N
100
0
l i r p A
i a M
e n u J
y l u J
t s u g u A
r e b m e t p e
r e b o t c O
r e b m e v o N
r e b m e c e D
y r a u n a J
0
Pipeline Aggregation Introduction • ... But remember, remember, aggregations normally normall y work on documents • A pipeline aggregation, will actually works on the output generated by another aggregation • There are multiple pipeline aggregations available: ! avg, max, min, sum, cumulative sum, derivative, moving avg, serial diff
• Let's dive into some of the most complex pipeline aggregations...
Derivative • The derivative is used to compare one bucket with the previous one:
Derivative • No value can be computed for the first bucket as it require at least 2 values
Derivative
Derivative • The derivative help having an idea of the difference between a bucket and its previous value ! the longer the bar is, the greater the difference from the previous one
Serial Differencing Aggregation • The serial differencing aggregation will compute the difference between two buckets separated by a defined lag ! the default lag has a value of one (which make this aggregation, with the default setting, equivalent to the derivative derivative))
Serial Differencing Aggregation • Let's apply the serial differencing with a lag of 9, since it appear that there is a recurrent pattern of period 9: ! The 9 first buckets will then not have any values
Serial Differencing Aggregation • The serial differencing is a very powerful aggregation when it comes to times series analysis as it will de-trend the time series
The Moving Average Average • Typically the moving average can be used to smooth variations of a time series in order to highlight general trends
Window Based • A window will be defined on top of the buckets ! The bucket falling into this window will be averaged ! It is possible to define how large the window will be
Window Based • Then the window is slided and a new average computed
Window Based • Until the end of all the buckets
Window Based • The end result is a smoother time series:
Advanced JSON input • The UI doesn't give access to all the parameters of an aggregation (some aggregations have a lot of parameters available!) • Hopefully the advanced JSON input can be used to customized certain parameters: ! changing the window size for a moving average ! changing the lag in a serial differencing aggregation ! ...
• To customize the aggregation: ! refer to the doc to find the parameters of the aggregations ! Add the parameter parameter to the advanced advanced JSON JSON input { "window": "3"}
Demo
Lesson Review
Summary • Pipeline aggregations are working on the output of another aggregation instead of documents • The default settings of the serial differencing aggregation makes it equivalent to the derivative aggregation • Pipeline aggregations have a lot of parameters that can be added using Advanced JSON input
Quiz 1. What parameter parameter should should be used to increas increase e the window interval of a moving average? 2. True or False Pipeline False Pipeline aggregations work on the documents directly 3. What is the difference between the derivative and the serial differencing aggregation?
143
Lab 9
More Types Types of Visualization
Pivot Table in Kibana • Aggregations and pivot table are two concepts that are very similar • By using the table table visualization visualization it is possible to have a "pivot table" directly in Kibana • Let's generate the following: ! What aggregations are required here?
Category
Country
Count
Average(age)
Category A
US
2
32
Category A
FR
1
30
Category B
US
2
33
Category B
DE
1
44
Pivot Table • A bucket aggregation of aggregation of type term term need need to be executed on the category field first • A sub-bucket aggregation of aggregation of type term need to be executed on the country field • A metric aggregation of aggregation of type count count (default (default metric aggregation) need to be applied on all generated buckets • Another metric aggregation of type average on the age the age field need to be applied on all the generated buckets
Table Visualization • By default Kibana table will have only one row (a count of all the documents)
Count 6
Table visualization • The row need to be split by split by applying the first term aggregation on aggregation on the field category
Category
Count
Category A
3
Category B
3
Table visualization • The rows can be split again by again by applying another term aggregation on aggregation on the country country field field
Category
Country
Count
Category A
US
2
Category A
FR
1
Category B
US
2
Category B
DE
1
Table visualization • A column column can can be added to the visualization by adding the average metric aggregation on aggregation on the field age age::
Category
Country
Count
Average(age)
Category A
US
2
32
Category A
FR
1
30
Category B
US
2
33
Category B
DE
1
44
6
139
Total
Table Customization • Every rows in a table is the result of bucket aggregations • The number of rows displayed in the visualization can be customized in the options • If all the rows are not displayed in the table then a system of pagination can be used • It is possible to apply the metric aggregations at the parent level using the options of the table • The total of all the metrics can be computed
Heat Map Overview • Heat map can map can be seen as a 2 dimensions matrix visualization of your data • When creating a heat map, 3 things need to be defined: ! the bucket aggregation on the x-axis ! the bucket aggregation on the y-axis ! the metric to use to compare the different values of the matrix
Heat Map Overview Metric Aggregation
Category A
Category B
Bucket Aggregations
DE
US
FR
Heatmap Example Country with high response time per bits downloaded 0k-10k 10k-20k 20k-30k 30k-40k 40k-50k 50k-60k 60k-70k 70k-80k Haiti
Jordan
Mayotte
Venezuela
Lebanon
Botswana
Tag Cloud Overview • Tag cloud is a simple visualization visual ization allowing the display of words in the documents • There is only two bucket aggregations possible: ! terms aggregation ! significan significantt term aggregation
Tag Cloud • Every word is the result of the bucket aggregation • The size of a word is a function of the result of the metric aggregation (default to count)
Marie B
J o h n
i l l Tom
Emma
Visualizing Geo Points • The Coordinate map allows the visualization of geo points on a map • A grid is generated on top of a Mercator map • The granularity of the grid will depend of the zoom level ! the more the user zoom on the map the higher the precision will be
• The visualized points need to be geo point in Elasticsearch
Low Precision
Higher Precision
Region Map Overview • The coordinate map allows the visualization of geo points, the region map on the other hand is helpful to visualize regions: ! country ! states ! provinces
• By default Kibana is shipped with multiple vectors maps: https://maps.elastic.co/v2/index.html# • The Elastic Map Service have a zoom level up to 10 (18 if the basic license is used) • It is possible to use other base maps: https:// www.elastic.co/blog/custom-basemaps-for-region-andcoordinate-maps-in-kibana
Region Map Creation • To create a region re gion map it is important to make sure that the documents inside Elasticsearch contain the join the join field • For instance to visualize the countries of the world the documents need to contain a join a join field, field, that could be: ! 2 letters country code: FR, US, DE, NL ! 3 letters country code: FRA, USA, DEU, NLD ! the country name: France, United States, Germany, Netherlands
• It is possible to define the join field that is going to be used in the options • A terms aggregation is then run on the join field • The defined metric aggregation is going to define the color intensity of the regions
Region Map
Demo
Lesson Review
Summary • Kibana is providing a lot of ways of displaying data, but it all rely on the concept of aggregations and where they are used in the visualization • When building a Heat map the x and y axis are used for the buckets aggregation and the metric aggregation will define the color intensity of the cells • When displaying data on a map, there is two solutions, using the coordinate map (documents should have a field with latitude and longitude) or using the region map (documents should have a field containing a region)
Quiz 1. What is the maximum maximum value of Elastic Elastic Map Service Service zoom? zoom? 2. How would you you create create the following following table: table:
Name
Count
Average Like
Smith
321
25.4
Goodwill
219
193.4
De Bourraine
200
149.4
Schwartz
143
123.8
167
Lab 10
Improving Visualizations
Comparing Multiple Metrics Bucket Aggregation
?
Metrics Aggregation
4
40
t 3 n u o c t n e2 m u c o D
30
20
1
10
0
0 US
FR
DE
e g a e g a r e v A
Multi Metrics Visualization • Multiple metrics can be displayed side by side in a visualization • Having multiple metrics can make the visualization hard to read for the following reasons: - the metrics have different scale - the metrics are stacked (or not, depending of the purpose of the visualization) - the metrics have the same representations when they are comparing different things (bars, lines, area) - poor color choice
• Hopefully each metrics can have their own style, axes, color,, chart type, .... color
Multi Metrics Visualization • Metrics can be customized with their own style: - colors - axes - properties
• Each metrics can be displayed differently: - lines - bars - areas
Multi Metric Visualization Two different Y axes
Different chart types
Bubble Chart • Building a bubble chart can be done by defining a new metric: 1. define the X-axis 2. define the Y-axis Y-axis 3. define the dot size 4. change the visualization to display a line 5. uncheck "show line"
Bubble Chart
Multi Charts • Having multiple buckets or sub-buckets on a single visualization is sometime not interesting: ! buckets or sub-buckets are maybe not related ! too many metrics on the same visualization can make it hard to read
• In Kibana it is possible to display multiple charts in the same visualization by using split chart
Single Chart Count
Average
A B
4
40
t 3 n u o c t n e2 m u c o D
30
20
1
10
0
0 US
FR
DE
e g a e g a r e v A
Multi Charts
Category A
t n u o c t n e m u c o D
2
40
1
20
0
0 US
Category B
t n u o c t n e m u c o D
e g a e g a r e v A
FR
2
40
1
20
0
0 US
DE
e g a e g a r e v A
Demo
Lesson Review
Summary • Multiple metrics can be displayed in a single visualization • Every metrics can be customized, having a dedicated axis, color, style, ... • It is possible to create multiple chart inside a single visualization by using a bucket aggregation to divide the data
Quiz 1. What is the minimum minimum number number of metrics required required for for a bubble chart? 2. True or False: False: Splitting a chart relies on the same concept as a bucket aggregation 3. True or False: False: It is possible to display 2 metrics that have very different scales on the same visualization
Lab 11
Lesson 4
Kibana Customized Dashboards
1
Kibana Fundamentals
2
Kibana Search
3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
Introduction to Dashboard
Limitations • Visualizations are interactive, it is possible: ! to filter out some values ! apply searches ! select a time window
• But there is still a lot of things that are not possible to do with visualizations: ! how does other visualizations behave for the same specific time window
! which documents are used in the visualization ! analyzing the current visualization using different criteria like a part of the world, specifics terms, etc...
Dashboards • When there is a will, there is a way and the way is called: dashboard:: dashboard
Dashboards Overview • To create a dashboard d ashboard it is required to have at least one visualization (or search) as a dashboard is a collection of visualizations and searches ! create a new dashboard ! add visualizations and searches ! organize the dashboard ! save the dashboard ! ... analyze?
Filters and Dashboards • One of the main advantage of dashboards is the ability to dynamically create filters using multiple visualizations • Filters can be generated by using different visualizations directly from the dashboard interface: ! filter on values using some buckets present in the different visualizations
! define filters by drawing areas on the map ! filter by using input control (this will be covered later on) ! using searches in the dashboard
Sharing Dashboard • Once a dashboard is built it there is the possibility to share it with with someone else • There are multiple options to share a dashboard (or a visualization): ! using a permalink ! using an Iframe (embedded) ! generating a report
• The person with whom the dashboard is shared need to have access to Kibana
Inline Frame (Iframe) and Permalink • Iframes allow a kibana dashboard (or visualization) to be embedded in an HTML document ! when copying the Iframe the iframe tag will already be present
• Permalinks are links that point directly to a Kibana dashboard or a Kibana visualization • As Kibana URL contains the state state of the application they are often quiet long ! permalink will clean the URLs ! it is possible to shorten the URL as well
• When sharing a dashboard there are two options: ! share a snapshot ! share the latest saved object
Snapshot of a Dashboard • What would happen if while an email containing a permalink is sent, the visualizations in the dashboard are changed? ! if a snapshot was used then the visualizations that are going to be seen in the dashboard using the link (or iframe) will not changed
! if a saved object was used then the visualizations are going to changed
• Take a snapshot of the visualization/dashboard visualization/dash board in order to have it "frozen" and then share it with someone else
Kibana Access • The person with whom the dashboard is shared need to have access to Kibana: ! be aware of the permissions that the user is going to have ! is a user allowed to remove visualizations, or allowed to update a dashboard?
• Elastic security can help configuring different users and different roles with different permissions • Security will be covered more in depth in another lesson
Iframe and Kibana Security • Using security may seems troublesome when it comes to iframe as the user will have to authenticate two times ! once in the web app that display the iframe ! a second time in the Kibana iframe
• But there are solutions to this issue: ! defining a dashboard only user in Kibana ! using a reverse proxy to pass the authentication of the dashboard only user
Demo
Lesson Review
Summary • Dashboards group visualizations and searches in a single place to facilitate analysis • Dashboards can be be shared in multiple ways with other users • Sharing a dashboard imply security considerations: ! has the person access to Kibana? ! has the person access to the shared data?
Quiz 1. True or False One False One of the limitations of visualizations is their inability to have filters 2. What is the difference difference between between sharing sharing a saved dashboard dashboard and a snapshot of a dashboard? 3. What should should someone someone be careful careful about when when sharing a dashboard with someone else?
Lab 12
Markdown and User Input
Markdown Overview • Markdown is a simple markup language that allows a text to be annotated and then formatted accordingly • It is not designed to do advanced formatting but it can do things like: ! defining links ! add images ! defining titles ! ...
Markdown Examples # This is a big title ## ... and a smaller one ----* A list can can be useful useful * ... to display display links to _dashboards_ * ... for __insta __instance_ nce__ _
This is a big title ... and a smaller one _________________ _________________ • A list can be useful useful • ... to display display links to • ... for for instance
dashboards
Markdown Visualization • It is possible to create in Kibana markdown visualizations • Such visualization can be used to: ! describe specific visualizations ! add complementary information about what to expect from the behavior of a visualization
! add links to other dashboards (that may be related) ! ...
Dashboards and Filters • In a dashboard it is possible to manually create filters • But a user with whom the dashboard have been shared may not be familiar with Kibana and the concept of filters • There is a visualization that can be used to intuitively create filters, the controls controls visualization visualization • In the controls controls visualization, visualization, it is possible to define: ! a slider for numeric values ! an option list to find keywords
Demo
Lesson Review
Summary • There is a dedicated markdown visualization to add textual information to your dashboard • Inside a markdown visualization it is possible to add links that point to other source of information • Controls visualization can be used to generate filters directly inside the dashboards
Quiz 1. True or False It False It is possible to have dynamic text in a markdown visualization 2. Which text will will be the the biggest: biggest: 1. # Tit Title le1 1 2. ## Titl itle e2 3. True or False The False The filter generated by the controls visualization are different than the one that you can generate manually
Lab 13
Anomaly Hunt Hunt
Demo
Lesson Review
Summary • To hunt for an anomaly, use the different visualizations present inside the dashboard to dive into a specific anomaly • Once an anomaly is detected, it is possible to remove the anomaly by creating the corresponding filter • A filter can be pinned in order to navigate through multiple dashboards while hunting an anomaly
Quiz 1. True or False: False: An anomaly is always shaped by a high value on a visualization 2. How can an anomaly anomaly be removed removed from the visualiza visualizations? tions? 3. Is that always always advantageous advantageous to remove remove anomalies anomalies in a dashboard?
Lab 14
Canvas
What is Canvas? • Canvas is a rich live infographic system • By using Canvas you will be able to create: ! dashboards ! reports ! anything to present data...
• Canvas started as a Kibana plugin, it is now shipped with Kibana directly • Canvas is Beta since 6.5
A Dashboard with Metrics
Demo Data • The best way to get started with canvas is to look at canvas that are already created • It is possible to load sample data from Kibana, it will include: ! data ! index pattern ! visualizations ! dashboards ! canvas
The Elements Behind a Canvas The Canvas
Markdown
Images
Visualization
Internal of an Element Creating an Element
Element Type
Data Source
Styling
Image
Elasticsearch SQL
Colors
Pie Chart
Timelion
Fonts
...
Demodata
...
Markdown
Elasticsearch Documents
Size
Demo
Lesson Review
Summary • Canvas is a rich live infographic system that can be used to display anything • Plenty of different elements can be added to a canvas, each element being independent of each other • Canvas is relying on a strong expression language and to master canvas someone need to understand the expression language
Quiz 1. True or False: False: It is possible to display the canvas in full screen? 2. What are the differ different ent data inputs inputs that can be used used in an element?
Lab 15
Lesson 5
Kibana Time Series Visualization
1
Kibana Fundamentals
2
Kibana Search
5
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
Time Series Visual Builder
Introduction • Combine an infinite number of aggregations and pipeline aggregations to display complex data in a meaningful way • Allows a lot more customization in how the chart are displayed: ! background colors ! axis ! ...
• There is the possibility of combining multiple index patterns in the same visualization and adding annotations.
More Aggregations and Flexibility • Time Series Visual Builder (TSVB) is having more aggregations available to users: ! math (to apply mathematic operations to the data) ! static values ! overall maximum and minimum ! and many more ....
• TSVB has more flexibility: ! shifting the time series ! cloning series ! styling ! multiple index patterns
Demo
Lesson Review
Summary • Time Series Visual Builder is a powerful tool that give a lot of flexibility when working with aggregations • TSVB has more flexibility than pre-built visualization, styling options, crossing index patterns, metrics manipulation, ...
Quiz 1. Give 3 advantages that Time Time Series Visual Visual Builder has over pre-built visualization 2. True or False: False: It is possible to define static value using Time Series Visual Builder 3. True or False: False: It is possible to define an offset on a time series to align patterns
Lab 16
TSVB and Other Visualizations
Other Visualizations • TSVB is not limited to visualizing time series analysis data, if can also be used for ! metrics ! gauges ! top N ! markdown ! tables
• Those visualizations allow the addition of conditional to change the color of the visualization based on certain criterion • Visualizations do not work on the whole time series but only on the last bucket of the time series
Demo
Lesson Review
Summary • TSVB can be used to create more than time series visualizations, but metrics, gauges, tables and markdown visualizations • When using none time series visualizations, the data displayed are the one from the last bucket of the time series
Quiz 1. What are the 6 types types of visualizatio visualization n that TSVB TSVB is supporting? 2. True or False: False: When creating a metric in TSVB the number displayed is the number of documents inside the index pattern 3. True or False: False: It is possible to add dynamic elements to a markdown visualization using TSVB
Lab 17
Timelion
Introduction • Instead of building visualizations using the mouse, Timelion uses an expression language • When using Timelion's expression language everything is a function that start by a "." • It is very practical to use Timelion if the user is familiar with programming • When working with Timelion for the first time, it is best to use the auto-completion feature
Everything is a Function
A group of functions define a time series
The first function is the data source
Functions can be chained
.data_source_function(parameter=value).function(), .data_source_function(parameter=value).function()
A function can have parameters
Multiple time series can be defined using a coma
The Different Functions • Data sources: sources: es(), worldbank(), quandl(), static(), value() • Single-number math: math: abs(), min(), max(), log(), range(), scale_interval(), precision() • Time-series math: math: sum(), add(), plus(), subtract(), multiply(), divide(), derivative(), cusum(), • Trending functions: functions : movingaverage(), trend(), holt(), movingstd() • Conditions Conditions:: condition(), if() • UI functions: functions: bars(), lines(), points(), color(), label(), legend(), title() • Fit function (for function (for filling null values): fit()
A Simple Example
Elasticsearch is used as a datasource
The time series is divided by another time series
The query used, selects all the documents that contain FR
.es( apachelogs-*).divide(.es (index= apachelogs-*, q=FR)), es(index= index=apachelogs-*).divide(. es( index=apachelogs-*, es(index= index=apachelogs-*).divide(. es( index=apachelogs-*, .es( apachelogs-*).divide(.es (index= apachelogs-*, q=DE))
The index pattern used to represent the time series
Demo
Lesson Review
Summary • TSVB can be used to create more than time series visualizations, but metrics, gauges, tables and markdown visualizations • When using time series visualizations, the data displayed are the one from the last bucket of the time series
Quiz 1. Write a Timelion expression that: uses any index starting with apachelogs* queries the field geoip.country_code2 for the value DE and has the time field @timestamp @timestamp.. 2. True or False: False: Timelion Timelion can be used to divide di vide 2 different time series 3. What is the expre expression ssion below below computing? computing? .es(index=apachelogs*, q=geoip.country_code2:FR, metric=sum:bytes).divide(.es(index=apachelogs*,metric=sum:by metric=sum:bytes).divide(.es(index= apachelogs*,metric=sum:by tes)).multiply(100)
Lab 18
Lesson 6
Kibana Management
1
Kibana Fundamentals
2
Kibana Search
3
Kibana Visualization
4
Kibana Customized Dashboards
5
Kibana Time Series Visualization
6
Kibana Management
Advanced Settings Settings
Scripted Fields • Scripted fields compute values on the fly from the data in your Elasticsearch indices ! scripted field values are computed at query time so time so they aren’t indexed
intensive and can have a direct ! scripted fields can be very resource intensive and impact on Kibana’s performance
! there is no validation! validation! Buggy scripts will generate exceptions when viewing visualization containing them
• Scripted fields by default use the "Painless" or "Lucene Expressions" scripting languages ! https://www.elastic.co/guide/en/elasticsearch/reference/current/ modules-scripting-painless.html
! https://www.elastic.co/guide/en/elasticsearch/reference/current/ modules-scripting-expression.html
Scripted Fields • Scripted fields are created directly in the index patterns management view • The list of all the scripted fields can be seen in the same view • Once a scripted fields has been created it can be used like any other regular fields: ! while building visualizations ! while searching (only if the Kibana Query Language Enhancement is used)
Quick Range • The default 15 minutes value minutes value of the time picker can be misleading • That can be changed in the advanced settings: ! using the setting: timepicker:timeDefaults
• It is possible to define the default picker using the 3 following option: ! quick ! absolute ! relative
• The from from and and to to parameters parameters need to be defined accordingly
Timezone • Kibana by default is using the timezone of the browser to define the value of the timestamps selected in the index pattern • The timezone can be changed by explicitly defining it in the parameter dateFormat:tz • If the documents does not contain timezone in the time field used, then the documents won't be changed by Kibana
Format • Number can be formatted differently depending of what they are representing: ! bytes ! duration ! percentage ! color ! ...
• It can be defined at the field level in the index pattern view pattern view or for all the fields directly in the advanced settings
Locale Number • Number from country to country are not represented the same way, way, for instance, the number 30,000: ! in French comma are used to represent the decimal so this number will be read as thirty
! in English speaking country this number will be read as thirty thousand
• It is possible to specify a locale representation of the number in Kibana by changing the parameter format:number:defaultLocale to format:number:defaultLocale to a specific country • Once done 3,000.01 3,000.01 will will be displayed as: ! 3 000,01 in French ! 3,000.01 in English
Displayed Documents • By default only the first 500 documents are displayed in the discover interface, it is possible to increase this number: ! updating the parameter discover:sampleSize to a bigger value allows the display of more documents
! be careful displaying too many document may have negative effects on the user experience
• Meta-fields can be removed from the discover interface: ! keep only the _source in the metafields ! the _source field is the original document that was provided to Elasticsearch
Search • Remember the Lucene syntax and the autocompletion feature? ! it can be used to search with wildcards .... ! ... but it is more expensive than a regular search ! especially leading wildcards are very expensive
• Leading wildcards can be disabled with two parameters: ! query:allowLeadingWildcards for the autocompletion ! query:queryString:options for the Lucene syntax
• Inside the query:queryString:options query:queryString:options parameter parameter a default field for search can be set as well: ! by default the search bar is searching on all the fields
Demo
Lesson Review
Summary • The time picker can be customized by adding pre-defined time intervals • Scripted field is a way to create a new field that is not initially present in your data: ! scripted fields have a significant computing cost ! if used a lot, that should be mentioned to the person indexing te data inside Elasticsearch
• Numbers can be formatted to reflect local representation of numbers
Quiz 1. True or False: False: Currencies representation are linked to the local? 2. True or False: False: It is not possible to search over scripted fields
3. What are 3 things things to be aware of when when using field scriptin scripting? g?
Lab 19
Reporting and Saved Objects
Generating Report • One of the many feature that the Elastic license is coming with is the ability for Kibana to generate reports by simply clicking on a button. • Reports can be generated using: ! the discovery interface, to generate a CSV that will contains the documents that your search is retrieving. It limits the results to the columns you defined.
! the dashboard interface to generate a pdf of the dashboard
• Once a report has been generated it is possible to retrieve it in Management -> Reporting
Reporting Automation • When it comes to reporting, automation is always appreciable • Using watcher it is possible to generate report automatically: ! be careful, the interval of the generation of reports need to be higher than the time it takes for a report to be generated
• Watcher can be technical, but using the following example make it easier to set it up: ! https://www.elastic.co/guide/en/kibana/current/automating-reportgeneration.html
Saved Objects • A lot of things that are going to be created in Kibana are objects: ! index pattern ! visualization ! dashboard ! search
• Once those objects are saved they will be appearing in Management -> Saved Objects • Those saved objects are extremely important because it is possible to export them and import others into our current Kibana • Internally saved objects are stored inside an Elasticsearch index
Saved Objects Relationship • Saved objects have relationships between each other for instance a saved object (dashboard) will contains other saved objects (visualizations) • Those relationships can be seen in the UI • Seeing the relationship helps when taking decisions like removing one saved object. It will give a warning that deleting a specific saved object may impact other saved objects
Demo
Lesson Review
Summary • Dashboards, searches and canvas have reporting options. They can be downloaded as a PDF or as a CSV • Saved objects are stored inside an index in Elasticsearch • Using watcher it is possible to automate the creation of report
Quiz 1. True or False: False: Saved objects are stored inside Kibana 2. True or False: False: Alerting can be used to automate the report generation 3. True or False: False: Kibana can be used to look at the dependencies between saved objects
Lab 20
Security
Security Considerations • The Elastic Stack does not offer authentication authentication or authorization by default • Make sure your stack is secure before going live: ! Elastic Security
• With Elastic Security it is possible to define authorization for specific users using the following concepts: ! ressources ! privilege ! permissions ! role ! user 279
Elastic Security • Enables Security Security (Gold/Platinum): (Gold/Platinum): Gold Encrypted communications Role-based access control Audit Logging IP filtering Native, LDAP, LDAP, PKI, Active Directory authentication SAML authentication Attribute-based access control Field and document level security Third-party authentication Encryption at rest support
Platinum
Ressource, Privilege and Permission • A resource is an entity for which the access will be restricted, it can be for example a document or an index for instance • Privileges are a groups representing one or more actions that a user may execute against one of the above defined resources • For instance: ! being able to read the data from a specific twitter index index Privilege
Ressource
• A permission is a set of one or more privileges applied on one or more resources 281
Privileges • There is two different kind of privileges that can be given to a user: ! cluster privileges ! indices privileges
• The cluster privileges are given when a user need to do operation at the cluster level, for instance monitoring a cluster,, managing users, etc .... cluster • The indices privileges are given to users that will need to perform operation on specific indices, will a user be able to read data from a specific index? Write data?
282
User and Role • Both roles and users can be defined directly in Kibana: ! a role will be a named set of permission ! there is multiple built-in roles
• A user will be defined by different characteristics: ! username ! full name ! email address ! ... and a set of roles
Built-in Users • There are already many built-in user • The kibana_dashboard_only_user is is a role that restrict a specific user: ! to only have access to the dashboard tab ! all edit option will be hidden
• The kibana_dashboard_only_user is a role and that means that the resources that the user should be able to access still need to be defined
Demo
Lesson Review
Summary • Users and roles can be defined through the Kibana interface • Kibana shipped with default users and roles including kibana_dashboard_only_user
Quiz 1. True or False: False: The role kibana_dashboard_only_user allows a user to see all the data in all the dashboards 2. How do you create create a user user that will will be able to see see only the dashboard tab and the data from the index *ecommerce*?
Lab 21
Spaces
Concept • Spaces is a feature that has been added in the 6.5 6.5 release release • Spaces are enabled by default, it is possible to disable them by defining in the configuration file: ! xpack.spaces.enabled: false
• Spaces are a way of grouping visualization, dashboards and other saved objects into meaningful categories • All the objects created inside a specific space will only be accessible through this space • When security is enabled it is possible to define which spaces a user will be able to access • Saved objects from a specific space will labeled as such inside the saved object.
Space Creation • Kibana is creating a default space called Default • If upgrading from a version < 6.5 to 6.5 to > 6.5 all the visualizations will be stored in this default space • Spaces only require a name to be created • A URL will be generated base on the name of the space to it is possible to customize his URL • Every space must have a unique space identifier (the part that will be added to the Kibana URL) • A description can be added to the space to explain what is the purpose this space
Spaces and Roles • Spaces coupled well with security as they can be combined together to define multiple tenants: ! roles can be defined in way that limit their access only to certain spaces
! because there is a space differentiator inside the saved objects, spaces can be used to limit what visualizations a user is having access to
Demo
Lesson Review
Summary • Spaces allows the separation of saved objects • Spaces and security can be used together to avoid unauthorized people to see the data from another group • A role can have access to one or more spaces
Quiz 1. Y 1. Yes es or No: No: If a user A with the role R1 is having read access to the space S1, can this user create new visualization in that space? 2. Y 2. Yes es or No: No: If a user A with the role R1 is having all permission access to the space S1, can this user see all the data inside the space S1?
Lab 22
Conclusions
Elastic Training Empowering Your People Immersive Learning Lab-based exercises and knowledge checks to help master new skills
N O I T A D N U O F
Solution-based Curriculum Real-world examples and common use cases Experienced Instructors Expertly trained and deeply rooted in everything Elastic Performance-based Certification Apply practical knowledge to real-world ‹#› use cases, in real-time
S N O I T A Z I L A I C E P S
LOGGING
METRICS
APM
ADVANCED SEARCH
SECURITY ANALYTICS
DATA SCIENCE
Thank you! Please complete the online survey
Quiz Answers
Unit 1 Quiz Qu iz Answers 1. Elastic Elasticsearch search,, Kibana, Beats, Beats, Logstash Logstash 2. Fa Fals lse e 3. * c* co* coo* cook* cooki* cooki* cookin* cookin* cooking* cooking* cooking_* cooking_* 4. Left: Time Time series dataset dataset Right: Static Static dataset
303
Unit 2 Quiz Qu iz Answers 1. The time picker picker and and the index patter pattern n 2. AND, OR, NOT 3. user:c user:claire laire AND age:<30 AND category:a category:a
304
Unit 3 Quiz Qu iz Answers 1. Met Metric rics s and bucket bucket 2. Tru rue e 3. Categor Categories ies A and B are generated using a bucket aggregation, countries US, FR, DE are generated using a sub-bucket aggregation and the size of the slices of the pie are generated using a metrics aggregation.
305
Kibana Data Analyst Course: Kibana Data Analyst Version 6.5.0 © 2015-2019 Elasticsearch BV. All rights reserved. Decompiling, copying, publishing and/or distribution without written consent of E lasticsearch BV is strictly prohibited.
306
