Kibana guide and How To's

What is Kibana
Kibana is an open source data exploration & visualization tool used for log and time series analytics, application monitoring and operation intelligence.

It sits on the top of the ELK Stack(or Elastic Stack).

ELK Stack

(in order of process it should have been LEK but ELK sounds better)

Logstash
Tool for collecting and monitoring logs from remote machines.

It is a data pipeline for Elasticsearch.

Elasticsearch
An Apache Lucene based search engine.

It is open source and developed using Java.

Kibana
Enables the searching and interaction with data in Elasticsearch.

Allows performing advanced analytics and creation of reports.

Enable creation and sharing of dynamic dashboards that get updated in real time.

How it works
A really nice tutorial about more insights about Elasticsearch and Kibana : An introduction to elasticsearch with kibana .

Kibana features
Data Visualization
Vega Grammar
Geo spatial Data
Advanced Time Series Analysis
Graph Exploration
Anomalies Exploration
Dashboard Sharing
Visualization & Dashboard Exporting
Data Addition
 Users & Roles Control
Pipeline Management
Developer Console
Search Profiler
Grok Debugger

HudsonMX use case

"It’s time to get serious about analyzing the logs"

You can access HudsonMX Kibana here .

Main features useful for AM team

1. Discover;
2. Visualize;
3. Dashboard.

Elasticsearch and Kibana work with json objects

All information can be found in the form of json objects.

Discover
The essence of Kibana, the Discover functionality lets you do powerful searches on multiple log locations on different time periods (ranging from second
to even years - at long as you have data in the system).

You can interactively explore your data from the Discover page. You have access to every document in every index that matches the selected index
pattern. You can submit search queries, filter the search results, and view document data. You can also see the number of documents that match the
search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in
a histogram at the top of the page.

Search bar is case insensitive.

Searching for <<message : "ERROR">> or <<message : "error">> the result is the same.

Visualize
Visualize enables you to create visualizations of the data in your Elasticsearch indices. You can then build dashboards that display related visualizations.

Kibana visualizations are based on Elasticsearch queries. By using a series of Elasticsearch aggregations to extract and process your data, you can create
charts that show you the trends, spikes, and dips you need to know about.

Using the Visualize, you can create a wide variety of graphs, tables, pies and different metrics to visualize the data in any form you may need.

A new visualization can be created by either:

using a search saved from Discover ;

start with a new search query;
cloning an existing visualization.

You can search in both a visualization and a dashboard

Both the visualize tool and the dashboard have a search bar (exactly like the one in the discover functionality) - it this way you can create
overlapping searches with multiple and complex filters.

Dashboard
A Kibana dashboard displays a collection of visualizations and searches. You can arrange, resize, and edit the dashboard content and then save the
dashboard so you can share it.
Project specific Kibana objects and how to use them
In order to help us in our daily tasks we have created several objects in Kibana. All the objects names start with "[AM]" in order to be found easily by
searching in one category or another.

Crt. Type Name Main Use case Info

no.

1.1 Search [AM] Production All Errors - Search Fetches all error log messages across all the hosts that are in production
(Discove [DO NOT MODIFY] Base for all the (host names start with "prod").
r) visualization and
dashboard objects.
Do not use it or
modify it!

1.2 Search [AM] Smoketesting : Production Fetches all log messages across production PreBuy Main service and PreBuy
(Discove Main Smoketesting database.
r) production
Checking logs in prod
PreBuy Main + PreBuy
database

1.3 Search [AM] Smoketesting : Production Fetches all log messages across production PreBuy Main database.
(Discove Main (db only) Smoketesting
r) production
Checking logs only in
prod PreBuy database

1.4 Search [AM] Smoketesting : Production Fetches all log messages across production Comscore TV query services
(Discove Comscore Smoketesting excluding (prod-cstvfs01 and prod-cstvfs02 - comscore file processors)
r) production
Checking logs in prod
Comscore TV query
services

1.5 Search [AM] Smoketesting : Production Fetches all log messages across production Nielsen TV and Radio query
(Discove Nielsen Smoketesting services.
r) production
Checking logs only in
prod Nielsen TV and
Radio query services

1.6 Search [AM] Smoketesting : PreProd Main Fetches all log messages across preprod PreBuy Main service and PreBuy
(Discove Smoketesting preprod database.
r) Checking logs in
preprod PreBuy Main +
PreBuy database

1.7 Search [AM] Smoketesting : PreProd Main Fetches all log messages across production PreBuy Main database.
(Discove (db only) Smoketesting preprod
r) Checking logs only in
preprod PreBuy
database

1.8 Search [AM] Smoketesting : PreProd BETA Fetches all log messages across preprod beta PreBuy Main service
(Discove Main Smoketesting preprod and PreBuy database.
r) beta
Checking logs in
preprod beta PreBuy
Main + PreBuy
database

1.9 Search [AM] Smoketesting : Stage Main Fetches all log messages across staging PreBuy Main service.
(Discove Smoketesting stage
r) Checking logs in stage
PreBuy Main
2.1 Visualiza [AM] Production All Errors - Line Shows a graph of the evolution of the number on log errors in a time interval.
tion (Vis Visualization Used in all the
ualize) dashboards
Line visualization of the
errors over a time
period

2.2 Visualiza [AM] Production All Errors - Table Shows a table with the number of log errors grouped by the host names.
tion (Vis Visualization Used in all the
ualize) dashboards
Table visualization of
the errors by the host
name

2.3 Visualiza [AM] Production All Errors - Pie Shows a pie chart with the number of log errors grouped by the host names.
tion (Vis Chart Visualization Used in all the
ualize) dashboards
Pie chart visualization
of the errors by the
host name

2.5 Visualiza [AM] Production All Errors - Count Shows the total number of log errors.
tion (Vis Visualization Used in all the
ualize) dashboards
Count visualization of
the total number of the
log errors

3.1 Dashboa [AM] [1.1] [Production] [PreBuy] Shows all the visualizations and the list of log errors for production PreBuy
rd [Main] Errors - Dashboard Daily log error main service.
investigation

3.2 Dashboa [AM] [1.2] [Production] [PreBuy] Shows all the visualizations and the list of log errors for production PreBuy
rd [db] Errors - Dashboard Daily log error database.
investigation

3.3 Dashboa [AM] [2.1] [Production] [Nielsen] Shows all the visualizations and the list of log errors for production Nielsen
rd [Local TV] [Main] Errors - Daily log error Local TV services and database.
Dashboard investigation

3.4 Dashboa [AM] [2.2] [Production] [Nielsen] Shows all the visualizations and the list of log errors for production Nielsen
rd [Local Radio] Errors - Dashboard Daily log error Local Radio services and database.
investigation

3.5 Dashboa [AM] [2.3] [Production] [Comscore] Shows all the visualizations and the list of log errors for production Comscore
rd [Local TV & POLK] Errors - Daily log error Local TV & POLK services and database.
Dashboard investigation

3.6 Dashboa [AM] [2.4] [Production] [National Shows all the visualizations and the list of log errors for production National
rd TV] Errors - Dashboard Daily log error TV and databases.
investigation

3.7 Dashboa [AM] [3] [Production] [BrandCentral, Shows all the visualizations and the list of log errors for production Brand
rd SellerAssist, ScheduleAssist] Errors Daily log error Central (Agency Central) service, Seller Assist service and Schedule Assist
- Dashboard investigation service.

3.8 Dashboa [AM] [4] [Production] [LiveBuy] Shows all the visualizations and the list of log errors for production Live Buy
rd Errors - Dashboard Daily log error services and database.
investigation

3.9 Dashboa [AM] [5] [Production] [Portal & Shows all the visualizations and the list of log errors for production Portal
rd Service Registry] Errors - Daily log error Service and Service Registry.
Dashboard investigation

3.10 Dashboa [AM] [6] [Production] [Other] Errors Shows all the visualizations and the list of log errors for production Vendor
rd - Dashboard Daily log error API, NCC, Finance Assist (TO BE MOVED IN STAND ALONE DASHBOARD)
investigation and other smaller services.

3.11 Dashboa [AM] [7] [Production] [Reporting Shows all the visualizations and the list of log errors for production Delivery
rd Services] Errors - Dashboard Daily log error Reporting service.
investigation
All the dashboards are using this search object (1.1 [AM] Production All Errors - Search [DO NOT MODIFY]) and visualize objects (all 4) +
adding additional filtering.

Warning

Some objects might have additional filters that could change the final result.