Miroslav

Lessev

Monitoring Microsoft SQL Server using the

Elastic Stack
Sponsors
Gold Sponsors

Innovation Sponsor Trusted Partner

Global Sponsor PASS

Swag Sponsor
About me
qDatabase & Cloud Enthusiast

q17 years of experience in designing, developing,

implementing and supporting databases

qPhD Candidate & Teaching assistant

Faculty of Economics and Business Administration at University
of Sofia "St. Kliment Ohridski”

qDatabase Development Team Lead

Software Group
Agenda
q Search engines
q Elastic Search
§ Brief history
§ Ecosystem
§ Architecture
§ Strengths
q Monitoring MSSQL with Elastic stack
q Demo
Elastic Stack
 Database Engine Ranking

Source: DB-Engines Ranking, 2018

 Search Engine Ranking

Source: DB-Engines Ranking, 2018

Top Search Engines

There are about 20 search engines to choose from, but, if you

are looking for a reliable and efficient solution for your web
application.

For 2019 the following three: Elasticsearch, Solr and Sphinx –

that are at the top.
Elastic Search
Search Engines
Solr
is not as quick as Elasticsearch and works best for static data. The reason is due to
caches.
In Solr, the caches are global, which means that, when even the slightest change
happens in the cache, all indexing demands a refresh. This is usually a time-consuming
process. In Elastic, on the other hand, the refreshing is made by segments.
Sphinx
is good for structured data, but it is not the best choice for projects that deal with
unstructured data (DOCs, PDFs, MP3s etc.), as it takes developers a lot of time and
effort to configure.
This, together with other difficulties on configuration, makes Sphinx less comfortable
to use than its competitors.
 Search Engines Comparison

Elasticsearch Solr Sphinx

Types of 1. Full-text 1. Full-text 1. Full-text
Search 2. Autocomplete 2. Autocomplete 2. Autocomplete
Features suggestions suggestions suggestions
3. Faceted 3. Faceted 3. Faceted
4. Multifield 4. Multifield 4. Multifield
5. Synonyms 5. Synonyms 5. Synonyms (called wordforms)
6. Fuzzy 6. Fuzzy 6. Geospatial
7. Geospatial 7. Highlighting 7. Highlighting (called snippets)
8. Geospatial 8. Spell checker (called qsuggest)
9. Spell checker

Source: https://greenice.net/elasticsearch-vs-solr-vs-sphinx-best-open-source-search-platform-comparison/
 Search Engines Comparison
ElasticSearch Solr Sphinx

Real Time Indexing Yes Yes Yes

Performance High High High

Scalability High High High

Data Scheme Schema-free Yes, but dynamic Yes

Can be storage Yes Yes No

Visualization of Data Kibana Banana plugin No

Machine Learning Yes Yes No

Source: https://greenice.net/elasticsearch-vs-solr-vs-sphinx-best-open-source-search-platform-comparison/
Elastic Search Brief History

Elastic Search Company

Elastic Search Elastic Stack
was first renamed to
company Release 1.0
released 0.4 by Elastic and Release 7.1
formed 2014
Shay Banon in Release 2.0
2012 2019
2010 2015
The Elastic Stack Ecosystem
Basic Elastic Architecture
Elasticsearch Logical Structure
ElasticSearch Sample Cluster
Elasticsearch Node Roles
Ø MASTER-ELIGIBLE NODE
A node that has node.master set to true (default), which makes it eligible to be elected as the master node, which controls the
cluster. Minimum_master_nodes = N/2 + 1
Ø DATA NODE
A node that has node.data set to true (default). Data nodes hold data and perform data related operations such as CRUD, search,
and aggregations.
Ø INGEST NODE
A node that has node.ingest set to true (default). Ingest nodes are able to apply an ingest pipeline to a document in order to
transform and enrich the document before indexing. With a heavy ingest load, it makes sense to use dedicated ingest nodes and to
mark the master and data nodes as node.ingest: false.
Ø TRIBE NODE (Deprecated in 5.4)
A tribe node, configured via the tribe.* settings, is a special type of coordinating only node that can connect to multiple clusters and
perform search and other operations across all connected clusters.
Ø COORDINATING NODES
Coordinating only nodes can benefit large clusters by offloading the coordinating node role from data and master-eligible nodes.
They join the cluster and receive the full cluster state, like every other node, and they use the cluster state to route requests directly
to the appropriate place(s).
ELASTICSEARCH’S STRENGTHS
1. NEAR REAL-TIME INDEXING
Elasticsearch is able to index rapidly changing data almost instantly (in less than 1 sec). It is appropriate to use it in projects where
a database is constantly updating.
For example, in Uber, Elasticsearch aggregates business metrics on dynamic (surge) pricing and supply positioning, in real time. It
is able to handle more than 1,000 queries per second at peak time.
2. HIGH SCALABILITY
When the database grows, it becomes more difficult to look up. But Elasticsearch scales up while your DB gets bigger, so the search
speed does not slow down.
Expedia, one of the biggest hotel and airline ticket aggregators, provides seeking through up to 1TB a day with 300K events per
second. With the help of Elasticsearch, they managed to improve their customers’ booking experience.
3. STORAGE
ES can be used not only as an indexer, but also as a data storage. Nevertheless, we would not recommend using it as your primary
storage, and we still keep data in the main DB for better security and reliability, using ES only to index data and store logs.
4. VISUALIZATION OF DATA
This is one of the trendy features today that is perfectly implemented in ES. Elastic Stack (the combination of ES,
Logstash and Kibana plugins) makes a great tool for analytics. It allows for real-time monitoring of traffic on your application
(total number of visitors, number of unique visitors, IP addresses, most popular queries, most requested pages, device and browsers
used, traffic logs by the time of day, and much more).
ELASTICSEARCH’S STRENGTHS
5. SECURITY ANALYTICS
Elastic Stack is also a great security analytics tool. The near real-time log analytics and visualization
allow you to identify security threats (problems with a web server, broken links, attempts for unauthorized
access, attack locations, etc.). You can learn more from this official Elastic.co video.
By migrating to ES, Dell increased their security by ensuring only authorized people could access their
cluster. Dell also reduced the number of their servers by 25-30%.

6. MACHINE LEARNING
Elasticsearch can benefit from Machine Learning features provided by the X-Pack commercial plugin.
The Machine Learning algorithms are focused on anomaly detection and outlier detection in time series
data.

7. AMAZON ELASTICSEARCH SERVICE

Amazon Elasticsearch Service allows for quick and easy setup and operates and scales Elasticsearch in
the cloud without the necessity to configure your own servers.
Data Definition
q Elasticsearch is “schemaless”

q But it provides defining schema – mapping

q Very important when setting up for search:

§ Data types: string, integer, float, date/tst, boolean, binary, array, object,
nested, geo, attachment

§ search analysers, boosting, etc.

Querying data
q Elasticsearch query
curl -XGET 'http://localhost:9200/maindex/_search?q=management.name:petar‘

q SQL query
select * from company
where exists(
select 1 from comp_management cm
inner join peron p on p.id=cm.id_pers
where lower(p.name) like '%peter%');
Demo
MSSQL Performance Counters
• page_splits_per_sec
• lock_waits_per_sec
• user_connections
• transactions
• active_temp_tables
• connections_reset_per_sec
• logins_per_sec
• logouts_per_sec
• recompilations_per_sec
• compilations_per_sec
• batch_requests_per_sec
• cache_hit.pct
• page_life_expectancy.sec
• buffer.checkpoint_pages_per_sec
• buffer.database_pages
• buffer.target_pages
Kibana
MSSQL Transaction Log metric set
• space_usage - This shows space usage information about the
transaction log, a key metric to understand the performance
of a Microsoft SQL Server. It also shows the time passed since
the last backup of the log, useful for disaster recovery

• stats - This returns summary level attributes and information

on transaction log files of databases. Use this information for
monitoring and diagnostics of transaction log health. Here you
can see active size, total size, or backup time for example.
Kibana
Create network and start MSSQL
• docker network create demo

• docker run -d --net=demo -e 'SA_PASSWORD=MyDemoP@ss' -e

'MSSQL_PID=Developer' -e 'ACCEPT_EULA=Y' -p 1433:1433 --name
demo-mssql microsoft/mssql-server-linux:latest
Deploy Elastic Stack
• docker network create demodocker run -d --net=demo --name demo-
elastic -p 9200:9200 -e "discovery.type=single-node"
elasticsearch:7.1.1

• docker run -d --net=demo --name demo-kibana -p 5601:5601 -e

ELASTICSEARCH_HOSTS:http://demo-elastic:9200 kibana:7.1.1

• docker run -d --net=demo --name demo-beat -v

/Users/mlessev/demo/metricbeat.yml:/usr/share/metricbeat/metricb
eat.yml docker.elastic.co/beats/metricbeat:7.1.1
ELASTICSEARCH’S Success Stories
Many world-known companies use Elastic for their applications. Here you can find
such names as TripAdvisor, Shopify, Mozilla, Foursquare, Etsy, Github, SoundCloud,
eBay, Yelp and Netflix among others.

With the help of ES, The Guardian got a powerful analytics system that is able to
process 40 million documents per day to create a vision of how content is consumed.

At Netflix, with 8 million events and 24GB per second during peak hours, ES is used for
real-time analytics of events like video viewing activities, UI activities, error logs,
performance, diagnostic events, etc.
References
Ø https://www.elastic.co/blog/monitoring-microsoft-sql-server-using-metricbeat-and-elasticsearch
Ø https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-mssql-
performance.html#metricbeat-metricset-mssql-performance
Ø https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-mssql.html
Ø https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-metricset-mssql-performance.html
Ø https://www.elastic.co/guide/en/beats/metricbeat/7.0/metricbeat-metricset-mssql-transaction_log.html
Ø https://db-engines.com/en/ranking
Ø https://www.slideshare.net/objectrocket/elasticsearch-and-the-database-market
Ø https://greenice.net/elasticsearch-vs-solr-vs-sphinx-best-open-source-search-platform-comparison/
Ø https://www.elastic.co/about/history-of-elasticsearch
Ø https://www.slideshare.net/objectrocket/exploring-mongodb-elasticsearch-better-together
Ø https://www.slideshare.net/clintongormley/elasticsearch-query-dsl-not-just-for-wizards?qid=604e3c4e-cac7-
46b3-ac19-d57b341416a4&v=&b=&from_search=24
Sponsors
Gold Sponsors

Innovation Sponsor Trusted Partner

Global Sponsor PASS

Swag Sponsor