Query Quake: Python's Approach to Anomaly Detection in Searches

1
T. Aditya Sai Srinivas
1
Jayaprakash Narayan College of Engineering

Abstract: Search Queries Anomaly Detection involves isolating queries deviating from
expected performance metrics, aiding businesses in uncovering issues or opportunities, like
unusually high or low Click-Through Rates (CTRs). This article guides readers through the
process of implementing Machine Learning for Search Queries Anomaly Detection using
Python. Learn how to discern outliers and enhance the capacity to identify critical patterns
within search query data, offering a comprehensive exploration of techniques to optimize
query analysis and strategically leverage anomalies for actionable insights in the dynamic
landscape of online search behavior.
Keywords: Anomaly Detection, Search Queries, Machine Learning(ML), Python,
Performance Metrics.
1. Introduction
Embarking on the journey of Search Queries Anomaly Detection involves a meticulous
process designed to uncover irregular or unexpected patterns within the realm of search query
data. This methodical approach unfolds in the following steps:
1. Data Gathering: Begin by amassing historical search query data from the chosen source, be
it a search engine or a website’s internal search functionality.
2. Initial Analysis: Conduct an in-depth preliminary analysis to discern the distribution of
search queries, their frequencies, and any discernible patterns or trends that may lay the
groundwork for anomaly detection.
3. Feature Engineering: Create pertinent features or attributes derived from the search query
data, strategically chosen to enhance the efficacy of anomaly detection.
4. Algorithm Selection: Opt for a suitable anomaly detection algorithm, with common
methods including statistical approaches such as Z-score analysis and machine learning
algorithms like Isolation Forests or One-Class SVM.
5. Model Training: Train the selected model using the meticulously prepared dataset, fine-
tuning its ability to recognize anomalies.
6. Application of the Model: Deploy the trained model to the search query data, utilizing its
acquired knowledge to pinpoint anomalies or outliers within the dataset.
In essence, the process commences with the collection of a robust dataset centered around
search queries, and an exemplary dataset for this task is available for download
https://statso.io/search-queries-anomalies-case-study/. This detailed workflow ensures a
comprehensive exploration of search query anomaly detection, facilitating actionable insights
in the dynamic landscape of search behavior analysis.
2. Implementation
Begin the journey into Search Queries Anomaly Detection by initiating the importation of
essential Python libraries and the dataset.

Before progressing further, let's examine the insights provided by the column:

Now, we will transform the CTR column from a percentage string format into a floating-point
representation:
Next, we will examine the prevalent words within each search query:

Now, let's examine the foremost queries based on clicks and impressions:

Next, we'll delve into an examination of queries showcasing the highest and lowest Click-
Through Rates (CTRs):
Now, let's examine the interplay and connections among various metrics:

Within this correlation matrix:

1. The relationship between Clicks and Impressions is positive, signifying that an increase in
Impressions tends to coincide with a rise in Clicks.
2. Clicks and CTR exhibit a weak positive correlation, indicating that an increase in Clicks
may marginally elevate the Click-Through Rate.
3. Clicks and Position demonstrate a weak negative correlation, suggesting that higher ad or
page Positions may lead to a decrease in Clicks.
4. Impressions and CTR showcase a negative correlation, suggesting that a surge in
Impressions is associated with a reduction in the Click-Through Rate.
5. Impressions and Position are positively correlated, implying that higher Positions result in
more Impressions.
6. CTR and Position display a strong negative correlation, revealing that higher Positions are
linked to lower Click-Through Rates.
 3. Uncovering Aberrations in Search Queries
Moving on to the process of identifying anomalies within search queries, numerous
techniques can be employed for this purpose. A notably straightforward and efficient method
involves the implementation of the Isolation Forest algorithm. This algorithm proves
effective across diverse data distributions and exhibits commendable efficiency, making it
particularly well-suited for handling extensive datasets.
Explore the identified anomalies to unravel their essence and ascertain whether they
genuinely signify outliers or if they stem from data errors. Here's the step-by-step guide to
dissecting the nature of the detected anomalies:

The anomalies present in our search query data transcend mere outliers; they serve as crucial
markers pointing towards potential areas for expansion, optimization, and strategic emphasis.
These anomalies act as mirrors reflecting emerging trends and burgeoning areas of interest.
Remaining attuned and responsive to these evolving trends is pivotal for preserving and
amplifying the website's relevance and fostering sustained user engagement.
4. Conclusion
Search Queries Anomaly Detection involves the identification of queries that deviate as
outliers based on their performance metrics. This process holds significant value for
businesses, serving as a vital tool to promptly identify potential issues or opportunities,
including instances of unexpectedly high or low Click-Through Rates (CTRs).
References
1. https://statso.io/search-queries-anomalies-case-study/
2. https://thecleverprogrammer.com/2023/11/20/search-queries-anomaly-detection-
using-python/?fbclid=PAAab_BX5CDNm-
qzVz8bXdPRRP7BbN3l5qmBXUKLG5rPJGlcfWjqmI0Iw9pZk
3. https://www.projectpro.io/article/anomaly-detection-using-machine-learning-in-
python-with-example/555
4. Alla, Sridhar, and Suman Kalyan Adari. Beginning anomaly detection using python-
based deep learning. New Jersey: Apress, 2019.
5. https://symbl.ai/developers/blog/performing-anomaly-detection-in-python/