Ad Prediction using Click Through Rate and
Machine Learning with Reinforcement Learning
2021 Fourth International Conference on Electrical, Computer and Communication Technologies (ICECCT) | 978-1-6654-1480-7/21/$31.00 ©2021 IEEE | DOI: 10.1109/ICECCT52121.2021.9616653
S.Ushanag
A.Lakshmanarao
Department of CSE
Department of Information Technology
University College of Engineering
Aditya Engineering College
Kakinada
Surampalem,India
JNTUK Kakinada, A.P, India
laxman1216@gmail.com
ushavasi582010@gmail.com
Abstract:- Predicting the click-through rate (CTR) is an
essential problem in enterprise systems such as online
advertising. It is a crucial factor of advertisements platforms.
It is fed into auctions to determine the final ranking of
advertising. Machine Learning techniques are often used to
tackle challenges involving human-computer interaction.
Almost every website on the internet displays advertisements.
Companies who want to promote their products use these
websites as a method of promotion. The goal is to determine
which of the company's several advertisement versions can get
the best conversion rate, i.e., the most number of ad clicks. The
major issue for firms that rely on ad revenue is ad placement
on websites. The placement of the ad has a significant impact
on whether or not the ad gets clicked. This kind of challenge
lends itself very well to Reinforcement Learning algorithms. In
this paper, we applied the machine learning approach for Ad
Prediction. We used a dataset from Kaggle and applied two
reinforcement learning algorithms Upper Confidence Bound,
Thompson Sampling for predicting Ad position based on ad
clicks and achieved a good prediction rate. All the
implementations are done in python.
Keywords—Click Through Rate, Machine Learning,
Reinforcement Learning, Kaggle, Python.
I. INTRODUCTION
Advertising via internet sources has become a significant
element in internet browsing. These advertisements are
generally paid for searches and are based on a keyword
auction idea. The company uses pay-per-click advertising
with cost-per-click billing. In the recent digital world, CTR
(Click Through Prediction) got lots of attention. Online
advertising is a massive sector with a market value of more
than $50 billion. Because of focused advertising, internet
advertisers are increasing. While major corporations can
have capacity to consult personally in coordination to
professional advertisers, minor scale organizations outsource
their internet advertising to ad networks such as those
supplied by Facebook, Google, and others. These ad
networks deliver dynamic, rich, and appealing material with
links to promote, and they reward their customers based on
consumer clicks on those links.To boost their revenue, most
e-commerce and search engine businesses are looking for
tailored advertising. In general, internet advertising displays
may be seen as different parties like media, marketers, and
consumers. In the realm of internet advertising, one of the
most important issues is how to promote to certain user
groups. Inappropriate ads might degrade the user experience.
Advertising does not always have the desired impact, and the
media may be influenced as well. Text advertising on the
internet is generally in the form of text, and marketers can
buy media adverts using a cost-per-click (CPC) model. CTR
is an indicator of online web users who intend to click
Authorized licensed use limited to: Istinye Universitesi. Downloaded on February 13,2023 at 16:14:18 UTC from IEEE Xplore. Restrictions apply.
B.Sundara Leela
Department of CSE
University College of Engineering
Kakinada
JNTUK Kakinada, A.P, India
sundaraleela.b@gmail.com
advertising when they view advertisements on their websites.
It's a ratio of the number of users that clicked on the ad to the
number of times it was shown. A higher CTR value is always
important when it comes to improving a company's income.
Machine Learning is a field of study where predict by
learning is famous in recent years. Machine Learning also
has success stories in the advertising industry. In the internet
serving of ads, machine learning has played a significant
role. There has been a lot of studies done with Machine
Learning in this area. The issue with displaying the ad in the
same location is that after some time, the user will begin to
ignore the space since he is accustomed to seeing ads in that
location. As a result, he will begin to ignore that specific
position in the future. As a result, the number of ad clicks
will decrease. The problem with the first approach, which is
to place them at random, is that it ignores the best places. We
applied machine learning for solving this issue. In this paper,
we used reinforcement learning models for ad click position
prediction.
II. PREVIOUS WORK
Applying machine learning to the advertising industry is
not a new idea.in the past, several authors applied Machine
Learning and deep learning for ad click prediction.
M.J.Effendi [1] et.al used a machine learning regressor for ad
click prediction and achieved a good prediction rate. First,
they applied feature selection techniques for the reducing
size of the dataset. Later, with reduced features and Linear
Regression, they achieved a good R-squared value.
R.Kumar[2] et.al proposed a classifier(logistic regression
classifier) for ad click prediction and achieved good results.
W. Deng [3] et.al proposed a deep learning model for ad
click prediction. They applied LSTM methodology and
achieved a good detection rate. As RNN is time-consuming,
they also proposed a simplified version of RNN to reduce the
time complexity. B.Liu [4] et.al proposed FGCNN (Feature
Generation by Convolutional Neural Network). Their
proposed model contains two steps. In the first step, features
are generated. In the second step, the deep classifier is
applied. I. Islek [5] discussed the difficulties faced while
applying machine learning algorithms for ad predictions.
R. Xu [6] et.al proposed a new architecture
named OCDBN (Optimally Connected Deep Belief Net).
They claimed that their new architecture outperforms
previous models for ad click prediction. T. Niu [7] et.al
proposed Convolutional Neural Networks for CTR
prediction. The proposed CNN is based on density matrix. S.
Zhang [8] et.al proposed two extreme learning models
namely WO-ELM(Weight ELM) and W-ELM(Weighted
ELM) for CTR prediction and achieved good results. A.
Lakshmanarao [9] et.al proposed a voting classifier and
achieved 98% accuracy. Bakhtyari [10] et.al proposed the
XDBoost model for advertisement click prediction. XDBoost
combines boosting techniques with deep factorization
machines. First, they identified and removed redundant data,
and later with XDBoost, they achieved a good accuracy rate.
K. Kim [11] et.al proposed a new model named Deep User
Segment Interest Network. With their novel architecture,
they outperform previous models. X. Wang [12] et.al
discussed various works done in CTR prediction for online
advertisements.

III. PROPOSED METHODOLOGY

First, we collected a “Ads_CTR_Optimisation” dataset
from Kaggle [13]. The dataset contains information about 10
ad clicks. After that, we checked for missing values in the
data. There are no missing values in the data. Later, we
applied two reinforcement learning algorithms for
predictions. The proposed framework is shown in figure-1.

A. Dataset Details

The dataset (from Kaggle) contains 10000 user

clicks(rows) with 10 Advertisements information. All the
values in the dataset contain values of 0 and 1.0 means
Advertisement is not clicked.1 means Ad is clicked. The
number of 0’s and 1’s for each Advertisement details are
shown in the Table-I. From Table-I, it is clearly observed
that for all 10 advertisements number of 0’s is more than
number of 1’s.That means number of clicks are less when
compared to unclicks.

B. Reinforcement Learning

Machine Learning includes the field of reinforcement

learning. It's all about taking the right steps to maximize
rewards in a given circumstance. It is used by a variety of
software and computers to determine the best feasible action
or path in a given situation. In supervised learning, the
dataset is categorized into two sets(train, test) The first set
data is utilized for learning phase, later a model is created
and this model is used for testing. In reinforcement learning,
there is no answer and the reinforcement agent determines
what to do to complete the job. It will learn from its
experience if there isn't a training dataset available. The main
components of this learning are Input, Output, Training.
I/P: This is starting state for the model to work from.
O/P: There are several alternative outputs, just as there are
numerous solutions to a given issue.
Training: The model will return a state depending on the
input, and the user will decide whether to reward or penalize
the model based on its output. Fig. 1. Proposed Framework for Ad click prediction
The model is always evolving. Based on the maximum
reward, best solution is decided.
C. Upper Confidence Bound Steps in UCB:
Step 1:In every round n,for each ad i,there are two
UCB is one of old and good reinforcement learning numbers:
algorithm. UCB action selection balances exploration versus
Ni(n) is no. of Times the ad selected until round n,
exploitation by using uncertainty in action-value estimations.
Ri(n) is sum of rewards of ad i until round n.Random
Selection Model.

Authorized licensed use limited to: Istinye Universitesi. Downloaded on February 13,2023 at 16:14:18 UTC from IEEE Xplore. Restrictions apply.
 Step 2: Computer following:
I. Average reward of ad i until round n.
ii.The confidence interval at round n.
Step 3: Select the ad i with maximum of average reward
+ confidence interval.

TABLE I. DETAILS OF DATASET

class value Number

0’s 8297
Ad1 1’s 1703
0’s 8705
Ad2 1’s 1295
Fig. 2. Actual cliks in the dataset
0’s 9272
Ad3 1’s 728
0’s 8804 A. Applying Random Selection
Ad4 1’s 1196 We would put the advertisements at random in certain
0’s 7305 places Random Selection function now replicate the identical
Ad5 1’s 2695 scenario. The random selection technique is given in Alg-
0’s 9874 1.The random selection algorithm cannot apply any logic,
it’s just randomly giving preferences to all the 10
Ad6 1’s 126
advertisements, so there is no guarantee that we could
0’s 8888 achieve better results.
Ad7 1’s 1112
0’s 7909 Alg-1:Random Selection
Ad8 1’s 2091 m = 10000
0’s 9048 x = 10
Ad9 1’s 952 seleted_ads = []
0’s 9511 reward-total = 0
Ad10 for n in range(0, m):
1’s 489
a = random. randrange(x)
D. Thompson Sampling seleted_ads. append(a)
rew = dataset. values[n, a]
Thompson Sampling is a method that uses exploration and reward-total =reward-total + rew
exploitation to increase the average rewards earned from
completing a task. Exploration is when an activity is
repeated numerous times, while exploitation is when further After executing a random selection algorithm with 1000
actions are undertaken to maximize the return depending on rounds, it is observed that Ad-9 is shown as the best Ad.
the consequences of the previous actions, either rewards or (Fig.3). We also applied random selection with 3000 and 500
penalties. In other words, new options are investigated to rounds, But it cannot identify the best Advertisement in any
maximize rewards while utilizing previously investigated case.
options.

Steps in Thompson Sampling:

Step 1:In every round n,for each ad i,there are two numbers:
Ni1(n) is no. of times the ad selected until round n,
Ni0(n) no. of times the ad not selected until round n.
Step 2: For every ad i,take random draw from distribution
and computer Theta(n).
Step 3: Select the ad i with maximum Theta(n).
IV. EXPERIMENTATION
We collected a dataset from Kaggle with 10 ads. Each ad
contains 10000 rows of data(10000 users) as clicked(1) or
not clicked(0).We need to find the best ad, which means the Fig. 3. Results of Random Selection
Ad with highest clicks. Before applying algorithms, we
B. Applying Upper Confidence Bound
identified the best ad as Ad-5(Fig.2).First we applied
Random Selection algorithm followed by two reinforcement Rather of choosing an arbitrary action with a fixed
learning algorithms namely Upper Confidence Bound, probability, the UCB algorithm adjusts the mix of
Thompson Sampling. exploration and exploitation as it learns more about the
environment It shifts from a focus on exploration, where

Authorized licensed use limited to: Istinye Universitesi. Downloaded on February 13,2023 at 16:14:18 UTC from IEEE Xplore. Restrictions apply.
activities that have been tried the fewest times are favored, to
a focus on exploitation, where the action with the largest
anticipated payoff is chosen. We applied Upper Confidence
Bound three times with 3000 rounds,1000 rounds,500
rounds. The plots for UCB was shown in Figure-4. From
Fig.4, it is observed that UCB has done well with 3000
rounds,1000 rounds. In both cases, it is shown that Ad-5 is
better. But, with 500 rounds, it is shown that Ad-8 is better.
Fig. 4. Results of UCB with 3000,1000,500 rounds
C. Applying Thompson Sampling
Thompson is a heuristic for deciding what to do in the multi-
armed bandit issue that solves the exploration-exploitation
conundrum.It entails selecting the course of action that
Authorized licensed use limited to: Istinye Universitesi. Downloaded on February 13,2023 at 16:14:18 UTC from IEEE Xplore. Restrictions apply.
maximizes the predicted payoff in comparison to a randomly
generated belief. We applied Thompson sampling with 3000
rounds,1000 rounds,500 rounds. The plots are shown in
Fig.5.
From Fig.5, it clearly observed that Thompson Sampling
identified Ad-5 as best Ad in all three cases (3000
rounds,1000 rounds,500 rounds.
Fig. 5. Results of Thompson Sampling with 3000,1000,500 rounds
D. Comparing Results
After applying two reinforcement learning algorithms,
we compared the results. For comparison, consider Fig 5,
which shows an actual number of clicks in the dataset. From
Figure-5, it is observed that users clicked Ad-5 the greatest
number of times. Thompson Sampling gave priority to Ad-5.
So, Thompson Sampling gave better prediction in two
algorithms.
V. CONCLUSION
In this paper, the authors utilized a Ads data(Kaggle).It
contains 10 ads with 10000 user clicks data. Then, we
applied random selection algorithm for identifying best Ad.
We applied random selection three times with
3000,1000,500 rounds.But random selection algorithm was
unable to identify best Ad. Later, we applied Upper
Confidence Bound three times with 3000,1000,500 rounds.
UCB given best results with 3000,1000 rounds, but unable to
handle 500 rounds. Finally, we applied thompson sampling
with 3000,1000,500 rounds. Thompson Sampling identified
the best Ad in all three cases correctly. So, Thompson
sampling given correct results when compared to UCB for
Ad prediction
REFERENCES
[1] M.J.Effendi,S.Abbas Ali, “Click Through Rate Prediction for
Contextual Advertisment Using Linear Regression,”
https://arxiv.org/abs/1701.08744v1-2016.
[2] R. Kumar,Naik et.al, "Predicting clicks: CTR estimation of
advertisements using Logistic Regression classifier,” 2015 IEEE
International Advance Computing Conference -IACC, 2015, pp.
1134-1138, doi: 10.1109/IADCC.2015.7154880.
[3] W.Deng,X. Ling,Qi et.al, “Ad Click Prediction in Sequence with
LSTM Networks: An Externality-Aware Model,” International
Conference(ACM) on R&D in Information Retrieval Association for
Computing Machinery-2018, (pp. 1065–1068).
Authorized licensed use limited to: Istinye Universitesi. Downloaded on February 13,2023 at 16:14:18 UTC from IEEE Xplore. Restrictions apply.
[4] B.Liu et.al,“Feature Generation by Convolutional Neural Network for
Click-Through Rate Prediction,” World Wide Web Conference-2019.
[5] I. Işlek, E. Karamatlı and A. T. Cemgil, "Large scale ad click
prediction system,” 2018 26th Signal Processing and
Communications Applications Conference (SIU), 2018, pp. 1-4, doi:
10.1109/SIU.2018.8404392.
[6] R. Xu, M. Wang and Y. Xie, "Optimally Connected Deep Belief Net
for Click Through Rate Prediction in Online Advertising," in IEEE
Access, vol. 6, pp. 43009-43020, 2018, doi:
10.1109/ACCESS.2018.2861429.
[7] T. Niu ,Y. Hou, "Density Matrix Based Convolutional Neural
Network for Click-Through Rate Prediction," 2020 3rd International
Conference on Artificial Intelligence and Big Data -ICAIBD,pp. 46-
50,2020.
[8] S. Zhang, Z. Liu,W. Xiao, "A Hierarchical Extreme Learning
Machine Algorithm for Advertisement Click-Through Rate
Prediction," in IEEE Access, volume No-6, 641-647,doi:
10.1109/ACCESS.2018.2868998,2018.
[9] A.L.Rao,A.Srisaila,T.S.R.Kiran, “An Efficient Ad-Click Prediction
System using Machine Learning Techniques” , International Journal
of Engineering and Advanced Technology,Volume-9 Iss-3, 2020.
[10] M. Bakhtyari and S. Mirzaei, “Click Through Rate Prediction with
Feature Engineering Boosting Algorithms” , International Computer
Conf., 2021, pp. 1-5, doi: 10.1109/CSICC52343.2021.9420546.
[11] K. Kim, E. Kwon,Park, “Deep User Segment Interest Network
Modeling for Click-Through Rate Prediction of Online
Advertising”,volume-9,2021,pp. 9812-9821, IEEE Access ,2021.
[12] X. Wang, “A Survey of Online Advertising Click-Through Rate
Prediction Models”, International Conference on Information
Technology,Big Data and Artificial Intelligence -2020-ICIBA, 2020,
pp. 516-521, doi: 10.1109/ICIBA50161.2020.9277337.
[13] https://www.kaggle.com/akram24/ads-ctr-optimisation