Experiment No: 7 Date:

Title: Implementation of KNN using sklearn.

Objective:
• To understand the basic python library as numpy, Pandas.
• To understand more about sklearn and importing dataset using sklearn.
Theory/Description:

❖ K-Nearest Neighbor(KNN) Algorithm for Machine Learning

o K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on

Supervised Learning technique.
o K-NN algorithm assumes the similarity between the new case/data and available cases and
put the new case into the category that is most similar to the available categories.
o K-NN algorithm stores all the available data and classifies a new data point based on the
similarity. This means when new data appears then it can be easily classified into a well suite
category by using K- NN algorithm.
o K-NN algorithm can be used for Regression as well as for Classification but mostly it is used
for the Classification problems.
o K-NN is a non-parametric algorithm, which means it does not make any assumption on
underlying data.
o It is also called a lazy learner algorithm because it does not learn from the training set
immediately instead it stores the dataset and at the time of classification, it performs an action
on the dataset.
o KNN algorithm at the training phase just stores the dataset and when it gets new data, then
it classifies that data into a category that is much similar to the new data.
o Example: Suppose, we have an image of a creature that looks similar to cat and dog, but we
want to know either it is a cat or dog. So for this identification, we can use the KNN algorithm,
as it works on a similarity measure. Our KNN model will find the similar features of the new
data set to the cats and dogs images and based on the most similar features it will put it in
either cat or dog category.

❖ Why do we need a K-NN Algorithm?

o Suppose there are two categories, i.e., Category A and Category B, and we have a new data
point x1, so this data point will lie in which of these categories. To solve this type of problem,
we need a K-NN algorithm. With the help of K-NN, we can easily identify the category or class
of a particular dataset. Consider the below diagram:
 ❖ How does K-NN work?

The K-NN working can be explained on the basis of the below algorithm:

o Step-1: Select the number K of the neighbours

o Step-2: Calculate the Euclidean distance of K number of neighbours
o Step-3: Take the K nearest neighbours as per the calculated Euclidean distance.
o Step-4: Among these k neighbours, count the number of the data points in each category.
o Step-5: Assign the new data points to that category for which the number of the neighbour
is maximum.
o Step-6: Our model is ready.

Suppose we have a new data point and we need to put it in the required category. Consider the
below image:

o Firstly, we will choose the number of neighbors, so we will choose the k=5.
o Next, we will calculate the Euclidean distance between the data points. The Euclidean
distance is the distance between two points, which we have already studied in geometry. It
can be calculated as:

o By calculating the Euclidean distance we got the nearest neighbours, as three nearest
neighbours in category A and two nearest neighbours in category B. Consider the below
image:
 o As we can see the 3 nearest neighbours are from category A, hence this new data point must
belong to category A.

❖ How to select the value of K in the K-NN Algorithm?

Below are some points to remember while selecting the value of K in the K-NN algorithm:

❖ There is no particular way to determine the best value for "K", so we need to try some values
to find the best out of them. The most preferred value for K is 5.

o A very low value for K such as K=1 or K=2, can be noisy and lead to the effects of outliers in
the model.
o Large values for K are good, but it may find some difficulties.

❖ Advantages of KNN Algorithm:

o It is simple to implement.
o It is robust to the noisy training data
o It can be more effective if the training data is large.

❖ Disadvantages of KNN Algorithm:

o Always needs to determine the value of K which may be complex some time.
o The computation cost is high because of calculating the distance between the data points
for all the training samples.

❖ Steps to implement the K-NN algorithm:

o Data Pre-processing step

o Fitting the K-NN algorithm to the Training set
o Predicting the test result
o Test accuracy of the result(Creation of Confusion matrix)
o Visualizing the test set result.

❖ Compute KNN distance metrics

o To recap, the goal of the k-nearest neighbour algorithm is to identify the nearest neighbours of a
given query point, so that we can assign a class label to that point. In order to do this, KNN has a
few requirements:
❖ Determine your distance metrics

o In order to determine which data points are closest to a given query point, the distance between the
query point and the other data points will need to be calculated. These distance metrics help to form
decision boundaries, which partitions query points into different regions. You commonly will see
decision boundaries visualized with Voronoi diagrams.

While there are several distance measures that you can choose from, this article will only cover the
following:
o Euclidean distance (p=2): This is the most commonly used distance measure, and it is limited to
real-valued vectors. Using the below formula, it measures a straight line between the query point
and the other point being measured.

Euclidean distance formula

o Manhattan distance (p=1): This is also another popular distance metric, which measures the
absolute value between two points. It is also referred to as taxicab distance or city block distance as
it is commonly visualized with a grid, illustrating how one might navigate from one address to
another via city streets.
 Manhattan distance formula

o Minkowski distance: This distance measure is the generalized form of Euclidean and Manhattan
distance metrics. The parameter, p, in the formula below, allows for the creation of other distance
metrics. Euclidean distance is represented by this formula when p is equal to two, and Manhattan
distance is denoted with p equal to one.

Minkowski distance formula

o Hamming distance: This technique is used typically used with Boolean or string vectors,
identifying the points where the vectors do not match. As a result, it has also been referred to as the
overlap metric. This can be represented with the following formula:
 Hamming distance formula

o As an example, if you had the following strings, the hamming distance would be 2 since only two of
the values differ.

Hamming distance example

❖ Compute KNN: defining k

o The k value in the k-NN algorithm defines how many neighbors will be checked to determine the
classification of a specific query point. For example, if k=1, the instance will be assigned to the
same class as its single nearest neighbor.
o Defining k can be a balancing act as different values can lead to overfitting or underfitting. Lower
values of k can have high variance, but low bias, and larger values of k may lead to high bias and
lower variance.
o The choice of k will largely depend on the input data as data with more outliers or noise will likely
perform better with higher values of k. Overall, it is recommended to have an odd number for k to
avoid ties in classification, and cross-validation tactics can help you choose the optimal k for your
dataset.

❖ k-nearest neighbours and python

o To delve deeper, you can learn more about the k-NN algorithm by using Python and scikit-learn
(also known as sklearn),which also contains other popular libraries, like NumPy, pandas, and
Matplotlib.
o The following code is an example of how to create and predict with a KNN model:

o from sklearn.neighbors import KNeighborsClassifier

model_name = ‘K-Nearest Neighbor Classifier’
knnClassifier = KNeighborsClassifier(n_neighbors = 5, metric = ‘minkowski’, p=2)
knn_model = Pipeline(steps=[(‘preprocessor’, preprocessorForFeatures),
(‘classifier’, knnClassifier)])
knn_model.fit(X_train, y_train)
y_pred = knn_model.predict(X_test)

❖ Applications of k-NN in machine learning

o The k-NN algorithm has been utilized within a variety of applications, largely within classification.
Some of these use cases include:

o - Data pre-processing: Datasets frequently have missing values, but the KNN algorithm can
estimate for those values in a process known as missing data imputation.
 o - Recommendation Engines: Using clickstream data from websites, the KNN algorithm has been
used to provide automatic recommendations to users on additional content. This research (link
resides outside of ibm.com) shows that the a user is assigned to a particular group, and based on that
group’s user behaviour, they are given a recommendation. However, given the scaling issues with
KNN, this approach may not be optimal for larger datasets.

o - Finance: It has also been used in a variety of finance and economic use cases. For example,
one paper (PDF, 391 KB) (link resides outside of ibm.com) shows how using KNN on credit data
can help banks assess risk of a loan to an organization or individual. It is used to determine the
credit-worthiness of a loan applicant. Another journal (PDF, 447 KB)(link resides outside of
ibm.com) highlights its use in stock market forecasting, currency exchange rates, trading futures,
and money laundering analyses.
o - Healthcare: KNN has also had application within the healthcare industry, making predictions on
the risk of heart attacks and prostate cancer. The algorithm works by calculating the most likely
gene expressions.
o - Pattern Recognition: KNN has also assisted in identifying patterns, such as in text and digit
classification (link resides outside of ibm.com). This has been particularly helpful in identifying.

Conclusion:

K-Nearest Neighbour is one of the simplest Machine Learning algorithms based on Supervised Learning
technique. K-NN algorithm assumes the similarity between the new case/data and available cases and put the
new case into the category that is most similar to the available categories. K-NN algorithm stores all the
available data and classifies a new data point based on the similarity. This means when new data appears then
it can be easily classified into a well suite category by using K- NN algorithm. KNN algorithm can be used
for Regression as well as for Classification but mostly it is used for the Classification problems. Fitting K-NN
classifier to the Training data: Now we will fit the K-NN classifier to the training data. To do this we will
import the KNeighborsClassifier class of Sklearn Neighbors library. After importing the class, we will create
the Classifier object of the class. The Parameter of this class will be n_neighbors: To define the required
neighbors of the algorithm. Usually, it takes 5. metric='minkowski': This is the default parameter and it decides
the distance between the points. p=2: It is equivalent to the standard Euclidean metric. And then we will fit
the classifier to the training data. Below is the code for it: As we can see the graph is showing the red point
and green points. The green points are for Purchased(1) and Red Points for not Purchased(0) variable. The
graph is showing an irregular boundary instead of showing any straight line or any curve because it is a K-NN
algorithm, i.e., finding the nearest neighbor. The graph has classified users in the correct categories as most
of the users who didn't buy the SUV are in the red region and users who bought the SUV are in the green
region. The graph is showing good result but still, there are some green points in the red region and red points
in the green region. But this is no big issue as by doing this model is prevented from overfitting issues. It is
very simple algorithm to understand and interpret. It is very useful for nonlinear data because there is no
assumption about data in this algorithm. It is a versatile algorithm as we can use it for classification as well as
regression. It has relatively high accuracy but there are much better supervised learning models than KNN. It
is computationally a bit expensive algorithm because it stores all the training data. High memory storage
required as compared to other supervised learning algorithms. Prediction is slow in case of big N. It is very
sensitive to the scale of data as well as irrelevant features. It helped to explain the basics of the KNN algorithm,
and how it classifies data based on certain features. We also examined when it’s best to use the K value and
when to use the KNN algorithm. Finally, we went through a use case demo to predict whether a person is
likely to develop diabetes or not. In this we have learn about KNN in detail. KNN means k nearest neighbor.
K-Nearest Neighbors is one of the simplest supervised machine learning algorithms used for classification.