K - Nearest Neigbors

and it’s Hyperparameter

Category B

Category A

New Data Point

What is KNN?
The k-Nearest Neighbors (kNN) algorithm is a
supervised machine learning algorithm used
for classification and regression tasks. It
works by finding a predefined number of
training samples closest in distance to a new
point and predicting the label based on their
labels.
Why to use KNN?
Simplicity: It's an intuitive and easy-to-
implement algorithm.

No Training Phase: kNN is a lazy learner,

which means it doesn't require a separate
training phase and starts working at the
time of prediction.

Adaptive: Can quickly adapt to changes as

it does not require retraining with the
addition of new data.

Useful for non-linear data: Can capture

complex decision boundaries.

Versatile: Can be used for both

classification and regression tasks.
Advantages of KNN

No Assumption: Does not make any

assumptions about the functional form of
the transformation, which makes it
suitable for any type of data.

Instance-Based Learning: Can adapt

easily to changes.

Multiclass Classification: Naturally

supports multi-class classification.
Disdvantages of KNN

Computationally Intensive: Requires the

storage of the entire dataset. Finding the
nearest neighbors can be computationally
expensive.

Sensitive to Irrelevant Features: Since it

relies on distance computation, irrelevant
features can impact performance.

Sensitive to the Scale: Features with

higher scales can dominate the distance
metric. Feature scaling is crucial.

Degrades with High Dimensionality: Due

to the curse of dimensionality, distance-
based methods like kNN can perform
poorly.
Hyperparameters in KNN

Number of Neighbors (k)

Distance Metric (e.g., Euclidean,

Manhattan, Minkowski)

Weights (Uniform, Distance, or Custom)

Algorithm (Brute-force, KD-Tree, Ball Tree)

Leaf Size (For tree-based algorithms)

P (Power parameter for Minkowski

distance)

Metric Params (Additional arguments for

metric function)
Python Code to Implement KNN