Classification Algorithm in Machine Learning

Abstract:
Classification algorithm are extensively used in machine learning problems in data science to separate
one class into positive values and others into negative values. But all the algorithms doesn’t provide
the same accuracy and efficiency at the same time for the same problem, therefore we have to select
the best model among all. So have to test the different models for the same problem and conduct a
study to compare all the algorithms for a given problem.
This paper provides the working of different classification algorithm and the comparison between
them for a given dataset out of which it was concluded that SVM provided the best accuracy among
all for a given dataset and different algorithm provides different accuracy for n no of problems

Introduction & Background-

Machine learning is one of the fastest-growing areas of computer science, with far-reaching
applications. It refers to the automated detection of meaningful patterns in data. Machine learning
tools are concerned with endowing programs with the ability to learn and adapt. Machine Learning
has become one of the mainstays of Information Technology and with that, a rather central, albeit
usually hidden, part of our life. With the ever-increasing amounts of data becoming available there is
a good reason to believe that smart data analysis will become even more pervasive as a necessary
ingredient for technological progress. There are several applications for Machine Learning (ML), the
most significant of which is data mining. People are often prone to making mistakes during analyses
or, possibly, when trying to establish relationships between multiple features.

History
Machine learning was first conceived from the mathematical modeling of neural networks. A paper
by logician Walter Pitts and neuroscientist Warren McCulloch, published in 1943, attempted to
mathematically map out thought processes and decision-making in human cognition.

In 1950, Alan Turning proposed the Turing Test, which became the litmus test for which machines
were deemed "intelligent" or "unintelligent." The criteria for a machine to receive status as an
"intelligent" machine, was for it to have the ability to convince a human being that it, the machine,
was also a human being. Soon after, a summer research program at Dartmouth College became the
official birthplace of AI.

From this point on, "intelligent" machine learning algorithms and computer programs started to
appear, doing everything from planning travel routes for salespeople, to playing board games with
humans such as checkers and tic-tac-toe.

Types of machine learning algorithm

Supervised vs. Unsupervised vs. Reinforcement Learning
The easiest way to distinguish supervised learning and unsupervised learning is to see whether the data
is labelled or not.
Supervised learning learns a function to make a prediction of a defined label based on the input data.
It can be either classifying data into a category (classification problem) or forecasting an outcome
(regression algorithms).
Unsupervised learning reveals the underlying pattern in the dataset that is not explicitly presented,
which can discover the similarity of data points (clustering algorithms) or uncover hidden relationships
of variables (association rule algorithms) …
Reinforcement learning is another type of machine learning, where the agents learn to take actions
based on their interaction with the environment, to maximize rewards. It is most similar to the learning
process of humans, following a trial-and-error approach.

Machine Learning Algorithms for Classification:

classification algorithm in machine learning comes under supervised learning. There are many
Classification algorithms of which some are:
1. Logistic Regression
Logistic regression: This is a classification function that uses a class for building and uses a single
multinomial logistic regression model with a single estimator. Logistic regression usually states
where the boundary between the classes exists, and also states the class probabilities depend on the
boundary, in a specific approach. This moves towards the extremes (0 and 1) more rapidly when the
data set is larger. These statements about probabilities make logistic regression more than just a
classifier. It makes stronger, more detailed predictions, and can be fit differently; but those strong
predictions could be wrong. Logistic regression is an approach to prediction, like Ordinary Least
Squares (OLS) regression. However, with logistic regression, prediction results in a dichotomous
outcome [13]. Logistic regression is one of the most commonly used tools for applied statistics and
discrete data analysis. Logistic regression is linear interpolation.
2. Decision Tree
Decision Trees (DT) are trees that classify instances by sorting them based on feature values. Each
node in a decision tree represents a feature in an instance to be classified, and each branch represents a
value that the node can assume. Instances are classified starting at the root node and sorted based on
their feature values. Decision tree learning, used in data mining and machine learning, uses a decision
tree as a predictive model which maps observations about an item to conclusions about the item's
target value. More descriptive names for such tree models are classification trees or regression trees.
Decision tree classifiers usually employ post-pruning techniques that evaluate the performance of
decision trees, as they are pruned by using a validation set. Any node can be removed and assigned
the most common class of the training instances that are sorted to it.

3. Random Forest .
random forest is a collection of decision trees. It is a common type of ensemble method that aggregates
results from multiple predictors. Random forest additionally utilizes a bagging technique that allows
each tree trained on a random sampling of the original dataset and takes the majority vote from
trees. Compared to a decision tree, it has better generalization but is less interpretable, because of
more layers added to the model.
4. Support Vector Machine (SVM)

Support Vector Machine (SVM) Support vector machines (SVMs) are one of the discriminative
classification methods commonly recognized as more accurate. The SVM classification method is
based on the Structural Risk Minimization principle from computational learning theory The idea of
this principle is to find a hypothesis to guarantee the lowest true error. Besides, the SVM is well-
founded that very open to theoretical understanding and analysis. The SVM need both positive and
negative training set which are uncommon for other classification methods. These positive and
negative training sets are needed for the SVM to seek the decision surface that best separates the
positive from the negative data in the n-dimensional space, so-called the hyperplane. The document
representatives which are closest to the decision surface are called the support vector. The
performance of the SVM classification remains unchanged if documents that do not belong to the
support vectors are removed from the set of training data

5. K-Nearest Neighbour (KNN)

nearest-neighbour (kNN) classification is one of the most fundamental and simple classification
methods and should be one of the first choices for a classification study when there is little or no prior
knowledge about the distribution of
the data. K-nearest- neighbor
classification was developed from
the need to perform discriminant
analysis when reliable
parametric estimates of probability
densities are unknown or difficult to determine. In an unpublished US Air Force School of Aviation
Medicine report in 1951, Fix and Hodges introduced a non-parametric method for pattern
classification that has since become known as the k-nearest neighbor rule (Fix & Hodges, 1951). Later
in 1967, some of the formal properties of the k-nearest-neighbor rule were worked out; for instance, it
was shown that for k=1 and n→∞ the k-nearest-neighbor classification error is bounded above by
twice the Bayes error rate (Cover & Hart, 1967).

Compare Classification Algorithms

Case study 1:

For comparing the algorithm author has used a dataset based on the following data set of which
he had to conclude whether the person purchased the iPhone or not.

He compared 6 classification algorithms — the ones I have covered in previous projects. Feel free to
add and test others as well.
  Logistic Regression
 KNN
 Kernel SVM
 Naive Bayes
 Decision Tree
 Random Forest

Using 10-fold cross-validation to evaluate each algorithm and we will find the mean accuracy and the
standard deviation accuracy. First, we will create a list and add objects of the different classifiers we
want to evaluate. Then we loop through the list and use the cross_val_score method to get the
accuracies.

Output:
Logistic Regression: Mean Accuracy = 82.75% — SD Accuracy = 11.37%
K Nearest Neighbor: Mean Accuracy = 90.50% — SD Accuracy = 7.73%
Kernel SVM: Mean Accuracy = 90.75% — SD Accuracy = 9.15%
Naive Bayes: Mean Accuracy = 85.25% — SD Accuracy = 10.34%
Decision Tree: Mean Accuracy = 84.50% — SD Accuracy = 8.50%
Random Forest: Mean Accuracy = 88.75% — SD Accuracy = 8.46%
Case study 2:
COVID-19 World Vaccination Progress Using Machine Learning Classification Algorithms:
In December 2019, SARS-CoV-2 caused coronavirus disease (COVID-19) distributed to all
countries, infecting thousands of people and causing deaths. COVID-19 induces mild sickness in
most cases, although it may render some people very ill. Therefore, vaccines are in various
phases of clinical progress, and some of them are being approved for national use. The current
state reveals that there is a critical need for a quick and timely solution to the Covid-19 vaccine
development. Non-clinical methods such as data mining and machine learning techniques may
help do this. This study will focus on the COVID-19 World Vaccination Progress using
Machine learning classification Algorithms. The findings of the paper show which algorithm is
better for a given dataset. Weka is used to run tests on real-world data, and four output
classification algorithms (Decision Tree, K-nearest neighbors, Random Tree, and Naive
Bayes)are used to analyze and draw conclusions. The comparison is based accuracy of a
performance period, and it s discovered that the Decision Tree outperforms other algorithms in
terms of time d accuracy.

Algorithm True positive rate False positive rate precisio Recall

n
Naïve Bayes 0.697 0.002 0.856 0.697
K-nearest neighbour’s 0.813 0.005 0.754 0.813
Decision Tree 1.000 0.000 0.000 1.000
Random Tree .962 0.014 0.819 0.962

Conclusion
Choosing the right algorithm for a particular data mining task is difficult. The best way is to perform a
validation of the performances of the different algorithms to choose the best one that gives an
appropriate result. This study performed a comparative analysis of the five classification algorithms;
Naïve Bayes, Decision Tree, KNN, Logistic Regression and To get the best predictive model, the
different classification algorithms were evaluated based on the performance metrics; accuracy,
precision, recall, F1 score, and AUC. From the results, we can see that different algorithm perform
different accuracy on different datasets, Kernel SVM  performed better than in the case study 1 dataset
and the decision tree algorithm performed better in the case of study 2 data set.
Future of machine learning algorithms:
The future of machine learning classification algorithms is shrouded in potential but also in
uncertainty. However, there are a few potential evolutions that could occur to make these algorithms
more accurate and reliable. First, the algorithms could be designed to work with a greater variety of
data types. This would allow for more accurate classification, as the algorithms would be able to take
into account a wider range of information. Additionally, the algorithms could be designed to be more
flexible, so that they could adapt to changes in the data more easily. This would make them more
robust and reliable, as they would be less likely to produce false positives or false negatives. Finally,
the algorithms could be made more transparent, so that users could understand how and why they are
making the classification decisions that they are. This would allow for greater trust in the algorithms,
as users would be able to see the logic behind their decisions.

References
https://towardsdatascience.com/machine-learning-project-17-compare-classification-
algorithms-87cb50e1cb60
https://towardsdatascience.com/top-machine-learning-algorithms-for-classification-
2197870ff501
https://www.researchgate.net/profile/J-E-T-Akinsola/publication/
318338750_Supervised_Machine_Learning_Algorithms_Classification_and_Comparison/
links/596481dd0f7e9b819497e265/Supervised-Machine-Learning-Algorithms-Classification-
and-Comparison.pdf

https://journal.qubahan.com/index.php/qaj/article/view/53/38