Professional Documents
Culture Documents
Data Mining: UNIT-3 Classification
Data Mining: UNIT-3 Classification
UNIT-3 Classification
B SUDHAKAR
Department of Computer Science
GURU NANAK INSTISTUTE OF TECHNOLOGY
Target marketing
Medical diagnosis
Fraud detection
or mathematical formulae
Model usage: for classifying future or unknown objects
Estimate accuracy of the model
will occur
If the accuracy is acceptable, use the model to classify data
Classification
Algorithms
Training
Data
Classifier
Testing
Data Unseen Data
(Jeff, Professor, 4)
NAME RANK YEARS TENURED
Tom Assistant Prof 2 no Tenured?
Merlisa Associate Prof 7 no
George Professor 5 yes
Joseph Assistant Prof 7 yes
June 13, 2021 Data Mining: Concepts and Techniques 6
Supervised vs. Unsupervised Learning
Data cleaning
Preprocess data in order to reduce noise and handle
missing values
Relevance analysis (feature selection)
Remove the irrelevant or redundant attributes
Data transformation
Generalize and/or normalize data
attributes
Speed
time to construct the model (training time)
age?
<=30 overcast
31..40 >40
no yes no yes
but gini{medium,high} is 0.30 and thus the best since it is the lowest
All attributes are assumed continuous-valued
May need other tools, e.g., clustering, to get the possible split values
Can be modified for categorical attributes
P( H | X) P(X | H ) P(H )
P(X)
Informally, this can be written as
posteriori = likelihood x prior/evidence
Predicts X belongs to C2 iff the probability P(Ci|X) is the
highest among all the P(Ck|X) for all the k classes
Practical difficulty: require initial knowledge of many
probabilities, significant computational cost
June 13, 2021 Data Mining: Concepts and Techniques 26
Towards Naïve Bayesian Classifier
Let D be a training set of tuples and their associated class
labels, and each tuple is represented by an n-D attribute
vector X = (x1, x2, …, xn)
Suppose there are m classes C1, C2, …, Cm.
Classification is to derive the maximum posteriori, i.e., the
maximal P(Ci|X)
This can be derived from Bayes’ theorem
P(X | C )P(C )
P(C | X) i i
i P(X)
Since P(X) is constant for all classes, only
P(C | X) P(X | C )P(C )
i i i
needs to be maximized
June 13, 2021 Data Mining: Concepts and Techniques 27
Derivation of Naïve Bayes Classifier
A simplified assumption: attributes are conditionally
independent (i.e., no dependence relation between
attributes): n
P( X | C i ) P ( x | C i ) P( x | C i ) P( x | C i ) ... P( x | C i )
k 1 2 n
k 1
This greatly reduces the computation cost: Only counts
the class distribution
If Ak is categorical, P(xk|Ci) is the # of tuples in Ci having
value xk for Ak divided by |Ci, D| (# of tuples of Ci in D)
If Ak is continous-valued, P(xk|Ci) is usually computed
based on Gaussian distribution with a mean μ and
standard deviation σ 1
( x ) 2
g ( x, , ) e 2 2
2
and P(xk|Ci) is
P ( X | C i ) g ( xk , C i , C i )
June 13, 2021 Data Mining: Concepts and Techniques 28
Naïve Bayesian Classifier: Training Dataset
age income studentcredit_rating
buys_compu
<=30 high no fair no
<=30 high no excellent no
Class: 31…40 high no fair yes
C1:buys_computer = ‘yes’ >40 medium no fair yes
C2:buys_computer = ‘no’ >40 low yes fair yes
>40 low yes excellent no
Data sample
31…40 low yes excellent yes
X = (age <=30,
<=30 medium no fair no
Income = medium,
Student = yes <=30 low yes fair yes
Credit_rating = Fair) >40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
June 13, 2021 Data Mining: Concepts and Techniques 29
Naïve Bayesian Classifier: An Example
P(Ci): P(buys_computer = “yes”) = 9/14 = 0.643
P(buys_computer = “no”) = 5/14= 0.357
counterparts
June 13, 2021 Data Mining: Concepts and Techniques 31
Naïve Bayesian Classifier: Comments
Advantages
Easy to implement
Disadvantages
Assumption: class conditional independence, therefore loss
of accuracy
Practically, dependencies exist among variables
Classifier
How to deal with these dependencies?
Bayesian Belief Networks
Instance-based learning:
Store training examples and delay the processing
space.
Locally weighted regression
Case-based reasoning
based inference
June 13, 2021 Data Mining: Concepts and Techniques 38
The k-Nearest Neighbor Algorithm
All instances correspond to points in the n-D space
The nearest neighbor are defined in terms of
Euclidean distance, dist(X1, X2)
Target function could be discrete- or real- valued
For discrete-valued, k-NN returns the most common
value among the k training examples nearest to xq
Vonoroi diagram: the decision surface induced by 1-
NN for a typical set of training examples
_
_
_ _ .
+
_ .
+
xq + . . .
June 13, 2021
_ + .
Data Mining: Concepts and Techniques 39
Discussion on the k-NN Algorithm
d d
d
d
( yi yi ' ) 2
Relative absolute error:
| y y '|
i 1 Relative squared error:
i i
i 1
d
d
| y y |
i 1
i (y
i 1
i y)2
obtained
Cross-validation (k-fold, where k = 10 is most popular)
Randomly partition the data into k mutually exclusive subsets,
Ensemble methods
Use a combination of models to increase accuracy
classifiers
Boosting: weighted vote with a collection of classifiers