Uit-3

1eqning,
Sopevvi sed learning: pavametvie and on-paratnetvic
pavame triC leavning : Bayesiao Decision Theor y- Sinqle
nig:
Feature ( ), Moltiple feotuves. Nonparametric eay
1list ogYom Estitatoy aod payz en Windows, k-Nea Irest
tlistogram
AeighboUr ( K-NN) Ctassifiation, Ay tificial NeoYal Aletwors,
kernel machines
Unit-y

Featve extraction and Selecion: ReduciDa Dimensi onahty

PreprocesSing Feabuve seleci on- tnteripbraclass Distare,
Subset Selechon, Featuye ebacion -principal.Conponent
AnalysiS, Linear Disc rìmDant Apalysis Unsupexvis ed
Cmeans
Tearning clustering rk-Meos clustering Fuzzy
dostering, tAggloroe vative) tierarchichol duSterirg
Unit classi
parametric method s of statiCal
parametric learning: estimat e
distribuhons and
-iCation requive pvobability them ch as
the
and
me

parameters derived from

Compact vepresentotion
Standard deviaion to pro uide
YOledisCvioinant analvsiS etCC:
ex-"BayesdeciSion
pvo babi li by fónct dis tri butions
Non-pavametiic lea vning: If the non-parametyic melhods,
muSt USe
Qre Dot khOwn then we
we Se
eStimate densiby functi0ns
K-NN YOle etc.
éXi-Par2 en windo approa Ch

Bayesian Decì Si on, ; Theory

baseo on quantifying9 trade0Ff blw
7 Ihis approach 9 probl bi ity CoSt
& OSt

uaribus classifi Cation dec sio nsing

deision
decisiob 3
that acCo mpany Such lithes known
velevant probabi aYe
7ASSUmed that all

Single Featuve (1p)

ctasses ie CloSs 1 (uo) and cloSS2 (us)
Let u9 ConSider tw
and príor proba bìlity known plw):0 &pluo-3. fIf a new
problen
that it is assigned to
Sa mple is qiven the plwi) pl)
which yule: ACCo rding tD be decision vule
in general S
The proba bility of classification erYOr I
oilw)pla)0
Plenor): p (choo se Lo,9). plu) tplch 0se
Since we aays ch00 Se
Plerro): plchuose i , ) - plo)
 have large no of Sa oples PdF wi
be Gaussian Shape
plBIA) PA)
Baye's vUle
PL6)
Ploi)
+1ere Plorlx) : P(xlw;).
P()

Qr decision YUle becomes

P(xlw)plo)', plzlw,).plw)
P()
PAL)
Choose 02
else
p lw)
PlMl w.) plu
chooseo,
else

pxluD.
ik elino od ratio: Pllw,)
Ly The yatio of Closs conditional den sity fonctions

’ cheCk but Problems on

Multipie featUres
Cannot achieve eror-free clas ificati on
>7 AS Singlefeatve
. the we Con Sider anDtheY
there on ovevlap Cf pdf's
feature that Should be UNCOTY ela ted.
ev- Es h with wid th as well
l as their tightbesS 3alm0n

an d sea ba 3 5)
Salmon SeQbaIs
upjdth‘
6) 1
a)

-5'Li9
;ahtheS
 ) 9 sbov ld be Salmon but
decision bound ory Cla 9Sifies
as sea bass
b)A deCiSi on bUnd ary
Yepres ents Optì rn al
tradeofe blw per formant
on braining and Simphcity
of clas si fier giving highes
CCCUraCy on n ewda tc

Non par ametvic eavning:

the input
>Thege are pYOblens wbe re
best,. suited foY
well-deined or tOo complex tÍ bemodelled
data is Dot
Dsing parametric algoithm
>That Algorith ms donot
Alorith make StrongeY asSUmpbons
are clled
about the form of the fohction
mappipg fU
Alon-parametvic teavning

tistog ra m eStimator and paYzen l irdows

independeht ample s
FoY 1 dinenSional Case i we have N
(- Yandom variablei,: The pdF QppyoXima te d
Loith bìng thatar
Samples
by pdF bìstogYan foYmed from

tave (arge h Then

DoDf ga mples Then
>k is no of Sam ples in abin, of midpoint Xi
) Probabil:ty that a sample is in that bìn
- probabiliy can be estimated by relali ve freqvenCy y
KJN and density at i KOnN
Cah by e estimated by klsXN )
Choo se ovigin qnd bin width
» Constructing a bìstogya m
is snall it is spiky if it iS tàvge it is
Ay if binwid th
Smoo thy
 (o) kernel fUnction
’Pay zen oindow: t is USed as Smo oth weightng
function , to get Sm00 they estimate
of p r is th en Sum Of kernels
7 Tbe estimate
placed at each data point

P)7
)esimationof pdF byq

H bistogia m

Ki)
k ernel

b) 0Sing a Gaussi an ker nal tu weight each 3Q mple io

training set
K-NeaveSt eibbour classi fi caton:(k-NN)
It assumes the Similavity betwe en new data a
available da ta KNN based On SUpervised teavning
ta and dassjfies
)It Sor es al the Quailable da
data
new da ta pojnt based On simjlarily
Non- pararmetric algov)thm it doesn't
yi t is q parametric
make any asSum pti on on'underlying date
)|t is also caued o'Lazy teaner algori thm
because it doeS not leayn from taining set t
juSt stores the dt a.
 KONClasifi er
Rea Son: Similar featuYes
Cat COmpat ed on
animal Predicted Olp and gtoved o ne

Algorithm:
Step : Select the DUmber k of neigbhours
Step Calculate cuctidioo distQnce of k po-0e neghbour
as per alculated
Step3 Take the knearest neighboYS
eucLidian diSta nce
count noof dat o
Stepy:Among these k peigh bor s
Pojots in eoch Category
new data points to that atcGory
Step5: ASSign the
pUmber of neig hbor is max imum
for whi ch the
S tep 6: oor mOdet iS y ady
n ew ata
Si: choose K5
example
sii fìd euclidìaDdistance
G

categoYYÂ blw Data point s

CategoryÀ

Sji) Nevest neig hbouy s

are. 3 froU at A and
d fro m Cat B

Siy,: ASS ign it to cat A

beause of maxìmUm neay
Pojn-s

Low K vaue ’ 0oisy & leads to oUttiers in mode!

High k aloe good but it may find SOme difficolties
Ady O SìmPle to implement
) vob0st to noisy taining dat a
 peed to deter mine k
Disa dv: æ Always
du Computahon Ost
Avtificial Neal AetuwoYks (A NN)
percepton: perform ing alul
pevceptyon foY
> MrFrank iovented Capabiliies
det ect ilp data
aions to S a tinear
block for AN It
Jpercepton is buildin9 leorning for various
for Supervised
ML alqovi thm Used
bìnary classifiers 4 main
netootk with
) ConSidey a Single tayer neural
Det sUm
weights and Baìs
Para meters ie, ilp values
and an actua tion functi oh

Weights -levor
wetilpfunction
ACtuati onlfun

tsigo
step
I-sign
Fg:Single layer inial dota into Syst e
accepts the
which value
ilp layey
ilp Dode cootains veal
fuYther processing Cach stength
for repyesen ts
weight pavarmeter Can be Considered
Ial eight and Bias:
betw@er units Bias
Of Connection in linear equction
ine of intercept
as
Sign istep i, Sigmoid
Actuaion function: d)
)Thi is uSed to take a Subjechve deci SiOn

Iovkin9
-twn Xn
Step :
impYOVe models perfov ma nCe
To
wi Xi +b
actuation fonchon
Stepa: ApplyI
6)
Y: f(s, wiXi+
mode
Single layer perCept1on
TYPes -
layer peYCeptto n model
multi-
Single layer perceptron model:
->lt con sists
’It feedfovwovd Detwoyk
Fo analy 2 Iineay ly Seperoble
>main objectve to

objects
molti layered percepton mode l
’9ame as Siglebut it Contain bidd en layer
The multj-ayer perceptyon model is also Known as
the EackpYopagation algovithm, exeutes dstoges
FoYWard Stoge: AChn ion fonctì ons Start from p layer
in Forward Stage nd end on olP tayers
weights bias are
Backwavd stage: In thìs stege
demande!
modified. to reduce erYoY blw actual and
Output stay ts from olp la yer and ends on ilp
layer
This mulilayer Uses achation fonction TanH)Reu)
Siqmoìd etc.
3|t Can implement AND; OR; xoR NAND, N0T XNa,
NOR
Adu pisad v
worKS, well for
Compl ex nonlìnear 5 ti me comsuming
probl ems ’ depend S on qvality of bainirg
’ wOr ks wel with
but Small e large
ile data Hidden layers
ilelayer Outpot Layey

Fig: ml ti- layered

perceplvon mode!

Igicat
perceptron Rule for, AND qate
A^B
an d AB
>ulhe|:2, w)0 b Thyeshold=1
Iearning vate n0 5

1) A=0, 60 Toarget=o

of So o)p-o AD
Olp Dot 9reateY than threshold of weight
Updahon
a) Ao Target- O

olpi3 0 NO need of weight

O6<Thresho Updation?
3) A=1 B =O Target o

(2 > Thresho ld >0|p is 1

need Of modification of weights
 Bwi: witolt-o)i )weight vpdation ro le

W) = 6 to5 l0-1) 0:06

Threshold-t
modified weightS are
Learning Rate n 0 5
again Check for the Cases Same procegS
Then
A:1 B-0/
A:0 B:lv

A | B-1

weighted
SU m olp.

Logical OR
Thres hold-1 A
A 6B A+ B

Learning.vate n:0 5
and
OB:0, Target: D
c)A

<Thxes hold = Olp 1SO

o
ond 7avget:1
\ A=0, B
wì*i -0¥0 +|Y0: 6 :0 6
o'6< Tbrechotd
Need of weight vdation

-) ,
W: ob +o5(l-0)
 (6)
modified weightS aye

NÊ:0b W2| Tyeshold & ieayning Yat e n0: 5

Cases BO
check

X.
(i) A:| B0
NOw Wi :vi +olt-)

kn:o5
Again check W l W)- Thyesho ld=1

foY A:0 6:0 v

weig hted
SUm

)
lcqical xOR
ABY

ol

a)
f(yn):
AÇtatiop fonC.Oif Ym0
ASsume oleorning vate
hl: W:1, ThyeShold: e n:5
W

weight upd ation

" Wij: Wij +nlt-o)Yi
5* (0-1) 0 1
- W|= 1
: -05
*W2 l 1 5 * (0-1)

Aowy weightSs Wj:1 & W :-05

Check al Cases (ojo) (o) ,0) Ci)
b) Tare b)

Intialy W))2- 1

(o)’ Z2in:.
w)

weight vpdatio
- Wij =wij tolt-o) i
-WI2 - t I 5 lo-) 1 -0 5

weights are WI2:05 W221

Check (o,0)
Then y: 21 (o)Z)

C
initial weightS V V2
tor):
(o,) yin:l tdjust
weight

(lo):7 Meuval
Netwr
olp

(Bo) yin:o --l010lololololO+e pesived

.-o|1o0|lo610ollo
Ial||k W-05
Finally W|) -o-5 &W22:1 Fig Mpltil ayer lnevyal ol leani,
to model the xoR fUn

-os

Kernel Machines:
are a SS of algorithms for pattern
clo ss
Skerne! machìneS
SUpport vectoy
aralysis, Who se best known member is
machine, (SVM
>It is Alon parametric leaYning
the sepavating bypeyplane
naXimzing rnarg ins aroung
Seem according to P YObably APprximately Covert
good
CPA) theory
w'xtbo ’ eq fOr seperati ng byperplane
where wkb are Para meters

Y: Cwxtb)/W| )dstance from example Xi,

to hypev plane
 The exampleS closest to the hypev plane ave called

upport vectoy s
two-class olassifievs.
’Svm's aYe bagicaly

linaviy' seperable we ityoduc

7If training 3et is not
Slack Variabes to al a mìscdassifi catiun of difficult
examples - sOft navqin

avgin
Fig: Suppost vecty s R clo ssifiaton