ATutorialonClusteringAlgorithms

Introduction|Kmeans|FuzzyCmeans|Hierarchical|MixtureofGaussians|Links

FuzzyCMeansClustering
TheAlgorithm
Fuzzycmeans(FCM)isamethodofclusteringwhichallowsonepieceofdatatobelongtotwoormoreclusters.Thismethod(developedbyDunn
in1973andimprovedbyBezdekin1981)isfrequentlyusedinpatternrecognition.Itisbasedonminimizationofthefollowingobjectivefunction:

wheremisanyrealnumbergreaterthan1,uijisthedegreeofmembershipofxiintheclusterj,xiistheithofddimensionalmeasureddata,cjisthe
ddimensioncenterofthecluster,and||*||isanynormexpressingthesimilaritybetweenanymeasureddataandthecenter.
Fuzzypartitioningiscarriedoutthroughaniterativeoptimizationoftheobjectivefunctionshownabove,withtheupdateofmembershipuijandthe
clustercenterscjby:

Thisiterationwillstopwhen ,where is a termination criterion between 0 and 1, whereas k are the iteration steps. This
procedureconvergestoalocalminimumorasaddlepointofJm.
Thealgorithmiscomposedofthefollowingsteps:

1.InitializeU=[uij]matrix,U(0)
2.Atkstep:calculatethecentersvectorsC(k)=[cj]withU(k)

 3.UpdateU(k),U(k+1)

4.If||U(k+1)U(k)||< thenSTOPotherwisereturntostep2.

Remarks
Asalreadytold,dataareboundtoeachclusterbymeansofaMembershipFunction,whichrepresentsthefuzzybehaviourofthisalgorithm.Todo
that,wesimplyhavetobuildanappropriatematrixnamedUwhosefactorsarenumbersbetween0and1,andrepresentthedegreeofmembership
betweendataandcentersofclusters.
Forabetterunderstanding,wemayconsiderthissimplemonodimensionalexample.Givenacertaindataset,supposetorepresentitasdistributedon
anaxis.Thefigurebelowshowsthis:

Lookingatthepicture,wemayidentifytwoclustersinproximityofthetwodataconcentrations.WewillrefertothemusingAandB.Inthefirst
approachshowninthistutorialthekmeansalgorithmweassociatedeachdatumtoaspecificcentroidtherefore,thismembershipfunctionlooked
likethis:
IntheFCMapproach,instead,thesamegivendatumdoesnotbelongexclusivelytoawelldefinedcluster,butitcanbeplacedinamiddleway.In
thiscase,themembershipfunctionfollowsasmootherlinetoindicatethateverydatummaybelongtoseveralclusterswithdifferentvaluesofthe
membershipcoefficient.

Inthefigureabove,thedatumshownasaredmarkedspotbelongsmoretotheBclusterratherthantheAcluster.Thevalue0.2ofmindicatesthe
degreeofmembershiptoAforsuchdatum.Now,insteadofusingagraphicalrepresentation,weintroduceamatrixUwhosefactorsaretheones
takenfromthemembershipfunctions:


(a)(b)

Thenumberofrowsandcolumnsdependsonhowmanydataandclustersweareconsidering.MoreexactlywehaveC=2columns(C=2clusters)
andNrows,whereCisthetotalnumberofclustersandNisthetotalnumberofdata.Thegenericelementissoindicated:uij.
Intheexamplesabovewehaveconsideredthekmeans(a)andFCM(b)cases.Wecannoticethatinthefirstcase(a)thecoefficientsarealways
unitary.Itissotoindicatethefactthateachdatumcanbelongonlytoonecluster.Otherpropertiesareshownbelow:

AnExample
Here,weconsiderthesimplecaseofamonodimensionalapplicationoftheFCM.Twentydataandthreeclustersareusedtoinitializethealgorithm
andtocomputetheUmatrix.Figuresbelow(takenfromourinteractivedemo)showthemembershipvalueforeachdatumandforeachcluster.The
colorofthedataisthatofthenearestclusteraccordingtothemembershipfunction.
Inthesimulationshowninthefigureabovewehaveusedafuzzynesscoefficientm=2andwehavealsoimposedtoterminatethealgorithmwhen
.Thepictureshowstheinitialconditionwherethefuzzydistributiondependsontheparticularpositionoftheclusters.No
stepisperformedyetsothatclustersarenotidentifiedverywell.Nowwecanrunthealgorithmuntilthestopconditionisverified.Thefigurebelow
showsthefinalconditionreachedatthe8thstepwithm=2and =0.3:

Isitpossibletodobetter?Certainly,wecoulduseanhigheraccuracybutwewouldhavealsotopayforabiggercomputationaleffort.Inthenext
figurewecanseeabetterresulthavingusedthesameinitialconditionsand =0.01,butweneeded37steps!

Itisalsoimportanttonoticethatdifferentinitializationscausedifferentevolutionsofthealgorithm.Infactitcouldconvergetothesameresultbut
probablywithadifferentnumberofiterationsteps.

Bibliography
J. C. Dunn (1973): "A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact WellSeparated Clusters", Journal of
Cybernetics3:3257
J.C.Bezdek(1981):"PatternRecognitionwithFuzzyObjectiveFunctionAlgoritms",PlenumPress,NewYork
TariqRashid:Clustering
http://www.cs.bris.ac.uk/home/tr1690/documentation/fuzzy_clustering_initial_report/node11.html
HansJoachimMuchaandHizirSofyan:NonhierarchicalClustering
http://www.quantlet.com/mdstat/scripts/xag/html/xaghtmlframe149.html

FuzzyCmeansinteractivedemo

Previouspage|Nextpage