See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261233398

Improvement of K-means clustering algorithm with better initial centroids

based on weighted average

Conference Paper · December 2012

DOI: 10.1109/ICECE.2012.6471633

CITATIONS READS

40 1,810

3 authors, including:

Md Sohrab Mahmud Md. Mostafizer Rahman

UNSW Canberra The University of Aizu
8 PUBLICATIONS   58 CITATIONS    12 PUBLICATIONS   48 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Md Sohrab Mahmud on 09 January 2019.

The user has requested enhancement of the downloaded file.

Improvement of K-means Clustering algorithm with
better initial centroids based on weighted average
Md. Sohrab Mahmud,1 Md. Mostafizer Rahman,2 and Md. Nasim Akhtar3
Department of Computer Science & Engineering
Dhaka University of Engineering & Technology, Gazipur
Bangladesh
1
sohrabmahmud@gmail.com, 2 mostafiz26@gmail.com, 3 nasimntu@yahoo.com
Abstract—Clustering is the process of grouping similar data
into a set of clusters. Cluster analysis is one of the major
data analysis techniques and k-means one of the most popular
partitioning clustering algorithm that is widely used. But the
original k-means algorithm is computationally expensive and the
resulting set of clusters strongly depends on the selection of
initial centroids. Several methods have been proposed to improve
the performance of k-means clustering algorithm. In this paper
we propose a heuristic method to find better initial centroids
as well as more accurate clusters with less computational time.
Experimental results show that the proposed algorithm generates
clusters with better accuracy thus improve the performance of
k-means clustering algorithm.
Index Terms—Data Mining, Clustering, K-means, Enhancing
kmeans, improved Initial centroids.
I. I NTRODUCTION
Clustering is an unsupervised classification of patterns
(observations, data items, or feature vectors) into groups
(clusters). Classification is defined as supervised learning
whereas clustering is defined as unsupervised ones.
Classification is a procedure where data objects are assigned
to predefined classes. Clustering methods attempt to organize
a set of data into clusters such that data within a given
cluster have a high degree of similarity (or related), whereas
objects belonging to different clusters have a high degree
of dissimilarity (or unrelated). The term unsupervised here
means that clustering of data objects doesn’t depend on
predefined classes. These techniques have been widely
used in various areas such as taxonomy, image processing,
information retrieval, data mining, etc. [1].
Clustering is one of the major data analysis tool in the
field of data mining. Clustering algorithms are basically
divided into two categories : Partitioning Algorithms and
Hierarchical algorithms. A partitioning clustering algorithm
separates data set into a defined number of set in a single
iteration while a hierarchical clustering divides data into
smaller subsets in hierarchical manner [3]. A good clustering
method will generate high quality of data cluster. The quality
of a clustering result depends on both the similarity measure
used by the method and its implementation and also by its
ability to discover some or all of the hidden patterns. A good
survey on clustering methods is found in [2].
A number of clustering methods have been proposed to
solve clustering problems [1]. K-means is the most exoteric
partitioning clustering method for its efficiency and simplicity
in clustering large data sets. The K-means clustering algo-
rithm is a partitioning clustering method that partition data
objects into k different clusters [2][4][5]. Accuracy of k-
means clustering method is better than other approaches. The
method is more efficient since its intelligence to cluster large
data set fast. K-means clustering is widely used for large
number of datasets. But K-means has a major drawback to
select initial seed points (centroids). As initial centroids have
a heavy impact in final cluster sets in depends exclusively on
the selection of initial seed points [2]. In original k-means
initial centroids are selected in random manner and thus it
affects the operational time and accuracy of cluster directly.
For different initial centroids selection resulting clusters will
be different. Also the operational complexity of the basic
k-means algorithm is high [2]. Various methods have been
proposed to enhance the efficiency and accuracy of k-means
clustering algorithm and suggests that better initial centroids
calculation can lead to accurate and efficient cluster groups as
well as better time complexity [6].
However, we propose a novel idea to overcome the drawback
of k-means algorithm for selecting initial seed point (cen-
troids).Working procedure of our propose algorithm will very
simple it just calculate a standard score based on data point.
Deep discussion will be later of this paper. Accuracy and
complexity of our novel idea is better than k-means algorithm.
This paper is organized as follows. Section 2 presents an
overview of related works on k-means with different initial
centroids calculation. In section 3 we discuss the original k-
means algorithm. Section 4 introduces the proposed algorithm
for better selection of initial centroids. Section 5 describes the
computational complexity of the proposed algorithm. Section
6 shows the experimental data and finally Section 7 describes
the conclusion and future work.
II. R ELATED W ORK
The classic k-means algorithm is strongly dependable on the
selection of initial centroids. Different selection of centroids
leads to different resulting clusters. So the accuracy and quality
of clusters depends heavily on initial selection points. A
number of methods have been proposed over the last few years
to improve the quality and accuracy of k-means [1].
A. M. Fahim et al. [4] proposed an enhanced algorithm to put
data points into the suitable cluster. In [4], the author proposed
an approach that requires less computational time compared
to the original k-means which is computationally expensive.
But initial centroids are still selected randomly in this method.
So, the method remains sensitive to the initial seed points and
doesn’t defined to generate accurate clustering results.
Chen Zhang et al. in [7] proposed a new algorithm for
selection of initial centroids in k-means clustering that avoided
the random selection of cluster seeds as well as improve
performance of k-means.
A. Bhattacharya et al. [8] proposed a heuristic clustering
algorithm which was named Divisive Correlation Clustering
Algorithm (DCCA) for grouping data objects. The method
generates cluster of data set without taking the initial centroids
and the value of the desired number of clusters k. The time
complexity of this algorithm is too high.
Kathiresan V. et al. in [9] proposed an algorithm to select
better initial centroids based on Z-Score Ranking method. In
this algorithm the author proposed to calculate the Z-Score
of each data point in the data set and then sorting the data
points based on the Z-Score values. After sorting the data set
is divided into k subsets, where k is the number of desired
clusters. Then the mean value of each subset is calculated
and finally any nearby value of each subset is taken as initial
centroid. Experimental result shows the algorithm improves
both accuracy and time complexity.
Madhu et al. proposed an enhanced method to find better initial
centroids [1]. The algorithm first checks if the data points
contains negative data. If it contains then they are converted
to positive data through a general conversion. The data points
are sorted based on the distance from the origin and partitioned
into k (number of desired clusters) equal sets. From each set
the middle point is taken as the initial centroids.
K. A. Abdul Nazeer er al proposed an iterative process to
select initial centroids [2]. The system mainly works in two
sequential phases. In first phase k initial centroids are selected
based on the relative distances of each data points. In the
second phase the clusters are computed based on the distance
of each point from the initial centroids. The algorithm is
constrained with the phase 1 it is proposed. It need s to
calculate the distance of each data points from all other data
points. A large set of data points may lead to a tremendous
computation.
III. K-M EANS ALGORITHM
K-means is a popular unsupervised learning algorithm and
partitioning method for clustering. The basic idea of K-means
algorithm is to classify the dataset D into k different clusters
where D is the dataset of n data, k is the number of desired
clusters. The algorithm consists of two basic phases [10]. The
first phase is to select the initial centroids for each cluster
randomly. The second and final phase is to take each point in
dataset and assign it to the nearest centroids [10]. To measure
the distance between points Euclidean Distance method is
used. When a new point is assigned to a cluster the cluster
mean is immediately updated by calculating the average of
all the points in that cluster [3]. After all the points are
included in some clusters the early grouping is done. Now
each data object is assigned to a cluster based on closeness
with cluster center where closeness is measured by Euclidean
distance. This process of assigning a data points to a cluster
and updating cluster centroids continues until the convergence
criteria is met or the centroids don’t differ between two
consecutive iterations. Once a situation is met where centroids
don’t move anymore the algorithm ends. The Pseudocode for
k-means clustering algorithm is given below [3].
A. K-means partitioning algorithm:
Input :
D = d1,d2,.....dn // set of n data items.
k = the number of desired clusters,
Output :
a set of k clusters.
Steps:
• Randomly choose k objects from n as the initial cluster
centers.
• Repeat
• Assign each object from n to one of k clusters to which
the object is most similar based on the mean value of
the objects in the cluster.
• Update the cluster means by taking the mean value of
the objects for each of k cluster.
• Until no change in cluster means/ min error E is reached.
IV. M ODIFIED K-M EANS
A. Initial Centroids Calculation
In the improved algorithm discussed in this paper a new
method is used to find a weighted average score of dataset.
Here we use an uniform method to find rank score by
averaging the attribute of each data point which will generate
initial centroids that follow the data distribution of the given
set. A sorting algorithm is applied to the score of each data
point and divided into k subsets where k is the number of
desired clusters. Finally the nearest value of mean from each
subset is taken as initial centroid.
In recent times we have to deal with multidimensional
data. For example a data set D is consists of n data
such as d1,d2,d3,.....,dn. Each data point of this set may
contain multiple attributes such as di may contain attributes

x1,x2,x3,.....,xm, where m is the number of attributes. In case
of multidimensional attributes we propose to set a weight
with each attribute to ensure that while we are calculating the
average of the attributes the score follows the data distribution
of that set. The another advantage of this method is that if
we want to enhance a particular feature of the data set we
can do that by increasing the weight related to that attribute
of the data objects. After multiplying the weight factor with
each attribute we sum the values and make an average by
dividing the total with m. The entire set of data points are
then sorted using Merge Sort. The sorted list of data points
are then divided into k subsets. The nearest possible value of
mean from each dataset becomes the initial centroids of the
cluster to be constructed. The Pseudocode for the proposed
algorithm is as follows.
Input :
D = d1, d2.......dn // set of n data items
k // Number of desired clusters
Output:
A set of k initial centroids.
Steps:
1.Calculate the average score of each data point;
i) di=x1,x2,x3,..,xn
ii) di(avg)=(w1*x1+w2*x2+w3*x3+ ...... +wm*xm)/m //
where x = the attributes value, m = number of attributes and
w = weight to multiply to ensure fair distribution of cluster
2. Sort the data based on average score;
3. Divide the data into k subsets;
4. Calculate the mean value of the each subset;
5. Take the nearest possible data point of the mean as the
initial centroid for each data subsets;
The above described method for finding initial centroids of the
clusters is more meaningful than the original k-means where
centroids are selected randomly. The algorithm converges
faster than the original k-means.
V. C OMPLEXITY A NALYSIS
In basic K-means algorithm, the initial centroids are randomly
calculated. For this reason, the cluster centroids are recalcu-
lated numerous times before the convergence criteria of the
algorithm are met and the data points are assigned to their
nearest centroids. Since, complete reassignment of data points
takes place according the new centroids, this method takes
time O(nkl) where n is the number of data points, k is the
TABLE I
P ERFORMANCE C OMPARISON OF THE PROPOSED ALGORITHM
DataSet Number of Algorithm Average
Clusters Time Taken (ms)
Diabetes 3 Standard K-means 0.0938
Proposed K-means 0.0781
Thyriod 3 Standard K-means 0.1293
Proposed K-means 0.1074
Blood Pressure 3 Standard K-means 0.1055
Proposed K-means 0.0924
number of clusters and l is the number of iterations.
The proposed algorithm discussed in this paper works in two
phases. In the first phase of the algorithm the time required to
calculate the weighted average of all the data points is O(n)
where n is the number of data points. The algorithm then
proposes to sort the data in ascending order. Sorting the data
points based on the weighted average of each item can be
done in O(nlogn) time using Merge Sort. Finally the overall
time complexity of the first phase of the proposed algorithm
is O(nlogn).
The second phase of the proposed algorithm is same
as like the original k-means algorithm. Distribution of
the data points to the nearest cluster and the consequent
recalculation of centroids are conducted repeatedly until the
convergence criteria reached. This process concluded with
a time complexity of O(nkl) where the symbols represent
the meaning mentioned above. The experimental data shows
that the algorithm converges in less number of iterations as
the initial centroids are calculated in a strategic way rather
than randomly. Thus the overall complexity of the proposed
algorithm is of O(n(kl + logn).
VI. E XPERIMENTAL R ESULTS
The multivariate data set, taken from the UCI repository of
machine learning databases, that is used to test the accuracy
and efficiency of the modified k-means algorithm. This same
data set is given as input to the standard k-means algorithm and
the modified k-means algorithm. The value of k, the number
of clusters, is taken as 3.
We have evaluated our algorithm on several datasets. We
have compared our results with standard k-means algorithm
in terms of the accuracy of cluster and total execution time.
The experimental results are shown in Table 1.
In standard k-means algorithm centroid will take randomly
but our modified k-means algorithm the dataset and the value
of k are the only inputs needed since the initial centroids
are computed automatically and find optimal cantroids by the
program.The percentage of accuracy and the execution time
taken for each experiment where modified k-means algorithm
shows better performance than standard k-means algorithm.
 VII. C ONCLUSION AND F UTURE W ORK
The k-means algorithm is widely used for clustering large
set of data. But standard k-means algorithm does not
always ensure good results as the accuracy of the final
clusters depend on the selection of initial centroids. The
computational complexity of the standard k-means algorithm
is high than our proposed k-means algorithm. This paper
presents a modified k-means algorithm which find the
approximate centroids that reduce the number of iteration
to assign the data into a cluster. One limitation of this
algorithm is that we still need to provide the desired cluster
number as input. A research issue on this point remains
open. Automating the value of k is suggested as a future work.
R EFERENCES
[1] Madhu Yedla, Srinivasa Rao Pathakota,T M SrinivasaEnhancing, K-means
Clustering Algorithm with Improved Initial Center, International Journal
of Computer Science and Information Technologies(IJCSIT), Vol. 1 (2) ,
2010, 121-125.
View publication stats
[2] K. A. Abdul Nazeer, M. P. Sebastian, Improving the Accuracy and
Efficiency of the k-means Clustering Algorithm, Proceedings of the World
Congress on Engineering 2009 Vol IWCE 2009, July 1 - 3, 2009, London,
U.K.
[3] Margaret H. Dunham, Data Mining-Introductory and Advanced Concepts,
Pearson Education, 2006
[4] A. M. Fahim, A. M. Salem, F. A. Torkey and M. A. Ramadan, “An
Efficient enhanced k-means clustering algorithm,” Journal of Zhejiang
University, 10(7): 16261633, 2006.
[5] Koheri Arai and Ali Ridho Barabakh, “Hierarchical K-means: an algo-
rithm for Centroids initialization for k-means,” Department of Information
Science and Electrical Engineering Politechnique in Surabaya, Faculty of
Science and Engineering, Saga University, Vol. 36, No. 1, 2007.
[6] K. A. Abdul Nazeer, S.D. Madhu Kumar and M. P. Sebastian, “Enhancing
the k-means clustering algorithm by using a O(n logn) heuristic method
for finding better initial centroids,” Second International Conference on
Emerging Applications of Information Technology, 2011.
[7] Chen Zhang and Shixiong Xia, “K-means clustering algorithm with im-
proved initial centroids ,” Second International Workshop on Knowledge
Discovery and Data Mining (WKDD), pp. 790-792, 2009.
[8] A Bhattacharya and R. k. De, “Divisive Correlation Clustering Algorithm
(DCCA) for grouping of genes: detecting varying patterns in expression
profiles,” bioinformatics, Vol. 24 pp. 1359-1366, 2008.
[9] Kathiresan V. and Dr. P Sumanthi, “An Efficient Clustering Algorithm
based on Z-Score Ranking method,” International Conference on Com-
puter Communication and Informatics (ICCCI-2012), Jan. 10-12, 2012.
[10] J. Han and M. Kamber, Data Mining Concepts and Techniques, Morgan
Kaufmann Publishers, San Diego, 2001.