Map-Reduce (Hadoop) based Data Clustering for
BigData: A Survey
A. J. Umbarkar1, S. A. Jadhav2, P.D.Sheth3 and A. S. Sathe4
Walchand College of Engineering, Sangli, Maharashtra, India
1
anantumbarkar@gmail.com
Abstract- Data clustering problems are challenging due to
bigdata. The volume, value, variety, velocity, veracity and
structure of data makes clustering more complex. Processing
time for data mining is directly proportional to the data to be
processed. To access big data efficiently, data clustering may be
a good solution. Traditional data clustering algorithms have
been considered as NP-Hard problem. The clustering of big
data also comes under the roof of the NP-Hard problem.
Clustering algorithms require huge computations to make the
cluster of data. An increase in data increases the number of
computations. Hence, research in use of distributed computing
for clustering algorithms and improvising resources utilization.
However, distributed computing gives the worst results when
data has dependencies. Distributed system like Hadoop run the
clustering algorithms as the same code on partitioned data on
different nodes of computing and finally gathers the result in
one place. This paper gives a survey of Map-Reduce based big
data clustering algorithms and problem solved.
Keywords- Big-data, Hadoop, Spark, Data Clustering, Map-
Reduce.
I. INTRODUCTION
Data storage has become cost-efficient and faster, but
accessing storage devices remains a bottleneck due to limited
I/O speed. Data can be structured or unstructured, with
unstructured data lacking a predefined data model. Database
management systems (DBMS) are effective in handling
structured data. However, when data surpasses processing
capacity or system RAM, it becomes big data, which requires
distributed processing across multiple computing nodes. This
consumes more computational power and storage resources
[1]. Hadoop and Spark have commonly used solutions for big
data processing and clustering.
By combining similar data and separating it from
unrelated data, the concept of clustering makes the process of
processing data more straightforward. In order to generate
clusters, a similarity matrix must be created and comparable
items must be chosen. NP-Hard problems exist in clustering.
[2]. In several disciplines, including artificial intelligence,
data mining, and image analysis, clustering is a critical task.
However, because of the data's increased complexity and
size, clustering large data can be difficult and time-
consuming. Although data reliance presents a problem that
must be overcome and lowers performance, parallelization
can be helpful. A simple programming framework that
provides a high degree of abstraction for parallelization, data
distribution, and load balancing is required to handle these
problems. To maximize the use of all resources, a 100%
parallel processing technique is also necessary.
To achieve faster clustering of big data, distributed
environments, and the Map-Reduce architecture are helpful.
The Map-Reduce library provides an abstraction layer that
simplifies parallelization, fault tolerance, data distribution,
and load balancing for programmers [3]. Developing
algorithms suitable for big data processing can be
challenging, but the Hadoop framework has been developed
for this purpose. Hadoop is a distributed environment
architecture that allows remote computers to communicate
and coordinate their actions using messages. HDFS is closely
integrated with Map-Reduce for efficient big-data processing
[4].
The architecture of Map-Reduce is intended to enable the
distribution of programming and the processing of massive
datasets. It consists of three primary functions: Shuffle,
Reduce, and Map. The Map function is responsible for data
mapping to different nodes in a distributed environment,
while the Reduce function performs a summary operation
and gathers all results in one location. The infrastructure or
framework of Map-Reduce manages parallel processing, data
transfers, and communication among different parts of the
system, while also ensuring fault tolerance and redundancy
[5].
II. LITERATURE REVIEW
There are various clustering techniques/algorithms to
distinguish dissimilar data. Here, taxonomy of the clustering
algorithm as listed below:
The K-means clustering algorithm creates "hard" clusters
by dividing M points in N dimensions into K clusters and
reducing the sum of squares of distances between points
within each cluster. [6], [7]. In contrast, the "soft" Fuzzy C-
means clustering algorithm assigns a degree of membership
to each feature vector in each cluster [7], [8]. Hierarchical
clustering can be either "Bottom-Up" (Agglomerative) or
"Top-Down" (Divisive) and uses metrics such as Euclidean
Distance, Square Euclidean Distance, and Manhattan
distance to calculate proximity [9]. Canopy Clustering is
often used as a pre-processing step for other clustering
algorithms, and variations of K-means, such as K-medians
and K-medoids, also exist [10].
Anchalia, Koundinya, and Srinath [11] focused on data
mining and aimed to improve information retrieval speed
through clustering. They proposed a parallel implementation
architecture to speed up data clustering. The authors
emphasized the importance of handling outliers and
suggested improvements in stopping criteria and cluster
initialization for better results. They provided algorithm and
implementation details for K-means on Map-Reduce
architecture.
Zhao, Ma, and He [12] proposed that clustering data has
become challenging with increasing data volume, and a more
complex clustering approach is required. The proposed
solution for dealing with the challenge of clustering large
datasets is a parallel K-means algorithm that uses Map-
Reduce. In this algorithm, the map function assigns each
sample to its closest center, while the reduce function
modifies the center values. A combiner function reduces
network communication, and the dataset is divided and
broadcast to all mappers To enable each mapper to calculate
the closest center point for each data point, K-means
constructs a universal variant center for every map task. The
intermediate values generated during the algorithm's
execution include the index of the closest center point and
related information.
Ene, Im, and Moseley [13] experimented on designing a
Map-Reduce algorithm for k-means, specifically for k-
median and k-center algorithms. They evaluated various
serial and parallel algorithms for the k-median problem, and
proposed a solution to reduce the overhead of storing more
information for each un-sampled point. The solution involves
selecting the nearest sampled point and assigning additional
weight to it based on the number of un-sampled points it is
closest to. However, assigning weight to each point in the
sample adds extra time to the Map-Reduce K-Median
algorithm, which needs to be addressed. The author
acknowledges that the Map-Reduce architecture is not the
best model for iterative programming.
Esteves and Rong [14] experimented on a large, realistic,
and noisy dataset using two data clustering algorithms,
namely k-means and FCM (fuzzy c-means), and evaluated
the results using Apache Mahout, a free cloud computing
solution, and the latest Wikipedia articles. They
demonstrated that dimensionality reduction plays a critical
role in document clustering and that FCM performs worse
than k-means in the presence of noise. The convergence
speed of both algorithms is affected by the initialization of
the cluster center. Different initialization methods provide
varying convergence times for large datasets. In general,
FCM is faster than k-means, but random initialization can
yield different results, and it is difficult to predict which
algorithm will be faster
Lou, Li and Liu [15] proposed a solution to improve the
performance of the fuzzy C-means (FCM) algorithm by
introducing a distance regularity factor. The use of Euclidean
distance as the similarity measurement criterion in the
conventional FCM algorithm can lead to unequal data
distribution and reduced clustering performance, especially
when there are variations in cluster shape and density. The
distance regularity factor was designed to overcome these
limitations by ensuring that correct similarity measurements
are assigned when calculating similarity criteria between the
cluster center and sample points. The distance regularity
factor introduced in the study takes into account the cluster
density, which provides information on the global
distribution of points within a cluster. It was applied to the
conventional FCM algorithm to correct for distance
measurement, and the results showed good performance
across various clustering densities.
Xie, Yin, Ruan, Ding, Tian,
Majors, Manzanares, and Qin [16] demonstrates that
neglecting data locality in heterogeneous environments can
TABLE I. PROS AND CONS OF MAP-REDUCE BASED DATA CLUSTERING
decrease Map-Reduce performance. Although the literature
focuses on clustering using Map-Reduce, the authors argue
that data clustering is a prerequisite for improving Map-
Reduce performance. They highlight the challenge of
distributing data in a distributed environment and Map-
Reduce architecture. Instead of distributing random data to
random nodes, the authors propose performing data
clustering first to group similar data together and transfer
them to the same node. Data clustering can enhance data
locality and improve performance for data-intensive
applications by balancing data distribution across nodes.
Cordeiro, Traina, Traina, López, Kang and Faloutsos [17]
identify bottlenecks as a problem when using Map-Reduce
for data clustering, and propose minimizing I/O costs by
considering existing data partitions that reduce network costs
between processing nodes. To address this, they introduce
the "Best of Both Worlds" (BOW) strategy, which uses a
cost function to select the best approach and achieves good
results without user-defined parameters. The approach was
evaluated on both actual and artificial data with billions of
points, utilizing up to 1,024 cores in parallel.
Ekanayake, Li, Zhang, Gunarathne Bae and Fox [18]
proposed Twister, a distributed in-memory Map-Reduce
runtime, with the goal of optimizing iterative operations or
computations over Map-Reduce. They presented the
extended programming model and architecture of Twister,
and compared it with Map-Reduce in terms of performance
on various applications including data clustering, computer
vision, and machine learning. The experiments showed that
Twister performed well and scaled effectively for many
iterative Map-Reduce computations.
Pal, Pal, Keller, and Bezdek [19] proposed (PFCM)
Clustering Algorithm; this clustering algorithm is less
sensitive to outliers and avoids coincident clusters. PFCM is
more suitable for clustering overlapping big data datasets.
Heidari, Alborzi, Radfar, Afsharkazemi and Ghatari [20]
proposed a Map-Reduce based framework for clustering big
data that addresses the challenge of varied densities using an
efficient and scalable parallel method. They proposed a new
algorithm to overcome the problem of varying densities in
density-based clustering and designed a three-layer Map-
Reduce based algorithm on top of the Hadoop platform to
analyze clusters with a variety of densities. Their approach
improved scalability and execution time while focusing on
local density of points.
Gerakidis, Megarchioti and Mamalis [21] experimented
two clustering techniques, namely, a K-Means-based fast
clustering technique called Big K-Clustering and a hybrid
clustering approach, for document clustering on large-scale
data. They adapted these techniques for use in the Map-
Reduce model.
Liu, He, He, Zhang, and Guizani [22] proposes a new
segmentation algorithm for agricultural image processing,
designed to handle large datasets using the Apache Spark
framework and provides experimental results on several real-
world datasets.
Table 1 shows pros and cons of various Map-Reduce
(Hadoop) Based Data Clustering articles.
 Article
k-means in parallel using
MapReduce [12]
Fast clustering using Map-Reduce
[13]
A comparison of K-means versus
fuzzy C-means in the cloud using
Mahout to cluster the most recent
articles from Wikipedia [14]
Fuzzy C-means clustering technique
enhancement based on cluster
density [15]
Increasing Map-Reduce performance
by putting data in diverse Hadoop
clusters [16]
Pros
- Proposes a Map-Reduce-based parallel K-means technique
that can handle massive amounts of data and be easily scaled
up.
- Introduces several optimizations to the algorithm to improve
its performance.
- Compares the performance of the proposed algorithm to
existing cutting-edge parallel K-means algorithms and
provides a thorough evaluation of the proposed algorithm on a
sizable dataset.
- The algorithm is designed to minimize data movement and
network communication, which can significantly improve the
overall performance of the clustering process.
- Provide a comprehensive evaluation of the proposed
algorithm on various datasets and show that it outperforms
several other state-of-the-art clustering algorithms.
- Compares the performance of two clustering algorithms, K-
means, and fuzzy C-means, for clustering Wikipedia's latest
articles, using the Mahout machine learning library in a cloud
environment.
- Highlights the advantages and limitations of each algorithm
and provides insights into their strengths and weaknesses in
different clustering scenarios.
- Introduces an improved version of the Fuzzy C-Means
clustering algorithm.
- Proposed algorithm considers both the distance between data
points and the density of data points in the cluster for assigning
membership.
- Paper proposes a data placement strategy to improve Map-
Reduce performance in heterogeneous Hadoop clusters.
- Proposed strategy considers the heterogeneity of nodes, data
size, and network bandwidth to optimize data placement.
- Experiments demonstrate that the proposed strategy can
significantly improve Map-Reduce performance in
Cons
- Assumes that the Map-Reduce framework
is available.
- The proposed algorithm is specifically
designed for K-means.
-The proposed algorithm is based on a single
dataset.
- Evaluation of the proposed algorithm is
based on a limited set of datasets, and further
testing and evaluation on other datasets and
platforms are needed to confirm the
generalizability of the proposed algorithm.
- Evaluation of the proposed algorithms is
limited to a specific dataset (Wikipedia
articles), and further testing on other datasets
and platforms may be needed to confirm the
generalizability of the results.
- Experimental evaluations are conducted on
a limited number of datasets, which may not
be representative of all clustering scenarios.
- Proposed strategy assumes a static
workload, which may not be applicable to
dynamic workloads.
- Paper lacks a detailed comparison with
other data placement strategies in the
literature.

Using Map-Reduce, clustering very
huge multidimensional datasets [17]
Twister: an iterative map runtime-
Reduce [18]
An technique for probabilistic fuzzy
c-means clustering [19]
Map-Reduce-based big data
clustering with varying densities [20]
Big text data clustering techniques
using Hadoop and Spark [21]
heterogeneous Hadoop clusters compared to baseline
approaches.
- Proposes a Map-Reduce based clustering algorithm for very
large multi-dimensional datasets.
- The proposed algorithm is scalable and can handle datasets
with billions of points and dimensions.
- Proposes twister, a runtime for iterative Map-Reduce
applications that can reduce overhead and improve performance
compared to traditional Map-Reduce.
- Supports multiple programming languages, and can run on
various platforms, including Hadoop and Mesos.
- Experiments demonstrate that Twister outperforms traditional
Map-Reduce for iterative applications, including PageRank, k-
means, and support vector machines.
-Proposes a new clustering algorithm that addresses some
limitations of existing algorithms, such as the sensitivity to
noise and the difficulty in handling overlapping clusters.
- Introduces a new parameter called the "possibilistic exponent"
to control the degree of overlap between clusters.
- Experiments demonstrate that the proposed algorithm
outperforms traditional clustering algorithms, including
DBSCAN and K-Means, on various large datasets.
- Paper proposes a clustering algorithm for big data based on
the (VDBSCAN) algorithm and Map-Reduce framework.
- Experiments demonstrate that the proposed algorithms can
- Proposed algorithm has high computational
overhead due to the use of Map-Reduce,
which may not be suitable for real-time
clustering applications.
- Paper does not discuss the performance
impact of the choice of the Map-Reduce
platform and configuration on the proposed
algorithm.
-Paper lacks a comparison with other
iterative Map-Reduce runtimes in the
literature, which may limit the
generalizability of the findings.
- Proposed runtime may not be suitable for
non-iterative Map-Reduce applications,
which are common in big data processing.
- Experimental evaluation may not be
comprehensive enough to fully assess the
strengths and weaknesses of the algorithm,
and it is unclear how sensitive the results are
to the choice of parameters.
- Paper does not discuss the computational
complexity of the algorithm or provide any
implementation details, which may limit its
practical applicability.
- Proposed algorithm assumes that the data
can be partitioned into equal-sized chunks,
which may not always be possible or
efficient in practice.
- Paper does not discuss the performance
impact of the choice of the Map-Reduce
platform and configuration on the proposed
algorithm.
- Proposed algorithms are limited to text
 handle large text datasets and achieve high clustering accuracy
compared to traditional clustering algorithms.
- Paper proposes two efficient text data clustering algorithms
based on the K-Means and Affinity Propagation algorithms,
implemented using Hadoop and Spark.
Agricultural Image Segmentation
Using Fuzzy c-Means [22]
- Paper proposes a new segmentation algorithm for agricultural
images processing.
- Provides test results on a number of real-world datasets.
III. DISCUSSION
Overlapping datasets present a challenge for data
clustering due to their mixed characteristics and the
limitations of hard clustering. Fuzzy C-means, are more
suitable for clustering overlapping big data datasets. The
Map-Reduce (Hadoop) architecture is useful on distributed
computing with automatic load balancing for handling large
datasets. Researchers are faced with the future challenge of
solving data clustering problems using Fuzzy clustering with
Map-Reduce for voluminous overlapping datasets or big
data.
When using the Fuzzy C Means algorithm for soft
clustering on large amounts of data through Map-Reduce, it's
important to carefully consider several factors. These include
choosing the right method for initializing cluster center
points, selecting efficient criteria for measuring similarity to
speed up the convergence of the algorithm, determining
appropriate stopping criteria, and correctly placing data in a
heterogeneous network as part of the pre-processing stage.
By varying the Fuzzy C Means algorithm in these ways, its
performance in soft clustering can be improved, and it's
necessary to parallelize these variations in order to evaluate
their performance effectively.
In future scope, Fuzzy C Means should be experimented
on spark for voluminous overlapping datasets or big data for
clustering problems. In a variety of data processing
applications, deep learning approaches like neural networks
and convolutional neural networks have shown considerable
potential, and there is a growing interest in exploring their
potential for clustering Big Data.
ACKNOWLEDGMENT
It seems like a statement of acknowledgment and
gratitude to other authors whose research work in the field of
data clustering and Map-Reduce has been published in
various conference proceedings and journals.
REFERENCES
[1] R. A. Arsekar, A. V. Chikhale, V. T. Kamble, and V. N. Malavade,
"Comparative Study of MapReduce and Pig in Big Data," in
data.
- Paper does not discuss the trade-off
between performance and ease of use of the
proposed algorithms compared to other text
data clustering algorithms.
- Paper does not discuss the limitations or
potential drawbacks of the proposed
algorithm, such as the trade-offs between
accuracy and computational complexity or
the sensitivity to noise and outliers.
International Journal of Current Engineering and Technology, vol. 5,
pp. 688-691, 2015.
[2] M. Mahajan, P. Nimbhorkar, and K. Varadarajan, "The planar k-
means problem is NP-hard," in WALCOM: Algorithms and
Computation, ed. Springer, 2009, pp. 274-285.
[3] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing
on Large Clusters," in Communications of the ACM, vol. 51, no. 1,
pp. 107-113, Jan. 2008.
[4] J. Dean and S. Ghemawat, "MapReduce: A Flexible Data Processing
Tool," Communications of the ACM, vol. 53, no. 1, pp. 72-77, Jan.
2010.
[5] T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy,
and R. Sears, "MapReduce Online," in Proceedings of the 7th
USENIX Symposium on Networked Systems Design and
Implementation (NSDI), 2010, pp. 20-34.
[6] R. Suganya and R. Shanthi, "Fuzzy C-Means Algorithm - A Review,"
International Journal of Scientific and Research Publications, vol. 2,
no. 12, pp. 1-3, Dec. 2012.
[7] J. A. Hartigan and M. A. Wong, "Algorithm AS 136: A k-means
clustering algorithm," Applied Statistics, vol. 28, no. 1, pp. 100-108,
1979.
[8] http://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/C/
fk_means.htm dated 03 April 2023.
[9] R. L. Cannon, J. V. Dave, and J. C. Bezdek, "Efficient
implementation of the fuzzy c-means clustering algorithms," IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no.
2, pp. 248-255, Mar. 1986, doi: 10.1109/TPAMI.1986.4767778.
[10] A. K. Jain, "Data clustering: 50 years beyond K-means," Pattern
Recognition Letters, vol. 31, no. 8, pp. 651-666, Jun. 2010.
[11] P. Anchalia, A. Koundinya, and N. Srinath, "MapReduce Design of
K-Means Clustering Algorithm," in Information Science and
Applications (ICISA), 2013 International Conference on, 2013, pp. 1-
5, DOI: 10.1109/ICISA.2013.6579448.
[12] W. Zhao, H. Ma and Q. He, "Parallel k-means clustering based on
Map-Reduce," in Cloud Computing, vol. 5931, pp. 674-679, Springer,
2009, doi: 10.1007/978-3-642-10665-1_71.
[13] A. Ene, S. Im, and B. Moseley, "Fast clustering using MapReduce," in
Proceedings of the 17th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pp. 681-689, ACM, 2011.
doi: 10.1145/2020408.2020515.
[14] R. M. Esteves and C. Rong, "Using Mahout for clustering Wikipedia's
latest articles: A comparison between k-means and fuzzy C-means in
the cloud," in Cloud Computing Technology and Science
(CloudCom), 2011 IEEE Third International Conference on, 2011, pp.
565-569, doi: 10.1109/CloudCom.2011.86.
[15] X. Lou, J. Li, and H. Liu, "Improved fuzzy C-means clustering
algorithm based on cluster density," Journal of Computational
Information Systems, vol. 8, pp. 727-737, 2012.
[16] J. Xie, S. Yin, X. Ruan, Z. Ding, et al., "Improving MapReduce MapReduce," J. Big Data, vol. 6, p. 77, 2019, doi: 10.1186/s40537-
performance through data placement in heterogeneous Hadoop 019-0236-x.
clusters," in Parallel & Distributed Processing, Workshops and Phd [21] S. Gerakidis, S. Megarchioti, and B. Mamalis, "Efficient big text data
Forum (IPDPSW), 2010 IEEE International Symposium on, 2010, pp. clustering algorithms using Hadoop and Spark," arXiv preprint
1-9. DOI: 10.1109/IPDPSW.2010.5470880. arXiv:2112.00200, 2021.
[17] R. L. F. Cordeiro, C. Traina, A. J. M. Traina, J. López, U. Kang, C. [22] B. Liu, S. He, D. He, Y. Zhang, and M. Guizani, "A Spark-Based
Faloutsos, "Clustering very large multi-dimensional datasets with Parallel Fuzzy c-Means Segmentation Algorithm for Agricultural
MapReduce," in Proc. 17th ACM SIGKDD Int. Conf. Knowl. Discov. Image Big Data," IEEE Access, vol. 7, pp. 42169-42180, 2019.
Data Min., 2011, pp. 690-698, doi: 10.1145/2020408.2020516. doi:10.1109/ACCESS.2019.2900635
[18] J. Ekanayake, H. Li, B. Zhang, T. Gunarathne, S.-H. Bae, and G. Fox,
"Twister: A runtime for iterative MapReduce," in Proc. 19th ACM
Int. Symp. High Perform. Distrib. Comput., 2010, pp. 810-818, doi:
10.1145/1851476.1851592.
[19] N. R. Pal, K. Pal, J. M. Keller, and J. C. Bezdek, "A possibilistic
fuzzy c-means clustering algorithm," IEEE Trans. Fuzzy Syst., vol.
13, pp. 517-530, 2005, doi: 10.1109/TFUZZ.2004.840099. .
[20] S. Heidari, M. Alborzi, R. Radfar, M.A. Afsharkazemi & A. R.
Ghatari, "Big data clustering with varied density based on