Load Balancing through Block Rearrangement Policy
for Hadoop Heterogeneous Cluster
Ankit Shah
Information Technology
Shankersinh Vaghela Bapu Institute of Technology
Gandhinagar, India
shah_ankit101@yahoo.co.in
Abstract— To store and analyze Big Data, Hadoop is the
most common tool for the researchers and scientists. The
storage of huge amount of data in Hadoop is done using
Hadoop Distributed File System (HDFS). HDFS uses block
placement policy to split a very large file into blocks and place
them across the cluster in a distributed manner. Basically,
Hadoop and HDFS have been designed in such a way that it
works efficiently on the homogeneous cluster. But in this era of
networking, we cannot imagine having a cluster of
homogeneous nodes only. So, there is the need of storage policy
that can work efficiently on both homogeneous as well as the
heterogeneous cluster. Thus, the needs of applications that can
be executed time-efficiently based on homogeneous as well as
the heterogeneous environment can be sufficed. Data locality in
Hadoop maps the data block to process in the same node, but
often when you’re dealing with Big Data, it is required to map
the data block to the processes across multiple nodes. To deal
with this Hadoop has functionality to copy the data block
where mappers are running. This creates a lot of performance
degradation especially on heterogeneous cluster due to I/O
delay or network congestions. Here we present a Novel
algorithm to balance the data blocks on specific nodes (i.e.
custom block placement) only by dividing total nodes among
two categories like: homogeneous vs. heterogeneous or high
performing nodes vs. low performing nodes. This policy helps
to achieve better load rearrangement among the nodes and we
can put data blocks actually where we want our data to be
placed for the processing.
Keywords— Load Balancing in Hadoop, Heterogeneous
Load Balancing, Ha-doop Performance Improvement, Improved
Block Arrangement Policy, Hadoop Balancer, Performance
optimization in Hadoop;
I. INTRODUCTION
Hadoop [1] has become more popular due to its capability
to utilize general-purpose computers. Hadoop has basically
three most important constituents: The Hadoop Distributed
File System (HDFS) [2], MapReduce [3] and Yet Another
Resource Negotiator (YARN) [4]. HDFS allows nodes to
store data on the distributed cluster. HDFS said to be fault-
tolerant and can be deployed on any machine. MapReduce is
pro-gramming framework for writing Map-Reduce
applications which can run parallel on the distributed
platform. MapReduce allows applications to run on Hadoop
environment.
978-1-5386-5314-2/18/$31.00 ©2018 IEEE
Mamta Padole
Computer Science & Engineering
The M.S. University of Baroda
Baroda, India
mpadole29@rediffmail.com
To place the data blocks on nodes, Hadoop uses the
HDFS block placement policy [2]. Hadoop cluster gets
imbalanced at times, due to overutilization of few nodes
against the less utilized nodes or newly added nodes with no
blocks stored on them. To resolve this situation, Hadoop has
a built-in tool called HDFS Balancer [2].
A. HDFS Block Placement Policy
1. Hadoop stores the data blocks on the different nodes of
the cluster based on block placement policy available
in HDFS. The policy of placing blocks in Hadoop,
helps to distribute the data block uniformly amongst the
cluster nodes. This policy places the data blocks
according to the following approach: Split the files into
blocks and replicates the blocks according to replication
factor that has been already defined in the hdfs-site.xml
file.
2. If a request comes from the datanode which is the node
of the cluster, put the first replica of the block on that
datanode itself. Otherwise, place it randomly on any
datanode of the cluster.
3. The second replica of the block will be placed on nodes
of other racks if available, or maybe placed on the same
rack of the first replica.
4. The third replica will be placed on any node of the rack
where the second replica is placed.
Figure 1 demonstrates the default HDFS block placement
policy.
B. HDFS Balancer
Sometimes in the cluster, load imbalance may occur if
the load (i.e. blocks) is not uniformly distributed amongst
the nodes. In the case of Hadoop, another possible cause of
cluster imbalance is due to its elasticity. We can add
datanodes into existing cluster at any time. This may lead to
cluster imbalance as newly added nodes will not have any
data blocks. It is very essential to have a balanced cluster to
get the efficient results. HDFS Balancer tool takes care of
load balancing of the Hadoop cluster.
230
 FIGURE I. DEFAULT HDFS BLOCK PLACEMENT POLICY [5] [A]: TWO RACKS [B]: SINGLE RACK
Rack 1
Data block to be placed
First Replica
Second Replica
Third Replica
For moving number of blocks it depends upon the specified
threshold limit of the cluster. When we run HDFS balancer
tool with its default threshold of 10%, it tries to maintain an
overall 10% difference of disk usage from the overall cluster
usage. That means if overall cluster usage is 60% of the
cluster capacity than each node will have disk usage in
between 50% to 70% [6].
There are few issues with HDFS balancer, first is that, it
moves the blocks of data without looking into the
processing capacity of nodes where data blocks are being
moved and second, there is no mechanism that can be used
to control the movement of data blocks of specified
file/datasets only. Moreover it is important to consider that
moving data blocks from one node to another requires high
bandwidth, so that cluster balancing can take place faster.
In this paper, we propose a novel approach to move all
blocks to specified or pre-identified nodes only. This will
process the job faster by minimizing the inter-rack and
internode transfer. We present the load rearrangement
algorithm for specific datasets only to achieve better
performance by categorizing nodes based on their
processing capacity or heterogeneity.
This paper is structured as follows. Section 2 describes
the related work on Hadoop load balancing. Section 3 briefs
the motivation of our wok. Section 4 and 5 explain the
proposed novel approach for load balancing/ block
rearrangement and the pro-posed algorithm for the same.
Section 6 shows the experimental setup of Grid’5000 [7].
Sections 7 explain the results of the proposed algorithm
compared with the ba-lancer and in the last section; the
DataNodes
Rack 2 Single Rack
Data block to be placed
First Replica
Second Replica
Third Replica
paper is concluded with concluding remarks and our future
work.
II. RELATED WORK
To solve the load balancing issue in Hadoop there are two
common ways: First, Rebalance the cluster by migrating
data or Second, Task Migration. Researchers are mainly
working on optimizing task migration against data
migration, as data transfer among the nodes increases the
time complexity. Hadoop schedulers can achieve load
balancing to minimize the data migration of job or task to
some extent [8]. Several researchers have developed
improved scheduling algorithms apart from built-in
scheduler provided by Hadoop to contribute it for load
balancing [9,10].
In paper [11] authors propose the load balancing based
upon the disk latency and disk space utilization. Authors
have tested their custom block placement approach on the
heterogeneous cluster and show up to 20% improvement in
job running time. Authors of paper [12] propose a dynamic
load balancing algorithm for heterogeneous clusters. The
algorithm uses a sliding window-based approach to achieve
significant improvement of the Hadoop job processing. In
paper [13] authors propose to use spatial-temporal
efficiency in the heterogeneous cluster. Authors have used
predictive machine learning approach for forecast the job
execution time and space occupation. Hadaps [14] tool has
been designed for load balancing using custom block
placement policy by considering hardware generation of the
nodes. A novel dynamic load rebalancing algorithm, as
proposed by authors of paper [15]. The algorithm is tested
231
on the heterogeneous environment and emphasizes on to
reduce block movement to achieve better performance. In
paper [16] authors propose block placement scheme based
on the computing capacity of heterogeneous cluster nodes.
This approach balances the block of data based upon
considering the computing capacity of each node.
Most of the research work for load balancing is either
focusing on load balancing during scheduling [9, 10] or
load balancing during MapReduce [12,13] phase only.
However, in paper [11, 14, 15, 16] authors tried to balance
the whole cluster based on disk latency, disk utilization or
custom block placement policy. These approaches achieve
the better results especially in case of the heterogeneous
cluster.
III. MOTIVATION: OPTIMIZING TASK MIGRATION OVER
DATA MIGRATION
Hadoop is more productive on a homogeneous distributed
environment compared to the heterogeneous one [3]. In
reality, Hadoop cluster may not be the cluster of
homogeneous nodes only. Hadoop cluster may contain
nodes of different storing and processing capacity.
HDFS Block placement policy plots the data on nodes as
discussed in the section 1. The major issue is that we do not
have any control over placing the data blocks according to
our choice. There are few issues in MapReduce if load
balancing is not done properly. First, if MapReduce does not
get data blocks for processing on that node then it gets the
block from the node which already contains that block. This
requires inter-node or inter-rack block transfer which
degrades the performance of MapReduce intensively.
Second, if MapReduce is processing the data blocks on the
node which has low computation capacity then the
performance gets skewed. Therefore, a job may require
longer execution time to finish the job. It is important to
consider the cost of MapReduce job depending primarily on
the data being processed; load balancing is strongly
influenced by how data is partitioned in HDFS and where it
is stored for processing. The standard Hadoop distribution
does not necessarily meet the needs, which leads to an
imbalance between the maps and reduce. Our proposed
approach solves both the problem discussed above.
As discussed in Section 1 HDFS block placement policy
places the block replica on client node itself if client node is
part of the cluster. This makes cluster pretty much
unbalanced as that node will store one copy of whole
dataset. If that node does not leverage for MapReduce
processing then it takes more time for data processing.
We know in case of distributed computing “Moving
Computation is Cheaper than Moving Data” [17]. Therefore,
232
we propose to store all blocks of specific file evenly on
selected nodes only so minimum data transfer will take
place for MapReduce job processing. Once we get our
blocks on specific node then we can apply Node label [18]
to get actual advantage of proposed scheme. Using node
label we can group this node under single roof and specify
our job where to run. The idea is if we can select our nodes
for processing it would be better if those nodes contain
blocks of that job so that minimal internode and inter-rack
transfer will take place and we can achieve better results.
IV. THE NOVEL APPROACH
Hadoop by default will move the job to where data is
already lying. But, the node where data is lying when placed
by default HDFS block placement policy. That node which
already has the data may not have sufficient processing
capability. Therefore, even if data is there, a job is moved
but resource capacity is not there it may take more time for
processing or other nodes will get the job done by moving
data to that node. Hence, we propose the algorithm which
will place the data according to the configuration
requirement of the job that is moved, which is pre-
identified. We propose a new algorithm that exploits the
best capability of the cluster.
We can start with two assumptions in mind: First, we
have a group of heterogeneous nodes with different
computational processing capacity. Second, nodes of cluster
are heterogeneous in nature, out of which few nodes are
homogeneous in nature. Initially HDFS block placement
policy places the data block as shown in Fig. 2.
In first case, we can group nodes by its processing
capacity. Consider a fact that few nodes of cluster are very
much slow in nature which degrades the performance of the
cluster. Assign the block priority=2 for the nodes where we
want to store all blocks and priority=1 which will not store
any block of data. This setting will take place in config.xml
file and our algorithm will use this parameter at runtime for
custom block placement. Therefore, we will make list of
nodes which has higher processing capacity and will
allocate all the data blocks to that nodes only. In second
case, we can group the homogeneous nodes and assign
priority=2 to homogeneous nodes and priority=1 to rest of
other nodes. While placing all blocks our proposed
algorithm also take care that blocks of file will distribute
equally over all the nodes by considering disk usage of each
node. Each time before placing the block of file it checks
that node should not contain replica of the same block. The
fact that Hadoop works better for homogeneous
environment, fulfilled by our strategy even though cluster is
heterogeneous. Fig. 3 demonstrates the proposed custom
block placement strategy.
 FIGURE II. HDFS DEFAULT BLOCK PLACEMENT POLICY

FIGURE III. PROPOSED APPROACH GROUP-1: PRIORITY=2, GROUP-2: PRIORITY=1

Step 5: Sort the nodes (i.e. priority=2) by disk utilization in

V. PROPOSED ALGORITHM
Input: HDFS location of input files to be balanced
File contains list of data blocks which are placed in HDFS
using default block placement policy.
Output: Data blocks will be placed to specific nodes only
based on given priority factor.
Step 1: Select all blocks of selected HDFS path from
NameNode metadata
Step 2: Select all data nodes from DatanodeInfo
Step 3: Assign the priority to the nodes (Priority=2 will
store all data blocks and Priority=1 will not store any block
of data) configure the config.xml file
Step 4: Prepare data node list with priority=1 (i.e.
node_list1) and priority=2 (i.e. node_list2) from config.xml
file
ascending order and store it in node_list2
Step 6: For each block follow below steps:
- Initialize block_list containing all blocks of HDFS
path
- Initialize node_list2, the nodes having priority=2
- For each block replica in block_list do the following:
o Select the nodes from node_list2 which does not
contain replica and has lower disk utilization
among the list
o Put the block in this node and remove it from
block_list
o Remove the selected node from node_list2

233

VI. EXPERIMENTAL TEST BED
To perform experiment we have used Grid’5000
heterogeneous cluster. Grid’5000 is large scale distributed
testbed for the researchers to experiment their research on
high configurable cluster. We have used 10 nodes for our
experiment. The configuration of nodes is shown in below
table 1.
VII. RESULTS
We have tested the proposed algorithm using 4 data sets:
1. Text data sets copied from local file system having size 3
GB and 6 GB. 2. Datasets generated by TeraGen having size
3 GB and 6 GB. The experiment is carried out on Hadoop
2.7.2 with default replication factor 3 and all nodes are
empty.
Experiment 1:
In first test, we have used two text datasets of size 3 GB
and 6 GB. Firstly we have placed the blocks using default
HDFS block placement policy and afterwards we applied
our proposed algorithm applying settings in table 2. The
results (Fig. 4 and 5)shows that strategy would place all data
blocks on the nodes with priority -2 and also try to maintain
the disk usage of each nodes, such that it does not differ
much.

TABLE I. DETAILS OF NODES OF THE CLUSTER TABLE II. SETTINGS FOR NODE PRIORITY

Parasilo-1 to 6 CPU: Intel Xeon E5-2630 V3 , Priority - 2 Priority - 1

[Parasilo-1 : 2.40 GHz 2 CPUs/node, parasilo-1.rennes.grid5000.fr parapide-1.rennes.grid5000.fr
Namenode & 8 cores/CPU parasilo-2.rennes.grid5000.fr parapide-2.rennes.grid5000.fr
Datanode] Memory: 128 GB parasilo-3.rennes.grid5000.fr parapide-3.rennes.grid5000.fr
Network: 1 Gigabit Ethernet parasilo-4.rennes.grid5000.fr parapide-4.rennes.grid5000.fr
Storage: 558 GB / HDD SATA parasilo-5.rennes.grid5000.fr
Parapide-1 to 4 CPU: Intel Xeon X5570 , parasilo-6.rennes.grid5000.fr
[Datanodes] 2.93GHz 2 CPUs/node,
4 cores/CPU
Memory: 24 GB
Network: 1 Gigabit Ethernet
Storage: 465 GB / HDD SATA

FIGURE IV. PROPOSED ALGORITHM FOR DATASET – 3 GB FIGURE V. PROPOSED ALGORITHM FOR DATASET – 6 GB

Experiment 2: generating tool of Hadoop. Using TeraGen we generates the

3GB and 6GB size of data files and test it using the same
In second test, we have used TeraGen which generates the
settings used in experiment 1 (Table 2).
data using MapReduce and places into nodes of the cluster.
The purpose is to check our algorithm with built-in data

234
 FIGURE VI. PROPOSED ALGORITHM FOR TERAGEN – 3 GB FIGURE VII. PROPOSED ALGORITHM FOR TERAGEN – 6 GB

From the Fig. 6 and 7, it is observed that after applying
the proposed algorithm the data blocks are placed in the
selected nodes only. It can be also observed that the
approximate uniform size of data is placed in each node i.e.
disk utilization is more or less same on each node.
Timing Analysis
Fig. 8 shows the execution time taken in experiment 1
and experiment 2 by proposed algorithm. The amount of
time taken for balancing the blocks can be compensated
during job processing as it reduces the internode and inter-
rack transfer.
FIGURE VIII. BALANCING TIME OF PROPOSED ALGORITHM
affected. Experimental results prove that this scheme can be
applied to both heterogeneous as well as to homogeneous
cluster.
The future enhancement focuses in using the concept of
“Node Labeling” along with proposed scheme, which will
allow the user to put the job on specific nodes only. Node
labeling combined with our proposed algorithm improves
the performance of Hadoop. This concept will facilitate to
do processing on selected nodes only and will also arrange
to can get data blocks on those nodes only. This will
optimize the processing further. In extended version, we
intend to test the proposed algorithm with an application, so
that actual benefit of the proposed scheme can be evaluated.

REFERENCES

[1] "Welcome to Apache™ Hadoop®!", Hadoop.apache.org, 2018.

[2]
[3]
[4]
[5]
VIII. CONCLUSION AND FUTURE WORK
We propose custom block placement policy which [6]
leverages the processing capacity of CPU, during block
placement. This approach will be helpful in MapReduce to
minimize the internode and inter-rack transfer. We have
[7]
demonstrated that we can place data blocks of specific file
to specific nodes only. This approach will not affect the
overall load balancing of cluster as rest of the files won’t be
[Online]. Available: http://hadoop.apache.org/. [Accessed: 09- Jul-
2018].
K. Shvachko, H. Kuang, S. Radia and R. Chansler, "The Hadoop
Distributed File System", 2010 IEEE 26th Symposium on Mass
Storage Systems and Technologies (MSST), 2010.
Dean and S. Ghemawat, "MapReduce", Communications of the ACM,
vol. 51, no. 1, p. 107, 2008.
Vavilapalli, S. Seth, B. Saha, C. Curino, O. O'Malley, S. Radia, B.
Reed, E. Baldeschwieler, A. Murthy, C. Douglas, S. Agarwal, M.
Konar, R. Evans, T. Graves, J. Lowe and H. Shah, "Apache Hadoop
YARN", Proceedings of the 4th annual Symposium on Cloud
Computing - SOCC '13, 2013.
Maheshwari, R. Nanduri and V. Varma, "Dynamic energy efficient
data placement and cluster reconfiguration algorithm for MapReduce
framework", Future Generation Computer Systems, vol. 28, no. 1, pp.
119-127, 2012.
"HDFS Balancers | 5.7.x | Cloudera Documentation", Cloudera.com,
2018. [Online]. Available:
https://www.cloudera.com/documentation/enterprise/5-7-
x/topics/admin_hdfs_balancer.html. [Accessed: 07- Jul- 2018].
"Grid5000", Grid5000.fr, 2018. [Online]. Available:
https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home.
[Accessed: 07- Jul- 2018].

235
[8] A. Shah and M. Padole, "Performance Analysis of Scheduling optimization in a heterogeneous cloud environment", Security and
Algorithms in Apache Hadoop." Communication Networks, vol. 9, no. 17, pp. 4002-4012, 2016.
[9] Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.,“ [14] "fluxroot/hadaps", GitHub, 2018. [Online]. Available:
Improving MapReduce Performance in Heterogeneous https://github.com/fluxroot/hadaps. [Accessed: 08- Jul- 2018].
Environments” , Proceedings of the 8th conference on Symposium on [15] Xie, S. Yin, X. Ruan, Z. Ding, Y. Tian, J. Majors, A. Manzanares and
Operating Systems Design and Implementation, , pp. 29–42. ,2008 X. Qin, "Improving MapReduce performance through data placement
[10] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker in heterogeneous Hadoop clusters", 2010 IEEE International
and I. Stoica, "Delay scheduling", Proceedings of the 5th European Symposium on Parallel & Distributed Processing, Workshops and
conference on Computer systems - EuroSys '10, 2010. Phd Forum (IPDPSW), 2010.
[11] Dharanipragada, S. Padala, B. Kammili and V. Kumar, "Tula: A disk [16] H. Hsiao, H. Chung, H. Shen and Y. Chao, "Load Rebalancing for
latency aware balancing and block placement strategy for Distributed File Systems in Clouds", IEEE Transactions on Parallel
Hadoop", 2017 IEEE International Conference on Big Data (Big and Distributed Systems, vol. 24, no. 5, pp. 951-962, 2013.
Data), 2017. [17] "Apache Hadoop 2.7.2 – HDFS Architecture", Hadoop.apache.org,
[12] Y. Liu, W. Jing, Y. Liu, L. Lv, M. Qi and Y. Xiang, "A sliding 2018. [Online]. Available:
window-based dynamic load balancing for heterogeneous Hadoop https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-
clusters", Concurrency and Computation: Practice and Experience, hdfs/HdfsDesign.html. [Accessed: 09- Jul- 2018].
vol. 29, no. 3, p. e3763, 2016. [18] "Apache Hadoop 2.7.2 – YARN Node Labels", Hadoop.apache.org,
[13] Q. Liu, W. Cai, J. Shen, Z. Fu, X. Liu and N. Linge, "A speculative 2018. [Online]. Available:
approach to spatial-temporal efficiency with multi-objective https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-
site/NodeLabel.html. [Accessed: 09- Jul- 2018].

236