JOIN ALGORITHMS USING MAPREDUCE: A SURVEY

VIKAS JADHAV1, JAGANNATH AGHAV1, SUNIL DORWANI2

1
Department of Computer Engineering, and Information Technology
2
SAS Research and Development (India) Pvt. Ltd., College of Engineering Pune, India.

Abstract- MapReduce framework is widely used in large scale data analysis. It is highly scalable and programming model
ofconvenience. However, performance of MapReduce is a concern when applied on complex data analytical task involving
multiplejoin of datasets for certain aggregates. Join operation is frequentin data processing. Original MapReduce
programming modeldoes not directly support processing multiple related heterogeneousdatasets thus MapReduce has been
modified to supportheterogeneous datasets. Big Data analytics uncovers the hidden patterns and otheruseful information and
a popular data processing engine forbig data is Hadoop MapReduce. Hadoop MapReduce is usedheavily for data analytical
tasks like data mining, SQL queries,etc. Hive is data warehousing resides on the top of Hadoopand provides sql-like
interface to HDFS files. In this paper,we survey various methods of joining datasets using HadoopMapReduce.

Keywords- Analytics, Big Data, Hive, Join, MapReduce, SQL.

I. INTRODUCTION

With exponential growth in data size distributed
processingof data has become important. Parallel
Database System [1] are based on shared- nothing-
node (separate CPU, memory, disk) and connected
through high-speed interconnect. Every Parallel
Database System uses horizontal data partitioning
along with partitioned execution of SQL queries.
Horizontal partitioning involves distributing rows of
relational table across nodes of cluster so that rows
can be processed in parallel. MapReduce [2] is
revolutionary platform for large scale data processing
in distributed manner. MapReduce is composed of a
master and multiple workers for conducting
MapReduce jobs. It can process huge amount of data
in small reasonable time using large number of
commodity hardware therefore valuable information
hidden in those big data can be extracted with much
less cost. MapReduce programs are automatically
parallelized and executed across cluster. Users don’t
need to handle parallelism and failures, MapReduce
framework automatically parallelizes user program
and reschedules failed task. Yahoo! has deployed
largest Hadoop cluster with 4000 nodes [3].
According to [1] MapReduce and Parallel DBMS are
complementary to each other. Hadoop MapReduce
[4] is open source implementation of Google’s
MapReduce [2]. Hadoop is developed by Yahoo! and
used at Facebook, Amazon, Last.fm and many other
companies and institutes [5].
Hadoop MapReduce is big data processing
framework thatrapidly became the de facto standard
in both academia andindustry [6]–[9]. To enable
experts in database field Facebookprovided interface
to Hadoop data using SQL like interfacecalled Hive
[10]. Hive converts query into Hadoop MapReduce
jobs and
executes these automatically on Hadoop cluster. Most
of the data Analytical queries involve joining of
Multiple datasets [11]–[14].
The rest of paper is organized as follows section II
describes Big Data, section III covers Hadoop,
section IV Bloom Filter, section V discusses various
methods for join using Hadoop MapReduce and
finally section VII covers conclusion.
II. BIG DATA
Big Data [15] refers to practice of collecting and
processing very large datasets and associated system
and algorithms used to analyse these massive datasets.
Big Data which is getting global attention can be
described with the help of three V’s, Volume,
Velocity, Variety [16]. Main issues related to big data
are capturing, storing searching, sharing, analysing
and visualization.
Big data has following properties:
1) Volume: Big Data has very large volume of
data. It may grow up to hundreds of gigabytes
to petabytes, and may have large number of
records.
2) Velocity: Rate at which data comes is very high
and this requires tools to respond quickly.
3) Variety: Big data may have range of data types
and data sources.
Architecture of Big Data generally spans across
multiple machines and clusters. Hadoop and HDFS
has become dominant platform for Big Data Analytics
[17].
III. HADOOP
The Apache Hadoop [4] is open source project which
develops software framework for distributed

International Conference on Electrical Engineering and Computer Science, 21st April-2013, Coimbatore, ISBN: 978-93-83060-02-3
40
 Join Algorithm
ms using MapRed
duce: A Survey

processinng of data in fault tolerantt manner. Haadoop Bloo om Filter aree initially set to zero. Forr inserting
frameworrk is designedd to scale froom single macchine elem
ment in Bloom m Filter, elemment is hashed k times
to thousaands of machinnes. withh k hash funcction and possitions in Blo oom Filter
arrayy correspondiing to hash vaalue are set to 1. To test
whether element is present iin Bloom Fiilter array
checck that all bitts of k hash positions aree 1 if yes
elem
ment is presennt in set otherw wise it is not present in
set. Bloom Filterr may yield fa false positivess but false
negaatives are nevver generated. Advantage of Bloom
Filteer is space efficiency. Bloom filteers allow
memmbership queries without the need off original
colleection. We canc concludee that elemen nt is not
present in the origginal collectioon if at least one
o of the
A. MapR Reduce positions computted by the hhash function ns of the
MapReduuce [2] is framework
fr foor processing and blooom filter pointts to a bit whhich is set to 0. Size of
generatinng large data sets.
s MapReduuce frameworrk for blooom filter is fixed
fi but therre is trade-offf between
data analyysis is effectivve when files are very largee and Bloo om Filter veector size m and falsee positive
are rarelyy updated. Proogrammers off MapReduce need probbability p. Probability
P off false posittive after
to speciffy two functiions map() annd reduce(). Both inserrting n numbeer of elementss is given by following
map() annd reduce() fuunction takes input as paiirs of equaation (1).
key and value. Map() function takees (key,value)) pair (1)
as input and
a produces set of interm mediate (key,vvalue)
pairs. Then reduce() fuunction takes these intermeediate
(key,valuue) pair as innput and mergges all the values
v V. JOIN
J USING
G MAPREDU
UCE
with samme intermediatee key.
Programss written usingg MapReducee Programminng are In this
t section we are goinng to discuss various
automaticcally parallellized and exeecuted acrosss the meth hods availabble for jooining dataset using
cluster using
u commmodity hardw ware. MapReeduce Map pReduce.
frameworrk takes caree of partitioniing of input data, A. Reduce-Side
R J
Join
schedulinng of user program on clustter and handlinng of Redu uce-Side Joinn [22]–[24] iss simple join technique
failure. Programmer
P n
need not to worry
w about these wheere we map overo both the datasets and emit join
issues.MaapReduce Proogramming Model: M MapReeduce key as intermediiate key and tuple as inttermediate
program takes set of o (key,valuee) as input and ue. To know from
valu f which ddataset record came we
produces set of (key,vvalue) as output. User exprresses tag intermediate (key,value) ppairs with dataaset name
computattion as two functions maap() and reduuce(). to produce
p taggeed (key,value)) pairs. Outpu uts of the
Singaturees of map and reduce are given below. mapppers are paartitioned, soorted and mergedm by
map(k1,v1) →list(k22,v2) fram
mework. All thhe record withh same join arre grouped
reduce(k2,listt(v2)) → list(kk3,v3) togeether and fed to reducer. Reducer sepaarates and
bufffers the input records into ttwo sets by using
u table
B. Hadooop Distributedd File System (HDFS)
( tag and
a then perfo forms cross prroduct of thesse records.
Hadoop uses file systtem called Hadoop
H Distribbuted (keyy,value) pairs with dataset nname to produ uce tagged
File Systtem (HDFS) [18]. HDFS stores file syystem (keyy,value) pairss. Outputs of the map ppers are
metadata and applicattion data sepaarately. HDFS S has partiitioned, sortedd and mergedd by frameworrk. All the
separate server to storee file system metadata called as reco
ord with same join are grouuped together and a fed to
NameNode and appliccation data iss stored on server redu
ucer. Reducerr separates aand buffers the input
called as DataNode. Unlike
U other file
f systems HDFS
H reco
ords intotwo sets by usinng table tag and then
does not use RAID [19] for data protection. HDFS H perfforms cross product of thesee records.
uses repplication tecchnique for data protecction.
Contents of file are repplicated on multiple
m DataNNodes
for reliaability purpose. This strrategy has added a
advantage that data trransfer bandw width is multiiplied
and has more opportuunity for locaating computtation
near dataa.

OOM FILTER
IV. BLO R

Bloom Filter [20], [211] is probabiliistic data struucture

used toccheck whetheer element iss member off set.
Bloom Filter
F consists of array of o m bits annd k
independdent hash funcctions. All thhe bits in arraay of

Internaational Conferencce on Electrical Engineering

E and Computer
C Sciencee, 21st April-20133, Coimbatore, ISBN: 978-93-8306
60-02-3
41
 Join Algorithms using MapReduce: A Survey
Basic idea behind Reduce-Side join is repartition both
the datasets by join key. Reduce-Side join approach is
not efficient since it involves shuffling of both the
datasets across the network.
B. Map-Side Join
Reduce-Side join method is natural way of joining
datasets; it uses built-in capacity of Hadoop
frameworks to sort intermediate (key,value) pairs
before they reach reducer. Hadoop offer another way
of joining datasets using Map-Side join [23]– [26]. If
both the datasets are partitioned and sorted on join
key attribute then joining of datasets can be
accomplished in parallel using map phase of
MapReduce job. We invoke map over one of the
(larger) dataset, in that mapper we read corresponding
part of another (smaller) dataset and perform merge
join. Reducers are not required unless programmer
want to repartition dataset or perform further
processing. Fig. 3 shows data flow for Map-Side Join
technique.
Map side joins are more efficient than reduce side join
since we are avoiding sorting and shuffling phases but
map side join has very stringent condition that both
the datasets should be sorted and partitioned on join
key.
C. Memory-Backed Join
Memory Backed Joins [26] are family of join
technique. Simplest version of memory-backed join is
broadcast join technique where one of the two dataset
completely fit into memory of each node in Hadoop
cluster. In this case we can load smaller dataset into
memory in every mapper so that it can be accessed by
using join key. Then mappers are applied to larger
dataset and for each key value pair, mapper probe in-
memory dataset to check whether there is tuple with
same join key. If there is then join is performed. This
method is known hash join [27] by database
community. Broadcast join in map only join
technique
Fig. 3 Data Flow for Map-Side Join.
 
Pre-processing step for broadcast join: Even though
there is no way to control physical placement of
International Conference on Electrical Engineering and Computer Science, 21st April-2013, Coimbatore, ISBN: 978-93-83060-02-3
42
replica in DFS. We can increase replication factor for
smaller dataset so that most of nodes in cluster will
have local copy of smaller dataset and they don’t need
to retrieve smaller dataset from another node.
If none of the dataset fit into memory then divide
smaller datasets into number of partitions. We choose
partition size such that it will fit into memory of node
and then run memory backed hash join. We need to
go n times through other dataset, where n is number
of partitions.
Use of distributed key value store can useful if neither
of dataset fit into memory. Distributed key-value store
can be used to store one dataset into memory of
multiple machines. Mappers can iterate over other
dataset and can query distributed key-value store in
parallel and perform join if joinkey matches. Open
source system memcached can be used for this
purpose thus this technique is called as memcached
join technique.
Fig. 4 Data Flow for Memory Backed Join
D. MapReduce Join using Bloom Filter
Method [28] discusses join processing using Bloom
Filter and Hadoop MapReduce. According [28]
MapReduce is not very efficient to perform join
operation because it always processes all records even
if small fraction of datasets are relevant for join
operation. We can improve performance by using
Bloom Filter in join operation. Bloom filter can be
constructed in distributed manner to filter out
redundant records. Section IV discusses Bloom Filter
in detail. For R 1 S operation, Let R is dataset on
which bloom filter constructed and another dataset S.
Method [28] can process only two way join involves
following steps.
1) Job submission: Job is submitted with m1 map
tasks for R and m2 map tasks for S and r reduce
tasks are created.
2) First map phase: Jobtracker assigns m1 map
tasks or reduce tasks to idle tasktrackers. A map
tasktrackers reads input split for the task and
converts it into (key,value) pairs.
 Join Algorithms using MapReduce: A Survey
3) Local filter construction: Intermediate
(key,value) pairs produced from map are sent to
r tasktrackers and also r Bloom filters are
constructed on keys in each partition. These are
local filters because they are built only on
intermediate result in single tasktracker.
4) Global filter merging: When all m1 tasktrackers
complete their map task then tasktracker send
their Bloom filters to Jobtracker. Jobtracker
constructs global filter of all local filter for
dataset R and then Jobtracker sends global filter
to all tasktrackers. Until now map task for S are
not assigned.
5) Second map phase: Jobtracker assigns m2 map
tasks for dataset Son tasktrackers. Tasktracker
runs assigned task by using received global
filters. Records which are not in global filter are
filtered out.
6) Reduce phase: This step is same as reduce
phase in Hadoop [4]. A Reduce tasktracker
reads corresponding intermediate pairs from all
map tasktrackers and it sorts all intermediate
pairs received and runs reduce function. Final
results are written output path in HDFS [18].
E. Map-Reduce-Merge
Map-Reduce-Merge [29] model enables processing
multiple heterogeneous datasets. Map-Reduce-Merge
[29] has following signature. Symbols α, β, γrepresent
dataset lineage and symbols k and v represent keys
and value respectively.
map : k1, v1 α k2, v2 α
reduce: k2, v2 α k2, v3 α
merge : k2, v3 α, k3, v4 β k4, v5
The map function transforms an input key and value
pair (k1,v1) into a list of intermediate key and value
pairs [(k2,v2)]. The reduce function aggregates the list
of values [v2] associated with k2and produces a list of
values [v3], which is also associated with k2. The
input and output belong to same dataset lineage α.
Similarly another pair of input belonging to dataset
lineage β also produces list of values [v4] associated
with k3. Merge function combines output of two
reduce function belonging to two different dataset
lineage α and β and produces final output with γ
dataset lineage. ifα = β then merge function does self-
merge which is similar to self-join.
Map-Reduce-Merge [29] is an extension to standard
MapReduce. By adding Merge Phase to perform
relational operations on dataset, it can merge dataset
naturally. MapReduce itself has performance problem
that pull operation of reduce to map may cause large
number of disk seeks and slow down effective disk
transfer rate. Map-Reduce-Merge does not help to
solve this problem instead it may aggravate this
phenomenon because amount of data transferred
becomes larger [30].
International Conference on Electrical Engineering and Computer Science, 21st April-2013, Coimbatore, ISBN: 978-93-83060-02-3
VI. OTHER METHODS
There has been lot of research going on for optimizing
MapReduce for the Analytical join queries. Method
[13] discusses optimization of analytic queries which
involve very large fact table which is to be joined
with smaller dimension table and queries which
involve paths through graph with high out degree.
According to [31] data analytical queries generally
involve joining multi-way join operation and
operators involved in join queries are more than equi-
join operator. This method calculates cost associated
with MapReduce job and also provides efficient
execution of chained type theta join queries and also
compares result with Pig [32] and Hive [8]. Map-Join-
Reduce [12] is another attempt to optimize join
operation using Hadoop MapReduce framework but
this approach focuses only on equi-join queries.
There are wide ranges of application where we
require topk similar records from database. Work
mentioned in [33] describes how we can exploit
Hadoop MapReduce for topk similarity join
algorithms. Hadoop MapReduce frameworks
implementation of join operation cannot handle skew
in input data which leads to poor load balancing this
can result in swamping of benefits of parallelization.
Atta, Viglas and Niazi introduced new technique
called SAND Join [34] which employs range
partitioning instead of hash partitioning for load
distribution.
VII. CONCLUSION
γ
The MapReduce framework enables us to process
very large dataset distributed among nodes in cluster.
For large dataset parallel processing reduces response
time since processing of subset of dataset is carried
out independently on multiple nodes. As every kind of
task can be benefited from parallel processing, join
operation also can be benefited.
Simple Technique for execution of join using
MapReduce is In-Memory. However memory-backed
technique can be used only when one of the two
datasets completely fit into memory. If both the
datasets are too large to fit into memory then we must
use map-side or reduce-side join.
Map-side join technique is more efficient than reduce
side join technique if input is sorted and partitioned,
because there is no need to shuffle datasets over the
network.
If there are only small fraction of records relevant to
join operation then applying Bloom Filter to filter out
records which will not satisfy join condition will help
in improving performance of join operation. Bloom
filter will filter out records in map phase thus only
subset of records will be transferred to reducer over
network.
43
 Join Algorithms using MapReduce: A Survey
REFERENCES
[1] M. Stonebraker, D. Abadi, D. J. DeWitt, S. Madden, E.
Paulson, A. Pavlo, and A. Rasin, “MapReduce and parallel
DBMSs: friends or foes?” Communications of the ACM,
vol. 53, no. 1, pp. 64–71, Jan.2010.
[2] J. Dean and S. Ghemawat, “MapReduce: simplified data
processing on large clusters,” CACM, vol. 51, no. 1, pp.
107–113,2008.
[3] http://www.dbms2.com/2011/07/06/petabyte-hadoop-
clusters/
[4] “Apache hadoop.” http://hadoop.apache.org/
[5] http://wiki.apache.org/hadoop/PoweredBy
[6] J. Dittrich, J.-A. Quiane-Ruiz,´ A. Jindal, Y. Kargin, V.
Setty, and J. Schad, “Hadoop++: making a yellow elephant
run like a cheetah (without it even noticing),” Proc. VLDB
Endow., vol. 3, no. 1-2, pp. 515–529, Sep. 2010.
[7] A. Jindal, J.-A. Quiane-Ruiz,´ and J. Dittrich, “Trojan data
layouts: right shoes for a running elephant,” in Proceedings
of the 2nd ACM Symposium on Cloud Computing, ser.
SOCC ’11. New York, NY, USA: ACM, 2011, pp. 21:1–
21:14.
[8] A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, N. Z.
0002, S. Anthony, H. Liu, and R. Murthy, “Hive - a petabyte
scale data warehouse using hadoop,” in ICDE, F. Li, M. M.
Moro, S. Ghandeharizadeh, J. R. Haritsa, G. Weikum, M. J.
Carey, F. Casati, E. Y. Chang, I. Manolescu, S. Mehrotra, U.
Dayal, and V. J. Tsotras, Eds. IEEE, 2010, pp. 996–1005.
[9] R. A. Brown, “Hadoop at home: large-scale computing at a
small college,” in Proceedings of the 40th ACM technical
symposium on Computer science education, ser. SIGCSE
’09. New York, NY, USA: ACM, 2009, pp. 106–110
[10] Hive http://hive.apache.org/
[11] A. Aji and F. Wang, “High performance spatial query
processing for large scale scientific data,” in
SIGMOD/PODS PhD Symposium, X. L. Dong and M. T.
Ozsu,¨ Eds. ACM, 2012, pp. 9–14. [Online].
[12] D. Jiang, A. K. H. Tung, and G. Chen, “MAP-JOIN-
REDUCE: Toward scalable and efficient data analysis on
large clusters,” IEEE Trans. Knowl. Data Eng, vol. 23, no.
9, pp. 1299–1311, 2011.
[13] F. N. Afrati and J. D. Ullman, “Optimizing joins in a
mapreduce environment,” in EDBT, ser. ACM International
Conference Proceeding Series, I. Manolescu, S.
Spaccapietra, J. Teubner, M. Kitsuregawa, A. Leger,´ F.
Naumann, A. Ailamaki, and F. Ozcan,¨ Eds., vol. 426.
ACM, 2010, pp. 99–110.
[14] A. F. Gates, O. Natkovich, S. Chopra, P. Kamath, S. M.
Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U.
Srivastava, “Building a high-level dataflow system on top of
map-reduce: the pig experience,” Proc. VLDB Endow., vol.
2, no. 2, pp. 1414–1425, Aug. 2009.
[15] E. Begoli and J. Horey, “Design principles for effective
knowledge discovery from big data,” in Software
Architecture (WICSA) and European Conference on
Software Architecture (ECSA), 2012 Joint Working
IEEE/IFIP Conference on, aug. 2012, pp. 215 –218.
[16] S. Madden, “From databases to big data,” IEEE Internet
Computing, vol. 16, no. 3, pp. 4–6, 2012.
[17] V. R. Borkar, M. J. Carey, and C. Li, “Big data platforms:
What’s next?” XRDS, vol. 19, no. 1, pp. 44–49, Sep. 2012.
”””
International Conference on Electrical Engineering and Computer Science, 21st April-2013, Coimbatore, ISBN: 978-93-83060-02-3
44
[18] K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The
hadoop distributed file system,” in Proceedings of the 2010
IEEE 26th Symposium on Mass Storage Systems and
Technologies (MSST), ser. MSST ’10. Washington, DC,
USA: IEEE Computer Society, 2010, pp. 1–10.
[19] P. M. Chen, E. K. Lee, G. A. Gibson, R. H. Katz, and D. A.
Patterson, “Raid: high-performance, reliable secondary
storage,” ACM Comput. Surv., vol. 26, no. 2, pp. 145–185,
Jun. 1994.
[20] B. H. Bloom, “Space/time trade-offs in hash coding with
allowable errors,” Commun. ACM, vol. 13, no. 7, pp. 422–
426, Jul. 1970.
[21] L. Michael, W. Nejd, O. Papapetrou, and W. Siberski,
“Improving distributed join efficiency with extended bloom
filter operations,” in Advanced Information Networking and
Applications, 2007. AINA ’07. 21st International
Conference on, may 2007, pp. 187 –194.
[22] S. Blanas, J. M. Patel, V. Ercegovac, J. Rao, E. J. Shekita,
and Y. Tian, “A comparison of join algorithms for log
processing in mapreduce,” in SIGMOD Conference, A. K.
Elmagarmid and D. Agrawal, Eds. ACM, 2010, pp. 975–
986.
[23] T. White, Hadoop - The Definitive Guide: MapReduce for
the Cloud. O’Reilly, 2009.
[24] K. Lee, Y. Lee, H. Choi, Y. Chung, and B. Moon, “Parallel
data processing with mapreduce: a survey,” ACM SIGMOD
Record, vol. 40, no. 4, pp. 11–20, 2012.
[25] J. Venner, Pro Hadoop, 1st ed. Berkely, CA, USA:
Apress, 2009.
[26] J. Lin and C. Dyer, ser. Synthesis Lectures on Human
Language Technologies. Morgan and Claypool Publishers,
2010.
[27] D. J. DeWitt, R. H. Katz, F. Olken, L. D. Shapiro, M. R.
Stonebraker, and D. A. Wood, “Implementation techniques
for main memory database systems,” SIGMOD Rec., vol.
14, no. 2, pp. 1–8, Jun. 1984.
[28] T. Lee, K. Kim, and H.-J. Kim, “Join processing using
bloom filter in mapreduce,” in Proceedings of the 2012
ACM Research in Applied Computation Symposium, ser.
RACS ’12. New York, NY, USA: ACM, 2012, pp. 100–
105.
[29] H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker,
“Map-reducemerge: simplified relational data processing on
large clusters,” in Proceedings of the 2007 ACM SIGMOD
international conference on Management of data, ser.
SIGMOD ’07. New York, NY, USA: ACM, 2007, pp.
1029–1040.
[30] W. Hu, L. Ma, X. Liu, H. Qi, L. Zha, H. Liao, and Y. Zhang,
“A hybrid join algorithm on top of map reduce,” in
Semantics Knowledge and Grid (SKG), 2011 Seventh
International Conference on, oct. 2011, pp. 44 –50.
[31] X. Zhang, L. Chen, and M. Wang, “Efficient multi-way
thetajoin processing using mapreduce,” CoRR, vol.
abs/1208.0081, 2012.
[32] Available: http://pig.apache.org/
[33] Y. Kim and K. Shim, “Parallel top-k similarity join
algorithms using mapreduce,” in Data Engineering (ICDE),
2012 IEEE 28th International Conference on, april 2012, pp.
510 –521.
[34] F. Atta, S. Viglas, and S. Niazi, “Sand join - a skew
handling join algorithm for google’smapreduce framework,”
in Multitopic Conference (INMIC), 2011 IEEE 14th
International, dec. 2011, pp. 170 –175.