2015 IEEE International Congress on Big Data
Big Data Open Source Platforms
Pedro Daniel Coimbra de Almeida
Department of Computer Engineering and Systems
ISEC – Coimbra Institute of Engineering
Coimbra, Portugal
a21180299@isec.pt
Abstract— In a global market the capacity to mine and
analyze user data is one way for companies to be as close in
time and accuracy to the needs of their users. Big Data
Platforms are one solution for companies to solve the
necessary challenges to accomplish these capacities.
Unfortunately the number of challenges that need to be
addressed, allied with the high number of different solutions
proposed, has led to the creation of a high number of
different platforms making it hard to name one definitive
and adequate platform for companies. In this paper we
compare six of the most important Big Data Open Source
Platforms to help companies or organizations choose the
most adequate one to their needs. We analyze the following
open source platforms - Apache Mahout, MOA, R Project,
Vowpal Wabbit, PEGASUS and GraphLab CreateTM.
Keywords- Big Data, Open Source, Data Mining, Data
Analysis
I. INTRODUCTION
Big Data Platforms are the combination of hardware
infrastructures and software tools developed to acquire,
store and analyze data in a timely manner. They allow
individuals and enterprises to extract insight and value
from data generated both inside and outside their business
or area of interest. In business environments these
platforms help managers have a better overview over their
business and lead to potential enhanced productivity,
greater innovation and stronger position in the market.
The potential value and knowledge inside data is huge
and of interest to all enterprises. Unfortunately very few
managers are educated to acknowledge such importance
and even less enterprises have the necessary budgets and
resources to tap into that source of power. In 2014 the
OECD (Organization for Economic Co-operation and
Development) registered that 95% of enterprises are
Small and Medium Enterprises that do not have resources
to spend studying the potential of data [1]. These
enterprises need to be educated about this potential and
this has to be done by showing practical solutions that
allow them to see results and value in the moment. To
complicate this problem the offer of systems that work
with Big Data is too vast. This happens because the
process of working with Big Data or data in general is not
a straightforward one step thing. It is a complex never
978-1-4673-7278-7/15 $31.00 © 2015 IEEE
DOI 10.1109/BigDataCongress.2015.45
Jorge Bernardino
Polytechnic of Coimbra
ISEC-CISUC
Coimbra, Portugal
jorge@isec.pt
ending process, where some steps can require repetition
until results are satisfactory, that goes from the point
where data is generated until its analysis. To add
complexity to the question we have that each developed
platform uses different approaches to tackle both
conceptual and technical challenges associated with each
phase of the process. And we shall not forget that because
this field of research is constantly evolving, solutions
deployed today may not be useful in a near future. What
is considered Big Data today may not be appropriately
Big tomorrow. Companies wanting to optimize their
resources cannot afford to waste time exploring all the
available platforms and need help and education in
finding the most suitable solution for their specific set of
needs. For most it is not even a matter of wanting or not
wanting but a matter of absolute impossibility due to lack
of resources and know-how. It has to be understood that is
impossible to name one definitive platform that can be
named as the best one and most suited for the needs of
everyone. One size fits all is not possible in this area. A
great deal of research has already been done on the field
of Big Data and Big Data Platforms [2, 3, 4, 5]. This
research can be essentially done in two ways. The first
one is to study the needs of one organization and create a
set of tools that answer the needs of that organization, and
that one only. This approach requires a large amount of
investment and time that only big enterprises are able to
afford. The second way works by studying the existing
platforms and choosing the one that is closer in
characteristics and capabilities to the requirements of each
enterprise. This approach is preferred for Small and
Medium Enterprises because it requires no further
investment or development. Proprietary solutions can
require a certain amount of investment but nowadays
there are a lot of open source solutions that are not only
free but more flexible to be adapted. In this paper we
study and compare Big Data Open Source Platforms
technical characteristics but also and most importantly
their capabilities for insertion in the segment of Small and
Medium Enterprises. We compare this capability through
analysis of parameters such as the size of hardware
infrastructure necessary for deployment, ease of use of the
interfaces and availability of programmers for the
language required to manage and use the platform. To
268
keep the focus of this work on platforms that mine and
analyze data we left out of the study the distributed
computing platforms over which Big Data Platforms are
built, such as Hadoop [6], Spark [7] and Storm [8].
The remainder of this paper is structured as follows.
Section 2 explains the Big Data paradigm/model. Section
3 overviews the features and requirements involved into
Big Data Open Source Platforms. Section 4 describes the
six platforms. Section 5 compares the tools and
establishes connections between their features and the
necessities of companies. Finally section 6 presents
concluding remarks and future work.
II. THE BIG DATA MODEL EXPLAINED
There is not one final definition about what exactly is Big
Data. In a very simple way Big Data can be defined just as
a volume of data that is bigger than the traditional volumes
[9]. Still this definition is more relative than actually
concrete while being very reductive at the same time. It
does not work because the volumes of data are getting
bigger day by day, making this definition constantly
outdated. This is why newer definitions consider Big Data
as the combination of two things – large volumes of data,
no matter what real size they are as long as they are big
enough that they can’t be managed with traditional
architectures and the combination of techniques and
technologies used to extract value from such information
[10]. Volume and value are only two of the dimensions
that need to be considered when treating Big Data. The
most recent definitions have evolved from the 3V’s
concept that considered only Volume, Velocity and
Variety [11] to a more complex 6V’s concept that adds
Value, Variability and Veracity to the list of characteristics
inherent to the model as can be seen in Figure 1:
• Volume represents a real quantity of generated
data that can be measured.
• Velocity is a combination of both the speed at
which data is generated and the speed at
which it is processed.
• Variety consists of the several forms but also
dimensions in which data can be represented
– the structured/unstructured division is the
most common definition.
• Value is the knowledge possible of being
extracted from data analysis and varies
depending on the needs of who is analyzing it.
• Variability is the inconsistency that can be
presented in data because of the high number
of distributed autonomous sources of data that
have no centralized control whatsoever.
• Veracity is what evaluates the quality of the
data as in a metric as to how much that data
can be trusted to provide acceptable
information for the goals of the analyzers.
These characteristics have to be ever present in the process
of developing technologies and platforms whether they act
upon one or more steps of the Big Data chain of value [12]
as seen in Figure 2. This chain of value is what divides the
269
Figure 1: The 6V of Big Data
different phases that data goes through from its generation
to its eventual death. Data Generation is the first step,
followed by Data Acquisition, Data Storage and finally
Data Analysis steps. For any individual or group user
wanting to put a Big Data Platform to work it is important
to note that before the platform itself is chosen and
deployed it is advisable to study and implement a Big Data
strategy that helps managers understand exactly what they
want from the eventual platform they are going to use [13].
A Big Data strategy consists mainly of three different
areas:
• Big Data Basics that usually represent the
acknowledgment of the diverse types of data
available such as social data, preprocessed data or
unstructured data.
• Big Data Assessment, an area that evaluates the
several data aspects such as the source, the
potential uses, volumes, estimated future growth
and privacy regulations.
• Big Data Strategy in itself that studies the
impacts of Big Data in the organization,
opportunities to be taken and business cases
where Big Data can be of use. It also studies
economic impacts such as the potential return of
investment.
Only after that strategy is documented and thoroughly
analyzed are organizations and managers able to effective
chose what functionalities they will have to look for in a
Big Data Platform.
III. OPEN SOURCE BIG DATA PLATFORMS OVERVIEW
Any platform capable of supporting the large kind of
datasets that are not manageable by traditional database
tools can be considered a Big Data Platform [14].
Figure 2: Big Data chain of value
A generic goal of such platforms can be formulated as to
grant the abilities to integrate privately acquired and
publicly available Big Data with data generated within an
enterprise and to analyze the combined set for value
extraction [15]. In further detail Big Data Platforms
should also possess a group of additional features:
• Being comprehensive and ready for enterprise
use.
• Being easily scalable and extensible.
• Provide capabilities for update of data with low
latency.
• Be robust and capable of supporting fault-
tolerance.
• Being corporative or open source depending on
the interest of the development and investment
teams.
• Require as little maintenance as possible.
But Big Data Platforms aren’t just features, they are also
technologies. It doesn’t have to be made of the newest
and most powerful technology. Sometimes the only thing
it takes is a new configuration or approach to an old one.
It all depends on the needs of the organization or person
wanting to use the platform. Unfortunately this solution is
not widely used because modifying decades of old
systems for the needs of the present come at a very high
cost that very few are able to afford. This is why new
platforms are arising to answer new requirements. Big
Data Platforms requirements can be divided into three
main phases – Data Acquisition, Data Organization and
Data Analysis as seen in Figure 3. In the Data Acquisition
phase systems are being called to provide lower latency
on data capture, shorter execution on data queries,
capacity for distributed environments and support for
most varied and flexible types of data structures. NoSQL
databases are currently the solution that is closer to such
specifications because they prioritize data capture over
data categorization thus eliminating the overhead caused
by the existence of a data scheme [16, 17]. When it comes
to Data Organization or data integration the preferred
solution of organizing all the data at one single location is
not possible anymore in the Big Data era. Moving around
significant amounts of data while keeping its integrity is
mandatory for big companies. In this area the main
requirements for the systems are the capacity for scaling
both vertical and horizontally,
Figure 3: The three phases of work of Big Data Platforms
270
allow distributed programming, provide high throughput
of data in between the entire infrastructure and provide
support for both structured and unstructured types of data.
Hadoop [6] and its Hadoop Distributed File System
(HDFS) are currently the most used solutions for this
phase of data treatment. But as the paradigm is shifting
from batch processing to real time stream processing,
Hadoop is being abandoned in favor of more suitable
platforms such as Spark, Storm or S4 [18]. Data Analysis
just like data integration, demands the use of distributed
environments to realize its tasks of deep analytics and
statistics on a broad variety of data types. Platforms must
comply with facts such as data being stored in different
systems and scaling up in terms of volume. They need to
be faster at delivering answers and reacting to changes in
data behavior. In some scenarios where data moving from
one place to the other raises privacy concerns but also
prohibitive costs for organizations to support, data mining
and analysis tasks performed by Big Data Platforms are
encouraged to be carried out in multiple locations with
intermediate results being then sent to a central location
that groups them back for another process of final
analysis. Still this brings even more complexity especially
for development at algorithm level because intermediate
results are not as precise as raw data, since noise and
arbitrariness can be introduced for privacy maintenance,
and reading and integrating them can cause both losses of
important data but also incorrect interpretation [19].
IV. OPEN SOURCE BIG DATA PLATFORMS
Based on the work of [20] we choose the following Big
Data Open Source Platforms: Apache Mahout, MOA, R
Project, Vowpal Wabbit, PEGASUS and GraphLab
CreateTM. In this section we describe these tools in some
detail.
A. Apache Mahout
Apache Mahout [21] is an open source project aiming to
build a comprehensive library of machine learning and
data mining algorithms. The main feature of Mahout is
that all the algorithms in its library are highly scalable and
able to perform well on both standalone machines and
distributed environments. It runs on top of the Hadoop
environment and make use of such technologies as HDFS
and MapReduce. Unlike other platforms that fail to do so,
Mahout provides elevated strength and stability in the
management of large data sets. Because most machine
learning algorithms are iterative and require multiple
loading steps from disk which causes a lot of processing
overhead, the Mahout developers are being encouraged to
abandon Hadoop in favor of Spark because Spark not
only maintains the distribution of the file system but also
provides more modern and powerful uses of parallel-
processing systems [22].
Mahout currently has implementations and support for
most of the machine learning tasks of supervised and
unsupervised nature such as Recommendation Mining,
Clustering and Classification. Recommendation Mining is
the algorithm that mines and analyzes user behavior and
builds recommendations on similar items the user might
like. Clustering is characterized as the set of algorithms
aimed at analyzing text documents and group them into
topic related groups of texts. Classification comprises the
set of algorithms that use previously obtained information
from already categorized documents to assign new ones
into the most suitable category. Other tasks available are
Collaborative Filtering, Dimension Reduction, Topic
Models and Frequent Pattern Mining [21].
It is important to refer that Mahout is only a library. It
does not have its own server or user interface thus
requiring the use of an external programming IDE with
support for Java to be used for development and testing. It
has gained a lot of popularity among Data Mining
developers because of the freedom of implementation. It
still has one critical issue as it is constantly under
development and not very well documented.
It is licensed under the Apache License 2.0. It needs Java
1.6 or greater and Maven 3.0 or greater to be installed on
the machine(s) that compose the system. The system
requirements are dependent on the particular algorithm to
be run. The latest stable version is 0.9 [21].
B. Massive Online Analysis (MOA)
Massive Online Analysis [23] is an open source software
oriented toward mining of data streams that present
conceptual drift. It allows both building and
experimentation on machine learning and data mining
algorithms to provide fast answers to the evolution of the
nature of the data streams. It is a very user-friendly
platform – having its own Graphical User Interface (GUI)
as seen in Figure 4 – command line and also making use of
the Java API, which makes it suitable for users with
distinct levels of experience. One of most recognized
abilities is its modularity. Besides the core program one
user can choose to expand only to modules of interest thus
saving time and effort in exploring features not relevant to
his/her work.
The main goal of MOA is to provide a framework for
benchmarking of existing machine learning algorithms that
operate on real-time big data streams.
This allows the community to early delete less efficient
algorithms while placing more investment in the
development of newer solutions. Unlike the WEKA [24]
platform from where MOA has derived it does not work
with batch-processing. To perform such tasks it provides a
set of essential tools such as real and synthetic examples of
data streams for testing, a library of existing algorithms
and measures of comparison between them. It does not
only allow working with content given in the platform but
it provides framework for the user to insert new types of
streams, algorithms and methods. It also permits the
storage of previously run benchmark results thus providing
the creation of scenarios to be used against the newer
algorithms [25]. The current version of MOA provides
271
Figure 4: MOA Graphical User Interface (source:
www.moa.cms.waikato.ac.nz/getting-started/)
collections able to perform such tasks as Classification,
Regression, Clustering, Outlier Detection, Recommender
Systems, Frequent Pattern Mining and Change Detection.
MOA is a framework that works mostly on algorithms that
work on single node machines and studies the capabilities
of algorithms to scale up rather than scale out. It has three
simulated environments each defined accordingly to
different memory requirements – Sensor Network,
Handheld Computer and Server – that use 100Kb, 32Mb,
and 400Mb of memory respectively. Custom environments
can be created with their own specifications although the
400Mb imposed by the Server environment has proven to
be enough for most of the algorithms run on MOA.
It is freely licensed under the GNU GPL license and is
written in Java. The latest stable version is 14.11 [23].
C. R Project
R Project [26] is the combination of a programming
language and an environment for statistical and graphics
computing. It has been designed with influence taken
from the programming languages S and Scheme but
unlike these two it is completely open-source [27].
So far R Project provides a variety of graphical
techniques and statistical ones that include linear and non-
linear modelling, classic statistic tests, time-series
analysis or more traditional features like classification and
clustering. It intends to be a fully planned ahead system
built with coherency rather than a basic suite where
different tools are simply added for extension of
functionalities [28].
Among its strengths are the high extensibility through the
use of packages for addition of new models, easy
integration with code made in C, C++ or Fortran for the
execution of intensive tasks and the ability to produce
robust well designed statistic plots for publication.
Because R is in itself a programming language it allows
users who feel comfortable with coding to add new
functionalities to the suite.
It is an integrated suite of software that allows performing
a full circle of data treatment including manipulation,
calculation and finally display. Besides including
effective methods for data storing and handling it includes
a group of operators to provide calculation with arrays
and matrices. Other features include an integrated
collection of tools for intermediate data analysis and
graphical facilities to facilitate visualization of such
analysis. The R programming language also provides a
simple way to develop functions with inclusion of
conditionals, loops, input and output facilities that as we
mentioned before can be integrated with functions written
in other languages that perform better at computationally
intensive tasks.
R Project has so much popularity among the community
working with statistical data analysis that it has led to the
creation of several tools to make it more user-friendly and
appealing. One of them is RStudio [29]. RStudio is an
IDE specifically aimed at working with R and developed
by the company of the same name. An example of the
RStudio interface is shown in Figure 5.
It is licensed under the terms of the GNU General Public
License and can be run in a variety of UNIX distributions
but also Windows and MacOS. The latest stable version is
3.2 [26].
D. Vowpal Wabbit
Vowpal Wabbit [30] is a project sponsored by Microsoft
Research with the goal of developing a single machine
learning algorithm that is inherently fast, able of being run
in both standalone machines and in parallel processing
environments and capable of handling datasets in the
scale of terabytes.
The creators and developers of Wabbit have decided from
the very beginning focus on building a single strong
multi-purpose algorithm rather than a library with many
algorithms and also to start at a high level of development
rather than wasting investment in making bottom up
development from slower and less efficient algorithms.
The development of the algorithm is encouraged to take
advantage of four main features that when used in
combination can achieve better results – input formats of
data, speed of learning, scalability of the data sets
analyzed and feature pairing. Currents efforts are being
put together not just to improve the algorithm in itself but
also to improve some more basic features such as the
speed of the input/output operations.
Wabbit presents a group of features that are so far unique
to its system. It is optimized by default to support online
learning rather than batch learning through a series of
modifications made to the stochastic gradient descent
methods that allow a more robust analysis on data sets of
considerable size [30]. It presents Feature Hashing, a
method that reduces the necessary pre-processing of data
making the analysis
272
Figure 5: Interface of R Studio (source:
www.rprogramming.net/download-and-install-rstudio/)
process not only faster but more accurate. The
combination of the aforementioned two features allows
Wabbit to make effective learning from any size of
information made available to the algorithm no matter
how small or big it is. The implementation of a reduction
stack within its core also allows it to provide a solution
for large scale advanced problems.. Like we mentioned
before, Wabbit runs mainly as a library or a standalone
daemon service but it is fully ready to be deployed in
cloud environments.
Wabbit is licensed under BSD and the latest stable
version is 7.10 [30].
E. PEGASUS
PEGASUS [31] is an open source platform developed by
the data mining group at the Carnegie Mellon University
and designed specifically for data mining in graph
structures of sizes ranging from a few gigabytes to
petabytes (see logo in Figure 6). Because datasets of such
size are no longer able of being processed in single node
machines, PEGASUS works with resort to parallel
programming by being implemented on top of Hadoop.
It is a more specific data mining platform than others
because it works with data that comes in form or graphs
and networks with billions of nodes and connections. This
is a considerable leap against previous systems that can
only work in the size of the millions. Graph structures are
suffering a rise in number and importance coming from
such various fields as mobile networks, social networks or
medical fields such as protein regulation [32].
PEGASUS unifies a vast number of graph mining tasks
such as the computing of the graph diameter, computing
the radius of each node or finding connections between
the graph nodes by making a generalization of matrix-
vector multiplication called GIM-V [32].

Figure 6: Project PEGASUS logo (source:
www.cs.cmu.edu/~pegasus/)
GIM-V is carefully implemented and optimized with
built-in graph mining operations such as Page Rank,
Random Walk with Restart and diameter estimation [32].
It provides linear scaling as to the number of edges to
analyze making it suitable to work on any number of
machines available.
The system is not fully developed yet and current efforts
are being put in place to expand the library to newer data
mining and machine learning algorithms but also to
provide more efficient methods for graph indexing.
Written in Java it runs on any operative system able to run
Hadoop with preference for UNIX machines. It is quite
heavy in terms of software requirements as it calls for not
only Hadoop, but also Apache Ant, Java, Python and
Gnuplot. It is licensed under the Apache License version
2.0 and the latest stable version is 2.0 [32].
F. GraphLab CreateTM
GraphLab CreateTM [33] formerly known only has
GraphLab before Dato Incorporation acquired it. It is a
platform for development of machine learning
applications working at various scales of dataset sizes. It
aims at providing means for applications to have all the
necessary iterative steps to make predictions based in data
mining results.
The main goal is to design and implement machine
learning algorithms that are efficient, accurate and able to
keep data consistency while taking the most advantage
from parallel processing [34]. Development focuses on
approaching existing common patterns in machine
learning and putting efforts on those, while ignoring more
exotic approaches that provide less utility. Some of the
already developed algorithms are belief propagation,
Gibbs sampling or Co-EM. All of these algorithms have
been optimized from their previous versions for parallel
processing. The platform works mainly with three areas
of data processing – Data Engineering, Data Intelligence
and Data Deployment. One of its strengths is the ease of
use for both beginner and experts in the area of data
science. Its main components are scalable data structures,
machine learning modules, methods for data visualization
and capacity to easily integrate with many data sources of
different types. In the field of data engineering it provides
an easy way to run the ETL (Extract, Transform and
Load) process on data as a means to clean data and save
time during the analysis process. For this it provides tools
273
to perform basic tasks such as data sort, group or dice in a
fast way on datasets with terabytes of size. It also
provides intuitive visualization of the generated tables and
graphs through the auxiliary tool GraphLab CanvasTM
whose example of interface can be seen in Figure 7. For
data intelligence there are tools to build recommenders
with Python code, make data analysis in images through
deep learning techniques that take advantage of powerful
computing GPU capabilities. There are also tools for
analysis of unstructured text, graph analysis and
supervised learning for outcome prediction. For data
deployment it provides tools to deploy easily coded
services for prediction of various types. Written in C++
with resource to a Python interface, it runs on all the main
operative systems such as MacOSX, Windows Virtual
Machines and UNIX distributions such as Ubuntu, Debian
or Rhel. It is licensed under the Apache License 2.0 and
the latest stable version is 1.3 [33].
V. OPEN SOURCE BIG DATA PLATFORMS
COMPARISON
To perform our comparison on the previously detailed
platforms we chose six parameters that we consider
relevant for business managers to analyze when choosing
an open source Big Data Platform. These parameters are a
mix of technical characteristics but also other defining
characteristics to follow when finding out if a platform is
or not suitable for Small and Medium Enterprise
environments. We start with the programming paradigm -
this is directly related to the size of the company and size
of the computer infrastructure where the system may
eventually be installed. Same applies for another set of
characteristics we are analyzing - the supported sizes of
datasets. Next we analyze the required programming
languages because we consider important for a manager
to know what kind of programmers s/he will need to run
the systems. To this end we also provide comparison on
the user interface each platform provides as we consider
the level of experience of the programmers and ease of
use and adoption of the platform can be related to the kind
of less or more user friendly interfaces available. We also
compare the types of data supported as a means to
categorize platforms as more generic or more specific.
Last but not the least we compare the variety of
algorithms available as a way to say which platforms are
more complete than the others. In Table 1 we show the
detailed listing of these characteristics for each platform.
Analyzing all the characteristics present in Table 1 we can
draw the following conclusions – both MOA and R
Project are the best tools for Serial Computing running in
a single node machine. The others are best for companies
with larger infrastructures that that allow parallel
computing.
Most of the platforms, except for R Project, make use of
common languages that have no lack of programmers
available.

Figure 7: GraphLab CanvasTm Interface (source:
www.blog.dato.com/)
MOA, R Project and GraphLab can be considered the
most user friendly as they provide Graphical User
Interfaces where the remaining ones do not, which makes
working on them considerably harder and with a bigger
learning curve, at least for less experienced users.
PEGASUS and GraphLab are graph oriented where others
platforms are made to work with all generic types of
datasets. Working with graphs only is not a limitation in
itself as this type of data structure is very common these
days. It is up to the companies to make this evaluation
accordingly to their own needs. Lastly, when comparing
available algorithms it is complicated to make clear
conclusions about the platforms. Mahout and R Project
are the ones with more participation from the community
and therefore the ones with more diverse algorithms
provided. GraphLab, PEGASUS and MOA also have a
significant number of algorithms ready to be used. Here is
Table 1. Open Source Big Data Platforms Comparison
where Wabbit is the weakest because as we already
explained it only works with a single algorithm that can
be powerful under specific conditions but useless for most
cases. Mahout, PEGASUS and GraphLab are the ones
who support bigger data sets while MOA is the weaker
one on this field supporting only datasets with a size of a
few megabytes.
In summary, MOA is definitely the best platform for
small companies with insignificant computer
infrastructures and that work with lesser amounts of data.
For larger companies with bigger infrastructures that
require parallel computing it is hard to decide between
Mahout and GraphLab. While the first has the advantage
of being broader in terms of data types supported, the
second provides more ease of use due to the existence of a
GUI that helps track the work being done.
VI. CONCLUSIONS AND FUTURE WORK
With the rise of Big Data and with more individuals and
organizations gaining awareness of the potential and
opportunities it brings, the number of Big Data Platforms
of both open source and proprietary nature is increasing
exponentially over the next years. Implementing a
platform that solves all the questions inherent to Big Data
is too heavy, expensive if not impossible at all. So the
path being more commonly followed is for each
organization to invest in developing its own platform that
is close to its specific needs, inwards or by calling to the
community. This creates a high number of available

Mahout MOA Wabbit R Project PEGASUS GraphLab

Programming Parallel Serial Parallel Serial Parallel Parallel

Paradigm Computing Computing Computing Computing Computing Computing

Programming R, S, C, C++,
Java Java C, C++ Java C++, Python
Language(s) Fortran

GUI, N/A (but GUI through

User Interface N/A command N/A RStudio GUI N/A GraphLab
line, Java API exists) CanvasTM

Data Types All All All All Graphs Graphs

Belief
Recommendation
Classification Page Rank, propagation,
Mining,
Available Regression, Own Single Random Walk Gibbs
Classification, Undefined
Algorithms Clustering Algorithm With Restart sampling or
Clustering and
and others and others Co-EM and
others
others

Scale of
Few Up to Up to Up to
Supported Up to Petabytes Up to Petabytes
Megabytes Terabytes Gigabytes Petabytes
Datasets

274
solutions which not only bring difficulty the choice for
people wanting to select a platform for their enterprise but
also create a high level of redundancy in solutions
presented. Educating not only business managers but also
people in general to the potential of Big Data is crucial to
guarantee the future of research and platform deployment.
In this paper we study and analyze six Big Data Open
Source Platforms and conclude that the best platform for
small companies with trivial computer infrastructures is
MOA, while Mahout and GraphLab are the best for
companies with larger computer infrastructures that
require the use of parallel computing. Mahout has more
types of data supported but GraphLab is more user
friendly because it has a GUI and Mahout does not. As
future work we pretend to analyze these six tools in real
environments and explore other available Big Data tools.
ACKNOWLEDGMENTS
This work was partially financed by iCIS – Intelligent
Computing in the Internet Services (CENTRO-07- ST24 –
FEDER – 002003), Portugal.
This work was also made possible with the help of
ISEC-Polytechnic of Coimbra, which provided some of
the facilities and infrastructures.
REFERENCES
[1] OECD (2014), "Small and medium-sized enterprises", in
OECD, OECD Factbook 2014: Economic, Environmental
and Social Statistics, OECD Publishing, Paris, doi:
10.1787/factbook-2014-en.
[2] V. Mayer-Schnberger, K. Cukier “Big Data: A Revolution
That Will Transform How We Live, Work and Think”,
John Murray Publishers, United Kingdom 2013,
ISBN:1848547927 9781848547926.
[3] O’Reilly Media “Big Data Now: 2012 Edition”, O’Reilly
Media Inc, October 2012, ASIN: B0097E4EBQ.
[4] Melyssa Barata, Jorge Bernardino, Pedro Furtado, “YCSB
and TPC-H: Big Data and Decision Support Benchmarks”.
BigData Congress 2014, pp. 800-801.
[5] Melyssa Barata, Jorge Bernardino, Pedro Furtado, “Survey
on Big Data and Decision Support Benchmarks”. Database
and Expert Systems Applications - 25th International
Conference, DEXA 2014, pp.174-182.
[6] Hadoop, https://hadoop.apache.org/.
[7] Spark, https://spark.apache.org/.
[8] Storm, https://storm.apache.org/.
[9] C. Snijders, U. Matzat, U. Reips “”Big Data”: Big Gaps of
Knowledge in the Field of Internet Science” International
Journal of Internet Science, vol 7, 2012, pp. 1-5.
[10] I. Hashem et al, “The rise of “big data” on cloud
computing: Review and open research issues” Information
Systems, vol 47, January 2015, pp. 98-115,
doi:10.1016/j.is.2014.07.006.
[11] D. Laney “3D Data Management: Controlling Data
Volume, Velocity and Variety” Gartner, File 494, February
2012.
[12] H. Hu, Y. Wen, T. Chua, X. Li “Toward Scalable Systems
for Big Data Analytics: A Technology Tutorial” IEEE
Access, volume 2, June 2014, pp. 652-687,
doi:10.1109/ACCESS.2014.2332453.
275
[13] M. Huddar, M. Ramannavar “A Survey on Big Data
Analytical Tools” International Journal of Latest Trends in
Engineering and Technology (IJLTET), 2013, pp 85-91.
[14] R. Gupta, S. Gupta, A. Singhal “Big Data : Overview”
International Journal of Computer Trends and Technology
(IJCTT), volume 9 number 5, March 2014, pp. 266-268,
ISSN: 2231-2803.
[15] J. Dijcks “Big Data for the Enterprise” An Oracle White
Paper, September 2014.
[16] V. Abramova, J. Bernardino. “NoSQL databases:
MongoDB vs Cassandra”. Sixth International C*
Conference on Computer Science & Software Engineering
C3S2E – 2013, pp. 14-22.
[17] J. Lourenço, V. Abramova, M. Vieira, B. Cabral, J.
Bernardino. Nosql Databases: A Software Engineering
Perspective. WorldCIST'15 - 3rd World Conference on
Information Systems and Technologies.
[18] Apache S4, http://incubator.apache.org/s4/.
[19] X. Wu, X. Zhu, G. Wu, W. Ding “Data Mining with Big Data”
IEEE Transactions on Knowledge & Data Engineering, vol
26, January 2014, pp 97-107,
[20] A. Bifet “Mining Big Data in Real Time” Informatica, vol
37, 2013, pp 15-20.
[21] Apache Mahout, http://mahout.apache.org.
[22] A. Fernández et al. “Big Data with Cloud Computing: an
insight on the computing environment, MapReduce, and
programming frameworks” WIREs Data Mining Knowl
Discov 2014, 4:380–409. doi: 10.1002/widm.1134.
[23] MOA, http://moa.cs.waikato.ac.nz.
[24] WEKA, http://www.cs.waikato.ac.nz/ml/weka/.
[25] A. Bifet et al. “MOA: Massive Online Analysis, a
Framework for Stream Classification and Clustering.”
Journal of Machine Learning Research (JMLR) Workshop
and Conference Proceedings.
Volume 11: Workshop on Applications of Pattern Analysis
(2010).
[26] R, http://www.r-project.org.
[27] K. Hornik “The R FAQ”, http://cran.r-
project.org/doc/FAQ/R-FAQ, 2015.
[28] F. Morandat, B. Hill, L. Osvald, J. Vitek “Evaluating the
design of the R language: objects and functions for data
analysis”, ECOOP'12 Proceedings of the 26th European
conference on Object-Oriented Programming, pp 104-131,
doi:10.1007/978-3-642-31057-7_6.
[29] RStudio, http://www.rstudio.com/products/rstudio.
[30] Vowpal Wabbit,
https://github.com/JohnLangford/vowpal_wabbit/wiki.
[31] PEGASUS, http://www.cs.cmu.edu/~pegasus.
[32] U. Kang, C. Tsourakakis, C. Faloutsos “PEGASUS: A
Peta-Scale Graph Mining System - Implementation and
Observations” ICDM '09. Ninth IEEE International
Conference on Data Mining, 2009, pp 229-238,
doi:10.1109/ICDM.2009.14.
[33] GraphLab CreateTM, https://dato.com/products/create.
[34] Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, J.
Hellerstein “GraphLab: A New Framework For Parallel
Machine Learning” Proceedings of the Twenty-Sixth
Conference on Uncertainty in Artificial Intelligence, 2014,
arXiv:1408.2041.