Big Data or Pig Data? Is Big Data worth the Hype?

Off late, in every conversation on new technology trends, Big Data invariably appears as a ubiquitous
entry. Big Data has been claimed to be the ultimate tool which will reveal the answer to the analytical
challenges and help in revealing the hidden needle in the mythical haystack.

Looking at all these overtly large claims about Big Data and its alleged ability to change the way
companies carry out analytical decision making, Cynics tend to pejoratively mock Big Data as PIG Data.
These cynics are those that question the implications of the approach itself, and those that question the
way it is currently done.

This brings us to ask this question: Is Big Data worth the Hype?

Answer to the question above lies in understanding a few things as ahead:

Well, the data has been there always, what is so special about Data & it being Big now?

Lets deliberate and try to arrive at a sane response to understand: How big really the phenomenon is
called Big Data?

Before we answer the first question, it is better to understand a response to the second question as it
will help us to understand the very need to measure and deploy big data.

Business have been there since centuries and the collection, dissection and analysis of data by
businesses to arrive at saner decisions are equally old phenomena running into centuries. What has
changed off late through mechanization and technology advancements is the ability to capture a vast set
of data.

Confused? Lets take a real life example: Within the four walls of a production facility, in a highly
mechanized shop floor, there are hundreds of sensors, actuators and field devices collecting all kinds of
data related to temperature, pressure, product quality, measurements and plethora of other production
parameters. Lets move out of factory premises, due to social media, mobility advancements, now
business have ability to capture on 24*7 basis, various parameters related to positioning of products in
the supply chain network, the factors that were behind a customer purchase decision.

This has given business the ability to have access to huge mountains of Data pile ups akin to those
garbage mountains risen out of landfills in the periphery of a metro city. The data that is being captured
today is profoundly BIG and insanely varied. As they say, special situation demands special response.
Hence, the driver to have a comprehensive solution about Big Data. Big data "size" is a constantly
moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data today. Hence,
there is a need to uncover large hidden values from large datasets that are diverse, complex, and of a
massive scale.
Big Data is characterized by Volume, Variety, Velocity, Variability, Veracity and above all Complexity. In
fact Complexity is nothing but a summation of all the characteristics starting with V and mentioned in
preceding sentence. Given all these, Big Data needs a robust solution to take it head on.

There are many tools and solutions available in the market today to process big data. A
2011 McKinsey report[37] suggests suitable technologies include A/B testing, crowdsourcing, data
fusion and integration, genetic algorithms, machine learning, natural language processing, signal
processing, simulation, time series analysis and visualization. Multidimensional big data can also be
represented as tensors, which can be more efficiently handled by tensor-based computation, such
as multi-linear subspace learning. Additional technologies being applied to big data include massively
parallel-processing (MPP) databases, search-based applications, data mining, distributed file systems,
distributed databases, cloud based infrastructure (applications, storage and computing resources) and
the Internet.

Ok, We can capture large volumes of data now, still why do we need a Big Data solution?

Big Data is worth more than the hype that it has generated. Analysis of Big data may throw up
interesting trends and conclusions that can change the way decisions are taken. Extending the example
of shop floor given above, a robust big data analytics solution may through up e.g. that the packing done
at a given temperature point could be a cause