Professional Documents
Culture Documents
CTG Big Data Review Cloudera Intel
CTG Big Data Review Cloudera Intel
Sponsored by
Contents
Executive summary p3
Research overview p4
Big Data goes mainstream p5
This document is property of Incisive Media Services Ltd. Reproduction and distribution of this
publication in any form without prior written permission is forbidden.
2 Research overview
The key objective of the Computing Big Data Review 2015 was to compare
research with that undertaken 12 months ago and published in the
Computing Big Data Review 2014, in order to establish the degree to which
the Big Data market has matured along with the causes and consequences.
This year, 18 per cent of respondents have already implemented Big Data
solutions at the operational level. When asked what technologies they
were using or considering and presented with a list of likely options, the
proportion opting for none of the above plummeted from 33 per cent in
2014 to 16 per cent this year.
2014
Next-generation data warehouses/ 24%
analytics databases 13%
Cloud-based Big Data services 26%
13%
Data integration and data quality 24%
platforms and tools 22%
Application development platforms 17%
applied to Big Data 11%
Analytic and transactional 15%
applications and services, as applied 13%
to Big Data use cases
NoSQL databases
25%
15%
Advanced analytics and data science 15%
platforms as applied to Big Data 13%
Hadoop software 21%
8%
In-memory databases as applied to 23%
Big Data 11%
Non-Hadoop Big Data platforms
12%
6%
None of these 33%
0% 10% 20% 30% 40% 50%
Considering or using Have skills in-house
only large organisations can implement. The cloud has ensured that even the
smallest of businesses can utilise Big Data. have become
much more
Cloud computing You dont have to have the same
potentially makes Big number of staff around tooling attuned to the
Data refining and analysis so businesses can take more advantage
much easier even for of services. The service-based nature problem they
modest and small firms of everything is totally changing our are trying to
IT manager, Research consumption of IT CIO, Services
solve and how
There have been other factors too. As increasing numbers of case studies that data might
become available the understanding of the benefits of Big Data has increased.
This process of education has been pushed along by the vendors themselves help them
who have become much better at explaining benefits for businesses. solve it
Vendors have also addressed issues of usability, smoothing off some of the
rough edges and offering training to users as part of enterprise subscription
packages. However, much of the increased awareness and takeup of of
Big Data has been driven by factors unique to individual industries, as the
following words from our interviewees demonstrate.
Computing asked: What changes have you noticed over the last 12 months
with respect to Big Data that are most likely to impact on your business or on
the market in general?
The most frequently chosen answer was Moving from analysing historical
data to real-time analytics. The commercial benefits of real-time data
analytics for businesses include better understanding of their customers and a
chance to proactively improve services. Two of our interviewees shared how
they were using this technology.
reiterated the message that a multi-departmental approach was necessary as
projects were initiated.
The final two participants highlight the fact that while IT teams may be
driving Big Data projects, the long-term goal of these projects is to make data
analytics widely accessible for the rest of the business.
a first glance would suggest. Some of our interviewees explain why this
may be the case.
These responses are indicative of the fact that many organisations are still
in the early phases of Big Data deployment. A logical first step is to start
by making efficiencies and proving the value of the project. Once that is
achieved you might look at external data with a view to providing new
opportunities.
We want the speculative. We want to move away from the
routine-based last year I sold these amounts of things; this year
I am going to sell five per cent more approach. There is nothing there
that says that five per cent is the right number, or that people are still
going to want to buy this stuff, or that a different segment of people
might want to buy this stuff Head of enterprise analytics, Large retail
combine to drive this process. Unlike traditional business intelligence (BI)
tools, anybody (within reason) can use them.
BI tools are not very agile. With the new Big Data visualisation tools
I dont need to build anything. I dont need any semantic layer or an
object-driven interface in order to grab data out, impose an hierarchy thats
relevant and then visualise something Head of enterprise analytics, Large retail
to drill down into the data at points of interest. Different employees can
interrogate the same pool of data in very different ways.
attention and they will be much through visualisation tools
more engaged... IT manager, Research Director, Consultancy
The managers of the business are much more interested in how well
they are performing, how they are complying with regulations, so
they will look at data through a completely different lens to the engineer
who will be looking at the fine details Software director, Utilities
can be more agile, rational and efficient, from top to bottom. Many of our
interviewees were excited by the possibilities.
The new way of analytics allows us to dig in, visualise, re-profile, cut,
slice and dice, which is a very dynamic and different way of managing
performance from the factory reports of how many employees youve got.
The second thing for us was the opportunity to bring information together
from multiple sources and apply these tools on top to give us insights that
otherwise we wouldnt be able to do CIO, Local authority
making process, ultimately leading to a high level of democratisation and
empowerment.
component of this wider transformation of business organisations and the
role of IT within them.
accuracy is a the benefits of data-driven decision-making to organisations resistant to
change. Some of our interviewees had experienced some real challenges here.
goal that is out
of reach for a Traditionally, leaders have been leaders for a reason and 99 per cent
of that is down to their gut instinct. What we are saying is we want
majority of to run our future using something else thats quite disruptive
organisations Director digital, analytics & Innovation, Business services
given budget Our field is still heavily based How do you move
in engineering. Theres been from gut feel to data-
limitations. a tradition of trusting the engineers based decision-making? In
However, the judgment and that the engineer is retail, you would think they
irreplaceable. With technology we are know their customer and
fact that a not trying to replace the engineer, but their local people: When the
to aid the engineers decision-making sun comes out I must put
majority of and allow the engineer to concentrate this in front of the store
respondents on the problems we cant solve But you cant work like
analytically or by machine learning. that nowadays, you cant
told us that That has required a cultural change. have a national chain thats
Its like the first time a pilot had to turn run by 700 different store
somewhere the auto-pilot on. Its about trusting it managers Head of enterprise
around 80 to Software director, Utilities analytics, Large retail
90 per cent
was good
enough was
telling
many cases, still reluctant to share it. Many organisations are still grappling
with the difficulties in persuading these parties to play nicely.
and the EU have different approaches to data protection causes significant
compliance headaches for CIOs.
their data outside of the what the issues are and occasionally get very
EU CTO, Technology rabid about it CIO, Education
actually making data available centrally, another problem is quality. The
implications of poor data quality can be far-reaching.
Computing asked about how accurate data would need to be in order for
it to be used as the basis for an important decision.
Thirty-six per cent of respondents chose 80 per cent accuracy, 32 per cent 90
per cent accuracy, and an optimistic 11 per cent went for 100 per cent (Fig. 7).
11% 11%
2% 4% 3%
0% 0% 1%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Data accuracy required
One-hundred per cent data accuracy is a goal that is out of reach for a
majority of organisations given budget limitations. However, the fact that
a majority of respondents told us that somewhere around 80 to 90 per cent
was good enough was telling.
Its about supporting gut feeling and experience, not replacing it.
If data is 80 per cent accurate you leave room to make the actual
decision Data infrastructure director, Global bank
The clear message is that when it comes to the data analytics and their wider
use in organisations, perfectionism is the enemy of progress.
problem. The need to train people internally in the relevant skillsets is
undoubtedly slowing down progress. the new raw
material and
We went out to The candidates that you need to
market to look for do this properly are largely PhD- information is
experienced Spark and holding post graduates. If you look for
Scala developers and they people doing ecommerce theyre incredibly
the new
are as rare as hens teeth expensive. We are now looking at academic currency
or they were looking for programmes, whereas before we would
salaries beyond what we be looking at someone with commercial
are able to offer, so we experience. Its the academic skills which
have gone and trained will make the use of our data in much
people up internally more interesting ways and thats going
CTO, Technology back to academia CIO, Online services
Interestingly, only 13 per cent stated that the raw data was the biggest issue.
The biggest challenges facing the architects dealing in raw data are those
already discussed simply knowing what is there and what is needed,
skills, knowing where to start and stop, the collation of data from numerous
locations, etc (Fig. 9).
39%
Information turning raw data into
something useful
something meaningful (48 per cent) and derive actionable insights from it
(39 per cent). As one of our interviewees succinctly stated:
I dont believe in this claptrap that data is the new currency; its not.
Data is the new raw material and information is the new currency
CIO, Government agency
Indeed. However, turning raw data into something useful tends to be the
preserve of BI teams and this can prove to be a procedural bottleneck.
came to turning data into information. The biggest problems were asking the
right questions and determining the real value of the information available.
With Big Data, theres always been too much data and not enough
science and its how to sift the wood from the trees. With the advent
of analytics and visualisation its getting a lot better, but I still think the
bottleneck is the volume of data against the question you are trying to
answer And the bottleneck is more about the skillsets than anything else
Head of research information & intelligence, Government department
Trickier still is turning information into knowledge because here the issues
are cultural as much as they are technical. It may be the analysts job to
translate information for the upper echelons of the business, but for true
data-driven decision-making this needs to go far wider. Changing the
organisational culture and building confidence in data-driven decision-
making were the two biggest challenges faced by respondents here.
Only the business can decide what questions need to be asked and only the
business can ultimately push the cultural changes required.
If you look at the C-suite 15 years ago, did half the people today exist
within that boardroom as they do now? Thats all come from the
technology. Now data and the governance around it are all going to change
it. It will have a lasting impact across the board Director, Consultancy
board was a key attribute for a project initiator. There was a tangible concern
it wants to that Big Data would be perceived as an IT issue and left on the side.
achieve, Its a C-suite and CEO There is still a view that IT
knows what conversation versus a CDO- puts laptops on peoples
CIO conversation. It will ultimately desks, so they sometimes
data it has and sit with a chief data or monitoring forget to include us in the
has decided officer, but clear strategy and
vision needs to be brought in from
conversation. We are missing a
presence on the board; the CIO
how people the top as its going to drive the is not on the board, he is one
business forward Director digital, step down Data analytics manager,
are going to analytics & innovation, Business services Infrastructure
Big Data (and indeed any other technology) projects can be broken into three
phases planning, implementation and exploitation of outcomes. Computing
asked participants in both qualitative and quantitative parts of our research
what they thought were the key factors for success in each stage.
and its movement between databases) 11%
Machine learning/automation 5%
FIG .13
Please pick up to five essential factors for
success in exploiting the outcomes of a Big Data project
Business buy-in 51%
Showing results quickly 39%
Trust in the information to enable data-driven
decision-making 39%
Having the vision to see the opportunities 38%
Building confidence among business users 37%
Simple end user tools 35%
Culture that embraces change 35%
Having a core understanding of the business 30%
Visualisation/engaging the audience/self-service tools 27%
The right skillsets 25%
Adapting strategies 23%
Data governance 23%
Integration with current BI tools 21%
Data provenance (tracing and recording the origins of data
and its movement between databases) 14%
Change internal processes 12%
Machine learning/automation 4%
seeing cant be disputed. You need a tool which makes it easier to prove
Head of enterprise analytics, Large retail
Its getting the trust and that can take Its getting
quite a long time. There can be a the results in
lack of trust because no one had seen this quickly that brings
before or had seen it in a different way you the buy-in
Solutions manager, NGO Solutions manager, NGO
going to use it. And, indeed, how it is going to persuade them to do so. Our
interviewees had more advice:
time. Its about balancing the attention, balancing the investment that
is how to mature CIO, Government agency
but businesses beginning to grapple with how they isolate the useful data and turn it into
knowledge and action.
are beginning
to grapple with The Internet of Things is going to produce a huge data explosion
over the next couple of years. I think CIOs have had their say on
how they where thats going and we are going to see more and more data from
connected devices, and there will be a requirement to analyse some of
isolate the that IT manager, Research
useful data
and turn it into Computing asked survey respondents about the relevance of the IoT to their
industries. The answers are shown in Figure 14. A striking finding was the 29
knowledge per cent stating that it was already making an impact not bad for a concept
at the top of the hype curve. A further 38 per cent stated that IoT would
and actionable affect the way that they worked but just not yet. Representatives from the
insight distribution and transport, telecoms/IT and public sectors were most likely
to state that they were already witnessing an impact on their organisations.
everything that is to do with the construction or maintenance of the
railway should be included in the design Data analytics manager, Infrastructure
22% 10%
Huge it is changing everything
11%
However, there are a few challenges to resolve before the IoT takes a firm
hold on the popular imagination and day-to-day life, as Figure 15 shows.
Analytics 27%
Bandwidth 23%
Storage 15%
Power supplies 4%
data protection law simply hasnt caught up with a world in which sensors are
proliferating at a lightning rate.
Some concerns are perhaps a little overplayed but it doesnt require a huge
leap to imagine personal medical data being used as a tool for some very
aggressive marketing or the prospect of critical public infrastructure being
used by those with nefarious intentions particularly given the spectacular
rise in targeted attacks that we have seen in the last 18 months.
Education was the second biggest area blocking further development of IoT,
according to our respondents both in terms of educating employees and
clients. However, that education can only be effective if security concerns
have been properly addressed. If they are not, misinformation will continue
to circulate and effective education becomes infinitely more difficult.
1
http://www.forbes.com/sites/parmyolson/2014/11/16/fitbit-data-court-room-personal-injury-claim
Turning this raw data into actual knowledge and action are the next steps
and it is in these areas rather than with gathering raw data that respondents
were experiencing their biggest challenges. Knowing which were the right
questions to be asking and determining what information was of genuine
value were big challenges, although not as big as the changes required in
organisational culture to build confidence in data-driven decision-making.
The Internet of Things (IoT) was changing organisations for nearly one
third of our research participants and the predicted explosion in data was
starting to happen. Although progress down the other side of the hype curve
is being impeded by real concerns about privacy and the ability of security
technology to protect it, the IoT is something we are likely to be exploring in
our Big Data Reviews in the years ahead.
For more information about our research, surveys and content creation
services please call Tom Wright, publisher, on (+44) 20 7316 9529 or John
Leonard, research and content editor, on (+44) 20 7316 9776.