Report for Amdocs

The future of big data

analytics in the telecoms
industry
7 July 2014

Justin van der Lande

Ref: 2000593-282

This document was sponsored by

 Contents

1 Executive summary 1

2 About this report 2

3 BDA for the telecoms market 2

3.1 Brief history and definition 2
3.2 Current size and status 4

4 Key challenges experienced by telecoms operators 4

4.1 Technical challenges 5
4.2 Staffing skills 6
4.3 Organizational challenges 6
4.4 Regulatory challenges 7

5 Solutions 7
5.1 Vendor developments to address the telecoms market 8
5.2 CSP approaches 10

6 Conclusions 11

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | ii

Copyright © 2014. The information contained herein is the property of Analysys Mason
Limited and is provided on condition that it will not be reproduced, copied, lent or
disclosed, directly or indirectly, nor used for any purpose other than that for which it was
specifically furnished.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 1

1 Executive summary

Big data analytics (BDA) is not just a passing trend; it is becoming an increasingly important part of
every aspect of a communications service provider’s (CSP) operations. The drive for efficiency gains,
coupled with the need to personalize customers’ experiences, is driving new installations. Falling
costs and the increasing functionality of BDA solutions are allowing CSPs to add more data to make
increasingly complex decisions without growing their costs. BDA tools are essential solutions that
provide insights and actionable intelligence, which deliver the required operational changes. A critical
element is the ability to deliver rapid insights; although not all processes need to be performed in real
time, most benefit from being done faster. There is a general movement by CSPs towards adopting
BDA techniques, in order to move every process towards real-time analysis.

In order to implement BDA, operators need to resolve five challenges:

 the adoption of new technology and use of associated new vendors

 organizational change, to provide the most efficient use of data and resources
 ensuring the right resources are used, including staff and external personnel
 the standardization of data in the enterprise, to ensure that big data can be best utilized across the
whole organization
 the impact of siloed infrastructure on the implementation.

BDA solutions address a huge variety of projects. When assessing the business case for such projects,
both the business benefits and the cost of implementation need to be considered; business cases where
data is readily available in a usable format are, therefore, more likely candidates for selection. Over
half of all BDA expenditure currently goes on projects that impact revenue and churn, which often
have the greatest potential for a return on the investment. Billing and usage-related data are both good
starting points, where ‘context aware’ marketing techniques can be used to tailor offers at the most
appropriate time in a highly personalized way.

A significant business case will help set up the framework on which further business cases can be
built. CSPs are adopting a much more open and heterogeneous set of technology on which to build
their BDA solutions when compared with more traditional data warehouse solutions. The ability to
use open source tools, such as that of Hadoop, helps to reduce the cost significantly, but there is still a
need to work with existing tools that have been optimized for current data and processes.

CSPs may choose to limit general-purpose tools that can be used as part of a BDA solution to a
smaller, strategic set used across the whole enterprise to provide conformity in all departments;
however, a more flexible approach may prove more viable. This approach allows for telecoms-
specific vendor solutions that can provide quicker implementations, industry expertise and ongoing
support, which can provide better value for a BDA solution. These specialized solutions are much
closer to an application than to a more general-purpose platform, providing much more ‘out of the
box’ functionality and saving CIOs from the necessity of developing skills to understand the
underlying data.
 The future of big data analytics in the telecoms industry | 2

2 About this report

This report was commissioned by Amdocs to look at the future of BDA within the telecoms industry,
in order to provide high-level guidance for operators planning their business with the support of BDA
tools and services.

The report begins with an assessment of the current situation and a definition of what is currently
meant by ‘BDA’, a widely used but ill-defined term. Although BDA solutions are not new, their use
has been recently accelerated through the advent of new technology and a significant lowering of
costs. This document also examines the business challenges that are being addressed by big data and
its analysis, and considers the different types of ‘use cases’ being delivered by operators.

New vendors have been quick to react to the changing technology and have adopted big data
techniques in their products and solutions. Established vendors with software tools already in place
are also reacting to market changes; they initially viewed the new technologies as potential threats,
but are now including them in their offerings.

This paper also assesses the various approaches that different types of vendors have adopted to
support BDA, and considers some of the impacts these have had on CSPs.

3 BDA for the telecoms market

3.1 Brief history and definition

Some people argue that CSPs have had access to big data for years already, and that the latest
discussion is ‘hype’ and will disappear over time until the next technology craze comes along. CIOs
have had to contend with earlier technology bubbles of statistical analysis, decision support, data
mining, knowledge management, business intelligence, business analytics and are, therefore, likely to
be skeptical when BDA is presented to them. It is true that CSPs have always had to manipulate and
analyze large data sets, but there is also a shift that is taking place beyond ‘business as usual’. This
has been triggered by a number of factors that are discussed later in this report; however, the net
impact in the underlying data is significant and makes BDA profoundly different from previous
trends.

Big data has three new characteristics:

 Volume: The sheer volume of big data has grown from petabytes to exabytes, and there are plans
underway to support zettabytes and even yottabytes. The scale of data means that analyzing it
would require a new set of systems that can cope with the mountain of data now available.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 3

 Variety: Data comes from various different sources (CDRs, data sessions, social networks,
internal reports, transaction-based systems) and in different formats (alphanumeric, XML, audio,
etc.). Established systems have been built around ‘structured’ data, allowed data to be stored and
accessed from a structured database. This also dictated the way data is loaded into storage,
needing to be transformed and structured before being saved to a data warehouse, for example.
Unstructured or semi-structured data does not need to be manipulated into a pre-defined structure
to be saved in newer data infrastructure that is supported by NoSQL databases (such as HBase,
MongoDB and Cassandra).

 Velocity: The velocity of data is the frequency at which it is generated. The high frequency of
streaming or real-time data leads to new opportunities for real-time management and reporting.

This combination of big-data characteristics is driving substantial changes in IT requirements, with

greater use of unstructured or semi-structured data changing requirements for storage and modelling.
Moreover, the ability to provide near real-time, cost-effective, predictive modelling and analysis on
the data has driven demand to use new technology such as the open source software of Apache
Hadoop. Hadoop is able to cluster commodity hardware to build a distributed file system – for
example, Hadoop’s own HDFS. Yahoo, Facebook and most of the Fortune 50 companies have been
significant users of Hadoop.

Figure 1. Big data technology is supporting three business challenges

CSPs have three significant business challenges that need to be addressed:

 Customers require increasing personalisation of the experience offered by CSPs, but at the same
time costs need to be driven down.

 Current processes across CSP enterprises need to be optimized to ensure, for example, that
network planning is considered in terms of a potential increase in revenue, not just from a
network-centric view.

 Revenue need to be increased or maintained as traditional services decline.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 4

The ability of data infrasture to deliver analysis or analytics more quickly or in real-time is critical for
CSPs’ responsiveness to improve business outcomes. Real-time or ‘in-line’ analytics can change; for
example, the selection of voice options on an interactive voice recording, which provide customers
with better support for their most likely reason for calling.

There is a virtuous circle between the relationship between big data and BDA, as the falling costs of
the storage and computing used for big data allows greater amounts of data to be analysed, and drives
the need and capability for yet more data.

3.2 Current size and status

Analysys Mason estimates the size of the analytics market in the telecoms sector to be US$2.1 billion
in 2013, and market growth is expected to be near 9% in 2014. The market is heavily biased towards
the developed markets of North America and Western Europe, as depicted in Figure 2.

This market size includes only the analytics software and associated visualization functions, and tools
for extracting, loading and transforming (ELT) data into an infrastructure and schemas specifically
built for the telecoms sector.

Figure 2: Regional Analytics market shares by revenue, worldwide, 2013 [Source: Analysys Mason, 2014]

4 Key challenges experienced by telecoms operators

BDA systems present a number of new challenges, which fall into three broad categories: technical,
organizational and staff skills. In an already complex data environment, in which all CSPs operate, the
ability to fully embrace the potential of big data is restricted by these factors.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 5

4.1 Technical challenges

BDA needs big-data infrastructure to support it. This may require significant changes to the current
technology and tools deployed by the CSPs. It may require upgrading current data warehouse
technology and the creation of a hybrid solution that will co-exist with both technologies, and support
structured and unstructured data. The use of each data set dictates the technology and location on
which it is stored. So where data is highly structured and is needed for high-speed transactions, for
example, it may be best served by a relational database – but location information from a handset on
location on a wireless network, or browser URL history, for example, may be better saved in an
unstructured data store. Therefore, understanding the business issues that are to be resolved with the
data becomes important before committing to underlying technology.

Big-data scale

The sheer volume and speed requirements need to be addressed to provide timely information that
fulfills a given business requires new systems. Initially, CSPs turned to their existing data
infrastructure providers for solutions that included the dividing of data into multiple databases or the
supply of higher-specification hardware. High-performance requirements also led to greater in-
memory solutions and higher-specification machines, but these have lately given way to be expanded
using newer open source and other parallel processing technology.

Real-time in-line actions

BDA is able to provide near real-time analysis of data that can be used to impact in-line processes.
CSPs increasingly need to process and action data in real-time to stay competitive. Context-aware
marketing needs instant analysis to ensure that the very latest context is incorporated into an offer, or
that customer care representatives have calls deflected when proactive messages are sent to customers
impacted by a network error. Recalibrating responses or interactions in near real-time and at a cost-
effective price requires BDA technology.

Real-time and big-data acquisition

BDA often utilizes new data sources; having the right tools in place to load data in the right format
and in a timely way presents a new set of challenges for CSPs. It is not unusual that one third of an
analytics project’s costs is from configuring and manipulating data to be used. Understanding the
types of data that CSPs use and providing a logical data model designed for them is essential, as
analytics requires a logical model on which to work. Where real-time or streaming data is required,
additional challenges can be placed on the performance of systems and networks providing the data.

BDA needs good visualization

Visualization of large data sets is challenging, but nonetheless still important to provide. Visualization
helps to spot the more obvious data quality issues, provides some analysis, and enables outlying or

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 6

clustered data to be quickly addressed. Big data visualization will need to be consolidated or grouped
in order for this to be achieved.

Analytics tools with pre-defined models help

CSPs need analysis quickly. The ability to provide data models that take months to create and deploy
will not help to provide the competitive edge that is needed. As the number of data attributes
increases, visualization and traditional models need to be supplemented with machine learning so that
models can be optimized. This allows, for example, personalization to be achieved on a scale that had
not been practical before. Having pre-defined model or optimized machine learning tools for a given
business case helps de-risk and reduce the price of projects for CSPs.

4.2 Staffing skills

There is no replacement to having staff with deep domain knowledge of data sets that are being used
in support of business cases. Understanding what data fields are significant or the elimination of
outliers can save considerable time and effort. General purpose analytics tools ensure that CSPs can
be supported from a wide range of resources. However, where there is strong demand for the same
resources, such as in the financial services sector, CSPs have to compete for staff on an open market.
Staff retention is a significant issue to ensure that CSPs can build up skills internally without having
to use external resources.

Two factors are being used to help resolve this challenge. First, the use of more powerful tools that
provide machine learning techniques or help non-technical staff manipulate data better are
increasingly available to CSPs. Second, the use of external resources where suppliers provide
managed services, professional services or even software as a service.

4.3 Organizational challenges

CSPs need to consider how to best support BDA requirements; for example, whether they require a
centralized team or one which is dispersed within departmental structures.

A central BDA team that provides a single enterprise-wide resource to address each project
requirement has its benefits in terms of best utilization of limited resource, but the use of a central
team inevitably leads to a bottleneck and management of the priority needs to be given to each
proposed project. This will often result in longer term, small impact work being pushed out. In
addition, the inflexibility can lead to departments using external services to resolve their business
challenges.

Potentially, the optimum model is a hybrid of approaches, where new models are initially created at a
central resource with the best and most skilled staff, with a view of all enterprise data. Departmental
teams can then adopt and refine the models to provide the additional tactical and incremental updates
needed to refine the models in a very timely way.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 7

4.4 Regulatory challenges

Data is increasingly being regulated. This includes the collection of personal data, the use of the data
and where it can be stored. There are also obligations for security and notification of security
breaches, if they occur. Regulations are increasingly requiring that consent is granted from the
individual before personal data can be passed to other organizations or be used for electronic
marketing.

5 Solutions

BDA solutions fall into four classes of products. Larger vendors will often have solutions in a number
of the classes, although each class continues to have specialist vendors focusing on the challenges that
they present. Figure 4 below depicts the different classes of tools within BDA solutions.

Where the ELT and extract/transform/load (ETL) tools enable streaming or batch mode extraction,
loading and transformation of data into the data infrastructure, the data store provides appropriate
schemas and embedded analytics within the databases for performance. Analysis tools enable data
scientists and data miners to search for partners in the data and create models to help predict future
outcomes. Output tools provide data to visualize data, hand off to automated processes, and offer data
through APIs or in reports.

Figure 4: Different classes of BDA tools

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 8

5.1 Vendor developments to address the telecoms market

New vendors continue to enter the BDA markets, encouraged by the growing demand by CSPs and
the availability of new, low-cost technology that is open source and is increasingly accepted by CSPs.
Vendors are categorized into four types based on their history and current capabilities, which provide
different view-points on open source and their access to the CSPs. These categories are storage
vendors, general-purpose analytics vendors, specialist vendors and new telecoms-specific analytics
vendors.

Figure 5: BDA vendors are entering the market with different backgrounds

5.1.1 Strengths and weaknesses of different vendor types

Established analytics and BI vendors

 Strengths: Established general-purpose analytics vendors provide the majority of the analytics
capabilities within CSPs today. Their robust products and tools are able to draw on long-
established pedigrees that have been built on knowledge from multiple industry sectors.
Established products have rich functionality and often cover all classes of tools within a BDA
environment to reduce integration and risk associated with complex BDA products. Analytics
tools are able to work with multiple data sources and are supported by professional services
teams. CSPs often already have skills and software licenses in place.

 Weaknesses: Telecoms-specific uses have to compete with other industry segments to develop
specific functionality. Though often a significant market, telecoms trails others – specifically
financial, retail and government. The ability to retain staff when trained on general-purpose tools
outweighs the benefit of being able to attract staff from a larger pool of trained staff. Deeper
knowledge or applications tend to be those that can be applied over multiple industry sectors and
not telecoms-specific.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 9

Storage tools vendors

 Strengths: At the heart of all BDA is the need to store and retrieve data. Vendors who have
storage and database technology see the potential in growing their products to provide data
analysis functionality. The ability to optimize the underlying data infrastructure to support
statistical models and queries helps drive performance. Storage vendors are also increasingly
supporting integration to the new technology, such as Hadoop, and are able to provide a robust
and supported integration with established enterprise data warehouse (EDW) deployments.

 Weaknesses: Although there are good reasons why established EDW solutions should be
continued to be used, vendors may be more reluctant to embrace new open source technology in
their solutions if they are vendors who supply EDW technology. In the same way as established
analytics vendors, the ability to address specific telecoms requirements will be compromised as
they need to address multiple markets. As storage and database vendors increasingly compete
with analytics vendors they may compromise current partnering arrangements with analytics tools
vendors.

Specialist software vendors

 Strengths: Specialist vendors will continue to have a significant role within BDA for CSPs. A
deep understanding of specific telecoms requirements and the underlying data provides CSPs with
valuable business benefits. To gain greater benefits, BDA vendors need to build solutions that are
solving specific telecoms-related business challenges. These will increasingly be positioned as
applications where BDA is a significant core component and the models, data sets and
presentation of data are closer to being pre-configured. The true added value is that vendors are
able to provide specialist resources to implement their solutions at each CSP.

 Weaknesses: Although some specialist vendors are able to expand from their specialist areas into
others in order to provide support for all BDA requirements the deployment of multiple vendors
tools will be needed.

Holistic telecoms vendors based on open-source software

 Strengths: There are some specific telecoms vendors that have been building their reputations
within the telecoms sector based on open source technology. Although specialist vendors are also
adopting Hadoop and other open source technologies, they have not necessarily developed a
platform or tools to address more general-purpose telecoms requirements. These vendors are not
building on, or restricted to, a specialist application area. Core skills in machine learning,
streaming technology and strong advocates of open source tools are helping them to establish
themselves as trusted providers of BDA tools outside of SI’s or general-purpose established tools
vendors. These vendors include both new vendors to the market, but also established telecoms-
specific vendors with deep knowledge of the telecoms sector that have a number of different
products already in place. The adoption of BDA provides a core platform asset that can be used to
support new analysis and functionality to both support current applications and develop new ones.

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 10

 Weaknesses: Although open source data tools are becoming more widely adopted within
telecoms, they are still not established with developments still underway to provide the same level
of support and stability current found on established vendor technology. Using a telecoms-specific
vendor will limit cross-industry benefits that could be derived from more general-purpose tools
that are sold into multiple sectors.

5.2 CSP approaches

BDA has become a very active topic of discussion within CSPs as they consider how best to make use
of the big data assets that are available to them. In addition, the relentless need to make business
processes better, cheaper and near to real-time is shaping CSPs’ BDA strategies. CSPs are moving
towards a real-time operations environment where, for example, CDRs are increasingly moving from
post-paid batch collection and rating to be real-time. Customer interactions are inevitably moving to a
real-time basis and collecting data every 15 minutes, as with most post-paid systems it is not sufficient
to provide a market-leading experience for the customer. Moving to a connection-oriented network
where potentially information is being updated every 70 nanoseconds as packets arrive on a 10GB IP
network means that not only is there a need to react faster, but the richness of information is far
deeper than most currently deployed systems can cope with.

The scope and amount of data that could be saved and processed in a BDA system is too expensive
and too time consuming to save all available to CSPs. Business cases for each must therefore be
considered.

Figure 6: The process flow considered by CSPs for BDA

Figure 6 outlines the generic steps that CSPs need to go through to decide what BDA infrastructure
and data is needed for each business case. This process works on the principal that there is already an
understanding of the key data attributes needed to support a specific business case. Each business case
justification will also need to explain the speed at which processing is required, what needs to be in
real-time, if anything, and what the output information will be. The building of models and algorithms

Ref: 2000593-282 .
 The future of big data analytics in the telecoms industry | 11

that take information and create knowledge can be built using machine learning tools, off-the-shelf
templates or data scientists tasked with investigating the business case.

Analytics tools, visualization tools and ETL/ELT tools that address each of the process functions as
outlined in Figure 4 are rarely from a single vendor and often involve multiple tools at each level. At
the heart of all business solutions is, however, the data on which models are created and often the
technical starting point for CSPs to build BDA solutions. The ability to access data sources such as
Amdoc’s Terascale quickly helps business cases built on that data. There is an element of a ‘chicken-
and-egg’ scenario, as to which is more significant: a business case that justifies the infrastructure, or
the available infrastructure and data that justify the business case. Billing and usage data is already
available and provides a rich set of business cases, such as context-aware marketing that can utilize
the current data sets far better.

CSPs are adopting a dual approach to BDA where there is both a technical set of projects that
consider the expansion of current data infrastructure to support new technology and the search for the
‘killer’ BDA project to help justify the expense. Over the past year, open source technology has
moved from a lab environment to being deployed in mainstream live processes. Where vendor
solutions are adopted they help remove the perceived risks associated with open source technology
and CSPs are more likely to adopt it. Moreover, if the solution provided by a vendor is an application-
based approach built around a business case, the adoption is made easier still.

6 Conclusions

BDA is not just a passing trend; it is becoming an increasingly important part of every aspect of a
communications service provider’s (CSP) operations. The drive for efficiency gains, coupled with the
need to personalize customers’ experiences, is driving new installations. Falling costs and growing
functionality are enabling CSPs to make increasingly complex decisions based on an expanding data
set without growing their costs. BDA tools are essential solutions that provide insights and actionable
intelligence, which deliver the required operational changes.

Although it is clear that due to each CSP’s unique requirements the need for generic tools will always
be a significant factor, it is often an applications-based approach that accelerates the justification for a
BDA framework.

Ref: 2000593-282 .