IDC PERSPECTIVE

Data Intelligence in the Future of Intelligence

Stewart Bond

EXECUTIVE SNAPSHOT

FIGURE 1

Executive Snapshot: Laying a Foundation of Data Intelligence to Build

Enterprise Intelligence

Source: IDC, 2023

March 2023, IDC #US49921623

SITUATION OVERVIEW

The future enterprise will look more digital than physical. At IDC, we've been tracking the progress of
organizations' digital journeys for the past six years. We've seen and lived through two chapters of the
digital transformation era, and IDC now believes we are moving beyond digital transformation into the
era of digital business. Organizations will still transform, but that is no longer the primary focus in a
digital-first world. The C-suite has recognized that at some point, transformations must be superseded
by a bigger and more purposeful long-term goal and that is about business outcomes built on a digital
foundation, which is digital business.

The most cited critical lever in running a digital business in 2023 is using data and intelligence
strategically, according to respondents to IDC's August 2022 Worldwide Future Enterprise Resiliency
and Spending Survey. Pulling that lever is not easy because building a data-driven business and
culture is also the most cited challenge organizations are facing in running a digital business. This is
where IDC's future of intelligence research offers insight and guidance to help organizations build an
enterprise with intelligence across pillars of information synthesis, insights delivery, collective learning,
and data culture (see Figure 2).

Pillars, however, cannot stand on their own, but they require a foundation underneath to reenforce the
pillars and distribute the load. The foundation of enterprise intelligence is data intelligence. Intelligence
about data supports and informs every data-driven learning, analytics, decision, action, and outcome.

©2023 IDC #US49921623 2

FIGURE 2

Future of Intelligence Framework

Source: IDC, 2023

The four pillars of enterprise intelligence and their relationship to the foundation of data intelligence are
discussed in Table 1.

©2023 IDC #US49921623 3

 TABLE 1

Enterprise Intelligence Pillars

Pillar Enterprise Intelligence Definition Data Intelligence Foundation

Information This pillar includes the ability to use
synthesis technologies (business intelligence [BI],
artificial intelligence [AI], data
warehouses, etc.) to integrate
information across all data sources in a
way that drives actionable insights.
Data intelligence defines and describes data in context,
providing the necessary information required to integrate,
cleanse, and transform data so it can be consumed by
operational and analytical processes that drive actionable
insight.

Insights delivery This pillar includes the ability to get
actionable insights to the right decision
makers within your organization (e.g.,
manager, director, C-level executive). It's
important that not only are these insights
available but that they are also timely
and contextual.
Insights cannot be delivered to the right decision makers
without intelligence about who owns the data being used
to derive the insight, the context in which the data should
be used, and the controls that need to be in place to
protect sensitive information. Data intelligence helps
ensure the right data gets to the right resources at the
right time and ensures it is being used for the right reason.

Collective This pillar includes the ability for the
learning enterprise to generate knowledge from
insights and share that knowledge. This
is about learning from the insights being
generated and sharing the insights and
learnings across different business areas
to make important connections and
improve the overall business
performance.
Data intelligence captures technical, business, lineage,
usage, and relationship metadata about data and
information assets in the enterprise. Data intelligence can
provide insight into how and where data is being shared
across the enterprise, and it facilitates the capture and
sharing of knowledge through crowdsourcing facilities.
Intelligence and knowledge collectively maintained
contributes to collective learning across the enterprise.

Data culture This pillar focuses on the following
questions: Do you have a clear vision
and mission within your organization to
be data centric? Do you have executive
sponsorship to sustain a data culture?
Does your organization understand the
value of data?
Data intelligence improves the productivity of data workers
by reducing the amount of time it takes to find and prepare
data, and therefore, more time can be spent in analytics
with faster time to insight. Through lineage and business
context, data intelligence also provides traceability of data
to business outcomes, so that the true value of data
managed by the enterprise can be derived.

Source: IDC, 2023

What Exactly Is Data Intelligence?

Data on its own does not hold a lot of value; data needs context in analytics and decision making, and
context is provided by intelligence about data. Several years ago, IDC started using the term data
intelligence to define a category of software that harvests and analyzes metadata to provide the
necessary context. Data context is also required for intelligent data management and control.

This data intelligence category of software has led to a market, that many software vendors have
aligned to, and IT buyers are searching in their procurement of data catalog, quality, mastering, and
data life-cycle management software. Data intelligence software mostly emerged out of the data

©2023 IDC #US49921623 4

integration software market, and we are now seeing solutions emerging from adjacent markets, such
as data privacy and compliance software. We are also now seeing growth of data intelligence software
outpacing growth of data integration software.

We believe this is partly due to a shift in how data is being moved around an organization's IT
environment, a shift away from extract, transform, and load (ETL) patterns toward ELT in modern data
environments, and the reorganization and consolidation of data made necessary to move to the public
cloud. But it is also a recognition of the value of data intelligence. There has been tremendous focus
on business analytics, artificial intelligence, and machine learning to extract business value from data,
but without a solid foundation of data intelligence, results are questionable and so too is the value. In
addition to getting insights from data, data also needs to be protected as cybersecurity attacks are on
the rise and the number of regional data regulations are also on the rise. Complying with regional data
regulations that specify data privacy and security requirements without a solid foundation of data
intelligence could mean you may not be protecting the right data, but more likely, not all the data that
needs to be protected is being protected, simply because there is a lack of intelligence about it.

If data intelligence is a foundation, metadata is the aggregate used in the material that forms the
foundation. Metadata includes technical definitions of data in databases and object storage
management systems. Metadata also exists in lineage, captured as data flows through an
organization. There is business metadata in the form of glossaries, business lineage of data, and
relationships to business processes. There is behavioral metadata that is generated every time data is
accessed, updated, added, and deleted by users and applications in the enterprise. Semantic
metadata is used to describe semantic relationships between data sets and elements within data sets.
There is performance and optimization metadata generated when data is processed. Metadata is also
used to add structure to the unstructured and used to describe contents of binary objects such as
video, audio, and images.

There are competing perspectives in the broader data management market right now as we hear
about data lakes, data lakehouses, cloud data warehouses, data fabric, and data mesh architectures.
At IDC, we recognize that each category of solution is not the silver bullet of resolution to take control
of the highly distributed, diverse, and dynamic data that organizations are managing in a digital-first
world. Data will always be in multiple places, and regardless of our desire to get rid of silos, silos will
continue to exist, and that is not necessarily a bad thing, if we know what the data means and how
data may be applied in different contexts. The speed of data is also increasing as we see the
emergence of another adjacent market of real-time streaming data and analytics software. If data
becomes as standard as electricity, and every application turns into just another appliance plugged
into the data flow, the need for intelligence, quality, and control increases substantially. This is why
IDC introduced an architectural approach called the data control plane.

IDC defines a data control plane as an architectural layer that sits over an end-to-end set of data
activities (e.g., integration, access, governance, and protection) to manage and control the holistic
behavior of people and processes in the use of distributed, diverse, and dynamic data. Figure 3 shows
the components of a data control plane architecture.

©2023 IDC #US49921623 5

FIGURE 3

Data Control Plane

Source: IDC, 2023

The data intelligence domain in the data control plane informs policy, control, movement, and
observation of data across all other domains. Data intelligence should be the first thing that
organizations harvest and curate before deciding whether the data needs to move into a data lake,
warehouse, or lakehouse, or whether it can be leveraged in situ by using a fabric or mesh approach.
Data intelligence provides all the information necessary to make decisions about where the data needs
to be based on functional and nonfunctional use case requirements and how it needs to be controlled
to be compliant with governance requirements.

In the research work we have done at IDC over the past several years, we have seen the rise of data
intelligence's importance in organizations. More organizations are dedicating resources to data
intelligence, and accountability is increasingly being held by executive roles, ultimately reporting into
the C-level. In a recent IDC survey on the topic of data management, we see data intelligence and
integration initiatives at the top of the priority list in 2023, as shown in Figure 4.

©2023 IDC #US49921623 6

FIGURE 4

Priority Data Management Initiatives in 2023

Q. Which of the following initiatives are being prioritized for organizational resources (budget,
people, skills) in 2023?

n = 1,021

Source: IDC's Data Management Survey, 2023

Data catalogs are the cornerstone of data intelligence foundations, as they collect, curate, and
centralize (or federate) intelligence and knowledge about data. Data catalogs are necessary to support
data governance, quality management, and access use cases in the enterprise. Data intelligence is
now fueling another software category of data marketplaces. Data marketplaces offer users the ability
to "shop for data" like how they might shop for consumer goods online. When you are looking at a
product online, you are looking at metadata about the product, not the product itself. Similarly, it is the
metadata and intelligence about the data that is being used to describe data in a data marketplace.
Data marketplaces can be internal, external, or both, but at their core are merchandising systems for
data products. This isn't a stretch for external data marketplaces as they are very much focused on
buying, selling, or trading data products. But it is a bit of a stretch for internal data marketplaces, as
DataOps teams need to think about data products instead of data sets. Data product owners will need
to consider product quality, maintenance, and merchandising, including managing the product
metadata in the marketplace.

IDC surveys have also indicated that while there is tremendous value of intelligence about data as
organizations work to improve their enterprise intelligence, metadata management is often low on the
list of investment priorities. The emergence of GDPR, CCPA, and many other regional data regulations
increased the demand for metadata, managed by data catalogs, but we believe another catalyst is
required to spark the next level of investment into data intelligence. The scale at which analytical
workloads can be run in the cloud makes it very attractive, but realization of the potentially significant
costs associated with running analytical workloads in the cloud has also surfaced. We have seen
emergence of a practice called FinOps, which at its core is about making intelligent decisions on where
to put data and how to best scale analytical compute, to optimize costs. With data intelligence

©2023 IDC #US49921623 7

providing many of the metrics about data volume, diversity, complexity, quality scores, and so forth,
FinOps is possibly the next catalyst for growth in data intelligence investment and maturity of software
in the market.

Implications
Data cataloging and metadata management is a top priority for organizations in 2023, but software
spend in this area is often lower priority. This is perhaps because one of the most popular data
catalogs being used in organizations is the spreadsheet, or it could be an under-appreciation of the
technology and its importance. It will be incumbent on the data management organization to take
inventory of its own data intelligence capabilities and assess whether current tools, technologies, and
processes are sufficient in building enterprise intelligence. Laying a foundation of data intelligence will
also require change management because, to build a proper foundation, it will require adoption and it
will change how people work.

Top Investment Priorities

As the cornerstone of data intelligence, organizations should prioritize investment in a modern data
catalog. Catalogs of data stored in static documents or spreadsheets will not be able to keep up with
the rate at which data, data movement, data transformation, and data consumption changes in modern
data environments. Starting with a data catalog — that can continuously harvest metadata from active
data repositories, data movement, transformation, and consumption — is a great beginning to
improving your odds at being successful in enterprise intelligence.

Alignment with Broader Business Outcomes

In the recent IDC's Data Management Survey, IDC was able to correlate capabilities in data
management to financial and operational business outcomes and data management improvements.
Organizations with a high level of data intelligence, compared with organizations with a low level of
data intelligence, experienced 40% higher financial improvements and 20% higher operational
improvements and 200% more organizations reported significant improvements in data management
metrics. Organizations with a high level of data intelligence are those with data catalogs and metadata
management implemented and operational at levels higher than 50%, whereas organizations with a
low level of data intelligence are those at 50% or less operationalization of data catalogs and metadata
management.

Note: All numbers in this document may not be exact due to rounding.

ADVICE FOR THE TECHNOLOGY BUYER

Organizations looking to build enterprise intelligence need to establish a solid foundation of data
intelligence. As you assess your current state of information synthesis, insight delivery, collective
learning, and data culture, look at where and how intelligence about data is being leveraged in each
pillar. Close gaps where needed to reenforce the foundation to build even stronger enterprise
intelligence.

Not every organization is at the same point along the path to having a solid foundation, and each
should consider the following advice based on their current state of data intelligence:

 No need to boil the ocean in building a foundation of data intelligence. Focus on areas of the
business, a process, or a few critical data elements that do not require hundreds of

©2023 IDC #US49921623 8

 stakeholders to be involved. But also select a focus area that is business relevant to
demonstrate real business value with the outcome.
 Data catalogs are the cornerstone of the data intelligence foundation. If your organization uses
manual processes and static technologies to capture intelligence about data, it will not be able
to keep up with the scale of distribution, diversity, and dynamics of the modern data
environment. Modern data catalogs continuously collect and curate intelligence about data, a
prerequisite capability in modern environments in support of enterprise intelligence.
 Implementing data intelligence technologies is not usually the biggest challenge, but getting
end users to adopt and embrace the technology requires change management because it will
change how people work. Herein lies one of the biggest challenges to success: The value that
can be derived from data intelligence technologies, not unlike anything else, is directly related
to how much effort is put into the solution. The more that people use these technologies, the
smarter they become, and the deeper the body of knowledge that is built up in the intelligence.
Change management efforts need to reenforce the value that data intelligence can deliver to
the data worker, demonstrating both quantifiable and qualifiable benefits.

LEARN MORE

Related Research
 IDC Market Glance: Data Control Plane, 1Q23 (IDC #US50088023, January 2023)
 Future of Intelligence Framework (IDC #US49859622, December 2022)
 IDC PeerScape: Best Practices in Data Cataloging (IDC #US48429422, December 2022)
 Data Intelligence and Integration Observations in IDC's Future Enterprise Resiliency and
Spending Research (IDC #US49861122, November 2022)
 IDC FutureScape: Worldwide Data and Content Technologies 2023 Predictions (IDC
#US48733222, October 2022)
 IDC MarketScape: Worldwide Data Catalog Software 2022 Vendor Assessment (IDC
#US48395622, August 2022)

Synopsis
This IDC Perspective describes how intelligence about data is required to lay the foundation for
building enterprise intelligence in the context of IDC's Future of Intelligence framework. It positions
how data intelligence can be leveraged in the synthesis of information, delivery of insights, collective
learning, and building a data culture.

"Enterprise intelligence relies on data for analytics and decision making, but definition ad context,
including relevance, is required to ensure the right data is being used by the right resource, at the best
time, and for the right reason. Data context is provided by intelligence about data, " says Stewart Bond,
vice president, Data Intelligence and Integration Software Research at IDC. "The foundation of
enterprise intelligence is data intelligence. Intelligence about data supports and informs every data-
driven learning, analytics, decision, action, and outcome."

©2023 IDC #US49921623 9

About IDC
International Data Corporation (IDC) is the premier global provider of market intelligence, advisory
services, and events for the information technology, telecommunications and consumer technology
markets. IDC helps IT professionals, business executives, and the investment community make fact-
based decisions on technology purchases and business strategy. More than 1,100 IDC analysts
provide global, regional, and local expertise on technology and industry opportunities and trends in
over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients
achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology
media, research, and events company.

Global Headquarters

140 Kendrick Street

Building B
Needham, MA 02494
USA
508.872.8200
Twitter: @IDC
blogs.idc.com
www.idc.com

Copyright Notice

This IDC research document was published as part of an IDC continuous intelligence service, providing written
research, analyst interactions, telebriefings, and conferences. Visit www.idc.com to learn more about IDC
subscription and consulting services. To view a list of IDC offices worldwide, visit www.idc.com/offices. Please
contact the IDC Hotline at 800.343.4952, ext. 7988 (or +1.508.988.7988) or sales@idc.com for information on
applying the price of this document toward the purchase of an IDC service or for information on additional copies
or web rights.

Copyright 2023 IDC. Reproduction is forbidden unless authorized. All rights reserved.