IDC PERSPECTIVE

A Framework for Evaluating Storage Efficiency Technologies in

Enterprise-Class All-Flash Arrays (AFAs)
Eric Burgener

EXECUTIVE SNAPSHOT

FIGURE 1

Executive Snapshot: Storage Efficiency Framework

Source: IDC, 2017

April 2017, IDC #US42464717

SITUATION OVERVIEW

Storage efficiency technologies are primarily used to increase effective storage capacity, decreasing
the cost to store and/or transmit information. They include a range of technologies that have evolved
over the years, based on the characteristics of the underlying storage media, and have been in use in
enterprise storage for decades. Compression is an example of a popular early storage efficiency
technology that is in widespread use, first with tapes as far back as the 1960s and then later adopted
for use with hard disk drives (HDDs) and solid state drives (SSDs). Other storage efficiency
technologies that have emerged include deduplication, thin provisioning, pattern recognition, write
minimization techniques, and the use of delta differentials in both snapshots and replication. The cost
to store and transmit data has always been a concern, but in this era of explosive data growth, it has
taken on increased importance. At its core, storage efficiency technologies identify redundancies,
patterns, and common workflows that can be modified to increase the amount of information that can
be stored on a given amount of storage capacity. Storage efficiency technologies will continue to
evolve over time but will be with us as long as information technology costs have to be managed.

The first step in evaluating storage efficiency technologies is to understand the different measures of
storage capacity and its importance. For the purposes of this discussion, there are three measures of
storage capacity: raw, usable, and effective.

 Raw capacity is the storage capacity a device (or array) offers when it is first shipped and
before it has been prepared for actual use. In all-flash arrays (AFAs), there is usually some
amount of flash media overprovisioning for reliability reasons and this varies by vendor but is
not often specifically called out. Higher rates of overprovisioning increase the endurance of the
stated raw capacity but also increase the cost per raw gigabyte (GB).
 Usable capacity is the storage capacity a device offers after it has been prepared for use by
formatting. The usable capacity will be smaller than the raw capacity, but the difference will
vary based on the type of formatting. In IDC's opinion, the usable capacity should also take
into account data protection overhead, which will vary by the chosen protection method
(mirroring or RAID 1, RAID 5, RAID 6, erasure coding, etc.). Adding data protection adds some
level of data redundancy that makes storage media resilient to failures (increasing capacity
consumption) and it results in some level of write amplification that varies by the method
chosen.
 Effective capacity is the storage capacity that the device appears to offer after all storage
efficiency technologies have been taken into account. For example, if a datacenter uses
compression and gets a 2:1 compression ratio (i.e., 1TB of information can be stored in
500GB of usable capacity), then the effective capacity is twice that of the usable capacity. In
IDC's opinion, the effective capacity measure should also take any additional capacity
consumption driven by data protection algorithms as well (as an input to the usable capacity).
Effective storage capacity is the key metric to use when calculating cost per gigabyte for
comparison purposes.
In most enterprise environments, multiple storage efficiency technologies are in simultaneous use
against any given workload. The combination of space-saving benefits of these technologies can be
referred to as the data reduction ratio. Different vendors may define the data reduction ratio differently
based on which storage efficiency technologies they offer in their portfolio. In a perfect world, the net
data reduction ratio should be defined as the cumulative impact of all storage efficiency technologies
and data protection (i.e., RAID) against a given data set — the amount of data written by an application
divided by the actual amount of data written to persistent storage in an array to store and protect it.

©2017 IDC #US42464717 2

Many enterprise storage arrays display a data reduction ratio on their GUIs, but what they are actually
tracking to calculate that number may vary. For most vendors, the tracked data reduction ratio is just
based on the combined effect of compression and deduplication — an approach that can understate the
actual savings obtained by the combined impact of all storage efficiency technologies. Those vendors
that take more of the storage efficiency technologies in use into account when displaying the data
reduction ratio are in fact providing a more accurate measure of it, which offers advantages to storage
administrators actively managing effective storage capacity. It is extremely important to understand
just what is being tracked by particular vendors if data reduction ratios will be compared across
different products.

Data reduction ratios are dependent on workload characteristics, the combination of storage efficiency
technologies in use, and how these technologies are actually implemented (in decreasing order of
importance). Different workloads will exhibit different data reduction ratios on the same platform while
the same workload will exhibit different data reduction ratios across different vendors' platforms. Data
reduction ratios are not static; depending on the combination of storage efficiency technologies, the
data stream and data set definition, and the workflows in use at the time, the data reduction ratio will
vary. Many vendors provide data reduction guarantees (typically somewhere in the 3:1 to 4:1 range)
and enterprises should definitely be aware of this and leverage it, but also understand that data
reduction ratios can vary day to day in a single shop or even against a single workload. When
evaluating the economic benefits of storage efficiency technologies, it is difficult to do it accurately up
front, and any anticipated benefits should take into account what the average data reduction ratio (and
the trajectory of its evolution) is over time.

How Flash Has Changed the Storage Efficiency Landscape

The typical I/O path for a write in an enterprise storage array encompasses the application (running on
a host) generating the write and the write flowing through the storage hierarchy (from CPU to memory
to any caches and eventually to persistent storage). The write path for a networked storage array will
additionally include transmission through a host bus adapter (HBA) connected to the server's I/O bus,
along a network transport of some kind (Fibre Channel, Ethernet, etc.) to an array controller, which
then writes through to persistent storage (sometimes using one or more array-based caches or tiers).
Given the write profiles of most enterprise applications, there are opportunities all along this path to
optimize, thereby minimizing the amount of write activity (and hence persistent storage) required to
service any given workload.

When most primary storage was composed of HDDs, it was more common to find storage efficiency
technologies that operated as a postprocess to the initial write. Because HDD latencies were very high
(relative to CPU cycle times), any attempt to perform any data reduction in line imposed noticeable
performance impacts, precluding their use for latency-sensitive applications. Compression technology,
which is an option on virtually every tape device, did operate in line but tape's very low performance
relative to HDDs had relegated its use to archive and other nonlatency-sensitive application
environments where the latencies imposed by inline data reduction were largely irrelevant. By 2005,
any applications that had any degree of performance sensitivity, including backup and disaster
recovery workloads, were using HDDs as persistent storage.

The use of disk in backup and other secondary storage environments opened up the use of interesting
new technologies for data protection, including snapshots, compression, and deduplication. Vendors
like Avamar, FalconStor, and Data Domain introduced backup appliances that had built-in data
reduction. Latency impacts due to the use of these technologies in HDD-based environments were not

©2017 IDC #US42464717 3

an issue since the backup use cases for them tended to not be latency sensitive. These technologies
were extremely valuable, because with the high degrees of redundancy in daily backups customers
could realistically achieve data reduction ratios of 10:1 or greater over long periods of time,
significantly lowering the cost of secondary storage capacity. Unfortunately, HDDs just did not offer the
latencies to support the use of inline data reduction technologies for more latency-sensitive primary
applications.

The use of flash as persistent, primary storage changed all of that. With consistent latencies that were
at least 10x lower than that offered by HDDs, flash enabled the use of a number of storage efficiency
technologies in line without unduly impacting primary application performance. (It impacts it, but just
not in a way noticeable by end users because of low flash latencies.) In 2005, most general-purpose
enterprise storage platforms did not support inline data reduction even as an option, and those that did
had relatively low utilization of those features for primary storage applications due to the latency
impacts. Today, literally every competitive AFA offers a portfolio of storage efficiency technologies,
and these technologies are in wide use across primary workloads of all types. Enterprise-class AFAs
began to ship in 2011 and while today only 15% of enterprises are actually running one in production,
76% of enterprises expressed the intent to evaluate and/or purchase one in the next 12 months as a
general-purpose enterprise storage platform for mixed workloads. Interestingly, IDC expects that by
2020, AFA revenue will account for almost 80% of all primary enterprise storage spend.

The use of inline storage efficiency techniques — algorithms that reduce the data before it is ever
written to persistent storage — offers significant benefits over postprocess approaches. When less data
must be written to represent any given piece of information, space savings occur not only in storage
capacity but in the bandwidth required to transmit that data (while in its data-reduced form). The need
to move less data results in an effective increase in bandwidth, which can result in performance gains.
When postprocess methods are used, the data is first written in its full form, then read back and
reduced, then written again to persistent storage. While this does eventually result in a better use of
storage capacity, systems must still have sufficient storage capacity to write the data in its full form
when it is first ingested, limiting the cost savings associated with purchased storage capacity.

The use of inline storage efficiency technologies provides other benefits important in all-flash systems
as well. Unlike spinning disk technologies, flash media wears each time data is written to it, so the
reduction in write volumes due to the use of inline storage efficiency technologies also helps to
increase flash media endurance and reliability. This combination of benefits, enabled by flash, makes
the use of inline storage efficiency technologies an absolute requirement in enterprise-class AFAs.

3rd Platform Computing Data Streams and Data Sets

Most enterprise workloads today are comprised of a mix of legacy applications, including relational
databases, collaboration and messaging platforms, and file shares, combined with a mix of next-
generation applications (NGAs) that service mobile, social media, big data/analytics, and cloud
workloads — all of which are primarily running on virtual infrastructure. This creates an I/O profile (i.e.,
data stream) that is very different from what most enterprises were servicing just a decade ago.
Today's I/O profile is much more dynamic, characterized by a variety of read/write ratios, a wide
distribution of block sizes, a heavy skew toward random (rather than serial) I/O, a significant amount of
data redundancy (i.e., the data is highly reducible using storage efficiency technologies), and "hot
spots" (I/O bands that drift over time). HDD technologies are not good at handling highly random I/O
with any degree of performance, driving the need for flash as a persistent storage medium, and the
much lower flash latency also opens up the possibility of leveraging inline storage efficiency

©2017 IDC #US42464717 4

technologies against today's highly reducible data sets to further minimize the cost of effective flash
storage capacity.

The Key Data Reduction Technologies and How They Work

There are six classes of storage efficiency technologies that should be available for use on all-flash
enterprise storage platforms, although all of them will not necessarily apply to all workloads all the
time. Because these technologies generate computational overhead and can add latency, it is
important to be able to apply them selectively to ensure optimal utilization of storage platform
resources.

Compression
Data compression involves encoding information using fewer bits than the original representation,
storing it in this more efficient form, and then decompressing it to restore the original representation
when the data is needed. Compression can be either "lossy" or "lossless." For enterprise applications,
compression algorithms tend to be lossless since there is less data that meets the definition of
"unnecessary" or "less important." Compression algorithms use a variety of different techniques,
including removing unneeded characters, inserting a single repeat character to indicate a string of
repeated characters, and substituting a smaller bit string for a frequently occurring one, to shrink the
amount of storage capacity needed to store a given piece of information. Compression has been
around since at least the 1960s, and good open source algorithms are available. Most enterprise
compression technologies used in storage arrays are based on modified versions of the open source
Lempel–Ziv set of lossless compression algorithms.

Certain workloads tend to benefit more from compression than others, although even for those that are
most amenable to it, data reduction ratios tend to be no higher than 2:1. Certain technologies that go
by the name of "compression," such as Oracle's Hybrid Columnar Compression, can produce much
higher data reduction ratios, but only against a single application (in this case, Oracle). When
compression is performed by an application, this takes host cycles (often driving the need for more
CPU cores and hence higher software licensing costs) and is not for general-purpose use (it works
only with that application). For enterprises looking to host mixed workloads on a single enterprise
storage platform, the more general-purpose compression algorithms that can be used with any
application tend to be a better fit as their value can be spread across the entire workload, not just a
single application. They also offload the computational workload for compression to the array, freeing
up more host cycles to drive application performance, and possibly, reducing software licensing costs.

Compression can be done on the host (as in with Oracle's Hybrid Columnar Compression), in storage
controller firmware, or in software that runs on the storage array. In the past, there have been
performance differences between controller and software-based compression, but with today's faster
processors, this difference can be very limited. Array-based implementations which run entirely in
DRAM can provide performance advantages in certain cases, with the trade-off being potentially
higher memory requirements. As DRAM costs come down and memory density increases, system
designs which move more of the storage efficiency workload to run in main memory will offer better
processing efficiencies as well as performance advantages.

Deduplication
Deduplication identifies and removes redundant "chunks" of data, replacing them with pointers to a
single logical copy of that chunk kept in an index. If the same chunk of data appears 1,000 times in an
array, it can save significant space to store one logical copy of it and 999 pointers to that copy instead

©2017 IDC #US42464717 5

of the full 1,000 copies. New chunks that have never been seen are written into the index, while each
time a chunk that has already been indexed would be written, the pointers are updated instead. When
a request for that chunk arises, the pointers refer to the single logical copy, and the chunk is read from
there. Deduplication indexes can be maintained within a single node (or system) or across multiple
nodes (an approach referred to as global deduplication). In most environments, global deduplication
produces higher data reduction ratios, more than offsetting any of the additional overhead incurred in
managing an index across multiple nodes.

Deduplication first appeared in backup appliances, as mentioned previously. Backup is a workload that
benefits significantly from deduplication. When backups are taken daily, it is typical that only 3–5% of
the data changes day to day. As each daily backup is deduplicated, only changed (or new) data is
written — all of the data that didn't change is not rewritten, pointers are just updated. This makes
backups faster, can result in significantly less data being transferred across networks (depending on
where the deduplication algorithm is implemented, on the backup source or the backup target), and
can easily result in the need for 90%+ less storage capacity.

Most enterprise environments have significant redundancy in their primary data as well. For example,
in a 1,000-seat virtual desktop environment, every desktop has a copy of Windows and likely of other
key applications (e.g., Microsoft Office, instant messaging, network file systems, antivirus software),
driving high degrees of redundancy within widely used applications and templates. But redundancies
can also be found in individual file data, an operation improved by the process of "chunking" data (i.e.,
slicing objects [e.g., a file] and data up into smaller chunks). Systems can look for redundancies at the
file level, but they tend to find more redundancies (and hence drive higher data reduction ratios) by
chunking the data at the subfile level and looking for redundant chunks.

The most efficient deduplication algorithms for general-purpose mixed workloads work at the subfile
level. Some systems work with fixed chunk sizes (e.g., 8K blocks), while others work with variable
block sizes. While vendors tout their specific implementations, in actual use, there has not been a
noticeable advantage to the use of fixed or variable block sizes that is consistent across all primary
workloads (although there can be advantages with specific applications to either approach). Variable
block size-based deduplication does appear to offer some advantages in some secondary workloads
like backup. What is important however, for primary workloads, is that the deduplication algorithms
work at the subfile level.

The emergence of flash as a persistent storage medium delivers latencies that allow deduplication to
be performed in line without unduly impacting application performance, enabling its use in latency-
sensitive primary storage environments as long as they were running on AFAs. While deduplication
could be performed either at the source or target, it is most often performed in a shared storage target
to offload servers of the computational overhead and better enable the use of global deduplication
repositories to drive higher data reduction ratios across data owned by multiple servers. Data
deduplication ratios will vary by workload and over time, and in primary storage environments, data
reduction ratios based on deduplication can be significantly higher than that achievable with general-
purpose compression — up to 10:1 or more in some cases but more generally in the range of 2:1 to 4:1
in mixed workload virtual environments.

Thin Provisioning
In the past, storage was provisioned in volumes, typically in sizes that would allow for some amount of
growth in storage usage before volume sizes needed to be increased. When a new volume was
created, often less than 10% of the capacity was actually used, while 90% was reserved and there to

©2017 IDC #US42464717 6

accommodate growth. The space was statically allocated specifically to that volume and not available
for use by any other volume, so in most systems, there was a lot of storage capacity allocated but not
actually in use much of the time. Thin provisioning effectively allows storage capacity provisioning to
take place dynamically and without direct operator involvement. While volumes can be of any specified
size (logically), their actual size (i.e., actual capacity consumption) is based on how much actual data
they contain. A 2GB volume with only 384KB of data written to it only consumes 384KB of actual
space, although it could potentially grow to 2GB in size over time. As more data is written to it, the
volume's actual capacity is increased as needed by the system (rather than an operator), pulling from
a global pool of capacity that is potentially available to all volumes in a system. Thin provisioning also
made it easier to expand capacity because storage could just be added to the global pool, at which
point it became available for use with any volumes as needed.

While thin provisioning makes for much more efficient utilization of existing storage capacity, it did
impose overhead which, with HDD-based systems, in many cases, noticeably impacted application
performance. As a result, it was used sparingly, and variants of it were developed that offered higher
performance in HDD-based systems (with certain trade-offs). As flash began to be more widely used
for persistent storage, it addressed the thin provisioning performance impact just as it did with
deduplication, and it is now in much more widespread use as it enables very efficient capacity
utilization without any noticeable performance impacts. IDC in fact views it as a baseline requirement
in enterprise-class AFAs.

The use of thin provisioning impacts effective storage capacity differently, depending on storage
provisioning and other workflows. It provides the most benefit in high growth systems where new
storage is being frequently allocated to or moved between users and projects. When a volume is
provisioned, storage administrators determine the maximum volume size. To make room for more
growth, they may have used larger volume sizes which (without thin provisioning) wasted a lot of
storage space. Alternatively, to make better use of space they might choose smaller volume sizes but
have to go in and expand them more often to accommodate growth. Relative to static provisioning, the
use of thin provisioning in systems where administrators want to allocate larger volume sizes to
minimize management overhead in environments where lots of new volumes are being created will
benefit the most from its use. With virtual infrastructure, environments where virtual machines are
being constantly created and deleted are dynamic environments where thin provisioning can offer
significant space savings. Relatively static systems not experiencing much growth or much new
volume creation will not benefit from the use of thin provisioning nearly as much. The magnitude of
space savings with the use of thin provisioning is generally much less than with compression and/or
deduplication, but it is an important feature for high data growth environments.

Pattern Recognition
There are certain patterns, such as repeating zeros when provisioning and/or reclaiming storage, that
are widely used. Instead of writing all these zeros, blocks of varying sizes can be defined as including
all zeros (and hence being available for use) without actually writing all the zeros. Other types of
patterns occur during formatting, data migration, and other administrative tasks where techniques like
"zero block elimination" can provide some space savings (as well as decrease the amount of data that
has to be actually moved to complete any operation). Because these approaches generally result in
less data being written to an array, they can also have positive impacts on flash endurance and
reliability as well as the speed with which certain storage operations get completed. Potential space
savings using these types of approaches varies by workload and workflow but are significantly smaller
than that achievable by technologies such as compression and deduplication.

©2017 IDC #US42464717 7

Write Minimization Techniques
In most legacy shared storage array designs used with latency-sensitive primary workloads, writes
were cached in resilient, DRAM-based caches (generally in the storage controllers), a write
acknowledgement was returned to the host, and the data was then later written through to persistent
storage. This is the classic definition of a write-back cache design, an approach which improves write
latencies but requires resilient cache recovery infrastructure (to ensure data is not lost in the event of a
storage controller failure for data in cache that had not yet been written through to persistent storage).
For arrays built around HDDs, write-through cache designs (where the data is written all the way
through to persistent storage before a write acknowledgement is returned to the application) were less
expensive to produce than write-back cache designs but generally resulted in higher write latencies.

Placing data in a small, high-speed cache before writing it out to persistent storage provides
opportunities for optimizations that can minimize the number of writes to persistent storage. For
example, a number of random, small block writes can be reorganized into a single, contiguous large
block write, making the eventual write-back to persistent storage more efficient in terms of processing
overhead and bandwidth consumption. A single block which is overwritten a number of times in quick
succession in cache needs to only have the latest version of it written when the time comes to move it
to persistent storage. Some writes to cache do not need to be persisted and can be identified and
discarded before they are ever written through to persistent storage. Data integrity is not affected by
these "write coalescing" algorithms since if the data needed to be accessed between the time the
block was first written to cache and when it will eventually be written out to persistent storage (or
discarded), the current data state is always accessible from cache (and protected against failures by
resilient cache designs). Taken together, these approaches can minimize writes (and the associated
overhead of moving every write individually to persistent storage) as much as 10–20% for certain
workloads over time.

Because these techniques minimize the number of writes that must be persisted to handle any given
data stream, they can result in better performance and, in AFAs, help increase flash endurance and
reliability. When end users are purchasing AFAs, they should absolutely look for these types of
features but also be realistic about the overall storage efficiency impact they are likely to have on their
workloads over time (as it is not large). As AFAs evolve to become the general-purpose primary
storage platform of choice in enterprises over the next several years, the use of these types of
techniques will become even more widespread — as they should, given their broad positive impact on
all-flash storage solutions.

Space-Efficient Snapshots and Delta Differential Technology

Certain datacenter workflows require the creation and retention of lots of copies of data. Clones and
snapshots are two ways to do that. Legacy HDD-based systems used clones (full block for block
copies) when higher performance was needed on either the source or target for read/write copies and
snapshots when administrators could give up some performance to achieve space efficiency and
faster creation times for read-only copies. Snapshots used pointers to refer to blocks that the target
shared with the source and did not require any data to be moved to create a read-only snapshot. When
read/write activity needed to be enabled for either the source or target snapshot and the highest write
performance was not required, two design approaches were used to handle the writes — copy on write
and redirect on write. Copy on write snapshots generate higher overhead on the source system and
original copy than redirect on write snapshots, which put more of that on the target system and copy.
Depending on how the snapshots will be used, one of these approaches may be better than the other.

©2017 IDC #US42464717 8

The performance impact with snapshots stemmed from the common use of shared blocks to achieve
space efficiency and faster creation times. Head/disk assembly contention when those shared blocks
were in high demand could cause performance problems in HDD-based systems. The use of flash
allowed space-efficient snapshot implementations to perform much faster, enabling their much broader
use across a number of different datacenter workflows. Since snapshots could now simultaneously
offer fast creation times, space efficiency, and high performance in very scalable implementations that
could support thousands of snapshots with very limited performance degradation, the need for clones
(full copies) was minimized. Most AFA platforms offer highly flash-optimized snapshot implementations
to provide scalable, high performance, but those that have moved much of the snapshot metadata
processing into RAM achieve the highest levels of scalability and performance. The art of leveraging
high-performance snapshot implementations in enterprise-class storage systems is referred to as
"copy data management," and it is enabling datacenter administrators to optimize workflows in ways
never before possible with the older, slower, and less scalable snapshot technology based around
HDDs.

For certain environments, significant space savings can be achieved by the use of high-performance,
very scalable snapshot implementations. Virtual desktop infrastructure (VDI) is a perfect example of
that. If we assume a 1,000-seat VDI environment, every desktop needs a copy of Windows, and
possibly other applications like Microsoft Office, instant messaging platforms, network file systems,
and antivirus software. The binaries for these applications are read only, so all 1,000 seats can
potentially be serviced by a single logical copy of those shared applications (this type of highly shared,
read-only snapshot is often referred to as a linked clone). If we assume that the storage footprint for
that collection of software is 15GB, then instead of deploying a dedicated copy of them for each
desktop (15GB x 1,000 = 15TB), you only deploy one that supports all of them (15GB), saving almost
15TB of space. Obviously, there will be a great deal of shared concurrent read access to those blocks,
particularly during certain times like boot, log-on, and logout storms, but flash-optimized snapshot
implementations running on flash can provide that performance. It is also interesting to note that there
are other administrative benefits to using this approach, such as any patches or updates to the shared
applications need be done only once to make them available to all desktops. Under this model, each
desktop also has some dedicated storage capacity for personal data.

Delta differentials identify redundant data between comparison data sets (e.g., two snapshots),
minimizing the data that has to be sent and/or stored to accurately represent two separate data states.
The most common use of this technology is with snapshot and replication technologies. Snapshots are
widely used in a number of different datacenter workflows, including test and development, data
protection, and business analytics. If successive snapshots are taken of the same data set as it
evolves over time and stored locally, it is much more efficient to track and store only the changes,
using pointers to refer to data that does not change. If snapshots are periodically replicated to a
remote site for data protection purposes, it is much more efficient to transmit and store only the
changes (after the initial full copy has been replicated) along with a set of pointers to the blocks that
did not change from the original snapshot.

While this approach can save significant storage capacity while retaining a lot of recovery points or
making available a number of copies of the same data set for testing or quality assurance purposes,
there are some caveats to its use. If redundant data (e.g., data that is shared across multiple
snapshots) becomes corrupted, then all of the snapshots that reference that shared block will also
become corrupted. This is a concern any time data redundancies are identified to aid in data reduction
operations (such as with deduplication as well). This makes the use of very resilient data protection
schemes, like dual parity RAID, multiple replicas, or erasure coding, that can transparently recover

©2017 IDC #US42464717 9

from multiple simultaneous failures, extremely important when storage efficiency technologies that
work based on identifying redundant data are in use.

The impact of delta differentials on capacity consumption is heavily based on workflows. Where lots of
snapshots or other copies of data are commonly used with processes that access that data on a single
storage system, or where slightly modified versions of the same data set over time are frequently
replicated to remote locations, delta differential technology can result in significant storage capacity
and bandwidth utilization savings. Workloads where this is extremely important include VDI, analytics,
test and development, and backup and disaster recovery.

Protecting the Data in AFAs

The broader use of AFAs will result in a significant increase in infrastructure density — in other words,
the floor and rack space, energy consumption, and acquisition cost will decrease to drive a given level
of business performance (e.g., transactions per second, page views per second). With this increase in
performance and storage density, concerns about overall system availability (and the related metrics
of reliability, resilience to failures, and recovery times) will also increase because a single component
failure can have a much larger impact on application performance. Data protection and system
recovery issues are not necessarily orthogonal to storage efficiency ones, though, since methods of
data protection and recovery impact overall considerations of storage latencies, capacity consumption,
and bandwidth utilization.

Most enterprise arrays use some form of RAID, rather than replication, to provide primary data
protection locally. With RAID, a given write is decomposed into some number of data fragments and
some number of parity fragments before it is written to persistent storage. Those fragments are then
placed across multiple physical devices. The original data is readable by accessing a subset of the
fragments. Single parity RAID implementations use a single parity fragment, and dual parity ones use
two, with the increase in parity fragments resulting in an ability to recover from multiple simultaneous
failures without data loss. Erasure coding, which is broadly used with distributed object stores, is very
similar to RAID but is usually used to refer to protection schemes that use a higher number of parity
fragments than RAID and offer the ability to tune resiliency by varying the number of parity fragments.

RAID implementations in broad use have evolved from single to dual parity approaches within the past
decade or so, often because disk rebuild times had grown so large as HDD capacities increased. The
concern with single parity RAID implementations and high-capacity drives with long rebuild times is the
possibility that a second failure could occur during the recovery period, thereby putting data at risk.
Flash media promises to increase device-level storage capacities by roughly an order of magnitude
over what HDD devices are capable of, but flash throughput of 10x to 100x more than HDDs helps
reduce device rebuild times. Still, almost every AFA offers at least a dual parity RAID option, providing
extra confidence in protecting data integrity and availability in the wake of device failures.

RAID adds the overhead of reading and writing to multiple physical devices for every write, and it
consumes additional storage capacity. For example, in a RAID 6 (4 + 2) protection configuration, there
are four data fragments and two parity fragments. The original data can be read by accessing all four
data fragments, or any three data fragments and either of the two parity fragments. If each fragment is
the same block size, the capacity overhead associated with this RAID 6 configuration is 33.3% (6
fragments are written, two of them are redundant [the parity fragments], and 2/6 = 33.3%). Because of
the increasing speed of processors and low-flash latencies, the processing overhead associated with
writing to multiple devices is negligible, and AFA vendors have come up with other multiparity RAID
implementations that introduce less capacity overhead (e.g., 8 + 2 or 14 + 2 RAID configurations

©2017 IDC #US42464717 10

instead of 4 + 2). Spreading data across more devices allows the use of smaller fragment sizes, so the
actual capacity consumed by the parity fragments ends up being less (e.g., 10% in the 8 + 2 RAID
example versus 33.2% in the conventional RAID 6 example). Assuming there are enough storage
devices internal to an array to allow each fragment to be written to a separate physical device (which
there generally are in enterprise arrays), these approaches are more efficient in terms of capacity
consumption yet still provide protection from multiple simultaneous device failures.

With the cost of raw flash capacity still at a premium over HDD capacity, the use of more space-
efficient RAID implementations that do not compromise protection are in broad use among AFA
vendors. Enterprises should check the available RAID options on AFAs, ensuring that they understand
the capacity overhead implications, and should plan to use at least a dual parity RAID implementation
in production, particularly when storage efficiency technologies that rely on data redundancy are in
use. As the use of higher capacity flash devices becomes more widespread, it also pays to understand
the rebuild times. Spreading the data fragments across more drives improves recovery performance as
it allows the throughput of multiple devices to be brought to bear when moving data. Recovery
mechanisms that just rebuild the lost data (rather than the entire failed device) are more efficient and
recover faster, minimizing the time data may be at risk during rebuild operations. They also tend to end
up writing less data to complete rebuild operations, an outcome that supports higher flash endurance
and reliability.

Future Outlook
With flash addressing many of the original performance concerns that existed with storage efficiency
technologies in HDD-based systems, end users are no longer shy about deploying them in latency-
sensitive primary storage environments. As AFAs become the mainstream general-purpose enterprise
storage platform over the next two to three years, the use of technologies like compression,
deduplication, thin provisioning, pattern recognition, write minimization, and delta differentials will only
increase. Across all AFA vendors running mixed workloads, the net data reduction ratios are in the 4:1
to 6:1 range today (although it is more like 2:1 in relational database-intensive environments and
higher than 6:1 in VDI, test and development, and other areas where high numbers of snapshots [both
read only and read/write] are in use). Net data reduction ratios are not expected to increase much in
the next few years, but the impact of their use is expected to be felt across a broader set of primary
storage workloads.

Another macro trend IDC sees in the enterprise storage market is a gradual migration away from
scale-up architectures toward scale-out architectures. Although scale-out architectures originally
gained traction in more technical computing markets, vendors have created enterprise-class versions
of these platforms that meet stringent requirements for balanced scalability, high availability and rapid
recovery without data loss, and nondisruptive operations. There are a number of enterprise storage
platforms built around scale-out designs that deliver "five-nines plus" availability, as measured across
their entire installed bases over time through telemetrics, so most customers have scale-out options
available to them to meet even the most stringent storage requirements. Because scale-out designs
offer options to scale both performance and capacity over time, it is a particularly interesting storage
architecture when a number of data services that impose overhead like storage efficiency and
encryption technologies are in broad use. Compute overhead increases as these data services must
be administered across additional storage capacity, but the ability to add performance as systems are
scaled can help ensure balanced scalability without performance degradation.

©2017 IDC #US42464717 11

ADVICE FOR THE TECHNOLOGY BUYER

Storage efficiency technologies have matured along with the increased penetration of AFAs for
primary storage workloads and should be available for use on any AFAs that will be used in production
environments. Features like compression, deduplication, thin provisioning, pattern recognition, write
optimization, and delta differentials do have some overlap, but they should all be used together
simultaneously to produce the highest net data reduction ratios. Different applications respond
differently to each of these technologies, but any enterprise that plans to run mixed workloads on their
AFA needs to have access to all of these. More powerful processors and extremely low-flash latencies
generally offer the option to engage all of these technologies simultaneously without performance
degradation, and let the system itself tease out the best data reduction ratios from each application.
The best data reduction ratios will be produced across the entire mixed workload when they are all
engaged simultaneously. Table 1 provides a framework for evaluating storage efficiency technologies
in enterprise-class AFAs.

TABLE 1

Storage Efficiency Evaluation Framework

Storage Efficiency Technology Key Features to Look For

Compression  Lossless (instead of lossy) compression

 General purpose (not specific to one application)

 Can be used in conjunction with encryption and other data services

Deduplication  Subfile-level chunking

 Global rather than single node implementation

 Both fixed or variable chunking acceptable

Thin provisioning  Should simultaneously deliver fast creation, space efficiency, and high
performance

Pattern recognition  Zero block elimination

 Identifies the difference between moving volumes or just data

 Vendor-specific pattern identification and optimization

Write optimization  Write-back cache that enables write coalescing and reordering

 Identification/removal of cache writes that do not need to be persisted

Delta differentials  Flash-optimized, space-efficient implementation

 Copy on write or redirect on write, depending upon prevalent workflows

 Copy data management orientation for broader snapshot utilization

 Space-efficient, snapshot-based replication as an option

Source: IDC, 2017

©2017 IDC #US42464717 12

There are several cases where storage efficiency technologies should not be used, including when
proscribed by compliance, regulatory, or privacy laws, when they generate a noticeable performance
degradation, or when they limit the use of other features. There still are some AFAs in the market
where other data services, like encryption, cannot be used without restriction when certain storage
efficiency technologies are in use. This is a limitation that can be avoided by confirming the ability to
use data services simultaneously up front, and staying away from systems that impose these types of
limitations where that can have a negative impact on anticipated workflows.

For primary workloads on all-flash systems, all storage efficiency technologies should operate in line.
Enterprises should take advantage of vendors' data reduction guarantees and monitor the achieved
ratio over time to ensure compliance, with the understanding that it will vary over time. Certain
workloads that see significant early savings from the use of thin provisioning and space-efficient
snapshots may see a degradation of that space savings over time as provisioned volumes start to fill
up (with writes) and snapshot targets begin to diverge more from their sources (again, due to
increased writes).

Suggested Recommendations
 When legacy enterprise storage platforms come up for technology refresh, consider
enterprise-class AFAs that demonstrate a proven track record for delivering "five-nines plus"
availability in the field.
 Designate a full complement of storage efficiency technologies high on the list of purchase
criteria, since that will provide the most flexibility for accommodating both evolving and new
workloads (that may not have initially been marked for AFA deployment), and a combination of
all technologies working together will result in higher data reduction ratios; administrators
must, however, be conscious of any noticeable performance impacts that may occur and
ensure that they can selectively turn off any storage efficiency algorithms whose overhead is
not justified by the achieved data reduction ratio.
 Inquire into the ways various vendors have flash optimized their enterprise storage platforms,
looking for designs that enhance storage performance, availability, space utilization, reliability,
and flash endurance.
 Explore the ways that scale-out architectures and RAM-based metadata processing impact
performance, scalability, and overall cost when storage efficiency technologies are in use.
 Plan to use at least dual parity RAID implementations for data protection that are flash
optimized for low-capacity consumption, and evaluate other array features that impact overall
availability like resilience, redundant, online replaceable components, recovery (including
rebuild times and array self-healing characteristics), and fault domain size.
 Take advantage of data reduction guarantees from the AFA vendors, making sure to
understand up front exactly what the vendors' definitions of "data reduction ratio" mean, and
monitor your actual ratios to ensure that your vendor meets their commitments to you in this
area.
 In evaluating overall data reduction ratios, understand where two different storage efficiency
technologies may cancel each other out; for example, in a VDI environment that uses linked
clones, deduplication may not add much to the overall ratio because the linked clone has
already effectively removed most of the data redundancy, but it will provide some added value
with redundancies in other data so it should still be used (but with adjusted expectations).

©2017 IDC #US42464717 13

LEARN MORE

Related Research
 HPE Hits a Home Run with the Acquisition of Midmarket Enterprise Storage Vendor Nimble
Storage (IDC #US42389717, March 2017)
 Worldwide All-Flash Array Market Shares, 3Q16: Top 5 Market Share Holders Consolidate
Gains (IDC #US42374817, March 2017)
 Evolving Deployment and Usage of All-Flash Arrays (IDC #US42267217, February 2017)
 The Evolving All-Flash Array Market (IDC #US42256317, January 2017)
 IDC's Worldwide Flash in the Datacenter Taxonomy, 2017 (IDC #US42214317, January 2017)
 End-to-End NVMe-Based External Enterprise Storage Makes Its Debut (IDC #US41952816,
December 2016)
 All-Flash Array Adoption: "Start Focused and Expand" (IDC #US41915616, November 2016)
 Market Analysis Perspective: Worldwide Storage Systems, 2016 — Growing Demand for
Software-Defined, Flash-Accelerated Persistence (IDC #US40357815, September 2016)
 Flash Media Packaging Decisions Secondary to System-Level Considerations in the All-Flash
Array Market (IDC #US41735716, September 2016)
 FlashBlade: Putting File- and Object-Based Storage Vendors En Garde (IDC #US41660216,
August 2016)

Synopsis
This IDC Perspective provides a framework for evaluating storage efficiency technologies in all-flash
arrays (AFAs). There are a number of different approaches in use, and this document defines the most
popular ones in enterprise storage, discusses how they each contribute to the overall efficiency of
storage capacity utilization, identifies related considerations, and provides a general framework for
evaluating the applicability of different options and implementations for various workloads. This
framework is not specific to a single vendor's products and can be used to evaluate different storage
efficiency technologies from any vendor of AFAs. This document is written for a technical storage
administrator audience.

"As budget constraints continue to pressure IT organizations, the use of inline storage efficiency
technologies is becoming a must have for production workloads of all types," said Eric Burgener,
research director, Storage. "The use of these technologies with mixed primary storage workloads
delivers the most benefit when run on primary flash AFAs designed as general-purpose platforms for
mixed workload consolidation."

©2017 IDC #US42464717 14

About IDC
International Data Corporation (IDC) is the premier global provider of market intelligence, advisory
services, and events for the information technology, telecommunications and consumer technology
markets. IDC helps IT professionals, business executives, and the investment community make fact-
based decisions on technology purchases and business strategy. More than 1,100 IDC analysts
provide global, regional, and local expertise on technology and industry opportunities and trends in
over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients
achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology
media, research, and events company.

Global Headquarters

5 Speen Street
Framingham, MA 01701
USA
508.872.8200
Twitter: @IDC
idc-community.com
www.idc.com

Copyright Notice

This IDC research document was published as part of an IDC continuous intelligence service, providing written
research, analyst interactions, telebriefings, and conferences. Visit www.idc.com to learn more about IDC
subscription and consulting services. To view a list of IDC offices worldwide, visit www.idc.com/offices. Please
contact the IDC Hotline at 800.343.4952, ext. 7988 (or +1.508.988.7988) or sales@idc.com for information on
applying the price of this document toward the purchase of an IDC service or for information on additional copies
or web rights.

Copyright 2017 IDC. Reproduction is forbidden unless authorized. All rights reserved.