When Bad Things Happen

to Good Businesses:
Moving from DIY Disaster Recovery
to Unfailing Business IT Resiliency

Table of Contents
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Changing Face of IT Resiliency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Are You Being Lulled into a False Sense of Security? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Where DIY Disaster Recovery Falls Short . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Its Not About Disaster Recovery Its About Business IT Resiliency. . . . . . . . . . . . . . . . . . . . .

11

Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

2 When Bad Things Happen to Good Businesses

Introduction
Whether youre in charge of your organizations
infrastructure or are responsible for minimizing
risk throughout the environment, IT infrastructure
resiliency is likely top of mind.
This E-book explores the costs and impacts
of unexpected business disruptions, and the
tendency for businesses to rely too heavily
on do-it-yourself (DIY) recovery methods. It
illustrates why the DIY approach often falls short,
and the consequences when DIY recovery fails.
It also outlines in what ways DIY solutions differ
while explaining other ways to ensure business
technology resiliency.

3 When Bad Things Happen to Good Businesses

The Changing Face of IT Resiliency

An Aberdeen survey of IT professionals in 2013
found that the average cost of downtime per hour
across companies of all sizes was a staggering
$163,674.1 While the cost of downtime often
gets lots of airplay, a number of other factors
are driving the need to make business IT
resiliency a priority. This includes numerous new
business realities and trends, such as the shift
to cloud computing, virtualization and mobility,
to name a few. The uptick in adoption of these
technologies means business users are now
accustomed to instant responses and data access,
and are less forgiving of latency or downtime. They
demand predictable outcomes when it comes to
the availability of applications and data.

Yet organizations are taking longer to recover

from disasters, which directly translates into lost
revenue. According to a study conducted by
Forrester and Disaster Recovery Journal, in 2013
median actual recovery times were 8 hours, up
from 3 hours in 2010.2
We know you grasp the importance of disaster
recovery. Sixty-six percent of IT decision makers at
enterprises rated improving BC/DR a critical or high
priority for 2014.3 The question is: Are you taking
the best approach to business IT resiliency?

Aberdeen Group, Downtime and Data Loss: How Much Can You Afford?, August 2013
DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014
3
Ibid
1
2

4 When Bad Things Happen to Good Businesses

Are You Being Lulled into a False Sense

of Security?
Organizations like yours have no choice but to
invest in IT recovery. Your stakeholders, customers
and employees are completely unforgiving of any
downtime they simply expect their apps and
systems to work at all times. But are you making
the right investments and taking the right approach
to ensure true resiliency?
Easy availability of computer resources such as
co-location and simple virtual machine recovery
backup tools have made disaster recovery
appear so easy that organizations sometimes
believe that alone is enough for business resiliency.
In fact, many CIOs are investing significant funds
to build secondary data centers to serve as a
replicated site, making co-location the second
highest area of investment for disaster recovery.4
While these technologies offer great promise and
have made life a little easier when it comes to
recovery, heres the problem:

Ibid

5 When Bad Things Happen to Good Businesses

None of these tools perform application or

infrastructure discovery to find the shadow
or forgotten IT platforms where critical
applications reside.
None of these approaches reveal the complex
interdependencies between hundreds of
applications that run the business nor
do they subsequently address the process
expertise needed to orchestrate recovery of
environments on disparate tiers.
None of these approaches handle the people
and staff impact. Specifically, can IT folks get
to work at that second data center in case of
an outage?

The result? Business processes stall and

companies come to a halt because there isnt
consistent recovery performance to bring all critical
applications, data and infrastructure back online
rapidly enough.
Knowing the facts weve just presented, are
you still confident about your organizations
resiliency strategy? Can you really meet your
defined RTO and RPOs? Often, defined RTOs are
vastly different than achieved RTOs.

Figure 1. Readily available technologies and compute

resources do not equate to business resiliency.

HA
Architecture

Easy access
to compute 1153 colo
providers

Even if you manage to achieve a stringent

SLA of say, a six-hour RTO, it means nothing if
stakeholders expect or assume a much faster time
frame given the real risks or needs of the business.

Second
Datacenter

Business
Resiliency

Load balancing
capability
with flexible
resource pools

Figure 2. The majority of companies receive a failing

grade for their disaster recovery program.

2%
9%
16%
51%
22%

6 When Bad Things Happen to Good Businesses

Where DIY Disaster Recovery Falls Short

As much as organizations like to think that disaster
recovery is only needed in the event of disasters,
such as hurricanes, storms, floods or man-made
malicious attacks, those account for a small
percentage of downtime. As shown in figure 3, the
top causes of downtime are largely man-made
or operational in nature.
Many businesses attempt to handle recoveries
on their own. But planning, implementing and
executing a recovery is complicated. Heres why
many DIY disaster recovery attempts fail:

Misconception. A misperception that the cloud

makes disaster recovery easy. Enabling full
recovery of all business processes end-to-end
requires far more than storing and accessing data
in the cloud. Every application relies on the network,
firewalls and load balancers. If youre running an
application and replicating it to the cloud, all of the
other components of that application stack must be
replicated to ensure a complete recovery. Otherwise,
you can recover your infrastructure and data, but not
your entire business.

Figure 3. The top causes of downtime are man-made or optional in nature.

60%

56%

44%

29%

26%

Human Error

Unexpected Patches
and Updates

Server Room
Environment Issues

Power Outages

Onsite Disasters

DRJ/State of Disaster Recovery Spring 2014 Survey

7 When Bad Things Happen to Good Businesses

67% of disaster
declarations are
caused by either
operational failures or
man-made mistakes.
- Forrester/Disaster Recovery Journal,
November 2013 Global Disaster Recovery

Complexity of hybrid IT environments and

change management. Most companies rely
on multiple operating systems and its hard to
recover such an environment. For the most part,
IT managers are on top of their game managing
integration technology to make all systems work
together smoothly at top efficiency. But ensuring
business continuity and disaster recovery for a
hybrid environment is much more complex and
risk-laden.
To reliably recover a hybrid environment and
your applications and data you need to replicate
the precise mix of servers, storage, operating
systems, hypervisors, networks and software
versions. And that means keeping abreast of
any and all changes to that mix at every moment
in time. After all, business applications are
interdependent on one another the unavailability
of even one system or application can trickle down
to impact many business processes.

8 When Bad Things Happen to Good Businesses

Understanding true costs. Dont know what

you dont know and costs quickly spiral out
of control. Sungard AS has found that 70% of
its customers lack up-to-date configurations and
40% have gaps in configurations. That means they
have a discovery problem because they cant be
certain about the make-up of their environment.
This lack of clarity can lead to big expenses. Think
about it this way: Costs associated with disaster
recovery span recovery infrastructure, backup,
labor and additional services. You can fairly easily
predict labor and backup costs. The cost for
additional services, such as consultants, is hard to
predict, but tends to be relatively small. But costs
for recovery infrastructure can be significant. At
the same time, they can vary greatly company
by company, so its not really possible to use a
benchmark as a predictor.

Recovery starts with

good data, but good
data requires reliable
backups. More than
40% of customers fail a
disaster recovery test
due to poor backups.
- Sungard AS Customer Survey

Lack of personnel and expertise. Many

enterprises think they can design a few tools
for disaster recovery and theyre all set. But
they forget the human cost and effort to recover the
entire business environment. For instance, your IT
manager may need to order equipment, but not be
able to get it delivered for a few days.

Or your organization may not have the right

person in-house to properly conduct a disaster
recovery test. In other words, it takes quite a bit of
coordination of technology, people and processes
to enable effective disaster recovery. And many
companies simply cant pull this off.

Virtual Machine Deployment

The ease of virtual machine deployment can contribute to one of the biggest
challenges in recovery. VMs can be created and deployed more rapidly than
change management processes can verify that all of the latest production
changes made it into the recovery environment. Even if everything is
identical, a bare-metal restore from a cloud provider can play havoc with
recovery time objectives.

9 When Bad Things Happen to Good Businesses

A summary of the DIY disaster recovery costs that

many companies commonly overlook:
Accounting for the additional burden that DIY
measures place on internal disaster recovery
and IT teams, including costs for travel time,
additional coordination, and the orchestration
required for the various platforms.
The cost and steep step-function associated
with storage and infrastructure lock-in at a
third-party secondary site.

A false sense of security due to overinvestment. Once they decide to go the DIY route,
many organizations over-invest in co-location, high
availability architecture and lots of technology. But
without the right people, processes and preparation
including testing the recovery environment
multiple times per year all of this investment is
for naught. And 23% of organizations admit they
never test.5

The operational costs if a recovery is delayed

or fails completely during a disaster event.
IT isnt given enough budget to address the
risks associated with disasters. Demands for
business uptime are on the rise, but the reality is
that IT staff is typically dedicated to managing core
business processes and applications; its secondary
responsibility is disaster recovery. In other
words, the majority of IT resources are focused
on the operating environment, and a very small
percentage to disaster recovery.

DRJ and Forrester BC/DR Market Study: The State of DR Preparedness, March 2014

10 When Bad Things Happen to Good Businesses

66% of IT decision
makers at enterprises
rated improving BC/
DR a critical or high
priority for 2014
but only 5.8% of IT
operational and capital
budget is allocated to
BC/DR.
Disaster Recovery Journal and Forrester
BC/DR Market Study: The State of DR
Preparedness, March 2014

Its Not About Disaster Recovery

Its About Business IT Resiliency
Ultimately, business stakeholders want all
employees to be productive and 100% of
applications running with as little interruption as
possible. However, disaster recovery is only one
aspect of business IT resiliency.
To ensure true resiliency, your company must
take disaster recovery to the next level by
leveraging the DR environment for testing
and development. By doing so, you continually
improve your IT infrastructure, ensure employee
productivity, and maintain customer satisfaction for
the sake of revenue growth.
In other words, what matters most is ensuring that
your business is conducting business, not just
performing recovery. Lets illustrate what this looks
like in the real world.

The Case of the Fragmented

Insurance Carrier
Without the proper disaster recovery processes
in place, a company can come to a halt, affecting
not only routine tasks, but putting its reputation on
the line for future business. Business IT resiliency
depends on addressing disaster recovery, but it
does much more.

11 When Bad Things Happen to Good Businesses

Over a period of 25 years, an insurance company

specializing in commercial and personal insurance
grew exponentially by making multiple strategic
acquisitions. Its IT environment grew along with
all the acquisitions. The company now manages
over 430 physical and virtual machines running a
mix of Oracle enterprise applications and custom
vertical applications.
Its business processes, such as processing
claims and claim adjustments, must all work in
concert to ensure accuracy and consistency
among all of the different platforms and
databases.That means its recovery plan
must address multiple applications and
data in a diverse IT infrastructure.

When the company was assembling a recovery

team in response to a natural disaster, a debilitating
DDoS attack brought down its commercial account
web portal. This portal housed customer data,
including client lists, SSNs, financial data and credit
scores. The inability to view client data in the field
made it impossible for the company to conduct
business as usual.

The Case of the Stormy

Distribution Company
How quickly critical applications and data can be
brought back online can make a difference to your
bottom line. A large-scale fine foods distributor
in the Midwest was operating at peak efficiency
when a sudden storm snapped the power lines to
its data center for 18 hours. Even as the recovery
team worked under fierce pressure to bring all the
systems back online simultaneously, collateral
damage rolled up and down the supply chain as all
the just-in-time warehousing data went blank.

12 When Bad Things Happen to Good Businesses

Critical retail and supply chain applications went

down. ERPs and custom-built apps for Warehouse
Management, Forecast & Replenishment and
Order Entry were not working as needed. The
result? A great deal of perishable food didnt get
moved before spoilagewaste set in and the
company found itself with a huge revenue loss.
The lesson in these tales? Companies cannot
afford downtime they need zero business
process interruption. But you do not have to
shoulder the responsibility alone. With the right
partner, the right cloud technology and the right
managed solution, enterprises can tap into cloudbased recovery for always-on, always-available
true business resilience.

Conclusion: What to Seek in a RaaS

Partner & Solution
While RaaS is a viable and low-cost option for
addressing your disaster recovery needs, its
critical to thoroughly vet your options. Heres what
to look for in a solution and vendor.
Ask if the service provider:
Assesses your disaster recovery needs
Manages the lifecycle of your DR program
Suggests and implements continuous
improvements
Specifically, does it:
oo Develop applications to support system
mapping and prioritization?
oo Provide DR run books?
oo Build the appropriate systems?
oo Restore OSes and installing backups?
oo Restore data and initiate networks?
oo Execute startup/restore procedures for
applications and databases?

13 When Bad Things Happen to Good Businesses

oo Manage pre-test planning and coordination

and all DR testing teams?
oo Monitor the DR test?
oo Provide post-test reporting, gap analyses
and remediation plans?
oo Identify and implement production changes
impacting recovery?
oo Maintain all DR technical docs
oo and configurations?
oo Provide real-time access to and reporting
on all program elements?
oo Provide credible testimonials?
A lot is riding on your disaster recovery
capabilities in fact, your very business
depends on it. Moving from DIY to RaaS is a
smart move. Just be certain to choose the vendor
and solution that can guarantee predictable uptime.

About Sungard Availability Services

Sungard Availability Services
650 E. Swedesford Road
Wayne, PA 19087
as.marketingevents@sungard.com

Sungard Availability Services provides managed IT services, information

availability consulting services, business continuity management software,
and disaster recovery services.
To learn more, visit www.sungardas.com or call 1-888-270-3657

Phone: 1-(888) 817-0925

Trademark information
Sungard Availability Services is a trademark of SunGard Data Systems
Inc. or its affiliate used under license. The Sungard Availability Services
logo by itself is a trademark of Sungard Availability Services Capital,
Inc. or its affiliate. All other trade names are trademarks or registered
trademarks of their respective holders.

14 When Bad Things Happen to Good Businesses