Business Continuity and the Customer Contact Center

November, 2007

Contents
Executive Summary .................................................................................................................................... 1 The Business Continuity Imperative......................................................................................................... 1 Evolving from disaster recovery planning to business continuity planning.................................. 1 Balancing cost and risk........................................................................................................................... 2 Causes of downtime ............................................................................................................................... 3 The Genesys Approach to Business Continuity ...................................................................................... 3 Resilience at the Software Level ................................................................................................................ 4 Reducing operator errors....................................................................................................................... 4 Fault tolerance ......................................................................................................................................... 5 Resilient Deployments with the Genesys HA Suite................................................................................ 6 Hot standby ............................................................................................................................................. 6 Warm standby ......................................................................................................................................... 7 N+1 load distribution ............................................................................................................................. 8 Protection from Site-wide Failures............................................................................................................ 9 Summary..................................................................................................................................................... 10 About Genesys ........................................................................................................................................... 11

Copyright 2007 Genesys Telecommunications Laboratories, Inc. All Rights Reserved

Executive Summary
Business continuity is taking on new urgency in many businesses. As a critical process in the business, the customer contact center needs protection from downtime and outages. Genesys helps companies balance cost and risk when protecting customer contact centers from outages. Genesys offers a multi-layered approach to business continuity, offering protection from outages at the software, component, and site-wide levels. Genesys supports its customers business continuity efforts with flexible licensing, best practices consulting and implementation services. This paper outlines the Genesys approach to business continuity, with resilient software components deployed in redundant and flexible configurations. Specifically, it describes at a high level the Genesys High Availability solution platform, enabling redundancy of key components with warm or hot standby.

The Business Continuity Imperative

Businesses worldwide are increasing their focus on business continuity for a wide variety of reasons. For some, regulatory requirements for protecting vital data are challenging companies to examine disaster recovery and business continuity plans. Server consolidation and virtualization efforts intensify the need for careful planning by consolidating vital business assets. Increasing competition in many industries makes the need to remain operational in case of a disaster even more important for long-term business survival. The need for business continuity planning has reached the customer contact center. Once a necessary cost of doing business, most businesses now realize the strategic and competitive advantages of efficient, high-quality and proactive customer contact. As the value of customer contact to essential business processes escalates, so does the necessity for careful planning to protect these operations in case of problems or disasters.

Evolving from disaster recovery planning to business continuity planning Over time, companies have expanded their focus and expectations from merely being able to recover from a major disaster to withstanding problems, both large and small, without business interruption. Disaster recovery (DR) is the ability to recover from a major outage or problem that affects more than one component of a system. Disasters may be highly localized, such as flooding from a nearby riverbank, or widespread, such as a hurricane. DR plans balance factors such as how long it takes to recover and how much data loss is acceptable. High availability (HA) is the ability of a service or infrastructure to remain operational when individual component problems arise. For example, in a highly available call center operation, the failure of a single server should not affect the ability of the call center to operate, although it may slow responsiveness or increase wait times.

Business Continuity and the Customer Contact Center Page 1

The objective of business continuity is to identify the essential services that need to remain functioning even during a disaster, or as quickly as possible thereafter, to keep the business running. Business continuity includes HA and DR technologies, but extends beyond the technology itself to consider all of the essential resources to run the business, including people. Over time, the focus on protecting systems and data has shifted from disaster mitigation and recovery to application availability and business continuity.

High availability, disaster recovery and business continuity are different ways of looking at the same problem: how do you reduce the risk to your business of outages of vital systems? Balancing cost and risk The objective of business continuity is to protect the business but not at all costs. Not all services are critical to the survival of the business. For some business areas, an outage of a week or two is acceptable when balanced against the cost of mitigating measures to provide true continuity, without outage. In determining where and how to invest in high availability and business continuity, you must consider the financial costs of downtimethe risk to the business from outages. For the customer contact center, potential costs of downtime include: Loss of revenues: Business may be lost because transactions cannot be completed. Customer dissatisfaction/opportunity costs: A poorly managed outage can result in customer dissatisfaction and reduction in trust. Opportunity costs: Customers unable to use business services may turn to a competitor during an outage. Agent productivity: Agents cannot properly perform their jobs when systems are unavailable. Loss of information: If interactions are processed in an inefficient (paper-based) manner, the business may lose important data about the transaction. If data is not properly backed up, it may be lost in the event of a hardware failure or environmental disaster.

Business Continuity and the Customer Contact Center Page 2

Some companies are leveraging the customer contact center to improve overall business process efficiency, integrating customer contact in key business processes. For these companies, the cost of downtime is even higher, as downtime affects core processes. Genesys understands that organizations need contact center operations that are resilient to outages and problems, but that these solutions cannot add undue cost to the contact center operation. Causes of downtime To protect against downtime, it is useful to understand its causes. Planned downtime is typically scheduled for operational purposes such as upgrading facilities or operating systems. It can be scheduled to minimize business impact. Potential sources of unplanned downtime include: Hardware failure: disk and network failures, processor unit and memory failures, loss of power supplies or system cooling. Software failure: operating system, middleware or application failure. Environment: climate, temperature and natural disasters People: mistakes, misunderstanding the way a system operates. According to IEEE research, people account for 15% of all unplanned outages. Inadequate security: hackers / denial of service attacks. The potential causes of downtime seem to be growing daily: bird flu, terrorist attacks, theft, etc. And businesses with consolidated contact center operations and/or data center operations also consolidate their risk, and must take particular efforts to protect their business from all kinds of downtime.

The Genesys Approach to Business Continuity

Genesys recognizes the increasingly important role of customer contact in many businesses, and has taken action at several levels to help its customers with their business continuity requirements. Support for business continuity spans product development, licensing, consulting and best practices, and support. The Genesys 7 product suite supports contact center availability and business continuity at several levels. As a flexible, standards-based platform, it enables a large range of deployments for different business needs. Support for SIP connectivity enables businesses to leverage geographically distributed personnel to handle customer contact functions when regular agents are not available. It is easy to outsource contact center capabilities, or deploy remote facilities, offering protection from localized problems. Genesys supports flexible, highly-available deployments through licensing.

Business Continuity and the Customer Contact Center Page 3

Looking at the potential sources of downtime, the Genesys 7 software platform offers several layers of protection from failures and downtime. At the application level, Genesys 7 software is fault-resilient and has secure and simplified administrative interfaces. The software is designed to reduce the possibility of user errors (mistakes and misunderstandings), while providing continued operation in the event of internal and external faults. At the component level: Genesys offers High Availability (HA) versions of critical components of the Genesys infrastructure, operating under the control of Genesys Management Layer. These versions, enable flexible, resilient deployments that are capable of withstanding the failure of a component, such as a server hosting a critical function. At the site level: Genesys helps companies deploy call center architectures that can withstand the loss of an entire site through techniques like site duplication, remote contact centers, and deployment of SIP-based agents. The Genesys software design and QA testing methodology ensure a high level of quality of the Genesys software itself.

The remainder of this paper discusses these different levels of business continuity support in more detail.

Resilience at the Software Level

The Genesys commitment to business continuity begins with our end-to-end design philosophy and strategy that ensures overall system quality, reliability and expandability. Genesys approach relies on a well thought-out architecture, engineering and testing, because the most effective approach to failures is to prevent them from occurring. All Genesys solutions are submitted to intensive review and testing at each step of their development, enabling defects to be detected and resolved early in the product lifecycle. At a systems level, Genesys interaction management solutions operate on a unified, highly open and scalable architecture that ensures flexibility and modularity for changing with the environment , such as offering new value-added services, adding agents, adding hosts, modifying skills, or changing routing strategies. In addition, Genesys offers a single point of control from which to administer and configure all aspects of the interaction management system. This architecture supports the ability to add functionality to a production system without disruption. Reducing operator errors Statistics reveal that human errors are a significant cause of downtime. While these errors can never be avoided completely, Genesys software solutions minimize their likelihood. In addition to providing comprehensive user manuals, training, support and consulting services, Genesys has incorporated measures to prevent operator errors due to incorrect data entry or action.

Business Continuity and the Customer Contact Center Page 4

Centralized configuration and integrity rules Genesys provides centralized configuration of data processing and storage to reduce the possibility of configuration inconsistencies. Configuration data objects are set once and distributed to all components that require their data, eliminating duplicate data entry errors. The software applies data integrity checking to validate configuration data before it is applied. Configuration Import Wizards (CIW) further eliminate manual errors and unnecessary work. The CIW includes ACD upload features to ensure the consistency of the Genesys database with the configuration data objects in the PBX. User access control Comprehensive access control prevents unauthorized or unskilled operators from making changes that affect system stability. Permissions are granted based on users security profiles. Permissions to access or modify objects can be granted to groups or individual users. Process automation and interfaces The Genesys interfaces provide operators with a step-by-step guide to simplify operator interventions, removing a major source of outages. SQL scripts are also provided as a part of the installation, to automate the creation of tables in the database.

Fault tolerance Fault tolerance is the ability of a system to continue operations in the presence of a fault. The Genesys software is designed for fault resilient along several levels. Detection: The delay of communication protocols such as TCP/IP to detect and signal failure or communication defects in a distributed architecture presents a serious challenge to the availability of any interaction management system. For example, the TCP/IP stack may take several minutes to report a failure associated with a hardware problem, such as a failed computer or disconnected cable. Genesys has implemented its own protocol to detect failures rapidly. The Advanced Disconnect Detection Protocol (ADDP) periodically polls the opposite process in case there is no actual activity through a given connection. Exception recovery: A software exception is an interruption to the normal flow of a program that occurs in response to an unexpected condition or software error. If an exception is not caught by the application, the operating system terminates the application, affecting all solutions that depend on it. Genesys software is designed to catch exceptions and recover gracefully where possible, hence avoiding a failure of the application.

Business Continuity and the Customer Contact Center Page 5

Resilient Deployments with the Genesys HA Suite

Although resilient software is an essential first step, business continuity requires protection from broader component-level failures. Genesys offers High Availability (HA) options for critical platform components, including: Genesys Customer Interaction Management Platform Genesys Framework Inbound Voice Outbound Voice Genesys Info Mart e-services: E-mail and Genesys Web Media Genesys Open Media Genesys Self Service The Genesys HA product suite enables business continuity by supporting redundancy of critical components in the call infrastructure. Genesys takes a software-based approach to HA, by deploying multiple software components in redundant configurations. HA versions are necessary only for those components that track session state or customer data; the HA versions help defend against data loss and ensure data consistency in the case of component failures. Depending on the particular component and its role in the infrastructure, the HA version of the software operates with either hot standby, warm standby, or N+1 load distributed redundancy. For other components of the Genesys solution that do not hold interaction data, simplex (non redundant) operation is enough. For example, if a single agents desktop system fails, incoming calls are simply directed to other agents. While an individual call may be dropped, overall system availability is maintained. There are no specific HA versions of these components.

Hot standby In the hot standby configuration, the standby component duplicates the current state of the primary component. Genesys Management Layer software monitors the operating status of the primary and standby components. External clients maintain connections to both components, but only the primary component sends data to clients. If the primary component fails, the standby component takes over all interactions currently in progress with little or no loss of data, as well as handling new interactions.

Business Continuity and the Customer Contact Center Page 6

Maintaining synchronization between the primary and standby components is the key challenge of the hot standby model. Genesys provides hot standby for certain critical components, such as T-Server and SIP Server. An example of this is Genesys SIP Server which supports a hot standby HA configuration, with ongoing synchronization between two SIP servers. Both servers receive all data, but only the primary SIP Server interacts with clients. The Genesys Local Control Agent and Solution Control Server communicate to monitor availability and manage failover when needed. In case of a problem, the standby SIP Server becomes the primary. When the problem on the failed server is resolved, it is brought in as the standby server. There is no call loss and, in most cases, no loss of interaction data

Warm standby In the warm standby configuration, the standby server is initialized and ready to take over the operations of the primary server, but remains idle and does not perform any processing.

The Solution Control Server acts as a control point for starting, stopping and managing Genesys solutions. When it detects a failure, it changes the standby server to primary server status. The new primary server immediately begins processing all new requests for service. Clients that were connected to the failed server automatically reconnect to the standby (now primary) server.

Business Continuity and the Customer Contact Center Page 7

Note that in the warm standby model, a small number of interactions in progress at the time of failure may be interrupted, in particular those interactions that changed their state between the actual occurrence of the failure and the moment it was detected. Only the current state of the interaction may be lost; its existence and all data collected with respect to the interaction will be maintained. N+1 load distribution In an N+1 load distribution configuration, a pool of similar components operate in parallel to share the overall workload. Each component handles its own interactions independently of the others. Sufficient components are deployed such that if one fails, the remaining components still have enough capacity to handle the total workload.

For example, the GVP Communication Server is normally deployed in an N+1 configuration. The number of servers deployed is one more than would be dictated by the incoming traffic load. The IVR drivers distribute the workload between the available Communication Servers. If one server fails, the remaining servers are still able to handle the total workload.

The driver uses a round robin algorithm to balance load between the different (IVR) communication servers. Any failure will be detected within milliseconds, and the server will be removed from the round robin cluster. In this situation, the availability solution also enhances performance by providing integrated load balancing. This approach is more cost effective at the hardware level and is used with IVR systems where loss of a call is typically seen as an inconvenience.

Business Continuity and the Customer Contact Center Page 8

Protection from Site-wide Failures

Beyond individual components, businesses need to protect themselves from larger interruptions or failures that can affect multiple components. Examples of these problems include: Wide area network outages isolating one data center from the outside world. Localized data center outages (flooding, power problems) Regional disasters (hurricanes, tornados)

For contact center operations, organizations need to consider the possibility of failure to the site hosting the contact center (including agents), as well as the data center hosting the essential software and servers in support. The contact center and data center may be co-located at the same site, or may reside in distinct, separate locations. To protect themselves from site-wide problems, many organizations deploy multiple contact centers and/or data centers, with the ability to transfer operations to another center if one center should fail. The Genesys platform supports a number of different deployments to provide business continuity during a site-wide failure, with flexible licensing support for these approaches. SIP technologies within the Genesys platform provide a great deal of flexibility for customers addressing site-wide failures of contact centers and agent locations. For example, using SIP technologies, it is relatively easy to deploy remote contact centers, or leverage experts or homebased personnel as agents if primary contact center agents are unavailable. A distributed contact center architecture, using the Genesys HA software, can withstand the failure of a single site.

If a single remote contact center agent site fails, new incoming calls are sent to other agents as well.

Business Continuity and the Customer Contact Center Page 9

If a contact center site-wide failure happens while a call is underway, it is impossible to prevent interruption to the individual call. However, deploying multiple contact centers in different locations offers business continuity in the case of failure. Customers who call in after the site failure are automatically routed to alternate centers. Data center failures are another key concern. In some cases, customers may replicate the entire data center at a remote location. This approach offers the highest possible level of resilience from site-wide failure, but introduces additional costs in term of equipment and facilities.

Again, the trade-off between cost and capacity needs to be evaluated based on the impact to the business if loss to those services were to occur; the flexible, software-based approach offered by Genesys enables a wide range of options. Note that in a disaster situation, call centers may have the double disadvantage of fewer available agents and increased call volume. The IP remote agent capability of the Genesys platform helps companies bring additional agents online quickly, leveraging branch or remote experts or homebased agents that are not affected by the outage to temporarily handle the increased call volume.

Summary
At Genesys Telecommunications, we understand the pressing need to protect the business from call center outagesa need that is escalating as contact centers serve increasingly strategic roles in the business. Our software-based approach to business continuity provides the flexibility to balance cost and risk, call center requirements and business continuity needs. Our commitment to enabling business continuity extends to flexible licensing options, consulting and services support, and ongoing software design and implementation.

Business Continuity and the Customer Contact Center Page 10

About Genesys
Genesys, an Alcatel-Lucent company, is the #1 provider of contact center software and is the leader in open standards voice platforms. The companys integrated, 100% software, self and agent-assisted service solutions empower contact centers to end customer frustration while driving operational efficiencies and accelerating business innovations. Over the past 15 years, Genesys has pioneered the evolution of open contact center and customer interaction management solutions and is today the world leader in setting standards and pioneering the evolution of self service for global enterprises. With more than 3000 customers in 80 countries, including Global 2000 enterprises, government agencies and many of the worlds fastest-growing mid-sized businesses, Genesys directs more than 100 million customer interactions every day. Genesys has more than 300 technology, systems integration and value-added reseller partners worldwide. The companys impressive list of global partners, such as Accenture, IBM, Microsoft and Oracle, are leaders in their industries and enable Genesys customers to deploy tightly integrated solutions. For more information, visit our Web site at www.genesyslab.com, or call 1-888-GENESYS (4363797).

Business Continuity and the Customer Contact Center Page 11