Emerging Software driven IVR Tuning

Emerging, Software-Driven IVR Tuning

Provided by Greg Borton, Listening Methods June, 2012 Some companies pursue IVR Tuning exercises once, twice or even four times a year, especially if they use speech recognition in their Voice Self-Service (VSS) solutions. Others leave their solutions alone for multiple years often they have DTMF-only, relatively static implementations. Complex speech Voice Self-Service solutions, especially if they evolve, need to be tuned to help improve or maintain VSS performance and customer experience. The movement towards natural language dialogues further drives the need to monitor solution performance, in some cases, justifying a continuous improvement process. This need is further emphasized when there are monthly or seasonal changes in the types of calls made to an organization. Emerging Software-driven IVR Tuning significantly improves the business value over traditional, consulting-based tuning engagements. This new approach provides solution-level and in-depth performance analyses, prioritizes issues, automatically measures negative customer experiences, and enables efficient, in-depth utterance analyses to support grammar improvements. More importantly, it uses these insights to find immediate opportunities to quickly improve VSS solutions. The analysts and consultants are better able to focus on the most important opportunities, recommend improvements to grammars better than before, and achieve insights that were not available in the past.

Current Approaches to IVR Tuning

Current IVR tuning projects reveal their recommendations via a consulting-based analysis that focuses on the top fifteen to twenty-five prompts by volume. Typically: 1. The analysis is often supported by basic reports that identify prompt volume, opt-outs and choices made by callers (for instance, the Nuance Analytics Reporting (NAR)). 2. Consultants listen to or transcribe fifty to several hundred randomly selected calls to learn more about caller interactions with the high volume prompts. 3. They may compare the call audio audit results with the grammar, with examples of utterances that are not working. 4. Recommendations for improving prompt performance, for instance, re-wording of queries, re-ordering of options on a menu prompt, removal or addition of an opt-out, or modification of the confidence level, are presented in Word or PowerPoint documents.

Provided by Greg Borton, Listening Methods greg.borton@listeningmethods.com; 415 623-8646

Emerging Software driven IVR Tuning

What is Software-driven IVR Tuning?

A traditional Tuning Engagement reveals and recommends a number of actionable insights. The purpose of Software-driven IVR Tuning is to super-power this process and the analysts. It supports higher value insights based on richer, automatically generated data, supported by query tools that sort among tens of thousands of calls to find the right calls for audits. As such, Software-driven IVR Tuning refers to the use of software-based, in-depth VSS analyses that fully integrate endto-end call audio with softwaredriven insights into callers interactions with the IVR to support analysts.
Prompt 1 Prompt 2 Queue

Agent

Survey

Response

Agent Request

Caller Response

Software-driven IVR Tuning

Mine audio to capture IVR events and caller choices Software analyzes the results to reveal solution and prompt level performance, issues and customer experience Dashboards show results and support queries to select book-marked call audio Ranking focuses analyst on high priority issues Analyst gather several thousand utterances to better improve grammar

If the call audio is used as the sole data source for Softwaredriven IVR Tuning (ie, the software automatically finds prompts and analyzes their performance simply from processing audio), there is no need to change or update any IVR software, logs or other portions of the IVR solution in order to capture the data necessary for the analysis. Such solutions may be installed in less than a day, providing in-depth results in a few weeks.
Get Statistically Accurate Data To: Measure VSS Solution & Prompt Performance Rank and Prioritize Prompts by Multiple Factors Select the right calls for listening Select the right calls for Utterance Analyses
Software-Driven IVR Tuning

Traditional IVR Tuning

50 to 300 Calls

Database details and recordings of 50,000 to 80,000 Calls

Analysts use the dashboards offered by such solutions to identify performance, prioritize and rank their focus on the most important issues, quickly capture utterances to support updates in prompt grammar. They also can identify other opportunities for improvement by listening to selected calls as well as multiple instances of a selected prompt across many calls. The Software-driven IVR Tuning provides the analysts with the tools needed to enhance their ability to identify opportunities for VSS improvement.

Provided by Greg Borton, Listening Methods greg.borton@listeningmethods.com; 415 623-8646

Emerging Software driven IVR Tuning

Empower Analysts to Increase the Benefits of Tuning at the Same Cost

The table outlines the additional benefits available to the traditional IVR tuning approach when enhanced with Software-driven IVR Tuning. As a note, the costs and time-frame of Software-driven and traditional IVR Tuning are similar.

Traditional IVR Tuning Performance Analysis

Software-driven IVR Tuning

Prompt level performance for Prompt level performance for the top 100+ Top 15 to 25 Prompts prompts Overall VSS Solution Level Performance Customer Experience Performance N/A Allocates Costs of Planned and Un-Planned Transfers to Agents for the overall VSS solution as well as for each Prompt Volume Agent Costs incurred via Planned or Un-Planned Transfers Callers Desire to Self-Serve, with a focus on Strong Desire AND Un-Planned Opt-Outs for immediate benefits Negative Customer Experience Events

Using Cost of Caller Transfer to Agent for Ranking Ranking Prompts by Caller Behavior for Focus

Volume

Call Audits

50 to 100 randomly selected calls

50,000+ calls available for audit via in-depth query tools; 100+ select calls may be fully audited. Caller Interactions with selected prompts from several thousand calls provides deep data.

Grammar Analysis

Yes, based on observations or Yes, grammar recommendations based on auditing on transcriptions of several 2,000 to 3,000 utterances for the key prompts hundred utterances selected via ranking Varies from none to impact at prompt success levels Few or No ROI or Cost Saving Estimates Yes A Word or Power Point document $25K to $75K Estimate of Impact on Prompt-level and Overall Solution Performance Estimates of Agent Cost Savings, Agent Headcount Savings and Overall ROI Yes Dashboard with all data, ability to listen to example calls and summary Power Point presentation $45K to $75K

Identifying Cost & Customer Experience Impact of Findings

Recommendations Presentation of Findings Cost of Tuning

Focus Analysts on the Most Important Opportunities via Ranking

Current tuning engagements focus on prompts with the highest volume, but this may not be the most important approach for achieving the business goals. With Software-driven IVR Tuning solutions that use end-to-end recorded

Provided by Greg Borton, Listening Methods greg.borton@listeningmethods.com; 415 623-8646

Emerging Software driven IVR Tuning

calls as their data source, analysts are able to prioritize prompts using additional data elements than just volume. Among the options for ranking, two are of high interest: 1. Agent Costs Incurred Upon Un-Planned Transfers: This provides the ability to rank prompts by the agent costs incurred when callers initiate transfers to agents from each prompt. Of particular interest are UnPlanned transfers. These transfers are not intended by the VSS solution designer, so when the caller initiates a transfer by requesting an agent (perhaps because the prompt has already failed once, captured incorrect information or the caller is confused) this indicates that the VSS solution has failed. The ranking of the prompt is assigned using the actual cost of agent talk time across all the calls in which this occurs. 2. Callers Desire to Self-Serve: Here, Software-driven IVR Tuning solutions evaluate how hard callers are trying to use Voice Self-Service. The solutions rank prompts by finding the those with the highest volume of Un-Planned Opt-Outs to agents when the callers also exhibit the highest Desire to Self-Serve. This focuses analysts on those prompts that, if improved, will almost immediately lead to agent costs savings (due to a reduction in Un-Planned Transfers to Agents) and reduce the negative customer experiences.

Analyst Priorities are Very Different if Reducing Operating Expenses is the Most Important Goal
Prompts Ranked by Usage Volume
1 2 3 4 5 6 Language Selection Greeting & Main Menu Caller Selection Type Support Menu Customer ID Request ZIP Code Request 98.7% 98.3% 86.1% 43.8% 41.2% 36.7% 1 2 3 4 5 6

Prompts Ranked by Cost of Un-Planned Transfers to Agents

Birth Date Request Billing Phone # Request Order Number Request Service Type Menu Invoice # Request Customer ID Request $657,239 $431,006 $405,768 $390,205 $367,984 $350,061

Getting Adequate Utterance Data for Improving Speech Recognition Grammar

The key data needed for improving grammar is an adequate sample of utterances made by callers when interacting with each prompt. As stated by Nuance, Of the two tasks, predicting and encoding [the grammar], predicting the set of responses is by far the more difficult. Even if you are just expecting a simple yes/no response, you will likely get a wide range of responses from real callers, such as yeah, yup, no way, correct, and others that you might not guess in advance (Nuance Speech Recognition System Version 8.5 Grammar Developers Guide).

Provided by Greg Borton, Listening Methods greg.borton@listeningmethods.com; 415 623-8646

Emerging Software driven IVR Tuning

If one uses a complex natural language menu prompt that supports more than several responses, it is important to capture upwards of several hundred or more utterances. They then need to be organized to reveal percent usage, core terms, and variations surrounding the core terms. Then VUI designer needs to decide on which core and filler portions of the grammar to implement, and then apply weight to the options in order to optimize the speech recognition engines ability to optimally choose the most likely option identifying the response of the caller. The more data the VUI designer has, the better they are able to do their job. Example: Prompt-focused Audio Playback across Multiple Calls for Efficient Utterance Analyses

An innovation provided by Software-driven IVR Tuning is the ability to choose a prompt and then quickly listen to the use of the prompt across many different calls. This enables the analyst to capture several hundred utterances in the matter of a morning. Without such tools, one is relegated to listening to random calls and hoping to find the prompt of interest in some of the calls. If the prompt is not one of the top several by volume, it becomes difficult or impossible to capture enough utterances to adequately optimize a complex menu. Conclusion Emerging Software-driven IVR Tuning tools enable voice self-service analysts to provide in-depth analyses of the IVR solutions with a sharper focus than available with the current approaches to IVR tuning. Utterance analysis for improving speech recognition is faster, offering statistically valid samples for improving grammar. Lastly, linking the IVR analytics with the actual call audio, via book-marked call playback available in dashboards enables insights simply not available. The cost and duration of the IVR projects is similar, thus increasing the effectiveness of these projects and providing a higher ROI.
Provided by Greg Borton, Listening Methods greg.borton@listeningmethods.com; 415 623-8646