Measuring What Matters:

Annual Customer Reference Program Benchmark Report

October 2010
Research by Mainstay Partners and the Metrics Special Interest Group

Mainstay Partners LLC 901 Mariners Island Blvd, Ste. 105 Research and analysis were conducted by Mainstay Partners, the leading
www.mainstaypartners.net provider of IT value assessments to enterprise IT organizations and high
San Mateo, California 94404-1592 technology solution providers. Mainstay does not warrant any results
650.638.0575ph 650.638.0578f derived from this study. Copyright © 2010 Mainstay Partners, LLC.

Table of Contents
Background
Key Findings
Profile of Respondents
Top Performance Metrics
Reference Materials
How References Are Used
Responsiveness to Requests
Customer Participation
Conclusion
For More Information
Abby Atkinson, chairperson,
Metrics SIG
abby.atkinson@infor.com
Amir Hartman, coordinator,
Metrics SIG
ahartman@mainstaypartners.net
1
BACKGROUND
1
2 About the Metrics SIG
In 2006, members of the Customer Reference Forum (CRF)—a community
4
of customer reference professionals representing some of the world's
8 leading corporations—formed the Metrics Special Interest Group (Metrics
SIG) to help companies of all sizes effectively measure, benchmark, and
10
communicate the value of reference organizations and assets. The Metrics
11 SIG includes representatives from Oracle, SAP, Microsoft, Intel, Dell, and
Infor.
13
To fulfill its mission, the Metrics SIG is currently working to:
14
 Develop a set of performance metrics that measure the effectiveness of
16
customer reference programs
 Conduct surveys to establish benchmarks for each performance metric,
document best practices, and uncover industry trends in customer
referencing
 Update performance metrics and benchmarks annually
 Regularly share the results with Metrics SIG members
 Discover future trends and goals for leading reference programs
About Mainstay Partners
Mainstay Partners is a leading provider of independent value assessment
and IT strategy services to industry leaders worldwide. For nearly a decade,
Mainstay has been delivering the evidence and insights businesses need to
make the most of their technology solutions. For more information, visit
www.mainstaypartners.net.
Survey Objectives and Methodology
CRF's Metrics SIG conducted the second annual survey of customer
reference programs in 2010, soliciting feedback from reference professionals
at 18 leading companies, all of them members of the Metrics SIG. The
members helped design and conduct the survey in association with Mainstay
Partners, a consulting firm with nearly a decade of experience in customer
reference and IT advisory services.
The survey results detailed here will be used to help reference program
managers benchmark their programs with those of their peers and share a
set of data and best practices. This second annual survey will also track
progress against last year and analyze trends in key performance areas. In
addition, the Metrics SIG plans to expand its survey to include more
programs within the Customer Reference Forum community.

The survey uncovered what
reference managers felt were the
most important indicators of
program success and the extent
to which companies measured
these performance metrics.
2
KEY FINDINGS
The survey paints a broad picture of customer reference programs operated by
companies predominately located in the U.S. and Europe. It describes a range of
operational characteristics and performance metrics, including productivity
(request fulfillment and collateral throughput rates), usage (how references and
materials are being deployed), and finances (budgets and sales impact).
This year’s survey reveals significant differences between programs—sometimes
but not always related to the size of the company—and lays the foundation for a
set of industry benchmarks to which companies can compare their own
performance. The survey also uncovers what reference managers feel are the
most important indicators of program success and the extent to which companies
actually measure these performance metrics.
Key Performance Indicators
 Survey respondents identified and ranked 10 key indicators of reference
program performance. In general, these metrics were related to expanding
the relevancy and coverage of reference programs. For example, programs
were concerned with having sufficient resources—annual budget and full-
time equivalents (FTE)—as well as increasing usage of and participation in
reference programs.
 The majority of programs are focused on “effectiveness” and “efficiency”
performance measures, compared to last year’s emphases on “output” and
“productivity.”
 Many of the key performance indicators, such as total contract value and
lead conversion rates with references, while judged to be critical, were still
not being measured by many reference programs.
What Reference Programs Are Doing Well
 Programs are very responsive to stakeholders, promptly fulfilling reference
requests. Over three-quarters of respondents reported fulfillment rates of
more than 89% within one week.
 Programs have maintained these high response rates despite reduced
budgets.
 Larger companies (>$1B) have increased their use of external FTEs to
leverage expertise in specialized areas such as writing and creative
services.
Areas Needing Improvement
 While programs are increasing production of multimedia assets such as
videos and audios, the demand for these assets continues to outpace
supply. Nevertheless, among the respondents answering in both the 2009
and 2010 survey, the number of videos produced increased by 72% over
last year, showing that program managers are taking significant steps to
make up the shortfall.
 Programs still have very limited visibility into the impact of references on
sales and revenue, with almost 84% of respondents unable to say how their
 3

reference activities affected sales and business performance. Less than half
even knew how often reference materials were included in bids.
 Average budgets and headcount, particularly for small and mid-sized
companies, have fallen, mainly as a result of economic conditions. To regain
resources, customer reference programs will need to integrate more
effectively with the sales process and demonstrate with hard numbers how
they are bringing value to the business.
 Demand for reference requests grew over last year, which meant that
reference program managers found themselves caught between the
pressure to contain costs and the still-rising demand for references.
Managers of reference programs need to find ways to keep up with the
growth for the demand for references while still allocating budget and time to
new initiatives such as integrating references with sales and acquiring new
reference customers.
 Customer reference programs need to do a better job of measuring their
overall value; this will require better defining the “soft” benefits they deliver,
such as enhancing customer relationships and protecting valuable customer
reference resources, as well as quantifying reference outputs more
rigorously.
 Once the mechanisms for measuring sales impact are in place, managers of
reference programs should consider tracking performance metrics such as
revenue impact (percentage of bids containing references, value of sales),
sales cycle acceleration, and percentage of references featuring top
customers (e.g., those that are most profitable and loyal).
 To increase visibility and value, reference program managers need to better
understand the business strategy and goals of key stakeholders, and then
translate that understanding into reference measures that count.
 4

PROFILE OF SURVEY RESPONDENTS

Company Demographics
The 2010 survey targeted leading companies within the Metrics SIG
membership, most of them representing high technology or related industries.
The majority were large global enterprises with more than 5,000 employees and
more than $1 billion in revenue.

Companies with more than $10
billion in revenue (28% of all
respondents) budgeted about $2
million on average for reference
programs, allocating nearly half
of that amount to outside
contractors.
5
Size and Budget of Reference Programs
Not surprisingly, reference program budgets varied according to the size of the
company. Companies with more than $10 billion in revenue (28% of all
respondents) budgeted about $2 million on average for reference programs—
down about 33% from $3 million in 2009—and allocated almost half of that
amount to outside contractors. Companies with $1-$10 billion in revenue (22% of
respondents) budgeted about $300,000 on average for reference programs,
down about 25% from $400,000 in 2009, with about 60% of it going to
contractors or agencies. Companies with less than $1 billion in revenue (50% of
all respondents) budgeted just $133,000 on average for reference programs,
down 20% from $166,000 in 2009, with about half of that going to third-party
contractors. (Note: Total budget figures do not include FTEs.)
Reference programs at the smallest companies (less than $1 billion in revenue)
were staffed with an average of 1.4 full-time employees and almost no full-time
contractors ("external FTEs"). For the next two tiers of companies ($1-$10 billion
and $10+ billion in revenue), reference programs were predictably larger,
employing about five and four internal FTEs respectively, and three and nine
external FTEs. Most notable was that large companies relied more on external
resources while using about the same number of internal resources.
6

Most of the programs surveyed
(78%) rely on outside contractors
or agencies for common services
such as writing.
7
Outsourcing
Most of the programs surveyed (78%) rely on outside contractors or agencies for
common services like writing. Similar to last year, respondents said that the most
common services that programs this year outsourced were:
 Writing
 Creative services
 Content development
The survey showed a moderately strong correlation (75%) between company
revenue and the total number of functions handled by outside contractors and
agencies, with companies over $10 billion in revenue usually outsourcing at least
three functions. The ratio of third-party FTEs to total FTEs also changed
significantly from 2009, as smaller companies shifted toward relying more on
internal resources while larger companies increased their use of outside
resources.

There was widespread
agreement on the importance of
the top two performance
metrics—new customers in the
past year and new reference
requests per month—with
almost every respondent
regarding them as critical to
understanding the effectiveness
of reference programs.
8
TOP PERFORMANCE METRICS
What are the key performance indicators for customer reference programs?
Survey respondents rated the relative importance of more than 40 performance
metrics and identified the top 10 (shown in Figure 9).
2010 witnessed a significant shift, with more programs emphasizing the
importance of “effectiveness and efficiency” metrics, such as the number of new
reference customers recruited, the number of new requests per month, and lead
time to fulfill a reference request. By contrast, in 2009 three of the top four
metrics were focused on how well references programs were impacting sales.
Financial measures such as the annual program budget and the number of
internal FTEs and contractors were also judged to be more important this year,
possibly due to tighter budgets and the general macroeconomic environment.
There was widespread agreement on the importance of the top two performance
metrics—the number of new customers recruited in the past year and new
requests per month—with almost every respondent regarding these measures as
critical to understanding the effectiveness of reference programs.
Respondents were mixed in their opinion of the value of other metrics, such as
the number internal FTEs, annual budget, and customer retention rate. Larger
companies (>$10B revenue) claimed that these metrics were very important
while smaller companies (<$1B revenue) considered them less critical.
Significantly, response rates to the performance metric question surged in 2010,
with 70% of respondents choosing to rank the importance of these metrics
compared to only 50% last year. As a result, the 2010 data can be considered a
more accurate reflection of the opinion of program managers regarding these
metrics.
 9

Key "downstream” performance metrics—such as total sales-contract value, lead

conversion rates, or internal customer satisfaction—showed little to no gains in
importance this year, but still ranked as one of the critical key metrics among
program managers. (Indeed, top performing programs in the survey were more
likely to measure impact on sales.)

Tracking downstream performance, however, remains a significant organizational

challenge, with about 44% of respondents saying they do not track sales-impact
data. Overcoming this challenge and gaining greater visibility into downstream
performance will be necessary if managers expect reference programs to
become stronger and more relevant partners in driving business value.

By contrast, program managers in 2010 reported being more focused on

“upstream” performance measurements—that is, metrics that indicate future
reference operational performance and productivity, such as new customers
recruited in the past year and new requests per month.

In any business organization, an abundance of metrics can be tracked to

measure performance. The challenge is to determine which metrics hold specific
value for the organization and to figure out what to do with the data once it’s
gathered. This year, the biggest “value-measurement gap”—that is, when
managers consider a performance metric important but fail to measure it—was in
the category of total contract value of deals influenced by references. Only 28%
of programs could measure this metric, although that figure is an improvement
over last year’s.

If government and education are
combined, more reference
customers are from the public
sector than any other industry.
Financial services topped the list
of private-sector industries that
generate customer references.
10
REFERENCE MATERIALS: TYPES, INDUSTRIES, AND
GEOGRAPHIES COVERED
The reference programs studied produced a wide spectrum of customer-focused
collateral and managed a high volume of "live referencing" activities, including
phone calls and meetings with prospects, analysts, and the media. In terms of
physical reference materials produced by these programs, the top three (by
quantity) were case studies, testimonials, and presentations. As was the case
last year, multimedia assets—including videos and podcasts—were the second
most commonly used asset, but they ranked fifth in terms of the amount
produced (about one-quarter as many multimedia assets were created as case
studies). Thus, while videos and audio are in high demand, time and cost
requirements remain limiting factors to boosting quantity.
As in 2009, program managers in 2010 said that they get more references from
the public sector than any other industry (“public sector” in this study combines
both government and education). If public sector is broken down into government
and education, financial services and healthcare were the top sources of
reference customers, with 40% of respondents listing financial services among
the top three industries represented. While the list of top industries stayed
somewhat similar, healthcare gained more than other industries.
Figure 12. Top Industries Covered (With Public Sector Broken Up)
1. Financial Services
2. Healthcare
3. Government
4. Manufacturing
5. Education

Respondents in 2010 had a
higher proportion of reference
customers in North America
compared to 2009, a trend that
may have been driven by a
greater emphasis on increasing
the volume of new reference
customers and the greater
proportion of smaller companies
in the 2010 survey pool.
11
The majority of “studies” produced by reference programs—including customer
case studies, success stories, and white papers—featured customers operating
in North America and EMEA (Europe, Middle East, and Africa). This year, more
studies shifted to North America, with customers from this region generating 65%
of all studies compared to 45% last year. The shift toward North America could
be explained by the greater proportion of smaller (<$1B) companies in the survey
pool this year (about 50% vs. 33% in 2009); on average, smaller companies in
the CRF survey have a much higher concentration (about 90%) of North
American reference customers, likely due to the proximity and greater proportion
of domestic business for smaller companies.
However, the majority of respondents this year continued to say they were less
concerned about the customer's location than about getting the right mix of
industries represented in their reference programs. About 28% of programs rated
geographical distribution more important—double the 2009 percentage—but
industry relevance remains more important than geographical proximity to the
target prospect or customer.
HOW CUSTOMER REFERENCES ARE USED
Like last year, reference managers surveyed in 2010 appear to have only limited
knowledge of whether referencing activities contribute to closing deals. In fact,
approximately a quarter of the respondents were unable to say even how often
references were being used as part of proposals or bids. The remaining
respondents, however, said that between 30% and 100% of bids were
accompanied by reference materials or activities, with the top three-quarters of
the respondents stating that references were used more than 75% of the time,
about the same frequency as in 2009 (see Figure 14).
In 2010, program managers placed more importance on the sheer number of
reference requests and reference customers, suggesting that the primary focus
was on increasing service levels and embedding references within the sales
process. With the squeeze on budgets and FTEs, expanding the use of
references as well as the available pool of reference customers may have helped
justify retaining as much budget and resources as possible for these programs.
 12

There continues to be a wide variance in the volume of reference requests that

For the most part, reference
managers appear to have only
limited knowledge of the extent
to which referencing activities
contribute to closing deals. While
programs must field, ranging from two per month to as many as 350. Excluding
outliers, the average number of reference requests per month increased from
22.4 to 37.7. Additionally, the median for the average number of reference
requests per month from 2009 to 2010 increased from 20 to 35, a 75% increase.
Among the responses, company size was the primary factor influencing the
number of reference requests, with companies under $1 billion in revenue
receiving about 18 reference requests per month on average, and companies
over $1 billion in revenue receiving about 66 requests per month.

total contract value influenced by

references was one of the most

important metrics for

respondents, less than 30% of

programs are actually able to

measure that data point.


Audio/video assets, while popular
among stakeholders, are still
produced in relatively low
quantity. However, among the
repeat survey respondents, the
average number of videos
increased from 18.6 to 32.0,
showing that programs are
making significant steps to boost
multimedia production.
13
What kinds of reference materials do stakeholders most often use? Again, limited
"downstream" visibility makes it hard to get an accurate picture, but respondents
said the following materials, on average, were used most frequently (in rank
order):
1. Case studies
2. Audio/Video
3. Product information sheets
4. Press release
5. Presentations
The usage list correlates well with the types of materials currently being
produced by programs. Case studies, for example, are both heavily used by
stakeholders and produced in large quantity. However, press releases and audio-
video assets, while popular among stakeholders, are produced in relatively low
quantity compared to case studies and presentations.
RESPONSIVENESS TO REFERENCE REQUESTS
Responding to reference requests in a timely manner and fulfilling a high
percentage of requests are both widely seen as two of the most important
reference program performance metrics. This year, a majority of the respondents
to the survey said they fulfill a very high percentage of reference requests, with
three-quarters saying they fulfill more than 90% on average. Top programs also
excelled at turnaround time in 2010, responding to requests within a few days on
average (a slight increase over 2009). Overall, respondents large and small
reported fulfillment rates of between 75% and 100%, and response times within
one week. There was also a moderately strong correlation (-0.72) between these
two metrics, indicating that programs that fulfill a high percentage of requests
also take less time to fulfill each request.

Stakeholders are generally
satisfied with reference program
services, but see room for
improvement. On average,
reference managers rated the
satisfaction level of internal
customers at 3.78 out of 5.
14
CUSTOMER PARTICIPATION AND SATISFACTION
Survey respondents this year reported almost no difference between the
customer’s willingness to participate in case studies (or other reference
materials) and "live references" such as calls and meetings. This marks a change
from 2009, when customer “acceptance rates” were significantly different
between referencing types, with much higher acceptance rates for reference
materials.
The top quarter of programs surveyed said that customers agreed to produce
reference materials over 48% of the time, and overall customer participation
rates ranged between 10% and 100%. Similarly, for live references, the top
quarter of programs succeeded in gaining customer participation about 47% of
the time, and overall participation rates ranged between 15% and 90%. By
contrast, in 2009 respondents said that customers agreed to participate in live
references only 25% of the time.
Figure 18. Customer Acceptance Rates
 15

Figure 19. Average Number of New References

Once recruited, customers were very likely to stay with reference programs from
year to year, respondents said. The top three-quarters of programs surveyed
achieved reference customer retention rates of at least 89%. However, about 20-
30% fewer new customers joined reference programs this year compared to last,
as shown in Figure 20. Tighter budgets and program resources may explain the
decline. Still, demand for references kept rising this year, as companies saw the
number of monthly reference requests for references go up by 68%.
 16

CONCLUSION
When companies invest in customer success, they create loyalty and advocacy
that sets them apart from the competition. As markets become more globalized
and competitive—and as buyers become more sophisticated and demanding—
the unique selling power of customer references takes on even greater
importance.

This survey found that reference managers are keenly interested in tracking and
boosting program performance by focusing more on effectiveness and efficiency,
especially as the tough economy forces companies to shrink budgets and put
greater importance on managing resources wisely. The average demand for new
reference requests per month increased since 2009, which is evidence that
further supports the importance of references.

However, there was a significant drop in the average number of new customers
joining reference programs, a trend that is likely attributable to the growing
demand for reference requests and the decrease in budgets and staff, resulting
in more time spent on maintaining operations and fulfilling those reference
requests, rather than on new initiatives to recruit more reference customers and
create new assets.

As the economy bounces back in the years ahead, it remains to be seen whether
reference programs will regain the budgets and resources needed to meet the
growing demand for references and to create new value by integrating programs
more tightly with the sales process. Opportunities for value creation also await
programs that can expand relationships with reference customers and delve
deeper into high-demand multimedia assets.

Reference programs have the unique power to gather and publicize valuable
customer intelligence. Often they are the strongest link between the business
and the customer, and one of the few post-sales organizations positioned to
positively impact the customer relationship. This year’s increase in customer
participation rates suggests that programs are fostering better relationships with
customers—and that could be good news for overall customer satisfaction and
loyalty.

While this survey focused more on the quantitative aspects of customer

reference programs, it does leave out some of the qualitative aspects, such as
the quality of the reference assets and customers.

Larger reference programs may hold the edge here, since they’re better able to
fund more sophisticated collateral and multimedia productions—often through
strategic outsourcing—as well as attract brand-name customers. Not surprisingly,
reference programs at the largest companies were found to outsource more
reference functions, especially in creative fields.

Program managers also want to better understand and monitor the impact
referencing has on sales and marketing, but as this survey confirms once again,
many are limited in their ability to consistently track this metric. While in 2009,
increasing the "downstream" visibility was a priority for programs, in 2010,
 17

reference managers were more concerned with growing the reach of their
programs, with increasing efficiency and throughput.

This report marks the second annual survey of reference programs designed to
establish a set of reliable metrics and industry benchmarks for evaluating
program performance and success. In subsequent reports, the Metrics SIG will
expand its survey to incorporate more programs, further refine the list of
performance measures, and continue to add to a growing database of results
and metrics from survey participants.

For more information, contact:

Abby Atkinson, chairperson, Metrics SIG

abby.atkinson@infor.com

Amir Hartman, Metrics SIG coordinator

ahartman@mainstaypartners.net