User Survey Analysis: Customers Rate Their CPM Vendors, 2012

4 June 2013 ID:G00247384 Analyst(s): Christopher Iervolino, John E. Van Decker

VIEW SUMMARY Gartner's survey of 275 customer references from 13 corporate performance management vendors between September and November 2012 has produced valuable insights and benchmarking data for IT leaders assessing CPM solutions. Don't just focus on the megavendors when evaluating prospective providers.

Overview
Key Findings

Among the megavendors, SAP has improved overall satisfaction ratings in relation to Oracle, and IBM has maintained an above-average overall rating; however, survey results highlight room for improvement for all three providers. Annualized cost comparisons indicate that software-as-a-service solutions are not significantly less expensive than many traditional on-premises offerings. For both options, business value attained and total cost of ownership should primarily guide buying decisions. Results indicate distinctive customer satisfaction differences among corporate performance management vendors in vendor-, product- and implementation-related scores. Statistics related to average number of users per implementation, popularity of modules, length of time used and cost also characterize vendor solutions. Many specialist CPM vendors score consistently higher than the megavendors, which shows they have credible solutions, despite their relatively small size and market share.

Recommendations

Extend product evaluations past the assessment of individual functions, and consider the cost and complexity of implementations and ongoing use, as well as the overall value of the vendor relationship. Conduct cost comparisons of SaaS versus on-premises solutions that consider a wide range of TCO factors, including ongoing consulting, IT support and upgrade expenditures. Consider specialist CPM vendors in assessments, and do not focus only on the megavendors.

TABLE OF CONTENTS CONTENTS

Survey Objective Data Insights

o o o o o o
TABLES

Vendor Experience Ratings Product Experience Ratings Implementation Experience Ratings Solution Usage Comparisons Solution Cost Comparisons Methodology

Table 1. Response by Vendor Customer Locations Table 2. Estimated Number of Years Solution Has Been in Use

FIGURES
Figure 1. Overall Satisfaction Figure 2. Overall Satisfaction Versus Average Implementation Size Figure 3. Vendor Experience Figure 4. Overall Satisfaction Versus the Top-Three Differentiating Vendor Attributes Figure 5. Product Experience Figure 6. Overall Satisfaction Versus the Top-Three Differentiating Product Attributes Figure 7. Implementation Ratings Figure 8. Overall Satisfaction Versus the Top-Three Differentiating Implementation Attributes Figure 9. Strategic CPM Usage Figure 10. Module Usage Figure 11. Cost Characteristics: Total Cost to Date Figure 12. Annual Cost Characteristics Figure 13. Organization Size (Number of Employees) Figure 14. Geographic Composition Figure 15. Industry Composition

Survey Objective
This document was revised on 6 June 2013. For more information, see the Corrections page.

Toward the end of 2012, Gartner conducted its annual corporate performance management (CPM) Magic Quadrant survey.1 In that survey, 275 CPM customers from 13 different vendors shared valuable insights for companies selecting CPM vendors and for customers benchmarking suppliers. The detailed results provide additional insights to support prospective customers in fine-tuning their RFPs, product selection criteria and project plan approaches for implementations and upgrades. This survey comprised a key component of the CPM Suite Magic Quadrant report (see "Magic Quadrant for Corporate Performance Management Suites"). The analysis of these detailed results is an important complement to that study.
Table of Contents

Data Insights
This research reveals customer satisfaction ratings as they relate to vendor, product and implementation attributes. It uses the three scores from each category with the greatest variances among vendors to highlight the most-differentiating factors. It then evaluates responses alongside the average number of users per implementation, and the levels of use for each module in the vendors' product suites. This research also sheds light on other differentiating solution cost characteristics. Table 1 shows the number of responses by the vendor and describes their customer locations. Table 1. Response by Vendor Customer Locations
Vendor
IBM Prophix Board International Host Analytics Longview Oracle Tagetik Bitam KCI Computing SAP SAS Infor prevero

No. of Survey Participants

30 29 26 24 24 23 21 19 17 17 17 15 13

Customer Locations
Worldwide Mainly North America, but also worldwide Mainly EMEA, but also worldwide Mainly North America, but also worldwide Mainly North America, but also worldwide Worldwide Mainly EMEA, but also worldwide Mainly North and South America, but also worldwide Mainly North America, but also worldwide Worldwide Worldwide Worldwide Mainly EMEA, but also worldwide

Source: Gartner (June 2013)

Table of Contents

Overall, most survey participants were satisfied with their CPM vendors (see Figure 1). On a scale of 1 to 7, the average rating was 6.09. This favorable rating, which is higher than those in previous years, indicates that CPM solutions are mature and are delivering value. However, the standard deviation was fairly high (0.94), which reflects a greater variation in scores, especially for KCI

Computing (standard deviation = 1.46) and prevero (standard deviation = 1.38). Most of the customers chosen by the vendors were satisfied with their solutions; however, the level of variability underscores the importance of matching the correct CPM vendor, product and implementation capabilities to a given set of organizational needs. Figure 1. Overall Satisfaction

The rating is an average of respondents' overall aggregate score by vendor. The figure represents customer perceptions, not Gartner's opinion. The graph may feature vendors that in Gartner's opinion don't deliver the benefits described (N = 275). Source: Gartner (June 2013)
Table of Contents

Although it's reasonable to assume that vendors with smaller implementations and few users would consistently score higher in questions related to vendor support, these survey results do not wholly support this. As in 2011, vendors Tagetik and Longview score above-average satisfaction ratings and have an above-average number of users per implementation. IBM also scored above average in both categories; however, Tagetik and Longview's higher scores are standouts in this area (see Figure 2). Figure 2. Overall Satisfaction Versus Average Implementation Size

Source: Gartner (June 2013)

Table of Contents

The overall customer satisfaction ratings are an aggregation of vendor, product and implementation scores. To gain insights into these overall ratings, it's necessary to drill down into more-detailed survey results that illustrate more-specific strengths and weaknesses of each vendor.
Table of Contents

Vendor Experience Ratings

The following information highlights results related to the vendor experience, such as sales, support and ongoing communication. The survey asked this question regarding the vendor experience: "Compared with experiences with other vendors, how was the experience with this vendor on the following attributes?"

Ongoing support and after-sales care Overall value for the money Ongoing communication Responsiveness of telephone support Experience throughout the sales process Additional implementation and deployment Ease of applying upgrades, fixes and patches Professional service implementation, and deployment training and handover

The attributes that resulted in the largest variations between high and low scores have the biggest impact on the overall client satisfaction score, and contribute most to differentiating the vendors within this category. The three attributes with the greatest variances, and their respective scores for each vendor, are represented in Figure 3. The question asked was, "Compared with experiences with other vendors, how was the experience with this vendor on the following attributes?" Figure 3. Vendor Experience

The figure represents customer perceptions, not Gartner's opinion, and may feature vendors that in Gartner's opinion don't deliver the functional capability described (N = 275). Source: Gartner (June 2013)
Table of Contents

Figure 4 shows overall customer satisfaction, compared with the top vendor-differentiating attributes.

Figure 4. Overall Satisfaction Versus the Top-Three Differentiating Vendor Attributes

The figure represents customer perceptions, not Gartner's opinion, and may feature vendors that in Gartner's opinion don't deliver the functional capability described (N = 275). Source: Gartner (June 2013)
Table of Contents

Similar to last year, Longview and Tagetik scored well in this category, and Board International's rating is also on a par with these vendors. Of the three, Tagetik's high rating stands out, given its above-average revenue growth rate of 25.7% in 2012. Host Analytics, which had ranked among these vendors in previous years, scored the highest in 2012. As a pure software as a service (SaaS) vendor, it naturally scored the highest rating for "ease of applying upgrades/patches/fixes"; however, it also scored highest on the other two differentiating ratings, including the important "overall value for the money" rating. Host Analytics has achieved this score during a period of high revenue growth (78.3%), which indicates its growing organizational maturity. (All vendor revenue growth figures can be found in "Market Share Analysis: Business Intelligence, Analytics and Performance Management, Worldwide, 2012.") The largest variation of all vendor, product and implementation ratings existed for "ease of applying upgrades/patches/fixes." This suggests that the greatest single difference between the vendors, from the customer's perspective, relates to upgrades, which encompasses ongoing maintenance and support cost sensitivity, as well as the desire for new product features available in more-current software versions. Consistent with previous years, all three megavendors (IBM, Oracle and SAP) scored below average in this vendor-specific category. Although Oracle scored the lowest in average, it also had the lowest "overall value for the money" rating. This low rating may have been affected by the increased awareness of less-expensive CPM options in the marketplace. Slowed CPM market

growth figures for 2012 seem to indicate that additional cost pressures are affecting CPM vendors differently; for example subscription-based options, such as Host Analytics, have increased their revenue in 2012 at higher-than-average rates. On the whole, Oracle's average score was most affected by "responsiveness of telephone support," where it received the lowest score of any other vendor rating. All three of the highest variations in overall vendor rating are related to cost. Furthermore, the largest variations occurred for "ease of applying upgrades/fixes/patches" and "responsiveness of telephone support." These ratings relate to ongoing solution effort and cost, indicating a growing gap in customer sentiment related to ease of use and cost considerations among the different vendors. Prospective customers should note the correlation of these vendor-specific criteria to overall customer satisfaction. The horizontal distance from average is an indication of how these vendor-specific factors have affected their overall customer satisfaction ratings.
Table of Contents

Product Experience Ratings

The survey asked the following question regarding product experience: "To what extent has the organization's vendor CPM solution met the following requirements?"

Improved the quality, accuracy and timeliness of financial and management reporting Added more sophistication and confidence to the budgeting, planning and forecasting (BP&F) processes Reduced the effort to produce budgets, and reduced the budget cycle time Allowed business users to get the information they need, with reduced reliance on IT Provided a better understanding of current performance Allowed a better prediction of financial and business performance Provided a linkage between strategic planning, financial budgets and operational activity Allowed a better understanding of profitability drivers Improved the ability to meet compliance requirements, including external disclosure

As with responses related to vendor experience, the attributes that resulted in the largest variations between high and low scores have the biggest impact on the overall client satisfaction score, and contribute most to differentiating the vendors within this product experience category. The three attributes with the greatest variances in this category and their respective scores for each vendor are represented in Figure 5. The question asked was, "To what extent has the organization's vendor CPM solution met the following requirements?" Figure 5. Product Experience

Source: Gartner (June 2013)

Table of Contents

Compared with last year's results, Host Analytics and Board International scored notably higher. This indicates a growing satisfaction with available product functionality in these areas and an increased sophistication of use for these vendors' solutions (see "User Survey Analysis: Customers Rate Their Corporate Performance Management Vendors"). These ratings indicate that the widest range of end-user satisfaction exists for some key product functionality compliance, giving insight into the drivers of profitability and providing a linkage between strategic financial planning, budgeting and operational planning. The rating for Host Analytics is somewhat surprising in that it is newer to the market than most of the others. A reasonable assumption is that Host Analytics does not have the breath of functionality of some of the more widely used solutions that have been around longer; however, these ratings are a measure of satisfaction with ease of use (accessibility of the functionality), as well as satisfaction with the functionality itself (robustness of the functionality). Vendors that scored well in this category are likely to be able to attribute their

success to customer satisfaction with a combination of feature accessibility and capability (see Figure 6). Figure 6. Overall Satisfaction Versus the Top-Three Differentiating Product Attributes

Source: Gartner (June 2013)

Table of Contents

The horizontal distance from average is an indication of how these product-specific factors have affected their overall customer satisfaction ratings. Of the three satisfaction categories surveyed (vendor, product and implementation), the most narrow range of scores exists for product ratings (between 4.63 and 5.72). That is, vendor- and implementation-related areas had the greatest impact on overall satisfaction scores. This indicates that customers are more satisfied with the product-specific functionality their solutions provide than with vendor- and implementation-specific items. As a result, prospective customers should ensure that they evaluate new solutions from a product functionality perspective, as well as in terms of information gathered from vendor references regarding their vendor and implementation experiences.
Table of Contents

Implementation Experience Ratings

The survey requested the following information regarding each customer's implementation experience by requesting its level of agreement with the following statements about the organization's implementation of the vendor's CPM solution:

The vendor's professional service staff had the skills and knowledge to make the project a success.

The CPM applications delivered the functionality expected. The anticipated business benefits from implementing the CPM applications were realized. It was easy to adapt the CPM applications to meet business requirements over time. Users found the system easy to use and manage. IT finds the CPM application technology straightforward to maintain and support. The solution gained wider acceptance than expected. The implementation was easier than expected. The implementation was less expensive than expected.

Because the attributes that resulted in the largest variations between high and low scores have the biggest impact on the overall client satisfaction score, and contribute most to differentiating the vendors within this product-experience-related category, the three attributes with the greatest variances within this category and their respective scores for each vendor are examined in Figure 7. The request was, "Indicate agreement with the following statements about the organization's implementation of the vendor's CPM solution." Figure 7. Implementation Ratings

Source: Gartner (June 2013)

Table of Contents

Some of these ratings are expected for example, it's natural that Host Analytics would score highest for ease of IT maintenance and support, because it's a SaaS solution. Interestingly, Board, which does not offer a SaaS solution, and Tagetik, whose customer respondents did not use its cloud-based offering, were not far behind. This shows that CPM apps are different from ERP and other business applications that require heavy IT support; IT complexity should not be a forgone conclusion. Another significant expectation-related result here is that every vendor had its lowest rating for the item, "our implementation was cheaper than expected." Although installation costs varied among the vendors, this indicates a consistent set of false expectations with CPM solutions. This may occur during the sales or the implementation scoping or a combination of the two. This result also reflects the importance of cost as a satisfaction factor. Prospective customers should ensure they include a rigorous customer reference effort that includes an exploration of consulting and support fees as a necessary component of any CPM purchase or project scoping exercise (see Figure 8). Figure 8. Overall Satisfaction Versus the Top-Three Differentiating Implementation Attributes

Source: Gartner (June 2013)

Table of Contents

The tight diagonal arrangement indicates close relationship between implementation and satisfaction. This is correlation is highest for these implementation ratings, underscoring the importance of considering implementation project needs alongside the software purchase when selecting a particular vendor's solution.

Table of Contents

Solution Usage Comparisons

Although most CPM implementations take advantage of office-of-finance CPM capability initially, organizations can realize significant improvements by taking advantage of strategic CPM. (see "Getting More Value From CPM: Strategic Versus Office-of-Finance CPM"). This graphic is one indication of the degree to which different vendor's solutions are used strategically. Figure 9 illustrates the relationship between overall satisfaction and two key metrics that indicate a morestrategic use of each vendor's CPM solution. The first is the average number of users per implementation, and the second is module use, which is represented by circle size (more information regarding module use can be found in Figure 10). Figure 9. Strategic CPM Usage

Source: Gartner (June 2013)

Table of Contents

Figure 10. Module Usage

Source: Gartner (June 2013)

Table of Contents

Those vendors with a larger number of users per implementation (distance to the right) and a larger circle in Figure 9 (more widely used set of CPM modules) indicate more strategic CPM use. Those that have also scored high levels of satisfaction (vertical position) are able to do so with more-complex installations that support a wider number of CPM processes. How many CPM modules are used is an indication of the popularity of the each of the vendors' modules. The survey asked the question, "How many users does the organization currently have for each of the following features of the vendor's CPM solution?" The answers were converted into percentages of the number of overall users for each vendor to provide consistent comparison mechanism among vendors. Total scores were ranked from highest to lowest. This information was then compared with the average number of users per implementation, another measure of CPM maturity, as well as the average customer satisfaction rating. By comparing how extensively the product is used (by estimating the modules used and number of users) the average level of CPM maturity can be estimated. Furthermore, by comparing these statistics against overall satisfaction, the ease in which the solution allows for additional CPM maturity for implementations of different sizes can be roughly estimated.

Although the customer sample size is small, compared with the total number of customers (this is especially true for IBM, Oracle and SAP), the survey data indicates patterns of usage among survey respondents, such as:

IBM's solutions were reported to be used for a broader set of CPM processes than were SAP's (and somewhat more than Oracle's as well). IBM also had a higher overall satisfaction rating than either Oracle or SAP. Longview and Tagetik were able to maintain high customer satisfaction ratings, even with a high number of average users. Also, a lower percentage of Tagetik customer respondents use their applications for BP&F, as compared with Longview's. Prophix, Board International, SAS Institute and Host Analytics respondents support a wide array of CPM processes, although each does so for implementations of different average size and, most likely, of varied complexity.

Table of Contents

Solution Cost Comparisons

Respondents were asked to provide the approximate total cost to date of their organizations' CPM solutions for each of five cost categories. This included subscription fees; external services (including consulting, system integration, managed services, hosting, and outsourced process); hardware; software; and internal labor and head count. Because IBM and KCI Computing each had one customer response with very high costs, median results were used, instead of averages, to reduce the impact of these two outliers. The breakdown of costs by vendor is shown in Figure 11. Figure 11. Cost Characteristics: Total Cost to Date

Source: Gartner (June 2013)

Table of Contents

These results clearly indicated that Oracle's CPM implementations are the most expensive, especially for external services, among survey respondents. However, Oracle's CPM solutions have, on average, been used for longer periods of time. Table 2 shows the median number of years that respondents reported their solution has been in use (if they have multiple solutions from the vendor, the solution that was used the longest was reported on). Table 2. Estimated Number of Years Solution Has Been in Use
Vendor
Oracle Longview SAS Institute

Years Used
8 7 6

Table 2. Estimated Number of Years Solution Has Been in Use

Vendor
IBM KCI Computing Bitam Board International Infor prevero Tagetik SAP Host Analytics Prophix

Years Used
5 5 4 4 3 3 3 2.5 2 2

Source: Gartner (June 2013)

Table of Contents

Clearly, vendors with customers that have used their solutions for longer periods of time (such as those using Longview and SAS Institute's solutions) now show lower comparable costs, because they have had more years over which to spread initial software purchase and implementation costs. Others that, on average, have relatively new installations, such as those using SAP's and Host Analytics' solutions, have higher per-year costs. Oracle remains the most expensive, both on a total and per year basis; however, it's important to consider that Oracle's customer respondents also estimated a relatively high number of average users. On a per-user basis Oracle's average costs per year are lower, as are Longview's and Tagetik's. These annualized cost comparisons also illustrate ongoing costs over time for example, Figure 12 highlights the higher annual cost of external services for both SAP and Oracle. Figure 12. Annual Cost Characteristics

Source: Gartner (June 2013)

Table of Contents

Host Analytics, a SaaS vendor, has a relatively low initial cost; however, when these costs are annualized, they are higher than many on-premises vendors. Also, because Host Analytics' respondents reported a relatively low number of users, costs per year, per user are also higher relative to many of the other solutions. Regarding SaaS versus on-premises cost comparisons, this survey was generally completed by finance users and internal labor and head count estimates may not always reflect accurate ongoing IT support costs. In addition, ongoing costs, such as software upgrades, are generally built into multitenant SaaS vendor costs; such is the case with Host Analytics. However, these results suggest that prospective customers should look past short-term costs and perform on-premises versus SaaS cost comparisons from a total cost of ownership (TCO) perspective over the anticipated life of the solution. Customers should consider all cost considerations including ongoing internal and external consulting expenses and both long-term subscription and license fees when selecting a CPM vendor.

On the whole, this more-detailed analysis of survey results illustrates the credibility of smaller CPM vendors across a number of areas for larger, more-complex CPM solutions or more-focused ones. A number of these vendors consistently score high marks in key satisfaction areas, such as overall value for the money and ease in adapting the solution for changing business requirements. Despite the consistent leader ratings of the megavendors in the Magic Quadrant, smaller vendors are highly credible options for many CPM needs and should be considered in a wide array of product evaluations. In addition, survey results regarding average implementation size, modules used, length of use and cost can help determine correct product fit and guide product selection processes.
Table of Contents

Methodology
As part of our research for the CPM Magic Quadrant, Gartner conducted an English-language Web survey of 275 CPM suite customers (up from 233 in 2011). The individuals surveyed were nominated by CPM vendors at Gartner's request (Magic Quadrant methodology mandates gathering data from vendor customers). The survey was composed of 21 questions regarding customer experiences in working with vendors. Gartner requested contacts knowledgeable about how the vendors' CPM products were used by customer organizations. The vendors did not see the questionnaire before the customers were nominated, and responders were made aware that their answers would remain strictly anonymous. The survey took 20 minutes and covered the customer's use of the nominating vendor's CPM solution. Gartner expected that, because the survey participants were vendor reference customers, the results would reflect a more positive experience of using and implementing each vendor's CPM solutions. However, the relative comparison of results is valid, since all respondents were nominated by their respective vendors. Although representative of this group, and providing useful indications of customer experiences and deployment characteristics, the survey results may not reflect the views of the general population using these products. Cost characteristics were calculated by converting rates to U.S. dollars as of the last day of the survey, 13 November 2012. Median results were then selected to reduce the effect of outliers. Size, geography and industry characteristics are shown in Figure 13, Figure 14 and Figure 15, respectively. Figure 13. Organization Size (Number of Employees)

Source: Gartner (June 2013)

Table of Contents

Figure 14. Geographic Composition

Source: Gartner (June 2013)

Table of Contents

Figure 15. Industry Composition

Source: Gartner (June 2013)

Table of Contents

EVIDENCE
1

This annual survey was conducted from early September until the middle of November 2012. A total of 275 completed surveys were returned. Additional details regarding this survey can be found in "Magic Quadrant for Corporate Performance Management Suites."

2013 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This publication may not be reproduced or distributed in any form without Gartners prior written permission. If you are authorized to access this publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This publication consists of the opinions of Gartners research organization and should not be construed as statements of fact. The opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research.

Gartners Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see Guiding Principles on Independence and Objectivity. About Gartner | Careers | Newsroom | Policies | Site Index | IT Glossary | Contact Gartner